Archives for: December 2005

12/31/05

Permalink 01:37:21 pm, Categories: Python, 496 words   English (US)

Making a text block flush left

Every now and then I come across some really nifty Python code. Terseness is really not a good virtue of Python code, but I just thought this snippet was very clever. It comes from a really nice online book called Text Processing in Python. The problem deals with flushing a code block to the left, keeping all other indentation the same. In other words, this:

   if blah:
      print 'hi'

gets changed to:

if blah:
   print 'hi'

Simple enough huh? Here is their solution:

from re import findall,sub
# What is the minimum line indentation of a block?
indent = lambda s: reduce(min,map(len,
   findall('(?m)^ *(?=\S)',s)))
# Remove the block-minimum indentation from each line?
flush_left = lambda s: sub('(?m)^ {%d}' % indent(s),'',s)

lambdas are small one-line functions in Python. This text talks alot about 'high order functions' which are basically function generators, and they make extensive use of lambdas. The above creates 2 functions stored in the variables indent and flush_left. flush_left would be called on the text to be processed.

First lets look at 'indent'. Essentially this function is trying to determine the minumum line indentation of a block to remove. It would result in 3 for my example. It's kinda confusing unless you break it down. findall is the innermost function, and it finds all regular expression matches for:

(?m)^ *(?=\S)

The '(?m)' is adding the multiline option to this regexp, so that it would span multiple lines. Then the regexp looks for 1 or more spaces at the beginning of a line given a lookahead assertion that there is a non-whitespace character right after the last space. Whew. So this will return a bunch of strings with some number of spaces.

Next is the call to map(). This function applies a certain function (in this case len()) to a list and returns that list. So the result of the map() call is a list of numbers which are the lengths of leftmost whitespace.

The next call is reduce(), which applies a certain function to pairs of elements in a list, returning one result. Essentially it reduces. In this case it is applying min(), which returns the minimum value. So the final result is the smallest number of leftmost spaces. This whole operation is assigned a lambda function called indent() that takes one argument, the block of code being analyzed.

Next is the flush_left() function:

flush_left = lambda s: sub('(?m)^ {%d}' % 
   indent(s),'',s)

What does this do? Well it does a regexp substitution, replacing '%d' number of spaces at the beginning of a line with nothing. %d is calculated by the indent() function defined above, which is the minimum number of spaces to remove in order to flush everything to the left.

Pretty nifty, but hard to understand at first glance. I'm not sure if I'd ever write such code, but it is fun to analyze. Obviously this wouldn't work with tab-indentation, but a few modifications and it would.

12/30/05

Permalink 11:24:03 am, Categories: Home, 160 words   English (US)

Moving

The movers came out this week to take my stuff as well as my car. There were 3 moving guys and they did a pretty good job, considering I had relatively very little to move.

One of the movers had broke something on my screen door while taking a sofa out. I mentioned this to the lead mover and he says to me with a straight face, "I have a 45 in the truck." At first I was like, 45? What does he mean? I didn't know what he was talking about. Then he says, "If you want to aim, I can pull the trigger." At that point I realized ohhh, he's talking about a gun! Hehehe, well at least he had a sense of humor. I found out that he likes busting his coworkers balls'.

Anyway, they were able to fix the door gaffe, so no harm done. I'll be on my way Jan 2nd to the other side of the country.

12/29/05

Permalink 11:28:30 pm, Categories: Television, 58 words   English (US)

The Sopranos downfall

I had a chance to see an episode from the first season of The Sopranos and realized how much anything after season 1 sucked. It was just so much more interesting back then. Now it's just a very silly soap opera. It seems like the writing completely changed. I wouldn't be surprised if it's now written by new authors.

Permalink 07:25:45 pm, Categories: Web, 5 words   English (US)

Translator

Neat little Ajax real-time translator.

Permalink 10:28:06 am, Categories: Science, 15 words   English (US)

Running in rain good?

Ever wondered whether running in the rain helps reduce your wetness? Check out this analysis.

12/28/05

Permalink 11:32:57 pm, Categories: Linux, 636 words   English (US)

Diagnosing MySQL load

Recently I looked at a client webserver where there were lots of MySQL processes eating up quite a bit of CPU. It's a fairly high traffic site, however when I see mysqld processes staying at 80-90% for more than a few seconds, that tells me something is wrong. It's usually an indication that either they are using some very inefficient queries, do not have the proper indexes, or the tables are getting too large.

So let's see how to debug this. First thing is by the looks of the processes, there must be 1 or 2 queries that are pegging the CPU, and we need to find out what queries are taking long. I started up the mysql client and did a 'show status':

show status;
...
| Slow_queries             | 12256     |
...

Hmm, lot's of slow queries. but that number isn't increasing rapidly. I believe this shows very slow queries only. Next I did a 'show processlist' a few times until something showed up:

show processlist;
...
| 2721 | web  | localhost | development | Query   |    0 | Copying to tmp table |
 SELECT DISTINCT wheel.make FROM inventory LEFT JOIN wheel ON
 inventory.id=wheel.id WHERE status="act |
...

Usually, if you see a query on 'show processlist' consistently, that query is probably unoptimized and taking a long time. The above only showed me part of the query, and I needed to find the whole query to test it. Now I could go digging in the client's source code, but it would be a waste of time trying to find the culprit query. The best way is to enable MySQL logging. MySQL is such a lightweight DB, restarting it usually goes unnoticed even on high traffic webservers. I added to /etc/my.cnf:

[mysqld]
log=/var/log/mysqld/mysqld.log

And made sure to create that logfile writable by the mysql user (it won't log it if it doesn't have the right permissions). Next I find the query I'm looking for:

SELECT DISTINCT diameter
FROM inventory
LEFT JOIN wheel_fitment ON wheel_fitment.wheel=wheel.id
LEFT JOIN wheel ON wheel_fitment.wheel=wheel.id
WHERE status="active" AND vehicle=4566  AND price!=0 AND
wheel_fitment.type="all"
ORDER BY diameter

Ok, I'm not too familiar with the multiple left joins, but I do know what a join is. I run this query manually and it takes anywhere from 6 to 10 seconds to return results. That's definitely too long for a query to take, and the tables are not even that big! Now is when you try to take everything out of the query except the bare necessity that makes it slow. I ended up with:

SELECT DISTINCT diameter 
FROM inventory
LEFT JOIN wheel ON wheel_fitment.wheel=wheel.id
LEFT JOIN wheel_fitment ON wheel_fitment.wheel=wheel.id
WHERE vehicle=4566

Now I needed to know something about these tables. As sysadmins, we usually come in the picture when something has gone wrong, and we really played no part in the development of the system we are diagnosing. I wanted to know whether there were indexes on the tables. I did 'show create table wheel' as well as wheel_fitment and inventory. They all seemed to have proper indexes on the fields being used in the query. Hm what next? Well the double 'LEFT JOINs' looked kinda wierd to me, they are just swapped. The resulting set if I take out the 'DISTINCT' is something like 250k rows, much larger than all of the tables combined. Clearly something is wierd.

I checked the mysql join docs for some help, and determined that the following provides the same results:

SELECT DISTINCT diameter
FROM inventory LEFT JOIN (wheel, wheel_fitment)
ON (wheel_fitment.wheel=wheel.id)
WHERE vehicle=4566

Only now the query takes less than .3 seconds, quite an improvement from 10 seconds! My guess is that MySQL is better able to optimize the latter query.

12/27/05

Permalink 09:35:41 am, Categories: Fun, 17 words   English (US)

The Winning Ticket

Wonder what it's like to win the lottery? Well here's a very mean prank played on someone.

12/26/05

Permalink 04:59:22 pm, Categories: Fun, 11 words   English (US)

Rocket launcher

Here's a cool video of a camera strapped onto a rocket :).

12/25/05

Permalink 04:55:02 pm, Categories: Religion, 127 words   English (US)

Merry Christmas

A prayer:

Another year is dawning
With the chance to start anew.
May I be kinder, wiser, Lord,
In all I say and do,

Not so caught up in selfish gain.
That I would fail to see
The things in life that mean the most
Cost not a fancy fee.

The warm, kind word that I can give,
The outstretched hand to help,
The prayers I pray for those in need --
More precious these than wealth.

I know not what may lie ahead
Of laughter or of tears;
I only need to know each day
That You are walking near.

I'm thankful for this brand new year
As now I humbly pray,
My hand secure in Yours, dear Lord,
Each step along the way.

- by Kay Hoffman

Permalink 12:23:30 pm, Categories: Fun, 4 words   English (US)

Simply Fired

Some pretty hilarious videos.

Permalink 12:00:35 am, Categories: Music, 16 words   English (US)

Carpark North

Just heard this Dutch band's music on an IFC video. Very cool, checkout the 'Human' video.

12/24/05

Permalink 03:16:34 pm, Categories: Work, 28 words   English (US)

Time Management for Sysadmins

This is an interesting video about time management for sysadmins. Alot of it may be common sense, but there are some useful tips and it is presented well.

Permalink 01:21:32 pm, Categories: Linux, 28 words   English (US)

Box PCs

Here's a cool little small form factor PC that comes bundled with Ubuntu Linux. Great if you just want to setup a simple hassle-free desktop PC for someone.

12/21/05

Permalink 05:20:32 pm, Categories: Health, 138 words   English (US)

Sick

Well for the past 4 days I've been very sick. Not sure what I have, but it is an intense and burning sore throat and fever, like nothing I've ever had before. The cough is extremely painful and the first few days I was on the brink of passing out due to the pain. I think I've coughed up at least a gallon of phlegm by now. I've started antibiotics, which I absolutely loathe taking. They make me feel even more like crap, and I have to take them for 5+ days :(. My fever seems to be gone now which I suppose is a good sign.

This morning I had some sort of coughing fit where I could not for the life of me stop coughing. A minute more of that and I would've had to go to the ER.

12/18/05

Permalink 04:22:39 pm, Categories: Work, 7 words   English (US)

Google hiring

Here's an interesting article on Google hiring.

12/17/05

Permalink 05:17:58 pm, Categories: Movies, 531 words   English (US)

King Kong

You gotta love interspecies love.

I'm one of the 2 on this earth that doesn't like Lord of the Rings. I believe I am missing the LOTR gene or something. I just can't get into that fantasyland crap, much less 4 hours of it. So naturally I wasn't a big fan of Peter Jackson or knew much of his capabilities. After seeing King Kong, I have much more respect for him. This movie is absolutely amazing, and is truly what a movie should be.

Indian (i.e. Bollywood) movies are known for their length and breadth in the number of emotions they try to eke out of the audience. It's hard for an American viewer to understand them, because the transition between emotions in scenes is so abrupt that it's almost cartoonish. When a movie can span every emotion and smoothly transition between them, that's real directorial skill. That's what King Kong does.

It's 3 hours long, but honestly it didn't feel long at all. That's again a director's skill in keeping the audience enthralled. I was reminded of the Indiana Jones movies, E.T., Close Encounters, and others that just captivated you. This movie's action sequences actually made me sweat in my seat. It's said this was Peter Jackson's labor of love, and it really shows.

It's really amazing how much science and our understanding of animals, especially apes, have contributed to this film. There is probably a stark difference just in this realm from the older version of the movie. When we see Kong and Naomi interacting in many scenes, we understand that Kong sees beauty in her, and is essentially trying to impress her. He goes through the animalistic alpha-male rituals to show his strength. He wants to possess her, and when she is taken away he gains enormous strength to get her back. It's as if he is unstoppable in his goal. I honestly thought of how much the mind can affect the body. That's love for you. Are humans really much different? We do the same things, but just in different ways. Sure we are less primitive, but that's only relatively speaking.

I noticed throughout the movie a connection between ape and human, almost as if Jackson is trying to show Darwin's thoughts. You can see in Kong's face and actions a struggle to reach humanity, and the end result proves this. You realize just how alone this so-called monster really is, and how much he struggles to find love. In the end he realizes the gap is too wide. But he gained the love he was searching for.

If I have any gripes, it has to do with what looks like actors in front of a blue screen. Don't get me wrong, the effects are amazing, but there are certain scenes where the physics just doesn't seem right. Another issue I had was during the dinosaur fight scene. I just couldn't imagine dinosaurs going through so much trouble for a meager piece of meat that is Naomi Watts. I mean she's hot and all, but come on...

I was not sure what to expect with Jack Black seriously acting. I was pleasantly surprised because he did really well.

12/16/05

Permalink 07:04:03 pm, Categories: Home, 22 words   English (US)

Apartment hunt

Here are some pictures I took of apartments in Mountain View, CA. I picked Avalon at Creekside which seemed a good price.

Permalink 04:12:21 pm, Categories: Fun, 6 words   English (US)

Robopet

Seems like a nice Christmas gift.

12/15/05

Permalink 01:50:13 am, Categories: Car, 232 words   English (US)

BMW Tire

I went out to dinner with some friends and we decided to change a low tire on a BMW 325. Sounds simple right? Well first of all they have this wierd contraption to jack the car and alter the space-time continuum. Once that was figured out, we lifted the car and took all the bolts out, but the wheel would not budge. After a ring to BMW, they said to kick it a few times. We threw some really hard kicks at it but still no budge.

Then we took the emergency brake off. Not a good idea. The car fell off the jack and the wheel came off due to the pressure. Luckily the wheel stayed on and the axle didn't hit the ground. We raised it again and put the new tire back one. Afterwards, the towing service came by and explained how to change a BMW tire. He said to first loosen the lugnuts a bit while the car is on the ground, then drive and slam the breaks a few times. This loosens the wheel. After that, jack the car up and take out the wheel. Hmm, who could possibly figure this out? The manual says nothing about it.

Just some advice on how to change a tire on your ultimate driving machine. But if you're driving one you may as well call for service than do it yourself.

12/14/05

Permalink 09:38:49 pm, Categories: Work, 153 words   English (US)

I'm feeling lucky

Some of you may already know this, but I have accepted a position of Cluster System Administrator at Google. This was a difficult choice for me due to my past decision to go back to school. However, I think this is a great opportunity and has honestly always been a dream job for me.

I'm in Mountain View, CA now looking for apartments. The rent is definitely up there compared to Florida, but I found a fairly reasonable one that is right next to a bike trail that goes about 2.8 miles to Google. I would really like to bike to work. That would be sweet. I took a walk down the trail and it took me about 1 hour, so I expect a bike to take 20 min or so. I actually walked by Counterpane's office, and thought about all of the Bruce Schneier books I have :).

Anyhow, I'll post more as things move forward.

12/12/05

Permalink 10:36:12 am, Categories: Health, 69 words   English (US)

Exercise without weights

Here is a great website with some workouts you can do without weights. If you do them right, even just workout 1 will leave you sore for a few days. The main thing is keeping the correct posture for all exercises. Trying to do those 'hindu' squats keeping your knees behind your toes is next to impossible if you are not in shape. The 'pyramid' reps are also a killer.

Permalink 10:26:45 am, Categories: Web, 19 words   English (US)

Web based instant messaging

I came across Meebo, which is a web-based IM client. It's a rather amazing AJAX application. Check it out.

12/10/05

Permalink 09:37:31 pm, Categories: Linux, 106 words   English (US)

Mysql 4gb limit

Well I came across Mysql's default 4gb limit today. I have Apache logging via mod_log_sql the tables are growing very fast. Just a week's worth of data hits 4gb.

When I think of a database limiting you to a 4gb table I think huh? It just doesn't make much sense to me, and makes me want to use PostgreSQL instead. Sure I can do all this alter table stuff to get it larger than 4gb, but why default it to such a low value in the first place? What's worse, comments from users claim such an alter table can lose records.

Gee, that's great.

Permalink 12:28:23 pm, Categories: Linux, 637 words   English (US)

Homemade Cluster File Servers

I read an interesting article in the latest Linux Journal about someone who used a Linksys NSLU2 file server, installed a Linux firmware on it, and created an iTunes streaming music server. He was able to use this small device to replace a full-blown PC for serving music.

The NSLU2 is a small nifty device with 2 USB ports and an Ethernet port. You hook up USB (v2) hard drives to it, and it has a web-based administrative interface to share out the disks via Samba. Once people were able to install Linux on it, all sorts of cool applications came forth.

But I have another cool idea for such a setup. Think about getting a bunch of these Linksys devices, a bunch of USB drives, and creating a cheap clustered file server. Something that could aggregate the storage of many of these units, and possibly provide some way to easily add or remove nodes on the fly. It would be cool if there was some sort of redundancy or failover as well if a node went down.

A 'node' in this case refers to the Linksys device. On each we could connect 2 hard drives, and then use software RAID 1 to mirror them. This would give us redundant data on one node. Now we would need something to consolidate all of these nodes into one mass storage. There are a few clustering filesystems that I've done some reading on.

Probably the simplest setup is to use (E)NBD, or network block device. This allows a Linux system to 'share' out local devices over the network to be mapped on another system. The local 'device' in this case would be our RAID 1 virtual device. The system that would map these NBDs could be another Linksys node that aggregates them into a RAID 0 device. On that same node we'd run Samba or some other filesharing system. It sounds like a homebrew implementation of Coraid.

This will handle single drive failures, but the problem is if one node goes down, that makes one unit of the RAID 0 inaccessible, and the RAID 0 will fail completely. One option is to use RAID 0 striping on the local node with 2 drives. Then the fileserving node would RAID 1 each pair of nodes, and finally RAID 0 stripe over the pairs. I guess this would be termed a RAID 0+1+0 setup :). That might be overkill for a home setup where you can tolerate a bit of downtime to replace a node (at least your data is still safe).

There is another driver called ddraid, but I'm not exactly sure how this would differ from an NBD RAID, and it is documented poorly.

An alternative to NBD is using a clustering filesystem. There are a few for Linux, but many seem to target big-spenders, and have high learning curves and jargon. The most simple and straightforward ones I found were Lustre and PVFS2. I like PVFS2 because it's documentation is easy to follow. It seems to provide easy aggregation of data storage on nodes, allowing addition and removal. It seems to create a virtual directory that is just the aggregate of all nodes. It does not provide redundancy on its own, but using software RAID and failover tools it is possible. I'm a bit confused to how it handles low disk space issues though.

Using RAID 0, I don't believe it would be easy to add more storage as you go. Maybe just using RAID 1 on the nodes, and then aggregating them with NBD and LVM would be enough. LVM would let us grow the filesystem as we added more nodes, and the mirroring saves in case of 1 drive failure.

I think creating a farm of slugs, as these Linksys units are referred to, would be a very cool home project to work on. I hope I can get some time.

12/09/05

Permalink 12:30:47 pm, Categories: Linux, 48 words   English (US)

Usability

There's a good article by Nat Friedman at the end of this month's Linux Journal. A quote:

There's a difference between software that is usable and software that is a pleasure to use. Until you watch people using your software, it's hard to know how well you're doing.

Permalink 11:27:30 am, Categories: Music, 174 words   English (US)

Pandora

I remember reading in Discover magazine about some company trying to design technology to analyze commonalities in a music database and predict what a person will like. I don't know know if this is from the same company, but Pandora seems awfully close to achieving this.

A friend pointed it out to me and I think I'm already hooked. Normally I listen to Shoutcast streams, which was rather difficult to find consistent music that I liked on any station. Pandora seems to generate personally tailored stations. I wonder how they are achieving this, because I can imagine it requires quite a bit of bandwidth to play so many streams, growing linearly with the number of users. I wonder if these seemingly unique streams are actually shared among many listeners.

Regardless of how they do it, it's very cool. It has consistently predicted music that I like. I don't think I'll be listening to Shoutcast streams anymore, for popular music at least. I'm not sure if I can get any drums and bass from Pandora :).

12/08/05

Permalink 09:38:02 am, Categories: Linux, 333 words   English (US)

Indians and free software

Here is an interesting view of how many Indians educated in India think of free software. I myself am Indian but I was born and raised in the US. I agree with the article that the university systems in India probably do not encourage creativity, but rather focus on employment.

However I think for a person to be interested in free software, they have to have a certain mindset created very early, perhaps much before college. I think the reason I became interested in OSS was not because of school, in fact I did most 'hacking' outside of school. I saw the benefits of OSS and its results, because every bit of 'hacking' or learning I did was dependent on OSS. I needed to have Linux on my system to learn Unix. I needed GCC to learn C, etc. These tools were there and I didn't need to spend money to gain the knowledge on how to use them. I saw that in general OSS seemed to work better than closed source in many cases, and that I could get better support from one lone developer than an army of help desk personnel in a commercial product's support department.

I also worked in the computer science department as a sysadmin, and doing so forced me look for solutions to problems with little budget. It also allowed me to change software to work the way I needed it to. It's one thing to know that the source is there and you can change it, but it's different when you actually do change it. You become much more entwined in OSS, and you realize the true power of it. You realize that this could never have been done if you used a closed source product. All of this contributed to how much I value OSS.

For someone to gain interest in becoming an OSS developer, you need to see all of these things and realize that you want to be part of it as well.

12/06/05

Permalink 08:46:53 pm, Categories: Science, 83 words   English (US)

Word of the day: Lobotomy


icepick

From my A&P textbook:

During the middle of the twentieth century, this rather drastic procedure, called prefrontal lobotomy, was used to "cure" a variety of mental illnesses, especially those associated with violent or antisocial behavior. After a lobotomy, the patient would no longer be concerned about what had previously been a major problem, whether psychological (hallucinations) or physical (severe pain). However, the individual was often equally unconcerned about tact, decorum, and toilet training.

A reader sent in this interesting lobotomy story.

12/04/05

Permalink 11:32:03 am, Categories: Medicine, 30 words   English (US)

Truly useless information

So I'm sitting here studying for exams and thinking to myself: How does it benefit me to know that a pearl fish lives inside the anus of a sea cucumber?

12/03/05

Permalink 10:28:00 pm, Categories: Programming, 75 words   English (US)

Sourceforge getting stupid

Is it me, or is Sourceforge getting extremely ugly and impossible to navigate? The place is filled with tons of ads and dumb Yahoo search results everywhere. What's worse is the ads and unrelated stuff looks exactly the same as the rest of the page and it is hard to dilineate ads from useful content. I guess that is their goal. For some reason, whenever I go to Sourceforge it feels like I gain 20 pounds.

Permalink 12:07:08 pm, Categories: Apple, 477 words   English (US)

Safari cookie bug?

I think I found a bug in how Safari 1.3.1 is handling cookies.

I've setup Trac, which is a nice Python wiki/bugtracker for a client. There are multiple projects setup, and each has its own authentication via htpasswd. The problem I'm having is when I login to one project, I can't login to the other project without logging out of the previous project. I described the problem fully on this bug posting.

Trac uses a trac_auth cookie to keep track of authentication. It has a specified path associated with it, so it's perfectly ok to have multiple trac_auth cookies for the same site and separate authentications for each. But this is simply not working with Safari. Firefox works fine with this. I have 2 projects setup, one with path /projects/splash and another with path /projects/splash_old. I started looking at some tcpdumps to see what Safari is sending. I noticed that when I tried logging into the 2nd project after logging into the 1st one, Safari sent:

Cookie: trac_auth=36b8db01607d7ab36506ad97d38196b3; trac_auth=eb167bd6b57b7a5dae9a3dee48ef13b2

Note the 2 cookies with the same name. There is only one cookie that is supposed to be associated with the path I'm going to (/projects/splash_old) so I don't understand how it could send 2. The cookie specification states that a client may send multiple values of the same cookie if the cookies are part of a parent path. But /projects/splash is not the parent of /projects/splash_old. Or am I missing something? It seems Safari is incorrectly making this assumption.

Looking at a tcpdump of Firefox shows it only sending one trac_auth cookie for the 2nd project and it works fine. Firefox also correctly shows 2 trac_auth cookies in the cookie browser, one associated with each path (/projects/splash and /projects/splash_old), whereas in Safari I could only see one.

Update: I created some simple scripts to reproduce this. Go to this page which will set 2 cookies. One cookie will be for test.php and another for test.php_notpath.php. Clearly the second php script is not a child of the first one. If you follow the link, you should see HTTP_COOKIE only having one value for SafariCookieBug because the resulting page is not a child page. That's how Firefox behaves, but Safari 1.3.1 shows 2 values.

Update: Well I found out that IE behaves like Safari. So maybe this is not a bug after all.

Update: Deeper into the rabbit hole I go. It turns out Trac uses the Python Cookie module, which doesn't have support for multiple cookies with the same name. It uses a dictionary to store the data. I'll probably make some changes to Trac to get it working, but I submitted a Python bug report.

Viraj's Weblog

Donate to keep this site going!

Amount USD $

December 2005
Mon Tue Wed Thu Fri Sat Sun
<<  <   >  >>
      1 2 3 4
5 6 7 8 9 10 11
12 13 14 15 16 17 18
19 20 21 22 23 24 25
26 27 28 29 30 31  

Search

Categories


Misc

Syndicate this blog XML

What is RSS?

powered by
b2evolution