Viraj's Weblog

I've always liked David Lynch's films. Blue Velvet and Lost Highway have got to be my favorites. Not to mention Mulholland Drive. I would call them psychological thrillers. Blue Velvet's Dennis Hopper character is almost evil personified, and it's just amazing to see him act in those scenes. The way Lynch paints film makes the images unforgettable. He also uniquely blends subtle and ominous music that enforces the effect.

So I had to see Inland Empire. It was playing at small arthouse theatre in SF. The movie is a whopping 3 hours long, and I considered not going because of its length. Lynch promoted the movie by sitting on the street with a cow. There are no cows in the movie, but the absurdity of that is equivalent to the absurdity of the movie. In fact, the opening credits included a company called ABSURDA.

So what is this movie about? I honestly don't know. Many would consider it an 'artsy' movie. It doesn't follow any plotline, and the timeline is so disjointed it's pointless to try to piece anything together. Many of the reviews state that you should just let the movie pass over your mind without trying to figure it out.

The movie is made with a digicam. Aside from a few unique effects, the production is rather bland. After about the 20th "slow walking through narrow hallway," I about had enough. Lynch's previous movies were very beautiful, and they blended music at the right times. Inland Empire is more chaotic, and has almost no music for the 1st hour. There are lots of ominous noises though. I cannot really get into a movie without music. When the music does come in, it is rather refreshing and it's a great haunting blues song. There are some scenes that really stick with you, such as a family with human torsos and rabbit heads. Supposedly there are a few references to Alice in Wonderland, but it's a much more horrific wonderland. The movie seems to be about guilt and pathological jealousy. It plays like a dream, where nothing at all makes any sense whatsoever.

The problem I have with this movie is the characters. Lynch's previous movies had all sorts of unique characters that you remembered for a long time. This movie only has Laura Dern. I felt it was too focused on her. Also, previous movies had some semblence of a plot, or at least some small snippets of plots. Think of the 'tailgating' scene in Lost Highway. There are unfortunately no such similar moments in Inland Empire. This movie is more dreamlike, with plotlines changing continously and furiously. You need alot of patience to sit through it.

I'd recommend it only if you are Lynch fan. Otherwise, you'd probably hate it.

Good critique on the ridiculous state of mainstream media.

I work on an application at work that does all sort of maintenance on large clusters. Without going into too much detail, it generally keeps an eye on our servers, fixes problems, and performs setup and maintenance. It's a rather important system, and it's all written in Python.

Now for the past few months we've been trying to debug an issue. At random times it would segfault. We had lots of people looking at the core dumps and it seemed to be some sort of memory corruption. We ruled out the machine, as it would crash on multiple machines.

So where did I start? The program is multithreaded, so it usually has lots of processes listed via 'ps'. On a whim (and most of my debugging was based on whims), I did straces on every process. When I did this, I observed the Heisenberg principle, the program did not crash. Running straces on the application wasn't really a solution so I had to look more into it.

I'm fairly involved with the development of this system, so I understand alot of the code. I did not understand why it would crash. I did know one thing: the application uses threads, and it also forks. But that didn't help me because I had no idea there was a problem mixing the two.

I decide to run a tcpdump to see if any strange network traffic may be causing it. After much analysis with ethereal, I noticed something strange. Crashes always seemed to coincide with a web request to the application. The app runs a web interface, and we have monitoring hitting this UI to make sure it is running properly. So it turned out, the 'random' crashes were not so random, and were happening when our monitoring system was hitting the app.

But why was it crashing? With any multithreaded app, you need to have proper locking of shared variables. You generally have to worry about multiple threads modifying/accessing the same area of memory at the same time. So I spent weeks analyzing the code, trying different locks in different areas, etc. I made our webserving function simply output nothing, so it didn't access any shared data. To my surprise, we still crashed.

One problem was I could not predict when it would crash. It could run up to 12 hours without crashing, and most of the monitoring hits of the UI worked fine. So I had work on replicating the bug more frequently. After alot of work using a live debugger that a co-worker built into the app, I was able to set off a sequence of events that crashed it within 2-3 minutes. It generally involved slamming the UI with parallel web requests, and kicking off some internal methods that cause a fork() (for example, SSH'ing to a machine to verify connectivity). Ok that's better for testing. But it told me something: the crash seemed to be involved with forking.

Our webserver in this app was using Python's generic BaseHTTPServer with a ThreadingMixin to handle requests with new threads. From parallel analysis of the core files by others, it did appear to be crashing on pthread create functions. Out on another whim, I changed ThreadingMixin to ForkingMixin, so the webserver would fork instead. Voila, no crash! So this told me it has something to do with forking and threading.

A co-worker pointed out this GNU C Library page warning about using threads and fork. The clincher was this sentence:

Because threads are not inherited across fork, issues arise. At the time of the call to fork, threads in the parent process other than the one calling fork may have been executing critical regions of code. As a result, the child process may get a copy of objects that are not in a well-defined state. This potential problem affects all components of the program.

So indeed we were doing something bad. The problem with changing our webserver to ForkingMixin is that removes any possibility of IPC, which we needed via our UI. i.e. A web request couldn't change the state of the program unless we created some specific IPC mechanism. So instead, we created a parent thread which started one BaseHTTPServer without any mixin. This means the requests would be processed serially, but would still have the ability to update shared data. The only issue was low throughput due to serially handled requests (and also, one request at a time). For our app, it sufficed because the number of UI requests we get is low, and mainly it's for monitoring.

This fixed our immediate issue, but I have now seen crashes still occurring, but at a much later time. The problem is that we are still using threads. It's not just in our webserving, but in other parts of the app as well. The only solution it seems is to remove such threading. We must fork because we do things like SSH.

What's the lesson here? Do not, I repeat, do not ever design an application that used threads and also forks. You will have no end of trouble. If using threads, stick with threads only. If using fork, stick with forks only.