Viraj's Weblog

This weekend I finally installed Leopard on my macs. I have an old Powerbook G4 15" and a Mac Mini. Both of them run it great (I have at least 1G RAM on each). At home I also have an Ubuntu server running on a laptop which I use for various Linux things. I've been sharing drives via Samba on it and hacked up my own backup scripts.

But with Leopard, there is this cool Time Machine thing I can use right? Well I wanted to use a network share via Samba with Time Machine. Turns out it is not so straightforward. OS X by default won't let you use the network drive. This post describes how to get around that. There is also a wealth of information here. Great, now I can select the network drive in Time Machine.

But now I was getting an error about creating the disk image. I watched a bit what it was actually trying to do on the Samba share and noticed that it was using the machine name in the filename being created, which is a 'sparsebundle' file. My machine name was something like 'Powerbook G4 15"' and I had a feeling the '"' or some other character in the filename was not playing well with Samba. So I changed my computer name (via Preferences -> Sharing) to some very simple name. Still no go .

After some searching I came across this post which seems to describe my problem. It generally suggests to create the disk image file manually and then copy it to the network share. But I wondered, why not try creating this directly on the network share? I should have the same problem Time Machine gave but perhaps I can get more information and debug further. So I tried it, but unfortunately the command given in the post was wrong. A few comments below describes the correct one (wtf don't they fix this?). Here is the command I ran:

hdiutil create -size 50g -fs HFS+J -type SPARSEBUNDLE -volname "Backup of powerbook" powerbook_000d93b43026.sparsebundle

And I got the unsexy error:

hdiutil: create failed - Operation not supported

Fooey. Ok, I didn't get much debug info there. It sounds like some operation is being tried on the Samba share, but what? Next I followed the rules and created this image on a local drive. That worked fine. Now the post states to copy this image over, but I instead tried to 'mv' it. This should try to replicate everything at the destination and perhaps give me more info. It did:

mv /tmp/powerbook_000d93b43026.sparsebundle .
mv: chown: ./powerbook_000d93b43026.sparsebundle/bands/62: Operation not supported
... lots more
mv: chown: ./powerbook_000d93b43026.sparsebundle: Operation not supported
mv: /bin/cp: terminated with 1 (non-zero) status: Cross-device link

Interesting! So it looks like the real culprit here is 'chown'. OS X is trying to chown these directories on the Samba drive, which of course is not supported. Looking at the files created by hdiutil, it looks like it is trying to chown the group 'wheel':

powerbook_000d93b43026.sparsebundle valankar$ ls -lRa|more
total 16
drwxr-xr-x@ 6 valankar wheel 204 May 24 10:27 .
drwxrwxrwt 10 root wheel 340 May 24 10:27 ..
-rw-r--r-- 1 valankar wheel 498 May 24 10:27 Info.bckup
-rw-r--r-- 1 valankar wheel 498 May 24 10:27 Info.plist
...

Whatever, knowing this is not helping me much. In fact, I just found another post where someone digged further. After copying it over (actually the 'mv' copied it just fine but bombed on the chowning), I enabled Time Machine and it generally worked. But it worked SLOW.

As an aside, I've generally had some slowness using Samba when it involves lots of intense disk access. Say I'm watching a video as well as recording something with EyeTV on the Samba share. It generally will clip the video pretty badly. No amount of Samba tuning on my part could improve this and I eventually gave up. But Time Machine was ridiculously slow, and it drove me to wonder, "WTF am I using Samba for anyway?" I mean seriously, I have no Windows boxes here. Why aren't I using Apple File Sharing? Surely Linux supports it right?

Thus began my journey into setting up netatalk (AFP) on Ubuntu. It was a short one and generally without hiccups. This page has some pretty good docs, however with Ubuntu there is one stickler. SSL support is not compiled into netatalk, and this is required for Leopard. The workaround is described here. Sheesh, why even include the package if it's essentially unusable? Well it works with Panther, albeit with a warning. But with Leopard you just get an error trying to connect.

Now how do I connect to my share? Well I have yet to figure out how to show my shares in Finder. Instead I have to go directly to Go -> Connect to server. In there I type the full path of the share, e.g.:

afp://system76/timemachine_powerbook

At this point I came across a problem. In my /etc/hosts file on the server I had something like:

127.0.1.1 system76 ...

And when I attempted to connect, it hung at trying to connect to 127.0.1.1. Evidently this IP is passed along over AFP. I fixed my /etc/hosts so my hostname was not pointing to a loopback (no idea why it was in the first place).

Voila, I have a network share now. Sadly, neither hdiutil nor Time Machine were able to create the disk image on the network drive, so I still needed to create the image locally and copy it over. But once I did it worked like a charm, and much faster than Samba. It's still pretty slow for the initial backup of about 20G (in fact I think it took like 12 hours), but after that it worked pretty well. I'm a bit worried how it will deal with large backups though, as it could be lots of traffic over the network.

Another thing I have to worry about is resizing the disk image. This post describes it somewhat, and it's likely I'll have to do that at some point.

I setup the same thing on my Mac Mini, so now I have 2 Time Machines going to the same share. Plus I have AFP setup and no more need for Samba. Good riddance!

One caveat. My syslogs are filling with:

May 26 09:02:33 system76 afpd[3651]: bad function 4F
May 26 09:03:05 system76 last message repeated 89 times
May 26 09:03:25 system76 last message repeated 163 times

Turns out this is some undocumented AFP call.

Update: And now I think I've lost faith in Time Machine network backups altogether. There is a great post describing the pitfalls of such a solution. Maybe it's time to move back to rsync snapshot backups.

These days the buzzword seems to be "hypervisor". It's a general term for what VMware is. One thing I've always found to be a good use case for virtualization is testing new and possibly broken configurations. These past few days I've been food poisoned and working from home recovering, but it did give me a chance to play around with the latest Linux virtualization: KVM.

KVM in this sense is not that hardware device to connect one monitor to multiple machines, so the choice of naming is unfortunate. It is based on QEMU, however there are kernel hooks to make things go faster by utilizing virtualization features on certain processors. What's great is recent distributions have it simply as a kernel module that doesn't require any kernel patching.

First some background. Why am I interested in this? Well in my case I run a website on Apache (who doesn't). I've been hearing some good things about lighttpd and I wanted to see what the fuss was about. However, I didn't want to muddy my current Apache server with lighttpd. I simply wanted another system to play around with, preferably one that didn't require hardware. So this seemed like a perfect case to try out the completely free KVM.

Now my host OS (the OS that will run KVM) is a Ubuntu 7.10 server. It actually runs on a closed laptop, so essentially I have no display on it and only connect to it remotely. Most virtualization software these days are GUI apps unless you buy the horribly expensive 'server' products. I was somewhat pleasantly surprised as to how KVM worked in my scenario. Ideally I wanted to run a guest OS the same as my host OS, Ubuntu 7.10, so that's what I started with.

First I installed the kvm package:

sudo aptitude install kvm

Easy enough, next I modprobe the right modules:

sudo modprobe kvm
sudo modprobe kvm_intel

I should point out you need to have a processor that supports KVM. See their web page for more info. Now it get's a bit confusing. In some documentation there is mention of running "qemu-system-x86_64", however on my box this is the unaccelerated version that makes no use of KVM's kernel modules. I'm not sure what's going on here, but I believe at some point QEMU and KVM will merge code, and maybe this is the reason for the documentation discrepancy. Anyhow, for my case, I had to use "kvm" to start up my virtual machine. So I downloaded the Ubuntu 7.10 server ISO and begun my journey. First I needed to create a disk image. Interestingly, these disk images only take up space as you add to them, similar to VMware:

qemu-img create -f qcow vdisk.img 10G 

This creates a 10G-maximum disk image. Now we're ready to begin the installation:

sudo kvm -hda vdisk.img \
  -cdrom ubuntu-7.10-server-i386.iso \
  -boot d -m 384 -vnc 192.168.1.137:0.0 \
  -no-acpi

This simply boots the VM with the Ubuntu server CD and 384M of RAM. It's recommended to use -no-acpi, so I did. Now BAM I get an "exception 6" and a crash immediately (as described here). My foray is not starting off well . After more searching I came across this bug which hinted that this was maybe fixed in a newer version of Ubuntu. Huh? So something in Ubuntu is causing this crash? Searching further I found this thread which gives more info:

Confirmed here with kvm-intel and KVM 39. Invalid opcode (#UD) is probably caused by the boot spash code which may be using big real mode code.

So indeed, something in Ubuntu was doing it. So I decided to try grabbing the bleeding edge Ubuntu "Hardy Heron" server. After some thumb twiddling it now boots!

Now remember how I have no display on this box? Well the convenient -vnc argument creates a VNC server that I can connect to remotely! This is all great, but kvm also supports X, and actually I do most of my work from an OS X X terminal. Why not just use the X interface? Well on OS X, the keyboard becomes completely unusable in KVM for some reason. It could be something retarded in the X server. Anyhow, for VNC, there is Chicken of the VNC, by far the stupidest package name ever. It generally works well though, however when I tried to connect to my VM I was getting some invalid rectangle error. It seems everything was against my trying to get this working . After much experimentation, I found that I had to simply disable Hextile encoding in the VNC connection profile and voila, I can connect.

Now I thought to myself, I'm going to be running a webserver on this, don't I need some kind of network setup? Well once I confirmed things are booting ok, I did some research on KVM networking. Essentially what I wanted is my VM to appear just like a separate machine on the network with full network access. KVM has all sorts of networking possibilities, but here is my setup. First I updated /etc/network/interfaces to look like:

auto lo
iface lo inet loopback

auto br0
iface br0 inet dhcp
  bridge_ports eth0
  bridge_maxwait 2
  up /sbin/ifconfig eth0 inet 0.0.0.0 promisc

auto eth0
iface eth0 inet static
  address 172.16.5.0
  netmask 255.255.255.0

This generally came from this Ubuntu KVM doc. The IP above is actually a bogus one that won't be used. Rebooting (alas, I did have to reboot) gave me a br0 and eth0 device listed via 'ifconfig'.

My host machine still worked so that's good.

Now how to start the VM with networking? Simple:

sudo kvm -hda vdisk.img -m 384 \
  -vnc 192.168.1.137:0.0 \
  -no-acpi -net nic -net tap

I went through the install with no problems at all and it got an IP via DHCP on my router. Sweet.

Now one thing that was bothering me about all of this is that spiny sudo prefix. I would've liked to not use sudo. This forum thread mentions a possible solution but I had no luck with tunctl and I didn't feel like spending too much time on it.

So I got my cool VM working, now what? For what reason am I on this Earth? On my main web server I have a Django site running under Apache. What I wanted to try is running this with FastCGI under lighttpd. Now I have to admit, lighttpd configuration is a much nicer experience than Apache configuration.

Take a look at my lighttpd.conf. My virtualhost is defined at the bottom, and is taken mostly from the Django fastcgi docs. lighttpd has a very simple and elegant configuration. Note that I had to modify some of the modules loaded, and the Django docs seem to indicate the ordering is important. Once I started up my Django site in fcgi mode my site was instantly accessible, and FAST. At least, very fast for a virtual machine!

So what have I learned from all of this? Well KVM is cool, and one of these days it is going to beat out its commercial counterparts. Also, lighttpd is cool. Cooler than Apache I must say. It will definitely be coming to more web servers near you.

I've been wanting to buy a Linux laptop. There are a bunch of vendors now selling them from the high-priced Emperor Linux to probably Walmart. I felt like supporting a small Linux laptop vendor, and narrowed my choices down to either Zareason or System 76. I heard about both on the Linux Action Show Podcast.

I decided to go with System 76 as it was a bit cheaper. I purchased a Pangolin Value laptop for about $1k. Here are the specs:

1 x Pangolin Value (PAN-V4) = $998.00
       Bluetooth no Bluetooth
       Extra AC Adapter no extra AC adapter
       Extra Battery no extra battery
       Hard Drive 80 GB 5400 RPM SATA
       Hardware Warranty 1 Yr. Ltd. Warranty and Technical Support
       Laptop Bag no bag
       Memory 2 GB - 2 x 1 GB DDR2 667 MHZ
       Operating System Ubuntu 7.04 (Feisty Fawn) Linux
       Optical Drive CD-RW / DVD-RW
       Portable Flash Drive no flash drive
       Processor Core 2 Duo T7300 2.0 GHz 800 MHz FSB 4 MB L2
       Wireless 802.11 abg

Sounds pretty nice right? Well I got the laptop and generally things were working. They installed Ubuntu with wifi drivers, etc. Immediately though I noticed one things: a super-sensitive tap-to-click touchpad. I straight off went to look for mouse preferences but could find no indication of tap-to-click. In fact, to the system it appeared I had a regular external mouse. So began my hell trying to get this laptop in a usable state.

You can see my cry for help on this support forum thread. Interestingly I think my original posting was deleted for some reason. Maybe they are worried about bad PR? The gist of it is they said there is no Linux driver and the 'solution' is to disable the touchpad. That's just fucking ridiculous, and I pretty much lost all respect for System 76. I mean, why even claim you are a Linux laptop vendor when you are selling laptops with proprietary hardware without Linux support?

This laptop was pretty much unusable to me. After some long hard searching I came across this Ubuntu bug report which eventually led to this kernel bug. That shed some light. Well at least it proved that someone else was going through the same pain.

I contemplated writing my own driver as it would be a good learning experience. But I'd likely need a Windows box to do any reverse engineering. I didn't have one, and this didn't sound like fun at all. A few weeks passed and then someone updated the bug stating that a driver had been written. The announcement was a patch asking for comments. I decided to try it.

Now I haven't compiled a kernel in a long time. I needed to find out the proper way to do so on Ubuntu. I decided to follow the steps on this Ubuntu page. After going through it, I think this howto would have been a better choice. Basically I had to manually apply the patch as the kernel source I got was different than what was assumed. It was only a few changes though. It took many hours to compile, and only afterwards did I realize it was compiling many different kernels for different architectures. Sigh, this whole package management stuff for kernels is a real pain.

Eventually I had a running kernel with the new driver. But I'm not at the end of the race yet. I'm in communication with the author debugging some issues. I did manage to disable the tap-to-click by writing the proper hex value to the registers provided via /sys by the driver. However there seems to be some problems with window focus where applications don't give up the pointer. Hopefully some debugging with Arjan (the driver author) will resolve this. Stay tuned.

Here's an interesting interview with Con Kolivas. Kind've depressing, but likely truthful in many ways.

I have an old Thinkpad laptop. Since the new Ubuntu 7.04 came out, I wanted to try it out. The laptop is pretty much useless as a desktop with 192M RAM, but I've been hearing good things about the server version of Ubuntu and decided to give it a whirl. Now I have a laptop server that I use for simple things like an SSH gateway for me to connect externally.

Ubuntu server is pretty nice, but as with typical Debian distros, is rather bare bones. I had to do alot of package downloading to get to a useable system. What do I think of Ubuntu as a server? Well I'm used to Redhat, and I've griped about Debian in the past, namely update-rc.d. The same problems I mentioned in that blog in 2005 still exist today. Searching around I found that 'rcconf' is another useful tool. Why not install that by default then?

I also have major gripes with the dbconfig stuff. This is an effort to make database configuration package-friendly. It works, at least until you start removing packages and trying to reinstall them. I understand the reasoning for it, but it is just more pain than necessary. It's very easy to screw up the configuration completely.

Anyway, once I got things setup, I liked it. Next up was trying to mount some filesystems on my Mac Mini. I installed netatalk, but soon realized this was for the opposite: to share linux filesystems to a mac. This page indicates that the afpfs filesystem module, which I needed, is unmaintained. Next I found afpfs-ng, which is a fuse module. That sounds great and all, but no matter what I did I could not get it to authenticate properly with my mac.

I decided to ditch AFP and setup my mac to export shares via Samba. It's tried and true and mounts on Linux flawlessly. But I had a few shares to mount, all containing multimedia stuff. Now comes the cool part: unionfs.

On my mac I have 2 directories with movies, one on an internal drive and one on an external. I shared them out and mounted them on Linux, with something like this in /etc/fstab (using \ as line continuation below):

//192.168.1.100/valankar /mnt/mini smbfs \
 credentials=/home/valankar/.smbcredentials, \
 uid=valankar,gid=valankar,fmask=600,dmask=700 \
 0 0
//192.168.1.100/stuff /mnt/stuff smbfs \
 credentials=/home/valankar/.smbcredentials, \
 uid=valankar,gid=valankar,fmask=600,dmask=700 \
 0 0

The credentials file is so I don't have to include my password in the world-readable /etc/fstab. So this is great, but what I'd really like to do is combine /mnt/stuff and /mnt/mini in to one virtual directory. After installing the unionfs-tools package, I added to fstab:

unionfs /mnt/movies unionfs \
 dirs=/mnt/mini/Downloads:/mnt/stuff/Movies \
 0 0

Now when I go to /mnt/movies and do 'ls', I see a combination of both directories. This is just so cool. When I write to the directory, the 1st real directory gets priority, but I thought, this would actually be a very cool filesystem if the writes were put on the drive with the most free space. There would need to be some coordination when making directories, but this could essentially let me mount many different types of drives and even disparate filesystems, and then combine them into one writable virtual filesystem.

Turns out there is one such experimental filesystem called SwitchFS. I'm not sure how active this project is but I might give it a whirl. I've been itching for some open source stuff to work on.

strace is the swiss army knife for sysadmins. Here is a nice article on debugging Apache with strace.