Viraj's Weblog

Today we received another small form factor PC and I set it up as a firewall. We need to statically NAT an external IP to an internal IP, and after some searching I found this article explaining the procedure.

The Stalker guys have always seemed to have an ambivalence towards Linux. I've been on their mailing list for quite some time, and there are alot of attacks on Linux having an inferior threading implementation. It's almost to the point where they almost say that if Communigate crashes, it's due to buggy Linux.

Today while testing Communigate on Debian Stable with kernel 2.4.29 it crashed when I was accessing the SSL webmail interface. This thread pretty much sums up the experience and Stalker's thoughts on it. In my opinion it's a long winded way of them saying "Linux sucks at threading." Especially biting is this message.

Me and some others asked about this issue awhile back. Take a look at the response. The hatred is palpable.

Still, Communigate is a very nice system when it works. You just have to find the right distro. From what I gather, it seems the best choice we have for running Communigate is to ditch Debian and run this in Fedora Core 3.

A co-worker pointed me to this flash parody.

A few years ago I heard of a company called Bynari that was working on an Exchange replacement. In fact at my previous job it came down to either Bynari Insight Server or Communigate Pro. The reason I picked the latter was twofold. One, I had pretty good experience with using Communigate in a large ISP environment (1 million users). Two, the storage system I had was a NAS mounted via NFS, and (at the time) Cyrus IMAP (which was included with Bynari) had many caveats listed on their site about using NFS. They didn't recommend it.

These days I have no such restriction and am free to experiment. I downloaded the latest eval version. What I like about Bynari is it's primarily based on open source tools: Apache, OpenLDAP, Cyrus IMAP, Postfix, ProFTPD, Amavis, SpamAssassin, Clamav, etc. Installation was a cinch on CentOS. Their site mentions Debian support as well, but strangely I only see RPMs.

So what's good about it? The web admin is surprisingly straightforward. It's probably the first interface that that actually makes LDAP comprehensible. It was easy for me to setup domains, add users, etc. The webmail/groupware interface is also pretty nice. This is probably the most important thing I'm looking for, because I want to remove dependency on Outlook. I don't much care about the Outlook Connectors and such. Bynari's webmail is not Outlook Web Access, but does have a decent calendar interface. You can schedule meetings and see shared folders via the web client. It's lightweight and things work as expected. I did like the way Cyrus was storing messages: plain text files. Maybe Cyrus was just configured to run this way, because I remember it using Berkeley DB files in the past.

Some problems I noticed was when someone accepted or rejected a meeting, the attendee response is not visible in the original calendar entry. You can't tell whether someone accepted or rejected it. This seems a bug to me, which I've asked their support about. Also, on the recipient end, the calendar entry does not include the attendee information. So I can imagine users having questions like "Who was this meeting with again?"

I tried to migrate a 1.9gig PST file. Bynari provides a PST import tool via the web interface. But come on, 1.9gigs submitted in a web form? It crashed my Safari, and Firefox said "blow me." That import method does not seem useful to me. The way I imported data in Communigate was to install their Outlook connector, and use Outlook to import the PST. Well I tried the same with Bynari, and it pretty much crashed Outlook... sigh. Next I tried just copying from PST to the IMAP server in Outlook, which again crashed. I do have to get data from Outlook users into whatever server we use, and I want that to be done easily.

Meanwhile Scalix got some good press at LinuxWorld. It seems their new version has improved the admin interface, which was sorely lacking in the version I tested. I like Scalix's webmail interface, but I hate the complexity of the HP openmail system. It reminds me of people being hired just to administer openmail. It's a fucking mail server, it doesn't have to be this complicated! Looking at the Scalix manual all you see are chapters filled with cryptic om(insert-jobsecurity-acronym-here) commands. It's just silly to expect someone to spend time learning this crap, because when something goes wrong in such a closed system, no one will know how to fix it.

It looks like Communigate is still the choice for us.

As I mentioned, we installed a small form factor PC as a router. It's currently in test mode. One thing I noticed is sometime after bootup the blue power LED starts flashing, maybe twice a second. The system appears normal. I was concerned it might be overheating or something, so I installed lm_sensors and the SMART (ide) utilities. That took a bit to get working, mainly because Debian Stable had old versions and I ultimately had to get the latest source.

When I check sensors output, it lists quite a few warnings about voltage, 1 out of 3 fans not running, and the CPU running at 88 degrees celsius. That temperature sounds insanely high, and I'm doubting whether the output is correct. I then ran smartctl on my IDE drive and it reported temperature of 40 C. I would assume if the CPU is very hot, the HD temperature would be high as well since it sits right above the CPU in the small case.

I haven't noticed any wierd hangs or crashes. Just the flashing blue power LED. Tomorrow at work I'm going to reboot it and quickly check the BIOS temperature reports. My co-worker says it could need a BIOS flash update. It's a cool little box, but would suck if it's overheating .

Update: Rebooted the system this morning and the BIOS reported a CPU temperature of 52 C. That seems reasonable. Just before the reboot sensors said 88 C, so I know it must be wrong, or there is some awesome cooldown system going on between the time it takes me to reboot and get to the BIOS . What's also wierd is now after reboot sensors says my CPU temp is 25 C, and reports a different temperature field as 42 C. Forget sensors on this motherbord, I don't trust that output.

I looked at my Tmobile bill today and noticed it was $10 higher than normal. I have their 'get more' plan that gives 600 'whenever' minutes. I never come even close to using my cell that much.

So I'm looking at the charges, and noticed that the $10 was from 411 calls! WTF? Tmobile is charging $1.25 for each 411 call. That is such a fucking ripoff. Why can't they use the 'whenever' minutes that I never used to pay for this crap?

On the bill envelope they are encouraging people to dial 411. Their desire to squeeze as much money as possible from customers is vividly apparent.

I'm working for a client to migrate a fairly busy web server to a new network. Obviously, downtime would like to be kept to a minumum.

The main issue in such migration scenarios is DNS. You need to be sure to have low TTL values so that when you switch the IP, users go to the new site fairly quickly. Usually you would update the TTL many days in advance. TTL does not always work as expected however, and there are some DNS servers that ignore TTL in an effort to cache more.

During the time between changing DNS and remote DNS servers getting the new IP, there will be connections to both servers. One possible solution to this is implement port forwarding on the old server to redirect requests to the new one. There are different ways to do this, such as port forwarding deamons, Apache doing proxying, etc. However I found the simplest method is using Linux's iptables. This of course assumes the old server is a Linux system.

Suppose we are on the old server and the new server is 5.6.7.8. We want to redirect HTTP and HTTPS traffic. The following commands will do it:

echo 1 > /proc/sys/net/ipv4/ip_forward
iptables -t nat -A PREROUTING -p tcp --dport 80 -i eth0 -j DNAT --to 5.6.7.8:80
iptables -t nat -A PREROUTING -p tcp --dport 443 -i eth0 -j DNAT --to 5.6.7.8:443
iptables -t nat -A POSTROUTING -o eth0 -j MASQUERADE

As soon as you run these commands, all connections are transparently sent to the new server. Just keep this on until there are no more requests coming to the old server.

I setup a Idotpc Ibox Crystal P4c small form factor PC with a Sangoma card and made it our T1 router through Bellsouth.

The PC is a 2.4gz Celeron with 1 gig RAM and 40 gig HD. It costs $500 and was the cheapest small form factor PC I found for the specs. Another $500 went to the T1 card, so for roughly $1k you have a Linux router. The Sangoma was not too difficult to setup. I had some hiccups using the 'stable' driver with Debian Woody, and was told by tech support to use the beta instead. That worked fine.

The small PC is rather nice looking, and has a purple light that looks cool in the dark. This is our first step at moving core services to small form factor PCs. We also plan to test a Mac Mini Debian server.

Contrast this with the monstrosity that is the Dell Poweredge 2800. We received one of these and when I saw the box on a wooden crate, I said WTF. Since when have Dell tower servers become so ridiculously large? The depth is 28 inches and it weighs 108 pounds!

Such big iron systems are becoming less and less appealing to me.

Today I installed 10 Hitachi Deskstar 120gig HDs into a Coraid box. My prediction was pretty much correct. I've setup the drives in a RAID 1+0 setup, where 5 pairs of mirrors are striped. I'm seeing about 24 MB/sec read, and 30 MB/sec write. Checkout the bonnie output.

The Daily Show mentioned a website that lists the worst things on TV. I had to check these out .

A friend sent me a link to someone criticizing Hula. He claims group calendaring should get people laid.

Came across the Hula project on Slashdot. This is an email and calendaring server that is striving to have a Gmail-like interface. The developers seem very optimistic.

I decided to download it to check it out. I have a CentOS 3.4 box at home, and installation was not too difficult. I was able to get it up an running to check out its webmail interface. Some notes:

• Download and install Subversion. Do a svn checkout as described on the Hula page
• Install the kernel-source RPM
• Because of some problems finding the atomic_ kernel functions, you need to do some trickery:

mv /usr/include/asm/atomic.h{,.old}
cp /usr/src/linux/include/asm/atomic.h /usr/include/asm/

• Configure with the following:

CPPFLAGS=-I/usr/kerberos/include ./configure --prefix=/opt/hula

The rest of the instructions should be fine. The system is definitely not ready for prime time, but it's cool to checkout. I hope it moves forward.

Google's front page links to a Valentine's Day search. Am I the only one who finds it hilarious that the first link has the text "Watch The St. Valentine's Day Massacre"? lol...

There's a very interesting article in the Feb 2005 issue of SysAdmin magazine titled Next Generation DHCP Deployments. Unfortunately the article is not available online.

It describes University of Kansas' DHCP setup, which has to deal with any crazy student plugging in their virus-laden PC into their network. They designed a very ingenious system that does many things. First off it does passive OS fingerprinting by looking at the DHCP request sent by a device. It turns out every device/OS sends a unique DHCP request packet with certain options sent. They've built a database of every possible device, from Linksys routers to Windows versions. They can deny/accept requests based on this data.

Another cool thing is they tag the DHCP request with VLAN information and are able to determine where exactly the device is. This lets them quickly track down security problems, and can be used to stop people from installing a hub at the network port connecting many systems to it.

So what about rogue DHCP servers that someone decides to setup? Well the network switches are configured to only allow a certain trusted port to send DHCP responses. Hmm, but I still don't understand how they handle systems setup with a static IP.

Anyhow, it reminded me of how much different it is working as sysadmin in a university. The problems there seem alot more grand than the problems at a small company, and the solutions are grand as well.

I heard a pretty cool cover of Britney Spears' Toxic on the radio by Local H. Searched on Itunes for it but wasn't there. Then I found it was downloadable as an MP3 from their site. Check it out, it's Nirvana doing Toxic . It sounds alot better than the original.

The latest update to the Linux on Mac Mini page informed me of how to burn ISO images in OS X. I have been normally using cdrecord, which has been giving me coasters unless I set the recording speed to 8. I wondered if there was a way to burn ISOs without cdrecord, and in fact there is. Disk Utility allows you to do it!

We received a Coraid shelf at work for some testing. I was always interested in testing out this system. The cost savings of being able to use commodity hard drives easily attachable to the network was very appealing. Here are my experiences with the product.

My main objective was to guage performance with software RAID and test failure scenarios. Each drive has an 100mb ethernet port. I was hoping for gigabit, but oh well. The preferred setup is to connect all drives to a separate switch with a gigabit uplink to the server. This way the aggregate throughput can scale well, and it seems to.

The Linux driver is very clean, and as you plug in new drives, they are automatically detected. On my 2.4 kernel, I get the found drives listed in /dev/etherd/stat:

/dev/etherd/e0.3 eth1 up
/dev/etherd/e0.1 eth1 up
/dev/etherd/e0.0 eth1 up
/dev/etherd/e0.2 eth1 up

Then you just use those devices just like hard drives. It is pretty amazing, and I'm impressed with how easy it was to get up and going. Now for the performance tests.

The max throughput in my tests to a specific drive, both read and write, was about 5 MB/sec. For a hard drive this is very slow, so RAID needs to be used to get more speed.

Using simple RAID 0 (striping across drives) results in the best speeds, but of course there is no redundancy. In such a case, the system seemed to scale well. I tested with 4 40gig hard drives, and my throughput, both and read and write, was about 20 MB/sec.

As expected, RAID 1 gives less than stellar performance. About 5 MB/sec read, and 5 MB/sec write when using 2 drives.

With RAID 5, I ran into lots of problems. In the 4 disk scenario, I got 20 MB/sec read, and 1 MB/sec write. Yes, that's 1 MB/sec. What is also wierd is a 3 disk RAID 5 did its resync at 1 MB/sec, whereas a 4 disk RAID 5 resync'ed at 3.5 MB/sec. I am still not sure what's going on here, but it appears to be something with the Linux RAID software or my CPU. I know that RAID 5 involves lots of XORing, and it is almost like this is CPU bound on my system (Dell Poweredge 1650 P3 1gz).

Thinking about this more, RAID 5 is not very appealing. I don't trust the rebuild times. Also if 2 drives die, you are dead. Next I tested RAID 1+0, which doesn't have these limitations. Well it does, but there is less chance of complete failure. This seems to be the best choice for us. With my 4 40 gig HDs, with RAID 1 across 2 drives, and RAID 0 across the 2 pairs, I see about 10 MB/sec both read and write. Now we are getting somewhere. I'm guessing this will continue to scale. We have ordered 10 120gig drives to put in the box, and if my predictions are right, I should see about 20-25 MB/sec. I plan to use this for a mail server, and I would consider those rates acceptable.

The system shelf is rather bare. Don't expect visual appeal like an IBM or Dell system. The shelf is smaller than it looks, which is good for us because we don't have much space. The blades are bare circuit boards, rather tough to get a grip and slide out when the drive is attached, and when the system is live I do sometimes wonder if I'll get shocked hot swapping them .

The technical support is very good. They are the ones who wrote the ATA over Ethernet driver, and I hear it is in the latest 2.6 kernels.

Well it looks like the fix I made was actually implemented in Communigate, so my script will soon go into my museum of dead code. There must have been other users requesting this:

* HTTP: the CalendarDataDel realm is supported now (same as the CalendarData realm, but PUT operations delete all existing iCalendar items first).

Well was fun to work on either way, and this fix is not in the stable release of Communigate yet.

I had some questions for the Mozilla Calendar development group. I posted a message to the newsgroup but didn't get any response. I did some research and found this bug which doesn't appear to have been updated in over 6 months. I decided to send an email to the email address the bug is assigned to, which is supposedly the Mozilla Calendar development group. I got an undeliverable message back about the account not existing . I even sent an email to the QA contact but he stated he is not part of that group.

As my message stated, I've been having ideas of adding code to Mozilla Calendar/Sunbird to store data in IMAP. It seems a large undertaking but it seems at least a few people are interested in developing this. However, it is not clear whether this is a good idea or not. Mark Crispin of UW IMAP tells me:

Basically, IMAP was designed for messaging, not calendaring; so it lacks tools which are needed for calendaring, and tools which could be used for calendaring aren't quite right.

There is a working group at IETF that is working on calendaring.  They haven't progressed very well, and I suspect that if we try to duplicate their work in IMAP we may have the same problems.

None of this is intended to discourage you from developing calendaring in IMAP.  Rather, I'm just sounding some notes of caution.

Good advice. This needs to be thought through. I'm torn between a 'build it and they will come' attitude verses wait until something else comes along. But judging from my research so far, robust calendaring seems to be at a standstill.

Update: I found this blog which decribes the future of Sunbird. It looks like they are going towards CalDev as the standard.

Dealing with groupware calendaring issues has been taking alot of my time. I usually like to do some programming every now and then to sharpen my skills. Programming to me falls under the use it or lose it category. I work on small projects occasionally as you can see from my home page, and needed to work on something new. It's sort of a catharsis for me .

Python is my language of choice these days. The language never ceases to amaze me. It's just beautiful and simple. I've noticed that the end result usually has less bugs than something I've written in other languages.

So, during my tests of group calendaring and Mozilla Sunbird, I found out that when used with the Communigate Pro mail server, deleting calendar events on a remote calendar does not work. It has to do with Sunbird issuing a HTTP PUT for its calendar data, and Communigate assuming this to be an append of data, rather than a replace.

Mozilla 'shared' calendaring in its current state is a complete hack that is not scalable at all. It involes first retrieving the whole calendar from a server, making a change, and then posting the whole calendar back to the server. Think about if you have a large calendar with 1meg of data. This gets extremely slow and inefficient.

Regardless, I wanted to get Sunbird working with Communigate. I looked at the available operations on CG's calendar interface, and realized what I need to do is convert the HTTP PUT from Sunbird into a combination of a HTTP DELETE and HTTP PUT. So began the idea of my next script.

It took me about 5 hours to make a multithreaded HTTP proxy to do this in Python. If you examine the code, you will see just how simple this is. Python includes a HTTP server class that is easily extended, and I got alot of help from the excellent book Foundations of Python Network Programming. What's also amazing is to make it multithreaded, I just needed to inherit a threading class! Nothing else at all, it just works!

I'll spare the details, but one thing I especially liked was this line of code:

self.respond_client(*self.cg_request('GET'))

This is the implementation of the GET request. It's a strange syntax, but here is what it's doing. respond_client() needs to have 2 arguments passed to it. cg_request() returns a tuple with 2 values. A tuple is essentially a read-only list. The * prepended to the function call makes the tuple expand into function arguments. Amazing I tell you!

It seems I'm digging myself deeper and deeper into group calendaring. Pete pointed out this interesting article, and we talked about an idea of getting Sunbird to store entries via IMAP in this iMIP format. The difficulty level of this looks to be high, but man it would be cool. I'm reading up on XUL now.

I posted the following on the Mozilla forum:

Proposal for Sunbird to support iMIP for calendar storage

As described here and other places, I think that it would be great if storage of calendar entries can be done on an IMAP folder.

Communigate Pro seems to do this internally, storing in the iMIP format. Essentially each calendar entry is a mail message in a folder. They implement a MAPI connector for Outlook that works over IMAP to access the calendar.

When deleting entries, just the one message is deleted. Assuming storage is maildir, this works pretty well. Likewise, additions just add one message. This seems alot better than downloading the whole calendar in iCal format, making a change, and posting the whole calendar back. I've run into slowness problems with large calendars.

What's great about this is if Sunbird could speak IMAP and use this storage, it would interoperate with Communigate's own web calendaring, and also could be used by any IMAP server. It seems standard and open enough that other clients would be able to work with it.

Does anyone have thoughts on this? Any plans to implement this currently in Sunbird? I would honestly like to embark on such a project, but I don't know too much about the Mozilla platform. If someone could provide some pointers on where I could start that would be great.

Today it was pretty much decided at work that we want to move away from Outlook. I had originally been looking into MAPI interfaces for Outlook, and doing testing with Communigate. However, it was brought up that if we implement tight integration with Outlook we would become dependent on it. Better to cut the umbilical cord sooner than later.

Our idea is to instead go with Mozilla Thunderbird for mail via IMAP and Sunbird for calendaring. We would decouple calendaring from mail. Contacts would be done via LDAP. There are alot of issues to consider, but the main ones are offline support and shared calendaring.

Thunderbird with IMAP will work fine for mail. I'm just not sure of the offline support. It is there, but you basically mark which folders you want to have available offline and it seems like you need to synchronize mail before going offline. I just know this may be problematic for users.

Sunbird can work off of a remote calendar as described here. With Communigate you can implement access restrictions to your calendar that a group of users could read/write to. This implements a basic shared calendar with delegation. Since these are just ical files, Communigate's web interface can also be used to access the calendar. This is very cool. You could most likely do this all without Communigate, but CG just makes it easier.

So what you have with the above setup is a calendar that others can access. Before you add a calendar entry in Sunbird, it first downloads your calendar from the server, makes the change, then posts it back. The one problem I have is it is prompting for the login information every time Sunbird starts and there is no option to save the password. It seems you can use a http://user:\pass@... format for the URL to post, but this seems rather scary to have your password cleartext. Other than that it seems to work well, but I'm not sure how this will work if someone has a very large calendar. Presumably the ical data is small.

The real problem is doing this offline. Sunbird appears to attempt retrieving the calendar from the server before you can make a change. It essentially ignores anything you do offline. It doesn't hang in OS X, but I've heard it does in Windows. It would be nice to configure it to allow some sort of offline editing.

I thought of even reading up on XUL and possibly coding something up that would attempt to post the local calendar every X minutes, ignoring errors, instead of first attempting to retrieve the remote one first.

So in a nutshell the problems are:

• Thunderbird requiring user intervention for synchronizing folders for offline work
• Sunbird prompting for login information at startup for remote calendar
• Sunbird not able to work offline with remote calendar

I would love to go to a standard format and ditch Outlook. If everyone's data is stored in these formats, it will be much easier to code up scripts to implement any other needs as they arise (meeting scheduling, etc). But it will be an uphill battle for sure.