Viraj's Weblog

We received a Coraid shelf at work for some testing. I was always interested in testing out this system. The cost savings of being able to use commodity hard drives easily attachable to the network was very appealing. Here are my experiences with the product.

My main objective was to guage performance with software RAID and test failure scenarios. Each drive has an 100mb ethernet port. I was hoping for gigabit, but oh well. The preferred setup is to connect all drives to a separate switch with a gigabit uplink to the server. This way the aggregate throughput can scale well, and it seems to.

The Linux driver is very clean, and as you plug in new drives, they are automatically detected. On my 2.4 kernel, I get the found drives listed in /dev/etherd/stat:

/dev/etherd/e0.3 eth1 up
/dev/etherd/e0.1 eth1 up
/dev/etherd/e0.0 eth1 up
/dev/etherd/e0.2 eth1 up

Then you just use those devices just like hard drives. It is pretty amazing, and I'm impressed with how easy it was to get up and going. Now for the performance tests.

The max throughput in my tests to a specific drive, both read and write, was about 5 MB/sec. For a hard drive this is very slow, so RAID needs to be used to get more speed.

Using simple RAID 0 (striping across drives) results in the best speeds, but of course there is no redundancy. In such a case, the system seemed to scale well. I tested with 4 40gig hard drives, and my throughput, both and read and write, was about 20 MB/sec.

As expected, RAID 1 gives less than stellar performance. About 5 MB/sec read, and 5 MB/sec write when using 2 drives.

With RAID 5, I ran into lots of problems. In the 4 disk scenario, I got 20 MB/sec read, and 1 MB/sec write. Yes, that's 1 MB/sec. What is also wierd is a 3 disk RAID 5 did its resync at 1 MB/sec, whereas a 4 disk RAID 5 resync'ed at 3.5 MB/sec. I am still not sure what's going on here, but it appears to be something with the Linux RAID software or my CPU. I know that RAID 5 involves lots of XORing, and it is almost like this is CPU bound on my system (Dell Poweredge 1650 P3 1gz).

Thinking about this more, RAID 5 is not very appealing. I don't trust the rebuild times. Also if 2 drives die, you are dead. Next I tested RAID 1+0, which doesn't have these limitations. Well it does, but there is less chance of complete failure. This seems to be the best choice for us. With my 4 40 gig HDs, with RAID 1 across 2 drives, and RAID 0 across the 2 pairs, I see about 10 MB/sec both read and write. Now we are getting somewhere. I'm guessing this will continue to scale. We have ordered 10 120gig drives to put in the box, and if my predictions are right, I should see about 20-25 MB/sec. I plan to use this for a mail server, and I would consider those rates acceptable.

The system shelf is rather bare. Don't expect visual appeal like an IBM or Dell system. The shelf is smaller than it looks, which is good for us because we don't have much space. The blades are bare circuit boards, rather tough to get a grip and slide out when the drive is attached, and when the system is live I do sometimes wonder if I'll get shocked hot swapping them .

The technical support is very good. They are the ones who wrote the ATA over Ethernet driver, and I hear it is in the latest 2.6 kernels.