Viraj's Weblog

At work they were testing a Business Continuation Plan, which pretty much is a setup that allows the company to continue business in the event of a disaster.

For Internet access we had a simple Linksys router with a DSL line. During the test, Internet connectivity was determined to be working by going to Google and making sure the page loaded up. A tester said they couldn't get their email, so I was pulled in to figure out what was wrong.

So I started simple, doing some pings to www.google.com. Response was ok. Then went to the Google site in Firefox and Internet Explorer without issues. Then I tried telnetting to our mail server port 143 (IMAP), 25 (SMTP), 80 (HTTP), 443 (HTTPS), and 110 (POP). All gave me connections, and I was able to send simple commands without issues. But Outlook was crapping out.

Outlook was setup to use CommuniGate Pro's MAPI connector, which essentially converts everything to IMAP. I really didn't think the connector was at fault because it has worked fine in the past.

I then tried going to our webmail and didn't get anything! What was going on? I could connect via telnet, but not via any browser. Next I tried another site besides Google, such as www.cnn.com. That page didn't load up either... neither did Yahoo. I thought maybe it was network routing issue, but I could ping the destinations just fine. And even when CNN didn't load up in the browser, I could still telnet to port 80. Some screwed up proxying by Bellsouth DSL?

I thought it was a browser issue, but both IE and Firefox has same results. Why would Google load up but not other sites? Next I had to connect a real system (i.e. Unix) to this network and do some tcpdumps. I started a tcpdump, and ran a wget:

wget -O - -q http://www.cnn.com/

I got nothing back as expected, and saw in the tcpdump:

sudo tcpdump -nlp -i en1 port 80
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on en1, link-type EN10MB (Ethernet), capture size 96 bytes
20:11:27.120888 IP 192.168.1.100.54924 > 64.236.24.28.80: S 1892952170:1892952170(0) win 65535 [mss 1460,nop,wscale 0,nop,nop,timestamp 4252212886 0]
20:11:27.187946 IP 64.236.24.28.80 > 192.168.1.100.54924: S 3676883317:3676883317(0) ack 1892952171 win 5840 [mss 1460]
20:11:27.188062 IP 192.168.1.100.54924 > 64.236.24.28.80: . ack 1 win 65535
20:11:27.188228 IP 192.168.1.100.54924 > 64.236.24.28.80: P 1:582(581) ack 1 win 65535

The 3-way handshake is there, but once the GET request is sent, nothing comes back.

Next I suspected some problem with the DSL line or the Linksys router. I looked through the web interface and saw an option for SPI, or Stateful Packet Inspection, with features to block ActiveX and other crap underneath it. I tried disabling SPI, and did my wget again. It worked!

So it boiled down to the Linksys SPI feature causing problems. It could be that it just doesn't work well for many sites. This explained why the Google page came up: because it was relatively simple HTML. Anything complex, such as CNN, would hang. I checked I was running the latest firmware on the Linksys, and I was.

Lesson learned, Linksys SPI (Stateful Packet Inspection) sucks. Don't use it.