Weird TCP problem

[Gordon Messmer]

Last week, I started seeing very strange behavior in one of the networks 
that I manage.

The office LAN uses a Linux firewall which masquerades their 
workstations over their DSL connection.  There are probably ~75 
workstations in the office LAN.

Their mail server is in a collocated facility nearby.  That server has 
an RFC1918 address; its router does SNAT to forward packets to the system.

Here's the weird part: If a machine running Linux in the office lan 
attempts to connect to the mail server on any TCP port, there's a small 
chance that the server will simply ignore the SYN packets.  It doesn't 
log any errors.  If I'm running tcpdump, I see the incoming SYN packets, 
but no reply.  If I use iptables to log the packets, information about 
the packet is saved in the messages file.  If I capture the packets and 
use wireshark to analyze them, I don't see anything odd: the checksums 
are good and I can't see any difference between a SYN packet that gets a 
SYN+ACK and one that's ignored (beyond the obvious: different timestamps 
and checksums).  The problem doesn't seem to affect Windows workstations 
in the office LAN.  As far as I can tell, only SYN packets are dropped. 
  I don't see delays in established connections.

I've attached a file that contains, first, the output of tcpdump which 
shows packets to or from the office's firewall address, as recorded by 
the destination server.  The first four SYN packets are ignored, but the 
kernel proceeds with the TCP handshake after the fifth SYN packet.

Second, the file contains the log messages which are recorded as a 
result of these iptables rules:

iptables -A INPUT -p tcp -s officefw --dport 22 -j LOG
iptables -A INPUT -p tcp -s officefw --dport 22 -j ACCEPT

Those are the only iptables rules present on the server accepting the 
connections.

Both of those appear to indicate that the server in the colo facility is 
receiving the SYN packets.  What possible reasons are there that it 
would not reply with SYN+ACK?
-------------- next part --------------
19:08:43.751579 IP officefw.57948 > remoteserver.ssh: S 3347102294:3347102294(0) win 5840 <mss1460,sackOK,timestamp 2705398041 0,nop,wscale 7>
19:08:46.751136 IP officefw.57948 > remoteserver.ssh: S 3347102294:3347102294(0) win 5840 <mss1460,sackOK,timestamp 2705401041 0,nop,wscale 7>
19:08:52.749305 IP officefw.57948 > remoteserver.ssh: S 3347102294:3347102294(0) win 5840 <mss1460,sackOK,timestamp 2705407041 0,nop,wscale 7>
19:09:04.747287 IP officefw.57948 > remoteserver.ssh: S 3347102294:3347102294(0) win 5840 <mss1460,sackOK,timestamp 2705419041 0,nop,wscale 7>
19:09:28.741854 IP officefw.57948 > remoteserver.ssh: S 3347102294:3347102294(0) win 5840 <mss1460,sackOK,timestamp 2705443041 0,nop,wscale 7>
19:09:28.742540 IP remoteserver.ssh > officefw.57948: S 3324501337:3324501337(0) ack 3347102295 win 5792 <mss1460,sackOK,timestamp 15960089 2705443041,nop,wscale 7>
19:09:28.783886 IP officefw.57948 > remoteserver.ssh: . ack 1 win 46 <nop,nop,timestamp 2705443083 15960089>
19:09:28.789814 IP remoteserver.ssh > officefw.57948: P 1:21(20) ack 1 win 46 <nop,nop,timestamp 159601372705443083>
19:09:28.829114 IP officefw.57948 > remoteserver.ssh: . ack 21 win 46 <nop,nop,timestamp 2705443129 15960137>


Sep  4 19:08:43 remoteserver kernel: IN=eth0 OUT= MAC=00:30:48:97:5a:3a:00:0a:b8:8e:53:29:08:00 SRC=officefw DST=remoteserver LEN=60 TOS=0x00 PREC=0x00 TTL=56 ID=41807 DF PROTO=TCP SPT=57948 DPT=22 WINDOW=5840 RES=0x00 SYN URGP=0 
Sep  4 19:08:46 remoteserver kernel: IN=eth0 OUT= MAC=00:30:48:97:5a:3a:00:0a:b8:8e:53:29:08:00 SRC=officefw DST=remoteserver LEN=60 TOS=0x00 PREC=0x00 TTL=56 ID=41808 DF PROTO=TCP SPT=57948 DPT=22 WINDOW=5840 RES=0x00 SYN URGP=0 
Sep  4 19:08:52 remoteserver kernel: IN=eth0 OUT= MAC=00:30:48:97:5a:3a:00:0a:b8:8e:53:29:08:00 SRC=officefw DST=remoteserver LEN=60 TOS=0x00 PREC=0x00 TTL=56 ID=41809 DF PROTO=TCP SPT=57948 DPT=22 WINDOW=5840 RES=0x00 SYN URGP=0 
Sep  4 19:09:04 remoteserver kernel: IN=eth0 OUT= MAC=00:30:48:97:5a:3a:00:0a:b8:8e:53:29:08:00 SRC=officefw DST=remoteserver LEN=60 TOS=0x00 PREC=0x00 TTL=56 ID=41810 DF PROTO=TCP SPT=57948 DPT=22 WINDOW=5840 RES=0x00 SYN URGP=0 
Sep  4 19:09:28 remoteserver kernel: IN=eth0 OUT= MAC=00:30:48:97:5a:3a:00:0a:b8:8e:53:29:08:00 SRC=officefw DST=remoteserver LEN=60 TOS=0x00 PREC=0x00 TTL=56 ID=41811 DF PROTO=TCP SPT=57948 DPT=22 WINDOW=5840 RES=0x00 SYN URGP=0 
Sep  4 19:09:28 remoteserver kernel: IN=eth0 OUT= MAC=00:30:48:97:5a:3a:00:0a:b8:8e:53:29:08:00 SRC=officefw DST=remoteserver LEN=52 TOS=0x00 PREC=0x00 TTL=56 ID=41812 DF PROTO=TCP SPT=57948 DPT=22 WINDOW=46 RES=0x00 ACK URGP=0 
Sep  4 19:09:28 remoteserver kernel: IN=eth0 OUT= MAC=00:30:48:97:5a:3a:00:0a:b8:8e:53:29:08:00 SRC=officefw DST=remoteserver LEN=52 TOS=0x00 PREC=0x00 TTL=56 ID=41813 DF PROTO=TCP SPT=57948 DPT=22 WINDOW=46 RES=0x00 ACK URGP=0