Strange network (?) problems

[Chris St. Pierre]

I have recently been experiencing a very strange problem with a number
of different RHEL 4 boxes providing a number of different services.
Basically, it appears that packets are occasionally delayed or
dropped, apparently randomly.

I first noticed it on our Samba server.  Mac clients were complaining
that, ever 15-45 minutes, they would get a message that the connection
to the server had been dropped.  Then we noticed it with email: IMAP
connections, usually when FCCing, would hang indefinitely.  Now I'm
seeing it with SVN checkouts and commits.

In every case, one party to the transaction claims to have sent a
response but the other party claims to never receive it.

As IMAP is the easiest to debug (thank you, Cyrus telemetry logging!),
I've focused mostly on that.  In the case of the hanging FCC, the
telemetry logs show the following activity:

<1171566733<000000cb RLIST "" INBOX.Sent
>1171566733>* LIST (\HasNoChildren) "." "INBOX.Sent"
000000cb OK Completed (0.000 secs 2 calls)
<1171566735<000000cc APPEND INBOX.Sent {4301}
>1171566735>+ go ahead

So the IMAP server claims that it's sent the go-ahead.  On the other
end, though, the client claims to never receive it, and eventually
times out because of a lack of response from the server.
Occasionally, the server "go ahead" will be received after 10-15
minutes and the transaction will continue.

Since all of these are sequential protocols, my best guess was that a
packet was getting dropped, but our network equipment shows no
significant errors of that sort.

I'm not even sure where to go from here WRT troubleshooting; I haven't
found a way to reliably reproduce the problem.  Does anyone have any
ideas of what I might look at to narrow down the issue?  I've thought
of doing a traffic capture, but it only happens to me (IMAP with Pine)
1-2 times a day -- eight hours of packets would not be fun to look
through. :(

All suggestions welcome!

Thanks!

Chris St. Pierre
Unix Systems Administrator
Nebraska Wesleyan University
----------------------------
Never send mail to thobrux at nebrwesleyan.edu