Sporadic low network performance

[redhat-list at demner.com]

I'm running RHEL ES (kernel version 2.4.21-27.0.1.ELsmp), and have
sporadic network problems on our production server.  90% of the time the
transfer rate is 500K/s but 10% it's 15K/s.  I don't know if it's my
network provider or something wrong with the server.  The tests on the
network so far have been inconclusive but points to a server issue.

This box is hardly hit - 10% CPU max, min 350M (of 1G) free memory.  To
narrow down the problem, so far on the server I've tried:

- VolanoMark local benchmark (same during slowdown and not)
- free (350M+ free always)
- top (10% CPU max always)
- cat /proc/mdstat (everything looks fine with the RAID array)
- all log files (nothing unusual)

Over the network we've tried

- VolanoMark network benchmark - slowdown seen during test
- ftp/sftp/apache tests to server from multiple external locations - all
show the slowdown symptoms
- ftp/sftp/apache tests from multiple external locations to server -
inconclusive
- ftp/sftp/apache tests to server from within ISPs internal network - no
slowdown
- ftp/sftp/apache tests to server from within same subnet - no slowdown
- ISP unplugging the server and plugging in their server - inconclusive
since it takes 15 minutes to update their ARP table and the one time we
tried it so far it was slow before the test, but fast after it.

I'm fast running out of ideas, as is the ISP.  Can anyone please please
please help me resolve this situation?

Thanks in advance,

David Demner