Apache webserver outage - need help with forensics

[Bob Brennan]

On 4/14/05, Nigel Wade <nmw at ion.le.ac.uk> wrote:
> Bob Brennan wrote:
> > I have a server which went completely unresponsive today on port 80
> > for 20 minutes and would appreciate any pointers as to what might have
> > happened.
> >
> > A bit of background:
> > * FC3, Up2Date
> > * The Apache webserver serves a dozen virtual websites
> > * Sendmail + Dovecot + Squirrelmail for all sites
> > * Spamassasin recently activated (yesterday)
> >
> > The problem + observations:
> > * All websites were inaccessible from 14:00 gmt to 14:20 today
> > * The mailserver was running and responsive during that time
> > * FTP was running and responsive during that time
> > * telnet theServer.com 80 timed out with no connection during that time
> >
> > What I checked:
> > * all access_log and error_log for all sites - showed 5 users using
> > the sites at the time but nothing unusual
> > * no evidence of a DOS attack (that I could see)
> > * no records of anything unusual in system logs
> > * no accesses or errors in any of the http logs during that time
> >
> > Thankfully the webserver came back as if by magic after 20 minutes and
> > was immediately responsive.
> >
> > Any insights into anything else I can check? Needless to say an
> > embarassing incident for a webmaster who whould like to prevent it
> > happening again.
> >
> > Thanks in advance,
> > bob
> >
> 
> Maybe either a deliberate or unintentional DoS attack.
> 
> How many clients is your server configured to handle simultaneously? Maybe
> there was a problem, or some deliberate attack, which meant the established
> clients communications stuck and no new client connections could be accepted.
> 
> Did you have netstat output to show what connections were established to
> port 80 at the time?

The number of simultaneous clients is the FC3 default. The system is
somewhat ram-bound at only 256m and I have experienced swapping
slowdowns in the past but it's only seconds of delay, this was 20
minutes. Also it was only httpd missing, other services were normally
responsive.

I have searched all access_logs and error_logs for all of my domains
around that time and there was no unusual activity.

I can view a current netstat but can't find any log or history
information for netstat. Where might I find that?