502 Bad Gateway

[Mike McGrath]

Thomas Chung wrote:
> On 4/5/07, Thomas Chung <tchung at fedoraproject.org> wrote:
>> Are we experiencing an issue with our wiki?
>> I'm seeing following all pages:
>>
>> Bad Gateway
>>
>> The proxy server received an invalid response from an upstream server.
>> Apache/2.0.52 (Red Hat) Server at fedoraproject.org Port 80
>
> It seems the problem has been resolved now. :)
> Regards,
This was actually one of our worst crashes.  I've got some plans I'm 
working on but we'll need the netapps for them to work.  Here's a 
roundup of what happened.

App2 (where the wiki is actually stored) was having an issue with httpd 
starting.  I haven't looked into this error (yet) but here's the message:

[Thu Apr 05 11:34:17 2007] [error] (28)No space left on device: 
mod_python: Failed to create global mutex 1 of 4 (/tmp/mpmtx309251).

I verified that the box had enough space and inodes (we're talking 70% 
free).  The box was behaving strangely so I decided to reboot it.  
Normally not a problem except this box still had the older iptables 
rules on it which causes excessive logging.  With connections coming 
into it from the proxy servers and app1 the box quickly got 
overwhelmed.  It locked and I couldn't log in via the console so I 
killed the box.  When it came back up the grub settings were off so I 
could not console into it to see why the boot was hanging (I suspect 
fsck).  We couldn't PXE that box because its on a different network from 
the DHCP server so no rescue mode.  Completely lucky for me, I happened 
to be here.  Stuck a CD in and we were off.

This was bad and could have been much worse.  I'm going to take a look 
at our options to make sure it doesn't happen again.

    -Mike