Restart TG apps for high mem-usage

[Toshio Kuratomi]

Luke Macken wrote:
> On Sun, Nov 25, 2007 at 01:00:53PM -0800, Toshio Kuratomi wrote:
>> Here's a short script to test our TG apps run via supervisor for excessive 
>> memory usage and restart them if necessary.  We could run this via cron in 
>> alternate hours on each app server.  Does this seem like a good or bad idea 
>> to people?
> 
> Probably not a bad idea; I think koji does something similar with
> apache.  However, I don't think we need this for bodhi, at least for the
> moment.  The only time bodhi's memory usage jumps is when it's pushing
> updates -- so if we were to use this script for bodhi, it would have to check
> if it is currently running mash.  But for now, I'm not sure that it is necessary
> seeing as how most of the time puppetd eats more memory than bodhi.
> 
Sounds good.  I've taken both Bodhi and transifex out of the script for 
now as neither one is load balanced (the idea being that the apps which 
are load balanced should continue to serve requests off the other 
instance while one is restarting).

I've also changed it to take a different memory limit for each app.  It 
currently has some generous guesses as to what the memory limit should 
be.  I'm running a cron that logs the rss of the apps on app3&4 and will 
refine the limits after we have more data.

-Toshio