Restart TG apps for high mem-usage

[Toshio Kuratomi]

Mike McGrath wrote:
> Toshio Kuratomi wrote:
>> Mike McGrath wrote:
>>> Bill Nottingham wrote:
>>>> Toshio Kuratomi (a.badger at gmail.com) said:
>>>>> Here's a short script to test our TG apps run via supervisor for 
>>>>> excessive memory usage and restart them if necessary.  We could run 
>>>>> this via cron in alternate hours on each app server.  Does this 
>>>>> seem like a good or bad idea to people?
>>>>>     
>>>>
>>>> It's a good idea if it's needed, but it's a bad idea that it is 
>>>> needed. What's
>>>> wrong with TG that it leads to this situation?
>>>>   
>>>
>>> I was wondering this myself, I know smolt recently had some major 
>>> changes to keep memory usage down.  Which TG apps are having this 
>>> issue and how often?  I know MM uses a lot of memory but, AFAIK, it 
>>> was determined that there's not much of a leak if there is one and 
>>> that all of that memory is actually used.
>>>
>> Looks like smolt was upgraded just before Thanksgiving so it could be 
>> that we've plugged the leaks we had to deal with that inspired me to 
>> write this.  Would it be a good idea to have this in place anyways? 
>> With it periodically checking, we would find out that we had problems 
>> when cron emails us a notice that the script had to restart a process. 
>> Without it, we'll be notified when nagios or a user tells us they're 
>> getting timeouts.
> 
> I think its a good idea if for no other reason then allows us to more 
> actively monitor this stuff, we'll get notified when the app restarts.  
> +1 from me with the intention that, over time, we get fewer and fewer 
> restarts.
> 
Cool.  I'll check it in and set up a cron job.

One further piece of information since I have output from testing this 
on app3 yesterday:

AppName        Uptime  RSS 11/25   RSS 11/26
mirrormanager  2d4h    714336      962268
packagedb      --- restarted 13h ago --
smolt          5d3h    299556      299556
transifex      5d3h    42744       42768

-Toshio