Builder monitoring

[Mike McGrath]

Jeremy Katz wrote:
> On Wed, 2007-08-15 at 23:03 -0500, Dennis Gilmore wrote:
>   
>> Once upon a time Wednesday 15 August 2007, Mike McGrath wrote:
>>     
>>> We've recently started taking a closer look at the builders and as a
>>> result we've been getting many notifications.  As a baseline, the
>>> builders need some work.  One thing we've been getting alerted of is
>>> memory usage, in particular less than 10% swap free.
>>>
>>> We have a couple of options.
>>>
>>> 1) stop monitoring swap on the builders
>>> 2) reduce resolution on the builders (check less often and require more
>>> failures in a row before notification, right now is 3 failures in a row
>>> results in a notification)
>>> 3) increase swap (presently about 1G / core so between 2G and 8G of swap
>>> on the boxes)
>>>       
> [snip]
>   
>> Some builders could really do with some more ram.  but in the mean time 
>> perhaps up swap to 1.5g/core 
>>     
>
> I'd say a combination of 2 and 3 is going to get the best results of
> improving reliability and reducing false positives
>
>   

Created a ticket:

https://hosted.fedoraproject.org/projects/fedora-infrastructure/ticket/124

    -Mike