Disk Space Issues - cvs-int/build hosts - (includes general note on Nagios Disk notifications)

[Mike Bonnet]

On Tue, 2008-05-13 at 08:58 -0500, Dennis Gilmore wrote:
> On Tuesday 13 May 2008, Nigel Jones wrote:
> > Hey guys,
> >
> > I seem to have deleted the nagios notification that I want to mention in
> > particular, but as I noted in SOP/Nagios the %ages noted in the e-mails
> > are what is left, so inode=99% doesn't mean that 99% of the inodes are
> > used, it means 99% are still free.
> >
> > Anyway, what this means, is that when nagios has been complaining about
> > cvs-int recently, in particular the fact that /git has reached WARNING.
> > After a bit of hunting around, I found /repo/pkgs using 168GiB of the
> > 192GiB available, which is understandable, Fedora has got huge.
> >
> > Problem here, is that there are a LOT of old tarballs in that folder,
> > which leaves me wondering if we should do a spring clean ~1 mo after
> > release.
> <snip>
> 
> We already have plans to move the lookaside cache to the netapp.  at least 
> that was the last plan i was aware of.
> 
> > Diskspace isn't cheap, so I like delete old tarballs, I also like this
> > option because it's not like they disappear completely, they should be
> > in the src.rpm's already on archive.fp.o and if we accidentally delete 1
> > or 2 that are still needed, well grab it from src.rpm...
> >
> > This leads on to my second item...
> >
> > xenbuilder2 has run out of diskspace in /, it's down to 32M, thankfully
> > koji has disabled it so it's safe for now, but wouldn't it be nice to
> > throw say a 50GB partition dedicated solely for /var/lib/mock &
> > /mnt/build?  Yes, yes, I know money, but once again, builds are getting
> > bigger so 'it'd be nice'.
> 
> xenbuilder1 and hammer2 are the oldest boxes we have in the buildsystem.  
> closely followed by ppc1  xenbuilder2 is also quite an old box now.  im 
> cleaning up /var/lib/mock on xenbuilder2 now.  failed builds dont get cleaned 
> up automatically.  we probably should reap them more often.  

Buildroots for failed Koji builds do get cleaned up after 4 hours (to
give someone time to diagnose the problem if desired).  However,
xenbuilder2 is also running plague, which never cleans up buildroots as
far as I know.