Disk Space Issues - cvs-int/build hosts - (includes general note on Nagios Disk notifications)

[Nigel Jones]

Hey guys,

I seem to have deleted the nagios notification that I want to mention in 
particular, but as I noted in SOP/Nagios the %ages noted in the e-mails 
are what is left, so inode=99% doesn't mean that 99% of the inodes are 
used, it means 99% are still free.

Anyway, what this means, is that when nagios has been complaining about 
cvs-int recently, in particular the fact that /git has reached WARNING.  
After a bit of hunting around, I found /repo/pkgs using 168GiB of the 
192GiB available, which is understandable, Fedora has got huge.

Problem here, is that there are a LOT of old tarballs in that folder, 
which leaves me wondering if we should do a spring clean ~1 mo after 
release.

Lets take banshee for example, a package which I adopted....

$ ls -l /repo/pkgs/banshee/
total 88
drwxrwsr-x  3 apache repoextras 4096 May  3  2006 banshee-0.10.10.tar.gz
drwxrwsr-x  3 apache repoextras 4096 Aug  7  2006 banshee-0.10.11.tar.gz
drwxrwsr-x  3 apache repoextras 4096 Aug 24  2006 banshee-0.10.12.tar.gz
drwxrwsr-x  3 apache repoextras 4096 Mar  4  2006 banshee-0.10.6.tar.gz
drwxrwsr-x  3 apache repoextras 4096 Mar  7  2006 banshee-0.10.7.tar.gz
drwxrwsr-x  3 apache repoextras 4096 Mar 14  2006 banshee-0.10.8.tar.gz
drwxrwsr-x  3 apache repoextras 4096 Apr 16  2006 banshee-0.10.9.tar.gz
drwxrwsr-x  3 apache repoextras 4096 Feb  2  2007 banshee-0.11.5.tar.bz2
drwxrwsr-x  3 apache repoextras 4096 Mar  7  2007 banshee-0.12.0.tar.bz2
drwxrwsr-x  3 apache repoextras 4096 Apr  5  2007 banshee-0.12.1.tar.bz2
drwxrwsr-x  3 apache repoextras 4096 Aug  7  2007 banshee-0.13.0.tar.bz2
drwxrwsr-x  3 apache repoextras 4096 Aug 31  2007 banshee-0.13.1.tar.bz2
drwxrwsr-x  3 apache repoextras 4096 Jan 14 15:58 banshee-0.13.2.tar.bz2
drwxrwsr-x  3 apache repoextras 4096 Apr 13 01:51 banshee-1-0.98.3.tar.bz2
drwxrwsr-x  3 apache repoextras 4096 May 10 03:41 banshee-1-0.99.1.tar.bz2

$ du -sh /repo/pkgs/banshee/
26M     /repo/pkgs/banshee/
(Approximately 2M/package)
At the most there should only be 4 tar balls there (R-2, R-1, R, 
Rawhide), R-2 only valid for one month after R has released.

Another couple of examples:
$ du -sh /repo/pkgs/kernel/
2.6G    /repo/pkgs/kernel/
(900ish tarballs dating back to 2004)

$ du -sh /repo/pkgs/kdeedu
1.4G    /repo/pkgs/kdeedu
(48 tarballs from KDE-3.0)

With plans such and Hans' plans of creating a 500M vegastrike data SRPM 
and the size growth and update schedules of some packages we are going 
to have these nightmares more and more frequently.

Two solutions I can think of:

Create a script, go thru ALL non dead.package's grab the tarball name 
from sources.list and basically create a bit of a database of what we 
are using, scan through /repo/pkgs and either:
* Move old tarballs to some archiving system (another use for archive.fp.o?)
* Delete old tarballs
Or throw more diskspace at cvs-int

Even if were to only remove 15% of the tarballs there (this is a very 
cautious estimate of the number of stale tarballs) we could potentially 
reach 72% diskspace available on that mount (down from 82%) - note this 
is very simplistic math, in essence, we could be no better off if we 
only removed the small stale tarballs :).

Diskspace isn't cheap, so I like delete old tarballs, I also like this 
option because it's not like they disappear completely, they should be 
in the src.rpm's already on archive.fp.o and if we accidentally delete 1 
or 2 that are still needed, well grab it from src.rpm...

This leads on to my second item...

xenbuilder2 has run out of diskspace in /, it's down to 32M, thankfully 
koji has disabled it so it's safe for now, but wouldn't it be nice to 
throw say a 50GB partition dedicated solely for /var/lib/mock & 
/mnt/build?  Yes, yes, I know money, but once again, builds are getting 
bigger so 'it'd be nice'.

Just thought I'd throw these two thoughts into the open, let the 
discussion begin!