change request (/mnt/koji fs issues)

[Mike McGrath]

So we're in a bit of trouble.  Right now the ext3 filesystem on /mnt/koji
is in a sort of funky state.  There's a section of the filesystem that is
pretty borked.  Running fsck.ext3 on it causes a segfault.  The core dump
is 2G right now.  This is likely from the short few hours we ran /mnt/koji
from a RHEL4 box (lesson learned there...)

Steps to proceed:

1) Snapshots - I've been doing all of my tests from snapshots and will
continue to do so.  The share itself doesn't seem to be catastrophically
broken, just that one section.

2) Contact some fs experts with the dump and see what is going on.

3) Fix the filesystem.

4) Also, for the last 4 nights at around 3 am my time the filesystem has
hit this part of the disk causing it to get remounted ro.  I'd like to
setup a script to auto correct this issue until we can come up with a
permanent fix.  Just saves me from having to get up in the middle of the
night.

Is anyone against this?  I'm mostly worried about step3 then anything.
Before we do that I want to make sure we have a proper backup and do
plenty of tests from snapshots.  It might be best to do 3) after the
release.

	-Mike