CVS is busted

[Mike McGrath]

On 18 Nov 2006 09:31:26 -0600, Jason L Tibbitts III <tibbs at math.uh.edu> wrote:
> >>>>> "AK" == Avi Kivity <avi at argo.co.il> writes:
>
> AK> Just curious: why the read-only mount?  shouldn't the RAID have
> AK> continued in degraded mode?
>
> Probably because something else bad happened that just completely
> screwed up the SCSI bus and corrupted data on multiple disks.
>
>  - J<
>

We're talking about multiple failures across multiple drives, possibly
a backplane.  Here's the current plan.

1) Move proxy 3-4 into the f.rh.c cluster so we can take our new dells back.

2) Grab one of the new Dells and build the new cvs box.  This will
allow us to A) trust the hardware (we're all a little wary about the
current cvs box) and B) build a new box with atleast access to the old
box if we're missing something.  It will also allow us greater
capacity with regards to future growth and the whole FC+FC=Fedora
thing.

3) Restore backups to the new cvs box.

4) test test test

5) Release to the wild and fix bugs as needed.

6) Take the old cvs box and run full diagnostics before we rebuild it
(it'll be come one of our db servers, either primary or backup)

Right now mgalgoci is working working on steps 1 and 2.  When they are
done I'll be on step 3 and we'll need a few people for 4.  We'll
probably discuss in #fedora-extras when the time comes.

Bottom line, this sucks but we're working on it.  Should be up and
better than ever by Monday.

               -Mike