download2.fedora.redhat.com is behind; download3 overloaded

[Matt Domsch]

Since download1 DNS is pointing at download3 right now, rsync
connections to that box often fail.  I think it's overloaded. Recall
download3 is in PHX.

Download2 is running "behind" in its snap-mirroring.  Or at least
that's the impression one gets when rsyncing from it - after several
several hours, it still hasn't finished picking up all the rawhide
pushes from each day.   It has been this way since at least mid last
week; I don't have better statistics than this.

As rawhide is now pushed from netapp in PHX, I suspect the overload of
download3 is also causing bandwidth to be unavailable for
snap-mirroring from the netapp in PHX to download2 in TPA.

As it stands, I am extremely worried about the Fedora 9 launch
scheduled for 29 April, data to flow to the mirrors starting ~24 April
if all goes well.  That's about 2.5 weeks from now.  If the
unavailability of download1 in RDU from Internet2 continues, we won't
be able to get the bits to the mirrors.  While we have intentionally
hidden download.fedora.redhat.com from the published mirrorlists
(since the Fedora 8 launch), if our mirrors don't get the bits, users
_will_ come searching for them, and _will_ find them on
download.fedora.redhat.com.  I would hate to have a repeat of the
Fedora 6 launch fiasco, where we knocked redhat.com offline for a
week.

What can we do, immediately, to get download1 again reachable over I2,
even if it's only via static routes to fedora-archives.ibiblio.org?  9
of our 10 "Tier 1" mirrors reach us over I2.  We can direct them all
to fedora-archives.ibiblio.org, if we can get the content over to them
in a timely manner.  (fwiw, 3 of the 4 mirrors.kernel.org servers
aren't on I2, but all our other Tier 1 mirrors are.)

I suspect this will resolve the overload of download3, and the network
bandwidth challenge (which I presume exists) which will resolve the
snap-mirror to download2.

Thanks,
Matt
Fedora Mirror Wrangler, and one who doesn't want to see the fallout of
knocking redhat.com offline for a week again...


-- 
Matt Domsch
Linux Technology Strategist, Dell Office of the CTO
linux.dell.com & www.dell.com/linux