up2date, mirror repositories, and performance

[Chris Adams]

Once upon a time, Gene C. <czar at czarc.net> said:
> The only other thing I can think of is some kind of central augmentation of 
> the mirror list which would reflect the sync status as well as somewhat rank 
> the servers in terms of capabilities (a big server attached to an OC48 can 
> take a lot bigger load than a small server on a T1).  This sounds good but I 
> am not sure it is do-able.

To start with, something like the CPAN system that checks "freshness"
would be good (although I'm not sure what's up with CPAN - I sent an
update of my record and nothing has happened).  I don't know if anything
uses that data, but it could be used.

However, I am not sure about multi-level tiered mirroring.  When I
started mirroring CPAN, I mirrored from another mirror.  They had
occasional problems, so I switched.  That mirror had problems and then
went away, so I just switched to the master.  Since then, the only time
I've touched my CPAN mirror is when I got a heads up one afternoon when
(IIRC) 5.8.0 was released (normally I sync in the morning so I did a
manual mid-day sync).

To have a tiered mirror system, there'd have to be some front-end tool
for the mirror admins to catch "upstream" problems and move to another
mirror (but you've got to handle different "freshness" - there are
occasional sync problems between some of the Red Hat master servers;
adding multiple tiers will not help).  Mirrors would need to be
registered with each other (to get sync in advance of release and for
notification of problems).

Mirror update schedules would also need to be synchronized; if I have my
update cron job as "0-23/6" in Central time and I mirror off someone
that has "0-23/6" but in Pacific time, I'll always be 4-12 hours behind.
How long should I schedule "behind" the next tier to get updates (15
minutes, 1 hour)?

Maybe something that pushes more of the bandwidth load to the mirrors
(like "us.download.fedora.redhat.com" be a round-robin DNS, with the
primary RH servers throttled back some) would help mirrors stay up to
date better?

There also does need to be some automated (or at least semi-automated
with a one-click RH review) way for mirror admins to maintain and update
listings.

Ranking by bandwidth might be tricky; if I ever get my patch for adding
route realm support to quagga done, my bandwidth cap will be variable
based on my available outbound bandwidth (probably anywhere from 30 to
75 Mbps or until the server falls over).

-- 
Chris Adams <cmadams at hiwaay.net>
Systems and Network Administrator - HiWAAY Internet Services
I don't speak for anybody but myself - that's enough trouble.