The updates firehose - automatic local mirrors

[Matt Domsch]

On Mon, Jun 11, 2007 at 09:21:00AM -0400, Jakub Jelinek wrote:
> On Mon, Jun 11, 2007 at 08:08:30AM -0500, Matt Domsch wrote:
> > > This would require rather simple updates to the 
> > > http://mirrors.fedoraproject.org/mirrorlist script.
> > 
> > I really don't want to trust that the only mirror in a given country
> > is active and fully up-to-date, and send everyone from that country to
> > that single mirror.  The thresshold is 3 per country to provide some
> > reasonable guarantee that at least one of those 3 is active and
> > up-to-date at any given instance.
> 
> Isn't that reasonable guarantee already the report_mirror data or
> crawler acquired data?
> Looking at current http://mirrors.fedoraproject.org/publiclist/Fedora/7/
> there are only very few countries for which non-global list is used
> (unlike FC6 times):
> Canada, Germany, Spain, France, Greece, Japan, Netherlands, Romania, USA
> E.g. Czech Rep. used to be listed, but supposedly two of the mirrors
> were kicked out for not being 100% up2date by the crawler and now we
> have just 2 mirrors, both stable though.

It's a lot better, yes, but it's not guaranteed.  The crawler takes
nearly 12 hours to get through all the mirror hosts.

Tell you what.  I need three changes, and we can make this a lot
better.

1) python-GeoIP needs to export the countrycode->continent mapping
   table, so we can list other mirrors on the same continent at least
   if the country has <3 hosts.  The code is in GeoIP, it's just not
   exported via python-GeoIP, and my Python C-binding foo is weak.

   I'd like the countrycode->countryname (e.g. FI -> 'Finland')
   mapping exported as well, so I can display the country names rather
   than just the country codes.

   https://bugzilla.redhat.com/bugzilla/show_bug.cgi?id=243696
   filed to track this.

2) mirrormanager mirrorlist algorithm adds an extra step.  If there
   are 3 or more mirrors in a country, return all those.  If not, add
   in mirrors on the same continent.  If still <3, add in the global
   list.

   In addition, mirrormanager then returns the mirrorlist in priority
   order, so the country-specific mirrors appear first, then the
   same-continent, then global if necessary.  For each geographic
   circle, randomize that sub-list so we get the same behavior as yum
   failovermethod=roundrobin, but we don't rely on yum for help.

3) Jesse releases a new fedora-release RPM that adds 
   failovermethod=priority
   to each of the [repository] sections.

   https://bugzilla.redhat.com/bugzilla/show_bug.cgi?id=243698
   filed to track this.


So, if someone can help with 1) above, the other two should be easy.

Thanks,
Matt

-- 
Matt Domsch
Software Architect
Dell Linux Solutions linux.dell.com & www.dell.com/linux
Linux on Dell mailing lists @ http://lists.us.dell.com