MirrorManager upgrades

[Matt Domsch]

----- Forwarded message from Matt Domsch <Matt_Domsch at dell.com> -----

Date: Sat, 4 Oct 2008 16:37:12 -0500
From: Matt Domsch <Matt_Domsch at dell.com>
To: mirror-list-d at redhat.com
Subject: MirrorManager upgrades
References: <20081003150210.GJ1105163 at hiwaay.net> <20081003151427.GK1105163 at hiwaay.net> <E9909A75A543064DB66E55B8E3BE41ECA22457 at ausx3mps307.aus.amer.dell.com> <20081003173951.GA1222229 at hiwaay.net> <20081003193533.GA16972 at auslistsprd01.us.dell.com> <20081003201159.GC1222229 at hiwaay.net>
Status: RO

On Fri, Oct 03, 2008 at 03:11:59PM -0500, Chris Adams wrote:
> Once upon a time, Matt Domsch <Matt_Domsch at dell.com> said:
> > Found "yet another bug". :-)
> > 
> > Please run report_mirror one more time, and check again after the top
> > of the next hour.
> 
> Well, they all came back good about half an hour ago, but now I'm back
> to only fedora-9/{i386,x86_64,ppc} come back with the preferred netblock
> and my mirror in the list (the other repos give a valid list, just not
> including my local mirror).

Chris and I traded private mails after this, and I believe everything
is working again as expected.  It helps if I deploy the new version to
all of the machines running in the cluster...  If anyone else has
trouble with MM, please let me know.

FWIW, I've been working on an upgrade to mirrormanager, to allow it to
expose the mirrorlist as a metalink file.  Longer-term, yum is growing
the capability to use a metalink file for the mirrorlist, which brings
with it the ability to check checksums and signatures of the
repository files (whats in the repodata/ directories).  It's not
looking like that feature will make Fedora 10, but at least the MM
code server-side will be in place.

One feature of metalinks is the ability for the user's client app to
download multiple chunks of a large file from multiple servers in
parallel.  For the moment I've disabled this feature in the metalink
files being published, by setting maxconnections=1, and by not
publishing info about smaller "chunks" of the ISO files.  I know
"download accelerators" have been problematic for our mirrors, and I
don't want to exacerbate the problems that adding metalinks might
bring.

If you want to see for yourself what is being published, take a normal
mirrorlist URL, such as:

http://mirrors.fedoraproject.org/mirrorlist?repo=rawhide&arch=i386
 and use 'metalink' instead of 'mirrorlist'.
http://mirrors.fedoraproject.org/metalink?repo=rawhide&arch=i386

Likewise, URLs to ISOs of the form
http://mirrors.fedoraproject.org/mirrorlist?path=pub/fedora/linux/releases/test/10-Beta/Live/x86_64/F10-Beta-x86_64-Live.iso
 become
http://mirrors.fedoraproject.org/metalink?path=pub/fedora/linux/releases/test/10-Beta/Live/x86_64/F10-Beta-x86_64-Live.iso

This shows a long list of mirrors, plus some metadata about them:
* a preference which is just the mirrormanager randomization algorithm
* country each mirror is in
* multiple protocols, if a mirror provides >1.  In the past, if both
  http and ftp were provided, only the http URL would have appeared in
  the mirrorlist.  This allows client-side tools (not yum yet, but
  perhaps in the future - yum-ftponly plugin anyone?) to determine the
  best protocol they can use.
* SHA1SUM for verification

In addition, mirrors.fedoraproject.org now can serve its content via
https, e.g.
https://mirrors.fedoraproject.org/metalink?repo=rawhide&arch=i386 for
added security (again, assuming your tools actually do certificate
checking, which ATM yum does not).

Thanks,
Matt

-- 
Matt Domsch
Linux Technology Strategist, Dell Office of the CTO
linux.dell.com & www.dell.com/linux

----- End forwarded message -----