Better repodata performance
Dag Wieers
dag at wieers.com
Thu Feb 10 16:47:26 UTC 2005
On Mon, 7 Feb 2005, Dag Wieers wrote:
> On Sun, 29 Jan 2005, Alexandre Oliva wrote:
> > On Jan 29, 2005, "Charles R. Anderson" <cra at WPI.EDU> wrote:
> >
> > > Why all the complication when a general-purpose algorithm, rsync,
> > > already exists to solve this problem?
> >
> > It doesn't do very well on compressed files, and there aren't a lot of
> > rsync-enabled web proxies out there.
>
> You may like this RFE:
>
> https://devel.linux.duke.edu/bugzilla/show_bug.cgi?id=391
>
> gzip has an --rsyncable option that is missing from the python zlib
> implementation. If python/zlib could be taught this, and createrepo
> is able to use it, the metadata would be rsyncable (improves
> transferspeed for repo maintainers/mirrors).
My repo-generation script now unzips the createrepo metadata, recompresses
with --rsyncable and recreates the repomd.xml after createrepo's run.
I synced the repository.
I then added a single update package to my repository
And then rsynced again (with a 12kB upstream limitation), and got:
fedora/3/en/i386/dag/repodata/
fedora/3/en/i386/dag/repodata/filelists.xml.gz
961353 100% 306.10kB/s 0:00:03 (68, 16.8% of 208800)
fedora/3/en/i386/dag/repodata/other.xml.gz
412335 100% 89.30kB/s 0:00:04 (69, 16.8% of 208800)
fedora/3/en/i386/dag/repodata/primary.xml.gz
668753 100% 154.79kB/s 0:00:04 (70, 16.8% of 208800)
fedora/3/en/i386/dag/repodata/repomd.xml
1128 100% 4.87kB/s 0:00:00 (71, 16.8% of 208800)
and
fedora/3/en/x86_64/dag/repodata/
fedora/3/en/x86_64/dag/repodata/filelists.xml.gz
1466742 100% 273.93kB/s 0:00:05 (86, 20.9% of 208800)
fedora/3/en/x86_64/dag/repodata/other.xml.gz
674072 100% 79.47kB/s 0:00:08 (87, 20.9% of 208800)
fedora/3/en/x86_64/dag/repodata/primary.xml.gz
1086239 100% 141.68kB/s 0:00:07 (88, 20.9% of 208800)
fedora/3/en/x86_64/dag/repodata/repomd.xml
1128 100% 1.46kB/s 0:00:00 (89, 20.9% of 208800)
Which is a between 6.6x and 25.6x speed improvement for something I have
to update every single sync. (Normally these files are synced with a
12kB/sec rate limitation taking on average 1min per file, now only 6secs)
Now imagine what this would do if Yum had a client-side rsync
implementation. zsync may be something to look at.
-- dag wieers, dag at wieers.com, http://dag.wieers.com/ --
[all I want is a warm bed and a kind word and unlimited power]
More information about the fedora-devel-list
mailing list