Better repodata performance
David Farning
dfarning at sbcglobal.net
Sat Jan 29 23:54:48 UTC 2005
seth vidal wrote:
> And how do you feed the repo creation system this data? Where do you get
> it to begin with? The only way you know this information is if you
> already have it - the only way you have it is if you checked all the
> packages for what has changed. Are you beginning to see the loop here?
>
Hey, great work on yum!!
Let me see if I a understand the problem and can provide a general solution.
When running yum, a check is preformed to insure that the most recent
versions of primary.xml is present, based on the timestamp found in
repomd.xml. If a more recent version of primary.xml is found then
download it.
Now, in most situations very few packages have been changed. Therefore,
between primary.xml versions few <package>...</packages> have changed
thus most of the information in primary.xml is redundant against
previous versions of primary.xml.
First, add a repo_build_number to repomd.xml. This should should be an
integer that increments every time createrepo is run. This allows us to
always know what build of a repo we are working with.
Createrepo:
saves the last runs primary.xml as primary.repo_build_number.xml.
Generates new primary.xml as usual.
Compares primary.xml to old primary.repo_build_number.xml to create
primary.increment-(age).xml. Where age indicates the packages changes
required to bring a primary of age or less builds old up to date.
A number of primary.increment-(age).xml files could be made available.
Ages of 1, 5, 10 could be made available. These files could have a
'rolling' age to cover a majority of updates without wasting too much
space.
On the Yum side of things:
Read local repomd.xml to determine local repo_build_number
dl new repomd.xml to determine current repo_build_number.
if remote repo_build_number = local repo_build_number = 0
local primary.xml is uptodate
if remote repo_build_number = local repo_build_number = 1
dl primary.increment-1.xml
merge primary.increment-1.xml with local primary.xml
if remote repo_build_number = local repo_build_number <= 5
dl primary.increment-5.xml
merge primary.increment-5.xml with local primary.xml
if remote repo_build_number = local repo_build_number <= 10
dl primary.increment-10.xml
merge primary.increment-1.xml with local primary.xml
if remote repo_build_number = local repo_build_number > 10
dl primary.xml
-dtf
More information about the fedora-devel-list
mailing list