Request for Comments: updating RPMs using binary deltas

[Lamar Owen]

On Thursday 08 January 2004 09:43 pm, Jef Spaleta wrote:
> Now...here's a wrinkle in your plan. Maybe you are not aware, and I
> myself had to be reminded of this fact....in the past Red Hat has had to
> respin isos for non-technical reasons. 

Sure.  There have been many respins over the years.  And I am aware of them.  
Yes, this is a wrinkle; the first respin update would need to be kept as a 
fully cooked package the that time the user had a pre-re-spin ISO.  Simple 
enough to work around.

> isos and images must take their place. Your rpmdiff idea can not CLEANLY
> take care of this sort of problem.

Sure it can.  You almost sound like you don't want it to work, unless you're 
just playing a Blue Devil's advocate.

The update package for the respin, or multiple packages, for that matter, can 
be kept as 'fully cooked' packages.  Most users will not need to download 
them.  Then the respun packages become the new baseline.  One would think 
that you would have though this through and suggested the solution, not just 
being stymied and saying 'It won't work! It won't work!', Jef.

> > When updates exceed 100MB _ALL_ users are low bandwidth, and dialup
> > users are locked completely out.  So they don't update at all, get
> > hacked, and blame Linux.

> There is an argument to be made that users shouuld expect to utilize the
> same networking methods to get updates that they used to get isos.

Windows users don't; why should Linux users not have a feature Windows users 
have grown to expect.  For all its cruft, Windows Update works fairly well; 
and you don't have to download 50MB everytime a minor patch to the kernel 
happens.  We can do better than this.  We already have better software.

> Whether that network is broadband or sneakernet or their friends
> stationwagon full of punchcards or even homing pigeons.

Oh puhlease.  It's common to get install media through a different channel; I 
have bought CheapBytes discs, official Red Hat disks, etc, and have pulled 
updates from myriads of on and offline sources.

> But maybe he didn't realize you meant to do this as part of the
> all-powerful buildserver. I'm making popcorn in preparation for watching
> more discussion from seth on this issue.

Why the animosity over a purely technical issue, Jef?  Have I rocked your 
little red wagon or something?

The buildfarm is already all powerful; without the build boxen, which by 
nature are the most heavily taxed boxen in the whole update scheme, we 
wouldn't have updates.  The build hosts can do this fairly easily.  The 
uploads then happen more quickly.  The rsync happens more quickly.  The 
download happen more quickly.  Users like it; it takes less time.

As to popcorn, well, Seth is a fine mirror operator, and has a great tool in 
yum.  But Seth doesn't seem to fully realize yet what I've said.  It could in 
the long run save a substantial amount of download time and reduce the 
latency of providing updates.

> And i want to comment on making this a replacement and not providing the
> fully baked rpms, you have to create the full update rpms to get the
> diffs, so they have to exist, and they really need to be available as
> part of a transparent process so people can verify the rpmdiff is
> reintegrating the rpm correctly. Because reintegration for some people
> WILL not work for them at some point, now you can come into #fedora irc
> chat room at any time and try to explain to them that it SHOULD work
> everytime for everybody...but users have an amazing ability to create
> their own problems..that you didn't expect to see.

You're reaching.  There would of course be nothing stopping any mirror from 
having a full set of 'fully cooked' rpms laying around.  SuSE is providing 
both at this time; maybe that is the best way.  Hey, I'm flexible; this is 
nothing worth getting too worked up about.

But I ask again: if SuSE is doing this, is it working for them and their users 
RIGHT NOW?  I don't have the answer; I just threw out the idea.  The few 
users that have posted in fedora-list have seemed enthusiastic.

> -jef"So...i wonder why noone has taken the time yet to attempt to build
> the equivalent full set of rpmdiffs for ALL the published rhl 8.0
> updates and started actually benchmarking this concept in a general use
> case for a full lifetimes worth of updates..working with both original
> boxset isos and the respun downloadable isos..and find an interested
> mirror maintainer in the fedora core mirror list already and actually
> start benchmarking how this is going to work in general use cases....we
> been talking about this for what a whole day now....but I'm happy to
> keep talking about...because i'm certaintly not interested in
> implementing it myself"spaleta

SuSE is doing this; let's find out how it's working for them and not reinvent 
the wheel, needlessly duplicate effort, and waste time.

So, looking at ftp://ftp.suse.com/pub/suse/i386/update/9.0/rpm/i586
I see several interesting things:
1.)	They aren't distributing a patch.rpm for kernel-source, which is 
disappointing (and the full rpm is 43.2MB) (the reason is the changed 
directory, I'm sure, which doesn't allow them to just drop in the changed 
files);
2.)	They do, however, distribute patch rpms for other large packages.  You can 
see the filesize differences yourself.  For instance, the updated 
kdelibs3-3.1.4-38.i586.rpm is 12.3MB.  The patch.rpm is 702KB.  That's 
substantial. The koffice rpm is 2.5MB, the patch.rpm 276KB.  The XFree86 
updates fare well; the perl update does not, with the patch weighing in at 
5.2MB vs the full RPM at 12MB.  The kopete package fares worse, with the 
patch.rpm at 2MB and the full at 3MB.

Ask a SuSE mirror operator what they think, or are we in Fedora-land afflicted 
with the dread NIHilist disease?
-- 
Lamar Owen
Director of Information Technology
Pisgah Astronomical Research Institute
1 PARI Drive
Rosman, NC  28772
(828)862-5554
www.pari.edu