Integrating deltarpm creation

[Jeremy Katz]

On Fri, 2007-06-22 at 16:00 -0400, Mike McLean wrote:
> Jeremy Katz wrote:
> > The first thing is that it makes the most sense for the deltas to be
> > created and stored by koji rather than as a secondary process.  This
> > adds the advantage that they're stored consistently with the packages
> > and also can be cached rather than recreated every time.  It feels
> > somewhat analagous to me to the current situation with signing.
> 
> While I see the semi-parallel with signatures, I'd rather not rush into 
> adding this to koji. I'd like to have a better understanding of how 
> these deltas need to be managed.

Fair enough...

> Do we anticipate Koji actually having a use for the deltas, or would it 
> just be storing them for other tools?

I suspect largely storage.  Since we recreate build roots every time,
the deltas aren't ever going to matter on the building packages side.
So the main advantage of doing it in koji is consistent storage and
retrieval.

> Can deltarpms be signed independently of the rpms it compares? If so, we 
> may need to think about tracking these signatures.

The deltas aren't independently signed.  The way that the deltas work is
that they take the bits off of the filesystem + the deltarpm itself to
recreate the original RPM.  You then have the original package, and you
verify it (including signature).

> How should we deal with the delta/signature interaction? Is there a 
> quick way to read the target's signature info from the delta (applydelta 
> -i doesn't seem to report it)? Each rpm in koji can have multiple 
> signatures, and we would presumably care which signature will be used 
> for the target rpm in the delta. This leads me to wonder about naming 
> schemes and api needs.

Yeah, given this we probably are going to want to be able to have
multiple deltas taking into account the signatures.  Although that just
makes me cringe a little in pain...

> With signatures, the cached files are tiny, there are unlikely to be 
> more than a handful of them per rpm, and it is clearly reasonable to 
> keep them for as long as the rpm is kept. Even deltas for trivial cases 
> seem to be much larger than a cached signature header, and one can 
> imagine accumulating a large number of deltas for an rpm. So the 
> question is, how long should deltas be kept, and what should trigger 
> their removal?

Removal should be triggered the same way as removal of the package -- I
don't think you'll want to do separate garbage collection there.  And
yes, larger than the signature.  But I don't see any way around that.
Generating them on-demand is going to be worse from the point of view of
regenerating the same bits over and over.  We've got to keep the
generated ones somewhere.  Keeping them outside of the buildsystem means
we have another data base, another file store, etc.

Jeremy