Yum, Proxy Cache Safety, Storage Backend

[Les Mikesell]

Nicolas Mailhot wrote:
> 
>> I'm missing something here.  A cache is only going to resend what it got 
>> the first time.  If the first user got the right thing, so will the 
>> second.
> 
> No it won't. Common failures are:
> 
> - T0: user A updates foo and in the course of doing it populates the
> proxy cache with yum T0 metadata and foo rpm only.
> - T1: upstream updates repo changing bar rpm version. Metadata is
> regenerated with the same filenames as before. For the proxy its T0 copy
> is still valid till the expiration delay  it got in http headers the
> first time.
> - T2: user B wants to update bar. The proxy happily serves yum the
> cached T0 metadata (since yum didn't request any special treatment for
> it). It tells yum the T0 bar is available. Unfortunately it's not - user
> A didn't download it so it's not in the cache, and upstream moved to a
> new version

Can't the T1->T2 transition happen while user A is in mid-update?  I'm 
fairly sure I've seen something like that without any proxies involved.

> - T0: user A updates foo and in the course of doing it populates the
> proxy cache with yum T0 metadata and foo rpm 
> - T1: upstream signs foo rpm and regenerates the repo. Metadata is
> regenerated with the same filenames as before. For the proxy its T0 copy
> is still valid till the expiration delay  it got in http headers the
> first time.
> - T2: proxy cache is contended, it evicts foo rpm since it takes a lot
> of room. T0 metadata is small so it's retained for now
> - T3: user B wants to update foo. The proxy happily serves yum the
> cached T0 metadata (since yum didn't request any special treatment for
> it). It tells yum foo is available with T0 checksum. Unfortunately it's
> not - T0 foo rpm was evicted from the cache, when user B requests the
> same URL as user A it gets T1 signed foo with a new checksum.
> 
> (you can invert this case with a proxy which strategy is to keep big
> files which are expensive to download, and drop small metadata)
> 
> In all those cases the proxy worked as designed and was fooled by
> yum/createrepo using the same URLs for different objects.

Again, I think you'll still have the same issues if user A's update 
spans any of these arbitrary Tx designations - at least the ones where 
you've got metadata and the rpms are changed before you get them.

> A proxy will always serve you a set of files as they existed on the
> original location at some time. But there is zero insurance they were
> taken from this location at the same time, so you *must* take care new
> content is created with new filenames, or your web server tells the
> proxy when old data will be replaced by new one (via expires), or your
> system is able to cope with a mix of files from different times.

But you are doing things that aren't guaranteed to work in the first 
place.  All the cache does is expand the window for breakage a bit.

-- 
   Les Mikesell
    lesmikesell at gmal.com