Proposal - "Slow updates" repo

[Luke Macken]

On Tue, Nov 18, 2008 at 01:53:37PM -0500, Casey Dahlin wrote:
> There's been a lot of talk about a stable release lately. Most agree  
> that its not what Fedora does best and not something that would be  
> natural for us to do. I have a sort of middle-ground proposal that might  
> be easy enough to implement that we could try it out.
>
> Currently, when a packager has an update, he runs "make update," and  
> bodhi goes and grabs the package out of koji and places it in the next  
> mash of the updates-testing repo (Luke, please shout when I start  
> getting this wrong). Then, bold subscribers to updates-testing install  
> it, check it out, and if two people report in that it didn't eat their  
> brane, it goes into the regular updates repo. That slim sanity check  
> prevents a whole helluva lot of breakage, but isn't a very broad testing  
> of the package.

Nope, the karma thresholds are completely configurable, and it defaults
to 3.  Yes, I agree that this is not a sufficient enough metric to
ensure that a package will not break anything.

> What I propose is having another repo called updates-slow. Like updates  
> testing this repo ships with Fedora but is disabled and hidden by  
> default. Interested parties would have to manually enable it, and  
> disable the regular updates repo. updates-slow would relate to updates  
> in a similar way that updates relates to updates-testing: packages would  
> go into it after people getting the general Fedora updates seem to be  
> ok. The criteria here would have to be much harder than the two karma  
> points it takes to move out of testing, I would suggest the people  
> interested in this feature form a group that decides what the entry  
> policy should be.
>
> The advantage is that this is fairly cheap to set up. Bodhi would need  
> some level of changes (Luke, what do you think?) but there's no  
> branching. We're basically just taking the ability which already exists  
> in yum to selectively take updates and providing a system to  
> collaboratively exploit this mechanism.

>From a bodhi perspective, I think it would be a minimally invasive
procedure.  It is currently written so that every release has a
Release.dist_tag + '-updates-testing' repo as well.  So, aside from a new koji
tag and mash config, and whatever interface changes the slowrepo policy
requires, it doesn't sound very difficult to implement.

However, all of this hinges on a policy that has yet to be created.  I
agree with you that we need an improved updates testing policy.  I don't
think we need a new SIG to provide it, as it sounds like a job for the
QA team.

> Not branching, however, also creates a disadvantage: with no ability to  
> create new packages, the slow repo can't take newer fixes without taking  
> an entire update. The slow repo must consume updates from Fedora in  
> whole packages; they don't have patch-level control over incoming  
> changes. Also, the slow repo has to synchronize at release time; you  
> can't have a late-F9-with-bits-of-F10 offering (hopefully though the  
> base set is the peak of mainline Fedora's QA).
>
> If the people interested in a stability feature are willing to help  
> govern and maintain this repo, and help with the bug load (we can't just  
> dump bugs against backdated versions directly on the maintainers), then  
> it could go a long way to meeting their needs.
>
> Thoughts?

I'm not quite sure that yet-another-repo will solve the actual problem at
hand, which is: we constantly break things in our stable releases.

Bodhi's karma system provides a very basic community-driven workflow for
"testing" updates.  The problem is that this "testing" is ambiguous,
unpredictable, and not documented.

Ok, so lets step back and figure out how we are actually breaking stuff.

    - Packaging issues.  Broken deps, %post, etc.
    - Bugs from new code in 'enhancement' updates.
    - Infrastructure->client-tool breakage 
      Bugs in infrastructure causing issues in the repos that the clients
      consume, causing bugs in their tools.  (Bug #471873 is a good example)
    - ...

So, how do we minimize that breakage?

    - Automated package sanity-checking.
      http://fedoraproject.org/wiki/QA/Beaker looks like it was the
      closest to solving that problem, but that initiative looks like it
      has since fizzled out.
    - Ensure that new features are tested, by humans and ideally a test suite.
    - Infrastructure bugs tend to decrease over time by nature, so we
      just need more developers to help out.
    - Having all upstream projects maintain a stable bugfix-only release
      would be great, but it's not going to happen.  Thus, we need to do
      the dirty work.

Once those issues are addressed, then it's a matter of letting our users
choose if they want 'incremental features' or 'a rock-solid slow
moving system'.  If we wanted to granularize our update repositories
even more, we could potentially make our 'stable' repository bug & security
fix only, and have a separate repo for new packages and enhancements....

luke