[fedora-virt] Improving release / update process for virt packages

[Mark McLoughlin]

On Mon, 2009-04-06 at 15:41 +0100, Daniel P. Berrange wrote:

> This mail is a braindump of some ideas we've been kicking around to
> try and improve the way we push new releases of upstream virt 
> packages to Fedora. We want to try and improve the quality of stable
> Fedora branches, and also improve testing of rawhide. 

Totally agree with the conclusions here.

> Taking libvirt as an example, currently new libvirt upstream releases
> get immediately built and pushed to both
> 
>  - Fedora rawhide
>  - Fedora stable updates-testing
> 
> Depending on the development phase rawhide is in, we get somewhere 
> between 'zero' and 'lots' of feedback for new builds in rawhide. 
> Typically before rawhide beta, we get near zero feedback, and from
> beta onwards we get alot of feedback.

I don't think "near zero" is quite accurate - there's been a fair bit of
bugzilla activity on virt bits since the Alpha. One example is that
Fedora rel-eng and QA seem to be doing lots of installs testing using
KVM and filing bugs when it's broken.

Sure, not everyone runs rawhide, but we still need to do what we can to
help make it less broken.

> We typically always get significant feedback when the new builds are
> pushed to updates-testing.

Personally, I see updates-testing as a "last chance QA" - i.e. it's only
for package updates that we really do believe are ready for stable
updates.

updates-testing is a distro-wide thing, so I expect most people who
enable it aren't specifically interested in testing virt, but they are
interested in being the last line of defence for updates before they hit
stable.

When libvirt-0.6.0 hit my laptop through updates-testing and was
significantly broken, my reaction was along the lines of "wait, I didn't
sign up for this!" - I suspect other people in that situation might
disable updates-testing as a result. That damages not only testing of
virt packages, but testing of the whole distro.

I think it was after that update that I wrote these guidelines and had
it approved by FESCo:

https://fedoraproject.org/wiki/PackageMaintainers/Package_update_guidelines

> The obvious problem with what we do for libvirt at the moment, is
> that we are introducing major new features into the stable release
> stream here. This risks destabalizing the stable Fedora streams, 
> and also compromises our 'Feature' process because stuff we're 
> pushing for Fedora 11 appears in Fedora 9 / 10 before F11 even comes
> out.

Yeah - that is another aspect. How could we pimp feature X in the F11
release notes if it is pushed to F9 updates even before F11 GA?

And another thing is pushing new features to updates doesn't mean we've
thought about whether the feature will work or not. I saw a bug report
today of PXE booting not working with latest virt-manager in F9
updates-testing because the virtio PXE ROM isn't available on F9. That
just wasn't an issue with the original F9 version.

> I think it would be desirable to get the stable Fedora releases onto a 
> pretty strong bugfix only policy, but we can only do that if we figure
> out a way to get more people testing the stuff we're pushing for rawhide.
> There simply isn't enough testing of rawhide prior to beta point. The
> bug reports / testing from the stable update-testing repo far exceeds
> the feedback we get from rawhide users.
> 
> We can keep telling people that rawhide is supposed to be functional
> enough for daily use, but frankly history shows otherwise and I don't
> see any sign of this ever changing. Things like the mass rebuild,
> rpm upgrade, xorg <-> kernel interface are inevitably changing and
> cause instability. Personally I do pretty much all of my day-to-day
> libvirt work on a stable Fedora 9 / 10 distro, only ever using rawhide
> if there is a specific thing I can't get in F9/10 packages.

I agree that it's not wise to use rawhide as your everyday machine,
especially early in the release cycle, but i do believe a significant
majority of the people seriously involved in Fedora have up-to-date
rawhide running on another machine and use this for their Fedora
development.

> There are many people who wish to test new virtualization pieces. They
> do not wish to have to debug changed RPM, or changed Xorg or changed
> kernel packages, and the rest of rawhide at the same time. While it is
> true that problems with rawhide can be resolved when they occur, it is
> still a significant waste of time. If I'm interested in testing virt,
> I only expect to have to debug virt packages,  not the entire OS.
> 
> What we need is a way to expose new virtualization features to people
> running the current latest stable Fedora release stream. So if they 
> are interested in testing the new features, they can update to those
> packages without having to include & debug the rest of rawhide.
> 
> Debian distros have this sort of idea in the 'backports' stream which
> gives optional early access to new app releases, separate from the
> normal updates and unstable streams. This is distro wide though.
> 
> We currently have people doing various adhoc personal repos from scratch
> builds, so my suggestion is that formalize this under the umbrella of the
> Fedora Virtualization SIG. Specifically, to provide a 'virt-preview' YUM
> repo for the most recent stable stream (ie F10, but not F9).
> 
> In the designated virtualization package set, anytime we do a build of a
> new release for rawhide, we would also do a build for the 'preview' stream.

I think this would be hugely useful to people interested in the latest
virt bits, but without a testing machine for running rawhide.

How about "virt-hide" ? :)

> We would explicitly not do these builds for the 'updates' repos, at least
> not without a very significant period of testing & positive feedback.
> 
> To allow us to do builds in both the preview stream, and updates stream,
> which are not scratch builds, this would unfortunately require another 
> CVS branch. We'd also need an NEVR that satisfies
> 
>  - Newer than current stable stream NEVR
>  - Older than the new build in rawhide
>  - Allows for newer build to be later made in 'updates' repo
> 
> One simple option is to have
> 
>  - CVS branch of F-10-VIRT_PREVIEW
>  - Release field for all builds in F-10VIRT-PREVIEW must have leading 0
> 
> The leading 0, allows us to later do a build in the main F-10 stable
> branch with newer NEVR, without risking that it then become newer
> than the rawhide build.
> 
> A convenient Makefile rule in the 'devel' branch could automate the
> initial work of copying the contents of devel/ to F-10-VIRT-PREVIEW
> and setting the N-E-V-R to suitable value. A few manual changes 
> might be needed, but hopefully not much before the maintainer could
> do a build in koji.

Another way would be to create a new CVS branch of 'devel' for each
build - that way you wouldn't have to sync 'devel' into the preview
branch each time. It should be possible to script tweaking the release
field and anything depending on the distro version could be
conditionalised in the spec file in 'devel'.

> There would then need to be a way to build a YUM repo from all builds
> in the F-10-VIRT-PREVIEW branch. It should automatically take all the
> latest builds from the branch - we want it to be a mini rawhide of
> just virt packages, not use the updates infrastructure.
> 
> If we could get Fedora infrastructure support for the branching that
> would be great, but we could easily do this with a private branch like
> (private-F-10-VIRT-PREVIEW) and a cron job to build the YUM repo and
> publish in a well-known location, eg http://virtmaint.fedorapeople.org/

Yep, I think we should do it that way first - if it works out well, we
can lobby rel-eng to generalize the concept for other areas of the
distro too.

> So in summary
> 
>  - All new upstream releases built in rawhide
>  - New upstream releases also built in stable preview branch if possible
>  - Only bugfixes built in stable updates/updates-testing branch
>  - In exceptional circumstances, rebase for preview branch can be
>    built to updates/updates-testing after alot of positive testing
> 
> This would
> 
>  - Ensure users of stable Fedora release have high confidence in 
>    quality of the updates/updates-testing stream
>  - Allow users to trivially test new upstream rebases while 
>    staying on the stable distro stream
>  - Improve testing coverage of major new rawhide features without using
>    the stable release stream users as guinea pigs

Excellent!

Cheers,
Mark.