Request for Comments: Proposed changes to Fedora development cycle

[Jesse Keating]

On Tue, 16 Oct 2007 19:13:55 +0200
Thorsten Leemhuis <fedora at leemhuis.info> wrote:

> I think having a "it is installable" as defined goal way more often
> would be good. Every Friday maybe?
> 
> Further: both Opensuse and Ubuntu have way more betas/rcs. I don't
> think we should follow that model completely, but maybe we can steal
> ideas from it? A installable rawhide snapshot every two weeks maybe
> (as ISO or as second dir (files hardlinked to rawhide) on the
> servers?)
> 
> Just thinking aloud.

While it wasn't stated in the proposal, and I should add it, we
definitely want to do more "snapshot" like releases for testing, like
what Jeremy has been doing with rawhide live snapshots.  They won't be
as heavily tested as even the "Alpha" release, but will still be made
sure to be mostly bootable and installable.  However the timing of
these would depend on changes in the tree and should target significant
changes and not just be a ticking clock, where as Alpha/Beta/RC/Final
are time based, the other snapshots can be feature or change based.

> 
> > [...]
> > === Release Candidate and Final release ===
> > In order to prepare the release candidate(s) and the final release a
> > final freeze is needed.  It is at this point that we would block
> > rawhide once again.  It is extremely important that we have as many
> > eyes on the bits as possible to make our release as best as we
> > can.  This freeze point is also the point in which we would create
> > any remaining needed branches in CVS for the release.  Any builds
> > from the branch would be held in an updates candidate tag to be
> > potential updates for after the release, or to be pulled in to the
> > release by request.  Builds from devel/ would be tagged for the
> > next release tag to show up in rawhide once the next development
> > cycle starts.   
> 
> Sounds to complicated and much of a "to much of a slowdown/next to
> full stop" to me.

Well it pretty much is.  We have to stop changes in order to finalize
the release and get it out the door.  I can't think of any way to stop
changes but let developers prepare updates for the release (and use
updates testing to get them tested!!)

> New stuff can't be tested, as rawhide stands still
> (I assume it's not pushed to the servers?). 

I'm still trying to come up with some scheme that would open a second
rawhide for the next release, or someway to have a nightly tree
somewhere of the pending release that is still seen and used by the
majority of our testers.  It's essential we get their feedback on the
tree until such time as we won't take in fixes they've found.

> Normal, non-critical
> updates for the release (once it happens) stay updates candidates and
> thus don't get much tested either.

They can be pushed to updates-testing at the 0day mark and thus get
testing there.

> 
> Some packagers will thus likely just push their updates into the
> current release without testing them in rawhide first. 

But rawhide isn't the place to test them, as we're using rawhide as the
most convenient method to get the majority of our community users to
test the bits that will be the release, not the release + potential
updates, the actual release.  After that rawhide would be the /next/
release and also not suitable as a testing ground for updates to a
release.

> That's bad in
> general already; but it's even more bad here, as it also create
> EVR-update troubles once the release is done.

How can it create EVR-update troubles?  Only if maintainers push
updates to previous releases before pushing them for the pending
release, and even so this is not a new problem, as updates to previous
releases happen all the time while the release is a static package set
that quickly has "lower" nvrs than the fully updated previous release.
Not a new problem, and no real solution other than using network at
upgrade time to reach the released updates for the release you're
upgrading to.

> 
> > In order to accommodate a
> > release candidate set plus the final release creation the freeze
> > point for this should happen between the current Test3 dates and
> > the final deep freeze dates.  
> 
> This proposal would IMHO be a bit easier to understand if you could
> write a potential F9 schedule with release, freeze and branching
> dates.

I can add some example dates, but they shouldn't be counted on until
the Board and FESCo et al set the schedule.

> > === CVS Branching ===
> > At some point in the tail end of development, either right after
> > Beta release or sometime between Beta and RC release, we allow for
> > early branching of software.  This allows developers to check in
> > new features and otherwise unstable changes that would not be
> > suitable to introduce to the current release.  At Release
> > Candidate / Final freeze time all unbranched packages will be
> > branched.  
> 
> I'd say we should split up rawhide and way to final release a bit
> earlier. See below.
> 
> > == Discussion ==
> > Questions or comments regarding this proposed change should be
> > directed to the
> > [https://www.redhat.com/mailman/listinfo/fedora-devel-list
> > fedora-devel-list] mailing list.  The ReleaseEngineering team is
> > driving this proposal.  
> 
> Thx again for this.
> 
> What I further like to see: get test-releases (and maybe also the
> releases itself) out more quickly; currently we release only on
> Tuesday, Wednesday and Thursday; the target being a Thursday. That
> afaics lead to a delay of multiple days sometimes -- e.g. release was
> targeted for Thursday but was delayed by one day due to problem; that
> meant a five day delay till next Tuesday to get that release out to
> the testers -- way to long IMHO...
> 
> IOW: if a test-release is ready wait max. 24 hours and then get that
> thing out. Maybe even release new test-versions as bittorrent only the
> first 24 hours and then open the normal mirrors?
> 
> Yes, sure, I know out mirrors are important, but not that important to
> delay the release that 5 days when the test window is four weeks (the
> usual cycle we had between test releases up to now).
> 

That puts an extreme pressure on the mirrors that I'd not like to do.
We've frequently seen feedback from them that if we just do torrent
first and delay the mirrors that they'd just as likely stop mirroring
us which doesn't help anybody.  The only /real/ way to get releases out
there quicker is to not have unstable trees all the time.  If people
were more careful about what they put into rawhide or more reactive to
broken deps reports or did more communication when they introduce wide
reaching changes we'd have less unstable trees and be able to produce
functional releases in quicker fashion and be able to get them to
mirrors on time.  We did a pretty good job with test3, and I hope it
continues with the final, and into next development cycle.

> Further: I know multiple people that don't run rawhide because we
> always tell everyone "once rawhide always rawhide"

Where exactly do "we" tell everyone this?

> and "rawhide easts
> babies". That' IMHO are our biggest problems. Thus i suggest:
> 
> - work hard to make rawhide not eat babies and kill kittens in the 5
> weeks approaching a release. Tell the developers so they are more
> careful during that timeframe (e.g. as careful as they would be for a
> update for a released version) and send ninjas if they are not.

Why 5 weeks, why not just all the time?

> - we need a way for testers that want to test the release that they
> want to continue to run afterwards without reinstalling it. Two ideas
> follow, I think we should do both:
> 
> (1.) support and document the "rawhide -> stable" way -- e.g. at the
> point where rawhide on the servers becomes targeting the next Fedora
> version give users the choice "Do you want to continue using rawhide
> or stick to the release that is happening now?".

Not sure how to "give them the choice" but we certainly can document
how to switch to the final release.  A good "Using Rawhide" page would
list this, and would fall inline with the other changes we want to make
to lead to rawhide being easier and more friendly to use (and thus more
used).

> 
> (2.) create a "Fedora <N> Early Adapters Release (EAR)"  which uses
> the final directory structure and a Release for "early adapters".
> E.g. what I mean is something like this:
> 
> T-30 days before official release
>  * publish a test3 (or call it RC if you like)
>  * hard free for rawhide -- just like the two/three lasts week before
> the release now
> 
> T-18 days before official release
>  * branch rawhide and release;
>  * create the final directory structure on the servers with what's in
> rawhide
>  * create and publish "Fedora <N> Early Adapters Release (EAR)" from
> that tree; its yum  configs point to that tree and not to rawhide
>  * open rawhide for release +1 -- rawhide users had enough time to
> report bugs; they won't find any new bugs anymore, thus we need a
> different kind of testers now which we get with that Fedora <N> EAR
> 
> T-18 until T-7 days
> * fix important stuff only like we in the past often did in the first
> ten days after a Fedora release; ship that as updates
> 
> T-7 days
> * respin the tree  on the servers, including all updates that happened
> over the past 10 days
> 
> T-4 days
> * prepare the final release
> 
> T-0
> * release
> 
> = EOF =
> 
> Just my 2 cent.

I had thought about this too.  It is challenging from an infrastructure
standpoint to manage the bits in this way, due to the permissions
typically on the release directories.  However it's not impossible and
could be a solution to the "second rawhide" type thing.  I'm certainly
open to discussing this more, but I'm not comfortable adding it to the
proposal just yet.

-- 
Jesse Keating
Fedora -- All my bits are free, are yours?
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 189 bytes
Desc: not available
URL: <http://listman.redhat.com/archives/fedora-devel-list/attachments/20071016/b7307539/attachment.sig>