Request for Comments: Proposed changes to Fedora development cycle

[Thorsten Leemhuis]

On 16.10.2007 19:38, Jesse Keating wrote:
> On Tue, 16 Oct 2007 19:13:55 +0200
> Thorsten Leemhuis <fedora at leemhuis.info> wrote:
>> I think having a "it is installable" as defined goal way more often
>> would be good. Every Friday maybe?
>> Further: both Opensuse and Ubuntu have way more betas/rcs. I don't
>> think we should follow that model completely, but maybe we can steal
>> ideas from it? A installable rawhide snapshot every two weeks maybe
>> (as ISO or as second dir (files hardlinked to rawhide) on the
>> servers?)
>> Just thinking aloud.
> While it wasn't stated in the proposal, and I should add it, we
> definitely want to do more "snapshot" like releases for testing, like
> what Jeremy has been doing with rawhide live snapshots. [...]

k, thx

>>> [...]
>>> === Release Candidate and Final release ===
>>> In order to prepare the release candidate(s) and the final release a
>>> final freeze is needed.  It is at this point that we would block
>>> rawhide once again.  It is extremely important that we have as many
>>> eyes on the bits as possible to make our release as best as we
>>> can.  This freeze point is also the point in which we would create
>>> any remaining needed branches in CVS for the release.  Any builds
>>> from the branch would be held in an updates candidate tag to be
>>> potential updates for after the release, or to be pulled in to the
>>> release by request.  Builds from devel/ would be tagged for the
>>> next release tag to show up in rawhide once the next development
>>> cycle starts.   
>> Sounds to complicated and much of a "to much of a slowdown/next to
>> full stop" to me.
> Well it pretty much is.

That's IMHO a problem -- the world rotates. Sure, it's not that bad to
let it rotating for a week, stop Fedora then try to catch up. But I'd
say it becomes really problematic when it's more then a week.

>  We have to stop changes in order to finalize
> the release and get it out the door.

A week IMHO should suffice. If it doesn't I'd say we need more point in
the middle to bring rawhide at least in a better shape beforehand.

> [...]
>> Normal, non-critical
>> updates for the release (once it happens) stay updates candidates and
>> thus don't get much tested either.
> They can be pushed to updates-testing at the 0day mark and thus get
> testing there.

Depends on how much lands into updates-testing over those weeks (and how
many weeks we are taking of; three to four afaics). IOW: it IMHO looks
bad to do a release and have more than <n> updates in updates-proper and
<n> in updates testing (<n> something like 10 or 20 each in both cases).

>> Some packagers will thus likely just push their updates into the
>> current release without testing them in rawhide first. 
> But rawhide isn't the place to test them,

Sure it is -- at least for me. Here is what I do when I get a new
upstream release in Fedora:

- test locally (but that's of course always limited, as apps get used
differently by different people)
- update cvs for devel and build for rawhide, as rawhide users are those
that won't yell to much if upstream released something new with multiple
bugs in it
- leave it there -- depending how crucial the package is for a few days
a week some weeks
- if nothing bad turned up in rawhide (e.g. if the new upstream release
didn't contain any new bugs) I ship it in F7 updates-testing for some days
- if nothing bad turns up move it to F7 updates proper
- if that package would be of interest for F6 I would put it in
updates-testing for F6 at this point (if one would exist)
- if nothing bad turned some days later up push to F6 updates proper

Shipping a update in multiple branches at the same time or in F7 earlier
then in rawhide (because it is blocked for release) seems like a bad
idea to me (I know, that's the standard behavior for a lot of people) --
if upstream got a bad bug in it all our users are effected by it and
that can easily be prevented by above scheme.

>> That's bad in
>> general already; but it's even more bad here, as it also create
>> EVR-update troubles once the release is done.
> How can it create EVR-update troubles?  Only if maintainers push
> updates to previous releases before pushing them for the pending
> release, and even so this is not a new problem, as updates to previous
> releases happen all the time while the release is a static package set
> that quickly has "lower" nvrs than the fully updated previous release.
> Not a new problem, [...]

No, but with the new scheme it becomes bigger afaics.

>>> In order to accommodate a
>>> release candidate set plus the final release creation the freeze
>>> point for this should happen between the current Test3 dates and
>>> the final deep freeze dates.  
>> This proposal would IMHO be a bit easier to understand if you could
>> write a potential F9 schedule with release, freeze and branching
>> dates.
> I can add some example dates, but they shouldn't be counted on until
> the Board and FESCo et al set the schedule.

thx for doing that!

> [...]
>> Yes, sure, I know out mirrors are important, but not that important to
>> delay the release that 5 days when the test window is four weeks (the
>> usual cycle we had between test releases up to now).
> That puts an extreme pressure on the mirrors that I'd not like to do.
> We've frequently seen feedback from them that if we just do torrent
> first and delay the mirrors that they'd just as likely stop mirroring
> us which doesn't help anybody.  The only /real/ way to get releases out
> there quicker is to not have unstable trees all the time.

I strongly disagree with the "only" in that sentence. OpenSuse and
Ubuntu afaics seem to get their test-release out way more quickly. We
should be as well for *test*releases. For the final, yeah, the old
behavior might fine. I'd say most mirror admins will understand that we
need to get test releases out more quickly.

>> Further: I know multiple people that don't run rawhide because we
>> always tell everyone "once rawhide always rawhide"
> Where exactly do "we" tell everyone this?

It's in peoples heads afaics and they mention it on the list if
questions come up "can I run a test release and later the final without
reinstalling?" -> "No".

Of did we document somewhere in between how to switch from rawhide to
release before rawhide moves on? If yes please point me to, then I'll
blog about it to get the old scheme out of peoples head.

>> and "rawhide easts
>> babies". That' IMHO are our biggest problems. Thus i suggest:
>> - work hard to make rawhide not eat babies and kill kittens in the 5
>> weeks approaching a release. Tell the developers so they are more
>> careful during that timeframe (e.g. as careful as they would be for a
>> update for a released version) and send ninjas if they are not.
> Why 5 weeks, why not just all the time?

Well, that's what we tried. Seemed it didn't work to well -- see for
example OpenEXR.

>> - we need a way for testers that want to test the release that they
>> want to continue to run afterwards without reinstalling it. Two ideas
>> follow, I think we should do both:
>> (1.) support and document the "rawhide -> stable" way -- e.g. at the
>> point where rawhide on the servers becomes targeting the next Fedora
>> version give users the choice "Do you want to continue using rawhide
>> or stick to the release that is happening now?".
> Not sure how to "give them the choice"

Yum-Plugin? Or something that disables all repos and asks which ones to
activate again: rawhide or release.

> but we certainly can document
> how to switch to the final release. 

thx -- in the past it seemed that was discouraged.

> A good "Using Rawhide" page would
> list this, and would fall inline with the other changes we want to make
> to lead to rawhide being easier and more friendly to use (and thus more
> used).

Sounds good. But it's nevertheless easy to miss the point, and then you
have a problem.

>> (2.) create a "Fedora <N> Early Adapters Release (EAR)"  which uses
>> the final directory structure and a Release for "early adapters".
>> E.g. what I mean is something like this:
>>
>> T-30 days before official release
>>  * publish a test3 (or call it RC if you like)
>>  * hard free for rawhide -- just like the two/three lasts week before
>> the release now
>>
>> T-18 days before official release
>>  * branch rawhide and release;
>>  * create the final directory structure on the servers with what's in
>> rawhide
>>  * create and publish "Fedora <N> Early Adapters Release (EAR)" from
>> that tree; its yum  configs point to that tree and not to rawhide
>>  * open rawhide for release +1 -- rawhide users had enough time to
>> report bugs; they won't find any new bugs anymore, thus we need a
>> different kind of testers now which we get with that Fedora <N> EAR
>>
>> T-18 until T-7 days
>> * fix important stuff only like we in the past often did in the first
>> ten days after a Fedora release; ship that as updates
>>
>> T-7 days
>> * respin the tree  on the servers, including all updates that happened
>> over the past 10 days
>>
>> T-4 days
>> * prepare the final release
>>
>> T-0
>> * release
>>
>> = EOF =
>>
>> Just my 2 cent.
> 
> I had thought about this too.  It is challenging from an infrastructure
> standpoint to manage the bits in this way, due to the permissions
> typically on the release directories.  However it's not impossible and
> could be a solution to the "second rawhide" type thing.  I'm certainly
> open to discussing this more, but I'm not comfortable adding it to the
> proposal just yet.

thx for considering something like this.

CU
knurd