Announcing Fedora Activity Day - Fedora Development Cycle 2009

[Jesse Keating]

On Tue, 2009-06-02 at 21:09 +0200, Thorsten Leemhuis wrote:
> Just a general comment, not relevant for the "Announcing Fedora Activity
> Day". I'm glad that you gave up that quest, as "Fedora gets lot's of
> updates" is afaics a reason why a lot people actually use or contribute
> to Fedora. But that doesn't matter much. What I really want to say:
> 
> I totally agree with this:
> 
> > It's just doesn't
> > seem to be in line with the desires of the package maintainers, whether
> > or not it's in line with the desires of (some of) the project leaders.
> 
> It IMHO shows a big and more and more pressing problem in Fedora:
> Packagers and leadership are not working towards the same direction.

Yes, that is a problem.  In other projects, the contributions from folks
who don't agree with where the project leaders are trying to take the
project are often rejected or ignored.  That isn't happening here, which
is a good thing I suppose, but it is a point of contention that will
have to be addressed.

> 
> Anyway, back to topic:
> 
> >> - other distributions seems to manage a whole lot more test releases
> >> (e.g. alphas, beta, RC, milestones, ...) per devel cycle; is that
> >> something we should aim for as well?
> > If there was a way to do it without adding stress and work needed to
> > teams like releng, installer, etc.. [...]
> 
> One comment: *from a outside* it often looks a bit like
> - some (not all) people go crazy for weeks or months and ignore some of
> those bugs that are not pressing, but nevertheless pressing (e.g. kind
> of bugs that tend to land on target or blocker tracker bugs or already
> are there)
> - then you send a reminder "there will be a a test release next week"
> - people suddenly wake up and try to fix those bugs for the test release
> - they notice: arghh, serious things are broken, we need more time; can
> we please slip?
> - we slip
> 
> Maybe more target dates where people should "get things into shape"
> might help to reduce the work for the real test/final releases.

Yeah, I can see how that would be the case from the outside.  From the
inside, the more freeze points we have, the more times  work in progress
has to be stifled and beat into some sort of working shape, even if it's
nothing like what the final product goal is, which results in a lot of
wasted effort and testing which gets thrown out as development continues
and changes everything that was "tested" earlier.

> >> - how about doing something like a "cp -l development devel-snapshot"
> >> now and then (once a week) when we know rawhide is mostly working?
> > The rawhide trees for at least the past month are kept in the Fedora
> > infrastructure and are even available by a public url.  We haven't
> > tagged any of them as "stable" though.
> 
> They are afaics way to far away/way to slow to reach to do a proper
> network install in an acceptable period of time (at least from Germany).
> Is rsync available -- I'm not aware of it, so I doubt it?

No, but the most important part of these are the boot.iso which has the
anaconda and other code that anaconda needs.  The repository that you
point the boot.iso at to install packages matters a lot less at least so
far as "completing the install".

> >> - how can we reduce our time between finishing a (test) release and
> >> releasing it dramatically? It seems other distributions get new (test)
> >> release out to the users a lot quicker then the three to five days days
> >> we require, which seems a whole lot for a devel cycle that takes 180
> >> days in total (and we all know how much rawhide can move on with a few days)
> > If we didn't care to let the mirrors sync up we could "release" it
> > earlier. However what good does that do when nobody can /get/ to the
> > release because none of the mirrors have it, and the ones that do can't
> > help sync to those that don't because users are tying up all the
> > bandwidth?
> 
> I didn't say to not care. But maybe shorten the time we wait for them
> and help to get the bits out more quickly; pushing users to use
> bittorrent more might help a lot as well.

I honestly feel that trying to shorten the length will only lead to
mistakes.  We're short as it is, nearly too short, with extremely little
time to recover from a mistake without slipping.

> 
> >> - I'd be glad if we could stick to our release targets a lot better.
> >> Delaying releases looks quite unprofessional. Delaying also creates
> >> trouble for those depending on our releases. Take computer magazines
> >> (which have hard deadlines for productions) that might want to ship with
> >> a new release on a CD or DVD together with the next issue -- due to our
> >> fame in missing deadlines it seems to me that we are a lot more
> >> unattractive than Ubuntu (which afaics is on the shelf's here in Germany
> >> with new computer magazines just a few days after it has been released)
> > What looks more unprofessional?  Delaying the release, or hitting our
> > date and releasing with bugs that eat people's data?  Or releasing with
> > broken graphics for large swaths of users?
> 
> I think you drifted a bit way to far away and into a opposition without
> need. I for example nowhere said that we should not slip if there is a
> strong reason to. But my comment was quite open, so it's partly my
> fault; so let me rephrase:
> 
> What is rel-eng doing in the next devel cycle to make sure we slip less
> then we used to in the past? Or does rel-eng think everything was fine
> and missing three and a half target dates (alpha, beta,
> release and the first slipped release target date) out of four is
> acceptable?

I don't think it's acceptable, and what we're doing is taking a look at
the problems with our development cycle, so that we can provide more
time for development and bugfixing without getting in the way of
maintainers, we're holding this FAD to investigate and brainstorm.

> >> - why do we have to slip by a whole week most of the time? can't we find
> >> ways to slip just a day or two if there really is no way around a delay?
> > The marketing machine has very strongly requested that we only do
> > releases on Tuesdays.
> 
> Some reasons why Tuesdays are "must" instead of "good" would be way more
> helpful then a simple statement "they said so". So all I can say is:
> Yes, let's target Tuesdays if that is idea (which I agree), but if there
> is a slip then slip only a day, two or three if the problem can be fixed
> within that timeframe (which for example was not the case for the first
> release slip for the final, but maybe for the second).

It wouldn't have been.  Nearly every time we tried to slip only for a
couple days, we've had to slip for longer.  Slipping usually means that
we didn't have a GOLD set at the point of no return, which means we have
to generate a fix for whatever is broken, re-create the release
candidate, and re-validate it through our test matrix.  It simply takes
more time to do than what we could fit into a 2 or 3 day slip.

> > [...]
> >> - I'd be glad if the final release directories (e.g.
> >> release/12/Everything" could be available earlier, even if what is in
> >> them is not yet what "12" actually will become
> > You'll have to enumerate why that is.
> 
> Kevin outlined some of the reasons already in a reply:
> https://www.redhat.com/archives/fedora-devel-list/2009-June/msg00073.html

I don't want to make assumptions.  If his reasons are yours, that's fine
we'll take those issues as the problems to solve.  What I'm not looking
for is solutions looking for a problem, or pre-determined solutions that
we'll try to make every problem fit into.  I want to look at the problem
set as a whole and /then/ work toward a solution.  The proposal on the
page is mostly a thought exercise to get people thinking about the scope
of things that we could change.  If we come out of this FAD with exactly
my proposal, then we'll have likely wasted a lot of time and money.

> 
> >  One reason I've avoided this is
> > added confusion as to when the "release" happens.  If we created that
> > directory and put content in there, would we have then released Fedora
> > 12?  When does it become "released" and thus trusted?
> 
> As Kevin said as well: It seems to work quite well for Ubuntu and
> actually avoids a lot of confusion. I'd say their scheme even works
> better then our scheme. I'd also think most "normal" users don't ever
> look on the servers.
> 
> BTW, one more general thing for the FAD: Would it make sense to make
> rawhide updates more than once a day in case something that bugs lot's
> of people can be fixed easily by a quick update?

Unfortunately with the addition of delta rpms, rawhide composes are back
up to taking 8+ hours.  There are places in the delta code paths were we
can optimize, but the fact that we're dealing with 8500 packages to
generate a lot more rpms across 4 arches, determining multilib on these,
and then generating / validating deltas for all of these means that
we're working on a scale that is very large, with a comparatively very
small budget for hardware to deal with it, which means that our composes
are not going to be quick.  Particularly when we're trying to also push
tonnes of updates for 3 other releases, which puts more stress on the
same pieces of hardware.  The scale is enormous, and only getting
bigger, which means slower.

-- 
Jesse Keating
Fedora -- Freedom² is a feature!
identi.ca: http://identi.ca/jkeating
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 197 bytes
Desc: This is a digitally signed message part
URL: <http://listman.redhat.com/archives/fedora-devel-list/attachments/20090602/fd861157/attachment.sig>