Enhancing Our Fedora 13 Release Criteria

John Poelstra poelstra at redhat.com
Sun Nov 22 22:00:26 UTC 2009


Adam Williamson said the following on 11/20/2009 08:37 PM Pacific Time:
 > On Fri, 2009-11-20 at 15:43 -0800, John Poelstra wrote:
 >> I've been been thinking for a while that it would benefit us to rework
 >> our release criteria
 >> (https://fedoraproject.org/wiki/Fedora_Release_Criteria) from being 
just
 >> a description of blocker bugs to more about the broader criteria that
 >> needs to be met to issue a public release.
 >
 > I was talking to James Laska about this earlier today, and we both have
 > thoughts on it too. I think it's a great idea to have revised and more
 > detailed criteria, and I especially like the organization - a main page
 > with an overview, and detailed criteria for each release.
 >
 > I had one major high-level suggestion. Release criteria are more or less
 > unavoidably partially subjective. I don't think it's feasible to come up
 > with concrete rules to cover every possible situation. Therefore, the
 > criteria should explicitly embrace and cover this subjectivity. It
 > should be made clear that things like 'boots successfully' are to some
 > degree dependent on subjective, contextual judgements; do we block the
 > Alpha for a bug that stops 0.1% of systems booting? 0.5%? 1%? 5%? I
 > think rather than trying to define something like this, we should just
 > explicitly acknowledge that it'll be a judgment call.
 >

I agree that there will always be some level of subjectivity.  The part 
that has bothered me in the past was the notion that because there is a 
level of subjectivity, defining things any more was impossible or a 
waste of time and that people should just be okay with it.  For the 
example you've given above I think we could extend it to say "most 
systems boot successfully" where 'most' becomes the part that is up for 
a judgement call.  Yes, I would agree that percentages for this example 
would not be helpful.  In other instances like "how many of the test 
cases need to be run or pass"--a percentage of completion or success can 
be useful depending on how it is written.

I agree that we can't come up with concrete rules to cover every 
scenario, but as you suggest I think we can add a few more parameters to 
explain the our thought process or be more explicit about where the 
judgement calls are and who will make them.

I'm expecting that creating more detailed release criteria will be an 
iterative process that we may not get right the first time and that is 
okay. This is exactly how the current feature process came to maturity. 
  We tweaked and changed it over two or three releases to the point that 
we rarely tweak it any more.  But when we do need to tweak it there is a 
larger framework to work within and we can make minor changes without 
having to change the whole process.

This is why I think the additional page layout will help the release 
criteria pages mature in the same way.  The feature process still isn't 
perfect, but it is worlds better than what we had before Fedora 8.  I'm 
advocating the same thing for our release criteria readiness 
decisions... where each release things get a little more specific, 
clearly documented and more widely understood by more people.  This is 
one way to scale and increase the chances of encouraging more new people 
to participate in our community.

John




More information about the fedora-test-list mailing list