Managing Fedora Testing

[Jef Spaleta]

Keith Lofstrom wrote:

> I would put myself in the 45% group, and include many of the folks >
that are complaining about booting or mirror problems.  The Fedora >
test process seems to be geared exclusively for the 5%.  As a 
> result, Fedora will release with less than 10% of the potentially 
> available testing effort, with the remaining effort 
> disproportionally focused.

I would sort of agree with that. I would agree that the aggressive 
timescale for testing for fedora core means relying on those 5% (I'd put
it closer to 1%) to be a strong component of the process. I would also
say that the dropping of the private betas that rhl use to have makes
the situation somewhat more dramatic looking than it is. As others have
said, the fedora testing process, is a more open process than what
happened with rhl. And more open translates directly into sharper edges
for the informed to impale themselves on while blundering around in the
dark.

There is a trade-off here, taking the very long time to do things a
piece at a time, means the pieces quickly become out of sync with
upstream development. Open source projects, especially 'desktop'
oriented projects are undergoing relative rapid pace of development
churn. Doing it the way your example describes with electronics hardware
breaks down, if the point is to really push things forward.
I point you to the list of objectives for fedora. #3 and #5 demand an
aggressive testing/release cycle. Doing it the very conservative way
isn't in line with the objectives. And arguing that the objectives are
bad...is just a waste. The objectives are what they are, the point is to
find a process that fits best to meet the stated objectives, not argue
for the best possible testing process that can take infinite time to
produce the perfect release. So i really don't see mucb room in tweaking
the stated timescales. 

That being said, i think there is a lot of room on how to organize the
more open testing process. If small group, invitation only, betas are
not something that can be reconsidered in an effort to reduce the
signal-to-noise then something should be done to have testers
self-identify themselves as to skill level and steps should be taken to
layout to new testers what the expectations are if they are going to
become involved. I would personally like to see at least a token effort
made to make a list of things that testers are agreeing to do and to the
expectations testers should have in their heads, before they venture
into running a test release.  Too many people show up on the irc
channels, or in the mailinglist, with the wrong expectations.
Sadly, I don't think better documenting a beta tester manifesto is going
to be enough. I think there has to be a way to confirm that testers are
prepared to actually help provide value instead of just noise. If i
could do it, I'd demand that all testers pass a short quiz before they
got the ability to file bug reports or become active in the
mailinglists. A quiz based on yet to be written documentation that lays
out what testers should expect from the releases and more importantly
what developers expect testers to do. I wish i had time to work on
documentation like that.  As it stands, learning how to approach test
releases is a trial by fire for most people...and fire burns.

-jef"i should stop writing emails while having a high fever, I'm sure
this one just sounds like gibberish"spaleta