Fedora 12 QA retrospective - feedback needed

[Adam Williamson]

On Wed, 2009-11-25 at 14:29 -0500, James Laska wrote:
> Greetings folks,
> 
> Whether you call it a post-mortem, retrospective or lessons learned ...
> the end result is the same.  I'd like to collect thoughts on how
> good/bad of a job the QA group did in planning and testing the Fedora 12
> release.  In keeping with the release-wide retrospective from Fedora 11
> [1], feel free to share any wishlist items as well.
> 
> I've started the discussion on the wiki at
> https://fedoraproject.org/wiki/Fedora_12_QA_Retrospective.  
> 
> Adding your thoughts is easy ...
>       * Edit the wiki directly (instructions provided for ~anonymous
>         feedback)
>       * Or, reply to this mail (I'll collect feedback and add to the
>         wiki)
> 
> Over the next week, I plan to organize any feedback and discuss the
> highlights during an upcoming QA team meeting.  The goal will be to
> prioritize the pain points and use as a basis for defining objectives
> for Fedora 13.
> 
> Thanks for your feedback!
> James

1) in the end, we focused heavily on just three component areas for
testing: anaconda, kernel, and X.org. This is primarily a function of
the fact that these are the most vital bits; it feels like we're still
at the point of doing 'let's make sure it's not totally broken' testing
than 'let's make sure it's really good' testing. We didn't do stuff like
making sure the desktop was polished.

2) there was clearly a lot of uncertainty about RAID issues; it's
something we obviously don't as a team test well enough (and some of us
personally don't understand enough :>). In the end there didn't turn out
to be any horrible issues, but the confusion was evident, and we did
miss the Intel BIOS RAID stuff-ups. For F13 we should have better RAID
testing both in Test Days and in pre-release test cycles.

3) We weren't completely on top of X.org bugs for this release. The ones
that wound up getting promoted to release blocker level were kind of an
arbitrary selection. I think we got nouveau mostly right as I had a
reasonable grip on nouveau triage, but we just had too few people to
triage server / intel / ati bugs during the cycle, so when we hit beta /
RC stage, we didn't have the whole bug set well enough triaged to be
able to be sure we picked the most important bugs as blockers. For F13
we should stay on top of triage better so we can do blocker
identification accurately. happily, matej is more active on X triage
again now and we have some more assistance from Chris Campbell (thank
you Chris!) further volunteers would be great. I will try to stay on top
of nouveau, again.

4) we have the big security thing to deal with. I did start a thread on
-devel about that.

5) test days went well again. it was nice to see how many 'independent'
test days there were.

that's what i've got so far :)

-- 
Adam Williamson
Fedora QA Community Monkey
IRC: adamw | Fedora Talk: adamwill AT fedoraproject DOT org
http://www.happyassassin.net