Fedora.us QA (was: Re: Prelink success story :))

[Toshio]

FWIW: I only discovered fedora.us in December.
Note: Heavily snipped.  Please alert me to anything you think I took out
of context.

On Thu, 2004-02-26 at 17:43, Erik LaBianca wrote:
> I do
> think that the barrier for entry that I found was entirely too high. I'm
> just now beginning to understand what is going on after getting some
> feedback on some of my reviews.

I think the need to get feedback _on_a_QA_ is the heart of the problem
right now.  QA'ers need mentoring to develop their skills.  Jef, how
would you feel if some bug day we had experienced QA'ers team with new
QA'ers to nitpick packages?  This could be a good first or second time
QA experience: irc with a more experienced QA'er and pick apart the good
and bad of a package.  Or we could assemble a team of experienced QA'ers
to do second reviews once a new user did the first one.  This could be
useful for a more advanced QA'er.  Kinda have a short apprenticeship
followed by a test what you know phase.

> 
> 1. Information how "how to do some useful qa" is scattered.
I agree that as a new QA'er this was hard to deal with.  I felt like
there were five or six or seven lists of things that I had to keep in
mind.  Nowadays I use my judgement, followed by a quick walkthrough of
the major QAChecklist (and any sublists that that pulls in) and a
run-through of rpmlint on the SRPM and Binary RPM.

What I would like to see (for both QA and packager) is an RPM Best
Practices Handbook.  Accumulated wisdom about how to do things well
organized by use cases with in depth justifications available as side
passages for those who are curious why %{buildroot} and
${RPM_BUILD_ROOT} have such strong proponents on each side :-)
 
> 2.
[...QA Checklist in list form...]
> There should be a template like I have above, that can be answered
> with simple yes or no answers, that covers all the showstoppers. That
> way a newbie can fill it in the blank with yes's or no's and no he's
> done something useful.

I ran across my first QA that had such a list in it the other day and it
was pretty awful.  Part of that's the way bugzilla's structured to make
you scroll down, down, down.  Another part's that it doesn't organize
along good/needs-fixing/minor-quibble sections.  And the third's that it
tends to lead to a false sense of having finished the job when the
checklist is complete.

I wanted to write a fedora-qatemplate script that could generate a
template complete with checklist.  But I still haven't figured out how
the template should be structured to address all the shortcomings I
listed above.  (My best thought so far has been to abandon the CLI and
instead have an interactive GUI script with checklist that outputs
certain security related boilerplate and whatever checklist items fail.)

As QA stands (with all volunteer QA'ers) we can't depend on a newbie
doing first review and a seasoned vet doing the second review to figure
out what the newbie left out.  Besides, the seasoned vet is likely to do
the same things the newbie did to make sure those things really do check
out so having the newbie _just_ run through a checklist isn't that
useful.  What is useful is if the new QA'er has the sense of
responsibility to run through the checklist and the curiosity to test
whatever looks broken, fragile, or otherwise could might need
improvement.

> 
> The non-showstoppers should be on a second list of "stuff to watch for".
> This could be far more detailed than the actual QA checklist, and as a
> newbie gets deeper into packaging lore, they would likely begin filling
> out more of it. 
> 
In my mind I divide things into showstopper and stylistic.  With the
possible exception of the present wording of RPM_BUILD_ROOT, I think
everything on the list is a showstopper.  And as Michael lists, there
are many more that aren't on the list (because they don't occur often
enough?  Because there isn't consensus yet?  Because no one wanted to
turn the Checklist into a pre-flight manual with seventy pages
sixty-nine of which don't apply to this particular package?) 

> 1. Does the package follow the Fedora Package Naming Guidelines? 
> This is pretty darn complicated for a newbie QA'er. They should be
> allowed to opt out.
My problem with opting out was stated above: what if two QA'ers review
the package and both opt-out?

> 2. Are the sources from upstream pristine and free from trojans?
I agree with your statements of importance, difficulty, and in general
with your thought that we need to outline steps to take.  This is the
heaviest responsibility in the Checklist and it needs extra hints to
show what can be done to attain a decent level of assurance.

> 3. Are the pre- and post(un)install scripts correct?
> ...Again, concrete steps would be better. ...
I think this one exceeds the step-by-step.  Way too many things could
happen in the scriptlets to say definitively when it's done (more
entries for a best practices book, OTOH...)

> 5. Are there no missing BuildRequires?
> 
I agree that this can be basically impossible as it now stands.  I first
tried fedora-rmdevelrpms but that made my machine close to unusable as a
development environment (to QA or program.. Hmmmm...)

I've been trying to get mach to work, but I seem to run from problem to
problem with the damn thing.  There should definitely be a fedora-mach
HOWTO/FAQ/IRC forum....  So I'm reduced to reading the configure.in and
watching the build output which leads to embarrasment when I miss
something or mention an implicitly included dependency.

Another solution is to move half of the BuildRequires to the
autobuilder.  (Humans still have to check for "optional" BuildRequires. 
But the autobuilder can check for required Requires.)

> "Does the package build under mach". And the "how to build using mach"
> instructions can be
> 
> yum install mach
[Remember to `mach clean` before subsequent runs]
> mach build mypackage.src.rpm
> 

> 1. Relax on the whole GPG thing. ....
> When they [a new QA'er] first start, their input is going
> to be suspect anyway, so why slow down the process
> 
I might be a little attached to public key cryptography, but I think
it's an important added protection.  Bugzilla accounts can be traced to
creation dates and email addresses.  GPG keys add third party
signatures.  If crackers ever start trying to package compromised files
and get them into Extras, we will have more information to scan on to
try to figure out if we need to do more background checking on the
packager/QA'ers of a package.  OTOH, that might be completely emotional
and the additional security might not be that great.

> 2. Provide some feedback. I QA'd some packages. I waited.
I agree that reviews of work done encourage learning.  Perhaps the use
of keywords needs to go on the QA Checklist page?  Should people get
used to changing keywords often?  (Positive=REVIEWED;
negative=NEEDSWORK; 2nd positive in a row=PUBLISH);

And perhaps some of my ideas way back at the beginning of this message?
(Mentoring new users...)

[snippage]
> I think it's imperative that packages make it through the QA process. It
> doesn't do any good if packages never make it into the repo.

Speaking as someone who responds when someone points out a packaging
error but has never had a package make it through the queue: Sometimes
you've got to accept it.  I only package things I need.  If it doesn't
make it out the door, then not enough other people found it equally
useful.  *shrug*

That said, the most frustrating packages are not the ones that sit there
unreviewed, but the ones that sit there half reviewed (or with an older
version reviewed but not a newer one.)  It implies interest, but not
enough among active QA'ers to make the push over the hump.

Speaking as a fedora user/developer -- I don't care so much that
packages wait in the QA queue.  If I see something I want there, I
download and review it and if it works well I start driving :-)  I'm
sure "normal" users that don't want to do QA would see this
differently....

-Toshio
-- 
Toshio <toshio at tiki-lounge.com>