fedora-startqa

[Erik LaBianca]

> 
> > - A NEEDSWORK review is just as valuable as a PUBLISH +1 review.  I'd
> > like to see the script generate that as well.
> 
> Good idea, right now, the idea is to stop if a QA showstopper is found (no
> signature, build fails in mach), and let the QA'er write the NEEDSWORK
> review. This can be automated a little I think. Added on the TODO list.
> 

My personal opinion is that there shouldn't be a PUBLISH++ in the review template that is automatically output unless we are automatically doing ALL showstopper checks correctly, and the package passes. Currently we can't automate name checking, installation / uninstallation, or complete source checking, so we shouldn't print it out.

Whether or not we should print out "NEEDSWORK++" or something similar is up for debate, probably a good idea.

> > - (Showing my ignorance of mach) How safe is it to build untrusted
> > sources within mach?  since mach builds the package before the user gets
> > a chance to go look at whether the Source URL is canonical, I was
> > wondering....

I think I tackled this on in another email. Synopsis: mach is defined as a secure build environment. If it breaks, we need to fix mach. The truly paranoid should do QA under a vserver, UML or even better on a dedicated machine.

> 
> > - Review has "Installs, runs, and uninstalls fine on FC1" but I haven't
> > done any of that yet -- should it be in TODO?
> 
> It is always in the TODO anyway. Erik also thinks that it should not be
> there, so I'll remove it, but I've put it there to remember the user to
> tell which distro he has tested the package on, and to check
> uninstallation. I think that nothing prevents a user from doing a false
> review anyway, and I wanted to make a template where nothing but the
> "notes" had to be added. Anyway, if the majority thinks it's wrong, let's
> remove it.
> 

My preference is for the review template to have a series of "blanks" to be filled in by the reviewer. A script like qa-assistant could take the output of our automated program and provide hand-holding for the user through filling in the rest of the items.

I prefer to have a series of lines like this:
Builds OK?:		YES (fc1,rh9) NO(rh8)
Name OK?:         unchecked
(Un)Installs OK?: unchecked
Secure?:          unchecked

The idea here being that the reviewer has a very brief idea of what is required to complete the review beyond what we do automatically, and must sign their name to the fact that they've checked those things when they change "unchecked" to YES.

If they didn't bother to do either, but posted the review anyway, it would still be a useful data point.

> > However, GPG signed SRPMs are equivalent to
> > checking a GPG signed md5sum file that has an  md5sum for the SRPM.  So
> > my view is if the GPG signature on the SRPM is good and the MD5SUM file
> > doesn't contradict it (ie: different signing keys, different MD5Sums for
> > the same file) it shouldn't error out.
> 
> Yes, there is this -c option to disable srpm md5sum checking.
> 

Agreed. MD5sum's should just be printed out and noted to be checked if they don't exist or mismatch in my opinion. See rant in previous email.

--erik