Fedora.us QA (was: Re: Prelink success story :))

[Michael Schwendt]

[changed subject line!]

On Thu, 26 Feb 2004 17:43:22 -0500, Erik LaBianca wrote:

> As a longtime redhat user (4+ yrs) and not-entirely-new rpm packager (2
> yrs) and professional system administrator / software engineer, it took
> me over 8 hours of work before I submitted my first QA review. This is
> partially because I hadn't been using GPG, partially because I didn't
> have a bugzilla account, partially due to general incompetence on my
> part, and partially due to the amount of material in the QA checklist.

... and, of course, because you've realized that you take a big portion of
responsibility with your approval of a package, so you wanted to aim
at perfection, didn't you? ;)  Yes, of course you wanted to avoid any
mistakes. :)

> 1. Information how "how to do some useful qa" is scattered.

Remains the question whether you could really instruct newbies without the
help of a book or a few courses.

> 2. Too much detail. The QA checklist is great. However, as it turns out,
> a lot of the stuff being checked there aren't really showstoppers. I
> ended up turning it into a list

I added an entry to the checklist yesterday. A show-stopper IMO.
To check file permissions and ownership. Has happened several times,
that a %defattr was missing or files/directories were inaccessible.

> There should be a template like I have above, that can be answered
> with simple yes or no answers, that covers all the showstoppers. That
> way a newbie can fill it in the blank with yes's or no's and no he's
> done something useful.

But would that be useful? Do you expect someone to go through the list a
newbie posted and check step by step whether he made a mistake?  I think
the checklist is for personal use only and should not be attached to
package requests as a Yes/No answered list. Approvals should be short and
to the point and concentrate on a few items. If I had to fill out a list
of what I check, that would slow me down a lot unless some special
software would aid me upon maintaining the list.

> Since all the checklist items are there, answered
> with yes or no questions, it's easy to know if they actually made a good
> faith effort to check the package.

Which increases the workload a lot, when many items are not needed
for a particular package or when many more items are added to the list
(because it isn't complete). Reviewers should develop a feeling for
what is important and what is not.
 
> Some critique of the list:
> 
> 1. Does the package follow the Fedora Package Naming Guidelines? 
> This is pretty darn complicated for a newbie QA'er. They should be
> allowed to opt out. A "senior developer" could look at the package in 10
> seconds and understand whether or not the name is ok.

But package naming and versioning issues don't stop anyone from reviewing
the rest of a package, do they?
 
> 2. Are the sources from upstream pristine and free from trojans?
> This is important, but exceedingly difficult to do "properly"... At what
> point does this become a showstopper? At some point it becomes
> impossible to verify the source without actually being the author of the
> code and inspecting it line by line. For a newbie this is very
> intimidating

Of course. Still, the least someone can do is compare the MD5 checksums
and also check packages from other distributions (not because they are
trusted ultimately, but to fetch the source from multiple locations) and
visit the home page (and e.g. directories like freshmeat.net) to check
whether a release might be official.

Asking upstream authors for MD5 checksums via e-mail doesn't work
well. I've tried without luck to get some to post clearsigned checksums in
the download area.

> 3. Are the pre- and post(un)install scripts correct?
> Honestly. How the heck is a newbie supposed to know this?

Depends on how new he's to RPM.

> Again, concrete steps would be better.

Somebody would need to create such documentation. There are entire books
about RPM. And you could fill more chapters with documentation on when a
%pre/%post script can be considered "correct".

> I'd suggest something like "are all
> pathnames in the pre/post(un) install scripts complete?

What does that mean? Macros in scripts are okay and are expanded at
build-time. Environment variables are not okay because they are evaluated
at run-time.

> Are macros used for all package files?

Depends on whether it makes sense. If the source code has some paths
hardcoded, it doesn't make much sense to use corresponding macros in the
spec file.

> Also. Wheres the "does this package appear to work" line item? Do we QA
> packages and not actually run them? It seems like this is pretty basic
> criteria.

Are step by step items on "does it build?", "does it install?", 
"does it upgrade?", "does it erase?" really necessary?

> 4. Epoch
> 
> This appear to be subject to debate, and isn't included in a lot of
> upstream rpms. That being said it's easy to check so leave it in.
> Couldn't / doesn't rpmlint check this?

It does.
 
> 5. Are there no missing BuildRequires?
> 
> This one took me a LONG time to check. The manual "remove all devel
> rpms" method is apparently deprecated according to the url above,
> however fedora.us doesn't even include a copy of mach. No less, mach is
> pretty complicated and you've got to figure out how to create a
> fedora.us enabled buildroot.

"mach" is not mandatory. I've tried it once, long ago. Use what works for
you. It can be embarrassing if an approved package fails in the build
system.
 
> 6. Is the Group tag an official one? See /usr/share/doc/rpm-*/GROUPS.
> 
> Not sure if this is relevant to anything, but again, it's easy to check.
> Couldn't rpmlint do this?

AFAIK, rpmlint checks whether a group is known, but not whether
a package is put into the proper category.

> 8. If the package has a DesktopEntry, are the VFolder categories ok?
> 
> I don't have any idea what any of this stuff is. I don't build desktop
> software. Some documentation about it might be good.

Documentation is linked (www.freedesktop.org).
 
> 10. Is the package free of unowned directories? 
> 
> This is also very hard to check. You mean I have rpm -ivvh this package
> and then scroll through a whole pile of cruft to see the unowned
> directories? And then I have to know which ones it's "supposed" to own,
> and which ones it's now? 

You can also "rpm -qlv |less" the package and make sure that it owns
its _own_ directories. ;)

> The packages I checked didn't own /usr, /usr/lib, /usr/bin. I assume
> they aren't supposed to?

$ rpm -qf /usr /usr/lib /usr/bin
filesystem-2.2.1-5
filesystem-2.2.1-5
flash-plugin-6.0.79-2
filesystem-2.2.1-5

Ooops! :)  Flash-plugin owns something it shouldn't.

Lots of directories belong to the filesystem and major core packages
(e.g. kdebase) already.

> This is a good thing to check, but theres gotta be a way to automate
> this, and if nothing else, explain what paths are ALLOWED to be unowned.

This depends on a package's dependencies, e.g. whether the package adds
files to the directory of a package it depends on. In that case it would
not need to own the directory.
 
> 14. Does the package handle a parallel build cleanly?
> 
> Does this really matter? Seems like a nice thing to check but not a
> showstopper. Make it go away.

It is a showstopper, because a package which "make"s with %_smp_mflags in
the spec file might break badly in the build system. It is not always
obvious whether such breakage is due to SMP make flags. Defining
"%_smp_mflags -j3" in your local ~/.rpmmacros file also on UP machines
can help detect breakage.
  
> 11. Does it pass rpmlint cleanly?

Not always possible. :)
 
> With good solid descriptions for 1-6 especially, I think it will become
> much easier to get useful information from would-be QA'ers. I'd like to
> see the how to help page say "Please do QA. Heres how"

And when issues not covered in the list come up, those people are fed up.
E.g. but not limited to:

 * build not accepting $RPM_OPT_FLAGS
 * build stripping binaries
 * bad -rpath found
 * archdependent files below %_datadir
 * not adhering to FHS
 * spec file sections in wrong order (e.g. builds in %install section)
 * libtool archives not deleted and not needed or wrong paths
   in libtool archives
 * *.so links in non-devel package and not needed there
 * python *.py{c,o} files not marked %ghost
 * perl modules installed into site location, not vendor location
 * macros in %changelog
 
> 2. Provide some feedback. I QA'd some packages. I waited. And waited. A
> week later, AFTER I saw a discussion about bugzilla keywords and added
> "NEEDSWORK" to the packages I had QA'd,

NEEDSWORK is to move packages out of the QA queue, because they don't
build or need more work before they are moved back into the QA queue.
This can also be set by the packager if major changes are planned.
REVIEWED is the new keyword to flag reviewed packages, so they are
easy to locate.

> I got my first piece of
> feedback. This doesn't encourage repeat customers. It's a whole lot more
> rewarding when you submit something, and an hour later you get a message
> back from somebody saying "Hey, you messed up, but thanks for trying,
> can you please check a, b, and c?". 

Lack of man power. More reviewers are needed. Especially since we don't
know how/what official Fedora Extras will change.

> I think it's imperative that packages make it through the QA process. It
> doesn't do any good if packages never make it into the repo. Right now,
> they appear to just sit there. Then the packager gets fed up, doesn't
> respond if anybody checks his package, and starts a new repo.

Not every packager can work on his packages as soon as someone found
issues. Sometimes you need to wait till the next weekend. If the packager
doesn't respond, simply move the request out of the package queue by
deleting the QA keyword and setting the NEEDSWORK keyword.

> This is
> not building community. This process has GOT to be fast enough that when
> I submit a package for something that people actually use, it gets QA'd
> SOON, before I've lost all interest in it.

If there are people with interest in a package, surely there should be
two reviewers, too?

--