Prelink success story :)

Thu Feb 26 22:43:22 UTC 2004

> What would that change? We've talked about it, criticism has been
noted,
> and as I've tried to make clear, the checklist should not be
> misunderstood. There is no silver bullet. One could create a different
> checklist for every different type of package. The biggest hurdle to
QA is
> lack of common sense. I don't want to spend a lot of time editing
> documentation in the Wiki to please a single individual (read "you")
who
> runs his own independent repository and doesn't really care. I'd
rather
> like to know how to lower the hurdle for other people who would like
to
> help, but who still don't know where to start. And that would mean
that
> they start talking about any problems they see.
> 

As a new-to-fedora.us prospective QA'er, I've been attempting to compose
an email on this very subject for the last several days, but haven't
managed to complete it. It's going to be long, so please forgive me

As a longtime redhat user (4+ yrs) and not-entirely-new rpm packager (2
yrs) and professional system administrator / software engineer, it took
me over 8 hours of work before I submitted my first QA review. This is
partially because I hadn't been using GPG, partially because I didn't
have a bugzilla account, partially due to general incompetence on my
part, and partially due to the amount of material in the QA checklist.

I'm happy to discuss this stuff in more detail, but I'm just going to
throw out some "newbie observations" for you all to think about. I do
think that the barrier for entry that I found was entirely too high. I'm
just now beginning to understand what is going on after getting some
feedback on some of my reviews.

1. Information how "how to do some useful qa" is scattered. Heres only
some of the url's I had to look at to figure out whats going. None of
them were complete in and of themselves, and my first two attempts at QA
were still sub-optimal.

http://www.fedora.us/wiki/PackageSubmissionQAPolicy

http://www.fedora.us/pipermail/fedora-devel/2003-March/000459.html

http://www.fedora.us/wiki/QAChecklist

http://www.fedora.us/wiki/HOWTOFindMissingBuildRequires

I also got some good mileage out of one of Jef Spaleta's bug triage
messages to the list.

2. Too much detail. The QA checklist is great. However, as it turns out,
a lot of the stuff being checked there aren't really showstoppers. I
ended up turning it into a list

1. Does the package follow the Fedora Package Naming Guidelines? 
2. Are the sources from upstream pristine and free from trojans?
3. Are the pre- and post(un)install scripts correct?
4. Epoch
5. Are there no missing BuildRequires?
6. Is the Group tag an official one? See /usr/share/doc/rpm-*/GROUPS.
7. Are all paths within the .spec replaced with macros?
8. If the package has a DesktopEntry, are the VFolder categories ok?
9. Is the package free of any %{buildroot} macros? 
10. Is the package free of unowned directories? 
11. Is the License tag used rather than Copyright?
12. Is the package free of default passwords? 
13. Are all daemon's which can be installed to run as non-root installed
as such? 
14. Does the package handle a parallel build cleanly?

Now. Some of these items apparently are subject to contention. If
there's any chance that they aren't showstoppers, lets get them off that
list! There should be a template like I have above, that can be answered
with simple yes or no answers, that covers all the showstoppers. That
way a newbie can fill it in the blank with yes's or no's and no he's
done something useful. Since all the checklist items are there, answered
with yes or no questions, it's easy to know if they actually made a good
faith effort to check the package.

The non-showstoppers should be on a second list of "stuff to watch for".
This could be far more detailed than the actual QA checklist, and as a
newbie gets deeper into packaging lore, they would likely begin filling
out more of it. 

Some critique of the list:

1. Does the package follow the Fedora Package Naming Guidelines? 
This is pretty darn complicated for a newbie QA'er. They should be
allowed to opt out. A "senior developer" could look at the package in 10
seconds and understand whether or not the name is ok.

2. Are the sources from upstream pristine and free from trojans?
This is important, but exceedingly difficult to do "properly"... At what
point does this become a showstopper? At some point it becomes
impossible to verify the source without actually being the author of the
code and inspecting it line by line. For a newbie this is very
intimidating

My suggestion would be to have a set of no-fail, simple steps that
result in a concrete answer, like the following: "verify the source url
exists, and check to see that it is at the official distribution site
for the software. Compare the md5 sums of their posted souce tarball and
the one included in the SRPM. If it doesn't, please detail your
findings".

3. Are the pre- and post(un)install scripts correct?
Honestly. How the heck is a newbie supposed to know this? Again,
concrete steps would be better. I'd suggest something like "are all
pathnames in the pre/post(un) install scripts complete? Are macros used
for all package files? Are all operations backed out? Is ldconfig being
run if there are shared libraries installed? Maybe these should be a
series of questions

Also. Wheres the "does this package appear to work" line item? Do we QA
packages and not actually run them? It seems like this is pretty basic
criteria.

4. Epoch

This appear to be subject to debate, and isn't included in a lot of
upstream rpms. That being said it's easy to check so leave it in.
Couldn't / doesn't rpmlint check this?

5. Are there no missing BuildRequires?

This one took me a LONG time to check. The manual "remove all devel
rpms" method is apparently deprecated according to the url above,
however fedora.us doesn't even include a copy of mach. No less, mach is
pretty complicated and you've got to figure out how to create a
fedora.us enabled buildroot.

After all that mach works great, except for I found out that it doesn't
understand perl(Net::Lib) style depends, so it doesn't handle on
buildrequires. 

There needs to be an official build environment. Mach works. Why not
spend the hour it takes to get a copy of mach in the repo, with
fc1+extras as the default buildroot? Then you can change this line to
"Does the package build under mach". And the "how to build using mach"
instructions can be

yum install mach
mach build mypackage.src.rpm

6. Is the Group tag an official one? See /usr/share/doc/rpm-*/GROUPS.

Not sure if this is relevant to anything, but again, it's easy to check.
Couldn't rpmlint do this?

7. Are all paths within the .spec replaced with macros?

Relatively easy to check, but I'd had to be tasked with this without
having built a few rpms in the past. A howto for this process might be
nice.

8. If the package has a DesktopEntry, are the VFolder categories ok?

I don't have any idea what any of this stuff is. I don't build desktop
software. Some documentation about it might be good.

9. Is the package free of any %{buildroot} macros? 

Apparently this is subject to debate. It's not a showstopper. Make it go
away.

10. Is the package free of unowned directories? 

This is also very hard to check. You mean I have rpm -ivvh this package
and then scroll through a whole pile of cruft to see the unowned
directories? And then I have to know which ones it's "supposed" to own,
and which ones it's now? 

The packages I checked didn't own /usr, /usr/lib, /usr/bin. I assume
they aren't supposed to?

This is a good thing to check, but theres gotta be a way to automate
this, and if nothing else, explain what paths are ALLOWED to be unowned.

11. Is the License tag used rather than Copyright?

Easy. Rpmlint should/can? Check this.
12. Is the package free of default passwords? 

Not necessarily easy to check, but easy to understand, and important.

13. Are all daemon's which can be installed to run as non-root installed
as such? 

Same as 12.

14. Does the package handle a parallel build cleanly?

Does this really matter? Seems like a nice thing to check but not a
showstopper. Make it go away.

Ok. So. Heres what I suggest to replace it all with

Showstoppers:
1. Are the sources from upstream pristine and free from trojans?
2. Does the package build clean under mach?
3. Does the package follow the Fedora Package Naming Guidelines?
4. Does the package appear to work?
5. Is the package free of default passwords? 
6. Are all daemon's which can be installed to run as non-root installed
as such?

Additional Critiques:
7. Are all paths within the .spec replaced with macros?
8. If the package has a DesktopEntry, are the VFolder categories ok?
9. Is the package free of unowned directories? <-- at least explain this
10. Does the package handle a parallel build cleanly?
11. Does it pass rpmlint cleanly?

With good solid descriptions for 1-6 especially, I think it will become
much easier to get useful information from would-be QA'ers. I'd like to
see the how to help page say "Please do QA. Heres how"

1. Create bugzilla login
2. Follow this link to the list of packages needing QA.
3. Follow these instructions and fill out this checklist and post the
results in a comment.

Some additional notes:

1. Relax on the whole GPG thing. GPG is great. I figured it out. So can
anybody else. But don't make it a barrier for entry. For a newbie, it's
utterly unnecessary. The first thing you want people to do is to go
through the package, create a bugzilla login, and post a completed
checklist like I did above. When they first start, their input is going
to be suspect anyway, so why slow down the process by an hour by
figuring out how to use GPG? Bugzilla passwords aren't entirely
insecure...

2. Provide some feedback. I QA'd some packages. I waited. And waited. A
week later, AFTER I saw a discussion about bugzilla keywords and added
"NEEDSWORK" to the packages I had QA'd, I got my first piece of
feedback. This doesn't encourage repeat customers. It's a whole lot more
rewarding when you submit something, and an hour later you get a message
back from somebody saying "Hey, you messed up, but thanks for trying,
can you please check a, b, and c?". 

I realize everybody's volunteering here, but there has GOT to be a way
to detect when somebody new has posted to bugzilla and go check out what
they did. If there were a "newbie comments" query available in bugzilla
it might help. It's a set the hook sort of thing.

3. Provide a way for people to QA packages which have already been QA'd,
so the 2 reviews happens. I think there has been some discussion about
this. A link to an appropriate bugzilla query from the "How do I get
started" page would be great. 

I think it's imperative that packages make it through the QA process. It
doesn't do any good if packages never make it into the repo. Right now,
they appear to just sit there. Then the packager gets fed up, doesn't
respond if anybody checks his package, and starts a new repo. This is
not building community. This process has GOT to be fast enough that when
I submit a package for something that people actually use, it gets QA'd
SOON, before I've lost all interest in it.

In addition, there needs to be a way for a motivated packager to
convince people to QA his work. Is sending email to fedora-devel saying
"please QA my new widget package?" acceptable? 

Maybe a mailing list that automatically gets CC'd all new Fedora Meta
New package Requests, NEEDSWORK, and REVIEWED keyword changes would
help? Or maybe they should just go to fedora-devel?

Anyway, just my $0.02. Hope you're still awake!

--erik