QA process was Re: RPM submission procedure

[Michael Schwendt]

On Sat, 10 Jan 2004 04:00:29 -0500, Gene C. wrote:

-- snip --

Disclaimer in front: I've posted so much to this topic, I think I need to
point out that all this is my personal opinion and in no way official for
fedora.us. Consult the fedora.us leadership for official statements.

> Right now I believe that ESR (and the 40 packages he maintains) has been
> definitely alienated by this discussion.

So? Is that a problem? What he has in mind is simply not possible yet.
Fedora.us is damned short on human resources. There's no automated build
system. There's no automated system where ESR could submit package updates
and get them built. Fedora.us does scale, provided that with each new
package, additional people join the "fun".

> I really like Alex's bleeding/testing/good/stable classification of packages.  
> It should be easy to get packages into the bleeding or testing categories

*What* is so difficult to get a package into fedora.us "testing" or
"unstable"? I still don't understand it.

> but 
> to get into either good or stable should require lots more QA.  By QA I mean 
> TESTING by lots of people ... not code reviews by a couple of "experts".

Where do you take these things from? Who does "code reviews" [*]? You can
be happy if one or two people test a package prior to publishing it. A
rock solid package can be in testing for a long time without a single bug
report being submitted. What does that tell you? Nothing. The next update
already could introduce a showstopper bug which must be caught prior to
release. But sometimes these are very difficult to find and affect only a
small target group. Even the code maintainers miss them until they receive
a bug report.

[*] Except maybe skimming over context diffs upon upgrades. Have you ever
watched the sized of a diff between two releases of e.g. the GIMP 2
development branch? It would be a full-time job to "review" something like
that at the level of source code.

> My 
> experience is that very subtle bugs (or intentional compromises) can fool 
> even experts who have access to source code but lots of testing (running the 
> software) by lots of people can reveal that the software does have problems 
> (even if they cannot point to specific causes).  

So what? Do you consider the upstream source tarball checksum verification
step unnecessary?

> Your do not get lots of 
> testing if the package is not "available".

Then let users vote on what they want to see published and what they will
test-drive. Package requests can be prioritized based on demand.

Btw, do you wait for any specific software to be published? Or what
is your intention?

> I am also seeing QA requirements well beyond what Red Hat does internally 
> (from my perspective) and what I believe is reasonable. 

*Where*? I don't see that. I see mostly low level technical reviews which
want to ensure that the software builds and installs correctly, is set up
properly, has good dependencies and that it "seems to work" (if the
reviewer does not consider himself qualified enough to do expert
tests). Sure, for some packages it is different. Some reviewers are very
familiar with some software and test-drive the software before they would
approve a release. But there is no such definition of QA at fedora.us.

> While I believe that 
> some QA rules are needed, lets make them realistic ... lots of rules about 
> package format are reasonable but the quality of the code in the package will 
> only be discovered through testing (actually running the software).

Package format is not an issue at all. Packagers are encouraged to keep
their spec files readable and easy to maintain and adapt what is
considered common practise. They are not forced to do that.

> As I see it, a lot of the Extras (fedora.us) packages are more like the gftp 
> package rather than the kernel, gcc, or glibc which need lots of testing 
> before becoming stable.  For gftp, Red Hat has taken the software package 
> from upstream and then packaged it for Red Hat / Fedora Core making sure that 
> it puts stuff into the correct directories, etc.  However, the Red Hat folks 
> do not make sure that the software is working as it should ... that is an 
> upstream responsibility.  The 2.0.14 version of gftp which is part of FC-1 
> has problems with http.  I worked with the upstream maintainer/developer to 
> fix these problems and most of the fixes are in 2.0.16 (with the remainder in 
> CVS).  While there is no gftp update for FC1, 2.0.16 is available in rawhide 
> (development) for those with the need.

Red Hat QA is not bullet-proof either. And they may have different
priorities from time to time.

A gftp 2.0.16 package can also be found in fedora.us "patches". It could
have been released as an official Update to FC1 if there had not been a
misunderstanding. The fedora.us package was "tweaked", i.e. based on a
different build script than Red Hat's package. In the future, it will be
better if external updates to packages in FC are based on Red Hat's
package and only modified where absolutely necessary.
 
--