How to do QA

[Michael Schwendt]

On Fri, 06 Feb 2004 02:14:19 +0100, Jonas Pasche wrote:

> I have put together a page that might help people doing QA:
> 
> http://jonaspasche.de/fedoralegacy/qa-howto.html

In such a detailed form this goes into a wrong direction IMHO. Some
comments following... but first:

  Disclaimer in front: This is no attempt at stopping anyone from trying
  to create a "how to do QA" document, although I think there's no general
  recipe on how to do QA. If there are people who find such documents
  useful and if it helps them actually to get started with package reviews
  and approvals, I shut up. At this time, however, I'd like to avoid that
  the "QA hurdle" is set too high or that it is considered as set too
  high. Apart from that, copying some things from the fedora.us checklist
  simply doesn't make any sense.

I like to point out that people who are new to reviewing packages should
follow a top-down approach to testing packages (and build teams with
people who look a level lower) instead of a bottom-up approach which
starts with low-level technical package reviews: If Fedora Legacy wants to
provide updated packages which fix security issues or critical bugs, the
primary objective is to provide packages which fix those issues and which
don't break anything else. This requires testing the built binary
packages. If no binary packages are provided, it may be necessary to
rebuild the source rpms. Like Eric Rostetter's page, an introduction to
testing packages should start with stuff like how to rpmbuild a src.rpm as
non-root user, how to extract a src.rpm, how to create an rpm diff against
the previous src.rpm, and maybe later continue with additional checks at
the level of the source rpm or binary rpm (e.g. ldd, rpm -qlvp and rpm -qR
checks), meeting the requirements of a build system (=> completing the
build dependencies). However, Fedora Legacy aims at modifying the source
rpms as little as necessary, so reviewers don't generally need to spend
any time reading the spec file beyond the diff against the previous
release.

Fedora Legacy has different requirements than Fedora.us. Such a checklist
gives a false impression of what kind of package reviews are needed. For
instance, the Fedora.us QA checklist has been misunderstood by people
repeatedly as it covers many low-level issues but doesn't spend a single
word on how much "in production" testing is required or when a package is
classified as "stable" instead of "testing" or "unstable", respectively.
A checklist makes the package verification procedure appear overly
complex, complicated [and maybe even pedantic]. Or people go through the
list step by step and don't examine anything else. Fedora.us has different
requirements because it deals with entirely new package build scripts,
which are often written from scratch or sometimes derived from other
distributions. Sometimes a packager doesn't even take the time to
understand what is done in the package build script and whether it does
what it is supposed to do. Such circumstances require a different level of
reviewing package submissions.

Quite some of the Fedora.us guidelines and policies have been created, so
the initial repository is filled with a solid base of usable packages,
which other packages can built upon and which also serve as examples and
inspiration, and to avoid that the repository is turned into a dumping
ground for a wild mixture of packages, which would probably require
increased maintenance upon updates and/or distribution upgrades (e.g. even
improperly set dependencies can break a repository easily). Other
requirements and procedures (e.g. different levels of trust and low-level
package reviews) are because in an open community project you meet and
work with people you don't know or haven't worked with before (do you
build from src.rpm without comparing the package contents with its
previous release?). Many of the steps in the linked/copied fedora.us
checklist don't apply at all to Fedora Legacy. I feel that starting with
this list as a basis is approaching the "QA problem" from the wrong
direction. Start at the top and refine the procedures and policies.

-- 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 189 bytes
Desc: not available
URL: <http://listman.redhat.com/archives/fedora-legacy-list/attachments/20040206/0f65c225/attachment.sig>