Risk Models and QA [Re: no mandatory QA testing at all [Re: crazy thought about how to ease QA testing]]

[David Eisenstein]

On Tue, 14 Feb 2006, Mike McCarty wrote:

> 
> >>Unless I hear major objections in two days, I'll start the two-week 
> >>clock (from today) for all the pending packages.
> 
> Ok then, it seems to me that there is no longer any distinction
> between the released repository, and the test repository.
> So, please send out an e-mail three days before the first
> "timed release" so I can pull a last tested version before
> removing the legacy repository from my yum configuration.

Hi Mike, and all,

I just want to let you know that I have reservations about this as well.
You are not alone in the way you feel about this proposed and now
implemented way of ... of getting stuff done here.

For my systems, I end up choosing not to use yum's automated updating
feature.  This is the way I've done it from day one, even when Red Hat was
maintaining the security updates and upgrades for FC1.  Even when packages
end up going through all the stages of Fedora Legacy's previous QA
methods, I still apply updates with caution, and always provide a way to
restore the previously installed version of a given software package in
case things suddenly stop working right after applying an update.

How one chooses to update one's system(s) ends up being a matter of either
personal policy or, if doing updates for a company, a matter of company
policy.  We have what I consider to be an excellent discussion of that
kind of local decisionmaking (when and how to use Automatic Updates) on
our wiki, at <http://fedoraproject.org/wiki/AutoUpdates>.  I highly
recommend reading it.

My experience with Fedora Legacy over the year-and-a-half that I've
participated in it is that the Legacy project does not have hard-and-fast
standards for what people do in the last stage of QA testing (that is, the
"QA Verify" testing -- testing of packages for release to updates, as
documented in http://fedoraproject.org/wiki/Legacy/QAVerify).

My experience is, in many cases, the people who do that QA Verify testing
for the Fedora Legacy Project (including myself) are simply installing the
package on their test system, running it through as little as a few paces
or as much as many days worth of running it in a production-like
environment, and then saying, "Ok, it works for me!"  In many if not most
cases of QA testing, it seems to me that the people who do it do *not*
specifically test the new package for the security vulnerabilities we are
purporting to fix.  At least part of the reason for that is that the test
cases are not available to us -- they're either embargoed somewhere, or
the vulnerability is rather theoretical and has never been proven to harm
anyone.

This is not a scientific nor rigorous way of doing QA testing.  But due to
the limited time and population of folks who do testing in our community,
it is the probably the best we can expect.  As hard as it is for a perfec-
tionist like me to accept, we're not in the business of excellence here 
so much as we're in the business of trade-offs.  With limited manpower and 
time and expertise, the issue to me becomes -- "Which is the least risk?"

When we can get more manpower to check packages more thoroughly, our risk
goes down.  When we have less manpower, the risk goes up.  But if we are
to be a viable project, the sense of the participants of this list as I 
read it is that we need to have a product pushed out in a timely fashion.

For some people, the Fedora Legacy Project risk model is not for them.  By 
the same token, for some people, the Fedora *Project* risk model is not 
for them, either.  Not unlike the Fedora Project, the use of Fedora Legacy 
packages comes with a different risk model than some project like CentOS 
or Debian.  Fedora Project is closer to the "bleeding edge," and so it is 
not unreasonable to expect that Fedora Legacy may also be that way.

We do the best we are able.  Anyone who wants to do QA testing may do so.
We welcome participation at whatever level you are willing and able to do,
Mike, and everyone.

Thank you for expressing your thoughts, Mike.  I appreciate them.

	Warm regards,
	David Eisenstein