crazy thought about how to ease QA testing

[Benjamin Smith]

On Friday 10 February 2006 11:22, Pekka Savola wrote:
> Jim, you're probably missing the fact that VERIFY QA doesn't include 
> the steps "test if the patch worked; test if the vulnerability is 
> fixed".  While some folks do perform more rigorous testing, it's not 
> required, and for a good reason.
> 
> Which one is better, not shipping any updates at all (or after months 
> and months of delays), or shipping "looks good" updates quickly and 
> fixing them (if issues come up) even faster?
> 
> Aiming for perfection doesn't cut it.  Contrary to common beliefs, FL 
> doesn't have the resources for thorough testing that some vendors have 
> the luxury of.  That's why we employ those vendors' fixes directly :-)

It may come as no surprise (given my past threads on creating tools to make it 
easier to report testing results, I'm with Pekka on this one. 

Updates are getting delayed because nobody's bothering to test them. It's a 
hassle. We all have day jobs. Yatta yatta. We need to make it as easy as 
possible. 

If you want more testing on critical packages, I'd suggest giving a testing 
package a certain number of points needed before it's ready for release. 
Important packages (eg: kernel) might need more testing than lesser important 
ones. (EG: Mozilla) 

Let's say that we give different levels of validation, 1 or 2 points for 
"installed ok", 3-4 points for "ran several key commands in the package and 
it worked out, 10 points for "tested the vuln fixed and it worked out". 

A single negative report ("-1, it didn't work out") would kill the package for 
release. 

If a package gathers enough points for release, then it's released. Packages 
that are critical (EG: Kernel) would either get installed in a large number 
of machines without any negative reports, (meaning, it's probably OK) or get 
extensively tested on a smaller number of hosts by tough, dedicated admins. 

Either way, automating this reporting process would: 
	1) Make it easier to do testing, and 
	2) Provide more extensive testing of a kernel than Mozilla. 

Jesse, what do you think of this idea? 

> --
> fedora-legacy-list mailing list
> fedora-legacy-list at redhat.com
> https://www.redhat.com/mailman/listinfo/fedora-legacy-list
> 
> -- 
> This message has been scanned for viruses and
> dangerous content by MailScanner, and is
> believed to be clean.
> 

-- 
"The best way to predict the future is to invent it."
- XEROX PARC slogan, circa 1978