Regression testing

[Mike A. Harris]

Chris Adams wrote:
> Once upon a time, Gregory Maxwell <gmaxwell at gmail.com> said:
> 
>>One way to accomplish this would be to have the spec optionally build
>>a packagename-regression rpm, which includes the scripts and data
>>needed to perform the automated regression testing.
> 
> 
> Some regression tests are designed to be run in the build tree, so
> building a package is probably not the best way to do this (the OpenSSH
> tests are this way for example).
> 
> A better idea would be for the %build or %install stage (as appropriate)
> to optionally run such tests, so if they fail, the RPM doesn't build.

The openssh tests also consume 99.9% of all CPUs even in 8way boxes, and
do this for like 16 hours straight.  This has caused massive buildsystem
contention in the past when we did have it enabled.

I agree that enabling post build regression tests is in general a good
idea, but before anyone enables tests in a given package, they should
determine how much buildsystem load the tests may cause, and for what
length of time, and also be mindful that the buildsystem is a shared
resource used by numerous people.

For tests that pound the buildsystem(s), they should be disabled by
default IMHO, and only enabled every now and then when the buildsystem
is very idle, and never at peak times of building.

Just some thoughts..