Regression testing

[Gregory Maxwell]

On Sat, 19 Mar 2005 20:14:55 -0600, Chris Adams <cmadams at hiwaay.net> wrote:
> How else do you run regression tests on an FTP client (for example)?

By including a FTP server, of course. ;)

But more seriously, .. you don't.. Or rather you don't put such tests
in the same class of tests with tens of thousands of other tests which
can happily be run with in a more limited environment.

The idea being is that if you have a thousand packages with a thousand
different testing requirements none of them will be tested at all,
because everyone will disable the tests because they fail in on
completely sane systems and the effort of sorting out which is a real
failure and which is something silly is too much, especially for a
package you're not familiar with.

If instead you split tests into two classes, an extensive class with
arbratarly tall requirements, and a basic class which should never
fail you will likely get a lot more useful testing... and those that
really care about a packages extensive tests will still undertake the
additional effort to execute them.
 
> The perl package that does network connections during "make test" (I've
> only seen one IIRC) does it because the modules are network client
> modules; the only real way to test them is to connect to network
> servers.

Well if anyone really cared they could override the socket calls with
a testing shim... a lot of work for not much gain.... but it's exactly
the sort of thing that should eventually be done for   important
infrastructure components that don't change often (libc stuff for
example).

Waiting for real users to encounter bugs in production is not really
acceptable when it can be avoided via testing... But with the number
of packages in a distro, and the complexity of the overall system...
the myriad of potential interactions, we miss a lot of potential
issues when regression testing is limited to the core developer
running on his single system before issuing a new release.

> Since OpenSSH includes both a server and a client, it can test both
> against each other.

And it would be fantastic for it to do that...