[libvirt] [test-API] RFC: Stabilization of libvirt-test-API

[Laine Stump]

On 03/29/2012 08:14 AM, Martin Kletzander wrote:
> So here are the things I would like to do definitely (the optional
> things follow later on):
>  - fix hard-coded options into real options (e.g. commit 65449e)

Forgive me if I'm suggesting things that already exist - I haven't had
time to go through all of the test-API code, but want to make sure this
gets brought up sooner rather than later.

Two things that I think are very important for a general purpose test
setup are:

1) The ability to have multiple physical machines, network devices of
various types (bridges, standard NICs, sr-iov NICs) in the test harness
available via standard names (implying, of course, that the
address/capabilities (and even the existence/non-existence) of at least
one other machine be configurable in a global config file referenced by
every test).

2) The ability to mark some tests as requiring certain standard objects
from the global configuration (e.g. the second machine, a bridge
interface, sr-iov NICs) and to either skip the test when some required
object is missing, or fail the test (the test itself would be marked
either OPTIONAL or REQUIRED).

This would allow us to have, for example, full testing of networking
capabilities present in the test suite, but without raising an entry
barrier for "random Joe" who only has a single machine, one NIC, etc.,
but wants to run the test suite. When Joe ran the tests, each test that
required multiple hosts (or sr-iov NICs or whatever esoteric piece of
hardware/driver) and was designated as an "OPTIONAL" test, would be
semi-silently skipped, but someone with a full complement of hardware
who demanded a thorough and complete test could set a switch and
guarantee that every single test would be run (or a failure saying, e.g.
"object REMOTE_HOST required by this test is missing in config", or
something like that).

Is there currently an allowance for these two items?