[Libguestfs] [PATCH 0/3] tests: Introduce test harness for running tests.

[Richard W.M. Jones]

These are my thoughts on adding a test harness to run tests instead of
using automake.  The aim of this exercise is to allow us to run the
full test suite on an installed copy of libguestfs.  Another aim is to
allow us to work around all the limitations and problems of automake.

The first patch makes an observation that since the ./run script sets
up $PATH to contain all the directories containing binaries, we can
change all the test scripts to use (eg) 'guestfish' instead of
'../fish/guestfish'.  This allows the test scripts to be reused to
test the installed binary, simply by not invoking the ./run script.
Note this change is completely compatible with the current test
system.

The second patch is just preparation.

The third patch adds (or rather, generates) a shell script in the top
level directory called 'test-harness'.  This script is what will run
the tests (replacing automake).

The test-harness script can be used both to run the tests on the built
tree, using:

  ./run test-harness [subdirectory]

or on an installed tree.  The plan would be to install all the test
scripts and data files required into @libdir@/guestfs/tests, and then
you could run the test suite by doing:

  cd /usr/lib/guestfs/tests
  ./test-harness
  ./test-harness --fast
  ./test-harness --slow --valgrind

The test-harness script is wired into the existing Makefile.am's by
having the generator also generate 'tests.mk' fragments which are
included in the existing Makefile.am's.  They contain rules for
'check', 'check-valgrind' etc which were previously written by hand.
The aim here is to have commands like:

  make check
  make -C inspector check-valgrind

just work as they do currently.

Some unfinished business in the current patch set:

 - Some tests need data files (tests/data and tests/guests) but these
   are probably too large to ship around with the test suite, and have
   to be generated because we don't want the test suite to require the
   appliance.  Exactly how this will work is not fully decided.

 - Only the inspector/ subdirectory has been converted.  All the other
   tests and directories would have to be converted.  Although this is
   quite a lot of compatibility, it's still going to be a long and
   painful process.

Rich.