[Libguestfs] Fixing the tests (was: Re: [PATCH] build: Enable parallel tests.)

[Richard W.M. Jones]

I had a think about what to do about tests.  I didn't come to any
conclusions :-( but here are some notes anyway.

The problem
-----------

At the moment you can run the tests by doing 'make check'.  This uses
the automake test framework which sucks greatly (like the rest of
automake) but is slightly better than the alternatives (like the rest
of automake).

What we want to do is to make the guests installable.  You would
install a package 'libguestfs-tests', run some script from that
package, and it would run the tests on the installed copy of
libguestfs.

This lets you separate building and testing into two separate tasks,
which is useful for Koji -- the current build + test can take hours.
It would let us do testing in more places, for example, building the
package in Koji and then testing it on baremetal -- currently not
possible with Koji.  We would also like to run the tests after other
packages have changed, especially qemu & the kernel -- tieing the
tests into the build makes this difficult.

Other issues
------------

We still want 'make check' to work, and it should always be possible
to run the tests as non-root even when installed.

Tests have dependencies between them.

It should be possible to run the tests in parallel, where dependencies
permit.

There should be a flexible way to skip tests which doesn't require
every test to parse a $SKIP_* environment variable, which is what
happens now.

The current system keeps many tests in the same directory as the
program being tested, eg. fish/ contains guestfish tests.  That's a
good thing, I think.  At the same time we have a tests/ directory that
has non-specific tests, and I think we should keep that too.

Language binding tests will probably have to stay as they are now,
since the tests use language-specific test harnesses
(eg. rubygem-minitest).

It shouldn't require installing any other software.

It would be nice if tests could generate temporary files without
needing to give them unique names.

Background reading
------------------

I'm quite intrigued by Colin's "InstalledTests" plan for GNOME:
https://wiki.gnome.org/Initiatives/GnomeGoals/InstalledTests
There are also some relevant links at the bottom of that page.

He is certainly trying to solve some of the same problems.

Automake installable tests
--------------------------

I think the make the tests installable, we are surely going to need to
write automake rules like:

  testsdir = $(libdir)/guestfs/tests/fish
  fish_testsdir = $(testsdir)/fish
  fish_tests_SCRIPTS = test-a.sh test-add-domain.sh #etc
  fish_tests_DATA = # any data files required for the test

This creates a directory of test scripts, but it doesn't solve how to
run them or how to express dependencies between them.

We could drop a Makefile into the test directory.  How to generate that?

We could have some sort of script as a test harness.

How do we ensure 'make check' still works?  Obvious plan is to have a
check-local rule that does:

  check-local:
    make install testsdir=$(top_builddir)/tmp/check/
    cd $(top_builddir)/tmp/check && run-the-tests

That gets ugly if you want to allow the user to run a single
subdirectory of tests, or to run single tests (something I often want
to do).

It seems like we need to have some kind of program to manage tests.
gnome-desktop-testing-runner is a contender here
(https://git.gnome.org/browse/gnome-desktop-testing/tree/src/gnome-desktop-testing-runner.c)

Rich.

-- 
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
libguestfs lets you edit virtual machines.  Supports shell scripting,
bindings from many languages.  http://libguestfs.org