[Libguestfs] [PATCH v5 0/7] tests: Introduce test harness for running tests.

[Pino Toscano]

Hi,

On Sunday 05 October 2014 14:08:34 Richard W.M. Jones wrote:
> Since v4:
> 
>  - More tests have been converted.
>  - Testing local guests fixed.
>  - Drop no-exec-stack test.

I've sent few particular notes already to the separate patches.

As a general notes: I look forward too to something to run the tests as 
installed, and we discussed (mostly on IRC) about it a couple of times.

What I don't like particularly is the generation of the test list at 
generator time, and some tests at configure time; currently it is easy 
to just drop a new test in some directory and add it to TESTS.
On the other hand, I understand the need to avoid duplication between 
in-tree and uninstalled tests.

A possible idea just came into my mind is installing the current tests, 
like:
  installedtestsdir = $(libdir)/guestfs/tests/somesubdir
  installedtests_SCRIPTS = $(TESTS)
at the bottom of each Makefile.am providing tests, maybe automating that 
using a small make function; this could make sure all the tests in each 
directory get installed properly, without moving the logic away from the 
build system.
Regarding the "how to find tests": maybe it is a waste, but an idea 
could be add, next to test.sh, a test.sh.testinfo (or whatever), in 
plain text format, with entries like:
  [Test]
  Need=appliance,fake-guests
  CompatSkip=SKIP_OLD_NAME
(we have a .ini parser in virt-builder, so that comes for free.)
For example, the runner could get a parameter to know whether the 
appliance has been enabled, and not run tests with "appliance" in Need 
key. The bigger drawback is that even the tests with nothing to specify 
in such .testinfo file would need one, even if just with an empty [Test] 
section.

-- 
Pino Toscano