[Libguestfs] Regression testing inspection code

[Richard W.M. Jones]

On Mon, Dec 02, 2013 at 01:33:53PM +0000, Matthew Booth wrote:
> On Mon, 2013-12-02 at 13:23 +0000, Richard W.M. Jones wrote:
> > [Continuing a discussion we had on IRC last week ...]
> > 
> > How to create meaningful regression tests for the inspection code?
> > We'd like developers to be able to test that changes to the inspection
> > code don't break inspection of existing guests.  We might not be able
> > to run these tests routinely, and we might have to accept that they
> > have to download hundreds of megabytes of test data.
> > 
> > 
> > Idea #1: Run virt-inspector on cloud images
> > ----------------------------------------------------------------------
> > 
> > eg:
> > 
> > virt-inspector -a http://download.fedoraproject.org/pub/fedora/linux/releases/19/Images/x86_64/Fedora-x86_64-19-20130627-sda.qcow2
> > 
> > Unfortunately this doesn't work for a couple of reasons.  Firstly the
> > Curl driver in qemu is broken (RHBZ#971790).  Secondly even when I
> > implemented a workaround, the command above doesn't work reliably when
> > connecting to a slow mirror, because of SCSI timeouts inside the
> > appliance.
> > 
> > 
> > Idea #2: Make "reduced" test images available
> > ----------------------------------------------------------------------
> > 
> > Delete bits of real test images so they are small enough to
> > distribute.  Inspection doesn't look at the vast majority of files in
> > a test image.  The hard bit is working out which bits inspection
> > *does* need to look at, either now or in the future.
> > 
> > I wrote a small program to test this.  Unfortunately it doesn't reduce
> > images that much.  We'd still be talking about ~50-100MB to host +
> > distribute per test case, and we'd ideally want >20 test cases so
> > that's still multiple gigabytes of test data.
> > 
> > I've attached the program to this email in case anyone wants to look
> > at this further.
> > 
> > 
> > Any other ideas ...?
> 
> guestconv builds several RHEL and Fedora based images for automated
> testing. The build process is separate from the test process, and tests
> are skipped if their image hasn't been built. It's very flexible, and
> works for guests we can build automatically.

Can you link to the code?  Does it use Oz?

Note the really interesting regression test cases (for inspection, not
for v2v) are obscure Linux distros, *BSD, etc.  These are the ones
which are likely to break without anyone noticing.

Rich.

-- 
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
virt-df lists disk usage of guests without needing to install any
software inside the virtual machine.  Supports Linux and Windows.
http://people.redhat.com/~rjones/virt-df/