[libvirt] [PATCH v2 3/7] tests: qemuxml2argv: Add infrastructure for testing with real qemuCaps

[Andrea Bolognani]

On Wed, 2018-04-18 at 17:22 +0200, Peter Krempa wrote:
> On Wed, Apr 18, 2018 at 14:43:55 +0200, Andrea Bolognani wrote:
> > I think using real-life capabilities instead of the syntetic data
> > we use now is a step in the right direction.
> > 
> > I'm still a bit uneasy about the "moving target" factor, though
> > I've spent some time trying to come up with possible failure
> > scenarios without much success...
> 
> There are few churn-scenarios, some avoidable, some not so much:
> 
> - When adding something which is added for a lot of configurations or all
>   of them (e.g. recent adding of the new seccomp code) the change will
>   generate a lot of changes.
> 
> - Machine type. We can't use any of the alias machine types in the XML
>   files as they will change when new capabilities with new machine types
>   are added. This can easily be avoided by using an explicit machine
>   type.
> 
> While the above may induce some churn in some scenarios, I think that
> the benefit of at least seeing that something changed in places where
> you did not expect any change may outweigh it.

I'm not really worried about churn, but about situations where
changes to libvirt would make it behave wrongly with old versions
of QEMU but correctly with newer ones, possibly with respect to
features that are not directly touched by the changes, in ways
unforeseen by the developer.

Right now that's easy to spot because everything is locked to some
well-defined set of capabilities, but if we started using moving
targets for most tests we'd decrease coverage of older QEMU in
favor of newer QEMU over time.

> > I think ideally with each new feature the author would introduce
> > three tests: a negative one locked to a QEMU version that doesn't
> > have the required bits, a positive one locked to a QEMU version
> > that has them, and a positive one against the latest QEMU version.
> 
> I'm considering whether it's necessary to have the one locked to the
> older-but-supporting version of qemu.
> 
> My idea was that if somebody were to add test which would change
> anything, the test can be forked prior to that. But it seems that it may
> be a better idea to lock-in the changes right away.

I believe so. This would also make it so our test coverage always
increases over time.


Now, a crazy idea: what if the test author would define a baseline
version QEMU where the test passes, and the test suite would
automatically test all versions between that one and the latest?

That would mean a lot of churn whenever we add new capabilities
data, but it should be feasible to detect output files that do not
change and turn them into symlinks to reduce diffs and save disk
space.

We would also need a way to run a test only in a range of versions,
for negative testing.

Too far fetched? :)

-- 
Andrea Bolognani / Red Hat / Virtualization