[libvirt PATCH 0/4] RFC: tests: introduce lavocado

Thu Jul 1 18:04:32 UTC 2021

On Wed, Jun 30, 2021 at 01:36:30PM -0300, Beraldo Leal wrote:
> lavocado aims to be an alternative test framework for the libvirt
> project using Python, python-libvirt and Avocado. This can be used to
> write unit, functional and integration tests and it is inspired by the
> libvirt-tck framework.
> 
> This series introduces the basic framework along with some basic test
> examples that I got from libvirt-tck. I would appreciate your comments
> on this RFC, to see if this fits this project's needs. Also, names and
> locations are just a proposal and can be changed.

Some high level thoughts

 - More extensive functional integration testing coverage is good

 - We need to actually run the functional tests regularly reporting
   via GitLab pipelines in some way

 - Using Python is way more contributor friendly than Perl

 - This does not need to live in libvirt.git as we don't follow
   a monolithic repo approach in libvirt, and it already depends
   on functionality provided by other repos.

When it comes to testing, I feel like there are several distinct
pieces to think about

  - The execution & reporting harness
  - Supporting infrastructure to aid writing tests
  - The set of tests themselves

If I look at the TCK

 - The harness is essentially the standard Perl harness
   with a thin CLI wrapper, thus essentially works with
   any test emitting TAP format
 - The support infra is all custom APIs using libvirt-perl
 - The tests are mostly written in Perl, but some are
   written in shell (yuk). They all output TAP format.

One key thing here is that the test harness is fairly loosely
coupled to the support infra & tests.

The TAP data format bridged the two, letting us write tests
in essentially any language. Of course writing tests in
non-Perl was/is tedious today, since there's no support
infra for that which exists today.

The TAP data format bridge also means we can easily throw
away the current TCK harness and replace it with anything
else that can consume tests emitting TAP data.

If I look at Avocado, I think (correct me if i'm wrong)

 1. The harness is essentially the standard Python harness
    with a thin CLI wrapper. Thus needs all tests to implement
    the Python test APIs
 2. The support infra is all custom APIs using libvirt-python
 3. The tests are to be written entirely in Python, to integrate
    with the python test harness

IIUC, 90% of this patch series is essentially about (2),
since (1) is fairly trivial and for (3) there's just
simple demo tests.

Libvirt has consumers writing applications in a variety of
languages, and periodically reporting bugs.  My general wish
for a test harness would be for something that can invoke
and consume arbitrary test cases. Essentially if someone
can provide us a piece of code that demonstrates a problem
in libvirt, I would like to be able to use that directly as
a functional test, regardless of language it was written
in.

In theory the libvirt TCK allowed for that, and indeed the
few tests written in shell essentially came to exist because
someone at IBM had written some shell code to test firewalls
and contributed that as a functional test. They just hacked
their script to emit TAP data.

> For now, this framework assumes that you are going to run the tests in a
> fresh clean environment, i.e. a VM. If you decide to use your local
> system, beware that execution of the tests may affect your system.

It is always good to be wary of functional test frameworks trashing
your system, but at the same time I think makes things more flexible
if they are able to be reasonably safely run on /any/ host. 

For the TCK we tried to be quite conservative in this regard, because
it was actually an explicit goal that you can run it on any Fedora
host to validate it correctly functioning for KVM. To achieve that
we tried to use some standard naming conventions for any resources
that tests created, and if we saw pre-existing resources named
differently we didn't touch them. ie all VMs were named 'tck-XXX',
and were free to be deleted, but other named VMs were ignored.

For anything special, such as testing PCI device assignment, the test
configuration file had to explicitly contain details of host devices
that were safe to use. This was also done for host block devices or
NICs, etc.  Thus a default invokation only ran a subset of tests
which were safe. The more dangerous tests required you to modify the
config file to grant access.

I think it would be good if the Avocado supporting test APIs had a
similar conceptual approach, especially wrt to a config file
granting access to host resources such as NICs/Block devs/PCI devs,
where you need prior explicit admin permission to avoid danger.

> One of the future goals of this framework is to utilize nested
> virtualization technologies and hence make sure an L1 guest is
> provisioned automatically for the tests to be executed in this
> environment and not tamper with your main system.

I think the test framework should not concern itself with this
kind of thing directly. We already have libvirt-ci, which has
tools for provisioning VMs with pre-defined package sets. The
test harness should just expect that this has been done, and
that it is already running in such an environment if the user
wanted it to be.

In the TCK config file we provided setting for a URI to connect
to other hosts, to enable multi-hosts tests like live migration
to be done.

> I'm adding more information with some details inside the README file.

Overall, I'm more enthusiastic about writing tests in Python
than Perl, for the long term, but would also potentially like
to write tests in Go too.

I'm wondering if we can't bridge the divide between what we
have already in libvirt-tck, and what you're bringing to the
table with avocado here. While we've not done much development
with the TCK recently, there are some very valuable tests
there, especially related to firewall support and I don't
fancy rewriting them.

Thus my suggestion is that we:

  - Put this avocado code into the libvirt-tck repository,
    with focus on the supporting infra for making it easy to
    write Python tests

  - Declare that all tests need a way to emit TAP format,
    no matter what language they're written in. This could
    either be the test directly emitting TAP, or it could
    be via use of a plugin. For example 'tappy' can make
    existing Python tests emit TAP, with no modifications
    to the tests themselves.

      https://tappy.readthedocs.io/en/latest/consumers.html

    IOW, you can still invoke the python tests using the
    standard Python test runner, and still invoke the perl
    tests using the stnadard Perl test runner if desired.

  - Switch the TCK configuration file to use JSON/YAML 
    instead of its current custom format. This is to enable
    the Python tests to share a config file with the Perl
    and Shell tests. Thus they can all have the same way
    to figure out what block devs, NICs, PCI devs, etc
    they are permitted to use, and what dirs it is safe
    to create VM images in, etc.

  - Replace the current Perl based execution harness
    with something using Python, so that it is less
    of a pain to install in terms of dependancies.
    Looks like we could just do something using the
    'tappy' Python APIs to consume the TAP format
    from all the tests and generate a junit report
    that GitLab can consume.

  - Rename the RPMs so they match the project name
    "libvirt-tck", instead of "perl-Sys-Virt-TCK"
    to emphasis that this is not a Perl project, it
    is a testing project, of which some parts were
    historically Perl, but new parts will be mostly
    Python (and possibly Go/etc in future if desired).
    Potentially have RPMs split:

       * libvirt-tck  - just the execution harness,
         currently the Perl one, but to be replaced
	 with a Python one.

       * libvirt-tck-tests-python - supporting APIs
         and tests written in Python - essentially
	 all of this series, and future tests written

       * libvirt-tck-tests-perl - supporting APIs
         and tests written in Perl - essentially
	 most of existing TCK stuff

       * libvirt-tck-tests-shell - supporting APIs
         and tests written in shell - mostly the
	 network/firewall related TCK pieces

Given the existance of the 'tappy' tool, it feels like the first
two points ought to be reasonably doable without any significant
changes to what you've already written. Just need to import it
as is, and then using tappy as a shim to let it invoked from
libvirt-tck. Can still have the existing way to invoke it
directly too as an alternative.

The common JSON/YAML config file, and adopting some standard
naming conventions and resource usage conventions are probably
where the real work would lie. That's likely not too bad at
this stage though, since you've not actually written a large
number of test cases.

> Beraldo Leal (4):
>   tests: introduce lavocado: initial code structure
>   tests.lavocado: adding basic transient domain tests
>   tests.lavocado: adding a .gitignore
>   tests.lavocado: adding a README and Makefile for convenience

Regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|