[libvirt PATCH 0/4] RFC: tests: introduce lavocado

[Cleber Rosa]

Hi Daniel,

On Thu, Jul 1, 2021 at 2:05 PM Daniel P. Berrangé <berrange at redhat.com> wrote:
>
> On Wed, Jun 30, 2021 at 01:36:30PM -0300, Beraldo Leal wrote:
> > lavocado aims to be an alternative test framework for the libvirt
> > project using Python, python-libvirt and Avocado. This can be used to
> > write unit, functional and integration tests and it is inspired by the
> > libvirt-tck framework.
> >
> > This series introduces the basic framework along with some basic test
> > examples that I got from libvirt-tck. I would appreciate your comments
> > on this RFC, to see if this fits this project's needs. Also, names and
> > locations are just a proposal and can be changed.
>
> Some high level thoughts
>
>  - More extensive functional integration testing coverage is good
>
>  - We need to actually run the functional tests regularly reporting
>    via GitLab pipelines in some way
>
>  - Using Python is way more contributor friendly than Perl
>
>  - This does not need to live in libvirt.git as we don't follow
>    a monolithic repo approach in libvirt, and it already depends
>    on functionality provided by other repos.
>
>
> When it comes to testing, I feel like there are several distinct
> pieces to think about
>
>   - The execution & reporting harness
>   - Supporting infrastructure to aid writing tests
>   - The set of tests themselves
>
> If I look at the TCK
>
>  - The harness is essentially the standard Perl harness
>    with a thin CLI wrapper, thus essentially works with
>    any test emitting TAP format
>  - The support infra is all custom APIs using libvirt-perl
>  - The tests are mostly written in Perl, but some are
>    written in shell (yuk). They all output TAP format.
>
> One key thing here is that the test harness is fairly loosely
> coupled to the support infra & tests.
>
> The TAP data format bridged the two, letting us write tests
> in essentially any language. Of course writing tests in
> non-Perl was/is tedious today, since there's no support
> infra for that which exists today.
>
> The TAP data format bridge also means we can easily throw
> away the current TCK harness and replace it with anything
> else that can consume tests emitting TAP data.
>
>
> If I look at Avocado, I think (correct me if i'm wrong)
>
>  1. The harness is essentially the standard Python harness
>     with a thin CLI wrapper. Thus needs all tests to implement
>     the Python test APIs

Not really.  Even though Avocado is mostly written in Python, there
have been provisions for accommodating foreign types of tests (in
different forms) since its inception.  The most basic way, is, of
course, simply treating a test as an executable.  But this is far from
the only way.  For instance, these are other possibilities:

a) if an executable generates TAP, it can consume the test's TAP
output and determine the test results
b) you can write support for completely new test types[1].  If you
want your test to be "discoverable" when running `avocado list ...`
you will have to write some Python code.  But you can skip that part,
and have an executable named "avocado-runner-$test-type" written in
any language whatsoever, which will be used to execute whatever your
tests are and feed the information to Avocado.  BTW, we already have a
plugin that can list go tests[2] and run them (via a Python wrapper
that uses `go test`).

>  2. The support infra is all custom APIs using libvirt-python
>  3. The tests are to be written entirely in Python, to integrate
>     with the python test harness
>
> IIUC, 90% of this patch series is essentially about (2),
> since (1) is fairly trivial and for (3) there's just
> simple demo tests.
>
>
> Libvirt has consumers writing applications in a variety of
> languages, and periodically reporting bugs.  My general wish
> for a test harness would be for something that can invoke
> and consume arbitrary test cases. Essentially if someone
> can provide us a piece of code that demonstrates a problem
> in libvirt, I would like to be able to use that directly as
> a functional test, regardless of language it was written
> in.
>
> In theory the libvirt TCK allowed for that, and indeed the
> few tests written in shell essentially came to exist because
> someone at IBM had written some shell code to test firewalls
> and contributed that as a functional test. They just hacked
> their script to emit TAP data.
>
>

I agree there's no reason to *not* continue to allow that.  I'd even
argue that Avocado can give libvirt even more options wrt integrating
different types of tests.

>
> > For now, this framework assumes that you are going to run the tests in a
> > fresh clean environment, i.e. a VM. If you decide to use your local
> > system, beware that execution of the tests may affect your system.
>
> It is always good to be wary of functional test frameworks trashing
> your system, but at the same time I think makes things more flexible
> if they are able to be reasonably safely run on /any/ host.
>

I'm on the same boat here.  All the QEMU/Libvirt testing
infrastructure that I was involved with, evolved from something that
needed to run as privileged users and could easily trash your system,
from something that is runnable as a regular user after a `pip install
--user`.  There are challenges, but most situations are handled with
tests being canceled if they do not have what they need to run, and by
always assuming a "good neighbor" policy (that is, you're not alone in
the system).

Having said that, I've seen how much care has been dedicated to
libvirt-ci so that test execution environments would be predictable,
reliable and easy to obtain.  I guess the original text from Beraldo
tried to make it clear that we envision an obvious/default integration
with those environments created by libvirt-ci, while not restricting
other environments or assuming every execution will be on those
environments.

> For the TCK we tried to be quite conservative in this regard, because
> it was actually an explicit goal that you can run it on any Fedora
> host to validate it correctly functioning for KVM. To achieve that
> we tried to use some standard naming conventions for any resources
> that tests created, and if we saw pre-existing resources named
> differently we didn't touch them. ie all VMs were named 'tck-XXX',
> and were free to be deleted, but other named VMs were ignored.
>
> For anything special, such as testing PCI device assignment, the test
> configuration file had to explicitly contain details of host devices
> that were safe to use. This was also done for host block devices or
> NICs, etc.  Thus a default invokation only ran a subset of tests
> which were safe. The more dangerous tests required you to modify the
> config file to grant access.
>
> I think it would be good if the Avocado supporting test APIs had a
> similar conceptual approach, especially wrt to a config file
> granting access to host resources such as NICs/Block devs/PCI devs,
> where you need prior explicit admin permission to avoid danger.
>

Absolutely agreed.

> > One of the future goals of this framework is to utilize nested
> > virtualization technologies and hence make sure an L1 guest is
> > provisioned automatically for the tests to be executed in this
> > environment and not tamper with your main system.
>
> I think the test framework should not concern itself with this
> kind of thing directly. We already have libvirt-ci, which has
> tools for provisioning VMs with pre-defined package sets. The
> test harness should just expect that this has been done, and
> that it is already running in such an environment if the user
> wanted it to be.
>

Right.  I know for a fact the original text was not about depending
exclusively on those VM environments, but having an obvious default
experience based on them.

> In the TCK config file we provided setting for a URI to connect
> to other hosts, to enable multi-hosts tests like live migration
> to be done.
>
> > I'm adding more information with some details inside the README file.
>
> Overall, I'm more enthusiastic about writing tests in Python
> than Perl, for the long term, but would also potentially like
> to write tests in Go too.
>
> I'm wondering if we can't bridge the divide between what we
> have already in libvirt-tck, and what you're bringing to the
> table with avocado here. While we've not done much development
> with the TCK recently, there are some very valuable tests
> there, especially related to firewall support and I don't
> fancy rewriting them.
>
> Thus my suggestion is that we:
>
>   - Put this avocado code into the libvirt-tck repository,
>     with focus on the supporting infra for making it easy to
>     write Python tests
>

One drawback would be the various supporting files for a perl+Python
package.  I mean, a Makefile.PL and a setup.py in the same repo can be
a source of confusion.  But, maybe I'm overreacting.

>   - Declare that all tests need a way to emit TAP format,
>     no matter what language they're written in. This could
>     either be the test directly emitting TAP, or it could
>     be via use of a plugin. For example 'tappy' can make
>     existing Python tests emit TAP, with no modifications
>     to the tests themselves.
>
>       https://tappy.readthedocs.io/en/latest/consumers.html
>

I honestly think Avocado can provide a better range of integration
opportunities.

>     IOW, you can still invoke the python tests using the
>     standard Python test runner, and still invoke the perl
>     tests using the stnadard Perl test runner if desired.
>
>   - Switch the TCK configuration file to use JSON/YAML
>     instead of its current custom format. This is to enable
>     the Python tests to share a config file with the Perl
>     and Shell tests. Thus they can all have the same way
>     to figure out what block devs, NICs, PCI devs, etc
>     they are permitted to use, and what dirs it is safe
>     to create VM images in, etc.
>

Makes sense.

>   - Replace the current Perl based execution harness
>     with something using Python, so that it is less
>     of a pain to install in terms of dependancies.
>     Looks like we could just do something using the
>     'tappy' Python APIs to consume the TAP format
>     from all the tests and generate a junit report
>     that GitLab can consume.
>

These are all things Avocado can do right away.

>   - Rename the RPMs so they match the project name
>     "libvirt-tck", instead of "perl-Sys-Virt-TCK"
>     to emphasis that this is not a Perl project, it
>     is a testing project, of which some parts were
>     historically Perl, but new parts will be mostly
>     Python (and possibly Go/etc in future if desired).
>     Potentially have RPMs split:
>
>        * libvirt-tck  - just the execution harness,
>          currently the Perl one, but to be replaced
>          with a Python one.
>
>        * libvirt-tck-tests-python - supporting APIs
>          and tests written in Python - essentially
>          all of this series, and future tests written
>
>        * libvirt-tck-tests-perl - supporting APIs
>          and tests written in Perl - essentially
>          most of existing TCK stuff
>
>        * libvirt-tck-tests-shell - supporting APIs
>          and tests written in shell - mostly the
>          network/firewall related TCK pieces
>
>
> Given the existance of the 'tappy' tool, it feels like the first
> two points ought to be reasonably doable without any significant
> changes to what you've already written. Just need to import it
> as is, and then using tappy as a shim to let it invoked from
> libvirt-tck. Can still have the existing way to invoke it
> directly too as an alternative.
>
> The common JSON/YAML config file, and adopting some standard
> naming conventions and resource usage conventions are probably
> where the real work would lie. That's likely not too bad at
> this stage though, since you've not actually written a large
> number of test cases.
>
> > Beraldo Leal (4):
> >   tests: introduce lavocado: initial code structure
> >   tests.lavocado: adding basic transient domain tests
> >   tests.lavocado: adding a .gitignore
> >   tests.lavocado: adding a README and Makefile for convenience
>
> Regards,
> Daniel
> --
> |: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
> |: https://libvirt.org         -o-            https://fstop138.berrange.com :|
> |: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|
>

Cheers,
- Cleber.

[1] - https://avocado-framework.readthedocs.io/en/89.0/guides/contributor/chapters/plugins.html#new-test-type-plugin-example
[2] - https://avocado-framework.readthedocs.io/en/89.0/plugins/optional/golang.html