[libvirt PATCH 0/4] RFC: tests: introduce lavocado

[Beraldo Leal]

On Thu, Jul 01, 2021 at 07:04:32PM +0100, Daniel P. Berrangé wrote:
> On Wed, Jun 30, 2021 at 01:36:30PM -0300, Beraldo Leal wrote:
>
> If I look at Avocado, I think (correct me if i'm wrong)
> 
>  1. The harness is essentially the standard Python harness
>     with a thin CLI wrapper. Thus needs all tests to implement
>     the Python test APIs
>  2. The support infra is all custom APIs using libvirt-python
>  3. The tests are to be written entirely in Python, to integrate
>     with the python test harness

The harness proposed here would be the Avocado itself. That doesn't just
run Python tests, it can run any executable. And also has support for
multiple test results, including TAP.

However, for developers trying to write Python tests will receive some
support infrastructure classes that will help them.

> IIUC, 90% of this patch series is essentially about (2),
> since (1) is fairly trivial and for (3) there's just
> simple demo tests.

Yes, and (1) is trivial because I’m delegating this to Avocado.

> Libvirt has consumers writing applications in a variety of
> languages, and periodically reporting bugs.  My general wish
> for a test harness would be for something that can invoke
> and consume arbitrary test cases. Essentially if someone
> can provide us a piece of code that demonstrates a problem
> in libvirt, I would like to be able to use that directly as
> a functional test, regardless of language it was written
> in.
> 
> In theory the libvirt TCK allowed for that, and indeed the
> few tests written in shell essentially came to exist because
> someone at IBM had written some shell code to test firewalls
> and contributed that as a functional test. They just hacked
> their script to emit TAP data.

For sure, I completely agree and understand. We take that into account
as well, but maybe it wasn't so clear in the RFC. With Avocado you can
do that, and there is no need for hacking the scripts to emit TAP data.
Avocado has a --tap option that will output results in TAP format:

  $ avocado run --tap - /bin/true /bin/false
  1..2
  ok 1 /bin/true
  not ok 2 /bin/false

> > For now, this framework assumes that you are going to run the tests in a
> > fresh clean environment, i.e. a VM. If you decide to use your local
> > system, beware that execution of the tests may affect your system.
> 
> It is always good to be wary of functional test frameworks trashing
> your system, but at the same time I think makes things more flexible
> if they are able to be reasonably safely run on /any/ host. 
> 
> For the TCK we tried to be quite conservative in this regard, because
> it was actually an explicit goal that you can run it on any Fedora
> host to validate it correctly functioning for KVM. To achieve that
> we tried to use some standard naming conventions for any resources
> that tests created, and if we saw pre-existing resources named
> differently we didn't touch them. ie all VMs were named 'tck-XXX',
> and were free to be deleted, but other named VMs were ignored.
> 
> For anything special, such as testing PCI device assignment, the test
> configuration file had to explicitly contain details of host devices
> that were safe to use. This was also done for host block devices or
> NICs, etc.  Thus a default invokation only ran a subset of tests
> which were safe. The more dangerous tests required you to modify the
> config file to grant access.
> 
> I think it would be good if the Avocado supporting test APIs had a
> similar conceptual approach, especially wrt to a config file
> granting access to host resources such as NICs/Block devs/PCI devs,
> where you need prior explicit admin permission to avoid danger.

Indeed. It is a good practice to isolate those things. Again, it was my
fault that wasn't clear on the RFC, and yes we have that in mind. As
mentioned lavocado it was inspired by libvirt-tck and I'm already using
the same approach with the prefix: when creating domains I'm prefixing
the `test.id` to the domain name (and this can be customizable).

On top of that, Avocado has the spawners concept, where we support right
now subprocess and podman. But a new spawner could be developed to
create a more isolated env.

We also have the Job API feature, where, using python code, you can
create custom job suites based on your logic.

> > One of the future goals of this framework is to utilize nested
> > virtualization technologies and hence make sure an L1 guest is
> > provisioned automatically for the tests to be executed in this
> > environment and not tamper with your main system.
> 
> I think the test framework should not concern itself with this
> kind of thing directly. We already have libvirt-ci, which has
> tools for provisioning VMs with pre-defined package sets. The
> test harness should just expect that this has been done, and
> that it is already running in such an environment if the user
> wanted it to be.

Sure, again I wasn't clear. We intend to delegate this to tools like
lcitool.

> In the TCK config file we provided setting for a URI to connect
> to other hosts, to enable multi-hosts tests like live migration
> to be done.

I notice that, and I did the same here on LIBVIRT_URI config option.  By
default it is pointing to "qemu:///system". But for sure, could be
customized. It is on my plans to extend this to support > 1 URI to
support live migration.

> > I'm adding more information with some details inside the README file.
> 
> Overall, I'm more enthusiastic about writing tests in Python
> than Perl, for the long term, but would also potentially like
> to write tests in Go too.
> 
> I'm wondering if we can't bridge the divide between what we
> have already in libvirt-tck, and what you're bringing to the
> table with avocado here. While we've not done much development
> with the TCK recently, there are some very valuable tests
> there, especially related to firewall support and I don't
> fancy rewriting them.
> 
> Thus my suggestion is that we:
> 
>   - Put this avocado code into the libvirt-tck repository,
>     with focus on the supporting infra for making it easy to
>     write Python tests
> 
>   - Declare that all tests need a way to emit TAP format,
>     no matter what language they're written in. This could
>     either be the test directly emitting TAP, or it could
>     be via use of a plugin. For example 'tappy' can make
>     existing Python tests emit TAP, with no modifications
>     to the tests themselves.
> 
>       https://tappy.readthedocs.io/en/latest/consumers.html
> 
>     IOW, you can still invoke the python tests using the
>     standard Python test runner, and still invoke the perl
>     tests using the stnadard Perl test runner if desired.

This is supported already:

$ avocado run --tap - --test-runner='nrunner' tests/domain/transient.py
1..3
ok 1 tests/domain/transient.py:TransientDomain.test_autostart
ok 2 tests/domain/transient.py:TransientDomain.test_lifecycle
ok 3 tests/domain/transient.py:TransientDomain.test_convert_transient_to_persistent


Thank you for your comments and review here.

--
Beraldo