[libvirt PATCH 0/4] RFC: tests: introduce lavocado

[Daniel P. Berrangé]

On Thu, Jul 01, 2021 at 06:09:47PM -0300, Beraldo Leal wrote:
> On Thu, Jul 01, 2021 at 07:04:32PM +0100, Daniel P. Berrangé wrote:
> > On Wed, Jun 30, 2021 at 01:36:30PM -0300, Beraldo Leal wrote:
> > > I'm adding more information with some details inside the README file.
> > 
> > Overall, I'm more enthusiastic about writing tests in Python
> > than Perl, for the long term, but would also potentially like
> > to write tests in Go too.
> > 
> > I'm wondering if we can't bridge the divide between what we
> > have already in libvirt-tck, and what you're bringing to the
> > table with avocado here. While we've not done much development
> > with the TCK recently, there are some very valuable tests
> > there, especially related to firewall support and I don't
> > fancy rewriting them.
> > 
> > Thus my suggestion is that we:
> > 
> >   - Put this avocado code into the libvirt-tck repository,
> >     with focus on the supporting infra for making it easy to
> >     write Python tests
> > 
> >   - Declare that all tests need a way to emit TAP format,
> >     no matter what language they're written in. This could
> >     either be the test directly emitting TAP, or it could
> >     be via use of a plugin. For example 'tappy' can make
> >     existing Python tests emit TAP, with no modifications
> >     to the tests themselves.
> > 
> >       https://tappy.readthedocs.io/en/latest/consumers.html
> > 
> >     IOW, you can still invoke the python tests using the
> >     standard Python test runner, and still invoke the perl
> >     tests using the stnadard Perl test runner if desired.
> 
> This is supported already:
> 
> $ avocado run --tap - --test-runner='nrunner' tests/domain/transient.py
> 1..3
> ok 1 tests/domain/transient.py:TransientDomain.test_autostart
> ok 2 tests/domain/transient.py:TransientDomain.test_lifecycle
> ok 3 tests/domain/transient.py:TransientDomain.test_convert_transient_to_persistent

This is nice, showing fine grained TAP output lines for each
individual test within the test program


I tried using the hints file that Cleber pointed to make avocado
*consume* TAP format for the Perl/Shell scripts:

$ cd libvirt-tck
$ cat .avocado.hint
[kinds]
tap = scripts/*/*.t

[tap]
uri = $testpath


And I can indeed invoke the scripts:

$ avocado run  ./scripts/domain/05*.t
JOB ID     : b5d596d909dc8024d986957c909fc8fb6b31e2dd
JOB LOG    : /home/berrange/avocado/job-results/job-2021-07-21T18.45-b5d596d/job.log
 (1/2) ./scripts/domain/050-transient-lifecycle.t: PASS (0.70 s)
 (2/2) ./scripts/domain/051-transient-autostart.t: PASS (0.76 s)
RESULTS    : PASS 2 | ERROR 0 | FAIL 0 | SKIP 0 | WARN 0 | INTERRUPT 0 | CANCEL 0
JOB HTML   : /home/berrange/avocado/job-results/job-2021-07-21T18.45-b5d596d/results.html
JOB TIME   : 1.90 s

which is good.

And also I can ask it to produce tap output too:

$ avocado run --tap - ./scripts/domain/05*.t
1..2
ok 1 ./scripts/domain/050-transient-lifecycle.t
ok 2 ./scripts/domain/051-transient-autostart.t


But this output isn't entirely what I was after. This is just summarizing
the results of each test program.

I can't find a way to make it show the fine grained tap output for the
individual tests, like it does for the python program

eg I'd like to be able to see something similar to:

$ ./scripts/domain/050-transient-lifecycle.t
1..2
# Creating a new transient domain
ok 1 - created transient domain object
# Destroying the transient domain
# Checking that transient domain has gone away
ok 2 - NO_DOMAIN error raised from missing domain


$ ./scripts/domain/051-transient-autostart.t
1..4
# Creating a new transient domain
ok 1 - created transient domain object
ok 2 - autostart is disabled for transient VMs
ok 3 - Set autostart not supported on transient VMs
# Destroying the transient domain
# Checking that transient domain has gone away
ok 4 - NO_DOMAIN error raised from missing domain


None the less this does seem like we're pretty close to being able
to do something useful in integration

Regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|