[libvirt PATCH 0/4] RFC: tests: introduce lavocado

[Beraldo Leal]

On Wed, Jul 21, 2021 at 06:50:03PM +0100, Daniel P. Berrangé wrote:
> On Thu, Jul 01, 2021 at 06:09:47PM -0300, Beraldo Leal wrote:
> > On Thu, Jul 01, 2021 at 07:04:32PM +0100, Daniel P. Berrangé wrote:
> > > On Wed, Jun 30, 2021 at 01:36:30PM -0300, Beraldo Leal wrote:
> > > > I'm adding more information with some details inside the README file.
> > > 
> > > Overall, I'm more enthusiastic about writing tests in Python
> > > than Perl, for the long term, but would also potentially like
> > > to write tests in Go too.
> > > 
> > > I'm wondering if we can't bridge the divide between what we
> > > have already in libvirt-tck, and what you're bringing to the
> > > table with avocado here. While we've not done much development
> > > with the TCK recently, there are some very valuable tests
> > > there, especially related to firewall support and I don't
> > > fancy rewriting them.
> > > 
> > > Thus my suggestion is that we:
> > > 
> > >   - Put this avocado code into the libvirt-tck repository,
> > >     with focus on the supporting infra for making it easy to
> > >     write Python tests
> > > 
> > >   - Declare that all tests need a way to emit TAP format,
> > >     no matter what language they're written in. This could
> > >     either be the test directly emitting TAP, or it could
> > >     be via use of a plugin. For example 'tappy' can make
> > >     existing Python tests emit TAP, with no modifications
> > >     to the tests themselves.
> > > 
> > >       https://tappy.readthedocs.io/en/latest/consumers.html
> > > 
> > >     IOW, you can still invoke the python tests using the
> > >     standard Python test runner, and still invoke the perl
> > >     tests using the stnadard Perl test runner if desired.
> > 
> > This is supported already:
> > 
> > $ avocado run --tap - --test-runner='nrunner' tests/domain/transient.py
> > 1..3
> > ok 1 tests/domain/transient.py:TransientDomain.test_autostart
> > ok 2 tests/domain/transient.py:TransientDomain.test_lifecycle
> > ok 3 tests/domain/transient.py:TransientDomain.test_convert_transient_to_persistent
> 
> This is nice, showing fine grained TAP output lines for each
> individual test within the test program
> 
> 
> I tried using the hints file that Cleber pointed to make avocado
> *consume* TAP format for the Perl/Shell scripts:
> 
> $ cd libvirt-tck
> $ cat .avocado.hint
> [kinds]
> tap = scripts/*/*.t
> 
> [tap]
> uri = $testpath
> 
> 
> And I can indeed invoke the scripts:
> 
> $ avocado run  ./scripts/domain/05*.t
> JOB ID     : b5d596d909dc8024d986957c909fc8fb6b31e2dd
> JOB LOG    : /home/berrange/avocado/job-results/job-2021-07-21T18.45-b5d596d/job.log
>  (1/2) ./scripts/domain/050-transient-lifecycle.t: PASS (0.70 s)
>  (2/2) ./scripts/domain/051-transient-autostart.t: PASS (0.76 s)
> RESULTS    : PASS 2 | ERROR 0 | FAIL 0 | SKIP 0 | WARN 0 | INTERRUPT 0 | CANCEL 0
> JOB HTML   : /home/berrange/avocado/job-results/job-2021-07-21T18.45-b5d596d/results.html
> JOB TIME   : 1.90 s
> 
> which is good.
> 
> And also I can ask it to produce tap output too:
> 
> $ avocado run --tap - ./scripts/domain/05*.t
> 1..2
> ok 1 ./scripts/domain/050-transient-lifecycle.t
> ok 2 ./scripts/domain/051-transient-autostart.t
> 
> 
> But this output isn't entirely what I was after. This is just summarizing
> the results of each test program.
> 
> I can't find a way to make it show the fine grained tap output for the
> individual tests, like it does for the python program

Actually, the first Python TAP output example is showing a coarse TAP
result. I have combined three transient tests into one single file but
splitting them into three different methods, so we could execute each one
individually. i.e: tests/domain/transient.py:TransientDomain.test_autostart.

So, I did some reorg when migrating to Python test.

In order to archive the same with Perl, we could do the same there,
because the way individual tests are written there, doesn't allow for
individual execution. 

Yes, we could do some tricks, to parse and combine outputs and list as
it was a more fine graned, but afaict, we could not individually execute
those. This is part of Avocado test definition where in order to be
called a test, we need to be able to execute those individually as well.

--
Beraldo