[libvirt PATCH 0/4] RFC: tests: introduce lavocado

[Daniel P. Berrangé]

On Wed, Jul 21, 2021 at 04:22:19PM -0300, Beraldo Leal wrote:
> On Wed, Jul 21, 2021 at 06:50:03PM +0100, Daniel P. Berrangé wrote:
> > On Thu, Jul 01, 2021 at 06:09:47PM -0300, Beraldo Leal wrote:
> > > On Thu, Jul 01, 2021 at 07:04:32PM +0100, Daniel P. Berrangé wrote:
> > > > On Wed, Jun 30, 2021 at 01:36:30PM -0300, Beraldo Leal wrote:
> > > > > I'm adding more information with some details inside the README file.
> > > > 
> > > > Overall, I'm more enthusiastic about writing tests in Python
> > > > than Perl, for the long term, but would also potentially like
> > > > to write tests in Go too.
> > > > 
> > > > I'm wondering if we can't bridge the divide between what we
> > > > have already in libvirt-tck, and what you're bringing to the
> > > > table with avocado here. While we've not done much development
> > > > with the TCK recently, there are some very valuable tests
> > > > there, especially related to firewall support and I don't
> > > > fancy rewriting them.
> > > > 
> > > > Thus my suggestion is that we:
> > > > 
> > > >   - Put this avocado code into the libvirt-tck repository,
> > > >     with focus on the supporting infra for making it easy to
> > > >     write Python tests
> > > > 
> > > >   - Declare that all tests need a way to emit TAP format,
> > > >     no matter what language they're written in. This could
> > > >     either be the test directly emitting TAP, or it could
> > > >     be via use of a plugin. For example 'tappy' can make
> > > >     existing Python tests emit TAP, with no modifications
> > > >     to the tests themselves.
> > > > 
> > > >       https://tappy.readthedocs.io/en/latest/consumers.html
> > > > 
> > > >     IOW, you can still invoke the python tests using the
> > > >     standard Python test runner, and still invoke the perl
> > > >     tests using the stnadard Perl test runner if desired.
> > > 
> > > This is supported already:
> > > 
> > > $ avocado run --tap - --test-runner='nrunner' tests/domain/transient.py
> > > 1..3
> > > ok 1 tests/domain/transient.py:TransientDomain.test_autostart
> > > ok 2 tests/domain/transient.py:TransientDomain.test_lifecycle
> > > ok 3 tests/domain/transient.py:TransientDomain.test_convert_transient_to_persistent
> > 
> > This is nice, showing fine grained TAP output lines for each
> > individual test within the test program
> > 
> > 
> > I tried using the hints file that Cleber pointed to make avocado
> > *consume* TAP format for the Perl/Shell scripts:
> > 
> > $ cd libvirt-tck
> > $ cat .avocado.hint
> > [kinds]
> > tap = scripts/*/*.t
> > 
> > [tap]
> > uri = $testpath
> > 
> > 
> > And I can indeed invoke the scripts:
> > 
> > $ avocado run  ./scripts/domain/05*.t
> > JOB ID     : b5d596d909dc8024d986957c909fc8fb6b31e2dd
> > JOB LOG    : /home/berrange/avocado/job-results/job-2021-07-21T18.45-b5d596d/job.log
> >  (1/2) ./scripts/domain/050-transient-lifecycle.t: PASS (0.70 s)
> >  (2/2) ./scripts/domain/051-transient-autostart.t: PASS (0.76 s)
> > RESULTS    : PASS 2 | ERROR 0 | FAIL 0 | SKIP 0 | WARN 0 | INTERRUPT 0 | CANCEL 0
> > JOB HTML   : /home/berrange/avocado/job-results/job-2021-07-21T18.45-b5d596d/results.html
> > JOB TIME   : 1.90 s
> > 
> > which is good.
> > 
> > And also I can ask it to produce tap output too:
> > 
> > $ avocado run --tap - ./scripts/domain/05*.t
> > 1..2
> > ok 1 ./scripts/domain/050-transient-lifecycle.t
> > ok 2 ./scripts/domain/051-transient-autostart.t
> > 
> > 
> > But this output isn't entirely what I was after. This is just summarizing
> > the results of each test program.
> > 
> > I can't find a way to make it show the fine grained tap output for the
> > individual tests, like it does for the python program
> 
> Actually, the first Python TAP output example is showing a coarse TAP
> result. I have combined three transient tests into one single file but
> splitting them into three different methods, so we could execute each one
> individually. i.e: tests/domain/transient.py:TransientDomain.test_autostart.
> 
> So, I did some reorg when migrating to Python test.
> 
> In order to archive the same with Perl, we could do the same there,
> because the way individual tests are written there, doesn't allow for
> individual execution. 
> 
> Yes, we could do some tricks, to parse and combine outputs and list as
> it was a more fine graned, but afaict, we could not individually execute
> those. This is part of Avocado test definition where in order to be
> called a test, we need to be able to execute those individually as well.

Ok, I'm not so fussed about whether avocado can ultimately preserve the
fine grained TAP output. Mostly I'm looking to understand how you should
debug failures when they go wrong, becuase the default output I see from
avocado is very terse giving no indiciation of the the failure - merely
that there has been a failure.

In an interactive environment you can just re-rnu the individual failed
test directly. In an automated CI environment you need the test harness
to display enough info to debug the failure directly. Ideally it should
dump the full stdout+stderr either to the console or a log file that
can be publish as an artifact from CI.  Meson tends todo the latter
creating a log file will full stdout/err from all tests.

Regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|