[libvirt PATCH 0/4] RFC: tests: introduce lavocado

Thu Jul 22 08:40:46 UTC 2021

On Wed, Jul 21, 2021 at 06:01:34PM -0400, Cleber Rosa wrote:
> 
> Daniel P. Berrangé writes:
> 
> > On Thu, Jul 01, 2021 at 06:09:47PM -0300, Beraldo Leal wrote:
> >> 
> >> This is supported already:
> >> 
> >> $ avocado run --tap - --test-runner='nrunner' tests/domain/transient.py
> >> 1..3
> >> ok 1 tests/domain/transient.py:TransientDomain.test_autostart
> >> ok 2 tests/domain/transient.py:TransientDomain.test_lifecycle
> >> ok 3 tests/domain/transient.py:TransientDomain.test_convert_transient_to_persistent
> >
> > This is nice, showing fine grained TAP output lines for each
> > individual test within the test program
> >
> >
> > I tried using the hints file that Cleber pointed to make avocado
> > *consume* TAP format for the Perl/Shell scripts:
> >
> > $ cd libvirt-tck
> > $ cat .avocado.hint
> > [kinds]
> > tap = scripts/*/*.t
> >
> > [tap]
> > uri = $testpath
> >
> >
> > And I can indeed invoke the scripts:
> >
> > $ avocado run  ./scripts/domain/05*.t
> > JOB ID     : b5d596d909dc8024d986957c909fc8fb6b31e2dd
> > JOB LOG    : /home/berrange/avocado/job-results/job-2021-07-21T18.45-b5d596d/job.log
> >  (1/2) ./scripts/domain/050-transient-lifecycle.t: PASS (0.70 s)
> >  (2/2) ./scripts/domain/051-transient-autostart.t: PASS (0.76 s)
> > RESULTS    : PASS 2 | ERROR 0 | FAIL 0 | SKIP 0 | WARN 0 | INTERRUPT 0 | CANCEL 0
> > JOB HTML   : /home/berrange/avocado/job-results/job-2021-07-21T18.45-b5d596d/results.html
> > JOB TIME   : 1.90 s
> >
> > which is good.
> >
> > And also I can ask it to produce tap output too:
> >
> > $ avocado run --tap - ./scripts/domain/05*.t
> > 1..2
> > ok 1 ./scripts/domain/050-transient-lifecycle.t
> > ok 2 ./scripts/domain/051-transient-autostart.t
> >
> >
> > But this output isn't entirely what I was after. This is just summarizing
> > the results of each test program.
> >
> 
> Hi Daniel,
> 
> From the user experience perspective, I completely understand your
> point.  I would side with this expectation and output any day if acting
> purely as a user.
> 
> But, as a framework, Avocado came up with some well defined concepts,
> and one of them is what a test is.  To be able to "replay only the
> failed tests of the latest job" or to "run a test completely isolated in
> a container or VM", there needs to be an specific and indistinguishable
> entry point for anything that is considered a test.

Ok, I see what you mean.

Regardless of what its written in, it is common that a test will
print arbitrary stuff to stdout or stderr if things go very badly
wrong, possibly including stack traces, etc.

What is the expected approach for debugging failed tests such that
you can see the full stdout/stderr ?  Are you supposed to just
directly invoke the individual test program without using the
avocado harness to see full output ?

> > I can't find a way to make it show the fine grained tap output for the
> > individual tests, like it does for the python program
> >
> > eg I'd like to be able to see something similar to:
> >
> > $ ./scripts/domain/050-transient-lifecycle.t
> > 1..2
> > # Creating a new transient domain
> > ok 1 - created transient domain object
> > # Destroying the transient domain
> > # Checking that transient domain has gone away
> > ok 2 - NO_DOMAIN error raised from missing domain
> >
> >
> > $ ./scripts/domain/051-transient-autostart.t
> > 1..4
> > # Creating a new transient domain
> > ok 1 - created transient domain object
> > ok 2 - autostart is disabled for transient VMs
> > ok 3 - Set autostart not supported on transient VMs
> > # Destroying the transient domain
> > # Checking that transient domain has gone away
> > ok 4 - NO_DOMAIN error raised from missing domain
> >
> 
> Results for tests that produce TAP can be seen in the granularity you
> list here, given some of the conditions like the one explained before.
> For instance, the GLib plugin[1] knows that it's possible to run a
> single test case on tests that use the g_test_*() APIs, by providing a
> "-p /test/case/name".
> 
> We could in theory reorganize the current Perl based tests, so that a
> test matches a function or some other type of code marker that can be
> used as an entry point.  Given that the overall result will always be
> valid and the log files will always contain the more granular
> information, I have mixed feelings about the cost/benefit of doing that,
> but you are surely a much better judge for that.

Yeah, that concept is alien to the Perl tests and not worth the effort
to try to change that - you'd be better off just rewriting from scratch
at that point. The perl re-execution granularity is purely at an
individual script level - the implication is that while in python you
might have a single test_foo.py with many tests inside, in Perl you'd
probably have many xxxx.t files.  

> > None the less this does seem like we're pretty close to being able
> > to do something useful in integration
> 
> PS: The GLib plugin was deprecated because we observed that most users
> would do fine with the TAP plugin by itself, but it can be reinstated if
> there is a demand.

GLib's test harness has two output formats - it can output its own
custom data format, or it can output in TAP format. The granularity
of its TAP format is pretty much the same as the way avocado reports
TAP format granularity for python programs.

Using TAP format, however, requires the test harness to pass an
extra --tap arg to the individual test program.  You can also still
use the -p arg. 

If you're deprecating the GLib plugin, is avocado TAP harness have
a way to be told to pass --tap, and a way to se -p to select individual
tests ?

I agre with no needing to support the GLib test output format - it
is more typical for GLib apps to be using its TAP format instead
these days.

> PS2: If a test needs to produce more detailed information about its
> status, such as for a long running test with different stages, we
> recommend that advanced logging is used[2], instead of calling the
> different stages of a single test (from Avocado's PoV).
> 
> [1] - https://avocado-framework.readthedocs.io/en/82.0/plugins/optional/glib.html
> [2] - https://avocado-framework.readthedocs.io/en/89.0/guides/writer/chapters/logging.html

That logging assume a python program ? Is there a way for non-python
programs to get logging back into avocado ?

Regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|