[libvirt PATCH 0/4] RFC: tests: introduce lavocado

[Cleber Rosa]

Daniel P. Berrangé writes:

> On Thu, Jul 01, 2021 at 06:09:47PM -0300, Beraldo Leal wrote:
>> 
>> This is supported already:
>> 
>> $ avocado run --tap - --test-runner='nrunner' tests/domain/transient.py
>> 1..3
>> ok 1 tests/domain/transient.py:TransientDomain.test_autostart
>> ok 2 tests/domain/transient.py:TransientDomain.test_lifecycle
>> ok 3 tests/domain/transient.py:TransientDomain.test_convert_transient_to_persistent
>
> This is nice, showing fine grained TAP output lines for each
> individual test within the test program
>
>
> I tried using the hints file that Cleber pointed to make avocado
> *consume* TAP format for the Perl/Shell scripts:
>
> $ cd libvirt-tck
> $ cat .avocado.hint
> [kinds]
> tap = scripts/*/*.t
>
> [tap]
> uri = $testpath
>
>
> And I can indeed invoke the scripts:
>
> $ avocado run  ./scripts/domain/05*.t
> JOB ID     : b5d596d909dc8024d986957c909fc8fb6b31e2dd
> JOB LOG    : /home/berrange/avocado/job-results/job-2021-07-21T18.45-b5d596d/job.log
>  (1/2) ./scripts/domain/050-transient-lifecycle.t: PASS (0.70 s)
>  (2/2) ./scripts/domain/051-transient-autostart.t: PASS (0.76 s)
> RESULTS    : PASS 2 | ERROR 0 | FAIL 0 | SKIP 0 | WARN 0 | INTERRUPT 0 | CANCEL 0
> JOB HTML   : /home/berrange/avocado/job-results/job-2021-07-21T18.45-b5d596d/results.html
> JOB TIME   : 1.90 s
>
> which is good.
>
> And also I can ask it to produce tap output too:
>
> $ avocado run --tap - ./scripts/domain/05*.t
> 1..2
> ok 1 ./scripts/domain/050-transient-lifecycle.t
> ok 2 ./scripts/domain/051-transient-autostart.t
>
>
> But this output isn't entirely what I was after. This is just summarizing
> the results of each test program.
>

Hi Daniel,

>From the user experience perspective, I completely understand your
point.  I would side with this expectation and output any day if acting
purely as a user.

But, as a framework, Avocado came up with some well defined concepts,
and one of them is what a test is.  To be able to "replay only the
failed tests of the latest job" or to "run a test completely isolated in
a container or VM", there needs to be an specific and indistinguishable
entry point for anything that is considered a test.

> I can't find a way to make it show the fine grained tap output for the
> individual tests, like it does for the python program
>
> eg I'd like to be able to see something similar to:
>
> $ ./scripts/domain/050-transient-lifecycle.t
> 1..2
> # Creating a new transient domain
> ok 1 - created transient domain object
> # Destroying the transient domain
> # Checking that transient domain has gone away
> ok 2 - NO_DOMAIN error raised from missing domain
>
>
> $ ./scripts/domain/051-transient-autostart.t
> 1..4
> # Creating a new transient domain
> ok 1 - created transient domain object
> ok 2 - autostart is disabled for transient VMs
> ok 3 - Set autostart not supported on transient VMs
> # Destroying the transient domain
> # Checking that transient domain has gone away
> ok 4 - NO_DOMAIN error raised from missing domain
>

Results for tests that produce TAP can be seen in the granularity you
list here, given some of the conditions like the one explained before.
For instance, the GLib plugin[1] knows that it's possible to run a
single test case on tests that use the g_test_*() APIs, by providing a
"-p /test/case/name".

We could in theory reorganize the current Perl based tests, so that a
test matches a function or some other type of code marker that can be
used as an entry point.  Given that the overall result will always be
valid and the log files will always contain the more granular
information, I have mixed feelings about the cost/benefit of doing that,
but you are surely a much better judge for that.

>
> None the less this does seem like we're pretty close to being able
> to do something useful in integration
>
> Regards,
> Daniel

Thanks for the feedback!
- Cleber.

PS: The GLib plugin was deprecated because we observed that most users
would do fine with the TAP plugin by itself, but it can be reinstated if
there is a demand.

PS2: If a test needs to produce more detailed information about its
status, such as for a long running test with different stages, we
recommend that advanced logging is used[2], instead of calling the
different stages of a single test (from Avocado's PoV).

[1] - https://avocado-framework.readthedocs.io/en/82.0/plugins/optional/glib.html
[2] - https://avocado-framework.readthedocs.io/en/89.0/guides/writer/chapters/logging.html