[libvirt PATCH 0/4] RFC: tests: introduce lavocado

Thu Jul 22 21:12:52 UTC 2021

On Thu, Jul 22, 2021 at 09:44:28AM +0100, Daniel P. Berrangé wrote:
> On Wed, Jul 21, 2021 at 04:22:19PM -0300, Beraldo Leal wrote:
> > On Wed, Jul 21, 2021 at 06:50:03PM +0100, Daniel P. Berrangé wrote:
> > > On Thu, Jul 01, 2021 at 06:09:47PM -0300, Beraldo Leal wrote:
> > > > On Thu, Jul 01, 2021 at 07:04:32PM +0100, Daniel P. Berrangé wrote:
> > > > > On Wed, Jun 30, 2021 at 01:36:30PM -0300, Beraldo Leal wrote:
> > > > > > I'm adding more information with some details inside the README file.
> > > > > 
> > > > > Overall, I'm more enthusiastic about writing tests in Python
> > > > > than Perl, for the long term, but would also potentially like
> > > > > to write tests in Go too.
> > > > > 
> > > > > I'm wondering if we can't bridge the divide between what we
> > > > > have already in libvirt-tck, and what you're bringing to the
> > > > > table with avocado here. While we've not done much development
> > > > > with the TCK recently, there are some very valuable tests
> > > > > there, especially related to firewall support and I don't
> > > > > fancy rewriting them.
> > > > > 
> > > > > Thus my suggestion is that we:
> > > > > 
> > > > >   - Put this avocado code into the libvirt-tck repository,
> > > > >     with focus on the supporting infra for making it easy to
> > > > >     write Python tests
> > > > > 
> > > > >   - Declare that all tests need a way to emit TAP format,
> > > > >     no matter what language they're written in. This could
> > > > >     either be the test directly emitting TAP, or it could
> > > > >     be via use of a plugin. For example 'tappy' can make
> > > > >     existing Python tests emit TAP, with no modifications
> > > > >     to the tests themselves.
> > > > > 
> > > > >       https://tappy.readthedocs.io/en/latest/consumers.html
> > > > > 
> > > > >     IOW, you can still invoke the python tests using the
> > > > >     standard Python test runner, and still invoke the perl
> > > > >     tests using the stnadard Perl test runner if desired.
> > > > 
> > > > This is supported already:
> > > > 
> > > > $ avocado run --tap - --test-runner='nrunner' tests/domain/transient.py
> > > > 1..3
> > > > ok 1 tests/domain/transient.py:TransientDomain.test_autostart
> > > > ok 2 tests/domain/transient.py:TransientDomain.test_lifecycle
> > > > ok 3 tests/domain/transient.py:TransientDomain.test_convert_transient_to_persistent
> > > 
> > > This is nice, showing fine grained TAP output lines for each
> > > individual test within the test program
> > > 
> > > 
> > > I tried using the hints file that Cleber pointed to make avocado
> > > *consume* TAP format for the Perl/Shell scripts:
> > > 
> > > $ cd libvirt-tck
> > > $ cat .avocado.hint
> > > [kinds]
> > > tap = scripts/*/*.t
> > > 
> > > [tap]
> > > uri = $testpath
> > > 
> > > 
> > > And I can indeed invoke the scripts:
> > > 
> > > $ avocado run  ./scripts/domain/05*.t
> > > JOB ID     : b5d596d909dc8024d986957c909fc8fb6b31e2dd
> > > JOB LOG    : /home/berrange/avocado/job-results/job-2021-07-21T18.45-b5d596d/job.log
> > >  (1/2) ./scripts/domain/050-transient-lifecycle.t: PASS (0.70 s)
> > >  (2/2) ./scripts/domain/051-transient-autostart.t: PASS (0.76 s)
> > > RESULTS    : PASS 2 | ERROR 0 | FAIL 0 | SKIP 0 | WARN 0 | INTERRUPT 0 | CANCEL 0
> > > JOB HTML   : /home/berrange/avocado/job-results/job-2021-07-21T18.45-b5d596d/results.html
> > > JOB TIME   : 1.90 s
> > > 
> > > which is good.
> > > 
> > > And also I can ask it to produce tap output too:
> > > 
> > > $ avocado run --tap - ./scripts/domain/05*.t
> > > 1..2
> > > ok 1 ./scripts/domain/050-transient-lifecycle.t
> > > ok 2 ./scripts/domain/051-transient-autostart.t
> > > 
> > > 
> > > But this output isn't entirely what I was after. This is just summarizing
> > > the results of each test program.
> > > 
> > > I can't find a way to make it show the fine grained tap output for the
> > > individual tests, like it does for the python program
> > 
> > Actually, the first Python TAP output example is showing a coarse TAP
> > result. I have combined three transient tests into one single file but
> > splitting them into three different methods, so we could execute each one
> > individually. i.e: tests/domain/transient.py:TransientDomain.test_autostart.
> > 
> > So, I did some reorg when migrating to Python test.
> > 
> > In order to archive the same with Perl, we could do the same there,
> > because the way individual tests are written there, doesn't allow for
> > individual execution. 
> > 
> > Yes, we could do some tricks, to parse and combine outputs and list as
> > it was a more fine graned, but afaict, we could not individually execute
> > those. This is part of Avocado test definition where in order to be
> > called a test, we need to be able to execute those individually as well.
> 
> Ok, I'm not so fussed about whether avocado can ultimately preserve the
> fine grained TAP output. Mostly I'm looking to understand how you should
> debug failures when they go wrong, becuase the default output I see from
> avocado is very terse giving no indiciation of the the failure - merely
> that there has been a failure.
> 
> In an interactive environment you can just re-rnu the individual failed
> test directly. In an automated CI environment you need the test harness
> to display enough info to debug the failure directly. Ideally it should
> dump the full stdout+stderr either to the console or a log file that
> can be publish as an artifact from CI.  Meson tends todo the latter
> creating a log file will full stdout/err from all tests.

Sure, I completely understand your concerns.

In this case, the output here is "very terse" because you are running
the "--tap" version. When using the "default/human" output, Avocado
outputs some pointers to where you could get more details.

We understand that debugging is a critical part of the process, so
Avocado stores the job information inside
"~/avocado/job-results/[job-id|latest]/" folder by default (regardless
of the output format used).

Inside this folder, you can find a "debug.log" file and a "test-results"
sub-folder (with per-test stdout/stderr files). The entire folder could
be uploaded as an artifact for any CI. It is also generated by default a
results.xml (xUnit format) for better debugging on CI that supports this
format (i.e: Gitlab). 

If you would like to see an example, QEMU project's test list[1] on
Gitlab is based on this XML file.

I understand that there is some room for improvements here and we are
glad to address them. In meanwhile I was working on a proposal for the
libvirt-tck based on the progress of this thread. Hopefully, this will
help with the Avocado bootstrap phase.

[1] - https://gitlab.com/qemu-project/qemu/-/pipelines/340613922/test_report
[2] - https://gitlab.com/beraldoleal/libvirt-tck/-/tree/avocado-libvirt/

Regards,
--
Beraldo