[libvirt PATCH 0/4] RFC: tests: introduce lavocado

[Cleber Rosa]

On Fri, Jul 2, 2021 at 7:55 AM Daniel P. Berrangé <berrange at redhat.com> wrote:
>
> On Fri, Jul 02, 2021 at 07:23:57AM -0400, Cleber Rosa wrote:
> > Hi Daniel,
> >
> > On Thu, Jul 1, 2021 at 2:05 PM Daniel P. Berrangé <berrange at redhat.com> wrote:
> > >
> > > On Wed, Jun 30, 2021 at 01:36:30PM -0300, Beraldo Leal wrote:
> > > > lavocado aims to be an alternative test framework for the libvirt
> > > > project using Python, python-libvirt and Avocado. This can be used to
> > > > write unit, functional and integration tests and it is inspired by the
> > > > libvirt-tck framework.
> > > >
> > > > This series introduces the basic framework along with some basic test
> > > > examples that I got from libvirt-tck. I would appreciate your comments
> > > > on this RFC, to see if this fits this project's needs. Also, names and
> > > > locations are just a proposal and can be changed.
> > >
> > > Some high level thoughts
> > >
> > >  - More extensive functional integration testing coverage is good
> > >
> > >  - We need to actually run the functional tests regularly reporting
> > >    via GitLab pipelines in some way
> > >
> > >  - Using Python is way more contributor friendly than Perl
> > >
> > >  - This does not need to live in libvirt.git as we don't follow
> > >    a monolithic repo approach in libvirt, and it already depends
> > >    on functionality provided by other repos.
> > >
> > >
> > > When it comes to testing, I feel like there are several distinct
> > > pieces to think about
> > >
> > >   - The execution & reporting harness
> > >   - Supporting infrastructure to aid writing tests
> > >   - The set of tests themselves
> > >
> > > If I look at the TCK
> > >
> > >  - The harness is essentially the standard Perl harness
> > >    with a thin CLI wrapper, thus essentially works with
> > >    any test emitting TAP format
> > >  - The support infra is all custom APIs using libvirt-perl
> > >  - The tests are mostly written in Perl, but some are
> > >    written in shell (yuk). They all output TAP format.
> > >
> > > One key thing here is that the test harness is fairly loosely
> > > coupled to the support infra & tests.
> > >
> > > The TAP data format bridged the two, letting us write tests
> > > in essentially any language. Of course writing tests in
> > > non-Perl was/is tedious today, since there's no support
> > > infra for that which exists today.
> > >
> > > The TAP data format bridge also means we can easily throw
> > > away the current TCK harness and replace it with anything
> > > else that can consume tests emitting TAP data.
> > >
> > >
> > > If I look at Avocado, I think (correct me if i'm wrong)
> > >
> > >  1. The harness is essentially the standard Python harness
> > >     with a thin CLI wrapper. Thus needs all tests to implement
> > >     the Python test APIs
> >
> > Not really.  Even though Avocado is mostly written in Python, there
> > have been provisions for accommodating foreign types of tests (in
> > different forms) since its inception.  The most basic way, is, of
> > course, simply treating a test as an executable.  But this is far from
> > the only way.  For instance, these are other possibilities:
> >
> > a) if an executable generates TAP, it can consume the test's TAP
> > output and determine the test results
>
> Can you show how to actually make this work, since from the manpage
> I can only see how to make it emit TAP, not consume it. I see there
> is a 'tap' plugin but that doesn't seem to provide a runner.
>

There are a couple of ways.  The simplest is hinting to Avocado that a
file is of kind "tap" with a hintfile.  Suppose you have a
"test-suite" directory, and in it, you have "test.sh":

#!/bin/sh -e
echo "1..2"
echo "ok 2 /bin/true"
echo "ok 3 /bin/uname"

And ".avocado.hint":

[kinds]
tap = *.sh

[tap]
uri = $testpath

If you "cd test-suite", and do "avocado list --resolver *sh" you get:

Type Test    Tag(s)
tap  test.sh

Resolver Reference Info

TEST TYPES SUMMARY
==================
tap: 1


And then you can run it with:

$ avocado run --test-runner=nrunner ./test.sh
JOB ID     : 2166ad93ffc25da4d5fc8e7db073f4555d55e81a
JOB LOG    : /home/cleber/avocado/job-results/job-2021-07-02T08.39-2166ad9/job.log
 (1/1) ./test.sh: STARTED
 (1/1) ./test.sh: PASS (0.01 s)
RESULTS    : PASS 1 | ERROR 0 | FAIL 0 | SKIP 0 | WARN 0 | INTERRUPT 0
| CANCEL 0
JOB HTML   : /home/cleber/avocado/job-results/job-2021-07-02T08.39-2166ad9/results.html
JOB TIME   : 1.31 s

This is a relevant documentation pointer:
https://avocado-framework.readthedocs.io/en/latest/guides/user/chapters/introduction.html#the-hint-files

And I'll make sure the man page is updated, thanks for noticing it.

Thanks,
- Cleber.

> Regards,
> Daniel
> --
> |: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
> |: https://libvirt.org         -o-            https://fstop138.berrange.com :|
> |: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|
>