[libvirt PATCH 0/4] RFC: tests: introduce lavocado

[Daniel P. Berrangé]

On Fri, Jul 02, 2021 at 07:23:57AM -0400, Cleber Rosa wrote:
> Hi Daniel,
> 
> On Thu, Jul 1, 2021 at 2:05 PM Daniel P. Berrangé <berrange at redhat.com> wrote:
> >
> > On Wed, Jun 30, 2021 at 01:36:30PM -0300, Beraldo Leal wrote:
> > > lavocado aims to be an alternative test framework for the libvirt
> > > project using Python, python-libvirt and Avocado. This can be used to
> > > write unit, functional and integration tests and it is inspired by the
> > > libvirt-tck framework.
> > >
> > > This series introduces the basic framework along with some basic test
> > > examples that I got from libvirt-tck. I would appreciate your comments
> > > on this RFC, to see if this fits this project's needs. Also, names and
> > > locations are just a proposal and can be changed.
> >
> > Some high level thoughts
> >
> >  - More extensive functional integration testing coverage is good
> >
> >  - We need to actually run the functional tests regularly reporting
> >    via GitLab pipelines in some way
> >
> >  - Using Python is way more contributor friendly than Perl
> >
> >  - This does not need to live in libvirt.git as we don't follow
> >    a monolithic repo approach in libvirt, and it already depends
> >    on functionality provided by other repos.
> >
> >
> > When it comes to testing, I feel like there are several distinct
> > pieces to think about
> >
> >   - The execution & reporting harness
> >   - Supporting infrastructure to aid writing tests
> >   - The set of tests themselves
> >
> > If I look at the TCK
> >
> >  - The harness is essentially the standard Perl harness
> >    with a thin CLI wrapper, thus essentially works with
> >    any test emitting TAP format
> >  - The support infra is all custom APIs using libvirt-perl
> >  - The tests are mostly written in Perl, but some are
> >    written in shell (yuk). They all output TAP format.
> >
> > One key thing here is that the test harness is fairly loosely
> > coupled to the support infra & tests.
> >
> > The TAP data format bridged the two, letting us write tests
> > in essentially any language. Of course writing tests in
> > non-Perl was/is tedious today, since there's no support
> > infra for that which exists today.
> >
> > The TAP data format bridge also means we can easily throw
> > away the current TCK harness and replace it with anything
> > else that can consume tests emitting TAP data.
> >
> >
> > If I look at Avocado, I think (correct me if i'm wrong)
> >
> >  1. The harness is essentially the standard Python harness
> >     with a thin CLI wrapper. Thus needs all tests to implement
> >     the Python test APIs
> 
> Not really.  Even though Avocado is mostly written in Python, there
> have been provisions for accommodating foreign types of tests (in
> different forms) since its inception.  The most basic way, is, of
> course, simply treating a test as an executable.  But this is far from
> the only way.  For instance, these are other possibilities:
> 
> a) if an executable generates TAP, it can consume the test's TAP
> output and determine the test results

Can you show how to actually make this work, since from the manpage
I can only see how to make it emit TAP, not consume it. I see there
is a 'tap' plugin but that doesn't seem to provide a runner.

Regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|