[Avocado-devel] N(ext) Runner - The road to maturity

[Cleber Rosa]

Intro
=====

The N(ext) Runner is an experiment started within the Avocado project,
with the overall goal of solving some fundamental problems with the
current architecture, and opening up the possibility of smoother
implementation of a number of advanced features and use case.

For a complete list of the limitations and use cases, please refer
to::

  https://avocado-framework.readthedocs.io/en/79.0/future/core/nrunner.html#motivation

On the last Avocado release (79.0), a comparison of the *current* and
*N(ext) Runner* components currently co-existing in Avocado was
posted.  It's a suggested reading for people interested in following
this discussion and contributing to this RFC::

  https://avocado-framework.readthedocs.io/en/79.0/future/core/nrunner.html#current-and-n-ext-runner-components-of-avocado

In addition the explanation linked above, it's also a good idea to
explain to go over the various development efforts in place, which to
some extent, contribute or influence or require some parts of the
N(ext) Runner architecture.

Job API
=======

Avocado, as a "test automation framework", provides a native API that
users can use to write tests from scratch.  These are, quite simply,
the "Test APIs"::

  https://avocado-framework.readthedocs.io/en/79.0/api/test/avocado.html#test-apis

But, we have noticed that a large part of the test automation process
in the real world also happens outside of individual tests.  To
facilitate the integration of the environment and the execution of a
number of tests, it was natural to look at the layer that encompasses
more responsibilities.

In Avocado terminology, a "Job" is an unique execution of a number of
tests.  An Avocado Job is traditionally created/executed by means of
the ``avocado run`` command.  The goal of the "Job API" work is to augment
the expressiveness of jobs, by removing the obligatory command
line interface, and giving users the possibility of using more flexible
job descriptions.

With the "Job API", users should be able to write advanced jobs in a
number ofcomplementary ways, listed here from least complex to most
complex:

* Providing a Python Dictionary configuration to the Job() class, which
  can be easily loaded, say, from a JSON file

* Writing Python code that creates the Python dictionary configuration
  at run time, and gives that to the Job() class

* Subclassing the core Job() implementation, customizing specific phases
  of the job workflow, such as the creation of the "test suite", that is
  the definition of the tests to be executed by the job.

Because Avocado is not interested in creating a new programming
language, most of that should be done with standard programming
languages, such as Avocado's primary language: pure Python.

Currently ongoing development
-----------------------------

There has been three major types of developments around the Job API:

1. Setting some examples "job files", and making use of custom jobs on
   Avocado's own code.  Users can find example job files in "examples/jobs"
   directory in the source code::

   https://github.com/avocado-framework/avocado/tree/master/examples/jobs

   And jobs that are used during the Avocado release process::

   https://github.com/avocado-framework/avocado/tree/master/jobs

2. Introducing and porting Avocado's code to new combined settings
   module.  This should allow the same experience for users of the
   command line tool giving option as command line arguments, users
   of the Job API providing the configuration as a dictionary, or
   users of both command line and Job API settings options with
   configuration files.

3. Making sure that features that work while running jobs on the
   command line (via "avocado run"), also work the same way in custom
   jobs.  Recently a number of issues with output not being created
   when running custom jobs were fixed, but there are certainly many
   more to be found.


Relationship with N(ext) Runner
-------------------------------

The N(ext) Runner will only be considered "feature complete" when it
is completely integrated into an Avocado Job.  This means that the job
files discussed on the previous section item #1 should behave the same
with the current runner, or the "nrunner" implementation.  The items
#2 and #3, should, as much as possible be finished *before* that, given
that not doing so will cast a number of questions on who the culprit
for a bug is.  Any bug will lead to the question: is it a Job API bug
or a N(ext) Runner bug?

The mechanism that a Job uses to select a runner implementation already
is in place.  Once can see the runner implementations available by
running::

  $ avocado plugins
  ...
  Plugins that run test suites on a job (runners):
  docker  *DEPRECATED* Runs on a Docker (or compatible) container
  nrunner *EXPERIMENTAL* nrunner based implementation of job compliant runner
  remote  *DEPRECATED* Runs on a remote machine using SSH
  runner  The conventional test runner
  vm      *DEPRECATED* Runs on a VM using SSH

And one can set a custom runner by setting the "test_runner" key
in an Avocado, as it can be seen in "examples/jobs/nrunner.py"::

  #!/usr/bin/env python3

  import sys
  from avocado.core.job import Job

  config = {
      'test_runner': 'nrunner',
      'run.references': [
          'selftests/unit/test_resolver.py',
          'selftests/functional/test_argument_parsing.py',
          '/bin/true',
          '/bin/false',
      ],
      }

  with Job(config) as j:
      sys.exit(j.run())

The ability to set the runner as part of the command line, has
also been proposed (and be merged by the time one read this)::

  https://github.com/avocado-framework/avocado/pull/3816

But setting the configuration in a configuration file and running all
tests should give a quick indication of the number of issues with the
N(ext) Runner.

Requirements Resolver
=====================

The requirements resolver is an abstraction and evolution of the
current "asset fetcher" utility.

Currently ongoing development
-----------------------------

There's a blue print document describing the motivation,
specification, etc::

  https://avocado-framework.readthedocs.io/en/79.0/blueprints/BP002.html

Relationship with N(ext) Runner
-------------------------------

While the ideas and features of the Requirements Resolver are
not bound to the N(ext) Runner, its implementation must be based
on one architecture.

Given the current plans to phase out and remove the current runner
implementation, it makes sense to implement it based on the N(ext)
Runner architecture, but because of the lack of a complete
implementation or specification, this can prove to be challenging.

One of the goals of this RFC, is to look at the proposal for the
N(ext) Runner pending development tasks, and discuss an architecture
that facilitates the implementation of the Requirements Resolver.

Some early brainstorming about how some of the N(ext) Runner bits
may be (or may NOT be) useful in the context of the Requirements
Resolver implementation happened here:

  https://github.com/avocado-framework/avocado/pull/3756

But as noticed there, a lot of the N(ext) Runner missing bits
themselves have to be defined.

Avocado-VT
==========

Avocado-VT, in its previous Virt-Test and Autotest names incarnations
and names, were the projects that gave birth to Avocado, so we'll
always be fond of them.

Having said that, we have to be careful with the promise of
compatibility for the N(ext) Runner and current Avocado-VT, specially
about most of the Avocado-VT workflows and expectations.

For instance, Avocado-VT tests will, as a general rule, access guest
image files in an installation wide location, say
"~/avocado/data/avocado-vt/images/f30-x86_64.qcow2".  There's even
a special plugin, "vt-joblock" whose goal is to prevent multiple jobs
for running simultaneously and prevent corruption of such images.

Typical Avocado-VT usage with the libvirt backend, will usually
require or suggest a fixed set of tests.  For instance, the first test
in a job may be one that "defines" a libvirt guest domain, the
following tests may be tests that operate on the guest domain defined
by the first test, and the final test is usually one that cleans up
the libvirt domain guest definition.

Now, thinking of N(ext) Runner, one of its main selling points is the
ability to run tests in parallel.  While this is a benefit to most
users, it's actually a problem to current Avocado-VT tests and
workflows.  Corruption and general test errors and failures would be
almost guaranteed to occur.

I can think of two action items to manage Avocado-VT, which are
complimentary:

1. Have an LTS release, say version 82.0, with the current runner
   implementation still supported.  This LTS version can be maintained
   for twice the time of previous LTS releases (~3 years), and the
   possibility to extend it even further.

2. Attempt to migrate Avocado-VT infrascture code and tests to
   behave well within the assumptions of the N(ext) Runner
   architecture.

Other?
======

If you're an Avocado user or developer, and you think there are
other topics that may be impacted by the N(ext) Runner architecture
and the deprecation and removal of the current runner architecture,
please join the discussion and raise your point.

Next
====

Following this, I'll be posting an architectural blue print proposal
for the missing pieces of the N(ext) Runner.  Feedback on the proposal
is much appreciated, and it's worth reminding everyone that there's no
unworthy suggestion or question.

Thanks!
- Cleber.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 833 bytes
Desc: not available
URL: <http://listman.redhat.com/archives/avocado-devel/attachments/20200513/317e6a41/attachment.sig>