[Avocado-devel] Parameter System Overhaul

[Cleber Rosa]

Even though Avocado has had a parameter passing system for
instrumented tests almost from day one, it has been intertwined with
the varianter (then multiplexer) and this is fundamentally wrong.  The
most obvious example of this broken design is the `mux-inject` command
line option::

  --mux-inject [MUX_INJECT [MUX_INJECT ...]]
                        Inject [path:]key:node values into the final
multiplex
                        tree.

This is broken design not because such a varianter implementations can
be tweaked over the command line, that's fine.  It's broken because it
is the recommended way of passing parameters on the command line.

The varianter (or any other subsystem) should be able to act as a
parameter provider, but can not dictate that parameters must first be
nodes/key/values of its own internal structure.

The proposed design
===================

A diagram has been used on a few different occasions, to describe how
the parameters and variants generation mechanism should be connected
to a test and to the overall Avocado architecture.  Here it is, in its
original form::

   +------------------------------+
   | Test                         |
   +------------------------------+
             |
             |
   +---------v---------+    +--------------------------------+
   | Parameters System |--->| Variants Generation Plugin API |
   +-------------------+    +--------------------------------+
             ^                                ^
             |                                |
   +--------------------------------------+   |
   | +--------------+ +-----------------+ |   |
   | | avocado-virt | | other providers | |   |
   | +--------------+ +-----------------+ |   |
   +--------------------------------------+   |
                                              |
                 +----------------------------+-----+
                 |                                  |
                 |                                  |
                 |                                  |
       +--------------------+           +-------------------------+
       | Multiplexer Plugin |           | Other variant plugin(s) |
       +--------------------+           +-------------------------+
             |
             |
       +-----v---------------------------+
       | +------------+ +--------------+ |
       | | --mux-yaml | | --mux-inject | |
       | +------------+ +--------------+ |
       +---------------------------------+


Given that the "Parameter System" is the entry point into the parameters
providers, it should provide two different interfaces:

 1) An interface for its users, that is, developers writing
    `avocado.Test` based tests

 2) An interface for developers of additional providers, such as the
    "avocado-virt" and "other providers" box on the diagram.

The current state of the the first interface is the ``self.params``
attribute.  Hopefeully, it will be possible to keep its current interface,
so that tests won't need any kind of compatibility adjustments.

The second item will probably mean the definition of a new class to
the ``avocado.core.plugin_interfaces`` module, together with a new
dispatcher(-like) implementation in ``avocado.core.dispatcher``.

Parameters availability: local .vs. external
============================================

Right now, all parameters are given to the test at instantiation time.
Let's say that in this scenario, all parameters are *local* to the
test.  Local parameters have the benefit that the test is self
contained and doesn't need any external communication.

In theory, this is also a limitation, because all parameters must be
available before the test is started.  Even if other parameter system
implementations are possible, with a local approach, there would be a
number of limitations.  For long running tests, that may depend on
parameters generated during the test, this would be a blocker.  Also,
if a huge number of parameters would be available (think of a huge
cloud or database of parameters) they would all have to be copied to
the test at instantiation time.  Finally, it also means that the
source of parameters would need to be able to iterate over all the
available parameters, so that they can be copied, which can be a
further limitation for cascading implementations.

An external approach to parameters, would be one that the test holds a
handle to a broker of parameter providers.  The parameter resolution
would be done at run time.  This avoids the copying of parameters, but
requires runtime communication with the parameter providers.  This can
make the test execution much more fragile and dependent on the external
communication.  Even by minimizing the number of communication
endpoints by communicating with the test runner only, it can still add
significant overhead, latency and point of failures to the test
execution.

I believe that, at this time, the limitations imposed by local
parameter availability do *not* outweigh the problems that an external
approach can bring.  In the future, if advanced use cases require an
external approach to parameters availability, this can be reevaluated.

Namespaces (AKA how/if should we merge parameters)
==================================================

Currently, the parameter fetching interface already contains at its
core the concept of paths[1].  In theory, this is sufficient to prevent
clashes of keys with the same names, but intended to configure different
aspects of a test.

Now, with the proposed implementation of multiple providers to the
parameter system, the question of how they will be combined comes up.

One way is for each implementation to claim, based on some unique
attribute such as its own name, a part of a tree path.  For instance,
for two implementations:

 1) variants
 2) plain_ini

Users could access parameters explicitly defined on those by referring
to paths such as:

   self.params.get('speed', path='/plain_ini', default=60)

or

   self.params.get('speed', path='/variants/path/inside/varianter',
default=60)

This clearly solves the clashes, but binds the implementation to the
tests, which should be avoided at all costs.

One other approach would be to merge everything into the tree root
node.  By doing this, one parameter provider could effectively
override the values in another parameter provider, given that both
used the same paths for a number of parameters.

Yet another approach would be to *not* use paths, and resort to
completely separate namespaces.  A parameter namespace would be an
additional level of isolation, which can quickly become exceedingly
complex.

As can be seen from the section name, I'm not proposing one solution
at this point, but hoping that a discussion on the topic would help
achieve the best possible design.

[1] -
http://avocado-framework.readthedocs.io/en/52.0/api/core/avocado.core.html#avocado.core.varianter.AvocadoParams.get

-- 
Cleber Rosa
[ Sr Software Engineer - Virtualization Team - Red Hat ]
[ Avocado Test Framework - avocado-framework.github.io ]
[  7ABB 96EB 8B46 B94D 5E0F  E9BB 657E 8D33 A5F2 09F3  ]

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 819 bytes
Desc: OpenPGP digital signature
URL: <http://listman.redhat.com/archives/avocado-devel/attachments/20170801/9927bb9a/attachment.sig>