[Avocado-devel] RFC: Avocado multiplexer plugin

[Ademar Reis]

On Thu, Jun 30, 2016 at 06:59:39PM +0200, Lukáš Doktor wrote:
> Hello guys,
> 
> the purpose of this RFC is to properly split and define the way test
> parameters are processed. There are several ways to split them apart, each
> with some benefits and drawbacks.
> 
> 
> Current params process
> ======================
> 
> `tree.TreeNode` -> Object allowing to store things (environment, filters,
> ...) in tree-structure
> `multiplexer.Mux` -> Interface between job and multiplexer. Reports number
> of tests and yields modified test_suite templates
> `multiplexer.MuxTree` -> Object representing part of the tree from the root
> to leaves or another multiplex domain. Recursively it creates multiplexed
> variants of the full tree.
> `multiplexer.AvocadoParams` -> Params object used to retrieve params from
> given path, allows defining several domains for relative paths matching
> defined as `mux_path`s.
> `multiplexer.AvocadoParam` -> Slice of the `AvocadoParams` which handles
> given `mux_path`.
> `test.Test.default_params` -> Dictionary which can define test's default
> values, it's intended for removal for some time.
> 
> 
> Creating variants
> -----------------
> 
> 1. app
> 2. parser -> creates the root tree `args.default_avocado_params =
> TreeNode()`
> 3. plugins.* -> inject key/value into `args.default_avocado_params` (or it's
> children). One example is `plugins.run`'s --mux-inject, the other is
> `avocado_virt`'s default values.
> 4. job -> creates multiplexer.Mux() object
>     a. If "-m" specified, parses and filters the yaml file(s), otherwise
> creates an empty TreeNode() called `mux_tree`
>     b. If `args.default_avocado_params` exists, it merges it into the
> `mux_tree` (no filtering of the default params)
>     c. Initializes `multiplexer.MuxTree` object using the `mux_tree`
> 5. job -> asks the Mux() object for number of tests
>     a. Mux iterates all MuxTree variants and reports `no_variants *
> no_tests`
> 6. runner -> iterates through test_suite
>     a. runner -> iterates through Mux:
>         i.  multiplexer.Mux -> iterates through MuxTree
>         	* multiplexer.MuxTree -> yields list of leaves of the `mux_tree`
>         ii, yields the modified test template
>     b. runs the test template:
>         i. Test.__init__: |
>             if isinstance(params, dict):
>                 # update test's default params
>             elif params is None:
>                 # New empty multiplexer.AvocadoParams are created
>             elif isinstance(params, tuple):
>                 # multiplexer.AvocadoParams are created from params
> 7. exit
> 
> AvocadoParams initialization
> ----------------------------
> 
>     def __init__(self, leaves, test_id, tag, mux_path, default_params):
>         """
>         :param leaves: List of TreeNode leaves defining current variant
>         :param test_id: test id
>         :param tag: test tag
>         :param mux_path: list of entry points
>         :param default_params: dict of params used when no matches found
>         """
> 
> 1. Iterates through `mux_path` and creates `AvocadoParam` slices containing
> params from only matching nodes, storing them in `self._rel_paths`
> 2. Creates `AvocadoParam` slice containing the remaining params, storing
> them in `self._abs_path`
> 
> Test params
> -----------
> 
>     def get(self, key, path=None, default=None):
>         """
>         Retrieve value associated with key from params
>         :param key: Key you're looking for
>         :param path: namespace ['*']
>         :param default: default value when not found
>         :raise KeyError: In case of multiple different values (params clash)
>         """
> 
> 1. Tries to obtain the (key,  path, default) from cache
> 2. Tries to get the (key, path) from `self._rel_paths`
> 3. When the `path` is abs_path it tries to get the param from
> `self._abs_path`
> 4. Looks into the Test's default params dictionary
> 5. Returns `default`
> 
> Overview
> --------
> 
> Basically now there are 3 components:
> 
> 1. params parser (yaml => tree)
> 2. variants generator (tree => iterator)
> 3. test parameters (key => value from variant values)
> 
> and we need to decide how do we want to split it and what should be part of
> the plugin API and what core API.
> 
> 
> Plugin parser->tree
> ===================
> 
> We can say the `tree` is the actual params API and we could only allow
> producing custom parsers to construct the tree. This would be the simplest
> method as it only requires us to move the yaml parsing into the module and
> the `AvocadoParams` would stay as they are. The cons is that the plugin
> writers would only be able to produce params compatible with the tree
> structure (flat, or tree-like).
> 
> If we decide to chose this method, we can keep the current avocado arguments
> and only allow replacing the parser plugin, eg. by `--multiplex-plugin
> NAME`. Alternatively we might even detect the suitable plugin based on the
> multiplex file and even allow combining them (`.cfg vs. .yaml, ...)
> 
> The plugin would have to support:
> 
> * parse_file(FILE) => tree_node
> * check_file(FILE) => BOOL	// in case we want automatic detection of
> file->plugin
> 
> Plugin parser->variant
> ======================
> 
> This would require deeper changes, but allow greater flexibility. We'd also
> have to chose whether we want to allow combinations, or whether the plugin
> should handle the whole workflow. I don't think we should allow combinations
> as that would imply another convention for storing the parsed results.
> 
> The user would configure in config or on cmdline which plugin he wants to
> use and the arguments would stay the same (optionally extended by the
> plugin's arguments)
> 
> The plugin would have to support:
> 
> * configure(parser)	// optional, add extended options like --mux-version
> * parse_file(FILE)	// does not return as it's up to plugin to store the
> results
> * inject_value(key, value, path)	// used by avocado-virt to inject default
> values
> * __len__()			// Return number of variants (we might want to extend this to
> accept TEMPLATE and allow different number of variants per TEMPLATE. That is
> currently not supported, but it might be handy
> * itervariants(TEMPLATE)	// yields modified TEMPLATE with params set in
> AvocadoParams understandable format
> 
> 
> Plugin AvocadoParams
> ====================
> 
> I don't think we should make the AvocadoParams replaceable, but if we want
> to we should strictly require `params.get` compatibility so all tests can
> run seamlessly with all params. Anyway if we decided to make AvocadoParams
> replaceable, then we can create a proxy between the params and the plugin.
> 
> 
> Conclusion
> ==========
> 
> I'm looking forward to cleaner multiplexer API. I don't think people would
> like to invest much time in developing fancy multiplexer plugins so I'd go
> with the `parser->tree` variant, which allows easy extensibility with some
> level of flexibility. The flexibility is for example sufficient to implement
> cartesian_config parser.
> 
> As for the automatic detection, I donẗ like the idea as people might want to
> use the same format with different custom tags.

Hi Lukáš.

I believe we're in sync, but I miss the high level overview, or
at least review, of how params, variants and the multiplexer or
other plugins are all related to each other.

Please check the definitions/examples below to see if we're in
sync:

Params
------

    A dictionary of key/values, with an optional path (we could
    simply call it prefix), which is used to identify the key
    when there are multiple versions of it. The path is
    interpreted from right to left to find a match.

    The Params data structure can be populated by multiple
    sources.

    Example:
    (implementation and API details are not discussed here)

    key: var1=a
    path: /foo/bar/baz

    key: var1=b
    path: /foo/bar

    key: var2=c
    path: NULL (empty)

    get(key=var1, path=/foo/) ==> error ("/foo/var1" not found)
    get(key=var1, path=/foo/*) ==> error (multiple var1)
    get(key=var1, path=/foo/bar/baz/weeee/) ==> error
    get(key=var1, path=/foo/bar/weeee/) ==> error

    get(key=var2) ==> c
    get(key=var2, path=foobar) ==> error ("foobar/var2" not found)

    get(key=var1, path=/foo/bar/baz/) ==> a
    (unique match for "/foo/bar/baz/var1")

    get(key=var1, path=/foo/bar/) ==> b
    (unique match for "/foo/bar/var1/")

    get(key=var1, path=baz) ==> a
    (unique match for "baz/var1")

    get(key=var1, path=bar) ==> b
    (unique match for "bar/var1")

    This kind of "get" API is exposed in the Test API.


Variants
--------

    Multiple sets of params, all with the same set of keys and
    paths, but potentially different values. Each variant is
    identified by a "Variant ID" (see the "Test ID RFC").

    The test runner is responsible for the association of tests
    and variants. That is, the component creating the
    variants has absolutely no visibility on which tests are
    going to be associated with variants.

    This is also completely abstract to tests: they don't have
    any visibility about which variant they're using, or which
    variants exist.

Given the above, the multiplexer (or any other component, like a
"cartesian config" implementation from Autotest) would be bound
to these APIs.

Thanks.
   - Ademar

-- 
Ademar Reis
Red Hat

^[:wq!