[Avocado-devel] RFC: Avocado multiplexer plugin

Fri Jul 1 13:57:31 UTC 2016

Dne 30.6.2016 v 22:57 Ademar Reis napsal(a):
> On Thu, Jun 30, 2016 at 06:59:39PM +0200, Lukáš Doktor wrote:
>> Hello guys,
>>
>> the purpose of this RFC is to properly split and define the way test
>> parameters are processed. There are several ways to split them apart, each
>> with some benefits and drawbacks.
>>
>>
>> Current params process
>> ======================
>>
>> `tree.TreeNode` -> Object allowing to store things (environment, filters,
>> ...) in tree-structure
>> `multiplexer.Mux` -> Interface between job and multiplexer. Reports number
>> of tests and yields modified test_suite templates
>> `multiplexer.MuxTree` -> Object representing part of the tree from the root
>> to leaves or another multiplex domain. Recursively it creates multiplexed
>> variants of the full tree.
>> `multiplexer.AvocadoParams` -> Params object used to retrieve params from
>> given path, allows defining several domains for relative paths matching
>> defined as `mux_path`s.
>> `multiplexer.AvocadoParam` -> Slice of the `AvocadoParams` which handles
>> given `mux_path`.
>> `test.Test.default_params` -> Dictionary which can define test's default
>> values, it's intended for removal for some time.
>>
>>
>> Creating variants
>> -----------------
>>
>> 1. app
>> 2. parser -> creates the root tree `args.default_avocado_params =
>> TreeNode()`
>> 3. plugins.* -> inject key/value into `args.default_avocado_params` (or it's
>> children). One example is `plugins.run`'s --mux-inject, the other is
>> `avocado_virt`'s default values.
>> 4. job -> creates multiplexer.Mux() object
>>     a. If "-m" specified, parses and filters the yaml file(s), otherwise
>> creates an empty TreeNode() called `mux_tree`
>>     b. If `args.default_avocado_params` exists, it merges it into the
>> `mux_tree` (no filtering of the default params)
>>     c. Initializes `multiplexer.MuxTree` object using the `mux_tree`
>> 5. job -> asks the Mux() object for number of tests
>>     a. Mux iterates all MuxTree variants and reports `no_variants *
>> no_tests`
>> 6. runner -> iterates through test_suite
>>     a. runner -> iterates through Mux:
>>         i.  multiplexer.Mux -> iterates through MuxTree
>>         	* multiplexer.MuxTree -> yields list of leaves of the `mux_tree`
>>         ii, yields the modified test template
>>     b. runs the test template:
>>         i. Test.__init__: |
>>             if isinstance(params, dict):
>>                 # update test's default params
>>             elif params is None:
>>                 # New empty multiplexer.AvocadoParams are created
>>             elif isinstance(params, tuple):
>>                 # multiplexer.AvocadoParams are created from params
>> 7. exit
>>
>> AvocadoParams initialization
>> ----------------------------
>>
>>     def __init__(self, leaves, test_id, tag, mux_path, default_params):
>>         """
>>         :param leaves: List of TreeNode leaves defining current variant
>>         :param test_id: test id
>>         :param tag: test tag
>>         :param mux_path: list of entry points
>>         :param default_params: dict of params used when no matches found
>>         """
>>
>> 1. Iterates through `mux_path` and creates `AvocadoParam` slices containing
>> params from only matching nodes, storing them in `self._rel_paths`
>> 2. Creates `AvocadoParam` slice containing the remaining params, storing
>> them in `self._abs_path`
>>
>> Test params
>> -----------
>>
>>     def get(self, key, path=None, default=None):
>>         """
>>         Retrieve value associated with key from params
>>         :param key: Key you're looking for
>>         :param path: namespace ['*']
>>         :param default: default value when not found
>>         :raise KeyError: In case of multiple different values (params clash)
>>         """
>>
>> 1. Tries to obtain the (key,  path, default) from cache
>> 2. Tries to get the (key, path) from `self._rel_paths`
>> 3. When the `path` is abs_path it tries to get the param from
>> `self._abs_path`
>> 4. Looks into the Test's default params dictionary
>> 5. Returns `default`
>>
>> Overview
>> --------
>>
>> Basically now there are 3 components:
>>
>> 1. params parser (yaml => tree)
>> 2. variants generator (tree => iterator)
>> 3. test parameters (key => value from variant values)
>>
>> and we need to decide how do we want to split it and what should be part of
>> the plugin API and what core API.
>>
>>
>> Plugin parser->tree
>> ===================
>>
>> We can say the `tree` is the actual params API and we could only allow
>> producing custom parsers to construct the tree. This would be the simplest
>> method as it only requires us to move the yaml parsing into the module and
>> the `AvocadoParams` would stay as they are. The cons is that the plugin
>> writers would only be able to produce params compatible with the tree
>> structure (flat, or tree-like).
>>
>> If we decide to chose this method, we can keep the current avocado arguments
>> and only allow replacing the parser plugin, eg. by `--multiplex-plugin
>> NAME`. Alternatively we might even detect the suitable plugin based on the
>> multiplex file and even allow combining them (`.cfg vs. .yaml, ...)
>>
>> The plugin would have to support:
>>
>> * parse_file(FILE) => tree_node
>> * check_file(FILE) => BOOL	// in case we want automatic detection of
>> file->plugin
>>
>> Plugin parser->variant
>> ======================
>>
>> This would require deeper changes, but allow greater flexibility. We'd also
>> have to chose whether we want to allow combinations, or whether the plugin
>> should handle the whole workflow. I don't think we should allow combinations
>> as that would imply another convention for storing the parsed results.
>>
>> The user would configure in config or on cmdline which plugin he wants to
>> use and the arguments would stay the same (optionally extended by the
>> plugin's arguments)
>>
>> The plugin would have to support:
>>
>> * configure(parser)	// optional, add extended options like --mux-version
>> * parse_file(FILE)	// does not return as it's up to plugin to store the
>> results
>> * inject_value(key, value, path)	// used by avocado-virt to inject default
>> values
>> * __len__()			// Return number of variants (we might want to extend this to
>> accept TEMPLATE and allow different number of variants per TEMPLATE. That is
>> currently not supported, but it might be handy
>> * itervariants(TEMPLATE)	// yields modified TEMPLATE with params set in
>> AvocadoParams understandable format
>>
>>
>> Plugin AvocadoParams
>> ====================
>>
>> I don't think we should make the AvocadoParams replaceable, but if we want
>> to we should strictly require `params.get` compatibility so all tests can
>> run seamlessly with all params. Anyway if we decided to make AvocadoParams
>> replaceable, then we can create a proxy between the params and the plugin.
>>
>>
>> Conclusion
>> ==========
>>
>> I'm looking forward to cleaner multiplexer API. I don't think people would
>> like to invest much time in developing fancy multiplexer plugins so I'd go
>> with the `parser->tree` variant, which allows easy extensibility with some
>> level of flexibility. The flexibility is for example sufficient to implement
>> cartesian_config parser.
>>
>> As for the automatic detection, I donẗ like the idea as people might want to
>> use the same format with different custom tags.
>
> Hi Lukáš.
>
> I believe we're in sync, but I miss the high level overview, or
> at least review, of how params, variants and the multiplexer or
> other plugins are all related to each other.
>
> Please check the definitions/examples below to see if we're in
> sync:
>
> Params
> ------
>
>     A dictionary of key/values, with an optional path (we could
>     simply call it prefix), which is used to identify the key
>     when there are multiple versions of it. The path is
>     interpreted from right to left to find a match.
>
>     The Params data structure can be populated by multiple
>     sources.
>
>     Example:
>     (implementation and API details are not discussed here)
>
>     key: var1=a
>     path: /foo/bar/baz
>
>     key: var1=b
>     path: /foo/bar
>
>     key: var2=c
>     path: NULL (empty)
>
>     get(key=var1, path=/foo/) ==> error ("/foo/var1" not found)
>     get(key=var1, path=/foo/*) ==> error (multiple var1)
>     get(key=var1, path=/foo/bar/baz/weeee/) ==> error
>     get(key=var1, path=/foo/bar/weeee/) ==> error
>
>     get(key=var2) ==> c
>     get(key=var2, path=foobar) ==> error ("foobar/var2" not found)
>
>     get(key=var1, path=/foo/bar/baz/) ==> a
>     (unique match for "/foo/bar/baz/var1")
>
>     get(key=var1, path=/foo/bar/) ==> b
>     (unique match for "/foo/bar/var1/")
>
>     get(key=var1, path=baz) ==> a
>     (unique match for "baz/var1")
>
>     get(key=var1, path=bar) ==> b
>     (unique match for "bar/var1")
>
>     This kind of "get" API is exposed in the Test API.
>
>
> Variants
> --------
>
>     Multiple sets of params, all with the same set of keys and
>     paths, but potentially different values. Each variant is
>     identified by a "Variant ID" (see the "Test ID RFC").
>
>     The test runner is responsible for the association of tests
>     and variants. That is, the component creating the
>     variants has absolutely no visibility on which tests are
>     going to be associated with variants.
>
>     This is also completely abstract to tests: they don't have
>     any visibility about which variant they're using, or which
>     variants exist.
>
Hello Ademar,

Thank you for the overview, I probably should have included it. I 
omitted it as it's described in the documentation, so I only mentioned 
in the `Plugin AvocadoParams` that I don't think we should turn that 
part into a plugin.

The variant, as described earlier, is the method which modifies the 
`test_template` and as you pointed out it compounds of `Variant ID` and 
`Params`. The way it works now it can go even further and alter all the 
test's arguments (name, methodName, params, base_logdir, tag, job, 
runner_queue) but it's not documented and might change in the future.

> Given the above, the multiplexer (or any other component, like a
> "cartesian config" implementation from Autotest) would be bound
> to these APIs.
The cartesian config is not related to params at all. Avocado-vt uses a 
hybrid mode and it replaces the params with their custom object, while 
keeping the `avocado` params in `test.avocado_params`. So in 
`avocado_vt` tests you don't have `self.params`, but you have 
`test.params` and `test.avocado_params`, where `test.params` is a 
dictionary and `test.avocado_params` the avocado params interface with 
path/key/value. Cartesian config produces variants not by creating test 
variants, but by adding multiple tests with different parameters to the 
test_suite.

Regards,
Lukáš

>
> Thanks.
>    - Ademar
>

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 473 bytes
Desc: OpenPGP digital signature
URL: <http://listman.redhat.com/archives/avocado-devel/attachments/20160701/98299a30/attachment.sig>