[Avocado-devel] RFC: Avocado multiplexer plugin

[Lukáš Doktor]

...

>> To summarize it here (once again):
>>
>> 1. The tree is a "database" which allows injecting values under certain keys
>> with given path (namespace would be also acceptable by me, I don't like the
>> tag, though). So far we agreed that we should turn the yaml=>tree
>> pseudoplugin to real plugin and we are looking for a name (I'd call it
>> yaml2params).
>> 2. Mux is the variant generation mechanism, which is responsible for walking
>> the database and producing variants (which are lists of path along with
>> their values)
>> 3. AvocadoParams is a per-variant database used inside tests to query for
>> values
>>
>> So (1) is actually a core API (independent on yaml parser) which feeds the
>> Mux, the (2) is used by the runner (or job API?) to produce multiple
>> variants of the same test and the (3) is the API available inside test to
>> query for the current test parameters.
>>
>> The (1), (2) and (3) are essential parts of avocado, independent on yaml and
>> currently should stay in `core`, even though that in the future we might
>> want to allow replacing the (1) and maybe even (2) with plugins. Currently
>> this is not important and we are not aware of anyone demanding it. We take
>> (3) as definite for now, so it should not change apart from optimizations.
>
> I think we have different views of what the abstraction layer
> should be. But I don't think this is going to be used anytime
> soon by 3rd parties and we can always make more changes in the
> future, so it's OK to disagree here.
Well I'm not saying this is the final stage, it's the bare minimum for 
removing yaml-dependent part from core and allow custom plugins as 
params sources.

>
>>
>> The workflow is:
>>
>> 1. params-feeder (eg. `yaml2params`, or `--mux-inject`) -> inject records
>> into `tree` (in the future it could be any arbitrary database, currently
>> hardcoded as tree)
>
> I think we need a better vocabulary. In particular, we need to
> define what params are. Are params branches/leaves of the tree?
> Or are params the result of the processing of a tree?
>
AvocadoParams is a driver to query data from the current variant. The 
current variant is a product of Mux which is currently defined as list 
of TreeNodes (as a consequence this currently means leaves, but one can 
manually supply any TreeNode object).

>> 2. `tree` merges all records and applies filters
>
> Filters are part of the tree structure? So are the multiplex
> flags? Again, if this is exposed at the avocado-core API level,
> we're exposing too many implementation details and using the
> (current) multiplex as the API, instead of something more generic
> and abstract. That's why I think the core API should be about
> params (as defined by me previously), not about trees to be
> processed.
Filters are applied on trees. They are applied by Mux object when the 
tree is final (my sentence before was a bit misleading, sorry about 
that). Think about trees as about special database which contain:

* path
* environment (key+value pairs)
* tags (!mux, !filters, ...)


>
>> 3. `Mux` object is created on top of the tree
>> 4. test runner uses `Mux` to produce variants and runs tests+variant
>> combinations
>> 5. each test creates AvocadoParams from the variant
>>
>> Currently it works a bit differently (inherited from the old days) because
>> the `yaml2param`, currently `core.tree.create_from_yaml`, actually creates
>> the tree database directly. This is not a problem now, it would be no
>> problem even after turning it into a proper plugin, but it'd become a
>> problem if we chose to replace `tree` with any arbitrary database, therefor
>> I think it's time to extract the `create_from_yaml` and instead of creating
>> the tree we should only ask the `tree` (or database) to inject/update/remove
>> variables/filters/flags. Then we have a clear interface.
>>
>> So the multiplexer is core and IMO should stay there. Only the
>> params-feeders should be extracted, which means the PyYAML would be
>> optional. Also multiplexer is just a database with combinations so it could
>> be used for anything, also we want to keep using one database for test
>> variants and not mixed settings/config_files/test_params.
>>
>> I hope everything is clear, because if not I'd have to draw a
>> chart :-D
>
> I think I get what you're proposing. I would do things
> differently, but this is all going to be internal for a while
> more and we can improve it incrementally.
Sure, this is the first (actually second, the first one was to split 
tree+multiplexer and create the Mux object to define the 
avocado->variant interface) step. How I see params now is:

1. params-feeders => --mux-inject or yaml parser which should inject 
path/environment/tags into the tree (or let's call it a database, should 
become an independent plugin in the close future)
2. tree (database) of path/environment/tags. It's purpose is to store 
key/value pairs as well as information which allows Mux to produce 
different variants (basically slices of the database's environment)
3. Mux which takes the tree (database), applies filters and allows 
producing variants
4. variant => list of path/environment mappings (slice of the tree), 
currently list of full TreeNode structures. Only their path+environment 
are used (so if you insist we can only send tuple(path, key+values))
5. AvocadoParams => driver to get values based on key+path from the variant

Lukáš

>
> Thanks.
>    - Ademar
>
>>
>> Regards,
>> Lukáš
>>
>>> Thanks.
>>>    - Ademar
>>>
>>>> 2. multiplexer - to produce variants
>>>> 3. avocado params - to obtain params from variant
>>>>
>>>> We should probably find a suitable name to ambiguously identify each part as
>>>> of today we only have multiplexer and avocado params, which can lead to
>>>> confusions. I can't come up with any good name so unless you have a good
>>>> name we maybe end up with the same name. The situation should be better,
>>>> though, because those parts will be really separated.
>>>>
>>>> Thank you for the feedback, let's get my hands dirty. :-)
>>>>
>>>> Regards,
>>>> Lukáš
>>>>
>>>>>
>>>>>> Regards,
>>>>>> Lukáš
>>>>>>
>>>>>
>>>>> [1] - https://docs.python.org/2.6/glossary.html#term-iterable
>>>>>
>>>>
>>>
>>>
>>>
>>>
>>
>
>
>
>

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 473 bytes
Desc: OpenPGP digital signature
URL: <http://listman.redhat.com/archives/avocado-devel/attachments/20160719/55f1a88d/attachment.sig>