[Avocado-devel] Multiplexer: mechanism for tests to retrieve variables

[Ademar Reis]

On Tue, Jan 20, 2015 at 07:54:27PM +0100, Lukáš Doktor wrote:
> Dne 20.1.2015 v 19:01 Ademar Reis napsal(a):
> >On Tue, Jan 20, 2015 at 05:57:10PM +0100, Lukáš Doktor wrote:

<snip>

> >>
> >>[Matching nodes - startswith]
> >>
> >>There is also an opposite problem with the beginning. Usually we encourage
> >>people to use simple yaml files to multiplex tests (eg. the sleeptest
> >>multiplex:
> >>
> >>     short:
> >>         sleep_length: 0.5
> >>     medium:
> >>         sleep_length: 1
> >>     long:
> >>         sleep_length: 5
> >>     longest:
> >>         sleep_length: 10
> >>
> >>The tree is:
> >>
> >>--- short
> >>  |- medium
> >>  |-long
> >>  \-longest
> >>
> >>so the result leaves are:
> >>
> >>[/short, /medium, /long, /longest]
> >>
> >>So when writing test for this simple version, we'd ask for params.get('/',
> >>'sleep_length').
> >
> >Or just params.get("sleep_lenght")
> 
> The problem here is, from where the `sleep_length` is used. It can be on
> multiple places if you combine this simple multiplex file with other
> existing files...
> 
> >
> >>
> >>But what if someones want's more complicated version and he puts this into
> >>another branch, eg:
> >>
> >>     tests:
> >>         sleeptest:
> >>             by_length:
> >>                 short:
> >>                 medium:
> >>                 long:
> >>                 longest:
> >>
> >>When he develops the test, he'd use params.get('/tests/sleeptest/by_length',
> >>'sleep_length') to obtain the value from the correct namespace. This would
> >>might cause trouble when executing this test with the simple version (the
> >>issue is more serious as most of the time it'd work fine, but when the keys
> >>are duplicate, other value might win).
> >
> >The simple call "params.get("sleep_lenght") will still work if
> >the user has a small mux file.
> >
> >Now, when the user combines multiple mux files, then you need a
> >hierarchical mechanism for the variable resolution (more below).
> 
> The main issue is, that people might combine multiple yaml files and
> multiple attitudes. Again, this is here to demonstrate possible issues which
> might be solved by the predefined namespaces.
> 
> >
> >>
> >>This might be eliminated a bit by separating framework-related and
> >>test-related multiplexing.
> >
> >Or just make a hierarchical resolution when there are multiple
> >options. Avocado could have a few pre-defined namespaces and a
> >hierarchy like the following to resolve
> >"params.get("sleep_lenght"):
> >
> >  1. Local namespace: anything that doesn't belong to a
> >  pre-defined namespace.
> >  2. Pre-defined namespaces, whose order would be documented.
> >  Example:
> >    2.1 /config
> >    2.2 /plugins
> >    2.3 /environment
> >    2.4 /setup
> >    2.5 /tests
> >    [...]
> 
> Well, that's the solution I prepared only with more namespaces predefined.
> I'd rather start with few of them and each plugin can define it's own.
> Anyway I'd hate paths like '/plugins/virt/hw/image_format'...

Another approach is to allow the resolution order to be specified
somewhere else.

Either way, the deeper the user goes into the tree, the more
specific the variant name becomes with less change of colisions.
"/plugins/virt/hw/" is probably overkill, but if you tell users
you resolution starts at local namespace, then goes to
pre-defined ones like I exemplified above, in a specific and
documented order, then "virt/hw/image_format" should be perfectly
fine and safe to use.

> >>
> >>1) framework-related (plugin-related) should have defined structure so we
> >>can safely assume `/virt/hw/nic` defines each key only once and is used to
> >>obtain information about the current `nic`.
> >>2) test-related should be unstructured and should extend the `/test`
> >>namespace. That way we don't mix values from other namespaces (other plugins
> >>or complex structures defined by users) and we only query for
> >>`params.get('/test', key)` or if we know we defined substructres for
> >>`params.get('/test/our_subvariant', key)`.
> >>
> >>But let me know if you know of a better solution (params.get('/', key)
> >>returns all of the leafs including /hw/nic/rtl8139, /os/type/linux/Fedora/8,
> >>... so one can only guess what's returned.
> >
> >That's expected, as I discussed above: if both [...]/rtl8139 and
> >[...]/Fedora/8 have a key with the same name (say: "foobar"), a
> >simple call such as params.get("foobar") will need to follow a
> >resolution hierarchy to return the right key.
> >
> >We should not encourage this kind of usage, it's a bad practice.
> >
> 
> Sure, this was about the beauty of named multiplex domains. But I can see we
> don't share the same view on that (which is fine, we have different usage in
> mind)

I have nothing against the multiplex domains idea. But I must
confess I didn't put much though on it. I've read your original
e-mail a couple of times but I still don't get it. I'm missing
something.

<snip>

> Also I thought about another approach I previously rejected for the sake of
> systematic approach. We can say that top level is always !multiplex. That
> way all simple yaml files would work the same way. You'd easily extend the
> pre-defined namespaces (test, plugin, config, ...) without even knowing
> !multiplex exists.
> 
> On the other hand you'd still be able to create deeper named !multiplex
> groups as proposed by me and welcomed by Paolo.
> 
> So this simple yaml file:
> 
>     short:
>         sleep_length: 0.5
>     medium:
>         sleep_length: 1
>     long:
>         sleep_length: 5
>     longest:
>         sleep_length: 10
> 
> would work perfectly without the need for any tags.

But what happens if you include the file above into a larger
tree?

> 
> More complex:
> 
> os:
>     Windows:
>         2000:
>         xp:
>     Linux:
>         RHEL:
>             7:
>                 0:
>                 1:
> 
> again no changes required as it's multiplexed only at the top level.
> 
> 
> And our favorited:
> 
> hw:  !multiplex
>     nic:
>         rtl8139:
>         virtio_net:
>     smp:
>         up:
>         smp2:
>     drive_format:
>         ide:
>         scsi:
> 
> Requires !multiplex to emphasize nic/smp/drive_format can share the same
> values and we should query for their values using their names (/hw/$name).
> Also we specifically say /hw/{nic,smp,drive_format} are different namespaces
> we might use in tests to say it make sense to multiplex this file with
> different nic types. (and in another test it make sense to use different
> drive_formats, ...).
> 
> 
> To me this combines the simplicity while forcing people to thing about the
> namespaces in more complex examples. What do you think about it?
> 

I think you really want to use !multiplex instead of !nomux. :-)

I tend to like consistency and symmetry and to me applying
!multiplexer only to the first level of a tree breaks it.

Besides, I strongly believe in teaching users by usage, slowly,
specially on something as complex and confuse as the multiplexer.
I would prefer users to start with a one-level tree, then someday
they add a second level, then a third. By then, they realize
there's a problem: sometimes they don't want all leaves to be
merged.  That's when they go check the documentation or google
about it and discover there's this "!nomux" tag.

Another approach is to use !mux_group and !mux_choice, as
proposed by Paolo. Did you discard that idea?

Thanks.
   - Ademar

-- 
Ademar de Souza Reis Jr.
Red Hat

^[:wq!