[Avocado-devel] Multiplexer design recent discussions summary
Lukáš Doktor
ldoktor at redhat.com
Wed Nov 25 11:45:13 UTC 2015
Hello guys,
let me resurrect this old thread...
Dne 23.4.2015 v 18:04 Ademar Reis napsal(a):
> Hi folks.
>
> Paolo spent some time looking at our multiplexer implementation
> and came to IRC to discuss some issues. Below is a summary of his
> feedback and ideas (Paolo, please correct me if I'm missing
> something):
>
...
>
> os:
> distro: !choice
> fedora:
> version: !choice
> 20:
> 21:
> flavor: !choice
> workstation:
> cloud:
> rhel:
> version: !choice
> 5:
> 6:
> arch: !choice
> i386:
> x86_64:
>
> Where:
> os is "just namespacing"
> os/distro is "multiplexing"
> os/distro/fedora is "just namespacing"
> os/distro/fedora.version is "multiplexing"
> os/distro/fedora.flavor is "multiplexing"
> os/distro/rhel is "just namespacing"
> os/distro/rhel.version is "multiplexing"
> os/arch is "multiplexing"
>
> This will generate what we want:
>
> Variant 1: /os/distro/fedora/version/20
> /os/distro/fedora/flavor/workstation
> /os/arch/i386
>
> Variant n: /os/distro/rhel/version/5
> /os/arch/i386
>
This is currently implemented, the tag was renamed to `!mux`
...
> 2. Overriding variables
>
> Say you add /os/distro/fedora/version/1 to this tree. But Fedora
> 1, being so old, wants the value of /os/arch/i386/arch_name to be
> 'i586' instead of 'i686'. How to override it?
>
> Proposal: override it with a variable that contains a path:
>
> distro:
> fedora:
> version:
> 1:
> /os/arch/i386/arch_name: 'i586'
> 20:
> 21:
> ...
> arch:
> i386:
> arch_name: 'i686'
> x86_64:
>
> The path may contain a blob to allow more flexibility (say
> /*/arch/i386/arch_name: 'i586').
This is something we most certainly want. I looked at it a bit and have
2 questions:
1. Format:
----------
We can say that keys can't contain '/' (or just start with '/' or '*' or
combination) and in case they do we consider them as variable overrides
where the last item is "key" (basically the example above). So the
format is:
$path/$key: $value
where path has to start with `/` or `*` and can contain multiple `/`
Additionally we should decide what `/*/` means. Does it means one level,
or multiple levels. For Jenkins it usually means one level and one has
to use `**` to make it recursive. I kind of like that idea...
Another way to specify where to inject variables is to use a special tag
for blocks, eg:
```
1:
!inject :
"**/arch/x86/32/**/name": "i386"
```
And we can go even further and separate the `path` and the `key`:
```
1:
!inject :
"**/arch/x86/32/**":
"name": "i386"
```
2. Behavior
-----------
Even bigger question is the behavior, when the target was expanded:
```
distro: !mux
fedora: !mux
version: !mux
1:
/os/arch/i386/arch_name: 'i586'
20:
21:
...
arch: !mux
i386: !mux
arch_name: 'i686'
486:
cpu: 486
arch_name: 'i386'
host:
cpu: host
x86_64:
```
We could either override the value in `/os/arch/i386` node, but that
value can be overridden (and is overridden in `486` leaf). The user
would have to use `/os/arch/i386/**/arch_name` to pass it to the leaves.
But I bet people would tend to forget about this and after years of
using a stable setup when they expand the `i386` branch of 2 variants,
it'd start behaving oddly (from time to time).
Therefor I think we should say that all values are injected into all
matching leaves by default, which would add another meaning to the last `/`
$path/$key: $value
where `/**/` is appended to the path
NOTE: Using this brings back the format of the output. For me it's
natural to use the last element of the path as `key` when
printing/accessing the values. I think we should reconsider doing it and
using it everywhere:
params['/hw/arch/arch_name'] => arch_name from "/hw/arch/**" path
params['arch/arch_name'] => arch name from "**/arch/**" path
print params => \
/hw/arch/i386/486/cpu: 486
/hw/arch/i386/486/arch_name: i386
nowadays it looks like this:
params.get("arch_name", "/hw/arch/*")
params.get("arch_name", "arch/*")
print params => \
/hw/arch/i386/486 cpu 486
/hw/arch/i386/486 arch_name i386
>
> 3. We also brainstormed a few ideas about filtering and the
> implementation details of the algorithm. Here's a quote:
>
> <bonzini> with all the tree manipulation operators that you're adding
> <bonzini> is it still possible to separate the multiplexing in two phases:
> <bonzini> 1) building a tree
> <bonzini> 2) expanding the variants
> <bonzini> ?
> <bonzini> or are they intertwined?
> <bonzini> i'm asking because (2) is already NP-hard
> <bonzini> and if you intertwine it with something else, it becomes too unwieldy to optimize it
>
> <bonzini> in the ideal world, 1) would build an AST
> <bonzini> and 2) would be implemented as a visitor on that AST
> <bonzini> receives the AST, does NP-hard magic, gives back variants
>
> <bonzini> the abstract syntax tree is a Composite
> <bonzini> the visitor is, well, a Visitor :)
> <bonzini> http://en.wikipedia.org/wiki/Composite_pattern
>
> <bonzini> see my old e-mail (http://permalink.gmane.org/gmane.linux.kernel.autotest.virt-test/2872)
> <bonzini> see the "class MuxBuilder(object):" part
>
I looked at it back then, but I was not able to understand it. Then
other tasks came and last week I tried to read it again, but I was not
any smarter. I don't say that maybe if I have one month exclusively to
study it that I won't be able to do something with it, but given the
time and other tasks I can't work on it. Still eventually when the
performance becomes important, we can open this again...
...
There is yet another chapter of this email:
Internal filters
================
Currently avocado supports only global/external filters, specified on
cmdline, modifying the tree by cutting it's branches. Internal filters
are suppose to do the same but only when the leaf containing the filter
is present in the variant:
```
os: !mux
linux:
windows:
!filter-out : /hw/cpu/ppc64
hw:
cpu: !mux
x86_64:
ppc64:
```
should result in:
1. linux + x86_64
2. linux + ppc64
3. windows + x86_64
and the windows + ppc64 should not be created (or at least not yielded).
This example was quite simple, but let's use more advanced example to
demonstrate something we have a different opinion on and we'd like to
get your feedback about:
```
!mux
machine: !mux
x86_64:
aarch64:
!filter-out : /os/fedora/variants
os: !mux
fedora:
variants: !mux
workstation:
server:
releases: !mux
21:
22:
gentoo:
```
If we stick to the external filters implementation, the
`/os/fedora/variants` should be simply cut out of the `aarch64` variant,
resulting in:
1. x86_64 + workstation + 21
2. x86_64 + workstation + 22
3. x86_64 + server + 21
4. x86_64 + server + 22
5. x86_64 + gentoo
6. aarch64 + 21
7. aarch64 + 22
8. aarch64 + gentoo
And the variants 6 and 7 would use default values instead of
/os/fedora/variants/* (the same way it works without multiplex file).
So the meaning of `!filter-out` is: "This leaf does not care about
values/combinations from given node(s)/branche(s)".
This does not make sense to me, because we loose the information, so I'm
proposing renaming the `--filter-only|out` to
`--mux-remove`|`--mux-keep`, which would modify the tree
(external/global filters). The internal filters should than filter (not
generate) values, which are marked as incompatible, so the results of
the tree above is:
1. x86_64 + workstation + 21
2. x86_64 + workstation + 22
3. x86_64 + server + 21
4. x86_64 + server + 22
5. x86_64 + gentoo
6. aarch64 + gentoo
and the meaning of the `!filter-out` would be: "This leaf is not
compatible with given node(s)/variant(s)".
Note that for leaf nodes the results are identical. The difference is
when we cut/filter mux nodes or nodes containing other mux(es).
To try it out you can use global filters to simulate the first approach
and https://github.com/avocado-framework/avocado/pull/612 to try the
filtering approach. (note it's terribly slow, I already know of ways to
improve the performance)
Kind regards,
Lukáš
More information about the Avocado-devel
mailing list