[Avocado-devel] Multiplexer design recent discussions summary

[Lukáš Doktor]

Dne 25.11.2015 v 14:45 Paolo Bonzini napsal(a):
>
>
> On 25/11/2015 12:45, Lukáš Doktor wrote:
>> 1. Format:
>> ----------
>>
>> Additionally we should decide what `/*/` means. Does it means one level,
>> or multiple levels. For Jenkins it usually means one level and one has
>> to use `**` to make it recursive. I kind of like that idea...
>
> Me too.
>
>> Another way to specify where to inject variables is to use a special tag
>> for blocks, eg:
>>
>> ```
>>      1:
>>          !inject :
>>              "**/arch/x86/32/**/name": "i386"
>> ```
>>
>> And we can go even further and separate the `path` and the `key`:
>>
>> ```
>>      1:
>>          !inject :
>>              "**/arch/x86/32/**":
>>                  "name": "i386"
>> ```
>
> I like this too.
>
>>
>> 2. Behavior
>> -----------
>>
>> Even bigger question is the behavior, when the target was expanded:
>>
>> Therefor I think we should say that all values are injected into all
>> matching leaves by default, which would add another meaning to the last `/`
>
> I agree.
>
>
>>> 3. We also brainstormed a few ideas about filtering and the
>>> implementation details of the algorithm.
>>
>> I looked at it back then, but I was not able to understand it. Then
>> other tasks came and last week I tried to read it again, but I was not
>> any smarter. I don't say that maybe if I have one month exclusively to
>> study it that I won't be able to do something with it, but given the
>> time and other tasks I can't work on it. Still eventually when the
>> performance becomes important, we can open this again...
>
> A few months ago I tried implementing something similar and it was
> really slow (not practical).  I guess we're back to heuristics to order
> mux expansion, as in the old cartesian config parser.
>
Yep, they are actually not all that different...

>> Internal filters
>> ================
>>
>> Currently avocado supports only global/external filters, specified on
>> cmdline, modifying the tree by cutting it's branches. Internal filters
>> are suppose to do the same but only when the leaf containing the filter
>> is present in the variant. [...]
>>
>> This example was quite simple, but let's use more advanced example to
>> demonstrate something we have a different opinion on and we'd like to
>> get your feedback about:
>>
>> ```
>>      !mux
>>      machine: !mux
>>          x86_64:
>>          aarch64:
>>              !filter-out : /os/fedora/variants
>>      os: !mux
>>          fedora:
>>              variants: !mux
>>                  workstation:
>>                  server:
>>              releases: !mux
>>                  21:
>>                  22:
>>          gentoo:
>> ```
>>
>> If we stick to the external filters implementation, the
>> `/os/fedora/variants` should be simply cut out of the `aarch64` variant,
>> resulting in:
>>
>> 1. x86_64 + workstation + 21
>> 2. x86_64 + workstation + 22
>> 3. x86_64 + server + 21
>> 4. x86_64 + server + 22
>> 5. x86_64 + gentoo
>> 6. aarch64 + 21
>> 7. aarch64 + 22
>> 8. aarch64 + gentoo
>
> Makes sense. But it's surprising.
>
>> So the meaning of `!filter-out` is: "This leaf does not care about
>> values/combinations from given node(s)/branche(s)".
>
> Sounds right.
>
>> Note that for leaf nodes the results are identical. The difference is
>> when we cut/filter mux nodes or nodes containing other mux(es).
>
> Snipping, because I have a question then.  Should we just forbid
> filter-only/filter-out of !mux paths?  What is the use case?
If we choose to filter-out, than yes. That would be a solution (and I 
really like it).

If we choose to cut the node out, than we have another problem:

```
     os: !mux
         fedora:
             desktop:
                 xserver: xorg
             kernel:
                 /proc/sys/vm/swappiness: 30
         gentoo:
```

the result is:

1. desktop + kernel
2. gentoo

if I cut-out `/os/fedora/desktop`, the result is:

1. kernel
2. gentoo

if I filter-out `/os/fedora/desktop`, the result is:

1. gentoo

To me cutting makes sense on global scale as opposite to `--mux-inject` 
and it should coexists to `--filter-out`.

Filtering also makes sense and actually on a global scale can be 
directly translated to cutting:

     filter-out /os/fedora/desktop => cut-out /os/fedora

(opposite direction does not work - not all cut-outs can be translated 
to filter-outs)

>
>> Note that for leaf nodes the results are identical.
>
> Not just for leaf nodes, also for intermediate nodes. "!filter-out:
> /os/fedora" should work and give
>
> 1. x86_64 + gentoo
> 2. aarch64 + gentoo
>
> Right?
Yep.

>
> Paolo
>

Btw when thinking about optimization (which is not part of this RFC) I 
thought we should translate all `filter-only` into `filter-out` before 
generating the combinations. Handling filter-only is quite power-hungry, 
on the other hand retrieving the parent node and getting all children 
but one is very quick.... The workflow would be:

1. Parse tree out of yaml
2. Modify global filter-out to cut-out
3. Cut the tree (global filters)
4. Pass values and internal filters to leaves
5. Replace filter-only with filter-outs
6. Compute product and yield variants that are not filtered

Eventually in the future we can try implementing your AST-based 
solution, or Ademar's tree-walking solution. For now I'd go with the 
recursive product, which is implemented in C so it has some benefits 
over pure python solutions... (also we don't process the values, only 
generate variant and ignore it if it's filtered-out).

Anyway these were just my optimization ideas, not related to the 
high-level intention of this RFC.

Lukáš