[Avocado-devel] Multiplexer design recent discussions summary

[Lukáš Doktor]

Dne 25.11.2015 v 17:42 Paolo Bonzini napsal(a):
>
>
> On 25/11/2015 17:34, Lukáš Doktor wrote:
>> Dne 25.11.2015 v 17:15 Paolo Bonzini napsal(a):
>>>
>>>
>>> On 25/11/2015 16:39, Lukáš Doktor wrote:
>>>>
>>>>       --mux-mux $path[:true|false] -- to set the node as mux/nomux
>>>>       --mux-inject [$path:]key:value -- to set value (and create the
>>>> node)
>>>>       --mux-cut $path -- remove any node (opposite of --mux-inject)
>>>
>>> --mux-remove?
>> that's better :D
>
>>>>       --mux-keep $path -- mux-cut siblings
>>>
>>> What's the usecase?
>> The same as filter-out vs. filter-only. Just convenience method to say
>> this is the only node that should stay in the tree (eg. when you combine
>> multiple yaml files and you don't know what other sibblings are there)
>
> But what is it used for?  For mux nodes there are clear usecases.  I
> would like to see a real world example for keep and remove.
>
> However, remove is the fundamental primitive for all of
> remove/keep/out/only, so I'm good with having it.  keep, not so much.
>
Imagine:

```
# upstream.yaml
     hw:
         nic:
             e1000:
             virtio_net:
             rtl8139:

# downstream.yaml
     hw:
         nic:
             hyper_super_ultra_nic:
```

When you include booth, you test on e1000, virtio_net, rtl8139 and 
hyper_super_ultra_nic. But from time to time you might want to use only 
your `hyper_super_ultra_nic` or any other nic. You can't use remove, 
because you don't know what other nics are there.

Yes, the above example can be realized with `only`, because nics are 
direct children of a mux node. But imagine similar example, where this 
does not apply and where `keep` and `only` differs. `keep` is more 
generic, than `only`, therefor I think it should be implemented.

PS: I don't know any real world scenario for `remove` and `keep`. To me 
`out` and `only` are the only commands I need. But I understand that 
they are not 100% generic and we can't achieve everything with them, 
therefor I think we should support booth. The theoretical usecases I can 
come up with is when combining multiple yaml files. Then one might need 
to remove some parts.


>>>>       --mux-skip $path -- remove variants containing this path
>>>>       --mux-only $path -- mux-out siblings
>>>>
>>>> yaml:
>>>>
>>>>       !mux -- set the node as multiplexed
>>>>       !skip : $path -- ignore variants with this $path
>>>>       !only : $path -- !skip siblings
>
> !skip is more consistent with the command line.
>
>> I really don't. Would you please show me an example where you have
>> multiplex file and on a local scale (only for certain variant) you want
>> to remove some nodes, which are not direct children of mux node? So
>> something like:
>>
>> ```
>>      !mux
>>      machine: !mux
>>          x86_64:
>>          aarch64:
>>              !remove : /os/fedora/variants
>>      os: !mux
>>          fedora:
>>              variants: !mux
>>                  workstation:
>>                  server:
>>              releases: !mux
>>                  21:
>>                  22:
>>          gentoo:
>> ```
>>
>> The difference is that in this case you'll get values of
>> `/os/fedora/releases`, but you won't get values from
>> `/os/fedora/variants`. This just does not make sense to me, instead I'd
>> use `!skip` which would result in no fedora variants on `aarch64`, or
>> `!only : fedora/variants/server` to have only one variant of the fedora.
>> But having fedora, without values from `variants` would be confusing to me.
>
> I think if it makes sense to use it on the command line, it makes sense
> to have it in the config file.  I'd rarely use patching on the command
> line for example.  If tests can take a long time, but I occasionally
> want to change some details, I prefer to modify an on-disk file so that
> I know exactly what was the last thing I've run.
>
I use opposite approach. In Jenkins I use generic multiplex files and 
for each variant I just cut some parts. It's easier than copying a file 
from server to the client node. Even now the yaml tree is part of the 
results, so you can easily reproduce it (and hopefully soon we'll 
support `replay` to do it automatically).

Btw now when I think about this a bit more, we already support 
`!remove-node` during tree merge. It's a different usecase than internal 
filters, but what I mean is we could support:

     --mux-skip - remove variants containing this path
     --mux-only - use only variant containing this path

which can be used in yaml to do per-branch filtering:

     !skip
     !only

and for up->downstream patching we have:

     --mux-cut - remove any node
     --mux-keep - remove sibblings

and for extending yaml files we have tags which during the merge do the 
same (so it's not per branch, but downstream yaml file can remove 
variants of the previously existing upstream branch):

     !remove-node
     !keep-node

The difference is in usage. The `skip` and `only` tags would be used to 
filter the variant they are in. The `remove` and `keep` would be used to 
modify the final tree before multiplexation when merging multiple nodes:

```
# upstream.yaml
     hw:
         nic:
             e1000:
             virtio_net:
             rtl8139:

# downstream.yaml
     hw:
         nic:
             !keep-node : virtio_net
             hyper_super_ultra_nic:
                 !only : disk/virtio_blk
```
would result in:

```
     hw:
         nic:
             virtio_net:
             hyper_super_ultra_nic:
                 !only : disk/virtio_blk
```
where `hyper_super_ultra_nic` would be only used with `virtio_blk` disk 
and the setup would work even when someone extends the `upstream.yaml` 
without the need to modify the `downstream.yaml` or global filters.

What I'm trying say is there are different usages and `remove`/`keep` 
allows everything. I can live without them, but from time to time it 
might help someone to achieve what he wants. (which means I'm not 
against not implementing them. I included them because they are the 
original behavior of `--filter-only`/`--filter-out`, which is supported now.

Lukáš


> Paolo
>
>> As I said before, in most cases there is no difference between `!skip`
>> and `!remove`. But so far I haven't found any example where `!remove`
>> would produce better results, than `!skip` (I'm not talking about global
>> scale and --mux-remove, it's useful for on-cmdline-patching) so I don't
>> think we need to implement this tag for internal filters.