[Avocado-devel] Multiplexer design recent discussions summary
Lukáš Doktor
ldoktor at redhat.com
Fri Apr 24 05:26:18 UTC 2015
Dne 23.4.2015 v 18:04 Ademar Reis napsal(a):
> Hi folks.
>
> Paolo spent some time looking at our multiplexer implementation
> and came to IRC to discuss some issues. Below is a summary of his
> feedback and ideas (Paolo, please correct me if I'm missing
> something):
Hello guys, please accept my apology for not being there.
>
> 0. Small issues:
>
> * Consistency in the usage of '/' versus '.' when dealing with
> multiplexer variants. For example, the get API and some
> examples use '.', while the output of avocado multiplex uses
> '/'. Consensus was to use '/' everywhere.
>
> * Branch names inside the yaml can contain a '/', resulting in
> some weird looking and confusing variants. Just create a
> variant with '/' in the middle and you'll see the odd-looking
> result in the list of variants.
>
> 1. !join behavior / design
yep, that's intended, it's a remain of the pre-namespace era. The new
API is not yet used.
>
> Consider the following example:
>
> os:
> distro:
> fedora:
> version:
> 20:
> something
> 21:
> flavor:
> workstation:
> cloud:
> rhel:
> version:
> 5:
> 6:
> arch:
> i386:
> arch_name: 'i686'
> x86_64:
> arch_name: 'x86_64'
>
> Without !join or anything, the current output will be like this:
>
> Variant 1: /os/distro/fedora/version/20
> /os/distro/fedora/flavor/workstation
> /os/distro/rhel/version/5
> /os/arch/i386
>
> Which is clearly not what we want given it includes both fedora
> and rhel variables in the same variant. We want something similar
> to:
>
> Variant 1: /os/distro/fedora/version/20
> /os/distro/fedora/flavor/workstation
> /os/arch/i386
>
> Variant n: /os/distro/rhel/version/5
> /os/arch/i386
>
> With the current implementation of !join, we can't get the
> desirable result and brainstorming a solution resulted in a
> messy-looking structure, full of "magic".
>
> So Paolo proposed something different, which is to split the
> multiplex structure into "namespaces" and "multiplexer choices",
> which should be easier to understand. It should be like this:
>
> (Lukas: this appears to be similar to what you proposed a while
> ago, using the !mux keyword)
>
> os:
> distro: !choice
> fedora:
> version: !choice
> 20:
> 21:
> flavor: !choice
> workstation:
> cloud:
> rhel:
> version: !choice
> 5:
> 6:
> arch: !choice
> i386:
> x86_64:
>
> Where:
> os is "just namespacing"
> os/distro is "multiplexing"
> os/distro/fedora is "just namespacing"
> os/distro/fedora.version is "multiplexing"
> os/distro/fedora.flavor is "multiplexing"
> os/distro/rhel is "just namespacing"
> os/distro/rhel.version is "multiplexing"
> os/arch is "multiplexing"
>
> This will generate what we want:
>
> Variant 1: /os/distro/fedora/version/20
> /os/distro/fedora/flavor/workstation
> /os/arch/i386
>
> Variant n: /os/distro/rhel/version/5
> /os/arch/i386
>
> Notice a tree without any 'choice' will generate only one
> variant, which will be the combination of all leafs. In this
> case:
>
> Variant 1: /os/distro/fedora/version/20
> /os/distro/fedora/version/21
> /os/distro/fedora/flavor/cloud
> /os/distro/fedora/flavor/workstation
> /os/distro/rhel/version/5
> /os/distro/rhel/version/6
> /os/arch/i386
> /os/arch/x86_64
I love you Paolo. You managed to talk others into what I proposed at the
beginning. Because what you proposed is actually `!mux`. It's ~10 times
faster in basic implementation and easier to understand by my brain.
Thank you, thank you, thank you and thank you.
>
>
> 2. Overriding variables
>
> Say you add /os/distro/fedora/version/1 to this tree. But Fedora
> 1, being so old, wants the value of /os/arch/i386/arch_name to be
> 'i586' instead of 'i686'. How to override it?
>
> Proposal: override it with a variable that contains a path:
>
> distro:
> fedora:
> version:
> 1:
> /os/arch/i386/arch_name: 'i586'
> 20:
> 21:
> ...
> arch:
> i386:
> arch_name: 'i686'
> x86_64:
>
> The path may contain a blob to allow more flexibility (say
> /*/arch/i386/arch_name: 'i586').
This is something which troubled me too. It's the equivalent of:
$tag:
...
in Autotest. I intended to investigate this after developing the
namespaces and cartesian_config2yaml translator (at least set of
recommendations). Because I wasn't sure if we can't do something like
that by different design.
So I think we might need it, but can't tell it for sure yet. I don't
want to tie myself to single idea, first I want to explore what's
possible with the current implementation.
>
> 3. We also brainstormed a few ideas about filtering and the
> implementation details of the algorithm. Here's a quote:
>
> <bonzini> with all the tree manipulation operators that you're adding
> <bonzini> is it still possible to separate the multiplexing in two phases:
> <bonzini> 1) building a tree
> <bonzini> 2) expanding the variants
> <bonzini> ?
> <bonzini> or are they intertwined?
> <bonzini> i'm asking because (2) is already NP-hard
> <bonzini> and if you intertwine it with something else, it becomes too unwieldy to optimize it
>
> <bonzini> in the ideal world, 1) would build an AST
> <bonzini> and 2) would be implemented as a visitor on that AST
> <bonzini> receives the AST, does NP-hard magic, gives back variants
>
> <bonzini> the abstract syntax tree is a Composite
> <bonzini> the visitor is, well, a Visitor :)
> <bonzini> http://en.wikipedia.org/wiki/Composite_pattern
>
> <bonzini> see my old e-mail (http://permalink.gmane.org/gmane.linux.kernel.autotest.virt-test/2872)
> <bonzini> see the "class MuxBuilder(object):" part
Hmm, I never seen the last email. I'll take a deeper look. Anyway apart
from the different node value overwrite it should be separable. When the
code is ready I'll investigate this last element and try to come up with
some designs and alternative solutions if possible.
>
> Thanks!
> - Ademar
>
Thank you, guys, for discussing whats still missing. It's still very
early and there is plenty of space for improvement. My idea is to try to
port virt-test's cartesian configs and see what are the options, what is
possible, what are the options and what kind of tree it leads to (to see
the direction we need to optimize for).
Kind regads,
Lukáš
PS: Thanks again for !choice, I hope it'll finally make it into upstream.
More information about the Avocado-devel
mailing list