[Avocado-devel] Multiplexer design recent discussions summary

[Ademar Reis]

Hi folks.

Paolo spent some time looking at our multiplexer implementation
and came to IRC to discuss some issues. Below is a summary of his
feedback and ideas (Paolo, please correct me if I'm missing
something):

0. Small issues:

  * Consistency in the usage of '/' versus '.' when dealing with
    multiplexer variants. For example, the get API and some
    examples use '.', while the output of avocado multiplex uses
    '/'. Consensus was to use '/' everywhere.

  * Branch names inside the yaml can contain a '/', resulting in
    some weird looking and confusing variants. Just create a
    variant with '/' in the middle and you'll see the odd-looking
    result in the list of variants.

1. !join behavior / design

Consider the following example:

os:
    distro:
        fedora:
            version:
                20:
                    something
                21:
            flavor: 
                workstation:
                cloud:
        rhel:
            version:
                5:
                6:
    arch:
        i386:
           arch_name: 'i686'
        x86_64:
           arch_name: 'x86_64'

Without !join or anything, the current output will be like this:

Variant 1: /os/distro/fedora/version/20
           /os/distro/fedora/flavor/workstation
           /os/distro/rhel/version/5
           /os/arch/i386

Which is clearly not what we want given it includes both fedora
and rhel variables in the same variant. We want something similar
to:

Variant 1: /os/distro/fedora/version/20
           /os/distro/fedora/flavor/workstation
           /os/arch/i386

Variant n: /os/distro/rhel/version/5
           /os/arch/i386

With the current implementation of !join, we can't get the
desirable result and brainstorming a solution resulted in a
messy-looking structure, full of "magic".

So Paolo proposed something different, which is to split the
multiplex structure into "namespaces" and "multiplexer choices",
which should be easier to understand. It should be like this:

(Lukas: this appears to be similar to what you proposed a while
ago, using the !mux keyword)

os:
    distro: !choice
        fedora:
            version: !choice
                20:
                21:
            flavor: !choice
                workstation:
                cloud:
        rhel:
            version: !choice
                5:
                6:
    arch: !choice
        i386:
        x86_64: 

Where:
    os is "just namespacing"
    os/distro is "multiplexing"
    os/distro/fedora is "just namespacing"
    os/distro/fedora.version is "multiplexing"
    os/distro/fedora.flavor is "multiplexing"
    os/distro/rhel is "just namespacing"
    os/distro/rhel.version is "multiplexing"
    os/arch is "multiplexing"

This will generate what we want:

Variant 1: /os/distro/fedora/version/20
           /os/distro/fedora/flavor/workstation
           /os/arch/i386

Variant n: /os/distro/rhel/version/5
           /os/arch/i386

Notice a tree without any 'choice' will generate only one
variant, which will be the combination of all leafs. In this
case:

Variant 1: /os/distro/fedora/version/20
           /os/distro/fedora/version/21
           /os/distro/fedora/flavor/cloud
           /os/distro/fedora/flavor/workstation
           /os/distro/rhel/version/5
           /os/distro/rhel/version/6
           /os/arch/i386
           /os/arch/x86_64


2. Overriding variables

Say you add /os/distro/fedora/version/1 to this tree. But Fedora
1, being so old, wants the value of /os/arch/i386/arch_name to be
'i586' instead of 'i686'. How to override it?

Proposal: override it with a variable that contains a path:

    distro:
         fedora:
             version:
                 1:
                     /os/arch/i386/arch_name: 'i586'
                 20:
                 21:
                 ...
    arch:
        i386:
            arch_name: 'i686'
        x86_64: 

The path may contain a blob to allow more flexibility (say
/*/arch/i386/arch_name: 'i586').

3. We also brainstormed a few ideas about filtering and the
implementation details of the algorithm. Here's a quote:

<bonzini> with all the tree manipulation operators that you're adding
<bonzini> is it still possible to separate the multiplexing in two phases:
<bonzini> 1) building a tree
<bonzini> 2) expanding the variants
<bonzini> ?
<bonzini> or are they intertwined?
<bonzini> i'm asking because (2) is already NP-hard
<bonzini> and if you intertwine it with something else, it becomes too unwieldy to optimize it

<bonzini> in the ideal world, 1) would build an AST
<bonzini> and 2) would be implemented as a visitor on that AST
<bonzini> receives the AST, does NP-hard magic, gives back variants

<bonzini> the abstract syntax tree is a Composite
<bonzini> the visitor is, well, a Visitor :)
<bonzini> http://en.wikipedia.org/wiki/Composite_pattern

<bonzini> see my old e-mail (http://permalink.gmane.org/gmane.linux.kernel.autotest.virt-test/2872)
<bonzini> see the "class MuxBuilder(object):" part

Thanks!
   - Ademar

-- 
Ademar Reis
Red Hat

^[:wq!