[Avocado-devel] Wrong behaviour with exclusion paths in variants

[Lukáš Doktor]

Dne 28.8.2018 v 16:21 Sergey Bronnikov napsal(a):
> Hello,
> 
> I am stuck with strange behavior with using option --mux-filter-only.
> I use following YAML config:
> 
> nightly: !mux
>     shutdown:
>         test_reference: io-github-autotest-qemu.shutdown
> 
>     boot:
>         test_reference: io-github-autotest-qemu.boot
> 
> versions: !mux
>     test_reference_resolver_class: avocado_vt.loader.VirtTestLoader
>     timeout: 720
>     fedora-27-x86_64:
>         mux_suite_test_name_prefix: 'fedora-27-x86_64 '
>         test_reference_resolver_args: !!python/dict
>             vt_guest_os: Fedora.27.x86_64
>             vt_machine_type: q35
>     jeos-27-x86_64:
>         mux_suite_test_name_prefix: 'jeos-27-x86_64 '
>         test_reference_resolver_args: !!python/dict
>             vt_guest_os: JeOS.27.x86_64
>             vt_machine_type: q35
> 
> Avocado successfully generate variants with it:
> 
> $ avocado variants -m os.yaml
> Multiplex variants (4):
> Variant shutdown-fedora-27-x86_64-f4a5:    /run/nightly/shutdown,
> /run/versions/fedora-27-x86_64
> Variant shutdown-jeos-27-x86_64-5c42:    /run/nightly/shutdown,
> /run/versions/jeos-27-x86_64
> Variant boot-fedora-27-x86_64-b296:    /run/nightly/boot,
> /run/versions/fedora-27-x86_64
> Variant boot-jeos-27-x86_64-cad3:    /run/nightly/boot,
> /run/versions/jeos-27-x86_64
> $
> 
> Let's imagine we want to exclude jeos-27-x86_64 and run tests with Fedora
> 27 only. Avocado shows variants with Fedora 27 guest OS only:
> 
> $ avocado variants -m os.yaml --mux-filter-only /run/versions/jeos-27-x86_64
> Multiplex variants (2):
> Variant shutdown-jeos-27-x86_64-5c42:    /run/nightly/shutdown,
> /run/versions/jeos-27-x86_64
> Variant boot-jeos-27-x86_64-cad3:    /run/nightly/boot,
> /run/versions/jeos-27-x86_64
> $
> 
> But avocado run _all_ tests when I'm trying to use same option with run
> command:
> 
> $ avocado run os.yaml --mux-filter-only /run/versions/jeos-27-x86_64

Hello Sergey,

the `--mux-filter-only` is used for `--mux-yaml` filtering (therefor test variants). To filter suite there are the `--mux-suite-only` and `--mux-suite-out` arguments.

Also one tip for your mux-suite, every variant gets a fresh loader, therefor discovering your suite results in 4x cartesian config parsing. For vt_loader it's possible to specify multiple test_references on a single line, so if you always want to execute "boot and shutdown", you can simply use:

```
nightly: !mux
    basic:
        test_reference: "io-github-autotest-qemu.boot io-github-autotest-qemu.shutdown "
```

there are some limitations, for example the order of tests is not kept (as in background it only works as list of filters) and you can specify multiple same tests ("boot boot" results in "boot") and it's not possible with file-loader (or any other loader as far as I know) but especially with many vt-tests it can speed-up the discovery as this snippet results in 2 cartesian config parsing (in my CI I'm combining those approaches).

Regards,
Lukáš

> JOB ID     : 20890fa3194cf83a7b40b42a12cb16255ddb9550
> JOB LOG    :
> /home/sergeyb/avocado/job-results/job-2018-08-28T17.17-20890fa/job.log
>  (1/4) fedora-27-x86_64 io-github-autotest-qemu.shutdown;-e630: PASS (45.06
> s)
>  (2/4) jeos-27-x86_64 io-github-autotest-qemu.shutdown;-e630: PASS (37.65 s)
>  (3/4) fedora-27-x86_64 io-github-autotest-qemu.boot;-e630: PASS (46.93 s)
>  (4/4) jeos-27-x86_64 io-github-autotest-qemu.boot;-e630: PASS (39.78 s)
> No handlers could be found for logger "avocado.test.stdout"
> RESULTS    : PASS 4 | ERROR 0 | FAIL 0 | SKIP 0 | WARN 0 | INTERRUPT 0 |
> CANCEL 0
> JOB TIME   : 177.34 s
> $
> 
> Is it a bug?
> 
> Sergey
> 


-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 488 bytes
Desc: OpenPGP digital signature
URL: <http://listman.redhat.com/archives/avocado-devel/attachments/20180902/f88a5e3d/attachment.sig>