[Avocado-devel] Wrong behaviour with exclusion paths in variants

[Sergey Bronnikov]

вс, 2 сент. 2018 г. в 18:13, Lukáš Doktor <ldoktor at redhat.com>:

> Dne 28.8.2018 v 16:21 Sergey Bronnikov napsal(a):
> > Hello,
> >
> > I am stuck with strange behavior with using option --mux-filter-only.
> > I use following YAML config:
> >
> > nightly: !mux
> >     shutdown:
> >         test_reference: io-github-autotest-qemu.shutdown
> >
> >     boot:
> >         test_reference: io-github-autotest-qemu.boot
> >
> > versions: !mux
> >     test_reference_resolver_class: avocado_vt.loader.VirtTestLoader
> >     timeout: 720
> >     fedora-27-x86_64:
> >         mux_suite_test_name_prefix: 'fedora-27-x86_64 '
> >         test_reference_resolver_args: !!python/dict
> >             vt_guest_os: Fedora.27.x86_64
> >             vt_machine_type: q35
> >     jeos-27-x86_64:
> >         mux_suite_test_name_prefix: 'jeos-27-x86_64 '
> >         test_reference_resolver_args: !!python/dict
> >             vt_guest_os: JeOS.27.x86_64
> >             vt_machine_type: q35
> >
> > Avocado successfully generate variants with it:
> >
> > $ avocado variants -m os.yaml
> > Multiplex variants (4):
> > Variant shutdown-fedora-27-x86_64-f4a5:    /run/nightly/shutdown,
> > /run/versions/fedora-27-x86_64
> > Variant shutdown-jeos-27-x86_64-5c42:    /run/nightly/shutdown,
> > /run/versions/jeos-27-x86_64
> > Variant boot-fedora-27-x86_64-b296:    /run/nightly/boot,
> > /run/versions/fedora-27-x86_64
> > Variant boot-jeos-27-x86_64-cad3:    /run/nightly/boot,
> > /run/versions/jeos-27-x86_64
> > $
> >
> > Let's imagine we want to exclude jeos-27-x86_64 and run tests with Fedora
> > 27 only. Avocado shows variants with Fedora 27 guest OS only:
> >
> > $ avocado variants -m os.yaml --mux-filter-only
> /run/versions/jeos-27-x86_64
> > Multiplex variants (2):
> > Variant shutdown-jeos-27-x86_64-5c42:    /run/nightly/shutdown,
> > /run/versions/jeos-27-x86_64
> > Variant boot-jeos-27-x86_64-cad3:    /run/nightly/boot,
> > /run/versions/jeos-27-x86_64
> > $
> >
> > But avocado run _all_ tests when I'm trying to use same option with run
> > command:
> >
> > $ avocado run os.yaml --mux-filter-only /run/versions/jeos-27-x86_64
>
> Hello Sergey,
>
> the `--mux-filter-only` is used for `--mux-yaml` filtering (therefor test
> variants). To filter suite there are the `--mux-suite-only` and
> `--mux-suite-out` arguments.
>

Many thanks! That's what I want.


> Also one tip for your mux-suite, every variant gets a fresh loader,
> therefor discovering your suite results in 4x cartesian config parsing. For
> vt_loader it's possible to specify multiple test_references on a single
> line, so if you always want to execute "boot and shutdown", you can simply
> use:
>
> ```
> nightly: !mux
>     basic:
>         test_reference: "io-github-autotest-qemu.boot
> io-github-autotest-qemu.shutdown "
> ```
>
>
Yes, I remember you told me about it. But I use one test per one
test_reference for two reasons:
1. it is more convenient to read testplan with test per new line
2. some my test_references contains custom test parameters and it wouldn't
be possible with a single test_reference

Sergey

there are some limitations, for example the order of tests is not kept (as
> in background it only works as list of filters) and you can specify
> multiple same tests ("boot boot" results in "boot") and it's not possible
> with file-loader (or any other loader as far as I know) but especially with
> many vt-tests it can speed-up the discovery as this snippet results in 2
> cartesian config parsing (in my CI I'm combining those approaches).
>
> Regards,
> Lukáš
>
> > JOB ID     : 20890fa3194cf83a7b40b42a12cb16255ddb9550
> > JOB LOG    :
> > /home/sergeyb/avocado/job-results/job-2018-08-28T17.17-20890fa/job.log
> >  (1/4) fedora-27-x86_64 io-github-autotest-qemu.shutdown;-e630: PASS
> (45.06
> > s)
> >  (2/4) jeos-27-x86_64 io-github-autotest-qemu.shutdown;-e630: PASS
> (37.65 s)
> >  (3/4) fedora-27-x86_64 io-github-autotest-qemu.boot;-e630: PASS (46.93
> s)
> >  (4/4) jeos-27-x86_64 io-github-autotest-qemu.boot;-e630: PASS (39.78 s)
> > No handlers could be found for logger "avocado.test.stdout"
> > RESULTS    : PASS 4 | ERROR 0 | FAIL 0 | SKIP 0 | WARN 0 | INTERRUPT 0 |
> > CANCEL 0
> > JOB TIME   : 177.34 s
> > $
> >
> > Is it a bug?
> >
> > Sergey
> >
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/avocado-devel/attachments/20180903/393910d0/attachment.htm>