[Avocado-devel] Avocado testplans

Thu Aug 2 14:08:01 UTC 2018

Dne 1.8.2018 v 13:19 Sergey Bronnikov napsal(a):
> Hello!
> 
> I'm trying to find a way to organize my tests for running without
> specifying all of them in command-line. Imagine we have a set of different
> tests: sleep.py, passtest.py and fail.py. To run all of them with required
> parameters (aka variants) we may execute a command:
> avocado run sleep.py fail.py passtest.py -m basic.yaml
> 
> Is there a way to run the same tests without explicit specifying them?
> For example: avocado run -m basic.yaml
> 
> Sergey
> 

Dear Sergey,

as a matter of fact, there are multiple ways.

1. filter-by-tag: http://avocado-framework.readthedocs.io/en/latest/WritingTests.html#categorizing-tests

    you can tag your tests eg. `sanity`, `release`, ... and then run `avocado run --filter-by-tag=sanity -- TEST_REPO` to only run tests tagged `sanity` (note the granularity is per-class or even per-test-method.

2. yaml loader optional plugin: http://avocado-framework.readthedocs.io/en/latest/optional_plugins/yaml_loader.html

    which is probably more what you're looking for, but currently only works on python2 (but should get py3 support, hopefully, soon). It works similarly to yaml_to_mux varianter plugin, but allows special keys like `test_reference` to specify what test will be used or even `test_reference_resolver_class` when your tests are not the default ones. This is for example useful when you want to include multiple external-runner tests with different external-runner. There are examples in: https://github.com/avocado-framework/avocado/tree/master/examples/yaml_to_mux_loader but let me share my slighly advanced example here:

```yaml
# Generated s390x mux-suite
!mux
qemu_make: !mux
    mux_suite_test_name_prefix: 'qemu_make/git '
    check:
        test_reference: /home/jenkins/s390x/qemu-master/avocado-qemu-make-check
    install:
        test_reference: /home/jenkins/s390x/qemu-master/avocado-qemu-make-install

unit:
    variants: !mux
        test_reference_resolver_class: avocado.core.loader.ExternalLoader
        timeout: 360
        rpm-tcg:
            mux_suite_test_name_prefix: 'unit/rpm/tcg '
            test_reference_resolver_extra: !!python/dict
                loader_options: /home/jenkins/s390x/kvm-unit-test/avocado-runner-rpm-tcg
        rpm-kvm:
            mux_suite_test_name_prefix: 'unit/rpm/kvm '
            test_reference_resolver_extra: !!python/dict
                loader_options: /home/jenkins/s390x/kvm-unit-test/avocado-runner-rpm-kvm
        git-tcg:
            mux_suite_test_name_prefix: 'unit/git/tcg '
            test_reference_resolver_extra: !!python/dict
                loader_options: /home/jenkins/s390x/kvm-unit-test/avocado-runner-git-tcg
        git-kvm:
            mux_suite_test_name_prefix: 'unit/git/kvm '
            test_reference_resolver_extra: !!python/dict
                loader_options: /home/jenkins/s390x/kvm-unit-test/avocado-runner-git-kvm
    tests: !mux
        cmm:
            test_reference: cmm
        diag10:
            test_reference: diag10
        emulator:
            test_reference: emulator
        gs:
            test_reference: gs
        iep:
            test_reference: iep
        intercept:
            test_reference: intercept
        pfmf:
            test_reference: pfmf
        selftest-setup:
            test_reference: selftest-setup
        sieve:
            test_reference: sieve
        skey:
            test_reference: skey
        sthyi:
            test_reference: sthyi
        vector:
            test_reference: vector
functional: !mux
    test_reference_resolver_class: avocado_vt.loader.VirtTestLoader
    s390x-rpm-libvirt/dvd:
        mux_suite_test_name_prefix: 'functional/rpm/libvirt/dvd '
        test_reference: unattended_install.cdrom.extra_cdrom_ks.default_install.aio_threads io-github-autotest-qemu.boot remove_guest.without_disk
        test_reference_resolver_args: !!python/dict
            vt_qemu_bin: /usr/libexec/qemu-kvm
            vt_dst_qemu_bin: /usr/libexec/qemu-kvm
            vt_type: libvirt
            vt_extra_params:
                - automem=no
                - url=$URL
                - vga=none
    s390x-rpm-libvirt/url:
        mux_suite_test_name_prefix: 'functional/rpm/libvirt/url '
        test_reference: unattended_install.url.extra_cdrom_ks.default_install.aio_threads io-github-autotest-qemu.boot remove_guest.without_disk
        test_reference_resolver_args: !!python/dict
            vt_qemu_bin: /usr/libexec/qemu-kvm
            vt_dst_qemu_bin: /usr/libexec/qemu-kvm
            vt_type: libvirt
            vt_extra_params:
                - automem=no
                - url=$URL
                - vga=none
    s390x-rpm-dvd:
        mux_suite_test_name_prefix: 'functional/rpm/dvd '
        test_reference: unattended_install.cdrom.extra_cdrom_ks.default_install.aio_threads boot migrate.with_reboot.tcp migrate.with_reboot.exec.gzip_exec migrate.default.tcp.with_post_copy
        test_reference_resolver_args: !!python/dict
            vt_qemu_bin: /usr/libexec/qemu-kvm
            vt_dst_qemu_bin: /usr/libexec/qemu-kvm
            vt_extra_params:
                - automem=no
                - url=$URL
                - ping_pong=5
                - vga=none
    s390x-rpm-url:
        mux_suite_test_name_prefix: 'functional/rpm/url '
        test_reference: unattended_install.url.extra_cdrom_ks.default_install.aio_threads boot migrate.with_reboot.tcp migrate.with_reboot.exec.gzip_exec migrate.default.tcp.with_post_copy
        test_reference_resolver_args: !!python/dict
            vt_qemu_bin: /usr/libexec/qemu-kvm
            vt_dst_qemu_bin: /usr/libexec/qemu-kvm
            vt_extra_params:
                - automem=no
                - url=$URL
                - ping_pong=5
                - vga=none
    s390x-git-dvd:
        mux_suite_test_name_prefix: 'functional/git/dvd '
        test_reference: unattended_install.cdrom.extra_cdrom_ks.default_install.aio_threads boot migrate.with_reboot.tcp migrate.with_reboot.exec.gzip_exec migrate.default.tcp.with_post_copy
        test_reference_resolver_args: !!python/dict
            vt_qemu_bin: /home/jenkins/s390x/qemu-master/build/bin/qemu-system-s390x
            vt_dst_qemu_bin: /home/jenkins/s390x/qemu-master/build/bin/qemu-system-s390x
            vt_extra_params:
                - automem=no
                - url=$URL
                - ping_pong=5
                - vga=none
    s390x-git-url:
        mux_suite_test_name_prefix: 'functional/git/url '
        test_reference: unattended_install.url.extra_cdrom_ks.default_install.aio_threads boot migrate.with_reboot.tcp migrate.with_reboot.exec.gzip_exec migrate.default.tcp.with_post_copy
        test_reference_resolver_args: !!python/dict
            vt_qemu_bin: /home/jenkins/s390x/qemu-master/build/bin/qemu-system-s390x
            vt_dst_qemu_bin: /home/jenkins/s390x/qemu-master/build/bin/qemu-system-s390x
            vt_extra_params:
                - automem=no
                - url=$URL
                - ping_pong=5
                - vga=none
```

(feel free to ask for details, but basically it uses SIMPLE tests, EXTERNAL_RUNNER tests and then Avocado-vt tests. The test always receives all the params so you can override the standard arguments. If you specify "test_reference_resolver_args" it also updates the loader parameters, which is what I use for Avocado-vt tests).

Theoretically there is yet another way and that is to create directory "sanity" and put symlinks to actual tests there, but the granularity there would be only per-test-class, which usually does not suffice.

Anyway, hopefully at least one method suits you, but feel free to elaborate more, we can always improve.

Kind regards,
Lukáš

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 488 bytes
Desc: OpenPGP digital signature
URL: <http://listman.redhat.com/archives/avocado-devel/attachments/20180802/8d296b3f/attachment.sig>