[Avocado-devel] Avocado testplans
Lukáš Doktor
ldoktor at redhat.com
Fri Aug 3 11:55:13 UTC 2018
Well the execution is actually simpler than you think, `avocado run basic.yaml` should does the trick. Anyway the yaml-file is slightly corrupted and even then some things won't probably result in what you expect. Let me explain it a bit and provide better example)
Dne 3.8.2018 v 10:20 Sergey Bronnikov napsal(a):
> Lukas, thanks for explanation! But I still have a question.
>
> Imagine we have an YAML config with a list of tests and their parameters:
>
> ```yaml
> basic: !mux
>
> test_reference_resolver_class: avocado_vt.loader.VirtTestLoader
> timeout: 360
> test_reference_resolver_args: !!python/dict
> vt_extra_params:
> - nic_model=rtl8139
> - extra_params=" -cpu host"
> - drive_format=ahci
> - mem=2048
> vt-machine-type: q35
this should be "vt_machine_type"
>
> shutdown:
> test_reference: io-github-autotest-qemu.shutdown
Here `test_reference_resolver_extra:` is missing.
> vt_extra_params:
this should be `avocado_vt_extra_params:` (see `avocado_vt/loader.py`)
> - shutdown_method = shell
>
> reboot:
> test_reference: io-github-autotest-qemu.reboot
Here `test_reference_resolver_extra:` is missing.
> vt_extra_params:
> - param1=1
> - param2=2
>
> stress:
> test_reference: io-github-autotest-qemu.linux_stress
>
> iofuzz:
> test_reference: io-github-autotest-qemu.iofuzz
Here `test_reference_resolver_extra:` is missing.
> vt_extra_params:
> - param1=1
> - param2=2
>
> ```
>
> How to execute tests from this testplan?
as mentioned earlier, it's recognized as test format, therefor simply `avocado run basic.yaml`.
Also one thing, the vt discovery is quite slow (yeah we parse cartesian config on each invocation as it might have different values). You can put multiple `test_reference` into the same line and separate them by space in avocado-vt loader. But beware, it works as filters, therefor the discovered tests will be re-oredered accordingly to cartesian config (rather than how you put them there) and they'll occur only once. Therefor:
test_reference: shutdown boot boot
will generate:
avocado list basic.yaml
VT io-github-autotest-qemu.boot
VT io-github-autotest-qemu.shutdown
Anyway the speed of that is incomparably faster, which is why I use it when possible.
Regards,
Lukáš
PS: If this happens to work for you, would you please write an Avocado-vt documentation about this? I'd prefer someone external doing so because (apparently) I take many things as given... (you might as well improve the Avocado documentation, but looking at this I think the Avocado-vt one is missing more)
>
> $ avocado run -m basic.yaml
> No test references provided nor any other arguments resolved into tests.
> Please double check the executed command.
>
>
> Sergey
>
>
>
> чт, 2 авг. 2018 г. в 17:08, Lukáš Doktor <ldoktor at redhat.com>:
>
>> Dne 1.8.2018 v 13:19 Sergey Bronnikov napsal(a):
>>> Hello!
>>>
>>> I'm trying to find a way to organize my tests for running without
>>> specifying all of them in command-line. Imagine we have a set of
>> different
>>> tests: sleep.py, passtest.py and fail.py. To run all of them with
>> required
>>> parameters (aka variants) we may execute a command:
>>> avocado run sleep.py fail.py passtest.py -m basic.yaml
>>>
>>> Is there a way to run the same tests without explicit specifying them?
>>> For example: avocado run -m basic.yaml
>>>
>>> Sergey
>>>
>>
>> Dear Sergey,
>>
>> as a matter of fact, there are multiple ways.
>>
>> 1. filter-by-tag:
>> http://avocado-framework.readthedocs.io/en/latest/WritingTests.html#categorizing-tests
>>
>> you can tag your tests eg. `sanity`, `release`, ... and then run
>> `avocado run --filter-by-tag=sanity -- TEST_REPO` to only run tests tagged
>> `sanity` (note the granularity is per-class or even per-test-method.
>>
>> 2. yaml loader optional plugin:
>> http://avocado-framework.readthedocs.io/en/latest/optional_plugins/yaml_loader.html
>>
>> which is probably more what you're looking for, but currently only
>> works on python2 (but should get py3 support, hopefully, soon). It works
>> similarly to yaml_to_mux varianter plugin, but allows special keys like
>> `test_reference` to specify what test will be used or even
>> `test_reference_resolver_class` when your tests are not the default ones.
>> This is for example useful when you want to include multiple
>> external-runner tests with different external-runner. There are examples
>> in:
>> https://github.com/avocado-framework/avocado/tree/master/examples/yaml_to_mux_loader
>> but let me share my slighly advanced example here:
>>
>> ```yaml
>> # Generated s390x mux-suite
>> !mux
>> qemu_make: !mux
>> mux_suite_test_name_prefix: 'qemu_make/git '
>> check:
>> test_reference:
>> /home/jenkins/s390x/qemu-master/avocado-qemu-make-check
>> install:
>> test_reference:
>> /home/jenkins/s390x/qemu-master/avocado-qemu-make-install
>>
>> unit:
>> variants: !mux
>> test_reference_resolver_class: avocado.core.loader.ExternalLoader
>> timeout: 360
>> rpm-tcg:
>> mux_suite_test_name_prefix: 'unit/rpm/tcg '
>> test_reference_resolver_extra: !!python/dict
>> loader_options:
>> /home/jenkins/s390x/kvm-unit-test/avocado-runner-rpm-tcg
>> rpm-kvm:
>> mux_suite_test_name_prefix: 'unit/rpm/kvm '
>> test_reference_resolver_extra: !!python/dict
>> loader_options:
>> /home/jenkins/s390x/kvm-unit-test/avocado-runner-rpm-kvm
>> git-tcg:
>> mux_suite_test_name_prefix: 'unit/git/tcg '
>> test_reference_resolver_extra: !!python/dict
>> loader_options:
>> /home/jenkins/s390x/kvm-unit-test/avocado-runner-git-tcg
>> git-kvm:
>> mux_suite_test_name_prefix: 'unit/git/kvm '
>> test_reference_resolver_extra: !!python/dict
>> loader_options:
>> /home/jenkins/s390x/kvm-unit-test/avocado-runner-git-kvm
>> tests: !mux
>> cmm:
>> test_reference: cmm
>> diag10:
>> test_reference: diag10
>> emulator:
>> test_reference: emulator
>> gs:
>> test_reference: gs
>> iep:
>> test_reference: iep
>> intercept:
>> test_reference: intercept
>> pfmf:
>> test_reference: pfmf
>> selftest-setup:
>> test_reference: selftest-setup
>> sieve:
>> test_reference: sieve
>> skey:
>> test_reference: skey
>> sthyi:
>> test_reference: sthyi
>> vector:
>> test_reference: vector
>> functional: !mux
>> test_reference_resolver_class: avocado_vt.loader.VirtTestLoader
>> s390x-rpm-libvirt/dvd:
>> mux_suite_test_name_prefix: 'functional/rpm/libvirt/dvd '
>> test_reference:
>> unattended_install.cdrom.extra_cdrom_ks.default_install.aio_threads
>> io-github-autotest-qemu.boot remove_guest.without_disk
>> test_reference_resolver_args: !!python/dict
>> vt_qemu_bin: /usr/libexec/qemu-kvm
>> vt_dst_qemu_bin: /usr/libexec/qemu-kvm
>> vt_type: libvirt
>> vt_extra_params:
>> - automem=no
>> - url=$URL
>> - vga=none
>> s390x-rpm-libvirt/url:
>> mux_suite_test_name_prefix: 'functional/rpm/libvirt/url '
>> test_reference:
>> unattended_install.url.extra_cdrom_ks.default_install.aio_threads
>> io-github-autotest-qemu.boot remove_guest.without_disk
>> test_reference_resolver_args: !!python/dict
>> vt_qemu_bin: /usr/libexec/qemu-kvm
>> vt_dst_qemu_bin: /usr/libexec/qemu-kvm
>> vt_type: libvirt
>> vt_extra_params:
>> - automem=no
>> - url=$URL
>> - vga=none
>> s390x-rpm-dvd:
>> mux_suite_test_name_prefix: 'functional/rpm/dvd '
>> test_reference:
>> unattended_install.cdrom.extra_cdrom_ks.default_install.aio_threads boot
>> migrate.with_reboot.tcp migrate.with_reboot.exec.gzip_exec
>> migrate.default.tcp.with_post_copy
>> test_reference_resolver_args: !!python/dict
>> vt_qemu_bin: /usr/libexec/qemu-kvm
>> vt_dst_qemu_bin: /usr/libexec/qemu-kvm
>> vt_extra_params:
>> - automem=no
>> - url=$URL
>> - ping_pong=5
>> - vga=none
>> s390x-rpm-url:
>> mux_suite_test_name_prefix: 'functional/rpm/url '
>> test_reference:
>> unattended_install.url.extra_cdrom_ks.default_install.aio_threads boot
>> migrate.with_reboot.tcp migrate.with_reboot.exec.gzip_exec
>> migrate.default.tcp.with_post_copy
>> test_reference_resolver_args: !!python/dict
>> vt_qemu_bin: /usr/libexec/qemu-kvm
>> vt_dst_qemu_bin: /usr/libexec/qemu-kvm
>> vt_extra_params:
>> - automem=no
>> - url=$URL
>> - ping_pong=5
>> - vga=none
>> s390x-git-dvd:
>> mux_suite_test_name_prefix: 'functional/git/dvd '
>> test_reference:
>> unattended_install.cdrom.extra_cdrom_ks.default_install.aio_threads boot
>> migrate.with_reboot.tcp migrate.with_reboot.exec.gzip_exec
>> migrate.default.tcp.with_post_copy
>> test_reference_resolver_args: !!python/dict
>> vt_qemu_bin:
>> /home/jenkins/s390x/qemu-master/build/bin/qemu-system-s390x
>> vt_dst_qemu_bin:
>> /home/jenkins/s390x/qemu-master/build/bin/qemu-system-s390x
>> vt_extra_params:
>> - automem=no
>> - url=$URL
>> - ping_pong=5
>> - vga=none
>> s390x-git-url:
>> mux_suite_test_name_prefix: 'functional/git/url '
>> test_reference:
>> unattended_install.url.extra_cdrom_ks.default_install.aio_threads boot
>> migrate.with_reboot.tcp migrate.with_reboot.exec.gzip_exec
>> migrate.default.tcp.with_post_copy
>> test_reference_resolver_args: !!python/dict
>> vt_qemu_bin:
>> /home/jenkins/s390x/qemu-master/build/bin/qemu-system-s390x
>> vt_dst_qemu_bin:
>> /home/jenkins/s390x/qemu-master/build/bin/qemu-system-s390x
>> vt_extra_params:
>> - automem=no
>> - url=$URL
>> - ping_pong=5
>> - vga=none
>> ```
>>
>> (feel free to ask for details, but basically it uses SIMPLE tests,
>> EXTERNAL_RUNNER tests and then Avocado-vt tests. The test always receives
>> all the params so you can override the standard arguments. If you specify
>> "test_reference_resolver_args" it also updates the loader parameters, which
>> is what I use for Avocado-vt tests).
>>
>> Theoretically there is yet another way and that is to create directory
>> "sanity" and put symlinks to actual tests there, but the granularity there
>> would be only per-test-class, which usually does not suffice.
>>
>> Anyway, hopefully at least one method suits you, but feel free to
>> elaborate more, we can always improve.
>>
>> Kind regards,
>> Lukáš
>>
>>
>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 488 bytes
Desc: OpenPGP digital signature
URL: <http://listman.redhat.com/archives/avocado-devel/attachments/20180803/5522785e/attachment.sig>
More information about the Avocado-devel
mailing list