[Avocado-devel] Avocado testplans
Lukáš Doktor
ldoktor at redhat.com
Mon Aug 6 16:03:09 UTC 2018
Dne 6.8.2018 v 09:52 Sergey Bronnikov napsal(a):
> пт, 3 авг. 2018 г. в 14:55, Lukáš Doktor <ldoktor at redhat.com>:
>
>> Well the execution is actually simpler than you think, `avocado run
>> basic.yaml` should does the trick. Anyway the yaml-file is slightly
>> corrupted and even then some things won't probably result in what you
>> expect. Let me explain it a bit and provide better example)
>>
>> Dne 3.8.2018 v 10:20 Sergey Bronnikov napsal(a):
>>> Lukas, thanks for explanation! But I still have a question.
>>>
>>> Imagine we have an YAML config with a list of tests and their parameters:
>>>
>>> ```yaml
>>> basic: !mux
>>>
>>> test_reference_resolver_class: avocado_vt.loader.VirtTestLoader
>>> timeout: 360
>>> test_reference_resolver_args: !!python/dict
>>> vt_extra_params:
>>> - nic_model=rtl8139
>>> - extra_params=" -cpu host"
>>> - drive_format=ahci
>>> - mem=2048
>>> vt-machine-type: q35
>>
>> this should be "vt_machine_type"
>>
>>>
>>> shutdown:
>>> test_reference: io-github-autotest-qemu.shutdown
>>
>> Here `test_reference_resolver_extra:` is missing.
>>
>>> vt_extra_params:
>>
>> this should be `avocado_vt_extra_params:` (see `avocado_vt/loader.py`)
>>
>>> - shutdown_method = shell
>>>
>>> reboot:
>>> test_reference: io-github-autotest-qemu.reboot
>>
>> Here `test_reference_resolver_extra:` is missing.
>>
>>> vt_extra_params:
>>> - param1=1
>>> - param2=2
>>>
>>> stress:
>>> test_reference: io-github-autotest-qemu.linux_stress
>>>
>>> iofuzz:
>>> test_reference: io-github-autotest-qemu.iofuzz
>>
>> Here `test_reference_resolver_extra:` is missing.
>>
>>> vt_extra_params:
>>> - param1=1
>>> - param2=2
>>>
>>> ```
>>>
>>> How to execute tests from this testplan?
>>
>> as mentioned earlier, it's recognized as test format, therefor simply
>> `avocado run basic.yaml`.
>>
>> Also one thing, the vt discovery is quite slow (yeah we parse cartesian
>> config on each invocation as it might have different values). You can put
>> multiple `test_reference` into the same line and separate them by space in
>> avocado-vt loader. But beware, it works as filters, therefor the discovered
>> tests will be re-oredered accordingly to cartesian config (rather than how
>> you put them there) and they'll occur only once. Therefor:
>>
>> test_reference: shutdown boot boot
>>
>> will generate:
>>
>> avocado list basic.yaml
>> VT io-github-autotest-qemu.boot
>> VT io-github-autotest-qemu.shutdown
>>
>> Anyway the speed of that is incomparably faster, which is why I use it
>> when possible.
>>
>
>
> Finally with corrections I got a following testplan:
Dear Sergey,
actually you unveiled a bug; fix can be found here: https://github.com/avocado-framework/avocado-vt/pull/1696 (in my CI I always use `--vt-extra-params` so I forgot it might not be set. Sorry about this).
With this there is still one little remaining issue in your `basic.yaml`, see below...
>
> basic: !mux
> test_reference_resolver_class: avocado_vt.loader.VirtTestLoader
> timeout: 360
> test_reference_resolver_extra: !!python/dict
This is supposed to be "test_reference_resolver_args" (you're modifying the `self.args` of Avocado for the purpose of this loader. Note this only affects things inside the loader, so you can't for example change the job-results-dir or so...)
> vt_extra_params:
> - mem=2048
> - nic_model=rtl8139
> - drive_format=ahci
> - disk_bus=ahci
> - extra_params=" -cpu host"
> vt_machine_type: q35
>
> shutdown:
> test_reference: io-github-autotest-qemu.shutdown
> test_reference_resolver_extra: !!python/dict
> avocado_vt_extra_params:
> - shutdown_method = shell
>
> reboot:
> test_reference: io-github-autotest-qemu.reboot
> test_reference_resolver_extra: !!python/dict
> avocado_vt_extra_params:
> - param1 = 1
> - param2 = 2
>
> stress:
> test_reference: io-github-autotest-qemu.linux_stress
>
> iofuzz:
> test_reference: io-github-autotest-qemu.iofuzz
> test_reference_resolver_extra: !!python/dict
> avocado_vt_extra_params:
> - param1 = 1
> - param2 = 2
>
> ~$ avocado variants -m testplans/basic.yaml
> Multiplex variants (4):
> Variant shutdown-edec: /run/basic/shutdown
> Variant reboot-c8ba: /run/basic/reboot
> Variant stress-fd6a: /run/basic/stress
> Variant iofuzz-e8bc: /run/basic/iofuzz
> ~$
>
> But it is still not accepted by 'avocado {list, run}':
> ~$ avocado list testplans/basic.yaml
> Unable to resolve reference(s) 'testplans/basic.yaml' with plugins(s)
> 'file', 'vt', 'external', try running 'avocado list -V
> testplans/basic.yaml' to see the details.
>
> Do you have any idea why it may happen?
With that single change it works well for me (tried changing the "test_reference_resolver_args" and first test's "test_reference_resolver_extra" and it successfully propagated to test params).
Regards,
Lukáš
>> autotest==0.16.4
> avocado-framework==63.0
> avocado-framework-plugin-varianter-yaml-to-mux==63.0
> avocado-plugins-vt==63.0
>
> Sergey
>
>
>>
>> Regards,
>> Lukáš
>>
>> PS: If this happens to work for you, would you please write an Avocado-vt
>> documentation about this? I'd prefer someone external doing so because
>> (apparently) I take many things as given... (you might as well improve the
>> Avocado documentation, but looking at this I think the Avocado-vt one is
>> missing more)
>>
>>>
>>> $ avocado run -m basic.yaml
>>> No test references provided nor any other arguments resolved into tests.
>>> Please double check the executed command.
>>>
>>>
>>> Sergey
>>>
>>
>>
>
>
>>>
>>> чт, 2 авг. 2018 г. в 17:08, Lukáš Doktor <ldoktor at redhat.com>:
>>>
>>>> Dne 1.8.2018 v 13:19 Sergey Bronnikov napsal(a):
>>>>> Hello!
>>>>>
>>>>> I'm trying to find a way to organize my tests for running without
>>>>> specifying all of them in command-line. Imagine we have a set of
>>>> different
>>>>> tests: sleep.py, passtest.py and fail.py. To run all of them with
>>>> required
>>>>> parameters (aka variants) we may execute a command:
>>>>> avocado run sleep.py fail.py passtest.py -m basic.yaml
>>>>>
>>>>> Is there a way to run the same tests without explicit specifying them?
>>>>> For example: avocado run -m basic.yaml
>>>>>
>>>>> Sergey
>>>>>
>>>>
>>>> Dear Sergey,
>>>>
>>>> as a matter of fact, there are multiple ways.
>>>>
>>>> 1. filter-by-tag:
>>>>
>> http://avocado-framework.readthedocs.io/en/latest/WritingTests.html#categorizing-tests
>>>>
>>>> you can tag your tests eg. `sanity`, `release`, ... and then run
>>>> `avocado run --filter-by-tag=sanity -- TEST_REPO` to only run tests
>> tagged
>>>> `sanity` (note the granularity is per-class or even per-test-method.
>>>>
>>>> 2. yaml loader optional plugin:
>>>>
>> http://avocado-framework.readthedocs.io/en/latest/optional_plugins/yaml_loader.html
>>>>
>>>> which is probably more what you're looking for, but currently only
>>>> works on python2 (but should get py3 support, hopefully, soon). It works
>>>> similarly to yaml_to_mux varianter plugin, but allows special keys like
>>>> `test_reference` to specify what test will be used or even
>>>> `test_reference_resolver_class` when your tests are not the default
>> ones.
>>>> This is for example useful when you want to include multiple
>>>> external-runner tests with different external-runner. There are examples
>>>> in:
>>>>
>> https://github.com/avocado-framework/avocado/tree/master/examples/yaml_to_mux_loader
>>>> but let me share my slighly advanced example here:
>>>>
>>>> ```yaml
>>>> # Generated s390x mux-suite
>>>> !mux
>>>> qemu_make: !mux
>>>> mux_suite_test_name_prefix: 'qemu_make/git '
>>>> check:
>>>> test_reference:
>>>> /home/jenkins/s390x/qemu-master/avocado-qemu-make-check
>>>> install:
>>>> test_reference:
>>>> /home/jenkins/s390x/qemu-master/avocado-qemu-make-install
>>>>
>>>> unit:
>>>> variants: !mux
>>>> test_reference_resolver_class:
>> avocado.core.loader.ExternalLoader
>>>> timeout: 360
>>>> rpm-tcg:
>>>> mux_suite_test_name_prefix: 'unit/rpm/tcg '
>>>> test_reference_resolver_extra: !!python/dict
>>>> loader_options:
>>>> /home/jenkins/s390x/kvm-unit-test/avocado-runner-rpm-tcg
>>>> rpm-kvm:
>>>> mux_suite_test_name_prefix: 'unit/rpm/kvm '
>>>> test_reference_resolver_extra: !!python/dict
>>>> loader_options:
>>>> /home/jenkins/s390x/kvm-unit-test/avocado-runner-rpm-kvm
>>>> git-tcg:
>>>> mux_suite_test_name_prefix: 'unit/git/tcg '
>>>> test_reference_resolver_extra: !!python/dict
>>>> loader_options:
>>>> /home/jenkins/s390x/kvm-unit-test/avocado-runner-git-tcg
>>>> git-kvm:
>>>> mux_suite_test_name_prefix: 'unit/git/kvm '
>>>> test_reference_resolver_extra: !!python/dict
>>>> loader_options:
>>>> /home/jenkins/s390x/kvm-unit-test/avocado-runner-git-kvm
>>>> tests: !mux
>>>> cmm:
>>>> test_reference: cmm
>>>> diag10:
>>>> test_reference: diag10
>>>> emulator:
>>>> test_reference: emulator
>>>> gs:
>>>> test_reference: gs
>>>> iep:
>>>> test_reference: iep
>>>> intercept:
>>>> test_reference: intercept
>>>> pfmf:
>>>> test_reference: pfmf
>>>> selftest-setup:
>>>> test_reference: selftest-setup
>>>> sieve:
>>>> test_reference: sieve
>>>> skey:
>>>> test_reference: skey
>>>> sthyi:
>>>> test_reference: sthyi
>>>> vector:
>>>> test_reference: vector
>>>> functional: !mux
>>>> test_reference_resolver_class: avocado_vt.loader.VirtTestLoader
>>>> s390x-rpm-libvirt/dvd:
>>>> mux_suite_test_name_prefix: 'functional/rpm/libvirt/dvd '
>>>> test_reference:
>>>> unattended_install.cdrom.extra_cdrom_ks.default_install.aio_threads
>>>> io-github-autotest-qemu.boot remove_guest.without_disk
>>>> test_reference_resolver_args: !!python/dict
>>>> vt_qemu_bin: /usr/libexec/qemu-kvm
>>>> vt_dst_qemu_bin: /usr/libexec/qemu-kvm
>>>> vt_type: libvirt
>>>> vt_extra_params:
>>>> - automem=no
>>>> - url=$URL
>>>> - vga=none
>>>> s390x-rpm-libvirt/url:
>>>> mux_suite_test_name_prefix: 'functional/rpm/libvirt/url '
>>>> test_reference:
>>>> unattended_install.url.extra_cdrom_ks.default_install.aio_threads
>>>> io-github-autotest-qemu.boot remove_guest.without_disk
>>>> test_reference_resolver_args: !!python/dict
>>>> vt_qemu_bin: /usr/libexec/qemu-kvm
>>>> vt_dst_qemu_bin: /usr/libexec/qemu-kvm
>>>> vt_type: libvirt
>>>> vt_extra_params:
>>>> - automem=no
>>>> - url=$URL
>>>> - vga=none
>>>> s390x-rpm-dvd:
>>>> mux_suite_test_name_prefix: 'functional/rpm/dvd '
>>>> test_reference:
>>>> unattended_install.cdrom.extra_cdrom_ks.default_install.aio_threads boot
>>>> migrate.with_reboot.tcp migrate.with_reboot.exec.gzip_exec
>>>> migrate.default.tcp.with_post_copy
>>>> test_reference_resolver_args: !!python/dict
>>>> vt_qemu_bin: /usr/libexec/qemu-kvm
>>>> vt_dst_qemu_bin: /usr/libexec/qemu-kvm
>>>> vt_extra_params:
>>>> - automem=no
>>>> - url=$URL
>>>> - ping_pong=5
>>>> - vga=none
>>>> s390x-rpm-url:
>>>> mux_suite_test_name_prefix: 'functional/rpm/url '
>>>> test_reference:
>>>> unattended_install.url.extra_cdrom_ks.default_install.aio_threads boot
>>>> migrate.with_reboot.tcp migrate.with_reboot.exec.gzip_exec
>>>> migrate.default.tcp.with_post_copy
>>>> test_reference_resolver_args: !!python/dict
>>>> vt_qemu_bin: /usr/libexec/qemu-kvm
>>>> vt_dst_qemu_bin: /usr/libexec/qemu-kvm
>>>> vt_extra_params:
>>>> - automem=no
>>>> - url=$URL
>>>> - ping_pong=5
>>>> - vga=none
>>>> s390x-git-dvd:
>>>> mux_suite_test_name_prefix: 'functional/git/dvd '
>>>> test_reference:
>>>> unattended_install.cdrom.extra_cdrom_ks.default_install.aio_threads boot
>>>> migrate.with_reboot.tcp migrate.with_reboot.exec.gzip_exec
>>>> migrate.default.tcp.with_post_copy
>>>> test_reference_resolver_args: !!python/dict
>>>> vt_qemu_bin:
>>>> /home/jenkins/s390x/qemu-master/build/bin/qemu-system-s390x
>>>> vt_dst_qemu_bin:
>>>> /home/jenkins/s390x/qemu-master/build/bin/qemu-system-s390x
>>>> vt_extra_params:
>>>> - automem=no
>>>> - url=$URL
>>>> - ping_pong=5
>>>> - vga=none
>>>> s390x-git-url:
>>>> mux_suite_test_name_prefix: 'functional/git/url '
>>>> test_reference:
>>>> unattended_install.url.extra_cdrom_ks.default_install.aio_threads boot
>>>> migrate.with_reboot.tcp migrate.with_reboot.exec.gzip_exec
>>>> migrate.default.tcp.with_post_copy
>>>> test_reference_resolver_args: !!python/dict
>>>> vt_qemu_bin:
>>>> /home/jenkins/s390x/qemu-master/build/bin/qemu-system-s390x
>>>> vt_dst_qemu_bin:
>>>> /home/jenkins/s390x/qemu-master/build/bin/qemu-system-s390x
>>>> vt_extra_params:
>>>> - automem=no
>>>> - url=$URL
>>>> - ping_pong=5
>>>> - vga=none
>>>> ```
>>>>
>>>> (feel free to ask for details, but basically it uses SIMPLE tests,
>>>> EXTERNAL_RUNNER tests and then Avocado-vt tests. The test always
>> receives
>>>> all the params so you can override the standard arguments. If you
>> specify
>>>> "test_reference_resolver_args" it also updates the loader parameters,
>> which
>>>> is what I use for Avocado-vt tests).
>>>>
>>>> Theoretically there is yet another way and that is to create directory
>>>> "sanity" and put symlinks to actual tests there, but the granularity
>> there
>>>> would be only per-test-class, which usually does not suffice.
>>>>
>>>> Anyway, hopefully at least one method suits you, but feel free to
>>>> elaborate more, we can always improve.
>>>>
>>>> Kind regards,
>>>> Lukáš
>>>>
>>>>
>>>
>>
>>
>>
>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 488 bytes
Desc: OpenPGP digital signature
URL: <http://listman.redhat.com/archives/avocado-devel/attachments/20180806/e47f9bac/attachment.sig>
More information about the Avocado-devel
mailing list