[Avocado-devel] Avocado testplans
Lukáš Doktor
ldoktor at redhat.com
Tue Aug 7 09:59:42 UTC 2018
Dne 7.8.2018 v 11:16 Sergey Bronnikov napsal(a):
> пн, 6 авг. 2018 г. в 19:03, Lukáš Doktor <ldoktor at redhat.com>:
>
>> Dne 6.8.2018 v 09:52 Sergey Bronnikov napsal(a):
>>> пт, 3 авг. 2018 г. в 14:55, Lukáš Doktor <ldoktor at redhat.com>:
>>>
>>>> Well the execution is actually simpler than you think, `avocado run
>>>> basic.yaml` should does the trick. Anyway the yaml-file is slightly
>>>> corrupted and even then some things won't probably result in what you
>>>> expect. Let me explain it a bit and provide better example)
>>>>
>>>> Dne 3.8.2018 v 10:20 Sergey Bronnikov napsal(a):
>>>>> Lukas, thanks for explanation! But I still have a question.
>>>>>
>>>>> Imagine we have an YAML config with a list of tests and their
>> parameters:
>>>>>
>>>>> ```yaml
>>>>> basic: !mux
>>>>>
>>>>> test_reference_resolver_class: avocado_vt.loader.VirtTestLoader
>>>>> timeout: 360
>>>>> test_reference_resolver_args: !!python/dict
>>>>> vt_extra_params:
>>>>> - nic_model=rtl8139
>>>>> - extra_params=" -cpu host"
>>>>> - drive_format=ahci
>>>>> - mem=2048
>>>>> vt-machine-type: q35
>>>>
>>>> this should be "vt_machine_type"
>>>>
>>>>>
>>>>> shutdown:
>>>>> test_reference: io-github-autotest-qemu.shutdown
>>>>
>>>> Here `test_reference_resolver_extra:` is missing.
>>>>
>>>>> vt_extra_params:
>>>>
>>>> this should be `avocado_vt_extra_params:` (see `avocado_vt/loader.py`)
>>>>
>>>>> - shutdown_method = shell
>>>>>
>>>>> reboot:
>>>>> test_reference: io-github-autotest-qemu.reboot
>>>>
>>>> Here `test_reference_resolver_extra:` is missing.
>>>>
>>>>> vt_extra_params:
>>>>> - param1=1
>>>>> - param2=2
>>>>>
>>>>> stress:
>>>>> test_reference: io-github-autotest-qemu.linux_stress
>>>>>
>>>>> iofuzz:
>>>>> test_reference: io-github-autotest-qemu.iofuzz
>>>>
>>>> Here `test_reference_resolver_extra:` is missing.
>>>>
>>>>> vt_extra_params:
>>>>> - param1=1
>>>>> - param2=2
>>>>>
>>>>> ```
>>>>>
>>>>> How to execute tests from this testplan?
>>>>
>>>> as mentioned earlier, it's recognized as test format, therefor simply
>>>> `avocado run basic.yaml`.
>>>>
>>>> Also one thing, the vt discovery is quite slow (yeah we parse cartesian
>>>> config on each invocation as it might have different values). You can
>> put
>>>> multiple `test_reference` into the same line and separate them by space
>> in
>>>> avocado-vt loader. But beware, it works as filters, therefor the
>> discovered
>>>> tests will be re-oredered accordingly to cartesian config (rather than
>> how
>>>> you put them there) and they'll occur only once. Therefor:
>>>>
>>>> test_reference: shutdown boot boot
>>>>
>>>> will generate:
>>>>
>>>> avocado list basic.yaml
>>>> VT io-github-autotest-qemu.boot
>>>> VT io-github-autotest-qemu.shutdown
>>>>
>>>> Anyway the speed of that is incomparably faster, which is why I use it
>>>> when possible.
>>>>
>>>
>>>
>>> Finally with corrections I got a following testplan:
>>
>> Dear Sergey,
>>
>> actually you unveiled a bug; fix can be found here:
>> https://github.com/avocado-framework/avocado-vt/pull/1696 (in my CI I
>> always use `--vt-extra-params` so I forgot it might not be set. Sorry about
>> this).
>>
>>
> `avocado list plan.yaml` still fails. With --verbose it awlays shows error
> for any testplan:
> "Unexpected character / on pos 7. Special chars are allowed only in
> variable assignation statement: 'only ../avocado-vt/plans/lukas.yaml'
> (<string>:1)"
>
> Common error for any YAML testplan:
> ~$ avocado list --verbose ../avocado-vt/plans/lukas.yaml
> Type
> Test
> Tag(s)
> NOT_A_TEST ../avocado-vt/plans/lukas.yaml: Not an INSTRUMENTED
> (avocado.Test based), PyUNITTEST (unittest.TestCase based) or SIMPLE
> (executable) test
> !VT ../avocado-vt/plans/lukas.yaml: Unexpected character / on pos
> 7. Special chars are allowed only in variable assignation statement: 'only
> ../avocado-vt/plans/lukas.yaml' (<string>:1)
>
> TEST TYPES SUMMARY
> ...
>
> ~$ curl
> https://github.com/avocado-framework/avocado/blob/master/examples/yaml_to_mux_loader/basic.yaml
I assume you used `curl https://raw.githubusercontent.com/avocado-framework/avocado/master/examples/yaml_to_mux_loader/basic.yaml`
>> ../avocado-vt/plans/basic.yaml
> ~$ avocado list --verbose ../avocado-vt/plans/basic.yaml
> Type
> Test
> Tag(s)
> NOT_A_TEST ../avocado-vt/plans/basic.yaml: Not an INSTRUMENTED
> (avocado.Test based), PyUNITTEST (unittest.TestCase based) or SIMPLE
> (executable) test
> !VT ../avocado-vt/plans/basic-plain.yaml: Unexpected character / on
> pos 7. Special chars are allowed only in variable assignation statement:
> 'only ../avocado-vt/plans/basic-plain.yaml' (<string>:1)
>
This is correct, it's NOT_A_TEST and !VT. Anyway there should be the "loader_yaml" installed which should understand that format and discover the tests defined by the varianter. What is the output of:
$ avocado list --loaders help
Unknown loader 'help'. Available plugins are:
glib: glib.GLIB
golang: golang.GOLANG
yaml_testsuite:
robot: robot.ROBOT
vt: vt.VT
file: file.SIMPLE, file.INSTRUMENTED, file.PyUNITTEST
external: external.EXTERNAL
If the "yaml_testsuite is not there you need to install it (cd optional_plugins/loader_yaml && python setup.py install). If you use `make link` it should be there but `make install` won't install the optional plugins.
$ avocado list --verbose examples/mux-lukas.yaml
Type Test Tag(s)
NOT_A_TEST examples/mux-lukas.yaml: Not an INSTRUMENTED (avocado.Test based), PyUNITTEST (unittest.TestCase based) or SIMPLE (executable) test
!GLIB examples/mux-lukas.yaml: No GLib-like tests found
!GOLANG examples/mux-lukas.yaml: Go binary not found.
INSTRUMENTED passtest.py:PassTest.test fast
INSTRUMENTED sleeptest.py:SleepTest.test
INSTRUMENTED failtest.py:FailTest.test failure_expected
INSTRUMENTED failtest.py:FailTest.test failure_expected
INSTRUMENTED failtest.py:FailTest.test failure_expected
!ROBOT examples/mux-lukas.yaml: Unsupported file format 'yaml'.
!VT examples/mux-lukas.yaml: Unexpected character / on pos 13. Special chars are allowed only in variable assignation statement: 'only examples/mux-lukas.yaml' (<string>:1)
$ avocado list examples/mux-lukas.yaml
INSTRUMENTED passtest.py:PassTest.test
INSTRUMENTED sleeptest.py:SleepTest.test
INSTRUMENTED failtest.py:FailTest.test
INSTRUMENTED failtest.py:FailTest.test
INSTRUMENTED failtest.py:FailTest.test
Hopefully it's just the missing plugin,
Lukáš
> TEST TYPES SUMMARY
> ==================
> !VT: 1
> ACCESS_DENIED: 0
> BROKEN_SYMLINK: 0
> EXTERNAL: 0
> INSTRUMENTED: 0
> MISSING: 0
> NOT_A_TEST: 1
> PYUNITTEST: 0
> SIMPLE: 0
> VT: 0
>
> Sergey
>
>
>> With this there is still one little remaining issue in your `basic.yaml`,
>> see below...
>>
>>>
>>> basic: !mux
>>> test_reference_resolver_class: avocado_vt.loader.VirtTestLoader
>>> timeout: 360
>>> test_reference_resolver_extra: !!python/dict
>>
>> This is supposed to be "test_reference_resolver_args" (you're modifying
>> the `self.args` of Avocado for the purpose of this loader. Note this only
>> affects things inside the loader, so you can't for example change the
>> job-results-dir or so...)
>>
>>> vt_extra_params:
>>> - mem=2048
>>> - nic_model=rtl8139
>>> - drive_format=ahci
>>> - disk_bus=ahci
>>> - extra_params=" -cpu host"
>>> vt_machine_type: q35
>>>
>>> shutdown:
>>> test_reference: io-github-autotest-qemu.shutdown
>>> test_reference_resolver_extra: !!python/dict
>>> avocado_vt_extra_params:
>>> - shutdown_method = shell
>>>
>>> reboot:
>>> test_reference: io-github-autotest-qemu.reboot
>>> test_reference_resolver_extra: !!python/dict
>>> avocado_vt_extra_params:
>>> - param1 = 1
>>> - param2 = 2
>>>
>>> stress:
>>> test_reference: io-github-autotest-qemu.linux_stress
>>>
>>> iofuzz:
>>> test_reference: io-github-autotest-qemu.iofuzz
>>> test_reference_resolver_extra: !!python/dict
>>> avocado_vt_extra_params:
>>> - param1 = 1
>>> - param2 = 2
>>>
>>> ~$ avocado variants -m testplans/basic.yaml
>>> Multiplex variants (4):
>>> Variant shutdown-edec: /run/basic/shutdown
>>> Variant reboot-c8ba: /run/basic/reboot
>>> Variant stress-fd6a: /run/basic/stress
>>> Variant iofuzz-e8bc: /run/basic/iofuzz
>>> ~$
>>>
>>> But it is still not accepted by 'avocado {list, run}':
>>> ~$ avocado list testplans/basic.yaml
>>> Unable to resolve reference(s) 'testplans/basic.yaml' with plugins(s)
>>> 'file', 'vt', 'external', try running 'avocado list -V
>>> testplans/basic.yaml' to see the details.
>>>
>>> Do you have any idea why it may happen?
>>
>> With that single change it works well for me (tried changing the
>> "test_reference_resolver_args" and first test's
>> "test_reference_resolver_extra" and it successfully propagated to test
>> params).
>>
>
>
>
>> Regards,
>> Lukáš
>>
>>>> autotest==0.16.4
>>> avocado-framework==63.0
>>> avocado-framework-plugin-varianter-yaml-to-mux==63.0
>>> avocado-plugins-vt==63.0
>>>
>>> Sergey
>>>
>>>
>>>>
>>>> Regards,
>>>> Lukáš
>>>>
>>>> PS: If this happens to work for you, would you please write an
>> Avocado-vt
>>>> documentation about this? I'd prefer someone external doing so because
>>>> (apparently) I take many things as given... (you might as well improve
>> the
>>>> Avocado documentation, but looking at this I think the Avocado-vt one is
>>>> missing more)
>>>>
>>>>>
>>>>> $ avocado run -m basic.yaml
>>>>> No test references provided nor any other arguments resolved into
>> tests.
>>>>> Please double check the executed command.
>>>>>
>>>>>
>>>>> Sergey
>>>>>
>>>>
>>>>
>>>
>>>
>>>>>
>>>>> чт, 2 авг. 2018 г. в 17:08, Lukáš Doktor <ldoktor at redhat.com>:
>>>>>
>>>>>> Dne 1.8.2018 v 13:19 Sergey Bronnikov napsal(a):
>>>>>>> Hello!
>>>>>>>
>>>>>>> I'm trying to find a way to organize my tests for running without
>>>>>>> specifying all of them in command-line. Imagine we have a set of
>>>>>> different
>>>>>>> tests: sleep.py, passtest.py and fail.py. To run all of them with
>>>>>> required
>>>>>>> parameters (aka variants) we may execute a command:
>>>>>>> avocado run sleep.py fail.py passtest.py -m basic.yaml
>>>>>>>
>>>>>>> Is there a way to run the same tests without explicit specifying
>> them?
>>>>>>> For example: avocado run -m basic.yaml
>>>>>>>
>>>>>>> Sergey
>>>>>>>
>>>>>>
>>>>>> Dear Sergey,
>>>>>>
>>>>>> as a matter of fact, there are multiple ways.
>>>>>>
>>>>>> 1. filter-by-tag:
>>>>>>
>>>>
>> http://avocado-framework.readthedocs.io/en/latest/WritingTests.html#categorizing-tests
>>>>>>
>>>>>> you can tag your tests eg. `sanity`, `release`, ... and then run
>>>>>> `avocado run --filter-by-tag=sanity -- TEST_REPO` to only run tests
>>>> tagged
>>>>>> `sanity` (note the granularity is per-class or even per-test-method.
>>>>>>
>>>>>> 2. yaml loader optional plugin:
>>>>>>
>>>>
>> http://avocado-framework.readthedocs.io/en/latest/optional_plugins/yaml_loader.html
>>>>>>
>>>>>> which is probably more what you're looking for, but currently only
>>>>>> works on python2 (but should get py3 support, hopefully, soon). It
>> works
>>>>>> similarly to yaml_to_mux varianter plugin, but allows special keys
>> like
>>>>>> `test_reference` to specify what test will be used or even
>>>>>> `test_reference_resolver_class` when your tests are not the default
>>>> ones.
>>>>>> This is for example useful when you want to include multiple
>>>>>> external-runner tests with different external-runner. There are
>> examples
>>>>>> in:
>>>>>>
>>>>
>> https://github.com/avocado-framework/avocado/tree/master/examples/yaml_to_mux_loader
>>>>>> but let me share my slighly advanced example here:
>>>>>>
>>>>>> ```yaml
>>>>>> # Generated s390x mux-suite
>>>>>> !mux
>>>>>> qemu_make: !mux
>>>>>> mux_suite_test_name_prefix: 'qemu_make/git '
>>>>>> check:
>>>>>> test_reference:
>>>>>> /home/jenkins/s390x/qemu-master/avocado-qemu-make-check
>>>>>> install:
>>>>>> test_reference:
>>>>>> /home/jenkins/s390x/qemu-master/avocado-qemu-make-install
>>>>>>
>>>>>> unit:
>>>>>> variants: !mux
>>>>>> test_reference_resolver_class:
>>>> avocado.core.loader.ExternalLoader
>>>>>> timeout: 360
>>>>>> rpm-tcg:
>>>>>> mux_suite_test_name_prefix: 'unit/rpm/tcg '
>>>>>> test_reference_resolver_extra: !!python/dict
>>>>>> loader_options:
>>>>>> /home/jenkins/s390x/kvm-unit-test/avocado-runner-rpm-tcg
>>>>>> rpm-kvm:
>>>>>> mux_suite_test_name_prefix: 'unit/rpm/kvm '
>>>>>> test_reference_resolver_extra: !!python/dict
>>>>>> loader_options:
>>>>>> /home/jenkins/s390x/kvm-unit-test/avocado-runner-rpm-kvm
>>>>>> git-tcg:
>>>>>> mux_suite_test_name_prefix: 'unit/git/tcg '
>>>>>> test_reference_resolver_extra: !!python/dict
>>>>>> loader_options:
>>>>>> /home/jenkins/s390x/kvm-unit-test/avocado-runner-git-tcg
>>>>>> git-kvm:
>>>>>> mux_suite_test_name_prefix: 'unit/git/kvm '
>>>>>> test_reference_resolver_extra: !!python/dict
>>>>>> loader_options:
>>>>>> /home/jenkins/s390x/kvm-unit-test/avocado-runner-git-kvm
>>>>>> tests: !mux
>>>>>> cmm:
>>>>>> test_reference: cmm
>>>>>> diag10:
>>>>>> test_reference: diag10
>>>>>> emulator:
>>>>>> test_reference: emulator
>>>>>> gs:
>>>>>> test_reference: gs
>>>>>> iep:
>>>>>> test_reference: iep
>>>>>> intercept:
>>>>>> test_reference: intercept
>>>>>> pfmf:
>>>>>> test_reference: pfmf
>>>>>> selftest-setup:
>>>>>> test_reference: selftest-setup
>>>>>> sieve:
>>>>>> test_reference: sieve
>>>>>> skey:
>>>>>> test_reference: skey
>>>>>> sthyi:
>>>>>> test_reference: sthyi
>>>>>> vector:
>>>>>> test_reference: vector
>>>>>> functional: !mux
>>>>>> test_reference_resolver_class: avocado_vt.loader.VirtTestLoader
>>>>>> s390x-rpm-libvirt/dvd:
>>>>>> mux_suite_test_name_prefix: 'functional/rpm/libvirt/dvd '
>>>>>> test_reference:
>>>>>> unattended_install.cdrom.extra_cdrom_ks.default_install.aio_threads
>>>>>> io-github-autotest-qemu.boot remove_guest.without_disk
>>>>>> test_reference_resolver_args: !!python/dict
>>>>>> vt_qemu_bin: /usr/libexec/qemu-kvm
>>>>>> vt_dst_qemu_bin: /usr/libexec/qemu-kvm
>>>>>> vt_type: libvirt
>>>>>> vt_extra_params:
>>>>>> - automem=no
>>>>>> - url=$URL
>>>>>> - vga=none
>>>>>> s390x-rpm-libvirt/url:
>>>>>> mux_suite_test_name_prefix: 'functional/rpm/libvirt/url '
>>>>>> test_reference:
>>>>>> unattended_install.url.extra_cdrom_ks.default_install.aio_threads
>>>>>> io-github-autotest-qemu.boot remove_guest.without_disk
>>>>>> test_reference_resolver_args: !!python/dict
>>>>>> vt_qemu_bin: /usr/libexec/qemu-kvm
>>>>>> vt_dst_qemu_bin: /usr/libexec/qemu-kvm
>>>>>> vt_type: libvirt
>>>>>> vt_extra_params:
>>>>>> - automem=no
>>>>>> - url=$URL
>>>>>> - vga=none
>>>>>> s390x-rpm-dvd:
>>>>>> mux_suite_test_name_prefix: 'functional/rpm/dvd '
>>>>>> test_reference:
>>>>>> unattended_install.cdrom.extra_cdrom_ks.default_install.aio_threads
>> boot
>>>>>> migrate.with_reboot.tcp migrate.with_reboot.exec.gzip_exec
>>>>>> migrate.default.tcp.with_post_copy
>>>>>> test_reference_resolver_args: !!python/dict
>>>>>> vt_qemu_bin: /usr/libexec/qemu-kvm
>>>>>> vt_dst_qemu_bin: /usr/libexec/qemu-kvm
>>>>>> vt_extra_params:
>>>>>> - automem=no
>>>>>> - url=$URL
>>>>>> - ping_pong=5
>>>>>> - vga=none
>>>>>> s390x-rpm-url:
>>>>>> mux_suite_test_name_prefix: 'functional/rpm/url '
>>>>>> test_reference:
>>>>>> unattended_install.url.extra_cdrom_ks.default_install.aio_threads boot
>>>>>> migrate.with_reboot.tcp migrate.with_reboot.exec.gzip_exec
>>>>>> migrate.default.tcp.with_post_copy
>>>>>> test_reference_resolver_args: !!python/dict
>>>>>> vt_qemu_bin: /usr/libexec/qemu-kvm
>>>>>> vt_dst_qemu_bin: /usr/libexec/qemu-kvm
>>>>>> vt_extra_params:
>>>>>> - automem=no
>>>>>> - url=$URL
>>>>>> - ping_pong=5
>>>>>> - vga=none
>>>>>> s390x-git-dvd:
>>>>>> mux_suite_test_name_prefix: 'functional/git/dvd '
>>>>>> test_reference:
>>>>>> unattended_install.cdrom.extra_cdrom_ks.default_install.aio_threads
>> boot
>>>>>> migrate.with_reboot.tcp migrate.with_reboot.exec.gzip_exec
>>>>>> migrate.default.tcp.with_post_copy
>>>>>> test_reference_resolver_args: !!python/dict
>>>>>> vt_qemu_bin:
>>>>>> /home/jenkins/s390x/qemu-master/build/bin/qemu-system-s390x
>>>>>> vt_dst_qemu_bin:
>>>>>> /home/jenkins/s390x/qemu-master/build/bin/qemu-system-s390x
>>>>>> vt_extra_params:
>>>>>> - automem=no
>>>>>> - url=$URL
>>>>>> - ping_pong=5
>>>>>> - vga=none
>>>>>> s390x-git-url:
>>>>>> mux_suite_test_name_prefix: 'functional/git/url '
>>>>>> test_reference:
>>>>>> unattended_install.url.extra_cdrom_ks.default_install.aio_threads boot
>>>>>> migrate.with_reboot.tcp migrate.with_reboot.exec.gzip_exec
>>>>>> migrate.default.tcp.with_post_copy
>>>>>> test_reference_resolver_args: !!python/dict
>>>>>> vt_qemu_bin:
>>>>>> /home/jenkins/s390x/qemu-master/build/bin/qemu-system-s390x
>>>>>> vt_dst_qemu_bin:
>>>>>> /home/jenkins/s390x/qemu-master/build/bin/qemu-system-s390x
>>>>>> vt_extra_params:
>>>>>> - automem=no
>>>>>> - url=$URL
>>>>>> - ping_pong=5
>>>>>> - vga=none
>>>>>> ```
>>>>>>
>>>>>> (feel free to ask for details, but basically it uses SIMPLE tests,
>>>>>> EXTERNAL_RUNNER tests and then Avocado-vt tests. The test always
>>>> receives
>>>>>> all the params so you can override the standard arguments. If you
>>>> specify
>>>>>> "test_reference_resolver_args" it also updates the loader parameters,
>>>> which
>>>>>> is what I use for Avocado-vt tests).
>>>>>>
>>>>>> Theoretically there is yet another way and that is to create directory
>>>>>> "sanity" and put symlinks to actual tests there, but the granularity
>>>> there
>>>>>> would be only per-test-class, which usually does not suffice.
>>>>>>
>>>>>> Anyway, hopefully at least one method suits you, but feel free to
>>>>>> elaborate more, we can always improve.
>>>>>>
>>>>>> Kind regards,
>>>>>> Lukáš
>>>>>>
>>>>>>
>>>>>
>>>>
>>>>
>>>>
>>>
>>
>>
>>
>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 488 bytes
Desc: OpenPGP digital signature
URL: <http://listman.redhat.com/archives/avocado-devel/attachments/20180807/fa6e33f1/attachment.sig>
More information about the Avocado-devel
mailing list