[Avocado-devel] Avocado testplans
Sergey Bronnikov
estetus at gmail.com
Tue Aug 7 09:16:03 UTC 2018
пн, 6 авг. 2018 г. в 19:03, Lukáš Doktor <ldoktor at redhat.com>:
> Dne 6.8.2018 v 09:52 Sergey Bronnikov napsal(a):
> > пт, 3 авг. 2018 г. в 14:55, Lukáš Doktor <ldoktor at redhat.com>:
> >
> >> Well the execution is actually simpler than you think, `avocado run
> >> basic.yaml` should does the trick. Anyway the yaml-file is slightly
> >> corrupted and even then some things won't probably result in what you
> >> expect. Let me explain it a bit and provide better example)
> >>
> >> Dne 3.8.2018 v 10:20 Sergey Bronnikov napsal(a):
> >>> Lukas, thanks for explanation! But I still have a question.
> >>>
> >>> Imagine we have an YAML config with a list of tests and their
> parameters:
> >>>
> >>> ```yaml
> >>> basic: !mux
> >>>
> >>> test_reference_resolver_class: avocado_vt.loader.VirtTestLoader
> >>> timeout: 360
> >>> test_reference_resolver_args: !!python/dict
> >>> vt_extra_params:
> >>> - nic_model=rtl8139
> >>> - extra_params=" -cpu host"
> >>> - drive_format=ahci
> >>> - mem=2048
> >>> vt-machine-type: q35
> >>
> >> this should be "vt_machine_type"
> >>
> >>>
> >>> shutdown:
> >>> test_reference: io-github-autotest-qemu.shutdown
> >>
> >> Here `test_reference_resolver_extra:` is missing.
> >>
> >>> vt_extra_params:
> >>
> >> this should be `avocado_vt_extra_params:` (see `avocado_vt/loader.py`)
> >>
> >>> - shutdown_method = shell
> >>>
> >>> reboot:
> >>> test_reference: io-github-autotest-qemu.reboot
> >>
> >> Here `test_reference_resolver_extra:` is missing.
> >>
> >>> vt_extra_params:
> >>> - param1=1
> >>> - param2=2
> >>>
> >>> stress:
> >>> test_reference: io-github-autotest-qemu.linux_stress
> >>>
> >>> iofuzz:
> >>> test_reference: io-github-autotest-qemu.iofuzz
> >>
> >> Here `test_reference_resolver_extra:` is missing.
> >>
> >>> vt_extra_params:
> >>> - param1=1
> >>> - param2=2
> >>>
> >>> ```
> >>>
> >>> How to execute tests from this testplan?
> >>
> >> as mentioned earlier, it's recognized as test format, therefor simply
> >> `avocado run basic.yaml`.
> >>
> >> Also one thing, the vt discovery is quite slow (yeah we parse cartesian
> >> config on each invocation as it might have different values). You can
> put
> >> multiple `test_reference` into the same line and separate them by space
> in
> >> avocado-vt loader. But beware, it works as filters, therefor the
> discovered
> >> tests will be re-oredered accordingly to cartesian config (rather than
> how
> >> you put them there) and they'll occur only once. Therefor:
> >>
> >> test_reference: shutdown boot boot
> >>
> >> will generate:
> >>
> >> avocado list basic.yaml
> >> VT io-github-autotest-qemu.boot
> >> VT io-github-autotest-qemu.shutdown
> >>
> >> Anyway the speed of that is incomparably faster, which is why I use it
> >> when possible.
> >>
> >
> >
> > Finally with corrections I got a following testplan:
>
> Dear Sergey,
>
> actually you unveiled a bug; fix can be found here:
> https://github.com/avocado-framework/avocado-vt/pull/1696 (in my CI I
> always use `--vt-extra-params` so I forgot it might not be set. Sorry about
> this).
>
>
`avocado list plan.yaml` still fails. With --verbose it awlays shows error
for any testplan:
"Unexpected character / on pos 7. Special chars are allowed only in
variable assignation statement: 'only ../avocado-vt/plans/lukas.yaml'
(<string>:1)"
Common error for any YAML testplan:
~$ avocado list --verbose ../avocado-vt/plans/lukas.yaml
Type
Test
Tag(s)
NOT_A_TEST ../avocado-vt/plans/lukas.yaml: Not an INSTRUMENTED
(avocado.Test based), PyUNITTEST (unittest.TestCase based) or SIMPLE
(executable) test
!VT ../avocado-vt/plans/lukas.yaml: Unexpected character / on pos
7. Special chars are allowed only in variable assignation statement: 'only
../avocado-vt/plans/lukas.yaml' (<string>:1)
TEST TYPES SUMMARY
...
~$ curl
https://github.com/avocado-framework/avocado/blob/master/examples/yaml_to_mux_loader/basic.yaml
> ../avocado-vt/plans/basic.yaml
~$ avocado list --verbose ../avocado-vt/plans/basic.yaml
Type
Test
Tag(s)
NOT_A_TEST ../avocado-vt/plans/basic.yaml: Not an INSTRUMENTED
(avocado.Test based), PyUNITTEST (unittest.TestCase based) or SIMPLE
(executable) test
!VT ../avocado-vt/plans/basic-plain.yaml: Unexpected character / on
pos 7. Special chars are allowed only in variable assignation statement:
'only ../avocado-vt/plans/basic-plain.yaml' (<string>:1)
TEST TYPES SUMMARY
==================
!VT: 1
ACCESS_DENIED: 0
BROKEN_SYMLINK: 0
EXTERNAL: 0
INSTRUMENTED: 0
MISSING: 0
NOT_A_TEST: 1
PYUNITTEST: 0
SIMPLE: 0
VT: 0
Sergey
> With this there is still one little remaining issue in your `basic.yaml`,
> see below...
>
> >
> > basic: !mux
> > test_reference_resolver_class: avocado_vt.loader.VirtTestLoader
> > timeout: 360
> > test_reference_resolver_extra: !!python/dict
>
> This is supposed to be "test_reference_resolver_args" (you're modifying
> the `self.args` of Avocado for the purpose of this loader. Note this only
> affects things inside the loader, so you can't for example change the
> job-results-dir or so...)
>
> > vt_extra_params:
> > - mem=2048
> > - nic_model=rtl8139
> > - drive_format=ahci
> > - disk_bus=ahci
> > - extra_params=" -cpu host"
> > vt_machine_type: q35
> >
> > shutdown:
> > test_reference: io-github-autotest-qemu.shutdown
> > test_reference_resolver_extra: !!python/dict
> > avocado_vt_extra_params:
> > - shutdown_method = shell
> >
> > reboot:
> > test_reference: io-github-autotest-qemu.reboot
> > test_reference_resolver_extra: !!python/dict
> > avocado_vt_extra_params:
> > - param1 = 1
> > - param2 = 2
> >
> > stress:
> > test_reference: io-github-autotest-qemu.linux_stress
> >
> > iofuzz:
> > test_reference: io-github-autotest-qemu.iofuzz
> > test_reference_resolver_extra: !!python/dict
> > avocado_vt_extra_params:
> > - param1 = 1
> > - param2 = 2
> >
> > ~$ avocado variants -m testplans/basic.yaml
> > Multiplex variants (4):
> > Variant shutdown-edec: /run/basic/shutdown
> > Variant reboot-c8ba: /run/basic/reboot
> > Variant stress-fd6a: /run/basic/stress
> > Variant iofuzz-e8bc: /run/basic/iofuzz
> > ~$
> >
> > But it is still not accepted by 'avocado {list, run}':
> > ~$ avocado list testplans/basic.yaml
> > Unable to resolve reference(s) 'testplans/basic.yaml' with plugins(s)
> > 'file', 'vt', 'external', try running 'avocado list -V
> > testplans/basic.yaml' to see the details.
> >
> > Do you have any idea why it may happen?
>
> With that single change it works well for me (tried changing the
> "test_reference_resolver_args" and first test's
> "test_reference_resolver_extra" and it successfully propagated to test
> params).
>
> Regards,
> Lukáš
>
> >> autotest==0.16.4
> > avocado-framework==63.0
> > avocado-framework-plugin-varianter-yaml-to-mux==63.0
> > avocado-plugins-vt==63.0
> >
> > Sergey
> >
> >
> >>
> >> Regards,
> >> Lukáš
> >>
> >> PS: If this happens to work for you, would you please write an
> Avocado-vt
> >> documentation about this? I'd prefer someone external doing so because
> >> (apparently) I take many things as given... (you might as well improve
> the
> >> Avocado documentation, but looking at this I think the Avocado-vt one is
> >> missing more)
> >>
> >>>
> >>> $ avocado run -m basic.yaml
> >>> No test references provided nor any other arguments resolved into
> tests.
> >>> Please double check the executed command.
> >>>
> >>>
> >>> Sergey
> >>>
> >>
> >>
> >
> >
> >>>
> >>> чт, 2 авг. 2018 г. в 17:08, Lukáš Doktor <ldoktor at redhat.com>:
> >>>
> >>>> Dne 1.8.2018 v 13:19 Sergey Bronnikov napsal(a):
> >>>>> Hello!
> >>>>>
> >>>>> I'm trying to find a way to organize my tests for running without
> >>>>> specifying all of them in command-line. Imagine we have a set of
> >>>> different
> >>>>> tests: sleep.py, passtest.py and fail.py. To run all of them with
> >>>> required
> >>>>> parameters (aka variants) we may execute a command:
> >>>>> avocado run sleep.py fail.py passtest.py -m basic.yaml
> >>>>>
> >>>>> Is there a way to run the same tests without explicit specifying
> them?
> >>>>> For example: avocado run -m basic.yaml
> >>>>>
> >>>>> Sergey
> >>>>>
> >>>>
> >>>> Dear Sergey,
> >>>>
> >>>> as a matter of fact, there are multiple ways.
> >>>>
> >>>> 1. filter-by-tag:
> >>>>
> >>
> http://avocado-framework.readthedocs.io/en/latest/WritingTests.html#categorizing-tests
> >>>>
> >>>> you can tag your tests eg. `sanity`, `release`, ... and then run
> >>>> `avocado run --filter-by-tag=sanity -- TEST_REPO` to only run tests
> >> tagged
> >>>> `sanity` (note the granularity is per-class or even per-test-method.
> >>>>
> >>>> 2. yaml loader optional plugin:
> >>>>
> >>
> http://avocado-framework.readthedocs.io/en/latest/optional_plugins/yaml_loader.html
> >>>>
> >>>> which is probably more what you're looking for, but currently only
> >>>> works on python2 (but should get py3 support, hopefully, soon). It
> works
> >>>> similarly to yaml_to_mux varianter plugin, but allows special keys
> like
> >>>> `test_reference` to specify what test will be used or even
> >>>> `test_reference_resolver_class` when your tests are not the default
> >> ones.
> >>>> This is for example useful when you want to include multiple
> >>>> external-runner tests with different external-runner. There are
> examples
> >>>> in:
> >>>>
> >>
> https://github.com/avocado-framework/avocado/tree/master/examples/yaml_to_mux_loader
> >>>> but let me share my slighly advanced example here:
> >>>>
> >>>> ```yaml
> >>>> # Generated s390x mux-suite
> >>>> !mux
> >>>> qemu_make: !mux
> >>>> mux_suite_test_name_prefix: 'qemu_make/git '
> >>>> check:
> >>>> test_reference:
> >>>> /home/jenkins/s390x/qemu-master/avocado-qemu-make-check
> >>>> install:
> >>>> test_reference:
> >>>> /home/jenkins/s390x/qemu-master/avocado-qemu-make-install
> >>>>
> >>>> unit:
> >>>> variants: !mux
> >>>> test_reference_resolver_class:
> >> avocado.core.loader.ExternalLoader
> >>>> timeout: 360
> >>>> rpm-tcg:
> >>>> mux_suite_test_name_prefix: 'unit/rpm/tcg '
> >>>> test_reference_resolver_extra: !!python/dict
> >>>> loader_options:
> >>>> /home/jenkins/s390x/kvm-unit-test/avocado-runner-rpm-tcg
> >>>> rpm-kvm:
> >>>> mux_suite_test_name_prefix: 'unit/rpm/kvm '
> >>>> test_reference_resolver_extra: !!python/dict
> >>>> loader_options:
> >>>> /home/jenkins/s390x/kvm-unit-test/avocado-runner-rpm-kvm
> >>>> git-tcg:
> >>>> mux_suite_test_name_prefix: 'unit/git/tcg '
> >>>> test_reference_resolver_extra: !!python/dict
> >>>> loader_options:
> >>>> /home/jenkins/s390x/kvm-unit-test/avocado-runner-git-tcg
> >>>> git-kvm:
> >>>> mux_suite_test_name_prefix: 'unit/git/kvm '
> >>>> test_reference_resolver_extra: !!python/dict
> >>>> loader_options:
> >>>> /home/jenkins/s390x/kvm-unit-test/avocado-runner-git-kvm
> >>>> tests: !mux
> >>>> cmm:
> >>>> test_reference: cmm
> >>>> diag10:
> >>>> test_reference: diag10
> >>>> emulator:
> >>>> test_reference: emulator
> >>>> gs:
> >>>> test_reference: gs
> >>>> iep:
> >>>> test_reference: iep
> >>>> intercept:
> >>>> test_reference: intercept
> >>>> pfmf:
> >>>> test_reference: pfmf
> >>>> selftest-setup:
> >>>> test_reference: selftest-setup
> >>>> sieve:
> >>>> test_reference: sieve
> >>>> skey:
> >>>> test_reference: skey
> >>>> sthyi:
> >>>> test_reference: sthyi
> >>>> vector:
> >>>> test_reference: vector
> >>>> functional: !mux
> >>>> test_reference_resolver_class: avocado_vt.loader.VirtTestLoader
> >>>> s390x-rpm-libvirt/dvd:
> >>>> mux_suite_test_name_prefix: 'functional/rpm/libvirt/dvd '
> >>>> test_reference:
> >>>> unattended_install.cdrom.extra_cdrom_ks.default_install.aio_threads
> >>>> io-github-autotest-qemu.boot remove_guest.without_disk
> >>>> test_reference_resolver_args: !!python/dict
> >>>> vt_qemu_bin: /usr/libexec/qemu-kvm
> >>>> vt_dst_qemu_bin: /usr/libexec/qemu-kvm
> >>>> vt_type: libvirt
> >>>> vt_extra_params:
> >>>> - automem=no
> >>>> - url=$URL
> >>>> - vga=none
> >>>> s390x-rpm-libvirt/url:
> >>>> mux_suite_test_name_prefix: 'functional/rpm/libvirt/url '
> >>>> test_reference:
> >>>> unattended_install.url.extra_cdrom_ks.default_install.aio_threads
> >>>> io-github-autotest-qemu.boot remove_guest.without_disk
> >>>> test_reference_resolver_args: !!python/dict
> >>>> vt_qemu_bin: /usr/libexec/qemu-kvm
> >>>> vt_dst_qemu_bin: /usr/libexec/qemu-kvm
> >>>> vt_type: libvirt
> >>>> vt_extra_params:
> >>>> - automem=no
> >>>> - url=$URL
> >>>> - vga=none
> >>>> s390x-rpm-dvd:
> >>>> mux_suite_test_name_prefix: 'functional/rpm/dvd '
> >>>> test_reference:
> >>>> unattended_install.cdrom.extra_cdrom_ks.default_install.aio_threads
> boot
> >>>> migrate.with_reboot.tcp migrate.with_reboot.exec.gzip_exec
> >>>> migrate.default.tcp.with_post_copy
> >>>> test_reference_resolver_args: !!python/dict
> >>>> vt_qemu_bin: /usr/libexec/qemu-kvm
> >>>> vt_dst_qemu_bin: /usr/libexec/qemu-kvm
> >>>> vt_extra_params:
> >>>> - automem=no
> >>>> - url=$URL
> >>>> - ping_pong=5
> >>>> - vga=none
> >>>> s390x-rpm-url:
> >>>> mux_suite_test_name_prefix: 'functional/rpm/url '
> >>>> test_reference:
> >>>> unattended_install.url.extra_cdrom_ks.default_install.aio_threads boot
> >>>> migrate.with_reboot.tcp migrate.with_reboot.exec.gzip_exec
> >>>> migrate.default.tcp.with_post_copy
> >>>> test_reference_resolver_args: !!python/dict
> >>>> vt_qemu_bin: /usr/libexec/qemu-kvm
> >>>> vt_dst_qemu_bin: /usr/libexec/qemu-kvm
> >>>> vt_extra_params:
> >>>> - automem=no
> >>>> - url=$URL
> >>>> - ping_pong=5
> >>>> - vga=none
> >>>> s390x-git-dvd:
> >>>> mux_suite_test_name_prefix: 'functional/git/dvd '
> >>>> test_reference:
> >>>> unattended_install.cdrom.extra_cdrom_ks.default_install.aio_threads
> boot
> >>>> migrate.with_reboot.tcp migrate.with_reboot.exec.gzip_exec
> >>>> migrate.default.tcp.with_post_copy
> >>>> test_reference_resolver_args: !!python/dict
> >>>> vt_qemu_bin:
> >>>> /home/jenkins/s390x/qemu-master/build/bin/qemu-system-s390x
> >>>> vt_dst_qemu_bin:
> >>>> /home/jenkins/s390x/qemu-master/build/bin/qemu-system-s390x
> >>>> vt_extra_params:
> >>>> - automem=no
> >>>> - url=$URL
> >>>> - ping_pong=5
> >>>> - vga=none
> >>>> s390x-git-url:
> >>>> mux_suite_test_name_prefix: 'functional/git/url '
> >>>> test_reference:
> >>>> unattended_install.url.extra_cdrom_ks.default_install.aio_threads boot
> >>>> migrate.with_reboot.tcp migrate.with_reboot.exec.gzip_exec
> >>>> migrate.default.tcp.with_post_copy
> >>>> test_reference_resolver_args: !!python/dict
> >>>> vt_qemu_bin:
> >>>> /home/jenkins/s390x/qemu-master/build/bin/qemu-system-s390x
> >>>> vt_dst_qemu_bin:
> >>>> /home/jenkins/s390x/qemu-master/build/bin/qemu-system-s390x
> >>>> vt_extra_params:
> >>>> - automem=no
> >>>> - url=$URL
> >>>> - ping_pong=5
> >>>> - vga=none
> >>>> ```
> >>>>
> >>>> (feel free to ask for details, but basically it uses SIMPLE tests,
> >>>> EXTERNAL_RUNNER tests and then Avocado-vt tests. The test always
> >> receives
> >>>> all the params so you can override the standard arguments. If you
> >> specify
> >>>> "test_reference_resolver_args" it also updates the loader parameters,
> >> which
> >>>> is what I use for Avocado-vt tests).
> >>>>
> >>>> Theoretically there is yet another way and that is to create directory
> >>>> "sanity" and put symlinks to actual tests there, but the granularity
> >> there
> >>>> would be only per-test-class, which usually does not suffice.
> >>>>
> >>>> Anyway, hopefully at least one method suits you, but feel free to
> >>>> elaborate more, we can always improve.
> >>>>
> >>>> Kind regards,
> >>>> Lukáš
> >>>>
> >>>>
> >>>
> >>
> >>
> >>
> >
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/avocado-devel/attachments/20180807/d1e3b19d/attachment.htm>
More information about the Avocado-devel
mailing list