[Avocado-devel] Avocado testplans
Sergey Bronnikov
estetus at gmail.com
Mon Aug 6 07:52:55 UTC 2018
пт, 3 авг. 2018 г. в 14:55, Lukáš Doktor <ldoktor at redhat.com>:
> Well the execution is actually simpler than you think, `avocado run
> basic.yaml` should does the trick. Anyway the yaml-file is slightly
> corrupted and even then some things won't probably result in what you
> expect. Let me explain it a bit and provide better example)
>
> Dne 3.8.2018 v 10:20 Sergey Bronnikov napsal(a):
> > Lukas, thanks for explanation! But I still have a question.
> >
> > Imagine we have an YAML config with a list of tests and their parameters:
> >
> > ```yaml
> > basic: !mux
> >
> > test_reference_resolver_class: avocado_vt.loader.VirtTestLoader
> > timeout: 360
> > test_reference_resolver_args: !!python/dict
> > vt_extra_params:
> > - nic_model=rtl8139
> > - extra_params=" -cpu host"
> > - drive_format=ahci
> > - mem=2048
> > vt-machine-type: q35
>
> this should be "vt_machine_type"
>
> >
> > shutdown:
> > test_reference: io-github-autotest-qemu.shutdown
>
> Here `test_reference_resolver_extra:` is missing.
>
> > vt_extra_params:
>
> this should be `avocado_vt_extra_params:` (see `avocado_vt/loader.py`)
>
> > - shutdown_method = shell
> >
> > reboot:
> > test_reference: io-github-autotest-qemu.reboot
>
> Here `test_reference_resolver_extra:` is missing.
>
> > vt_extra_params:
> > - param1=1
> > - param2=2
> >
> > stress:
> > test_reference: io-github-autotest-qemu.linux_stress
> >
> > iofuzz:
> > test_reference: io-github-autotest-qemu.iofuzz
>
> Here `test_reference_resolver_extra:` is missing.
>
> > vt_extra_params:
> > - param1=1
> > - param2=2
> >
> > ```
> >
> > How to execute tests from this testplan?
>
> as mentioned earlier, it's recognized as test format, therefor simply
> `avocado run basic.yaml`.
>
> Also one thing, the vt discovery is quite slow (yeah we parse cartesian
> config on each invocation as it might have different values). You can put
> multiple `test_reference` into the same line and separate them by space in
> avocado-vt loader. But beware, it works as filters, therefor the discovered
> tests will be re-oredered accordingly to cartesian config (rather than how
> you put them there) and they'll occur only once. Therefor:
>
> test_reference: shutdown boot boot
>
> will generate:
>
> avocado list basic.yaml
> VT io-github-autotest-qemu.boot
> VT io-github-autotest-qemu.shutdown
>
> Anyway the speed of that is incomparably faster, which is why I use it
> when possible.
>
Finally with corrections I got a following testplan:
basic: !mux
test_reference_resolver_class: avocado_vt.loader.VirtTestLoader
timeout: 360
test_reference_resolver_extra: !!python/dict
vt_extra_params:
- mem=2048
- nic_model=rtl8139
- drive_format=ahci
- disk_bus=ahci
- extra_params=" -cpu host"
vt_machine_type: q35
shutdown:
test_reference: io-github-autotest-qemu.shutdown
test_reference_resolver_extra: !!python/dict
avocado_vt_extra_params:
- shutdown_method = shell
reboot:
test_reference: io-github-autotest-qemu.reboot
test_reference_resolver_extra: !!python/dict
avocado_vt_extra_params:
- param1 = 1
- param2 = 2
stress:
test_reference: io-github-autotest-qemu.linux_stress
iofuzz:
test_reference: io-github-autotest-qemu.iofuzz
test_reference_resolver_extra: !!python/dict
avocado_vt_extra_params:
- param1 = 1
- param2 = 2
~$ avocado variants -m testplans/basic.yaml
Multiplex variants (4):
Variant shutdown-edec: /run/basic/shutdown
Variant reboot-c8ba: /run/basic/reboot
Variant stress-fd6a: /run/basic/stress
Variant iofuzz-e8bc: /run/basic/iofuzz
~$
But it is still not accepted by 'avocado {list, run}':
~$ avocado list testplans/basic.yaml
Unable to resolve reference(s) 'testplans/basic.yaml' with plugins(s)
'file', 'vt', 'external', try running 'avocado list -V
testplans/basic.yaml' to see the details.
Do you have any idea why it may happen?
autotest==0.16.4
avocado-framework==63.0
avocado-framework-plugin-varianter-yaml-to-mux==63.0
avocado-plugins-vt==63.0
Sergey
>
> Regards,
> Lukáš
>
> PS: If this happens to work for you, would you please write an Avocado-vt
> documentation about this? I'd prefer someone external doing so because
> (apparently) I take many things as given... (you might as well improve the
> Avocado documentation, but looking at this I think the Avocado-vt one is
> missing more)
>
> >
> > $ avocado run -m basic.yaml
> > No test references provided nor any other arguments resolved into tests.
> > Please double check the executed command.
> >
> >
> > Sergey
> >
>
>
> >
> > чт, 2 авг. 2018 г. в 17:08, Lukáš Doktor <ldoktor at redhat.com>:
> >
> >> Dne 1.8.2018 v 13:19 Sergey Bronnikov napsal(a):
> >>> Hello!
> >>>
> >>> I'm trying to find a way to organize my tests for running without
> >>> specifying all of them in command-line. Imagine we have a set of
> >> different
> >>> tests: sleep.py, passtest.py and fail.py. To run all of them with
> >> required
> >>> parameters (aka variants) we may execute a command:
> >>> avocado run sleep.py fail.py passtest.py -m basic.yaml
> >>>
> >>> Is there a way to run the same tests without explicit specifying them?
> >>> For example: avocado run -m basic.yaml
> >>>
> >>> Sergey
> >>>
> >>
> >> Dear Sergey,
> >>
> >> as a matter of fact, there are multiple ways.
> >>
> >> 1. filter-by-tag:
> >>
> http://avocado-framework.readthedocs.io/en/latest/WritingTests.html#categorizing-tests
> >>
> >> you can tag your tests eg. `sanity`, `release`, ... and then run
> >> `avocado run --filter-by-tag=sanity -- TEST_REPO` to only run tests
> tagged
> >> `sanity` (note the granularity is per-class or even per-test-method.
> >>
> >> 2. yaml loader optional plugin:
> >>
> http://avocado-framework.readthedocs.io/en/latest/optional_plugins/yaml_loader.html
> >>
> >> which is probably more what you're looking for, but currently only
> >> works on python2 (but should get py3 support, hopefully, soon). It works
> >> similarly to yaml_to_mux varianter plugin, but allows special keys like
> >> `test_reference` to specify what test will be used or even
> >> `test_reference_resolver_class` when your tests are not the default
> ones.
> >> This is for example useful when you want to include multiple
> >> external-runner tests with different external-runner. There are examples
> >> in:
> >>
> https://github.com/avocado-framework/avocado/tree/master/examples/yaml_to_mux_loader
> >> but let me share my slighly advanced example here:
> >>
> >> ```yaml
> >> # Generated s390x mux-suite
> >> !mux
> >> qemu_make: !mux
> >> mux_suite_test_name_prefix: 'qemu_make/git '
> >> check:
> >> test_reference:
> >> /home/jenkins/s390x/qemu-master/avocado-qemu-make-check
> >> install:
> >> test_reference:
> >> /home/jenkins/s390x/qemu-master/avocado-qemu-make-install
> >>
> >> unit:
> >> variants: !mux
> >> test_reference_resolver_class:
> avocado.core.loader.ExternalLoader
> >> timeout: 360
> >> rpm-tcg:
> >> mux_suite_test_name_prefix: 'unit/rpm/tcg '
> >> test_reference_resolver_extra: !!python/dict
> >> loader_options:
> >> /home/jenkins/s390x/kvm-unit-test/avocado-runner-rpm-tcg
> >> rpm-kvm:
> >> mux_suite_test_name_prefix: 'unit/rpm/kvm '
> >> test_reference_resolver_extra: !!python/dict
> >> loader_options:
> >> /home/jenkins/s390x/kvm-unit-test/avocado-runner-rpm-kvm
> >> git-tcg:
> >> mux_suite_test_name_prefix: 'unit/git/tcg '
> >> test_reference_resolver_extra: !!python/dict
> >> loader_options:
> >> /home/jenkins/s390x/kvm-unit-test/avocado-runner-git-tcg
> >> git-kvm:
> >> mux_suite_test_name_prefix: 'unit/git/kvm '
> >> test_reference_resolver_extra: !!python/dict
> >> loader_options:
> >> /home/jenkins/s390x/kvm-unit-test/avocado-runner-git-kvm
> >> tests: !mux
> >> cmm:
> >> test_reference: cmm
> >> diag10:
> >> test_reference: diag10
> >> emulator:
> >> test_reference: emulator
> >> gs:
> >> test_reference: gs
> >> iep:
> >> test_reference: iep
> >> intercept:
> >> test_reference: intercept
> >> pfmf:
> >> test_reference: pfmf
> >> selftest-setup:
> >> test_reference: selftest-setup
> >> sieve:
> >> test_reference: sieve
> >> skey:
> >> test_reference: skey
> >> sthyi:
> >> test_reference: sthyi
> >> vector:
> >> test_reference: vector
> >> functional: !mux
> >> test_reference_resolver_class: avocado_vt.loader.VirtTestLoader
> >> s390x-rpm-libvirt/dvd:
> >> mux_suite_test_name_prefix: 'functional/rpm/libvirt/dvd '
> >> test_reference:
> >> unattended_install.cdrom.extra_cdrom_ks.default_install.aio_threads
> >> io-github-autotest-qemu.boot remove_guest.without_disk
> >> test_reference_resolver_args: !!python/dict
> >> vt_qemu_bin: /usr/libexec/qemu-kvm
> >> vt_dst_qemu_bin: /usr/libexec/qemu-kvm
> >> vt_type: libvirt
> >> vt_extra_params:
> >> - automem=no
> >> - url=$URL
> >> - vga=none
> >> s390x-rpm-libvirt/url:
> >> mux_suite_test_name_prefix: 'functional/rpm/libvirt/url '
> >> test_reference:
> >> unattended_install.url.extra_cdrom_ks.default_install.aio_threads
> >> io-github-autotest-qemu.boot remove_guest.without_disk
> >> test_reference_resolver_args: !!python/dict
> >> vt_qemu_bin: /usr/libexec/qemu-kvm
> >> vt_dst_qemu_bin: /usr/libexec/qemu-kvm
> >> vt_type: libvirt
> >> vt_extra_params:
> >> - automem=no
> >> - url=$URL
> >> - vga=none
> >> s390x-rpm-dvd:
> >> mux_suite_test_name_prefix: 'functional/rpm/dvd '
> >> test_reference:
> >> unattended_install.cdrom.extra_cdrom_ks.default_install.aio_threads boot
> >> migrate.with_reboot.tcp migrate.with_reboot.exec.gzip_exec
> >> migrate.default.tcp.with_post_copy
> >> test_reference_resolver_args: !!python/dict
> >> vt_qemu_bin: /usr/libexec/qemu-kvm
> >> vt_dst_qemu_bin: /usr/libexec/qemu-kvm
> >> vt_extra_params:
> >> - automem=no
> >> - url=$URL
> >> - ping_pong=5
> >> - vga=none
> >> s390x-rpm-url:
> >> mux_suite_test_name_prefix: 'functional/rpm/url '
> >> test_reference:
> >> unattended_install.url.extra_cdrom_ks.default_install.aio_threads boot
> >> migrate.with_reboot.tcp migrate.with_reboot.exec.gzip_exec
> >> migrate.default.tcp.with_post_copy
> >> test_reference_resolver_args: !!python/dict
> >> vt_qemu_bin: /usr/libexec/qemu-kvm
> >> vt_dst_qemu_bin: /usr/libexec/qemu-kvm
> >> vt_extra_params:
> >> - automem=no
> >> - url=$URL
> >> - ping_pong=5
> >> - vga=none
> >> s390x-git-dvd:
> >> mux_suite_test_name_prefix: 'functional/git/dvd '
> >> test_reference:
> >> unattended_install.cdrom.extra_cdrom_ks.default_install.aio_threads boot
> >> migrate.with_reboot.tcp migrate.with_reboot.exec.gzip_exec
> >> migrate.default.tcp.with_post_copy
> >> test_reference_resolver_args: !!python/dict
> >> vt_qemu_bin:
> >> /home/jenkins/s390x/qemu-master/build/bin/qemu-system-s390x
> >> vt_dst_qemu_bin:
> >> /home/jenkins/s390x/qemu-master/build/bin/qemu-system-s390x
> >> vt_extra_params:
> >> - automem=no
> >> - url=$URL
> >> - ping_pong=5
> >> - vga=none
> >> s390x-git-url:
> >> mux_suite_test_name_prefix: 'functional/git/url '
> >> test_reference:
> >> unattended_install.url.extra_cdrom_ks.default_install.aio_threads boot
> >> migrate.with_reboot.tcp migrate.with_reboot.exec.gzip_exec
> >> migrate.default.tcp.with_post_copy
> >> test_reference_resolver_args: !!python/dict
> >> vt_qemu_bin:
> >> /home/jenkins/s390x/qemu-master/build/bin/qemu-system-s390x
> >> vt_dst_qemu_bin:
> >> /home/jenkins/s390x/qemu-master/build/bin/qemu-system-s390x
> >> vt_extra_params:
> >> - automem=no
> >> - url=$URL
> >> - ping_pong=5
> >> - vga=none
> >> ```
> >>
> >> (feel free to ask for details, but basically it uses SIMPLE tests,
> >> EXTERNAL_RUNNER tests and then Avocado-vt tests. The test always
> receives
> >> all the params so you can override the standard arguments. If you
> specify
> >> "test_reference_resolver_args" it also updates the loader parameters,
> which
> >> is what I use for Avocado-vt tests).
> >>
> >> Theoretically there is yet another way and that is to create directory
> >> "sanity" and put symlinks to actual tests there, but the granularity
> there
> >> would be only per-test-class, which usually does not suffice.
> >>
> >> Anyway, hopefully at least one method suits you, but feel free to
> >> elaborate more, we can always improve.
> >>
> >> Kind regards,
> >> Lukáš
> >>
> >>
> >
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/avocado-devel/attachments/20180806/6b85b2f0/attachment.htm>
More information about the Avocado-devel
mailing list