[Avocado-devel] How to execute different tests of avocado-vt in one round

[Lukáš Doktor]

Dne 09. 01. 19 v 15:57 Vit Pelcak napsal(a):
> Hello.
> 
> Dne 08. 01. 19 v 17:47 Cleber Rosa napsal(a):
>>
>>
>> On 1/8/19 9:20 AM, Brian J. Murrell wrote:
>>> On Tue, 2019-01-08 at 14:57 +0100, Vit Pelcak wrote:
>>>> Hello.
>>>
>>> Hi,
>>>
>>> While not really answering your question, per se but perhaps you are
>>> trying to solve the wrong problem...
>>>
>>>>
>>>> That will mean extra 10 minutes for each of them as
>>>> VM
>>>> will have to be started each time, resulting in extra 30 minutes time
>>>> in
>>>> total.
>>>
>>> Your virtual machine really takes 10 minutes to start?  Why so long? 
>>> It should be a minute or two at most.  Under a minute would not be
>>> absurd.
>>>
>>> Cheers,
>>> b.
>>>
>>
>> Unless you're running under CPU emulation, or nested virt, low powered
>> and highly loaded environments...
>>
>> Still, based on experience, we strongly suggest you start a fresh VM on
>> every new test: the results can be easily spoiled if you don't do that.
> 
> Thank you very much for advice.
> 
> And if I do this:
> 
> 
> https://github.com/avocado-framework/avocado-vt/blob/master/etc/avocado/conf.d/vt.conf#L7
> 
> # Keep guest running between tests (faster, but unsafe)
> keep_guest_running=False
> 
> Is it possible to trigger VM restart?
> 

As mentioned in my other email, VM gets recreated when test requires different devices/setting. You can also execute "shutdown" test. Last but the most powerful would be the yaml_to_mux loader that allows you to fine-tune your setting. Excerpt from my CI:

```
# Generated s390x mux-suite
!mux
qemu_make: !mux
    mux_suite_test_name_prefix: 'qemu_make/git '
    check:
        test_reference: /home/jenkins/s390x/qemu-master/avocado-qemu-make-check
    install:
        test_reference: /home/jenkins/s390x/qemu-master/avocado-qemu-make-install

unit:
    variants: !mux
        test_reference_resolver_class: avocado.core.loader.ExternalLoader
        timeout: 360
        rpm-tcg:
            mux_suite_test_name_prefix: 'unit/rpm/tcg '
            test_reference_resolver_extra: !!python/dict
                loader_options: /home/jenkins/s390x/kvm-unit-test/avocado-runner-rpm-tcg
        rpm-kvm:
            mux_suite_test_name_prefix: 'unit/rpm/kvm '
            test_reference_resolver_extra: !!python/dict
                loader_options: /home/jenkins/s390x/kvm-unit-test/avocado-runner-rpm-kvm
        # ...
    tests: !mux
        cmm:
            test_reference: cmm
        diag10:
            test_reference: diag10
        emulator:
            test_reference: emulator
        # ...

functional: !mux
    test_reference_resolver_class: avocado_vt.loader.VirtTestLoader
    s390x-rpm-libvirt/dvdfetch_dvd/:
        mux_suite_test_name_prefix: 'functional/rpm/libvirt/dvd '
        test_reference_resolver_class: avocado.core.loader.ExternalLoader
        test_reference_resolver_extra: !!python/dict
            loader_options: /bin/sh -c
        test_reference: "'avocado_get_rhel7_cd s390x \"URL\" '"
    s390x-rpm-libvirt/dvd:
        mux_suite_test_name_prefix: 'functional/rpm/libvirt/dvd '
        test_reference: unattended_install.cdrom.extra_cdrom_ks.default_install.aio_threads io-github-autotest-qemu.boot remove_guest.without_disk
        test_reference_resolver_args: !!python/dict
            vt_qemu_bin: /usr/libexec/qemu-kvm
            vt_dst_qemu_bin: /usr/libexec/qemu-kvm
            vt_type: libvirt
            vt_extra_params:
                - automem=no
                - qemu_sandbox=''
                - url=URL
                - vga=none
                - mem=2048
    s390x-rpm-libvirt/url:
        mux_suite_test_name_prefix: 'functional/rpm/libvirt/url '
        test_reference: unattended_install.url.extra_cdrom_ks.default_install.aio_threads io-github-autotest-qemu.boot remove_guest.without_disk
        test_reference_resolver_args: !!python/dict
            vt_qemu_bin: /usr/libexec/qemu-kvm
            vt_dst_qemu_bin: /usr/libexec/qemu-kvm
            vt_type: libvirt
            vt_extra_params:
                - automem=no
                - qemu_sandbox=''
                - url=URL
                - vga=none
                - mem=2048
    s390x-rpm-RHEL-7/install/dvd/fetch_dvd/:
        mux_suite_test_name_prefix: 'functional/rpm/RHEL-7/install/dvd/ '
        test_reference_resolver_class: avocado.core.loader.ExternalLoader
        test_reference_resolver_extra: !!python/dict
            loader_options: /bin/sh -c
        test_reference: "'avocado_get_rhel7_cd s390x \"URL\" '"
    s390x-rpm-RHEL-7/install/dvd/:
        mux_suite_test_name_prefix: 'functional/rpm/RHEL-7/install/dvd/ '
        test_reference: unattended_install.cdrom.extra_cdrom_ks.default_install.aio_threads
        test_reference_resolver_args: !!python/dict
            vt_qemu_bin: /usr/libexec/qemu-kvm
            vt_dst_qemu_bin: /usr/libexec/qemu-kvm
            vt_extra_params:
                - automem=no
                - qemu_sandbox=''
                - url=URL
                - ping_pong=5
                - vga=none
                - mem=2048
                - timeout=3600
    # ...

```

Which is simply stored in "suite.yaml" file and executed by "avocado run suite.yaml". It executes few SIMPLE tests, then few EXTERNAL_RUNNER tests and then some VT tests with different --vt... options. (the "# ..." means I skipped some boring parts; URLs were also removed...)

Regards,
Lukáš

> I am thinking that it could make sense to group tests, disable VM
> restart and then restart the machine after each group of tests is finished.
> 
>> Regards!
>>
> 


-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 488 bytes
Desc: OpenPGP digital signature
URL: <http://listman.redhat.com/archives/avocado-devel/attachments/20190109/55f3f1e7/attachment.sig>