[Avocado-devel] Tips for making a standalone test script Avocado-friendly?
Cleber Rosa
crosa at redhat.com
Thu Apr 6 16:38:31 UTC 2017
On 04/06/2017 11:00 AM, Lukáš Doktor wrote:
> Dne 5.4.2017 v 17:06 Cleber Rosa napsal(a):
>>
>> On 04/05/2017 09:00 AM, Eduardo Habkost wrote:
>>>
>>> Hi,
>>>
>>> I have been writing a few standalone Python scripts[1] to test
>>> QEMU recently, and I would like to make them more useful for
>>> people running tests using Avocado.
>>>
>>> Most of them work this way:
>>> 1) Query QEMU to check which
>>> architectures/machine-types/CPU-models/devices/options
>>> it supports
>>> 2) Run QEMU multiple times for each
>>> architectures/machine-types/CPU-models/devices/options
>>> combination I want to test
>>> 3) Report success/failure/skip results (sometimes including
>>> warnings) for each combination
>>>
>>
>> I went ahead and tried one of them:
>>
>> QTEST_QEMU_BINARY=../x86_64-softmmu/qemu-system-x86_64 python
>> query-cpu-model-test.py
>> ..
>> ----------------------------------------------------------------------
>> Ran 2 tests in 34.227s
>>
>> OK
>>
>> One very important aspect here is result granularity. You're currently
>> using Python's unittest.TestCase, which in this case gives you a
>> granularity of two tests. At this point, Avocado could find those two
>> tests (with some help[1]):
>>
>> $ ~/src/avocado/avocado/contrib/scripts/avocado-find-unittests
>> query-cpu-model-test.py
>> query-cpu-model-test.CPUModelTest.testTCGModels
>> query-cpu-model-test.CPUModelTest.testKVMModels
>>
>> The other granularity level that can be achieved here is a test per
>> executed script:
>>
>> $ QTEST_QEMU_BINARY=../x86_64-softmmu/qemu-system-x86_64 avocado run
>> query-cpu-model-test.py
>> JOB ID : 11f0a5fbb02e6eb67580dd33270867b039806585
>> JOB LOG :
>> /home/cleber/avocado/job-results/job-2017-04-05T10.31-11f0a5f/job.log
>> (1/1) query-cpu-model-test.py: PASS (33.22 s)
>> RESULTS : PASS 1 | ERROR 0 | FAIL 0 | SKIP 0 | WARN 0 | INTERRUPT 0 |
>> CANCEL 0
>> TESTS TIME : 33.22 s
>> JOB HTML :
>> /home/cleber/avocado/job-results/job-2017-04-05T10.31-11f0a5f/html/results.html
>>
>>
>>
>> Which is nothing but our SIMPLE[2] tests support.
>>
>> About reporting results, PASS, FAIL and WARN are available for SIMPLE
>> tests. PASS is exit status 0, FAIL is anything but 0, and WARN is a PASS
>> that looks for a pattern in the test output[3][4].
>>
>>> I would like to keep the test scripts easy to run without
>>> installing extra dependencies, so I want them to keep working as
>>> standalone scripts even if Avocado modules aren't available.
>>
>> That's OK, but I would like to add a few notes:
>>
>> 1) When removing the unittest dependency, you'll most probably end up
>> creating custom mechanisms for locating and executing test entry points
>> and reporting those results.
>>
>> 2) We've worked to make sure Avocado is really easy to depend on. For
>> instance, it's available on stock Fedora (package name is
>> python-avocado) and on PyPI (installing it just requires a `pip install
>> avocado-framework`).
>>
>>> Adding a few "if avocado_available:" lines to the script would be
>>> OK, though.
>>>
>>
>> I can't really recommend this optional check of Avocado for INSTRUMENTED
>> tests. I really think the complexity added would bring more bad than
>> good.
>>
>> If your tests are going to be treated as SIMPLE tests, you could just
>> make a check for environment variables that Avocado makes available[5].
>>
>>> Do you have any suggestions for making the test result output
>>> from those scripts easily consumable by the Avocado test runner?
>>>
>>
>> If those tests are treated as SIMPLE tests, then all of their output
>> will automatically be added to the Avocado logs.
>>
>> Now, I do understand what your hopes are (or were after my possibly
>> disappointing recommendations :). Ideally, Avocado would be able to
>> find the tests available on your standalone scripts[6], would know how
>> to run the individual tests[7]. Right now, this requires writing a
>> plugin. I can see some ideas for a "Python universal" plugin here, that
>> would find other hints of tests in plain Python source code. If you
>> think that makes sense, let's discuss it further.
>>
> Actually we already have `robot` plugin which is able to discover
> `robot`-like tests. How about creating a `unittest` loader, which would
> do the `avocado-find-unittest` and discover them individually? By
> default nothing would change as the `.py` files would be recognized as
> instrumented/simple tests, but when you change the order of loaders it'd
> detect them as unittests:
>
Eduardo mentioned that he'd eventually get rid of the unittest
dependency. That's why I was suggesting/brainstorming about an even
more generic way of providing hints that there are tests on a Python
source file.
Still, I agree that it shouldn't be too hard.
- Cleber.
> ```
> avocado run query-cpu-model-test.py => query-cpu-model-test.py
> avocado run --loaders python-unittest =>
> query-cpu-model-test.CPUModelTest.testTCGModels;
> query-cpu-model-test.CPUModelTest.testKVMModels
> ```
>
> This should be really simple to develop (even simpler than robot as all
> things are already in place).
>
> Lukáš
>
>>>
>>> [1] Some examples:
>>>
>>> https://github.com/ehabkost/qemu-hacks/blob/work/device-crash-script/scripts/device-crash-test.py
>>>
>>>
>>> https://github.com/ehabkost/qemu-hacks/blob/work/x86-query-cpu-expansion-test/tests/query-cpu-model-test.py
>>>
>>>
>>> https://github.com/ehabkost/qemu-hacks/blob/work/query-machines-bus-info/tests/qmp-machine-info.py
>>>
>>> (Note that some of the scripts use the unittest module, but I
>>> will probably get rid of it, because the list of test cases I
>>> want to run will be generated at runtime. I've even wrote
>>> code to add test methods dynamically to the test class, but I
>>> will probably remove that hack because it's not worth the
>>> extra complexity)
>>>
>>
>> [1] -
>> https://github.com/avocado-framework/avocado/blob/master/contrib/scripts/avocado-find-unittests
>>
>> -
>> http://avocado-framework.readthedocs.io/en/48.0/GetStartedGuide.html#running-tests-with-an-external-runner
>>
>>
>> [2] -
>> http://avocado-framework.readthedocs.io/en/48.0/GetStartedGuide.html#writing-a-simple-test
>>
>>
>> [3] -
>> https://github.com/avocado-framework/avocado/blob/master/examples/tests/simplewarning.sh
>>
>>
>> [4] -
>> http://avocado-framework.readthedocs.io/en/48.0/WritingTests.html#test-statuses
>>
>>
>> [5] -
>> http://avocado-framework.readthedocs.io/en/48.0/WritingTests.html#environment-variables-for-simple-tests
>>
>>
>> [6] -
>> https://github.com/avocado-framework/avocado/blob/master/optional_plugins/robot/avocado_robot/__init__.py#L79
>>
>>
>> [7] -
>> https://github.com/avocado-framework/avocado/blob/master/optional_plugins/robot/avocado_robot/__init__.py#L52
>>
>>
>
--
Cleber Rosa
[ Sr Software Engineer - Virtualization Team - Red Hat ]
[ Avocado Test Framework - avocado-framework.github.io ]
[ 7ABB 96EB 8B46 B94D 5E0F E9BB 657E 8D33 A5F2 09F3 ]
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 819 bytes
Desc: OpenPGP digital signature
URL: <http://listman.redhat.com/archives/avocado-devel/attachments/20170406/b3da02a3/attachment.sig>
More information about the Avocado-devel
mailing list