[Avocado-devel] Tips for making a standalone test script Avocado-friendly?

Wed Apr 5 15:06:52 UTC 2017

On 04/05/2017 09:00 AM, Eduardo Habkost wrote:
> 
> Hi,
> 
> I have been writing a few standalone Python scripts[1] to test
> QEMU recently, and I would like to make them more useful for
> people running tests using Avocado.
> 
> Most of them work this way:
> 1) Query QEMU to check which
>    architectures/machine-types/CPU-models/devices/options
>    it supports
> 2) Run QEMU multiple times for each
>    architectures/machine-types/CPU-models/devices/options
>    combination I want to test
> 3) Report success/failure/skip results (sometimes including
>    warnings) for each combination
> 

I went ahead and tried one of them:

QTEST_QEMU_BINARY=../x86_64-softmmu/qemu-system-x86_64 python
query-cpu-model-test.py
..
----------------------------------------------------------------------
Ran 2 tests in 34.227s

OK

One very important aspect here is result granularity.  You're currently
using Python's unittest.TestCase, which in this case gives you a
granularity of two tests.  At this point, Avocado could find those two
tests (with some help[1]):

$ ~/src/avocado/avocado/contrib/scripts/avocado-find-unittests
query-cpu-model-test.py
query-cpu-model-test.CPUModelTest.testTCGModels
query-cpu-model-test.CPUModelTest.testKVMModels

The other granularity level that can be achieved here is a test per
executed script:

$ QTEST_QEMU_BINARY=../x86_64-softmmu/qemu-system-x86_64 avocado run
query-cpu-model-test.py
JOB ID     : 11f0a5fbb02e6eb67580dd33270867b039806585
JOB LOG    :
/home/cleber/avocado/job-results/job-2017-04-05T10.31-11f0a5f/job.log
 (1/1) query-cpu-model-test.py: PASS (33.22 s)
RESULTS    : PASS 1 | ERROR 0 | FAIL 0 | SKIP 0 | WARN 0 | INTERRUPT 0 |
CANCEL 0
TESTS TIME : 33.22 s
JOB HTML   :
/home/cleber/avocado/job-results/job-2017-04-05T10.31-11f0a5f/html/results.html

Which is nothing but our SIMPLE[2] tests support.

About reporting results, PASS, FAIL and WARN are available for SIMPLE
tests. PASS is exit status 0, FAIL is anything but 0, and WARN is a PASS
that looks for a pattern in the test output[3][4].

> I would like to keep the test scripts easy to run without
> installing extra dependencies, so I want them to keep working as
> standalone scripts even if Avocado modules aren't available.

That's OK, but I would like to add a few notes:

1) When removing the unittest dependency, you'll most probably end up
creating custom mechanisms for locating and executing test entry points
and reporting those results.

2) We've worked to make sure Avocado is really easy to depend on.  For
instance, it's available on stock Fedora (package name is
python-avocado) and on PyPI (installing it just requires a `pip install
avocado-framework`).

> Adding a few "if avocado_available:" lines to the script would be
> OK, though.
> 

I can't really recommend this optional check of Avocado for INSTRUMENTED
tests.  I really think the complexity added would bring more bad than good.

If your tests are going to be treated as SIMPLE tests, you could just
make a check for environment variables that Avocado makes available[5].

> Do you have any suggestions for making the test result output
> from those scripts easily consumable by the Avocado test runner?
> 

If those tests are treated as SIMPLE tests, then all of their output
will automatically be added to the Avocado logs.

Now, I do understand what your hopes are (or were after my possibly
disappointing recommendations :).  Ideally, Avocado would be able to
find the tests available on your standalone scripts[6], would know how
to run the individual tests[7].  Right now, this requires writing a
plugin.  I can see some ideas for a "Python universal" plugin here, that
would find other hints of tests in plain Python source code.  If you
think that makes sense, let's discuss it further.

> 
> [1] Some examples:
>     https://github.com/ehabkost/qemu-hacks/blob/work/device-crash-script/scripts/device-crash-test.py
>     https://github.com/ehabkost/qemu-hacks/blob/work/x86-query-cpu-expansion-test/tests/query-cpu-model-test.py
>     https://github.com/ehabkost/qemu-hacks/blob/work/query-machines-bus-info/tests/qmp-machine-info.py
>     (Note that some of the scripts use the unittest module, but I
>     will probably get rid of it, because the list of test cases I
>     want to run will be generated at runtime. I've even wrote
>     code to add test methods dynamically to the test class, but I
>     will probably remove that hack because it's not worth the
>     extra complexity)
> 

[1] -
https://github.com/avocado-framework/avocado/blob/master/contrib/scripts/avocado-find-unittests
    -
http://avocado-framework.readthedocs.io/en/48.0/GetStartedGuide.html#running-tests-with-an-external-runner

[2] -
http://avocado-framework.readthedocs.io/en/48.0/GetStartedGuide.html#writing-a-simple-test

[3] -
https://github.com/avocado-framework/avocado/blob/master/examples/tests/simplewarning.sh

[4] -
http://avocado-framework.readthedocs.io/en/48.0/WritingTests.html#test-statuses

[5] -
http://avocado-framework.readthedocs.io/en/48.0/WritingTests.html#environment-variables-for-simple-tests

[6] -
https://github.com/avocado-framework/avocado/blob/master/optional_plugins/robot/avocado_robot/__init__.py#L79

[7] -
https://github.com/avocado-framework/avocado/blob/master/optional_plugins/robot/avocado_robot/__init__.py#L52

-- 
Cleber Rosa
[ Sr Software Engineer - Virtualization Team - Red Hat ]
[ Avocado Test Framework - avocado-framework.github.io ]
[  7ABB 96EB 8B46 B94D 5E0F  E9BB 657E 8D33 A5F2 09F3  ]

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 819 bytes
Desc: OpenPGP digital signature
URL: <http://listman.redhat.com/archives/avocado-devel/attachments/20170405/2b663aec/attachment.sig>