[Avocado-devel] General test running questions

[Lukáš Doktor]

Dne 2.12.2015 v 07:29 Lucas Meneghel Rodrigues napsal(a):
>
>
> On Tue, Dec 1, 2015 at 1:38 PM Olav Philipp Henschel
> <olavph at linux.vnet.ibm.com <mailto:olavph at linux.vnet.ibm.com>> wrote:
>
>     I have a few questions regarding test running, the multiplexer and test
>     variants:
>
>     In avocado-vt, each test has a cartesian config file that is applied
>     only to that single test. Is there any plan to do something similar with
>     multiplex files? Sometimes I want to create variants to one test but not
>     the other.
>
>
> Strictly speaking, each individual .cfg file ends up merged on a goddamn
> giant subtests.cfg file.
>
> I'd say we want to support modularity between multiplex files, but I'm
> not sure if we already have an include system implemented there. Have
> we, Lukas?

Hello Olav, as Lucas said, this is not supported. I send an RFC which 
tried to make it possible to specify some default multiplexations year ago:

https://www.redhat.com/archives/virt-test-devel/2014-November/msg00004.html

but so far we had not time, nor consensus to really work on it. 
Multiplexer is still quite young and we need to improve it. And it is 
also possible, that we might come up with completely different design 
(then the RFC, we won't probably change the multiplexer itself, it seems 
to work quite well).

>
>     Is there currently any way of making a test dependent on another? For
>     example, if an unattended_install test fails, I want avocado to skip the
>     other tests.
>
>
> For avocado-vt, yes, sure. The cartesian config has provisions for that.
> However, for avocado-virt (AKA the next generation), we had endless
> discussions about that and reached the conclusion that having a
> dependency system is bad design. Each test should be able to run
> independently of other tests. If you need an installed vm for the tests,
> you should be able to specify that as a requirement for your test. The
> requires system is something still being designed.

This question caused the delay as I don't have any official info yet. 
The thing is that I'm also solving similar issues in our grid. Currently 
I used jenkins pipelines to define the relations (unattended_install -> 
tests | cleanup). It'd be definitely nice to have it in avocado itself, 
but we need to come up with flexible, yet simple to write definitions.

I quite like the way pipeline works in jenkins, so I can imagine similar 
approach. Alternatively we can reuse yaml to define the flow.

To wrap it up, I don't have an official statement, I don't know whether 
this is the priority right now, but I'd like to work on it. Please bear 
with us (before Christmas there are many PTOs delaying the discussions). 
We'd also welcome your ideas, or suggestions.

>
>     Is there any way of specifying a set of tests to run, instead of passing
>     each one of them? In avocado-vt, we can specify the test provider, or a
>     substring of the complete test name. It would be useful if we could pass
>     globs or something like that to execute all tests in a directory.
>
>
> You can pass the test directory to avocado. The test resolver will find
> and run all the tests inside that directory:
>
> avocado run examples/tests
> JOB ID     : 90da56e7af74bc493a6c8990d805c585380abc6b
> JOB LOG    :
> /home/lmr/avocado/job-results/job-2015-12-02T04.27-90da56e/job.log
> TESTS      : 57
>   (1/57) examples/tests/warntest.py:WarnTest.test: WARN (0.00 s)
>   (2/57) examples/tests/passtest.py:PassTest.test: PASS (0.00 s)
>   ...
>
> So I believe that covers your use case.

Yep, we execute programs from shell, which means you can use shell to 
specify what should be executed. Feel free to use:

     avocado run examples/tests/a*
     avocado run examples/tests/doublefree*.py
     avocado run `find examples/ -iname 'a*.py' | grep -v "foo"`

or other bashisms...