[Avocado-devel] General test running questions

[Olav Philipp Henschel]

On 02-12-2015 15:49, Lukáš Doktor wrote:
> Dne 2.12.2015 v 07:29 Lucas Meneghel Rodrigues napsal(a):
>>
>>
>> On Tue, Dec 1, 2015 at 1:38 PM Olav Philipp Henschel
>> <olavph at linux.vnet.ibm.com <mailto:olavph at linux.vnet.ibm.com>> wrote:
>>
>>     I have a few questions regarding test running, the multiplexer 
>> and test
>>     variants:
>>
>>     In avocado-vt, each test has a cartesian config file that is applied
>>     only to that single test. Is there any plan to do something 
>> similar with
>>     multiplex files? Sometimes I want to create variants to one test 
>> but not
>>     the other.
>>
>>
>> Strictly speaking, each individual .cfg file ends up merged on a goddamn
>> giant subtests.cfg file.
>>
>> I'd say we want to support modularity between multiplex files, but I'm
>> not sure if we already have an include system implemented there. Have
>> we, Lukas?
>
> Hello Olav, as Lucas said, this is not supported. I send an RFC which 
> tried to make it possible to specify some default multiplexations year 
> ago:
>
> https://www.redhat.com/archives/virt-test-devel/2014-November/msg00004.html 
>
>
> but so far we had not time, nor consensus to really work on it. 
> Multiplexer is still quite young and we need to improve it. And it is 
> also possible, that we might come up with completely different design 
> (then the RFC, we won't probably change the multiplexer itself, it 
> seems to work quite well).

Ok, as I understand, for now, if I want to apply variants to a single 
test, I'd have to run that test separately, right?
It'd be nice to have test specific files or filters.

>
>>
>>     Is there currently any way of making a test dependent on another? 
>> For
>>     example, if an unattended_install test fails, I want avocado to 
>> skip the
>>     other tests.
>>
>>
>> For avocado-vt, yes, sure. The cartesian config has provisions for that.
>> However, for avocado-virt (AKA the next generation), we had endless
>> discussions about that and reached the conclusion that having a
>> dependency system is bad design. Each test should be able to run
>> independently of other tests. If you need an installed vm for the tests,
>> you should be able to specify that as a requirement for your test. The
>> requires system is something still being designed.
>
> This question caused the delay as I don't have any official info yet. 
> The thing is that I'm also solving similar issues in our grid. 
> Currently I used jenkins pipelines to define the relations 
> (unattended_install -> tests | cleanup). It'd be definitely nice to 
> have it in avocado itself, but we need to come up with flexible, yet 
> simple to write definitions.
>
> I quite like the way pipeline works in jenkins, so I can imagine 
> similar approach. Alternatively we can reuse yaml to define the flow.
>
> To wrap it up, I don't have an official statement, I don't know 
> whether this is the priority right now, but I'd like to work on it. 
> Please bear with us (before Christmas there are many PTOs delaying the 
> discussions). We'd also welcome your ideas, or suggestions.

I've opened an issue 
(https://github.com/avocado-framework/avocado-vt/issues/282), because, 
apparently, avocado-vt dependencies are not working.
I will think about possible ways to support this.

>
>>
>>     Is there any way of specifying a set of tests to run, instead of 
>> passing
>>     each one of them? In avocado-vt, we can specify the test 
>> provider, or a
>>     substring of the complete test name. It would be useful if we 
>> could pass
>>     globs or something like that to execute all tests in a directory.
>>
>>
>> You can pass the test directory to avocado. The test resolver will find
>> and run all the tests inside that directory:
>>
>> avocado run examples/tests
>> JOB ID     : 90da56e7af74bc493a6c8990d805c585380abc6b
>> JOB LOG    :
>> /home/lmr/avocado/job-results/job-2015-12-02T04.27-90da56e/job.log
>> TESTS      : 57
>>   (1/57) examples/tests/warntest.py:WarnTest.test: WARN (0.00 s)
>>   (2/57) examples/tests/passtest.py:PassTest.test: PASS (0.00 s)
>>   ...
>>
>> So I believe that covers your use case.
>
> Yep, we execute programs from shell, which means you can use shell to 
> specify what should be executed. Feel free to use:
>
>     avocado run examples/tests/a*
>     avocado run examples/tests/doublefree*.py
>     avocado run `find examples/ -iname 'a*.py' | grep -v "foo"`
>
> or other bashisms...

Nice, I didn't know that worked.

Thank you for your time,
Olav P. Henschel