[Avocado-devel] Is it possible to handle variants only for some tests from cli interface?

[Jan Richter]

On 6/16/23 12:33, Uladzimir Bely wrote:
> Hello all.
> 
> I have some "tests.py" with two test classes I called MuxTest and NoMuxTest.
> 
> I want to run tests from MuxTest several times with different options, while tests from NoMuxTest should be run only once (they don't have any parameters for change).
> 
> I'm able to do it with python avocado interface using Job/TestSuite:
> 
> dict_variants.py:
> 
> ```
> import sys
> 
> from avocado.core.job import Job  
> from avocado.core.suite import TestSuite
> 
> MUX_CONFIG = {  
> "resolver.references": ["tests.py:MuxTest"],  
> "run.dict_variants.variant_id_keys": ["dut"],  
> "run.dict_variants": [  
> {"dut": "rpi-arm-v7l"},  
> {"dut": "de0-nano-soc"},  
> {"dut": "stm32mp15x"}  
> ],  
> }
> 
> NOMUX_CONFIG = {  
> "resolver.references": ["tests.py:NoMuxTest"],  
> }
> 
> with Job(  
> test_suites=[  
> TestSuite.from_config(MUX_CONFIG, name="mux"),  
> TestSuite.from_config(NOMUX_CONFIG, name="nomux"),  
> ]  
> ) as j:  
> sys.exit(j.run())
> ```
> 
> It works like expected. When running dict_variants.py I get:
> 
> ```
> $ ./dict_variants.py 
> JOB ID     : 37bc94579fe96aad053439615ce4f76ffb91406a
> JOB LOG    : /home/wiselord/avocado/job-results/job-2023-06-16T13.21-37bc945/job.log
>  (mtda-1/3) mtda.py:MuxTest.test_hw;rpi-arm-v7l: STARTED
>  (mtda-1/3) mtda.py:MuxTest.test_hw;rpi-arm-v7l: PASS (0.01 s)
>  (mtda-2/3) mtda.py:MuxTest.test_hw;de0-nano-soc: STARTED
>  (mtda-2/3) mtda.py:MuxTest.test_hw;de0-nano-soc: PASS (0.01 s)
>  (mtda-3/3) mtda.py:MuxTest.test_hw;stm32mp15x: STARTED
>  (mtda-3/3) mtda.py:MuxTest.test_hw;stm32mp15x: PASS (0.01 s)
>  (nomux-1/1) mtda.py:NoMuxTest.test_nomux: STARTED
>  (nomux-1/1) mtda.py:NoMuxTest.test_nomux: PASS (0.01 s)
> RESULTS    : PASS 4 | ERROR 0 | FAIL 0 | SKIP 0 | WARN 0 | INTERRUPT 0 | CANCEL 0
> JOB TIME   : 1.98 s
> ```
> 
> mtda.py:MuxTest.test_hw is run 3 times with different parameters, while mtda.py:NoMuxTest.test_nomux is run only once.
> 
> My question is the following:
> 
> - Is it possible to achieve the same behaviour using avocado's command line interface (e.g. "avocado run mtda.py ...")?
> 
> I tried to experiment, but either variants applied for both tests (so, I have 6 runs totally), or to none (so I have 2 runs).
> 

Hi Uladzimir,

avocado's command line interface creates only one Job with one TestSuite
and right now avocado doesn't support multiple TestSuites through the
command line. Therefore, you can't achieve the same results with command
line interface. IMO, your approach with Job API is correct. If you
really need to use command line interface, you have to use two different
avocado run commands, but your tests will be in two different avocado jobs.

— Jan