[Avocado-devel] Is it possible to handle variants only for some tests from cli interface?

[Cleber Rosa]

On Mon, Jun 19, 2023 at 3:58 AM Jan Richter <jarichte at redhat.com> wrote:
>
> On 6/16/23 12:33, Uladzimir Bely wrote:
> > Hello all.
> >
> > I have some "tests.py" with two test classes I called MuxTest and NoMuxTest.
> >
> > I want to run tests from MuxTest several times with different options, while tests from NoMuxTest should be run only once (they don't have any parameters for change).
> >
> > I'm able to do it with python avocado interface using Job/TestSuite:
> >
> > dict_variants.py:
> >
> > ```
> > import sys
> >
> > from avocado.core.job import Job
> > from avocado.core.suite import TestSuite
> >
> > MUX_CONFIG = {
> > "resolver.references": ["tests.py:MuxTest"],
> > "run.dict_variants.variant_id_keys": ["dut"],
> > "run.dict_variants": [
> > {"dut": "rpi-arm-v7l"},
> > {"dut": "de0-nano-soc"},
> > {"dut": "stm32mp15x"}
> > ],
> > }
> >
> > NOMUX_CONFIG = {
> > "resolver.references": ["tests.py:NoMuxTest"],
> > }
> >
> > with Job(
> > test_suites=[
> > TestSuite.from_config(MUX_CONFIG, name="mux"),
> > TestSuite.from_config(NOMUX_CONFIG, name="nomux"),
> > ]
> > ) as j:
> > sys.exit(j.run())
> > ```
> >
> > It works like expected. When running dict_variants.py I get:
> >
> > ```
> > $ ./dict_variants.py
> > JOB ID     : 37bc94579fe96aad053439615ce4f76ffb91406a
> > JOB LOG    : /home/wiselord/avocado/job-results/job-2023-06-16T13.21-37bc945/job.log
> >  (mtda-1/3) mtda.py:MuxTest.test_hw;rpi-arm-v7l: STARTED
> >  (mtda-1/3) mtda.py:MuxTest.test_hw;rpi-arm-v7l: PASS (0.01 s)
> >  (mtda-2/3) mtda.py:MuxTest.test_hw;de0-nano-soc: STARTED
> >  (mtda-2/3) mtda.py:MuxTest.test_hw;de0-nano-soc: PASS (0.01 s)
> >  (mtda-3/3) mtda.py:MuxTest.test_hw;stm32mp15x: STARTED
> >  (mtda-3/3) mtda.py:MuxTest.test_hw;stm32mp15x: PASS (0.01 s)
> >  (nomux-1/1) mtda.py:NoMuxTest.test_nomux: STARTED
> >  (nomux-1/1) mtda.py:NoMuxTest.test_nomux: PASS (0.01 s)
> > RESULTS    : PASS 4 | ERROR 0 | FAIL 0 | SKIP 0 | WARN 0 | INTERRUPT 0 | CANCEL 0
> > JOB TIME   : 1.98 s
> > ```
> >
> > mtda.py:MuxTest.test_hw is run 3 times with different parameters, while mtda.py:NoMuxTest.test_nomux is run only once.
> >
> > My question is the following:
> >
> > - Is it possible to achieve the same behaviour using avocado's command line interface (e.g. "avocado run mtda.py ...")?
> >
> > I tried to experiment, but either variants applied for both tests (so, I have 6 runs totally), or to none (so I have 2 runs).
> >
>
> Hi Uladzimir,
>
> avocado's command line interface creates only one Job with one TestSuite
> and right now avocado doesn't support multiple TestSuites through the
> command line. Therefore, you can't achieve the same results with command
> line interface. IMO, your approach with Job API is correct. If you
> really need to use command line interface, you have to use two different
> avocado run commands, but your tests will be in two different avocado jobs.
>
> — Jan
>

Hi everyone,

Just adding to the (correct) answer given by Jan.

The reason for not allowing that from the command line is that we
found no way to express such advanced conditions (which variants or
any other option map to which tests) by using command line arguments
only.  If anyone has any ideas about how to do something like this in
a sane and predictable way, I would be very much interested in
pursuing its implementation.

One other possibility is to use the Job API to create your own runner,
that will parse its own command line options.  Because this won't be
the generic "avocado" runner command line, the options can be more
specialized.  Example:

$ my-runner --tests-nomux=tests.py:NoMuxTes ...

Regards,
- Cleber.