[Avocado-devel] RFC: Guidelines for categorizing tests

Mon May 22 18:58:18 UTC 2017

On 05/22/2017 02:00 PM, Ademar Reis wrote:
> On Wed, May 17, 2017 at 05:49:36PM -0400, Cleber Rosa wrote:
>> Introduction
>> ============
>>
>> Avocado allows users to select tests to run based on free form "tags".
>> These tags are given as "docstring directives", that is, special
>> entries on a class or function docstring.
>>
>> As a user of an Avocado based test suite, I'd see value in not **having**
>> to look at all the test tags before realizing that to not run tests that
>> require "super user" permissions I should run::
>>
>>   $ avocado run test.py --filter-by-tags=-root
>>
>> Instead of::
>>
>>   $ avocado run test.py --filter-by-tags=-privileged
>>
>> Not even that, by having different tests as part of the same job,
>> the following very odd sequence of command line options may be
>> needed::
>>
>>   $ avocado run test.py test2.py --filter-by-tags=-root,-privileged
>>
>> So the goal here is to let users familiar with a given Avocado based
>> test, to have fair expectations when running another Avocado based
>> tests.
>>
>> This was initially going to be a documentation update, but I felt that
>> it was not fair to make a formal proposal without without some initial
>> brainstorming.
>>
>> Proposal
>> ========
>>
>> To set the tone for my proposal, I'd like to make most things simple
>> and easy, while allowing for "everything else" to be doable.
>>
>> My general impression is that higher level information about the test
>> itself and its requirements are going to be the most commonly used
>> tags, so they must be easily set.  Some examples follow.
>>
>> Simple (standalone) tags
>> ------------------------
>>
>> Tags by functional area:
>>
>>  * cpu - Exercises a system's CPU
>>  * net - Exercises a system's network devices or networking stack
>>  * storage - Exercises a system's local storage
>>  * fs - Exercises a system's file system
>>
>> Tags by architecture:
>>
>>  * x86_64 - Requires a x86_64 architecture
>>  * ppc64 - Requries a ppc64
>>
>> Tags by access privileges:
>>
>>  * privileged - requires the test to be run with the most privileged,
>>    unrestricted privileges.  For Linux systems, this usually means the
>>    root account
>>
>> Composed (key:value) tags
>> -------------------------
>>
>> The more specific tags can be achieved by composing a predefined key
>> with a value.  For instance, to tag a test as needing a specific
>> CPU flag:
>>
>>  * cpu_flag:vmx
>>
>> Or a specific PCI device:
>>
>>  * pci_id:8086:08b2
>>
>> Or even a software package:
>>
>>  * package:gcc
>>
>> Or a package group altogether:
>>
>>  * package_group:development-tools
>>
>> Some examples
>> -------------
>>
>>  * ``cpu,x86_64`` - The test exercises the CPU and requires a
>>    ``x86_64`` based platform
>>
>>  * ``net,privileged,pci_id:14e4:1657`` - The test exercises either a
>>    network device or the network stack, needs super user privileges
>>    and a "Broadcom Limited NetXtreme BCM5719 Gigabit Ethernet PCIe
>>    (rev 01)" device.
> 
> In my understanding, one of the key design aspects of tags is
> that they're user-defined, optional, and totally arbitrary to the
> rest of Avocado.  In other words, to Avocado there's no semantics
> in a tag called "ppc64", "priviledged" or "pci_id:8086:08b2".
> 

Right, the fact that there are no semantics in tags, and that Avocado
itself will *not* attempt to interpret the tags has been repeated a
couple of times.  I hope it's somewhat clearer now.

> This should be clear in the documentation and in this RFC,
> otherwise users might be tempted to start tagging tests following
> some sort of "official list of tags" provided by Avocado, or
> coding features that depend on some specific tag.
> 

Users tagging tests following some sort of "official list of tags" is
indeed what I'm pursuing here.  And I'm pursuing it here because there
has been an understanding that it'd be beneficial across test repos.
BTW, it'd be an "official" *recommendation*, but not a requirement.

> (or is this actually your intent?)
> 

Partially, yes.  See answer above.  BTW, the task generated this RFC is
called "Created a guideline for tagging tests":

https://trello.com/c/ZMmFqA5j/880-created-a-guideline-for-tagging-tests

Which IMO adheres to the idea of "official list of tags", but doesn't
enforce them.

>>
>> Looking at test tags
>> ====================
>>
>> Currently, there's no way to actually list the test tags from the
>> command line, alongside the test name themselves.  An RFE for fixing
>> this has been filed at https://trello.com/c/NXrbKEJC .
> 
> +1 to this.
> 
>>
>> Do users have to provide all the ``--filter-by-tags`` themselves?
>> =================================================================
>>
>> The test runner can certainly help here, getting system information
>> when the job starts, and feeding them to the filtering.  This is yet
>> another reason why coming up with a good set of guidelines for tagging
>> tests is important.
>>
>> In some ways, this can be seen similar to a dependency resolution
>> mechanism for tests, only that at this point it will not resolve the
>> requirements.  It will only filter out tests that can't (or shouldn't)
>> be loaded on the current system.
>>
>> Effectively, instead of many in-tests checks, and many SKIPs/CANCELs,
>> the system information can be loaded once, and the only relevant tests
>> will be part of the tests suite.
>>
>> A list of the tests that were filtered out from the job test suite can
>> certainly be a useful part of the job results.
> 
> This is close to what I had in mind years ago when I proposed a
> simple dependency resolution mechanism, I think we discussed
> it in the past:
> 

Yes, it is similar.  The difference is that we were discussing a
scheduler feature, and a way to choose an environment (a machine?) where
the tests *could* be run.  This is limited only to the gathering of
facts of the local environment (local machine?) and passing it on to the
loader automatically, according to the documented tags guidelines
published before.

Bear in mind that this is completely optional to the original proposal.

>  - Tests could have a set of tags listed as dependencies (I don't
>    think generic tags should be used for it);

Do you mean another "docstring directive" such as ":avocado: dep=" or
the use of specific tags as depencies such as ":avocado:
tag=dep:package:foo"?

>  - Environments where tests are run provide a list of tags as
>    capabilities;

Yes, this is what this would be limited to.  And the "environment" here
would be only the environment in which the tests are being resolved.

>  - Avocado, when running tests, only runs the test in
>    environments where all dependencies are matched by
>    capabilities.
> 

This is different: avocado, when resolving tests, will only resolve the
ones which have all tags that were required.  Some of the tags required
*may* come from this "automatic system information tag generation".

>>
>> As in every RFC, feedback is extremely welcome!
>>
> 
> I think the generic tags mechanism should be kept arbitrary and
> abstract. My first reaction is that there should be no
> interpretation of the contents of the tags by other parts of the
> system or "official list of tags and their semantics". And I'm
> not sure if this is what you're proposing. :-)
> 

In no situation in this proposal Avocado would attempt to interpret the
meaning of a tag.

Now, I'm not sure if you believe writing a guideline for the usage of
tags mean adding semantics to it.  If you do, then we need to adjust the
task description and outcome, because it's really about proposing  a
somewhat official list recommended tags.

> Thanks.
>    - Ademar
> 

Thanks for the feedback!  Looking forward for further clarification of
the few points I didn't fully understand.

-- 
Cleber Rosa
[ Sr Software Engineer - Virtualization Team - Red Hat ]
[ Avocado Test Framework - avocado-framework.github.io ]
[  7ABB 96EB 8B46 B94D 5E0F  E9BB 657E 8D33 A5F2 09F3  ]

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 819 bytes
Desc: OpenPGP digital signature
URL: <http://listman.redhat.com/archives/avocado-devel/attachments/20170522/3bfb1dec/attachment.sig>