[Avocado-devel] RFC: Guidelines for categorizing tests

[Narasimhan V]

Hi Cleber,

I agree the proposal.
Please find some of my comments in-line.

On 2017-05-18 03:19, Cleber Rosa wrote:
> Introduction
> ============
> 
> Avocado allows users to select tests to run based on free form "tags".
> These tags are given as "docstring directives", that is, special
> entries on a class or function docstring.

I like this kind of a proposal :-)

> 
> As a user of an Avocado based test suite, I'd see value in not 
> **having**
> to look at all the test tags before realizing that to not run tests 
> that
> require "super user" permissions I should run::
> 
>   $ avocado run test.py --filter-by-tags=-root
> 
> Instead of::
> 
>   $ avocado run test.py --filter-by-tags=-privileged
> 
> Not even that, by having different tests as part of the same job,
> the following very odd sequence of command line options may be
> needed::
> 
>   $ avocado run test.py test2.py --filter-by-tags=-root,-privileged
> 
> So the goal here is to let users familiar with a given Avocado based
> test, to have fair expectations when running another Avocado based
> tests.
> 
> This was initially going to be a documentation update, but I felt that
> it was not fair to make a formal proposal without without some initial
> brainstorming.
> 
> Proposal
> ========
> 
> To set the tone for my proposal, I'd like to make most things simple
> and easy, while allowing for "everything else" to be doable.
> 
> My general impression is that higher level information about the test
> itself and its requirements are going to be the most commonly used
> tags, so they must be easily set.  Some examples follow.
> 
> Simple (standalone) tags
> ------------------------
> 
> Tags by functional area:
> 
>  * cpu - Exercises a system's CPU
>  * net - Exercises a system's network devices or networking stack
>  * storage - Exercises a system's local storage
>  * fs - Exercises a system's file system
> 
> Tags by architecture:
> 
>  * x86_64 - Requires a x86_64 architecture
>  * ppc64 - Requries a ppc64
> 
> Tags by access privileges:
> 
>  * privileged - requires the test to be run with the most privileged,
>    unrestricted privileges.  For Linux systems, this usually means the
>    root account
> 

How are these tags going to be differentiated, ie, tag names that can be
common across types/areas.
Examples, if already on your mind, can help here.

> Composed (key:value) tags
> -------------------------
> 
> The more specific tags can be achieved by composing a predefined key
> with a value.  For instance, to tag a test as needing a specific
> CPU flag:
> 
>  * cpu_flag:vmx
> 
> Or a specific PCI device:
> 
>  * pci_id:8086:08b2
> 
> Or even a software package:
> 
>  * package:gcc
> 
> Or a package group altogether:
> 
>  * package_group:development-tools
> 
> Some examples
> -------------
> 
>  * ``cpu,x86_64`` - The test exercises the CPU and requires a
>    ``x86_64`` based platform
> 
>  * ``net,privileged,pci_id:14e4:1657`` - The test exercises either a
>    network device or the network stack, needs super user privileges
>    and a "Broadcom Limited NetXtreme BCM5719 Gigabit Ethernet PCIe
>    (rev 01)" device.

I would be interested in seeing how we can implement this.


> 
> Looking at test tags
> ====================
> 
> Currently, there's no way to actually list the test tags from the
> command line, alongside the test name themselves.  An RFE for fixing
> this has been filed at https://trello.com/c/NXrbKEJC .
> 
> Do users have to provide all the ``--filter-by-tags`` themselves?
> =================================================================
> 
> The test runner can certainly help here, getting system information
> when the job starts, and feeding them to the filtering.  This is yet
> another reason why coming up with a good set of guidelines for tagging
> tests is important.
> 
> In some ways, this can be seen similar to a dependency resolution
> mechanism for tests, only that at this point it will not resolve the
> requirements.  It will only filter out tests that can't (or shouldn't)
> be loaded on the current system.
> 
> Effectively, instead of many in-tests checks, and many SKIPs/CANCELs,
> the system information can be loaded once, and the only relevant tests
> will be part of the tests suite.
> 
> A list of the tests that were filtered out from the job test suite can
> certainly be a useful part of the job results.
> 
> As in every RFC, feedback is extremely welcome!

-- 
Regards
Narasimhan V