[Avocado-devel] RFC: Guidelines for categorizing tests

[Satheesh Rajendran]

On Fri, 2017-05-19 at 10:28 -0400, Cleber Rosa wrote:
> 
> On 05/18/2017 01:07 AM, Satheesh Rajendran wrote:
> > 
> > On Wed, 2017-05-17 at 17:49 -0400, Cleber Rosa wrote:
> > > 
> > > Introduction
> > > ============
> > > 
> > > Avocado allows users to select tests to run based on free form
> > > "tags".
> > > These tags are given as "docstring directives", that is, special
> > > entries on a class or function docstring.
> > > 
> > This would be very helpful and becomes easy way to 
> > create a dynamic testsuite based on needs.
> > +1
> I'm not sure if it was clear enough, but this functionality already
> exists in Avocado:
> 
> http://avocado-framework.readthedocs.io/en/50.0/WritingTests.html#cat
> egorizing-tests
> 
> This proposal is about proposing a set of guidelines for using that
> feature.
> 
Oh Sure, need to explore on that, Thanks :-)
> > 
> > > 
> > > As a user of an Avocado based test suite, I'd see value in not
> > > **having**
> > > to look at all the test tags before realizing that to not run
> > > tests
> > > that
> > > require "super user" permissions I should run::
> > > 
> > >   $ avocado run test.py --filter-by-tags=-root
> > > 
> > > Instead of::
> > > 
> > >   $ avocado run test.py --filter-by-tags=-privileged
> > > 
> > > Not even that, by having different tests as part of the same job,
> > > the following very odd sequence of command line options may be
> > > needed::
> > > 
> > >   $ avocado run test.py test2.py --filter-by-tags=-root,-
> > > privileged
> > > 
> > > So the goal here is to let users familiar with a given Avocado
> > > based
> > > test, to have fair expectations when running another Avocado
> > > based
> > > tests.
> > > 
> > > This was initially going to be a documentation update, but I felt
> > > that
> > > it was not fair to make a formal proposal without without some
> > > initial
> > > brainstorming.
> > > 
> > > Proposal
> > > ========
> > > 
> > > To set the tone for my proposal, I'd like to make most things
> > > simple
> > > and easy, while allowing for "everything else" to be doable.
> > > 
> > > My general impression is that higher level information about the
> > > test
> > > itself and its requirements are going to be the most commonly
> > > used
> > > tags, so they must be easily set.  Some examples follow.
> > > 
> > > Simple (standalone) tags
> > > ------------------------
> > > 
> > > Tags by functional area:
> > > 
> > >  * cpu - Exercises a system's CPU
> > >  * net - Exercises a system's network devices or networking stack
> > >  * storage - Exercises a system's local storage
> > >  * fs - Exercises a system's file system
> > > 
> > > Tags by architecture:
> > > 
> > >  * x86_64 - Requires a x86_64 architecture
> > >  * ppc64 - Requries a ppc64
> > > 
> > > Tags by access privileges:
> > > 
> > >  * privileged - requires the test to be run with the most
> > > privileged,
> > >    unrestricted privileges.  For Linux systems, this usually
> > > means
> > > the
> > >    root account
> > > 
> > > Composed (key:value) tags
> > > -------------------------
> > > 
> > > The more specific tags can be achieved by composing a predefined
> > > key
> > > with a value.  For instance, to tag a test as needing a specific
> > > CPU flag:
> > > 
> > >  * cpu_flag:vmx
> > > 
> > > Or a specific PCI device:
> > > 
> > >  * pci_id:8086:08b2
> > > 
> > > Or even a software package:
> > > 
> > >  * package:gcc
> > > 
> > > Or a package group altogether:
> > > 
> > >  * package_group:development-tools
> > > 
> > > Some examples
> > > -------------
> > > 
> > >  * ``cpu,x86_64`` - The test exercises the CPU and requires a
> > >    ``x86_64`` based platform
> > > 
> > >  * ``net,privileged,pci_id:14e4:1657`` - The test exercises
> > > either a
> > >    network device or the network stack, needs super user
> > > privileges
> > >    and a "Broadcom Limited NetXtreme BCM5719 Gigabit Ethernet
> > > PCIe
> > >    (rev 01)" device.
> > > 
> > > Looking at test tags
> > > ====================
> > > 
> > > Currently, there's no way to actually list the test tags from the
> > > command line, alongside the test name themselves.  An RFE for
> > > fixing
> > > this has been filed at https://trello.com/c/NXrbKEJC .
> > > 
> > > Do users have to provide all the ``--filter-by-tags`` themselves?
> > > =================================================================
> > > 
> > > The test runner can certainly help here, getting system
> > > information
> > > when the job starts, and feeding them to the filtering.  This is
> > > yet
> > > another reason why coming up with a good set of guidelines for
> > > tagging
> > > tests is important.
> > > 
> > > In some ways, this can be seen similar to a dependency resolution
> > > mechanism for tests, only that at this point it will not resolve
> > > the
> > > requirements.  It will only filter out tests that can't (or
> > > shouldn't)
> > > be loaded on the current system.
> > > 
> > > Effectively, instead of many in-tests checks, and many
> > > SKIPs/CANCELs,
> > > the system information can be loaded once, and the only relevant
> > > tests
> > > will be part of the tests suite.
> > > 
> > > A list of the tests that were filtered out from the job test
> > > suite
> > > can
> > > certainly be a useful part of the job results.
> > > 
> > > As in every RFC, feedback is extremely welcome!
> > > 
> > I assume this has to be at the overall testcase level, can not
> > be applied to multiplexed params. 
> > for example:
> > certain multiplexer options to be filtered in the same testcase.
> > 
> > As the proposal is to use doc-strings, I assume it can not be done?
> > 
> Exactly, it's done at the test level, before they're combined with
> variants.
> 
> I'm not quite sure that it can't be done, but definitely it's out of
> the
> scope of this specific RFC.
> 
Sure, makes sense.

Regards,
-Satheesh.
> Cheers!
>