[Avocado-devel] RFC: Guidelines for categorizing tests

Thu May 18 19:34:43 UTC 2017

On 2017-05-18 20:18, Cleber Rosa wrote:
> On 05/18/2017 12:22 AM, Narasimhan V wrote:
>> Hi Cleber,
>> 
>> I agree the proposal.
>> Please find some of my comments in-line.
>> 
> 
> Hi Narasimhan,
> 
> Thanks for the feedback.  My responses are inline.
> 
>> On 2017-05-18 03:19, Cleber Rosa wrote:
>>> Introduction
>>> ============
>>> 
>>> Avocado allows users to select tests to run based on free form 
>>> "tags".
>>> These tags are given as "docstring directives", that is, special
>>> entries on a class or function docstring.
>> 
>> I like this kind of a proposal :-)
>> 
> 
> 
> I'm not sure if it was clear enough, but this functionality already
> exists in Avocado:
> 
> http://avocado-framework.readthedocs.io/en/50.0/WritingTests.html#categorizing-tests
> 
> This proposal is about proposing a set of guidelines for using that 
> feature.
> 

Thanks for pointing this out.

>>> 
>>> As a user of an Avocado based test suite, I'd see value in not 
>>> **having**
>>> to look at all the test tags before realizing that to not run tests 
>>> that
>>> require "super user" permissions I should run::
>>> 
>>>   $ avocado run test.py --filter-by-tags=-root
>>> 
>>> Instead of::
>>> 
>>>   $ avocado run test.py --filter-by-tags=-privileged
>>> 
>>> Not even that, by having different tests as part of the same job,
>>> the following very odd sequence of command line options may be
>>> needed::
>>> 
>>>   $ avocado run test.py test2.py --filter-by-tags=-root,-privileged
>>> 
>>> So the goal here is to let users familiar with a given Avocado based
>>> test, to have fair expectations when running another Avocado based
>>> tests.
>>> 
>>> This was initially going to be a documentation update, but I felt 
>>> that
>>> it was not fair to make a formal proposal without without some 
>>> initial
>>> brainstorming.
>>> 
>>> Proposal
>>> ========
>>> 
>>> To set the tone for my proposal, I'd like to make most things simple
>>> and easy, while allowing for "everything else" to be doable.
>>> 
>>> My general impression is that higher level information about the test
>>> itself and its requirements are going to be the most commonly used
>>> tags, so they must be easily set.  Some examples follow.
>>> 
>>> Simple (standalone) tags
>>> ------------------------
>>> 
>>> Tags by functional area:
>>> 
>>>  * cpu - Exercises a system's CPU
>>>  * net - Exercises a system's network devices or networking stack
>>>  * storage - Exercises a system's local storage
>>>  * fs - Exercises a system's file system
>>> 
>>> Tags by architecture:
>>> 
>>>  * x86_64 - Requires a x86_64 architecture
>>>  * ppc64 - Requries a ppc64
>>> 
>>> Tags by access privileges:
>>> 
>>>  * privileged - requires the test to be run with the most privileged,
>>>    unrestricted privileges.  For Linux systems, this usually means 
>>> the
>>>    root account
>>> 
>> 
>> How are these tags going to be differentiated, ie, tag names that can 
>> be
>> common across types/areas.
>> Examples, if already on your mind, can help here.
>> 
> 
> Well, that's exactly the point of this proposal.  If users have access
> to two different test repos, and both are related to hardware/software
> testing, they can expect that by choosing only tests with "cpu" tags,
> they will be exercising their Central Processing Units.
> 
> Conversely, if a test repo is concerned about testing Hospital
> procedures (this is very hypothetical), they may choose to use "cpu" as
> a shorthand for "Care of Patients with Urgency".  They would be going
> against the general Avocado guidelines, users may expect those tests to
> be about "Central Processing Units", and be frustrated.
> 
> Does this example make sense?
> 

Are we looking at tree structure for tags, as this gets more use cases ?

>>> Composed (key:value) tags
>>> -------------------------
>>> 
>>> The more specific tags can be achieved by composing a predefined key
>>> with a value.  For instance, to tag a test as needing a specific
>>> CPU flag:
>>> 
>>>  * cpu_flag:vmx
>>> 
>>> Or a specific PCI device:
>>> 
>>>  * pci_id:8086:08b2
>>> 
>>> Or even a software package:
>>> 
>>>  * package:gcc
>>> 
>>> Or a package group altogether:
>>> 
>>>  * package_group:development-tools
>>> 
>>> Some examples
>>> -------------
>>> 
>>>  * ``cpu,x86_64`` - The test exercises the CPU and requires a
>>>    ``x86_64`` based platform
>>> 
>>>  * ``net,privileged,pci_id:14e4:1657`` - The test exercises either a
>>>    network device or the network stack, needs super user privileges
>>>    and a "Broadcom Limited NetXtreme BCM5719 Gigabit Ethernet PCIe
>>>    (rev 01)" device.
>> 
>> I would be interested in seeing how we can implement this.
>> 
>> 
> 
> We are actually free to start using this right away.  Actually, it'd be
> better to start doing this once we settle on a set of guidelines.
> 
> The point is that tags are already accepted by docstring statements, 
> and
> recognized by `--filter-by-tags`.  The next evolutionary step is on the
> "Do users have to provide all the ``--filter-by-tags`` themselves?"
> section bellow.

So, are we going to tag a test as privileged based on the tag 
``superuser`` ?

And, how do we use the key value pairs (pci_id14e4:1657) to run / skip 
the tests ?
Some examples there would help as well.

> 
> - Cleber.
> 
>>> 
>>> Looking at test tags
>>> ====================
>>> 
>>> Currently, there's no way to actually list the test tags from the
>>> command line, alongside the test name themselves.  An RFE for fixing
>>> this has been filed at https://trello.com/c/NXrbKEJC .
>>> 
>>> Do users have to provide all the ``--filter-by-tags`` themselves?
>>> =================================================================
>>> 
>>> The test runner can certainly help here, getting system information
>>> when the job starts, and feeding them to the filtering.  This is yet
>>> another reason why coming up with a good set of guidelines for 
>>> tagging
>>> tests is important.
>>> 
>>> In some ways, this can be seen similar to a dependency resolution
>>> mechanism for tests, only that at this point it will not resolve the
>>> requirements.  It will only filter out tests that can't (or 
>>> shouldn't)
>>> be loaded on the current system.
>>> 
>>> Effectively, instead of many in-tests checks, and many SKIPs/CANCELs,
>>> the system information can be loaded once, and the only relevant 
>>> tests
>>> will be part of the tests suite.
>>> 
>>> A list of the tests that were filtered out from the job test suite 
>>> can
>>> certainly be a useful part of the job results.
>>> 
>>> As in every RFC, feedback is extremely welcome!
>> 

-- 
Regards
Narasimhan V