[Avocado-devel] RFC: Guidelines for categorizing tests
Cleber Rosa
crosa at redhat.com
Thu May 18 20:02:27 UTC 2017
On 05/18/2017 03:34 PM, Narasimhan V wrote:
> On 2017-05-18 20:18, Cleber Rosa wrote:
>> On 05/18/2017 12:22 AM, Narasimhan V wrote:
>>> Hi Cleber,
>>>
>>> I agree the proposal.
>>> Please find some of my comments in-line.
>>>
>>
>> Hi Narasimhan,
>>
>> Thanks for the feedback. My responses are inline.
>>
>>> On 2017-05-18 03:19, Cleber Rosa wrote:
>>>> Introduction
>>>> ============
>>>>
>>>> Avocado allows users to select tests to run based on free form "tags".
>>>> These tags are given as "docstring directives", that is, special
>>>> entries on a class or function docstring.
>>>
>>> I like this kind of a proposal :-)
>>>
>>
>>
>> I'm not sure if it was clear enough, but this functionality already
>> exists in Avocado:
>>
>> http://avocado-framework.readthedocs.io/en/50.0/WritingTests.html#categorizing-tests
>>
>>
>> This proposal is about proposing a set of guidelines for using that
>> feature.
>>
>
> Thanks for pointing this out.
>
>>>>
>>>> As a user of an Avocado based test suite, I'd see value in not
>>>> **having**
>>>> to look at all the test tags before realizing that to not run tests
>>>> that
>>>> require "super user" permissions I should run::
>>>>
>>>> $ avocado run test.py --filter-by-tags=-root
>>>>
>>>> Instead of::
>>>>
>>>> $ avocado run test.py --filter-by-tags=-privileged
>>>>
>>>> Not even that, by having different tests as part of the same job,
>>>> the following very odd sequence of command line options may be
>>>> needed::
>>>>
>>>> $ avocado run test.py test2.py --filter-by-tags=-root,-privileged
>>>>
>>>> So the goal here is to let users familiar with a given Avocado based
>>>> test, to have fair expectations when running another Avocado based
>>>> tests.
>>>>
>>>> This was initially going to be a documentation update, but I felt that
>>>> it was not fair to make a formal proposal without without some initial
>>>> brainstorming.
>>>>
>>>> Proposal
>>>> ========
>>>>
>>>> To set the tone for my proposal, I'd like to make most things simple
>>>> and easy, while allowing for "everything else" to be doable.
>>>>
>>>> My general impression is that higher level information about the test
>>>> itself and its requirements are going to be the most commonly used
>>>> tags, so they must be easily set. Some examples follow.
>>>>
>>>> Simple (standalone) tags
>>>> ------------------------
>>>>
>>>> Tags by functional area:
>>>>
>>>> * cpu - Exercises a system's CPU
>>>> * net - Exercises a system's network devices or networking stack
>>>> * storage - Exercises a system's local storage
>>>> * fs - Exercises a system's file system
>>>>
>>>> Tags by architecture:
>>>>
>>>> * x86_64 - Requires a x86_64 architecture
>>>> * ppc64 - Requries a ppc64
>>>>
>>>> Tags by access privileges:
>>>>
>>>> * privileged - requires the test to be run with the most privileged,
>>>> unrestricted privileges. For Linux systems, this usually means the
>>>> root account
>>>>
>>>
>>> How are these tags going to be differentiated, ie, tag names that can be
>>> common across types/areas.
>>> Examples, if already on your mind, can help here.
>>>
>>
>> Well, that's exactly the point of this proposal. If users have access
>> to two different test repos, and both are related to hardware/software
>> testing, they can expect that by choosing only tests with "cpu" tags,
>> they will be exercising their Central Processing Units.
>>
>> Conversely, if a test repo is concerned about testing Hospital
>> procedures (this is very hypothetical), they may choose to use "cpu" as
>> a shorthand for "Care of Patients with Urgency". They would be going
>> against the general Avocado guidelines, users may expect those tests to
>> be about "Central Processing Units", and be frustrated.
>>
>> Does this example make sense?
>>
>
> Are we looking at tree structure for tags, as this gets more use cases ?
>
I'm assuming you're talking about a filesystem (tree) structure that
would be mapped to tags, right? That is, a "cpu" dir would implicitly
set a ":avocado: tags=cpu" docstring directive. Is that what you mean?
This is not planned, but you're free to file an RFE, send an RFC and get
the discussion started. IMHO, directory structure and tags can be
complimentary. For instance, by running:
$ avocado run cpu/*.py
You already get most, if not all, of what you'd get by having a "tree
structure for tags" based on directories. And I see them as
complimentary, because you could still do:
$ avocado run cpu/*.py --filter-by-tags=-privileged
>>>> Composed (key:value) tags
>>>> -------------------------
>>>>
>>>> The more specific tags can be achieved by composing a predefined key
>>>> with a value. For instance, to tag a test as needing a specific
>>>> CPU flag:
>>>>
>>>> * cpu_flag:vmx
>>>>
>>>> Or a specific PCI device:
>>>>
>>>> * pci_id:8086:08b2
>>>>
>>>> Or even a software package:
>>>>
>>>> * package:gcc
>>>>
>>>> Or a package group altogether:
>>>>
>>>> * package_group:development-tools
>>>>
>>>> Some examples
>>>> -------------
>>>>
>>>> * ``cpu,x86_64`` - The test exercises the CPU and requires a
>>>> ``x86_64`` based platform
>>>>
>>>> * ``net,privileged,pci_id:14e4:1657`` - The test exercises either a
>>>> network device or the network stack, needs super user privileges
>>>> and a "Broadcom Limited NetXtreme BCM5719 Gigabit Ethernet PCIe
>>>> (rev 01)" device.
>>>
>>> I would be interested in seeing how we can implement this.
>>>
>>>
>>
>> We are actually free to start using this right away. Actually, it'd be
>> better to start doing this once we settle on a set of guidelines.
>>
>> The point is that tags are already accepted by docstring statements, and
>> recognized by `--filter-by-tags`. The next evolutionary step is on the
>> "Do users have to provide all the ``--filter-by-tags`` themselves?"
>> section bellow.
>
> So, are we going to tag a test as privileged based on the tag
> ``superuser`` ?
>
Giving good names is hard, and that's one of the reasons I'm asking for
feedback here. A few name options that are common for that:
* root - plain and simple, direct but pretty much has meaning on UNIX/Linux
* superuser - a little more abstract, but still somehow attached to
UNIX/Linux
* privileged - completely abstract, but still accurate. IMHO, the best
option here.
> And, how do we use the key value pairs (pci_id14e4:1657) to run / skip
> the tests ?
> Some examples there would help as well.
>
All tags can be used the same way, they carry no meaning on themselves.
So, if you write a test such as:
from avocado import Test
class PCI(Test):
def test_vendor_foo(self):
"""
:avocado: tags=pci_id:14e4:1657
"""
pass
def test_vendor_bar(self):
"""
:avocado: tags=pci_id:dead:beef
"""
pass
You can filter with:
$ avocado run pci.py --filter-by-tags=pci_id:14e4:1657
And the result is:
avocado run pci.py --filter-by-tags=pci_id:14e4:1657
JOB ID : a8b9348d9a544f4fec6b944eaa6cc5be4bf11035
JOB LOG :
/home/cleber/avocado/job-results/job-2017-05-18T16.01-a8b9348/job.log
(1/1) pci.py:PCI.test_vendor_foo: PASS (0.01 s)
RESULTS : PASS 1 | ERROR 0 | FAIL 0 | SKIP 0 | WARN 0 | INTERRUPT
0 | CANCEL 0
JOB TIME : 0.15 s
JOB HTML :
/home/cleber/avocado/job-results/job-2017-05-18T16.01-a8b9348/html/results.html
It it clear? Questions? Comments?
--
Cleber Rosa
[ Sr Software Engineer - Virtualization Team - Red Hat ]
[ Avocado Test Framework - avocado-framework.github.io ]
[ 7ABB 96EB 8B46 B94D 5E0F E9BB 657E 8D33 A5F2 09F3 ]
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 819 bytes
Desc: OpenPGP digital signature
URL: <http://listman.redhat.com/archives/avocado-devel/attachments/20170518/c6781415/attachment.sig>
More information about the Avocado-devel
mailing list