[Avocado-devel] RFC: Guidelines for categorizing tests
Satheesh Rajendran
sathnaga at linux.vnet.ibm.com
Thu May 18 05:07:45 UTC 2017
On Wed, 2017-05-17 at 17:49 -0400, Cleber Rosa wrote:
> Introduction
> ============
>
> Avocado allows users to select tests to run based on free form
> "tags".
> These tags are given as "docstring directives", that is, special
> entries on a class or function docstring.
>
This would be very helpful and becomes easy way to
create a dynamic testsuite based on needs.
+1
> As a user of an Avocado based test suite, I'd see value in not
> **having**
> to look at all the test tags before realizing that to not run tests
> that
> require "super user" permissions I should run::
>
> $ avocado run test.py --filter-by-tags=-root
>
> Instead of::
>
> $ avocado run test.py --filter-by-tags=-privileged
>
> Not even that, by having different tests as part of the same job,
> the following very odd sequence of command line options may be
> needed::
>
> $ avocado run test.py test2.py --filter-by-tags=-root,-privileged
>
> So the goal here is to let users familiar with a given Avocado based
> test, to have fair expectations when running another Avocado based
> tests.
>
> This was initially going to be a documentation update, but I felt
> that
> it was not fair to make a formal proposal without without some
> initial
> brainstorming.
>
> Proposal
> ========
>
> To set the tone for my proposal, I'd like to make most things simple
> and easy, while allowing for "everything else" to be doable.
>
> My general impression is that higher level information about the test
> itself and its requirements are going to be the most commonly used
> tags, so they must be easily set. Some examples follow.
>
> Simple (standalone) tags
> ------------------------
>
> Tags by functional area:
>
> * cpu - Exercises a system's CPU
> * net - Exercises a system's network devices or networking stack
> * storage - Exercises a system's local storage
> * fs - Exercises a system's file system
>
> Tags by architecture:
>
> * x86_64 - Requires a x86_64 architecture
> * ppc64 - Requries a ppc64
>
> Tags by access privileges:
>
> * privileged - requires the test to be run with the most privileged,
> unrestricted privileges. For Linux systems, this usually means
> the
> root account
>
> Composed (key:value) tags
> -------------------------
>
> The more specific tags can be achieved by composing a predefined key
> with a value. For instance, to tag a test as needing a specific
> CPU flag:
>
> * cpu_flag:vmx
>
> Or a specific PCI device:
>
> * pci_id:8086:08b2
>
> Or even a software package:
>
> * package:gcc
>
> Or a package group altogether:
>
> * package_group:development-tools
>
> Some examples
> -------------
>
> * ``cpu,x86_64`` - The test exercises the CPU and requires a
> ``x86_64`` based platform
>
> * ``net,privileged,pci_id:14e4:1657`` - The test exercises either a
> network device or the network stack, needs super user privileges
> and a "Broadcom Limited NetXtreme BCM5719 Gigabit Ethernet PCIe
> (rev 01)" device.
>
> Looking at test tags
> ====================
>
> Currently, there's no way to actually list the test tags from the
> command line, alongside the test name themselves. An RFE for fixing
> this has been filed at https://trello.com/c/NXrbKEJC .
>
> Do users have to provide all the ``--filter-by-tags`` themselves?
> =================================================================
>
> The test runner can certainly help here, getting system information
> when the job starts, and feeding them to the filtering. This is yet
> another reason why coming up with a good set of guidelines for
> tagging
> tests is important.
>
> In some ways, this can be seen similar to a dependency resolution
> mechanism for tests, only that at this point it will not resolve the
> requirements. It will only filter out tests that can't (or
> shouldn't)
> be loaded on the current system.
>
> Effectively, instead of many in-tests checks, and many SKIPs/CANCELs,
> the system information can be loaded once, and the only relevant
> tests
> will be part of the tests suite.
>
> A list of the tests that were filtered out from the job test suite
> can
> certainly be a useful part of the job results.
>
> As in every RFC, feedback is extremely welcome!
>
I assume this has to be at the overall testcase level, can not
be applied to multiplexed params.
for example:
certain multiplexer options to be filtered in the same testcase.
As the proposal is to use doc-strings, I assume it can not be done?
>
More information about the Avocado-devel
mailing list