[Avocado-devel] [RFC] Avocado Test Loader: Test types and extended status

[Cleber Rosa]

=====================================================
 Avocado Test Loader: Test types and extended status
=====================================================

The current architecture of the Avocado Test Loader has a number of
shortcomings.

First, although not directly related to the major topic being present
here, it's worth noticing that the various test loader implementations
are still not implemented the same way that other extensible areas of
Avocado are.  Basically, it does *not* describe and interface at
`avocado.core.plugin_interfaces`, which should be followed by the
actual test loader implementations.

This can be explained by the fact that the test loader implementation
preceeds the "new" plugin architecture, and has not yet been ported.
Given that this document will suggest changes to the Avocado Test
Loader subsystem, it makes sense to also consider porting it to the
"new" architeture.  This task can will be tracked here:

https://trello.com/c/H3hSd3Lm/1123-test-loader-adapt-implementation-to-the-new-plugin-architecture

But the most urgent violation in the architecture is one the most
visible to users: "test types" which are not really tests.

Abuse of test types
===================

A sample command execution can quickly show the current breakage
when it comes to test types::

  $ avocado list -V /dev/null
  Type       Test                                                                                NOT_A_TEST /dev/null: Not an INSTRUMENTED (avocado.Test based), PyUNITTEST (unittest.TestCase based) or SIMPLE (executable) test
  !GLIB      /dev/null: No GLib-like tests found
  !GOLANG    /dev/null: Go binary not found.
  !ROBOT     /dev/null: Data source does not exist.

  TEST TYPES SUMMARY
  ==================
  !GLIB: 1
  !GOLANG: 1
  !ROBOT: 1
  ACCESS_DENIED: 0
  BROKEN_SYMLINK: 0
  EXTERNAL: 0
  GLIB: 0
  GOLANG: 0
  INSTRUMENTED: 0
  MISSING: 0
  NOT_A_TEST: 1
  PYUNITTEST: 0
  ROBOT: 0
  SIMPLE: 0

A partial list of the problems that this output reveals:

 1) `NOT_A_TEST` and `!ROBOT` (and many others) listed under "Type"
    are not really test types, but representations of the loader
    "findings" on a given test reference, which I'll call "test
    reference resolution result" from now on.  The same applies to
    `ACCESS_DENIED`, `BROKEN_SYMLINK` and `MISSING`, which are
    listed as test types, but in fact are not.
 2) `NOT_A_TEST` is a "test reference resolution result" that is only
    used for `INSTRUMENTED` and `SIMPLE` tests (although the given
    message only makes it clear for the INSTRUMENTED test type)
 3) The "TEST TYPE SUMMARY" reports a total count of 2 for one given
    reference, not because two tests were found, but because they
    were *not* found by two different loaders (FileLoader and
    RobotLoader).

One may argue that having well defined "test types" such as
`ACCESS_DENIED` are useful so that the user can tell what was wrong
with the loading of the test.  But, the fact is that the same
information can be given without categorizing it as a test type.

Expected behavior
=================

The following command output would be produced if the problems listed
previously did not exist.  Fist, if an only a test reference that can
not be resolved into a valid test type were given, output similar to
the following would be expected::

  $ avocado list /dev/null
  Unable to resolve reference(s): '/dev/null'

  Resolution process results:
  ===========================
  Reference   Plugin            Result    Info
  /dev/null   file
  /dev/null   yaml_testsuite
  /dev/null   robot
  /dev/null   external

The key point here is that *no test* was found when a number of
plugins attempted to find one.  Plugins should be free to add
additional information, such as::

  $ avocado list /does/not/exist
  Unable to resolve reference(s): '/does/not/exist'

  Resolution process results:
  ===========================
  Reference         Plugin            Result    Info
  /does/not/exist   file                        FileNotFoundError: [Errno 2] No such file or directory
  /does/not/exist   yaml_testsuite              Failed to parse YAML file
  /does/not/exist   robot
  /does/not/exist   external

Proposed Solution
=================

Instead of delving into too much theory, a PR has been posted with a
RFC implementation of a new resolver architecture.  To avoid
disruption, that PR introduces a "resolve" command (instead of "list")
and hooks itself into the "nrunner" experimental test runner.

Feedback to this and to the PR "[RFC] Introduce avocado.core.resolver
as an alternative to loader" are extremely welcome.