[Avocado-devel] RFC: "output check" record/testing

[Ademar Reis]

https://trello.com/c/1pSOhco8/181-output-check-record-testing

Some projects use an approach for testing that works like this:

 * Collect the stdout/stderr of the test (and all processes
   started by it)
 * Diff it against a reference file

We should provide a convenience in avocado for this kind of
testing. What I'm proposing is that avocado supports this out of
the box for any test if the user records a reference file. When
running the test, a non-empty "diff" is considered a fail.
Naturally, our logs would include the stdout/stderr as usual,
plus the diff that is responsible for the failure.

Proposal:

  * Add --output-check-record{=all,stdout,stderr} (default = all)
    to avocado run: when this option is used, avocado will, for
    every test being run, record one or two files in the test
    data dir called "stdout.expected" and/or "stderr.expected"
    (I'm assuming their contents is obvious);
    * Datadir should be created if it doesn't exist;
    * Existing files should be overwritten;
    * No output means an empty file (which is different than a
      non-existing file);
  * When avocado runs a test that has one or two of these
    .expected files in its datadir, avocado will, besides running
    the test as usual, compare the test stdout/stderr to the
    reference files and FAIL the test if there's any difference;
  * Add --disable-output-check to disable the behavior mentioned
    above;
  * Add an API option that allows the test writer to start a
    program with record disabled (a parameter to the existing API
    that starts a program, something like
    allow_output_record={all,none,stdout,stderr}, default is
    all).
    * When --output-check-record is used but a program output is
      not recorded due to a restriction imposed by the test
      writer using the API above, a warning should be shown to
      the user.

The feature should work with all kinds of tests (including simple
shell scripts - the so called "drop-in tests").

As a side note, apparently if users print something to
stdout/stderr from inside a test, it "leaks" to the console. This
should be changed: stdout/stderr, besides being handled as
described above, should go to log files instead.

Thanks!
   - Ademar

-- 
Ademar de Souza Reis Jr.
Red Hat

^[:wq!