[Avocado-devel] [RFC] Pre/Post test hooks

[Jeff Nelson]

On Fri, Apr 01, 2016 at 11:00:39AM -0300, Cleber Rosa wrote:
>MOTIVATION
>==========
>
>The idea of adding hooks to be run by Avocado before and after tests 
>is general enough, and may be used by the community in unpredictable 
>ways. And that is good.
>
>For this team, the initial motivation was to be able to bring back an 
>Autotest feature that some of our users are missing: the ability to 
>set the system-wide "kernel core pattern" configuration for tests.
>
>Having a pre-test hook would allow "/proc/sys/kernel/core_pattern" to 
>be read, saved and modified to point to the test results directory. 
>Having a post-test hook would allow "/proc/sys/kernel/core_pattern" to 
>be reverted back to its original state.
>
>Other currently core features such as sysinfo collection, could be 
>re-implemented as pre/post test hooks.
>
>GENERAL DESIGN POINTS
>=====================
>
>These are the most important design decisions to be acknowledged or 
>questioned. Please reply with either ACK or your 
>questions/suggestions.

I have some questions (hope you don't mind).

What are the outputs of pre- and post-test hooks?

Are there limits to the actions that are permitted in pre- and
post-test hooks? Of course, the primary use-case of the pre-test hook
is to set up the environment for the test--so environment changes are
permitted--and the use-case for a matching post-hook is to restore the
environment. About the only operation I can imagine NOT being
permitted is to abort (kill itself, or kill its controlling parent
process).

Can a pre-test hook return a status that causes the test execution to
be skipped? I can imagine this being done for another use-case:
validate the test environment (e.g., check to see if required hardware
is present).

Can a post-test hook alter the result (status) of the test?

Has there been any thought given to having pre- and post-job hooks?
For example, setting the kernel core pattern is something I would want
to do globally, for all tests in a job. It would be faster and more
convenient to do this just once. But I admit this is a +1 optimization
and need not be considered now. (It also complicates things when tests
run on multiple machines.)

Can there be multiple hooks for a given test? If so, how does one
define the order in which they are executed? Since there are pre- and
post-test hooks, there are really two orders to consider.

I found myself wanting to make an assumption so I better ask: must
hooks come in pairs (for every pre-test there must be a post-test, and
vice-versa)?

That's all for now.

-Jeff



>
>1) Hooks are implemented as plugin classes, based on a given defined 
>interface, in the same way current "CLICmd" and "CLI" interfaces allow 
>plugin writers to extend Avocado and give it new commands and command 
>line options.
>
>2) The hooks are executed by the *runner*, and not by the test 
>process. The goal is not interfere with the test itself. The pre and 
>post code that runs before and after the test should not *directly* 
>change the test behavior and outcome. Of course, the test environment 
>can be changed in a way (say having packages removed) that a test may 
>fail because of hook actions.
>
>3) Test execution time should not be changed by pre and post hooks. If 
>a pre-test hook takes "n" seconds to run, "n" should not be added to 
>the test run time.
>
>4) Job run time: right now, Avocado times a Job based on the sum of 
>individual test run times. With pre and post test hooks, this can be 
>very different from job "wall clock" times. My instinct is to change 
>that, so that a Job run time is the job "wall clock" time. I'm unsure 
>if we should add yet another time measure, that is, the sum of 
>individual test run time. This is also bound to be broken when 
>parallel run of tests is implemented.
>
>5) The pre test hook is given the test "early status". Information 
>such as the test tagged name, the fact that it has not yet started to 
>run and the test results directory are all part of the early status.
>
>6) Because of point #5, the test is instantiated on the test process, 
>its early state is sent, but the test execution itself is held until 
>the runner finishes running the pre-test hooks.
>
>7) The post test hook is given the last test status, which is also 
>used by the runner to identify test success, failure, etc.
>
>
>Thanks,
> - Cleber.
>
>-- 
>Cleber Rosa
>[ Sr Software Engineer - Virtualization Team - Red Hat ]
>[ Avocado Test Framework - avocado-framework.github.io ]
>
>_______________________________________________
>Avocado-devel mailing list
>Avocado-devel at redhat.com
>https://www.redhat.com/mailman/listinfo/avocado-devel