[Avocado-devel] [RFC] Pre/Post test hooks

[Cleber Rosa]

On 04/05/2016 03:23 PM, Jeff Nelson wrote:
> On Fri, Apr 01, 2016 at 11:00:39AM -0300, Cleber Rosa wrote:
>> MOTIVATION
>> ==========
>>
>> The idea of adding hooks to be run by Avocado before and after tests
>> is general enough, and may be used by the community in unpredictable
>> ways. And that is good.
>>
>> For this team, the initial motivation was to be able to bring back an
>> Autotest feature that some of our users are missing: the ability to
>> set the system-wide "kernel core pattern" configuration for tests.
>>
>> Having a pre-test hook would allow "/proc/sys/kernel/core_pattern" to
>> be read, saved and modified to point to the test results directory.
>> Having a post-test hook would allow "/proc/sys/kernel/core_pattern" to
>> be reverted back to its original state.
>>
>> Other currently core features such as sysinfo collection, could be
>> re-implemented as pre/post test hooks.
>>
>> GENERAL DESIGN POINTS
>> =====================
>>
>> These are the most important design decisions to be acknowledged or
>> questioned. Please reply with either ACK or your questions/suggestions.
>
> I have some questions (hope you don't mind).
>
> What are the outputs of pre- and post-test hooks?
>

The outputs are defined by the actual plugin "hooked in the hook". The 
current interface doesn't define it. They can be completely silent, they 
can generate output to the UI, they can write to files at the test's 
result directory, etc.

> Are there limits to the actions that are permitted in pre- and
> post-test hooks? Of course, the primary use-case of the pre-test hook
> is to set up the environment for the test--so environment changes are
> permitted--and the use-case for a matching post-hook is to restore the
> environment. About the only operation I can imagine NOT being
> permitted is to abort (kill itself, or kill its controlling parent
> process).
>

The pre/post test hooks are fed the test status, so they can gather 
information, but not (directly) influence its outcome.

> Can a pre-test hook return a status that causes the test execution to
> be skipped? I can imagine this being done for another use-case:
> validate the test environment (e.g., check to see if required hardware
> is present).
>

No, this should be done at the test's "setUp" stage. Pointers:

https://github.com/avocado-framework/avocado/blob/master/avocado/core/test.py#L368

But I understand the value in influencing all tests with a single, 
plugable, block of code. Still, this looks like something that 
could/should be addressed by what we're calling "Job API". With some 
Python code and such an API, you could select the exact tests you want 
to run.

> Can a post-test hook alter the result (status) of the test?
>

In the current implementation draft it could. But just because we're 
sharing the same (last, final) test state among the post plugins and 
test result handlers. I'm not sure that a plugin *should* change it, though.

> Has there been any thought given to having pre- and post-job hooks?
> For example, setting the kernel core pattern is something I would want
> to do globally, for all tests in a job. It would be faster and more
> convenient to do this just once. But I admit this is a +1 optimization
> and need not be considered now. (It also complicates things when tests
> run on multiple machines.)

Pre/Post job hooks are actually simpler, and implemented in the 
following PR:

https://github.com/avocado-framework/avocado/pull/1106

>
> Can there be multiple hooks for a given test? If so, how does one
> define the order in which they are executed? Since there are pre- and
> post-test hooks, there are really two orders to consider.
>

You mean multiple plugins registered under a given hook, or simply put, 
multiple pre (and/or) post plugins active, right? The answer is yes.

We have not implemented any kind of ordering management, simply because 
we have not hit a use case that requires it.

> I found myself wanting to make an assumption so I better ask: must
> hooks come in pairs (for every pre-test there must be a post-test, and
> vice-versa)?

Not at all. You can have only pre hooks enabled, only post hooks enabled 
or both.

The given *job* mail notification plugin example, for instance, is 
something that makes more sense activated only in a post hook:

https://github.com/clebergnu/avocado/blob/job_pre_post/examples/plugins/job-pre-post/mail/avocado_job_mail.py

>
> That's all for now.
>
> -Jeff
>

Thanks a lot for the feedback!
- Cleber.