[sos-devel] Proposal: Synergy of upstream and downstream testing of sos

[Miroslav Hradilek]

Hello,

I was thinking about this for a long time and I think that the following 
could work.

Problem in downstream testing:

For downstream, we develop tests for a) specific issues, b) whole 
plugins, c) basic functionality. Unlike sos internal test suite these 
are blackbox integration tests written in bash as opposed to python 
whitebox tests checking the functions themselves. Sometimes the tests 
are run on real environment but very often the files and commands to be 
collected are mocked. To make our lifes easier there are libraries of 
setup functions, mock functions, assertions and so on.

To ensure these tests can be reused, a lot of branching needs to be done 
within code depending on the environment but mostly on version of the 
sosreport and patches applied downstream. Code branching is very 
inefficient and takes a lot of effort. When developing the tests library 
is developed along with it and needs to be maintained too. Upstream gets 
only bug reports from these efforts.

Problems I suggest to solve:
   1. Avoid downstream branching and test library development.
   2. Extend upstream test coverage and test library by downstream testers.
   3. Write tests library functions and tests so that they can be run in 
mocked environment as well as in real environment with the flick of a 
switch.

Extra work I suggest to put on our shoulders:
   1. Developers and plugin contributors forced to modify tests and 
libraries to ensure they work with their commit.
   2. Downstream testers extending the upstream test suite and libraries 
to later use it downstream.

Proposal I make:
   * Let's chose test framework + assertion library, develop our library 
and fixtures upstream. Develop at least part of the integration test 
suite [ b) and c) ] upstream and use upstream library for a) downstream.
   * Write tests and mock functions so that they use real file chunks 
and, by change of the environment, setups can be disabled therefore 
collecting and asserting real files and commands.
* Require integration test suite to pass in order to accept commit. 
Encourage submitting integration tests with new plugins and plugins 
functionality.


What do you think?


-- 
Miroslav Hradilek
Quality Assurance Engineer
Base OS Quality Engineering

Red Hat Czech, s. r. o.
Purkynova 99
612 45 Brno, Czech Republic