[sos-devel] Proposal: Synergy of upstream and downstream testing of sos

[Miroslav Hradilek]

On 11/02/2016 01:20 PM, Miroslav Hradilek wrote:
>>> To ensure these tests can be reused, a lot of branching needs to be done
>>> within code depending on the environment but mostly on version of the
>>> sosreport and patches applied downstream. Code branching is very inefficient
>>>
>>> and takes a lot of effort. When developing the tests library is developed
>>>
>>> along with it and needs to be maintained too. Upstream gets only bug reports
>>>
>>> from these efforts.
>>
>> It might help some of the other readers on the list to give a brief example,
>>
>> or overview, of the downstream testing we do in RHEL, and why maintaining
>> multiple branches becomes painful.
>>
>
> I guess I could do that. I will sent another email explaining it.

We are directed by downstream needs. We get bugs reported, features 
requested and new plugins asked for downstream.

Our version of sosreport is upstream version of sos frozen at some point 
in time + patches.

Patch can fix an issue, add feature or even plugin. Mostly patches are 
downstream backports of upstream features of sos.

If the feature or fix is requested downstream and there is no reason not 
to, it is first fixed upstream in sos github repo. The fix is used 
downstream in a form of a patch.

To verify that the fix works a test is written downstream. The test is 
then used to make sure the fix works in later releases too. The most 
interesting part is when new feature or whole plugin is added. We then 
write downstream test which is testing that the feature works. Or we are 
covering most, if not all, of the plugin functionality.

If we can extract enough information from reporters and we are able to 
reproduce the real environment, the test runs on real data. If it is too 
complex or information is too difficult to obtain, we mock the files and 
commands. To make writing tests easier we maintain library to run 
sosreport, grep listings, mock files, cleanup mocks,... downstream.

Later in releases, the environment, implementation or a plugin changes. 
If the change was not anticipated, the test fails and needs to be 
branched to reflect the change. IF/ELSE branching within code was chosen 
so that we do not have to maintain SCM branches for the large matrix of 
(sos version) x (os version) x (environmental dimension 1) x ... 
(environmental dimension N) [1]. The change can affect the library too.

Pain points that I believe could be solved by doing it upstream:

  1) If sos itself or it's plugin changes, the test suite reflects this 
change. We get the test at working state downstream. Benefit: More 
coverage for upstream and other downstreams.

  2) If the implementation in sos changes, the library changes and we 
get working state of the library downstream. We can write downstream 
tests using working library functions. Benefit: Common test library API 
matching the functionality of the current version of sos. Library to 
support seamless work on real data. Perhaps it could be shielding QE's 
from downstream differences.

  3) If the test is submitted with the plugin and real sample data is 
used for mocking, we can a) Avoid time consuming investigations. b) base 
tests on real data so that we can later assert it on real environment. 
Benefit: More coverage upstream and downstream.

~~~~
[1] Environmental dimension can be anything that changes the behavior 
like (systemd vs systemv).

-- 
Miroslav Hradilek
Quality Assurance Engineer
Base OS Quality Engineering

Red Hat Czech, s. r. o.
Purkynova 99
612 45 Brno, Czech Republic