[sos-devel] Proposal: Synergy of upstream and downstream testing of sos

[Vasant Hegde]

On 11/02/2016 08:09 PM, Louis Bouchard wrote:
> Hello,
>
> Le 02/11/2016 à 15:15, Miroslav Hradilek a écrit :
>> On 11/02/2016 01:20 PM, Miroslav Hradilek wrote:
>>>>> To ensure these tests can be reused, a lot of branching needs to be done
>>>>> within code depending on the environment but mostly on version of the
>>>>> sosreport and patches applied downstream. Code branching is very inefficient
>>>>>
>>>>> and takes a lot of effort. When developing the tests library is developed
>>>>>
>>>>> along with it and needs to be maintained too. Upstream gets only bug reports
>>>>>
>>>>> from these efforts.
>>>>
>>>> It might help some of the other readers on the list to give a brief example,
>>>>
>>>> or overview, of the downstream testing we do in RHEL, and why maintaining
>>>> multiple branches becomes painful.
>>>>
>>>
>>> I guess I could do that. I will sent another email explaining it.
>>
>> We are directed by downstream needs. We get bugs reported, features requested
>> and new plugins asked for downstream.
>>
>> Our version of sosreport is upstream version of sos frozen at some point in time
>> + patches.
>>
>> Patch can fix an issue, add feature or even plugin. Mostly patches are
>> downstream backports of upstream features of sos.
>>
>> If the feature or fix is requested downstream and there is no reason not to, it
>> is first fixed upstream in sos github repo. The fix is used downstream in a form
>> of a patch.
>>
>> To verify that the fix works a test is written downstream. The test is then used
>> to make sure the fix works in later releases too. The most interesting part is
>> when new feature or whole plugin is added. We then write downstream test which
>> is testing that the feature works. Or we are covering most, if not all, of the
>> plugin functionality.
>>
>> If we can extract enough information from reporters and we are able to reproduce
>> the real environment, the test runs on real data. If it is too complex or
>> information is too difficult to obtain, we mock the files and commands. To make
>> writing tests easier we maintain library to run sosreport, grep listings, mock
>> files, cleanup mocks,... downstream.
>>
>> Later in releases, the environment, implementation or a plugin changes. If the
>> change was not anticipated, the test fails and needs to be branched to reflect
>> the change. IF/ELSE branching within code was chosen so that we do not have to
>> maintain SCM branches for the large matrix of (sos version) x (os version) x
>> (environmental dimension 1) x ... (environmental dimension N) [1]. The change
>> can affect the library too.
>>
>> Pain points that I believe could be solved by doing it upstream:
>>
>>   1) If sos itself or it's plugin changes, the test suite reflects this change.
>> We get the test at working state downstream. Benefit: More coverage for upstream
>> and other downstreams.
>>
>>   2) If the implementation in sos changes, the library changes and we get working
>> state of the library downstream. We can write downstream tests using working
>> library functions. Benefit: Common test library API matching the functionality
>> of the current version of sos. Library to support seamless work on real data.
>> Perhaps it could be shielding QE's from downstream differences.
>>
>>   3) If the test is submitted with the plugin and real sample data is used for
>> mocking, we can a) Avoid time consuming investigations. b) base tests on real
>> data so that we can later assert it on real environment. Benefit: More coverage
>> upstream and downstream.
>>
>> ~~~~
>> [1] Environmental dimension can be anything that changes the behavior like
>> (systemd vs systemv).
>>
>

Louis, Bryn, Miroslav,

Thanks for bring up the topic and giving detailed explanation of how you are 
testing today.

The IBM PowerKVM version uses sosreport as default data collection tool. Also we 
recommend sosreport on most of the distros running on Power System and has 
sosreport package.

Our test team does some validation like
   sosreport output generated or not,
   Few sosreport options
   Whether it collected some of the Power specific data or not (ex: Device tree).

As a platform people we have different problem.  We will have different 
sosreport versions on different distros :-(

IMO its good if we have test cases in upstream project itself. Then downstream 
guys can grab upstream tests and add additional testcases if required.

-Vasant