From mhradile at redhat.com Wed Nov 2 12:20:16 2016 From: mhradile at redhat.com (Miroslav Hradilek) Date: Wed, 2 Nov 2016 13:20:16 +0100 Subject: [sos-devel] Proposal: Synergy of upstream and downstream testing of sos In-Reply-To: <20161026111210.GB9882@dhcp-24-182.fab.redhat.com> References: <20161026111210.GB9882@dhcp-24-182.fab.redhat.com> Message-ID: <8cc72725-1e0e-63c9-7d08-4c71c699ab55@redhat.com> Hello Bryn, answers inline: On 10/26/2016 01:12 PM, Bryn M. Reeves wrote: > On Wed, Oct 26, 2016 at 10:41:51AM +0200, Miroslav Hradilek wrote: >> For downstream, we develop tests for a) specific issues, b) whole plugins, >> c) basic functionality. Unlike sos internal test suite these are blackbox >> integration tests written in bash as opposed to python whitebox tests >> checking the functions themselves. Sometimes the tests are run on real >> environment but very often the files and commands to be collected are >> mocked. To make our lifes easier there are libraries of setup functions, >> mock functions, assertions and so on. > > Right: these are integration tests (that work on 'sosreport' as a whole), > where the upstream suite is primarily unit testing. > > One thing to remember here though: there are more downstreams than just the > Red Hat / Fedora world today: sos is actively maintained in both Debian and > Ubuntu, is used in several hypervisor products (notably IBM's PowerKVM), > and has at least some users on SuSE and other distributions. > Yes, I'm not planing to test these but I believe that their respective QE people can too benefit from the tests being maintained upstream. People who actually do the testing on those systems would have to keep an eye on the test code so that it runs smoothly on their systems too. I'd like the test code and libraries to support this variability. >> To ensure these tests can be reused, a lot of branching needs to be done >> within code depending on the environment but mostly on version of the >> sosreport and patches applied downstream. Code branching is very inefficient >> and takes a lot of effort. When developing the tests library is developed >> along with it and needs to be maintained too. Upstream gets only bug reports >> from these efforts. > > It might help some of the other readers on the list to give a brief example, > or overview, of the downstream testing we do in RHEL, and why maintaining > multiple branches becomes painful. > I guess I could do that. I will sent another email explaining it. One thing to note now is that we are not maintaining multiple branches. Instead we branch the differences in the test code using IF/ELSE statements where the test differs. Maintaining different git branch for each version and environment would make a really large matrix of test code to backport patches into. This wold be the major benefit of the test being maintained upstream. There would no longer be need for any IF statement caused by sosreport version differences because the test would be working just for the version of sosreport source the test suite is part of. There would still be need for branching (IF/ELSE or git) based on the environmental differences. >> Problems I suggest to solve: >> 1. Avoid downstream branching and test library development. >> 2. Extend upstream test coverage and test library by downstream testers. >> 3. Write tests library functions and tests so that they can be run in >> mocked environment as well as in real environment with the flick of a >> switch. > > I think this is an excellent goal. We tried a few years ago to get a docker > based testing system together, that was trying to address similar needs: > > - coverage of multiple distributions (policies/environments) > - reproducible environments for testing > - repeatable tests > > Some of the ideas were tracked in the following GitHub issue: > > https://github.com/sosreport/sos/issues/335 > Unfortunately, I do not plan to solve these for sos: * The way to obtain, prepare, maintain images of environment setups. * Scheduling (test) x (environment) matrices. * Collecting, interpreting and reviewing test results. * ... What I'd want to work on is tests and test libraries that will run on any system (eg. CI, developers workstation...) via mocking and will work seamlessly on real environments when the things mentioned above are solved by sos. >> Extra work I suggest to put on our shoulders: >> 1. Developers and plugin contributors forced to modify tests and libraries >> to ensure they work with their commit. > > Ack. This is something we've gotten a little better at for API changes - > asking developers to also amend the unit test suite - but extending this > to also include integration testing would be really useful. > Cool. >> 2. Downstream testers extending the upstream test suite and libraries to >> later use it downstream. >> >> Proposal I make: >> * Let's chose test framework + assertion library, develop our library and >> fixtures upstream. Develop at least part of the integration test suite [ b) >> and c) ] upstream and use upstream library for a) downstream. >> * Write tests and mock functions so that they use real file chunks and, by >> change of the environment, setups can be disabled therefore collecting and >> asserting real files and commands. >> * Require integration test suite to pass in order to accept commit. >> Encourage submitting integration tests with new plugins and plugins >> functionality. > > I'm curious what it would look like, and how we would manage to support > the range of different downstreams from a single upstream testing base, > but I think it's definitely worth more investigation. > > As you're more experienced with this kind of testing, and the available > tools than most of the team, would you be OK to write up a more detailed > proposal, or something that demos how some of this would work? > Thanks, I will try to do this and let you know when I have something. I needed to know that it makes sense to put an effort to it. > Regards, > Bryn. > Hopefully you are not disappointed that my goals are actually less ambitious then they sounded. -- Miroslav Hradilek Quality Assurance Engineer Base OS Quality Engineering Red Hat Czech, s. r. o. Purkynova 99 612 45 Brno, Czech Republic From mhradile at redhat.com Wed Nov 2 14:15:10 2016 From: mhradile at redhat.com (Miroslav Hradilek) Date: Wed, 2 Nov 2016 15:15:10 +0100 Subject: [sos-devel] Proposal: Synergy of upstream and downstream testing of sos In-Reply-To: <8cc72725-1e0e-63c9-7d08-4c71c699ab55@redhat.com> References: <20161026111210.GB9882@dhcp-24-182.fab.redhat.com> <8cc72725-1e0e-63c9-7d08-4c71c699ab55@redhat.com> Message-ID: <816551db-e123-606c-f031-fc89b726bb59@redhat.com> On 11/02/2016 01:20 PM, Miroslav Hradilek wrote: >>> To ensure these tests can be reused, a lot of branching needs to be done >>> within code depending on the environment but mostly on version of the >>> sosreport and patches applied downstream. Code branching is very inefficient >>> >>> and takes a lot of effort. When developing the tests library is developed >>> >>> along with it and needs to be maintained too. Upstream gets only bug reports >>> >>> from these efforts. >> >> It might help some of the other readers on the list to give a brief example, >> >> or overview, of the downstream testing we do in RHEL, and why maintaining >> multiple branches becomes painful. >> > > I guess I could do that. I will sent another email explaining it. We are directed by downstream needs. We get bugs reported, features requested and new plugins asked for downstream. Our version of sosreport is upstream version of sos frozen at some point in time + patches. Patch can fix an issue, add feature or even plugin. Mostly patches are downstream backports of upstream features of sos. If the feature or fix is requested downstream and there is no reason not to, it is first fixed upstream in sos github repo. The fix is used downstream in a form of a patch. To verify that the fix works a test is written downstream. The test is then used to make sure the fix works in later releases too. The most interesting part is when new feature or whole plugin is added. We then write downstream test which is testing that the feature works. Or we are covering most, if not all, of the plugin functionality. If we can extract enough information from reporters and we are able to reproduce the real environment, the test runs on real data. If it is too complex or information is too difficult to obtain, we mock the files and commands. To make writing tests easier we maintain library to run sosreport, grep listings, mock files, cleanup mocks,... downstream. Later in releases, the environment, implementation or a plugin changes. If the change was not anticipated, the test fails and needs to be branched to reflect the change. IF/ELSE branching within code was chosen so that we do not have to maintain SCM branches for the large matrix of (sos version) x (os version) x (environmental dimension 1) x ... (environmental dimension N) [1]. The change can affect the library too. Pain points that I believe could be solved by doing it upstream: 1) If sos itself or it's plugin changes, the test suite reflects this change. We get the test at working state downstream. Benefit: More coverage for upstream and other downstreams. 2) If the implementation in sos changes, the library changes and we get working state of the library downstream. We can write downstream tests using working library functions. Benefit: Common test library API matching the functionality of the current version of sos. Library to support seamless work on real data. Perhaps it could be shielding QE's from downstream differences. 3) If the test is submitted with the plugin and real sample data is used for mocking, we can a) Avoid time consuming investigations. b) base tests on real data so that we can later assert it on real environment. Benefit: More coverage upstream and downstream. ~~~~ [1] Environmental dimension can be anything that changes the behavior like (systemd vs systemv). -- Miroslav Hradilek Quality Assurance Engineer Base OS Quality Engineering Red Hat Czech, s. r. o. Purkynova 99 612 45 Brno, Czech Republic From louis.bouchard at canonical.com Wed Nov 2 14:39:08 2016 From: louis.bouchard at canonical.com (Louis Bouchard) Date: Wed, 2 Nov 2016 15:39:08 +0100 Subject: [sos-devel] Proposal: Synergy of upstream and downstream testing of sos In-Reply-To: <816551db-e123-606c-f031-fc89b726bb59@redhat.com> References: <20161026111210.GB9882@dhcp-24-182.fab.redhat.com> <8cc72725-1e0e-63c9-7d08-4c71c699ab55@redhat.com> <816551db-e123-606c-f031-fc89b726bb59@redhat.com> Message-ID: <51a041cd-5cbc-4cc0-dfca-52792adb9c82@canonical.com> Hello, Le 02/11/2016 ? 15:15, Miroslav Hradilek a ?crit : > On 11/02/2016 01:20 PM, Miroslav Hradilek wrote: >>>> To ensure these tests can be reused, a lot of branching needs to be done >>>> within code depending on the environment but mostly on version of the >>>> sosreport and patches applied downstream. Code branching is very inefficient >>>> >>>> and takes a lot of effort. When developing the tests library is developed >>>> >>>> along with it and needs to be maintained too. Upstream gets only bug reports >>>> >>>> from these efforts. >>> >>> It might help some of the other readers on the list to give a brief example, >>> >>> or overview, of the downstream testing we do in RHEL, and why maintaining >>> multiple branches becomes painful. >>> >> >> I guess I could do that. I will sent another email explaining it. > > We are directed by downstream needs. We get bugs reported, features requested > and new plugins asked for downstream. > > Our version of sosreport is upstream version of sos frozen at some point in time > + patches. > > Patch can fix an issue, add feature or even plugin. Mostly patches are > downstream backports of upstream features of sos. > > If the feature or fix is requested downstream and there is no reason not to, it > is first fixed upstream in sos github repo. The fix is used downstream in a form > of a patch. > > To verify that the fix works a test is written downstream. The test is then used > to make sure the fix works in later releases too. The most interesting part is > when new feature or whole plugin is added. We then write downstream test which > is testing that the feature works. Or we are covering most, if not all, of the > plugin functionality. > > If we can extract enough information from reporters and we are able to reproduce > the real environment, the test runs on real data. If it is too complex or > information is too difficult to obtain, we mock the files and commands. To make > writing tests easier we maintain library to run sosreport, grep listings, mock > files, cleanup mocks,... downstream. > > Later in releases, the environment, implementation or a plugin changes. If the > change was not anticipated, the test fails and needs to be branched to reflect > the change. IF/ELSE branching within code was chosen so that we do not have to > maintain SCM branches for the large matrix of (sos version) x (os version) x > (environmental dimension 1) x ... (environmental dimension N) [1]. The change > can affect the library too. > > Pain points that I believe could be solved by doing it upstream: > > 1) If sos itself or it's plugin changes, the test suite reflects this change. > We get the test at working state downstream. Benefit: More coverage for upstream > and other downstreams. > > 2) If the implementation in sos changes, the library changes and we get working > state of the library downstream. We can write downstream tests using working > library functions. Benefit: Common test library API matching the functionality > of the current version of sos. Library to support seamless work on real data. > Perhaps it could be shielding QE's from downstream differences. > > 3) If the test is submitted with the plugin and real sample data is used for > mocking, we can a) Avoid time consuming investigations. b) base tests on real > data so that we can later assert it on real environment. Benefit: More coverage > upstream and downstream. > > ~~~~ > [1] Environmental dimension can be anything that changes the behavior like > (systemd vs systemv). > The Ubuntu/Debian downstream requirements are somewhat similar and slightly different between Ubuntu and Debian. Debian Sid contains a frozen version of sosreport (currently 3.3) plus patches. Plugins are added as patches above the frozen version. This version is synchronized to Ubuntu's development version (currently Zesty). When bugs are found, they are either fixed in Debian, synchronized to Ubuntu development release and fixes are brought in the stable releases using the "Stable Release Update (SRU)" process to stable releases. This introduces an increasing gap between the stable releases and Debian/Ubuntu-devel as the time goes by. This is not a standard procedure, but merely the side effect of having the same person maintaining sosreport in Debian and Ubuntu (i.e. me). Unit tests are run during the packaging phase. We also recently introduced DEP8 tests[1] (aka autopkgtests). Those are executed when the package builds on Ubuntu as well as each time a dependancy of sosreport gets built. Our plan is to propose those tests to the upstream project so they can also be run and improved. Currently, the DEP8 tests[2] verify that : - sosreport doesn't run plugins as non-root - sosreport can create a tarball - tarball's MD5 matches the tarball's md5 - Default plugin run does not produces errors Adding the DEP8 tests means that a sosreport that closely matches the upstream version can be pushed to the stable releases while being sure that no regression is introduced in basic functionalities. They are currently debian specific, but could easily be made part of the upstream tree and used by debian. I hope that this description can be helpful to others, Kind regards, ...Louis [1] https://anonscm.debian.org/cgit/autopkgtest/autopkgtest.git/tree/doc/README.package-tests.rst [2] https://anonscm.debian.org/cgit/collab-maint/sosreport.git/tree/debian/tests -- Louis Bouchard Software engineer, Cloud & Sustaining eng. Canonical Ltd Ubuntu developer Debian Maintainer GPG : 429D 7A3B DD05 B6F8 AF63 B9C4 8B3D 867C 823E 7A61 From hegdevasant at linux.vnet.ibm.com Thu Nov 3 08:27:17 2016 From: hegdevasant at linux.vnet.ibm.com (Vasant Hegde) Date: Thu, 3 Nov 2016 13:57:17 +0530 Subject: [sos-devel] Proposal: Synergy of upstream and downstream testing of sos In-Reply-To: <51a041cd-5cbc-4cc0-dfca-52792adb9c82@canonical.com> References: <20161026111210.GB9882@dhcp-24-182.fab.redhat.com> <8cc72725-1e0e-63c9-7d08-4c71c699ab55@redhat.com> <816551db-e123-606c-f031-fc89b726bb59@redhat.com> <51a041cd-5cbc-4cc0-dfca-52792adb9c82@canonical.com> Message-ID: <581AF4E5.7090105@linux.vnet.ibm.com> On 11/02/2016 08:09 PM, Louis Bouchard wrote: > Hello, > > Le 02/11/2016 ? 15:15, Miroslav Hradilek a ?crit : >> On 11/02/2016 01:20 PM, Miroslav Hradilek wrote: >>>>> To ensure these tests can be reused, a lot of branching needs to be done >>>>> within code depending on the environment but mostly on version of the >>>>> sosreport and patches applied downstream. Code branching is very inefficient >>>>> >>>>> and takes a lot of effort. When developing the tests library is developed >>>>> >>>>> along with it and needs to be maintained too. Upstream gets only bug reports >>>>> >>>>> from these efforts. >>>> >>>> It might help some of the other readers on the list to give a brief example, >>>> >>>> or overview, of the downstream testing we do in RHEL, and why maintaining >>>> multiple branches becomes painful. >>>> >>> >>> I guess I could do that. I will sent another email explaining it. >> >> We are directed by downstream needs. We get bugs reported, features requested >> and new plugins asked for downstream. >> >> Our version of sosreport is upstream version of sos frozen at some point in time >> + patches. >> >> Patch can fix an issue, add feature or even plugin. Mostly patches are >> downstream backports of upstream features of sos. >> >> If the feature or fix is requested downstream and there is no reason not to, it >> is first fixed upstream in sos github repo. The fix is used downstream in a form >> of a patch. >> >> To verify that the fix works a test is written downstream. The test is then used >> to make sure the fix works in later releases too. The most interesting part is >> when new feature or whole plugin is added. We then write downstream test which >> is testing that the feature works. Or we are covering most, if not all, of the >> plugin functionality. >> >> If we can extract enough information from reporters and we are able to reproduce >> the real environment, the test runs on real data. If it is too complex or >> information is too difficult to obtain, we mock the files and commands. To make >> writing tests easier we maintain library to run sosreport, grep listings, mock >> files, cleanup mocks,... downstream. >> >> Later in releases, the environment, implementation or a plugin changes. If the >> change was not anticipated, the test fails and needs to be branched to reflect >> the change. IF/ELSE branching within code was chosen so that we do not have to >> maintain SCM branches for the large matrix of (sos version) x (os version) x >> (environmental dimension 1) x ... (environmental dimension N) [1]. The change >> can affect the library too. >> >> Pain points that I believe could be solved by doing it upstream: >> >> 1) If sos itself or it's plugin changes, the test suite reflects this change. >> We get the test at working state downstream. Benefit: More coverage for upstream >> and other downstreams. >> >> 2) If the implementation in sos changes, the library changes and we get working >> state of the library downstream. We can write downstream tests using working >> library functions. Benefit: Common test library API matching the functionality >> of the current version of sos. Library to support seamless work on real data. >> Perhaps it could be shielding QE's from downstream differences. >> >> 3) If the test is submitted with the plugin and real sample data is used for >> mocking, we can a) Avoid time consuming investigations. b) base tests on real >> data so that we can later assert it on real environment. Benefit: More coverage >> upstream and downstream. >> >> ~~~~ >> [1] Environmental dimension can be anything that changes the behavior like >> (systemd vs systemv). >> > Louis, Bryn, Miroslav, Thanks for bring up the topic and giving detailed explanation of how you are testing today. The IBM PowerKVM version uses sosreport as default data collection tool. Also we recommend sosreport on most of the distros running on Power System and has sosreport package. Our test team does some validation like sosreport output generated or not, Few sosreport options Whether it collected some of the Power specific data or not (ex: Device tree). As a platform people we have different problem. We will have different sosreport versions on different distros :-( IMO its good if we have test cases in upstream project itself. Then downstream guys can grab upstream tests and add additional testcases if required. -Vasant