[Pulp-dev] Proposal: Basic Functional Test Requirement with each Feature or Bugfix

[Matthias Dellweg]

+1

On Tue, Aug 18, 2020 at 10:39 AM Pavel Picka <ppicka at redhat.com> wrote:
>
> Big +1 to require (at least) one basic/positive functional test if possible.
> Maybe we can set up a review checklist (contains 'check for basic test').
>
> We already have some docs which we can update a bit [0] to be yet more plugin writer friendly and point back to it from plugins' docs.
> Like extending it with the way the test can be run when pulp is installed e.g. by the rpm package.
>
> [0] https://docs.pulpproject.org/contributing/tests.html
>
> On Mon, Aug 17, 2020 at 10:07 PM Brian Bouterse <bmbouter at redhat.com> wrote:
>>
>> I have two goals with this:
>>
>> 1. Improve the stability of pulpcore and it's plugins
>>
>> 2. Enable all downstreams and packagers with tests and information on how to run those tests, so they can run the automated tests even as the downstream is curated with a series of backports. This closely follows the model of a downstream maintainer who backports a feature/fix from upstream into a RPM and runs the upstream tests to make sure their patch didn't break functionality.
>>
>> # Proposal
>> Upstream could require a basic functional test for each feature or a test for each bug fix. This test would be in the same commit as the feature or fix causing it to "follow the code" so when it's cherry picked downstream or into a package, the test goes with it. I believe we don't change the test during cherry pick time because although the code implementation may change what the feature or fix provides will not.
>>
>> Additionally, upstream should document how to run these tests easily and have the goal of making that easy.
>>
>> This policy change would ultimately be decided by each plugin and pulpcore separately. I don't want to make it a requirement to be part of the github.com/pulp/ organization in any way, so if a plugin can't, or won't, make this change that's ok, they can still host at github.com/pulp/.
>>
>> As a pulpcore and pulp_file maintainer, I'm proposing this policy for both of those repos specifically.
>>
>> In terms of enforcement, I'm pitching manual enforcement by review as a start. We can automate it later, but I hope that's a separate discussion to focus on the policy change and keep it simple.
>>
>> # Brian's take
>> This is a non-trivial contribution requirement so we need to really think about it. My personal take is that at this point in the 3.y release line it's time to trade feature/fix velocity for stability. Also backporting is about to become a regular activity, so I think we have a practical motivation to adopt this, even though it will slow down the future features and bug fix velocity.
>>
>> What's your perspective?
>>
>> _______________________________________________
>> Pulp-dev mailing list
>> Pulp-dev at redhat.com
>> https://www.redhat.com/mailman/listinfo/pulp-dev
>
>
>
> --
> Pavel Picka
> Red Hat
> _______________________________________________
> Pulp-dev mailing list
> Pulp-dev at redhat.com
> https://www.redhat.com/mailman/listinfo/pulp-dev