[Pulp-dev] Proposal: Basic Functional Test Requirement with each Feature or Bugfix

[Fabricio Aguiar]

I think we can do silly automation in one specific plugin, after some
trial, expand the idea or not:

pulpcore on  6844 [$] via 🐍 v3.8.0 (venv)
❯ git diff --name-only master
CHANGES/6844.feature
functest_requirements.txt
setup.py

pulpcore on  6844 [$] via 🐍 v3.8.0 (venv) took 2s
❯ git diff --name-only master | grep -E 'feature|bugfix'
CHANGES/6844.feature

pulpcore on  6844 [$] via 🐍 v3.8.0 (venv)
❯ git diff --name-only master | grep -E '^test_'

Best regards,
Fabricio Aguiar
Software Engineer, Pulp Project
Red Hat Brazil - Latam <https://www.redhat.com/>
+55 11 999652368


On Tue, Aug 18, 2020 at 11:08 AM David Davis <daviddavis at redhat.com> wrote:

> +1 from me.
>
> One of the problems I foresee though is that this could make cherry
> picking difficult if we have tests that depend on other tests. For example,
> suppose you have change A that adds some tests and then change B adds some
> tests on top of A's tests. It'll make cherry picking B without A tricky.
>
> We'll need to be careful to avoid such situations.
>
> David
>
>
> On Tue, Aug 18, 2020 at 4:48 AM Matthias Dellweg <mdellweg at redhat.com>
> wrote:
>
>> +1
>>
>> On Tue, Aug 18, 2020 at 10:39 AM Pavel Picka <ppicka at redhat.com> wrote:
>> >
>> > Big +1 to require (at least) one basic/positive functional test if
>> possible.
>> > Maybe we can set up a review checklist (contains 'check for basic
>> test').
>> >
>> > We already have some docs which we can update a bit [0] to be yet more
>> plugin writer friendly and point back to it from plugins' docs.
>> > Like extending it with the way the test can be run when pulp is
>> installed e.g. by the rpm package.
>> >
>> > [0] https://docs.pulpproject.org/contributing/tests.html
>> >
>> > On Mon, Aug 17, 2020 at 10:07 PM Brian Bouterse <bmbouter at redhat.com>
>> wrote:
>> >>
>> >> I have two goals with this:
>> >>
>> >> 1. Improve the stability of pulpcore and it's plugins
>> >>
>> >> 2. Enable all downstreams and packagers with tests and information on
>> how to run those tests, so they can run the automated tests even as the
>> downstream is curated with a series of backports. This closely follows the
>> model of a downstream maintainer who backports a feature/fix from upstream
>> into a RPM and runs the upstream tests to make sure their patch didn't
>> break functionality.
>> >>
>> >> # Proposal
>> >> Upstream could require a basic functional test for each feature or a
>> test for each bug fix. This test would be in the same commit as the feature
>> or fix causing it to "follow the code" so when it's cherry picked
>> downstream or into a package, the test goes with it. I believe we don't
>> change the test during cherry pick time because although the code
>> implementation may change what the feature or fix provides will not.
>> >>
>> >> Additionally, upstream should document how to run these tests easily
>> and have the goal of making that easy.
>> >>
>> >> This policy change would ultimately be decided by each plugin and
>> pulpcore separately. I don't want to make it a requirement to be part of
>> the github.com/pulp/ organization in any way, so if a plugin can't, or
>> won't, make this change that's ok, they can still host at
>> github.com/pulp/.
>> >>
>> >> As a pulpcore and pulp_file maintainer, I'm proposing this policy for
>> both of those repos specifically.
>> >>
>> >> In terms of enforcement, I'm pitching manual enforcement by review as
>> a start. We can automate it later, but I hope that's a separate discussion
>> to focus on the policy change and keep it simple.
>> >>
>> >> # Brian's take
>> >> This is a non-trivial contribution requirement so we need to really
>> think about it. My personal take is that at this point in the 3.y release
>> line it's time to trade feature/fix velocity for stability. Also
>> backporting is about to become a regular activity, so I think we have a
>> practical motivation to adopt this, even though it will slow down the
>> future features and bug fix velocity.
>> >>
>> >> What's your perspective?
>> >>
>> >> _______________________________________________
>> >> Pulp-dev mailing list
>> >> Pulp-dev at redhat.com
>> >> https://www.redhat.com/mailman/listinfo/pulp-dev
>> >
>> >
>> >
>> > --
>> > Pavel Picka
>> > Red Hat
>> > _______________________________________________
>> > Pulp-dev mailing list
>> > Pulp-dev at redhat.com
>> > https://www.redhat.com/mailman/listinfo/pulp-dev
>>
>>
>> _______________________________________________
>> Pulp-dev mailing list
>> Pulp-dev at redhat.com
>> https://www.redhat.com/mailman/listinfo/pulp-dev
>>
>> _______________________________________________
> Pulp-dev mailing list
> Pulp-dev at redhat.com
> https://www.redhat.com/mailman/listinfo/pulp-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/pulp-dev/attachments/20200818/351b6342/attachment.htm>