[Pulp-dev] Proposal: Basic Functional Test Requirement with each Feature or Bugfix

[David Davis]

+1 from me.

One of the problems I foresee though is that this could make cherry picking
difficult if we have tests that depend on other tests. For example, suppose
you have change A that adds some tests and then change B adds some tests on
top of A's tests. It'll make cherry picking B without A tricky.

We'll need to be careful to avoid such situations.

David


On Tue, Aug 18, 2020 at 4:48 AM Matthias Dellweg <mdellweg at redhat.com>
wrote:

> +1
>
> On Tue, Aug 18, 2020 at 10:39 AM Pavel Picka <ppicka at redhat.com> wrote:
> >
> > Big +1 to require (at least) one basic/positive functional test if
> possible.
> > Maybe we can set up a review checklist (contains 'check for basic test').
> >
> > We already have some docs which we can update a bit [0] to be yet more
> plugin writer friendly and point back to it from plugins' docs.
> > Like extending it with the way the test can be run when pulp is
> installed e.g. by the rpm package.
> >
> > [0] https://docs.pulpproject.org/contributing/tests.html
> >
> > On Mon, Aug 17, 2020 at 10:07 PM Brian Bouterse <bmbouter at redhat.com>
> wrote:
> >>
> >> I have two goals with this:
> >>
> >> 1. Improve the stability of pulpcore and it's plugins
> >>
> >> 2. Enable all downstreams and packagers with tests and information on
> how to run those tests, so they can run the automated tests even as the
> downstream is curated with a series of backports. This closely follows the
> model of a downstream maintainer who backports a feature/fix from upstream
> into a RPM and runs the upstream tests to make sure their patch didn't
> break functionality.
> >>
> >> # Proposal
> >> Upstream could require a basic functional test for each feature or a
> test for each bug fix. This test would be in the same commit as the feature
> or fix causing it to "follow the code" so when it's cherry picked
> downstream or into a package, the test goes with it. I believe we don't
> change the test during cherry pick time because although the code
> implementation may change what the feature or fix provides will not.
> >>
> >> Additionally, upstream should document how to run these tests easily
> and have the goal of making that easy.
> >>
> >> This policy change would ultimately be decided by each plugin and
> pulpcore separately. I don't want to make it a requirement to be part of
> the github.com/pulp/ organization in any way, so if a plugin can't, or
> won't, make this change that's ok, they can still host at github.com/pulp/
> .
> >>
> >> As a pulpcore and pulp_file maintainer, I'm proposing this policy for
> both of those repos specifically.
> >>
> >> In terms of enforcement, I'm pitching manual enforcement by review as a
> start. We can automate it later, but I hope that's a separate discussion to
> focus on the policy change and keep it simple.
> >>
> >> # Brian's take
> >> This is a non-trivial contribution requirement so we need to really
> think about it. My personal take is that at this point in the 3.y release
> line it's time to trade feature/fix velocity for stability. Also
> backporting is about to become a regular activity, so I think we have a
> practical motivation to adopt this, even though it will slow down the
> future features and bug fix velocity.
> >>
> >> What's your perspective?
> >>
> >> _______________________________________________
> >> Pulp-dev mailing list
> >> Pulp-dev at redhat.com
> >> https://www.redhat.com/mailman/listinfo/pulp-dev
> >
> >
> >
> > --
> > Pavel Picka
> > Red Hat
> > _______________________________________________
> > Pulp-dev mailing list
> > Pulp-dev at redhat.com
> > https://www.redhat.com/mailman/listinfo/pulp-dev
>
>
> _______________________________________________
> Pulp-dev mailing list
> Pulp-dev at redhat.com
> https://www.redhat.com/mailman/listinfo/pulp-dev
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/pulp-dev/attachments/20200818/97cbadbb/attachment.htm>