[Pulp-dev] Pull Request builder job changes for plugins

[Dennis Kliban]

What if we ran our plugin unit tests against both the latest GA build and
nightly build of core?

If the tests pass with the GA version, the job is marked as successful. If
not, core packages are upgraded to the latest nightly and unit tests are
run again. If the unit tests fail again, the job is marked as failed. If
the unit tests pass with the latest nightly, the job is marked as
successful.

On Sun, Jun 4, 2017 at 12:03 PM, Brian Bouterse <bbouters at redhat.com> wrote:

> After thinking about this more, I realized that for the remainder of Pulp2
> at least, we need to have the plugin unittest runner test against the
> nightly version of core and not the latest GA. Using the GAs won't work
> because not only is the 'I need unreleased code from platform' a problem
> with the PR that needs it but also a problem for all subsequent PRs after
> its merged. That second part makes using GA core as the basis for plugin
> testing probably a non-starter. Assuming that, the next step for 2751 is to
> update the GA urls to be the "stable nightly" URLs.
>
> We also need to look into the nightlies to check on their reliability. In
> theory, each night an "unstable nightly" of core gets built nightly in
> Jenkins, tested with pulp-smash, and if all tests pass it gets "promoted"
> to a separate URL for "stable nightlies". Let me know if we should move
> this to another thread, but I've got these questions about nightlies.
>
> 1. Who investigates when the "unstable nightly" fails to build?
> 2. Who investigates when a "unstable nightly" fails to be promoted to a
> "stable nightly" due to pulp-smash failures?
> 3. Who is in charge of maintaining these Jenkins jobs over time and are
> they currently maintained?
> 4. Who is in charge of managing the directory structure on
> repos.fedorapeople.org?
> 5. Where are the docs on ^?
>
> With Pulp3 I think we can switch to using the latest GA as the basis for
> plugin testing which would be better in several ways.
>
> -Brian
>
> On Fri, Jun 2, 2017 at 5:06 PM, Brian Bouterse <bbouters at redhat.com>
> wrote:
>
>> That is a good point, and one we are giving some thought to through convo
>> on #pulp-dev and the issue [0]. The case of a plugin needing an unreleased
>> change from core would fail with this change. It's a tradeoff though
>> because if we go with nightlies as the version of core that is used,
>> whenever the nightlies break, the unittest PR runners also will, which has
>> been a reliability issue with the plugin unittest runner for a while.
>>
>> I wrote some on the issue about it, but I see the 'plugin needs
>> unreleased code from core' as a special case, not a normal case. It used to
>> be common, but it's getting less common, which is good, because
>> contributing to a plugin should not involve changes to the core as the
>> norm. It will happen from time to time, so we can handle the special case,
>> specially by running the unittests locally with the necessary unreleased
>> version of platform and posting the results as evidence that its safe to
>> merge.
>>
>> [0]: https://pulp.plan.io/issues/2751
>>
>> On Fri, Jun 2, 2017 at 4:43 PM, Michael Hrivnak <mhrivnak at redhat.com>
>> wrote:
>>
>>> What about cases where a plugin wants to use something that's new in the
>>> unreleased core? The master branch of a plugin will usually be released
>>> with the master branch of the core in the next 2.y release for example.
>>> That seems like a normal scenario; is it facilitated somehow with this
>>> testing change?
>>>
>>> On Fri, Jun 2, 2017 at 4:33 PM, Dennis Kliban <dkliban at redhat.com>
>>> wrote:
>>>
>>>> In an effort to resolve issue 2751[0], I updated the PR builder job for
>>>> plugins. Each PR for a plugin will now be tested against the latest stable
>>>> release of the core found here[1]. This will ensure that the plugin is
>>>> maintaining compatibility with the latest stable core and that we are only
>>>> testing one change at a time.
>>>>
>>>>
>>>> [0] https://pulp.plan.io/issues/2751
>>>> [1] https://repos.fedorapeople.org/pulp/pulp/stable/latest/
>>>>
>>>>
>>>> -Dennis
>>>>
>>>> _______________________________________________
>>>> Pulp-dev mailing list
>>>> Pulp-dev at redhat.com
>>>> https://www.redhat.com/mailman/listinfo/pulp-dev
>>>>
>>>>
>>>
>>>
>>> --
>>>
>>> Michael Hrivnak
>>>
>>> Principal Software Engineer, RHCE
>>>
>>> Red Hat
>>>
>>> _______________________________________________
>>> Pulp-dev mailing list
>>> Pulp-dev at redhat.com
>>> https://www.redhat.com/mailman/listinfo/pulp-dev
>>>
>>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/pulp-dev/attachments/20170605/b3ffdc7d/attachment.htm>