[Pulp-dev] Pulp 3 release quality

Dana Walker dawalker at redhat.com
Tue Apr 17 20:28:53 UTC 2018


How about running the full test suite nightly to catch things from all PRs
that day rather than waiting until the next release?  Is that being done?
We should want things to fail sooner rather than later if there's
regressions.


--Dana

Dana Walker

Associate Software Engineer

Red Hat

<https://www.redhat.com>
<https://red.ht/sig>

On Tue, Apr 17, 2018 at 4:22 PM, Daniel Alley <dalley at redhat.com> wrote:

> bmbouter is fixing the docs link in a PR: https://github.com/pulp/pulp/p
> ull/3446
>
> My understanding is that the smoke tests are for PRs, but actual releases
> would still be tested against the full test suite.
>
>
> On Tue, Apr 17, 2018 at 3:25 PM, Preethi Thomas <pthomas at redhat.com>
> wrote:
>
>>
>>
>> On Tue, Apr 17, 2018 at 8:32 AM, David Davis <daviddavis at redhat.com>
>> wrote:
>>
>>> A couple things. First, we’re only running pulpcore and pulp_file tests
>>> against pulpcore PRs. Also, we’re not running all pulp-smash tests for each
>>> PR—only a certain subset labeled as "smoke tests". That said, we’ll still
>>> want to keep an eye on pulp-smash tests over time to make sure it doesn’t
>>> become a bottleneck.
>>>
>>
>> This raises another concern.
>>
>> If the "smoke tests" are the only tests run before releasing to PyPi,
>> what would be plan to run the complete integration test suite?
>>
>> Also, looking at
>> https://docs.pulpproject.org/en/3.0/nightly/contributing/con
>> tinuous_integration.html
>>
>> "The unit tests for both pulpcore and pulpcore-plugin live in the tests
>> <https://github.com/pulp/pulp/blob/3.0-dev/tests/> folder"
>>
>> The link seems to rendering a 404
>>
>>
>>>
>>> David
>>>
>>> On Tue, Apr 17, 2018 at 8:24 AM, Preethi Thomas <pthomas at redhat.com>
>>> wrote:
>>>
>>>> "The plan for quality involves the continuous delivery of Pulp 3 where
>>>> both unit and integration are run with each PR and prior to each release to
>>>> PyPI"
>>>>
>>>> My concern about the above statement is that as the number of tests
>>>> increases, the time it will take to run the pulp-smash integration suite
>>>> will also increase. If each PR is required to run the integration suite
>>>> then pulp-smash will become the bottleneck.
>>>>
>>>> Just for comparison purpose, it takes about 2.5 hours to run pulp-smash
>>>> for Pulp 2.
>>>>
>>>>
>>>>
>>>> On Mon, Apr 16, 2018 at 4:30 PM, Dennis Kliban <dkliban at redhat.com>
>>>> wrote:
>>>>
>>>>> The documentation has been updated[0].
>>>>>
>>>>> [0] https://docs.pulpproject.org/en/3.0/nightly/contributing/con
>>>>> tinuous_integration.html
>>>>>
>>>>> On Fri, Apr 6, 2018 at 2:52 PM, Dennis Kliban <dkliban at redhat.com>
>>>>> wrote:
>>>>>
>>>>>> I have updated the redmine issue[0] to include a page in the
>>>>>> contributors guide on "Continuous Integration" as a deliverable. That page
>>>>>> will include that quote.
>>>>>>
>>>>>>
>>>>>> [0] https://pulp.plan.io/issues/3530
>>>>>>
>>>>>> On Fri, Apr 6, 2018 at 2:06 PM, Robin Chan <rchan at redhat.com> wrote:
>>>>>>
>>>>>>> Dennis,
>>>>>>>
>>>>>>> Thanks for putting this together. I don't see any responses on this
>>>>>>> thread and take that to mean there were no concerns about this proposal.
>>>>>>>
>>>>>>> Would this process/responsibility change need to go anywhere? (Side
>>>>>>> questions, was this technically a PUP?)
>>>>>>> "author of the PR would need to be responsible for making additional
>>>>>>> PRs to pulp_file and/or pulp-smash to fix the smash tests"
>>>>>>>
>>>>>>> I think this is minor enough to skip some process, however I would
>>>>>>> like to see the above quote go into the developers guide. A new contributor
>>>>>>> shouldn't have to glean through mail list archives to understand this new
>>>>>>> responsibility.
>>>>>>>
>>>>>>> Thanks,
>>>>>>> Robin
>>>>>>>
>>>>>>> On Tue, Mar 27, 2018 at 8:22 PM, Dennis Kliban <dkliban at redhat.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> One of the requirements for this plan to work is making sure that
>>>>>>>> pulp, pulp_file, and pulp-smash always continue working together. This
>>>>>>>> means that any time we have a PR that breaks pulp-smash tests, the author
>>>>>>>> of the PR would need to be responsible for making additional PRs to
>>>>>>>> pulp_file and/or pulp-smash to fix the smash tests. To enforce this
>>>>>>>> requirement using Travis, I have filed 2 tasks[0,1] in redmine that I would
>>>>>>>> like to get groomed and added to the sprint. I plan on working on these
>>>>>>>> tasks as I introduce a change that will result from resolving issue 3488[2].
>>>>>>>>
>>>>>>>> [0] https://pulp.plan.io/issues/3530
>>>>>>>> [1] https://pulp.plan.io/issues/3531
>>>>>>>> [2] https://pulp.plan.io/issues/3488
>>>>>>>>
>>>>>>>> On Fri, Mar 23, 2018 at 2:37 PM, Dennis Kliban <dkliban at redhat.com>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> I've started putting together a Continuous Delivery of Pulp 3
>>>>>>>>> page[0] on our wiki.
>>>>>>>>>
>>>>>>>>> This page outlines a plan for how we can ensure and prove the
>>>>>>>>> quality of Pulp 3 releases by relying on pulp-smash tests and unit tests.
>>>>>>>>>
>>>>>>>>> This plan enables anyone to improve the quality of Pulp 3 releases
>>>>>>>>> through contributions to pulp-smash and unit tests.
>>>>>>>>>
>>>>>>>>> Please take a look at the plan and provide feedback on this thread
>>>>>>>>> or feel free to make edits directly on the page.
>>>>>>>>>
>>>>>>>>> [0] https://pulp.plan.io/projects/pulp/wiki/Continuous_Delivery_
>>>>>>>>> of_Pulp_3
>>>>>>>>>
>>>>>>>>> -Dennis
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> _______________________________________________
>>>>>>>> Pulp-dev mailing list
>>>>>>>> Pulp-dev at redhat.com
>>>>>>>> https://www.redhat.com/mailman/listinfo/pulp-dev
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>> _______________________________________________
>>>>> Pulp-dev mailing list
>>>>> Pulp-dev at redhat.com
>>>>> https://www.redhat.com/mailman/listinfo/pulp-dev
>>>>>
>>>>>
>>>>
>>>> _______________________________________________
>>>> Pulp-dev mailing list
>>>> Pulp-dev at redhat.com
>>>> https://www.redhat.com/mailman/listinfo/pulp-dev
>>>>
>>>>
>>>
>>
>> _______________________________________________
>> Pulp-dev mailing list
>> Pulp-dev at redhat.com
>> https://www.redhat.com/mailman/listinfo/pulp-dev
>>
>>
>
> _______________________________________________
> Pulp-dev mailing list
> Pulp-dev at redhat.com
> https://www.redhat.com/mailman/listinfo/pulp-dev
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/pulp-dev/attachments/20180417/2baad5cd/attachment.htm>


More information about the Pulp-dev mailing list