[Pulp-dev] Pulp 3 release quality

Wed Apr 18 20:50:30 UTC 2018

Thanks for the suggestion Dana! David and I have added daily cron jobs to
Travis that run the same tests that are run for each PR. The latest status
of the build for pulp repository is displayed at the top of the README[0].
The unit test coverage report status is also displayed in the same
location. Similar badges will be added to the pulp_file repository.

We currently run all of Pulp 3 pulp-smash tests[1,2] for the pulp and
pulp_file PRs.

[0] https://github.com/pulp/pulp/blob/3.0-dev/README.md
[1] https://github.com/pulp/pulp_file/blob/master/.travis/script.sh#L20
[2] https://github.com/pulp/pulp/blob/3.0-dev/.travis/script.sh#L38

On Tue, Apr 17, 2018 at 4:28 PM, Dana Walker <dawalker at redhat.com> wrote:

> How about running the full test suite nightly to catch things from all PRs
> that day rather than waiting until the next release?  Is that being done?
> We should want things to fail sooner rather than later if there's
> regressions.
>
>
> --Dana
>
> Dana Walker
>
> Associate Software Engineer
>
> Red Hat
>
> <https://www.redhat.com>
> <https://red.ht/sig>
>
> On Tue, Apr 17, 2018 at 4:22 PM, Daniel Alley <dalley at redhat.com> wrote:
>
>> bmbouter is fixing the docs link in a PR: https://github.com/pulp/pulp/p
>> ull/3446
>>
>> My understanding is that the smoke tests are for PRs, but actual releases
>> would still be tested against the full test suite.
>>
>>
>> On Tue, Apr 17, 2018 at 3:25 PM, Preethi Thomas <pthomas at redhat.com>
>> wrote:
>>
>>>
>>>
>>> On Tue, Apr 17, 2018 at 8:32 AM, David Davis <daviddavis at redhat.com>
>>> wrote:
>>>
>>>> A couple things. First, we’re only running pulpcore and pulp_file tests
>>>> against pulpcore PRs. Also, we’re not running all pulp-smash tests for each
>>>> PR—only a certain subset labeled as "smoke tests". That said, we’ll still
>>>> want to keep an eye on pulp-smash tests over time to make sure it doesn’t
>>>> become a bottleneck.
>>>>
>>>
>>> This raises another concern.
>>>
>>> If the "smoke tests" are the only tests run before releasing to PyPi,
>>> what would be plan to run the complete integration test suite?
>>>
>>> Also, looking at
>>> https://docs.pulpproject.org/en/3.0/nightly/contributing/con
>>> tinuous_integration.html
>>>
>>> "The unit tests for both pulpcore and pulpcore-plugin live in the tests
>>> <https://github.com/pulp/pulp/blob/3.0-dev/tests/> folder"
>>>
>>> The link seems to rendering a 404
>>>
>>>
>>>>
>>>> David
>>>>
>>>> On Tue, Apr 17, 2018 at 8:24 AM, Preethi Thomas <pthomas at redhat.com>
>>>> wrote:
>>>>
>>>>> "The plan for quality involves the continuous delivery of Pulp 3 where
>>>>> both unit and integration are run with each PR and prior to each release to
>>>>> PyPI"
>>>>>
>>>>> My concern about the above statement is that as the number of tests
>>>>> increases, the time it will take to run the pulp-smash integration suite
>>>>> will also increase. If each PR is required to run the integration suite
>>>>> then pulp-smash will become the bottleneck.
>>>>>
>>>>> Just for comparison purpose, it takes about 2.5 hours to run
>>>>> pulp-smash for Pulp 2.
>>>>>
>>>>>
>>>>>
>>>>> On Mon, Apr 16, 2018 at 4:30 PM, Dennis Kliban <dkliban at redhat.com>
>>>>> wrote:
>>>>>
>>>>>> The documentation has been updated[0].
>>>>>>
>>>>>> [0] https://docs.pulpproject.org/en/3.0/nightly/contributing/con
>>>>>> tinuous_integration.html
>>>>>>
>>>>>> On Fri, Apr 6, 2018 at 2:52 PM, Dennis Kliban <dkliban at redhat.com>
>>>>>> wrote:
>>>>>>
>>>>>>> I have updated the redmine issue[0] to include a page in the
>>>>>>> contributors guide on "Continuous Integration" as a deliverable. That page
>>>>>>> will include that quote.
>>>>>>>
>>>>>>>
>>>>>>> [0] https://pulp.plan.io/issues/3530
>>>>>>>
>>>>>>> On Fri, Apr 6, 2018 at 2:06 PM, Robin Chan <rchan at redhat.com> wrote:
>>>>>>>
>>>>>>>> Dennis,
>>>>>>>>
>>>>>>>> Thanks for putting this together. I don't see any responses on this
>>>>>>>> thread and take that to mean there were no concerns about this proposal.
>>>>>>>>
>>>>>>>> Would this process/responsibility change need to go anywhere? (Side
>>>>>>>> questions, was this technically a PUP?)
>>>>>>>> "author of the PR would need to be responsible for making
>>>>>>>> additional PRs to pulp_file and/or pulp-smash to fix the smash tests"
>>>>>>>>
>>>>>>>> I think this is minor enough to skip some process, however I would
>>>>>>>> like to see the above quote go into the developers guide. A new contributor
>>>>>>>> shouldn't have to glean through mail list archives to understand this new
>>>>>>>> responsibility.
>>>>>>>>
>>>>>>>> Thanks,
>>>>>>>> Robin
>>>>>>>>
>>>>>>>> On Tue, Mar 27, 2018 at 8:22 PM, Dennis Kliban <dkliban at redhat.com>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> One of the requirements for this plan to work is making sure that
>>>>>>>>> pulp, pulp_file, and pulp-smash always continue working together. This
>>>>>>>>> means that any time we have a PR that breaks pulp-smash tests, the author
>>>>>>>>> of the PR would need to be responsible for making additional PRs to
>>>>>>>>> pulp_file and/or pulp-smash to fix the smash tests. To enforce this
>>>>>>>>> requirement using Travis, I have filed 2 tasks[0,1] in redmine that I would
>>>>>>>>> like to get groomed and added to the sprint. I plan on working on these
>>>>>>>>> tasks as I introduce a change that will result from resolving issue 3488[2].
>>>>>>>>>
>>>>>>>>> [0] https://pulp.plan.io/issues/3530
>>>>>>>>> [1] https://pulp.plan.io/issues/3531
>>>>>>>>> [2] https://pulp.plan.io/issues/3488
>>>>>>>>>
>>>>>>>>> On Fri, Mar 23, 2018 at 2:37 PM, Dennis Kliban <dkliban at redhat.com
>>>>>>>>> > wrote:
>>>>>>>>>
>>>>>>>>>> I've started putting together a Continuous Delivery of Pulp 3
>>>>>>>>>> page[0] on our wiki.
>>>>>>>>>>
>>>>>>>>>> This page outlines a plan for how we can ensure and prove the
>>>>>>>>>> quality of Pulp 3 releases by relying on pulp-smash tests and unit tests.
>>>>>>>>>>
>>>>>>>>>> This plan enables anyone to improve the quality of Pulp 3
>>>>>>>>>> releases through contributions to pulp-smash and unit tests.
>>>>>>>>>>
>>>>>>>>>> Please take a look at the plan and provide feedback on this
>>>>>>>>>> thread or feel free to make edits directly on the page.
>>>>>>>>>>
>>>>>>>>>> [0] https://pulp.plan.io/projects/pulp/wiki/Continuous_Delivery_
>>>>>>>>>> of_Pulp_3
>>>>>>>>>>
>>>>>>>>>> -Dennis
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> _______________________________________________
>>>>>>>>> Pulp-dev mailing list
>>>>>>>>> Pulp-dev at redhat.com
>>>>>>>>> https://www.redhat.com/mailman/listinfo/pulp-dev
>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>> _______________________________________________
>>>>>> Pulp-dev mailing list
>>>>>> Pulp-dev at redhat.com
>>>>>> https://www.redhat.com/mailman/listinfo/pulp-dev
>>>>>>
>>>>>>
>>>>>
>>>>> _______________________________________________
>>>>> Pulp-dev mailing list
>>>>> Pulp-dev at redhat.com
>>>>> https://www.redhat.com/mailman/listinfo/pulp-dev
>>>>>
>>>>>
>>>>
>>>
>>> _______________________________________________
>>> Pulp-dev mailing list
>>> Pulp-dev at redhat.com
>>> https://www.redhat.com/mailman/listinfo/pulp-dev
>>>
>>>
>>
>> _______________________________________________
>> Pulp-dev mailing list
>> Pulp-dev at redhat.com
>> https://www.redhat.com/mailman/listinfo/pulp-dev
>>
>>
>
> _______________________________________________
> Pulp-dev mailing list
> Pulp-dev at redhat.com
> https://www.redhat.com/mailman/listinfo/pulp-dev
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/pulp-dev/attachments/20180418/f5c6c2f5/attachment.htm>