<div dir="ltr"><div>How about running the full test suite nightly to catch things from all PRs that day rather than waiting until the next release? Is that being done? We should want things to fail sooner rather than later if there's regressions.<br><br><br></div>--Dana<br></div><div class="gmail_extra"><br clear="all"><div><div class="gmail_signature" data-smartmail="gmail_signature"><div dir="ltr"><div>
<p style="font-weight:bold;margin:0;padding:0;font-size:14px;text-transform:uppercase;margin-bottom:0"><span>Dana</span> <span>Walker</span></p>
<p style="font-weight:normal;font-size:10px;margin:0px 0px 4px;text-transform:uppercase"><span>Associate Software Engineer</span><span style="font-weight:normal;color:#aaa;margin:0"></span></p>
<p style="font-weight:normal;margin:0;font-size:10px;color:#999"><a style="color:#0088ce;font-size:10px;margin:0;text-decoration:none;font-family:'overpass',sans-serif" href="https://www.redhat.com" target="_blank">Red Hat <span><br><br></span></a></p>
<table border="0"><tbody><tr><td width="100px"><a href="https://red.ht/sig" target="_blank"> <img src="https://www.redhat.com/files/brand/email/sig-redhat.png" width="90" height="auto"></a> </td>
</tr></tbody></table>
</div></div></div></div>
<br><div class="gmail_quote">On Tue, Apr 17, 2018 at 4:22 PM, Daniel Alley <span dir="ltr"><<a href="mailto:dalley@redhat.com" target="_blank">dalley@redhat.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div>bmbouter is fixing the docs link in a PR: <a href="https://github.com/pulp/pulp/pull/3446" target="_blank">https://github.com/pulp/pulp/p<wbr>ull/3446</a><br><br></div>My understanding is that the smoke tests are for PRs, but actual releases would still be tested against the full test suite.<div><div class="h5"><br><div class="gmail_extra"><br><div class="gmail_quote">On Tue, Apr 17, 2018 at 3:25 PM, Preethi Thomas <span dir="ltr"><<a href="mailto:pthomas@redhat.com" target="_blank">pthomas@redhat.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><br><div class="gmail_extra"><br><div class="gmail_quote"><span>On Tue, Apr 17, 2018 at 8:32 AM, David Davis <span dir="ltr"><<a href="mailto:daviddavis@redhat.com" target="_blank">daviddavis@redhat.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">A couple things. First, we’re only running pulpcore and pulp_file tests against pulpcore PRs. Also, we’re not running all pulp-smash tests for each PR—only a certain subset labeled as "smoke tests". That said, we’ll still want to keep an eye on pulp-smash tests over time to make sure it doesn’t become a bottleneck.</div></blockquote><div><br></div></span><div>This raises another concern.</div><div><br></div><div>If the "smoke tests" are the only tests run before releasing to PyPi, what would be plan to run the complete integration test suite? </div><div><br></div><div>Also, looking at</div><div><a href="https://docs.pulpproject.org/en/3.0/nightly/contributing/continuous_integration.html" target="_blank">https://docs.pulpproject.org/e<wbr>n/3.0/nightly/contributing/con<wbr>tinuous_integration.html</a><br></div><div><br></div><div><span style="color:rgb(64,64,64);font-family:Lato,proxima-nova,"Helvetica Neue",Arial,sans-serif;font-size:16px;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(252,252,252);text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline">"The unit tests for both<span> </span></span><cite style="box-sizing:border-box;color:rgb(64,64,64);font-family:Lato,proxima-nova,"Helvetica Neue",Arial,sans-serif;font-size:16px;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(252,252,252);text-decoration-style:initial;text-decoration-color:initial">pulpcore</cite><span style="color:rgb(64,64,64);font-family:Lato,proxima-nova,"Helvetica Neue",Arial,sans-serif;font-size:16px;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(252,252,252);text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline"><span> </span>and<span> </span></span><cite style="box-sizing:border-box;color:rgb(64,64,64);font-family:Lato,proxima-nova,"Helvetica Neue",Arial,sans-serif;font-size:16px;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(252,252,252);text-decoration-style:initial;text-decoration-color:initial">pulpcore-plu<wbr>gin</cite><span style="color:rgb(64,64,64);font-family:Lato,proxima-nova,"Helvetica Neue",Arial,sans-serif;font-size:16px;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(252,252,252);text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline"><span> </span>live in the<span> </span></span><a class="m_-5797965817316461138m_-5365321129638063712m_-8158081997275588642gmail-reference m_-5797965817316461138m_-5365321129638063712m_-8158081997275588642external" href="https://github.com/pulp/pulp/blob/3.0-dev/tests/" style="box-sizing:border-box;color:rgb(155,89,182);text-decoration:none;font-family:Lato,proxima-nova,"Helvetica Neue",Arial,sans-serif;font-size:16px;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(252,252,252)" target="_blank">tests</a><span style="color:rgb(64,64,64);font-family:Lato,proxima-nova,"Helvetica Neue",Arial,sans-serif;font-size:16px;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(252,252,252);text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline"><span> </span>folder"</span></div><div><br></div><div><font face="Lato, proxima-nova, Helvetica Neue, Arial, sans-serif" color="#404040"><span style="font-size:16px;background-color:rgb(252,252,252)">The link seems to rendering a 404</span></font></div><div><div class="m_-5797965817316461138m_-5365321129638063712h5"><div><br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div class="gmail_extra"><span class="m_-5797965817316461138m_-5365321129638063712m_-8158081997275588642gmail-HOEnZb"><font color="#888888"><br clear="all"><div><div class="m_-5797965817316461138m_-5365321129638063712m_-8158081997275588642gmail-m_2334848424775229229gmail_signature"><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div><br></div><div>David<br></div></div></div></div></div></div></div></div></font></span><div><div class="m_-5797965817316461138m_-5365321129638063712m_-8158081997275588642gmail-h5">
<br><div class="gmail_quote">On Tue, Apr 17, 2018 at 8:24 AM, Preethi Thomas <span dir="ltr"><<a href="mailto:pthomas@redhat.com" target="_blank">pthomas@redhat.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><span style="color:rgb(61,69,76);font-family:ProximaNova-Regular,helvetica,arial,sans-serif;font-size:14px;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline">"The plan for quality involves the continuous delivery of Pulp 3 where both unit and integration are run with each PR and prior to each release to PyPI"</span><br><div><span style="color:rgb(61,69,76);font-family:ProximaNova-Regular,helvetica,arial,sans-serif;font-size:14px;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline"><br></span></div><div><font face="ProximaNova-Regular, helvetica, arial, sans-serif" color="#3d454c"><span style="font-size:14px">My concern about the above statement is that as the number of tests increases, the time it will take to run the pulp-smash integration suite will also increase. If each PR is required to run the integration suite then pulp-smash will become the bottleneck.</span></font></div><div><font face="ProximaNova-Regular, helvetica, arial, sans-serif" color="#3d454c"><span style="font-size:14px"><br></span></font></div><div><font face="ProximaNova-Regular, helvetica, arial, sans-serif" color="#3d454c"><span style="font-size:14px">Just for comparison purpose, it takes about 2.5 hours to run pulp-smash for Pulp 2.</span></font></div><div><font face="ProximaNova-Regular, helvetica, arial, sans-serif" color="#3d454c"><span style="font-size:14px"><br></span></font></div><div><font face="ProximaNova-Regular, helvetica, arial, sans-serif" color="#3d454c"><span style="font-size:14px"> </span></font></div></div><div class="m_-5797965817316461138m_-5365321129638063712m_-8158081997275588642gmail-m_2334848424775229229HOEnZb"><div class="m_-5797965817316461138m_-5365321129638063712m_-8158081997275588642gmail-m_2334848424775229229h5"><div class="gmail_extra"><br><div class="gmail_quote">On Mon, Apr 16, 2018 at 4:30 PM, Dennis Kliban <span dir="ltr"><<a href="mailto:dkliban@redhat.com" target="_blank">dkliban@redhat.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">The documentation has been updated[0].<br><br>[0] <a href="https://docs.pulpproject.org/en/3.0/nightly/contributing/continuous_integration.html" target="_blank">https://docs.pulpproject.org/e<wbr>n/3.0/nightly/contributing/con<wbr>tinuous_integration.html</a><br></div><div class="gmail_extra"><br><div class="gmail_quote">On Fri, Apr 6, 2018 at 2:52 PM, Dennis Kliban <span dir="ltr"><<a href="mailto:dkliban@redhat.com" target="_blank">dkliban@redhat.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">I have updated the redmine issue[0] to include a page in the contributors guide on "Continuous Integration" as a deliverable. That page will include that quote. <br><br><br>[0] <a href="https://pulp.plan.io/issues/3530" target="_blank">https://pulp.plan.io/issues/35<wbr>30</a><br></div><div class="m_-5797965817316461138m_-5365321129638063712m_-8158081997275588642gmail-m_2334848424775229229m_-6039743806705167138m_-2724850298467085680HOEnZb"><div class="m_-5797965817316461138m_-5365321129638063712m_-8158081997275588642gmail-m_2334848424775229229m_-6039743806705167138m_-2724850298467085680h5"><div class="gmail_extra"><br><div class="gmail_quote">On Fri, Apr 6, 2018 at 2:06 PM, Robin Chan <span dir="ltr"><<a href="mailto:rchan@redhat.com" target="_blank">rchan@redhat.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div><div><div><div><div>Dennis,<br></div><br></div>Thanks for putting this together. I don't see any responses on this thread and take that to mean there were no concerns about this proposal.<br><br>Would this process/responsibility change need to go anywhere? (Side questions, was this technically a PUP?)<span><br>"author of the PR would need to be responsible for making additional PRs to pulp_file and/or pulp-smash to fix the smash tests"<br><br></span></div>I think this is minor enough to skip some process, however I would like to see the above quote go into the developers guide. A new contributor shouldn't have to glean through mail list archives to understand this new responsibility.<br><br></div>Thanks,<br></div>Robin<br></div><div class="gmail_extra"><br><div class="gmail_quote"><div><div class="m_-5797965817316461138m_-5365321129638063712m_-8158081997275588642gmail-m_2334848424775229229m_-6039743806705167138m_-2724850298467085680m_-2286539430938277476h5">On Tue, Mar 27, 2018 at 8:22 PM, Dennis Kliban <span dir="ltr"><<a href="mailto:dkliban@redhat.com" target="_blank">dkliban@redhat.com</a>></span> wrote:<br></div></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div><div class="m_-5797965817316461138m_-5365321129638063712m_-8158081997275588642gmail-m_2334848424775229229m_-6039743806705167138m_-2724850298467085680m_-2286539430938277476h5"><div dir="ltr">One of the requirements for this plan to work is making sure that pulp, pulp_file, and pulp-smash always continue working together. This means that any time we have a PR that breaks pulp-smash tests, the author of the PR would need to be responsible for making additional PRs to pulp_file and/or pulp-smash to fix the smash tests. To enforce this requirement using Travis, I have filed 2 tasks[0,1] in redmine that I would like to get groomed and added to the sprint. I plan on working on these tasks as I introduce a change that will result from resolving issue 3488[2].<br><br>[0] <a href="https://pulp.plan.io/issues/3530" target="_blank">https://pulp.plan.io/issues/35<wbr>30</a><br>[1] <a href="https://pulp.plan.io/issues/3531" target="_blank">https://pulp.plan.io/issues/35<wbr>31</a><br>[2] <a href="https://pulp.plan.io/issues/3488" target="_blank">https://pulp.plan.io/issues/34<wbr>88</a><br></div><div class="m_-5797965817316461138m_-5365321129638063712m_-8158081997275588642gmail-m_2334848424775229229m_-6039743806705167138m_-2724850298467085680m_-2286539430938277476m_3845429498153912837HOEnZb"><div class="m_-5797965817316461138m_-5365321129638063712m_-8158081997275588642gmail-m_2334848424775229229m_-6039743806705167138m_-2724850298467085680m_-2286539430938277476m_3845429498153912837h5"><div class="gmail_extra"><br><div class="gmail_quote">On Fri, Mar 23, 2018 at 2:37 PM, Dennis Kliban <span dir="ltr"><<a href="mailto:dkliban@redhat.com" target="_blank">dkliban@redhat.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div>I've started putting together a Continuous Delivery of Pulp 3 page[0] on our wiki.<br><br>This page outlines a plan for how we can ensure and prove the quality of Pulp 3 releases by relying on pulp-smash tests and unit tests. <br><br></div><div>This plan enables anyone to improve the quality of Pulp 3 releases through contributions to pulp-smash and unit tests.<br><br></div><div>Please take a look at the plan and provide feedback on this thread or feel free to make edits directly on the page. <br></div><div><br>[0] <a href="https://pulp.plan.io/projects/pulp/wiki/Continuous_Delivery_of_Pulp_3" target="_blank">https://pulp.plan.io/projects/<wbr>pulp/wiki/Continuous_Delivery_<wbr>of_Pulp_3</a><span class="m_-5797965817316461138m_-5365321129638063712m_-8158081997275588642gmail-m_2334848424775229229m_-6039743806705167138m_-2724850298467085680m_-2286539430938277476m_3845429498153912837m_1826061612564265352HOEnZb"><font color="#888888"><br><br></font></span></div><span class="m_-5797965817316461138m_-5365321129638063712m_-8158081997275588642gmail-m_2334848424775229229m_-6039743806705167138m_-2724850298467085680m_-2286539430938277476m_3845429498153912837m_1826061612564265352HOEnZb"><font color="#888888"><div>-Dennis<br></div></font></span></div>
</blockquote></div><br></div>
</div></div><br></div></div>______________________________<wbr>_________________<br>
Pulp-dev mailing list<br>
<a href="mailto:Pulp-dev@redhat.com" target="_blank">Pulp-dev@redhat.com</a><br>
<a href="https://www.redhat.com/mailman/listinfo/pulp-dev" rel="noreferrer" target="_blank">https://www.redhat.com/mailman<wbr>/listinfo/pulp-dev</a><br>
<br></blockquote></div><br></div>
</blockquote></div><br></div>
</div></div></blockquote></div><br></div>
<br>______________________________<wbr>_________________<br>
Pulp-dev mailing list<br>
<a href="mailto:Pulp-dev@redhat.com" target="_blank">Pulp-dev@redhat.com</a><br>
<a href="https://www.redhat.com/mailman/listinfo/pulp-dev" rel="noreferrer" target="_blank">https://www.redhat.com/mailman<wbr>/listinfo/pulp-dev</a><br>
<br></blockquote></div><br></div>
</div></div><br>______________________________<wbr>_________________<br>
Pulp-dev mailing list<br>
<a href="mailto:Pulp-dev@redhat.com" target="_blank">Pulp-dev@redhat.com</a><br>
<a href="https://www.redhat.com/mailman/listinfo/pulp-dev" rel="noreferrer" target="_blank">https://www.redhat.com/mailman<wbr>/listinfo/pulp-dev</a><br>
<br></blockquote></div><br></div></div></div>
</blockquote></div></div></div><br></div></div>
<br>______________________________<wbr>_________________<br>
Pulp-dev mailing list<br>
<a href="mailto:Pulp-dev@redhat.com" target="_blank">Pulp-dev@redhat.com</a><br>
<a href="https://www.redhat.com/mailman/listinfo/pulp-dev" rel="noreferrer" target="_blank">https://www.redhat.com/mailman<wbr>/listinfo/pulp-dev</a><br>
<br></blockquote></div><br></div></div></div></div>
<br>______________________________<wbr>_________________<br>
Pulp-dev mailing list<br>
<a href="mailto:Pulp-dev@redhat.com">Pulp-dev@redhat.com</a><br>
<a href="https://www.redhat.com/mailman/listinfo/pulp-dev" rel="noreferrer" target="_blank">https://www.redhat.com/<wbr>mailman/listinfo/pulp-dev</a><br>
<br></blockquote></div><br></div>