[Pulp-dev] Pulp 3 Unit Test Plan Proposal

[Robin Chan]

Thank you David & Brian on driving this plan forward.

I am happy we will release a Pulp 3.0 Core Beta with a known % coverage for
unit tests, a policy in place and mechanisms in place to ensure that number
only goes up.

In the future, I would like for us to flush out more the definition of
valuable unit tests vs. "not valuable," since I heard this a lot
generically but not spelled out in any way. I think this would be a good
way to provide transparency and ensure that when the % goes up in a
valuable way, but I don't believe that to be highest on our priority list.

Thank you!
Robin


On Fri, Apr 6, 2018 at 2:57 PM, Brian Bouterse <bbouters at redhat.com> wrote:

> There was a lot of +1 support on the PR. The basic Pulp3 unit test policy
> can be seen here: https://docs.pulpproject.org/
> en/3.0/nightly/contributing/unit_tests.html
>
> On Thu, Apr 5, 2018 at 1:06 PM, Brian Bouterse <bbouters at redhat.com>
> wrote:
>
>> This is a wrap-up update with the last next steps (for now) on the Pulp3
>> unit test discussion.
>>
>> 1. Here is a docs PR adopting simple but specific policy language
>> recommending unit tests for Pulp3: https://github.com/pulp/pulp/pull/3411
>> 2. We need to move our existing unit tests into their "forever home" so
>> @daviddavis and I groomed this issue and put it on the sprint:
>> https://pulp.plan.io/issues/3553
>>
>> For any areas of Pulp3 untested code that you want to add unit tests for
>> please file a Task in Redmine for that. A few of those have been filed AIUI.
>>
>> If there are any issues with these changes please bring them up. Any
>> aspect of them can be rethought. David and I have tried to facilitate what
>> we've heard from the group.
>>
>>
>>
>> On Mon, Mar 26, 2018 at 5:13 PM, Brian Bouterse <bbouters at redhat.com>
>> wrote:
>>
>>> I want to summarize what I've heard to facilitate some next steps and
>>> further discussion.
>>>
>>> There seems to be broad support and no -1 votes to the idea of a soft
>>> check that tracks unit test coverage, so we wanted to get that out of the
>>> way. Daviddavis enabled unit test coverage reporting for all Pulp3 PRs (
>>> https://github.com/pulp/pulp/pull/3397) and it will report on all PRs
>>> now. Currently, it shows 54.98% for pulpcore. That number is surprisingly
>>> high but not awesome. When looking at the report, it is mainly all import
>>> statements and function definitions since we have few/zero unit tests but
>>> also not much code.
>>>
>>> Based on feedback it sounds like leaving it at a soft check and highly
>>> encouraging unit tests with each PR is something we could all agree on. I
>>> want us to get to specific language that we can add into the Pulp3 docs as
>>> a new section called "Unit Tests" here: https://docs.pulpproject.org/e
>>> n/3.0/nightly/contributing/index.html Here is a starting point, please
>>> send suggestions:  "All new code is highly encouraged to have basic unit
>>> tests that demonstrate its functionality. A Pull Request that has failing
>>> unit tests cannot be merged."
>>>
>>>
>>> Also from the convo on-list and on-irc, here are some questions I would
>>> like help answering:
>>>
>>> * What areas in the existing codebase would really benefit from unit
>>> testing? I think we need a classpath list of modules and classes. I made an
>>> etherpad here: https://etherpad.net/p/Pulp_Unit_Test_Candidates
>>>
>>> * What are the existing unit tests and where do they live?
>>>
>>> * What docs need to be added to make contributing unit tests reasonable?
>>>
>>>
>>> Thanks for all the discussion!
>>> -Brian
>>>
>>>
>>> On Mon, Mar 26, 2018 at 4:01 PM, Jeremy Audet <jaudet at redhat.com> wrote:
>>>
>>>> > I'm also generally -1 against trying to pick a number (100%, 80%,
>>>> 60%) up-front.  We should unit test what makes sense to unit test, push
>>>> that number as high as reasonable, and otherwise focus on pulp-smash, which
>>>> I think has historically been more useful.
>>>>
>>>> QE is flattered by the focus on Pulp Smash. We're happy that the smoke
>>>> tests are being executed as a pull request check.
>>>>
>>>> However, it's important to remember that unit tests are far faster
>>>> than integration tests, typically by several orders of magnitude. In
>>>> addition, Pulp Smash's smoke tests are intentionally limited. They're
>>>> designed to execute quickly and to detect catastrophic regressions.
>>>> They're not intended to be comprehensive. In fact, some of the
>>>> already-written test cases may be stripped of their "smoke test"
>>>> status for the sake of speed. Psychology is important here: it's bad
>>>> if a developer locally fires off smoke tests, gets bored, and opens up
>>>> a new web browser tab.
>>>>
>>>> Please keep this limitation in mind when deciding on policies
>>>> regarding smoke tests.
>>>>
>>>> _______________________________________________
>>>> Pulp-dev mailing list
>>>> Pulp-dev at redhat.com
>>>> https://www.redhat.com/mailman/listinfo/pulp-dev
>>>>
>>>
>>>
>>
>
> _______________________________________________
> Pulp-dev mailing list
> Pulp-dev at redhat.com
> https://www.redhat.com/mailman/listinfo/pulp-dev
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/pulp-dev/attachments/20180406/a49adfc8/attachment.htm>