[Pulp-dev] Pulp 3 Unit Test Plan Proposal

[David Davis]

We haven't had a unit test policy in Pulp 3, and the community and core
committers would all like one. The general desire we've heard so far is to
change course and encourage developers to add unit tests to their changes
to Pulp 3.

The policy we're suggesting is to add a coveralls[0] check for Pull
Requests against the pulpcore 3.0-dev branch that shows the overall
coverage percentage, e.g. 12.89%. This check would pass if and only if
coverage increases or remains the same with the PR. We think this will
eventually get us on the path to 100% unit test coverage.

We propose the coveralls check be a soft check that allow for merging if it
fails. We would document the policy and try to adhere to it even though it
wouldn't formally block merging. At a future point when pulp3 (maybe the
GA?) we could make this a hard check.

Benefits:
- It's easy, simple, and automatic
- It's pretty objective and there's little room to argue with a number.
- Helps us raise our unit test coverage gradually over time

Downsides:
- Could discourage community contributions
- It can be a bit strict and unforgiving at times (especially if there's a
hard check)
- It only provides a guarantee around quantity of unit testing and not
quality


*Q: What about the existing functionality? Do we need to write unit tests
for it?*

We're not sure about this. We'd like community feedback. Is anyone
interested in writing backfill unit tests? If so, then maybe we should.

*Q: Will this also affect Pulp 2?*

We're not planning on it but if we like this enough, we can look at adding
it to Pulp 2.

*Q: Will this affect plugins?*

We want to start out with just pulpcore now and see how we like it. Then
we'll look at adding it to pulp_file. In the future, we can also look at
ways to make it easy for plugins to set this up.

*Q: Do I no longer need to write pulp-smash test plan issues in Github for
Pulp 3 features?*

No, you should still do that.

[0] https://coveralls.io/
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/pulp-dev/attachments/20180323/1f066d1a/attachment.htm>