[Freeipa-devel] Switching to pytest

[Petr Viktorin]

On 10/06/2014 11:21 PM, John Dennis wrote:
> On 10/03/2014 09:24 AM, Petr Viktorin wrote:
>> https://fedorahosted.org/freeipa/ticket/4610
>>
>> Our test suite is currently not very maintainable. I want to dedicate
>> some time to improve this.
>> The biggest part of this effort will be switching to a different test
>> framework, [pytest]. Compared to Nose, it makes complicated tests easier
>> to write -- at the expense of using more "magic".
>
> Just looking for information, why pytest?

Nose is not enough, and knowing nothing clearly better (easier, more 
supportable, more popular*) than pytest, I went with what I know.

* unittest/nose is more popular than pytest, of course

> Here is why I'm asking. After leaving IPA I went to another large Python
> based project, OpenStack. OpenStack is really a number of loosely
> affiliated autonomous projects, each of which gets to decide how they
> are going to write and run their unit tests. As might be expected it's a
> bit of a mess. But there has been some convergence lately using tox,
> testr and testtools in combination.Python like Java has no shortage of
> test frameworks and that diversity is actually a pain point. What
> happens is very few folks actually understand the intricacies of test
> construction and execution (because there are so many options, it can be
> convoluted, there are many layers and each project does things
> differently). The net result is a lot of wasted time, I groan every time
> tests get ported to the test framework de jour, which seems to happen
> with alarming frequency.
>
> I'd really like to see some convergence in the Python community on test
> frameworks so as a developer you only have to be an expert in one
> framework because debugging test failures (including simply reproducing
> the failure) is currently a time sink and just when you think you
> understand what the heck is going on someone announces we're moving to
> the next big thing. The proliferation of test frameworks is further
> compounded by weak documentation. If something goes "boom" it can
> literally take days to piece together enough information to diagnose the
> issue (often because there are so many layers).

I sympathize.
Do you have a solution, though? Staying with Nose is not one at this 
point. I'm not too convinced by convergence in OpenStack; it's still 
just one project. If you have some other arguments for testr/testtools, 
let me know; I don't see the advantages over pytest.

> So how does pytest fit in with everything else? Is stable, long term and
> widely adopted? What about mocking? Is there a consensus on fixtures vs.
> mocks (that isn't religious). Can pytest handle both?

I'm not sure I understand correctly what you mean, but you can wrap 
mocks/monkeypatching in fixtures. So yes, pytest can handle both.

> I guess what I'm asking is there some convergence with Python test
> tools? Or with testing are we in the equivalent of the SCM wars of 5-7
> years ago until most everyone figured out git was superior and had the
> most traction?

Yeah, I guess you could say the testing war is raging on.
But even the SCM situation is not clear-cut; e.g. Mozilla or Python use 
Mercurial, and given the availability of bridge tools, they aren't 
likely to switch any time soon. And the Linux package manager war is 
also raging on, with no clear winner.

> Picking the test framework other than the one that bubbles to the top
> and becomes the de facto standard has consequences. I clearly do not
> have enough information to make a judgment here and any additional
> information would be very much appreciated.

Perhaps it's because I'm in Europe, but every major Python conference I 
go to has one or two pytest talks. Also, pytest was written by the same 
people as tox, which (I dare say) is the de-facto standard for test 
*orchestration* in Python.
So in the test framework war I'm betting on this one.

-- 
Petr³