[Freeipa-devel] Switching to pytest

[John Dennis]

On 10/03/2014 09:24 AM, Petr Viktorin wrote:
> https://fedorahosted.org/freeipa/ticket/4610
> 
> Our test suite is currently not very maintainable. I want to dedicate 
> some time to improve this.
> The biggest part of this effort will be switching to a different test 
> framework, [pytest]. Compared to Nose, it makes complicated tests easier 
> to write -- at the expense of using more "magic".

Just looking for information, why pytest?

Here is why I'm asking. After leaving IPA I went to another large Python
based project, OpenStack. OpenStack is really a number of loosely
affiliated autonomous projects, each of which gets to decide how they
are going to write and run their unit tests. As might be expected it's a
bit of a mess. But there has been some convergence lately using tox,
testr and testtools in combination. Python like Java has no shortage of
test frameworks and that diversity is actually a pain point. What
happens is very few folks actually understand the intricacies of test
construction and execution (because there are so many options, it can be
convoluted, there are many layers and each project does things
differently). The net result is a lot of wasted time, I groan every time
tests get ported to the test framework de jour, which seems to happen
with alarming frequency.

I'd really like to see some convergence in the Python community on test
frameworks so as a developer you only have to be an expert in one
framework because debugging test failures (including simply reproducing
the failure) is currently a time sink and just when you think you
understand what the heck is going on someone announces we're moving to
the next big thing. The proliferation of test frameworks is further
compounded by weak documentation. If something goes "boom" it can
literally take days to piece together enough information to diagnose the
issue (often because there are so many layers).

So how does pytest fit in with everything else? Is stable, long term and
widely adopted? What about mocking? Is there a consensus on fixtures vs.
mocks (that isn't religious). Can pytest handle both?

I guess what I'm asking is there some convergence with Python test
tools? Or with testing are we in the equivalent of the SCM wars of 5-7
years ago until most everyone figured out git was superior and had the
most traction?

Picking the test framework other than the one that bubbles to the top
and becomes the de facto standard has consequences. I clearly do not
have enough information to make a judgment here and any additional
information would be very much appreciated.



-- 
John