[Freeipa-devel] Switching to pytest

[Petr Viktorin]

https://fedorahosted.org/freeipa/ticket/4610

Our test suite is currently not very maintainable. I want to dedicate 
some time to improve this.
The biggest part of this effort will be switching to a different test 
framework, [pytest]. Compared to Nose, it makes complicated tests easier 
to write -- at the expense of using more "magic".

Probably the most visible features in pytest are:

- Fixtures. Not so easily explained if you haven't used it before, but 
I'll try.
Basically, setup/teardown, test parametrization, and dependencies for 
injection can be packaged in a named function/object, which individual 
tests can then very simply use. In IPA, this would for example allow us 
to say "this tests needs a user in LDAP", and the user will be 
created/destroyed before/after the test is run (and, depending on 
confiiguration, it can be reused in the next test to save time).
Fixtures can also depend on other fixtures. For example the above user 
fixture can depend on a LDAP server being installed. This could be a 
pre-installed server, or a server the tests install automatically, or 
some kind of mock -- and this could be selectable by configuration/options.
Fixtures can also be used select/skip tests, which ticket #4610 callls for.

- Assertion rewriting: you can write a simple assert statement, and 
py.test will give detailed information about what failed in it. This 
reduces the need for boilerplate-y methods like "assertEqual". (though 
it's not a silver bullet)

I was certainly rather distrustful when I first saw all this -- it's 
quite questionable from SW engineering best practices point of view. But 
after using it, I can assure you that it's very well thought out. 
Testing does require a slightly different set of practices than building 
software.


Pytest is successfully used to test many projects; closest to home for 
us is probably 389-ds.

Most Nose tests can be run as-is with pytest. IPA's "xmlrpc" suite is, 
unfortunately, weird enough to be the exception, but since it's 
declarative it should be possible to port.


There are three stages to this effort:
- Converting the existing suite to run under pytest
- Updating the infrastructure (./make-test, plugins, CI config)
- Rewriting individual tests to make use of the new features

It's rather hard to plan this in more detail, since the first stage is 
"overcome unknown obstacles". I'll keep you updated.

Feedback welcome.

[pytest]: http://pytest.org/latest/

-- 
Petr³