[Freeipa-devel] Thoughts on tests (unit, integration, self-test)

[Jason Gerard DeRose]

On Thu, 2009-02-19 at 11:47 -0500, Rob Crittenden wrote:
> Jason Gerard DeRose wrote:
> > This is a brain dump on all things related to the freeIPA tests included
> > in its source tree...
> > 
> > One of my big goals with the Python code I've written for v2 is to make
> > freeIPA easier to test (especially easier to quickly test as you code,
> > while running everything in-tree).  This is a challenging problem
> > because running full blown freeIPA requires some fairly invasive
> > configuration changes... you don't want to make these changes to your
> > workstation unless you're actually part of an IPA realm, and if you're
> > part of an IPA realm, you don't want to run these tests against (and
> > possibly break) a production realm.
> 
> [ snip ]
> 
> I generally agree with this approach, particularly when it comes to 
> separating the integration tests from the unit tests. I like the idea of 
> selftests but I haven't yet wrapped my mind around it. I keep thinking 
> of Jurassic park where you are only testing what you are expecting and 
> therefore all tests pass (even though there are raptors about to bite you).

Well, I'm trying to wrap my mind how this made you think of Jurassic
Park.  ;)

The self-tests just fake the behavior of the backend plugins... the
command plugins still do all the param normalization, conversion, and
validation, and all the logic in Command.execute() is still getting
checked.  So it gives the code in the command plugin itself a full
workout.

> I think we should have 2 types of integration testing too: lite and 
> full. Lite testing would utilize the python-litexml.py script and a full 
> test would test against Apache running the XML-RPC module. In theory 
> they should work the same way, this will confirm the theory. It should 
> be as simple as changing a config option on the client so running one vs 
> the other should be straightforward. It might be nice to be able to run 
> this with a remote IPA server as well as a local one too.

Agreed.  It will very be good to know that lite-xmlrpc.py and Apache
truly are behaving the same.

> So I think you are on the right track here, we just need to flesh out 
> what the SelfTest might look like. Would it essentially be one test 
> per-method or would we be able to fake negative testing too (to test 
> throwing exceptions)?

A given command will have as many self-tests as it needs.  So we would
have a self-test for, say, user_show when the user doesn't exist and
make sure user_show raises a NotFound exception.

If you look at the DummyMethod and DummyClass classes in
tests/util.py... these implement a test-harness similar to what we need
for the self-tests (at least for emulating the backend plugins).

DummyClass.__process() checks that the calls are made in the right order
to the right methods, with the right args and options.  If the return
value is an Exception instance, the exception is raised.  Otherwise the
return value is returned.

> I gather you envision this as separate make-test scripts (or make targets)?

Maybe we have "unittest" and "selftest" targets, and the "test" target
will run both.

> I'm a little less jaded when it comes to setting up an environment. I 
> drop and re-create my IPA server on an almost daily basis it seems 
> testing one thing or another. Running in a VM is a good idea though 
> simply because IPA is so invasive in overwriting config files.

Well, maybe I should be bolder about testing on my host system, but so
far I just stick to the VM.  Mostly I want the setup procedure to be
codified so that the tests are easily repeatable by others.

> Great start.
> 
> rob
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 835 bytes
Desc: This is a digitally signed message part
URL: <http://listman.redhat.com/archives/freeipa-devel/attachments/20090219/ca1b995c/attachment.sig>