[Freeipa-devel] Thoughts on tests (unit, integration, self-test)

Thu Feb 19 03:48:34 UTC 2009

This is a brain dump on all things related to the freeIPA tests included
in its source tree...

One of my big goals with the Python code I've written for v2 is to make
freeIPA easier to test (especially easier to quickly test as you code,
while running everything in-tree).  This is a challenging problem
because running full blown freeIPA requires some fairly invasive
configuration changes... you don't want to make these changes to your
workstation unless you're actually part of an IPA realm, and if you're
part of an IPA realm, you don't want to run these tests against (and
possibly break) a production realm.

To get around this problem, everyone on the Red Hat team uses virtual
machines extensively to test freeIPA.  The problem is, we each do this
somewhat by hand, with slightly differing setups and procedures.  It
will help everyone be more productive if these "invasive" sorts of tests
are fully automated and repeatable.  I also personally feel it's very
important that the community be able to test their work the same way we
do at Red Hat.  So part of what I want to work on next is formalizing
the setup for these invasive tests into the freeIPA code.

Of course, many (if not most) of our tests will be of the non-invasive
variety.  Probably 90% of our current unit tests are non-invasive.
Whenever possible, we should write non-invasive tests because they're
fast to run and give the programmers immediate feedback.  Unit tests can
drastically improved productivity, but only if you run them very
frequently.  You want the time between when you break a test and when
you find out you broke a test to be as small as possible... ideally not
more than 5 or 10 minutes.  That way you have fresh in your head what
you changed, what likely caused the breakage.  If a breakage is left for
a few weeks, it can take you or another programmer hours to figure out
what is wrong.

In some cases we need to find ways to test stuff in a non-invasive way
that we currently can only test in an invasive way.  In particular, we
need a non-invasive way to test the code in the command plugins, even if
it doesn't test the full code path of a production server.  Also, the
non-invasive tests should be run automatically when the package is built
(whereas the invasive tests will have to be manually started).

So I'm proposing dividing the in-tree tests into three categories: unit
tests; integration tests; and self-tests.  The terminology doesn't
exactly fit, but I haven't thought of anything better yet (suggestions
welcome).

1. Unit tests (non-invasive)
----------------------------

I'm giving "unit test" some special meaning in this context: unit tests
are completely non-invasive... they can be run without Kerberos or LDAP
configured, can be run as a normal user, will not open network
connections nor contact external services, etc.  They will typically
test a fairly isolated layer of code, but might also test a broader
swath of the code path (as long as they're still non-invasive).

We're doing our Python unit tests using nose.  The core library has
extensive unit tests, all located in the tests/ directory.  

Additionally, Rob has written some tests for individual command plugins
(forwarded over XML-RPC, talking to live LDAP).  Currently these tests
get executed by nose automatically if the lite-xmlrpc.py script seems to
be running when you run the tests.  However, as they're what I'm calling
"invasive" tests, I believe these should be moved into the integration
tests below.  They can also be supplemented with the self-tests, my 3rd
category. 

So in summary, unit tests can be run without any unexpected side affects
and without having any external services configured.  Unit tests are
excellent for testing the core library, but not as useful for testing
individual plugins (especially backend plugins that talk to an external
service like LDAP).

2. Integration tests (invasive)
-------------------------------

I'm also giving "integration test" some special meaning in this context:
integration tests are invasive simply because of what they test... they
test the interaction with live LDAP, Kerberos, etc.  These are tests you
will only want to run in a virtual machine or on a dedicated test
machine.  They will often require you to run them as root or at least
kinit to get Kerboros credentials.  Even if a test is testing a fairly
isolated component, I'm still calling it an integration test if it's
invasive.

Like I said, I'd like Rob's xmlrpc tests to be moved into the
integration tests.  And all the integration tests should include the
full setup procedure so they're fully automatic and repeatable.

In my opinion, the first integration tests we write should be for
backend plugins like Backend.ldap.  In the case of the LDAP backend
plugin, the rest of the code is not supposed to use the python-ldap
binding directly, but instead just use the API provided by
api.Backend.ldap.  So Obviously this API needs to be well tested in an
isolated fashion.  The integration test would:

1. Configure Kerberos and DS, put DS into a known initial state.

2. Run tests against api.Backend.ldap like this:

    >>> from ipalib import api
    >>> api.bootstrap(in_server=True)
    >>> api.finalize()
    >>> ldap = api.Backend.ldap
    >>> # And now test the ldap plugin...

Rob's tests for individual command plugins can take a similar setup
approach (if not just reuse the same).  However, there's another cool
way we can test individual command plugins that I think will give us a
big productivity boost...

3. Self-tests (non-invasive, installed with plugins)
----------------------------------------------------

I don't recall if I ever made it clear why I was so insistent that for
things like LDAP, plugins should interact strictly with api.Backend.ldap
and never with the python-ldap bindings directly.  The reason is it
allows us to register dummy backend plugins and do very useful
non-invasive tests where we otherwise couldn't.  We can't test the full
code path this way, but we can test that the command plugins do their
part of the processing correctly and that they correctly call whatever
backend plugins do the heavy lifting for them.

This allows us to test via a transitive relationship: if A uses B
correctly and B uses C correctly, then A uses C correctly.  For example,
we know Command.user_add calls Backend.ldap correctly, and we know
Backend.ldap talks to FDS correctly, so we're pretty darn sure
Command.user_add talks to FDS correctly.

Obviously we will also want to test Command.user_add in the integration
tests, where we test the full code path against a live LDAP server.  But
the point of the self-tests is to move more testing into the
non-invasive realm so we can immediately get feedback while coding.
Make a small change to Command.user_add, run the self-tests, and if they
pass, you're 95% sure the live version will also work correctly.

The self-tests also make it easier for plugin authors (even 3rd-party)
to add tests because the self-tests are defined in the same module
defining the plugins.  As a consequence, the self-tests will ship in the
distribution packages (.rpm, .deb, .whatever), whereas the unit and
integration tests are only in the source tarball.  This allows end users
to run a lot of diagnostics easily.

The self-tests for commands will be totally declarative.  They're
defined in the same module as the corresponding command plugins,
although the exact mechanism is yet to be decided.  But they'll look
something like this (still using user_add as the example):

SelfTest(
    # Calling this command with these args and options:
    'user_add', tuple(),  dict(givenname=u'Jason', sn=u'DeRose'),

    # Will return this value:
    dict(uid=u'jderose', givenname=u'Jason', sn=u'DeRose'),

    # And result in these calls to Backend.ldap:
    [
# 1st call is to ldap.make_user_dn() with these args, kw:
        'ldap.make_user_dn', (u'jderose'), {},

# And will return this value:
        u'uid=jderose,cn=users,cn=accounts,dc=example,dc=com'
    ],
    [
        # 2nd call is to ldap.create() with these args, kw:
        'ldap.create', tuple(), dict(
           dn=u'uid=jderose,cn=users,cn=accounts,dc=example,dc=com',    
    uid=u'jderose',
            givenname=u'Jason',
            sn=u'DeRose',
        ),

        # And will return this value:
        dict(uid=u'jderose', givenname=u'Jason', sn=u'DeRose'),
    ],
)

So both the calls that the command plugin should make to various backend
plugin methods and the values those calls will return are provided in
the self-test.  This illustrative self-test above would (if completed)
test whether user_add processes its args and options correctly, whether
user_add makes the correct sequence of calls to Backend.ldap, whether it
does the right thing with the return values from Backend.ldap, and
whether it ultimately returns to correct result.  The exact API is yet
do be decided, but that's the idea of it anyway.

Obviously I got some ideas (and the "self-test" term) from my favorite
embeddable DVCS, Bazaar.  But as far as I know, Bazaar doesn't use its
self-tests to make otherwise invasive tests into non-invasive ones (all
its tests can easily be non-invasive, but it's also a different beast
altogether).

So there's my ramblings.  Thoughts?

Cheers,
Jason

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 835 bytes
Desc: This is a digitally signed message part
URL: <http://listman.redhat.com/archives/freeipa-devel/attachments/20090218/501629b4/attachment.sig>