[Freeipa-devel] Thoughts on tests (unit, integration, self-test)

Thu Feb 19 15:00:14 UTC 2009

Jason Gerard DeRose wrote:
> This is a brain dump on all things related to the freeIPA tests included
> in its source tree...
>
> One of my big goals with the Python code I've written for v2 is to make
> freeIPA easier to test (especially easier to quickly test as you code,
> while running everything in-tree).  This is a challenging problem
> because running full blown freeIPA requires some fairly invasive
> configuration changes... you don't want to make these changes to your
> workstation unless you're actually part of an IPA realm, and if you're
> part of an IPA realm, you don't want to run these tests against (and
> possibly break) a production realm.
>
> To get around this problem, everyone on the Red Hat team uses virtual
> machines extensively to test freeIPA.  The problem is, we each do this
> somewhat by hand, with slightly differing setups and procedures.  It
> will help everyone be more productive if these "invasive" sorts of tests
> are fully automated and repeatable.  I also personally feel it's very
> important that the community be able to test their work the same way we
> do at Red Hat.  So part of what I want to work on next is formalizing
> the setup for these invasive tests into the freeIPA code.
>
> Of course, many (if not most) of our tests will be of the non-invasive
> variety.  Probably 90% of our current unit tests are non-invasive.
> Whenever possible, we should write non-invasive tests because they're
> fast to run and give the programmers immediate feedback.  Unit tests can
> drastically improved productivity, but only if you run them very
> frequently.  You want the time between when you break a test and when
> you find out you broke a test to be as small as possible... ideally not
> more than 5 or 10 minutes.  That way you have fresh in your head what
> you changed, what likely caused the breakage.  If a breakage is left for
> a few weeks, it can take you or another programmer hours to figure out
> what is wrong.
>
> In some cases we need to find ways to test stuff in a non-invasive way
> that we currently can only test in an invasive way.  In particular, we
> need a non-invasive way to test the code in the command plugins, even if
> it doesn't test the full code path of a production server.  Also, the
> non-invasive tests should be run automatically when the package is built
> (whereas the invasive tests will have to be manually started).
>
> So I'm proposing dividing the in-tree tests into three categories: unit
> tests; integration tests; and self-tests.  The terminology doesn't
> exactly fit, but I haven't thought of anything better yet (suggestions
> welcome).
>
>
> 1. Unit tests (non-invasive)
> ----------------------------
>
> I'm giving "unit test" some special meaning in this context: unit tests
> are completely non-invasive... they can be run without Kerberos or LDAP
> configured, can be run as a normal user, will not open network
> connections nor contact external services, etc.  They will typically
> test a fairly isolated layer of code, but might also test a broader
> swath of the code path (as long as they're still non-invasive).
>
> We're doing our Python unit tests using nose.  The core library has
> extensive unit tests, all located in the tests/ directory.  
>
> Additionally, Rob has written some tests for individual command plugins
> (forwarded over XML-RPC, talking to live LDAP).  Currently these tests
> get executed by nose automatically if the lite-xmlrpc.py script seems to
> be running when you run the tests.  However, as they're what I'm calling
> "invasive" tests, I believe these should be moved into the integration
> tests below.  They can also be supplemented with the self-tests, my 3rd
> category. 
>
> So in summary, unit tests can be run without any unexpected side affects
> and without having any external services configured.  Unit tests are
> excellent for testing the core library, but not as useful for testing
> individual plugins (especially backend plugins that talk to an external
> service like LDAP).
>
>
> 2. Integration tests (invasive)
> -------------------------------
>
> I'm also giving "integration test" some special meaning in this context:
> integration tests are invasive simply because of what they test... they
> test the interaction with live LDAP, Kerberos, etc.  These are tests you
> will only want to run in a virtual machine or on a dedicated test
> machine.  They will often require you to run them as root or at least
> kinit to get Kerboros credentials.  Even if a test is testing a fairly
> isolated component, I'm still calling it an integration test if it's
> invasive.
>
> Like I said, I'd like Rob's xmlrpc tests to be moved into the
> integration tests.  And all the integration tests should include the
> full setup procedure so they're fully automatic and repeatable.
>
> In my opinion, the first integration tests we write should be for
> backend plugins like Backend.ldap.  In the case of the LDAP backend
> plugin, the rest of the code is not supposed to use the python-ldap
> binding directly, but instead just use the API provided by
> api.Backend.ldap.  So Obviously this API needs to be well tested in an
> isolated fashion.  The integration test would:
>
> 1. Configure Kerberos and DS, put DS into a known initial state.
>
> 2. Run tests against api.Backend.ldap like this:
>
>     >>> from ipalib import api
>     >>> api.bootstrap(in_server=True)
>     >>> api.finalize()
>     >>> ldap = api.Backend.ldap
>     >>> # And now test the ldap plugin...
>
> Rob's tests for individual command plugins can take a similar setup
> approach (if not just reuse the same).  However, there's another cool
> way we can test individual command plugins that I think will give us a
> big productivity boost...
>
>
> 3. Self-tests (non-invasive, installed with plugins)
> ----------------------------------------------------
>
> I don't recall if I ever made it clear why I was so insistent that for
> things like LDAP, plugins should interact strictly with api.Backend.ldap
> and never with the python-ldap bindings directly.  The reason is it
> allows us to register dummy backend plugins and do very useful
> non-invasive tests where we otherwise couldn't.  We can't test the full
> code path this way, but we can test that the command plugins do their
> part of the processing correctly and that they correctly call whatever
> backend plugins do the heavy lifting for them.
>
> This allows us to test via a transitive relationship: if A uses B
> correctly and B uses C correctly, then A uses C correctly.  For example,
> we know Command.user_add calls Backend.ldap correctly, and we know
> Backend.ldap talks to FDS correctly, so we're pretty darn sure
> Command.user_add talks to FDS correctly.
>
> Obviously we will also want to test Command.user_add in the integration
> tests, where we test the full code path against a live LDAP server.  But
> the point of the self-tests is to move more testing into the
> non-invasive realm so we can immediately get feedback while coding.
> Make a small change to Command.user_add, run the self-tests, and if they
> pass, you're 95% sure the live version will also work correctly.
>
> The self-tests also make it easier for plugin authors (even 3rd-party)
> to add tests because the self-tests are defined in the same module
> defining the plugins.  As a consequence, the self-tests will ship in the
> distribution packages (.rpm, .deb, .whatever), whereas the unit and
> integration tests are only in the source tarball.  This allows end users
> to run a lot of diagnostics easily.
>
> The self-tests for commands will be totally declarative.  They're
> defined in the same module as the corresponding command plugins,
> although the exact mechanism is yet to be decided.  But they'll look
> something like this (still using user_add as the example):
>
> SelfTest(
>     # Calling this command with these args and options:
>     'user_add', tuple(),  dict(givenname=u'Jason', sn=u'DeRose'),
>
>     # Will return this value:
>     dict(uid=u'jderose', givenname=u'Jason', sn=u'DeRose'),
>
>     # And result in these calls to Backend.ldap:
>     [
> # 1st call is to ldap.make_user_dn() with these args, kw:
>         'ldap.make_user_dn', (u'jderose'), {},
>
> # And will return this value:
>         u'uid=jderose,cn=users,cn=accounts,dc=example,dc=com'
>     ],
>     [
>         # 2nd call is to ldap.create() with these args, kw:
>         'ldap.create', tuple(), dict(
>            dn=u'uid=jderose,cn=users,cn=accounts,dc=example,dc=com',    
>     uid=u'jderose',
>             givenname=u'Jason',
>             sn=u'DeRose',
>         ),
>
>         # And will return this value:
>         dict(uid=u'jderose', givenname=u'Jason', sn=u'DeRose'),
>     ],
> )
>
> So both the calls that the command plugin should make to various backend
> plugin methods and the values those calls will return are provided in
> the self-test.  This illustrative self-test above would (if completed)
> test whether user_add processes its args and options correctly, whether
> user_add makes the correct sequence of calls to Backend.ldap, whether it
> does the right thing with the return values from Backend.ldap, and
> whether it ultimately returns to correct result.  The exact API is yet
> do be decided, but that's the idea of it anyway.
>
> Obviously I got some ideas (and the "self-test" term) from my favorite
> embeddable DVCS, Bazaar.  But as far as I know, Bazaar doesn't use its
> self-tests to make otherwise invasive tests into non-invasive ones (all
> its tests can easily be non-invasive, but it's also a different beast
> altogether).
>
>
> So there's my ramblings.  Thoughts?
>
> Cheers,
> Jason
>
>
>        
>     
>   
I will defer to experts but this seems like a good idea to me.
I do not know what are the conventions and best practices about the 
inclusion of the tests into the final deliverables.
If we do not violate any convention with such approach I do not see a 
reason why we should not follow it.
Rob, John and comments?

Thanks
Dmitri
>     
>
>
>
>
>
>
>   
> ------------------------------------------------------------------------
>
> _______________________________________________
> Freeipa-devel mailing list
> Freeipa-devel at redhat.com
> https://www.redhat.com/mailman/listinfo/freeipa-devel

-- 
Thank you,
Dmitri Pal

Engineering Manager IPA project,
Red Hat Inc.

-------------------------------
Looking to carve out IT costs?
www.redhat.com/carveoutcosts/