[Freeipa-devel] Thoughts on tests (unit, integration, self-test)

[Rob Crittenden]

Rich Megginson wrote:
> Rob Crittenden wrote:
>> Jason Gerard DeRose wrote:
>>> This is a brain dump on all things related to the freeIPA tests included
>>> in its source tree...
>>>
>>> One of my big goals with the Python code I've written for v2 is to make
>>> freeIPA easier to test (especially easier to quickly test as you code,
>>> while running everything in-tree).  This is a challenging problem
>>> because running full blown freeIPA requires some fairly invasive
>>> configuration changes... you don't want to make these changes to your
>>> workstation unless you're actually part of an IPA realm, and if you're
>>> part of an IPA realm, you don't want to run these tests against (and
>>> possibly break) a production realm.
>>
>> [ snip ]
>>
>> I generally agree with this approach, particularly when it comes to 
>> separating the integration tests from the unit tests. I like the idea 
>> of selftests but I haven't yet wrapped my mind around it. I keep 
>> thinking of Jurassic park where you are only testing what you are 
>> expecting and therefore all tests pass (even though there are raptors 
>> about to bite you).
>>
>> I think we should have 2 types of integration testing too: lite and 
>> full. Lite testing would utilize the python-litexml.py script and a 
>> full test would test against Apache running the XML-RPC module. In 
>> theory they should work the same way, this will confirm the theory. It 
>> should be as simple as changing a config option on the client so 
>> running one vs the other should be straightforward. It might be nice 
>> to be able to run this with a remote IPA server as well as a local one 
>> too.
>>
>> So I think you are on the right track here, we just need to flesh out 
>> what the SelfTest might look like. Would it essentially be one test 
>> per-method or would we be able to fake negative testing too (to test 
>> throwing exceptions)?
>>
>> I gather you envision this as separate make-test scripts (or make 
>> targets)?
>>
>> I'm a little less jaded when it comes to setting up an environment. I 
>> drop and re-create my IPA server on an almost daily basis it seems 
>> testing one thing or another. Running in a VM is a good idea though 
>> simply because IPA is so invasive in overwriting config files.
> Another sort of half-way option - using mock to run tests.  mock is 
> great if you need a full-blown OS file system with your packages 
> installed and you also need root access.  mock can run any rhel or 
> fedora operating system (even 32-bit and 64-bit on a 64-bit machine).  
> mock is much more lightweight than running a VM.  The only problem is 
> network connections e.g. you cannot have two directory servers running 
> at the same time both listening to port 389 - there are probably other 
> gotchas as well with hostnames (but some sort of /etc/hosts hack in the 
> chroot might be possible).
> 
> I have run the directory server acceptance test suite inside mock, 
> including such tests as 4 way MMR.  It's pretty nifty to be able to use 
> mock to build and test packages in F-8, F-9, F-10, and even rhel 
> platforms, from a single machine with no VM.
>

I actually thought about that but figured it would be too much of a pain 
to set up. Can you provide any details/tips?

thanks

rob