[Freeipa-devel] [PATCHES] 0230-0240 Integration testing framework

Thu Jun 13 12:16:36 UTC 2013

On 06/10/2013 04:48 PM, Petr Viktorin wrote:
> On 05/31/2013 01:46 PM, Petr Viktorin wrote:
>> Apply on top of my patches 0227-0234.
>>
>> These patches add an initial integration testing framework.
>>
>> Patch 0230 adds a plugin for ordered test classes.
>> Nose orders methods within a test suite alphabetically, but we generally
>> want to run them in the order defined. This adds the @ordered decorator
>> that causes Nose to do just that, provided the plugin is loaded and
>> enabled, and that the methods are defined in the same file. The
>> ipa-run-tests wrapper is changed to enable the plugin.
>> In the future we may want to use this for unit tests as well. It might
>> also make sense to separate it from the FreeIPA project altogether.
>>
>> Patch 0231 adds configuration for tests. This reads environment
>> variables like:
>> - MASTER (FQDN of initial server)
>> - REPLICA (space-separated FQDNs of replicas)
>> - CLIENT (space-separated FQDNs of clients),
>> - IPATEST_DIR (directory the tests use on the remote machines)
>> etc., and loads them into an easy-to use Python object.
>> A tool called ipa-test-config is provided that generates a full set of
>> environment variables for shell-based tests from these, either global or
>> specific for a given host.
>> If environment variables don't work for us, alternate configuration
>> methods can be added in the future.
>>
>> Patch 0232 adds an integration test framework.
>> This extends Host object available from the configuration with methods
>> to run commands and copy files on the remote host, and adds a base class
>> for integration tests which can currently install and uninstall IPA in a
>> "star" topology. (In the future, the install/uninstall code should also
>> be made available as a shell command.)
>> A simple test for user replication between two masters is provided.
>> Log files from the remote hosts can be marked for collection, but the
>> actual collection is left to a Nose plugin.
>> The base class uses the @ordered decorator mentioned above.
>>
>> Patch 0233 improves on how commands are run on remote hosts.
>> In the previous patch, the process's stdin and stdout were combined as a
>> quick hack to avoid the problem that if we first read stdout and then
>> stderr, then stderr's buffer can fill up and we'd deadlock (and the
>> other way around). With this patch the streams are read in parallel.
>> In the future this can be extended to calling whole commands in parallel
>> (e.g. uninstalling IPA on all the hosts at once).
>>
>> Patch 0234 adds log collection to the BeakerLib integration plugin.
>> This tars up the marked logs, downloads then, and calls rlFileSubmit on
>> them.
>>
>> ---
>
> Attaching additional patches:
>
> Patch 0237 configures logging in ipa-run-tests to forward messages from
> the IPA logging machinery to a normal Python logger. This way the logs
> are captured
> The logs are also printed to stderr so that there's some activity on the
> terminal after you run the utility.
>
> Patch 0238 makes it possible to use RSA private SSH keys to log in to
> the remote machines. The key is given in $IPA_ROOT_SSH_KEY, and used if
> $IPA_ROOT_SSH_PASSWORD is empty.
> I've added this to the design page.
>
> Patch 0239 makes test setup change the hostname, /etc/hosts, and
> /etc/resolv.conf to match the configured values. These should be
> equivalent to the fixes in
> https://github.com/freeipa/tests/blob/master/ipa-tests/beaker/ipa-server/shared/ipa-install.sh#L733
>
> In test teardown, the changes are undone.

I've rebased the patrchset and added small fixes for patches 0232 and 0239.

New patch 0240 contains a few fixes/improvements to the Host class that 
were not trivial to rebase into previous patches.

The WIP patch adds a sketch of some of the tests for CA-less 
(http://www.freeipa.org/page/V3/CA-less_install#Test_Plan). Please 
comment if you can see where things can be improved for test authors!

For your convenience, there is a Git repo with the changes available at 
git://github.com/encukou/freeipa.git, branch integration-testing.

-- 
Petr³

-------------- next part --------------
A non-text attachment was scrubbed...
Name: freeipa-pviktori-0230.2-Add-a-plugin-for-test-ordering.patch
Type: text/x-patch
Size: 4190 bytes
Desc: not available
URL: <http://listman.redhat.com/archives/freeipa-devel/attachments/20130613/2307de91/attachment.bin>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: freeipa-pviktori-0231.2-Add-a-framework-for-integration-test-configuration.patch
Type: text/x-patch
Size: 21290 bytes
Desc: not available
URL: <http://listman.redhat.com/archives/freeipa-devel/attachments/20130613/2307de91/attachment-0001.bin>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: freeipa-pviktori-0232.2-Add-a-framework-for-integration-testing.patch
Type: text/x-patch
Size: 22712 bytes
Desc: not available
URL: <http://listman.redhat.com/archives/freeipa-devel/attachments/20130613/2307de91/attachment-0002.bin>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: freeipa-pviktori-0233.2-Introduce-a-class-for-remote-commands.patch
Type: text/x-patch
Size: 10810 bytes
Desc: not available
URL: <http://listman.redhat.com/archives/freeipa-devel/attachments/20130613/2307de91/attachment-0003.bin>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: freeipa-pviktori-0234.2-Collect-logs-from-tests.patch
Type: text/x-patch
Size: 7020 bytes
Desc: not available
URL: <http://listman.redhat.com/archives/freeipa-devel/attachments/20130613/2307de91/attachment-0004.bin>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: freeipa-pviktori-0237.2-Show-logs-in-failed-tests.patch
Type: text/x-patch
Size: 3124 bytes
Desc: not available
URL: <http://listman.redhat.com/archives/freeipa-devel/attachments/20130613/2307de91/attachment-0005.bin>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: freeipa-pviktori-0238.2-tests-Allow-public-keys-for-authentication-to-the-re.patch
Type: text/x-patch
Size: 4173 bytes
Desc: not available
URL: <http://listman.redhat.com/archives/freeipa-devel/attachments/20130613/2307de91/attachment-0006.bin>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: freeipa-pviktori-0239.2-tests-Configure-unconfigure-remote-hosts.patch
Type: text/x-patch
Size: 9826 bytes
Desc: not available
URL: <http://listman.redhat.com/archives/freeipa-devel/attachments/20130613/2307de91/attachment-0007.bin>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: freeipa-pviktori-0240.2-Host-class-improvements.patch
Type: text/x-patch
Size: 6259 bytes
Desc: not available
URL: <http://listman.redhat.com/archives/freeipa-devel/attachments/20130613/2307de91/attachment-0008.bin>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: WIP-Add-initial-CA-less-installation-tests.patch
Type: text/x-patch
Size: 21357 bytes
Desc: not available
URL: <http://listman.redhat.com/archives/freeipa-devel/attachments/20130613/2307de91/attachment-0009.bin>