[Freeipa-devel] [PATCHES] 0230-0240 Integration testing framework

Fri Jun 14 15:44:35 UTC 2013

On 06/14/2013 04:41 PM, Jan Cholasta wrote:
> On 14.6.2013 16:19, Jan Cholasta wrote:
>> Hi,
>>
>> On 13.6.2013 14:16, Petr Viktorin wrote:
>>> On 06/10/2013 04:48 PM, Petr Viktorin wrote:
>>>> On 05/31/2013 01:46 PM, Petr Viktorin wrote:
>>>>> Apply on top of my patches 0227-0234.
>>>>>
>>>>> These patches add an initial integration testing framework.
>>>>>
>>>>> Patch 0230 adds a plugin for ordered test classes.
>>>>> Nose orders methods within a test suite alphabetically, but we
>>>>> generally
>>>>> want to run them in the order defined. This adds the @ordered
>>>>> decorator
>>>>> that causes Nose to do just that, provided the plugin is loaded and
>>>>> enabled, and that the methods are defined in the same file. The
>>>>> ipa-run-tests wrapper is changed to enable the plugin.
>>>>> In the future we may want to use this for unit tests as well. It might
>>>>> also make sense to separate it from the FreeIPA project altogether.
>>>>>
>>>>> Patch 0231 adds configuration for tests. This reads environment
>>>>> variables like:
>>>>> - MASTER (FQDN of initial server)
>>>>> - REPLICA (space-separated FQDNs of replicas)
>>>>> - CLIENT (space-separated FQDNs of clients),
>>>>> - IPATEST_DIR (directory the tests use on the remote machines)
>>>>> etc., and loads them into an easy-to use Python object.
>>>>> A tool called ipa-test-config is provided that generates a full set of
>>>>> environment variables for shell-based tests from these, either
>>>>> global or
>>>>> specific for a given host.
>>>>> If environment variables don't work for us, alternate configuration
>>>>> methods can be added in the future.
>>
>> I think you forgot to add "%dir
>> %{python_sitelib}/ipatests/test_integration" to the spec file.
>>
>> Is the "self = cls()" line at the beginning of Config.from_env()
>> intentional?
>>
>> +        self = cls()
>> +        env_normalize(env)
>> +
>> +        self = cls(test_dir=env.get('IPATEST_DIR') or '/root/ipatests',

No. I removed it, thanks for thee catch.

> Also typo in commit message: "Integration tests are be configured ..."

Thanks, fixed.

>>>>>
>>>>> Patch 0232 adds an integration test framework.
>>>>> This extends Host object available from the configuration with methods
>>>>> to run commands and copy files on the remote host, and adds a base
>>>>> class
>>>>> for integration tests which can currently install and uninstall IPA
>>>>> in a
>>>>> "star" topology. (In the future, the install/uninstall code should
>>>>> also
>>>>> be made available as a shell command.)
>>>>> A simple test for user replication between two masters is provided.
>>>>> Log files from the remote hosts can be marked for collection, but the
>>>>> actual collection is left to a Nose plugin.
>>>>> The base class uses the @ordered decorator mentioned above.
>>>>>
>>>>> Patch 0233 improves on how commands are run on remote hosts.
>>>>> In the previous patch, the process's stdin and stdout were combined
>>>>> as a
>>>>> quick hack to avoid the problem that if we first read stdout and then
>>>>> stderr, then stderr's buffer can fill up and we'd deadlock (and the
>>>>> other way around). With this patch the streams are read in parallel.
>>>>> In the future this can be extended to calling whole commands in
>>>>> parallel
>>>>> (e.g. uninstalling IPA on all the hosts at once).
>>>>>
>>>>> Patch 0234 adds log collection to the BeakerLib integration plugin.
>>>>> This tars up the marked logs, downloads then, and calls
>>>>> rlFileSubmit on
>>>>> them.
>
> Missing space in commit message: "... log files fromthe remote ..."

Thanks, fixed.

>>>>>
>>>>> ---
>>>>
>>>> Attaching additional patches:
>>>>
>>>> Patch 0237 configures logging in ipa-run-tests to forward messages from
>>>> the IPA logging machinery to a normal Python logger. This way the logs
>>>> are captured
>>>> The logs are also printed to stderr so that there's some activity on
>>>> the
>>>> terminal after you run the utility.
>>>>
>>>> Patch 0238 makes it possible to use RSA private SSH keys to log in to
>>>> the remote machines. The key is given in $IPA_ROOT_SSH_KEY, and used if
>>>> $IPA_ROOT_SSH_PASSWORD is empty.
>>>> I've added this to the design page.
>>
>> It seems that the code prefers password authentication over public key
>> authentication if both are configured. IMO it would be better if it did
>> the opposite.

Good point, fixed, design page updated.

>>>> Patch 0239 makes test setup change the hostname, /etc/hosts, and
>>>> /etc/resolv.conf to match the configured values. These should be
>>>> equivalent to the fixes in
>>>> https://github.com/freeipa/tests/blob/master/ipa-tests/beaker/ipa-server/shared/ipa-install.sh#L733
>>>>
>>>>
>>>>
>>>>
>>>> In test teardown, the changes are undone.
>>>
>>> I've rebased the patrchset and added small fixes for patches 0232 and
>>> 0239.
>>>
>>> New patch 0240 contains a few fixes/improvements to the Host class that
>>> were not trivial to rebase into previous patches.
>>>
>>> The WIP patch adds a sketch of some of the tests for CA-less
>>> (http://www.freeipa.org/page/V3/CA-less_install#Test_Plan). Please
>>> comment if you can see where things can be improved for test authors!
>>
>> Just a word of warning, there are still a few test cases I need to add
>> to the CA-less test plan.

Sure. I did this mainly to see how things look from the test author's 
point of view.

-- 
Petr³

-------------- next part --------------
A non-text attachment was scrubbed...
Name: freeipa-pviktori-0227.3-Make-an-ipa-tests-package.patch
Type: text/x-patch
Size: 71201 bytes
Desc: not available
URL: <http://listman.redhat.com/archives/freeipa-devel/attachments/20130614/8c2fc7ca/attachment.bin>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: freeipa-pviktori-0228.3-Add-ipa-run-tests-command.patch
Type: text/x-patch
Size: 2517 bytes
Desc: not available
URL: <http://listman.redhat.com/archives/freeipa-devel/attachments/20130614/8c2fc7ca/attachment-0001.bin>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: freeipa-pviktori-0229.3-Add-Nose-plugin-for-BeakerLib-integration.patch
Type: text/x-patch
Size: 11056 bytes
Desc: not available
URL: <http://listman.redhat.com/archives/freeipa-devel/attachments/20130614/8c2fc7ca/attachment-0002.bin>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: freeipa-pviktori-0230.3-Add-a-plugin-for-test-ordering.patch
Type: text/x-patch
Size: 4190 bytes
Desc: not available
URL: <http://listman.redhat.com/archives/freeipa-devel/attachments/20130614/8c2fc7ca/attachment-0003.bin>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: freeipa-pviktori-0231.3-Add-a-framework-for-integration-test-configuration.patch
Type: text/x-patch
Size: 21265 bytes
Desc: not available
URL: <http://listman.redhat.com/archives/freeipa-devel/attachments/20130614/8c2fc7ca/attachment-0004.bin>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: freeipa-pviktori-0232.3-Add-a-framework-for-integration-testing.patch
Type: text/x-patch
Size: 22712 bytes
Desc: not available
URL: <http://listman.redhat.com/archives/freeipa-devel/attachments/20130614/8c2fc7ca/attachment-0005.bin>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: freeipa-pviktori-0233.3-Introduce-a-class-for-remote-commands.patch
Type: text/x-patch
Size: 10810 bytes
Desc: not available
URL: <http://listman.redhat.com/archives/freeipa-devel/attachments/20130614/8c2fc7ca/attachment-0006.bin>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: freeipa-pviktori-0234.3-Collect-logs-from-tests.patch
Type: text/x-patch
Size: 7021 bytes
Desc: not available
URL: <http://listman.redhat.com/archives/freeipa-devel/attachments/20130614/8c2fc7ca/attachment-0007.bin>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: freeipa-pviktori-0235.3-Do-not-hardcode-path-to-ipa-getkeytab-in-tests.patch
Type: text/x-patch
Size: 3287 bytes
Desc: not available
URL: <http://listman.redhat.com/archives/freeipa-devel/attachments/20130614/8c2fc7ca/attachment-0008.bin>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: freeipa-pviktori-0237.3-Show-logs-in-failed-tests.patch
Type: text/x-patch
Size: 3124 bytes
Desc: not available
URL: <http://listman.redhat.com/archives/freeipa-devel/attachments/20130614/8c2fc7ca/attachment-0009.bin>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: freeipa-pviktori-0238.3-tests-Allow-public-keys-for-authentication-to-the-re.patch
Type: text/x-patch
Size: 4538 bytes
Desc: not available
URL: <http://listman.redhat.com/archives/freeipa-devel/attachments/20130614/8c2fc7ca/attachment-0010.bin>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: freeipa-pviktori-0239.3-tests-Configure-unconfigure-remote-hosts.patch
Type: text/x-patch
Size: 9826 bytes
Desc: not available
URL: <http://listman.redhat.com/archives/freeipa-devel/attachments/20130614/8c2fc7ca/attachment-0011.bin>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: freeipa-pviktori-0240.3-Host-class-improvements.patch
Type: text/x-patch
Size: 6270 bytes
Desc: not available
URL: <http://listman.redhat.com/archives/freeipa-devel/attachments/20130614/8c2fc7ca/attachment-0012.bin>