[Freeipa-devel] [PATCHES] 0230-0240 Integration testing framework

Tue Jun 25 12:08:28 UTC 2013

On 06/14/2013 05:44 PM, Petr Viktorin wrote:
> On 06/14/2013 04:41 PM, Jan Cholasta wrote:
>> On 14.6.2013 16:19, Jan Cholasta wrote:
>>> Hi,
>>>
>>> On 13.6.2013 14:16, Petr Viktorin wrote:
>>>> On 06/10/2013 04:48 PM, Petr Viktorin wrote:
>>>>> On 05/31/2013 01:46 PM, Petr Viktorin wrote:
>>>>>> Apply on top of my patches 0227-0234.
>>>>>>
>>>>>> These patches add an initial integration testing framework.
>>>>>>
>>>>>> Patch 0230 adds a plugin for ordered test classes.
>>>>>> Nose orders methods within a test suite alphabetically, but we
>>>>>> generally
>>>>>> want to run them in the order defined. This adds the @ordered
>>>>>> decorator
>>>>>> that causes Nose to do just that, provided the plugin is loaded and
>>>>>> enabled, and that the methods are defined in the same file. The
>>>>>> ipa-run-tests wrapper is changed to enable the plugin.
>>>>>> In the future we may want to use this for unit tests as well. It
>>>>>> might
>>>>>> also make sense to separate it from the FreeIPA project altogether.
>>>>>>
>>>>>> Patch 0231 adds configuration for tests. This reads environment
>>>>>> variables like:
>>>>>> - MASTER (FQDN of initial server)
>>>>>> - REPLICA (space-separated FQDNs of replicas)
>>>>>> - CLIENT (space-separated FQDNs of clients),
>>>>>> - IPATEST_DIR (directory the tests use on the remote machines)
>>>>>> etc., and loads them into an easy-to use Python object.
>>>>>> A tool called ipa-test-config is provided that generates a full
>>>>>> set of
>>>>>> environment variables for shell-based tests from these, either
>>>>>> global or
>>>>>> specific for a given host.
>>>>>> If environment variables don't work for us, alternate configuration
>>>>>> methods can be added in the future.
>>>
>>> I think you forgot to add "%dir
>>> %{python_sitelib}/ipatests/test_integration" to the spec file.
>>>
>>> Is the "self = cls()" line at the beginning of Config.from_env()
>>> intentional?
>>>
>>> +        self = cls()
>>> +        env_normalize(env)
>>> +
>>> +        self = cls(test_dir=env.get('IPATEST_DIR') or '/root/ipatests',
>
> No. I removed it, thanks for thee catch.
>
>> Also typo in commit message: "Integration tests are be configured ..."
>
> Thanks, fixed.
>
>>>>>>
>>>>>> Patch 0232 adds an integration test framework.
>>>>>> This extends Host object available from the configuration with
>>>>>> methods
>>>>>> to run commands and copy files on the remote host, and adds a base
>>>>>> class
>>>>>> for integration tests which can currently install and uninstall IPA
>>>>>> in a
>>>>>> "star" topology. (In the future, the install/uninstall code should
>>>>>> also
>>>>>> be made available as a shell command.)
>>>>>> A simple test for user replication between two masters is provided.
>>>>>> Log files from the remote hosts can be marked for collection, but the
>>>>>> actual collection is left to a Nose plugin.
>>>>>> The base class uses the @ordered decorator mentioned above.
>>>>>>
>>>>>> Patch 0233 improves on how commands are run on remote hosts.
>>>>>> In the previous patch, the process's stdin and stdout were combined
>>>>>> as a
>>>>>> quick hack to avoid the problem that if we first read stdout and then
>>>>>> stderr, then stderr's buffer can fill up and we'd deadlock (and the
>>>>>> other way around). With this patch the streams are read in parallel.
>>>>>> In the future this can be extended to calling whole commands in
>>>>>> parallel
>>>>>> (e.g. uninstalling IPA on all the hosts at once).
>>>>>>
>>>>>> Patch 0234 adds log collection to the BeakerLib integration plugin.
>>>>>> This tars up the marked logs, downloads then, and calls
>>>>>> rlFileSubmit on
>>>>>> them.
>>
>> Missing space in commit message: "... log files fromthe remote ..."
>
> Thanks, fixed.
>
>>>>>>
>>>>>> ---
>>>>>
>>>>> Attaching additional patches:
>>>>>
>>>>> Patch 0237 configures logging in ipa-run-tests to forward messages
>>>>> from
>>>>> the IPA logging machinery to a normal Python logger. This way the logs
>>>>> are captured
>>>>> The logs are also printed to stderr so that there's some activity on
>>>>> the
>>>>> terminal after you run the utility.
>>>>>
>>>>> Patch 0238 makes it possible to use RSA private SSH keys to log in to
>>>>> the remote machines. The key is given in $IPA_ROOT_SSH_KEY, and
>>>>> used if
>>>>> $IPA_ROOT_SSH_PASSWORD is empty.
>>>>> I've added this to the design page.
>>>
>>> It seems that the code prefers password authentication over public key
>>> authentication if both are configured. IMO it would be better if it did
>>> the opposite.
>
> Good point, fixed, design page updated.
>
>>>>> Patch 0239 makes test setup change the hostname, /etc/hosts, and
>>>>> /etc/resolv.conf to match the configured values. These should be
>>>>> equivalent to the fixes in
>>>>> https://github.com/freeipa/tests/blob/master/ipa-tests/beaker/ipa-server/shared/ipa-install.sh#L733
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> In test teardown, the changes are undone.
>>>>
>>>> I've rebased the patrchset and added small fixes for patches 0232 and
>>>> 0239.
>>>>
>>>> New patch 0240 contains a few fixes/improvements to the Host class that
>>>> were not trivial to rebase into previous patches.
>>>>
>>>> The WIP patch adds a sketch of some of the tests for CA-less
>>>> (http://www.freeipa.org/page/V3/CA-less_install#Test_Plan). Please
>>>> comment if you can see where things can be improved for test authors!
>>>
>>> Just a word of warning, there are still a few test cases I need to add
>>> to the CA-less test plan.
>
> Sure. I did this mainly to see how things look from the test author's
> point of view.
>

Adding two additional patches for better Beaker integration:
Patch 0241 allows e.g. adding ticket numbers for automatic test case 
management
Patch 0242 should bring the BeakerLib logging closer to what 
traditional Beaker tests output

-- 
Petr³

-------------- next part --------------
A non-text attachment was scrubbed...
Name: freeipa-pviktori-0241.3-Use-dosctrings-in-BeakerLib-phase-descriptions.patch
Type: text/x-patch
Size: 1906 bytes
Desc: not available
URL: <http://listman.redhat.com/archives/freeipa-devel/attachments/20130625/f5e92506/attachment.bin>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: freeipa-pviktori-0242.3-Make-BeakerLib-logging-less-verbose.patch
Type: text/x-patch
Size: 7517 bytes
Desc: not available
URL: <http://listman.redhat.com/archives/freeipa-devel/attachments/20130625/f5e92506/attachment-0001.bin>