[Freeipa-devel] [PATCHES] 0230-0244 Integration testing framework

[Jan Cholasta]

On 10.7.2013 17:50, Petr Viktorin wrote:
> On 07/10/2013 02:03 PM, Jan Cholasta wrote:
>> make test seems to run fine with patches 230-242 applied, however
>> ipa-run-tests produces the following output:
>>
> [...Skipping nose output...]
>>
>>
>> I guess the location of the test certificate should be made configurable
>> in order to fix the host and service plugin test failures.
>
> Making the existing test suite pass out of tree it's not a goal for this
> patchset. There is a pending patch for the service cert.

Better report it now than forget it later.

>
>> Also, there is a lot of debugging messages in ipa-run-tests output which
>> wasn't there before IIRC, is that intentional?
>
> Yes, I believe that for the integration tests it's better to see what is
> going on. Both for manual runs, and also this way the information is
> more easily picked up by CI tools.

It's harder to see what's going on with so much noise IMHO, I would 
prefer less verbose output for manual runs by default.

>
>> If $MASTER (and possible other host names) is not resolvable,
>> ipa-test-config --global crashes:
>>
>> $ ipa-test-config --global
>> Traceback (most recent call last):
>>    File "/usr/bin/ipa-test-config", line 104, in <module>
>>      print main(sys.argv[1:]),
>>    File "/usr/bin/ipa-test-config", line 58, in main
>>      return config.env_to_script(get_object(conf, args).to_env(**kwargs))
>>    File
>> "/usr/lib/python2.7/site-packages/ipatests/test_integration/config.py",
>> line 168, in to_env
>>      env['MASTER'] = default_domain.master.hostname
>>    File
>> "/usr/lib/python2.7/site-packages/ipatests/test_integration/config.py",
>> line 282, in master
>>      return self.masters[0]
>> IndexError: list index out of range
>
> Fixed.
> Now, if the name is not resolvable, reading the config will fail.
> Alternatively, the IP address can be given in variables such as
> $BEAKERREPLICA1_IP_env1 (no, I didn't invent the name).

This fixed ipa-test-config as advertised, but "ipa-run-tests 
test_integration/test_simple_replication.py" still fails.

>
>> Both the tests in test_simple_replication.py fail for me. I suspect it
>> is because the data isn't replicated fast enough, a little delay between
>> user-show and user-add might fix this.
>
> Hm, they worked for me.
>   I've added a delay. It seems fragile, I wonder what the proper way to
> do this would be.

Hmm, the "replica -> master" test still fails for me, even with delay 
increased to 20 s.

Something less fragile than delay would certainly be nice.

>
>
> I've made some other changes, mainly BeakerLib plugin output.
>
> Patch 240: Avoid infinite recursion that happened with some cases of bad
> SSH credentials
> Patch 241-243: Rework the BeakerLib plugin output to better match
> traditional Beaker tests
> Patch 244: Make it possible to explicitly specify IP addresses of hosts

To sum things up, there are still some little bugs, but these can be 
fixed after the beta release, in general everything seems to work, so ACK.

Honza

-- 
Jan Cholasta