[Freeipa-devel] [RFE] Integration testing

[Petr Vobornik]

On 06/04/2013 07:19 PM, Petr Viktorin wrote:
> On 06/04/2013 06:15 PM, Petr Vobornik wrote:
>> Hello,
>>
>> Some Qs:
>>
>> 1) does the configuration only mean: "We have this topology, run all
>> test which fits into it"?
>
> Not quite, it means "We have these machines, run tests whose topology
> can fit on them".
>
>> IDK how should Web UI or XML-RPC tests fit
>> into it.
>
> The existing test suite, including the RPC tests, stays as it is.

> The
> single-machine tests generally expect IPA to be set up on the local
> machine.
>

I wouldn't make that assumption. We should allow to have test runner and 
FreeIPA server on different machines.

>> The difference is that configuration for Web UI tests should
>> mean: "This is how FreeIPA and related stuff is installed. Tests all
>> available functionality and don't test the missing." IE. if I install
>> freeipa server without CA, the UI tests needs to get that info (ie. by
>> no_ca flag) and then skip certificate tests or parts of other tests
>> which touches this feature (like navigation tests). The same for no-dns
>> and has-trust...
>>
>> Complete UI testing would be something like following:
>>    a) set configuration A
>>    b) install server with configuration A
>>    c) run all UI tests (some may be skipped)
>>         i) in Firefox
>>       ii) in IE
>>          iii) in Chrome
>>    d) uninstall server
>>    e) set configuration B
>>    f) install server with configuration B
>>    g) run all UI tests (some may be skipped)
>>        i) in Firefox
>>       ii) in IE
>>          iii) in Chrome
>>    h) uninstall server
>>    i) repeat for config C
>>    ...
>>
>> Note:: browser change is also done by change of configuration.
>>
>> Is it possible? Can the configuration change or should there be other
>> level of configuration which can change?  Do we have time to do it? It
>> may take almost a day (sequentially, with full coverage).
>
> This would be configured in Jenkins as several jobs:
>
> 1) Run UI tests with configuration A + Firefox
> 2) Run UI tests with configuration A + IE
> 3) Run UI tests with configuration A + Chrome
> 4) Run UI tests with configuration B + Firefox
> 5) Run UI tests with configuration B + IE
> 6) Run UI tests with configuration B + Chrome
>
> If we wanted exhaustive tests it could be done as a "matrix job" ([A,
> B]×[FF, IE, Chrome]), but as you say we don't have resources for that,
> so we'll have to use some sane subset.
>
> (Note: Jenkins can aggregate results from multiple jobs, so we'd get a
> single report from all these)
>
>> Does python nose support some master tests or can it be somehow
>> configured in Jenkins so we can automate installation of IPA with
>> different configurations (IE by the 'To-be-done command-line tools').
>
> I'm not sure what you mean by master tests.
> Jenkins can be configured to run different tests using different
> configurations.
> But the way I see it, the different topologies would be in different
> tests, and run different checks. remember that we want a CI, not
> exhaustive testing of all the possible cases.

Thanks for the info. So it would be safe to have only one job for Web UI 
in CI, the one with most complete feature set and most supported 
browser(FF). All tests for various config/browser combinations can be 
executed manually when needed.

>
> The command-line tools would be for external shell-based tests; I've
> heard some people might want to write those :)
>
>> 2) There is no configuration options for trusts. IMO we would like to
>> test that.
>
> Options for that can be added when the trust tests are written.
>
>> 3) Test runner needs to connect to remote machine as root. Should a
>> config option for root password be added or can we safely assume that
>> there will always be other authentication method available?
>
> My bad, I forgot to add $IPA_ROOT_SSH_PASSWORD to the wiki page. Fixed.
>
> Other authentication methods can be added if needed.
>


-- 
Petr Vobornik