[Freeipa-devel] [RFE] Integration testing

[Petr Viktorin]

On 06/04/2013 06:15 PM, Petr Vobornik wrote:
> Hello,
>
> Some Qs:
>
> 1) does the configuration only mean: "We have this topology, run all
> test which fits into it"?

Not quite, it means "We have these machines, run tests whose topology 
can fit on them".

> IDK how should Web UI or XML-RPC tests fit
> into it.

The existing test suite, including the RPC tests, stays as it is. The 
single-machine tests generally expect IPA to be set up on the local machine.

> The difference is that configuration for Web UI tests should
> mean: "This is how FreeIPA and related stuff is installed. Tests all
> available functionality and don't test the missing." IE. if I install
> freeipa server without CA, the UI tests needs to get that info (ie. by
> no_ca flag) and then skip certificate tests or parts of other tests
> which touches this feature (like navigation tests). The same for no-dns
> and has-trust...
>
> Complete UI testing would be something like following:
>    a) set configuration A
>    b) install server with configuration A
>    c) run all UI tests (some may be skipped)
>         i) in Firefox
>       ii) in IE
>          iii) in Chrome
>    d) uninstall server
>    e) set configuration B
>    f) install server with configuration B
>    g) run all UI tests (some may be skipped)
>        i) in Firefox
>       ii) in IE
>          iii) in Chrome
>    h) uninstall server
>    i) repeat for config C
>    ...
>
> Note:: browser change is also done by change of configuration.
>
> Is it possible? Can the configuration change or should there be other
> level of configuration which can change?  Do we have time to do it? It
> may take almost a day (sequentially, with full coverage).

This would be configured in Jenkins as several jobs:

1) Run UI tests with configuration A + Firefox
2) Run UI tests with configuration A + IE
3) Run UI tests with configuration A + Chrome
4) Run UI tests with configuration B + Firefox
5) Run UI tests with configuration B + IE
6) Run UI tests with configuration B + Chrome

If we wanted exhaustive tests it could be done as a "matrix job" ([A, 
B]×[FF, IE, Chrome]), but as you say we don't have resources for that, 
so we'll have to use some sane subset.

(Note: Jenkins can aggregate results from multiple jobs, so we'd get a 
single report from all these)

> Does python nose support some master tests or can it be somehow
> configured in Jenkins so we can automate installation of IPA with
> different configurations (IE by the 'To-be-done command-line tools').

I'm not sure what you mean by master tests.
Jenkins can be configured to run different tests using different 
configurations.
But the way I see it, the different topologies would be in different 
tests, and run different checks. remember that we want a CI, not 
exhaustive testing of all the possible cases.

The command-line tools would be for external shell-based tests; I've 
heard some people might want to write those :)

> 2) There is no configuration options for trusts. IMO we would like to
> test that.

Options for that can be added when the trust tests are written.

> 3) Test runner needs to connect to remote machine as root. Should a
> config option for root password be added or can we safely assume that
> there will always be other authentication method available?

My bad, I forgot to add $IPA_ROOT_SSH_PASSWORD to the wiki page. Fixed.

Other authentication methods can be added if needed.

-- 
Petr³