[Freeipa-devel] [RFE] Integration testing

[Petr Viktorin]

On 06/05/2013 12:49 PM, Petr Vobornik wrote:
> On 06/04/2013 07:19 PM, Petr Viktorin wrote:
>> On 06/04/2013 06:15 PM, Petr Vobornik wrote:
>>> Hello,
>>>
>>> Some Qs:
>>>
>>> 1) does the configuration only mean: "We have this topology, run all
>>> test which fits into it"?
>>
>> Not quite, it means "We have these machines, run tests whose topology
>> can fit on them".
>>
>>> IDK how should Web UI or XML-RPC tests fit
>>> into it.
>>
>> The existing test suite, including the RPC tests, stays as it is.
>
>> The
>> single-machine tests generally expect IPA to be set up on the local
>> machine.
>>
>
> I wouldn't make that assumption. We should allow to have test runner and
> FreeIPA server on different machines.

That would mean changing our entire existing test suite.
For a single machine, the test runner would basically just SSH into the 
server and run a command or script there. You (or Jenkins) can do that 
easily.

>>> The difference is that configuration for Web UI tests should
>>> mean: "This is how FreeIPA and related stuff is installed. Tests all
>>> available functionality and don't test the missing." IE. if I install
>>> freeipa server without CA, the UI tests needs to get that info (ie. by
>>> no_ca flag) and then skip certificate tests or parts of other tests
>>> which touches this feature (like navigation tests). The same for no-dns
>>> and has-trust...
>>>
>>> Complete UI testing would be something like following:
>>>    a) set configuration A
>>>    b) install server with configuration A
>>>    c) run all UI tests (some may be skipped)
>>>         i) in Firefox
>>>       ii) in IE
>>>          iii) in Chrome
>>>    d) uninstall server
>>>    e) set configuration B
>>>    f) install server with configuration B
>>>    g) run all UI tests (some may be skipped)
>>>        i) in Firefox
>>>       ii) in IE
>>>          iii) in Chrome
>>>    h) uninstall server
>>>    i) repeat for config C
>>>    ...
>>>
>>> Note:: browser change is also done by change of configuration.
>>>
>>> Is it possible? Can the configuration change or should there be other
>>> level of configuration which can change?  Do we have time to do it? It
>>> may take almost a day (sequentially, with full coverage).
>>
>> This would be configured in Jenkins as several jobs:
>>
>> 1) Run UI tests with configuration A + Firefox
>> 2) Run UI tests with configuration A + IE
>> 3) Run UI tests with configuration A + Chrome
>> 4) Run UI tests with configuration B + Firefox
>> 5) Run UI tests with configuration B + IE
>> 6) Run UI tests with configuration B + Chrome
>>
>> If we wanted exhaustive tests it could be done as a "matrix job" ([A,
>> B]×[FF, IE, Chrome]), but as you say we don't have resources for that,
>> so we'll have to use some sane subset.
>>
>> (Note: Jenkins can aggregate results from multiple jobs, so we'd get a
>> single report from all these)
>>
>>> Does python nose support some master tests or can it be somehow
>>> configured in Jenkins so we can automate installation of IPA with
>>> different configurations (IE by the 'To-be-done command-line tools').
>>
>> I'm not sure what you mean by master tests.
>> Jenkins can be configured to run different tests using different
>> configurations.
>> But the way I see it, the different topologies would be in different
>> tests, and run different checks. remember that we want a CI, not
>> exhaustive testing of all the possible cases.
>
> Thanks for the info. So it would be safe to have only one job for Web UI
> in CI, the one with most complete feature set and most supported
> browser(FF). All tests for various config/browser combinations can be
> executed manually when needed.

Yes.
Additionally, if there is some subset of tests that we want to always 
run on another browser, we can mark them with Nose's attrib plugin, and 
run them in a separate job.

-- 
Petr³