[Freeipa-devel] [freeipa PR#310][opened] WIP: CLI testing

[mirielka]

   URL: https://github.com/freeipa/freeipa/pull/310
Author: mirielka
 Title: #310: WIP: CLI testing
Action: opened

PR body:
"""
Here is basic part of CLI testing for you to take a look at and provide feedback, before it's all done and polished up.

How it works: so far it's only tuned to run stageuser tests, so use `./make-tests ipatests/test_xmlrpc/test_stageuser_plugin.py --cli` to run in CLI mode, or of course without the --cli option to run original API tests. Note that last three tests are expected to fail in CLI mode as group tracker has not been modified for CLI testing yet.

What gets changed: 
- '--cli' option is added
- base tracker: changes that should ensure the basic functionality to run existing tests in CLI mode, this includes way how to execute commands and mapping of defined API options and output values to the CLI style (the idea of the mapping was to convert the text output from CLI to API style output so that more extensive methods to compare CLI output with expected values do not have to be created)
- specific trackers: changes are needed because CLI calls have different output than API calls (some messages are different, some items of API output are not present in CLI). So far I've modified stageuser tracker fully (i.e. that one should be final) and user tracker partly (just so that stageuser tests that use user tracker work properly, but user tests will not work in CLI mode yet).
- xmlrpc_test.py: unfortunately I failed to find a way how to raise the exception from subprocess so that it would fit the exception comparison using raises_exact method, so I chose to raise ExecutionError when it occurs in subprocess and compare just the error message in stderr. Since I expect the CLI tests to be run along with API tests, any change in type of raised error should be cought by API tests.

So far I observed:
- positives: no tests need to be changed, all the necessary changes are in trackers (which was the intention)
- negatives: only works for tracker based tests, i.e. not for declarative tests or tests that use just api.Command style; tests run longer that API tests

So please, if you have any comment and suggestions, I'll be glad to hear them so that I can polish this and finish the modifications.
"""

To pull the PR as Git branch:
git remote add ghfreeipa https://github.com/freeipa/freeipa
git fetch ghfreeipa pull/310/head:pr310
git checkout pr310
-------------- next part --------------
A non-text attachment was scrubbed...
Name: freeipa-pr-310.patch
Type: text/x-diff
Size: 40729 bytes
Desc: not available
URL: <http://listman.redhat.com/archives/freeipa-devel/attachments/20161206/0f9d1919/attachment.bin>