[Freeipa-devel] [PATCHES] 0230-0240 Integration testing framework

[Jan Cholasta]

On 25.6.2013 14:08, Petr Viktorin wrote:
> Adding two additional patches for better Beaker integration:
> Patch 0241 allows e.g. adding ticket numbers for automatic test case
> management
> Patch 0242 should bring the BeakerLib logging closer to what traditional
> Beaker tests output
>

make test seems to run fine with patches 230-242 applied, however 
ipa-run-tests produces the following output:

======================================================================
ERROR: Test that `ipa help` only writes to stdout
----------------------------------------------------------------------
Traceback (most recent call last):
   File "/usr/lib/python2.7/site-packages/nose/case.py", line 197, in 
runTest
     self.test(*self.arg)
   File 
"/usr/lib/python2.7/site-packages/ipatests/test_cmdline/test_help.py", 
line 63, in test_ipa_help
     return_value = api.Backend.cli.run(['help'])
   File "/usr/lib/python2.7/site-packages/ipalib/cli.py", line 1067, in run
     result = self.execute(name, **kw)
   File "/usr/lib/python2.7/site-packages/ipalib/backend.py", line 146, 
in execute
     raise error #pylint: disable=E0702
InternalError: an internal error has occurred
-------------------- >> begin captured logging << --------------------
IPA: DEBUG: [ipa.ipalib.cli.help] raw: help(None, version=u'2.60')
IPA: DEBUG: [ipa.ipalib.cli.help] help(None, version=u'2.60')
IPA: ERROR: [ipa.ipalib.cli.cli] non-public: AttributeError: 'API' 
object has no attribute 'parser'
Traceback (most recent call last):
   File "/usr/lib/python2.7/site-packages/ipalib/backend.py", line 129, 
in execute
     result = self.Command[_name](*args, **options)
   File "/usr/lib/python2.7/site-packages/ipalib/frontend.py", line 435, 
in __call__
     ret = self.run(*args, **options)
   File "/usr/lib/python2.7/site-packages/ipalib/cli.py", line 765, in run
     self.api.parser.print_help(outfile)
AttributeError: 'API' object has no attribute 'parser'
IPA: DEBUG: [ipa.ipalib.rpc.xmlclient] Destroyed connection 
context.xmlclient
--------------------- >> end captured logging << ---------------------

======================================================================
ERROR: Failure: IOError ([Errno 2] No such file or directory: 
'ipatests/test_xmlrpc/service.crt')
----------------------------------------------------------------------
Traceback (most recent call last):
   File "/usr/lib/python2.7/site-packages/nose/loader.py", line 413, in 
loadTestsFromName
     addr.filename, addr.module)
   File "/usr/lib/python2.7/site-packages/nose/importer.py", line 47, in 
importFromPath
     return self.importFromDir(dir_path, fqname)
   File "/usr/lib/python2.7/site-packages/nose/importer.py", line 94, in 
importFromDir
     mod = load_module(part_fqname, fh, filename, desc)
   File 
"/usr/lib/python2.7/site-packages/ipatests/test_xmlrpc/test_host_plugin.py", 
line 59, in <module>
     fd = open('ipatests/test_xmlrpc/service.crt', 'r')
IOError: [Errno 2] No such file or directory: 
'ipatests/test_xmlrpc/service.crt'

======================================================================
ERROR: Failure: IOError ([Errno 2] No such file or directory: 
'ipatests/test_xmlrpc/service.crt')
----------------------------------------------------------------------
Traceback (most recent call last):
   File "/usr/lib/python2.7/site-packages/nose/loader.py", line 413, in 
loadTestsFromName
     addr.filename, addr.module)
   File "/usr/lib/python2.7/site-packages/nose/importer.py", line 47, in 
importFromPath
     return self.importFromDir(dir_path, fqname)
   File "/usr/lib/python2.7/site-packages/nose/importer.py", line 94, in 
importFromDir
     mod = load_module(part_fqname, fh, filename, desc)
   File 
"/usr/lib/python2.7/site-packages/ipatests/test_xmlrpc/test_service_plugin.py", 
line 42, in <module>
     fd = open('ipatests/test_xmlrpc/service.crt', 'r')
IOError: [Errno 2] No such file or directory: 
'ipatests/test_xmlrpc/service.crt'

======================================================================
FAIL: Test that `ipa` errors out, and prints the help to stderr
----------------------------------------------------------------------
Traceback (most recent call last):
   File "/usr/lib/python2.7/site-packages/nose/case.py", line 197, in 
runTest
     self.test(*self.arg)
   File 
"/usr/lib/python2.7/site-packages/ipatests/test_cmdline/test_help.py", 
line 71, in test_ipa_without_arguments
     api.Backend.cli.run([])
   File 
"/usr/lib/python2.7/site-packages/ipatests/test_cmdline/test_help.py", 
line 55, in __exit__
     assert isinstance(exc_value, self.exception), exc_value
AssertionError: 'API' object has no attribute 'parser'
-------------------- >> begin captured logging << --------------------
IPA: DEBUG: [ipa.ipalib.cli.help] raw: help(None, 
outfile=<StringIO.StringIO instance at 0x486cb48>)
IPA: DEBUG: [ipa.ipalib.cli.help] help(None, outfile=<StringIO.StringIO 
instance at 0x486cb48>)
--------------------- >> end captured logging << ---------------------


I guess the location of the test certificate should be made configurable 
in order to fix the host and service plugin test failures.

Also, there is a lot of debugging messages in ipa-run-tests output which 
wasn't there before IIRC, is that intentional?


ipa-test-config should do more command line options validation:

$ ipa-test-config --master
Traceback (most recent call last):
   File "/usr/bin/ipa-test-config", line 104, in <module>
     print main(sys.argv[1:]),
   File "/usr/bin/ipa-test-config", line 58, in main
     return config.env_to_script(get_object(conf, args).to_env(**kwargs))
   File "/usr/bin/ipa-test-config", line 84, in get_object
     domain = conf.domain[0]
AttributeError: 'Config' object has no attribute 'domain'


If $MASTER (and possible other host names) is not resolvable, 
ipa-test-config --global crashes:

$ ipa-test-config --global
Traceback (most recent call last):
   File "/usr/bin/ipa-test-config", line 104, in <module>
     print main(sys.argv[1:]),
   File "/usr/bin/ipa-test-config", line 58, in main
     return config.env_to_script(get_object(conf, args).to_env(**kwargs))
   File 
"/usr/lib/python2.7/site-packages/ipatests/test_integration/config.py", 
line 168, in to_env
     env['MASTER'] = default_domain.master.hostname
   File 
"/usr/lib/python2.7/site-packages/ipatests/test_integration/config.py", 
line 282, in master
     return self.masters[0]
IndexError: list index out of range


Both the tests in test_simple_replication.py fail for me. I suspect it 
is because the data isn't replicated fast enough, a little delay between 
user-show and user-add might fix this.

Honza

-- 
Jan Cholasta