[Freeipa-devel] [PATCH] 0029 Check expected error messages in tests

[Rob Crittenden]

Petr Viktorin wrote:
> On 03/26/2012 09:49 PM, Rob Crittenden wrote:
>> Petr Viktorin wrote:
>>> On 03/20/2012 01:39 PM, Petr Viktorin wrote:
>>>> This patch adds checking error messages, not just types, to the XML-RPC
>>>> tests.
>>>> The checking is still somewhat hackish, since XML-RPC doesn't give us
>>>> structured error info, but it should protect against regressions on
>>>> issues like whether we put name or cli_name in a ValidationError.
>>>>
>>>> https://fedorahosted.org/freeipa/ticket/2549
>>>>
>>>
>>> Updated and rebased to current master.
>>
>> NACK
>>
>> automember wrongly was testing for non-existent users rather than
>> automember rules but those should still be tested IMHO, perhaps with
>> both types.
>>
>> There is also some inconsistency. In host you use substitution to set
>> the hostname in the error: '%s: host not found' % fqdn1 but in others
>> (group, hostgroup for example) the name is hardcoded. I also noticed
>> that some reasons are unicode and others are not.
>>
>> rob
>
> Added tests for automember, made all the reasons unicode, using
> substitutions when variables are involved.
>
> The patch still only updates tests that didn't pass the error message
> check.
>

I'm seeing three failures that I think are due to recently pushed 
patches. Otherwise it looks good.

rob


======================================================================
FAIL: test_netgroup[1]: netgroup_mod: Try to update non-existent 
u'netgroup1'
----------------------------------------------------------------------
Traceback (most recent call last):
   File "/usr/lib/python2.7/site-packages/nose/case.py", line 187, in 
runTest
     self.test(*self.arg)
   File 
"/home/rcrit/redhat/freeipa-beta1/tests/test_xmlrpc/xmlrpc_test.py", 
line 247, in <lambda>
     func = lambda: self.check(nice, **test)
   File 
"/home/rcrit/redhat/freeipa-beta1/tests/test_xmlrpc/xmlrpc_test.py", 
line 260, in check
     self.check_exception(nice, cmd, args, options, expected)
   File 
"/home/rcrit/redhat/freeipa-beta1/tests/test_xmlrpc/xmlrpc_test.py", 
line 284, in check_exception
     assert_deepequal(expected.strerror, e.strerror)
   File "/home/rcrit/redhat/freeipa-beta1/tests/util.py", line 328, in 
assert_deepequal
     VALUE % (doc, expected, got, stack)
AssertionError: assert_deepequal: expected != got.

   expected = u'netgroup1: netgroup not found'
   got = u'no such entry'
   path = ()

======================================================================
FAIL: test_netgroup[4]: netgroup_add: Test an invalid nisdomain1 name 
u'domain1,domain2'
----------------------------------------------------------------------
Traceback (most recent call last):
   File "/usr/lib/python2.7/site-packages/nose/case.py", line 187, in 
runTest
     self.test(*self.arg)
   File 
"/home/rcrit/redhat/freeipa-beta1/tests/test_xmlrpc/xmlrpc_test.py", 
line 247, in <lambda>
     func = lambda: self.check(nice, **test)
   File 
"/home/rcrit/redhat/freeipa-beta1/tests/test_xmlrpc/xmlrpc_test.py", 
line 260, in check
     self.check_exception(nice, cmd, args, options, expected)
   File 
"/home/rcrit/redhat/freeipa-beta1/tests/test_xmlrpc/xmlrpc_test.py", 
line 284, in check_exception
     assert_deepequal(expected.strerror, e.strerror)
   File "/home/rcrit/redhat/freeipa-beta1/tests/util.py", line 328, in 
assert_deepequal
     VALUE % (doc, expected, got, stack)
AssertionError: assert_deepequal: expected != got.

   expected = u"invalid 'nisdomainname': may only include letters, 
numbers, _, - and ."
   got = u"invalid 'nisdomain': may only include letters, numbers, _, -, 
and ."
   path = ()

======================================================================
FAIL: test_netgroup[5]: netgroup_add: Test an invalid nisdomain2 name 
u'+invalidnisdomain'
----------------------------------------------------------------------
Traceback (most recent call last):
   File "/usr/lib/python2.7/site-packages/nose/case.py", line 187, in 
runTest
     self.test(*self.arg)
   File 
"/home/rcrit/redhat/freeipa-beta1/tests/test_xmlrpc/xmlrpc_test.py", 
line 247, in <lambda>
     func = lambda: self.check(nice, **test)
   File 
"/home/rcrit/redhat/freeipa-beta1/tests/test_xmlrpc/xmlrpc_test.py", 
line 260, in check
     self.check_exception(nice, cmd, args, options, expected)
   File 
"/home/rcrit/redhat/freeipa-beta1/tests/test_xmlrpc/xmlrpc_test.py", 
line 284, in check_exception
     assert_deepequal(expected.strerror, e.strerror)
   File "/home/rcrit/redhat/freeipa-beta1/tests/util.py", line 328, in 
assert_deepequal
     VALUE % (doc, expected, got, stack)
AssertionError: assert_deepequal: expected != got.

   expected = u"invalid 'nisdomainname': may only include letters, 
numbers, _, - and ."
   got = u"invalid 'nisdomain': may only include letters, numbers, _, -, 
and ."
   path = ()