[Freeipa-devel] [PATCH] 0029 Check expected error messages in tests

[Rob Crittenden]

Petr Viktorin wrote:
> On 03/29/2012 11:17 PM, Rob Crittenden wrote:
>> Petr Viktorin wrote:
>>> On 03/26/2012 09:49 PM, Rob Crittenden wrote:
>>>> Petr Viktorin wrote:
>>>>> On 03/20/2012 01:39 PM, Petr Viktorin wrote:
>>>>>> This patch adds checking error messages, not just types, to the
>>>>>> XML-RPC
>>>>>> tests.
>>>>>> The checking is still somewhat hackish, since XML-RPC doesn't give us
>>>>>> structured error info, but it should protect against regressions on
>>>>>> issues like whether we put name or cli_name in a ValidationError.
>>>>>>
>>>>>> https://fedorahosted.org/freeipa/ticket/2549
>>>>>>
>>>>>
>>>>> Updated and rebased to current master.
>>>>
>>>> NACK
>>>>
>>>> automember wrongly was testing for non-existent users rather than
>>>> automember rules but those should still be tested IMHO, perhaps with
>>>> both types.
>>>>
>>>> There is also some inconsistency. In host you use substitution to set
>>>> the hostname in the error: '%s: host not found' % fqdn1 but in others
>>>> (group, hostgroup for example) the name is hardcoded. I also noticed
>>>> that some reasons are unicode and others are not.
>>>>
>>>> rob
>>>
>>> Added tests for automember, made all the reasons unicode, using
>>> substitutions when variables are involved.
>>>
>>> The patch still only updates tests that didn't pass the error message
>>> check.
>>>
>>
>> I'm seeing three failures that I think are due to recently pushed
>> patches. Otherwise it looks good.
>>
>> rob
>>
>>
>> ======================================================================
>> FAIL: test_netgroup[1]: netgroup_mod: Try to update non-existent
>> u'netgroup1'
>> ----------------------------------------------------------------------
>> Traceback (most recent call last):
>> File "/usr/lib/python2.7/site-packages/nose/case.py", line 187, in
>> runTest
>> self.test(*self.arg)
>> File
>> "/home/rcrit/redhat/freeipa-beta1/tests/test_xmlrpc/xmlrpc_test.py",
>> line 247, in <lambda>
>> func = lambda: self.check(nice, **test)
>> File
>> "/home/rcrit/redhat/freeipa-beta1/tests/test_xmlrpc/xmlrpc_test.py",
>> line 260, in check
>> self.check_exception(nice, cmd, args, options, expected)
>> File
>> "/home/rcrit/redhat/freeipa-beta1/tests/test_xmlrpc/xmlrpc_test.py",
>> line 284, in check_exception
>> assert_deepequal(expected.strerror, e.strerror)
>> File "/home/rcrit/redhat/freeipa-beta1/tests/util.py", line 328, in
>> assert_deepequal
>> VALUE % (doc, expected, got, stack)
>> AssertionError: assert_deepequal: expected != got.
>>
>> expected = u'netgroup1: netgroup not found'
>> got = u'no such entry'
>> path = ()
>>
>> ======================================================================
>> FAIL: test_netgroup[4]: netgroup_add: Test an invalid nisdomain1 name
>> u'domain1,domain2'
>> ----------------------------------------------------------------------
>> Traceback (most recent call last):
>> File "/usr/lib/python2.7/site-packages/nose/case.py", line 187, in
>> runTest
>> self.test(*self.arg)
>> File
>> "/home/rcrit/redhat/freeipa-beta1/tests/test_xmlrpc/xmlrpc_test.py",
>> line 247, in <lambda>
>> func = lambda: self.check(nice, **test)
>> File
>> "/home/rcrit/redhat/freeipa-beta1/tests/test_xmlrpc/xmlrpc_test.py",
>> line 260, in check
>> self.check_exception(nice, cmd, args, options, expected)
>> File
>> "/home/rcrit/redhat/freeipa-beta1/tests/test_xmlrpc/xmlrpc_test.py",
>> line 284, in check_exception
>> assert_deepequal(expected.strerror, e.strerror)
>> File "/home/rcrit/redhat/freeipa-beta1/tests/util.py", line 328, in
>> assert_deepequal
>> VALUE % (doc, expected, got, stack)
>> AssertionError: assert_deepequal: expected != got.
>>
>> expected = u"invalid 'nisdomainname': may only include letters, numbers,
>> _, - and ."
>> got = u"invalid 'nisdomain': may only include letters, numbers, _, -,
>> and ."
>> path = ()
>>
>> ======================================================================
>> FAIL: test_netgroup[5]: netgroup_add: Test an invalid nisdomain2 name
>> u'+invalidnisdomain'
>> ----------------------------------------------------------------------
>> Traceback (most recent call last):
>> File "/usr/lib/python2.7/site-packages/nose/case.py", line 187, in
>> runTest
>> self.test(*self.arg)
>> File
>> "/home/rcrit/redhat/freeipa-beta1/tests/test_xmlrpc/xmlrpc_test.py",
>> line 247, in <lambda>
>> func = lambda: self.check(nice, **test)
>> File
>> "/home/rcrit/redhat/freeipa-beta1/tests/test_xmlrpc/xmlrpc_test.py",
>> line 260, in check
>> self.check_exception(nice, cmd, args, options, expected)
>> File
>> "/home/rcrit/redhat/freeipa-beta1/tests/test_xmlrpc/xmlrpc_test.py",
>> line 284, in check_exception
>> assert_deepequal(expected.strerror, e.strerror)
>> File "/home/rcrit/redhat/freeipa-beta1/tests/util.py", line 328, in
>> assert_deepequal
>> VALUE % (doc, expected, got, stack)
>> AssertionError: assert_deepequal: expected != got.
>>
>> expected = u"invalid 'nisdomainname': may only include letters, numbers,
>> _, - and ."
>> got = u"invalid 'nisdomain': may only include letters, numbers, _, -,
>> and ."
>> path = ()
>>
>
> Updated patch attached.

Works great. ACK, pushed to master and ipa-2-2

rob