[Freeipa-devel] Unit tests failing on F18

[Rob Crittenden]

Martin Kosek wrote:
> On 10/12/2012 06:21 PM, Rob Crittenden wrote:
>> John Dennis wrote:
>>> On 10/12/2012 11:20 AM, Martin Kosek wrote:
>>>> Hello,
>>>>
>>>> I was investigating global unit test failure on Fedora 18 for most of
>>>> today, I
>>>> would like to share results I found so far.
>>>>
>>>> Unit test and its related scripts on F18 now reports NSS BUSY
>>>> exception, just
>>>> like this one:
>>>>
>>>> # ./make-testcert
>>>> Traceback (most recent call last):
>>>>     File "./make-testcert", line 134, in <module>
>>>>       sys.exit(makecert(reqdir))
>>>>     File "./make-testcert", line 111, in makecert
>>>>       add=True)
>>>>     File "./make-testcert", line 68, in run
>>>>       result = self.execute(method, *args, **options)
>>>>     File "/root/freeipa-master2/ipalib/backend.py", line 146, in execute
>>>>       raise error #pylint: disable=E0702
>>>> ipalib.errors.NetworkError: cannot connect to
>>>> 'http://vm-042.idm.lab.bos.redhat.com/ipa/session/xml': [Errno -8053]
>>>> (SEC_ERROR_BUSY) NSS could not shutdown. Objects are still in use.
>>>>
>>>> Something In F18 must have changed, this worked before... But leaked
>>>> NSSConnection objects without proper close() now ends with the
>>>> exception above.
>>>>
>>>> In case of make-testcert script, the exception is raised because the
>>>> script
>>>> does the following procedure:
>>>>
>>>> 1) connect, do one command
>>>> 2) disconnect
>>>> 3) connect, do second command
>>>>
>>>> However, during disconnect, NSSConnection is leaked which makes NSS very
>>>> uncomfortable during second connection atempt (and nss_shutdown()). I
>>>> managed
>>>> to fix this issue with attached patch. ./make-testcert or "./make-test
>>>> tests/test_xmlrpc/test_group_plugin.py" works fine now.
>>>>
>>>> But global "./make-test" still fails, I think there is some remaining
>>>> NSSConnection leak, I suspect there is something wrong with how we use
>>>> our
>>>> context (threading.local object). It looses a connection or some other
>>>> thread
>>>> invoked in ldap2 module may be kicking in, here is my debug output:
>>>>
>>>> CONTEXT[xmlclient] = <ipalib.request.Connection object at 0x9a1f5ec>
>>>>
>>>> Test a simple LDAP bind using ldap2 ... SKIP: No directory manager
>>>> password in
>>>> /root/.ipa/.dmpw
>>>> Test the `ipaserver.rpcserver.jsonserver.unmarshal` method. ... ok
>>>> tests.test_ipaserver.test_rpcserver.test_session.test_mount ... CONTEXT
>>>> 150714476: GET languages
>>>>
>>>> CONTEXT[xmlclient] = None
>>>>
>>>> The connection is in the context, but then something happens and it is
>>>> gone.
>>>> Then, unit tests try to connect again and NSS fails.
>>>>
>>>> I would be really glad if somebody with a knowledge of NSS or how
>>>> threads in
>>>> Python/IPA work could give me some advice...
>>>
>>> O.K. I'll take a look at it. I seem to recall Rob looked into something
>>> similar a couple of days ago. Rob, do you have any additional
>>> information to share?
>>>
>>>
>>
>> Nothing to add. I looked at this briefly yesterday but nothing jumped out at
>> me. I think this may be unrelated to the previous problem I investigated. It
>> very well could be we are not closing a connection somewhere.
>>
>> rob
>
> Yes, we are not. I fixed it in one place (patch in my original e-mail). With
> the patch, one could run at least some parts of the unit test. I assume we are
> simply not treating initializing and shutting down of NSS correctly in some
> places (also with regards to threads), but I have not found a fix yet.

If you modify tests/test_xmlrpc/xmlrpc_test.py and remove 
request.destroy_context() from tearDown then the tests will pass. But 
this shouldn't be necessary, we really want to clean up the context 
between requests.

I'm guessing that we're being affected by an external change. It could 
be in a number of places: httplib, xmlrpclib, NSS, etc.

rob