[Freeipa-devel] Unit tests failing on F18

[Martin Kosek]

On 10/12/2012 06:21 PM, Rob Crittenden wrote:
> John Dennis wrote:
>> On 10/12/2012 11:20 AM, Martin Kosek wrote:
>>> Hello,
>>>
>>> I was investigating global unit test failure on Fedora 18 for most of
>>> today, I
>>> would like to share results I found so far.
>>>
>>> Unit test and its related scripts on F18 now reports NSS BUSY
>>> exception, just
>>> like this one:
>>>
>>> # ./make-testcert
>>> Traceback (most recent call last):
>>>    File "./make-testcert", line 134, in <module>
>>>      sys.exit(makecert(reqdir))
>>>    File "./make-testcert", line 111, in makecert
>>>      add=True)
>>>    File "./make-testcert", line 68, in run
>>>      result = self.execute(method, *args, **options)
>>>    File "/root/freeipa-master2/ipalib/backend.py", line 146, in execute
>>>      raise error #pylint: disable=E0702
>>> ipalib.errors.NetworkError: cannot connect to
>>> 'http://vm-042.idm.lab.bos.redhat.com/ipa/session/xml': [Errno -8053]
>>> (SEC_ERROR_BUSY) NSS could not shutdown. Objects are still in use.
>>>
>>> Something In F18 must have changed, this worked before... But leaked
>>> NSSConnection objects without proper close() now ends with the
>>> exception above.
>>>
>>> In case of make-testcert script, the exception is raised because the
>>> script
>>> does the following procedure:
>>>
>>> 1) connect, do one command
>>> 2) disconnect
>>> 3) connect, do second command
>>>
>>> However, during disconnect, NSSConnection is leaked which makes NSS very
>>> uncomfortable during second connection atempt (and nss_shutdown()). I
>>> managed
>>> to fix this issue with attached patch. ./make-testcert or "./make-test
>>> tests/test_xmlrpc/test_group_plugin.py" works fine now.
>>>
>>> But global "./make-test" still fails, I think there is some remaining
>>> NSSConnection leak, I suspect there is something wrong with how we use
>>> our
>>> context (threading.local object). It looses a connection or some other
>>> thread
>>> invoked in ldap2 module may be kicking in, here is my debug output:
>>>
>>> CONTEXT[xmlclient] = <ipalib.request.Connection object at 0x9a1f5ec>
>>>
>>> Test a simple LDAP bind using ldap2 ... SKIP: No directory manager
>>> password in
>>> /root/.ipa/.dmpw
>>> Test the `ipaserver.rpcserver.jsonserver.unmarshal` method. ... ok
>>> tests.test_ipaserver.test_rpcserver.test_session.test_mount ... CONTEXT
>>> 150714476: GET languages
>>>
>>> CONTEXT[xmlclient] = None
>>>
>>> The connection is in the context, but then something happens and it is
>>> gone.
>>> Then, unit tests try to connect again and NSS fails.
>>>
>>> I would be really glad if somebody with a knowledge of NSS or how
>>> threads in
>>> Python/IPA work could give me some advice...
>>
>> O.K. I'll take a look at it. I seem to recall Rob looked into something
>> similar a couple of days ago. Rob, do you have any additional
>> information to share?
>>
>>
> 
> Nothing to add. I looked at this briefly yesterday but nothing jumped out at
> me. I think this may be unrelated to the previous problem I investigated. It
> very well could be we are not closing a connection somewhere.
> 
> rob

Yes, we are not. I fixed it in one place (patch in my original e-mail). With
the patch, one could run at least some parts of the unit test. I assume we are
simply not treating initializing and shutting down of NSS correctly in some
places (also with regards to threads), but I have not found a fix yet.

Martin