[Freeipa-users] ipa replica failed PR_DeleteSemaphore

[Andrew E. Bruno]

On Wed, Mar 09, 2016 at 05:21:50PM +0100, Ludwig Krispenz wrote:
> 
> On 03/09/2016 04:46 PM, Andrew E. Bruno wrote:
> >On Wed, Mar 09, 2016 at 10:37:05AM -0500, Andrew E. Bruno wrote:
> >>On Wed, Mar 09, 2016 at 04:13:28PM +0100, Ludwig Krispenz wrote:
> >>>if the process hangs, could you get a pstack from the process ?
> >>
> >>I'd be happy to provide a pstack but can't seem to get the correct debuginfo
> >>packages installed.. we're running centos7 and  389-ds-base 1.3.3.1. We haven't
> >>upgraded to 1.3.4.0. How can I get the debuginfo packages installed for that
> >>specific version.
> >Nevermind.. i got the debuginfo packages. Attached is the stacktrace of
> >our second failed replicate that's currently hung.
> not sure, but the process could be in a deadlock, there are threads in the
> retro cl and memberof plugin and we have seen deadlocks there. In that case
> restart or stop would not be able to stop the ds process. You can try ipactl
> stop and if the ds process is still running, you have to kill it

Our first master came back up after the restart and appears to be
working again. The second replica that was hung, we did a ipactl stop
and it killed the ds process. Running a ipactl start now. We got the
same error:


[09/Mar/2016:11:33:03 -0500] NSMMReplicationPlugin - changelog program - _cl5NewDBFile: PR_DeleteSemaphore: /var/lib/dirsrv/slapd-CBLS-CCR-BUFFALO-EDU/cldb/ed35d212-2cb811e5-af63d574-de3f6355.sema; NSPR error - -5943


We're going to let this run and hopefully will come back up. 

Just want to confirm again that these can can be safely ignored:

[09/Mar/2016:10:23:10 -0500] DSRetroclPlugin - delete_changerecord: could not delete change record 11272989 (rc: 32)
[09/Mar/2016:10:23:10 -0500] DSRetroclPlugin - delete_changerecord: could not delete change record 11272990 (rc: 32)

They fill up the logs when bringing the ds back up.

We seem to keep getting bit by this deadlock [1,2]. Replicas become
unresponsive, file descriptor counts increase. Other than a pstack, if there's
any other info we can provide/check let us know. 

We'll be upgrading to centos 7.2 and 389-ds-base 1.3.4 next week.

As always, thanks again for the help and quick responses. 

Best,

--Andrew

[1] https://www.redhat.com/archives/freeipa-users/2015-September/msg00006.html
[2] https://www.redhat.com/archives/freeipa-users/2015-June/msg00389.html