[Freeipa-users] ipa replica failed PR_DeleteSemaphore

[Ludwig Krispenz]

On 03/09/2016 05:51 PM, Andrew E. Bruno wrote:
> On Wed, Mar 09, 2016 at 05:21:50PM +0100, Ludwig Krispenz wrote:
>> On 03/09/2016 04:46 PM, Andrew E. Bruno wrote:
>>> On Wed, Mar 09, 2016 at 10:37:05AM -0500, Andrew E. Bruno wrote:
>>>> On Wed, Mar 09, 2016 at 04:13:28PM +0100, Ludwig Krispenz wrote:
>>>>> if the process hangs, could you get a pstack from the process ?
>>>> I'd be happy to provide a pstack but can't seem to get the correct debuginfo
>>>> packages installed.. we're running centos7 and  389-ds-base 1.3.3.1. We haven't
>>>> upgraded to 1.3.4.0. How can I get the debuginfo packages installed for that
>>>> specific version.
>>> Nevermind.. i got the debuginfo packages. Attached is the stacktrace of
>>> our second failed replicate that's currently hung.
>> not sure, but the process could be in a deadlock, there are threads in the
>> retro cl and memberof plugin and we have seen deadlocks there. In that case
>> restart or stop would not be able to stop the ds process. You can try ipactl
>> stop and if the ds process is still running, you have to kill it
> Our first master came back up after the restart and appears to be
> working again. The second replica that was hung, we did a ipactl stop
> and it killed the ds process. Running a ipactl start now. We got the
> same error:
>
>
> [09/Mar/2016:11:33:03 -0500] NSMMReplicationPlugin - changelog program - _cl5NewDBFile: PR_DeleteSemaphore: /var/lib/dirsrv/slapd-CBLS-CCR-BUFFALO-EDU/cldb/ed35d212-2cb811e5-af63d574-de3f6355.sema; NSPR error - -5943
if ds is cleanly shutdown this file should be removed, if ds is killed 
it remains and should be recreated at restart, which fails. could you 
try another stop, remove the file manually and start again ?
>
>
> We're going to let this run and hopefully will come back up.
>
> Just want to confirm again that these can can be safely ignored:
>
> [09/Mar/2016:10:23:10 -0500] DSRetroclPlugin - delete_changerecord: could not delete change record 11272989 (rc: 32)
> [09/Mar/2016:10:23:10 -0500] DSRetroclPlugin - delete_changerecord: could not delete change record 11272990 (rc: 32)
there is something wrong with defining the starting point for changelog 
trimming, so it will skip many entries, this is annoying and we will 
have to fix it. Apart from spamming the logs and keeping the retor cl 
busy for a while it should not do any harm.
>
> They fill up the logs when bringing the ds back up.
>
> We seem to keep getting bit by this deadlock [1,2]. Replicas become
> unresponsive, file descriptor counts increase. Other than a pstack, if there's
> any other info we can provide/check let us know.
>
> We'll be upgrading to centos 7.2 and 389-ds-base 1.3.4 next week.
>
> As always, thanks again for the help and quick responses.
>
> Best,
>
> --Andrew
>
> [1] https://www.redhat.com/archives/freeipa-users/2015-September/msg00006.html
> [2] https://www.redhat.com/archives/freeipa-users/2015-June/msg00389.html

-- 
Red Hat GmbH, http://www.de.redhat.com/, Registered seat: Grasbrunn,
Commercial register: Amtsgericht Muenchen, HRB 153243,
Managing Directors: Paul Argiry, Charles Cachera, Michael Cunningham, Michael O'Neill