[Freeipa-users] RHEL 7.2 update - ns-slapd hanging system

[Andy Thompson]

> >>
> >> 		-----Original Message-----
> >> 		From: freeipa-users-bounces at redhat.com <mailto:freeipa-
> >> users-bounces at redhat.com>  [mailto:freeipa-users-
> >> 		bounces at redhat.com <mailto:bounces at redhat.com> ] On
> Behalf Of Petr
> >> Spacek
> >> 		Sent: Thursday, December 3, 2015 3:04 AM
> >> 		To: freeipa-users at redhat.com <mailto:freeipa-
> users at redhat.com>
> >> 		Subject: Re: [Freeipa-users] RHEL 7.2 update - ns-slapd
> hanging
> >> system
> >>
> >> 		On 2.12.2015 22:02, Alexander Bokovoy wrote:
> >>
> >> 			On Wed, 02 Dec 2015, Andy Thompson wrote:
> >>
> >> 				Since updating to RHEL 7.2 I've got issues with
> ns-slapd hanging
> >> the
> >> 				system up after a period of time.  The
> directory becomes
> >> unresponsive
> >> 				to searches or any connections.  After a
> restart I see
> >>
> >> 				[02/Dec/2015:15:27:41 -0500] - slapd started.
> >> Listening on All
> >> 				Interfaces port 389 for LDAP requests
> >> 				[02/Dec/2015:15:27:41 -0500] - Listening on
> All Interfaces port
> >> 636
> >> 				for LDAPS requests
> >> 				[02/Dec/2015:15:27:41 -0500] - Listening on
> >> 				/var/run/slapd-MHBENP-LIN.socket for
> LDAPI requests
> >> 				[02/Dec/2015:15:27:44 -0500]
> >> NSMMReplicationPlugin -
> >> 				agmt="cn=meTomdhixnpipa02.mhbenp.lin"
> >> (mdhixnpipa02:389):
> >>
> >> 		Replication
> >>
> >> 				bind with GSSAPI auth resumed
> >> 				[02/Dec/2015:15:27:47 -0500]
> >> NSMMReplicationPlugin - replication keep
> >> 				alive entry <cn=repl keep alive
> >> 4,dc=mhbenp,dc=lin> already exists
> >>
> >> 				In the logs and occasionally the keepalive
> entry message is seen
> >> a
> >> 				few times and then eventually the ns-slapd
> taps the system.  100%
> >> 				util, system load crawls up between 30 and 40
> and eventually I
> >> have
> >> 				to restart the service to get it to respond
> again.  Memory usage
> >> is
> >> 				normal, db and entry cache is sufficient..
> >> possibly a little on the
> >> 				high side but resource is sitting there asking
> to be used :)
> >>
> >> 				Running 389-ds-base-1.3.4.0-19.el7.x86_64
> after the update
> >> yesterday.
> >>
> >> 				What additional information can I provide?
> >>
> >> 			install debuginfo for 389-ds-base and slapi-nis, and
> take a pstack
> >> 			output for ns-slapd pid.
> >>
> >>
> >> 		For detailed instructions please see
> >>
> >> 	http://directory.fedoraproject.org/docs/389ds/FAQ/faq.html#debug
> >> _hangs
> >>
> >>
> >>
> >> 	Here is the resulting stacktrace during the last hang.
> >>
> >>
> >> The server is idle at this point.  None of the threads are doing any work, or
> >> are blocked/deadlocked.  It does not appear hung at all.
> >>
> >> When the server is in the "hung" state again, use ldapsearch (e.g. -s base -
> b
> >> "") to "ping" the server to see if it is entirely unresponsive.
> >>
> >>
> >>
> > No ldapsearch does not respond, it just hangs and doesn't ever return.
> 
> Try doing an strace of ldapsearch to see in what system call it is stuck
> in (or you can just do an strace/pstack on the running, hung ldapsearch
> process).
> 


I ended up opening a ticket with Redhat but wanted to pass along that Thierry tracked it down to this bug 

https://fedorahosted.org/freeipa/ticket/5464

I bumped up the nsslapd-threadnumber and things have settled down it appears, been running since Friday with no issues.

-andy