<html> <head> <meta content="text/html; charset=windows-1252" http-equiv="Content-Type"> </head> <body bgcolor="#FFFFFF" text="#000000"> <div class="moz-cite-prefix">On 07/22/2015 11:47 PM, Torsten Harenberg wrote: </div> <blockquote cite="mid:55B07FDC.5080803@physik.uni-wuppertal.de" type="cite"> <pre wrap="">Good morning, Am 22.07.15 um 19:25 schrieb Rich Megginson: </pre> <blockquote type="cite"> <pre wrap="">On 07/22/2015 11:03 AM, Torsten Harenberg wrote: </pre> <blockquote type="cite"> <pre wrap="">Dear Rich, Am 22.07.2015 um 17:03 schrieb Rich Megginson: </pre> <blockquote type="cite"> <pre wrap="">It might be helpful to do a # debuginfo-install 389-ds-base ipa-server slapi-nis and follow the directions at <a class="moz-txt-link-freetext" href="http://www.port389.org/docs/389ds/FAQ/faq.html#debugging-hangs">http://www.port389.org/docs/389ds/FAQ/faq.html#debugging-hangs</a> to get a full stack trace </pre> </blockquote> <pre wrap="">thanks for the hint. Did that. But assume I need to wait until it hangs again, right? </pre> </blockquote> <pre wrap=""> Right. </pre> </blockquote> <pre wrap=""> problem happend again, this time with slightly different symtoms: [root@wn108 ~]# id atlasprd020 id: atlasprd020: No such user after restarting dirserv: [root@wn108 ~]# id atlasprd020 uid=18970(atlasprd020) gid=1407(atlasprd) groups=1407(atlasprd) [root@wn108 ~]# So of course the whole site was down. the dirserv log file has not much: [22/Jul/2015:11:03:35 +0200] - Skipping CoS Definition cn=Password Policy,cn=accounts,dc=pleiades,dc=uni-wuppertal,dc=de--no CoS Templates found, which should be add ed before the CoS Definition. [22/Jul/2015:11:03:38 +0200] NSMMReplicationPlugin - replica_check_for_data_reload: Warning: disordely shutdown for replica dc=pleiades,dc=uni-wuppertal,dc=de. Check if DB RUV needs to be updated [22/Jul/2015:11:03:38 +0200] NSMMReplicationPlugin - Force update of database RUV (from CL RUV) -> 55af7af3000e00040000 [22/Jul/2015:11:03:39 +0200] set_krb5_creds - Could not get initial credentials for principal [<a class="moz-txt-link-abbreviated" href="mailto:ldap/ipa.pleiades.uni-wuppertal.de@PLEIADES.UNI-WUPPERTAL.DE">ldap/ipa.pleiades.uni-wuppertal.de@PLEIADES.UNI-WUPPERTAL.DE</a>] in keyta b [<a class="moz-txt-link-freetext" href="FILE:/etc/dirsrv/ds.keytab">FILE:/etc/dirsrv/ds.keytab</a>]: -1765328228 (Cannot contact any KDC for requested realm) [22/Jul/2015:11:03:39 +0200] - Skipping CoS Definition cn=Password Policy,cn=accounts,dc=pleiades,dc=uni-wuppertal,dc=de--no CoS Templates found, which should be add ed before the CoS Definition. [22/Jul/2015:11:03:39 +0200] slapd_ldap_sasl_interactive_bind - Error: could not perform interactive bind for id [] mech [GSSAPI]: LDAP error -2 (Local error) (SASL( -1): generic failure: GSSAPI Error: Unspecified GSS failure. Minor code may provide more information (No Kerberos credentials available)) errno 0 (Success) [22/Jul/2015:11:03:39 +0200] slapi_ldap_bind - Error: could not perform interactive bind for id [] authentication mechanism [GSSAPI]: error -2 (Local error) [22/Jul/2015:11:03:39 +0200] NSMMReplicationPlugin - agmt="cn=meToipa2.pleiades.uni-wuppertal.de" (ipa2:389): Replication bind with GSSAPI auth failed: LDAP error -2 (Local error) (SASL(-1): generic failure: GSSAPI Error: Unspecified GSS failure. Minor code may provide more information (No Kerberos credentials available)) [22/Jul/2015:11:03:39 +0200] - slapd started. Listening on All Interfaces port 389 for LDAP requests [22/Jul/2015:11:03:39 +0200] - Listening on All Interfaces port 636 for LDAPS requests [22/Jul/2015:11:03:39 +0200] - Listening on /var/run/slapd-PLEIADES-UNI-WUPPERTAL-DE.socket for LDAPI requests [22/Jul/2015:11:03:43 +0200] NSMMReplicationPlugin - agmt="cn=meToipa2.pleiades.uni-wuppertal.de" (ipa2:389): Replication bind with GSSAPI auth resumed [22/Jul/2015:15:15:41 +0200] find_sid_for_ldap_entry - [file ipa_sidgen_common.c, line 522]: Cannot convert Posix ID [51351] into an unused SID. [22/Jul/2015:15:15:41 +0200] ipa_sidgen_add_post_op - [file ipa_sidgen.c, line 149]: Cannot add SID to new entry. [23/Jul/2015:07:33:05 +0200] - slapd shutting down - signaling operation threads - op stack size 48 max work q size 22 max work q stack size 22 [23/Jul/2015:07:33:05 +0200] - slapd shutting down - waiting for 1 thread to terminate [23/Jul/2015:07:33:05 +0200] - slapd shutting down - closing down internal subsystems and plugins [23/Jul/2015:07:33:05 +0200] NSMMReplicationPlugin - agmt="cn=meToipa2.pleiades.uni-wuppertal.de" (ipa2:389): Warning: Attempting to release replica, but unable to r eceive endReplication extended operation response from the replica. Error -5 (Timed out) [23/Jul/2015:07:33:06 +0200] nis-plugin - error sending request to portmap or rpcbind on 9: Connection refused [23/Jul/2015:07:33:06 +0200] nis-plugin - error sending request to portmap or rpcbind on 9: Connection refused [23/Jul/2015:07:33:06 +0200] nis-plugin - timeout registering with portmap service [23/Jul/2015:07:33:06 +0200] nis-plugin - error sending request to portmap or rpcbind on 9: Connection refused [23/Jul/2015:07:33:06 +0200] nis-plugin - error sending request to portmap or rpcbind on 9: Connection refused [23/Jul/2015:07:33:06 +0200] nis-plugin - error sending request to portmap or rpcbind on 9: Connection refused [23/Jul/2015:07:33:06 +0200] nis-plugin - timeout registering with portmap service [23/Jul/2015:07:33:06 +0200] nis-plugin - error sending request to portmap or rpcbind on 9: Connection refused [23/Jul/2015:07:33:06 +0200] nis-plugin - error sending request to portmap or rpcbind on 9: Connection refused [23/Jul/2015:07:33:06 +0200] nis-plugin - timeout registering with portmap service [23/Jul/2015:07:33:06 +0200] nis-plugin - error sending request to portmap or rpcbind on 9: Connection refused [23/Jul/2015:07:33:06 +0200] nis-plugin - error sending request to portmap or rpcbind on 9: Connection refused [23/Jul/2015:07:33:06 +0200] nis-plugin - error sending request to portmap or rpcbind on 9: Connection refused [23/Jul/2015:07:33:06 +0200] nis-plugin - timeout registering with portmap service [23/Jul/2015:07:33:06 +0200] nis-plugin - error sending request to portmap or rpcbind on 9: Connection refused [23/Jul/2015:07:33:06 +0200] nis-plugin - error sending request to portmap or rpcbind on 9: Connection refused [23/Jul/2015:07:33:06 +0200] nis-plugin - timeout registering with portmap service [23/Jul/2015:07:33:06 +0200] nis-plugin - error sending request to portmap or rpcbind on 9: Connection refused [23/Jul/2015:07:33:06 +0200] nis-plugin - error sending request to portmap or rpcbind on 9: Connection refused [23/Jul/2015:07:33:06 +0200] nis-plugin - error sending request to portmap or rpcbind on 9: Connection refused [23/Jul/2015:07:33:06 +0200] nis-plugin - timeout registering with portmap service [23/Jul/2015:07:33:06 +0200] - Waiting for 4 database threads to stop [23/Jul/2015:07:33:06 +0200] - All database threads now stopped [23/Jul/2015:07:33:06 +0200] - slapd shutting down - freed 22 work q stack objects - freed 49 op stack objects [23/Jul/2015:07:33:06 +0200] - slapd stopped. [23/Jul/2015:07:33:08 +0200] nis-plugin - error connecting rpcbind client socket to the service [23/Jul/2015:07:33:08 +0200] SSL Initialization - Configured SSL version range: min: TLS1.0, max: TLS1.2 [23/Jul/2015:07:33:08 +0200] - SSL alert: Configured NSS Ciphers [...] [23/Jul/2015:07:33:12 +0200] set_krb5_creds - Could not get initial credentials for principal [ldap/ipa@] in keytab [<a class="moz-txt-link-freetext" href="FILE:/etc/dirsrv/ds.keytab">FILE:/etc/dirsrv/ds.keytab</a>]: -1765328203 (Key table entry not found) [23/Jul/2015:07:33:12 +0200] slapd_ldap_sasl_interactive_bind - Error: could not perform interactive bind for id [] mech [GSSAPI]: LDAP error -2 (Local error) (SASL(-1): generic failure: GSSAPI Error: Unspecified GSS failure. Minor code may provide more information (No Kerberos credentials available)) errno 0 (Success) [23/Jul/2015:07:33:12 +0200] slapi_ldap_bind - Error: could not perform interactive bind for id [] authentication mechanism [GSSAPI]: error -2 (Local error) [23/Jul/2015:07:33:12 +0200] NSMMReplicationPlugin - agmt="cn=meToipa2.pleiades.uni-wuppertal.de" (ipa2:389): Replication bind with GSSAPI auth failed: LDAP error -2 (Local error) (SASL(-1): generic failure: GSSAPI Error: Unspecified GSS failure. Minor code may provide more information (No Kerberos credentials available)) [23/Jul/2015:07:33:12 +0200] - slapd started. Listening on All Interfaces port 389 for LDAP requests [23/Jul/2015:07:33:12 +0200] - Listening on All Interfaces port 636 for LDAPS requests [23/Jul/2015:07:33:12 +0200] - Listening on /var/run/slapd-PLEIADES-UNI-WUPPERTAL-DE.socket for LDAPI requests [23/Jul/2015:07:33:15 +0200] set_krb5_creds - Could not get initial credentials for principal [ldap/ipa@] in keytab [<a class="moz-txt-link-freetext" href="FILE:/etc/dirsrv/ds.keytab">FILE:/etc/dirsrv/ds.keytab</a>]: -1765328203 (Key table entry not found) [23/Jul/2015:07:33:15 +0200] slapd_ldap_sasl_interactive_bind - Error: could not perform interactive bind for id [] mech [GSSAPI]: LDAP error -2 (Local error) (SASL(-1): generic failure: GSSAPI Error: Unspecified GSS failure. Minor code may provide more information (No Kerberos credentials available)) errno 0 (Success) [23/Jul/2015:07:33:15 +0200] slapi_ldap_bind - Error: could not perform interactive bind for id [] authentication mechanism [GSSAPI]: error -2 (Local error) [23/Jul/2015:07:33:21 +0200] set_krb5_creds - Could not get initial credentials for principal [ldap/ipa@] in keytab [<a class="moz-txt-link-freetext" href="FILE:/etc/dirsrv/ds.keytab">FILE:/etc/dirsrv/ds.keytab</a>]: -1765328203 (Key table entry not found) [23/Jul/2015:07:33:21 +0200] slapd_ldap_sasl_interactive_bind - Error: could not perform interactive bind for id [] mech [GSSAPI]: LDAP error -2 (Local error) (SASL(-1): generic failure: GSSAPI Error: Unspecified GSS failure. Minor code may provide more information (No Kerberos credentials available)) errno 0 (Success) [23/Jul/2015:07:33:21 +0200] slapi_ldap_bind - Error: could not perform interactive bind for id [] authentication mechanism [GSSAPI]: error -2 (Local error) [23/Jul/2015:07:33:33 +0200] set_krb5_creds - Could not get initial credentials for principal [ldap/ipa@] in keytab [<a class="moz-txt-link-freetext" href="FILE:/etc/dirsrv/ds.keytab">FILE:/etc/dirsrv/ds.keytab</a>]: -1765328203 (Key table entry not found) [23/Jul/2015:07:33:33 +0200] slapd_ldap_sasl_interactive_bind - Error: could not perform interactive bind for id [] mech [GSSAPI]: LDAP error -2 (Local error) (SASL(-1): generic failure: GSSAPI Error: Unspecified GSS failure. Minor code may provide more information (No Kerberos credentials available)) errno 0 (Success) [23/Jul/2015:07:33:33 +0200] slapi_ldap_bind - Error: could not perform interactive bind for id [] authentication mechanism [GSSAPI]: error -2 (Local error) [23/Jul/2015:07:33:57 +0200] set_krb5_creds - Could not get initial credentials for principal [ldap/ipa@] in keytab [<a class="moz-txt-link-freetext" href="FILE:/etc/dirsrv/ds.keytab">FILE:/etc/dirsrv/ds.keytab</a>]: -1765328203 (Key table entry not found) [23/Jul/2015:07:33:57 +0200] slapd_ldap_sasl_interactive_bind - Error: could not perform interactive bind for id [] mech [GSSAPI]: LDAP error -2 (Local error) (SASL(-1): generic failure: GSSAPI Error: Unspecified GSS failure. Minor code may provide more information (No Kerberos credentials available)) errno 0 (Success) [23/Jul/2015:07:33:57 +0200] slapi_ldap_bind - Error: could not perform interactive bind for id [] authentication mechanism [GSSAPI]: error -2 (Local error) [23/Jul/2015:07:34:45 +0200] set_krb5_creds - Could not get initial credentials for principal [ldap/ipa@] in keytab [<a class="moz-txt-link-freetext" href="FILE:/etc/dirsrv/ds.keytab">FILE:/etc/dirsrv/ds.keytab</a>]: -1765328203 (Key table entry not found) [23/Jul/2015:07:34:45 +0200] slapd_ldap_sasl_interactive_bind - Error: could not perform interactive bind for id [] mech [GSSAPI]: LDAP error -2 (Local error) (SASL(-1): generic failure: GSSAPI Error: Unspecified GSS failure. Minor code may provide more information (No Kerberos credentials available)) errno 0 (Success) [23/Jul/2015:07:34:45 +0200] slapi_ldap_bind - Error: could not perform interactive bind for id [] authentication mechanism [GSSAPI]: error -2 (Local error) attached is the strace of slapd as instructed earlier.</pre> </blockquote> Thanks.� This was taken during slapd unresponsive?� The server does not appear to be under a load at all.� It is almost completely idle except for a thread sending replication updates. I think we need a better test for slapd being unresponsive, so we can narrow the problem down to slapd or something else.� When the directory server appears to be hung, try this: # ldapsearch -xLLL -D "cn=directory manager" -W -s base -b "dc=uni-wuppertal,dc=de" This search should return immediately.� If it hangs, then the problem is in slapd, and get a stack trace as before. <blockquote cite="mid:55B07FDC.5080803@physik.uni-wuppertal.de" type="cite"> <pre wrap=""> I am really a bit lost. Even though we have deployed two IPA servers, the whole site is down about twice a day with authentication problems. Hope anybody can help. Best regards, Torsten </pre> <fieldset class="mimeAttachmentHeader"></fieldset> </blockquote> </body> </html>