<html><body> Has anyone seen these before? First Master IPA DNS logs show: Looks like the host names are getting the domain twice domain.local.domain.local client 10.x.x.x#58094: query failed (SERVFAIL) for server1.domain.local.domain.local/IN/AAAA at query.c:6569 timeout in ldap_pool_getconnection(): try to raise 'connections' parameter; potential deadlock? client 10.x.x.x#44147: query failed (SERVFAIL) for x.x.x.10.in-addr.arpa/IN/PTR at query.c:6569 timeout in ldap_pool_getconnection(): try to raise 'connections' parameter; potential deadlock? client 10.x.x.x#56466: query failed (SERVFAIL) for x.x.x.10.in-addr.arpa/IN/PTR at query.c:6569 timeout in ldap_pool_getconnection(): try to raise 'connections' parameter; potential deadlock? client 10.x.x.x53367: query failed (SERVFAIL) for server2.domain.local.domain.local/IN/A at query.c:6569 timeout in ldap_pool_getconnection(): try to raise 'connections' parameter; potential deadlock? client 10.x.x.x#53367: query failed (SERVFAIL) for server2.domain.local.domain.local/IN/AAAA at query.c:6569 So enrolls are failing at this point when tyring to enroll to a replica: [bob@server1 log]# ipa-client-install –enable-dns-updates Discovery was successful! Hostname: server1.watson.local Realm: DOMAIN.LOCAL DNS Domain: domain.local IPA Server: ipareplica.domain.local BaseDN: dc=domain,dc=local Continue to configure the system with these values? [no]: yes User authorized to enroll computers: bob Synchronizing time with KDC... Password for bob@DOMAIN.LOCAL: Successfully retrieved CA cert Subject: CN=Certificate Authority,O=DOMAIN.LOCAL Issuer: CN=Certificate Authority,O=DOMAIN.LOCAL Valid From: Tue Jan 06 19:37:09 2015 UTC Valid Until: Sat Jan 06 19:37:09 2035 UTC Enrolled in IPA realm DOMAIN.LOCAL Attempting to get host TGT... Created /etc/ipa/default.conf New SSSD config will be created Configured sudoers in /etc/nsswitch.conf Configured /etc/sssd/sssd.conf Configured /etc/krb5.conf for IPA realm DOMAIN.LOCAL trying <a href="https://rtpvxl0069.watson.local/ipa/xml">https://ipareplica.domain.local/ipa/xml</a> Cannot connect to the server due to Kerberos error: Kerberos error: Kerberos error: ('Unspecified GSS failure. Minor code may provide more information', 851968)/('KDC returned error string: PROCESS_TGS', -1765328324)/. Trying with delegate=True trying <a href="https://rtpvxl0069.watson.local/ipa/xml">https://ipareplica.domain.local/ipa/xml</a> Second connect with delegate=True also failed: Kerberos error: Kerberos error: ('Unspecified GSS failure. Minor code may provide more information', 851968)/('KDC returned error string: PROCESS_TGS', -1765328324)/ Cannot connect to the IPA server XML-RPC interface: Kerberos error: Kerberos error: ('Unspecified GSS failure. Minor code may provide more information', 851968)/('KDC returned error string: PROCESS_TGS', -1765328324)/ Installation failed. Rolling back changes. Unenrolling client from IPA server Unenrolling host failed: Error obtaining initial credentials: Generic error (see e-text). Removing Kerberos service principals from /etc/krb5.keytab Disabling client Kerberos and LDAP configurations Redundant SSSD configuration file /etc/sssd/sssd.conf was moved to /etc/sssd/sssd.conf.deleted Restoring client configuration files nscd daemon is not installed, skip configuration nslcd daemon is not installed, skip configuration Client uninstall complete. Sean Hogan <img width="16" height="16" src="cid:2__=88BBF54ADFF4633B8f9e8a93df938690918c88B@" border="0" alt="Inactive hide details for Sean Hogan---06/20/2016 12:49:27 PM---Also seeing this in the upgrade log on the first master but not">Sean Hogan---06/20/2016 12:49:27 PM---Also seeing this in the upgrade log on the first master but not on the 7 ipas. ERROR Failed to resta From: Sean Hogan/Durham/IBM To: Sean Hogan/Durham/IBM@IBMUS Cc: freeipa-users <freeipa-users@redhat.com> Date: 06/20/2016 12:49 PM Subject: Re: [Freeipa-users] IPA 3.0.47 to 3.0.50 Upgrade problem <hr width="100%" size="2" align="left" noshade style="color:#8091A5; "> Also seeing this in the upgrade log on the first master but not on the 7 ipas. ERROR Failed to restart named: Command '/sbin/service named restart ' returned non-zero exit status 7 which led me to <a href="https://bugzilla.redhat.com/show_bug.cgi?id=895298">https://bugzilla.redhat.com/show_bug.cgi?id=895298</a> Sean Hogan <img width="16" height="16" src="cid:2__=88BBF54ADFF4633B8f9e8a93df938690918c88B@" border="0" alt="Inactive hide details for Sean Hogan---06/20/2016 11:46:54 AM---Hi All.. I thought we fixed this issue by rebooting the KVM h">Sean Hogan---06/20/2016 11:46:54 AM---Hi All.. I thought we fixed this issue by rebooting the KVM host but it is showing From: Sean Hogan/Durham/IBM@IBMUS To: freeipa-users <freeipa-users@redhat.com> Date: 06/20/2016 11:46 AM Subject: Re: [Freeipa-users] IPA 3.0.47 to 3.0.50 Upgrade problem Sent by: freeipa-users-bounces@redhat.com <hr width="100%" size="2" align="left" noshade style="color:#8091A5; "> Hi All.. I thought we fixed this issue by rebooting the KVM host but it is showing again. Our First Master IPA is being rebooted 2 -5 times a day now just to keep it alive. What we are seeing: God@FirstMaster log]# kinit admin kinit: Cannot contact any KDC for realm 'Domain.LOCAL' while getting initial credentials DNS is not working as nslookup is failing to a replica.... think once we lose DNS it all goes down hill which makes sense. [god@FirstMaster log]# ipactl stop -----> Just hangs forever.. no replies.. no error.. nothing I try service named stop and nothing happens I have the box hard shutdown from KVM console. Reboot it and it works for a little while but eventually back to same behavior. At this point I can service named stop and it responds... ipactl status and it responds.. but when if I try service named restart I get [god@FirstMaster log]# service named stop Stopping named: ...... [god@Firstmaster log]# service named start Starting named: [FAILED] [god@FirstMaster log]# service named status rndc: connect failed: 127.0.0.1#953: connection refused named dead but pid file exists Rebooted box and it is hung on shutting down domain-local and never fully shuts down.. have to get it hard shutdown again. During an attempt to gracefully shut down we see this Shutting Down dirsrv: PKI-IPA OK DOMAIN-LOCAL FAILED *** Error: 1 instance(s) unsuccessfully stopped FAILED Then it moves on to shut other things down and returns to dirsrv Shutting Down dirsrv: PKI-IPA....server already stopped FAILED {Makes sense.. it died earlier} DOMAIN-LOCAL... {this sits here til we hard shutdown} bind-libs-9.8.2-0.47.rc1.el6.x86_64 bind-9.8.2-0.47.rc1.el6.x86_64 bind-utils-9.8.2-0.47.rc1.el6.x86_64 ipa-client-3.0.0-50.el6.1.x86_64 ipa-server-selinux-3.0.0-50.el6.1.x86_64 ipa-server-3.0.0-50.el6.1.x86_64 sssd-ipa-1.13.3-22.el6.x86_64 /var/log/dirsrv/slapd-DOMAIN-LOCAL [20/Jun/2016:13:29:06 -0400] - 389-Directory/1.2.11.15 B2016.063.2110 starting up [20/Jun/2016:13:29:06 -0400] schema-compat-plugin - warning: no entries set up under cn=computers, cn=compat,dc=domain,dc=local [20/Jun/2016:13:29:07 -0400] NSMMReplicationPlugin - ruv_compare_ruv: RUV [database RUV] does not contain element [{replica 7} 55ca26a0000900070000 5688d8e6001000070000] which is present in RUV [changelog max RUV] [20/Jun/2016:13:29:07 -0400] NSMMReplicationPlugin - replica_check_for_data_reload: Warning: for replica dc=domain,dc=local there were some differences between the changelog max RUV and the database RUV. If there are obsolete elements in the database RUV, you should remove them using the CLEANALLRUV task. If they are not obsolete, you should check their status to see why there are no changes from those servers in the changelog. [20/Jun/2016:13:29:07 -0400] set_krb5_creds - Could not get initial credentials for principal [ldap/server1.domain.local@DOMAIN.LOCAL] in keytab [FILE:/etc/dirsrv/ds.keytab]: -1765328228 (Cannot contact any KDC for requested realm) [20/Jun/2016:13:29:07 -0400] set_krb5_creds - Could not get initial credentials for principal [ldap/server1.domain.local@DOMAIN.LOCAL] in keytab [FILE:/etc/dirsrv/ds.keytab]: -1765328228 (Cannot contact any KDC for requested realm) [20/Jun/2016:13:29:07 -0400] set_krb5_creds - Could not get initial credentials for principal [ldap/server1.domain.local@DOMAIN.LOCAL] in keytab [FILE:/etc/dirsrv/ds.keytab]: -1765328228 (Cannot contact any KDC for requested realm) [20/Jun/2016:13:29:07 -0400] slapd_ldap_sasl_interactive_bind - Error: could not perform interactive bind for id [] mech [GSSAPI]: LDAP error -2 (Local error) (SASL(-1): generic failure: GSSAPI Error: Unspecified GSS failure. Minor code may provide more information (Credentials cache file '/tmp/krb5cc_495' not found)) errno 2 (No such file or directory) [20/Jun/2016:13:29:07 -0400] slapi_ldap_bind - Error: could not perform interactive bind for id [] mech [GSSAPI]: error -2 (Local error) [20/Jun/2016:13:29:07 -0400] NSMMReplicationPlugin - agmt="cn=meToserver8.domain.local" (server8:389): Replication bind with GSSAPI auth failed: LDAP error -2 (Local error) (SASL(-1): generic failure: GSSAPI Error: Unspecified GSS failure. Minor code may provide more information (Credentials cache file '/tmp/krb5cc_495' not found)) [20/Jun/2016:13:29:07 -0400] set_krb5_creds - Could not get initial credentials for principal [ldap/server1.domain.local@DOMAIN.LOCAL] in keytab [FILE:/etc/dirsrv/ds.keytab]: -1765328228 (Cannot contact any KDC for requested realm) [20/Jun/2016:13:29:07 -0400] slapd_ldap_sasl_interactive_bind - Error: could not perform interactive bind for id [] mech [GSSAPI]: LDAP error -2 (Local error) (SASL(-1): generic failure: GSSAPI Error: Unspecified GSS failure. Minor code may provide more information (Credentials cache file '/tmp/krb5cc_495' not found)) errno 2 (No such file or directory) [20/Jun/2016:13:29:07 -0400] slapi_ldap_bind - Error: could not perform interactive bind for id [] mech [GSSAPI]: error -2 (Local error) [20/Jun/2016:13:29:07 -0400] NSMMReplicationPlugin - agmt="cn=meToserver9.domain.local" (server9:389): Replication bind with GSSAPI auth failed: LDAP error -2 (Local error) (SASL(-1): generic failure: GSSAPI Error: Unspecified GSS failure. Minor code may provide more information (Credentials cache file '/tmp/krb5cc_495' not found)) [20/Jun/2016:13:29:07 -0400] set_krb5_creds - Could not get initial credentials for principal [ldap/server1.domain.local@DOMAIN.LOCAL] in keytab [FILE:/etc/dirsrv/ds.keytab]: -1765328228 (Cannot contact any KDC for requested realm) [20/Jun/2016:13:29:07 -0400] slapd_ldap_sasl_interactive_bind - Error: could not perform interactive bind for id [] mech [GSSAPI]: LDAP error -2 (Local error) (SASL(-1): generic failure: GSSAPI Error: Unspecified GSS failure. Minor code may provide more information (Credentials cache file '/tmp/krb5cc_495' not found)) errno 2 (No such file or directory) [20/Jun/2016:13:29:07 -0400] slapi_ldap_bind - Error: could not perform interactive bind for id [] mech [GSSAPI]: error -2 (Local error) [20/Jun/2016:13:29:07 -0400] NSMMReplicationPlugin - agmt="cn=meToserver3.domain.local" (server3:389): Replication bind with GSSAPI auth failed: LDAP error -2 (Local error) (SASL(-1): generic failure: GSSAPI Error: Unspecified GSS failure. Minor code may provide more information (Credentials cache file '/tmp/krb5cc_495' not found)) [20/Jun/2016:13:29:07 -0400] set_krb5_creds - Could not get initial credentials for principal [ldap/server1.domain.local@DOMAIN.LOCAL] in keytab [FILE:/etc/dirsrv/ds.keytab]: -1765328228 (Cannot contact any KDC for requested realm) [20/Jun/2016:13:29:07 -0400] - slapd started. Listening on All Interfaces port 389 for LDAP requests [20/Jun/2016:13:29:07 -0400] - Listening on All Interfaces port 636 for LDAPS requests [20/Jun/2016:13:29:07 -0400] - Listening on /var/run/slapd-DOMAIN-LOCAL.socket for LDAPI requests [20/Jun/2016:13:29:07 -0400] set_krb5_creds - Could not get initial credentials for principal [ldap/server1.domain.local@DOMAIN.LOCAL] in keytab [FILE:/etc/dirsrv/ds.keytab]: -1765328228 (Cannot contact any KDC for requested realm) [20/Jun/2016:13:29:07 -0400] slapd_ldap_sasl_interactive_bind - Error: could not perform interactive bind for id [] mech [GSSAPI]: LDAP error -2 (Local error) (SASL(-1): generic failure: GSSAPI Error: Unspecified GSS failure. Minor code may provide more information (Credentials cache file '/tmp/krb5cc_495' not found)) errno 2 (No such file or directory) [20/Jun/2016:13:29:07 -0400] slapi_ldap_bind - Error: could not perform interactive bind for id [] mech [GSSAPI]: error -2 (Local error) [20/Jun/2016:13:29:07 -0400] NSMMReplicationPlugin - agmt="cn=meToserver4.domain.local" (server4:389): Replication bind with GSSAPI auth failed: LDAP error -2 (Local error) (SASL(-1): generic failure: GSSAPI Error: Unspecified GSS failure. Minor code may provide more information (Credentials cache file '/tmp/krb5cc_495' not found)) [20/Jun/2016:13:29:07 -0400] slapd_ldap_sasl_interactive_bind - Error: could not perform interactive bind for id [] mech [GSSAPI]: LDAP error -2 (Local error) (SASL(-1): generic failure: GSSAPI Error: Unspecified GSS failure. Minor code may provide more information (Credentials cache file '/tmp/krb5cc_495' not found)) errno 2 (No such file or directory) [20/Jun/2016:13:29:07 -0400] slapi_ldap_bind - Error: could not perform interactive bind for id [] mech [GSSAPI]: error -2 (Local error) [20/Jun/2016:13:29:07 -0400] NSMMReplicationPlugin - agmt="cn=meTo1server.domain.local" (1server:389): Replication bind with GSSAPI auth failed: LDAP error -2 (Local error) (SASL(-1): generic failure: GSSAPI Error: Unspecified GSS failure. Minor code may provide more information (Credentials cache file '/tmp/krb5cc_495' not found)) [20/Jun/2016:13:29:07 -0400] slapd_ldap_sasl_interactive_bind - Error: could not perform interactive bind for id [] mech [GSSAPI]: LDAP error -2 (Local error) (SASL(-1): generic failure: GSSAPI Error: Unspecified GSS failure. Minor code may provide more information (Credentials cache file '/tmp/krb5cc_495' not found)) errno 2 (No such file or directory) [20/Jun/2016:13:29:07 -0400] slapi_ldap_bind - Error: could not perform interactive bind for id [] mech [GSSAPI]: error -2 (Local error) [20/Jun/2016:13:29:07 -0400] NSMMReplicationPlugin - agmt="cn=meToserver7.domain.local" (server7:389): Replication bind with GSSAPI auth failed: LDAP error -2 (Local error) (SASL(-1): generic failure: GSSAPI Error: Unspecified GSS failure. Minor code may provide more information (Credentials cache file '/tmp/krb5cc_495' not found)) [20/Jun/2016:13:29:07 -0400] slapd_ldap_sasl_interactive_bind - Error: could not perform interactive bind for id [] mech [GSSAPI]: LDAP error -2 (Local error) (SASL(-1): generic failure: GSSAPI Error: Unspecified GSS failure. Minor code may provide more information (Credentials cache file '/tmp/krb5cc_495' not found)) errno 0 (Success) [20/Jun/2016:13:29:07 -0400] slapi_ldap_bind - Error: could not perform interactive bind for id [] mech [GSSAPI]: error -2 (Local error) [20/Jun/2016:13:29:07 -0400] NSMMReplicationPlugin - agmt="cn=meToserver2.domain.local" (server2:389): Replication bind with GSSAPI auth failed: LDAP error -2 (Local error) (SASL(-1): generic failure: GSSAPI Error: Unspecified GSS failure. Minor code may provide more information (Credentials cache file '/tmp/krb5cc_495' not found)) [20/Jun/2016:13:29:10 -0400] NSMMReplicationPlugin - agmt="cn=meToserver9.domain.local" (server9:389): Replication bind with GSSAPI auth resumed [20/Jun/2016:13:29:10 -0400] slapd_ldap_sasl_interactive_bind - Error: could not perform interactive bind for id [] mech [GSSAPI]: LDAP error 49 (Invalid credentials) (SASL(-13): authentication failure: GSSAPI Failure: gss_accept_sec_context) errno 0 (Success) [20/Jun/2016:13:29:10 -0400] slapi_ldap_bind - Error: could not perform interactive bind for id [] mech [GSSAPI]: error 49 (Invalid credentials) [20/Jun/2016:13:29:10 -0400] NSMMReplicationPlugin - agmt="cn=meToserver3.domain.local" (server3:389): Replication bind with GSSAPI auth failed: LDAP error 49 (Invalid credentials) (SASL(-13): authentication failure: GSSAPI Failure: gss_accept_sec_context) [20/Jun/2016:13:29:10 -0400] NSMMReplicationPlugin - agmt="cn=meToserver2.domain.local" (server2:389): Replication bind with GSSAPI auth resumed [20/Jun/2016:13:29:10 -0400] slapi_ldap_bind - Error: could not perform interactive bind for id [] mech [GSSAPI]: error -2 (Local error) [20/Jun/2016:13:29:10 -0400] NSMMReplicationPlugin - agmt="cn=meToserver8.domain.local" (server8:389): Replication bind with GSSAPI auth resumed [20/Jun/2016:13:29:10 -0400] slapd_ldap_sasl_interactive_bind - Error: could not perform interactive bind for id [] mech [GSSAPI]: LDAP error -2 (Local error) (SASL(-1): generic failure: GSSAPI Error: Unspecified GSS failure. Minor code may provide more information (No credentials cache found)) errno 2 (No such file or directory) [20/Jun/2016:13:29:10 -0400] slapi_ldap_bind - Error: could not perform interactive bind for id [] mech [GSSAPI]: error -2 (Local error) [20/Jun/2016:13:29:10 -0400] slapd_ldap_sasl_interactive_bind - Error: could not perform interactive bind for id [] mech [GSSAPI]: LDAP error -2 (Local error) (SASL(-1): generic failure: GSSAPI Error: Unspecified GSS failure. Minor code may provide more information (No credentials cache found)) errno 2 (No such file or directory) [20/Jun/2016:13:29:10 -0400] slapi_ldap_bind - Error: could not perform interactive bind for id [] mech [GSSAPI]: error -2 (Local error) [20/Jun/2016:13:29:16 -0400] slapd_ldap_sasl_interactive_bind - Error: could not perform interactive bind for id [] mech [GSSAPI]: LDAP error 49 (Invalid credentials) (SASL(-13): authentication failure: GSSAPI Failure: gss_accept_sec_context) errno 0 (Success) [20/Jun/2016:13:29:16 -0400] slapi_ldap_bind - Error: could not perform interactive bind for id [] mech [GSSAPI]: error 49 (Invalid credentials) [20/Jun/2016:13:59:00 -0400] - 389-Directory/1.2.11.15 B2016.063.2110 starting up [20/Jun/2016:13:59:00 -0400] - Detected Disorderly Shutdown last time Directory Server was running, recovering database. [20/Jun/2016:13:59:01 -0400] schema-compat-plugin - warning: no entries set up under cn=computers, cn=compat,dc=domain,dc=local [20/Jun/2016:13:59:48 -0400] NSMMReplicationPlugin - ruv_compare_ruv: RUV [database RUV] does not contain element [{replica 7} 55ca26a0000900070000 5688d8e6001000070000] which is present in RUV [changelog max RUV] [20/Jun/2016:13:59:48 -0400] NSMMReplicationPlugin - replica_check_for_data_reload: Warning: for replica dc=domain,dc=local there were some differences between the changelog max RUV and the database RUV. If there are obsolete elements in the database RUV, you should remove them using the CLEANALLRUV task. If they are not obsolete, you should check their status to see why there are no changes from those servers in the changelog. [20/Jun/2016:13:59:48 -0400] set_krb5_creds - Could not get initial credentials for principal [ldap/server1.domain.local@DOMAIN.LOCAL] in keytab [FILE:/etc/dirsrv/ds.keytab]: -1765328228 (Cannot contact any KDC for requested realm) [20/Jun/2016:13:59:48 -0400] set_krb5_creds - Could not get initial credentials for principal [ldap/server1.domain.local@DOMAIN.LOCAL] in keytab [FILE:/etc/dirsrv/ds.keytab]: -1765328228 (Cannot contact any KDC for requested realm) [20/Jun/2016:13:59:48 -0400] set_krb5_creds - Could not get initial credentials for principal [ldap/server1.domain.local@DOMAIN.LOCAL] in keytab [FILE:/etc/dirsrv/ds.keytab]: -1765328228 (Cannot contact any KDC for requested realm) [20/Jun/2016:13:59:48 -0400] slapd_ldap_sasl_interactive_bind - Error: could not perform interactive bind for id [] mech [GSSAPI]: LDAP error -2 (Local error) (SASL(-1): generic failure: GSSAPI Error: Unspecified GSS failure. Minor code may provide more information (Credentials cache file '/tmp/krb5cc_495' not found)) errno 2 (No such file or directory) [20/Jun/2016:13:59:48 -0400] slapi_ldap_bind - Error: could not perform interactive bind for id [] mech [GSSAPI]: error -2 (Local error) [20/Jun/2016:13:59:48 -0400] slapd_ldap_sasl_interactive_bind - Error: could not perform interactive bind for id [] mech [GSSAPI]: LDAP error -2 (Local error) (SASL(-1): generic failure: GSSAPI Error: Unspecified GSS failure. Minor code may provide more information (Credentials cache file '/tmp/krb5cc_495' not found)) errno 2 (No such file or directory) [20/Jun/2016:13:59:48 -0400] slapi_ldap_bind - Error: could not perform interactive bind for id [] mech [GSSAPI]: error -2 (Local error) [20/Jun/2016:13:59:48 -0400] NSMMReplicationPlugin - agmt="cn=meToserver9.domain.local" (server9:389): Replication bind with GSSAPI auth failed: LDAP error -2 (Local error) (SASL(-1): generic failure: GSSAPI Error: Unspecified GSS failure. Minor code may provide more information (Credentials cache file '/tmp/krb5cc_495' not found)) [20/Jun/2016:13:59:48 -0400] set_krb5_creds - Could not get initial credentials for principal [ldap/server1.domain.local@DOMAIN.LOCAL] in keytab [FILE:/etc/dirsrv/ds.keytab]: -1765328228 (Cannot contact any KDC for requested realm) [20/Jun/2016:13:59:48 -0400] NSMMReplicationPlugin - agmt="cn=meToserver8.domain.local" (server8:389): Replication bind with GSSAPI auth failed: LDAP error -2 (Local error) (SASL(-1): generic failure: GSSAPI Error: Unspecified GSS failure. Minor code may provide more information (Credentials cache file '/tmp/krb5cc_495' not found)) [20/Jun/2016:13:59:48 -0400] slapd_ldap_sasl_interactive_bind - Error: could not perform interactive bind for id [] mech [GSSAPI]: LDAP error -2 (Local error) (SASL(-1): generic failure: GSSAPI Error: Unspecified GSS failure. Minor code may provide more information (Credentials cache file '/tmp/krb5cc_495' not found)) errno 2 (No such file or directory) [20/Jun/2016:13:59:48 -0400] slapi_ldap_bind - Error: could not perform interactive bind for id [] mech [GSSAPI]: error -2 (Local error) [20/Jun/2016:13:59:48 -0400] NSMMReplicationPlugin - agmt="cn=meTobldvxl0011.domain.local" (1server:389): Replication bind with GSSAPI auth failed: LDAP error -2 (Local error) (SASL(-1): generic failure: GSSAPI Error: Unspecified GSS failure. Minor code may provide more information (Credentials cache file '/tmp/krb5cc_495' not found)) [20/Jun/2016:13:59:48 -0400] slapd_ldap_sasl_interactive_bind - Error: could not perform interactive bind for id [] mech [GSSAPI]: LDAP error -2 (Local error) (SASL(-1): generic failure: GSSAPI Error: Unspecified GSS failure. Minor code may provide more information (Credentials cache file '/tmp/krb5cc_495' not found)) errno 2 (No such file or directory) [20/Jun/2016:13:59:48 -0400] slapi_ldap_bind - Error: could not perform interactive bind for id [] mech [GSSAPI]: error -2 (Local error) [20/Jun/2016:13:59:48 -0400] NSMMReplicationPlugin - agmt="cn=meToserver3.domain.local" (server3:389): Replication bind with GSSAPI auth failed: LDAP error -2 (Local error) (SASL(-1): generic failure: GSSAPI Error: Unspecified GSS failure. Minor code may provide more information (Credentials cache file '/tmp/krb5cc_495' not found)) [20/Jun/2016:13:59:48 -0400] set_krb5_creds - Could not get initial credentials for principal [ldap/server1.domain.local@DOMAIN.LOCAL] in keytab [FILE:/etc/dirsrv/ds.keytab]: -1765328228 (Cannot contact any KDC for requested realm) [20/Jun/2016:13:59:48 -0400] set_krb5_creds - Could not get initial credentials for principal [ldap/server1.domain.local@DOMAIN.LOCAL] in keytab [FILE:/etc/dirsrv/ds.keytab]: -1765328228 (Cannot contact any KDC for requested realm) [20/Jun/2016:13:59:48 -0400] slapd_ldap_sasl_interactive_bind - Error: could not perform interactive bind for id [] mech [GSSAPI]: LDAP error -2 (Local error) (SASL(-1): generic failure: GSSAPI Error: Unspecified GSS failure. Minor code may provide more information (Credentials cache file '/tmp/krb5cc_495' not found)) errno 2 (No such file or directory) [20/Jun/2016:13:59:48 -0400] slapi_ldap_bind - Error: could not perform interactive bind for id [] mech [GSSAPI]: error -2 (Local error) [20/Jun/2016:13:59:48 -0400] NSMMReplicationPlugin - agmt="cn=meToserver7.domain.local" (server7:389): Replication bind with GSSAPI auth failed: LDAP error -2 (Local error) (SASL(-1): generic failure: GSSAPI Error: Unspecified GSS failure. Minor code may provide more information (Credentials cache file '/tmp/krb5cc_495' not found)) [20/Jun/2016:13:59:48 -0400] - slapd started. Listening on All Interfaces port 389 for LDAP requests [20/Jun/2016:13:59:48 -0400] - Listening on All Interfaces port 636 for LDAPS requests [20/Jun/2016:13:59:48 -0400] - Listening on /var/run/slapd-DOMAIN-LOCAL.socket for LDAPI requests [20/Jun/2016:13:59:48 -0400] slapd_ldap_sasl_interactive_bind - Error: could not perform interactive bind for id [] mech [GSSAPI]: LDAP error -2 (Local error) (SASL(-1): generic failure: GSSAPI Error: Unspecified GSS failure. Minor code may provide more information (Credentials cache file '/tmp/krb5cc_495' not found)) errno 2 (No such file or directory) [20/Jun/2016:13:59:48 -0400] slapi_ldap_bind - Error: could not perform interactive bind for id [] mech [GSSAPI]: error -2 (Local error) [20/Jun/2016:13:59:48 -0400] NSMMReplicationPlugin - agmt="cn=meToserver4.domain.local" (server4:389): Replication bind with GSSAPI auth failed: LDAP error -2 (Local error) (SASL(-1): generic failure: GSSAPI Error: Unspecified GSS failure. Minor code may provide more information (Credentials cache file '/tmp/krb5cc_495' not found)) [20/Jun/2016:13:59:48 -0400] set_krb5_creds - Could not get initial credentials for principal [ldap/server1.domain.local@DOMAIN.LOCAL] in keytab [FILE:/etc/dirsrv/ds.keytab]: -1765328228 (Cannot contact any KDC for requested realm) [20/Jun/2016:13:59:48 -0400] slapd_ldap_sasl_interactive_bind - Error: could not perform interactive bind for id [] mech [GSSAPI]: LDAP error -2 (Local error) (SASL(-1): generic failure: GSSAPI Error: Unspecified GSS failure. Minor code may provide more information (Credentials cache file '/tmp/krb5cc_495' not found)) errno 0 (Success) [20/Jun/2016:13:59:48 -0400] slapi_ldap_bind - Error: could not perform interactive bind for id [] mech [GSSAPI]: error -2 (Local error) [20/Jun/2016:13:59:48 -0400] NSMMReplicationPlugin - agmt="cn=meToserver2.domain.local" (server2:389): Replication bind with GSSAPI auth failed: LDAP error -2 (Local error) (SASL(-1): generic failure: GSSAPI Error: Unspecified GSS failure. Minor code may provide more information (Credentials cache file '/tmp/krb5cc_495' not found)) [20/Jun/2016:13:59:51 -0400] NSMMReplicationPlugin - agmt="cn=meToserver8.domain.local" (server8:389): Replication bind with GSSAPI auth resumed [20/Jun/2016:13:59:51 -0400] slapd_ldap_sasl_interactive_bind - Error: could not perform interactive bind for id [] mech [GSSAPI]: LDAP error 49 (Invalid credentials) (SASL(-13): authentication failure: GSSAPI Failure: gss_accept_sec_context) errno 0 (Success) [20/Jun/2016:13:59:51 -0400] slapi_ldap_bind - Error: could not perform interactive bind for id [] mech [GSSAPI]: error 49 (Invalid credentials) [20/Jun/2016:13:59:51 -0400] NSMMReplicationPlugin - agmt="cn=meToserver3.domain.local" (server3:389): Replication bind with GSSAPI auth failed: LDAP error 49 (Invalid credentials) (SASL(-13): authentication failure: GSSAPI Failure: gss_accept_sec_context) [20/Jun/2016:13:59:51 -0400] slapd_ldap_sasl_interactive_bind - Error: could not perform interactive bind for id [] mech [GSSAPI]: LDAP error -2 (Local error) (SASL(-1): generic failure: GSSAPI Error: Unspecified GSS failure. Minor code may provide more information (No credentials cache found)) errno 2 (No such file or directory) [20/Jun/2016:13:59:51 -0400] slapi_ldap_bind - Error: could not perform interactive bind for id [] mech [GSSAPI]: error -2 (Local error) [20/Jun/2016:13:59:51 -0400] slapd_ldap_sasl_interactive_bind - Error: could not perform interactive bind for id [] mech [GSSAPI]: LDAP error -2 (Local error) (SASL(-1): generic failure: GSSAPI Error: Unspecified GSS failure. Minor code may provide more information (No credentials cache found)) errno 2 (No such file or directory) [20/Jun/2016:13:59:51 -0400] slapi_ldap_bind - Error: could not perform interactive bind for id [] mech [GSSAPI]: error -2 (Local error) [20/Jun/2016:13:59:51 -0400] slapd_ldap_sasl_interactive_bind - Error: could not perform interactive bind for id [] mech [GSSAPI]: LDAP error -2 (Local error) (SASL(-1): generic failure: GSSAPI Error: Unspecified GSS failure. Minor code may provide more information (No credentials cache found)) errno 2 (No such file or directory) [20/Jun/2016:13:59:51 -0400] slapi_ldap_bind - Error: could not perform interactive bind for id [] mech [GSSAPI]: error -2 (Local error) [20/Jun/2016:13:59:51 -0400] slapd_ldap_sasl_interactive_bind - Error: could not perform interactive bind for id [] mech [GSSAPI]: LDAP error -2 (Local error) (SASL(-1): generic failure: GSSAPI Error: Unspecified GSS failure. Minor code may provide more information (No credentials cache found)) errno 2 (No such file or directory) [20/Jun/2016:13:59:51 -0400] slapi_ldap_bind - Error: could not perform interactive bind for id [] mech [GSSAPI]: error -2 (Local error) [20/Jun/2016:13:59:51 -0400] slapd_ldap_sasl_interactive_bind - Error: could not perform interactive bind for id [] mech [GSSAPI]: LDAP error -2 (Local error) (SASL(-1): generic failure: GSSAPI Error: Unspecified GSS failure. Minor code may provide more information (No credentials cache found)) errno 2 (No such file or directory) [20/Jun/2016:13:59:51 -0400] slapi_ldap_bind - Error: could not perform interactive bind for id [] mech [GSSAPI]: error -2 (Local error) [20/Jun/2016:13:59:57 -0400] slapd_ldap_sasl_interactive_bind - Error: could not perform interactive bind for id [] mech [GSSAPI]: LDAP error 49 (Invalid credentials) (SASL(-13): authentication failure: GSSAPI Failure: gss_accept_sec_context) errno 0 (Success) [20/Jun/2016:13:59:57 -0400] slapi_ldap_bind - Error: could not perform interactive bind for id [] mech [GSSAPI]: error 49 (Invalid credentials) [20/Jun/2016:13:59:57 -0400] NSMMReplicationPlugin - agmt="cn=meToserver2.domain.local" (server2:389): Replication bind with GSSAPI auth resumed Sean Hogan <img src="cid:2__=88BBF54ADFF4633B8f9e8a93df938690918c88B@" width="16" height="16" alt="Inactive hide details for Sean Hogan---06/02/2016 09:24:39 AM---Hello All, Recently went from RHEL 6.7 IPA 3.0.47 to 6.8 IPA 3.">Sean Hogan---06/02/2016 09:24:39 AM---Hello All, Recently went from RHEL 6.7 IPA 3.0.47 to 6.8 IPA 3.0.50. I also think (not sure on this From: Sean Hogan/Durham/IBM To: freeipa-users <freeipa-users@redhat.com> Date: 06/02/2016 09:24 AM Subject: IPA 3.0.47 to 3.0.50 Upgrade problem <hr width="100%" size="2" align="left" noshade style="color:#000000; "> Hello All, Recently went from RHEL 6.7 IPA 3.0.47 to 6.8 IPA 3.0.50. I also think (not sure on this yet) that they changed ntp.. ntp used to point at my ipas.. but they look like they are now pointing elsewhere. Everything was stable at 6.7 3.0.47 pointing to IPA for NTP. However.. they all seem to have the same date. My master first IPA is acting up. Replication is off, kerberos seems to be off, DNS is off and I think IPA in general on it is toast. We do have 8 IPAs.. only FirstMaster is acting up it seems right now and all either running on KVM or ESXI. [God@FirstMasterIPA slapd-DOMAIN-LOCAL]# kinit admin kinit: Generic error (see e-text) while getting initial credential slapd-DOMAIN-LOCAL [01/Jun/2016:18:25:43 -0400] slapd_ldap_sasl_interactive_bind - Error: could not perform interactive bind for id [] mech [GSSAPI]: LDAP error -2 (Local error) (SASL(-1): generic failure: GSSAPI Error: Unspecified GSS failure. Minor code may provide more information (Cannot contact any KDC for realm 'DOMAIN.LOCAL')) errno 115 (Operation now in progress) [01/Jun/2016:18:25:43 -0400] slapi_ldap_bind - Error: could not perform interactive bind for id [] mech [GSSAPI]: error -2 (Local error) [01/Jun/2016:18:25:48 -0400] NSMMReplicationPlugin - agmt="cn=meToipaserv2.domain.local" (ipaserv2:389): Replication bind with GSSAPI auth resumed [01/Jun/2016:18:25:48 -0400] NSMMReplicationPlugin - agmt="cn=meToipaserv3.domain.local" (ipaserv3:389): Replication bind with GSSAPI auth resumed [01/Jun/2016:18:25:48 -0400] NSMMReplicationPlugin - agmt="cn=meToipaserv4.domain.local" (ipaserv4:389): Replication bind with GSSAPI auth resumed [01/Jun/2016:18:25:48 -0400] NSMMReplicationPlugin - agmt="cn=meToipaserv5.domain.local" (ipaserv5:389): Replication bind with GSSAPI auth resumed [01/Jun/2016:18:28:04 -0400] slapd_ldap_sasl_interactive_bind - Error: could not perform interactive bind for id [] mech [GSSAPI]: LDAP error 49 (Invalid credentials) (SASL(-13): authentication failure: GSSAPI Failure: gss_accept_sec_context) errno 0 (Success) [01/Jun/2016:18:28:04 -0400] slapi_ldap_bind - Error: could not perform interactive bind for id [] mech [GSSAPI]: error 49 (Invalid credentials) [01/Jun/2016:18:28:13 -0400] slapd_ldap_sasl_interactive_bind - Error: could not perform interactive bind for id [] mech [GSSAPI]: LDAP error -2 (Local error) (SASL(-1): generic failure: GSSAPI Error: Unspecified GSS failure. Minor code may provide more information (No credentials cache found)) errno 2 (No such file or directory) [01/Jun/2016:18:28:13 -0400] slapi_ldap_bind - Error: could not perform interactive bind for id [] mech [GSSAPI]: error -2 (Local error) [01/Jun/2016:18:33:03 -0400] slapd_ldap_sasl_interactive_bind - Error: could not perform interactive bind for id [] mech [GSSAPI]: LDAP error 49 (Invalid credentials) (SASL(-13): authentication failure: GSSAPI Failure: gss_accept_sec_context) errno 0 (Success) [01/Jun/2016:18:33:03 -0400] slapi_ldap_bind - Error: could not perform interactive bind for id [] mech [GSSAPI]: error 49 (Invalid credentials) [01/Jun/2016:18:33:18 -0400] slapd_ldap_sasl_interactive_bind - Error: could not perform interactive bind for id [] mech [GSSAPI]: LDAP error -2 (Local error) (SASL(-1): generic failure: GSSAPI Error: Unspecified GSS failure. Minor code may provide more information (No credentials cache found)) errno 2 (No such file or directory) [01/Jun/2016:18:33:18 -0400] slapi_ldap_bind - Error: could not perform interactive bind for id [] mech [GSSAPI]: error -2 (Local error) [01/Jun/2016:18:38:03 -0400] slapd_ldap_sasl_interactive_bind - Error: could not perform interactive bind for id [] mech [GSSAPI]: LDAP error 49 (Invalid credentials) (SASL(-13): authentication failure: GSSAPI Failure: gss_accept_sec_context) errno 0 (Success) [God@FirstMasterIPA slapd-DOMAIN-LOCAL]# ipa-replica-manage -v list --------------> just hangs and never returns [God@FirstMasterIPA slapd-DOMAIN-LOCAL]# ipactl start ------------->Just hangs here as well.. never gets to the KDC. Starting Directory Service Starting dirsrv: PKI-IPA... already running [ OK ] DOMAIN-LOCAL... already running [ OK ] If I run nslookup it fails over to a Replica for the DNS resolution instead of resolving ips itself. PKI log shows a bunch of this: [02/Jun/2016:11:15:25 -0400] NSMMReplicationPlugin - agmt="cn=masterAgreement1-ipaserver2.domain.local-pki-ca" (ipaserver2:7389): Replication bind with SIMPLE auth failed: LDAP error -1 (Can't contact LDAP server) ((null)) [02/Jun/2016:11:15:34 -0400] NSMMReplicationPlugin - agmt="cn=masterAgreement1-ipaserver2.domain.local-pki-ca" (ipaserver2:7389): Replication bind with SIMPLE auth resumed [02/Jun/2016:11:16:36 -0400] slapi_ldap_bind - Error: could not send startTLS request: error -1 (Can't contact LDAP server) errno 0 (Success) [02/Jun/2016:11:16:51 -0400] slapi_ldap_bind - Error: could not send startTLS request: error -1 (Can't contact LDAP server) errno 107 (Transport endpoint is not connected) [02/Jun/2016:11:21:51 -0400] slapi_ldap_bind - Error: could not send startTLS request: error -1 (Can't contact LDAP server) errno 0 (Success) [02/Jun/2016:11:22:06 -0400] slapi_ldap_bind - Error: could not send startTLS request: error -1 (Can't contact LDAP server) errno 107 (Transport endpoint is not connected) [02/Jun/2016:11:26:36 -0400] slapi_ldap_bind - Error: could not send startTLS request: error -1 (Can't contact LDAP server) errno 0 (Success) [02/Jun/2016:11:26:41 -0400] slapi_ldap_bind - Error: could not send startTLS request: error -1 (Can't contact LDAP server) errno 107 (Transport endpoint is not connected) [02/Jun/2016:11:31:36 -0400] slapi_ldap_bind - Error: could not send startTLS request: error -1 (Can't contact LDAP server) errno 0 (Success) [02/Jun/2016:11:31:41 -0400] slapi_ldap_bind - Error: could not send startTLS request: error -1 (Can't contact LDAP server) errno 107 (Transport endpoint is not connected) [02/Jun/2016:11:36:36 -0400] slapi_ldap_bind - Error: could not send startTLS request: error -1 (Can't contact LDAP server) errno 0 (Success) [02/Jun/2016:11:36:41 -0400] slapi_ldap_bind - Error: could not send startTLS request: error -1 (Can't contact LDAP server) errno 107 (Transport endpoint is not connected) [02/Jun/2016:11:41:46 -0400] slapi_ldap_bind - Error: could not send startTLS request: error -1 (Can't contact LDAP server) errno 0 (Success) [02/Jun/2016:11:41:51 -0400] slapi_ldap_bind - Error: could not send startTLS request: error -1 (Can't contact LDAP server) errno 107 (Transport endpoint is not connected) [02/Jun/2016:11:45:16 -0400] slapi_ldap_bind - Error: could not send startTLS request: error -1 (Can't contact LDAP server) errno 0 (Success) [02/Jun/2016:11:45:16 -0400] NSMMReplicationPlugin - agmt="cn=masterAgreement1-ipaserver3.domain.local-pki-ca" (ipaserver3:7389): Replication bind with SIMPLE auth failed: LDAP error -1 (Can't contact LDAP server) ((null)) [02/Jun/2016:11:45:25 -0400] NSMMReplicationPlugin - agmt="cn=masterAgreement1-ipaserver3.domain.local-pki-ca" (ipaserver3:7389): Replication bind with SIMPLE auth resumed [02/Jun/2016:11:46:51 -0400] slapi_ldap_bind - Error: could not send startTLS request: error -1 (Can't contact LDAP server) errno 0 (Success) [02/Jun/2016:11:46:56 -0400] slapi_ldap_bind - Error: could not send startTLS request: error -1 (Can't contact LDAP server) errno 107 (Transport endpoint is not connected) [02/Jun/2016:11:51:36 -0400] slapi_ldap_bind - Error: could not send startTLS request: error -1 (Can't contact LDAP server) errno 0 (Success) [02/Jun/2016:11:51:41 -0400] slapi_ldap_bind - Error: could not send startTLS request: error -1 (Can't contact LDAP server) errno 107 (Transport endpoint is not connected) [02/Jun/2016:11:56:46 -0400] slapi_ldap_bind - Error: could not send startTLS request: error -1 (Can't contact LDAP server) errno 0 (Success) [02/Jun/2016:11:56:51 -0400] slapi_ldap_bind - Error: could not send startTLS request: error -1 (Can't contact LDAP server) errno 107 (Transport endpoint is not connected) [02/Jun/2016:12:01:36 -0400] slapi_ldap_bind - Error: could not send startTLS request: error -1 (Can't contact LDAP server) errno 0 (Success) [02/Jun/2016:12:01:41 -0400] slapi_ldap_bind - Error: could not send startTLS request: error -1 (Can't contact LDAP server) errno 107 (Transport endpoint is not connected) [02/Jun/2016:12:05:33 -0400] slapi_ldap_bind - Error: could not send startTLS request: error -1 (Can't contact LDAP server) errno 0 (Success) [02/Jun/2016:12:05:33 -0400] NSMMReplicationPlugin - agmt="cn=masterAgreement1-ipaserver3.domain.local-pki-ca" (ipaserver3:7389): Replication bind with SIMPLE auth failed: LDAP error -1 (Can't contact LDAP server) ((null)) [02/Jun/2016:12:06:01 -0400] slapi_ldap_bind - Error: could not send startTLS request: error -1 (Can't contact LDAP server) errno 0 (Success) [02/Jun/2016:12:06:06 -0400] NSMMReplicationPlugin - agmt="cn=masterAgreement1-ipaserver3.domain.local-pki-ca" (ipaserver3:7389): Replication bind with SIMPLE auth resumed [02/Jun/2016:12:06:31 -0400] slapi_ldap_bind - Error: could not send startTLS request: error -1 (Can't contact LDAP server) errno 107 (Transport endpoint is not connected) [02/Jun/2016:12:06:41 -0400] slapi_ldap_bind - Error: could not send startTLS request: error -1 (Can't contact LDAP server) errno 0 (Success) NTP seems OK [God@FirstMasterIPA slapd-PKI-IPA]# date Thu Jun 2 12:23:00 EDT 2016 [God@ipaserver3 ~]# date Thu Jun 2 12:23:02 EDT 2016 Sean Hogan <tt>-- Manage your subscription for the Freeipa-users mailing list: </tt><tt><a href="https://www.redhat.com/mailman/listinfo/freeipa-users">https://www.redhat.com/mailman/listinfo/freeipa-users</a></tt><tt> Go to </tt><tt><a href="http://freeipa.org">http://freeipa.org</a></tt><tt> for more info on the project</tt> </body></html>