<html><body>Hello All, Recently went from RHEL 6.7 IPA 3.0.47 to 6.8 IPA 3.0.50. I also think (not sure on this yet) that they changed ntp.. ntp used to point at my ipas.. but they look like they are now pointing elsewhere. Everything was stable at 6.7 3.0.47 pointing to IPA for NTP. However.. they all seem to have the same date. My master first IPA is acting up. Replication is off, kerberos seems to be off, DNS is off and I think IPA in general on it is toast. We do have 8 IPAs.. only FirstMaster is acting up it seems right now and all either running on KVM or ESXI. [God@FirstMasterIPA slapd-DOMAIN-LOCAL]# kinit admin kinit: Generic error (see e-text) while getting initial credential slapd-DOMAIN-LOCAL [01/Jun/2016:18:25:43 -0400] slapd_ldap_sasl_interactive_bind - Error: could not perform interactive bind for id [] mech [GSSAPI]: LDAP error -2 (Local error) (SASL(-1): generic failure: GSSAPI Error: Unspecified GSS failure. Minor code may provide more information (Cannot contact any KDC for realm 'DOMAIN.LOCAL')) errno 115 (Operation now in progress) [01/Jun/2016:18:25:43 -0400] slapi_ldap_bind - Error: could not perform interactive bind for id [] mech [GSSAPI]: error -2 (Local error) [01/Jun/2016:18:25:48 -0400] NSMMReplicationPlugin - agmt="cn=meToipaserv2.domain.local" (ipaserv2:389): Replication bind with GSSAPI auth resumed [01/Jun/2016:18:25:48 -0400] NSMMReplicationPlugin - agmt="cn=meToipaserv3.domain.local" (ipaserv3:389): Replication bind with GSSAPI auth resumed [01/Jun/2016:18:25:48 -0400] NSMMReplicationPlugin - agmt="cn=meToipaserv4.domain.local" (ipaserv4:389): Replication bind with GSSAPI auth resumed [01/Jun/2016:18:25:48 -0400] NSMMReplicationPlugin - agmt="cn=meToipaserv5.domain.local" (ipaserv5:389): Replication bind with GSSAPI auth resumed [01/Jun/2016:18:28:04 -0400] slapd_ldap_sasl_interactive_bind - Error: could not perform interactive bind for id [] mech [GSSAPI]: LDAP error 49 (Invalid credentials) (SASL(-13): authentication failure: GSSAPI Failure: gss_accept_sec_context) errno 0 (Success) [01/Jun/2016:18:28:04 -0400] slapi_ldap_bind - Error: could not perform interactive bind for id [] mech [GSSAPI]: error 49 (Invalid credentials) [01/Jun/2016:18:28:13 -0400] slapd_ldap_sasl_interactive_bind - Error: could not perform interactive bind for id [] mech [GSSAPI]: LDAP error -2 (Local error) (SASL(-1): generic failure: GSSAPI Error: Unspecified GSS failure. Minor code may provide more information (No credentials cache found)) errno 2 (No such file or directory) [01/Jun/2016:18:28:13 -0400] slapi_ldap_bind - Error: could not perform interactive bind for id [] mech [GSSAPI]: error -2 (Local error) [01/Jun/2016:18:33:03 -0400] slapd_ldap_sasl_interactive_bind - Error: could not perform interactive bind for id [] mech [GSSAPI]: LDAP error 49 (Invalid credentials) (SASL(-13): authentication failure: GSSAPI Failure: gss_accept_sec_context) errno 0 (Success) [01/Jun/2016:18:33:03 -0400] slapi_ldap_bind - Error: could not perform interactive bind for id [] mech [GSSAPI]: error 49 (Invalid credentials) [01/Jun/2016:18:33:18 -0400] slapd_ldap_sasl_interactive_bind - Error: could not perform interactive bind for id [] mech [GSSAPI]: LDAP error -2 (Local error) (SASL(-1): generic failure: GSSAPI Error: Unspecified GSS failure. Minor code may provide more information (No credentials cache found)) errno 2 (No such file or directory) [01/Jun/2016:18:33:18 -0400] slapi_ldap_bind - Error: could not perform interactive bind for id [] mech [GSSAPI]: error -2 (Local error) [01/Jun/2016:18:38:03 -0400] slapd_ldap_sasl_interactive_bind - Error: could not perform interactive bind for id [] mech [GSSAPI]: LDAP error 49 (Invalid credentials) (SASL(-13): authentication failure: GSSAPI Failure: gss_accept_sec_context) errno 0 (Success) [God@FirstMasterIPA slapd-DOMAIN-LOCAL]# ipa-replica-manage -v list --------------> just hangs and never returns [God@FirstMasterIPA slapd-DOMAIN-LOCAL]# ipactl start ------------->Just hangs here as well.. never gets to the KDC. Starting Directory Service Starting dirsrv: PKI-IPA... already running [ OK ] DOMAIN-LOCAL... already running [ OK ] If I run nslookup it fails over to a Replica for the DNS resolution instead of resolving ips itself. PKI log shows a bunch of this: [02/Jun/2016:11:15:25 -0400] NSMMReplicationPlugin - agmt="cn=masterAgreement1-ipaserver2.domain.local-pki-ca" (ipaserver2:7389): Replication bind with SIMPLE auth failed: LDAP error -1 (Can't contact LDAP server) ((null)) [02/Jun/2016:11:15:34 -0400] NSMMReplicationPlugin - agmt="cn=masterAgreement1-ipaserver2.domain.local-pki-ca" (ipaserver2:7389): Replication bind with SIMPLE auth resumed [02/Jun/2016:11:16:36 -0400] slapi_ldap_bind - Error: could not send startTLS request: error -1 (Can't contact LDAP server) errno 0 (Success) [02/Jun/2016:11:16:51 -0400] slapi_ldap_bind - Error: could not send startTLS request: error -1 (Can't contact LDAP server) errno 107 (Transport endpoint is not connected) [02/Jun/2016:11:21:51 -0400] slapi_ldap_bind - Error: could not send startTLS request: error -1 (Can't contact LDAP server) errno 0 (Success) [02/Jun/2016:11:22:06 -0400] slapi_ldap_bind - Error: could not send startTLS request: error -1 (Can't contact LDAP server) errno 107 (Transport endpoint is not connected) [02/Jun/2016:11:26:36 -0400] slapi_ldap_bind - Error: could not send startTLS request: error -1 (Can't contact LDAP server) errno 0 (Success) [02/Jun/2016:11:26:41 -0400] slapi_ldap_bind - Error: could not send startTLS request: error -1 (Can't contact LDAP server) errno 107 (Transport endpoint is not connected) [02/Jun/2016:11:31:36 -0400] slapi_ldap_bind - Error: could not send startTLS request: error -1 (Can't contact LDAP server) errno 0 (Success) [02/Jun/2016:11:31:41 -0400] slapi_ldap_bind - Error: could not send startTLS request: error -1 (Can't contact LDAP server) errno 107 (Transport endpoint is not connected) [02/Jun/2016:11:36:36 -0400] slapi_ldap_bind - Error: could not send startTLS request: error -1 (Can't contact LDAP server) errno 0 (Success) [02/Jun/2016:11:36:41 -0400] slapi_ldap_bind - Error: could not send startTLS request: error -1 (Can't contact LDAP server) errno 107 (Transport endpoint is not connected) [02/Jun/2016:11:41:46 -0400] slapi_ldap_bind - Error: could not send startTLS request: error -1 (Can't contact LDAP server) errno 0 (Success) [02/Jun/2016:11:41:51 -0400] slapi_ldap_bind - Error: could not send startTLS request: error -1 (Can't contact LDAP server) errno 107 (Transport endpoint is not connected) [02/Jun/2016:11:45:16 -0400] slapi_ldap_bind - Error: could not send startTLS request: error -1 (Can't contact LDAP server) errno 0 (Success) [02/Jun/2016:11:45:16 -0400] NSMMReplicationPlugin - agmt="cn=masterAgreement1-ipaserver3.domain.local-pki-ca" (ipaserver3:7389): Replication bind with SIMPLE auth failed: LDAP error -1 (Can't contact LDAP server) ((null)) [02/Jun/2016:11:45:25 -0400] NSMMReplicationPlugin - agmt="cn=masterAgreement1-ipaserver3.domain.local-pki-ca" (ipaserver3:7389): Replication bind with SIMPLE auth resumed [02/Jun/2016:11:46:51 -0400] slapi_ldap_bind - Error: could not send startTLS request: error -1 (Can't contact LDAP server) errno 0 (Success) [02/Jun/2016:11:46:56 -0400] slapi_ldap_bind - Error: could not send startTLS request: error -1 (Can't contact LDAP server) errno 107 (Transport endpoint is not connected) [02/Jun/2016:11:51:36 -0400] slapi_ldap_bind - Error: could not send startTLS request: error -1 (Can't contact LDAP server) errno 0 (Success) [02/Jun/2016:11:51:41 -0400] slapi_ldap_bind - Error: could not send startTLS request: error -1 (Can't contact LDAP server) errno 107 (Transport endpoint is not connected) [02/Jun/2016:11:56:46 -0400] slapi_ldap_bind - Error: could not send startTLS request: error -1 (Can't contact LDAP server) errno 0 (Success) [02/Jun/2016:11:56:51 -0400] slapi_ldap_bind - Error: could not send startTLS request: error -1 (Can't contact LDAP server) errno 107 (Transport endpoint is not connected) [02/Jun/2016:12:01:36 -0400] slapi_ldap_bind - Error: could not send startTLS request: error -1 (Can't contact LDAP server) errno 0 (Success) [02/Jun/2016:12:01:41 -0400] slapi_ldap_bind - Error: could not send startTLS request: error -1 (Can't contact LDAP server) errno 107 (Transport endpoint is not connected) [02/Jun/2016:12:05:33 -0400] slapi_ldap_bind - Error: could not send startTLS request: error -1 (Can't contact LDAP server) errno 0 (Success) [02/Jun/2016:12:05:33 -0400] NSMMReplicationPlugin - agmt="cn=masterAgreement1-ipaserver3.domain.local-pki-ca" (ipaserver3:7389): Replication bind with SIMPLE auth failed: LDAP error -1 (Can't contact LDAP server) ((null)) [02/Jun/2016:12:06:01 -0400] slapi_ldap_bind - Error: could not send startTLS request: error -1 (Can't contact LDAP server) errno 0 (Success) [02/Jun/2016:12:06:06 -0400] NSMMReplicationPlugin - agmt="cn=masterAgreement1-ipaserver3.domain.local-pki-ca" (ipaserver3:7389): Replication bind with SIMPLE auth resumed [02/Jun/2016:12:06:31 -0400] slapi_ldap_bind - Error: could not send startTLS request: error -1 (Can't contact LDAP server) errno 107 (Transport endpoint is not connected) [02/Jun/2016:12:06:41 -0400] slapi_ldap_bind - Error: could not send startTLS request: error -1 (Can't contact LDAP server) errno 0 (Success) NTP seems OK [God@FirstMasterIPA slapd-PKI-IPA]# date Thu Jun 2 12:23:00 EDT 2016 [God@ipaserver3 ~]# date Thu Jun 2 12:23:02 EDT 2016 Sean Hogan </body></html>