[Freeipa-users] IPA 3.0.47 to 3.0.50 Upgrade problem

Thu Jun 2 16:24:39 UTC 2016

Hello All,

Recently went from RHEL 6.7 IPA 3.0.47 to 6.8 IPA 3.0.50.  I also think
(not sure on this yet) that they changed ntp.. ntp used to point at my
ipas.. but they look like they are now pointing elsewhere.  Everything was
stable at 6.7 3.0.47 pointing to IPA for NTP.  However.. they all seem to
have the same date.

My master first IPA is acting up.  Replication is off, kerberos seems to be
off, DNS is off and I think IPA in general on it is toast.
We do have 8 IPAs.. only FirstMaster is acting up it seems right now and
all either running on KVM or ESXI.

[God at FirstMasterIPA slapd-DOMAIN-LOCAL]# kinit admin
kinit: Generic error (see e-text) while getting initial credential

slapd-DOMAIN-LOCAL
[01/Jun/2016:18:25:43 -0400] slapd_ldap_sasl_interactive_bind - Error:
could not perform interactive bind for id [] mech [GSSAPI]: LDAP error -2
(Local error) (SASL(-1): generic failure: GSSAPI Error: Unspecified GSS
failure.  Minor code may provide more information (Cannot contact any KDC
for realm 'DOMAIN.LOCAL')) errno 115 (Operation now in progress)
[01/Jun/2016:18:25:43 -0400] slapi_ldap_bind - Error: could not perform
interactive bind for id [] mech [GSSAPI]: error -2 (Local error)
[01/Jun/2016:18:25:48 -0400] NSMMReplicationPlugin -
agmt="cn=meToipaserv2.domain.local" (ipaserv2:389): Replication bind with
GSSAPI auth resumed
[01/Jun/2016:18:25:48 -0400] NSMMReplicationPlugin -
agmt="cn=meToipaserv3.domain.local" (ipaserv3:389): Replication bind with
GSSAPI auth resumed
[01/Jun/2016:18:25:48 -0400] NSMMReplicationPlugin -
agmt="cn=meToipaserv4.domain.local" (ipaserv4:389): Replication bind with
GSSAPI auth resumed
[01/Jun/2016:18:25:48 -0400] NSMMReplicationPlugin -
agmt="cn=meToipaserv5.domain.local" (ipaserv5:389): Replication bind with
GSSAPI auth resumed
[01/Jun/2016:18:28:04 -0400] slapd_ldap_sasl_interactive_bind - Error:
could not perform interactive bind for id [] mech [GSSAPI]: LDAP error 49
(Invalid credentials) (SASL(-13): authentication failure: GSSAPI Failure:
gss_accept_sec_context) errno 0 (Success)
[01/Jun/2016:18:28:04 -0400] slapi_ldap_bind - Error: could not perform
interactive bind for id [] mech [GSSAPI]: error 49 (Invalid credentials)
[01/Jun/2016:18:28:13 -0400] slapd_ldap_sasl_interactive_bind - Error:
could not perform interactive bind for id [] mech [GSSAPI]: LDAP error -2
(Local error) (SASL(-1): generic failure: GSSAPI Error: Unspecified GSS
failure.  Minor code may provide more information (No credentials cache
found)) errno 2 (No such file or directory)
[01/Jun/2016:18:28:13 -0400] slapi_ldap_bind - Error: could not perform
interactive bind for id [] mech [GSSAPI]: error -2 (Local error)
[01/Jun/2016:18:33:03 -0400] slapd_ldap_sasl_interactive_bind - Error:
could not perform interactive bind for id [] mech [GSSAPI]: LDAP error 49
(Invalid credentials) (SASL(-13): authentication failure: GSSAPI Failure:
gss_accept_sec_context) errno 0 (Success)
[01/Jun/2016:18:33:03 -0400] slapi_ldap_bind - Error: could not perform
interactive bind for id [] mech [GSSAPI]: error 49 (Invalid credentials)
[01/Jun/2016:18:33:18 -0400] slapd_ldap_sasl_interactive_bind - Error:
could not perform interactive bind for id [] mech [GSSAPI]: LDAP error -2
(Local error) (SASL(-1): generic failure: GSSAPI Error: Unspecified GSS
failure.  Minor code may provide more information (No credentials cache
found)) errno 2 (No such file or directory)
[01/Jun/2016:18:33:18 -0400] slapi_ldap_bind - Error: could not perform
interactive bind for id [] mech [GSSAPI]: error -2 (Local error)
[01/Jun/2016:18:38:03 -0400] slapd_ldap_sasl_interactive_bind - Error:
could not perform interactive bind for id [] mech [GSSAPI]: LDAP error 49
(Invalid credentials) (SASL(-13): authentication failure: GSSAPI Failure:
gss_accept_sec_context) errno 0 (Success)

[God at FirstMasterIPA slapd-DOMAIN-LOCAL]# ipa-replica-manage -v  list
--------------> just hangs and never returns

[God at FirstMasterIPA slapd-DOMAIN-LOCAL]# ipactl start    ------------->Just
hangs here as well.. never gets to the  KDC.

Starting Directory Service
Starting dirsrv:
    PKI-IPA... already running                             [  OK  ]
    DOMAIN-LOCAL... already running                        [  OK  ]

If I run nslookup it fails over to a Replica for the DNS resolution instead
of resolving ips itself.

PKI log shows a bunch of this:
[02/Jun/2016:11:15:25 -0400] NSMMReplicationPlugin -
agmt="cn=masterAgreement1-ipaserver2.domain.local-pki-ca" (ipaserver2:7389):
 Replication bind with SIMPLE auth failed: LDAP error -1 (Can't contact
LDAP server) ((null))
[02/Jun/2016:11:15:34 -0400] NSMMReplicationPlugin -
agmt="cn=masterAgreement1-ipaserver2.domain.local-pki-ca" (ipaserver2:7389):
 Replication bind with SIMPLE auth resumed
[02/Jun/2016:11:16:36 -0400] slapi_ldap_bind - Error: could not send
startTLS request: error -1 (Can't contact LDAP server) errno 0 (Success)
[02/Jun/2016:11:16:51 -0400] slapi_ldap_bind - Error: could not send
startTLS request: error -1 (Can't contact LDAP server) errno 107 (Transport
endpoint is not connected)
[02/Jun/2016:11:21:51 -0400] slapi_ldap_bind - Error: could not send
startTLS request: error -1 (Can't contact LDAP server) errno 0 (Success)
[02/Jun/2016:11:22:06 -0400] slapi_ldap_bind - Error: could not send
startTLS request: error -1 (Can't contact LDAP server) errno 107 (Transport
endpoint is not connected)
[02/Jun/2016:11:26:36 -0400] slapi_ldap_bind - Error: could not send
startTLS request: error -1 (Can't contact LDAP server) errno 0 (Success)
[02/Jun/2016:11:26:41 -0400] slapi_ldap_bind - Error: could not send
startTLS request: error -1 (Can't contact LDAP server) errno 107 (Transport
endpoint is not connected)
[02/Jun/2016:11:31:36 -0400] slapi_ldap_bind - Error: could not send
startTLS request: error -1 (Can't contact LDAP server) errno 0 (Success)
[02/Jun/2016:11:31:41 -0400] slapi_ldap_bind - Error: could not send
startTLS request: error -1 (Can't contact LDAP server) errno 107 (Transport
endpoint is not connected)
[02/Jun/2016:11:36:36 -0400] slapi_ldap_bind - Error: could not send
startTLS request: error -1 (Can't contact LDAP server) errno 0 (Success)
[02/Jun/2016:11:36:41 -0400] slapi_ldap_bind - Error: could not send
startTLS request: error -1 (Can't contact LDAP server) errno 107 (Transport
endpoint is not connected)
[02/Jun/2016:11:41:46 -0400] slapi_ldap_bind - Error: could not send
startTLS request: error -1 (Can't contact LDAP server) errno 0 (Success)
[02/Jun/2016:11:41:51 -0400] slapi_ldap_bind - Error: could not send
startTLS request: error -1 (Can't contact LDAP server) errno 107 (Transport
endpoint is not connected)
[02/Jun/2016:11:45:16 -0400] slapi_ldap_bind - Error: could not send
startTLS request: error -1 (Can't contact LDAP server) errno 0 (Success)
[02/Jun/2016:11:45:16 -0400] NSMMReplicationPlugin -
agmt="cn=masterAgreement1-ipaserver3.domain.local-pki-ca" (ipaserver3:7389):
 Replication bind with SIMPLE auth failed: LDAP error -1 (Can't contact
LDAP server) ((null))
[02/Jun/2016:11:45:25 -0400] NSMMReplicationPlugin -
agmt="cn=masterAgreement1-ipaserver3.domain.local-pki-ca" (ipaserver3:7389):
 Replication bind with SIMPLE auth resumed
[02/Jun/2016:11:46:51 -0400] slapi_ldap_bind - Error: could not send
startTLS request: error -1 (Can't contact LDAP server) errno 0 (Success)
[02/Jun/2016:11:46:56 -0400] slapi_ldap_bind - Error: could not send
startTLS request: error -1 (Can't contact LDAP server) errno 107 (Transport
endpoint is not connected)
[02/Jun/2016:11:51:36 -0400] slapi_ldap_bind - Error: could not send
startTLS request: error -1 (Can't contact LDAP server) errno 0 (Success)
[02/Jun/2016:11:51:41 -0400] slapi_ldap_bind - Error: could not send
startTLS request: error -1 (Can't contact LDAP server) errno 107 (Transport
endpoint is not connected)
[02/Jun/2016:11:56:46 -0400] slapi_ldap_bind - Error: could not send
startTLS request: error -1 (Can't contact LDAP server) errno 0 (Success)
[02/Jun/2016:11:56:51 -0400] slapi_ldap_bind - Error: could not send
startTLS request: error -1 (Can't contact LDAP server) errno 107 (Transport
endpoint is not connected)
[02/Jun/2016:12:01:36 -0400] slapi_ldap_bind - Error: could not send
startTLS request: error -1 (Can't contact LDAP server) errno 0 (Success)
[02/Jun/2016:12:01:41 -0400] slapi_ldap_bind - Error: could not send
startTLS request: error -1 (Can't contact LDAP server) errno 107 (Transport
endpoint is not connected)
[02/Jun/2016:12:05:33 -0400] slapi_ldap_bind - Error: could not send
startTLS request: error -1 (Can't contact LDAP server) errno 0 (Success)
[02/Jun/2016:12:05:33 -0400] NSMMReplicationPlugin -
agmt="cn=masterAgreement1-ipaserver3.domain.local-pki-ca" (ipaserver3:7389):
 Replication bind with SIMPLE auth failed: LDAP error -1 (Can't contact
LDAP server) ((null))
[02/Jun/2016:12:06:01 -0400] slapi_ldap_bind - Error: could not send
startTLS request: error -1 (Can't contact LDAP server) errno 0 (Success)
[02/Jun/2016:12:06:06 -0400] NSMMReplicationPlugin -
agmt="cn=masterAgreement1-ipaserver3.domain.local-pki-ca" (ipaserver3:7389):
 Replication bind with SIMPLE auth resumed
[02/Jun/2016:12:06:31 -0400] slapi_ldap_bind - Error: could not send
startTLS request: error -1 (Can't contact LDAP server) errno 107 (Transport
endpoint is not connected)
[02/Jun/2016:12:06:41 -0400] slapi_ldap_bind - Error: could not send
startTLS request: error -1 (Can't contact LDAP server) errno 0 (Success)

NTP seems OK
[God at FirstMasterIPA slapd-PKI-IPA]# date
Thu Jun  2 12:23:00 EDT 2016

[God at ipaserver3 ~]# date
Thu Jun  2 12:23:02 EDT 2016

Sean Hogan

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/freeipa-users/attachments/20160602/46cc09e8/attachment.htm>