[Freeipa-users] IPA 3.0.47 to 3.0.50 Upgrade problem

Thu Jun 2 21:42:07 UTC 2016

Sean Hogan wrote:
> Hello All,
>
> Recently went from RHEL 6.7 IPA 3.0.47 to 6.8 IPA 3.0.50. I also think
> (not sure on this yet) that they changed ntp.. ntp used to point at my
> ipas.. but they look like they are now pointing elsewhere. Everything
> was stable at 6.7 3.0.47 pointing to IPA for NTP. However.. they all
> seem to have the same date.
>
>
> My master first IPA is acting up. Replication is off, kerberos seems to
> be off, DNS is off and I think IPA in general on it is toast.
> We do have 8 IPAs.. only FirstMaster is acting up it seems right now and
> all either running on KVM or ESXI.
>
>
> [God at FirstMasterIPA slapd-DOMAIN-LOCAL]# kinit admin
> kinit: Generic error (see e-text) while getting initial credential

ipactl status should show what services are running. It looks like the 
KDC is responding but can't talk to the LDAP backend.
>
>
> slapd-DOMAIN-LOCAL
> [01/Jun/2016:18:25:43 -0400] slapd_ldap_sasl_interactive_bind - Error:
> could not perform interactive bind for id [] mech [GSSAPI]: LDAP error
> -2 (Local error) (SASL(-1): generic failure: GSSAPI Error: Unspecified
> GSS failure. Minor code may provide more information (Cannot contact any
> KDC for realm 'DOMAIN.LOCAL')) errno 115 (Operation now in progress)
> [01/Jun/2016:18:25:43 -0400] slapi_ldap_bind - Error: could not perform
> interactive bind for id [] mech [GSSAPI]: error -2 (Local error)
> [01/Jun/2016:18:25:48 -0400] NSMMReplicationPlugin -
> agmt="cn=meToipaserv2.domain.local" (ipaserv2:389): Replication bind
> with GSSAPI auth resumed
> [01/Jun/2016:18:25:48 -0400] NSMMReplicationPlugin -
> agmt="cn=meToipaserv3.domain.local" (ipaserv3:389): Replication bind
> with GSSAPI auth resumed
> [01/Jun/2016:18:25:48 -0400] NSMMReplicationPlugin -
> agmt="cn=meToipaserv4.domain.local" (ipaserv4:389): Replication bind
> with GSSAPI auth resumed
> [01/Jun/2016:18:25:48 -0400] NSMMReplicationPlugin -
> agmt="cn=meToipaserv5.domain.local" (ipaserv5:389): Replication bind
> with GSSAPI auth resumed
> [01/Jun/2016:18:28:04 -0400] slapd_ldap_sasl_interactive_bind - Error:
> could not perform interactive bind for id [] mech [GSSAPI]: LDAP error
> 49 (Invalid credentials) (SASL(-13): authentication failure: GSSAPI
> Failure: gss_accept_sec_context) errno 0 (Success)
> [01/Jun/2016:18:28:04 -0400] slapi_ldap_bind - Error: could not perform
> interactive bind for id [] mech [GSSAPI]: error 49 (Invalid credentials)
> [01/Jun/2016:18:28:13 -0400] slapd_ldap_sasl_interactive_bind - Error:
> could not perform interactive bind for id [] mech [GSSAPI]: LDAP error
> -2 (Local error) (SASL(-1): generic failure: GSSAPI Error: Unspecified
> GSS failure. Minor code may provide more information (No credentials
> cache found)) errno 2 (No such file or directory)
> [01/Jun/2016:18:28:13 -0400] slapi_ldap_bind - Error: could not perform
> interactive bind for id [] mech [GSSAPI]: error -2 (Local error)
> [01/Jun/2016:18:33:03 -0400] slapd_ldap_sasl_interactive_bind - Error:
> could not perform interactive bind for id [] mech [GSSAPI]: LDAP error
> 49 (Invalid credentials) (SASL(-13): authentication failure: GSSAPI
> Failure: gss_accept_sec_context) errno 0 (Success)
> [01/Jun/2016:18:33:03 -0400] slapi_ldap_bind - Error: could not perform
> interactive bind for id [] mech [GSSAPI]: error 49 (Invalid credentials)
> [01/Jun/2016:18:33:18 -0400] slapd_ldap_sasl_interactive_bind - Error:
> could not perform interactive bind for id [] mech [GSSAPI]: LDAP error
> -2 (Local error) (SASL(-1): generic failure: GSSAPI Error: Unspecified
> GSS failure. Minor code may provide more information (No credentials
> cache found)) errno 2 (No such file or directory)
> [01/Jun/2016:18:33:18 -0400] slapi_ldap_bind - Error: could not perform
> interactive bind for id [] mech [GSSAPI]: error -2 (Local error)
> [01/Jun/2016:18:38:03 -0400] slapd_ldap_sasl_interactive_bind - Error:
> could not perform interactive bind for id [] mech [GSSAPI]: LDAP error
> 49 (Invalid credentials) (SASL(-13): authentication failure: GSSAPI
> Failure: gss_accept_sec_context) errno 0 (Success)

And this makes it look like it can't talk to the KDC.

I'd check for SELinux errors, ausearch -m AVC -ts recent

I think the rest is just indication that something is wrong with either 
the LDAP servers, the KDC or both.

You may also want to look at /var/log/ipaupgrade.log to ensure that the 
upgrade was successful.

rob

>
> [God at FirstMasterIPA slapd-DOMAIN-LOCAL]# ipa-replica-manage -v list
> --------------> just hangs and never returns
>
>
> [God at FirstMasterIPA slapd-DOMAIN-LOCAL]# ipactl start ------------->Just
> hangs here as well.. never gets to the KDC.
>
> Starting Directory Service
> Starting dirsrv:
> PKI-IPA... already running [ OK ]
> DOMAIN-LOCAL... already running [ OK ]
>
>
> If I run nslookup it fails over to a Replica for the DNS resolution
> instead of resolving ips itself.
>
>
>
> PKI log shows a bunch of this:
> [02/Jun/2016:11:15:25 -0400] NSMMReplicationPlugin -
> agmt="cn=masterAgreement1-ipaserver2.domain.local-pki-ca"
> (ipaserver2:7389): Replication bind with SIMPLE auth failed: LDAP error
> -1 (Can't contact LDAP server) ((null))
> [02/Jun/2016:11:15:34 -0400] NSMMReplicationPlugin -
> agmt="cn=masterAgreement1-ipaserver2.domain.local-pki-ca"
> (ipaserver2:7389): Replication bind with SIMPLE auth resumed
> [02/Jun/2016:11:16:36 -0400] slapi_ldap_bind - Error: could not send
> startTLS request: error -1 (Can't contact LDAP server) errno 0 (Success)
> [02/Jun/2016:11:16:51 -0400] slapi_ldap_bind - Error: could not send
> startTLS request: error -1 (Can't contact LDAP server) errno 107
> (Transport endpoint is not connected)
> [02/Jun/2016:11:21:51 -0400] slapi_ldap_bind - Error: could not send
> startTLS request: error -1 (Can't contact LDAP server) errno 0 (Success)
> [02/Jun/2016:11:22:06 -0400] slapi_ldap_bind - Error: could not send
> startTLS request: error -1 (Can't contact LDAP server) errno 107
> (Transport endpoint is not connected)
> [02/Jun/2016:11:26:36 -0400] slapi_ldap_bind - Error: could not send
> startTLS request: error -1 (Can't contact LDAP server) errno 0 (Success)
> [02/Jun/2016:11:26:41 -0400] slapi_ldap_bind - Error: could not send
> startTLS request: error -1 (Can't contact LDAP server) errno 107
> (Transport endpoint is not connected)
> [02/Jun/2016:11:31:36 -0400] slapi_ldap_bind - Error: could not send
> startTLS request: error -1 (Can't contact LDAP server) errno 0 (Success)
> [02/Jun/2016:11:31:41 -0400] slapi_ldap_bind - Error: could not send
> startTLS request: error -1 (Can't contact LDAP server) errno 107
> (Transport endpoint is not connected)
> [02/Jun/2016:11:36:36 -0400] slapi_ldap_bind - Error: could not send
> startTLS request: error -1 (Can't contact LDAP server) errno 0 (Success)
> [02/Jun/2016:11:36:41 -0400] slapi_ldap_bind - Error: could not send
> startTLS request: error -1 (Can't contact LDAP server) errno 107
> (Transport endpoint is not connected)
> [02/Jun/2016:11:41:46 -0400] slapi_ldap_bind - Error: could not send
> startTLS request: error -1 (Can't contact LDAP server) errno 0 (Success)
> [02/Jun/2016:11:41:51 -0400] slapi_ldap_bind - Error: could not send
> startTLS request: error -1 (Can't contact LDAP server) errno 107
> (Transport endpoint is not connected)
> [02/Jun/2016:11:45:16 -0400] slapi_ldap_bind - Error: could not send
> startTLS request: error -1 (Can't contact LDAP server) errno 0 (Success)
> [02/Jun/2016:11:45:16 -0400] NSMMReplicationPlugin -
> agmt="cn=masterAgreement1-ipaserver3.domain.local-pki-ca"
> (ipaserver3:7389): Replication bind with SIMPLE auth failed: LDAP error
> -1 (Can't contact LDAP server) ((null))
> [02/Jun/2016:11:45:25 -0400] NSMMReplicationPlugin -
> agmt="cn=masterAgreement1-ipaserver3.domain.local-pki-ca"
> (ipaserver3:7389): Replication bind with SIMPLE auth resumed
> [02/Jun/2016:11:46:51 -0400] slapi_ldap_bind - Error: could not send
> startTLS request: error -1 (Can't contact LDAP server) errno 0 (Success)
> [02/Jun/2016:11:46:56 -0400] slapi_ldap_bind - Error: could not send
> startTLS request: error -1 (Can't contact LDAP server) errno 107
> (Transport endpoint is not connected)
> [02/Jun/2016:11:51:36 -0400] slapi_ldap_bind - Error: could not send
> startTLS request: error -1 (Can't contact LDAP server) errno 0 (Success)
> [02/Jun/2016:11:51:41 -0400] slapi_ldap_bind - Error: could not send
> startTLS request: error -1 (Can't contact LDAP server) errno 107
> (Transport endpoint is not connected)
> [02/Jun/2016:11:56:46 -0400] slapi_ldap_bind - Error: could not send
> startTLS request: error -1 (Can't contact LDAP server) errno 0 (Success)
> [02/Jun/2016:11:56:51 -0400] slapi_ldap_bind - Error: could not send
> startTLS request: error -1 (Can't contact LDAP server) errno 107
> (Transport endpoint is not connected)
> [02/Jun/2016:12:01:36 -0400] slapi_ldap_bind - Error: could not send
> startTLS request: error -1 (Can't contact LDAP server) errno 0 (Success)
> [02/Jun/2016:12:01:41 -0400] slapi_ldap_bind - Error: could not send
> startTLS request: error -1 (Can't contact LDAP server) errno 107
> (Transport endpoint is not connected)
> [02/Jun/2016:12:05:33 -0400] slapi_ldap_bind - Error: could not send
> startTLS request: error -1 (Can't contact LDAP server) errno 0 (Success)
> [02/Jun/2016:12:05:33 -0400] NSMMReplicationPlugin -
> agmt="cn=masterAgreement1-ipaserver3.domain.local-pki-ca"
> (ipaserver3:7389): Replication bind with SIMPLE auth failed: LDAP error
> -1 (Can't contact LDAP server) ((null))
> [02/Jun/2016:12:06:01 -0400] slapi_ldap_bind - Error: could not send
> startTLS request: error -1 (Can't contact LDAP server) errno 0 (Success)
> [02/Jun/2016:12:06:06 -0400] NSMMReplicationPlugin -
> agmt="cn=masterAgreement1-ipaserver3.domain.local-pki-ca"
> (ipaserver3:7389): Replication bind with SIMPLE auth resumed
> [02/Jun/2016:12:06:31 -0400] slapi_ldap_bind - Error: could not send
> startTLS request: error -1 (Can't contact LDAP server) errno 107
> (Transport endpoint is not connected)
> [02/Jun/2016:12:06:41 -0400] slapi_ldap_bind - Error: could not send
> startTLS request: error -1 (Can't contact LDAP server) errno 0 (Success)
>
>
>
>
> NTP seems OK
> [God at FirstMasterIPA slapd-PKI-IPA]# date
> Thu Jun 2 12:23:00 EDT 2016
>
> [God at ipaserver3 ~]# date
> Thu Jun 2 12:23:02 EDT 2016
>
>
>
> Sean Hogan
>
>
>
>
>