[Freeipa-users] IPA 3.0.47 to 3.0.50 Upgrade problem

Fri Jun 3 14:23:05 UTC 2016

Hi Robert..

  Thanks for the reply.  Think I might have found the issue.  The KVM host
my master was running on was showing redhat release 6.5 but the libvrt
packages were showing 6.6.  I think the managers of the kvm host did not
reboot it after an update with new kernel.  Asked them to reboot the KVM
host after I gracefully shut down my NFS profile server and Master IPA
(both run on that host).  However Master IPA would not shutdown so they
rebooted it with the IPA server still running.  Once it was back up and the
2 servers were back up I had to gracefully shutdown the Master IPA and this
time it did shutdown.  Powered back up and it seems to be running fine now.
BTW... there is a lot of info in the upgrade log but will overview it more
later.

Thanks

Sean Hogan

From:	Rob Crittenden <rcritten at redhat.com>
To:	Sean Hogan/Durham/IBM at IBMUS, freeipa-users
            <freeipa-users at redhat.com>
Date:	06/02/2016 02:42 PM
Subject:	Re: [Freeipa-users] IPA 3.0.47 to 3.0.50 Upgrade problem

Sean Hogan wrote:
> Hello All,
>
> Recently went from RHEL 6.7 IPA 3.0.47 to 6.8 IPA 3.0.50. I also think
> (not sure on this yet) that they changed ntp.. ntp used to point at my
> ipas.. but they look like they are now pointing elsewhere. Everything
> was stable at 6.7 3.0.47 pointing to IPA for NTP. However.. they all
> seem to have the same date.
>
>
> My master first IPA is acting up. Replication is off, kerberos seems to
> be off, DNS is off and I think IPA in general on it is toast.
> We do have 8 IPAs.. only FirstMaster is acting up it seems right now and
> all either running on KVM or ESXI.
>
>
> [God at FirstMasterIPA slapd-DOMAIN-LOCAL]# kinit admin
> kinit: Generic error (see e-text) while getting initial credential

ipactl status should show what services are running. It looks like the
KDC is responding but can't talk to the LDAP backend.
>
>
> slapd-DOMAIN-LOCAL
> [01/Jun/2016:18:25:43 -0400] slapd_ldap_sasl_interactive_bind - Error:
> could not perform interactive bind for id [] mech [GSSAPI]: LDAP error
> -2 (Local error) (SASL(-1): generic failure: GSSAPI Error: Unspecified
> GSS failure. Minor code may provide more information (Cannot contact any
> KDC for realm 'DOMAIN.LOCAL')) errno 115 (Operation now in progress)
> [01/Jun/2016:18:25:43 -0400] slapi_ldap_bind - Error: could not perform
> interactive bind for id [] mech [GSSAPI]: error -2 (Local error)
> [01/Jun/2016:18:25:48 -0400] NSMMReplicationPlugin -
> agmt="cn=meToipaserv2.domain.local" (ipaserv2:389): Replication bind
> with GSSAPI auth resumed
> [01/Jun/2016:18:25:48 -0400] NSMMReplicationPlugin -
> agmt="cn=meToipaserv3.domain.local" (ipaserv3:389): Replication bind
> with GSSAPI auth resumed
> [01/Jun/2016:18:25:48 -0400] NSMMReplicationPlugin -
> agmt="cn=meToipaserv4.domain.local" (ipaserv4:389): Replication bind
> with GSSAPI auth resumed
> [01/Jun/2016:18:25:48 -0400] NSMMReplicationPlugin -
> agmt="cn=meToipaserv5.domain.local" (ipaserv5:389): Replication bind
> with GSSAPI auth resumed
> [01/Jun/2016:18:28:04 -0400] slapd_ldap_sasl_interactive_bind - Error:
> could not perform interactive bind for id [] mech [GSSAPI]: LDAP error
> 49 (Invalid credentials) (SASL(-13): authentication failure: GSSAPI
> Failure: gss_accept_sec_context) errno 0 (Success)
> [01/Jun/2016:18:28:04 -0400] slapi_ldap_bind - Error: could not perform
> interactive bind for id [] mech [GSSAPI]: error 49 (Invalid credentials)
> [01/Jun/2016:18:28:13 -0400] slapd_ldap_sasl_interactive_bind - Error:
> could not perform interactive bind for id [] mech [GSSAPI]: LDAP error
> -2 (Local error) (SASL(-1): generic failure: GSSAPI Error: Unspecified
> GSS failure. Minor code may provide more information (No credentials
> cache found)) errno 2 (No such file or directory)
> [01/Jun/2016:18:28:13 -0400] slapi_ldap_bind - Error: could not perform
> interactive bind for id [] mech [GSSAPI]: error -2 (Local error)
> [01/Jun/2016:18:33:03 -0400] slapd_ldap_sasl_interactive_bind - Error:
> could not perform interactive bind for id [] mech [GSSAPI]: LDAP error
> 49 (Invalid credentials) (SASL(-13): authentication failure: GSSAPI
> Failure: gss_accept_sec_context) errno 0 (Success)
> [01/Jun/2016:18:33:03 -0400] slapi_ldap_bind - Error: could not perform
> interactive bind for id [] mech [GSSAPI]: error 49 (Invalid credentials)
> [01/Jun/2016:18:33:18 -0400] slapd_ldap_sasl_interactive_bind - Error:
> could not perform interactive bind for id [] mech [GSSAPI]: LDAP error
> -2 (Local error) (SASL(-1): generic failure: GSSAPI Error: Unspecified
> GSS failure. Minor code may provide more information (No credentials
> cache found)) errno 2 (No such file or directory)
> [01/Jun/2016:18:33:18 -0400] slapi_ldap_bind - Error: could not perform
> interactive bind for id [] mech [GSSAPI]: error -2 (Local error)
> [01/Jun/2016:18:38:03 -0400] slapd_ldap_sasl_interactive_bind - Error:
> could not perform interactive bind for id [] mech [GSSAPI]: LDAP error
> 49 (Invalid credentials) (SASL(-13): authentication failure: GSSAPI
> Failure: gss_accept_sec_context) errno 0 (Success)

And this makes it look like it can't talk to the KDC.

I'd check for SELinux errors, ausearch -m AVC -ts recent

I think the rest is just indication that something is wrong with either
the LDAP servers, the KDC or both.

You may also want to look at /var/log/ipaupgrade.log to ensure that the
upgrade was successful.

rob

>
> [God at FirstMasterIPA slapd-DOMAIN-LOCAL]# ipa-replica-manage -v list
> --------------> just hangs and never returns
>
>
> [God at FirstMasterIPA slapd-DOMAIN-LOCAL]# ipactl start ------------->Just
> hangs here as well.. never gets to the KDC.
>
> Starting Directory Service
> Starting dirsrv:
> PKI-IPA... already running [ OK ]
> DOMAIN-LOCAL... already running [ OK ]
>
>
> If I run nslookup it fails over to a Replica for the DNS resolution
> instead of resolving ips itself.
>
>
>
> PKI log shows a bunch of this:
> [02/Jun/2016:11:15:25 -0400] NSMMReplicationPlugin -
> agmt="cn=masterAgreement1-ipaserver2.domain.local-pki-ca"
> (ipaserver2:7389): Replication bind with SIMPLE auth failed: LDAP error
> -1 (Can't contact LDAP server) ((null))
> [02/Jun/2016:11:15:34 -0400] NSMMReplicationPlugin -
> agmt="cn=masterAgreement1-ipaserver2.domain.local-pki-ca"
> (ipaserver2:7389): Replication bind with SIMPLE auth resumed
> [02/Jun/2016:11:16:36 -0400] slapi_ldap_bind - Error: could not send
> startTLS request: error -1 (Can't contact LDAP server) errno 0 (Success)
> [02/Jun/2016:11:16:51 -0400] slapi_ldap_bind - Error: could not send
> startTLS request: error -1 (Can't contact LDAP server) errno 107
> (Transport endpoint is not connected)
> [02/Jun/2016:11:21:51 -0400] slapi_ldap_bind - Error: could not send
> startTLS request: error -1 (Can't contact LDAP server) errno 0 (Success)
> [02/Jun/2016:11:22:06 -0400] slapi_ldap_bind - Error: could not send
> startTLS request: error -1 (Can't contact LDAP server) errno 107
> (Transport endpoint is not connected)
> [02/Jun/2016:11:26:36 -0400] slapi_ldap_bind - Error: could not send
> startTLS request: error -1 (Can't contact LDAP server) errno 0 (Success)
> [02/Jun/2016:11:26:41 -0400] slapi_ldap_bind - Error: could not send
> startTLS request: error -1 (Can't contact LDAP server) errno 107
> (Transport endpoint is not connected)
> [02/Jun/2016:11:31:36 -0400] slapi_ldap_bind - Error: could not send
> startTLS request: error -1 (Can't contact LDAP server) errno 0 (Success)
> [02/Jun/2016:11:31:41 -0400] slapi_ldap_bind - Error: could not send
> startTLS request: error -1 (Can't contact LDAP server) errno 107
> (Transport endpoint is not connected)
> [02/Jun/2016:11:36:36 -0400] slapi_ldap_bind - Error: could not send
> startTLS request: error -1 (Can't contact LDAP server) errno 0 (Success)
> [02/Jun/2016:11:36:41 -0400] slapi_ldap_bind - Error: could not send
> startTLS request: error -1 (Can't contact LDAP server) errno 107
> (Transport endpoint is not connected)
> [02/Jun/2016:11:41:46 -0400] slapi_ldap_bind - Error: could not send
> startTLS request: error -1 (Can't contact LDAP server) errno 0 (Success)
> [02/Jun/2016:11:41:51 -0400] slapi_ldap_bind - Error: could not send
> startTLS request: error -1 (Can't contact LDAP server) errno 107
> (Transport endpoint is not connected)
> [02/Jun/2016:11:45:16 -0400] slapi_ldap_bind - Error: could not send
> startTLS request: error -1 (Can't contact LDAP server) errno 0 (Success)
> [02/Jun/2016:11:45:16 -0400] NSMMReplicationPlugin -
> agmt="cn=masterAgreement1-ipaserver3.domain.local-pki-ca"
> (ipaserver3:7389): Replication bind with SIMPLE auth failed: LDAP error
> -1 (Can't contact LDAP server) ((null))
> [02/Jun/2016:11:45:25 -0400] NSMMReplicationPlugin -
> agmt="cn=masterAgreement1-ipaserver3.domain.local-pki-ca"
> (ipaserver3:7389): Replication bind with SIMPLE auth resumed
> [02/Jun/2016:11:46:51 -0400] slapi_ldap_bind - Error: could not send
> startTLS request: error -1 (Can't contact LDAP server) errno 0 (Success)
> [02/Jun/2016:11:46:56 -0400] slapi_ldap_bind - Error: could not send
> startTLS request: error -1 (Can't contact LDAP server) errno 107
> (Transport endpoint is not connected)
> [02/Jun/2016:11:51:36 -0400] slapi_ldap_bind - Error: could not send
> startTLS request: error -1 (Can't contact LDAP server) errno 0 (Success)
> [02/Jun/2016:11:51:41 -0400] slapi_ldap_bind - Error: could not send
> startTLS request: error -1 (Can't contact LDAP server) errno 107
> (Transport endpoint is not connected)
> [02/Jun/2016:11:56:46 -0400] slapi_ldap_bind - Error: could not send
> startTLS request: error -1 (Can't contact LDAP server) errno 0 (Success)
> [02/Jun/2016:11:56:51 -0400] slapi_ldap_bind - Error: could not send
> startTLS request: error -1 (Can't contact LDAP server) errno 107
> (Transport endpoint is not connected)
> [02/Jun/2016:12:01:36 -0400] slapi_ldap_bind - Error: could not send
> startTLS request: error -1 (Can't contact LDAP server) errno 0 (Success)
> [02/Jun/2016:12:01:41 -0400] slapi_ldap_bind - Error: could not send
> startTLS request: error -1 (Can't contact LDAP server) errno 107
> (Transport endpoint is not connected)
> [02/Jun/2016:12:05:33 -0400] slapi_ldap_bind - Error: could not send
> startTLS request: error -1 (Can't contact LDAP server) errno 0 (Success)
> [02/Jun/2016:12:05:33 -0400] NSMMReplicationPlugin -
> agmt="cn=masterAgreement1-ipaserver3.domain.local-pki-ca"
> (ipaserver3:7389): Replication bind with SIMPLE auth failed: LDAP error
> -1 (Can't contact LDAP server) ((null))
> [02/Jun/2016:12:06:01 -0400] slapi_ldap_bind - Error: could not send
> startTLS request: error -1 (Can't contact LDAP server) errno 0 (Success)
> [02/Jun/2016:12:06:06 -0400] NSMMReplicationPlugin -
> agmt="cn=masterAgreement1-ipaserver3.domain.local-pki-ca"
> (ipaserver3:7389): Replication bind with SIMPLE auth resumed
> [02/Jun/2016:12:06:31 -0400] slapi_ldap_bind - Error: could not send
> startTLS request: error -1 (Can't contact LDAP server) errno 107
> (Transport endpoint is not connected)
> [02/Jun/2016:12:06:41 -0400] slapi_ldap_bind - Error: could not send
> startTLS request: error -1 (Can't contact LDAP server) errno 0 (Success)
>
>
>
>
> NTP seems OK
> [God at FirstMasterIPA slapd-PKI-IPA]# date
> Thu Jun 2 12:23:00 EDT 2016
>
> [God at ipaserver3 ~]# date
> Thu Jun 2 12:23:02 EDT 2016
>
>
>
> Sean Hogan
>
>
>
>
>

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/freeipa-users/attachments/20160603/9627326b/attachment.htm>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: graycol.gif
Type: image/gif
Size: 105 bytes
Desc: not available
URL: <http://listman.redhat.com/archives/freeipa-users/attachments/20160603/9627326b/attachment.gif>