[Freeipa-users] IPA 3.0.47 to 3.0.50 Upgrade problem

Wed Jun 29 07:50:21 UTC 2016

On 28.6.2016 20:21, Sean Hogan wrote:
> Thanks Petr,
> 
>   Since the last recycle of the Host hosting the First Master it has been
> stable for about a week now.  Only thing I did was to spread out my
> replication agreements.  I had 8 replications hitting it but now have 4
> going to it and the other 4 to its backup replica with the first master and
> the backup replica having an agreement.
> 
> 
> Not sure that fixed it or not but it seems to be stable at this point and I
> know the docs say no more than 4 replications agreements so maybe it was
> the cause.

Generally more replication agreements mean more load on the server. Many
replication agreements should not cause problems by itself if the server has
sufficient performance.

Petr^2 Spacek

> Sean Hogan
> 
> 
> 
> 
> 
> 
> 
> From:	Petr Spacek <pspacek at redhat.com>
> To:	Sean Hogan/Durham/IBM at IBMUS
> Cc:	freeipa-users at redhat.com
> Date:	06/28/2016 10:24 AM
> Subject:	Re: [Freeipa-users] IPA 3.0.47 to 3.0.50 Upgrade problem
> 
> 
> 
> On 22.6.2016 23:09, Sean Hogan wrote:
>> SLAPD showing
>>
>> 22/Jun/2016:17:01:59 -0400] slapi_ldap_bind - Error: could not perform
>> interactive bind for id [] mech [GSSAPI]: error 49 (Invalid credentials)
>> [22/Jun/2016:17:06:59 -0400] slapd_ldap_sasl_interactive_bind - Error:
>> could not perform interactive bind for id [] mech [GSSAPI]: LDAP error 49
>> (Invalid credentials) (SASL(-13): authentication failure: GSSAPI Failure:
>> gss_accept_sec_context) errno 0 (Success)
>>
>>
>> where would these creds be and what ID?  I am using SASL so I assume it
> to
>> be sasl_user DNS/FirstMaster.watson.local  or something like that?
> 
> These are in /etc/dirsrv/ds.keytab.
> 
> I would start with
> # klist -kt /etc/dirsrv/ds.keytab
> and try to proceed with kinit etc. (very similarly to the bind-dyndb-ldap
> how-to).
> 
> I hope it helps.
> 
> Petr^2 Spacek
> 
> 
>> From:		 Sean Hogan/Durham/IBM at IBMUS
>> To:		 Petr Spacek <pspacek at redhat.com>
>> Cc:		 freeipa-users at redhat.com
>> Date:		 06/22/2016 08:36 AM
>> Subject:		 Re: [Freeipa-users] IPA 3.0.47 to 3.0.50 Upgrade problem
>> Sent by:		 freeipa-users-bounces at redhat.com
>>
>>
>>
>> Hi Peter...
>>
>> Yes..... this has me doing loops in my head to /dev/null
>>
>> You are correct I could not complete the BIND steps... I did them
> yesterday
>> but did not post results as I wanted to stop bugging you all :)
>> The initial credential section of that I could not complete nor can I get
>> an keytab without it and I don't think I have an issue with cert versions
>> (used the SASL section). The upgrade log from 3.47 to 3.50 on this one
>> server did show an error with named though.
>>
>> I had the box powered down again last night after testing the BIND
>> procedures... and its been up since then. Which makes we really not sure
>> what is going on(DNS DOS from internal maybe? I get a lot of outside
>> requests showing network unreachable and I don't forward to a outside
> DNS).
>> If it was a password/cert/cipher/file perm issue then I don't see how it
>> can work at all after a reboot.
>>
>> I am thinking it needs a rebuild.. I have not done this on a First Master
>> IPA is there anything I need to be take into consider with it being first
>> master? Right now I have 8 IPAs all DNS, NTP and CAs on differ vlans but
>> the first master is the fail back IPA(on the only vlan that can talk to
> the
>> others) in case there local vlan IPA dies. First Master is also the
> master
>> CA in the realm where everything is enrolled to originally. We then mod
>> everything to point to the vlan IPA with the Firstmaster as secondary
> with
>> our vlan-specific scripts we run after ipa client install.
>>
>> With the box rebooted last night I am now getting normal functionality
> but
>> it prob wont last long as indicated from the past...
>>
>> Working
>> [bob at FirstMaster ~]# kinit admin
>> Password for admin at DOMAIN.LOCAL:
>> Warning: Your password will expire in 6 days on Tue Jun 28 14:55:52 2016
>> [bob at FirstMaster ~]#
>>
>> I did post ldap logs in my first email though... will readd them to this
>> and when it dies off again I will add more.
>>
>>
>>> [20/Jun/2016:13:59:00 -0400] - Detected Disorderly Shutdown last time
>>> Directory Server was running, recovering database.
>>> [20/Jun/2016:13:59:01 -0400] schema-compat-plugin - warning: no entries
>> set
>>> up under cn=computers, cn=compat,dc=domain,dc=local
>>> [20/Jun/2016:13:59:48 -0400] NSMMReplicationPlugin - ruv_compare_ruv:
> RUV
>>> [database RUV] does not contain element [{replica 7}
> 55ca26a0000900070000
>>> 5688d8e6001000070000] which is present in RUV [changelog max RUV]
>>> [20/Jun/2016:13:59:48 -0400] NSMMReplicationPlugin -
>>> replica_check_for_data_reload: Warning: for replica dc=domain,dc=local
>>> there were some differences between the changelog max RUV and the
>> database
>>> RUV. If there are obsolete elements in the database RUV, you should
>> remove
>>> them using the CLEANALLRUV task. If they are not obsolete, you should
>> check
>>> their status to see why there are no changes from those servers in the
>>> changelog.
>>> [20/Jun/2016:13:59:48 -0400] set_krb5_creds - Could not get initial
>>> credentials for principal [ldap/server1.domain.local at DOMAIN.LOCAL] in
>>> keytab [FILE:/etc/dirsrv/ds.keytab]: -1765328228 (Cannot contact any KDC
>>> for requested realm)
>>> [20/Jun/2016:13:59:48 -0400] set_krb5_creds - Could not get initial
>>> credentials for principal [ldap/server1.domain.local at DOMAIN.LOCAL] in
>>> keytab [FILE:/etc/dirsrv/ds.keytab]: -1765328228 (Cannot contact any KDC
>>> for requested realm)
>>> [20/Jun/2016:13:59:48 -0400] set_krb5_creds - Could not get initial
>>> credentials for principal [ldap/server1.domain.local at DOMAIN.LOCAL] in
>>> keytab [FILE:/etc/dirsrv/ds.keytab]: -1765328228 (Cannot contact any KDC
>>> for requested realm)
>>> [20/Jun/2016:13:59:48 -0400] slapd_ldap_sasl_interactive_bind - Error:
>>> could not perform interactive bind for id [] mech [GSSAPI]: LDAP error
> -2
>>> (Local error) (SASL(-1): generic failure: GSSAPI Error: Unspecified GSS
>>> failure. Minor code may provide more information (Credentials cache file
>>> '/tmp/krb5cc_495' not found)) errno 2 (No such file or directory)
>>> [20/Jun/2016:13:59:48 -0400] slapi_ldap_bind - Error: could not perform
>>> interactive bind for id [] mech [GSSAPI]: error -2 (Local error)
>>> [20/Jun/2016:13:59:48 -0400] slapd_ldap_sasl_interactive_bind - Error:
>>> could not perform interactive bind for id [] mech [GSSAPI]: LDAP error
> -2
>>> (Local error) (SASL(-1): generic failure: GSSAPI Error: Unspecified GSS
>>> failure. Minor code may provide more information (Credentials cache file
>>> '/tmp/krb5cc_495' not found)) errno 2 (No such file or directory)
>>> [20/Jun/2016:13:59:48 -0400] slapi_ldap_bind - Error: could not perform
>>> interactive bind for id [] mech [GSSAPI]: error -2 (Local error)
>>> [20/Jun/2016:13:59:48 -0400] NSMMReplicationPlugin -
>>> agmt="cn=meToserver9.domain.local" (server9:389): Replication bind with
>>> GSSAPI auth failed: LDAP error -2 (Local error) (SASL(-1): generic
>> failure:
>>> GSSAPI Error: Unspecified GSS failure. Minor code may provide more
>>> information (Credentials cache file '/tmp/krb5cc_495' not found))
>>> [20/Jun/2016:13:59:48 -0400] set_krb5_creds - Could not get initial
>>> credentials for principal [ldap/server1.domain.local at DOMAIN.LOCAL] in
>>> keytab [FILE:/etc/dirsrv/ds.keytab]: -1765328228 (Cannot contact any KDC
>>> for requested realm)
>>> [20/Jun/2016:13:59:48 -0400] NSMMReplicationPlugin -
>>> agmt="cn=meToserver8.domain.local" (server8:389): Replication bind with
>>> GSSAPI auth failed: LDAP error -2 (Local error) (SASL(-1): generic
>> failure:
>>> GSSAPI Error: Unspecified GSS failure. Minor code may provide more
>>> information (Credentials cache file '/tmp/krb5cc_495' not found))
>>> [20/Jun/2016:13:59:48 -0400] slapd_ldap_sasl_interactive_bind - Error:
>>> could not perform interactive bind for id [] mech [GSSAPI]: LDAP error
> -2
>>> (Local error) (SASL(-1): generic failure: GSSAPI Error: Unspecified GSS
>>> failure. Minor code may provide more information (Credentials cache file
>>> '/tmp/krb5cc_495' not found)) errno 2 (No such file or directory)
>>> [20/Jun/2016:13:59:48 -0400] slapi_ldap_bind - Error: could not perform
>>> interactive bind for id [] mech [GSSAPI]: error -2 (Local error)
>>> [20/Jun/2016:13:59:48 -0400] NSMMReplicationPlugin -
>>> agmt="cn=meTobldvxl0011.domain.local" (1server:389): Replication bind
>> with
>>> GSSAPI auth failed: LDAP error -2 (Local error) (SASL(-1): generic
>> failure:
>>> GSSAPI Error: Unspecified GSS failure. Minor code may provide more
>>> information (Credentials cache file '/tmp/krb5cc_495' not found))
>>> [20/Jun/2016:13:59:48 -0400] slapd_ldap_sasl_interactive_bind - Error:
>>> could not perform interactive bind for id [] mech [GSSAPI]: LDAP error
> -2
>>> (Local error) (SASL(-1): generic failure: GSSAPI Error: Unspecified GSS
>>> failure. Minor code may provide more information (Credentials cache file
>>> '/tmp/krb5cc_495' not found)) errno 2 (No such file or directory)
>>> [20/Jun/2016:13:59:48 -0400] slapi_ldap_bind - Error: could not perform
>>> interactive bind for id [] mech [GSSAPI]: error -2 (Local error)
>>> [20/Jun/2016:13:59:48 -0400] NSMMReplicationPlugin -
>>> agmt="cn=meToserver3.domain.local" (server3:389): Replication bind with
>>> GSSAPI auth failed: LDAP error -2 (Local error) (SASL(-1): generic
>> failure:
>>> GSSAPI Error: Unspecified GSS failure. Minor code may provide more
>>> information (Credentials cache file '/tmp/krb5cc_495' not found))
>>> [20/Jun/2016:13:59:48 -0400] set_krb5_creds - Could not get initial
>>> credentials for principal [ldap/server1.domain.local at DOMAIN.LOCAL] in
>>> keytab [FILE:/etc/dirsrv/ds.keytab]: -1765328228 (Cannot contact any KDC
>>> for requested realm)
>>> [20/Jun/2016:13:59:48 -0400] set_krb5_creds - Could not get initial
>>> credentials for principal [ldap/server1.domain.local at DOMAIN.LOCAL] in
>>> keytab [FILE:/etc/dirsrv/ds.keytab]: -1765328228 (Cannot contact any KDC
>>> for requested realm)
>>> [20/Jun/2016:13:59:48 -0400] slapd_ldap_sasl_interactive_bind - Error:
>>> could not perform interactive bind for id [] mech [GSSAPI]: LDAP error
> -2
>>> (Local error) (SASL(-1): generic failure: GSSAPI Error: Unspecified GSS
>>> failure. Minor code may provide more information (Credentials cache file
>>> '/tmp/krb5cc_495' not found)) errno 2 (No such file or directory)
>>> [20/Jun/2016:13:59:48 -0400] slapi_ldap_bind - Error: could not perform
>>> interactive bind for id [] mech [GSSAPI]: error -2 (Local error)
>>> [20/Jun/2016:13:59:48 -0400] NSMMReplicationPlugin -
>>> agmt="cn=meToserver7.domain.local" (server7:389): Replication bind with
>>> GSSAPI auth failed: LDAP error -2 (Local error) (SASL(-1): generic
>> failure:
>>> GSSAPI Error: Unspecified GSS failure. Minor code may provide more
>>> information (Credentials cache file '/tmp/krb5cc_495' not found))
>>> [20/Jun/2016:13:59:48 -0400] - slapd started. Listening on All
> Interfaces
>>> port 389 for LDAP requests
>>> [20/Jun/2016:13:59:48 -0400] - Listening on All Interfaces port 636 for
>>> LDAPS requests
>>> [20/Jun/2016:13:59:48 -0400] - Listening
>>> on /var/run/slapd-DOMAIN-LOCAL.socket for LDAPI requests
>>> [20/Jun/2016:13:59:48 -0400] slapd_ldap_sasl_interactive_bind - Error:
>>> could not perform interactive bind for id [] mech [GSSAPI]: LDAP error
> -2
>>> (Local error) (SASL(-1): generic failure: GSSAPI Error: Unspecified GSS
>>> failure. Minor code may provide more information (Credentials cache file
>>> '/tmp/krb5cc_495' not found)) errno 2 (No such file or directory)
>>> [20/Jun/2016:13:59:48 -0400] slapi_ldap_bind - Error: could not perform
>>> interactive bind for id [] mech [GSSAPI]: error -2 (Local error)
>>> [20/Jun/2016:13:59:48 -0400] NSMMReplicationPlugin -
>>> agmt="cn=meToserver4.domain.local" (server4:389): Replication bind with
>>> GSSAPI auth failed: LDAP error -2 (Local error) (SASL(-1): generic
>> failure:
>>> GSSAPI Error: Unspecified GSS failure. Minor code may provide more
>>> information (Credentials cache file '/tmp/krb5cc_495' not found))
>>> [20/Jun/2016:13:59:48 -0400] set_krb5_creds - Could not get initial
>>> credentials for principal [ldap/server1.domain.local at DOMAIN.LOCAL] in
>>> keytab [FILE:/etc/dirsrv/ds.keytab]: -1765328228 (Cannot contact any KDC
>>> for requested realm)
>>> [20/Jun/2016:13:59:48 -0400] slapd_ldap_sasl_interactive_bind - Error:
>>> could not perform interactive bind for id [] mech [GSSAPI]: LDAP error
> -2
>>> (Local error) (SASL(-1): generic failure: GSSAPI Error: Unspecified GSS
>>> failure. Minor code may provide more information (Credentials cache file
>>> '/tmp/krb5cc_495' not found)) errno 0 (Success)
>>> [20/Jun/2016:13:59:48 -0400] slapi_ldap_bind - Error: could not perform
>>> interactive bind for id [] mech [GSSAPI]: error -2 (Local error)
>>> [20/Jun/2016:13:59:48 -0400] NSMMReplicationPlugin -
>>> agmt="cn=meToserver2.domain.local" (server2:389): Replication bind with
>>> GSSAPI auth failed: LDAP error -2 (Local error) (SASL(-1): generic
>> failure:
>>> GSSAPI Error: Unspecified GSS failure. Minor code may provide more
>>> information (Credentials cache file '/tmp/krb5cc_495' not found))
>>> [20/Jun/2016:13:59:51 -0400] NSMMReplicationPlugin -
>>> agmt="cn=meToserver8.domain.local" (server8:389): Replication bind with
>>> GSSAPI auth resumed
>>> [20/Jun/2016:13:59:51 -0400] slapd_ldap_sasl_interactive_bind - Error:
>>> could not perform interactive bind for id [] mech [GSSAPI]: LDAP error
> 49
>>> (Invalid credentials) (SASL(-13): authentication failure: GSSAPI
> Failure:
>>> gss_accept_sec_context) errno 0 (Success)
>>> [20/Jun/2016:13:59:51 -0400] slapi_ldap_bind - Error: could not perform
>>> interactive bind for id [] mech [GSSAPI]: error 49 (Invalid credentials)
>>> [20/Jun/2016:13:59:51 -0400] NSMMReplicationPlugin -
>>> agmt="cn=meToserver3.domain.local" (server3:389): Replication bind with
>>> GSSAPI auth failed: LDAP error 49 (Invalid credentials) (SASL(-13):
>>> authentication failure: GSSAPI Failure: gss_accept_sec_context)
>>> [20/Jun/2016:13:59:51 -0400] slapd_ldap_sasl_interactive_bind - Error:
>>> could not perform interactive bind for id [] mech [GSSAPI]: LDAP error
> -2
>>> (Local error) (SASL(-1): generic failure: GSSAPI Error: Unspecified GSS
>>> failure. Minor code may provide more information (No credentials cache
>>> found)) errno 2 (No such file or directory)
>>> [20/Jun/2016:13:59:51 -0400] slapi_ldap_bind - Error: could not perform
>>> interactive bind for id [] mech [GSSAPI]: error -2 (Local error)
>>> [20/Jun/2016:13:59:51 -0400] slapd_ldap_sasl_interactive_bind - Error:
>>> could not perform interactive bind for id [] mech [GSSAPI]: LDAP error
> -2
>>> (Local error) (SASL(-1): generic failure: GSSAPI Error: Unspecified GSS
>>> failure. Minor code may provide more information (No credentials cache
>>> found)) errno 2 (No such file or directory)
>>> [20/Jun/2016:13:59:51 -0400] slapi_ldap_bind - Error: could not perform
>>> interactive bind for id [] mech [GSSAPI]: error -2 (Local error)
>>> [20/Jun/2016:13:59:51 -0400] slapd_ldap_sasl_interactive_bind - Error:
>>> could not perform interactive bind for id [] mech [GSSAPI]: LDAP error
> -2
>>> (Local error) (SASL(-1): generic failure: GSSAPI Error: Unspecified GSS
>>> failure. Minor code may provide more information (No credentials cache
>>> found)) errno 2 (No such file or directory)
>>> [20/Jun/2016:13:59:51 -0400] slapi_ldap_bind - Error: could not perform
>>> interactive bind for id [] mech [GSSAPI]: error -2 (Local error)
>>> [20/Jun/2016:13:59:51 -0400] slapd_ldap_sasl_interactive_bind - Error:
>>> could not perform interactive bind for id [] mech [GSSAPI]: LDAP error
> -2
>>> (Local error) (SASL(-1): generic failure: GSSAPI Error: Unspecified GSS
>>> failure. Minor code may provide more information (No credentials cache
>>> found)) errno 2 (No such file or directory)
>>> [20/Jun/2016:13:59:51 -0400] slapi_ldap_bind - Error: could not perform
>>> interactive bind for id [] mech [GSSAPI]: error -2 (Local error)
>>> [20/Jun/2016:13:59:51 -0400] slapd_ldap_sasl_interactive_bind - Error:
>>> could not perform interactive bind for id [] mech [GSSAPI]: LDAP error
> -2
>>> (Local error) (SASL(-1): generic failure: GSSAPI Error: Unspecified GSS
>>> failure. Minor code may provide more information (No credentials cache
>>> found)) errno 2 (No such file or directory)
>>> [20/Jun/2016:13:59:51 -0400] slapi_ldap_bind - Error: could not perform
>>> interactive bind for id [] mech [GSSAPI]: error -2 (Local error)
>>> [20/Jun/2016:13:59:57 -0400] slapd_ldap_sasl_interactive_bind - Error:
>>> could not perform interactive bind for id [] mech [GSSAPI]: LDAP error
> 49
>>> (Invalid credentials) (SASL(-13): authentication failure: GSSAPI
> Failure:
>>> gss_accept_sec_context) errno 0 (Success)
>>> [20/Jun/2016:13:59:57 -0400] slapi_ldap_bind - Error: could not perform
>>> interactive bind for id [] mech [GSSAPI]: error 49 (Invalid credentials)
>>> [20/Jun/2016:13:59:57 -0400] NSMMReplicationPlugin -
>>> agmt="cn=meToserver2.domain.local" (server2:389): Replication bind with
>>> GSSAPI auth resumed
>>
>>
>>
>> Sean Hogan
>>
>>
>>
>>
>>
>> Inactive hide details for Petr Spacek ---06/21/2016 10:20:43 PM---On
>> 22.6.2016 02:56, Sean Hogan wrote: > More infoPetr Spacek ---06/21/2016
>> 10:20:43 PM---On 22.6.2016 02:56, Sean Hogan wrote: > More info
>>
>> From: Petr Spacek <pspacek at redhat.com>
>> To: freeipa-users at redhat.com
>> Date: 06/21/2016 10:20 PM
>> Subject: Re: [Freeipa-users] IPA 3.0.47 to 3.0.50 Upgrade problem
>> Sent by: freeipa-users-bounces at redhat.com
>>
>>
>>
>> On 22.6.2016 02:56, Sean Hogan wrote:
>>> More info
>>>
>>>
>>> Krb5 log is showing:
>>> Jun 21 20:42:47 Firstmaster.domain.local krb5kdc[2141](info): AS_REQ (4
>>> etypes {18 17 16 23}) 10.x.x.x: LOOKING_UP_CLIENT: admin at domain.LOCAL
> for
>>> krbtgt/DOMAIN.LOCAL at DOMAIN.LOCAL, Server error
>>
>>
>> Hello,
>>
>> this is really fishy. I would bet that there is a problem with LDAP
> server
>> and
>> DNS errors are just consequence of it.
>>
>> I suspect that you will not be able to finish steps mentioned in
>>
> https://fedorahosted.org/bind-dyndb-ldap/wiki/BIND9/NamedCannotStart#a3.FailedtoinitcredentialsorFailedtogetinitialcredentialsDecryptintegritycheckfailedorClientscredentialshavebeenrevoked
> 
>>
>>
>> If it is the case I would turn your attention to krb5kdc.log and LDAP
>> server
>> logs in /var/log/dirsrv/*
>>
>> There must be something wrong with the LDAP server.
>>
>> Petr^2 Spacek
>>
>>
>>>
>>> [bob at Firstmaster etc]# kinit -v admin
>>> kinit: Credentials cache file '/tmp/krb5cc_0' not found while validating
>>> credentials
>>>
>>>
>>>
>>>
>>>
>>>
>>> Sean Hogan
>>>
>>>
>>>
>>>
>>>
>>>
>>> From: Sean Hogan/Durham/IBM
>>> To: freeipa-users <freeipa-users at redhat.com>
>>> Date: 06/21/2016 12:02 PM
>>> Subject: Re: [Freeipa-users] IPA 3.0.47 to 3.0.50 Upgrade problem
>>>
>>>
>>>   Has anyone seen these before?
>>>
>>>
>>>
>>> First Master IPA DNS logs show:   Looks like the host names are getting
>> the
>>> domain twice domain.local.domain.local
>>>
>>>
>>> client 10.x.x.x#58094: query failed (SERVFAIL) for
>>> server1.domain.local.domain.local/IN/AAAA at query.c:6569
>>> timeout in ldap_pool_getconnection(): try to raise 'connections'
>> parameter;
>>> potential deadlock?
>>> client 10.x.x.x#44147: query failed (SERVFAIL) for
>>> x.x.x.10.in-addr.arpa/IN/PTR at query.c:6569
>>> timeout in ldap_pool_getconnection(): try to raise 'connections'
>> parameter;
>>> potential deadlock?
>>> client 10.x.x.x#56466: query failed (SERVFAIL) for
>>> x.x.x.10.in-addr.arpa/IN/PTR at query.c:6569
>>> timeout in ldap_pool_getconnection(): try to raise 'connections'
>> parameter;
>>> potential deadlock?
>>> client 10.x.x.x53367: query failed (SERVFAIL) for
>>> server2.domain.local.domain.local/IN/A at query.c:6569
>>> timeout in ldap_pool_getconnection(): try to raise 'connections'
>> parameter;
>>> potential deadlock?
>>> client 10.x.x.x#53367: query failed (SERVFAIL) for
>>> server2.domain.local.domain.local/IN/AAAA at query.c:6569
>>>
>>>
>>>
>>> So enrolls are failing at this point when tyring to enroll to a replica:
>>>
>>> [bob at server1 log]# ipa-client-install –enable-dns-updates
>>> Discovery was successful!
>>> Hostname: server1.watson.local
>>> Realm: DOMAIN.LOCAL
>>> DNS Domain: domain.local
>>> IPA Server: ipareplica.domain.local
>>> BaseDN: dc=domain,dc=local
>>>
>>> Continue to configure the system with these values? [no]: yes
>>> User authorized to enroll computers: bob
>>> Synchronizing time with KDC...
>>> Password for bob at DOMAIN.LOCAL:
>>> Successfully retrieved CA cert
>>>     Subject:     CN=Certificate Authority,O=DOMAIN.LOCAL
>>>     Issuer:      CN=Certificate Authority,O=DOMAIN.LOCAL
>>>     Valid From:  Tue Jan 06 19:37:09 2015 UTC
>>>     Valid Until: Sat Jan 06 19:37:09 2035 UTC
>>>
>>> Enrolled in IPA realm DOMAIN.LOCAL
>>> Attempting to get host TGT...
>>> Created /etc/ipa/default.conf
>>> New SSSD config will be created
>>> Configured sudoers in /etc/nsswitch.conf
>>> Configured /etc/sssd/sssd.conf
>>> Configured /etc/krb5.conf for IPA realm DOMAIN.LOCAL
>>> trying https://ipareplica.domain.local/ipa/xml
>>> Cannot connect to the server due to Kerberos error: Kerberos error:
>>> Kerberos error: ('Unspecified GSS failure.  Minor code may provide more
>>> information', 851968)/('KDC returned error string: PROCESS_TGS',
>>> -1765328324)/. Trying with delegate=True
>>> trying https://ipareplica.domain.local/ipa/xml
>>> Second connect with delegate=True also failed: Kerberos error: Kerberos
>>> error: ('Unspecified GSS failure.  Minor code may provide more
>>> information', 851968)/('KDC returned error string: PROCESS_TGS',
>>> -1765328324)/
>>> Cannot connect to the IPA server XML-RPC interface: Kerberos error:
>>> Kerberos error: ('Unspecified GSS failure.  Minor code may provide more
>>> information', 851968)/('KDC returned error string: PROCESS_TGS',
>>> -1765328324)/
>>> Installation failed. Rolling back changes.
>>> Unenrolling client from IPA server
>>> Unenrolling host failed: Error obtaining initial credentials: Generic
>> error
>>> (see e-text).
>>>
>>> Removing Kerberos service principals from /etc/krb5.keytab
>>> Disabling client Kerberos and LDAP configurations
>>> Redundant SSSD configuration file /etc/sssd/sssd.conf was moved
>>> to /etc/sssd/sssd.conf.deleted
>>> Restoring client configuration files
>>> nscd daemon is not installed, skip configuration
>>> nslcd daemon is not installed, skip configuration
>>> Client uninstall complete.
>>>
>>>
>>> Sean Hogan
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>> From: Sean Hogan/Durham/IBM
>>> To: Sean Hogan/Durham/IBM at IBMUS
>>> Cc: freeipa-users <freeipa-users at redhat.com>
>>> Date: 06/20/2016 12:49 PM
>>> Subject: Re: [Freeipa-users] IPA 3.0.47 to 3.0.50 Upgrade problem
>>>
>>>
>>> Also seeing this in the upgrade log on the first master but not on the 7
>>> ipas.
>>>
>>> ERROR Failed to restart named: Command '/sbin/service named restart '
>>> returned non-zero exit status 7
>>>
>>>
>>> which led me to
>>>
>>> https://bugzilla.redhat.com/show_bug.cgi?id=895298
>>>
>>>
>>>
>>>
>>>
>>> Sean Hogan
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>> From: Sean Hogan/Durham/IBM at IBMUS
>>> To: freeipa-users <freeipa-users at redhat.com>
>>> Date: 06/20/2016 11:46 AM
>>> Subject: Re: [Freeipa-users] IPA 3.0.47 to 3.0.50 Upgrade problem
>>> Sent by: freeipa-users-bounces at redhat.com
>>>
>>>
>>>
>>> Hi All..
>>>
>>> I thought we fixed this issue by rebooting the KVM host but it is
> showing
>>> again. Our First Master IPA is being rebooted 2 -5 times a day now just
>> to
>>> keep it alive.
>>>
>>> What we are seeing:
>>>
>>> God at FirstMaster log]# kinit admin
>>> kinit: Cannot contact any KDC for realm 'Domain.LOCAL' while getting
>>> initial credentials
>>>
>>> DNS is not working as nslookup is failing to a replica.... think once we
>>> lose DNS it all goes down hill which makes sense.
>>>
>>> [god at FirstMaster log]# ipactl stop -----> Just hangs forever.. no
>> replies..
>>> no error.. nothing
>>>
>>> I try service named stop and nothing happens
>>>
>>> I have the box hard shutdown from KVM console. Reboot it and it works
> for
>> a
>>> little while but eventually back to same behavior.
>>>
>>> At this point I can service named stop and it responds... ipactl status
>> and
>>> it responds.. but when if I try service named restart I get
>>>
>>> [god at FirstMaster log]# service named stop
>>> Stopping named: ......
>>>
>>> [god at Firstmaster log]# service named start
>>> Starting named: [FAILED]
>>>
>>> [god at FirstMaster log]# service named status
>>> rndc: connect failed: 127.0.0.1#953: connection refused
>>> named dead but pid file exists
>>>
>>> Rebooted box and it is hung on shutting down domain-local and never
> fully
>>> shuts down.. have to get it hard shutdown again.
>>> During an attempt to gracefully shut down we see this
>>>
>>> Shutting Down dirsrv:
>>> PKI-IPA OK
>>> DOMAIN-LOCAL FAILED
>>> *** Error: 1 instance(s) unsuccessfully stopped FAILED
>>>
>>> Then it moves on to shut other things down and returns to dirsrv
>>> Shutting Down dirsrv:
>>> PKI-IPA....server already stopped FAILED {Makes sense.. it died earlier}
>>> DOMAIN-LOCAL... {this sits here til we hard shutdown}
>>>
>>>
>>>
>>> bind-libs-9.8.2-0.47.rc1.el6.x86_64
>>> bind-9.8.2-0.47.rc1.el6.x86_64
>>> bind-utils-9.8.2-0.47.rc1.el6.x86_64
>>>
>>>
>>> ipa-client-3.0.0-50.el6.1.x86_64
>>> ipa-server-selinux-3.0.0-50.el6.1.x86_64
>>> ipa-server-3.0.0-50.el6.1.x86_64
>>> sssd-ipa-1.13.3-22.el6.x86_64
>>>
>>>
>>> /var/log/dirsrv/slapd-DOMAIN-LOCAL
>>> [20/Jun/2016:13:29:06 -0400] - 389-Directory/1.2.11.15 B2016.063.2110
>>> starting up
>>> [20/Jun/2016:13:29:06 -0400] schema-compat-plugin - warning: no entries
>> set
>>> up under cn=computers, cn=compat,dc=domain,dc=local
>>> [20/Jun/2016:13:29:07 -0400] NSMMReplicationPlugin - ruv_compare_ruv:
> RUV
>>> [database RUV] does not contain element [{replica 7}
> 55ca26a0000900070000
>>> 5688d8e6001000070000] which is present in RUV [changelog max RUV]
>>> [20/Jun/2016:13:29:07 -0400] NSMMReplicationPlugin -
>>> replica_check_for_data_reload: Warning: for replica dc=domain,dc=local
>>> there were some differences between the changelog max RUV and the
>> database
>>> RUV. If there are obsolete elements in the database RUV, you should
>> remove
>>> them using the CLEANALLRUV task. If they are not obsolete, you should
>> check
>>> their status to see why there are no changes from those servers in the
>>> changelog.
>>> [20/Jun/2016:13:29:07 -0400] set_krb5_creds - Could not get initial
>>> credentials for principal [ldap/server1.domain.local at DOMAIN.LOCAL] in
>>> keytab [FILE:/etc/dirsrv/ds.keytab]: -1765328228 (Cannot contact any KDC
>>> for requested realm)
>>> [20/Jun/2016:13:29:07 -0400] set_krb5_creds - Could not get initial
>>> credentials for principal [ldap/server1.domain.local at DOMAIN.LOCAL] in
>>> keytab [FILE:/etc/dirsrv/ds.keytab]: -1765328228 (Cannot contact any KDC
>>> for requested realm)
>>> [20/Jun/2016:13:29:07 -0400] set_krb5_creds - Could not get initial
>>> credentials for principal [ldap/server1.domain.local at DOMAIN.LOCAL] in
>>> keytab [FILE:/etc/dirsrv/ds.keytab]: -1765328228 (Cannot contact any KDC
>>> for requested realm)
>>> [20/Jun/2016:13:29:07 -0400] slapd_ldap_sasl_interactive_bind - Error:
>>> could not perform interactive bind for id [] mech [GSSAPI]: LDAP error
> -2
>>> (Local error) (SASL(-1): generic failure: GSSAPI Error: Unspecified GSS
>>> failure. Minor code may provide more information (Credentials cache file
>>> '/tmp/krb5cc_495' not found)) errno 2 (No such file or directory)
>>> [20/Jun/2016:13:29:07 -0400] slapi_ldap_bind - Error: could not perform
>>> interactive bind for id [] mech [GSSAPI]: error -2 (Local error)
>>> [20/Jun/2016:13:29:07 -0400] NSMMReplicationPlugin -
>>> agmt="cn=meToserver8.domain.local" (server8:389): Replication bind with
>>> GSSAPI auth failed: LDAP error -2 (Local error) (SASL(-1): generic
>> failure:
>>> GSSAPI Error: Unspecified GSS failure. Minor code may provide more
>>> information (Credentials cache file '/tmp/krb5cc_495' not found))
>>> [20/Jun/2016:13:29:07 -0400] set_krb5_creds - Could not get initial
>>> credentials for principal [ldap/server1.domain.local at DOMAIN.LOCAL] in
>>> keytab [FILE:/etc/dirsrv/ds.keytab]: -1765328228 (Cannot contact any KDC
>>> for requested realm)
>>> [20/Jun/2016:13:29:07 -0400] slapd_ldap_sasl_interactive_bind - Error:
>>> could not perform interactive bind for id [] mech [GSSAPI]: LDAP error
> -2
>>> (Local error) (SASL(-1): generic failure: GSSAPI Error: Unspecified GSS
>>> failure. Minor code may provide more information (Credentials cache file
>>> '/tmp/krb5cc_495' not found)) errno 2 (No such file or directory)
>>> [20/Jun/2016:13:29:07 -0400] slapi_ldap_bind - Error: could not perform
>>> interactive bind for id [] mech [GSSAPI]: error -2 (Local error)
>>> [20/Jun/2016:13:29:07 -0400] NSMMReplicationPlugin -
>>> agmt="cn=meToserver9.domain.local" (server9:389): Replication bind with
>>> GSSAPI auth failed: LDAP error -2 (Local error) (SASL(-1): generic
>> failure:
>>> GSSAPI Error: Unspecified GSS failure. Minor code may provide more
>>> information (Credentials cache file '/tmp/krb5cc_495' not found))
>>> [20/Jun/2016:13:29:07 -0400] set_krb5_creds - Could not get initial
>>> credentials for principal [ldap/server1.domain.local at DOMAIN.LOCAL] in
>>> keytab [FILE:/etc/dirsrv/ds.keytab]: -1765328228 (Cannot contact any KDC
>>> for requested realm)
>>> [20/Jun/2016:13:29:07 -0400] slapd_ldap_sasl_interactive_bind - Error:
>>> could not perform interactive bind for id [] mech [GSSAPI]: LDAP error
> -2
>>> (Local error) (SASL(-1): generic failure: GSSAPI Error: Unspecified GSS
>>> failure. Minor code may provide more information (Credentials cache file
>>> '/tmp/krb5cc_495' not found)) errno 2 (No such file or directory)
>>> [20/Jun/2016:13:29:07 -0400] slapi_ldap_bind - Error: could not perform
>>> interactive bind for id [] mech [GSSAPI]: error -2 (Local error)
>>> [20/Jun/2016:13:29:07 -0400] NSMMReplicationPlugin -
>>> agmt="cn=meToserver3.domain.local" (server3:389): Replication bind with
>>> GSSAPI auth failed: LDAP error -2 (Local error) (SASL(-1): generic
>> failure:
>>> GSSAPI Error: Unspecified GSS failure. Minor code may provide more
>>> information (Credentials cache file '/tmp/krb5cc_495' not found))
>>> [20/Jun/2016:13:29:07 -0400] set_krb5_creds - Could not get initial
>>> credentials for principal [ldap/server1.domain.local at DOMAIN.LOCAL] in
>>> keytab [FILE:/etc/dirsrv/ds.keytab]: -1765328228 (Cannot contact any KDC
>>> for requested realm)
>>> [20/Jun/2016:13:29:07 -0400] - slapd started. Listening on All
> Interfaces
>>> port 389 for LDAP requests
>>> [20/Jun/2016:13:29:07 -0400] - Listening on All Interfaces port 636 for
>>> LDAPS requests
>>> [20/Jun/2016:13:29:07 -0400] - Listening
>>> on /var/run/slapd-DOMAIN-LOCAL.socket for LDAPI requests
>>> [20/Jun/2016:13:29:07 -0400] set_krb5_creds - Could not get initial
>>> credentials for principal [ldap/server1.domain.local at DOMAIN.LOCAL] in
>>> keytab [FILE:/etc/dirsrv/ds.keytab]: -1765328228 (Cannot contact any KDC
>>> for requested realm)
>>> [20/Jun/2016:13:29:07 -0400] slapd_ldap_sasl_interactive_bind - Error:
>>> could not perform interactive bind for id [] mech [GSSAPI]: LDAP error
> -2
>>> (Local error) (SASL(-1): generic failure: GSSAPI Error: Unspecified GSS
>>> failure. Minor code may provide more information (Credentials cache file
>>> '/tmp/krb5cc_495' not found)) errno 2 (No such file or directory)
>>> [20/Jun/2016:13:29:07 -0400] slapi_ldap_bind - Error: could not perform
>>> interactive bind for id [] mech [GSSAPI]: error -2 (Local error)
>>> [20/Jun/2016:13:29:07 -0400] NSMMReplicationPlugin -
>>> agmt="cn=meToserver4.domain.local" (server4:389): Replication bind with
>>> GSSAPI auth failed: LDAP error -2 (Local error) (SASL(-1): generic
>> failure:
>>> GSSAPI Error: Unspecified GSS failure. Minor code may provide more
>>> information (Credentials cache file '/tmp/krb5cc_495' not found))
>>> [20/Jun/2016:13:29:07 -0400] slapd_ldap_sasl_interactive_bind - Error:
>>> could not perform interactive bind for id [] mech [GSSAPI]: LDAP error
> -2
>>> (Local error) (SASL(-1): generic failure: GSSAPI Error: Unspecified GSS
>>> failure. Minor code may provide more information (Credentials cache file
>>> '/tmp/krb5cc_495' not found)) errno 2 (No such file or directory)
>>> [20/Jun/2016:13:29:07 -0400] slapi_ldap_bind - Error: could not perform
>>> interactive bind for id [] mech [GSSAPI]: error -2 (Local error)
>>> [20/Jun/2016:13:29:07 -0400] NSMMReplicationPlugin -
>>> agmt="cn=meTo1server.domain.local" (1server:389): Replication bind with
>>> GSSAPI auth failed: LDAP error -2 (Local error) (SASL(-1): generic
>> failure:
>>> GSSAPI Error: Unspecified GSS failure. Minor code may provide more>> information (Credentials cache file '/tmp/krb5cc_495' not found))
>>> [20/Jun/2016:13:29:07 -0400] slapd_ldap_sasl_interactive_bind - Error:
>>> could not perform interactive bind for id [] mech [GSSAPI]: LDAP error
> -2
>>> (Local error) (SASL(-1): generic failure: GSSAPI Error: Unspecified GSS
>>> failure. Minor code may provide more information (Credentials cache file
>>> '/tmp/krb5cc_495' not found)) errno 2 (No such file or directory)
>>> [20/Jun/2016:13:29:07 -0400] slapi_ldap_bind - Error: could not perform
>>> interactive bind for id [] mech [GSSAPI]: error -2 (Local error)
>>> [20/Jun/2016:13:29:07 -0400] NSMMReplicationPlugin -
>>> agmt="cn=meToserver7.domain.local" (server7:389): Replication bind with
>>> GSSAPI auth failed: LDAP error -2 (Local error) (SASL(-1): generic
>> failure:
>>> GSSAPI Error: Unspecified GSS failure. Minor code may provide more
>>> information (Credentials cache file '/tmp/krb5cc_495' not found))
>>> [20/Jun/2016:13:29:07 -0400] slapd_ldap_sasl_interactive_bind - Error:
>>> could not perform interactive bind for id [] mech [GSSAPI]: LDAP error
> -2
>>> (Local error) (SASL(-1): generic failure: GSSAPI Error: Unspecified GSS
>>> failure. Minor code may provide more information (Credentials cache file
>>> '/tmp/krb5cc_495' not found)) errno 0 (Success)
>>> [20/Jun/2016:13:29:07 -0400] slapi_ldap_bind - Error: could not perform
>>> interactive bind for id [] mech [GSSAPI]: error -2 (Local error)
>>> [20/Jun/2016:13:29:07 -0400] NSMMReplicationPlugin -
>>> agmt="cn=meToserver2.domain.local" (server2:389): Replication bind with
>>> GSSAPI auth failed: LDAP error -2 (Local error) (SASL(-1): generic
>> failure:
>>> GSSAPI Error: Unspecified GSS failure. Minor code may provide more
>>> information (Credentials cache file '/tmp/krb5cc_495' not found))
>>> [20/Jun/2016:13:29:10 -0400] NSMMReplicationPlugin -