[Freeipa-users] krb5kdc will not start (kerberos authentication error)

Tue Nov 10 16:42:29 UTC 2015

This gave me a huge return!  Appears to be a long list of all the servers and applications whose users authenticate to the IPA servers.

ldapsearch -x -h 172.16.100.161 -D "cn=directory manager" -W -b "dc=itmodev,dc=gov" '(objectclass=krbprincipal)'

# search result
search: 2
result: 0 Success

# numResponses: 142
# numEntries: 141

-----Original Message-----
From: freeipa-users-bounces at redhat.com [mailto:freeipa-users-bounces at redhat.com] On Behalf Of Ludwig Krispenz
Sent: Tuesday, November 10, 2015 11:37 AM
To: freeipa-users at redhat.com
Subject: Re: [Freeipa-users] krb5kdc will not start (kerberos authentication error)

what do you get if you search for "objectclass=krbprincipal" ?

On 11/10/2015 05:27 PM, Rich Megginson wrote:
> On 11/10/2015 09:16 AM, Gronde, Christopher (Contractor) wrote:
>> Neither came back with anything
>>
>> # ldapsearch -x -h 172.16.100.161 -D "cn=directory manager" -W -b 
>> "dc=itmodev,dc=gov" '(uid=ldap/comipa01.itmodev.gov)'
>> Enter LDAP Password:
>> # extended LDIF
>> #
>> # LDAPv3
>> # base <dc=itmodev,dc=gov> with scope subtree # filter: 
>> (uid=ldap/comipa01.itmodev.gov) # requesting: ALL #
>>
>> # search result
>> search: 2
>> result: 0 Success
>>
>> # numResponses: 1
>> [root at comipa02 ~]# ldapsearch -x -h 172.16.100.161 -D "cn=directory 
>> manager" -W -b "dc=itmodev,dc=gov" '(uid=ldap/*.gov)' uid Enter LDAP 
>> Password:
>> # extended LDIF
>> #
>> # LDAPv3
>> # base <dc=itmodev,dc=gov> with scope subtree # filter: 
>> (uid=ldap/*.gov) # requesting: uid #
>>
>> # search result
>> search: 2
>> result: 0 Success
>>
>> # numResponses: 1
>
> That means this server has no LDAP service principals?  I'm not sure 
> how to recover IPA from this scenario.
>
>>
>> -----Original Message-----
>> From: freeipa-users-bounces at redhat.com 
>> [mailto:freeipa-users-bounces at redhat.com] On Behalf Of Rich Megginson
>> Sent: Tuesday, November 10, 2015 11:04 AM
>> To: freeipa-users at redhat.com
>> Subject: Re: [Freeipa-users] krb5kdc will not start (kerberos 
>> authentication error)
>>
>> On 11/10/2015 08:18 AM, Gronde, Christopher (Contractor) wrote:
>>> Thank you!  I should have caught that...
>>>
>>> I changed the log level and then restarted dirsrv and attempted to 
>>> start krb5kdc and got the following...
>> <snip>
>>
>> [10/Nov/2015:10:12:02 -0500] conn=5 fd=64 slot=64 connection from
>> 172.16.100.208 to 172.16.100.161
>> [10/Nov/2015:10:12:02 -0500] conn=5 op=0 BIND dn="" method=sasl
>> version=3 mech=GSSAPI
>> [10/Nov/2015:10:12:03 -0500] conn=5 op=0 RESULT err=14 tag=97
>> nentries=0 etime=1, SASL bind in progress
>> [10/Nov/2015:10:12:03 -0500] conn=5 op=1 BIND dn="" method=sasl
>> version=3 mech=GSSAPI
>> [10/Nov/2015:10:12:03 -0500] conn=5 op=1 RESULT err=14 tag=97
>> nentries=0 etime=0, SASL bind in progress
>> [10/Nov/2015:10:12:03 -0500] conn=5 op=2 BIND dn="" method=sasl
>> version=3 mech=GSSAPI
>> [10/Nov/2015:10:12:03 -0500] conn=Internal op=-1 SRCH 
>> base="dc=itmodev,dc=gov" scope=2 
>> filter="(uid=ldap/comipa01.itmodev.gov)" attrs=ALL
>> [10/Nov/2015:10:12:03 -0500] conn=Internal op=-1 RESULT err=0 tag=48
>> nentries=0 etime=0
>> [10/Nov/2015:10:12:03 -0500] conn=5 op=2 RESULT err=49 tag=97 
>> nentries=0
>> etime=0
>> [10/Nov/2015:10:12:03 -0500] conn=5 op=3 UNBIND
>> [10/Nov/2015:10:12:03 -0500] conn=5 op=3 fd=64 closed - U1
>>
>> <snip>
>>
>> This is the SASL bind.  It thinks the principal in the Kerberos 
>> credential is "ldap/comipa01.itmodev.gov", and the SASL map tells the 
>> code to look for something with uid=ldap/comipa01.itmodev.gov under 
>> dc=itmodev,dc=gov.  However, this entry is not found: RESULT err=0
>> tag=48 nentries=0.  nentries=0 means no entries matched the search 
>> criteria.
>>
>> You can do the search yourself with ldapsearch:
>>
>> ldapsearch -x -h 172.16.100.161 -D "cn=directory manager" -W -b 
>> "dc=itmodev,dc=gov" '(uid=ldap/comipa01.itmodev.gov)'
>>
>> If you want to find out if there is some other ldap principal, do a 
>> search like this:
>>
>> ldapsearch -x -h 172.16.100.161 -D "cn=directory manager" -W -b 
>> "dc=itmodev,dc=gov" '(uid=ldap/*.gov)' uid
>>
>>>> Ran into an error trying to set that
>>>>
>>>> # ldapmodify -a -D "cn=directory manager" -W Enter LDAP Password:
>>>> dn: cn=config
>>>> changetype: modify
>>>> replace: nsslapd-acesslog-level
>>>> : 260
>>>>
>>>> modifying entry "cn=config"
>>>> ldap_modify: Server is unwilling to perform (53)
>>>>            additional info: Unknown attribute 
>>>> nsslapd-acesslog-level will be ignored
>>>>
>>>> [root at comipa02 ~]# ldapmodify -a -D "cn=config" -W Enter LDAP
>>>> Password:
>>>> ldap_bind: Inappropriate authentication (48)
>>>>
>>>> -----Original Message-----
>>>> From: Ludwig Krispenz [mailto:lkrispen at redhat.com]
>>>> Sent: Tuesday, November 10, 2015 9:48 AM
>>>> To: Gronde, Christopher (Contractor) 
>>>> <Christopher.Gronde at fincen.gov>
>>>> Cc: freeipa-users at redhat.com
>>>> Subject: Re: [Freeipa-users] krb5kdc will not start (kerberos 
>>>> authentication error)
>>>>
>>>>
>>>> On 11/10/2015 03:32 PM, Gronde, Christopher (Contractor) wrote:
>>>>> How do I change that log setting? Is that done in LDAP?  Using 
>>>>> ldapmodify?
>>>> yes,
>>>> ldapmodify ...
>>>> dn: cn=config
>>>> changetype: modify
>>>> replace: nsslapd-acesslog-level
>>>> nsslapd-acesslog-level: 260
>>>>> -----Original Message-----
>>>>> From: freeipa-users-bounces at redhat.com 
>>>>> [mailto:freeipa-users-bounces at redhat.com] On Behalf Of Ludwig 
>>>>> Krispenz
>>>>> Sent: Tuesday, November 10, 2015 9:03 AM
>>>>> To: freeipa-users at redhat.com
>>>>> Subject: Re: [Freeipa-users] krb5kdc will not start (kerberos 
>>>>> authentication error)
>>>>>
>>>>>
>>>>> On 11/10/2015 02:40 PM, Alexander Bokovoy wrote:
>>>>>> On Tue, 10 Nov 2015, Gronde, Christopher (Contractor) wrote:
>>>>>>> Where can I verify or change the credentials it is trying to use?
>>>>>>> Is it my LDAP password?
>>>>>> No, according to your logs, it is your LDAP master trying to 
>>>>>> replicate (push changes) to your LDAP replica:
>>>>>>>> [09/Nov/2015:15:02:01 -0500] conn=2 fd=64 slot=64 connection 
>>>>>>>> from <MASTER_IP> to <REPLICA_IP>
>>>>>>>> [09/Nov/2015:15:02:01 -0500] conn=2 op=0 BIND dn="" method=sasl
>>>>>>>> version=3 mech=GSSAPI
>>>>> err=49 could also be a result if the entry which is mapped from 
>>>>> the principal is not found in the directory. A bit more info could 
>>>>> be gained by enabling logging of internal searches.
>>>>> Set nsslapd-acesslog-level: 260
>>>>>
>>>>> and then look what internal searches are done during the gssapi 
>>>>> authentication
>>>>>> If that is true, it would be ldap/<master> Kerberos principal 
>>>>>> talking to ldap/<replica> Kerberos principal. If that fails, it 
>>>>>> means master and replica KDCs have different understanding of 
>>>>>> both ldap/<master> and ldap/<replica> keys which most likely 
>>>>>> means keys were rotated on master and weren't propagated to replica.
>>>>>>
>>>>>> How to solve it? One possibility is to set master's hostname as 
>>>>>> KDC address in krb5.conf on replica, forcing LDAP server on 
>>>>>> replica to use master's KDC. I'm absolutely not sure this will 
>>>>>> actually work but at least it allows to see if we are indeed 
>>>>>> dealing with inconsistent state of service principals' keys.
>>>>>>
>>>>>>> -----Original Message-----
>>>>>>> From: Alexander Bokovoy [mailto:abokovoy at redhat.com]
>>>>>>> Sent: Tuesday, November 10, 2015 8:18 AM
>>>>>>> To: Gronde, Christopher (Contractor) 
>>>>>>> <Christopher.Gronde at fincen.gov>
>>>>>>> Cc: Rob Crittenden <rcritten at redhat.com>; 
>>>>>>> freeipa-users at redhat.com
>>>>>>> Subject: Re: [Freeipa-users] krb5kdc will not start (kerberos 
>>>>>>> authentication error)
>>>>>>>
>>>>>>> On Tue, 10 Nov 2015, Gronde, Christopher (Contractor) wrote:
>>>>>>>> When I tried to start the service again I got no response from 
>>>>>>>> tail of the log, but this is a repeating entry I see in the 
>>>>>>>> access log
>>>>>>>>
>>>>>>>> [09/Nov/2015:15:01:04 -0500] conn=1 fd=64 slot=64 connection 
>>>>>>>> from
>>>>>>>> 127.0.0.1 to 127.0.0.1
>>>>>>>> [09/Nov/2015:15:01:04 -0500] conn=1 op=-1 fd=64 closed - B1
>>>>>>>> [09/Nov/2015:15:02:01 -0500] conn=2 fd=64 slot=64 connection 
>>>>>>>> from <MASTER_IP> to <REPLICA_IP>
>>>>>>>> [09/Nov/2015:15:02:01 -0500] conn=2 op=0 BIND dn="" method=sasl
>>>>>>>> version=3 mech=GSSAPI
>>>>>>>> [09/Nov/2015:15:02:01 -0500] conn=2 op=0 RESULT err=14 tag=97
>>>>>>>> nentries=0 etime=0, SASL bind in progress
>>>>>>>> [09/Nov/2015:15:02:01 -0500] conn=2 op=1 BIND dn="" method=sasl
>>>>>>>> version=3 mech=GSSAPI
>>>>>>>> [09/Nov/2015:15:02:01 -0500] conn=2 op=1 RESULT err=14 tag=97
>>>>>>>> nentries=0 etime=0, SASL bind in progress
>>>>>>>> [09/Nov/2015:15:02:01 -0500] conn=2 op=2 BIND dn="" method=sasl
>>>>>>>> version=3 mech=GSSAPI
>>>>>>>> [09/Nov/2015:15:02:01 -0500] conn=2 op=2 RESULT err=49 tag=97
>>>>>>>> nentries=0 etime=0
>>>>>>>> [09/Nov/2015:15:02:01 -0500] conn=2 op=3 UNBIND
>>>>>>>> [09/Nov/2015:15:02:01 -0500] conn=2 op=3 fd=64 closed - U1
>>>>>>>>
>>>>>>>> Does anyone know what err=14 or err=49 are?
>>>>>>> err=14 means SASL bind in progress -- i.e. multi-round 
>>>>>>> processing is ongoing. This is normal for SASL GSSAPI.
>>>>>>>
>>>>>>> err=49 is wrong password or username, i.e. credentials were 
>>>>>>> incorrect.
>>>>>>> It may also mean that LDAP server side was unable to process 
>>>>>>> Kerberos negotiation due to not having a current Kerberos ticket 
>>>>>>> for own service
>>>>>>> (LDAP) and trying to request it from the Kerberos KDC but 
>>>>>>> Kerberos KDC is down.
>>>>>>>
>>>>>>>> -----Original Message-----
>>>>>>>> From: Rob Crittenden [mailto:rcritten at redhat.com]
>>>>>>>> Sent: Monday, November 09, 2015 3:26 PM
>>>>>>>> To: Gronde, Christopher (Contractor) 
>>>>>>>> <Christopher.Gronde at fincen.gov>; Alexander Bokovoy 
>>>>>>>> <abokovoy at redhat.com>
>>>>>>>> Cc: freeipa-users at redhat.com
>>>>>>>> Subject: Re: [Freeipa-users] krb5kdc will not start (kerberos 
>>>>>>>> authentication error)
>>>>>>>>
>>>>>>>> Gronde, Christopher (Contractor) wrote:
>>>>>>>>> Nothing bad came back and there is definitely data in the tree.
>>>>>>>> Ok, I guess I'd try to start the kdc again and then watch the 
>>>>>>>> 389-ds access log (buffered) to:
>>>>>>>>
>>>>>>>> 1. See if it is binding at all
>>>>>>>> 2. See what the search is and what, if any, results were 
>>>>>>>> returned
>>>>>>>>
>>>>>>>> This would be in /var/log/dirsrv/slapd-YOUR_REALM/access
>>>>>>>>
>>>>>>>> rob
>>>>>>>>
>>>>>>>>> -----Original Message-----
>>>>>>>>> From: Rob Crittenden [mailto:rcritten at redhat.com]
>>>>>>>>> Sent: Monday, November 09, 2015 11:46 AM
>>>>>>>>> To: Gronde, Christopher (Contractor) 
>>>>>>>>> <Christopher.Gronde at fincen.gov>; Alexander Bokovoy 
>>>>>>>>> <abokovoy at redhat.com>
>>>>>>>>> Cc: freeipa-users at redhat.com
>>>>>>>>> Subject: Re: [Freeipa-users] krb5kdc will not start (kerberos 
>>>>>>>>> authentication error)
>>>>>>>>>
>>>>>>>>> Gronde, Christopher (Contractor) wrote:
>>>>>>>>>> I restarted dirsrv and attempted to start krb5kdc and this is 
>>>>>>>>>> what the error log shows
>>>>>>>>>>
>>>>>>>>>> # tail /var/log/dirsrv/slapd-ITMODEV-GOV/errors
>>>>>>>>>> [09/Nov/2015:11:01:02 -0500] - WARNING: userRoot: entry cache 
>>>>>>>>>> size 10485760B is less than db size 28016640B; We recommend 
>>>>>>>>>> to increase the entry cache size nsslapd-cachememsize.
>>>>>>>>>> [09/Nov/2015:11:01:02 -0500] - slapd started. Listening on 
>>>>>>>>>> All Interfaces port 389 for LDAP requests
>>>>>>>>>> [09/Nov/2015:11:06:04 -0500] - slapd shutting down - 
>>>>>>>>>> signaling operation threads
>>>>>>>>>> [09/Nov/2015:11:06:04 -0500] - slapd shutting down - closing 
>>>>>>>>>> down internal subsystems and plugins
>>>>>>>>>> [09/Nov/2015:11:06:04 -0500] - Waiting for 4 database threads 
>>>>>>>>>> to stop
>>>>>>>>>> [09/Nov/2015:11:06:04 -0500] - All database threads now 
>>>>>>>>>> stopped
>>>>>>>>>> [09/Nov/2015:11:06:04 -0500] - slapd stopped.
>>>>>>>>>> [09/Nov/2015:11:14:20 -0500] - 389-Directory/1.2.11.15
>>>>>>>>>> B2015.247.1737 starting up
>>>>>>>>>> [09/Nov/2015:11:14:20 -0500] - WARNING: userRoot: entry cache 
>>>>>>>>>> size 10485760B is less than db size 28016640B; We recommend 
>>>>>>>>>> to increase the entry cache size nsslapd-cachememsize.
>>>>>>>>>> [09/Nov/2015:11:14:20 -0500] - slapd started. Listening on 
>>>>>>>>>> All Interfaces port 389 for LDAP requests
>>>>>>>>> Ok, that's good.
>>>>>>>>>
>>>>>>>>> I'd do something like this to see what is in the db 
>>>>>>>>> (substitute example.com with your domain):
>>>>>>>>>
>>>>>>>>> $ ldapsearch -x -D 'cn=Directory Manager' -W -s one -b 
>>>>>>>>> cn=kerberos,dc=example,dc=com
>>>>>>>>>
>>>>>>>>> (don't post the output as it would include the kerberos master 
>>>>>>>>> key).
>>>>>>>>>
>>>>>>>>> If that returns nothing that's bad.
>>>>>>>>>
>>>>>>>>> If it succeeds I'd broaden the search base a bit to see what 
>>>>>>>>> data you do
>>>>>>>>> have:
>>>>>>>>>
>>>>>>>>> $ ldapsearch -x -D 'cn=Directory Manager' -W -b 
>>>>>>>>> cn=groups,cn=accounts,dc=example,dc=com
>>>>>>>>>
>>>>>>>>> I picked groups because usually groups << users in numbers. 
>>>>>>>>> This is just to see if you have data in the tree.
>>>>>>>>>
>>>>>>>>> Let us know if either or both turns up nothing.
>>>>>>>>>
>>>>>>>>> rob
>>>>>>>>>
>>>>>>>>>> -----Original Message-----
>>>>>>>>>> From: Alexander Bokovoy [mailto:abokovoy at redhat.com]
>>>>>>>>>> Sent: Monday, November 09, 2015 10:51 AM
>>>>>>>>>> To: Gronde, Christopher (Contractor) 
>>>>>>>>>> <Christopher.Gronde at fincen.gov>
>>>>>>>>>> Cc: freeipa-users at redhat.com
>>>>>>>>>> Subject: Re: [Freeipa-users] krb5kdc will not start (kerberos 
>>>>>>>>>> authentication error)
>>>>>>>>>>
>>>>>>>>>> On Mon, 09 Nov 2015, Gronde, Christopher (Contractor) wrote:
>>>>>>>>>>> Hello all!
>>>>>>>>>>>
>>>>>>>>>>> On my replica IPA server after fixing a cert issue that had 
>>>>>>>>>>> been going on for sometime, I have all my certs figured out 
>>>>>>>>>>> but the krb5kdc service will not start.
>>>>>>>>>>>
>>>>>>>>>>> # service krb5kdc start
>>>>>>>>>>> Starting Kerberos 5 KDC: krb5kdc: cannot initialize realm
>>>>>>>>>>> ITMODEV.GOV - see log file for details                  
>>>>>>>>>>> [FAILED]
>>>>>>>>>>>
>>>>>>>>>>> # cat /var/log/krb5kdc.log
>>>>>>>>>>> krb5kdc: Server error - while fetching master key K/M for 
>>>>>>>>>>> realm ITMODEV.GOV
>>>>>>>>>>> krb5kdc: Server error - while fetching master key K/M for 
>>>>>>>>>>> realm ITMODEV.GOV
>>>>>>>>>>> krb5kdc: Server error - while fetching master key K/M for 
>>>>>>>>>>> realm ITMODEV.GOV
>>>>>>>>>>>
>>>>>>>>>>> I found this article online:
>>>>>>>>>>> http://research.imb.uq.edu.au/~l.rathbone/ldap/kerberos.shtm
>>>>>>>>>>> l
>>>>>>>>>>>
>>>>>>>>>>> Which stated it might be because The slave KDC does not have 
>>>>>>>>>>> a stash file (.k5.EXAMPLE.COM). You need to create one. 
>>>>>>>>>>> Tried the command
>>>>>>>>>>> listed:
>>>>>>>>>>>
>>>>>>>>>>> # kdb5_util stash
>>>>>>>>>>> kdb5_util: Server error while retrieving master entry
>>>>>>>>>>>
>>>>>>>>>>> No further information found on the proceeding error above 
>>>>>>>>>>> for the kdb5_util command.
>>>>>>>>>>>
>>>>>>>>>>> Any thoughts?
>>>>>>>>>> First: don't use instructions which are not related to IPA, 
>>>>>>>>>> please.
>>>>>>>>>>
>>>>>>>>>> FreeIPA has its own LDAP driver for KDC and instructions for 
>>>>>>>>>> anything else do not apply here at all.
>>>>>>>>>>
>>>>>>>>>> If you see 'Server error - while fetching master key ..' it 
>>>>>>>>>> means KDC LDAP driver was unable to contact LDAP server. Does 
>>>>>>>>>> LDAP server work on the replica? What is in its error log 
>>>>>>>>>> (/var/log/dirsrv/slapd-ITMODEV-GOV/errors)?
>>>>>>>>>>
>>>>>>>>>> --
>>>>>>>>>> / Alexander Bokovoy
>>>>>>>>>>
>>>>>>>>>>
>>>>>>> --
>>>>>>> / Alexander Bokovoy
>>>>>>>
>>>>> --
>>>>> Manage your subscription for the Freeipa-users mailing list:
>>>>> https://www.redhat.com/mailman/listinfo/freeipa-users
>>>>> Go to http://freeipa.org for more info on the project
>>>>>
>>>
>> --
>> Manage your subscription for the Freeipa-users mailing list:
>> https://www.redhat.com/mailman/listinfo/freeipa-users
>> Go to http://freeipa.org for more info on the project
>>
>

--
Manage your subscription for the Freeipa-users mailing list:
https://www.redhat.com/mailman/listinfo/freeipa-users
Go to http://freeipa.org for more info on the project