[Freeipa-users] krb5kdc will not start (kerberos authentication error)

Tue Nov 10 16:16:48 UTC 2015

Neither came back with anything

# ldapsearch -x -h 172.16.100.161 -D "cn=directory manager" -W -b "dc=itmodev,dc=gov" '(uid=ldap/comipa01.itmodev.gov)'
Enter LDAP Password:
# extended LDIF
#
# LDAPv3
# base <dc=itmodev,dc=gov> with scope subtree
# filter: (uid=ldap/comipa01.itmodev.gov)
# requesting: ALL
#

# search result
search: 2
result: 0 Success

# numResponses: 1
[root at comipa02 ~]# ldapsearch -x -h 172.16.100.161 -D "cn=directory manager" -W -b "dc=itmodev,dc=gov" '(uid=ldap/*.gov)' uid
Enter LDAP Password:
# extended LDIF
#
# LDAPv3
# base <dc=itmodev,dc=gov> with scope subtree
# filter: (uid=ldap/*.gov)
# requesting: uid
#

# search result
search: 2
result: 0 Success

# numResponses: 1

-----Original Message-----
From: freeipa-users-bounces at redhat.com [mailto:freeipa-users-bounces at redhat.com] On Behalf Of Rich Megginson
Sent: Tuesday, November 10, 2015 11:04 AM
To: freeipa-users at redhat.com
Subject: Re: [Freeipa-users] krb5kdc will not start (kerberos authentication error)

On 11/10/2015 08:18 AM, Gronde, Christopher (Contractor) wrote:
> Thank you!  I should have caught that...
>
> I changed the log level and then restarted dirsrv and attempted to start krb5kdc and got the following...
<snip>

[10/Nov/2015:10:12:02 -0500] conn=5 fd=64 slot=64 connection from
172.16.100.208 to 172.16.100.161
[10/Nov/2015:10:12:02 -0500] conn=5 op=0 BIND dn="" method=sasl
version=3 mech=GSSAPI
[10/Nov/2015:10:12:03 -0500] conn=5 op=0 RESULT err=14 tag=97 nentries=0 etime=1, SASL bind in progress
[10/Nov/2015:10:12:03 -0500] conn=5 op=1 BIND dn="" method=sasl
version=3 mech=GSSAPI
[10/Nov/2015:10:12:03 -0500] conn=5 op=1 RESULT err=14 tag=97 nentries=0 etime=0, SASL bind in progress
[10/Nov/2015:10:12:03 -0500] conn=5 op=2 BIND dn="" method=sasl
version=3 mech=GSSAPI
[10/Nov/2015:10:12:03 -0500] conn=Internal op=-1 SRCH base="dc=itmodev,dc=gov" scope=2 filter="(uid=ldap/comipa01.itmodev.gov)" attrs=ALL
[10/Nov/2015:10:12:03 -0500] conn=Internal op=-1 RESULT err=0 tag=48
nentries=0 etime=0
[10/Nov/2015:10:12:03 -0500] conn=5 op=2 RESULT err=49 tag=97 nentries=0
etime=0
[10/Nov/2015:10:12:03 -0500] conn=5 op=3 UNBIND
[10/Nov/2015:10:12:03 -0500] conn=5 op=3 fd=64 closed - U1

<snip>

This is the SASL bind.  It thinks the principal in the Kerberos credential is "ldap/comipa01.itmodev.gov", and the SASL map tells the code to look for something with uid=ldap/comipa01.itmodev.gov under dc=itmodev,dc=gov.  However, this entry is not found: RESULT err=0
tag=48 nentries=0.  nentries=0 means no entries matched the search criteria.

You can do the search yourself with ldapsearch:

ldapsearch -x -h 172.16.100.161 -D "cn=directory manager" -W -b "dc=itmodev,dc=gov" '(uid=ldap/comipa01.itmodev.gov)'

If you want to find out if there is some other ldap principal, do a search like this:

ldapsearch -x -h 172.16.100.161 -D "cn=directory manager" -W -b "dc=itmodev,dc=gov" '(uid=ldap/*.gov)' uid

>> Ran into an error trying to set that
>>
>> # ldapmodify -a -D "cn=directory manager" -W Enter LDAP Password:
>> dn: cn=config
>> changetype: modify
>> replace: nsslapd-acesslog-level
>> : 260
>>
>> modifying entry "cn=config"
>> ldap_modify: Server is unwilling to perform (53)
>>           additional info: Unknown attribute nsslapd-acesslog-level 
>> will be ignored
>>
>> [root at comipa02 ~]# ldapmodify -a -D "cn=config" -W Enter LDAP
>> Password:
>> ldap_bind: Inappropriate authentication (48)
>>
>> -----Original Message-----
>> From: Ludwig Krispenz [mailto:lkrispen at redhat.com]
>> Sent: Tuesday, November 10, 2015 9:48 AM
>> To: Gronde, Christopher (Contractor) <Christopher.Gronde at fincen.gov>
>> Cc: freeipa-users at redhat.com
>> Subject: Re: [Freeipa-users] krb5kdc will not start (kerberos 
>> authentication error)
>>
>>
>> On 11/10/2015 03:32 PM, Gronde, Christopher (Contractor) wrote:
>>> How do I change that log setting?  Is that done in LDAP?  Using ldapmodify?
>> yes,
>> ldapmodify ...
>> dn: cn=config
>> changetype: modify
>> replace: nsslapd-acesslog-level
>> nsslapd-acesslog-level: 260
>>> -----Original Message-----
>>> From: freeipa-users-bounces at redhat.com 
>>> [mailto:freeipa-users-bounces at redhat.com] On Behalf Of Ludwig 
>>> Krispenz
>>> Sent: Tuesday, November 10, 2015 9:03 AM
>>> To: freeipa-users at redhat.com
>>> Subject: Re: [Freeipa-users] krb5kdc will not start (kerberos 
>>> authentication error)
>>>
>>>
>>> On 11/10/2015 02:40 PM, Alexander Bokovoy wrote:
>>>> On Tue, 10 Nov 2015, Gronde, Christopher (Contractor) wrote:
>>>>> Where can I verify or change the credentials it is trying to use?
>>>>> Is it my LDAP password?
>>>> No, according to your logs, it is your LDAP master trying to 
>>>> replicate (push changes) to your LDAP replica:
>>>>>> [09/Nov/2015:15:02:01 -0500] conn=2 fd=64 slot=64 connection from 
>>>>>> <MASTER_IP> to <REPLICA_IP>
>>>>>> [09/Nov/2015:15:02:01 -0500] conn=2 op=0 BIND dn="" method=sasl
>>>>>> version=3 mech=GSSAPI
>>> err=49 could also be a result if the entry which is mapped from the principal is not found in the directory. A bit more info could be gained by enabling logging of internal searches.
>>> Set nsslapd-acesslog-level: 260
>>>
>>> and then look what internal searches are done during the gssapi 
>>> authentication
>>>> If that is true, it would be ldap/<master> Kerberos principal 
>>>> talking to ldap/<replica> Kerberos principal. If that fails, it 
>>>> means master and replica KDCs have different understanding of both 
>>>> ldap/<master> and ldap/<replica> keys which most likely means keys 
>>>> were rotated on master and weren't propagated to replica.
>>>>
>>>> How to solve it? One possibility is to set master's hostname as KDC 
>>>> address in krb5.conf on replica, forcing LDAP server on replica to 
>>>> use master's KDC. I'm absolutely not sure this will actually work 
>>>> but at least it allows to see if we are indeed dealing with 
>>>> inconsistent state of service principals' keys.
>>>>
>>>>> -----Original Message-----
>>>>> From: Alexander Bokovoy [mailto:abokovoy at redhat.com]
>>>>> Sent: Tuesday, November 10, 2015 8:18 AM
>>>>> To: Gronde, Christopher (Contractor) 
>>>>> <Christopher.Gronde at fincen.gov>
>>>>> Cc: Rob Crittenden <rcritten at redhat.com>; freeipa-users at redhat.com
>>>>> Subject: Re: [Freeipa-users] krb5kdc will not start (kerberos 
>>>>> authentication error)
>>>>>
>>>>> On Tue, 10 Nov 2015, Gronde, Christopher (Contractor) wrote:
>>>>>> When I tried to start the service again I got no response from 
>>>>>> tail of the log, but this is a repeating entry I see in the 
>>>>>> access log
>>>>>>
>>>>>> [09/Nov/2015:15:01:04 -0500] conn=1 fd=64 slot=64 connection from
>>>>>> 127.0.0.1 to 127.0.0.1
>>>>>> [09/Nov/2015:15:01:04 -0500] conn=1 op=-1 fd=64 closed - B1
>>>>>> [09/Nov/2015:15:02:01 -0500] conn=2 fd=64 slot=64 connection from 
>>>>>> <MASTER_IP> to <REPLICA_IP>
>>>>>> [09/Nov/2015:15:02:01 -0500] conn=2 op=0 BIND dn="" method=sasl
>>>>>> version=3 mech=GSSAPI
>>>>>> [09/Nov/2015:15:02:01 -0500] conn=2 op=0 RESULT err=14 tag=97
>>>>>> nentries=0 etime=0, SASL bind in progress
>>>>>> [09/Nov/2015:15:02:01 -0500] conn=2 op=1 BIND dn="" method=sasl
>>>>>> version=3 mech=GSSAPI
>>>>>> [09/Nov/2015:15:02:01 -0500] conn=2 op=1 RESULT err=14 tag=97
>>>>>> nentries=0 etime=0, SASL bind in progress
>>>>>> [09/Nov/2015:15:02:01 -0500] conn=2 op=2 BIND dn="" method=sasl
>>>>>> version=3 mech=GSSAPI
>>>>>> [09/Nov/2015:15:02:01 -0500] conn=2 op=2 RESULT err=49 tag=97
>>>>>> nentries=0 etime=0
>>>>>> [09/Nov/2015:15:02:01 -0500] conn=2 op=3 UNBIND
>>>>>> [09/Nov/2015:15:02:01 -0500] conn=2 op=3 fd=64 closed - U1
>>>>>>
>>>>>> Does anyone know what err=14 or err=49 are?
>>>>> err=14 means SASL bind in progress -- i.e. multi-round processing 
>>>>> is ongoing. This is normal for SASL GSSAPI.
>>>>>
>>>>> err=49 is wrong password or username, i.e. credentials were incorrect.
>>>>> It may also mean that LDAP server side was unable to process 
>>>>> Kerberos negotiation due to not having a current Kerberos ticket 
>>>>> for own service
>>>>> (LDAP) and trying to request it from the Kerberos KDC but Kerberos 
>>>>> KDC is down.
>>>>>
>>>>>> -----Original Message-----
>>>>>> From: Rob Crittenden [mailto:rcritten at redhat.com]
>>>>>> Sent: Monday, November 09, 2015 3:26 PM
>>>>>> To: Gronde, Christopher (Contractor) 
>>>>>> <Christopher.Gronde at fincen.gov>; Alexander Bokovoy 
>>>>>> <abokovoy at redhat.com>
>>>>>> Cc: freeipa-users at redhat.com
>>>>>> Subject: Re: [Freeipa-users] krb5kdc will not start (kerberos 
>>>>>> authentication error)
>>>>>>
>>>>>> Gronde, Christopher (Contractor) wrote:
>>>>>>> Nothing bad came back and there is definitely data in the tree.
>>>>>> Ok, I guess I'd try to start the kdc again and then watch the 
>>>>>> 389-ds access log (buffered) to:
>>>>>>
>>>>>> 1. See if it is binding at all
>>>>>> 2. See what the search is and what, if any, results were returned
>>>>>>
>>>>>> This would be in /var/log/dirsrv/slapd-YOUR_REALM/access
>>>>>>
>>>>>> rob
>>>>>>
>>>>>>> -----Original Message-----
>>>>>>> From: Rob Crittenden [mailto:rcritten at redhat.com]
>>>>>>> Sent: Monday, November 09, 2015 11:46 AM
>>>>>>> To: Gronde, Christopher (Contractor) 
>>>>>>> <Christopher.Gronde at fincen.gov>; Alexander Bokovoy 
>>>>>>> <abokovoy at redhat.com>
>>>>>>> Cc: freeipa-users at redhat.com
>>>>>>> Subject: Re: [Freeipa-users] krb5kdc will not start (kerberos 
>>>>>>> authentication error)
>>>>>>>
>>>>>>> Gronde, Christopher (Contractor) wrote:
>>>>>>>> I restarted dirsrv and attempted to start krb5kdc and this is 
>>>>>>>> what the error log shows
>>>>>>>>
>>>>>>>> # tail /var/log/dirsrv/slapd-ITMODEV-GOV/errors
>>>>>>>> [09/Nov/2015:11:01:02 -0500] - WARNING: userRoot: entry cache 
>>>>>>>> size 10485760B is less than db size 28016640B; We recommend to 
>>>>>>>> increase the entry cache size nsslapd-cachememsize.
>>>>>>>> [09/Nov/2015:11:01:02 -0500] - slapd started.  Listening on All 
>>>>>>>> Interfaces port 389 for LDAP requests
>>>>>>>> [09/Nov/2015:11:06:04 -0500] - slapd shutting down - signaling 
>>>>>>>> operation threads
>>>>>>>> [09/Nov/2015:11:06:04 -0500] - slapd shutting down - closing 
>>>>>>>> down internal subsystems and plugins
>>>>>>>> [09/Nov/2015:11:06:04 -0500] - Waiting for 4 database threads 
>>>>>>>> to stop
>>>>>>>> [09/Nov/2015:11:06:04 -0500] - All database threads now stopped
>>>>>>>> [09/Nov/2015:11:06:04 -0500] - slapd stopped.
>>>>>>>> [09/Nov/2015:11:14:20 -0500] - 389-Directory/1.2.11.15
>>>>>>>> B2015.247.1737 starting up
>>>>>>>> [09/Nov/2015:11:14:20 -0500] - WARNING: userRoot: entry cache 
>>>>>>>> size 10485760B is less than db size 28016640B; We recommend to 
>>>>>>>> increase the entry cache size nsslapd-cachememsize.
>>>>>>>> [09/Nov/2015:11:14:20 -0500] - slapd started.  Listening on All 
>>>>>>>> Interfaces port 389 for LDAP requests
>>>>>>> Ok, that's good.
>>>>>>>
>>>>>>> I'd do something like this to see what is in the db (substitute 
>>>>>>> example.com with your domain):
>>>>>>>
>>>>>>> $ ldapsearch -x -D 'cn=Directory Manager' -W -s one -b 
>>>>>>> cn=kerberos,dc=example,dc=com
>>>>>>>
>>>>>>> (don't post the output as it would include the kerberos master key).
>>>>>>>
>>>>>>> If that returns nothing that's bad.
>>>>>>>
>>>>>>> If it succeeds I'd broaden the search base a bit to see what 
>>>>>>> data you do
>>>>>>> have:
>>>>>>>
>>>>>>> $ ldapsearch -x -D 'cn=Directory Manager' -W -b 
>>>>>>> cn=groups,cn=accounts,dc=example,dc=com
>>>>>>>
>>>>>>> I picked groups because usually groups << users in numbers. This 
>>>>>>> is just to see if you have data in the tree.
>>>>>>>
>>>>>>> Let us know if either or both turns up nothing.
>>>>>>>
>>>>>>> rob
>>>>>>>
>>>>>>>> -----Original Message-----
>>>>>>>> From: Alexander Bokovoy [mailto:abokovoy at redhat.com]
>>>>>>>> Sent: Monday, November 09, 2015 10:51 AM
>>>>>>>> To: Gronde, Christopher (Contractor) 
>>>>>>>> <Christopher.Gronde at fincen.gov>
>>>>>>>> Cc: freeipa-users at redhat.com
>>>>>>>> Subject: Re: [Freeipa-users] krb5kdc will not start (kerberos 
>>>>>>>> authentication error)
>>>>>>>>
>>>>>>>> On Mon, 09 Nov 2015, Gronde, Christopher (Contractor) wrote:
>>>>>>>>> Hello all!
>>>>>>>>>
>>>>>>>>> On my replica IPA server after fixing a cert issue that had 
>>>>>>>>> been going on for sometime, I have all my certs figured out 
>>>>>>>>> but the krb5kdc service will not start.
>>>>>>>>>
>>>>>>>>> # service krb5kdc start
>>>>>>>>> Starting Kerberos 5 KDC: krb5kdc: cannot initialize realm
>>>>>>>>> ITMODEV.GOV - see log file for details                  [FAILED]
>>>>>>>>>
>>>>>>>>> # cat /var/log/krb5kdc.log
>>>>>>>>> krb5kdc: Server error - while fetching master key K/M for 
>>>>>>>>> realm ITMODEV.GOV
>>>>>>>>> krb5kdc: Server error - while fetching master key K/M for 
>>>>>>>>> realm ITMODEV.GOV
>>>>>>>>> krb5kdc: Server error - while fetching master key K/M for 
>>>>>>>>> realm ITMODEV.GOV
>>>>>>>>>
>>>>>>>>> I found this article online:
>>>>>>>>> http://research.imb.uq.edu.au/~l.rathbone/ldap/kerberos.shtml
>>>>>>>>>
>>>>>>>>> Which stated it might be because The slave KDC does not have a 
>>>>>>>>> stash file (.k5.EXAMPLE.COM). You need to create one. Tried 
>>>>>>>>> the command
>>>>>>>>> listed:
>>>>>>>>>
>>>>>>>>> # kdb5_util stash
>>>>>>>>> kdb5_util: Server error while retrieving master entry
>>>>>>>>>
>>>>>>>>> No further information found on the proceeding error above for 
>>>>>>>>> the kdb5_util command.
>>>>>>>>>
>>>>>>>>> Any thoughts?
>>>>>>>> First: don't use instructions which are not related to IPA, please.
>>>>>>>>
>>>>>>>> FreeIPA has its own LDAP driver for KDC and instructions for 
>>>>>>>> anything else do not apply here at all.
>>>>>>>>
>>>>>>>> If you see 'Server error - while fetching master key ..' it 
>>>>>>>> means KDC LDAP driver was unable to contact LDAP server. Does 
>>>>>>>> LDAP server work on the replica? What is in its error log 
>>>>>>>> (/var/log/dirsrv/slapd-ITMODEV-GOV/errors)?
>>>>>>>>
>>>>>>>> --
>>>>>>>> / Alexander Bokovoy
>>>>>>>>
>>>>>>>>
>>>>> --
>>>>> / Alexander Bokovoy
>>>>>
>>> --
>>> Manage your subscription for the Freeipa-users mailing list:
>>> https://www.redhat.com/mailman/listinfo/freeipa-users
>>> Go to http://freeipa.org for more info on the project
>>>
>
>

--
Manage your subscription for the Freeipa-users mailing list:
https://www.redhat.com/mailman/listinfo/freeipa-users
Go to http://freeipa.org for more info on the project