[Freeipa-users] IPA Master Issue - Not starting

Wed Aug 20 08:13:00 UTC 2014

On 20.8.2014 10:02, Peter Grant wrote:
> Hi Petr,
>
> Thanks for your help the other day.
>
> Something is bringing down my master instance.
>
> i am seeing mismatch on master
>
> [root at master init.d]# kvno DNS/master.domain.com at domain.COM
> DNS/master.domain.com at domain.COM: kvno = 8
> [root at master init.d]# klist -kt /etc/named.keytab
> Keytab name: FILE:/etc/named.keytab
> KVNO Timestamp         Principal
> ---- ----------------- --------------------------------------------------------
>    33 08/20/14 16:41:42 DNS/master.domain.com at domain.COM
>    33 08/20/14 16:41:42 DNS/master.domain.com at domain.COM
>    33 08/20/14 16:41:42 DNS/master.domain.com at domain.COM
>    33 08/20/14 16:41:42 DNS/master.domain.com at domain.COM
>    34 08/20/14 16:53:29 DNS/master.domain.com at domain.COM
>    34 08/20/14 16:53:29 DNS/master.domain.com at domain.COM
>    34 08/20/14 16:53:29 DNS/master.domain.com at domain.COM
>    34 08/20/14 16:53:29 DNS/master.domain.com at domain.COM
>    35 08/20/14 16:59:37 DNS/master.domain.com at domain.COM
>    35 08/20/14 16:59:37 DNS/master.domain.com at domain.COM
>    35 08/20/14 16:59:37 DNS/master.domain.com at domain.COM
>    35 08/20/14 16:59:37 DNS/master.domain.com at domain.COM
>    38 08/20/14 17:02:30 DNS/master.domain.com at domain.COM
>    38 08/20/14 17:02:30 DNS/master.domain.com at domain.COM
>    38 08/20/14 17:02:30 DNS/master.domain.com at domain.COM
>    38 08/20/14 17:02:30 DNS/master.domain.com at domain.COM
>    41 08/20/14 17:07:45 DNS/master.domain.com at domain.COM
>    41 08/20/14 17:07:45 DNS/master.domain.com at domain.COM
>    41 08/20/14 17:07:45 DNS/master.domain.com at domain.COM
>    41 08/20/14 17:07:45 DNS/master.domain.com at domain.COM
>    42 08/20/14 17:13:17 DNS/master.domain.com at domain.COM
>    42 08/20/14 17:13:17 DNS/master.domain.com at domain.COM
>    42 08/20/14 17:13:17 DNS/master.domain.com at domain.COM
>    42 08/20/14 17:13:17 DNS/master.domain.com at domain.COM
>    45 08/20/14 17:20:34 DNS/master.domain.com at domain.COM
>    45 08/20/14 17:20:34 DNS/master.domain.com at domain.COM
>    45 08/20/14 17:20:34 DNS/master.domain.com at domain.COM
>    45 08/20/14 17:20:34 DNS/master.domain.com at domain.COM
>    46 08/20/14 17:35:00 DNS/master.domain.com at domain.COM
>    46 08/20/14 17:35:00 DNS/master.domain.com at domain.COM
>    46 08/20/14 17:35:00 DNS/master.domain.com at domain.COM
>    46 08/20/14 17:35:00 DNS/master.domain.com at domain.COM
>    47 08/20/14 17:37:43 DNS/master.domain.com at domain.COM
>    47 08/20/14 17:37:43 DNS/master.domain.com at domain.COM
>    47 08/20/14 17:37:43 DNS/master.domain.com at domain.COM
>    47 08/20/14 17:37:43 DNS/master.domain.com at domain.COM
>    48 08/20/14 17:41:42 DNS/master.domain.com at domain.COM
>    48 08/20/14 17:41:42 DNS/master.domain.com at domain.COM
>    48 08/20/14 17:41:42 DNS/master.domain.com at domain.COM
>    48 08/20/14 17:41:42 DNS/master.domain.com at domain.COM
>    49 08/20/14 17:43:43 DNS/master.domain.com at domain.COM
>    49 08/20/14 17:43:44 DNS/master.domain.com at domain.COM
>    49 08/20/14 17:43:44 DNS/master.domain.com at domain.COM
>    49 08/20/14 17:43:44 DNS/master.domain.com at domain.COM
> [root at master init.d]#
>
>
> also here is output from /var/log/messages whilst trying to ipactl start
>
>
>
> [root at master init.d]# sudo ipactl start
> Starting Directory Service
> Starting dirsrv:
>      domain-COM...                                   [  OK  ]
>      PKI-IPA...                                             [  OK  ]
> Starting KDC Service
> Starting Kerberos 5 KDC:                                   [  OK  ]
> Starting KPASSWD Service
> Starting Kerberos 5 Admin Server:                          [  OK  ]
> Starting DNS Service
> Starting named: 2014-08-20T18:00:22.098747+10:00 master named[20827]: starting BIND 9.8.2rc1-RedHat-9.8.2-0.23.rc1.el6_5.1 -u named
> 2014-08-20T18:00:22.099552+10:00 master named[20827]: built with '--build=x86_64-redhat-linux-gnu' '--host=x86_64-redhat-linux-gnu' '--target=x86_64-redhat-linux-gnu' '--program-prefix=' '--prefix=/usr' '--exec-prefix=/usr' '--bindir=/usr/bin' '--sbindir=/usr/sbin' '--sysconfdir=/etc' '--datadir=/usr/share' '--includedir=/usr/include' '--libdir=/usr/lib64' '--libexecdir=/usr/libexec' '--sharedstatedir=/var/lib' '--mandir=/usr/share/man' '--infodir=/usr/share/info' '--with-libtool' '--localstatedir=/var' '--enable-threads' '--enable-ipv6' '--with-pic' '--disable-static' '--disable-openssl-version-check' '--with-dlz-ldap=yes' '--with-dlz-postgres=yes' '--with-dlz-mysql=yes' '--with-dlz-filesystem=yes' '--with-gssapi=yes' '--disable-isc-spnego' '--with-docbook-xsl=/usr/share/sgml/docbook/xsl-stylesheets' '--enable-fixed-rrset' 'build_alias=x86_64-redhat-linux-gnu' 'host_alias=x86_64-redhat-linux-gnu' 'target_alias=x86_64-redhat-linux-gnu' 'CFLAGS= -O2 -g -pipe -Wall -Wp,-D_FO!
 RTIFY_SOUR
CE=2 -fexceptions -fstack-protector --param=ssp-buffer-size=4 -m64 -mtune=generic' 'CPPFLAGS= -DDIG_SIGCHASE'
> 2014-08-20T18:00:22.099633+10:00 master named[20827]: ----------------------------------------------------
> 2014-08-20T18:00:22.099688+10:00 master named[20827]: BIND 9 is maintained by Internet Systems Consortium,
> 2014-08-20T18:00:22.099750+10:00 master named[20827]: Inc. (ISC), a non-profit 501(c)(3) public-benefit
> 2014-08-20T18:00:22.099803+10:00 master named[20827]: corporation.  Support and training for BIND 9 are
> 2014-08-20T18:00:22.099864+10:00 master named[20827]: available at https://www.isc.org/support
> 2014-08-20T18:00:22.099925+10:00 master named[20827]: ----------------------------------------------------
> 2014-08-20T18:00:22.099998+10:00 master named[20827]: adjusted limit on open files from 62000 to 1048576
> 2014-08-20T18:00:22.100207+10:00 master named[20827]: found 1 CPU, using 1 worker thread
> 2014-08-20T18:00:22.100484+10:00 master named[20827]: using up to 4096 sockets
> 2014-08-20T18:00:22.103796+10:00 master named[20827]: loading configuration from '/etc/named.conf'
> 2014-08-20T18:00:22.104495+10:00 master named[20827]: using default UDP/IPv4 port range: [1024, 65535]
> 2014-08-20T18:00:22.104728+10:00 master named[20827]: using default UDP/IPv6 port range: [1024, 65535]
> 2014-08-20T18:00:22.106090+10:00 master named[20827]: listening on IPv6 interfaces, port 53
> 2014-08-20T18:00:22.108167+10:00 master named[20827]: listening on IPv4 interface lo, 127.0.0.1#53
> 2014-08-20T18:00:22.108571+10:00 master named[20827]: listening on IPv4 interface eth0, 10.3.11.16#53
> 2014-08-20T18:00:22.109760+10:00 master named[20827]: generating session key for dynamic DNS
> 2014-08-20T18:00:22.109997+10:00 master named[20827]: sizing zone task pool based on 5 zones
> 2014-08-20T18:00:22.112660+10:00 master named[20827]: set up managed keys zone for view _default, file 'dynamic/managed-keys.bind'
> 2014-08-20T18:00:22.129607+10:00 master named[20827]: Failed to init credentials (Generic preauthentication failure)
> 2014-08-20T18:00:22.130031+10:00 master named[20827]: loading configuration: failure
> 2014-08-20T18:00:22.130285+10:00 master named[20827]: exiting (due to fatal error)
>                                                             [FAILED]
> Failed to start DNS Service
> Shutting down
> Stopping Kerberos 5 KDC:                                   [  OK  ]
> Stopping Kerberos 5 Admin Server: 2014-08-20T18:00:23.833115+10:00 master ns-slapd: GSSAPI Error: Unspecified GSS failure.  Minor code may provide more information (Server krbtgt/LOCALDOMAIN at domain.COM not found in Kerberos database)

This seems to be more serious - I suspect that replication between replicas 
doesn't work because replica is not able to authenticate.

The error message is suspicious but I'm not sure that it is not result of 
obfuscation. Please try to apply this article to ns-slapd on your broken master:

https://fedorahosted.org/bind-dyndb-ldap/wiki/BIND9/NamedCannotStart#a2.Serverldapsrv01EXAMPLE.COMnotfoundinKerberosdatabase

Maybe /etc/hosts is somehow misconfigured.

>                                                             [  OK  ]
> Stopping named:                                            [  OK  ]
> Stopping httpd:                                            [FAILED]
> Stopping pki-ca:                                           [  OK  ]
> Shutting down dirsrv:
>      domain-COM...                                   [  OK  ]
>      PKI-IPA...                                             [  OK  ]
> Aborting ipactl
> [root at master init.d]#
>
> however there is still a mismatch when i try to get key tab from secondary using command
> ipa-getkeytab -s secondary.domain.com -p DNS/master.domain.com at domain.COM -k /etc/named.keytab

Maybe it is caused by broken replication (one KDC have different keys than the 
other KDC). I would start with replication problems and focus on named later.

Petr^2 Spacek

>
> i am unable to regenerate the key tab on the master as ldap is not running.
>
>
> Any ideas?
>
>
> Thankyou,
>
> Peter.
>
>
>> On 15 Aug 2014, at 5:10 pm, Petr Spacek <pspacek at redhat.com> wrote:
>>
>> Hello,
>>
>> On 15.8.2014 03:52, Peter Grant wrote:
>>> 2014-08-15T11:43:46.434383+10:00 host named[6470]: Failed to init credentials (Decrypt integrity check failed)
>>>
>>> 2014-08-15T11:43:46.434884+10:00 host named[6470]: loading configuration: failure
>>>
>>> 2014-08-15T11:43:46.434991+10:00 host named[6470]: exiting (due to fatal error)
>>>
>>> 2014-08-15T11:43:47.435187+10:00 host ns-slapd: GSSAPI Error: Unspecified GSS failure.  Minor code may provide more information (Cannot contact any KDC for realm ‘DOMAIN.COM')
>>
>> For named issue please follow instructions on
>> https://fedorahosted.org/bind-dyndb-ldap/wiki/BIND9/NamedCannotStart#a3.FailedtoinitcredentialsorFailedtogetinitialcredentialsDecryptintegritycheckfailedorClientscredentialshavebeenrevoked
>>
>> It seems that /etc/named.keytab is somehow corrupted or obsolete.
>>
>> Also, KDC logs in /var/log/krb5kdc.log can tell you more.
>>
>> I hope that others will add ideas about other errors.