[Freeipa-users] Kerberos hanging approx. once a day

Torsten Harenberg harenberg at physik.uni-wuppertal.de
Wed Jul 22 09:06:53 UTC 2015 

Dear community,

we just moved our infrastructure (about 200 node cluster plus about 30
workstations) from NIS to FreeIPA (version 4.1.4 on FC 21).

We have two IPA servers (called "ipa" and "ipa2" both paravirtualized on
Xen4).

Approx once a day, the Kerberos service on the primary server suddenly
stops working and I am unable to re-start the service. Only a "full"
reboot helps and during that, the Kerberos shutdown takes about 2
minutes (unsure if it really finishes or if it's the final timeout of
the shutdown script).

Trying to collect as many log messages as possible:


Jul 22 10:52:06 ipa.pleiades.uni-wuppertal.de krb5kdc[1114](info):
AS_REQ (4 etypes {18 17 16 23}) 132.195.124.213: LOOKING_UP_CLIENT:
host/proton.pleiades.uni-wuppertal.de at PLEIADES.UNI-WUPPERTAL.DE for
krbtgt/PLEIADES.UNI-WUPPERTAL.DE at PLEIADES.UNI-WUPPERTAL.DE, Server error
Jul 22 10:52:11 ipa.pleiades.uni-wuppertal.de krb5kdc[1114](info):
AS_REQ (4 etypes {18 17 16 23}) 132.195.125.171: LOOKING_UP_CLIENT:
host/wn161.pleiades.uni-wuppertal.de at PLEIADES.UNI-WUPPERTAL.DE for
krbtgt/PLEIADES.UNI-WUPPERTAL.DE at PLEIADES.UNI-WUPPERTAL.DE, Server error


[root at ipa ~]# systemctl status krb5kdc.service
● krb5kdc.service - Kerberos 5 KDC
   Loaded: loaded (/usr/lib/systemd/system/krb5kdc.service; disabled)
   Active: failed (Result: exit-code) since Mi 2015-07-22 10:54:22 CEST;
10s ago
  Process: 11910 ExecStart=/usr/sbin/krb5kdc -P /var/run/krb5kdc.pid
$KRB5KDC_ARGS (code=exited, status=1/FAILURE)
 Main PID: 1114 (code=exited, status=0/SUCCESS)

Jul 22 10:54:22 ipa.pleiades.uni-wuppertal.de krb5kdc[11910]: krb5kdc:
cannot initialize realm PLEIADES.UNI-WUPPERTAL.DE - see log file for details
Jul 22 10:54:22 ipa.pleiades.uni-wuppertal.de systemd[1]:
krb5kdc.service: control process exited, code=exited status=1
Jul 22 10:54:22 ipa.pleiades.uni-wuppertal.de systemd[1]: Failed to
start Kerberos 5 KDC.
Jul 22 10:54:22 ipa.pleiades.uni-wuppertal.de systemd[1]: Unit
krb5kdc.service entered failed state.
Jul 22 10:54:22 ipa.pleiades.uni-wuppertal.de systemd[1]:
krb5kdc.service failed.
[root at ipa ~]# tail -f /var/log/krb5kdc.log
Jul 22 10:52:11 ipa.pleiades.uni-wuppertal.de krb5kdc[1114](info):
closing down fd 13
Jul 22 10:52:11 ipa.pleiades.uni-wuppertal.de krb5kdc[1114](info):
DISPATCH: repeated (retransmitted?) request from 132.195.124.213,
resending previous response
Jul 22 10:52:11 ipa.pleiades.uni-wuppertal.de krb5kdc[1114](info):
DISPATCH: repeated (retransmitted?) request from 132.195.125.171,
resending previous response
Jul 22 10:54:12 ipa.pleiades.uni-wuppertal.de krb5kdc[1114](debug): Got
signal to request exit
Jul 22 10:54:12 ipa.pleiades.uni-wuppertal.de krb5kdc[1114](info):
closing down fd 10
Jul 22 10:54:12 ipa.pleiades.uni-wuppertal.de krb5kdc[1114](info):
closing down fd 11
Jul 22 10:54:12 ipa.pleiades.uni-wuppertal.de krb5kdc[1114](info):
closing down fd 9
Jul 22 10:54:12 ipa.pleiades.uni-wuppertal.de krb5kdc[1114](info):
closing down fd 8
Jul 22 10:54:12 ipa.pleiades.uni-wuppertal.de krb5kdc[1114](info):
shutting down
krb5kdc: Server error - while fetching master key K/M for realm
PLEIADES.UNI-WUPPERTAL.DE


[root at ipa ~]# journalctl -xe
Jul 22 10:53:15 ipa.pleiades.uni-wuppertal.de ntpd[11903]: Listen and
drop on 1 v6wildcard :: UDP 123
Jul 22 10:53:15 ipa.pleiades.uni-wuppertal.de ntpd[11903]: Listen
normally on 2 lo 127.0.0.1 UDP 123
Jul 22 10:53:15 ipa.pleiades.uni-wuppertal.de ntpd[11903]: Listen
normally on 3 eth0 132.195.124.12 UDP 123
Jul 22 10:53:15 ipa.pleiades.uni-wuppertal.de ntpd[11903]: Listen
normally on 4 lo ::1 UDP 123
Jul 22 10:53:15 ipa.pleiades.uni-wuppertal.de ntpd[11903]: Listen
normally on 5 eth0 fe80::216:3eff:fe14:c27a UDP 123
Jul 22 10:53:15 ipa.pleiades.uni-wuppertal.de ntpd[11903]: Listening on
routing socket on fd #22 for interface updates
Jul 22 10:53:15 ipa.pleiades.uni-wuppertal.de systemd[1]: Started
Network Time Service.
-- Subject: Unit ntpd.service has finished start-up
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
--
-- Unit ntpd.service has finished starting up.
--
-- The start-up result is done.
Jul 22 10:53:15 ipa.pleiades.uni-wuppertal.de ntpd[11903]: 0.0.0.0 c016
06 restart
Jul 22 10:53:15 ipa.pleiades.uni-wuppertal.de ntpd[11903]: 0.0.0.0 c012
02 freq_set ntpd -23.557 PPM
Jul 22 10:53:16 ipa.pleiades.uni-wuppertal.de ntpd[11903]: 0.0.0.0 c615
05 clock_sync
Jul 22 10:54:12 ipa.pleiades.uni-wuppertal.de systemd[1]: Stopping
Kerberos 5 KDC...
-- Subject: Unit krb5kdc.service has begun shutting down
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
--
-- Unit krb5kdc.service has begun shutting down.
Jul 22 10:54:12 ipa.pleiades.uni-wuppertal.de systemd[1]: Starting
Kerberos 5 KDC...
-- Subject: Unit krb5kdc.service has begun with start-up
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
--
-- Unit krb5kdc.service has begun starting up.
Jul 22 10:54:22 ipa.pleiades.uni-wuppertal.de krb5kdc[11910]: krb5kdc:
cannot initialize realm PLEIADES.UNI-WUPPERTAL.DE - see log file for details
Jul 22 10:54:22 ipa.pleiades.uni-wuppertal.de systemd[1]:
krb5kdc.service: control process exited, code=exited status=1
Jul 22 10:54:22 ipa.pleiades.uni-wuppertal.de systemd[1]: Failed to
start Kerberos 5 KDC.
-- Subject: Unit krb5kdc.service has failed
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
--
-- Unit krb5kdc.service has failed.
--
-- The result is failed.
Jul 22 10:54:22 ipa.pleiades.uni-wuppertal.de systemd[1]: Unit
krb5kdc.service entered failed state.
Jul 22 10:54:22 ipa.pleiades.uni-wuppertal.de systemd[1]:
krb5kdc.service failed.
[root at ipa ~]#


[root at ipa ~]# rpm -qi freeipa-server
Name        : freeipa-server
Version     : 4.1.4
Release     : 1.fc21
Architecture: x86_64
Install Date: Di 28 Apr 2015 14:30:33 CEST
Group       : System Environment/Base
Size        : 4521059
License     : GPLv3+
Signature   : RSA/SHA256, Do 26 Mär 2015 23:58:02 CET, Key ID
89ad4e8795a43f54
Source RPM  : freeipa-4.1.4-1.fc21.src.rpm
Build Date  : Do 26 Mär 2015 16:16:19 CET
Build Host  : buildhw-07.phx2.fedoraproject.org
Relocations : (not relocatable)
Packager    : Fedora Project
Vendor      : Fedora Project
URL         : http://www.freeipa.org/
Summary     : The IPA authentication server
Description :
IPA is an integrated solution to provide centrally managed Identity
(machine,
user, virtual machines, groups, authentication credentials), Policy
(configuration settings, access control information) and Audit (events,
logs, analysis thereof). If you are installing an IPA server you need
to install this package (in other words, most people should NOT install
this package).
[root at ipa ~]#

We already enlarged the capacity of the primary server (now two
exclusive CPU cores and 8 GB RAM).

Any idea is appreciated, we are pretty new to IPA.

Kind regards,

 Torsten


-- 
<><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><>
<>                                                              <>
<> Dr. Torsten Harenberg     harenberg at physik.uni-wuppertal.de  <>
<> Bergische Universitaet                                       <>
<> FB C - Physik             Tel.: +49 (0)202 439-3521          <>
<> Gaussstr. 20              Fax : +49 (0)202 439-2811          <>
<> 42097 Wuppertal                                              <>
<>                                                              <>
<><><><><><><>< Of course it runs NetBSD http://www.netbsd.org ><>

More information about the Freeipa-users mailing list