[Freeipa-users] Server randomly will stop accepting krb requests

[Andrew Tranquada]

I have 6 servers setup as freeipa replicas. 
5 are working great, no problems.
They are all running ipa-server-3.0.0-26.el6_4.4.x86_64
However, the same one will randomly stop working. By stop working I mean the following:
(domain name and ips have been redacted)

I cannot kinit as any user on that machine:
[root at badserver ~]# kinit admin
kinit: Generic error (see e-text) while getting initial credentials

I cannot connect on 389 or 636 to that server:

 telnet badserver 636

telnet: Unable to connect to remote host: Connection refused

slapd is running and listening on port 389 according to netstat:
[root at badserver ~]# netstat -lpn | grep 389
tcp        0      0 :::7389                     :::*                        LISTEN      16419/ns-slapd

but nothing is returned for port 636


in the /var/log/slapd-PKI* or slapd-<DOMAIN> error files, the last error is from over a week ago, actually the last entry period is from there. 

[18/Sep/2013:01:09:34 -0400] slapd_ldap_sasl_interactive_bind - Error: could not perform interactive bind for id [] mech [GSSAPI]: LDAP error -2 (Local error) (SASL(-1): generic failure: GSSAPI Error: Unspecified GSS failure.  Minor code may provide more information (KDC returned error string: PROCESS_TGS)) errno 2 (No such file or directory)


/var/log/krb5kdc.log shows
Sep 30 12:22:24 badserver krb5kdc[32063](info): AS_REQ (4 etypes {18 17 16 23}) <ip>: LOOKING_UP_CLIENT: admin at EXAMPLE.COM for krbtgt/EXAMPLE.COM at EXAMPLE.COM, Server error

a service ipa restart ALWAYS fixes it. 
I added debug=true to /etc/ipa/default.conf but I do not see anything that is helpful.
The only things listed in default.conf are things related to "importing plugin module"


Any guidance/advice/docs to read would be greatly appreciated! The fact that it seems to be so random and the other 5 ipa servers are working great makes it even more frustrating!

Thanks!