[Freeipa-users] FreeIPA "crashes" after many mystery connections

[Andy Singleton]

There isn't much in the krb5kdc.logs.
Server A has a few entries about a minute before the incident. Then nothing until we had to reboot the box.

<krb5kdc.log>
Oct 22 12:27:53 a.office.tipp24.de krb5kdc[2114](info): TGS_REQ (1 etypes {18}) 192.168.0.11: IS
SUE: authtime 1255946532, etypes {rep=18 tkt=18 ses=18}, user1 at LIVE.TIPP24.NET for krbtgt/LIVE.TIPP2
4.NET at LIVE.TIPP24.NET
Oct 22 12:28:08 a.office.tipp24.de krb5kdc[2114](info): AS_REQ (7 etypes {18 17 16 23 1 3 2}) 192.168.0.12: CLIENT_NOT_FOUND: root at LIVE.TIPP24.NET for krbtgt/LIVE.TIPP24.NET at LIVE.TIPP24.NET, Clien
t not found in Kerberos database
Oct 22 12:28:13 a.office.tipp24.de krb5kdc[2114](info): AS_REQ (7 etypes {18 17 16 23 1 3 2}) 192.168.0.12: NEEDED_PREAUTH: user1 at LIVE.TIPP24.NET for krbtgt/LIVE.TIPP24.NET at LIVE.TIPP24.NET, Additi
onal pre-authentication required
Oct 22 12:28:13 a.office.tipp24.de krb5kdc[2114](info): AS_REQ (7 etypes {18 17 16 23 1 3 2}) 192.168.0.12: ISSUE: authtime 1256207293, etypes {rep=18 tkt=18 ses=18}, user1 at LIVE.TIPP24.NET for krb
tgt/LIVE.TIPP24.NET at LIVE.TIPP24.NET
Oct 22 13:21:40 a.office.tipp24.de krb5kdc[2080](info): setting up network...
<end>

Server B has even less: No entries for an hour before it gets the same problem.

<krb5kdc.log>
Oct 22 11:32:34 b.office.tipp24.de krb5kdc[11838](info): AS_REQ (7 etypes {18 17 16 23 1 3 2})
 192.168.0.10: NEEDED_PREAUTH: user2 at LIVE.TIPP24.NET for krbtgt/LIVE.TIPP24.NET at LIVE.TIPP24.NET, 
Additional pre-authentication required
Oct 22 11:32:34 b.office.tipp24.de krb5kdc[11838](info): AS_REQ (7 etypes {18 17 16 23 1 3 2})
 192.168.0.10: ISSUE: authtime 1256203954, etypes {rep=18 tkt=18 ses=18}, user2 at LIVE.TIPP24.NET f
or krbtgt/LIVE.TIPP24.NET at LIVE.TIPP24.NET
</end>


All hostnames and users have been changed to protect the innocent.
Andy

-----Original Message-----
From: Simo Sorce [mailto:ssorce at redhat.com] 
Sent: 22 October 2009 18:02
To: Andy Singleton
Cc: freeipa-users at redhat.com
Subject: Re: [Freeipa-users] FreeIPA "crashes" after many mystery connections

On Thu, 2009-10-22 at 16:22 +0100, Andy Singleton wrote:
> Hello,
> 
>  
> 
> I am trying to solve a mystery. We have 2 replicated FreeIPA servers.
> 
> Today they both stopped receiving requests because the Directory
> Server had begun to refuse connections.
> 
> The relevant message is “Not listening for new connections - too many
> fds open”
> 
>  
> 
> That’s all well and good: I can increase the file descriptor
> allowance.
> 
> However, the reason the fds limit was reached was a massive number of
> connections from the servers themselves.
> 
> Can someone provide me with an idea for what this might be?
> 
>  
> 
> We received 1024 connections in under 1 second: Here is an example
> dirsrv access log entry:
> 
>  
> 
> [22/Oct/2009:12:29:53 +0200] conn=679021 fd=464 slot=464 connection
> from 127.0.0.1 to 127.0.0.1
> 
> [22/Oct/2009:12:29:53 +0200] conn=679021 op=0 BIND
> dn="uid=kdc,cn=sysaccounts,cn=etc,dc=live,dc=tipp
> 
> 24,dc=net" method=128 version=3
> 
> [22/Oct/2009:12:29:53 +0200] conn=679021 op=0 RESULT err=0 tag=97
> nentries=0 etime=0 dn="uid=kdc,cn=
> 
> sysaccounts,cn=etc,dc=live,dc=tipp24,dc=net"
> 
>  
> 
>  
> 
> Some final notes:
> 
> Both servers stopped one after the other. First server A, then 1
> second afterwards, server B.
> 
>  
> 
> I’m pretty stuck as to what might have caused this.

Can you check the krb5kdc logs ?

dn="uid=kdc,cn=sysaccounts,cn=etc,dc=live,dc=tipp24,dc=net" is the
account used by the kdc (in v1). So it looks like the KDC went crazy
trying to connect to the ldap server.

Simo.

-- 
Simo Sorce * Red Hat, Inc * New York