[Freeipa-users] FreeIPA unresponsive - Causes DOS situations
Rich Megginson
rmeggins at redhat.com
Tue Nov 11 18:03:06 UTC 2014
On 11/11/2014 10:37 AM, Martin Basti wrote:
> On 11/11/14 15:58, Rich Megginson wrote:
>> On 11/11/2014 06:20 AM, Ludwig Krispenz wrote:
>>>
>>> On 11/11/2014 02:14 PM, Martin Basti wrote:
>>>> Ludiwg (CCed) this seems like old (fixed?) DS bug.
>>> hmm, it says limit is 2097152, so it already has the new setting,
>>> but the error message says the packet is 800MB*
>>> *
>>
>> *Right. That usually means the server was expecting an encrypted
>> SASL buffer from the client, but instead the client thinks SASL
>> encryption negotiation failed and just sent a plain LDAP buffer.
>> What version of 389-ds-base are you using? rpm -q 389-ds-base
>>
>> https://fedorahosted.org/389/ticket/47416
>>
>> So, DO NOT increase your sasl io buffer size - it will not fix the
>> problem, and it will leave you open to DoS attacks.
>> *
>
> He is using
>
> CentOS release 6.5 (Final)
> 389-ds-base.x86_64 1.2.11.15-34.el6_5
Ignore the "SASL encrypted packet length exceeds maximum allowed limit"
error. All it means is the client has some error and is canceling the
connection.
The bugzilla associated with *47416 is targeted for RHEL 7.1. The
problem is that we were never able to figure out what error the client
was getting that caused the client to stop establishing the *SASL
encryption, and the original customer who reported the problem dropped
the case - everything just started working, no further errors. Note
that 47416 doesn't really fix the problem or address the root cause -
the root cause is some error on the client side that causes it to stop
doing encrypted SASL I/O and send an LDAP unbind request.
Let's get back to the actual problem - what is the actual problem? The
ldap server is hanging or unresponsive? If so, then see
http://www.port389.org/docs/389ds/FAQ/faq.html#debugging-hangs. Let's
get some dirsrv stack traces during the period of time when it appears
to be unresponsive.
>
>> *
>> *
>>> **
>>>>
>>>> On 11/11/14 13:13, Walter van Lille wrote:
>>>>> I've just cleaned out a ton of slapd_poll timed out messages from
>>>>> the output and changed the names to protect the innocent, :-)
>>>>> Here is the output as requested:
>>>>>
>>>>>
>>>>> *[05/Nov/2014:11:44:05 +0200] - SASL encrypted packet length
>>>>> exceeds maximum allowed limit (length=805565, limit=2097152).
>>>>> Change the nsslapd-maxsasliosize attribute in cn=config to
>>>>> increase limit.*
>>>>> *
>>>>> *
>>>>> *[10/Nov/2014:14:45:19 +0200] - slapd_poll(115) timed out*
>>>>> *[10/Nov/2014:14:45:19 +0200] sasl_io_enable - Cannot enable SASL
>>>>> security on connection in CLOSING state*
>>>>> *[10/Nov/2014:14:45:19 +0200] - Error: could not add/remove IO
>>>>> layers from connection*
>>>>> *[11/Nov/2014:11:48:09 +0200] - slapd shutting down - signaling
>>>>> operation threads*
>>>>> *[11/Nov/2014:11:48:09 +0200] - slapd shutting down - waiting for
>>>>> 30 threads to terminate*
>>>>> *[11/Nov/2014:13:14:12 +0200] - slapd shutting down - closing down
>>>>> internal subsystems and plugins*
>>>>> *[11/Nov/2014:13:14:12 +0200] - Waiting for 4 database threads to
>>>>> stop*
>>>>> *[11/Nov/2014:13:14:13 +0200] - All database threads now stopped*
>>>>> *[11/Nov/2014:13:14:13 +0200] - slapd stopped.*
>>>>> *[11/Nov/2014:13:26:35 +0200] - 389-Directory/1.2.11.15
>>>>> <http://1.2.11.15> B2014.219.179 starting up*
>>>>> *[11/Nov/2014:13:26:35 +0200] schema-compat-plugin - warning: no
>>>>> entries set up under cn=computers, cn=compat,dc=sample,dc=example*
>>>>> *[11/Nov/2014:13:26:36 +0200] - Skipping CoS Definition
>>>>> cn=Password Policy,cn=accounts,dc=sample,dc=example--no CoS
>>>>> Templates found, which should be added before the CoS Definition.*
>>>>> *[11/Nov/2014:13:26:36 +0200] - Skipping CoS Definition
>>>>> cn=Password Policy,cn=accounts,dc=sample,dc=example--no CoS
>>>>> Templates found, which should be added before the CoS Definition.*
>>>>> *[11/Nov/2014:13:26:36 +0200] - slapd started. Listening on All
>>>>> Interfaces port 389 for LDAP requests*
>>>>> *[11/Nov/2014:13:26:36 +0200] - Listening on All Interfaces port
>>>>> 636 for LDAPS requests*
>>>>> *[11/Nov/2014:13:26:36 +0200] - Listening on
>>>>> /var/run/slapd-SAMPLE-EXAMPLE.socket for LDAPI requests*
>>>>> *[11/Nov/2014:13:57:08 +0200] - slapd_poll(78) timed out*
>>>>> *
>>>>> *
>>>>> *
>>>>> *
>>>>> *
>>>>> *
>>>>>
>>>>>
>>>>> On Tue, Nov 11, 2014 at 1:19 PM, Martin Basti <mbasti at redhat.com
>>>>> <mailto:mbasti at redhat.com>> wrote:
>>>>>
>>>>> IMHO It's DS bug, can you share DS error log?
>>>>> pspacek CCed to examine named logs.
>>>>>
>>>>> Martin^2
>>>>>
>>>>>
>>>>> On 11/11/14 12:13, Walter van Lille wrote:
>>>>>> Hi Martin, thanks for the reply.
>>>>>> My version: bind-dyndb-ldap-2.3-5.el6.x86_64
>>>>>> The server doesn't have journalctl installed but I have the
>>>>>> outputs from the messages and named.run files that I included
>>>>>> here:
>>>>>>
>>>>>> Messages:
>>>>>>
>>>>>> *Nov 11 12:30:13 freeipa named[1481]: error (network
>>>>>> unreachable) resolving
>>>>>> 'example.example.com.10.123.123.123/A/IN': 2001:500:2f::f#53*
>>>>>> *Nov 11 12:30:23 freeipa named[1481]: LDAP query timed out.
>>>>>> Try to adjust "timeout" parameter*
>>>>>> *Nov 11 12:30:23 freeipa named[1481]: LDAP query timed out.
>>>>>> Try to adjust "timeout" parameter*
>>>>>> *Nov 11 12:30:33 freeipa named[1481]: LDAP query timed out.
>>>>>> Try to adjust "timeout" parameter*
>>>>>> *Nov 11 12:30:33 freeipa named[1481]: LDAP query timed out.
>>>>>> Try to adjust "timeout" parameter*
>>>>>>
>>>>>> Named.run:
>>>>>>
>>>>>> *client 10.123.123.123#42639: transfer of
>>>>>> 'example.example/IN': AXFR-style IXFR started*
>>>>>> *client 10.123.123.123#42639: transfer of
>>>>>> ''example.example/IN': AXFR-style IXFR ended*
>>>>>> *client 10.123.123.123#46912: transfer of
>>>>>> '10.123.123.123.in-addr.arpa/IN': AXFR-style IXFR started*
>>>>>> *client 10.123.123.123#46912: transfer of
>>>>>> '10.123.123.123.in-addr.arpa/IN': AXFR-style IXFR ended*
>>>>>> *LDAP query timed out. Try to adjust "timeout" parameter*
>>>>>> *LDAP query timed out. Try to adjust "timeout" parameter*
>>>>>> *LDAP query timed out. Try to adjust "timeout" parameter*
>>>>>>
>>>>>> I just replaced the IPs and the actual names with something
>>>>>> more generic.
>>>>>>
>>>>>> Regards,
>>>>>>
>>>>>> Walter
>>>>>>
>>>>>> On Thu, Nov 6, 2014 at 5:00 PM, Martin Basti
>>>>>> <mbasti at redhat.com <mailto:mbasti at redhat.com>> wrote:
>>>>>>
>>>>>> On 06/11/14 14:58, Walter van Lille wrote:
>>>>>>> Hi,
>>>>>>>
>>>>>>> I need some assistance please.
>>>>>>> I've taken over an IPA server to manage a few months
>>>>>>> ago, and it was working fine until recently when it
>>>>>>> started acting up seemingly off its own accord.
>>>>>>> When I do an ipactl status it basically gives an output
>>>>>>> as shown below:
>>>>>>>
>>>>>>>
>>>>>>> *Directory Service: RUNNING
>>>>>>> *
>>>>>>> *
>>>>>>> *
>>>>>>> *Loooooooooooooooooooooooooooooooooooooooooooooooooong
>>>>>>> pause... (To the tune of 7 minutes sometimes)*
>>>>>>> *
>>>>>>> *
>>>>>>> *KDC Service: RUNNING*
>>>>>>> *KPASSWD Service: RUNNING*
>>>>>>> *DNS Service: RUNNING*
>>>>>>> *MEMCACHE Service: RUNNING*
>>>>>>> *HTTP Service: RUNNING*
>>>>>>> *CA Service: RUNNING*
>>>>>>> *ADTRUST Service: RUNNING*
>>>>>>> *EXTID Service: RUNNING*
>>>>>>>
>>>>>>> Running top showed that ns-slapd was munching almost all
>>>>>>> my resources, but I got that fixed by upping the cache.
>>>>>>> Unfortunately this did not correct the issue and it
>>>>>>> still reacts in the same fashion, although the resources
>>>>>>> have been freed up now.
>>>>>>> I've noticed that when I run dig on either the local
>>>>>>> server or a remote machine that the query basically just
>>>>>>> times out as shown here:
>>>>>>>
>>>>>>> *dig freeipa.myexample.sample*
>>>>>>> *
>>>>>>> *
>>>>>>> *; <<>> DiG 9.8.2rc1-RedHat-9.8.2-0.23.rc1.el6_5.1 <<>>
>>>>>>> freeipa.myexample.sample*
>>>>>>> *;; global options: +cmd*
>>>>>>> *;; connection timed out; no servers could be reached*
>>>>>>>
>>>>>>> When the KDC service fails to start, then name lookups
>>>>>>> seem OK, but authentication fails. otherwise it's dead
>>>>>>> in the water.
>>>>>>>
>>>>>>> This also happens:
>>>>>>>
>>>>>>> *sudo ipactl status*
>>>>>>> *Directory Service: RUNNING*
>>>>>>> *Unknown error when retrieving list of services from LDAP:*
>>>>>>> *
>>>>>>> *
>>>>>>> My software setup is as follows:
>>>>>>>
>>>>>>> *CentOS release 6.5 (Final)
>>>>>>> *
>>>>>>> *389-ds-base.x86_64 1.2.11.15-34.el6_5
>>>>>>> *
>>>>>>> *bind.x86_64 32:9.8.2-0.23.rc1.el6_5.1
>>>>>>> *
>>>>>>> *bind-dyndb-ldap.x86_64*
>>>>>>> *bind-libs.x86_64 32:9.8.2-0.23.rc1.el6_5.1*
>>>>>>> *bind-utils.x86_64 32:9.8.2-0.23.rc1.el6_5.1*
>>>>>>> *rpcbind.x86_64 0.2.0-11.el6
>>>>>>> @anaconda-CentOS-201311291202.x86_64/6.5*
>>>>>>> *samba4-winbind.x86_64*
>>>>>>> *krb5-server.x86_64 1.10.3-15.el6_5.1
>>>>>>> *
>>>>>>> *
>>>>>>> *
>>>>>>> *Linux 2.6.32-431.29.2.el6.x86_64 #1 SMP Tue Sep 9
>>>>>>> 21:36:05 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
>>>>>>> *
>>>>>>>
>>>>>>> It's not a permanent situation as it sometimes runs 100%
>>>>>>> for a while, but 80% of the time it is unusable. If
>>>>>>> anybody can assist me, please be so kind.
>>>>>>>
>>>>>>> Regards,
>>>>>>>
>>>>>>> Walter
>>>>>>>
>>>>>> Hello please which version of bind-dyndb-ldap do you use?
>>>>>> I had similar issue with bind-dyndb-ldap, but it was
>>>>>> development version, I'm not sure if this is your case.
>>>>>> When named was failing, dirserv was really slow.
>>>>>>
>>>>>> Can you send journalctl -b -u named log when dig doesn't
>>>>>> work??
>>>>>>
>>>>>> --
>>>>>> Martin Basti
>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Martin Basti
>>>>>
>>>>>
>>>>
>>>>
>>>> --
>>>> Martin Basti
>>>
>>>
>>>
>>
>>
>>
>
>
> --
> Martin Basti
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/freeipa-users/attachments/20141111/42ea3239/attachment.htm>
More information about the Freeipa-users
mailing list