[Freeipa-users] FreeIPA unresponsive - Causes DOS situations
Walter van Lille
walter.van.lille at gmail.com
Wed Nov 12 12:42:44 UTC 2014
Thanks again for the assistance guys. I have saved two files and included
it here. Hope it tells you more than it does me.
Regards,
Wallter
On Tue, Nov 11, 2014 at 8:03 PM, Rich Megginson <rmeggins at redhat.com> wrote:
> On 11/11/2014 10:37 AM, Martin Basti wrote:
>
> On 11/11/14 15:58, Rich Megginson wrote:
>
> On 11/11/2014 06:20 AM, Ludwig Krispenz wrote:
>
>
> On 11/11/2014 02:14 PM, Martin Basti wrote:
>
> Ludiwg (CCed) this seems like old (fixed?) DS bug.
>
> hmm, it says limit is 2097152, so it already has the new setting, but the
> error message says the packet is 800MB
>
>
>
>
>
>
>
> *Right. That usually means the server was expecting an encrypted SASL
> buffer from the client, but instead the client thinks SASL encryption
> negotiation failed and just sent a plain LDAP buffer. What version of
> 389-ds-base are you using? rpm -q 389-ds-base
> https://fedorahosted.org/389/ticket/47416
> <https://fedorahosted.org/389/ticket/47416> So, DO NOT increase your sasl
> io buffer size - it will not fix the problem, and it will leave you open to
> DoS attacks. *
>
>
> He is using
>
> CentOS release 6.5 (Final)
> 389-ds-base.x86_64 1.2.11.15-34.el6_5
>
>
>
> Ignore the "SASL encrypted packet length exceeds maximum allowed limit"
> error. All it means is the client has some error and is canceling the
> connection.
>
> The bugzilla associated with *47416
> <https://fedorahosted.org/389/ticket/47416> is targeted for RHEL 7.1. The
> problem is that we were never able to figure out what error the client was
> getting that caused the client to stop establishing the *SASL encryption,
> and the original customer who reported the problem dropped the case -
> everything just started working, no further errors. Note that 47416
> doesn't really fix the problem or address the root cause - the root cause
> is some error on the client side that causes it to stop doing encrypted
> SASL I/O and send an LDAP unbind request.
>
> Let's get back to the actual problem - what is the actual problem? The
> ldap server is hanging or unresponsive? If so, then see
> http://www.port389.org/docs/389ds/FAQ/faq.html#debugging-hangs. Let's
> get some dirsrv stack traces during the period of time when it appears to
> be unresponsive.
>
>
>
>
>
> On 11/11/14 13:13, Walter van Lille wrote:
>
> I've just cleaned out a ton of slapd_poll timed out messages from the
> output and changed the names to protect the innocent, :-)
> Here is the output as requested:
>
>
> *[05/Nov/2014:11:44:05 +0200] - SASL encrypted packet length exceeds
> maximum allowed limit (length=805565, limit=2097152). Change the
> nsslapd-maxsasliosize attribute in cn=config to increase limit.*
>
> *[10/Nov/2014:14:45:19 +0200] - slapd_poll(115) timed out*
> *[10/Nov/2014:14:45:19 +0200] sasl_io_enable - Cannot enable SASL security
> on connection in CLOSING state*
> *[10/Nov/2014:14:45:19 +0200] - Error: could not add/remove IO layers from
> connection*
> *[11/Nov/2014:11:48:09 +0200] - slapd shutting down - signaling
> operation threads*
> *[11/Nov/2014:11:48:09 +0200] - slapd shutting down - waiting for 30
> threads to terminate*
> *[11/Nov/2014:13:14:12 +0200] - slapd shutting down - closing down
> internal subsystems and plugins*
> *[11/Nov/2014:13:14:12 +0200] - Waiting for 4 database threads to stop*
> *[11/Nov/2014:13:14:13 +0200] - All database threads now stopped*
> *[11/Nov/2014:13:14:13 +0200] - slapd stopped.*
> *[11/Nov/2014:13:26:35 +0200] - 389-Directory/1.2.11.15 <http://1.2.11.15>
> B2014.219.179 starting up*
> *[11/Nov/2014:13:26:35 +0200] schema-compat-plugin - warning: no entries
> set up under cn=computers, cn=compat,dc=sample,dc=example*
> *[11/Nov/2014:13:26:36 +0200] - Skipping CoS Definition cn=Password
> Policy,cn=accounts,dc=sample,dc=example--no CoS Templates found, which
> should be added before the CoS Definition.*
> *[11/Nov/2014:13:26:36 +0200] - Skipping CoS Definition cn=Password
> Policy,cn=accounts,dc=sample,dc=example--no CoS Templates found, which
> should be added before the CoS Definition.*
> *[11/Nov/2014:13:26:36 +0200] - slapd started. Listening on All
> Interfaces port 389 for LDAP requests*
> *[11/Nov/2014:13:26:36 +0200] - Listening on All Interfaces port 636 for
> LDAPS requests*
> *[11/Nov/2014:13:26:36 +0200] - Listening on
> /var/run/slapd-SAMPLE-EXAMPLE.socket for LDAPI requests*
> *[11/Nov/2014:13:57:08 +0200] - slapd_poll(78) timed out*
>
>
>
>
>
> On Tue, Nov 11, 2014 at 1:19 PM, Martin Basti <mbasti at redhat.com> wrote:
>
>> IMHO It's DS bug, can you share DS error log?
>> pspacek CCed to examine named logs.
>>
>> Martin^2
>>
>>
>> On 11/11/14 12:13, Walter van Lille wrote:
>>
>> Hi Martin, thanks for the reply.
>> My version: bind-dyndb-ldap-2.3-5.el6.x86_64
>> The server doesn't have journalctl installed but I have the outputs from
>> the messages and named.run files that I included here:
>>
>> Messages:
>>
>> *Nov 11 12:30:13 freeipa named[1481]: error (network unreachable)
>> resolving 'example.example.com.10.123.123.123/A/IN': 2001:500:2f::f#53*
>> *Nov 11 12:30:23 freeipa named[1481]: LDAP query timed out. Try to adjust
>> "timeout" parameter*
>> *Nov 11 12:30:23 freeipa named[1481]: LDAP query timed out. Try to adjust
>> "timeout" parameter*
>> *Nov 11 12:30:33 freeipa named[1481]: LDAP query timed out. Try to adjust
>> "timeout" parameter*
>> *Nov 11 12:30:33 freeipa named[1481]: LDAP query timed out. Try to adjust
>> "timeout" parameter*
>>
>> Named.run:
>>
>> *client 10.123.123.123#42639: transfer of 'example.example/IN':
>> AXFR-style IXFR started*
>> *client 10.123.123.123#42639: transfer of ''example.example/IN':
>> AXFR-style IXFR ended*
>> *client 10.123.123.123#46912: transfer of
>> '10.123.123.123.in-addr.arpa/IN': AXFR-style IXFR started*
>> *client 10.123.123.123#46912: transfer of
>> '10.123.123.123.in-addr.arpa/IN': AXFR-style IXFR ended*
>> *LDAP query timed out. Try to adjust "timeout" parameter*
>> *LDAP query timed out. Try to adjust "timeout" parameter*
>> *LDAP query timed out. Try to adjust "timeout" parameter*
>>
>> I just replaced the IPs and the actual names with something more
>> generic.
>>
>> Regards,
>>
>> Walter
>>
>> On Thu, Nov 6, 2014 at 5:00 PM, Martin Basti <mbasti at redhat.com> wrote:
>>
>>> On 06/11/14 14:58, Walter van Lille wrote:
>>>
>>> Hi,
>>>
>>> I need some assistance please.
>>> I've taken over an IPA server to manage a few months ago, and it was
>>> working fine until recently when it started acting up seemingly off its own
>>> accord.
>>> When I do an ipactl status it basically gives an output as shown below:
>>>
>>>
>>>
>>> *Directory Service: RUNNING *
>>>
>>> *Loooooooooooooooooooooooooooooooooooooooooooooooooong pause... (To
>>> the tune of 7 minutes sometimes)*
>>>
>>> *KDC Service: RUNNING*
>>> *KPASSWD Service: RUNNING*
>>> *DNS Service: RUNNING*
>>> *MEMCACHE Service: RUNNING*
>>> *HTTP Service: RUNNING*
>>> *CA Service: RUNNING*
>>> *ADTRUST Service: RUNNING*
>>> *EXTID Service: RUNNING*
>>>
>>> Running top showed that ns-slapd was munching almost all my resources,
>>> but I got that fixed by upping the cache. Unfortunately this did not
>>> correct the issue and it still reacts in the same fashion, although the
>>> resources have been freed up now.
>>> I've noticed that when I run dig on either the local server or a remote
>>> machine that the query basically just times out as shown here:
>>>
>>> *dig freeipa.myexample.sample*
>>>
>>> *; <<>> DiG 9.8.2rc1-RedHat-9.8.2-0.23.rc1.el6_5.1 <<>>
>>> freeipa.myexample.sample*
>>> *;; global options: +cmd*
>>> *;; connection timed out; no servers could be reached*
>>>
>>> When the KDC service fails to start, then name lookups seem OK, but
>>> authentication fails. otherwise it's dead in the water.
>>>
>>> This also happens:
>>>
>>> *sudo ipactl status*
>>> *Directory Service: RUNNING*
>>> *Unknown error when retrieving list of services from LDAP:*
>>>
>>> My software setup is as follows:
>>>
>>>
>>> *CentOS release 6.5 (Final) *
>>>
>>> *389-ds-base.x86_64 1.2.11.15-34.el6_5 *
>>>
>>> *bind.x86_64 32:9.8.2-0.23.rc1.el6_5.1 *
>>> *bind-dyndb-ldap.x86_64*
>>> *bind-libs.x86_64 32:9.8.2-0.23.rc1.el6_5.1*
>>> *bind-utils.x86_64 32:9.8.2-0.23.rc1.el6_5.1*
>>> *rpcbind.x86_64 0.2.0-11.el6
>>> @anaconda-CentOS-201311291202.x86_64/6.5*
>>> *samba4-winbind.x86_64*
>>>
>>> *krb5-server.x86_64 1.10.3-15.el6_5.1 *
>>>
>>>
>>> *Linux 2.6.32-431.29.2.el6.x86_64 #1 SMP Tue Sep 9 21:36:05 UTC 2014
>>> x86_64 x86_64 x86_64 GNU/Linux *
>>>
>>> It's not a permanent situation as it sometimes runs 100% for a while,
>>> but 80% of the time it is unusable. If anybody can assist me, please be so
>>> kind.
>>>
>>> Regards,
>>>
>>> Walter
>>>
>>> Hello please which version of bind-dyndb-ldap do you use?
>>> I had similar issue with bind-dyndb-ldap, but it was development
>>> version, I'm not sure if this is your case.
>>> When named was failing, dirserv was really slow.
>>>
>>> Can you send journalctl -b -u named log when dig doesn't work??
>>>
>>> --
>>> Martin Basti
>>>
>>>
>>
>>
>> --
>> Martin Basti
>>
>>
>
>
> --
> Martin Basti
>
>
>
>
>
>
>
>
>
> --
> Martin Basti
>
>
>
>
>
> --
> Manage your subscription for the Freeipa-users mailing list:
> https://www.redhat.com/mailman/listinfo/freeipa-users
> Go To http://freeipa.org for more info on the project
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/freeipa-users/attachments/20141112/305f7dac/attachment.htm>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: stacktracefiles.zip
Type: application/zip
Size: 7270 bytes
Desc: not available
URL: <http://listman.redhat.com/archives/freeipa-users/attachments/20141112/305f7dac/attachment.zip>
More information about the Freeipa-users
mailing list