[Freeipa-users] FreeIPA unresponsive - Causes DOS situations
Rich Megginson
rmeggins at redhat.com
Thu Nov 13 14:26:27 UTC 2014
On 11/13/2014 03:02 AM, Walter van Lille wrote:
> Thanks Rich, I have installed the packages and run gdb again.
> Hopefully the attached file is more useful.
The symbols are there. However, the server is almost completely idle -
no hangs, no deadlocks, no waiting on I/O.
You must catch dirsrv while it is hung. You might want to run that gdb
script every few seconds during the time it appears to be hung.
>
> On Wed, Nov 12, 2014 at 4:16 PM, Rich Megginson <rmeggins at redhat.com
> <mailto:rmeggins at redhat.com>> wrote:
>
> On 11/12/2014 05:42 AM, Walter van Lille wrote:
>> Thanks again for the assistance guys. I have saved two files and
>> included it here. Hope it tells you more than it does me.
>
> These stack traces contain no useful symbols.
>
> Please read
> http://www.port389.org/docs/389ds/FAQ/faq.html#debugging-crashes
> to find out how to install the correct debuginfo packages on your
> system. For IPA you will also need debuginfo packages for ipa and
> slapi-nis
>
> If debuginfo-install is not working, see
> https://www.centos.org/forums/viewtopic.php?t=1710
>
> If you simply cannot figure out how to install the debuginfo
> packages, please email me with the following information:
>
> $ rpm -q ipa-server slapi-nis 389-ds-base openldap db4 nss nspr glibc
>
> and I will make them available to you
>
> NOTE: With debuginfo packages, the version of the debuginfo
> package _must exactly match_ the version of the associated package
> e.g. for 389-ds-base.x86_64 1.2.11.15-34.el6_5 you must have
> 389-ds-base-debuginfo.x86_64 1.2.11.15-34.el6_5
>
> If the versions do not match exactly, you will not get the symbols.
>
>>
>> Regards,
>>
>> Wallter
>>
>> On Tue, Nov 11, 2014 at 8:03 PM, Rich Megginson
>> <rmeggins at redhat.com <mailto:rmeggins at redhat.com>> wrote:
>>
>> On 11/11/2014 10:37 AM, Martin Basti wrote:
>>> On 11/11/14 15:58, Rich Megginson wrote:
>>>> On 11/11/2014 06:20 AM, Ludwig Krispenz wrote:
>>>>>
>>>>> On 11/11/2014 02:14 PM, Martin Basti wrote:
>>>>>> Ludiwg (CCed) this seems like old (fixed?) DS bug.
>>>>> hmm, it says limit is 2097152, so it already has the new
>>>>> setting, but the error message says the packet is 800MB*
>>>>> *
>>>>
>>>> *Right. That usually means the server was expecting an
>>>> encrypted SASL buffer from the client, but instead the
>>>> client thinks SASL encryption negotiation failed and just
>>>> sent a plain LDAP buffer. What version of 389-ds-base are
>>>> you using? rpm -q 389-ds-base
>>>>
>>>> https://fedorahosted.org/389/ticket/47416
>>>>
>>>> So, DO NOT increase your sasl io buffer size - it will not
>>>> fix the problem, and it will leave you open to DoS attacks.
>>>> *
>>>
>>> He is using
>>>
>>> CentOS release 6.5 (Final)
>>> 389-ds-base.x86_64 1.2.11.15-34.el6_5
>>
>>
>> Ignore the "SASL encrypted packet length exceeds maximum
>> allowed limit" error. All it means is the client has some
>> error and is canceling the connection.
>>
>> The bugzilla associated with *47416
>> <https://fedorahosted.org/389/ticket/47416> is targeted for
>> RHEL 7.1. The problem is that we were never able to figure
>> out what error the client was getting that caused the client
>> to stop establishing the *SASL encryption, and the original
>> customer who reported the problem dropped the case -
>> everything just started working, no further errors. Note
>> that 47416 doesn't really fix the problem or address the root
>> cause - the root cause is some error on the client side that
>> causes it to stop doing encrypted SASL I/O and send an LDAP
>> unbind request.
>>
>> Let's get back to the actual problem - what is the actual
>> problem? The ldap server is hanging or unresponsive? If so,
>> then see
>> http://www.port389.org/docs/389ds/FAQ/faq.html#debugging-hangs.
>> Let's get some dirsrv stack traces during the period of time
>> when it appears to be unresponsive.
>>
>>
>>>
>>>> *
>>>> *
>>>>> **
>>>>>>
>>>>>> On 11/11/14 13:13, Walter van Lille wrote:
>>>>>>> I've just cleaned out a ton of slapd_poll timed out
>>>>>>> messages from the output and changed the names to
>>>>>>> protect the innocent, :-)
>>>>>>> Here is the output as requested:
>>>>>>>
>>>>>>>
>>>>>>> *[05/Nov/2014:11:44:05 +0200] - SASL encrypted packet
>>>>>>> length exceeds maximum allowed limit (length=805565,
>>>>>>> limit=2097152). Change the nsslapd-maxsasliosize
>>>>>>> attribute in cn=config to increase limit.*
>>>>>>> *
>>>>>>> *
>>>>>>> *[10/Nov/2014:14:45:19 +0200] - slapd_poll(115) timed out*
>>>>>>> *[10/Nov/2014:14:45:19 +0200] sasl_io_enable - Cannot
>>>>>>> enable SASL security on connection in CLOSING state*
>>>>>>> *[10/Nov/2014:14:45:19 +0200] - Error: could not
>>>>>>> add/remove IO layers from connection*
>>>>>>> *[11/Nov/2014:11:48:09 +0200] - slapd shutting down -
>>>>>>> signaling operation threads*
>>>>>>> *[11/Nov/2014:11:48:09 +0200] - slapd shutting down -
>>>>>>> waiting for 30 threads to terminate*
>>>>>>> *[11/Nov/2014:13:14:12 +0200] - slapd shutting down -
>>>>>>> closing down internal subsystems and plugins*
>>>>>>> *[11/Nov/2014:13:14:12 +0200] - Waiting for 4 database
>>>>>>> threads to stop*
>>>>>>> *[11/Nov/2014:13:14:13 +0200] - All database threads now
>>>>>>> stopped*
>>>>>>> *[11/Nov/2014:13:14:13 +0200] - slapd stopped.*
>>>>>>> *[11/Nov/2014:13:26:35 +0200] - 389-Directory/1.2.11.15
>>>>>>> <http://1.2.11.15> B2014.219.179 starting up*
>>>>>>> *[11/Nov/2014:13:26:35 +0200] schema-compat-plugin -
>>>>>>> warning: no entries set up under cn=computers,
>>>>>>> cn=compat,dc=sample,dc=example*
>>>>>>> *[11/Nov/2014:13:26:36 +0200] - Skipping CoS Definition
>>>>>>> cn=Password Policy,cn=accounts,dc=sample,dc=example--no
>>>>>>> CoS Templates found, which should be added before the
>>>>>>> CoS Definition.*
>>>>>>> *[11/Nov/2014:13:26:36 +0200] - Skipping CoS Definition
>>>>>>> cn=Password Policy,cn=accounts,dc=sample,dc=example--no
>>>>>>> CoS Templates found, which should be added before the
>>>>>>> CoS Definition.*
>>>>>>> *[11/Nov/2014:13:26:36 +0200] - slapd started.
>>>>>>> Listening on All Interfaces port 389 for LDAP requests*
>>>>>>> *[11/Nov/2014:13:26:36 +0200] - Listening on All
>>>>>>> Interfaces port 636 for LDAPS requests*
>>>>>>> *[11/Nov/2014:13:26:36 +0200] - Listening on
>>>>>>> /var/run/slapd-SAMPLE-EXAMPLE.socket for LDAPI requests*
>>>>>>> *[11/Nov/2014:13:57:08 +0200] - slapd_poll(78) timed out*
>>>>>>> *
>>>>>>> *
>>>>>>> *
>>>>>>> *
>>>>>>> *
>>>>>>> *
>>>>>>>
>>>>>>>
>>>>>>> On Tue, Nov 11, 2014 at 1:19 PM, Martin Basti
>>>>>>> <mbasti at redhat.com <mailto:mbasti at redhat.com>> wrote:
>>>>>>>
>>>>>>> IMHO It's DS bug, can you share DS error log?
>>>>>>> pspacek CCed to examine named logs.
>>>>>>>
>>>>>>> Martin^2
>>>>>>>
>>>>>>>
>>>>>>> On 11/11/14 12:13, Walter van Lille wrote:
>>>>>>>> Hi Martin, thanks for the reply.
>>>>>>>> My version: bind-dyndb-ldap-2.3-5.el6.x86_64
>>>>>>>> The server doesn't have journalctl installed but I
>>>>>>>> have the outputs from the messages and named.run
>>>>>>>> files that I included here:
>>>>>>>>
>>>>>>>> Messages:
>>>>>>>>
>>>>>>>> *Nov 11 12:30:13 freeipa named[1481]: error
>>>>>>>> (network unreachable) resolving
>>>>>>>> 'example.example.com.10.123.123.123/A/IN':
>>>>>>>> 2001:500:2f::f#53*
>>>>>>>> *Nov 11 12:30:23 freeipa named[1481]: LDAP query
>>>>>>>> timed out. Try to adjust "timeout" parameter*
>>>>>>>> *Nov 11 12:30:23 freeipa named[1481]: LDAP query
>>>>>>>> timed out. Try to adjust "timeout" parameter*
>>>>>>>> *Nov 11 12:30:33 freeipa named[1481]: LDAP query
>>>>>>>> timed out. Try to adjust "timeout" parameter*
>>>>>>>> *Nov 11 12:30:33 freeipa named[1481]: LDAP query
>>>>>>>> timed out. Try to adjust "timeout" parameter*
>>>>>>>>
>>>>>>>> Named.run:
>>>>>>>>
>>>>>>>> *client 10.123.123.123#42639: transfer of
>>>>>>>> 'example.example/IN': AXFR-style IXFR started*
>>>>>>>> *client 10.123.123.123#42639: transfer of
>>>>>>>> ''example.example/IN': AXFR-style IXFR ended*
>>>>>>>> *client 10.123.123.123#46912: transfer of
>>>>>>>> '10.123.123.123.in-addr.arpa/IN': AXFR-style IXFR
>>>>>>>> started*
>>>>>>>> *client 10.123.123.123#46912: transfer of
>>>>>>>> '10.123.123.123.in-addr.arpa/IN': AXFR-style IXFR
>>>>>>>> ended*
>>>>>>>> *LDAP query timed out. Try to adjust "timeout"
>>>>>>>> parameter*
>>>>>>>> *LDAP query timed out. Try to adjust "timeout"
>>>>>>>> parameter*
>>>>>>>> *LDAP query timed out. Try to adjust "timeout"
>>>>>>>> parameter*
>>>>>>>>
>>>>>>>> I just replaced the IPs and the actual names with
>>>>>>>> something more generic.
>>>>>>>>
>>>>>>>> Regards,
>>>>>>>>
>>>>>>>> Walter
>>>>>>>>
>>>>>>>> On Thu, Nov 6, 2014 at 5:00 PM, Martin Basti
>>>>>>>> <mbasti at redhat.com <mailto:mbasti at redhat.com>> wrote:
>>>>>>>>
>>>>>>>> On 06/11/14 14:58, Walter van Lille wrote:
>>>>>>>>> Hi,
>>>>>>>>>
>>>>>>>>> I need some assistance please.
>>>>>>>>> I've taken over an IPA server to manage a few
>>>>>>>>> months ago, and it was working fine until
>>>>>>>>> recently when it started acting up seemingly
>>>>>>>>> off its own accord.
>>>>>>>>> When I do an ipactl status it basically gives
>>>>>>>>> an output as shown below:
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> *Directory Service: RUNNING
>>>>>>>>> *
>>>>>>>>> *
>>>>>>>>> *
>>>>>>>>> *Loooooooooooooooooooooooooooooooooooooooooooooooooong
>>>>>>>>> pause... (To the tune of 7 minutes sometimes)*
>>>>>>>>> *
>>>>>>>>> *
>>>>>>>>> *KDC Service: RUNNING*
>>>>>>>>> *KPASSWD Service: RUNNING*
>>>>>>>>> *DNS Service: RUNNING*
>>>>>>>>> *MEMCACHE Service: RUNNING*
>>>>>>>>> *HTTP Service: RUNNING*
>>>>>>>>> *CA Service: RUNNING*
>>>>>>>>> *ADTRUST Service: RUNNING*
>>>>>>>>> *EXTID Service: RUNNING*
>>>>>>>>>
>>>>>>>>> Running top showed that ns-slapd was munching
>>>>>>>>> almost all my resources, but I got that fixed
>>>>>>>>> by upping the cache. Unfortunately this did
>>>>>>>>> not correct the issue and it still reacts in
>>>>>>>>> the same fashion, although the resources have
>>>>>>>>> been freed up now.
>>>>>>>>> I've noticed that when I run dig on either the
>>>>>>>>> local server or a remote machine that the
>>>>>>>>> query basically just times out as shown here:
>>>>>>>>>
>>>>>>>>> *dig freeipa.myexample.sample*
>>>>>>>>> *
>>>>>>>>> *
>>>>>>>>> *; <<>> DiG
>>>>>>>>> 9.8.2rc1-RedHat-9.8.2-0.23.rc1.el6_5.1 <<>>
>>>>>>>>> freeipa.myexample.sample*
>>>>>>>>> *;; global options: +cmd*
>>>>>>>>> *;; connection timed out; no servers could be
>>>>>>>>> reached*
>>>>>>>>>
>>>>>>>>> When the KDC service fails to start, then name
>>>>>>>>> lookups seem OK, but authentication fails.
>>>>>>>>> otherwise it's dead in the water.
>>>>>>>>>
>>>>>>>>> This also happens:
>>>>>>>>>
>>>>>>>>> *sudo ipactl status*
>>>>>>>>> *Directory Service: RUNNING*
>>>>>>>>> *Unknown error when retrieving list of
>>>>>>>>> services from LDAP:*
>>>>>>>>> *
>>>>>>>>> *
>>>>>>>>> My software setup is as follows:
>>>>>>>>>
>>>>>>>>> *CentOS release 6.5 (Final)
>>>>>>>>> *
>>>>>>>>> *389-ds-base.x86_64 1.2.11.15-34.el6_5
>>>>>>>>> *
>>>>>>>>> *bind.x86_64 32:9.8.2-0.23.rc1.el6_5.1
>>>>>>>>> *
>>>>>>>>> *bind-dyndb-ldap.x86_64*
>>>>>>>>> *bind-libs.x86_64 32:9.8.2-0.23.rc1.el6_5.1*
>>>>>>>>> *bind-utils.x86_64 32:9.8.2-0.23.rc1.el6_5.1*
>>>>>>>>> *rpcbind.x86_64 0.2.0-11.el6
>>>>>>>>> @anaconda-CentOS-201311291202.x86_64/6.5*
>>>>>>>>> *samba4-winbind.x86_64*
>>>>>>>>> *krb5-server.x86_64 1.10.3-15.el6_5.1
>>>>>>>>> *
>>>>>>>>> *
>>>>>>>>> *
>>>>>>>>> *Linux 2.6.32-431.29.2.el6.x86_64 #1 SMP Tue
>>>>>>>>> Sep 9 21:36:05 UTC 2014 x86_64 x86_64 x86_64
>>>>>>>>> GNU/Linux
>>>>>>>>> *
>>>>>>>>>
>>>>>>>>> It's not a permanent situation as it sometimes
>>>>>>>>> runs 100% for a while, but 80% of the time it
>>>>>>>>> is unusable. If anybody can assist me, please
>>>>>>>>> be so kind.
>>>>>>>>>
>>>>>>>>> Regards,
>>>>>>>>>
>>>>>>>>> Walter
>>>>>>>>>
>>>>>>>> Hello please which version of bind-dyndb-ldap
>>>>>>>> do you use?
>>>>>>>> I had similar issue with bind-dyndb-ldap, but
>>>>>>>> it was development version, I'm not sure if
>>>>>>>> this is your case.
>>>>>>>> When named was failing, dirserv was really slow.
>>>>>>>>
>>>>>>>> Can you send journalctl -b -u named log when
>>>>>>>> dig doesn't work??
>>>>>>>>
>>>>>>>> --
>>>>>>>> Martin Basti
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> Martin Basti
>>>>>>>
>>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> Martin Basti
>>>>>
>>>>>
>>>>>
>>>>
>>>>
>>>>
>>>
>>>
>>> --
>>> Martin Basti
>>>
>>>
>>
>>
>> --
>> Manage your subscription for the Freeipa-users mailing list:
>> https://www.redhat.com/mailman/listinfo/freeipa-users
>> Go To http://freeipa.org for more info on the project
>>
>>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/freeipa-users/attachments/20141113/52760eeb/attachment.htm>
More information about the Freeipa-users
mailing list