[Freeipa-users] FreeIPA unresponsive - Causes DOS situations
Ludwig Krispenz
lkrispen at redhat.com
Thu Nov 13 10:27:02 UTC 2014
Hmm, the symbols are there now, but all threads are idle, DS is just
waiting on work to do. Which client do you expect to connect to DS,
maybe you need to debug this client.
On 11/13/2014 11:02 AM, Walter van Lille wrote:
> Thanks Rich, I have installed the packages and run gdb again.
> Hopefully the attached file is more useful.
>
> On Wed, Nov 12, 2014 at 4:16 PM, Rich Megginson <rmeggins at redhat.com
> <mailto:rmeggins at redhat.com>> wrote:
>
> On 11/12/2014 05:42 AM, Walter van Lille wrote:
>> Thanks again for the assistance guys. I have saved two files and
>> included it here. Hope it tells you more than it does me.
>
> These stack traces contain no useful symbols.
>
> Please read
> http://www.port389.org/docs/389ds/FAQ/faq.html#debugging-crashes
> to find out how to install the correct debuginfo packages on your
> system. For IPA you will also need debuginfo packages for ipa and
> slapi-nis
>
> If debuginfo-install is not working, see
> https://www.centos.org/forums/viewtopic.php?t=1710
>
> If you simply cannot figure out how to install the debuginfo
> packages, please email me with the following information:
>
> $ rpm -q ipa-server slapi-nis 389-ds-base openldap db4 nss nspr glibc
>
> and I will make them available to you
>
> NOTE: With debuginfo packages, the version of the debuginfo
> package _must exactly match_ the version of the associated package
> e.g. for 389-ds-base.x86_64 1.2.11.15-34.el6_5 you must have
> 389-ds-base-debuginfo.x86_64 1.2.11.15-34.el6_5
>
> If the versions do not match exactly, you will not get the symbols.
>
>>
>> Regards,
>>
>> Wallter
>>
>> On Tue, Nov 11, 2014 at 8:03 PM, Rich Megginson
>> <rmeggins at redhat.com <mailto:rmeggins at redhat.com>> wrote:
>>
>> On 11/11/2014 10:37 AM, Martin Basti wrote:
>>> On 11/11/14 15:58, Rich Megginson wrote:
>>>> On 11/11/2014 06:20 AM, Ludwig Krispenz wrote:
>>>>>
>>>>> On 11/11/2014 02:14 PM, Martin Basti wrote:
>>>>>> Ludiwg (CCed) this seems like old (fixed?) DS bug.
>>>>> hmm, it says limit is 2097152, so it already has the new
>>>>> setting, but the error message says the packet is 800MB*
>>>>> *
>>>>
>>>> *Right. That usually means the server was expecting an
>>>> encrypted SASL buffer from the client, but instead the
>>>> client thinks SASL encryption negotiation failed and just
>>>> sent a plain LDAP buffer. What version of 389-ds-base are
>>>> you using? rpm -q 389-ds-base
>>>>
>>>> https://fedorahosted.org/389/ticket/47416
>>>>
>>>> So, DO NOT increase your sasl io buffer size - it will not
>>>> fix the problem, and it will leave you open to DoS attacks.
>>>> *
>>>
>>> He is using
>>>
>>> CentOS release 6.5 (Final)
>>> 389-ds-base.x86_64 1.2.11.15-34.el6_5
>>
>>
>> Ignore the "SASL encrypted packet length exceeds maximum
>> allowed limit" error. All it means is the client has some
>> error and is canceling the connection.
>>
>> The bugzilla associated with *47416
>> <https://fedorahosted.org/389/ticket/47416> is targeted for
>> RHEL 7.1. The problem is that we were never able to figure
>> out what error the client was getting that caused the client
>> to stop establishing the *SASL encryption, and the original
>> customer who reported the problem dropped the case -
>> everything just started working, no further errors. Note
>> that 47416 doesn't really fix the problem or address the root
>> cause - the root cause is some error on the client side that
>> causes it to stop doing encrypted SASL I/O and send an LDAP
>> unbind request.
>>
>> Let's get back to the actual problem - what is the actual
>> problem? The ldap server is hanging or unresponsive? If so,
>> then see
>> http://www.port389.org/docs/389ds/FAQ/faq.html#debugging-hangs.
>> Let's get some dirsrv stack traces during the period of time
>> when it appears to be unresponsive.
>>
>>
>>>
>>>> *
>>>> *
>>>>> **
>>>>>>
>>>>>> On 11/11/14 13:13, Walter van Lille wrote:
>>>>>>> I've just cleaned out a ton of slapd_poll timed out
>>>>>>> messages from the output and changed the names to
>>>>>>> protect the innocent, :-)
>>>>>>> Here is the output as requested:
>>>>>>>
>>>>>>>
>>>>>>> *[05/Nov/2014:11:44:05 +0200] - SASL encrypted packet
>>>>>>> length exceeds maximum allowed limit (length=805565,
>>>>>>> limit=2097152). Change the nsslapd-maxsasliosize
>>>>>>> attribute in cn=config to increase limit.*
>>>>>>> *
>>>>>>> *
>>>>>>> *[10/Nov/2014:14:45:19 +0200] - slapd_poll(115) timed out*
>>>>>>> *[10/Nov/2014:14:45:19 +0200] sasl_io_enable - Cannot
>>>>>>> enable SASL security on connection in CLOSING state*
>>>>>>> *[10/Nov/2014:14:45:19 +0200] - Error: could not
>>>>>>> add/remove IO layers from connection*
>>>>>>> *[11/Nov/2014:11:48:09 +0200] - slapd shutting down -
>>>>>>> signaling operation threads*
>>>>>>> *[11/Nov/2014:11:48:09 +0200] - slapd shutting down -
>>>>>>> waiting for 30 threads to terminate*
>>>>>>> *[11/Nov/2014:13:14:12 +0200] - slapd shutting down -
>>>>>>> closing down internal subsystems and plugins*
>>>>>>> *[11/Nov/2014:13:14:12 +0200] - Waiting for 4 database
>>>>>>> threads to stop*
>>>>>>> *[11/Nov/2014:13:14:13 +0200] - All database threads now
>>>>>>> stopped*
>>>>>>> *[11/Nov/2014:13:14:13 +0200] - slapd stopped.*
>>>>>>> *[11/Nov/2014:13:26:35 +0200] - 389-Directory/1.2.11.15
>>>>>>> <http://1.2.11.15> B2014.219.179 starting up*
>>>>>>> *[11/Nov/2014:13:26:35 +0200] schema-compat-plugin -
>>>>>>> warning: no entries set up under cn=computers,
>>>>>>> cn=compat,dc=sample,dc=example*
>>>>>>> *[11/Nov/2014:13:26:36 +0200] - Skipping CoS Definition
>>>>>>> cn=Password Policy,cn=accounts,dc=sample,dc=example--no
>>>>>>> CoS Templates found, which should be added before the
>>>>>>> CoS Definition.*
>>>>>>> *[11/Nov/2014:13:26:36 +0200] - Skipping CoS Definition
>>>>>>> cn=Password Policy,cn=accounts,dc=sample,dc=example--no
>>>>>>> CoS Templates found, which should be added before the
>>>>>>> CoS Definition.*
>>>>>>> *[11/Nov/2014:13:26:36 +0200] - slapd started.
>>>>>>> Listening on All Interfaces port 389 for LDAP requests*
>>>>>>> *[11/Nov/2014:13:26:36 +0200] - Listening on All
>>>>>>> Interfaces port 636 for LDAPS requests*
>>>>>>> *[11/Nov/2014:13:26:36 +0200] - Listening on
>>>>>>> /var/run/slapd-SAMPLE-EXAMPLE.socket for LDAPI requests*
>>>>>>> *[11/Nov/2014:13:57:08 +0200] - slapd_poll(78) timed out*
>>>>>>> *
>>>>>>> *
>>>>>>> *
>>>>>>> *
>>>>>>> *
>>>>>>> *
>>>>>>>
>>>>>>>
>>>>>>> On Tue, Nov 11, 2014 at 1:19 PM, Martin Basti
>>>>>>> <mbasti at redhat.com <mailto:mbasti at redhat.com>> wrote:
>>>>>>>
>>>>>>> IMHO It's DS bug, can you share DS error log?
>>>>>>> pspacek CCed to examine named logs.
>>>>>>>
>>>>>>> Martin^2
>>>>>>>
>>>>>>>
>>>>>>> On 11/11/14 12:13, Walter van Lille wrote:
>>>>>>>> Hi Martin, thanks for the reply.
>>>>>>>> My version: bind-dyndb-ldap-2.3-5.el6.x86_64
>>>>>>>> The server doesn't have journalctl installed but I
>>>>>>>> have the outputs from the messages and named.run
>>>>>>>> files that I included here:
>>>>>>>>
>>>>>>>> Messages:
>>>>>>>>
>>>>>>>> *Nov 11 12:30:13 freeipa named[1481]: error
>>>>>>>> (network unreachable) resolving
>>>>>>>> 'example.example.com.10.123.123.123/A/IN':
>>>>>>>> 2001:500:2f::f#53*
>>>>>>>> *Nov 11 12:30:23 freeipa named[1481]: LDAP query
>>>>>>>> timed out. Try to adjust "timeout" parameter*
>>>>>>>> *Nov 11 12:30:23 freeipa named[1481]: LDAP query
>>>>>>>> timed out. Try to adjust "timeout" parameter*
>>>>>>>> *Nov 11 12:30:33 freeipa named[1481]: LDAP query
>>>>>>>> timed out. Try to adjust "timeout" parameter*
>>>>>>>> *Nov 11 12:30:33 freeipa named[1481]: LDAP query
>>>>>>>> timed out. Try to adjust "timeout" parameter*
>>>>>>>>
>>>>>>>> Named.run:
>>>>>>>>
>>>>>>>> *client 10.123.123.123#42639: transfer of
>>>>>>>> 'example.example/IN': AXFR-style IXFR started*
>>>>>>>> *client 10.123.123.123#42639: transfer of
>>>>>>>> ''example.example/IN': AXFR-style IXFR ended*
>>>>>>>> *client 10.123.123.123#46912: transfer of
>>>>>>>> '10.123.123.123.in-addr.arpa/IN': AXFR-style IXFR
>>>>>>>> started*
>>>>>>>> *client 10.123.123.123#46912: transfer of
>>>>>>>> '10.123.123.123.in-addr.arpa/IN': AXFR-style IXFR
>>>>>>>> ended*
>>>>>>>> *LDAP query timed out. Try to adjust "timeout"
>>>>>>>> parameter*
>>>>>>>> *LDAP query timed out. Try to adjust "timeout"
>>>>>>>> parameter*
>>>>>>>> *LDAP query timed out. Try to adjust "timeout"
>>>>>>>> parameter*
>>>>>>>>
>>>>>>>> I just replaced the IPs and the actual names with
>>>>>>>> something more generic.
>>>>>>>>
>>>>>>>> Regards,
>>>>>>>>
>>>>>>>> Walter
>>>>>>>>
>>>>>>>> On Thu, Nov 6, 2014 at 5:00 PM, Martin Basti
>>>>>>>> <mbasti at redhat.com <mailto:mbasti at redhat.com>> wrote:
>>>>>>>>
>>>>>>>> On 06/11/14 14:58, Walter van Lille wrote:
>>>>>>>>> Hi,
>>>>>>>>>
>>>>>>>>> I need some assistance please.
>>>>>>>>> I've taken over an IPA server to manage a few
>>>>>>>>> months ago, and it was working fine until
>>>>>>>>> recently when it started acting up seemingly
>>>>>>>>> off its own accord.
>>>>>>>>> When I do an ipactl status it basically gives
>>>>>>>>> an output as shown below:
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> *Directory Service: RUNNING
>>>>>>>>> *
>>>>>>>>> *
>>>>>>>>> *
>>>>>>>>> *Loooooooooooooooooooooooooooooooooooooooooooooooooong
>>>>>>>>> pause... (To the tune of 7 minutes sometimes)*
>>>>>>>>> *
>>>>>>>>> *
>>>>>>>>> *KDC Service: RUNNING*
>>>>>>>>> *KPASSWD Service: RUNNING*
>>>>>>>>> *DNS Service: RUNNING*
>>>>>>>>> *MEMCACHE Service: RUNNING*
>>>>>>>>> *HTTP Service: RUNNING*
>>>>>>>>> *CA Service: RUNNING*
>>>>>>>>> *ADTRUST Service: RUNNING*
>>>>>>>>> *EXTID Service: RUNNING*
>>>>>>>>>
>>>>>>>>> Running top showed that ns-slapd was munching
>>>>>>>>> almost all my resources, but I got that fixed
>>>>>>>>> by upping the cache. Unfortunately this did
>>>>>>>>> not correct the issue and it still reacts in
>>>>>>>>> the same fashion, although the resources have
>>>>>>>>> been freed up now.
>>>>>>>>> I've noticed that when I run dig on either the
>>>>>>>>> local server or a remote machine that the
>>>>>>>>> query basically just times out as shown here:
>>>>>>>>>
>>>>>>>>> *dig freeipa.myexample.sample*
>>>>>>>>> *
>>>>>>>>> *
>>>>>>>>> *; <<>> DiG
>>>>>>>>> 9.8.2rc1-RedHat-9.8.2-0.23.rc1.el6_5.1 <<>>
>>>>>>>>> freeipa.myexample.sample*
>>>>>>>>> *;; global options: +cmd*
>>>>>>>>> *;; connection timed out; no servers could be
>>>>>>>>> reached*
>>>>>>>>>
>>>>>>>>> When the KDC service fails to start, then name
>>>>>>>>> lookups seem OK, but authentication fails.
>>>>>>>>> otherwise it's dead in the water.
>>>>>>>>>
>>>>>>>>> This also happens:
>>>>>>>>>
>>>>>>>>> *sudo ipactl status*
>>>>>>>>> *Directory Service: RUNNING*
>>>>>>>>> *Unknown error when retrieving list of
>>>>>>>>> services from LDAP:*
>>>>>>>>> *
>>>>>>>>> *
>>>>>>>>> My software setup is as follows:
>>>>>>>>>
>>>>>>>>> *CentOS release 6.5 (Final)
>>>>>>>>> *
>>>>>>>>> *389-ds-base.x86_64 1.2.11.15-34.el6_5
>>>>>>>>> *
>>>>>>>>> *bind.x86_64 32:9.8.2-0.23.rc1.el6_5.1
>>>>>>>>> *
>>>>>>>>> *bind-dyndb-ldap.x86_64*
>>>>>>>>> *bind-libs.x86_64 32:9.8.2-0.23.rc1.el6_5.1*
>>>>>>>>> *bind-utils.x86_64 32:9.8.2-0.23.rc1.el6_5.1*
>>>>>>>>> *rpcbind.x86_64 0.2.0-11.el6
>>>>>>>>> @anaconda-CentOS-201311291202.x86_64/6.5*
>>>>>>>>> *samba4-winbind.x86_64*
>>>>>>>>> *krb5-server.x86_64 1.10.3-15.el6_5.1
>>>>>>>>> *
>>>>>>>>> *
>>>>>>>>> *
>>>>>>>>> *Linux 2.6.32-431.29.2.el6.x86_64 #1 SMP Tue
>>>>>>>>> Sep 9 21:36:05 UTC 2014 x86_64 x86_64 x86_64
>>>>>>>>> GNU/Linux
>>>>>>>>> *
>>>>>>>>>
>>>>>>>>> It's not a permanent situation as it sometimes
>>>>>>>>> runs 100% for a while, but 80% of the time it
>>>>>>>>> is unusable. If anybody can assist me, please
>>>>>>>>> be so kind.
>>>>>>>>>
>>>>>>>>> Regards,
>>>>>>>>>
>>>>>>>>> Walter
>>>>>>>>>
>>>>>>>> Hello please which version of bind-dyndb-ldap
>>>>>>>> do you use?
>>>>>>>> I had similar issue with bind-dyndb-ldap, but
>>>>>>>> it was development version, I'm not sure if
>>>>>>>> this is your case.
>>>>>>>> When named was failing, dirserv was really slow.
>>>>>>>>
>>>>>>>> Can you send journalctl -b -u named log when
>>>>>>>> dig doesn't work??
>>>>>>>>
>>>>>>>> --
>>>>>>>> Martin Basti
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> Martin Basti
>>>>>>>
>>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> Martin Basti
>>>>>
>>>>>
>>>>>
>>>>
>>>>
>>>>
>>>
>>>
>>> --
>>> Martin Basti
>>>
>>>
>>
>>
>> --
>> Manage your subscription for the Freeipa-users mailing list:
>> https://www.redhat.com/mailman/listinfo/freeipa-users
>> Go To http://freeipa.org for more info on the project
>>
>>
>
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/freeipa-users/attachments/20141113/3a1689cf/attachment.htm>
More information about the Freeipa-users
mailing list