[Freeipa-users] FreeIPA unresponsive - Causes DOS situations

Thu Nov 13 10:02:03 UTC 2014

Thanks Rich, I have installed the packages and run gdb again. Hopefully the
attached file is more useful.

On Wed, Nov 12, 2014 at 4:16 PM, Rich Megginson <rmeggins at redhat.com> wrote:

>  On 11/12/2014 05:42 AM, Walter van Lille wrote:
>
> Thanks again for the assistance guys. I have saved two files and included
> it here. Hope it tells you more than it does me.
>
>
> These stack traces contain no useful symbols.
>
> Please read
> http://www.port389.org/docs/389ds/FAQ/faq.html#debugging-crashes to find
> out how to install the correct debuginfo packages on your system.  For IPA
> you will also need debuginfo packages for ipa and slapi-nis
>
> If debuginfo-install is not working, see
> https://www.centos.org/forums/viewtopic.php?t=1710
>
> If you simply cannot figure out how to install the debuginfo packages,
> please email me with the following information:
>
> $ rpm -q ipa-server slapi-nis 389-ds-base openldap db4 nss nspr glibc
>
> and I will make them available to you
>
> NOTE: With debuginfo packages, the version of the debuginfo package _must
> exactly match_ the version of the associated package e.g. for 389-ds-base.x86_64
>   1.2.11.15-34.el6_5 you must have
> 389-ds-base-debuginfo.x86_64   1.2.11.15-34.el6_5
>
> If the versions do not match exactly, you will not get the symbols.
>
>
>  Regards,
>
>  Wallter
>
> On Tue, Nov 11, 2014 at 8:03 PM, Rich Megginson <rmeggins at redhat.com>
> wrote:
>
>>  On 11/11/2014 10:37 AM, Martin Basti wrote:
>>
>> On 11/11/14 15:58, Rich Megginson wrote:
>>
>> On 11/11/2014 06:20 AM, Ludwig Krispenz wrote:
>>
>>
>> On 11/11/2014 02:14 PM, Martin Basti wrote:
>>
>> Ludiwg (CCed) this seems like old (fixed?) DS bug.
>>
>> hmm, it says limit is 2097152, so it already has the new setting, but the
>> error message says the packet is 800MB
>>
>>
>>
>>
>>
>>
>>
>> *Right.  That usually means the server was expecting an encrypted SASL
>> buffer from the client, but instead the client thinks SASL encryption
>> negotiation failed and just sent a plain LDAP buffer.  What version of
>> 389-ds-base are you using?  rpm -q 389-ds-base
>> https://fedorahosted.org/389/ticket/47416
>> <https://fedorahosted.org/389/ticket/47416> So, DO NOT increase your sasl
>> io buffer size - it will not fix the problem, and it will leave you open to
>> DoS attacks. *
>>
>>
>> He is using
>>
>>  CentOS release 6.5 (Final)
>>  389-ds-base.x86_64   1.2.11.15-34.el6_5
>>
>>
>>
>>  Ignore the "SASL encrypted packet length exceeds maximum allowed limit"
>> error.  All it means is the client has some error and is canceling the
>> connection.
>>
>> The bugzilla associated with *47416
>> <https://fedorahosted.org/389/ticket/47416> is targeted for RHEL 7.1.  The
>> problem is that we were never able to figure out what error the client was
>> getting that caused the client to stop establishing the *SASL
>> encryption, and the original customer who reported the problem dropped the
>> case - everything just started working, no further errors.  Note that 47416
>> doesn't really fix the problem or address the root cause - the root cause
>> is some error on the client side that causes it to stop doing encrypted
>> SASL I/O and send an LDAP unbind request.
>>
>> Let's get back to the actual problem - what is the actual problem?  The
>> ldap server is hanging or unresponsive?  If so, then see
>> http://www.port389.org/docs/389ds/FAQ/faq.html#debugging-hangs.  Let's
>> get some dirsrv stack traces during the period of time when it appears to
>> be unresponsive.
>>
>>
>>
>>
>>
>> On 11/11/14 13:13, Walter van Lille wrote:
>>
>>  I've just cleaned out a ton of slapd_poll timed out messages from the
>> output and changed the names to protect the innocent, :-)
>> Here is the output as requested:
>>
>>
>>  *[05/Nov/2014:11:44:05 +0200] - SASL encrypted packet length exceeds
>> maximum allowed limit (length=805565, limit=2097152).  Change the
>> nsslapd-maxsasliosize attribute in cn=config to increase limit.*
>>
>>  *[10/Nov/2014:14:45:19 +0200] - slapd_poll(115) timed out*
>> *[10/Nov/2014:14:45:19 +0200] sasl_io_enable - Cannot enable SASL
>> security on connection in CLOSING state*
>> *[10/Nov/2014:14:45:19 +0200] - Error: could not add/remove IO layers
>> from connection*
>>  *[11/Nov/2014:11:48:09 +0200] - slapd shutting down - signaling
>> operation threads*
>> *[11/Nov/2014:11:48:09 +0200] - slapd shutting down - waiting for 30
>> threads to terminate*
>>  *[11/Nov/2014:13:14:12 +0200] - slapd shutting down - closing down
>> internal subsystems and plugins*
>> *[11/Nov/2014:13:14:12 +0200] - Waiting for 4 database threads to stop*
>> *[11/Nov/2014:13:14:13 +0200] - All database threads now stopped*
>> *[11/Nov/2014:13:14:13 +0200] - slapd stopped.*
>> *[11/Nov/2014:13:26:35 +0200] - 389-Directory/1.2.11.15
>> <http://1.2.11.15> B2014.219.179 starting up*
>> *[11/Nov/2014:13:26:35 +0200] schema-compat-plugin - warning: no entries
>> set up under cn=computers, cn=compat,dc=sample,dc=example*
>> *[11/Nov/2014:13:26:36 +0200] - Skipping CoS Definition cn=Password
>> Policy,cn=accounts,dc=sample,dc=example--no CoS Templates found, which
>> should be added before the CoS Definition.*
>> *[11/Nov/2014:13:26:36 +0200] - Skipping CoS Definition cn=Password
>> Policy,cn=accounts,dc=sample,dc=example--no CoS Templates found, which
>> should be added before the CoS Definition.*
>> *[11/Nov/2014:13:26:36 +0200] - slapd started.  Listening on All
>> Interfaces port 389 for LDAP requests*
>> *[11/Nov/2014:13:26:36 +0200] - Listening on All Interfaces port 636 for
>> LDAPS requests*
>> *[11/Nov/2014:13:26:36 +0200] - Listening on
>> /var/run/slapd-SAMPLE-EXAMPLE.socket for LDAPI requests*
>> *[11/Nov/2014:13:57:08 +0200] - slapd_poll(78) timed out*
>>
>>
>>
>>
>>
>> On Tue, Nov 11, 2014 at 1:19 PM, Martin Basti <mbasti at redhat.com> wrote:
>>
>>>  IMHO It's DS bug, can you share DS error log?
>>> pspacek CCed to examine named logs.
>>>
>>> Martin^2
>>>
>>>
>>> On 11/11/14 12:13, Walter van Lille wrote:
>>>
>>> Hi Martin, thanks for the reply.
>>> My version: bind-dyndb-ldap-2.3-5.el6.x86_64
>>> The server doesn't have journalctl installed but I have the outputs from
>>> the messages and named.run files that I included here:
>>>
>>>  Messages:
>>>
>>>  *Nov 11 12:30:13 freeipa named[1481]: error (network unreachable)
>>> resolving 'example.example.com.10.123.123.123/A/IN': 2001:500:2f::f#53*
>>> *Nov 11 12:30:23 freeipa named[1481]: LDAP query timed out. Try to
>>> adjust "timeout" parameter*
>>> *Nov 11 12:30:23 freeipa named[1481]: LDAP query timed out. Try to
>>> adjust "timeout" parameter*
>>> *Nov 11 12:30:33 freeipa named[1481]: LDAP query timed out. Try to
>>> adjust "timeout" parameter*
>>> *Nov 11 12:30:33 freeipa named[1481]: LDAP query timed out. Try to
>>> adjust "timeout" parameter*
>>>
>>>  Named.run:
>>>
>>>  *client 10.123.123.123#42639: transfer of 'example.example/IN':
>>> AXFR-style IXFR started*
>>> *client 10.123.123.123#42639: transfer of ''example.example/IN':
>>> AXFR-style IXFR ended*
>>> *client 10.123.123.123#46912: transfer of
>>> '10.123.123.123.in-addr.arpa/IN': AXFR-style IXFR started*
>>> *client 10.123.123.123#46912: transfer of
>>> '10.123.123.123.in-addr.arpa/IN': AXFR-style IXFR ended*
>>> *LDAP query timed out. Try to adjust "timeout" parameter*
>>> *LDAP query timed out. Try to adjust "timeout" parameter*
>>> *LDAP query timed out. Try to adjust "timeout" parameter*
>>>
>>>  I just replaced the IPs and the actual names with something more
>>> generic.
>>>
>>>  Regards,
>>>
>>>  Walter
>>>
>>> On Thu, Nov 6, 2014 at 5:00 PM, Martin Basti <mbasti at redhat.com> wrote:
>>>
>>>>   On 06/11/14 14:58, Walter van Lille wrote:
>>>>
>>>> Hi,
>>>>
>>>>  I need some assistance please.
>>>> I've taken over an IPA server to manage a few months ago, and it was
>>>> working fine until recently when it started acting up seemingly off its own
>>>> accord.
>>>> When I do an ipactl status it basically gives an output as shown below:
>>>>
>>>>
>>>>
>>>> *Directory Service: RUNNING *
>>>>
>>>>  *Loooooooooooooooooooooooooooooooooooooooooooooooooong pause... (To
>>>> the tune of 7 minutes sometimes)*
>>>>
>>>>  *KDC Service: RUNNING*
>>>> *KPASSWD Service: RUNNING*
>>>> *DNS Service: RUNNING*
>>>> *MEMCACHE Service: RUNNING*
>>>> *HTTP Service: RUNNING*
>>>> *CA Service: RUNNING*
>>>> *ADTRUST Service: RUNNING*
>>>> *EXTID Service: RUNNING*
>>>>
>>>>  Running top showed that ns-slapd was munching almost all my
>>>> resources, but I got that fixed by upping the cache. Unfortunately this did
>>>> not correct the issue and it still reacts in the same fashion, although the
>>>> resources have been freed up now.
>>>> I've noticed that when I run dig on either the local server or a remote
>>>> machine that the query basically just times out as shown here:
>>>>
>>>>   *dig freeipa.myexample.sample*
>>>>
>>>>  *; <<>> DiG 9.8.2rc1-RedHat-9.8.2-0.23.rc1.el6_5.1 <<>>
>>>> freeipa.myexample.sample*
>>>> *;; global options: +cmd*
>>>> *;; connection timed out; no servers could be reached*
>>>>
>>>>  When the KDC service fails to start, then name lookups seem OK, but
>>>> authentication fails. otherwise it's dead in the water.
>>>>
>>>>  This also happens:
>>>>
>>>>  *sudo ipactl status*
>>>> *Directory Service: RUNNING*
>>>> *Unknown error when retrieving list of services from LDAP:*
>>>>
>>>>  My software setup is as follows:
>>>>
>>>>
>>>> *CentOS release 6.5 (Final) *
>>>>
>>>> *389-ds-base.x86_64   1.2.11.15-34.el6_5 *
>>>>
>>>> *bind.x86_64          32:9.8.2-0.23.rc1.el6_5.1 *
>>>>  *bind-dyndb-ldap.x86_64*
>>>> *bind-libs.x86_64     32:9.8.2-0.23.rc1.el6_5.1*
>>>> *bind-utils.x86_64    32:9.8.2-0.23.rc1.el6_5.1*
>>>> *rpcbind.x86_64       0.2.0-11.el6
>>>> @anaconda-CentOS-201311291202.x86_64/6.5*
>>>> *samba4-winbind.x86_64*
>>>>
>>>> *krb5-server.x86_64   1.10.3-15.el6_5.1 *
>>>>
>>>>
>>>> *Linux 2.6.32-431.29.2.el6.x86_64 #1 SMP Tue Sep 9 21:36:05 UTC 2014
>>>> x86_64 x86_64 x86_64 GNU/Linux *
>>>>
>>>>  It's not a permanent situation as it sometimes runs 100% for a while,
>>>> but 80% of the time it is unusable. If anybody can assist me, please be so
>>>> kind.
>>>>
>>>>  Regards,
>>>>
>>>>  Walter
>>>>
>>>>    Hello please which version of bind-dyndb-ldap do you use?
>>>> I had similar issue with bind-dyndb-ldap, but it was development
>>>> version, I'm not sure if this is your case.
>>>> When named was failing, dirserv was really slow.
>>>>
>>>> Can you send journalctl -b -u named log when dig doesn't work??
>>>>
>>>> --
>>>> Martin Basti
>>>>
>>>>
>>>
>>>
>>>   --
>>> Martin Basti
>>>
>>>
>>
>>
>> --
>> Martin Basti
>>
>>
>>
>>
>>
>>
>>
>>
>>
>> --
>> Martin Basti
>>
>>
>>
>>
>>
>> --
>> Manage your subscription for the Freeipa-users mailing list:
>> https://www.redhat.com/mailman/listinfo/freeipa-users
>> Go To http://freeipa.org for more info on the project
>>
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/freeipa-users/attachments/20141113/4fb10ee5/attachment.htm>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: stacktrace.1415871697.zip
Type: application/zip
Size: 8953 bytes
Desc: not available
URL: <http://listman.redhat.com/archives/freeipa-users/attachments/20141113/4fb10ee5/attachment.zip>