[Freeipa-users] FreeIPA unresponsive - Causes DOS situations
Walter van Lille
walter.van.lille at gmail.com
Fri Nov 14 11:16:40 UTC 2014
Roughly 128 clients.. But when the services start going gaa-gaa it also
causes time-outs with the naming service running on the same server. I may
be wrong, but wouldn't that basically kill any chance of a client
connecting to the server anyway? I'm just lucky that the clients aren't
really there for general usage, so the only times that users actually try
to log in is when it is support staff that needs to check or restart a
pretty static, stable service.
I just want to add that the server is not hung when doing an ipactl status
and the KDC service is not started as below, so are we barking up the right
tree looking at the DS service? (PS, the KDC service was not manually
stopped by me.
*sudo ipactl status*
*Directory Service: RUNNING*
*KDC Service: STOPPED*
*KPASSWD Service: RUNNING*
*DNS Service: RUNNING*
*MEMCACHE Service: RUNNING*
*HTTP Service: RUNNING*
*CA Service: RUNNING*
*ADTRUST Service: RUNNING*
*EXTID Service: RUNNING*
On Thu, Nov 13, 2014 at 12:27 PM, Ludwig Krispenz <lkrispen at redhat.com>
wrote:
> Hmm, the symbols are there now, but all threads are idle, DS is just
> waiting on work to do. Which client do you expect to connect to DS, maybe
> you need to debug this client.
>
>
> On 11/13/2014 11:02 AM, Walter van Lille wrote:
>
> Thanks Rich, I have installed the packages and run gdb again. Hopefully
> the attached file is more useful.
>
> On Wed, Nov 12, 2014 at 4:16 PM, Rich Megginson <rmeggins at redhat.com>
> wrote:
>
>> On 11/12/2014 05:42 AM, Walter van Lille wrote:
>>
>> Thanks again for the assistance guys. I have saved two files and included
>> it here. Hope it tells you more than it does me.
>>
>>
>> These stack traces contain no useful symbols.
>>
>> Please read
>> http://www.port389.org/docs/389ds/FAQ/faq.html#debugging-crashes to find
>> out how to install the correct debuginfo packages on your system. For IPA
>> you will also need debuginfo packages for ipa and slapi-nis
>>
>> If debuginfo-install is not working, see
>> https://www.centos.org/forums/viewtopic.php?t=1710
>>
>> If you simply cannot figure out how to install the debuginfo packages,
>> please email me with the following information:
>>
>> $ rpm -q ipa-server slapi-nis 389-ds-base openldap db4 nss nspr glibc
>>
>> and I will make them available to you
>>
>> NOTE: With debuginfo packages, the version of the debuginfo package _must
>> exactly match_ the version of the associated package e.g. for 389-ds-base.x86_64
>> 1.2.11.15-34.el6_5 you must have
>> 389-ds-base-debuginfo.x86_64 1.2.11.15-34.el6_5
>>
>> If the versions do not match exactly, you will not get the symbols.
>>
>>
>> Regards,
>>
>> Wallter
>>
>> On Tue, Nov 11, 2014 at 8:03 PM, Rich Megginson <rmeggins at redhat.com>
>> wrote:
>>
>>> On 11/11/2014 10:37 AM, Martin Basti wrote:
>>>
>>> On 11/11/14 15:58, Rich Megginson wrote:
>>>
>>> On 11/11/2014 06:20 AM, Ludwig Krispenz wrote:
>>>
>>>
>>> On 11/11/2014 02:14 PM, Martin Basti wrote:
>>>
>>> Ludiwg (CCed) this seems like old (fixed?) DS bug.
>>>
>>> hmm, it says limit is 2097152, so it already has the new setting, but
>>> the error message says the packet is 800MB
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>> *Right. That usually means the server was expecting an encrypted SASL
>>> buffer from the client, but instead the client thinks SASL encryption
>>> negotiation failed and just sent a plain LDAP buffer. What version of
>>> 389-ds-base are you using? rpm -q 389-ds-base
>>> https://fedorahosted.org/389/ticket/47416
>>> <https://fedorahosted.org/389/ticket/47416> So, DO NOT increase your sasl
>>> io buffer size - it will not fix the problem, and it will leave you open to
>>> DoS attacks. *
>>>
>>>
>>> He is using
>>>
>>> CentOS release 6.5 (Final)
>>> 389-ds-base.x86_64 1.2.11.15-34.el6_5
>>>
>>>
>>>
>>> Ignore the "SASL encrypted packet length exceeds maximum allowed limit"
>>> error. All it means is the client has some error and is canceling the
>>> connection.
>>>
>>> The bugzilla associated with *47416
>>> <https://fedorahosted.org/389/ticket/47416> is targeted for RHEL 7.1. The
>>> problem is that we were never able to figure out what error the client was
>>> getting that caused the client to stop establishing the *SASL
>>> encryption, and the original customer who reported the problem dropped the
>>> case - everything just started working, no further errors. Note that 47416
>>> doesn't really fix the problem or address the root cause - the root cause
>>> is some error on the client side that causes it to stop doing encrypted
>>> SASL I/O and send an LDAP unbind request.
>>>
>>> Let's get back to the actual problem - what is the actual problem? The
>>> ldap server is hanging or unresponsive? If so, then see
>>> http://www.port389.org/docs/389ds/FAQ/faq.html#debugging-hangs. Let's
>>> get some dirsrv stack traces during the period of time when it appears to
>>> be unresponsive.
>>>
>>>
>>>
>>>
>>>
>>> On 11/11/14 13:13, Walter van Lille wrote:
>>>
>>> I've just cleaned out a ton of slapd_poll timed out messages from the
>>> output and changed the names to protect the innocent, :-)
>>> Here is the output as requested:
>>>
>>>
>>> *[05/Nov/2014:11:44:05 +0200] - SASL encrypted packet length exceeds
>>> maximum allowed limit (length=805565, limit=2097152). Change the
>>> nsslapd-maxsasliosize attribute in cn=config to increase limit.*
>>>
>>> *[10/Nov/2014:14:45:19 +0200] - slapd_poll(115) timed out*
>>> *[10/Nov/2014:14:45:19 +0200] sasl_io_enable - Cannot enable SASL
>>> security on connection in CLOSING state*
>>> *[10/Nov/2014:14:45:19 +0200] - Error: could not add/remove IO layers
>>> from connection*
>>> *[11/Nov/2014:11:48:09 +0200] - slapd shutting down - signaling
>>> operation threads*
>>> *[11/Nov/2014:11:48:09 +0200] - slapd shutting down - waiting for 30
>>> threads to terminate*
>>> *[11/Nov/2014:13:14:12 +0200] - slapd shutting down - closing down
>>> internal subsystems and plugins*
>>> *[11/Nov/2014:13:14:12 +0200] - Waiting for 4 database threads to stop*
>>> *[11/Nov/2014:13:14:13 +0200] - All database threads now stopped*
>>> *[11/Nov/2014:13:14:13 +0200] - slapd stopped.*
>>> *[11/Nov/2014:13:26:35 +0200] - 389-Directory/1.2.11.15
>>> <http://1.2.11.15> B2014.219.179 starting up*
>>> *[11/Nov/2014:13:26:35 +0200] schema-compat-plugin - warning: no entries
>>> set up under cn=computers, cn=compat,dc=sample,dc=example*
>>> *[11/Nov/2014:13:26:36 +0200] - Skipping CoS Definition cn=Password
>>> Policy,cn=accounts,dc=sample,dc=example--no CoS Templates found, which
>>> should be added before the CoS Definition.*
>>> *[11/Nov/2014:13:26:36 +0200] - Skipping CoS Definition cn=Password
>>> Policy,cn=accounts,dc=sample,dc=example--no CoS Templates found, which
>>> should be added before the CoS Definition.*
>>> *[11/Nov/2014:13:26:36 +0200] - slapd started. Listening on All
>>> Interfaces port 389 for LDAP requests*
>>> *[11/Nov/2014:13:26:36 +0200] - Listening on All Interfaces port 636 for
>>> LDAPS requests*
>>> *[11/Nov/2014:13:26:36 +0200] - Listening on
>>> /var/run/slapd-SAMPLE-EXAMPLE.socket for LDAPI requests*
>>> *[11/Nov/2014:13:57:08 +0200] - slapd_poll(78) timed out*
>>>
>>>
>>>
>>>
>>>
>>> On Tue, Nov 11, 2014 at 1:19 PM, Martin Basti <mbasti at redhat.com> wrote:
>>>
>>>> IMHO It's DS bug, can you share DS error log?
>>>> pspacek CCed to examine named logs.
>>>>
>>>> Martin^2
>>>>
>>>>
>>>> On 11/11/14 12:13, Walter van Lille wrote:
>>>>
>>>> Hi Martin, thanks for the reply.
>>>> My version: bind-dyndb-ldap-2.3-5.el6.x86_64
>>>> The server doesn't have journalctl installed but I have the outputs
>>>> from the messages and named.run files that I included here:
>>>>
>>>> Messages:
>>>>
>>>> *Nov 11 12:30:13 freeipa named[1481]: error (network unreachable)
>>>> resolving 'example.example.com.10.123.123.123/A/IN': 2001:500:2f::f#53*
>>>> *Nov 11 12:30:23 freeipa named[1481]: LDAP query timed out. Try to
>>>> adjust "timeout" parameter*
>>>> *Nov 11 12:30:23 freeipa named[1481]: LDAP query timed out. Try to
>>>> adjust "timeout" parameter*
>>>> *Nov 11 12:30:33 freeipa named[1481]: LDAP query timed out. Try to
>>>> adjust "timeout" parameter*
>>>> *Nov 11 12:30:33 freeipa named[1481]: LDAP query timed out. Try to
>>>> adjust "timeout" parameter*
>>>>
>>>> Named.run:
>>>>
>>>> *client 10.123.123.123#42639: transfer of 'example.example/IN':
>>>> AXFR-style IXFR started*
>>>> *client 10.123.123.123#42639: transfer of ''example.example/IN':
>>>> AXFR-style IXFR ended*
>>>> *client 10.123.123.123#46912: transfer of
>>>> '10.123.123.123.in-addr.arpa/IN': AXFR-style IXFR started*
>>>> *client 10.123.123.123#46912: transfer of
>>>> '10.123.123.123.in-addr.arpa/IN': AXFR-style IXFR ended*
>>>> *LDAP query timed out. Try to adjust "timeout" parameter*
>>>> *LDAP query timed out. Try to adjust "timeout" parameter*
>>>> *LDAP query timed out. Try to adjust "timeout" parameter*
>>>>
>>>> I just replaced the IPs and the actual names with something more
>>>> generic.
>>>>
>>>> Regards,
>>>>
>>>> Walter
>>>>
>>>> On Thu, Nov 6, 2014 at 5:00 PM, Martin Basti <mbasti at redhat.com> wrote:
>>>>
>>>>> On 06/11/14 14:58, Walter van Lille wrote:
>>>>>
>>>>> Hi,
>>>>>
>>>>> I need some assistance please.
>>>>> I've taken over an IPA server to manage a few months ago, and it was
>>>>> working fine until recently when it started acting up seemingly off its own
>>>>> accord.
>>>>> When I do an ipactl status it basically gives an output as shown below:
>>>>>
>>>>>
>>>>>
>>>>> *Directory Service: RUNNING *
>>>>>
>>>>> *Loooooooooooooooooooooooooooooooooooooooooooooooooong pause... (To
>>>>> the tune of 7 minutes sometimes)*
>>>>>
>>>>> *KDC Service: RUNNING*
>>>>> *KPASSWD Service: RUNNING*
>>>>> *DNS Service: RUNNING*
>>>>> *MEMCACHE Service: RUNNING*
>>>>> *HTTP Service: RUNNING*
>>>>> *CA Service: RUNNING*
>>>>> *ADTRUST Service: RUNNING*
>>>>> *EXTID Service: RUNNING*
>>>>>
>>>>> Running top showed that ns-slapd was munching almost all my
>>>>> resources, but I got that fixed by upping the cache. Unfortunately this did
>>>>> not correct the issue and it still reacts in the same fashion, although the
>>>>> resources have been freed up now.
>>>>> I've noticed that when I run dig on either the local server or a
>>>>> remote machine that the query basically just times out as shown here:
>>>>>
>>>>> *dig freeipa.myexample.sample*
>>>>>
>>>>> *; <<>> DiG 9.8.2rc1-RedHat-9.8.2-0.23.rc1.el6_5.1 <<>>
>>>>> freeipa.myexample.sample*
>>>>> *;; global options: +cmd*
>>>>> *;; connection timed out; no servers could be reached*
>>>>>
>>>>> When the KDC service fails to start, then name lookups seem OK, but
>>>>> authentication fails. otherwise it's dead in the water.
>>>>>
>>>>> This also happens:
>>>>>
>>>>> *sudo ipactl status*
>>>>> *Directory Service: RUNNING*
>>>>> *Unknown error when retrieving list of services from LDAP:*
>>>>>
>>>>> My software setup is as follows:
>>>>>
>>>>>
>>>>> *CentOS release 6.5 (Final) *
>>>>>
>>>>> *389-ds-base.x86_64 1.2.11.15-34.el6_5 *
>>>>>
>>>>> *bind.x86_64 32:9.8.2-0.23.rc1.el6_5.1 *
>>>>> *bind-dyndb-ldap.x86_64*
>>>>> *bind-libs.x86_64 32:9.8.2-0.23.rc1.el6_5.1*
>>>>> *bind-utils.x86_64 32:9.8.2-0.23.rc1.el6_5.1*
>>>>> *rpcbind.x86_64 0.2.0-11.el6
>>>>> @anaconda-CentOS-201311291202.x86_64/6.5*
>>>>> *samba4-winbind.x86_64*
>>>>>
>>>>> *krb5-server.x86_64 1.10.3-15.el6_5.1 *
>>>>>
>>>>>
>>>>> *Linux 2.6.32-431.29.2.el6.x86_64 #1 SMP Tue Sep 9 21:36:05 UTC 2014
>>>>> x86_64 x86_64 x86_64 GNU/Linux *
>>>>>
>>>>> It's not a permanent situation as it sometimes runs 100% for a
>>>>> while, but 80% of the time it is unusable. If anybody can assist me, please
>>>>> be so kind.
>>>>>
>>>>> Regards,
>>>>>
>>>>> Walter
>>>>>
>>>>> Hello please which version of bind-dyndb-ldap do you use?
>>>>> I had similar issue with bind-dyndb-ldap, but it was development
>>>>> version, I'm not sure if this is your case.
>>>>> When named was failing, dirserv was really slow.
>>>>>
>>>>> Can you send journalctl -b -u named log when dig doesn't work??
>>>>>
>>>>> --
>>>>> Martin Basti
>>>>>
>>>>>
>>>>
>>>>
>>>> --
>>>> Martin Basti
>>>>
>>>>
>>>
>>>
>>> --
>>> Martin Basti
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>> --
>>> Martin Basti
>>>
>>>
>>>
>>>
>>>
>>> --
>>> Manage your subscription for the Freeipa-users mailing list:
>>> https://www.redhat.com/mailman/listinfo/freeipa-users
>>> Go To http://freeipa.org for more info on the project
>>>
>>
>>
>>
>
>
>
>
> --
> Manage your subscription for the Freeipa-users mailing list:
> https://www.redhat.com/mailman/listinfo/freeipa-users
> Go To http://freeipa.org for more info on the project
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/freeipa-users/attachments/20141114/bb3886e3/attachment.htm>
More information about the Freeipa-users
mailing list