[Freeipa-users] IPA very very slow

Fri Jun 12 22:32:59 UTC 2015

On 06/12/2015 03:25 PM, William Graboyes wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA512
>
> Hi Ken,
>
> I ran this command back to back, I am snipping some of the results.
>
> First time I ran the command:
>
> time ldapsearch -x -h 127.0.0.1 "(uid=admin)"
> # extended LDIF
> #
> # LDAPv3
> # base <dc=foo,dc=org> (default) with scope subtree
> # filter: (uid=admin)
> # requesting: ALL
> #
>
> - --snip--
>
> # search result
> search: 2
> result: 0 Success
>
> # numResponses: 3
> # numEntries: 2
>
> real	0m0.056s
> user	0m0.003s
> sys	0m0.004s
>
>
> Run on the same server not 5 seconds after the previous command:
>
> time ldapsearch -x -h 127.0.0.1 "(uid=admin)"
> # extended LDIF
> #
> # LDAPv3
> # base <dc=foo,dc=org> (default) with scope subtree
> # filter: (uid=admin)
> # requesting: ALL
> #
>
> - -- snip --
>
> # search result
> search: 2
> result: 0 Success
>
> # numResponses: 3
> # numEntries: 2
>
> real	0m31.756s
> user	0m0.003s
> sys	0m0.005s

Ok.  First, see 
http://www.port389.org/docs/389ds/FAQ/faq.html#debugging-crashes

You'll also have to do
# debuginfo-install ipa-server slapi-nis
to get all of the ipa packages.

Next, see http://www.port389.org/docs/389ds/FAQ/faq.html#debugging-hangs

Reproduce the problem, and during the 30 seconds the directory server is 
processing the search request, run the gdb command several times to get 
stack traces during the search request.

>
>
> I am starting to see this error in the dirserv logs:
>
> [12/Jun/2015:14:06:51 -0700] slapi_ldap_bind - Error: could not send
> startTLS request: error -1 (Can't contact LDAP server) errno 107
> (Transport endpoint is not connected)
> [12/Jun/2015:14:11:51 -0700] slapi_ldap_bind - Error: could not send
> startTLS request: error -1 (Can't contact LDAP server) errno 107
> (Transport endpoint is not connected)
> [12/Jun/2015:14:16:51 -0700] slapi_ldap_bind - Error: could not send
> startTLS request: error -1 (Can't contact LDAP server) errno 107
> (Transport endpoint is not connected)
> [12/Jun/2015:14:21:51 -0700] slapi_ldap_bind - Error: could not send
> startTLS request: error -1 (Can't contact LDAP server) errno 107
> (Transport endpoint is not connected)

I doubt this is related to the performance.  This looks like the server 
is attempting to contact a replica which is down, and has backed off for 
the full 5 minute max backoff.

>
> Thanks,
> Bill Graboyes
>
>
> On 6/12/15 1:36 PM, Rich Megginson wrote:
>> On 06/12/2015 02:10 PM, Martin Kosek wrote:
>>> On 06/12/2015 09:15 PM, William Graboyes wrote:
>> Hi Martin,
>>
>> Here are the outputs of the various commands, cleaned of course:
>>
>> time ldapsearch SASL/EXTERNAL authentication started
>> ldap_sasl_interactive_bind_s: Unknown authentication method (-6)
>> additional info: SASL(-4): no mechanism available:
>>
>> real    0m32.464s user    0m0.385s sys    0m0.052s
>>>> This is quite long time. We should check respective dirsrv
>>>> errors and access logs snippets.
>>>>
>>>> Also, the command above did not exit successfully, I would
>>>> recommend doing at least
>>>>
>>>> # ldapsearch -x -h `hostname` "(uid=admin)"
>>> To eliminate DNS from the equation, use
>>> # time ldapsearch -x -h 127.0.0.1 "(uid=admin)"
>> time host ipa-server-2.foo.org <-- server with issues
>> ipa-server-2.foo.org has address 10.0.0.2
>>
>> real    0m0.070s user    0m0.010s sys    0m0.006s
>>
>> time host ipa-server-1.foo.org <-- replicant with no issues
>> ipa-server-1.foo.org has address 10.0.0.3
>>
>> real    0m0.073s user    0m0.012s sys    0m0.006s
>>
>> time kinit kinit: Cannot contact any KDC for realm 'FOO.ORG' while
>> getting initial credentials
>>
>> real    0m27.049s user    0m0.013s sys    0m0.004s
>>
>> ^^^ has been something I have been seeing intermittently
>>
>>
>>
>> On 6/12/15 12:11 AM, Martin Kosek wrote:
>>>>>>> Hi List,
>>>>>>>
>>>>>>> This is a problem that has surfaced after a reboot of
>>>>>>> this system in particular. It is being really, really
>>>>>>> slow.  In terms of hardware usage issues, there are none.
>>>>>>> It is taking 3-5 minutes to list users in the gui.
>>>>>>> Running commands like ipa-replica-manage list is taking
>>>>>>> between 30seconds and 3 minutes.  Memory usage is low,
>>>>>>> cpu usage is low, iops are low.  I really have no idea
>>>>>>> where to start here, there is noting really damning in
>>>>>>> the logs.  I have tried restarting IPA (ipactl restart)
>>>>>>> stopping and starting IPA (ipactl stop wait... ipactl
>>>>>>> start), and rebooting the entire server.
>>>>>>>
>>>>>>> The oddest thing is that there have been some krb errors
>>>>>>> saying that they cannot contact the krb server.. logging
>>>>>>> into the gui saying your session has timed out..
>>>>>>>
>>>>>>> It is just general strangeness.
>>>>>>>
>>>>>>> ipa-server-4.1.0-18.el7.centos.3.x86_64
>>>>>>> sssd-ipa-1.12.2-58.el7_1.6.x86_64
>>>>>>> krb5-server-1.12.2-14.el7.x86_64
>>>>>>>
>>>>>>> Any help would be greatly appreciated.
>>>>>>>
>>>>>>> Thanks, Bill
>>>>>> I would recommend starting with simple things, seeing the
>>>>>> performance and then following with more complex stuff:
>>>>>>
>>>>>> - Try bare "ldapsearch" against the FreeIPA LDAP server,
>>>>>> see the response rate. If it is also slow, we have the root
>>>>>> cause. Before ringing on DS people doors, see if for
>>>>>> example DNS is not slow and there are no DNS timeouts in
>>>>>> play - "host ipa.server.test" will tell you that
>>>>>>
>>>>>> - If DS is OK, try Kerberos - kinit, kvno commands
>>>>>>
>>>>>> - If Kerberos is also OK and "ipa-replica-manage list" is
>>>>>> still slow, maybe we should just "strace" it to see what it
>>>>>> waits on.
>>>>>>
>>>>>> HTH, Martin
>>>>>>
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG/MacGPG2 v2
> Comment: GPGTools - https://gpgtools.org
>
> iQIcBAEBCgAGBQJVe05nAAoJEJFMz73A1+zrg7QP/3s19crgzjSeic4KYZ3nYn80
> 9CWVLlm2/m7XU8Zeazm0nmlfMDTeBWJOLG0bXQKV3MYcGChSnX/vxQ9hqWJtzzvq
> 30MpgfyRKCNFOUcfAXB4YDINFd6/RrWl/lRii0eNksli+DXDlzarXsby+11G42kn
> XtRp/7EPmZixdy8G+CLYzY2mgzpyTheMWAk8+CQORjLJTi/hmMrkKxC5Ij8Q5Vtp
> qG2oUXgMeoBnCQyij+AQ1IqrlByt3iTtXsx5PdxB8eQ/kswOghFVokM83a1IqfOL
> yvspUpnCg5XgU9fN7+HDt45d/i2ZcXcM7gQjlAUmFtE2c0kcuu7LTiahD56ESyMc
> DkDQqI/MO/X/nb6JD7QNXy4bCjHiAPB2LyVbimqDepoyiW1QwuptdBuZmBZ6DXkj
> l2mbVUOma9ql61Cl/YTw4v7lsQS7Vf3Hc8Qua6o12fLJIYOwPL9FgDTznGh7S8F3
> DhUA0m1kzaZFB+7Js52UoiV9Qh3sRCSx0RyZ5hfPX3LgZyw+XORvjNQvPTYhSQ7A
> SKAK7/TEwlLxSVWikWvwfpMankVdbSVo06BsgHEkGdM/O8ymbxbLqGZo1FwFaocA
> Uocf4p1K7JBz/FfNb5OtI4o3JTiWs7LLGEYGZwGtlHHFZV42VfWdyeA3V/v0GUuW
> UXKUprDG3PjvK5HG2rP1
> =hr/W
> -----END PGP SIGNATURE-----
>