[Freeipa-users] IPA very very slow

Fri Jun 12 23:07:52 UTC 2015

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA512

Martin, Et al,

Now that debugging is installed and running, I cannot duplicate.
Isn't that always the way though?

I'll let you know if it happens again.

Thanks,
Bill

On 6/12/15 3:32 PM, Rich Megginson wrote:
> On 06/12/2015 03:25 PM, William Graboyes wrote: Hi Ken,
> 
> I ran this command back to back, I am snipping some of the
> results.
> 
> First time I ran the command:
> 
> time ldapsearch -x -h 127.0.0.1 "(uid=admin)" # extended LDIF # #
> LDAPv3 # base <dc=foo,dc=org> (default) with scope subtree #
> filter: (uid=admin) # requesting: ALL #
> 
> --snip--
> 
> # search result search: 2 result: 0 Success
> 
> # numResponses: 3 # numEntries: 2
> 
> real    0m0.056s user    0m0.003s sys    0m0.004s
> 
> 
> Run on the same server not 5 seconds after the previous command:
> 
> time ldapsearch -x -h 127.0.0.1 "(uid=admin)" # extended LDIF # #
> LDAPv3 # base <dc=foo,dc=org> (default) with scope subtree #
> filter: (uid=admin) # requesting: ALL #
> 
> -- snip --
> 
> # search result search: 2 result: 0 Success
> 
> # numResponses: 3 # numEntries: 2
> 
> real    0m31.756s user    0m0.003s sys    0m0.005s
> 
>> Ok.  First, see 
>> http://www.port389.org/docs/389ds/FAQ/faq.html#debugging-crashes
> 
>> You'll also have to do # debuginfo-install ipa-server slapi-nis 
>> to get all of the ipa packages.
> 
>> Next, see
>> http://www.port389.org/docs/389ds/FAQ/faq.html#debugging-hangs
> 
>> Reproduce the problem, and during the 30 seconds the directory
>> server is processing the search request, run the gdb command
>> several times to get stack traces during the search request.
> 
> 
> 
> I am starting to see this error in the dirserv logs:
> 
> [12/Jun/2015:14:06:51 -0700] slapi_ldap_bind - Error: could not
> send startTLS request: error -1 (Can't contact LDAP server) errno
> 107 (Transport endpoint is not connected) [12/Jun/2015:14:11:51
> -0700] slapi_ldap_bind - Error: could not send startTLS request:
> error -1 (Can't contact LDAP server) errno 107 (Transport endpoint
> is not connected) [12/Jun/2015:14:16:51 -0700] slapi_ldap_bind -
> Error: could not send startTLS request: error -1 (Can't contact
> LDAP server) errno 107 (Transport endpoint is not connected) 
> [12/Jun/2015:14:21:51 -0700] slapi_ldap_bind - Error: could not
> send startTLS request: error -1 (Can't contact LDAP server) errno
> 107 (Transport endpoint is not connected)
> 
>> I doubt this is related to the performance.  This looks like the
>> server is attempting to contact a replica which is down, and has
>> backed off for the full 5 minute max backoff.
> 
> 
> Thanks, Bill Graboyes
> 
> 
> On 6/12/15 1:36 PM, Rich Megginson wrote:
>>>> On 06/12/2015 02:10 PM, Martin Kosek wrote:
>>>>> On 06/12/2015 09:15 PM, William Graboyes wrote:
>>>> Hi Martin,
>>>> 
>>>> Here are the outputs of the various commands, cleaned of
>>>> course:
>>>> 
>>>> time ldapsearch SASL/EXTERNAL authentication started 
>>>> ldap_sasl_interactive_bind_s: Unknown authentication method
>>>> (-6) additional info: SASL(-4): no mechanism available:
>>>> 
>>>> real    0m32.464s user    0m0.385s sys    0m0.052s
>>>>>> This is quite long time. We should check respective
>>>>>> dirsrv errors and access logs snippets.
>>>>>> 
>>>>>> Also, the command above did not exit successfully, I
>>>>>> would recommend doing at least
>>>>>> 
>>>>>> # ldapsearch -x -h `hostname` "(uid=admin)"
>>>>> To eliminate DNS from the equation, use # time ldapsearch
>>>>> -x -h 127.0.0.1 "(uid=admin)"
>>>> time host ipa-server-2.foo.org <-- server with issues 
>>>> ipa-server-2.foo.org has address 10.0.0.2
>>>> 
>>>> real    0m0.070s user    0m0.010s sys    0m0.006s
>>>> 
>>>> time host ipa-server-1.foo.org <-- replicant with no issues 
>>>> ipa-server-1.foo.org has address 10.0.0.3
>>>> 
>>>> real    0m0.073s user    0m0.012s sys    0m0.006s
>>>> 
>>>> time kinit kinit: Cannot contact any KDC for realm 'FOO.ORG'
>>>> while getting initial credentials
>>>> 
>>>> real    0m27.049s user    0m0.013s sys    0m0.004s
>>>> 
>>>> ^^^ has been something I have been seeing intermittently
>>>> 
>>>> 
>>>> 
>>>> On 6/12/15 12:11 AM, Martin Kosek wrote:
>>>>>>>>> Hi List,
>>>>>>>>> 
>>>>>>>>> This is a problem that has surfaced after a reboot
>>>>>>>>> of this system in particular. It is being really,
>>>>>>>>> really slow.  In terms of hardware usage issues,
>>>>>>>>> there are none. It is taking 3-5 minutes to list
>>>>>>>>> users in the gui. Running commands like
>>>>>>>>> ipa-replica-manage list is taking between 30seconds
>>>>>>>>> and 3 minutes.  Memory usage is low, cpu usage is
>>>>>>>>> low, iops are low.  I really have no idea where to
>>>>>>>>> start here, there is noting really damning in the
>>>>>>>>> logs.  I have tried restarting IPA (ipactl
>>>>>>>>> restart) stopping and starting IPA (ipactl stop
>>>>>>>>> wait... ipactl start), and rebooting the entire
>>>>>>>>> server.
>>>>>>>>> 
>>>>>>>>> The oddest thing is that there have been some krb
>>>>>>>>> errors saying that they cannot contact the krb
>>>>>>>>> server.. logging into the gui saying your session
>>>>>>>>> has timed out..
>>>>>>>>> 
>>>>>>>>> It is just general strangeness.
>>>>>>>>> 
>>>>>>>>> ipa-server-4.1.0-18.el7.centos.3.x86_64 
>>>>>>>>> sssd-ipa-1.12.2-58.el7_1.6.x86_64 
>>>>>>>>> krb5-server-1.12.2-14.el7.x86_64
>>>>>>>>> 
>>>>>>>>> Any help would be greatly appreciated.
>>>>>>>>> 
>>>>>>>>> Thanks, Bill
>>>>>>>> I would recommend starting with simple things, seeing
>>>>>>>> the performance and then following with more complex
>>>>>>>> stuff:
>>>>>>>> 
>>>>>>>> - Try bare "ldapsearch" against the FreeIPA LDAP
>>>>>>>> server, see the response rate. If it is also slow, we
>>>>>>>> have the root cause. Before ringing on DS people
>>>>>>>> doors, see if for example DNS is not slow and there
>>>>>>>> are no DNS timeouts in play - "host ipa.server.test"
>>>>>>>> will tell you that
>>>>>>>> 
>>>>>>>> - If DS is OK, try Kerberos - kinit, kvno commands
>>>>>>>> 
>>>>>>>> - If Kerberos is also OK and "ipa-replica-manage
>>>>>>>> list" is still slow, maybe we should just "strace" it
>>>>>>>> to see what it waits on.
>>>>>>>> 
>>>>>>>> HTH, Martin
>>>>>>>> 
>> 
> 
-----BEGIN PGP SIGNATURE-----
Version: GnuPG/MacGPG2 v2
Comment: GPGTools - https://gpgtools.org

iQIcBAEBCgAGBQJVe2ZIAAoJEJFMz73A1+zr8qgP/jmKVGOwFMVZFowdB8rPBHOT
//TQ11V5E0nhMz3r0zWoUDOEMYyQy5Oy9d4CmoMa5oWimnTHJUJanUMK5hWHlo/g
d8lSyvEDmQbXpNCduwIeMwupI0C669X5/EYesvth3khebsHxlz7kov69XyknBcj2
8czY4eUAkS8jB60Ua4UOMa0ruuBtYU40FCdS8GiWFBYxjfnLLqUX4vsfyke+vJ0B
AYPlzsf0ipKdPVPhjxaWmjHJmU/Y0tK5/a8CrDpkQH19UzYjFX8BSpyAJBGOQVYw
4ZlhZHXmiGhuDnyoIZIHFOeo0BmGugiN85zLf4G4mFxkn2TNOp28+w94EBZxb9kI
rQ1vEE2eUF0f9n5usdXb+gHwm3yhnhOvOkV+MLhJXNTeTlEo9Kl/EEnWZamh4wRy
hsMP2j6/XeDDzNFd4q1JaiScGVwfzIAizFGzxz6jkluA8B/aCz05pjMVDf/HmLPh
64OygyzhYtkLTe6DTH/WwoLV664IDlzs6LMxEDix37dI+9e8TsLfdp9ZexQV24sR
qZEYqWcPqDfXPvjkXjsqmeU7mJMOaQsu7be1Ad9isoOocn1WorIx7eCFtHIT5xgF
D58uhXk6hHj1tYA3fDQY2ybWgD0HhST52XbRezwQQ2Mw4F5QcUBt/WtqEPwMVlDU
Pfp6LpG4V0Kph4zc3EF3
=IOQh
-----END PGP SIGNATURE-----