[Freeipa-users] ns-slapd using all CPU ressources
Martin Basti
mbasti at redhat.com
Wed Jan 20 09:15:44 UTC 2016
On 20.01.2016 09:29, Domingues Luis Filipe wrote:
> Hi,
>
> Thanks, this is actually the version we are running.
>
> Do you have a link to the ticket? I tried to find it on the bug tracer but I have always a ticket not found.
>
> Luis
Link to DS ticket
https://fedorahosted.org/389/ticket/48379
>
> -----Original Message-----
> From: freeipa-users-bounces at redhat.com [mailto:freeipa-users-bounces at redhat.com] On Behalf Of Ludwig Krispenz
> Sent: mardi 19 janvier 2016 16:59
> To: freeipa-users at redhat.com
> Subject: Re: [Freeipa-users] ns-slapd using all CPU ressources
>
> Hi,
> if you are running 389-ds 1.3.4+ you may hit, ticket #48379. It id fixed and a new build is in preparation
>
> Ludwig
>
> On 01/19/2016 03:39 PM, Domingues Luis Filipe wrote:
>> Hi,
>>
>> Reading the backtrace I have 30 threads with the same stack:
>>
>> Thread 6 (Thread 0x7f572efed700 (LWP 1335)):
>> #0 0x00007f576f80a877 in sched_yield () from /lib64/libc.so.6 No
>> symbol table info available.
>> #1 0x00007f577014df28 in PR_Sleep () from /lib64/libnspr4.so No
>> symbol table info available.
>> #2 0x000055c939e9e7c7 in connection_threadmain () No symbol table
>> info available.
>> #3 0x00007f577014d5cb in _pt_root () from /lib64/libnspr4.so No
>> symbol table info available.
>> #4 0x00007f576faec60a in start_thread () from /lib64/libpthread.so.0
>> No symbol table info available.
>> #5 0x00007f576f826a4d in clone () from /lib64/libc.so.6 No symbol
>> table info available.
>>
>> While the other instance which is running fine, almost all threads are waiting on a cond_wait, with thise stack:
>> Thread 48 (Thread 0x7fced53a9700 (LWP 1871)):
>> #0 0x00007fcee9269b10 in pthread_cond_wait@@GLIBC_2.3.2 () from
>> /lib64/libpthread.so.0 No symbol table info available.
>> #1 0x00007fcee98bfcf0 in PR_WaitCondVar () from /lib64/libnspr4.so No
>> symbol table info available.
>> #2 0x00007fceeb7172c8 in slapi_wait_condvar () from
>> /usr/lib64/dirsrv/libslapd.so.0 No symbol table info available.
>> #3 0x00007fcee127a67e in cos_cache_wait_on_change () from
>> /usr/lib64/dirsrv/plugins/libcos-plugin.so
>> No symbol table info available.
>> #4 0x00007fcee98c55cb in _pt_root () from /lib64/libnspr4.so No
>> symbol table info available.
>> #5 0x00007fcee926460a in start_thread () from /lib64/libpthread.so.0
>> No symbol table info available.
>> #6 0x00007fcee8f9ea4d in clone () from /lib64/libc.so.6 No symbol
>> table info available.
>>
>> Luis.
>> ________________________________________
>> From: Rob Crittenden [rcritten at redhat.com]
>> Sent: Friday, January 15, 2016 3:51 PM
>> To: Domingues Luis Filipe; freeipa-users at redhat.com
>> Cc: Aviolat Romain
>> Subject: Re: [Freeipa-users] ns-slapd using all CPU ressources
>>
>> Domingues Luis Filipe wrote:
>>> Hi all,
>>>
>>> On our infra, we have two machines running Fedora with FreeIPA installed.
>>>
>>> we have an issue with ns-slapd using 100% of CPU after a while. If we
>>> restart the service, it starts to use all CPU resources after one day.
>>>
>>> Outpute of the command strace -c -p <ns-slapd PID> running for 4 minutes is:
>>>
>>> % time seconds usecs/call calls errors syscall
>>> ------ ----------- ----------- --------- --------- ----------------
>>> 99.80 229.603633 11247 20415 poll
>>> 0.15 0.340032 10 32983 4 futex
>>> 0.05 0.114068 114068 1 restart_syscall
>>> 0.00 0.003464 0 20420 20416 getpeername
>>> 0.00 0.002752 0 20416 clock_gettime
>>> 0.00 0.001920 0 9840 read
>>> 0.00 0.000205 5 45 close
>>> 0.00 0.000036 2 22 access
>>> 0.00 0.000017 1 22 open
>>> 0.00 0.000016 1 24 accept
>>> 0.00 0.000012 0 45 setsockopt
>>> 0.00 0.000007 0 22 fstat
>>> 0.00 0.000000 0 22 stat
>>> 0.00 0.000000 0 1 sendto
>>> 0.00 0.000000 0 24 getsockname
>>> 0.00 0.000000 0 4 getsockopt
>>> 0.00 0.000000 0 70 fcntl
>>> 0.00 0.000000 0 22 gettimeofday
>>> ------ ----------- ----------- --------- --------- ----------------
>>> 100.00 230.066162 104398 20420 total
>>>
>>>
>>>
>>> Plus we looked at the syscalls using FTrace:
>>>
>>> ns-slapd-7963 [000] .... 4063846.395630: sys_sched_yield()
>>> ns-slapd-7956 [000] .... 4063846.395631: sys_sched_yield -> 0x0
>>> ns-slapd-7956 [000] .... 4063846.395632: sys_sched_yield()
>>> ns-slapd-7973 [000] .... 4063846.395633: sys_sched_yield -> 0x0
>>> ns-slapd-7973 [000] .... 4063846.395634: sys_sched_yield()
>>> ns-slapd-7965 [000] .... 4063846.395635: sys_sched_yield -> 0x0
>>> ns-slapd-7965 [000] .... 4063846.395637: sys_sched_yield()
>>> ns-slapd-7963 [000] .... 4063846.395637: sys_sched_yield -> 0x0
>>> ns-slapd-7963 [000] .... 4063846.395639: sys_sched_yield()
>>> ns-slapd-7956 [000] .... 4063846.395640: sys_sched_yield -> 0x0
>>> ns-slapd-7956 [000] .... 4063846.395641: sys_sched_yield()
>>> ns-slapd-7973 [000] .... 4063846.395642: sys_sched_yield -> 0x0
>>> ns-slapd-7973 [000] .... 4063846.395643: sys_sched_yield()
>>> ns-slapd-7965 [000] .... 4063846.395644: sys_sched_yield -> 0x0
>>>
>>> The sys_sched_yield function is called almost every 2 microseconds. It seems too much.
>> Your best bet is to get a pstack or full backtrace to see what 389-ds
>> is doing. See
>> http://directory.fedoraproject.org/docs/389ds/FAQ/faq.html#debugging-h
>> angs
>>
>> rob
>>
> --
> Manage your subscription for the Freeipa-users mailing list:
> https://www.redhat.com/mailman/listinfo/freeipa-users
> Go to http://freeipa.org for more info on the project
>
More information about the Freeipa-users
mailing list