<html>
<head>
<meta content="text/html; charset=utf-8" http-equiv="Content-Type">
</head>
<body text="#000000" bgcolor="#FFFFFF">
Hi Rakesh,<br>
<br>
Thanks you soooo much for you continuous help on this !!<br>
<br>
If the hang is still going on, would you dump the DB lock
information with:<br>
<blockquote>db_stat -CA -N -h
/var/lib/dirsrv/slapd-<instance>/db >/tmp/db_lock_info<br>
<br>
</blockquote>
It is looking like several threads are waiting for one/several db
locks. With the lock info we will know which thread acquired
it/them.<br>
A guess is that it could be the checkpointing thread but this one is
also waiting (on a memory pool).<br>
<br>
I am copying Ludwig on this thread in case this pstack rings any
bell.<br>
<br>
best regards<br>
thierry<br>
<br>
<div class="moz-cite-prefix">On 09/05/2016 02:05 PM, Rakesh
Rajasekharan wrote:<br>
</div>
<blockquote
cite="mid:CANAMAkrhnL5Qyi5v1HL08J5a+ApZanWJurf2Ta6Oyu7VCBnjSQ@mail.gmail.com"
type="cite">
<div dir="ltr">
<div>
<div>
<div>
<div>
<div>Hi Thierry,<br>
<br>
</div>
I was able to now reporduce the issue.. <br>
<br>
</div>
the ns-slapd process just got hung,.. however, no core
file was generate..<br>
<br>
</div>
I manually created a coredump using gcore and then took a
strace of it<br>
this is strace from the coredump generated<br>
<br>
GNU gdb (GDB) Red Hat Enterprise Linux 7.6.1-80.el7<br>
Copyright (C) 2013 Free Software Foundation, Inc.<br>
License GPLv3+: GNU GPL version 3 or later <<a
moz-do-not-send="true"
href="http://gnu.org/licenses/gpl.html"><a class="moz-txt-link-freetext" href="http://gnu.org/licenses/gpl.html">http://gnu.org/licenses/gpl.html</a></a>><br>
This is free software: you are free to change and
redistribute it.<br>
There is NO WARRANTY, to the extent permitted by law. Type
"show copying"<br>
and "show warranty" for details.<br>
This GDB was configured as "x86_64-redhat-linux-gnu".<br>
For bug reporting instructions, please see:<br>
<<a moz-do-not-send="true"
href="http://www.gnu.org/software/gdb/bugs/">http://www.gnu.org/software/gdb/bugs/</a>>...<br>
Reading symbols from /usr/sbin/ns-slapd...Reading symbols
from /usr/sbin/ns-slapd...(no debugging symbols
found)...done.<br>
(no debugging symbols found)...done.<br>
[New LWP 105159]<br>
[New LWP 105160]<br>
[New LWP 105161]<br>
[New LWP 105162]<br>
[New LWP 105163]<br>
[New LWP 105164]<br>
[New LWP 105165]<br>
[New LWP 105166]<br>
[New LWP 105167]<br>
[New LWP 105168]<br>
[New LWP 105171]<br>
[New LWP 105172]<br>
[New LWP 105173]<br>
[New LWP 105174]<br>
[New LWP 105175]<br>
[New LWP 105176]<br>
[New LWP 105177]<br>
[New LWP 105178]<br>
[New LWP 105179]<br>
[New LWP 105180]<br>
[New LWP 105181]<br>
[New LWP 105182]<br>
[New LWP 105183]<br>
[New LWP 105184]<br>
[New LWP 105185]<br>
[New LWP 105186]<br>
[New LWP 105187]<br>
[New LWP 105188]<br>
[New LWP 105189]<br>
[New LWP 105190]<br>
[New LWP 105191]<br>
[New LWP 105192]<br>
[New LWP 105193]<br>
[New LWP 105194]<br>
[New LWP 105195]<br>
[New LWP 105196]<br>
[New LWP 105197]<br>
[New LWP 105198]<br>
[New LWP 105199]<br>
[New LWP 105200]<br>
[New LWP 105201]<br>
[New LWP 105203]<br>
[New LWP 105215]<br>
[New LWP 105158]<br>
[Thread debugging using libthread_db enabled]<br>
Using host libthread_db library "/lib64/libthread_db.so.1".<br>
Core was generated by `/usr/sbin/ns-slapd'.<br>
#0 0x00007f1b48ae69b3 in select () from /lib64/libc.so.6<br>
<br>
Thread 44 (Thread 0x7f1b4b6d9840 (LWP 105158)):<br>
#0 0x00007f1b48dc56d5 in <a class="moz-txt-link-abbreviated" href="mailto:pthread_cond_wait@@GLIBC_2.3.2">pthread_cond_wait@@GLIBC_2.3.2</a> ()
from /lib64/libpthread.so.0<br>
No symbol table info available.<br>
#1 0x00007f1b4941b2c3 in PR_EnterMonitor () from
/lib64/libnspr4.so<br>
No symbol table info available.<br>
#2 0x00007f1b4b7125ed in setup_pr_read_pds ()<br>
No symbol table info available.<br>
#3 0x00007f1b4b715fd9 in slapd_daemon ()<br>
No symbol table info available.<br>
#4 0x00007f1b4b707c6c in main ()<br>
No symbol table info available.<br>
<br>
Thread 43 (Thread 0x7f1a5df0f700 (LWP 105215)):<br>
#0 0x00007f1b48dc56d5 in <a class="moz-txt-link-abbreviated" href="mailto:pthread_cond_wait@@GLIBC_2.3.2">pthread_cond_wait@@GLIBC_2.3.2</a> ()
from /lib64/libpthread.so.0<br>
No symbol table info available.<br>
#1 0x00007f1b4941b050 in PR_WaitCondVar () from
/lib64/libnspr4.so<br>
No symbol table info available.<br>
#2 0x00007f1b4b71df25 in ps_send_results ()<br>
No symbol table info available.<br>
#3 0x00007f1b494207bb in _pt_root () from
/lib64/libnspr4.so<br>
No symbol table info available.<br>
#4 0x00007f1b48dc1dc5 in start_thread () from
/lib64/libpthread.so.0<br>
No symbol table info available.<br>
#5 0x00007f1b48aef28d in clone () from /lib64/libc.so.6<br>
No symbol table info available.<br>
<br>
Thread 42 (Thread 0x7f1b03fe7700 (LWP 105203)):<br>
#0 0x00007f1b48dc56d5 in <a class="moz-txt-link-abbreviated" href="mailto:pthread_cond_wait@@GLIBC_2.3.2">pthread_cond_wait@@GLIBC_2.3.2</a> ()
from /lib64/libpthread.so.0<br>
No symbol table info available.<br>
#1 0x00007f1b4941b050 in PR_WaitCondVar () from
/lib64/libnspr4.so<br>
No symbol table info available.<br>
#2 0x00007f1b4b71df25 in ps_send_results ()<br>
No symbol table info available.<br>
#3 0x00007f1b494207bb in _pt_root () from
/lib64/libnspr4.so<br>
No symbol table info available.<br>
#4 0x00007f1b48dc1dc5 in start_thread () from
/lib64/libpthread.so.0<br>
No symbol table info available.<br>
#5 0x00007f1b48aef28d in clone () from /lib64/libc.so.6<br>
No symbol table info available.<br>
<br>
Thread 41 (Thread 0x7f1b047e8700 (LWP 105201)):<br>
#0 0x00007f1b48ae69b3 in select () from /lib64/libc.so.6<br>
No symbol table info available.<br>
#1 0x00007f1b4b2710e9 in DS_Sleep () from
/usr/lib64/dirsrv/libslapd.so.0<br>
No symbol table info available.<br>
#2 0x00007f1b4b712325 in time_thread ()<br>
No symbol table info available.<br>
#3 0x00007f1b494207bb in _pt_root () from
/lib64/libnspr4.so<br>
No symbol table info available.<br>
#4 0x00007f1b48dc1dc5 in start_thread () from
/lib64/libpthread.so.0<br>
No symbol table info available.<br>
#5 0x00007f1b48aef28d in clone () from /lib64/libc.so.6<br>
No symbol table info available.<br>
<br>
Thread 40 (Thread 0x7f1b04fe9700 (LWP 105200)):<br>
#0 0x00007f1b48dc56d5 in <a class="moz-txt-link-abbreviated" href="mailto:pthread_cond_wait@@GLIBC_2.3.2">pthread_cond_wait@@GLIBC_2.3.2</a> ()
from /lib64/libpthread.so.0<br>
No symbol table info available.<br>
#1 0x00007f1b433212f3 in __db_hybrid_mutex_suspend () from
/lib64/<a moz-do-not-send="true" href="http://libdb-5.3.so">libdb-5.3.so</a><br>
No symbol table info available.<br>
#2 0x00007f1b43320640 in __db_tas_mutex_lock () from
/lib64/<a moz-do-not-send="true" href="http://libdb-5.3.so">libdb-5.3.so</a><br>
No symbol table info available.<br>
#3 0x00007f1b4343ae1b in __memp_fget () from /lib64/<a
moz-do-not-send="true" href="http://libdb-5.3.so">libdb-5.3.so</a><br>
No symbol table info available.<br>
#4 0x00007f1b4333e211 in __bam_search () from /lib64/<a
moz-do-not-send="true" href="http://libdb-5.3.so">libdb-5.3.so</a><br>
No symbol table info available.<br>
#5 0x00007f1b43329256 in __bamc_search () from /lib64/<a
moz-do-not-send="true" href="http://libdb-5.3.so">libdb-5.3.so</a><br>
No symbol table info available.<br>
#6 0x00007f1b4332d6c4 in __bamc_put () from /lib64/<a
moz-do-not-send="true" href="http://libdb-5.3.so">libdb-5.3.so</a><br>
No symbol table info available.<br>
#7 0x00007f1b433e46d5 in __dbc_iput () from /lib64/<a
moz-do-not-send="true" href="http://libdb-5.3.so">libdb-5.3.so</a><br>
No symbol table info available.<br>
#8 0x00007f1b433dfa0e in __db_put () from /lib64/<a
moz-do-not-send="true" href="http://libdb-5.3.so">libdb-5.3.so</a><br>
No symbol table info available.<br>
#9 0x00007f1b433f4fa4 in __db_put_pp () from /lib64/<a
moz-do-not-send="true" href="http://libdb-5.3.so">libdb-5.3.so</a><br>
No symbol table info available.<br>
#10 0x00007f1b3e8cf4cf in idl_new_insert_key () from
/usr/lib64/dirsrv/plugins/libback-ldbm.so<br>
No symbol table info available.<br>
#11 0x00007f1b3e8de5b5 in addordel_values_sv.isra.3 () from
/usr/lib64/dirsrv/plugins/libback-ldbm.so<br>
No symbol table info available.<br>
#12 0x00007f1b3e8df25a in index_addordel_values_ext_sv ()
from /usr/lib64/dirsrv/plugins/libback-ldbm.so<br>
No symbol table info available.<br>
#13 0x00007f1b3e8df694 in index_addordel_values_sv () from
/usr/lib64/dirsrv/plugins/libback-ldbm.so<br>
No symbol table info available.<br>
#14 0x00007f1b3e8df75a in index_addordel_entry () from
/usr/lib64/dirsrv/plugins/libback-ldbm.so<br>
No symbol table info available.<br>
#15 0x00007f1b3e8e25fa in ldbm_back_add () from
/usr/lib64/dirsrv/plugins/libback-ldbm.so<br>
No symbol table info available.<br>
#16 0x00007f1b4b1e4cca in op_shared_add () from
/usr/lib64/dirsrv/libslapd.so.0<br>
No symbol table info available.<br>
#17 0x00007f1b4b1e6020 in do_add () from
/usr/lib64/dirsrv/libslapd.so.0<br>
No symbol table info available.<br>
#18 0x00007f1b4b711303 in connection_threadmain ()<br>
No symbol table info available.<br>
#19 0x00007f1b494207bb in _pt_root () from
/lib64/libnspr4.so<br>
No symbol table info available.<br>
#20 0x00007f1b48dc1dc5 in start_thread () from
/lib64/libpthread.so.0<br>
No symbol table info available.<br>
#21 0x00007f1b48aef28d in clone () from /lib64/libc.so.6<br>
No symbol table info available.<br>
<br>
Thread 39 (Thread 0x7f1b057ea700 (LWP 105199)):<br>
#0 0x00007f1b48dc56d5 in <a class="moz-txt-link-abbreviated" href="mailto:pthread_cond_wait@@GLIBC_2.3.2">pthread_cond_wait@@GLIBC_2.3.2</a> ()
from /lib64/libpthread.so.0<br>
No symbol table info available.<br>
#1 0x00007f1b4941b050 in PR_WaitCondVar () from
/lib64/libnspr4.so<br>
No symbol table info available.<br>
#2 0x00007f1b4b70efee in connection_wait_for_new_work ()<br>
No symbol table info available.<br>
#3 0x00007f1b4b710719 in connection_threadmain ()<br>
No symbol table info available.<br>
#4 0x00007f1b494207bb in _pt_root () from
/lib64/libnspr4.so<br>
No symbol table info available.<br>
#5 0x00007f1b48dc1dc5 in start_thread () from
/lib64/libpthread.so.0<br>
No symbol table info available.<br>
#6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6<br>
No symbol table info available.<br>
<br>
Thread 38 (Thread 0x7f1b05feb700 (LWP 105198)):<br>
#0 0x00007f1b48dc56d5 in <a class="moz-txt-link-abbreviated" href="mailto:pthread_cond_wait@@GLIBC_2.3.2">pthread_cond_wait@@GLIBC_2.3.2</a> ()
from /lib64/libpthread.so.0<br>
No symbol table info available.<br>
#1 0x00007f1b4941b050 in PR_WaitCondVar () from
/lib64/libnspr4.so<br>
No symbol table info available.<br>
#2 0x00007f1b4b70efee in connection_wait_for_new_work ()<br>
No symbol table info available.<br>
#3 0x00007f1b4b710719 in connection_threadmain ()<br>
No symbol table info available.<br>
#4 0x00007f1b494207bb in _pt_root () from
/lib64/libnspr4.so<br>
No symbol table info available.<br>
#5 0x00007f1b48dc1dc5 in start_thread () from
/lib64/libpthread.so.0<br>
No symbol table info available.<br>
#6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6<br>
No symbol table info available.<br>
<br>
Thread 37 (Thread 0x7f1b067ec700 (LWP 105197)):<br>
#0 0x00007f1b48dc56d5 in <a class="moz-txt-link-abbreviated" href="mailto:pthread_cond_wait@@GLIBC_2.3.2">pthread_cond_wait@@GLIBC_2.3.2</a> ()
from /lib64/libpthread.so.0<br>
No symbol table info available.<br>
#1 0x00007f1b4941b050 in PR_WaitCondVar () from
/lib64/libnspr4.so<br>
No symbol table info available.<br>
#2 0x00007f1b4b70efee in connection_wait_for_new_work ()<br>
No symbol table info available.<br>
#3 0x00007f1b4b710719 in connection_threadmain ()<br>
No symbol table info available.<br>
#4 0x00007f1b494207bb in _pt_root () from
/lib64/libnspr4.so<br>
No symbol table info available.<br>
#5 0x00007f1b48dc1dc5 in start_thread () from
/lib64/libpthread.so.0<br>
No symbol table info available.<br>
#6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6<br>
No symbol table info available.<br>
<br>
Thread 36 (Thread 0x7f1b06fed700 (LWP 105196)):<br>
#0 0x00007f1b48dc56d5 in <a class="moz-txt-link-abbreviated" href="mailto:pthread_cond_wait@@GLIBC_2.3.2">pthread_cond_wait@@GLIBC_2.3.2</a> ()
from /lib64/libpthread.so.0<br>
No symbol table info available.<br>
#1 0x00007f1b4941b050 in PR_WaitCondVar () from
/lib64/libnspr4.so<br>
No symbol table info available.<br>
#2 0x00007f1b4b70efee in connection_wait_for_new_work ()<br>
No symbol table info available.<br>
#3 0x00007f1b4b710719 in connection_threadmain ()<br>
No symbol table info available.<br>
#4 0x00007f1b494207bb in _pt_root () from
/lib64/libnspr4.so<br>
No symbol table info available.<br>
#5 0x00007f1b48dc1dc5 in start_thread () from
/lib64/libpthread.so.0<br>
No symbol table info available.<br>
#6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6<br>
No symbol table info available.<br>
<br>
Thread 35 (Thread 0x7f1b077ee700 (LWP 105195)):<br>
#0 0x00007f1b48dc56d5 in <a class="moz-txt-link-abbreviated" href="mailto:pthread_cond_wait@@GLIBC_2.3.2">pthread_cond_wait@@GLIBC_2.3.2</a> ()
from /lib64/libpthread.so.0<br>
No symbol table info available.<br>
#1 0x00007f1b4941b2c3 in PR_EnterMonitor () from
/lib64/libnspr4.so<br>
No symbol table info available.<br>
#2 0x00007f1b3e8c23e6 in dblayer_txn_begin () from
/usr/lib64/dirsrv/plugins/libback-ldbm.so<br>
No symbol table info available.<br>
#3 0x00007f1b3e8fe007 in ldbm_back_modify () from
/usr/lib64/dirsrv/plugins/libback-ldbm.so<br>
No symbol table info available.<br>
#4 0x00007f1b4b22b541 in op_shared_modify () from
/usr/lib64/dirsrv/libslapd.so.0<br>
No symbol table info available.<br>
#5 0x00007f1b4b22c004 in modify_internal_pb () from
/usr/lib64/dirsrv/libslapd.so.0<br>
No symbol table info available.<br>
#6 0x00007f1b3eb48b60 in ipapwd_apply_mods () from
/usr/lib64/dirsrv/plugins/libipa_pwd_extop.so<br>
No symbol table info available.<br>
#7 0x00007f1b3eb4c8e0 in store_new_keys () from
/usr/lib64/dirsrv/plugins/libipa_pwd_extop.so<br>
No symbol table info available.<br>
#8 0x00007f1b3eb4ead0 in ipapwd_extop () from
/usr/lib64/dirsrv/plugins/libipa_pwd_extop.so<br>
No symbol table info available.<br>
#9 0x00007f1b4b23bba2 in plugin_call_exop_plugins () from
/usr/lib64/dirsrv/libslapd.so.0<br>
No symbol table info available.<br>
#10 0x00007f1b4b717efe in do_extended ()<br>
No symbol table info available.<br>
#11 0x00007f1b4b71141a in connection_threadmain ()<br>
No symbol table info available.<br>
#12 0x00007f1b494207bb in _pt_root () from
/lib64/libnspr4.so<br>
No symbol table info available.<br>
#13 0x00007f1b48dc1dc5 in start_thread () from
/lib64/libpthread.so.0<br>
No symbol table info available.<br>
#14 0x00007f1b48aef28d in clone () from /lib64/libc.so.6<br>
No symbol table info available.<br>
<br>
Thread 34 (Thread 0x7f1b07fef700 (LWP 105194)):<br>
#0 0x00007f1b48dc56d5 in <a class="moz-txt-link-abbreviated" href="mailto:pthread_cond_wait@@GLIBC_2.3.2">pthread_cond_wait@@GLIBC_2.3.2</a> ()
from /lib64/libpthread.so.0<br>
No symbol table info available.<br>
#1 0x00007f1b4941b050 in PR_WaitCondVar () from
/lib64/libnspr4.so<br>
No symbol table info available.<br>
#2 0x00007f1b4b70efee in connection_wait_for_new_work ()<br>
No symbol table info available.<br>
#3 0x00007f1b4b710719 in connection_threadmain ()<br>
No symbol table info available.<br>
#4 0x00007f1b494207bb in _pt_root () from
/lib64/libnspr4.so<br>
No symbol table info available.<br>
#5 0x00007f1b48dc1dc5 in start_thread () from
/lib64/libpthread.so.0<br>
No symbol table info available.<br>
#6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6<br>
No symbol table info available.<br>
<br>
Thread 33 (Thread 0x7f1b087f0700 (LWP 105193)):<br>
#0 0x00007f1b48dc56d5 in <a class="moz-txt-link-abbreviated" href="mailto:pthread_cond_wait@@GLIBC_2.3.2">pthread_cond_wait@@GLIBC_2.3.2</a> ()
from /lib64/libpthread.so.0<br>
No symbol table info available.<br>
#1 0x00007f1b4941b050 in PR_WaitCondVar () from
/lib64/libnspr4.so<br>
No symbol table info available.<br>
#2 0x00007f1b4b70efee in connection_wait_for_new_work ()<br>
No symbol table info available.<br>
#3 0x00007f1b4b710719 in connection_threadmain ()<br>
No symbol table info available.<br>
#4 0x00007f1b494207bb in _pt_root () from
/lib64/libnspr4.so<br>
No symbol table info available.<br>
#5 0x00007f1b48dc1dc5 in start_thread () from
/lib64/libpthread.so.0<br>
No symbol table info available.<br>
#6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6<br>
No symbol table info available.<br>
<br>
Thread 32 (Thread 0x7f1b08ff1700 (LWP 105192)):<br>
#0 0x00007f1b48dc56d5 in <a class="moz-txt-link-abbreviated" href="mailto:pthread_cond_wait@@GLIBC_2.3.2">pthread_cond_wait@@GLIBC_2.3.2</a> ()
from /lib64/libpthread.so.0<br>
No symbol table info available.<br>
#1 0x00007f1b4941b050 in PR_WaitCondVar () from
/lib64/libnspr4.so<br>
No symbol table info available.<br>
#2 0x00007f1b4b70efee in connection_wait_for_new_work ()<br>
No symbol table info available.<br>
#3 0x00007f1b4b710719 in connection_threadmain ()<br>
No symbol table info available.<br>
#4 0x00007f1b494207bb in _pt_root () from
/lib64/libnspr4.so<br>
No symbol table info available.<br>
#5 0x00007f1b48dc1dc5 in start_thread () from
/lib64/libpthread.so.0<br>
No symbol table info available.<br>
#6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6<br>
No symbol table info available.<br>
<br>
Thread 31 (Thread 0x7f1b097f2700 (LWP 105191)):<br>
#0 0x00007f1b48dc56d5 in <a class="moz-txt-link-abbreviated" href="mailto:pthread_cond_wait@@GLIBC_2.3.2">pthread_cond_wait@@GLIBC_2.3.2</a> ()
from /lib64/libpthread.so.0<br>
No symbol table info available.<br>
#1 0x00007f1b4941b050 in PR_WaitCondVar () from
/lib64/libnspr4.so<br>
No symbol table info available.<br>
#2 0x00007f1b4b70efee in connection_wait_for_new_work ()<br>
No symbol table info available.<br>
#3 0x00007f1b4b710719 in connection_threadmain ()<br>
No symbol table info available.<br>
#4 0x00007f1b494207bb in _pt_root () from
/lib64/libnspr4.so<br>
No symbol table info available.<br>
#5 0x00007f1b48dc1dc5 in start_thread () from
/lib64/libpthread.so.0<br>
No symbol table info available.<br>
#6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6<br>
No symbol table info available.<br>
<br>
Thread 30 (Thread 0x7f1b09ff3700 (LWP 105190)):<br>
#0 0x00007f1b48dc56d5 in <a class="moz-txt-link-abbreviated" href="mailto:pthread_cond_wait@@GLIBC_2.3.2">pthread_cond_wait@@GLIBC_2.3.2</a> ()
from /lib64/libpthread.so.0<br>
No symbol table info available.<br>
#1 0x00007f1b433212f3 in __db_hybrid_mutex_suspend () from
/lib64/<a moz-do-not-send="true" href="http://libdb-5.3.so">libdb-5.3.so</a><br>
No symbol table info available.<br>
#2 0x00007f1b43320640 in __db_tas_mutex_lock () from
/lib64/<a moz-do-not-send="true" href="http://libdb-5.3.so">libdb-5.3.so</a><br>
No symbol table info available.<br>
#3 0x00007f1b433cad3a in __lock_get_internal () from
/lib64/<a moz-do-not-send="true" href="http://libdb-5.3.so">libdb-5.3.so</a><br>
No symbol table info available.<br>
#4 0x00007f1b433cb820 in __lock_get () from /lib64/<a
moz-do-not-send="true" href="http://libdb-5.3.so">libdb-5.3.so</a><br>
No symbol table info available.<br>
#5 0x00007f1b433f7142 in __db_lget () from /lib64/<a
moz-do-not-send="true" href="http://libdb-5.3.so">libdb-5.3.so</a><br>
No symbol table info available.<br>
#6 0x00007f1b4333e605 in __bam_search () from /lib64/<a
moz-do-not-send="true" href="http://libdb-5.3.so">libdb-5.3.so</a><br>
No symbol table info available.<br>
#7 0x00007f1b43329256 in __bamc_search () from /lib64/<a
moz-do-not-send="true" href="http://libdb-5.3.so">libdb-5.3.so</a><br>
No symbol table info available.<br>
#8 0x00007f1b4332ad0f in __bamc_get () from /lib64/<a
moz-do-not-send="true" href="http://libdb-5.3.so">libdb-5.3.so</a><br>
No symbol table info available.<br>
#9 0x00007f1b433e3ca6 in __dbc_iget () from /lib64/<a
moz-do-not-send="true" href="http://libdb-5.3.so">libdb-5.3.so</a><br>
No symbol table info available.<br>
#10 0x00007f1b433f2b02 in __dbc_get_pp () from /lib64/<a
moz-do-not-send="true" href="http://libdb-5.3.so">libdb-5.3.so</a><br>
No symbol table info available.<br>
#11 0x00007f1b3e8ce680 in idl_new_fetch () from
/usr/lib64/dirsrv/plugins/libback-ldbm.so<br>
No symbol table info available.<br>
#12 0x00007f1b3e8dcb26 in index_read_ext_allids () from
/usr/lib64/dirsrv/plugins/libback-ldbm.so<br>
No symbol table info available.<br>
#13 0x00007f1b3e8c7284 in keys2idl () from
/usr/lib64/dirsrv/plugins/libback-ldbm.so<br>
No symbol table info available.<br>
#14 0x00007f1b3e8c79e3 in ava_candidates.isra.0 () from
/usr/lib64/dirsrv/plugins/libback-ldbm.so<br>
No symbol table info available.<br>
#15 0x00007f1b3e8c7fd2 in filter_candidates_ext () from
/usr/lib64/dirsrv/plugins/libback-ldbm.so<br>
No symbol table info available.<br>
#16 0x00007f1b3e8c9096 in list_candidates () from
/usr/lib64/dirsrv/plugins/libback-ldbm.so<br>
No symbol table info available.<br>
#17 0x00007f1b3e8c7f40 in filter_candidates_ext () from
/usr/lib64/dirsrv/plugins/libback-ldbm.so<br>
No symbol table info available.<br>
#18 0x00007f1b3e8c9096 in list_candidates () from
/usr/lib64/dirsrv/plugins/libback-ldbm.so<br>
No symbol table info available.<br>
#19 0x00007f1b3e8c7f40 in filter_candidates_ext () from
/usr/lib64/dirsrv/plugins/libback-ldbm.so<br>
No symbol table info available.<br>
#20 0x00007f1b3e8c9096 in list_candidates () from
/usr/lib64/dirsrv/plugins/libback-ldbm.so<br>
No symbol table info available.<br>
#21 0x00007f1b3e8c7f40 in filter_candidates_ext () from
/usr/lib64/dirsrv/plugins/libback-ldbm.so<br>
No symbol table info available.<br>
#22 0x00007f1b3e90380c in subtree_candidates () from
/usr/lib64/dirsrv/plugins/libback-ldbm.so<br>
No symbol table info available.<br>
#23 0x00007f1b3e904e7f in ldbm_back_search () from
/usr/lib64/dirsrv/plugins/libback-ldbm.so<br>
No symbol table info available.<br>
#24 0x00007f1b4b2321d6 in op_shared_search () from
/usr/lib64/dirsrv/libslapd.so.0<br>
No symbol table info available.<br>
#25 0x00007f1b4b72333e in do_search ()<br>
No symbol table info available.<br>
#26 0x00007f1b4b7113c3 in connection_threadmain ()<br>
No symbol table info available.<br>
#27 0x00007f1b494207bb in _pt_root () from
/lib64/libnspr4.so<br>
No symbol table info available.<br>
#28 0x00007f1b48dc1dc5 in start_thread () from
/lib64/libpthread.so.0<br>
No symbol table info available.<br>
#29 0x00007f1b48aef28d in clone () from /lib64/libc.so.6<br>
No symbol table info available.<br>
<br>
Thread 29 (Thread 0x7f1b0a7f4700 (LWP 105189)):<br>
#0 0x00007f1b48dc56d5 in <a class="moz-txt-link-abbreviated" href="mailto:pthread_cond_wait@@GLIBC_2.3.2">pthread_cond_wait@@GLIBC_2.3.2</a> ()
from /lib64/libpthread.so.0<br>
No symbol table info available.<br>
#1 0x00007f1b4941b050 in PR_WaitCondVar () from
/lib64/libnspr4.so<br>
No symbol table info available.<br>
#2 0x00007f1b4b70efee in connection_wait_for_new_work ()<br>
No symbol table info available.<br>
#3 0x00007f1b4b710719 in connection_threadmain ()<br>
No symbol table info available.<br>
#4 0x00007f1b494207bb in _pt_root () from
/lib64/libnspr4.so<br>
No symbol table info available.<br>
#5 0x00007f1b48dc1dc5 in start_thread () from
/lib64/libpthread.so.0<br>
No symbol table info available.<br>
#6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6<br>
No symbol table info available.<br>
<br>
Thread 28 (Thread 0x7f1b0aff5700 (LWP 105188)):<br>
#0 0x00007f1b48dc56d5 in <a class="moz-txt-link-abbreviated" href="mailto:pthread_cond_wait@@GLIBC_2.3.2">pthread_cond_wait@@GLIBC_2.3.2</a> ()
from /lib64/libpthread.so.0<br>
No symbol table info available.<br>
#1 0x00007f1b4941b050 in PR_WaitCondVar () from
/lib64/libnspr4.so<br>
No symbol table info available.<br>
#2 0x00007f1b4b70efee in connection_wait_for_new_work ()<br>
No symbol table info available.<br>
#3 0x00007f1b4b710719 in connection_threadmain ()<br>
No symbol table info available.<br>
#4 0x00007f1b494207bb in _pt_root () from
/lib64/libnspr4.so<br>
No symbol table info available.<br>
#5 0x00007f1b48dc1dc5 in start_thread () from
/lib64/libpthread.so.0<br>
No symbol table info available.<br>
#6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6<br>
No symbol table info available.<br>
<br>
Thread 27 (Thread 0x7f1b0b7f6700 (LWP 105187)):<br>
#0 0x00007f1b48dc56d5 in <a class="moz-txt-link-abbreviated" href="mailto:pthread_cond_wait@@GLIBC_2.3.2">pthread_cond_wait@@GLIBC_2.3.2</a> ()
from /lib64/libpthread.so.0<br>
No symbol table info available.<br>
#1 0x00007f1b4941b050 in PR_WaitCondVar () from
/lib64/libnspr4.so<br>
No symbol table info available.<br>
#2 0x00007f1b4b70efee in connection_wait_for_new_work ()<br>
No symbol table info available.<br>
#3 0x00007f1b4b710719 in connection_threadmain ()<br>
No symbol table info available.<br>
#4 0x00007f1b494207bb in _pt_root () from
/lib64/libnspr4.so<br>
No symbol table info available.<br>
#5 0x00007f1b48dc1dc5 in start_thread () from
/lib64/libpthread.so.0<br>
No symbol table info available.<br>
#6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6<br>
No symbol table info available.<br>
<br>
Thread 26 (Thread 0x7f1b0bff7700 (LWP 105186)):<br>
#0 0x00007f1b48dc56d5 in <a class="moz-txt-link-abbreviated" href="mailto:pthread_cond_wait@@GLIBC_2.3.2">pthread_cond_wait@@GLIBC_2.3.2</a> ()
from /lib64/libpthread.so.0<br>
No symbol table info available.<br>
#1 0x00007f1b433212f3 in __db_hybrid_mutex_suspend () from
/lib64/<a moz-do-not-send="true" href="http://libdb-5.3.so">libdb-5.3.so</a><br>
No symbol table info available.<br>
#2 0x00007f1b43320640 in __db_tas_mutex_lock () from
/lib64/<a moz-do-not-send="true" href="http://libdb-5.3.so">libdb-5.3.so</a><br>
No symbol table info available.<br>
#3 0x00007f1b433cad3a in __lock_get_internal () from
/lib64/<a moz-do-not-send="true" href="http://libdb-5.3.so">libdb-5.3.so</a><br>
No symbol table info available.<br>
#4 0x00007f1b433cb820 in __lock_get () from /lib64/<a
moz-do-not-send="true" href="http://libdb-5.3.so">libdb-5.3.so</a><br>
No symbol table info available.<br>
#5 0x00007f1b433f7142 in __db_lget () from /lib64/<a
moz-do-not-send="true" href="http://libdb-5.3.so">libdb-5.3.so</a><br>
No symbol table info available.<br>
#6 0x00007f1b4333e605 in __bam_search () from /lib64/<a
moz-do-not-send="true" href="http://libdb-5.3.so">libdb-5.3.so</a><br>
No symbol table info available.<br>
#7 0x00007f1b43329256 in __bamc_search () from /lib64/<a
moz-do-not-send="true" href="http://libdb-5.3.so">libdb-5.3.so</a><br>
No symbol table info available.<br>
#8 0x00007f1b4332ad0f in __bamc_get () from /lib64/<a
moz-do-not-send="true" href="http://libdb-5.3.so">libdb-5.3.so</a><br>
No symbol table info available.<br>
#9 0x00007f1b433e3ca6 in __dbc_iget () from /lib64/<a
moz-do-not-send="true" href="http://libdb-5.3.so">libdb-5.3.so</a><br>
No symbol table info available.<br>
#10 0x00007f1b433f2b02 in __dbc_get_pp () from /lib64/<a
moz-do-not-send="true" href="http://libdb-5.3.so">libdb-5.3.so</a><br>
No symbol table info available.<br>
#11 0x00007f1b3e8ce680 in idl_new_fetch () from
/usr/lib64/dirsrv/plugins/libback-ldbm.so<br>
No symbol table info available.<br>
#12 0x00007f1b3e8dcb26 in index_read_ext_allids () from
/usr/lib64/dirsrv/plugins/libback-ldbm.so<br>
No symbol table info available.<br>
#13 0x00007f1b3e8c7284 in keys2idl () from
/usr/lib64/dirsrv/plugins/libback-ldbm.so<br>
No symbol table info available.<br>
#14 0x00007f1b3e8c79e3 in ava_candidates.isra.0 () from
/usr/lib64/dirsrv/plugins/libback-ldbm.so<br>
No symbol table info available.<br>
#15 0x00007f1b3e8c7fd2 in filter_candidates_ext () from
/usr/lib64/dirsrv/plugins/libback-ldbm.so<br>
No symbol table info available.<br>
#16 0x00007f1b3e8c9096 in list_candidates () from
/usr/lib64/dirsrv/plugins/libback-ldbm.so<br>
No symbol table info available.<br>
#17 0x00007f1b3e8c7f40 in filter_candidates_ext () from
/usr/lib64/dirsrv/plugins/libback-ldbm.so<br>
No symbol table info available.<br>
#18 0x00007f1b3e90380c in subtree_candidates () from
/usr/lib64/dirsrv/plugins/libback-ldbm.so<br>
No symbol table info available.<br>
#19 0x00007f1b3e904e7f in ldbm_back_search () from
/usr/lib64/dirsrv/plugins/libback-ldbm.so<br>
No symbol table info available.<br>
#20 0x00007f1b4b2321d6 in op_shared_search () from
/usr/lib64/dirsrv/libslapd.so.0<br>
No symbol table info available.<br>
#21 0x00007f1b4b24294e in search_internal_callback_pb ()
from /usr/lib64/dirsrv/libslapd.so.0<br>
No symbol table info available.<br>
#22 0x00007f1b4b242be8 in search_internal_pb () from
/usr/lib64/dirsrv/libslapd.so.0<br>
No symbol table info available.<br>
#23 0x00007f1b4b72149b in ids_sasl_canon_user ()<br>
No symbol table info available.<br>
#24 0x00007f1b491e193b in _sasl_canon_user () from
/lib64/libsasl2.so.3<br>
No symbol table info available.<br>
#25 0x00007f1b491e1c4c in _sasl_canon_user_lookup () from
/lib64/libsasl2.so.3<br>
No symbol table info available.<br>
#26 0x00007f1b430e9fb9 in gssapi_server_mech_step () from
/usr/lib64/sasl2/libgssapiv2.so<br>
No symbol table info available.<br>
#27 0x00007f1b491edb9b in sasl_server_step () from
/lib64/libsasl2.so.3<br>
No symbol table info available.<br>
#28 0x00007f1b4b72283b in ids_sasl_check_bind ()<br>
No symbol table info available.<br>
#29 0x00007f1b4b70a61a in do_bind ()<br>
No symbol table info available.<br>
#30 0x00007f1b4b7113fd in connection_threadmain ()<br>
No symbol table info available.<br>
#31 0x00007f1b494207bb in _pt_root () from
/lib64/libnspr4.so<br>
No symbol table info available.<br>
#32 0x00007f1b48dc1dc5 in start_thread () from
/lib64/libpthread.so.0<br>
No symbol table info available.<br>
#33 0x00007f1b48aef28d in clone () from /lib64/libc.so.6<br>
No symbol table info available.<br>
<br>
Thread 25 (Thread 0x7f1b0c7f8700 (LWP 105185)):<br>
#0 0x00007f1b48dc56d5 in <a class="moz-txt-link-abbreviated" href="mailto:pthread_cond_wait@@GLIBC_2.3.2">pthread_cond_wait@@GLIBC_2.3.2</a> ()
from /lib64/libpthread.so.0<br>
No symbol table info available.<br>
#1 0x00007f1b4941b050 in PR_WaitCondVar () from
/lib64/libnspr4.so<br>
No symbol table info available.<br>
#2 0x00007f1b4b70efee in connection_wait_for_new_work ()<br>
No symbol table info available.<br>
#3 0x00007f1b4b710719 in connection_threadmain ()<br>
No symbol table info available.<br>
#4 0x00007f1b494207bb in _pt_root () from
/lib64/libnspr4.so<br>
No symbol table info available.<br>
#5 0x00007f1b48dc1dc5 in start_thread () from
/lib64/libpthread.so.0<br>
No symbol table info available.<br>
#6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6<br>
No symbol table info available.<br>
<br>
Thread 24 (Thread 0x7f1b0cff9700 (LWP 105184)):<br>
#0 0x00007f1b48dc56d5 in <a class="moz-txt-link-abbreviated" href="mailto:pthread_cond_wait@@GLIBC_2.3.2">pthread_cond_wait@@GLIBC_2.3.2</a> ()
from /lib64/libpthread.so.0<br>
No symbol table info available.<br>
#1 0x00007f1b4941b050 in PR_WaitCondVar () from
/lib64/libnspr4.so<br>
No symbol table info available.<br>
#2 0x00007f1b4b70efee in connection_wait_for_new_work ()<br>
No symbol table info available.<br>
#3 0x00007f1b4b710719 in connection_threadmain ()<br>
No symbol table info available.<br>
#4 0x00007f1b494207bb in _pt_root () from
/lib64/libnspr4.so<br>
No symbol table info available.<br>
#5 0x00007f1b48dc1dc5 in start_thread () from
/lib64/libpthread.so.0<br>
No symbol table info available.<br>
#6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6<br>
No symbol table info available.<br>
<br>
Thread 23 (Thread 0x7f1b0d7fa700 (LWP 105183)):<br>
#0 0x00007f1b48dc56d5 in <a class="moz-txt-link-abbreviated" href="mailto:pthread_cond_wait@@GLIBC_2.3.2">pthread_cond_wait@@GLIBC_2.3.2</a> ()
from /lib64/libpthread.so.0<br>
No symbol table info available.<br>
#1 0x00007f1b4941b050 in PR_WaitCondVar () from
/lib64/libnspr4.so<br>
No symbol table info available.<br>
#2 0x00007f1b4b70efee in connection_wait_for_new_work ()<br>
No symbol table info available.<br>
#3 0x00007f1b4b710719 in connection_threadmain ()<br>
No symbol table info available.<br>
#4 0x00007f1b494207bb in _pt_root () from
/lib64/libnspr4.so<br>
No symbol table info available.<br>
#5 0x00007f1b48dc1dc5 in start_thread () from
/lib64/libpthread.so.0<br>
No symbol table info available.<br>
#6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6<br>
No symbol table info available.<br>
<br>
Thread 22 (Thread 0x7f1b0dffb700 (LWP 105182)):<br>
#0 0x00007f1b48dc56d5 in <a class="moz-txt-link-abbreviated" href="mailto:pthread_cond_wait@@GLIBC_2.3.2">pthread_cond_wait@@GLIBC_2.3.2</a> ()
from /lib64/libpthread.so.0<br>
No symbol table info available.<br>
#1 0x00007f1b4941b050 in PR_WaitCondVar () from
/lib64/libnspr4.so<br>
No symbol table info available.<br>
#2 0x00007f1b4b70efee in connection_wait_for_new_work ()<br>
No symbol table info available.<br>
#3 0x00007f1b4b710719 in connection_threadmain ()<br>
No symbol table info available.<br>
#4 0x00007f1b494207bb in _pt_root () from
/lib64/libnspr4.so<br>
No symbol table info available.<br>
#5 0x00007f1b48dc1dc5 in start_thread () from
/lib64/libpthread.so.0<br>
No symbol table info available.<br>
#6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6<br>
No symbol table info available.<br>
<br>
Thread 21 (Thread 0x7f1b0e7fc700 (LWP 105181)):<br>
#0 0x00007f1b48dc56d5 in <a class="moz-txt-link-abbreviated" href="mailto:pthread_cond_wait@@GLIBC_2.3.2">pthread_cond_wait@@GLIBC_2.3.2</a> ()
from /lib64/libpthread.so.0<br>
No symbol table info available.<br>
#1 0x00007f1b4941b050 in PR_WaitCondVar () from
/lib64/libnspr4.so<br>
No symbol table info available.<br>
#2 0x00007f1b4b70efee in connection_wait_for_new_work ()<br>
No symbol table info available.<br>
#3 0x00007f1b4b710719 in connection_threadmain ()<br>
No symbol table info available.<br>
#4 0x00007f1b494207bb in _pt_root () from
/lib64/libnspr4.so<br>
No symbol table info available.<br>
#5 0x00007f1b48dc1dc5 in start_thread () from
/lib64/libpthread.so.0<br>
No symbol table info available.<br>
#6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6<br>
No symbol table info available.<br>
<br>
Thread 20 (Thread 0x7f1b0effd700 (LWP 105180)):<br>
#0 0x00007f1b48dc56d5 in <a class="moz-txt-link-abbreviated" href="mailto:pthread_cond_wait@@GLIBC_2.3.2">pthread_cond_wait@@GLIBC_2.3.2</a> ()
from /lib64/libpthread.so.0<br>
No symbol table info available.<br>
#1 0x00007f1b433212f3 in __db_hybrid_mutex_suspend () from
/lib64/<a moz-do-not-send="true" href="http://libdb-5.3.so">libdb-5.3.so</a><br>
No symbol table info available.<br>
#2 0x00007f1b43320b37 in __db_tas_mutex_readlock () from
/lib64/<a moz-do-not-send="true" href="http://libdb-5.3.so">libdb-5.3.so</a><br>
No symbol table info available.<br>
#3 0x00007f1b4343ad35 in __memp_fget () from /lib64/<a
moz-do-not-send="true" href="http://libdb-5.3.so">libdb-5.3.so</a><br>
No symbol table info available.<br>
#4 0x00007f1b433e4333 in __dbc_iget () from /lib64/<a
moz-do-not-send="true" href="http://libdb-5.3.so">libdb-5.3.so</a><br>
No symbol table info available.<br>
#5 0x00007f1b433f2b02 in __dbc_get_pp () from /lib64/<a
moz-do-not-send="true" href="http://libdb-5.3.so">libdb-5.3.so</a><br>
No symbol table info available.<br>
#6 0x00007f1b3e8ce822 in idl_new_fetch () from
/usr/lib64/dirsrv/plugins/libback-ldbm.so<br>
No symbol table info available.<br>
#7 0x00007f1b3e8dcb26 in index_read_ext_allids () from
/usr/lib64/dirsrv/plugins/libback-ldbm.so<br>
No symbol table info available.<br>
#8 0x00007f1b3e8c7284 in keys2idl () from
/usr/lib64/dirsrv/plugins/libback-ldbm.so<br>
No symbol table info available.<br>
#9 0x00007f1b3e8c79e3 in ava_candidates.isra.0 () from
/usr/lib64/dirsrv/plugins/libback-ldbm.so<br>
No symbol table info available.<br>
#10 0x00007f1b3e8c7fd2 in filter_candidates_ext () from
/usr/lib64/dirsrv/plugins/libback-ldbm.so<br>
No symbol table info available.<br>
#11 0x00007f1b3e8c9096 in list_candidates () from
/usr/lib64/dirsrv/plugins/libback-ldbm.so<br>
No symbol table info available.<br>
#12 0x00007f1b3e8c7f40 in filter_candidates_ext () from
/usr/lib64/dirsrv/plugins/libback-ldbm.so<br>
No symbol table info available.<br>
#13 0x00007f1b3e8c9096 in list_candidates () from
/usr/lib64/dirsrv/plugins/libback-ldbm.so<br>
No symbol table info available.<br>
#14 0x00007f1b3e8c7f40 in filter_candidates_ext () from
/usr/lib64/dirsrv/plugins/libback-ldbm.so<br>
No symbol table info available.<br>
#15 0x00007f1b3e8c9096 in list_candidates () from
/usr/lib64/dirsrv/plugins/libback-ldbm.so<br>
No symbol table info available.<br>
#16 0x00007f1b3e8c7f40 in filter_candidates_ext () from
/usr/lib64/dirsrv/plugins/libback-ldbm.so<br>
No symbol table info available.<br>
#17 0x00007f1b3e90380c in subtree_candidates () from
/usr/lib64/dirsrv/plugins/libback-ldbm.so<br>
No symbol table info available.<br>
#18 0x00007f1b3e904e7f in ldbm_back_search () from
/usr/lib64/dirsrv/plugins/libback-ldbm.so<br>
No symbol table info available.<br>
#19 0x00007f1b4b2321d6 in op_shared_search () from
/usr/lib64/dirsrv/libslapd.so.0<br>
No symbol table info available.<br>
#20 0x00007f1b4b72333e in do_search ()<br>
No symbol table info available.<br>
#21 0x00007f1b4b7113c3 in connection_threadmain ()<br>
No symbol table info available.<br>
#22 0x00007f1b494207bb in _pt_root () from
/lib64/libnspr4.so<br>
No symbol table info available.<br>
#23 0x00007f1b48dc1dc5 in start_thread () from
/lib64/libpthread.so.0<br>
No symbol table info available.<br>
#24 0x00007f1b48aef28d in clone () from /lib64/libc.so.6<br>
No symbol table info available.<br>
<br>
Thread 19 (Thread 0x7f1b0f7fe700 (LWP 105179)):<br>
#0 0x00007f1b48dc56d5 in <a class="moz-txt-link-abbreviated" href="mailto:pthread_cond_wait@@GLIBC_2.3.2">pthread_cond_wait@@GLIBC_2.3.2</a> ()
from /lib64/libpthread.so.0<br>
No symbol table info available.<br>
#1 0x00007f1b433212f3 in __db_hybrid_mutex_suspend () from
/lib64/<a moz-do-not-send="true" href="http://libdb-5.3.so">libdb-5.3.so</a><br>
No symbol table info available.<br>
#2 0x00007f1b43320640 in __db_tas_mutex_lock () from
/lib64/<a moz-do-not-send="true" href="http://libdb-5.3.so">libdb-5.3.so</a><br>
No symbol table info available.<br>
#3 0x00007f1b433cad3a in __lock_get_internal () from
/lib64/<a moz-do-not-send="true" href="http://libdb-5.3.so">libdb-5.3.so</a><br>
No symbol table info available.<br>
#4 0x00007f1b433cb820 in __lock_get () from /lib64/<a
moz-do-not-send="true" href="http://libdb-5.3.so">libdb-5.3.so</a><br>
No symbol table info available.<br>
#5 0x00007f1b433f7142 in __db_lget () from /lib64/<a
moz-do-not-send="true" href="http://libdb-5.3.so">libdb-5.3.so</a><br>
No symbol table info available.<br>
#6 0x00007f1b4333e605 in __bam_search () from /lib64/<a
moz-do-not-send="true" href="http://libdb-5.3.so">libdb-5.3.so</a><br>
No symbol table info available.<br>
#7 0x00007f1b43329256 in __bamc_search () from /lib64/<a
moz-do-not-send="true" href="http://libdb-5.3.so">libdb-5.3.so</a><br>
No symbol table info available.<br>
#8 0x00007f1b4332ad0f in __bamc_get () from /lib64/<a
moz-do-not-send="true" href="http://libdb-5.3.so">libdb-5.3.so</a><br>
No symbol table info available.<br>
#9 0x00007f1b433e3ca6 in __dbc_iget () from /lib64/<a
moz-do-not-send="true" href="http://libdb-5.3.so">libdb-5.3.so</a><br>
No symbol table info available.<br>
#10 0x00007f1b433f2b02 in __dbc_get_pp () from /lib64/<a
moz-do-not-send="true" href="http://libdb-5.3.so">libdb-5.3.so</a><br>
No symbol table info available.<br>
#11 0x00007f1b3e8ce680 in idl_new_fetch () from
/usr/lib64/dirsrv/plugins/libback-ldbm.so<br>
No symbol table info available.<br>
#12 0x00007f1b3e8dcb26 in index_read_ext_allids () from
/usr/lib64/dirsrv/plugins/libback-ldbm.so<br>
No symbol table info available.<br>
#13 0x00007f1b3e8c7284 in keys2idl () from
/usr/lib64/dirsrv/plugins/libback-ldbm.so<br>
No symbol table info available.<br>
#14 0x00007f1b3e8c79e3 in ava_candidates.isra.0 () from
/usr/lib64/dirsrv/plugins/libback-ldbm.so<br>
No symbol table info available.<br>
#15 0x00007f1b3e8c7fd2 in filter_candidates_ext () from
/usr/lib64/dirsrv/plugins/libback-ldbm.so<br>
No symbol table info available.<br>
#16 0x00007f1b3e8c9096 in list_candidates () from
/usr/lib64/dirsrv/plugins/libback-ldbm.so<br>
No symbol table info available.<br>
#17 0x00007f1b3e8c7f40 in filter_candidates_ext () from
/usr/lib64/dirsrv/plugins/libback-ldbm.so<br>
No symbol table info available.<br>
#18 0x00007f1b3e8c9096 in list_candidates () from
/usr/lib64/dirsrv/plugins/libback-ldbm.so<br>
No symbol table info available.<br>
#19 0x00007f1b3e8c7f40 in filter_candidates_ext () from
/usr/lib64/dirsrv/plugins/libback-ldbm.so<br>
No symbol table info available.<br>
#20 0x00007f1b3e8c9096 in list_candidates () from
/usr/lib64/dirsrv/plugins/libback-ldbm.so<br>
No symbol table info available.<br>
#21 0x00007f1b3e8c7f40 in filter_candidates_ext () from
/usr/lib64/dirsrv/plugins/libback-ldbm.so<br>
No symbol table info available.<br>
#22 0x00007f1b3e90380c in subtree_candidates () from
/usr/lib64/dirsrv/plugins/libback-ldbm.so<br>
No symbol table info available.<br>
#23 0x00007f1b3e904e7f in ldbm_back_search () from
/usr/lib64/dirsrv/plugins/libback-ldbm.so<br>
No symbol table info available.<br>
#24 0x00007f1b4b2321d6 in op_shared_search () from
/usr/lib64/dirsrv/libslapd.so.0<br>
No symbol table info available.<br>
#25 0x00007f1b4b72333e in do_search ()<br>
No symbol table info available.<br>
#26 0x00007f1b4b7113c3 in connection_threadmain ()<br>
No symbol table info available.<br>
#27 0x00007f1b494207bb in _pt_root () from
/lib64/libnspr4.so<br>
No symbol table info available.<br>
#28 0x00007f1b48dc1dc5 in start_thread () from
/lib64/libpthread.so.0<br>
No symbol table info available.<br>
#29 0x00007f1b48aef28d in clone () from /lib64/libc.so.6<br>
No symbol table info available.<br>
<br>
Thread 18 (Thread 0x7f1b0ffff700 (LWP 105178)):<br>
#0 0x00007f1b48dc56d5 in <a class="moz-txt-link-abbreviated" href="mailto:pthread_cond_wait@@GLIBC_2.3.2">pthread_cond_wait@@GLIBC_2.3.2</a> ()
from /lib64/libpthread.so.0<br>
No symbol table info available.<br>
#1 0x00007f1b4941b050 in PR_WaitCondVar () from
/lib64/libnspr4.so<br>
No symbol table info available.<br>
#2 0x00007f1b4b70efee in connection_wait_for_new_work ()<br>
No symbol table info available.<br>
#3 0x00007f1b4b710719 in connection_threadmain ()<br>
No symbol table info available.<br>
#4 0x00007f1b494207bb in _pt_root () from
/lib64/libnspr4.so<br>
No symbol table info available.<br>
#5 0x00007f1b48dc1dc5 in start_thread () from
/lib64/libpthread.so.0<br>
No symbol table info available.<br>
#6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6<br>
No symbol table info available.<br>
<br>
Thread 17 (Thread 0x7f1b14d28700 (LWP 105177)):<br>
#0 0x00007f1b48dc56d5 in <a class="moz-txt-link-abbreviated" href="mailto:pthread_cond_wait@@GLIBC_2.3.2">pthread_cond_wait@@GLIBC_2.3.2</a> ()
from /lib64/libpthread.so.0<br>
No symbol table info available.<br>
#1 0x00007f1b4941b050 in PR_WaitCondVar () from
/lib64/libnspr4.so<br>
No symbol table info available.<br>
#2 0x00007f1b4b70efee in connection_wait_for_new_work ()<br>
No symbol table info available.<br>
#3 0x00007f1b4b710719 in connection_threadmain ()<br>
No symbol table info available.<br>
#4 0x00007f1b494207bb in _pt_root () from
/lib64/libnspr4.so<br>
No symbol table info available.<br>
#5 0x00007f1b48dc1dc5 in start_thread () from
/lib64/libpthread.so.0<br>
No symbol table info available.<br>
#6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6<br>
No symbol table info available.<br>
<br>
Thread 16 (Thread 0x7f1b15529700 (LWP 105176)):<br>
#0 0x00007f1b48dc56d5 in <a class="moz-txt-link-abbreviated" href="mailto:pthread_cond_wait@@GLIBC_2.3.2">pthread_cond_wait@@GLIBC_2.3.2</a> ()
from /lib64/libpthread.so.0<br>
No symbol table info available.<br>
#1 0x00007f1b4941b050 in PR_WaitCondVar () from
/lib64/libnspr4.so<br>
No symbol table info available.<br>
#2 0x00007f1b4b70efee in connection_wait_for_new_work ()<br>
No symbol table info available.<br>
#3 0x00007f1b4b710719 in connection_threadmain ()<br>
No symbol table info available.<br>
#4 0x00007f1b494207bb in _pt_root () from
/lib64/libnspr4.so<br>
No symbol table info available.<br>
#5 0x00007f1b48dc1dc5 in start_thread () from
/lib64/libpthread.so.0<br>
No symbol table info available.<br>
#6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6<br>
No symbol table info available.<br>
<br>
Thread 15 (Thread 0x7f1b15d2a700 (LWP 105175)):<br>
#0 0x00007f1b48dc56d5 in <a class="moz-txt-link-abbreviated" href="mailto:pthread_cond_wait@@GLIBC_2.3.2">pthread_cond_wait@@GLIBC_2.3.2</a> ()
from /lib64/libpthread.so.0<br>
No symbol table info available.<br>
#1 0x00007f1b4941b050 in PR_WaitCondVar () from
/lib64/libnspr4.so<br>
No symbol table info available.<br>
#2 0x00007f1b4b70efee in connection_wait_for_new_work ()<br>
No symbol table info available.<br>
#3 0x00007f1b4b710719 in connection_threadmain ()<br>
No symbol table info available.<br>
#4 0x00007f1b494207bb in _pt_root () from
/lib64/libnspr4.so<br>
No symbol table info available.<br>
#5 0x00007f1b48dc1dc5 in start_thread () from
/lib64/libpthread.so.0<br>
No symbol table info available.<br>
#6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6<br>
No symbol table info available.<br>
<br>
Thread 14 (Thread 0x7f1b1652b700 (LWP 105174)):<br>
#0 0x00007f1b48dc56d5 in <a class="moz-txt-link-abbreviated" href="mailto:pthread_cond_wait@@GLIBC_2.3.2">pthread_cond_wait@@GLIBC_2.3.2</a> ()
from /lib64/libpthread.so.0<br>
No symbol table info available.<br>
#1 0x00007f1b4941b050 in PR_WaitCondVar () from
/lib64/libnspr4.so<br>
No symbol table info available.<br>
#2 0x00007f1b4b70efee in connection_wait_for_new_work ()<br>
No symbol table info available.<br>
#3 0x00007f1b4b710719 in connection_threadmain ()<br>
No symbol table info available.<br>
#4 0x00007f1b494207bb in _pt_root () from
/lib64/libnspr4.so<br>
No symbol table info available.<br>
#5 0x00007f1b48dc1dc5 in start_thread () from
/lib64/libpthread.so.0<br>
No symbol table info available.<br>
#6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6<br>
No symbol table info available.<br>
<br>
Thread 13 (Thread 0x7f1b16d2c700 (LWP 105173)):<br>
#0 0x00007f1b48dc56d5 in <a class="moz-txt-link-abbreviated" href="mailto:pthread_cond_wait@@GLIBC_2.3.2">pthread_cond_wait@@GLIBC_2.3.2</a> ()
from /lib64/libpthread.so.0<br>
No symbol table info available.<br>
#1 0x00007f1b433212f3 in __db_hybrid_mutex_suspend () from
/lib64/<a moz-do-not-send="true" href="http://libdb-5.3.so">libdb-5.3.so</a><br>
No symbol table info available.<br>
#2 0x00007f1b43320640 in __db_tas_mutex_lock () from
/lib64/<a moz-do-not-send="true" href="http://libdb-5.3.so">libdb-5.3.so</a><br>
No symbol table info available.<br>
#3 0x00007f1b433cad3a in __lock_get_internal () from
/lib64/<a moz-do-not-send="true" href="http://libdb-5.3.so">libdb-5.3.so</a><br>
No symbol table info available.<br>
#4 0x00007f1b433cb820 in __lock_get () from /lib64/<a
moz-do-not-send="true" href="http://libdb-5.3.so">libdb-5.3.so</a><br>
No symbol table info available.<br>
#5 0x00007f1b433f7142 in __db_lget () from /lib64/<a
moz-do-not-send="true" href="http://libdb-5.3.so">libdb-5.3.so</a><br>
No symbol table info available.<br>
#6 0x00007f1b4333e605 in __bam_search () from /lib64/<a
moz-do-not-send="true" href="http://libdb-5.3.so">libdb-5.3.so</a><br>
No symbol table info available.<br>
#7 0x00007f1b43329256 in __bamc_search () from /lib64/<a
moz-do-not-send="true" href="http://libdb-5.3.so">libdb-5.3.so</a><br>
No symbol table info available.<br>
#8 0x00007f1b4332ad0f in __bamc_get () from /lib64/<a
moz-do-not-send="true" href="http://libdb-5.3.so">libdb-5.3.so</a><br>
No symbol table info available.<br>
#9 0x00007f1b433e3ca6 in __dbc_iget () from /lib64/<a
moz-do-not-send="true" href="http://libdb-5.3.so">libdb-5.3.so</a><br>
No symbol table info available.<br>
#10 0x00007f1b433f2b02 in __dbc_get_pp () from /lib64/<a
moz-do-not-send="true" href="http://libdb-5.3.so">libdb-5.3.so</a><br>
No symbol table info available.<br>
#11 0x00007f1b3e8ce680 in idl_new_fetch () from
/usr/lib64/dirsrv/plugins/libback-ldbm.so<br>
No symbol table info available.<br>
#12 0x00007f1b3e8dcb26 in index_read_ext_allids () from
/usr/lib64/dirsrv/plugins/libback-ldbm.so<br>
No symbol table info available.<br>
#13 0x00007f1b3e8c7284 in keys2idl () from
/usr/lib64/dirsrv/plugins/libback-ldbm.so<br>
No symbol table info available.<br>
#14 0x00007f1b3e8c79e3 in ava_candidates.isra.0 () from
/usr/lib64/dirsrv/plugins/libback-ldbm.so<br>
No symbol table info available.<br>
#15 0x00007f1b3e8c7fd2 in filter_candidates_ext () from
/usr/lib64/dirsrv/plugins/libback-ldbm.so<br>
No symbol table info available.<br>
#16 0x00007f1b3e8c9096 in list_candidates () from
/usr/lib64/dirsrv/plugins/libback-ldbm.so<br>
No symbol table info available.<br>
#17 0x00007f1b3e8c7f40 in filter_candidates_ext () from
/usr/lib64/dirsrv/plugins/libback-ldbm.so<br>
No symbol table info available.<br>
#18 0x00007f1b3e8c9096 in list_candidates () from
/usr/lib64/dirsrv/plugins/libback-ldbm.so<br>
No symbol table info available.<br>
#19 0x00007f1b3e8c7f40 in filter_candidates_ext () from
/usr/lib64/dirsrv/plugins/libback-ldbm.so<br>
No symbol table info available.<br>
#20 0x00007f1b3e8c9096 in list_candidates () from
/usr/lib64/dirsrv/plugins/libback-ldbm.so<br>
No symbol table info available.<br>
#21 0x00007f1b3e8c7f40 in filter_candidates_ext () from
/usr/lib64/dirsrv/plugins/libback-ldbm.so<br>
No symbol table info available.<br>
#22 0x00007f1b3e90380c in subtree_candidates () from
/usr/lib64/dirsrv/plugins/libback-ldbm.so<br>
No symbol table info available.<br>
#23 0x00007f1b3e904e7f in ldbm_back_search () from
/usr/lib64/dirsrv/plugins/libback-ldbm.so<br>
No symbol table info available.<br>
#24 0x00007f1b4b2321d6 in op_shared_search () from
/usr/lib64/dirsrv/libslapd.so.0<br>
No symbol table info available.<br>
#25 0x00007f1b4b72333e in do_search ()<br>
No symbol table info available.<br>
#26 0x00007f1b4b7113c3 in connection_threadmain ()<br>
No symbol table info available.<br>
#27 0x00007f1b494207bb in _pt_root () from
/lib64/libnspr4.so<br>
No symbol table info available.<br>
#28 0x00007f1b48dc1dc5 in start_thread () from
/lib64/libpthread.so.0<br>
No symbol table info available.<br>
#29 0x00007f1b48aef28d in clone () from /lib64/libc.so.6<br>
No symbol table info available.<br>
<br>
Thread 12 (Thread 0x7f1b1752d700 (LWP 105172)):<br>
#0 0x00007f1b48dc56d5 in <a class="moz-txt-link-abbreviated" href="mailto:pthread_cond_wait@@GLIBC_2.3.2">pthread_cond_wait@@GLIBC_2.3.2</a> ()
from /lib64/libpthread.so.0<br>
No symbol table info available.<br>
#1 0x00007f1b4941b050 in PR_WaitCondVar () from
/lib64/libnspr4.so<br>
No symbol table info available.<br>
#2 0x00007f1b4b70efee in connection_wait_for_new_work ()<br>
No symbol table info available.<br>
#3 0x00007f1b4b710719 in connection_threadmain ()<br>
No symbol table info available.<br>
#4 0x00007f1b494207bb in _pt_root () from
/lib64/libnspr4.so<br>
No symbol table info available.<br>
#5 0x00007f1b48dc1dc5 in start_thread () from
/lib64/libpthread.so.0<br>
No symbol table info available.<br>
#6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6<br>
No symbol table info available.<br>
<br>
Thread 11 (Thread 0x7f1b17d2e700 (LWP 105171)):<br>
#0 0x00007f1b48dc56d5 in <a class="moz-txt-link-abbreviated" href="mailto:pthread_cond_wait@@GLIBC_2.3.2">pthread_cond_wait@@GLIBC_2.3.2</a> ()
from /lib64/libpthread.so.0<br>
No symbol table info available.<br>
#1 0x00007f1b4941b050 in PR_WaitCondVar () from
/lib64/libnspr4.so<br>
No symbol table info available.<br>
#2 0x00007f1b4b70efee in connection_wait_for_new_work ()<br>
No symbol table info available.<br>
#3 0x00007f1b4b710719 in connection_threadmain ()<br>
No symbol table info available.<br>
#4 0x00007f1b494207bb in _pt_root () from
/lib64/libnspr4.so<br>
No symbol table info available.<br>
#5 0x00007f1b48dc1dc5 in start_thread () from
/lib64/libpthread.so.0<br>
No symbol table info available.<br>
#6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6<br>
No symbol table info available.<br>
<br>
Thread 10 (Thread 0x7f1b2509d700 (LWP 105168)):<br>
#0 0x00007f1b48dc5a82 in
<a class="moz-txt-link-abbreviated" href="mailto:pthread_cond_timedwait@@GLIBC_2.3.2">pthread_cond_timedwait@@GLIBC_2.3.2</a> () from
/lib64/libpthread.so.0<br>
No symbol table info available.<br>
#1 0x00007f1b4941ab07 in pt_TimedWait () from
/lib64/libnspr4.so<br>
No symbol table info available.<br>
#2 0x00007f1b4941afce in PR_WaitCondVar () from
/lib64/libnspr4.so<br>
No symbol table info available.<br>
#3 0x00007f1b4b2061e8 in eq_loop () from
/usr/lib64/dirsrv/libslapd.so.0<br>
No symbol table info available.<br>
#4 0x00007f1b494207bb in _pt_root () from
/lib64/libnspr4.so<br>
No symbol table info available.<br>
#5 0x00007f1b48dc1dc5 in start_thread () from
/lib64/libpthread.so.0<br>
No symbol table info available.<br>
#6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6<br>
No symbol table info available.<br>
<br>
Thread 9 (Thread 0x7f1b2589e700 (LWP 105167)):<br>
#0 0x00007f1b48dc5a82 in
<a class="moz-txt-link-abbreviated" href="mailto:pthread_cond_timedwait@@GLIBC_2.3.2">pthread_cond_timedwait@@GLIBC_2.3.2</a> () from
/lib64/libpthread.so.0<br>
No symbol table info available.<br>
#1 0x00007f1b4941ab07 in pt_TimedWait () from
/lib64/libnspr4.so<br>
No symbol table info available.<br>
#2 0x00007f1b4941afce in PR_WaitCondVar () from
/lib64/libnspr4.so<br>
No symbol table info available.<br>
#3 0x00007f1b4b719d03 in housecleaning ()<br>
No symbol table info available.<br>
#4 0x00007f1b494207bb in _pt_root () from
/lib64/libnspr4.so<br>
No symbol table info available.<br>
#5 0x00007f1b48dc1dc5 in start_thread () from
/lib64/libpthread.so.0<br>
No symbol table info available.<br>
#6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6<br>
No symbol table info available.<br>
<br>
Thread 8 (Thread 0x7f1b2609f700 (LWP 105166)):<br>
#0 0x00007f1b48dc56d5 in <a class="moz-txt-link-abbreviated" href="mailto:pthread_cond_wait@@GLIBC_2.3.2">pthread_cond_wait@@GLIBC_2.3.2</a> ()
from /lib64/libpthread.so.0<br>
No symbol table info available.<br>
#1 0x00007f1b4941b050 in PR_WaitCondVar () from
/lib64/libnspr4.so<br>
No symbol table info available.<br>
#2 0x00007f1b4b260198 in slapi_wait_condvar () from
/usr/lib64/dirsrv/libslapd.so.0<br>
No symbol table info available.<br>
#3 0x00007f1b3cf5deed in roles_cache_wait_on_change () from
/usr/lib64/dirsrv/plugins/libroles-plugin.so<br>
No symbol table info available.<br>
#4 0x00007f1b494207bb in _pt_root () from
/lib64/libnspr4.so<br>
No symbol table info available.<br>
#5 0x00007f1b48dc1dc5 in start_thread () from
/lib64/libpthread.so.0<br>
No symbol table info available.<br>
#6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6<br>
No symbol table info available.<br>
<br>
Thread 7 (Thread 0x7f1b268a0700 (LWP 105165)):<br>
#0 0x00007f1b48dc56d5 in <a class="moz-txt-link-abbreviated" href="mailto:pthread_cond_wait@@GLIBC_2.3.2">pthread_cond_wait@@GLIBC_2.3.2</a> ()
from /lib64/libpthread.so.0<br>
No symbol table info available.<br>
#1 0x00007f1b4941b050 in PR_WaitCondVar () from
/lib64/libnspr4.so<br>
No symbol table info available.<br>
#2 0x00007f1b4b260198 in slapi_wait_condvar () from
/usr/lib64/dirsrv/libslapd.so.0<br>
No symbol table info available.<br>
#3 0x00007f1b3cf5deed in roles_cache_wait_on_change () from
/usr/lib64/dirsrv/plugins/libroles-plugin.so<br>
No symbol table info available.<br>
#4 0x00007f1b494207bb in _pt_root () from
/lib64/libnspr4.so<br>
No symbol table info available.<br>
#5 0x00007f1b48dc1dc5 in start_thread () from
/lib64/libpthread.so.0<br>
No symbol table info available.<br>
#6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6<br>
No symbol table info available.<br>
<br>
Thread 6 (Thread 0x7f1b270a1700 (LWP 105164)):<br>
#0 0x00007f1b48dc56d5 in <a class="moz-txt-link-abbreviated" href="mailto:pthread_cond_wait@@GLIBC_2.3.2">pthread_cond_wait@@GLIBC_2.3.2</a> ()
from /lib64/libpthread.so.0<br>
No symbol table info available.<br>
#1 0x00007f1b4941b050 in PR_WaitCondVar () from
/lib64/libnspr4.so<br>
No symbol table info available.<br>
#2 0x00007f1b4b260198 in slapi_wait_condvar () from
/usr/lib64/dirsrv/libslapd.so.0<br>
No symbol table info available.<br>
#3 0x00007f1b3cf5deed in roles_cache_wait_on_change () from
/usr/lib64/dirsrv/plugins/libroles-plugin.so<br>
No symbol table info available.<br>
#4 0x00007f1b494207bb in _pt_root () from
/lib64/libnspr4.so<br>
No symbol table info available.<br>
#5 0x00007f1b48dc1dc5 in start_thread () from
/lib64/libpthread.so.0<br>
No symbol table info available.<br>
#6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6<br>
No symbol table info available.<br>
<br>
Thread 5 (Thread 0x7f1b278a2700 (LWP 105163)):<br>
#0 0x00007f1b48dc56d5 in <a class="moz-txt-link-abbreviated" href="mailto:pthread_cond_wait@@GLIBC_2.3.2">pthread_cond_wait@@GLIBC_2.3.2</a> ()
from /lib64/libpthread.so.0<br>
No symbol table info available.<br>
#1 0x00007f1b4941b050 in PR_WaitCondVar () from
/lib64/libnspr4.so<br>
No symbol table info available.<br>
#2 0x00007f1b4b260198 in slapi_wait_condvar () from
/usr/lib64/dirsrv/libslapd.so.0<br>
No symbol table info available.<br>
#3 0x00007f1b415f262e in cos_cache_wait_on_change () from
/usr/lib64/dirsrv/plugins/libcos-plugin.so<br>
No symbol table info available.<br>
#4 0x00007f1b494207bb in _pt_root () from
/lib64/libnspr4.so<br>
No symbol table info available.<br>
#5 0x00007f1b48dc1dc5 in start_thread () from
/lib64/libpthread.so.0<br>
No symbol table info available.<br>
#6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6<br>
No symbol table info available.<br>
<br>
Thread 4 (Thread 0x7f1b282ac700 (LWP 105162)):<br>
#0 0x00007f1b48ae69b3 in select () from /lib64/libc.so.6<br>
No symbol table info available.<br>
#1 0x00007f1b4b2710e9 in DS_Sleep () from
/usr/lib64/dirsrv/libslapd.so.0<br>
No symbol table info available.<br>
#2 0x00007f1b3e8ba707 in perf_threadmain () from
/usr/lib64/dirsrv/plugins/libback-ldbm.so<br>
No symbol table info available.<br>
#3 0x00007f1b494207bb in _pt_root () from
/lib64/libnspr4.so<br>
No symbol table info available.<br>
#4 0x00007f1b48dc1dc5 in start_thread () from
/lib64/libpthread.so.0<br>
No symbol table info available.<br>
#5 0x00007f1b48aef28d in clone () from /lib64/libc.so.6<br>
No symbol table info available.<br>
<br>
Thread 3 (Thread 0x7f1b28aad700 (LWP 105161)):<br>
#0 0x00007f1b48ae69b3 in select () from /lib64/libc.so.6<br>
No symbol table info available.<br>
#1 0x00007f1b4b2710e9 in DS_Sleep () from
/usr/lib64/dirsrv/libslapd.so.0<br>
No symbol table info available.<br>
#2 0x00007f1b3e8bfb7f in trickle_threadmain () from
/usr/lib64/dirsrv/plugins/libback-ldbm.so<br>
No symbol table info available.<br>
#3 0x00007f1b494207bb in _pt_root () from
/lib64/libnspr4.so<br>
No symbol table info available.<br>
#4 0x00007f1b48dc1dc5 in start_thread () from
/lib64/libpthread.so.0<br>
No symbol table info available.<br>
#5 0x00007f1b48aef28d in clone () from /lib64/libc.so.6<br>
No symbol table info available.<br>
<br>
Thread 2 (Thread 0x7f1b292ae700 (LWP 105160)):<br>
#0 0x00007f1b48ae69b3 in select () from /lib64/libc.so.6<br>
No symbol table info available.<br>
#1 0x00007f1b4344dbad in __os_yield () from /lib64/<a
moz-do-not-send="true" href="http://libdb-5.3.so">libdb-5.3.so</a><br>
No symbol table info available.<br>
#2 0x00007f1b434492b3 in __memp_sync_int () from /lib64/<a
moz-do-not-send="true" href="http://libdb-5.3.so">libdb-5.3.so</a><br>
No symbol table info available.<br>
#3 0x00007f1b43459752 in __txn_checkpoint () from /lib64/<a
moz-do-not-send="true" href="http://libdb-5.3.so">libdb-5.3.so</a><br>
No symbol table info available.<br>
#4 0x00007f1b43459b74 in __txn_checkpoint_pp () from
/lib64/<a moz-do-not-send="true" href="http://libdb-5.3.so">libdb-5.3.so</a><br>
No symbol table info available.<br>
#5 0x00007f1b3e8c3a87 in checkpoint_threadmain () from
/usr/lib64/dirsrv/plugins/libback-ldbm.so<br>
No symbol table info available.<br>
#6 0x00007f1b494207bb in _pt_root () from
/lib64/libnspr4.so<br>
No symbol table info available.<br>
#7 0x00007f1b48dc1dc5 in start_thread () from
/lib64/libpthread.so.0<br>
No symbol table info available.<br>
#8 0x00007f1b48aef28d in clone () from /lib64/libc.so.6<br>
No symbol table info available.<br>
<br>
Thread 1 (Thread 0x7f1b29aaf700 (LWP 105159)):<br>
#0 0x00007f1b48ae69b3 in select () from /lib64/libc.so.6<br>
No symbol table info available.<br>
#1 0x00007f1b4b2710e9 in DS_Sleep () from
/usr/lib64/dirsrv/libslapd.so.0<br>
No symbol table info available.<br>
#2 0x00007f1b3e8bf907 in deadlock_threadmain () from
/usr/lib64/dirsrv/plugins/libback-ldbm.so<br>
No symbol table info available.<br>
#3 0x00007f1b494207bb in _pt_root () from
/lib64/libnspr4.so<br>
No symbol table info available.<br>
#4 0x00007f1b48dc1dc5 in start_thread () from
/lib64/libpthread.so.0<br>
No symbol table info available.<br>
#5 0x00007f1b48aef28d in clone () from /lib64/libc.so.6<br>
No symbol table info available.<br>
<br>
<br>
<br>
</div>
Thanks,<br>
<br>
</div>
Rakesh<br>
<div>
<div><br>
</div>
</div>
</div>
<div class="gmail_extra"><br>
<div class="gmail_quote">On Mon, Sep 5, 2016 at 4:48 PM, thierry
bordaz <span dir="ltr"><<a moz-do-not-send="true"
href="mailto:tbordaz@redhat.com" target="_blank">tbordaz@redhat.com</a>></span>
wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0
.8ex;border-left:1px #ccc solid;padding-left:1ex">
<div text="#000000" bgcolor="#FFFFFF"> <br>
<br>
<div>On 09/05/2016 12:05 PM, Rakesh Rajasekharan wrote:<br>
</div>
<blockquote type="cite">
<div dir="ltr">
<div>
<div>
<div>
<div>
<div>Hi Thierry,<br>
<br>
<br>
</div>
<div>I was getting the hang issue while
running ipa-client-install simultaneously on
few clients..<br>
</div>
<div>However, today, I am not able to
replicate that. <br>
</div>
<div><br>
</div>
<div>I could not get a gdb . But i will try
getting that the next time I face this
issue.<br>
<br>
</div>
The CPU does not stay high.. it just
momentarily touches a high value and then
drops down to around 2-7%<br>
<br>
</div>
One question I have is , is it ok to set it
nsslapd-threadnumber to a very high value .<br>
</div>
I have around 4000 clients and with
nsslapd-maxthreadsperconn set to 5..So, can I set
nsslapd-threadnumber to around 25000.<br>
</div>
</div>
</div>
</blockquote>
<br>
Hello,<br>
<br>
I know some users running in production with several
hunderds of threads (>600) and this without problem.<br>
<br>
I do not recall having suggested to increase that number
and for what reason.<br>
Usually 30 workers is a good enough value. It can create
bootleneck if for some reason each operation is very long
to satisfy and exhaust the number of workers. You can
monitor the work queue:<br>
<blockquote><tt>ldapsearch -D "cn=directory manager" -w
xxx -LLL -b "cn=monitor" -s base opsinitiated
opscompleted</tt><br>
</blockquote>
<br>
If opscompleted-opsinitiated remains close to
threadnumber, then yes it would be valuable to increase
it.<br>
<br>
The computation #client * #async_op_per_client sound an
overkill. Even if all clients send at the exact same time
all their requests, it is very likely that some common
resource (db page, log, allocator...) will serialize them.
If you monitor a need to increase the work, you would for
example set it to 50, then monitor, then set it to 100,
then monitor... until you find a good enough value.<br>
Note the increasing the #thread, increases the memory
footprint that will reduce the efficiency of file system
cache and can increase the response time.<br>
<br>
<br>
best regards<br>
thierry <br>
<br>
<blockquote type="cite">
<div dir="ltr">
<div><br>
</div>
Thanks<br>
</div>
<div class="gmail_extra"><br>
<div class="gmail_quote">On Mon, Sep 5, 2016 at 1:03
PM, thierry bordaz <span dir="ltr"><<a
moz-do-not-send="true"
href="mailto:tbordaz@redhat.com" target="_blank"><a class="moz-txt-link-abbreviated" href="mailto:tbordaz@redhat.com">tbordaz@redhat.com</a></a>></span>
wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0
.8ex;border-left:1px #ccc solid;padding-left:1ex">
<div text="#000000" bgcolor="#FFFFFF"> <br>
Hi Rakesh,<br>
<br>
Were you able to get a pstack or full stack with
gdb (<a moz-do-not-send="true"
href="http://www.port389.org/docs/"
target="_blank">http://www.port389.org/docs/</a>3<wbr>89ds/FAQ/faq.html#debugging-cr<wbr>ashes)
when the server hangs ?<br>
<br>
If it happens with 500 threads as well as with
30, using 30 threads is a better choice to debug
this issue.<br>
I will try to reproduce using 150 parallel 'ipa
user-find p-testipa' commands<br>
<br>
Something I am unsure is if the CPU consumption
stays high (you mentioned 340% CPU usage) as
long as the hang happens or if after a suddent
shot up to 340% (that marks the beginning of the
hang) it drops and stay hanging ?<br>
<br>
thanks<br>
thierry<br>
<br>
<div>On 09/04/2016 08:40 PM, Rakesh Rajasekharan
wrote:<br>
</div>
<blockquote type="cite">
<div dir="ltr">
<div>starce on the slapd process actually
had this in the output.. <br>
FUTEX_WAIT_PRIVATE<br>
<br>
</div>
and checking for the number of threads slapd
had.. there were 5015 threads<br>
<div><br>
ps -efL|grep slapd|wc -l<br>
5015<br>
<br>
strace on most of the threads gave this
output <br>
<br>
strace -p 67411<br>
Process 67411 attached<br>
futex(0x7f3f0226b41c, FUTEX_WAIT_PRIVATE,
1, NULL) = -1 EAGAIN (Resource temporarily
unavailable)<br>
futex(0x7f3f0226b41c, FUTEX_WAIT_PRIVATE,
2, NULL^CProcess 67411 detached<br>
<br>
<br>
<br>
<br>
</div>
</div>
<div class="gmail_extra"><br>
<div class="gmail_quote">On Sun, Sep 4, 2016
at 5:34 PM, Rakesh Rajasekharan <span
dir="ltr"><<a moz-do-not-send="true"
href="mailto:rakesh.rajasekharan@gmail.com" target="_blank">rakesh.rajasekharan@gmail.com</a><wbr>></span>
wrote:<br>
<blockquote class="gmail_quote"
style="margin:0 0 0 .8ex;border-left:1px
#ccc solid;padding-left:1ex">
<div dir="ltr">
<div>
<div>
<div>
<div>
<div>I have again got the
issue of IPA hanging.. The
issue came up when i tried
to run ipa-client-isntall on
142 clients simultaneously<br>
<br>
<br>
</div>
<div>None of the IPA commands
are responding, and I see
this error<br>
<br>
ipa user-find p-testipa<br>
ipa: ERROR: Insufficient
access: SASL(-1): generic
failure: GSSAPI Error:
Unspecified GSS failure.
Minor code may provide more
information (KDC returned
error string: PROCESS_TGS)<br>
<br>
KRB5_TRACE=/dev/stdout
kinit admin<br>
[41178] 1472984115.233214:
Getting initial credentials
for <a
moz-do-not-send="true"
href="mailto:admin@XYZ.COM"
target="_blank"><a class="moz-txt-link-abbreviated" href="mailto:admin@XYZ.COM">admin@XYZ.COM</a></a><br>
[41178] 1472984115.235257:
Sending request (167 bytes)
to <a
moz-do-not-send="true"
href="http://XYZ.COM"
target="_blank">XYZ.COM</a><br>
[41178] 1472984115.235419:
Initiating TCP connection to
stream <a
moz-do-not-send="true"
href="http://10.1.3.36:88"
target="_blank">10.1.3.36:88</a><br>
[41178] 1472984115.235685:
Sending TCP request to
stream <a
moz-do-not-send="true"
href="http://10.1.3.36:88"
target="_blank">10.1.3.36:88</a><br>
[41178] 1472984120.238914:
Received answer (174 bytes)
from stream <a
moz-do-not-send="true"
href="http://10.1.3.36:88"
target="_blank">10.1.3.36:88</a><br>
[41178] 1472984120.238925:
Terminating TCP connection
to stream <a
moz-do-not-send="true"
href="http://10.1.3.36:88"
target="_blank">10.1.3.36:88</a><br>
[41178] 1472984120.238993:
Response was from master KDC<br>
</div>
<div>[41<br>
<br>
<br>
</div>
<div>Running an ldapsearch to
see the db.. does not give
any results and just hangs
there<br>
<br>
ldapsearch -x -D
'cn=Directory Manager' -W -s
one -b
'cn=kerberos,dc=xyz,dc=com'<br>
Enter LDAP Password:<br>
<br>
</div>
<div>even an ldapsearch -x
does not respond<br>
</div>
<div>At this point, am sure
that slapd is the one
causing issues<br>
</div>
<div><br>
</div>
<div>Running an strace against
the hung slapd itself seems
to get stuck does not
proceed after saying
"attaching to process"<br>
<br>
</div>
<div>From some others posts I
read Thierry suggesting to
increase the
nsslapd-threadnumber value<br>
<br>
</div>
<div>It was set to 30, I think
that might be too low.<br>
<br>
</div>
<div>I have raised it to 500<br>
</div>
<br>
</div>
<div>Now after restarting the
service .. ldapsearch starts
responding.<br>
</div>
But running the test to add a
sudden high number of clients
again left ns-slapd to hung
state<br>
<br>
</div>
When i attempted adding the
clients.. the ns-slapd cpu usage
shot up to 340% and after that
ns-slapd stopped responding<br>
<br>
</div>
So now, atleast I know what might be
causing the issue and I can now
easily reproduce it.<br>
<br>
</div>
<div>Is there a way I can make
ns-slapd handle a sudden bump in
incoming request for
ipa-client-install<br>
<br>
</div>
<div>Thanks<span><font color="#888888"><br>
</font></span></div>
<span><font color="#888888">
<div>Rakesh<br>
</div>
<br>
<div>
<div>
<div>
<div>
<div>
<div><br>
</div>
<div><br>
<br>
<br>
</div>
</div>
</div>
</div>
</div>
</div>
</font></span></div>
<div class="gmail_extra"><br>
<div class="gmail_quote">
<div>
<div>On Mon, Aug 29, 2016 at 11:18
PM, Rich Megginson <span
dir="ltr"><<a
moz-do-not-send="true"
href="mailto:rmeggins@redhat.com"
target="_blank"><a class="moz-txt-link-abbreviated" href="mailto:rmeggins@redhat.com">rmeggins@redhat.com</a></a>></span>
wrote:<br>
</div>
</div>
<blockquote class="gmail_quote"
style="margin:0 0 0
.8ex;border-left:1px #ccc
solid;padding-left:1ex">
<div>
<div>
<div bgcolor="#FFFFFF"
text="#000000"><span>
<div>On 08/29/2016 10:53
AM, Rakesh Rajasekharan
wrote:<br>
</div>
<blockquote type="cite">
<div dir="ltr">
<div>
<div>Hi Thierry,<br>
<br>
My machine has
30GB RAM ..and
389-ds version is
1.3.4<br>
<br>
</div>
ldapsearch shows the
values for
nsslapd-cachememsize
updated to 200MB.<br>
<br>
ldapsearch -LLL -o
ldif-wrap=no -D
"cn=directory
manager" -w
'mypassword' -b
'cn=userRoot,cn=ldbm
database,cn=plugins,cn=config'<wbr>|grep nsslapd-cachememsize<br>
nsslapd-cachememsize:
209715200<br>
<br>
<br>
So, it seems to have
updated though
seeing that
warning(WARNING:
ipaca: entry cache
size 10485760B is
less than db size
11599872B) in the
log confuses me a
bit.<br>
<br>
</div>
<div>Thers one more
entry that I found
from the ldapsearch
to be bit low<br>
<br>
nsslapd-dncachememsize:
10485760<br>
maxdncachesize:
10485760<br>
<br>
</div>
<div>Should I update
these as well to a
higher value<br>
<br>
</div>
<div>At the time when
the issue happened,
the memory usage as
well as the overall
load of the system
was very low . <br>
I will try
reproducing the
issue atleast in my
QA env..probably by
trying to mock
simultaneous
parallel logins to a
large number of
hosts <br>
</div>
</div>
</blockquote>
<br>
</span> To monitor your
cache sizes, please use the
dbmon.sh tool provided with
your distro. If that is not
available with your
particular distro, see <a
moz-do-not-send="true"
href="https://github.com/richm/scripts/wiki/dbmon.sh"
target="_blank"><a class="moz-txt-link-freetext" href="https://github.com/richm/scrip">https://github.com/richm/scrip</a><wbr>ts/wiki/dbmon.sh</a>
<div>
<div><br>
<br>
<blockquote type="cite">
<div dir="ltr">
<div><br>
<br>
</div>
<div>thanks<br>
</div>
<div>Rakesh<br>
</div>
<div><br>
</div>
<div><br>
</div>
<br>
</div>
<div
class="gmail_extra"><br>
<div
class="gmail_quote">On
Mon, Aug 29, 2016
at 8:16 PM,
thierry bordaz <span
dir="ltr"><<a
moz-do-not-send="true" href="mailto:tbordaz@redhat.com" target="_blank"><a class="moz-txt-link-abbreviated" href="mailto:tbordaz@redhat.com">tbordaz@redhat.com</a></a>></span>
wrote:<br>
<blockquote
class="gmail_quote"
style="margin:0
0 0
.8ex;border-left:1px
#ccc
solid;padding-left:1ex">
<div
text="#000000"
bgcolor="#FFFFFF"> Hi Rakesh,<br>
<br>
Those tuning
may depend on
the memory
available on
your machine.
<br>
nsslapd-cachememsize
allows the
entry cache to
consume up to
200Mb but its
memory
footprint is
known to go
above. <br>
200Mb both
looks pretty
good to me.
How large is
your machine ?
What is your
version of
389-ds ?<br>
<br>
Those warnings
do not change
your settings.
It just raise
that entry
cache of
'ipaca' and
'retrocl' are
small but it
is fine. The
size of the
entry cache is
important
mostly in
userRoot.<br>
You may double
check the
actual values,
after restart,
with
ldapsearch on
'cn=userRoot,cn=ldbm
database,cn=plugins,cn=config'
and
'cn=config,cn=ldbm
database,cn=plugins,cn=config'<wbr>.<br>
<br>
A step is to
know what will
be response
time of DS to
know if it is
responsible of
the hang or
not.<br>
The logs and
possibly
pstack during
those
intermittent
hangs will
help to
determine
that.<br>
<br>
regards<span><font
color="#888888"><br>
thierry</font></span>
<div>
<div><br>
<br>
<br>
<br>
<br>
<div>On
08/29/2016
04:25 PM,
Rakesh
Rajasekharan
wrote:<br>
</div>
<blockquote
type="cite">
<div dir="ltr">
<div>
<div>
<div>
<div>
<div>
<div>I tried
increasing the
nsslapd-dbcachesize
and
nsslapd-cachememsize
in my QA envs
to 200MB.<br>
<br>
</div>
However, in my
log files, I
still see this
message<br>
[29/Aug/2016:04:34:37
+0000] -
WARNING:
ipaca: entry
cache size
10485760B is
less than db
size
11599872B; We
recommend to
increase the
entry cache
size
nsslapd-cachememsize.<br>
[29/Aug/2016:04:34:37
+0000] -
WARNING:
changelog:
entry cache
size 2097152B
is less than
db size
441647104B; We
recommend to
increase the
entry cache
size
nsslapd-cachememsize.<br>
<br>
</div>
these are my
ldif files
that i used to
modify the
values<br>
modify entry
cache size<br>
cat
modify-cache-mem-size.ldif<br>
dn:
cn=userRoot,cn=ldbm
database,cn=plugins,cn=config<br>
changetype:
modify<br>
replace:
nsslapd-cachememsize<br>
nsslapd-cachememsize:
209715200<br>
<br>
modify db
cache size<br>
cat
modfy-db-cache-size.ldif<br>
dn:
cn=config,cn=ldbm
database,cn=plugins,cn=config<br>
changetype:
modify<br>
replace:
nsslapd-dbcachesize<br>
nsslapd-dbcachesize:
209715200<br>
<br>
</div>
After
modifying , i
restarted IPA
services<br>
<br>
</div>
Is there
anything else
that I need
to take care
of as the logs
suggest its
still not
getting the
updated values<br>
<br>
</div>
Thanks<br>
</div>
Rakesh<br>
</div>
<div
class="gmail_extra"><br>
<div
class="gmail_quote">On
Mon, Aug 29,
2016 at 6:07
PM, Rakesh
Rajasekharan <span
dir="ltr"><<a
moz-do-not-send="true" href="mailto:rakesh.rajasekharan@gmail.com"
target="_blank"><a class="moz-txt-link-abbreviated" href="mailto:rakesh.rajasekharan@gmail.com">rakesh.rajasekharan@gmail.com</a></a><wbr>></span>
wrote:<br>
<blockquote
class="gmail_quote"
style="margin:0
0 0
.8ex;border-left:1px
#ccc
solid;padding-left:1ex">
<div dir="ltr">
<div>
<div>
<div>
<div>
<div>
<div>Hi
Thierry,<br>
<br>
</div>
Coz of the
issues we had
to revert back
to earlier
running
openldap in
production.<br>
<br>
</div>
I have now
done a few TCP
related
changes in
sysctl.conf
and have also
increased the
nsslapd-dbcachesize
and
nsslapd-cachememsize
to 200MB<br>
<br>
</div>
I will again
start
migrating
hosts back to
IPA and see if
I face the
earlier issue.<br>
<br>
</div>
I will update
back once I
have something<br>
<br>
<br>
</div>
Thanks,<br>
</div>
Rakesh<br>
<div>
<div>
<div>
<div><br>
<br>
</div>
</div>
</div>
</div>
</div>
<div>
<div>
<div
class="gmail_extra"><br>
<div
class="gmail_quote">On
Thu, Aug 25,
2016 at 2:17
PM, thierry
bordaz <span
dir="ltr"><<a
moz-do-not-send="true" href="mailto:tbordaz@redhat.com" target="_blank"><a class="moz-txt-link-abbreviated" href="mailto:tbordaz@redhat.com">tbordaz@redhat.com</a></a>></span>
wrote:<br>
<blockquote
class="gmail_quote"
style="margin:0
0 0
.8ex;border-left:1px
#ccc
solid;padding-left:1ex">
<div
text="#000000"
bgcolor="#FFFFFF">
<div>
<div> <br>
<br>
<div>On
08/25/2016
10:15 AM,
Rakesh
Rajasekharan
wrote:<br>
</div>
<blockquote
type="cite">
<div dir="ltr">
<div>All of
the
troubleshooting
seems fine.<br>
<br>
<br>
</div>
<div>However,
Running <a
moz-do-not-send="true"
href="http://libconv.pl" target="_blank">libconv.pl</a> gives me this
output<br>
<br>
-----
Recommendations
-----<br>
<br>
1. You have
unindexed
components,
this can be
caused from a
search on an
unindexed
attribute, or
your returned
results
exceeded the
allidsthreshold.
Unindexed
components are
not
recommended.
To refuse
unindexed
searches,
switch
'nsslapd-require-index'
to 'on' under
your database
entry (e.g.
cn=UserRoot,cn=ldbm
database,cn=plugins,cn=config)<wbr>.<br>
<br>
2. You have
a significant
difference
between binds
and unbinds.
You may want
to investigate
this
difference.<br>
<br>
</div>
<div><br>
</div>
<div>I feel,
this could be
a pointer to
things going
slow.. and IPA
hanging. I
think i now
have something
that I can try
and nail down
this issue.<br>
<br>
On a sidenote,
I was earlier
running
openldap and
migrated over
to Freeipa, <br>
<br>
</div>
<div>Thanks<br>
</div>
<div>Rakesh<br>
</div>
<div><br>
<br>
</div>
</div>
<div
class="gmail_extra"><br>
<div
class="gmail_quote">On
Wed, Aug 24,
2016 at 12:38
PM, Petr
Spacek <span
dir="ltr"><<a
moz-do-not-send="true" href="mailto:pspacek@redhat.com" target="_blank"><a class="moz-txt-link-abbreviated" href="mailto:pspacek@redhat.com">pspacek@redhat.com</a></a>></span>
wrote:<br>
<blockquote
class="gmail_quote"
style="margin:0
0 0
.8ex;border-left:1px
#ccc
solid;padding-left:1ex"><span>On
23.8.2016
18:44, Rakesh
Rajasekharan
wrote:<br>
> I think
thers
something
seriously
wrong with my
system<br>
><br>
> not able
to run any
IPA commands<br>
><br>
> klist<br>
> Ticket
cache:
KEYRING:persistent:0:0<br>
> Default
principal: <a
moz-do-not-send="true" href="mailto:admin@XYZ.COM" target="_blank"><a class="moz-txt-link-abbreviated" href="mailto:admin@XYZ.COM">admin@XYZ.COM</a></a><br>
><br>
> Valid
starting
Expires
Service
principal<br>
>
2016-08-23T16:26:36
2016-08-24T16:26:22
krbtgt/<a
moz-do-not-send="true"
href="mailto:XYZ.COM@XYZ.COM" target="_blank"><a class="moz-txt-link-abbreviated" href="mailto:XYZ.COM@XYZ.COM">XYZ.COM@XYZ.COM</a></a><br>
><br>
><br>
>
[root@prod-ipa-master-1a
:~] ipactl
status<br>
> Directory
Service:
RUNNING<br>
> krb5kdc
Service:
RUNNING<br>
> kadmin
Service:
RUNNING<br>
>
ipa_memcached
Service:
RUNNING<br>
> httpd
Service:
RUNNING<br>
>
pki-tomcatd
Service:
RUNNING<br>
> ipa-otpd
Service:
RUNNING<br>
> ipa:
INFO: The
ipactl command
was successful<br>
><br>
><br>
><br>
>
[root@prod-ipa-master
:~] ipa
user-find
p-testuser<br>
> ipa:
ERROR:
Kerberos
error:
('Unspecified
GSS failure.
Minor code may<br>
> provide
more
information',
851968)/("Cannot
contact any
KDC for realm
'<br>
> <a
moz-do-not-send="true"
href="http://XYZ.COM" rel="noreferrer" target="_blank">XYZ.COM</a>'",
-1765328228)<br>
</span></blockquote>
</div>
</div>
</blockquote>
<br>
</div>
</div>
Hi Rakesh,<br>
<br>
<blockquote>Having
a reproducible
test case
would you
rerun the
command above.<br>
During its
processing you
may monitor DS
process load
(top). If it
is high, you
may get some
pstacks of it.<br>
Also would you
attach the
part of DS
access logs
taken during
the command.<br>
<br>
regards<br>
thierry<br>
</blockquote>
<div>
<div>
<blockquote
type="cite">
<div
class="gmail_extra">
<div
class="gmail_quote">
<blockquote
class="gmail_quote"
style="margin:0
0 0
.8ex;border-left:1px
#ccc
solid;padding-left:1ex"><span>
><br>
<br>
</span>This is
weird because
the server
seems to be
up.<br>
<br>
Please follow<br>
<a
moz-do-not-send="true"
href="http://www.freeipa.org/page/Tr" target="_blank"><a class="moz-txt-link-freetext" href="http://www.freeipa.org/page/Tr">http://www.freeipa.org/page/Tr</a></a><wbr>oubleshooting#Authentication.2<wbr>FKerberos<br>
<br>
Petr^2 Spacek<br>
<div>
<div><br>
><br>
><br>
> Thanks<br>
><br>
> Rakesh<br>
><br>
> On Tue,
Aug 23, 2016
at 10:01 PM,
Rakesh
Rajasekharan
<<br>
> <a
moz-do-not-send="true"
href="mailto:rakesh.rajasekharan@gmail.com" target="_blank"><a class="moz-txt-link-abbreviated" href="mailto:rakesh.rajasekharan@gmail.com">rakesh.rajasekharan@gmail.com</a></a>>
wrote:<br>
><br>
>> i
changed the
loggin level
to 4 .
Modifying
nsslapd-accesslog-level<br>
>><br>
>> But,
the hang is
still there.
though I dont
see the
sigfault now<br>
>><br>
>><br>
>><br>
>><br>
>> On
Tue, Aug 23,
2016 at 9:02
PM, Rakesh
Rajasekharan
<<br>
>> <a
moz-do-not-send="true"
href="mailto:rakesh.rajasekharan@gmail.com" target="_blank"><a class="moz-txt-link-abbreviated" href="mailto:rakesh.rajasekharan@gmail.com">rakesh.rajasekharan@gmail.com</a></a>>
wrote:<br>
>><br>
>>>
My disk was
getting filled
too fast<br>
>>><br>
>>>
logs under
/var/log/dirsrv
was coming
around 5 gb
quickly
filling up<br>
>>><br>
>>>
Is there a way
to make the
logging less
verbose<br>
>>><br>
>>><br>
>>><br>
>>>
On Tue, Aug
23, 2016 at
6:41 PM, Petr
Spacek <<a
moz-do-not-send="true" href="mailto:pspacek@redhat.com" target="_blank"><a class="moz-txt-link-abbreviated" href="mailto:pspacek@redhat.com">pspacek@redhat.com</a></a>>
wrote:<br>
>>><br>
>>>>
On 23.8.2016
15:07, Rakesh
Rajasekharan
wrote:<br>
>>>>>
I was able to
fix that may
be
temporarily...
when i checked
the<br>
>>>>
network..<br>
>>>>>
there was
another
process that
was running
and consuming
a lot of<br>
>>>>
network (<br>
>>>>>
i have no idea
who did that.
I need to
seriously
start
restricting<br>
>>>>
people<br>
>>>>>
access to this
machine )<br>
>>>>><br>
>>>>>
after killing
that
perfomance
improved
drastically<br>
>>>>><br>
>>>>>
But now,
suddenly I
started
experiencing
the same hang.<br>
>>>>><br>
>>>>>
This time , I
gert the
following
error when
checked dmesg<br>
>>>>><br>
>>>>>
[ 301.236976]
ns-slapd[3124]:
segfault at 0
ip
00007f1de416951c
sp<br>
>>>>>
00007f1dee1dba70
error 4 in
libcos-plugin.so[7f1de4166000+<wbr>b000]<br>
>>>>>
[ 1116.248431]
TCP:
request_sock_TCP:
Possible SYN
flooding on
port 88.<br>
>>>>>
Sending
cookies.
Check SNMP
counters.<br>
>>>>>
[11831.397037]
ns-slapd[22550]:
segfault at 0
ip
00007f533d82251c
sp<br>
>>>>>
00007f5347894a70
error 4 in
libcos-plugin.so[7f533d81f000+<wbr>b000]<br>
>>>>>
[11832.727989]
ns-slapd[22606]:
segfault at 0
ip
00007f6231eb951c
sp<br>
>>>>>
00007f623bf2ba70
error 4 in
libcos-plugin.so[7f6231eb6000+<wbr>b00<br>
>>>><br>
>>>>
Okay, this one
is serious.
The LDAP
server
crashed.<br>
>>>><br>
>>>>
1. Make sure
all your
packages are
up-to-date.<br>
>>>><br>
>>>>
Please see<br>
>>>>
<a
moz-do-not-send="true"
href="http://directory.fedoraproject" target="_blank"><a class="moz-txt-link-freetext" href="http://directory.fedoraproject">http://directory.fedoraproject</a></a><wbr>.org/docs/389ds/FAQ/faq.html#d<br>
>>>>
ebugging-crashes<br>
>>>>
for further
instructions
how to debug
this.<br>
>>>><br>
>>>>
Petr^2 Spacek<br>
>>>><br>
>>>>><br>
>>>>>
and in
/var/log/dirsrv/example-com/er<wbr>rors<br>
>>>>><br>
>>>>>
[23/Aug/2016:12:49:36
+0000]
DSRetroclPlugin
-
delete_changerecord:<br>
>>>>
could<br>
>>>>>
not delete
change record
3291138 (rc:
32)<br>
>>>>>
[23/Aug/2016:12:49:36
+0000]
DSRetroclPlugin
-
delete_changerecord:<br>
>>>>
could<br>
>>>>>
not delete
change record
3291139 (rc:
32)<br>
>>>>>
[23/Aug/2016:12:49:36
+0000]
DSRetroclPlugin
-
delete_changerecord:<br>
>>>>
could<br>
>>>>>
not delete
change record
3291140 (rc:
32)<br>
>>>>>
[23/Aug/2016:12:49:36
+0000]
DSRetroclPlugin
-
delete_changerecord:<br>
>>>>
could<br>
>>>>>
not delete
change record
3291141 (rc:
32)<br>
>>>>>
[23/Aug/2016:12:49:36
+0000]
DSRetroclPlugin
-
delete_changerecord:<br>
>>>>
could<br>
>>>>>
not delete
change record
3291142 (rc:
32)<br>
>>>>>
[23/Aug/2016:12:49:36
+0000]
DSRetroclPlugin
-
delete_changerecord:<br>
>>>>
could<br>
>>>>>
not delete
change record
3291143 (rc:
32)<br>
>>>>>
[23/Aug/2016:12:49:36
+0000]
DSRetroclPlugin
-
delete_changerecord:<br>
>>>>
could<br>
>>>>>
not delete
change record
3291144 (rc:
32)<br>
>>>>>
[23/Aug/2016:12:49:36
+0000]
DSRetroclPlugin
-
delete_changerecord:<br>
>>>>
could<br>
>>>>>
not delete
change record
3291145 (rc:
32)<br>
>>>>>
[23/Aug/2016:12:49:50
+0000] - Retry
count exceeded
in delete<br>
>>>>>
[23/Aug/2016:12:49:50
+0000]
DSRetroclPlugin
-
delete_changerecord:<br>
>>>>
could<br>
>>>>>
not delete
change record
3292734 (rc:
51)<br>
>>>>><br>
>>>>><br>
>>>>>
Can i do
something
about this
error.. I
treid to
restart ipa a
couple<br>
>>>>
of<br>
>>>>>
time but that
did not help<br>
>>>>><br>
>>>>>
Thanks<br>
>>>>>
Rakesh<br>
>>>>><br>
>>>>>
On Mon, Aug
22, 2016 at
2:27 PM, Petr
Spacek <<a
moz-do-not-send="true" href="mailto:pspacek@redhat.com" target="_blank"><a class="moz-txt-link-abbreviated" href="mailto:pspacek@redhat.com">pspacek@redhat.com</a></a>><br>
>>>>
wrote:<br>
>>>>><br>
>>>>>>
On 19.8.2016
19:32, Rakesh
Rajasekharan
wrote:<br>
>>>>>>>
I am running
my set up on
AWS cloud, and
entropy is low
at around<br>
>>>>
180 .<br>
>>>>>>><br>
>>>>>>>
I plan to
increase it bu
installing
haveged . But,
would low
entropy<br>
>>>>
by<br>
>>>>>>
any<br>
>>>>>>>
chance cause
this issue of
intermittent
hang .<br>
>>>>>>>
Also, the hang
is mostly
observed when
registering
around 20
clients<br>
>>>>>>>
together<br>
>>>>>><br>
>>>>>>
Possibly, I'm
not sure. If
you want to
dig into this,
I would do
this:<br>
>>>>>>
1. look what
process hangs
on client
(using pstree
command or so)<br>
>>>>>>
$ pstree<br>
>>>>>><br>
>>>>>>
2. look to
what server
and port is
the hanging
client
connected to<br>
>>>>>>
$ lsof -p
<PID of the
hanging
process><br>
>>>>>><br>
>>>>>>
3. jump to
server and see
what process
is bound to
the target
port<br>
>>>>>>
$ netstat -pn<br>
>>>>>><br>
>>>>>>
4. see where
the process if
hanging<br>
>>>>>>
$ strace -p
<PID of the
hanging
process><br>
>>>>>><br>
>>>>>>
I hope it
helps.<br>
>>>>>><br>
>>>>>>
Petr^2 Spacek<br>
>>>>>><br>
>>>>>>>
On Fri, Aug
19, 2016 at
7:24 PM,
Rakesh
Rajasekharan
<<br>
>>>>>>>
<a
moz-do-not-send="true"
href="mailto:rakesh.rajasekharan@gmail.com" target="_blank"><a class="moz-txt-link-abbreviated" href="mailto:rakesh.rajasekharan@gmail.com">rakesh.rajasekharan@gmail.com</a></a>>
wrote:<br>
>>>>>>><br>
>>>>>>>>
yes there
seems to be
something
thats
worrying.. I
have faced
this<br>
>>>>
today<br>
>>>>>>>>
as well.<br>
>>>>>>>>
There are few
hosts around
280 odd left
and when i try
adding them<br>
>>>>
to<br>
>>>>>>
IPA<br>
>>>>>>>>
, the slowness
begins..<br>
>>>>>>>><br>
>>>>>>>>
all the ipa
commands like
ipa
user-find..
etc becomes
very slow in<br>
>>>>>>>>
responding.<br>
>>>>>>>><br>
>>>>>>>>
the SYNC_RECV
are not many
though just
around 80-90
and today that<br>
>>>>
was<br>
>>>>>>>>
around 20 only<br>
>>>>>>>><br>
>>>>>>>><br>
>>>>>>>>
I have for now
increased
tcp_max_syn_backlog
to 5000.<br>
>>>>>>>>
For now the
slowness seems
to have gone..
but I will do
a try<br>
>>>>
adding the<br>
>>>>>>>>
clients again
tomorrow and
see how it
goes<br>
>>>>>>>><br>
>>>>>>>>
Thanks<br>
>>>>>>>>
Rakesh<br>
>>>>>>>><br>
>>>>>>>>
The issues<br>
>>>>>>>><br>
>>>>>>>>
On Fri, Aug
19, 2016 at
12:58 PM, Petr
Spacek <<a
moz-do-not-send="true" href="mailto:pspacek@redhat.com" target="_blank"><a class="moz-txt-link-abbreviated" href="mailto:pspacek@redhat.com">pspacek@redhat.com</a></a>><br>
>>>>>>
wrote:<br>
>>>>>>>><br>
>>>>>>>>>
On 18.8.2016
17:23, Rakesh
Rajasekharan
wrote:<br>
>>>>>>>>>>
Hi<br>
>>>>>>>>>><br>
>>>>>>>>>>
I am migrating
to freeipa
from openldap
and have
around 4000<br>
>>>>
clients<br>
>>>>>>>>>><br>
>>>>>>>>>>
I had openned
a another
thread on
that, but
chose to start
a new<br>
>>>>
one<br>
>>>>>>>>>
here<br>
>>>>>>>>>>
as its a
separate issue<br>
>>>>>>>>>><br>
>>>>>>>>>>
I was able to
change the
nssslapd-maxdescriptors
adding an ldif<br>
>>>>
file<br>
>>>>>>>>>><br>
>>>>>>>>>>
cat
nsslapd-modify.ldif<br>
>>>>>>>>>>
dn: cn=config<br>
>>>>>>>>>>
changetype:
modify<br>
>>>>>>>>>>
replace:
nsslapd-maxdescriptors<br>
>>>>>>>>>>
nsslapd-maxdescriptors:
17000<br>
>>>>>>>>>><br>
>>>>>>>>>>
and running
the ldapmodify
command<br>
>>>>>>>>>><br>
>>>>>>>>>>
I have now
started moving
clients
running an
openldap to
Freeipa<br>
>>>>
and<br>
>>>>>>>>>
have<br>
>>>>>>>>>>
today moved
close to 2000
clients<br>
>>>>>>>>>><br>
>>>>>>>>>>
However, I
have noticed
that IPA hangs
intermittently.<br>
>>>>>>>>>><br>
>>>>>>>>>>
running a
kinit admin
returns the
below error<br>
>>>>>>>>>>
kinit: Generic
error (see
e-text) while
getting
initial<br>
>>>>
credentials<br>
>>>>>>>>>><br>
>>>>>>>>>>
from the
/var/log/messages,
I see this
entry<br>
>>>>>>>>>><br>
>>>>>>>>>>
prod-ipa-master-int
kernel:
[104090.315801]
TCP:<br>
>>>>
request_sock_TCP:<br>
>>>>>>>>>>
Possible SYN
flooding on
port 88.
Sending
cookies.
Check SNMP<br>
>>>>>>
counters.<br>
>>>>>>>>><br>
>>>>>>>>>
I would be
worried about
this message.
Maybe
kernel/firewall
is<br>
>>>>
doing<br>
>>>>>>>>>
something
fishy behind
your back and
blocking some
connections or<br>
>>>>
so.<br>
>>>>>>>>><br>
>>>>>>>>>
Petr^2 Spacek<br>
>>>>>>>>><br>
>>>>>>>>><br>
>>>>>>>>>>
Aug 18
13:00:01
prod-ipa-master-int
systemd[1]:
Started
Session<br>
>>>>
4885<br>
>>>>>>
of<br>
>>>>>>>>>>
user root.<br>
>>>>>>>>>>
Aug 18
13:00:01
prod-ipa-master-int
systemd[1]:
Starting
Session<br>
>>>>
4885<br>
>>>>>>
of<br>
>>>>>>>>>>
user root.<br>
>>>>>>>>>>
Aug 18
13:01:01
prod-ipa-master-int
systemd[1]:
Started
Session<br>
>>>>
4886<br>
>>>>>>
of<br>
>>>>>>>>>>
user root.<br>
>>>>>>>>>>
Aug 18
13:01:01
prod-ipa-master-int
systemd[1]:
Starting
Session<br>
>>>>
4886<br>
>>>>>>
of<br>
>>>>>>>>>>
user root.<br>
>>>>>>>>>>
Aug 18
13:02:40
prod-ipa-master-int
python[28984]:
ansible-command<br>
>>>>>>>>>
Invoked<br>
>>>>>>>>>>
with
creates=None
executable=None
shell=True
args=
removes=None<br>
>>>>>>>>>
warn=True<br>
>>>>>>>>>>
chdir=None<br>
>>>>>>>>>>
Aug 18
13:04:37
prod-ipa-master-int
sssd_be:
GSSAPI Error:<br>
>>>>
Unspecified<br>
>>>>>>>>>
GSS<br>
>>>>>>>>>>
failure.
Minor code may
provide more
information
(KDC returned<br>
>>>>
error<br>
>>>>>>>>>>
string:
PROCESS_TGS)<br>
>>>>>>>>>><br>
>>>>>>>>>>
Could it be
possible that
its due to the
initial load
of adding<br>
>>>>
the<br>
>>>>>>>>>
clients<br>
>>>>>>>>>>
or is there
something else
that I need to
take care of.<br>
</div>
</div>
</blockquote>
</div>
<br>
</div>
<br>
<fieldset></fieldset>
<br>
</blockquote>
<br>
</div>
</div>
</div>
</blockquote>
</div>
<br>
</div>
</div>
</div>
</blockquote>
</div>
<br>
</div>
</blockquote>
<br>
</div>
</div>
</div>
</blockquote>
</div>
<br>
</div>
<br>
<fieldset></fieldset>
<br>
</blockquote>
<p><br>
<span class="HOEnZb"><font
color="#888888"> <span><font
color="#888888"> </font></span></font></span></p>
<span class="HOEnZb"><font
color="#888888"> <span><font
color="#888888">
</font></span></font></span></div>
<span class="HOEnZb"><font
color="#888888"> <span><font
color="#888888"> </font></span></font></span></div>
<span class="HOEnZb"><font
color="#888888"> <span><font
color="#888888"> </font></span></font></span></div>
<span class="HOEnZb"><font
color="#888888"> <span><font
color="#888888"> <br>
</font></span></font></span></div>
<span class="HOEnZb"><font
color="#888888"> <span><font
color="#888888"> </font></span></font></span></div>
<span class="HOEnZb"><font
color="#888888"> <span><font
color="#888888"> <span>--<br>
Manage your subscription
for the Freeipa-users
mailing list:<br>
<a
moz-do-not-send="true"
href="https://www.redhat.com/mailman/listinfo/freeipa-users"
rel="noreferrer"
target="_blank"><a class="moz-txt-link-freetext" href="https://www.redhat.com/mailman">https://www.redhat.com/mailman</a><wbr>/listinfo/freeipa-users</a><br>
Go to <a
moz-do-not-send="true"
href="http://freeipa.org" rel="noreferrer" target="_blank"><a class="moz-txt-link-freetext" href="http://freeipa.org">http://freeipa.org</a></a>
for more info on the
project<br>
</span></font></span></font></span></blockquote>
<span class="HOEnZb"><font
color="#888888"> <span><font
color="#888888"> </font></span></font></span></div>
<span class="HOEnZb"><font
color="#888888"> <span><font
color="#888888"> <br>
</font></span></font></span></div>
<span class="HOEnZb"><font
color="#888888"> <span><font
color="#888888"> </font></span></font></span></blockquote>
<span class="HOEnZb"><font color="#888888">
<span><font color="#888888"> </font></span></font></span></div>
<span class="HOEnZb"><font color="#888888">
<span><font color="#888888"> <br>
</font></span></font></span></div>
<span class="HOEnZb"><font color="#888888"> <span><font
color="#888888"> <br>
<fieldset></fieldset>
<br>
</font></span></font></span></blockquote>
<span class="HOEnZb"><font color="#888888"> <br>
</font></span></div>
<span class="HOEnZb"><font color="#888888"> </font></span></blockquote>
<span class="HOEnZb"><font color="#888888"> </font></span></div>
<span class="HOEnZb"><font color="#888888"> <br>
</font></span></div>
</blockquote>
<br>
</div>
</blockquote>
</div>
<br>
</div>
</blockquote>
<br>
</body>
</html>