[Freeipa-users] Freeipa 4.2.0 hangs intermittently
Ludwig Krispenz
lkrispen at redhat.com
Mon Sep 5 12:44:45 UTC 2016
On 09/05/2016 02:31 PM, thierry bordaz wrote:
> Hi Rakesh,
>
> Thanks you soooo much for you continuous help on this !!
>
> If the hang is still going on, would you dump the DB lock information
> with:
>
> db_stat -CA -N -h /var/lib/dirsrv/slapd-<instance>/db
> >/tmp/db_lock_info
>
> It is looking like several threads are waiting for one/several db
> locks. With the lock info we will know which thread acquired it/them.
> A guess is that it could be the checkpointing thread but this one is
> also waiting (on a memory pool).
>
> I am copying Ludwig on this thread in case this pstack rings any bell.
the frequent presence of __db_tas_mutex_lock() looks like the NUMA
problem of libdb, although the threads already have stopped spinning
(maybe this was an effect of gcore).
If the hang aoccurs again, could you check cpu consumption
top -H -p <slapd-pid>
and get a pstack
pstack <slapd-pid>
>
> best regards
> thierry
>
> On 09/05/2016 02:05 PM, Rakesh Rajasekharan wrote:
>> Hi Thierry,
>>
>> I was able to now reporduce the issue..
>>
>> the ns-slapd process just got hung,.. however, no core file was
>> generate..
>>
>> I manually created a coredump using gcore and then took a strace of it
>> this is strace from the coredump generated
>>
>> GNU gdb (GDB) Red Hat Enterprise Linux 7.6.1-80.el7
>> Copyright (C) 2013 Free Software Foundation, Inc.
>> License GPLv3+: GNU GPL version 3 or later
>> <http://gnu.org/licenses/gpl.html>
>> This is free software: you are free to change and redistribute it.
>> There is NO WARRANTY, to the extent permitted by law. Type "show copying"
>> and "show warranty" for details.
>> This GDB was configured as "x86_64-redhat-linux-gnu".
>> For bug reporting instructions, please see:
>> <http://www.gnu.org/software/gdb/bugs/>...
>> Reading symbols from /usr/sbin/ns-slapd...Reading symbols from
>> /usr/sbin/ns-slapd...(no debugging symbols found)...done.
>> (no debugging symbols found)...done.
>> [New LWP 105159]
>> [New LWP 105160]
>> [New LWP 105161]
>> [New LWP 105162]
>> [New LWP 105163]
>> [New LWP 105164]
>> [New LWP 105165]
>> [New LWP 105166]
>> [New LWP 105167]
>> [New LWP 105168]
>> [New LWP 105171]
>> [New LWP 105172]
>> [New LWP 105173]
>> [New LWP 105174]
>> [New LWP 105175]
>> [New LWP 105176]
>> [New LWP 105177]
>> [New LWP 105178]
>> [New LWP 105179]
>> [New LWP 105180]
>> [New LWP 105181]
>> [New LWP 105182]
>> [New LWP 105183]
>> [New LWP 105184]
>> [New LWP 105185]
>> [New LWP 105186]
>> [New LWP 105187]
>> [New LWP 105188]
>> [New LWP 105189]
>> [New LWP 105190]
>> [New LWP 105191]
>> [New LWP 105192]
>> [New LWP 105193]
>> [New LWP 105194]
>> [New LWP 105195]
>> [New LWP 105196]
>> [New LWP 105197]
>> [New LWP 105198]
>> [New LWP 105199]
>> [New LWP 105200]
>> [New LWP 105201]
>> [New LWP 105203]
>> [New LWP 105215]
>> [New LWP 105158]
>> [Thread debugging using libthread_db enabled]
>> Using host libthread_db library "/lib64/libthread_db.so.1".
>> Core was generated by `/usr/sbin/ns-slapd'.
>> #0 0x00007f1b48ae69b3 in select () from /lib64/libc.so.6
>>
>> Thread 44 (Thread 0x7f1b4b6d9840 (LWP 105158)):
>> #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 () from
>> /lib64/libpthread.so.0
>> No symbol table info available.
>> #1 0x00007f1b4941b2c3 in PR_EnterMonitor () from /lib64/libnspr4.so
>> No symbol table info available.
>> #2 0x00007f1b4b7125ed in setup_pr_read_pds ()
>> No symbol table info available.
>> #3 0x00007f1b4b715fd9 in slapd_daemon ()
>> No symbol table info available.
>> #4 0x00007f1b4b707c6c in main ()
>> No symbol table info available.
>>
>> Thread 43 (Thread 0x7f1a5df0f700 (LWP 105215)):
>> #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 () from
>> /lib64/libpthread.so.0
>> No symbol table info available.
>> #1 0x00007f1b4941b050 in PR_WaitCondVar () from /lib64/libnspr4.so
>> No symbol table info available.
>> #2 0x00007f1b4b71df25 in ps_send_results ()
>> No symbol table info available.
>> #3 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so
>> No symbol table info available.
>> #4 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0
>> No symbol table info available.
>> #5 0x00007f1b48aef28d in clone () from /lib64/libc.so.6
>> No symbol table info available.
>>
>> Thread 42 (Thread 0x7f1b03fe7700 (LWP 105203)):
>> #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 () from
>> /lib64/libpthread.so.0
>> No symbol table info available.
>> #1 0x00007f1b4941b050 in PR_WaitCondVar () from /lib64/libnspr4.so
>> No symbol table info available.
>> #2 0x00007f1b4b71df25 in ps_send_results ()
>> No symbol table info available.
>> #3 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so
>> No symbol table info available.
>> #4 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0
>> No symbol table info available.
>> #5 0x00007f1b48aef28d in clone () from /lib64/libc.so.6
>> No symbol table info available.
>>
>> Thread 41 (Thread 0x7f1b047e8700 (LWP 105201)):
>> #0 0x00007f1b48ae69b3 in select () from /lib64/libc.so.6
>> No symbol table info available.
>> #1 0x00007f1b4b2710e9 in DS_Sleep () from
>> /usr/lib64/dirsrv/libslapd.so.0
>> No symbol table info available.
>> #2 0x00007f1b4b712325 in time_thread ()
>> No symbol table info available.
>> #3 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so
>> No symbol table info available.
>> #4 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0
>> No symbol table info available.
>> #5 0x00007f1b48aef28d in clone () from /lib64/libc.so.6
>> No symbol table info available.
>>
>> Thread 40 (Thread 0x7f1b04fe9700 (LWP 105200)):
>> #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 () from
>> /lib64/libpthread.so.0
>> No symbol table info available.
>> #1 0x00007f1b433212f3 in __db_hybrid_mutex_suspend () from
>> /lib64/libdb-5.3.so <http://libdb-5.3.so>
>> No symbol table info available.
>> #2 0x00007f1b43320640 in __db_tas_mutex_lock () from
>> /lib64/libdb-5.3.so <http://libdb-5.3.so>
>> No symbol table info available.
>> #3 0x00007f1b4343ae1b in __memp_fget () from /lib64/libdb-5.3.so
>> <http://libdb-5.3.so>
>> No symbol table info available.
>> #4 0x00007f1b4333e211 in __bam_search () from /lib64/libdb-5.3.so
>> <http://libdb-5.3.so>
>> No symbol table info available.
>> #5 0x00007f1b43329256 in __bamc_search () from /lib64/libdb-5.3.so
>> <http://libdb-5.3.so>
>> No symbol table info available.
>> #6 0x00007f1b4332d6c4 in __bamc_put () from /lib64/libdb-5.3.so
>> <http://libdb-5.3.so>
>> No symbol table info available.
>> #7 0x00007f1b433e46d5 in __dbc_iput () from /lib64/libdb-5.3.so
>> <http://libdb-5.3.so>
>> No symbol table info available.
>> #8 0x00007f1b433dfa0e in __db_put () from /lib64/libdb-5.3.so
>> <http://libdb-5.3.so>
>> No symbol table info available.
>> #9 0x00007f1b433f4fa4 in __db_put_pp () from /lib64/libdb-5.3.so
>> <http://libdb-5.3.so>
>> No symbol table info available.
>> #10 0x00007f1b3e8cf4cf in idl_new_insert_key () from
>> /usr/lib64/dirsrv/plugins/libback-ldbm.so
>> No symbol table info available.
>> #11 0x00007f1b3e8de5b5 in addordel_values_sv.isra.3 () from
>> /usr/lib64/dirsrv/plugins/libback-ldbm.so
>> No symbol table info available.
>> #12 0x00007f1b3e8df25a in index_addordel_values_ext_sv () from
>> /usr/lib64/dirsrv/plugins/libback-ldbm.so
>> No symbol table info available.
>> #13 0x00007f1b3e8df694 in index_addordel_values_sv () from
>> /usr/lib64/dirsrv/plugins/libback-ldbm.so
>> No symbol table info available.
>> #14 0x00007f1b3e8df75a in index_addordel_entry () from
>> /usr/lib64/dirsrv/plugins/libback-ldbm.so
>> No symbol table info available.
>> #15 0x00007f1b3e8e25fa in ldbm_back_add () from
>> /usr/lib64/dirsrv/plugins/libback-ldbm.so
>> No symbol table info available.
>> #16 0x00007f1b4b1e4cca in op_shared_add () from
>> /usr/lib64/dirsrv/libslapd.so.0
>> No symbol table info available.
>> #17 0x00007f1b4b1e6020 in do_add () from /usr/lib64/dirsrv/libslapd.so.0
>> No symbol table info available.
>> #18 0x00007f1b4b711303 in connection_threadmain ()
>> No symbol table info available.
>> #19 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so
>> No symbol table info available.
>> #20 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0
>> No symbol table info available.
>> #21 0x00007f1b48aef28d in clone () from /lib64/libc.so.6
>> No symbol table info available.
>>
>> Thread 39 (Thread 0x7f1b057ea700 (LWP 105199)):
>> #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 () from
>> /lib64/libpthread.so.0
>> No symbol table info available.
>> #1 0x00007f1b4941b050 in PR_WaitCondVar () from /lib64/libnspr4.so
>> No symbol table info available.
>> #2 0x00007f1b4b70efee in connection_wait_for_new_work ()
>> No symbol table info available.
>> #3 0x00007f1b4b710719 in connection_threadmain ()
>> No symbol table info available.
>> #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so
>> No symbol table info available.
>> #5 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0
>> No symbol table info available.
>> #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6
>> No symbol table info available.
>>
>> Thread 38 (Thread 0x7f1b05feb700 (LWP 105198)):
>> #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 () from
>> /lib64/libpthread.so.0
>> No symbol table info available.
>> #1 0x00007f1b4941b050 in PR_WaitCondVar () from /lib64/libnspr4.so
>> No symbol table info available.
>> #2 0x00007f1b4b70efee in connection_wait_for_new_work ()
>> No symbol table info available.
>> #3 0x00007f1b4b710719 in connection_threadmain ()
>> No symbol table info available.
>> #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so
>> No symbol table info available.
>> #5 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0
>> No symbol table info available.
>> #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6
>> No symbol table info available.
>>
>> Thread 37 (Thread 0x7f1b067ec700 (LWP 105197)):
>> #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 () from
>> /lib64/libpthread.so.0
>> No symbol table info available.
>> #1 0x00007f1b4941b050 in PR_WaitCondVar () from /lib64/libnspr4.so
>> No symbol table info available.
>> #2 0x00007f1b4b70efee in connection_wait_for_new_work ()
>> No symbol table info available.
>> #3 0x00007f1b4b710719 in connection_threadmain ()
>> No symbol table info available.
>> #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so
>> No symbol table info available.
>> #5 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0
>> No symbol table info available.
>> #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6
>> No symbol table info available.
>>
>> Thread 36 (Thread 0x7f1b06fed700 (LWP 105196)):
>> #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 () from
>> /lib64/libpthread.so.0
>> No symbol table info available.
>> #1 0x00007f1b4941b050 in PR_WaitCondVar () from /lib64/libnspr4.so
>> No symbol table info available.
>> #2 0x00007f1b4b70efee in connection_wait_for_new_work ()
>> No symbol table info available.
>> #3 0x00007f1b4b710719 in connection_threadmain ()
>> No symbol table info available.
>> #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so
>> No symbol table info available.
>> #5 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0
>> No symbol table info available.
>> #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6
>> No symbol table info available.
>>
>> Thread 35 (Thread 0x7f1b077ee700 (LWP 105195)):
>> #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 () from
>> /lib64/libpthread.so.0
>> No symbol table info available.
>> #1 0x00007f1b4941b2c3 in PR_EnterMonitor () from /lib64/libnspr4.so
>> No symbol table info available.
>> #2 0x00007f1b3e8c23e6 in dblayer_txn_begin () from
>> /usr/lib64/dirsrv/plugins/libback-ldbm.so
>> No symbol table info available.
>> #3 0x00007f1b3e8fe007 in ldbm_back_modify () from
>> /usr/lib64/dirsrv/plugins/libback-ldbm.so
>> No symbol table info available.
>> #4 0x00007f1b4b22b541 in op_shared_modify () from
>> /usr/lib64/dirsrv/libslapd.so.0
>> No symbol table info available.
>> #5 0x00007f1b4b22c004 in modify_internal_pb () from
>> /usr/lib64/dirsrv/libslapd.so.0
>> No symbol table info available.
>> #6 0x00007f1b3eb48b60 in ipapwd_apply_mods () from
>> /usr/lib64/dirsrv/plugins/libipa_pwd_extop.so
>> No symbol table info available.
>> #7 0x00007f1b3eb4c8e0 in store_new_keys () from
>> /usr/lib64/dirsrv/plugins/libipa_pwd_extop.so
>> No symbol table info available.
>> #8 0x00007f1b3eb4ead0 in ipapwd_extop () from
>> /usr/lib64/dirsrv/plugins/libipa_pwd_extop.so
>> No symbol table info available.
>> #9 0x00007f1b4b23bba2 in plugin_call_exop_plugins () from
>> /usr/lib64/dirsrv/libslapd.so.0
>> No symbol table info available.
>> #10 0x00007f1b4b717efe in do_extended ()
>> No symbol table info available.
>> #11 0x00007f1b4b71141a in connection_threadmain ()
>> No symbol table info available.
>> #12 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so
>> No symbol table info available.
>> #13 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0
>> No symbol table info available.
>> #14 0x00007f1b48aef28d in clone () from /lib64/libc.so.6
>> No symbol table info available.
>>
>> Thread 34 (Thread 0x7f1b07fef700 (LWP 105194)):
>> #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 () from
>> /lib64/libpthread.so.0
>> No symbol table info available.
>> #1 0x00007f1b4941b050 in PR_WaitCondVar () from /lib64/libnspr4.so
>> No symbol table info available.
>> #2 0x00007f1b4b70efee in connection_wait_for_new_work ()
>> No symbol table info available.
>> #3 0x00007f1b4b710719 in connection_threadmain ()
>> No symbol table info available.
>> #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so
>> No symbol table info available.
>> #5 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0
>> No symbol table info available.
>> #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6
>> No symbol table info available.
>>
>> Thread 33 (Thread 0x7f1b087f0700 (LWP 105193)):
>> #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 () from
>> /lib64/libpthread.so.0
>> No symbol table info available.
>> #1 0x00007f1b4941b050 in PR_WaitCondVar () from /lib64/libnspr4.so
>> No symbol table info available.
>> #2 0x00007f1b4b70efee in connection_wait_for_new_work ()
>> No symbol table info available.
>> #3 0x00007f1b4b710719 in connection_threadmain ()
>> No symbol table info available.
>> #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so
>> No symbol table info available.
>> #5 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0
>> No symbol table info available.
>> #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6
>> No symbol table info available.
>>
>> Thread 32 (Thread 0x7f1b08ff1700 (LWP 105192)):
>> #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 () from
>> /lib64/libpthread.so.0
>> No symbol table info available.
>> #1 0x00007f1b4941b050 in PR_WaitCondVar () from /lib64/libnspr4.so
>> No symbol table info available.
>> #2 0x00007f1b4b70efee in connection_wait_for_new_work ()
>> No symbol table info available.
>> #3 0x00007f1b4b710719 in connection_threadmain ()
>> No symbol table info available.
>> #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so
>> No symbol table info available.
>> #5 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0
>> No symbol table info available.
>> #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6
>> No symbol table info available.
>>
>> Thread 31 (Thread 0x7f1b097f2700 (LWP 105191)):
>> #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 () from
>> /lib64/libpthread.so.0
>> No symbol table info available.
>> #1 0x00007f1b4941b050 in PR_WaitCondVar () from /lib64/libnspr4.so
>> No symbol table info available.
>> #2 0x00007f1b4b70efee in connection_wait_for_new_work ()
>> No symbol table info available.
>> #3 0x00007f1b4b710719 in connection_threadmain ()
>> No symbol table info available.
>> #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so
>> No symbol table info available.
>> #5 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0
>> No symbol table info available.
>> #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6
>> No symbol table info available.
>>
>> Thread 30 (Thread 0x7f1b09ff3700 (LWP 105190)):
>> #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 () from
>> /lib64/libpthread.so.0
>> No symbol table info available.
>> #1 0x00007f1b433212f3 in __db_hybrid_mutex_suspend () from
>> /lib64/libdb-5.3.so <http://libdb-5.3.so>
>> No symbol table info available.
>> #2 0x00007f1b43320640 in __db_tas_mutex_lock () from
>> /lib64/libdb-5.3.so <http://libdb-5.3.so>
>> No symbol table info available.
>> #3 0x00007f1b433cad3a in __lock_get_internal () from
>> /lib64/libdb-5.3.so <http://libdb-5.3.so>
>> No symbol table info available.
>> #4 0x00007f1b433cb820 in __lock_get () from /lib64/libdb-5.3.so
>> <http://libdb-5.3.so>
>> No symbol table info available.
>> #5 0x00007f1b433f7142 in __db_lget () from /lib64/libdb-5.3.so
>> <http://libdb-5.3.so>
>> No symbol table info available.
>> #6 0x00007f1b4333e605 in __bam_search () from /lib64/libdb-5.3.so
>> <http://libdb-5.3.so>
>> No symbol table info available.
>> #7 0x00007f1b43329256 in __bamc_search () from /lib64/libdb-5.3.so
>> <http://libdb-5.3.so>
>> No symbol table info available.
>> #8 0x00007f1b4332ad0f in __bamc_get () from /lib64/libdb-5.3.so
>> <http://libdb-5.3.so>
>> No symbol table info available.
>> #9 0x00007f1b433e3ca6 in __dbc_iget () from /lib64/libdb-5.3.so
>> <http://libdb-5.3.so>
>> No symbol table info available.
>> #10 0x00007f1b433f2b02 in __dbc_get_pp () from /lib64/libdb-5.3.so
>> <http://libdb-5.3.so>
>> No symbol table info available.
>> #11 0x00007f1b3e8ce680 in idl_new_fetch () from
>> /usr/lib64/dirsrv/plugins/libback-ldbm.so
>> No symbol table info available.
>> #12 0x00007f1b3e8dcb26 in index_read_ext_allids () from
>> /usr/lib64/dirsrv/plugins/libback-ldbm.so
>> No symbol table info available.
>> #13 0x00007f1b3e8c7284 in keys2idl () from
>> /usr/lib64/dirsrv/plugins/libback-ldbm.so
>> No symbol table info available.
>> #14 0x00007f1b3e8c79e3 in ava_candidates.isra.0 () from
>> /usr/lib64/dirsrv/plugins/libback-ldbm.so
>> No symbol table info available.
>> #15 0x00007f1b3e8c7fd2 in filter_candidates_ext () from
>> /usr/lib64/dirsrv/plugins/libback-ldbm.so
>> No symbol table info available.
>> #16 0x00007f1b3e8c9096 in list_candidates () from
>> /usr/lib64/dirsrv/plugins/libback-ldbm.so
>> No symbol table info available.
>> #17 0x00007f1b3e8c7f40 in filter_candidates_ext () from
>> /usr/lib64/dirsrv/plugins/libback-ldbm.so
>> No symbol table info available.
>> #18 0x00007f1b3e8c9096 in list_candidates () from
>> /usr/lib64/dirsrv/plugins/libback-ldbm.so
>> No symbol table info available.
>> #19 0x00007f1b3e8c7f40 in filter_candidates_ext () from
>> /usr/lib64/dirsrv/plugins/libback-ldbm.so
>> No symbol table info available.
>> #20 0x00007f1b3e8c9096 in list_candidates () from
>> /usr/lib64/dirsrv/plugins/libback-ldbm.so
>> No symbol table info available.
>> #21 0x00007f1b3e8c7f40 in filter_candidates_ext () from
>> /usr/lib64/dirsrv/plugins/libback-ldbm.so
>> No symbol table info available.
>> #22 0x00007f1b3e90380c in subtree_candidates () from
>> /usr/lib64/dirsrv/plugins/libback-ldbm.so
>> No symbol table info available.
>> #23 0x00007f1b3e904e7f in ldbm_back_search () from
>> /usr/lib64/dirsrv/plugins/libback-ldbm.so
>> No symbol table info available.
>> #24 0x00007f1b4b2321d6 in op_shared_search () from
>> /usr/lib64/dirsrv/libslapd.so.0
>> No symbol table info available.
>> #25 0x00007f1b4b72333e in do_search ()
>> No symbol table info available.
>> #26 0x00007f1b4b7113c3 in connection_threadmain ()
>> No symbol table info available.
>> #27 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so
>> No symbol table info available.
>> #28 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0
>> No symbol table info available.
>> #29 0x00007f1b48aef28d in clone () from /lib64/libc.so.6
>> No symbol table info available.
>>
>> Thread 29 (Thread 0x7f1b0a7f4700 (LWP 105189)):
>> #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 () from
>> /lib64/libpthread.so.0
>> No symbol table info available.
>> #1 0x00007f1b4941b050 in PR_WaitCondVar () from /lib64/libnspr4.so
>> No symbol table info available.
>> #2 0x00007f1b4b70efee in connection_wait_for_new_work ()
>> No symbol table info available.
>> #3 0x00007f1b4b710719 in connection_threadmain ()
>> No symbol table info available.
>> #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so
>> No symbol table info available.
>> #5 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0
>> No symbol table info available.
>> #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6
>> No symbol table info available.
>>
>> Thread 28 (Thread 0x7f1b0aff5700 (LWP 105188)):
>> #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 () from
>> /lib64/libpthread.so.0
>> No symbol table info available.
>> #1 0x00007f1b4941b050 in PR_WaitCondVar () from /lib64/libnspr4.so
>> No symbol table info available.
>> #2 0x00007f1b4b70efee in connection_wait_for_new_work ()
>> No symbol table info available.
>> #3 0x00007f1b4b710719 in connection_threadmain ()
>> No symbol table info available.
>> #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so
>> No symbol table info available.
>> #5 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0
>> No symbol table info available.
>> #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6
>> No symbol table info available.
>>
>> Thread 27 (Thread 0x7f1b0b7f6700 (LWP 105187)):
>> #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 () from
>> /lib64/libpthread.so.0
>> No symbol table info available.
>> #1 0x00007f1b4941b050 in PR_WaitCondVar () from /lib64/libnspr4.so
>> No symbol table info available.
>> #2 0x00007f1b4b70efee in connection_wait_for_new_work ()
>> No symbol table info available.
>> #3 0x00007f1b4b710719 in connection_threadmain ()
>> No symbol table info available.
>> #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so
>> No symbol table info available.
>> #5 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0
>> No symbol table info available.
>> #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6
>> No symbol table info available.
>>
>> Thread 26 (Thread 0x7f1b0bff7700 (LWP 105186)):
>> #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 () from
>> /lib64/libpthread.so.0
>> No symbol table info available.
>> #1 0x00007f1b433212f3 in __db_hybrid_mutex_suspend () from
>> /lib64/libdb-5.3.so <http://libdb-5.3.so>
>> No symbol table info available.
>> #2 0x00007f1b43320640 in __db_tas_mutex_lock () from
>> /lib64/libdb-5.3.so <http://libdb-5.3.so>
>> No symbol table info available.
>> #3 0x00007f1b433cad3a in __lock_get_internal () from
>> /lib64/libdb-5.3.so <http://libdb-5.3.so>
>> No symbol table info available.
>> #4 0x00007f1b433cb820 in __lock_get () from /lib64/libdb-5.3.so
>> <http://libdb-5.3.so>
>> No symbol table info available.
>> #5 0x00007f1b433f7142 in __db_lget () from /lib64/libdb-5.3.so
>> <http://libdb-5.3.so>
>> No symbol table info available.
>> #6 0x00007f1b4333e605 in __bam_search () from /lib64/libdb-5.3.so
>> <http://libdb-5.3.so>
>> No symbol table info available.
>> #7 0x00007f1b43329256 in __bamc_search () from /lib64/libdb-5.3.so
>> <http://libdb-5.3.so>
>> No symbol table info available.
>> #8 0x00007f1b4332ad0f in __bamc_get () from /lib64/libdb-5.3.so
>> <http://libdb-5.3.so>
>> No symbol table info available.
>> #9 0x00007f1b433e3ca6 in __dbc_iget () from /lib64/libdb-5.3.so
>> <http://libdb-5.3.so>
>> No symbol table info available.
>> #10 0x00007f1b433f2b02 in __dbc_get_pp () from /lib64/libdb-5.3.so
>> <http://libdb-5.3.so>
>> No symbol table info available.
>> #11 0x00007f1b3e8ce680 in idl_new_fetch () from
>> /usr/lib64/dirsrv/plugins/libback-ldbm.so
>> No symbol table info available.
>> #12 0x00007f1b3e8dcb26 in index_read_ext_allids () from
>> /usr/lib64/dirsrv/plugins/libback-ldbm.so
>> No symbol table info available.
>> #13 0x00007f1b3e8c7284 in keys2idl () from
>> /usr/lib64/dirsrv/plugins/libback-ldbm.so
>> No symbol table info available.
>> #14 0x00007f1b3e8c79e3 in ava_candidates.isra.0 () from
>> /usr/lib64/dirsrv/plugins/libback-ldbm.so
>> No symbol table info available.
>> #15 0x00007f1b3e8c7fd2 in filter_candidates_ext () from
>> /usr/lib64/dirsrv/plugins/libback-ldbm.so
>> No symbol table info available.
>> #16 0x00007f1b3e8c9096 in list_candidates () from
>> /usr/lib64/dirsrv/plugins/libback-ldbm.so
>> No symbol table info available.
>> #17 0x00007f1b3e8c7f40 in filter_candidates_ext () from
>> /usr/lib64/dirsrv/plugins/libback-ldbm.so
>> No symbol table info available.
>> #18 0x00007f1b3e90380c in subtree_candidates () from
>> /usr/lib64/dirsrv/plugins/libback-ldbm.so
>> No symbol table info available.
>> #19 0x00007f1b3e904e7f in ldbm_back_search () from
>> /usr/lib64/dirsrv/plugins/libback-ldbm.so
>> No symbol table info available.
>> #20 0x00007f1b4b2321d6 in op_shared_search () from
>> /usr/lib64/dirsrv/libslapd.so.0
>> No symbol table info available.
>> #21 0x00007f1b4b24294e in search_internal_callback_pb () from
>> /usr/lib64/dirsrv/libslapd.so.0
>> No symbol table info available.
>> #22 0x00007f1b4b242be8 in search_internal_pb () from
>> /usr/lib64/dirsrv/libslapd.so.0
>> No symbol table info available.
>> #23 0x00007f1b4b72149b in ids_sasl_canon_user ()
>> No symbol table info available.
>> #24 0x00007f1b491e193b in _sasl_canon_user () from /lib64/libsasl2.so.3
>> No symbol table info available.
>> #25 0x00007f1b491e1c4c in _sasl_canon_user_lookup () from
>> /lib64/libsasl2.so.3
>> No symbol table info available.
>> #26 0x00007f1b430e9fb9 in gssapi_server_mech_step () from
>> /usr/lib64/sasl2/libgssapiv2.so
>> No symbol table info available.
>> #27 0x00007f1b491edb9b in sasl_server_step () from /lib64/libsasl2.so.3
>> No symbol table info available.
>> #28 0x00007f1b4b72283b in ids_sasl_check_bind ()
>> No symbol table info available.
>> #29 0x00007f1b4b70a61a in do_bind ()
>> No symbol table info available.
>> #30 0x00007f1b4b7113fd in connection_threadmain ()
>> No symbol table info available.
>> #31 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so
>> No symbol table info available.
>> #32 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0
>> No symbol table info available.
>> #33 0x00007f1b48aef28d in clone () from /lib64/libc.so.6
>> No symbol table info available.
>>
>> Thread 25 (Thread 0x7f1b0c7f8700 (LWP 105185)):
>> #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 () from
>> /lib64/libpthread.so.0
>> No symbol table info available.
>> #1 0x00007f1b4941b050 in PR_WaitCondVar () from /lib64/libnspr4.so
>> No symbol table info available.
>> #2 0x00007f1b4b70efee in connection_wait_for_new_work ()
>> No symbol table info available.
>> #3 0x00007f1b4b710719 in connection_threadmain ()
>> No symbol table info available.
>> #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so
>> No symbol table info available.
>> #5 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0
>> No symbol table info available.
>> #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6
>> No symbol table info available.
>>
>> Thread 24 (Thread 0x7f1b0cff9700 (LWP 105184)):
>> #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 () from
>> /lib64/libpthread.so.0
>> No symbol table info available.
>> #1 0x00007f1b4941b050 in PR_WaitCondVar () from /lib64/libnspr4.so
>> No symbol table info available.
>> #2 0x00007f1b4b70efee in connection_wait_for_new_work ()
>> No symbol table info available.
>> #3 0x00007f1b4b710719 in connection_threadmain ()
>> No symbol table info available.
>> #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so
>> No symbol table info available.
>> #5 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0
>> No symbol table info available.
>> #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6
>> No symbol table info available.
>>
>> Thread 23 (Thread 0x7f1b0d7fa700 (LWP 105183)):
>> #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 () from
>> /lib64/libpthread.so.0
>> No symbol table info available.
>> #1 0x00007f1b4941b050 in PR_WaitCondVar () from /lib64/libnspr4.so
>> No symbol table info available.
>> #2 0x00007f1b4b70efee in connection_wait_for_new_work ()
>> No symbol table info available.
>> #3 0x00007f1b4b710719 in connection_threadmain ()
>> No symbol table info available.
>> #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so
>> No symbol table info available.
>> #5 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0
>> No symbol table info available.
>> #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6
>> No symbol table info available.
>>
>> Thread 22 (Thread 0x7f1b0dffb700 (LWP 105182)):
>> #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 () from
>> /lib64/libpthread.so.0
>> No symbol table info available.
>> #1 0x00007f1b4941b050 in PR_WaitCondVar () from /lib64/libnspr4.so
>> No symbol table info available.
>> #2 0x00007f1b4b70efee in connection_wait_for_new_work ()
>> No symbol table info available.
>> #3 0x00007f1b4b710719 in connection_threadmain ()
>> No symbol table info available.
>> #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so
>> No symbol table info available.
>> #5 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0
>> No symbol table info available.
>> #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6
>> No symbol table info available.
>>
>> Thread 21 (Thread 0x7f1b0e7fc700 (LWP 105181)):
>> #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 () from
>> /lib64/libpthread.so.0
>> No symbol table info available.
>> #1 0x00007f1b4941b050 in PR_WaitCondVar () from /lib64/libnspr4.so
>> No symbol table info available.
>> #2 0x00007f1b4b70efee in connection_wait_for_new_work ()
>> No symbol table info available.
>> #3 0x00007f1b4b710719 in connection_threadmain ()
>> No symbol table info available.
>> #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so
>> No symbol table info available.
>> #5 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0
>> No symbol table info available.
>> #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6
>> No symbol table info available.
>>
>> Thread 20 (Thread 0x7f1b0effd700 (LWP 105180)):
>> #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 () from
>> /lib64/libpthread.so.0
>> No symbol table info available.
>> #1 0x00007f1b433212f3 in __db_hybrid_mutex_suspend () from
>> /lib64/libdb-5.3.so <http://libdb-5.3.so>
>> No symbol table info available.
>> #2 0x00007f1b43320b37 in __db_tas_mutex_readlock () from
>> /lib64/libdb-5.3.so <http://libdb-5.3.so>
>> No symbol table info available.
>> #3 0x00007f1b4343ad35 in __memp_fget () from /lib64/libdb-5.3.so
>> <http://libdb-5.3.so>
>> No symbol table info available.
>> #4 0x00007f1b433e4333 in __dbc_iget () from /lib64/libdb-5.3.so
>> <http://libdb-5.3.so>
>> No symbol table info available.
>> #5 0x00007f1b433f2b02 in __dbc_get_pp () from /lib64/libdb-5.3.so
>> <http://libdb-5.3.so>
>> No symbol table info available.
>> #6 0x00007f1b3e8ce822 in idl_new_fetch () from
>> /usr/lib64/dirsrv/plugins/libback-ldbm.so
>> No symbol table info available.
>> #7 0x00007f1b3e8dcb26 in index_read_ext_allids () from
>> /usr/lib64/dirsrv/plugins/libback-ldbm.so
>> No symbol table info available.
>> #8 0x00007f1b3e8c7284 in keys2idl () from
>> /usr/lib64/dirsrv/plugins/libback-ldbm.so
>> No symbol table info available.
>> #9 0x00007f1b3e8c79e3 in ava_candidates.isra.0 () from
>> /usr/lib64/dirsrv/plugins/libback-ldbm.so
>> No symbol table info available.
>> #10 0x00007f1b3e8c7fd2 in filter_candidates_ext () from
>> /usr/lib64/dirsrv/plugins/libback-ldbm.so
>> No symbol table info available.
>> #11 0x00007f1b3e8c9096 in list_candidates () from
>> /usr/lib64/dirsrv/plugins/libback-ldbm.so
>> No symbol table info available.
>> #12 0x00007f1b3e8c7f40 in filter_candidates_ext () from
>> /usr/lib64/dirsrv/plugins/libback-ldbm.so
>> No symbol table info available.
>> #13 0x00007f1b3e8c9096 in list_candidates () from
>> /usr/lib64/dirsrv/plugins/libback-ldbm.so
>> No symbol table info available.
>> #14 0x00007f1b3e8c7f40 in filter_candidates_ext () from
>> /usr/lib64/dirsrv/plugins/libback-ldbm.so
>> No symbol table info available.
>> #15 0x00007f1b3e8c9096 in list_candidates () from
>> /usr/lib64/dirsrv/plugins/libback-ldbm.so
>> No symbol table info available.
>> #16 0x00007f1b3e8c7f40 in filter_candidates_ext () from
>> /usr/lib64/dirsrv/plugins/libback-ldbm.so
>> No symbol table info available.
>> #17 0x00007f1b3e90380c in subtree_candidates () from
>> /usr/lib64/dirsrv/plugins/libback-ldbm.so
>> No symbol table info available.
>> #18 0x00007f1b3e904e7f in ldbm_back_search () from
>> /usr/lib64/dirsrv/plugins/libback-ldbm.so
>> No symbol table info available.
>> #19 0x00007f1b4b2321d6 in op_shared_search () from
>> /usr/lib64/dirsrv/libslapd.so.0
>> No symbol table info available.
>> #20 0x00007f1b4b72333e in do_search ()
>> No symbol table info available.
>> #21 0x00007f1b4b7113c3 in connection_threadmain ()
>> No symbol table info available.
>> #22 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so
>> No symbol table info available.
>> #23 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0
>> No symbol table info available.
>> #24 0x00007f1b48aef28d in clone () from /lib64/libc.so.6
>> No symbol table info available.
>>
>> Thread 19 (Thread 0x7f1b0f7fe700 (LWP 105179)):
>> #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 () from
>> /lib64/libpthread.so.0
>> No symbol table info available.
>> #1 0x00007f1b433212f3 in __db_hybrid_mutex_suspend () from
>> /lib64/libdb-5.3.so <http://libdb-5.3.so>
>> No symbol table info available.
>> #2 0x00007f1b43320640 in __db_tas_mutex_lock () from
>> /lib64/libdb-5.3.so <http://libdb-5.3.so>
>> No symbol table info available.
>> #3 0x00007f1b433cad3a in __lock_get_internal () from
>> /lib64/libdb-5.3.so <http://libdb-5.3.so>
>> No symbol table info available.
>> #4 0x00007f1b433cb820 in __lock_get () from /lib64/libdb-5.3.so
>> <http://libdb-5.3.so>
>> No symbol table info available.
>> #5 0x00007f1b433f7142 in __db_lget () from /lib64/libdb-5.3.so
>> <http://libdb-5.3.so>
>> No symbol table info available.
>> #6 0x00007f1b4333e605 in __bam_search () from /lib64/libdb-5.3.so
>> <http://libdb-5.3.so>
>> No symbol table info available.
>> #7 0x00007f1b43329256 in __bamc_search () from /lib64/libdb-5.3.so
>> <http://libdb-5.3.so>
>> No symbol table info available.
>> #8 0x00007f1b4332ad0f in __bamc_get () from /lib64/libdb-5.3.so
>> <http://libdb-5.3.so>
>> No symbol table info available.
>> #9 0x00007f1b433e3ca6 in __dbc_iget () from /lib64/libdb-5.3.so
>> <http://libdb-5.3.so>
>> No symbol table info available.
>> #10 0x00007f1b433f2b02 in __dbc_get_pp () from /lib64/libdb-5.3.so
>> <http://libdb-5.3.so>
>> No symbol table info available.
>> #11 0x00007f1b3e8ce680 in idl_new_fetch () from
>> /usr/lib64/dirsrv/plugins/libback-ldbm.so
>> No symbol table info available.
>> #12 0x00007f1b3e8dcb26 in index_read_ext_allids () from
>> /usr/lib64/dirsrv/plugins/libback-ldbm.so
>> No symbol table info available.
>> #13 0x00007f1b3e8c7284 in keys2idl () from
>> /usr/lib64/dirsrv/plugins/libback-ldbm.so
>> No symbol table info available.
>> #14 0x00007f1b3e8c79e3 in ava_candidates.isra.0 () from
>> /usr/lib64/dirsrv/plugins/libback-ldbm.so
>> No symbol table info available.
>> #15 0x00007f1b3e8c7fd2 in filter_candidates_ext () from
>> /usr/lib64/dirsrv/plugins/libback-ldbm.so
>> No symbol table info available.
>> #16 0x00007f1b3e8c9096 in list_candidates () from
>> /usr/lib64/dirsrv/plugins/libback-ldbm.so
>> No symbol table info available.
>> #17 0x00007f1b3e8c7f40 in filter_candidates_ext () from
>> /usr/lib64/dirsrv/plugins/libback-ldbm.so
>> No symbol table info available.
>> #18 0x00007f1b3e8c9096 in list_candidates () from
>> /usr/lib64/dirsrv/plugins/libback-ldbm.so
>> No symbol table info available.
>> #19 0x00007f1b3e8c7f40 in filter_candidates_ext () from
>> /usr/lib64/dirsrv/plugins/libback-ldbm.so
>> No symbol table info available.
>> #20 0x00007f1b3e8c9096 in list_candidates () from
>> /usr/lib64/dirsrv/plugins/libback-ldbm.so
>> No symbol table info available.
>> #21 0x00007f1b3e8c7f40 in filter_candidates_ext () from
>> /usr/lib64/dirsrv/plugins/libback-ldbm.so
>> No symbol table info available.
>> #22 0x00007f1b3e90380c in subtree_candidates () from
>> /usr/lib64/dirsrv/plugins/libback-ldbm.so
>> No symbol table info available.
>> #23 0x00007f1b3e904e7f in ldbm_back_search () from
>> /usr/lib64/dirsrv/plugins/libback-ldbm.so
>> No symbol table info available.
>> #24 0x00007f1b4b2321d6 in op_shared_search () from
>> /usr/lib64/dirsrv/libslapd.so.0
>> No symbol table info available.
>> #25 0x00007f1b4b72333e in do_search ()
>> No symbol table info available.
>> #26 0x00007f1b4b7113c3 in connection_threadmain ()
>> No symbol table info available.
>> #27 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so
>> No symbol table info available.
>> #28 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0
>> No symbol table info available.
>> #29 0x00007f1b48aef28d in clone () from /lib64/libc.so.6
>> No symbol table info available.
>>
>> Thread 18 (Thread 0x7f1b0ffff700 (LWP 105178)):
>> #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 () from
>> /lib64/libpthread.so.0
>> No symbol table info available.
>> #1 0x00007f1b4941b050 in PR_WaitCondVar () from /lib64/libnspr4.so
>> No symbol table info available.
>> #2 0x00007f1b4b70efee in connection_wait_for_new_work ()
>> No symbol table info available.
>> #3 0x00007f1b4b710719 in connection_threadmain ()
>> No symbol table info available.
>> #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so
>> No symbol table info available.
>> #5 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0
>> No symbol table info available.
>> #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6
>> No symbol table info available.
>>
>> Thread 17 (Thread 0x7f1b14d28700 (LWP 105177)):
>> #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 () from
>> /lib64/libpthread.so.0
>> No symbol table info available.
>> #1 0x00007f1b4941b050 in PR_WaitCondVar () from /lib64/libnspr4.so
>> No symbol table info available.
>> #2 0x00007f1b4b70efee in connection_wait_for_new_work ()
>> No symbol table info available.
>> #3 0x00007f1b4b710719 in connection_threadmain ()
>> No symbol table info available.
>> #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so
>> No symbol table info available.
>> #5 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0
>> No symbol table info available.
>> #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6
>> No symbol table info available.
>>
>> Thread 16 (Thread 0x7f1b15529700 (LWP 105176)):
>> #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 () from
>> /lib64/libpthread.so.0
>> No symbol table info available.
>> #1 0x00007f1b4941b050 in PR_WaitCondVar () from /lib64/libnspr4.so
>> No symbol table info available.
>> #2 0x00007f1b4b70efee in connection_wait_for_new_work ()
>> No symbol table info available.
>> #3 0x00007f1b4b710719 in connection_threadmain ()
>> No symbol table info available.
>> #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so
>> No symbol table info available.
>> #5 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0
>> No symbol table info available.
>> #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6
>> No symbol table info available.
>>
>> Thread 15 (Thread 0x7f1b15d2a700 (LWP 105175)):
>> #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 () from
>> /lib64/libpthread.so.0
>> No symbol table info available.
>> #1 0x00007f1b4941b050 in PR_WaitCondVar () from /lib64/libnspr4.so
>> No symbol table info available.
>> #2 0x00007f1b4b70efee in connection_wait_for_new_work ()
>> No symbol table info available.
>> #3 0x00007f1b4b710719 in connection_threadmain ()
>> No symbol table info available.
>> #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so
>> No symbol table info available.
>> #5 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0
>> No symbol table info available.
>> #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6
>> No symbol table info available.
>>
>> Thread 14 (Thread 0x7f1b1652b700 (LWP 105174)):
>> #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 () from
>> /lib64/libpthread.so.0
>> No symbol table info available.
>> #1 0x00007f1b4941b050 in PR_WaitCondVar () from /lib64/libnspr4.so
>> No symbol table info available.
>> #2 0x00007f1b4b70efee in connection_wait_for_new_work ()
>> No symbol table info available.
>> #3 0x00007f1b4b710719 in connection_threadmain ()
>> No symbol table info available.
>> #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so
>> No symbol table info available.
>> #5 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0
>> No symbol table info available.
>> #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6
>> No symbol table info available.
>>
>> Thread 13 (Thread 0x7f1b16d2c700 (LWP 105173)):
>> #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 () from
>> /lib64/libpthread.so.0
>> No symbol table info available.
>> #1 0x00007f1b433212f3 in __db_hybrid_mutex_suspend () from
>> /lib64/libdb-5.3.so <http://libdb-5.3.so>
>> No symbol table info available.
>> #2 0x00007f1b43320640 in __db_tas_mutex_lock () from
>> /lib64/libdb-5.3.so <http://libdb-5.3.so>
>> No symbol table info available.
>> #3 0x00007f1b433cad3a in __lock_get_internal () from
>> /lib64/libdb-5.3.so <http://libdb-5.3.so>
>> No symbol table info available.
>> #4 0x00007f1b433cb820 in __lock_get () from /lib64/libdb-5.3.so
>> <http://libdb-5.3.so>
>> No symbol table info available.
>> #5 0x00007f1b433f7142 in __db_lget () from /lib64/libdb-5.3.so
>> <http://libdb-5.3.so>
>> No symbol table info available.
>> #6 0x00007f1b4333e605 in __bam_search () from /lib64/libdb-5.3.so
>> <http://libdb-5.3.so>
>> No symbol table info available.
>> #7 0x00007f1b43329256 in __bamc_search () from /lib64/libdb-5.3.so
>> <http://libdb-5.3.so>
>> No symbol table info available.
>> #8 0x00007f1b4332ad0f in __bamc_get () from /lib64/libdb-5.3.so
>> <http://libdb-5.3.so>
>> No symbol table info available.
>> #9 0x00007f1b433e3ca6 in __dbc_iget () from /lib64/libdb-5.3.so
>> <http://libdb-5.3.so>
>> No symbol table info available.
>> #10 0x00007f1b433f2b02 in __dbc_get_pp () from /lib64/libdb-5.3.so
>> <http://libdb-5.3.so>
>> No symbol table info available.
>> #11 0x00007f1b3e8ce680 in idl_new_fetch () from
>> /usr/lib64/dirsrv/plugins/libback-ldbm.so
>> No symbol table info available.
>> #12 0x00007f1b3e8dcb26 in index_read_ext_allids () from
>> /usr/lib64/dirsrv/plugins/libback-ldbm.so
>> No symbol table info available.
>> #13 0x00007f1b3e8c7284 in keys2idl () from
>> /usr/lib64/dirsrv/plugins/libback-ldbm.so
>> No symbol table info available.
>> #14 0x00007f1b3e8c79e3 in ava_candidates.isra.0 () from
>> /usr/lib64/dirsrv/plugins/libback-ldbm.so
>> No symbol table info available.
>> #15 0x00007f1b3e8c7fd2 in filter_candidates_ext () from
>> /usr/lib64/dirsrv/plugins/libback-ldbm.so
>> No symbol table info available.
>> #16 0x00007f1b3e8c9096 in list_candidates () from
>> /usr/lib64/dirsrv/plugins/libback-ldbm.so
>> No symbol table info available.
>> #17 0x00007f1b3e8c7f40 in filter_candidates_ext () from
>> /usr/lib64/dirsrv/plugins/libback-ldbm.so
>> No symbol table info available.
>> #18 0x00007f1b3e8c9096 in list_candidates () from
>> /usr/lib64/dirsrv/plugins/libback-ldbm.so
>> No symbol table info available.
>> #19 0x00007f1b3e8c7f40 in filter_candidates_ext () from
>> /usr/lib64/dirsrv/plugins/libback-ldbm.so
>> No symbol table info available.
>> #20 0x00007f1b3e8c9096 in list_candidates () from
>> /usr/lib64/dirsrv/plugins/libback-ldbm.so
>> No symbol table info available.
>> #21 0x00007f1b3e8c7f40 in filter_candidates_ext () from
>> /usr/lib64/dirsrv/plugins/libback-ldbm.so
>> No symbol table info available.
>> #22 0x00007f1b3e90380c in subtree_candidates () from
>> /usr/lib64/dirsrv/plugins/libback-ldbm.so
>> No symbol table info available.
>> #23 0x00007f1b3e904e7f in ldbm_back_search () from
>> /usr/lib64/dirsrv/plugins/libback-ldbm.so
>> No symbol table info available.
>> #24 0x00007f1b4b2321d6 in op_shared_search () from
>> /usr/lib64/dirsrv/libslapd.so.0
>> No symbol table info available.
>> #25 0x00007f1b4b72333e in do_search ()
>> No symbol table info available.
>> #26 0x00007f1b4b7113c3 in connection_threadmain ()
>> No symbol table info available.
>> #27 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so
>> No symbol table info available.
>> #28 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0
>> No symbol table info available.
>> #29 0x00007f1b48aef28d in clone () from /lib64/libc.so.6
>> No symbol table info available.
>>
>> Thread 12 (Thread 0x7f1b1752d700 (LWP 105172)):
>> #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 () from
>> /lib64/libpthread.so.0
>> No symbol table info available.
>> #1 0x00007f1b4941b050 in PR_WaitCondVar () from /lib64/libnspr4.so
>> No symbol table info available.
>> #2 0x00007f1b4b70efee in connection_wait_for_new_work ()
>> No symbol table info available.
>> #3 0x00007f1b4b710719 in connection_threadmain ()
>> No symbol table info available.
>> #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so
>> No symbol table info available.
>> #5 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0
>> No symbol table info available.
>> #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6
>> No symbol table info available.
>>
>> Thread 11 (Thread 0x7f1b17d2e700 (LWP 105171)):
>> #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 () from
>> /lib64/libpthread.so.0
>> No symbol table info available.
>> #1 0x00007f1b4941b050 in PR_WaitCondVar () from /lib64/libnspr4.so
>> No symbol table info available.
>> #2 0x00007f1b4b70efee in connection_wait_for_new_work ()
>> No symbol table info available.
>> #3 0x00007f1b4b710719 in connection_threadmain ()
>> No symbol table info available.
>> #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so
>> No symbol table info available.
>> #5 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0
>> No symbol table info available.
>> #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6
>> No symbol table info available.
>>
>> Thread 10 (Thread 0x7f1b2509d700 (LWP 105168)):
>> #0 0x00007f1b48dc5a82 in pthread_cond_timedwait@@GLIBC_2.3.2 () from
>> /lib64/libpthread.so.0
>> No symbol table info available.
>> #1 0x00007f1b4941ab07 in pt_TimedWait () from /lib64/libnspr4.so
>> No symbol table info available.
>> #2 0x00007f1b4941afce in PR_WaitCondVar () from /lib64/libnspr4.so
>> No symbol table info available.
>> #3 0x00007f1b4b2061e8 in eq_loop () from /usr/lib64/dirsrv/libslapd.so.0
>> No symbol table info available.
>> #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so
>> No symbol table info available.
>> #5 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0
>> No symbol table info available.
>> #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6
>> No symbol table info available.
>>
>> Thread 9 (Thread 0x7f1b2589e700 (LWP 105167)):
>> #0 0x00007f1b48dc5a82 in pthread_cond_timedwait@@GLIBC_2.3.2 () from
>> /lib64/libpthread.so.0
>> No symbol table info available.
>> #1 0x00007f1b4941ab07 in pt_TimedWait () from /lib64/libnspr4.so
>> No symbol table info available.
>> #2 0x00007f1b4941afce in PR_WaitCondVar () from /lib64/libnspr4.so
>> No symbol table info available.
>> #3 0x00007f1b4b719d03 in housecleaning ()
>> No symbol table info available.
>> #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so
>> No symbol table info available.
>> #5 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0
>> No symbol table info available.
>> #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6
>> No symbol table info available.
>>
>> Thread 8 (Thread 0x7f1b2609f700 (LWP 105166)):
>> #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 () from
>> /lib64/libpthread.so.0
>> No symbol table info available.
>> #1 0x00007f1b4941b050 in PR_WaitCondVar () from /lib64/libnspr4.so
>> No symbol table info available.
>> #2 0x00007f1b4b260198 in slapi_wait_condvar () from
>> /usr/lib64/dirsrv/libslapd.so.0
>> No symbol table info available.
>> #3 0x00007f1b3cf5deed in roles_cache_wait_on_change () from
>> /usr/lib64/dirsrv/plugins/libroles-plugin.so
>> No symbol table info available.
>> #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so
>> No symbol table info available.
>> #5 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0
>> No symbol table info available.
>> #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6
>> No symbol table info available.
>>
>> Thread 7 (Thread 0x7f1b268a0700 (LWP 105165)):
>> #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 () from
>> /lib64/libpthread.so.0
>> No symbol table info available.
>> #1 0x00007f1b4941b050 in PR_WaitCondVar () from /lib64/libnspr4.so
>> No symbol table info available.
>> #2 0x00007f1b4b260198 in slapi_wait_condvar () from
>> /usr/lib64/dirsrv/libslapd.so.0
>> No symbol table info available.
>> #3 0x00007f1b3cf5deed in roles_cache_wait_on_change () from
>> /usr/lib64/dirsrv/plugins/libroles-plugin.so
>> No symbol table info available.
>> #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so
>> No symbol table info available.
>> #5 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0
>> No symbol table info available.
>> #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6
>> No symbol table info available.
>>
>> Thread 6 (Thread 0x7f1b270a1700 (LWP 105164)):
>> #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 () from
>> /lib64/libpthread.so.0
>> No symbol table info available.
>> #1 0x00007f1b4941b050 in PR_WaitCondVar () from /lib64/libnspr4.so
>> No symbol table info available.
>> #2 0x00007f1b4b260198 in slapi_wait_condvar () from
>> /usr/lib64/dirsrv/libslapd.so.0
>> No symbol table info available.
>> #3 0x00007f1b3cf5deed in roles_cache_wait_on_change () from
>> /usr/lib64/dirsrv/plugins/libroles-plugin.so
>> No symbol table info available.
>> #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so
>> No symbol table info available.
>> #5 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0
>> No symbol table info available.
>> #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6
>> No symbol table info available.
>>
>> Thread 5 (Thread 0x7f1b278a2700 (LWP 105163)):
>> #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 () from
>> /lib64/libpthread.so.0
>> No symbol table info available.
>> #1 0x00007f1b4941b050 in PR_WaitCondVar () from /lib64/libnspr4.so
>> No symbol table info available.
>> #2 0x00007f1b4b260198 in slapi_wait_condvar () from
>> /usr/lib64/dirsrv/libslapd.so.0
>> No symbol table info available.
>> #3 0x00007f1b415f262e in cos_cache_wait_on_change () from
>> /usr/lib64/dirsrv/plugins/libcos-plugin.so
>> No symbol table info available.
>> #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so
>> No symbol table info available.
>> #5 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0
>> No symbol table info available.
>> #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6
>> No symbol table info available.
>>
>> Thread 4 (Thread 0x7f1b282ac700 (LWP 105162)):
>> #0 0x00007f1b48ae69b3 in select () from /lib64/libc.so.6
>> No symbol table info available.
>> #1 0x00007f1b4b2710e9 in DS_Sleep () from
>> /usr/lib64/dirsrv/libslapd.so.0
>> No symbol table info available.
>> #2 0x00007f1b3e8ba707 in perf_threadmain () from
>> /usr/lib64/dirsrv/plugins/libback-ldbm.so
>> No symbol table info available.
>> #3 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so
>> No symbol table info available.
>> #4 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0
>> No symbol table info available.
>> #5 0x00007f1b48aef28d in clone () from /lib64/libc.so.6
>> No symbol table info available.
>>
>> Thread 3 (Thread 0x7f1b28aad700 (LWP 105161)):
>> #0 0x00007f1b48ae69b3 in select () from /lib64/libc.so.6
>> No symbol table info available.
>> #1 0x00007f1b4b2710e9 in DS_Sleep () from
>> /usr/lib64/dirsrv/libslapd.so.0
>> No symbol table info available.
>> #2 0x00007f1b3e8bfb7f in trickle_threadmain () from
>> /usr/lib64/dirsrv/plugins/libback-ldbm.so
>> No symbol table info available.
>> #3 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so
>> No symbol table info available.
>> #4 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0
>> No symbol table info available.
>> #5 0x00007f1b48aef28d in clone () from /lib64/libc.so.6
>> No symbol table info available.
>>
>> Thread 2 (Thread 0x7f1b292ae700 (LWP 105160)):
>> #0 0x00007f1b48ae69b3 in select () from /lib64/libc.so.6
>> No symbol table info available.
>> #1 0x00007f1b4344dbad in __os_yield () from /lib64/libdb-5.3.so
>> <http://libdb-5.3.so>
>> No symbol table info available.
>> #2 0x00007f1b434492b3 in __memp_sync_int () from /lib64/libdb-5.3.so
>> <http://libdb-5.3.so>
>> No symbol table info available.
>> #3 0x00007f1b43459752 in __txn_checkpoint () from
>> /lib64/libdb-5.3.so <http://libdb-5.3.so>
>> No symbol table info available.
>> #4 0x00007f1b43459b74 in __txn_checkpoint_pp () from
>> /lib64/libdb-5.3.so <http://libdb-5.3.so>
>> No symbol table info available.
>> #5 0x00007f1b3e8c3a87 in checkpoint_threadmain () from
>> /usr/lib64/dirsrv/plugins/libback-ldbm.so
>> No symbol table info available.
>> #6 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so
>> No symbol table info available.
>> #7 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0
>> No symbol table info available.
>> #8 0x00007f1b48aef28d in clone () from /lib64/libc.so.6
>> No symbol table info available.
>>
>> Thread 1 (Thread 0x7f1b29aaf700 (LWP 105159)):
>> #0 0x00007f1b48ae69b3 in select () from /lib64/libc.so.6
>> No symbol table info available.
>> #1 0x00007f1b4b2710e9 in DS_Sleep () from
>> /usr/lib64/dirsrv/libslapd.so.0
>> No symbol table info available.
>> #2 0x00007f1b3e8bf907 in deadlock_threadmain () from
>> /usr/lib64/dirsrv/plugins/libback-ldbm.so
>> No symbol table info available.
>> #3 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so
>> No symbol table info available.
>> #4 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0
>> No symbol table info available.
>> #5 0x00007f1b48aef28d in clone () from /lib64/libc.so.6
>> No symbol table info available.
>>
>>
>>
>> Thanks,
>>
>> Rakesh
>>
>>
>> On Mon, Sep 5, 2016 at 4:48 PM, thierry bordaz <tbordaz at redhat.com
>> <mailto:tbordaz at redhat.com>> wrote:
>>
>>
>>
>> On 09/05/2016 12:05 PM, Rakesh Rajasekharan wrote:
>>> Hi Thierry,
>>>
>>>
>>> I was getting the hang issue while running ipa-client-install
>>> simultaneously on few clients..
>>> However, today, I am not able to replicate that.
>>>
>>> I could not get a gdb . But i will try getting that the next
>>> time I face this issue.
>>>
>>> The CPU does not stay high.. it just momentarily touches a high
>>> value and then drops down to around 2-7%
>>>
>>> One question I have is , is it ok to set it nsslapd-threadnumber
>>> to a very high value .
>>> I have around 4000 clients and with nsslapd-maxthreadsperconn
>>> set to 5..So, can I set nsslapd-threadnumber to around 25000.
>>
>> Hello,
>>
>> I know some users running in production with several hunderds of
>> threads (>600) and this without problem.
>>
>> I do not recall having suggested to increase that number and for
>> what reason.
>> Usually 30 workers is a good enough value. It can create
>> bootleneck if for some reason each operation is very long to
>> satisfy and exhaust the number of workers. You can monitor the
>> work queue:
>>
>> ldapsearch -D "cn=directory manager" -w xxx -LLL -b
>> "cn=monitor" -s base opsinitiated opscompleted
>>
>>
>> If opscompleted-opsinitiated remains close to threadnumber, then
>> yes it would be valuable to increase it.
>>
>> The computation #client * #async_op_per_client sound an overkill.
>> Even if all clients send at the exact same time all their
>> requests, it is very likely that some common resource (db page,
>> log, allocator...) will serialize them. If you monitor a need to
>> increase the work, you would for example set it to 50, then
>> monitor, then set it to 100, then monitor... until you find a
>> good enough value.
>> Note the increasing the #thread, increases the memory footprint
>> that will reduce the efficiency of file system cache and can
>> increase the response time.
>>
>>
>> best regards
>> thierry
>>
>>>
>>> Thanks
>>>
>>> On Mon, Sep 5, 2016 at 1:03 PM, thierry bordaz
>>> <tbordaz at redhat.com> wrote:
>>>
>>>
>>> Hi Rakesh,
>>>
>>> Were you able to get a pstack or full stack with gdb
>>> (http://www.port389.org/docs/389ds/FAQ/faq.html#debugging-crashes)
>>> when the server hangs ?
>>>
>>> If it happens with 500 threads as well as with 30, using 30
>>> threads is a better choice to debug this issue.
>>> I will try to reproduce using 150 parallel 'ipa user-find
>>> p-testipa' commands
>>>
>>> Something I am unsure is if the CPU consumption stays high
>>> (you mentioned 340% CPU usage) as long as the hang happens
>>> or if after a suddent shot up to 340% (that marks the
>>> beginning of the hang) it drops and stay hanging ?
>>>
>>> thanks
>>> thierry
>>>
>>> On 09/04/2016 08:40 PM, Rakesh Rajasekharan wrote:
>>>> starce on the slapd process actually had this in the output..
>>>> FUTEX_WAIT_PRIVATE
>>>>
>>>> and checking for the number of threads slapd had.. there
>>>> were 5015 threads
>>>>
>>>> ps -efL|grep slapd|wc -l
>>>> 5015
>>>>
>>>> strace on most of the threads gave this output
>>>>
>>>> strace -p 67411
>>>> Process 67411 attached
>>>> futex(0x7f3f0226b41c, FUTEX_WAIT_PRIVATE, 1, NULL) = -1
>>>> EAGAIN (Resource temporarily unavailable)
>>>> futex(0x7f3f0226b41c, FUTEX_WAIT_PRIVATE, 2, NULL^CProcess
>>>> 67411 detached
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> On Sun, Sep 4, 2016 at 5:34 PM, Rakesh Rajasekharan
>>>> <rakesh.rajasekharan at gmail.com
>>>> <mailto:rakesh.rajasekharan at gmail.com>> wrote:
>>>>
>>>> I have again got the issue of IPA hanging.. The issue
>>>> came up when i tried to run ipa-client-isntall on 142
>>>> clients simultaneously
>>>>
>>>>
>>>> None of the IPA commands are responding, and I see this
>>>> error
>>>>
>>>> ipa user-find p-testipa
>>>> ipa: ERROR: Insufficient access: SASL(-1): generic
>>>> failure: GSSAPI Error: Unspecified GSS failure. Minor
>>>> code may provide more information (KDC returned error
>>>> string: PROCESS_TGS)
>>>>
>>>> KRB5_TRACE=/dev/stdout kinit admin
>>>> [41178] 1472984115.233214: Getting initial credentials
>>>> for admin at XYZ.COM
>>>> [41178] 1472984115.235257: Sending request (167 bytes)
>>>> to XYZ.COM <http://XYZ.COM>
>>>> [41178] 1472984115.235419: Initiating TCP connection to
>>>> stream 10.1.3.36:88 <http://10.1.3.36:88>
>>>> [41178] 1472984115.235685: Sending TCP request to
>>>> stream 10.1.3.36:88 <http://10.1.3.36:88>
>>>> [41178] 1472984120.238914: Received answer (174 bytes)
>>>> from stream 10.1.3.36:88 <http://10.1.3.36:88>
>>>> [41178] 1472984120.238925: Terminating TCP connection
>>>> to stream 10.1.3.36:88 <http://10.1.3.36:88>
>>>> [41178] 1472984120.238993: Response was from master KDC
>>>> [41
>>>>
>>>>
>>>> Running an ldapsearch to see the db.. does not give any
>>>> results and just hangs there
>>>>
>>>> ldapsearch -x -D 'cn=Directory Manager' -W -s one -b
>>>> 'cn=kerberos,dc=xyz,dc=com'
>>>> Enter LDAP Password:
>>>>
>>>> even an ldapsearch -x does not respond
>>>> At this point, am sure that slapd is the one causing issues
>>>>
>>>> Running an strace against the hung slapd itself seems
>>>> to get stuck does not proceed after saying "attaching
>>>> to process"
>>>>
>>>> From some others posts I read Thierry suggesting to
>>>> increase the nsslapd-threadnumber value
>>>>
>>>> It was set to 30, I think that might be too low.
>>>>
>>>> I have raised it to 500
>>>>
>>>> Now after restarting the service .. ldapsearch starts
>>>> responding.
>>>> But running the test to add a sudden high number of
>>>> clients again left ns-slapd to hung state
>>>>
>>>> When i attempted adding the clients.. the ns-slapd cpu
>>>> usage shot up to 340% and after that ns-slapd stopped
>>>> responding
>>>>
>>>> So now, atleast I know what might be causing the issue
>>>> and I can now easily reproduce it.
>>>>
>>>> Is there a way I can make ns-slapd handle a sudden bump
>>>> in incoming request for ipa-client-install
>>>>
>>>> Thanks
>>>> Rakesh
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> On Mon, Aug 29, 2016 at 11:18 PM, Rich Megginson
>>>> <rmeggins at redhat.com> wrote:
>>>>
>>>> On 08/29/2016 10:53 AM, Rakesh Rajasekharan wrote:
>>>>> Hi Thierry,
>>>>>
>>>>> My machine has 30GB RAM ..and 389-ds version is 1.3.4
>>>>>
>>>>> ldapsearch shows the values for
>>>>> nsslapd-cachememsize updated to 200MB.
>>>>>
>>>>> ldapsearch -LLL -o ldif-wrap=no -D "cn=directory
>>>>> manager" -w 'mypassword' -b 'cn=userRoot,cn=ldbm
>>>>> database,cn=plugins,cn=config'|grep
>>>>> nsslapd-cachememsize
>>>>> nsslapd-cachememsize: 209715200
>>>>>
>>>>>
>>>>> So, it seems to have updated though seeing that
>>>>> warning(WARNING: ipaca: entry cache size 10485760B
>>>>> is less than db size 11599872B) in the log
>>>>> confuses me a bit.
>>>>>
>>>>> Thers one more entry that I found from the
>>>>> ldapsearch to be bit low
>>>>>
>>>>> nsslapd-dncachememsize: 10485760
>>>>> maxdncachesize: 10485760
>>>>>
>>>>> Should I update these as well to a higher value
>>>>>
>>>>> At the time when the issue happened, the memory
>>>>> usage as well as the overall load of the system
>>>>> was very low .
>>>>> I will try reproducing the issue atleast in my QA
>>>>> env..probably by trying to mock simultaneous
>>>>> parallel logins to a large number of hosts
>>>>
>>>> To monitor your cache sizes, please use the
>>>> dbmon.sh tool provided with your distro. If that
>>>> is not available with your particular distro, see
>>>> https://github.com/richm/scripts/wiki/dbmon.sh
>>>>
>>>>
>>>>>
>>>>>
>>>>> thanks
>>>>> Rakesh
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> On Mon, Aug 29, 2016 at 8:16 PM, thierry bordaz
>>>>> <tbordaz at redhat.com> wrote:
>>>>>
>>>>> Hi Rakesh,
>>>>>
>>>>> Those tuning may depend on the memory
>>>>> available on your machine.
>>>>> nsslapd-cachememsize allows the entry cache to
>>>>> consume up to 200Mb but its memory footprint
>>>>> is known to go above.
>>>>> 200Mb both looks pretty good to me. How large
>>>>> is your machine ? What is your version of 389-ds ?
>>>>>
>>>>> Those warnings do not change your settings. It
>>>>> just raise that entry cache of 'ipaca' and
>>>>> 'retrocl' are small but it is fine. The size
>>>>> of the entry cache is important mostly in
>>>>> userRoot.
>>>>> You may double check the actual values, after
>>>>> restart, with ldapsearch on
>>>>> 'cn=userRoot,cn=ldbm
>>>>> database,cn=plugins,cn=config' and
>>>>> 'cn=config,cn=ldbm database,cn=plugins,cn=config'.
>>>>>
>>>>> A step is to know what will be response time
>>>>> of DS to know if it is responsible of the hang
>>>>> or not.
>>>>> The logs and possibly pstack during those
>>>>> intermittent hangs will help to determine that.
>>>>>
>>>>> regards
>>>>> thierry
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> On 08/29/2016 04:25 PM, Rakesh Rajasekharan wrote:
>>>>>> I tried increasing the nsslapd-dbcachesize
>>>>>> and nsslapd-cachememsize in my QA envs to 200MB.
>>>>>>
>>>>>> However, in my log files, I still see this
>>>>>> message
>>>>>> [29/Aug/2016:04:34:37 +0000] - WARNING:
>>>>>> ipaca: entry cache size 10485760B is less
>>>>>> than db size 11599872B; We recommend to
>>>>>> increase the entry cache size
>>>>>> nsslapd-cachememsize.
>>>>>> [29/Aug/2016:04:34:37 +0000] - WARNING:
>>>>>> changelog: entry cache size 2097152B is less
>>>>>> than db size 441647104B; We recommend to
>>>>>> increase the entry cache size
>>>>>> nsslapd-cachememsize.
>>>>>>
>>>>>> these are my ldif files that i used to modify
>>>>>> the values
>>>>>> modify entry cache size
>>>>>> cat modify-cache-mem-size.ldif
>>>>>> dn: cn=userRoot,cn=ldbm
>>>>>> database,cn=plugins,cn=config
>>>>>> changetype: modify
>>>>>> replace: nsslapd-cachememsize
>>>>>> nsslapd-cachememsize: 209715200
>>>>>>
>>>>>> modify db cache size
>>>>>> cat modfy-db-cache-size.ldif
>>>>>> dn: cn=config,cn=ldbm
>>>>>> database,cn=plugins,cn=config
>>>>>> changetype: modify
>>>>>> replace: nsslapd-dbcachesize
>>>>>> nsslapd-dbcachesize: 209715200
>>>>>>
>>>>>> After modifying , i restarted IPA services
>>>>>>
>>>>>> Is there anything else that I need to take
>>>>>> care of as the logs suggest its still not
>>>>>> getting the updated values
>>>>>>
>>>>>> Thanks
>>>>>> Rakesh
>>>>>>
>>>>>> On Mon, Aug 29, 2016 at 6:07 PM, Rakesh
>>>>>> Rajasekharan <rakesh.rajasekharan at gmail.com>
>>>>>> wrote:
>>>>>>
>>>>>> Hi Thierry,
>>>>>>
>>>>>> Coz of the issues we had to revert back
>>>>>> to earlier running openldap in production.
>>>>>>
>>>>>> I have now done a few TCP related changes
>>>>>> in sysctl.conf and have also increased
>>>>>> the nsslapd-dbcachesize and
>>>>>> nsslapd-cachememsize to 200MB
>>>>>>
>>>>>> I will again start migrating hosts back
>>>>>> to IPA and see if I face the earlier issue.
>>>>>>
>>>>>> I will update back once I have something
>>>>>>
>>>>>>
>>>>>> Thanks,
>>>>>> Rakesh
>>>>>>
>>>>>>
>>>>>>
>>>>>> On Thu, Aug 25, 2016 at 2:17 PM, thierry
>>>>>> bordaz <tbordaz at redhat.com> wrote:
>>>>>>
>>>>>>
>>>>>>
>>>>>> On 08/25/2016 10:15 AM, Rakesh
>>>>>> Rajasekharan wrote:
>>>>>>> All of the troubleshooting seems fine.
>>>>>>>
>>>>>>>
>>>>>>> However, Running libconv.pl
>>>>>>> <http://libconv.pl> gives me this output
>>>>>>>
>>>>>>> ----- Recommendations -----
>>>>>>>
>>>>>>> 1. You have unindexed components,
>>>>>>> this can be caused from a search on
>>>>>>> an unindexed attribute, or your
>>>>>>> returned results exceeded the
>>>>>>> allidsthreshold. Unindexed
>>>>>>> components are not recommended. To
>>>>>>> refuse unindexed searches, switch
>>>>>>> 'nsslapd-require-index' to 'on'
>>>>>>> under your database entry (e.g.
>>>>>>> cn=UserRoot,cn=ldbm
>>>>>>> database,cn=plugins,cn=config).
>>>>>>>
>>>>>>> 2. You have a significant
>>>>>>> difference between binds and
>>>>>>> unbinds. You may want to investigate
>>>>>>> this difference.
>>>>>>>
>>>>>>>
>>>>>>> I feel, this could be a pointer to
>>>>>>> things going slow.. and IPA hanging.
>>>>>>> I think i now have something that I
>>>>>>> can try and nail down this issue.
>>>>>>>
>>>>>>> On a sidenote, I was earlier running
>>>>>>> openldap and migrated over to Freeipa,
>>>>>>>
>>>>>>> Thanks
>>>>>>> Rakesh
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On Wed, Aug 24, 2016 at 12:38 PM,
>>>>>>> Petr Spacek <pspacek at redhat.com> wrote:
>>>>>>>
>>>>>>> On 23.8.2016 18:44, Rakesh
>>>>>>> Rajasekharan wrote:
>>>>>>> > I think thers something
>>>>>>> seriously wrong with my system
>>>>>>> >
>>>>>>> > not able to run any IPA commands
>>>>>>> >
>>>>>>> > klist
>>>>>>> > Ticket cache:
>>>>>>> KEYRING:persistent:0:0
>>>>>>> > Default principal: admin at XYZ.COM
>>>>>>> >
>>>>>>> > Valid starting Expires
>>>>>>> Service principal
>>>>>>> > 2016-08-23T16:26:36
>>>>>>> 2016-08-24T16:26:22
>>>>>>> krbtgt/XYZ.COM at XYZ.COM
>>>>>>> >
>>>>>>> >
>>>>>>> > [root at prod-ipa-master-1a :~]
>>>>>>> ipactl status
>>>>>>> > Directory Service: RUNNING
>>>>>>> > krb5kdc Service: RUNNING
>>>>>>> > kadmin Service: RUNNING
>>>>>>> > ipa_memcached Service: RUNNING
>>>>>>> > httpd Service: RUNNING
>>>>>>> > pki-tomcatd Service: RUNNING
>>>>>>> > ipa-otpd Service: RUNNING
>>>>>>> > ipa: INFO: The ipactl command
>>>>>>> was successful
>>>>>>> >
>>>>>>> >
>>>>>>> >
>>>>>>> > [root at prod-ipa-master :~] ipa
>>>>>>> user-find p-testuser
>>>>>>> > ipa: ERROR: Kerberos error:
>>>>>>> ('Unspecified GSS failure. Minor
>>>>>>> code may
>>>>>>> > provide more information',
>>>>>>> 851968)/("Cannot contact any KDC
>>>>>>> for realm '
>>>>>>> > XYZ.COM <http://XYZ.COM>'",
>>>>>>> -1765328228)
>>>>>>>
>>>>>>
>>>>>> Hi Rakesh,
>>>>>>
>>>>>> Having a reproducible test case
>>>>>> would you rerun the command above.
>>>>>> During its processing you may
>>>>>> monitor DS process load (top). If
>>>>>> it is high, you may get some
>>>>>> pstacks of it.
>>>>>> Also would you attach the part of
>>>>>> DS access logs taken during the
>>>>>> command.
>>>>>>
>>>>>> regards
>>>>>> thierry
>>>>>>
>>>>>>> >
>>>>>>>
>>>>>>> This is weird because the server
>>>>>>> seems to be up.
>>>>>>>
>>>>>>> Please follow
>>>>>>> http://www.freeipa.org/page/Troubleshooting#Authentication.2FKerberos
>>>>>>>
>>>>>>> Petr^2 Spacek
>>>>>>>
>>>>>>> >
>>>>>>> >
>>>>>>> > Thanks
>>>>>>> >
>>>>>>> > Rakesh
>>>>>>> >
>>>>>>> > On Tue, Aug 23, 2016 at 10:01
>>>>>>> PM, Rakesh Rajasekharan <
>>>>>>> > rakesh.rajasekharan at gmail.com>
>>>>>>> wrote:
>>>>>>> >
>>>>>>> >> i changed the loggin level to
>>>>>>> 4 . Modifying
>>>>>>> nsslapd-accesslog-level
>>>>>>> >>
>>>>>>> >> But, the hang is still there.
>>>>>>> though I dont see the sigfault now
>>>>>>> >>
>>>>>>> >>
>>>>>>> >>
>>>>>>> >>
>>>>>>> >> On Tue, Aug 23, 2016 at 9:02
>>>>>>> PM, Rakesh Rajasekharan <
>>>>>>> >>
>>>>>>> rakesh.rajasekharan at gmail.com>
>>>>>>> wrote:
>>>>>>> >>
>>>>>>> >>> My disk was getting filled
>>>>>>> too fast
>>>>>>> >>>
>>>>>>> >>> logs under /var/log/dirsrv
>>>>>>> was coming around 5 gb quickly
>>>>>>> filling up
>>>>>>> >>>
>>>>>>> >>> Is there a way to make the
>>>>>>> logging less verbose
>>>>>>> >>>
>>>>>>> >>>
>>>>>>> >>>
>>>>>>> >>> On Tue, Aug 23, 2016 at 6:41
>>>>>>> PM, Petr Spacek
>>>>>>> <pspacek at redhat.com> wrote:
>>>>>>> >>>
>>>>>>> >>>> On 23.8.2016 15:07, Rakesh
>>>>>>> Rajasekharan wrote:
>>>>>>> >>>>> I was able to fix that may
>>>>>>> be temporarily... when i checked the
>>>>>>> >>>> network..
>>>>>>> >>>>> there was another process
>>>>>>> that was running and consuming a
>>>>>>> lot of
>>>>>>> >>>> network (
>>>>>>> >>>>> i have no idea who did
>>>>>>> that. I need to seriously start
>>>>>>> restricting
>>>>>>> >>>> people
>>>>>>> >>>>> access to this machine )
>>>>>>> >>>>>
>>>>>>> >>>>> after killing that
>>>>>>> perfomance improved drastically
>>>>>>> >>>>>
>>>>>>> >>>>> But now, suddenly I
>>>>>>> started experiencing the same hang.
>>>>>>> >>>>>
>>>>>>> >>>>> This time , I gert the
>>>>>>> following error when checked dmesg
>>>>>>> >>>>>
>>>>>>> >>>>> [ 301.236976]
>>>>>>> ns-slapd[3124]: segfault at 0 ip
>>>>>>> 00007f1de416951c sp
>>>>>>> >>>>> 00007f1dee1dba70 error 4
>>>>>>> in
>>>>>>> libcos-plugin.so[7f1de4166000+b000]
>>>>>>> >>>>> [ 1116.248431] TCP:
>>>>>>> request_sock_TCP: Possible SYN
>>>>>>> flooding on port 88.
>>>>>>> >>>>> Sending cookies. Check
>>>>>>> SNMP counters.
>>>>>>> >>>>> [11831.397037]
>>>>>>> ns-slapd[22550]: segfault at 0
>>>>>>> ip 00007f533d82251c sp
>>>>>>> >>>>> 00007f5347894a70 error 4
>>>>>>> in
>>>>>>> libcos-plugin.so[7f533d81f000+b000]
>>>>>>> >>>>> [11832.727989]
>>>>>>> ns-slapd[22606]: segfault at 0
>>>>>>> ip 00007f6231eb951c sp
>>>>>>> >>>>> 00007f623bf2ba70 error 4
>>>>>>> in libcos-plugin.so[7f6231eb6000+b00
>>>>>>> >>>>
>>>>>>> >>>> Okay, this one is serious.
>>>>>>> The LDAP server crashed.
>>>>>>> >>>>
>>>>>>> >>>> 1. Make sure all your
>>>>>>> packages are up-to-date.
>>>>>>> >>>>
>>>>>>> >>>> Please see
>>>>>>> >>>>
>>>>>>> http://directory.fedoraproject.org/docs/389ds/FAQ/faq.html#d
>>>>>>> >>>> ebugging-crashes
>>>>>>> >>>> for further instructions
>>>>>>> how to debug this.
>>>>>>> >>>>
>>>>>>> >>>> Petr^2 Spacek
>>>>>>> >>>>
>>>>>>> >>>>>
>>>>>>> >>>>> and in
>>>>>>> /var/log/dirsrv/example-com/errors
>>>>>>> >>>>>
>>>>>>> >>>>> [23/Aug/2016:12:49:36
>>>>>>> +0000] DSRetroclPlugin -
>>>>>>> delete_changerecord:
>>>>>>> >>>> could
>>>>>>> >>>>> not delete change record
>>>>>>> 3291138 (rc: 32)
>>>>>>> >>>>> [23/Aug/2016:12:49:36
>>>>>>> +0000] DSRetroclPlugin -
>>>>>>> delete_changerecord:
>>>>>>> >>>> could
>>>>>>> >>>>> not delete change record
>>>>>>> 3291139 (rc: 32)
>>>>>>> >>>>> [23/Aug/2016:12:49:36
>>>>>>> +0000] DSRetroclPlugin -
>>>>>>> delete_changerecord:
>>>>>>> >>>> could
>>>>>>> >>>>> not delete change record
>>>>>>> 3291140 (rc: 32)
>>>>>>> >>>>> [23/Aug/2016:12:49:36
>>>>>>> +0000] DSRetroclPlugin -
>>>>>>> delete_changerecord:
>>>>>>> >>>> could
>>>>>>> >>>>> not delete change record
>>>>>>> 3291141 (rc: 32)
>>>>>>> >>>>> [23/Aug/2016:12:49:36
>>>>>>> +0000] DSRetroclPlugin -
>>>>>>> delete_changerecord:
>>>>>>> >>>> could
>>>>>>> >>>>> not delete change record
>>>>>>> 3291142 (rc: 32)
>>>>>>> >>>>> [23/Aug/2016:12:49:36
>>>>>>> +0000] DSRetroclPlugin -
>>>>>>> delete_changerecord:
>>>>>>> >>>> could
>>>>>>> >>>>> not delete change record
>>>>>>> 3291143 (rc: 32)
>>>>>>> >>>>> [23/Aug/2016:12:49:36
>>>>>>> +0000] DSRetroclPlugin -
>>>>>>> delete_changerecord:
>>>>>>> >>>> could
>>>>>>> >>>>> not delete change record
>>>>>>> 3291144 (rc: 32)
>>>>>>> >>>>> [23/Aug/2016:12:49:36
>>>>>>> +0000] DSRetroclPlugin -
>>>>>>> delete_changerecord:
>>>>>>> >>>> could
>>>>>>> >>>>> not delete change record
>>>>>>> 3291145 (rc: 32)
>>>>>>> >>>>> [23/Aug/2016:12:49:50
>>>>>>> +0000] - Retry count exceeded in
>>>>>>> delete
>>>>>>> >>>>> [23/Aug/2016:12:49:50
>>>>>>> +0000] DSRetroclPlugin -
>>>>>>> delete_changerecord:
>>>>>>> >>>> could
>>>>>>> >>>>> not delete change record
>>>>>>> 3292734 (rc: 51)
>>>>>>> >>>>>
>>>>>>> >>>>>
>>>>>>> >>>>> Can i do something about
>>>>>>> this error.. I treid to restart
>>>>>>> ipa a couple
>>>>>>> >>>> of
>>>>>>> >>>>> time but that did not help
>>>>>>> >>>>>
>>>>>>> >>>>> Thanks
>>>>>>> >>>>> Rakesh
>>>>>>> >>>>>
>>>>>>> >>>>> On Mon, Aug 22, 2016 at
>>>>>>> 2:27 PM, Petr Spacek
>>>>>>> <pspacek at redhat.com>
>>>>>>> >>>> wrote:
>>>>>>> >>>>>
>>>>>>> >>>>>> On 19.8.2016 19:32,
>>>>>>> Rakesh Rajasekharan wrote:
>>>>>>> >>>>>>> I am running my set up
>>>>>>> on AWS cloud, and entropy is low
>>>>>>> at around
>>>>>>> >>>> 180 .
>>>>>>> >>>>>>>
>>>>>>> >>>>>>> I plan to increase it bu
>>>>>>> installing haveged . But, would
>>>>>>> low entropy
>>>>>>> >>>> by
>>>>>>> >>>>>> any
>>>>>>> >>>>>>> chance cause this issue
>>>>>>> of intermittent hang .
>>>>>>> >>>>>>> Also, the hang is mostly
>>>>>>> observed when registering around
>>>>>>> 20 clients
>>>>>>> >>>>>>> together
>>>>>>> >>>>>>
>>>>>>> >>>>>> Possibly, I'm not sure.
>>>>>>> If you want to dig into this, I
>>>>>>> would do this:
>>>>>>> >>>>>> 1. look what process
>>>>>>> hangs on client (using pstree
>>>>>>> command or so)
>>>>>>> >>>>>> $ pstree
>>>>>>> >>>>>>
>>>>>>> >>>>>> 2. look to what server
>>>>>>> and port is the hanging client
>>>>>>> connected to
>>>>>>> >>>>>> $ lsof -p <PID of the
>>>>>>> hanging process>
>>>>>>> >>>>>>
>>>>>>> >>>>>> 3. jump to server and see
>>>>>>> what process is bound to the
>>>>>>> target port
>>>>>>> >>>>>> $ netstat -pn
>>>>>>> >>>>>>
>>>>>>> >>>>>> 4. see where the process
>>>>>>> if hanging
>>>>>>> >>>>>> $ strace -p <PID of the
>>>>>>> hanging process>
>>>>>>> >>>>>>
>>>>>>> >>>>>> I hope it helps.
>>>>>>> >>>>>>
>>>>>>> >>>>>> Petr^2 Spacek
>>>>>>> >>>>>>
>>>>>>> >>>>>>> On Fri, Aug 19, 2016 at
>>>>>>> 7:24 PM, Rakesh Rajasekharan <
>>>>>>> >>>>>>>
>>>>>>> rakesh.rajasekharan at gmail.com>
>>>>>>> wrote:
>>>>>>> >>>>>>>
>>>>>>> >>>>>>>> yes there seems to be
>>>>>>> something thats worrying.. I
>>>>>>> have faced this
>>>>>>> >>>> today
>>>>>>> >>>>>>>> as well.
>>>>>>> >>>>>>>> There are few hosts
>>>>>>> around 280 odd left and when i
>>>>>>> try adding them
>>>>>>> >>>> to
>>>>>>> >>>>>> IPA
>>>>>>> >>>>>>>> , the slowness begins..
>>>>>>> >>>>>>>>
>>>>>>> >>>>>>>> all the ipa commands
>>>>>>> like ipa user-find.. etc becomes
>>>>>>> very slow in
>>>>>>> >>>>>>>> responding.
>>>>>>> >>>>>>>>
>>>>>>> >>>>>>>> the SYNC_RECV are not
>>>>>>> many though just around 80-90
>>>>>>> and today that
>>>>>>> >>>> was
>>>>>>> >>>>>>>> around 20 only
>>>>>>> >>>>>>>>
>>>>>>> >>>>>>>>
>>>>>>> >>>>>>>> I have for now
>>>>>>> increased tcp_max_syn_backlog to
>>>>>>> 5000.
>>>>>>> >>>>>>>> For now the slowness
>>>>>>> seems to have gone.. but I will
>>>>>>> do a try
>>>>>>> >>>> adding the
>>>>>>> >>>>>>>> clients again tomorrow
>>>>>>> and see how it goes
>>>>>>> >>>>>>>>
>>>>>>> >>>>>>>> Thanks
>>>>>>> >>>>>>>> Rakesh
>>>>>>> >>>>>>>>
>>>>>>> >>>>>>>> The issues
>>>>>>> >>>>>>>>
>>>>>>> >>>>>>>> On Fri, Aug 19, 2016 at
>>>>>>> 12:58 PM, Petr Spacek
>>>>>>> <pspacek at redhat.com>
>>>>>>> >>>>>> wrote:
>>>>>>> >>>>>>>>
>>>>>>> >>>>>>>>> On 18.8.2016 17:23,
>>>>>>> Rakesh Rajasekharan wrote:
>>>>>>> >>>>>>>>>> Hi
>>>>>>> >>>>>>>>>>
>>>>>>> >>>>>>>>>> I am migrating to
>>>>>>> freeipa from openldap and have
>>>>>>> around 4000
>>>>>>> >>>> clients
>>>>>>> >>>>>>>>>>
>>>>>>> >>>>>>>>>> I had openned a
>>>>>>> another thread on that, but
>>>>>>> chose to start a new
>>>>>>> >>>> one
>>>>>>> >>>>>>>>> here
>>>>>>> >>>>>>>>>> as its a separate issue
>>>>>>> >>>>>>>>>>
>>>>>>> >>>>>>>>>> I was able to change
>>>>>>> the nssslapd-maxdescriptors
>>>>>>> adding an ldif
>>>>>>> >>>> file
>>>>>>> >>>>>>>>>>
>>>>>>> >>>>>>>>>> cat nsslapd-modify.ldif
>>>>>>> >>>>>>>>>> dn: cn=config
>>>>>>> >>>>>>>>>> changetype: modify
>>>>>>> >>>>>>>>>> replace:
>>>>>>> nsslapd-maxdescriptors
>>>>>>> >>>>>>>>>>
>>>>>>> nsslapd-maxdescriptors: 17000
>>>>>>> >>>>>>>>>>
>>>>>>> >>>>>>>>>> and running the
>>>>>>> ldapmodify command
>>>>>>> >>>>>>>>>>
>>>>>>> >>>>>>>>>> I have now started
>>>>>>> moving clients running an
>>>>>>> openldap to Freeipa
>>>>>>> >>>> and
>>>>>>> >>>>>>>>> have
>>>>>>> >>>>>>>>>> today moved close to
>>>>>>> 2000 clients
>>>>>>> >>>>>>>>>>
>>>>>>> >>>>>>>>>> However, I have
>>>>>>> noticed that IPA hangs
>>>>>>> intermittently.
>>>>>>> >>>>>>>>>>
>>>>>>> >>>>>>>>>> running a kinit admin
>>>>>>> returns the below error
>>>>>>> >>>>>>>>>> kinit: Generic error
>>>>>>> (see e-text) while getting initial
>>>>>>> >>>> credentials
>>>>>>> >>>>>>>>>>
>>>>>>> >>>>>>>>>> from the
>>>>>>> /var/log/messages, I see this entry
>>>>>>> >>>>>>>>>>
>>>>>>> >>>>>>>>>> prod-ipa-master-int
>>>>>>> kernel: [104090.315801] TCP:
>>>>>>> >>>> request_sock_TCP:
>>>>>>> >>>>>>>>>> Possible SYN flooding
>>>>>>> on port 88. Sending cookies.
>>>>>>> Check SNMP
>>>>>>> >>>>>> counters.
>>>>>>> >>>>>>>>>
>>>>>>> >>>>>>>>> I would be worried
>>>>>>> about this message. Maybe
>>>>>>> kernel/firewall is
>>>>>>> >>>> doing
>>>>>>> >>>>>>>>> something fishy behind
>>>>>>> your back and blocking some
>>>>>>> connections or
>>>>>>> >>>> so.
>>>>>>> >>>>>>>>>
>>>>>>> >>>>>>>>> Petr^2 Spacek
>>>>>>> >>>>>>>>>
>>>>>>> >>>>>>>>>
>>>>>>> >>>>>>>>>> Aug 18 13:00:01
>>>>>>> prod-ipa-master-int systemd[1]:
>>>>>>> Started Session
>>>>>>> >>>> 4885
>>>>>>> >>>>>> of
>>>>>>> >>>>>>>>>> user root.
>>>>>>> >>>>>>>>>> Aug 18 13:00:01
>>>>>>> prod-ipa-master-int systemd[1]:
>>>>>>> Starting Session
>>>>>>> >>>> 4885
>>>>>>> >>>>>> of
>>>>>>> >>>>>>>>>> user root.
>>>>>>> >>>>>>>>>> Aug 18 13:01:01
>>>>>>> prod-ipa-master-int systemd[1]:
>>>>>>> Started Session
>>>>>>> >>>> 4886
>>>>>>> >>>>>> of
>>>>>>> >>>>>>>>>> user root.
>>>>>>> >>>>>>>>>> Aug 18 13:01:01
>>>>>>> prod-ipa-master-int systemd[1]:
>>>>>>> Starting Session
>>>>>>> >>>> 4886
>>>>>>> >>>>>> of
>>>>>>> >>>>>>>>>> user root.
>>>>>>> >>>>>>>>>> Aug 18 13:02:40
>>>>>>> prod-ipa-master-int
>>>>>>> python[28984]: ansible-command
>>>>>>> >>>>>>>>> Invoked
>>>>>>> >>>>>>>>>> with creates=None
>>>>>>> executable=None shell=True args=
>>>>>>> removes=None
>>>>>>> >>>>>>>>> warn=True
>>>>>>> >>>>>>>>>> chdir=None
>>>>>>> >>>>>>>>>> Aug 18 13:04:37
>>>>>>> prod-ipa-master-int sssd_be:
>>>>>>> GSSAPI Error:
>>>>>>> >>>> Unspecified
>>>>>>> >>>>>>>>> GSS
>>>>>>> >>>>>>>>>> failure. Minor code
>>>>>>> may provide more information
>>>>>>> (KDC returned
>>>>>>> >>>> error
>>>>>>> >>>>>>>>>> string: PROCESS_TGS)
>>>>>>> >>>>>>>>>>
>>>>>>> >>>>>>>>>> Could it be possible
>>>>>>> that its due to the initial load
>>>>>>> of adding
>>>>>>> >>>> the
>>>>>>> >>>>>>>>> clients
>>>>>>> >>>>>>>>>> or is there something
>>>>>>> else that I need to take care of.
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>
>>>>
>>>> --
>>>> Manage your subscription for the Freeipa-users
>>>> mailing list:
>>>> https://www.redhat.com/mailman/listinfo/freeipa-users
>>>> Go to http://freeipa.org for more info on the project
>>>>
>>>>
>>>>
>>>>
>>>>
>>>
>>>
>>
>>
>
--
Red Hat GmbH, http://www.de.redhat.com/, Registered seat: Grasbrunn,
Commercial register: Amtsgericht Muenchen, HRB 153243,
Managing Directors: Charles Cachera, Michael Cunningham, Michael O'Neill, Eric Shander
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/freeipa-users/attachments/20160905/5ce42b72/attachment.htm>
More information about the Freeipa-users
mailing list