[Freeipa-users] Freeipa 4.2.0 hangs intermittently
thierry bordaz
tbordaz at redhat.com
Mon Sep 5 12:31:23 UTC 2016
Hi Rakesh,
Thanks you soooo much for you continuous help on this !!
If the hang is still going on, would you dump the DB lock information with:
db_stat -CA -N -h /var/lib/dirsrv/slapd-<instance>/db
>/tmp/db_lock_info
It is looking like several threads are waiting for one/several db locks.
With the lock info we will know which thread acquired it/them.
A guess is that it could be the checkpointing thread but this one is
also waiting (on a memory pool).
I am copying Ludwig on this thread in case this pstack rings any bell.
best regards
thierry
On 09/05/2016 02:05 PM, Rakesh Rajasekharan wrote:
> Hi Thierry,
>
> I was able to now reporduce the issue..
>
> the ns-slapd process just got hung,.. however, no core file was generate..
>
> I manually created a coredump using gcore and then took a strace of it
> this is strace from the coredump generated
>
> GNU gdb (GDB) Red Hat Enterprise Linux 7.6.1-80.el7
> Copyright (C) 2013 Free Software Foundation, Inc.
> License GPLv3+: GNU GPL version 3 or later
> <http://gnu.org/licenses/gpl.html>
> This is free software: you are free to change and redistribute it.
> There is NO WARRANTY, to the extent permitted by law. Type "show copying"
> and "show warranty" for details.
> This GDB was configured as "x86_64-redhat-linux-gnu".
> For bug reporting instructions, please see:
> <http://www.gnu.org/software/gdb/bugs/>...
> Reading symbols from /usr/sbin/ns-slapd...Reading symbols from
> /usr/sbin/ns-slapd...(no debugging symbols found)...done.
> (no debugging symbols found)...done.
> [New LWP 105159]
> [New LWP 105160]
> [New LWP 105161]
> [New LWP 105162]
> [New LWP 105163]
> [New LWP 105164]
> [New LWP 105165]
> [New LWP 105166]
> [New LWP 105167]
> [New LWP 105168]
> [New LWP 105171]
> [New LWP 105172]
> [New LWP 105173]
> [New LWP 105174]
> [New LWP 105175]
> [New LWP 105176]
> [New LWP 105177]
> [New LWP 105178]
> [New LWP 105179]
> [New LWP 105180]
> [New LWP 105181]
> [New LWP 105182]
> [New LWP 105183]
> [New LWP 105184]
> [New LWP 105185]
> [New LWP 105186]
> [New LWP 105187]
> [New LWP 105188]
> [New LWP 105189]
> [New LWP 105190]
> [New LWP 105191]
> [New LWP 105192]
> [New LWP 105193]
> [New LWP 105194]
> [New LWP 105195]
> [New LWP 105196]
> [New LWP 105197]
> [New LWP 105198]
> [New LWP 105199]
> [New LWP 105200]
> [New LWP 105201]
> [New LWP 105203]
> [New LWP 105215]
> [New LWP 105158]
> [Thread debugging using libthread_db enabled]
> Using host libthread_db library "/lib64/libthread_db.so.1".
> Core was generated by `/usr/sbin/ns-slapd'.
> #0 0x00007f1b48ae69b3 in select () from /lib64/libc.so.6
>
> Thread 44 (Thread 0x7f1b4b6d9840 (LWP 105158)):
> #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 () from
> /lib64/libpthread.so.0
> No symbol table info available.
> #1 0x00007f1b4941b2c3 in PR_EnterMonitor () from /lib64/libnspr4.so
> No symbol table info available.
> #2 0x00007f1b4b7125ed in setup_pr_read_pds ()
> No symbol table info available.
> #3 0x00007f1b4b715fd9 in slapd_daemon ()
> No symbol table info available.
> #4 0x00007f1b4b707c6c in main ()
> No symbol table info available.
>
> Thread 43 (Thread 0x7f1a5df0f700 (LWP 105215)):
> #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 () from
> /lib64/libpthread.so.0
> No symbol table info available.
> #1 0x00007f1b4941b050 in PR_WaitCondVar () from /lib64/libnspr4.so
> No symbol table info available.
> #2 0x00007f1b4b71df25 in ps_send_results ()
> No symbol table info available.
> #3 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so
> No symbol table info available.
> #4 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0
> No symbol table info available.
> #5 0x00007f1b48aef28d in clone () from /lib64/libc.so.6
> No symbol table info available.
>
> Thread 42 (Thread 0x7f1b03fe7700 (LWP 105203)):
> #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 () from
> /lib64/libpthread.so.0
> No symbol table info available.
> #1 0x00007f1b4941b050 in PR_WaitCondVar () from /lib64/libnspr4.so
> No symbol table info available.
> #2 0x00007f1b4b71df25 in ps_send_results ()
> No symbol table info available.
> #3 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so
> No symbol table info available.
> #4 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0
> No symbol table info available.
> #5 0x00007f1b48aef28d in clone () from /lib64/libc.so.6
> No symbol table info available.
>
> Thread 41 (Thread 0x7f1b047e8700 (LWP 105201)):
> #0 0x00007f1b48ae69b3 in select () from /lib64/libc.so.6
> No symbol table info available.
> #1 0x00007f1b4b2710e9 in DS_Sleep () from /usr/lib64/dirsrv/libslapd.so.0
> No symbol table info available.
> #2 0x00007f1b4b712325 in time_thread ()
> No symbol table info available.
> #3 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so
> No symbol table info available.
> #4 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0
> No symbol table info available.
> #5 0x00007f1b48aef28d in clone () from /lib64/libc.so.6
> No symbol table info available.
>
> Thread 40 (Thread 0x7f1b04fe9700 (LWP 105200)):
> #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 () from
> /lib64/libpthread.so.0
> No symbol table info available.
> #1 0x00007f1b433212f3 in __db_hybrid_mutex_suspend () from
> /lib64/libdb-5.3.so <http://libdb-5.3.so>
> No symbol table info available.
> #2 0x00007f1b43320640 in __db_tas_mutex_lock () from
> /lib64/libdb-5.3.so <http://libdb-5.3.so>
> No symbol table info available.
> #3 0x00007f1b4343ae1b in __memp_fget () from /lib64/libdb-5.3.so
> <http://libdb-5.3.so>
> No symbol table info available.
> #4 0x00007f1b4333e211 in __bam_search () from /lib64/libdb-5.3.so
> <http://libdb-5.3.so>
> No symbol table info available.
> #5 0x00007f1b43329256 in __bamc_search () from /lib64/libdb-5.3.so
> <http://libdb-5.3.so>
> No symbol table info available.
> #6 0x00007f1b4332d6c4 in __bamc_put () from /lib64/libdb-5.3.so
> <http://libdb-5.3.so>
> No symbol table info available.
> #7 0x00007f1b433e46d5 in __dbc_iput () from /lib64/libdb-5.3.so
> <http://libdb-5.3.so>
> No symbol table info available.
> #8 0x00007f1b433dfa0e in __db_put () from /lib64/libdb-5.3.so
> <http://libdb-5.3.so>
> No symbol table info available.
> #9 0x00007f1b433f4fa4 in __db_put_pp () from /lib64/libdb-5.3.so
> <http://libdb-5.3.so>
> No symbol table info available.
> #10 0x00007f1b3e8cf4cf in idl_new_insert_key () from
> /usr/lib64/dirsrv/plugins/libback-ldbm.so
> No symbol table info available.
> #11 0x00007f1b3e8de5b5 in addordel_values_sv.isra.3 () from
> /usr/lib64/dirsrv/plugins/libback-ldbm.so
> No symbol table info available.
> #12 0x00007f1b3e8df25a in index_addordel_values_ext_sv () from
> /usr/lib64/dirsrv/plugins/libback-ldbm.so
> No symbol table info available.
> #13 0x00007f1b3e8df694 in index_addordel_values_sv () from
> /usr/lib64/dirsrv/plugins/libback-ldbm.so
> No symbol table info available.
> #14 0x00007f1b3e8df75a in index_addordel_entry () from
> /usr/lib64/dirsrv/plugins/libback-ldbm.so
> No symbol table info available.
> #15 0x00007f1b3e8e25fa in ldbm_back_add () from
> /usr/lib64/dirsrv/plugins/libback-ldbm.so
> No symbol table info available.
> #16 0x00007f1b4b1e4cca in op_shared_add () from
> /usr/lib64/dirsrv/libslapd.so.0
> No symbol table info available.
> #17 0x00007f1b4b1e6020 in do_add () from /usr/lib64/dirsrv/libslapd.so.0
> No symbol table info available.
> #18 0x00007f1b4b711303 in connection_threadmain ()
> No symbol table info available.
> #19 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so
> No symbol table info available.
> #20 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0
> No symbol table info available.
> #21 0x00007f1b48aef28d in clone () from /lib64/libc.so.6
> No symbol table info available.
>
> Thread 39 (Thread 0x7f1b057ea700 (LWP 105199)):
> #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 () from
> /lib64/libpthread.so.0
> No symbol table info available.
> #1 0x00007f1b4941b050 in PR_WaitCondVar () from /lib64/libnspr4.so
> No symbol table info available.
> #2 0x00007f1b4b70efee in connection_wait_for_new_work ()
> No symbol table info available.
> #3 0x00007f1b4b710719 in connection_threadmain ()
> No symbol table info available.
> #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so
> No symbol table info available.
> #5 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0
> No symbol table info available.
> #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6
> No symbol table info available.
>
> Thread 38 (Thread 0x7f1b05feb700 (LWP 105198)):
> #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 () from
> /lib64/libpthread.so.0
> No symbol table info available.
> #1 0x00007f1b4941b050 in PR_WaitCondVar () from /lib64/libnspr4.so
> No symbol table info available.
> #2 0x00007f1b4b70efee in connection_wait_for_new_work ()
> No symbol table info available.
> #3 0x00007f1b4b710719 in connection_threadmain ()
> No symbol table info available.
> #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so
> No symbol table info available.
> #5 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0
> No symbol table info available.
> #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6
> No symbol table info available.
>
> Thread 37 (Thread 0x7f1b067ec700 (LWP 105197)):
> #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 () from
> /lib64/libpthread.so.0
> No symbol table info available.
> #1 0x00007f1b4941b050 in PR_WaitCondVar () from /lib64/libnspr4.so
> No symbol table info available.
> #2 0x00007f1b4b70efee in connection_wait_for_new_work ()
> No symbol table info available.
> #3 0x00007f1b4b710719 in connection_threadmain ()
> No symbol table info available.
> #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so
> No symbol table info available.
> #5 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0
> No symbol table info available.
> #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6
> No symbol table info available.
>
> Thread 36 (Thread 0x7f1b06fed700 (LWP 105196)):
> #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 () from
> /lib64/libpthread.so.0
> No symbol table info available.
> #1 0x00007f1b4941b050 in PR_WaitCondVar () from /lib64/libnspr4.so
> No symbol table info available.
> #2 0x00007f1b4b70efee in connection_wait_for_new_work ()
> No symbol table info available.
> #3 0x00007f1b4b710719 in connection_threadmain ()
> No symbol table info available.
> #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so
> No symbol table info available.
> #5 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0
> No symbol table info available.
> #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6
> No symbol table info available.
>
> Thread 35 (Thread 0x7f1b077ee700 (LWP 105195)):
> #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 () from
> /lib64/libpthread.so.0
> No symbol table info available.
> #1 0x00007f1b4941b2c3 in PR_EnterMonitor () from /lib64/libnspr4.so
> No symbol table info available.
> #2 0x00007f1b3e8c23e6 in dblayer_txn_begin () from
> /usr/lib64/dirsrv/plugins/libback-ldbm.so
> No symbol table info available.
> #3 0x00007f1b3e8fe007 in ldbm_back_modify () from
> /usr/lib64/dirsrv/plugins/libback-ldbm.so
> No symbol table info available.
> #4 0x00007f1b4b22b541 in op_shared_modify () from
> /usr/lib64/dirsrv/libslapd.so.0
> No symbol table info available.
> #5 0x00007f1b4b22c004 in modify_internal_pb () from
> /usr/lib64/dirsrv/libslapd.so.0
> No symbol table info available.
> #6 0x00007f1b3eb48b60 in ipapwd_apply_mods () from
> /usr/lib64/dirsrv/plugins/libipa_pwd_extop.so
> No symbol table info available.
> #7 0x00007f1b3eb4c8e0 in store_new_keys () from
> /usr/lib64/dirsrv/plugins/libipa_pwd_extop.so
> No symbol table info available.
> #8 0x00007f1b3eb4ead0 in ipapwd_extop () from
> /usr/lib64/dirsrv/plugins/libipa_pwd_extop.so
> No symbol table info available.
> #9 0x00007f1b4b23bba2 in plugin_call_exop_plugins () from
> /usr/lib64/dirsrv/libslapd.so.0
> No symbol table info available.
> #10 0x00007f1b4b717efe in do_extended ()
> No symbol table info available.
> #11 0x00007f1b4b71141a in connection_threadmain ()
> No symbol table info available.
> #12 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so
> No symbol table info available.
> #13 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0
> No symbol table info available.
> #14 0x00007f1b48aef28d in clone () from /lib64/libc.so.6
> No symbol table info available.
>
> Thread 34 (Thread 0x7f1b07fef700 (LWP 105194)):
> #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 () from
> /lib64/libpthread.so.0
> No symbol table info available.
> #1 0x00007f1b4941b050 in PR_WaitCondVar () from /lib64/libnspr4.so
> No symbol table info available.
> #2 0x00007f1b4b70efee in connection_wait_for_new_work ()
> No symbol table info available.
> #3 0x00007f1b4b710719 in connection_threadmain ()
> No symbol table info available.
> #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so
> No symbol table info available.
> #5 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0
> No symbol table info available.
> #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6
> No symbol table info available.
>
> Thread 33 (Thread 0x7f1b087f0700 (LWP 105193)):
> #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 () from
> /lib64/libpthread.so.0
> No symbol table info available.
> #1 0x00007f1b4941b050 in PR_WaitCondVar () from /lib64/libnspr4.so
> No symbol table info available.
> #2 0x00007f1b4b70efee in connection_wait_for_new_work ()
> No symbol table info available.
> #3 0x00007f1b4b710719 in connection_threadmain ()
> No symbol table info available.
> #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so
> No symbol table info available.
> #5 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0
> No symbol table info available.
> #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6
> No symbol table info available.
>
> Thread 32 (Thread 0x7f1b08ff1700 (LWP 105192)):
> #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 () from
> /lib64/libpthread.so.0
> No symbol table info available.
> #1 0x00007f1b4941b050 in PR_WaitCondVar () from /lib64/libnspr4.so
> No symbol table info available.
> #2 0x00007f1b4b70efee in connection_wait_for_new_work ()
> No symbol table info available.
> #3 0x00007f1b4b710719 in connection_threadmain ()
> No symbol table info available.
> #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so
> No symbol table info available.
> #5 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0
> No symbol table info available.
> #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6
> No symbol table info available.
>
> Thread 31 (Thread 0x7f1b097f2700 (LWP 105191)):
> #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 () from
> /lib64/libpthread.so.0
> No symbol table info available.
> #1 0x00007f1b4941b050 in PR_WaitCondVar () from /lib64/libnspr4.so
> No symbol table info available.
> #2 0x00007f1b4b70efee in connection_wait_for_new_work ()
> No symbol table info available.
> #3 0x00007f1b4b710719 in connection_threadmain ()
> No symbol table info available.
> #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so
> No symbol table info available.
> #5 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0
> No symbol table info available.
> #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6
> No symbol table info available.
>
> Thread 30 (Thread 0x7f1b09ff3700 (LWP 105190)):
> #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 () from
> /lib64/libpthread.so.0
> No symbol table info available.
> #1 0x00007f1b433212f3 in __db_hybrid_mutex_suspend () from
> /lib64/libdb-5.3.so <http://libdb-5.3.so>
> No symbol table info available.
> #2 0x00007f1b43320640 in __db_tas_mutex_lock () from
> /lib64/libdb-5.3.so <http://libdb-5.3.so>
> No symbol table info available.
> #3 0x00007f1b433cad3a in __lock_get_internal () from
> /lib64/libdb-5.3.so <http://libdb-5.3.so>
> No symbol table info available.
> #4 0x00007f1b433cb820 in __lock_get () from /lib64/libdb-5.3.so
> <http://libdb-5.3.so>
> No symbol table info available.
> #5 0x00007f1b433f7142 in __db_lget () from /lib64/libdb-5.3.so
> <http://libdb-5.3.so>
> No symbol table info available.
> #6 0x00007f1b4333e605 in __bam_search () from /lib64/libdb-5.3.so
> <http://libdb-5.3.so>
> No symbol table info available.
> #7 0x00007f1b43329256 in __bamc_search () from /lib64/libdb-5.3.so
> <http://libdb-5.3.so>
> No symbol table info available.
> #8 0x00007f1b4332ad0f in __bamc_get () from /lib64/libdb-5.3.so
> <http://libdb-5.3.so>
> No symbol table info available.
> #9 0x00007f1b433e3ca6 in __dbc_iget () from /lib64/libdb-5.3.so
> <http://libdb-5.3.so>
> No symbol table info available.
> #10 0x00007f1b433f2b02 in __dbc_get_pp () from /lib64/libdb-5.3.so
> <http://libdb-5.3.so>
> No symbol table info available.
> #11 0x00007f1b3e8ce680 in idl_new_fetch () from
> /usr/lib64/dirsrv/plugins/libback-ldbm.so
> No symbol table info available.
> #12 0x00007f1b3e8dcb26 in index_read_ext_allids () from
> /usr/lib64/dirsrv/plugins/libback-ldbm.so
> No symbol table info available.
> #13 0x00007f1b3e8c7284 in keys2idl () from
> /usr/lib64/dirsrv/plugins/libback-ldbm.so
> No symbol table info available.
> #14 0x00007f1b3e8c79e3 in ava_candidates.isra.0 () from
> /usr/lib64/dirsrv/plugins/libback-ldbm.so
> No symbol table info available.
> #15 0x00007f1b3e8c7fd2 in filter_candidates_ext () from
> /usr/lib64/dirsrv/plugins/libback-ldbm.so
> No symbol table info available.
> #16 0x00007f1b3e8c9096 in list_candidates () from
> /usr/lib64/dirsrv/plugins/libback-ldbm.so
> No symbol table info available.
> #17 0x00007f1b3e8c7f40 in filter_candidates_ext () from
> /usr/lib64/dirsrv/plugins/libback-ldbm.so
> No symbol table info available.
> #18 0x00007f1b3e8c9096 in list_candidates () from
> /usr/lib64/dirsrv/plugins/libback-ldbm.so
> No symbol table info available.
> #19 0x00007f1b3e8c7f40 in filter_candidates_ext () from
> /usr/lib64/dirsrv/plugins/libback-ldbm.so
> No symbol table info available.
> #20 0x00007f1b3e8c9096 in list_candidates () from
> /usr/lib64/dirsrv/plugins/libback-ldbm.so
> No symbol table info available.
> #21 0x00007f1b3e8c7f40 in filter_candidates_ext () from
> /usr/lib64/dirsrv/plugins/libback-ldbm.so
> No symbol table info available.
> #22 0x00007f1b3e90380c in subtree_candidates () from
> /usr/lib64/dirsrv/plugins/libback-ldbm.so
> No symbol table info available.
> #23 0x00007f1b3e904e7f in ldbm_back_search () from
> /usr/lib64/dirsrv/plugins/libback-ldbm.so
> No symbol table info available.
> #24 0x00007f1b4b2321d6 in op_shared_search () from
> /usr/lib64/dirsrv/libslapd.so.0
> No symbol table info available.
> #25 0x00007f1b4b72333e in do_search ()
> No symbol table info available.
> #26 0x00007f1b4b7113c3 in connection_threadmain ()
> No symbol table info available.
> #27 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so
> No symbol table info available.
> #28 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0
> No symbol table info available.
> #29 0x00007f1b48aef28d in clone () from /lib64/libc.so.6
> No symbol table info available.
>
> Thread 29 (Thread 0x7f1b0a7f4700 (LWP 105189)):
> #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 () from
> /lib64/libpthread.so.0
> No symbol table info available.
> #1 0x00007f1b4941b050 in PR_WaitCondVar () from /lib64/libnspr4.so
> No symbol table info available.
> #2 0x00007f1b4b70efee in connection_wait_for_new_work ()
> No symbol table info available.
> #3 0x00007f1b4b710719 in connection_threadmain ()
> No symbol table info available.
> #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so
> No symbol table info available.
> #5 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0
> No symbol table info available.
> #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6
> No symbol table info available.
>
> Thread 28 (Thread 0x7f1b0aff5700 (LWP 105188)):
> #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 () from
> /lib64/libpthread.so.0
> No symbol table info available.
> #1 0x00007f1b4941b050 in PR_WaitCondVar () from /lib64/libnspr4.so
> No symbol table info available.
> #2 0x00007f1b4b70efee in connection_wait_for_new_work ()
> No symbol table info available.
> #3 0x00007f1b4b710719 in connection_threadmain ()
> No symbol table info available.
> #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so
> No symbol table info available.
> #5 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0
> No symbol table info available.
> #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6
> No symbol table info available.
>
> Thread 27 (Thread 0x7f1b0b7f6700 (LWP 105187)):
> #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 () from
> /lib64/libpthread.so.0
> No symbol table info available.
> #1 0x00007f1b4941b050 in PR_WaitCondVar () from /lib64/libnspr4.so
> No symbol table info available.
> #2 0x00007f1b4b70efee in connection_wait_for_new_work ()
> No symbol table info available.
> #3 0x00007f1b4b710719 in connection_threadmain ()
> No symbol table info available.
> #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so
> No symbol table info available.
> #5 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0
> No symbol table info available.
> #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6
> No symbol table info available.
>
> Thread 26 (Thread 0x7f1b0bff7700 (LWP 105186)):
> #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 () from
> /lib64/libpthread.so.0
> No symbol table info available.
> #1 0x00007f1b433212f3 in __db_hybrid_mutex_suspend () from
> /lib64/libdb-5.3.so <http://libdb-5.3.so>
> No symbol table info available.
> #2 0x00007f1b43320640 in __db_tas_mutex_lock () from
> /lib64/libdb-5.3.so <http://libdb-5.3.so>
> No symbol table info available.
> #3 0x00007f1b433cad3a in __lock_get_internal () from
> /lib64/libdb-5.3.so <http://libdb-5.3.so>
> No symbol table info available.
> #4 0x00007f1b433cb820 in __lock_get () from /lib64/libdb-5.3.so
> <http://libdb-5.3.so>
> No symbol table info available.
> #5 0x00007f1b433f7142 in __db_lget () from /lib64/libdb-5.3.so
> <http://libdb-5.3.so>
> No symbol table info available.
> #6 0x00007f1b4333e605 in __bam_search () from /lib64/libdb-5.3.so
> <http://libdb-5.3.so>
> No symbol table info available.
> #7 0x00007f1b43329256 in __bamc_search () from /lib64/libdb-5.3.so
> <http://libdb-5.3.so>
> No symbol table info available.
> #8 0x00007f1b4332ad0f in __bamc_get () from /lib64/libdb-5.3.so
> <http://libdb-5.3.so>
> No symbol table info available.
> #9 0x00007f1b433e3ca6 in __dbc_iget () from /lib64/libdb-5.3.so
> <http://libdb-5.3.so>
> No symbol table info available.
> #10 0x00007f1b433f2b02 in __dbc_get_pp () from /lib64/libdb-5.3.so
> <http://libdb-5.3.so>
> No symbol table info available.
> #11 0x00007f1b3e8ce680 in idl_new_fetch () from
> /usr/lib64/dirsrv/plugins/libback-ldbm.so
> No symbol table info available.
> #12 0x00007f1b3e8dcb26 in index_read_ext_allids () from
> /usr/lib64/dirsrv/plugins/libback-ldbm.so
> No symbol table info available.
> #13 0x00007f1b3e8c7284 in keys2idl () from
> /usr/lib64/dirsrv/plugins/libback-ldbm.so
> No symbol table info available.
> #14 0x00007f1b3e8c79e3 in ava_candidates.isra.0 () from
> /usr/lib64/dirsrv/plugins/libback-ldbm.so
> No symbol table info available.
> #15 0x00007f1b3e8c7fd2 in filter_candidates_ext () from
> /usr/lib64/dirsrv/plugins/libback-ldbm.so
> No symbol table info available.
> #16 0x00007f1b3e8c9096 in list_candidates () from
> /usr/lib64/dirsrv/plugins/libback-ldbm.so
> No symbol table info available.
> #17 0x00007f1b3e8c7f40 in filter_candidates_ext () from
> /usr/lib64/dirsrv/plugins/libback-ldbm.so
> No symbol table info available.
> #18 0x00007f1b3e90380c in subtree_candidates () from
> /usr/lib64/dirsrv/plugins/libback-ldbm.so
> No symbol table info available.
> #19 0x00007f1b3e904e7f in ldbm_back_search () from
> /usr/lib64/dirsrv/plugins/libback-ldbm.so
> No symbol table info available.
> #20 0x00007f1b4b2321d6 in op_shared_search () from
> /usr/lib64/dirsrv/libslapd.so.0
> No symbol table info available.
> #21 0x00007f1b4b24294e in search_internal_callback_pb () from
> /usr/lib64/dirsrv/libslapd.so.0
> No symbol table info available.
> #22 0x00007f1b4b242be8 in search_internal_pb () from
> /usr/lib64/dirsrv/libslapd.so.0
> No symbol table info available.
> #23 0x00007f1b4b72149b in ids_sasl_canon_user ()
> No symbol table info available.
> #24 0x00007f1b491e193b in _sasl_canon_user () from /lib64/libsasl2.so.3
> No symbol table info available.
> #25 0x00007f1b491e1c4c in _sasl_canon_user_lookup () from
> /lib64/libsasl2.so.3
> No symbol table info available.
> #26 0x00007f1b430e9fb9 in gssapi_server_mech_step () from
> /usr/lib64/sasl2/libgssapiv2.so
> No symbol table info available.
> #27 0x00007f1b491edb9b in sasl_server_step () from /lib64/libsasl2.so.3
> No symbol table info available.
> #28 0x00007f1b4b72283b in ids_sasl_check_bind ()
> No symbol table info available.
> #29 0x00007f1b4b70a61a in do_bind ()
> No symbol table info available.
> #30 0x00007f1b4b7113fd in connection_threadmain ()
> No symbol table info available.
> #31 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so
> No symbol table info available.
> #32 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0
> No symbol table info available.
> #33 0x00007f1b48aef28d in clone () from /lib64/libc.so.6
> No symbol table info available.
>
> Thread 25 (Thread 0x7f1b0c7f8700 (LWP 105185)):
> #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 () from
> /lib64/libpthread.so.0
> No symbol table info available.
> #1 0x00007f1b4941b050 in PR_WaitCondVar () from /lib64/libnspr4.so
> No symbol table info available.
> #2 0x00007f1b4b70efee in connection_wait_for_new_work ()
> No symbol table info available.
> #3 0x00007f1b4b710719 in connection_threadmain ()
> No symbol table info available.
> #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so
> No symbol table info available.
> #5 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0
> No symbol table info available.
> #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6
> No symbol table info available.
>
> Thread 24 (Thread 0x7f1b0cff9700 (LWP 105184)):
> #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 () from
> /lib64/libpthread.so.0
> No symbol table info available.
> #1 0x00007f1b4941b050 in PR_WaitCondVar () from /lib64/libnspr4.so
> No symbol table info available.
> #2 0x00007f1b4b70efee in connection_wait_for_new_work ()
> No symbol table info available.
> #3 0x00007f1b4b710719 in connection_threadmain ()
> No symbol table info available.
> #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so
> No symbol table info available.
> #5 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0
> No symbol table info available.
> #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6
> No symbol table info available.
>
> Thread 23 (Thread 0x7f1b0d7fa700 (LWP 105183)):
> #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 () from
> /lib64/libpthread.so.0
> No symbol table info available.
> #1 0x00007f1b4941b050 in PR_WaitCondVar () from /lib64/libnspr4.so
> No symbol table info available.
> #2 0x00007f1b4b70efee in connection_wait_for_new_work ()
> No symbol table info available.
> #3 0x00007f1b4b710719 in connection_threadmain ()
> No symbol table info available.
> #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so
> No symbol table info available.
> #5 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0
> No symbol table info available.
> #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6
> No symbol table info available.
>
> Thread 22 (Thread 0x7f1b0dffb700 (LWP 105182)):
> #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 () from
> /lib64/libpthread.so.0
> No symbol table info available.
> #1 0x00007f1b4941b050 in PR_WaitCondVar () from /lib64/libnspr4.so
> No symbol table info available.
> #2 0x00007f1b4b70efee in connection_wait_for_new_work ()
> No symbol table info available.
> #3 0x00007f1b4b710719 in connection_threadmain ()
> No symbol table info available.
> #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so
> No symbol table info available.
> #5 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0
> No symbol table info available.
> #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6
> No symbol table info available.
>
> Thread 21 (Thread 0x7f1b0e7fc700 (LWP 105181)):
> #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 () from
> /lib64/libpthread.so.0
> No symbol table info available.
> #1 0x00007f1b4941b050 in PR_WaitCondVar () from /lib64/libnspr4.so
> No symbol table info available.
> #2 0x00007f1b4b70efee in connection_wait_for_new_work ()
> No symbol table info available.
> #3 0x00007f1b4b710719 in connection_threadmain ()
> No symbol table info available.
> #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so
> No symbol table info available.
> #5 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0
> No symbol table info available.
> #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6
> No symbol table info available.
>
> Thread 20 (Thread 0x7f1b0effd700 (LWP 105180)):
> #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 () from
> /lib64/libpthread.so.0
> No symbol table info available.
> #1 0x00007f1b433212f3 in __db_hybrid_mutex_suspend () from
> /lib64/libdb-5.3.so <http://libdb-5.3.so>
> No symbol table info available.
> #2 0x00007f1b43320b37 in __db_tas_mutex_readlock () from
> /lib64/libdb-5.3.so <http://libdb-5.3.so>
> No symbol table info available.
> #3 0x00007f1b4343ad35 in __memp_fget () from /lib64/libdb-5.3.so
> <http://libdb-5.3.so>
> No symbol table info available.
> #4 0x00007f1b433e4333 in __dbc_iget () from /lib64/libdb-5.3.so
> <http://libdb-5.3.so>
> No symbol table info available.
> #5 0x00007f1b433f2b02 in __dbc_get_pp () from /lib64/libdb-5.3.so
> <http://libdb-5.3.so>
> No symbol table info available.
> #6 0x00007f1b3e8ce822 in idl_new_fetch () from
> /usr/lib64/dirsrv/plugins/libback-ldbm.so
> No symbol table info available.
> #7 0x00007f1b3e8dcb26 in index_read_ext_allids () from
> /usr/lib64/dirsrv/plugins/libback-ldbm.so
> No symbol table info available.
> #8 0x00007f1b3e8c7284 in keys2idl () from
> /usr/lib64/dirsrv/plugins/libback-ldbm.so
> No symbol table info available.
> #9 0x00007f1b3e8c79e3 in ava_candidates.isra.0 () from
> /usr/lib64/dirsrv/plugins/libback-ldbm.so
> No symbol table info available.
> #10 0x00007f1b3e8c7fd2 in filter_candidates_ext () from
> /usr/lib64/dirsrv/plugins/libback-ldbm.so
> No symbol table info available.
> #11 0x00007f1b3e8c9096 in list_candidates () from
> /usr/lib64/dirsrv/plugins/libback-ldbm.so
> No symbol table info available.
> #12 0x00007f1b3e8c7f40 in filter_candidates_ext () from
> /usr/lib64/dirsrv/plugins/libback-ldbm.so
> No symbol table info available.
> #13 0x00007f1b3e8c9096 in list_candidates () from
> /usr/lib64/dirsrv/plugins/libback-ldbm.so
> No symbol table info available.
> #14 0x00007f1b3e8c7f40 in filter_candidates_ext () from
> /usr/lib64/dirsrv/plugins/libback-ldbm.so
> No symbol table info available.
> #15 0x00007f1b3e8c9096 in list_candidates () from
> /usr/lib64/dirsrv/plugins/libback-ldbm.so
> No symbol table info available.
> #16 0x00007f1b3e8c7f40 in filter_candidates_ext () from
> /usr/lib64/dirsrv/plugins/libback-ldbm.so
> No symbol table info available.
> #17 0x00007f1b3e90380c in subtree_candidates () from
> /usr/lib64/dirsrv/plugins/libback-ldbm.so
> No symbol table info available.
> #18 0x00007f1b3e904e7f in ldbm_back_search () from
> /usr/lib64/dirsrv/plugins/libback-ldbm.so
> No symbol table info available.
> #19 0x00007f1b4b2321d6 in op_shared_search () from
> /usr/lib64/dirsrv/libslapd.so.0
> No symbol table info available.
> #20 0x00007f1b4b72333e in do_search ()
> No symbol table info available.
> #21 0x00007f1b4b7113c3 in connection_threadmain ()
> No symbol table info available.
> #22 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so
> No symbol table info available.
> #23 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0
> No symbol table info available.
> #24 0x00007f1b48aef28d in clone () from /lib64/libc.so.6
> No symbol table info available.
>
> Thread 19 (Thread 0x7f1b0f7fe700 (LWP 105179)):
> #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 () from
> /lib64/libpthread.so.0
> No symbol table info available.
> #1 0x00007f1b433212f3 in __db_hybrid_mutex_suspend () from
> /lib64/libdb-5.3.so <http://libdb-5.3.so>
> No symbol table info available.
> #2 0x00007f1b43320640 in __db_tas_mutex_lock () from
> /lib64/libdb-5.3.so <http://libdb-5.3.so>
> No symbol table info available.
> #3 0x00007f1b433cad3a in __lock_get_internal () from
> /lib64/libdb-5.3.so <http://libdb-5.3.so>
> No symbol table info available.
> #4 0x00007f1b433cb820 in __lock_get () from /lib64/libdb-5.3.so
> <http://libdb-5.3.so>
> No symbol table info available.
> #5 0x00007f1b433f7142 in __db_lget () from /lib64/libdb-5.3.so
> <http://libdb-5.3.so>
> No symbol table info available.
> #6 0x00007f1b4333e605 in __bam_search () from /lib64/libdb-5.3.so
> <http://libdb-5.3.so>
> No symbol table info available.
> #7 0x00007f1b43329256 in __bamc_search () from /lib64/libdb-5.3.so
> <http://libdb-5.3.so>
> No symbol table info available.
> #8 0x00007f1b4332ad0f in __bamc_get () from /lib64/libdb-5.3.so
> <http://libdb-5.3.so>
> No symbol table info available.
> #9 0x00007f1b433e3ca6 in __dbc_iget () from /lib64/libdb-5.3.so
> <http://libdb-5.3.so>
> No symbol table info available.
> #10 0x00007f1b433f2b02 in __dbc_get_pp () from /lib64/libdb-5.3.so
> <http://libdb-5.3.so>
> No symbol table info available.
> #11 0x00007f1b3e8ce680 in idl_new_fetch () from
> /usr/lib64/dirsrv/plugins/libback-ldbm.so
> No symbol table info available.
> #12 0x00007f1b3e8dcb26 in index_read_ext_allids () from
> /usr/lib64/dirsrv/plugins/libback-ldbm.so
> No symbol table info available.
> #13 0x00007f1b3e8c7284 in keys2idl () from
> /usr/lib64/dirsrv/plugins/libback-ldbm.so
> No symbol table info available.
> #14 0x00007f1b3e8c79e3 in ava_candidates.isra.0 () from
> /usr/lib64/dirsrv/plugins/libback-ldbm.so
> No symbol table info available.
> #15 0x00007f1b3e8c7fd2 in filter_candidates_ext () from
> /usr/lib64/dirsrv/plugins/libback-ldbm.so
> No symbol table info available.
> #16 0x00007f1b3e8c9096 in list_candidates () from
> /usr/lib64/dirsrv/plugins/libback-ldbm.so
> No symbol table info available.
> #17 0x00007f1b3e8c7f40 in filter_candidates_ext () from
> /usr/lib64/dirsrv/plugins/libback-ldbm.so
> No symbol table info available.
> #18 0x00007f1b3e8c9096 in list_candidates () from
> /usr/lib64/dirsrv/plugins/libback-ldbm.so
> No symbol table info available.
> #19 0x00007f1b3e8c7f40 in filter_candidates_ext () from
> /usr/lib64/dirsrv/plugins/libback-ldbm.so
> No symbol table info available.
> #20 0x00007f1b3e8c9096 in list_candidates () from
> /usr/lib64/dirsrv/plugins/libback-ldbm.so
> No symbol table info available.
> #21 0x00007f1b3e8c7f40 in filter_candidates_ext () from
> /usr/lib64/dirsrv/plugins/libback-ldbm.so
> No symbol table info available.
> #22 0x00007f1b3e90380c in subtree_candidates () from
> /usr/lib64/dirsrv/plugins/libback-ldbm.so
> No symbol table info available.
> #23 0x00007f1b3e904e7f in ldbm_back_search () from
> /usr/lib64/dirsrv/plugins/libback-ldbm.so
> No symbol table info available.
> #24 0x00007f1b4b2321d6 in op_shared_search () from
> /usr/lib64/dirsrv/libslapd.so.0
> No symbol table info available.
> #25 0x00007f1b4b72333e in do_search ()
> No symbol table info available.
> #26 0x00007f1b4b7113c3 in connection_threadmain ()
> No symbol table info available.
> #27 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so
> No symbol table info available.
> #28 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0
> No symbol table info available.
> #29 0x00007f1b48aef28d in clone () from /lib64/libc.so.6
> No symbol table info available.
>
> Thread 18 (Thread 0x7f1b0ffff700 (LWP 105178)):
> #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 () from
> /lib64/libpthread.so.0
> No symbol table info available.
> #1 0x00007f1b4941b050 in PR_WaitCondVar () from /lib64/libnspr4.so
> No symbol table info available.
> #2 0x00007f1b4b70efee in connection_wait_for_new_work ()
> No symbol table info available.
> #3 0x00007f1b4b710719 in connection_threadmain ()
> No symbol table info available.
> #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so
> No symbol table info available.
> #5 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0
> No symbol table info available.
> #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6
> No symbol table info available.
>
> Thread 17 (Thread 0x7f1b14d28700 (LWP 105177)):
> #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 () from
> /lib64/libpthread.so.0
> No symbol table info available.
> #1 0x00007f1b4941b050 in PR_WaitCondVar () from /lib64/libnspr4.so
> No symbol table info available.
> #2 0x00007f1b4b70efee in connection_wait_for_new_work ()
> No symbol table info available.
> #3 0x00007f1b4b710719 in connection_threadmain ()
> No symbol table info available.
> #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so
> No symbol table info available.
> #5 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0
> No symbol table info available.
> #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6
> No symbol table info available.
>
> Thread 16 (Thread 0x7f1b15529700 (LWP 105176)):
> #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 () from
> /lib64/libpthread.so.0
> No symbol table info available.
> #1 0x00007f1b4941b050 in PR_WaitCondVar () from /lib64/libnspr4.so
> No symbol table info available.
> #2 0x00007f1b4b70efee in connection_wait_for_new_work ()
> No symbol table info available.
> #3 0x00007f1b4b710719 in connection_threadmain ()
> No symbol table info available.
> #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so
> No symbol table info available.
> #5 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0
> No symbol table info available.
> #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6
> No symbol table info available.
>
> Thread 15 (Thread 0x7f1b15d2a700 (LWP 105175)):
> #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 () from
> /lib64/libpthread.so.0
> No symbol table info available.
> #1 0x00007f1b4941b050 in PR_WaitCondVar () from /lib64/libnspr4.so
> No symbol table info available.
> #2 0x00007f1b4b70efee in connection_wait_for_new_work ()
> No symbol table info available.
> #3 0x00007f1b4b710719 in connection_threadmain ()
> No symbol table info available.
> #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so
> No symbol table info available.
> #5 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0
> No symbol table info available.
> #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6
> No symbol table info available.
>
> Thread 14 (Thread 0x7f1b1652b700 (LWP 105174)):
> #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 () from
> /lib64/libpthread.so.0
> No symbol table info available.
> #1 0x00007f1b4941b050 in PR_WaitCondVar () from /lib64/libnspr4.so
> No symbol table info available.
> #2 0x00007f1b4b70efee in connection_wait_for_new_work ()
> No symbol table info available.
> #3 0x00007f1b4b710719 in connection_threadmain ()
> No symbol table info available.
> #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so
> No symbol table info available.
> #5 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0
> No symbol table info available.
> #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6
> No symbol table info available.
>
> Thread 13 (Thread 0x7f1b16d2c700 (LWP 105173)):
> #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 () from
> /lib64/libpthread.so.0
> No symbol table info available.
> #1 0x00007f1b433212f3 in __db_hybrid_mutex_suspend () from
> /lib64/libdb-5.3.so <http://libdb-5.3.so>
> No symbol table info available.
> #2 0x00007f1b43320640 in __db_tas_mutex_lock () from
> /lib64/libdb-5.3.so <http://libdb-5.3.so>
> No symbol table info available.
> #3 0x00007f1b433cad3a in __lock_get_internal () from
> /lib64/libdb-5.3.so <http://libdb-5.3.so>
> No symbol table info available.
> #4 0x00007f1b433cb820 in __lock_get () from /lib64/libdb-5.3.so
> <http://libdb-5.3.so>
> No symbol table info available.
> #5 0x00007f1b433f7142 in __db_lget () from /lib64/libdb-5.3.so
> <http://libdb-5.3.so>
> No symbol table info available.
> #6 0x00007f1b4333e605 in __bam_search () from /lib64/libdb-5.3.so
> <http://libdb-5.3.so>
> No symbol table info available.
> #7 0x00007f1b43329256 in __bamc_search () from /lib64/libdb-5.3.so
> <http://libdb-5.3.so>
> No symbol table info available.
> #8 0x00007f1b4332ad0f in __bamc_get () from /lib64/libdb-5.3.so
> <http://libdb-5.3.so>
> No symbol table info available.
> #9 0x00007f1b433e3ca6 in __dbc_iget () from /lib64/libdb-5.3.so
> <http://libdb-5.3.so>
> No symbol table info available.
> #10 0x00007f1b433f2b02 in __dbc_get_pp () from /lib64/libdb-5.3.so
> <http://libdb-5.3.so>
> No symbol table info available.
> #11 0x00007f1b3e8ce680 in idl_new_fetch () from
> /usr/lib64/dirsrv/plugins/libback-ldbm.so
> No symbol table info available.
> #12 0x00007f1b3e8dcb26 in index_read_ext_allids () from
> /usr/lib64/dirsrv/plugins/libback-ldbm.so
> No symbol table info available.
> #13 0x00007f1b3e8c7284 in keys2idl () from
> /usr/lib64/dirsrv/plugins/libback-ldbm.so
> No symbol table info available.
> #14 0x00007f1b3e8c79e3 in ava_candidates.isra.0 () from
> /usr/lib64/dirsrv/plugins/libback-ldbm.so
> No symbol table info available.
> #15 0x00007f1b3e8c7fd2 in filter_candidates_ext () from
> /usr/lib64/dirsrv/plugins/libback-ldbm.so
> No symbol table info available.
> #16 0x00007f1b3e8c9096 in list_candidates () from
> /usr/lib64/dirsrv/plugins/libback-ldbm.so
> No symbol table info available.
> #17 0x00007f1b3e8c7f40 in filter_candidates_ext () from
> /usr/lib64/dirsrv/plugins/libback-ldbm.so
> No symbol table info available.
> #18 0x00007f1b3e8c9096 in list_candidates () from
> /usr/lib64/dirsrv/plugins/libback-ldbm.so
> No symbol table info available.
> #19 0x00007f1b3e8c7f40 in filter_candidates_ext () from
> /usr/lib64/dirsrv/plugins/libback-ldbm.so
> No symbol table info available.
> #20 0x00007f1b3e8c9096 in list_candidates () from
> /usr/lib64/dirsrv/plugins/libback-ldbm.so
> No symbol table info available.
> #21 0x00007f1b3e8c7f40 in filter_candidates_ext () from
> /usr/lib64/dirsrv/plugins/libback-ldbm.so
> No symbol table info available.
> #22 0x00007f1b3e90380c in subtree_candidates () from
> /usr/lib64/dirsrv/plugins/libback-ldbm.so
> No symbol table info available.
> #23 0x00007f1b3e904e7f in ldbm_back_search () from
> /usr/lib64/dirsrv/plugins/libback-ldbm.so
> No symbol table info available.
> #24 0x00007f1b4b2321d6 in op_shared_search () from
> /usr/lib64/dirsrv/libslapd.so.0
> No symbol table info available.
> #25 0x00007f1b4b72333e in do_search ()
> No symbol table info available.
> #26 0x00007f1b4b7113c3 in connection_threadmain ()
> No symbol table info available.
> #27 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so
> No symbol table info available.
> #28 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0
> No symbol table info available.
> #29 0x00007f1b48aef28d in clone () from /lib64/libc.so.6
> No symbol table info available.
>
> Thread 12 (Thread 0x7f1b1752d700 (LWP 105172)):
> #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 () from
> /lib64/libpthread.so.0
> No symbol table info available.
> #1 0x00007f1b4941b050 in PR_WaitCondVar () from /lib64/libnspr4.so
> No symbol table info available.
> #2 0x00007f1b4b70efee in connection_wait_for_new_work ()
> No symbol table info available.
> #3 0x00007f1b4b710719 in connection_threadmain ()
> No symbol table info available.
> #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so
> No symbol table info available.
> #5 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0
> No symbol table info available.
> #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6
> No symbol table info available.
>
> Thread 11 (Thread 0x7f1b17d2e700 (LWP 105171)):
> #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 () from
> /lib64/libpthread.so.0
> No symbol table info available.
> #1 0x00007f1b4941b050 in PR_WaitCondVar () from /lib64/libnspr4.so
> No symbol table info available.
> #2 0x00007f1b4b70efee in connection_wait_for_new_work ()
> No symbol table info available.
> #3 0x00007f1b4b710719 in connection_threadmain ()
> No symbol table info available.
> #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so
> No symbol table info available.
> #5 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0
> No symbol table info available.
> #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6
> No symbol table info available.
>
> Thread 10 (Thread 0x7f1b2509d700 (LWP 105168)):
> #0 0x00007f1b48dc5a82 in pthread_cond_timedwait@@GLIBC_2.3.2 () from
> /lib64/libpthread.so.0
> No symbol table info available.
> #1 0x00007f1b4941ab07 in pt_TimedWait () from /lib64/libnspr4.so
> No symbol table info available.
> #2 0x00007f1b4941afce in PR_WaitCondVar () from /lib64/libnspr4.so
> No symbol table info available.
> #3 0x00007f1b4b2061e8 in eq_loop () from /usr/lib64/dirsrv/libslapd.so.0
> No symbol table info available.
> #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so
> No symbol table info available.
> #5 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0
> No symbol table info available.
> #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6
> No symbol table info available.
>
> Thread 9 (Thread 0x7f1b2589e700 (LWP 105167)):
> #0 0x00007f1b48dc5a82 in pthread_cond_timedwait@@GLIBC_2.3.2 () from
> /lib64/libpthread.so.0
> No symbol table info available.
> #1 0x00007f1b4941ab07 in pt_TimedWait () from /lib64/libnspr4.so
> No symbol table info available.
> #2 0x00007f1b4941afce in PR_WaitCondVar () from /lib64/libnspr4.so
> No symbol table info available.
> #3 0x00007f1b4b719d03 in housecleaning ()
> No symbol table info available.
> #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so
> No symbol table info available.
> #5 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0
> No symbol table info available.
> #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6
> No symbol table info available.
>
> Thread 8 (Thread 0x7f1b2609f700 (LWP 105166)):
> #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 () from
> /lib64/libpthread.so.0
> No symbol table info available.
> #1 0x00007f1b4941b050 in PR_WaitCondVar () from /lib64/libnspr4.so
> No symbol table info available.
> #2 0x00007f1b4b260198 in slapi_wait_condvar () from
> /usr/lib64/dirsrv/libslapd.so.0
> No symbol table info available.
> #3 0x00007f1b3cf5deed in roles_cache_wait_on_change () from
> /usr/lib64/dirsrv/plugins/libroles-plugin.so
> No symbol table info available.
> #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so
> No symbol table info available.
> #5 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0
> No symbol table info available.
> #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6
> No symbol table info available.
>
> Thread 7 (Thread 0x7f1b268a0700 (LWP 105165)):
> #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 () from
> /lib64/libpthread.so.0
> No symbol table info available.
> #1 0x00007f1b4941b050 in PR_WaitCondVar () from /lib64/libnspr4.so
> No symbol table info available.
> #2 0x00007f1b4b260198 in slapi_wait_condvar () from
> /usr/lib64/dirsrv/libslapd.so.0
> No symbol table info available.
> #3 0x00007f1b3cf5deed in roles_cache_wait_on_change () from
> /usr/lib64/dirsrv/plugins/libroles-plugin.so
> No symbol table info available.
> #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so
> No symbol table info available.
> #5 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0
> No symbol table info available.
> #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6
> No symbol table info available.
>
> Thread 6 (Thread 0x7f1b270a1700 (LWP 105164)):
> #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 () from
> /lib64/libpthread.so.0
> No symbol table info available.
> #1 0x00007f1b4941b050 in PR_WaitCondVar () from /lib64/libnspr4.so
> No symbol table info available.
> #2 0x00007f1b4b260198 in slapi_wait_condvar () from
> /usr/lib64/dirsrv/libslapd.so.0
> No symbol table info available.
> #3 0x00007f1b3cf5deed in roles_cache_wait_on_change () from
> /usr/lib64/dirsrv/plugins/libroles-plugin.so
> No symbol table info available.
> #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so
> No symbol table info available.
> #5 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0
> No symbol table info available.
> #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6
> No symbol table info available.
>
> Thread 5 (Thread 0x7f1b278a2700 (LWP 105163)):
> #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 () from
> /lib64/libpthread.so.0
> No symbol table info available.
> #1 0x00007f1b4941b050 in PR_WaitCondVar () from /lib64/libnspr4.so
> No symbol table info available.
> #2 0x00007f1b4b260198 in slapi_wait_condvar () from
> /usr/lib64/dirsrv/libslapd.so.0
> No symbol table info available.
> #3 0x00007f1b415f262e in cos_cache_wait_on_change () from
> /usr/lib64/dirsrv/plugins/libcos-plugin.so
> No symbol table info available.
> #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so
> No symbol table info available.
> #5 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0
> No symbol table info available.
> #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6
> No symbol table info available.
>
> Thread 4 (Thread 0x7f1b282ac700 (LWP 105162)):
> #0 0x00007f1b48ae69b3 in select () from /lib64/libc.so.6
> No symbol table info available.
> #1 0x00007f1b4b2710e9 in DS_Sleep () from /usr/lib64/dirsrv/libslapd.so.0
> No symbol table info available.
> #2 0x00007f1b3e8ba707 in perf_threadmain () from
> /usr/lib64/dirsrv/plugins/libback-ldbm.so
> No symbol table info available.
> #3 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so
> No symbol table info available.
> #4 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0
> No symbol table info available.
> #5 0x00007f1b48aef28d in clone () from /lib64/libc.so.6
> No symbol table info available.
>
> Thread 3 (Thread 0x7f1b28aad700 (LWP 105161)):
> #0 0x00007f1b48ae69b3 in select () from /lib64/libc.so.6
> No symbol table info available.
> #1 0x00007f1b4b2710e9 in DS_Sleep () from /usr/lib64/dirsrv/libslapd.so.0
> No symbol table info available.
> #2 0x00007f1b3e8bfb7f in trickle_threadmain () from
> /usr/lib64/dirsrv/plugins/libback-ldbm.so
> No symbol table info available.
> #3 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so
> No symbol table info available.
> #4 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0
> No symbol table info available.
> #5 0x00007f1b48aef28d in clone () from /lib64/libc.so.6
> No symbol table info available.
>
> Thread 2 (Thread 0x7f1b292ae700 (LWP 105160)):
> #0 0x00007f1b48ae69b3 in select () from /lib64/libc.so.6
> No symbol table info available.
> #1 0x00007f1b4344dbad in __os_yield () from /lib64/libdb-5.3.so
> <http://libdb-5.3.so>
> No symbol table info available.
> #2 0x00007f1b434492b3 in __memp_sync_int () from /lib64/libdb-5.3.so
> <http://libdb-5.3.so>
> No symbol table info available.
> #3 0x00007f1b43459752 in __txn_checkpoint () from /lib64/libdb-5.3.so
> <http://libdb-5.3.so>
> No symbol table info available.
> #4 0x00007f1b43459b74 in __txn_checkpoint_pp () from
> /lib64/libdb-5.3.so <http://libdb-5.3.so>
> No symbol table info available.
> #5 0x00007f1b3e8c3a87 in checkpoint_threadmain () from
> /usr/lib64/dirsrv/plugins/libback-ldbm.so
> No symbol table info available.
> #6 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so
> No symbol table info available.
> #7 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0
> No symbol table info available.
> #8 0x00007f1b48aef28d in clone () from /lib64/libc.so.6
> No symbol table info available.
>
> Thread 1 (Thread 0x7f1b29aaf700 (LWP 105159)):
> #0 0x00007f1b48ae69b3 in select () from /lib64/libc.so.6
> No symbol table info available.
> #1 0x00007f1b4b2710e9 in DS_Sleep () from /usr/lib64/dirsrv/libslapd.so.0
> No symbol table info available.
> #2 0x00007f1b3e8bf907 in deadlock_threadmain () from
> /usr/lib64/dirsrv/plugins/libback-ldbm.so
> No symbol table info available.
> #3 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so
> No symbol table info available.
> #4 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0
> No symbol table info available.
> #5 0x00007f1b48aef28d in clone () from /lib64/libc.so.6
> No symbol table info available.
>
>
>
> Thanks,
>
> Rakesh
>
>
> On Mon, Sep 5, 2016 at 4:48 PM, thierry bordaz <tbordaz at redhat.com
> <mailto:tbordaz at redhat.com>> wrote:
>
>
>
> On 09/05/2016 12:05 PM, Rakesh Rajasekharan wrote:
>> Hi Thierry,
>>
>>
>> I was getting the hang issue while running ipa-client-install
>> simultaneously on few clients..
>> However, today, I am not able to replicate that.
>>
>> I could not get a gdb . But i will try getting that the next time
>> I face this issue.
>>
>> The CPU does not stay high.. it just momentarily touches a high
>> value and then drops down to around 2-7%
>>
>> One question I have is , is it ok to set it nsslapd-threadnumber
>> to a very high value .
>> I have around 4000 clients and with nsslapd-maxthreadsperconn set
>> to 5..So, can I set nsslapd-threadnumber to around 25000.
>
> Hello,
>
> I know some users running in production with several hunderds of
> threads (>600) and this without problem.
>
> I do not recall having suggested to increase that number and for
> what reason.
> Usually 30 workers is a good enough value. It can create
> bootleneck if for some reason each operation is very long to
> satisfy and exhaust the number of workers. You can monitor the
> work queue:
>
> ldapsearch -D "cn=directory manager" -w xxx -LLL -b
> "cn=monitor" -s base opsinitiated opscompleted
>
>
> If opscompleted-opsinitiated remains close to threadnumber, then
> yes it would be valuable to increase it.
>
> The computation #client * #async_op_per_client sound an overkill.
> Even if all clients send at the exact same time all their
> requests, it is very likely that some common resource (db page,
> log, allocator...) will serialize them. If you monitor a need to
> increase the work, you would for example set it to 50, then
> monitor, then set it to 100, then monitor... until you find a good
> enough value.
> Note the increasing the #thread, increases the memory footprint
> that will reduce the efficiency of file system cache and can
> increase the response time.
>
>
> best regards
> thierry
>
>>
>> Thanks
>>
>> On Mon, Sep 5, 2016 at 1:03 PM, thierry bordaz
>> <tbordaz at redhat.com <mailto:tbordaz at redhat.com>> wrote:
>>
>>
>> Hi Rakesh,
>>
>> Were you able to get a pstack or full stack with gdb
>> (http://www.port389.org/docs/389ds/FAQ/faq.html#debugging-crashes)
>> when the server hangs ?
>>
>> If it happens with 500 threads as well as with 30, using 30
>> threads is a better choice to debug this issue.
>> I will try to reproduce using 150 parallel 'ipa user-find
>> p-testipa' commands
>>
>> Something I am unsure is if the CPU consumption stays high
>> (you mentioned 340% CPU usage) as long as the hang happens or
>> if after a suddent shot up to 340% (that marks the beginning
>> of the hang) it drops and stay hanging ?
>>
>> thanks
>> thierry
>>
>> On 09/04/2016 08:40 PM, Rakesh Rajasekharan wrote:
>>> starce on the slapd process actually had this in the output..
>>> FUTEX_WAIT_PRIVATE
>>>
>>> and checking for the number of threads slapd had.. there
>>> were 5015 threads
>>>
>>> ps -efL|grep slapd|wc -l
>>> 5015
>>>
>>> strace on most of the threads gave this output
>>>
>>> strace -p 67411
>>> Process 67411 attached
>>> futex(0x7f3f0226b41c, FUTEX_WAIT_PRIVATE, 1, NULL) = -1
>>> EAGAIN (Resource temporarily unavailable)
>>> futex(0x7f3f0226b41c, FUTEX_WAIT_PRIVATE, 2, NULL^CProcess
>>> 67411 detached
>>>
>>>
>>>
>>>
>>>
>>> On Sun, Sep 4, 2016 at 5:34 PM, Rakesh Rajasekharan
>>> <rakesh.rajasekharan at gmail.com
>>> <mailto:rakesh.rajasekharan at gmail.com>> wrote:
>>>
>>> I have again got the issue of IPA hanging.. The issue
>>> came up when i tried to run ipa-client-isntall on 142
>>> clients simultaneously
>>>
>>>
>>> None of the IPA commands are responding, and I see this
>>> error
>>>
>>> ipa user-find p-testipa
>>> ipa: ERROR: Insufficient access: SASL(-1): generic
>>> failure: GSSAPI Error: Unspecified GSS failure. Minor
>>> code may provide more information (KDC returned error
>>> string: PROCESS_TGS)
>>>
>>> KRB5_TRACE=/dev/stdout kinit admin
>>> [41178] 1472984115.233214: Getting initial credentials
>>> for admin at XYZ.COM <mailto:admin at XYZ.COM>
>>> [41178] 1472984115.235257: Sending request (167 bytes)
>>> to XYZ.COM <http://XYZ.COM>
>>> [41178] 1472984115.235419: Initiating TCP connection to
>>> stream 10.1.3.36:88 <http://10.1.3.36:88>
>>> [41178] 1472984115.235685: Sending TCP request to stream
>>> 10.1.3.36:88 <http://10.1.3.36:88>
>>> [41178] 1472984120.238914: Received answer (174 bytes)
>>> from stream 10.1.3.36:88 <http://10.1.3.36:88>
>>> [41178] 1472984120.238925: Terminating TCP connection to
>>> stream 10.1.3.36:88 <http://10.1.3.36:88>
>>> [41178] 1472984120.238993: Response was from master KDC
>>> [41
>>>
>>>
>>> Running an ldapsearch to see the db.. does not give any
>>> results and just hangs there
>>>
>>> ldapsearch -x -D 'cn=Directory Manager' -W -s one -b
>>> 'cn=kerberos,dc=xyz,dc=com'
>>> Enter LDAP Password:
>>>
>>> even an ldapsearch -x does not respond
>>> At this point, am sure that slapd is the one causing issues
>>>
>>> Running an strace against the hung slapd itself seems to
>>> get stuck does not proceed after saying "attaching to
>>> process"
>>>
>>> From some others posts I read Thierry suggesting to
>>> increase the nsslapd-threadnumber value
>>>
>>> It was set to 30, I think that might be too low.
>>>
>>> I have raised it to 500
>>>
>>> Now after restarting the service .. ldapsearch starts
>>> responding.
>>> But running the test to add a sudden high number of
>>> clients again left ns-slapd to hung state
>>>
>>> When i attempted adding the clients.. the ns-slapd cpu
>>> usage shot up to 340% and after that ns-slapd stopped
>>> responding
>>>
>>> So now, atleast I know what might be causing the issue
>>> and I can now easily reproduce it.
>>>
>>> Is there a way I can make ns-slapd handle a sudden bump
>>> in incoming request for ipa-client-install
>>>
>>> Thanks
>>> Rakesh
>>>
>>>
>>>
>>>
>>>
>>>
>>> On Mon, Aug 29, 2016 at 11:18 PM, Rich Megginson
>>> <rmeggins at redhat.com <mailto:rmeggins at redhat.com>> wrote:
>>>
>>> On 08/29/2016 10:53 AM, Rakesh Rajasekharan wrote:
>>>> Hi Thierry,
>>>>
>>>> My machine has 30GB RAM ..and 389-ds version is 1.3.4
>>>>
>>>> ldapsearch shows the values for
>>>> nsslapd-cachememsize updated to 200MB.
>>>>
>>>> ldapsearch -LLL -o ldif-wrap=no -D "cn=directory
>>>> manager" -w 'mypassword' -b 'cn=userRoot,cn=ldbm
>>>> database,cn=plugins,cn=config'|grep
>>>> nsslapd-cachememsize
>>>> nsslapd-cachememsize: 209715200
>>>>
>>>>
>>>> So, it seems to have updated though seeing that
>>>> warning(WARNING: ipaca: entry cache size 10485760B
>>>> is less than db size 11599872B) in the log confuses
>>>> me a bit.
>>>>
>>>> Thers one more entry that I found from the
>>>> ldapsearch to be bit low
>>>>
>>>> nsslapd-dncachememsize: 10485760
>>>> maxdncachesize: 10485760
>>>>
>>>> Should I update these as well to a higher value
>>>>
>>>> At the time when the issue happened, the memory
>>>> usage as well as the overall load of the system was
>>>> very low .
>>>> I will try reproducing the issue atleast in my QA
>>>> env..probably by trying to mock simultaneous
>>>> parallel logins to a large number of hosts
>>>
>>> To monitor your cache sizes, please use the dbmon.sh
>>> tool provided with your distro. If that is not
>>> available with your particular distro, see
>>> https://github.com/richm/scripts/wiki/dbmon.sh
>>> <https://github.com/richm/scripts/wiki/dbmon.sh>
>>>
>>>
>>>>
>>>>
>>>> thanks
>>>> Rakesh
>>>>
>>>>
>>>>
>>>>
>>>> On Mon, Aug 29, 2016 at 8:16 PM, thierry bordaz
>>>> <tbordaz at redhat.com <mailto:tbordaz at redhat.com>> wrote:
>>>>
>>>> Hi Rakesh,
>>>>
>>>> Those tuning may depend on the memory available
>>>> on your machine.
>>>> nsslapd-cachememsize allows the entry cache to
>>>> consume up to 200Mb but its memory footprint is
>>>> known to go above.
>>>> 200Mb both looks pretty good to me. How large
>>>> is your machine ? What is your version of 389-ds ?
>>>>
>>>> Those warnings do not change your settings. It
>>>> just raise that entry cache of 'ipaca' and
>>>> 'retrocl' are small but it is fine. The size of
>>>> the entry cache is important mostly in userRoot.
>>>> You may double check the actual values, after
>>>> restart, with ldapsearch on
>>>> 'cn=userRoot,cn=ldbm
>>>> database,cn=plugins,cn=config' and
>>>> 'cn=config,cn=ldbm database,cn=plugins,cn=config'.
>>>>
>>>> A step is to know what will be response time of
>>>> DS to know if it is responsible of the hang or not.
>>>> The logs and possibly pstack during those
>>>> intermittent hangs will help to determine that.
>>>>
>>>> regards
>>>> thierry
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> On 08/29/2016 04:25 PM, Rakesh Rajasekharan wrote:
>>>>> I tried increasing the nsslapd-dbcachesize and
>>>>> nsslapd-cachememsize in my QA envs to 200MB.
>>>>>
>>>>> However, in my log files, I still see this message
>>>>> [29/Aug/2016:04:34:37 +0000] - WARNING: ipaca:
>>>>> entry cache size 10485760B is less than db
>>>>> size 11599872B; We recommend to increase the
>>>>> entry cache size nsslapd-cachememsize.
>>>>> [29/Aug/2016:04:34:37 +0000] - WARNING:
>>>>> changelog: entry cache size 2097152B is less
>>>>> than db size 441647104B; We recommend to
>>>>> increase the entry cache size
>>>>> nsslapd-cachememsize.
>>>>>
>>>>> these are my ldif files that i used to modify
>>>>> the values
>>>>> modify entry cache size
>>>>> cat modify-cache-mem-size.ldif
>>>>> dn: cn=userRoot,cn=ldbm
>>>>> database,cn=plugins,cn=config
>>>>> changetype: modify
>>>>> replace: nsslapd-cachememsize
>>>>> nsslapd-cachememsize: 209715200
>>>>>
>>>>> modify db cache size
>>>>> cat modfy-db-cache-size.ldif
>>>>> dn: cn=config,cn=ldbm
>>>>> database,cn=plugins,cn=config
>>>>> changetype: modify
>>>>> replace: nsslapd-dbcachesize
>>>>> nsslapd-dbcachesize: 209715200
>>>>>
>>>>> After modifying , i restarted IPA services
>>>>>
>>>>> Is there anything else that I need to take
>>>>> care of as the logs suggest its still not
>>>>> getting the updated values
>>>>>
>>>>> Thanks
>>>>> Rakesh
>>>>>
>>>>> On Mon, Aug 29, 2016 at 6:07 PM, Rakesh
>>>>> Rajasekharan <rakesh.rajasekharan at gmail.com
>>>>> <mailto:rakesh.rajasekharan at gmail.com>> wrote:
>>>>>
>>>>> Hi Thierry,
>>>>>
>>>>> Coz of the issues we had to revert back to
>>>>> earlier running openldap in production.
>>>>>
>>>>> I have now done a few TCP related changes
>>>>> in sysctl.conf and have also increased the
>>>>> nsslapd-dbcachesize and
>>>>> nsslapd-cachememsize to 200MB
>>>>>
>>>>> I will again start migrating hosts back to
>>>>> IPA and see if I face the earlier issue.
>>>>>
>>>>> I will update back once I have something
>>>>>
>>>>>
>>>>> Thanks,
>>>>> Rakesh
>>>>>
>>>>>
>>>>>
>>>>> On Thu, Aug 25, 2016 at 2:17 PM, thierry
>>>>> bordaz <tbordaz at redhat.com
>>>>> <mailto:tbordaz at redhat.com>> wrote:
>>>>>
>>>>>
>>>>>
>>>>> On 08/25/2016 10:15 AM, Rakesh
>>>>> Rajasekharan wrote:
>>>>>> All of the troubleshooting seems fine.
>>>>>>
>>>>>>
>>>>>> However, Running libconv.pl
>>>>>> <http://libconv.pl> gives me this output
>>>>>>
>>>>>> ----- Recommendations -----
>>>>>>
>>>>>> 1. You have unindexed components,
>>>>>> this can be caused from a search on
>>>>>> an unindexed attribute, or your
>>>>>> returned results exceeded the
>>>>>> allidsthreshold. Unindexed components
>>>>>> are not recommended. To refuse
>>>>>> unindexed searches, switch
>>>>>> 'nsslapd-require-index' to 'on' under
>>>>>> your database entry (e.g.
>>>>>> cn=UserRoot,cn=ldbm
>>>>>> database,cn=plugins,cn=config).
>>>>>>
>>>>>> 2. You have a significant
>>>>>> difference between binds and unbinds.
>>>>>> You may want to investigate this
>>>>>> difference.
>>>>>>
>>>>>>
>>>>>> I feel, this could be a pointer to
>>>>>> things going slow.. and IPA hanging.
>>>>>> I think i now have something that I
>>>>>> can try and nail down this issue.
>>>>>>
>>>>>> On a sidenote, I was earlier running
>>>>>> openldap and migrated over to Freeipa,
>>>>>>
>>>>>> Thanks
>>>>>> Rakesh
>>>>>>
>>>>>>
>>>>>>
>>>>>> On Wed, Aug 24, 2016 at 12:38 PM,
>>>>>> Petr Spacek <pspacek at redhat.com
>>>>>> <mailto:pspacek at redhat.com>> wrote:
>>>>>>
>>>>>> On 23.8.2016 18:44, Rakesh
>>>>>> Rajasekharan wrote:
>>>>>> > I think thers something
>>>>>> seriously wrong with my system
>>>>>> >
>>>>>> > not able to run any IPA commands
>>>>>> >
>>>>>> > klist
>>>>>> > Ticket cache:
>>>>>> KEYRING:persistent:0:0
>>>>>> > Default principal:
>>>>>> admin at XYZ.COM <mailto:admin at XYZ.COM>
>>>>>> >
>>>>>> > Valid starting Expires Service
>>>>>> principal
>>>>>> > 2016-08-23T16:26:36
>>>>>> 2016-08-24T16:26:22
>>>>>> krbtgt/XYZ.COM at XYZ.COM
>>>>>> <mailto:XYZ.COM at XYZ.COM>
>>>>>> >
>>>>>> >
>>>>>> > [root at prod-ipa-master-1a :~]
>>>>>> ipactl status
>>>>>> > Directory Service: RUNNING
>>>>>> > krb5kdc Service: RUNNING
>>>>>> > kadmin Service: RUNNING
>>>>>> > ipa_memcached Service: RUNNING
>>>>>> > httpd Service: RUNNING
>>>>>> > pki-tomcatd Service: RUNNING
>>>>>> > ipa-otpd Service: RUNNING
>>>>>> > ipa: INFO: The ipactl command
>>>>>> was successful
>>>>>> >
>>>>>> >
>>>>>> >
>>>>>> > [root at prod-ipa-master :~] ipa
>>>>>> user-find p-testuser
>>>>>> > ipa: ERROR: Kerberos error:
>>>>>> ('Unspecified GSS failure. Minor
>>>>>> code may
>>>>>> > provide more information',
>>>>>> 851968)/("Cannot contact any KDC
>>>>>> for realm '
>>>>>> > XYZ.COM <http://XYZ.COM>'",
>>>>>> -1765328228)
>>>>>>
>>>>>
>>>>> Hi Rakesh,
>>>>>
>>>>> Having a reproducible test case
>>>>> would you rerun the command above.
>>>>> During its processing you may
>>>>> monitor DS process load (top). If
>>>>> it is high, you may get some
>>>>> pstacks of it.
>>>>> Also would you attach the part of
>>>>> DS access logs taken during the
>>>>> command.
>>>>>
>>>>> regards
>>>>> thierry
>>>>>
>>>>>> >
>>>>>>
>>>>>> This is weird because the server
>>>>>> seems to be up.
>>>>>>
>>>>>> Please follow
>>>>>> http://www.freeipa.org/page/Troubleshooting#Authentication.2FKerberos
>>>>>>
>>>>>> Petr^2 Spacek
>>>>>>
>>>>>> >
>>>>>> >
>>>>>> > Thanks
>>>>>> >
>>>>>> > Rakesh
>>>>>> >
>>>>>> > On Tue, Aug 23, 2016 at 10:01
>>>>>> PM, Rakesh Rajasekharan <
>>>>>> > rakesh.rajasekharan at gmail.com
>>>>>> <mailto:rakesh.rajasekharan at gmail.com>>
>>>>>> wrote:
>>>>>> >
>>>>>> >> i changed the loggin level to
>>>>>> 4 . Modifying nsslapd-accesslog-level
>>>>>> >>
>>>>>> >> But, the hang is still there.
>>>>>> though I dont see the sigfault now
>>>>>> >>
>>>>>> >>
>>>>>> >>
>>>>>> >>
>>>>>> >> On Tue, Aug 23, 2016 at 9:02
>>>>>> PM, Rakesh Rajasekharan <
>>>>>> >> rakesh.rajasekharan at gmail.com
>>>>>> <mailto:rakesh.rajasekharan at gmail.com>>
>>>>>> wrote:
>>>>>> >>
>>>>>> >>> My disk was getting filled
>>>>>> too fast
>>>>>> >>>
>>>>>> >>> logs under /var/log/dirsrv
>>>>>> was coming around 5 gb quickly
>>>>>> filling up
>>>>>> >>>
>>>>>> >>> Is there a way to make the
>>>>>> logging less verbose
>>>>>> >>>
>>>>>> >>>
>>>>>> >>>
>>>>>> >>> On Tue, Aug 23, 2016 at 6:41
>>>>>> PM, Petr Spacek
>>>>>> <pspacek at redhat.com
>>>>>> <mailto:pspacek at redhat.com>> wrote:
>>>>>> >>>
>>>>>> >>>> On 23.8.2016 15:07, Rakesh
>>>>>> Rajasekharan wrote:
>>>>>> >>>>> I was able to fix that may
>>>>>> be temporarily... when i checked the
>>>>>> >>>> network..
>>>>>> >>>>> there was another process
>>>>>> that was running and consuming a
>>>>>> lot of
>>>>>> >>>> network (
>>>>>> >>>>> i have no idea who did
>>>>>> that. I need to seriously start
>>>>>> restricting
>>>>>> >>>> people
>>>>>> >>>>> access to this machine )
>>>>>> >>>>>
>>>>>> >>>>> after killing that
>>>>>> perfomance improved drastically
>>>>>> >>>>>
>>>>>> >>>>> But now, suddenly I started
>>>>>> experiencing the same hang.
>>>>>> >>>>>
>>>>>> >>>>> This time , I gert the
>>>>>> following error when checked dmesg
>>>>>> >>>>>
>>>>>> >>>>> [ 301.236976]
>>>>>> ns-slapd[3124]: segfault at 0 ip
>>>>>> 00007f1de416951c sp
>>>>>> >>>>> 00007f1dee1dba70 error 4 in
>>>>>> libcos-plugin.so[7f1de4166000+b000]
>>>>>> >>>>> [ 1116.248431] TCP:
>>>>>> request_sock_TCP: Possible SYN
>>>>>> flooding on port 88.
>>>>>> >>>>> Sending cookies. Check SNMP
>>>>>> counters.
>>>>>> >>>>> [11831.397037]
>>>>>> ns-slapd[22550]: segfault at 0 ip
>>>>>> 00007f533d82251c sp
>>>>>> >>>>> 00007f5347894a70 error 4 in
>>>>>> libcos-plugin.so[7f533d81f000+b000]
>>>>>> >>>>> [11832.727989]
>>>>>> ns-slapd[22606]: segfault at 0 ip
>>>>>> 00007f6231eb951c sp
>>>>>> >>>>> 00007f623bf2ba70 error 4 in
>>>>>> libcos-plugin.so[7f6231eb6000+b00
>>>>>> >>>>
>>>>>> >>>> Okay, this one is serious.
>>>>>> The LDAP server crashed.
>>>>>> >>>>
>>>>>> >>>> 1. Make sure all your
>>>>>> packages are up-to-date.
>>>>>> >>>>
>>>>>> >>>> Please see
>>>>>> >>>>
>>>>>> http://directory.fedoraproject.org/docs/389ds/FAQ/faq.html#d
>>>>>> >>>> ebugging-crashes
>>>>>> >>>> for further instructions how
>>>>>> to debug this.
>>>>>> >>>>
>>>>>> >>>> Petr^2 Spacek
>>>>>> >>>>
>>>>>> >>>>>
>>>>>> >>>>> and in
>>>>>> /var/log/dirsrv/example-com/errors
>>>>>> >>>>>
>>>>>> >>>>> [23/Aug/2016:12:49:36
>>>>>> +0000] DSRetroclPlugin -
>>>>>> delete_changerecord:
>>>>>> >>>> could
>>>>>> >>>>> not delete change record
>>>>>> 3291138 (rc: 32)
>>>>>> >>>>> [23/Aug/2016:12:49:36
>>>>>> +0000] DSRetroclPlugin -
>>>>>> delete_changerecord:
>>>>>> >>>> could
>>>>>> >>>>> not delete change record
>>>>>> 3291139 (rc: 32)
>>>>>> >>>>> [23/Aug/2016:12:49:36
>>>>>> +0000] DSRetroclPlugin -
>>>>>> delete_changerecord:
>>>>>> >>>> could
>>>>>> >>>>> not delete change record
>>>>>> 3291140 (rc: 32)
>>>>>> >>>>> [23/Aug/2016:12:49:36
>>>>>> +0000] DSRetroclPlugin -
>>>>>> delete_changerecord:
>>>>>> >>>> could
>>>>>> >>>>> not delete change record
>>>>>> 3291141 (rc: 32)
>>>>>> >>>>> [23/Aug/2016:12:49:36
>>>>>> +0000] DSRetroclPlugin -
>>>>>> delete_changerecord:
>>>>>> >>>> could
>>>>>> >>>>> not delete change record
>>>>>> 3291142 (rc: 32)
>>>>>> >>>>> [23/Aug/2016:12:49:36
>>>>>> +0000] DSRetroclPlugin -
>>>>>> delete_changerecord:
>>>>>> >>>> could
>>>>>> >>>>> not delete change record
>>>>>> 3291143 (rc: 32)
>>>>>> >>>>> [23/Aug/2016:12:49:36
>>>>>> +0000] DSRetroclPlugin -
>>>>>> delete_changerecord:
>>>>>> >>>> could
>>>>>> >>>>> not delete change record
>>>>>> 3291144 (rc: 32)
>>>>>> >>>>> [23/Aug/2016:12:49:36
>>>>>> +0000] DSRetroclPlugin -
>>>>>> delete_changerecord:
>>>>>> >>>> could
>>>>>> >>>>> not delete change record
>>>>>> 3291145 (rc: 32)
>>>>>> >>>>> [23/Aug/2016:12:49:50
>>>>>> +0000] - Retry count exceeded in
>>>>>> delete
>>>>>> >>>>> [23/Aug/2016:12:49:50
>>>>>> +0000] DSRetroclPlugin -
>>>>>> delete_changerecord:
>>>>>> >>>> could
>>>>>> >>>>> not delete change record
>>>>>> 3292734 (rc: 51)
>>>>>> >>>>>
>>>>>> >>>>>
>>>>>> >>>>> Can i do something about
>>>>>> this error.. I treid to restart
>>>>>> ipa a couple
>>>>>> >>>> of
>>>>>> >>>>> time but that did not help
>>>>>> >>>>>
>>>>>> >>>>> Thanks
>>>>>> >>>>> Rakesh
>>>>>> >>>>>
>>>>>> >>>>> On Mon, Aug 22, 2016 at
>>>>>> 2:27 PM, Petr Spacek
>>>>>> <pspacek at redhat.com
>>>>>> <mailto:pspacek at redhat.com>>
>>>>>> >>>> wrote:
>>>>>> >>>>>
>>>>>> >>>>>> On 19.8.2016 19:32, Rakesh
>>>>>> Rajasekharan wrote:
>>>>>> >>>>>>> I am running my set up on
>>>>>> AWS cloud, and entropy is low at
>>>>>> around
>>>>>> >>>> 180 .
>>>>>> >>>>>>>
>>>>>> >>>>>>> I plan to increase it bu
>>>>>> installing haveged . But, would
>>>>>> low entropy
>>>>>> >>>> by
>>>>>> >>>>>> any
>>>>>> >>>>>>> chance cause this issue
>>>>>> of intermittent hang .
>>>>>> >>>>>>> Also, the hang is mostly
>>>>>> observed when registering around
>>>>>> 20 clients
>>>>>> >>>>>>> together
>>>>>> >>>>>>
>>>>>> >>>>>> Possibly, I'm not sure. If
>>>>>> you want to dig into this, I
>>>>>> would do this:
>>>>>> >>>>>> 1. look what process hangs
>>>>>> on client (using pstree command
>>>>>> or so)
>>>>>> >>>>>> $ pstree
>>>>>> >>>>>>
>>>>>> >>>>>> 2. look to what server and
>>>>>> port is the hanging client
>>>>>> connected to
>>>>>> >>>>>> $ lsof -p <PID of the
>>>>>> hanging process>
>>>>>> >>>>>>
>>>>>> >>>>>> 3. jump to server and see
>>>>>> what process is bound to the
>>>>>> target port
>>>>>> >>>>>> $ netstat -pn
>>>>>> >>>>>>
>>>>>> >>>>>> 4. see where the process
>>>>>> if hanging
>>>>>> >>>>>> $ strace -p <PID of the
>>>>>> hanging process>
>>>>>> >>>>>>
>>>>>> >>>>>> I hope it helps.
>>>>>> >>>>>>
>>>>>> >>>>>> Petr^2 Spacek
>>>>>> >>>>>>
>>>>>> >>>>>>> On Fri, Aug 19, 2016 at
>>>>>> 7:24 PM, Rakesh Rajasekharan <
>>>>>> >>>>>>>
>>>>>> rakesh.rajasekharan at gmail.com
>>>>>> <mailto:rakesh.rajasekharan at gmail.com>>
>>>>>> wrote:
>>>>>> >>>>>>>
>>>>>> >>>>>>>> yes there seems to be
>>>>>> something thats worrying.. I have
>>>>>> faced this
>>>>>> >>>> today
>>>>>> >>>>>>>> as well.
>>>>>> >>>>>>>> There are few hosts
>>>>>> around 280 odd left and when i
>>>>>> try adding them
>>>>>> >>>> to
>>>>>> >>>>>> IPA
>>>>>> >>>>>>>> , the slowness begins..
>>>>>> >>>>>>>>
>>>>>> >>>>>>>> all the ipa commands
>>>>>> like ipa user-find.. etc becomes
>>>>>> very slow in
>>>>>> >>>>>>>> responding.
>>>>>> >>>>>>>>
>>>>>> >>>>>>>> the SYNC_RECV are not
>>>>>> many though just around 80-90 and
>>>>>> today that
>>>>>> >>>> was
>>>>>> >>>>>>>> around 20 only
>>>>>> >>>>>>>>
>>>>>> >>>>>>>>
>>>>>> >>>>>>>> I have for now increased
>>>>>> tcp_max_syn_backlog to 5000.
>>>>>> >>>>>>>> For now the slowness
>>>>>> seems to have gone.. but I will
>>>>>> do a try
>>>>>> >>>> adding the
>>>>>> >>>>>>>> clients again tomorrow
>>>>>> and see how it goes
>>>>>> >>>>>>>>
>>>>>> >>>>>>>> Thanks
>>>>>> >>>>>>>> Rakesh
>>>>>> >>>>>>>>
>>>>>> >>>>>>>> The issues
>>>>>> >>>>>>>>
>>>>>> >>>>>>>> On Fri, Aug 19, 2016 at
>>>>>> 12:58 PM, Petr Spacek
>>>>>> <pspacek at redhat.com
>>>>>> <mailto:pspacek at redhat.com>>
>>>>>> >>>>>> wrote:
>>>>>> >>>>>>>>
>>>>>> >>>>>>>>> On 18.8.2016 17:23,
>>>>>> Rakesh Rajasekharan wrote:
>>>>>> >>>>>>>>>> Hi
>>>>>> >>>>>>>>>>
>>>>>> >>>>>>>>>> I am migrating to
>>>>>> freeipa from openldap and have
>>>>>> around 4000
>>>>>> >>>> clients
>>>>>> >>>>>>>>>>
>>>>>> >>>>>>>>>> I had openned a
>>>>>> another thread on that, but chose
>>>>>> to start a new
>>>>>> >>>> one
>>>>>> >>>>>>>>> here
>>>>>> >>>>>>>>>> as its a separate issue
>>>>>> >>>>>>>>>>
>>>>>> >>>>>>>>>> I was able to change
>>>>>> the nssslapd-maxdescriptors
>>>>>> adding an ldif
>>>>>> >>>> file
>>>>>> >>>>>>>>>>
>>>>>> >>>>>>>>>> cat nsslapd-modify.ldif
>>>>>> >>>>>>>>>> dn: cn=config
>>>>>> >>>>>>>>>> changetype: modify
>>>>>> >>>>>>>>>> replace:
>>>>>> nsslapd-maxdescriptors
>>>>>> >>>>>>>>>>
>>>>>> nsslapd-maxdescriptors: 17000
>>>>>> >>>>>>>>>>
>>>>>> >>>>>>>>>> and running the
>>>>>> ldapmodify command
>>>>>> >>>>>>>>>>
>>>>>> >>>>>>>>>> I have now started
>>>>>> moving clients running an
>>>>>> openldap to Freeipa
>>>>>> >>>> and
>>>>>> >>>>>>>>> have
>>>>>> >>>>>>>>>> today moved close to
>>>>>> 2000 clients
>>>>>> >>>>>>>>>>
>>>>>> >>>>>>>>>> However, I have
>>>>>> noticed that IPA hangs
>>>>>> intermittently.
>>>>>> >>>>>>>>>>
>>>>>> >>>>>>>>>> running a kinit admin
>>>>>> returns the below error
>>>>>> >>>>>>>>>> kinit: Generic error
>>>>>> (see e-text) while getting initial
>>>>>> >>>> credentials
>>>>>> >>>>>>>>>>
>>>>>> >>>>>>>>>> from the
>>>>>> /var/log/messages, I see this entry
>>>>>> >>>>>>>>>>
>>>>>> >>>>>>>>>> prod-ipa-master-int
>>>>>> kernel: [104090.315801] TCP:
>>>>>> >>>> request_sock_TCP:
>>>>>> >>>>>>>>>> Possible SYN flooding
>>>>>> on port 88. Sending cookies.
>>>>>> Check SNMP
>>>>>> >>>>>> counters.
>>>>>> >>>>>>>>>
>>>>>> >>>>>>>>> I would be worried
>>>>>> about this message. Maybe
>>>>>> kernel/firewall is
>>>>>> >>>> doing
>>>>>> >>>>>>>>> something fishy behind
>>>>>> your back and blocking some
>>>>>> connections or
>>>>>> >>>> so.
>>>>>> >>>>>>>>>
>>>>>> >>>>>>>>> Petr^2 Spacek
>>>>>> >>>>>>>>>
>>>>>> >>>>>>>>>
>>>>>> >>>>>>>>>> Aug 18 13:00:01
>>>>>> prod-ipa-master-int systemd[1]:
>>>>>> Started Session
>>>>>> >>>> 4885
>>>>>> >>>>>> of
>>>>>> >>>>>>>>>> user root.
>>>>>> >>>>>>>>>> Aug 18 13:00:01
>>>>>> prod-ipa-master-int systemd[1]:
>>>>>> Starting Session
>>>>>> >>>> 4885
>>>>>> >>>>>> of
>>>>>> >>>>>>>>>> user root.
>>>>>> >>>>>>>>>> Aug 18 13:01:01
>>>>>> prod-ipa-master-int systemd[1]:
>>>>>> Started Session
>>>>>> >>>> 4886
>>>>>> >>>>>> of
>>>>>> >>>>>>>>>> user root.
>>>>>> >>>>>>>>>> Aug 18 13:01:01
>>>>>> prod-ipa-master-int systemd[1]:
>>>>>> Starting Session
>>>>>> >>>> 4886
>>>>>> >>>>>> of
>>>>>> >>>>>>>>>> user root.
>>>>>> >>>>>>>>>> Aug 18 13:02:40
>>>>>> prod-ipa-master-int
>>>>>> python[28984]: ansible-command
>>>>>> >>>>>>>>> Invoked
>>>>>> >>>>>>>>>> with creates=None
>>>>>> executable=None shell=True args=
>>>>>> removes=None
>>>>>> >>>>>>>>> warn=True
>>>>>> >>>>>>>>>> chdir=None
>>>>>> >>>>>>>>>> Aug 18 13:04:37
>>>>>> prod-ipa-master-int sssd_be:
>>>>>> GSSAPI Error:
>>>>>> >>>> Unspecified
>>>>>> >>>>>>>>> GSS
>>>>>> >>>>>>>>>> failure. Minor code
>>>>>> may provide more information (KDC
>>>>>> returned
>>>>>> >>>> error
>>>>>> >>>>>>>>>> string: PROCESS_TGS)
>>>>>> >>>>>>>>>>
>>>>>> >>>>>>>>>> Could it be possible
>>>>>> that its due to the initial load
>>>>>> of adding
>>>>>> >>>> the
>>>>>> >>>>>>>>> clients
>>>>>> >>>>>>>>>> or is there something
>>>>>> else that I need to take care of.
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>>
>>>>
>>>>
>>>>
>>>>
>>>
>>>
>>> --
>>> Manage your subscription for the Freeipa-users
>>> mailing list:
>>> https://www.redhat.com/mailman/listinfo/freeipa-users <https://www.redhat.com/mailman/listinfo/freeipa-users>
>>> Go to http://freeipa.org for more info on the project
>>>
>>>
>>>
>>>
>>>
>>
>>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/freeipa-users/attachments/20160905/4fc9dfd4/attachment.htm>
More information about the Freeipa-users
mailing list