[Freeipa-users] Freeipa 4.2.0 hangs intermittently
thierry bordaz
tbordaz at redhat.com
Mon Sep 5 12:37:53 UTC 2016
Hi Rakesh,
Looking at this monitoring I do not see any reason to increase the
number of worker threads.
It shows that (before the hang), only one thread (out of 30) was busy.
When the hang occurred, 7 threads (out of 30) were busy.
regards
thierry
On 09/05/2016 02:14 PM, Rakesh Rajasekharan wrote:
> btw the values for opscompleted and opsinitiated are
>
> ldapsearch -D "cn=directory manager" -w xxxx -LLL -b "cn=monitor" -s
> base opsinitiated opscompleted
> dn: cn=monitor
> opsinitiated: 25655
> opscompleted: 25654
>
> Thanks
>
> On Mon, Sep 5, 2016 at 5:35 PM, Rakesh Rajasekharan
> <rakesh.rajasekharan at gmail.com <mailto:rakesh.rajasekharan at gmail.com>>
> wrote:
>
> Hi Thierry,
>
> I was able to now reporduce the issue..
>
> the ns-slapd process just got hung,.. however, no core file was
> generate..
>
> I manually created a coredump using gcore and then took a strace of it
> this is strace from the coredump generated
>
> GNU gdb (GDB) Red Hat Enterprise Linux 7.6.1-80.el7
> Copyright (C) 2013 Free Software Foundation, Inc.
> License GPLv3+: GNU GPL version 3 or later
> <http://gnu.org/licenses/gpl.html <http://gnu.org/licenses/gpl.html>>
> This is free software: you are free to change and redistribute it.
> There is NO WARRANTY, to the extent permitted by law. Type "show
> copying"
> and "show warranty" for details.
> This GDB was configured as "x86_64-redhat-linux-gnu".
> For bug reporting instructions, please see:
> <http://www.gnu.org/software/gdb/bugs/
> <http://www.gnu.org/software/gdb/bugs/>>...
> Reading symbols from /usr/sbin/ns-slapd...Reading symbols from
> /usr/sbin/ns-slapd...(no debugging symbols found)...done.
> (no debugging symbols found)...done.
> [New LWP 105159]
> [New LWP 105160]
> [New LWP 105161]
> [New LWP 105162]
> [New LWP 105163]
> [New LWP 105164]
> [New LWP 105165]
> [New LWP 105166]
> [New LWP 105167]
> [New LWP 105168]
> [New LWP 105171]
> [New LWP 105172]
> [New LWP 105173]
> [New LWP 105174]
> [New LWP 105175]
> [New LWP 105176]
> [New LWP 105177]
> [New LWP 105178]
> [New LWP 105179]
> [New LWP 105180]
> [New LWP 105181]
> [New LWP 105182]
> [New LWP 105183]
> [New LWP 105184]
> [New LWP 105185]
> [New LWP 105186]
> [New LWP 105187]
> [New LWP 105188]
> [New LWP 105189]
> [New LWP 105190]
> [New LWP 105191]
> [New LWP 105192]
> [New LWP 105193]
> [New LWP 105194]
> [New LWP 105195]
> [New LWP 105196]
> [New LWP 105197]
> [New LWP 105198]
> [New LWP 105199]
> [New LWP 105200]
> [New LWP 105201]
> [New LWP 105203]
> [New LWP 105215]
> [New LWP 105158]
> [Thread debugging using libthread_db enabled]
> Using host libthread_db library "/lib64/libthread_db.so.1".
> Core was generated by `/usr/sbin/ns-slapd'.
> #0 0x00007f1b48ae69b3 in select () from /lib64/libc.so.6
>
> Thread 44 (Thread 0x7f1b4b6d9840 (LWP 105158)):
> #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 () from
> /lib64/libpthread.so.0
> No symbol table info available.
> #1 0x00007f1b4941b2c3 in PR_EnterMonitor () from /lib64/libnspr4.so
> No symbol table info available.
> #2 0x00007f1b4b7125ed in setup_pr_read_pds ()
> No symbol table info available.
> #3 0x00007f1b4b715fd9 in slapd_daemon ()
> No symbol table info available.
> #4 0x00007f1b4b707c6c in main ()
> No symbol table info available.
>
> Thread 43 (Thread 0x7f1a5df0f700 (LWP 105215)):
> #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 () from
> /lib64/libpthread.so.0
> No symbol table info available.
> #1 0x00007f1b4941b050 in PR_WaitCondVar () from /lib64/libnspr4.so
> No symbol table info available.
> #2 0x00007f1b4b71df25 in ps_send_results ()
> No symbol table info available.
> #3 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so
> No symbol table info available.
> #4 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0
> No symbol table info available.
> #5 0x00007f1b48aef28d in clone () from /lib64/libc.so.6
> No symbol table info available.
>
> Thread 42 (Thread 0x7f1b03fe7700 (LWP 105203)):
> #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 () from
> /lib64/libpthread.so.0
> No symbol table info available.
> #1 0x00007f1b4941b050 in PR_WaitCondVar () from /lib64/libnspr4.so
> No symbol table info available.
> #2 0x00007f1b4b71df25 in ps_send_results ()
> No symbol table info available.
> #3 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so
> No symbol table info available.
> #4 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0
> No symbol table info available.
> #5 0x00007f1b48aef28d in clone () from /lib64/libc.so.6
> No symbol table info available.
>
> Thread 41 (Thread 0x7f1b047e8700 (LWP 105201)):
> #0 0x00007f1b48ae69b3 in select () from /lib64/libc.so.6
> No symbol table info available.
> #1 0x00007f1b4b2710e9 in DS_Sleep () from
> /usr/lib64/dirsrv/libslapd.so.0
> No symbol table info available.
> #2 0x00007f1b4b712325 in time_thread ()
> No symbol table info available.
> #3 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so
> No symbol table info available.
> #4 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0
> No symbol table info available.
> #5 0x00007f1b48aef28d in clone () from /lib64/libc.so.6
> No symbol table info available.
>
> Thread 40 (Thread 0x7f1b04fe9700 (LWP 105200)):
> #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 () from
> /lib64/libpthread.so.0
> No symbol table info available.
> #1 0x00007f1b433212f3 in __db_hybrid_mutex_suspend () from
> /lib64/libdb-5.3.so <http://libdb-5.3.so>
> No symbol table info available.
> #2 0x00007f1b43320640 in __db_tas_mutex_lock () from
> /lib64/libdb-5.3.so <http://libdb-5.3.so>
> No symbol table info available.
> #3 0x00007f1b4343ae1b in __memp_fget () from /lib64/libdb-5.3.so
> <http://libdb-5.3.so>
> No symbol table info available.
> #4 0x00007f1b4333e211 in __bam_search () from /lib64/libdb-5.3.so
> <http://libdb-5.3.so>
> No symbol table info available.
> #5 0x00007f1b43329256 in __bamc_search () from
> /lib64/libdb-5.3.so <http://libdb-5.3.so>
> No symbol table info available.
> #6 0x00007f1b4332d6c4 in __bamc_put () from /lib64/libdb-5.3.so
> <http://libdb-5.3.so>
> No symbol table info available.
> #7 0x00007f1b433e46d5 in __dbc_iput () from /lib64/libdb-5.3.so
> <http://libdb-5.3.so>
> No symbol table info available.
> #8 0x00007f1b433dfa0e in __db_put () from /lib64/libdb-5.3.so
> <http://libdb-5.3.so>
> No symbol table info available.
> #9 0x00007f1b433f4fa4 in __db_put_pp () from /lib64/libdb-5.3.so
> <http://libdb-5.3.so>
> No symbol table info available.
> #10 0x00007f1b3e8cf4cf in idl_new_insert_key () from
> /usr/lib64/dirsrv/plugins/libback-ldbm.so
> No symbol table info available.
> #11 0x00007f1b3e8de5b5 in addordel_values_sv.isra.3 () from
> /usr/lib64/dirsrv/plugins/libback-ldbm.so
> No symbol table info available.
> #12 0x00007f1b3e8df25a in index_addordel_values_ext_sv () from
> /usr/lib64/dirsrv/plugins/libback-ldbm.so
> No symbol table info available.
> #13 0x00007f1b3e8df694 in index_addordel_values_sv () from
> /usr/lib64/dirsrv/plugins/libback-ldbm.so
> No symbol table info available.
> #14 0x00007f1b3e8df75a in index_addordel_entry () from
> /usr/lib64/dirsrv/plugins/libback-ldbm.so
> No symbol table info available.
> #15 0x00007f1b3e8e25fa in ldbm_back_add () from
> /usr/lib64/dirsrv/plugins/libback-ldbm.so
> No symbol table info available.
> #16 0x00007f1b4b1e4cca in op_shared_add () from
> /usr/lib64/dirsrv/libslapd.so.0
> No symbol table info available.
> #17 0x00007f1b4b1e6020 in do_add () from
> /usr/lib64/dirsrv/libslapd.so.0
> No symbol table info available.
> #18 0x00007f1b4b711303 in connection_threadmain ()
> No symbol table info available.
> #19 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so
> No symbol table info available.
> #20 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0
> No symbol table info available.
> #21 0x00007f1b48aef28d in clone () from /lib64/libc.so.6
> No symbol table info available.
>
> Thread 39 (Thread 0x7f1b057ea700 (LWP 105199)):
> #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 () from
> /lib64/libpthread.so.0
> No symbol table info available.
> #1 0x00007f1b4941b050 in PR_WaitCondVar () from /lib64/libnspr4.so
> No symbol table info available.
> #2 0x00007f1b4b70efee in connection_wait_for_new_work ()
> No symbol table info available.
> #3 0x00007f1b4b710719 in connection_threadmain ()
> No symbol table info available.
> #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so
> No symbol table info available.
> #5 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0
> No symbol table info available.
> #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6
> No symbol table info available.
>
> Thread 38 (Thread 0x7f1b05feb700 (LWP 105198)):
> #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 () from
> /lib64/libpthread.so.0
> No symbol table info available.
> #1 0x00007f1b4941b050 in PR_WaitCondVar () from /lib64/libnspr4.so
> No symbol table info available.
> #2 0x00007f1b4b70efee in connection_wait_for_new_work ()
> No symbol table info available.
> #3 0x00007f1b4b710719 in connection_threadmain ()
> No symbol table info available.
> #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so
> No symbol table info available.
> #5 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0
> No symbol table info available.
> #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6
> No symbol table info available.
>
> Thread 37 (Thread 0x7f1b067ec700 (LWP 105197)):
> #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 () from
> /lib64/libpthread.so.0
> No symbol table info available.
> #1 0x00007f1b4941b050 in PR_WaitCondVar () from /lib64/libnspr4.so
> No symbol table info available.
> #2 0x00007f1b4b70efee in connection_wait_for_new_work ()
> No symbol table info available.
> #3 0x00007f1b4b710719 in connection_threadmain ()
> No symbol table info available.
> #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so
> No symbol table info available.
> #5 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0
> No symbol table info available.
> #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6
> No symbol table info available.
>
> Thread 36 (Thread 0x7f1b06fed700 (LWP 105196)):
> #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 () from
> /lib64/libpthread.so.0
> No symbol table info available.
> #1 0x00007f1b4941b050 in PR_WaitCondVar () from /lib64/libnspr4.so
> No symbol table info available.
> #2 0x00007f1b4b70efee in connection_wait_for_new_work ()
> No symbol table info available.
> #3 0x00007f1b4b710719 in connection_threadmain ()
> No symbol table info available.
> #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so
> No symbol table info available.
> #5 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0
> No symbol table info available.
> #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6
> No symbol table info available.
>
> Thread 35 (Thread 0x7f1b077ee700 (LWP 105195)):
> #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 () from
> /lib64/libpthread.so.0
> No symbol table info available.
> #1 0x00007f1b4941b2c3 in PR_EnterMonitor () from /lib64/libnspr4.so
> No symbol table info available.
> #2 0x00007f1b3e8c23e6 in dblayer_txn_begin () from
> /usr/lib64/dirsrv/plugins/libback-ldbm.so
> No symbol table info available.
> #3 0x00007f1b3e8fe007 in ldbm_back_modify () from
> /usr/lib64/dirsrv/plugins/libback-ldbm.so
> No symbol table info available.
> #4 0x00007f1b4b22b541 in op_shared_modify () from
> /usr/lib64/dirsrv/libslapd.so.0
> No symbol table info available.
> #5 0x00007f1b4b22c004 in modify_internal_pb () from
> /usr/lib64/dirsrv/libslapd.so.0
> No symbol table info available.
> #6 0x00007f1b3eb48b60 in ipapwd_apply_mods () from
> /usr/lib64/dirsrv/plugins/libipa_pwd_extop.so
> No symbol table info available.
> #7 0x00007f1b3eb4c8e0 in store_new_keys () from
> /usr/lib64/dirsrv/plugins/libipa_pwd_extop.so
> No symbol table info available.
> #8 0x00007f1b3eb4ead0 in ipapwd_extop () from
> /usr/lib64/dirsrv/plugins/libipa_pwd_extop.so
> No symbol table info available.
> #9 0x00007f1b4b23bba2 in plugin_call_exop_plugins () from
> /usr/lib64/dirsrv/libslapd.so.0
> No symbol table info available.
> #10 0x00007f1b4b717efe in do_extended ()
> No symbol table info available.
> #11 0x00007f1b4b71141a in connection_threadmain ()
> No symbol table info available.
> #12 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so
> No symbol table info available.
> #13 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0
> No symbol table info available.
> #14 0x00007f1b48aef28d in clone () from /lib64/libc.so.6
> No symbol table info available.
>
> Thread 34 (Thread 0x7f1b07fef700 (LWP 105194)):
> #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 () from
> /lib64/libpthread.so.0
> No symbol table info available.
> #1 0x00007f1b4941b050 in PR_WaitCondVar () from /lib64/libnspr4.so
> No symbol table info available.
> #2 0x00007f1b4b70efee in connection_wait_for_new_work ()
> No symbol table info available.
> #3 0x00007f1b4b710719 in connection_threadmain ()
> No symbol table info available.
> #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so
> No symbol table info available.
> #5 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0
> No symbol table info available.
> #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6
> No symbol table info available.
>
> Thread 33 (Thread 0x7f1b087f0700 (LWP 105193)):
> #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 () from
> /lib64/libpthread.so.0
> No symbol table info available.
> #1 0x00007f1b4941b050 in PR_WaitCondVar () from /lib64/libnspr4.so
> No symbol table info available.
> #2 0x00007f1b4b70efee in connection_wait_for_new_work ()
> No symbol table info available.
> #3 0x00007f1b4b710719 in connection_threadmain ()
> No symbol table info available.
> #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so
> No symbol table info available.
> #5 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0
> No symbol table info available.
> #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6
> No symbol table info available.
>
> Thread 32 (Thread 0x7f1b08ff1700 (LWP 105192)):
> #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 () from
> /lib64/libpthread.so.0
> No symbol table info available.
> #1 0x00007f1b4941b050 in PR_WaitCondVar () from /lib64/libnspr4.so
> No symbol table info available.
> #2 0x00007f1b4b70efee in connection_wait_for_new_work ()
> No symbol table info available.
> #3 0x00007f1b4b710719 in connection_threadmain ()
> No symbol table info available.
> #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so
> No symbol table info available.
> #5 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0
> No symbol table info available.
> #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6
> No symbol table info available.
>
> Thread 31 (Thread 0x7f1b097f2700 (LWP 105191)):
> #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 () from
> /lib64/libpthread.so.0
> No symbol table info available.
> #1 0x00007f1b4941b050 in PR_WaitCondVar () from /lib64/libnspr4.so
> No symbol table info available.
> #2 0x00007f1b4b70efee in connection_wait_for_new_work ()
> No symbol table info available.
> #3 0x00007f1b4b710719 in connection_threadmain ()
> No symbol table info available.
> #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so
> No symbol table info available.
> #5 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0
> No symbol table info available.
> #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6
> No symbol table info available.
>
> Thread 30 (Thread 0x7f1b09ff3700 (LWP 105190)):
> #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 () from
> /lib64/libpthread.so.0
> No symbol table info available.
> #1 0x00007f1b433212f3 in __db_hybrid_mutex_suspend () from
> /lib64/libdb-5.3.so <http://libdb-5.3.so>
> No symbol table info available.
> #2 0x00007f1b43320640 in __db_tas_mutex_lock () from
> /lib64/libdb-5.3.so <http://libdb-5.3.so>
> No symbol table info available.
> #3 0x00007f1b433cad3a in __lock_get_internal () from
> /lib64/libdb-5.3.so <http://libdb-5.3.so>
> No symbol table info available.
> #4 0x00007f1b433cb820 in __lock_get () from /lib64/libdb-5.3.so
> <http://libdb-5.3.so>
> No symbol table info available.
> #5 0x00007f1b433f7142 in __db_lget () from /lib64/libdb-5.3.so
> <http://libdb-5.3.so>
> No symbol table info available.
> #6 0x00007f1b4333e605 in __bam_search () from /lib64/libdb-5.3.so
> <http://libdb-5.3.so>
> No symbol table info available.
> #7 0x00007f1b43329256 in __bamc_search () from
> /lib64/libdb-5.3.so <http://libdb-5.3.so>
> No symbol table info available.
> #8 0x00007f1b4332ad0f in __bamc_get () from /lib64/libdb-5.3.so
> <http://libdb-5.3.so>
> No symbol table info available.
> #9 0x00007f1b433e3ca6 in __dbc_iget () from /lib64/libdb-5.3.so
> <http://libdb-5.3.so>
> No symbol table info available.
> #10 0x00007f1b433f2b02 in __dbc_get_pp () from /lib64/libdb-5.3.so
> <http://libdb-5.3.so>
> No symbol table info available.
> #11 0x00007f1b3e8ce680 in idl_new_fetch () from
> /usr/lib64/dirsrv/plugins/libback-ldbm.so
> No symbol table info available.
> #12 0x00007f1b3e8dcb26 in index_read_ext_allids () from
> /usr/lib64/dirsrv/plugins/libback-ldbm.so
> No symbol table info available.
> #13 0x00007f1b3e8c7284 in keys2idl () from
> /usr/lib64/dirsrv/plugins/libback-ldbm.so
> No symbol table info available.
> #14 0x00007f1b3e8c79e3 in ava_candidates.isra.0 () from
> /usr/lib64/dirsrv/plugins/libback-ldbm.so
> No symbol table info available.
> #15 0x00007f1b3e8c7fd2 in filter_candidates_ext () from
> /usr/lib64/dirsrv/plugins/libback-ldbm.so
> No symbol table info available.
> #16 0x00007f1b3e8c9096 in list_candidates () from
> /usr/lib64/dirsrv/plugins/libback-ldbm.so
> No symbol table info available.
> #17 0x00007f1b3e8c7f40 in filter_candidates_ext () from
> /usr/lib64/dirsrv/plugins/libback-ldbm.so
> No symbol table info available.
> #18 0x00007f1b3e8c9096 in list_candidates () from
> /usr/lib64/dirsrv/plugins/libback-ldbm.so
> No symbol table info available.
> #19 0x00007f1b3e8c7f40 in filter_candidates_ext () from
> /usr/lib64/dirsrv/plugins/libback-ldbm.so
> No symbol table info available.
> #20 0x00007f1b3e8c9096 in list_candidates () from
> /usr/lib64/dirsrv/plugins/libback-ldbm.so
> No symbol table info available.
> #21 0x00007f1b3e8c7f40 in filter_candidates_ext () from
> /usr/lib64/dirsrv/plugins/libback-ldbm.so
> No symbol table info available.
> #22 0x00007f1b3e90380c in subtree_candidates () from
> /usr/lib64/dirsrv/plugins/libback-ldbm.so
> No symbol table info available.
> #23 0x00007f1b3e904e7f in ldbm_back_search () from
> /usr/lib64/dirsrv/plugins/libback-ldbm.so
> No symbol table info available.
> #24 0x00007f1b4b2321d6 in op_shared_search () from
> /usr/lib64/dirsrv/libslapd.so.0
> No symbol table info available.
> #25 0x00007f1b4b72333e in do_search ()
> No symbol table info available.
> #26 0x00007f1b4b7113c3 in connection_threadmain ()
> No symbol table info available.
> #27 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so
> No symbol table info available.
> #28 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0
> No symbol table info available.
> #29 0x00007f1b48aef28d in clone () from /lib64/libc.so.6
> No symbol table info available.
>
> Thread 29 (Thread 0x7f1b0a7f4700 (LWP 105189)):
> #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 () from
> /lib64/libpthread.so.0
> No symbol table info available.
> #1 0x00007f1b4941b050 in PR_WaitCondVar () from /lib64/libnspr4.so
> No symbol table info available.
> #2 0x00007f1b4b70efee in connection_wait_for_new_work ()
> No symbol table info available.
> #3 0x00007f1b4b710719 in connection_threadmain ()
> No symbol table info available.
> #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so
> No symbol table info available.
> #5 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0
> No symbol table info available.
> #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6
> No symbol table info available.
>
> Thread 28 (Thread 0x7f1b0aff5700 (LWP 105188)):
> #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 () from
> /lib64/libpthread.so.0
> No symbol table info available.
> #1 0x00007f1b4941b050 in PR_WaitCondVar () from /lib64/libnspr4.so
> No symbol table info available.
> #2 0x00007f1b4b70efee in connection_wait_for_new_work ()
> No symbol table info available.
> #3 0x00007f1b4b710719 in connection_threadmain ()
> No symbol table info available.
> #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so
> No symbol table info available.
> #5 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0
> No symbol table info available.
> #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6
> No symbol table info available.
>
> Thread 27 (Thread 0x7f1b0b7f6700 (LWP 105187)):
> #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 () from
> /lib64/libpthread.so.0
> No symbol table info available.
> #1 0x00007f1b4941b050 in PR_WaitCondVar () from /lib64/libnspr4.so
> No symbol table info available.
> #2 0x00007f1b4b70efee in connection_wait_for_new_work ()
> No symbol table info available.
> #3 0x00007f1b4b710719 in connection_threadmain ()
> No symbol table info available.
> #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so
> No symbol table info available.
> #5 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0
> No symbol table info available.
> #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6
> No symbol table info available.
>
> Thread 26 (Thread 0x7f1b0bff7700 (LWP 105186)):
> #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 () from
> /lib64/libpthread.so.0
> No symbol table info available.
> #1 0x00007f1b433212f3 in __db_hybrid_mutex_suspend () from
> /lib64/libdb-5.3.so <http://libdb-5.3.so>
> No symbol table info available.
> #2 0x00007f1b43320640 in __db_tas_mutex_lock () from
> /lib64/libdb-5.3.so <http://libdb-5.3.so>
> No symbol table info available.
> #3 0x00007f1b433cad3a in __lock_get_internal () from
> /lib64/libdb-5.3.so <http://libdb-5.3.so>
> No symbol table info available.
> #4 0x00007f1b433cb820 in __lock_get () from /lib64/libdb-5.3.so
> <http://libdb-5.3.so>
> No symbol table info available.
> #5 0x00007f1b433f7142 in __db_lget () from /lib64/libdb-5.3.so
> <http://libdb-5.3.so>
> No symbol table info available.
> #6 0x00007f1b4333e605 in __bam_search () from /lib64/libdb-5.3.so
> <http://libdb-5.3.so>
> No symbol table info available.
> #7 0x00007f1b43329256 in __bamc_search () from
> /lib64/libdb-5.3.so <http://libdb-5.3.so>
> No symbol table info available.
> #8 0x00007f1b4332ad0f in __bamc_get () from /lib64/libdb-5.3.so
> <http://libdb-5.3.so>
> No symbol table info available.
> #9 0x00007f1b433e3ca6 in __dbc_iget () from /lib64/libdb-5.3.so
> <http://libdb-5.3.so>
> No symbol table info available.
> #10 0x00007f1b433f2b02 in __dbc_get_pp () from /lib64/libdb-5.3.so
> <http://libdb-5.3.so>
> No symbol table info available.
> #11 0x00007f1b3e8ce680 in idl_new_fetch () from
> /usr/lib64/dirsrv/plugins/libback-ldbm.so
> No symbol table info available.
> #12 0x00007f1b3e8dcb26 in index_read_ext_allids () from
> /usr/lib64/dirsrv/plugins/libback-ldbm.so
> No symbol table info available.
> #13 0x00007f1b3e8c7284 in keys2idl () from
> /usr/lib64/dirsrv/plugins/libback-ldbm.so
> No symbol table info available.
> #14 0x00007f1b3e8c79e3 in ava_candidates.isra.0 () from
> /usr/lib64/dirsrv/plugins/libback-ldbm.so
> No symbol table info available.
> #15 0x00007f1b3e8c7fd2 in filter_candidates_ext () from
> /usr/lib64/dirsrv/plugins/libback-ldbm.so
> No symbol table info available.
> #16 0x00007f1b3e8c9096 in list_candidates () from
> /usr/lib64/dirsrv/plugins/libback-ldbm.so
> No symbol table info available.
> #17 0x00007f1b3e8c7f40 in filter_candidates_ext () from
> /usr/lib64/dirsrv/plugins/libback-ldbm.so
> No symbol table info available.
> #18 0x00007f1b3e90380c in subtree_candidates () from
> /usr/lib64/dirsrv/plugins/libback-ldbm.so
> No symbol table info available.
> #19 0x00007f1b3e904e7f in ldbm_back_search () from
> /usr/lib64/dirsrv/plugins/libback-ldbm.so
> No symbol table info available.
> #20 0x00007f1b4b2321d6 in op_shared_search () from
> /usr/lib64/dirsrv/libslapd.so.0
> No symbol table info available.
> #21 0x00007f1b4b24294e in search_internal_callback_pb () from
> /usr/lib64/dirsrv/libslapd.so.0
> No symbol table info available.
> #22 0x00007f1b4b242be8 in search_internal_pb () from
> /usr/lib64/dirsrv/libslapd.so.0
> No symbol table info available.
> #23 0x00007f1b4b72149b in ids_sasl_canon_user ()
> No symbol table info available.
> #24 0x00007f1b491e193b in _sasl_canon_user () from
> /lib64/libsasl2.so.3
> No symbol table info available.
> #25 0x00007f1b491e1c4c in _sasl_canon_user_lookup () from
> /lib64/libsasl2.so.3
> No symbol table info available.
> #26 0x00007f1b430e9fb9 in gssapi_server_mech_step () from
> /usr/lib64/sasl2/libgssapiv2.so
> No symbol table info available.
> #27 0x00007f1b491edb9b in sasl_server_step () from
> /lib64/libsasl2.so.3
> No symbol table info available.
> #28 0x00007f1b4b72283b in ids_sasl_check_bind ()
> No symbol table info available.
> #29 0x00007f1b4b70a61a in do_bind ()
> No symbol table info available.
> #30 0x00007f1b4b7113fd in connection_threadmain ()
> No symbol table info available.
> #31 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so
> No symbol table info available.
> #32 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0
> No symbol table info available.
> #33 0x00007f1b48aef28d in clone () from /lib64/libc.so.6
> No symbol table info available.
>
> Thread 25 (Thread 0x7f1b0c7f8700 (LWP 105185)):
> #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 () from
> /lib64/libpthread.so.0
> No symbol table info available.
> #1 0x00007f1b4941b050 in PR_WaitCondVar () from /lib64/libnspr4.so
> No symbol table info available.
> #2 0x00007f1b4b70efee in connection_wait_for_new_work ()
> No symbol table info available.
> #3 0x00007f1b4b710719 in connection_threadmain ()
> No symbol table info available.
> #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so
> No symbol table info available.
> #5 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0
> No symbol table info available.
> #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6
> No symbol table info available.
>
> Thread 24 (Thread 0x7f1b0cff9700 (LWP 105184)):
> #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 () from
> /lib64/libpthread.so.0
> No symbol table info available.
> #1 0x00007f1b4941b050 in PR_WaitCondVar () from /lib64/libnspr4.so
> No symbol table info available.
> #2 0x00007f1b4b70efee in connection_wait_for_new_work ()
> No symbol table info available.
> #3 0x00007f1b4b710719 in connection_threadmain ()
> No symbol table info available.
> #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so
> No symbol table info available.
> #5 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0
> No symbol table info available.
> #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6
> No symbol table info available.
>
> Thread 23 (Thread 0x7f1b0d7fa700 (LWP 105183)):
> #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 () from
> /lib64/libpthread.so.0
> No symbol table info available.
> #1 0x00007f1b4941b050 in PR_WaitCondVar () from /lib64/libnspr4.so
> No symbol table info available.
> #2 0x00007f1b4b70efee in connection_wait_for_new_work ()
> No symbol table info available.
> #3 0x00007f1b4b710719 in connection_threadmain ()
> No symbol table info available.
> #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so
> No symbol table info available.
> #5 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0
> No symbol table info available.
> #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6
> No symbol table info available.
>
> Thread 22 (Thread 0x7f1b0dffb700 (LWP 105182)):
> #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 () from
> /lib64/libpthread.so.0
> No symbol table info available.
> #1 0x00007f1b4941b050 in PR_WaitCondVar () from /lib64/libnspr4.so
> No symbol table info available.
> #2 0x00007f1b4b70efee in connection_wait_for_new_work ()
> No symbol table info available.
> #3 0x00007f1b4b710719 in connection_threadmain ()
> No symbol table info available.
> #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so
> No symbol table info available.
> #5 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0
> No symbol table info available.
> #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6
> No symbol table info available.
>
> Thread 21 (Thread 0x7f1b0e7fc700 (LWP 105181)):
> #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 () from
> /lib64/libpthread.so.0
> No symbol table info available.
> #1 0x00007f1b4941b050 in PR_WaitCondVar () from /lib64/libnspr4.so
> No symbol table info available.
> #2 0x00007f1b4b70efee in connection_wait_for_new_work ()
> No symbol table info available.
> #3 0x00007f1b4b710719 in connection_threadmain ()
> No symbol table info available.
> #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so
> No symbol table info available.
> #5 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0
> No symbol table info available.
> #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6
> No symbol table info available.
>
> Thread 20 (Thread 0x7f1b0effd700 (LWP 105180)):
> #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 () from
> /lib64/libpthread.so.0
> No symbol table info available.
> #1 0x00007f1b433212f3 in __db_hybrid_mutex_suspend () from
> /lib64/libdb-5.3.so <http://libdb-5.3.so>
> No symbol table info available.
> #2 0x00007f1b43320b37 in __db_tas_mutex_readlock () from
> /lib64/libdb-5.3.so <http://libdb-5.3.so>
> No symbol table info available.
> #3 0x00007f1b4343ad35 in __memp_fget () from /lib64/libdb-5.3.so
> <http://libdb-5.3.so>
> No symbol table info available.
> #4 0x00007f1b433e4333 in __dbc_iget () from /lib64/libdb-5.3.so
> <http://libdb-5.3.so>
> No symbol table info available.
> #5 0x00007f1b433f2b02 in __dbc_get_pp () from /lib64/libdb-5.3.so
> <http://libdb-5.3.so>
> No symbol table info available.
> #6 0x00007f1b3e8ce822 in idl_new_fetch () from
> /usr/lib64/dirsrv/plugins/libback-ldbm.so
> No symbol table info available.
> #7 0x00007f1b3e8dcb26 in index_read_ext_allids () from
> /usr/lib64/dirsrv/plugins/libback-ldbm.so
> No symbol table info available.
> #8 0x00007f1b3e8c7284 in keys2idl () from
> /usr/lib64/dirsrv/plugins/libback-ldbm.so
> No symbol table info available.
> #9 0x00007f1b3e8c79e3 in ava_candidates.isra.0 () from
> /usr/lib64/dirsrv/plugins/libback-ldbm.so
> No symbol table info available.
> #10 0x00007f1b3e8c7fd2 in filter_candidates_ext () from
> /usr/lib64/dirsrv/plugins/libback-ldbm.so
> No symbol table info available.
> #11 0x00007f1b3e8c9096 in list_candidates () from
> /usr/lib64/dirsrv/plugins/libback-ldbm.so
> No symbol table info available.
> #12 0x00007f1b3e8c7f40 in filter_candidates_ext () from
> /usr/lib64/dirsrv/plugins/libback-ldbm.so
> No symbol table info available.
> #13 0x00007f1b3e8c9096 in list_candidates () from
> /usr/lib64/dirsrv/plugins/libback-ldbm.so
> No symbol table info available.
> #14 0x00007f1b3e8c7f40 in filter_candidates_ext () from
> /usr/lib64/dirsrv/plugins/libback-ldbm.so
> No symbol table info available.
> #15 0x00007f1b3e8c9096 in list_candidates () from
> /usr/lib64/dirsrv/plugins/libback-ldbm.so
> No symbol table info available.
> #16 0x00007f1b3e8c7f40 in filter_candidates_ext () from
> /usr/lib64/dirsrv/plugins/libback-ldbm.so
> No symbol table info available.
> #17 0x00007f1b3e90380c in subtree_candidates () from
> /usr/lib64/dirsrv/plugins/libback-ldbm.so
> No symbol table info available.
> #18 0x00007f1b3e904e7f in ldbm_back_search () from
> /usr/lib64/dirsrv/plugins/libback-ldbm.so
> No symbol table info available.
> #19 0x00007f1b4b2321d6 in op_shared_search () from
> /usr/lib64/dirsrv/libslapd.so.0
> No symbol table info available.
> #20 0x00007f1b4b72333e in do_search ()
> No symbol table info available.
> #21 0x00007f1b4b7113c3 in connection_threadmain ()
> No symbol table info available.
> #22 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so
> No symbol table info available.
> #23 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0
> No symbol table info available.
> #24 0x00007f1b48aef28d in clone () from /lib64/libc.so.6
> No symbol table info available.
>
> Thread 19 (Thread 0x7f1b0f7fe700 (LWP 105179)):
> #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 () from
> /lib64/libpthread.so.0
> No symbol table info available.
> #1 0x00007f1b433212f3 in __db_hybrid_mutex_suspend () from
> /lib64/libdb-5.3.so <http://libdb-5.3.so>
> No symbol table info available.
> #2 0x00007f1b43320640 in __db_tas_mutex_lock () from
> /lib64/libdb-5.3.so <http://libdb-5.3.so>
> No symbol table info available.
> #3 0x00007f1b433cad3a in __lock_get_internal () from
> /lib64/libdb-5.3.so <http://libdb-5.3.so>
> No symbol table info available.
> #4 0x00007f1b433cb820 in __lock_get () from /lib64/libdb-5.3.so
> <http://libdb-5.3.so>
> No symbol table info available.
> #5 0x00007f1b433f7142 in __db_lget () from /lib64/libdb-5.3.so
> <http://libdb-5.3.so>
> No symbol table info available.
> #6 0x00007f1b4333e605 in __bam_search () from /lib64/libdb-5.3.so
> <http://libdb-5.3.so>
> No symbol table info available.
> #7 0x00007f1b43329256 in __bamc_search () from
> /lib64/libdb-5.3.so <http://libdb-5.3.so>
> No symbol table info available.
> #8 0x00007f1b4332ad0f in __bamc_get () from /lib64/libdb-5.3.so
> <http://libdb-5.3.so>
> No symbol table info available.
> #9 0x00007f1b433e3ca6 in __dbc_iget () from /lib64/libdb-5.3.so
> <http://libdb-5.3.so>
> No symbol table info available.
> #10 0x00007f1b433f2b02 in __dbc_get_pp () from /lib64/libdb-5.3.so
> <http://libdb-5.3.so>
> No symbol table info available.
> #11 0x00007f1b3e8ce680 in idl_new_fetch () from
> /usr/lib64/dirsrv/plugins/libback-ldbm.so
> No symbol table info available.
> #12 0x00007f1b3e8dcb26 in index_read_ext_allids () from
> /usr/lib64/dirsrv/plugins/libback-ldbm.so
> No symbol table info available.
> #13 0x00007f1b3e8c7284 in keys2idl () from
> /usr/lib64/dirsrv/plugins/libback-ldbm.so
> No symbol table info available.
> #14 0x00007f1b3e8c79e3 in ava_candidates.isra.0 () from
> /usr/lib64/dirsrv/plugins/libback-ldbm.so
> No symbol table info available.
> #15 0x00007f1b3e8c7fd2 in filter_candidates_ext () from
> /usr/lib64/dirsrv/plugins/libback-ldbm.so
> No symbol table info available.
> #16 0x00007f1b3e8c9096 in list_candidates () from
> /usr/lib64/dirsrv/plugins/libback-ldbm.so
> No symbol table info available.
> #17 0x00007f1b3e8c7f40 in filter_candidates_ext () from
> /usr/lib64/dirsrv/plugins/libback-ldbm.so
> No symbol table info available.
> #18 0x00007f1b3e8c9096 in list_candidates () from
> /usr/lib64/dirsrv/plugins/libback-ldbm.so
> No symbol table info available.
> #19 0x00007f1b3e8c7f40 in filter_candidates_ext () from
> /usr/lib64/dirsrv/plugins/libback-ldbm.so
> No symbol table info available.
> #20 0x00007f1b3e8c9096 in list_candidates () from
> /usr/lib64/dirsrv/plugins/libback-ldbm.so
> No symbol table info available.
> #21 0x00007f1b3e8c7f40 in filter_candidates_ext () from
> /usr/lib64/dirsrv/plugins/libback-ldbm.so
> No symbol table info available.
> #22 0x00007f1b3e90380c in subtree_candidates () from
> /usr/lib64/dirsrv/plugins/libback-ldbm.so
> No symbol table info available.
> #23 0x00007f1b3e904e7f in ldbm_back_search () from
> /usr/lib64/dirsrv/plugins/libback-ldbm.so
> No symbol table info available.
> #24 0x00007f1b4b2321d6 in op_shared_search () from
> /usr/lib64/dirsrv/libslapd.so.0
> No symbol table info available.
> #25 0x00007f1b4b72333e in do_search ()
> No symbol table info available.
> #26 0x00007f1b4b7113c3 in connection_threadmain ()
> No symbol table info available.
> #27 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so
> No symbol table info available.
> #28 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0
> No symbol table info available.
> #29 0x00007f1b48aef28d in clone () from /lib64/libc.so.6
> No symbol table info available.
>
> Thread 18 (Thread 0x7f1b0ffff700 (LWP 105178)):
> #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 () from
> /lib64/libpthread.so.0
> No symbol table info available.
> #1 0x00007f1b4941b050 in PR_WaitCondVar () from /lib64/libnspr4.so
> No symbol table info available.
> #2 0x00007f1b4b70efee in connection_wait_for_new_work ()
> No symbol table info available.
> #3 0x00007f1b4b710719 in connection_threadmain ()
> No symbol table info available.
> #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so
> No symbol table info available.
> #5 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0
> No symbol table info available.
> #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6
> No symbol table info available.
>
> Thread 17 (Thread 0x7f1b14d28700 (LWP 105177)):
> #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 () from
> /lib64/libpthread.so.0
> No symbol table info available.
> #1 0x00007f1b4941b050 in PR_WaitCondVar () from /lib64/libnspr4.so
> No symbol table info available.
> #2 0x00007f1b4b70efee in connection_wait_for_new_work ()
> No symbol table info available.
> #3 0x00007f1b4b710719 in connection_threadmain ()
> No symbol table info available.
> #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so
> No symbol table info available.
> #5 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0
> No symbol table info available.
> #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6
> No symbol table info available.
>
> Thread 16 (Thread 0x7f1b15529700 (LWP 105176)):
> #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 () from
> /lib64/libpthread.so.0
> No symbol table info available.
> #1 0x00007f1b4941b050 in PR_WaitCondVar () from /lib64/libnspr4.so
> No symbol table info available.
> #2 0x00007f1b4b70efee in connection_wait_for_new_work ()
> No symbol table info available.
> #3 0x00007f1b4b710719 in connection_threadmain ()
> No symbol table info available.
> #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so
> No symbol table info available.
> #5 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0
> No symbol table info available.
> #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6
> No symbol table info available.
>
> Thread 15 (Thread 0x7f1b15d2a700 (LWP 105175)):
> #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 () from
> /lib64/libpthread.so.0
> No symbol table info available.
> #1 0x00007f1b4941b050 in PR_WaitCondVar () from /lib64/libnspr4.so
> No symbol table info available.
> #2 0x00007f1b4b70efee in connection_wait_for_new_work ()
> No symbol table info available.
> #3 0x00007f1b4b710719 in connection_threadmain ()
> No symbol table info available.
> #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so
> No symbol table info available.
> #5 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0
> No symbol table info available.
> #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6
> No symbol table info available.
>
> Thread 14 (Thread 0x7f1b1652b700 (LWP 105174)):
> #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 () from
> /lib64/libpthread.so.0
> No symbol table info available.
> #1 0x00007f1b4941b050 in PR_WaitCondVar () from /lib64/libnspr4.so
> No symbol table info available.
> #2 0x00007f1b4b70efee in connection_wait_for_new_work ()
> No symbol table info available.
> #3 0x00007f1b4b710719 in connection_threadmain ()
> No symbol table info available.
> #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so
> No symbol table info available.
> #5 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0
> No symbol table info available.
> #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6
> No symbol table info available.
>
> Thread 13 (Thread 0x7f1b16d2c700 (LWP 105173)):
> #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 () from
> /lib64/libpthread.so.0
> No symbol table info available.
> #1 0x00007f1b433212f3 in __db_hybrid_mutex_suspend () from
> /lib64/libdb-5.3.so <http://libdb-5.3.so>
> No symbol table info available.
> #2 0x00007f1b43320640 in __db_tas_mutex_lock () from
> /lib64/libdb-5.3.so <http://libdb-5.3.so>
> No symbol table info available.
> #3 0x00007f1b433cad3a in __lock_get_internal () from
> /lib64/libdb-5.3.so <http://libdb-5.3.so>
> No symbol table info available.
> #4 0x00007f1b433cb820 in __lock_get () from /lib64/libdb-5.3.so
> <http://libdb-5.3.so>
> No symbol table info available.
> #5 0x00007f1b433f7142 in __db_lget () from /lib64/libdb-5.3.so
> <http://libdb-5.3.so>
> No symbol table info available.
> #6 0x00007f1b4333e605 in __bam_search () from /lib64/libdb-5.3.so
> <http://libdb-5.3.so>
> No symbol table info available.
> #7 0x00007f1b43329256 in __bamc_search () from
> /lib64/libdb-5.3.so <http://libdb-5.3.so>
> No symbol table info available.
> #8 0x00007f1b4332ad0f in __bamc_get () from /lib64/libdb-5.3.so
> <http://libdb-5.3.so>
> No symbol table info available.
> #9 0x00007f1b433e3ca6 in __dbc_iget () from /lib64/libdb-5.3.so
> <http://libdb-5.3.so>
> No symbol table info available.
> #10 0x00007f1b433f2b02 in __dbc_get_pp () from /lib64/libdb-5.3.so
> <http://libdb-5.3.so>
> No symbol table info available.
> #11 0x00007f1b3e8ce680 in idl_new_fetch () from
> /usr/lib64/dirsrv/plugins/libback-ldbm.so
> No symbol table info available.
> #12 0x00007f1b3e8dcb26 in index_read_ext_allids () from
> /usr/lib64/dirsrv/plugins/libback-ldbm.so
> No symbol table info available.
> #13 0x00007f1b3e8c7284 in keys2idl () from
> /usr/lib64/dirsrv/plugins/libback-ldbm.so
> No symbol table info available.
> #14 0x00007f1b3e8c79e3 in ava_candidates.isra.0 () from
> /usr/lib64/dirsrv/plugins/libback-ldbm.so
> No symbol table info available.
> #15 0x00007f1b3e8c7fd2 in filter_candidates_ext () from
> /usr/lib64/dirsrv/plugins/libback-ldbm.so
> No symbol table info available.
> #16 0x00007f1b3e8c9096 in list_candidates () from
> /usr/lib64/dirsrv/plugins/libback-ldbm.so
> No symbol table info available.
> #17 0x00007f1b3e8c7f40 in filter_candidates_ext () from
> /usr/lib64/dirsrv/plugins/libback-ldbm.so
> No symbol table info available.
> #18 0x00007f1b3e8c9096 in list_candidates () from
> /usr/lib64/dirsrv/plugins/libback-ldbm.so
> No symbol table info available.
> #19 0x00007f1b3e8c7f40 in filter_candidates_ext () from
> /usr/lib64/dirsrv/plugins/libback-ldbm.so
> No symbol table info available.
> #20 0x00007f1b3e8c9096 in list_candidates () from
> /usr/lib64/dirsrv/plugins/libback-ldbm.so
> No symbol table info available.
> #21 0x00007f1b3e8c7f40 in filter_candidates_ext () from
> /usr/lib64/dirsrv/plugins/libback-ldbm.so
> No symbol table info available.
> #22 0x00007f1b3e90380c in subtree_candidates () from
> /usr/lib64/dirsrv/plugins/libback-ldbm.so
> No symbol table info available.
> #23 0x00007f1b3e904e7f in ldbm_back_search () from
> /usr/lib64/dirsrv/plugins/libback-ldbm.so
> No symbol table info available.
> #24 0x00007f1b4b2321d6 in op_shared_search () from
> /usr/lib64/dirsrv/libslapd.so.0
> No symbol table info available.
> #25 0x00007f1b4b72333e in do_search ()
> No symbol table info available.
> #26 0x00007f1b4b7113c3 in connection_threadmain ()
> No symbol table info available.
> #27 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so
> No symbol table info available.
> #28 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0
> No symbol table info available.
> #29 0x00007f1b48aef28d in clone () from /lib64/libc.so.6
> No symbol table info available.
>
> Thread 12 (Thread 0x7f1b1752d700 (LWP 105172)):
> #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 () from
> /lib64/libpthread.so.0
> No symbol table info available.
> #1 0x00007f1b4941b050 in PR_WaitCondVar () from /lib64/libnspr4.so
> No symbol table info available.
> #2 0x00007f1b4b70efee in connection_wait_for_new_work ()
> No symbol table info available.
> #3 0x00007f1b4b710719 in connection_threadmain ()
> No symbol table info available.
> #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so
> No symbol table info available.
> #5 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0
> No symbol table info available.
> #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6
> No symbol table info available.
>
> Thread 11 (Thread 0x7f1b17d2e700 (LWP 105171)):
> #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 () from
> /lib64/libpthread.so.0
> No symbol table info available.
> #1 0x00007f1b4941b050 in PR_WaitCondVar () from /lib64/libnspr4.so
> No symbol table info available.
> #2 0x00007f1b4b70efee in connection_wait_for_new_work ()
> No symbol table info available.
> #3 0x00007f1b4b710719 in connection_threadmain ()
> No symbol table info available.
> #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so
> No symbol table info available.
> #5 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0
> No symbol table info available.
> #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6
> No symbol table info available.
>
> Thread 10 (Thread 0x7f1b2509d700 (LWP 105168)):
> #0 0x00007f1b48dc5a82 in pthread_cond_timedwait@@GLIBC_2.3.2 ()
> from /lib64/libpthread.so.0
> No symbol table info available.
> #1 0x00007f1b4941ab07 in pt_TimedWait () from /lib64/libnspr4.so
> No symbol table info available.
> #2 0x00007f1b4941afce in PR_WaitCondVar () from /lib64/libnspr4.so
> No symbol table info available.
> #3 0x00007f1b4b2061e8 in eq_loop () from
> /usr/lib64/dirsrv/libslapd.so.0
> No symbol table info available.
> #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so
> No symbol table info available.
> #5 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0
> No symbol table info available.
> #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6
> No symbol table info available.
>
> Thread 9 (Thread 0x7f1b2589e700 (LWP 105167)):
> #0 0x00007f1b48dc5a82 in pthread_cond_timedwait@@GLIBC_2.3.2 ()
> from /lib64/libpthread.so.0
> No symbol table info available.
> #1 0x00007f1b4941ab07 in pt_TimedWait () from /lib64/libnspr4.so
> No symbol table info available.
> #2 0x00007f1b4941afce in PR_WaitCondVar () from /lib64/libnspr4.so
> No symbol table info available.
> #3 0x00007f1b4b719d03 in housecleaning ()
> No symbol table info available.
> #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so
> No symbol table info available.
> #5 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0
> No symbol table info available.
> #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6
> No symbol table info available.
>
> Thread 8 (Thread 0x7f1b2609f700 (LWP 105166)):
> #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 () from
> /lib64/libpthread.so.0
> No symbol table info available.
> #1 0x00007f1b4941b050 in PR_WaitCondVar () from /lib64/libnspr4.so
> No symbol table info available.
> #2 0x00007f1b4b260198 in slapi_wait_condvar () from
> /usr/lib64/dirsrv/libslapd.so.0
> No symbol table info available.
> #3 0x00007f1b3cf5deed in roles_cache_wait_on_change () from
> /usr/lib64/dirsrv/plugins/libroles-plugin.so
> No symbol table info available.
> #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so
> No symbol table info available.
> #5 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0
> No symbol table info available.
> #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6
> No symbol table info available.
>
> Thread 7 (Thread 0x7f1b268a0700 (LWP 105165)):
> #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 () from
> /lib64/libpthread.so.0
> No symbol table info available.
> #1 0x00007f1b4941b050 in PR_WaitCondVar () from /lib64/libnspr4.so
> No symbol table info available.
> #2 0x00007f1b4b260198 in slapi_wait_condvar () from
> /usr/lib64/dirsrv/libslapd.so.0
> No symbol table info available.
> #3 0x00007f1b3cf5deed in roles_cache_wait_on_change () from
> /usr/lib64/dirsrv/plugins/libroles-plugin.so
> No symbol table info available.
> #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so
> No symbol table info available.
> #5 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0
> No symbol table info available.
> #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6
> No symbol table info available.
>
> Thread 6 (Thread 0x7f1b270a1700 (LWP 105164)):
> #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 () from
> /lib64/libpthread.so.0
> No symbol table info available.
> #1 0x00007f1b4941b050 in PR_WaitCondVar () from /lib64/libnspr4.so
> No symbol table info available.
> #2 0x00007f1b4b260198 in slapi_wait_condvar () from
> /usr/lib64/dirsrv/libslapd.so.0
> No symbol table info available.
> #3 0x00007f1b3cf5deed in roles_cache_wait_on_change () from
> /usr/lib64/dirsrv/plugins/libroles-plugin.so
> No symbol table info available.
> #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so
> No symbol table info available.
> #5 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0
> No symbol table info available.
> #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6
> No symbol table info available.
>
> Thread 5 (Thread 0x7f1b278a2700 (LWP 105163)):
> #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 () from
> /lib64/libpthread.so.0
> No symbol table info available.
> #1 0x00007f1b4941b050 in PR_WaitCondVar () from /lib64/libnspr4.so
> No symbol table info available.
> #2 0x00007f1b4b260198 in slapi_wait_condvar () from
> /usr/lib64/dirsrv/libslapd.so.0
> No symbol table info available.
> #3 0x00007f1b415f262e in cos_cache_wait_on_change () from
> /usr/lib64/dirsrv/plugins/libcos-plugin.so
> No symbol table info available.
> #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so
> No symbol table info available.
> #5 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0
> No symbol table info available.
> #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6
> No symbol table info available.
>
> Thread 4 (Thread 0x7f1b282ac700 (LWP 105162)):
> #0 0x00007f1b48ae69b3 in select () from /lib64/libc.so.6
> No symbol table info available.
> #1 0x00007f1b4b2710e9 in DS_Sleep () from
> /usr/lib64/dirsrv/libslapd.so.0
> No symbol table info available.
> #2 0x00007f1b3e8ba707 in perf_threadmain () from
> /usr/lib64/dirsrv/plugins/libback-ldbm.so
> No symbol table info available.
> #3 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so
> No symbol table info available.
> #4 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0
> No symbol table info available.
> #5 0x00007f1b48aef28d in clone () from /lib64/libc.so.6
> No symbol table info available.
>
> Thread 3 (Thread 0x7f1b28aad700 (LWP 105161)):
> #0 0x00007f1b48ae69b3 in select () from /lib64/libc.so.6
> No symbol table info available.
> #1 0x00007f1b4b2710e9 in DS_Sleep () from
> /usr/lib64/dirsrv/libslapd.so.0
> No symbol table info available.
> #2 0x00007f1b3e8bfb7f in trickle_threadmain () from
> /usr/lib64/dirsrv/plugins/libback-ldbm.so
> No symbol table info available.
> #3 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so
> No symbol table info available.
> #4 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0
> No symbol table info available.
> #5 0x00007f1b48aef28d in clone () from /lib64/libc.so.6
> No symbol table info available.
>
> Thread 2 (Thread 0x7f1b292ae700 (LWP 105160)):
> #0 0x00007f1b48ae69b3 in select () from /lib64/libc.so.6
> No symbol table info available.
> #1 0x00007f1b4344dbad in __os_yield () from /lib64/libdb-5.3.so
> <http://libdb-5.3.so>
> No symbol table info available.
> #2 0x00007f1b434492b3 in __memp_sync_int () from
> /lib64/libdb-5.3.so <http://libdb-5.3.so>
> No symbol table info available.
> #3 0x00007f1b43459752 in __txn_checkpoint () from
> /lib64/libdb-5.3.so <http://libdb-5.3.so>
> No symbol table info available.
> #4 0x00007f1b43459b74 in __txn_checkpoint_pp () from
> /lib64/libdb-5.3.so <http://libdb-5.3.so>
> No symbol table info available.
> #5 0x00007f1b3e8c3a87 in checkpoint_threadmain () from
> /usr/lib64/dirsrv/plugins/libback-ldbm.so
> No symbol table info available.
> #6 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so
> No symbol table info available.
> #7 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0
> No symbol table info available.
> #8 0x00007f1b48aef28d in clone () from /lib64/libc.so.6
> No symbol table info available.
>
> Thread 1 (Thread 0x7f1b29aaf700 (LWP 105159)):
> #0 0x00007f1b48ae69b3 in select () from /lib64/libc.so.6
> No symbol table info available.
> #1 0x00007f1b4b2710e9 in DS_Sleep () from
> /usr/lib64/dirsrv/libslapd.so.0
> No symbol table info available.
> #2 0x00007f1b3e8bf907 in deadlock_threadmain () from
> /usr/lib64/dirsrv/plugins/libback-ldbm.so
> No symbol table info available.
> #3 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so
> No symbol table info available.
> #4 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0
> No symbol table info available.
> #5 0x00007f1b48aef28d in clone () from /lib64/libc.so.6
> No symbol table info available.
>
>
>
> Thanks,
>
> Rakesh
>
>
> On Mon, Sep 5, 2016 at 4:48 PM, thierry bordaz <tbordaz at redhat.com
> <mailto:tbordaz at redhat.com>> wrote:
>
>
>
> On 09/05/2016 12:05 PM, Rakesh Rajasekharan wrote:
>> Hi Thierry,
>>
>>
>> I was getting the hang issue while running ipa-client-install
>> simultaneously on few clients..
>> However, today, I am not able to replicate that.
>>
>> I could not get a gdb . But i will try getting that the next
>> time I face this issue.
>>
>> The CPU does not stay high.. it just momentarily touches a
>> high value and then drops down to around 2-7%
>>
>> One question I have is , is it ok to set it
>> nsslapd-threadnumber to a very high value .
>> I have around 4000 clients and with nsslapd-maxthreadsperconn
>> set to 5..So, can I set nsslapd-threadnumber to around 25000.
>
> Hello,
>
> I know some users running in production with several hunderds
> of threads (>600) and this without problem.
>
> I do not recall having suggested to increase that number and
> for what reason.
> Usually 30 workers is a good enough value. It can create
> bootleneck if for some reason each operation is very long to
> satisfy and exhaust the number of workers. You can monitor the
> work queue:
>
> ldapsearch -D "cn=directory manager" -w xxx -LLL -b
> "cn=monitor" -s base opsinitiated opscompleted
>
>
> If opscompleted-opsinitiated remains close to threadnumber,
> then yes it would be valuable to increase it.
>
> The computation #client * #async_op_per_client sound an
> overkill. Even if all clients send at the exact same time all
> their requests, it is very likely that some common resource
> (db page, log, allocator...) will serialize them. If you
> monitor a need to increase the work, you would for example set
> it to 50, then monitor, then set it to 100, then monitor...
> until you find a good enough value.
> Note the increasing the #thread, increases the memory
> footprint that will reduce the efficiency of file system cache
> and can increase the response time.
>
>
> best regards
> thierry
>
>>
>> Thanks
>>
>> On Mon, Sep 5, 2016 at 1:03 PM, thierry bordaz
>> <tbordaz at redhat.com <mailto:tbordaz at redhat.com>> wrote:
>>
>>
>> Hi Rakesh,
>>
>> Were you able to get a pstack or full stack with gdb
>> (http://www.port389.org/docs/389ds/FAQ/faq.html#debugging-crashes)
>> when the server hangs ?
>>
>> If it happens with 500 threads as well as with 30, using
>> 30 threads is a better choice to debug this issue.
>> I will try to reproduce using 150 parallel 'ipa user-find
>> p-testipa' commands
>>
>> Something I am unsure is if the CPU consumption stays
>> high (you mentioned 340% CPU usage) as long as the hang
>> happens or if after a suddent shot up to 340% (that marks
>> the beginning of the hang) it drops and stay hanging ?
>>
>> thanks
>> thierry
>>
>> On 09/04/2016 08:40 PM, Rakesh Rajasekharan wrote:
>>> starce on the slapd process actually had this in the
>>> output..
>>> FUTEX_WAIT_PRIVATE
>>>
>>> and checking for the number of threads slapd had.. there
>>> were 5015 threads
>>>
>>> ps -efL|grep slapd|wc -l
>>> 5015
>>>
>>> strace on most of the threads gave this output
>>>
>>> strace -p 67411
>>> Process 67411 attached
>>> futex(0x7f3f0226b41c, FUTEX_WAIT_PRIVATE, 1, NULL) = -1
>>> EAGAIN (Resource temporarily unavailable)
>>> futex(0x7f3f0226b41c, FUTEX_WAIT_PRIVATE, 2,
>>> NULL^CProcess 67411 detached
>>>
>>>
>>>
>>>
>>>
>>> On Sun, Sep 4, 2016 at 5:34 PM, Rakesh Rajasekharan
>>> <rakesh.rajasekharan at gmail.com
>>> <mailto:rakesh.rajasekharan at gmail.com>> wrote:
>>>
>>> I have again got the issue of IPA hanging.. The
>>> issue came up when i tried to run ipa-client-isntall
>>> on 142 clients simultaneously
>>>
>>>
>>> None of the IPA commands are responding, and I see
>>> this error
>>>
>>> ipa user-find p-testipa
>>> ipa: ERROR: Insufficient access: SASL(-1): generic
>>> failure: GSSAPI Error: Unspecified GSS failure.
>>> Minor code may provide more information (KDC
>>> returned error string: PROCESS_TGS)
>>>
>>> KRB5_TRACE=/dev/stdout kinit admin
>>> [41178] 1472984115.233214: Getting initial
>>> credentials for admin at XYZ.COM <mailto:admin at XYZ.COM>
>>> [41178] 1472984115.235257: Sending request (167
>>> bytes) to XYZ.COM <http://XYZ.COM>
>>> [41178] 1472984115.235419: Initiating TCP connection
>>> to stream 10.1.3.36:88 <http://10.1.3.36:88>
>>> [41178] 1472984115.235685: Sending TCP request to
>>> stream 10.1.3.36:88 <http://10.1.3.36:88>
>>> [41178] 1472984120.238914: Received answer (174
>>> bytes) from stream 10.1.3.36:88 <http://10.1.3.36:88>
>>> [41178] 1472984120.238925: Terminating TCP
>>> connection to stream 10.1.3.36:88 <http://10.1.3.36:88>
>>> [41178] 1472984120.238993: Response was from master KDC
>>> [41
>>>
>>>
>>> Running an ldapsearch to see the db.. does not give
>>> any results and just hangs there
>>>
>>> ldapsearch -x -D 'cn=Directory Manager' -W -s one -b
>>> 'cn=kerberos,dc=xyz,dc=com'
>>> Enter LDAP Password:
>>>
>>> even an ldapsearch -x does not respond
>>> At this point, am sure that slapd is the one causing
>>> issues
>>>
>>> Running an strace against the hung slapd itself
>>> seems to get stuck does not proceed after saying
>>> "attaching to process"
>>>
>>> From some others posts I read Thierry suggesting to
>>> increase the nsslapd-threadnumber value
>>>
>>> It was set to 30, I think that might be too low.
>>>
>>> I have raised it to 500
>>>
>>> Now after restarting the service .. ldapsearch
>>> starts responding.
>>> But running the test to add a sudden high number of
>>> clients again left ns-slapd to hung state
>>>
>>> When i attempted adding the clients.. the ns-slapd
>>> cpu usage shot up to 340% and after that ns-slapd
>>> stopped responding
>>>
>>> So now, atleast I know what might be causing the
>>> issue and I can now easily reproduce it.
>>>
>>> Is there a way I can make ns-slapd handle a sudden
>>> bump in incoming request for ipa-client-install
>>>
>>> Thanks
>>> Rakesh
>>>
>>>
>>>
>>>
>>>
>>>
>>> On Mon, Aug 29, 2016 at 11:18 PM, Rich Megginson
>>> <rmeggins at redhat.com <mailto:rmeggins at redhat.com>>
>>> wrote:
>>>
>>> On 08/29/2016 10:53 AM, Rakesh Rajasekharan wrote:
>>>> Hi Thierry,
>>>>
>>>> My machine has 30GB RAM ..and 389-ds version
>>>> is 1.3.4
>>>>
>>>> ldapsearch shows the values for
>>>> nsslapd-cachememsize updated to 200MB.
>>>>
>>>> ldapsearch -LLL -o ldif-wrap=no -D
>>>> "cn=directory manager" -w 'mypassword' -b
>>>> 'cn=userRoot,cn=ldbm
>>>> database,cn=plugins,cn=config'|grep
>>>> nsslapd-cachememsize
>>>> nsslapd-cachememsize: 209715200
>>>>
>>>>
>>>> So, it seems to have updated though seeing that
>>>> warning(WARNING: ipaca: entry cache size
>>>> 10485760B is less than db size 11599872B) in
>>>> the log confuses me a bit.
>>>>
>>>> Thers one more entry that I found from the
>>>> ldapsearch to be bit low
>>>>
>>>> nsslapd-dncachememsize: 10485760
>>>> maxdncachesize: 10485760
>>>>
>>>> Should I update these as well to a higher value
>>>>
>>>> At the time when the issue happened, the memory
>>>> usage as well as the overall load of the system
>>>> was very low .
>>>> I will try reproducing the issue atleast in my
>>>> QA env..probably by trying to mock simultaneous
>>>> parallel logins to a large number of hosts
>>>
>>> To monitor your cache sizes, please use the
>>> dbmon.sh tool provided with your distro. If
>>> that is not available with your particular
>>> distro, see
>>> https://github.com/richm/scripts/wiki/dbmon.sh
>>> <https://github.com/richm/scripts/wiki/dbmon.sh>
>>>
>>>
>>>>
>>>>
>>>> thanks
>>>> Rakesh
>>>>
>>>>
>>>>
>>>>
>>>> On Mon, Aug 29, 2016 at 8:16 PM, thierry bordaz
>>>> <tbordaz at redhat.com
>>>> <mailto:tbordaz at redhat.com>> wrote:
>>>>
>>>> Hi Rakesh,
>>>>
>>>> Those tuning may depend on the memory
>>>> available on your machine.
>>>> nsslapd-cachememsize allows the entry cache
>>>> to consume up to 200Mb but its memory
>>>> footprint is known to go above.
>>>> 200Mb both looks pretty good to me. How
>>>> large is your machine ? What is your
>>>> version of 389-ds ?
>>>>
>>>> Those warnings do not change your settings.
>>>> It just raise that entry cache of 'ipaca'
>>>> and 'retrocl' are small but it is fine. The
>>>> size of the entry cache is important mostly
>>>> in userRoot.
>>>> You may double check the actual values,
>>>> after restart, with ldapsearch on
>>>> 'cn=userRoot,cn=ldbm
>>>> database,cn=plugins,cn=config' and
>>>> 'cn=config,cn=ldbm
>>>> database,cn=plugins,cn=config'.
>>>>
>>>> A step is to know what will be response
>>>> time of DS to know if it is responsible of
>>>> the hang or not.
>>>> The logs and possibly pstack during those
>>>> intermittent hangs will help to determine that.
>>>>
>>>> regards
>>>> thierry
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> On 08/29/2016 04:25 PM, Rakesh Rajasekharan
>>>> wrote:
>>>>> I tried increasing the nsslapd-dbcachesize
>>>>> and nsslapd-cachememsize in my QA envs to
>>>>> 200MB.
>>>>>
>>>>> However, in my log files, I still see this
>>>>> message
>>>>> [29/Aug/2016:04:34:37 +0000] - WARNING:
>>>>> ipaca: entry cache size 10485760B is less
>>>>> than db size 11599872B; We recommend to
>>>>> increase the entry cache size
>>>>> nsslapd-cachememsize.
>>>>> [29/Aug/2016:04:34:37 +0000] - WARNING:
>>>>> changelog: entry cache size 2097152B is
>>>>> less than db size 441647104B; We recommend
>>>>> to increase the entry cache size
>>>>> nsslapd-cachememsize.
>>>>>
>>>>> these are my ldif files that i used to
>>>>> modify the values
>>>>> modify entry cache size
>>>>> cat modify-cache-mem-size.ldif
>>>>> dn: cn=userRoot,cn=ldbm
>>>>> database,cn=plugins,cn=config
>>>>> changetype: modify
>>>>> replace: nsslapd-cachememsize
>>>>> nsslapd-cachememsize: 209715200
>>>>>
>>>>> modify db cache size
>>>>> cat modfy-db-cache-size.ldif
>>>>> dn: cn=config,cn=ldbm
>>>>> database,cn=plugins,cn=config
>>>>> changetype: modify
>>>>> replace: nsslapd-dbcachesize
>>>>> nsslapd-dbcachesize: 209715200
>>>>>
>>>>> After modifying , i restarted IPA services
>>>>>
>>>>> Is there anything else that I need to
>>>>> take care of as the logs suggest its still
>>>>> not getting the updated values
>>>>>
>>>>> Thanks
>>>>> Rakesh
>>>>>
>>>>> On Mon, Aug 29, 2016 at 6:07 PM, Rakesh
>>>>> Rajasekharan
>>>>> <rakesh.rajasekharan at gmail.com
>>>>> <mailto:rakesh.rajasekharan at gmail.com>> wrote:
>>>>>
>>>>> Hi Thierry,
>>>>>
>>>>> Coz of the issues we had to revert
>>>>> back to earlier running openldap in
>>>>> production.
>>>>>
>>>>> I have now done a few TCP related
>>>>> changes in sysctl.conf and have also
>>>>> increased the nsslapd-dbcachesize and
>>>>> nsslapd-cachememsize to 200MB
>>>>>
>>>>> I will again start migrating hosts
>>>>> back to IPA and see if I face the
>>>>> earlier issue.
>>>>>
>>>>> I will update back once I have something
>>>>>
>>>>>
>>>>> Thanks,
>>>>> Rakesh
>>>>>
>>>>>
>>>>>
>>>>> On Thu, Aug 25, 2016 at 2:17 PM,
>>>>> thierry bordaz <tbordaz at redhat.com
>>>>> <mailto:tbordaz at redhat.com>> wrote:
>>>>>
>>>>>
>>>>>
>>>>> On 08/25/2016 10:15 AM, Rakesh
>>>>> Rajasekharan wrote:
>>>>>> All of the troubleshooting seems
>>>>>> fine.
>>>>>>
>>>>>>
>>>>>> However, Running libconv.pl
>>>>>> <http://libconv.pl> gives me this
>>>>>> output
>>>>>>
>>>>>> ----- Recommendations -----
>>>>>>
>>>>>> 1. You have unindexed
>>>>>> components, this can be caused
>>>>>> from a search on an unindexed
>>>>>> attribute, or your returned
>>>>>> results exceeded the
>>>>>> allidsthreshold. Unindexed
>>>>>> components are not recommended.
>>>>>> To refuse unindexed searches,
>>>>>> switch 'nsslapd-require-index' to
>>>>>> 'on' under your database entry
>>>>>> (e.g. cn=UserRoot,cn=ldbm
>>>>>> database,cn=plugins,cn=config).
>>>>>>
>>>>>> 2. You have a significant
>>>>>> difference between binds and
>>>>>> unbinds. You may want to
>>>>>> investigate this difference.
>>>>>>
>>>>>>
>>>>>> I feel, this could be a pointer
>>>>>> to things going slow.. and IPA
>>>>>> hanging. I think i now have
>>>>>> something that I can try and nail
>>>>>> down this issue.
>>>>>>
>>>>>> On a sidenote, I was earlier
>>>>>> running openldap and migrated
>>>>>> over to Freeipa,
>>>>>>
>>>>>> Thanks
>>>>>> Rakesh
>>>>>>
>>>>>>
>>>>>>
>>>>>> On Wed, Aug 24, 2016 at 12:38 PM,
>>>>>> Petr Spacek <pspacek at redhat.com
>>>>>> <mailto:pspacek at redhat.com>> wrote:
>>>>>>
>>>>>> On 23.8.2016 18:44, Rakesh
>>>>>> Rajasekharan wrote:
>>>>>> > I think thers something
>>>>>> seriously wrong with my system
>>>>>> >
>>>>>> > not able to run any IPA
>>>>>> commands
>>>>>> >
>>>>>> > klist
>>>>>> > Ticket cache:
>>>>>> KEYRING:persistent:0:0
>>>>>> > Default principal:
>>>>>> admin at XYZ.COM
>>>>>> <mailto:admin at XYZ.COM>
>>>>>> >
>>>>>> > Valid starting Expires
>>>>>> Service principal
>>>>>> > 2016-08-23T16:26:36
>>>>>> 2016-08-24T16:26:22
>>>>>> krbtgt/XYZ.COM at XYZ.COM
>>>>>> <mailto:XYZ.COM at XYZ.COM>
>>>>>> >
>>>>>> >
>>>>>> > [root at prod-ipa-master-1a
>>>>>> :~] ipactl status
>>>>>> > Directory Service: RUNNING
>>>>>> > krb5kdc Service: RUNNING
>>>>>> > kadmin Service: RUNNING
>>>>>> > ipa_memcached Service: RUNNING
>>>>>> > httpd Service: RUNNING
>>>>>> > pki-tomcatd Service: RUNNING
>>>>>> > ipa-otpd Service: RUNNING
>>>>>> > ipa: INFO: The ipactl
>>>>>> command was successful
>>>>>> >
>>>>>> >
>>>>>> >
>>>>>> > [root at prod-ipa-master :~]
>>>>>> ipa user-find p-testuser
>>>>>> > ipa: ERROR: Kerberos error:
>>>>>> ('Unspecified GSS failure.
>>>>>> Minor code may
>>>>>> > provide more information',
>>>>>> 851968)/("Cannot contact any
>>>>>> KDC for realm '
>>>>>> > XYZ.COM <http://XYZ.COM>'",
>>>>>> -1765328228)
>>>>>>
>>>>>
>>>>> Hi Rakesh,
>>>>>
>>>>> Having a reproducible test
>>>>> case would you rerun the
>>>>> command above.
>>>>> During its processing you may
>>>>> monitor DS process load (top).
>>>>> If it is high, you may get
>>>>> some pstacks of it.
>>>>> Also would you attach the part
>>>>> of DS access logs taken during
>>>>> the command.
>>>>>
>>>>> regards
>>>>> thierry
>>>>>
>>>>>> >
>>>>>>
>>>>>> This is weird because the
>>>>>> server seems to be up.
>>>>>>
>>>>>> Please follow
>>>>>> http://www.freeipa.org/page/Troubleshooting#Authentication.2FKerberos
>>>>>>
>>>>>> Petr^2 Spacek
>>>>>>
>>>>>> >
>>>>>> >
>>>>>> > Thanks
>>>>>> >
>>>>>> > Rakesh
>>>>>> >
>>>>>> > On Tue, Aug 23, 2016 at
>>>>>> 10:01 PM, Rakesh Rajasekharan <
>>>>>> >
>>>>>> rakesh.rajasekharan at gmail.com
>>>>>> <mailto:rakesh.rajasekharan at gmail.com>>
>>>>>> wrote:
>>>>>> >
>>>>>> >> i changed the loggin level
>>>>>> to 4 . Modifying
>>>>>> nsslapd-accesslog-level
>>>>>> >>
>>>>>> >> But, the hang is still
>>>>>> there. though I dont see the
>>>>>> sigfault now
>>>>>> >>
>>>>>> >>
>>>>>> >>
>>>>>> >>
>>>>>> >> On Tue, Aug 23, 2016 at
>>>>>> 9:02 PM, Rakesh Rajasekharan <
>>>>>> >>
>>>>>> rakesh.rajasekharan at gmail.com
>>>>>> <mailto:rakesh.rajasekharan at gmail.com>>
>>>>>> wrote:
>>>>>> >>
>>>>>> >>> My disk was getting
>>>>>> filled too fast
>>>>>> >>>
>>>>>> >>> logs under
>>>>>> /var/log/dirsrv was coming
>>>>>> around 5 gb quickly filling up
>>>>>> >>>
>>>>>> >>> Is there a way to make
>>>>>> the logging less verbose
>>>>>> >>>
>>>>>> >>>
>>>>>> >>>
>>>>>> >>> On Tue, Aug 23, 2016 at
>>>>>> 6:41 PM, Petr Spacek
>>>>>> <pspacek at redhat.com
>>>>>> <mailto:pspacek at redhat.com>>
>>>>>> wrote:
>>>>>> >>>
>>>>>> >>>> On 23.8.2016 15:07,
>>>>>> Rakesh Rajasekharan wrote:
>>>>>> >>>>> I was able to fix that
>>>>>> may be temporarily... when i
>>>>>> checked the
>>>>>> >>>> network..
>>>>>> >>>>> there was another
>>>>>> process that was running and
>>>>>> consuming a lot of
>>>>>> >>>> network (
>>>>>> >>>>> i have no idea who did
>>>>>> that. I need to seriously
>>>>>> start restricting
>>>>>> >>>> people
>>>>>> >>>>> access to this machine )
>>>>>> >>>>>
>>>>>> >>>>> after killing that
>>>>>> perfomance improved drastically
>>>>>> >>>>>
>>>>>> >>>>> But now, suddenly I
>>>>>> started experiencing the same
>>>>>> hang.
>>>>>> >>>>>
>>>>>> >>>>> This time , I gert the
>>>>>> following error when checked
>>>>>> dmesg
>>>>>> >>>>>
>>>>>> >>>>> [ 301.236976]
>>>>>> ns-slapd[3124]: segfault at 0
>>>>>> ip 00007f1de416951c sp
>>>>>> >>>>> 00007f1dee1dba70 error
>>>>>> 4 in
>>>>>> libcos-plugin.so[7f1de4166000+b000]
>>>>>> >>>>> [ 1116.248431] TCP:
>>>>>> request_sock_TCP: Possible
>>>>>> SYN flooding on port 88.
>>>>>> >>>>> Sending cookies. Check
>>>>>> SNMP counters.
>>>>>> >>>>> [11831.397037]
>>>>>> ns-slapd[22550]: segfault at
>>>>>> 0 ip 00007f533d82251c sp
>>>>>> >>>>> 00007f5347894a70 error
>>>>>> 4 in
>>>>>> libcos-plugin.so[7f533d81f000+b000]
>>>>>> >>>>> [11832.727989]
>>>>>> ns-slapd[22606]: segfault at
>>>>>> 0 ip 00007f6231eb951c sp
>>>>>> >>>>> 00007f623bf2ba70 error
>>>>>> 4 in
>>>>>> libcos-plugin.so[7f6231eb6000+b00
>>>>>> >>>>
>>>>>> >>>> Okay, this one is
>>>>>> serious. The LDAP server crashed.
>>>>>> >>>>
>>>>>> >>>> 1. Make sure all your
>>>>>> packages are up-to-date.
>>>>>> >>>>
>>>>>> >>>> Please see
>>>>>> >>>>
>>>>>> http://directory.fedoraproject.org/docs/389ds/FAQ/faq.html#d
>>>>>> >>>> ebugging-crashes
>>>>>> >>>> for further instructions
>>>>>> how to debug this.
>>>>>> >>>>
>>>>>> >>>> Petr^2 Spacek
>>>>>> >>>>
>>>>>> >>>>>
>>>>>> >>>>> and in
>>>>>> /var/log/dirsrv/example-com/errors
>>>>>> >>>>>
>>>>>> >>>>> [23/Aug/2016:12:49:36
>>>>>> +0000] DSRetroclPlugin -
>>>>>> delete_changerecord:
>>>>>> >>>> could
>>>>>> >>>>> not delete change
>>>>>> record 3291138 (rc: 32)
>>>>>> >>>>> [23/Aug/2016:12:49:36
>>>>>> +0000] DSRetroclPlugin -
>>>>>> delete_changerecord:
>>>>>> >>>> could
>>>>>> >>>>> not delete change
>>>>>> record 3291139 (rc: 32)
>>>>>> >>>>> [23/Aug/2016:12:49:36
>>>>>> +0000] DSRetroclPlugin -
>>>>>> delete_changerecord:
>>>>>> >>>> could
>>>>>> >>>>> not delete change
>>>>>> record 3291140 (rc: 32)
>>>>>> >>>>> [23/Aug/2016:12:49:36
>>>>>> +0000] DSRetroclPlugin -
>>>>>> delete_changerecord:
>>>>>> >>>> could
>>>>>> >>>>> not delete change
>>>>>> record 3291141 (rc: 32)
>>>>>> >>>>> [23/Aug/2016:12:49:36
>>>>>> +0000] DSRetroclPlugin -
>>>>>> delete_changerecord:
>>>>>> >>>> could
>>>>>> >>>>> not delete change
>>>>>> record 3291142 (rc: 32)
>>>>>> >>>>> [23/Aug/2016:12:49:36
>>>>>> +0000] DSRetroclPlugin -
>>>>>> delete_changerecord:
>>>>>> >>>> could
>>>>>> >>>>> not delete change
>>>>>> record 3291143 (rc: 32)
>>>>>> >>>>> [23/Aug/2016:12:49:36
>>>>>> +0000] DSRetroclPlugin -
>>>>>> delete_changerecord:
>>>>>> >>>> could
>>>>>> >>>>> not delete change
>>>>>> record 3291144 (rc: 32)
>>>>>> >>>>> [23/Aug/2016:12:49:36
>>>>>> +0000] DSRetroclPlugin -
>>>>>> delete_changerecord:
>>>>>> >>>> could
>>>>>> >>>>> not delete change
>>>>>> record 3291145 (rc: 32)
>>>>>> >>>>> [23/Aug/2016:12:49:50
>>>>>> +0000] - Retry count exceeded
>>>>>> in delete
>>>>>> >>>>> [23/Aug/2016:12:49:50
>>>>>> +0000] DSRetroclPlugin -
>>>>>> delete_changerecord:
>>>>>> >>>> could
>>>>>> >>>>> not delete change
>>>>>> record 3292734 (rc: 51)
>>>>>> >>>>>
>>>>>> >>>>>
>>>>>> >>>>> Can i do something
>>>>>> about this error.. I treid to
>>>>>> restart ipa a couple
>>>>>> >>>> of
>>>>>> >>>>> time but that did not help
>>>>>> >>>>>
>>>>>> >>>>> Thanks
>>>>>> >>>>> Rakesh
>>>>>> >>>>>
>>>>>> >>>>> On Mon, Aug 22, 2016 at
>>>>>> 2:27 PM, Petr Spacek
>>>>>> <pspacek at redhat.com
>>>>>> <mailto:pspacek at redhat.com>>
>>>>>> >>>> wrote:
>>>>>> >>>>>
>>>>>> >>>>>> On 19.8.2016 19:32,
>>>>>> Rakesh Rajasekharan wrote:
>>>>>> >>>>>>> I am running my set
>>>>>> up on AWS cloud, and entropy
>>>>>> is low at around
>>>>>> >>>> 180 .
>>>>>> >>>>>>>
>>>>>> >>>>>>> I plan to increase it
>>>>>> bu installing haveged . But,
>>>>>> would low entropy
>>>>>> >>>> by
>>>>>> >>>>>> any
>>>>>> >>>>>>> chance cause this
>>>>>> issue of intermittent hang .
>>>>>> >>>>>>> Also, the hang is
>>>>>> mostly observed when
>>>>>> registering around 20 clients
>>>>>> >>>>>>> together
>>>>>> >>>>>>
>>>>>> >>>>>> Possibly, I'm not
>>>>>> sure. If you want to dig into
>>>>>> this, I would do this:
>>>>>> >>>>>> 1. look what process
>>>>>> hangs on client (using pstree
>>>>>> command or so)
>>>>>> >>>>>> $ pstree
>>>>>> >>>>>>
>>>>>> >>>>>> 2. look to what server
>>>>>> and port is the hanging
>>>>>> client connected to
>>>>>> >>>>>> $ lsof -p <PID of the
>>>>>> hanging process>
>>>>>> >>>>>>
>>>>>> >>>>>> 3. jump to server and
>>>>>> see what process is bound to
>>>>>> the target port
>>>>>> >>>>>> $ netstat -pn
>>>>>> >>>>>>
>>>>>> >>>>>> 4. see where the
>>>>>> process if hanging
>>>>>> >>>>>> $ strace -p <PID of
>>>>>> the hanging process>
>>>>>> >>>>>>
>>>>>> >>>>>> I hope it helps.
>>>>>> >>>>>>
>>>>>> >>>>>> Petr^2 Spacek
>>>>>> >>>>>>
>>>>>> >>>>>>> On Fri, Aug 19, 2016
>>>>>> at 7:24 PM, Rakesh Rajasekharan <
>>>>>> >>>>>>>
>>>>>> rakesh.rajasekharan at gmail.com
>>>>>> <mailto:rakesh.rajasekharan at gmail.com>>
>>>>>> wrote:
>>>>>> >>>>>>>
>>>>>> >>>>>>>> yes there seems to
>>>>>> be something thats worrying..
>>>>>> I have faced this
>>>>>> >>>> today
>>>>>> >>>>>>>> as well.
>>>>>> >>>>>>>> There are few hosts
>>>>>> around 280 odd left and when
>>>>>> i try adding them
>>>>>> >>>> to
>>>>>> >>>>>> IPA
>>>>>> >>>>>>>> , the slowness begins..
>>>>>> >>>>>>>>
>>>>>> >>>>>>>> all the ipa commands
>>>>>> like ipa user-find.. etc
>>>>>> becomes very slow in
>>>>>> >>>>>>>> responding.
>>>>>> >>>>>>>>
>>>>>> >>>>>>>> the SYNC_RECV are
>>>>>> not many though just around
>>>>>> 80-90 and today that
>>>>>> >>>> was
>>>>>> >>>>>>>> around 20 only
>>>>>> >>>>>>>>
>>>>>> >>>>>>>>
>>>>>> >>>>>>>> I have for now
>>>>>> increased tcp_max_syn_backlog
>>>>>> to 5000.
>>>>>> >>>>>>>> For now the slowness
>>>>>> seems to have gone.. but I
>>>>>> will do a try
>>>>>> >>>> adding the
>>>>>> >>>>>>>> clients again
>>>>>> tomorrow and see how it goes
>>>>>> >>>>>>>>
>>>>>> >>>>>>>> Thanks
>>>>>> >>>>>>>> Rakesh
>>>>>> >>>>>>>>
>>>>>> >>>>>>>> The issues
>>>>>> >>>>>>>>
>>>>>> >>>>>>>> On Fri, Aug 19, 2016
>>>>>> at 12:58 PM, Petr Spacek
>>>>>> <pspacek at redhat.com
>>>>>> <mailto:pspacek at redhat.com>>
>>>>>> >>>>>> wrote:
>>>>>> >>>>>>>>
>>>>>> >>>>>>>>> On 18.8.2016 17:23,
>>>>>> Rakesh Rajasekharan wrote:
>>>>>> >>>>>>>>>> Hi
>>>>>> >>>>>>>>>>
>>>>>> >>>>>>>>>> I am migrating to
>>>>>> freeipa from openldap and
>>>>>> have around 4000
>>>>>> >>>> clients
>>>>>> >>>>>>>>>>
>>>>>> >>>>>>>>>> I had openned a
>>>>>> another thread on that, but
>>>>>> chose to start a new
>>>>>> >>>> one
>>>>>> >>>>>>>>> here
>>>>>> >>>>>>>>>> as its a separate
>>>>>> issue
>>>>>> >>>>>>>>>>
>>>>>> >>>>>>>>>> I was able to
>>>>>> change the
>>>>>> nssslapd-maxdescriptors
>>>>>> adding an ldif
>>>>>> >>>> file
>>>>>> >>>>>>>>>>
>>>>>> >>>>>>>>>> cat
>>>>>> nsslapd-modify.ldif
>>>>>> >>>>>>>>>> dn: cn=config
>>>>>> >>>>>>>>>> changetype: modify
>>>>>> >>>>>>>>>> replace:
>>>>>> nsslapd-maxdescriptors
>>>>>> >>>>>>>>>>
>>>>>> nsslapd-maxdescriptors: 17000
>>>>>> >>>>>>>>>>
>>>>>> >>>>>>>>>> and running the
>>>>>> ldapmodify command
>>>>>> >>>>>>>>>>
>>>>>> >>>>>>>>>> I have now started
>>>>>> moving clients running an
>>>>>> openldap to Freeipa
>>>>>> >>>> and
>>>>>> >>>>>>>>> have
>>>>>> >>>>>>>>>> today moved close
>>>>>> to 2000 clients
>>>>>> >>>>>>>>>>
>>>>>> >>>>>>>>>> However, I have
>>>>>> noticed that IPA hangs
>>>>>> intermittently.
>>>>>> >>>>>>>>>>
>>>>>> >>>>>>>>>> running a kinit
>>>>>> admin returns the below error
>>>>>> >>>>>>>>>> kinit: Generic
>>>>>> error (see e-text) while
>>>>>> getting initial
>>>>>> >>>> credentials
>>>>>> >>>>>>>>>>
>>>>>> >>>>>>>>>> from the
>>>>>> /var/log/messages, I see this
>>>>>> entry
>>>>>> >>>>>>>>>>
>>>>>> >>>>>>>>>>
>>>>>> prod-ipa-master-int kernel:
>>>>>> [104090.315801] TCP:
>>>>>> >>>> request_sock_TCP:
>>>>>> >>>>>>>>>> Possible SYN
>>>>>> flooding on port 88. Sending
>>>>>> cookies. Check SNMP
>>>>>> >>>>>> counters.
>>>>>> >>>>>>>>>
>>>>>> >>>>>>>>> I would be worried
>>>>>> about this message. Maybe
>>>>>> kernel/firewall is
>>>>>> >>>> doing
>>>>>> >>>>>>>>> something fishy
>>>>>> behind your back and blocking
>>>>>> some connections or
>>>>>> >>>> so.
>>>>>> >>>>>>>>>
>>>>>> >>>>>>>>> Petr^2 Spacek
>>>>>> >>>>>>>>>
>>>>>> >>>>>>>>>
>>>>>> >>>>>>>>>> Aug 18 13:00:01
>>>>>> prod-ipa-master-int
>>>>>> systemd[1]: Started Session
>>>>>> >>>> 4885
>>>>>> >>>>>> of
>>>>>> >>>>>>>>>> user root.
>>>>>> >>>>>>>>>> Aug 18 13:00:01
>>>>>> prod-ipa-master-int
>>>>>> systemd[1]: Starting Session
>>>>>> >>>> 4885
>>>>>> >>>>>> of
>>>>>> >>>>>>>>>> user root.
>>>>>> >>>>>>>>>> Aug 18 13:01:01
>>>>>> prod-ipa-master-int
>>>>>> systemd[1]: Started Session
>>>>>> >>>> 4886
>>>>>> >>>>>> of
>>>>>> >>>>>>>>>> user root.
>>>>>> >>>>>>>>>> Aug 18 13:01:01
>>>>>> prod-ipa-master-int
>>>>>> systemd[1]: Starting Session
>>>>>> >>>> 4886
>>>>>> >>>>>> of
>>>>>> >>>>>>>>>> user root.
>>>>>> >>>>>>>>>> Aug 18 13:02:40
>>>>>> prod-ipa-master-int
>>>>>> python[28984]: ansible-command
>>>>>> >>>>>>>>> Invoked
>>>>>> >>>>>>>>>> with creates=None
>>>>>> executable=None shell=True
>>>>>> args= removes=None
>>>>>> >>>>>>>>> warn=True
>>>>>> >>>>>>>>>> chdir=None
>>>>>> >>>>>>>>>> Aug 18 13:04:37
>>>>>> prod-ipa-master-int sssd_be:
>>>>>> GSSAPI Error:
>>>>>> >>>> Unspecified
>>>>>> >>>>>>>>> GSS
>>>>>> >>>>>>>>>> failure. Minor
>>>>>> code may provide more
>>>>>> information (KDC returned
>>>>>> >>>> error
>>>>>> >>>>>>>>>> string: PROCESS_TGS)
>>>>>> >>>>>>>>>>
>>>>>> >>>>>>>>>> Could it be
>>>>>> possible that its due to the
>>>>>> initial load of adding
>>>>>> >>>> the
>>>>>> >>>>>>>>> clients
>>>>>> >>>>>>>>>> or is there
>>>>>> something else that I need to
>>>>>> take care of.
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>>
>>>>
>>>>
>>>>
>>>>
>>>
>>>
>>> --
>>> Manage your subscription for the Freeipa-users
>>> mailing list:
>>> https://www.redhat.com/mailman/listinfo/freeipa-users
>>> <https://www.redhat.com/mailman/listinfo/freeipa-users>
>>> Go to http://freeipa.org for more info on the
>>> project
>>>
>>>
>>>
>>>
>>>
>>
>>
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/freeipa-users/attachments/20160905/392213b0/attachment.htm>
More information about the Freeipa-users
mailing list