[Freeipa-users] Freeipa 4.2.0 hangs intermittently
Rakesh Rajasekharan
rakesh.rajasekharan at gmail.com
Mon Sep 5 12:14:09 UTC 2016
btw the values for opscompleted and opsinitiated are
ldapsearch -D "cn=directory manager" -w xxxx -LLL -b "cn=monitor" -s base
opsinitiated opscompleted
dn: cn=monitor
opsinitiated: 25655
opscompleted: 25654
Thanks
On Mon, Sep 5, 2016 at 5:35 PM, Rakesh Rajasekharan <
rakesh.rajasekharan at gmail.com> wrote:
> Hi Thierry,
>
> I was able to now reporduce the issue..
>
> the ns-slapd process just got hung,.. however, no core file was generate..
>
> I manually created a coredump using gcore and then took a strace of it
> this is strace from the coredump generated
>
> GNU gdb (GDB) Red Hat Enterprise Linux 7.6.1-80.el7
> Copyright (C) 2013 Free Software Foundation, Inc.
> License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.
> html>
> This is free software: you are free to change and redistribute it.
> There is NO WARRANTY, to the extent permitted by law. Type "show copying"
> and "show warranty" for details.
> This GDB was configured as "x86_64-redhat-linux-gnu".
> For bug reporting instructions, please see:
> <http://www.gnu.org/software/gdb/bugs/>...
> Reading symbols from /usr/sbin/ns-slapd...Reading symbols from
> /usr/sbin/ns-slapd...(no debugging symbols found)...done.
> (no debugging symbols found)...done.
> [New LWP 105159]
> [New LWP 105160]
> [New LWP 105161]
> [New LWP 105162]
> [New LWP 105163]
> [New LWP 105164]
> [New LWP 105165]
> [New LWP 105166]
> [New LWP 105167]
> [New LWP 105168]
> [New LWP 105171]
> [New LWP 105172]
> [New LWP 105173]
> [New LWP 105174]
> [New LWP 105175]
> [New LWP 105176]
> [New LWP 105177]
> [New LWP 105178]
> [New LWP 105179]
> [New LWP 105180]
> [New LWP 105181]
> [New LWP 105182]
> [New LWP 105183]
> [New LWP 105184]
> [New LWP 105185]
> [New LWP 105186]
> [New LWP 105187]
> [New LWP 105188]
> [New LWP 105189]
> [New LWP 105190]
> [New LWP 105191]
> [New LWP 105192]
> [New LWP 105193]
> [New LWP 105194]
> [New LWP 105195]
> [New LWP 105196]
> [New LWP 105197]
> [New LWP 105198]
> [New LWP 105199]
> [New LWP 105200]
> [New LWP 105201]
> [New LWP 105203]
> [New LWP 105215]
> [New LWP 105158]
> [Thread debugging using libthread_db enabled]
> Using host libthread_db library "/lib64/libthread_db.so.1".
> Core was generated by `/usr/sbin/ns-slapd'.
> #0 0x00007f1b48ae69b3 in select () from /lib64/libc.so.6
>
> Thread 44 (Thread 0x7f1b4b6d9840 (LWP 105158)):
> #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 () from
> /lib64/libpthread.so.0
> No symbol table info available.
> #1 0x00007f1b4941b2c3 in PR_EnterMonitor () from /lib64/libnspr4.so
> No symbol table info available.
> #2 0x00007f1b4b7125ed in setup_pr_read_pds ()
> No symbol table info available.
> #3 0x00007f1b4b715fd9 in slapd_daemon ()
> No symbol table info available.
> #4 0x00007f1b4b707c6c in main ()
> No symbol table info available.
>
> Thread 43 (Thread 0x7f1a5df0f700 (LWP 105215)):
> #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 () from
> /lib64/libpthread.so.0
> No symbol table info available.
> #1 0x00007f1b4941b050 in PR_WaitCondVar () from /lib64/libnspr4.so
> No symbol table info available.
> #2 0x00007f1b4b71df25 in ps_send_results ()
> No symbol table info available.
> #3 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so
> No symbol table info available.
> #4 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0
> No symbol table info available.
> #5 0x00007f1b48aef28d in clone () from /lib64/libc.so.6
> No symbol table info available.
>
> Thread 42 (Thread 0x7f1b03fe7700 (LWP 105203)):
> #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 () from
> /lib64/libpthread.so.0
> No symbol table info available.
> #1 0x00007f1b4941b050 in PR_WaitCondVar () from /lib64/libnspr4.so
> No symbol table info available.
> #2 0x00007f1b4b71df25 in ps_send_results ()
> No symbol table info available.
> #3 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so
> No symbol table info available.
> #4 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0
> No symbol table info available.
> #5 0x00007f1b48aef28d in clone () from /lib64/libc.so.6
> No symbol table info available.
>
> Thread 41 (Thread 0x7f1b047e8700 (LWP 105201)):
> #0 0x00007f1b48ae69b3 in select () from /lib64/libc.so.6
> No symbol table info available.
> #1 0x00007f1b4b2710e9 in DS_Sleep () from /usr/lib64/dirsrv/libslapd.so.0
> No symbol table info available.
> #2 0x00007f1b4b712325 in time_thread ()
> No symbol table info available.
> #3 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so
> No symbol table info available.
> #4 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0
> No symbol table info available.
> #5 0x00007f1b48aef28d in clone () from /lib64/libc.so.6
> No symbol table info available.
>
> Thread 40 (Thread 0x7f1b04fe9700 (LWP 105200)):
> #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 () from
> /lib64/libpthread.so.0
> No symbol table info available.
> #1 0x00007f1b433212f3 in __db_hybrid_mutex_suspend () from /lib64/
> libdb-5.3.so
> No symbol table info available.
> #2 0x00007f1b43320640 in __db_tas_mutex_lock () from /lib64/libdb-5.3.so
> No symbol table info available.
> #3 0x00007f1b4343ae1b in __memp_fget () from /lib64/libdb-5.3.so
> No symbol table info available.
> #4 0x00007f1b4333e211 in __bam_search () from /lib64/libdb-5.3.so
> No symbol table info available.
> #5 0x00007f1b43329256 in __bamc_search () from /lib64/libdb-5.3.so
> No symbol table info available.
> #6 0x00007f1b4332d6c4 in __bamc_put () from /lib64/libdb-5.3.so
> No symbol table info available.
> #7 0x00007f1b433e46d5 in __dbc_iput () from /lib64/libdb-5.3.so
> No symbol table info available.
> #8 0x00007f1b433dfa0e in __db_put () from /lib64/libdb-5.3.so
> No symbol table info available.
> #9 0x00007f1b433f4fa4 in __db_put_pp () from /lib64/libdb-5.3.so
> No symbol table info available.
> #10 0x00007f1b3e8cf4cf in idl_new_insert_key () from
> /usr/lib64/dirsrv/plugins/libback-ldbm.so
> No symbol table info available.
> #11 0x00007f1b3e8de5b5 in addordel_values_sv.isra.3 () from
> /usr/lib64/dirsrv/plugins/libback-ldbm.so
> No symbol table info available.
> #12 0x00007f1b3e8df25a in index_addordel_values_ext_sv () from
> /usr/lib64/dirsrv/plugins/libback-ldbm.so
> No symbol table info available.
> #13 0x00007f1b3e8df694 in index_addordel_values_sv () from
> /usr/lib64/dirsrv/plugins/libback-ldbm.so
> No symbol table info available.
> #14 0x00007f1b3e8df75a in index_addordel_entry () from
> /usr/lib64/dirsrv/plugins/libback-ldbm.so
> No symbol table info available.
> #15 0x00007f1b3e8e25fa in ldbm_back_add () from /usr/lib64/dirsrv/plugins/
> libback-ldbm.so
> No symbol table info available.
> #16 0x00007f1b4b1e4cca in op_shared_add () from
> /usr/lib64/dirsrv/libslapd.so.0
> No symbol table info available.
> #17 0x00007f1b4b1e6020 in do_add () from /usr/lib64/dirsrv/libslapd.so.0
> No symbol table info available.
> #18 0x00007f1b4b711303 in connection_threadmain ()
> No symbol table info available.
> #19 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so
> No symbol table info available.
> #20 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0
> No symbol table info available.
> #21 0x00007f1b48aef28d in clone () from /lib64/libc.so.6
> No symbol table info available.
>
> Thread 39 (Thread 0x7f1b057ea700 (LWP 105199)):
> #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 () from
> /lib64/libpthread.so.0
> No symbol table info available.
> #1 0x00007f1b4941b050 in PR_WaitCondVar () from /lib64/libnspr4.so
> No symbol table info available.
> #2 0x00007f1b4b70efee in connection_wait_for_new_work ()
> No symbol table info available.
> #3 0x00007f1b4b710719 in connection_threadmain ()
> No symbol table info available.
> #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so
> No symbol table info available.
> #5 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0
> No symbol table info available.
> #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6
> No symbol table info available.
>
> Thread 38 (Thread 0x7f1b05feb700 (LWP 105198)):
> #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 () from
> /lib64/libpthread.so.0
> No symbol table info available.
> #1 0x00007f1b4941b050 in PR_WaitCondVar () from /lib64/libnspr4.so
> No symbol table info available.
> #2 0x00007f1b4b70efee in connection_wait_for_new_work ()
> No symbol table info available.
> #3 0x00007f1b4b710719 in connection_threadmain ()
> No symbol table info available.
> #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so
> No symbol table info available.
> #5 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0
> No symbol table info available.
> #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6
> No symbol table info available.
>
> Thread 37 (Thread 0x7f1b067ec700 (LWP 105197)):
> #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 () from
> /lib64/libpthread.so.0
> No symbol table info available.
> #1 0x00007f1b4941b050 in PR_WaitCondVar () from /lib64/libnspr4.so
> No symbol table info available.
> #2 0x00007f1b4b70efee in connection_wait_for_new_work ()
> No symbol table info available.
> #3 0x00007f1b4b710719 in connection_threadmain ()
> No symbol table info available.
> #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so
> No symbol table info available.
> #5 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0
> No symbol table info available.
> #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6
> No symbol table info available.
>
> Thread 36 (Thread 0x7f1b06fed700 (LWP 105196)):
> #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 () from
> /lib64/libpthread.so.0
> No symbol table info available.
> #1 0x00007f1b4941b050 in PR_WaitCondVar () from /lib64/libnspr4.so
> No symbol table info available.
> #2 0x00007f1b4b70efee in connection_wait_for_new_work ()
> No symbol table info available.
> #3 0x00007f1b4b710719 in connection_threadmain ()
> No symbol table info available.
> #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so
> No symbol table info available.
> #5 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0
> No symbol table info available.
> #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6
> No symbol table info available.
>
> Thread 35 (Thread 0x7f1b077ee700 (LWP 105195)):
> #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 () from
> /lib64/libpthread.so.0
> No symbol table info available.
> #1 0x00007f1b4941b2c3 in PR_EnterMonitor () from /lib64/libnspr4.so
> No symbol table info available.
> #2 0x00007f1b3e8c23e6 in dblayer_txn_begin () from
> /usr/lib64/dirsrv/plugins/libback-ldbm.so
> No symbol table info available.
> #3 0x00007f1b3e8fe007 in ldbm_back_modify () from
> /usr/lib64/dirsrv/plugins/libback-ldbm.so
> No symbol table info available.
> #4 0x00007f1b4b22b541 in op_shared_modify () from
> /usr/lib64/dirsrv/libslapd.so.0
> No symbol table info available.
> #5 0x00007f1b4b22c004 in modify_internal_pb () from
> /usr/lib64/dirsrv/libslapd.so.0
> No symbol table info available.
> #6 0x00007f1b3eb48b60 in ipapwd_apply_mods () from
> /usr/lib64/dirsrv/plugins/libipa_pwd_extop.so
> No symbol table info available.
> #7 0x00007f1b3eb4c8e0 in store_new_keys () from /usr/lib64/dirsrv/plugins/
> libipa_pwd_extop.so
> No symbol table info available.
> #8 0x00007f1b3eb4ead0 in ipapwd_extop () from /usr/lib64/dirsrv/plugins/
> libipa_pwd_extop.so
> No symbol table info available.
> #9 0x00007f1b4b23bba2 in plugin_call_exop_plugins () from
> /usr/lib64/dirsrv/libslapd.so.0
> No symbol table info available.
> #10 0x00007f1b4b717efe in do_extended ()
> No symbol table info available.
> #11 0x00007f1b4b71141a in connection_threadmain ()
> No symbol table info available.
> #12 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so
> No symbol table info available.
> #13 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0
> No symbol table info available.
> #14 0x00007f1b48aef28d in clone () from /lib64/libc.so.6
> No symbol table info available.
>
> Thread 34 (Thread 0x7f1b07fef700 (LWP 105194)):
> #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 () from
> /lib64/libpthread.so.0
> No symbol table info available.
> #1 0x00007f1b4941b050 in PR_WaitCondVar () from /lib64/libnspr4.so
> No symbol table info available.
> #2 0x00007f1b4b70efee in connection_wait_for_new_work ()
> No symbol table info available.
> #3 0x00007f1b4b710719 in connection_threadmain ()
> No symbol table info available.
> #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so
> No symbol table info available.
> #5 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0
> No symbol table info available.
> #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6
> No symbol table info available.
>
> Thread 33 (Thread 0x7f1b087f0700 (LWP 105193)):
> #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 () from
> /lib64/libpthread.so.0
> No symbol table info available.
> #1 0x00007f1b4941b050 in PR_WaitCondVar () from /lib64/libnspr4.so
> No symbol table info available.
> #2 0x00007f1b4b70efee in connection_wait_for_new_work ()
> No symbol table info available.
> #3 0x00007f1b4b710719 in connection_threadmain ()
> No symbol table info available.
> #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so
> No symbol table info available.
> #5 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0
> No symbol table info available.
> #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6
> No symbol table info available.
>
> Thread 32 (Thread 0x7f1b08ff1700 (LWP 105192)):
> #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 () from
> /lib64/libpthread.so.0
> No symbol table info available.
> #1 0x00007f1b4941b050 in PR_WaitCondVar () from /lib64/libnspr4.so
> No symbol table info available.
> #2 0x00007f1b4b70efee in connection_wait_for_new_work ()
> No symbol table info available.
> #3 0x00007f1b4b710719 in connection_threadmain ()
> No symbol table info available.
> #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so
> No symbol table info available.
> #5 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0
> No symbol table info available.
> #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6
> No symbol table info available.
>
> Thread 31 (Thread 0x7f1b097f2700 (LWP 105191)):
> #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 () from
> /lib64/libpthread.so.0
> No symbol table info available.
> #1 0x00007f1b4941b050 in PR_WaitCondVar () from /lib64/libnspr4.so
> No symbol table info available.
> #2 0x00007f1b4b70efee in connection_wait_for_new_work ()
> No symbol table info available.
> #3 0x00007f1b4b710719 in connection_threadmain ()
> No symbol table info available.
> #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so
> No symbol table info available.
> #5 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0
> No symbol table info available.
> #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6
> No symbol table info available.
>
> Thread 30 (Thread 0x7f1b09ff3700 (LWP 105190)):
> #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 () from
> /lib64/libpthread.so.0
> No symbol table info available.
> #1 0x00007f1b433212f3 in __db_hybrid_mutex_suspend () from /lib64/
> libdb-5.3.so
> No symbol table info available.
> #2 0x00007f1b43320640 in __db_tas_mutex_lock () from /lib64/libdb-5.3.so
> No symbol table info available.
> #3 0x00007f1b433cad3a in __lock_get_internal () from /lib64/libdb-5.3.so
> No symbol table info available.
> #4 0x00007f1b433cb820 in __lock_get () from /lib64/libdb-5.3.so
> No symbol table info available.
> #5 0x00007f1b433f7142 in __db_lget () from /lib64/libdb-5.3.so
> No symbol table info available.
> #6 0x00007f1b4333e605 in __bam_search () from /lib64/libdb-5.3.so
> No symbol table info available.
> #7 0x00007f1b43329256 in __bamc_search () from /lib64/libdb-5.3.so
> No symbol table info available.
> #8 0x00007f1b4332ad0f in __bamc_get () from /lib64/libdb-5.3.so
> No symbol table info available.
> #9 0x00007f1b433e3ca6 in __dbc_iget () from /lib64/libdb-5.3.so
> No symbol table info available.
> #10 0x00007f1b433f2b02 in __dbc_get_pp () from /lib64/libdb-5.3.so
> No symbol table info available.
> #11 0x00007f1b3e8ce680 in idl_new_fetch () from /usr/lib64/dirsrv/plugins/
> libback-ldbm.so
> No symbol table info available.
> #12 0x00007f1b3e8dcb26 in index_read_ext_allids () from
> /usr/lib64/dirsrv/plugins/libback-ldbm.so
> No symbol table info available.
> #13 0x00007f1b3e8c7284 in keys2idl () from /usr/lib64/dirsrv/plugins/
> libback-ldbm.so
> No symbol table info available.
> #14 0x00007f1b3e8c79e3 in ava_candidates.isra.0 () from
> /usr/lib64/dirsrv/plugins/libback-ldbm.so
> No symbol table info available.
> #15 0x00007f1b3e8c7fd2 in filter_candidates_ext () from
> /usr/lib64/dirsrv/plugins/libback-ldbm.so
> No symbol table info available.
> #16 0x00007f1b3e8c9096 in list_candidates () from
> /usr/lib64/dirsrv/plugins/libback-ldbm.so
> No symbol table info available.
> #17 0x00007f1b3e8c7f40 in filter_candidates_ext () from
> /usr/lib64/dirsrv/plugins/libback-ldbm.so
> No symbol table info available.
> #18 0x00007f1b3e8c9096 in list_candidates () from
> /usr/lib64/dirsrv/plugins/libback-ldbm.so
> No symbol table info available.
> #19 0x00007f1b3e8c7f40 in filter_candidates_ext () from
> /usr/lib64/dirsrv/plugins/libback-ldbm.so
> No symbol table info available.
> #20 0x00007f1b3e8c9096 in list_candidates () from
> /usr/lib64/dirsrv/plugins/libback-ldbm.so
> No symbol table info available.
> #21 0x00007f1b3e8c7f40 in filter_candidates_ext () from
> /usr/lib64/dirsrv/plugins/libback-ldbm.so
> No symbol table info available.
> #22 0x00007f1b3e90380c in subtree_candidates () from
> /usr/lib64/dirsrv/plugins/libback-ldbm.so
> No symbol table info available.
> #23 0x00007f1b3e904e7f in ldbm_back_search () from
> /usr/lib64/dirsrv/plugins/libback-ldbm.so
> No symbol table info available.
> #24 0x00007f1b4b2321d6 in op_shared_search () from
> /usr/lib64/dirsrv/libslapd.so.0
> No symbol table info available.
> #25 0x00007f1b4b72333e in do_search ()
> No symbol table info available.
> #26 0x00007f1b4b7113c3 in connection_threadmain ()
> No symbol table info available.
> #27 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so
> No symbol table info available.
> #28 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0
> No symbol table info available.
> #29 0x00007f1b48aef28d in clone () from /lib64/libc.so.6
> No symbol table info available.
>
> Thread 29 (Thread 0x7f1b0a7f4700 (LWP 105189)):
> #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 () from
> /lib64/libpthread.so.0
> No symbol table info available.
> #1 0x00007f1b4941b050 in PR_WaitCondVar () from /lib64/libnspr4.so
> No symbol table info available.
> #2 0x00007f1b4b70efee in connection_wait_for_new_work ()
> No symbol table info available.
> #3 0x00007f1b4b710719 in connection_threadmain ()
> No symbol table info available.
> #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so
> No symbol table info available.
> #5 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0
> No symbol table info available.
> #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6
> No symbol table info available.
>
> Thread 28 (Thread 0x7f1b0aff5700 (LWP 105188)):
> #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 () from
> /lib64/libpthread.so.0
> No symbol table info available.
> #1 0x00007f1b4941b050 in PR_WaitCondVar () from /lib64/libnspr4.so
> No symbol table info available.
> #2 0x00007f1b4b70efee in connection_wait_for_new_work ()
> No symbol table info available.
> #3 0x00007f1b4b710719 in connection_threadmain ()
> No symbol table info available.
> #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so
> No symbol table info available.
> #5 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0
> No symbol table info available.
> #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6
> No symbol table info available.
>
> Thread 27 (Thread 0x7f1b0b7f6700 (LWP 105187)):
> #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 () from
> /lib64/libpthread.so.0
> No symbol table info available.
> #1 0x00007f1b4941b050 in PR_WaitCondVar () from /lib64/libnspr4.so
> No symbol table info available.
> #2 0x00007f1b4b70efee in connection_wait_for_new_work ()
> No symbol table info available.
> #3 0x00007f1b4b710719 in connection_threadmain ()
> No symbol table info available.
> #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so
> No symbol table info available.
> #5 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0
> No symbol table info available.
> #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6
> No symbol table info available.
>
> Thread 26 (Thread 0x7f1b0bff7700 (LWP 105186)):
> #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 () from
> /lib64/libpthread.so.0
> No symbol table info available.
> #1 0x00007f1b433212f3 in __db_hybrid_mutex_suspend () from /lib64/
> libdb-5.3.so
> No symbol table info available.
> #2 0x00007f1b43320640 in __db_tas_mutex_lock () from /lib64/libdb-5.3.so
> No symbol table info available.
> #3 0x00007f1b433cad3a in __lock_get_internal () from /lib64/libdb-5.3.so
> No symbol table info available.
> #4 0x00007f1b433cb820 in __lock_get () from /lib64/libdb-5.3.so
> No symbol table info available.
> #5 0x00007f1b433f7142 in __db_lget () from /lib64/libdb-5.3.so
> No symbol table info available.
> #6 0x00007f1b4333e605 in __bam_search () from /lib64/libdb-5.3.so
> No symbol table info available.
> #7 0x00007f1b43329256 in __bamc_search () from /lib64/libdb-5.3.so
> No symbol table info available.
> #8 0x00007f1b4332ad0f in __bamc_get () from /lib64/libdb-5.3.so
> No symbol table info available.
> #9 0x00007f1b433e3ca6 in __dbc_iget () from /lib64/libdb-5.3.so
> No symbol table info available.
> #10 0x00007f1b433f2b02 in __dbc_get_pp () from /lib64/libdb-5.3.so
> No symbol table info available.
> #11 0x00007f1b3e8ce680 in idl_new_fetch () from /usr/lib64/dirsrv/plugins/
> libback-ldbm.so
> No symbol table info available.
> #12 0x00007f1b3e8dcb26 in index_read_ext_allids () from
> /usr/lib64/dirsrv/plugins/libback-ldbm.so
> No symbol table info available.
> #13 0x00007f1b3e8c7284 in keys2idl () from /usr/lib64/dirsrv/plugins/
> libback-ldbm.so
> No symbol table info available.
> #14 0x00007f1b3e8c79e3 in ava_candidates.isra.0 () from
> /usr/lib64/dirsrv/plugins/libback-ldbm.so
> No symbol table info available.
> #15 0x00007f1b3e8c7fd2 in filter_candidates_ext () from
> /usr/lib64/dirsrv/plugins/libback-ldbm.so
> No symbol table info available.
> #16 0x00007f1b3e8c9096 in list_candidates () from
> /usr/lib64/dirsrv/plugins/libback-ldbm.so
> No symbol table info available.
> #17 0x00007f1b3e8c7f40 in filter_candidates_ext () from
> /usr/lib64/dirsrv/plugins/libback-ldbm.so
> No symbol table info available.
> #18 0x00007f1b3e90380c in subtree_candidates () from
> /usr/lib64/dirsrv/plugins/libback-ldbm.so
> No symbol table info available.
> #19 0x00007f1b3e904e7f in ldbm_back_search () from
> /usr/lib64/dirsrv/plugins/libback-ldbm.so
> No symbol table info available.
> #20 0x00007f1b4b2321d6 in op_shared_search () from
> /usr/lib64/dirsrv/libslapd.so.0
> No symbol table info available.
> #21 0x00007f1b4b24294e in search_internal_callback_pb () from
> /usr/lib64/dirsrv/libslapd.so.0
> No symbol table info available.
> #22 0x00007f1b4b242be8 in search_internal_pb () from
> /usr/lib64/dirsrv/libslapd.so.0
> No symbol table info available.
> #23 0x00007f1b4b72149b in ids_sasl_canon_user ()
> No symbol table info available.
> #24 0x00007f1b491e193b in _sasl_canon_user () from /lib64/libsasl2.so.3
> No symbol table info available.
> #25 0x00007f1b491e1c4c in _sasl_canon_user_lookup () from
> /lib64/libsasl2.so.3
> No symbol table info available.
> #26 0x00007f1b430e9fb9 in gssapi_server_mech_step () from
> /usr/lib64/sasl2/libgssapiv2.so
> No symbol table info available.
> #27 0x00007f1b491edb9b in sasl_server_step () from /lib64/libsasl2.so.3
> No symbol table info available.
> #28 0x00007f1b4b72283b in ids_sasl_check_bind ()
> No symbol table info available.
> #29 0x00007f1b4b70a61a in do_bind ()
> No symbol table info available.
> #30 0x00007f1b4b7113fd in connection_threadmain ()
> No symbol table info available.
> #31 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so
> No symbol table info available.
> #32 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0
> No symbol table info available.
> #33 0x00007f1b48aef28d in clone () from /lib64/libc.so.6
> No symbol table info available.
>
> Thread 25 (Thread 0x7f1b0c7f8700 (LWP 105185)):
> #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 () from
> /lib64/libpthread.so.0
> No symbol table info available.
> #1 0x00007f1b4941b050 in PR_WaitCondVar () from /lib64/libnspr4.so
> No symbol table info available.
> #2 0x00007f1b4b70efee in connection_wait_for_new_work ()
> No symbol table info available.
> #3 0x00007f1b4b710719 in connection_threadmain ()
> No symbol table info available.
> #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so
> No symbol table info available.
> #5 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0
> No symbol table info available.
> #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6
> No symbol table info available.
>
> Thread 24 (Thread 0x7f1b0cff9700 (LWP 105184)):
> #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 () from
> /lib64/libpthread.so.0
> No symbol table info available.
> #1 0x00007f1b4941b050 in PR_WaitCondVar () from /lib64/libnspr4.so
> No symbol table info available.
> #2 0x00007f1b4b70efee in connection_wait_for_new_work ()
> No symbol table info available.
> #3 0x00007f1b4b710719 in connection_threadmain ()
> No symbol table info available.
> #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so
> No symbol table info available.
> #5 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0
> No symbol table info available.
> #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6
> No symbol table info available.
>
> Thread 23 (Thread 0x7f1b0d7fa700 (LWP 105183)):
> #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 () from
> /lib64/libpthread.so.0
> No symbol table info available.
> #1 0x00007f1b4941b050 in PR_WaitCondVar () from /lib64/libnspr4.so
> No symbol table info available.
> #2 0x00007f1b4b70efee in connection_wait_for_new_work ()
> No symbol table info available.
> #3 0x00007f1b4b710719 in connection_threadmain ()
> No symbol table info available.
> #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so
> No symbol table info available.
> #5 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0
> No symbol table info available.
> #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6
> No symbol table info available.
>
> Thread 22 (Thread 0x7f1b0dffb700 (LWP 105182)):
> #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 () from
> /lib64/libpthread.so.0
> No symbol table info available.
> #1 0x00007f1b4941b050 in PR_WaitCondVar () from /lib64/libnspr4.so
> No symbol table info available.
> #2 0x00007f1b4b70efee in connection_wait_for_new_work ()
> No symbol table info available.
> #3 0x00007f1b4b710719 in connection_threadmain ()
> No symbol table info available.
> #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so
> No symbol table info available.
> #5 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0
> No symbol table info available.
> #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6
> No symbol table info available.
>
> Thread 21 (Thread 0x7f1b0e7fc700 (LWP 105181)):
> #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 () from
> /lib64/libpthread.so.0
> No symbol table info available.
> #1 0x00007f1b4941b050 in PR_WaitCondVar () from /lib64/libnspr4.so
> No symbol table info available.
> #2 0x00007f1b4b70efee in connection_wait_for_new_work ()
> No symbol table info available.
> #3 0x00007f1b4b710719 in connection_threadmain ()
> No symbol table info available.
> #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so
> No symbol table info available.
> #5 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0
> No symbol table info available.
> #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6
> No symbol table info available.
>
> Thread 20 (Thread 0x7f1b0effd700 (LWP 105180)):
> #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 () from
> /lib64/libpthread.so.0
> No symbol table info available.
> #1 0x00007f1b433212f3 in __db_hybrid_mutex_suspend () from /lib64/
> libdb-5.3.so
> No symbol table info available.
> #2 0x00007f1b43320b37 in __db_tas_mutex_readlock () from /lib64/
> libdb-5.3.so
> No symbol table info available.
> #3 0x00007f1b4343ad35 in __memp_fget () from /lib64/libdb-5.3.so
> No symbol table info available.
> #4 0x00007f1b433e4333 in __dbc_iget () from /lib64/libdb-5.3.so
> No symbol table info available.
> #5 0x00007f1b433f2b02 in __dbc_get_pp () from /lib64/libdb-5.3.so
> No symbol table info available.
> #6 0x00007f1b3e8ce822 in idl_new_fetch () from /usr/lib64/dirsrv/plugins/
> libback-ldbm.so
> No symbol table info available.
> #7 0x00007f1b3e8dcb26 in index_read_ext_allids () from
> /usr/lib64/dirsrv/plugins/libback-ldbm.so
> No symbol table info available.
> #8 0x00007f1b3e8c7284 in keys2idl () from /usr/lib64/dirsrv/plugins/
> libback-ldbm.so
> No symbol table info available.
> #9 0x00007f1b3e8c79e3 in ava_candidates.isra.0 () from
> /usr/lib64/dirsrv/plugins/libback-ldbm.so
> No symbol table info available.
> #10 0x00007f1b3e8c7fd2 in filter_candidates_ext () from
> /usr/lib64/dirsrv/plugins/libback-ldbm.so
> No symbol table info available.
> #11 0x00007f1b3e8c9096 in list_candidates () from
> /usr/lib64/dirsrv/plugins/libback-ldbm.so
> No symbol table info available.
> #12 0x00007f1b3e8c7f40 in filter_candidates_ext () from
> /usr/lib64/dirsrv/plugins/libback-ldbm.so
> No symbol table info available.
> #13 0x00007f1b3e8c9096 in list_candidates () from
> /usr/lib64/dirsrv/plugins/libback-ldbm.so
> No symbol table info available.
> #14 0x00007f1b3e8c7f40 in filter_candidates_ext () from
> /usr/lib64/dirsrv/plugins/libback-ldbm.so
> No symbol table info available.
> #15 0x00007f1b3e8c9096 in list_candidates () from
> /usr/lib64/dirsrv/plugins/libback-ldbm.so
> No symbol table info available.
> #16 0x00007f1b3e8c7f40 in filter_candidates_ext () from
> /usr/lib64/dirsrv/plugins/libback-ldbm.so
> No symbol table info available.
> #17 0x00007f1b3e90380c in subtree_candidates () from
> /usr/lib64/dirsrv/plugins/libback-ldbm.so
> No symbol table info available.
> #18 0x00007f1b3e904e7f in ldbm_back_search () from
> /usr/lib64/dirsrv/plugins/libback-ldbm.so
> No symbol table info available.
> #19 0x00007f1b4b2321d6 in op_shared_search () from
> /usr/lib64/dirsrv/libslapd.so.0
> No symbol table info available.
> #20 0x00007f1b4b72333e in do_search ()
> No symbol table info available.
> #21 0x00007f1b4b7113c3 in connection_threadmain ()
> No symbol table info available.
> #22 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so
> No symbol table info available.
> #23 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0
> No symbol table info available.
> #24 0x00007f1b48aef28d in clone () from /lib64/libc.so.6
> No symbol table info available.
>
> Thread 19 (Thread 0x7f1b0f7fe700 (LWP 105179)):
> #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 () from
> /lib64/libpthread.so.0
> No symbol table info available.
> #1 0x00007f1b433212f3 in __db_hybrid_mutex_suspend () from /lib64/
> libdb-5.3.so
> No symbol table info available.
> #2 0x00007f1b43320640 in __db_tas_mutex_lock () from /lib64/libdb-5.3.so
> No symbol table info available.
> #3 0x00007f1b433cad3a in __lock_get_internal () from /lib64/libdb-5.3.so
> No symbol table info available.
> #4 0x00007f1b433cb820 in __lock_get () from /lib64/libdb-5.3.so
> No symbol table info available.
> #5 0x00007f1b433f7142 in __db_lget () from /lib64/libdb-5.3.so
> No symbol table info available.
> #6 0x00007f1b4333e605 in __bam_search () from /lib64/libdb-5.3.so
> No symbol table info available.
> #7 0x00007f1b43329256 in __bamc_search () from /lib64/libdb-5.3.so
> No symbol table info available.
> #8 0x00007f1b4332ad0f in __bamc_get () from /lib64/libdb-5.3.so
> No symbol table info available.
> #9 0x00007f1b433e3ca6 in __dbc_iget () from /lib64/libdb-5.3.so
> No symbol table info available.
> #10 0x00007f1b433f2b02 in __dbc_get_pp () from /lib64/libdb-5.3.so
> No symbol table info available.
> #11 0x00007f1b3e8ce680 in idl_new_fetch () from /usr/lib64/dirsrv/plugins/
> libback-ldbm.so
> No symbol table info available.
> #12 0x00007f1b3e8dcb26 in index_read_ext_allids () from
> /usr/lib64/dirsrv/plugins/libback-ldbm.so
> No symbol table info available.
> #13 0x00007f1b3e8c7284 in keys2idl () from /usr/lib64/dirsrv/plugins/
> libback-ldbm.so
> No symbol table info available.
> #14 0x00007f1b3e8c79e3 in ava_candidates.isra.0 () from
> /usr/lib64/dirsrv/plugins/libback-ldbm.so
> No symbol table info available.
> #15 0x00007f1b3e8c7fd2 in filter_candidates_ext () from
> /usr/lib64/dirsrv/plugins/libback-ldbm.so
> No symbol table info available.
> #16 0x00007f1b3e8c9096 in list_candidates () from
> /usr/lib64/dirsrv/plugins/libback-ldbm.so
> No symbol table info available.
> #17 0x00007f1b3e8c7f40 in filter_candidates_ext () from
> /usr/lib64/dirsrv/plugins/libback-ldbm.so
> No symbol table info available.
> #18 0x00007f1b3e8c9096 in list_candidates () from
> /usr/lib64/dirsrv/plugins/libback-ldbm.so
> No symbol table info available.
> #19 0x00007f1b3e8c7f40 in filter_candidates_ext () from
> /usr/lib64/dirsrv/plugins/libback-ldbm.so
> No symbol table info available.
> #20 0x00007f1b3e8c9096 in list_candidates () from
> /usr/lib64/dirsrv/plugins/libback-ldbm.so
> No symbol table info available.
> #21 0x00007f1b3e8c7f40 in filter_candidates_ext () from
> /usr/lib64/dirsrv/plugins/libback-ldbm.so
> No symbol table info available.
> #22 0x00007f1b3e90380c in subtree_candidates () from
> /usr/lib64/dirsrv/plugins/libback-ldbm.so
> No symbol table info available.
> #23 0x00007f1b3e904e7f in ldbm_back_search () from
> /usr/lib64/dirsrv/plugins/libback-ldbm.so
> No symbol table info available.
> #24 0x00007f1b4b2321d6 in op_shared_search () from
> /usr/lib64/dirsrv/libslapd.so.0
> No symbol table info available.
> #25 0x00007f1b4b72333e in do_search ()
> No symbol table info available.
> #26 0x00007f1b4b7113c3 in connection_threadmain ()
> No symbol table info available.
> #27 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so
> No symbol table info available.
> #28 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0
> No symbol table info available.
> #29 0x00007f1b48aef28d in clone () from /lib64/libc.so.6
> No symbol table info available.
>
> Thread 18 (Thread 0x7f1b0ffff700 (LWP 105178)):
> #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 () from
> /lib64/libpthread.so.0
> No symbol table info available.
> #1 0x00007f1b4941b050 in PR_WaitCondVar () from /lib64/libnspr4.so
> No symbol table info available.
> #2 0x00007f1b4b70efee in connection_wait_for_new_work ()
> No symbol table info available.
> #3 0x00007f1b4b710719 in connection_threadmain ()
> No symbol table info available.
> #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so
> No symbol table info available.
> #5 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0
> No symbol table info available.
> #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6
> No symbol table info available.
>
> Thread 17 (Thread 0x7f1b14d28700 (LWP 105177)):
> #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 () from
> /lib64/libpthread.so.0
> No symbol table info available.
> #1 0x00007f1b4941b050 in PR_WaitCondVar () from /lib64/libnspr4.so
> No symbol table info available.
> #2 0x00007f1b4b70efee in connection_wait_for_new_work ()
> No symbol table info available.
> #3 0x00007f1b4b710719 in connection_threadmain ()
> No symbol table info available.
> #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so
> No symbol table info available.
> #5 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0
> No symbol table info available.
> #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6
> No symbol table info available.
>
> Thread 16 (Thread 0x7f1b15529700 (LWP 105176)):
> #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 () from
> /lib64/libpthread.so.0
> No symbol table info available.
> #1 0x00007f1b4941b050 in PR_WaitCondVar () from /lib64/libnspr4.so
> No symbol table info available.
> #2 0x00007f1b4b70efee in connection_wait_for_new_work ()
> No symbol table info available.
> #3 0x00007f1b4b710719 in connection_threadmain ()
> No symbol table info available.
> #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so
> No symbol table info available.
> #5 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0
> No symbol table info available.
> #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6
> No symbol table info available.
>
> Thread 15 (Thread 0x7f1b15d2a700 (LWP 105175)):
> #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 () from
> /lib64/libpthread.so.0
> No symbol table info available.
> #1 0x00007f1b4941b050 in PR_WaitCondVar () from /lib64/libnspr4.so
> No symbol table info available.
> #2 0x00007f1b4b70efee in connection_wait_for_new_work ()
> No symbol table info available.
> #3 0x00007f1b4b710719 in connection_threadmain ()
> No symbol table info available.
> #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so
> No symbol table info available.
> #5 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0
> No symbol table info available.
> #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6
> No symbol table info available.
>
> Thread 14 (Thread 0x7f1b1652b700 (LWP 105174)):
> #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 () from
> /lib64/libpthread.so.0
> No symbol table info available.
> #1 0x00007f1b4941b050 in PR_WaitCondVar () from /lib64/libnspr4.so
> No symbol table info available.
> #2 0x00007f1b4b70efee in connection_wait_for_new_work ()
> No symbol table info available.
> #3 0x00007f1b4b710719 in connection_threadmain ()
> No symbol table info available.
> #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so
> No symbol table info available.
> #5 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0
> No symbol table info available.
> #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6
> No symbol table info available.
>
> Thread 13 (Thread 0x7f1b16d2c700 (LWP 105173)):
> #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 () from
> /lib64/libpthread.so.0
> No symbol table info available.
> #1 0x00007f1b433212f3 in __db_hybrid_mutex_suspend () from /lib64/
> libdb-5.3.so
> No symbol table info available.
> #2 0x00007f1b43320640 in __db_tas_mutex_lock () from /lib64/libdb-5.3.so
> No symbol table info available.
> #3 0x00007f1b433cad3a in __lock_get_internal () from /lib64/libdb-5.3.so
> No symbol table info available.
> #4 0x00007f1b433cb820 in __lock_get () from /lib64/libdb-5.3.so
> No symbol table info available.
> #5 0x00007f1b433f7142 in __db_lget () from /lib64/libdb-5.3.so
> No symbol table info available.
> #6 0x00007f1b4333e605 in __bam_search () from /lib64/libdb-5.3.so
> No symbol table info available.
> #7 0x00007f1b43329256 in __bamc_search () from /lib64/libdb-5.3.so
> No symbol table info available.
> #8 0x00007f1b4332ad0f in __bamc_get () from /lib64/libdb-5.3.so
> No symbol table info available.
> #9 0x00007f1b433e3ca6 in __dbc_iget () from /lib64/libdb-5.3.so
> No symbol table info available.
> #10 0x00007f1b433f2b02 in __dbc_get_pp () from /lib64/libdb-5.3.so
> No symbol table info available.
> #11 0x00007f1b3e8ce680 in idl_new_fetch () from /usr/lib64/dirsrv/plugins/
> libback-ldbm.so
> No symbol table info available.
> #12 0x00007f1b3e8dcb26 in index_read_ext_allids () from
> /usr/lib64/dirsrv/plugins/libback-ldbm.so
> No symbol table info available.
> #13 0x00007f1b3e8c7284 in keys2idl () from /usr/lib64/dirsrv/plugins/
> libback-ldbm.so
> No symbol table info available.
> #14 0x00007f1b3e8c79e3 in ava_candidates.isra.0 () from
> /usr/lib64/dirsrv/plugins/libback-ldbm.so
> No symbol table info available.
> #15 0x00007f1b3e8c7fd2 in filter_candidates_ext () from
> /usr/lib64/dirsrv/plugins/libback-ldbm.so
> No symbol table info available.
> #16 0x00007f1b3e8c9096 in list_candidates () from
> /usr/lib64/dirsrv/plugins/libback-ldbm.so
> No symbol table info available.
> #17 0x00007f1b3e8c7f40 in filter_candidates_ext () from
> /usr/lib64/dirsrv/plugins/libback-ldbm.so
> No symbol table info available.
> #18 0x00007f1b3e8c9096 in list_candidates () from
> /usr/lib64/dirsrv/plugins/libback-ldbm.so
> No symbol table info available.
> #19 0x00007f1b3e8c7f40 in filter_candidates_ext () from
> /usr/lib64/dirsrv/plugins/libback-ldbm.so
> No symbol table info available.
> #20 0x00007f1b3e8c9096 in list_candidates () from
> /usr/lib64/dirsrv/plugins/libback-ldbm.so
> No symbol table info available.
> #21 0x00007f1b3e8c7f40 in filter_candidates_ext () from
> /usr/lib64/dirsrv/plugins/libback-ldbm.so
> No symbol table info available.
> #22 0x00007f1b3e90380c in subtree_candidates () from
> /usr/lib64/dirsrv/plugins/libback-ldbm.so
> No symbol table info available.
> #23 0x00007f1b3e904e7f in ldbm_back_search () from
> /usr/lib64/dirsrv/plugins/libback-ldbm.so
> No symbol table info available.
> #24 0x00007f1b4b2321d6 in op_shared_search () from
> /usr/lib64/dirsrv/libslapd.so.0
> No symbol table info available.
> #25 0x00007f1b4b72333e in do_search ()
> No symbol table info available.
> #26 0x00007f1b4b7113c3 in connection_threadmain ()
> No symbol table info available.
> #27 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so
> No symbol table info available.
> #28 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0
> No symbol table info available.
> #29 0x00007f1b48aef28d in clone () from /lib64/libc.so.6
> No symbol table info available.
>
> Thread 12 (Thread 0x7f1b1752d700 (LWP 105172)):
> #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 () from
> /lib64/libpthread.so.0
> No symbol table info available.
> #1 0x00007f1b4941b050 in PR_WaitCondVar () from /lib64/libnspr4.so
> No symbol table info available.
> #2 0x00007f1b4b70efee in connection_wait_for_new_work ()
> No symbol table info available.
> #3 0x00007f1b4b710719 in connection_threadmain ()
> No symbol table info available.
> #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so
> No symbol table info available.
> #5 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0
> No symbol table info available.
> #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6
> No symbol table info available.
>
> Thread 11 (Thread 0x7f1b17d2e700 (LWP 105171)):
> #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 () from
> /lib64/libpthread.so.0
> No symbol table info available.
> #1 0x00007f1b4941b050 in PR_WaitCondVar () from /lib64/libnspr4.so
> No symbol table info available.
> #2 0x00007f1b4b70efee in connection_wait_for_new_work ()
> No symbol table info available.
> #3 0x00007f1b4b710719 in connection_threadmain ()
> No symbol table info available.
> #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so
> No symbol table info available.
> #5 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0
> No symbol table info available.
> #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6
> No symbol table info available.
>
> Thread 10 (Thread 0x7f1b2509d700 (LWP 105168)):
> #0 0x00007f1b48dc5a82 in pthread_cond_timedwait@@GLIBC_2.3.2 () from
> /lib64/libpthread.so.0
> No symbol table info available.
> #1 0x00007f1b4941ab07 in pt_TimedWait () from /lib64/libnspr4.so
> No symbol table info available.
> #2 0x00007f1b4941afce in PR_WaitCondVar () from /lib64/libnspr4.so
> No symbol table info available.
> #3 0x00007f1b4b2061e8 in eq_loop () from /usr/lib64/dirsrv/libslapd.so.0
> No symbol table info available.
> #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so
> No symbol table info available.
> #5 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0
> No symbol table info available.
> #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6
> No symbol table info available.
>
> Thread 9 (Thread 0x7f1b2589e700 (LWP 105167)):
> #0 0x00007f1b48dc5a82 in pthread_cond_timedwait@@GLIBC_2.3.2 () from
> /lib64/libpthread.so.0
> No symbol table info available.
> #1 0x00007f1b4941ab07 in pt_TimedWait () from /lib64/libnspr4.so
> No symbol table info available.
> #2 0x00007f1b4941afce in PR_WaitCondVar () from /lib64/libnspr4.so
> No symbol table info available.
> #3 0x00007f1b4b719d03 in housecleaning ()
> No symbol table info available.
> #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so
> No symbol table info available.
> #5 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0
> No symbol table info available.
> #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6
> No symbol table info available.
>
> Thread 8 (Thread 0x7f1b2609f700 (LWP 105166)):
> #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 () from
> /lib64/libpthread.so.0
> No symbol table info available.
> #1 0x00007f1b4941b050 in PR_WaitCondVar () from /lib64/libnspr4.so
> No symbol table info available.
> #2 0x00007f1b4b260198 in slapi_wait_condvar () from
> /usr/lib64/dirsrv/libslapd.so.0
> No symbol table info available.
> #3 0x00007f1b3cf5deed in roles_cache_wait_on_change () from
> /usr/lib64/dirsrv/plugins/libroles-plugin.so
> No symbol table info available.
> #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so
> No symbol table info available.
> #5 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0
> No symbol table info available.
> #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6
> No symbol table info available.
>
> Thread 7 (Thread 0x7f1b268a0700 (LWP 105165)):
> #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 () from
> /lib64/libpthread.so.0
> No symbol table info available.
> #1 0x00007f1b4941b050 in PR_WaitCondVar () from /lib64/libnspr4.so
> No symbol table info available.
> #2 0x00007f1b4b260198 in slapi_wait_condvar () from
> /usr/lib64/dirsrv/libslapd.so.0
> No symbol table info available.
> #3 0x00007f1b3cf5deed in roles_cache_wait_on_change () from
> /usr/lib64/dirsrv/plugins/libroles-plugin.so
> No symbol table info available.
> #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so
> No symbol table info available.
> #5 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0
> No symbol table info available.
> #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6
> No symbol table info available.
>
> Thread 6 (Thread 0x7f1b270a1700 (LWP 105164)):
> #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 () from
> /lib64/libpthread.so.0
> No symbol table info available.
> #1 0x00007f1b4941b050 in PR_WaitCondVar () from /lib64/libnspr4.so
> No symbol table info available.
> #2 0x00007f1b4b260198 in slapi_wait_condvar () from
> /usr/lib64/dirsrv/libslapd.so.0
> No symbol table info available.
> #3 0x00007f1b3cf5deed in roles_cache_wait_on_change () from
> /usr/lib64/dirsrv/plugins/libroles-plugin.so
> No symbol table info available.
> #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so
> No symbol table info available.
> #5 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0
> No symbol table info available.
> #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6
> No symbol table info available.
>
> Thread 5 (Thread 0x7f1b278a2700 (LWP 105163)):
> #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 () from
> /lib64/libpthread.so.0
> No symbol table info available.
> #1 0x00007f1b4941b050 in PR_WaitCondVar () from /lib64/libnspr4.so
> No symbol table info available.
> #2 0x00007f1b4b260198 in slapi_wait_condvar () from
> /usr/lib64/dirsrv/libslapd.so.0
> No symbol table info available.
> #3 0x00007f1b415f262e in cos_cache_wait_on_change () from
> /usr/lib64/dirsrv/plugins/libcos-plugin.so
> No symbol table info available.
> #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so
> No symbol table info available.
> #5 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0
> No symbol table info available.
> #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6
> No symbol table info available.
>
> Thread 4 (Thread 0x7f1b282ac700 (LWP 105162)):
> #0 0x00007f1b48ae69b3 in select () from /lib64/libc.so.6
> No symbol table info available.
> #1 0x00007f1b4b2710e9 in DS_Sleep () from /usr/lib64/dirsrv/libslapd.so.0
> No symbol table info available.
> #2 0x00007f1b3e8ba707 in perf_threadmain () from
> /usr/lib64/dirsrv/plugins/libback-ldbm.so
> No symbol table info available.
> #3 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so
> No symbol table info available.
> #4 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0
> No symbol table info available.
> #5 0x00007f1b48aef28d in clone () from /lib64/libc.so.6
> No symbol table info available.
>
> Thread 3 (Thread 0x7f1b28aad700 (LWP 105161)):
> #0 0x00007f1b48ae69b3 in select () from /lib64/libc.so.6
> No symbol table info available.
> #1 0x00007f1b4b2710e9 in DS_Sleep () from /usr/lib64/dirsrv/libslapd.so.0
> No symbol table info available.
> #2 0x00007f1b3e8bfb7f in trickle_threadmain () from
> /usr/lib64/dirsrv/plugins/libback-ldbm.so
> No symbol table info available.
> #3 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so
> No symbol table info available.
> #4 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0
> No symbol table info available.
> #5 0x00007f1b48aef28d in clone () from /lib64/libc.so.6
> No symbol table info available.
>
> Thread 2 (Thread 0x7f1b292ae700 (LWP 105160)):
> #0 0x00007f1b48ae69b3 in select () from /lib64/libc.so.6
> No symbol table info available.
> #1 0x00007f1b4344dbad in __os_yield () from /lib64/libdb-5.3.so
> No symbol table info available.
> #2 0x00007f1b434492b3 in __memp_sync_int () from /lib64/libdb-5.3.so
> No symbol table info available.
> #3 0x00007f1b43459752 in __txn_checkpoint () from /lib64/libdb-5.3.so
> No symbol table info available.
> #4 0x00007f1b43459b74 in __txn_checkpoint_pp () from /lib64/libdb-5.3.so
> No symbol table info available.
> #5 0x00007f1b3e8c3a87 in checkpoint_threadmain () from
> /usr/lib64/dirsrv/plugins/libback-ldbm.so
> No symbol table info available.
> #6 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so
> No symbol table info available.
> #7 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0
> No symbol table info available.
> #8 0x00007f1b48aef28d in clone () from /lib64/libc.so.6
> No symbol table info available.
>
> Thread 1 (Thread 0x7f1b29aaf700 (LWP 105159)):
> #0 0x00007f1b48ae69b3 in select () from /lib64/libc.so.6
> No symbol table info available.
> #1 0x00007f1b4b2710e9 in DS_Sleep () from /usr/lib64/dirsrv/libslapd.so.0
> No symbol table info available.
> #2 0x00007f1b3e8bf907 in deadlock_threadmain () from
> /usr/lib64/dirsrv/plugins/libback-ldbm.so
> No symbol table info available.
> #3 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so
> No symbol table info available.
> #4 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0
> No symbol table info available.
> #5 0x00007f1b48aef28d in clone () from /lib64/libc.so.6
> No symbol table info available.
>
>
>
> Thanks,
>
> Rakesh
>
>
> On Mon, Sep 5, 2016 at 4:48 PM, thierry bordaz <tbordaz at redhat.com> wrote:
>
>>
>>
>> On 09/05/2016 12:05 PM, Rakesh Rajasekharan wrote:
>>
>> Hi Thierry,
>>
>>
>> I was getting the hang issue while running ipa-client-install
>> simultaneously on few clients..
>> However, today, I am not able to replicate that.
>>
>> I could not get a gdb . But i will try getting that the next time I face
>> this issue.
>>
>> The CPU does not stay high.. it just momentarily touches a high value and
>> then drops down to around 2-7%
>>
>> One question I have is , is it ok to set it nsslapd-threadnumber to a
>> very high value .
>> I have around 4000 clients and with nsslapd-maxthreadsperconn set to
>> 5..So, can I set nsslapd-threadnumber to around 25000.
>>
>>
>> Hello,
>>
>> I know some users running in production with several hunderds of threads
>> (>600) and this without problem.
>>
>> I do not recall having suggested to increase that number and for what
>> reason.
>> Usually 30 workers is a good enough value. It can create bootleneck if
>> for some reason each operation is very long to satisfy and exhaust the
>> number of workers. You can monitor the work queue:
>>
>> ldapsearch -D "cn=directory manager" -w xxx -LLL -b "cn=monitor" -s base
>> opsinitiated opscompleted
>>
>>
>> If opscompleted-opsinitiated remains close to threadnumber, then yes it
>> would be valuable to increase it.
>>
>> The computation #client * #async_op_per_client sound an overkill. Even if
>> all clients send at the exact same time all their requests, it is very
>> likely that some common resource (db page, log, allocator...) will
>> serialize them. If you monitor a need to increase the work, you would for
>> example set it to 50, then monitor, then set it to 100, then monitor...
>> until you find a good enough value.
>> Note the increasing the #thread, increases the memory footprint that will
>> reduce the efficiency of file system cache and can increase the response
>> time.
>>
>>
>> best regards
>> thierry
>>
>>
>> Thanks
>>
>> On Mon, Sep 5, 2016 at 1:03 PM, thierry bordaz <tbordaz at redhat.com>
>> wrote:
>>
>>>
>>> Hi Rakesh,
>>>
>>> Were you able to get a pstack or full stack with gdb (
>>> <http://www.port389.org/docs/389ds/FAQ/faq.html#debugging-crashes>
>>> http://www.port389.org/docs/389ds/FAQ/faq.html#debugging-crashes) when
>>> the server hangs ?
>>>
>>> If it happens with 500 threads as well as with 30, using 30 threads is a
>>> better choice to debug this issue.
>>> I will try to reproduce using 150 parallel 'ipa user-find p-testipa'
>>> commands
>>>
>>> Something I am unsure is if the CPU consumption stays high (you
>>> mentioned 340% CPU usage) as long as the hang happens or if after a suddent
>>> shot up to 340% (that marks the beginning of the hang) it drops and stay
>>> hanging ?
>>>
>>> thanks
>>> thierry
>>>
>>> On 09/04/2016 08:40 PM, Rakesh Rajasekharan wrote:
>>>
>>> starce on the slapd process actually had this in the output..
>>> FUTEX_WAIT_PRIVATE
>>>
>>> and checking for the number of threads slapd had.. there were 5015
>>> threads
>>>
>>> ps -efL|grep slapd|wc -l
>>> 5015
>>>
>>> strace on most of the threads gave this output
>>>
>>> strace -p 67411
>>> Process 67411 attached
>>> futex(0x7f3f0226b41c, FUTEX_WAIT_PRIVATE, 1, NULL) = -1 EAGAIN (Resource
>>> temporarily unavailable)
>>> futex(0x7f3f0226b41c, FUTEX_WAIT_PRIVATE, 2, NULL^CProcess 67411 detached
>>>
>>>
>>>
>>>
>>>
>>> On Sun, Sep 4, 2016 at 5:34 PM, Rakesh Rajasekharan <
>>> <rakesh.rajasekharan at gmail.com>rakesh.rajasekharan at gmail.com> wrote:
>>>
>>>> I have again got the issue of IPA hanging.. The issue came up when i
>>>> tried to run ipa-client-isntall on 142 clients simultaneously
>>>>
>>>>
>>>> None of the IPA commands are responding, and I see this error
>>>>
>>>> ipa user-find p-testipa
>>>> ipa: ERROR: Insufficient access: SASL(-1): generic failure: GSSAPI
>>>> Error: Unspecified GSS failure. Minor code may provide more information
>>>> (KDC returned error string: PROCESS_TGS)
>>>>
>>>> KRB5_TRACE=/dev/stdout kinit admin
>>>> [41178] 1472984115.233214: Getting initial credentials for
>>>> <admin at XYZ.COM>admin at XYZ.COM
>>>> [41178] 1472984115.235257: Sending request (167 bytes) to XYZ.COM
>>>> [41178] 1472984115.235419: Initiating TCP connection to stream
>>>> 10.1.3.36:88
>>>> [41178] 1472984115.235685: Sending TCP request to stream 10.1.3.36:88
>>>> [41178] 1472984120.238914: Received answer (174 bytes) from stream
>>>> 10.1.3.36:88
>>>> [41178] 1472984120.238925: Terminating TCP connection to stream
>>>> 10.1.3.36:88
>>>> [41178] 1472984120.238993: Response was from master KDC
>>>> [41
>>>>
>>>>
>>>> Running an ldapsearch to see the db.. does not give any results and
>>>> just hangs there
>>>>
>>>> ldapsearch -x -D 'cn=Directory Manager' -W -s one -b
>>>> 'cn=kerberos,dc=xyz,dc=com'
>>>> Enter LDAP Password:
>>>>
>>>> even an ldapsearch -x does not respond
>>>> At this point, am sure that slapd is the one causing issues
>>>>
>>>> Running an strace against the hung slapd itself seems to get stuck does
>>>> not proceed after saying "attaching to process"
>>>>
>>>> From some others posts I read Thierry suggesting to increase the
>>>> nsslapd-threadnumber value
>>>>
>>>> It was set to 30, I think that might be too low.
>>>>
>>>> I have raised it to 500
>>>>
>>>> Now after restarting the service .. ldapsearch starts responding.
>>>> But running the test to add a sudden high number of clients again left
>>>> ns-slapd to hung state
>>>>
>>>> When i attempted adding the clients.. the ns-slapd cpu usage shot up to
>>>> 340% and after that ns-slapd stopped responding
>>>>
>>>> So now, atleast I know what might be causing the issue and I can now
>>>> easily reproduce it.
>>>>
>>>> Is there a way I can make ns-slapd handle a sudden bump in incoming
>>>> request for ipa-client-install
>>>>
>>>> Thanks
>>>> Rakesh
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> On Mon, Aug 29, 2016 at 11:18 PM, Rich Megginson <
>>>> <rmeggins at redhat.com>rmeggins at redhat.com> wrote:
>>>>
>>>>> On 08/29/2016 10:53 AM, Rakesh Rajasekharan wrote:
>>>>>
>>>>> Hi Thierry,
>>>>>
>>>>> My machine has 30GB RAM ..and 389-ds version is 1.3.4
>>>>>
>>>>> ldapsearch shows the values for nsslapd-cachememsize updated to 200MB.
>>>>>
>>>>> ldapsearch -LLL -o ldif-wrap=no -D "cn=directory manager" -w
>>>>> 'mypassword' -b 'cn=userRoot,cn=ldbm database,cn=plugins,cn=config'|grep
>>>>> nsslapd-cachememsize
>>>>> nsslapd-cachememsize: 209715200
>>>>>
>>>>>
>>>>> So, it seems to have updated though seeing that warning(WARNING:
>>>>> ipaca: entry cache size 10485760B is less than db size 11599872B) in the
>>>>> log confuses me a bit.
>>>>>
>>>>> Thers one more entry that I found from the ldapsearch to be bit low
>>>>>
>>>>> nsslapd-dncachememsize: 10485760
>>>>> maxdncachesize: 10485760
>>>>>
>>>>> Should I update these as well to a higher value
>>>>>
>>>>> At the time when the issue happened, the memory usage as well as the
>>>>> overall load of the system was very low .
>>>>> I will try reproducing the issue atleast in my QA env..probably by
>>>>> trying to mock simultaneous parallel logins to a large number of hosts
>>>>>
>>>>>
>>>>> To monitor your cache sizes, please use the dbmon.sh tool provided
>>>>> with your distro. If that is not available with your particular distro,
>>>>> see https://github.com/richm/scripts/wiki/dbmon.sh
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> thanks
>>>>> Rakesh
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> On Mon, Aug 29, 2016 at 8:16 PM, thierry bordaz < <tbordaz at redhat.com>
>>>>> tbordaz at redhat.com> wrote:
>>>>>
>>>>>> Hi Rakesh,
>>>>>>
>>>>>> Those tuning may depend on the memory available on your machine.
>>>>>> nsslapd-cachememsize allows the entry cache to consume up to 200Mb
>>>>>> but its memory footprint is known to go above.
>>>>>> 200Mb both looks pretty good to me. How large is your machine ? What
>>>>>> is your version of 389-ds ?
>>>>>>
>>>>>> Those warnings do not change your settings. It just raise that entry
>>>>>> cache of 'ipaca' and 'retrocl' are small but it is fine. The size of the
>>>>>> entry cache is important mostly in userRoot.
>>>>>> You may double check the actual values, after restart, with
>>>>>> ldapsearch on 'cn=userRoot,cn=ldbm database,cn=plugins,cn=config' and
>>>>>> 'cn=config,cn=ldbm database,cn=plugins,cn=config'.
>>>>>>
>>>>>> A step is to know what will be response time of DS to know if it is
>>>>>> responsible of the hang or not.
>>>>>> The logs and possibly pstack during those intermittent hangs will
>>>>>> help to determine that.
>>>>>>
>>>>>> regards
>>>>>> thierry
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> On 08/29/2016 04:25 PM, Rakesh Rajasekharan wrote:
>>>>>>
>>>>>> I tried increasing the nsslapd-dbcachesize and nsslapd-cachememsize
>>>>>> in my QA envs to 200MB.
>>>>>>
>>>>>> However, in my log files, I still see this message
>>>>>> [29/Aug/2016:04:34:37 +0000] - WARNING: ipaca: entry cache size
>>>>>> 10485760B is less than db size 11599872B; We recommend to increase the
>>>>>> entry cache size nsslapd-cachememsize.
>>>>>> [29/Aug/2016:04:34:37 +0000] - WARNING: changelog: entry cache size
>>>>>> 2097152B is less than db size 441647104B; We recommend to increase the
>>>>>> entry cache size nsslapd-cachememsize.
>>>>>>
>>>>>> these are my ldif files that i used to modify the values
>>>>>> modify entry cache size
>>>>>> cat modify-cache-mem-size.ldif
>>>>>> dn: cn=userRoot,cn=ldbm database,cn=plugins,cn=config
>>>>>> changetype: modify
>>>>>> replace: nsslapd-cachememsize
>>>>>> nsslapd-cachememsize: 209715200
>>>>>>
>>>>>> modify db cache size
>>>>>> cat modfy-db-cache-size.ldif
>>>>>> dn: cn=config,cn=ldbm database,cn=plugins,cn=config
>>>>>> changetype: modify
>>>>>> replace: nsslapd-dbcachesize
>>>>>> nsslapd-dbcachesize: 209715200
>>>>>>
>>>>>> After modifying , i restarted IPA services
>>>>>>
>>>>>> Is there anything else that I need to take care of as the logs
>>>>>> suggest its still not getting the updated values
>>>>>>
>>>>>> Thanks
>>>>>> Rakesh
>>>>>>
>>>>>> On Mon, Aug 29, 2016 at 6:07 PM, Rakesh Rajasekharan <
>>>>>> <rakesh.rajasekharan at gmail.com>rakesh.rajasekharan at gmail.com> wrote:
>>>>>>
>>>>>>> Hi Thierry,
>>>>>>>
>>>>>>> Coz of the issues we had to revert back to earlier running openldap
>>>>>>> in production.
>>>>>>>
>>>>>>> I have now done a few TCP related changes in sysctl.conf and have
>>>>>>> also increased the nsslapd-dbcachesize and nsslapd-cachememsize to 200MB
>>>>>>>
>>>>>>> I will again start migrating hosts back to IPA and see if I face the
>>>>>>> earlier issue.
>>>>>>>
>>>>>>> I will update back once I have something
>>>>>>>
>>>>>>>
>>>>>>> Thanks,
>>>>>>> Rakesh
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On Thu, Aug 25, 2016 at 2:17 PM, thierry bordaz <
>>>>>>> <tbordaz at redhat.com>tbordaz at redhat.com> wrote:
>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> On 08/25/2016 10:15 AM, Rakesh Rajasekharan wrote:
>>>>>>>>
>>>>>>>> All of the troubleshooting seems fine.
>>>>>>>>
>>>>>>>>
>>>>>>>> However, Running libconv.pl gives me this output
>>>>>>>>
>>>>>>>> ----- Recommendations -----
>>>>>>>>
>>>>>>>> 1. You have unindexed components, this can be caused from a
>>>>>>>> search on an unindexed attribute, or your returned results exceeded the
>>>>>>>> allidsthreshold. Unindexed components are not recommended. To refuse
>>>>>>>> unindexed searches, switch 'nsslapd-require-index' to 'on' under your
>>>>>>>> database entry (e.g. cn=UserRoot,cn=ldbm database,cn=plugins,cn=config)
>>>>>>>> .
>>>>>>>>
>>>>>>>> 2. You have a significant difference between binds and unbinds.
>>>>>>>> You may want to investigate this difference.
>>>>>>>>
>>>>>>>>
>>>>>>>> I feel, this could be a pointer to things going slow.. and IPA
>>>>>>>> hanging. I think i now have something that I can try and nail down this
>>>>>>>> issue.
>>>>>>>>
>>>>>>>> On a sidenote, I was earlier running openldap and migrated over to
>>>>>>>> Freeipa,
>>>>>>>>
>>>>>>>> Thanks
>>>>>>>> Rakesh
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> On Wed, Aug 24, 2016 at 12:38 PM, Petr Spacek <
>>>>>>>> <pspacek at redhat.com>pspacek at redhat.com> wrote:
>>>>>>>>
>>>>>>>>> On 23.8.2016 18:44, Rakesh Rajasekharan wrote:
>>>>>>>>> > I think thers something seriously wrong with my system
>>>>>>>>> >
>>>>>>>>> > not able to run any IPA commands
>>>>>>>>> >
>>>>>>>>> > klist
>>>>>>>>> > Ticket cache: KEYRING:persistent:0:0
>>>>>>>>> > Default principal: <admin at XYZ.COM>admin at XYZ.COM
>>>>>>>>> >
>>>>>>>>> > Valid starting Expires Service principal
>>>>>>>>> > 2016-08-23T16:26:36 2016-08-24T16:26:22 krbtgt/
>>>>>>>>> <XYZ.COM at XYZ.COM>XYZ.COM at XYZ.COM
>>>>>>>>> >
>>>>>>>>> >
>>>>>>>>> > [root at prod-ipa-master-1a :~] ipactl status
>>>>>>>>> > Directory Service: RUNNING
>>>>>>>>> > krb5kdc Service: RUNNING
>>>>>>>>> > kadmin Service: RUNNING
>>>>>>>>> > ipa_memcached Service: RUNNING
>>>>>>>>> > httpd Service: RUNNING
>>>>>>>>> > pki-tomcatd Service: RUNNING
>>>>>>>>> > ipa-otpd Service: RUNNING
>>>>>>>>> > ipa: INFO: The ipactl command was successful
>>>>>>>>> >
>>>>>>>>> >
>>>>>>>>> >
>>>>>>>>> > [root at prod-ipa-master :~] ipa user-find p-testuser
>>>>>>>>> > ipa: ERROR: Kerberos error: ('Unspecified GSS failure. Minor
>>>>>>>>> code may
>>>>>>>>> > provide more information', 851968)/("Cannot contact any KDC for
>>>>>>>>> realm '
>>>>>>>>> > XYZ.COM'", -1765328228)
>>>>>>>>>
>>>>>>>>
>>>>>>>> Hi Rakesh,
>>>>>>>>
>>>>>>>> Having a reproducible test case would you rerun the command above.
>>>>>>>> During its processing you may monitor DS process load (top). If it
>>>>>>>> is high, you may get some pstacks of it.
>>>>>>>> Also would you attach the part of DS access logs taken during the
>>>>>>>> command.
>>>>>>>>
>>>>>>>> regards
>>>>>>>> thierry
>>>>>>>>
>>>>>>>> >
>>>>>>>>>
>>>>>>>>> This is weird because the server seems to be up.
>>>>>>>>>
>>>>>>>>> Please follow
>>>>>>>>> <http://www.freeipa.org/page/Tr>http://www.freeipa.org/page/Tr
>>>>>>>>> oubleshooting#Authentication.2FKerberos
>>>>>>>>>
>>>>>>>>> Petr^2 Spacek
>>>>>>>>>
>>>>>>>>> >
>>>>>>>>> >
>>>>>>>>> > Thanks
>>>>>>>>> >
>>>>>>>>> > Rakesh
>>>>>>>>> >
>>>>>>>>> > On Tue, Aug 23, 2016 at 10:01 PM, Rakesh Rajasekharan <
>>>>>>>>> > <rakesh.rajasekharan at gmail.com>rakesh.rajasekharan at gmail.com>
>>>>>>>>> wrote:
>>>>>>>>> >
>>>>>>>>> >> i changed the loggin level to 4 . Modifying
>>>>>>>>> nsslapd-accesslog-level
>>>>>>>>> >>
>>>>>>>>> >> But, the hang is still there. though I dont see the sigfault now
>>>>>>>>> >>
>>>>>>>>> >>
>>>>>>>>> >>
>>>>>>>>> >>
>>>>>>>>> >> On Tue, Aug 23, 2016 at 9:02 PM, Rakesh Rajasekharan <
>>>>>>>>> >> <rakesh.rajasekharan at gmail.com>rakesh.rajasekharan at gmail.com>
>>>>>>>>> wrote:
>>>>>>>>> >>
>>>>>>>>> >>> My disk was getting filled too fast
>>>>>>>>> >>>
>>>>>>>>> >>> logs under /var/log/dirsrv was coming around 5 gb quickly
>>>>>>>>> filling up
>>>>>>>>> >>>
>>>>>>>>> >>> Is there a way to make the logging less verbose
>>>>>>>>> >>>
>>>>>>>>> >>>
>>>>>>>>> >>>
>>>>>>>>> >>> On Tue, Aug 23, 2016 at 6:41 PM, Petr Spacek <
>>>>>>>>> <pspacek at redhat.com>pspacek at redhat.com> wrote:
>>>>>>>>> >>>
>>>>>>>>> >>>> On 23.8.2016 15:07, Rakesh Rajasekharan wrote:
>>>>>>>>> >>>>> I was able to fix that may be temporarily... when i checked
>>>>>>>>> the
>>>>>>>>> >>>> network..
>>>>>>>>> >>>>> there was another process that was running and consuming a
>>>>>>>>> lot of
>>>>>>>>> >>>> network (
>>>>>>>>> >>>>> i have no idea who did that. I need to seriously start
>>>>>>>>> restricting
>>>>>>>>> >>>> people
>>>>>>>>> >>>>> access to this machine )
>>>>>>>>> >>>>>
>>>>>>>>> >>>>> after killing that perfomance improved drastically
>>>>>>>>> >>>>>
>>>>>>>>> >>>>> But now, suddenly I started experiencing the same hang.
>>>>>>>>> >>>>>
>>>>>>>>> >>>>> This time , I gert the following error when checked dmesg
>>>>>>>>> >>>>>
>>>>>>>>> >>>>> [ 301.236976] ns-slapd[3124]: segfault at 0 ip
>>>>>>>>> 00007f1de416951c sp
>>>>>>>>> >>>>> 00007f1dee1dba70 error 4 in libcos-plugin.so[7f1de4166000+
>>>>>>>>> b000]
>>>>>>>>> >>>>> [ 1116.248431] TCP: request_sock_TCP: Possible SYN flooding
>>>>>>>>> on port 88.
>>>>>>>>> >>>>> Sending cookies. Check SNMP counters.
>>>>>>>>> >>>>> [11831.397037] ns-slapd[22550]: segfault at 0 ip
>>>>>>>>> 00007f533d82251c sp
>>>>>>>>> >>>>> 00007f5347894a70 error 4 in libcos-plugin.so[7f533d81f000+
>>>>>>>>> b000]
>>>>>>>>> >>>>> [11832.727989] ns-slapd[22606]: segfault at 0 ip
>>>>>>>>> 00007f6231eb951c sp
>>>>>>>>> >>>>> 00007f623bf2ba70 error 4 in libcos-plugin.so[7f6231eb6000+
>>>>>>>>> b00
>>>>>>>>> >>>>
>>>>>>>>> >>>> Okay, this one is serious. The LDAP server crashed.
>>>>>>>>> >>>>
>>>>>>>>> >>>> 1. Make sure all your packages are up-to-date.
>>>>>>>>> >>>>
>>>>>>>>> >>>> Please see
>>>>>>>>> >>>> <http://directory.fedoraproject>
>>>>>>>>> http://directory.fedoraproject.org/docs/389ds/FAQ/faq.html#d
>>>>>>>>> >>>> ebugging-crashes
>>>>>>>>> >>>> for further instructions how to debug this.
>>>>>>>>> >>>>
>>>>>>>>> >>>> Petr^2 Spacek
>>>>>>>>> >>>>
>>>>>>>>> >>>>>
>>>>>>>>> >>>>> and in /var/log/dirsrv/example-com/errors
>>>>>>>>> >>>>>
>>>>>>>>> >>>>> [23/Aug/2016:12:49:36 +0000] DSRetroclPlugin -
>>>>>>>>> delete_changerecord:
>>>>>>>>> >>>> could
>>>>>>>>> >>>>> not delete change record 3291138 (rc: 32)
>>>>>>>>> >>>>> [23/Aug/2016:12:49:36 +0000] DSRetroclPlugin -
>>>>>>>>> delete_changerecord:
>>>>>>>>> >>>> could
>>>>>>>>> >>>>> not delete change record 3291139 (rc: 32)
>>>>>>>>> >>>>> [23/Aug/2016:12:49:36 +0000] DSRetroclPlugin -
>>>>>>>>> delete_changerecord:
>>>>>>>>> >>>> could
>>>>>>>>> >>>>> not delete change record 3291140 (rc: 32)
>>>>>>>>> >>>>> [23/Aug/2016:12:49:36 +0000] DSRetroclPlugin -
>>>>>>>>> delete_changerecord:
>>>>>>>>> >>>> could
>>>>>>>>> >>>>> not delete change record 3291141 (rc: 32)
>>>>>>>>> >>>>> [23/Aug/2016:12:49:36 +0000] DSRetroclPlugin -
>>>>>>>>> delete_changerecord:
>>>>>>>>> >>>> could
>>>>>>>>> >>>>> not delete change record 3291142 (rc: 32)
>>>>>>>>> >>>>> [23/Aug/2016:12:49:36 +0000] DSRetroclPlugin -
>>>>>>>>> delete_changerecord:
>>>>>>>>> >>>> could
>>>>>>>>> >>>>> not delete change record 3291143 (rc: 32)
>>>>>>>>> >>>>> [23/Aug/2016:12:49:36 +0000] DSRetroclPlugin -
>>>>>>>>> delete_changerecord:
>>>>>>>>> >>>> could
>>>>>>>>> >>>>> not delete change record 3291144 (rc: 32)
>>>>>>>>> >>>>> [23/Aug/2016:12:49:36 +0000] DSRetroclPlugin -
>>>>>>>>> delete_changerecord:
>>>>>>>>> >>>> could
>>>>>>>>> >>>>> not delete change record 3291145 (rc: 32)
>>>>>>>>> >>>>> [23/Aug/2016:12:49:50 +0000] - Retry count exceeded in delete
>>>>>>>>> >>>>> [23/Aug/2016:12:49:50 +0000] DSRetroclPlugin -
>>>>>>>>> delete_changerecord:
>>>>>>>>> >>>> could
>>>>>>>>> >>>>> not delete change record 3292734 (rc: 51)
>>>>>>>>> >>>>>
>>>>>>>>> >>>>>
>>>>>>>>> >>>>> Can i do something about this error.. I treid to restart
>>>>>>>>> ipa a couple
>>>>>>>>> >>>> of
>>>>>>>>> >>>>> time but that did not help
>>>>>>>>> >>>>>
>>>>>>>>> >>>>> Thanks
>>>>>>>>> >>>>> Rakesh
>>>>>>>>> >>>>>
>>>>>>>>> >>>>> On Mon, Aug 22, 2016 at 2:27 PM, Petr Spacek <
>>>>>>>>> <pspacek at redhat.com>pspacek at redhat.com>
>>>>>>>>> >>>> wrote:
>>>>>>>>> >>>>>
>>>>>>>>> >>>>>> On 19.8.2016 19:32, Rakesh Rajasekharan wrote:
>>>>>>>>> >>>>>>> I am running my set up on AWS cloud, and entropy is low at
>>>>>>>>> around
>>>>>>>>> >>>> 180 .
>>>>>>>>> >>>>>>>
>>>>>>>>> >>>>>>> I plan to increase it bu installing haveged . But, would
>>>>>>>>> low entropy
>>>>>>>>> >>>> by
>>>>>>>>> >>>>>> any
>>>>>>>>> >>>>>>> chance cause this issue of intermittent hang .
>>>>>>>>> >>>>>>> Also, the hang is mostly observed when registering around
>>>>>>>>> 20 clients
>>>>>>>>> >>>>>>> together
>>>>>>>>> >>>>>>
>>>>>>>>> >>>>>> Possibly, I'm not sure. If you want to dig into this, I
>>>>>>>>> would do this:
>>>>>>>>> >>>>>> 1. look what process hangs on client (using pstree command
>>>>>>>>> or so)
>>>>>>>>> >>>>>> $ pstree
>>>>>>>>> >>>>>>
>>>>>>>>> >>>>>> 2. look to what server and port is the hanging client
>>>>>>>>> connected to
>>>>>>>>> >>>>>> $ lsof -p <PID of the hanging process>
>>>>>>>>> >>>>>>
>>>>>>>>> >>>>>> 3. jump to server and see what process is bound to the
>>>>>>>>> target port
>>>>>>>>> >>>>>> $ netstat -pn
>>>>>>>>> >>>>>>
>>>>>>>>> >>>>>> 4. see where the process if hanging
>>>>>>>>> >>>>>> $ strace -p <PID of the hanging process>
>>>>>>>>> >>>>>>
>>>>>>>>> >>>>>> I hope it helps.
>>>>>>>>> >>>>>>
>>>>>>>>> >>>>>> Petr^2 Spacek
>>>>>>>>> >>>>>>
>>>>>>>>> >>>>>>> On Fri, Aug 19, 2016 at 7:24 PM, Rakesh Rajasekharan <
>>>>>>>>> >>>>>>> <rakesh.rajasekharan at gmail.com>
>>>>>>>>> rakesh.rajasekharan at gmail.com> wrote:
>>>>>>>>> >>>>>>>
>>>>>>>>> >>>>>>>> yes there seems to be something thats worrying.. I have
>>>>>>>>> faced this
>>>>>>>>> >>>> today
>>>>>>>>> >>>>>>>> as well.
>>>>>>>>> >>>>>>>> There are few hosts around 280 odd left and when i try
>>>>>>>>> adding them
>>>>>>>>> >>>> to
>>>>>>>>> >>>>>> IPA
>>>>>>>>> >>>>>>>> , the slowness begins..
>>>>>>>>> >>>>>>>>
>>>>>>>>> >>>>>>>> all the ipa commands like ipa user-find.. etc becomes
>>>>>>>>> very slow in
>>>>>>>>> >>>>>>>> responding.
>>>>>>>>> >>>>>>>>
>>>>>>>>> >>>>>>>> the SYNC_RECV are not many though just around 80-90 and
>>>>>>>>> today that
>>>>>>>>> >>>> was
>>>>>>>>> >>>>>>>> around 20 only
>>>>>>>>> >>>>>>>>
>>>>>>>>> >>>>>>>>
>>>>>>>>> >>>>>>>> I have for now increased tcp_max_syn_backlog to 5000.
>>>>>>>>> >>>>>>>> For now the slowness seems to have gone.. but I will do a
>>>>>>>>> try
>>>>>>>>> >>>> adding the
>>>>>>>>> >>>>>>>> clients again tomorrow and see how it goes
>>>>>>>>> >>>>>>>>
>>>>>>>>> >>>>>>>> Thanks
>>>>>>>>> >>>>>>>> Rakesh
>>>>>>>>> >>>>>>>>
>>>>>>>>> >>>>>>>> The issues
>>>>>>>>> >>>>>>>>
>>>>>>>>> >>>>>>>> On Fri, Aug 19, 2016 at 12:58 PM, Petr Spacek <
>>>>>>>>> <pspacek at redhat.com>pspacek at redhat.com>
>>>>>>>>> >>>>>> wrote:
>>>>>>>>> >>>>>>>>
>>>>>>>>> >>>>>>>>> On 18.8.2016 17:23, Rakesh Rajasekharan wrote:
>>>>>>>>> >>>>>>>>>> Hi
>>>>>>>>> >>>>>>>>>>
>>>>>>>>> >>>>>>>>>> I am migrating to freeipa from openldap and have around
>>>>>>>>> 4000
>>>>>>>>> >>>> clients
>>>>>>>>> >>>>>>>>>>
>>>>>>>>> >>>>>>>>>> I had openned a another thread on that, but chose to
>>>>>>>>> start a new
>>>>>>>>> >>>> one
>>>>>>>>> >>>>>>>>> here
>>>>>>>>> >>>>>>>>>> as its a separate issue
>>>>>>>>> >>>>>>>>>>
>>>>>>>>> >>>>>>>>>> I was able to change the nssslapd-maxdescriptors adding
>>>>>>>>> an ldif
>>>>>>>>> >>>> file
>>>>>>>>> >>>>>>>>>>
>>>>>>>>> >>>>>>>>>> cat nsslapd-modify.ldif
>>>>>>>>> >>>>>>>>>> dn: cn=config
>>>>>>>>> >>>>>>>>>> changetype: modify
>>>>>>>>> >>>>>>>>>> replace: nsslapd-maxdescriptors
>>>>>>>>> >>>>>>>>>> nsslapd-maxdescriptors: 17000
>>>>>>>>> >>>>>>>>>>
>>>>>>>>> >>>>>>>>>> and running the ldapmodify command
>>>>>>>>> >>>>>>>>>>
>>>>>>>>> >>>>>>>>>> I have now started moving clients running an openldap
>>>>>>>>> to Freeipa
>>>>>>>>> >>>> and
>>>>>>>>> >>>>>>>>> have
>>>>>>>>> >>>>>>>>>> today moved close to 2000 clients
>>>>>>>>> >>>>>>>>>>
>>>>>>>>> >>>>>>>>>> However, I have noticed that IPA hangs intermittently.
>>>>>>>>> >>>>>>>>>>
>>>>>>>>> >>>>>>>>>> running a kinit admin returns the below error
>>>>>>>>> >>>>>>>>>> kinit: Generic error (see e-text) while getting initial
>>>>>>>>> >>>> credentials
>>>>>>>>> >>>>>>>>>>
>>>>>>>>> >>>>>>>>>> from the /var/log/messages, I see this entry
>>>>>>>>> >>>>>>>>>>
>>>>>>>>> >>>>>>>>>> prod-ipa-master-int kernel: [104090.315801] TCP:
>>>>>>>>> >>>> request_sock_TCP:
>>>>>>>>> >>>>>>>>>> Possible SYN flooding on port 88. Sending cookies.
>>>>>>>>> Check SNMP
>>>>>>>>> >>>>>> counters.
>>>>>>>>> >>>>>>>>>
>>>>>>>>> >>>>>>>>> I would be worried about this message. Maybe
>>>>>>>>> kernel/firewall is
>>>>>>>>> >>>> doing
>>>>>>>>> >>>>>>>>> something fishy behind your back and blocking some
>>>>>>>>> connections or
>>>>>>>>> >>>> so.
>>>>>>>>> >>>>>>>>>
>>>>>>>>> >>>>>>>>> Petr^2 Spacek
>>>>>>>>> >>>>>>>>>
>>>>>>>>> >>>>>>>>>
>>>>>>>>> >>>>>>>>>> Aug 18 13:00:01 prod-ipa-master-int systemd[1]: Started
>>>>>>>>> Session
>>>>>>>>> >>>> 4885
>>>>>>>>> >>>>>> of
>>>>>>>>> >>>>>>>>>> user root.
>>>>>>>>> >>>>>>>>>> Aug 18 13:00:01 prod-ipa-master-int systemd[1]:
>>>>>>>>> Starting Session
>>>>>>>>> >>>> 4885
>>>>>>>>> >>>>>> of
>>>>>>>>> >>>>>>>>>> user root.
>>>>>>>>> >>>>>>>>>> Aug 18 13:01:01 prod-ipa-master-int systemd[1]: Started
>>>>>>>>> Session
>>>>>>>>> >>>> 4886
>>>>>>>>> >>>>>> of
>>>>>>>>> >>>>>>>>>> user root.
>>>>>>>>> >>>>>>>>>> Aug 18 13:01:01 prod-ipa-master-int systemd[1]:
>>>>>>>>> Starting Session
>>>>>>>>> >>>> 4886
>>>>>>>>> >>>>>> of
>>>>>>>>> >>>>>>>>>> user root.
>>>>>>>>> >>>>>>>>>> Aug 18 13:02:40 prod-ipa-master-int python[28984]:
>>>>>>>>> ansible-command
>>>>>>>>> >>>>>>>>> Invoked
>>>>>>>>> >>>>>>>>>> with creates=None executable=None shell=True args=
>>>>>>>>> removes=None
>>>>>>>>> >>>>>>>>> warn=True
>>>>>>>>> >>>>>>>>>> chdir=None
>>>>>>>>> >>>>>>>>>> Aug 18 13:04:37 prod-ipa-master-int sssd_be: GSSAPI
>>>>>>>>> Error:
>>>>>>>>> >>>> Unspecified
>>>>>>>>> >>>>>>>>> GSS
>>>>>>>>> >>>>>>>>>> failure. Minor code may provide more information (KDC
>>>>>>>>> returned
>>>>>>>>> >>>> error
>>>>>>>>> >>>>>>>>>> string: PROCESS_TGS)
>>>>>>>>> >>>>>>>>>>
>>>>>>>>> >>>>>>>>>> Could it be possible that its due to the initial load
>>>>>>>>> of adding
>>>>>>>>> >>>> the
>>>>>>>>> >>>>>>>>> clients
>>>>>>>>> >>>>>>>>>> or is there something else that I need to take care of.
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Manage your subscription for the Freeipa-users mailing list:
>>>>> https://www.redhat.com/mailman/listinfo/freeipa-users
>>>>> Go to http://freeipa.org for more info on the project
>>>>>
>>>>
>>>>
>>>
>>>
>>>
>>>
>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/freeipa-users/attachments/20160905/ae89313a/attachment.htm>
More information about the Freeipa-users
mailing list