[Freeipa-users] Problem with replication
thierry bordaz
tbordaz at redhat.com
Wed May 6 09:47:11 UTC 2015
This is looking like thread 13 prevents thread 12 run (and all the others).
Now thread 13 is likely waiting for db page? We may need output of
db_stat (db_state -N -h /var/lib/dirsrv/slapd-xxx/db/ -CA)
thanks
thierry
On 05/06/2015 11:31 AM, Łukasz Jaworski wrote:
>>> ldapsearch hangs. Dirsrv is not responding now.
>> if the server is hanging, can you get a pstack
> Thread 45 (Thread 0x7fc6a562d700 (LWP 1868)):
> #0 0x00007fc6b2f1aae3 in select () from /lib64/libc.so.6
> #1 0x00007fc6b5492a99 in DS_Sleep () from /usr/lib64/dirsrv/libslapd.so.0
> #2 0x00007fc6a961f987 in deadlock_threadmain () from /usr/lib64/dirsrv/plugins/libback-ldbm.so
> #3 0x00007fc6b3847cab in _pt_root () from /lib64/libnspr4.so
> #4 0x00007fc6b31e852a in start_thread () from /lib64/libpthread.so.0
> #5 0x00007fc6b2f2422d in clone () from /lib64/libc.so.6
> Thread 44 (Thread 0x7fc6a4e2c700 (LWP 1869)):
> #0 0x00007fc6b2f1aae3 in select () from /lib64/libc.so.6
> #1 0x00007fc6b5492a99 in DS_Sleep () from /usr/lib64/dirsrv/libslapd.so.0
> #2 0x00007fc6a9623a4e in checkpoint_threadmain () from /usr/lib64/dirsrv/plugins/libback-ldbm.so
> #3 0x00007fc6b3847cab in _pt_root () from /lib64/libnspr4.so
> #4 0x00007fc6b31e852a in start_thread () from /lib64/libpthread.so.0
> #5 0x00007fc6b2f2422d in clone () from /lib64/libc.so.6
> Thread 43 (Thread 0x7fc6a462b700 (LWP 1870)):
> #0 0x00007fc6b2f1aae3 in select () from /lib64/libc.so.6
> #1 0x00007fc6b5492a99 in DS_Sleep () from /usr/lib64/dirsrv/libslapd.so.0
> #2 0x00007fc6a961fc0f in trickle_threadmain () from /usr/lib64/dirsrv/plugins/libback-ldbm.so
> #3 0x00007fc6b3847cab in _pt_root () from /lib64/libnspr4.so
> #4 0x00007fc6b31e852a in start_thread () from /lib64/libpthread.so.0
> #5 0x00007fc6b2f2422d in clone () from /lib64/libc.so.6
> Thread 42 (Thread 0x7fc6a3e2a700 (LWP 1871)):
> #0 0x00007fc6b2f1aae3 in select () from /lib64/libc.so.6
> #1 0x00007fc6b5492a99 in DS_Sleep () from /usr/lib64/dirsrv/libslapd.so.0
> #2 0x00007fc6a961a667 in perf_threadmain () from /usr/lib64/dirsrv/plugins/libback-ldbm.so
> #3 0x00007fc6b3847cab in _pt_root () from /lib64/libnspr4.so
> #4 0x00007fc6b31e852a in start_thread () from /lib64/libpthread.so.0
> #5 0x00007fc6b2f2422d in clone () from /lib64/libc.so.6
> Thread 41 (Thread 0x7fc6a3629700 (LWP 1900)):
> #0 0x00007fc6b31ed590 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
> #1 0x00007fc6b3842440 in PR_WaitCondVar () from /lib64/libnspr4.so
> #2 0x00007fc6b5482c68 in slapi_wait_condvar () from /usr/lib64/dirsrv/libslapd.so.0
> #3 0x00007fc6abd455be in cos_cache_wait_on_change () from /usr/lib64/dirsrv/plugins/libcos-plugin.so
> #4 0x00007fc6b3847cab in _pt_root () from /lib64/libnspr4.so
> #5 0x00007fc6b31e852a in start_thread () from /lib64/libpthread.so.0
> #6 0x00007fc6b2f2422d in clone () from /lib64/libc.so.6
> Thread 40 (Thread 0x7fc6b5735700 (LWP 1901)):
> #0 0x00007fc6b31ed939 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
> #1 0x00007fc6b3841ef8 in pt_TimedWait () from /lib64/libnspr4.so
> #2 0x00007fc6b38423be in PR_WaitCondVar () from /lib64/libnspr4.so
> #3 0x00007fc6a937be84 in _cl5TrimMain () from /usr/lib64/dirsrv/plugins/libreplication-plugin.so
> #4 0x00007fc6b3847cab in _pt_root () from /lib64/libnspr4.so
> #5 0x00007fc6b31e852a in start_thread () from /lib64/libpthread.so.0
> #6 0x00007fc6b2f2422d in clone () from /lib64/libc.so.6
> Thread 39 (Thread 0x7fc6a2e28700 (LWP 1902)):
> #0 0x00007fc6b31ed590 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
> #1 0x00007fc6b3842440 in PR_WaitCondVar () from /lib64/libnspr4.so
> #2 0x00007fc6a93928d4 in protocol_sleep () from /usr/lib64/dirsrv/plugins/libreplication-plugin.so
> #3 0x00007fc6a93949e5 in repl5_inc_run () from /usr/lib64/dirsrv/plugins/libreplication-plugin.so
> #4 0x00007fc6a9399421 in prot_thread_main () from /usr/lib64/dirsrv/plugins/libreplication-plugin.so
> #5 0x00007fc6b3847cab in _pt_root () from /lib64/libnspr4.so
> #6 0x00007fc6b31e852a in start_thread () from /lib64/libpthread.so.0
> #7 0x00007fc6b2f2422d in clone () from /lib64/libc.so.6
> Thread 38 (Thread 0x7fc6a2627700 (LWP 1903)):
> #0 0x00007fc6b31ed939 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
> #1 0x00007fc6b3841ef8 in pt_TimedWait () from /lib64/libnspr4.so
> #2 0x00007fc6b38423be in PR_WaitCondVar () from /lib64/libnspr4.so
> #3 0x00007fc6a93928d4 in protocol_sleep () from /usr/lib64/dirsrv/plugins/libreplication-plugin.so
> #4 0x00007fc6a9395158 in repl5_inc_run () from /usr/lib64/dirsrv/plugins/libreplication-plugin.so
> #5 0x00007fc6a9399421 in prot_thread_main () from /usr/lib64/dirsrv/plugins/libreplication-plugin.so
> #6 0x00007fc6b3847cab in _pt_root () from /lib64/libnspr4.so
> #7 0x00007fc6b31e852a in start_thread () from /lib64/libpthread.so.0
> #8 0x00007fc6b2f2422d in clone () from /lib64/libc.so.6
> Thread 37 (Thread 0x7fc6a1a04700 (LWP 1906)):
> #0 0x00007fc6b31ed590 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
> #1 0x00007fc6b3842440 in PR_WaitCondVar () from /lib64/libnspr4.so
> #2 0x00007fc6b5482c68 in slapi_wait_condvar () from /usr/lib64/dirsrv/libslapd.so.0
> #3 0x00007fc6a7cbef5d in roles_cache_wait_on_change () from /usr/lib64/dirsrv/plugins/libroles-plugin.so
> #4 0x00007fc6b3847cab in _pt_root () from /lib64/libnspr4.so
> #5 0x00007fc6b31e852a in start_thread () from /lib64/libpthread.so.0
> #6 0x00007fc6b2f2422d in clone () from /lib64/libc.so.6
> Thread 36 (Thread 0x7fc6a1203700 (LWP 1907)):
> #0 0x00007fc6b31ed590 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
> #1 0x00007fc6b3842440 in PR_WaitCondVar () from /lib64/libnspr4.so
> #2 0x00007fc6b5482c68 in slapi_wait_condvar () from /usr/lib64/dirsrv/libslapd.so.0
> #3 0x00007fc6a7cbef5d in roles_cache_wait_on_change () from /usr/lib64/dirsrv/plugins/libroles-plugin.so
> #4 0x00007fc6b3847cab in _pt_root () from /lib64/libnspr4.so
> #5 0x00007fc6b31e852a in start_thread () from /lib64/libpthread.so.0
> #6 0x00007fc6b2f2422d in clone () from /lib64/libc.so.6
> Thread 35 (Thread 0x7fc6a0a02700 (LWP 1908)):
> #0 0x00007fc6b31ed590 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
> #1 0x00007fc6b3842440 in PR_WaitCondVar () from /lib64/libnspr4.so
> #2 0x00007fc6b5482c68 in slapi_wait_condvar () from /usr/lib64/dirsrv/libslapd.so.0
> #3 0x00007fc6a7cbef5d in roles_cache_wait_on_change () from /usr/lib64/dirsrv/plugins/libroles-plugin.so
> #4 0x00007fc6b3847cab in _pt_root () from /lib64/libnspr4.so
> #5 0x00007fc6b31e852a in start_thread () from /lib64/libpthread.so.0
> #6 0x00007fc6b2f2422d in clone () from /lib64/libc.so.6
> Thread 34 (Thread 0x7fc693fff700 (LWP 1909)):
> #0 0x00007fc6b31ed939 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
> #1 0x00007fc6b3841ef8 in pt_TimedWait () from /lib64/libnspr4.so
> #2 0x00007fc6b38423be in PR_WaitCondVar () from /lib64/libnspr4.so
> #3 0x00007fc6b593a773 in housecleaning ()
> #4 0x00007fc6b3847cab in _pt_root () from /lib64/libnspr4.so
> #5 0x00007fc6b31e852a in start_thread () from /lib64/libpthread.so.0
> #6 0x00007fc6b2f2422d in clone () from /lib64/libc.so.6
> Thread 33 (Thread 0x7fc6937fe700 (LWP 1910)):
> #0 0x00007fc6b31ed590 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
> #1 0x00007fc6b38426b3 in PR_EnterMonitor () from /lib64/libnspr4.so
> #2 0x00007fc6a9622456 in dblayer_txn_begin () from /usr/lib64/dirsrv/plugins/libback-ldbm.so
> #3 0x00007fc6a965d615 in ldbm_back_modify () from /usr/lib64/dirsrv/plugins/libback-ldbm.so
> #4 0x00007fc6b544e905 in op_shared_modify () from /usr/lib64/dirsrv/libslapd.so.0
> #5 0x00007fc6b544f387 in modify_internal_pb () from /usr/lib64/dirsrv/libslapd.so.0
> #6 0x00007fc6a939f76f in replica_write_ruv () from /usr/lib64/dirsrv/plugins/libreplication-plugin.so
> #7 0x00007fc6a939f9fe in replica_update_state () from /usr/lib64/dirsrv/plugins/libreplication-plugin.so
> #8 0x00007fc6b542965a in eq_loop () from /usr/lib64/dirsrv/libslapd.so.0
> #9 0x00007fc6b3847cab in _pt_root () from /lib64/libnspr4.so
> #10 0x00007fc6b31e852a in start_thread () from /lib64/libpthread.so.0
> #11 0x00007fc6b2f2422d in clone () from /lib64/libc.so.6
> Thread 32 (Thread 0x7fc692924700 (LWP 1912)):
> #0 0x00007fc6b31ed590 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
> #1 0x00007fc6b3842440 in PR_WaitCondVar () from /lib64/libnspr4.so
> #2 0x00007fc6b5930b9e in connection_wait_for_new_work ()
> #3 0x00007fc6b5931dc9 in connection_threadmain ()
> #4 0x00007fc6b3847cab in _pt_root () from /lib64/libnspr4.so
> #5 0x00007fc6b31e852a in start_thread () from /lib64/libpthread.so.0
> #6 0x00007fc6b2f2422d in clone () from /lib64/libc.so.6
> Thread 31 (Thread 0x7fc692123700 (LWP 1913)):
> #0 0x00007fc6b31ed590 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
> #1 0x00007fc6b3842440 in PR_WaitCondVar () from /lib64/libnspr4.so
> #2 0x00007fc6b5930b9e in connection_wait_for_new_work ()
> #3 0x00007fc6b5931dc9 in connection_threadmain ()
> #4 0x00007fc6b3847cab in _pt_root () from /lib64/libnspr4.so
> #5 0x00007fc6b31e852a in start_thread () from /lib64/libpthread.so.0
> #6 0x00007fc6b2f2422d in clone () from /lib64/libc.so.6
> Thread 30 (Thread 0x7fc691922700 (LWP 1914)):
> #0 0x00007fc6b31ed590 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
> #1 0x00007fc6adc811fb in __db_hybrid_mutex_suspend () from /lib64/libdb-5.3.so
> #2 0x00007fc6adc8059b in __db_tas_mutex_lock () from /lib64/libdb-5.3.so
> #3 0x00007fc6add2aa91 in __lock_get_internal () from /lib64/libdb-5.3.so
> #4 0x00007fc6add2b657 in __lock_get () from /lib64/libdb-5.3.so
> #5 0x00007fc6add576af in __db_lget () from /lib64/libdb-5.3.so
> #6 0x00007fc6adc9d5d9 in __bam_get_root () from /lib64/libdb-5.3.so
> #7 0x00007fc6adc9d91b in __bam_search () from /lib64/libdb-5.3.so
> #8 0x00007fc6adc89144 in __bamc_search () from /lib64/libdb-5.3.so
> #9 0x00007fc6adc8adff in __bamc_get () from /lib64/libdb-5.3.so
> #10 0x00007fc6add43ba3 in __dbc_iget () from /lib64/libdb-5.3.so
> #11 0x00007fc6add53092 in __dbc_get_pp () from /lib64/libdb-5.3.so
> #12 0x00007fc6a962e6de in idl_new_fetch () from /usr/lib64/dirsrv/plugins/libback-ldbm.so
> #13 0x00007fc6a963cc2b in index_read_ext_allids () from /usr/lib64/dirsrv/plugins/libback-ldbm.so
> #14 0x00007fc6a96272ed in keys2idl () from /usr/lib64/dirsrv/plugins/libback-ldbm.so
> #15 0x00007fc6a9627afe in ava_candidates.isra () from /usr/lib64/dirsrv/plugins/libback-ldbm.so
> #16 0x00007fc6a96280fa in filter_candidates_ext () from /usr/lib64/dirsrv/plugins/libback-ldbm.so
> #17 0x00007fc6a96290ce in list_candidates () from /usr/lib64/dirsrv/plugins/libback-ldbm.so
> #18 0x00007fc6a9628062 in filter_candidates_ext () from /usr/lib64/dirsrv/plugins/libback-ldbm.so
> #19 0x00007fc6a96631a7 in subtree_candidates () from /usr/lib64/dirsrv/plugins/libback-ldbm.so
> #20 0x00007fc6a9664811 in ldbm_back_search () from /usr/lib64/dirsrv/plugins/libback-ldbm.so
> #21 0x00007fc6b5455410 in op_shared_search () from /usr/lib64/dirsrv/libslapd.so.0
> #22 0x00007fc6b5943fc6 in do_search ()
> #23 0x00007fc6b5932b4c in connection_threadmain ()
> #24 0x00007fc6b3847cab in _pt_root () from /lib64/libnspr4.so
> #25 0x00007fc6b31e852a in start_thread () from /lib64/libpthread.so.0
> #26 0x00007fc6b2f2422d in clone () from /lib64/libc.so.6
> Thread 29 (Thread 0x7fc691121700 (LWP 1915)):
> #0 0x00007fc6b31ed590 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
> #1 0x00007fc6adc811fb in __db_hybrid_mutex_suspend () from /lib64/libdb-5.3.so
> #2 0x00007fc6adc8059b in __db_tas_mutex_lock () from /lib64/libdb-5.3.so
> #3 0x00007fc6add2aa91 in __lock_get_internal () from /lib64/libdb-5.3.so
> #4 0x00007fc6add2b657 in __lock_get () from /lib64/libdb-5.3.so
> #5 0x00007fc6add576af in __db_lget () from /lib64/libdb-5.3.so
> #6 0x00007fc6adc9d5d9 in __bam_get_root () from /lib64/libdb-5.3.so
> #7 0x00007fc6adc9d91b in __bam_search () from /lib64/libdb-5.3.so
> #8 0x00007fc6adc89144 in __bamc_search () from /lib64/libdb-5.3.so
> #9 0x00007fc6adc8adff in __bamc_get () from /lib64/libdb-5.3.so
> #10 0x00007fc6add43ba3 in __dbc_iget () from /lib64/libdb-5.3.so
> #11 0x00007fc6add53092 in __dbc_get_pp () from /lib64/libdb-5.3.so
> #12 0x00007fc6a962e6de in idl_new_fetch () from /usr/lib64/dirsrv/plugins/libback-ldbm.so
> #13 0x00007fc6a963cc2b in index_read_ext_allids () from /usr/lib64/dirsrv/plugins/libback-ldbm.so
> #14 0x00007fc6a96272ed in keys2idl () from /usr/lib64/dirsrv/plugins/libback-ldbm.so
> #15 0x00007fc6a9627afe in ava_candidates.isra () from /usr/lib64/dirsrv/plugins/libback-ldbm.so
> #16 0x00007fc6a96280fa in filter_candidates_ext () from /usr/lib64/dirsrv/plugins/libback-ldbm.so
> #17 0x00007fc6a96290ce in list_candidates () from /usr/lib64/dirsrv/plugins/libback-ldbm.so
> #18 0x00007fc6a9628062 in filter_candidates_ext () from /usr/lib64/dirsrv/plugins/libback-ldbm.so
> #19 0x00007fc6a96631a7 in subtree_candidates () from /usr/lib64/dirsrv/plugins/libback-ldbm.so
> #20 0x00007fc6a9664811 in ldbm_back_search () from /usr/lib64/dirsrv/plugins/libback-ldbm.so
> #21 0x00007fc6b5455410 in op_shared_search () from /usr/lib64/dirsrv/libslapd.so.0
> #22 0x00007fc6b5943fc6 in do_search ()
> #23 0x00007fc6b5932b4c in connection_threadmain ()
> #24 0x00007fc6b3847cab in _pt_root () from /lib64/libnspr4.so
> #25 0x00007fc6b31e852a in start_thread () from /lib64/libpthread.so.0
> #26 0x00007fc6b2f2422d in clone () from /lib64/libc.so.6
> Thread 28 (Thread 0x7fc690920700 (LWP 1916)):
> #0 0x00007fc6b31ed590 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
> #1 0x00007fc6adc811fb in __db_hybrid_mutex_suspend () from /lib64/libdb-5.3.so
> #2 0x00007fc6adc8059b in __db_tas_mutex_lock () from /lib64/libdb-5.3.so
> #3 0x00007fc6add2aa91 in __lock_get_internal () from /lib64/libdb-5.3.so
> #4 0x00007fc6add2b657 in __lock_get () from /lib64/libdb-5.3.so
> #5 0x00007fc6add576af in __db_lget () from /lib64/libdb-5.3.so
> #6 0x00007fc6adc9d5d9 in __bam_get_root () from /lib64/libdb-5.3.so
> #7 0x00007fc6adc9d91b in __bam_search () from /lib64/libdb-5.3.so
> #8 0x00007fc6adc89144 in __bamc_search () from /lib64/libdb-5.3.so
> #9 0x00007fc6adc8adff in __bamc_get () from /lib64/libdb-5.3.so
> #10 0x00007fc6add43ba3 in __dbc_iget () from /lib64/libdb-5.3.so
> #11 0x00007fc6add53092 in __dbc_get_pp () from /lib64/libdb-5.3.so
> #12 0x00007fc6a962e6de in idl_new_fetch () from /usr/lib64/dirsrv/plugins/libback-ldbm.so
> #13 0x00007fc6a963cc2b in index_read_ext_allids () from /usr/lib64/dirsrv/plugins/libback-ldbm.so
> #14 0x00007fc6a96272ed in keys2idl () from /usr/lib64/dirsrv/plugins/libback-ldbm.so
> #15 0x00007fc6a9627afe in ava_candidates.isra () from /usr/lib64/dirsrv/plugins/libback-ldbm.so
> #16 0x00007fc6a96280fa in filter_candidates_ext () from /usr/lib64/dirsrv/plugins/libback-ldbm.so
> #17 0x00007fc6a96290ce in list_candidates () from /usr/lib64/dirsrv/plugins/libback-ldbm.so
> #18 0x00007fc6a9628062 in filter_candidates_ext () from /usr/lib64/dirsrv/plugins/libback-ldbm.so
> #19 0x00007fc6a96631a7 in subtree_candidates () from /usr/lib64/dirsrv/plugins/libback-ldbm.so
> #20 0x00007fc6a9664811 in ldbm_back_search () from /usr/lib64/dirsrv/plugins/libback-ldbm.so
> #21 0x00007fc6b5455410 in op_shared_search () from /usr/lib64/dirsrv/libslapd.so.0
> #22 0x00007fc6b5943fc6 in do_search ()
> #23 0x00007fc6b5932b4c in connection_threadmain ()
> #24 0x00007fc6b3847cab in _pt_root () from /lib64/libnspr4.so
> #25 0x00007fc6b31e852a in start_thread () from /lib64/libpthread.so.0
> #26 0x00007fc6b2f2422d in clone () from /lib64/libc.so.6
> Thread 27 (Thread 0x7fc67ffff700 (LWP 1917)):
> #0 0x00007fc6b31ed590 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
> #1 0x00007fc6b3842440 in PR_WaitCondVar () from /lib64/libnspr4.so
> #2 0x00007fc6b5930b9e in connection_wait_for_new_work ()
> #3 0x00007fc6b5931dc9 in connection_threadmain ()
> #4 0x00007fc6b3847cab in _pt_root () from /lib64/libnspr4.so
> #5 0x00007fc6b31e852a in start_thread () from /lib64/libpthread.so.0
> #6 0x00007fc6b2f2422d in clone () from /lib64/libc.so.6
> Thread 26 (Thread 0x7fc67f7fe700 (LWP 1918)):
> #0 0x00007fc6b31ed590 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
> #1 0x00007fc6adc811fb in __db_hybrid_mutex_suspend () from /lib64/libdb-5.3.so
> #2 0x00007fc6adc8059b in __db_tas_mutex_lock () from /lib64/libdb-5.3.so
> #3 0x00007fc6add2aa91 in __lock_get_internal () from /lib64/libdb-5.3.so
> #4 0x00007fc6add2b657 in __lock_get () from /lib64/libdb-5.3.so
> #5 0x00007fc6add576af in __db_lget () from /lib64/libdb-5.3.so
> #6 0x00007fc6adc9e4f0 in __bam_search () from /lib64/libdb-5.3.so
> #7 0x00007fc6adc89144 in __bamc_search () from /lib64/libdb-5.3.so
> #8 0x00007fc6adc8adff in __bamc_get () from /lib64/libdb-5.3.so
> #9 0x00007fc6add43ba3 in __dbc_iget () from /lib64/libdb-5.3.so
> #10 0x00007fc6add50ced in __db_get () from /lib64/libdb-5.3.so
> #11 0x00007fc6add5473b in __db_get_pp () from /lib64/libdb-5.3.so
> #12 0x00007fc6a962a9bb in id2entry () from /usr/lib64/dirsrv/plugins/libback-ldbm.so
> #13 0x00007fc6a9626cbd in dn2entry_ext () from /usr/lib64/dirsrv/plugins/libback-ldbm.so
> #14 0x00007fc6a9629bb1 in find_entry_internal.isra () from /usr/lib64/dirsrv/plugins/libback-ldbm.so
> #15 0x00007fc6a9629e99 in find_entry () from /usr/lib64/dirsrv/plugins/libback-ldbm.so
> #16 0x00007fc6a9663b5b in ldbm_back_search () from /usr/lib64/dirsrv/plugins/libback-ldbm.so
> #17 0x00007fc6b5455410 in op_shared_search () from /usr/lib64/dirsrv/libslapd.so.0
> #18 0x00007fc6b546553e in search_internal_callback_pb () from /usr/lib64/dirsrv/libslapd.so.0
> #19 0x00007fc6b54657c8 in search_internal_pb () from /usr/lib64/dirsrv/libslapd.so.0
> #20 0x00007fc6b5465b73 in slapi_search_internal_get_entry () from /usr/lib64/dirsrv/libslapd.so.0
> #21 0x00007fc6aad02234 in ipalockout_preop () from /usr/lib64/dirsrv/plugins/libipa_lockout.so
> #22 0x00007fc6b546156f in plugin_call_func () from /usr/lib64/dirsrv/libslapd.so.0
> #23 0x00007fc6b5461893 in plugin_call_plugins () from /usr/lib64/dirsrv/libslapd.so.0
> #24 0x00007fc6b592c3d8 in do_bind ()
> #25 0x00007fc6b5932974 in connection_threadmain ()
> #26 0x00007fc6b3847cab in _pt_root () from /lib64/libnspr4.so
> #27 0x00007fc6b31e852a in start_thread () from /lib64/libpthread.so.0
> #28 0x00007fc6b2f2422d in clone () from /lib64/libc.so.6
> Thread 25 (Thread 0x7fc67effd700 (LWP 1919)):
> #0 0x00007fc6b31ed590 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
> #1 0x00007fc6b3842440 in PR_WaitCondVar () from /lib64/libnspr4.so
> #2 0x00007fc6b5930b9e in connection_wait_for_new_work ()
> #3 0x00007fc6b5931dc9 in connection_threadmain ()
> #4 0x00007fc6b3847cab in _pt_root () from /lib64/libnspr4.so
> #5 0x00007fc6b31e852a in start_thread () from /lib64/libpthread.so.0
> #6 0x00007fc6b2f2422d in clone () from /lib64/libc.so.6
> Thread 24 (Thread 0x7fc67e7fc700 (LWP 1920)):
> #0 0x00007fc6b31ed590 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
> #1 0x00007fc6b3842440 in PR_WaitCondVar () from /lib64/libnspr4.so
> #2 0x00007fc6b5930b9e in connection_wait_for_new_work ()
> #3 0x00007fc6b5931dc9 in connection_threadmain ()
> #4 0x00007fc6b3847cab in _pt_root () from /lib64/libnspr4.so
> #5 0x00007fc6b31e852a in start_thread () from /lib64/libpthread.so.0
> #6 0x00007fc6b2f2422d in clone () from /lib64/libc.so.6
> Thread 23 (Thread 0x7fc67dffb700 (LWP 1921)):
> #0 0x00007fc6b31ed590 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
> #1 0x00007fc6b3842440 in PR_WaitCondVar () from /lib64/libnspr4.so
> #2 0x00007fc6b5930b9e in connection_wait_for_new_work ()
> #3 0x00007fc6b5931dc9 in connection_threadmain ()
> #4 0x00007fc6b3847cab in _pt_root () from /lib64/libnspr4.so
> #5 0x00007fc6b31e852a in start_thread () from /lib64/libpthread.so.0
> #6 0x00007fc6b2f2422d in clone () from /lib64/libc.so.6
> Thread 22 (Thread 0x7fc67d7fa700 (LWP 1922)):
> #0 0x00007fc6b31ed590 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
> #1 0x00007fc6b38426b3 in PR_EnterMonitor () from /lib64/libnspr4.so
> #2 0x00007fc6a9622456 in dblayer_txn_begin () from /usr/lib64/dirsrv/plugins/libback-ldbm.so
> #3 0x00007fc6a965d615 in ldbm_back_modify () from /usr/lib64/dirsrv/plugins/libback-ldbm.so
> #4 0x00007fc6b544e905 in op_shared_modify () from /usr/lib64/dirsrv/libslapd.so.0
> #5 0x00007fc6b544f387 in modify_internal_pb () from /usr/lib64/dirsrv/libslapd.so.0
> #6 0x00007fc6aad02bd3 in ipalockout_postop () from /usr/lib64/dirsrv/plugins/libipa_lockout.so
> #7 0x00007fc6b546156f in plugin_call_func () from /usr/lib64/dirsrv/libslapd.so.0
> #8 0x00007fc6b5461893 in plugin_call_plugins () from /usr/lib64/dirsrv/libslapd.so.0
> #9 0x00007fc6b592c22d in do_bind ()
> #10 0x00007fc6b5932974 in connection_threadmain ()
> #11 0x00007fc6b3847cab in _pt_root () from /lib64/libnspr4.so
> #12 0x00007fc6b31e852a in start_thread () from /lib64/libpthread.so.0
> #13 0x00007fc6b2f2422d in clone () from /lib64/libc.so.6
> Thread 21 (Thread 0x7fc67cff9700 (LWP 1923)):
> #0 0x00007fc6b31ed590 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
> #1 0x00007fc6b38426b3 in PR_EnterMonitor () from /lib64/libnspr4.so
> #2 0x00007fc6a9622456 in dblayer_txn_begin () from /usr/lib64/dirsrv/plugins/libback-ldbm.so
> #3 0x00007fc6a965d615 in ldbm_back_modify () from /usr/lib64/dirsrv/plugins/libback-ldbm.so
> #4 0x00007fc6b544e905 in op_shared_modify () from /usr/lib64/dirsrv/libslapd.so.0
> #5 0x00007fc6b544f387 in modify_internal_pb () from /usr/lib64/dirsrv/libslapd.so.0
> #6 0x00007fc6aad02bd3 in ipalockout_postop () from /usr/lib64/dirsrv/plugins/libipa_lockout.so
> #7 0x00007fc6b546156f in plugin_call_func () from /usr/lib64/dirsrv/libslapd.so.0
> #8 0x00007fc6b5461893 in plugin_call_plugins () from /usr/lib64/dirsrv/libslapd.so.0
> #9 0x00007fc6b592c22d in do_bind ()
> #10 0x00007fc6b5932974 in connection_threadmain ()
> #11 0x00007fc6b3847cab in _pt_root () from /lib64/libnspr4.so
> #12 0x00007fc6b31e852a in start_thread () from /lib64/libpthread.so.0
> #13 0x00007fc6b2f2422d in clone () from /lib64/libc.so.6
> Thread 20 (Thread 0x7fc67c7f8700 (LWP 1924)):
> #0 0x00007fc6b31ed590 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
> #1 0x00007fc6b3842440 in PR_WaitCondVar () from /lib64/libnspr4.so
> #2 0x00007fc6b5930b9e in connection_wait_for_new_work ()
> #3 0x00007fc6b5931dc9 in connection_threadmain ()
> #4 0x00007fc6b3847cab in _pt_root () from /lib64/libnspr4.so
> #5 0x00007fc6b31e852a in start_thread () from /lib64/libpthread.so.0
> #6 0x00007fc6b2f2422d in clone () from /lib64/libc.so.6
> Thread 19 (Thread 0x7fc67bff7700 (LWP 1925)):
> #0 0x00007fc6b31ed590 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
> #1 0x00007fc6b38426b3 in PR_EnterMonitor () from /lib64/libnspr4.so
> #2 0x00007fc6a9622456 in dblayer_txn_begin () from /usr/lib64/dirsrv/plugins/libback-ldbm.so
> #3 0x00007fc6a965d615 in ldbm_back_modify () from /usr/lib64/dirsrv/plugins/libback-ldbm.so
> #4 0x00007fc6b544e905 in op_shared_modify () from /usr/lib64/dirsrv/libslapd.so.0
> #5 0x00007fc6b544f387 in modify_internal_pb () from /usr/lib64/dirsrv/libslapd.so.0
> #6 0x00007fc6aad02bd3 in ipalockout_postop () from /usr/lib64/dirsrv/plugins/libipa_lockout.so
> #7 0x00007fc6b546156f in plugin_call_func () from /usr/lib64/dirsrv/libslapd.so.0
> #8 0x00007fc6b5461893 in plugin_call_plugins () from /usr/lib64/dirsrv/libslapd.so.0
> #9 0x00007fc6b592c22d in do_bind ()
> #10 0x00007fc6b5932974 in connection_threadmain ()
> #11 0x00007fc6b3847cab in _pt_root () from /lib64/libnspr4.so
> #12 0x00007fc6b31e852a in start_thread () from /lib64/libpthread.so.0
> #13 0x00007fc6b2f2422d in clone () from /lib64/libc.so.6
> Thread 18 (Thread 0x7fc67b7f6700 (LWP 1926)):
> #0 0x00007fc6b31ed590 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
> #1 0x00007fc6b3842440 in PR_WaitCondVar () from /lib64/libnspr4.so
> #2 0x00007fc6b5930b9e in connection_wait_for_new_work ()
> #3 0x00007fc6b5931dc9 in connection_threadmain ()
> #4 0x00007fc6b3847cab in _pt_root () from /lib64/libnspr4.so
> #5 0x00007fc6b31e852a in start_thread () from /lib64/libpthread.so.0
> #6 0x00007fc6b2f2422d in clone () from /lib64/libc.so.6
> Thread 17 (Thread 0x7fc67aff5700 (LWP 1927)):
> #0 0x00007fc6b31ed590 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
> #1 0x00007fc6b3842440 in PR_WaitCondVar () from /lib64/libnspr4.so
> #2 0x00007fc6b5930b9e in connection_wait_for_new_work ()
> #3 0x00007fc6b5931dc9 in connection_threadmain ()
> #4 0x00007fc6b3847cab in _pt_root () from /lib64/libnspr4.so
> #5 0x00007fc6b31e852a in start_thread () from /lib64/libpthread.so.0
> #6 0x00007fc6b2f2422d in clone () from /lib64/libc.so.6
> Thread 16 (Thread 0x7fc67a7f4700 (LWP 1928)):
> #0 0x00007fc6b31ed590 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
> #1 0x00007fc6b3842440 in PR_WaitCondVar () from /lib64/libnspr4.so
> #2 0x00007fc6b5930b9e in connection_wait_for_new_work ()
> #3 0x00007fc6b5931dc9 in connection_threadmain ()
> #4 0x00007fc6b3847cab in _pt_root () from /lib64/libnspr4.so
> #5 0x00007fc6b31e852a in start_thread () from /lib64/libpthread.so.0
> #6 0x00007fc6b2f2422d in clone () from /lib64/libc.so.6
> Thread 15 (Thread 0x7fc679ff3700 (LWP 1929)):
> #0 0x00007fc6b31ed590 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
> #1 0x00007fc6b3842440 in PR_WaitCondVar () from /lib64/libnspr4.so
> #2 0x00007fc6b5930b9e in connection_wait_for_new_work ()
> #3 0x00007fc6b5931dc9 in connection_threadmain ()
> #4 0x00007fc6b3847cab in _pt_root () from /lib64/libnspr4.so
> #5 0x00007fc6b31e852a in start_thread () from /lib64/libpthread.so.0
> #6 0x00007fc6b2f2422d in clone () from /lib64/libc.so.6
> Thread 14 (Thread 0x7fc6797f2700 (LWP 1930)):
> #0 0x00007fc6b31eff1d in __lll_lock_wait () from /lib64/libpthread.so.0
> #1 0x00007fc6b31ea9be in pthread_mutex_lock () from /lib64/libpthread.so.0
> #2 0x00007fc6b38420b9 in PR_Lock () from /lib64/libnspr4.so
> #3 0x00007fc6a7ec99b0 in retrocl_postob () from /usr/lib64/dirsrv/plugins/libretrocl-plugin.so
> #4 0x00007fc6b546156f in plugin_call_func () from /usr/lib64/dirsrv/libslapd.so.0
> #5 0x00007fc6b5461893 in plugin_call_plugins () from /usr/lib64/dirsrv/libslapd.so.0
> #6 0x00007fc6a965d231 in ldbm_back_modify () from /usr/lib64/dirsrv/plugins/libback-ldbm.so
> #7 0x00007fc6b544e905 in op_shared_modify () from /usr/lib64/dirsrv/libslapd.so.0
> #8 0x00007fc6b544f387 in modify_internal_pb () from /usr/lib64/dirsrv/libslapd.so.0
> #9 0x00007fc6a8d354a2 in memberof_fix_memberof_callback () from /usr/lib64/dirsrv/plugins/libmemberof-plugin.so
> #10 0x00007fc6a8d35f65 in memberof_modop_one_replace_r () from /usr/lib64/dirsrv/plugins/libmemberof-plugin.so
> #11 0x00007fc6a8d362a6 in memberof_mod_smod_list () from /usr/lib64/dirsrv/plugins/libmemberof-plugin.so
> #12 0x00007fc6a8d3758d in memberof_postop_modify () from /usr/lib64/dirsrv/plugins/libmemberof-plugin.so
> #13 0x00007fc6b546156f in plugin_call_func () from /usr/lib64/dirsrv/libslapd.so.0
> #14 0x00007fc6b5461893 in plugin_call_plugins () from /usr/lib64/dirsrv/libslapd.so.0
> #15 0x00007fc6a965d231 in ldbm_back_modify () from /usr/lib64/dirsrv/plugins/libback-ldbm.so
> #16 0x00007fc6b544e905 in op_shared_modify () from /usr/lib64/dirsrv/libslapd.so.0
> #17 0x00007fc6b544fbe7 in do_modify () from /usr/lib64/dirsrv/libslapd.so.0
> #18 0x00007fc6b5932925 in connection_threadmain ()
> #19 0x00007fc6b3847cab in _pt_root () from /lib64/libnspr4.so
> #20 0x00007fc6b31e852a in start_thread () from /lib64/libpthread.so.0
> #21 0x00007fc6b2f2422d in clone () from /lib64/libc.so.6
> Thread 13 (Thread 0x7fc678ff1700 (LWP 1931)):
> #0 0x00007fc6b31ed590 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
> #1 0x00007fc6adc811fb in __db_hybrid_mutex_suspend () from /lib64/libdb-5.3.so
> #2 0x00007fc6adc8059b in __db_tas_mutex_lock () from /lib64/libdb-5.3.so
> #3 0x00007fc6add2aa91 in __lock_get_internal () from /lib64/libdb-5.3.so
> #4 0x00007fc6add2b657 in __lock_get () from /lib64/libdb-5.3.so
> #5 0x00007fc6add576af in __db_lget () from /lib64/libdb-5.3.so
> #6 0x00007fc6adc9e4f0 in __bam_search () from /lib64/libdb-5.3.so
> #7 0x00007fc6adc89144 in __bamc_search () from /lib64/libdb-5.3.so
> #8 0x00007fc6adc8b074 in __bamc_get () from /lib64/libdb-5.3.so
> #9 0x00007fc6add43ba3 in __dbc_iget () from /lib64/libdb-5.3.so
> #10 0x00007fc6add53092 in __dbc_get_pp () from /lib64/libdb-5.3.so
> #11 0x00007fc6a9672b70 in ldbm_back_seq () from /usr/lib64/dirsrv/plugins/libback-ldbm.so
> #12 0x00007fc6b54650f9 in seq_internal_callback_pb () from /usr/lib64/dirsrv/libslapd.so.0
> #13 0x00007fc6b54652a9 in slapi_seq_callback () from /usr/lib64/dirsrv/libslapd.so.0
> #14 0x00007fc6a7ec8919 in retrocl_update_lastchangenumber () from /usr/lib64/dirsrv/plugins/libretrocl-plugin.so
> #15 0x00007fc6a7ec89bb in retrocl_assign_changenumber () from /usr/lib64/dirsrv/plugins/libretrocl-plugin.so
> #16 0x00007fc6a7ec99b5 in retrocl_postob () from /usr/lib64/dirsrv/plugins/libretrocl-plugin.so
> #17 0x00007fc6b546156f in plugin_call_func () from /usr/lib64/dirsrv/libslapd.so.0
> #18 0x00007fc6b5461893 in plugin_call_plugins () from /usr/lib64/dirsrv/libslapd.so.0
> #19 0x00007fc6a965d231 in ldbm_back_modify () from /usr/lib64/dirsrv/plugins/libback-ldbm.so
> #20 0x00007fc6b544e905 in op_shared_modify () from /usr/lib64/dirsrv/libslapd.so.0
> #21 0x00007fc6b544fbe7 in do_modify () from /usr/lib64/dirsrv/libslapd.so.0
> #22 0x00007fc6b5932925 in connection_threadmain ()
> #23 0x00007fc6b3847cab in _pt_root () from /lib64/libnspr4.so
> #24 0x00007fc6b31e852a in start_thread () from /lib64/libpthread.so.0
> #25 0x00007fc6b2f2422d in clone () from /lib64/libc.so.6
> Thread 12 (Thread 0x7fc6787f0700 (LWP 1932)):
> #0 0x00007fc6b31ed590 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
> #1 0x00007fc6b3842440 in PR_WaitCondVar () from /lib64/libnspr4.so
> #2 0x00007fc6b5930b9e in connection_wait_for_new_work ()
> #3 0x00007fc6b5931dc9 in connection_threadmain ()
> #4 0x00007fc6b3847cab in _pt_root () from /lib64/libnspr4.so
> #5 0x00007fc6b31e852a in start_thread () from /lib64/libpthread.so.0
> #6 0x00007fc6b2f2422d in clone () from /lib64/libc.so.6
> Thread 11 (Thread 0x7fc677fef700 (LWP 1933)):
> #0 0x00007fc6b31ed590 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
> #1 0x00007fc6b3842440 in PR_WaitCondVar () from /lib64/libnspr4.so
> #2 0x00007fc6b5930b9e in connection_wait_for_new_work ()
> #3 0x00007fc6b5931dc9 in connection_threadmain ()
> #4 0x00007fc6b3847cab in _pt_root () from /lib64/libnspr4.so
> #5 0x00007fc6b31e852a in start_thread () from /lib64/libpthread.so.0
> #6 0x00007fc6b2f2422d in clone () from /lib64/libc.so.6
> Thread 10 (Thread 0x7fc6777ee700 (LWP 1934)):
> #0 0x00007fc6b31ed590 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
> #1 0x00007fc6b38426b3 in PR_EnterMonitor () from /lib64/libnspr4.so
> #2 0x00007fc6a9622456 in dblayer_txn_begin () from /usr/lib64/dirsrv/plugins/libback-ldbm.so
> #3 0x00007fc6a965d615 in ldbm_back_modify () from /usr/lib64/dirsrv/plugins/libback-ldbm.so
> #4 0x00007fc6b544e905 in op_shared_modify () from /usr/lib64/dirsrv/libslapd.so.0
> #5 0x00007fc6b544f387 in modify_internal_pb () from /usr/lib64/dirsrv/libslapd.so.0
> #6 0x00007fc6aad02bd3 in ipalockout_postop () from /usr/lib64/dirsrv/plugins/libipa_lockout.so
> #7 0x00007fc6b546156f in plugin_call_func () from /usr/lib64/dirsrv/libslapd.so.0
> #8 0x00007fc6b5461893 in plugin_call_plugins () from /usr/lib64/dirsrv/libslapd.so.0
> #9 0x00007fc6b592c22d in do_bind ()
> #10 0x00007fc6b5932974 in connection_threadmain ()
> #11 0x00007fc6b3847cab in _pt_root () from /lib64/libnspr4.so
> #12 0x00007fc6b31e852a in start_thread () from /lib64/libpthread.so.0
> #13 0x00007fc6b2f2422d in clone () from /lib64/libc.so.6
> Thread 9 (Thread 0x7fc676fed700 (LWP 1935)):
> #0 0x00007fc6b31ed590 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
> #1 0x00007fc6b38426b3 in PR_EnterMonitor () from /lib64/libnspr4.so
> #2 0x00007fc6a9622456 in dblayer_txn_begin () from /usr/lib64/dirsrv/plugins/libback-ldbm.so
> #3 0x00007fc6a965d615 in ldbm_back_modify () from /usr/lib64/dirsrv/plugins/libback-ldbm.so
> #4 0x00007fc6b544e905 in op_shared_modify () from /usr/lib64/dirsrv/libslapd.so.0
> #5 0x00007fc6b544f387 in modify_internal_pb () from /usr/lib64/dirsrv/libslapd.so.0
> #6 0x00007fc6aad02bd3 in ipalockout_postop () from /usr/lib64/dirsrv/plugins/libipa_lockout.so
> #7 0x00007fc6b546156f in plugin_call_func () from /usr/lib64/dirsrv/libslapd.so.0
> #8 0x00007fc6b5461893 in plugin_call_plugins () from /usr/lib64/dirsrv/libslapd.so.0
> #9 0x00007fc6b592c22d in do_bind ()
> #10 0x00007fc6b5932974 in connection_threadmain ()
> #11 0x00007fc6b3847cab in _pt_root () from /lib64/libnspr4.so
> #12 0x00007fc6b31e852a in start_thread () from /lib64/libpthread.so.0
> #13 0x00007fc6b2f2422d in clone () from /lib64/libc.so.6
> Thread 8 (Thread 0x7fc6767ec700 (LWP 1936)):
> #0 0x00007fc6b31ed590 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
> #1 0x00007fc6b3842440 in PR_WaitCondVar () from /lib64/libnspr4.so
> #2 0x00007fc6b5930b9e in connection_wait_for_new_work ()
> #3 0x00007fc6b5931dc9 in connection_threadmain ()
> #4 0x00007fc6b3847cab in _pt_root () from /lib64/libnspr4.so
> #5 0x00007fc6b31e852a in start_thread () from /lib64/libpthread.so.0
> #6 0x00007fc6b2f2422d in clone () from /lib64/libc.so.6
> Thread 7 (Thread 0x7fc675feb700 (LWP 1937)):
> #0 0x00007fc6b31ed590 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
> #1 0x00007fc6adc811fb in __db_hybrid_mutex_suspend () from /lib64/libdb-5.3.so
> #2 0x00007fc6adc8059b in __db_tas_mutex_lock () from /lib64/libdb-5.3.so
> #3 0x00007fc6add2aa91 in __lock_get_internal () from /lib64/libdb-5.3.so
> #4 0x00007fc6add2b657 in __lock_get () from /lib64/libdb-5.3.so
> #5 0x00007fc6add576af in __db_lget () from /lib64/libdb-5.3.so
> #6 0x00007fc6adc9d5d9 in __bam_get_root () from /lib64/libdb-5.3.so
> #7 0x00007fc6adc9d91b in __bam_search () from /lib64/libdb-5.3.so
> #8 0x00007fc6adc89144 in __bamc_search () from /lib64/libdb-5.3.so
> #9 0x00007fc6adc8adff in __bamc_get () from /lib64/libdb-5.3.so
> #10 0x00007fc6add43ba3 in __dbc_iget () from /lib64/libdb-5.3.so
> #11 0x00007fc6add53092 in __dbc_get_pp () from /lib64/libdb-5.3.so
> #12 0x00007fc6a962e6de in idl_new_fetch () from /usr/lib64/dirsrv/plugins/libback-ldbm.so
> #13 0x00007fc6a963cc2b in index_read_ext_allids () from /usr/lib64/dirsrv/plugins/libback-ldbm.so
> #14 0x00007fc6a96272ed in keys2idl () from /usr/lib64/dirsrv/plugins/libback-ldbm.so
> #15 0x00007fc6a9627afe in ava_candidates.isra () from /usr/lib64/dirsrv/plugins/libback-ldbm.so
> #16 0x00007fc6a96280fa in filter_candidates_ext () from /usr/lib64/dirsrv/plugins/libback-ldbm.so
> #17 0x00007fc6a96290ce in list_candidates () from /usr/lib64/dirsrv/plugins/libback-ldbm.so
> #18 0x00007fc6a9628062 in filter_candidates_ext () from /usr/lib64/dirsrv/plugins/libback-ldbm.so
> #19 0x00007fc6a96631a7 in subtree_candidates () from /usr/lib64/dirsrv/plugins/libback-ldbm.so
> #20 0x00007fc6a9664811 in ldbm_back_search () from /usr/lib64/dirsrv/plugins/libback-ldbm.so
> #21 0x00007fc6b5455410 in op_shared_search () from /usr/lib64/dirsrv/libslapd.so.0
> #22 0x00007fc6b5943fc6 in do_search ()
> #23 0x00007fc6b5932b4c in connection_threadmain ()
> #24 0x00007fc6b3847cab in _pt_root () from /lib64/libnspr4.so
> #25 0x00007fc6b31e852a in start_thread () from /lib64/libpthread.so.0
> #26 0x00007fc6b2f2422d in clone () from /lib64/libc.so.6
> Thread 6 (Thread 0x7fc6757ea700 (LWP 1938)):
> #0 0x00007fc6b31ed590 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
> #1 0x00007fc6b3842440 in PR_WaitCondVar () from /lib64/libnspr4.so
> #2 0x00007fc6b5930b9e in connection_wait_for_new_work ()
> #3 0x00007fc6b5931dc9 in connection_threadmain ()
> #4 0x00007fc6b3847cab in _pt_root () from /lib64/libnspr4.so
> #5 0x00007fc6b31e852a in start_thread () from /lib64/libpthread.so.0
> #6 0x00007fc6b2f2422d in clone () from /lib64/libc.so.6
> Thread 5 (Thread 0x7fc674fe9700 (LWP 1939)):
> #0 0x00007fc6b31ed590 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
> #1 0x00007fc6b3842440 in PR_WaitCondVar () from /lib64/libnspr4.so
> #2 0x00007fc6b5930b9e in connection_wait_for_new_work ()
> #3 0x00007fc6b5931dc9 in connection_threadmain ()
> #4 0x00007fc6b3847cab in _pt_root () from /lib64/libnspr4.so
> #5 0x00007fc6b31e852a in start_thread () from /lib64/libpthread.so.0
> #6 0x00007fc6b2f2422d in clone () from /lib64/libc.so.6
> Thread 4 (Thread 0x7fc6747e8700 (LWP 1940)):
> #0 0x00007fc6b31ed590 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
> #1 0x00007fc6adc811fb in __db_hybrid_mutex_suspend () from /lib64/libdb-5.3.so
> #2 0x00007fc6adc8059b in __db_tas_mutex_lock () from /lib64/libdb-5.3.so
> #3 0x00007fc6add2aa91 in __lock_get_internal () from /lib64/libdb-5.3.so
> #4 0x00007fc6add2b657 in __lock_get () from /lib64/libdb-5.3.so
> #5 0x00007fc6add576af in __db_lget () from /lib64/libdb-5.3.so
> #6 0x00007fc6adc9d5d9 in __bam_get_root () from /lib64/libdb-5.3.so
> #7 0x00007fc6adc9d91b in __bam_search () from /lib64/libdb-5.3.so
> #8 0x00007fc6adc89144 in __bamc_search () from /lib64/libdb-5.3.so
> #9 0x00007fc6adc8adff in __bamc_get () from /lib64/libdb-5.3.so
> #10 0x00007fc6add43ba3 in __dbc_iget () from /lib64/libdb-5.3.so
> #11 0x00007fc6add53092 in __dbc_get_pp () from /lib64/libdb-5.3.so
> #12 0x00007fc6a962e6de in idl_new_fetch () from /usr/lib64/dirsrv/plugins/libback-ldbm.so
> #13 0x00007fc6a963cc2b in index_read_ext_allids () from /usr/lib64/dirsrv/plugins/libback-ldbm.so
> #14 0x00007fc6a96272ed in keys2idl () from /usr/lib64/dirsrv/plugins/libback-ldbm.so
> #15 0x00007fc6a9627afe in ava_candidates.isra () from /usr/lib64/dirsrv/plugins/libback-ldbm.so
> #16 0x00007fc6a96280fa in filter_candidates_ext () from /usr/lib64/dirsrv/plugins/libback-ldbm.so
> #17 0x00007fc6a96290ce in list_candidates () from /usr/lib64/dirsrv/plugins/libback-ldbm.so
> #18 0x00007fc6a9628062 in filter_candidates_ext () from /usr/lib64/dirsrv/plugins/libback-ldbm.so
> #19 0x00007fc6a96631a7 in subtree_candidates () from /usr/lib64/dirsrv/plugins/libback-ldbm.so
> #20 0x00007fc6a9664811 in ldbm_back_search () from /usr/lib64/dirsrv/plugins/libback-ldbm.so
> #21 0x00007fc6b5455410 in op_shared_search () from /usr/lib64/dirsrv/libslapd.so.0
> #22 0x00007fc6b5943fc6 in do_search ()
> #23 0x00007fc6b5932b4c in connection_threadmain ()
> #24 0x00007fc6b3847cab in _pt_root () from /lib64/libnspr4.so
> #25 0x00007fc6b31e852a in start_thread () from /lib64/libpthread.so.0
> #26 0x00007fc6b2f2422d in clone () from /lib64/libc.so.6
> Thread 3 (Thread 0x7fc673fe7700 (LWP 1941)):
> #0 0x00007fc6b31ed590 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
> #1 0x00007fc6b3842440 in PR_WaitCondVar () from /lib64/libnspr4.so
> #2 0x00007fc6b5930b9e in connection_wait_for_new_work ()
> #3 0x00007fc6b5931dc9 in connection_threadmain ()
> #4 0x00007fc6b3847cab in _pt_root () from /lib64/libnspr4.so
> #5 0x00007fc6b31e852a in start_thread () from /lib64/libpthread.so.0
> #6 0x00007fc6b2f2422d in clone () from /lib64/libc.so.6
> Thread 2 (Thread 0x7fc6737e6700 (LWP 1942)):
> #0 0x00007fc6b2f1aae3 in select () from /lib64/libc.so.6
> #1 0x00007fc6b5492a99 in DS_Sleep () from /usr/lib64/dirsrv/libslapd.so.0
> #2 0x00007fc6b5933855 in time_thread ()
> #3 0x00007fc6b3847cab in _pt_root () from /lib64/libnspr4.so
> #4 0x00007fc6b31e852a in start_thread () from /lib64/libpthread.so.0
> #5 0x00007fc6b2f2422d in clone () from /lib64/libc.so.6
> Thread 1 (Thread 0x7fc6b58fb800 (LWP 1863)):
> #0 0x00007fc6b2f18c8d in poll () from /lib64/libc.so.6
> #1 0x00007fc6b3843da8 in _pr_poll_with_poll () from /lib64/libnspr4.so
> #2 0x00007fc6b5936b11 in slapd_daemon ()
> #3 0x00007fc6b59294f4 in main ()
>
>
>>> This replica is on virtual machine (ganeti). We had problems with replication to vm, but after force-sync all was fine. On physical servers all works fine.
>> the messages indicate there could be many concurrent operations, because individual ops are not fast enough, could your VM have less/slower resources than the physical machines ?
> VMs have less rsources and slower than physical machines.
> VM: 4 cpu, hd via drbd on sas drives,
>
> physical: more cores, faster (ssd) drives
>
> Best regards,
> Lukasz Jaworski 'Ender'
>
>
>
>>> Lukasz Jaworski 'Ender'
>>>
>>> Wiadomość napisana przez Ludwig Krispenz <lkrispen at redhat.com> w dniu 6 maj 2015, o godz. 10:52:
>>>
>>>> Hi,
>>>>
>>>> there seem to be different issues,
>>>> - I don't know what the ipactl status is looking for when it generates the error message about no matching master,
>>>> but I don't think it is related to the retro changelog.
>>>>
>>>> - the retro changelog errors for adding and deleting
>>>> -- the add failures are about aborted transactions because a page cannot be accessed, this maybe caused by concurrent mods on different backends, which want to update teh shared retro cl database.
>>>> the changenumber reprted seems to be increasing, one error is about changenumber 44975, the next about 45577, so it looks like changes into the changelog are written and teh changenumber increases
>>>> -- i'm not sure about the delete errors, but normally trimming would go on after such an error message, the changenumber attempted to delete are increasing.
>>>> Could you verify which changes are in the changelog, and if these are changing:
>>>> ldapsearch -b "cn=changelog" dn
>>>>
>>>> On 05/06/2015 09:52 AM, Łukasz Jaworski wrote:
>>>>> Hi,
>>>>>
>>>>> One of our replica hanged up morning. Error log after dirsrv restart:
>>>>> [06/May/2015:09:28:15 +0200] - Retry count exceeded in delete
>>>>> [06/May/2015:09:28:15 +0200] DSRetroclPlugin - delete_changerecord: could not delete change record 38376 (rc: 51)
>>>>> [06/May/2015:09:28:15 +0200] - Operation error fetching Null DN (6368aeb7-f3c111e4-ae70ce39-9b469c1f), error -30993.
>>>>> [06/May/2015:09:28:15 +0200] - dn2entry_ext: Failed to get id for changenumber=44975,cn=changelog from entryrdn index (-30993)
>>>>> [06/May/2015:09:28:15 +0200] - Operation error fetching changenumber=44975,cn=changelog (null), error -30993.
>>>>> [06/May/2015:09:28:15 +0200] DSRetroclPlugin - replog: an error occured while adding change number 44975, dn = changenumber=44975,cn=changelog: Operations error.
>>>>> [06/May/2015:09:28:15 +0200] retrocl-plugin - retrocl_postob: operation failure [1]
>>>>> [06/May/2015:09:28:15 +0200] - ldbm_back_seq deadlock retry BAD 1601, err=0 BDB0062 Successful return: 0
>>>>> [06/May/2015:09:30:03 +0200] - ldbm_back_seq deadlock retry BAD 1601, err=0 BDB0062 Successful return: 0
>>>>> [06/May/2015:09:30:06 +0200] - Retry count exceeded in delete
>>>>> [06/May/2015:09:30:06 +0200] DSRetroclPlugin - delete_changerecord: could not delete change record 39297 (rc: 51)
>>>>>
>>>>> I did "re-initialize" from other replica.
>>>>>
>>>>> Now ipactl doesn't work. Shows: Configured hostname 'replica09.local' does not match any master server in LDAP. On lists replica09 is exists (twice)
>>>>>
>>>>> # ipactl status
>>>>> Failed to get list of services to probe status!
>>>>> Configured hostname 'replica09.local' does not match any master server in LDAP:
>>>>> replica01.local
>>>>> replica02.local
>>>>> replica03.local
>>>>> replica04.local
>>>>> replica05.local
>>>>> replica06.local
>>>>> replica07.local
>>>>> replica08.local
>>>>> replica09.local
>>>>> replica10.local
>>>>> replica09.local
>>>>>
>>>>> After dirsrv stop/start:
>>>>>
>>>>> In error logs there are many:
>>>>> [06/May/2015:09:50:30 +0200] DSRetroclPlugin - delete_changerecord: could not delete change record 39290 (rc: 32)
>>>>> [06/May/2015:09:50:30 +0200] DSRetroclPlugin - delete_changerecord: could not delete change record 39291 (rc: 32)
>>>>> [06/May/2015:09:50:30 +0200] DSRetroclPlugin - delete_changerecord: could not delete change record 39292 (rc: 32)
>>>>> [06/May/2015:09:50:30 +0200] DSRetroclPlugin - delete_changerecord: could not delete change record 39293 (rc: 32)
>>>>> [06/May/2015:09:50:30 +0200] DSRetroclPlugin - delete_changerecord: could not delete change record 39294 (rc: 32)
>>>>> [06/May/2015:09:50:30 +0200] DSRetroclPlugin - delete_changerecord: could not delete change record 39295 (rc: 32)
>>>>> [06/May/2015:09:50:30 +0200] DSRetroclPlugin - delete_changerecord: could not delete change record 39296 (rc: 32)
>>>>> etc.
>>>>>
>>>>> [06/May/2015:09:51:08 +0200] - Operation error fetching Null DN (9f51430a-f3c411e4-927ece39-9b469c1f), error -30993.
>>>>> [06/May/2015:09:51:08 +0200] - dn2entry_ext: Failed to get id for changenumber=45577,cn=changelog from entryrdn index (-30993)
>>>>> [06/May/2015:09:51:08 +0200] - Operation error fetching changenumber=45577,cn=changelog (null), error -30993.
>>>>> [06/May/2015:09:51:08 +0200] DSRetroclPlugin - replog: an error occured while adding change number 45577, dn = changenumber=45577,cn=changelog: Operations error.
>>>>> [06/May/2015:09:51:08 +0200] retrocl-plugin - retrocl_postob: operation failure [1]
>>>>> [06/May/2015:09:51:08 +0200] - ldbm_back_seq deadlock retry BAD 1601, err=0 BDB0062 Successful return: 0
>>>>>
>>>>> Packages:
>>>>> freeipa-server-4.1.3-2.fc21.x86_64
>>>>> 389-ds-base-1.3.3.8-1.fc21.x86_64
>>>>> 389-ds-base-libs-1.3.3.8-1.fc21.x86_64
>>>>>
>>>>> Best regards,
>>>>> Ender
>>>>>
>>>> --
>>>> Manage your subscription for the Freeipa-users mailing list:
>>>> https://www.redhat.com/mailman/listinfo/freeipa-users
>>>> Go to http://freeipa.org for more info on the project
>
More information about the Freeipa-users
mailing list