[Freeipa-users] Problem with replication
Dmitri Pal
dpal at redhat.com
Thu May 7 00:05:10 UTC 2015
On 05/06/2015 07:29 AM, thierry bordaz wrote:
> This is looking like a deadlock between thread 14 and thread 13 where
> both threads acquired lock in the opposite order.
> Thread 14 is processing a betxn_postop but thread 13 is doing a
> postop. I wonder if the problem is that the test/set of the
> changenumber is done outside of the backend lock.
Is this enough info to open a ticket?
>
>
> Thread 14 holds some db page locks and tries to acquire retrocl plugin
> lock (retrocl_internal_lock)
> 800000f6 dd= 6 locks held 108 write locks 57 pid/thread
> 1863/140490418628352 flags 0 priority 100
> 800000f6 WRITE 1 HELD userRoot/id2entry.db page 2495
> 800000f6 READ 1 HELD userRoot/id2entry.db page 2495
> 800000f6 WRITE 3 HELD changelog/objectclass.db page 1
> 800000f6 WRITE 1 HELD changelog/changenumber.db
> page 68
>
> Thread 14 (Thread 0x7fc6797f2700 (LWP 1930)):
> #0 0x00007fc6b31eff1d in __lll_lock_wait () from /lib64/libpthread.so.0
> #1 0x00007fc6b31ea9be in pthread_mutex_lock () from
> /lib64/libpthread.so.0
> #2 0x00007fc6b38420b9 in PR_Lock () from /lib64/libnspr4.so
> #3 0x00007fc6a7ec99b0 in retrocl_postob () from
> /usr/lib64/dirsrv/plugins/libretrocl-plugin.so
> #4 0x00007fc6b546156f in plugin_call_func () from
> /usr/lib64/dirsrv/libslapd.so.0
> #5 0x00007fc6b5461893 in plugin_call_plugins () from
> /usr/lib64/dirsrv/libslapd.so.0
> #6 0x00007fc6a965d231 in ldbm_back_modify () from
> /usr/lib64/dirsrv/plugins/libback-ldbm.so
> #7 0x00007fc6b544e905 in op_shared_modify () from
> /usr/lib64/dirsrv/libslapd.so.0
> #8 0x00007fc6b544f387 in modify_internal_pb () from
> /usr/lib64/dirsrv/libslapd.so.0
> #9 0x00007fc6a8d354a2 in memberof_fix_memberof_callback () from
> /usr/lib64/dirsrv/plugins/libmemberof-plugin.so
> #10 0x00007fc6a8d35f65 in memberof_modop_one_replace_r () from
> /usr/lib64/dirsrv/plugins/libmemberof-plugin.so
> #11 0x00007fc6a8d362a6 in memberof_mod_smod_list () from
> /usr/lib64/dirsrv/plugins/libmemberof-plugin.so
> #12 0x00007fc6a8d3758d in memberof_postop_modify () from
> /usr/lib64/dirsrv/plugins/libmemberof-plugin.so
> #13 0x00007fc6b546156f in plugin_call_func () from
> /usr/lib64/dirsrv/libslapd.so.0
> #14 0x00007fc6b5461893 in plugin_call_plugins () from
> /usr/lib64/dirsrv/libslapd.so.0
> #15 0x00007fc6a965d231 in ldbm_back_modify () from
> /usr/lib64/dirsrv/plugins/libback-ldbm.so
> #16 0x00007fc6b544e905 in op_shared_modify () from
> /usr/lib64/dirsrv/libslapd.so.0
> #17 0x00007fc6b544fbe7 in do_modify () from
> /usr/lib64/dirsrv/libslapd.so.0
> #18 0x00007fc6b5932925 in connection_threadmain ()
> #19 0x00007fc6b3847cab in _pt_root () from /lib64/libnspr4.so
> #20 0x00007fc6b31e852a in start_thread () from /lib64/libpthread.so.0
> #21 0x00007fc6b2f2422d in clone () from /lib64/libc.so.6
>
> While thread 13 which hold the retrocl plugin lock, tries to acquire
> a page lock (held by thread 14)
>
> Thread 13 (Thread 0x7fc678ff1700 (LWP 1931)):
> 65 READ 1 WAIT changelog/changenumber.db page 68
>
> Thread 13 (Thread 0x7fc678ff1700 (LWP 1931)):
> #0 0x00007fc6b31ed590 inpthread_cond_wait@@GLIBC_2.3.2 () from
> /lib64/libpthread.so.0
> #1 0x00007fc6adc811fb in __db_hybrid_mutex_suspend () from
> /lib64/libdb-5.3.so
> #2 0x00007fc6adc8059b in __db_tas_mutex_lock () from /lib64/libdb-5.3.so
> #3 0x00007fc6add2aa91 in __lock_get_internal () from /lib64/libdb-5.3.so
> #4 0x00007fc6add2b657 in __lock_get () from /lib64/libdb-5.3.so
> #5 0x00007fc6add576af in __db_lget () from /lib64/libdb-5.3.so
> #6 0x00007fc6adc9e4f0 in __bam_search () from /lib64/libdb-5.3.so
> #7 0x00007fc6adc89144 in __bamc_search () from /lib64/libdb-5.3.so
> #8 0x00007fc6adc8b074 in __bamc_get () from /lib64/libdb-5.3.so
> #9 0x00007fc6add43ba3 in __dbc_iget () from /lib64/libdb-5.3.so
> #10 0x00007fc6add53092 in __dbc_get_pp () from /lib64/libdb-5.3.so
> #11 0x00007fc6a9672b70 in ldbm_back_seq () from
> /usr/lib64/dirsrv/plugins/libback-ldbm.so
> #12 0x00007fc6b54650f9 in seq_internal_callback_pb () from
> /usr/lib64/dirsrv/libslapd.so.0
> #13 0x00007fc6b54652a9 in slapi_seq_callback () from
> /usr/lib64/dirsrv/libslapd.so.0
> #14 0x00007fc6a7ec8919 in retrocl_update_lastchangenumber () from
> /usr/lib64/dirsrv/plugins/libretrocl-plugin.so
> #15 0x00007fc6a7ec89bb in retrocl_assign_changenumber () from
> /usr/lib64/dirsrv/plugins/libretrocl-plugin.so
> #16 0x00007fc6a7ec99b5 in retrocl_postob () from
> /usr/lib64/dirsrv/plugins/libretrocl-plugin.so
> #17 0x00007fc6b546156f in plugin_call_func () from
> /usr/lib64/dirsrv/libslapd.so.0
> #18 0x00007fc6b5461893 in plugin_call_plugins () from
> /usr/lib64/dirsrv/libslapd.so.0
> #19 0x00007fc6a965d231 in ldbm_back_modify () from
> /usr/lib64/dirsrv/plugins/libback-ldbm.so
> #20 0x00007fc6b544e905 in op_shared_modify () from
> /usr/lib64/dirsrv/libslapd.so.0
> #21 0x00007fc6b544fbe7 in do_modify () from
> /usr/lib64/dirsrv/libslapd.so.0
> #22 0x00007fc6b5932925 in connection_threadmain ()
> #23 0x00007fc6b3847cab in _pt_root () from /lib64/libnspr4.so
> #24 0x00007fc6b31e852a in start_thread () from /lib64/libpthread.so.0
> #25 0x00007fc6b2f2422d in clone () from /lib64/libc.so.6
>
> Many others threads are waiting for others pages held by thread 14:
>
>
> Thread 4 (Thread 0x7fc6747e8700 (LWP 1940)):
> Thread 7 (Thread 0x7fc675feb700 (LWP 1937)):
> Thread 30 (Thread 0x7fc691922700 (LWP 1914)):
> Thread 28 (Thread 0x7fc690920700 (LWP 1916)):
> Thread 29 (Thread 0x7fc691121700 (LWP 1915)):
> 49 READ 1 WAIT changelog/objectclass.db page 1
>
> Thread 13 (Thread 0x7fc678ff1700 (LWP 1931)):
> 65 READ 1 WAIT changelog/changenumber.db page 68
>
> Thread 26 (Thread 0x7fc67f7fe700 (LWP 1918)):
> 2 READ 1 WAIT userRoot/id2entry.db page 2495
>
>
>
> On 05/06/2015 12:01 PM, Łukasz Jaworski wrote:
>> dbstat:
>>
>> MacBookPro-10DDB1EAF1CC-1522:~ ender$ cat FILE
>> Default locking region information:
>> 139 Last allocated locker ID
>> 0x7fffffff Current maximum unused locker ID
>> 9 Number of lock modes
>> 200 Initial number of locks allocated
>> 0 Initial number of lockers allocated
>> 200 Initial number of lock objects allocated
>> 10000 Maximum number of locks possible
>> 10000 Maximum number of lockers possible
>> 10000 Maximum number of lock objects possible
>> 312 Current number of locks allocated
>> 151 Current number of lockers allocated
>> 250 Current number of lock objects allocated
>> 40 Number of lock object partitions
>> 8191 Size of object hash table
>> 275 Number of current locks
>> 303 Maximum number of locks at any one time
>> 6 Maximum number of locks in any one bucket
>> 174 Maximum number of locks stolen by for an empty partition
>> 13 Maximum number of locks stolen for any one partition
>> 124 Number of current lockers
>> 124 Maximum number of lockers at any one time
>> 223 Number of current lock objects
>> 233 Maximum number of lock objects at any one time
>> 3 Maximum number of lock objects in any one bucket
>> 49 Maximum number of objects stolen by for an empty partition
>> 5 Maximum number of objects stolen for any one partition
>> 82905 Total number of locks requested
>> 82018 Total number of locks released
>> 0 Total number of locks upgraded
>> 68 Total number of locks downgraded
>> 8 Lock requests not available due to conflicts, for which we waited
>> 12 Lock requests not available due to conflicts, for which we did
>> not wait
>> 0 Number of deadlocks
>> 0 Lock timeout value
>> 0 Number of locks that have timed out
>> 0 Transaction timeout value
>> 0 Number of transactions that have timed out
>> 2MB 304KB Region size
>> 0 The number of partition locks that required waiting (0%)
>> 0 The maximum number of times any partition lock was waited for (0%)
>> 0 The number of object queue operations that required waiting (0%)
>> 0 The number of locker allocations that required waiting (0%)
>> 4 The number of region locks that required waiting (0%)
>> 5 Maximum hash bucket length
>> =-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
>> Lock REGINFO information:
>> Environment Region type
>> 1 Region ID
>> /var/lib/dirsrv/slapd-xxxx/db/__db.001 Region name
>> 0x7fd376ff3000 Region address
>> 0x7fd376ff30a0 Region allocation head
>> 0x7fd376ffb2b0 Region primary address
>> 0 Region maximum allocation
>> 0 Region allocated
>> Region allocations: 796 allocations, 0 failures, 539 frees, 3 longest
>> Allocations by power-of-two sizes:
>> 1KB 781
>> 2KB 3
>> 4KB 6
>> 8KB 3
>> 16KB 0
>> 32KB 1
>> 64KB 0
>> 128KB 0
>> 256KB 2
>> 512KB 0
>> 1024KB 1
>> REGION_SHARED Region flags
>> =-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
>> Lock region parameters:
>> 2 Lock region region mutex [4/3168 0% !Own] <wakeups 0/2>
>> 16381 locker table size
>> 8191 object table size
>> 34128 obj_off
>> 889656 locker_off
>> 0 need_dd
>> =-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
>> Lock conflict matrix:
>> =-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
>> Locks grouped by lockers:
>> Locker Mode Count Status ----------------- Object
>> ---------------
>> 1 dd=122 locks held 1 write locks 0 pid/thread
>> 1863/140491426347008 flags 10 priority 100
>> 1 READ 1 HELD userRoot/id2entry.db handle 0
>> 2 dd=121 locks held 0 write locks 0 pid/thread
>> 1863/140490519340800 flags 0 priority 100
>> 2 READ 1 WAIT userRoot/id2entry.db page 2495
>> 3 dd=120 locks held 1 write locks 0 pid/thread
>> 1863/140491426347008 flags 10 priority 100
>> 3 READ 1 HELD ipaca/id2entry.db handle 0
>> 4 dd=119 locks held 0 write locks 0 pid/thread
>> 1863/140491426347008 flags 0 priority 100
>> 5 dd=118 locks held 1 write locks 0 pid/thread
>> 1863/140491426347008 flags 10 priority 100
>> 5 READ 1 HELD ipaca/entryrdn.db handle 0
>> 6 dd=117 locks held 0 write locks 0 pid/thread
>> 1863/140491426347008 flags 0 priority 100
>> 7 dd=116 locks held 0 write locks 0 pid/thread
>> 1863/140491426347008 flags 0 priority 100
>> 8 dd=115 locks held 0 write locks 0 pid/thread
>> 1863/140491426347008 flags 0 priority 100
>> 9 dd=114 locks held 1 write locks 0 pid/thread
>> 1863/140491426347008 flags 10 priority 100
>> 9 READ 1 HELD ipaca/vlv#allcertspkitomcatindex.db
>> handle 0
>> d dd=113 locks held 1 write locks 0 pid/thread
>> 1863/140491426347008 flags 10 priority 100
>> d READ 1 HELD
>> ipaca/vlv#allnonrevokedcertspkitomcatindex.db handle 0
>> f dd=112 locks held 1 write locks 0 pid/thread
>> 1863/140491426347008 flags 10 priority 100
>> f READ 1 HELD
>> ipaca/vlv#allrevokedcertspkitomcatindex.db handle 0
>> 10 dd=111 locks held 1 write locks 0 pid/thread
>> 1863/140491426347008 flags 10 priority 100
>> 10 READ 1 HELD
>> ipaca/vlv#allrevokedcertsnotafterpkitomcatindex.db handle 0
>> 13 dd=110 locks held 1 write locks 0 pid/thread
>> 1863/140491426347008 flags 10 priority 100
>> 13 READ 1 HELD
>> ipaca/vlv#allrevokedorrevokedexpiredcertspkitomcatindex.db
>> handle 0
>> 14 dd=109 locks held 1 write locks 0 pid/thread
>> 1863/140491426347008 flags 10 priority 100
>> 14 READ 1 HELD
>> ipaca/vlv#allvalidcertspkitomcatindex.db handle 0
>> 15 dd=108 locks held 1 write locks 0 pid/thread
>> 1863/140491426347008 flags 10 priority 100
>> 15 READ 1 HELD
>> ipaca/vlv#allvalidcertsnotafterpkitomcatindex.db handle 0
>> 16 dd=107 locks held 1 write locks 0 pid/thread
>> 1863/140491426347008 flags 10 priority 100
>> 16 READ 1 HELD
>> ipaca/vlv#allvalidorrevokedcertspkitomcatindex.db handle 0
>> 17 dd=106 locks held 1 write locks 0 pid/thread
>> 1863/140491426347008 flags 10 priority 100
>> 17 READ 1 HELD ipaca/vlv#caallpkitomcatindex.db
>> handle 0
>> 1c dd=105 locks held 1 write locks 0 pid/thread
>> 1863/140491426347008 flags 10 priority 100
>> 1c READ 1 HELD ipaca/vlv#cacompletepkitomcatindex.db
>> handle 0
>> 1d dd=104 locks held 1 write locks 0 pid/thread
>> 1863/140491426347008 flags 10 priority 100
>> 1d READ 1 HELD
>> ipaca/vlv#cacompleteenrollmentpkitomcatindex.db handle 0
>> 1f dd=103 locks held 1 write locks 0 pid/thread
>> 1863/140491426347008 flags 10 priority 100
>> 1f READ 1 HELD
>> ipaca/vlv#cacompleterevocationpkitomcatindex.db handle 0
>> 20 dd=102 locks held 1 write locks 0 pid/thread
>> 1863/140491426347008 flags 10 priority 100
>> 20 READ 1 HELD
>> ipaca/vlv#caenrollmentpkitomcatindex.db handle 0
>> 25 dd=101 locks held 1 write locks 0 pid/thread
>> 1863/140491426347008 flags 10 priority 100
>> 25 READ 1 HELD ipaca/vlv#carejectedpkitomcatindex.db
>> handle 0
>> 26 dd=100 locks held 1 write locks 0 pid/thread
>> 1863/140491426347008 flags 10 priority 100
>> 26 READ 1 HELD
>> ipaca/vlv#carejectedenrollmentpkitomcatindex.db handle 0
>> 2a dd=99 locks held 1 write locks 0 pid/thread
>> 1863/140491426347008 flags 10 priority 100
>> 2a READ 1 HELD
>> ipaca/vlv#carevocationpkitomcatindex.db handle 0
>> 2b dd=98 locks held 1 write locks 0 pid/thread
>> 1863/140491426347008 flags 10 priority 100
>> 2b READ 1 HELD changelog/id2entry.db handle 0
>> 2c dd=97 locks held 0 write locks 0 pid/thread
>> 1863/140490468984576 flags 0 priority 100
>> 2d dd=96 locks held 1 write locks 0 pid/thread
>> 1863/140491426347008 flags 10 priority 100
>> 2d READ 1 HELD changelog/entryusn.db handle 0
>> 2e dd=95 locks held 0 write locks 0 pid/thread
>> 1863/140491426347008 flags 0 priority 100
>> 2f dd=94 locks held 1 write locks 0 pid/thread
>> 1863/140491426347008 flags 10 priority 100
>> 2f READ 1 HELD userRoot/entryusn.db handle 0
>> 30 dd=93 locks held 0 write locks 0 pid/thread
>> 1863/140491426347008 flags 0 priority 100
>> 31 dd=92 locks held 1 write locks 0 pid/thread
>> 1863/140491426347008 flags 10 priority 100
>> 31 READ 1 HELD ipaca/entryusn.db handle 0
>> 32 dd=91 locks held 0 write locks 0 pid/thread
>> 1863/140491426347008 flags 0 priority 100
>> 33 dd=90 locks held 1 write locks 0 pid/thread
>> 1863/140491426347008 flags 10 priority 100
>> 33 READ 1 HELD userRoot/entryrdn.db handle 0
>> 34 dd=89 locks held 0 write locks 0 pid/thread
>> 1863/140490839312128 flags 0 priority 100
>> 35 dd=88 locks held 0 write locks 0 pid/thread
>> 1863/140490510948096 flags 0 priority 100
>> 36 dd=87 locks held 0 write locks 0 pid/thread
>> 1863/140490519340800 flags 0 priority 100
>> 37 dd=86 locks held 1 write locks 0 pid/thread
>> 1863/140491426347008 flags 10 priority 100
>> 37 READ 1 HELD userRoot/objectclass.db
>> handle 0
>> 38 dd=85 locks held 0 write locks 0 pid/thread
>> 1863/140490351486720 flags 0 priority 100
>> 39 dd=84 locks held 1 write locks 0 pid/thread
>> 1863/140491426347008 flags 10 priority 100
>> 39 READ 1 HELD userRoot/ancestorid.db handle 0
>> 3a dd=83 locks held 0 write locks 0 pid/thread
>> 1863/140491426347008 flags 0 priority 100
>> 3b dd=82 locks held 1 write locks 0 pid/thread
>> 1863/140491426347008 flags 10 priority 100
>> 3b READ 1 HELD userRoot/macAddress.db handle 0
>> 3c dd=81 locks held 0 write locks 0 pid/thread
>> 1863/140491426347008 flags 0 priority 100
>> 3d dd=80 locks held 0 write locks 0 pid/thread
>> 1863/140490435413760 flags 0 priority 100
>> 3e dd=79 locks held 0 write locks 0 pid/thread
>> 1863/140490343094016 flags 0 priority 100
>> 3f dd=78 locks held 0 write locks 0 pid/thread
>> 1863/140491426347008 flags 0 priority 100
>> 40 dd=77 locks held 0 write locks 0 pid/thread
>> 1863/140490839312128 flags 0 priority 100
>> 41 dd=76 locks held 0 write locks 0 pid/thread
>> 1863/140491426347008 flags 0 priority 100
>> 42 dd=75 locks held 0 write locks 0 pid/thread
>> 1863/140490510948096 flags 0 priority 100
>> 43 dd=74 locks held 0 write locks 0 pid/thread
>> 1863/140491426347008 flags 0 priority 100
>> 44 dd=73 locks held 1 write locks 0 pid/thread
>> 1863/140491426347008 flags 10 priority 100
>> 44 READ 1 HELD changelog/entryrdn.db handle 0
>> 45 dd=72 locks held 0 write locks 0 pid/thread
>> 1863/140490494162688 flags 0 priority 100
>> 46 dd=71 locks held 0 write locks 0 pid/thread
>> 1863/140490468984576 flags 0 priority 100
>> 47 dd=70 locks held 0 write locks 0 pid/thread
>> 1863/140491426347008 flags 0 priority 100
>> 48 dd=69 locks held 1 write locks 0 pid/thread
>> 1863/140491426347008 flags 10 priority 100
>> 48 READ 1 HELD changelog/objectclass.db
>> handle 0
>> 49 dd=68 locks held 0 write locks 0 pid/thread
>> 1863/140490334701312 flags 0 priority 100
>> 49 READ 1 WAIT changelog/objectclass.db
>> page 1
>> 4a dd=67 locks held 0 write locks 0 pid/thread
>> 1863/140491066709760 flags 0 priority 100
>> 4b dd=66 locks held 1 write locks 0 pid/thread
>> 1863/140491426347008 flags 10 priority 100
>> 4b READ 1 HELD ipaca/objectclass.db handle 0
>> 4c dd=65 locks held 0 write locks 0 pid/thread
>> 1863/140490468984576 flags 0 priority 100
>> 4d dd=64 locks held 0 write locks 0 pid/thread
>> 1863/140491426347008 flags 0 priority 100
>> 4e dd=63 locks held 1 write locks 0 pid/thread
>> 1863/140491426347008 flags 10 priority 100
>> 4e READ 1 HELD changelog/aci.db handle 0
>> 4f dd=62 locks held 0 write locks 0 pid/thread
>> 1863/140491426347008 flags 0 priority 100
>> 50 dd=61 locks held 1 write locks 0 pid/thread
>> 1863/140491426347008 flags 10 priority 100
>> 50 READ 1 HELD userRoot/aci.db handle 0
>> 51 dd=60 locks held 0 write locks 0 pid/thread
>> 1863/140491426347008 flags 0 priority 100
>> 52 dd=59 locks held 1 write locks 0 pid/thread
>> 1863/140491426347008 flags 10 priority 100
>> 52 READ 1 HELD ipaca/aci.db handle 0
>> 53 dd=58 locks held 0 write locks 0 pid/thread
>> 1863/140491426347008 flags 0 priority 100
>> 54 dd=57 locks held 1 write locks 0 pid/thread
>> 1863/140491426347008 flags 10 priority 100
>> 54 READ 1 HELD userRoot/parentid.db handle 0
>> 55 dd=56 locks held 0 write locks 0 pid/thread
>> 1863/140491426347008 flags 0 priority 100
>> 56 dd=55 locks held 0 write locks 0 pid/thread
>> 1863/140490468984576 flags 0 priority 100
>> 57 dd=54 locks held 1 write locks 0 pid/thread
>> 1863/140491426347008 flags 10 priority 100
>> 57 READ 1 HELD userRoot/nsuniqueid.db handle 0
>> 58 dd=53 locks held 0 write locks 0 pid/thread
>> 1863/140490494162688 flags 0 priority 100
>> 59 dd=52 locks held 1 write locks 0 pid/thread
>> 1863/140491426347008 flags 10 priority 100
>> 59 READ 1 HELD ipaca/nsuniqueid.db handle 0
>> 5a dd=51 locks held 0 write locks 0 pid/thread
>> 1863/140491426347008 flags 0 priority 100
>> 5b dd=50 locks held 1 write locks 0 pid/thread
>> 1863/140491426347008 flags 10 priority 100
>> 5b READ 1 HELD
>> /var/lib/dirsrv/slapd-xxxx/cldb/9890a88b-d15511e4-8d35ce39-9b469c1f_550feacc000000040000.db
>> handle 0
>> 5c dd=49 locks held 0 write locks 0 pid/thread
>> 1863/140491426347008 flags 0 priority 100
>> 5d dd=48 locks held 0 write locks 0 pid/thread
>> 1863/140491426347008 flags 0 priority 100
>> 5e dd=47 locks held 1 write locks 0 pid/thread
>> 1863/140491426347008 flags 10 priority 100
>> 5e READ 1 HELD
>> /var/lib/dirsrv/slapd-xxxx/cldb/f3c29b1e-d15511e4-8d35ce39-9b469c1f_550feb15000000600000.db
>> handle 0
>> 5f dd=46 locks held 0 write locks 0 pid/thread
>> 1863/140491104614144 flags 0 priority 100
>> 60 dd=45 locks held 0 write locks 0 pid/thread
>> 1863/140491104614144 flags 0 priority 100
>> 61 dd=44 locks held 0 write locks 0 pid/thread
>> 1863/140491426347008 flags 0 priority 100
>> 62 dd=43 locks held 0 write locks 0 pid/thread
>> 1863/140490468984576 flags 0 priority 100
>> 63 dd=42 locks held 1 write locks 0 pid/thread
>> 1863/140491426347008 flags 10 priority 100
>> 63 READ 1 HELD changelog/nsuniqueid.db
>> handle 0
>> 64 dd=41 locks held 1 write locks 0 pid/thread
>> 1863/140491426347008 flags 10 priority 100
>> 64 READ 1 HELD changelog/changenumber.db
>> handle 0
>> 65 dd=40 locks held 0 write locks 0 pid/thread
>> 1863/140490410235648 flags 0 priority 100
>> 65 READ 1 WAIT changelog/changenumber.db
>> page 68
>> 66 dd=39 locks held 0 write locks 0 pid/thread
>> 1863/140490854885120 flags 0 priority 100
>> 67 dd=38 locks held 1 write locks 0 pid/thread
>> 1863/140491066709760 flags 10 priority 100
>> 67 READ 1 HELD changelog/targetuniqueid.db
>> handle 0
>> 68 dd=37 locks held 1 write locks 0 pid/thread
>> 1863/140491066709760 flags 10 priority 100
>> 68 READ 1 HELD changelog/parentid.db handle 0
>> 69 dd=36 locks held 1 write locks 0 pid/thread
>> 1863/140491066709760 flags 10 priority 100
>> 69 READ 1 HELD changelog/ancestorid.db
>> handle 0
>> 6a dd=35 locks held 1 write locks 0 pid/thread
>> 1863/140491066709760 flags 10 priority 100
>> 6a READ 1 HELD changelog/numsubordinates.db
>> handle 0
>> 6b dd=34 locks held 0 write locks 0 pid/thread
>> 1863/140490359879424 flags 0 priority 100
>> 6b READ 1 WAIT changelog/objectclass.db
>> page 1
>> 6c dd=33 locks held 1 write locks 0 pid/thread
>> 1863/140490854885120 flags 10 priority 100
>> 6c READ 1 HELD userRoot/nsTombstoneCSN.db
>> handle 0
>> 6d dd=32 locks held 1 write locks 0 pid/thread
>> 1863/140490854885120 flags 10 priority 100
>> 6d READ 1 HELD userRoot/nscpEntryDN.db
>> handle 0
>> 6e dd=31 locks held 1 write locks 0 pid/thread
>> 1863/140490854885120 flags 10 priority 100
>> 6e READ 1 HELD userRoot/numsubordinates.db
>> handle 0
>> 6f dd=30 locks held 1 write locks 0 pid/thread
>> 1863/140490854885120 flags 10 priority 100
>> 6f READ 1 HELD userRoot/member.db handle 0
>> 70 dd=29 locks held 1 write locks 0 pid/thread
>> 1863/140490854885120 flags 10 priority 100
>> 70 READ 1 HELD userRoot/uniquemember.db
>> handle 0
>> 71 dd=28 locks held 1 write locks 0 pid/thread
>> 1863/140490854885120 flags 10 priority 100
>> 71 READ 1 HELD userRoot/owner.db handle 0
>> 72 dd=27 locks held 1 write locks 0 pid/thread
>> 1863/140490854885120 flags 10 priority 100
>> 72 READ 1 HELD userRoot/seeAlso.db handle 0
>> 73 dd=26 locks held 1 write locks 0 pid/thread
>> 1863/140490854885120 flags 10 priority 100
>> 73 READ 1 HELD userRoot/manager.db handle 0
>> 74 dd=25 locks held 1 write locks 0 pid/thread
>> 1863/140490854885120 flags 10 priority 100
>> 74 READ 1 HELD userRoot/secretary.db handle 0
>> 75 dd=24 locks held 1 write locks 0 pid/thread
>> 1863/140490854885120 flags 10 priority 100
>> 75 READ 1 HELD userRoot/memberUser.db handle 0
>> 76 dd=23 locks held 1 write locks 0 pid/thread
>> 1863/140490854885120 flags 10 priority 100
>> 76 READ 1 HELD userRoot/memberHost.db handle 0
>> 77 dd=22 locks held 1 write locks 0 pid/thread
>> 1863/140490854885120 flags 10 priority 100
>> 77 READ 1 HELD userRoot/sourcehost.db handle 0
>> 78 dd=21 locks held 1 write locks 0 pid/thread
>> 1863/140490854885120 flags 10 priority 100
>> 78 READ 1 HELD userRoot/memberservice.db
>> handle 0
>> 79 dd=20 locks held 1 write locks 0 pid/thread
>> 1863/140490854885120 flags 10 priority 100
>> 79 READ 1 HELD userRoot/managedby.db handle 0
>> 7a dd=19 locks held 1 write locks 0 pid/thread
>> 1863/140490854885120 flags 10 priority 100
>> 7a READ 1 HELD userRoot/memberallowcmd.db
>> handle 0
>> 7b dd=18 locks held 1 write locks 0 pid/thread
>> 1863/140490854885120 flags 10 priority 100
>> 7b READ 1 HELD userRoot/memberdenycmd.db
>> handle 0
>> 7c dd=17 locks held 1 write locks 0 pid/thread
>> 1863/140490854885120 flags 10 priority 100
>> 7c READ 1 HELD userRoot/ipasudorunas.db
>> handle 0
>> 7d dd=16 locks held 1 write locks 0 pid/thread
>> 1863/140490854885120 flags 10 priority 100
>> 7d READ 1 HELD userRoot/ipasudorunasgroup.db
>> handle 0
>> 7e dd=15 locks held 1 write locks 0 pid/thread
>> 1863/140490854885120 flags 10 priority 100
>> 7e READ 1 HELD userRoot/ipatokenradiusconfiglink.db
>> handle 0
>> 7f dd=14 locks held 1 write locks 0 pid/thread
>> 1863/140490854885120 flags 10 priority 100
>> 7f READ 1 HELD userRoot/ipaassignedidview.db
>> handle 0
>> 80 dd=13 locks held 0 write locks 0 pid/thread
>> 1863/140490494162688 flags 0 priority 100
>> 81 dd=12 locks held 0 write locks 0 pid/thread
>> 1863/140490468984576 flags 0 priority 100
>> 82 dd=11 locks held 1 write locks 0 pid/thread
>> 1863/140490510948096 flags 10 priority 100
>> 82 READ 1 HELD userRoot/krbPrincipalName.db
>> handle 0
>> 83 dd=10 locks held 0 write locks 0 pid/thread
>> 1863/140490435413760 flags 0 priority 100
>> 84 dd= 9 locks held 0 write locks 0 pid/thread
>> 1863/140490510948096 flags 0 priority 100
>> 85 dd= 8 locks held 1 write locks 0 pid/thread
>> 1863/140490452199168 flags 10 priority 100
>> 85 READ 1 HELD ipaca/requeststate.db handle 0
>> 86 dd= 7 locks held 1 write locks 0 pid/thread
>> 1863/140490452199168 flags 10 priority 100
>> 86 READ 1 HELD ipaca/requesttype.db handle 0
>> 87 dd= 4 locks held 1 write locks 0 pid/thread
>> 1863/140490418628352 flags 10 priority 100
>> 87 READ 1 HELD userRoot/memberOf.db handle 0
>> 88 dd= 3 locks held 0 write locks 0 pid/thread
>> 1863/140490822526720 flags 0 priority 100
>> 88 READ 1 WAIT changelog/objectclass.db
>> page 1
>> 89 dd= 2 locks held 0 write locks 0 pid/thread
>> 1863/140490805741312 flags 0 priority 100
>> 89 READ 1 WAIT changelog/objectclass.db
>> page 1
>> 8a dd= 1 locks held 0 write locks 0 pid/thread
>> 1863/140490839312128 flags 0 priority 100
>> 8b dd= 0 locks held 0 write locks 0 pid/thread
>> 1863/140490814134016 flags 0 priority 100
>> 8b READ 1 WAIT changelog/objectclass.db
>> page 1
>> 800000f6 dd= 6 locks held 108 write locks 57 pid/thread
>> 1863/140490418628352 flags 0 priority 100
>> 800000f6 READ 4 HELD userRoot/member.db page 356
>> 800000f6 READ 4 HELD userRoot/member.db page 360
>> 800000f6 READ 2 HELD userRoot/member.db page 403
>> 800000f6 READ 1 HELD userRoot/id2entry.db page 1722
>> 800000f6 READ 1 HELD userRoot/id2entry.db page 1714
>> 800000f6 READ 5 HELD userRoot/entryrdn.db page 255
>> 800000f6 READ 4 HELD userRoot/id2entry.db page 1712
>> 800000f6 READ 1 HELD userRoot/entryrdn.db page 35
>> 800000f6 READ 1 HELD userRoot/id2entry.db page 1679
>> 800000f6 READ 1 HELD userRoot/id2entry.db page 1730
>> 800000f6 READ 4 HELD userRoot/member.db page 405
>> 800000f6 READ 3 HELD userRoot/entryrdn.db page 262
>> 800000f6 READ 1 HELD userRoot/id2entry.db page 1726
>> 800000f6 READ 4 HELD userRoot/entryrdn.db page 249
>> 800000f6 READ 4 HELD userRoot/id2entry.db page 2905
>> 800000f6 READ 3 HELD userRoot/memberHost.db page 96
>> 800000f6 READ 10 HELD userRoot/memberUser.db page 166
>> 800000f6 READ 15 HELD userRoot/member.db page 70
>> 800000f6 READ 4 HELD userRoot/entryrdn.db page 229
>> 800000f6 READ 4 HELD userRoot/id2entry.db page 1663
>> 800000f6 READ 3 HELD userRoot/memberHost.db page 209
>> 800000f6 READ 6 HELD userRoot/memberUser.db page 14
>> 800000f6 READ 3 HELD userRoot/member.db page 26
>> 800000f6 READ 1 HELD userRoot/ancestorid.db page 2
>> 800000f6 READ 5 HELD userRoot/memberHost.db page 87
>> 800000f6 READ 10 HELD userRoot/member.db page 50
>> 800000f6 READ 1 HELD userRoot/memberHost.db page 28
>> 800000f6 READ 1 HELD userRoot/memberUser.db page 92
>> 800000f6 READ 1 HELD userRoot/member.db page 43
>> 800000f6 READ 2 HELD userRoot/memberUser.db page 103
>> 800000f6 READ 2 HELD userRoot/member.db page 337
>> 800000f6 READ 2 HELD userRoot/memberallowcmd.db
>> page 44
>> 800000f6 READ 2 HELD userRoot/memberdenycmd.db
>> page 1
>> 800000f6 READ 2 HELD userRoot/ipasudorunas.db
>> page 1
>> 800000f6 READ 2 HELD userRoot/ipasudorunasgroup.db
>> page 1
>> 800000f6 READ 2 HELD userRoot/objectclass.db page 10
>> 800000f6 READ 15 HELD userRoot/member.db page 406
>> 800000f6 READ 61 HELD userRoot/objectclass.db page 61
>> 800000f6 READ 36 HELD userRoot/memberUser.db page 81
>> 800000f6 READ 35 HELD userRoot/memberHost.db page 195
>> 800000f6 READ 2 HELD userRoot/objectclass.db page 50
>> 800000f6 READ 1 HELD changelog/nsuniqueid.db page 191
>> 800000f6 READ 3 HELD changelog/entryrdn.db page 387
>> 800000f6 READ 2 HELD changelog/entryrdn.db page 388
>> 800000f6 WRITE 1 HELD changelog/id2entry.db page 0
>> 800000f6 WRITE 1 HELD changelog/id2entry.db page 7353
>> 800000f6 WRITE 1 HELD changelog/id2entry.db page 5573
>> 800000f6 WRITE 2 HELD changelog/id2entry.db page 5564
>> 800000f6 WRITE 3 HELD changelog/objectclass.db
>> page 1
>> 800000f6 WRITE 1 HELD changelog/targetuniqueid.db
>> page 47
>> 800000f6 WRITE 1 HELD changelog/changenumber.db
>> page 68
>> 800000f6 WRITE 1 HELD changelog/nsuniqueid.db page 191
>> 800000f6 WRITE 1 HELD changelog/parentid.db page 1
>> 800000f6 WRITE 1 HELD changelog/entryusn.db page 68
>> 800000f6 WRITE 1 HELD changelog/ancestorid.db page 1
>> 800000f6 WRITE 1 HELD changelog/entryrdn.db page 388
>> 800000f6 WRITE 1 HELD changelog/entryrdn.db page 917
>> 800000f6 WRITE 1 HELD changelog/entryrdn.db page 919
>> 800000f6 WRITE 1 HELD changelog/id2entry.db page 2
>> 800000f6 WRITE 1 HELD changelog/numsubordinates.db
>> page 1
>> 800000f6 WRITE 3 HELD userRoot/entryusn.db page 27
>> 800000f6 READ 1 HELD userRoot/entryusn.db page 27
>> 800000f6 WRITE 1 HELD userRoot/memberUser.db page 19
>> 800000f6 WRITE 1 HELD userRoot/memberUser.db page 82
>> 800000f6 WRITE 1 HELD userRoot/memberUser.db page 192
>> 800000f6 WRITE 1 HELD userRoot/memberUser.db page 23
>> 800000f6 WRITE 1 HELD userRoot/memberUser.db page 109
>> 800000f6 WRITE 1 HELD userRoot/memberUser.db page 22
>> 800000f6 WRITE 1 HELD userRoot/memberUser.db page 167
>> 800000f6 WRITE 1 HELD userRoot/memberUser.db page 12
>> 800000f6 WRITE 1 HELD userRoot/memberUser.db page 3
>> 800000f6 WRITE 1 HELD userRoot/memberUser.db page 20
>> 800000f6 WRITE 1 HELD userRoot/memberUser.db page 26
>> 800000f6 WRITE 1 HELD userRoot/memberUser.db page 35
>> 800000f6 WRITE 1 HELD userRoot/memberUser.db page 11
>> 800000f6 WRITE 1 HELD userRoot/memberUser.db page 15
>> 800000f6 WRITE 1 HELD userRoot/memberUser.db page 8
>> 800000f6 WRITE 1 HELD userRoot/memberUser.db page 25
>> 800000f6 WRITE 1 HELD userRoot/memberUser.db page 190
>> 800000f6 WRITE 1 HELD userRoot/memberUser.db page 193
>> 800000f6 WRITE 1 HELD userRoot/memberUser.db page 9
>> 800000f6 WRITE 1 HELD userRoot/memberUser.db page 189
>> 800000f6 WRITE 1 HELD userRoot/memberUser.db page 10
>> 800000f6 WRITE 1 HELD userRoot/memberUser.db page 104
>> 800000f6 WRITE 3 HELD userRoot/memberUser.db page 28
>> 800000f6 WRITE 2 HELD userRoot/memberUser.db page 2
>> 800000f6 WRITE 1 HELD userRoot/memberUser.db page 13
>> 800000f6 WRITE 1 HELD userRoot/memberUser.db page 74
>> 800000f6 WRITE 1 HELD userRoot/memberUser.db page 24
>> 800000f6 WRITE 1 HELD userRoot/memberUser.db page 205
>> 800000f6 WRITE 1 HELD userRoot/memberUser.db page 191
>> 800000f6 WRITE 3 HELD userRoot/memberUser.db page 5
>> 800000f6 WRITE 1 HELD userRoot/memberUser.db page 97
>> 800000f6 WRITE 1 HELD userRoot/memberUser.db page 84
>> 800000f6 WRITE 2 HELD userRoot/memberUser.db page 6
>> 800000f6 WRITE 1 HELD userRoot/memberUser.db page 164
>> 800000f6 WRITE 1 HELD userRoot/memberUser.db page 163
>> 800000f6 WRITE 2 HELD userRoot/memberUser.db page 34
>> 800000f6 WRITE 1 HELD userRoot/memberUser.db page 27
>> 800000f6 WRITE 1 HELD userRoot/memberUser.db page 103
>> 800000f6 WRITE 4 HELD userRoot/memberUser.db page 168
>> 800000f6 WRITE 1 HELD userRoot/id2entry.db page 2495
>> 800000f6 READ 18 HELD userRoot/entryrdn.db page 466
>> 800000f6 READ 18 HELD userRoot/entryrdn.db page 93
>> 800000f6 READ 18 HELD userRoot/entryrdn.db page 83
>> 800000f6 READ 1 HELD userRoot/entryrdn.db page 495
>> 800000f6 READ 1 HELD userRoot/id2entry.db page 2495
>> 800000f6 READ 2 HELD userRoot/nsuniqueid.db page 29
>> 80000106 dd= 5 locks held 17 write locks 13 pid/thread
>> 1863/140490410235648 flags 0 priority 100
>> 80000106 WRITE 2 HELD ipaca/vlv#caenrollmentpkitomcatindex.db
>> page 1
>> 80000106 WRITE 4 HELD ipaca/vlv#caenrollmentpkitomcatindex.db
>> page 5
>> 80000106 WRITE 2 HELD
>> ipaca/vlv#cacompleteenrollmentpkitomcatindex.db page 1
>> 80000106 WRITE 4 HELD
>> ipaca/vlv#cacompleteenrollmentpkitomcatindex.db page 5
>> 80000106 WRITE 2 HELD ipaca/vlv#cacompletepkitomcatindex.db
>> page 1
>> 80000106 WRITE 4 HELD ipaca/vlv#cacompletepkitomcatindex.db
>> page 5
>> 80000106 WRITE 2 HELD ipaca/vlv#caallpkitomcatindex.db
>> page 1
>> 80000106 WRITE 4 HELD ipaca/vlv#caallpkitomcatindex.db
>> page 5
>> 80000106 WRITE 3 HELD ipaca/entryusn.db page 13
>> 80000106 READ 1 HELD ipaca/entryusn.db page 13
>> 80000106 WRITE 4 HELD ipaca/requesttype.db page 1
>> 80000106 READ 1 HELD ipaca/requesttype.db page 1
>> 80000106 WRITE 4 HELD ipaca/requeststate.db page 1
>> 80000106 READ 1 HELD ipaca/requeststate.db page 1
>> 80000106 WRITE 4 HELD ipaca/id2entry.db page 0
>> 80000106 WRITE 1 HELD ipaca/id2entry.db page 301
>> 80000106 READ 2 HELD ipaca/nsuniqueid.db page 21
>> 800001eb dd=4294967295 locks held 75 write locks 39 pid/thread
>> 1863/140490418628352 flags 0 priority 100
>> 800001eb WRITE 3 HELD userRoot/entryusn.db page 27
>> 800001eb READ 1 HELD userRoot/entryusn.db page 27
>> 800001eb WRITE 1 HELD userRoot/memberOf.db page 37
>> 800001eb WRITE 3 HELD userRoot/memberOf.db page 219
>> 800001eb READ 1 HELD userRoot/memberOf.db page 219
>> 800001eb WRITE 3 HELD userRoot/memberOf.db page 259
>> 800001eb READ 1 HELD userRoot/memberOf.db page 259
>> 800001eb WRITE 3 HELD userRoot/memberOf.db page 253
>> 800001eb READ 1 HELD userRoot/memberOf.db page 253
>> 800001eb WRITE 3 HELD userRoot/memberOf.db page 207
>> 800001eb READ 1 HELD userRoot/memberOf.db page 207
>> 800001eb WRITE 3 HELD userRoot/memberOf.db page 91
>> 800001eb READ 1 HELD userRoot/memberOf.db page 91
>> 800001eb WRITE 3 HELD userRoot/memberOf.db page 10
>> 800001eb READ 1 HELD userRoot/memberOf.db page 10
>> 800001eb WRITE 3 HELD userRoot/memberOf.db page 211
>> 800001eb READ 1 HELD userRoot/memberOf.db page 211
>> 800001eb WRITE 3 HELD userRoot/memberOf.db page 143
>> 800001eb READ 1 HELD userRoot/memberOf.db page 143
>> 800001eb WRITE 3 HELD userRoot/memberOf.db page 121
>> 800001eb READ 1 HELD userRoot/memberOf.db page 121
>> 800001eb WRITE 3 HELD userRoot/memberOf.db page 228
>> 800001eb READ 1 HELD userRoot/memberOf.db page 228
>> 800001eb WRITE 3 HELD userRoot/memberOf.db page 78
>> 800001eb READ 1 HELD userRoot/memberOf.db page 78
>> 800001eb WRITE 3 HELD userRoot/memberOf.db page 272
>> 800001eb READ 1 HELD userRoot/memberOf.db page 272
>> 800001eb WRITE 3 HELD userRoot/memberOf.db page 189
>> 800001eb READ 1 HELD userRoot/memberOf.db page 189
>> 800001eb WRITE 3 HELD userRoot/memberOf.db page 156
>> 800001eb READ 1 HELD userRoot/memberOf.db page 156
>> 800001eb WRITE 3 HELD userRoot/memberOf.db page 148
>> 800001eb READ 1 HELD userRoot/memberOf.db page 148
>> 800001eb WRITE 3 HELD userRoot/memberOf.db page 142
>> 800001eb READ 1 HELD userRoot/memberOf.db page 142
>> 800001eb WRITE 3 HELD userRoot/memberOf.db page 258
>> 800001eb READ 1 HELD userRoot/memberOf.db page 258
>> 800001eb WRITE 9 HELD userRoot/memberOf.db page 260
>> 800001eb READ 3 HELD userRoot/memberOf.db page 260
>> 800001eb WRITE 3 HELD userRoot/memberOf.db page 218
>> 800001eb READ 1 HELD userRoot/memberOf.db page 218
>> 800001eb WRITE 3 HELD userRoot/memberOf.db page 106
>> 800001eb READ 1 HELD userRoot/memberOf.db page 106
>> 800001eb WRITE 3 HELD userRoot/memberOf.db page 141
>> 800001eb READ 1 HELD userRoot/memberOf.db page 141
>> 800001eb WRITE 3 HELD userRoot/memberOf.db page 120
>> 800001eb READ 1 HELD userRoot/memberOf.db page 120
>> 800001eb WRITE 6 HELD userRoot/memberOf.db page 126
>> 800001eb READ 2 HELD userRoot/memberOf.db page 126
>> 800001eb WRITE 3 HELD userRoot/memberOf.db page 97
>> 800001eb READ 1 HELD userRoot/memberOf.db page 97
>> 800001eb WRITE 3 HELD userRoot/memberOf.db page 196
>> 800001eb READ 1 HELD userRoot/memberOf.db page 196
>> 800001eb WRITE 3 HELD userRoot/memberOf.db page 240
>> 800001eb READ 1 HELD userRoot/memberOf.db page 240
>> 800001eb WRITE 3 HELD userRoot/memberOf.db page 73
>> 800001eb READ 1 HELD userRoot/memberOf.db page 73
>> 800001eb WRITE 3 HELD userRoot/memberOf.db page 42
>> 800001eb READ 1 HELD userRoot/memberOf.db page 42
>> 800001eb WRITE 3 HELD userRoot/memberOf.db page 251
>> 800001eb READ 1 HELD userRoot/memberOf.db page 251
>> 800001eb WRITE 3 HELD userRoot/memberOf.db page 75
>> 800001eb READ 1 HELD userRoot/memberOf.db page 75
>> 800001eb WRITE 6 HELD userRoot/memberOf.db page 271
>> 800001eb READ 2 HELD userRoot/memberOf.db page 271
>> 800001eb WRITE 3 HELD userRoot/memberOf.db page 262
>> 800001eb READ 1 HELD userRoot/memberOf.db page 262
>> 800001eb WRITE 9 HELD userRoot/memberOf.db page 255
>> 800001eb READ 3 HELD userRoot/memberOf.db page 255
>> 800001eb WRITE 3 HELD userRoot/memberOf.db page 245
>> 800001eb READ 1 HELD userRoot/memberOf.db page 245
>> 800001eb WRITE 3 HELD userRoot/memberOf.db page 19
>> 800001eb READ 1 HELD userRoot/memberOf.db page 19
>> 800001eb WRITE 2 HELD userRoot/id2entry.db page 0
>> 800001eb WRITE 1 HELD userRoot/id2entry.db page 1224
>> =-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
>> Locks grouped by object:
>> Locker Mode Count Status ----------------- Object
>> ---------------
>> 800000f6 WRITE 1 HELD changelog/id2entry.db page 5573
>>
>> 800000f6 WRITE 2 HELD changelog/id2entry.db page 5564
>>
>> 2d READ 1 HELD changelog/entryusn.db handle 0
>>
>> 39 READ 1 HELD userRoot/ancestorid.db handle 0
>>
>> 800000f6 READ 1 HELD userRoot/ancestorid.db page 2
>>
>> 9 READ 1 HELD ipaca/vlv#allcertspkitomcatindex.db
>> handle 0
>>
>> 44 READ 1 HELD changelog/entryrdn.db handle 0
>>
>> 800000f6 READ 1 HELD changelog/nsuniqueid.db page 191
>> 800000f6 WRITE 1 HELD changelog/nsuniqueid.db page 191
>>
>> 63 READ 1 HELD changelog/nsuniqueid.db
>> handle 0
>>
>> 800000f6 WRITE 1 HELD changelog/entryrdn.db page 917
>>
>> 800000f6 WRITE 1 HELD changelog/entryrdn.db page 919
>>
>> 59 READ 1 HELD ipaca/nsuniqueid.db handle 0
>>
>> 80000106 READ 2 HELD ipaca/nsuniqueid.db page 21
>>
>> 7e READ 1 HELD userRoot/ipatokenradiusconfiglink.db
>> handle 0
>>
>> d READ 1 HELD
>> ipaca/vlv#allnonrevokedcertspkitomcatindex.db handle 0
>>
>> 75 READ 1 HELD userRoot/memberUser.db handle 0
>>
>> 800000f6 WRITE 1 HELD userRoot/memberUser.db page 3
>>
>> 800000f6 WRITE 2 HELD userRoot/memberUser.db page 2
>>
>> 800000f6 WRITE 3 HELD userRoot/memberUser.db page 5
>>
>> 800000f6 WRITE 2 HELD userRoot/memberUser.db page 6
>>
>> 800000f6 WRITE 1 HELD userRoot/memberUser.db page 9
>>
>> 800000f6 WRITE 1 HELD userRoot/memberUser.db page 8
>>
>> 800000f6 WRITE 1 HELD userRoot/memberUser.db page 11
>>
>> 800000f6 WRITE 1 HELD userRoot/memberUser.db page 10
>>
>> 800000f6 WRITE 1 HELD userRoot/memberUser.db page 13
>>
>> 800000f6 WRITE 1 HELD userRoot/memberUser.db page 12
>>
>> 800000f6 WRITE 1 HELD userRoot/memberUser.db page 15
>>
>> 800000f6 READ 6 HELD userRoot/memberUser.db page 14
>>
>> 800000f6 WRITE 1 HELD userRoot/memberUser.db page 19
>>
>> 800000f6 WRITE 1 HELD userRoot/memberUser.db page 20
>>
>> 800000f6 WRITE 1 HELD userRoot/memberUser.db page 23
>>
>> 800000f6 WRITE 1 HELD userRoot/memberUser.db page 22
>>
>> 800000f6 WRITE 1 HELD userRoot/memberUser.db page 25
>>
>> 800000f6 WRITE 1 HELD userRoot/memberUser.db page 24
>>
>> 800000f6 WRITE 1 HELD userRoot/memberUser.db page 27
>>
>> 800000f6 WRITE 1 HELD userRoot/memberUser.db page 26
>>
>> 800000f6 WRITE 3 HELD userRoot/memberUser.db page 28
>>
>> 800000f6 WRITE 1 HELD userRoot/memberUser.db page 35
>>
>> 800000f6 WRITE 2 HELD userRoot/memberUser.db page 34
>>
>> 800000f6 WRITE 1 HELD userRoot/memberUser.db page 74
>>
>> 800000f6 READ 36 HELD userRoot/memberUser.db page 81
>>
>> 800000f6 WRITE 1 HELD userRoot/memberUser.db page 82
>>
>> 800000f6 WRITE 1 HELD userRoot/memberUser.db page 84
>>
>> 800000f6 READ 1 HELD userRoot/memberUser.db page 92
>>
>> 800000f6 WRITE 1 HELD userRoot/memberUser.db page 97
>>
>> 800000f6 WRITE 1 HELD userRoot/memberUser.db page 103
>> 800000f6 READ 2 HELD userRoot/memberUser.db page 103
>>
>> 800000f6 WRITE 1 HELD userRoot/memberUser.db page 104
>>
>> 800000f6 WRITE 1 HELD userRoot/memberUser.db page 109
>>
>> 800000f6 WRITE 1 HELD userRoot/memberUser.db page 163
>>
>> 800000f6 WRITE 1 HELD userRoot/memberUser.db page 164
>>
>> 800000f6 WRITE 1 HELD userRoot/memberUser.db page 167
>>
>> 800000f6 READ 10 HELD userRoot/memberUser.db page 166
>>
>> 800000f6 WRITE 4 HELD userRoot/memberUser.db page 168
>>
>> 800000f6 WRITE 1 HELD userRoot/memberUser.db page 189
>>
>> 800000f6 WRITE 1 HELD userRoot/memberUser.db page 191
>>
>> 800000f6 WRITE 1 HELD userRoot/memberUser.db page 190
>>
>> 800000f6 WRITE 1 HELD userRoot/memberUser.db page 193
>>
>> 800000f6 WRITE 1 HELD userRoot/memberUser.db page 192
>>
>> 68 READ 1 HELD changelog/parentid.db handle 0
>>
>> 800000f6 WRITE 1 HELD changelog/parentid.db page 1
>>
>> 800000f6 WRITE 1 HELD userRoot/memberUser.db page 205
>>
>> 14 READ 1 HELD
>> ipaca/vlv#allvalidcertspkitomcatindex.db handle 0
>>
>> 50 READ 1 HELD userRoot/aci.db handle 0
>>
>> 7f READ 1 HELD userRoot/ipaassignedidview.db
>> handle 0
>>
>> 72 READ 1 HELD userRoot/seeAlso.db handle 0
>>
>> 70 READ 1 HELD userRoot/uniquemember.db
>> handle 0
>>
>> 800000f6 READ 15 HELD userRoot/member.db page 406
>>
>> 800000f6 READ 4 HELD userRoot/member.db page 405
>>
>> 800000f6 READ 2 HELD userRoot/member.db page 403
>>
>> 800001eb WRITE 1 HELD userRoot/id2entry.db page 1224
>>
>> 16 READ 1 HELD
>> ipaca/vlv#allvalidorrevokedcertspkitomcatindex.db handle 0
>>
>> 52 READ 1 HELD ipaca/aci.db handle 0
>>
>> 800000f6 READ 2 HELD userRoot/member.db page 337
>>
>> 800000f6 READ 4 HELD userRoot/member.db page 360
>>
>> 800000f6 READ 4 HELD userRoot/member.db page 356
>>
>> 73 READ 1 HELD userRoot/manager.db handle 0
>>
>> 7c READ 1 HELD userRoot/ipasudorunas.db
>> handle 0
>>
>> 800000f6 READ 2 HELD userRoot/ipasudorunas.db
>> page 1
>>
>> 7d READ 1 HELD userRoot/ipasudorunasgroup.db
>> handle 0
>>
>> 800000f6 READ 2 HELD userRoot/ipasudorunasgroup.db
>> page 1
>>
>> 800000f6 READ 4 HELD userRoot/id2entry.db page 1663
>>
>> 800000f6 READ 3 HELD userRoot/member.db page 26
>>
>> 6f READ 1 HELD userRoot/member.db handle 0
>>
>> 800000f6 READ 1 HELD userRoot/id2entry.db page 1714
>>
>> 800000f6 READ 10 HELD userRoot/member.db page 50
>>
>> 800000f6 READ 4 HELD userRoot/id2entry.db page 1712
>>
>> 800000f6 READ 1 HELD userRoot/id2entry.db page 1726
>>
>> 800000f6 READ 1 HELD userRoot/id2entry.db page 1722
>>
>> 800000f6 READ 1 HELD userRoot/member.db page 43
>>
>> 800000f6 READ 15 HELD userRoot/member.db page 70
>>
>> 800000f6 READ 1 HELD userRoot/id2entry.db page 1679
>>
>> 800001eb READ 1 HELD userRoot/memberOf.db page 272
>> 800001eb WRITE 3 HELD userRoot/memberOf.db page 272
>>
>> 800001eb READ 1 HELD userRoot/memberOf.db page 259
>> 800001eb WRITE 3 HELD userRoot/memberOf.db page 259
>>
>> 800001eb READ 1 HELD userRoot/memberOf.db page 258
>> 800001eb WRITE 3 HELD userRoot/memberOf.db page 258
>>
>> 800001eb READ 3 HELD userRoot/memberOf.db page 260
>> 800001eb WRITE 9 HELD userRoot/memberOf.db page 260
>>
>> 800001eb READ 1 HELD userRoot/memberOf.db page 262
>> 800001eb WRITE 3 HELD userRoot/memberOf.db page 262
>>
>> 800001eb READ 2 HELD userRoot/memberOf.db page 271
>> 800001eb WRITE 6 HELD userRoot/memberOf.db page 271
>>
>> 800000f6 READ 1 HELD userRoot/id2entry.db page 1730
>>
>> 26 READ 1 HELD
>> ipaca/vlv#carejectedenrollmentpkitomcatindex.db handle 0
>>
>> 1f READ 1 HELD
>> ipaca/vlv#cacompleterevocationpkitomcatindex.db handle 0
>>
>> 800001eb READ 1 HELD userRoot/memberOf.db page 19
>> 800001eb WRITE 3 HELD userRoot/memberOf.db page 19
>>
>> 87 READ 1 HELD userRoot/memberOf.db handle 0
>>
>> 800001eb READ 1 HELD userRoot/memberOf.db page 10
>> 800001eb WRITE 3 HELD userRoot/memberOf.db page 10
>>
>> 800001eb WRITE 1 HELD userRoot/memberOf.db page 37
>>
>> 800001eb READ 1 HELD userRoot/memberOf.db page 42
>> 800001eb WRITE 3 HELD userRoot/memberOf.db page 42
>>
>> 800001eb READ 1 HELD userRoot/memberOf.db page 91
>> 800001eb WRITE 3 HELD userRoot/memberOf.db page 91
>>
>> 800001eb READ 1 HELD userRoot/memberOf.db page 73
>> 800001eb WRITE 3 HELD userRoot/memberOf.db page 73
>>
>> 800001eb READ 1 HELD userRoot/memberOf.db page 75
>> 800001eb WRITE 3 HELD userRoot/memberOf.db page 75
>>
>> 800001eb READ 1 HELD userRoot/memberOf.db page 78
>> 800001eb WRITE 3 HELD userRoot/memberOf.db page 78
>>
>> 800001eb READ 1 HELD userRoot/memberOf.db page 121
>> 800001eb WRITE 3 HELD userRoot/memberOf.db page 121
>>
>> 800001eb READ 1 HELD userRoot/memberOf.db page 120
>> 800001eb WRITE 3 HELD userRoot/memberOf.db page 120
>>
>> 800001eb WRITE 2 HELD userRoot/id2entry.db page 0
>>
>> 1 READ 1 HELD userRoot/id2entry.db handle 0
>>
>> 800001eb READ 2 HELD userRoot/memberOf.db page 126
>> 800001eb WRITE 6 HELD userRoot/memberOf.db page 126
>>
>> 800001eb READ 1 HELD userRoot/memberOf.db page 97
>> 800001eb WRITE 3 HELD userRoot/memberOf.db page 97
>>
>> 800001eb READ 1 HELD userRoot/memberOf.db page 106
>> 800001eb WRITE 3 HELD userRoot/memberOf.db page 106
>>
>> 800001eb READ 1 HELD userRoot/memberOf.db page 148
>> 800001eb WRITE 3 HELD userRoot/memberOf.db page 148
>>
>> 800001eb READ 1 HELD userRoot/memberOf.db page 156
>> 800001eb WRITE 3 HELD userRoot/memberOf.db page 156
>>
>> 800001eb READ 1 HELD userRoot/memberOf.db page 141
>> 800001eb WRITE 3 HELD userRoot/memberOf.db page 141
>>
>> 800001eb READ 1 HELD userRoot/memberOf.db page 143
>> 800001eb WRITE 3 HELD userRoot/memberOf.db page 143
>>
>> 800001eb READ 1 HELD userRoot/memberOf.db page 142
>> 800001eb WRITE 3 HELD userRoot/memberOf.db page 142
>>
>> 800001eb READ 1 HELD userRoot/memberOf.db page 189
>> 800001eb WRITE 3 HELD userRoot/memberOf.db page 189
>>
>> 800001eb READ 1 HELD userRoot/memberOf.db page 211
>> 800001eb WRITE 3 HELD userRoot/memberOf.db page 211
>>
>> 800001eb READ 1 HELD userRoot/memberOf.db page 219
>> 800001eb WRITE 3 HELD userRoot/memberOf.db page 219
>>
>> 800001eb READ 1 HELD userRoot/memberOf.db page 218
>> 800001eb WRITE 3 HELD userRoot/memberOf.db page 218
>>
>> 800001eb READ 1 HELD userRoot/memberOf.db page 196
>> 800001eb WRITE 3 HELD userRoot/memberOf.db page 196
>>
>> f READ 1 HELD
>> ipaca/vlv#allrevokedcertspkitomcatindex.db handle 0
>>
>> 800001eb READ 1 HELD userRoot/memberOf.db page 207
>> 800001eb WRITE 3 HELD userRoot/memberOf.db page 207
>>
>> 800001eb READ 1 HELD userRoot/memberOf.db page 240
>> 800001eb WRITE 3 HELD userRoot/memberOf.db page 240
>>
>> 800001eb READ 1 HELD userRoot/memberOf.db page 245
>> 800001eb WRITE 3 HELD userRoot/memberOf.db page 245
>>
>> 78 READ 1 HELD userRoot/memberservice.db
>> handle 0
>>
>> 800001eb READ 1 HELD userRoot/memberOf.db page 251
>> 800001eb WRITE 3 HELD userRoot/memberOf.db page 251
>>
>> 800001eb READ 1 HELD userRoot/memberOf.db page 253
>> 800001eb WRITE 3 HELD userRoot/memberOf.db page 253
>>
>> 800001eb READ 3 HELD userRoot/memberOf.db page 255
>> 800001eb WRITE 9 HELD userRoot/memberOf.db page 255
>>
>> 800001eb READ 1 HELD userRoot/memberOf.db page 228
>> 800001eb WRITE 3 HELD userRoot/memberOf.db page 228
>>
>> 80000106 WRITE 2 HELD ipaca/vlv#cacompletepkitomcatindex.db
>> page 1
>>
>> 1c READ 1 HELD ipaca/vlv#cacompletepkitomcatindex.db
>> handle 0
>>
>> 80000106 WRITE 4 HELD ipaca/vlv#cacompletepkitomcatindex.db
>> page 5
>>
>> 13 READ 1 HELD
>> ipaca/vlv#allrevokedorrevokedexpiredcertspkitomcatindex.db
>> handle 0
>>
>> 2a READ 1 HELD
>> ipaca/vlv#carevocationpkitomcatindex.db handle 0
>>
>> 74 READ 1 HELD userRoot/secretary.db handle 0
>>
>> 5 READ 1 HELD ipaca/entryrdn.db handle 0
>>
>> 85 READ 1 HELD ipaca/requeststate.db handle 0
>>
>> 80000106 READ 1 HELD ipaca/requeststate.db page 1
>> 80000106 WRITE 4 HELD ipaca/requeststate.db page 1
>>
>> 3b READ 1 HELD userRoot/macAddress.db handle 0
>>
>> 6e READ 1 HELD userRoot/numsubordinates.db
>> handle 0
>>
>> 80000106 WRITE 2 HELD ipaca/vlv#caallpkitomcatindex.db
>> page 1
>>
>> 17 READ 1 HELD ipaca/vlv#caallpkitomcatindex.db
>> handle 0
>>
>> 80000106 WRITE 4 HELD ipaca/vlv#caallpkitomcatindex.db
>> page 5
>>
>> 10 READ 1 HELD
>> ipaca/vlv#allrevokedcertsnotafterpkitomcatindex.db handle 0
>>
>> 800000f6 READ 1 HELD userRoot/id2entry.db page 2495
>> 800000f6 WRITE 1 HELD userRoot/id2entry.db page 2495
>> 2 READ 1 WAIT userRoot/id2entry.db page 2495
>>
>> 800000f6 READ 2 HELD userRoot/memberdenycmd.db
>> page 1
>>
>> 7b READ 1 HELD userRoot/memberdenycmd.db
>> handle 0
>>
>> 5e READ 1 HELD
>> /var/lib/dirsrv/slapd-xxxx/cldb/f3c29b1e-d15511e4-8d35ce39-9b469c1f_550feb15000000600000.db
>> handle 0
>>
>> 4e READ 1 HELD changelog/aci.db handle 0
>>
>> 67 READ 1 HELD changelog/targetuniqueid.db
>> handle 0
>>
>> 800000f6 WRITE 1 HELD changelog/targetuniqueid.db
>> page 47
>>
>> 800000f6 WRITE 1 HELD changelog/id2entry.db page 2
>>
>> 800000f6 WRITE 1 HELD changelog/id2entry.db page 0
>>
>> 2b READ 1 HELD changelog/id2entry.db handle 0
>>
>> 25 READ 1 HELD ipaca/vlv#carejectedpkitomcatindex.db
>> handle 0
>>
>> 800000f6 READ 4 HELD userRoot/id2entry.db page 2905
>>
>> 2f READ 1 HELD userRoot/entryusn.db handle 0
>>
>> 31 READ 1 HELD ipaca/entryusn.db handle 0
>>
>> 80000106 READ 1 HELD ipaca/entryusn.db page 13
>> 80000106 WRITE 3 HELD ipaca/entryusn.db page 13
>>
>> 800000f6 READ 1 HELD userRoot/entryusn.db page 27
>> 800000f6 WRITE 3 HELD userRoot/entryusn.db page 27
>> 800001eb READ 1 HELD userRoot/entryusn.db page 27
>> 800001eb WRITE 3 HELD userRoot/entryusn.db page 27
>>
>> 800000f6 WRITE 1 HELD changelog/changenumber.db
>> page 68
>> 65 READ 1 WAIT changelog/changenumber.db
>> page 68
>>
>> 64 READ 1 HELD changelog/changenumber.db
>> handle 0
>>
>> 800000f6 READ 1 HELD userRoot/entryrdn.db page 35
>>
>> 82 READ 1 HELD userRoot/krbPrincipalName.db
>> handle 0
>>
>> 800000f6 READ 2 HELD userRoot/memberallowcmd.db
>> page 44
>>
>> 33 READ 1 HELD userRoot/entryrdn.db handle 0
>>
>> 80000106 READ 1 HELD ipaca/requesttype.db page 1
>> 80000106 WRITE 4 HELD ipaca/requesttype.db page 1
>>
>> 86 READ 1 HELD ipaca/requesttype.db handle 0
>>
>> 7a READ 1 HELD userRoot/memberallowcmd.db
>> handle 0
>>
>> 800000f6 READ 18 HELD userRoot/entryrdn.db page 93
>>
>> 800000f6 READ 18 HELD userRoot/entryrdn.db page 83
>>
>> 800000f6 READ 4 HELD userRoot/entryrdn.db page 249
>>
>> 37 READ 1 HELD userRoot/objectclass.db
>> handle 0
>>
>> 800000f6 READ 5 HELD userRoot/entryrdn.db page 255
>>
>> 800000f6 READ 2 HELD userRoot/objectclass.db page 10
>>
>> 800000f6 READ 4 HELD userRoot/entryrdn.db page 229
>>
>> 77 READ 1 HELD userRoot/sourcehost.db handle 0
>>
>> 800000f6 READ 2 HELD userRoot/objectclass.db page 50
>>
>> 800000f6 WRITE 1 HELD changelog/id2entry.db page 7353
>>
>> 48 READ 1 HELD changelog/objectclass.db
>> handle 0
>>
>> 800000f6 WRITE 3 HELD changelog/objectclass.db
>> page 1
>> 6b READ 1 WAIT changelog/objectclass.db
>> page 1
>> 49 READ 1 WAIT changelog/objectclass.db
>> page 1
>> 88 READ 1 WAIT changelog/objectclass.db
>> page 1
>> 89 READ 1 WAIT changelog/objectclass.db
>> page 1
>> 8b READ 1 WAIT changelog/objectclass.db
>> page 1
>>
>> 800000f6 READ 61 HELD userRoot/objectclass.db page 61
>>
>> 80000106 WRITE 4 HELD ipaca/vlv#caenrollmentpkitomcatindex.db
>> page 5
>>
>> 20 READ 1 HELD
>> ipaca/vlv#caenrollmentpkitomcatindex.db handle 0
>>
>> 80000106 WRITE 2 HELD ipaca/vlv#caenrollmentpkitomcatindex.db
>> page 1
>>
>> 800000f6 READ 3 HELD userRoot/entryrdn.db page 262
>>
>> 6a READ 1 HELD changelog/numsubordinates.db
>> handle 0
>>
>> 800000f6 WRITE 1 HELD changelog/numsubordinates.db
>> page 1
>>
>> 800000f6 READ 1 HELD userRoot/entryrdn.db page 495
>>
>> 800000f6 READ 18 HELD userRoot/entryrdn.db page 466
>>
>> 54 READ 1 HELD userRoot/parentid.db handle 0
>>
>> 80000106 WRITE 1 HELD ipaca/id2entry.db page 301
>>
>> 800000f6 WRITE 1 HELD changelog/ancestorid.db page 1
>>
>> 69 READ 1 HELD changelog/ancestorid.db
>> handle 0
>>
>> 4b READ 1 HELD ipaca/objectclass.db handle 0
>>
>> 57 READ 1 HELD userRoot/nsuniqueid.db handle 0
>>
>> 800000f6 READ 2 HELD userRoot/nsuniqueid.db page 29
>>
>> 5b READ 1 HELD
>> /var/lib/dirsrv/slapd-xxxx/cldb/9890a88b-d15511e4-8d35ce39-9b469c1f_550feacc000000040000.db
>> handle 0
>>
>> 6d READ 1 HELD userRoot/nscpEntryDN.db
>> handle 0
>>
>> 80000106 WRITE 4 HELD ipaca/id2entry.db page 0
>>
>> 3 READ 1 HELD ipaca/id2entry.db handle 0
>>
>> 6c READ 1 HELD userRoot/nsTombstoneCSN.db
>> handle 0
>>
>> 800000f6 READ 5 HELD userRoot/memberHost.db page 87
>>
>> 800000f6 READ 3 HELD userRoot/memberHost.db page 96
>>
>> 800000f6 READ 1 HELD userRoot/memberHost.db page 28
>>
>> 76 READ 1 HELD userRoot/memberHost.db handle 0
>>
>> 71 READ 1 HELD userRoot/owner.db handle 0
>>
>> 800000f6 READ 3 HELD userRoot/memberHost.db page 209
>>
>> 800000f6 READ 35 HELD userRoot/memberHost.db page 195
>>
>> 15 READ 1 HELD
>> ipaca/vlv#allvalidcertsnotafterpkitomcatindex.db handle 0
>>
>> 80000106 WRITE 4 HELD
>> ipaca/vlv#cacompleteenrollmentpkitomcatindex.db page 5
>>
>> 1d READ 1 HELD
>> ipaca/vlv#cacompleteenrollmentpkitomcatindex.db handle 0
>>
>> 80000106 WRITE 2 HELD
>> ipaca/vlv#cacompleteenrollmentpkitomcatindex.db page 1
>>
>> 79 READ 1 HELD userRoot/managedby.db handle 0
>>
>> 800000f6 READ 3 HELD changelog/entryrdn.db page 387
>>
>> 800000f6 READ 2 HELD changelog/entryrdn.db page 388
>> 800000f6 WRITE 1 HELD changelog/entryrdn.db page 388
>>
>> 800000f6 WRITE 1 HELD changelog/entryusn.db page 68
>>
>>
>> Lukasz Jaworski 'Ender'
>>
>>
>> Wiadomość napisana przez thierry bordaz <tbordaz at redhat.com> w dniu 6
>> maj 2015, o godz. 11:47:
>>
>>> This is looking like thread 13 prevents thread 12 run (and all the
>>> others).
>>> Now thread 13 is likely waiting for db page? We may need output of
>>> db_stat (db_state -N -h /var/lib/dirsrv/slapd-xxx/db/ -CA)
>>>
>>> thanks
>>> thierry
>>> On 05/06/2015 11:31 AM, Łukasz Jaworski wrote:
>>>>>> ldapsearch hangs. Dirsrv is not responding now.
>>>>> if the server is hanging, can you get a pstack
>>>> Thread 45 (Thread 0x7fc6a562d700 (LWP 1868)):
>>>> #0 0x00007fc6b2f1aae3 in select () from /lib64/libc.so.6
>>>> #1 0x00007fc6b5492a99 in DS_Sleep () from
>>>> /usr/lib64/dirsrv/libslapd.so.0
>>>> #2 0x00007fc6a961f987 in deadlock_threadmain () from
>>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so
>>>> #3 0x00007fc6b3847cab in _pt_root () from /lib64/libnspr4.so
>>>> #4 0x00007fc6b31e852a in start_thread () from /lib64/libpthread.so.0
>>>> #5 0x00007fc6b2f2422d in clone () from /lib64/libc.so.6
>>>> Thread 44 (Thread 0x7fc6a4e2c700 (LWP 1869)):
>>>> #0 0x00007fc6b2f1aae3 in select () from /lib64/libc.so.6
>>>> #1 0x00007fc6b5492a99 in DS_Sleep () from
>>>> /usr/lib64/dirsrv/libslapd.so.0
>>>> #2 0x00007fc6a9623a4e in checkpoint_threadmain () from
>>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so
>>>> #3 0x00007fc6b3847cab in _pt_root () from /lib64/libnspr4.so
>>>> #4 0x00007fc6b31e852a in start_thread () from /lib64/libpthread.so.0
>>>> #5 0x00007fc6b2f2422d in clone () from /lib64/libc.so.6
>>>> Thread 43 (Thread 0x7fc6a462b700 (LWP 1870)):
>>>> #0 0x00007fc6b2f1aae3 in select () from /lib64/libc.so.6
>>>> #1 0x00007fc6b5492a99 in DS_Sleep () from
>>>> /usr/lib64/dirsrv/libslapd.so.0
>>>> #2 0x00007fc6a961fc0f in trickle_threadmain () from
>>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so
>>>> #3 0x00007fc6b3847cab in _pt_root () from /lib64/libnspr4.so
>>>> #4 0x00007fc6b31e852a in start_thread () from /lib64/libpthread.so.0
>>>> #5 0x00007fc6b2f2422d in clone () from /lib64/libc.so.6
>>>> Thread 42 (Thread 0x7fc6a3e2a700 (LWP 1871)):
>>>> #0 0x00007fc6b2f1aae3 in select () from /lib64/libc.so.6
>>>> #1 0x00007fc6b5492a99 in DS_Sleep () from
>>>> /usr/lib64/dirsrv/libslapd.so.0
>>>> #2 0x00007fc6a961a667 in perf_threadmain () from
>>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so
>>>> #3 0x00007fc6b3847cab in _pt_root () from /lib64/libnspr4.so
>>>> #4 0x00007fc6b31e852a in start_thread () from /lib64/libpthread.so.0
>>>> #5 0x00007fc6b2f2422d in clone () from /lib64/libc.so.6
>>>> Thread 41 (Thread 0x7fc6a3629700 (LWP 1900)):
>>>> #0 0x00007fc6b31ed590 in pthread_cond_wait@@GLIBC_2.3.2 () from
>>>> /lib64/libpthread.so.0
>>>> #1 0x00007fc6b3842440 in PR_WaitCondVar () from /lib64/libnspr4.so
>>>> #2 0x00007fc6b5482c68 in slapi_wait_condvar () from
>>>> /usr/lib64/dirsrv/libslapd.so.0
>>>> #3 0x00007fc6abd455be in cos_cache_wait_on_change () from
>>>> /usr/lib64/dirsrv/plugins/libcos-plugin.so
>>>> #4 0x00007fc6b3847cab in _pt_root () from /lib64/libnspr4.so
>>>> #5 0x00007fc6b31e852a in start_thread () from /lib64/libpthread.so.0
>>>> #6 0x00007fc6b2f2422d in clone () from /lib64/libc.so.6
>>>> Thread 40 (Thread 0x7fc6b5735700 (LWP 1901)):
>>>> #0 0x00007fc6b31ed939 in pthread_cond_timedwait@@GLIBC_2.3.2 ()
>>>> from /lib64/libpthread.so.0
>>>> #1 0x00007fc6b3841ef8 in pt_TimedWait () from /lib64/libnspr4.so
>>>> #2 0x00007fc6b38423be in PR_WaitCondVar () from /lib64/libnspr4.so
>>>> #3 0x00007fc6a937be84 in _cl5TrimMain () from
>>>> /usr/lib64/dirsrv/plugins/libreplication-plugin.so
>>>> #4 0x00007fc6b3847cab in _pt_root () from /lib64/libnspr4.so
>>>> #5 0x00007fc6b31e852a in start_thread () from /lib64/libpthread.so.0
>>>> #6 0x00007fc6b2f2422d in clone () from /lib64/libc.so.6
>>>> Thread 39 (Thread 0x7fc6a2e28700 (LWP 1902)):
>>>> #0 0x00007fc6b31ed590 in pthread_cond_wait@@GLIBC_2.3.2 () from
>>>> /lib64/libpthread.so.0
>>>> #1 0x00007fc6b3842440 in PR_WaitCondVar () from /lib64/libnspr4.so
>>>> #2 0x00007fc6a93928d4 in protocol_sleep () from
>>>> /usr/lib64/dirsrv/plugins/libreplication-plugin.so
>>>> #3 0x00007fc6a93949e5 in repl5_inc_run () from
>>>> /usr/lib64/dirsrv/plugins/libreplication-plugin.so
>>>> #4 0x00007fc6a9399421 in prot_thread_main () from
>>>> /usr/lib64/dirsrv/plugins/libreplication-plugin.so
>>>> #5 0x00007fc6b3847cab in _pt_root () from /lib64/libnspr4.so
>>>> #6 0x00007fc6b31e852a in start_thread () from /lib64/libpthread.so.0
>>>> #7 0x00007fc6b2f2422d in clone () from /lib64/libc.so.6
>>>> Thread 38 (Thread 0x7fc6a2627700 (LWP 1903)):
>>>> #0 0x00007fc6b31ed939 in pthread_cond_timedwait@@GLIBC_2.3.2 ()
>>>> from /lib64/libpthread.so.0
>>>> #1 0x00007fc6b3841ef8 in pt_TimedWait () from /lib64/libnspr4.so
>>>> #2 0x00007fc6b38423be in PR_WaitCondVar () from /lib64/libnspr4.so
>>>> #3 0x00007fc6a93928d4 in protocol_sleep () from
>>>> /usr/lib64/dirsrv/plugins/libreplication-plugin.so
>>>> #4 0x00007fc6a9395158 in repl5_inc_run () from
>>>> /usr/lib64/dirsrv/plugins/libreplication-plugin.so
>>>> #5 0x00007fc6a9399421 in prot_thread_main () from
>>>> /usr/lib64/dirsrv/plugins/libreplication-plugin.so
>>>> #6 0x00007fc6b3847cab in _pt_root () from /lib64/libnspr4.so
>>>> #7 0x00007fc6b31e852a in start_thread () from /lib64/libpthread.so.0
>>>> #8 0x00007fc6b2f2422d in clone () from /lib64/libc.so.6
>>>> Thread 37 (Thread 0x7fc6a1a04700 (LWP 1906)):
>>>> #0 0x00007fc6b31ed590 in pthread_cond_wait@@GLIBC_2.3.2 () from
>>>> /lib64/libpthread.so.0
>>>> #1 0x00007fc6b3842440 in PR_WaitCondVar () from /lib64/libnspr4.so
>>>> #2 0x00007fc6b5482c68 in slapi_wait_condvar () from
>>>> /usr/lib64/dirsrv/libslapd.so.0
>>>> #3 0x00007fc6a7cbef5d in roles_cache_wait_on_change () from
>>>> /usr/lib64/dirsrv/plugins/libroles-plugin.so
>>>> #4 0x00007fc6b3847cab in _pt_root () from /lib64/libnspr4.so
>>>> #5 0x00007fc6b31e852a in start_thread () from /lib64/libpthread.so.0
>>>> #6 0x00007fc6b2f2422d in clone () from /lib64/libc.so.6
>>>> Thread 36 (Thread 0x7fc6a1203700 (LWP 1907)):
>>>> #0 0x00007fc6b31ed590 in pthread_cond_wait@@GLIBC_2.3.2 () from
>>>> /lib64/libpthread.so.0
>>>> #1 0x00007fc6b3842440 in PR_WaitCondVar () from /lib64/libnspr4.so
>>>> #2 0x00007fc6b5482c68 in slapi_wait_condvar () from
>>>> /usr/lib64/dirsrv/libslapd.so.0
>>>> #3 0x00007fc6a7cbef5d in roles_cache_wait_on_change () from
>>>> /usr/lib64/dirsrv/plugins/libroles-plugin.so
>>>> #4 0x00007fc6b3847cab in _pt_root () from /lib64/libnspr4.so
>>>> #5 0x00007fc6b31e852a in start_thread () from /lib64/libpthread.so.0
>>>> #6 0x00007fc6b2f2422d in clone () from /lib64/libc.so.6
>>>> Thread 35 (Thread 0x7fc6a0a02700 (LWP 1908)):
>>>> #0 0x00007fc6b31ed590 in pthread_cond_wait@@GLIBC_2.3.2 () from
>>>> /lib64/libpthread.so.0
>>>> #1 0x00007fc6b3842440 in PR_WaitCondVar () from /lib64/libnspr4.so
>>>> #2 0x00007fc6b5482c68 in slapi_wait_condvar () from
>>>> /usr/lib64/dirsrv/libslapd.so.0
>>>> #3 0x00007fc6a7cbef5d in roles_cache_wait_on_change () from
>>>> /usr/lib64/dirsrv/plugins/libroles-plugin.so
>>>> #4 0x00007fc6b3847cab in _pt_root () from /lib64/libnspr4.so
>>>> #5 0x00007fc6b31e852a in start_thread () from /lib64/libpthread.so.0
>>>> #6 0x00007fc6b2f2422d in clone () from /lib64/libc.so.6
>>>> Thread 34 (Thread 0x7fc693fff700 (LWP 1909)):
>>>> #0 0x00007fc6b31ed939 in pthread_cond_timedwait@@GLIBC_2.3.2 ()
>>>> from /lib64/libpthread.so.0
>>>> #1 0x00007fc6b3841ef8 in pt_TimedWait () from /lib64/libnspr4.so
>>>> #2 0x00007fc6b38423be in PR_WaitCondVar () from /lib64/libnspr4.so
>>>> #3 0x00007fc6b593a773 in housecleaning ()
>>>> #4 0x00007fc6b3847cab in _pt_root () from /lib64/libnspr4.so
>>>> #5 0x00007fc6b31e852a in start_thread () from /lib64/libpthread.so.0
>>>> #6 0x00007fc6b2f2422d in clone () from /lib64/libc.so.6
>>>> Thread 33 (Thread 0x7fc6937fe700 (LWP 1910)):
>>>> #0 0x00007fc6b31ed590 in pthread_cond_wait@@GLIBC_2.3.2 () from
>>>> /lib64/libpthread.so.0
>>>> #1 0x00007fc6b38426b3 in PR_EnterMonitor () from /lib64/libnspr4.so
>>>> #2 0x00007fc6a9622456 in dblayer_txn_begin () from
>>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so
>>>> #3 0x00007fc6a965d615 in ldbm_back_modify () from
>>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so
>>>> #4 0x00007fc6b544e905 in op_shared_modify () from
>>>> /usr/lib64/dirsrv/libslapd.so.0
>>>> #5 0x00007fc6b544f387 in modify_internal_pb () from
>>>> /usr/lib64/dirsrv/libslapd.so.0
>>>> #6 0x00007fc6a939f76f in replica_write_ruv () from
>>>> /usr/lib64/dirsrv/plugins/libreplication-plugin.so
>>>> #7 0x00007fc6a939f9fe in replica_update_state () from
>>>> /usr/lib64/dirsrv/plugins/libreplication-plugin.so
>>>> #8 0x00007fc6b542965a in eq_loop () from
>>>> /usr/lib64/dirsrv/libslapd.so.0
>>>> #9 0x00007fc6b3847cab in _pt_root () from /lib64/libnspr4.so
>>>> #10 0x00007fc6b31e852a in start_thread () from /lib64/libpthread.so.0
>>>> #11 0x00007fc6b2f2422d in clone () from /lib64/libc.so.6
>>>> Thread 32 (Thread 0x7fc692924700 (LWP 1912)):
>>>> #0 0x00007fc6b31ed590 in pthread_cond_wait@@GLIBC_2.3.2 () from
>>>> /lib64/libpthread.so.0
>>>> #1 0x00007fc6b3842440 in PR_WaitCondVar () from /lib64/libnspr4.so
>>>> #2 0x00007fc6b5930b9e in connection_wait_for_new_work ()
>>>> #3 0x00007fc6b5931dc9 in connection_threadmain ()
>>>> #4 0x00007fc6b3847cab in _pt_root () from /lib64/libnspr4.so
>>>> #5 0x00007fc6b31e852a in start_thread () from /lib64/libpthread.so.0
>>>> #6 0x00007fc6b2f2422d in clone () from /lib64/libc.so.6
>>>> Thread 31 (Thread 0x7fc692123700 (LWP 1913)):
>>>> #0 0x00007fc6b31ed590 in pthread_cond_wait@@GLIBC_2.3.2 () from
>>>> /lib64/libpthread.so.0
>>>> #1 0x00007fc6b3842440 in PR_WaitCondVar () from /lib64/libnspr4.so
>>>> #2 0x00007fc6b5930b9e in connection_wait_for_new_work ()
>>>> #3 0x00007fc6b5931dc9 in connection_threadmain ()
>>>> #4 0x00007fc6b3847cab in _pt_root () from /lib64/libnspr4.so
>>>> #5 0x00007fc6b31e852a in start_thread () from /lib64/libpthread.so.0
>>>> #6 0x00007fc6b2f2422d in clone () from /lib64/libc.so.6
>>>> Thread 30 (Thread 0x7fc691922700 (LWP 1914)):
>>>> #0 0x00007fc6b31ed590 in pthread_cond_wait@@GLIBC_2.3.2 () from
>>>> /lib64/libpthread.so.0
>>>> #1 0x00007fc6adc811fb in __db_hybrid_mutex_suspend () from
>>>> /lib64/libdb-5.3.so
>>>> #2 0x00007fc6adc8059b in __db_tas_mutex_lock () from
>>>> /lib64/libdb-5.3.so
>>>> #3 0x00007fc6add2aa91 in __lock_get_internal () from
>>>> /lib64/libdb-5.3.so
>>>> #4 0x00007fc6add2b657 in __lock_get () from /lib64/libdb-5.3.so
>>>> #5 0x00007fc6add576af in __db_lget () from /lib64/libdb-5.3.so
>>>> #6 0x00007fc6adc9d5d9 in __bam_get_root () from /lib64/libdb-5.3.so
>>>> #7 0x00007fc6adc9d91b in __bam_search () from /lib64/libdb-5.3.so
>>>> #8 0x00007fc6adc89144 in __bamc_search () from /lib64/libdb-5.3.so
>>>> #9 0x00007fc6adc8adff in __bamc_get () from /lib64/libdb-5.3.so
>>>> #10 0x00007fc6add43ba3 in __dbc_iget () from /lib64/libdb-5.3.so
>>>> #11 0x00007fc6add53092 in __dbc_get_pp () from /lib64/libdb-5.3.so
>>>> #12 0x00007fc6a962e6de in idl_new_fetch () from
>>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so
>>>> #13 0x00007fc6a963cc2b in index_read_ext_allids () from
>>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so
>>>> #14 0x00007fc6a96272ed in keys2idl () from
>>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so
>>>> #15 0x00007fc6a9627afe in ava_candidates.isra () from
>>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so
>>>> #16 0x00007fc6a96280fa in filter_candidates_ext () from
>>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so
>>>> #17 0x00007fc6a96290ce in list_candidates () from
>>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so
>>>> #18 0x00007fc6a9628062 in filter_candidates_ext () from
>>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so
>>>> #19 0x00007fc6a96631a7 in subtree_candidates () from
>>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so
>>>> #20 0x00007fc6a9664811 in ldbm_back_search () from
>>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so
>>>> #21 0x00007fc6b5455410 in op_shared_search () from
>>>> /usr/lib64/dirsrv/libslapd.so.0
>>>> #22 0x00007fc6b5943fc6 in do_search ()
>>>> #23 0x00007fc6b5932b4c in connection_threadmain ()
>>>> #24 0x00007fc6b3847cab in _pt_root () from /lib64/libnspr4.so
>>>> #25 0x00007fc6b31e852a in start_thread () from /lib64/libpthread.so.0
>>>> #26 0x00007fc6b2f2422d in clone () from /lib64/libc.so.6
>>>> Thread 29 (Thread 0x7fc691121700 (LWP 1915)):
>>>> #0 0x00007fc6b31ed590 in pthread_cond_wait@@GLIBC_2.3.2 () from
>>>> /lib64/libpthread.so.0
>>>> #1 0x00007fc6adc811fb in __db_hybrid_mutex_suspend () from
>>>> /lib64/libdb-5.3.so
>>>> #2 0x00007fc6adc8059b in __db_tas_mutex_lock () from
>>>> /lib64/libdb-5.3.so
>>>> #3 0x00007fc6add2aa91 in __lock_get_internal () from
>>>> /lib64/libdb-5.3.so
>>>> #4 0x00007fc6add2b657 in __lock_get () from /lib64/libdb-5.3.so
>>>> #5 0x00007fc6add576af in __db_lget () from /lib64/libdb-5.3.so
>>>> #6 0x00007fc6adc9d5d9 in __bam_get_root () from /lib64/libdb-5.3.so
>>>> #7 0x00007fc6adc9d91b in __bam_search () from /lib64/libdb-5.3.so
>>>> #8 0x00007fc6adc89144 in __bamc_search () from /lib64/libdb-5.3.so
>>>> #9 0x00007fc6adc8adff in __bamc_get () from /lib64/libdb-5.3.so
>>>> #10 0x00007fc6add43ba3 in __dbc_iget () from /lib64/libdb-5.3.so
>>>> #11 0x00007fc6add53092 in __dbc_get_pp () from /lib64/libdb-5.3.so
>>>> #12 0x00007fc6a962e6de in idl_new_fetch () from
>>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so
>>>> #13 0x00007fc6a963cc2b in index_read_ext_allids () from
>>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so
>>>> #14 0x00007fc6a96272ed in keys2idl () from
>>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so
>>>> #15 0x00007fc6a9627afe in ava_candidates.isra () from
>>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so
>>>> #16 0x00007fc6a96280fa in filter_candidates_ext () from
>>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so
>>>> #17 0x00007fc6a96290ce in list_candidates () from
>>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so
>>>> #18 0x00007fc6a9628062 in filter_candidates_ext () from
>>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so
>>>> #19 0x00007fc6a96631a7 in subtree_candidates () from
>>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so
>>>> #20 0x00007fc6a9664811 in ldbm_back_search () from
>>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so
>>>> #21 0x00007fc6b5455410 in op_shared_search () from
>>>> /usr/lib64/dirsrv/libslapd.so.0
>>>> #22 0x00007fc6b5943fc6 in do_search ()
>>>> #23 0x00007fc6b5932b4c in connection_threadmain ()
>>>> #24 0x00007fc6b3847cab in _pt_root () from /lib64/libnspr4.so
>>>> #25 0x00007fc6b31e852a in start_thread () from /lib64/libpthread.so.0
>>>> #26 0x00007fc6b2f2422d in clone () from /lib64/libc.so.6
>>>> Thread 28 (Thread 0x7fc690920700 (LWP 1916)):
>>>> #0 0x00007fc6b31ed590 in pthread_cond_wait@@GLIBC_2.3.2 () from
>>>> /lib64/libpthread.so.0
>>>> #1 0x00007fc6adc811fb in __db_hybrid_mutex_suspend () from
>>>> /lib64/libdb-5.3.so
>>>> #2 0x00007fc6adc8059b in __db_tas_mutex_lock () from
>>>> /lib64/libdb-5.3.so
>>>> #3 0x00007fc6add2aa91 in __lock_get_internal () from
>>>> /lib64/libdb-5.3.so
>>>> #4 0x00007fc6add2b657 in __lock_get () from /lib64/libdb-5.3.so
>>>> #5 0x00007fc6add576af in __db_lget () from /lib64/libdb-5.3.so
>>>> #6 0x00007fc6adc9d5d9 in __bam_get_root () from /lib64/libdb-5.3.so
>>>> #7 0x00007fc6adc9d91b in __bam_search () from /lib64/libdb-5.3.so
>>>> #8 0x00007fc6adc89144 in __bamc_search () from /lib64/libdb-5.3.so
>>>> #9 0x00007fc6adc8adff in __bamc_get () from /lib64/libdb-5.3.so
>>>> #10 0x00007fc6add43ba3 in __dbc_iget () from /lib64/libdb-5.3.so
>>>> #11 0x00007fc6add53092 in __dbc_get_pp () from /lib64/libdb-5.3.so
>>>> #12 0x00007fc6a962e6de in idl_new_fetch () from
>>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so
>>>> #13 0x00007fc6a963cc2b in index_read_ext_allids () from
>>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so
>>>> #14 0x00007fc6a96272ed in keys2idl () from
>>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so
>>>> #15 0x00007fc6a9627afe in ava_candidates.isra () from
>>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so
>>>> #16 0x00007fc6a96280fa in filter_candidates_ext () from
>>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so
>>>> #17 0x00007fc6a96290ce in list_candidates () from
>>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so
>>>> #18 0x00007fc6a9628062 in filter_candidates_ext () from
>>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so
>>>> #19 0x00007fc6a96631a7 in subtree_candidates () from
>>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so
>>>> #20 0x00007fc6a9664811 in ldbm_back_search () from
>>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so
>>>> #21 0x00007fc6b5455410 in op_shared_search () from
>>>> /usr/lib64/dirsrv/libslapd.so.0
>>>> #22 0x00007fc6b5943fc6 in do_search ()
>>>> #23 0x00007fc6b5932b4c in connection_threadmain ()
>>>> #24 0x00007fc6b3847cab in _pt_root () from /lib64/libnspr4.so
>>>> #25 0x00007fc6b31e852a in start_thread () from /lib64/libpthread.so.0
>>>> #26 0x00007fc6b2f2422d in clone () from /lib64/libc.so.6
>>>> Thread 27 (Thread 0x7fc67ffff700 (LWP 1917)):
>>>> #0 0x00007fc6b31ed590 in pthread_cond_wait@@GLIBC_2.3.2 () from
>>>> /lib64/libpthread.so.0
>>>> #1 0x00007fc6b3842440 in PR_WaitCondVar () from /lib64/libnspr4.so
>>>> #2 0x00007fc6b5930b9e in connection_wait_for_new_work ()
>>>> #3 0x00007fc6b5931dc9 in connection_threadmain ()
>>>> #4 0x00007fc6b3847cab in _pt_root () from /lib64/libnspr4.so
>>>> #5 0x00007fc6b31e852a in start_thread () from /lib64/libpthread.so.0
>>>> #6 0x00007fc6b2f2422d in clone () from /lib64/libc.so.6
>>>> Thread 26 (Thread 0x7fc67f7fe700 (LWP 1918)):
>>>> #0 0x00007fc6b31ed590 in pthread_cond_wait@@GLIBC_2.3.2 () from
>>>> /lib64/libpthread.so.0
>>>> #1 0x00007fc6adc811fb in __db_hybrid_mutex_suspend () from
>>>> /lib64/libdb-5.3.so
>>>> #2 0x00007fc6adc8059b in __db_tas_mutex_lock () from
>>>> /lib64/libdb-5.3.so
>>>> #3 0x00007fc6add2aa91 in __lock_get_internal () from
>>>> /lib64/libdb-5.3.so
>>>> #4 0x00007fc6add2b657 in __lock_get () from /lib64/libdb-5.3.so
>>>> #5 0x00007fc6add576af in __db_lget () from /lib64/libdb-5.3.so
>>>> #6 0x00007fc6adc9e4f0 in __bam_search () from /lib64/libdb-5.3.so
>>>> #7 0x00007fc6adc89144 in __bamc_search () from /lib64/libdb-5.3.so
>>>> #8 0x00007fc6adc8adff in __bamc_get () from /lib64/libdb-5.3.so
>>>> #9 0x00007fc6add43ba3 in __dbc_iget () from /lib64/libdb-5.3.so
>>>> #10 0x00007fc6add50ced in __db_get () from /lib64/libdb-5.3.so
>>>> #11 0x00007fc6add5473b in __db_get_pp () from /lib64/libdb-5.3.so
>>>> #12 0x00007fc6a962a9bb in id2entry () from
>>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so
>>>> #13 0x00007fc6a9626cbd in dn2entry_ext () from
>>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so
>>>> #14 0x00007fc6a9629bb1 in find_entry_internal.isra () from
>>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so
>>>> #15 0x00007fc6a9629e99 in find_entry () from
>>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so
>>>> #16 0x00007fc6a9663b5b in ldbm_back_search () from
>>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so
>>>> #17 0x00007fc6b5455410 in op_shared_search () from
>>>> /usr/lib64/dirsrv/libslapd.so.0
>>>> #18 0x00007fc6b546553e in search_internal_callback_pb () from
>>>> /usr/lib64/dirsrv/libslapd.so.0
>>>> #19 0x00007fc6b54657c8 in search_internal_pb () from
>>>> /usr/lib64/dirsrv/libslapd.so.0
>>>> #20 0x00007fc6b5465b73 in slapi_search_internal_get_entry () from
>>>> /usr/lib64/dirsrv/libslapd.so.0
>>>> #21 0x00007fc6aad02234 in ipalockout_preop () from
>>>> /usr/lib64/dirsrv/plugins/libipa_lockout.so
>>>> #22 0x00007fc6b546156f in plugin_call_func () from
>>>> /usr/lib64/dirsrv/libslapd.so.0
>>>> #23 0x00007fc6b5461893 in plugin_call_plugins () from
>>>> /usr/lib64/dirsrv/libslapd.so.0
>>>> #24 0x00007fc6b592c3d8 in do_bind ()
>>>> #25 0x00007fc6b5932974 in connection_threadmain ()
>>>> #26 0x00007fc6b3847cab in _pt_root () from /lib64/libnspr4.so
>>>> #27 0x00007fc6b31e852a in start_thread () from /lib64/libpthread.so.0
>>>> #28 0x00007fc6b2f2422d in clone () from /lib64/libc.so.6
>>>> Thread 25 (Thread 0x7fc67effd700 (LWP 1919)):
>>>> #0 0x00007fc6b31ed590 in pthread_cond_wait@@GLIBC_2.3.2 () from
>>>> /lib64/libpthread.so.0
>>>> #1 0x00007fc6b3842440 in PR_WaitCondVar () from /lib64/libnspr4.so
>>>> #2 0x00007fc6b5930b9e in connection_wait_for_new_work ()
>>>> #3 0x00007fc6b5931dc9 in connection_threadmain ()
>>>> #4 0x00007fc6b3847cab in _pt_root () from /lib64/libnspr4.so
>>>> #5 0x00007fc6b31e852a in start_thread () from /lib64/libpthread.so.0
>>>> #6 0x00007fc6b2f2422d in clone () from /lib64/libc.so.6
>>>> Thread 24 (Thread 0x7fc67e7fc700 (LWP 1920)):
>>>> #0 0x00007fc6b31ed590 in pthread_cond_wait@@GLIBC_2.3.2 () from
>>>> /lib64/libpthread.so.0
>>>> #1 0x00007fc6b3842440 in PR_WaitCondVar () from /lib64/libnspr4.so
>>>> #2 0x00007fc6b5930b9e in connection_wait_for_new_work ()
>>>> #3 0x00007fc6b5931dc9 in connection_threadmain ()
>>>> #4 0x00007fc6b3847cab in _pt_root () from /lib64/libnspr4.so
>>>> #5 0x00007fc6b31e852a in start_thread () from /lib64/libpthread.so.0
>>>> #6 0x00007fc6b2f2422d in clone () from /lib64/libc.so.6
>>>> Thread 23 (Thread 0x7fc67dffb700 (LWP 1921)):
>>>> #0 0x00007fc6b31ed590 in pthread_cond_wait@@GLIBC_2.3.2 () from
>>>> /lib64/libpthread.so.0
>>>> #1 0x00007fc6b3842440 in PR_WaitCondVar () from /lib64/libnspr4.so
>>>> #2 0x00007fc6b5930b9e in connection_wait_for_new_work ()
>>>> #3 0x00007fc6b5931dc9 in connection_threadmain ()
>>>> #4 0x00007fc6b3847cab in _pt_root () from /lib64/libnspr4.so
>>>> #5 0x00007fc6b31e852a in start_thread () from /lib64/libpthread.so.0
>>>> #6 0x00007fc6b2f2422d in clone () from /lib64/libc.so.6
>>>> Thread 22 (Thread 0x7fc67d7fa700 (LWP 1922)):
>>>> #0 0x00007fc6b31ed590 in pthread_cond_wait@@GLIBC_2.3.2 () from
>>>> /lib64/libpthread.so.0
>>>> #1 0x00007fc6b38426b3 in PR_EnterMonitor () from /lib64/libnspr4.so
>>>> #2 0x00007fc6a9622456 in dblayer_txn_begin () from
>>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so
>>>> #3 0x00007fc6a965d615 in ldbm_back_modify () from
>>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so
>>>> #4 0x00007fc6b544e905 in op_shared_modify () from
>>>> /usr/lib64/dirsrv/libslapd.so.0
>>>> #5 0x00007fc6b544f387 in modify_internal_pb () from
>>>> /usr/lib64/dirsrv/libslapd.so.0
>>>> #6 0x00007fc6aad02bd3 in ipalockout_postop () from
>>>> /usr/lib64/dirsrv/plugins/libipa_lockout.so
>>>> #7 0x00007fc6b546156f in plugin_call_func () from
>>>> /usr/lib64/dirsrv/libslapd.so.0
>>>> #8 0x00007fc6b5461893 in plugin_call_plugins () from
>>>> /usr/lib64/dirsrv/libslapd.so.0
>>>> #9 0x00007fc6b592c22d in do_bind ()
>>>> #10 0x00007fc6b5932974 in connection_threadmain ()
>>>> #11 0x00007fc6b3847cab in _pt_root () from /lib64/libnspr4.so
>>>> #12 0x00007fc6b31e852a in start_thread () from /lib64/libpthread.so.0
>>>> #13 0x00007fc6b2f2422d in clone () from /lib64/libc.so.6
>>>> Thread 21 (Thread 0x7fc67cff9700 (LWP 1923)):
>>>> #0 0x00007fc6b31ed590 in pthread_cond_wait@@GLIBC_2.3.2 () from
>>>> /lib64/libpthread.so.0
>>>> #1 0x00007fc6b38426b3 in PR_EnterMonitor () from /lib64/libnspr4.so
>>>> #2 0x00007fc6a9622456 in dblayer_txn_begin () from
>>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so
>>>> #3 0x00007fc6a965d615 in ldbm_back_modify () from
>>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so
>>>> #4 0x00007fc6b544e905 in op_shared_modify () from
>>>> /usr/lib64/dirsrv/libslapd.so.0
>>>> #5 0x00007fc6b544f387 in modify_internal_pb () from
>>>> /usr/lib64/dirsrv/libslapd.so.0
>>>> #6 0x00007fc6aad02bd3 in ipalockout_postop () from
>>>> /usr/lib64/dirsrv/plugins/libipa_lockout.so
>>>> #7 0x00007fc6b546156f in plugin_call_func () from
>>>> /usr/lib64/dirsrv/libslapd.so.0
>>>> #8 0x00007fc6b5461893 in plugin_call_plugins () from
>>>> /usr/lib64/dirsrv/libslapd.so.0
>>>> #9 0x00007fc6b592c22d in do_bind ()
>>>> #10 0x00007fc6b5932974 in connection_threadmain ()
>>>> #11 0x00007fc6b3847cab in _pt_root () from /lib64/libnspr4.so
>>>> #12 0x00007fc6b31e852a in start_thread () from /lib64/libpthread.so.0
>>>> #13 0x00007fc6b2f2422d in clone () from /lib64/libc.so.6
>>>> Thread 20 (Thread 0x7fc67c7f8700 (LWP 1924)):
>>>> #0 0x00007fc6b31ed590 in pthread_cond_wait@@GLIBC_2.3.2 () from
>>>> /lib64/libpthread.so.0
>>>> #1 0x00007fc6b3842440 in PR_WaitCondVar () from /lib64/libnspr4.so
>>>> #2 0x00007fc6b5930b9e in connection_wait_for_new_work ()
>>>> #3 0x00007fc6b5931dc9 in connection_threadmain ()
>>>> #4 0x00007fc6b3847cab in _pt_root () from /lib64/libnspr4.so
>>>> #5 0x00007fc6b31e852a in start_thread () from /lib64/libpthread.so.0
>>>> #6 0x00007fc6b2f2422d in clone () from /lib64/libc.so.6
>>>> Thread 19 (Thread 0x7fc67bff7700 (LWP 1925)):
>>>> #0 0x00007fc6b31ed590 in pthread_cond_wait@@GLIBC_2.3.2 () from
>>>> /lib64/libpthread.so.0
>>>> #1 0x00007fc6b38426b3 in PR_EnterMonitor () from /lib64/libnspr4.so
>>>> #2 0x00007fc6a9622456 in dblayer_txn_begin () from
>>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so
>>>> #3 0x00007fc6a965d615 in ldbm_back_modify () from
>>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so
>>>> #4 0x00007fc6b544e905 in op_shared_modify () from
>>>> /usr/lib64/dirsrv/libslapd.so.0
>>>> #5 0x00007fc6b544f387 in modify_internal_pb () from
>>>> /usr/lib64/dirsrv/libslapd.so.0
>>>> #6 0x00007fc6aad02bd3 in ipalockout_postop () from
>>>> /usr/lib64/dirsrv/plugins/libipa_lockout.so
>>>> #7 0x00007fc6b546156f in plugin_call_func () from
>>>> /usr/lib64/dirsrv/libslapd.so.0
>>>> #8 0x00007fc6b5461893 in plugin_call_plugins () from
>>>> /usr/lib64/dirsrv/libslapd.so.0
>>>> #9 0x00007fc6b592c22d in do_bind ()
>>>> #10 0x00007fc6b5932974 in connection_threadmain ()
>>>> #11 0x00007fc6b3847cab in _pt_root () from /lib64/libnspr4.so
>>>> #12 0x00007fc6b31e852a in start_thread () from /lib64/libpthread.so.0
>>>> #13 0x00007fc6b2f2422d in clone () from /lib64/libc.so.6
>>>> Thread 18 (Thread 0x7fc67b7f6700 (LWP 1926)):
>>>> #0 0x00007fc6b31ed590 in pthread_cond_wait@@GLIBC_2.3.2 () from
>>>> /lib64/libpthread.so.0
>>>> #1 0x00007fc6b3842440 in PR_WaitCondVar () from /lib64/libnspr4.so
>>>> #2 0x00007fc6b5930b9e in connection_wait_for_new_work ()
>>>> #3 0x00007fc6b5931dc9 in connection_threadmain ()
>>>> #4 0x00007fc6b3847cab in _pt_root () from /lib64/libnspr4.so
>>>> #5 0x00007fc6b31e852a in start_thread () from /lib64/libpthread.so.0
>>>> #6 0x00007fc6b2f2422d in clone () from /lib64/libc.so.6
>>>> Thread 17 (Thread 0x7fc67aff5700 (LWP 1927)):
>>>> #0 0x00007fc6b31ed590 in pthread_cond_wait@@GLIBC_2.3.2 () from
>>>> /lib64/libpthread.so.0
>>>> #1 0x00007fc6b3842440 in PR_WaitCondVar () from /lib64/libnspr4.so
>>>> #2 0x00007fc6b5930b9e in connection_wait_for_new_work ()
>>>> #3 0x00007fc6b5931dc9 in connection_threadmain ()
>>>> #4 0x00007fc6b3847cab in _pt_root () from /lib64/libnspr4.so
>>>> #5 0x00007fc6b31e852a in start_thread () from /lib64/libpthread.so.0
>>>> #6 0x00007fc6b2f2422d in clone () from /lib64/libc.so.6
>>>> Thread 16 (Thread 0x7fc67a7f4700 (LWP 1928)):
>>>> #0 0x00007fc6b31ed590 in pthread_cond_wait@@GLIBC_2.3.2 () from
>>>> /lib64/libpthread.so.0
>>>> #1 0x00007fc6b3842440 in PR_WaitCondVar () from /lib64/libnspr4.so
>>>> #2 0x00007fc6b5930b9e in connection_wait_for_new_work ()
>>>> #3 0x00007fc6b5931dc9 in connection_threadmain ()
>>>> #4 0x00007fc6b3847cab in _pt_root () from /lib64/libnspr4.so
>>>> #5 0x00007fc6b31e852a in start_thread () from /lib64/libpthread.so.0
>>>> #6 0x00007fc6b2f2422d in clone () from /lib64/libc.so.6
>>>> Thread 15 (Thread 0x7fc679ff3700 (LWP 1929)):
>>>> #0 0x00007fc6b31ed590 in pthread_cond_wait@@GLIBC_2.3.2 () from
>>>> /lib64/libpthread.so.0
>>>> #1 0x00007fc6b3842440 in PR_WaitCondVar () from /lib64/libnspr4.so
>>>> #2 0x00007fc6b5930b9e in connection_wait_for_new_work ()
>>>> #3 0x00007fc6b5931dc9 in connection_threadmain ()
>>>> #4 0x00007fc6b3847cab in _pt_root () from /lib64/libnspr4.so
>>>> #5 0x00007fc6b31e852a in start_thread () from /lib64/libpthread.so.0
>>>> #6 0x00007fc6b2f2422d in clone () from /lib64/libc.so.6
>>>> Thread 14 (Thread 0x7fc6797f2700 (LWP 1930)):
>>>> #0 0x00007fc6b31eff1d in __lll_lock_wait () from
>>>> /lib64/libpthread.so.0
>>>> #1 0x00007fc6b31ea9be in pthread_mutex_lock () from
>>>> /lib64/libpthread.so.0
>>>> #2 0x00007fc6b38420b9 in PR_Lock () from /lib64/libnspr4.so
>>>> #3 0x00007fc6a7ec99b0 in retrocl_postob () from
>>>> /usr/lib64/dirsrv/plugins/libretrocl-plugin.so
>>>> #4 0x00007fc6b546156f in plugin_call_func () from
>>>> /usr/lib64/dirsrv/libslapd.so.0
>>>> #5 0x00007fc6b5461893 in plugin_call_plugins () from
>>>> /usr/lib64/dirsrv/libslapd.so.0
>>>> #6 0x00007fc6a965d231 in ldbm_back_modify () from
>>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so
>>>> #7 0x00007fc6b544e905 in op_shared_modify () from
>>>> /usr/lib64/dirsrv/libslapd.so.0
>>>> #8 0x00007fc6b544f387 in modify_internal_pb () from
>>>> /usr/lib64/dirsrv/libslapd.so.0
>>>> #9 0x00007fc6a8d354a2 in memberof_fix_memberof_callback () from
>>>> /usr/lib64/dirsrv/plugins/libmemberof-plugin.so
>>>> #10 0x00007fc6a8d35f65 in memberof_modop_one_replace_r () from
>>>> /usr/lib64/dirsrv/plugins/libmemberof-plugin.so
>>>> #11 0x00007fc6a8d362a6 in memberof_mod_smod_list () from
>>>> /usr/lib64/dirsrv/plugins/libmemberof-plugin.so
>>>> #12 0x00007fc6a8d3758d in memberof_postop_modify () from
>>>> /usr/lib64/dirsrv/plugins/libmemberof-plugin.so
>>>> #13 0x00007fc6b546156f in plugin_call_func () from
>>>> /usr/lib64/dirsrv/libslapd.so.0
>>>> #14 0x00007fc6b5461893 in plugin_call_plugins () from
>>>> /usr/lib64/dirsrv/libslapd.so.0
>>>> #15 0x00007fc6a965d231 in ldbm_back_modify () from
>>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so
>>>> #16 0x00007fc6b544e905 in op_shared_modify () from
>>>> /usr/lib64/dirsrv/libslapd.so.0
>>>> #17 0x00007fc6b544fbe7 in do_modify () from
>>>> /usr/lib64/dirsrv/libslapd.so.0
>>>> #18 0x00007fc6b5932925 in connection_threadmain ()
>>>> #19 0x00007fc6b3847cab in _pt_root () from /lib64/libnspr4.so
>>>> #20 0x00007fc6b31e852a in start_thread () from /lib64/libpthread.so.0
>>>> #21 0x00007fc6b2f2422d in clone () from /lib64/libc.so.6
>>>> Thread 13 (Thread 0x7fc678ff1700 (LWP 1931)):
>>>> #0 0x00007fc6b31ed590 in pthread_cond_wait@@GLIBC_2.3.2 () from
>>>> /lib64/libpthread.so.0
>>>> #1 0x00007fc6adc811fb in __db_hybrid_mutex_suspend () from
>>>> /lib64/libdb-5.3.so
>>>> #2 0x00007fc6adc8059b in __db_tas_mutex_lock () from
>>>> /lib64/libdb-5.3.so
>>>> #3 0x00007fc6add2aa91 in __lock_get_internal () from
>>>> /lib64/libdb-5.3.so
>>>> #4 0x00007fc6add2b657 in __lock_get () from /lib64/libdb-5.3.so
>>>> #5 0x00007fc6add576af in __db_lget () from /lib64/libdb-5.3.so
>>>> #6 0x00007fc6adc9e4f0 in __bam_search () from /lib64/libdb-5.3.so
>>>> #7 0x00007fc6adc89144 in __bamc_search () from /lib64/libdb-5.3.so
>>>> #8 0x00007fc6adc8b074 in __bamc_get () from /lib64/libdb-5.3.so
>>>> #9 0x00007fc6add43ba3 in __dbc_iget () from /lib64/libdb-5.3.so
>>>> #10 0x00007fc6add53092 in __dbc_get_pp () from /lib64/libdb-5.3.so
>>>> #11 0x00007fc6a9672b70 in ldbm_back_seq () from
>>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so
>>>> #12 0x00007fc6b54650f9 in seq_internal_callback_pb () from
>>>> /usr/lib64/dirsrv/libslapd.so.0
>>>> #13 0x00007fc6b54652a9 in slapi_seq_callback () from
>>>> /usr/lib64/dirsrv/libslapd.so.0
>>>> #14 0x00007fc6a7ec8919 in retrocl_update_lastchangenumber () from
>>>> /usr/lib64/dirsrv/plugins/libretrocl-plugin.so
>>>> #15 0x00007fc6a7ec89bb in retrocl_assign_changenumber () from
>>>> /usr/lib64/dirsrv/plugins/libretrocl-plugin.so
>>>> #16 0x00007fc6a7ec99b5 in retrocl_postob () from
>>>> /usr/lib64/dirsrv/plugins/libretrocl-plugin.so
>>>> #17 0x00007fc6b546156f in plugin_call_func () from
>>>> /usr/lib64/dirsrv/libslapd.so.0
>>>> #18 0x00007fc6b5461893 in plugin_call_plugins () from
>>>> /usr/lib64/dirsrv/libslapd.so.0
>>>> #19 0x00007fc6a965d231 in ldbm_back_modify () from
>>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so
>>>> #20 0x00007fc6b544e905 in op_shared_modify () from
>>>> /usr/lib64/dirsrv/libslapd.so.0
>>>> #21 0x00007fc6b544fbe7 in do_modify () from
>>>> /usr/lib64/dirsrv/libslapd.so.0
>>>> #22 0x00007fc6b5932925 in connection_threadmain ()
>>>> #23 0x00007fc6b3847cab in _pt_root () from /lib64/libnspr4.so
>>>> #24 0x00007fc6b31e852a in start_thread () from /lib64/libpthread.so.0
>>>> #25 0x00007fc6b2f2422d in clone () from /lib64/libc.so.6
>>>> Thread 12 (Thread 0x7fc6787f0700 (LWP 1932)):
>>>> #0 0x00007fc6b31ed590 in pthread_cond_wait@@GLIBC_2.3.2 () from
>>>> /lib64/libpthread.so.0
>>>> #1 0x00007fc6b3842440 in PR_WaitCondVar () from /lib64/libnspr4.so
>>>> #2 0x00007fc6b5930b9e in connection_wait_for_new_work ()
>>>> #3 0x00007fc6b5931dc9 in connection_threadmain ()
>>>> #4 0x00007fc6b3847cab in _pt_root () from /lib64/libnspr4.so
>>>> #5 0x00007fc6b31e852a in start_thread () from /lib64/libpthread.so.0
>>>> #6 0x00007fc6b2f2422d in clone () from /lib64/libc.so.6
>>>> Thread 11 (Thread 0x7fc677fef700 (LWP 1933)):
>>>> #0 0x00007fc6b31ed590 in pthread_cond_wait@@GLIBC_2.3.2 () from
>>>> /lib64/libpthread.so.0
>>>> #1 0x00007fc6b3842440 in PR_WaitCondVar () from /lib64/libnspr4.so
>>>> #2 0x00007fc6b5930b9e in connection_wait_for_new_work ()
>>>> #3 0x00007fc6b5931dc9 in connection_threadmain ()
>>>> #4 0x00007fc6b3847cab in _pt_root () from /lib64/libnspr4.so
>>>> #5 0x00007fc6b31e852a in start_thread () from /lib64/libpthread.so.0
>>>> #6 0x00007fc6b2f2422d in clone () from /lib64/libc.so.6
>>>> Thread 10 (Thread 0x7fc6777ee700 (LWP 1934)):
>>>> #0 0x00007fc6b31ed590 in pthread_cond_wait@@GLIBC_2.3.2 () from
>>>> /lib64/libpthread.so.0
>>>> #1 0x00007fc6b38426b3 in PR_EnterMonitor () from /lib64/libnspr4.so
>>>> #2 0x00007fc6a9622456 in dblayer_txn_begin () from
>>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so
>>>> #3 0x00007fc6a965d615 in ldbm_back_modify () from
>>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so
>>>> #4 0x00007fc6b544e905 in op_shared_modify () from
>>>> /usr/lib64/dirsrv/libslapd.so.0
>>>> #5 0x00007fc6b544f387 in modify_internal_pb () from
>>>> /usr/lib64/dirsrv/libslapd.so.0
>>>> #6 0x00007fc6aad02bd3 in ipalockout_postop () from
>>>> /usr/lib64/dirsrv/plugins/libipa_lockout.so
>>>> #7 0x00007fc6b546156f in plugin_call_func () from
>>>> /usr/lib64/dirsrv/libslapd.so.0
>>>> #8 0x00007fc6b5461893 in plugin_call_plugins () from
>>>> /usr/lib64/dirsrv/libslapd.so.0
>>>> #9 0x00007fc6b592c22d in do_bind ()
>>>> #10 0x00007fc6b5932974 in connection_threadmain ()
>>>> #11 0x00007fc6b3847cab in _pt_root () from /lib64/libnspr4.so
>>>> #12 0x00007fc6b31e852a in start_thread () from /lib64/libpthread.so.0
>>>> #13 0x00007fc6b2f2422d in clone () from /lib64/libc.so.6
>>>> Thread 9 (Thread 0x7fc676fed700 (LWP 1935)):
>>>> #0 0x00007fc6b31ed590 in pthread_cond_wait@@GLIBC_2.3.2 () from
>>>> /lib64/libpthread.so.0
>>>> #1 0x00007fc6b38426b3 in PR_EnterMonitor () from /lib64/libnspr4.so
>>>> #2 0x00007fc6a9622456 in dblayer_txn_begin () from
>>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so
>>>> #3 0x00007fc6a965d615 in ldbm_back_modify () from
>>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so
>>>> #4 0x00007fc6b544e905 in op_shared_modify () from
>>>> /usr/lib64/dirsrv/libslapd.so.0
>>>> #5 0x00007fc6b544f387 in modify_internal_pb () from
>>>> /usr/lib64/dirsrv/libslapd.so.0
>>>> #6 0x00007fc6aad02bd3 in ipalockout_postop () from
>>>> /usr/lib64/dirsrv/plugins/libipa_lockout.so
>>>> #7 0x00007fc6b546156f in plugin_call_func () from
>>>> /usr/lib64/dirsrv/libslapd.so.0
>>>> #8 0x00007fc6b5461893 in plugin_call_plugins () from
>>>> /usr/lib64/dirsrv/libslapd.so.0
>>>> #9 0x00007fc6b592c22d in do_bind ()
>>>> #10 0x00007fc6b5932974 in connection_threadmain ()
>>>> #11 0x00007fc6b3847cab in _pt_root () from /lib64/libnspr4.so
>>>> #12 0x00007fc6b31e852a in start_thread () from /lib64/libpthread.so.0
>>>> #13 0x00007fc6b2f2422d in clone () from /lib64/libc.so.6
>>>> Thread 8 (Thread 0x7fc6767ec700 (LWP 1936)):
>>>> #0 0x00007fc6b31ed590 in pthread_cond_wait@@GLIBC_2.3.2 () from
>>>> /lib64/libpthread.so.0
>>>> #1 0x00007fc6b3842440 in PR_WaitCondVar () from /lib64/libnspr4.so
>>>> #2 0x00007fc6b5930b9e in connection_wait_for_new_work ()
>>>> #3 0x00007fc6b5931dc9 in connection_threadmain ()
>>>> #4 0x00007fc6b3847cab in _pt_root () from /lib64/libnspr4.so
>>>> #5 0x00007fc6b31e852a in start_thread () from /lib64/libpthread.so.0
>>>> #6 0x00007fc6b2f2422d in clone () from /lib64/libc.so.6
>>>> Thread 7 (Thread 0x7fc675feb700 (LWP 1937)):
>>>> #0 0x00007fc6b31ed590 in pthread_cond_wait@@GLIBC_2.3.2 () from
>>>> /lib64/libpthread.so.0
>>>> #1 0x00007fc6adc811fb in __db_hybrid_mutex_suspend () from
>>>> /lib64/libdb-5.3.so
>>>> #2 0x00007fc6adc8059b in __db_tas_mutex_lock () from
>>>> /lib64/libdb-5.3.so
>>>> #3 0x00007fc6add2aa91 in __lock_get_internal () from
>>>> /lib64/libdb-5.3.so
>>>> #4 0x00007fc6add2b657 in __lock_get () from /lib64/libdb-5.3.so
>>>> #5 0x00007fc6add576af in __db_lget () from /lib64/libdb-5.3.so
>>>> #6 0x00007fc6adc9d5d9 in __bam_get_root () from /lib64/libdb-5.3.so
>>>> #7 0x00007fc6adc9d91b in __bam_search () from /lib64/libdb-5.3.so
>>>> #8 0x00007fc6adc89144 in __bamc_search () from /lib64/libdb-5.3.so
>>>> #9 0x00007fc6adc8adff in __bamc_get () from /lib64/libdb-5.3.so
>>>> #10 0x00007fc6add43ba3 in __dbc_iget () from /lib64/libdb-5.3.so
>>>> #11 0x00007fc6add53092 in __dbc_get_pp () from /lib64/libdb-5.3.so
>>>> #12 0x00007fc6a962e6de in idl_new_fetch () from
>>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so
>>>> #13 0x00007fc6a963cc2b in index_read_ext_allids () from
>>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so
>>>> #14 0x00007fc6a96272ed in keys2idl () from
>>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so
>>>> #15 0x00007fc6a9627afe in ava_candidates.isra () from
>>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so
>>>> #16 0x00007fc6a96280fa in filter_candidates_ext () from
>>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so
>>>> #17 0x00007fc6a96290ce in list_candidates () from
>>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so
>>>> #18 0x00007fc6a9628062 in filter_candidates_ext () from
>>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so
>>>> #19 0x00007fc6a96631a7 in subtree_candidates () from
>>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so
>>>> #20 0x00007fc6a9664811 in ldbm_back_search () from
>>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so
>>>> #21 0x00007fc6b5455410 in op_shared_search () from
>>>> /usr/lib64/dirsrv/libslapd.so.0
>>>> #22 0x00007fc6b5943fc6 in do_search ()
>>>> #23 0x00007fc6b5932b4c in connection_threadmain ()
>>>> #24 0x00007fc6b3847cab in _pt_root () from /lib64/libnspr4.so
>>>> #25 0x00007fc6b31e852a in start_thread () from /lib64/libpthread.so.0
>>>> #26 0x00007fc6b2f2422d in clone () from /lib64/libc.so.6
>>>> Thread 6 (Thread 0x7fc6757ea700 (LWP 1938)):
>>>> #0 0x00007fc6b31ed590 in pthread_cond_wait@@GLIBC_2.3.2 () from
>>>> /lib64/libpthread.so.0
>>>> #1 0x00007fc6b3842440 in PR_WaitCondVar () from /lib64/libnspr4.so
>>>> #2 0x00007fc6b5930b9e in connection_wait_for_new_work ()
>>>> #3 0x00007fc6b5931dc9 in connection_threadmain ()
>>>> #4 0x00007fc6b3847cab in _pt_root () from /lib64/libnspr4.so
>>>> #5 0x00007fc6b31e852a in start_thread () from /lib64/libpthread.so.0
>>>> #6 0x00007fc6b2f2422d in clone () from /lib64/libc.so.6
>>>> Thread 5 (Thread 0x7fc674fe9700 (LWP 1939)):
>>>> #0 0x00007fc6b31ed590 in pthread_cond_wait@@GLIBC_2.3.2 () from
>>>> /lib64/libpthread.so.0
>>>> #1 0x00007fc6b3842440 in PR_WaitCondVar () from /lib64/libnspr4.so
>>>> #2 0x00007fc6b5930b9e in connection_wait_for_new_work ()
>>>> #3 0x00007fc6b5931dc9 in connection_threadmain ()
>>>> #4 0x00007fc6b3847cab in _pt_root () from /lib64/libnspr4.so
>>>> #5 0x00007fc6b31e852a in start_thread () from /lib64/libpthread.so.0
>>>> #6 0x00007fc6b2f2422d in clone () from /lib64/libc.so.6
>>>> Thread 4 (Thread 0x7fc6747e8700 (LWP 1940)):
>>>> #0 0x00007fc6b31ed590 in pthread_cond_wait@@GLIBC_2.3.2 () from
>>>> /lib64/libpthread.so.0
>>>> #1 0x00007fc6adc811fb in __db_hybrid_mutex_suspend () from
>>>> /lib64/libdb-5.3.so
>>>> #2 0x00007fc6adc8059b in __db_tas_mutex_lock () from
>>>> /lib64/libdb-5.3.so
>>>> #3 0x00007fc6add2aa91 in __lock_get_internal () from
>>>> /lib64/libdb-5.3.so
>>>> #4 0x00007fc6add2b657 in __lock_get () from /lib64/libdb-5.3.so
>>>> #5 0x00007fc6add576af in __db_lget () from /lib64/libdb-5.3.so
>>>> #6 0x00007fc6adc9d5d9 in __bam_get_root () from /lib64/libdb-5.3.so
>>>> #7 0x00007fc6adc9d91b in __bam_search () from /lib64/libdb-5.3.so
>>>> #8 0x00007fc6adc89144 in __bamc_search () from /lib64/libdb-5.3.so
>>>> #9 0x00007fc6adc8adff in __bamc_get () from /lib64/libdb-5.3.so
>>>> #10 0x00007fc6add43ba3 in __dbc_iget () from /lib64/libdb-5.3.so
>>>> #11 0x00007fc6add53092 in __dbc_get_pp () from /lib64/libdb-5.3.so
>>>> #12 0x00007fc6a962e6de in idl_new_fetch () from
>>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so
>>>> #13 0x00007fc6a963cc2b in index_read_ext_allids () from
>>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so
>>>> #14 0x00007fc6a96272ed in keys2idl () from
>>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so
>>>> #15 0x00007fc6a9627afe in ava_candidates.isra () from
>>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so
>>>> #16 0x00007fc6a96280fa in filter_candidates_ext () from
>>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so
>>>> #17 0x00007fc6a96290ce in list_candidates () from
>>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so
>>>> #18 0x00007fc6a9628062 in filter_candidates_ext () from
>>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so
>>>> #19 0x00007fc6a96631a7 in subtree_candidates () from
>>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so
>>>> #20 0x00007fc6a9664811 in ldbm_back_search () from
>>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so
>>>> #21 0x00007fc6b5455410 in op_shared_search () from
>>>> /usr/lib64/dirsrv/libslapd.so.0
>>>> #22 0x00007fc6b5943fc6 in do_search ()
>>>> #23 0x00007fc6b5932b4c in connection_threadmain ()
>>>> #24 0x00007fc6b3847cab in _pt_root () from /lib64/libnspr4.so
>>>> #25 0x00007fc6b31e852a in start_thread () from /lib64/libpthread.so.0
>>>> #26 0x00007fc6b2f2422d in clone () from /lib64/libc.so.6
>>>> Thread 3 (Thread 0x7fc673fe7700 (LWP 1941)):
>>>> #0 0x00007fc6b31ed590 in pthread_cond_wait@@GLIBC_2.3.2 () from
>>>> /lib64/libpthread.so.0
>>>> #1 0x00007fc6b3842440 in PR_WaitCondVar () from /lib64/libnspr4.so
>>>> #2 0x00007fc6b5930b9e in connection_wait_for_new_work ()
>>>> #3 0x00007fc6b5931dc9 in connection_threadmain ()
>>>> #4 0x00007fc6b3847cab in _pt_root () from /lib64/libnspr4.so
>>>> #5 0x00007fc6b31e852a in start_thread () from /lib64/libpthread.so.0
>>>> #6 0x00007fc6b2f2422d in clone () from /lib64/libc.so.6
>>>> Thread 2 (Thread 0x7fc6737e6700 (LWP 1942)):
>>>> #0 0x00007fc6b2f1aae3 in select () from /lib64/libc.so.6
>>>> #1 0x00007fc6b5492a99 in DS_Sleep () from
>>>> /usr/lib64/dirsrv/libslapd.so.0
>>>> #2 0x00007fc6b5933855 in time_thread ()
>>>> #3 0x00007fc6b3847cab in _pt_root () from /lib64/libnspr4.so
>>>> #4 0x00007fc6b31e852a in start_thread () from /lib64/libpthread.so.0
>>>> #5 0x00007fc6b2f2422d in clone () from /lib64/libc.so.6
>>>> Thread 1 (Thread 0x7fc6b58fb800 (LWP 1863)):
>>>> #0 0x00007fc6b2f18c8d in poll () from /lib64/libc.so.6
>>>> #1 0x00007fc6b3843da8 in _pr_poll_with_poll () from
>>>> /lib64/libnspr4.so
>>>> #2 0x00007fc6b5936b11 in slapd_daemon ()
>>>> #3 0x00007fc6b59294f4 in main ()
>>>>
>>>>
>>>>>> This replica is on virtual machine (ganeti). We had problems with
>>>>>> replication to vm, but after force-sync all was fine. On physical
>>>>>> servers all works fine.
>>>>> the messages indicate there could be many concurrent operations,
>>>>> because individual ops are not fast enough, could your VM have
>>>>> less/slower resources than the physical machines ?
>>>> VMs have less rsources and slower than physical machines.
>>>> VM: 4 cpu, hd via drbd on sas drives,
>>>>
>>>> physical: more cores, faster (ssd) drives
>>>>
>>>> Best regards,
>>>> Lukasz Jaworski 'Ender'
>>>>
>>>>
>>>>
>>>>>> Lukasz Jaworski 'Ender'
>>>>>>
>>>>>> Wiadomość napisana przez Ludwig Krispenz <lkrispen at redhat.com> w
>>>>>> dniu 6 maj 2015, o godz. 10:52:
>>>>>>
>>>>>>> Hi,
>>>>>>>
>>>>>>> there seem to be different issues,
>>>>>>> - I don't know what the ipactl status is looking for when it
>>>>>>> generates the error message about no matching master,
>>>>>>> but I don't think it is related to the retro changelog.
>>>>>>>
>>>>>>> - the retro changelog errors for adding and deleting
>>>>>>> -- the add failures are about aborted transactions because a
>>>>>>> page cannot be accessed, this maybe caused by concurrent mods on
>>>>>>> different backends, which want to update teh shared retro cl
>>>>>>> database.
>>>>>>> the changenumber reprted seems to be increasing, one error is
>>>>>>> about changenumber 44975, the next about 45577, so it looks like
>>>>>>> changes into the changelog are written and teh changenumber
>>>>>>> increases
>>>>>>> -- i'm not sure about the delete errors, but normally trimming
>>>>>>> would go on after such an error message, the changenumber
>>>>>>> attempted to delete are increasing.
>>>>>>> Could you verify which changes are in the changelog, and if
>>>>>>> these are changing:
>>>>>>> ldapsearch -b "cn=changelog" dn
>>>>>>>
>>>>>>> On 05/06/2015 09:52 AM, Łukasz Jaworski wrote:
>>>>>>>> Hi,
>>>>>>>>
>>>>>>>> One of our replica hanged up morning. Error log after dirsrv
>>>>>>>> restart:
>>>>>>>> [06/May/2015:09:28:15 +0200] - Retry count exceeded in delete
>>>>>>>> [06/May/2015:09:28:15 +0200] DSRetroclPlugin -
>>>>>>>> delete_changerecord: could not delete change record 38376 (rc: 51)
>>>>>>>> [06/May/2015:09:28:15 +0200] - Operation error fetching Null DN
>>>>>>>> (6368aeb7-f3c111e4-ae70ce39-9b469c1f), error -30993.
>>>>>>>> [06/May/2015:09:28:15 +0200] - dn2entry_ext: Failed to get id
>>>>>>>> for changenumber=44975,cn=changelog from entryrdn index (-30993)
>>>>>>>> [06/May/2015:09:28:15 +0200] - Operation error fetching
>>>>>>>> changenumber=44975,cn=changelog (null), error -30993.
>>>>>>>> [06/May/2015:09:28:15 +0200] DSRetroclPlugin - replog: an error
>>>>>>>> occured while adding change number 44975, dn =
>>>>>>>> changenumber=44975,cn=changelog: Operations error.
>>>>>>>> [06/May/2015:09:28:15 +0200] retrocl-plugin - retrocl_postob:
>>>>>>>> operation failure [1]
>>>>>>>> [06/May/2015:09:28:15 +0200] - ldbm_back_seq deadlock retry BAD
>>>>>>>> 1601, err=0 BDB0062 Successful return: 0
>>>>>>>> [06/May/2015:09:30:03 +0200] - ldbm_back_seq deadlock retry BAD
>>>>>>>> 1601, err=0 BDB0062 Successful return: 0
>>>>>>>> [06/May/2015:09:30:06 +0200] - Retry count exceeded in delete
>>>>>>>> [06/May/2015:09:30:06 +0200] DSRetroclPlugin -
>>>>>>>> delete_changerecord: could not delete change record 39297 (rc: 51)
>>>>>>>>
>>>>>>>> I did "re-initialize" from other replica.
>>>>>>>>
>>>>>>>> Now ipactl doesn't work. Shows: Configured hostname
>>>>>>>> 'replica09.local' does not match any master server in LDAP. On
>>>>>>>> lists replica09 is exists (twice)
>>>>>>>>
>>>>>>>> # ipactl status
>>>>>>>> Failed to get list of services to probe status!
>>>>>>>> Configured hostname 'replica09.local' does not match any master
>>>>>>>> server in LDAP:
>>>>>>>> replica01.local
>>>>>>>> replica02.local
>>>>>>>> replica03.local
>>>>>>>> replica04.local
>>>>>>>> replica05.local
>>>>>>>> replica06.local
>>>>>>>> replica07.local
>>>>>>>> replica08.local
>>>>>>>> replica09.local
>>>>>>>> replica10.local
>>>>>>>> replica09.local
>>>>>>>>
>>>>>>>> After dirsrv stop/start:
>>>>>>>>
>>>>>>>> In error logs there are many:
>>>>>>>> [06/May/2015:09:50:30 +0200] DSRetroclPlugin -
>>>>>>>> delete_changerecord: could not delete change record 39290 (rc: 32)
>>>>>>>> [06/May/2015:09:50:30 +0200] DSRetroclPlugin -
>>>>>>>> delete_changerecord: could not delete change record 39291 (rc: 32)
>>>>>>>> [06/May/2015:09:50:30 +0200] DSRetroclPlugin -
>>>>>>>> delete_changerecord: could not delete change record 39292 (rc: 32)
>>>>>>>> [06/May/2015:09:50:30 +0200] DSRetroclPlugin -
>>>>>>>> delete_changerecord: could not delete change record 39293 (rc: 32)
>>>>>>>> [06/May/2015:09:50:30 +0200] DSRetroclPlugin -
>>>>>>>> delete_changerecord: could not delete change record 39294 (rc: 32)
>>>>>>>> [06/May/2015:09:50:30 +0200] DSRetroclPlugin -
>>>>>>>> delete_changerecord: could not delete change record 39295 (rc: 32)
>>>>>>>> [06/May/2015:09:50:30 +0200] DSRetroclPlugin -
>>>>>>>> delete_changerecord: could not delete change record 39296 (rc: 32)
>>>>>>>> etc.
>>>>>>>>
>>>>>>>> [06/May/2015:09:51:08 +0200] - Operation error fetching Null DN
>>>>>>>> (9f51430a-f3c411e4-927ece39-9b469c1f), error -30993.
>>>>>>>> [06/May/2015:09:51:08 +0200] - dn2entry_ext: Failed to get id
>>>>>>>> for changenumber=45577,cn=changelog from entryrdn index (-30993)
>>>>>>>> [06/May/2015:09:51:08 +0200] - Operation error fetching
>>>>>>>> changenumber=45577,cn=changelog (null), error -30993.
>>>>>>>> [06/May/2015:09:51:08 +0200] DSRetroclPlugin - replog: an error
>>>>>>>> occured while adding change number 45577, dn =
>>>>>>>> changenumber=45577,cn=changelog: Operations error.
>>>>>>>> [06/May/2015:09:51:08 +0200] retrocl-plugin - retrocl_postob:
>>>>>>>> operation failure [1]
>>>>>>>> [06/May/2015:09:51:08 +0200] - ldbm_back_seq deadlock retry BAD
>>>>>>>> 1601, err=0 BDB0062 Successful return: 0
>>>>>>>>
>>>>>>>> Packages:
>>>>>>>> freeipa-server-4.1.3-2.fc21.x86_64
>>>>>>>> 389-ds-base-1.3.3.8-1.fc21.x86_64
>>>>>>>> 389-ds-base-libs-1.3.3.8-1.fc21.x86_64
>>>>>>>>
>>>>>>>> Best regards,
>>>>>>>> Ender
>>>>>>>>
>>>>>>> --
>>>>>>> Manage your subscription for the Freeipa-users mailing list:
>>>>>>> https://www.redhat.com/mailman/listinfo/freeipa-users
>>>>>>> Go to http://freeipa.org for more info on the project
>
--
Thank you,
Dmitri Pal
Director of Engineering for IdM portfolio
Red Hat, Inc.
More information about the Freeipa-users
mailing list