[Freeipa-users] Freeipa 4.2.0 hangs intermittently
Ludwig Krispenz
lkrispen at redhat.com
Tue Sep 6 10:43:50 UTC 2016
Hi,
I think you are running into this bug:
https://bugzilla.redhat.com/show_bug.cgi?id=1349779
it was the outcome of an analysis started in this one:
https://bugzilla.redhat.com/show_bug.cgi?id=1347800
the problem is that it looks like the checkpoint thread is holding a
mutex and otehr threads are blocked on it. It looks like an issue in
BerkeleyDB although there is no explanation from the BDB code why this
could happen, we also had a support case with Oracle to investigate
this, with no solution. Oracle at the end was providing a diagnostic
patch, which we did not test since the customer with the original
problem had worked around this by changing their configuration.
I don't know the status of this patch and if we can make it available,
probably depends on your support status, you need to contact support for
this.
About the workarounds:
you can try to change timing for the database update and checkpointing
eg by reducing the dbcache size, changing the checkpoint interval. In
the customer case they identified one substring index, which was
frequently updated but not really required - and removing this index helped.
Regards,
Ludwig
On 09/06/2016 11:31 AM, thierry bordaz wrote:
> Hi,
>
> The following thread is hanging most of the others threads
>
> Thread 20 (Thread 0x7f68007f8700 (LWP 4873)):
> #0 0x00007f683d0506d5 in pthread_cond_wait@@GLIBC_2.3.2 () from
> /lib64/libpthread.so.0
> #1 0x00007f68375ac2f3 in __db_hybrid_mutex_suspend () from
> /lib64/libdb-5.3.so
> #2 0x00007f68375ab640 in __db_tas_mutex_lock () from
> /lib64/libdb-5.3.so
> #3 0x00007f68376c5e1b in __memp_fget () from /lib64/libdb-5.3.so
> #4 0x00007f68375c9211 in __bam_search () from /lib64/libdb-5.3.so
> #5 0x00007f68375b4256 in __bamc_search () from /lib64/libdb-5.3.so
> #6 0x00007f68375b86c4 in __bamc_put () from /lib64/libdb-5.3.so
> #7 0x00007f683766f6d5 in __dbc_iput () from /lib64/libdb-5.3.so
> #8 0x00007f683766aa0e in __db_put () from /lib64/libdb-5.3.so
> #9 0x00007f683767ffa4 in __db_put_pp () from /lib64/libdb-5.3.so
> #10 0x00007f6832b5a4cf in idl_new_insert_key () from
> /usr/lib64/dirsrv/plugins/libback-ldbm.so
> #11 0x00007f6832b695b5 in addordel_values_sv.isra.3 () from
> /usr/lib64/dirsrv/plugins/libback-ldbm.so
> #12 0x00007f6832b6a25a in index_addordel_values_ext_sv () from
> /usr/lib64/dirsrv/plugins/libback-ldbm.so
> #13 0x00007f6832b6a694 in index_addordel_values_sv () from
> /usr/lib64/dirsrv/plugins/libback-ldbm.so
> #14 0x00007f6832b6a75a in index_addordel_entry () from
> /usr/lib64/dirsrv/plugins/libback-ldbm.so
> #15 0x00007f6832b6d5fa in ldbm_back_add () from
> /usr/lib64/dirsrv/plugins/libback-ldbm.so
> #16 0x00007f683f46fcca in op_shared_add () from
> /usr/lib64/dirsrv/libslapd.so.0
> #17 0x00007f683f471020 in do_add () from
> /usr/lib64/dirsrv/libslapd.so.0
> #18 0x00007f683f99c303 in connection_threadmain ()
> #19 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so
> #20 0x00007f683d04cdc5 in start_thread () from /lib64/libpthread.so.0
> #21 0x00007f683cd7a28d in clone () from /lib64/libc.so.6
>
> This thread is likely not hang on 'db_tas_mutex_lock' as the db_lock
> does not show it waiting for something.
> Unfortunately it is not shown in the output of top -H, did you get the
> complete output ?
> Ludwig mentioned a known issue that DS is not scaling well on NUMA
> arch, do you know if the machine is NUMA ?
> A possibility is that this thread get severely slowed down by a NUMA
> remote lock and showing DS has hung.
> This is why it would be interesting to know if it consums CPU.
>
> thanks
> theirry
>
> On 09/06/2016 10:59 AM, Rakesh Rajasekharan wrote:
>> Hi Thierry/Ludwig,
>>
>> I again hit the hang issue. This is the pstack of the ns-slapd
>>
>> So was able to this time get the pstack and db locks
>>
>> db locks details
>> Default locking region information:
>> 451 Last allocated locker ID
>> 0x7fffffff Current maximum unused locker ID
>> 9 Number of lock modes
>> 800 Initial number of locks allocated
>> 0 Initial number of lockers allocated
>> 800 Initial number of lock objects allocated
>> 10000 Maximum number of locks possible
>> 10000 Maximum number of lockers possible
>> 10000 Maximum number of lock objects possible
>> 800 Current number of locks allocated
>> 366 Current number of lockers allocated
>> 800 Current number of lock objects allocated
>> 160 Number of lock object partitions
>> 8191 Size of object hash table
>> 126 Number of current locks
>> 1005 Maximum number of locks at any one time
>> 33 Maximum number of locks in any one bucket
>> 3987 Maximum number of locks stolen by for an empty partition
>> 75 Maximum number of locks stolen for any one partition
>> 309 Number of current lockers
>> 313 Maximum number of lockers at any one time
>> 121 Number of current lock objects
>> 676 Maximum number of lock objects at any one time
>> 3 Maximum number of lock objects in any one bucket
>> 69 Maximum number of objects stolen by for an empty partition
>> 5 Maximum number of objects stolen for any one partition
>> 22M Total number of locks requested (22009642)
>> 21M Total number of locks released (21992878)
>> 0 Total number of locks upgraded
>> 95 Total number of locks downgraded
>> 3348 Lock requests not available due to conflicts, for which we
>> waited
>> 12646 Lock requests not available due to conflicts, for which we
>> did not wait
>> 0 Number of deadlocks
>> 0 Lock timeout value
>> 0 Number of locks that have timed out
>> 0 Transaction timeout value
>> 0 Number of transactions that have timed out
>> 2MB 560KB Region size
>> 71719 The number of partition locks that required waiting (0%)
>> 23105 The maximum number of times any partition lock was waited
>> for (0%)
>> 0 The number of object queue operations that required
>> waiting (0%)
>> 163 The number of locker allocations that required waiting (0%)
>> 3172 The number of region locks that required waiting (0%)
>> 5 Maximum hash bucket length
>> =-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
>> Lock REGINFO information:
>> Environment Region type
>> 1 Region ID
>> /var/lib/dirsrv/slapd-SPRINKLR-COM/db/__db.001 Region name
>> 0x7f67ef869000 Region address
>> 0x7f67ef8690a0 Region allocation head
>> 0x7f67ef8712b0 Region primary address
>> 0 Region maximum allocation
>> 0 Region allocated
>> Region allocations: 1581345 allocations, 0 failures, 1580794 frees, 6
>> longest
>> Allocations by power-of-two sizes:
>> 1KB 1581329
>> 2KB 3
>> 4KB 4
>> 8KB 3
>> 16KB 3
>> 32KB 1
>> 64KB 0
>> 128KB 0
>> 256KB 2
>> 512KB 0
>> 1024KB 1
>> REGION_SHARED Region flags
>> =-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
>> Lock region parameters:
>> 2 Lock region region mutex [3172/14M 0% !Own] <wakeups 0/1143>
>> 16381 locker table size
>> 8191 object table size
>> 34128 obj_off
>> 900216 locker_off
>> 0 need_dd
>> =-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
>> Lock conflict matrix:
>> =-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
>> Locks grouped by lockers:
>> Locker Mode Count Status ----------------- Object ---------------
>> 2 dd=308 locks held 1 write locks 0 pid/thread
>> 4845/140085720139840 flags 10 priority 100
>> 2 READ 1 HELD userRoot/id2entry.db handle 0
>> 3 dd=307 locks held 0 write locks 0 pid/thread
>> 4845/140084829472512 flags 0 priority 100
>> 4 dd=306 locks held 1 write locks 0 pid/thread
>> 4845/140085720139840 flags 10 priority 100
>> 4 READ 1 HELD ipaca/id2entry.db handle 0
>> 5 dd=305 locks held 0 write locks 0 pid/thread
>> 4845/140084837865216 flags 0 priority 100
>> 6 dd=304 locks held 1 write locks 0 pid/thread
>> 4845/140085720139840 flags 10 priority 100
>> 6 READ 1 HELD ipaca/entryrdn.db handle 0
>> 7 dd=303 locks held 0 write locks 0 pid/thread
>> 4845/140084804294400 flags 0 priority 100
>> 8 dd=302 locks held 0 write locks 0 pid/thread
>> 4845/140084720432896 flags 0 priority 100
>> 9 dd=301 locks held 0 write locks 0 pid/thread
>> 4845/140084837865216 flags 0 priority 100
>> a dd=300 locks held 1 write locks 0 pid/thread
>> 4845/140085720139840 flags 10 priority 100
>> a READ 1 HELD ipaca/vlv#allcertspkitomcatindex.db
>> handle 0
>> e dd=299 locks held 1 write locks 0 pid/thread
>> 4845/140085720139840 flags 10 priority 100
>> e READ 1 HELD
>> ipaca/vlv#allnonrevokedcertspkitomcatindex.db handle 0
>> 10 dd=298 locks held 1 write locks 0 pid/thread
>> 4845/140085720139840 flags 10 priority 100
>> 10 READ 1 HELD
>> ipaca/vlv#allrevokedcertspkitomcatindex.db handle 0
>> 11 dd=297 locks held 1 write locks 0 pid/thread
>> 4845/140085720139840 flags 10 priority 100
>> 11 READ 1 HELD
>> ipaca/vlv#allrevokedcertsnotafterpkitomcatindex.db handle 0
>> 14 dd=296 locks held 1 write locks 0 pid/thread
>> 4845/140085720139840 flags 10 priority 100
>> 14 READ 1 HELD
>> ipaca/vlv#allrevokedorrevokedexpiredcertspkitomcatindex.db
>> handle 0
>> 15 dd=295 locks held 1 write locks 0 pid/thread
>> 4845/140085720139840 flags 10 priority 100
>> 15 READ 1 HELD
>> ipaca/vlv#allvalidcertspkitomcatindex.db handle 0
>> 16 dd=294 locks held 1 write locks 0 pid/thread
>> 4845/140085720139840 flags 10 priority 100
>> 16 READ 1 HELD
>> ipaca/vlv#allvalidcertsnotafterpkitomcatindex.db handle 0
>> 17 dd=293 locks held 1 write locks 0 pid/thread
>> 4845/140085720139840 flags 10 priority 100
>> 17 READ 1 HELD
>> ipaca/vlv#allvalidorrevokedcertspkitomcatindex.db handle 0
>> 18 dd=292 locks held 1 write locks 0 pid/thread
>> 4845/140085720139840 flags 10 priority 100
>> 18 READ 1 HELD ipaca/vlv#caallpkitomcatindex.db
>> handle 0
>> 1d dd=291 locks held 1 write locks 0 pid/thread
>> 4845/140085720139840 flags 10 priority 100
>> 1d READ 1 HELD ipaca/vlv#cacompletepkitomcatindex.db
>> handle 0
>> 1e dd=290 locks held 1 write locks 0 pid/thread
>> 4845/140085720139840 flags 10 priority 100
>> 1e READ 1 HELD
>> ipaca/vlv#cacompleteenrollmentpkitomcatindex.db handle 0
>> 20 dd=289 locks held 1 write locks 0 pid/thread
>> 4845/140085720139840 flags 10 priority 100
>> 20 READ 1 HELD
>> ipaca/vlv#cacompleterevocationpkitomcatindex.db handle 0
>> 21 dd=288 locks held 1 write locks 0 pid/thread
>> 4845/140085720139840 flags 10 priority 100
>> 21 READ 1 HELD ipaca/vlv#caenrollmentpkitomcatindex.db
>> handle 0
>> 2b dd=287 locks held 1 write locks 0 pid/thread
>> 4845/140085720139840 flags 10 priority 100
>> 2b READ 1 HELD ipaca/vlv#carevocationpkitomcatindex.db
>> handle 0
>> 2c dd=286 locks held 1 write locks 0 pid/thread
>> 4845/140085720139840 flags 10 priority 100
>> 2c READ 1 HELD changelog/id2entry.db handle 0
>> 2d dd=285 locks held 0 write locks 0 pid/thread
>> 4845/140085719607040 flags 0 priority 100
>> 2e dd=284 locks held 1 write locks 0 pid/thread
>> 4845/140085720139840 flags 10 priority 100
>> 2e READ 1 HELD changelog/entryusn.db handle 0
>> 2f dd=283 locks held 0 write locks 0 pid/thread
>> 4845/140085720139840 flags 0 priority 100
>> 30 dd=282 locks held 1 write locks 0 pid/thread
>> 4845/140085720139840 flags 10 priority 100
>> 30 READ 1 HELD userRoot/entryusn.db handle 0
>> 31 dd=281 locks held 0 write locks 0 pid/thread
>> 4845/140085720139840 flags 0 priority 100
>> 32 dd=280 locks held 1 write locks 0 pid/thread
>> 4845/140085720139840 flags 10 priority 100
>> 32 READ 1 HELD ipaca/entryusn.db handle 0
>> 33 dd=279 locks held 0 write locks 0 pid/thread
>> 4845/140085720139840 flags 0 priority 100
>> 34 dd=278 locks held 1 write locks 0 pid/thread
>> 4845/140085720139840 flags 10 priority 100
>> 34 READ 1 HELD userRoot/entryrdn.db handle 0
>> 35 dd=277 locks held 0 write locks 0 pid/thread
>> 4845/140084544186112 flags 0 priority 100
>> 36 dd=276 locks held 0 write locks 0 pid/thread
>> 4845/140084636505856 flags 0 priority 100
>> 37 dd=275 locks held 0 write locks 0 pid/thread
>> 4845/140084854650624 flags 0 priority 100
>> 38 dd=274 locks held 1 write locks 0 pid/thread
>> 4845/140085720139840 flags 10 priority 100
>> 38 READ 1 HELD userRoot/objectclass.db handle 0
>> 39 dd=273 locks held 0 write locks 0 pid/thread
>> 4845/140084569364224 flags 0 priority 100
>> 3a dd=272 locks held 1 write locks 0 pid/thread
>> 4845/140085720139840 flags 10 priority 100
>> 3a READ 1 HELD userRoot/ancestorid.db handle 0
>> 3b dd=271 locks held 0 write locks 0 pid/thread
>> 4845/140084854650624 flags 0 priority 100
>> 3c dd=270 locks held 1 write locks 0 pid/thread
>> 4845/140085720139840 flags 10 priority 100
>> 3c READ 1 HELD changelog/entryrdn.db handle 0
>> 3d dd=269 locks held 0 write locks 0 pid/thread
>> 4845/140085719607040 flags 0 priority 100
>> 3e dd=268 locks held 0 write locks 0 pid/thread
>> 4845/140084602935040 flags 0 priority 100
>> 3f dd=267 locks held 0 write locks 0 pid/thread
>> 4845/140085719607040 flags 0 priority 100
>> 40 dd=266 locks held 1 write locks 0 pid/thread
>> 4845/140085720139840 flags 10 priority 100
>> 40 READ 1 HELD changelog/objectclass.db handle 0
>> 41 dd=265 locks held 0 write locks 0 pid/thread
>> 4845/140085719607040 flags 0 priority 100
>> 42 dd=264 locks held 0 write locks 0 pid/thread
>> 4845/140085719607040 flags 0 priority 100
>> 43 dd=263 locks held 0 write locks 0 pid/thread
>> 4845/140084653291264 flags 0 priority 100
>> 44 dd=262 locks held 1 write locks 0 pid/thread
>> 4845/140085720139840 flags 10 priority 100
>> 44 READ 1 HELD ipaca/objectclass.db handle 0
>> 45 dd=261 locks held 0 write locks 0 pid/thread
>> 4845/140084577756928 flags 0 priority 100
>> 46 dd=260 locks held 0 write locks 0 pid/thread
>> 4845/140084720432896 flags 0 priority 100
>> 47 dd=259 locks held 1 write locks 0 pid/thread
>> 4845/140085720139840 flags 10 priority 100
>> 47 READ 1 HELD changelog/aci.db handle 0
>> 48 dd=258 locks held 0 write locks 0 pid/thread
>> 4845/140085720139840 flags 0 priority 100
>> 49 dd=257 locks held 1 write locks 0 pid/thread
>> 4845/140085720139840 flags 10 priority 100
>> 49 READ 1 HELD userRoot/aci.db handle 0
>> 4a dd=256 locks held 0 write locks 0 pid/thread
>> 4845/140085720139840 flags 0 priority 100
>> 4b dd=255 locks held 0 write locks 0 pid/thread
>> 4845/140084535793408 flags 0 priority 100
>> 4c dd=254 locks held 1 write locks 0 pid/thread
>> 4845/140085720139840 flags 10 priority 100
>> 4c READ 1 HELD ipaca/aci.db handle 0
>> 4d dd=253 locks held 0 write locks 0 pid/thread
>> 4845/140085720139840 flags 0 priority 100
>> 4e dd=252 locks held 1 write locks 0 pid/thread
>> 4845/140085720139840 flags 10 priority 100
>> 4e READ 1 HELD userRoot/parentid.db handle 0
>> 4f dd=251 locks held 0 write locks 0 pid/thread
>> 4845/140085720139840 flags 0 priority 100
>> 50 dd=250 locks held 0 write locks 0 pid/thread
>> 4845/140085719607040 flags 0 priority 100
>> 51 dd=249 locks held 1 write locks 0 pid/thread
>> 4845/140085720139840 flags 10 priority 100
>> 51 READ 1 HELD changelog/nsuniqueid.db handle 0
>> 52 dd=248 locks held 1 write locks 0 pid/thread
>> 4845/140085720139840 flags 10 priority 100
>> 52 READ 1 HELD changelog/changenumber.db handle 0
>> 53 dd=247 locks held 0 write locks 0 pid/thread
>> 4845/140084602935040 flags 0 priority 100
>> 54 dd=246 locks held 0 write locks 0 pid/thread
>> 4845/140085719607040 flags 0 priority 100
>> 55 dd=245 locks held 1 write locks 0 pid/thread
>> 4845/140085719607040 flags 10 priority 100
>> 55 READ 1 HELD changelog/targetuniqueid.db handle 0
>> 56 dd=244 locks held 1 write locks 0 pid/thread
>> 4845/140085719607040 flags 10 priority 100
>> 56 READ 1 HELD changelog/parentid.db handle 0
>> 57 dd=243 locks held 1 write locks 0 pid/thread
>> 4845/140085719607040 flags 10 priority 100
>> 57 READ 1 HELD changelog/ancestorid.db handle 0
>> 58 dd=242 locks held 1 write locks 0 pid/thread
>> 4845/140085719607040 flags 10 priority 100
>> 58 READ 1 HELD changelog/numsubordinates.db
>> handle 0
>> 59 dd=241 locks held 0 write locks 0 pid/thread
>> 4845/140085719607040 flags 0 priority 100
>> 5a dd=240 locks held 0 write locks 0 pid/thread
>> 4845/140084544186112 flags 0 priority 100
>> 5b dd=239 locks held 1 write locks 0 pid/thread
>> 4845/140084577756928 flags 10 priority 100
>> 5b READ 1 HELD userRoot/krbPrincipalName.db
>> handle 0
>> 5c dd=238 locks held 0 write locks 0 pid/thread
>> 4845/140084636505856 flags 0 priority 100
>> 5d dd=237 locks held 0 write locks 0 pid/thread
>> 4845/140084695254784 flags 0 priority 100
>> 5e dd=236 locks held 1 write locks 0 pid/thread
>> 4845/140084560971520 flags 10 priority 100
>> 5e READ 1 HELD userRoot/ipakrbprincipalalias.db
>> handle 0
>> 5f dd=235 locks held 0 write locks 0 pid/thread
>> 4845/140084703647488 flags 0 priority 100
>> 60 dd=234 locks held 0 write locks 0 pid/thread
>> 4845/140084653291264 flags 0 priority 100
>> 61 dd=233 locks held 0 write locks 0 pid/thread
>> 4845/140084577756928 flags 0 priority 100
>> 62 dd=232 locks held 0 write locks 0 pid/thread
>> 4845/140084686862080 flags 0 priority 100
>> 63 dd=231 locks held 1 write locks 0 pid/thread
>> 4845/140084611327744 flags 10 priority 100
>> 63 READ 1 HELD changelog/seeAlso.db handle 0
>> 64 dd=230 locks held 0 write locks 0 pid/thread
>> 4845/140084686862080 flags 0 priority 100
>> 65 dd=229 locks held 1 write locks 0 pid/thread
>> 4845/140084611327744 flags 10 priority 100
>> 65 READ 1 HELD userRoot/seeAlso.db handle 0
>> 66 dd=228 locks held 0 write locks 0 pid/thread
>> 4845/140084636505856 flags 0 priority 100
>> 67 dd=227 locks held 1 write locks 0 pid/thread
>> 4845/140084611327744 flags 10 priority 100
>> 67 READ 1 HELD ipaca/seeAlso.db handle 0
>> 68 dd=226 locks held 0 write locks 0 pid/thread
>> 4845/140084686862080 flags 0 priority 100
>> 69 dd=225 locks held 1 write locks 0 pid/thread
>> 4845/140084678469376 flags 10 priority 100
>> 69 READ 1 HELD ipaca/parentid.db handle 0
>> 6a dd=224 locks held 0 write locks 0 pid/thread
>> 4845/140084678469376 flags 0 priority 100
>> 6b dd=223 locks held 0 write locks 0 pid/thread
>> 4845/140084829472512 flags 0 priority 100
>> 6c dd=222 locks held 0 write locks 0 pid/thread
>> 4845/140084653291264 flags 0 priority 100
>> 6d dd=221 locks held 0 write locks 0 pid/thread
>> 4845/140084569364224 flags 0 priority 100
>> 6e dd=220 locks held 0 write locks 0 pid/thread
>> 4845/140084636505856 flags 0 priority 100
>> 6e READ 1 WAIT userRoot/objectclass.db page 18
>> 6f dd=219 locks held 0 write locks 0 pid/thread
>> 4845/140084544186112 flags 0 priority 100
>> 70 dd=218 locks held 0 write locks 0 pid/thread
>> 4845/140084569364224 flags 0 priority 100
>> 71 dd=217 locks held 0 write locks 0 pid/thread
>> 4845/140084619720448 flags 0 priority 100
>> 72 dd=216 locks held 0 write locks 0 pid/thread
>> 4845/140084552578816 flags 0 priority 100
>> 73 dd=215 locks held 0 write locks 0 pid/thread
>> 4845/140084829472512 flags 0 priority 100
>> 74 dd=214 locks held 0 write locks 0 pid/thread
>> 4845/140084628113152 flags 0 priority 100
>> 75 dd=213 locks held 0 write locks 0 pid/thread
>> 4845/140084686862080 flags 0 priority 100
>> 76 dd=212 locks held 1 write locks 0 pid/thread
>> 4845/140084560971520 flags 10 priority 100
>> 76 READ 1 HELD userRoot/ipauniqueid.db handle 0
>> 77 dd=211 locks held 0 write locks 0 pid/thread
>> 4845/140084661683968 flags 0 priority 100
>> 78 dd=210 locks held 0 write locks 0 pid/thread
>> 4845/140084846257920 flags 0 priority 100
>> 79 dd=209 locks held 0 write locks 0 pid/thread
>> 4845/140084628113152 flags 0 priority 100
>> 7a dd=208 locks held 1 write locks 0 pid/thread
>> 4845/140084560971520 flags 10 priority 100
>> 7a READ 1 HELD userRoot/nsuniqueid.db handle 0
>> 7b dd=207 locks held 1 write locks 0 pid/thread
>> 4845/140084560971520 flags 10 priority 100
>> 7b READ 1 HELD userRoot/cn.db handle 0
>> 7c dd=206 locks held 1 write locks 0 pid/thread
>> 4845/140084560971520 flags 10 priority 100
>> 7c READ 1 HELD userRoot/fqdn.db handle 0
>> 7d dd=205 locks held 1 write locks 0 pid/thread
>> 4845/140084560971520 flags 10 priority 100
>> 7d READ 1 HELD userRoot/managedby.db handle 0
>> 7e dd=204 locks held 1 write locks 0 pid/thread
>> 4845/140084560971520 flags 10 priority 100
>> 7e READ 1 HELD userRoot/numsubordinates.db handle 0
>> 7f dd=203 locks held 1 write locks 0 pid/thread
>> 4845/140084678469376 flags 10 priority 100
>> 7f READ 1 HELD userRoot/member.db handle 0
>> 80 dd=202 locks held 0 write locks 0 pid/thread
>> 4845/140084712040192 flags 0 priority 100
>> 81 dd=201 locks held 1 write locks 0 pid/thread
>> 4845/140084678469376 flags 10 priority 100
>> 81 READ 1 HELD userRoot/memberUser.db handle 0
>> 82 dd=200 locks held 0 write locks 0 pid/thread
>> 4845/140084577756928 flags 0 priority 100
>> 83 dd=199 locks held 1 write locks 0 pid/thread
>> 4845/140084678469376 flags 10 priority 100
>> 83 READ 1 HELD userRoot/memberHost.db handle 0
>> 84 dd=198 locks held 0 write locks 0 pid/thread
>> 4845/140084712040192 flags 0 priority 100
>> 85 dd=197 locks held 0 write locks 0 pid/thread
>> 4845/140084577756928 flags 0 priority 100
>> 86 dd=196 locks held 0 write locks 0 pid/thread
>> 4845/140084670076672 flags 0 priority 100
>> 87 dd=195 locks held 0 write locks 0 pid/thread
>> 4845/140084577756928 flags 0 priority 100
>> 88 dd=194 locks held 0 write locks 0 pid/thread
>> 4845/140084846257920 flags 0 priority 100
>> 89 dd=193 locks held 0 write locks 0 pid/thread
>> 4845/140084552578816 flags 0 priority 100
>> 8a dd=192 locks held 0 write locks 0 pid/thread
>> 4845/140084703647488 flags 0 priority 100
>> 8b dd=191 locks held 0 write locks 0 pid/thread
>> 4845/140084544186112 flags 0 priority 100
>> 8c dd=190 locks held 0 write locks 0 pid/thread
>> 4845/140084846257920 flags 0 priority 100
>> 8d dd=189 locks held 0 write locks 0 pid/thread
>> 4845/140084661683968 flags 0 priority 100
>> 8e dd=188 locks held 1 write locks 0 pid/thread
>> 4845/140085067675392 flags 10 priority 100
>> 8e READ 1 HELD userRoot/macAddress.db handle 0
>> 8f dd=187 locks held 0 write locks 0 pid/thread
>> 4845/140085067675392 flags 0 priority 100
>> 90 dd=186 locks held 0 write locks 0 pid/thread
>> 4845/140084703647488 flags 0 priority 100
>> 91 dd=185 locks held 0 write locks 0 pid/thread
>> 4845/140084552578816 flags 0 priority 100
>> 92 dd=184 locks held 0 write locks 0 pid/thread
>> 4845/140084560971520 flags 0 priority 100
>> 92 READ 1 WAIT userRoot/objectclass.db page 9
>> 93 dd=183 locks held 0 write locks 0 pid/thread
>> 4845/140084577756928 flags 0 priority 100
>> 94 dd=182 locks held 0 write locks 0 pid/thread
>> 4845/140084821079808 flags 0 priority 100
>> 94 READ 1 WAIT userRoot/objectclass.db page 18
>> 95 dd=181 locks held 0 write locks 0 pid/thread
>> 4845/140084544186112 flags 0 priority 100
>> 96 dd=180 locks held 0 write locks 0 pid/thread
>> 4845/140084804294400 flags 0 priority 100
>> 97 dd=179 locks held 1 write locks 0 pid/thread
>> 4845/140084712040192 flags 10 priority 100
>> 97 READ 1 HELD userRoot/uid.db handle 0
>> 98 dd=178 locks held 0 write locks 0 pid/thread
>> 4845/140084552578816 flags 0 priority 100
>> 99 dd=177 locks held 0 write locks 0 pid/thread
>> 4845/140084577756928 flags 0 priority 100
>> 9a dd=176 locks held 0 write locks 0 pid/thread
>> 4845/140084602935040 flags 0 priority 100
>> 9b dd=175 locks held 1 write locks 0 pid/thread
>> 4845/140085076068096 flags 10 priority 100
>> 9b READ 1 HELD userRoot/uniquemember.db handle 0
>> 9c dd=174 locks held 1 write locks 0 pid/thread
>> 4845/140085076068096 flags 10 priority 100
>> 9c READ 1 HELD userRoot/owner.db handle 0
>> 9d dd=173 locks held 1 write locks 0 pid/thread
>> 4845/140085076068096 flags 10 priority 100
>> 9d READ 1 HELD userRoot/ipaallowedtarget.db
>> handle 0
>> 9e dd=172 locks held 1 write locks 0 pid/thread
>> 4845/140085076068096 flags 10 priority 100
>> 9e READ 1 HELD userRoot/ipaMemberCertProfile.db
>> handle 0
>> 9f dd=171 locks held 1 write locks 0 pid/thread
>> 4845/140085076068096 flags 10 priority 100
>> 9f READ 1 HELD userRoot/ipaassignedidview.db
>> handle 0
>> a0 dd=170 locks held 1 write locks 0 pid/thread
>> 4845/140085076068096 flags 10 priority 100
>> a0 READ 1 HELD userRoot/secretary.db handle 0
>> a1 dd=169 locks held 1 write locks 0 pid/thread
>> 4845/140085076068096 flags 10 priority 100
>> a1 READ 1 HELD userRoot/memberdenycmd.db handle 0
>> a2 dd=168 locks held 1 write locks 0 pid/thread
>> 4845/140085076068096 flags 10 priority 100
>> a2 READ 1 HELD userRoot/memberallowcmd.db handle 0
>> a3 dd=167 locks held 1 write locks 0 pid/thread
>> 4845/140085076068096 flags 10 priority 100
>> a3 READ 1 HELD userRoot/manager.db handle 0
>> a4 dd=166 locks held 1 write locks 0 pid/thread
>> 4845/140085076068096 flags 10 priority 100
>> a4 READ 1 HELD userRoot/ipasudorunas.db handle 0
>> a5 dd=165 locks held 1 write locks 0 pid/thread
>> 4845/140085076068096 flags 10 priority 100
>> a5 READ 1 HELD userRoot/ipaMemberCa.db handle 0
>> a6 dd=164 locks held 1 write locks 0 pid/thread
>> 4845/140085076068096 flags 10 priority 100
>> a6 READ 1 HELD userRoot/ipasudorunasgroup.db
>> handle 0
>> a7 dd=163 locks held 1 write locks 0 pid/thread
>> 4845/140085076068096 flags 10 priority 100
>> a7 READ 1 HELD userRoot/sourcehost.db handle 0
>> a8 dd=162 locks held 1 write locks 0 pid/thread
>> 4845/140085076068096 flags 10 priority 100
>> a8 READ 1 HELD userRoot/memberservice.db handle 0
>> a9 dd=161 locks held 1 write locks 0 pid/thread
>> 4845/140085076068096 flags 10 priority 100
>> a9 READ 1 HELD userRoot/ipatokenradiusconfiglink.db
>> handle 0
>> aa dd=160 locks held 0 write locks 0 pid/thread
>> 4845/140084703647488 flags 0 priority 100
>> ab dd=159 locks held 0 write locks 0 pid/thread
>> 4845/140084821079808 flags 0 priority 100
>> ac dd=158 locks held 0 write locks 0 pid/thread
>> 4845/140084628113152 flags 0 priority 100
>> ad dd=157 locks held 0 write locks 0 pid/thread
>> 4845/140084829472512 flags 0 priority 100
>> ae dd=156 locks held 0 write locks 0 pid/thread
>> 4845/140084829472512 flags 0 priority 100
>> af dd=155 locks held 0 write locks 0 pid/thread
>> 4845/140084854650624 flags 0 priority 100
>> b0 dd=154 locks held 0 write locks 0 pid/thread
>> 4845/140084854650624 flags 0 priority 100
>> b1 dd=153 locks held 0 write locks 0 pid/thread
>> 4845/140084837865216 flags 0 priority 100
>> b2 dd=152 locks held 0 write locks 0 pid/thread
>> 4845/140084804294400 flags 0 priority 100
>> b3 dd=151 locks held 0 write locks 0 pid/thread
>> 4845/140084812687104 flags 0 priority 100
>> b4 dd=150 locks held 0 write locks 0 pid/thread
>> 4845/140084804294400 flags 0 priority 100
>> b5 dd=149 locks held 0 write locks 0 pid/thread
>> 4845/140084837865216 flags 0 priority 100
>> b6 dd=148 locks held 0 write locks 0 pid/thread
>> 4845/140084569364224 flags 0 priority 100
>> b7 dd=147 locks held 0 write locks 0 pid/thread
>> 4845/140084829472512 flags 0 priority 100
>> b8 dd=146 locks held 0 write locks 0 pid/thread
>> 4845/140084586149632 flags 0 priority 100
>> b9 dd=145 locks held 0 write locks 0 pid/thread
>> 4845/140084653291264 flags 0 priority 100
>> ba dd=144 locks held 0 write locks 0 pid/thread
>> 4845/140084636505856 flags 0 priority 100
>> bb dd=143 locks held 0 write locks 0 pid/thread
>> 4845/140084804294400 flags 0 priority 100
>> bc dd=142 locks held 0 write locks 0 pid/thread
>> 4845/140084804294400 flags 0 priority 100
>> bd dd=141 locks held 0 write locks 0 pid/thread
>> 4845/140084695254784 flags 0 priority 100
>> be dd=140 locks held 0 write locks 0 pid/thread
>> 4845/140084670076672 flags 0 priority 100
>> bf dd=139 locks held 0 write locks 0 pid/thread
>> 4845/140084628113152 flags 0 priority 100
>> c0 dd=138 locks held 0 write locks 0 pid/thread
>> 4845/140084661683968 flags 0 priority 100
>> c1 dd=137 locks held 0 write locks 0 pid/thread
>> 4845/140084712040192 flags 0 priority 100
>> c2 dd=136 locks held 0 write locks 0 pid/thread
>> 4845/140084661683968 flags 0 priority 100
>> c3 dd=135 locks held 0 write locks 0 pid/thread
>> 4845/140084577756928 flags 0 priority 100
>> c4 dd=134 locks held 0 write locks 0 pid/thread
>> 4845/140084854650624 flags 0 priority 100
>> c5 dd=133 locks held 1 write locks 0 pid/thread
>> 4845/140084644898560 flags 10 priority 100
>> c5 READ 1 HELD userRoot/uidnumber.db handle 0
>> c6 dd=132 locks held 0 write locks 0 pid/thread
>> 4845/140084594542336 flags 0 priority 100
>> c8 dd=131 locks held 1 write locks 0 pid/thread
>> 4845/140084678469376 flags 10 priority 100
>> c8 READ 1 HELD ipaca/certstatus.db handle 0
>> c9 dd=130 locks held 0 write locks 0 pid/thread
>> 4845/140084720432896 flags 0 priority 100
>> ca dd=129 locks held 0 write locks 0 pid/thread
>> 4845/140084619720448 flags 0 priority 100
>> cb dd=128 locks held 0 write locks 0 pid/thread
>> 4845/140084821079808 flags 0 priority 100
>> cc dd=127 locks held 0 write locks 0 pid/thread
>> 4845/140084594542336 flags 0 priority 100
>> cd dd=126 locks held 0 write locks 0 pid/thread
>> 4845/140084560971520 flags 0 priority 100
>> ce dd=125 locks held 1 write locks 0 pid/thread
>> 4845/140084837865216 flags 10 priority 100
>> ce READ 1 HELD userRoot/gidnumber.db handle 0
>> cf dd=124 locks held 0 write locks 0 pid/thread
>> 4845/140084720432896 flags 0 priority 100
>> d0 dd=123 locks held 0 write locks 0 pid/thread
>> 4845/140084569364224 flags 0 priority 100
>> d1 dd=122 locks held 1 write locks 0 pid/thread
>> 4845/140084678469376 flags 10 priority 100
>> d1 READ 1 HELD userRoot/nscpEntryDN.db handle 0
>> d2 dd=121 locks held 0 write locks 0 pid/thread
>> 4845/140084695254784 flags 0 priority 100
>> d3 dd=120 locks held 0 write locks 0 pid/thread
>> 4845/140084569364224 flags 0 priority 100
>> d4 dd=119 locks held 0 write locks 0 pid/thread
>> 4845/140084653291264 flags 0 priority 100
>> d5 dd=118 locks held 0 write locks 0 pid/thread
>> 4845/140084829472512 flags 0 priority 100
>> d6 dd=117 locks held 0 write locks 0 pid/thread
>> 4845/140084661683968 flags 0 priority 100
>> d7 dd=116 locks held 0 write locks 0 pid/thread
>> 4845/140084804294400 flags 0 priority 100
>> d8 dd=115 locks held 1 write locks 0 pid/thread
>> 4845/140084821079808 flags 10 priority 100
>> d8 READ 1 HELD ipaca/description.db handle 0
>> d9 dd=114 locks held 0 write locks 0 pid/thread
>> 4845/140084661683968 flags 0 priority 100
>> da dd=113 locks held 0 write locks 0 pid/thread
>> 4845/140084837865216 flags 0 priority 100
>> db dd=112 locks held 1 write locks 0 pid/thread
>> 4845/140084678469376 flags 10 priority 100
>> db READ 1 HELD ipaca/cn.db handle 0
>> dc dd=111 locks held 0 write locks 0 pid/thread
>> 4845/140084577756928 flags 0 priority 100
>> dd dd=110 locks held 1 write locks 0 pid/thread
>> 4845/140084678469376 flags 10 priority 100
>> dd READ 1 HELD ipaca/ancestorid.db handle 0
>> de dd=109 locks held 0 write locks 0 pid/thread
>> 4845/140084577756928 flags 0 priority 100
>> df dd=108 locks held 0 write locks 0 pid/thread
>> 4845/140084703647488 flags 0 priority 100
>> e0 dd=107 locks held 0 write locks 0 pid/thread
>> 4845/140084569364224 flags 0 priority 100
>> e1 dd=106 locks held 0 write locks 0 pid/thread
>> 4845/140084569364224 flags 0 priority 100
>> e2 dd=105 locks held 0 write locks 0 pid/thread
>> 4845/140084821079808 flags 0 priority 100
>> e3 dd=104 locks held 1 write locks 0 pid/thread
>> 4845/140084686862080 flags 10 priority 100
>> e3 READ 1 HELD ipaca/serialno.db handle 0
>> e4 dd=103 locks held 0 write locks 0 pid/thread
>> 4845/140084837865216 flags 0 priority 100
>> e5 dd=102 locks held 1 write locks 0 pid/thread
>> 4845/140084695254784 flags 10 priority 100
>> e5 READ 1 HELD ipaca/revInfo.db handle 0
>> e6 dd=101 locks held 1 write locks 0 pid/thread
>> 4845/140084695254784 flags 10 priority 100
>> e6 READ 1 HELD ipaca/revokedby.db handle 0
>> e7 dd=100 locks held 1 write locks 0 pid/thread
>> 4845/140084695254784 flags 10 priority 100
>> e7 READ 1 HELD ipaca/revokedOn.db handle 0
>> e8 dd=99 locks held 0 write locks 0 pid/thread
>> 4845/140084829472512 flags 0 priority 100
>> e9 dd=98 locks held 0 write locks 0 pid/thread
>> 4845/140084846257920 flags 0 priority 100
>> ea dd=97 locks held 1 write locks 0 pid/thread
>> 4845/140084703647488 flags 10 priority 100
>> ea READ 1 HELD ipaca/nsuniqueid.db handle 0
>> eb dd=96 locks held 1 write locks 0 pid/thread
>> 4845/140084703647488 flags 10 priority 100
>> eb READ 1 HELD ipaca/requestid.db handle 0
>> ec dd=95 locks held 1 write locks 0 pid/thread
>> 4845/140084703647488 flags 10 priority 100
>> ec READ 1 HELD ipaca/requeststate.db handle 0
>> ed dd=94 locks held 1 write locks 0 pid/thread
>> 4845/140084703647488 flags 10 priority 100
>> ed READ 1 HELD ipaca/dateOfCreate.db handle 0
>> ee dd=93 locks held 1 write locks 0 pid/thread
>> 4845/140084703647488 flags 10 priority 100
>> ee READ 1 HELD ipaca/requesttype.db handle 0
>> ef dd=92 locks held 1 write locks 0 pid/thread
>> 4845/140084703647488 flags 10 priority 100
>> ef READ 1 HELD ipaca/numsubordinates.db handle 0
>> f0 dd=91 locks held 1 write locks 0 pid/thread
>> 4845/140084611327744 flags 10 priority 100
>> f0 READ 1 HELD userRoot/userCertificate.db handle 0
>> f1 dd=90 locks held 0 write locks 0 pid/thread
>> 4845/140084821079808 flags 0 priority 100
>> f2 dd=89 locks held 0 write locks 0 pid/thread
>> 4845/140084569364224 flags 0 priority 100
>> f3 dd=88 locks held 0 write locks 0 pid/thread
>> 4845/140084602935040 flags 0 priority 100
>> f4 dd=87 locks held 0 write locks 0 pid/thread
>> 4845/140084720432896 flags 0 priority 100
>> f5 dd=86 locks held 0 write locks 0 pid/thread
>> 4845/140084628113152 flags 0 priority 100
>> f6 dd=85 locks held 0 write locks 0 pid/thread
>> 4845/140084703647488 flags 0 priority 100
>> f7 dd=84 locks held 0 write locks 0 pid/thread
>> 4845/140084611327744 flags 0 priority 100
>> f8 dd=83 locks held 0 write locks 0 pid/thread
>> 4845/140084661683968 flags 0 priority 100
>> f9 dd=82 locks held 0 write locks 0 pid/thread
>> 4845/140084837865216 flags 0 priority 100
>> fa dd=81 locks held 0 write locks 0 pid/thread
>> 4845/140084544186112 flags 0 priority 100
>> fb dd=80 locks held 0 write locks 0 pid/thread
>> 4845/140084703647488 flags 0 priority 100
>> fc dd=79 locks held 0 write locks 0 pid/thread
>> 4845/140084544186112 flags 0 priority 100
>> fd dd=78 locks held 0 write locks 0 pid/thread
>> 4845/140084535793408 flags 0 priority 100
>> fe dd=77 locks held 0 write locks 0 pid/thread
>> 4845/140084586149632 flags 0 priority 100
>> ff dd=76 locks held 0 write locks 0 pid/thread
>> 4845/140084569364224 flags 0 priority 100
>> 100 dd=75 locks held 0 write locks 0 pid/thread
>> 4845/140084804294400 flags 0 priority 100
>> 101 dd=74 locks held 0 write locks 0 pid/thread
>> 4845/140084703647488 flags 0 priority 100
>> 102 dd=73 locks held 0 write locks 0 pid/thread
>> 4845/140084703647488 flags 0 priority 100
>> 103 dd=72 locks held 0 write locks 0 pid/thread
>> 4845/140084821079808 flags 0 priority 100
>> 104 dd=71 locks held 0 write locks 0 pid/thread
>> 4845/140084695254784 flags 0 priority 100
>> 105 dd=70 locks held 0 write locks 0 pid/thread
>> 4845/140084544186112 flags 0 priority 100
>> 106 dd=69 locks held 0 write locks 0 pid/thread
>> 4845/140084854650624 flags 0 priority 100
>> 107 dd=68 locks held 0 write locks 0 pid/thread
>> 4845/140084628113152 flags 0 priority 100
>> 108 dd=67 locks held 0 write locks 0 pid/thread
>> 4845/140084569364224 flags 0 priority 100
>> 109 dd=66 locks held 0 write locks 0 pid/thread
>> 4845/140084594542336 flags 0 priority 100
>> 10a dd=65 locks held 0 write locks 0 pid/thread
>> 4845/140084661683968 flags 0 priority 100
>> 10b dd=64 locks held 0 write locks 0 pid/thread
>> 4845/140084653291264 flags 0 priority 100
>> 10c dd=63 locks held 0 write locks 0 pid/thread
>> 4845/140084695254784 flags 0 priority 100
>> 10d dd=62 locks held 0 write locks 0 pid/thread
>> 4845/140084720432896 flags 0 priority 100
>> 10e dd=61 locks held 0 write locks 0 pid/thread
>> 4845/140084619720448 flags 0 priority 100
>> 10f dd=60 locks held 0 write locks 0 pid/thread
>> 4845/140084720432896 flags 0 priority 100
>> 110 dd=59 locks held 0 write locks 0 pid/thread
>> 4845/140084569364224 flags 0 priority 100
>> 111 dd=58 locks held 0 write locks 0 pid/thread
>> 4845/140084695254784 flags 0 priority 100
>> 112 dd=57 locks held 0 write locks 0 pid/thread
>> 4845/140084653291264 flags 0 priority 100
>> 113 dd=56 locks held 0 write locks 0 pid/thread
>> 4845/140084712040192 flags 0 priority 100
>> 114 dd=55 locks held 0 write locks 0 pid/thread
>> 4845/140084686862080 flags 0 priority 100
>> 115 dd=54 locks held 0 write locks 0 pid/thread
>> 4845/140084577756928 flags 0 priority 100
>> 115 READ 1 WAIT userRoot/objectclass.db page 9
>> 116 dd=53 locks held 0 write locks 0 pid/thread
>> 4845/140084837865216 flags 0 priority 100
>> 116 READ 1 WAIT userRoot/objectclass.db page 9
>> 117 dd=52 locks held 0 write locks 0 pid/thread
>> 4845/140084577756928 flags 0 priority 100
>> 118 dd=51 locks held 0 write locks 0 pid/thread
>> 4845/140084569364224 flags 0 priority 100
>> 119 dd=50 locks held 0 write locks 0 pid/thread
>> 4845/140084560971520 flags 0 priority 100
>> 11a dd=49 locks held 1 write locks 0 pid/thread
>> 4845/140084695254784 flags 10 priority 100
>> 11a READ 1 HELD ipaca/metaInfo.db handle 0
>> 11b dd=48 locks held 1 write locks 0 pid/thread
>> 4845/140084695254784 flags 10 priority 100
>> 11b READ 1 HELD ipaca/notbefore.db handle 0
>> 11c dd=47 locks held 1 write locks 0 pid/thread
>> 4845/140084695254784 flags 10 priority 100
>> 11c READ 1 HELD ipaca/notafter.db handle 0
>> 11d dd=46 locks held 1 write locks 0 pid/thread
>> 4845/140084695254784 flags 10 priority 100
>> 11d READ 1 HELD ipaca/duration.db handle 0
>> 11e dd=45 locks held 1 write locks 0 pid/thread
>> 4845/140084695254784 flags 10 priority 100
>> 11e READ 1 HELD ipaca/subjectname.db handle 0
>> 11f dd=44 locks held 1 write locks 0 pid/thread
>> 4845/140084695254784 flags 10 priority 100
>> 11f READ 1 HELD ipaca/publicKeyData.db handle 0
>> 120 dd=43 locks held 1 write locks 0 pid/thread
>> 4845/140084695254784 flags 10 priority 100
>> 120 READ 1 HELD ipaca/extension.db handle 0
>> 121 dd=42 locks held 1 write locks 0 pid/thread
>> 4845/140084695254784 flags 10 priority 100
>> 121 READ 1 HELD ipaca/issuedby.db handle 0
>> 122 dd=41 locks held 0 write locks 0 pid/thread
>> 4845/140084712040192 flags 0 priority 100
>> 123 dd=40 locks held 0 write locks 0 pid/thread
>> 4845/140084812687104 flags 0 priority 100
>> 124 dd=39 locks held 0 write locks 0 pid/thread
>> 4845/140084544186112 flags 0 priority 100
>> 125 dd=38 locks held 0 write locks 0 pid/thread
>> 4845/140084686862080 flags 0 priority 100
>> 126 dd=37 locks held 0 write locks 0 pid/thread
>> 4845/140084552578816 flags 0 priority 100
>> 128 dd=36 locks held 0 write locks 0 pid/thread
>> 4845/140084686862080 flags 0 priority 100
>> 12a dd=35 locks held 0 write locks 0 pid/thread
>> 4845/140084720432896 flags 0 priority 100
>> 12b dd=34 locks held 0 write locks 0 pid/thread
>> 4845/140084552578816 flags 0 priority 100
>> 12c dd=33 locks held 0 write locks 0 pid/thread
>> 4845/140084670076672 flags 0 priority 100
>> 12d dd=32 locks held 0 write locks 0 pid/thread
>> 4845/140084577756928 flags 0 priority 100
>> 12e dd=31 locks held 0 write locks 0 pid/thread
>> 4845/140084712040192 flags 0 priority 100
>> 12f dd=30 locks held 0 write locks 0 pid/thread
>> 4845/140084670076672 flags 0 priority 100
>> 130 dd=29 locks held 0 write locks 0 pid/thread
>> 4845/140084577756928 flags 0 priority 100
>> 131 dd=28 locks held 0 write locks 0 pid/thread
>> 4845/140084560971520 flags 0 priority 100
>> 132 dd=27 locks held 0 write locks 0 pid/thread
>> 4845/140084670076672 flags 0 priority 100
>> 133 dd=26 locks held 0 write locks 0 pid/thread
>> 4845/140084602935040 flags 0 priority 100
>> 134 dd=25 locks held 0 write locks 0 pid/thread
>> 4845/140084552578816 flags 0 priority 100
>> 135 dd=24 locks held 0 write locks 0 pid/thread
>> 4845/140084678469376 flags 0 priority 100
>> 136 dd=23 locks held 0 write locks 0 pid/thread
>> 4845/140084544186112 flags 0 priority 100
>> 137 dd=22 locks held 0 write locks 0 pid/thread
>> 4845/140084577756928 flags 0 priority 100
>> 138 dd=21 locks held 0 write locks 0 pid/thread
>> 4845/140084703647488 flags 0 priority 100
>> 139 dd=20 locks held 0 write locks 0 pid/thread
>> 4845/140084678469376 flags 0 priority 100
>> 13b dd=19 locks held 0 write locks 0 pid/thread
>> 4845/140084812687104 flags 0 priority 100
>> 144 dd=18 locks held 0 write locks 0 pid/thread
>> 4845/140084670076672 flags 0 priority 100
>> 156 dd=17 locks held 0 write locks 0 pid/thread
>> 4845/140084594542336 flags 0 priority 100
>> 15a dd=16 locks held 0 write locks 0 pid/thread
>> 4845/140084577756928 flags 0 priority 100
>> 168 dd=15 locks held 0 write locks 0 pid/thread
>> 4845/140084653291264 flags 0 priority 100
>> 16a dd=14 locks held 0 write locks 0 pid/thread
>> 4845/140084829472512 flags 0 priority 100
>> 17d dd=13 locks held 0 write locks 0 pid/thread
>> 4845/140084829472512 flags 0 priority 100
>> 17e dd=12 locks held 0 write locks 0 pid/thread
>> 4845/140084829472512 flags 0 priority 100
>> 17f dd=11 locks held 0 write locks 0 pid/thread
>> 4845/140084586149632 flags 0 priority 100
>> 180 dd=10 locks held 0 write locks 0 pid/thread
>> 4845/140084594542336 flags 0 priority 100
>> 181 dd= 9 locks held 0 write locks 0 pid/thread
>> 4845/140084577756928 flags 0 priority 100
>> 193 dd= 8 locks held 0 write locks 0 pid/thread
>> 4845/140084812687104 flags 0 priority 100
>> 196 dd= 7 locks held 0 write locks 0 pid/thread
>> 4845/140084695254784 flags 0 priority 100
>> 1a1 dd= 6 locks held 0 write locks 0 pid/thread
>> 4845/140084720432896 flags 0 priority 100
>> 1a2 dd= 5 locks held 0 write locks 0 pid/thread
>> 4845/140084577756928 flags 0 priority 100
>> 1a3 dd= 4 locks held 0 write locks 0 pid/thread
>> 4845/140084812687104 flags 0 priority 100
>> 1a4 dd= 3 locks held 0 write locks 0 pid/thread
>> 4845/140084812687104 flags 0 priority 100
>> 1a5 dd= 2 locks held 0 write locks 0 pid/thread
>> 4845/140084577756928 flags 0 priority 100
>> 1a6 dd= 1 locks held 0 write locks 0 pid/thread
>> 4845/140084594542336 flags 0 priority 100
>> 800b3a33 dd= 0 locks held 26 write locks 22 pid/thread
>> 4845/140084661683968 flags 0 priority 100
>> 800b3a33 WRITE 1 HELD userRoot/objectclass.db page 9
>> 800b3a33 WRITE 1 HELD userRoot/objectclass.db page 8
>> 800b3a33 WRITE 1 HELD userRoot/objectclass.db page 18
>> 800b3a33 WRITE 1 HELD userRoot/objectclass.db page 7
>> 800b3a33 WRITE 1 HELD userRoot/objectclass.db page 14
>> 800b3a33 WRITE 1 HELD userRoot/objectclass.db page 11
>> 800b3a33 WRITE 2 HELD userRoot/objectclass.db page 3
>> 800b3a33 WRITE 1 HELD userRoot/cn.db page 150
>> 800b3a33 WRITE 1 HELD userRoot/cn.db page 89
>> 800b3a33 WRITE 1 HELD userRoot/cn.db page 53
>> 800b3a33 WRITE 1 HELD userRoot/cn.db page 131
>> 800b3a33 WRITE 1 HELD userRoot/cn.db page 117
>> 800b3a33 WRITE 1 HELD userRoot/cn.db page 11
>> 800b3a33 WRITE 2 HELD userRoot/cn.db page 141
>> 800b3a33 WRITE 1 HELD userRoot/cn.db page 41
>> 800b3a33 WRITE 1 HELD userRoot/cn.db page 40
>> 800b3a33 WRITE 1 HELD userRoot/cn.db page 143
>> 800b3a33 WRITE 1 HELD userRoot/cn.db page 30
>> 800b3a33 WRITE 2 HELD userRoot/id2entry.db page 8184
>> 800b3a33 WRITE 1 HELD userRoot/id2entry.db page 8183
>> 800b3a33 WRITE 1 HELD userRoot/id2entry.db page 7779
>> 800b3a33 WRITE 1 HELD userRoot/id2entry.db page 0
>> 800b3a33 READ 3 HELD userRoot/entryrdn.db page 3
>> 800b3a33 READ 1 HELD userRoot/entryrdn.db page 36
>> 800b3a33 READ 1 HELD userRoot/entryrdn.db page 4
>> 800b3a33 READ 1 HELD userRoot/nsuniqueid.db page 86
>> =-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
>> Locks grouped by object:
>> Locker Mode Count Status ----------------- Object ---------------
>> a4 READ 1 HELD userRoot/ipasudorunas.db handle 0
>>
>> 9e READ 1 HELD userRoot/ipaMemberCertProfile.db
>> handle 0
>>
>> 10 READ 1 HELD
>> ipaca/vlv#allrevokedcertspkitomcatindex.db handle 0
>>
>> 800b3a33 WRITE 2 HELD userRoot/id2entry.db page 8184
>>
>> 800b3a33 WRITE 1 HELD userRoot/id2entry.db page 8183
>>
>> 800b3a33 WRITE 1 HELD userRoot/id2entry.db page 7779
>>
>> a1 READ 1 HELD userRoot/memberdenycmd.db handle 0
>>
>> a READ 1 HELD ipaca/vlv#allcertspkitomcatindex.db
>> handle 0
>>
>> a7 READ 1 HELD userRoot/sourcehost.db handle 0
>>
>> 57 READ 1 HELD changelog/ancestorid.db handle 0
>>
>> e5 READ 1 HELD ipaca/revInfo.db handle 0
>>
>> e6 READ 1 HELD ipaca/revokedby.db handle 0
>>
>> 11 READ 1 HELD
>> ipaca/vlv#allrevokedcertsnotafterpkitomcatindex.db handle 0
>>
>> 4e READ 1 HELD userRoot/parentid.db handle 0
>>
>> 4 READ 1 HELD ipaca/id2entry.db handle 0
>>
>> 58 READ 1 HELD changelog/numsubordinates.db
>> handle 0
>>
>> 67 READ 1 HELD ipaca/seeAlso.db handle 0
>>
>> 2b READ 1 HELD ipaca/vlv#carevocationpkitomcatindex.db
>> handle 0
>>
>> ec READ 1 HELD ipaca/requeststate.db handle 0
>>
>> 11b READ 1 HELD ipaca/notbefore.db handle 0
>>
>> 7e READ 1 HELD userRoot/numsubordinates.db handle 0
>>
>> a2 READ 1 HELD userRoot/memberallowcmd.db handle 0
>>
>> 9c READ 1 HELD userRoot/owner.db handle 0
>>
>> 1e READ 1 HELD
>> ipaca/vlv#cacompleteenrollmentpkitomcatindex.db handle 0
>>
>> 11c READ 1 HELD ipaca/notafter.db handle 0
>>
>> 40 READ 1 HELD changelog/objectclass.db handle 0
>>
>> 2e READ 1 HELD changelog/entryusn.db handle 0
>>
>> 16 READ 1 HELD
>> ipaca/vlv#allvalidcertsnotafterpkitomcatindex.db handle 0
>>
>> 15 READ 1 HELD
>> ipaca/vlv#allvalidcertspkitomcatindex.db handle 0
>>
>> a5 READ 1 HELD userRoot/ipaMemberCa.db handle 0
>>
>> 18 READ 1 HELD ipaca/vlv#caallpkitomcatindex.db
>> handle 0
>>
>> a0 READ 1 HELD userRoot/secretary.db handle 0
>>
>> 81 READ 1 HELD userRoot/memberUser.db handle 0
>>
>> 9f READ 1 HELD userRoot/ipaassignedidview.db
>> handle 0
>>
>> a3 READ 1 HELD userRoot/manager.db handle 0
>>
>> d8 READ 1 HELD ipaca/description.db handle 0
>>
>> a8 READ 1 HELD userRoot/memberservice.db handle 0
>>
>> 44 READ 1 HELD ipaca/objectclass.db handle 0
>>
>> ed READ 1 HELD ipaca/dateOfCreate.db handle 0
>>
>> 800b3a33 WRITE 1 HELD userRoot/id2entry.db page 0
>>
>> 2 READ 1 HELD userRoot/id2entry.db handle 0
>>
>> 5b READ 1 HELD userRoot/krbPrincipalName.db
>> handle 0
>>
>> ee READ 1 HELD ipaca/requesttype.db handle 0
>>
>> c8 READ 1 HELD ipaca/certstatus.db handle 0
>>
>> 56 READ 1 HELD changelog/parentid.db handle 0
>>
>> 83 READ 1 HELD userRoot/memberHost.db handle 0
>>
>> 5e READ 1 HELD userRoot/ipakrbprincipalalias.db
>> handle 0
>>
>> 121 READ 1 HELD ipaca/issuedby.db handle 0
>>
>> 6 READ 1 HELD ipaca/entryrdn.db handle 0
>>
>> e7 READ 1 HELD ipaca/revokedOn.db handle 0
>>
>> 4c READ 1 HELD ipaca/aci.db handle 0
>>
>> 11d READ 1 HELD ipaca/duration.db handle 0
>>
>> 14 READ 1 HELD
>> ipaca/vlv#allrevokedorrevokedexpiredcertspkitomcatindex.db
>> handle 0
>>
>> 21 READ 1 HELD ipaca/vlv#caenrollmentpkitomcatindex.db
>> handle 0
>>
>> 55 READ 1 HELD changelog/targetuniqueid.db handle 0
>>
>> 800b3a33 READ 1 HELD userRoot/entryrdn.db page 36
>>
>> 2c READ 1 HELD changelog/id2entry.db handle 0
>>
>> 800b3a33 READ 1 HELD userRoot/entryrdn.db page 4
>>
>> 34 READ 1 HELD userRoot/entryrdn.db handle 0
>>
>> 800b3a33 READ 3 HELD userRoot/entryrdn.db page 3
>>
>> eb READ 1 HELD ipaca/requestid.db handle 0
>>
>> 49 READ 1 HELD userRoot/aci.db handle 0
>>
>> 76 READ 1 HELD userRoot/ipauniqueid.db handle 0
>>
>> 63 READ 1 HELD changelog/seeAlso.db handle 0
>>
>> 7c READ 1 HELD userRoot/fqdn.db handle 0
>>
>> db READ 1 HELD ipaca/cn.db handle 0
>>
>> ef READ 1 HELD ipaca/numsubordinates.db handle 0
>>
>> 32 READ 1 HELD ipaca/entryusn.db handle 0
>>
>> 1d READ 1 HELD ipaca/vlv#cacompletepkitomcatindex.db
>> handle 0
>>
>> dd READ 1 HELD ipaca/ancestorid.db handle 0
>>
>> a9 READ 1 HELD userRoot/ipatokenradiusconfiglink.db
>> handle 0
>>
>> f0 READ 1 HELD userRoot/userCertificate.db handle 0
>>
>> 11a READ 1 HELD ipaca/metaInfo.db handle 0
>>
>> 7d READ 1 HELD userRoot/managedby.db handle 0
>>
>> ea READ 1 HELD ipaca/nsuniqueid.db handle 0
>>
>> c5 READ 1 HELD userRoot/uidnumber.db handle 0
>>
>> d1 READ 1 HELD userRoot/nscpEntryDN.db handle 0
>>
>> 120 READ 1 HELD ipaca/extension.db handle 0
>>
>> 800b3a33 WRITE 1 HELD userRoot/cn.db page 150
>>
>> 69 READ 1 HELD ipaca/parentid.db handle 0
>>
>> 800b3a33 WRITE 1 HELD userRoot/cn.db page 131
>>
>> 800b3a33 WRITE 1 HELD userRoot/cn.db page 143
>>
>> 800b3a33 WRITE 2 HELD userRoot/cn.db page 141
>>
>> 20 READ 1 HELD
>> ipaca/vlv#cacompleterevocationpkitomcatindex.db handle 0
>>
>> 8e READ 1 HELD userRoot/macAddress.db handle 0
>>
>> e READ 1 HELD
>> ipaca/vlv#allnonrevokedcertspkitomcatindex.db handle 0
>>
>> 17 READ 1 HELD
>> ipaca/vlv#allvalidorrevokedcertspkitomcatindex.db handle 0
>>
>> 800b3a33 WRITE 1 HELD userRoot/cn.db page 30
>>
>> 800b3a33 WRITE 1 HELD userRoot/objectclass.db page 14
>>
>> 800b3a33 WRITE 1 HELD userRoot/objectclass.db page 9
>> 116 READ 1 WAIT userRoot/objectclass.db page 9
>> 115 READ 1 WAIT userRoot/objectclass.db page 9
>> 92 READ 1 WAIT userRoot/objectclass.db page 9
>>
>> 800b3a33 WRITE 1 HELD userRoot/objectclass.db page 8
>>
>> 7b READ 1 HELD userRoot/cn.db handle 0
>>
>> 800b3a33 WRITE 1 HELD userRoot/objectclass.db page 11
>>
>> 800b3a33 WRITE 1 HELD userRoot/objectclass.db page 7
>>
>> 800b3a33 WRITE 1 HELD userRoot/cn.db page 11
>>
>> 38 READ 1 HELD userRoot/objectclass.db handle 0
>>
>> 800b3a33 WRITE 2 HELD userRoot/objectclass.db page 3
>>
>> 800b3a33 WRITE 1 HELD userRoot/cn.db page 53
>>
>> 800b3a33 WRITE 1 HELD userRoot/objectclass.db page 18
>> 6e READ 1 WAIT userRoot/objectclass.db page 18
>> 94 READ 1 WAIT userRoot/objectclass.db page 18
>>
>> 800b3a33 WRITE 1 HELD userRoot/cn.db page 41
>>
>> 800b3a33 WRITE 1 HELD userRoot/cn.db page 40
>>
>> 800b3a33 WRITE 1 HELD userRoot/cn.db page 89
>>
>> 800b3a33 WRITE 1 HELD userRoot/cn.db page 117
>>
>> 30 READ 1 HELD userRoot/entryusn.db handle 0
>>
>> 3c READ 1 HELD changelog/entryrdn.db handle 0
>>
>> 97 READ 1 HELD userRoot/uid.db handle 0
>>
>> 47 READ 1 HELD changelog/aci.db handle 0
>>
>> e3 READ 1 HELD ipaca/serialno.db handle 0
>>
>> 7a READ 1 HELD userRoot/nsuniqueid.db handle 0
>>
>> 800b3a33 READ 1 HELD userRoot/nsuniqueid.db page 86
>>
>> 51 READ 1 HELD changelog/nsuniqueid.db handle 0
>>
>> 11e READ 1 HELD ipaca/subjectname.db handle 0
>>
>> 7f READ 1 HELD userRoot/member.db handle 0
>>
>> 11f READ 1 HELD ipaca/publicKeyData.db handle 0
>>
>> 9b READ 1 HELD userRoot/uniquemember.db handle 0
>>
>> 52 READ 1 HELD changelog/changenumber.db handle 0
>>
>> ce READ 1 HELD userRoot/gidnumber.db handle 0
>>
>> 65 READ 1 HELD userRoot/seeAlso.db handle 0
>>
>> 9d READ 1 HELD userRoot/ipaallowedtarget.db
>> handle 0
>>
>> a6 READ 1 HELD userRoot/ipasudorunasgroup.db
>> handle 0
>>
>> 3a READ 1 HELD userRoot/ancestorid.db handle 0
>>
>>
>>
>>
>> pstack output
>>
>> root at ipa-master ]# pstack 4845
>> Thread 44 (Thread 0x7f681dd3a700 (LWP 4847)):
>> #0 0x00007f683cd719b3 in select () from /lib64/libc.so.6
>> #1 0x00007f683f4fc0e9 in DS_Sleep () from
>> /usr/lib64/dirsrv/libslapd.so.0
>> #2 0x00007f6832b4a907 in deadlock_threadmain () from
>> /usr/lib64/dirsrv/plugins/libback-ldbm.so
>> #3 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so
>> #4 0x00007f683d04cdc5 in start_thread () from /lib64/libpthread.so.0
>> #5 0x00007f683cd7a28d in clone () from /lib64/libc.so.6
>> Thread 43 (Thread 0x7f681d539700 (LWP 4848)):
>> #0 0x00007f683cd719b3 in select () from /lib64/libc.so.6
>> #1 0x00007f68376d8bad in __os_yield () from /lib64/libdb-5.3.so
>> <http://libdb-5.3.so>
>> #2 0x00007f68376d42b3 in __memp_sync_int () from /lib64/libdb-5.3.so
>> <http://libdb-5.3.so>
>> #3 0x00007f68376e4752 in __txn_checkpoint () from
>> /lib64/libdb-5.3.so <http://libdb-5.3.so>
>> #4 0x00007f68376e4b74 in __txn_checkpoint_pp () from
>> /lib64/libdb-5.3.so <http://libdb-5.3.so>
>> #5 0x00007f6832b4ea87 in checkpoint_threadmain () from
>> /usr/lib64/dirsrv/plugins/libback-ldbm.so
>> #6 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so
>> #7 0x00007f683d04cdc5 in start_thread () from /lib64/libpthread.so.0
>> #8 0x00007f683cd7a28d in clone () from /lib64/libc.so.6
>> Thread 42 (Thread 0x7f681cd38700 (LWP 4849)):
>> #0 0x00007f683cd719b3 in select () from /lib64/libc.so.6
>> #1 0x00007f683f4fc0e9 in DS_Sleep () from
>> /usr/lib64/dirsrv/libslapd.so.0
>> #2 0x00007f6832b4ab7f in trickle_threadmain () from
>> /usr/lib64/dirsrv/plugins/libback-ldbm.so
>> #3 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so
>> #4 0x00007f683d04cdc5 in start_thread () from /lib64/libpthread.so.0
>> #5 0x00007f683cd7a28d in clone () from /lib64/libc.so.6
>> Thread 41 (Thread 0x7f681c537700 (LWP 4850)):
>> #0 0x00007f683cd719b3 in select () from /lib64/libc.so.6
>> #1 0x00007f683f4fc0e9 in DS_Sleep () from
>> /usr/lib64/dirsrv/libslapd.so.0
>> #2 0x00007f6832b45707 in perf_threadmain () from
>> /usr/lib64/dirsrv/plugins/libback-ldbm.so
>> #3 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so
>> #4 0x00007f683d04cdc5 in start_thread () from /lib64/libpthread.so.0
>> #5 0x00007f683cd7a28d in clone () from /lib64/libc.so.6
>> Thread 40 (Thread 0x7f681bb2d700 (LWP 4851)):
>> #0 0x00007f683d0506d5 in pthread_cond_wait@@GLIBC_2.3.2 () from
>> /lib64/libpthread.so.0
>> #1 0x00007f683d6a6050 in PR_WaitCondVar () from /lib64/libnspr4.so
>> #2 0x00007f683f4eb198 in slapi_wait_condvar () from
>> /usr/lib64/dirsrv/libslapd.so.0
>> #3 0x00007f683587d62e in cos_cache_wait_on_change () from
>> /usr/lib64/dirsrv/plugins/libcos-plugin.so
>> #4 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so
>> #5 0x00007f683d04cdc5 in start_thread () from /lib64/libpthread.so.0
>> #6 0x00007f683cd7a28d in clone () from /lib64/libc.so.6
>> Thread 39 (Thread 0x7f681b32c700 (LWP 4852)):
>> #0 0x00007f683d0506d5 in pthread_cond_wait@@GLIBC_2.3.2 () from
>> /lib64/libpthread.so.0
>> #1 0x00007f683d6a6050 in PR_WaitCondVar () from /lib64/libnspr4.so
>> #2 0x00007f683f4eb198 in slapi_wait_condvar () from
>> /usr/lib64/dirsrv/libslapd.so.0
>> #3 0x00007f68311e8eed in roles_cache_wait_on_change () from
>> /usr/lib64/dirsrv/plugins/libroles-plugin.so
>> #4 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so
>> #5 0x00007f683d04cdc5 in start_thread () from /lib64/libpthread.so.0
>> #6 0x00007f683cd7a28d in clone () from /lib64/libc.so.6
>> Thread 38 (Thread 0x7f681ab2b700 (LWP 4853)):
>> #0 0x00007f683d0506d5 in pthread_cond_wait@@GLIBC_2.3.2 () from
>> /lib64/libpthread.so.0
>> #1 0x00007f683d6a6050 in PR_WaitCondVar () from /lib64/libnspr4.so
>> #2 0x00007f683f4eb198 in slapi_wait_condvar () from
>> /usr/lib64/dirsrv/libslapd.so.0
>> #3 0x00007f68311e8eed in roles_cache_wait_on_change () from
>> /usr/lib64/dirsrv/plugins/libroles-plugin.so
>> #4 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so
>> #5 0x00007f683d04cdc5 in start_thread () from /lib64/libpthread.so.0
>> #6 0x00007f683cd7a28d in clone () from /lib64/libc.so.6
>> Thread 37 (Thread 0x7f681a32a700 (LWP 4854)):
>> #0 0x00007f683d0506d5 in pthread_cond_wait@@GLIBC_2.3.2 () from
>> /lib64/libpthread.so.0
>> #1 0x00007f683d6a6050 in PR_WaitCondVar () from /lib64/libnspr4.so
>> #2 0x00007f683f4eb198 in slapi_wait_condvar () from
>> /usr/lib64/dirsrv/libslapd.so.0
>> #3 0x00007f68311e8eed in roles_cache_wait_on_change () from
>> /usr/lib64/dirsrv/plugins/libroles-plugin.so
>> #4 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so
>> #5 0x00007f683d04cdc5 in start_thread () from /lib64/libpthread.so.0
>> #6 0x00007f683cd7a28d in clone () from /lib64/libc.so.6
>> Thread 36 (Thread 0x7f6819b29700 (LWP 4855)):
>> #0 0x00007f683d050a82 in pthread_cond_timedwait@@GLIBC_2.3.2 () from
>> /lib64/libpthread.so.0
>> #1 0x00007f683d6a5b07 in pt_TimedWait () from /lib64/libnspr4.so
>> #2 0x00007f683d6a5fce in PR_WaitCondVar () from /lib64/libnspr4.so
>> #3 0x00007f683f9a4d03 in housecleaning ()
>> #4 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so
>> #5 0x00007f683d04cdc5 in start_thread () from /lib64/libpthread.so.0
>> #6 0x00007f683cd7a28d in clone () from /lib64/libc.so.6
>> Thread 35 (Thread 0x7f6819328700 (LWP 4856)):
>> #0 0x00007f683d050a82 in pthread_cond_timedwait@@GLIBC_2.3.2 () from
>> /lib64/libpthread.so.0
>> #1 0x00007f683d6a5b07 in pt_TimedWait () from /lib64/libnspr4.so
>> #2 0x00007f683d6a5fce in PR_WaitCondVar () from /lib64/libnspr4.so
>> #3 0x00007f683f4911e8 in eq_loop () from /usr/lib64/dirsrv/libslapd.so.0
>> #4 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so
>> #5 0x00007f683d04cdc5 in start_thread () from /lib64/libpthread.so.0
>> #6 0x00007f683cd7a28d in clone () from /lib64/libc.so.6
>> Thread 34 (Thread 0x7f680bfff700 (LWP 4859)):
>> #0 0x00007f683d0506d5 in pthread_cond_wait@@GLIBC_2.3.2 () from
>> /lib64/libpthread.so.0
>> #1 0x00007f683d6a6050 in PR_WaitCondVar () from /lib64/libnspr4.so
>> #2 0x00007f683f999fee in connection_wait_for_new_work ()
>> #3 0x00007f683f99b719 in connection_threadmain ()
>> #4 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so
>> #5 0x00007f683d04cdc5 in start_thread () from /lib64/libpthread.so.0
>> #6 0x00007f683cd7a28d in clone () from /lib64/libc.so.6
>> Thread 33 (Thread 0x7f680b7fe700 (LWP 4860)):
>> #0 0x00007f683d0506d5 in pthread_cond_wait@@GLIBC_2.3.2 () from
>> /lib64/libpthread.so.0
>> #1 0x00007f683d6a6050 in PR_WaitCondVar () from /lib64/libnspr4.so
>> #2 0x00007f683f999fee in connection_wait_for_new_work ()
>> #3 0x00007f683f99b719 in connection_threadmain ()
>> #4 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so
>> #5 0x00007f683d04cdc5 in start_thread () from /lib64/libpthread.so.0
>> #6 0x00007f683cd7a28d in clone () from /lib64/libc.so.6
>> Thread 32 (Thread 0x7f680affd700 (LWP 4861)):
>> #0 0x00007f683d0506d5 in pthread_cond_wait@@GLIBC_2.3.2 () from
>> /lib64/libpthread.so.0
>> #1 0x00007f68375ac2f3 in __db_hybrid_mutex_suspend () from
>> /lib64/libdb-5.3.so <http://libdb-5.3.so>
>> #2 0x00007f68375ab640 in __db_tas_mutex_lock () from
>> /lib64/libdb-5.3.so <http://libdb-5.3.so>
>> #3 0x00007f6837655d3a in __lock_get_internal () from
>> /lib64/libdb-5.3.so <http://libdb-5.3.so>
>> #4 0x00007f6837656820 in __lock_get () from /lib64/libdb-5.3.so
>> <http://libdb-5.3.so>
>> #5 0x00007f6837682142 in __db_lget () from /lib64/libdb-5.3.so
>> <http://libdb-5.3.so>
>> #6 0x00007f68375c9605 in __bam_search () from /lib64/libdb-5.3.so
>> <http://libdb-5.3.so>
>> #7 0x00007f68375b4256 in __bamc_search () from /lib64/libdb-5.3.so
>> <http://libdb-5.3.so>
>> #8 0x00007f68375b5d0f in __bamc_get () from /lib64/libdb-5.3.so
>> <http://libdb-5.3.so>
>> #9 0x00007f683766eca6 in __dbc_iget () from /lib64/libdb-5.3.so
>> <http://libdb-5.3.so>
>> #10 0x00007f683767db02 in __dbc_get_pp () from /lib64/libdb-5.3.so
>> <http://libdb-5.3.so>
>> #11 0x00007f6832b59680 in idl_new_fetch () from
>> /usr/lib64/dirsrv/plugins/libback-ldbm.so
>> #12 0x00007f6832b67b26 in index_read_ext_allids () from
>> /usr/lib64/dirsrv/plugins/libback-ldbm.so
>> #13 0x00007f6832b52284 in keys2idl () from
>> /usr/lib64/dirsrv/plugins/libback-ldbm.so
>> #14 0x00007f6832b529e3 in ava_candidates.isra.0 () from
>> /usr/lib64/dirsrv/plugins/libback-ldbm.so
>> #15 0x00007f6832b52fd2 in filter_candidates_ext () from
>> /usr/lib64/dirsrv/plugins/libback-ldbm.so
>> #16 0x00007f6832b54096 in list_candidates () from
>> /usr/lib64/dirsrv/plugins/libback-ldbm.so
>> #17 0x00007f6832b52f40 in filter_candidates_ext () from
>> /usr/lib64/dirsrv/plugins/libback-ldbm.so
>> #18 0x00007f6832b54096 in list_candidates () from
>> /usr/lib64/dirsrv/plugins/libback-ldbm.so
>> #19 0x00007f6832b52f40 in filter_candidates_ext () from
>> /usr/lib64/dirsrv/plugins/libback-ldbm.so
>> #20 0x00007f6832b54096 in list_candidates () from
>> /usr/lib64/dirsrv/plugins/libback-ldbm.so
>> #21 0x00007f6832b52f40 in filter_candidates_ext () from
>> /usr/lib64/dirsrv/plugins/libback-ldbm.so
>> #22 0x00007f6832b8e80c in subtree_candidates () from
>> /usr/lib64/dirsrv/plugins/libback-ldbm.so
>> #23 0x00007f6832b8fe7f in ldbm_back_search () from
>> /usr/lib64/dirsrv/plugins/libback-ldbm.so
>> #24 0x00007f683f4bd1d6 in op_shared_search () from
>> /usr/lib64/dirsrv/libslapd.so.0
>> #25 0x00007f683f9ae33e in do_search ()
>> #26 0x00007f683f99c3c3 in connection_threadmain ()
>> #27 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so
>> #28 0x00007f683d04cdc5 in start_thread () from /lib64/libpthread.so.0
>> #29 0x00007f683cd7a28d in clone () from /lib64/libc.so.6
>> Thread 31 (Thread 0x7f680a7fc700 (LWP 4862)):
>> #0 0x00007f683d0506d5 in pthread_cond_wait@@GLIBC_2.3.2 () from
>> /lib64/libpthread.so.0
>> #1 0x00007f683d6a62c3 in PR_EnterMonitor () from /lib64/libnspr4.so
>> #2 0x00007f6832b4d3e6 in dblayer_txn_begin () from
>> /usr/lib64/dirsrv/plugins/libback-ldbm.so
>> #3 0x00007f6832b89007 in ldbm_back_modify () from
>> /usr/lib64/dirsrv/plugins/libback-ldbm.so
>> #4 0x00007f683f4b6541 in op_shared_modify () from
>> /usr/lib64/dirsrv/libslapd.so.0
>> #5 0x00007f683f4b7004 in modify_internal_pb () from
>> /usr/lib64/dirsrv/libslapd.so.0
>> #6 0x00007f683483bbbb in ipalockout_postop () from
>> /usr/lib64/dirsrv/plugins/libipa_lockout.so
>> #7 0x00007f683f4c8848 in plugin_call_func () from
>> /usr/lib64/dirsrv/libslapd.so.0
>> #8 0x00007f683f4c8b13 in plugin_call_plugins () from
>> /usr/lib64/dirsrv/libslapd.so.0
>> #9 0x00007f683f9954d8 in do_bind ()
>> #10 0x00007f683f99c3fd in connection_threadmain ()
>> #11 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so
>> #12 0x00007f683d04cdc5 in start_thread () from /lib64/libpthread.so.0
>> #13 0x00007f683cd7a28d in clone () from /lib64/libc.so.6
>> Thread 30 (Thread 0x7f6809ffb700 (LWP 4863)):
>> #0 0x00007f683d0506d5 in pthread_cond_wait@@GLIBC_2.3.2 () from
>> /lib64/libpthread.so.0
>> #1 0x00007f68375ac2f3 in __db_hybrid_mutex_suspend () from
>> /lib64/libdb-5.3.so <http://libdb-5.3.so>
>> #2 0x00007f68375ab640 in __db_tas_mutex_lock () from
>> /lib64/libdb-5.3.so <http://libdb-5.3.so>
>> #3 0x00007f6837655d3a in __lock_get_internal () from
>> /lib64/libdb-5.3.so <http://libdb-5.3.so>
>> #4 0x00007f6837656820 in __lock_get () from /lib64/libdb-5.3.so
>> <http://libdb-5.3.so>
>> #5 0x00007f6837682142 in __db_lget () from /lib64/libdb-5.3.so
>> <http://libdb-5.3.so>
>> #6 0x00007f68375c9605 in __bam_search () from /lib64/libdb-5.3.so
>> <http://libdb-5.3.so>
>> #7 0x00007f68375b4256 in __bamc_search () from /lib64/libdb-5.3.so
>> <http://libdb-5.3.so>
>> #8 0x00007f68375b5d0f in __bamc_get () from /lib64/libdb-5.3.so
>> <http://libdb-5.3.so>
>> #9 0x00007f683766eca6 in __dbc_iget () from /lib64/libdb-5.3.so
>> <http://libdb-5.3.so>
>> #10 0x00007f683767db02 in __dbc_get_pp () from /lib64/libdb-5.3.so
>> <http://libdb-5.3.so>
>> #11 0x00007f6832b59680 in idl_new_fetch () from
>> /usr/lib64/dirsrv/plugins/libback-ldbm.so
>> #12 0x00007f6832b67b26 in index_read_ext_allids () from
>> /usr/lib64/dirsrv/plugins/libback-ldbm.so
>> #13 0x00007f6832b52284 in keys2idl () from
>> /usr/lib64/dirsrv/plugins/libback-ldbm.so
>> #14 0x00007f6832b529e3 in ava_candidates.isra.0 () from
>> /usr/lib64/dirsrv/plugins/libback-ldbm.so
>> #15 0x00007f6832b52fd2 in filter_candidates_ext () from
>> /usr/lib64/dirsrv/plugins/libback-ldbm.so
>> #16 0x00007f6832b54096 in list_candidates () from
>> /usr/lib64/dirsrv/plugins/libback-ldbm.so
>> #17 0x00007f6832b52f40 in filter_candidates_ext () from
>> /usr/lib64/dirsrv/plugins/libback-ldbm.so
>> #18 0x00007f6832b8e80c in subtree_candidates () from
>> /usr/lib64/dirsrv/plugins/libback-ldbm.so
>> #19 0x00007f6832b8fe7f in ldbm_back_search () from
>> /usr/lib64/dirsrv/plugins/libback-ldbm.so
>> #20 0x00007f683f4bd1d6 in op_shared_search () from
>> /usr/lib64/dirsrv/libslapd.so.0
>> #21 0x00007f683f4cd94e in search_internal_callback_pb () from
>> /usr/lib64/dirsrv/libslapd.so.0
>> #22 0x00007f683f4cdbe8 in search_internal_pb () from
>> /usr/lib64/dirsrv/libslapd.so.0
>> #23 0x00007f683f9ac49b in ids_sasl_canon_user ()
>> #24 0x00007f683d46c93b in _sasl_canon_user () from /lib64/libsasl2.so.3
>> #25 0x00007f683d46cc4c in _sasl_canon_user_lookup () from
>> /lib64/libsasl2.so.3
>> #26 0x00007f6837374fb9 in gssapi_server_mech_step () from
>> /usr/lib64/sasl2/libgssapiv2.so
>> #27 0x00007f683d478b9b in sasl_server_step () from /lib64/libsasl2.so.3
>> #28 0x00007f683f9ad83b in ids_sasl_check_bind ()
>> #29 0x00007f683f99561a in do_bind ()
>> #30 0x00007f683f99c3fd in connection_threadmain ()
>> #31 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so
>> #32 0x00007f683d04cdc5 in start_thread () from /lib64/libpthread.so.0
>> #33 0x00007f683cd7a28d in clone () from /lib64/libc.so.6
>> Thread 29 (Thread 0x7f68097fa700 (LWP 4864)):
>> #0 0x00007f683d0506d5 in pthread_cond_wait@@GLIBC_2.3.2 () from
>> /lib64/libpthread.so.0
>> #1 0x00007f683d6a6050 in PR_WaitCondVar () from /lib64/libnspr4.so
>> #2 0x00007f683f999fee in connection_wait_for_new_work ()
>> #3 0x00007f683f99b719 in connection_threadmain ()
>> #4 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so
>> #5 0x00007f683d04cdc5 in start_thread () from /lib64/libpthread.so.0
>> #6 0x00007f683cd7a28d in clone () from /lib64/libc.so.6
>> Thread 28 (Thread 0x7f6808ff9700 (LWP 4865)):
>> #0 0x00007f683d0506d5 in pthread_cond_wait@@GLIBC_2.3.2 () from
>> /lib64/libpthread.so.0
>> #1 0x00007f683d6a6050 in PR_WaitCondVar () from /lib64/libnspr4.so
>> #2 0x00007f683f999fee in connection_wait_for_new_work ()
>> #3 0x00007f683f99b719 in connection_threadmain ()
>> #4 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so
>> #5 0x00007f683d04cdc5 in start_thread () from /lib64/libpthread.so.0
>> #6 0x00007f683cd7a28d in clone () from /lib64/libc.so.6
>> Thread 27 (Thread 0x7f6803fff700 (LWP 4866)):
>> #0 0x00007f683d0506d5 in pthread_cond_wait@@GLIBC_2.3.2 () from
>> /lib64/libpthread.so.0
>> #1 0x00007f683d6a6050 in PR_WaitCondVar () from /lib64/libnspr4.so
>> #2 0x00007f683f999fee in connection_wait_for_new_work ()
>> #3 0x00007f683f99b719 in connection_threadmain ()
>> #4 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so
>> #5 0x00007f683d04cdc5 in start_thread () from /lib64/libpthread.so.0
>> #6 0x00007f683cd7a28d in clone () from /lib64/libc.so.6
>> Thread 26 (Thread 0x7f68037fe700 (LWP 4867)):
>> #0 0x00007f683d0506d5 in pthread_cond_wait@@GLIBC_2.3.2 () from
>> /lib64/libpthread.so.0
>> #1 0x00007f683d6a6050 in PR_WaitCondVar () from /lib64/libnspr4.so
>> #2 0x00007f683f999fee in connection_wait_for_new_work ()
>> #3 0x00007f683f99b719 in connection_threadmain ()
>> #4 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so
>> #5 0x00007f683d04cdc5 in start_thread () from /lib64/libpthread.so.0
>> #6 0x00007f683cd7a28d in clone () from /lib64/libc.so.6
>> Thread 25 (Thread 0x7f6802ffd700 (LWP 4868)):
>> #0 0x00007f683d0506d5 in pthread_cond_wait@@GLIBC_2.3.2 () from
>> /lib64/libpthread.so.0
>> #1 0x00007f683d6a6050 in PR_WaitCondVar () from /lib64/libnspr4.so
>> #2 0x00007f683f999fee in connection_wait_for_new_work ()
>> #3 0x00007f683f99b719 in connection_threadmain ()
>> #4 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so
>> #5 0x00007f683d04cdc5 in start_thread () from /lib64/libpthread.so.0
>> #6 0x00007f683cd7a28d in clone () from /lib64/libc.so.6
>> Thread 24 (Thread 0x7f68027fc700 (LWP 4869)):
>> #0 0x00007f683d0506d5 in pthread_cond_wait@@GLIBC_2.3.2 () from
>> /lib64/libpthread.so.0
>> #1 0x00007f68375ac2f3 in __db_hybrid_mutex_suspend () from
>> /lib64/libdb-5.3.so <http://libdb-5.3.so>
>> #2 0x00007f68375abb37 in __db_tas_mutex_readlock () from
>> /lib64/libdb-5.3.so <http://libdb-5.3.so>
>> #3 0x00007f68376c5d35 in __memp_fget () from /lib64/libdb-5.3.so
>> <http://libdb-5.3.so>
>> #4 0x00007f683766f333 in __dbc_iget () from /lib64/libdb-5.3.so
>> <http://libdb-5.3.so>
>> #5 0x00007f683767db02 in __dbc_get_pp () from /lib64/libdb-5.3.so
>> <http://libdb-5.3.so>
>> #6 0x00007f6832b59822 in idl_new_fetch () from
>> /usr/lib64/dirsrv/plugins/libback-ldbm.so
>> #7 0x00007f6832b67b26 in index_read_ext_allids () from
>> /usr/lib64/dirsrv/plugins/libback-ldbm.so
>> #8 0x00007f6832b52284 in keys2idl () from
>> /usr/lib64/dirsrv/plugins/libback-ldbm.so
>> #9 0x00007f6832b529e3 in ava_candidates.isra.0 () from
>> /usr/lib64/dirsrv/plugins/libback-ldbm.so
>> #10 0x00007f6832b52fd2 in filter_candidates_ext () from
>> /usr/lib64/dirsrv/plugins/libback-ldbm.so
>> #11 0x00007f6832b54096 in list_candidates () from
>> /usr/lib64/dirsrv/plugins/libback-ldbm.so
>> #12 0x00007f6832b52f40 in filter_candidates_ext () from
>> /usr/lib64/dirsrv/plugins/libback-ldbm.so
>> #13 0x00007f6832b54096 in list_candidates () from
>> /usr/lib64/dirsrv/plugins/libback-ldbm.so
>> #14 0x00007f6832b52f40 in filter_candidates_ext () from
>> /usr/lib64/dirsrv/plugins/libback-ldbm.so
>> #15 0x00007f6832b54096 in list_candidates () from
>> /usr/lib64/dirsrv/plugins/libback-ldbm.so
>> #16 0x00007f6832b52f40 in filter_candidates_ext () from
>> /usr/lib64/dirsrv/plugins/libback-ldbm.so
>> #17 0x00007f6832b8e80c in subtree_candidates () from
>> /usr/lib64/dirsrv/plugins/libback-ldbm.so
>> #18 0x00007f6832b8fe7f in ldbm_back_search () from
>> /usr/lib64/dirsrv/plugins/libback-ldbm.so
>> #19 0x00007f683f4bd1d6 in op_shared_search () from
>> /usr/lib64/dirsrv/libslapd.so.0
>> #20 0x00007f683f9ae33e in do_search ()
>> #21 0x00007f683f99c3c3 in connection_threadmain ()
>> #22 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so
>> #23 0x00007f683d04cdc5 in start_thread () from /lib64/libpthread.so.0
>> #24 0x00007f683cd7a28d in clone () from /lib64/libc.so.6
>> Thread 23 (Thread 0x7f6801ffb700 (LWP 4870)):
>> #0 0x00007f683d0506d5 in pthread_cond_wait@@GLIBC_2.3.2 () from
>> /lib64/libpthread.so.0
>> #1 0x00007f683d6a6050 in PR_WaitCondVar () from /lib64/libnspr4.so
>> #2 0x00007f683f999fee in connection_wait_for_new_work ()
>> #3 0x00007f683f99b719 in connection_threadmain ()
>> #4 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so
>> #5 0x00007f683d04cdc5 in start_thread () from /lib64/libpthread.so.0
>> #6 0x00007f683cd7a28d in clone () from /lib64/libc.so.6
>> Thread 22 (Thread 0x7f68017fa700 (LWP 4871)):
>> #0 0x00007f683d0506d5 in pthread_cond_wait@@GLIBC_2.3.2 () from
>> /lib64/libpthread.so.0
>> #1 0x00007f683d6a6050 in PR_WaitCondVar () from /lib64/libnspr4.so
>> #2 0x00007f683f999fee in connection_wait_for_new_work ()
>> #3 0x00007f683f99b719 in connection_threadmain ()
>> #4 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so
>> #5 0x00007f683d04cdc5 in start_thread () from /lib64/libpthread.so.0
>> #6 0x00007f683cd7a28d in clone () from /lib64/libc.so.6
>> Thread 21 (Thread 0x7f6800ff9700 (LWP 4872)):
>> #0 0x00007f683d0506d5 in pthread_cond_wait@@GLIBC_2.3.2 () from
>> /lib64/libpthread.so.0
>> #1 0x00007f683d6a6050 in PR_WaitCondVar () from /lib64/libnspr4.so
>> #2 0x00007f683f999fee in connection_wait_for_new_work ()
>> #3 0x00007f683f99b719 in connection_threadmain ()
>> #4 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so
>> #5 0x00007f683d04cdc5 in start_thread () from /lib64/libpthread.so.0
>> #6 0x00007f683cd7a28d in clone () from /lib64/libc.so.6
>> Thread 20 (Thread 0x7f68007f8700 (LWP 4873)):
>> #0 0x00007f683d0506d5 in pthread_cond_wait@@GLIBC_2.3.2 () from
>> /lib64/libpthread.so.0
>> #1 0x00007f68375ac2f3 in __db_hybrid_mutex_suspend () from
>> /lib64/libdb-5.3.so <http://libdb-5.3.so>
>> #2 0x00007f68375ab640 in __db_tas_mutex_lock () from
>> /lib64/libdb-5.3.so <http://libdb-5.3.so>
>> #3 0x00007f68376c5e1b in __memp_fget () from /lib64/libdb-5.3.so
>> <http://libdb-5.3.so>
>> #4 0x00007f68375c9211 in __bam_search () from /lib64/libdb-5.3.so
>> <http://libdb-5.3.so>
>> #5 0x00007f68375b4256 in __bamc_search () from /lib64/libdb-5.3.so
>> <http://libdb-5.3.so>
>> #6 0x00007f68375b86c4 in __bamc_put () from /lib64/libdb-5.3.so
>> <http://libdb-5.3.so>
>> #7 0x00007f683766f6d5 in __dbc_iput () from /lib64/libdb-5.3.so
>> <http://libdb-5.3.so>
>> #8 0x00007f683766aa0e in __db_put () from /lib64/libdb-5.3.so
>> <http://libdb-5.3.so>
>> #9 0x00007f683767ffa4 in __db_put_pp () from /lib64/libdb-5.3.so
>> <http://libdb-5.3.so>
>> #10 0x00007f6832b5a4cf in idl_new_insert_key () from
>> /usr/lib64/dirsrv/plugins/libback-ldbm.so
>> #11 0x00007f6832b695b5 in addordel_values_sv.isra.3 () from
>> /usr/lib64/dirsrv/plugins/libback-ldbm.so
>> #12 0x00007f6832b6a25a in index_addordel_values_ext_sv () from
>> /usr/lib64/dirsrv/plugins/libback-ldbm.so
>> #13 0x00007f6832b6a694 in index_addordel_values_sv () from
>> /usr/lib64/dirsrv/plugins/libback-ldbm.so
>> #14 0x00007f6832b6a75a in index_addordel_entry () from
>> /usr/lib64/dirsrv/plugins/libback-ldbm.so
>> #15 0x00007f6832b6d5fa in ldbm_back_add () from
>> /usr/lib64/dirsrv/plugins/libback-ldbm.so
>> #16 0x00007f683f46fcca in op_shared_add () from
>> /usr/lib64/dirsrv/libslapd.so.0
>> #17 0x00007f683f471020 in do_add () from /usr/lib64/dirsrv/libslapd.so.0
>> #18 0x00007f683f99c303 in connection_threadmain ()
>> #19 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so
>> #20 0x00007f683d04cdc5 in start_thread () from /lib64/libpthread.so.0
>> #21 0x00007f683cd7a28d in clone () from /lib64/libc.so.6
>> Thread 19 (Thread 0x7f67ffff7700 (LWP 4874)):
>> #0 0x00007f683d0506d5 in pthread_cond_wait@@GLIBC_2.3.2 () from
>> /lib64/libpthread.so.0
>> #1 0x00007f683d6a62c3 in PR_EnterMonitor () from /lib64/libnspr4.so
>> #2 0x00007f6832b4d3e6 in dblayer_txn_begin () from
>> /usr/lib64/dirsrv/plugins/libback-ldbm.so
>> #3 0x00007f6832b89007 in ldbm_back_modify () from
>> /usr/lib64/dirsrv/plugins/libback-ldbm.so
>> #4 0x00007f683f4b6541 in op_shared_modify () from
>> /usr/lib64/dirsrv/libslapd.so.0
>> #5 0x00007f683f4b7004 in modify_internal_pb () from
>> /usr/lib64/dirsrv/libslapd.so.0
>> #6 0x00007f6832dd3b60 in ipapwd_apply_mods () from
>> /usr/lib64/dirsrv/plugins/libipa_pwd_extop.so
>> #7 0x00007f6832dd78e0 in store_new_keys () from
>> /usr/lib64/dirsrv/plugins/libipa_pwd_extop.so
>> #8 0x00007f6832dd9ad0 in ipapwd_extop () from
>> /usr/lib64/dirsrv/plugins/libipa_pwd_extop.so
>> #9 0x00007f683f4c6ba2 in plugin_call_exop_plugins () from
>> /usr/lib64/dirsrv/libslapd.so.0
>> #10 0x00007f683f9a2efe in do_extended ()
>> #11 0x00007f683f99c41a in connection_threadmain ()
>> #12 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so
>> #13 0x00007f683d04cdc5 in start_thread () from /lib64/libpthread.so.0
>> #14 0x00007f683cd7a28d in clone () from /lib64/libc.so.6
>> Thread 18 (Thread 0x7f67ff7f6700 (LWP 4875)):
>> #0 0x00007f683d0506d5 in pthread_cond_wait@@GLIBC_2.3.2 () from
>> /lib64/libpthread.so.0
>> #1 0x00007f683d6a6050 in PR_WaitCondVar () from /lib64/libnspr4.so
>> #2 0x00007f683f999fee in connection_wait_for_new_work ()
>> #3 0x00007f683f99b719 in connection_threadmain ()
>> #4 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so
>> #5 0x00007f683d04cdc5 in start_thread () from /lib64/libpthread.so.0
>> #6 0x00007f683cd7a28d in clone () from /lib64/libc.so.6
>> Thread 17 (Thread 0x7f67feff5700 (LWP 4876)):
>> #0 0x00007f683d0506d5 in pthread_cond_wait@@GLIBC_2.3.2 () from
>> /lib64/libpthread.so.0
>> #1 0x00007f68375ac2f3 in __db_hybrid_mutex_suspend () from
>> /lib64/libdb-5.3.so <http://libdb-5.3.so>
>> #2 0x00007f68375ab640 in __db_tas_mutex_lock () from
>> /lib64/libdb-5.3.so <http://libdb-5.3.so>
>> #3 0x00007f6837655d3a in __lock_get_internal () from
>> /lib64/libdb-5.3.so <http://libdb-5.3.so>
>> #4 0x00007f6837656820 in __lock_get () from /lib64/libdb-5.3.so
>> <http://libdb-5.3.so>
>> #5 0x00007f6837682142 in __db_lget () from /lib64/libdb-5.3.so
>> <http://libdb-5.3.so>
>> #6 0x00007f68375c9605 in __bam_search () from /lib64/libdb-5.3.so
>> <http://libdb-5.3.so>
>> #7 0x00007f68375b4256 in __bamc_search () from /lib64/libdb-5.3.so
>> <http://libdb-5.3.so>
>> #8 0x00007f68375b5d0f in __bamc_get () from /lib64/libdb-5.3.so
>> <http://libdb-5.3.so>
>> #9 0x00007f683766eca6 in __dbc_iget () from /lib64/libdb-5.3.so
>> <http://libdb-5.3.so>
>> #10 0x00007f683767db02 in __dbc_get_pp () from /lib64/libdb-5.3.so
>> <http://libdb-5.3.so>
>> #11 0x00007f6832b59680 in idl_new_fetch () from
>> /usr/lib64/dirsrv/plugins/libback-ldbm.so
>> #12 0x00007f6832b67b26 in index_read_ext_allids () from
>> /usr/lib64/dirsrv/plugins/libback-ldbm.so
>> #13 0x00007f6832b52284 in keys2idl () from
>> /usr/lib64/dirsrv/plugins/libback-ldbm.so
>> #14 0x00007f6832b529e3 in ava_candidates.isra.0 () from
>> /usr/lib64/dirsrv/plugins/libback-ldbm.so
>> #15 0x00007f6832b52fd2 in filter_candidates_ext () from
>> /usr/lib64/dirsrv/plugins/libback-ldbm.so
>> #16 0x00007f6832b54096 in list_candidates () from
>> /usr/lib64/dirsrv/plugins/libback-ldbm.so
>> #17 0x00007f6832b52f40 in filter_candidates_ext () from
>> /usr/lib64/dirsrv/plugins/libback-ldbm.so
>> #18 0x00007f6832b8e80c in subtree_candidates () from
>> /usr/lib64/dirsrv/plugins/libback-ldbm.so
>> #19 0x00007f6832b8fe7f in ldbm_back_search () from
>> /usr/lib64/dirsrv/plugins/libback-ldbm.so
>> #20 0x00007f683f4bd1d6 in op_shared_search () from
>> /usr/lib64/dirsrv/libslapd.so.0
>> #21 0x00007f683f4cd94e in search_internal_callback_pb () from
>> /usr/lib64/dirsrv/libslapd.so.0
>> #22 0x00007f683f4cdbe8 in search_internal_pb () from
>> /usr/lib64/dirsrv/libslapd.so.0
>> #23 0x00007f6832dd75dc in get_entry_by_principal () from
>> /usr/lib64/dirsrv/plugins/libipa_pwd_extop.so
>> #24 0x00007f6832dd8eb1 in ipapwd_extop () from
>> /usr/lib64/dirsrv/plugins/libipa_pwd_extop.so
>> #25 0x00007f683f4c6ba2 in plugin_call_exop_plugins () from
>> /usr/lib64/dirsrv/libslapd.so.0
>> #26 0x00007f683f9a2efe in do_extended ()
>> #27 0x00007f683f99c41a in connection_threadmain ()
>> #28 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so
>> #29 0x00007f683d04cdc5 in start_thread () from /lib64/libpthread.so.0
>> #30 0x00007f683cd7a28d in clone () from /lib64/libc.so.6
>> Thread 16 (Thread 0x7f67fe7f4700 (LWP 4877)):
>> #0 0x00007f683d0506d5 in pthread_cond_wait@@GLIBC_2.3.2 () from
>> /lib64/libpthread.so.0
>> #1 0x00007f683d6a6050 in PR_WaitCondVar () from /lib64/libnspr4.so
>> #2 0x00007f683f999fee in connection_wait_for_new_work ()
>> #3 0x00007f683f99b719 in connection_threadmain ()
>> #4 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so
>> #5 0x00007f683d04cdc5 in start_thread () from /lib64/libpthread.so.0
>> #6 0x00007f683cd7a28d in clone () from /lib64/libc.so.6
>> Thread 15 (Thread 0x7f67fdff3700 (LWP 4878)):
>> #0 0x00007f683d0506d5 in pthread_cond_wait@@GLIBC_2.3.2 () from
>> /lib64/libpthread.so.0
>> #1 0x00007f683d6a6050 in PR_WaitCondVar () from /lib64/libnspr4.so
>> #2 0x00007f683f999fee in connection_wait_for_new_work ()
>> #3 0x00007f683f99b719 in connection_threadmain ()
>> #4 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so
>> #5 0x00007f683d04cdc5 in start_thread () from /lib64/libpthread.so.0
>> #6 0x00007f683cd7a28d in clone () from /lib64/libc.so.6
>> Thread 14 (Thread 0x7f67fd7f2700 (LWP 4879)):
>> #0 0x00007f683d0506d5 in pthread_cond_wait@@GLIBC_2.3.2 () from
>> /lib64/libpthread.so.0
>> #1 0x00007f683d6a6050 in PR_WaitCondVar () from /lib64/libnspr4.so
>> #2 0x00007f683f999fee in connection_wait_for_new_work ()
>> #3 0x00007f683f99b719 in connection_threadmain ()
>> #4 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so
>> #5 0x00007f683d04cdc5 in start_thread () from /lib64/libpthread.so.0
>> #6 0x00007f683cd7a28d in clone () from /lib64/libc.so.6
>> Thread 13 (Thread 0x7f67fcff1700 (LWP 4880)):
>> #0 0x00007f683d0506d5 in pthread_cond_wait@@GLIBC_2.3.2 () from
>> /lib64/libpthread.so.0
>> #1 0x00007f683d6a6050 in PR_WaitCondVar () from /lib64/libnspr4.so
>> #2 0x00007f683f999fee in connection_wait_for_new_work ()
>> #3 0x00007f683f99b719 in connection_threadmain ()
>> #4 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so
>> #5 0x00007f683d04cdc5 in start_thread () from /lib64/libpthread.so.0
>> #6 0x00007f683cd7a28d in clone () from /lib64/libc.so.6
>> Thread 12 (Thread 0x7f67fc7f0700 (LWP 4881)):
>> #0 0x00007f683d0506d5 in pthread_cond_wait@@GLIBC_2.3.2 () from
>> /lib64/libpthread.so.0
>> #1 0x00007f683d6a6050 in PR_WaitCondVar () from /lib64/libnspr4.so
>> #2 0x00007f683f999fee in connection_wait_for_new_work ()
>> #3 0x00007f683f99b719 in connection_threadmain ()
>> #4 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so
>> #5 0x00007f683d04cdc5 in start_thread () from /lib64/libpthread.so.0
>> #6 0x00007f683cd7a28d in clone () from /lib64/libc.so.6
>> Thread 11 (Thread 0x7f67fbfef700 (LWP 4882)):
>> #0 0x00007f683d0506d5 in pthread_cond_wait@@GLIBC_2.3.2 () from
>> /lib64/libpthread.so.0
>> #1 0x00007f683d6a6050 in PR_WaitCondVar () from /lib64/libnspr4.so
>> #2 0x00007f683f999fee in connection_wait_for_new_work ()
>> #3 0x00007f683f99b719 in connection_threadmain ()
>> #4 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so
>> #5 0x00007f683d04cdc5 in start_thread () from /lib64/libpthread.so.0
>> #6 0x00007f683cd7a28d in clone () from /lib64/libc.so.6
>> Thread 10 (Thread 0x7f67fb7ee700 (LWP 4883)):
>> #0 0x00007f683d0506d5 in pthread_cond_wait@@GLIBC_2.3.2 () from
>> /lib64/libpthread.so.0
>> #1 0x00007f68375ac2f3 in __db_hybrid_mutex_suspend () from
>> /lib64/libdb-5.3.so <http://libdb-5.3.so>
>> #2 0x00007f68375ab640 in __db_tas_mutex_lock () from
>> /lib64/libdb-5.3.so <http://libdb-5.3.so>
>> #3 0x00007f6837655d3a in __lock_get_internal () from
>> /lib64/libdb-5.3.so <http://libdb-5.3.so>
>> #4 0x00007f6837656820 in __lock_get () from /lib64/libdb-5.3.so
>> <http://libdb-5.3.so>
>> #5 0x00007f6837682142 in __db_lget () from /lib64/libdb-5.3.so
>> <http://libdb-5.3.so>
>> #6 0x00007f68375c9605 in __bam_search () from /lib64/libdb-5.3.so
>> <http://libdb-5.3.so>
>> #7 0x00007f68375b4256 in __bamc_search () from /lib64/libdb-5.3.so
>> <http://libdb-5.3.so>
>> #8 0x00007f68375b5d0f in __bamc_get () from /lib64/libdb-5.3.so
>> <http://libdb-5.3.so>
>> #9 0x00007f683766eca6 in __dbc_iget () from /lib64/libdb-5.3.so
>> <http://libdb-5.3.so>
>> #10 0x00007f683767db02 in __dbc_get_pp () from /lib64/libdb-5.3.so
>> <http://libdb-5.3.so>
>> #11 0x00007f6832b59680 in idl_new_fetch () from
>> /usr/lib64/dirsrv/plugins/libback-ldbm.so
>> #12 0x00007f6832b67b26 in index_read_ext_allids () from
>> /usr/lib64/dirsrv/plugins/libback-ldbm.so
>> #13 0x00007f6832b52284 in keys2idl () from
>> /usr/lib64/dirsrv/plugins/libback-ldbm.so
>> #14 0x00007f6832b529e3 in ava_candidates.isra.0 () from
>> /usr/lib64/dirsrv/plugins/libback-ldbm.so
>> #15 0x00007f6832b52fd2 in filter_candidates_ext () from
>> /usr/lib64/dirsrv/plugins/libback-ldbm.so
>> #16 0x00007f6832b54096 in list_candidates () from
>> /usr/lib64/dirsrv/plugins/libback-ldbm.so
>> #17 0x00007f6832b52f40 in filter_candidates_ext () from
>> /usr/lib64/dirsrv/plugins/libback-ldbm.so
>> #18 0x00007f6832b54096 in list_candidates () from
>> /usr/lib64/dirsrv/plugins/libback-ldbm.so
>> #19 0x00007f6832b52f40 in filter_candidates_ext () from
>> /usr/lib64/dirsrv/plugins/libback-ldbm.so
>> #20 0x00007f6832b54096 in list_candidates () from
>> /usr/lib64/dirsrv/plugins/libback-ldbm.so
>> #21 0x00007f6832b52f40 in filter_candidates_ext () from
>> /usr/lib64/dirsrv/plugins/libback-ldbm.so
>> #22 0x00007f6832b8e80c in subtree_candidates () from
>> /usr/lib64/dirsrv/plugins/libback-ldbm.so
>> #23 0x00007f6832b8fe7f in ldbm_back_search () from
>> /usr/lib64/dirsrv/plugins/libback-ldbm.so
>> #24 0x00007f683f4bd1d6 in op_shared_search () from
>> /usr/lib64/dirsrv/libslapd.so.0
>> #25 0x00007f683f9ae33e in do_search ()
>> #26 0x00007f683f99c3c3 in connection_threadmain ()
>> #27 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so
>> #28 0x00007f683d04cdc5 in start_thread () from /lib64/libpthread.so.0
>> #29 0x00007f683cd7a28d in clone () from /lib64/libc.so.6
>> Thread 9 (Thread 0x7f67fafed700 (LWP 4884)):
>> #0 0x00007f683d0506d5 in pthread_cond_wait@@GLIBC_2.3.2 () from
>> /lib64/libpthread.so.0
>> #1 0x00007f683d6a6050 in PR_WaitCondVar () from /lib64/libnspr4.so
>> #2 0x00007f683f999fee in connection_wait_for_new_work ()
>> #3 0x00007f683f99b719 in connection_threadmain ()
>> #4 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so
>> #5 0x00007f683d04cdc5 in start_thread () from /lib64/libpthread.so.0
>> #6 0x00007f683cd7a28d in clone () from /lib64/libc.so.6
>> Thread 8 (Thread 0x7f67fa7ec700 (LWP 4885)):
>> #0 0x00007f683d0506d5 in pthread_cond_wait@@GLIBC_2.3.2 () from
>> /lib64/libpthread.so.0
>> #1 0x00007f68375ac2f3 in __db_hybrid_mutex_suspend () from
>> /lib64/libdb-5.3.so <http://libdb-5.3.so>
>> #2 0x00007f68375ab640 in __db_tas_mutex_lock () from
>> /lib64/libdb-5.3.so <http://libdb-5.3.so>
>> #3 0x00007f6837655d3a in __lock_get_internal () from
>> /lib64/libdb-5.3.so <http://libdb-5.3.so>
>> #4 0x00007f6837656820 in __lock_get () from /lib64/libdb-5.3.so
>> <http://libdb-5.3.so>
>> #5 0x00007f6837682142 in __db_lget () from /lib64/libdb-5.3.so
>> <http://libdb-5.3.so>
>> #6 0x00007f68375c9605 in __bam_search () from /lib64/libdb-5.3.so
>> <http://libdb-5.3.so>
>> #7 0x00007f68375b4256 in __bamc_search () from /lib64/libdb-5.3.so
>> <http://libdb-5.3.so>
>> #8 0x00007f68375b5d0f in __bamc_get () from /lib64/libdb-5.3.so
>> <http://libdb-5.3.so>
>> #9 0x00007f683766eca6 in __dbc_iget () from /lib64/libdb-5.3.so
>> <http://libdb-5.3.so>
>> #10 0x00007f683767db02 in __dbc_get_pp () from /lib64/libdb-5.3.so
>> <http://libdb-5.3.so>
>> #11 0x00007f6832b59680 in idl_new_fetch () from
>> /usr/lib64/dirsrv/plugins/libback-ldbm.so
>> #12 0x00007f6832b67b26 in index_read_ext_allids () from
>> /usr/lib64/dirsrv/plugins/libback-ldbm.so
>> #13 0x00007f6832b52284 in keys2idl () from
>> /usr/lib64/dirsrv/plugins/libback-ldbm.so
>> #14 0x00007f6832b529e3 in ava_candidates.isra.0 () from
>> /usr/lib64/dirsrv/plugins/libback-ldbm.so
>> #15 0x00007f6832b52fd2 in filter_candidates_ext () from
>> /usr/lib64/dirsrv/plugins/libback-ldbm.so
>> #16 0x00007f6832b54096 in list_candidates () from
>> /usr/lib64/dirsrv/plugins/libback-ldbm.so
>> #17 0x00007f6832b52f40 in filter_candidates_ext () from
>> /usr/lib64/dirsrv/plugins/libback-ldbm.so
>> #18 0x00007f6832b54096 in list_candidates () from
>> /usr/lib64/dirsrv/plugins/libback-ldbm.so
>> #19 0x00007f6832b52f40 in filter_candidates_ext () from
>> /usr/lib64/dirsrv/plugins/libback-ldbm.so
>> #20 0x00007f6832b54096 in list_candidates () from
>> /usr/lib64/dirsrv/plugins/libback-ldbm.so
>> #21 0x00007f6832b52f40 in filter_candidates_ext () from
>> /usr/lib64/dirsrv/plugins/libback-ldbm.so
>> #22 0x00007f6832b8e80c in subtree_candidates () from
>> /usr/lib64/dirsrv/plugins/libback-ldbm.so
>> #23 0x00007f6832b8fe7f in ldbm_back_search () from
>> /usr/lib64/dirsrv/plugins/libback-ldbm.so
>> #24 0x00007f683f4bd1d6 in op_shared_search () from
>> /usr/lib64/dirsrv/libslapd.so.0
>> #25 0x00007f683f9ae33e in do_search ()
>> #26 0x00007f683f99c3c3 in connection_threadmain ()
>> #27 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so
>> #28 0x00007f683d04cdc5 in start_thread () from /lib64/libpthread.so.0
>> #29 0x00007f683cd7a28d in clone () from /lib64/libc.so.6
>> Thread 7 (Thread 0x7f67f9feb700 (LWP 4886)):
>> #0 0x00007f683d0506d5 in pthread_cond_wait@@GLIBC_2.3.2 () from
>> /lib64/libpthread.so.0
>> #1 0x00007f683d6a6050 in PR_WaitCondVar () from /lib64/libnspr4.so
>> #2 0x00007f683f999fee in connection_wait_for_new_work ()
>> #3 0x00007f683f99b719 in connection_threadmain ()
>> #4 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so
>> #5 0x00007f683d04cdc5 in start_thread () from /lib64/libpthread.so.0
>> #6 0x00007f683cd7a28d in clone () from /lib64/libc.so.6
>> Thread 6 (Thread 0x7f67f97ea700 (LWP 4887)):
>> #0 0x00007f683d0506d5 in pthread_cond_wait@@GLIBC_2.3.2 () from
>> /lib64/libpthread.so.0
>> #1 0x00007f683d6a6050 in PR_WaitCondVar () from /lib64/libnspr4.so
>> #2 0x00007f683f999fee in connection_wait_for_new_work ()
>> #3 0x00007f683f99b719 in connection_threadmain ()
>> #4 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so
>> #5 0x00007f683d04cdc5 in start_thread () from /lib64/libpthread.so.0
>> #6 0x00007f683cd7a28d in clone () from /lib64/libc.so.6
>> Thread 5 (Thread 0x7f67f8fe9700 (LWP 4888)):
>> #0 0x00007f683d0506d5 in pthread_cond_wait@@GLIBC_2.3.2 () from
>> /lib64/libpthread.so.0
>> #1 0x00007f683d6a6050 in PR_WaitCondVar () from /lib64/libnspr4.so
>> #2 0x00007f683f999fee in connection_wait_for_new_work ()
>> #3 0x00007f683f99b719 in connection_threadmain ()
>> #4 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so
>> #5 0x00007f683d04cdc5 in start_thread () from /lib64/libpthread.so.0
>> #6 0x00007f683cd7a28d in clone () from /lib64/libc.so.6
>> Thread 4 (Thread 0x7f67f87e8700 (LWP 4889)):
>> #0 0x00007f683cd719b3 in select () from /lib64/libc.so.6
>> #1 0x00007f683f4fc0e9 in DS_Sleep () from
>> /usr/lib64/dirsrv/libslapd.so.0
>> #2 0x00007f683f99d325 in time_thread ()
>> #3 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so
>> #4 0x00007f683d04cdc5 in start_thread () from /lib64/libpthread.so.0
>> #5 0x00007f683cd7a28d in clone () from /lib64/libc.so.6
>> Thread 3 (Thread 0x7f67f7de6700 (LWP 4892)):
>> #0 0x00007f683d0506d5 in pthread_cond_wait@@GLIBC_2.3.2 () from
>> /lib64/libpthread.so.0
>> #1 0x00007f683d6a6050 in PR_WaitCondVar () from /lib64/libnspr4.so
>> #2 0x00007f683f9a8f25 in ps_send_results ()
>> #3 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so
>> #4 0x00007f683d04cdc5 in start_thread () from /lib64/libpthread.so.0
>> #5 0x00007f683cd7a28d in clone () from /lib64/libc.so.6
>> Thread 2 (Thread 0x7f67f75e5700 (LWP 5084)):
>> #0 0x00007f683d0506d5 in pthread_cond_wait@@GLIBC_2.3.2 () from
>> /lib64/libpthread.so.0
>> #1 0x00007f683d6a6050 in PR_WaitCondVar () from /lib64/libnspr4.so
>> #2 0x00007f683f9a8f25 in ps_send_results ()
>> #3 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so
>> #4 0x00007f683d04cdc5 in start_thread () from /lib64/libpthread.so.0
>> #5 0x00007f683cd7a28d in clone () from /lib64/libc.so.6
>> Thread 1 (Thread 0x7f683f964840 (LWP 4845)):
>> #0 0x00007f683d0506d5 in pthread_cond_wait@@GLIBC_2.3.2 () from
>> /lib64/libpthread.so.0
>> #1 0x00007f683d6a62c3 in PR_EnterMonitor () from /lib64/libnspr4.so
>> #2 0x00007f683f99d5ed in setup_pr_read_pds ()
>> #3 0x00007f683f9a0fd9 in slapd_daemon ()
>> #4 0x00007f683f992c6c in main ()
>>
>>
>>
>> the top output seems ok, there was just a sudden spike to around 340%
>> cpu then it comes down
>>
>> top - 08:53:12 up 6 days, 23:20, 1 user, load average: 0.00, 0.01, 0.12
>> Threads: 44 total, 0 running, 44 sleeping, 0 stopped, 0 zombie
>> %Cpu(s): 0.0 us, 0.0 sy, 0.0 ni,100.0 id, 0.0 wa, 0.0 hi, 0.0
>> si, 0.0 st
>> KiB Mem : 65533176 total, 52165676 free, 6135088 used, 7232412
>> buff/cache
>> KiB Swap: 3999996 total, 3999996 free, 0 used. 58452040
>> avail Mem
>>
>> PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+
>> COMMAND
>> 4845 dirsrv 20 0 5133196 3.150g 281272 S 0.0 5.0 31:34.51
>> ns-slapd
>> 4847 dirsrv 20 0 5133196 3.150g 281272 S 0.0 5.0 0:13.53
>> ns-slapd
>> 4848 dirsrv 20 0 5133196 3.150g 281272 S 0.0 5.0 0:13.16
>> ns-slapd
>> 4849 dirsrv 20 0 5133196 3.150g 281272 S 0.0 5.0 0:45.27
>> ns-slapd
>> 4850 dirsrv 20 0 5133196 3.150g 281272 S 0.0 5.0 0:01.60
>> ns-slapd
>> 4851 dirsrv 20 0 5133196 3.150g 281272 S 0.0 5.0 0:00.00
>> ns-slapd
>> 4852 dirsrv 20 0 5133196 3.150g 281272 S 0.0 5.0 0:00.00
>> ns-slapd
>> 4853 dirsrv 20 0 5133196 3.150g 281272 S 0.0 5.0 0:00.00
>> ns-slapd
>> 4854 dirsrv 20 0 5133196 3.150g 281272 S 0.0 5.0 0:00.00
>> ns-slapd
>> 4855 dirsrv 20 0 5133196 3.150g 281272 S 0.0 5.0 0:03.02
>> ns-slapd
>> 4856 dirsrv 20 0 5133196 3.150g 281272 S 0.0 5.0 0:00.26
>> ns-slapd
>> 4859 dirsrv 20 0 5133196 3.150g 281272 S 0.0 5.0 3:18.67
>> ns-slapd
>> 4860 dirsrv 20 0 5133196 3.150g 281272 S 0.0 5.0 3:19.82
>> ns-slapd
>> 4861 dirsrv 20 0 5133196 3.150g 281272 S 0.0 5.0 3:20.23
>> ns-slapd
>> 4862 dirsrv 20 0 5133196 3.150g 281272 S 0.0 5.0 3:22.65
>> ns-slapd
>> 4863 dirsrv 20 0 5133196 3.150g 281272 S 0.0 5.0 3:16.64
>> ns-slapd
>> 4864 dirsrv 20 0 5133196 3.150g 281272 S 0.0 5.0 3:17.39
>> ns-slapd
>> 4865 dirsrv 20 0 5133196 3.150g 281272 S 0.0 5.0 3:17.80
>> ns-slapd
>>
>>
>> Thanks
>> Rakesh
>>
>> On Tue, Sep 6, 2016 at 12:09 AM, Rakesh Rajasekharan
>> <rakesh.rajasekharan at gmail.com
>> <mailto:rakesh.rajasekharan at gmail.com>> wrote:
>>
>> Hi Thierry/Ludwig,
>>
>> Unfortunately I had by then restarted the service and since then
>> have been trying to hit the hang issue again but no luck this time
>>
>> It has been running fine for the last few hours now in-spite of
>> trying all that I believe could cause the hang like making
>> simultaneous client addition requests.
>>
>> I will continue testing this and next time I get it I will make
>> sure I get the pstack and also get the dump of db lock info.
>>
>> Thanks,
>> Rakesh
>>
>>
>>
>> On Mon, Sep 5, 2016 at 6:14 PM, Ludwig Krispenz
>> <lkrispen at redhat.com> wrote:
>>
>>
>> On 09/05/2016 02:31 PM, thierry bordaz wrote:
>>> Hi Rakesh,
>>>
>>> Thanks you soooo much for you continuous help on this !!
>>>
>>> If the hang is still going on, would you dump the DB lock
>>> information with:
>>>
>>> db_stat -CA -N -h /var/lib/dirsrv/slapd-<instance>/db
>>> >/tmp/db_lock_info
>>>
>>> It is looking like several threads are waiting for
>>> one/several db locks. With the lock info we will know which
>>> thread acquired it/them.
>>> A guess is that it could be the checkpointing thread but
>>> this one is also waiting (on a memory pool).
>>>
>>> I am copying Ludwig on this thread in case this pstack rings
>>> any bell.
>> the frequent presence of __db_tas_mutex_lock() looks like the
>> NUMA problem of libdb, although the threads already have
>> stopped spinning (maybe this was an effect of gcore).
>> If the hang aoccurs again, could you check cpu consumption
>> top -H -p <slapd-pid>
>> and get a pstack
>> pstack <slapd-pid>
>>>
>>> best regards
>>> thierry
>>>
>>> On 09/05/2016 02:05 PM, Rakesh Rajasekharan wrote:
>>>> Hi Thierry,
>>>>
>>>> I was able to now reporduce the issue..
>>>>
>>>> the ns-slapd process just got hung,.. however, no core file
>>>> was generate..
>>>>
>>>> I manually created a coredump using gcore and then took a
>>>> strace of it
>>>> this is strace from the coredump generated
>>>>
>>>> GNU gdb (GDB) Red Hat Enterprise Linux 7.6.1-80.el7
>>>> Copyright (C) 2013 Free Software Foundation, Inc.
>>>> License GPLv3+: GNU GPL version 3 or later
>>>> <http://gnu.org/licenses/gpl.html>
>>>> This is free software: you are free to change and
>>>> redistribute it.
>>>> There is NO WARRANTY, to the extent permitted by law. Type
>>>> "show copying"
>>>> and "show warranty" for details.
>>>> This GDB was configured as "x86_64-redhat-linux-gnu".
>>>> For bug reporting instructions, please see:
>>>> <http://www.gnu.org/software/gdb/bugs/
>>>> <http://www.gnu.org/software/gdb/bugs/>>...
>>>> Reading symbols from /usr/sbin/ns-slapd...Reading symbols
>>>> from /usr/sbin/ns-slapd...(no debugging symbols found)...done.
>>>> (no debugging symbols found)...done.
>>>> [New LWP 105159]
>>>> [New LWP 105160]
>>>> [New LWP 105161]
>>>> [New LWP 105162]
>>>> [New LWP 105163]
>>>> [New LWP 105164]
>>>> [New LWP 105165]
>>>> [New LWP 105166]
>>>> [New LWP 105167]
>>>> [New LWP 105168]
>>>> [New LWP 105171]
>>>> [New LWP 105172]
>>>> [New LWP 105173]
>>>> [New LWP 105174]
>>>> [New LWP 105175]
>>>> [New LWP 105176]
>>>> [New LWP 105177]
>>>> [New LWP 105178]
>>>> [New LWP 105179]
>>>> [New LWP 105180]
>>>> [New LWP 105181]
>>>> [New LWP 105182]
>>>> [New LWP 105183]
>>>> [New LWP 105184]
>>>> [New LWP 105185]
>>>> [New LWP 105186]
>>>> [New LWP 105187]
>>>> [New LWP 105188]
>>>> [New LWP 105189]
>>>> [New LWP 105190]
>>>> [New LWP 105191]
>>>> [New LWP 105192]
>>>> [New LWP 105193]
>>>> [New LWP 105194]
>>>> [New LWP 105195]
>>>> [New LWP 105196]
>>>> [New LWP 105197]
>>>> [New LWP 105198]
>>>> [New LWP 105199]
>>>> [New LWP 105200]
>>>> [New LWP 105201]
>>>> [New LWP 105203]
>>>> [New LWP 105215]
>>>> [New LWP 105158]
>>>> [Thread debugging using libthread_db enabled]
>>>> Using host libthread_db library "/lib64/libthread_db.so.1".
>>>> Core was generated by `/usr/sbin/ns-slapd'.
>>>> #0 0x00007f1b48ae69b3 in select () from /lib64/libc.so.6
>>>>
>>>> Thread 44 (Thread 0x7f1b4b6d9840 (LWP 105158)):
>>>> #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 ()
>>>> from /lib64/libpthread.so.0
>>>> No symbol table info available.
>>>> #1 0x00007f1b4941b2c3 in PR_EnterMonitor () from
>>>> /lib64/libnspr4.so
>>>> No symbol table info available.
>>>> #2 0x00007f1b4b7125ed in setup_pr_read_pds ()
>>>> No symbol table info available.
>>>> #3 0x00007f1b4b715fd9 in slapd_daemon ()
>>>> No symbol table info available.
>>>> #4 0x00007f1b4b707c6c in main ()
>>>> No symbol table info available.
>>>>
>>>> Thread 43 (Thread 0x7f1a5df0f700 (LWP 105215)):
>>>> #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 ()
>>>> from /lib64/libpthread.so.0
>>>> No symbol table info available.
>>>> #1 0x00007f1b4941b050 in PR_WaitCondVar () from
>>>> /lib64/libnspr4.so
>>>> No symbol table info available.
>>>> #2 0x00007f1b4b71df25 in ps_send_results ()
>>>> No symbol table info available.
>>>> #3 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so
>>>> No symbol table info available.
>>>> #4 0x00007f1b48dc1dc5 in start_thread () from
>>>> /lib64/libpthread.so.0
>>>> No symbol table info available.
>>>> #5 0x00007f1b48aef28d in clone () from /lib64/libc.so.6
>>>> No symbol table info available.
>>>>
>>>> Thread 42 (Thread 0x7f1b03fe7700 (LWP 105203)):
>>>> #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 ()
>>>> from /lib64/libpthread.so.0
>>>> No symbol table info available.
>>>> #1 0x00007f1b4941b050 in PR_WaitCondVar () from
>>>> /lib64/libnspr4.so
>>>> No symbol table info available.
>>>> #2 0x00007f1b4b71df25 in ps_send_results ()
>>>> No symbol table info available.
>>>> #3 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so
>>>> No symbol table info available.
>>>> #4 0x00007f1b48dc1dc5 in start_thread () from
>>>> /lib64/libpthread.so.0
>>>> No symbol table info available.
>>>> #5 0x00007f1b48aef28d in clone () from /lib64/libc.so.6
>>>> No symbol table info available.
>>>>
>>>> Thread 41 (Thread 0x7f1b047e8700 (LWP 105201)):
>>>> #0 0x00007f1b48ae69b3 in select () from /lib64/libc.so.6
>>>> No symbol table info available.
>>>> #1 0x00007f1b4b2710e9 in DS_Sleep () from
>>>> /usr/lib64/dirsrv/libslapd.so.0
>>>> No symbol table info available.
>>>> #2 0x00007f1b4b712325 in time_thread ()
>>>> No symbol table info available.
>>>> #3 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so
>>>> No symbol table info available.
>>>> #4 0x00007f1b48dc1dc5 in start_thread () from
>>>> /lib64/libpthread.so.0
>>>> No symbol table info available.
>>>> #5 0x00007f1b48aef28d in clone () from /lib64/libc.so.6
>>>> No symbol table info available.
>>>>
>>>> Thread 40 (Thread 0x7f1b04fe9700 (LWP 105200)):
>>>> #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 ()
>>>> from /lib64/libpthread.so.0
>>>> No symbol table info available.
>>>> #1 0x00007f1b433212f3 in __db_hybrid_mutex_suspend () from
>>>> /lib64/libdb-5.3.so <http://libdb-5.3.so>
>>>> No symbol table info available.
>>>> #2 0x00007f1b43320640 in __db_tas_mutex_lock () from
>>>> /lib64/libdb-5.3.so <http://libdb-5.3.so>
>>>> No symbol table info available.
>>>> #3 0x00007f1b4343ae1b in __memp_fget () from
>>>> /lib64/libdb-5.3.so <http://libdb-5.3.so>
>>>> No symbol table info available.
>>>> #4 0x00007f1b4333e211 in __bam_search () from
>>>> /lib64/libdb-5.3.so <http://libdb-5.3.so>
>>>> No symbol table info available.
>>>> #5 0x00007f1b43329256 in __bamc_search () from
>>>> /lib64/libdb-5.3.so <http://libdb-5.3.so>
>>>> No symbol table info available.
>>>> #6 0x00007f1b4332d6c4 in __bamc_put () from
>>>> /lib64/libdb-5.3.so <http://libdb-5.3.so>
>>>> No symbol table info available.
>>>> #7 0x00007f1b433e46d5 in __dbc_iput () from
>>>> /lib64/libdb-5.3.so <http://libdb-5.3.so>
>>>> No symbol table info available.
>>>> #8 0x00007f1b433dfa0e in __db_put () from
>>>> /lib64/libdb-5.3.so <http://libdb-5.3.so>
>>>> No symbol table info available.
>>>> #9 0x00007f1b433f4fa4 in __db_put_pp () from
>>>> /lib64/libdb-5.3.so <http://libdb-5.3.so>
>>>> No symbol table info available.
>>>> #10 0x00007f1b3e8cf4cf in idl_new_insert_key () from
>>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so
>>>> No symbol table info available.
>>>> #11 0x00007f1b3e8de5b5 in addordel_values_sv.isra.3 () from
>>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so
>>>> No symbol table info available.
>>>> #12 0x00007f1b3e8df25a in index_addordel_values_ext_sv ()
>>>> from /usr/lib64/dirsrv/plugins/libback-ldbm.so
>>>> No symbol table info available.
>>>> #13 0x00007f1b3e8df694 in index_addordel_values_sv () from
>>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so
>>>> No symbol table info available.
>>>> #14 0x00007f1b3e8df75a in index_addordel_entry () from
>>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so
>>>> No symbol table info available.
>>>> #15 0x00007f1b3e8e25fa in ldbm_back_add () from
>>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so
>>>> No symbol table info available.
>>>> #16 0x00007f1b4b1e4cca in op_shared_add () from
>>>> /usr/lib64/dirsrv/libslapd.so.0
>>>> No symbol table info available.
>>>> #17 0x00007f1b4b1e6020 in do_add () from
>>>> /usr/lib64/dirsrv/libslapd.so.0
>>>> No symbol table info available.
>>>> #18 0x00007f1b4b711303 in connection_threadmain ()
>>>> No symbol table info available.
>>>> #19 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so
>>>> No symbol table info available.
>>>> #20 0x00007f1b48dc1dc5 in start_thread () from
>>>> /lib64/libpthread.so.0
>>>> No symbol table info available.
>>>> #21 0x00007f1b48aef28d in clone () from /lib64/libc.so.6
>>>> No symbol table info available.
>>>>
>>>> Thread 39 (Thread 0x7f1b057ea700 (LWP 105199)):
>>>> #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 ()
>>>> from /lib64/libpthread.so.0
>>>> No symbol table info available.
>>>> #1 0x00007f1b4941b050 in PR_WaitCondVar () from
>>>> /lib64/libnspr4.so
>>>> No symbol table info available.
>>>> #2 0x00007f1b4b70efee in connection_wait_for_new_work ()
>>>> No symbol table info available.
>>>> #3 0x00007f1b4b710719 in connection_threadmain ()
>>>> No symbol table info available.
>>>> #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so
>>>> No symbol table info available.
>>>> #5 0x00007f1b48dc1dc5 in start_thread () from
>>>> /lib64/libpthread.so.0
>>>> No symbol table info available.
>>>> #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6
>>>> No symbol table info available.
>>>>
>>>> Thread 38 (Thread 0x7f1b05feb700 (LWP 105198)):
>>>> #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 ()
>>>> from /lib64/libpthread.so.0
>>>> No symbol table info available.
>>>> #1 0x00007f1b4941b050 in PR_WaitCondVar () from
>>>> /lib64/libnspr4.so
>>>> No symbol table info available.
>>>> #2 0x00007f1b4b70efee in connection_wait_for_new_work ()
>>>> No symbol table info available.
>>>> #3 0x00007f1b4b710719 in connection_threadmain ()
>>>> No symbol table info available.
>>>> #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so
>>>> No symbol table info available.
>>>> #5 0x00007f1b48dc1dc5 in start_thread () from
>>>> /lib64/libpthread.so.0
>>>> No symbol table info available.
>>>> #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6
>>>> No symbol table info available.
>>>>
>>>> Thread 37 (Thread 0x7f1b067ec700 (LWP 105197)):
>>>> #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 ()
>>>> from /lib64/libpthread.so.0
>>>> No symbol table info available.
>>>> #1 0x00007f1b4941b050 in PR_WaitCondVar () from
>>>> /lib64/libnspr4.so
>>>> No symbol table info available.
>>>> #2 0x00007f1b4b70efee in connection_wait_for_new_work ()
>>>> No symbol table info available.
>>>> #3 0x00007f1b4b710719 in connection_threadmain ()
>>>> No symbol table info available.
>>>> #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so
>>>> No symbol table info available.
>>>> #5 0x00007f1b48dc1dc5 in start_thread () from
>>>> /lib64/libpthread.so.0
>>>> No symbol table info available.
>>>> #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6
>>>> No symbol table info available.
>>>>
>>>> Thread 36 (Thread 0x7f1b06fed700 (LWP 105196)):
>>>> #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 ()
>>>> from /lib64/libpthread.so.0
>>>> No symbol table info available.
>>>> #1 0x00007f1b4941b050 in PR_WaitCondVar () from
>>>> /lib64/libnspr4.so
>>>> No symbol table info available.
>>>> #2 0x00007f1b4b70efee in connection_wait_for_new_work ()
>>>> No symbol table info available.
>>>> #3 0x00007f1b4b710719 in connection_threadmain ()
>>>> No symbol table info available.
>>>> #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so
>>>> No symbol table info available.
>>>> #5 0x00007f1b48dc1dc5 in start_thread () from
>>>> /lib64/libpthread.so.0
>>>> No symbol table info available.
>>>> #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6
>>>> No symbol table info available.
>>>>
>>>> Thread 35 (Thread 0x7f1b077ee700 (LWP 105195)):
>>>> #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 ()
>>>> from /lib64/libpthread.so.0
>>>> No symbol table info available.
>>>> #1 0x00007f1b4941b2c3 in PR_EnterMonitor () from
>>>> /lib64/libnspr4.so
>>>> No symbol table info available.
>>>> #2 0x00007f1b3e8c23e6 in dblayer_txn_begin () from
>>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so
>>>> No symbol table info available.
>>>> #3 0x00007f1b3e8fe007 in ldbm_back_modify () from
>>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so
>>>> No symbol table info available.
>>>> #4 0x00007f1b4b22b541 in op_shared_modify () from
>>>> /usr/lib64/dirsrv/libslapd.so.0
>>>> No symbol table info available.
>>>> #5 0x00007f1b4b22c004 in modify_internal_pb () from
>>>> /usr/lib64/dirsrv/libslapd.so.0
>>>> No symbol table info available.
>>>> #6 0x00007f1b3eb48b60 in ipapwd_apply_mods () from
>>>> /usr/lib64/dirsrv/plugins/libipa_pwd_extop.so
>>>> No symbol table info available.
>>>> #7 0x00007f1b3eb4c8e0 in store_new_keys () from
>>>> /usr/lib64/dirsrv/plugins/libipa_pwd_extop.so
>>>> No symbol table info available.
>>>> #8 0x00007f1b3eb4ead0 in ipapwd_extop () from
>>>> /usr/lib64/dirsrv/plugins/libipa_pwd_extop.so
>>>> No symbol table info available.
>>>> #9 0x00007f1b4b23bba2 in plugin_call_exop_plugins () from
>>>> /usr/lib64/dirsrv/libslapd.so.0
>>>> No symbol table info available.
>>>> #10 0x00007f1b4b717efe in do_extended ()
>>>> No symbol table info available.
>>>> #11 0x00007f1b4b71141a in connection_threadmain ()
>>>> No symbol table info available.
>>>> #12 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so
>>>> No symbol table info available.
>>>> #13 0x00007f1b48dc1dc5 in start_thread () from
>>>> /lib64/libpthread.so.0
>>>> No symbol table info available.
>>>> #14 0x00007f1b48aef28d in clone () from /lib64/libc.so.6
>>>> No symbol table info available.
>>>>
>>>> Thread 34 (Thread 0x7f1b07fef700 (LWP 105194)):
>>>> #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 ()
>>>> from /lib64/libpthread.so.0
>>>> No symbol table info available.
>>>> #1 0x00007f1b4941b050 in PR_WaitCondVar () from
>>>> /lib64/libnspr4.so
>>>> No symbol table info available.
>>>> #2 0x00007f1b4b70efee in connection_wait_for_new_work ()
>>>> No symbol table info available.
>>>> #3 0x00007f1b4b710719 in connection_threadmain ()
>>>> No symbol table info available.
>>>> #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so
>>>> No symbol table info available.
>>>> #5 0x00007f1b48dc1dc5 in start_thread () from
>>>> /lib64/libpthread.so.0
>>>> No symbol table info available.
>>>> #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6
>>>> No symbol table info available.
>>>>
>>>> Thread 33 (Thread 0x7f1b087f0700 (LWP 105193)):
>>>> #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 ()
>>>> from /lib64/libpthread.so.0
>>>> No symbol table info available.
>>>> #1 0x00007f1b4941b050 in PR_WaitCondVar () from
>>>> /lib64/libnspr4.so
>>>> No symbol table info available.
>>>> #2 0x00007f1b4b70efee in connection_wait_for_new_work ()
>>>> No symbol table info available.
>>>> #3 0x00007f1b4b710719 in connection_threadmain ()
>>>> No symbol table info available.
>>>> #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so
>>>> No symbol table info available.
>>>> #5 0x00007f1b48dc1dc5 in start_thread () from
>>>> /lib64/libpthread.so.0
>>>> No symbol table info available.
>>>> #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6
>>>> No symbol table info available.
>>>>
>>>> Thread 32 (Thread 0x7f1b08ff1700 (LWP 105192)):
>>>> #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 ()
>>>> from /lib64/libpthread.so.0
>>>> No symbol table info available.
>>>> #1 0x00007f1b4941b050 in PR_WaitCondVar () from
>>>> /lib64/libnspr4.so
>>>> No symbol table info available.
>>>> #2 0x00007f1b4b70efee in connection_wait_for_new_work ()
>>>> No symbol table info available.
>>>> #3 0x00007f1b4b710719 in connection_threadmain ()
>>>> No symbol table info available.
>>>> #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so
>>>> No symbol table info available.
>>>> #5 0x00007f1b48dc1dc5 in start_thread () from
>>>> /lib64/libpthread.so.0
>>>> No symbol table info available.
>>>> #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6
>>>> No symbol table info available.
>>>>
>>>> Thread 31 (Thread 0x7f1b097f2700 (LWP 105191)):
>>>> #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 ()
>>>> from /lib64/libpthread.so.0
>>>> No symbol table info available.
>>>> #1 0x00007f1b4941b050 in PR_WaitCondVar () from
>>>> /lib64/libnspr4.so
>>>> No symbol table info available.
>>>> #2 0x00007f1b4b70efee in connection_wait_for_new_work ()
>>>> No symbol table info available.
>>>> #3 0x00007f1b4b710719 in connection_threadmain ()
>>>> No symbol table info available.
>>>> #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so
>>>> No symbol table info available.
>>>> #5 0x00007f1b48dc1dc5 in start_thread () from
>>>> /lib64/libpthread.so.0
>>>> No symbol table info available.
>>>> #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6
>>>> No symbol table info available.
>>>>
>>>> Thread 30 (Thread 0x7f1b09ff3700 (LWP 105190)):
>>>> #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 ()
>>>> from /lib64/libpthread.so.0
>>>> No symbol table info available.
>>>> #1 0x00007f1b433212f3 in __db_hybrid_mutex_suspend () from
>>>> /lib64/libdb-5.3.so <http://libdb-5.3.so>
>>>> No symbol table info available.
>>>> #2 0x00007f1b43320640 in __db_tas_mutex_lock () from
>>>> /lib64/libdb-5.3.so <http://libdb-5.3.so>
>>>> No symbol table info available.
>>>> #3 0x00007f1b433cad3a in __lock_get_internal () from
>>>> /lib64/libdb-5.3.so <http://libdb-5.3.so>
>>>> No symbol table info available.
>>>> #4 0x00007f1b433cb820 in __lock_get () from
>>>> /lib64/libdb-5.3.so <http://libdb-5.3.so>
>>>> No symbol table info available.
>>>> #5 0x00007f1b433f7142 in __db_lget () from
>>>> /lib64/libdb-5.3.so <http://libdb-5.3.so>
>>>> No symbol table info available.
>>>> #6 0x00007f1b4333e605 in __bam_search () from
>>>> /lib64/libdb-5.3.so <http://libdb-5.3.so>
>>>> No symbol table info available.
>>>> #7 0x00007f1b43329256 in __bamc_search () from
>>>> /lib64/libdb-5.3.so <http://libdb-5.3.so>
>>>> No symbol table info available.
>>>> #8 0x00007f1b4332ad0f in __bamc_get () from
>>>> /lib64/libdb-5.3.so <http://libdb-5.3.so>
>>>> No symbol table info available.
>>>> #9 0x00007f1b433e3ca6 in __dbc_iget () from
>>>> /lib64/libdb-5.3.so <http://libdb-5.3.so>
>>>> No symbol table info available.
>>>> #10 0x00007f1b433f2b02 in __dbc_get_pp () from
>>>> /lib64/libdb-5.3.so <http://libdb-5.3.so>
>>>> No symbol table info available.
>>>> #11 0x00007f1b3e8ce680 in idl_new_fetch () from
>>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so
>>>> No symbol table info available.
>>>> #12 0x00007f1b3e8dcb26 in index_read_ext_allids () from
>>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so
>>>> No symbol table info available.
>>>> #13 0x00007f1b3e8c7284 in keys2idl () from
>>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so
>>>> No symbol table info available.
>>>> #14 0x00007f1b3e8c79e3 in ava_candidates.isra.0 () from
>>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so
>>>> No symbol table info available.
>>>> #15 0x00007f1b3e8c7fd2 in filter_candidates_ext () from
>>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so
>>>> No symbol table info available.
>>>> #16 0x00007f1b3e8c9096 in list_candidates () from
>>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so
>>>> No symbol table info available.
>>>> #17 0x00007f1b3e8c7f40 in filter_candidates_ext () from
>>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so
>>>> No symbol table info available.
>>>> #18 0x00007f1b3e8c9096 in list_candidates () from
>>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so
>>>> No symbol table info available.
>>>> #19 0x00007f1b3e8c7f40 in filter_candidates_ext () from
>>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so
>>>> No symbol table info available.
>>>> #20 0x00007f1b3e8c9096 in list_candidates () from
>>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so
>>>> No symbol table info available.
>>>> #21 0x00007f1b3e8c7f40 in filter_candidates_ext () from
>>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so
>>>> No symbol table info available.
>>>> #22 0x00007f1b3e90380c in subtree_candidates () from
>>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so
>>>> No symbol table info available.
>>>> #23 0x00007f1b3e904e7f in ldbm_back_search () from
>>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so
>>>> No symbol table info available.
>>>> #24 0x00007f1b4b2321d6 in op_shared_search () from
>>>> /usr/lib64/dirsrv/libslapd.so.0
>>>> No symbol table info available.
>>>> #25 0x00007f1b4b72333e in do_search ()
>>>> No symbol table info available.
>>>> #26 0x00007f1b4b7113c3 in connection_threadmain ()
>>>> No symbol table info available.
>>>> #27 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so
>>>> No symbol table info available.
>>>> #28 0x00007f1b48dc1dc5 in start_thread () from
>>>> /lib64/libpthread.so.0
>>>> No symbol table info available.
>>>> #29 0x00007f1b48aef28d in clone () from /lib64/libc.so.6
>>>> No symbol table info available.
>>>>
>>>> Thread 29 (Thread 0x7f1b0a7f4700 (LWP 105189)):
>>>> #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 ()
>>>> from /lib64/libpthread.so.0
>>>> No symbol table info available.
>>>> #1 0x00007f1b4941b050 in PR_WaitCondVar () from
>>>> /lib64/libnspr4.so
>>>> No symbol table info available.
>>>> #2 0x00007f1b4b70efee in connection_wait_for_new_work ()
>>>> No symbol table info available.
>>>> #3 0x00007f1b4b710719 in connection_threadmain ()
>>>> No symbol table info available.
>>>> #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so
>>>> No symbol table info available.
>>>> #5 0x00007f1b48dc1dc5 in start_thread () from
>>>> /lib64/libpthread.so.0
>>>> No symbol table info available.
>>>> #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6
>>>> No symbol table info available.
>>>>
>>>> Thread 28 (Thread 0x7f1b0aff5700 (LWP 105188)):
>>>> #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 ()
>>>> from /lib64/libpthread.so.0
>>>> No symbol table info available.
>>>> #1 0x00007f1b4941b050 in PR_WaitCondVar () from
>>>> /lib64/libnspr4.so
>>>> No symbol table info available.
>>>> #2 0x00007f1b4b70efee in connection_wait_for_new_work ()
>>>> No symbol table info available.
>>>> #3 0x00007f1b4b710719 in connection_threadmain ()
>>>> No symbol table info available.
>>>> #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so
>>>> No symbol table info available.
>>>> #5 0x00007f1b48dc1dc5 in start_thread () from
>>>> /lib64/libpthread.so.0
>>>> No symbol table info available.
>>>> #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6
>>>> No symbol table info available.
>>>>
>>>> Thread 27 (Thread 0x7f1b0b7f6700 (LWP 105187)):
>>>> #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 ()
>>>> from /lib64/libpthread.so.0
>>>> No symbol table info available.
>>>> #1 0x00007f1b4941b050 in PR_WaitCondVar () from
>>>> /lib64/libnspr4.so
>>>> No symbol table info available.
>>>> #2 0x00007f1b4b70efee in connection_wait_for_new_work ()
>>>> No symbol table info available.
>>>> #3 0x00007f1b4b710719 in connection_threadmain ()
>>>> No symbol table info available.
>>>> #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so
>>>> No symbol table info available.
>>>> #5 0x00007f1b48dc1dc5 in start_thread () from
>>>> /lib64/libpthread.so.0
>>>> No symbol table info available.
>>>> #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6
>>>> No symbol table info available.
>>>>
>>>> Thread 26 (Thread 0x7f1b0bff7700 (LWP 105186)):
>>>> #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 ()
>>>> from /lib64/libpthread.so.0
>>>> No symbol table info available.
>>>> #1 0x00007f1b433212f3 in __db_hybrid_mutex_suspend () from
>>>> /lib64/libdb-5.3.so <http://libdb-5.3.so>
>>>> No symbol table info available.
>>>> #2 0x00007f1b43320640 in __db_tas_mutex_lock () from
>>>> /lib64/libdb-5.3.so <http://libdb-5.3.so>
>>>> No symbol table info available.
>>>> #3 0x00007f1b433cad3a in __lock_get_internal () from
>>>> /lib64/libdb-5.3.so <http://libdb-5.3.so>
>>>> No symbol table info available.
>>>> #4 0x00007f1b433cb820 in __lock_get () from
>>>> /lib64/libdb-5.3.so <http://libdb-5.3.so>
>>>> No symbol table info available.
>>>> #5 0x00007f1b433f7142 in __db_lget () from
>>>> /lib64/libdb-5.3.so <http://libdb-5.3.so>
>>>> No symbol table info available.
>>>> #6 0x00007f1b4333e605 in __bam_search () from
>>>> /lib64/libdb-5.3.so <http://libdb-5.3.so>
>>>> No symbol table info available.
>>>> #7 0x00007f1b43329256 in __bamc_search () from
>>>> /lib64/libdb-5.3.so <http://libdb-5.3.so>
>>>> No symbol table info available.
>>>> #8 0x00007f1b4332ad0f in __bamc_get () from
>>>> /lib64/libdb-5.3.so <http://libdb-5.3.so>
>>>> No symbol table info available.
>>>> #9 0x00007f1b433e3ca6 in __dbc_iget () from
>>>> /lib64/libdb-5.3.so <http://libdb-5.3.so>
>>>> No symbol table info available.
>>>> #10 0x00007f1b433f2b02 in __dbc_get_pp () from
>>>> /lib64/libdb-5.3.so <http://libdb-5.3.so>
>>>> No symbol table info available.
>>>> #11 0x00007f1b3e8ce680 in idl_new_fetch () from
>>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so
>>>> No symbol table info available.
>>>> #12 0x00007f1b3e8dcb26 in index_read_ext_allids () from
>>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so
>>>> No symbol table info available.
>>>> #13 0x00007f1b3e8c7284 in keys2idl () from
>>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so
>>>> No symbol table info available.
>>>> #14 0x00007f1b3e8c79e3 in ava_candidates.isra.0 () from
>>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so
>>>> No symbol table info available.
>>>> #15 0x00007f1b3e8c7fd2 in filter_candidates_ext () from
>>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so
>>>> No symbol table info available.
>>>> #16 0x00007f1b3e8c9096 in list_candidates () from
>>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so
>>>> No symbol table info available.
>>>> #17 0x00007f1b3e8c7f40 in filter_candidates_ext () from
>>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so
>>>> No symbol table info available.
>>>> #18 0x00007f1b3e90380c in subtree_candidates () from
>>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so
>>>> No symbol table info available.
>>>> #19 0x00007f1b3e904e7f in ldbm_back_search () from
>>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so
>>>> No symbol table info available.
>>>> #20 0x00007f1b4b2321d6 in op_shared_search () from
>>>> /usr/lib64/dirsrv/libslapd.so.0
>>>> No symbol table info available.
>>>> #21 0x00007f1b4b24294e in search_internal_callback_pb ()
>>>> from /usr/lib64/dirsrv/libslapd.so.0
>>>> No symbol table info available.
>>>> #22 0x00007f1b4b242be8 in search_internal_pb () from
>>>> /usr/lib64/dirsrv/libslapd.so.0
>>>> No symbol table info available.
>>>> #23 0x00007f1b4b72149b in ids_sasl_canon_user ()
>>>> No symbol table info available.
>>>> #24 0x00007f1b491e193b in _sasl_canon_user () from
>>>> /lib64/libsasl2.so.3
>>>> No symbol table info available.
>>>> #25 0x00007f1b491e1c4c in _sasl_canon_user_lookup () from
>>>> /lib64/libsasl2.so.3
>>>> No symbol table info available.
>>>> #26 0x00007f1b430e9fb9 in gssapi_server_mech_step () from
>>>> /usr/lib64/sasl2/libgssapiv2.so
>>>> No symbol table info available.
>>>> #27 0x00007f1b491edb9b in sasl_server_step () from
>>>> /lib64/libsasl2.so.3
>>>> No symbol table info available.
>>>> #28 0x00007f1b4b72283b in ids_sasl_check_bind ()
>>>> No symbol table info available.
>>>> #29 0x00007f1b4b70a61a in do_bind ()
>>>> No symbol table info available.
>>>> #30 0x00007f1b4b7113fd in connection_threadmain ()
>>>> No symbol table info available.
>>>> #31 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so
>>>> No symbol table info available.
>>>> #32 0x00007f1b48dc1dc5 in start_thread () from
>>>> /lib64/libpthread.so.0
>>>> No symbol table info available.
>>>> #33 0x00007f1b48aef28d in clone () from /lib64/libc.so.6
>>>> No symbol table info available.
>>>>
>>>> Thread 25 (Thread 0x7f1b0c7f8700 (LWP 105185)):
>>>> #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 ()
>>>> from /lib64/libpthread.so.0
>>>> No symbol table info available.
>>>> #1 0x00007f1b4941b050 in PR_WaitCondVar () from
>>>> /lib64/libnspr4.so
>>>> No symbol table info available.
>>>> #2 0x00007f1b4b70efee in connection_wait_for_new_work ()
>>>> No symbol table info available.
>>>> #3 0x00007f1b4b710719 in connection_threadmain ()
>>>> No symbol table info available.
>>>> #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so
>>>> No symbol table info available.
>>>> #5 0x00007f1b48dc1dc5 in start_thread () from
>>>> /lib64/libpthread.so.0
>>>> No symbol table info available.
>>>> #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6
>>>> No symbol table info available.
>>>>
>>>> Thread 24 (Thread 0x7f1b0cff9700 (LWP 105184)):
>>>> #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 ()
>>>> from /lib64/libpthread.so.0
>>>> No symbol table info available.
>>>> #1 0x00007f1b4941b050 in PR_WaitCondVar () from
>>>> /lib64/libnspr4.so
>>>> No symbol table info available.
>>>> #2 0x00007f1b4b70efee in connection_wait_for_new_work ()
>>>> No symbol table info available.
>>>> #3 0x00007f1b4b710719 in connection_threadmain ()
>>>> No symbol table info available.
>>>> #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so
>>>> No symbol table info available.
>>>> #5 0x00007f1b48dc1dc5 in start_thread () from
>>>> /lib64/libpthread.so.0
>>>> No symbol table info available.
>>>> #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6
>>>> No symbol table info available.
>>>>
>>>> Thread 23 (Thread 0x7f1b0d7fa700 (LWP 105183)):
>>>> #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 ()
>>>> from /lib64/libpthread.so.0
>>>> No symbol table info available.
>>>> #1 0x00007f1b4941b050 in PR_WaitCondVar () from
>>>> /lib64/libnspr4.so
>>>> No symbol table info available.
>>>> #2 0x00007f1b4b70efee in connection_wait_for_new_work ()
>>>> No symbol table info available.
>>>> #3 0x00007f1b4b710719 in connection_threadmain ()
>>>> No symbol table info available.
>>>> #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so
>>>> No symbol table info available.
>>>> #5 0x00007f1b48dc1dc5 in start_thread () from
>>>> /lib64/libpthread.so.0
>>>> No symbol table info available.
>>>> #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6
>>>> No symbol table info available.
>>>>
>>>> Thread 22 (Thread 0x7f1b0dffb700 (LWP 105182)):
>>>> #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 ()
>>>> from /lib64/libpthread.so.0
>>>> No symbol table info available.
>>>> #1 0x00007f1b4941b050 in PR_WaitCondVar () from
>>>> /lib64/libnspr4.so
>>>> No symbol table info available.
>>>> #2 0x00007f1b4b70efee in connection_wait_for_new_work ()
>>>> No symbol table info available.
>>>> #3 0x00007f1b4b710719 in connection_threadmain ()
>>>> No symbol table info available.
>>>> #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so
>>>> No symbol table info available.
>>>> #5 0x00007f1b48dc1dc5 in start_thread () from
>>>> /lib64/libpthread.so.0
>>>> No symbol table info available.
>>>> #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6
>>>> No symbol table info available.
>>>>
>>>> Thread 21 (Thread 0x7f1b0e7fc700 (LWP 105181)):
>>>> #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 ()
>>>> from /lib64/libpthread.so.0
>>>> No symbol table info available.
>>>> #1 0x00007f1b4941b050 in PR_WaitCondVar () from
>>>> /lib64/libnspr4.so
>>>> No symbol table info available.
>>>> #2 0x00007f1b4b70efee in connection_wait_for_new_work ()
>>>> No symbol table info available.
>>>> #3 0x00007f1b4b710719 in connection_threadmain ()
>>>> No symbol table info available.
>>>> #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so
>>>> No symbol table info available.
>>>> #5 0x00007f1b48dc1dc5 in start_thread () from
>>>> /lib64/libpthread.so.0
>>>> No symbol table info available.
>>>> #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6
>>>> No symbol table info available.
>>>>
>>>> Thread 20 (Thread 0x7f1b0effd700 (LWP 105180)):
>>>> #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 ()
>>>> from /lib64/libpthread.so.0
>>>> No symbol table info available.
>>>> #1 0x00007f1b433212f3 in __db_hybrid_mutex_suspend () from
>>>> /lib64/libdb-5.3.so <http://libdb-5.3.so>
>>>> No symbol table info available.
>>>> #2 0x00007f1b43320b37 in __db_tas_mutex_readlock () from
>>>> /lib64/libdb-5.3.so <http://libdb-5.3.so>
>>>> No symbol table info available.
>>>> #3 0x00007f1b4343ad35 in __memp_fget () from
>>>> /lib64/libdb-5.3.so <http://libdb-5.3.so>
>>>> No symbol table info available.
>>>> #4 0x00007f1b433e4333 in __dbc_iget () from
>>>> /lib64/libdb-5.3.so <http://libdb-5.3.so>
>>>> No symbol table info available.
>>>> #5 0x00007f1b433f2b02 in __dbc_get_pp () from
>>>> /lib64/libdb-5.3.so <http://libdb-5.3.so>
>>>> No symbol table info available.
>>>> #6 0x00007f1b3e8ce822 in idl_new_fetch () from
>>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so
>>>> No symbol table info available.
>>>> #7 0x00007f1b3e8dcb26 in index_read_ext_allids () from
>>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so
>>>> No symbol table info available.
>>>> #8 0x00007f1b3e8c7284 in keys2idl () from
>>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so
>>>> No symbol table info available.
>>>> #9 0x00007f1b3e8c79e3 in ava_candidates.isra.0 () from
>>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so
>>>> No symbol table info available.
>>>> #10 0x00007f1b3e8c7fd2 in filter_candidates_ext () from
>>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so
>>>> No symbol table info available.
>>>> #11 0x00007f1b3e8c9096 in list_candidates () from
>>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so
>>>> No symbol table info available.
>>>> #12 0x00007f1b3e8c7f40 in filter_candidates_ext () from
>>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so
>>>> No symbol table info available.
>>>> #13 0x00007f1b3e8c9096 in list_candidates () from
>>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so
>>>> No symbol table info available.
>>>> #14 0x00007f1b3e8c7f40 in filter_candidates_ext () from
>>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so
>>>> No symbol table info available.
>>>> #15 0x00007f1b3e8c9096 in list_candidates () from
>>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so
>>>> No symbol table info available.
>>>> #16 0x00007f1b3e8c7f40 in filter_candidates_ext () from
>>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so
>>>> No symbol table info available.
>>>> #17 0x00007f1b3e90380c in subtree_candidates () from
>>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so
>>>> No symbol table info available.
>>>> #18 0x00007f1b3e904e7f in ldbm_back_search () from
>>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so
>>>> No symbol table info available.
>>>> #19 0x00007f1b4b2321d6 in op_shared_search () from
>>>> /usr/lib64/dirsrv/libslapd.so.0
>>>> No symbol table info available.
>>>> #20 0x00007f1b4b72333e in do_search ()
>>>> No symbol table info available.
>>>> #21 0x00007f1b4b7113c3 in connection_threadmain ()
>>>> No symbol table info available.
>>>> #22 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so
>>>> No symbol table info available.
>>>> #23 0x00007f1b48dc1dc5 in start_thread () from
>>>> /lib64/libpthread.so.0
>>>> No symbol table info available.
>>>> #24 0x00007f1b48aef28d in clone () from /lib64/libc.so.6
>>>> No symbol table info available.
>>>>
>>>> Thread 19 (Thread 0x7f1b0f7fe700 (LWP 105179)):
>>>> #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 ()
>>>> from /lib64/libpthread.so.0
>>>> No symbol table info available.
>>>> #1 0x00007f1b433212f3 in __db_hybrid_mutex_suspend () from
>>>> /lib64/libdb-5.3.so <http://libdb-5.3.so>
>>>> No symbol table info available.
>>>> #2 0x00007f1b43320640 in __db_tas_mutex_lock () from
>>>> /lib64/libdb-5.3.so <http://libdb-5.3.so>
>>>> No symbol table info available.
>>>> #3 0x00007f1b433cad3a in __lock_get_internal () from
>>>> /lib64/libdb-5.3.so <http://libdb-5.3.so>
>>>> No symbol table info available.
>>>> #4 0x00007f1b433cb820 in __lock_get () from
>>>> /lib64/libdb-5.3.so <http://libdb-5.3.so>
>>>> No symbol table info available.
>>>> #5 0x00007f1b433f7142 in __db_lget () from
>>>> /lib64/libdb-5.3.so <http://libdb-5.3.so>
>>>> No symbol table info available.
>>>> #6 0x00007f1b4333e605 in __bam_search () from
>>>> /lib64/libdb-5.3.so <http://libdb-5.3.so>
>>>> No symbol table info available.
>>>> #7 0x00007f1b43329256 in __bamc_search () from
>>>> /lib64/libdb-5.3.so <http://libdb-5.3.so>
>>>> No symbol table info available.
>>>> #8 0x00007f1b4332ad0f in __bamc_get () from
>>>> /lib64/libdb-5.3.so <http://libdb-5.3.so>
>>>> No symbol table info available.
>>>> #9 0x00007f1b433e3ca6 in __dbc_iget () from
>>>> /lib64/libdb-5.3.so <http://libdb-5.3.so>
>>>> No symbol table info available.
>>>> #10 0x00007f1b433f2b02 in __dbc_get_pp () from
>>>> /lib64/libdb-5.3.so <http://libdb-5.3.so>
>>>> No symbol table info available.
>>>> #11 0x00007f1b3e8ce680 in idl_new_fetch () from
>>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so
>>>> No symbol table info available.
>>>> #12 0x00007f1b3e8dcb26 in index_read_ext_allids () from
>>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so
>>>> No symbol table info available.
>>>> #13 0x00007f1b3e8c7284 in keys2idl () from
>>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so
>>>> No symbol table info available.
>>>> #14 0x00007f1b3e8c79e3 in ava_candidates.isra.0 () from
>>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so
>>>> No symbol table info available.
>>>> #15 0x00007f1b3e8c7fd2 in filter_candidates_ext () from
>>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so
>>>> No symbol table info available.
>>>> #16 0x00007f1b3e8c9096 in list_candidates () from
>>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so
>>>> No symbol table info available.
>>>> #17 0x00007f1b3e8c7f40 in filter_candidates_ext () from
>>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so
>>>> No symbol table info available.
>>>> #18 0x00007f1b3e8c9096 in list_candidates () from
>>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so
>>>> No symbol table info available.
>>>> #19 0x00007f1b3e8c7f40 in filter_candidates_ext () from
>>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so
>>>> No symbol table info available.
>>>> #20 0x00007f1b3e8c9096 in list_candidates () from
>>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so
>>>> No symbol table info available.
>>>> #21 0x00007f1b3e8c7f40 in filter_candidates_ext () from
>>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so
>>>> No symbol table info available.
>>>> #22 0x00007f1b3e90380c in subtree_candidates () from
>>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so
>>>> No symbol table info available.
>>>> #23 0x00007f1b3e904e7f in ldbm_back_search () from
>>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so
>>>> No symbol table info available.
>>>> #24 0x00007f1b4b2321d6 in op_shared_search () from
>>>> /usr/lib64/dirsrv/libslapd.so.0
>>>> No symbol table info available.
>>>> #25 0x00007f1b4b72333e in do_search ()
>>>> No symbol table info available.
>>>> #26 0x00007f1b4b7113c3 in connection_threadmain ()
>>>> No symbol table info available.
>>>> #27 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so
>>>> No symbol table info available.
>>>> #28 0x00007f1b48dc1dc5 in start_thread () from
>>>> /lib64/libpthread.so.0
>>>> No symbol table info available.
>>>> #29 0x00007f1b48aef28d in clone () from /lib64/libc.so.6
>>>> No symbol table info available.
>>>>
>>>> Thread 18 (Thread 0x7f1b0ffff700 (LWP 105178)):
>>>> #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 ()
>>>> from /lib64/libpthread.so.0
>>>> No symbol table info available.
>>>> #1 0x00007f1b4941b050 in PR_WaitCondVar () from
>>>> /lib64/libnspr4.so
>>>> No symbol table info available.
>>>> #2 0x00007f1b4b70efee in connection_wait_for_new_work ()
>>>> No symbol table info available.
>>>> #3 0x00007f1b4b710719 in connection_threadmain ()
>>>> No symbol table info available.
>>>> #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so
>>>> No symbol table info available.
>>>> #5 0x00007f1b48dc1dc5 in start_thread () from
>>>> /lib64/libpthread.so.0
>>>> No symbol table info available.
>>>> #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6
>>>> No symbol table info available.
>>>>
>>>> Thread 17 (Thread 0x7f1b14d28700 (LWP 105177)):
>>>> #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 ()
>>>> from /lib64/libpthread.so.0
>>>> No symbol table info available.
>>>> #1 0x00007f1b4941b050 in PR_WaitCondVar () from
>>>> /lib64/libnspr4.so
>>>> No symbol table info available.
>>>> #2 0x00007f1b4b70efee in connection_wait_for_new_work ()
>>>> No symbol table info available.
>>>> #3 0x00007f1b4b710719 in connection_threadmain ()
>>>> No symbol table info available.
>>>> #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so
>>>> No symbol table info available.
>>>> #5 0x00007f1b48dc1dc5 in start_thread () from
>>>> /lib64/libpthread.so.0
>>>> No symbol table info available.
>>>> #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6
>>>> No symbol table info available.
>>>>
>>>> Thread 16 (Thread 0x7f1b15529700 (LWP 105176)):
>>>> #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 ()
>>>> from /lib64/libpthread.so.0
>>>> No symbol table info available.
>>>> #1 0x00007f1b4941b050 in PR_WaitCondVar () from
>>>> /lib64/libnspr4.so
>>>> No symbol table info available.
>>>> #2 0x00007f1b4b70efee in connection_wait_for_new_work ()
>>>> No symbol table info available.
>>>> #3 0x00007f1b4b710719 in connection_threadmain ()
>>>> No symbol table info available.
>>>> #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so
>>>> No symbol table info available.
>>>> #5 0x00007f1b48dc1dc5 in start_thread () from
>>>> /lib64/libpthread.so.0
>>>> No symbol table info available.
>>>> #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6
>>>> No symbol table info available.
>>>>
>>>> Thread 15 (Thread 0x7f1b15d2a700 (LWP 105175)):
>>>> #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 ()
>>>> from /lib64/libpthread.so.0
>>>> No symbol table info available.
>>>> #1 0x00007f1b4941b050 in PR_WaitCondVar () from
>>>> /lib64/libnspr4.so
>>>> No symbol table info available.
>>>> #2 0x00007f1b4b70efee in connection_wait_for_new_work ()
>>>> No symbol table info available.
>>>> #3 0x00007f1b4b710719 in connection_threadmain ()
>>>> No symbol table info available.
>>>> #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so
>>>> No symbol table info available.
>>>> #5 0x00007f1b48dc1dc5 in start_thread () from
>>>> /lib64/libpthread.so.0
>>>> No symbol table info available.
>>>> #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6
>>>> No symbol table info available.
>>>>
>>>> Thread 14 (Thread 0x7f1b1652b700 (LWP 105174)):
>>>> #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 ()
>>>> from /lib64/libpthread.so.0
>>>> No symbol table info available.
>>>> #1 0x00007f1b4941b050 in PR_WaitCondVar () from
>>>> /lib64/libnspr4.so
>>>> No symbol table info available.
>>>> #2 0x00007f1b4b70efee in connection_wait_for_new_work ()
>>>> No symbol table info available.
>>>> #3 0x00007f1b4b710719 in connection_threadmain ()
>>>> No symbol table info available.
>>>> #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so
>>>> No symbol table info available.
>>>> #5 0x00007f1b48dc1dc5 in start_thread () from
>>>> /lib64/libpthread.so.0
>>>> No symbol table info available.
>>>> #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6
>>>> No symbol table info available.
>>>>
>>>> Thread 13 (Thread 0x7f1b16d2c700 (LWP 105173)):
>>>> #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 ()
>>>> from /lib64/libpthread.so.0
>>>> No symbol table info available.
>>>> #1 0x00007f1b433212f3 in __db_hybrid_mutex_suspend () from
>>>> /lib64/libdb-5.3.so <http://libdb-5.3.so>
>>>> No symbol table info available.
>>>> #2 0x00007f1b43320640 in __db_tas_mutex_lock () from
>>>> /lib64/libdb-5.3.so <http://libdb-5.3.so>
>>>> No symbol table info available.
>>>> #3 0x00007f1b433cad3a in __lock_get_internal () from
>>>> /lib64/libdb-5.3.so <http://libdb-5.3.so>
>>>> No symbol table info available.
>>>> #4 0x00007f1b433cb820 in __lock_get () from
>>>> /lib64/libdb-5.3.so <http://libdb-5.3.so>
>>>> No symbol table info available.
>>>> #5 0x00007f1b433f7142 in __db_lget () from
>>>> /lib64/libdb-5.3.so <http://libdb-5.3.so>
>>>> No symbol table info available.
>>>> #6 0x00007f1b4333e605 in __bam_search () from
>>>> /lib64/libdb-5.3.so <http://libdb-5.3.so>
>>>> No symbol table info available.
>>>> #7 0x00007f1b43329256 in __bamc_search () from
>>>> /lib64/libdb-5.3.so <http://libdb-5.3.so>
>>>> No symbol table info available.
>>>> #8 0x00007f1b4332ad0f in __bamc_get () from
>>>> /lib64/libdb-5.3.so <http://libdb-5.3.so>
>>>> No symbol table info available.
>>>> #9 0x00007f1b433e3ca6 in __dbc_iget () from
>>>> /lib64/libdb-5.3.so <http://libdb-5.3.so>
>>>> No symbol table info available.
>>>> #10 0x00007f1b433f2b02 in __dbc_get_pp () from
>>>> /lib64/libdb-5.3.so <http://libdb-5.3.so>
>>>> No symbol table info available.
>>>> #11 0x00007f1b3e8ce680 in idl_new_fetch () from
>>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so
>>>> No symbol table info available.
>>>> #12 0x00007f1b3e8dcb26 in index_read_ext_allids () from
>>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so
>>>> No symbol table info available.
>>>> #13 0x00007f1b3e8c7284 in keys2idl () from
>>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so
>>>> No symbol table info available.
>>>> #14 0x00007f1b3e8c79e3 in ava_candidates.isra.0 () from
>>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so
>>>> No symbol table info available.
>>>> #15 0x00007f1b3e8c7fd2 in filter_candidates_ext () from
>>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so
>>>> No symbol table info available.
>>>> #16 0x00007f1b3e8c9096 in list_candidates () from
>>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so
>>>> No symbol table info available.
>>>> #17 0x00007f1b3e8c7f40 in filter_candidates_ext () from
>>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so
>>>> No symbol table info available.
>>>> #18 0x00007f1b3e8c9096 in list_candidates () from
>>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so
>>>> No symbol table info available.
>>>> #19 0x00007f1b3e8c7f40 in filter_candidates_ext () from
>>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so
>>>> No symbol table info available.
>>>> #20 0x00007f1b3e8c9096 in list_candidates () from
>>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so
>>>> No symbol table info available.
>>>> #21 0x00007f1b3e8c7f40 in filter_candidates_ext () from
>>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so
>>>> No symbol table info available.
>>>> #22 0x00007f1b3e90380c in subtree_candidates () from
>>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so
>>>> No symbol table info available.
>>>> #23 0x00007f1b3e904e7f in ldbm_back_search () from
>>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so
>>>> No symbol table info available.
>>>> #24 0x00007f1b4b2321d6 in op_shared_search () from
>>>> /usr/lib64/dirsrv/libslapd.so.0
>>>> No symbol table info available.
>>>> #25 0x00007f1b4b72333e in do_search ()
>>>> No symbol table info available.
>>>> #26 0x00007f1b4b7113c3 in connection_threadmain ()
>>>> No symbol table info available.
>>>> #27 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so
>>>> No symbol table info available.
>>>> #28 0x00007f1b48dc1dc5 in start_thread () from
>>>> /lib64/libpthread.so.0
>>>> No symbol table info available.
>>>> #29 0x00007f1b48aef28d in clone () from /lib64/libc.so.6
>>>> No symbol table info available.
>>>>
>>>> Thread 12 (Thread 0x7f1b1752d700 (LWP 105172)):
>>>> #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 ()
>>>> from /lib64/libpthread.so.0
>>>> No symbol table info available.
>>>> #1 0x00007f1b4941b050 in PR_WaitCondVar () from
>>>> /lib64/libnspr4.so
>>>> No symbol table info available.
>>>> #2 0x00007f1b4b70efee in connection_wait_for_new_work ()
>>>> No symbol table info available.
>>>> #3 0x00007f1b4b710719 in connection_threadmain ()
>>>> No symbol table info available.
>>>> #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so
>>>> No symbol table info available.
>>>> #5 0x00007f1b48dc1dc5 in start_thread () from
>>>> /lib64/libpthread.so.0
>>>> No symbol table info available.
>>>> #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6
>>>> No symbol table info available.
>>>>
>>>> Thread 11 (Thread 0x7f1b17d2e700 (LWP 105171)):
>>>> #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 ()
>>>> from /lib64/libpthread.so.0
>>>> No symbol table info available.
>>>> #1 0x00007f1b4941b050 in PR_WaitCondVar () from
>>>> /lib64/libnspr4.so
>>>> No symbol table info available.
>>>> #2 0x00007f1b4b70efee in connection_wait_for_new_work ()
>>>> No symbol table info available.
>>>> #3 0x00007f1b4b710719 in connection_threadmain ()
>>>> No symbol table info available.
>>>> #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so
>>>> No symbol table info available.
>>>> #5 0x00007f1b48dc1dc5 in start_thread () from
>>>> /lib64/libpthread.so.0
>>>> No symbol table info available.
>>>> #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6
>>>> No symbol table info available.
>>>>
>>>> Thread 10 (Thread 0x7f1b2509d700 (LWP 105168)):
>>>> #0 0x00007f1b48dc5a82 in
>>>> pthread_cond_timedwait@@GLIBC_2.3.2
>>>> <mailto:pthread_cond_timedwait@@GLIBC_2.3.2> () from
>>>> /lib64/libpthread.so.0
>>>> No symbol table info available.
>>>> #1 0x00007f1b4941ab07 in pt_TimedWait () from
>>>> /lib64/libnspr4.so
>>>> No symbol table info available.
>>>> #2 0x00007f1b4941afce in PR_WaitCondVar () from
>>>> /lib64/libnspr4.so
>>>> No symbol table info available.
>>>> #3 0x00007f1b4b2061e8 in eq_loop () from
>>>> /usr/lib64/dirsrv/libslapd.so.0
>>>> No symbol table info available.
>>>> #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so
>>>> No symbol table info available.
>>>> #5 0x00007f1b48dc1dc5 in start_thread () from
>>>> /lib64/libpthread.so.0
>>>> No symbol table info available.
>>>> #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6
>>>> No symbol table info available.
>>>>
>>>> Thread 9 (Thread 0x7f1b2589e700 (LWP 105167)):
>>>> #0 0x00007f1b48dc5a82 in
>>>> pthread_cond_timedwait@@GLIBC_2.3.2
>>>> <mailto:pthread_cond_timedwait@@GLIBC_2.3.2> () from
>>>> /lib64/libpthread.so.0
>>>> No symbol table info available.
>>>> #1 0x00007f1b4941ab07 in pt_TimedWait () from
>>>> /lib64/libnspr4.so
>>>> No symbol table info available.
>>>> #2 0x00007f1b4941afce in PR_WaitCondVar () from
>>>> /lib64/libnspr4.so
>>>> No symbol table info available.
>>>> #3 0x00007f1b4b719d03 in housecleaning ()
>>>> No symbol table info available.
>>>> #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so
>>>> No symbol table info available.
>>>> #5 0x00007f1b48dc1dc5 in start_thread () from
>>>> /lib64/libpthread.so.0
>>>> No symbol table info available.
>>>> #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6
>>>> No symbol table info available.
>>>>
>>>> Thread 8 (Thread 0x7f1b2609f700 (LWP 105166)):
>>>> #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 ()
>>>> from /lib64/libpthread.so.0
>>>> No symbol table info available.
>>>> #1 0x00007f1b4941b050 in PR_WaitCondVar () from
>>>> /lib64/libnspr4.so
>>>> No symbol table info available.
>>>> #2 0x00007f1b4b260198 in slapi_wait_condvar () from
>>>> /usr/lib64/dirsrv/libslapd.so.0
>>>> No symbol table info available.
>>>> #3 0x00007f1b3cf5deed in roles_cache_wait_on_change ()
>>>> from /usr/lib64/dirsrv/plugins/libroles-plugin.so
>>>> No symbol table info available.
>>>> #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so
>>>> No symbol table info available.
>>>> #5 0x00007f1b48dc1dc5 in start_thread () from
>>>> /lib64/libpthread.so.0
>>>> No symbol table info available.
>>>> #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6
>>>> No symbol table info available.
>>>>
>>>> Thread 7 (Thread 0x7f1b268a0700 (LWP 105165)):
>>>> #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 ()
>>>> from /lib64/libpthread.so.0
>>>> No symbol table info available.
>>>> #1 0x00007f1b4941b050 in PR_WaitCondVar () from
>>>> /lib64/libnspr4.so
>>>> No symbol table info available.
>>>> #2 0x00007f1b4b260198 in slapi_wait_condvar () from
>>>> /usr/lib64/dirsrv/libslapd.so.0
>>>> No symbol table info available.
>>>> #3 0x00007f1b3cf5deed in roles_cache_wait_on_change ()
>>>> from /usr/lib64/dirsrv/plugins/libroles-plugin.so
>>>> No symbol table info available.
>>>> #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so
>>>> No symbol table info available.
>>>> #5 0x00007f1b48dc1dc5 in start_thread () from
>>>> /lib64/libpthread.so.0
>>>> No symbol table info available.
>>>> #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6
>>>> No symbol table info available.
>>>>
>>>> Thread 6 (Thread 0x7f1b270a1700 (LWP 105164)):
>>>> #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 ()
>>>> from /lib64/libpthread.so.0
>>>> No symbol table info available.
>>>> #1 0x00007f1b4941b050 in PR_WaitCondVar () from
>>>> /lib64/libnspr4.so
>>>> No symbol table info available.
>>>> #2 0x00007f1b4b260198 in slapi_wait_condvar () from
>>>> /usr/lib64/dirsrv/libslapd.so.0
>>>> No symbol table info available.
>>>> #3 0x00007f1b3cf5deed in roles_cache_wait_on_change ()
>>>> from /usr/lib64/dirsrv/plugins/libroles-plugin.so
>>>> No symbol table info available.
>>>> #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so
>>>> No symbol table info available.
>>>> #5 0x00007f1b48dc1dc5 in start_thread () from
>>>> /lib64/libpthread.so.0
>>>> No symbol table info available.
>>>> #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6
>>>> No symbol table info available.
>>>>
>>>> Thread 5 (Thread 0x7f1b278a2700 (LWP 105163)):
>>>> #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 ()
>>>> from /lib64/libpthread.so.0
>>>> No symbol table info available.
>>>> #1 0x00007f1b4941b050 in PR_WaitCondVar () from
>>>> /lib64/libnspr4.so
>>>> No symbol table info available.
>>>> #2 0x00007f1b4b260198 in slapi_wait_condvar () from
>>>> /usr/lib64/dirsrv/libslapd.so.0
>>>> No symbol table info available.
>>>> #3 0x00007f1b415f262e in cos_cache_wait_on_change () from
>>>> /usr/lib64/dirsrv/plugins/libcos-plugin.so
>>>> No symbol table info available.
>>>> #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so
>>>> No symbol table info available.
>>>> #5 0x00007f1b48dc1dc5 in start_thread () from
>>>> /lib64/libpthread.so.0
>>>> No symbol table info available.
>>>> #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6
>>>> No symbol table info available.
>>>>
>>>> Thread 4 (Thread 0x7f1b282ac700 (LWP 105162)):
>>>> #0 0x00007f1b48ae69b3 in select () from /lib64/libc.so.6
>>>> No symbol table info available.
>>>> #1 0x00007f1b4b2710e9 in DS_Sleep () from
>>>> /usr/lib64/dirsrv/libslapd.so.0
>>>> No symbol table info available.
>>>> #2 0x00007f1b3e8ba707 in perf_threadmain () from
>>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so
>>>> No symbol table info available.
>>>> #3 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so
>>>> No symbol table info available.
>>>> #4 0x00007f1b48dc1dc5 in start_thread () from
>>>> /lib64/libpthread.so.0
>>>> No symbol table info available.
>>>> #5 0x00007f1b48aef28d in clone () from /lib64/libc.so.6
>>>> No symbol table info available.
>>>>
>>>> Thread 3 (Thread 0x7f1b28aad700 (LWP 105161)):
>>>> #0 0x00007f1b48ae69b3 in select () from /lib64/libc.so.6
>>>> No symbol table info available.
>>>> #1 0x00007f1b4b2710e9 in DS_Sleep () from
>>>> /usr/lib64/dirsrv/libslapd.so.0
>>>> No symbol table info available.
>>>> #2 0x00007f1b3e8bfb7f in trickle_threadmain () from
>>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so
>>>> No symbol table info available.
>>>> #3 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so
>>>> No symbol table info available.
>>>> #4 0x00007f1b48dc1dc5 in start_thread () from
>>>> /lib64/libpthread.so.0
>>>> No symbol table info available.
>>>> #5 0x00007f1b48aef28d in clone () from /lib64/libc.so.6
>>>> No symbol table info available.
>>>>
>>>> Thread 2 (Thread 0x7f1b292ae700 (LWP 105160)):
>>>> #0 0x00007f1b48ae69b3 in select () from /lib64/libc.so.6
>>>> No symbol table info available.
>>>> #1 0x00007f1b4344dbad in __os_yield () from
>>>> /lib64/libdb-5.3.so <http://libdb-5.3.so>
>>>> No symbol table info available.
>>>> #2 0x00007f1b434492b3 in __memp_sync_int () from
>>>> /lib64/libdb-5.3.so <http://libdb-5.3.so>
>>>> No symbol table info available.
>>>> #3 0x00007f1b43459752 in __txn_checkpoint () from
>>>> /lib64/libdb-5.3.so <http://libdb-5.3.so>
>>>> No symbol table info available.
>>>> #4 0x00007f1b43459b74 in __txn_checkpoint_pp () from
>>>> /lib64/libdb-5.3.so <http://libdb-5.3.so>
>>>> No symbol table info available.
>>>> #5 0x00007f1b3e8c3a87 in checkpoint_threadmain () from
>>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so
>>>> No symbol table info available.
>>>> #6 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so
>>>> No symbol table info available.
>>>> #7 0x00007f1b48dc1dc5 in start_thread () from
>>>> /lib64/libpthread.so.0
>>>> No symbol table info available.
>>>> #8 0x00007f1b48aef28d in clone () from /lib64/libc.so.6
>>>> No symbol table info available.
>>>>
>>>> Thread 1 (Thread 0x7f1b29aaf700 (LWP 105159)):
>>>> #0 0x00007f1b48ae69b3 in select () from /lib64/libc.so.6
>>>> No symbol table info available.
>>>> #1 0x00007f1b4b2710e9 in DS_Sleep () from
>>>> /usr/lib64/dirsrv/libslapd.so.0
>>>> No symbol table info available.
>>>> #2 0x00007f1b3e8bf907 in deadlock_threadmain () from
>>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so
>>>> No symbol table info available.
>>>> #3 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so
>>>> No symbol table info available.
>>>> #4 0x00007f1b48dc1dc5 in start_thread () from
>>>> /lib64/libpthread.so.0
>>>> No symbol table info available.
>>>> #5 0x00007f1b48aef28d in clone () from /lib64/libc.so.6
>>>> No symbol table info available.
>>>>
>>>>
>>>>
>>>> Thanks,
>>>>
>>>> Rakesh
>>>>
>>>>
>>>> On Mon, Sep 5, 2016 at 4:48 PM, thierry bordaz
>>>> <tbordaz at redhat.com> wrote:
>>>>
>>>>
>>>>
>>>> On 09/05/2016 12:05 PM, Rakesh Rajasekharan wrote:
>>>>> Hi Thierry,
>>>>>
>>>>>
>>>>> I was getting the hang issue while running
>>>>> ipa-client-install simultaneously on few clients..
>>>>> However, today, I am not able to replicate that.
>>>>>
>>>>> I could not get a gdb . But i will try getting that
>>>>> the next time I face this issue.
>>>>>
>>>>> The CPU does not stay high.. it just momentarily
>>>>> touches a high value and then drops down to around 2-7%
>>>>>
>>>>> One question I have is , is it ok to set it
>>>>> nsslapd-threadnumber to a very high value .
>>>>> I have around 4000 clients and with
>>>>> nsslapd-maxthreadsperconn set to 5..So, can I set
>>>>> nsslapd-threadnumber to around 25000.
>>>>
>>>> Hello,
>>>>
>>>> I know some users running in production with several
>>>> hunderds of threads (>600) and this without problem.
>>>>
>>>> I do not recall having suggested to increase that
>>>> number and for what reason.
>>>> Usually 30 workers is a good enough value. It can
>>>> create bootleneck if for some reason each operation is
>>>> very long to satisfy and exhaust the number of workers.
>>>> You can monitor the work queue:
>>>>
>>>> ldapsearch -D "cn=directory manager" -w xxx -LLL
>>>> -b "cn=monitor" -s base opsinitiated opscompleted
>>>>
>>>>
>>>> If opscompleted-opsinitiated remains close to
>>>> threadnumber, then yes it would be valuable to increase it.
>>>>
>>>> The computation #client * #async_op_per_client sound an
>>>> overkill. Even if all clients send at the exact same
>>>> time all their requests, it is very likely that some
>>>> common resource (db page, log, allocator...) will
>>>> serialize them. If you monitor a need to increase the
>>>> work, you would for example set it to 50, then monitor,
>>>> then set it to 100, then monitor... until you find a
>>>> good enough value.
>>>> Note the increasing the #thread, increases the memory
>>>> footprint that will reduce the efficiency of file
>>>> system cache and can increase the response time.
>>>>
>>>>
>>>> best regards
>>>> thierry
>>>>
>>>>>
>>>>> Thanks
>>>>>
>>>>> On Mon, Sep 5, 2016 at 1:03 PM, thierry bordaz
>>>>> <tbordaz at redhat.com> wrote:
>>>>>
>>>>>
>>>>> Hi Rakesh,
>>>>>
>>>>> Were you able to get a pstack or full stack with
>>>>> gdb
>>>>> (http://www.port389.org/docs/389ds/FAQ/faq.html#debugging-crashes)
>>>>> when the server hangs ?
>>>>>
>>>>> If it happens with 500 threads as well as with 30,
>>>>> using 30 threads is a better choice to debug this
>>>>> issue.
>>>>> I will try to reproduce using 150 parallel 'ipa
>>>>> user-find p-testipa' commands
>>>>>
>>>>> Something I am unsure is if the CPU consumption
>>>>> stays high (you mentioned 340% CPU usage) as long
>>>>> as the hang happens or if after a suddent shot up
>>>>> to 340% (that marks the beginning of the hang) it
>>>>> drops and stay hanging ?
>>>>>
>>>>> thanks
>>>>> thierry
>>>>>
>>>>> On 09/04/2016 08:40 PM, Rakesh Rajasekharan wrote:
>>>>>> starce on the slapd process actually had this in
>>>>>> the output..
>>>>>> FUTEX_WAIT_PRIVATE
>>>>>>
>>>>>> and checking for the number of threads slapd
>>>>>> had.. there were 5015 threads
>>>>>>
>>>>>> ps -efL|grep slapd|wc -l
>>>>>> 5015
>>>>>>
>>>>>> strace on most of the threads gave this output
>>>>>>
>>>>>> strace -p 67411
>>>>>> Process 67411 attached
>>>>>> futex(0x7f3f0226b41c, FUTEX_WAIT_PRIVATE, 1,
>>>>>> NULL) = -1 EAGAIN (Resource temporarily unavailable)
>>>>>> futex(0x7f3f0226b41c, FUTEX_WAIT_PRIVATE, 2,
>>>>>> NULL^CProcess 67411 detached
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> On Sun, Sep 4, 2016 at 5:34 PM, Rakesh
>>>>>> Rajasekharan <rakesh.rajasekharan at gmail.com> wrote:
>>>>>>
>>>>>> I have again got the issue of IPA hanging..
>>>>>> The issue came up when i tried to run
>>>>>> ipa-client-isntall on 142 clients simultaneously
>>>>>>
>>>>>>
>>>>>> None of the IPA commands are responding, and
>>>>>> I see this error
>>>>>>
>>>>>> ipa user-find p-testipa
>>>>>> ipa: ERROR: Insufficient access: SASL(-1):
>>>>>> generic failure: GSSAPI Error: Unspecified
>>>>>> GSS failure. Minor code may provide more
>>>>>> information (KDC returned error string:
>>>>>> PROCESS_TGS)
>>>>>>
>>>>>> KRB5_TRACE=/dev/stdout kinit admin
>>>>>> [41178] 1472984115.233214: Getting initial
>>>>>> credentials for admin at XYZ.COM
>>>>>> [41178] 1472984115.235257: Sending request
>>>>>> (167 bytes) to XYZ.COM <http://XYZ.COM>
>>>>>> [41178] 1472984115.235419: Initiating TCP
>>>>>> connection to stream 10.1.3.36:88
>>>>>> <http://10.1.3.36:88>
>>>>>> [41178] 1472984115.235685: Sending TCP
>>>>>> request to stream 10.1.3.36:88
>>>>>> <http://10.1.3.36:88>
>>>>>> [41178] 1472984120.238914: Received answer
>>>>>> (174 bytes) from stream 10.1.3.36:88
>>>>>> <http://10.1.3.36:88>
>>>>>> [41178] 1472984120.238925: Terminating TCP
>>>>>> connection to stream 10.1.3.36:88
>>>>>> <http://10.1.3.36:88>
>>>>>> [41178] 1472984120.238993: Response was from
>>>>>> master KDC
>>>>>> [41
>>>>>>
>>>>>>
>>>>>> Running an ldapsearch to see the db.. does
>>>>>> not give any results and just hangs there
>>>>>>
>>>>>> ldapsearch -x -D 'cn=Directory Manager' -W -s
>>>>>> one -b 'cn=kerberos,dc=xyz,dc=com'
>>>>>> Enter LDAP Password:
>>>>>>
>>>>>> even an ldapsearch -x does not respond
>>>>>> At this point, am sure that slapd is the one
>>>>>> causing issues
>>>>>>
>>>>>> Running an strace against the hung slapd
>>>>>> itself seems to get stuck does not proceed
>>>>>> after saying "attaching to process"
>>>>>>
>>>>>> From some others posts I read Thierry
>>>>>> suggesting to increase the
>>>>>> nsslapd-threadnumber value
>>>>>>
>>>>>> It was set to 30, I think that might be too low.
>>>>>>
>> ...
>>
>> [Message clipped]
>>
>>
>>
>
--
Red Hat GmbH, http://www.de.redhat.com/, Registered seat: Grasbrunn,
Commercial register: Amtsgericht Muenchen, HRB 153243,
Managing Directors: Charles Cachera, Michael Cunningham, Michael O'Neill, Eric Shander
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/freeipa-users/attachments/20160906/273890dd/attachment.htm>
More information about the Freeipa-users
mailing list