<div dir="ltr"><div><div><div><div><div><div>Hi, </div>With script provided by Rich, I looked at my db cache usage and it does not look to be high enough # under each db are the list of selected indexes specified with INDEXLIST dbname count free free% size changelog:ent 5 2076333 99.0 4163.8 changelog:dn 15976 9303557 88.7 74.0 userroot:ent 4763 175321074 83.6 7221.1 userroot:dn 4763 9995056 95.3 103.0 ipaca:ent 425 6337866 60.4 9759.8 ipaca:dn 425 10451081 99.7 81.6 </div>Right now I set it to 200MB. So, I am planning to bring it down to 100MB </div>Also, <code>nsslapd-db-checkpoint-interval </code>will change it to 80 or 90 from default of 60 </div>I will check this out and the impact it has on the hang issues and get back </div>Thanks </div>Rakesh </div><div class="gmail_extra"> <div class="gmail_quote">On Tue, Sep 6, 2016 at 4:13 PM, Ludwig Krispenz <<a href="mailto:lkrispen@redhat.com" target="_blank">lkrispen@redhat.com</a>> wrote: <blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"> <div bgcolor="#FFFFFF" text="#000000"> Hi, I think you are running into this bug: <a href="https://bugzilla.redhat.com/show_bug.cgi?id=1349779" target="_blank">https://bugzilla.redhat.com/show_bug.cgi?id=1349779</a> it was the outcome of an analysis started in this one: <a href="https://bugzilla.redhat.com/show_bug.cgi?id=1347800" target="_blank">https://bugzilla.redhat.com/show_bug.cgi?id=1347800</a> the problem is that it looks like the checkpoint thread is holding a mutex and otehr threads are blocked on it. It looks like an issue in BerkeleyDB although there is no explanation from the BDB code why this could happen, we also had a support case with Oracle to investigate this, with no solution. Oracle at the end was providing a diagnostic patch, which we did not test since the customer with the original problem had worked around this by changing their configuration. I don't know the status of this patch and if we can make it available, probably depends on your support status, you need to contact support for this. About the workarounds: you can try to change timing for the database update and checkpointing eg by reducing the dbcache size, changing the checkpoint interval. In the customer case they identified one substring index, which was frequently updated but not really required - and removing this index helped. Regards, Ludwig <div>On 09/06/2016 11:31 AM, thierry bordaz wrote: </div> <blockquote type="cite"> Hi, The following thread is hanging most of the others threads <blockquote><tt>Thread 20 (Thread 0x7f68007f8700 (LWP 4873)): </tt> <tt>#0 0x00007f683d0506d5 in <a href="mailto:pthread_cond_wait@@GLIBC_2.3.2" target="_blank">pthread_cond_wait@@GLIBC_2.3.2</a> () from /lib64/libpthread.so.0</tt> <tt>#1 0x00007f68375ac2f3 in __db_hybrid_mutex_suspend () from /lib64/<a href="http://libdb-5.3.so" target="_blank">libdb-5.3.so</a></tt> <tt>#2 0x00007f68375ab640 in __db_tas_mutex_lock () from /lib64/<a href="http://libdb-5.3.so" target="_blank">libdb-5.3.so</a></tt> <tt>#3 0x00007f68376c5e1b in __memp_fget () from /lib64/<a href="http://libdb-5.3.so" target="_blank">libdb-5.3.so</a></tt> <tt>#4 0x00007f68375c9211 in __bam_search () from /lib64/<a href="http://libdb-5.3.so" target="_blank">libdb-5.3.so</a></tt> <tt>#5 0x00007f68375b4256 in __bamc_search () from /lib64/<a href="http://libdb-5.3.so" target="_blank">libdb-5.3.so</a></tt> <tt>#6 0x00007f68375b86c4 in __bamc_put () from /lib64/<a href="http://libdb-5.3.so" target="_blank">libdb-5.3.so</a></tt> <tt>#7 0x00007f683766f6d5 in __dbc_iput () from /lib64/<a href="http://libdb-5.3.so" target="_blank">libdb-5.3.so</a></tt> <tt>#8 0x00007f683766aa0e in __db_put () from /lib64/<a href="http://libdb-5.3.so" target="_blank">libdb-5.3.so</a></tt> <tt>#9 0x00007f683767ffa4 in __db_put_pp () from /lib64/<a href="http://libdb-5.3.so" target="_blank">libdb-5.3.so</a></tt> <tt>#10 0x00007f6832b5a4cf in idl_new_insert_key () from /usr/lib64/dirsrv/plugins/libback-ldbm.so</tt> <tt>#11 0x00007f6832b695b5 in addordel_values_sv.isra.3 () from /usr/lib64/dirsrv/plugins/libback-ldbm.so</tt> <tt>#12 0x00007f6832b6a25a in index_addordel_values_ext_sv () from /usr/lib64/dirsrv/plugins/libback-ldbm.so</tt> <tt>#13 0x00007f6832b6a694 in index_addordel_values_sv () from /usr/lib64/dirsrv/plugins/libback-ldbm.so</tt> <tt>#14 0x00007f6832b6a75a in index_addordel_entry () from /usr/lib64/dirsrv/plugins/libback-ldbm.so</tt> <tt>#15 0x00007f6832b6d5fa in ldbm_back_add () from /usr/lib64/dirsrv/plugins/libback-ldbm.so</tt> <tt>#16 0x00007f683f46fcca in op_shared_add () from /usr/lib64/dirsrv/libslapd.so.0</tt> <tt>#17 0x00007f683f471020 in do_add () from /usr/lib64/dirsrv/libslapd.so.0</tt> <tt>#18 0x00007f683f99c303 in connection_threadmain ()</tt> <tt>#19 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so</tt> <tt>#20 0x00007f683d04cdc5 in start_thread () from /lib64/libpthread.so.0</tt> <tt>#21 0x00007f683cd7a28d in clone () from /lib64/libc.so.6</tt> </blockquote> This thread is likely not hang on 'db_tas_mutex_lock' as the db_lock does not show it waiting for something. Unfortunately it is not shown in the output of top -H, did you get the complete output ? Ludwig mentioned a known issue that DS is not scaling well on NUMA arch, do you know if the machine is NUMA ? A possibility is that this thread get severely slowed down by a NUMA remote lock and showing DS has hung. This is why it would be interesting to know if it consums CPU. thanks theirry <div>On 09/06/2016 10:59 AM, Rakesh Rajasekharan wrote: </div> <blockquote type="cite"> <div dir="ltr"> <div> <div> <div> <div> <div>Hi Thierry/Ludwig, </div> I again hit the hang issue. This is the pstack of the ns-slapd </div> <div>So was able to this time get the pstack and db locks </div> <div>db locks details Default locking region information: 451 Last allocated locker ID 0x7fffffff Current maximum unused locker ID 9 Number of lock modes 800 Initial number of locks allocated 0 Initial number of lockers allocated 800 Initial number of lock objects allocated 10000 Maximum number of locks possible 10000 Maximum number of lockers possible 10000 Maximum number of lock objects possible 800 Current number of locks allocated 366 Current number of lockers allocated 800 Current number of lock objects allocated 160 Number of lock object partitions 8191 Size of object hash table 126 Number of current locks 1005 Maximum number of locks at any one time 33 Maximum number of locks in any one bucket 3987 Maximum number of locks stolen by for an empty partition 75 Maximum number of locks stolen for any one partition 309 Number of current lockers 313 Maximum number of lockers at any one time 121 Number of current lock objects 676 Maximum number of lock objects at any one time 3 Maximum number of lock objects in any one bucket 69 Maximum number of objects stolen by for an empty partition 5 Maximum number of objects stolen for any one partition 22M Total number of locks requested (22009642) 21M Total number of locks released (21992878) 0 Total number of locks upgraded 95 Total number of locks downgraded 3348 Lock requests not available due to conflicts, for which we waited 12646 Lock requests not available due to conflicts, for which we did not wait 0 Number of deadlocks 0 Lock timeout value 0 Number of locks that have timed out 0 Transaction timeout value 0 Number of transactions that have timed out 2MB 560KB Region size 71719 The number of partition locks that required waiting (0%) 23105 The maximum number of times any partition lock was waited for (0%) 0 The number of object queue operations that required waiting (0%) 163 The number of locker allocations that required waiting (0%) 3172 The number of region locks that required waiting (0%) 5 Maximum hash bucket length =-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-= Lock REGINFO information: Environment Region type 1 Region ID /var/lib/dirsrv/slapd-SPRINKLR-COM/db/__db.001 Region name 0x7f67ef869000 Region address 0x7f67ef8690a0 Region allocation head 0x7f67ef8712b0 Region primary address 0 Region maximum allocation 0 Region allocated Region allocations: 1581345 allocations, 0 failures, 1580794 frees, 6 longest Allocations by power-of-two sizes: 1KB 1581329 2KB 3 4KB 4 8KB 3 16KB 3 32KB 1 64KB 0 128KB 0 256KB 2 512KB 0 1024KB 1 REGION_SHARED Region flags =-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-= Lock region parameters: 2 Lock region region mutex [3172/14M 0% !Own] <wakeups 0/1143> 16381 locker table size 8191 object table size 34128 obj_off 900216 locker_off 0 need_dd =-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-= Lock conflict matrix: =-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-= Locks grouped by lockers: Locker Mode Count Status ----------------- Object --------------- 2 dd=308 locks held 1 write locks 0 pid/thread 4845/140085720139840 flags 10 priority 100 2 READ 1 HELD userRoot/id2entry.db handle 0 3 dd=307 locks held 0 write locks 0 pid/thread 4845/140084829472512 flags 0 priority 100 4 dd=306 locks held 1 write locks 0 pid/thread 4845/140085720139840 flags 10 priority 100 4 READ 1 HELD ipaca/id2entry.db handle 0 5 dd=305 locks held 0 write locks 0 pid/thread 4845/140084837865216 flags 0 priority 100 6 dd=304 locks held 1 write locks 0 pid/thread 4845/140085720139840 flags 10 priority 100 6 READ 1 HELD ipaca/entryrdn.db handle 0 7 dd=303 locks held 0 write locks 0 pid/thread 4845/140084804294400 flags 0 priority 100 8 dd=302 locks held 0 write locks 0 pid/thread 4845/140084720432896 flags 0 priority 100 9 dd=301 locks held 0 write locks 0 pid/thread 4845/140084837865216 flags 0 priority 100 a dd=300 locks held 1 write locks 0 pid/thread 4845/140085720139840 flags 10 priority 100 a READ 1 HELD ipaca/vlv#allcertspkitomcatindex.db handle 0 e dd=299 locks held 1 write locks 0 pid/thread 4845/140085720139840 flags 10 priority 100 e READ 1 HELD ipaca/vlv#allnonrevokedcertspkitomcatindex.db handle 0 10 dd=298 locks held 1 write locks 0 pid/thread 4845/140085720139840 flags 10 priority 100 10 READ 1 HELD ipaca/vlv#allrevokedcertspkitomcatindex.db handle 0 11 dd=297 locks held 1 write locks 0 pid/thread 4845/140085720139840 flags 10 priority 100 11 READ 1 HELD ipaca/vlv#allrevokedcertsnotafterpkitomcatindex.db handle 0 14 dd=296 locks held 1 write locks 0 pid/thread 4845/140085720139840 flags 10 priority 100 14 READ 1 HELD ipaca/vlv#allrevokedorrevokedexpiredcertspkitomcatindex.db handle 0 15 dd=295 locks held 1 write locks 0 pid/thread 4845/140085720139840 flags 10 priority 100 15 READ 1 HELD ipaca/vlv#allvalidcertspkitomcatindex.db handle 0 16 dd=294 locks held 1 write locks 0 pid/thread 4845/140085720139840 flags 10 priority 100 16 READ 1 HELD ipaca/vlv#allvalidcertsnotafterpkitomcatindex.db handle 0 17 dd=293 locks held 1 write locks 0 pid/thread 4845/140085720139840 flags 10 priority 100 17 READ 1 HELD ipaca/vlv#allvalidorrevokedcertspkitomcatindex.db handle 0 18 dd=292 locks held 1 write locks 0 pid/thread 4845/140085720139840 flags 10 priority 100 18 READ 1 HELD ipaca/vlv#caallpkitomcatindex.db handle 0 1d dd=291 locks held 1 write locks 0 pid/thread 4845/140085720139840 flags 10 priority 100 1d READ 1 HELD ipaca/vlv#cacompletepkitomcatindex.db handle 0 1e dd=290 locks held 1 write locks 0 pid/thread 4845/140085720139840 flags 10 priority 100 1e READ 1 HELD ipaca/vlv#cacompleteenrollmentpkitomcatindex.db handle 0 20 dd=289 locks held 1 write locks 0 pid/thread 4845/140085720139840 flags 10 priority 100 20 READ 1 HELD ipaca/vlv#cacompleterevocationpkitomcatindex.db handle 0 21 dd=288 locks held 1 write locks 0 pid/thread 4845/140085720139840 flags 10 priority 100 21 READ 1 HELD ipaca/vlv#caenrollmentpkitomcatindex.db handle 0 2b dd=287 locks held 1 write locks 0 pid/thread 4845/140085720139840 flags 10 priority 100 2b READ 1 HELD ipaca/vlv#carevocationpkitomcatindex.db handle 0 2c dd=286 locks held 1 write locks 0 pid/thread 4845/140085720139840 flags 10 priority 100 2c READ 1 HELD changelog/id2entry.db handle 0 2d dd=285 locks held 0 write locks 0 pid/thread 4845/140085719607040 flags 0 priority 100 2e dd=284 locks held 1 write locks 0 pid/thread 4845/140085720139840 flags 10 priority 100 2e READ 1 HELD changelog/entryusn.db handle 0 2f dd=283 locks held 0 write locks 0 pid/thread 4845/140085720139840 flags 0 priority 100 30 dd=282 locks held 1 write locks 0 pid/thread 4845/140085720139840 flags 10 priority 100 30 READ 1 HELD userRoot/entryusn.db handle 0 31 dd=281 locks held 0 write locks 0 pid/thread 4845/140085720139840 flags 0 priority 100 32 dd=280 locks held 1 write locks 0 pid/thread 4845/140085720139840 flags 10 priority 100 32 READ 1 HELD ipaca/entryusn.db handle 0 33 dd=279 locks held 0 write locks 0 pid/thread 4845/140085720139840 flags 0 priority 100 34 dd=278 locks held 1 write locks 0 pid/thread 4845/140085720139840 flags 10 priority 100 34 READ 1 HELD userRoot/entryrdn.db handle 0 35 dd=277 locks held 0 write locks 0 pid/thread 4845/140084544186112 flags 0 priority 100 36 dd=276 locks held 0 write locks 0 pid/thread 4845/140084636505856 flags 0 priority 100 37 dd=275 locks held 0 write locks 0 pid/thread 4845/140084854650624 flags 0 priority 100 38 dd=274 locks held 1 write locks 0 pid/thread 4845/140085720139840 flags 10 priority 100 38 READ 1 HELD userRoot/objectclass.db handle 0 39 dd=273 locks held 0 write locks 0 pid/thread 4845/140084569364224 flags 0 priority 100 3a dd=272 locks held 1 write locks 0 pid/thread 4845/140085720139840 flags 10 priority 100 3a READ 1 HELD userRoot/ancestorid.db handle 0 3b dd=271 locks held 0 write locks 0 pid/thread 4845/140084854650624 flags 0 priority 100 3c dd=270 locks held 1 write locks 0 pid/thread 4845/140085720139840 flags 10 priority 100 3c READ 1 HELD changelog/entryrdn.db handle 0 3d dd=269 locks held 0 write locks 0 pid/thread 4845/140085719607040 flags 0 priority 100 3e dd=268 locks held 0 write locks 0 pid/thread 4845/140084602935040 flags 0 priority 100 3f dd=267 locks held 0 write locks 0 pid/thread 4845/140085719607040 flags 0 priority 100 40 dd=266 locks held 1 write locks 0 pid/thread 4845/140085720139840 flags 10 priority 100 40 READ 1 HELD changelog/objectclass.db handle 0 41 dd=265 locks held 0 write locks 0 pid/thread 4845/140085719607040 flags 0 priority 100 42 dd=264 locks held 0 write locks 0 pid/thread 4845/140085719607040 flags 0 priority 100 43 dd=263 locks held 0 write locks 0 pid/thread 4845/140084653291264 flags 0 priority 100 44 dd=262 locks held 1 write locks 0 pid/thread 4845/140085720139840 flags 10 priority 100 44 READ 1 HELD ipaca/objectclass.db handle 0 45 dd=261 locks held 0 write locks 0 pid/thread 4845/140084577756928 flags 0 priority 100 46 dd=260 locks held 0 write locks 0 pid/thread 4845/140084720432896 flags 0 priority 100 47 dd=259 locks held 1 write locks 0 pid/thread 4845/140085720139840 flags 10 priority 100 47 READ 1 HELD changelog/aci.db handle 0 48 dd=258 locks held 0 write locks 0 pid/thread 4845/140085720139840 flags 0 priority 100 49 dd=257 locks held 1 write locks 0 pid/thread 4845/140085720139840 flags 10 priority 100 49 READ 1 HELD userRoot/aci.db handle 0 4a dd=256 locks held 0 write locks 0 pid/thread 4845/140085720139840 flags 0 priority 100 4b dd=255 locks held 0 write locks 0 pid/thread 4845/140084535793408 flags 0 priority 100 4c dd=254 locks held 1 write locks 0 pid/thread 4845/140085720139840 flags 10 priority 100 4c READ 1 HELD ipaca/aci.db handle 0 4d dd=253 locks held 0 write locks 0 pid/thread 4845/140085720139840 flags 0 priority 100 4e dd=252 locks held 1 write locks 0 pid/thread 4845/140085720139840 flags 10 priority 100 4e READ 1 HELD userRoot/parentid.db handle 0 4f dd=251 locks held 0 write locks 0 pid/thread 4845/140085720139840 flags 0 priority 100 50 dd=250 locks held 0 write locks 0 pid/thread 4845/140085719607040 flags 0 priority 100 51 dd=249 locks held 1 write locks 0 pid/thread 4845/140085720139840 flags 10 priority 100 51 READ 1 HELD changelog/nsuniqueid.db handle 0 52 dd=248 locks held 1 write locks 0 pid/thread 4845/140085720139840 flags 10 priority 100 52 READ 1 HELD changelog/changenumber.db handle 0 53 dd=247 locks held 0 write locks 0 pid/thread 4845/140084602935040 flags 0 priority 100 54 dd=246 locks held 0 write locks 0 pid/thread 4845/140085719607040 flags 0 priority 100 55 dd=245 locks held 1 write locks 0 pid/thread 4845/140085719607040 flags 10 priority 100 55 READ 1 HELD changelog/targetuniqueid.db handle 0 56 dd=244 locks held 1 write locks 0 pid/thread 4845/140085719607040 flags 10 priority 100 56 READ 1 HELD changelog/parentid.db handle 0 57 dd=243 locks held 1 write locks 0 pid/thread 4845/140085719607040 flags 10 priority 100 57 READ 1 HELD changelog/ancestorid.db handle 0 58 dd=242 locks held 1 write locks 0 pid/thread 4845/140085719607040 flags 10 priority 100 58 READ 1 HELD changelog/numsubordinates.db handle 0 59 dd=241 locks held 0 write locks 0 pid/thread 4845/140085719607040 flags 0 priority 100 5a dd=240 locks held 0 write locks 0 pid/thread 4845/140084544186112 flags 0 priority 100 5b dd=239 locks held 1 write locks 0 pid/thread 4845/140084577756928 flags 10 priority 100 5b READ 1 HELD userRoot/krbPrincipalName.db handle 0 5c dd=238 locks held 0 write locks 0 pid/thread 4845/140084636505856 flags 0 priority 100 5d dd=237 locks held 0 write locks 0 pid/thread 4845/140084695254784 flags 0 priority 100 5e dd=236 locks held 1 write locks 0 pid/thread 4845/140084560971520 flags 10 priority 100 5e READ 1 HELD userRoot/ipakrbprincipalalias.db handle 0 5f dd=235 locks held 0 write locks 0 pid/thread 4845/140084703647488 flags 0 priority 100 60 dd=234 locks held 0 write locks 0 pid/thread 4845/140084653291264 flags 0 priority 100 61 dd=233 locks held 0 write locks 0 pid/thread 4845/140084577756928 flags 0 priority 100 62 dd=232 locks held 0 write locks 0 pid/thread 4845/140084686862080 flags 0 priority 100 63 dd=231 locks held 1 write locks 0 pid/thread 4845/140084611327744 flags 10 priority 100 63 READ 1 HELD changelog/seeAlso.db handle 0 64 dd=230 locks held 0 write locks 0 pid/thread 4845/140084686862080 flags 0 priority 100 65 dd=229 locks held 1 write locks 0 pid/thread 4845/140084611327744 flags 10 priority 100 65 READ 1 HELD userRoot/seeAlso.db handle 0 66 dd=228 locks held 0 write locks 0 pid/thread 4845/140084636505856 flags 0 priority 100 67 dd=227 locks held 1 write locks 0 pid/thread 4845/140084611327744 flags 10 priority 100 67 READ 1 HELD ipaca/seeAlso.db handle 0 68 dd=226 locks held 0 write locks 0 pid/thread 4845/140084686862080 flags 0 priority 100 69 dd=225 locks held 1 write locks 0 pid/thread 4845/140084678469376 flags 10 priority 100 69 READ 1 HELD ipaca/parentid.db handle 0 6a dd=224 locks held 0 write locks 0 pid/thread 4845/140084678469376 flags 0 priority 100 6b dd=223 locks held 0 write locks 0 pid/thread 4845/140084829472512 flags 0 priority 100 6c dd=222 locks held 0 write locks 0 pid/thread 4845/140084653291264 flags 0 priority 100 6d dd=221 locks held 0 write locks 0 pid/thread 4845/140084569364224 flags 0 priority 100 6e dd=220 locks held 0 write locks 0 pid/thread 4845/140084636505856 flags 0 priority 100 6e READ 1 WAIT userRoot/objectclass.db page 18 6f dd=219 locks held 0 write locks 0 pid/thread 4845/140084544186112 flags 0 priority 100 70 dd=218 locks held 0 write locks 0 pid/thread 4845/140084569364224 flags 0 priority 100 71 dd=217 locks held 0 write locks 0 pid/thread 4845/140084619720448 flags 0 priority 100 72 dd=216 locks held 0 write locks 0 pid/thread 4845/140084552578816 flags 0 priority 100 73 dd=215 locks held 0 write locks 0 pid/thread 4845/140084829472512 flags 0 priority 100 74 dd=214 locks held 0 write locks 0 pid/thread 4845/140084628113152 flags 0 priority 100 75 dd=213 locks held 0 write locks 0 pid/thread 4845/140084686862080 flags 0 priority 100 76 dd=212 locks held 1 write locks 0 pid/thread 4845/140084560971520 flags 10 priority 100 76 READ 1 HELD userRoot/ipauniqueid.db handle 0 77 dd=211 locks held 0 write locks 0 pid/thread 4845/140084661683968 flags 0 priority 100 78 dd=210 locks held 0 write locks 0 pid/thread 4845/140084846257920 flags 0 priority 100 79 dd=209 locks held 0 write locks 0 pid/thread 4845/140084628113152 flags 0 priority 100 7a dd=208 locks held 1 write locks 0 pid/thread 4845/140084560971520 flags 10 priority 100 7a READ 1 HELD userRoot/nsuniqueid.db handle 0 7b dd=207 locks held 1 write locks 0 pid/thread 4845/140084560971520 flags 10 priority 100 7b READ 1 HELD userRoot/cn.db handle 0 7c dd=206 locks held 1 write locks 0 pid/thread 4845/140084560971520 flags 10 priority 100 7c READ 1 HELD userRoot/fqdn.db handle 0 7d dd=205 locks held 1 write locks 0 pid/thread 4845/140084560971520 flags 10 priority 100 7d READ 1 HELD userRoot/managedby.db handle 0 7e dd=204 locks held 1 write locks 0 pid/thread 4845/140084560971520 flags 10 priority 100 7e READ 1 HELD userRoot/numsubordinates.db handle 0 7f dd=203 locks held 1 write locks 0 pid/thread 4845/140084678469376 flags 10 priority 100 7f READ 1 HELD userRoot/member.db handle 0 80 dd=202 locks held 0 write locks 0 pid/thread 4845/140084712040192 flags 0 priority 100 81 dd=201 locks held 1 write locks 0 pid/thread 4845/140084678469376 flags 10 priority 100 81 READ 1 HELD userRoot/memberUser.db handle 0 82 dd=200 locks held 0 write locks 0 pid/thread 4845/140084577756928 flags 0 priority 100 83 dd=199 locks held 1 write locks 0 pid/thread 4845/140084678469376 flags 10 priority 100 83 READ 1 HELD userRoot/memberHost.db handle 0 84 dd=198 locks held 0 write locks 0 pid/thread 4845/140084712040192 flags 0 priority 100 85 dd=197 locks held 0 write locks 0 pid/thread 4845/140084577756928 flags 0 priority 100 86 dd=196 locks held 0 write locks 0 pid/thread 4845/140084670076672 flags 0 priority 100 87 dd=195 locks held 0 write locks 0 pid/thread 4845/140084577756928 flags 0 priority 100 88 dd=194 locks held 0 write locks 0 pid/thread 4845/140084846257920 flags 0 priority 100 89 dd=193 locks held 0 write locks 0 pid/thread 4845/140084552578816 flags 0 priority 100 8a dd=192 locks held 0 write locks 0 pid/thread 4845/140084703647488 flags 0 priority 100 8b dd=191 locks held 0 write locks 0 pid/thread 4845/140084544186112 flags 0 priority 100 8c dd=190 locks held 0 write locks 0 pid/thread 4845/140084846257920 flags 0 priority 100 8d dd=189 locks held 0 write locks 0 pid/thread 4845/140084661683968 flags 0 priority 100 8e dd=188 locks held 1 write locks 0 pid/thread 4845/140085067675392 flags 10 priority 100 8e READ 1 HELD userRoot/macAddress.db handle 0 8f dd=187 locks held 0 write locks 0 pid/thread 4845/140085067675392 flags 0 priority 100 90 dd=186 locks held 0 write locks 0 pid/thread 4845/140084703647488 flags 0 priority 100 91 dd=185 locks held 0 write locks 0 pid/thread 4845/140084552578816 flags 0 priority 100 92 dd=184 locks held 0 write locks 0 pid/thread 4845/140084560971520 flags 0 priority 100 92 READ 1 WAIT userRoot/objectclass.db page 9 93 dd=183 locks held 0 write locks 0 pid/thread 4845/140084577756928 flags 0 priority 100 94 dd=182 locks held 0 write locks 0 pid/thread 4845/140084821079808 flags 0 priority 100 94 READ 1 WAIT userRoot/objectclass.db page 18 95 dd=181 locks held 0 write locks 0 pid/thread 4845/140084544186112 flags 0 priority 100 96 dd=180 locks held 0 write locks 0 pid/thread 4845/140084804294400 flags 0 priority 100 97 dd=179 locks held 1 write locks 0 pid/thread 4845/140084712040192 flags 10 priority 100 97 READ 1 HELD userRoot/uid.db handle 0 98 dd=178 locks held 0 write locks 0 pid/thread 4845/140084552578816 flags 0 priority 100 99 dd=177 locks held 0 write locks 0 pid/thread 4845/140084577756928 flags 0 priority 100 9a dd=176 locks held 0 write locks 0 pid/thread 4845/140084602935040 flags 0 priority 100 9b dd=175 locks held 1 write locks 0 pid/thread 4845/140085076068096 flags 10 priority 100 9b READ 1 HELD userRoot/uniquemember.db handle 0 9c dd=174 locks held 1 write locks 0 pid/thread 4845/140085076068096 flags 10 priority 100 9c READ 1 HELD userRoot/owner.db handle 0 9d dd=173 locks held 1 write locks 0 pid/thread 4845/140085076068096 flags 10 priority 100 9d READ 1 HELD userRoot/ipaallowedtarget.db handle 0 9e dd=172 locks held 1 write locks 0 pid/thread 4845/140085076068096 flags 10 priority 100 9e READ 1 HELD userRoot/ipaMemberCertProfile.db handle 0 9f dd=171 locks held 1 write locks 0 pid/thread 4845/140085076068096 flags 10 priority 100 9f READ 1 HELD userRoot/ipaassignedidview.db handle 0 a0 dd=170 locks held 1 write locks 0 pid/thread 4845/140085076068096 flags 10 priority 100 a0 READ 1 HELD userRoot/secretary.db handle 0 a1 dd=169 locks held 1 write locks 0 pid/thread 4845/140085076068096 flags 10 priority 100 a1 READ 1 HELD userRoot/memberdenycmd.db handle 0 a2 dd=168 locks held 1 write locks 0 pid/thread 4845/140085076068096 flags 10 priority 100 a2 READ 1 HELD userRoot/memberallowcmd.db handle 0 a3 dd=167 locks held 1 write locks 0 pid/thread 4845/140085076068096 flags 10 priority 100 a3 READ 1 HELD userRoot/manager.db handle 0 a4 dd=166 locks held 1 write locks 0 pid/thread 4845/140085076068096 flags 10 priority 100 a4 READ 1 HELD userRoot/ipasudorunas.db handle 0 a5 dd=165 locks held 1 write locks 0 pid/thread 4845/140085076068096 flags 10 priority 100 a5 READ 1 HELD userRoot/ipaMemberCa.db handle 0 a6 dd=164 locks held 1 write locks 0 pid/thread 4845/140085076068096 flags 10 priority 100 a6 READ 1 HELD userRoot/ipasudorunasgroup.db handle 0 a7 dd=163 locks held 1 write locks 0 pid/thread 4845/140085076068096 flags 10 priority 100 a7 READ 1 HELD userRoot/sourcehost.db handle 0 a8 dd=162 locks held 1 write locks 0 pid/thread 4845/140085076068096 flags 10 priority 100 a8 READ 1 HELD userRoot/memberservice.db handle 0 a9 dd=161 locks held 1 write locks 0 pid/thread 4845/140085076068096 flags 10 priority 100 a9 READ 1 HELD userRoot/ipatokenradiusconfiglink.db handle 0 aa dd=160 locks held 0 write locks 0 pid/thread 4845/140084703647488 flags 0 priority 100 ab dd=159 locks held 0 write locks 0 pid/thread 4845/140084821079808 flags 0 priority 100 ac dd=158 locks held 0 write locks 0 pid/thread 4845/140084628113152 flags 0 priority 100 ad dd=157 locks held 0 write locks 0 pid/thread 4845/140084829472512 flags 0 priority 100 ae dd=156 locks held 0 write locks 0 pid/thread 4845/140084829472512 flags 0 priority 100 af dd=155 locks held 0 write locks 0 pid/thread 4845/140084854650624 flags 0 priority 100 b0 dd=154 locks held 0 write locks 0 pid/thread 4845/140084854650624 flags 0 priority 100 b1 dd=153 locks held 0 write locks 0 pid/thread 4845/140084837865216 flags 0 priority 100 b2 dd=152 locks held 0 write locks 0 pid/thread 4845/140084804294400 flags 0 priority 100 b3 dd=151 locks held 0 write locks 0 pid/thread 4845/140084812687104 flags 0 priority 100 b4 dd=150 locks held 0 write locks 0 pid/thread 4845/140084804294400 flags 0 priority 100 b5 dd=149 locks held 0 write locks 0 pid/thread 4845/140084837865216 flags 0 priority 100 b6 dd=148 locks held 0 write locks 0 pid/thread 4845/140084569364224 flags 0 priority 100 b7 dd=147 locks held 0 write locks 0 pid/thread 4845/140084829472512 flags 0 priority 100 b8 dd=146 locks held 0 write locks 0 pid/thread 4845/140084586149632 flags 0 priority 100 b9 dd=145 locks held 0 write locks 0 pid/thread 4845/140084653291264 flags 0 priority 100 ba dd=144 locks held 0 write locks 0 pid/thread 4845/140084636505856 flags 0 priority 100 bb dd=143 locks held 0 write locks 0 pid/thread 4845/140084804294400 flags 0 priority 100 bc dd=142 locks held 0 write locks 0 pid/thread 4845/140084804294400 flags 0 priority 100 bd dd=141 locks held 0 write locks 0 pid/thread 4845/140084695254784 flags 0 priority 100 be dd=140 locks held 0 write locks 0 pid/thread 4845/140084670076672 flags 0 priority 100 bf dd=139 locks held 0 write locks 0 pid/thread 4845/140084628113152 flags 0 priority 100 c0 dd=138 locks held 0 write locks 0 pid/thread 4845/140084661683968 flags 0 priority 100 c1 dd=137 locks held 0 write locks 0 pid/thread 4845/140084712040192 flags 0 priority 100 c2 dd=136 locks held 0 write locks 0 pid/thread 4845/140084661683968 flags 0 priority 100 c3 dd=135 locks held 0 write locks 0 pid/thread 4845/140084577756928 flags 0 priority 100 c4 dd=134 locks held 0 write locks 0 pid/thread 4845/140084854650624 flags 0 priority 100 c5 dd=133 locks held 1 write locks 0 pid/thread 4845/140084644898560 flags 10 priority 100 c5 READ 1 HELD userRoot/uidnumber.db handle 0 c6 dd=132 locks held 0 write locks 0 pid/thread 4845/140084594542336 flags 0 priority 100 c8 dd=131 locks held 1 write locks 0 pid/thread 4845/140084678469376 flags 10 priority 100 c8 READ 1 HELD ipaca/certstatus.db handle 0 c9 dd=130 locks held 0 write locks 0 pid/thread 4845/140084720432896 flags 0 priority 100 ca dd=129 locks held 0 write locks 0 pid/thread 4845/140084619720448 flags 0 priority 100 cb dd=128 locks held 0 write locks 0 pid/thread 4845/140084821079808 flags 0 priority 100 cc dd=127 locks held 0 write locks 0 pid/thread 4845/140084594542336 flags 0 priority 100 cd dd=126 locks held 0 write locks 0 pid/thread 4845/140084560971520 flags 0 priority 100 ce dd=125 locks held 1 write locks 0 pid/thread 4845/140084837865216 flags 10 priority 100 ce READ 1 HELD userRoot/gidnumber.db handle 0 cf dd=124 locks held 0 write locks 0 pid/thread 4845/140084720432896 flags 0 priority 100 d0 dd=123 locks held 0 write locks 0 pid/thread 4845/140084569364224 flags 0 priority 100 d1 dd=122 locks held 1 write locks 0 pid/thread 4845/140084678469376 flags 10 priority 100 d1 READ 1 HELD userRoot/nscpEntryDN.db handle 0 d2 dd=121 locks held 0 write locks 0 pid/thread 4845/140084695254784 flags 0 priority 100 d3 dd=120 locks held 0 write locks 0 pid/thread 4845/140084569364224 flags 0 priority 100 d4 dd=119 locks held 0 write locks 0 pid/thread 4845/140084653291264 flags 0 priority 100 d5 dd=118 locks held 0 write locks 0 pid/thread 4845/140084829472512 flags 0 priority 100 d6 dd=117 locks held 0 write locks 0 pid/thread 4845/140084661683968 flags 0 priority 100 d7 dd=116 locks held 0 write locks 0 pid/thread 4845/140084804294400 flags 0 priority 100 d8 dd=115 locks held 1 write locks 0 pid/thread 4845/140084821079808 flags 10 priority 100 d8 READ 1 HELD ipaca/description.db handle 0 d9 dd=114 locks held 0 write locks 0 pid/thread 4845/140084661683968 flags 0 priority 100 da dd=113 locks held 0 write locks 0 pid/thread 4845/140084837865216 flags 0 priority 100 db dd=112 locks held 1 write locks 0 pid/thread 4845/140084678469376 flags 10 priority 100 db READ 1 HELD ipaca/cn.db handle 0 dc dd=111 locks held 0 write locks 0 pid/thread 4845/140084577756928 flags 0 priority 100 dd dd=110 locks held 1 write locks 0 pid/thread 4845/140084678469376 flags 10 priority 100 dd READ 1 HELD ipaca/ancestorid.db handle 0 de dd=109 locks held 0 write locks 0 pid/thread 4845/140084577756928 flags 0 priority 100 df dd=108 locks held 0 write locks 0 pid/thread 4845/140084703647488 flags 0 priority 100 e0 dd=107 locks held 0 write locks 0 pid/thread 4845/140084569364224 flags 0 priority 100 e1 dd=106 locks held 0 write locks 0 pid/thread 4845/140084569364224 flags 0 priority 100 e2 dd=105 locks held 0 write locks 0 pid/thread 4845/140084821079808 flags 0 priority 100 e3 dd=104 locks held 1 write locks 0 pid/thread 4845/140084686862080 flags 10 priority 100 e3 READ 1 HELD ipaca/serialno.db handle 0 e4 dd=103 locks held 0 write locks 0 pid/thread 4845/140084837865216 flags 0 priority 100 e5 dd=102 locks held 1 write locks 0 pid/thread 4845/140084695254784 flags 10 priority 100 e5 READ 1 HELD ipaca/revInfo.db handle 0 e6 dd=101 locks held 1 write locks 0 pid/thread 4845/140084695254784 flags 10 priority 100 e6 READ 1 HELD ipaca/revokedby.db handle 0 e7 dd=100 locks held 1 write locks 0 pid/thread 4845/140084695254784 flags 10 priority 100 e7 READ 1 HELD ipaca/revokedOn.db handle 0 e8 dd=99 locks held 0 write locks 0 pid/thread 4845/140084829472512 flags 0 priority 100 e9 dd=98 locks held 0 write locks 0 pid/thread 4845/140084846257920 flags 0 priority 100 ea dd=97 locks held 1 write locks 0 pid/thread 4845/140084703647488 flags 10 priority 100 ea READ 1 HELD ipaca/nsuniqueid.db handle 0 eb dd=96 locks held 1 write locks 0 pid/thread 4845/140084703647488 flags 10 priority 100 eb READ 1 HELD ipaca/requestid.db handle 0 ec dd=95 locks held 1 write locks 0 pid/thread 4845/140084703647488 flags 10 priority 100 ec READ 1 HELD ipaca/requeststate.db handle 0 ed dd=94 locks held 1 write locks 0 pid/thread 4845/140084703647488 flags 10 priority 100 ed READ 1 HELD ipaca/dateOfCreate.db handle 0 ee dd=93 locks held 1 write locks 0 pid/thread 4845/140084703647488 flags 10 priority 100 ee READ 1 HELD ipaca/requesttype.db handle 0 ef dd=92 locks held 1 write locks 0 pid/thread 4845/140084703647488 flags 10 priority 100 ef READ 1 HELD ipaca/numsubordinates.db handle 0 f0 dd=91 locks held 1 write locks 0 pid/thread 4845/140084611327744 flags 10 priority 100 f0 READ 1 HELD userRoot/userCertificate.db handle 0 f1 dd=90 locks held 0 write locks 0 pid/thread 4845/140084821079808 flags 0 priority 100 f2 dd=89 locks held 0 write locks 0 pid/thread 4845/140084569364224 flags 0 priority 100 f3 dd=88 locks held 0 write locks 0 pid/thread 4845/140084602935040 flags 0 priority 100 f4 dd=87 locks held 0 write locks 0 pid/thread 4845/140084720432896 flags 0 priority 100 f5 dd=86 locks held 0 write locks 0 pid/thread 4845/140084628113152 flags 0 priority 100 f6 dd=85 locks held 0 write locks 0 pid/thread 4845/140084703647488 flags 0 priority 100 f7 dd=84 locks held 0 write locks 0 pid/thread 4845/140084611327744 flags 0 priority 100 f8 dd=83 locks held 0 write locks 0 pid/thread 4845/140084661683968 flags 0 priority 100 f9 dd=82 locks held 0 write locks 0 pid/thread 4845/140084837865216 flags 0 priority 100 fa dd=81 locks held 0 write locks 0 pid/thread 4845/140084544186112 flags 0 priority 100 fb dd=80 locks held 0 write locks 0 pid/thread 4845/140084703647488 flags 0 priority 100 fc dd=79 locks held 0 write locks 0 pid/thread 4845/140084544186112 flags 0 priority 100 fd dd=78 locks held 0 write locks 0 pid/thread 4845/140084535793408 flags 0 priority 100 fe dd=77 locks held 0 write locks 0 pid/thread 4845/140084586149632 flags 0 priority 100 ff dd=76 locks held 0 write locks 0 pid/thread 4845/140084569364224 flags 0 priority 100 100 dd=75 locks held 0 write locks 0 pid/thread 4845/140084804294400 flags 0 priority 100 101 dd=74 locks held 0 write locks 0 pid/thread 4845/140084703647488 flags 0 priority 100 102 dd=73 locks held 0 write locks 0 pid/thread 4845/140084703647488 flags 0 priority 100 103 dd=72 locks held 0 write locks 0 pid/thread 4845/140084821079808 flags 0 priority 100 104 dd=71 locks held 0 write locks 0 pid/thread 4845/140084695254784 flags 0 priority 100 105 dd=70 locks held 0 write locks 0 pid/thread 4845/140084544186112 flags 0 priority 100 106 dd=69 locks held 0 write locks 0 pid/thread 4845/140084854650624 flags 0 priority 100 107 dd=68 locks held 0 write locks 0 pid/thread 4845/140084628113152 flags 0 priority 100 108 dd=67 locks held 0 write locks 0 pid/thread 4845/140084569364224 flags 0 priority 100 109 dd=66 locks held 0 write locks 0 pid/thread 4845/140084594542336 flags 0 priority 100 10a dd=65 locks held 0 write locks 0 pid/thread 4845/140084661683968 flags 0 priority 100 10b dd=64 locks held 0 write locks 0 pid/thread 4845/140084653291264 flags 0 priority 100 10c dd=63 locks held 0 write locks 0 pid/thread 4845/140084695254784 flags 0 priority 100 10d dd=62 locks held 0 write locks 0 pid/thread 4845/140084720432896 flags 0 priority 100 10e dd=61 locks held 0 write locks 0 pid/thread 4845/140084619720448 flags 0 priority 100 10f dd=60 locks held 0 write locks 0 pid/thread 4845/140084720432896 flags 0 priority 100 110 dd=59 locks held 0 write locks 0 pid/thread 4845/140084569364224 flags 0 priority 100 111 dd=58 locks held 0 write locks 0 pid/thread 4845/140084695254784 flags 0 priority 100 112 dd=57 locks held 0 write locks 0 pid/thread 4845/140084653291264 flags 0 priority 100 113 dd=56 locks held 0 write locks 0 pid/thread 4845/140084712040192 flags 0 priority 100 114 dd=55 locks held 0 write locks 0 pid/thread 4845/140084686862080 flags 0 priority 100 115 dd=54 locks held 0 write locks 0 pid/thread 4845/140084577756928 flags 0 priority 100 115 READ 1 WAIT userRoot/objectclass.db page 9 116 dd=53 locks held 0 write locks 0 pid/thread 4845/140084837865216 flags 0 priority 100 116 READ 1 WAIT userRoot/objectclass.db page 9 117 dd=52 locks held 0 write locks 0 pid/thread 4845/140084577756928 flags 0 priority 100 118 dd=51 locks held 0 write locks 0 pid/thread 4845/140084569364224 flags 0 priority 100 119 dd=50 locks held 0 write locks 0 pid/thread 4845/140084560971520 flags 0 priority 100 11a dd=49 locks held 1 write locks 0 pid/thread 4845/140084695254784 flags 10 priority 100 11a READ 1 HELD ipaca/metaInfo.db handle 0 11b dd=48 locks held 1 write locks 0 pid/thread 4845/140084695254784 flags 10 priority 100 11b READ 1 HELD ipaca/notbefore.db handle 0 11c dd=47 locks held 1 write locks 0 pid/thread 4845/140084695254784 flags 10 priority 100 11c READ 1 HELD ipaca/notafter.db handle 0 11d dd=46 locks held 1 write locks 0 pid/thread 4845/140084695254784 flags 10 priority 100 11d READ 1 HELD ipaca/duration.db handle 0 11e dd=45 locks held 1 write locks 0 pid/thread 4845/140084695254784 flags 10 priority 100 11e READ 1 HELD ipaca/subjectname.db handle 0 11f dd=44 locks held 1 write locks 0 pid/thread 4845/140084695254784 flags 10 priority 100 11f READ 1 HELD ipaca/publicKeyData.db handle 0 120 dd=43 locks held 1 write locks 0 pid/thread 4845/140084695254784 flags 10 priority 100 120 READ 1 HELD ipaca/extension.db handle 0 121 dd=42 locks held 1 write locks 0 pid/thread 4845/140084695254784 flags 10 priority 100 121 READ 1 HELD ipaca/issuedby.db handle 0 122 dd=41 locks held 0 write locks 0 pid/thread 4845/140084712040192 flags 0 priority 100 123 dd=40 locks held 0 write locks 0 pid/thread 4845/140084812687104 flags 0 priority 100 124 dd=39 locks held 0 write locks 0 pid/thread 4845/140084544186112 flags 0 priority 100 125 dd=38 locks held 0 write locks 0 pid/thread 4845/140084686862080 flags 0 priority 100 126 dd=37 locks held 0 write locks 0 pid/thread 4845/140084552578816 flags 0 priority 100 128 dd=36 locks held 0 write locks 0 pid/thread 4845/140084686862080 flags 0 priority 100 12a dd=35 locks held 0 write locks 0 pid/thread 4845/140084720432896 flags 0 priority 100 12b dd=34 locks held 0 write locks 0 pid/thread 4845/140084552578816 flags 0 priority 100 12c dd=33 locks held 0 write locks 0 pid/thread 4845/140084670076672 flags 0 priority 100 12d dd=32 locks held 0 write locks 0 pid/thread 4845/140084577756928 flags 0 priority 100 12e dd=31 locks held 0 write locks 0 pid/thread 4845/140084712040192 flags 0 priority 100 12f dd=30 locks held 0 write locks 0 pid/thread 4845/140084670076672 flags 0 priority 100 130 dd=29 locks held 0 write locks 0 pid/thread 4845/140084577756928 flags 0 priority 100 131 dd=28 locks held 0 write locks 0 pid/thread 4845/140084560971520 flags 0 priority 100 132 dd=27 locks held 0 write locks 0 pid/thread 4845/140084670076672 flags 0 priority 100 133 dd=26 locks held 0 write locks 0 pid/thread 4845/140084602935040 flags 0 priority 100 134 dd=25 locks held 0 write locks 0 pid/thread 4845/140084552578816 flags 0 priority 100 135 dd=24 locks held 0 write locks 0 pid/thread 4845/140084678469376 flags 0 priority 100 136 dd=23 locks held 0 write locks 0 pid/thread 4845/140084544186112 flags 0 priority 100 137 dd=22 locks held 0 write locks 0 pid/thread 4845/140084577756928 flags 0 priority 100 138 dd=21 locks held 0 write locks 0 pid/thread 4845/140084703647488 flags 0 priority 100 139 dd=20 locks held 0 write locks 0 pid/thread 4845/140084678469376 flags 0 priority 100 13b dd=19 locks held 0 write locks 0 pid/thread 4845/140084812687104 flags 0 priority 100 144 dd=18 locks held 0 write locks 0 pid/thread 4845/140084670076672 flags 0 priority 100 156 dd=17 locks held 0 write locks 0 pid/thread 4845/140084594542336 flags 0 priority 100 15a dd=16 locks held 0 write locks 0 pid/thread 4845/140084577756928 flags 0 priority 100 168 dd=15 locks held 0 write locks 0 pid/thread 4845/140084653291264 flags 0 priority 100 16a dd=14 locks held 0 write locks 0 pid/thread 4845/140084829472512 flags 0 priority 100 17d dd=13 locks held 0 write locks 0 pid/thread 4845/140084829472512 flags 0 priority 100 17e dd=12 locks held 0 write locks 0 pid/thread 4845/140084829472512 flags 0 priority 100 17f dd=11 locks held 0 write locks 0 pid/thread 4845/140084586149632 flags 0 priority 100 180 dd=10 locks held 0 write locks 0 pid/thread 4845/140084594542336 flags 0 priority 100 181 dd= 9 locks held 0 write locks 0 pid/thread 4845/140084577756928 flags 0 priority 100 193 dd= 8 locks held 0 write locks 0 pid/thread 4845/140084812687104 flags 0 priority 100 196 dd= 7 locks held 0 write locks 0 pid/thread 4845/140084695254784 flags 0 priority 100 1a1 dd= 6 locks held 0 write locks 0 pid/thread 4845/140084720432896 flags 0 priority 100 1a2 dd= 5 locks held 0 write locks 0 pid/thread 4845/140084577756928 flags 0 priority 100 1a3 dd= 4 locks held 0 write locks 0 pid/thread 4845/140084812687104 flags 0 priority 100 1a4 dd= 3 locks held 0 write locks 0 pid/thread 4845/140084812687104 flags 0 priority 100 1a5 dd= 2 locks held 0 write locks 0 pid/thread 4845/140084577756928 flags 0 priority 100 1a6 dd= 1 locks held 0 write locks 0 pid/thread 4845/140084594542336 flags 0 priority 100 800b3a33 dd= 0 locks held 26 write locks 22 pid/thread 4845/140084661683968 flags 0 priority 100 800b3a33 WRITE 1 HELD userRoot/objectclass.db page 9 800b3a33 WRITE 1 HELD userRoot/objectclass.db page 8 800b3a33 WRITE 1 HELD userRoot/objectclass.db page 18 800b3a33 WRITE 1 HELD userRoot/objectclass.db page 7 800b3a33 WRITE 1 HELD userRoot/objectclass.db page 14 800b3a33 WRITE 1 HELD userRoot/objectclass.db page 11 800b3a33 WRITE 2 HELD userRoot/objectclass.db page 3 800b3a33 WRITE 1 HELD userRoot/cn.db page 150 800b3a33 WRITE 1 HELD userRoot/cn.db page 89 800b3a33 WRITE 1 HELD userRoot/cn.db page 53 800b3a33 WRITE 1 HELD userRoot/cn.db page 131 800b3a33 WRITE 1 HELD userRoot/cn.db page 117 800b3a33 WRITE 1 HELD userRoot/cn.db page 11 800b3a33 WRITE 2 HELD userRoot/cn.db page 141 800b3a33 WRITE 1 HELD userRoot/cn.db page 41 800b3a33 WRITE 1 HELD userRoot/cn.db page 40 800b3a33 WRITE 1 HELD userRoot/cn.db page 143 800b3a33 WRITE 1 HELD userRoot/cn.db page 30 800b3a33 WRITE 2 HELD userRoot/id2entry.db page 8184 800b3a33 WRITE 1 HELD userRoot/id2entry.db page 8183 800b3a33 WRITE 1 HELD userRoot/id2entry.db page 7779 800b3a33 WRITE 1 HELD userRoot/id2entry.db page 0 800b3a33 READ 3 HELD userRoot/entryrdn.db page 3 800b3a33 READ 1 HELD userRoot/entryrdn.db page 36 800b3a33 READ 1 HELD userRoot/entryrdn.db page 4 800b3a33 READ 1 HELD userRoot/nsuniqueid.db page 86 =-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-= Locks grouped by object: Locker Mode Count Status ----------------- Object --------------- a4 READ 1 HELD userRoot/ipasudorunas.db handle 0 9e READ 1 HELD userRoot/ipaMemberCertProfile.db handle 0 10 READ 1 HELD ipaca/vlv#allrevokedcertspkitomcatindex.db handle 0 800b3a33 WRITE 2 HELD userRoot/id2entry.db page 8184 800b3a33 WRITE 1 HELD userRoot/id2entry.db page 8183 800b3a33 WRITE 1 HELD userRoot/id2entry.db page 7779 a1 READ 1 HELD userRoot/memberdenycmd.db handle 0 a READ 1 HELD ipaca/vlv#allcertspkitomcatindex.db handle 0 a7 READ 1 HELD userRoot/sourcehost.db handle 0 57 READ 1 HELD changelog/ancestorid.db handle 0 e5 READ 1 HELD ipaca/revInfo.db handle 0 e6 READ 1 HELD ipaca/revokedby.db handle 0 11 READ 1 HELD ipaca/vlv#allrevokedcertsnotafterpkitomcatindex.db handle 0 4e READ 1 HELD userRoot/parentid.db handle 0 4 READ 1 HELD ipaca/id2entry.db handle 0 58 READ 1 HELD changelog/numsubordinates.db handle 0 67 READ 1 HELD ipaca/seeAlso.db handle 0 2b READ 1 HELD ipaca/vlv#carevocationpkitomcatindex.db handle 0 ec READ 1 HELD ipaca/requeststate.db handle 0 11b READ 1 HELD ipaca/notbefore.db handle 0 7e READ 1 HELD userRoot/numsubordinates.db handle 0 a2 READ 1 HELD userRoot/memberallowcmd.db handle 0 9c READ 1 HELD userRoot/owner.db handle 0 1e READ 1 HELD ipaca/vlv#cacompleteenrollmentpkitomcatindex.db handle 0 11c READ 1 HELD ipaca/notafter.db handle 0 40 READ 1 HELD changelog/objectclass.db handle 0 2e READ 1 HELD changelog/entryusn.db handle 0 16 READ 1 HELD ipaca/vlv#allvalidcertsnotafterpkitomcatindex.db handle 0 15 READ 1 HELD ipaca/vlv#allvalidcertspkitomcatindex.db handle 0 a5 READ 1 HELD userRoot/ipaMemberCa.db handle 0 18 READ 1 HELD ipaca/vlv#caallpkitomcatindex.db handle 0 a0 READ 1 HELD userRoot/secretary.db handle 0 81 READ 1 HELD userRoot/memberUser.db handle 0 9f READ 1 HELD userRoot/ipaassignedidview.db handle 0 a3 READ 1 HELD userRoot/manager.db handle 0 d8 READ 1 HELD ipaca/description.db handle 0 a8 READ 1 HELD userRoot/memberservice.db handle 0 44 READ 1 HELD ipaca/objectclass.db handle 0 ed READ 1 HELD ipaca/dateOfCreate.db handle 0 800b3a33 WRITE 1 HELD userRoot/id2entry.db page 0 2 READ 1 HELD userRoot/id2entry.db handle 0 5b READ 1 HELD userRoot/krbPrincipalName.db handle 0 ee READ 1 HELD ipaca/requesttype.db handle 0 c8 READ 1 HELD ipaca/certstatus.db handle 0 56 READ 1 HELD changelog/parentid.db handle 0 83 READ 1 HELD userRoot/memberHost.db handle 0 5e READ 1 HELD userRoot/ipakrbprincipalalias.db handle 0 121 READ 1 HELD ipaca/issuedby.db handle 0 6 READ 1 HELD ipaca/entryrdn.db handle 0 e7 READ 1 HELD ipaca/revokedOn.db handle 0 4c READ 1 HELD ipaca/aci.db handle 0 11d READ 1 HELD ipaca/duration.db handle 0 14 READ 1 HELD ipaca/vlv#allrevokedorrevokedexpiredcertspkitomcatindex.db handle 0 21 READ 1 HELD ipaca/vlv#caenrollmentpkitomcatindex.db handle 0 55 READ 1 HELD changelog/targetuniqueid.db handle 0 800b3a33 READ 1 HELD userRoot/entryrdn.db page 36 2c READ 1 HELD changelog/id2entry.db handle 0 800b3a33 READ 1 HELD userRoot/entryrdn.db page 4 34 READ 1 HELD userRoot/entryrdn.db handle 0 800b3a33 READ 3 HELD userRoot/entryrdn.db page 3 eb READ 1 HELD ipaca/requestid.db handle 0 49 READ 1 HELD userRoot/aci.db handle 0 76 READ 1 HELD userRoot/ipauniqueid.db handle 0 63 READ 1 HELD changelog/seeAlso.db handle 0 7c READ 1 HELD userRoot/fqdn.db handle 0 db READ 1 HELD ipaca/cn.db handle 0 ef READ 1 HELD ipaca/numsubordinates.db handle 0 32 READ 1 HELD ipaca/entryusn.db handle 0 1d READ 1 HELD ipaca/vlv#cacompletepkitomcatindex.db handle 0 dd READ 1 HELD ipaca/ancestorid.db handle 0 a9 READ 1 HELD userRoot/ipatokenradiusconfiglink.db handle 0 f0 READ 1 HELD userRoot/userCertificate.db handle 0 11a READ 1 HELD ipaca/metaInfo.db handle 0 7d READ 1 HELD userRoot/managedby.db handle 0 ea READ 1 HELD ipaca/nsuniqueid.db handle 0 c5 READ 1 HELD userRoot/uidnumber.db handle 0 d1 READ 1 HELD userRoot/nscpEntryDN.db handle 0 120 READ 1 HELD ipaca/extension.db handle 0 800b3a33 WRITE 1 HELD userRoot/cn.db page 150 69 READ 1 HELD ipaca/parentid.db handle 0 800b3a33 WRITE 1 HELD userRoot/cn.db page 131 800b3a33 WRITE 1 HELD userRoot/cn.db page 143 800b3a33 WRITE 2 HELD userRoot/cn.db page 141 20 READ 1 HELD ipaca/vlv#cacompleterevocationpkitomcatindex.db handle 0 8e READ 1 HELD userRoot/macAddress.db handle 0 e READ 1 HELD ipaca/vlv#allnonrevokedcertspkitomcatindex.db handle 0 17 READ 1 HELD ipaca/vlv#allvalidorrevokedcertspkitomcatindex.db handle 0 800b3a33 WRITE 1 HELD userRoot/cn.db page 30 800b3a33 WRITE 1 HELD userRoot/objectclass.db page 14 800b3a33 WRITE 1 HELD userRoot/objectclass.db page 9 116 READ 1 WAIT userRoot/objectclass.db page 9 115 READ 1 WAIT userRoot/objectclass.db page 9 92 READ 1 WAIT userRoot/objectclass.db page 9 800b3a33 WRITE 1 HELD userRoot/objectclass.db page 8 7b READ 1 HELD userRoot/cn.db handle 0 800b3a33 WRITE 1 HELD userRoot/objectclass.db page 11 800b3a33 WRITE 1 HELD userRoot/objectclass.db page 7 800b3a33 WRITE 1 HELD userRoot/cn.db page 11 38 READ 1 HELD userRoot/objectclass.db handle 0 800b3a33 WRITE 2 HELD userRoot/objectclass.db page 3 800b3a33 WRITE 1 HELD userRoot/cn.db page 53 800b3a33 WRITE 1 HELD userRoot/objectclass.db page 18 6e READ 1 WAIT userRoot/objectclass.db page 18 94 READ 1 WAIT userRoot/objectclass.db page 18 800b3a33 WRITE 1 HELD userRoot/cn.db page 41 800b3a33 WRITE 1 HELD userRoot/cn.db page 40 800b3a33 WRITE 1 HELD userRoot/cn.db page 89 800b3a33 WRITE 1 HELD userRoot/cn.db page 117 30 READ 1 HELD userRoot/entryusn.db handle 0 3c READ 1 HELD changelog/entryrdn.db handle 0 97 READ 1 HELD userRoot/uid.db handle 0 47 READ 1 HELD changelog/aci.db handle 0 e3 READ 1 HELD ipaca/serialno.db handle 0 7a READ 1 HELD userRoot/nsuniqueid.db handle 0 800b3a33 READ 1 HELD userRoot/nsuniqueid.db page 86 51 READ 1 HELD changelog/nsuniqueid.db handle 0 11e READ 1 HELD ipaca/subjectname.db handle 0 7f READ 1 HELD userRoot/member.db handle 0 11f READ 1 HELD ipaca/publicKeyData.db handle 0 9b READ 1 HELD userRoot/uniquemember.db handle 0 52 READ 1 HELD changelog/changenumber.db handle 0 ce READ 1 HELD userRoot/gidnumber.db handle 0 65 READ 1 HELD userRoot/seeAlso.db handle 0 9d READ 1 HELD userRoot/ipaallowedtarget.db handle 0 a6 READ 1 HELD userRoot/ipasudorunasgroup.db handle 0 3a READ 1 HELD userRoot/ancestorid.db handle 0 </div> <div>pstack output </div> <div> root@ipa-master ]# pstack 4845 Thread 44 (Thread 0x7f681dd3a700 (LWP 4847)): #0 0x00007f683cd719b3 in select () from /lib64/libc.so.6 #1 0x00007f683f4fc0e9 in DS_Sleep () from /usr/lib64/dirsrv/libslapd.so.0 #2 0x00007f6832b4a907 in deadlock_threadmain () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #3 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so #4 0x00007f683d04cdc5 in start_thread () from /lib64/libpthread.so.0 #5 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 Thread 43 (Thread 0x7f681d539700 (LWP 4848)): #0 0x00007f683cd719b3 in select () from /lib64/libc.so.6 #1 0x00007f68376d8bad in __os_yield () from /lib64/<a href="http://libdb-5.3.so" target="_blank">libdb-5.3.so</a> #2 0x00007f68376d42b3 in __memp_sync_int () from /lib64/<a href="http://libdb-5.3.so" target="_blank">libdb-5.3.so</a> #3 0x00007f68376e4752 in __txn_checkpoint () from /lib64/<a href="http://libdb-5.3.so" target="_blank">libdb-5.3.so</a> #4 0x00007f68376e4b74 in __txn_checkpoint_pp () from /lib64/<a href="http://libdb-5.3.so" target="_blank">libdb-5.3.so</a> #5 0x00007f6832b4ea87 in checkpoint_threadmain () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #6 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so #7 0x00007f683d04cdc5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 Thread 42 (Thread 0x7f681cd38700 (LWP 4849)): #0 0x00007f683cd719b3 in select () from /lib64/libc.so.6 #1 0x00007f683f4fc0e9 in DS_Sleep () from /usr/lib64/dirsrv/libslapd.so.0 #2 0x00007f6832b4ab7f in trickle_threadmain () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #3 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so #4 0x00007f683d04cdc5 in start_thread () from /lib64/libpthread.so.0 #5 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 Thread 41 (Thread 0x7f681c537700 (LWP 4850)): #0 0x00007f683cd719b3 in select () from /lib64/libc.so.6 #1 0x00007f683f4fc0e9 in DS_Sleep () from /usr/lib64/dirsrv/libslapd.so.0 #2 0x00007f6832b45707 in perf_threadmain () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #3 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so #4 0x00007f683d04cdc5 in start_thread () from /lib64/libpthread.so.0 #5 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 Thread 40 (Thread 0x7f681bb2d700 (LWP 4851)): #0 0x00007f683d0506d5 in <a href="mailto:pthread_cond_wait@@GLIBC_2.3.2" target="_blank">pthread_cond_wait@@GLIBC_2.3.2</a> () from /lib64/libpthread.so.0 #1 0x00007f683d6a6050 in PR_WaitCondVar () from /lib64/libnspr4.so #2 0x00007f683f4eb198 in slapi_wait_condvar () from /usr/lib64/dirsrv/libslapd.so.0 #3 0x00007f683587d62e in cos_cache_wait_on_change () from /usr/lib64/dirsrv/plugins/libcos-plugin.so #4 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so #5 0x00007f683d04cdc5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 Thread 39 (Thread 0x7f681b32c700 (LWP 4852)): #0 0x00007f683d0506d5 in <a href="mailto:pthread_cond_wait@@GLIBC_2.3.2" target="_blank">pthread_cond_wait@@GLIBC_2.3.2</a> () from /lib64/libpthread.so.0 #1 0x00007f683d6a6050 in PR_WaitCondVar () from /lib64/libnspr4.so #2 0x00007f683f4eb198 in slapi_wait_condvar () from /usr/lib64/dirsrv/libslapd.so.0 #3 0x00007f68311e8eed in roles_cache_wait_on_change () from /usr/lib64/dirsrv/plugins/libroles-plugin.so #4 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so #5 0x00007f683d04cdc5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 Thread 38 (Thread 0x7f681ab2b700 (LWP 4853)): #0 0x00007f683d0506d5 in <a href="mailto:pthread_cond_wait@@GLIBC_2.3.2" target="_blank">pthread_cond_wait@@GLIBC_2.3.2</a> () from /lib64/libpthread.so.0 #1 0x00007f683d6a6050 in PR_WaitCondVar () from /lib64/libnspr4.so #2 0x00007f683f4eb198 in slapi_wait_condvar () from /usr/lib64/dirsrv/libslapd.so.0 #3 0x00007f68311e8eed in roles_cache_wait_on_change () from /usr/lib64/dirsrv/plugins/libroles-plugin.so #4 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so #5 0x00007f683d04cdc5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 Thread 37 (Thread 0x7f681a32a700 (LWP 4854)): #0 0x00007f683d0506d5 in <a href="mailto:pthread_cond_wait@@GLIBC_2.3.2" target="_blank">pthread_cond_wait@@GLIBC_2.3.2</a> () from /lib64/libpthread.so.0 #1 0x00007f683d6a6050 in PR_WaitCondVar () from /lib64/libnspr4.so #2 0x00007f683f4eb198 in slapi_wait_condvar () from /usr/lib64/dirsrv/libslapd.so.0 #3 0x00007f68311e8eed in roles_cache_wait_on_change () from /usr/lib64/dirsrv/plugins/libroles-plugin.so #4 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so #5 0x00007f683d04cdc5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 Thread 36 (Thread 0x7f6819b29700 (LWP 4855)): #0 0x00007f683d050a82 in <a href="mailto:pthread_cond_timedwait@@GLIBC_2.3.2" target="_blank">pthread_cond_timedwait@@GLIBC_2.3.2</a> () from /lib64/libpthread.so.0 #1 0x00007f683d6a5b07 in pt_TimedWait () from /lib64/libnspr4.so #2 0x00007f683d6a5fce in PR_WaitCondVar () from /lib64/libnspr4.so #3 0x00007f683f9a4d03 in housecleaning () #4 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so #5 0x00007f683d04cdc5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 Thread 35 (Thread 0x7f6819328700 (LWP 4856)): #0 0x00007f683d050a82 in <a href="mailto:pthread_cond_timedwait@@GLIBC_2.3.2" target="_blank">pthread_cond_timedwait@@GLIBC_2.3.2</a> () from /lib64/libpthread.so.0 #1 0x00007f683d6a5b07 in pt_TimedWait () from /lib64/libnspr4.so #2 0x00007f683d6a5fce in PR_WaitCondVar () from /lib64/libnspr4.so #3 0x00007f683f4911e8 in eq_loop () from /usr/lib64/dirsrv/libslapd.so.0 #4 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so #5 0x00007f683d04cdc5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 Thread 34 (Thread 0x7f680bfff700 (LWP 4859)): #0 0x00007f683d0506d5 in <a href="mailto:pthread_cond_wait@@GLIBC_2.3.2" target="_blank">pthread_cond_wait@@GLIBC_2.3.2</a> () from /lib64/libpthread.so.0 #1 0x00007f683d6a6050 in PR_WaitCondVar () from /lib64/libnspr4.so #2 0x00007f683f999fee in connection_wait_for_new_work () #3 0x00007f683f99b719 in connection_threadmain () #4 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so #5 0x00007f683d04cdc5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 Thread 33 (Thread 0x7f680b7fe700 (LWP 4860)): #0 0x00007f683d0506d5 in <a href="mailto:pthread_cond_wait@@GLIBC_2.3.2" target="_blank">pthread_cond_wait@@GLIBC_2.3.2</a> () from /lib64/libpthread.so.0 #1 0x00007f683d6a6050 in PR_WaitCondVar () from /lib64/libnspr4.so #2 0x00007f683f999fee in connection_wait_for_new_work () #3 0x00007f683f99b719 in connection_threadmain () #4 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so #5 0x00007f683d04cdc5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 Thread 32 (Thread 0x7f680affd700 (LWP 4861)): #0 0x00007f683d0506d5 in <a href="mailto:pthread_cond_wait@@GLIBC_2.3.2" target="_blank">pthread_cond_wait@@GLIBC_2.3.2</a> () from /lib64/libpthread.so.0 #1 0x00007f68375ac2f3 in __db_hybrid_mutex_suspend () from /lib64/<a href="http://libdb-5.3.so" target="_blank">libdb-5.3.so</a> #2 0x00007f68375ab640 in __db_tas_mutex_lock () from /lib64/<a href="http://libdb-5.3.so" target="_blank">libdb-5.3.so</a> #3 0x00007f6837655d3a in __lock_get_internal () from /lib64/<a href="http://libdb-5.3.so" target="_blank">libdb-5.3.so</a> #4 0x00007f6837656820 in __lock_get () from /lib64/<a href="http://libdb-5.3.so" target="_blank">libdb-5.3.so</a> #5 0x00007f6837682142 in __db_lget () from /lib64/<a href="http://libdb-5.3.so" target="_blank">libdb-5.3.so</a> #6 0x00007f68375c9605 in __bam_search () from /lib64/<a href="http://libdb-5.3.so" target="_blank">libdb-5.3.so</a> #7 0x00007f68375b4256 in __bamc_search () from /lib64/<a href="http://libdb-5.3.so" target="_blank">libdb-5.3.so</a> #8 0x00007f68375b5d0f in __bamc_get () from /lib64/<a href="http://libdb-5.3.so" target="_blank">libdb-5.3.so</a> #9 0x00007f683766eca6 in __dbc_iget () from /lib64/<a href="http://libdb-5.3.so" target="_blank">libdb-5.3.so</a> #10 0x00007f683767db02 in __dbc_get_pp () from /lib64/<a href="http://libdb-5.3.so" target="_blank">libdb-5.3.so</a> #11 0x00007f6832b59680 in idl_new_fetch () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #12 0x00007f6832b67b26 in index_read_ext_allids () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #13 0x00007f6832b52284 in keys2idl () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #14 0x00007f6832b529e3 in ava_candidates.isra.0 () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #15 0x00007f6832b52fd2 in filter_candidates_ext () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #16 0x00007f6832b54096 in list_candidates () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #17 0x00007f6832b52f40 in filter_candidates_ext () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #18 0x00007f6832b54096 in list_candidates () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #19 0x00007f6832b52f40 in filter_candidates_ext () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #20 0x00007f6832b54096 in list_candidates () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #21 0x00007f6832b52f40 in filter_candidates_ext () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #22 0x00007f6832b8e80c in subtree_candidates () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #23 0x00007f6832b8fe7f in ldbm_back_search () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #24 0x00007f683f4bd1d6 in op_shared_search () from /usr/lib64/dirsrv/libslapd.so.0 #25 0x00007f683f9ae33e in do_search () #26 0x00007f683f99c3c3 in connection_threadmain () #27 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so #28 0x00007f683d04cdc5 in start_thread () from /lib64/libpthread.so.0 #29 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 Thread 31 (Thread 0x7f680a7fc700 (LWP 4862)): #0 0x00007f683d0506d5 in <a href="mailto:pthread_cond_wait@@GLIBC_2.3.2" target="_blank">pthread_cond_wait@@GLIBC_2.3.2</a> () from /lib64/libpthread.so.0 #1 0x00007f683d6a62c3 in PR_EnterMonitor () from /lib64/libnspr4.so #2 0x00007f6832b4d3e6 in dblayer_txn_begin () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #3 0x00007f6832b89007 in ldbm_back_modify () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #4 0x00007f683f4b6541 in op_shared_modify () from /usr/lib64/dirsrv/libslapd.so.0 #5 0x00007f683f4b7004 in modify_internal_pb () from /usr/lib64/dirsrv/libslapd.so.0 #6 0x00007f683483bbbb in ipalockout_postop () from /usr/lib64/dirsrv/plugins/libipa_lockout.so #7 0x00007f683f4c8848 in plugin_call_func () from /usr/lib64/dirsrv/libslapd.so.0 #8 0x00007f683f4c8b13 in plugin_call_plugins () from /usr/lib64/dirsrv/libslapd.so.0 #9 0x00007f683f9954d8 in do_bind () #10 0x00007f683f99c3fd in connection_threadmain () #11 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so #12 0x00007f683d04cdc5 in start_thread () from /lib64/libpthread.so.0 #13 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 Thread 30 (Thread 0x7f6809ffb700 (LWP 4863)): #0 0x00007f683d0506d5 in <a href="mailto:pthread_cond_wait@@GLIBC_2.3.2" target="_blank">pthread_cond_wait@@GLIBC_2.3.2</a> () from /lib64/libpthread.so.0 #1 0x00007f68375ac2f3 in __db_hybrid_mutex_suspend () from /lib64/<a href="http://libdb-5.3.so" target="_blank">libdb-5.3.so</a> #2 0x00007f68375ab640 in __db_tas_mutex_lock () from /lib64/<a href="http://libdb-5.3.so" target="_blank">libdb-5.3.so</a> #3 0x00007f6837655d3a in __lock_get_internal () from /lib64/<a href="http://libdb-5.3.so" target="_blank">libdb-5.3.so</a> #4 0x00007f6837656820 in __lock_get () from /lib64/<a href="http://libdb-5.3.so" target="_blank">libdb-5.3.so</a> #5 0x00007f6837682142 in __db_lget () from /lib64/<a href="http://libdb-5.3.so" target="_blank">libdb-5.3.so</a> #6 0x00007f68375c9605 in __bam_search () from /lib64/<a href="http://libdb-5.3.so" target="_blank">libdb-5.3.so</a> #7 0x00007f68375b4256 in __bamc_search () from /lib64/<a href="http://libdb-5.3.so" target="_blank">libdb-5.3.so</a> #8 0x00007f68375b5d0f in __bamc_get () from /lib64/<a href="http://libdb-5.3.so" target="_blank">libdb-5.3.so</a> #9 0x00007f683766eca6 in __dbc_iget () from /lib64/<a href="http://libdb-5.3.so" target="_blank">libdb-5.3.so</a> #10 0x00007f683767db02 in __dbc_get_pp () from /lib64/<a href="http://libdb-5.3.so" target="_blank">libdb-5.3.so</a> #11 0x00007f6832b59680 in idl_new_fetch () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #12 0x00007f6832b67b26 in index_read_ext_allids () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #13 0x00007f6832b52284 in keys2idl () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #14 0x00007f6832b529e3 in ava_candidates.isra.0 () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #15 0x00007f6832b52fd2 in filter_candidates_ext () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #16 0x00007f6832b54096 in list_candidates () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #17 0x00007f6832b52f40 in filter_candidates_ext () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #18 0x00007f6832b8e80c in subtree_candidates () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #19 0x00007f6832b8fe7f in ldbm_back_search () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #20 0x00007f683f4bd1d6 in op_shared_search () from /usr/lib64/dirsrv/libslapd.so.0 #21 0x00007f683f4cd94e in search_internal_callback_pb () from /usr/lib64/dirsrv/libslapd.so.0 #22 0x00007f683f4cdbe8 in search_internal_pb () from /usr/lib64/dirsrv/libslapd.so.0 #23 0x00007f683f9ac49b in ids_sasl_canon_user () #24 0x00007f683d46c93b in _sasl_canon_user () from /lib64/libsasl2.so.3 #25 0x00007f683d46cc4c in _sasl_canon_user_lookup () from /lib64/libsasl2.so.3 #26 0x00007f6837374fb9 in gssapi_server_mech_step () from /usr/lib64/sasl2/libgssapiv2.so #27 0x00007f683d478b9b in sasl_server_step () from /lib64/libsasl2.so.3 #28 0x00007f683f9ad83b in ids_sasl_check_bind () #29 0x00007f683f99561a in do_bind () #30 0x00007f683f99c3fd in connection_threadmain () #31 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so #32 0x00007f683d04cdc5 in start_thread () from /lib64/libpthread.so.0 #33 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 Thread 29 (Thread 0x7f68097fa700 (LWP 4864)): #0 0x00007f683d0506d5 in <a href="mailto:pthread_cond_wait@@GLIBC_2.3.2" target="_blank">pthread_cond_wait@@GLIBC_2.3.2</a> () from /lib64/libpthread.so.0 #1 0x00007f683d6a6050 in PR_WaitCondVar () from /lib64/libnspr4.so #2 0x00007f683f999fee in connection_wait_for_new_work () #3 0x00007f683f99b719 in connection_threadmain () #4 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so #5 0x00007f683d04cdc5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 Thread 28 (Thread 0x7f6808ff9700 (LWP 4865)): #0 0x00007f683d0506d5 in <a href="mailto:pthread_cond_wait@@GLIBC_2.3.2" target="_blank">pthread_cond_wait@@GLIBC_2.3.2</a> () from /lib64/libpthread.so.0 #1 0x00007f683d6a6050 in PR_WaitCondVar () from /lib64/libnspr4.so #2 0x00007f683f999fee in connection_wait_for_new_work () #3 0x00007f683f99b719 in connection_threadmain () #4 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so #5 0x00007f683d04cdc5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 Thread 27 (Thread 0x7f6803fff700 (LWP 4866)): #0 0x00007f683d0506d5 in <a href="mailto:pthread_cond_wait@@GLIBC_2.3.2" target="_blank">pthread_cond_wait@@GLIBC_2.3.2</a> () from /lib64/libpthread.so.0 #1 0x00007f683d6a6050 in PR_WaitCondVar () from /lib64/libnspr4.so #2 0x00007f683f999fee in connection_wait_for_new_work () #3 0x00007f683f99b719 in connection_threadmain () #4 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so #5 0x00007f683d04cdc5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 Thread 26 (Thread 0x7f68037fe700 (LWP 4867)): #0 0x00007f683d0506d5 in <a href="mailto:pthread_cond_wait@@GLIBC_2.3.2" target="_blank">pthread_cond_wait@@GLIBC_2.3.2</a> () from /lib64/libpthread.so.0 #1 0x00007f683d6a6050 in PR_WaitCondVar () from /lib64/libnspr4.so #2 0x00007f683f999fee in connection_wait_for_new_work () #3 0x00007f683f99b719 in connection_threadmain () #4 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so #5 0x00007f683d04cdc5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 Thread 25 (Thread 0x7f6802ffd700 (LWP 4868)): #0 0x00007f683d0506d5 in <a href="mailto:pthread_cond_wait@@GLIBC_2.3.2" target="_blank">pthread_cond_wait@@GLIBC_2.3.2</a> () from /lib64/libpthread.so.0 #1 0x00007f683d6a6050 in PR_WaitCondVar () from /lib64/libnspr4.so #2 0x00007f683f999fee in connection_wait_for_new_work () #3 0x00007f683f99b719 in connection_threadmain () #4 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so #5 0x00007f683d04cdc5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 Thread 24 (Thread 0x7f68027fc700 (LWP 4869)): #0 0x00007f683d0506d5 in <a href="mailto:pthread_cond_wait@@GLIBC_2.3.2" target="_blank">pthread_cond_wait@@GLIBC_2.3.2</a> () from /lib64/libpthread.so.0 #1 0x00007f68375ac2f3 in __db_hybrid_mutex_suspend () from /lib64/<a href="http://libdb-5.3.so" target="_blank">libdb-5.3.so</a> #2 0x00007f68375abb37 in __db_tas_mutex_readlock () from /lib64/<a href="http://libdb-5.3.so" target="_blank">libdb-5.3.so</a> #3 0x00007f68376c5d35 in __memp_fget () from /lib64/<a href="http://libdb-5.3.so" target="_blank">libdb-5.3.so</a> #4 0x00007f683766f333 in __dbc_iget () from /lib64/<a href="http://libdb-5.3.so" target="_blank">libdb-5.3.so</a> #5 0x00007f683767db02 in __dbc_get_pp () from /lib64/<a href="http://libdb-5.3.so" target="_blank">libdb-5.3.so</a> #6 0x00007f6832b59822 in idl_new_fetch () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #7 0x00007f6832b67b26 in index_read_ext_allids () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #8 0x00007f6832b52284 in keys2idl () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #9 0x00007f6832b529e3 in ava_candidates.isra.0 () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #10 0x00007f6832b52fd2 in filter_candidates_ext () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #11 0x00007f6832b54096 in list_candidates () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #12 0x00007f6832b52f40 in filter_candidates_ext () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #13 0x00007f6832b54096 in list_candidates () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #14 0x00007f6832b52f40 in filter_candidates_ext () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #15 0x00007f6832b54096 in list_candidates () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #16 0x00007f6832b52f40 in filter_candidates_ext () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #17 0x00007f6832b8e80c in subtree_candidates () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #18 0x00007f6832b8fe7f in ldbm_back_search () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #19 0x00007f683f4bd1d6 in op_shared_search () from /usr/lib64/dirsrv/libslapd.so.0 #20 0x00007f683f9ae33e in do_search () #21 0x00007f683f99c3c3 in connection_threadmain () #22 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so #23 0x00007f683d04cdc5 in start_thread () from /lib64/libpthread.so.0 #24 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 Thread 23 (Thread 0x7f6801ffb700 (LWP 4870)): #0 0x00007f683d0506d5 in <a href="mailto:pthread_cond_wait@@GLIBC_2.3.2" target="_blank">pthread_cond_wait@@GLIBC_2.3.2</a> () from /lib64/libpthread.so.0 #1 0x00007f683d6a6050 in PR_WaitCondVar () from /lib64/libnspr4.so #2 0x00007f683f999fee in connection_wait_for_new_work () #3 0x00007f683f99b719 in connection_threadmain () #4 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so #5 0x00007f683d04cdc5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 Thread 22 (Thread 0x7f68017fa700 (LWP 4871)): #0 0x00007f683d0506d5 in <a href="mailto:pthread_cond_wait@@GLIBC_2.3.2" target="_blank">pthread_cond_wait@@GLIBC_2.3.2</a> () from /lib64/libpthread.so.0 #1 0x00007f683d6a6050 in PR_WaitCondVar () from /lib64/libnspr4.so #2 0x00007f683f999fee in connection_wait_for_new_work () #3 0x00007f683f99b719 in connection_threadmain () #4 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so #5 0x00007f683d04cdc5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 Thread 21 (Thread 0x7f6800ff9700 (LWP 4872)): #0 0x00007f683d0506d5 in <a href="mailto:pthread_cond_wait@@GLIBC_2.3.2" target="_blank">pthread_cond_wait@@GLIBC_2.3.2</a> () from /lib64/libpthread.so.0 #1 0x00007f683d6a6050 in PR_WaitCondVar () from /lib64/libnspr4.so #2 0x00007f683f999fee in connection_wait_for_new_work () #3 0x00007f683f99b719 in connection_threadmain () #4 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so #5 0x00007f683d04cdc5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 Thread 20 (Thread 0x7f68007f8700 (LWP 4873)): #0 0x00007f683d0506d5 in <a href="mailto:pthread_cond_wait@@GLIBC_2.3.2" target="_blank">pthread_cond_wait@@GLIBC_2.3.2</a> () from /lib64/libpthread.so.0 #1 0x00007f68375ac2f3 in __db_hybrid_mutex_suspend () from /lib64/<a href="http://libdb-5.3.so" target="_blank">libdb-5.3.so</a> #2 0x00007f68375ab640 in __db_tas_mutex_lock () from /lib64/<a href="http://libdb-5.3.so" target="_blank">libdb-5.3.so</a> #3 0x00007f68376c5e1b in __memp_fget () from /lib64/<a href="http://libdb-5.3.so" target="_blank">libdb-5.3.so</a> #4 0x00007f68375c9211 in __bam_search () from /lib64/<a href="http://libdb-5.3.so" target="_blank">libdb-5.3.so</a> #5 0x00007f68375b4256 in __bamc_search () from /lib64/<a href="http://libdb-5.3.so" target="_blank">libdb-5.3.so</a> #6 0x00007f68375b86c4 in __bamc_put () from /lib64/<a href="http://libdb-5.3.so" target="_blank">libdb-5.3.so</a> #7 0x00007f683766f6d5 in __dbc_iput () from /lib64/<a href="http://libdb-5.3.so" target="_blank">libdb-5.3.so</a> #8 0x00007f683766aa0e in __db_put () from /lib64/<a href="http://libdb-5.3.so" target="_blank">libdb-5.3.so</a> #9 0x00007f683767ffa4 in __db_put_pp () from /lib64/<a href="http://libdb-5.3.so" target="_blank">libdb-5.3.so</a> #10 0x00007f6832b5a4cf in idl_new_insert_key () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #11 0x00007f6832b695b5 in addordel_values_sv.isra.3 () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #12 0x00007f6832b6a25a in index_addordel_values_ext_sv () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #13 0x00007f6832b6a694 in index_addordel_values_sv () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #14 0x00007f6832b6a75a in index_addordel_entry () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #15 0x00007f6832b6d5fa in ldbm_back_add () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #16 0x00007f683f46fcca in op_shared_add () from /usr/lib64/dirsrv/libslapd.so.0 #17 0x00007f683f471020 in do_add () from /usr/lib64/dirsrv/libslapd.so.0 #18 0x00007f683f99c303 in connection_threadmain () #19 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so #20 0x00007f683d04cdc5 in start_thread () from /lib64/libpthread.so.0 #21 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 Thread 19 (Thread 0x7f67ffff7700 (LWP 4874)): #0 0x00007f683d0506d5 in <a href="mailto:pthread_cond_wait@@GLIBC_2.3.2" target="_blank">pthread_cond_wait@@GLIBC_2.3.2</a> () from /lib64/libpthread.so.0 #1 0x00007f683d6a62c3 in PR_EnterMonitor () from /lib64/libnspr4.so #2 0x00007f6832b4d3e6 in dblayer_txn_begin () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #3 0x00007f6832b89007 in ldbm_back_modify () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #4 0x00007f683f4b6541 in op_shared_modify () from /usr/lib64/dirsrv/libslapd.so.0 #5 0x00007f683f4b7004 in modify_internal_pb () from /usr/lib64/dirsrv/libslapd.so.0 #6 0x00007f6832dd3b60 in ipapwd_apply_mods () from /usr/lib64/dirsrv/plugins/libipa_pwd_extop.so #7 0x00007f6832dd78e0 in store_new_keys () from /usr/lib64/dirsrv/plugins/libipa_pwd_extop.so #8 0x00007f6832dd9ad0 in ipapwd_extop () from /usr/lib64/dirsrv/plugins/libipa_pwd_extop.so #9 0x00007f683f4c6ba2 in plugin_call_exop_plugins () from /usr/lib64/dirsrv/libslapd.so.0 #10 0x00007f683f9a2efe in do_extended () #11 0x00007f683f99c41a in connection_threadmain () #12 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so #13 0x00007f683d04cdc5 in start_thread () from /lib64/libpthread.so.0 #14 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 Thread 18 (Thread 0x7f67ff7f6700 (LWP 4875)): #0 0x00007f683d0506d5 in <a href="mailto:pthread_cond_wait@@GLIBC_2.3.2" target="_blank">pthread_cond_wait@@GLIBC_2.3.2</a> () from /lib64/libpthread.so.0 #1 0x00007f683d6a6050 in PR_WaitCondVar () from /lib64/libnspr4.so #2 0x00007f683f999fee in connection_wait_for_new_work () #3 0x00007f683f99b719 in connection_threadmain () #4 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so #5 0x00007f683d04cdc5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 Thread 17 (Thread 0x7f67feff5700 (LWP 4876)): #0 0x00007f683d0506d5 in <a href="mailto:pthread_cond_wait@@GLIBC_2.3.2" target="_blank">pthread_cond_wait@@GLIBC_2.3.2</a> () from /lib64/libpthread.so.0 #1 0x00007f68375ac2f3 in __db_hybrid_mutex_suspend () from /lib64/<a href="http://libdb-5.3.so" target="_blank">libdb-5.3.so</a> #2 0x00007f68375ab640 in __db_tas_mutex_lock () from /lib64/<a href="http://libdb-5.3.so" target="_blank">libdb-5.3.so</a> #3 0x00007f6837655d3a in __lock_get_internal () from /lib64/<a href="http://libdb-5.3.so" target="_blank">libdb-5.3.so</a> #4 0x00007f6837656820 in __lock_get () from /lib64/<a href="http://libdb-5.3.so" target="_blank">libdb-5.3.so</a> #5 0x00007f6837682142 in __db_lget () from /lib64/<a href="http://libdb-5.3.so" target="_blank">libdb-5.3.so</a> #6 0x00007f68375c9605 in __bam_search () from /lib64/<a href="http://libdb-5.3.so" target="_blank">libdb-5.3.so</a> #7 0x00007f68375b4256 in __bamc_search () from /lib64/<a href="http://libdb-5.3.so" target="_blank">libdb-5.3.so</a> #8 0x00007f68375b5d0f in __bamc_get () from /lib64/<a href="http://libdb-5.3.so" target="_blank">libdb-5.3.so</a> #9 0x00007f683766eca6 in __dbc_iget () from /lib64/<a href="http://libdb-5.3.so" target="_blank">libdb-5.3.so</a> #10 0x00007f683767db02 in __dbc_get_pp () from /lib64/<a href="http://libdb-5.3.so" target="_blank">libdb-5.3.so</a> #11 0x00007f6832b59680 in idl_new_fetch () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #12 0x00007f6832b67b26 in index_read_ext_allids () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #13 0x00007f6832b52284 in keys2idl () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #14 0x00007f6832b529e3 in ava_candidates.isra.0 () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #15 0x00007f6832b52fd2 in filter_candidates_ext () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #16 0x00007f6832b54096 in list_candidates () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #17 0x00007f6832b52f40 in filter_candidates_ext () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #18 0x00007f6832b8e80c in subtree_candidates () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #19 0x00007f6832b8fe7f in ldbm_back_search () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #20 0x00007f683f4bd1d6 in op_shared_search () from /usr/lib64/dirsrv/libslapd.so.0 #21 0x00007f683f4cd94e in search_internal_callback_pb () from /usr/lib64/dirsrv/libslapd.so.0 #22 0x00007f683f4cdbe8 in search_internal_pb () from /usr/lib64/dirsrv/libslapd.so.0 #23 0x00007f6832dd75dc in get_entry_by_principal () from /usr/lib64/dirsrv/plugins/libipa_pwd_extop.so #24 0x00007f6832dd8eb1 in ipapwd_extop () from /usr/lib64/dirsrv/plugins/libipa_pwd_extop.so #25 0x00007f683f4c6ba2 in plugin_call_exop_plugins () from /usr/lib64/dirsrv/libslapd.so.0 #26 0x00007f683f9a2efe in do_extended () #27 0x00007f683f99c41a in connection_threadmain () #28 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so #29 0x00007f683d04cdc5 in start_thread () from /lib64/libpthread.so.0 #30 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 Thread 16 (Thread 0x7f67fe7f4700 (LWP 4877)): #0 0x00007f683d0506d5 in <a href="mailto:pthread_cond_wait@@GLIBC_2.3.2" target="_blank">pthread_cond_wait@@GLIBC_2.3.2</a> () from /lib64/libpthread.so.0 #1 0x00007f683d6a6050 in PR_WaitCondVar () from /lib64/libnspr4.so #2 0x00007f683f999fee in connection_wait_for_new_work () #3 0x00007f683f99b719 in connection_threadmain () #4 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so #5 0x00007f683d04cdc5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 Thread 15 (Thread 0x7f67fdff3700 (LWP 4878)): #0 0x00007f683d0506d5 in <a href="mailto:pthread_cond_wait@@GLIBC_2.3.2" target="_blank">pthread_cond_wait@@GLIBC_2.3.2</a> () from /lib64/libpthread.so.0 #1 0x00007f683d6a6050 in PR_WaitCondVar () from /lib64/libnspr4.so #2 0x00007f683f999fee in connection_wait_for_new_work () #3 0x00007f683f99b719 in connection_threadmain () #4 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so #5 0x00007f683d04cdc5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 Thread 14 (Thread 0x7f67fd7f2700 (LWP 4879)): #0 0x00007f683d0506d5 in <a href="mailto:pthread_cond_wait@@GLIBC_2.3.2" target="_blank">pthread_cond_wait@@GLIBC_2.3.2</a> () from /lib64/libpthread.so.0 #1 0x00007f683d6a6050 in PR_WaitCondVar () from /lib64/libnspr4.so #2 0x00007f683f999fee in connection_wait_for_new_work () #3 0x00007f683f99b719 in connection_threadmain () #4 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so #5 0x00007f683d04cdc5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 Thread 13 (Thread 0x7f67fcff1700 (LWP 4880)): #0 0x00007f683d0506d5 in <a href="mailto:pthread_cond_wait@@GLIBC_2.3.2" target="_blank">pthread_cond_wait@@GLIBC_2.3.2</a> () from /lib64/libpthread.so.0 #1 0x00007f683d6a6050 in PR_WaitCondVar () from /lib64/libnspr4.so #2 0x00007f683f999fee in connection_wait_for_new_work () #3 0x00007f683f99b719 in connection_threadmain () #4 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so #5 0x00007f683d04cdc5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 Thread 12 (Thread 0x7f67fc7f0700 (LWP 4881)): #0 0x00007f683d0506d5 in <a href="mailto:pthread_cond_wait@@GLIBC_2.3.2" target="_blank">pthread_cond_wait@@GLIBC_2.3.2</a> () from /lib64/libpthread.so.0 #1 0x00007f683d6a6050 in PR_WaitCondVar () from /lib64/libnspr4.so #2 0x00007f683f999fee in connection_wait_for_new_work () #3 0x00007f683f99b719 in connection_threadmain () #4 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so #5 0x00007f683d04cdc5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 Thread 11 (Thread 0x7f67fbfef700 (LWP 4882)): #0 0x00007f683d0506d5 in <a href="mailto:pthread_cond_wait@@GLIBC_2.3.2" target="_blank">pthread_cond_wait@@GLIBC_2.3.2</a> () from /lib64/libpthread.so.0 #1 0x00007f683d6a6050 in PR_WaitCondVar () from /lib64/libnspr4.so #2 0x00007f683f999fee in connection_wait_for_new_work () #3 0x00007f683f99b719 in connection_threadmain () #4 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so #5 0x00007f683d04cdc5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 Thread 10 (Thread 0x7f67fb7ee700 (LWP 4883)): #0 0x00007f683d0506d5 in <a href="mailto:pthread_cond_wait@@GLIBC_2.3.2" target="_blank">pthread_cond_wait@@GLIBC_2.3.2</a> () from /lib64/libpthread.so.0 #1 0x00007f68375ac2f3 in __db_hybrid_mutex_suspend () from /lib64/<a href="http://libdb-5.3.so" target="_blank">libdb-5.3.so</a> #2 0x00007f68375ab640 in __db_tas_mutex_lock () from /lib64/<a href="http://libdb-5.3.so" target="_blank">libdb-5.3.so</a> #3 0x00007f6837655d3a in __lock_get_internal () from /lib64/<a href="http://libdb-5.3.so" target="_blank">libdb-5.3.so</a> #4 0x00007f6837656820 in __lock_get () from /lib64/<a href="http://libdb-5.3.so" target="_blank">libdb-5.3.so</a> #5 0x00007f6837682142 in __db_lget () from /lib64/<a href="http://libdb-5.3.so" target="_blank">libdb-5.3.so</a> #6 0x00007f68375c9605 in __bam_search () from /lib64/<a href="http://libdb-5.3.so" target="_blank">libdb-5.3.so</a> #7 0x00007f68375b4256 in __bamc_search () from /lib64/<a href="http://libdb-5.3.so" target="_blank">libdb-5.3.so</a> #8 0x00007f68375b5d0f in __bamc_get () from /lib64/<a href="http://libdb-5.3.so" target="_blank">libdb-5.3.so</a> #9 0x00007f683766eca6 in __dbc_iget () from /lib64/<a href="http://libdb-5.3.so" target="_blank">libdb-5.3.so</a> #10 0x00007f683767db02 in __dbc_get_pp () from /lib64/<a href="http://libdb-5.3.so" target="_blank">libdb-5.3.so</a> #11 0x00007f6832b59680 in idl_new_fetch () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #12 0x00007f6832b67b26 in index_read_ext_allids () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #13 0x00007f6832b52284 in keys2idl () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #14 0x00007f6832b529e3 in ava_candidates.isra.0 () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #15 0x00007f6832b52fd2 in filter_candidates_ext () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #16 0x00007f6832b54096 in list_candidates () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #17 0x00007f6832b52f40 in filter_candidates_ext () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #18 0x00007f6832b54096 in list_candidates () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #19 0x00007f6832b52f40 in filter_candidates_ext () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #20 0x00007f6832b54096 in list_candidates () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #21 0x00007f6832b52f40 in filter_candidates_ext () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #22 0x00007f6832b8e80c in subtree_candidates () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #23 0x00007f6832b8fe7f in ldbm_back_search () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #24 0x00007f683f4bd1d6 in op_shared_search () from /usr/lib64/dirsrv/libslapd.so.0 #25 0x00007f683f9ae33e in do_search () #26 0x00007f683f99c3c3 in connection_threadmain () #27 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so #28 0x00007f683d04cdc5 in start_thread () from /lib64/libpthread.so.0 #29 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 Thread 9 (Thread 0x7f67fafed700 (LWP 4884)): #0 0x00007f683d0506d5 in <a href="mailto:pthread_cond_wait@@GLIBC_2.3.2" target="_blank">pthread_cond_wait@@GLIBC_2.3.2</a> () from /lib64/libpthread.so.0 #1 0x00007f683d6a6050 in PR_WaitCondVar () from /lib64/libnspr4.so #2 0x00007f683f999fee in connection_wait_for_new_work () #3 0x00007f683f99b719 in connection_threadmain () #4 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so #5 0x00007f683d04cdc5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 Thread 8 (Thread 0x7f67fa7ec700 (LWP 4885)): #0 0x00007f683d0506d5 in <a href="mailto:pthread_cond_wait@@GLIBC_2.3.2" target="_blank">pthread_cond_wait@@GLIBC_2.3.2</a> () from /lib64/libpthread.so.0 #1 0x00007f68375ac2f3 in __db_hybrid_mutex_suspend () from /lib64/<a href="http://libdb-5.3.so" target="_blank">libdb-5.3.so</a> #2 0x00007f68375ab640 in __db_tas_mutex_lock () from /lib64/<a href="http://libdb-5.3.so" target="_blank">libdb-5.3.so</a> #3 0x00007f6837655d3a in __lock_get_internal () from /lib64/<a href="http://libdb-5.3.so" target="_blank">libdb-5.3.so</a> #4 0x00007f6837656820 in __lock_get () from /lib64/<a href="http://libdb-5.3.so" target="_blank">libdb-5.3.so</a> #5 0x00007f6837682142 in __db_lget () from /lib64/<a href="http://libdb-5.3.so" target="_blank">libdb-5.3.so</a> #6 0x00007f68375c9605 in __bam_search () from /lib64/<a href="http://libdb-5.3.so" target="_blank">libdb-5.3.so</a> #7 0x00007f68375b4256 in __bamc_search () from /lib64/<a href="http://libdb-5.3.so" target="_blank">libdb-5.3.so</a> #8 0x00007f68375b5d0f in __bamc_get () from /lib64/<a href="http://libdb-5.3.so" target="_blank">libdb-5.3.so</a> #9 0x00007f683766eca6 in __dbc_iget () from /lib64/<a href="http://libdb-5.3.so" target="_blank">libdb-5.3.so</a> #10 0x00007f683767db02 in __dbc_get_pp () from /lib64/<a href="http://libdb-5.3.so" target="_blank">libdb-5.3.so</a> #11 0x00007f6832b59680 in idl_new_fetch () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #12 0x00007f6832b67b26 in index_read_ext_allids () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #13 0x00007f6832b52284 in keys2idl () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #14 0x00007f6832b529e3 in ava_candidates.isra.0 () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #15 0x00007f6832b52fd2 in filter_candidates_ext () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #16 0x00007f6832b54096 in list_candidates () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #17 0x00007f6832b52f40 in filter_candidates_ext () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #18 0x00007f6832b54096 in list_candidates () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #19 0x00007f6832b52f40 in filter_candidates_ext () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #20 0x00007f6832b54096 in list_candidates () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #21 0x00007f6832b52f40 in filter_candidates_ext () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #22 0x00007f6832b8e80c in subtree_candidates () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #23 0x00007f6832b8fe7f in ldbm_back_search () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #24 0x00007f683f4bd1d6 in op_shared_search () from /usr/lib64/dirsrv/libslapd.so.0 #25 0x00007f683f9ae33e in do_search () #26 0x00007f683f99c3c3 in connection_threadmain () #27 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so #28 0x00007f683d04cdc5 in start_thread () from /lib64/libpthread.so.0 #29 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 Thread 7 (Thread 0x7f67f9feb700 (LWP 4886)): #0 0x00007f683d0506d5 in <a href="mailto:pthread_cond_wait@@GLIBC_2.3.2" target="_blank">pthread_cond_wait@@GLIBC_2.3.2</a> () from /lib64/libpthread.so.0 #1 0x00007f683d6a6050 in PR_WaitCondVar () from /lib64/libnspr4.so #2 0x00007f683f999fee in connection_wait_for_new_work () #3 0x00007f683f99b719 in connection_threadmain () #4 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so #5 0x00007f683d04cdc5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 Thread 6 (Thread 0x7f67f97ea700 (LWP 4887)): #0 0x00007f683d0506d5 in <a href="mailto:pthread_cond_wait@@GLIBC_2.3.2" target="_blank">pthread_cond_wait@@GLIBC_2.3.2</a> () from /lib64/libpthread.so.0 #1 0x00007f683d6a6050 in PR_WaitCondVar () from /lib64/libnspr4.so #2 0x00007f683f999fee in connection_wait_for_new_work () #3 0x00007f683f99b719 in connection_threadmain () #4 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so #5 0x00007f683d04cdc5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 Thread 5 (Thread 0x7f67f8fe9700 (LWP 4888)): #0 0x00007f683d0506d5 in <a href="mailto:pthread_cond_wait@@GLIBC_2.3.2" target="_blank">pthread_cond_wait@@GLIBC_2.3.2</a> () from /lib64/libpthread.so.0 #1 0x00007f683d6a6050 in PR_WaitCondVar () from /lib64/libnspr4.so #2 0x00007f683f999fee in connection_wait_for_new_work () #3 0x00007f683f99b719 in connection_threadmain () #4 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so #5 0x00007f683d04cdc5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 Thread 4 (Thread 0x7f67f87e8700 (LWP 4889)): #0 0x00007f683cd719b3 in select () from /lib64/libc.so.6 #1 0x00007f683f4fc0e9 in DS_Sleep () from /usr/lib64/dirsrv/libslapd.so.0 #2 0x00007f683f99d325 in time_thread () #3 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so #4 0x00007f683d04cdc5 in start_thread () from /lib64/libpthread.so.0 #5 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 Thread 3 (Thread 0x7f67f7de6700 (LWP 4892)): #0 0x00007f683d0506d5 in <a href="mailto:pthread_cond_wait@@GLIBC_2.3.2" target="_blank">pthread_cond_wait@@GLIBC_2.3.2</a> () from /lib64/libpthread.so.0 #1 0x00007f683d6a6050 in PR_WaitCondVar () from /lib64/libnspr4.so #2 0x00007f683f9a8f25 in ps_send_results () #3 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so #4 0x00007f683d04cdc5 in start_thread () from /lib64/libpthread.so.0 #5 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 Thread 2 (Thread 0x7f67f75e5700 (LWP 5084)): #0 0x00007f683d0506d5 in <a href="mailto:pthread_cond_wait@@GLIBC_2.3.2" target="_blank">pthread_cond_wait@@GLIBC_2.3.2</a> () from /lib64/libpthread.so.0 #1 0x00007f683d6a6050 in PR_WaitCondVar () from /lib64/libnspr4.so #2 0x00007f683f9a8f25 in ps_send_results () #3 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so #4 0x00007f683d04cdc5 in start_thread () from /lib64/libpthread.so.0 #5 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 Thread 1 (Thread 0x7f683f964840 (LWP 4845)): #0 0x00007f683d0506d5 in <a href="mailto:pthread_cond_wait@@GLIBC_2.3.2" target="_blank">pthread_cond_wait@@GLIBC_2.3.2</a> () from /lib64/libpthread.so.0 #1 0x00007f683d6a62c3 in PR_EnterMonitor () from /lib64/libnspr4.so #2 0x00007f683f99d5ed in setup_pr_read_pds () #3 0x00007f683f9a0fd9 in slapd_daemon () #4 0x00007f683f992c6c in main () </div> the top output seems ok, there was just a sudden spike to around 340% cpu then it comes down </div> top - 08:53:12 up 6 days, 23:20, 1 user, load average: 0.00, 0.01, 0.12 Threads: 44 total, 0 running, 44 sleeping, 0 stopped, 0 zombie %Cpu(s): 0.0 us, 0.0 sy, 0.0 ni,100.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st KiB Mem : 65533176 total, 52165676 free, 6135088 used, 7232412 buff/cache KiB Swap: 3999996 total, 3999996 free, 0 used. 58452040 avail Mem PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 4845 dirsrv 20 0 5133196 3.150g 281272 S 0.0 5.0 31:34.51 ns-slapd 4847 dirsrv 20 0 5133196 3.150g 281272 S 0.0 5.0 0:13.53 ns-slapd 4848 dirsrv 20 0 5133196 3.150g 281272 S 0.0 5.0 0:13.16 ns-slapd 4849 dirsrv 20 0 5133196 3.150g 281272 S 0.0 5.0 0:45.27 ns-slapd 4850 dirsrv 20 0 5133196 3.150g 281272 S 0.0 5.0 0:01.60 ns-slapd 4851 dirsrv 20 0 5133196 3.150g 281272 S 0.0 5.0 0:00.00 ns-slapd 4852 dirsrv 20 0 5133196 3.150g 281272 S 0.0 5.0 0:00.00 ns-slapd 4853 dirsrv 20 0 5133196 3.150g 281272 S 0.0 5.0 0:00.00 ns-slapd 4854 dirsrv 20 0 5133196 3.150g 281272 S 0.0 5.0 0:00.00 ns-slapd 4855 dirsrv 20 0 5133196 3.150g 281272 S 0.0 5.0 0:03.02 ns-slapd 4856 dirsrv 20 0 5133196 3.150g 281272 S 0.0 5.0 0:00.26 ns-slapd 4859 dirsrv 20 0 5133196 3.150g 281272 S 0.0 5.0 3:18.67 ns-slapd 4860 dirsrv 20 0 5133196 3.150g 281272 S 0.0 5.0 3:19.82 ns-slapd 4861 dirsrv 20 0 5133196 3.150g 281272 S 0.0 5.0 3:20.23 ns-slapd 4862 dirsrv 20 0 5133196 3.150g 281272 S 0.0 5.0 3:22.65 ns-slapd 4863 dirsrv 20 0 5133196 3.150g 281272 S 0.0 5.0 3:16.64 ns-slapd 4864 dirsrv 20 0 5133196 3.150g 281272 S 0.0 5.0 3:17.39 ns-slapd 4865 dirsrv 20 0 5133196 3.150g 281272 S 0.0 5.0 3:17.80 ns-slapd </div> Thanks </div> Rakesh </div> <div class="gmail_extra"> <div class="gmail_quote">On Tue, Sep 6, 2016 at 12:09 AM, Rakesh Rajasekharan <<a href="mailto:rakesh.rajasekharan@gmail.com" target="_blank">rakesh.rajasekharan@gmail.com</a>> wrote: <blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"> <div dir="ltr"> <div> <div> <div> <div> <div>Hi Thierry/Ludwig, </div> Unfortunately I had by then restarted the service and since then have been trying to hit the hang issue again but no luck this time </div> It has been running fine for the last few hours now in-spite of trying all that I believe could cause the hang like making simultaneous client addition requests. </div> I will continue testing this and next time I get it I will make sure I get the pstack and also get the dump of db lock info. </div> Thanks, </div> Rakesh <div> <div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div>On Mon, Sep 5, 2016 at 6:14 PM, Ludwig Krispenz <<a href="mailto:lkrispen@redhat.com" target="_blank">lkrispen@redhat.com</a>> wrote: </div> </div> <blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"> <div> <div> <div bgcolor="#FFFFFF" text="#000000"> <div>On 09/05/2016 02:31 PM, thierry bordaz wrote: </div> <blockquote type="cite"> Hi Rakesh, Thanks you soooo much for you continuous help on this !! If the hang is still going on, would you dump the DB lock information with: <blockquote>db_stat -CA -N -h /var/lib/dirsrv/slapd-<instance>/db >/tmp/db_lock_info </blockquote> It is looking like several threads are waiting for one/several db locks. With the lock info we will know which thread acquired it/them. A guess is that it could be the checkpointing thread but this one is also waiting (on a memory pool). I am copying Ludwig on this thread in case this pstack rings any bell. </blockquote> the frequent presence of __db_tas_mutex_lock() looks like the NUMA problem of libdb, although the threads already have stopped spinning (maybe this was an effect of gcore). If the hang aoccurs again, could you check cpu consumption top -H -p <slapd-pid> and get a pstack pstack <slapd-pid> <div> <div> <blockquote type="cite"> best regards thierry <div>On 09/05/2016 02:05 PM, Rakesh Rajasekharan wrote: </div> <blockquote type="cite"> <div dir="ltr"> <div> <div> <div> <div> <div>Hi Thierry, </div> I was able to now reporduce the issue.. </div> the ns-slapd process just got hung,.. however, no core file was generate.. </div> I manually created a coredump using gcore and then took a strace of it this is strace from the coredump generated GNU gdb (GDB) Red Hat Enterprise Linux 7.6.1-80.el7 Copyright (C) 2013 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later <<a href="http://gnu.org/licenses/gpl.h" target="_blank">http://gnu.org/licenses/gpl.h</a>tml> This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type "show copying" and "show warranty" for details. This GDB was configured as "x86_64-redhat-linux-gnu". For bug reporting instructions, please see: <<a href="http://www.gnu.org/software/gdb/bugs/" target="_blank">http://www.gnu.org/software/gdb/bugs/</a>>... Reading symbols from /usr/sbin/ns-slapd...Reading symbols from /usr/sbin/ns-slapd...(no debugging symbols found)...done. (no debugging symbols found)...done. [New LWP 105159] [New LWP 105160] [New LWP 105161] [New LWP 105162] [New LWP 105163] [New LWP 105164] [New LWP 105165] [New LWP 105166] [New LWP 105167] [New LWP 105168] [New LWP 105171] [New LWP 105172] [New LWP 105173] [New LWP 105174] [New LWP 105175] [New LWP 105176] [New LWP 105177] [New LWP 105178] [New LWP 105179] [New LWP 105180] [New LWP 105181] [New LWP 105182] [New LWP 105183] [New LWP 105184] [New LWP 105185] [New LWP 105186] [New LWP 105187] [New LWP 105188] [New LWP 105189] [New LWP 105190] [New LWP 105191] [New LWP 105192] [New LWP 105193] [New LWP 105194] [New LWP 105195] [New LWP 105196] [New LWP 105197] [New LWP 105198] [New LWP 105199] [New LWP 105200] [New LWP 105201] [New LWP 105203] [New LWP 105215] [New LWP 105158] [Thread debugging using libthread_db enabled] Using host libthread_db library "/lib64/libthread_db.so.1". Core was generated by `/usr/sbin/ns-slapd'. #0 0x00007f1b48ae69b3 in select () from /lib64/libc.so.6 Thread 44 (Thread 0x7f1b4b6d9840 (LWP 105158)): #0 0x00007f1b48dc56d5 in <a href="mailto:pthread_cond_wait@@GLIBC_2.3.2" target="_blank">pthread_cond_wait@@GLIBC_2.3.2</a> () from /lib64/libpthread.so.0 No symbol table info available. #1 0x00007f1b4941b2c3 in PR_EnterMonitor () from /lib64/libnspr4.so No symbol table info available. #2 0x00007f1b4b7125ed in setup_pr_read_pds () No symbol table info available. #3 0x00007f1b4b715fd9 in slapd_daemon () No symbol table info available. #4 0x00007f1b4b707c6c in main () No symbol table info available. Thread 43 (Thread 0x7f1a5df0f700 (LWP 105215)): #0 0x00007f1b48dc56d5 in <a href="mailto:pthread_cond_wait@@GLIBC_2.3.2" target="_blank">pthread_cond_wait@@GLIBC_2.3.2</a> () from /lib64/libpthread.so.0 No symbol table info available. #1 0x00007f1b4941b050 in PR_WaitCondVar () from /lib64/libnspr4.so No symbol table info available. #2 0x00007f1b4b71df25 in ps_send_results () No symbol table info available. #3 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so No symbol table info available. #4 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 No symbol table info available. #5 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 No symbol table info available. Thread 42 (Thread 0x7f1b03fe7700 (LWP 105203)): #0 0x00007f1b48dc56d5 in <a href="mailto:pthread_cond_wait@@GLIBC_2.3.2" target="_blank">pthread_cond_wait@@GLIBC_2.3.2</a> () from /lib64/libpthread.so.0 No symbol table info available. #1 0x00007f1b4941b050 in PR_WaitCondVar () from /lib64/libnspr4.so No symbol table info available. #2 0x00007f1b4b71df25 in ps_send_results () No symbol table info available. #3 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so No symbol table info available. #4 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 No symbol table info available. #5 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 No symbol table info available. Thread 41 (Thread 0x7f1b047e8700 (LWP 105201)): #0 0x00007f1b48ae69b3 in select () from /lib64/libc.so.6 No symbol table info available. #1 0x00007f1b4b2710e9 in DS_Sleep () from /usr/lib64/dirsrv/libslapd.so.0 No symbol table info available. #2 0x00007f1b4b712325 in time_thread () No symbol table info available. #3 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so No symbol table info available. #4 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 No symbol table info available. #5 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 No symbol table info available. Thread 40 (Thread 0x7f1b04fe9700 (LWP 105200)): #0 0x00007f1b48dc56d5 in <a href="mailto:pthread_cond_wait@@GLIBC_2.3.2" target="_blank">pthread_cond_wait@@GLIBC_2.3.2</a> () from /lib64/libpthread.so.0 No symbol table info available. #1 0x00007f1b433212f3 in __db_hybrid_mutex_suspend () from /lib64/<a href="http://libdb-5.3.so" target="_blank">libdb-5.3.so</a> No symbol table info available. #2 0x00007f1b43320640 in __db_tas_mutex_lock () from /lib64/<a href="http://libdb-5.3.so" target="_blank">libdb-5.3.so</a> No symbol table info available. #3 0x00007f1b4343ae1b in __memp_fget () from /lib64/<a href="http://libdb-5.3.so" target="_blank">libdb-5.3.so</a> No symbol table info available. #4 0x00007f1b4333e211 in __bam_search () from /lib64/<a href="http://libdb-5.3.so" target="_blank">libdb-5.3.so</a> No symbol table info available. #5 0x00007f1b43329256 in __bamc_search () from /lib64/<a href="http://libdb-5.3.so" target="_blank">libdb-5.3.so</a> No symbol table info available. #6 0x00007f1b4332d6c4 in __bamc_put () from /lib64/<a href="http://libdb-5.3.so" target="_blank">libdb-5.3.so</a> No symbol table info available. #7 0x00007f1b433e46d5 in __dbc_iput () from /lib64/<a href="http://libdb-5.3.so" target="_blank">libdb-5.3.so</a> No symbol table info available. #8 0x00007f1b433dfa0e in __db_put () from /lib64/<a href="http://libdb-5.3.so" target="_blank">libdb-5.3.so</a> No symbol table info available. #9 0x00007f1b433f4fa4 in __db_put_pp () from /lib64/<a href="http://libdb-5.3.so" target="_blank">libdb-5.3.so</a> No symbol table info available. #10 0x00007f1b3e8cf4cf in idl_new_insert_key () from /usr/lib64/dirsrv/plugins/libback-ldbm.so No symbol table info available. #11 0x00007f1b3e8de5b5 in addordel_values_sv.isra.3 () from /usr/lib64/dirsrv/plugins/libback-ldbm.so No symbol table info available. #12 0x00007f1b3e8df25a in index_addordel_values_ext_sv () from /usr/lib64/dirsrv/plugins/libback-ldbm.so No symbol table info available. #13 0x00007f1b3e8df694 in index_addordel_values_sv () from /usr/lib64/dirsrv/plugins/libback-ldbm.so No symbol table info available. #14 0x00007f1b3e8df75a in index_addordel_entry () from /usr/lib64/dirsrv/plugins/libback-ldbm.so No symbol table info available. #15 0x00007f1b3e8e25fa in ldbm_back_add () from /usr/lib64/dirsrv/plugins/libback-ldbm.so No symbol table info available. #16 0x00007f1b4b1e4cca in op_shared_add () from /usr/lib64/dirsrv/libslapd.so.0 No symbol table info available. #17 0x00007f1b4b1e6020 in do_add () from /usr/lib64/dirsrv/libslapd.so.0 No symbol table info available. #18 0x00007f1b4b711303 in connection_threadmain () No symbol table info available. #19 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so No symbol table info available. #20 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 No symbol table info available. #21 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 No symbol table info available. Thread 39 (Thread 0x7f1b057ea700 (LWP 105199)): #0 0x00007f1b48dc56d5 in <a href="mailto:pthread_cond_wait@@GLIBC_2.3.2" target="_blank">pthread_cond_wait@@GLIBC_2.3.2</a> () from /lib64/libpthread.so.0 No symbol table info available. #1 0x00007f1b4941b050 in PR_WaitCondVar () from /lib64/libnspr4.so No symbol table info available. #2 0x00007f1b4b70efee in connection_wait_for_new_work () No symbol table info available. #3 0x00007f1b4b710719 in connection_threadmain () No symbol table info available. #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so No symbol table info available. #5 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 No symbol table info available. #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 No symbol table info available. Thread 38 (Thread 0x7f1b05feb700 (LWP 105198)): #0 0x00007f1b48dc56d5 in <a href="mailto:pthread_cond_wait@@GLIBC_2.3.2" target="_blank">pthread_cond_wait@@GLIBC_2.3.2</a> () from /lib64/libpthread.so.0 No symbol table info available. #1 0x00007f1b4941b050 in PR_WaitCondVar () from /lib64/libnspr4.so No symbol table info available. #2 0x00007f1b4b70efee in connection_wait_for_new_work () No symbol table info available. #3 0x00007f1b4b710719 in connection_threadmain () No symbol table info available. #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so No symbol table info available. #5 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 No symbol table info available. #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 No symbol table info available. Thread 37 (Thread 0x7f1b067ec700 (LWP 105197)): #0 0x00007f1b48dc56d5 in <a href="mailto:pthread_cond_wait@@GLIBC_2.3.2" target="_blank">pthread_cond_wait@@GLIBC_2.3.2</a> () from /lib64/libpthread.so.0 No symbol table info available. #1 0x00007f1b4941b050 in PR_WaitCondVar () from /lib64/libnspr4.so No symbol table info available. #2 0x00007f1b4b70efee in connection_wait_for_new_work () No symbol table info available. #3 0x00007f1b4b710719 in connection_threadmain () No symbol table info available. #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so No symbol table info available. #5 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 No symbol table info available. #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 No symbol table info available. Thread 36 (Thread 0x7f1b06fed700 (LWP 105196)): #0 0x00007f1b48dc56d5 in <a href="mailto:pthread_cond_wait@@GLIBC_2.3.2" target="_blank">pthread_cond_wait@@GLIBC_2.3.2</a> () from /lib64/libpthread.so.0 No symbol table info available. #1 0x00007f1b4941b050 in PR_WaitCondVar () from /lib64/libnspr4.so No symbol table info available. #2 0x00007f1b4b70efee in connection_wait_for_new_work () No symbol table info available. #3 0x00007f1b4b710719 in connection_threadmain () No symbol table info available. #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so No symbol table info available. #5 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 No symbol table info available. #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 No symbol table info available. Thread 35 (Thread 0x7f1b077ee700 (LWP 105195)): #0 0x00007f1b48dc56d5 in <a href="mailto:pthread_cond_wait@@GLIBC_2.3.2" target="_blank">pthread_cond_wait@@GLIBC_2.3.2</a> () from /lib64/libpthread.so.0 No symbol table info available. #1 0x00007f1b4941b2c3 in PR_EnterMonitor () from /lib64/libnspr4.so No symbol table info available. #2 0x00007f1b3e8c23e6 in dblayer_txn_begin () from /usr/lib64/dirsrv/plugins/libback-ldbm.so No symbol table info available. #3 0x00007f1b3e8fe007 in ldbm_back_modify () from /usr/lib64/dirsrv/plugins/libback-ldbm.so No symbol table info available. #4 0x00007f1b4b22b541 in op_shared_modify () from /usr/lib64/dirsrv/libslapd.so.0 No symbol table info available. #5 0x00007f1b4b22c004 in modify_internal_pb () from /usr/lib64/dirsrv/libslapd.so.0 No symbol table info available. #6 0x00007f1b3eb48b60 in ipapwd_apply_mods () from /usr/lib64/dirsrv/plugins/libipa_pwd_extop.so No symbol table info available. #7 0x00007f1b3eb4c8e0 in store_new_keys () from /usr/lib64/dirsrv/plugins/libipa_pwd_extop.so No symbol table info available. #8 0x00007f1b3eb4ead0 in ipapwd_extop () from /usr/lib64/dirsrv/plugins/libipa_pwd_extop.so No symbol table info available. #9 0x00007f1b4b23bba2 in plugin_call_exop_plugins () from /usr/lib64/dirsrv/libslapd.so.0 No symbol table info available. #10 0x00007f1b4b717efe in do_extended () No symbol table info available. #11 0x00007f1b4b71141a in connection_threadmain () No symbol table info available. #12 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so No symbol table info available. #13 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 No symbol table info available. #14 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 No symbol table info available. Thread 34 (Thread 0x7f1b07fef700 (LWP 105194)): #0 0x00007f1b48dc56d5 in <a href="mailto:pthread_cond_wait@@GLIBC_2.3.2" target="_blank">pthread_cond_wait@@GLIBC_2.3.2</a> () from /lib64/libpthread.so.0 No symbol table info available. #1 0x00007f1b4941b050 in PR_WaitCondVar () from /lib64/libnspr4.so No symbol table info available. #2 0x00007f1b4b70efee in connection_wait_for_new_work () No symbol table info available. #3 0x00007f1b4b710719 in connection_threadmain () No symbol table info available. #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so No symbol table info available. #5 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 No symbol table info available. #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 No symbol table info available. Thread 33 (Thread 0x7f1b087f0700 (LWP 105193)): #0 0x00007f1b48dc56d5 in <a href="mailto:pthread_cond_wait@@GLIBC_2.3.2" target="_blank">pthread_cond_wait@@GLIBC_2.3.2</a> () from /lib64/libpthread.so.0 No symbol table info available. #1 0x00007f1b4941b050 in PR_WaitCondVar () from /lib64/libnspr4.so No symbol table info available. #2 0x00007f1b4b70efee in connection_wait_for_new_work () No symbol table info available. #3 0x00007f1b4b710719 in connection_threadmain () No symbol table info available. #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so No symbol table info available. #5 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 No symbol table info available. #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 No symbol table info available. Thread 32 (Thread 0x7f1b08ff1700 (LWP 105192)): #0 0x00007f1b48dc56d5 in <a href="mailto:pthread_cond_wait@@GLIBC_2.3.2" target="_blank">pthread_cond_wait@@GLIBC_2.3.2</a> () from /lib64/libpthread.so.0 No symbol table info available. #1 0x00007f1b4941b050 in PR_WaitCondVar () from /lib64/libnspr4.so No symbol table info available. #2 0x00007f1b4b70efee in connection_wait_for_new_work () No symbol table info available. #3 0x00007f1b4b710719 in connection_threadmain () No symbol table info available. #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so No symbol table info available. #5 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 No symbol table info available. #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 No symbol table info available. Thread 31 (Thread 0x7f1b097f2700 (LWP 105191)): #0 0x00007f1b48dc56d5 in <a href="mailto:pthread_cond_wait@@GLIBC_2.3.2" target="_blank">pthread_cond_wait@@GLIBC_2.3.2</a> () from /lib64/libpthread.so.0 No symbol table info available. #1 0x00007f1b4941b050 in PR_WaitCondVar () from /lib64/libnspr4.so No symbol table info available. #2 0x00007f1b4b70efee in connection_wait_for_new_work () No symbol table info available. #3 0x00007f1b4b710719 in connection_threadmain () No symbol table info available. #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so No symbol table info available. #5 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 No symbol table info available. #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 No symbol table info available. Thread 30 (Thread 0x7f1b09ff3700 (LWP 105190)): #0 0x00007f1b48dc56d5 in <a href="mailto:pthread_cond_wait@@GLIBC_2.3.2" target="_blank">pthread_cond_wait@@GLIBC_2.3.2</a> () from /lib64/libpthread.so.0 No symbol table info available. #1 0x00007f1b433212f3 in __db_hybrid_mutex_suspend () from /lib64/<a href="http://libdb-5.3.so" target="_blank">libdb-5.3.so</a> No symbol table info available. #2 0x00007f1b43320640 in __db_tas_mutex_lock () from /lib64/<a href="http://libdb-5.3.so" target="_blank">libdb-5.3.so</a> No symbol table info available. #3 0x00007f1b433cad3a in __lock_get_internal () from /lib64/<a href="http://libdb-5.3.so" target="_blank">libdb-5.3.so</a> No symbol table info available. #4 0x00007f1b433cb820 in __lock_get () from /lib64/<a href="http://libdb-5.3.so" target="_blank">libdb-5.3.so</a> No symbol table info available. #5 0x00007f1b433f7142 in __db_lget () from /lib64/<a href="http://libdb-5.3.so" target="_blank">libdb-5.3.so</a> No symbol table info available. #6 0x00007f1b4333e605 in __bam_search () from /lib64/<a href="http://libdb-5.3.so" target="_blank">libdb-5.3.so</a> No symbol table info available. #7 0x00007f1b43329256 in __bamc_search () from /lib64/<a href="http://libdb-5.3.so" target="_blank">libdb-5.3.so</a> No symbol table info available. #8 0x00007f1b4332ad0f in __bamc_get () from /lib64/<a href="http://libdb-5.3.so" target="_blank">libdb-5.3.so</a> No symbol table info available. #9 0x00007f1b433e3ca6 in __dbc_iget () from /lib64/<a href="http://libdb-5.3.so" target="_blank">libdb-5.3.so</a> No symbol table info available. #10 0x00007f1b433f2b02 in __dbc_get_pp () from /lib64/<a href="http://libdb-5.3.so" target="_blank">libdb-5.3.so</a> No symbol table info available. #11 0x00007f1b3e8ce680 in idl_new_fetch () from /usr/lib64/dirsrv/plugins/libback-ldbm.so No symbol table info available. #12 0x00007f1b3e8dcb26 in index_read_ext_allids () from /usr/lib64/dirsrv/plugins/libback-ldbm.so No symbol table info available. #13 0x00007f1b3e8c7284 in keys2idl () from /usr/lib64/dirsrv/plugins/libback-ldbm.so No symbol table info available. #14 0x00007f1b3e8c79e3 in ava_candidates.isra.0 () from /usr/lib64/dirsrv/plugins/libback-ldbm.so No symbol table info available. #15 0x00007f1b3e8c7fd2 in filter_candidates_ext () from /usr/lib64/dirsrv/plugins/libback-ldbm.so No symbol table info available. #16 0x00007f1b3e8c9096 in list_candidates () from /usr/lib64/dirsrv/plugins/libback-ldbm.so No symbol table info available. #17 0x00007f1b3e8c7f40 in filter_candidates_ext () from /usr/lib64/dirsrv/plugins/libback-ldbm.so No symbol table info available. #18 0x00007f1b3e8c9096 in list_candidates () from /usr/lib64/dirsrv/plugins/libback-ldbm.so No symbol table info available. #19 0x00007f1b3e8c7f40 in filter_candidates_ext () from /usr/lib64/dirsrv/plugins/libback-ldbm.so No symbol table info available. #20 0x00007f1b3e8c9096 in list_candidates () from /usr/lib64/dirsrv/plugins/libback-ldbm.so No symbol table info available. #21 0x00007f1b3e8c7f40 in filter_candidates_ext () from /usr/lib64/dirsrv/plugins/libback-ldbm.so No symbol table info available. #22 0x00007f1b3e90380c in subtree_candidates () from /usr/lib64/dirsrv/plugins/libback-ldbm.so No symbol table info available. #23 0x00007f1b3e904e7f in ldbm_back_search () from /usr/lib64/dirsrv/plugins/libback-ldbm.so No symbol table info available. #24 0x00007f1b4b2321d6 in op_shared_search () from /usr/lib64/dirsrv/libslapd.so.0 No symbol table info available. #25 0x00007f1b4b72333e in do_search () No symbol table info available. #26 0x00007f1b4b7113c3 in connection_threadmain () No symbol table info available. #27 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so No symbol table info available. #28 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 No symbol table info available. #29 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 No symbol table info available. Thread 29 (Thread 0x7f1b0a7f4700 (LWP 105189)): #0 0x00007f1b48dc56d5 in <a href="mailto:pthread_cond_wait@@GLIBC_2.3.2" target="_blank">pthread_cond_wait@@GLIBC_2.3.2</a> () from /lib64/libpthread.so.0 No symbol table info available. #1 0x00007f1b4941b050 in PR_WaitCondVar () from /lib64/libnspr4.so No symbol table info available. #2 0x00007f1b4b70efee in connection_wait_for_new_work () No symbol table info available. #3 0x00007f1b4b710719 in connection_threadmain () No symbol table info available. #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so No symbol table info available. #5 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 No symbol table info available. #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 No symbol table info available. Thread 28 (Thread 0x7f1b0aff5700 (LWP 105188)): #0 0x00007f1b48dc56d5 in <a href="mailto:pthread_cond_wait@@GLIBC_2.3.2" target="_blank">pthread_cond_wait@@GLIBC_2.3.2</a> () from /lib64/libpthread.so.0 No symbol table info available. #1 0x00007f1b4941b050 in PR_WaitCondVar () from /lib64/libnspr4.so No symbol table info available. #2 0x00007f1b4b70efee in connection_wait_for_new_work () No symbol table info available. #3 0x00007f1b4b710719 in connection_threadmain () No symbol table info available. #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so No symbol table info available. #5 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 No symbol table info available. #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 No symbol table info available. Thread 27 (Thread 0x7f1b0b7f6700 (LWP 105187)): #0 0x00007f1b48dc56d5 in <a href="mailto:pthread_cond_wait@@GLIBC_2.3.2" target="_blank">pthread_cond_wait@@GLIBC_2.3.2</a> () from /lib64/libpthread.so.0 No symbol table info available. #1 0x00007f1b4941b050 in PR_WaitCondVar () from /lib64/libnspr4.so No symbol table info available. #2 0x00007f1b4b70efee in connection_wait_for_new_work () No symbol table info available. #3 0x00007f1b4b710719 in connection_threadmain () No symbol table info available. #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so No symbol table info available. #5 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 No symbol table info available. #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 No symbol table info available. Thread 26 (Thread 0x7f1b0bff7700 (LWP 105186)): #0 0x00007f1b48dc56d5 in <a href="mailto:pthread_cond_wait@@GLIBC_2.3.2" target="_blank">pthread_cond_wait@@GLIBC_2.3.2</a> () from /lib64/libpthread.so.0 No symbol table info available. #1 0x00007f1b433212f3 in __db_hybrid_mutex_suspend () from /lib64/<a href="http://libdb-5.3.so" target="_blank">libdb-5.3.so</a> No symbol table info available. #2 0x00007f1b43320640 in __db_tas_mutex_lock () from /lib64/<a href="http://libdb-5.3.so" target="_blank">libdb-5.3.so</a> No symbol table info available. #3 0x00007f1b433cad3a in __lock_get_internal () from /lib64/<a href="http://libdb-5.3.so" target="_blank">libdb-5.3.so</a> No symbol table info available. #4 0x00007f1b433cb820 in __lock_get () from /lib64/<a href="http://libdb-5.3.so" target="_blank">libdb-5.3.so</a> No symbol table info available. #5 0x00007f1b433f7142 in __db_lget () from /lib64/<a href="http://libdb-5.3.so" target="_blank">libdb-5.3.so</a> No symbol table info available. #6 0x00007f1b4333e605 in __bam_search () from /lib64/<a href="http://libdb-5.3.so" target="_blank">libdb-5.3.so</a> No symbol table info available. #7 0x00007f1b43329256 in __bamc_search () from /lib64/<a href="http://libdb-5.3.so" target="_blank">libdb-5.3.so</a> No symbol table info available. #8 0x00007f1b4332ad0f in __bamc_get () from /lib64/<a href="http://libdb-5.3.so" target="_blank">libdb-5.3.so</a> No symbol table info available. #9 0x00007f1b433e3ca6 in __dbc_iget () from /lib64/<a href="http://libdb-5.3.so" target="_blank">libdb-5.3.so</a> No symbol table info available. #10 0x00007f1b433f2b02 in __dbc_get_pp () from /lib64/<a href="http://libdb-5.3.so" target="_blank">libdb-5.3.so</a> No symbol table info available. #11 0x00007f1b3e8ce680 in idl_new_fetch () from /usr/lib64/dirsrv/plugins/libback-ldbm.so No symbol table info available. #12 0x00007f1b3e8dcb26 in index_read_ext_allids () from /usr/lib64/dirsrv/plugins/libback-ldbm.so No symbol table info available. #13 0x00007f1b3e8c7284 in keys2idl () from /usr/lib64/dirsrv/plugins/libback-ldbm.so No symbol table info available. #14 0x00007f1b3e8c79e3 in ava_candidates.isra.0 () from /usr/lib64/dirsrv/plugins/libback-ldbm.so No symbol table info available. #15 0x00007f1b3e8c7fd2 in filter_candidates_ext () from /usr/lib64/dirsrv/plugins/libback-ldbm.so No symbol table info available. #16 0x00007f1b3e8c9096 in list_candidates () from /usr/lib64/dirsrv/plugins/libback-ldbm.so No symbol table info available. #17 0x00007f1b3e8c7f40 in filter_candidates_ext () from /usr/lib64/dirsrv/plugins/libback-ldbm.so No symbol table info available. #18 0x00007f1b3e90380c in subtree_candidates () from /usr/lib64/dirsrv/plugins/libback-ldbm.so No symbol table info available. #19 0x00007f1b3e904e7f in ldbm_back_search () from /usr/lib64/dirsrv/plugins/libback-ldbm.so No symbol table info available. #20 0x00007f1b4b2321d6 in op_shared_search () from /usr/lib64/dirsrv/libslapd.so.0 No symbol table info available. #21 0x00007f1b4b24294e in search_internal_callback_pb () from /usr/lib64/dirsrv/libslapd.so.0 No symbol table info available. #22 0x00007f1b4b242be8 in search_internal_pb () from /usr/lib64/dirsrv/libslapd.so.0 No symbol table info available. #23 0x00007f1b4b72149b in ids_sasl_canon_user () No symbol table info available. #24 0x00007f1b491e193b in _sasl_canon_user () from /lib64/libsasl2.so.3 No symbol table info available. #25 0x00007f1b491e1c4c in _sasl_canon_user_lookup () from /lib64/libsasl2.so.3 No symbol table info available. #26 0x00007f1b430e9fb9 in gssapi_server_mech_step () from /usr/lib64/sasl2/libgssapiv2.so No symbol table info available. #27 0x00007f1b491edb9b in sasl_server_step () from /lib64/libsasl2.so.3 No symbol table info available. #28 0x00007f1b4b72283b in ids_sasl_check_bind () No symbol table info available. #29 0x00007f1b4b70a61a in do_bind () No symbol table info available. #30 0x00007f1b4b7113fd in connection_threadmain () No symbol table info available. #31 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so No symbol table info available. #32 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 No symbol table info available. #33 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 No symbol table info available. Thread 25 (Thread 0x7f1b0c7f8700 (LWP 105185)): #0 0x00007f1b48dc56d5 in <a href="mailto:pthread_cond_wait@@GLIBC_2.3.2" target="_blank">pthread_cond_wait@@GLIBC_2.3.2</a> () from /lib64/libpthread.so.0 No symbol table info available. #1 0x00007f1b4941b050 in PR_WaitCondVar () from /lib64/libnspr4.so No symbol table info available. #2 0x00007f1b4b70efee in connection_wait_for_new_work () No symbol table info available. #3 0x00007f1b4b710719 in connection_threadmain () No symbol table info available. #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so No symbol table info available. #5 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 No symbol table info available. #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 No symbol table info available. Thread 24 (Thread 0x7f1b0cff9700 (LWP 105184)): #0 0x00007f1b48dc56d5 in <a href="mailto:pthread_cond_wait@@GLIBC_2.3.2" target="_blank">pthread_cond_wait@@GLIBC_2.3.2</a> () from /lib64/libpthread.so.0 No symbol table info available. #1 0x00007f1b4941b050 in PR_WaitCondVar () from /lib64/libnspr4.so No symbol table info available. #2 0x00007f1b4b70efee in connection_wait_for_new_work () No symbol table info available. #3 0x00007f1b4b710719 in connection_threadmain () No symbol table info available. #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so No symbol table info available. #5 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 No symbol table info available. #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 No symbol table info available. Thread 23 (Thread 0x7f1b0d7fa700 (LWP 105183)): #0 0x00007f1b48dc56d5 in <a href="mailto:pthread_cond_wait@@GLIBC_2.3.2" target="_blank">pthread_cond_wait@@GLIBC_2.3.2</a> () from /lib64/libpthread.so.0 No symbol table info available. #1 0x00007f1b4941b050 in PR_WaitCondVar () from /lib64/libnspr4.so No symbol table info available. #2 0x00007f1b4b70efee in connection_wait_for_new_work () No symbol table info available. #3 0x00007f1b4b710719 in connection_threadmain () No symbol table info available. #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so No symbol table info available. #5 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 No symbol table info available. #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 No symbol table info available. Thread 22 (Thread 0x7f1b0dffb700 (LWP 105182)): #0 0x00007f1b48dc56d5 in <a href="mailto:pthread_cond_wait@@GLIBC_2.3.2" target="_blank">pthread_cond_wait@@GLIBC_2.3.2</a> () from /lib64/libpthread.so.0 No symbol table info available. #1 0x00007f1b4941b050 in PR_WaitCondVar () from /lib64/libnspr4.so No symbol table info available. #2 0x00007f1b4b70efee in connection_wait_for_new_work () No symbol table info available. #3 0x00007f1b4b710719 in connection_threadmain () No symbol table info available. #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so No symbol table info available. #5 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 No symbol table info available. #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 No symbol table info available. Thread 21 (Thread 0x7f1b0e7fc700 (LWP 105181)): #0 0x00007f1b48dc56d5 in <a href="mailto:pthread_cond_wait@@GLIBC_2.3.2" target="_blank">pthread_cond_wait@@GLIBC_2.3.2</a> () from /lib64/libpthread.so.0 No symbol table info available. #1 0x00007f1b4941b050 in PR_WaitCondVar () from /lib64/libnspr4.so No symbol table info available. #2 0x00007f1b4b70efee in connection_wait_for_new_work () No symbol table info available. #3 0x00007f1b4b710719 in connection_threadmain () No symbol table info available. #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so No symbol table info available. #5 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 No symbol table info available. #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 No symbol table info available. Thread 20 (Thread 0x7f1b0effd700 (LWP 105180)): #0 0x00007f1b48dc56d5 in <a href="mailto:pthread_cond_wait@@GLIBC_2.3.2" target="_blank">pthread_cond_wait@@GLIBC_2.3.2</a> () from /lib64/libpthread.so.0 No symbol table info available. #1 0x00007f1b433212f3 in __db_hybrid_mutex_suspend () from /lib64/<a href="http://libdb-5.3.so" target="_blank">libdb-5.3.so</a> No symbol table info available. #2 0x00007f1b43320b37 in __db_tas_mutex_readlock () from /lib64/<a href="http://libdb-5.3.so" target="_blank">libdb-5.3.so</a> No symbol table info available. #3 0x00007f1b4343ad35 in __memp_fget () from /lib64/<a href="http://libdb-5.3.so" target="_blank">libdb-5.3.so</a> No symbol table info available. #4 0x00007f1b433e4333 in __dbc_iget () from /lib64/<a href="http://libdb-5.3.so" target="_blank">libdb-5.3.so</a> No symbol table info available. #5 0x00007f1b433f2b02 in __dbc_get_pp () from /lib64/<a href="http://libdb-5.3.so" target="_blank">libdb-5.3.so</a> No symbol table info available. #6 0x00007f1b3e8ce822 in idl_new_fetch () from /usr/lib64/dirsrv/plugins/libback-ldbm.so No symbol table info available. #7 0x00007f1b3e8dcb26 in index_read_ext_allids () from /usr/lib64/dirsrv/plugins/libback-ldbm.so No symbol table info available. #8 0x00007f1b3e8c7284 in keys2idl () from /usr/lib64/dirsrv/plugins/libback-ldbm.so No symbol table info available. #9 0x00007f1b3e8c79e3 in ava_candidates.isra.0 () from /usr/lib64/dirsrv/plugins/libback-ldbm.so No symbol table info available. #10 0x00007f1b3e8c7fd2 in filter_candidates_ext () from /usr/lib64/dirsrv/plugins/libback-ldbm.so No symbol table info available. #11 0x00007f1b3e8c9096 in list_candidates () from /usr/lib64/dirsrv/plugins/libback-ldbm.so No symbol table info available. #12 0x00007f1b3e8c7f40 in filter_candidates_ext () from /usr/lib64/dirsrv/plugins/libback-ldbm.so No symbol table info available. #13 0x00007f1b3e8c9096 in list_candidates () from /usr/lib64/dirsrv/plugins/libback-ldbm.so No symbol table info available. #14 0x00007f1b3e8c7f40 in filter_candidates_ext () from /usr/lib64/dirsrv/plugins/libback-ldbm.so No symbol table info available. #15 0x00007f1b3e8c9096 in list_candidates () from /usr/lib64/dirsrv/plugins/libback-ldbm.so No symbol table info available. #16 0x00007f1b3e8c7f40 in filter_candidates_ext () from /usr/lib64/dirsrv/plugins/libback-ldbm.so No symbol table info available. #17 0x00007f1b3e90380c in subtree_candidates () from /usr/lib64/dirsrv/plugins/libback-ldbm.so No symbol table info available. #18 0x00007f1b3e904e7f in ldbm_back_search () from /usr/lib64/dirsrv/plugins/libback-ldbm.so No symbol table info available. #19 0x00007f1b4b2321d6 in op_shared_search () from /usr/lib64/dirsrv/libslapd.so.0 No symbol table info available. #20 0x00007f1b4b72333e in do_search () No symbol table info available. #21 0x00007f1b4b7113c3 in connection_threadmain () No symbol table info available. #22 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so No symbol table info available. #23 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 No symbol table info available. #24 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 No symbol table info available. Thread 19 (Thread 0x7f1b0f7fe700 (LWP 105179)): #0 0x00007f1b48dc56d5 in <a href="mailto:pthread_cond_wait@@GLIBC_2.3.2" target="_blank">pthread_cond_wait@@GLIBC_2.3.2</a> () from /lib64/libpthread.so.0 No symbol table info available. #1 0x00007f1b433212f3 in __db_hybrid_mutex_suspend () from /lib64/<a href="http://libdb-5.3.so" target="_blank">libdb-5.3.so</a> No symbol table info available. #2 0x00007f1b43320640 in __db_tas_mutex_lock () from /lib64/<a href="http://libdb-5.3.so" target="_blank">libdb-5.3.so</a> No symbol table info available. #3 0x00007f1b433cad3a in __lock_get_internal () from /lib64/<a href="http://libdb-5.3.so" target="_blank">libdb-5.3.so</a> No symbol table info available. #4 0x00007f1b433cb820 in __lock_get () from /lib64/<a href="http://libdb-5.3.so" target="_blank">libdb-5.3.so</a> No symbol table info available. #5 0x00007f1b433f7142 in __db_lget () from /lib64/<a href="http://libdb-5.3.so" target="_blank">libdb-5.3.so</a> No symbol table info available. #6 0x00007f1b4333e605 in __bam_search () from /lib64/<a href="http://libdb-5.3.so" target="_blank">libdb-5.3.so</a> No symbol table info available. #7 0x00007f1b43329256 in __bamc_search () from /lib64/<a href="http://libdb-5.3.so" target="_blank">libdb-5.3.so</a> No symbol table info available. #8 0x00007f1b4332ad0f in __bamc_get () from /lib64/<a href="http://libdb-5.3.so" target="_blank">libdb-5.3.so</a> No symbol table info available. #9 0x00007f1b433e3ca6 in __dbc_iget () from /lib64/<a href="http://libdb-5.3.so" target="_blank">libdb-5.3.so</a> No symbol table info available. #10 0x00007f1b433f2b02 in __dbc_get_pp () from /lib64/<a href="http://libdb-5.3.so" target="_blank">libdb-5.3.so</a> No symbol table info available. #11 0x00007f1b3e8ce680 in idl_new_fetch () from /usr/lib64/dirsrv/plugins/libback-ldbm.so No symbol table info available. #12 0x00007f1b3e8dcb26 in index_read_ext_allids () from /usr/lib64/dirsrv/plugins/libback-ldbm.so No symbol table info available. #13 0x00007f1b3e8c7284 in keys2idl () from /usr/lib64/dirsrv/plugins/libback-ldbm.so No symbol table info available. #14 0x00007f1b3e8c79e3 in ava_candidates.isra.0 () from /usr/lib64/dirsrv/plugins/libback-ldbm.so No symbol table info available. #15 0x00007f1b3e8c7fd2 in filter_candidates_ext () from /usr/lib64/dirsrv/plugins/libback-ldbm.so No symbol table info available. #16 0x00007f1b3e8c9096 in list_candidates () from /usr/lib64/dirsrv/plugins/libback-ldbm.so No symbol table info available. #17 0x00007f1b3e8c7f40 in filter_candidates_ext () from /usr/lib64/dirsrv/plugins/libback-ldbm.so No symbol table info available. #18 0x00007f1b3e8c9096 in list_candidates () from /usr/lib64/dirsrv/plugins/libback-ldbm.so No symbol table info available. #19 0x00007f1b3e8c7f40 in filter_candidates_ext () from /usr/lib64/dirsrv/plugins/libback-ldbm.so No symbol table info available. #20 0x00007f1b3e8c9096 in list_candidates () from /usr/lib64/dirsrv/plugins/libback-ldbm.so No symbol table info available. #21 0x00007f1b3e8c7f40 in filter_candidates_ext () from /usr/lib64/dirsrv/plugins/libback-ldbm.so No symbol table info available. #22 0x00007f1b3e90380c in subtree_candidates () from /usr/lib64/dirsrv/plugins/libback-ldbm.so No symbol table info available. #23 0x00007f1b3e904e7f in ldbm_back_search () from /usr/lib64/dirsrv/plugins/libback-ldbm.so No symbol table info available. #24 0x00007f1b4b2321d6 in op_shared_search () from /usr/lib64/dirsrv/libslapd.so.0 No symbol table info available. #25 0x00007f1b4b72333e in do_search () No symbol table info available. #26 0x00007f1b4b7113c3 in connection_threadmain () No symbol table info available. #27 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so No symbol table info available. #28 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 No symbol table info available. #29 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 No symbol table info available. Thread 18 (Thread 0x7f1b0ffff700 (LWP 105178)): #0 0x00007f1b48dc56d5 in <a href="mailto:pthread_cond_wait@@GLIBC_2.3.2" target="_blank">pthread_cond_wait@@GLIBC_2.3.2</a> () from /lib64/libpthread.so.0 No symbol table info available. #1 0x00007f1b4941b050 in PR_WaitCondVar () from /lib64/libnspr4.so No symbol table info available. #2 0x00007f1b4b70efee in connection_wait_for_new_work () No symbol table info available. #3 0x00007f1b4b710719 in connection_threadmain () No symbol table info available. #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so No symbol table info available. #5 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 No symbol table info available. #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 No symbol table info available. Thread 17 (Thread 0x7f1b14d28700 (LWP 105177)): #0 0x00007f1b48dc56d5 in <a href="mailto:pthread_cond_wait@@GLIBC_2.3.2" target="_blank">pthread_cond_wait@@GLIBC_2.3.2</a> () from /lib64/libpthread.so.0 No symbol table info available. #1 0x00007f1b4941b050 in PR_WaitCondVar () from /lib64/libnspr4.so No symbol table info available. #2 0x00007f1b4b70efee in connection_wait_for_new_work () No symbol table info available. #3 0x00007f1b4b710719 in connection_threadmain () No symbol table info available. #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so No symbol table info available. #5 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 No symbol table info available. #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 No symbol table info available. Thread 16 (Thread 0x7f1b15529700 (LWP 105176)): #0 0x00007f1b48dc56d5 in <a href="mailto:pthread_cond_wait@@GLIBC_2.3.2" target="_blank">pthread_cond_wait@@GLIBC_2.3.2</a> () from /lib64/libpthread.so.0 No symbol table info available. #1 0x00007f1b4941b050 in PR_WaitCondVar () from /lib64/libnspr4.so No symbol table info available. #2 0x00007f1b4b70efee in connection_wait_for_new_work () No symbol table info available. #3 0x00007f1b4b710719 in connection_threadmain () No symbol table info available. #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so No symbol table info available. #5 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 No symbol table info available. #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 No symbol table info available. Thread 15 (Thread 0x7f1b15d2a700 (LWP 105175)): #0 0x00007f1b48dc56d5 in <a href="mailto:pthread_cond_wait@@GLIBC_2.3.2" target="_blank">pthread_cond_wait@@GLIBC_2.3.2</a> () from /lib64/libpthread.so.0 No symbol table info available. #1 0x00007f1b4941b050 in PR_WaitCondVar () from /lib64/libnspr4.so No symbol table info available. #2 0x00007f1b4b70efee in connection_wait_for_new_work () No symbol table info available. #3 0x00007f1b4b710719 in connection_threadmain () No symbol table info available. #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so No symbol table info available. #5 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 No symbol table info available. #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 No symbol table info available. Thread 14 (Thread 0x7f1b1652b700 (LWP 105174)): #0 0x00007f1b48dc56d5 in <a href="mailto:pthread_cond_wait@@GLIBC_2.3.2" target="_blank">pthread_cond_wait@@GLIBC_2.3.2</a> () from /lib64/libpthread.so.0 No symbol table info available. #1 0x00007f1b4941b050 in PR_WaitCondVar () from /lib64/libnspr4.so No symbol table info available. #2 0x00007f1b4b70efee in connection_wait_for_new_work () No symbol table info available. #3 0x00007f1b4b710719 in connection_threadmain () No symbol table info available. #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so No symbol table info available. #5 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 No symbol table info available. #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 No symbol table info available. Thread 13 (Thread 0x7f1b16d2c700 (LWP 105173)): #0 0x00007f1b48dc56d5 in <a href="mailto:pthread_cond_wait@@GLIBC_2.3.2" target="_blank">pthread_cond_wait@@GLIBC_2.3.2</a> () from /lib64/libpthread.so.0 No symbol table info available. #1 0x00007f1b433212f3 in __db_hybrid_mutex_suspend () from /lib64/<a href="http://libdb-5.3.so" target="_blank">libdb-5.3.so</a> No symbol table info available. #2 0x00007f1b43320640 in __db_tas_mutex_lock () from /lib64/<a href="http://libdb-5.3.so" target="_blank">libdb-5.3.so</a> No symbol table info available. #3 0x00007f1b433cad3a in __lock_get_internal () from /lib64/<a href="http://libdb-5.3.so" target="_blank">libdb-5.3.so</a> No symbol table info available. #4 0x00007f1b433cb820 in __lock_get () from /lib64/<a href="http://libdb-5.3.so" target="_blank">libdb-5.3.so</a> No symbol table info available. #5 0x00007f1b433f7142 in __db_lget () from /lib64/<a href="http://libdb-5.3.so" target="_blank">libdb-5.3.so</a> No symbol table info available. #6 0x00007f1b4333e605 in __bam_search () from /lib64/<a href="http://libdb-5.3.so" target="_blank">libdb-5.3.so</a> No symbol table info available. #7 0x00007f1b43329256 in __bamc_search () from /lib64/<a href="http://libdb-5.3.so" target="_blank">libdb-5.3.so</a> No symbol table info available. #8 0x00007f1b4332ad0f in __bamc_get () from /lib64/<a href="http://libdb-5.3.so" target="_blank">libdb-5.3.so</a> No symbol table info available. #9 0x00007f1b433e3ca6 in __dbc_iget () from /lib64/<a href="http://libdb-5.3.so" target="_blank">libdb-5.3.so</a> No symbol table info available. #10 0x00007f1b433f2b02 in __dbc_get_pp () from /lib64/<a href="http://libdb-5.3.so" target="_blank">libdb-5.3.so</a> No symbol table info available. #11 0x00007f1b3e8ce680 in idl_new_fetch () from /usr/lib64/dirsrv/plugins/libback-ldbm.so No symbol table info available. #12 0x00007f1b3e8dcb26 in index_read_ext_allids () from /usr/lib64/dirsrv/plugins/libback-ldbm.so No symbol table info available. #13 0x00007f1b3e8c7284 in keys2idl () from /usr/lib64/dirsrv/plugins/libback-ldbm.so No symbol table info available. #14 0x00007f1b3e8c79e3 in ava_candidates.isra.0 () from /usr/lib64/dirsrv/plugins/libback-ldbm.so No symbol table info available. #15 0x00007f1b3e8c7fd2 in filter_candidates_ext () from /usr/lib64/dirsrv/plugins/libback-ldbm.so No symbol table info available. #16 0x00007f1b3e8c9096 in list_candidates () from /usr/lib64/dirsrv/plugins/libback-ldbm.so No symbol table info available. #17 0x00007f1b3e8c7f40 in filter_candidates_ext () from /usr/lib64/dirsrv/plugins/libback-ldbm.so No symbol table info available. #18 0x00007f1b3e8c9096 in list_candidates () from /usr/lib64/dirsrv/plugins/libback-ldbm.so No symbol table info available. #19 0x00007f1b3e8c7f40 in filter_candidates_ext () from /usr/lib64/dirsrv/plugins/libback-ldbm.so No symbol table info available. #20 0x00007f1b3e8c9096 in list_candidates () from /usr/lib64/dirsrv/plugins/libback-ldbm.so No symbol table info available. #21 0x00007f1b3e8c7f40 in filter_candidates_ext () from /usr/lib64/dirsrv/plugins/libback-ldbm.so No symbol table info available. #22 0x00007f1b3e90380c in subtree_candidates () from /usr/lib64/dirsrv/plugins/libback-ldbm.so No symbol table info available. #23 0x00007f1b3e904e7f in ldbm_back_search () from /usr/lib64/dirsrv/plugins/libback-ldbm.so No symbol table info available. #24 0x00007f1b4b2321d6 in op_shared_search () from /usr/lib64/dirsrv/libslapd.so.0 No symbol table info available. #25 0x00007f1b4b72333e in do_search () No symbol table info available. #26 0x00007f1b4b7113c3 in connection_threadmain () No symbol table info available. #27 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so No symbol table info available. #28 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 No symbol table info available. #29 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 No symbol table info available. Thread 12 (Thread 0x7f1b1752d700 (LWP 105172)): #0 0x00007f1b48dc56d5 in <a href="mailto:pthread_cond_wait@@GLIBC_2.3.2" target="_blank">pthread_cond_wait@@GLIBC_2.3.2</a> () from /lib64/libpthread.so.0 No symbol table info available. #1 0x00007f1b4941b050 in PR_WaitCondVar () from /lib64/libnspr4.so No symbol table info available. #2 0x00007f1b4b70efee in connection_wait_for_new_work () No symbol table info available. #3 0x00007f1b4b710719 in connection_threadmain () No symbol table info available. #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so No symbol table info available. #5 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 No symbol table info available. #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 No symbol table info available. Thread 11 (Thread 0x7f1b17d2e700 (LWP 105171)): #0 0x00007f1b48dc56d5 in <a href="mailto:pthread_cond_wait@@GLIBC_2.3.2" target="_blank">pthread_cond_wait@@GLIBC_2.3.2</a> () from /lib64/libpthread.so.0 No symbol table info available. #1 0x00007f1b4941b050 in PR_WaitCondVar () from /lib64/libnspr4.so No symbol table info available. #2 0x00007f1b4b70efee in connection_wait_for_new_work () No symbol table info available. #3 0x00007f1b4b710719 in connection_threadmain () No symbol table info available. #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so No symbol table info available. #5 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 No symbol table info available. #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 No symbol table info available. Thread 10 (Thread 0x7f1b2509d700 (LWP 105168)): #0 0x00007f1b48dc5a82 in <a href="mailto:pthread_cond_timedwait@@GLIBC_2.3.2" target="_blank">pthread_cond_timedwait@@GLIBC_2.3.2</a> () from /lib64/libpthread.so.0 No symbol table info available. #1 0x00007f1b4941ab07 in pt_TimedWait () from /lib64/libnspr4.so No symbol table info available. #2 0x00007f1b4941afce in PR_WaitCondVar () from /lib64/libnspr4.so No symbol table info available. #3 0x00007f1b4b2061e8 in eq_loop () from /usr/lib64/dirsrv/libslapd.so.0 No symbol table info available. #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so No symbol table info available. #5 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 No symbol table info available. #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 No symbol table info available. Thread 9 (Thread 0x7f1b2589e700 (LWP 105167)): #0 0x00007f1b48dc5a82 in <a href="mailto:pthread_cond_timedwait@@GLIBC_2.3.2" target="_blank">pthread_cond_timedwait@@GLIBC_2.3.2</a> () from /lib64/libpthread.so.0 No symbol table info available. #1 0x00007f1b4941ab07 in pt_TimedWait () from /lib64/libnspr4.so No symbol table info available. #2 0x00007f1b4941afce in PR_WaitCondVar () from /lib64/libnspr4.so No symbol table info available. #3 0x00007f1b4b719d03 in housecleaning () No symbol table info available. #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so No symbol table info available. #5 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 No symbol table info available. #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 No symbol table info available. Thread 8 (Thread 0x7f1b2609f700 (LWP 105166)): #0 0x00007f1b48dc56d5 in <a href="mailto:pthread_cond_wait@@GLIBC_2.3.2" target="_blank">pthread_cond_wait@@GLIBC_2.3.2</a> () from /lib64/libpthread.so.0 No symbol table info available. #1 0x00007f1b4941b050 in PR_WaitCondVar () from /lib64/libnspr4.so No symbol table info available. #2 0x00007f1b4b260198 in slapi_wait_condvar () from /usr/lib64/dirsrv/libslapd.so.0 No symbol table info available. #3 0x00007f1b3cf5deed in roles_cache_wait_on_change () from /usr/lib64/dirsrv/plugins/libroles-plugin.so No symbol table info available. #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so No symbol table info available. #5 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 No symbol table info available. #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 No symbol table info available. Thread 7 (Thread 0x7f1b268a0700 (LWP 105165)): #0 0x00007f1b48dc56d5 in <a href="mailto:pthread_cond_wait@@GLIBC_2.3.2" target="_blank">pthread_cond_wait@@GLIBC_2.3.2</a> () from /lib64/libpthread.so.0 No symbol table info available. #1 0x00007f1b4941b050 in PR_WaitCondVar () from /lib64/libnspr4.so No symbol table info available. #2 0x00007f1b4b260198 in slapi_wait_condvar () from /usr/lib64/dirsrv/libslapd.so.0 No symbol table info available. #3 0x00007f1b3cf5deed in roles_cache_wait_on_change () from /usr/lib64/dirsrv/plugins/libroles-plugin.so No symbol table info available. #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so No symbol table info available. #5 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 No symbol table info available. #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 No symbol table info available. Thread 6 (Thread 0x7f1b270a1700 (LWP 105164)): #0 0x00007f1b48dc56d5 in <a href="mailto:pthread_cond_wait@@GLIBC_2.3.2" target="_blank">pthread_cond_wait@@GLIBC_2.3.2</a> () from /lib64/libpthread.so.0 No symbol table info available. #1 0x00007f1b4941b050 in PR_WaitCondVar () from /lib64/libnspr4.so No symbol table info available. #2 0x00007f1b4b260198 in slapi_wait_condvar () from /usr/lib64/dirsrv/libslapd.so.0 No symbol table info available. #3 0x00007f1b3cf5deed in roles_cache_wait_on_change () from /usr/lib64/dirsrv/plugins/libroles-plugin.so No symbol table info available. #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so No symbol table info available. #5 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 No symbol table info available. #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 No symbol table info available. Thread 5 (Thread 0x7f1b278a2700 (LWP 105163)): #0 0x00007f1b48dc56d5 in <a href="mailto:pthread_cond_wait@@GLIBC_2.3.2" target="_blank">pthread_cond_wait@@GLIBC_2.3.2</a> () from /lib64/libpthread.so.0 No symbol table info available. #1 0x00007f1b4941b050 in PR_WaitCondVar () from /lib64/libnspr4.so No symbol table info available. #2 0x00007f1b4b260198 in slapi_wait_condvar () from /usr/lib64/dirsrv/libslapd.so.0 No symbol table info available. #3 0x00007f1b415f262e in cos_cache_wait_on_change () from /usr/lib64/dirsrv/plugins/libcos-plugin.so No symbol table info available. #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so No symbol table info available. #5 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 No symbol table info available. #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 No symbol table info available. Thread 4 (Thread 0x7f1b282ac700 (LWP 105162)): #0 0x00007f1b48ae69b3 in select () from /lib64/libc.so.6 No symbol table info available. #1 0x00007f1b4b2710e9 in DS_Sleep () from /usr/lib64/dirsrv/libslapd.so.0 No symbol table info available. #2 0x00007f1b3e8ba707 in perf_threadmain () from /usr/lib64/dirsrv/plugins/libback-ldbm.so No symbol table info available. #3 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so No symbol table info available. #4 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 No symbol table info available. #5 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 No symbol table info available. Thread 3 (Thread 0x7f1b28aad700 (LWP 105161)): #0 0x00007f1b48ae69b3 in select () from /lib64/libc.so.6 No symbol table info available. #1 0x00007f1b4b2710e9 in DS_Sleep () from /usr/lib64/dirsrv/libslapd.so.0 No symbol table info available. #2 0x00007f1b3e8bfb7f in trickle_threadmain () from /usr/lib64/dirsrv/plugins/libback-ldbm.so No symbol table info available. #3 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so No symbol table info available. #4 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 No symbol table info available. #5 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 No symbol table info available. Thread 2 (Thread 0x7f1b292ae700 (LWP 105160)): #0 0x00007f1b48ae69b3 in select () from /lib64/libc.so.6 No symbol table info available. #1 0x00007f1b4344dbad in __os_yield () from /lib64/<a href="http://libdb-5.3.so" target="_blank">libdb-5.3.so</a> No symbol table info available. #2 0x00007f1b434492b3 in __memp_sync_int () from /lib64/<a href="http://libdb-5.3.so" target="_blank">libdb-5.3.so</a> No symbol table info available. #3 0x00007f1b43459752 in __txn_checkpoint () from /lib64/<a href="http://libdb-5.3.so" target="_blank">libdb-5.3.so</a> No symbol table info available. #4 0x00007f1b43459b74 in __txn_checkpoint_pp () from /lib64/<a href="http://libdb-5.3.so" target="_blank">libdb-5.3.so</a> No symbol table info available. #5 0x00007f1b3e8c3a87 in checkpoint_threadmain () from /usr/lib64/dirsrv/plugins/libback-ldbm.so No symbol table info available. #6 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so No symbol table info available. #7 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 No symbol table info available. #8 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 No symbol table info available. Thread 1 (Thread 0x7f1b29aaf700 (LWP 105159)): #0 0x00007f1b48ae69b3 in select () from /lib64/libc.so.6 No symbol table info available. #1 0x00007f1b4b2710e9 in DS_Sleep () from /usr/lib64/dirsrv/libslapd.so.0 No symbol table info available. #2 0x00007f1b3e8bf907 in deadlock_threadmain () from /usr/lib64/dirsrv/plugins/libback-ldbm.so No symbol table info available. #3 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so No symbol table info available. #4 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 No symbol table info available. #5 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 No symbol table info available. </div> Thanks, </div> Rakesh <div> <div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote">On Mon, Sep 5, 2016 at 4:48 PM, thierry bordaz <<a href="mailto:tbordaz@redhat.com" target="_blank">tbordaz@redhat.com</a>> wrote: <blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"> <div text="#000000" bgcolor="#FFFFFF"> <div>On 09/05/2016 12:05 PM, Rakesh Rajasekharan wrote: </div> <blockquote type="cite"> <div dir="ltr"> <div> <div> <div> <div> <div>Hi Thierry, </div> <div>I was getting the hang issue while running ipa-client-install simultaneously on few clients.. </div> <div>However, today, I am not able to replicate that. </div> <div> </div> <div>I could not get a gdb . But i will try getting that the next time I face this issue. </div> The CPU does not stay high.. it just momentarily touches a high value and then drops down to around 2-7% </div> One question I have is , is it ok to set it nsslapd-threadnumber to a very high value . </div> I have around 4000 clients and with nsslapd-maxthreadsperconn set to 5..So, can I set nsslapd-threadnumber to around 25000. </div> </div> </div> </blockquote> Hello, I know some users running in production with several hunderds of threads (>600) and this without problem. I do not recall having suggested to increase that number and for what reason. Usually 30 workers is a good enough value. It can create bootleneck if for some reason each operation is very long to satisfy and exhaust the number of workers. You can monitor the work queue: <blockquote><tt>ldapsearch -D "cn=directory manager" -w xxx -LLL -b "cn=monitor" -s base opsinitiated opscompleted</tt> </blockquote> If opscompleted-opsinitiated remains close to threadnumber, then yes it would be valuable to increase it. The computation #client * #async_op_per_client sound an overkill. Even if all clients send at the exact same time all their requests, it is very likely that some common resource (db page, log, allocator...) will serialize them. If you monitor a need to increase the work, you would for example set it to 50, then monitor, then set it to 100, then monitor... until you find a good enough value. Note the increasing the #thread, increases the memory footprint that will reduce the efficiency of file system cache and can increase the response time. best regards thierry <blockquote type="cite"> <div dir="ltr"> <div> </div> Thanks </div> <div class="gmail_extra"> <div class="gmail_quote">On Mon, Sep 5, 2016 at 1:03 PM, thierry bordaz <<a href="mailto:tbordaz@redhat.com" target="_blank">tbordaz@redhat.com</a>> wrote: <blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"> <div text="#000000" bgcolor="#FFFFFF"> Hi Rakesh, Were you able to get a pstack or full stack with gdb (<a href="http://www.port389.org/docs/" target="_blank">http://www.port389.org/docs/</a>389ds/FAQ/faq.html#debugging-crashes) when the server hangs ? If it happens with 500 threads as well as with 30, using 30 threads is a better choice to debug this issue. I will try to reproduce using 150 parallel 'ipa user-find p-testipa' commands Something I am unsure is if the CPU consumption stays high (you mentioned 340% CPU usage) as long as the hang happens or if after a suddent shot up to 340% (that marks the beginning of the hang) it drops and stay hanging ? thanks thierry <div>On 09/04/2016 08:40 PM, Rakesh Rajasekharan wrote: </div> <blockquote type="cite"> <div dir="ltr"> <div>starce on the slapd process actually had this in the output.. FUTEX_WAIT_PRIVATE </div> and checking for the number of threads slapd had.. there were 5015 threads <div> ps -efL|grep slapd|wc -l 5015 strace on most of the threads gave this output strace -p 67411 Process 67411 attached futex(0x7f3f0226b41c, FUTEX_WAIT_PRIVATE, 1, NULL) = -1 EAGAIN (Resource temporarily unavailable) futex(0x7f3f0226b41c, FUTEX_WAIT_PRIVATE, 2, NULL^CProcess 67411 detached </div> </div> <div class="gmail_extra"> <div class="gmail_quote">On Sun, Sep 4, 2016 at 5:34 PM, Rakesh Rajasekharan <<a href="mailto:rakesh.rajasekharan@gmail.com" target="_blank">rakesh.rajasekharan@gmail.com</a>> wrote: <blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"> <div dir="ltr"> <div> <div> <div> <div> <div>I have again got the issue of IPA hanging.. The issue came up when i tried to run ipa-client-isntall on 142 clients simultaneously </div> <div>None of the IPA commands are responding, and I see this error ipa user-find p-testipa ipa: ERROR: Insufficient access: SASL(-1): generic failure: GSSAPI Error: Unspecified GSS failure. Minor code may provide more information (KDC returned error string: PROCESS_TGS) KRB5_TRACE=/dev/stdout kinit admin [41178] 1472984115.233214: Getting initial credentials for <a href="mailto:admin@XYZ.COM" target="_blank">admin@XYZ.COM</a> [41178] 1472984115.235257: Sending request (167 bytes) to <a href="http://XYZ.COM" target="_blank">XYZ.COM</a> [41178] 1472984115.235419: Initiating TCP connection to stream <a href="http://10.1.3.36:88" target="_blank">10.1.3.36:88</a> [41178] 1472984115.235685: Sending TCP request to stream <a href="http://10.1.3.36:88" target="_blank">10.1.3.36:88</a> [41178] 1472984120.238914: Received answer (174 bytes) from stream <a href="http://10.1.3.36:88" target="_blank">10.1.3.36:88</a> [41178] 1472984120.238925: Terminating TCP connection to stream <a href="http://10.1.3.36:88" target="_blank">10.1.3.36:88</a> [41178] 1472984120.238993: Response was from master KDC </div> <div>[41 </div> <div>Running an ldapsearch to see the db.. does not give any results and just hangs there ldapsearch -x -D 'cn=Directory Manager' -W -s one -b 'cn=kerberos,dc=xyz,dc=com' Enter LDAP Password: </div> <div>even an ldapsearch -x does not respond </div> <div>At this point, am sure that slapd is the one causing issues </div> <div> </div> <div>Running an strace against the hung slapd itself seems to get stuck does not proceed after saying "attaching to process" </div> <div>From some others posts I read Thierry suggesting to increase the nsslapd-threadnumber value </div> <div>It was set to 30, I think that might be too low. </div> </div> </div> </div> </div> </div> </blockquote> </div> </div> </blockquote> </div> </blockquote> </div> </div> </blockquote> </div> </blockquote> </div> </div> </blockquote> </blockquote> </div> </div> </div> </div> </div> ... [Message clipped] </blockquote> </div> </div> </blockquote> </div> </div> </blockquote> </blockquote> <pre cols="72">-- Red Hat GmbH, <a href="http://www.de.redhat.com/" target="_blank">http://www.de.redhat.com/</a>, Registered seat: Grasbrunn, Commercial register: Amtsgericht Muenchen, HRB 153243, Managing Directors: Charles Cachera, Michael Cunningham, Michael O'Neill, Eric Shander</pre> </div> </blockquote></div> </div>