From abokovoy at redhat.com Thu Sep 1 03:25:39 2016 From: abokovoy at redhat.com (Alexander Bokovoy) Date: Thu, 1 Sep 2016 06:25:39 +0300 Subject: [Freeipa-users] pfSense/FreeIPA LDAP Extended Query Fails In-Reply-To: <34BE6DEF-B105-45E0-BA21-EEC47416FE2A@flowjo.com> References: <34BE6DEF-B105-45E0-BA21-EEC47416FE2A@flowjo.com> Message-ID: <20160901032538.xmw4gkidnkf54gnu@redhat.com> On Wed, 31 Aug 2016, Mike Jacobacci wrote: >Hi, > >I have just got authentication against my FreeIPA system working by >following this: https://ask.fedoraproject.org/en/que...uthentication/ > > >The only change I had to make was to set the Search Scope level to >"entire subtree" and I also left the extended query unchecked... With >that setup I am able to authenticate using >"Diagnostics->Authentication". > >I really want to restrict access so I can use FreeIPA for our VPN auth >so I tried using the following extended query but it fails: >&(memberOf=cn=admins,cn=groups,cn=accounts,dc=doma in,dc=com) > >Looking in pfSense logs, using the extended query (fails): > >[24/Aug/2016:11:07:16 -0700] conn=1396 fd=116 slot=116 SSL connection from * to * >[24/Aug/2016:11:07:16 -0700] conn=1396 TLS1.2 256-bit AES-GCM >[24/Aug/2016:11:07:16 -0700] conn=1396 op=0 BIND dn="" method=128 version=3 >[24/Aug/2016:11:07:16 -0700] conn=1396 op=0 RESULT err=0 tag=97 nentries=0 etime=0 dn="" >[24/Aug/2016:11:07:16 -0700] conn=1396 op=1 SRCH base="cn=accounts,dc=domain,dc=com" scope=2 filter="(&(uid=user)(&(memberOf=cn=admins,cn=group s,cn=accounts,dc=domain,dc=com)))" attrs=ALL >[24/Aug/2016:11:07:16 -0700] conn=1396 op=1 RESULT err=0 tag=101 nentries=0 etime=0 >[24/Aug/2016:11:07:16 -0700] conn=1396 op=2 UNBIND >[24/Aug/2016:11:07:16 -0700] conn=1396 op=2 fd=116 closed - U1 > >Without the query (success): >[30/Aug/2016:10:23:25 -0700] conn=6432 fd=110 slot=110 SSL connection from * to * >[30/Aug/2016:10:23:25 -0700] conn=6432 TLS1.2 256-bit AES-GCM >[30/Aug/2016:10:23:25 -0700] conn=6432 op=0 BIND dn="" method=128 version=3 >[30/Aug/2016:10:23:25 -0700] conn=6432 op=0 RESULT err=0 tag=97 nentries=0 etime=0 dn="" >[30/Aug/2016:10:23:25 -0700] conn=6432 op=1 SRCH base="cn=compat,dc=domain,dc=com" scope=2 filter="(uid=user1)? attrs=ALL >[30/Aug/2016:10:23:25 -0700] conn=6432 op=1 RESULT err=0 tag=101 nentries=1 etime=0 >[30/Aug/2016:10:23:25 -0700] conn=6432 op=2 BIND dn="uid=user1,cn=users,cn=compat,dc=domain,dc=com " method=128 version=3 >[30/Aug/2016:10:23:25 -0700] conn=6432 op=2 RESULT err=0 tag=97 nentries=0 etime=0 dn="uid=user1,cn=users,cn=accounts,dc=domain,dc=co m" >[30/Aug/2016:10:23:25 -0700] conn=6433 fd=118 slot=118 SSL connection from * to * >[30/Aug/2016:10:23:25 -0700] conn=6432 op=3 UNBIND >[30/Aug/2016:10:23:25 -0700] conn=6432 op=3 fd=110 closed - U1 >[30/Aug/2016:10:23:25 -0700] conn=6433 TLS1.2 256-bit AES-GCM >[30/Aug/2016:10:23:25 -0700] conn=6433 op=0 BIND dn="" method=128 version=3 >[30/Aug/2016:10:23:25 -0700] conn=6433 op=0 RESULT err=0 tag=97 nentries=0 etime=0 dn="" >[30/Aug/2016:10:23:25 -0700] conn=6433 op=1 SRCH base="uid=user1,cn=users,cn=compat,dc=domain,dc=co m" scope=2 filter="(uid=user1)? attrs="memberOf" >[30/Aug/2016:10:23:25 -0700] conn=6433 op=1 RESULT err=0 tag=101 nentries=1 etime=0 >[30/Aug/2016:10:23:25 -0700] conn=6433 op=2 UNBIND >[30/Aug/2016:10:23:25 -0700] conn=6433 op=2 fd=118 closed - U1 > >I changed the cn from accounts to compat for the auth container, but >that doesn't make a difference. The last search shows attrs="memberOf", >but anytime I add an extended query the logs show attrs="all", not sure >if that means anything. I tried adding the full memberOf path under the >group member attribute, but that didn't restrict access although the >auth is still success. > >[30/Aug/2016:10:42:12 -0700] conn=6460 op=1 SRCH base="uid=user3,cn=users,cn=compat,dc=domain,dc=co m" scope=2 filter="(uid=user3)" attrs="memberof=cn=admins,cn=groups,cn=compat,dc=d omain,dc=com" >[30/Aug/2016:10:42:12 -0700] conn=6460 op=1 RESULT err=0 tag=101 nentries=1 etime=0 > >When doing an ldapsearch, I can see the group: > ># admins, groups, compat, domain.com >dn: cn=admins,cn=groups,cn=compat,dc=domain,dc=com >ipaAnchorUUID:: >gidNumber: 50000 >memberUid: admin >memberUid: user1 >memberUid: user2 >objectClass: posixGroup >objectClass: ipaOverrideTarget >objectClass: ipaexternalgroup >objectClass: top >cn: admins > >Any help would be greatly appreciated. FreeIPA 4.x requires authenticated bind to be able to see member attributes of the groups in the main subtree. Your pfSense is using anonymous bind, thus not being able to see them. Also, don't use cn=compat,$suffix subtree, it does not help for your task. Your pfSense device expects different schema than the one provided by the Compatibility Tree. -- / Alexander Bokovoy From a.rogovsky at gmail.com Thu Sep 1 03:55:50 2016 From: a.rogovsky at gmail.com (Andrey Rogovsky) Date: Thu, 1 Sep 2016 06:55:50 +0300 Subject: [Freeipa-users] Command-line replication is not works in FreeIPA-Master In-Reply-To: <4e37eb09-8c96-114a-aee1-3c6ee568194f@redhat.com> References: <4e37eb09-8c96-114a-aee1-3c6ee568194f@redhat.com> Message-ID: Hi! Thanks for your advices! I'm try start replica and get this errors in log: [01/Sep/2016:03:24:23 +0000] slapi_ldap_bind - Error: could not bind id [cn=replication manager,cn=config] authentication mechanism [SIMPLE]: error 32 (No such object) errno 0 (Success) [01/Sep/2016:03:24:23 +0000] NSMMReplicationPlugin - agmt="cn=ExampleAgreement" (ldap2:389): Replication bind with SIMPLE auth failed: LDAP error 32 (No such object) () This is my current replica: filter: (objectclass=nsds5replica) requesting: All userApplication attributes # extended LDIF # # LDAPv3 # base with scope subtree # filter: (objectclass=nsds5replica) # requesting: ALL # # replica, dc\3Dexample\2Cdc\3Dcom, mapping tree, config dn: cn=replica,cn=dc\3Dexample\2Cdc\3Dcom,cn=mapping tree,cn=config objectClass: top objectClass: nsds5replica objectClass: extensibleObject cn: replica nsDS5ReplicaRoot: dc=example,dc=com nsDS5ReplicaId: 7 nsDS5ReplicaType: 3 nsDS5Flags: 1 nsds5ReplicaPurgeDelay: 604800 nsDS5ReplicaBindDN: cn=replication manager,cn=config nsState:: BwAAAAAAAADqnMdXAAAAAAAAAAAAAAAAAAAAAAAAAAABAAAAAAAAAA== nsDS5ReplicaName: 496dba82-6f7a11e6-9d5ba359-5196ffe4 nsds5ReplicaChangeCount: 118 nsds5replicareapactive: 0 # search result search: 2 result: 0 Success # numResponses: 2 # numEntries: 1 This is my current agreement: # extended LDIF # # LDAPv3 # base with scope subtree # filter: (objectclass=nsds5ReplicationAgreement) # requesting: ALL # # ExampleAgreement, replica, dc\3Dexample\2Cdc\3Dcom, mapping tree, config dn: cn=ExampleAgreement,cn=replica,cn=dc\3Dexample\2Cdc\3Dcom,cn=mapping tree, cn=config objectClass: top objectClass: nsds5replicationagreement cn: ExampleAgreement nsDS5ReplicaHost: ldap2 nsDS5ReplicaPort: 389 nsDS5ReplicaBindDN: cn=replication manager,cn=config nsDS5ReplicaBindMethod: SIMPLE nsDS5ReplicaRoot: dc=example,dc=com description: agreement between supplier1 and consumer1 nsDS5ReplicaUpdateSchedule: 0000-0500 1 nsDS5ReplicatedAttributeList: (objectclass=*) $ EXCLUDE authorityRevocationLis t nsDS5ReplicaCredentials: {AES-TUhNR0NTcUdTSWIzRFFFRkRUQm1NRVVHQ1NxR1NJYjNEUUVG RERBNEJDUmxPVFl4TlRsbU5DMWtaV0UyTXpZeA0KTVMxaU1UYzFaREF3Wmkwek5qRmxNalkxWkFBQ 0FRSUNBU0F3Q2dZSUtvWklodmNOQWdjd0hRWUpZSVpJQVdVRA0KQkFFcUJCQU1Dc25vTkVzZVJ4b3 N2WVlEMXRpbQ==}a21h3uqnbcAZ1cX+NheCeg== nsds5replicareapactive: 0 nsds5replicaLastUpdateStart: 19700101000000Z nsds5replicaLastUpdateEnd: 19700101000000Z nsds5replicaChangesSentSinceStartup: nsds5replicaLastUpdateStatus: 0 No replication sessions started since server s tartup nsds5replicaUpdateInProgress: FALSE nsds5replicaLastInitStart: 20160901032423Z nsds5replicaLastInitEnd: 19700101000000Z nsds5replicaLastInitStatus: 32 - LDAP error: No such object # search result search: 2 result: 0 Success # numResponses: 2 # numEntries: 1 I'm try delete agreement, replica, user, changelog and create again. This not help, same error: [01/Sep/2016:03:42:37 +0000] NSMMReplicationPlugin - agmt_delete: begin [01/Sep/2016:03:45:35 +0000] NSMMReplicationPlugin - replica_config_delete: Warning: The changelog for replica dc=example,dc=com is no longer valid since the replica config is being deleted. Removing the changelog. [01/Sep/2016:03:53:18 +0000] slapi_ldap_bind - Error: could not bind id [cn=replication manager,cn=config] authentication mechanism [SIMPLE]: error 32 (No such object) errno 0 (Success) [01/Sep/2016:03:53:18 +0000] NSMMReplicationPlugin - agmt="cn=ExampleAgreement" (ldap2:389): Replication bind with SIMPLE auth failed: LDAP error 32 (No such object) () 2016-08-31 20:09 GMT+03:00 Mark Reynolds : > > > On 08/31/2016 12:39 PM, Andrey Rogovsky wrote: > > Hi, Mark! > > Thanks for explain. Now I create replication manager: (I hope) > [root at ldap1 ~]# ldapsearch -h ldap1.example.com -p 389 -xLLL -D > "cn=directory manager" -W -b cn=config "cn=replication manager" > Enter LDAP Password: > dn: cn=replication manager,cn=config > objectClass: inetorgperson > objectClass: person > objectClass: top > objectClass: organizationalPerson > cn: replication manager > sn: RM > userPassword:: e1NTSEF9N1JiRmNXWTFXNDA1cmdYSU > dCNWJtV3RzOElNQXBhakhXam94WlE9PQ= > = > > What is next? I use manual from 8 version and this a bit obsoleted. > > Now you should be able to initialize your standalone server by updating > the agreement on the ipa DS: > > dn: cn=ExampleAgreement,cn=replica,cn="dc=example,dc=com",cn=mapping > tree,cn=config > changetype: modify > replace: nsds5beginreplicarefresh > nsds5beginreplicarefresh: start > > If something goes wrong let us know what's in the errors log again. > > Mark > > > > 2016-08-31 19:30 GMT+03:00 Mark Reynolds : > >> Hi Andrey, >> >> It looks like you still did not create the replication manager entry. >> You must create that manager entry on the standalone server. Please read >> the link I sent you: >> >> https://access.redhat.com/documentation/en-US/Red_Hat_Direct >> ory_Server/10/html/Administration_Guide/Creating_the_Supplie >> r_Bind_DN_Entry.html >> >> You can verify its existence by doing this search against the standalone >> server: >> >> ldapsearch -h ldap1.example.com -p 389 -xLLL -D "cn=directory manager" >> -W -b cn=config "cn=replication manager" >> >> Mark >> >> >> On 08/31/2016 11:50 AM, Andrey Rogovsky wrote: >> >> Hi! >> Thank you for fast reply. >> Yes, I want use standalone 389DS to replica from FreeIPA. >> There is my replica: >> filter: (objectclass=nsds5replica) >> requesting: All userApplication attributes >> # extended LDIF >> # >> # LDAPv3 >> # base with scope subtree >> # filter: (objectclass=nsds5replica) >> # requesting: ALL >> # >> >> # replica, dc\3Dexample\2Cdc\3Dcom, mapping tree, config >> dn: cn=replica,cn=dc\3Dexample\2Cdc\3Dcom,cn=mapping tree,cn=config >> objectClass: top >> objectClass: nsds5replica >> objectClass: extensibleObject >> cn: replica >> nsDS5ReplicaRoot: dc=example,dc=com >> nsDS5ReplicaId: 7 >> nsDS5ReplicaType: 3 >> nsDS5Flags: 1 >> nsds5ReplicaPurgeDelay: 604800 >> nsDS5ReplicaBindDN: cn=replication manager,cn=config >> nsState:: BwAAAAAAAABZ98ZXAAAAAAAAAAAAAAAAAAAAAAAAAAABAAAAAAAAAA== >> nsDS5ReplicaName: 496dba82-6f7a11e6-9d5ba359-5196ffe4 >> nsds5ReplicaChangeCount: 22 >> nsds5replicareapactive: 0 >> >> # search result >> search: 2 >> result: 0 Success >> >> # numResponses: 2 >> # numEntries: 1 >> >> So, my replica have entry "cn=replication manager" >> >> But I try add entry in agreement. Unforthunalty this is not help, error >> is present: >> [root at ldap1 ~]# ldapmodify -v -h ldap1.example.com -p 389 -D >> "cn=directory manager" -w ... >> ldap_initialize( ldap://ldap1.example.com:389 ) >> dn: cn=ExampleAgreement,cn=replica,cn="dc=example,dc=com",cn=mapping >> tree,cn=config >> changetype: modify >> replace: nsds5ReplicaBindDN >> nsds5ReplicaBindDN: cn=replication manager,cn=config >> replace nsds5ReplicaBindDN: >> cn=replication manager,cn=config >> modifying entry "cn=ExampleAgreement,cn=replic >> a,cn="dc=example,dc=com",cn=mapping tree,cn=config" >> modify complete >> >> [root at ldap1 ~]# tail -f /var/log/dirsrv/slapd-EXAMPLE-COM/errors >> [31/Aug/2016:11:11:09 +0000] schema-compat-plugin - schema-compat-plugin >> tree scan will start in about 5 seconds! >> [31/Aug/2016:11:11:09 +0000] - slapd started. Listening on All >> Interfaces port 389 for LDAP requests >> [31/Aug/2016:11:11:09 +0000] - Listening on All Interfaces port 636 for >> LDAPS requests >> [31/Aug/2016:11:11:09 +0000] - Listening on /var/run/slapd-EXAMPLE-COM.socket >> for LDAPI requests >> [31/Aug/2016:11:11:13 +0000] schema-compat-plugin - warning: no entries >> set up under ou=sudoers,dc=example,dc=com >> [31/Aug/2016:11:11:14 +0000] schema-compat-plugin - warning: no entries >> set up under cn=ng, cn=compat,dc=example,dc=com >> [31/Aug/2016:11:11:14 +0000] schema-compat-plugin - warning: no entries >> set up under cn=computers, cn=compat,dc=example,dc=com >> [31/Aug/2016:11:11:14 +0000] schema-compat-plugin - Finished plugin >> initialization. >> [31/Aug/2016:13:38:01 +0000] slapi_ldap_bind - Error: could not bind id >> [cn=replication manager] authentication mechanism [SIMPLE]: error 32 (No >> such object) errno 0 (Success) >> [31/Aug/2016:13:38:01 +0000] NSMMReplicationPlugin - >> agmt="cn=ExampleAgreement" (ldap2:389): Replication bind with SIMPLE auth >> failed: LDAP error 32 (No such object) () >> ^C >> [root at ldap1 ~]# ldapmodify -v -h ldap1.example.com -p 389 -D >> "cn=directory manager" -w ... >> ldap_initialize( ldap://ldap1.example.com:389 ) >> dn: cn=ExampleAgreement,cn=replica,cn="dc=example,dc=com",cn=mapping >> tree,cn=config >> changetype: modify >> replace: nsds5beginreplicarefresh >> nsds5beginreplicarefresh: start >> replace nsds5beginreplicarefresh: >> start >> modifying entry "cn=ExampleAgreement,cn=replic >> a,cn="dc=example,dc=com",cn=mapping tree,cn=config" >> modify complete >> >> [root at ldap1 ~]# tail -f /var/log/dirsrv/slapd-EXAMPLE-COM/errors >> [31/Aug/2016:11:11:09 +0000] - slapd started. Listening on All >> Interfaces port 389 for LDAP requests >> [31/Aug/2016:11:11:09 +0000] - Listening on All Interfaces port 636 for >> LDAPS requests >> [31/Aug/2016:11:11:09 +0000] - Listening on /var/run/slapd-EXAMPLE-COM.socket >> for LDAPI requests >> [31/Aug/2016:11:11:13 +0000] schema-compat-plugin - warning: no entries >> set up under ou=sudoers,dc=example,dc=com >> [31/Aug/2016:11:11:14 +0000] schema-compat-plugin - warning: no entries >> set up under cn=ng, cn=compat,dc=example,dc=com >> [31/Aug/2016:11:11:14 +0000] schema-compat-plugin - warning: no entries >> set up under cn=computers, cn=compat,dc=example,dc=com >> [31/Aug/2016:11:11:14 +0000] schema-compat-plugin - Finished plugin >> initialization. >> [31/Aug/2016:13:38:01 +0000] slapi_ldap_bind - Error: could not bind id >> [cn=replication manager] authentication mechanism [SIMPLE]: error 32 (No >> such object) errno 0 (Success) >> [31/Aug/2016:13:38:01 +0000] NSMMReplicationPlugin - >> agmt="cn=ExampleAgreement" (ldap2:389): Replication bind with SIMPLE auth >> failed: LDAP error 32 (No such object) () >> [31/Aug/2016:15:48:36 +0000] slapi_ldap_bind - Error: could not bind id >> [cn=replication manager,cn=config] authentication mechanism [SIMPLE]: error >> 32 (No such object) errno 0 (Success) >> ^C >> [root at ldap1 ~]# >> >> >> 2016-08-31 18:15 GMT+03:00 Mark Reynolds : >> >>> >>> >>> On 08/31/2016 09:50 AM, Andrey Rogovsky wrote: >>> >>> Hi! >>> >>> I try configure manual replica from FreeIPA DS to 389 DS. >>> I have two VM: ldap1.example.com and ldap2.example.com >>> I was used this manual https://www.centos.org/ >>> docs/5/html/CDS/ag/8.0/Managing_Replication-Configuring-Repl >>> ication-cmd.html for configure relica >>> >>> There was replica agreement before starting: >>> >>> # extended LDIF >>> # >>> # LDAPv3 >>> # base with scope subtree >>> # filter: (objectclass=nsds5ReplicationAgreement) >>> # requesting: ALL >>> # >>> >>> # ExampleAgreement, replica, dc\3Dexample\2Cdc\3Dcom, mapping tree, >>> config >>> dn: cn=ExampleAgreement,cn=replica,cn=dc\3Dexample\2Cdc\3Dcom,cn=mapping >>> tree, >>> cn=config >>> objectClass: top >>> objectClass: nsds5replicationagreement >>> cn: ExampleAgreement >>> nsDS5ReplicaHost: ldap2 >>> nsDS5ReplicaPort: 389 >>> nsDS5ReplicaBindDN: cn=replication manager >>> nsDS5ReplicaBindMethod: SIMPLE >>> nsDS5ReplicaRoot: dc=example,dc=com >>> description: agreement between supplier1 and consumer1 >>> nsDS5ReplicaUpdateSchedule: 0000-0500 1 >>> nsDS5ReplicatedAttributeList: (objectclass=*) $ EXCLUDE >>> authorityRevocationLis >>> t >>> nsDS5ReplicaCredentials: {AES-TUhNR0NTcUdTSWIzRFFFRkRUQ >>> m1NRVVHQ1NxR1NJYjNEUUVG >>> RERBNEJDUmxPVFl4TlRsbU5DMWtaV0UyTXpZeA0KTVMxaU1UYzFaREF3Wmk >>> wek5qRmxNalkxWkFBQ >>> 0FRSUNBU0F3Q2dZSUtvWklodmNOQWdjd0hRWUpZSVpJQVdVRA0KQkFFcUJC >>> QUVJckpINmE0S3RFYl >>> NhLzkxL01qZg==}Wo+c0XfBnaDhg/a36yguXg== >>> nsds5replicareapactive: 0 >>> nsds5replicaLastUpdateStart: 19700101000000Z >>> nsds5replicaLastUpdateEnd: 19700101000000Z >>> nsds5replicaChangesSentSinceStartup: >>> nsds5replicaLastUpdateStatus: 0 No replication sessions started since >>> server s >>> tartup >>> nsds5replicaUpdateInProgress: FALSE >>> nsds5replicaLastInitStart: 19700101000000Z >>> nsds5replicaLastInitEnd: 19700101000000Z >>> >>> # search result >>> search: 2 >>> result: 0 Success >>> >>> # numResponses: 2 >>> # numEntries: >>> >>> >>> There is errors which I get when start replica: >>> >>> >>> [root at ldap1 ~]# ldapmodify -v -h ldap1.example.com -p 389 -D >>> "cn=directory manager" -w ... >>> ldap_initialize( ldap://ldap1.example.com:389 ) >>> dn: cn=ExampleAgreement,cn=replica,cn="dc=example,dc=com",cn=mapping >>> tree,cn=config >>> changetype: modify >>> replace: nsds5beginreplicarefresh >>> nsds5beginreplicarefresh: start >>> replace nsds5beginreplicarefresh: >>> start >>> modifying entry "cn=ExampleAgreement,cn=replic >>> a,cn="dc=example,dc=com",cn=mapping tree,cn=config" >>> modify complete >>> >>> [root at ldap1 ~]# tail -f /var/log/dirsrv/slapd-EXAMPLE-COM/errors >>> [31/Aug/2016:11:11:09 +0000] schema-compat-plugin - schema-compat-plugin >>> tree scan will start in about 5 seconds! >>> [31/Aug/2016:11:11:09 +0000] - slapd started. Listening on All >>> Interfaces port 389 for LDAP requests >>> [31/Aug/2016:11:11:09 +0000] - Listening on All Interfaces port 636 for >>> LDAPS requests >>> [31/Aug/2016:11:11:09 +0000] - Listening on >>> /var/run/slapd-EXAMPLE-COM.socket for LDAPI requests >>> [31/Aug/2016:11:11:13 +0000] schema-compat-plugin - warning: no entries >>> set up under ou=sudoers,dc=example,dc=com >>> [31/Aug/2016:11:11:14 +0000] schema-compat-plugin - warning: no entries >>> set up under cn=ng, cn=compat,dc=example,dc=com >>> [31/Aug/2016:11:11:14 +0000] schema-compat-plugin - warning: no entries >>> set up under cn=computers, cn=compat,dc=example,dc=com >>> [31/Aug/2016:11:11:14 +0000] schema-compat-plugin - Finished plugin >>> initialization. >>> [31/Aug/2016:13:38:01 +0000] slapi_ldap_bind - Error: could not bind id >>> [cn=replication manager] authentication mechanism [SIMPLE]: error 32 (No >>> such object) errno 0 (Success) >>> [31/Aug/2016:13:38:01 +0000] NSMMReplicationPlugin - >>> agmt="cn=ExampleAgreement" (ldap2:389): Replication bind with SIMPLE auth >>> failed: LDAP error 32 (No such object) () >>> ^C >>> >>> I'm assuming this is just a standalone 389 Directory Server you are >>> trying to replicate to(not a freeIPA installation). If it is a freeipa >>> installation, then you should use the freeipa CLI for setting up >>> replication. >>> >>> The error 32 (no such object) you are getting is because the replica >>> does not have an entry "cn=replication manager". Looking at the >>> replication agreement: >>> >>> nsDS5ReplicaBindDN: cn=replication manager >>> >>> This is not a valid DN as there is no base suffix: For example, I would >>> expect to see something like "cn=replication manager,cn=config" >>> >>> https://access.redhat.com/documentation/en-US/Red_Hat_Direct >>> ory_Server/10/html/Administration_Guide/Creating_the_Supplie >>> r_Bind_DN_Entry.html >>> >>> Regards, >>> Mark >>> >>> >>> Please help me fix this >>> >>> >>> >>> >>> >>> >> >> >> >> > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From abokovoy at redhat.com Thu Sep 1 04:14:56 2016 From: abokovoy at redhat.com (Alexander Bokovoy) Date: Thu, 1 Sep 2016 07:14:56 +0300 Subject: [Freeipa-users] Command-line replication is not works in FreeIPA-Master In-Reply-To: References: <4e37eb09-8c96-114a-aee1-3c6ee568194f@redhat.com> Message-ID: <20160901041455.4xsgwvv5duxloi6g@redhat.com> On Thu, 01 Sep 2016, Andrey Rogovsky wrote: >Hi! >Thanks for your advices! >I'm try start replica and get this errors in log: >[01/Sep/2016:03:24:23 +0000] slapi_ldap_bind - Error: could not bind id >[cn=replication manager,cn=config] authentication mechanism [SIMPLE]: error >32 (No such object) errno 0 (Success) >[01/Sep/2016:03:24:23 +0000] NSMMReplicationPlugin - >agmt="cn=ExampleAgreement" (ldap2:389): Replication bind with SIMPLE auth >failed: LDAP error 32 (No such object) () You've been told already that you should have replication manager object created at both sides. Your 'cn=replicaton manager,cn=config' does not exist at the replica. You should read RHDS Administration Guide, at least the part about supplier bind DN entry, but preferrably the whole chapter it is part of: https://access.redhat.com/documentation/en-US/Red_Hat_Directory_Server/10/html/Administration_Guide/Creating_the_Supplier_Bind_DN_Entry.html > >This is my current replica: >filter: (objectclass=nsds5replica) >requesting: All userApplication attributes ># extended LDIF ># ># LDAPv3 ># base with scope subtree ># filter: (objectclass=nsds5replica) ># requesting: ALL ># > ># replica, dc\3Dexample\2Cdc\3Dcom, mapping tree, config >dn: cn=replica,cn=dc\3Dexample\2Cdc\3Dcom,cn=mapping tree,cn=config >objectClass: top >objectClass: nsds5replica >objectClass: extensibleObject >cn: replica >nsDS5ReplicaRoot: dc=example,dc=com >nsDS5ReplicaId: 7 >nsDS5ReplicaType: 3 >nsDS5Flags: 1 >nsds5ReplicaPurgeDelay: 604800 >nsDS5ReplicaBindDN: cn=replication manager,cn=config >nsState:: BwAAAAAAAADqnMdXAAAAAAAAAAAAAAAAAAAAAAAAAAABAAAAAAAAAA== >nsDS5ReplicaName: 496dba82-6f7a11e6-9d5ba359-5196ffe4 >nsds5ReplicaChangeCount: 118 >nsds5replicareapactive: 0 > ># search result >search: 2 >result: 0 Success > ># numResponses: 2 ># numEntries: 1 > >This is my current agreement: > ># extended LDIF ># ># LDAPv3 ># base with scope subtree ># filter: (objectclass=nsds5ReplicationAgreement) ># requesting: ALL ># > ># ExampleAgreement, replica, dc\3Dexample\2Cdc\3Dcom, mapping tree, config >dn: cn=ExampleAgreement,cn=replica,cn=dc\3Dexample\2Cdc\3Dcom,cn=mapping >tree, > cn=config >objectClass: top >objectClass: nsds5replicationagreement >cn: ExampleAgreement >nsDS5ReplicaHost: ldap2 >nsDS5ReplicaPort: 389 >nsDS5ReplicaBindDN: cn=replication manager,cn=config >nsDS5ReplicaBindMethod: SIMPLE >nsDS5ReplicaRoot: dc=example,dc=com >description: agreement between supplier1 and consumer1 >nsDS5ReplicaUpdateSchedule: 0000-0500 1 >nsDS5ReplicatedAttributeList: (objectclass=*) $ EXCLUDE >authorityRevocationLis > t >nsDS5ReplicaCredentials: >{AES-TUhNR0NTcUdTSWIzRFFFRkRUQm1NRVVHQ1NxR1NJYjNEUUVG > RERBNEJDUmxPVFl4TlRsbU5DMWtaV0UyTXpZeA0KTVMxaU1UYzFaREF3Wmkwek5qRmxNalkxWkFBQ > 0FRSUNBU0F3Q2dZSUtvWklodmNOQWdjd0hRWUpZSVpJQVdVRA0KQkFFcUJCQU1Dc25vTkVzZVJ4b3 > N2WVlEMXRpbQ==}a21h3uqnbcAZ1cX+NheCeg== >nsds5replicareapactive: 0 >nsds5replicaLastUpdateStart: 19700101000000Z >nsds5replicaLastUpdateEnd: 19700101000000Z >nsds5replicaChangesSentSinceStartup: >nsds5replicaLastUpdateStatus: 0 No replication sessions started since >server s > tartup >nsds5replicaUpdateInProgress: FALSE >nsds5replicaLastInitStart: 20160901032423Z >nsds5replicaLastInitEnd: 19700101000000Z >nsds5replicaLastInitStatus: 32 - LDAP error: No such object > ># search result >search: 2 >result: 0 Success > ># numResponses: 2 ># numEntries: 1 > >I'm try delete agreement, replica, user, changelog and create again. This >not help, same error: > >[01/Sep/2016:03:42:37 +0000] NSMMReplicationPlugin - agmt_delete: begin >[01/Sep/2016:03:45:35 +0000] NSMMReplicationPlugin - replica_config_delete: >Warning: The changelog for replica dc=example,dc=com is no longer valid >since the replica config is being deleted. Removing the changelog. >[01/Sep/2016:03:53:18 +0000] slapi_ldap_bind - Error: could not bind id >[cn=replication manager,cn=config] authentication mechanism [SIMPLE]: error >32 (No such object) errno 0 (Success) >[01/Sep/2016:03:53:18 +0000] NSMMReplicationPlugin - >agmt="cn=ExampleAgreement" (ldap2:389): Replication bind with SIMPLE auth >failed: LDAP error 32 (No such object) () > > > >2016-08-31 20:09 GMT+03:00 Mark Reynolds : > >> >> >> On 08/31/2016 12:39 PM, Andrey Rogovsky wrote: >> >> Hi, Mark! >> >> Thanks for explain. Now I create replication manager: (I hope) >> [root at ldap1 ~]# ldapsearch -h ldap1.example.com -p 389 -xLLL -D >> "cn=directory manager" -W -b cn=config "cn=replication manager" >> Enter LDAP Password: >> dn: cn=replication manager,cn=config >> objectClass: inetorgperson >> objectClass: person >> objectClass: top >> objectClass: organizationalPerson >> cn: replication manager >> sn: RM >> userPassword:: e1NTSEF9N1JiRmNXWTFXNDA1cmdYSU >> dCNWJtV3RzOElNQXBhakhXam94WlE9PQ= >> = >> >> What is next? I use manual from 8 version and this a bit obsoleted. >> >> Now you should be able to initialize your standalone server by updating >> the agreement on the ipa DS: >> >> dn: cn=ExampleAgreement,cn=replica,cn="dc=example,dc=com",cn=mapping >> tree,cn=config >> changetype: modify >> replace: nsds5beginreplicarefresh >> nsds5beginreplicarefresh: start >> >> If something goes wrong let us know what's in the errors log again. >> >> Mark >> >> >> >> 2016-08-31 19:30 GMT+03:00 Mark Reynolds : >> >>> Hi Andrey, >>> >>> It looks like you still did not create the replication manager entry. >>> You must create that manager entry on the standalone server. Please read >>> the link I sent you: >>> >>> https://access.redhat.com/documentation/en-US/Red_Hat_Direct >>> ory_Server/10/html/Administration_Guide/Creating_the_Supplie >>> r_Bind_DN_Entry.html >>> >>> You can verify its existence by doing this search against the standalone >>> server: >>> >>> ldapsearch -h ldap1.example.com -p 389 -xLLL -D "cn=directory manager" >>> -W -b cn=config "cn=replication manager" >>> >>> Mark >>> >>> >>> On 08/31/2016 11:50 AM, Andrey Rogovsky wrote: >>> >>> Hi! >>> Thank you for fast reply. >>> Yes, I want use standalone 389DS to replica from FreeIPA. >>> There is my replica: >>> filter: (objectclass=nsds5replica) >>> requesting: All userApplication attributes >>> # extended LDIF >>> # >>> # LDAPv3 >>> # base with scope subtree >>> # filter: (objectclass=nsds5replica) >>> # requesting: ALL >>> # >>> >>> # replica, dc\3Dexample\2Cdc\3Dcom, mapping tree, config >>> dn: cn=replica,cn=dc\3Dexample\2Cdc\3Dcom,cn=mapping tree,cn=config >>> objectClass: top >>> objectClass: nsds5replica >>> objectClass: extensibleObject >>> cn: replica >>> nsDS5ReplicaRoot: dc=example,dc=com >>> nsDS5ReplicaId: 7 >>> nsDS5ReplicaType: 3 >>> nsDS5Flags: 1 >>> nsds5ReplicaPurgeDelay: 604800 >>> nsDS5ReplicaBindDN: cn=replication manager,cn=config >>> nsState:: BwAAAAAAAABZ98ZXAAAAAAAAAAAAAAAAAAAAAAAAAAABAAAAAAAAAA== >>> nsDS5ReplicaName: 496dba82-6f7a11e6-9d5ba359-5196ffe4 >>> nsds5ReplicaChangeCount: 22 >>> nsds5replicareapactive: 0 >>> >>> # search result >>> search: 2 >>> result: 0 Success >>> >>> # numResponses: 2 >>> # numEntries: 1 >>> >>> So, my replica have entry "cn=replication manager" >>> >>> But I try add entry in agreement. Unforthunalty this is not help, error >>> is present: >>> [root at ldap1 ~]# ldapmodify -v -h ldap1.example.com -p 389 -D >>> "cn=directory manager" -w ... >>> ldap_initialize( ldap://ldap1.example.com:389 ) >>> dn: cn=ExampleAgreement,cn=replica,cn="dc=example,dc=com",cn=mapping >>> tree,cn=config >>> changetype: modify >>> replace: nsds5ReplicaBindDN >>> nsds5ReplicaBindDN: cn=replication manager,cn=config >>> replace nsds5ReplicaBindDN: >>> cn=replication manager,cn=config >>> modifying entry "cn=ExampleAgreement,cn=replic >>> a,cn="dc=example,dc=com",cn=mapping tree,cn=config" >>> modify complete >>> >>> [root at ldap1 ~]# tail -f /var/log/dirsrv/slapd-EXAMPLE-COM/errors >>> [31/Aug/2016:11:11:09 +0000] schema-compat-plugin - schema-compat-plugin >>> tree scan will start in about 5 seconds! >>> [31/Aug/2016:11:11:09 +0000] - slapd started. Listening on All >>> Interfaces port 389 for LDAP requests >>> [31/Aug/2016:11:11:09 +0000] - Listening on All Interfaces port 636 for >>> LDAPS requests >>> [31/Aug/2016:11:11:09 +0000] - Listening on /var/run/slapd-EXAMPLE-COM.socket >>> for LDAPI requests >>> [31/Aug/2016:11:11:13 +0000] schema-compat-plugin - warning: no entries >>> set up under ou=sudoers,dc=example,dc=com >>> [31/Aug/2016:11:11:14 +0000] schema-compat-plugin - warning: no entries >>> set up under cn=ng, cn=compat,dc=example,dc=com >>> [31/Aug/2016:11:11:14 +0000] schema-compat-plugin - warning: no entries >>> set up under cn=computers, cn=compat,dc=example,dc=com >>> [31/Aug/2016:11:11:14 +0000] schema-compat-plugin - Finished plugin >>> initialization. >>> [31/Aug/2016:13:38:01 +0000] slapi_ldap_bind - Error: could not bind id >>> [cn=replication manager] authentication mechanism [SIMPLE]: error 32 (No >>> such object) errno 0 (Success) >>> [31/Aug/2016:13:38:01 +0000] NSMMReplicationPlugin - >>> agmt="cn=ExampleAgreement" (ldap2:389): Replication bind with SIMPLE auth >>> failed: LDAP error 32 (No such object) () >>> ^C >>> [root at ldap1 ~]# ldapmodify -v -h ldap1.example.com -p 389 -D >>> "cn=directory manager" -w ... >>> ldap_initialize( ldap://ldap1.example.com:389 ) >>> dn: cn=ExampleAgreement,cn=replica,cn="dc=example,dc=com",cn=mapping >>> tree,cn=config >>> changetype: modify >>> replace: nsds5beginreplicarefresh >>> nsds5beginreplicarefresh: start >>> replace nsds5beginreplicarefresh: >>> start >>> modifying entry "cn=ExampleAgreement,cn=replic >>> a,cn="dc=example,dc=com",cn=mapping tree,cn=config" >>> modify complete >>> >>> [root at ldap1 ~]# tail -f /var/log/dirsrv/slapd-EXAMPLE-COM/errors >>> [31/Aug/2016:11:11:09 +0000] - slapd started. Listening on All >>> Interfaces port 389 for LDAP requests >>> [31/Aug/2016:11:11:09 +0000] - Listening on All Interfaces port 636 for >>> LDAPS requests >>> [31/Aug/2016:11:11:09 +0000] - Listening on /var/run/slapd-EXAMPLE-COM.socket >>> for LDAPI requests >>> [31/Aug/2016:11:11:13 +0000] schema-compat-plugin - warning: no entries >>> set up under ou=sudoers,dc=example,dc=com >>> [31/Aug/2016:11:11:14 +0000] schema-compat-plugin - warning: no entries >>> set up under cn=ng, cn=compat,dc=example,dc=com >>> [31/Aug/2016:11:11:14 +0000] schema-compat-plugin - warning: no entries >>> set up under cn=computers, cn=compat,dc=example,dc=com >>> [31/Aug/2016:11:11:14 +0000] schema-compat-plugin - Finished plugin >>> initialization. >>> [31/Aug/2016:13:38:01 +0000] slapi_ldap_bind - Error: could not bind id >>> [cn=replication manager] authentication mechanism [SIMPLE]: error 32 (No >>> such object) errno 0 (Success) >>> [31/Aug/2016:13:38:01 +0000] NSMMReplicationPlugin - >>> agmt="cn=ExampleAgreement" (ldap2:389): Replication bind with SIMPLE auth >>> failed: LDAP error 32 (No such object) () >>> [31/Aug/2016:15:48:36 +0000] slapi_ldap_bind - Error: could not bind id >>> [cn=replication manager,cn=config] authentication mechanism [SIMPLE]: error >>> 32 (No such object) errno 0 (Success) >>> ^C >>> [root at ldap1 ~]# >>> >>> >>> 2016-08-31 18:15 GMT+03:00 Mark Reynolds : >>> >>>> >>>> >>>> On 08/31/2016 09:50 AM, Andrey Rogovsky wrote: >>>> >>>> Hi! >>>> >>>> I try configure manual replica from FreeIPA DS to 389 DS. >>>> I have two VM: ldap1.example.com and ldap2.example.com >>>> I was used this manual https://www.centos.org/ >>>> docs/5/html/CDS/ag/8.0/Managing_Replication-Configuring-Repl >>>> ication-cmd.html for configure relica >>>> >>>> There was replica agreement before starting: >>>> >>>> # extended LDIF >>>> # >>>> # LDAPv3 >>>> # base with scope subtree >>>> # filter: (objectclass=nsds5ReplicationAgreement) >>>> # requesting: ALL >>>> # >>>> >>>> # ExampleAgreement, replica, dc\3Dexample\2Cdc\3Dcom, mapping tree, >>>> config >>>> dn: cn=ExampleAgreement,cn=replica,cn=dc\3Dexample\2Cdc\3Dcom,cn=mapping >>>> tree, >>>> cn=config >>>> objectClass: top >>>> objectClass: nsds5replicationagreement >>>> cn: ExampleAgreement >>>> nsDS5ReplicaHost: ldap2 >>>> nsDS5ReplicaPort: 389 >>>> nsDS5ReplicaBindDN: cn=replication manager >>>> nsDS5ReplicaBindMethod: SIMPLE >>>> nsDS5ReplicaRoot: dc=example,dc=com >>>> description: agreement between supplier1 and consumer1 >>>> nsDS5ReplicaUpdateSchedule: 0000-0500 1 >>>> nsDS5ReplicatedAttributeList: (objectclass=*) $ EXCLUDE >>>> authorityRevocationLis >>>> t >>>> nsDS5ReplicaCredentials: {AES-TUhNR0NTcUdTSWIzRFFFRkRUQ >>>> m1NRVVHQ1NxR1NJYjNEUUVG >>>> RERBNEJDUmxPVFl4TlRsbU5DMWtaV0UyTXpZeA0KTVMxaU1UYzFaREF3Wmk >>>> wek5qRmxNalkxWkFBQ >>>> 0FRSUNBU0F3Q2dZSUtvWklodmNOQWdjd0hRWUpZSVpJQVdVRA0KQkFFcUJC >>>> QUVJckpINmE0S3RFYl >>>> NhLzkxL01qZg==}Wo+c0XfBnaDhg/a36yguXg== >>>> nsds5replicareapactive: 0 >>>> nsds5replicaLastUpdateStart: 19700101000000Z >>>> nsds5replicaLastUpdateEnd: 19700101000000Z >>>> nsds5replicaChangesSentSinceStartup: >>>> nsds5replicaLastUpdateStatus: 0 No replication sessions started since >>>> server s >>>> tartup >>>> nsds5replicaUpdateInProgress: FALSE >>>> nsds5replicaLastInitStart: 19700101000000Z >>>> nsds5replicaLastInitEnd: 19700101000000Z >>>> >>>> # search result >>>> search: 2 >>>> result: 0 Success >>>> >>>> # numResponses: 2 >>>> # numEntries: >>>> >>>> >>>> There is errors which I get when start replica: >>>> >>>> >>>> [root at ldap1 ~]# ldapmodify -v -h ldap1.example.com -p 389 -D >>>> "cn=directory manager" -w ... >>>> ldap_initialize( ldap://ldap1.example.com:389 ) >>>> dn: cn=ExampleAgreement,cn=replica,cn="dc=example,dc=com",cn=mapping >>>> tree,cn=config >>>> changetype: modify >>>> replace: nsds5beginreplicarefresh >>>> nsds5beginreplicarefresh: start >>>> replace nsds5beginreplicarefresh: >>>> start >>>> modifying entry "cn=ExampleAgreement,cn=replic >>>> a,cn="dc=example,dc=com",cn=mapping tree,cn=config" >>>> modify complete >>>> >>>> [root at ldap1 ~]# tail -f /var/log/dirsrv/slapd-EXAMPLE-COM/errors >>>> [31/Aug/2016:11:11:09 +0000] schema-compat-plugin - schema-compat-plugin >>>> tree scan will start in about 5 seconds! >>>> [31/Aug/2016:11:11:09 +0000] - slapd started. Listening on All >>>> Interfaces port 389 for LDAP requests >>>> [31/Aug/2016:11:11:09 +0000] - Listening on All Interfaces port 636 for >>>> LDAPS requests >>>> [31/Aug/2016:11:11:09 +0000] - Listening on >>>> /var/run/slapd-EXAMPLE-COM.socket for LDAPI requests >>>> [31/Aug/2016:11:11:13 +0000] schema-compat-plugin - warning: no entries >>>> set up under ou=sudoers,dc=example,dc=com >>>> [31/Aug/2016:11:11:14 +0000] schema-compat-plugin - warning: no entries >>>> set up under cn=ng, cn=compat,dc=example,dc=com >>>> [31/Aug/2016:11:11:14 +0000] schema-compat-plugin - warning: no entries >>>> set up under cn=computers, cn=compat,dc=example,dc=com >>>> [31/Aug/2016:11:11:14 +0000] schema-compat-plugin - Finished plugin >>>> initialization. >>>> [31/Aug/2016:13:38:01 +0000] slapi_ldap_bind - Error: could not bind id >>>> [cn=replication manager] authentication mechanism [SIMPLE]: error 32 (No >>>> such object) errno 0 (Success) >>>> [31/Aug/2016:13:38:01 +0000] NSMMReplicationPlugin - >>>> agmt="cn=ExampleAgreement" (ldap2:389): Replication bind with SIMPLE auth >>>> failed: LDAP error 32 (No such object) () >>>> ^C >>>> >>>> I'm assuming this is just a standalone 389 Directory Server you are >>>> trying to replicate to(not a freeIPA installation). If it is a freeipa >>>> installation, then you should use the freeipa CLI for setting up >>>> replication. >>>> >>>> The error 32 (no such object) you are getting is because the replica >>>> does not have an entry "cn=replication manager". Looking at the >>>> replication agreement: >>>> >>>> nsDS5ReplicaBindDN: cn=replication manager >>>> >>>> This is not a valid DN as there is no base suffix: For example, I would >>>> expect to see something like "cn=replication manager,cn=config" >>>> >>>> https://access.redhat.com/documentation/en-US/Red_Hat_Direct >>>> ory_Server/10/html/Administration_Guide/Creating_the_Supplie >>>> r_Bind_DN_Entry.html >>>> >>>> Regards, >>>> Mark >>>> >>>> >>>> Please help me fix this >>>> >>>> >>>> >>>> >>>> >>>> >>> >>> >>> >>> >> >> >> >> >-- >Manage your subscription for the Freeipa-users mailing list: >https://www.redhat.com/mailman/listinfo/freeipa-users >Go to http://freeipa.org for more info on the project -- / Alexander Bokovoy From a.rogovsky at gmail.com Thu Sep 1 04:25:44 2016 From: a.rogovsky at gmail.com (Andrey Rogovsky) Date: Thu, 1 Sep 2016 07:25:44 +0300 Subject: [Freeipa-users] Command-line replication is not works in FreeIPA-Master In-Reply-To: <20160901041455.4xsgwvv5duxloi6g@redhat.com> References: <4e37eb09-8c96-114a-aee1-3c6ee568194f@redhat.com> <20160901041455.4xsgwvv5duxloi6g@redhat.com> Message-ID: Hi, Alexander! Thank for fast reply. I have replication manager object: filter: (objectclass=organizationalPerson) requesting: All userApplication attributes # extended LDIF # # LDAPv3 # base with scope subtree # filter: (objectclass=organizationalPerson) # requesting: ALL # # replication manager, config dn: cn=replication manager,cn=config objectClass: inetorgperson objectClass: person objectClass: top objectClass: organizationalPerson cn: replication manager sn: RM userPassword:: e1NTSEF9d281RGZOTTlCSEVWTEhxY1lTcGs0WHdjRXplemU4S280S3EwWnc9PQ= = # search result search: 2 result: 0 Success # numResponses: 2 # numEntries: 1 But error is present. 2016-09-01 7:14 GMT+03:00 Alexander Bokovoy : > On Thu, 01 Sep 2016, Andrey Rogovsky wrote: > >> Hi! >> Thanks for your advices! >> I'm try start replica and get this errors in log: >> [01/Sep/2016:03:24:23 +0000] slapi_ldap_bind - Error: could not bind id >> [cn=replication manager,cn=config] authentication mechanism [SIMPLE]: >> error >> 32 (No such object) errno 0 (Success) >> [01/Sep/2016:03:24:23 +0000] NSMMReplicationPlugin - >> agmt="cn=ExampleAgreement" (ldap2:389): Replication bind with SIMPLE auth >> failed: LDAP error 32 (No such object) () >> > You've been told already that you should have replication manager object > created at both sides. Your 'cn=replicaton manager,cn=config' does not > exist at the replica. > > You should read RHDS Administration Guide, at least the part about > supplier bind DN entry, but preferrably the whole chapter it is part of: > https://access.redhat.com/documentation/en-US/Red_Hat_Direct > ory_Server/10/html/Administration_Guide/Creating_the_ > Supplier_Bind_DN_Entry.html > > > > >> This is my current replica: >> filter: (objectclass=nsds5replica) >> requesting: All userApplication attributes >> # extended LDIF >> # >> # LDAPv3 >> # base with scope subtree >> # filter: (objectclass=nsds5replica) >> # requesting: ALL >> # >> >> # replica, dc\3Dexample\2Cdc\3Dcom, mapping tree, config >> dn: cn=replica,cn=dc\3Dexample\2Cdc\3Dcom,cn=mapping tree,cn=config >> objectClass: top >> objectClass: nsds5replica >> objectClass: extensibleObject >> cn: replica >> nsDS5ReplicaRoot: dc=example,dc=com >> nsDS5ReplicaId: 7 >> nsDS5ReplicaType: 3 >> nsDS5Flags: 1 >> nsds5ReplicaPurgeDelay: 604800 >> nsDS5ReplicaBindDN: cn=replication manager,cn=config >> nsState:: BwAAAAAAAADqnMdXAAAAAAAAAAAAAAAAAAAAAAAAAAABAAAAAAAAAA== >> nsDS5ReplicaName: 496dba82-6f7a11e6-9d5ba359-5196ffe4 >> nsds5ReplicaChangeCount: 118 >> nsds5replicareapactive: 0 >> >> # search result >> search: 2 >> result: 0 Success >> >> # numResponses: 2 >> # numEntries: 1 >> >> This is my current agreement: >> >> # extended LDIF >> # >> # LDAPv3 >> # base with scope subtree >> # filter: (objectclass=nsds5ReplicationAgreement) >> # requesting: ALL >> # >> >> # ExampleAgreement, replica, dc\3Dexample\2Cdc\3Dcom, mapping tree, config >> dn: cn=ExampleAgreement,cn=replica,cn=dc\3Dexample\2Cdc\3Dcom,cn=mapping >> tree, >> cn=config >> objectClass: top >> objectClass: nsds5replicationagreement >> cn: ExampleAgreement >> nsDS5ReplicaHost: ldap2 >> nsDS5ReplicaPort: 389 >> nsDS5ReplicaBindDN: cn=replication manager,cn=config >> nsDS5ReplicaBindMethod: SIMPLE >> nsDS5ReplicaRoot: dc=example,dc=com >> description: agreement between supplier1 and consumer1 >> nsDS5ReplicaUpdateSchedule: 0000-0500 1 >> nsDS5ReplicatedAttributeList: (objectclass=*) $ EXCLUDE >> authorityRevocationLis >> t >> nsDS5ReplicaCredentials: >> {AES-TUhNR0NTcUdTSWIzRFFFRkRUQm1NRVVHQ1NxR1NJYjNEUUVG >> RERBNEJDUmxPVFl4TlRsbU5DMWtaV0UyTXpZeA0KTVMxaU1UYzFaREF3Wmkw >> ek5qRmxNalkxWkFBQ >> 0FRSUNBU0F3Q2dZSUtvWklodmNOQWdjd0hRWUpZSVpJQVdVRA0KQkFFcUJCQ >> U1Dc25vTkVzZVJ4b3 >> N2WVlEMXRpbQ==}a21h3uqnbcAZ1cX+NheCeg== >> nsds5replicareapactive: 0 >> nsds5replicaLastUpdateStart: 19700101000000Z >> nsds5replicaLastUpdateEnd: 19700101000000Z >> nsds5replicaChangesSentSinceStartup: >> nsds5replicaLastUpdateStatus: 0 No replication sessions started since >> server s >> tartup >> nsds5replicaUpdateInProgress: FALSE >> nsds5replicaLastInitStart: 20160901032423Z >> nsds5replicaLastInitEnd: 19700101000000Z >> nsds5replicaLastInitStatus: 32 - LDAP error: No such object >> >> # search result >> search: 2 >> result: 0 Success >> >> # numResponses: 2 >> # numEntries: 1 >> >> I'm try delete agreement, replica, user, changelog and create again. This >> not help, same error: >> >> [01/Sep/2016:03:42:37 +0000] NSMMReplicationPlugin - agmt_delete: begin >> [01/Sep/2016:03:45:35 +0000] NSMMReplicationPlugin - >> replica_config_delete: >> Warning: The changelog for replica dc=example,dc=com is no longer valid >> since the replica config is being deleted. Removing the changelog. >> [01/Sep/2016:03:53:18 +0000] slapi_ldap_bind - Error: could not bind id >> [cn=replication manager,cn=config] authentication mechanism [SIMPLE]: >> error >> 32 (No such object) errno 0 (Success) >> [01/Sep/2016:03:53:18 +0000] NSMMReplicationPlugin - >> agmt="cn=ExampleAgreement" (ldap2:389): Replication bind with SIMPLE auth >> failed: LDAP error 32 (No such object) () >> >> >> >> 2016-08-31 20:09 GMT+03:00 Mark Reynolds : >> >> >>> >>> On 08/31/2016 12:39 PM, Andrey Rogovsky wrote: >>> >>> Hi, Mark! >>> >>> Thanks for explain. Now I create replication manager: (I hope) >>> [root at ldap1 ~]# ldapsearch -h ldap1.example.com -p 389 -xLLL -D >>> "cn=directory manager" -W -b cn=config "cn=replication manager" >>> Enter LDAP Password: >>> dn: cn=replication manager,cn=config >>> objectClass: inetorgperson >>> objectClass: person >>> objectClass: top >>> objectClass: organizationalPerson >>> cn: replication manager >>> sn: RM >>> userPassword:: e1NTSEF9N1JiRmNXWTFXNDA1cmdYSU >>> dCNWJtV3RzOElNQXBhakhXam94WlE9PQ= >>> = >>> >>> What is next? I use manual from 8 version and this a bit obsoleted. >>> >>> Now you should be able to initialize your standalone server by updating >>> the agreement on the ipa DS: >>> >>> dn: cn=ExampleAgreement,cn=replica,cn="dc=example,dc=com",cn=mapping >>> tree,cn=config >>> changetype: modify >>> replace: nsds5beginreplicarefresh >>> nsds5beginreplicarefresh: start >>> >>> If something goes wrong let us know what's in the errors log again. >>> >>> Mark >>> >>> >>> >>> 2016-08-31 19:30 GMT+03:00 Mark Reynolds : >>> >>> Hi Andrey, >>>> >>>> It looks like you still did not create the replication manager entry. >>>> You must create that manager entry on the standalone server. Please >>>> read >>>> the link I sent you: >>>> >>>> https://access.redhat.com/documentation/en-US/Red_Hat_Direct >>>> ory_Server/10/html/Administration_Guide/Creating_the_Supplie >>>> r_Bind_DN_Entry.html >>>> >>>> You can verify its existence by doing this search against the standalone >>>> server: >>>> >>>> ldapsearch -h ldap1.example.com -p 389 -xLLL -D "cn=directory manager" >>>> -W -b cn=config "cn=replication manager" >>>> >>>> Mark >>>> >>>> >>>> On 08/31/2016 11:50 AM, Andrey Rogovsky wrote: >>>> >>>> Hi! >>>> Thank you for fast reply. >>>> Yes, I want use standalone 389DS to replica from FreeIPA. >>>> There is my replica: >>>> filter: (objectclass=nsds5replica) >>>> requesting: All userApplication attributes >>>> # extended LDIF >>>> # >>>> # LDAPv3 >>>> # base with scope subtree >>>> # filter: (objectclass=nsds5replica) >>>> # requesting: ALL >>>> # >>>> >>>> # replica, dc\3Dexample\2Cdc\3Dcom, mapping tree, config >>>> dn: cn=replica,cn=dc\3Dexample\2Cdc\3Dcom,cn=mapping tree,cn=config >>>> objectClass: top >>>> objectClass: nsds5replica >>>> objectClass: extensibleObject >>>> cn: replica >>>> nsDS5ReplicaRoot: dc=example,dc=com >>>> nsDS5ReplicaId: 7 >>>> nsDS5ReplicaType: 3 >>>> nsDS5Flags: 1 >>>> nsds5ReplicaPurgeDelay: 604800 >>>> nsDS5ReplicaBindDN: cn=replication manager,cn=config >>>> nsState:: BwAAAAAAAABZ98ZXAAAAAAAAAAAAAAAAAAAAAAAAAAABAAAAAAAAAA== >>>> nsDS5ReplicaName: 496dba82-6f7a11e6-9d5ba359-5196ffe4 >>>> nsds5ReplicaChangeCount: 22 >>>> nsds5replicareapactive: 0 >>>> >>>> # search result >>>> search: 2 >>>> result: 0 Success >>>> >>>> # numResponses: 2 >>>> # numEntries: 1 >>>> >>>> So, my replica have entry "cn=replication manager" >>>> >>>> But I try add entry in agreement. Unforthunalty this is not help, error >>>> is present: >>>> [root at ldap1 ~]# ldapmodify -v -h ldap1.example.com -p 389 -D >>>> "cn=directory manager" -w ... >>>> ldap_initialize( ldap://ldap1.example.com:389 ) >>>> dn: cn=ExampleAgreement,cn=replica,cn="dc=example,dc=com",cn=mapping >>>> tree,cn=config >>>> changetype: modify >>>> replace: nsds5ReplicaBindDN >>>> nsds5ReplicaBindDN: cn=replication manager,cn=config >>>> replace nsds5ReplicaBindDN: >>>> cn=replication manager,cn=config >>>> modifying entry "cn=ExampleAgreement,cn=replic >>>> a,cn="dc=example,dc=com",cn=mapping tree,cn=config" >>>> modify complete >>>> >>>> [root at ldap1 ~]# tail -f /var/log/dirsrv/slapd-EXAMPLE-COM/errors >>>> [31/Aug/2016:11:11:09 +0000] schema-compat-plugin - schema-compat-plugin >>>> tree scan will start in about 5 seconds! >>>> [31/Aug/2016:11:11:09 +0000] - slapd started. Listening on All >>>> Interfaces port 389 for LDAP requests >>>> [31/Aug/2016:11:11:09 +0000] - Listening on All Interfaces port 636 for >>>> LDAPS requests >>>> [31/Aug/2016:11:11:09 +0000] - Listening on >>>> /var/run/slapd-EXAMPLE-COM.socket >>>> for LDAPI requests >>>> [31/Aug/2016:11:11:13 +0000] schema-compat-plugin - warning: no entries >>>> set up under ou=sudoers,dc=example,dc=com >>>> [31/Aug/2016:11:11:14 +0000] schema-compat-plugin - warning: no entries >>>> set up under cn=ng, cn=compat,dc=example,dc=com >>>> [31/Aug/2016:11:11:14 +0000] schema-compat-plugin - warning: no entries >>>> set up under cn=computers, cn=compat,dc=example,dc=com >>>> [31/Aug/2016:11:11:14 +0000] schema-compat-plugin - Finished plugin >>>> initialization. >>>> [31/Aug/2016:13:38:01 +0000] slapi_ldap_bind - Error: could not bind id >>>> [cn=replication manager] authentication mechanism [SIMPLE]: error 32 (No >>>> such object) errno 0 (Success) >>>> [31/Aug/2016:13:38:01 +0000] NSMMReplicationPlugin - >>>> agmt="cn=ExampleAgreement" (ldap2:389): Replication bind with SIMPLE >>>> auth >>>> failed: LDAP error 32 (No such object) () >>>> ^C >>>> [root at ldap1 ~]# ldapmodify -v -h ldap1.example.com -p 389 -D >>>> "cn=directory manager" -w ... >>>> ldap_initialize( ldap://ldap1.example.com:389 ) >>>> dn: cn=ExampleAgreement,cn=replica,cn="dc=example,dc=com",cn=mapping >>>> tree,cn=config >>>> changetype: modify >>>> replace: nsds5beginreplicarefresh >>>> nsds5beginreplicarefresh: start >>>> replace nsds5beginreplicarefresh: >>>> start >>>> modifying entry "cn=ExampleAgreement,cn=replic >>>> a,cn="dc=example,dc=com",cn=mapping tree,cn=config" >>>> modify complete >>>> >>>> [root at ldap1 ~]# tail -f /var/log/dirsrv/slapd-EXAMPLE-COM/errors >>>> [31/Aug/2016:11:11:09 +0000] - slapd started. Listening on All >>>> Interfaces port 389 for LDAP requests >>>> [31/Aug/2016:11:11:09 +0000] - Listening on All Interfaces port 636 for >>>> LDAPS requests >>>> [31/Aug/2016:11:11:09 +0000] - Listening on >>>> /var/run/slapd-EXAMPLE-COM.socket >>>> for LDAPI requests >>>> [31/Aug/2016:11:11:13 +0000] schema-compat-plugin - warning: no entries >>>> set up under ou=sudoers,dc=example,dc=com >>>> [31/Aug/2016:11:11:14 +0000] schema-compat-plugin - warning: no entries >>>> set up under cn=ng, cn=compat,dc=example,dc=com >>>> [31/Aug/2016:11:11:14 +0000] schema-compat-plugin - warning: no entries >>>> set up under cn=computers, cn=compat,dc=example,dc=com >>>> [31/Aug/2016:11:11:14 +0000] schema-compat-plugin - Finished plugin >>>> initialization. >>>> [31/Aug/2016:13:38:01 +0000] slapi_ldap_bind - Error: could not bind id >>>> [cn=replication manager] authentication mechanism [SIMPLE]: error 32 (No >>>> such object) errno 0 (Success) >>>> [31/Aug/2016:13:38:01 +0000] NSMMReplicationPlugin - >>>> agmt="cn=ExampleAgreement" (ldap2:389): Replication bind with SIMPLE >>>> auth >>>> failed: LDAP error 32 (No such object) () >>>> [31/Aug/2016:15:48:36 +0000] slapi_ldap_bind - Error: could not bind id >>>> [cn=replication manager,cn=config] authentication mechanism [SIMPLE]: >>>> error >>>> 32 (No such object) errno 0 (Success) >>>> ^C >>>> [root at ldap1 ~]# >>>> >>>> >>>> 2016-08-31 18:15 GMT+03:00 Mark Reynolds : >>>> >>>> >>>>> >>>>> On 08/31/2016 09:50 AM, Andrey Rogovsky wrote: >>>>> >>>>> Hi! >>>>> >>>>> I try configure manual replica from FreeIPA DS to 389 DS. >>>>> I have two VM: ldap1.example.com and ldap2.example.com >>>>> I was used this manual https://www.centos.org/ >>>>> docs/5/html/CDS/ag/8.0/Managing_Replication-Configuring-Repl >>>>> ication-cmd.html for configure relica >>>>> >>>>> There was replica agreement before starting: >>>>> >>>>> # extended LDIF >>>>> # >>>>> # LDAPv3 >>>>> # base with scope subtree >>>>> # filter: (objectclass=nsds5ReplicationAgreement) >>>>> # requesting: ALL >>>>> # >>>>> >>>>> # ExampleAgreement, replica, dc\3Dexample\2Cdc\3Dcom, mapping tree, >>>>> config >>>>> dn: cn=ExampleAgreement,cn=replica,cn=dc\3Dexample\2Cdc\3Dcom, >>>>> cn=mapping >>>>> tree, >>>>> cn=config >>>>> objectClass: top >>>>> objectClass: nsds5replicationagreement >>>>> cn: ExampleAgreement >>>>> nsDS5ReplicaHost: ldap2 >>>>> nsDS5ReplicaPort: 389 >>>>> nsDS5ReplicaBindDN: cn=replication manager >>>>> nsDS5ReplicaBindMethod: SIMPLE >>>>> nsDS5ReplicaRoot: dc=example,dc=com >>>>> description: agreement between supplier1 and consumer1 >>>>> nsDS5ReplicaUpdateSchedule: 0000-0500 1 >>>>> nsDS5ReplicatedAttributeList: (objectclass=*) $ EXCLUDE >>>>> authorityRevocationLis >>>>> t >>>>> nsDS5ReplicaCredentials: {AES-TUhNR0NTcUdTSWIzRFFFRkRUQ >>>>> m1NRVVHQ1NxR1NJYjNEUUVG >>>>> RERBNEJDUmxPVFl4TlRsbU5DMWtaV0UyTXpZeA0KTVMxaU1UYzFaREF3Wmk >>>>> wek5qRmxNalkxWkFBQ >>>>> 0FRSUNBU0F3Q2dZSUtvWklodmNOQWdjd0hRWUpZSVpJQVdVRA0KQkFFcUJC >>>>> QUVJckpINmE0S3RFYl >>>>> NhLzkxL01qZg==}Wo+c0XfBnaDhg/a36yguXg== >>>>> nsds5replicareapactive: 0 >>>>> nsds5replicaLastUpdateStart: 19700101000000Z >>>>> nsds5replicaLastUpdateEnd: 19700101000000Z >>>>> nsds5replicaChangesSentSinceStartup: >>>>> nsds5replicaLastUpdateStatus: 0 No replication sessions started since >>>>> server s >>>>> tartup >>>>> nsds5replicaUpdateInProgress: FALSE >>>>> nsds5replicaLastInitStart: 19700101000000Z >>>>> nsds5replicaLastInitEnd: 19700101000000Z >>>>> >>>>> # search result >>>>> search: 2 >>>>> result: 0 Success >>>>> >>>>> # numResponses: 2 >>>>> # numEntries: >>>>> >>>>> >>>>> There is errors which I get when start replica: >>>>> >>>>> >>>>> [root at ldap1 ~]# ldapmodify -v -h ldap1.example.com -p 389 -D >>>>> "cn=directory manager" -w ... >>>>> ldap_initialize( ldap://ldap1.example.com:389 ) >>>>> dn: cn=ExampleAgreement,cn=replica,cn="dc=example,dc=com",cn=mapping >>>>> tree,cn=config >>>>> changetype: modify >>>>> replace: nsds5beginreplicarefresh >>>>> nsds5beginreplicarefresh: start >>>>> replace nsds5beginreplicarefresh: >>>>> start >>>>> modifying entry "cn=ExampleAgreement,cn=replic >>>>> a,cn="dc=example,dc=com",cn=mapping tree,cn=config" >>>>> modify complete >>>>> >>>>> [root at ldap1 ~]# tail -f /var/log/dirsrv/slapd-EXAMPLE-COM/errors >>>>> [31/Aug/2016:11:11:09 +0000] schema-compat-plugin - >>>>> schema-compat-plugin >>>>> tree scan will start in about 5 seconds! >>>>> [31/Aug/2016:11:11:09 +0000] - slapd started. Listening on All >>>>> Interfaces port 389 for LDAP requests >>>>> [31/Aug/2016:11:11:09 +0000] - Listening on All Interfaces port 636 for >>>>> LDAPS requests >>>>> [31/Aug/2016:11:11:09 +0000] - Listening on >>>>> /var/run/slapd-EXAMPLE-COM.socket for LDAPI requests >>>>> [31/Aug/2016:11:11:13 +0000] schema-compat-plugin - warning: no entries >>>>> set up under ou=sudoers,dc=example,dc=com >>>>> [31/Aug/2016:11:11:14 +0000] schema-compat-plugin - warning: no entries >>>>> set up under cn=ng, cn=compat,dc=example,dc=com >>>>> [31/Aug/2016:11:11:14 +0000] schema-compat-plugin - warning: no entries >>>>> set up under cn=computers, cn=compat,dc=example,dc=com >>>>> [31/Aug/2016:11:11:14 +0000] schema-compat-plugin - Finished plugin >>>>> initialization. >>>>> [31/Aug/2016:13:38:01 +0000] slapi_ldap_bind - Error: could not bind id >>>>> [cn=replication manager] authentication mechanism [SIMPLE]: error 32 >>>>> (No >>>>> such object) errno 0 (Success) >>>>> [31/Aug/2016:13:38:01 +0000] NSMMReplicationPlugin - >>>>> agmt="cn=ExampleAgreement" (ldap2:389): Replication bind with SIMPLE >>>>> auth >>>>> failed: LDAP error 32 (No such object) () >>>>> ^C >>>>> >>>>> I'm assuming this is just a standalone 389 Directory Server you are >>>>> trying to replicate to(not a freeIPA installation). If it is a freeipa >>>>> installation, then you should use the freeipa CLI for setting up >>>>> replication. >>>>> >>>>> The error 32 (no such object) you are getting is because the replica >>>>> does not have an entry "cn=replication manager". Looking at the >>>>> replication agreement: >>>>> >>>>> nsDS5ReplicaBindDN: cn=replication manager >>>>> >>>>> This is not a valid DN as there is no base suffix: For example, I >>>>> would >>>>> expect to see something like "cn=replication manager,cn=config" >>>>> >>>>> https://access.redhat.com/documentation/en-US/Red_Hat_Direct >>>>> ory_Server/10/html/Administration_Guide/Creating_the_Supplie >>>>> r_Bind_DN_Entry.html >>>>> >>>>> Regards, >>>>> Mark >>>>> >>>>> >>>>> Please help me fix this >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>> >>>> >>>> >>>> >>> >>> >>> >>> > -- >> Manage your subscription for the Freeipa-users mailing list: >> https://www.redhat.com/mailman/listinfo/freeipa-users >> Go to http://freeipa.org for more info on the project >> > > > -- > / Alexander Bokovoy > -------------- next part -------------- An HTML attachment was scrubbed... URL: From abokovoy at redhat.com Thu Sep 1 05:57:24 2016 From: abokovoy at redhat.com (Alexander Bokovoy) Date: Thu, 1 Sep 2016 08:57:24 +0300 Subject: [Freeipa-users] Command-line replication is not works in FreeIPA-Master In-Reply-To: References: <4e37eb09-8c96-114a-aee1-3c6ee568194f@redhat.com> <20160901041455.4xsgwvv5duxloi6g@redhat.com> Message-ID: <20160901055724.lph3h65eequi4x44@redhat.com> On Thu, 01 Sep 2016, Andrey Rogovsky wrote: >Hi, Alexander! > >Thank for fast reply. >I have replication manager object: >filter: (objectclass=organizationalPerson) >requesting: All userApplication attributes ># extended LDIF ># ># LDAPv3 ># base with scope subtree ># filter: (objectclass=organizationalPerson) ># requesting: ALL ># > ># replication manager, config >dn: cn=replication manager,cn=config >objectClass: inetorgperson >objectClass: person >objectClass: top >objectClass: organizationalPerson >cn: replication manager >sn: RM >userPassword:: >e1NTSEF9d281RGZOTTlCSEVWTEhxY1lTcGs0WHdjRXplemU4S280S3EwWnc9PQ= > = > ># search result >search: 2 >result: 0 Success > ># numResponses: 2 ># numEntries: 1 > >But error is present. You have two LDAP servers. If you have replication going in both directions, you need to have the replication bind entry defined on both servers. If you have replication going in one direction, then the target server should have this replication bind entry defined. Where do you have this entry? -- / Alexander Bokovoy From a.rogovsky at gmail.com Thu Sep 1 06:50:59 2016 From: a.rogovsky at gmail.com (Andrey Rogovsky) Date: Thu, 1 Sep 2016 09:50:59 +0300 Subject: [Freeipa-users] Command-line replication is not works in FreeIPA-Master In-Reply-To: <20160901055724.lph3h65eequi4x44@redhat.com> References: <4e37eb09-8c96-114a-aee1-3c6ee568194f@redhat.com> <20160901041455.4xsgwvv5duxloi6g@redhat.com> <20160901055724.lph3h65eequi4x44@redhat.com> Message-ID: Hi, Alexander! I have ldap1 - FreeIPA (master) and ldap2 - 389DS (slave) I want one-way replica from ldap1 to ldap2 On ldap1 I was define dn replication user, replica and agreement On ldap2 I was define replica only: filter: (objectclass=nsds5replica) requesting: All userApplication attributes # extended LDIF # # LDAPv3 # base with scope subtree # filter: (objectclass=nsds5replica) # requesting: ALL # # replica, dc\3Dexample\2Cdc\3Dcom, mapping tree, config dn: cn=replica,cn=dc\3Dexample\2Cdc\3Dcom,cn=mapping tree,cn=config objectClass: top objectClass: nsds5replica objectClass: extensibleObject cn: replica nsDS5ReplicaRoot: dc=example,dc=com nsDS5ReplicaType: 2 nsDS5ReplicaBindDN: cn=replication manager,cn=config nsDS5Flags: 0 nsDS5ReplicaId: 65535 nsState:: //8AAAAAAABY2sZXAAAAAAAAAAAAAAAAAAAAAAAAAAABAAAAAAAAAA== nsDS5ReplicaName: 06154b02-6f7e11e6-b236be05-3db8a3e8 nsds5ReplicaChangeCount: 0 nsds5replicareapactive: 0 # search result search: 2 result: 0 Success # numResponses: 2 # numEntries: 1 Does I need define DN replication user on ldap2? 2016-09-01 8:57 GMT+03:00 Alexander Bokovoy : > On Thu, 01 Sep 2016, Andrey Rogovsky wrote: > >> Hi, Alexander! >> >> Thank for fast reply. >> I have replication manager object: >> filter: (objectclass=organizationalPerson) >> requesting: All userApplication attributes >> # extended LDIF >> # >> # LDAPv3 >> # base with scope subtree >> # filter: (objectclass=organizationalPerson) >> # requesting: ALL >> # >> >> # replication manager, config >> dn: cn=replication manager,cn=config >> objectClass: inetorgperson >> objectClass: person >> objectClass: top >> objectClass: organizationalPerson >> cn: replication manager >> sn: RM >> userPassword:: >> e1NTSEF9d281RGZOTTlCSEVWTEhxY1lTcGs0WHdjRXplemU4S280S3EwWnc9PQ= >> = >> >> # search result >> search: 2 >> result: 0 Success >> >> # numResponses: 2 >> # numEntries: 1 >> >> But error is present. >> > You have two LDAP servers. If you have replication going in both > directions, you need to have the replication bind entry defined on both > servers. > > If you have replication going in one direction, then the target server > should have this replication bind entry defined. > > Where do you have this entry? > > > > -- > / Alexander Bokovoy > -------------- next part -------------- An HTML attachment was scrubbed... URL: From abokovoy at redhat.com Thu Sep 1 07:01:26 2016 From: abokovoy at redhat.com (Alexander Bokovoy) Date: Thu, 1 Sep 2016 10:01:26 +0300 Subject: [Freeipa-users] Command-line replication is not works in FreeIPA-Master In-Reply-To: References: <4e37eb09-8c96-114a-aee1-3c6ee568194f@redhat.com> <20160901041455.4xsgwvv5duxloi6g@redhat.com> <20160901055724.lph3h65eequi4x44@redhat.com> Message-ID: <20160901070126.xxb4u3iyqllll2oy@redhat.com> On Thu, 01 Sep 2016, Andrey Rogovsky wrote: >Hi, Alexander! > >I have ldap1 - FreeIPA (master) and ldap2 - 389DS (slave) >I want one-way replica from ldap1 to ldap2 >On ldap1 I was define dn replication user, replica and agreement >On ldap2 I was define replica only: This is what you are doing wrong. Your ldap1 server will attempt to connect to ldap2 server using the replication user credentials. It is ldap2 which will be authenticating this request. Where would it take information about the replication user? >filter: (objectclass=nsds5replica) >requesting: All userApplication attributes ># extended LDIF ># ># LDAPv3 ># base with scope subtree ># filter: (objectclass=nsds5replica) ># requesting: ALL ># > ># replica, dc\3Dexample\2Cdc\3Dcom, mapping tree, config >dn: cn=replica,cn=dc\3Dexample\2Cdc\3Dcom,cn=mapping tree,cn=config >objectClass: top >objectClass: nsds5replica >objectClass: extensibleObject >cn: replica >nsDS5ReplicaRoot: dc=example,dc=com >nsDS5ReplicaType: 2 >nsDS5ReplicaBindDN: cn=replication manager,cn=config >nsDS5Flags: 0 >nsDS5ReplicaId: 65535 >nsState:: //8AAAAAAABY2sZXAAAAAAAAAAAAAAAAAAAAAAAAAAABAAAAAAAAAA== >nsDS5ReplicaName: 06154b02-6f7e11e6-b236be05-3db8a3e8 >nsds5ReplicaChangeCount: 0 >nsds5replicareapactive: 0 > ># search result >search: 2 >result: 0 Success > ># numResponses: 2 ># numEntries: 1 > >Does I need define DN replication user on ldap2? > > >2016-09-01 8:57 GMT+03:00 Alexander Bokovoy : > >> On Thu, 01 Sep 2016, Andrey Rogovsky wrote: >> >>> Hi, Alexander! >>> >>> Thank for fast reply. >>> I have replication manager object: >>> filter: (objectclass=organizationalPerson) >>> requesting: All userApplication attributes >>> # extended LDIF >>> # >>> # LDAPv3 >>> # base with scope subtree >>> # filter: (objectclass=organizationalPerson) >>> # requesting: ALL >>> # >>> >>> # replication manager, config >>> dn: cn=replication manager,cn=config >>> objectClass: inetorgperson >>> objectClass: person >>> objectClass: top >>> objectClass: organizationalPerson >>> cn: replication manager >>> sn: RM >>> userPassword:: >>> e1NTSEF9d281RGZOTTlCSEVWTEhxY1lTcGs0WHdjRXplemU4S280S3EwWnc9PQ= >>> = >>> >>> # search result >>> search: 2 >>> result: 0 Success >>> >>> # numResponses: 2 >>> # numEntries: 1 >>> >>> But error is present. >>> >> You have two LDAP servers. If you have replication going in both >> directions, you need to have the replication bind entry defined on both >> servers. >> >> If you have replication going in one direction, then the target server >> should have this replication bind entry defined. >> >> Where do you have this entry? >> >> >> >> -- >> / Alexander Bokovoy >> >-- >Manage your subscription for the Freeipa-users mailing list: >https://www.redhat.com/mailman/listinfo/freeipa-users >Go to http://freeipa.org for more info on the project -- / Alexander Bokovoy From a.rogovsky at gmail.com Thu Sep 1 07:13:49 2016 From: a.rogovsky at gmail.com (Andrey Rogovsky) Date: Thu, 1 Sep 2016 10:13:49 +0300 Subject: [Freeipa-users] Command-line replication is not works in FreeIPA-Master In-Reply-To: <20160901070126.xxb4u3iyqllll2oy@redhat.com> References: <4e37eb09-8c96-114a-aee1-3c6ee568194f@redhat.com> <20160901041455.4xsgwvv5duxloi6g@redhat.com> <20160901055724.lph3h65eequi4x44@redhat.com> <20160901070126.xxb4u3iyqllll2oy@redhat.com> Message-ID: Hi, Alexander! Than you very much for help. Now I able to start replica, but have one issue - schemes is not replicated: [01/Sep/2016:07:04:53 +0000] NSMMReplicationPlugin - Warning: unable to replicate schema to host ldap2, port 389. Continuing with total update session. [01/Sep/2016:07:04:53 +0000] NSMMReplicationPlugin - Beginning total update of replica "agmt="cn=ExampleAgreement" (ldap2:389)". [01/Sep/2016:07:04:53 +0000] NSMMReplicationPlugin - Need to create replication keep alive entry [01/Sep/2016:07:04:53 +0000] NSMMReplicationPlugin - add dn: cn=repl keep alive 7,dc=example,dc=com objectclass: top objectclass: ldapsubentry objectclass: extensibleObject cn: repl keep alive 7 [01/Sep/2016:07:04:58 +0000] NSMMReplicationPlugin - Finished total update of replica "agmt="cn=ExampleAgreement" (ldap2:389)". Sent 415 entries. Can you help me with schemes? 2016-09-01 10:01 GMT+03:00 Alexander Bokovoy : > On Thu, 01 Sep 2016, Andrey Rogovsky wrote: > >> Hi, Alexander! >> >> I have ldap1 - FreeIPA (master) and ldap2 - 389DS (slave) >> I want one-way replica from ldap1 to ldap2 >> On ldap1 I was define dn replication user, replica and agreement >> On ldap2 I was define replica only: >> > This is what you are doing wrong. Your ldap1 server will attempt to > connect to ldap2 server using the replication user credentials. It is > ldap2 which will be authenticating this request. Where would it take > information about the replication user? > > > filter: (objectclass=nsds5replica) >> requesting: All userApplication attributes >> # extended LDIF >> # >> # LDAPv3 >> # base with scope subtree >> # filter: (objectclass=nsds5replica) >> # requesting: ALL >> # >> >> # replica, dc\3Dexample\2Cdc\3Dcom, mapping tree, config >> dn: cn=replica,cn=dc\3Dexample\2Cdc\3Dcom,cn=mapping tree,cn=config >> objectClass: top >> objectClass: nsds5replica >> objectClass: extensibleObject >> cn: replica >> nsDS5ReplicaRoot: dc=example,dc=com >> nsDS5ReplicaType: 2 >> nsDS5ReplicaBindDN: cn=replication manager,cn=config >> nsDS5Flags: 0 >> nsDS5ReplicaId: 65535 >> nsState:: //8AAAAAAABY2sZXAAAAAAAAAAAAAAAAAAAAAAAAAAABAAAAAAAAAA== >> nsDS5ReplicaName: 06154b02-6f7e11e6-b236be05-3db8a3e8 >> nsds5ReplicaChangeCount: 0 >> nsds5replicareapactive: 0 >> >> # search result >> search: 2 >> result: 0 Success >> >> # numResponses: 2 >> # numEntries: 1 >> >> Does I need define DN replication user on ldap2? >> >> >> 2016-09-01 8:57 GMT+03:00 Alexander Bokovoy : >> >> On Thu, 01 Sep 2016, Andrey Rogovsky wrote: >>> >>> Hi, Alexander! >>>> >>>> Thank for fast reply. >>>> I have replication manager object: >>>> filter: (objectclass=organizationalPerson) >>>> requesting: All userApplication attributes >>>> # extended LDIF >>>> # >>>> # LDAPv3 >>>> # base with scope subtree >>>> # filter: (objectclass=organizationalPerson) >>>> # requesting: ALL >>>> # >>>> >>>> # replication manager, config >>>> dn: cn=replication manager,cn=config >>>> objectClass: inetorgperson >>>> objectClass: person >>>> objectClass: top >>>> objectClass: organizationalPerson >>>> cn: replication manager >>>> sn: RM >>>> userPassword:: >>>> e1NTSEF9d281RGZOTTlCSEVWTEhxY1lTcGs0WHdjRXplemU4S280S3EwWnc9PQ= >>>> = >>>> >>>> # search result >>>> search: 2 >>>> result: 0 Success >>>> >>>> # numResponses: 2 >>>> # numEntries: 1 >>>> >>>> But error is present. >>>> >>>> You have two LDAP servers. If you have replication going in both >>> directions, you need to have the replication bind entry defined on both >>> servers. >>> >>> If you have replication going in one direction, then the target server >>> should have this replication bind entry defined. >>> >>> Where do you have this entry? >>> >>> >>> >>> -- >>> / Alexander Bokovoy >>> >>> > -- >> Manage your subscription for the Freeipa-users mailing list: >> https://www.redhat.com/mailman/listinfo/freeipa-users >> Go to http://freeipa.org for more info on the project >> > > > -- > / Alexander Bokovoy > -------------- next part -------------- An HTML attachment was scrubbed... URL: From abokovoy at redhat.com Thu Sep 1 08:10:50 2016 From: abokovoy at redhat.com (Alexander Bokovoy) Date: Thu, 1 Sep 2016 11:10:50 +0300 Subject: [Freeipa-users] Command-line replication is not works in FreeIPA-Master In-Reply-To: References: <4e37eb09-8c96-114a-aee1-3c6ee568194f@redhat.com> <20160901041455.4xsgwvv5duxloi6g@redhat.com> <20160901055724.lph3h65eequi4x44@redhat.com> <20160901070126.xxb4u3iyqllll2oy@redhat.com> Message-ID: <20160901081050.txeakqy2ag4t7u3i@redhat.com> On Thu, 01 Sep 2016, Andrey Rogovsky wrote: >Hi, Alexander! > >Than you very much for help. Now I able to start replica, but have one >issue - schemes is not replicated: > >[01/Sep/2016:07:04:53 +0000] NSMMReplicationPlugin - Warning: unable to >replicate schema to host ldap2, port 389. Continuing with total update >session. >[01/Sep/2016:07:04:53 +0000] NSMMReplicationPlugin - Beginning total update >of replica "agmt="cn=ExampleAgreement" (ldap2:389)". >[01/Sep/2016:07:04:53 +0000] NSMMReplicationPlugin - Need to create >replication keep alive entry >[01/Sep/2016:07:04:53 +0000] NSMMReplicationPlugin - add dn: cn=repl keep >alive 7,dc=example,dc=com >objectclass: top >objectclass: ldapsubentry >objectclass: extensibleObject >cn: repl keep alive 7 >[01/Sep/2016:07:04:58 +0000] NSMMReplicationPlugin - Finished total update >of replica "agmt="cn=ExampleAgreement" (ldap2:389)". Sent 415 entries. > >Can you help me with schemes? I'm afraid I cannot help with that. You need to read RHDS documentation and design your own mechanism to make schema compatible on both sides. Replicating schema between IPA and plain RHDS is not really supported as nobody did that before, so you are on your own research and exploration path to see what configuration should be. https://access.redhat.com/documentation/en-US/Red_Hat_Directory_Server/10/html/Deployment_Guide/Designing_the_Replication_Process-Using_Replication_with_Other_DS_Features.html#Using_Replication_with_Other_DS_Features-Schema_Replication -- / Alexander Bokovoy From a.rogovsky at gmail.com Thu Sep 1 10:01:03 2016 From: a.rogovsky at gmail.com (Andrey Rogovsky) Date: Thu, 1 Sep 2016 13:01:03 +0300 Subject: [Freeipa-users] Command-line replication is not works in FreeIPA-Master In-Reply-To: <20160901081050.txeakqy2ag4t7u3i@redhat.com> References: <4e37eb09-8c96-114a-aee1-3c6ee568194f@redhat.com> <20160901041455.4xsgwvv5duxloi6g@redhat.com> <20160901055724.lph3h65eequi4x44@redhat.com> <20160901070126.xxb4u3iyqllll2oy@redhat.com> <20160901081050.txeakqy2ag4t7u3i@redhat.com> Message-ID: Hi, Alexander! Thank for your reply I was read your link, but it not related my issue. I will start new thread, couse replica problem is resloved. 2016-09-01 11:10 GMT+03:00 Alexander Bokovoy : > On Thu, 01 Sep 2016, Andrey Rogovsky wrote: > >> Hi, Alexander! >> >> Than you very much for help. Now I able to start replica, but have one >> issue - schemes is not replicated: >> >> [01/Sep/2016:07:04:53 +0000] NSMMReplicationPlugin - Warning: unable to >> replicate schema to host ldap2, port 389. Continuing with total update >> session. >> [01/Sep/2016:07:04:53 +0000] NSMMReplicationPlugin - Beginning total >> update >> of replica "agmt="cn=ExampleAgreement" (ldap2:389)". >> [01/Sep/2016:07:04:53 +0000] NSMMReplicationPlugin - Need to create >> replication keep alive entry >> [01/Sep/2016:07:04:53 +0000] NSMMReplicationPlugin - add dn: cn=repl keep >> alive 7,dc=example,dc=com >> objectclass: top >> objectclass: ldapsubentry >> objectclass: extensibleObject >> cn: repl keep alive 7 >> [01/Sep/2016:07:04:58 +0000] NSMMReplicationPlugin - Finished total update >> of replica "agmt="cn=ExampleAgreement" (ldap2:389)". Sent 415 entries. >> >> Can you help me with schemes? >> > I'm afraid I cannot help with that. You need to read RHDS documentation > and design your own mechanism to make schema compatible on both sides. > Replicating schema between IPA and plain RHDS is not really supported as > nobody did that before, so you are on your own research and exploration > path to see what configuration should be. > > https://access.redhat.com/documentation/en-US/Red_Hat_Direct > ory_Server/10/html/Deployment_Guide/Designing_the_ > Replication_Process-Using_Replication_with_Other_DS_Features > .html#Using_Replication_with_Other_DS_Features-Schema_Replication > > -- > / Alexander Bokovoy > -------------- next part -------------- An HTML attachment was scrubbed... URL: From a.rogovsky at gmail.com Thu Sep 1 10:13:10 2016 From: a.rogovsky at gmail.com (Andrey Rogovsky) Date: Thu, 1 Sep 2016 13:13:10 +0300 Subject: [Freeipa-users] Replication scheme problem Message-ID: Hi! I have 2 servers - ldap1 is FreeIPA (master) and ldap2 is 389 DS (slave). One way replication ldap1 -> ldap2 is enabled but scheme is not replicated: Log file ldap1 have this line: [01/Sep/2016:07:04:53 +0000] NSMMReplicationPlugin - Warning: unable to replicate schema to host ldap2, port 389. Continuing with total update session. There is current status: filter: (objectclass=nsds5replicationagreement) requesting: All userApplication attributes # extended LDIF # # LDAPv3 # base with scope subtree # filter: (objectclass=nsds5replicationagreement) # requesting: ALL # # ExampleAgreement, replica, dc\3Dexample\2Cdc\3Dcom, mapping tree, config dn: cn=ExampleAgreement,cn=replica,cn=dc\3Dexample\2Cdc\3Dcom,cn=mapping tree, cn=config objectClass: top objectClass: nsds5replicationagreement cn: ExampleAgreement nsDS5ReplicaHost: ldap2 nsDS5ReplicaPort: 389 nsDS5ReplicaBindDN: cn=replication manager,cn=config nsDS5ReplicaBindMethod: SIMPLE nsDS5ReplicaRoot: dc=example,dc=com description: agreement between supplier1 and consumer1 nsDS5ReplicaUpdateSchedule: 0000-0500 1 nsDS5ReplicatedAttributeList: (objectclass=*) $ EXCLUDE authorityRevocationLis t nsDS5ReplicaCredentials: {AES-TUhNR0NTcUdTSWIzRFFFRkRUQm1NRVVHQ1NxR1NJYjNEUUVG RERBNEJDUmxPVFl4TlRsbU5DMWtaV0UyTXpZeA0KTVMxaU1UYzFaREF3Wmkwek5qRmxNalkxWkFBQ 0FRSUNBU0F3Q2dZSUtvWklodmNOQWdjd0hRWUpZSVpJQVdVRA0KQkFFcUJCQURJbjhNUWpLM1VqU1 M1SGZEUTY0TA==}mwxKHUYWXjNeyo1AGRWe9A== nsds5replicareapactive: 0 nsds5replicaLastUpdateStart: 19700101000000Z nsds5replicaLastUpdateEnd: 19700101000000Z nsds5replicaChangesSentSinceStartup: nsds5replicaLastUpdateStatus: 0 No replication sessions started since server s tartup nsds5replicaUpdateInProgress: FALSE nsds5replicaLastInitStart: 20160901070452Z nsds5replicaLastInitEnd: 20160901070455Z nsds5replicaLastInitStatus: 0 Total update succeeded # search result search: 2 result: 0 Success # numResponses: 2 # numEntries: 1 After execute schema-reload.pl on ldap2 I have this lines in log: Failed to add task entry "cn=schema_reload_2016_9_1_10_6_17, cn=schema reload task, cn=tasks, cn=config" error (49) [01/Sep/2016:07:04:59 +0000] NSACLPlugin - Error: This ((targetattr = "gidnumber || krbprincipalname || uidnumber")(version 3.0;acl "permission:System: Read system trust accounts";allow (compare,read,search) groupdn = "ldap:///cn=System: Read system trust accounts,cn=permissions,cn=pbac,dc=example,dc=com";)) ACL will not be considered for evaluation because of syntax errors. [01/Sep/2016:07:04:59 +0000] NSACLPlugin - __aclp__init_targetattr: targetattr "ipaanchoruuid" does not exist in schema. Please add attributeTypes "ipaanchoruuid" to schema if necessary. [01/Sep/2016:07:04:59 +0000] NSACLPlugin - ACL PARSE ERR(rv=-5): (targetattr = "cn [01/Sep/2016:07:04:59 +0000] NSACLPlugin - Error: This ((targetattr = "cn || createtimestamp || description || entryusn || gidnumber || ipaanchoruuid || modifytimestamp || objectclass")(targetfilter = "(objectclass=ipaGroupOverride)")(version 3.0;acl "permission:System: Read Group ID Overrides";allow (compare,read,search) userdn = "ldap:///all";)) ACL will not be considered for evaluation because of syntax errors. [01/Sep/2016:07:04:59 +0000] NSACLPlugin - __aclp__init_targetattr: targetattr "ipaanchoruuid" does not exist in schema. Please add attributeTypes "ipaanchoruuid" to schema if necessary. [01/Sep/2016:07:04:59 +0000] NSACLPlugin - ACL PARSE ERR(rv=-5): (targetattr = "createtimestamp [01/Sep/2016:07:04:59 +0000] NSACLPlugin - Error: This ((targetattr = "createtimestamp || description || entryusn || gecos || gidnumber || homedirectory || ipaanchoruuid || ipaoriginaluid || ipasshpubkey || loginshell || modifytimestamp || objectclass || uid || uidnumber")(targetfilter = "(objectclass=ipaUserOverride)")(version 3.0;acl "permission:System: Read User ID Overrides";allow (compare,read,search) userdn = "ldap:///all";)) ACL will not be considered for evaluation because of syntax errors. [01/Sep/2016:07:04:59 +0000] NSACLPlugin - __aclp__init_targetattr: targetattr "a6record" does not exist in schema. Please add attributeTypes "a6record" to schema if necessary. [01/Sep/2016:07:04:59 +0000] NSACLPlugin - ACL PARSE ERR(rv=-5): (targetattr = "a6record [01/Sep/2016:07:04:59 +0000] NSACLPlugin - Error: This ((targetattr = "a6record || aaaarecord || afsdbrecord || aplrecord || arecord || certrecord || cn || cnamerecord || dhcidrecord || dlvrecord || dnamerecord || dnsclass || dnsttl || dsrecord || hinforecord || hiprecord || idnsallowdynupdate || idnsallowquery || idnsallowsyncptr || idnsallowtransfer || idnsforwarders || idnsforwardpolicy || idnsname || idnssecinlinesigning || idnssoaexpire || idnssoaminimum || idnssoamname || idnssoarefresh || idnssoaretry || idnssoarname || idnssoaserial || idnsupdatepolicy || idnszoneactive || ipseckeyrecord || keyrecord || kxrecord || locrecord || mdrecord || minforecord || mxrecord || naptrrecord || nsecrecord || nsec3paramrecord || nsrecord || nxtrecord || ptrrecord || rprecord || rrsigrecord || sigrecord || spfrecord || srvrecord || sshfprecord || tlsarecord || txtrecord || unknownrecord ")(target = "ldap:///idnsname=*,cn=dns,dc=example,dc=com")(version 3.0;acl "Update DNS entries in a zone";allow (write) userattr = "parent[0,1].managedby#GROUPDN";)) ACL will not be considered for evaluation because of syntax errors. Looks, like ipaanchoruuid is missing. There is ldap scheme update: filter: (objectclass=*) requesting: 00core.ldif 01core389.ldif 02common.ldif 05rfc2927.ldif 05rfc4523.ldif 05rfc4524.ldif 06inetorgperson.ldif 10automember-plugin.ldif 10dna-plugin.ldif 10mep-plugin.ldif 10rfc2307.ldif 20subscriber.ldif 25java-object.ldif 28pilot.ldif 30ns-common.ldif 50ns-admin.ldif 50ns-certificate.ldif 50ns-directory.ldif 50ns-mail.ldif 50ns-value.ldif 50ns-web.ldif 60acctpolicy.ldif 60autofs.ldif 60eduperson.ldif 60mozilla.ldif 60nss-ldap.ldif 60pam-plugin.ldif 60posix-winsync-plugin.ldif 60pureftpd.ldif 60rfc2739.ldif 60rfc3712.ldif 60sabayon.ldif 60sudo.ldif 60trust.ldif 99user.ldif No one scheme files from ldap2 have no entry for ipaanchoruuid. But from ldap1 they have: root at ldap1 schema]# grep -i ipaanchoruuid * 71idviews.ldif:attributeTypes: (2.16.840.1.113730.3.8.11.62 NAME 'ipaAnchorUUID' DESC 'Unique Anchor Identifier' EQUALITY caseIgnoreMatch ORDERING caseIgnoreOrderingMatch SYNTAX 1.3.6.1.4.1.1466.115.121.1.15 SINGLE-VALUE X-ORIGIN 'IPA v4') 71idviews.ldif:objectClasses: (2.16.840.1.113730.3.8.12.30 NAME 'ipaOverrideAnchor' SUP top STRUCTURAL MUST ( ipaAnchorUUID ) MAY ( description ) X-ORIGIN 'IPA v4' ) 71idviews.ldif:objectClasses: (2.16.840.1.113730.3.8.12.35 NAME 'ipaOverrideTarget' SUP top STRUCTURAL MUST ( ipaAnchorUUID ) X-ORIGIN 'IPA v4' ) [root at ldap1 schema]# How to resolve this issue? Just copy schemes files from ldap1 to ldap2? -------------- next part -------------- An HTML attachment was scrubbed... URL: From rcritten at redhat.com Thu Sep 1 13:47:09 2016 From: rcritten at redhat.com (Rob Crittenden) Date: Thu, 1 Sep 2016 09:47:09 -0400 Subject: [Freeipa-users] IPA port 80 In-Reply-To: References: <459142cb-4db3-3158-2b43-6e85d147736a@0xc0dedbad.com> Message-ID: <57C8315D.6010903@redhat.com> Sean Hogan wrote: > Thanks Peter, > > > So the set up is each vlan has an IPA replica within the firewall > boundary acting as its primary auth/policy server. If it goes down.. > then the clients can reach back thru the firewall to our backup IPAs. So > I am trying to pinpoint the actual ports required to be open on the > firewall to allow the clients the ability to get back to the back up IPAs. > > It comes down to opening ports thru the firewalls back to our IPA backup > servers. If port 80 is not required for the clients or servers to get to > IPA behind the firewall then there is no need in opening more ports than > required and getting 443 open adheres more to our security policy than > 80. So if everything is redirected to 443 and 80 is not required as it > is all redirected then the docs I am using are not correct. > > I am hoping Simo can weigh in on this Peter is right about OCSP/CRL. If you don't need them, and don't want a user-friendly redirect if your users don't specify https then yeah, you can probably do without port 80, assuming none of your clients REQUIRE an OCSP response (e.g. security.OCSP.require in Firefox, false by default). Another, rarely used path for port 80 is retrieval of the CA certificate when enrolling clients. Normally it is retrieved over authenticated LDAP but if that fails, and one isn't pre-positioned, it will fall back to trying to get it over port 80 (last because this isn't exactly safe). rob > > > Redhat link shows this for firewall port openings > _https://access.redhat.com/solutions/357673_ > with <-> seeming to indicate bidirectional. Not sure why NTP requires > that for the clients. > > *Resolution** > IdM Server <-> Clients* > *Name* > > *Destination-port / Type* > > *Purpose* > HTTP/HTTPS 80 / 443 TCP WebUI and IPA CLI admin tools communication. > LDAP/LDAPS 389 / 636 TCP directory service communication. > Kerberos 88 / 464 TCP and UDP communication for authentication > DNS 53 TCP and UDP nameservice, used also for autodiscovery, > autoregistration and High Availability Authentication(sssd), optional > NTP 123 UDP network time protocol, optional > kadmind 464 / 749 TCP used for principal generation, password changes etc. > > * > IdM Server <-> IdM Server (i.e. Replica)* > *Name* > > *Destination-port/Type* > > *Purpose* > HTTP/HTTPS 80 / 443 TCP WebUI and IPA CLI admin tools communication. > LDAP/LDAPS 389 / 636 TCP directory service communication. > Kerberos 88 / 464 TCP and UDP communication for authentication > DNS 53 / TCP and UDP nameservice, used also for autodiscovery, > autoregistration and High Availability Authentication(sssd), *optional* > NTP 123 UDP network time protocol, *optional* > kadmind 464 / 749 TCP used only via localhost > dogtag 7389 TCP Server and replica communication > replica conf 9443 / 9444 / 9445 TCP Recplica configuration, only needed > during initial replica installation -- IPAv3/RHEL6 only (not required at > all in IPAv4/RHEL7) > > *Note:* In RHEL 7, 389 port is used for replication instead of 7389 port. > > > Sean Hogan > > > > > > Inactive hide details for Peter Fern ---08/31/2016 04:01:30 PM---You > need to serve CRLs and OCSP via HTTP to avoid clients failPeter Fern > ---08/31/2016 04:01:30 PM---You need to serve CRLs and OCSP via HTTP to > avoid clients failing to verify the cert of the host ser > > From: Peter Fern > To: freeipa-users > Date: 08/31/2016 04:01 PM > Subject: Re: [Freeipa-users] IPA port 80 > Sent by: freeipa-users-bounces at redhat.com > > ------------------------------------------------------------------------ > > > > You need to serve CRLs and OCSP via HTTP to avoid clients failing to > verify the cert of the host serving the CRL/OCSP when the cert on that > host needs to be verified at itself. > > I'm not sure why you'd particularly care though - reading the Apache > configs and you should see that other than a couple of exceptions, all > HTTP traffic is redirected to HTTPS. > > On 01/09/16 07:22, Sean Hogan wrote: > > Hi all, > > Been reading a lot about Port 80 for IPA and firewalls but have > not found a concrete answer. I know the redhat docs indicate > port 80 is required bidirectional however I need to investigate > if it is truly needed. > > GUI only responds to 443 so not sure what else would be > utilizing port 80. I have seen some references that dogtag > proxies its ports to 80 and 443 but if the gui is running on 443 > does that mean dogtag is proxying via 443 only? Or is there a > way to tell? Has anyone attempted not opening port 80 from IPA > Server to IPA Server and clients to IPA server? > ipa-server-3.0.0-50.el6.1.x86_64 > > > > > Sean Hogan > > > > > -- > Manage your subscription for the Freeipa-users mailing list: > https://www.redhat.com/mailman/listinfo/freeipa-users > Go to http://freeipa.org for more info on the project > > > > From simo at redhat.com Thu Sep 1 13:56:06 2016 From: simo at redhat.com (Simo Sorce) Date: Thu, 01 Sep 2016 09:56:06 -0400 Subject: [Freeipa-users] IPA port 80 In-Reply-To: References: <1472682951.5257.166.camel@redhat.com> Message-ID: <1472738166.10392.13.camel@redhat.com> On Thu, 2016-09-01 at 09:33 +1000, Peter Fern wrote: > On 01/09/16 08:35, Simo Sorce wrote: > > Port 80 is not required, the only thing you'll find there is a redirect > > to the HTTPS port. > > What about CRL/OCSP (and possibly others)? The Apache configs > explicitly do not redirect to HTTPS except for the /ipa path for this > reason. Oh right Peter, you are completely right. Simo. -- Simo Sorce * Red Hat, Inc * New York From schogan at us.ibm.com Thu Sep 1 15:16:42 2016 From: schogan at us.ibm.com (Sean Hogan) Date: Thu, 1 Sep 2016 08:16:42 -0700 Subject: [Freeipa-users] IPA port 80 In-Reply-To: <57C8315D.6010903@redhat.com> References: <459142cb-4db3-3158-2b43-6e85d147736a@0xc0dedbad.com> <57C8315D.6010903@redhat.com> Message-ID: Thank You for the clarification all. Sean Hogan From: Rob Crittenden To: Sean Hogan/Durham/IBM at IBMUS, Peter Fern Cc: freeipa-users Date: 09/01/2016 06:47 AM Subject: Re: [Freeipa-users] IPA port 80 Sean Hogan wrote: > Thanks Peter, > > > So the set up is each vlan has an IPA replica within the firewall > boundary acting as its primary auth/policy server. If it goes down.. > then the clients can reach back thru the firewall to our backup IPAs. So > I am trying to pinpoint the actual ports required to be open on the > firewall to allow the clients the ability to get back to the back up IPAs. > > It comes down to opening ports thru the firewalls back to our IPA backup > servers. If port 80 is not required for the clients or servers to get to > IPA behind the firewall then there is no need in opening more ports than > required and getting 443 open adheres more to our security policy than > 80. So if everything is redirected to 443 and 80 is not required as it > is all redirected then the docs I am using are not correct. > > I am hoping Simo can weigh in on this Peter is right about OCSP/CRL. If you don't need them, and don't want a user-friendly redirect if your users don't specify https then yeah, you can probably do without port 80, assuming none of your clients REQUIRE an OCSP response (e.g. security.OCSP.require in Firefox, false by default). Another, rarely used path for port 80 is retrieval of the CA certificate when enrolling clients. Normally it is retrieved over authenticated LDAP but if that fails, and one isn't pre-positioned, it will fall back to trying to get it over port 80 (last because this isn't exactly safe). rob > > > Redhat link shows this for firewall port openings > _https://access.redhat.com/solutions/357673_ > with <-> seeming to indicate bidirectional. Not sure why NTP requires > that for the clients. > > *Resolution** > IdM Server <-> Clients* > *Name* > > *Destination-port / Type* > > *Purpose* > HTTP/HTTPS 80 / 443 TCP WebUI and IPA CLI admin tools communication. > LDAP/LDAPS 389 / 636 TCP directory service communication. > Kerberos 88 / 464 TCP and UDP communication for authentication > DNS 53 TCP and UDP nameservice, used also for autodiscovery, > autoregistration and High Availability Authentication(sssd), optional > NTP 123 UDP network time protocol, optional > kadmind 464 / 749 TCP used for principal generation, password changes etc. > > * > IdM Server <-> IdM Server (i.e. Replica)* > *Name* > > *Destination-port/Type* > > *Purpose* > HTTP/HTTPS 80 / 443 TCP WebUI and IPA CLI admin tools communication. > LDAP/LDAPS 389 / 636 TCP directory service communication. > Kerberos 88 / 464 TCP and UDP communication for authentication > DNS 53 / TCP and UDP nameservice, used also for autodiscovery, > autoregistration and High Availability Authentication(sssd), *optional* > NTP 123 UDP network time protocol, *optional* > kadmind 464 / 749 TCP used only via localhost > dogtag 7389 TCP Server and replica communication > replica conf 9443 / 9444 / 9445 TCP Recplica configuration, only needed > during initial replica installation -- IPAv3/RHEL6 only (not required at > all in IPAv4/RHEL7) > > *Note:* In RHEL 7, 389 port is used for replication instead of 7389 port. > > > Sean Hogan > > > > > > Inactive hide details for Peter Fern ---08/31/2016 04:01:30 PM---You > need to serve CRLs and OCSP via HTTP to avoid clients failPeter Fern > ---08/31/2016 04:01:30 PM---You need to serve CRLs and OCSP via HTTP to > avoid clients failing to verify the cert of the host ser > > From: Peter Fern > To: freeipa-users > Date: 08/31/2016 04:01 PM > Subject: Re: [Freeipa-users] IPA port 80 > Sent by: freeipa-users-bounces at redhat.com > > ------------------------------------------------------------------------ > > > > You need to serve CRLs and OCSP via HTTP to avoid clients failing to > verify the cert of the host serving the CRL/OCSP when the cert on that > host needs to be verified at itself. > > I'm not sure why you'd particularly care though - reading the Apache > configs and you should see that other than a couple of exceptions, all > HTTP traffic is redirected to HTTPS. > > On 01/09/16 07:22, Sean Hogan wrote: > > Hi all, > > Been reading a lot about Port 80 for IPA and firewalls but have > not found a concrete answer. I know the redhat docs indicate > port 80 is required bidirectional however I need to investigate > if it is truly needed. > > GUI only responds to 443 so not sure what else would be > utilizing port 80. I have seen some references that dogtag > proxies its ports to 80 and 443 but if the gui is running on 443 > does that mean dogtag is proxying via 443 only? Or is there a > way to tell? Has anyone attempted not opening port 80 from IPA > Server to IPA Server and clients to IPA server? > ipa-server-3.0.0-50.el6.1.x86_64 > > > > > Sean Hogan > > > > > -- > Manage your subscription for the Freeipa-users mailing list: > https://www.redhat.com/mailman/listinfo/freeipa-users > Go to http://freeipa.org for more info on the project > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: graycol.gif Type: image/gif Size: 105 bytes Desc: not available URL: From mbasti at redhat.com Thu Sep 1 16:27:15 2016 From: mbasti at redhat.com (Martin Basti) Date: Thu, 1 Sep 2016 18:27:15 +0200 Subject: [Freeipa-users] Announcing FreeIPA 4.4.1 Message-ID: <13e4828e-bfcd-6bc7-6dde-52a9902be9b6@redhat.com> The FreeIPA team would like to announce FreeIPA v4.4.1 release! It can be downloaded from http://www.freeipa.org/page/Downloads. Builds for Fedora 24 will be available in the official COPR repository . == Highlights in 4.4.1 == === Enhancements === * Kerberos KDC now takes Authentication Indicators into account when issuing service tickets. This allows, for example, to require two-factor authenticated Kerberos credentials prior to obtaining tickets to a VPN service. * FreeIPA Certificate Authority now is able to create subordinate CAs to issue certificates with a specific scope * Web UI and API end-points now can be configured to log-in with client certificates and smart cards. Additional configuration details are described in the External Authentication design page . * Web UI now suggests to have redundancy in Certificate Authority topology * Custom FreeIPA plugins can now be built without modifying core FreeIPA code * When establishing trust to an Active Directory forest, FreeIPA now is capable on automatically resolving DNS namespace conflicts with another Active Directory forest. === Known Issues === * Interactive CLI input for dnsrecord-* commands does not work properly for multipart records * ipa-ca-install fails on replica when master is CA-less * Lightweight sub-CA certs are not tracked by certmonger after `ipa-replica-install` * Certificate revocation in service-del and host-del isn't aware of Sub CAs and causes command to fail when Sub CA cert is used === Bug fixes === FreeIPA 4.4.1 is a stabilization release for the features delivered as a part of 4.4.0. There are more than 140 bug-fixes which details can be seen in the list of resolved tickets below. == Upgrading == Upgrade instructions are available on [[Upgrade]] page. == Feedback == Please provide comments, bugs and other feedback via the freeipa-users mailing list (http://www.redhat.com/mailman/listinfo/freeipa-users) or #freeipa channel on Freenode. == Detailed changelog since 4.4.0 == === Abhijeet Kasurde (4) === * Minor fix in ipa-replica-manage MAN page * Corrected minor spell check in AD Trust information doc messages * Removed unwanted line break from RefererError Dialog message * Handled empty hostname in server-del command === Alexander Bokovoy (9) === * service: add flag to allow S4U2Self * support schema files from third-party plugins * ipaserver/dcerpc: reformat to make the code closer to pep8 * trust: automatically resolve DNS trust conflicts for triangle trusts * trust: make sure external trust topology is correctly rendered * trust: make sure ID range is created for the child domain even if it exists * ipa-kdb: simplify trusted domain parent search * support multiple uid values in schema compatibility tree * freeipa.spec.in: move ipa CLI utility to freeipa-client === Ben Lipton (3) === * Fix several small typos * Use existing HostKey config to test sshd * Silence sshd messages during install === Christian Heimes (5) === * Correct path to HTTPD's systemd service directory * RedHatCAService should wait for local Dogtag instance * Remove Custodia server keys from LDAP * Secure permissions of Custodia server.keys * Require httpd 2.4.6-31 with mod_proxy Unix socket support === David Kupka (21) === * schema: Fix subtopic -> topic mapping * help: Add dnsserver commands to help topic 'dns' * vault: Catch correct exception in decrypt * schema: Speed up schema cache * frontend: Change doc, summary, topic and NO_CLI to class properties * schema: Introduce schema cache format * schema: Generate bits for help load them on request * help: Do not create instances to get information about commands and topics * compat: Save server's API version in for pre-schema servers * schema cache: Do not reset ServerInfo dirty flag * schema cache: Do not read fingerprint and format from cache * Access data for help separately * frontent: Add summary class property to CommandOverride * schema cache: Read server info only once * schema cache: Store API schema cache in memory * client: Do not create instance just to check isinstance * schema cache: Read schema instead of rewriting it when SchemaUpToDate * schema check: Check current client language against cached one * compat: Fix ping command call * schema cache: Fallback to 'en_us' when locale is not available * otptoken, permission: Convert custom type parameters on server === Florence Blanc-Renaud (4) === * Show full error message for selinuxusermap-add-hostgroup * server uninstall fails to remove krb principals * Fix session cookies * Fix ipa hbactest output === Fraser Tweedale (11) === * uninstall: untrack lightweight CA certs * caacl: expand plugin documentation * spec: require Dogtag >= 10.3.3-3 * Create server and host certs with DNS altname * caacl: fix regression in rule instantiation * cert-revoke: fix permission check bypass (CVE-2016-5404) * Move GeneralName parsing code to ipalib.x509 * x509: fix SAN directoryName parsing * x509: use NSS enums and OIDs to identify SAN types * x509: include otherName DER value in GeneralNameInfo * cert-show: show subject alternative names === Ganna Kaihorodova (2) === * Fix conflict between "got" and "expected" values * Fix for integration tests replication layouts === Jan Cholasta (19) === * frontend: copy command arguments to output params on client * Revert "Enable vault-* commands on client" * client: fix hiding of commands which lack server support * compat: fix ping call * install: fix external CA cert validation * vault: add missing salt option to vault_mod * Revert "spec: add conflict with bind-chroot to freeipa-server-dns" * parameters: move the `confirm` kwarg to Param * client: add missing output params to client-side commands * cert: speed up cert-find * cert: do not crash on invalid data in cert-find * server install: do not prompt for cert file PIN repeatedly * tests: fix test_ipalib.test_frontend.test_Object * custodia: include known CA certs in the PKCS#12 file for Dogtag * cert: add missing param values to cert-find output * cert: include CA name in cert command output * rpcserver: assume version 1 for unversioned command calls * custodia: force reconnect before retrieving CA certs from LDAP * rpcserver: fix crash in XML-RPC system commands === Lenka Doudova (26) === * Tests: Tracker class for services * Tests: Authentication indicators xmlrpc tests * Tests: Authentication indicators integration tests * Tests: External trust * Tests: Support of UPN for trusted domains * Tests: Improve handling of rename operation by user tracker * Tests: IPA user can kinit using enterprise principal with IPA domain * Tests: Removing manipulation with /etc/hosts file from integration tests * Tests: Remove has_keytab from list of expected keys of update command * Tests: Add data attribute to messages * Tests: test_ipalib/test_output fails due to change of Output behaviour * Fix malformed or missing docstrings in ipalib/messages * Tests: Fix failing tests in test_ipalib/test_parameters * Tests: Fix failing tests in test_ipalib/test_frontend * Tests: ID views tests do not recognize ipakrboktoauthasdelegate sttribute * Tests: Duplicate declaration on variables in ID views tests * Tests: ID views tests do not recognize krbcanonicalname attribute * Tests: Host tracker does not recognize 'ipakrboktoauthasdelegate' attribute * Tests: Service tracker and tests don't recognize 'ipakrboktoauthasdelegate' attribute * Tests: Failing test_ipalib/test_rpc * Tests: Failing test_ipaserver/test_ldap test * Tests: Failing tests in test_ipalib/test_plugable * Raise error when running ipa-adtrust-install with empty netbios--name * Tests: Random issuer certificate can be added to a service * Tests: Add missing attributes to test_xmlrpc/test_trust tests * Tests: Avoid skipping tests due to missing files === Luk?? Slebodn?k (4) === * ipa_pwd_extop: Fix warning declaration shadows previous local * ipa-pwd-extop: Fix warning assignment discards ?const? qualifier from pointer * ipa-kdb: Allow to build with samba 4.5 * ipa-kdb: Fix unit test after packaging changes in krb5 === Martin Babinsky (20) === * Fix incorrect check for principal type when evaluating CA ACLs * ipa-nis-manage: Use server API to retrieve plugin status * ipa-compat-manage: use server API to retrieve plugin status * ipa-advise: correct handling of plugin namespace iteration * vault-add: set the default vault type on the client side if none was given * Preserve user principal aliases during rename operation * messages: specify message type for ResultFormattingError * DNS install: Ensure that DNS servers container exists * Use server API in com.redhat.idm.trust-fetch-domains oddjob helper * allow 'value' output param in commands without primary key * allow multiple dashes in the components of server hostname * expose `--secret` option in radiusproxy-* commands * prevent search for RADIUS proxy servers by secret * trust-add: handle `--all/--raw` options properly * baseldap: Fix MidairCollision instantiation during entry modification * Create indexes for krbCanonicalName attribute * harden the check for trust namespace overlap in new principals * re-set canonical principal name on migrated users * add python-libsss_nss_idmap and python-sss to BuildRequires * do not use trusted forest name to construct domain admin principal === Martin Ba?ti (18) === * Enable vault-* commands on client * host-find: do not show SSH key by default * CI: DNS locations * Host-del: fix behavior of --updatedns and PTR records * DNS Locations: fix update-system-records unpacking error * Use copy when replacing files to keep SELinux context * CI tests: improve log collecting * CI tests: fix SSSD log collecting * idrange: fix unassigned global variable * Do not initialize API in ipa-client-automount uninstall * Increase default length of auto generated passwords * ipa-backup: backup /etc/tmpfiles.d/dirsrv-.conf * Fix: container owner should be able to add vault * Remove forgotten print from DN.__str__ implementation * Raise DuplicatedEnrty error when user exists in delete_container * Update translations * Print to debug output answer from CA * Revert "Enable LDAPS in replica promotion" === Milan Kub?k (12) === * ipatests: Tracker implementation for Sub CA feature * ipatests: Extend CAACL suite to cover Sub CA members * ipatests: Test Sub CA with CAACL and certificate profile * ipatests: remove ipacertbase option from test CSR configuration * ipatests: Add tracker class for kerberos principal aliases * ipatests: Extend the MockLDAP utility class * ipatests: Provide a context manager for mocking a trust in RPC tests * ipatests: Move trust mock helper functions to a separate module * ipapython: Extend kinit_password to support principal canonicalization * ipatests: Allow change_principal context manager to use canonicalization * ipatests: Add kerberos principal alias tests * ipatests: Fix wrong fixture in kerberos principal alias test === Oleg Fayans (7) === * Test for incorrect client domain * Fixed import error * Fixed incorrect return code assert * Fixed incorrect domainlevel determination in tests * Fixed incorrect sequence of method calls in tasks.py * Added a sleep interval after domainlevel raise in tests * Disabled raiseonerr in kinit call during topology level check === Pavel Vomacka (12) === * Close host adder dialog before showing 4304 dialog * Remove navigation using breadcrumb menus * Fix test_navigation tests * Fix test which checks removing of user * Set default delete action name to 'delete' * Remove full name from adding user to user group dialog * Add function which check whether the field is empty * Add jslint into Makefile * Fix unicode characters in ca and domain adders * Add warning about only one existing CA server * Set servers list as default facet in topology facet group * Add 'trusted to auth as user' checkbox === Peter Lacko (1) === * Test URIs in certificate. === Petr Voborn?k (2) === * unite log file name of ipa-ca-install * ca-less tests: fix getting cert in pem format from nssdb === Petr ?pa?ek (15) === * client-install: log exceptions from certmonger.request_cert * replica-install: Fix --domain * Fix ipa-replica-prepare's error message about missing local CA instance * client: RPM require initscripts to get *-domainname.service * server-install: Fix --hostname option to always override api.env values * install: Call hostnamectl set-hostname only if --hostname option is used * DNS server upgrade: do not fail when DNS server did not respond * server upgrade: do not start BIND if it was not running before the upgrade * DNS: allow to add forward zone to already broken sub-domain * adtrust-install: Mention AD GC port 3286 in list of required ports. * config-mod: normalize attribute names for --usersearch/--groupsearch * migrate-ds: Mention --enable-migration in error message about migration mode * Fix man page ipa-replica-manage: remove duplicate -c option from --no-lookup * Tests: fix test_forward_zones in test_xmlrpc/test_dns_plugin * Tests: fix test_forward_zones in test_xmlrpc/test_dns_plugin === Simo Sorce (4) === * Simplify date manipulation in pwd plugin * Regenerate asn1 code * Additional coverity fixes. * Fix CA ACL Check on SubjectAltNames === Stanislav Laznicka (7) === * Removed unused method parameter from migrate-ds * Improvements for the ipa-cacert-manage man and help * Removed objectclass from LDAP*ReverseMember based tests * Don't show --force-ntpd option in replica install * Remove sys.exit from install modules and scripts * Fail on topology disconnect/last role removal * Don't ignore --ignore-last-of-role for last CA === Sumit Bose (1) === * kdb: check for local realm in enterprise principals === Thierry Bordaz (2) === * Heap corruption in ipapwd plugin * ipa-pwd-extop memory leak during passord update === Tiboris (1) === * Added new authentication method === Tomas Krizek (5) === * Update ipa-replica-install documentation * Fix ipa-caalc-add-service error message * Validate key in otptoken-add * Fix ipa-server-install in pure IPv6 environment * Enable LDAPS in replica promotion === gkaihoro (1) === * Test for caacl-add-service === tester (4) === * Add possibility to choose parent element by css * TEST: managing user certificates * TEST: managing host certificates * TEST: managing service certificates From william.muriithi at gmail.com Thu Sep 1 19:20:13 2016 From: william.muriithi at gmail.com (William Muriithi) Date: Thu, 1 Sep 2016 15:20:13 -0400 Subject: [Freeipa-users] openLDAP to FreeIPA user migration Message-ID: Afternoon, I have an openLDAP system that lack a required attribute. This result in the migration script rejecting all the user import. I have googled externsively, read ever line of ipa migration --help doc and it doesn't seem I will be able to use this migration script. I wonder if there is anybody here who have been able to overcome this problem in the past. [root at hydrogen ~]# ipa -v migrate-ds --with-compat --bind-dn="cn=admin,dc=eng.example,dc=com" --user-ignore-attribute="sn" --user-container="ou=People,dc=eng.example,dc=com" --group-container="ou=Group,dc=eng.example,dc=com" --group-objectclass="posixGroup" --user-objectclass="account" ldap://192.168.20.18:389 ipa: INFO: trying https://hydrogen.eng.example.com/ipa/session/json Password: ipa: INFO: Forwarding 'migrate_ds' to json server 'https://hydrogen.eng.example.com/ipa/session/json' ----------- migrate-ds: ----------- Migrated: Failed user: aagrim: missing attribute "sn" required by object class "organizationalPerson" acctemp: missing attribute "sn" required by object class "organizationalPerson" ........... Regards, William From abokovoy at redhat.com Thu Sep 1 19:26:57 2016 From: abokovoy at redhat.com (Alexander Bokovoy) Date: Thu, 1 Sep 2016 22:26:57 +0300 Subject: [Freeipa-users] openLDAP to FreeIPA user migration In-Reply-To: References: Message-ID: <20160901192657.mamcazshrv2ircb6@redhat.com> On Thu, 01 Sep 2016, William Muriithi wrote: >Afternoon, > >I have an openLDAP system that lack a required attribute. This result >in the migration script rejecting all the user import. > >I have googled externsively, read ever line of ipa migration --help >doc and it doesn't seem I will be able to use this migration script. >I wonder if there is anybody here who have been able to overcome this >problem in the past. > >[root at hydrogen ~]# ipa -v migrate-ds --with-compat >--bind-dn="cn=admin,dc=eng.example,dc=com" >--user-ignore-attribute="sn" >--user-container="ou=People,dc=eng.example,dc=com" >--group-container="ou=Group,dc=eng.example,dc=com" >--group-objectclass="posixGroup" --user-objectclass="account" >ldap://192.168.20.18:389 >ipa: INFO: trying https://hydrogen.eng.example.com/ipa/session/json >Password: >ipa: INFO: Forwarding 'migrate_ds' to json server >'https://hydrogen.eng.example.com/ipa/session/json' >----------- >migrate-ds: >----------- >Migrated: >Failed user: > aagrim: missing attribute "sn" required by object class "organizationalPerson" > acctemp: missing attribute "sn" required by object class >"organizationalPerson" > ........... This looks like a common problem. I had recently made a small 'hack' to solve this problem. Following small fixup plugin could be used to affect how entries are generated. If you add it to /usr/lib/python2.7/site-packages/ipalib/plugins on IPA master and restart httpd service, the plugin would modify migrate-ds command so that 'sn' attribute would be set to a 'Migrated User Last Name' for all entries that miss 'sn' attribute before they actually get added into IPA LDAP. This is an experimental hack, of course, but it should work. Once migration is finished, don't forget to remove the file and restart httpd service again. -- / Alexander Bokovoy -------------- next part -------------- from .migration import migrate_ds _fixup_pre_callback_user = None def _pre_callback_user(ldap, pkey, dn, entry_attrs, failed, config, ctx, **kwargs): dn = _fixup_pre_callback_user(ldap, pkey, dn, entry_attrs, failed, config, ctx, **kwargs) if entry_attrs.get('sn', None) is None: entry_attrs['sn'] = [u'Migrated User Last Name'] return dn _fixup_pre_callback_user = migrate_ds.migrate_objects['user']['pre_callback'] migrate_ds.migrate_objects['user']['pre_callback'] = _pre_callback_user From kashmancy at gmail.com Fri Sep 2 00:05:28 2016 From: kashmancy at gmail.com (Harry Kashouli) Date: Thu, 1 Sep 2016 17:05:28 -0700 Subject: [Freeipa-users] Remote users and passwords Message-ID: Hi all, I have FreeIPA set up on my home server (Fedora 24), and I would like for remote users to be able to set up new passwords, after I set them up with a default one. Most likely, they will be running Windows. What is the best/suggested/correct method to do this? Thanks, -Harry -------------- next part -------------- An HTML attachment was scrubbed... URL: From lslebodn at redhat.com Fri Sep 2 07:27:57 2016 From: lslebodn at redhat.com (Lukas Slebodnik) Date: Fri, 2 Sep 2016 09:27:57 +0200 Subject: [Freeipa-users] SUDO and group lookup in AD trust In-Reply-To: <20160826055407.sok4s6c7ehtwoe3q@hendrix> References: <2050149863.1012561.1471958268423.JavaMail.zimbra@casalogic.dk> <20160824075009.rj73yous3ptypj54@hendrix> <1433616466.7119.1472107348774.JavaMail.zimbra@casalogic.dk> <1121029446.9457.1472109874936.JavaMail.zimbra@casalogic.dk> <260135086.10187.1472112336235.JavaMail.zimbra@casalogic.dk> <20160825084820.GB30315@10.4.128.1> <1042341051.13710.1472117452652.JavaMail.zimbra@casalogic.dk> <20160825204152.GA6360@10.4.128.1> <20160826055407.sok4s6c7ehtwoe3q@hendrix> Message-ID: <20160902072757.GA6086@10.4.128.1> On (26/08/16 07:54), Jakub Hrozek wrote: >On Thu, Aug 25, 2016 at 10:41:53PM +0200, Lukas Slebodnik wrote: >> On (25/08/16 11:30), Troels Hansen wrote: >> >Hmm, adding the CentOS SSSD 1.14 copr repo and running yum upgrade, >> >getting a version 1.14.1, clean cache DB (complaing about cache being >> >old version), >> Upgrade to 1.14.1 should not require puring sssd cache. >> If you are able to reproduce then please provide steps. >> Or file a sssd bug https://fedorahosted.org/sssd/newticket >> >> >I can getent users, but log log in for no obvious reason (system error in secure.log). >> > >> system error sounds bad. Please provide log files with >> high debug level in domain section sssd.conf >> >> https://fedorahosted.org/sssd/wiki/Troubleshooting#SSSDdebuglogs > >We were debugging this yesterday with Troels and the logs said it's: > https://fedorahosted.org/sssd/ticket/3127 > Fixed version is in 1.14 copr LS From jhrozek at redhat.com Fri Sep 2 07:56:29 2016 From: jhrozek at redhat.com (Jakub Hrozek) Date: Fri, 2 Sep 2016 09:56:29 +0200 Subject: [Freeipa-users] SUDO and group lookup in AD trust In-Reply-To: <20160902072757.GA6086@10.4.128.1> References: <2050149863.1012561.1471958268423.JavaMail.zimbra@casalogic.dk> <20160824075009.rj73yous3ptypj54@hendrix> <1433616466.7119.1472107348774.JavaMail.zimbra@casalogic.dk> <1121029446.9457.1472109874936.JavaMail.zimbra@casalogic.dk> <260135086.10187.1472112336235.JavaMail.zimbra@casalogic.dk> <20160825084820.GB30315@10.4.128.1> <1042341051.13710.1472117452652.JavaMail.zimbra@casalogic.dk> <20160825204152.GA6360@10.4.128.1> <20160826055407.sok4s6c7ehtwoe3q@hendrix> <20160902072757.GA6086@10.4.128.1> Message-ID: <20160902075629.djs3yy22zk4p3kiz@hendrix> On Fri, Sep 02, 2016 at 09:27:57AM +0200, Lukas Slebodnik wrote: > On (26/08/16 07:54), Jakub Hrozek wrote: > >On Thu, Aug 25, 2016 at 10:41:53PM +0200, Lukas Slebodnik wrote: > >> On (25/08/16 11:30), Troels Hansen wrote: > >> >Hmm, adding the CentOS SSSD 1.14 copr repo and running yum upgrade, > >> >getting a version 1.14.1, clean cache DB (complaing about cache being > >> >old version), > >> Upgrade to 1.14.1 should not require puring sssd cache. > >> If you are able to reproduce then please provide steps. > >> Or file a sssd bug https://fedorahosted.org/sssd/newticket > >> > >> >I can getent users, but log log in for no obvious reason (system error in secure.log). > >> > > >> system error sounds bad. Please provide log files with > >> high debug level in domain section sssd.conf > >> > >> https://fedorahosted.org/sssd/wiki/Troubleshooting#SSSDdebuglogs > > > >We were debugging this yesterday with Troels and the logs said it's: > > https://fedorahosted.org/sssd/ticket/3127 > > > Fixed version is in 1.14 copr Thank you, btw another affected user confirmed that the patch fixes the problem. From rene.trippen at gmail.com Fri Sep 2 09:39:00 2016 From: rene.trippen at gmail.com (Rene Trippen) Date: Fri, 2 Sep 2016 11:39:00 +0200 Subject: [Freeipa-users] Migrate users with password from one IPA to another In-Reply-To: References: <57BF2E8C.80800@redhat.com> Message-ID: Hi, is it possible to transfer the Kerberos Master Key to the new IPA Server? - rene On 31.08.2016 10:57, Rene Trippen wrote: > On 25.08.2016 19:44, Rob Crittenden wrote: >> Rene Trippen wrote: >>> Hi, >>> >>> I`ve got an IPA with a broken CA infrastructure (don`t know what >>> happened, but new clients cannot be registered) >>> It is even not possible to setup a new replica. >> >> It may be fairly straightforward to getting the CA back up. How is it >> broken? >> > I don't know how that happened exactly, we had an IPA 3.x Server, then > we migrated it to another machine and upgraded to IPA 4.1, later, we > upgraded (on the same machine) to IPA 4.2. > The IPA Server is basically working, but when I want to register a new > machine, the registration process fails with following (I think these > are the relevant lines) error > > 2016-08-30T22:40:25Z DEBUG flushing ldap://ipa.internal.domain:389 > from SchemaCache > 2016-08-30T22:40:25Z DEBUG retrieving schema for SchemaCache > url=ldap://ipa.internal.domain:389 > conn= > 2016-08-30T22:40:26Z DEBUG Adding CA certificates to the IPA NSS > database. > 2016-08-30T22:40:26Z DEBUG Starting external process > 2016-08-30T22:40:26Z DEBUG args='/usr/bin/certutil' '-d' > '/etc/ipa/nssdb' '-A' '-n' 'INTERNAL.DOMAIN IPA CA' '-t' 'CT,C,C' > 2016-08-30T22:40:26Z DEBUG Process finished, return code=0 > 2016-08-30T22:40:26Z DEBUG stdout= > 2016-08-30T22:40:26Z DEBUG stderr= > 2016-08-30T22:40:26Z DEBUG Starting external process > 2016-08-30T22:40:26Z DEBUG args='/usr/bin/certutil' '-d' > '/etc/ipa/nssdb' '-A' '-n' 'INTERNAL.DOMAIN IPA CA' '-t' 'CT,C,C' > 2016-08-30T22:40:26Z DEBUG Process finished, return code=255 > 2016-08-30T22:40:26Z DEBUG stdout= > 2016-08-30T22:40:26Z DEBUG stderr=certutil: could not add certificate > to token or database: SEC_ERROR_ADDING_CERT: Error adding certificate > to database. > > 2016-08-30T22:40:26Z ERROR Failed to add INTERNAL.DOMAIN IPA CA to the > IPA NSS database. > 2016-08-30T22:40:26Z ERROR Installation failed. Rolling back changes. > > > The client tries to add 2 certificates, but fails with the second, I > think, it is because we have 2 CA certificates (one from the old IPA > 3.x server and one from the new 4.x server). My current workaround is > to register the client with an ipa3.x client, then I do an upgrade to > the 4.x client > > I've tried many ways to setup a new CA: > - tried ipa-cacert-manage renew > - tried to setup a new replica with new CA, but the setup failed with > the same problems described above > - tried to remove all old certificates refering to the old ipa server > (but I think I failed somewhere) > > My thoughts are, the CA is in a bad condition, and I spent much time > in trying to fix it, with no success. And, my fears are, if I find > some crude, not documented workaround for the CA problem, the problem > maybe pops up at the next update. So, setting up a fresh IPA and > migrating everything (except the clients), was my hope to get an IPA > running without all the CA problems. Migrating the clients is not the > problem, that can be done by script (spacewalk or ansible), but > migrating the users is not that easy, because the users cannot be > scripted :) > > >>> So, I wanted to setup a new IPA Server with new CA, and I want to move >>> all users with their passwords to the new IPA instance. >>> I`ve tried with 'ipa migrate-ds' >>> >>> ipa migrate-ds --continue --bind-dn="cn=Directory Manager" >>> --user-container=cn=users,cn=accounts >>> --group-container=cn=groups,cn=accounts --group-objectclass=posixgroup >>> --group-overwrite-gid --with-compat ldap:// >>> >>> The output is OK >>> ======= >>> Passwords have been migrated in pre-hashed format. >>> IPA is unable to generate Kerberos keys unless provided >>> with clear text passwords. All migrated users need to >>> login at https://your.domain/ipa/migration/ before they >>> can use their Kerberos accounts. >>> ======== >>> >>> But the ipa/migration website is not working for me. >>> Anyway, is there a way to export the users with passwords? I think I >>> have to export some kerberos specific stuff from the old IPA? >> >> The log file /var/log/httpd/error_log may have details on what isn't >> working. > > Sorry, that was not clearly described: > > The site is basically working, but when I enter the password, nothing > happens in the backend (I cannot login with my user on the ipa login > site). > > - rene > >> >> The way to export users with passwords is the method you've already >> tried. To not have to change a password at all would require the same >> Kerberos master key and these are generated randomly at install time. >> >> rob >> > From ezajko at root.ba Fri Sep 2 10:24:17 2016 From: ezajko at root.ba (Ernedin Zajko) Date: Fri, 2 Sep 2016 12:24:17 +0200 Subject: [Freeipa-users] openLDAP to FreeIPA user migration In-Reply-To: <20160901192657.mamcazshrv2ircb6@redhat.com> References: <20160901192657.mamcazshrv2ircb6@redhat.com> Message-ID: Hi Alexander, thank you for this - i think this should even work for missing some mandatory (gid) attributes... regards, --- Ernedin ZAJKO ezajko at root.ba > 340282366920938463463374607431768211456 On Thu, Sep 1, 2016 at 9:26 PM, Alexander Bokovoy wrote: > On Thu, 01 Sep 2016, William Muriithi wrote: >> >> Afternoon, >> >> I have an openLDAP system that lack a required attribute. This result >> in the migration script rejecting all the user import. >> >> I have googled externsively, read ever line of ipa migration --help >> doc and it doesn't seem I will be able to use this migration script. >> I wonder if there is anybody here who have been able to overcome this >> problem in the past. >> >> [root at hydrogen ~]# ipa -v migrate-ds --with-compat >> --bind-dn="cn=admin,dc=eng.example,dc=com" >> --user-ignore-attribute="sn" >> --user-container="ou=People,dc=eng.example,dc=com" >> --group-container="ou=Group,dc=eng.example,dc=com" >> --group-objectclass="posixGroup" --user-objectclass="account" >> ldap://192.168.20.18:389 >> ipa: INFO: trying https://hydrogen.eng.example.com/ipa/session/json >> Password: >> ipa: INFO: Forwarding 'migrate_ds' to json server >> 'https://hydrogen.eng.example.com/ipa/session/json' >> ----------- >> migrate-ds: >> ----------- >> Migrated: >> Failed user: >> aagrim: missing attribute "sn" required by object class >> "organizationalPerson" >> acctemp: missing attribute "sn" required by object class >> "organizationalPerson" >> ........... > > This looks like a common problem. I had recently made a small 'hack' to > solve this problem. > > Following small fixup plugin could be used to affect how entries are > generated. If you add it to /usr/lib/python2.7/site-packages/ipalib/plugins > on IPA master and restart httpd service, the plugin would modify migrate-ds > command so > that 'sn' attribute would be set to a 'Migrated User Last Name' for all > entries that miss 'sn' attribute before they actually get added into IPA > LDAP. > > This is an experimental hack, of course, but it should work. Once > migration is finished, don't forget to remove the file and restart httpd > service again. > > -- > / Alexander Bokovoy > > -- > Manage your subscription for the Freeipa-users mailing list: > https://www.redhat.com/mailman/listinfo/freeipa-users > Go to http://freeipa.org for more info on the project From abokovoy at redhat.com Fri Sep 2 10:31:11 2016 From: abokovoy at redhat.com (Alexander Bokovoy) Date: Fri, 2 Sep 2016 13:31:11 +0300 Subject: [Freeipa-users] openLDAP to FreeIPA user migration In-Reply-To: References: <20160901192657.mamcazshrv2ircb6@redhat.com> Message-ID: <20160902103111.hs6ww45kfpc72iyw@redhat.com> On Fri, 02 Sep 2016, Ernedin Zajko wrote: >Hi Alexander, > >thank you for this - i think this should even work for missing some >mandatory (gid) attributes... Yes, this fixup module can be used for anything to inject. > >regards, > >--- Ernedin ZAJKO > ezajko at root.ba > >> 340282366920938463463374607431768211456 > > > >On Thu, Sep 1, 2016 at 9:26 PM, Alexander Bokovoy wrote: >> On Thu, 01 Sep 2016, William Muriithi wrote: >>> >>> Afternoon, >>> >>> I have an openLDAP system that lack a required attribute. This result >>> in the migration script rejecting all the user import. >>> >>> I have googled externsively, read ever line of ipa migration --help >>> doc and it doesn't seem I will be able to use this migration script. >>> I wonder if there is anybody here who have been able to overcome this >>> problem in the past. >>> >>> [root at hydrogen ~]# ipa -v migrate-ds --with-compat >>> --bind-dn="cn=admin,dc=eng.example,dc=com" >>> --user-ignore-attribute="sn" >>> --user-container="ou=People,dc=eng.example,dc=com" >>> --group-container="ou=Group,dc=eng.example,dc=com" >>> --group-objectclass="posixGroup" --user-objectclass="account" >>> ldap://192.168.20.18:389 >>> ipa: INFO: trying https://hydrogen.eng.example.com/ipa/session/json >>> Password: >>> ipa: INFO: Forwarding 'migrate_ds' to json server >>> 'https://hydrogen.eng.example.com/ipa/session/json' >>> ----------- >>> migrate-ds: >>> ----------- >>> Migrated: >>> Failed user: >>> aagrim: missing attribute "sn" required by object class >>> "organizationalPerson" >>> acctemp: missing attribute "sn" required by object class >>> "organizationalPerson" >>> ........... >> >> This looks like a common problem. I had recently made a small 'hack' to >> solve this problem. >> >> Following small fixup plugin could be used to affect how entries are >> generated. If you add it to /usr/lib/python2.7/site-packages/ipalib/plugins >> on IPA master and restart httpd service, the plugin would modify migrate-ds >> command so >> that 'sn' attribute would be set to a 'Migrated User Last Name' for all >> entries that miss 'sn' attribute before they actually get added into IPA >> LDAP. >> >> This is an experimental hack, of course, but it should work. Once >> migration is finished, don't forget to remove the file and restart httpd >> service again. >> >> -- >> / Alexander Bokovoy >> >> -- >> Manage your subscription for the Freeipa-users mailing list: >> https://www.redhat.com/mailman/listinfo/freeipa-users >> Go to http://freeipa.org for more info on the project > >-- >Manage your subscription for the Freeipa-users mailing list: >https://www.redhat.com/mailman/listinfo/freeipa-users >Go to http://freeipa.org for more info on the project -- / Alexander Bokovoy From william.muriithi at gmail.com Fri Sep 2 11:44:38 2016 From: william.muriithi at gmail.com (William Muriithi) Date: Fri, 2 Sep 2016 07:44:38 -0400 Subject: [Freeipa-users] openLDAP to FreeIPA user migration Message-ID: Morning Alexander, >>Failed user: >> aagrim: missing attribute "sn" required by object class "organizationalPerson" >> acctemp: missing attribute "sn" required by object class >>"organizationalPerson" >> ........... > This looks like a common problem. I had recently made a small 'hack' to > solve this problem. > > Following small fixup plugin could be used to affect how entries are > generated. If you add it to /usr/lib/python2.7/site-packages/ipalib/plugins > on IPA master and restart httpd service, the plugin would modify migrate-ds command so > that 'sn' attribute would be set to a 'Migrated User Last Name' for all > entries that miss 'sn' attribute before they actually get added into IPA > LDAP. > > This is an experimental hack, of course, but it should work. Once > migration is finished, don't forget to remove the file and restart httpd > service again. Worked for me, thank you. Curious, would this qualify for inclusion in future IPA release considering its a common problem that show up often? > Regards, William From mareynol at redhat.com Fri Sep 2 13:41:38 2016 From: mareynol at redhat.com (Mark Reynolds) Date: Fri, 2 Sep 2016 09:41:38 -0400 Subject: [Freeipa-users] Replication scheme problem In-Reply-To: References: Message-ID: On 09/01/2016 06:13 AM, Andrey Rogovsky wrote: > Hi! > I have 2 servers - ldap1 is FreeIPA (master) and ldap2 is 389 DS (slave). > One way replication ldap1 -> ldap2 is enabled but scheme is not > replicated: What version of 389-ds-base are you using? rpm -qa | grep 389-ds-base > > Log file ldap1 have this line: > [01/Sep/2016:07:04:53 +0000] NSMMReplicationPlugin - Warning: unable > to replicate schema to host ldap2, port 389. Continuing with total > update session. Is there anything in ldap2's errors/access log from this time (01/Sep/2016:07:04:53)? > > There is current status: > filter: (objectclass=nsds5replicationagreement) > requesting: All userApplication attributes > # extended LDIF > # > # LDAPv3 > # base with scope subtree > # filter: (objectclass=nsds5replicationagreement) > # requesting: ALL > # > > # ExampleAgreement, replica, dc\3Dexample\2Cdc\3Dcom, mapping tree, config > dn: > cn=ExampleAgreement,cn=replica,cn=dc\3Dexample\2Cdc\3Dcom,cn=mapping tree, > cn=config > objectClass: top > objectClass: nsds5replicationagreement > cn: ExampleAgreement > nsDS5ReplicaHost: ldap2 > nsDS5ReplicaPort: 389 > nsDS5ReplicaBindDN: cn=replication manager,cn=config > nsDS5ReplicaBindMethod: SIMPLE > nsDS5ReplicaRoot: dc=example,dc=com > description: agreement between supplier1 and consumer1 > nsDS5ReplicaUpdateSchedule: 0000-0500 1 > nsDS5ReplicatedAttributeList: (objectclass=*) $ EXCLUDE > authorityRevocationLis > t > nsDS5ReplicaCredentials: > {AES-TUhNR0NTcUdTSWIzRFFFRkRUQm1NRVVHQ1NxR1NJYjNEUUVG > RERBNEJDUmxPVFl4TlRsbU5DMWtaV0UyTXpZeA0KTVMxaU1UYzFaREF3Wmkwek5qRmxNalkxWkFBQ > 0FRSUNBU0F3Q2dZSUtvWklodmNOQWdjd0hRWUpZSVpJQVdVRA0KQkFFcUJCQURJbjhNUWpLM1VqU1 > M1SGZEUTY0TA==}mwxKHUYWXjNeyo1AGRWe9A== > nsds5replicareapactive: 0 > nsds5replicaLastUpdateStart: 19700101000000Z > nsds5replicaLastUpdateEnd: 19700101000000Z > nsds5replicaChangesSentSinceStartup: > nsds5replicaLastUpdateStatus: 0 No replication sessions started since > server s > tartup > nsds5replicaUpdateInProgress: FALSE > nsds5replicaLastInitStart: 20160901070452Z > nsds5replicaLastInitEnd: 20160901070455Z > nsds5replicaLastInitStatus: 0 Total update succeeded > > # search result > search: 2 > result: 0 Success > > # numResponses: 2 > # numEntries: 1 > > > After execute schema-reload.pl on ldap2 I > have this lines in log: > Failed to add task entry "cn=schema_reload_2016_9_1_10_6_17, cn=schema > reload task, cn=tasks, cn=config" error (49) Error 49 = invalid credentials. You entered the wrong password - this prevented the schema reload task from taking place. You can also restart the directory server which will do the same thing as the schema reload task. The schema reload task is just so you can reload new schema files without having to restart the server. > [01/Sep/2016:07:04:59 +0000] NSACLPlugin - Error: This ((targetattr = > "gidnumber || krbprincipalname || uidnumber")(version 3.0;acl > "permission:System: Read system trust accounts";allow > (compare,read,search) groupdn = "ldap:///cn=System: Read system trust > accounts,cn=permissions,cn=pbac,dc=example,dc=com";)) ACL will not be > considered for evaluation because of syntax errors. > [01/Sep/2016:07:04:59 +0000] NSACLPlugin - __aclp__init_targetattr: > targetattr "ipaanchoruuid" does not exist in schema. Please add > attributeTypes "ipaanchoruuid" to schema if necessary. > [01/Sep/2016:07:04:59 +0000] NSACLPlugin - ACL PARSE ERR(rv=-5): > (targetattr = "cn > [01/Sep/2016:07:04:59 +0000] NSACLPlugin - Error: This ((targetattr = > "cn || createtimestamp || description || entryusn || gidnumber || > ipaanchoruuid || modifytimestamp || objectclass")(targetfilter = > "(objectclass=ipaGroupOverride)")(version 3.0;acl "permission:System: > Read Group ID Overrides";allow (compare,read,search) userdn = > "ldap:///all";)) ACL will not be considered for evaluation because of > syntax errors. > [01/Sep/2016:07:04:59 +0000] NSACLPlugin - __aclp__init_targetattr: > targetattr "ipaanchoruuid" does not exist in schema. Please add > attributeTypes "ipaanchoruuid" to schema if necessary. > [01/Sep/2016:07:04:59 +0000] NSACLPlugin - ACL PARSE ERR(rv=-5): > (targetattr = "createtimestamp > [01/Sep/2016:07:04:59 +0000] NSACLPlugin - Error: This ((targetattr = > "createtimestamp || description || entryusn || gecos || gidnumber || > homedirectory || ipaanchoruuid || ipaoriginaluid || ipasshpubkey || > loginshell || modifytimestamp || objectclass || uid || > uidnumber")(targetfilter = "(objectclass=ipaUserOverride)")(version > 3.0;acl "permission:System: Read User ID Overrides";allow > (compare,read,search) userdn = "ldap:///all";)) ACL will not be > considered for evaluation because of syntax errors. > [01/Sep/2016:07:04:59 +0000] NSACLPlugin - __aclp__init_targetattr: > targetattr "a6record" does not exist in schema. Please add > attributeTypes "a6record" to schema if necessary. > [01/Sep/2016:07:04:59 +0000] NSACLPlugin - ACL PARSE ERR(rv=-5): > (targetattr = "a6record > [01/Sep/2016:07:04:59 +0000] NSACLPlugin - Error: This ((targetattr = > "a6record || aaaarecord || afsdbrecord || aplrecord || arecord || > certrecord || cn || cnamerecord || dhcidrecord || dlvrecord || > dnamerecord || dnsclass || dnsttl || dsrecord || hinforecord || > hiprecord || idnsallowdynupdate || idnsallowquery || idnsallowsyncptr > || idnsallowtransfer || idnsforwarders || idnsforwardpolicy || > idnsname || idnssecinlinesigning || idnssoaexpire || idnssoaminimum || > idnssoamname || idnssoarefresh || idnssoaretry || idnssoarname || > idnssoaserial || idnsupdatepolicy || idnszoneactive || ipseckeyrecord > || keyrecord || kxrecord || locrecord || mdrecord || minforecord || > mxrecord || naptrrecord || nsecrecord || nsec3paramrecord || nsrecord > || nxtrecord || ptrrecord || rprecord || rrsigrecord || sigrecord || > spfrecord || srvrecord || sshfprecord || tlsarecord || txtrecord || > unknownrecord ")(target = > "ldap:///idnsname=*,cn=dns,dc=example,dc=com")(version 3.0;acl "Update > DNS entries in a zone";allow (write) userattr = > "parent[0,1].managedby#GROUPDN";)) ACL will not be considered for > evaluation because of syntax errors. > > Looks, like ipaanchoruuid is missing. There is ldap scheme update: > filter: (objectclass=*) > requesting: 00core.ldif 01core389.ldif 02common.ldif 05rfc2927.ldif > 05rfc4523.ldif 05rfc4524.ldif 06inetorgperson.ldif > 10automember-plugin.ldif 10dna-plugin.ldif 10mep-plugin.ldif > 10rfc2307.ldif 20subscriber.ldif 25java-object.ldif 28pilot.ldif > 30ns-common.ldif 50ns-admin.ldif 50ns-certificate.ldif > 50ns-directory.ldif 50ns-mail.ldif 50ns-value.ldif 50ns-web.ldif > 60acctpolicy.ldif 60autofs.ldif 60eduperson.ldif 60mozilla.ldif > 60nss-ldap.ldif 60pam-plugin.ldif 60posix-winsync-plugin.ldif > 60pureftpd.ldif 60rfc2739.ldif 60rfc3712.ldif 60sabayon.ldif > 60sudo.ldif 60trust.ldif 99user.ldif > > No one scheme files from ldap2 have no entry for ipaanchoruuid. But > from ldap1 they have: > root at ldap1 schema]# grep -i ipaanchoruuid * > 71idviews.ldif:attributeTypes: (2.16.840.1.113730.3.8.11.62 NAME > 'ipaAnchorUUID' DESC 'Unique Anchor Identifier' EQUALITY > caseIgnoreMatch ORDERING caseIgnoreOrderingMatch SYNTAX > 1.3.6.1.4.1.1466.115.121.1.15 SINGLE-VALUE X-ORIGIN 'IPA v4') > 71idviews.ldif:objectClasses: (2.16.840.1.113730.3.8.12.30 NAME > 'ipaOverrideAnchor' SUP top STRUCTURAL MUST ( ipaAnchorUUID ) MAY ( > description ) X-ORIGIN 'IPA v4' ) > 71idviews.ldif:objectClasses: (2.16.840.1.113730.3.8.12.35 NAME > 'ipaOverrideTarget' SUP top STRUCTURAL MUST ( ipaAnchorUUID ) X-ORIGIN > 'IPA v4' ) > [root at ldap1 schema]# > > How to resolve this issue? Just copy schemes files from ldap1 to ldap2? That will work, but you need to restart the server for it to take effect. > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From mike.driscoll at oracle.com Fri Sep 2 14:15:57 2016 From: mike.driscoll at oracle.com (Mike Driscoll) Date: Fri, 2 Sep 2016 07:15:57 -0700 Subject: [Freeipa-users] Password change rights In-Reply-To: References: Message-ID: <0A9933C5-F686-40BC-94D5-725AE0A68A4C@oracle.com> Hello. I want to script the new user creation process. I read in section 9.4 that "any user who has password change rights can change a password and no password policies are applied, but the other user must reset the password at the next login.? I want to create an account with this limited capability for inclusion in a script. But I can?t figure out how to configure an account to have this capability without being a full admin. How can I create new user accounts and set initial passwords in a script? Mike From abokovoy at redhat.com Fri Sep 2 14:59:23 2016 From: abokovoy at redhat.com (Alexander Bokovoy) Date: Fri, 2 Sep 2016 17:59:23 +0300 Subject: [Freeipa-users] Password change rights In-Reply-To: <0A9933C5-F686-40BC-94D5-725AE0A68A4C@oracle.com> References: <0A9933C5-F686-40BC-94D5-725AE0A68A4C@oracle.com> Message-ID: <20160902145923.oer6wy3z5czo35ph@redhat.com> On Fri, 02 Sep 2016, Mike Driscoll wrote: >Hello. I want to script the new user creation process. I read in >section 9.4 that "any user who has password change rights can change a >password and no password policies are applied, but the other user must >reset the password at the next login.? I want to create an account >with this limited capability for inclusion in a script. But I can?t >figure out how to configure an account to have this capability without >being a full admin. How can I create new user accounts and set initial >passwords in a script? You need to create a permission that allows to write to password attributes. Then create a privilege and role that utilize this permission. Then you would assign the user that is capable to reset passwords to that role and it should be enough. I recently wrote an article how to create new permissions: https://vda.li/en/posts/2016/08/30/Creating-permissions-in-FreeIPA/ You only need to look at selfservice 'Self can write own password' and create a normal permission with similar effective attributes: # ipa selfservice-show 'Self can write own password' Self-service name: Self can write own password Permissions: write Attributes: userpassword, krbprincipalkey, sambalmpassword, sambantpassword Note the difference between selfservice and permission -- the former is always executed against SELFDN of a bind identity, e.g. those who authenticate, the latter can take care of both the target and the bind identity. -- / Alexander Bokovoy From rcritten at redhat.com Fri Sep 2 15:17:49 2016 From: rcritten at redhat.com (Rob Crittenden) Date: Fri, 2 Sep 2016 11:17:49 -0400 Subject: [Freeipa-users] Password change rights In-Reply-To: <20160902145923.oer6wy3z5czo35ph@redhat.com> References: <0A9933C5-F686-40BC-94D5-725AE0A68A4C@oracle.com> <20160902145923.oer6wy3z5czo35ph@redhat.com> Message-ID: <57C9981D.6010502@redhat.com> Alexander Bokovoy wrote: > On Fri, 02 Sep 2016, Mike Driscoll wrote: >> Hello. I want to script the new user creation process. I read in >> section 9.4 that "any user who has password change rights can change a >> password and no password policies are applied, but the other user must >> reset the password at the next login.? I want to create an account >> with this limited capability for inclusion in a script. But I can?t >> figure out how to configure an account to have this capability without >> being a full admin. How can I create new user accounts and set initial >> passwords in a script? > You need to create a permission that allows to write to password > attributes. Then create a privilege and role that utilize this > permission. > > Then you would assign the user that is capable to reset passwords to > that role and it should be enough. > > I recently wrote an article how to create new permissions: > https://vda.li/en/posts/2016/08/30/Creating-permissions-in-FreeIPA/ > > You only need to look at selfservice 'Self can write own > password' and create a normal permission with similar effective > attributes: > # ipa selfservice-show 'Self can write own password' > Self-service name: Self can write own password > Permissions: write > Attributes: userpassword, krbprincipalkey, sambalmpassword, > sambantpassword > > Note the difference between selfservice and permission -- the former is > always executed against SELFDN of a bind identity, e.g. those who > authenticate, the latter can take care of both the target and the bind > identity. There already is such a permission, "System: Change User password" rob From abokovoy at redhat.com Fri Sep 2 15:55:08 2016 From: abokovoy at redhat.com (Alexander Bokovoy) Date: Fri, 2 Sep 2016 18:55:08 +0300 Subject: [Freeipa-users] Password change rights In-Reply-To: <57C9981D.6010502@redhat.com> References: <0A9933C5-F686-40BC-94D5-725AE0A68A4C@oracle.com> <20160902145923.oer6wy3z5czo35ph@redhat.com> <57C9981D.6010502@redhat.com> Message-ID: <20160902155508.m6vu5iidmyxtcnwn@redhat.com> On Fri, 02 Sep 2016, Rob Crittenden wrote: >Alexander Bokovoy wrote: >>On Fri, 02 Sep 2016, Mike Driscoll wrote: >>>Hello. I want to script the new user creation process. I read in >>>section 9.4 that "any user who has password change rights can change a >>>password and no password policies are applied, but the other user must >>>reset the password at the next login.? I want to create an account >>>with this limited capability for inclusion in a script. But I can?t >>>figure out how to configure an account to have this capability without >>>being a full admin. How can I create new user accounts and set initial >>>passwords in a script? >>You need to create a permission that allows to write to password >>attributes. Then create a privilege and role that utilize this >>permission. >> >>Then you would assign the user that is capable to reset passwords to >>that role and it should be enough. >> >>I recently wrote an article how to create new permissions: >>https://vda.li/en/posts/2016/08/30/Creating-permissions-in-FreeIPA/ >> >>You only need to look at selfservice 'Self can write own >>password' and create a normal permission with similar effective >>attributes: >># ipa selfservice-show 'Self can write own password' >> Self-service name: Self can write own password >> Permissions: write >> Attributes: userpassword, krbprincipalkey, sambalmpassword, >>sambantpassword >> >>Note the difference between selfservice and permission -- the former is >>always executed against SELFDN of a bind identity, e.g. those who >>authenticate, the latter can take care of both the target and the bind >>identity. > >There already is such a permission, "System: Change User password" Thank you, Rob. My bad: we have so many permissions now that default size limit kicks in and I don't see it: # ipa permission-find --sizelimit=0 |grep -i 'permission name:'|wc -l 237 # ipa permission-find --sizelimit=0 |grep -i 'permission name:'|grep -i 'change user password' Permission name: System: Change User password -- / Alexander Bokovoy From deepak_dimri at hotmail.com Fri Sep 2 18:06:52 2016 From: deepak_dimri at hotmail.com (Deepak Dimri) Date: Fri, 2 Sep 2016 14:06:52 -0400 Subject: [Freeipa-users] General query regarding nameserver enrtry Message-ID: Hi All, My ipa-client-install fails until etc/resolve.conf gets updated with IPA nameserver entry. I want to avoid a task of updating resolve.conf in my automation script. Is there a way i can get my IPA client installation successful without updating resolve.conf? what options do i have? Many Thanks,Deepak -------------- next part -------------- An HTML attachment was scrubbed... URL: From coy.hile at coyhile.com Fri Sep 2 18:56:42 2016 From: coy.hile at coyhile.com (Coy Hile) Date: Fri, 2 Sep 2016 14:56:42 -0400 Subject: [Freeipa-users] Question about ID views Message-ID: <006C1C38-0365-425D-8DEF-1AE5A6EB3001@coyhile.com> In looking at the ID Views functionality in FreeIPA, it looks like I can accomplish overrides (such as users? shell in LDAP is /bin/bash, but on a certain subset of hosts, users get /some/restrictive/shell instead? (Use case #1: a bastion host or jump box where admins might want to validate that users are only attempting to access authorized internal destinations.) However, it looks like one has to list each user individually? Is it possible to do things like this? *:::::/usr/bin/restricted_shell /somepath/home/jdoe -- Coy Hile coy.hile at coyhile.com From orion at cora.nwra.com Fri Sep 2 19:12:16 2016 From: orion at cora.nwra.com (Orion Poplawski) Date: Fri, 2 Sep 2016 13:12:16 -0600 Subject: [Freeipa-users] Default gid for AD trust users In-Reply-To: References: Message-ID: <656bfc4e-b802-f2ac-8d7e-db0d156db137@cora.nwra.com> FWIW - I've filed https://fedorahosted.org/freeipa/ticket/6293 to request the ability to set the primary group for AD trust users. On 08/24/2016 11:42 AM, Orion Poplawski wrote: > While that is definitely *a* convention, it's not the one we've used which > puts users by default in shared groups (nwra, visitors, etc). For example: > > uid=2941(user) gid=1991(nwra) > > We may be fine changing conventions, but I'm researching whether or not we > have to. > > Thanks. > > On 08/24/2016 11:19 AM, Justin Stephenson wrote: >> Could you please explain further what you are trying to accomplish with an AD >> trust default group? I believe we are following the standard linux convention >> of creating a user private group using the ID number which matches the uid >> number for AD trust users. >> >> Kind regards, >> >> Justin Stephenson >> >> >> On 08/23/2016 06:27 PM, Orion Poplawski wrote: >>> Is there any way to control the default gid for AD trust users? At the moment >>> each user has it's own default group, e.g.: >>> >>> uid=22603(user at ad.domain) gid=22603(user at ad.domain) >>> >>> It would be nice to be able to set this to an actual group. >>> >>> Thanks. >>> >> > > -- Orion Poplawski Technical Manager 303-415-9701 x222 NWRA, Boulder/CoRA Office FAX: 303-415-9702 3380 Mitchell Lane orion at nwra.com Boulder, CO 80301 http://www.nwra.com From tjyang2001 at gmail.com Fri Sep 2 20:28:12 2016 From: tjyang2001 at gmail.com (T.J. Yang) Date: Fri, 2 Sep 2016 15:28:12 -0500 Subject: [Freeipa-users] freeip-4.4.1 on CentOS 7.x ? Message-ID: Hi I was able to try out freeipa-4.4.1 on fedora 24 server by quick dnf enable at https://copr.fedorainfracloud.org/coprs/g/freeipa/freeipa-4-4/ But for production, I am hoping to run 4.4.1 on CentOS 7 Where is the doc explaining on this howto for CentOS 7 ? tj -- T.J. Yang -------------- next part -------------- An HTML attachment was scrubbed... URL: From rcritten at redhat.com Fri Sep 2 20:34:01 2016 From: rcritten at redhat.com (Rob Crittenden) Date: Fri, 2 Sep 2016 16:34:01 -0400 Subject: [Freeipa-users] freeip-4.4.1 on CentOS 7.x ? In-Reply-To: References: Message-ID: <57C9E239.90908@redhat.com> T.J. Yang wrote: > Hi > > I was able to try out freeipa-4.4.1 on fedora 24 server by quick dnf > enable at > > https://copr.fedorainfracloud.org/coprs/g/freeipa/freeipa-4-4/ > > But for production, I am hoping to run 4.4.1 on CentOS 7 > > Where is the doc explaining on this howto for CentOS 7 ? It hasn't been released for RHEL/CentOS 7 yet. There is a beta though AFAIU. rob From lslebodn at redhat.com Fri Sep 2 21:15:37 2016 From: lslebodn at redhat.com (Lukas Slebodnik) Date: Fri, 2 Sep 2016 23:15:37 +0200 Subject: [Freeipa-users] Default gid for AD trust users In-Reply-To: References: Message-ID: <20160902211536.GA29320@10.4.128.1> On (24/08/16 11:42), Orion Poplawski wrote: >While that is definitely *a* convention, it's not the one we've used which >puts users by default in shared groups (nwra, visitors, etc). For example: > >uid=2941(user) gid=1991(nwra) > The user "user" should be a member "nwra" group. If no then you have other issues. Why does it matter whether it is a primary group or no? LS From rakesh.rajasekharan at gmail.com Sun Sep 4 12:04:26 2016 From: rakesh.rajasekharan at gmail.com (Rakesh Rajasekharan) Date: Sun, 4 Sep 2016 17:34:26 +0530 Subject: [Freeipa-users] Freeipa 4.2.0 hangs intermittently In-Reply-To: References: <7850c6f9-f79a-355e-4730-93bb67fbacf1@redhat.com> <57BEB0AB.2090506@redhat.com> <57C44AC6.1030402@redhat.com> Message-ID: I have again got the issue of IPA hanging.. The issue came up when i tried to run ipa-client-isntall on 142 clients simultaneously None of the IPA commands are responding, and I see this error ipa user-find p-testipa ipa: ERROR: Insufficient access: SASL(-1): generic failure: GSSAPI Error: Unspecified GSS failure. Minor code may provide more information (KDC returned error string: PROCESS_TGS) KRB5_TRACE=/dev/stdout kinit admin [41178] 1472984115.233214: Getting initial credentials for admin at XYZ.COM [41178] 1472984115.235257: Sending request (167 bytes) to XYZ.COM [41178] 1472984115.235419: Initiating TCP connection to stream 10.1.3.36:88 [41178] 1472984115.235685: Sending TCP request to stream 10.1.3.36:88 [41178] 1472984120.238914: Received answer (174 bytes) from stream 10.1.3.36:88 [41178] 1472984120.238925: Terminating TCP connection to stream 10.1.3.36:88 [41178] 1472984120.238993: Response was from master KDC [41 Running an ldapsearch to see the db.. does not give any results and just hangs there ldapsearch -x -D 'cn=Directory Manager' -W -s one -b 'cn=kerberos,dc=xyz,dc=com' Enter LDAP Password: even an ldapsearch -x does not respond At this point, am sure that slapd is the one causing issues Running an strace against the hung slapd itself seems to get stuck does not proceed after saying "attaching to process" >From some others posts I read Thierry suggesting to increase the nsslapd-threadnumber value It was set to 30, I think that might be too low. I have raised it to 500 Now after restarting the service .. ldapsearch starts responding. But running the test to add a sudden high number of clients again left ns-slapd to hung state When i attempted adding the clients.. the ns-slapd cpu usage shot up to 340% and after that ns-slapd stopped responding So now, atleast I know what might be causing the issue and I can now easily reproduce it. Is there a way I can make ns-slapd handle a sudden bump in incoming request for ipa-client-install Thanks Rakesh On Mon, Aug 29, 2016 at 11:18 PM, Rich Megginson wrote: > On 08/29/2016 10:53 AM, Rakesh Rajasekharan wrote: > > Hi Thierry, > > My machine has 30GB RAM ..and 389-ds version is 1.3.4 > > ldapsearch shows the values for nsslapd-cachememsize updated to 200MB. > > ldapsearch -LLL -o ldif-wrap=no -D "cn=directory manager" -w 'mypassword' > -b 'cn=userRoot,cn=ldbm database,cn=plugins,cn=config'|grep > nsslapd-cachememsize > nsslapd-cachememsize: 209715200 > > > So, it seems to have updated though seeing that warning(WARNING: ipaca: > entry cache size 10485760B is less than db size 11599872B) in the log > confuses me a bit. > > Thers one more entry that I found from the ldapsearch to be bit low > > nsslapd-dncachememsize: 10485760 > maxdncachesize: 10485760 > > Should I update these as well to a higher value > > At the time when the issue happened, the memory usage as well as the > overall load of the system was very low . > I will try reproducing the issue atleast in my QA env..probably by trying > to mock simultaneous parallel logins to a large number of hosts > > > To monitor your cache sizes, please use the dbmon.sh tool provided with > your distro. If that is not available with your particular distro, see > https://github.com/richm/scripts/wiki/dbmon.sh > > > > > thanks > Rakesh > > > > > On Mon, Aug 29, 2016 at 8:16 PM, thierry bordaz > wrote: > >> Hi Rakesh, >> >> Those tuning may depend on the memory available on your machine. >> nsslapd-cachememsize allows the entry cache to consume up to 200Mb but >> its memory footprint is known to go above. >> 200Mb both looks pretty good to me. How large is your machine ? What is >> your version of 389-ds ? >> >> Those warnings do not change your settings. It just raise that entry >> cache of 'ipaca' and 'retrocl' are small but it is fine. The size of the >> entry cache is important mostly in userRoot. >> You may double check the actual values, after restart, with ldapsearch on >> 'cn=userRoot,cn=ldbm database,cn=plugins,cn=config' and 'cn=config,cn=ldbm >> database,cn=plugins,cn=config'. >> >> A step is to know what will be response time of DS to know if it is >> responsible of the hang or not. >> The logs and possibly pstack during those intermittent hangs will help to >> determine that. >> >> regards >> thierry >> >> >> >> >> >> On 08/29/2016 04:25 PM, Rakesh Rajasekharan wrote: >> >> I tried increasing the nsslapd-dbcachesize and nsslapd-cachememsize in my >> QA envs to 200MB. >> >> However, in my log files, I still see this message >> [29/Aug/2016:04:34:37 +0000] - WARNING: ipaca: entry cache size 10485760B >> is less than db size 11599872B; We recommend to increase the entry cache >> size nsslapd-cachememsize. >> [29/Aug/2016:04:34:37 +0000] - WARNING: changelog: entry cache size >> 2097152B is less than db size 441647104B; We recommend to increase the >> entry cache size nsslapd-cachememsize. >> >> these are my ldif files that i used to modify the values >> modify entry cache size >> cat modify-cache-mem-size.ldif >> dn: cn=userRoot,cn=ldbm database,cn=plugins,cn=config >> changetype: modify >> replace: nsslapd-cachememsize >> nsslapd-cachememsize: 209715200 >> >> modify db cache size >> cat modfy-db-cache-size.ldif >> dn: cn=config,cn=ldbm database,cn=plugins,cn=config >> changetype: modify >> replace: nsslapd-dbcachesize >> nsslapd-dbcachesize: 209715200 >> >> After modifying , i restarted IPA services >> >> Is there anything else that I need to take care of as the logs suggest >> its still not getting the updated values >> >> Thanks >> Rakesh >> >> On Mon, Aug 29, 2016 at 6:07 PM, Rakesh Rajasekharan < >> rakesh.rajasekharan at gmail.com> wrote: >> >>> Hi Thierry, >>> >>> Coz of the issues we had to revert back to earlier running openldap in >>> production. >>> >>> I have now done a few TCP related changes in sysctl.conf and have also >>> increased the nsslapd-dbcachesize and nsslapd-cachememsize to 200MB >>> >>> I will again start migrating hosts back to IPA and see if I face the >>> earlier issue. >>> >>> I will update back once I have something >>> >>> >>> Thanks, >>> Rakesh >>> >>> >>> >>> On Thu, Aug 25, 2016 at 2:17 PM, thierry bordaz >>> wrote: >>> >>>> >>>> >>>> On 08/25/2016 10:15 AM, Rakesh Rajasekharan wrote: >>>> >>>> All of the troubleshooting seems fine. >>>> >>>> >>>> However, Running libconv.pl gives me this output >>>> >>>> ----- Recommendations ----- >>>> >>>> 1. You have unindexed components, this can be caused from a search on >>>> an unindexed attribute, or your returned results exceeded the >>>> allidsthreshold. Unindexed components are not recommended. To refuse >>>> unindexed searches, switch 'nsslapd-require-index' to 'on' under your >>>> database entry (e.g. cn=UserRoot,cn=ldbm database,cn=plugins,cn=config) >>>> . >>>> >>>> 2. You have a significant difference between binds and unbinds. You >>>> may want to investigate this difference. >>>> >>>> >>>> I feel, this could be a pointer to things going slow.. and IPA hanging. >>>> I think i now have something that I can try and nail down this issue. >>>> >>>> On a sidenote, I was earlier running openldap and migrated over to >>>> Freeipa, >>>> >>>> Thanks >>>> Rakesh >>>> >>>> >>>> >>>> On Wed, Aug 24, 2016 at 12:38 PM, Petr Spacek >>>> wrote: >>>> >>>>> On 23.8.2016 18:44, Rakesh Rajasekharan wrote: >>>>> > I think thers something seriously wrong with my system >>>>> > >>>>> > not able to run any IPA commands >>>>> > >>>>> > klist >>>>> > Ticket cache: KEYRING:persistent:0:0 >>>>> > Default principal: admin at XYZ.COM >>>>> > >>>>> > Valid starting Expires Service principal >>>>> > 2016-08-23T16:26:36 2016-08-24T16:26:22 krbtgt/XYZ.COM at XYZ.COM >>>>> > >>>>> > >>>>> > [root at prod-ipa-master-1a :~] ipactl status >>>>> > Directory Service: RUNNING >>>>> > krb5kdc Service: RUNNING >>>>> > kadmin Service: RUNNING >>>>> > ipa_memcached Service: RUNNING >>>>> > httpd Service: RUNNING >>>>> > pki-tomcatd Service: RUNNING >>>>> > ipa-otpd Service: RUNNING >>>>> > ipa: INFO: The ipactl command was successful >>>>> > >>>>> > >>>>> > >>>>> > [root at prod-ipa-master :~] ipa user-find p-testuser >>>>> > ipa: ERROR: Kerberos error: ('Unspecified GSS failure. Minor code >>>>> may >>>>> > provide more information', 851968)/("Cannot contact any KDC for >>>>> realm ' >>>>> > XYZ.COM'", -1765328228) >>>>> >>>> >>>> Hi Rakesh, >>>> >>>> Having a reproducible test case would you rerun the command above. >>>> During its processing you may monitor DS process load (top). If it is >>>> high, you may get some pstacks of it. >>>> Also would you attach the part of DS access logs taken during the >>>> command. >>>> >>>> regards >>>> thierry >>>> >>>> > >>>>> >>>>> This is weird because the server seems to be up. >>>>> >>>>> Please follow >>>>> http://www.freeipa.org/page/Troubleshooting#Authentication.2FKerberos >>>>> >>>>> Petr^2 Spacek >>>>> >>>>> > >>>>> > >>>>> > Thanks >>>>> > >>>>> > Rakesh >>>>> > >>>>> > On Tue, Aug 23, 2016 at 10:01 PM, Rakesh Rajasekharan < >>>>> > rakesh.rajasekharan at gmail.com> wrote: >>>>> > >>>>> >> i changed the loggin level to 4 . Modifying nsslapd-accesslog-level >>>>> >> >>>>> >> But, the hang is still there. though I dont see the sigfault now >>>>> >> >>>>> >> >>>>> >> >>>>> >> >>>>> >> On Tue, Aug 23, 2016 at 9:02 PM, Rakesh Rajasekharan < >>>>> >> rakesh.rajasekharan at gmail.com> wrote: >>>>> >> >>>>> >>> My disk was getting filled too fast >>>>> >>> >>>>> >>> logs under /var/log/dirsrv was coming around 5 gb quickly filling >>>>> up >>>>> >>> >>>>> >>> Is there a way to make the logging less verbose >>>>> >>> >>>>> >>> >>>>> >>> >>>>> >>> On Tue, Aug 23, 2016 at 6:41 PM, Petr Spacek >>>>> wrote: >>>>> >>> >>>>> >>>> On 23.8.2016 15:07, Rakesh Rajasekharan wrote: >>>>> >>>>> I was able to fix that may be temporarily... when i checked the >>>>> >>>> network.. >>>>> >>>>> there was another process that was running and consuming a lot of >>>>> >>>> network ( >>>>> >>>>> i have no idea who did that. I need to seriously start >>>>> restricting >>>>> >>>> people >>>>> >>>>> access to this machine ) >>>>> >>>>> >>>>> >>>>> after killing that perfomance improved drastically >>>>> >>>>> >>>>> >>>>> But now, suddenly I started experiencing the same hang. >>>>> >>>>> >>>>> >>>>> This time , I gert the following error when checked dmesg >>>>> >>>>> >>>>> >>>>> [ 301.236976] ns-slapd[3124]: segfault at 0 ip 00007f1de416951c >>>>> sp >>>>> >>>>> 00007f1dee1dba70 error 4 in libcos-plugin.so[7f1de4166000+b000] >>>>> >>>>> [ 1116.248431] TCP: request_sock_TCP: Possible SYN flooding on >>>>> port 88. >>>>> >>>>> Sending cookies. Check SNMP counters. >>>>> >>>>> [11831.397037] ns-slapd[22550]: segfault at 0 ip >>>>> 00007f533d82251c sp >>>>> >>>>> 00007f5347894a70 error 4 in libcos-plugin.so[7f533d81f000+b000] >>>>> >>>>> [11832.727989] ns-slapd[22606]: segfault at 0 ip >>>>> 00007f6231eb951c sp >>>>> >>>>> 00007f623bf2ba70 error 4 in libcos-plugin.so[7f6231eb6000+b00 >>>>> >>>> >>>>> >>>> Okay, this one is serious. The LDAP server crashed. >>>>> >>>> >>>>> >>>> 1. Make sure all your packages are up-to-date. >>>>> >>>> >>>>> >>>> Please see >>>>> >>>> http://directory.fedoraproject.org/docs/389ds/FAQ/faq.html#d >>>>> >>>> ebugging-crashes >>>>> >>>> for further instructions how to debug this. >>>>> >>>> >>>>> >>>> Petr^2 Spacek >>>>> >>>> >>>>> >>>>> >>>>> >>>>> and in /var/log/dirsrv/example-com/errors >>>>> >>>>> >>>>> >>>>> [23/Aug/2016:12:49:36 +0000] DSRetroclPlugin - >>>>> delete_changerecord: >>>>> >>>> could >>>>> >>>>> not delete change record 3291138 (rc: 32) >>>>> >>>>> [23/Aug/2016:12:49:36 +0000] DSRetroclPlugin - >>>>> delete_changerecord: >>>>> >>>> could >>>>> >>>>> not delete change record 3291139 (rc: 32) >>>>> >>>>> [23/Aug/2016:12:49:36 +0000] DSRetroclPlugin - >>>>> delete_changerecord: >>>>> >>>> could >>>>> >>>>> not delete change record 3291140 (rc: 32) >>>>> >>>>> [23/Aug/2016:12:49:36 +0000] DSRetroclPlugin - >>>>> delete_changerecord: >>>>> >>>> could >>>>> >>>>> not delete change record 3291141 (rc: 32) >>>>> >>>>> [23/Aug/2016:12:49:36 +0000] DSRetroclPlugin - >>>>> delete_changerecord: >>>>> >>>> could >>>>> >>>>> not delete change record 3291142 (rc: 32) >>>>> >>>>> [23/Aug/2016:12:49:36 +0000] DSRetroclPlugin - >>>>> delete_changerecord: >>>>> >>>> could >>>>> >>>>> not delete change record 3291143 (rc: 32) >>>>> >>>>> [23/Aug/2016:12:49:36 +0000] DSRetroclPlugin - >>>>> delete_changerecord: >>>>> >>>> could >>>>> >>>>> not delete change record 3291144 (rc: 32) >>>>> >>>>> [23/Aug/2016:12:49:36 +0000] DSRetroclPlugin - >>>>> delete_changerecord: >>>>> >>>> could >>>>> >>>>> not delete change record 3291145 (rc: 32) >>>>> >>>>> [23/Aug/2016:12:49:50 +0000] - Retry count exceeded in delete >>>>> >>>>> [23/Aug/2016:12:49:50 +0000] DSRetroclPlugin - >>>>> delete_changerecord: >>>>> >>>> could >>>>> >>>>> not delete change record 3292734 (rc: 51) >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> Can i do something about this error.. I treid to restart ipa a >>>>> couple >>>>> >>>> of >>>>> >>>>> time but that did not help >>>>> >>>>> >>>>> >>>>> Thanks >>>>> >>>>> Rakesh >>>>> >>>>> >>>>> >>>>> On Mon, Aug 22, 2016 at 2:27 PM, Petr Spacek >>>> > >>>>> >>>> wrote: >>>>> >>>>> >>>>> >>>>>> On 19.8.2016 19:32, Rakesh Rajasekharan wrote: >>>>> >>>>>>> I am running my set up on AWS cloud, and entropy is low at >>>>> around >>>>> >>>> 180 . >>>>> >>>>>>> >>>>> >>>>>>> I plan to increase it bu installing haveged . But, would low >>>>> entropy >>>>> >>>> by >>>>> >>>>>> any >>>>> >>>>>>> chance cause this issue of intermittent hang . >>>>> >>>>>>> Also, the hang is mostly observed when registering around 20 >>>>> clients >>>>> >>>>>>> together >>>>> >>>>>> >>>>> >>>>>> Possibly, I'm not sure. If you want to dig into this, I would >>>>> do this: >>>>> >>>>>> 1. look what process hangs on client (using pstree command or >>>>> so) >>>>> >>>>>> $ pstree >>>>> >>>>>> >>>>> >>>>>> 2. look to what server and port is the hanging client connected >>>>> to >>>>> >>>>>> $ lsof -p >>>>> >>>>>> >>>>> >>>>>> 3. jump to server and see what process is bound to the target >>>>> port >>>>> >>>>>> $ netstat -pn >>>>> >>>>>> >>>>> >>>>>> 4. see where the process if hanging >>>>> >>>>>> $ strace -p >>>>> >>>>>> >>>>> >>>>>> I hope it helps. >>>>> >>>>>> >>>>> >>>>>> Petr^2 Spacek >>>>> >>>>>> >>>>> >>>>>>> On Fri, Aug 19, 2016 at 7:24 PM, Rakesh Rajasekharan < >>>>> >>>>>>> rakesh.rajasekharan at gmail.com> wrote: >>>>> >>>>>>> >>>>> >>>>>>>> yes there seems to be something thats worrying.. I have faced >>>>> this >>>>> >>>> today >>>>> >>>>>>>> as well. >>>>> >>>>>>>> There are few hosts around 280 odd left and when i try adding >>>>> them >>>>> >>>> to >>>>> >>>>>> IPA >>>>> >>>>>>>> , the slowness begins.. >>>>> >>>>>>>> >>>>> >>>>>>>> all the ipa commands like ipa user-find.. etc becomes very >>>>> slow in >>>>> >>>>>>>> responding. >>>>> >>>>>>>> >>>>> >>>>>>>> the SYNC_RECV are not many though just around 80-90 and today >>>>> that >>>>> >>>> was >>>>> >>>>>>>> around 20 only >>>>> >>>>>>>> >>>>> >>>>>>>> >>>>> >>>>>>>> I have for now increased tcp_max_syn_backlog to 5000. >>>>> >>>>>>>> For now the slowness seems to have gone.. but I will do a try >>>>> >>>> adding the >>>>> >>>>>>>> clients again tomorrow and see how it goes >>>>> >>>>>>>> >>>>> >>>>>>>> Thanks >>>>> >>>>>>>> Rakesh >>>>> >>>>>>>> >>>>> >>>>>>>> The issues >>>>> >>>>>>>> >>>>> >>>>>>>> On Fri, Aug 19, 2016 at 12:58 PM, Petr Spacek < >>>>> pspacek at redhat.com> >>>>> >>>>>> wrote: >>>>> >>>>>>>> >>>>> >>>>>>>>> On 18.8.2016 17:23, Rakesh Rajasekharan wrote: >>>>> >>>>>>>>>> Hi >>>>> >>>>>>>>>> >>>>> >>>>>>>>>> I am migrating to freeipa from openldap and have around 4000 >>>>> >>>> clients >>>>> >>>>>>>>>> >>>>> >>>>>>>>>> I had openned a another thread on that, but chose to start >>>>> a new >>>>> >>>> one >>>>> >>>>>>>>> here >>>>> >>>>>>>>>> as its a separate issue >>>>> >>>>>>>>>> >>>>> >>>>>>>>>> I was able to change the nssslapd-maxdescriptors adding an >>>>> ldif >>>>> >>>> file >>>>> >>>>>>>>>> >>>>> >>>>>>>>>> cat nsslapd-modify.ldif >>>>> >>>>>>>>>> dn: cn=config >>>>> >>>>>>>>>> changetype: modify >>>>> >>>>>>>>>> replace: nsslapd-maxdescriptors >>>>> >>>>>>>>>> nsslapd-maxdescriptors: 17000 >>>>> >>>>>>>>>> >>>>> >>>>>>>>>> and running the ldapmodify command >>>>> >>>>>>>>>> >>>>> >>>>>>>>>> I have now started moving clients running an openldap to >>>>> Freeipa >>>>> >>>> and >>>>> >>>>>>>>> have >>>>> >>>>>>>>>> today moved close to 2000 clients >>>>> >>>>>>>>>> >>>>> >>>>>>>>>> However, I have noticed that IPA hangs intermittently. >>>>> >>>>>>>>>> >>>>> >>>>>>>>>> running a kinit admin returns the below error >>>>> >>>>>>>>>> kinit: Generic error (see e-text) while getting initial >>>>> >>>> credentials >>>>> >>>>>>>>>> >>>>> >>>>>>>>>> from the /var/log/messages, I see this entry >>>>> >>>>>>>>>> >>>>> >>>>>>>>>> prod-ipa-master-int kernel: [104090.315801] TCP: >>>>> >>>> request_sock_TCP: >>>>> >>>>>>>>>> Possible SYN flooding on port 88. Sending cookies. Check >>>>> SNMP >>>>> >>>>>> counters. >>>>> >>>>>>>>> >>>>> >>>>>>>>> I would be worried about this message. Maybe kernel/firewall >>>>> is >>>>> >>>> doing >>>>> >>>>>>>>> something fishy behind your back and blocking some >>>>> connections or >>>>> >>>> so. >>>>> >>>>>>>>> >>>>> >>>>>>>>> Petr^2 Spacek >>>>> >>>>>>>>> >>>>> >>>>>>>>> >>>>> >>>>>>>>>> Aug 18 13:00:01 prod-ipa-master-int systemd[1]: Started >>>>> Session >>>>> >>>> 4885 >>>>> >>>>>> of >>>>> >>>>>>>>>> user root. >>>>> >>>>>>>>>> Aug 18 13:00:01 prod-ipa-master-int systemd[1]: Starting >>>>> Session >>>>> >>>> 4885 >>>>> >>>>>> of >>>>> >>>>>>>>>> user root. >>>>> >>>>>>>>>> Aug 18 13:01:01 prod-ipa-master-int systemd[1]: Started >>>>> Session >>>>> >>>> 4886 >>>>> >>>>>> of >>>>> >>>>>>>>>> user root. >>>>> >>>>>>>>>> Aug 18 13:01:01 prod-ipa-master-int systemd[1]: Starting >>>>> Session >>>>> >>>> 4886 >>>>> >>>>>> of >>>>> >>>>>>>>>> user root. >>>>> >>>>>>>>>> Aug 18 13:02:40 prod-ipa-master-int python[28984]: >>>>> ansible-command >>>>> >>>>>>>>> Invoked >>>>> >>>>>>>>>> with creates=None executable=None shell=True args= >>>>> removes=None >>>>> >>>>>>>>> warn=True >>>>> >>>>>>>>>> chdir=None >>>>> >>>>>>>>>> Aug 18 13:04:37 prod-ipa-master-int sssd_be: GSSAPI Error: >>>>> >>>> Unspecified >>>>> >>>>>>>>> GSS >>>>> >>>>>>>>>> failure. Minor code may provide more information (KDC >>>>> returned >>>>> >>>> error >>>>> >>>>>>>>>> string: PROCESS_TGS) >>>>> >>>>>>>>>> >>>>> >>>>>>>>>> Could it be possible that its due to the initial load of >>>>> adding >>>>> >>>> the >>>>> >>>>>>>>> clients >>>>> >>>>>>>>>> or is there something else that I need to take care of. >>>>> >>>> >>>> >>>> >>>> >>>> >>> >> >> > > > > > -- > Manage your subscription for the Freeipa-users mailing list: > https://www.redhat.com/mailman/listinfo/freeipa-users > Go to http://freeipa.org for more info on the project > -------------- next part -------------- An HTML attachment was scrubbed... URL: From rakesh.rajasekharan at gmail.com Sun Sep 4 18:40:43 2016 From: rakesh.rajasekharan at gmail.com (Rakesh Rajasekharan) Date: Mon, 5 Sep 2016 00:10:43 +0530 Subject: [Freeipa-users] Freeipa 4.2.0 hangs intermittently In-Reply-To: References: <7850c6f9-f79a-355e-4730-93bb67fbacf1@redhat.com> <57BEB0AB.2090506@redhat.com> <57C44AC6.1030402@redhat.com> Message-ID: starce on the slapd process actually had this in the output.. FUTEX_WAIT_PRIVATE and checking for the number of threads slapd had.. there were 5015 threads ps -efL|grep slapd|wc -l 5015 strace on most of the threads gave this output strace -p 67411 Process 67411 attached futex(0x7f3f0226b41c, FUTEX_WAIT_PRIVATE, 1, NULL) = -1 EAGAIN (Resource temporarily unavailable) futex(0x7f3f0226b41c, FUTEX_WAIT_PRIVATE, 2, NULL^CProcess 67411 detached On Sun, Sep 4, 2016 at 5:34 PM, Rakesh Rajasekharan < rakesh.rajasekharan at gmail.com> wrote: > I have again got the issue of IPA hanging.. The issue came up when i tried > to run ipa-client-isntall on 142 clients simultaneously > > > None of the IPA commands are responding, and I see this error > > ipa user-find p-testipa > ipa: ERROR: Insufficient access: SASL(-1): generic failure: GSSAPI Error: > Unspecified GSS failure. Minor code may provide more information (KDC > returned error string: PROCESS_TGS) > > KRB5_TRACE=/dev/stdout kinit admin > [41178] 1472984115.233214: Getting initial credentials for admin at XYZ.COM > [41178] 1472984115.235257: Sending request (167 bytes) to XYZ.COM > [41178] 1472984115.235419: Initiating TCP connection to stream > 10.1.3.36:88 > [41178] 1472984115.235685: Sending TCP request to stream 10.1.3.36:88 > [41178] 1472984120.238914: Received answer (174 bytes) from stream > 10.1.3.36:88 > [41178] 1472984120.238925: Terminating TCP connection to stream > 10.1.3.36:88 > [41178] 1472984120.238993: Response was from master KDC > [41 > > > Running an ldapsearch to see the db.. does not give any results and just > hangs there > > ldapsearch -x -D 'cn=Directory Manager' -W -s one -b > 'cn=kerberos,dc=xyz,dc=com' > Enter LDAP Password: > > even an ldapsearch -x does not respond > At this point, am sure that slapd is the one causing issues > > Running an strace against the hung slapd itself seems to get stuck does > not proceed after saying "attaching to process" > > From some others posts I read Thierry suggesting to increase the > nsslapd-threadnumber value > > It was set to 30, I think that might be too low. > > I have raised it to 500 > > Now after restarting the service .. ldapsearch starts responding. > But running the test to add a sudden high number of clients again left > ns-slapd to hung state > > When i attempted adding the clients.. the ns-slapd cpu usage shot up to > 340% and after that ns-slapd stopped responding > > So now, atleast I know what might be causing the issue and I can now > easily reproduce it. > > Is there a way I can make ns-slapd handle a sudden bump in incoming > request for ipa-client-install > > Thanks > Rakesh > > > > > > > On Mon, Aug 29, 2016 at 11:18 PM, Rich Megginson > wrote: > >> On 08/29/2016 10:53 AM, Rakesh Rajasekharan wrote: >> >> Hi Thierry, >> >> My machine has 30GB RAM ..and 389-ds version is 1.3.4 >> >> ldapsearch shows the values for nsslapd-cachememsize updated to 200MB. >> >> ldapsearch -LLL -o ldif-wrap=no -D "cn=directory manager" -w 'mypassword' >> -b 'cn=userRoot,cn=ldbm database,cn=plugins,cn=config'|grep >> nsslapd-cachememsize >> nsslapd-cachememsize: 209715200 >> >> >> So, it seems to have updated though seeing that warning(WARNING: ipaca: >> entry cache size 10485760B is less than db size 11599872B) in the log >> confuses me a bit. >> >> Thers one more entry that I found from the ldapsearch to be bit low >> >> nsslapd-dncachememsize: 10485760 >> maxdncachesize: 10485760 >> >> Should I update these as well to a higher value >> >> At the time when the issue happened, the memory usage as well as the >> overall load of the system was very low . >> I will try reproducing the issue atleast in my QA env..probably by trying >> to mock simultaneous parallel logins to a large number of hosts >> >> >> To monitor your cache sizes, please use the dbmon.sh tool provided with >> your distro. If that is not available with your particular distro, see >> https://github.com/richm/scripts/wiki/dbmon.sh >> >> >> >> >> thanks >> Rakesh >> >> >> >> >> On Mon, Aug 29, 2016 at 8:16 PM, thierry bordaz >> wrote: >> >>> Hi Rakesh, >>> >>> Those tuning may depend on the memory available on your machine. >>> nsslapd-cachememsize allows the entry cache to consume up to 200Mb but >>> its memory footprint is known to go above. >>> 200Mb both looks pretty good to me. How large is your machine ? What is >>> your version of 389-ds ? >>> >>> Those warnings do not change your settings. It just raise that entry >>> cache of 'ipaca' and 'retrocl' are small but it is fine. The size of the >>> entry cache is important mostly in userRoot. >>> You may double check the actual values, after restart, with ldapsearch >>> on 'cn=userRoot,cn=ldbm database,cn=plugins,cn=config' and >>> 'cn=config,cn=ldbm database,cn=plugins,cn=config'. >>> >>> A step is to know what will be response time of DS to know if it is >>> responsible of the hang or not. >>> The logs and possibly pstack during those intermittent hangs will help >>> to determine that. >>> >>> regards >>> thierry >>> >>> >>> >>> >>> >>> On 08/29/2016 04:25 PM, Rakesh Rajasekharan wrote: >>> >>> I tried increasing the nsslapd-dbcachesize and nsslapd-cachememsize in >>> my QA envs to 200MB. >>> >>> However, in my log files, I still see this message >>> [29/Aug/2016:04:34:37 +0000] - WARNING: ipaca: entry cache size >>> 10485760B is less than db size 11599872B; We recommend to increase the >>> entry cache size nsslapd-cachememsize. >>> [29/Aug/2016:04:34:37 +0000] - WARNING: changelog: entry cache size >>> 2097152B is less than db size 441647104B; We recommend to increase the >>> entry cache size nsslapd-cachememsize. >>> >>> these are my ldif files that i used to modify the values >>> modify entry cache size >>> cat modify-cache-mem-size.ldif >>> dn: cn=userRoot,cn=ldbm database,cn=plugins,cn=config >>> changetype: modify >>> replace: nsslapd-cachememsize >>> nsslapd-cachememsize: 209715200 >>> >>> modify db cache size >>> cat modfy-db-cache-size.ldif >>> dn: cn=config,cn=ldbm database,cn=plugins,cn=config >>> changetype: modify >>> replace: nsslapd-dbcachesize >>> nsslapd-dbcachesize: 209715200 >>> >>> After modifying , i restarted IPA services >>> >>> Is there anything else that I need to take care of as the logs suggest >>> its still not getting the updated values >>> >>> Thanks >>> Rakesh >>> >>> On Mon, Aug 29, 2016 at 6:07 PM, Rakesh Rajasekharan < >>> rakesh.rajasekharan at gmail.com> wrote: >>> >>>> Hi Thierry, >>>> >>>> Coz of the issues we had to revert back to earlier running openldap in >>>> production. >>>> >>>> I have now done a few TCP related changes in sysctl.conf and have also >>>> increased the nsslapd-dbcachesize and nsslapd-cachememsize to 200MB >>>> >>>> I will again start migrating hosts back to IPA and see if I face the >>>> earlier issue. >>>> >>>> I will update back once I have something >>>> >>>> >>>> Thanks, >>>> Rakesh >>>> >>>> >>>> >>>> On Thu, Aug 25, 2016 at 2:17 PM, thierry bordaz >>>> wrote: >>>> >>>>> >>>>> >>>>> On 08/25/2016 10:15 AM, Rakesh Rajasekharan wrote: >>>>> >>>>> All of the troubleshooting seems fine. >>>>> >>>>> >>>>> However, Running libconv.pl gives me this output >>>>> >>>>> ----- Recommendations ----- >>>>> >>>>> 1. You have unindexed components, this can be caused from a search >>>>> on an unindexed attribute, or your returned results exceeded the >>>>> allidsthreshold. Unindexed components are not recommended. To refuse >>>>> unindexed searches, switch 'nsslapd-require-index' to 'on' under your >>>>> database entry (e.g. cn=UserRoot,cn=ldbm database,cn=plugins,cn=config) >>>>> . >>>>> >>>>> 2. You have a significant difference between binds and unbinds. You >>>>> may want to investigate this difference. >>>>> >>>>> >>>>> I feel, this could be a pointer to things going slow.. and IPA >>>>> hanging. I think i now have something that I can try and nail down this >>>>> issue. >>>>> >>>>> On a sidenote, I was earlier running openldap and migrated over to >>>>> Freeipa, >>>>> >>>>> Thanks >>>>> Rakesh >>>>> >>>>> >>>>> >>>>> On Wed, Aug 24, 2016 at 12:38 PM, Petr Spacek >>>>> wrote: >>>>> >>>>>> On 23.8.2016 18:44, Rakesh Rajasekharan wrote: >>>>>> > I think thers something seriously wrong with my system >>>>>> > >>>>>> > not able to run any IPA commands >>>>>> > >>>>>> > klist >>>>>> > Ticket cache: KEYRING:persistent:0:0 >>>>>> > Default principal: admin at XYZ.COM >>>>>> > >>>>>> > Valid starting Expires Service principal >>>>>> > 2016-08-23T16:26:36 2016-08-24T16:26:22 krbtgt/XYZ.COM at XYZ.COM >>>>>> > >>>>>> > >>>>>> > [root at prod-ipa-master-1a :~] ipactl status >>>>>> > Directory Service: RUNNING >>>>>> > krb5kdc Service: RUNNING >>>>>> > kadmin Service: RUNNING >>>>>> > ipa_memcached Service: RUNNING >>>>>> > httpd Service: RUNNING >>>>>> > pki-tomcatd Service: RUNNING >>>>>> > ipa-otpd Service: RUNNING >>>>>> > ipa: INFO: The ipactl command was successful >>>>>> > >>>>>> > >>>>>> > >>>>>> > [root at prod-ipa-master :~] ipa user-find p-testuser >>>>>> > ipa: ERROR: Kerberos error: ('Unspecified GSS failure. Minor code >>>>>> may >>>>>> > provide more information', 851968)/("Cannot contact any KDC for >>>>>> realm ' >>>>>> > XYZ.COM'", -1765328228) >>>>>> >>>>> >>>>> Hi Rakesh, >>>>> >>>>> Having a reproducible test case would you rerun the command above. >>>>> During its processing you may monitor DS process load (top). If it is >>>>> high, you may get some pstacks of it. >>>>> Also would you attach the part of DS access logs taken during the >>>>> command. >>>>> >>>>> regards >>>>> thierry >>>>> >>>>> > >>>>>> >>>>>> This is weird because the server seems to be up. >>>>>> >>>>>> Please follow >>>>>> http://www.freeipa.org/page/Troubleshooting#Authentication.2FKerberos >>>>>> >>>>>> Petr^2 Spacek >>>>>> >>>>>> > >>>>>> > >>>>>> > Thanks >>>>>> > >>>>>> > Rakesh >>>>>> > >>>>>> > On Tue, Aug 23, 2016 at 10:01 PM, Rakesh Rajasekharan < >>>>>> > rakesh.rajasekharan at gmail.com> wrote: >>>>>> > >>>>>> >> i changed the loggin level to 4 . Modifying nsslapd-accesslog-level >>>>>> >> >>>>>> >> But, the hang is still there. though I dont see the sigfault now >>>>>> >> >>>>>> >> >>>>>> >> >>>>>> >> >>>>>> >> On Tue, Aug 23, 2016 at 9:02 PM, Rakesh Rajasekharan < >>>>>> >> rakesh.rajasekharan at gmail.com> wrote: >>>>>> >> >>>>>> >>> My disk was getting filled too fast >>>>>> >>> >>>>>> >>> logs under /var/log/dirsrv was coming around 5 gb quickly filling >>>>>> up >>>>>> >>> >>>>>> >>> Is there a way to make the logging less verbose >>>>>> >>> >>>>>> >>> >>>>>> >>> >>>>>> >>> On Tue, Aug 23, 2016 at 6:41 PM, Petr Spacek >>>>>> wrote: >>>>>> >>> >>>>>> >>>> On 23.8.2016 15:07, Rakesh Rajasekharan wrote: >>>>>> >>>>> I was able to fix that may be temporarily... when i checked the >>>>>> >>>> network.. >>>>>> >>>>> there was another process that was running and consuming a lot >>>>>> of >>>>>> >>>> network ( >>>>>> >>>>> i have no idea who did that. I need to seriously start >>>>>> restricting >>>>>> >>>> people >>>>>> >>>>> access to this machine ) >>>>>> >>>>> >>>>>> >>>>> after killing that perfomance improved drastically >>>>>> >>>>> >>>>>> >>>>> But now, suddenly I started experiencing the same hang. >>>>>> >>>>> >>>>>> >>>>> This time , I gert the following error when checked dmesg >>>>>> >>>>> >>>>>> >>>>> [ 301.236976] ns-slapd[3124]: segfault at 0 ip >>>>>> 00007f1de416951c sp >>>>>> >>>>> 00007f1dee1dba70 error 4 in libcos-plugin.so[7f1de4166000+b000] >>>>>> >>>>> [ 1116.248431] TCP: request_sock_TCP: Possible SYN flooding on >>>>>> port 88. >>>>>> >>>>> Sending cookies. Check SNMP counters. >>>>>> >>>>> [11831.397037] ns-slapd[22550]: segfault at 0 ip >>>>>> 00007f533d82251c sp >>>>>> >>>>> 00007f5347894a70 error 4 in libcos-plugin.so[7f533d81f000+b000] >>>>>> >>>>> [11832.727989] ns-slapd[22606]: segfault at 0 ip >>>>>> 00007f6231eb951c sp >>>>>> >>>>> 00007f623bf2ba70 error 4 in libcos-plugin.so[7f6231eb6000+b00 >>>>>> >>>> >>>>>> >>>> Okay, this one is serious. The LDAP server crashed. >>>>>> >>>> >>>>>> >>>> 1. Make sure all your packages are up-to-date. >>>>>> >>>> >>>>>> >>>> Please see >>>>>> >>>> http://directory.fedoraproject.org/docs/389ds/FAQ/faq.html#d >>>>>> >>>> ebugging-crashes >>>>>> >>>> for further instructions how to debug this. >>>>>> >>>> >>>>>> >>>> Petr^2 Spacek >>>>>> >>>> >>>>>> >>>>> >>>>>> >>>>> and in /var/log/dirsrv/example-com/errors >>>>>> >>>>> >>>>>> >>>>> [23/Aug/2016:12:49:36 +0000] DSRetroclPlugin - >>>>>> delete_changerecord: >>>>>> >>>> could >>>>>> >>>>> not delete change record 3291138 (rc: 32) >>>>>> >>>>> [23/Aug/2016:12:49:36 +0000] DSRetroclPlugin - >>>>>> delete_changerecord: >>>>>> >>>> could >>>>>> >>>>> not delete change record 3291139 (rc: 32) >>>>>> >>>>> [23/Aug/2016:12:49:36 +0000] DSRetroclPlugin - >>>>>> delete_changerecord: >>>>>> >>>> could >>>>>> >>>>> not delete change record 3291140 (rc: 32) >>>>>> >>>>> [23/Aug/2016:12:49:36 +0000] DSRetroclPlugin - >>>>>> delete_changerecord: >>>>>> >>>> could >>>>>> >>>>> not delete change record 3291141 (rc: 32) >>>>>> >>>>> [23/Aug/2016:12:49:36 +0000] DSRetroclPlugin - >>>>>> delete_changerecord: >>>>>> >>>> could >>>>>> >>>>> not delete change record 3291142 (rc: 32) >>>>>> >>>>> [23/Aug/2016:12:49:36 +0000] DSRetroclPlugin - >>>>>> delete_changerecord: >>>>>> >>>> could >>>>>> >>>>> not delete change record 3291143 (rc: 32) >>>>>> >>>>> [23/Aug/2016:12:49:36 +0000] DSRetroclPlugin - >>>>>> delete_changerecord: >>>>>> >>>> could >>>>>> >>>>> not delete change record 3291144 (rc: 32) >>>>>> >>>>> [23/Aug/2016:12:49:36 +0000] DSRetroclPlugin - >>>>>> delete_changerecord: >>>>>> >>>> could >>>>>> >>>>> not delete change record 3291145 (rc: 32) >>>>>> >>>>> [23/Aug/2016:12:49:50 +0000] - Retry count exceeded in delete >>>>>> >>>>> [23/Aug/2016:12:49:50 +0000] DSRetroclPlugin - >>>>>> delete_changerecord: >>>>>> >>>> could >>>>>> >>>>> not delete change record 3292734 (rc: 51) >>>>>> >>>>> >>>>>> >>>>> >>>>>> >>>>> Can i do something about this error.. I treid to restart ipa a >>>>>> couple >>>>>> >>>> of >>>>>> >>>>> time but that did not help >>>>>> >>>>> >>>>>> >>>>> Thanks >>>>>> >>>>> Rakesh >>>>>> >>>>> >>>>>> >>>>> On Mon, Aug 22, 2016 at 2:27 PM, Petr Spacek < >>>>>> pspacek at redhat.com> >>>>>> >>>> wrote: >>>>>> >>>>> >>>>>> >>>>>> On 19.8.2016 19:32, Rakesh Rajasekharan wrote: >>>>>> >>>>>>> I am running my set up on AWS cloud, and entropy is low at >>>>>> around >>>>>> >>>> 180 . >>>>>> >>>>>>> >>>>>> >>>>>>> I plan to increase it bu installing haveged . But, would low >>>>>> entropy >>>>>> >>>> by >>>>>> >>>>>> any >>>>>> >>>>>>> chance cause this issue of intermittent hang . >>>>>> >>>>>>> Also, the hang is mostly observed when registering around 20 >>>>>> clients >>>>>> >>>>>>> together >>>>>> >>>>>> >>>>>> >>>>>> Possibly, I'm not sure. If you want to dig into this, I would >>>>>> do this: >>>>>> >>>>>> 1. look what process hangs on client (using pstree command or >>>>>> so) >>>>>> >>>>>> $ pstree >>>>>> >>>>>> >>>>>> >>>>>> 2. look to what server and port is the hanging client >>>>>> connected to >>>>>> >>>>>> $ lsof -p >>>>>> >>>>>> >>>>>> >>>>>> 3. jump to server and see what process is bound to the target >>>>>> port >>>>>> >>>>>> $ netstat -pn >>>>>> >>>>>> >>>>>> >>>>>> 4. see where the process if hanging >>>>>> >>>>>> $ strace -p >>>>>> >>>>>> >>>>>> >>>>>> I hope it helps. >>>>>> >>>>>> >>>>>> >>>>>> Petr^2 Spacek >>>>>> >>>>>> >>>>>> >>>>>>> On Fri, Aug 19, 2016 at 7:24 PM, Rakesh Rajasekharan < >>>>>> >>>>>>> rakesh.rajasekharan at gmail.com> wrote: >>>>>> >>>>>>> >>>>>> >>>>>>>> yes there seems to be something thats worrying.. I have >>>>>> faced this >>>>>> >>>> today >>>>>> >>>>>>>> as well. >>>>>> >>>>>>>> There are few hosts around 280 odd left and when i try >>>>>> adding them >>>>>> >>>> to >>>>>> >>>>>> IPA >>>>>> >>>>>>>> , the slowness begins.. >>>>>> >>>>>>>> >>>>>> >>>>>>>> all the ipa commands like ipa user-find.. etc becomes very >>>>>> slow in >>>>>> >>>>>>>> responding. >>>>>> >>>>>>>> >>>>>> >>>>>>>> the SYNC_RECV are not many though just around 80-90 and >>>>>> today that >>>>>> >>>> was >>>>>> >>>>>>>> around 20 only >>>>>> >>>>>>>> >>>>>> >>>>>>>> >>>>>> >>>>>>>> I have for now increased tcp_max_syn_backlog to 5000. >>>>>> >>>>>>>> For now the slowness seems to have gone.. but I will do a try >>>>>> >>>> adding the >>>>>> >>>>>>>> clients again tomorrow and see how it goes >>>>>> >>>>>>>> >>>>>> >>>>>>>> Thanks >>>>>> >>>>>>>> Rakesh >>>>>> >>>>>>>> >>>>>> >>>>>>>> The issues >>>>>> >>>>>>>> >>>>>> >>>>>>>> On Fri, Aug 19, 2016 at 12:58 PM, Petr Spacek < >>>>>> pspacek at redhat.com> >>>>>> >>>>>> wrote: >>>>>> >>>>>>>> >>>>>> >>>>>>>>> On 18.8.2016 17:23, Rakesh Rajasekharan wrote: >>>>>> >>>>>>>>>> Hi >>>>>> >>>>>>>>>> >>>>>> >>>>>>>>>> I am migrating to freeipa from openldap and have around >>>>>> 4000 >>>>>> >>>> clients >>>>>> >>>>>>>>>> >>>>>> >>>>>>>>>> I had openned a another thread on that, but chose to start >>>>>> a new >>>>>> >>>> one >>>>>> >>>>>>>>> here >>>>>> >>>>>>>>>> as its a separate issue >>>>>> >>>>>>>>>> >>>>>> >>>>>>>>>> I was able to change the nssslapd-maxdescriptors adding an >>>>>> ldif >>>>>> >>>> file >>>>>> >>>>>>>>>> >>>>>> >>>>>>>>>> cat nsslapd-modify.ldif >>>>>> >>>>>>>>>> dn: cn=config >>>>>> >>>>>>>>>> changetype: modify >>>>>> >>>>>>>>>> replace: nsslapd-maxdescriptors >>>>>> >>>>>>>>>> nsslapd-maxdescriptors: 17000 >>>>>> >>>>>>>>>> >>>>>> >>>>>>>>>> and running the ldapmodify command >>>>>> >>>>>>>>>> >>>>>> >>>>>>>>>> I have now started moving clients running an openldap to >>>>>> Freeipa >>>>>> >>>> and >>>>>> >>>>>>>>> have >>>>>> >>>>>>>>>> today moved close to 2000 clients >>>>>> >>>>>>>>>> >>>>>> >>>>>>>>>> However, I have noticed that IPA hangs intermittently. >>>>>> >>>>>>>>>> >>>>>> >>>>>>>>>> running a kinit admin returns the below error >>>>>> >>>>>>>>>> kinit: Generic error (see e-text) while getting initial >>>>>> >>>> credentials >>>>>> >>>>>>>>>> >>>>>> >>>>>>>>>> from the /var/log/messages, I see this entry >>>>>> >>>>>>>>>> >>>>>> >>>>>>>>>> prod-ipa-master-int kernel: [104090.315801] TCP: >>>>>> >>>> request_sock_TCP: >>>>>> >>>>>>>>>> Possible SYN flooding on port 88. Sending cookies. Check >>>>>> SNMP >>>>>> >>>>>> counters. >>>>>> >>>>>>>>> >>>>>> >>>>>>>>> I would be worried about this message. Maybe >>>>>> kernel/firewall is >>>>>> >>>> doing >>>>>> >>>>>>>>> something fishy behind your back and blocking some >>>>>> connections or >>>>>> >>>> so. >>>>>> >>>>>>>>> >>>>>> >>>>>>>>> Petr^2 Spacek >>>>>> >>>>>>>>> >>>>>> >>>>>>>>> >>>>>> >>>>>>>>>> Aug 18 13:00:01 prod-ipa-master-int systemd[1]: Started >>>>>> Session >>>>>> >>>> 4885 >>>>>> >>>>>> of >>>>>> >>>>>>>>>> user root. >>>>>> >>>>>>>>>> Aug 18 13:00:01 prod-ipa-master-int systemd[1]: Starting >>>>>> Session >>>>>> >>>> 4885 >>>>>> >>>>>> of >>>>>> >>>>>>>>>> user root. >>>>>> >>>>>>>>>> Aug 18 13:01:01 prod-ipa-master-int systemd[1]: Started >>>>>> Session >>>>>> >>>> 4886 >>>>>> >>>>>> of >>>>>> >>>>>>>>>> user root. >>>>>> >>>>>>>>>> Aug 18 13:01:01 prod-ipa-master-int systemd[1]: Starting >>>>>> Session >>>>>> >>>> 4886 >>>>>> >>>>>> of >>>>>> >>>>>>>>>> user root. >>>>>> >>>>>>>>>> Aug 18 13:02:40 prod-ipa-master-int python[28984]: >>>>>> ansible-command >>>>>> >>>>>>>>> Invoked >>>>>> >>>>>>>>>> with creates=None executable=None shell=True args= >>>>>> removes=None >>>>>> >>>>>>>>> warn=True >>>>>> >>>>>>>>>> chdir=None >>>>>> >>>>>>>>>> Aug 18 13:04:37 prod-ipa-master-int sssd_be: GSSAPI Error: >>>>>> >>>> Unspecified >>>>>> >>>>>>>>> GSS >>>>>> >>>>>>>>>> failure. Minor code may provide more information (KDC >>>>>> returned >>>>>> >>>> error >>>>>> >>>>>>>>>> string: PROCESS_TGS) >>>>>> >>>>>>>>>> >>>>>> >>>>>>>>>> Could it be possible that its due to the initial load of >>>>>> adding >>>>>> >>>> the >>>>>> >>>>>>>>> clients >>>>>> >>>>>>>>>> or is there something else that I need to take care of. >>>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>> >>> >>> >> >> >> >> >> -- >> Manage your subscription for the Freeipa-users mailing list: >> https://www.redhat.com/mailman/listinfo/freeipa-users >> Go to http://freeipa.org for more info on the project >> > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From th at casalogic.dk Mon Sep 5 07:02:04 2016 From: th at casalogic.dk (Troels Hansen) Date: Mon, 5 Sep 2016 09:02:04 +0200 (CEST) Subject: [Freeipa-users] SUDO and group lookup in AD trust In-Reply-To: <20160902075629.djs3yy22zk4p3kiz@hendrix> References: <2050149863.1012561.1471958268423.JavaMail.zimbra@casalogic.dk> <260135086.10187.1472112336235.JavaMail.zimbra@casalogic.dk> <20160825084820.GB30315@10.4.128.1> <1042341051.13710.1472117452652.JavaMail.zimbra@casalogic.dk> <20160825204152.GA6360@10.4.128.1> <20160826055407.sok4s6c7ehtwoe3q@hendrix> <20160902072757.GA6086@10.4.128.1> <20160902075629.djs3yy22zk4p3kiz@hendrix> Message-ID: <793201723.148000.1473058924080.JavaMail.zimbra@casalogic.dk> ----- On Sep 2, 2016, at 9:56 AM, Jakub Hrozek jhrozek at redhat.com wrote: >> >We were debugging this yesterday with Troels and the logs said it's: >> > https://fedorahosted.org/sssd/ticket/3127 >> > >> Fixed version is in 1.14 copr > > Thank you, btw another affected user confirmed that the patch fixes the > problem. > Hi, one question. I can see that ticket 2919 targeted for 1.14.0 and 3127 which fixes our problem is targeted for SSSD 1.14.2 and RHEL 7.3beta currently runs 1.14.0 Will RHEL7.3 have the currently working SSSD 1.14.1 from jhrozek's COPR repo (test build https://copr.fedorainfracloud.org/coprs/jhrozek/sssd-principal-test/) From mbasti at redhat.com Mon Sep 5 07:12:08 2016 From: mbasti at redhat.com (Martin Basti) Date: Mon, 5 Sep 2016 09:12:08 +0200 Subject: [Freeipa-users] General query regarding nameserver enrtry In-Reply-To: References: Message-ID: <4d9223c9-216b-d79d-5c93-ec81466c75c6@redhat.com> On 02.09.2016 20:06, Deepak Dimri wrote: > Hi All, > > My ipa-client-install fails until etc/resolve.conf gets updated with > IPA nameserver entry. I want to avoid a task of updating > resolve.conf in my automation script. Is there a way i can get my IPA > client installation successful without updating resolve.conf? what > options do i have? > > > Many Thanks, > Deepak > > Hello, do you have proper zone delegation? With proper zone delegation it should be able to resolve IPA from every nameserver. Martin -------------- next part -------------- An HTML attachment was scrubbed... URL: From jhrozek at redhat.com Mon Sep 5 07:26:39 2016 From: jhrozek at redhat.com (Jakub Hrozek) Date: Mon, 5 Sep 2016 09:26:39 +0200 Subject: [Freeipa-users] SUDO and group lookup in AD trust In-Reply-To: <793201723.148000.1473058924080.JavaMail.zimbra@casalogic.dk> References: <2050149863.1012561.1471958268423.JavaMail.zimbra@casalogic.dk> <260135086.10187.1472112336235.JavaMail.zimbra@casalogic.dk> <20160825084820.GB30315@10.4.128.1> <1042341051.13710.1472117452652.JavaMail.zimbra@casalogic.dk> <20160825204152.GA6360@10.4.128.1> <20160826055407.sok4s6c7ehtwoe3q@hendrix> <20160902072757.GA6086@10.4.128.1> <20160902075629.djs3yy22zk4p3kiz@hendrix> <793201723.148000.1473058924080.JavaMail.zimbra@casalogic.dk> Message-ID: <20160905072639.ifofnqwuvcxv54tx@hendrix> On Mon, Sep 05, 2016 at 09:02:04AM +0200, Troels Hansen wrote: > > > ----- On Sep 2, 2016, at 9:56 AM, Jakub Hrozek jhrozek at redhat.com wrote: > >> >We were debugging this yesterday with Troels and the logs said it's: > >> > https://fedorahosted.org/sssd/ticket/3127 > >> > > >> Fixed version is in 1.14 copr > > > > Thank you, btw another affected user confirmed that the patch fixes the > > problem. > > > > > Hi, one question. I can see that ticket 2919 targeted for 1.14.0 and 3127 which fixes our problem is targeted for SSSD 1.14.2 and RHEL 7.3beta currently runs 1.14.0 > > Will RHEL7.3 have the currently working SSSD 1.14.1 from jhrozek's COPR repo (test build https://copr.fedorainfracloud.org/coprs/jhrozek/sssd-principal-test/) Not 1.14.1, but 7.3 will have the patch I put in my COPR repo (that's in general how RHEL works, we can rebase only up to a certain point and then we cherry-pick selected approved patches to avoid breaking RHEL with some unrelated upstream change..) From tbordaz at redhat.com Mon Sep 5 07:33:48 2016 From: tbordaz at redhat.com (thierry bordaz) Date: Mon, 5 Sep 2016 09:33:48 +0200 Subject: [Freeipa-users] Freeipa 4.2.0 hangs intermittently In-Reply-To: References: <57BEB0AB.2090506@redhat.com> <57C44AC6.1030402@redhat.com> Message-ID: <57CD1FDC.2010109@redhat.com> Hi Rakesh, Were you able to get a pstack or full stack with gdb (http://www.port389.org/docs/389ds/FAQ/faq.html#debugging-crashes) when the server hangs ? If it happens with 500 threads as well as with 30, using 30 threads is a better choice to debug this issue. I will try to reproduce using 150 parallel 'ipa user-find p-testipa' commands Something I am unsure is if the CPU consumption stays high (you mentioned 340% CPU usage) as long as the hang happens or if after a suddent shot up to 340% (that marks the beginning of the hang) it drops and stay hanging ? thanks thierry On 09/04/2016 08:40 PM, Rakesh Rajasekharan wrote: > starce on the slapd process actually had this in the output.. > FUTEX_WAIT_PRIVATE > > and checking for the number of threads slapd had.. there were 5015 threads > > ps -efL|grep slapd|wc -l > 5015 > > strace on most of the threads gave this output > > strace -p 67411 > Process 67411 attached > futex(0x7f3f0226b41c, FUTEX_WAIT_PRIVATE, 1, NULL) = -1 EAGAIN > (Resource temporarily unavailable) > futex(0x7f3f0226b41c, FUTEX_WAIT_PRIVATE, 2, NULL^CProcess 67411 detached > > > > > > On Sun, Sep 4, 2016 at 5:34 PM, Rakesh Rajasekharan > > > wrote: > > I have again got the issue of IPA hanging.. The issue came up when > i tried to run ipa-client-isntall on 142 clients simultaneously > > > None of the IPA commands are responding, and I see this error > > ipa user-find p-testipa > ipa: ERROR: Insufficient access: SASL(-1): generic failure: GSSAPI > Error: Unspecified GSS failure. Minor code may provide more > information (KDC returned error string: PROCESS_TGS) > > KRB5_TRACE=/dev/stdout kinit admin > [41178] 1472984115.233214: Getting initial credentials for > admin at XYZ.COM > [41178] 1472984115.235257: Sending request (167 bytes) to XYZ.COM > > [41178] 1472984115.235419: Initiating TCP connection to stream > 10.1.3.36:88 > [41178] 1472984115.235685: Sending TCP request to stream > 10.1.3.36:88 > [41178] 1472984120.238914: Received answer (174 bytes) from stream > 10.1.3.36:88 > [41178] 1472984120.238925: Terminating TCP connection to stream > 10.1.3.36:88 > [41178] 1472984120.238993: Response was from master KDC > [41 > > > Running an ldapsearch to see the db.. does not give any results > and just hangs there > > ldapsearch -x -D 'cn=Directory Manager' -W -s one -b > 'cn=kerberos,dc=xyz,dc=com' > Enter LDAP Password: > > even an ldapsearch -x does not respond > At this point, am sure that slapd is the one causing issues > > Running an strace against the hung slapd itself seems to get stuck > does not proceed after saying "attaching to process" > > From some others posts I read Thierry suggesting to increase the > nsslapd-threadnumber value > > It was set to 30, I think that might be too low. > > I have raised it to 500 > > Now after restarting the service .. ldapsearch starts responding. > But running the test to add a sudden high number of clients again > left ns-slapd to hung state > > When i attempted adding the clients.. the ns-slapd cpu usage shot > up to 340% and after that ns-slapd stopped responding > > So now, atleast I know what might be causing the issue and I can > now easily reproduce it. > > Is there a way I can make ns-slapd handle a sudden bump in > incoming request for ipa-client-install > > Thanks > Rakesh > > > > > > > On Mon, Aug 29, 2016 at 11:18 PM, Rich Megginson > > wrote: > > On 08/29/2016 10:53 AM, Rakesh Rajasekharan wrote: >> Hi Thierry, >> >> My machine has 30GB RAM ..and 389-ds version is 1.3.4 >> >> ldapsearch shows the values for nsslapd-cachememsize updated >> to 200MB. >> >> ldapsearch -LLL -o ldif-wrap=no -D "cn=directory manager" -w >> 'mypassword' -b 'cn=userRoot,cn=ldbm >> database,cn=plugins,cn=config'|grep nsslapd-cachememsize >> nsslapd-cachememsize: 209715200 >> >> >> So, it seems to have updated though seeing that >> warning(WARNING: ipaca: entry cache size 10485760B is less >> than db size 11599872B) in the log confuses me a bit. >> >> Thers one more entry that I found from the ldapsearch to be >> bit low >> >> nsslapd-dncachememsize: 10485760 >> maxdncachesize: 10485760 >> >> Should I update these as well to a higher value >> >> At the time when the issue happened, the memory usage as well >> as the overall load of the system was very low . >> I will try reproducing the issue atleast in my QA >> env..probably by trying to mock simultaneous parallel logins >> to a large number of hosts > > To monitor your cache sizes, please use the dbmon.sh tool > provided with your distro. If that is not available with your > particular distro, see > https://github.com/richm/scripts/wiki/dbmon.sh > > > >> >> >> thanks >> Rakesh >> >> >> >> >> On Mon, Aug 29, 2016 at 8:16 PM, thierry bordaz >> > wrote: >> >> Hi Rakesh, >> >> Those tuning may depend on the memory available on your >> machine. >> nsslapd-cachememsize allows the entry cache to consume up >> to 200Mb but its memory footprint is known to go above. >> 200Mb both looks pretty good to me. How large is your >> machine ? What is your version of 389-ds ? >> >> Those warnings do not change your settings. It just raise >> that entry cache of 'ipaca' and 'retrocl' are small but >> it is fine. The size of the entry cache is important >> mostly in userRoot. >> You may double check the actual values, after restart, >> with ldapsearch on 'cn=userRoot,cn=ldbm >> database,cn=plugins,cn=config' and 'cn=config,cn=ldbm >> database,cn=plugins,cn=config'. >> >> A step is to know what will be response time of DS to >> know if it is responsible of the hang or not. >> The logs and possibly pstack during those intermittent >> hangs will help to determine that. >> >> regards >> thierry >> >> >> >> >> >> On 08/29/2016 04:25 PM, Rakesh Rajasekharan wrote: >>> I tried increasing the nsslapd-dbcachesize and >>> nsslapd-cachememsize in my QA envs to 200MB. >>> >>> However, in my log files, I still see this message >>> [29/Aug/2016:04:34:37 +0000] - WARNING: ipaca: entry >>> cache size 10485760B is less than db size 11599872B; We >>> recommend to increase the entry cache size >>> nsslapd-cachememsize. >>> [29/Aug/2016:04:34:37 +0000] - WARNING: changelog: entry >>> cache size 2097152B is less than db size 441647104B; We >>> recommend to increase the entry cache size >>> nsslapd-cachememsize. >>> >>> these are my ldif files that i used to modify the values >>> modify entry cache size >>> cat modify-cache-mem-size.ldif >>> dn: cn=userRoot,cn=ldbm database,cn=plugins,cn=config >>> changetype: modify >>> replace: nsslapd-cachememsize >>> nsslapd-cachememsize: 209715200 >>> >>> modify db cache size >>> cat modfy-db-cache-size.ldif >>> dn: cn=config,cn=ldbm database,cn=plugins,cn=config >>> changetype: modify >>> replace: nsslapd-dbcachesize >>> nsslapd-dbcachesize: 209715200 >>> >>> After modifying , i restarted IPA services >>> >>> Is there anything else that I need to take care of as >>> the logs suggest its still not getting the updated values >>> >>> Thanks >>> Rakesh >>> >>> On Mon, Aug 29, 2016 at 6:07 PM, Rakesh Rajasekharan >>> >> > wrote: >>> >>> Hi Thierry, >>> >>> Coz of the issues we had to revert back to earlier >>> running openldap in production. >>> >>> I have now done a few TCP related changes in >>> sysctl.conf and have also increased the >>> nsslapd-dbcachesize and nsslapd-cachememsize to 200MB >>> >>> I will again start migrating hosts back to IPA and >>> see if I face the earlier issue. >>> >>> I will update back once I have something >>> >>> >>> Thanks, >>> Rakesh >>> >>> >>> >>> On Thu, Aug 25, 2016 at 2:17 PM, thierry bordaz >>> > wrote: >>> >>> >>> >>> On 08/25/2016 10:15 AM, Rakesh Rajasekharan wrote: >>>> All of the troubleshooting seems fine. >>>> >>>> >>>> However, Running libconv.pl >>>> gives me this output >>>> >>>> ----- Recommendations ----- >>>> >>>> 1. You have unindexed components, this can be >>>> caused from a search on an unindexed attribute, >>>> or your returned results exceeded the >>>> allidsthreshold. Unindexed components are not >>>> recommended. To refuse unindexed searches, >>>> switch 'nsslapd-require-index' to 'on' under >>>> your database entry (e.g. cn=UserRoot,cn=ldbm >>>> database,cn=plugins,cn=config). >>>> >>>> 2. You have a significant difference between >>>> binds and unbinds. You may want to investigate >>>> this difference. >>>> >>>> >>>> I feel, this could be a pointer to things going >>>> slow.. and IPA hanging. I think i now have >>>> something that I can try and nail down this issue. >>>> >>>> On a sidenote, I was earlier running openldap >>>> and migrated over to Freeipa, >>>> >>>> Thanks >>>> Rakesh >>>> >>>> >>>> >>>> On Wed, Aug 24, 2016 at 12:38 PM, Petr Spacek >>>> >>> > wrote: >>>> >>>> On 23.8.2016 18:44, Rakesh Rajasekharan wrote: >>>> > I think thers something seriously wrong >>>> with my system >>>> > >>>> > not able to run any IPA commands >>>> > >>>> > klist >>>> > Ticket cache: KEYRING:persistent:0:0 >>>> > Default principal: admin at XYZ.COM >>>> >>>> > >>>> > Valid starting Expires Service principal >>>> > 2016-08-23T16:26:36 2016-08-24T16:26:22 >>>> krbtgt/XYZ.COM at XYZ.COM >>>> > >>>> > >>>> > [root at prod-ipa-master-1a :~] ipactl status >>>> > Directory Service: RUNNING >>>> > krb5kdc Service: RUNNING >>>> > kadmin Service: RUNNING >>>> > ipa_memcached Service: RUNNING >>>> > httpd Service: RUNNING >>>> > pki-tomcatd Service: RUNNING >>>> > ipa-otpd Service: RUNNING >>>> > ipa: INFO: The ipactl command was successful >>>> > >>>> > >>>> > >>>> > [root at prod-ipa-master :~] ipa user-find >>>> p-testuser >>>> > ipa: ERROR: Kerberos error: ('Unspecified >>>> GSS failure. Minor code may >>>> > provide more information', >>>> 851968)/("Cannot contact any KDC for realm ' >>>> > XYZ.COM '", -1765328228) >>>> >>> >>> Hi Rakesh, >>> >>> Having a reproducible test case would you >>> rerun the command above. >>> During its processing you may monitor DS >>> process load (top). If it is high, you may >>> get some pstacks of it. >>> Also would you attach the part of DS access >>> logs taken during the command. >>> >>> regards >>> thierry >>> >>>> > >>>> >>>> This is weird because the server seems to >>>> be up. >>>> >>>> Please follow >>>> http://www.freeipa.org/page/Troubleshooting#Authentication.2FKerberos >>>> >>>> >>>> Petr^2 Spacek >>>> >>>> > >>>> > >>>> > Thanks >>>> > >>>> > Rakesh >>>> > >>>> > On Tue, Aug 23, 2016 at 10:01 PM, Rakesh >>>> Rajasekharan < >>>> > rakesh.rajasekharan at gmail.com >>>> > wrote: >>>> > >>>> >> i changed the loggin level to 4 . >>>> Modifying nsslapd-accesslog-level >>>> >> >>>> >> But, the hang is still there. though I >>>> dont see the sigfault now >>>> >> >>>> >> >>>> >> >>>> >> >>>> >> On Tue, Aug 23, 2016 at 9:02 PM, Rakesh >>>> Rajasekharan < >>>> >> rakesh.rajasekharan at gmail.com >>>> > wrote: >>>> >> >>>> >>> My disk was getting filled too fast >>>> >>> >>>> >>> logs under /var/log/dirsrv was coming >>>> around 5 gb quickly filling up >>>> >>> >>>> >>> Is there a way to make the logging less >>>> verbose >>>> >>> >>>> >>> >>>> >>> >>>> >>> On Tue, Aug 23, 2016 at 6:41 PM, Petr >>>> Spacek >>> > wrote: >>>> >>> >>>> >>>> On 23.8.2016 15:07, Rakesh >>>> Rajasekharan wrote: >>>> >>>>> I was able to fix that may be >>>> temporarily... when i checked the >>>> >>>> network.. >>>> >>>>> there was another process that was >>>> running and consuming a lot of >>>> >>>> network ( >>>> >>>>> i have no idea who did that. I need >>>> to seriously start restricting >>>> >>>> people >>>> >>>>> access to this machine ) >>>> >>>>> >>>> >>>>> after killing that perfomance >>>> improved drastically >>>> >>>>> >>>> >>>>> But now, suddenly I started >>>> experiencing the same hang. >>>> >>>>> >>>> >>>>> This time , I gert the following >>>> error when checked dmesg >>>> >>>>> >>>> >>>>> [ 301.236976] ns-slapd[3124]: >>>> segfault at 0 ip 00007f1de416951c sp >>>> >>>>> 00007f1dee1dba70 error 4 in >>>> libcos-plugin.so[7f1de4166000+b000] >>>> >>>>> [ 1116.248431] TCP: request_sock_TCP: >>>> Possible SYN flooding on port 88. >>>> >>>>> Sending cookies. Check SNMP counters. >>>> >>>>> [11831.397037] ns-slapd[22550]: >>>> segfault at 0 ip 00007f533d82251c sp >>>> >>>>> 00007f5347894a70 error 4 in >>>> libcos-plugin.so[7f533d81f000+b000] >>>> >>>>> [11832.727989] ns-slapd[22606]: >>>> segfault at 0 ip 00007f6231eb951c sp >>>> >>>>> 00007f623bf2ba70 error 4 in >>>> libcos-plugin.so[7f6231eb6000+b00 >>>> >>>> >>>> >>>> Okay, this one is serious. The LDAP >>>> server crashed. >>>> >>>> >>>> >>>> 1. Make sure all your packages are >>>> up-to-date. >>>> >>>> >>>> >>>> Please see >>>> >>>> >>>> http://directory.fedoraproject.org/docs/389ds/FAQ/faq.html#d >>>> >>>> ebugging-crashes >>>> >>>> for further instructions how to debug >>>> this. >>>> >>>> >>>> >>>> Petr^2 Spacek >>>> >>>> >>>> >>>>> >>>> >>>>> and in /var/log/dirsrv/example-com/errors >>>> >>>>> >>>> >>>>> [23/Aug/2016:12:49:36 +0000] >>>> DSRetroclPlugin - delete_changerecord: >>>> >>>> could >>>> >>>>> not delete change record 3291138 (rc: 32) >>>> >>>>> [23/Aug/2016:12:49:36 +0000] >>>> DSRetroclPlugin - delete_changerecord: >>>> >>>> could >>>> >>>>> not delete change record 3291139 (rc: 32) >>>> >>>>> [23/Aug/2016:12:49:36 +0000] >>>> DSRetroclPlugin - delete_changerecord: >>>> >>>> could >>>> >>>>> not delete change record 3291140 (rc: 32) >>>> >>>>> [23/Aug/2016:12:49:36 +0000] >>>> DSRetroclPlugin - delete_changerecord: >>>> >>>> could >>>> >>>>> not delete change record 3291141 (rc: 32) >>>> >>>>> [23/Aug/2016:12:49:36 +0000] >>>> DSRetroclPlugin - delete_changerecord: >>>> >>>> could >>>> >>>>> not delete change record 3291142 (rc: 32) >>>> >>>>> [23/Aug/2016:12:49:36 +0000] >>>> DSRetroclPlugin - delete_changerecord: >>>> >>>> could >>>> >>>>> not delete change record 3291143 (rc: 32) >>>> >>>>> [23/Aug/2016:12:49:36 +0000] >>>> DSRetroclPlugin - delete_changerecord: >>>> >>>> could >>>> >>>>> not delete change record 3291144 (rc: 32) >>>> >>>>> [23/Aug/2016:12:49:36 +0000] >>>> DSRetroclPlugin - delete_changerecord: >>>> >>>> could >>>> >>>>> not delete change record 3291145 (rc: 32) >>>> >>>>> [23/Aug/2016:12:49:50 +0000] - Retry >>>> count exceeded in delete >>>> >>>>> [23/Aug/2016:12:49:50 +0000] >>>> DSRetroclPlugin - delete_changerecord: >>>> >>>> could >>>> >>>>> not delete change record 3292734 (rc: 51) >>>> >>>>> >>>> >>>>> >>>> >>>>> Can i do something about this >>>> error.. I treid to restart ipa a couple >>>> >>>> of >>>> >>>>> time but that did not help >>>> >>>>> >>>> >>>>> Thanks >>>> >>>>> Rakesh >>>> >>>>> >>>> >>>>> On Mon, Aug 22, 2016 at 2:27 PM, Petr >>>> Spacek >>> > >>>> >>>> wrote: >>>> >>>>> >>>> >>>>>> On 19.8.2016 19:32, Rakesh >>>> Rajasekharan wrote: >>>> >>>>>>> I am running my set up on AWS >>>> cloud, and entropy is low at around >>>> >>>> 180 . >>>> >>>>>>> >>>> >>>>>>> I plan to increase it bu installing >>>> haveged . But, would low entropy >>>> >>>> by >>>> >>>>>> any >>>> >>>>>>> chance cause this issue of >>>> intermittent hang . >>>> >>>>>>> Also, the hang is mostly observed >>>> when registering around 20 clients >>>> >>>>>>> together >>>> >>>>>> >>>> >>>>>> Possibly, I'm not sure. If you want >>>> to dig into this, I would do this: >>>> >>>>>> 1. look what process hangs on client >>>> (using pstree command or so) >>>> >>>>>> $ pstree >>>> >>>>>> >>>> >>>>>> 2. look to what server and port is >>>> the hanging client connected to >>>> >>>>>> $ lsof -p >>>> >>>>>> >>>> >>>>>> 3. jump to server and see what >>>> process is bound to the target port >>>> >>>>>> $ netstat -pn >>>> >>>>>> >>>> >>>>>> 4. see where the process if hanging >>>> >>>>>> $ strace -p >>>> >>>>>> >>>> >>>>>> I hope it helps. >>>> >>>>>> >>>> >>>>>> Petr^2 Spacek >>>> >>>>>> >>>> >>>>>>> On Fri, Aug 19, 2016 at 7:24 PM, >>>> Rakesh Rajasekharan < >>>> >>>>>>> rakesh.rajasekharan at gmail.com >>>> > wrote: >>>> >>>>>>> >>>> >>>>>>>> yes there seems to be something >>>> thats worrying.. I have faced this >>>> >>>> today >>>> >>>>>>>> as well. >>>> >>>>>>>> There are few hosts around 280 odd >>>> left and when i try adding them >>>> >>>> to >>>> >>>>>> IPA >>>> >>>>>>>> , the slowness begins.. >>>> >>>>>>>> >>>> >>>>>>>> all the ipa commands like ipa >>>> user-find.. etc becomes very slow in >>>> >>>>>>>> responding. >>>> >>>>>>>> >>>> >>>>>>>> the SYNC_RECV are not many though >>>> just around 80-90 and today that >>>> >>>> was >>>> >>>>>>>> around 20 only >>>> >>>>>>>> >>>> >>>>>>>> >>>> >>>>>>>> I have for now increased >>>> tcp_max_syn_backlog to 5000. >>>> >>>>>>>> For now the slowness seems to have >>>> gone.. but I will do a try >>>> >>>> adding the >>>> >>>>>>>> clients again tomorrow and see how >>>> it goes >>>> >>>>>>>> >>>> >>>>>>>> Thanks >>>> >>>>>>>> Rakesh >>>> >>>>>>>> >>>> >>>>>>>> The issues >>>> >>>>>>>> >>>> >>>>>>>> On Fri, Aug 19, 2016 at 12:58 PM, >>>> Petr Spacek >>> > >>>> >>>>>> wrote: >>>> >>>>>>>> >>>> >>>>>>>>> On 18.8.2016 17:23, Rakesh >>>> Rajasekharan wrote: >>>> >>>>>>>>>> Hi >>>> >>>>>>>>>> >>>> >>>>>>>>>> I am migrating to freeipa from >>>> openldap and have around 4000 >>>> >>>> clients >>>> >>>>>>>>>> >>>> >>>>>>>>>> I had openned a another thread >>>> on that, but chose to start a new >>>> >>>> one >>>> >>>>>>>>> here >>>> >>>>>>>>>> as its a separate issue >>>> >>>>>>>>>> >>>> >>>>>>>>>> I was able to change the >>>> nssslapd-maxdescriptors adding an ldif >>>> >>>> file >>>> >>>>>>>>>> >>>> >>>>>>>>>> cat nsslapd-modify.ldif >>>> >>>>>>>>>> dn: cn=config >>>> >>>>>>>>>> changetype: modify >>>> >>>>>>>>>> replace: nsslapd-maxdescriptors >>>> >>>>>>>>>> nsslapd-maxdescriptors: 17000 >>>> >>>>>>>>>> >>>> >>>>>>>>>> and running the ldapmodify command >>>> >>>>>>>>>> >>>> >>>>>>>>>> I have now started moving >>>> clients running an openldap to Freeipa >>>> >>>> and >>>> >>>>>>>>> have >>>> >>>>>>>>>> today moved close to 2000 clients >>>> >>>>>>>>>> >>>> >>>>>>>>>> However, I have noticed that IPA >>>> hangs intermittently. >>>> >>>>>>>>>> >>>> >>>>>>>>>> running a kinit admin returns >>>> the below error >>>> >>>>>>>>>> kinit: Generic error (see >>>> e-text) while getting initial >>>> >>>> credentials >>>> >>>>>>>>>> >>>> >>>>>>>>>> from the /var/log/messages, I >>>> see this entry >>>> >>>>>>>>>> >>>> >>>>>>>>>> prod-ipa-master-int kernel: >>>> [104090.315801] TCP: >>>> >>>> request_sock_TCP: >>>> >>>>>>>>>> Possible SYN flooding on port >>>> 88. Sending cookies. Check SNMP >>>> >>>>>> counters. >>>> >>>>>>>>> >>>> >>>>>>>>> I would be worried about this >>>> message. Maybe kernel/firewall is >>>> >>>> doing >>>> >>>>>>>>> something fishy behind your back >>>> and blocking some connections or >>>> >>>> so. >>>> >>>>>>>>> >>>> >>>>>>>>> Petr^2 Spacek >>>> >>>>>>>>> >>>> >>>>>>>>> >>>> >>>>>>>>>> Aug 18 13:00:01 >>>> prod-ipa-master-int systemd[1]: Started Session >>>> >>>> 4885 >>>> >>>>>> of >>>> >>>>>>>>>> user root. >>>> >>>>>>>>>> Aug 18 13:00:01 >>>> prod-ipa-master-int systemd[1]: Starting >>>> Session >>>> >>>> 4885 >>>> >>>>>> of >>>> >>>>>>>>>> user root. >>>> >>>>>>>>>> Aug 18 13:01:01 >>>> prod-ipa-master-int systemd[1]: Started Session >>>> >>>> 4886 >>>> >>>>>> of >>>> >>>>>>>>>> user root. >>>> >>>>>>>>>> Aug 18 13:01:01 >>>> prod-ipa-master-int systemd[1]: Starting >>>> Session >>>> >>>> 4886 >>>> >>>>>> of >>>> >>>>>>>>>> user root. >>>> >>>>>>>>>> Aug 18 13:02:40 >>>> prod-ipa-master-int python[28984]: >>>> ansible-command >>>> >>>>>>>>> Invoked >>>> >>>>>>>>>> with creates=None >>>> executable=None shell=True args= removes=None >>>> >>>>>>>>> warn=True >>>> >>>>>>>>>> chdir=None >>>> >>>>>>>>>> Aug 18 13:04:37 >>>> prod-ipa-master-int sssd_be: GSSAPI Error: >>>> >>>> Unspecified >>>> >>>>>>>>> GSS >>>> >>>>>>>>>> failure. Minor code may provide >>>> more information (KDC returned >>>> >>>> error >>>> >>>>>>>>>> string: PROCESS_TGS) >>>> >>>>>>>>>> >>>> >>>>>>>>>> Could it be possible that its >>>> due to the initial load of adding >>>> >>>> the >>>> >>>>>>>>> clients >>>> >>>>>>>>>> or is there something else that >>>> I need to take care of. >>>> >>>> >>>> >>>> >>> >>> >>> >> >> >> >> > > > -- > Manage your subscription for the Freeipa-users mailing list: > https://www.redhat.com/mailman/listinfo/freeipa-users > > Go to http://freeipa.org for more info on the project > > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From rakesh.rajasekharan at gmail.com Mon Sep 5 10:05:53 2016 From: rakesh.rajasekharan at gmail.com (Rakesh Rajasekharan) Date: Mon, 5 Sep 2016 15:35:53 +0530 Subject: [Freeipa-users] Freeipa 4.2.0 hangs intermittently In-Reply-To: <57CD1FDC.2010109@redhat.com> References: <57BEB0AB.2090506@redhat.com> <57C44AC6.1030402@redhat.com> <57CD1FDC.2010109@redhat.com> Message-ID: Hi Thierry, I was getting the hang issue while running ipa-client-install simultaneously on few clients.. However, today, I am not able to replicate that. I could not get a gdb . But i will try getting that the next time I face this issue. The CPU does not stay high.. it just momentarily touches a high value and then drops down to around 2-7% One question I have is , is it ok to set it nsslapd-threadnumber to a very high value . I have around 4000 clients and with nsslapd-maxthreadsperconn set to 5..So, can I set nsslapd-threadnumber to around 25000. Thanks On Mon, Sep 5, 2016 at 1:03 PM, thierry bordaz wrote: > > Hi Rakesh, > > Were you able to get a pstack or full stack with gdb ( > http://www.port389.org/docs/389ds/FAQ/faq.html#debugging-crashes) when > the server hangs ? > > If it happens with 500 threads as well as with 30, using 30 threads is a > better choice to debug this issue. > I will try to reproduce using 150 parallel 'ipa user-find p-testipa' > commands > > Something I am unsure is if the CPU consumption stays high (you mentioned > 340% CPU usage) as long as the hang happens or if after a suddent shot up > to 340% (that marks the beginning of the hang) it drops and stay hanging ? > > thanks > thierry > > On 09/04/2016 08:40 PM, Rakesh Rajasekharan wrote: > > starce on the slapd process actually had this in the output.. > FUTEX_WAIT_PRIVATE > > and checking for the number of threads slapd had.. there were 5015 threads > > ps -efL|grep slapd|wc -l > 5015 > > strace on most of the threads gave this output > > strace -p 67411 > Process 67411 attached > futex(0x7f3f0226b41c, FUTEX_WAIT_PRIVATE, 1, NULL) = -1 EAGAIN (Resource > temporarily unavailable) > futex(0x7f3f0226b41c, FUTEX_WAIT_PRIVATE, 2, NULL^CProcess 67411 detached > > > > > > On Sun, Sep 4, 2016 at 5:34 PM, Rakesh Rajasekharan < > rakesh.rajasekharan at gmail.com> wrote: > >> I have again got the issue of IPA hanging.. The issue came up when i >> tried to run ipa-client-isntall on 142 clients simultaneously >> >> >> None of the IPA commands are responding, and I see this error >> >> ipa user-find p-testipa >> ipa: ERROR: Insufficient access: SASL(-1): generic failure: GSSAPI Error: >> Unspecified GSS failure. Minor code may provide more information (KDC >> returned error string: PROCESS_TGS) >> >> KRB5_TRACE=/dev/stdout kinit admin >> [41178] 1472984115.233214: Getting initial credentials for admin at XYZ.COM >> [41178] 1472984115.235257: Sending request (167 bytes) to XYZ.COM >> [41178] 1472984115.235419: Initiating TCP connection to stream >> 10.1.3.36:88 >> [41178] 1472984115.235685: Sending TCP request to stream 10.1.3.36:88 >> [41178] 1472984120.238914: Received answer (174 bytes) from stream >> 10.1.3.36:88 >> [41178] 1472984120.238925: Terminating TCP connection to stream >> 10.1.3.36:88 >> [41178] 1472984120.238993: Response was from master KDC >> [41 >> >> >> Running an ldapsearch to see the db.. does not give any results and just >> hangs there >> >> ldapsearch -x -D 'cn=Directory Manager' -W -s one -b >> 'cn=kerberos,dc=xyz,dc=com' >> Enter LDAP Password: >> >> even an ldapsearch -x does not respond >> At this point, am sure that slapd is the one causing issues >> >> Running an strace against the hung slapd itself seems to get stuck does >> not proceed after saying "attaching to process" >> >> From some others posts I read Thierry suggesting to increase the >> nsslapd-threadnumber value >> >> It was set to 30, I think that might be too low. >> >> I have raised it to 500 >> >> Now after restarting the service .. ldapsearch starts responding. >> But running the test to add a sudden high number of clients again left >> ns-slapd to hung state >> >> When i attempted adding the clients.. the ns-slapd cpu usage shot up to >> 340% and after that ns-slapd stopped responding >> >> So now, atleast I know what might be causing the issue and I can now >> easily reproduce it. >> >> Is there a way I can make ns-slapd handle a sudden bump in incoming >> request for ipa-client-install >> >> Thanks >> Rakesh >> >> >> >> >> >> >> On Mon, Aug 29, 2016 at 11:18 PM, Rich Megginson < >> rmeggins at redhat.com> wrote: >> >>> On 08/29/2016 10:53 AM, Rakesh Rajasekharan wrote: >>> >>> Hi Thierry, >>> >>> My machine has 30GB RAM ..and 389-ds version is 1.3.4 >>> >>> ldapsearch shows the values for nsslapd-cachememsize updated to 200MB. >>> >>> ldapsearch -LLL -o ldif-wrap=no -D "cn=directory manager" -w >>> 'mypassword' -b 'cn=userRoot,cn=ldbm database,cn=plugins,cn=config'|grep >>> nsslapd-cachememsize >>> nsslapd-cachememsize: 209715200 >>> >>> >>> So, it seems to have updated though seeing that warning(WARNING: ipaca: >>> entry cache size 10485760B is less than db size 11599872B) in the log >>> confuses me a bit. >>> >>> Thers one more entry that I found from the ldapsearch to be bit low >>> >>> nsslapd-dncachememsize: 10485760 >>> maxdncachesize: 10485760 >>> >>> Should I update these as well to a higher value >>> >>> At the time when the issue happened, the memory usage as well as the >>> overall load of the system was very low . >>> I will try reproducing the issue atleast in my QA env..probably by >>> trying to mock simultaneous parallel logins to a large number of hosts >>> >>> >>> To monitor your cache sizes, please use the dbmon.sh tool provided with >>> your distro. If that is not available with your particular distro, see >>> https://github.com/richm/scripts/wiki/dbmon.sh >>> >>> >>> >>> >>> thanks >>> Rakesh >>> >>> >>> >>> >>> On Mon, Aug 29, 2016 at 8:16 PM, thierry bordaz < >>> tbordaz at redhat.com> wrote: >>> >>>> Hi Rakesh, >>>> >>>> Those tuning may depend on the memory available on your machine. >>>> nsslapd-cachememsize allows the entry cache to consume up to 200Mb but >>>> its memory footprint is known to go above. >>>> 200Mb both looks pretty good to me. How large is your machine ? What is >>>> your version of 389-ds ? >>>> >>>> Those warnings do not change your settings. It just raise that entry >>>> cache of 'ipaca' and 'retrocl' are small but it is fine. The size of the >>>> entry cache is important mostly in userRoot. >>>> You may double check the actual values, after restart, with ldapsearch >>>> on 'cn=userRoot,cn=ldbm database,cn=plugins,cn=config' and >>>> 'cn=config,cn=ldbm database,cn=plugins,cn=config'. >>>> >>>> A step is to know what will be response time of DS to know if it is >>>> responsible of the hang or not. >>>> The logs and possibly pstack during those intermittent hangs will help >>>> to determine that. >>>> >>>> regards >>>> thierry >>>> >>>> >>>> >>>> >>>> >>>> On 08/29/2016 04:25 PM, Rakesh Rajasekharan wrote: >>>> >>>> I tried increasing the nsslapd-dbcachesize and nsslapd-cachememsize in >>>> my QA envs to 200MB. >>>> >>>> However, in my log files, I still see this message >>>> [29/Aug/2016:04:34:37 +0000] - WARNING: ipaca: entry cache size >>>> 10485760B is less than db size 11599872B; We recommend to increase the >>>> entry cache size nsslapd-cachememsize. >>>> [29/Aug/2016:04:34:37 +0000] - WARNING: changelog: entry cache size >>>> 2097152B is less than db size 441647104B; We recommend to increase the >>>> entry cache size nsslapd-cachememsize. >>>> >>>> these are my ldif files that i used to modify the values >>>> modify entry cache size >>>> cat modify-cache-mem-size.ldif >>>> dn: cn=userRoot,cn=ldbm database,cn=plugins,cn=config >>>> changetype: modify >>>> replace: nsslapd-cachememsize >>>> nsslapd-cachememsize: 209715200 >>>> >>>> modify db cache size >>>> cat modfy-db-cache-size.ldif >>>> dn: cn=config,cn=ldbm database,cn=plugins,cn=config >>>> changetype: modify >>>> replace: nsslapd-dbcachesize >>>> nsslapd-dbcachesize: 209715200 >>>> >>>> After modifying , i restarted IPA services >>>> >>>> Is there anything else that I need to take care of as the logs suggest >>>> its still not getting the updated values >>>> >>>> Thanks >>>> Rakesh >>>> >>>> On Mon, Aug 29, 2016 at 6:07 PM, Rakesh Rajasekharan < >>>> rakesh.rajasekharan at gmail.com> wrote: >>>> >>>>> Hi Thierry, >>>>> >>>>> Coz of the issues we had to revert back to earlier running openldap in >>>>> production. >>>>> >>>>> I have now done a few TCP related changes in sysctl.conf and have also >>>>> increased the nsslapd-dbcachesize and nsslapd-cachememsize to 200MB >>>>> >>>>> I will again start migrating hosts back to IPA and see if I face the >>>>> earlier issue. >>>>> >>>>> I will update back once I have something >>>>> >>>>> >>>>> Thanks, >>>>> Rakesh >>>>> >>>>> >>>>> >>>>> On Thu, Aug 25, 2016 at 2:17 PM, thierry bordaz < >>>>> tbordaz at redhat.com> wrote: >>>>> >>>>>> >>>>>> >>>>>> On 08/25/2016 10:15 AM, Rakesh Rajasekharan wrote: >>>>>> >>>>>> All of the troubleshooting seems fine. >>>>>> >>>>>> >>>>>> However, Running libconv.pl gives me this output >>>>>> >>>>>> ----- Recommendations ----- >>>>>> >>>>>> 1. You have unindexed components, this can be caused from a search >>>>>> on an unindexed attribute, or your returned results exceeded the >>>>>> allidsthreshold. Unindexed components are not recommended. To refuse >>>>>> unindexed searches, switch 'nsslapd-require-index' to 'on' under your >>>>>> database entry (e.g. cn=UserRoot,cn=ldbm database,cn=plugins,cn=config) >>>>>> . >>>>>> >>>>>> 2. You have a significant difference between binds and unbinds. >>>>>> You may want to investigate this difference. >>>>>> >>>>>> >>>>>> I feel, this could be a pointer to things going slow.. and IPA >>>>>> hanging. I think i now have something that I can try and nail down this >>>>>> issue. >>>>>> >>>>>> On a sidenote, I was earlier running openldap and migrated over to >>>>>> Freeipa, >>>>>> >>>>>> Thanks >>>>>> Rakesh >>>>>> >>>>>> >>>>>> >>>>>> On Wed, Aug 24, 2016 at 12:38 PM, Petr Spacek < >>>>>> pspacek at redhat.com> wrote: >>>>>> >>>>>>> On 23.8.2016 18:44, Rakesh Rajasekharan wrote: >>>>>>> > I think thers something seriously wrong with my system >>>>>>> > >>>>>>> > not able to run any IPA commands >>>>>>> > >>>>>>> > klist >>>>>>> > Ticket cache: KEYRING:persistent:0:0 >>>>>>> > Default principal: admin at XYZ.COM >>>>>>> > >>>>>>> > Valid starting Expires Service principal >>>>>>> > 2016-08-23T16:26:36 2016-08-24T16:26:22 krbtgt/ >>>>>>> XYZ.COM at XYZ.COM >>>>>>> > >>>>>>> > >>>>>>> > [root at prod-ipa-master-1a :~] ipactl status >>>>>>> > Directory Service: RUNNING >>>>>>> > krb5kdc Service: RUNNING >>>>>>> > kadmin Service: RUNNING >>>>>>> > ipa_memcached Service: RUNNING >>>>>>> > httpd Service: RUNNING >>>>>>> > pki-tomcatd Service: RUNNING >>>>>>> > ipa-otpd Service: RUNNING >>>>>>> > ipa: INFO: The ipactl command was successful >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > [root at prod-ipa-master :~] ipa user-find p-testuser >>>>>>> > ipa: ERROR: Kerberos error: ('Unspecified GSS failure. Minor code >>>>>>> may >>>>>>> > provide more information', 851968)/("Cannot contact any KDC for >>>>>>> realm ' >>>>>>> > XYZ.COM'", -1765328228) >>>>>>> >>>>>> >>>>>> Hi Rakesh, >>>>>> >>>>>> Having a reproducible test case would you rerun the command above. >>>>>> During its processing you may monitor DS process load (top). If it is >>>>>> high, you may get some pstacks of it. >>>>>> Also would you attach the part of DS access logs taken during the >>>>>> command. >>>>>> >>>>>> regards >>>>>> thierry >>>>>> >>>>>> > >>>>>>> >>>>>>> This is weird because the server seems to be up. >>>>>>> >>>>>>> Please follow >>>>>>> >>>>>>> >>>>>>> http://www.freeipa.org/page/Troubleshooting#Authentication.2 >>>>>>> FKerberos >>>>>>> >>>>>>> Petr^2 Spacek >>>>>>> >>>>>>> > >>>>>>> > >>>>>>> > Thanks >>>>>>> > >>>>>>> > Rakesh >>>>>>> > >>>>>>> > On Tue, Aug 23, 2016 at 10:01 PM, Rakesh Rajasekharan < >>>>>>> > rakesh.rajasekharan at gmail.com> >>>>>>> wrote: >>>>>>> > >>>>>>> >> i changed the loggin level to 4 . Modifying >>>>>>> nsslapd-accesslog-level >>>>>>> >> >>>>>>> >> But, the hang is still there. though I dont see the sigfault now >>>>>>> >> >>>>>>> >> >>>>>>> >> >>>>>>> >> >>>>>>> >> On Tue, Aug 23, 2016 at 9:02 PM, Rakesh Rajasekharan < >>>>>>> >> rakesh.rajasekharan at gmail.com> >>>>>>> wrote: >>>>>>> >> >>>>>>> >>> My disk was getting filled too fast >>>>>>> >>> >>>>>>> >>> logs under /var/log/dirsrv was coming around 5 gb quickly >>>>>>> filling up >>>>>>> >>> >>>>>>> >>> Is there a way to make the logging less verbose >>>>>>> >>> >>>>>>> >>> >>>>>>> >>> >>>>>>> >>> On Tue, Aug 23, 2016 at 6:41 PM, Petr Spacek < >>>>>>> pspacek at redhat.com> wrote: >>>>>>> >>> >>>>>>> >>>> On 23.8.2016 15:07, Rakesh Rajasekharan wrote: >>>>>>> >>>>> I was able to fix that may be temporarily... when i checked the >>>>>>> >>>> network.. >>>>>>> >>>>> there was another process that was running and consuming a lot >>>>>>> of >>>>>>> >>>> network ( >>>>>>> >>>>> i have no idea who did that. I need to seriously start >>>>>>> restricting >>>>>>> >>>> people >>>>>>> >>>>> access to this machine ) >>>>>>> >>>>> >>>>>>> >>>>> after killing that perfomance improved drastically >>>>>>> >>>>> >>>>>>> >>>>> But now, suddenly I started experiencing the same hang. >>>>>>> >>>>> >>>>>>> >>>>> This time , I gert the following error when checked dmesg >>>>>>> >>>>> >>>>>>> >>>>> [ 301.236976] ns-slapd[3124]: segfault at 0 ip >>>>>>> 00007f1de416951c sp >>>>>>> >>>>> 00007f1dee1dba70 error 4 in libcos-plugin.so[7f1de4166000+ >>>>>>> b000] >>>>>>> >>>>> [ 1116.248431] TCP: request_sock_TCP: Possible SYN flooding on >>>>>>> port 88. >>>>>>> >>>>> Sending cookies. Check SNMP counters. >>>>>>> >>>>> [11831.397037] ns-slapd[22550]: segfault at 0 ip >>>>>>> 00007f533d82251c sp >>>>>>> >>>>> 00007f5347894a70 error 4 in libcos-plugin.so[7f533d81f000+ >>>>>>> b000] >>>>>>> >>>>> [11832.727989] ns-slapd[22606]: segfault at 0 ip >>>>>>> 00007f6231eb951c sp >>>>>>> >>>>> 00007f623bf2ba70 error 4 in libcos-plugin.so[7f6231eb6000+b00 >>>>>>> >>>> >>>>>>> >>>> Okay, this one is serious. The LDAP server crashed. >>>>>>> >>>> >>>>>>> >>>> 1. Make sure all your packages are up-to-date. >>>>>>> >>>> >>>>>>> >>>> Please see >>>>>>> >>>> http://directory.fedoraproject >>>>>>> .org/docs/389ds/FAQ/faq.html#d >>>>>>> >>>> ebugging-crashes >>>>>>> >>>> for further instructions how to debug this. >>>>>>> >>>> >>>>>>> >>>> Petr^2 Spacek >>>>>>> >>>> >>>>>>> >>>>> >>>>>>> >>>>> and in /var/log/dirsrv/example-com/errors >>>>>>> >>>>> >>>>>>> >>>>> [23/Aug/2016:12:49:36 +0000] DSRetroclPlugin - >>>>>>> delete_changerecord: >>>>>>> >>>> could >>>>>>> >>>>> not delete change record 3291138 (rc: 32) >>>>>>> >>>>> [23/Aug/2016:12:49:36 +0000] DSRetroclPlugin - >>>>>>> delete_changerecord: >>>>>>> >>>> could >>>>>>> >>>>> not delete change record 3291139 (rc: 32) >>>>>>> >>>>> [23/Aug/2016:12:49:36 +0000] DSRetroclPlugin - >>>>>>> delete_changerecord: >>>>>>> >>>> could >>>>>>> >>>>> not delete change record 3291140 (rc: 32) >>>>>>> >>>>> [23/Aug/2016:12:49:36 +0000] DSRetroclPlugin - >>>>>>> delete_changerecord: >>>>>>> >>>> could >>>>>>> >>>>> not delete change record 3291141 (rc: 32) >>>>>>> >>>>> [23/Aug/2016:12:49:36 +0000] DSRetroclPlugin - >>>>>>> delete_changerecord: >>>>>>> >>>> could >>>>>>> >>>>> not delete change record 3291142 (rc: 32) >>>>>>> >>>>> [23/Aug/2016:12:49:36 +0000] DSRetroclPlugin - >>>>>>> delete_changerecord: >>>>>>> >>>> could >>>>>>> >>>>> not delete change record 3291143 (rc: 32) >>>>>>> >>>>> [23/Aug/2016:12:49:36 +0000] DSRetroclPlugin - >>>>>>> delete_changerecord: >>>>>>> >>>> could >>>>>>> >>>>> not delete change record 3291144 (rc: 32) >>>>>>> >>>>> [23/Aug/2016:12:49:36 +0000] DSRetroclPlugin - >>>>>>> delete_changerecord: >>>>>>> >>>> could >>>>>>> >>>>> not delete change record 3291145 (rc: 32) >>>>>>> >>>>> [23/Aug/2016:12:49:50 +0000] - Retry count exceeded in delete >>>>>>> >>>>> [23/Aug/2016:12:49:50 +0000] DSRetroclPlugin - >>>>>>> delete_changerecord: >>>>>>> >>>> could >>>>>>> >>>>> not delete change record 3292734 (rc: 51) >>>>>>> >>>>> >>>>>>> >>>>> >>>>>>> >>>>> Can i do something about this error.. I treid to restart ipa >>>>>>> a couple >>>>>>> >>>> of >>>>>>> >>>>> time but that did not help >>>>>>> >>>>> >>>>>>> >>>>> Thanks >>>>>>> >>>>> Rakesh >>>>>>> >>>>> >>>>>>> >>>>> On Mon, Aug 22, 2016 at 2:27 PM, Petr Spacek < >>>>>>> pspacek at redhat.com> >>>>>>> >>>> wrote: >>>>>>> >>>>> >>>>>>> >>>>>> On 19.8.2016 19:32, Rakesh Rajasekharan wrote: >>>>>>> >>>>>>> I am running my set up on AWS cloud, and entropy is low at >>>>>>> around >>>>>>> >>>> 180 . >>>>>>> >>>>>>> >>>>>>> >>>>>>> I plan to increase it bu installing haveged . But, would low >>>>>>> entropy >>>>>>> >>>> by >>>>>>> >>>>>> any >>>>>>> >>>>>>> chance cause this issue of intermittent hang . >>>>>>> >>>>>>> Also, the hang is mostly observed when registering around 20 >>>>>>> clients >>>>>>> >>>>>>> together >>>>>>> >>>>>> >>>>>>> >>>>>> Possibly, I'm not sure. If you want to dig into this, I would >>>>>>> do this: >>>>>>> >>>>>> 1. look what process hangs on client (using pstree command or >>>>>>> so) >>>>>>> >>>>>> $ pstree >>>>>>> >>>>>> >>>>>>> >>>>>> 2. look to what server and port is the hanging client >>>>>>> connected to >>>>>>> >>>>>> $ lsof -p >>>>>>> >>>>>> >>>>>>> >>>>>> 3. jump to server and see what process is bound to the target >>>>>>> port >>>>>>> >>>>>> $ netstat -pn >>>>>>> >>>>>> >>>>>>> >>>>>> 4. see where the process if hanging >>>>>>> >>>>>> $ strace -p >>>>>>> >>>>>> >>>>>>> >>>>>> I hope it helps. >>>>>>> >>>>>> >>>>>>> >>>>>> Petr^2 Spacek >>>>>>> >>>>>> >>>>>>> >>>>>>> On Fri, Aug 19, 2016 at 7:24 PM, Rakesh Rajasekharan < >>>>>>> >>>>>>> rakesh.rajasekharan at gmail.com> >>>>>>> wrote: >>>>>>> >>>>>>> >>>>>>> >>>>>>>> yes there seems to be something thats worrying.. I have >>>>>>> faced this >>>>>>> >>>> today >>>>>>> >>>>>>>> as well. >>>>>>> >>>>>>>> There are few hosts around 280 odd left and when i try >>>>>>> adding them >>>>>>> >>>> to >>>>>>> >>>>>> IPA >>>>>>> >>>>>>>> , the slowness begins.. >>>>>>> >>>>>>>> >>>>>>> >>>>>>>> all the ipa commands like ipa user-find.. etc becomes very >>>>>>> slow in >>>>>>> >>>>>>>> responding. >>>>>>> >>>>>>>> >>>>>>> >>>>>>>> the SYNC_RECV are not many though just around 80-90 and >>>>>>> today that >>>>>>> >>>> was >>>>>>> >>>>>>>> around 20 only >>>>>>> >>>>>>>> >>>>>>> >>>>>>>> >>>>>>> >>>>>>>> I have for now increased tcp_max_syn_backlog to 5000. >>>>>>> >>>>>>>> For now the slowness seems to have gone.. but I will do a >>>>>>> try >>>>>>> >>>> adding the >>>>>>> >>>>>>>> clients again tomorrow and see how it goes >>>>>>> >>>>>>>> >>>>>>> >>>>>>>> Thanks >>>>>>> >>>>>>>> Rakesh >>>>>>> >>>>>>>> >>>>>>> >>>>>>>> The issues >>>>>>> >>>>>>>> >>>>>>> >>>>>>>> On Fri, Aug 19, 2016 at 12:58 PM, Petr Spacek < >>>>>>> pspacek at redhat.com> >>>>>>> >>>>>> wrote: >>>>>>> >>>>>>>> >>>>>>> >>>>>>>>> On 18.8.2016 17:23, Rakesh Rajasekharan wrote: >>>>>>> >>>>>>>>>> Hi >>>>>>> >>>>>>>>>> >>>>>>> >>>>>>>>>> I am migrating to freeipa from openldap and have around >>>>>>> 4000 >>>>>>> >>>> clients >>>>>>> >>>>>>>>>> >>>>>>> >>>>>>>>>> I had openned a another thread on that, but chose to >>>>>>> start a new >>>>>>> >>>> one >>>>>>> >>>>>>>>> here >>>>>>> >>>>>>>>>> as its a separate issue >>>>>>> >>>>>>>>>> >>>>>>> >>>>>>>>>> I was able to change the nssslapd-maxdescriptors adding >>>>>>> an ldif >>>>>>> >>>> file >>>>>>> >>>>>>>>>> >>>>>>> >>>>>>>>>> cat nsslapd-modify.ldif >>>>>>> >>>>>>>>>> dn: cn=config >>>>>>> >>>>>>>>>> changetype: modify >>>>>>> >>>>>>>>>> replace: nsslapd-maxdescriptors >>>>>>> >>>>>>>>>> nsslapd-maxdescriptors: 17000 >>>>>>> >>>>>>>>>> >>>>>>> >>>>>>>>>> and running the ldapmodify command >>>>>>> >>>>>>>>>> >>>>>>> >>>>>>>>>> I have now started moving clients running an openldap to >>>>>>> Freeipa >>>>>>> >>>> and >>>>>>> >>>>>>>>> have >>>>>>> >>>>>>>>>> today moved close to 2000 clients >>>>>>> >>>>>>>>>> >>>>>>> >>>>>>>>>> However, I have noticed that IPA hangs intermittently. >>>>>>> >>>>>>>>>> >>>>>>> >>>>>>>>>> running a kinit admin returns the below error >>>>>>> >>>>>>>>>> kinit: Generic error (see e-text) while getting initial >>>>>>> >>>> credentials >>>>>>> >>>>>>>>>> >>>>>>> >>>>>>>>>> from the /var/log/messages, I see this entry >>>>>>> >>>>>>>>>> >>>>>>> >>>>>>>>>> prod-ipa-master-int kernel: [104090.315801] TCP: >>>>>>> >>>> request_sock_TCP: >>>>>>> >>>>>>>>>> Possible SYN flooding on port 88. Sending cookies. Check >>>>>>> SNMP >>>>>>> >>>>>> counters. >>>>>>> >>>>>>>>> >>>>>>> >>>>>>>>> I would be worried about this message. Maybe >>>>>>> kernel/firewall is >>>>>>> >>>> doing >>>>>>> >>>>>>>>> something fishy behind your back and blocking some >>>>>>> connections or >>>>>>> >>>> so. >>>>>>> >>>>>>>>> >>>>>>> >>>>>>>>> Petr^2 Spacek >>>>>>> >>>>>>>>> >>>>>>> >>>>>>>>> >>>>>>> >>>>>>>>>> Aug 18 13:00:01 prod-ipa-master-int systemd[1]: Started >>>>>>> Session >>>>>>> >>>> 4885 >>>>>>> >>>>>> of >>>>>>> >>>>>>>>>> user root. >>>>>>> >>>>>>>>>> Aug 18 13:00:01 prod-ipa-master-int systemd[1]: Starting >>>>>>> Session >>>>>>> >>>> 4885 >>>>>>> >>>>>> of >>>>>>> >>>>>>>>>> user root. >>>>>>> >>>>>>>>>> Aug 18 13:01:01 prod-ipa-master-int systemd[1]: Started >>>>>>> Session >>>>>>> >>>> 4886 >>>>>>> >>>>>> of >>>>>>> >>>>>>>>>> user root. >>>>>>> >>>>>>>>>> Aug 18 13:01:01 prod-ipa-master-int systemd[1]: Starting >>>>>>> Session >>>>>>> >>>> 4886 >>>>>>> >>>>>> of >>>>>>> >>>>>>>>>> user root. >>>>>>> >>>>>>>>>> Aug 18 13:02:40 prod-ipa-master-int python[28984]: >>>>>>> ansible-command >>>>>>> >>>>>>>>> Invoked >>>>>>> >>>>>>>>>> with creates=None executable=None shell=True args= >>>>>>> removes=None >>>>>>> >>>>>>>>> warn=True >>>>>>> >>>>>>>>>> chdir=None >>>>>>> >>>>>>>>>> Aug 18 13:04:37 prod-ipa-master-int sssd_be: GSSAPI Error: >>>>>>> >>>> Unspecified >>>>>>> >>>>>>>>> GSS >>>>>>> >>>>>>>>>> failure. Minor code may provide more information (KDC >>>>>>> returned >>>>>>> >>>> error >>>>>>> >>>>>>>>>> string: PROCESS_TGS) >>>>>>> >>>>>>>>>> >>>>>>> >>>>>>>>>> Could it be possible that its due to the initial load of >>>>>>> adding >>>>>>> >>>> the >>>>>>> >>>>>>>>> clients >>>>>>> >>>>>>>>>> or is there something else that I need to take care of. >>>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>> >>>> >>>> >>> >>> >>> >>> >>> -- >>> Manage your subscription for the Freeipa-users mailing list: >>> https://www.redhat.com/mailman/listinfo/freeipa-users >>> Go to http://freeipa.org for more info on the project >>> >> >> > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From tbordaz at redhat.com Mon Sep 5 11:18:32 2016 From: tbordaz at redhat.com (thierry bordaz) Date: Mon, 5 Sep 2016 13:18:32 +0200 Subject: [Freeipa-users] Freeipa 4.2.0 hangs intermittently In-Reply-To: References: <57BEB0AB.2090506@redhat.com> <57C44AC6.1030402@redhat.com> <57CD1FDC.2010109@redhat.com> Message-ID: <57CD5488.60404@redhat.com> On 09/05/2016 12:05 PM, Rakesh Rajasekharan wrote: > Hi Thierry, > > > I was getting the hang issue while running ipa-client-install > simultaneously on few clients.. > However, today, I am not able to replicate that. > > I could not get a gdb . But i will try getting that the next time I > face this issue. > > The CPU does not stay high.. it just momentarily touches a high value > and then drops down to around 2-7% > > One question I have is , is it ok to set it nsslapd-threadnumber to a > very high value . > I have around 4000 clients and with nsslapd-maxthreadsperconn set to > 5..So, can I set nsslapd-threadnumber to around 25000. Hello, I know some users running in production with several hunderds of threads (>600) and this without problem. I do not recall having suggested to increase that number and for what reason. Usually 30 workers is a good enough value. It can create bootleneck if for some reason each operation is very long to satisfy and exhaust the number of workers. You can monitor the work queue: ldapsearch -D "cn=directory manager" -w xxx -LLL -b "cn=monitor" -s base opsinitiated opscompleted If opscompleted-opsinitiated remains close to threadnumber, then yes it would be valuable to increase it. The computation #client * #async_op_per_client sound an overkill. Even if all clients send at the exact same time all their requests, it is very likely that some common resource (db page, log, allocator...) will serialize them. If you monitor a need to increase the work, you would for example set it to 50, then monitor, then set it to 100, then monitor... until you find a good enough value. Note the increasing the #thread, increases the memory footprint that will reduce the efficiency of file system cache and can increase the response time. best regards thierry > > Thanks > > On Mon, Sep 5, 2016 at 1:03 PM, thierry bordaz > wrote: > > > Hi Rakesh, > > Were you able to get a pstack or full stack with gdb > (http://www.port389.org/docs/389ds/FAQ/faq.html#debugging-crashes > ) when > the server hangs ? > > If it happens with 500 threads as well as with 30, using 30 > threads is a better choice to debug this issue. > I will try to reproduce using 150 parallel 'ipa user-find > p-testipa' commands > > Something I am unsure is if the CPU consumption stays high (you > mentioned 340% CPU usage) as long as the hang happens or if after > a suddent shot up to 340% (that marks the beginning of the hang) > it drops and stay hanging ? > > thanks > thierry > > On 09/04/2016 08:40 PM, Rakesh Rajasekharan wrote: >> starce on the slapd process actually had this in the output.. >> FUTEX_WAIT_PRIVATE >> >> and checking for the number of threads slapd had.. there were >> 5015 threads >> >> ps -efL|grep slapd|wc -l >> 5015 >> >> strace on most of the threads gave this output >> >> strace -p 67411 >> Process 67411 attached >> futex(0x7f3f0226b41c, FUTEX_WAIT_PRIVATE, 1, NULL) = -1 EAGAIN >> (Resource temporarily unavailable) >> futex(0x7f3f0226b41c, FUTEX_WAIT_PRIVATE, 2, NULL^CProcess 67411 >> detached >> >> >> >> >> >> On Sun, Sep 4, 2016 at 5:34 PM, Rakesh Rajasekharan >> > > wrote: >> >> I have again got the issue of IPA hanging.. The issue came up >> when i tried to run ipa-client-isntall on 142 clients >> simultaneously >> >> >> None of the IPA commands are responding, and I see this error >> >> ipa user-find p-testipa >> ipa: ERROR: Insufficient access: SASL(-1): generic failure: >> GSSAPI Error: Unspecified GSS failure. Minor code may >> provide more information (KDC returned error string: PROCESS_TGS) >> >> KRB5_TRACE=/dev/stdout kinit admin >> [41178] 1472984115.233214: Getting initial credentials for >> admin at XYZ.COM >> [41178] 1472984115.235257: Sending request (167 bytes) to >> XYZ.COM >> [41178] 1472984115.235419: Initiating TCP connection to >> stream 10.1.3.36:88 >> [41178] 1472984115.235685: Sending TCP request to stream >> 10.1.3.36:88 >> [41178] 1472984120.238914: Received answer (174 bytes) from >> stream 10.1.3.36:88 >> [41178] 1472984120.238925: Terminating TCP connection to >> stream 10.1.3.36:88 >> [41178] 1472984120.238993: Response was from master KDC >> [41 >> >> >> Running an ldapsearch to see the db.. does not give any >> results and just hangs there >> >> ldapsearch -x -D 'cn=Directory Manager' -W -s one -b >> 'cn=kerberos,dc=xyz,dc=com' >> Enter LDAP Password: >> >> even an ldapsearch -x does not respond >> At this point, am sure that slapd is the one causing issues >> >> Running an strace against the hung slapd itself seems to get >> stuck does not proceed after saying "attaching to process" >> >> From some others posts I read Thierry suggesting to increase >> the nsslapd-threadnumber value >> >> It was set to 30, I think that might be too low. >> >> I have raised it to 500 >> >> Now after restarting the service .. ldapsearch starts responding. >> But running the test to add a sudden high number of clients >> again left ns-slapd to hung state >> >> When i attempted adding the clients.. the ns-slapd cpu usage >> shot up to 340% and after that ns-slapd stopped responding >> >> So now, atleast I know what might be causing the issue and I >> can now easily reproduce it. >> >> Is there a way I can make ns-slapd handle a sudden bump in >> incoming request for ipa-client-install >> >> Thanks >> Rakesh >> >> >> >> >> >> >> On Mon, Aug 29, 2016 at 11:18 PM, Rich Megginson >> > wrote: >> >> On 08/29/2016 10:53 AM, Rakesh Rajasekharan wrote: >>> Hi Thierry, >>> >>> My machine has 30GB RAM ..and 389-ds version is 1.3.4 >>> >>> ldapsearch shows the values for nsslapd-cachememsize >>> updated to 200MB. >>> >>> ldapsearch -LLL -o ldif-wrap=no -D "cn=directory >>> manager" -w 'mypassword' -b 'cn=userRoot,cn=ldbm >>> database,cn=plugins,cn=config'|grep nsslapd-cachememsize >>> nsslapd-cachememsize: 209715200 >>> >>> >>> So, it seems to have updated though seeing that >>> warning(WARNING: ipaca: entry cache size 10485760B is >>> less than db size 11599872B) in the log confuses me a bit. >>> >>> Thers one more entry that I found from the ldapsearch to >>> be bit low >>> >>> nsslapd-dncachememsize: 10485760 >>> maxdncachesize: 10485760 >>> >>> Should I update these as well to a higher value >>> >>> At the time when the issue happened, the memory usage as >>> well as the overall load of the system was very low . >>> I will try reproducing the issue atleast in my QA >>> env..probably by trying to mock simultaneous parallel >>> logins to a large number of hosts >> >> To monitor your cache sizes, please use the dbmon.sh tool >> provided with your distro. If that is not available with >> your particular distro, see >> https://github.com/richm/scripts/wiki/dbmon.sh >> >> >> >>> >>> >>> thanks >>> Rakesh >>> >>> >>> >>> >>> On Mon, Aug 29, 2016 at 8:16 PM, thierry bordaz >>> > wrote: >>> >>> Hi Rakesh, >>> >>> Those tuning may depend on the memory available on >>> your machine. >>> nsslapd-cachememsize allows the entry cache to >>> consume up to 200Mb but its memory footprint is >>> known to go above. >>> 200Mb both looks pretty good to me. How large is >>> your machine ? What is your version of 389-ds ? >>> >>> Those warnings do not change your settings. It just >>> raise that entry cache of 'ipaca' and 'retrocl' are >>> small but it is fine. The size of the entry cache is >>> important mostly in userRoot. >>> You may double check the actual values, after >>> restart, with ldapsearch on 'cn=userRoot,cn=ldbm >>> database,cn=plugins,cn=config' and >>> 'cn=config,cn=ldbm database,cn=plugins,cn=config'. >>> >>> A step is to know what will be response time of DS >>> to know if it is responsible of the hang or not. >>> The logs and possibly pstack during those >>> intermittent hangs will help to determine that. >>> >>> regards >>> thierry >>> >>> >>> >>> >>> >>> On 08/29/2016 04:25 PM, Rakesh Rajasekharan wrote: >>>> I tried increasing the nsslapd-dbcachesize and >>>> nsslapd-cachememsize in my QA envs to 200MB. >>>> >>>> However, in my log files, I still see this message >>>> [29/Aug/2016:04:34:37 +0000] - WARNING: ipaca: >>>> entry cache size 10485760B is less than db size >>>> 11599872B; We recommend to increase the entry cache >>>> size nsslapd-cachememsize. >>>> [29/Aug/2016:04:34:37 +0000] - WARNING: changelog: >>>> entry cache size 2097152B is less than db size >>>> 441647104B; We recommend to increase the entry >>>> cache size nsslapd-cachememsize. >>>> >>>> these are my ldif files that i used to modify the >>>> values >>>> modify entry cache size >>>> cat modify-cache-mem-size.ldif >>>> dn: cn=userRoot,cn=ldbm database,cn=plugins,cn=config >>>> changetype: modify >>>> replace: nsslapd-cachememsize >>>> nsslapd-cachememsize: 209715200 >>>> >>>> modify db cache size >>>> cat modfy-db-cache-size.ldif >>>> dn: cn=config,cn=ldbm database,cn=plugins,cn=config >>>> changetype: modify >>>> replace: nsslapd-dbcachesize >>>> nsslapd-dbcachesize: 209715200 >>>> >>>> After modifying , i restarted IPA services >>>> >>>> Is there anything else that I need to take care of >>>> as the logs suggest its still not getting the >>>> updated values >>>> >>>> Thanks >>>> Rakesh >>>> >>>> On Mon, Aug 29, 2016 at 6:07 PM, Rakesh >>>> Rajasekharan >>> > wrote: >>>> >>>> Hi Thierry, >>>> >>>> Coz of the issues we had to revert back to >>>> earlier running openldap in production. >>>> >>>> I have now done a few TCP related changes in >>>> sysctl.conf and have also increased the >>>> nsslapd-dbcachesize and nsslapd-cachememsize to >>>> 200MB >>>> >>>> I will again start migrating hosts back to IPA >>>> and see if I face the earlier issue. >>>> >>>> I will update back once I have something >>>> >>>> >>>> Thanks, >>>> Rakesh >>>> >>>> >>>> >>>> On Thu, Aug 25, 2016 at 2:17 PM, thierry bordaz >>>> >>> > wrote: >>>> >>>> >>>> >>>> On 08/25/2016 10:15 AM, Rakesh Rajasekharan >>>> wrote: >>>>> All of the troubleshooting seems fine. >>>>> >>>>> >>>>> However, Running libconv.pl >>>>> gives me this output >>>>> >>>>> ----- Recommendations ----- >>>>> >>>>> 1. You have unindexed components, this >>>>> can be caused from a search on an >>>>> unindexed attribute, or your returned >>>>> results exceeded the allidsthreshold. >>>>> Unindexed components are not recommended. >>>>> To refuse unindexed searches, switch >>>>> 'nsslapd-require-index' to 'on' under your >>>>> database entry (e.g. cn=UserRoot,cn=ldbm >>>>> database,cn=plugins,cn=config). >>>>> >>>>> 2. You have a significant difference >>>>> between binds and unbinds. You may want to >>>>> investigate this difference. >>>>> >>>>> >>>>> I feel, this could be a pointer to things >>>>> going slow.. and IPA hanging. I think i >>>>> now have something that I can try and nail >>>>> down this issue. >>>>> >>>>> On a sidenote, I was earlier running >>>>> openldap and migrated over to Freeipa, >>>>> >>>>> Thanks >>>>> Rakesh >>>>> >>>>> >>>>> >>>>> On Wed, Aug 24, 2016 at 12:38 PM, Petr >>>>> Spacek >>>> > wrote: >>>>> >>>>> On 23.8.2016 18:44, Rakesh >>>>> Rajasekharan wrote: >>>>> > I think thers something seriously >>>>> wrong with my system >>>>> > >>>>> > not able to run any IPA commands >>>>> > >>>>> > klist >>>>> > Ticket cache: KEYRING:persistent:0:0 >>>>> > Default principal: admin at XYZ.COM >>>>> >>>>> > >>>>> > Valid starting Expires Service >>>>> principal >>>>> > 2016-08-23T16:26:36 >>>>> 2016-08-24T16:26:22 >>>>> krbtgt/XYZ.COM at XYZ.COM >>>>> >>>>> > >>>>> > >>>>> > [root at prod-ipa-master-1a :~] ipactl >>>>> status >>>>> > Directory Service: RUNNING >>>>> > krb5kdc Service: RUNNING >>>>> > kadmin Service: RUNNING >>>>> > ipa_memcached Service: RUNNING >>>>> > httpd Service: RUNNING >>>>> > pki-tomcatd Service: RUNNING >>>>> > ipa-otpd Service: RUNNING >>>>> > ipa: INFO: The ipactl command was >>>>> successful >>>>> > >>>>> > >>>>> > >>>>> > [root at prod-ipa-master :~] ipa >>>>> user-find p-testuser >>>>> > ipa: ERROR: Kerberos error: >>>>> ('Unspecified GSS failure. Minor code may >>>>> > provide more information', >>>>> 851968)/("Cannot contact any KDC for >>>>> realm ' >>>>> > XYZ.COM '", -1765328228) >>>>> >>>> >>>> Hi Rakesh, >>>> >>>> Having a reproducible test case would >>>> you rerun the command above. >>>> During its processing you may monitor >>>> DS process load (top). If it is high, >>>> you may get some pstacks of it. >>>> Also would you attach the part of DS >>>> access logs taken during the command. >>>> >>>> regards >>>> thierry >>>> >>>>> > >>>>> >>>>> This is weird because the server seems >>>>> to be up. >>>>> >>>>> Please follow >>>>> http://www.freeipa.org/page/Troubleshooting#Authentication.2FKerberos >>>>> >>>>> Petr^2 Spacek >>>>> >>>>> > >>>>> > >>>>> > Thanks >>>>> > >>>>> > Rakesh >>>>> > >>>>> > On Tue, Aug 23, 2016 at 10:01 PM, >>>>> Rakesh Rajasekharan < >>>>> > rakesh.rajasekharan at gmail.com >>>>> > wrote: >>>>> > >>>>> >> i changed the loggin level to 4 . >>>>> Modifying nsslapd-accesslog-level >>>>> >> >>>>> >> But, the hang is still there. >>>>> though I dont see the sigfault now >>>>> >> >>>>> >> >>>>> >> >>>>> >> >>>>> >> On Tue, Aug 23, 2016 at 9:02 PM, >>>>> Rakesh Rajasekharan < >>>>> >> rakesh.rajasekharan at gmail.com >>>>> > wrote: >>>>> >> >>>>> >>> My disk was getting filled too fast >>>>> >>> >>>>> >>> logs under /var/log/dirsrv was >>>>> coming around 5 gb quickly filling up >>>>> >>> >>>>> >>> Is there a way to make the logging >>>>> less verbose >>>>> >>> >>>>> >>> >>>>> >>> >>>>> >>> On Tue, Aug 23, 2016 at 6:41 PM, >>>>> Petr Spacek >>>> > wrote: >>>>> >>> >>>>> >>>> On 23.8.2016 15:07, Rakesh >>>>> Rajasekharan wrote: >>>>> >>>>> I was able to fix that may be >>>>> temporarily... when i checked the >>>>> >>>> network.. >>>>> >>>>> there was another process that >>>>> was running and consuming a lot of >>>>> >>>> network ( >>>>> >>>>> i have no idea who did that. I >>>>> need to seriously start restricting >>>>> >>>> people >>>>> >>>>> access to this machine ) >>>>> >>>>> >>>>> >>>>> after killing that perfomance >>>>> improved drastically >>>>> >>>>> >>>>> >>>>> But now, suddenly I started >>>>> experiencing the same hang. >>>>> >>>>> >>>>> >>>>> This time , I gert the following >>>>> error when checked dmesg >>>>> >>>>> >>>>> >>>>> [ 301.236976] ns-slapd[3124]: >>>>> segfault at 0 ip 00007f1de416951c sp >>>>> >>>>> 00007f1dee1dba70 error 4 in >>>>> libcos-plugin.so[7f1de4166000+b000] >>>>> >>>>> [ 1116.248431] TCP: >>>>> request_sock_TCP: Possible SYN >>>>> flooding on port 88. >>>>> >>>>> Sending cookies. Check SNMP >>>>> counters. >>>>> >>>>> [11831.397037] ns-slapd[22550]: >>>>> segfault at 0 ip 00007f533d82251c sp >>>>> >>>>> 00007f5347894a70 error 4 in >>>>> libcos-plugin.so[7f533d81f000+b000] >>>>> >>>>> [11832.727989] ns-slapd[22606]: >>>>> segfault at 0 ip 00007f6231eb951c sp >>>>> >>>>> 00007f623bf2ba70 error 4 in >>>>> libcos-plugin.so[7f6231eb6000+b00 >>>>> >>>> >>>>> >>>> Okay, this one is serious. The >>>>> LDAP server crashed. >>>>> >>>> >>>>> >>>> 1. Make sure all your packages >>>>> are up-to-date. >>>>> >>>> >>>>> >>>> Please see >>>>> >>>> >>>>> http://directory.fedoraproject.org/docs/389ds/FAQ/faq.html#d >>>>> >>>> ebugging-crashes >>>>> >>>> for further instructions how to >>>>> debug this. >>>>> >>>> >>>>> >>>> Petr^2 Spacek >>>>> >>>> >>>>> >>>>> >>>>> >>>>> and in >>>>> /var/log/dirsrv/example-com/errors >>>>> >>>>> >>>>> >>>>> [23/Aug/2016:12:49:36 +0000] >>>>> DSRetroclPlugin - delete_changerecord: >>>>> >>>> could >>>>> >>>>> not delete change record 3291138 >>>>> (rc: 32) >>>>> >>>>> [23/Aug/2016:12:49:36 +0000] >>>>> DSRetroclPlugin - delete_changerecord: >>>>> >>>> could >>>>> >>>>> not delete change record 3291139 >>>>> (rc: 32) >>>>> >>>>> [23/Aug/2016:12:49:36 +0000] >>>>> DSRetroclPlugin - delete_changerecord: >>>>> >>>> could >>>>> >>>>> not delete change record 3291140 >>>>> (rc: 32) >>>>> >>>>> [23/Aug/2016:12:49:36 +0000] >>>>> DSRetroclPlugin - delete_changerecord: >>>>> >>>> could >>>>> >>>>> not delete change record 3291141 >>>>> (rc: 32) >>>>> >>>>> [23/Aug/2016:12:49:36 +0000] >>>>> DSRetroclPlugin - delete_changerecord: >>>>> >>>> could >>>>> >>>>> not delete change record 3291142 >>>>> (rc: 32) >>>>> >>>>> [23/Aug/2016:12:49:36 +0000] >>>>> DSRetroclPlugin - delete_changerecord: >>>>> >>>> could >>>>> >>>>> not delete change record 3291143 >>>>> (rc: 32) >>>>> >>>>> [23/Aug/2016:12:49:36 +0000] >>>>> DSRetroclPlugin - delete_changerecord: >>>>> >>>> could >>>>> >>>>> not delete change record 3291144 >>>>> (rc: 32) >>>>> >>>>> [23/Aug/2016:12:49:36 +0000] >>>>> DSRetroclPlugin - delete_changerecord: >>>>> >>>> could >>>>> >>>>> not delete change record 3291145 >>>>> (rc: 32) >>>>> >>>>> [23/Aug/2016:12:49:50 +0000] - >>>>> Retry count exceeded in delete >>>>> >>>>> [23/Aug/2016:12:49:50 +0000] >>>>> DSRetroclPlugin - delete_changerecord: >>>>> >>>> could >>>>> >>>>> not delete change record 3292734 >>>>> (rc: 51) >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> Can i do something about this >>>>> error.. I treid to restart ipa a couple >>>>> >>>> of >>>>> >>>>> time but that did not help >>>>> >>>>> >>>>> >>>>> Thanks >>>>> >>>>> Rakesh >>>>> >>>>> >>>>> >>>>> On Mon, Aug 22, 2016 at 2:27 PM, >>>>> Petr Spacek >>>> > >>>>> >>>> wrote: >>>>> >>>>> >>>>> >>>>>> On 19.8.2016 19:32, Rakesh >>>>> Rajasekharan wrote: >>>>> >>>>>>> I am running my set up on AWS >>>>> cloud, and entropy is low at around >>>>> >>>> 180 . >>>>> >>>>>>> >>>>> >>>>>>> I plan to increase it bu >>>>> installing haveged . But, would low >>>>> entropy >>>>> >>>> by >>>>> >>>>>> any >>>>> >>>>>>> chance cause this issue of >>>>> intermittent hang . >>>>> >>>>>>> Also, the hang is mostly >>>>> observed when registering around 20 >>>>> clients >>>>> >>>>>>> together >>>>> >>>>>> >>>>> >>>>>> Possibly, I'm not sure. If you >>>>> want to dig into this, I would do this: >>>>> >>>>>> 1. look what process hangs on >>>>> client (using pstree command or so) >>>>> >>>>>> $ pstree >>>>> >>>>>> >>>>> >>>>>> 2. look to what server and port >>>>> is the hanging client connected to >>>>> >>>>>> $ lsof -p >>>> process> >>>>> >>>>>> >>>>> >>>>>> 3. jump to server and see what >>>>> process is bound to the target port >>>>> >>>>>> $ netstat -pn >>>>> >>>>>> >>>>> >>>>>> 4. see where the process if hanging >>>>> >>>>>> $ strace -p >>>> process> >>>>> >>>>>> >>>>> >>>>>> I hope it helps. >>>>> >>>>>> >>>>> >>>>>> Petr^2 Spacek >>>>> >>>>>> >>>>> >>>>>>> On Fri, Aug 19, 2016 at 7:24 >>>>> PM, Rakesh Rajasekharan < >>>>> >>>>>>> rakesh.rajasekharan at gmail.com >>>>> > wrote: >>>>> >>>>>>> >>>>> >>>>>>>> yes there seems to be >>>>> something thats worrying.. I have >>>>> faced this >>>>> >>>> today >>>>> >>>>>>>> as well. >>>>> >>>>>>>> There are few hosts around >>>>> 280 odd left and when i try adding them >>>>> >>>> to >>>>> >>>>>> IPA >>>>> >>>>>>>> , the slowness begins.. >>>>> >>>>>>>> >>>>> >>>>>>>> all the ipa commands like ipa >>>>> user-find.. etc becomes very slow in >>>>> >>>>>>>> responding. >>>>> >>>>>>>> >>>>> >>>>>>>> the SYNC_RECV are not many >>>>> though just around 80-90 and today that >>>>> >>>> was >>>>> >>>>>>>> around 20 only >>>>> >>>>>>>> >>>>> >>>>>>>> >>>>> >>>>>>>> I have for now increased >>>>> tcp_max_syn_backlog to 5000. >>>>> >>>>>>>> For now the slowness seems to >>>>> have gone.. but I will do a try >>>>> >>>> adding the >>>>> >>>>>>>> clients again tomorrow and >>>>> see how it goes >>>>> >>>>>>>> >>>>> >>>>>>>> Thanks >>>>> >>>>>>>> Rakesh >>>>> >>>>>>>> >>>>> >>>>>>>> The issues >>>>> >>>>>>>> >>>>> >>>>>>>> On Fri, Aug 19, 2016 at 12:58 >>>>> PM, Petr Spacek >>>> > >>>>> >>>>>> wrote: >>>>> >>>>>>>> >>>>> >>>>>>>>> On 18.8.2016 17:23, Rakesh >>>>> Rajasekharan wrote: >>>>> >>>>>>>>>> Hi >>>>> >>>>>>>>>> >>>>> >>>>>>>>>> I am migrating to freeipa >>>>> from openldap and have around 4000 >>>>> >>>> clients >>>>> >>>>>>>>>> >>>>> >>>>>>>>>> I had openned a another >>>>> thread on that, but chose to start a new >>>>> >>>> one >>>>> >>>>>>>>> here >>>>> >>>>>>>>>> as its a separate issue >>>>> >>>>>>>>>> >>>>> >>>>>>>>>> I was able to change the >>>>> nssslapd-maxdescriptors adding an ldif >>>>> >>>> file >>>>> >>>>>>>>>> >>>>> >>>>>>>>>> cat nsslapd-modify.ldif >>>>> >>>>>>>>>> dn: cn=config >>>>> >>>>>>>>>> changetype: modify >>>>> >>>>>>>>>> replace: nsslapd-maxdescriptors >>>>> >>>>>>>>>> nsslapd-maxdescriptors: 17000 >>>>> >>>>>>>>>> >>>>> >>>>>>>>>> and running the ldapmodify >>>>> command >>>>> >>>>>>>>>> >>>>> >>>>>>>>>> I have now started moving >>>>> clients running an openldap to Freeipa >>>>> >>>> and >>>>> >>>>>>>>> have >>>>> >>>>>>>>>> today moved close to 2000 >>>>> clients >>>>> >>>>>>>>>> >>>>> >>>>>>>>>> However, I have noticed >>>>> that IPA hangs intermittently. >>>>> >>>>>>>>>> >>>>> >>>>>>>>>> running a kinit admin >>>>> returns the below error >>>>> >>>>>>>>>> kinit: Generic error (see >>>>> e-text) while getting initial >>>>> >>>> credentials >>>>> >>>>>>>>>> >>>>> >>>>>>>>>> from the /var/log/messages, >>>>> I see this entry >>>>> >>>>>>>>>> >>>>> >>>>>>>>>> prod-ipa-master-int kernel: >>>>> [104090.315801] TCP: >>>>> >>>> request_sock_TCP: >>>>> >>>>>>>>>> Possible SYN flooding on >>>>> port 88. Sending cookies. Check SNMP >>>>> >>>>>> counters. >>>>> >>>>>>>>> >>>>> >>>>>>>>> I would be worried about >>>>> this message. Maybe kernel/firewall is >>>>> >>>> doing >>>>> >>>>>>>>> something fishy behind your >>>>> back and blocking some connections or >>>>> >>>> so. >>>>> >>>>>>>>> >>>>> >>>>>>>>> Petr^2 Spacek >>>>> >>>>>>>>> >>>>> >>>>>>>>> >>>>> >>>>>>>>>> Aug 18 13:00:01 >>>>> prod-ipa-master-int systemd[1]: >>>>> Started Session >>>>> >>>> 4885 >>>>> >>>>>> of >>>>> >>>>>>>>>> user root. >>>>> >>>>>>>>>> Aug 18 13:00:01 >>>>> prod-ipa-master-int systemd[1]: >>>>> Starting Session >>>>> >>>> 4885 >>>>> >>>>>> of >>>>> >>>>>>>>>> user root. >>>>> >>>>>>>>>> Aug 18 13:01:01 >>>>> prod-ipa-master-int systemd[1]: >>>>> Started Session >>>>> >>>> 4886 >>>>> >>>>>> of >>>>> >>>>>>>>>> user root. >>>>> >>>>>>>>>> Aug 18 13:01:01 >>>>> prod-ipa-master-int systemd[1]: >>>>> Starting Session >>>>> >>>> 4886 >>>>> >>>>>> of >>>>> >>>>>>>>>> user root. >>>>> >>>>>>>>>> Aug 18 13:02:40 >>>>> prod-ipa-master-int python[28984]: >>>>> ansible-command >>>>> >>>>>>>>> Invoked >>>>> >>>>>>>>>> with creates=None >>>>> executable=None shell=True args= >>>>> removes=None >>>>> >>>>>>>>> warn=True >>>>> >>>>>>>>>> chdir=None >>>>> >>>>>>>>>> Aug 18 13:04:37 >>>>> prod-ipa-master-int sssd_be: GSSAPI Error: >>>>> >>>> Unspecified >>>>> >>>>>>>>> GSS >>>>> >>>>>>>>>> failure. Minor code may >>>>> provide more information (KDC returned >>>>> >>>> error >>>>> >>>>>>>>>> string: PROCESS_TGS) >>>>> >>>>>>>>>> >>>>> >>>>>>>>>> Could it be possible that >>>>> its due to the initial load of adding >>>>> >>>> the >>>>> >>>>>>>>> clients >>>>> >>>>>>>>>> or is there something else >>>>> that I need to take care of. >>>>> >>>>> >>>>> >>>>> >>>> >>>> >>>> >>> >>> >>> >>> >> >> >> -- >> Manage your subscription for the Freeipa-users mailing list: >> https://www.redhat.com/mailman/listinfo/freeipa-users >> >> Go to http://freeipa.org for more info on the project >> >> >> >> >> > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From rakesh.rajasekharan at gmail.com Mon Sep 5 12:05:46 2016 From: rakesh.rajasekharan at gmail.com (Rakesh Rajasekharan) Date: Mon, 5 Sep 2016 17:35:46 +0530 Subject: [Freeipa-users] Freeipa 4.2.0 hangs intermittently In-Reply-To: <57CD5488.60404@redhat.com> References: <57BEB0AB.2090506@redhat.com> <57C44AC6.1030402@redhat.com> <57CD1FDC.2010109@redhat.com> <57CD5488.60404@redhat.com> Message-ID: Hi Thierry, I was able to now reporduce the issue.. the ns-slapd process just got hung,.. however, no core file was generate.. I manually created a coredump using gcore and then took a strace of it this is strace from the coredump generated GNU gdb (GDB) Red Hat Enterprise Linux 7.6.1-80.el7 Copyright (C) 2013 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type "show copying" and "show warranty" for details. This GDB was configured as "x86_64-redhat-linux-gnu". For bug reporting instructions, please see: ... Reading symbols from /usr/sbin/ns-slapd...Reading symbols from /usr/sbin/ns-slapd...(no debugging symbols found)...done. (no debugging symbols found)...done. [New LWP 105159] [New LWP 105160] [New LWP 105161] [New LWP 105162] [New LWP 105163] [New LWP 105164] [New LWP 105165] [New LWP 105166] [New LWP 105167] [New LWP 105168] [New LWP 105171] [New LWP 105172] [New LWP 105173] [New LWP 105174] [New LWP 105175] [New LWP 105176] [New LWP 105177] [New LWP 105178] [New LWP 105179] [New LWP 105180] [New LWP 105181] [New LWP 105182] [New LWP 105183] [New LWP 105184] [New LWP 105185] [New LWP 105186] [New LWP 105187] [New LWP 105188] [New LWP 105189] [New LWP 105190] [New LWP 105191] [New LWP 105192] [New LWP 105193] [New LWP 105194] [New LWP 105195] [New LWP 105196] [New LWP 105197] [New LWP 105198] [New LWP 105199] [New LWP 105200] [New LWP 105201] [New LWP 105203] [New LWP 105215] [New LWP 105158] [Thread debugging using libthread_db enabled] Using host libthread_db library "/lib64/libthread_db.so.1". Core was generated by `/usr/sbin/ns-slapd'. #0 0x00007f1b48ae69b3 in select () from /lib64/libc.so.6 Thread 44 (Thread 0x7f1b4b6d9840 (LWP 105158)): #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 No symbol table info available. #1 0x00007f1b4941b2c3 in PR_EnterMonitor () from /lib64/libnspr4.so No symbol table info available. #2 0x00007f1b4b7125ed in setup_pr_read_pds () No symbol table info available. #3 0x00007f1b4b715fd9 in slapd_daemon () No symbol table info available. #4 0x00007f1b4b707c6c in main () No symbol table info available. Thread 43 (Thread 0x7f1a5df0f700 (LWP 105215)): #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 No symbol table info available. #1 0x00007f1b4941b050 in PR_WaitCondVar () from /lib64/libnspr4.so No symbol table info available. #2 0x00007f1b4b71df25 in ps_send_results () No symbol table info available. #3 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so No symbol table info available. #4 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 No symbol table info available. #5 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 No symbol table info available. Thread 42 (Thread 0x7f1b03fe7700 (LWP 105203)): #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 No symbol table info available. #1 0x00007f1b4941b050 in PR_WaitCondVar () from /lib64/libnspr4.so No symbol table info available. #2 0x00007f1b4b71df25 in ps_send_results () No symbol table info available. #3 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so No symbol table info available. #4 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 No symbol table info available. #5 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 No symbol table info available. Thread 41 (Thread 0x7f1b047e8700 (LWP 105201)): #0 0x00007f1b48ae69b3 in select () from /lib64/libc.so.6 No symbol table info available. #1 0x00007f1b4b2710e9 in DS_Sleep () from /usr/lib64/dirsrv/libslapd.so.0 No symbol table info available. #2 0x00007f1b4b712325 in time_thread () No symbol table info available. #3 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so No symbol table info available. #4 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 No symbol table info available. #5 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 No symbol table info available. Thread 40 (Thread 0x7f1b04fe9700 (LWP 105200)): #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 No symbol table info available. #1 0x00007f1b433212f3 in __db_hybrid_mutex_suspend () from /lib64/ libdb-5.3.so No symbol table info available. #2 0x00007f1b43320640 in __db_tas_mutex_lock () from /lib64/libdb-5.3.so No symbol table info available. #3 0x00007f1b4343ae1b in __memp_fget () from /lib64/libdb-5.3.so No symbol table info available. #4 0x00007f1b4333e211 in __bam_search () from /lib64/libdb-5.3.so No symbol table info available. #5 0x00007f1b43329256 in __bamc_search () from /lib64/libdb-5.3.so No symbol table info available. #6 0x00007f1b4332d6c4 in __bamc_put () from /lib64/libdb-5.3.so No symbol table info available. #7 0x00007f1b433e46d5 in __dbc_iput () from /lib64/libdb-5.3.so No symbol table info available. #8 0x00007f1b433dfa0e in __db_put () from /lib64/libdb-5.3.so No symbol table info available. #9 0x00007f1b433f4fa4 in __db_put_pp () from /lib64/libdb-5.3.so No symbol table info available. #10 0x00007f1b3e8cf4cf in idl_new_insert_key () from /usr/lib64/dirsrv/plugins/libback-ldbm.so No symbol table info available. #11 0x00007f1b3e8de5b5 in addordel_values_sv.isra.3 () from /usr/lib64/dirsrv/plugins/libback-ldbm.so No symbol table info available. #12 0x00007f1b3e8df25a in index_addordel_values_ext_sv () from /usr/lib64/dirsrv/plugins/libback-ldbm.so No symbol table info available. #13 0x00007f1b3e8df694 in index_addordel_values_sv () from /usr/lib64/dirsrv/plugins/libback-ldbm.so No symbol table info available. #14 0x00007f1b3e8df75a in index_addordel_entry () from /usr/lib64/dirsrv/plugins/libback-ldbm.so No symbol table info available. #15 0x00007f1b3e8e25fa in ldbm_back_add () from /usr/lib64/dirsrv/plugins/libback-ldbm.so No symbol table info available. #16 0x00007f1b4b1e4cca in op_shared_add () from /usr/lib64/dirsrv/libslapd.so.0 No symbol table info available. #17 0x00007f1b4b1e6020 in do_add () from /usr/lib64/dirsrv/libslapd.so.0 No symbol table info available. #18 0x00007f1b4b711303 in connection_threadmain () No symbol table info available. #19 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so No symbol table info available. #20 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 No symbol table info available. #21 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 No symbol table info available. Thread 39 (Thread 0x7f1b057ea700 (LWP 105199)): #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 No symbol table info available. #1 0x00007f1b4941b050 in PR_WaitCondVar () from /lib64/libnspr4.so No symbol table info available. #2 0x00007f1b4b70efee in connection_wait_for_new_work () No symbol table info available. #3 0x00007f1b4b710719 in connection_threadmain () No symbol table info available. #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so No symbol table info available. #5 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 No symbol table info available. #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 No symbol table info available. Thread 38 (Thread 0x7f1b05feb700 (LWP 105198)): #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 No symbol table info available. #1 0x00007f1b4941b050 in PR_WaitCondVar () from /lib64/libnspr4.so No symbol table info available. #2 0x00007f1b4b70efee in connection_wait_for_new_work () No symbol table info available. #3 0x00007f1b4b710719 in connection_threadmain () No symbol table info available. #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so No symbol table info available. #5 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 No symbol table info available. #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 No symbol table info available. Thread 37 (Thread 0x7f1b067ec700 (LWP 105197)): #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 No symbol table info available. #1 0x00007f1b4941b050 in PR_WaitCondVar () from /lib64/libnspr4.so No symbol table info available. #2 0x00007f1b4b70efee in connection_wait_for_new_work () No symbol table info available. #3 0x00007f1b4b710719 in connection_threadmain () No symbol table info available. #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so No symbol table info available. #5 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 No symbol table info available. #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 No symbol table info available. Thread 36 (Thread 0x7f1b06fed700 (LWP 105196)): #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 No symbol table info available. #1 0x00007f1b4941b050 in PR_WaitCondVar () from /lib64/libnspr4.so No symbol table info available. #2 0x00007f1b4b70efee in connection_wait_for_new_work () No symbol table info available. #3 0x00007f1b4b710719 in connection_threadmain () No symbol table info available. #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so No symbol table info available. #5 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 No symbol table info available. #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 No symbol table info available. Thread 35 (Thread 0x7f1b077ee700 (LWP 105195)): #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 No symbol table info available. #1 0x00007f1b4941b2c3 in PR_EnterMonitor () from /lib64/libnspr4.so No symbol table info available. #2 0x00007f1b3e8c23e6 in dblayer_txn_begin () from /usr/lib64/dirsrv/plugins/libback-ldbm.so No symbol table info available. #3 0x00007f1b3e8fe007 in ldbm_back_modify () from /usr/lib64/dirsrv/plugins/libback-ldbm.so No symbol table info available. #4 0x00007f1b4b22b541 in op_shared_modify () from /usr/lib64/dirsrv/libslapd.so.0 No symbol table info available. #5 0x00007f1b4b22c004 in modify_internal_pb () from /usr/lib64/dirsrv/libslapd.so.0 No symbol table info available. #6 0x00007f1b3eb48b60 in ipapwd_apply_mods () from /usr/lib64/dirsrv/plugins/libipa_pwd_extop.so No symbol table info available. #7 0x00007f1b3eb4c8e0 in store_new_keys () from /usr/lib64/dirsrv/plugins/libipa_pwd_extop.so No symbol table info available. #8 0x00007f1b3eb4ead0 in ipapwd_extop () from /usr/lib64/dirsrv/plugins/libipa_pwd_extop.so No symbol table info available. #9 0x00007f1b4b23bba2 in plugin_call_exop_plugins () from /usr/lib64/dirsrv/libslapd.so.0 No symbol table info available. #10 0x00007f1b4b717efe in do_extended () No symbol table info available. #11 0x00007f1b4b71141a in connection_threadmain () No symbol table info available. #12 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so No symbol table info available. #13 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 No symbol table info available. #14 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 No symbol table info available. Thread 34 (Thread 0x7f1b07fef700 (LWP 105194)): #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 No symbol table info available. #1 0x00007f1b4941b050 in PR_WaitCondVar () from /lib64/libnspr4.so No symbol table info available. #2 0x00007f1b4b70efee in connection_wait_for_new_work () No symbol table info available. #3 0x00007f1b4b710719 in connection_threadmain () No symbol table info available. #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so No symbol table info available. #5 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 No symbol table info available. #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 No symbol table info available. Thread 33 (Thread 0x7f1b087f0700 (LWP 105193)): #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 No symbol table info available. #1 0x00007f1b4941b050 in PR_WaitCondVar () from /lib64/libnspr4.so No symbol table info available. #2 0x00007f1b4b70efee in connection_wait_for_new_work () No symbol table info available. #3 0x00007f1b4b710719 in connection_threadmain () No symbol table info available. #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so No symbol table info available. #5 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 No symbol table info available. #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 No symbol table info available. Thread 32 (Thread 0x7f1b08ff1700 (LWP 105192)): #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 No symbol table info available. #1 0x00007f1b4941b050 in PR_WaitCondVar () from /lib64/libnspr4.so No symbol table info available. #2 0x00007f1b4b70efee in connection_wait_for_new_work () No symbol table info available. #3 0x00007f1b4b710719 in connection_threadmain () No symbol table info available. #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so No symbol table info available. #5 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 No symbol table info available. #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 No symbol table info available. Thread 31 (Thread 0x7f1b097f2700 (LWP 105191)): #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 No symbol table info available. #1 0x00007f1b4941b050 in PR_WaitCondVar () from /lib64/libnspr4.so No symbol table info available. #2 0x00007f1b4b70efee in connection_wait_for_new_work () No symbol table info available. #3 0x00007f1b4b710719 in connection_threadmain () No symbol table info available. #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so No symbol table info available. #5 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 No symbol table info available. #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 No symbol table info available. Thread 30 (Thread 0x7f1b09ff3700 (LWP 105190)): #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 No symbol table info available. #1 0x00007f1b433212f3 in __db_hybrid_mutex_suspend () from /lib64/ libdb-5.3.so No symbol table info available. #2 0x00007f1b43320640 in __db_tas_mutex_lock () from /lib64/libdb-5.3.so No symbol table info available. #3 0x00007f1b433cad3a in __lock_get_internal () from /lib64/libdb-5.3.so No symbol table info available. #4 0x00007f1b433cb820 in __lock_get () from /lib64/libdb-5.3.so No symbol table info available. #5 0x00007f1b433f7142 in __db_lget () from /lib64/libdb-5.3.so No symbol table info available. #6 0x00007f1b4333e605 in __bam_search () from /lib64/libdb-5.3.so No symbol table info available. #7 0x00007f1b43329256 in __bamc_search () from /lib64/libdb-5.3.so No symbol table info available. #8 0x00007f1b4332ad0f in __bamc_get () from /lib64/libdb-5.3.so No symbol table info available. #9 0x00007f1b433e3ca6 in __dbc_iget () from /lib64/libdb-5.3.so No symbol table info available. #10 0x00007f1b433f2b02 in __dbc_get_pp () from /lib64/libdb-5.3.so No symbol table info available. #11 0x00007f1b3e8ce680 in idl_new_fetch () from /usr/lib64/dirsrv/plugins/libback-ldbm.so No symbol table info available. #12 0x00007f1b3e8dcb26 in index_read_ext_allids () from /usr/lib64/dirsrv/plugins/libback-ldbm.so No symbol table info available. #13 0x00007f1b3e8c7284 in keys2idl () from /usr/lib64/dirsrv/plugins/libback-ldbm.so No symbol table info available. #14 0x00007f1b3e8c79e3 in ava_candidates.isra.0 () from /usr/lib64/dirsrv/plugins/libback-ldbm.so No symbol table info available. #15 0x00007f1b3e8c7fd2 in filter_candidates_ext () from /usr/lib64/dirsrv/plugins/libback-ldbm.so No symbol table info available. #16 0x00007f1b3e8c9096 in list_candidates () from /usr/lib64/dirsrv/plugins/libback-ldbm.so No symbol table info available. #17 0x00007f1b3e8c7f40 in filter_candidates_ext () from /usr/lib64/dirsrv/plugins/libback-ldbm.so No symbol table info available. #18 0x00007f1b3e8c9096 in list_candidates () from /usr/lib64/dirsrv/plugins/libback-ldbm.so No symbol table info available. #19 0x00007f1b3e8c7f40 in filter_candidates_ext () from /usr/lib64/dirsrv/plugins/libback-ldbm.so No symbol table info available. #20 0x00007f1b3e8c9096 in list_candidates () from /usr/lib64/dirsrv/plugins/libback-ldbm.so No symbol table info available. #21 0x00007f1b3e8c7f40 in filter_candidates_ext () from /usr/lib64/dirsrv/plugins/libback-ldbm.so No symbol table info available. #22 0x00007f1b3e90380c in subtree_candidates () from /usr/lib64/dirsrv/plugins/libback-ldbm.so No symbol table info available. #23 0x00007f1b3e904e7f in ldbm_back_search () from /usr/lib64/dirsrv/plugins/libback-ldbm.so No symbol table info available. #24 0x00007f1b4b2321d6 in op_shared_search () from /usr/lib64/dirsrv/libslapd.so.0 No symbol table info available. #25 0x00007f1b4b72333e in do_search () No symbol table info available. #26 0x00007f1b4b7113c3 in connection_threadmain () No symbol table info available. #27 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so No symbol table info available. #28 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 No symbol table info available. #29 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 No symbol table info available. Thread 29 (Thread 0x7f1b0a7f4700 (LWP 105189)): #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 No symbol table info available. #1 0x00007f1b4941b050 in PR_WaitCondVar () from /lib64/libnspr4.so No symbol table info available. #2 0x00007f1b4b70efee in connection_wait_for_new_work () No symbol table info available. #3 0x00007f1b4b710719 in connection_threadmain () No symbol table info available. #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so No symbol table info available. #5 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 No symbol table info available. #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 No symbol table info available. Thread 28 (Thread 0x7f1b0aff5700 (LWP 105188)): #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 No symbol table info available. #1 0x00007f1b4941b050 in PR_WaitCondVar () from /lib64/libnspr4.so No symbol table info available. #2 0x00007f1b4b70efee in connection_wait_for_new_work () No symbol table info available. #3 0x00007f1b4b710719 in connection_threadmain () No symbol table info available. #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so No symbol table info available. #5 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 No symbol table info available. #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 No symbol table info available. Thread 27 (Thread 0x7f1b0b7f6700 (LWP 105187)): #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 No symbol table info available. #1 0x00007f1b4941b050 in PR_WaitCondVar () from /lib64/libnspr4.so No symbol table info available. #2 0x00007f1b4b70efee in connection_wait_for_new_work () No symbol table info available. #3 0x00007f1b4b710719 in connection_threadmain () No symbol table info available. #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so No symbol table info available. #5 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 No symbol table info available. #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 No symbol table info available. Thread 26 (Thread 0x7f1b0bff7700 (LWP 105186)): #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 No symbol table info available. #1 0x00007f1b433212f3 in __db_hybrid_mutex_suspend () from /lib64/ libdb-5.3.so No symbol table info available. #2 0x00007f1b43320640 in __db_tas_mutex_lock () from /lib64/libdb-5.3.so No symbol table info available. #3 0x00007f1b433cad3a in __lock_get_internal () from /lib64/libdb-5.3.so No symbol table info available. #4 0x00007f1b433cb820 in __lock_get () from /lib64/libdb-5.3.so No symbol table info available. #5 0x00007f1b433f7142 in __db_lget () from /lib64/libdb-5.3.so No symbol table info available. #6 0x00007f1b4333e605 in __bam_search () from /lib64/libdb-5.3.so No symbol table info available. #7 0x00007f1b43329256 in __bamc_search () from /lib64/libdb-5.3.so No symbol table info available. #8 0x00007f1b4332ad0f in __bamc_get () from /lib64/libdb-5.3.so No symbol table info available. #9 0x00007f1b433e3ca6 in __dbc_iget () from /lib64/libdb-5.3.so No symbol table info available. #10 0x00007f1b433f2b02 in __dbc_get_pp () from /lib64/libdb-5.3.so No symbol table info available. #11 0x00007f1b3e8ce680 in idl_new_fetch () from /usr/lib64/dirsrv/plugins/libback-ldbm.so No symbol table info available. #12 0x00007f1b3e8dcb26 in index_read_ext_allids () from /usr/lib64/dirsrv/plugins/libback-ldbm.so No symbol table info available. #13 0x00007f1b3e8c7284 in keys2idl () from /usr/lib64/dirsrv/plugins/libback-ldbm.so No symbol table info available. #14 0x00007f1b3e8c79e3 in ava_candidates.isra.0 () from /usr/lib64/dirsrv/plugins/libback-ldbm.so No symbol table info available. #15 0x00007f1b3e8c7fd2 in filter_candidates_ext () from /usr/lib64/dirsrv/plugins/libback-ldbm.so No symbol table info available. #16 0x00007f1b3e8c9096 in list_candidates () from /usr/lib64/dirsrv/plugins/libback-ldbm.so No symbol table info available. #17 0x00007f1b3e8c7f40 in filter_candidates_ext () from /usr/lib64/dirsrv/plugins/libback-ldbm.so No symbol table info available. #18 0x00007f1b3e90380c in subtree_candidates () from /usr/lib64/dirsrv/plugins/libback-ldbm.so No symbol table info available. #19 0x00007f1b3e904e7f in ldbm_back_search () from /usr/lib64/dirsrv/plugins/libback-ldbm.so No symbol table info available. #20 0x00007f1b4b2321d6 in op_shared_search () from /usr/lib64/dirsrv/libslapd.so.0 No symbol table info available. #21 0x00007f1b4b24294e in search_internal_callback_pb () from /usr/lib64/dirsrv/libslapd.so.0 No symbol table info available. #22 0x00007f1b4b242be8 in search_internal_pb () from /usr/lib64/dirsrv/libslapd.so.0 No symbol table info available. #23 0x00007f1b4b72149b in ids_sasl_canon_user () No symbol table info available. #24 0x00007f1b491e193b in _sasl_canon_user () from /lib64/libsasl2.so.3 No symbol table info available. #25 0x00007f1b491e1c4c in _sasl_canon_user_lookup () from /lib64/libsasl2.so.3 No symbol table info available. #26 0x00007f1b430e9fb9 in gssapi_server_mech_step () from /usr/lib64/sasl2/libgssapiv2.so No symbol table info available. #27 0x00007f1b491edb9b in sasl_server_step () from /lib64/libsasl2.so.3 No symbol table info available. #28 0x00007f1b4b72283b in ids_sasl_check_bind () No symbol table info available. #29 0x00007f1b4b70a61a in do_bind () No symbol table info available. #30 0x00007f1b4b7113fd in connection_threadmain () No symbol table info available. #31 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so No symbol table info available. #32 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 No symbol table info available. #33 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 No symbol table info available. Thread 25 (Thread 0x7f1b0c7f8700 (LWP 105185)): #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 No symbol table info available. #1 0x00007f1b4941b050 in PR_WaitCondVar () from /lib64/libnspr4.so No symbol table info available. #2 0x00007f1b4b70efee in connection_wait_for_new_work () No symbol table info available. #3 0x00007f1b4b710719 in connection_threadmain () No symbol table info available. #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so No symbol table info available. #5 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 No symbol table info available. #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 No symbol table info available. Thread 24 (Thread 0x7f1b0cff9700 (LWP 105184)): #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 No symbol table info available. #1 0x00007f1b4941b050 in PR_WaitCondVar () from /lib64/libnspr4.so No symbol table info available. #2 0x00007f1b4b70efee in connection_wait_for_new_work () No symbol table info available. #3 0x00007f1b4b710719 in connection_threadmain () No symbol table info available. #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so No symbol table info available. #5 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 No symbol table info available. #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 No symbol table info available. Thread 23 (Thread 0x7f1b0d7fa700 (LWP 105183)): #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 No symbol table info available. #1 0x00007f1b4941b050 in PR_WaitCondVar () from /lib64/libnspr4.so No symbol table info available. #2 0x00007f1b4b70efee in connection_wait_for_new_work () No symbol table info available. #3 0x00007f1b4b710719 in connection_threadmain () No symbol table info available. #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so No symbol table info available. #5 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 No symbol table info available. #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 No symbol table info available. Thread 22 (Thread 0x7f1b0dffb700 (LWP 105182)): #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 No symbol table info available. #1 0x00007f1b4941b050 in PR_WaitCondVar () from /lib64/libnspr4.so No symbol table info available. #2 0x00007f1b4b70efee in connection_wait_for_new_work () No symbol table info available. #3 0x00007f1b4b710719 in connection_threadmain () No symbol table info available. #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so No symbol table info available. #5 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 No symbol table info available. #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 No symbol table info available. Thread 21 (Thread 0x7f1b0e7fc700 (LWP 105181)): #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 No symbol table info available. #1 0x00007f1b4941b050 in PR_WaitCondVar () from /lib64/libnspr4.so No symbol table info available. #2 0x00007f1b4b70efee in connection_wait_for_new_work () No symbol table info available. #3 0x00007f1b4b710719 in connection_threadmain () No symbol table info available. #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so No symbol table info available. #5 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 No symbol table info available. #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 No symbol table info available. Thread 20 (Thread 0x7f1b0effd700 (LWP 105180)): #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 No symbol table info available. #1 0x00007f1b433212f3 in __db_hybrid_mutex_suspend () from /lib64/ libdb-5.3.so No symbol table info available. #2 0x00007f1b43320b37 in __db_tas_mutex_readlock () from /lib64/ libdb-5.3.so No symbol table info available. #3 0x00007f1b4343ad35 in __memp_fget () from /lib64/libdb-5.3.so No symbol table info available. #4 0x00007f1b433e4333 in __dbc_iget () from /lib64/libdb-5.3.so No symbol table info available. #5 0x00007f1b433f2b02 in __dbc_get_pp () from /lib64/libdb-5.3.so No symbol table info available. #6 0x00007f1b3e8ce822 in idl_new_fetch () from /usr/lib64/dirsrv/plugins/libback-ldbm.so No symbol table info available. #7 0x00007f1b3e8dcb26 in index_read_ext_allids () from /usr/lib64/dirsrv/plugins/libback-ldbm.so No symbol table info available. #8 0x00007f1b3e8c7284 in keys2idl () from /usr/lib64/dirsrv/plugins/libback-ldbm.so No symbol table info available. #9 0x00007f1b3e8c79e3 in ava_candidates.isra.0 () from /usr/lib64/dirsrv/plugins/libback-ldbm.so No symbol table info available. #10 0x00007f1b3e8c7fd2 in filter_candidates_ext () from /usr/lib64/dirsrv/plugins/libback-ldbm.so No symbol table info available. #11 0x00007f1b3e8c9096 in list_candidates () from /usr/lib64/dirsrv/plugins/libback-ldbm.so No symbol table info available. #12 0x00007f1b3e8c7f40 in filter_candidates_ext () from /usr/lib64/dirsrv/plugins/libback-ldbm.so No symbol table info available. #13 0x00007f1b3e8c9096 in list_candidates () from /usr/lib64/dirsrv/plugins/libback-ldbm.so No symbol table info available. #14 0x00007f1b3e8c7f40 in filter_candidates_ext () from /usr/lib64/dirsrv/plugins/libback-ldbm.so No symbol table info available. #15 0x00007f1b3e8c9096 in list_candidates () from /usr/lib64/dirsrv/plugins/libback-ldbm.so No symbol table info available. #16 0x00007f1b3e8c7f40 in filter_candidates_ext () from /usr/lib64/dirsrv/plugins/libback-ldbm.so No symbol table info available. #17 0x00007f1b3e90380c in subtree_candidates () from /usr/lib64/dirsrv/plugins/libback-ldbm.so No symbol table info available. #18 0x00007f1b3e904e7f in ldbm_back_search () from /usr/lib64/dirsrv/plugins/libback-ldbm.so No symbol table info available. #19 0x00007f1b4b2321d6 in op_shared_search () from /usr/lib64/dirsrv/libslapd.so.0 No symbol table info available. #20 0x00007f1b4b72333e in do_search () No symbol table info available. #21 0x00007f1b4b7113c3 in connection_threadmain () No symbol table info available. #22 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so No symbol table info available. #23 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 No symbol table info available. #24 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 No symbol table info available. Thread 19 (Thread 0x7f1b0f7fe700 (LWP 105179)): #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 No symbol table info available. #1 0x00007f1b433212f3 in __db_hybrid_mutex_suspend () from /lib64/ libdb-5.3.so No symbol table info available. #2 0x00007f1b43320640 in __db_tas_mutex_lock () from /lib64/libdb-5.3.so No symbol table info available. #3 0x00007f1b433cad3a in __lock_get_internal () from /lib64/libdb-5.3.so No symbol table info available. #4 0x00007f1b433cb820 in __lock_get () from /lib64/libdb-5.3.so No symbol table info available. #5 0x00007f1b433f7142 in __db_lget () from /lib64/libdb-5.3.so No symbol table info available. #6 0x00007f1b4333e605 in __bam_search () from /lib64/libdb-5.3.so No symbol table info available. #7 0x00007f1b43329256 in __bamc_search () from /lib64/libdb-5.3.so No symbol table info available. #8 0x00007f1b4332ad0f in __bamc_get () from /lib64/libdb-5.3.so No symbol table info available. #9 0x00007f1b433e3ca6 in __dbc_iget () from /lib64/libdb-5.3.so No symbol table info available. #10 0x00007f1b433f2b02 in __dbc_get_pp () from /lib64/libdb-5.3.so No symbol table info available. #11 0x00007f1b3e8ce680 in idl_new_fetch () from /usr/lib64/dirsrv/plugins/libback-ldbm.so No symbol table info available. #12 0x00007f1b3e8dcb26 in index_read_ext_allids () from /usr/lib64/dirsrv/plugins/libback-ldbm.so No symbol table info available. #13 0x00007f1b3e8c7284 in keys2idl () from /usr/lib64/dirsrv/plugins/libback-ldbm.so No symbol table info available. #14 0x00007f1b3e8c79e3 in ava_candidates.isra.0 () from /usr/lib64/dirsrv/plugins/libback-ldbm.so No symbol table info available. #15 0x00007f1b3e8c7fd2 in filter_candidates_ext () from /usr/lib64/dirsrv/plugins/libback-ldbm.so No symbol table info available. #16 0x00007f1b3e8c9096 in list_candidates () from /usr/lib64/dirsrv/plugins/libback-ldbm.so No symbol table info available. #17 0x00007f1b3e8c7f40 in filter_candidates_ext () from /usr/lib64/dirsrv/plugins/libback-ldbm.so No symbol table info available. #18 0x00007f1b3e8c9096 in list_candidates () from /usr/lib64/dirsrv/plugins/libback-ldbm.so No symbol table info available. #19 0x00007f1b3e8c7f40 in filter_candidates_ext () from /usr/lib64/dirsrv/plugins/libback-ldbm.so No symbol table info available. #20 0x00007f1b3e8c9096 in list_candidates () from /usr/lib64/dirsrv/plugins/libback-ldbm.so No symbol table info available. #21 0x00007f1b3e8c7f40 in filter_candidates_ext () from /usr/lib64/dirsrv/plugins/libback-ldbm.so No symbol table info available. #22 0x00007f1b3e90380c in subtree_candidates () from /usr/lib64/dirsrv/plugins/libback-ldbm.so No symbol table info available. #23 0x00007f1b3e904e7f in ldbm_back_search () from /usr/lib64/dirsrv/plugins/libback-ldbm.so No symbol table info available. #24 0x00007f1b4b2321d6 in op_shared_search () from /usr/lib64/dirsrv/libslapd.so.0 No symbol table info available. #25 0x00007f1b4b72333e in do_search () No symbol table info available. #26 0x00007f1b4b7113c3 in connection_threadmain () No symbol table info available. #27 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so No symbol table info available. #28 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 No symbol table info available. #29 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 No symbol table info available. Thread 18 (Thread 0x7f1b0ffff700 (LWP 105178)): #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 No symbol table info available. #1 0x00007f1b4941b050 in PR_WaitCondVar () from /lib64/libnspr4.so No symbol table info available. #2 0x00007f1b4b70efee in connection_wait_for_new_work () No symbol table info available. #3 0x00007f1b4b710719 in connection_threadmain () No symbol table info available. #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so No symbol table info available. #5 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 No symbol table info available. #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 No symbol table info available. Thread 17 (Thread 0x7f1b14d28700 (LWP 105177)): #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 No symbol table info available. #1 0x00007f1b4941b050 in PR_WaitCondVar () from /lib64/libnspr4.so No symbol table info available. #2 0x00007f1b4b70efee in connection_wait_for_new_work () No symbol table info available. #3 0x00007f1b4b710719 in connection_threadmain () No symbol table info available. #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so No symbol table info available. #5 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 No symbol table info available. #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 No symbol table info available. Thread 16 (Thread 0x7f1b15529700 (LWP 105176)): #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 No symbol table info available. #1 0x00007f1b4941b050 in PR_WaitCondVar () from /lib64/libnspr4.so No symbol table info available. #2 0x00007f1b4b70efee in connection_wait_for_new_work () No symbol table info available. #3 0x00007f1b4b710719 in connection_threadmain () No symbol table info available. #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so No symbol table info available. #5 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 No symbol table info available. #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 No symbol table info available. Thread 15 (Thread 0x7f1b15d2a700 (LWP 105175)): #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 No symbol table info available. #1 0x00007f1b4941b050 in PR_WaitCondVar () from /lib64/libnspr4.so No symbol table info available. #2 0x00007f1b4b70efee in connection_wait_for_new_work () No symbol table info available. #3 0x00007f1b4b710719 in connection_threadmain () No symbol table info available. #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so No symbol table info available. #5 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 No symbol table info available. #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 No symbol table info available. Thread 14 (Thread 0x7f1b1652b700 (LWP 105174)): #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 No symbol table info available. #1 0x00007f1b4941b050 in PR_WaitCondVar () from /lib64/libnspr4.so No symbol table info available. #2 0x00007f1b4b70efee in connection_wait_for_new_work () No symbol table info available. #3 0x00007f1b4b710719 in connection_threadmain () No symbol table info available. #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so No symbol table info available. #5 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 No symbol table info available. #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 No symbol table info available. Thread 13 (Thread 0x7f1b16d2c700 (LWP 105173)): #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 No symbol table info available. #1 0x00007f1b433212f3 in __db_hybrid_mutex_suspend () from /lib64/ libdb-5.3.so No symbol table info available. #2 0x00007f1b43320640 in __db_tas_mutex_lock () from /lib64/libdb-5.3.so No symbol table info available. #3 0x00007f1b433cad3a in __lock_get_internal () from /lib64/libdb-5.3.so No symbol table info available. #4 0x00007f1b433cb820 in __lock_get () from /lib64/libdb-5.3.so No symbol table info available. #5 0x00007f1b433f7142 in __db_lget () from /lib64/libdb-5.3.so No symbol table info available. #6 0x00007f1b4333e605 in __bam_search () from /lib64/libdb-5.3.so No symbol table info available. #7 0x00007f1b43329256 in __bamc_search () from /lib64/libdb-5.3.so No symbol table info available. #8 0x00007f1b4332ad0f in __bamc_get () from /lib64/libdb-5.3.so No symbol table info available. #9 0x00007f1b433e3ca6 in __dbc_iget () from /lib64/libdb-5.3.so No symbol table info available. #10 0x00007f1b433f2b02 in __dbc_get_pp () from /lib64/libdb-5.3.so No symbol table info available. #11 0x00007f1b3e8ce680 in idl_new_fetch () from /usr/lib64/dirsrv/plugins/libback-ldbm.so No symbol table info available. #12 0x00007f1b3e8dcb26 in index_read_ext_allids () from /usr/lib64/dirsrv/plugins/libback-ldbm.so No symbol table info available. #13 0x00007f1b3e8c7284 in keys2idl () from /usr/lib64/dirsrv/plugins/libback-ldbm.so No symbol table info available. #14 0x00007f1b3e8c79e3 in ava_candidates.isra.0 () from /usr/lib64/dirsrv/plugins/libback-ldbm.so No symbol table info available. #15 0x00007f1b3e8c7fd2 in filter_candidates_ext () from /usr/lib64/dirsrv/plugins/libback-ldbm.so No symbol table info available. #16 0x00007f1b3e8c9096 in list_candidates () from /usr/lib64/dirsrv/plugins/libback-ldbm.so No symbol table info available. #17 0x00007f1b3e8c7f40 in filter_candidates_ext () from /usr/lib64/dirsrv/plugins/libback-ldbm.so No symbol table info available. #18 0x00007f1b3e8c9096 in list_candidates () from /usr/lib64/dirsrv/plugins/libback-ldbm.so No symbol table info available. #19 0x00007f1b3e8c7f40 in filter_candidates_ext () from /usr/lib64/dirsrv/plugins/libback-ldbm.so No symbol table info available. #20 0x00007f1b3e8c9096 in list_candidates () from /usr/lib64/dirsrv/plugins/libback-ldbm.so No symbol table info available. #21 0x00007f1b3e8c7f40 in filter_candidates_ext () from /usr/lib64/dirsrv/plugins/libback-ldbm.so No symbol table info available. #22 0x00007f1b3e90380c in subtree_candidates () from /usr/lib64/dirsrv/plugins/libback-ldbm.so No symbol table info available. #23 0x00007f1b3e904e7f in ldbm_back_search () from /usr/lib64/dirsrv/plugins/libback-ldbm.so No symbol table info available. #24 0x00007f1b4b2321d6 in op_shared_search () from /usr/lib64/dirsrv/libslapd.so.0 No symbol table info available. #25 0x00007f1b4b72333e in do_search () No symbol table info available. #26 0x00007f1b4b7113c3 in connection_threadmain () No symbol table info available. #27 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so No symbol table info available. #28 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 No symbol table info available. #29 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 No symbol table info available. Thread 12 (Thread 0x7f1b1752d700 (LWP 105172)): #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 No symbol table info available. #1 0x00007f1b4941b050 in PR_WaitCondVar () from /lib64/libnspr4.so No symbol table info available. #2 0x00007f1b4b70efee in connection_wait_for_new_work () No symbol table info available. #3 0x00007f1b4b710719 in connection_threadmain () No symbol table info available. #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so No symbol table info available. #5 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 No symbol table info available. #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 No symbol table info available. Thread 11 (Thread 0x7f1b17d2e700 (LWP 105171)): #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 No symbol table info available. #1 0x00007f1b4941b050 in PR_WaitCondVar () from /lib64/libnspr4.so No symbol table info available. #2 0x00007f1b4b70efee in connection_wait_for_new_work () No symbol table info available. #3 0x00007f1b4b710719 in connection_threadmain () No symbol table info available. #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so No symbol table info available. #5 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 No symbol table info available. #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 No symbol table info available. Thread 10 (Thread 0x7f1b2509d700 (LWP 105168)): #0 0x00007f1b48dc5a82 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 No symbol table info available. #1 0x00007f1b4941ab07 in pt_TimedWait () from /lib64/libnspr4.so No symbol table info available. #2 0x00007f1b4941afce in PR_WaitCondVar () from /lib64/libnspr4.so No symbol table info available. #3 0x00007f1b4b2061e8 in eq_loop () from /usr/lib64/dirsrv/libslapd.so.0 No symbol table info available. #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so No symbol table info available. #5 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 No symbol table info available. #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 No symbol table info available. Thread 9 (Thread 0x7f1b2589e700 (LWP 105167)): #0 0x00007f1b48dc5a82 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 No symbol table info available. #1 0x00007f1b4941ab07 in pt_TimedWait () from /lib64/libnspr4.so No symbol table info available. #2 0x00007f1b4941afce in PR_WaitCondVar () from /lib64/libnspr4.so No symbol table info available. #3 0x00007f1b4b719d03 in housecleaning () No symbol table info available. #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so No symbol table info available. #5 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 No symbol table info available. #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 No symbol table info available. Thread 8 (Thread 0x7f1b2609f700 (LWP 105166)): #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 No symbol table info available. #1 0x00007f1b4941b050 in PR_WaitCondVar () from /lib64/libnspr4.so No symbol table info available. #2 0x00007f1b4b260198 in slapi_wait_condvar () from /usr/lib64/dirsrv/libslapd.so.0 No symbol table info available. #3 0x00007f1b3cf5deed in roles_cache_wait_on_change () from /usr/lib64/dirsrv/plugins/libroles-plugin.so No symbol table info available. #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so No symbol table info available. #5 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 No symbol table info available. #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 No symbol table info available. Thread 7 (Thread 0x7f1b268a0700 (LWP 105165)): #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 No symbol table info available. #1 0x00007f1b4941b050 in PR_WaitCondVar () from /lib64/libnspr4.so No symbol table info available. #2 0x00007f1b4b260198 in slapi_wait_condvar () from /usr/lib64/dirsrv/libslapd.so.0 No symbol table info available. #3 0x00007f1b3cf5deed in roles_cache_wait_on_change () from /usr/lib64/dirsrv/plugins/libroles-plugin.so No symbol table info available. #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so No symbol table info available. #5 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 No symbol table info available. #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 No symbol table info available. Thread 6 (Thread 0x7f1b270a1700 (LWP 105164)): #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 No symbol table info available. #1 0x00007f1b4941b050 in PR_WaitCondVar () from /lib64/libnspr4.so No symbol table info available. #2 0x00007f1b4b260198 in slapi_wait_condvar () from /usr/lib64/dirsrv/libslapd.so.0 No symbol table info available. #3 0x00007f1b3cf5deed in roles_cache_wait_on_change () from /usr/lib64/dirsrv/plugins/libroles-plugin.so No symbol table info available. #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so No symbol table info available. #5 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 No symbol table info available. #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 No symbol table info available. Thread 5 (Thread 0x7f1b278a2700 (LWP 105163)): #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 No symbol table info available. #1 0x00007f1b4941b050 in PR_WaitCondVar () from /lib64/libnspr4.so No symbol table info available. #2 0x00007f1b4b260198 in slapi_wait_condvar () from /usr/lib64/dirsrv/libslapd.so.0 No symbol table info available. #3 0x00007f1b415f262e in cos_cache_wait_on_change () from /usr/lib64/dirsrv/plugins/libcos-plugin.so No symbol table info available. #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so No symbol table info available. #5 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 No symbol table info available. #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 No symbol table info available. Thread 4 (Thread 0x7f1b282ac700 (LWP 105162)): #0 0x00007f1b48ae69b3 in select () from /lib64/libc.so.6 No symbol table info available. #1 0x00007f1b4b2710e9 in DS_Sleep () from /usr/lib64/dirsrv/libslapd.so.0 No symbol table info available. #2 0x00007f1b3e8ba707 in perf_threadmain () from /usr/lib64/dirsrv/plugins/libback-ldbm.so No symbol table info available. #3 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so No symbol table info available. #4 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 No symbol table info available. #5 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 No symbol table info available. Thread 3 (Thread 0x7f1b28aad700 (LWP 105161)): #0 0x00007f1b48ae69b3 in select () from /lib64/libc.so.6 No symbol table info available. #1 0x00007f1b4b2710e9 in DS_Sleep () from /usr/lib64/dirsrv/libslapd.so.0 No symbol table info available. #2 0x00007f1b3e8bfb7f in trickle_threadmain () from /usr/lib64/dirsrv/plugins/libback-ldbm.so No symbol table info available. #3 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so No symbol table info available. #4 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 No symbol table info available. #5 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 No symbol table info available. Thread 2 (Thread 0x7f1b292ae700 (LWP 105160)): #0 0x00007f1b48ae69b3 in select () from /lib64/libc.so.6 No symbol table info available. #1 0x00007f1b4344dbad in __os_yield () from /lib64/libdb-5.3.so No symbol table info available. #2 0x00007f1b434492b3 in __memp_sync_int () from /lib64/libdb-5.3.so No symbol table info available. #3 0x00007f1b43459752 in __txn_checkpoint () from /lib64/libdb-5.3.so No symbol table info available. #4 0x00007f1b43459b74 in __txn_checkpoint_pp () from /lib64/libdb-5.3.so No symbol table info available. #5 0x00007f1b3e8c3a87 in checkpoint_threadmain () from /usr/lib64/dirsrv/plugins/libback-ldbm.so No symbol table info available. #6 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so No symbol table info available. #7 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 No symbol table info available. #8 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 No symbol table info available. Thread 1 (Thread 0x7f1b29aaf700 (LWP 105159)): #0 0x00007f1b48ae69b3 in select () from /lib64/libc.so.6 No symbol table info available. #1 0x00007f1b4b2710e9 in DS_Sleep () from /usr/lib64/dirsrv/libslapd.so.0 No symbol table info available. #2 0x00007f1b3e8bf907 in deadlock_threadmain () from /usr/lib64/dirsrv/plugins/libback-ldbm.so No symbol table info available. #3 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so No symbol table info available. #4 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 No symbol table info available. #5 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 No symbol table info available. Thanks, Rakesh On Mon, Sep 5, 2016 at 4:48 PM, thierry bordaz wrote: > > > On 09/05/2016 12:05 PM, Rakesh Rajasekharan wrote: > > Hi Thierry, > > > I was getting the hang issue while running ipa-client-install > simultaneously on few clients.. > However, today, I am not able to replicate that. > > I could not get a gdb . But i will try getting that the next time I face > this issue. > > The CPU does not stay high.. it just momentarily touches a high value and > then drops down to around 2-7% > > One question I have is , is it ok to set it nsslapd-threadnumber to a very > high value . > I have around 4000 clients and with nsslapd-maxthreadsperconn set to > 5..So, can I set nsslapd-threadnumber to around 25000. > > > Hello, > > I know some users running in production with several hunderds of threads > (>600) and this without problem. > > I do not recall having suggested to increase that number and for what > reason. > Usually 30 workers is a good enough value. It can create bootleneck if for > some reason each operation is very long to satisfy and exhaust the number > of workers. You can monitor the work queue: > > ldapsearch -D "cn=directory manager" -w xxx -LLL -b "cn=monitor" -s base > opsinitiated opscompleted > > > If opscompleted-opsinitiated remains close to threadnumber, then yes it > would be valuable to increase it. > > The computation #client * #async_op_per_client sound an overkill. Even if > all clients send at the exact same time all their requests, it is very > likely that some common resource (db page, log, allocator...) will > serialize them. If you monitor a need to increase the work, you would for > example set it to 50, then monitor, then set it to 100, then monitor... > until you find a good enough value. > Note the increasing the #thread, increases the memory footprint that will > reduce the efficiency of file system cache and can increase the response > time. > > > best regards > thierry > > > Thanks > > On Mon, Sep 5, 2016 at 1:03 PM, thierry bordaz wrote: > >> >> Hi Rakesh, >> >> Were you able to get a pstack or full stack with gdb ( >> >> http://www.port389.org/docs/389ds/FAQ/faq.html#debugging-crashes) when >> the server hangs ? >> >> If it happens with 500 threads as well as with 30, using 30 threads is a >> better choice to debug this issue. >> I will try to reproduce using 150 parallel 'ipa user-find p-testipa' >> commands >> >> Something I am unsure is if the CPU consumption stays high (you mentioned >> 340% CPU usage) as long as the hang happens or if after a suddent shot up >> to 340% (that marks the beginning of the hang) it drops and stay hanging ? >> >> thanks >> thierry >> >> On 09/04/2016 08:40 PM, Rakesh Rajasekharan wrote: >> >> starce on the slapd process actually had this in the output.. >> FUTEX_WAIT_PRIVATE >> >> and checking for the number of threads slapd had.. there were 5015 threads >> >> ps -efL|grep slapd|wc -l >> 5015 >> >> strace on most of the threads gave this output >> >> strace -p 67411 >> Process 67411 attached >> futex(0x7f3f0226b41c, FUTEX_WAIT_PRIVATE, 1, NULL) = -1 EAGAIN (Resource >> temporarily unavailable) >> futex(0x7f3f0226b41c, FUTEX_WAIT_PRIVATE, 2, NULL^CProcess 67411 detached >> >> >> >> >> >> On Sun, Sep 4, 2016 at 5:34 PM, Rakesh Rajasekharan < >> rakesh.rajasekharan at gmail.com> wrote: >> >>> I have again got the issue of IPA hanging.. The issue came up when i >>> tried to run ipa-client-isntall on 142 clients simultaneously >>> >>> >>> None of the IPA commands are responding, and I see this error >>> >>> ipa user-find p-testipa >>> ipa: ERROR: Insufficient access: SASL(-1): generic failure: GSSAPI >>> Error: Unspecified GSS failure. Minor code may provide more information >>> (KDC returned error string: PROCESS_TGS) >>> >>> KRB5_TRACE=/dev/stdout kinit admin >>> [41178] 1472984115.233214: Getting initial credentials for >>> admin at XYZ.COM >>> [41178] 1472984115.235257: Sending request (167 bytes) to XYZ.COM >>> [41178] 1472984115.235419: Initiating TCP connection to stream >>> 10.1.3.36:88 >>> [41178] 1472984115.235685: Sending TCP request to stream 10.1.3.36:88 >>> [41178] 1472984120.238914: Received answer (174 bytes) from stream >>> 10.1.3.36:88 >>> [41178] 1472984120.238925: Terminating TCP connection to stream >>> 10.1.3.36:88 >>> [41178] 1472984120.238993: Response was from master KDC >>> [41 >>> >>> >>> Running an ldapsearch to see the db.. does not give any results and just >>> hangs there >>> >>> ldapsearch -x -D 'cn=Directory Manager' -W -s one -b >>> 'cn=kerberos,dc=xyz,dc=com' >>> Enter LDAP Password: >>> >>> even an ldapsearch -x does not respond >>> At this point, am sure that slapd is the one causing issues >>> >>> Running an strace against the hung slapd itself seems to get stuck does >>> not proceed after saying "attaching to process" >>> >>> From some others posts I read Thierry suggesting to increase the >>> nsslapd-threadnumber value >>> >>> It was set to 30, I think that might be too low. >>> >>> I have raised it to 500 >>> >>> Now after restarting the service .. ldapsearch starts responding. >>> But running the test to add a sudden high number of clients again left >>> ns-slapd to hung state >>> >>> When i attempted adding the clients.. the ns-slapd cpu usage shot up to >>> 340% and after that ns-slapd stopped responding >>> >>> So now, atleast I know what might be causing the issue and I can now >>> easily reproduce it. >>> >>> Is there a way I can make ns-slapd handle a sudden bump in incoming >>> request for ipa-client-install >>> >>> Thanks >>> Rakesh >>> >>> >>> >>> >>> >>> >>> On Mon, Aug 29, 2016 at 11:18 PM, Rich Megginson < >>> rmeggins at redhat.com> wrote: >>> >>>> On 08/29/2016 10:53 AM, Rakesh Rajasekharan wrote: >>>> >>>> Hi Thierry, >>>> >>>> My machine has 30GB RAM ..and 389-ds version is 1.3.4 >>>> >>>> ldapsearch shows the values for nsslapd-cachememsize updated to 200MB. >>>> >>>> ldapsearch -LLL -o ldif-wrap=no -D "cn=directory manager" -w >>>> 'mypassword' -b 'cn=userRoot,cn=ldbm database,cn=plugins,cn=config'|grep >>>> nsslapd-cachememsize >>>> nsslapd-cachememsize: 209715200 >>>> >>>> >>>> So, it seems to have updated though seeing that warning(WARNING: ipaca: >>>> entry cache size 10485760B is less than db size 11599872B) in the log >>>> confuses me a bit. >>>> >>>> Thers one more entry that I found from the ldapsearch to be bit low >>>> >>>> nsslapd-dncachememsize: 10485760 >>>> maxdncachesize: 10485760 >>>> >>>> Should I update these as well to a higher value >>>> >>>> At the time when the issue happened, the memory usage as well as the >>>> overall load of the system was very low . >>>> I will try reproducing the issue atleast in my QA env..probably by >>>> trying to mock simultaneous parallel logins to a large number of hosts >>>> >>>> >>>> To monitor your cache sizes, please use the dbmon.sh tool provided with >>>> your distro. If that is not available with your particular distro, see >>>> https://github.com/richm/scripts/wiki/dbmon.sh >>>> >>>> >>>> >>>> >>>> thanks >>>> Rakesh >>>> >>>> >>>> >>>> >>>> On Mon, Aug 29, 2016 at 8:16 PM, thierry bordaz < >>>> tbordaz at redhat.com> wrote: >>>> >>>>> Hi Rakesh, >>>>> >>>>> Those tuning may depend on the memory available on your machine. >>>>> nsslapd-cachememsize allows the entry cache to consume up to 200Mb but >>>>> its memory footprint is known to go above. >>>>> 200Mb both looks pretty good to me. How large is your machine ? What >>>>> is your version of 389-ds ? >>>>> >>>>> Those warnings do not change your settings. It just raise that entry >>>>> cache of 'ipaca' and 'retrocl' are small but it is fine. The size of the >>>>> entry cache is important mostly in userRoot. >>>>> You may double check the actual values, after restart, with ldapsearch >>>>> on 'cn=userRoot,cn=ldbm database,cn=plugins,cn=config' and >>>>> 'cn=config,cn=ldbm database,cn=plugins,cn=config'. >>>>> >>>>> A step is to know what will be response time of DS to know if it is >>>>> responsible of the hang or not. >>>>> The logs and possibly pstack during those intermittent hangs will help >>>>> to determine that. >>>>> >>>>> regards >>>>> thierry >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> On 08/29/2016 04:25 PM, Rakesh Rajasekharan wrote: >>>>> >>>>> I tried increasing the nsslapd-dbcachesize and nsslapd-cachememsize in >>>>> my QA envs to 200MB. >>>>> >>>>> However, in my log files, I still see this message >>>>> [29/Aug/2016:04:34:37 +0000] - WARNING: ipaca: entry cache size >>>>> 10485760B is less than db size 11599872B; We recommend to increase the >>>>> entry cache size nsslapd-cachememsize. >>>>> [29/Aug/2016:04:34:37 +0000] - WARNING: changelog: entry cache size >>>>> 2097152B is less than db size 441647104B; We recommend to increase the >>>>> entry cache size nsslapd-cachememsize. >>>>> >>>>> these are my ldif files that i used to modify the values >>>>> modify entry cache size >>>>> cat modify-cache-mem-size.ldif >>>>> dn: cn=userRoot,cn=ldbm database,cn=plugins,cn=config >>>>> changetype: modify >>>>> replace: nsslapd-cachememsize >>>>> nsslapd-cachememsize: 209715200 >>>>> >>>>> modify db cache size >>>>> cat modfy-db-cache-size.ldif >>>>> dn: cn=config,cn=ldbm database,cn=plugins,cn=config >>>>> changetype: modify >>>>> replace: nsslapd-dbcachesize >>>>> nsslapd-dbcachesize: 209715200 >>>>> >>>>> After modifying , i restarted IPA services >>>>> >>>>> Is there anything else that I need to take care of as the logs >>>>> suggest its still not getting the updated values >>>>> >>>>> Thanks >>>>> Rakesh >>>>> >>>>> On Mon, Aug 29, 2016 at 6:07 PM, Rakesh Rajasekharan < >>>>> rakesh.rajasekharan at gmail.com> wrote: >>>>> >>>>>> Hi Thierry, >>>>>> >>>>>> Coz of the issues we had to revert back to earlier running openldap >>>>>> in production. >>>>>> >>>>>> I have now done a few TCP related changes in sysctl.conf and have >>>>>> also increased the nsslapd-dbcachesize and nsslapd-cachememsize to 200MB >>>>>> >>>>>> I will again start migrating hosts back to IPA and see if I face the >>>>>> earlier issue. >>>>>> >>>>>> I will update back once I have something >>>>>> >>>>>> >>>>>> Thanks, >>>>>> Rakesh >>>>>> >>>>>> >>>>>> >>>>>> On Thu, Aug 25, 2016 at 2:17 PM, thierry bordaz < >>>>>> tbordaz at redhat.com> wrote: >>>>>> >>>>>>> >>>>>>> >>>>>>> On 08/25/2016 10:15 AM, Rakesh Rajasekharan wrote: >>>>>>> >>>>>>> All of the troubleshooting seems fine. >>>>>>> >>>>>>> >>>>>>> However, Running libconv.pl gives me this output >>>>>>> >>>>>>> ----- Recommendations ----- >>>>>>> >>>>>>> 1. You have unindexed components, this can be caused from a search >>>>>>> on an unindexed attribute, or your returned results exceeded the >>>>>>> allidsthreshold. Unindexed components are not recommended. To refuse >>>>>>> unindexed searches, switch 'nsslapd-require-index' to 'on' under your >>>>>>> database entry (e.g. cn=UserRoot,cn=ldbm database,cn=plugins,cn=config) >>>>>>> . >>>>>>> >>>>>>> 2. You have a significant difference between binds and unbinds. >>>>>>> You may want to investigate this difference. >>>>>>> >>>>>>> >>>>>>> I feel, this could be a pointer to things going slow.. and IPA >>>>>>> hanging. I think i now have something that I can try and nail down this >>>>>>> issue. >>>>>>> >>>>>>> On a sidenote, I was earlier running openldap and migrated over to >>>>>>> Freeipa, >>>>>>> >>>>>>> Thanks >>>>>>> Rakesh >>>>>>> >>>>>>> >>>>>>> >>>>>>> On Wed, Aug 24, 2016 at 12:38 PM, Petr Spacek < >>>>>>> pspacek at redhat.com> wrote: >>>>>>> >>>>>>>> On 23.8.2016 18:44, Rakesh Rajasekharan wrote: >>>>>>>> > I think thers something seriously wrong with my system >>>>>>>> > >>>>>>>> > not able to run any IPA commands >>>>>>>> > >>>>>>>> > klist >>>>>>>> > Ticket cache: KEYRING:persistent:0:0 >>>>>>>> > Default principal: admin at XYZ.COM >>>>>>>> > >>>>>>>> > Valid starting Expires Service principal >>>>>>>> > 2016-08-23T16:26:36 2016-08-24T16:26:22 krbtgt/ >>>>>>>> XYZ.COM at XYZ.COM >>>>>>>> > >>>>>>>> > >>>>>>>> > [root at prod-ipa-master-1a :~] ipactl status >>>>>>>> > Directory Service: RUNNING >>>>>>>> > krb5kdc Service: RUNNING >>>>>>>> > kadmin Service: RUNNING >>>>>>>> > ipa_memcached Service: RUNNING >>>>>>>> > httpd Service: RUNNING >>>>>>>> > pki-tomcatd Service: RUNNING >>>>>>>> > ipa-otpd Service: RUNNING >>>>>>>> > ipa: INFO: The ipactl command was successful >>>>>>>> > >>>>>>>> > >>>>>>>> > >>>>>>>> > [root at prod-ipa-master :~] ipa user-find p-testuser >>>>>>>> > ipa: ERROR: Kerberos error: ('Unspecified GSS failure. Minor >>>>>>>> code may >>>>>>>> > provide more information', 851968)/("Cannot contact any KDC for >>>>>>>> realm ' >>>>>>>> > XYZ.COM'", -1765328228) >>>>>>>> >>>>>>> >>>>>>> Hi Rakesh, >>>>>>> >>>>>>> Having a reproducible test case would you rerun the command above. >>>>>>> During its processing you may monitor DS process load (top). If it >>>>>>> is high, you may get some pstacks of it. >>>>>>> Also would you attach the part of DS access logs taken during the >>>>>>> command. >>>>>>> >>>>>>> regards >>>>>>> thierry >>>>>>> >>>>>>> > >>>>>>>> >>>>>>>> This is weird because the server seems to be up. >>>>>>>> >>>>>>>> Please follow >>>>>>>> http://www.freeipa.org/page/Tr >>>>>>>> oubleshooting#Authentication.2FKerberos >>>>>>>> >>>>>>>> Petr^2 Spacek >>>>>>>> >>>>>>>> > >>>>>>>> > >>>>>>>> > Thanks >>>>>>>> > >>>>>>>> > Rakesh >>>>>>>> > >>>>>>>> > On Tue, Aug 23, 2016 at 10:01 PM, Rakesh Rajasekharan < >>>>>>>> > rakesh.rajasekharan at gmail.com> >>>>>>>> wrote: >>>>>>>> > >>>>>>>> >> i changed the loggin level to 4 . Modifying >>>>>>>> nsslapd-accesslog-level >>>>>>>> >> >>>>>>>> >> But, the hang is still there. though I dont see the sigfault now >>>>>>>> >> >>>>>>>> >> >>>>>>>> >> >>>>>>>> >> >>>>>>>> >> On Tue, Aug 23, 2016 at 9:02 PM, Rakesh Rajasekharan < >>>>>>>> >> rakesh.rajasekharan at gmail.com> >>>>>>>> wrote: >>>>>>>> >> >>>>>>>> >>> My disk was getting filled too fast >>>>>>>> >>> >>>>>>>> >>> logs under /var/log/dirsrv was coming around 5 gb quickly >>>>>>>> filling up >>>>>>>> >>> >>>>>>>> >>> Is there a way to make the logging less verbose >>>>>>>> >>> >>>>>>>> >>> >>>>>>>> >>> >>>>>>>> >>> On Tue, Aug 23, 2016 at 6:41 PM, Petr Spacek < >>>>>>>> pspacek at redhat.com> wrote: >>>>>>>> >>> >>>>>>>> >>>> On 23.8.2016 15:07, Rakesh Rajasekharan wrote: >>>>>>>> >>>>> I was able to fix that may be temporarily... when i checked >>>>>>>> the >>>>>>>> >>>> network.. >>>>>>>> >>>>> there was another process that was running and consuming a >>>>>>>> lot of >>>>>>>> >>>> network ( >>>>>>>> >>>>> i have no idea who did that. I need to seriously start >>>>>>>> restricting >>>>>>>> >>>> people >>>>>>>> >>>>> access to this machine ) >>>>>>>> >>>>> >>>>>>>> >>>>> after killing that perfomance improved drastically >>>>>>>> >>>>> >>>>>>>> >>>>> But now, suddenly I started experiencing the same hang. >>>>>>>> >>>>> >>>>>>>> >>>>> This time , I gert the following error when checked dmesg >>>>>>>> >>>>> >>>>>>>> >>>>> [ 301.236976] ns-slapd[3124]: segfault at 0 ip >>>>>>>> 00007f1de416951c sp >>>>>>>> >>>>> 00007f1dee1dba70 error 4 in libcos-plugin.so[7f1de4166000+ >>>>>>>> b000] >>>>>>>> >>>>> [ 1116.248431] TCP: request_sock_TCP: Possible SYN flooding >>>>>>>> on port 88. >>>>>>>> >>>>> Sending cookies. Check SNMP counters. >>>>>>>> >>>>> [11831.397037] ns-slapd[22550]: segfault at 0 ip >>>>>>>> 00007f533d82251c sp >>>>>>>> >>>>> 00007f5347894a70 error 4 in libcos-plugin.so[7f533d81f000+ >>>>>>>> b000] >>>>>>>> >>>>> [11832.727989] ns-slapd[22606]: segfault at 0 ip >>>>>>>> 00007f6231eb951c sp >>>>>>>> >>>>> 00007f623bf2ba70 error 4 in libcos-plugin.so[7f6231eb6000+b00 >>>>>>>> >>>> >>>>>>>> >>>> Okay, this one is serious. The LDAP server crashed. >>>>>>>> >>>> >>>>>>>> >>>> 1. Make sure all your packages are up-to-date. >>>>>>>> >>>> >>>>>>>> >>>> Please see >>>>>>>> >>>> http://directory.fedoraproject >>>>>>>> .org/docs/389ds/FAQ/faq.html#d >>>>>>>> >>>> ebugging-crashes >>>>>>>> >>>> for further instructions how to debug this. >>>>>>>> >>>> >>>>>>>> >>>> Petr^2 Spacek >>>>>>>> >>>> >>>>>>>> >>>>> >>>>>>>> >>>>> and in /var/log/dirsrv/example-com/errors >>>>>>>> >>>>> >>>>>>>> >>>>> [23/Aug/2016:12:49:36 +0000] DSRetroclPlugin - >>>>>>>> delete_changerecord: >>>>>>>> >>>> could >>>>>>>> >>>>> not delete change record 3291138 (rc: 32) >>>>>>>> >>>>> [23/Aug/2016:12:49:36 +0000] DSRetroclPlugin - >>>>>>>> delete_changerecord: >>>>>>>> >>>> could >>>>>>>> >>>>> not delete change record 3291139 (rc: 32) >>>>>>>> >>>>> [23/Aug/2016:12:49:36 +0000] DSRetroclPlugin - >>>>>>>> delete_changerecord: >>>>>>>> >>>> could >>>>>>>> >>>>> not delete change record 3291140 (rc: 32) >>>>>>>> >>>>> [23/Aug/2016:12:49:36 +0000] DSRetroclPlugin - >>>>>>>> delete_changerecord: >>>>>>>> >>>> could >>>>>>>> >>>>> not delete change record 3291141 (rc: 32) >>>>>>>> >>>>> [23/Aug/2016:12:49:36 +0000] DSRetroclPlugin - >>>>>>>> delete_changerecord: >>>>>>>> >>>> could >>>>>>>> >>>>> not delete change record 3291142 (rc: 32) >>>>>>>> >>>>> [23/Aug/2016:12:49:36 +0000] DSRetroclPlugin - >>>>>>>> delete_changerecord: >>>>>>>> >>>> could >>>>>>>> >>>>> not delete change record 3291143 (rc: 32) >>>>>>>> >>>>> [23/Aug/2016:12:49:36 +0000] DSRetroclPlugin - >>>>>>>> delete_changerecord: >>>>>>>> >>>> could >>>>>>>> >>>>> not delete change record 3291144 (rc: 32) >>>>>>>> >>>>> [23/Aug/2016:12:49:36 +0000] DSRetroclPlugin - >>>>>>>> delete_changerecord: >>>>>>>> >>>> could >>>>>>>> >>>>> not delete change record 3291145 (rc: 32) >>>>>>>> >>>>> [23/Aug/2016:12:49:50 +0000] - Retry count exceeded in delete >>>>>>>> >>>>> [23/Aug/2016:12:49:50 +0000] DSRetroclPlugin - >>>>>>>> delete_changerecord: >>>>>>>> >>>> could >>>>>>>> >>>>> not delete change record 3292734 (rc: 51) >>>>>>>> >>>>> >>>>>>>> >>>>> >>>>>>>> >>>>> Can i do something about this error.. I treid to restart ipa >>>>>>>> a couple >>>>>>>> >>>> of >>>>>>>> >>>>> time but that did not help >>>>>>>> >>>>> >>>>>>>> >>>>> Thanks >>>>>>>> >>>>> Rakesh >>>>>>>> >>>>> >>>>>>>> >>>>> On Mon, Aug 22, 2016 at 2:27 PM, Petr Spacek < >>>>>>>> pspacek at redhat.com> >>>>>>>> >>>> wrote: >>>>>>>> >>>>> >>>>>>>> >>>>>> On 19.8.2016 19:32, Rakesh Rajasekharan wrote: >>>>>>>> >>>>>>> I am running my set up on AWS cloud, and entropy is low at >>>>>>>> around >>>>>>>> >>>> 180 . >>>>>>>> >>>>>>> >>>>>>>> >>>>>>> I plan to increase it bu installing haveged . But, would >>>>>>>> low entropy >>>>>>>> >>>> by >>>>>>>> >>>>>> any >>>>>>>> >>>>>>> chance cause this issue of intermittent hang . >>>>>>>> >>>>>>> Also, the hang is mostly observed when registering around >>>>>>>> 20 clients >>>>>>>> >>>>>>> together >>>>>>>> >>>>>> >>>>>>>> >>>>>> Possibly, I'm not sure. If you want to dig into this, I >>>>>>>> would do this: >>>>>>>> >>>>>> 1. look what process hangs on client (using pstree command >>>>>>>> or so) >>>>>>>> >>>>>> $ pstree >>>>>>>> >>>>>> >>>>>>>> >>>>>> 2. look to what server and port is the hanging client >>>>>>>> connected to >>>>>>>> >>>>>> $ lsof -p >>>>>>>> >>>>>> >>>>>>>> >>>>>> 3. jump to server and see what process is bound to the >>>>>>>> target port >>>>>>>> >>>>>> $ netstat -pn >>>>>>>> >>>>>> >>>>>>>> >>>>>> 4. see where the process if hanging >>>>>>>> >>>>>> $ strace -p >>>>>>>> >>>>>> >>>>>>>> >>>>>> I hope it helps. >>>>>>>> >>>>>> >>>>>>>> >>>>>> Petr^2 Spacek >>>>>>>> >>>>>> >>>>>>>> >>>>>>> On Fri, Aug 19, 2016 at 7:24 PM, Rakesh Rajasekharan < >>>>>>>> >>>>>>> >>>>>>>> rakesh.rajasekharan at gmail.com> wrote: >>>>>>>> >>>>>>> >>>>>>>> >>>>>>>> yes there seems to be something thats worrying.. I have >>>>>>>> faced this >>>>>>>> >>>> today >>>>>>>> >>>>>>>> as well. >>>>>>>> >>>>>>>> There are few hosts around 280 odd left and when i try >>>>>>>> adding them >>>>>>>> >>>> to >>>>>>>> >>>>>> IPA >>>>>>>> >>>>>>>> , the slowness begins.. >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> all the ipa commands like ipa user-find.. etc becomes very >>>>>>>> slow in >>>>>>>> >>>>>>>> responding. >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> the SYNC_RECV are not many though just around 80-90 and >>>>>>>> today that >>>>>>>> >>>> was >>>>>>>> >>>>>>>> around 20 only >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> I have for now increased tcp_max_syn_backlog to 5000. >>>>>>>> >>>>>>>> For now the slowness seems to have gone.. but I will do a >>>>>>>> try >>>>>>>> >>>> adding the >>>>>>>> >>>>>>>> clients again tomorrow and see how it goes >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> Thanks >>>>>>>> >>>>>>>> Rakesh >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> The issues >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> On Fri, Aug 19, 2016 at 12:58 PM, Petr Spacek < >>>>>>>> pspacek at redhat.com> >>>>>>>> >>>>>> wrote: >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>>> On 18.8.2016 17:23, Rakesh Rajasekharan wrote: >>>>>>>> >>>>>>>>>> Hi >>>>>>>> >>>>>>>>>> >>>>>>>> >>>>>>>>>> I am migrating to freeipa from openldap and have around >>>>>>>> 4000 >>>>>>>> >>>> clients >>>>>>>> >>>>>>>>>> >>>>>>>> >>>>>>>>>> I had openned a another thread on that, but chose to >>>>>>>> start a new >>>>>>>> >>>> one >>>>>>>> >>>>>>>>> here >>>>>>>> >>>>>>>>>> as its a separate issue >>>>>>>> >>>>>>>>>> >>>>>>>> >>>>>>>>>> I was able to change the nssslapd-maxdescriptors adding >>>>>>>> an ldif >>>>>>>> >>>> file >>>>>>>> >>>>>>>>>> >>>>>>>> >>>>>>>>>> cat nsslapd-modify.ldif >>>>>>>> >>>>>>>>>> dn: cn=config >>>>>>>> >>>>>>>>>> changetype: modify >>>>>>>> >>>>>>>>>> replace: nsslapd-maxdescriptors >>>>>>>> >>>>>>>>>> nsslapd-maxdescriptors: 17000 >>>>>>>> >>>>>>>>>> >>>>>>>> >>>>>>>>>> and running the ldapmodify command >>>>>>>> >>>>>>>>>> >>>>>>>> >>>>>>>>>> I have now started moving clients running an openldap to >>>>>>>> Freeipa >>>>>>>> >>>> and >>>>>>>> >>>>>>>>> have >>>>>>>> >>>>>>>>>> today moved close to 2000 clients >>>>>>>> >>>>>>>>>> >>>>>>>> >>>>>>>>>> However, I have noticed that IPA hangs intermittently. >>>>>>>> >>>>>>>>>> >>>>>>>> >>>>>>>>>> running a kinit admin returns the below error >>>>>>>> >>>>>>>>>> kinit: Generic error (see e-text) while getting initial >>>>>>>> >>>> credentials >>>>>>>> >>>>>>>>>> >>>>>>>> >>>>>>>>>> from the /var/log/messages, I see this entry >>>>>>>> >>>>>>>>>> >>>>>>>> >>>>>>>>>> prod-ipa-master-int kernel: [104090.315801] TCP: >>>>>>>> >>>> request_sock_TCP: >>>>>>>> >>>>>>>>>> Possible SYN flooding on port 88. Sending cookies. >>>>>>>> Check SNMP >>>>>>>> >>>>>> counters. >>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>>>> I would be worried about this message. Maybe >>>>>>>> kernel/firewall is >>>>>>>> >>>> doing >>>>>>>> >>>>>>>>> something fishy behind your back and blocking some >>>>>>>> connections or >>>>>>>> >>>> so. >>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>>>> Petr^2 Spacek >>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>>>>> Aug 18 13:00:01 prod-ipa-master-int systemd[1]: Started >>>>>>>> Session >>>>>>>> >>>> 4885 >>>>>>>> >>>>>> of >>>>>>>> >>>>>>>>>> user root. >>>>>>>> >>>>>>>>>> Aug 18 13:00:01 prod-ipa-master-int systemd[1]: Starting >>>>>>>> Session >>>>>>>> >>>> 4885 >>>>>>>> >>>>>> of >>>>>>>> >>>>>>>>>> user root. >>>>>>>> >>>>>>>>>> Aug 18 13:01:01 prod-ipa-master-int systemd[1]: Started >>>>>>>> Session >>>>>>>> >>>> 4886 >>>>>>>> >>>>>> of >>>>>>>> >>>>>>>>>> user root. >>>>>>>> >>>>>>>>>> Aug 18 13:01:01 prod-ipa-master-int systemd[1]: Starting >>>>>>>> Session >>>>>>>> >>>> 4886 >>>>>>>> >>>>>> of >>>>>>>> >>>>>>>>>> user root. >>>>>>>> >>>>>>>>>> Aug 18 13:02:40 prod-ipa-master-int python[28984]: >>>>>>>> ansible-command >>>>>>>> >>>>>>>>> Invoked >>>>>>>> >>>>>>>>>> with creates=None executable=None shell=True args= >>>>>>>> removes=None >>>>>>>> >>>>>>>>> warn=True >>>>>>>> >>>>>>>>>> chdir=None >>>>>>>> >>>>>>>>>> Aug 18 13:04:37 prod-ipa-master-int sssd_be: GSSAPI >>>>>>>> Error: >>>>>>>> >>>> Unspecified >>>>>>>> >>>>>>>>> GSS >>>>>>>> >>>>>>>>>> failure. Minor code may provide more information (KDC >>>>>>>> returned >>>>>>>> >>>> error >>>>>>>> >>>>>>>>>> string: PROCESS_TGS) >>>>>>>> >>>>>>>>>> >>>>>>>> >>>>>>>>>> Could it be possible that its due to the initial load of >>>>>>>> adding >>>>>>>> >>>> the >>>>>>>> >>>>>>>>> clients >>>>>>>> >>>>>>>>>> or is there something else that I need to take care of. >>>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>> >>>>> >>>>> >>>> >>>> >>>> >>>> >>>> -- >>>> Manage your subscription for the Freeipa-users mailing list: >>>> https://www.redhat.com/mailman/listinfo/freeipa-users >>>> Go to http://freeipa.org for more info on the project >>>> >>> >>> >> >> >> >> > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From rakesh.rajasekharan at gmail.com Mon Sep 5 12:14:09 2016 From: rakesh.rajasekharan at gmail.com (Rakesh Rajasekharan) Date: Mon, 5 Sep 2016 17:44:09 +0530 Subject: [Freeipa-users] Freeipa 4.2.0 hangs intermittently In-Reply-To: References: <57BEB0AB.2090506@redhat.com> <57C44AC6.1030402@redhat.com> <57CD1FDC.2010109@redhat.com> <57CD5488.60404@redhat.com> Message-ID: btw the values for opscompleted and opsinitiated are ldapsearch -D "cn=directory manager" -w xxxx -LLL -b "cn=monitor" -s base opsinitiated opscompleted dn: cn=monitor opsinitiated: 25655 opscompleted: 25654 Thanks On Mon, Sep 5, 2016 at 5:35 PM, Rakesh Rajasekharan < rakesh.rajasekharan at gmail.com> wrote: > Hi Thierry, > > I was able to now reporduce the issue.. > > the ns-slapd process just got hung,.. however, no core file was generate.. > > I manually created a coredump using gcore and then took a strace of it > this is strace from the coredump generated > > GNU gdb (GDB) Red Hat Enterprise Linux 7.6.1-80.el7 > Copyright (C) 2013 Free Software Foundation, Inc. > License GPLv3+: GNU GPL version 3 or later html> > This is free software: you are free to change and redistribute it. > There is NO WARRANTY, to the extent permitted by law. Type "show copying" > and "show warranty" for details. > This GDB was configured as "x86_64-redhat-linux-gnu". > For bug reporting instructions, please see: > ... > Reading symbols from /usr/sbin/ns-slapd...Reading symbols from > /usr/sbin/ns-slapd...(no debugging symbols found)...done. > (no debugging symbols found)...done. > [New LWP 105159] > [New LWP 105160] > [New LWP 105161] > [New LWP 105162] > [New LWP 105163] > [New LWP 105164] > [New LWP 105165] > [New LWP 105166] > [New LWP 105167] > [New LWP 105168] > [New LWP 105171] > [New LWP 105172] > [New LWP 105173] > [New LWP 105174] > [New LWP 105175] > [New LWP 105176] > [New LWP 105177] > [New LWP 105178] > [New LWP 105179] > [New LWP 105180] > [New LWP 105181] > [New LWP 105182] > [New LWP 105183] > [New LWP 105184] > [New LWP 105185] > [New LWP 105186] > [New LWP 105187] > [New LWP 105188] > [New LWP 105189] > [New LWP 105190] > [New LWP 105191] > [New LWP 105192] > [New LWP 105193] > [New LWP 105194] > [New LWP 105195] > [New LWP 105196] > [New LWP 105197] > [New LWP 105198] > [New LWP 105199] > [New LWP 105200] > [New LWP 105201] > [New LWP 105203] > [New LWP 105215] > [New LWP 105158] > [Thread debugging using libthread_db enabled] > Using host libthread_db library "/lib64/libthread_db.so.1". > Core was generated by `/usr/sbin/ns-slapd'. > #0 0x00007f1b48ae69b3 in select () from /lib64/libc.so.6 > > Thread 44 (Thread 0x7f1b4b6d9840 (LWP 105158)): > #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 () from > /lib64/libpthread.so.0 > No symbol table info available. > #1 0x00007f1b4941b2c3 in PR_EnterMonitor () from /lib64/libnspr4.so > No symbol table info available. > #2 0x00007f1b4b7125ed in setup_pr_read_pds () > No symbol table info available. > #3 0x00007f1b4b715fd9 in slapd_daemon () > No symbol table info available. > #4 0x00007f1b4b707c6c in main () > No symbol table info available. > > Thread 43 (Thread 0x7f1a5df0f700 (LWP 105215)): > #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 () from > /lib64/libpthread.so.0 > No symbol table info available. > #1 0x00007f1b4941b050 in PR_WaitCondVar () from /lib64/libnspr4.so > No symbol table info available. > #2 0x00007f1b4b71df25 in ps_send_results () > No symbol table info available. > #3 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so > No symbol table info available. > #4 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 > No symbol table info available. > #5 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 > No symbol table info available. > > Thread 42 (Thread 0x7f1b03fe7700 (LWP 105203)): > #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 () from > /lib64/libpthread.so.0 > No symbol table info available. > #1 0x00007f1b4941b050 in PR_WaitCondVar () from /lib64/libnspr4.so > No symbol table info available. > #2 0x00007f1b4b71df25 in ps_send_results () > No symbol table info available. > #3 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so > No symbol table info available. > #4 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 > No symbol table info available. > #5 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 > No symbol table info available. > > Thread 41 (Thread 0x7f1b047e8700 (LWP 105201)): > #0 0x00007f1b48ae69b3 in select () from /lib64/libc.so.6 > No symbol table info available. > #1 0x00007f1b4b2710e9 in DS_Sleep () from /usr/lib64/dirsrv/libslapd.so.0 > No symbol table info available. > #2 0x00007f1b4b712325 in time_thread () > No symbol table info available. > #3 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so > No symbol table info available. > #4 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 > No symbol table info available. > #5 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 > No symbol table info available. > > Thread 40 (Thread 0x7f1b04fe9700 (LWP 105200)): > #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 () from > /lib64/libpthread.so.0 > No symbol table info available. > #1 0x00007f1b433212f3 in __db_hybrid_mutex_suspend () from /lib64/ > libdb-5.3.so > No symbol table info available. > #2 0x00007f1b43320640 in __db_tas_mutex_lock () from /lib64/libdb-5.3.so > No symbol table info available. > #3 0x00007f1b4343ae1b in __memp_fget () from /lib64/libdb-5.3.so > No symbol table info available. > #4 0x00007f1b4333e211 in __bam_search () from /lib64/libdb-5.3.so > No symbol table info available. > #5 0x00007f1b43329256 in __bamc_search () from /lib64/libdb-5.3.so > No symbol table info available. > #6 0x00007f1b4332d6c4 in __bamc_put () from /lib64/libdb-5.3.so > No symbol table info available. > #7 0x00007f1b433e46d5 in __dbc_iput () from /lib64/libdb-5.3.so > No symbol table info available. > #8 0x00007f1b433dfa0e in __db_put () from /lib64/libdb-5.3.so > No symbol table info available. > #9 0x00007f1b433f4fa4 in __db_put_pp () from /lib64/libdb-5.3.so > No symbol table info available. > #10 0x00007f1b3e8cf4cf in idl_new_insert_key () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > No symbol table info available. > #11 0x00007f1b3e8de5b5 in addordel_values_sv.isra.3 () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > No symbol table info available. > #12 0x00007f1b3e8df25a in index_addordel_values_ext_sv () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > No symbol table info available. > #13 0x00007f1b3e8df694 in index_addordel_values_sv () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > No symbol table info available. > #14 0x00007f1b3e8df75a in index_addordel_entry () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > No symbol table info available. > #15 0x00007f1b3e8e25fa in ldbm_back_add () from /usr/lib64/dirsrv/plugins/ > libback-ldbm.so > No symbol table info available. > #16 0x00007f1b4b1e4cca in op_shared_add () from > /usr/lib64/dirsrv/libslapd.so.0 > No symbol table info available. > #17 0x00007f1b4b1e6020 in do_add () from /usr/lib64/dirsrv/libslapd.so.0 > No symbol table info available. > #18 0x00007f1b4b711303 in connection_threadmain () > No symbol table info available. > #19 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so > No symbol table info available. > #20 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 > No symbol table info available. > #21 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 > No symbol table info available. > > Thread 39 (Thread 0x7f1b057ea700 (LWP 105199)): > #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 () from > /lib64/libpthread.so.0 > No symbol table info available. > #1 0x00007f1b4941b050 in PR_WaitCondVar () from /lib64/libnspr4.so > No symbol table info available. > #2 0x00007f1b4b70efee in connection_wait_for_new_work () > No symbol table info available. > #3 0x00007f1b4b710719 in connection_threadmain () > No symbol table info available. > #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so > No symbol table info available. > #5 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 > No symbol table info available. > #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 > No symbol table info available. > > Thread 38 (Thread 0x7f1b05feb700 (LWP 105198)): > #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 () from > /lib64/libpthread.so.0 > No symbol table info available. > #1 0x00007f1b4941b050 in PR_WaitCondVar () from /lib64/libnspr4.so > No symbol table info available. > #2 0x00007f1b4b70efee in connection_wait_for_new_work () > No symbol table info available. > #3 0x00007f1b4b710719 in connection_threadmain () > No symbol table info available. > #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so > No symbol table info available. > #5 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 > No symbol table info available. > #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 > No symbol table info available. > > Thread 37 (Thread 0x7f1b067ec700 (LWP 105197)): > #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 () from > /lib64/libpthread.so.0 > No symbol table info available. > #1 0x00007f1b4941b050 in PR_WaitCondVar () from /lib64/libnspr4.so > No symbol table info available. > #2 0x00007f1b4b70efee in connection_wait_for_new_work () > No symbol table info available. > #3 0x00007f1b4b710719 in connection_threadmain () > No symbol table info available. > #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so > No symbol table info available. > #5 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 > No symbol table info available. > #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 > No symbol table info available. > > Thread 36 (Thread 0x7f1b06fed700 (LWP 105196)): > #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 () from > /lib64/libpthread.so.0 > No symbol table info available. > #1 0x00007f1b4941b050 in PR_WaitCondVar () from /lib64/libnspr4.so > No symbol table info available. > #2 0x00007f1b4b70efee in connection_wait_for_new_work () > No symbol table info available. > #3 0x00007f1b4b710719 in connection_threadmain () > No symbol table info available. > #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so > No symbol table info available. > #5 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 > No symbol table info available. > #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 > No symbol table info available. > > Thread 35 (Thread 0x7f1b077ee700 (LWP 105195)): > #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 () from > /lib64/libpthread.so.0 > No symbol table info available. > #1 0x00007f1b4941b2c3 in PR_EnterMonitor () from /lib64/libnspr4.so > No symbol table info available. > #2 0x00007f1b3e8c23e6 in dblayer_txn_begin () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > No symbol table info available. > #3 0x00007f1b3e8fe007 in ldbm_back_modify () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > No symbol table info available. > #4 0x00007f1b4b22b541 in op_shared_modify () from > /usr/lib64/dirsrv/libslapd.so.0 > No symbol table info available. > #5 0x00007f1b4b22c004 in modify_internal_pb () from > /usr/lib64/dirsrv/libslapd.so.0 > No symbol table info available. > #6 0x00007f1b3eb48b60 in ipapwd_apply_mods () from > /usr/lib64/dirsrv/plugins/libipa_pwd_extop.so > No symbol table info available. > #7 0x00007f1b3eb4c8e0 in store_new_keys () from /usr/lib64/dirsrv/plugins/ > libipa_pwd_extop.so > No symbol table info available. > #8 0x00007f1b3eb4ead0 in ipapwd_extop () from /usr/lib64/dirsrv/plugins/ > libipa_pwd_extop.so > No symbol table info available. > #9 0x00007f1b4b23bba2 in plugin_call_exop_plugins () from > /usr/lib64/dirsrv/libslapd.so.0 > No symbol table info available. > #10 0x00007f1b4b717efe in do_extended () > No symbol table info available. > #11 0x00007f1b4b71141a in connection_threadmain () > No symbol table info available. > #12 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so > No symbol table info available. > #13 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 > No symbol table info available. > #14 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 > No symbol table info available. > > Thread 34 (Thread 0x7f1b07fef700 (LWP 105194)): > #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 () from > /lib64/libpthread.so.0 > No symbol table info available. > #1 0x00007f1b4941b050 in PR_WaitCondVar () from /lib64/libnspr4.so > No symbol table info available. > #2 0x00007f1b4b70efee in connection_wait_for_new_work () > No symbol table info available. > #3 0x00007f1b4b710719 in connection_threadmain () > No symbol table info available. > #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so > No symbol table info available. > #5 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 > No symbol table info available. > #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 > No symbol table info available. > > Thread 33 (Thread 0x7f1b087f0700 (LWP 105193)): > #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 () from > /lib64/libpthread.so.0 > No symbol table info available. > #1 0x00007f1b4941b050 in PR_WaitCondVar () from /lib64/libnspr4.so > No symbol table info available. > #2 0x00007f1b4b70efee in connection_wait_for_new_work () > No symbol table info available. > #3 0x00007f1b4b710719 in connection_threadmain () > No symbol table info available. > #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so > No symbol table info available. > #5 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 > No symbol table info available. > #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 > No symbol table info available. > > Thread 32 (Thread 0x7f1b08ff1700 (LWP 105192)): > #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 () from > /lib64/libpthread.so.0 > No symbol table info available. > #1 0x00007f1b4941b050 in PR_WaitCondVar () from /lib64/libnspr4.so > No symbol table info available. > #2 0x00007f1b4b70efee in connection_wait_for_new_work () > No symbol table info available. > #3 0x00007f1b4b710719 in connection_threadmain () > No symbol table info available. > #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so > No symbol table info available. > #5 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 > No symbol table info available. > #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 > No symbol table info available. > > Thread 31 (Thread 0x7f1b097f2700 (LWP 105191)): > #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 () from > /lib64/libpthread.so.0 > No symbol table info available. > #1 0x00007f1b4941b050 in PR_WaitCondVar () from /lib64/libnspr4.so > No symbol table info available. > #2 0x00007f1b4b70efee in connection_wait_for_new_work () > No symbol table info available. > #3 0x00007f1b4b710719 in connection_threadmain () > No symbol table info available. > #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so > No symbol table info available. > #5 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 > No symbol table info available. > #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 > No symbol table info available. > > Thread 30 (Thread 0x7f1b09ff3700 (LWP 105190)): > #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 () from > /lib64/libpthread.so.0 > No symbol table info available. > #1 0x00007f1b433212f3 in __db_hybrid_mutex_suspend () from /lib64/ > libdb-5.3.so > No symbol table info available. > #2 0x00007f1b43320640 in __db_tas_mutex_lock () from /lib64/libdb-5.3.so > No symbol table info available. > #3 0x00007f1b433cad3a in __lock_get_internal () from /lib64/libdb-5.3.so > No symbol table info available. > #4 0x00007f1b433cb820 in __lock_get () from /lib64/libdb-5.3.so > No symbol table info available. > #5 0x00007f1b433f7142 in __db_lget () from /lib64/libdb-5.3.so > No symbol table info available. > #6 0x00007f1b4333e605 in __bam_search () from /lib64/libdb-5.3.so > No symbol table info available. > #7 0x00007f1b43329256 in __bamc_search () from /lib64/libdb-5.3.so > No symbol table info available. > #8 0x00007f1b4332ad0f in __bamc_get () from /lib64/libdb-5.3.so > No symbol table info available. > #9 0x00007f1b433e3ca6 in __dbc_iget () from /lib64/libdb-5.3.so > No symbol table info available. > #10 0x00007f1b433f2b02 in __dbc_get_pp () from /lib64/libdb-5.3.so > No symbol table info available. > #11 0x00007f1b3e8ce680 in idl_new_fetch () from /usr/lib64/dirsrv/plugins/ > libback-ldbm.so > No symbol table info available. > #12 0x00007f1b3e8dcb26 in index_read_ext_allids () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > No symbol table info available. > #13 0x00007f1b3e8c7284 in keys2idl () from /usr/lib64/dirsrv/plugins/ > libback-ldbm.so > No symbol table info available. > #14 0x00007f1b3e8c79e3 in ava_candidates.isra.0 () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > No symbol table info available. > #15 0x00007f1b3e8c7fd2 in filter_candidates_ext () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > No symbol table info available. > #16 0x00007f1b3e8c9096 in list_candidates () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > No symbol table info available. > #17 0x00007f1b3e8c7f40 in filter_candidates_ext () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > No symbol table info available. > #18 0x00007f1b3e8c9096 in list_candidates () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > No symbol table info available. > #19 0x00007f1b3e8c7f40 in filter_candidates_ext () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > No symbol table info available. > #20 0x00007f1b3e8c9096 in list_candidates () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > No symbol table info available. > #21 0x00007f1b3e8c7f40 in filter_candidates_ext () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > No symbol table info available. > #22 0x00007f1b3e90380c in subtree_candidates () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > No symbol table info available. > #23 0x00007f1b3e904e7f in ldbm_back_search () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > No symbol table info available. > #24 0x00007f1b4b2321d6 in op_shared_search () from > /usr/lib64/dirsrv/libslapd.so.0 > No symbol table info available. > #25 0x00007f1b4b72333e in do_search () > No symbol table info available. > #26 0x00007f1b4b7113c3 in connection_threadmain () > No symbol table info available. > #27 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so > No symbol table info available. > #28 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 > No symbol table info available. > #29 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 > No symbol table info available. > > Thread 29 (Thread 0x7f1b0a7f4700 (LWP 105189)): > #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 () from > /lib64/libpthread.so.0 > No symbol table info available. > #1 0x00007f1b4941b050 in PR_WaitCondVar () from /lib64/libnspr4.so > No symbol table info available. > #2 0x00007f1b4b70efee in connection_wait_for_new_work () > No symbol table info available. > #3 0x00007f1b4b710719 in connection_threadmain () > No symbol table info available. > #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so > No symbol table info available. > #5 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 > No symbol table info available. > #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 > No symbol table info available. > > Thread 28 (Thread 0x7f1b0aff5700 (LWP 105188)): > #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 () from > /lib64/libpthread.so.0 > No symbol table info available. > #1 0x00007f1b4941b050 in PR_WaitCondVar () from /lib64/libnspr4.so > No symbol table info available. > #2 0x00007f1b4b70efee in connection_wait_for_new_work () > No symbol table info available. > #3 0x00007f1b4b710719 in connection_threadmain () > No symbol table info available. > #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so > No symbol table info available. > #5 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 > No symbol table info available. > #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 > No symbol table info available. > > Thread 27 (Thread 0x7f1b0b7f6700 (LWP 105187)): > #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 () from > /lib64/libpthread.so.0 > No symbol table info available. > #1 0x00007f1b4941b050 in PR_WaitCondVar () from /lib64/libnspr4.so > No symbol table info available. > #2 0x00007f1b4b70efee in connection_wait_for_new_work () > No symbol table info available. > #3 0x00007f1b4b710719 in connection_threadmain () > No symbol table info available. > #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so > No symbol table info available. > #5 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 > No symbol table info available. > #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 > No symbol table info available. > > Thread 26 (Thread 0x7f1b0bff7700 (LWP 105186)): > #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 () from > /lib64/libpthread.so.0 > No symbol table info available. > #1 0x00007f1b433212f3 in __db_hybrid_mutex_suspend () from /lib64/ > libdb-5.3.so > No symbol table info available. > #2 0x00007f1b43320640 in __db_tas_mutex_lock () from /lib64/libdb-5.3.so > No symbol table info available. > #3 0x00007f1b433cad3a in __lock_get_internal () from /lib64/libdb-5.3.so > No symbol table info available. > #4 0x00007f1b433cb820 in __lock_get () from /lib64/libdb-5.3.so > No symbol table info available. > #5 0x00007f1b433f7142 in __db_lget () from /lib64/libdb-5.3.so > No symbol table info available. > #6 0x00007f1b4333e605 in __bam_search () from /lib64/libdb-5.3.so > No symbol table info available. > #7 0x00007f1b43329256 in __bamc_search () from /lib64/libdb-5.3.so > No symbol table info available. > #8 0x00007f1b4332ad0f in __bamc_get () from /lib64/libdb-5.3.so > No symbol table info available. > #9 0x00007f1b433e3ca6 in __dbc_iget () from /lib64/libdb-5.3.so > No symbol table info available. > #10 0x00007f1b433f2b02 in __dbc_get_pp () from /lib64/libdb-5.3.so > No symbol table info available. > #11 0x00007f1b3e8ce680 in idl_new_fetch () from /usr/lib64/dirsrv/plugins/ > libback-ldbm.so > No symbol table info available. > #12 0x00007f1b3e8dcb26 in index_read_ext_allids () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > No symbol table info available. > #13 0x00007f1b3e8c7284 in keys2idl () from /usr/lib64/dirsrv/plugins/ > libback-ldbm.so > No symbol table info available. > #14 0x00007f1b3e8c79e3 in ava_candidates.isra.0 () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > No symbol table info available. > #15 0x00007f1b3e8c7fd2 in filter_candidates_ext () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > No symbol table info available. > #16 0x00007f1b3e8c9096 in list_candidates () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > No symbol table info available. > #17 0x00007f1b3e8c7f40 in filter_candidates_ext () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > No symbol table info available. > #18 0x00007f1b3e90380c in subtree_candidates () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > No symbol table info available. > #19 0x00007f1b3e904e7f in ldbm_back_search () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > No symbol table info available. > #20 0x00007f1b4b2321d6 in op_shared_search () from > /usr/lib64/dirsrv/libslapd.so.0 > No symbol table info available. > #21 0x00007f1b4b24294e in search_internal_callback_pb () from > /usr/lib64/dirsrv/libslapd.so.0 > No symbol table info available. > #22 0x00007f1b4b242be8 in search_internal_pb () from > /usr/lib64/dirsrv/libslapd.so.0 > No symbol table info available. > #23 0x00007f1b4b72149b in ids_sasl_canon_user () > No symbol table info available. > #24 0x00007f1b491e193b in _sasl_canon_user () from /lib64/libsasl2.so.3 > No symbol table info available. > #25 0x00007f1b491e1c4c in _sasl_canon_user_lookup () from > /lib64/libsasl2.so.3 > No symbol table info available. > #26 0x00007f1b430e9fb9 in gssapi_server_mech_step () from > /usr/lib64/sasl2/libgssapiv2.so > No symbol table info available. > #27 0x00007f1b491edb9b in sasl_server_step () from /lib64/libsasl2.so.3 > No symbol table info available. > #28 0x00007f1b4b72283b in ids_sasl_check_bind () > No symbol table info available. > #29 0x00007f1b4b70a61a in do_bind () > No symbol table info available. > #30 0x00007f1b4b7113fd in connection_threadmain () > No symbol table info available. > #31 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so > No symbol table info available. > #32 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 > No symbol table info available. > #33 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 > No symbol table info available. > > Thread 25 (Thread 0x7f1b0c7f8700 (LWP 105185)): > #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 () from > /lib64/libpthread.so.0 > No symbol table info available. > #1 0x00007f1b4941b050 in PR_WaitCondVar () from /lib64/libnspr4.so > No symbol table info available. > #2 0x00007f1b4b70efee in connection_wait_for_new_work () > No symbol table info available. > #3 0x00007f1b4b710719 in connection_threadmain () > No symbol table info available. > #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so > No symbol table info available. > #5 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 > No symbol table info available. > #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 > No symbol table info available. > > Thread 24 (Thread 0x7f1b0cff9700 (LWP 105184)): > #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 () from > /lib64/libpthread.so.0 > No symbol table info available. > #1 0x00007f1b4941b050 in PR_WaitCondVar () from /lib64/libnspr4.so > No symbol table info available. > #2 0x00007f1b4b70efee in connection_wait_for_new_work () > No symbol table info available. > #3 0x00007f1b4b710719 in connection_threadmain () > No symbol table info available. > #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so > No symbol table info available. > #5 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 > No symbol table info available. > #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 > No symbol table info available. > > Thread 23 (Thread 0x7f1b0d7fa700 (LWP 105183)): > #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 () from > /lib64/libpthread.so.0 > No symbol table info available. > #1 0x00007f1b4941b050 in PR_WaitCondVar () from /lib64/libnspr4.so > No symbol table info available. > #2 0x00007f1b4b70efee in connection_wait_for_new_work () > No symbol table info available. > #3 0x00007f1b4b710719 in connection_threadmain () > No symbol table info available. > #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so > No symbol table info available. > #5 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 > No symbol table info available. > #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 > No symbol table info available. > > Thread 22 (Thread 0x7f1b0dffb700 (LWP 105182)): > #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 () from > /lib64/libpthread.so.0 > No symbol table info available. > #1 0x00007f1b4941b050 in PR_WaitCondVar () from /lib64/libnspr4.so > No symbol table info available. > #2 0x00007f1b4b70efee in connection_wait_for_new_work () > No symbol table info available. > #3 0x00007f1b4b710719 in connection_threadmain () > No symbol table info available. > #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so > No symbol table info available. > #5 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 > No symbol table info available. > #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 > No symbol table info available. > > Thread 21 (Thread 0x7f1b0e7fc700 (LWP 105181)): > #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 () from > /lib64/libpthread.so.0 > No symbol table info available. > #1 0x00007f1b4941b050 in PR_WaitCondVar () from /lib64/libnspr4.so > No symbol table info available. > #2 0x00007f1b4b70efee in connection_wait_for_new_work () > No symbol table info available. > #3 0x00007f1b4b710719 in connection_threadmain () > No symbol table info available. > #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so > No symbol table info available. > #5 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 > No symbol table info available. > #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 > No symbol table info available. > > Thread 20 (Thread 0x7f1b0effd700 (LWP 105180)): > #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 () from > /lib64/libpthread.so.0 > No symbol table info available. > #1 0x00007f1b433212f3 in __db_hybrid_mutex_suspend () from /lib64/ > libdb-5.3.so > No symbol table info available. > #2 0x00007f1b43320b37 in __db_tas_mutex_readlock () from /lib64/ > libdb-5.3.so > No symbol table info available. > #3 0x00007f1b4343ad35 in __memp_fget () from /lib64/libdb-5.3.so > No symbol table info available. > #4 0x00007f1b433e4333 in __dbc_iget () from /lib64/libdb-5.3.so > No symbol table info available. > #5 0x00007f1b433f2b02 in __dbc_get_pp () from /lib64/libdb-5.3.so > No symbol table info available. > #6 0x00007f1b3e8ce822 in idl_new_fetch () from /usr/lib64/dirsrv/plugins/ > libback-ldbm.so > No symbol table info available. > #7 0x00007f1b3e8dcb26 in index_read_ext_allids () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > No symbol table info available. > #8 0x00007f1b3e8c7284 in keys2idl () from /usr/lib64/dirsrv/plugins/ > libback-ldbm.so > No symbol table info available. > #9 0x00007f1b3e8c79e3 in ava_candidates.isra.0 () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > No symbol table info available. > #10 0x00007f1b3e8c7fd2 in filter_candidates_ext () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > No symbol table info available. > #11 0x00007f1b3e8c9096 in list_candidates () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > No symbol table info available. > #12 0x00007f1b3e8c7f40 in filter_candidates_ext () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > No symbol table info available. > #13 0x00007f1b3e8c9096 in list_candidates () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > No symbol table info available. > #14 0x00007f1b3e8c7f40 in filter_candidates_ext () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > No symbol table info available. > #15 0x00007f1b3e8c9096 in list_candidates () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > No symbol table info available. > #16 0x00007f1b3e8c7f40 in filter_candidates_ext () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > No symbol table info available. > #17 0x00007f1b3e90380c in subtree_candidates () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > No symbol table info available. > #18 0x00007f1b3e904e7f in ldbm_back_search () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > No symbol table info available. > #19 0x00007f1b4b2321d6 in op_shared_search () from > /usr/lib64/dirsrv/libslapd.so.0 > No symbol table info available. > #20 0x00007f1b4b72333e in do_search () > No symbol table info available. > #21 0x00007f1b4b7113c3 in connection_threadmain () > No symbol table info available. > #22 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so > No symbol table info available. > #23 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 > No symbol table info available. > #24 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 > No symbol table info available. > > Thread 19 (Thread 0x7f1b0f7fe700 (LWP 105179)): > #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 () from > /lib64/libpthread.so.0 > No symbol table info available. > #1 0x00007f1b433212f3 in __db_hybrid_mutex_suspend () from /lib64/ > libdb-5.3.so > No symbol table info available. > #2 0x00007f1b43320640 in __db_tas_mutex_lock () from /lib64/libdb-5.3.so > No symbol table info available. > #3 0x00007f1b433cad3a in __lock_get_internal () from /lib64/libdb-5.3.so > No symbol table info available. > #4 0x00007f1b433cb820 in __lock_get () from /lib64/libdb-5.3.so > No symbol table info available. > #5 0x00007f1b433f7142 in __db_lget () from /lib64/libdb-5.3.so > No symbol table info available. > #6 0x00007f1b4333e605 in __bam_search () from /lib64/libdb-5.3.so > No symbol table info available. > #7 0x00007f1b43329256 in __bamc_search () from /lib64/libdb-5.3.so > No symbol table info available. > #8 0x00007f1b4332ad0f in __bamc_get () from /lib64/libdb-5.3.so > No symbol table info available. > #9 0x00007f1b433e3ca6 in __dbc_iget () from /lib64/libdb-5.3.so > No symbol table info available. > #10 0x00007f1b433f2b02 in __dbc_get_pp () from /lib64/libdb-5.3.so > No symbol table info available. > #11 0x00007f1b3e8ce680 in idl_new_fetch () from /usr/lib64/dirsrv/plugins/ > libback-ldbm.so > No symbol table info available. > #12 0x00007f1b3e8dcb26 in index_read_ext_allids () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > No symbol table info available. > #13 0x00007f1b3e8c7284 in keys2idl () from /usr/lib64/dirsrv/plugins/ > libback-ldbm.so > No symbol table info available. > #14 0x00007f1b3e8c79e3 in ava_candidates.isra.0 () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > No symbol table info available. > #15 0x00007f1b3e8c7fd2 in filter_candidates_ext () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > No symbol table info available. > #16 0x00007f1b3e8c9096 in list_candidates () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > No symbol table info available. > #17 0x00007f1b3e8c7f40 in filter_candidates_ext () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > No symbol table info available. > #18 0x00007f1b3e8c9096 in list_candidates () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > No symbol table info available. > #19 0x00007f1b3e8c7f40 in filter_candidates_ext () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > No symbol table info available. > #20 0x00007f1b3e8c9096 in list_candidates () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > No symbol table info available. > #21 0x00007f1b3e8c7f40 in filter_candidates_ext () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > No symbol table info available. > #22 0x00007f1b3e90380c in subtree_candidates () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > No symbol table info available. > #23 0x00007f1b3e904e7f in ldbm_back_search () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > No symbol table info available. > #24 0x00007f1b4b2321d6 in op_shared_search () from > /usr/lib64/dirsrv/libslapd.so.0 > No symbol table info available. > #25 0x00007f1b4b72333e in do_search () > No symbol table info available. > #26 0x00007f1b4b7113c3 in connection_threadmain () > No symbol table info available. > #27 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so > No symbol table info available. > #28 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 > No symbol table info available. > #29 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 > No symbol table info available. > > Thread 18 (Thread 0x7f1b0ffff700 (LWP 105178)): > #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 () from > /lib64/libpthread.so.0 > No symbol table info available. > #1 0x00007f1b4941b050 in PR_WaitCondVar () from /lib64/libnspr4.so > No symbol table info available. > #2 0x00007f1b4b70efee in connection_wait_for_new_work () > No symbol table info available. > #3 0x00007f1b4b710719 in connection_threadmain () > No symbol table info available. > #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so > No symbol table info available. > #5 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 > No symbol table info available. > #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 > No symbol table info available. > > Thread 17 (Thread 0x7f1b14d28700 (LWP 105177)): > #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 () from > /lib64/libpthread.so.0 > No symbol table info available. > #1 0x00007f1b4941b050 in PR_WaitCondVar () from /lib64/libnspr4.so > No symbol table info available. > #2 0x00007f1b4b70efee in connection_wait_for_new_work () > No symbol table info available. > #3 0x00007f1b4b710719 in connection_threadmain () > No symbol table info available. > #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so > No symbol table info available. > #5 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 > No symbol table info available. > #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 > No symbol table info available. > > Thread 16 (Thread 0x7f1b15529700 (LWP 105176)): > #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 () from > /lib64/libpthread.so.0 > No symbol table info available. > #1 0x00007f1b4941b050 in PR_WaitCondVar () from /lib64/libnspr4.so > No symbol table info available. > #2 0x00007f1b4b70efee in connection_wait_for_new_work () > No symbol table info available. > #3 0x00007f1b4b710719 in connection_threadmain () > No symbol table info available. > #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so > No symbol table info available. > #5 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 > No symbol table info available. > #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 > No symbol table info available. > > Thread 15 (Thread 0x7f1b15d2a700 (LWP 105175)): > #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 () from > /lib64/libpthread.so.0 > No symbol table info available. > #1 0x00007f1b4941b050 in PR_WaitCondVar () from /lib64/libnspr4.so > No symbol table info available. > #2 0x00007f1b4b70efee in connection_wait_for_new_work () > No symbol table info available. > #3 0x00007f1b4b710719 in connection_threadmain () > No symbol table info available. > #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so > No symbol table info available. > #5 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 > No symbol table info available. > #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 > No symbol table info available. > > Thread 14 (Thread 0x7f1b1652b700 (LWP 105174)): > #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 () from > /lib64/libpthread.so.0 > No symbol table info available. > #1 0x00007f1b4941b050 in PR_WaitCondVar () from /lib64/libnspr4.so > No symbol table info available. > #2 0x00007f1b4b70efee in connection_wait_for_new_work () > No symbol table info available. > #3 0x00007f1b4b710719 in connection_threadmain () > No symbol table info available. > #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so > No symbol table info available. > #5 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 > No symbol table info available. > #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 > No symbol table info available. > > Thread 13 (Thread 0x7f1b16d2c700 (LWP 105173)): > #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 () from > /lib64/libpthread.so.0 > No symbol table info available. > #1 0x00007f1b433212f3 in __db_hybrid_mutex_suspend () from /lib64/ > libdb-5.3.so > No symbol table info available. > #2 0x00007f1b43320640 in __db_tas_mutex_lock () from /lib64/libdb-5.3.so > No symbol table info available. > #3 0x00007f1b433cad3a in __lock_get_internal () from /lib64/libdb-5.3.so > No symbol table info available. > #4 0x00007f1b433cb820 in __lock_get () from /lib64/libdb-5.3.so > No symbol table info available. > #5 0x00007f1b433f7142 in __db_lget () from /lib64/libdb-5.3.so > No symbol table info available. > #6 0x00007f1b4333e605 in __bam_search () from /lib64/libdb-5.3.so > No symbol table info available. > #7 0x00007f1b43329256 in __bamc_search () from /lib64/libdb-5.3.so > No symbol table info available. > #8 0x00007f1b4332ad0f in __bamc_get () from /lib64/libdb-5.3.so > No symbol table info available. > #9 0x00007f1b433e3ca6 in __dbc_iget () from /lib64/libdb-5.3.so > No symbol table info available. > #10 0x00007f1b433f2b02 in __dbc_get_pp () from /lib64/libdb-5.3.so > No symbol table info available. > #11 0x00007f1b3e8ce680 in idl_new_fetch () from /usr/lib64/dirsrv/plugins/ > libback-ldbm.so > No symbol table info available. > #12 0x00007f1b3e8dcb26 in index_read_ext_allids () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > No symbol table info available. > #13 0x00007f1b3e8c7284 in keys2idl () from /usr/lib64/dirsrv/plugins/ > libback-ldbm.so > No symbol table info available. > #14 0x00007f1b3e8c79e3 in ava_candidates.isra.0 () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > No symbol table info available. > #15 0x00007f1b3e8c7fd2 in filter_candidates_ext () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > No symbol table info available. > #16 0x00007f1b3e8c9096 in list_candidates () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > No symbol table info available. > #17 0x00007f1b3e8c7f40 in filter_candidates_ext () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > No symbol table info available. > #18 0x00007f1b3e8c9096 in list_candidates () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > No symbol table info available. > #19 0x00007f1b3e8c7f40 in filter_candidates_ext () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > No symbol table info available. > #20 0x00007f1b3e8c9096 in list_candidates () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > No symbol table info available. > #21 0x00007f1b3e8c7f40 in filter_candidates_ext () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > No symbol table info available. > #22 0x00007f1b3e90380c in subtree_candidates () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > No symbol table info available. > #23 0x00007f1b3e904e7f in ldbm_back_search () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > No symbol table info available. > #24 0x00007f1b4b2321d6 in op_shared_search () from > /usr/lib64/dirsrv/libslapd.so.0 > No symbol table info available. > #25 0x00007f1b4b72333e in do_search () > No symbol table info available. > #26 0x00007f1b4b7113c3 in connection_threadmain () > No symbol table info available. > #27 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so > No symbol table info available. > #28 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 > No symbol table info available. > #29 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 > No symbol table info available. > > Thread 12 (Thread 0x7f1b1752d700 (LWP 105172)): > #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 () from > /lib64/libpthread.so.0 > No symbol table info available. > #1 0x00007f1b4941b050 in PR_WaitCondVar () from /lib64/libnspr4.so > No symbol table info available. > #2 0x00007f1b4b70efee in connection_wait_for_new_work () > No symbol table info available. > #3 0x00007f1b4b710719 in connection_threadmain () > No symbol table info available. > #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so > No symbol table info available. > #5 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 > No symbol table info available. > #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 > No symbol table info available. > > Thread 11 (Thread 0x7f1b17d2e700 (LWP 105171)): > #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 () from > /lib64/libpthread.so.0 > No symbol table info available. > #1 0x00007f1b4941b050 in PR_WaitCondVar () from /lib64/libnspr4.so > No symbol table info available. > #2 0x00007f1b4b70efee in connection_wait_for_new_work () > No symbol table info available. > #3 0x00007f1b4b710719 in connection_threadmain () > No symbol table info available. > #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so > No symbol table info available. > #5 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 > No symbol table info available. > #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 > No symbol table info available. > > Thread 10 (Thread 0x7f1b2509d700 (LWP 105168)): > #0 0x00007f1b48dc5a82 in pthread_cond_timedwait@@GLIBC_2.3.2 () from > /lib64/libpthread.so.0 > No symbol table info available. > #1 0x00007f1b4941ab07 in pt_TimedWait () from /lib64/libnspr4.so > No symbol table info available. > #2 0x00007f1b4941afce in PR_WaitCondVar () from /lib64/libnspr4.so > No symbol table info available. > #3 0x00007f1b4b2061e8 in eq_loop () from /usr/lib64/dirsrv/libslapd.so.0 > No symbol table info available. > #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so > No symbol table info available. > #5 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 > No symbol table info available. > #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 > No symbol table info available. > > Thread 9 (Thread 0x7f1b2589e700 (LWP 105167)): > #0 0x00007f1b48dc5a82 in pthread_cond_timedwait@@GLIBC_2.3.2 () from > /lib64/libpthread.so.0 > No symbol table info available. > #1 0x00007f1b4941ab07 in pt_TimedWait () from /lib64/libnspr4.so > No symbol table info available. > #2 0x00007f1b4941afce in PR_WaitCondVar () from /lib64/libnspr4.so > No symbol table info available. > #3 0x00007f1b4b719d03 in housecleaning () > No symbol table info available. > #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so > No symbol table info available. > #5 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 > No symbol table info available. > #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 > No symbol table info available. > > Thread 8 (Thread 0x7f1b2609f700 (LWP 105166)): > #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 () from > /lib64/libpthread.so.0 > No symbol table info available. > #1 0x00007f1b4941b050 in PR_WaitCondVar () from /lib64/libnspr4.so > No symbol table info available. > #2 0x00007f1b4b260198 in slapi_wait_condvar () from > /usr/lib64/dirsrv/libslapd.so.0 > No symbol table info available. > #3 0x00007f1b3cf5deed in roles_cache_wait_on_change () from > /usr/lib64/dirsrv/plugins/libroles-plugin.so > No symbol table info available. > #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so > No symbol table info available. > #5 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 > No symbol table info available. > #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 > No symbol table info available. > > Thread 7 (Thread 0x7f1b268a0700 (LWP 105165)): > #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 () from > /lib64/libpthread.so.0 > No symbol table info available. > #1 0x00007f1b4941b050 in PR_WaitCondVar () from /lib64/libnspr4.so > No symbol table info available. > #2 0x00007f1b4b260198 in slapi_wait_condvar () from > /usr/lib64/dirsrv/libslapd.so.0 > No symbol table info available. > #3 0x00007f1b3cf5deed in roles_cache_wait_on_change () from > /usr/lib64/dirsrv/plugins/libroles-plugin.so > No symbol table info available. > #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so > No symbol table info available. > #5 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 > No symbol table info available. > #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 > No symbol table info available. > > Thread 6 (Thread 0x7f1b270a1700 (LWP 105164)): > #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 () from > /lib64/libpthread.so.0 > No symbol table info available. > #1 0x00007f1b4941b050 in PR_WaitCondVar () from /lib64/libnspr4.so > No symbol table info available. > #2 0x00007f1b4b260198 in slapi_wait_condvar () from > /usr/lib64/dirsrv/libslapd.so.0 > No symbol table info available. > #3 0x00007f1b3cf5deed in roles_cache_wait_on_change () from > /usr/lib64/dirsrv/plugins/libroles-plugin.so > No symbol table info available. > #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so > No symbol table info available. > #5 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 > No symbol table info available. > #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 > No symbol table info available. > > Thread 5 (Thread 0x7f1b278a2700 (LWP 105163)): > #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 () from > /lib64/libpthread.so.0 > No symbol table info available. > #1 0x00007f1b4941b050 in PR_WaitCondVar () from /lib64/libnspr4.so > No symbol table info available. > #2 0x00007f1b4b260198 in slapi_wait_condvar () from > /usr/lib64/dirsrv/libslapd.so.0 > No symbol table info available. > #3 0x00007f1b415f262e in cos_cache_wait_on_change () from > /usr/lib64/dirsrv/plugins/libcos-plugin.so > No symbol table info available. > #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so > No symbol table info available. > #5 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 > No symbol table info available. > #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 > No symbol table info available. > > Thread 4 (Thread 0x7f1b282ac700 (LWP 105162)): > #0 0x00007f1b48ae69b3 in select () from /lib64/libc.so.6 > No symbol table info available. > #1 0x00007f1b4b2710e9 in DS_Sleep () from /usr/lib64/dirsrv/libslapd.so.0 > No symbol table info available. > #2 0x00007f1b3e8ba707 in perf_threadmain () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > No symbol table info available. > #3 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so > No symbol table info available. > #4 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 > No symbol table info available. > #5 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 > No symbol table info available. > > Thread 3 (Thread 0x7f1b28aad700 (LWP 105161)): > #0 0x00007f1b48ae69b3 in select () from /lib64/libc.so.6 > No symbol table info available. > #1 0x00007f1b4b2710e9 in DS_Sleep () from /usr/lib64/dirsrv/libslapd.so.0 > No symbol table info available. > #2 0x00007f1b3e8bfb7f in trickle_threadmain () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > No symbol table info available. > #3 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so > No symbol table info available. > #4 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 > No symbol table info available. > #5 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 > No symbol table info available. > > Thread 2 (Thread 0x7f1b292ae700 (LWP 105160)): > #0 0x00007f1b48ae69b3 in select () from /lib64/libc.so.6 > No symbol table info available. > #1 0x00007f1b4344dbad in __os_yield () from /lib64/libdb-5.3.so > No symbol table info available. > #2 0x00007f1b434492b3 in __memp_sync_int () from /lib64/libdb-5.3.so > No symbol table info available. > #3 0x00007f1b43459752 in __txn_checkpoint () from /lib64/libdb-5.3.so > No symbol table info available. > #4 0x00007f1b43459b74 in __txn_checkpoint_pp () from /lib64/libdb-5.3.so > No symbol table info available. > #5 0x00007f1b3e8c3a87 in checkpoint_threadmain () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > No symbol table info available. > #6 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so > No symbol table info available. > #7 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 > No symbol table info available. > #8 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 > No symbol table info available. > > Thread 1 (Thread 0x7f1b29aaf700 (LWP 105159)): > #0 0x00007f1b48ae69b3 in select () from /lib64/libc.so.6 > No symbol table info available. > #1 0x00007f1b4b2710e9 in DS_Sleep () from /usr/lib64/dirsrv/libslapd.so.0 > No symbol table info available. > #2 0x00007f1b3e8bf907 in deadlock_threadmain () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > No symbol table info available. > #3 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so > No symbol table info available. > #4 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 > No symbol table info available. > #5 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 > No symbol table info available. > > > > Thanks, > > Rakesh > > > On Mon, Sep 5, 2016 at 4:48 PM, thierry bordaz wrote: > >> >> >> On 09/05/2016 12:05 PM, Rakesh Rajasekharan wrote: >> >> Hi Thierry, >> >> >> I was getting the hang issue while running ipa-client-install >> simultaneously on few clients.. >> However, today, I am not able to replicate that. >> >> I could not get a gdb . But i will try getting that the next time I face >> this issue. >> >> The CPU does not stay high.. it just momentarily touches a high value and >> then drops down to around 2-7% >> >> One question I have is , is it ok to set it nsslapd-threadnumber to a >> very high value . >> I have around 4000 clients and with nsslapd-maxthreadsperconn set to >> 5..So, can I set nsslapd-threadnumber to around 25000. >> >> >> Hello, >> >> I know some users running in production with several hunderds of threads >> (>600) and this without problem. >> >> I do not recall having suggested to increase that number and for what >> reason. >> Usually 30 workers is a good enough value. It can create bootleneck if >> for some reason each operation is very long to satisfy and exhaust the >> number of workers. You can monitor the work queue: >> >> ldapsearch -D "cn=directory manager" -w xxx -LLL -b "cn=monitor" -s base >> opsinitiated opscompleted >> >> >> If opscompleted-opsinitiated remains close to threadnumber, then yes it >> would be valuable to increase it. >> >> The computation #client * #async_op_per_client sound an overkill. Even if >> all clients send at the exact same time all their requests, it is very >> likely that some common resource (db page, log, allocator...) will >> serialize them. If you monitor a need to increase the work, you would for >> example set it to 50, then monitor, then set it to 100, then monitor... >> until you find a good enough value. >> Note the increasing the #thread, increases the memory footprint that will >> reduce the efficiency of file system cache and can increase the response >> time. >> >> >> best regards >> thierry >> >> >> Thanks >> >> On Mon, Sep 5, 2016 at 1:03 PM, thierry bordaz >> wrote: >> >>> >>> Hi Rakesh, >>> >>> Were you able to get a pstack or full stack with gdb ( >>> >>> http://www.port389.org/docs/389ds/FAQ/faq.html#debugging-crashes) when >>> the server hangs ? >>> >>> If it happens with 500 threads as well as with 30, using 30 threads is a >>> better choice to debug this issue. >>> I will try to reproduce using 150 parallel 'ipa user-find p-testipa' >>> commands >>> >>> Something I am unsure is if the CPU consumption stays high (you >>> mentioned 340% CPU usage) as long as the hang happens or if after a suddent >>> shot up to 340% (that marks the beginning of the hang) it drops and stay >>> hanging ? >>> >>> thanks >>> thierry >>> >>> On 09/04/2016 08:40 PM, Rakesh Rajasekharan wrote: >>> >>> starce on the slapd process actually had this in the output.. >>> FUTEX_WAIT_PRIVATE >>> >>> and checking for the number of threads slapd had.. there were 5015 >>> threads >>> >>> ps -efL|grep slapd|wc -l >>> 5015 >>> >>> strace on most of the threads gave this output >>> >>> strace -p 67411 >>> Process 67411 attached >>> futex(0x7f3f0226b41c, FUTEX_WAIT_PRIVATE, 1, NULL) = -1 EAGAIN (Resource >>> temporarily unavailable) >>> futex(0x7f3f0226b41c, FUTEX_WAIT_PRIVATE, 2, NULL^CProcess 67411 detached >>> >>> >>> >>> >>> >>> On Sun, Sep 4, 2016 at 5:34 PM, Rakesh Rajasekharan < >>> rakesh.rajasekharan at gmail.com> wrote: >>> >>>> I have again got the issue of IPA hanging.. The issue came up when i >>>> tried to run ipa-client-isntall on 142 clients simultaneously >>>> >>>> >>>> None of the IPA commands are responding, and I see this error >>>> >>>> ipa user-find p-testipa >>>> ipa: ERROR: Insufficient access: SASL(-1): generic failure: GSSAPI >>>> Error: Unspecified GSS failure. Minor code may provide more information >>>> (KDC returned error string: PROCESS_TGS) >>>> >>>> KRB5_TRACE=/dev/stdout kinit admin >>>> [41178] 1472984115.233214: Getting initial credentials for >>>> admin at XYZ.COM >>>> [41178] 1472984115.235257: Sending request (167 bytes) to XYZ.COM >>>> [41178] 1472984115.235419: Initiating TCP connection to stream >>>> 10.1.3.36:88 >>>> [41178] 1472984115.235685: Sending TCP request to stream 10.1.3.36:88 >>>> [41178] 1472984120.238914: Received answer (174 bytes) from stream >>>> 10.1.3.36:88 >>>> [41178] 1472984120.238925: Terminating TCP connection to stream >>>> 10.1.3.36:88 >>>> [41178] 1472984120.238993: Response was from master KDC >>>> [41 >>>> >>>> >>>> Running an ldapsearch to see the db.. does not give any results and >>>> just hangs there >>>> >>>> ldapsearch -x -D 'cn=Directory Manager' -W -s one -b >>>> 'cn=kerberos,dc=xyz,dc=com' >>>> Enter LDAP Password: >>>> >>>> even an ldapsearch -x does not respond >>>> At this point, am sure that slapd is the one causing issues >>>> >>>> Running an strace against the hung slapd itself seems to get stuck does >>>> not proceed after saying "attaching to process" >>>> >>>> From some others posts I read Thierry suggesting to increase the >>>> nsslapd-threadnumber value >>>> >>>> It was set to 30, I think that might be too low. >>>> >>>> I have raised it to 500 >>>> >>>> Now after restarting the service .. ldapsearch starts responding. >>>> But running the test to add a sudden high number of clients again left >>>> ns-slapd to hung state >>>> >>>> When i attempted adding the clients.. the ns-slapd cpu usage shot up to >>>> 340% and after that ns-slapd stopped responding >>>> >>>> So now, atleast I know what might be causing the issue and I can now >>>> easily reproduce it. >>>> >>>> Is there a way I can make ns-slapd handle a sudden bump in incoming >>>> request for ipa-client-install >>>> >>>> Thanks >>>> Rakesh >>>> >>>> >>>> >>>> >>>> >>>> >>>> On Mon, Aug 29, 2016 at 11:18 PM, Rich Megginson < >>>> rmeggins at redhat.com> wrote: >>>> >>>>> On 08/29/2016 10:53 AM, Rakesh Rajasekharan wrote: >>>>> >>>>> Hi Thierry, >>>>> >>>>> My machine has 30GB RAM ..and 389-ds version is 1.3.4 >>>>> >>>>> ldapsearch shows the values for nsslapd-cachememsize updated to 200MB. >>>>> >>>>> ldapsearch -LLL -o ldif-wrap=no -D "cn=directory manager" -w >>>>> 'mypassword' -b 'cn=userRoot,cn=ldbm database,cn=plugins,cn=config'|grep >>>>> nsslapd-cachememsize >>>>> nsslapd-cachememsize: 209715200 >>>>> >>>>> >>>>> So, it seems to have updated though seeing that warning(WARNING: >>>>> ipaca: entry cache size 10485760B is less than db size 11599872B) in the >>>>> log confuses me a bit. >>>>> >>>>> Thers one more entry that I found from the ldapsearch to be bit low >>>>> >>>>> nsslapd-dncachememsize: 10485760 >>>>> maxdncachesize: 10485760 >>>>> >>>>> Should I update these as well to a higher value >>>>> >>>>> At the time when the issue happened, the memory usage as well as the >>>>> overall load of the system was very low . >>>>> I will try reproducing the issue atleast in my QA env..probably by >>>>> trying to mock simultaneous parallel logins to a large number of hosts >>>>> >>>>> >>>>> To monitor your cache sizes, please use the dbmon.sh tool provided >>>>> with your distro. If that is not available with your particular distro, >>>>> see https://github.com/richm/scripts/wiki/dbmon.sh >>>>> >>>>> >>>>> >>>>> >>>>> thanks >>>>> Rakesh >>>>> >>>>> >>>>> >>>>> >>>>> On Mon, Aug 29, 2016 at 8:16 PM, thierry bordaz < >>>>> tbordaz at redhat.com> wrote: >>>>> >>>>>> Hi Rakesh, >>>>>> >>>>>> Those tuning may depend on the memory available on your machine. >>>>>> nsslapd-cachememsize allows the entry cache to consume up to 200Mb >>>>>> but its memory footprint is known to go above. >>>>>> 200Mb both looks pretty good to me. How large is your machine ? What >>>>>> is your version of 389-ds ? >>>>>> >>>>>> Those warnings do not change your settings. It just raise that entry >>>>>> cache of 'ipaca' and 'retrocl' are small but it is fine. The size of the >>>>>> entry cache is important mostly in userRoot. >>>>>> You may double check the actual values, after restart, with >>>>>> ldapsearch on 'cn=userRoot,cn=ldbm database,cn=plugins,cn=config' and >>>>>> 'cn=config,cn=ldbm database,cn=plugins,cn=config'. >>>>>> >>>>>> A step is to know what will be response time of DS to know if it is >>>>>> responsible of the hang or not. >>>>>> The logs and possibly pstack during those intermittent hangs will >>>>>> help to determine that. >>>>>> >>>>>> regards >>>>>> thierry >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> On 08/29/2016 04:25 PM, Rakesh Rajasekharan wrote: >>>>>> >>>>>> I tried increasing the nsslapd-dbcachesize and nsslapd-cachememsize >>>>>> in my QA envs to 200MB. >>>>>> >>>>>> However, in my log files, I still see this message >>>>>> [29/Aug/2016:04:34:37 +0000] - WARNING: ipaca: entry cache size >>>>>> 10485760B is less than db size 11599872B; We recommend to increase the >>>>>> entry cache size nsslapd-cachememsize. >>>>>> [29/Aug/2016:04:34:37 +0000] - WARNING: changelog: entry cache size >>>>>> 2097152B is less than db size 441647104B; We recommend to increase the >>>>>> entry cache size nsslapd-cachememsize. >>>>>> >>>>>> these are my ldif files that i used to modify the values >>>>>> modify entry cache size >>>>>> cat modify-cache-mem-size.ldif >>>>>> dn: cn=userRoot,cn=ldbm database,cn=plugins,cn=config >>>>>> changetype: modify >>>>>> replace: nsslapd-cachememsize >>>>>> nsslapd-cachememsize: 209715200 >>>>>> >>>>>> modify db cache size >>>>>> cat modfy-db-cache-size.ldif >>>>>> dn: cn=config,cn=ldbm database,cn=plugins,cn=config >>>>>> changetype: modify >>>>>> replace: nsslapd-dbcachesize >>>>>> nsslapd-dbcachesize: 209715200 >>>>>> >>>>>> After modifying , i restarted IPA services >>>>>> >>>>>> Is there anything else that I need to take care of as the logs >>>>>> suggest its still not getting the updated values >>>>>> >>>>>> Thanks >>>>>> Rakesh >>>>>> >>>>>> On Mon, Aug 29, 2016 at 6:07 PM, Rakesh Rajasekharan < >>>>>> rakesh.rajasekharan at gmail.com> wrote: >>>>>> >>>>>>> Hi Thierry, >>>>>>> >>>>>>> Coz of the issues we had to revert back to earlier running openldap >>>>>>> in production. >>>>>>> >>>>>>> I have now done a few TCP related changes in sysctl.conf and have >>>>>>> also increased the nsslapd-dbcachesize and nsslapd-cachememsize to 200MB >>>>>>> >>>>>>> I will again start migrating hosts back to IPA and see if I face the >>>>>>> earlier issue. >>>>>>> >>>>>>> I will update back once I have something >>>>>>> >>>>>>> >>>>>>> Thanks, >>>>>>> Rakesh >>>>>>> >>>>>>> >>>>>>> >>>>>>> On Thu, Aug 25, 2016 at 2:17 PM, thierry bordaz < >>>>>>> tbordaz at redhat.com> wrote: >>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> On 08/25/2016 10:15 AM, Rakesh Rajasekharan wrote: >>>>>>>> >>>>>>>> All of the troubleshooting seems fine. >>>>>>>> >>>>>>>> >>>>>>>> However, Running libconv.pl gives me this output >>>>>>>> >>>>>>>> ----- Recommendations ----- >>>>>>>> >>>>>>>> 1. You have unindexed components, this can be caused from a >>>>>>>> search on an unindexed attribute, or your returned results exceeded the >>>>>>>> allidsthreshold. Unindexed components are not recommended. To refuse >>>>>>>> unindexed searches, switch 'nsslapd-require-index' to 'on' under your >>>>>>>> database entry (e.g. cn=UserRoot,cn=ldbm database,cn=plugins,cn=config) >>>>>>>> . >>>>>>>> >>>>>>>> 2. You have a significant difference between binds and unbinds. >>>>>>>> You may want to investigate this difference. >>>>>>>> >>>>>>>> >>>>>>>> I feel, this could be a pointer to things going slow.. and IPA >>>>>>>> hanging. I think i now have something that I can try and nail down this >>>>>>>> issue. >>>>>>>> >>>>>>>> On a sidenote, I was earlier running openldap and migrated over to >>>>>>>> Freeipa, >>>>>>>> >>>>>>>> Thanks >>>>>>>> Rakesh >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> On Wed, Aug 24, 2016 at 12:38 PM, Petr Spacek < >>>>>>>> pspacek at redhat.com> wrote: >>>>>>>> >>>>>>>>> On 23.8.2016 18:44, Rakesh Rajasekharan wrote: >>>>>>>>> > I think thers something seriously wrong with my system >>>>>>>>> > >>>>>>>>> > not able to run any IPA commands >>>>>>>>> > >>>>>>>>> > klist >>>>>>>>> > Ticket cache: KEYRING:persistent:0:0 >>>>>>>>> > Default principal: admin at XYZ.COM >>>>>>>>> > >>>>>>>>> > Valid starting Expires Service principal >>>>>>>>> > 2016-08-23T16:26:36 2016-08-24T16:26:22 krbtgt/ >>>>>>>>> XYZ.COM at XYZ.COM >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > [root at prod-ipa-master-1a :~] ipactl status >>>>>>>>> > Directory Service: RUNNING >>>>>>>>> > krb5kdc Service: RUNNING >>>>>>>>> > kadmin Service: RUNNING >>>>>>>>> > ipa_memcached Service: RUNNING >>>>>>>>> > httpd Service: RUNNING >>>>>>>>> > pki-tomcatd Service: RUNNING >>>>>>>>> > ipa-otpd Service: RUNNING >>>>>>>>> > ipa: INFO: The ipactl command was successful >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > [root at prod-ipa-master :~] ipa user-find p-testuser >>>>>>>>> > ipa: ERROR: Kerberos error: ('Unspecified GSS failure. Minor >>>>>>>>> code may >>>>>>>>> > provide more information', 851968)/("Cannot contact any KDC for >>>>>>>>> realm ' >>>>>>>>> > XYZ.COM'", -1765328228) >>>>>>>>> >>>>>>>> >>>>>>>> Hi Rakesh, >>>>>>>> >>>>>>>> Having a reproducible test case would you rerun the command above. >>>>>>>> During its processing you may monitor DS process load (top). If it >>>>>>>> is high, you may get some pstacks of it. >>>>>>>> Also would you attach the part of DS access logs taken during the >>>>>>>> command. >>>>>>>> >>>>>>>> regards >>>>>>>> thierry >>>>>>>> >>>>>>>> > >>>>>>>>> >>>>>>>>> This is weird because the server seems to be up. >>>>>>>>> >>>>>>>>> Please follow >>>>>>>>> http://www.freeipa.org/page/Tr >>>>>>>>> oubleshooting#Authentication.2FKerberos >>>>>>>>> >>>>>>>>> Petr^2 Spacek >>>>>>>>> >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > Thanks >>>>>>>>> > >>>>>>>>> > Rakesh >>>>>>>>> > >>>>>>>>> > On Tue, Aug 23, 2016 at 10:01 PM, Rakesh Rajasekharan < >>>>>>>>> > rakesh.rajasekharan at gmail.com> >>>>>>>>> wrote: >>>>>>>>> > >>>>>>>>> >> i changed the loggin level to 4 . Modifying >>>>>>>>> nsslapd-accesslog-level >>>>>>>>> >> >>>>>>>>> >> But, the hang is still there. though I dont see the sigfault now >>>>>>>>> >> >>>>>>>>> >> >>>>>>>>> >> >>>>>>>>> >> >>>>>>>>> >> On Tue, Aug 23, 2016 at 9:02 PM, Rakesh Rajasekharan < >>>>>>>>> >> rakesh.rajasekharan at gmail.com> >>>>>>>>> wrote: >>>>>>>>> >> >>>>>>>>> >>> My disk was getting filled too fast >>>>>>>>> >>> >>>>>>>>> >>> logs under /var/log/dirsrv was coming around 5 gb quickly >>>>>>>>> filling up >>>>>>>>> >>> >>>>>>>>> >>> Is there a way to make the logging less verbose >>>>>>>>> >>> >>>>>>>>> >>> >>>>>>>>> >>> >>>>>>>>> >>> On Tue, Aug 23, 2016 at 6:41 PM, Petr Spacek < >>>>>>>>> pspacek at redhat.com> wrote: >>>>>>>>> >>> >>>>>>>>> >>>> On 23.8.2016 15:07, Rakesh Rajasekharan wrote: >>>>>>>>> >>>>> I was able to fix that may be temporarily... when i checked >>>>>>>>> the >>>>>>>>> >>>> network.. >>>>>>>>> >>>>> there was another process that was running and consuming a >>>>>>>>> lot of >>>>>>>>> >>>> network ( >>>>>>>>> >>>>> i have no idea who did that. I need to seriously start >>>>>>>>> restricting >>>>>>>>> >>>> people >>>>>>>>> >>>>> access to this machine ) >>>>>>>>> >>>>> >>>>>>>>> >>>>> after killing that perfomance improved drastically >>>>>>>>> >>>>> >>>>>>>>> >>>>> But now, suddenly I started experiencing the same hang. >>>>>>>>> >>>>> >>>>>>>>> >>>>> This time , I gert the following error when checked dmesg >>>>>>>>> >>>>> >>>>>>>>> >>>>> [ 301.236976] ns-slapd[3124]: segfault at 0 ip >>>>>>>>> 00007f1de416951c sp >>>>>>>>> >>>>> 00007f1dee1dba70 error 4 in libcos-plugin.so[7f1de4166000+ >>>>>>>>> b000] >>>>>>>>> >>>>> [ 1116.248431] TCP: request_sock_TCP: Possible SYN flooding >>>>>>>>> on port 88. >>>>>>>>> >>>>> Sending cookies. Check SNMP counters. >>>>>>>>> >>>>> [11831.397037] ns-slapd[22550]: segfault at 0 ip >>>>>>>>> 00007f533d82251c sp >>>>>>>>> >>>>> 00007f5347894a70 error 4 in libcos-plugin.so[7f533d81f000+ >>>>>>>>> b000] >>>>>>>>> >>>>> [11832.727989] ns-slapd[22606]: segfault at 0 ip >>>>>>>>> 00007f6231eb951c sp >>>>>>>>> >>>>> 00007f623bf2ba70 error 4 in libcos-plugin.so[7f6231eb6000+ >>>>>>>>> b00 >>>>>>>>> >>>> >>>>>>>>> >>>> Okay, this one is serious. The LDAP server crashed. >>>>>>>>> >>>> >>>>>>>>> >>>> 1. Make sure all your packages are up-to-date. >>>>>>>>> >>>> >>>>>>>>> >>>> Please see >>>>>>>>> >>>> >>>>>>>>> http://directory.fedoraproject.org/docs/389ds/FAQ/faq.html#d >>>>>>>>> >>>> ebugging-crashes >>>>>>>>> >>>> for further instructions how to debug this. >>>>>>>>> >>>> >>>>>>>>> >>>> Petr^2 Spacek >>>>>>>>> >>>> >>>>>>>>> >>>>> >>>>>>>>> >>>>> and in /var/log/dirsrv/example-com/errors >>>>>>>>> >>>>> >>>>>>>>> >>>>> [23/Aug/2016:12:49:36 +0000] DSRetroclPlugin - >>>>>>>>> delete_changerecord: >>>>>>>>> >>>> could >>>>>>>>> >>>>> not delete change record 3291138 (rc: 32) >>>>>>>>> >>>>> [23/Aug/2016:12:49:36 +0000] DSRetroclPlugin - >>>>>>>>> delete_changerecord: >>>>>>>>> >>>> could >>>>>>>>> >>>>> not delete change record 3291139 (rc: 32) >>>>>>>>> >>>>> [23/Aug/2016:12:49:36 +0000] DSRetroclPlugin - >>>>>>>>> delete_changerecord: >>>>>>>>> >>>> could >>>>>>>>> >>>>> not delete change record 3291140 (rc: 32) >>>>>>>>> >>>>> [23/Aug/2016:12:49:36 +0000] DSRetroclPlugin - >>>>>>>>> delete_changerecord: >>>>>>>>> >>>> could >>>>>>>>> >>>>> not delete change record 3291141 (rc: 32) >>>>>>>>> >>>>> [23/Aug/2016:12:49:36 +0000] DSRetroclPlugin - >>>>>>>>> delete_changerecord: >>>>>>>>> >>>> could >>>>>>>>> >>>>> not delete change record 3291142 (rc: 32) >>>>>>>>> >>>>> [23/Aug/2016:12:49:36 +0000] DSRetroclPlugin - >>>>>>>>> delete_changerecord: >>>>>>>>> >>>> could >>>>>>>>> >>>>> not delete change record 3291143 (rc: 32) >>>>>>>>> >>>>> [23/Aug/2016:12:49:36 +0000] DSRetroclPlugin - >>>>>>>>> delete_changerecord: >>>>>>>>> >>>> could >>>>>>>>> >>>>> not delete change record 3291144 (rc: 32) >>>>>>>>> >>>>> [23/Aug/2016:12:49:36 +0000] DSRetroclPlugin - >>>>>>>>> delete_changerecord: >>>>>>>>> >>>> could >>>>>>>>> >>>>> not delete change record 3291145 (rc: 32) >>>>>>>>> >>>>> [23/Aug/2016:12:49:50 +0000] - Retry count exceeded in delete >>>>>>>>> >>>>> [23/Aug/2016:12:49:50 +0000] DSRetroclPlugin - >>>>>>>>> delete_changerecord: >>>>>>>>> >>>> could >>>>>>>>> >>>>> not delete change record 3292734 (rc: 51) >>>>>>>>> >>>>> >>>>>>>>> >>>>> >>>>>>>>> >>>>> Can i do something about this error.. I treid to restart >>>>>>>>> ipa a couple >>>>>>>>> >>>> of >>>>>>>>> >>>>> time but that did not help >>>>>>>>> >>>>> >>>>>>>>> >>>>> Thanks >>>>>>>>> >>>>> Rakesh >>>>>>>>> >>>>> >>>>>>>>> >>>>> On Mon, Aug 22, 2016 at 2:27 PM, Petr Spacek < >>>>>>>>> pspacek at redhat.com> >>>>>>>>> >>>> wrote: >>>>>>>>> >>>>> >>>>>>>>> >>>>>> On 19.8.2016 19:32, Rakesh Rajasekharan wrote: >>>>>>>>> >>>>>>> I am running my set up on AWS cloud, and entropy is low at >>>>>>>>> around >>>>>>>>> >>>> 180 . >>>>>>>>> >>>>>>> >>>>>>>>> >>>>>>> I plan to increase it bu installing haveged . But, would >>>>>>>>> low entropy >>>>>>>>> >>>> by >>>>>>>>> >>>>>> any >>>>>>>>> >>>>>>> chance cause this issue of intermittent hang . >>>>>>>>> >>>>>>> Also, the hang is mostly observed when registering around >>>>>>>>> 20 clients >>>>>>>>> >>>>>>> together >>>>>>>>> >>>>>> >>>>>>>>> >>>>>> Possibly, I'm not sure. If you want to dig into this, I >>>>>>>>> would do this: >>>>>>>>> >>>>>> 1. look what process hangs on client (using pstree command >>>>>>>>> or so) >>>>>>>>> >>>>>> $ pstree >>>>>>>>> >>>>>> >>>>>>>>> >>>>>> 2. look to what server and port is the hanging client >>>>>>>>> connected to >>>>>>>>> >>>>>> $ lsof -p >>>>>>>>> >>>>>> >>>>>>>>> >>>>>> 3. jump to server and see what process is bound to the >>>>>>>>> target port >>>>>>>>> >>>>>> $ netstat -pn >>>>>>>>> >>>>>> >>>>>>>>> >>>>>> 4. see where the process if hanging >>>>>>>>> >>>>>> $ strace -p >>>>>>>>> >>>>>> >>>>>>>>> >>>>>> I hope it helps. >>>>>>>>> >>>>>> >>>>>>>>> >>>>>> Petr^2 Spacek >>>>>>>>> >>>>>> >>>>>>>>> >>>>>>> On Fri, Aug 19, 2016 at 7:24 PM, Rakesh Rajasekharan < >>>>>>>>> >>>>>>> >>>>>>>>> rakesh.rajasekharan at gmail.com> wrote: >>>>>>>>> >>>>>>> >>>>>>>>> >>>>>>>> yes there seems to be something thats worrying.. I have >>>>>>>>> faced this >>>>>>>>> >>>> today >>>>>>>>> >>>>>>>> as well. >>>>>>>>> >>>>>>>> There are few hosts around 280 odd left and when i try >>>>>>>>> adding them >>>>>>>>> >>>> to >>>>>>>>> >>>>>> IPA >>>>>>>>> >>>>>>>> , the slowness begins.. >>>>>>>>> >>>>>>>> >>>>>>>>> >>>>>>>> all the ipa commands like ipa user-find.. etc becomes >>>>>>>>> very slow in >>>>>>>>> >>>>>>>> responding. >>>>>>>>> >>>>>>>> >>>>>>>>> >>>>>>>> the SYNC_RECV are not many though just around 80-90 and >>>>>>>>> today that >>>>>>>>> >>>> was >>>>>>>>> >>>>>>>> around 20 only >>>>>>>>> >>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>>>> >>>>>>>> I have for now increased tcp_max_syn_backlog to 5000. >>>>>>>>> >>>>>>>> For now the slowness seems to have gone.. but I will do a >>>>>>>>> try >>>>>>>>> >>>> adding the >>>>>>>>> >>>>>>>> clients again tomorrow and see how it goes >>>>>>>>> >>>>>>>> >>>>>>>>> >>>>>>>> Thanks >>>>>>>>> >>>>>>>> Rakesh >>>>>>>>> >>>>>>>> >>>>>>>>> >>>>>>>> The issues >>>>>>>>> >>>>>>>> >>>>>>>>> >>>>>>>> On Fri, Aug 19, 2016 at 12:58 PM, Petr Spacek < >>>>>>>>> pspacek at redhat.com> >>>>>>>>> >>>>>> wrote: >>>>>>>>> >>>>>>>> >>>>>>>>> >>>>>>>>> On 18.8.2016 17:23, Rakesh Rajasekharan wrote: >>>>>>>>> >>>>>>>>>> Hi >>>>>>>>> >>>>>>>>>> >>>>>>>>> >>>>>>>>>> I am migrating to freeipa from openldap and have around >>>>>>>>> 4000 >>>>>>>>> >>>> clients >>>>>>>>> >>>>>>>>>> >>>>>>>>> >>>>>>>>>> I had openned a another thread on that, but chose to >>>>>>>>> start a new >>>>>>>>> >>>> one >>>>>>>>> >>>>>>>>> here >>>>>>>>> >>>>>>>>>> as its a separate issue >>>>>>>>> >>>>>>>>>> >>>>>>>>> >>>>>>>>>> I was able to change the nssslapd-maxdescriptors adding >>>>>>>>> an ldif >>>>>>>>> >>>> file >>>>>>>>> >>>>>>>>>> >>>>>>>>> >>>>>>>>>> cat nsslapd-modify.ldif >>>>>>>>> >>>>>>>>>> dn: cn=config >>>>>>>>> >>>>>>>>>> changetype: modify >>>>>>>>> >>>>>>>>>> replace: nsslapd-maxdescriptors >>>>>>>>> >>>>>>>>>> nsslapd-maxdescriptors: 17000 >>>>>>>>> >>>>>>>>>> >>>>>>>>> >>>>>>>>>> and running the ldapmodify command >>>>>>>>> >>>>>>>>>> >>>>>>>>> >>>>>>>>>> I have now started moving clients running an openldap >>>>>>>>> to Freeipa >>>>>>>>> >>>> and >>>>>>>>> >>>>>>>>> have >>>>>>>>> >>>>>>>>>> today moved close to 2000 clients >>>>>>>>> >>>>>>>>>> >>>>>>>>> >>>>>>>>>> However, I have noticed that IPA hangs intermittently. >>>>>>>>> >>>>>>>>>> >>>>>>>>> >>>>>>>>>> running a kinit admin returns the below error >>>>>>>>> >>>>>>>>>> kinit: Generic error (see e-text) while getting initial >>>>>>>>> >>>> credentials >>>>>>>>> >>>>>>>>>> >>>>>>>>> >>>>>>>>>> from the /var/log/messages, I see this entry >>>>>>>>> >>>>>>>>>> >>>>>>>>> >>>>>>>>>> prod-ipa-master-int kernel: [104090.315801] TCP: >>>>>>>>> >>>> request_sock_TCP: >>>>>>>>> >>>>>>>>>> Possible SYN flooding on port 88. Sending cookies. >>>>>>>>> Check SNMP >>>>>>>>> >>>>>> counters. >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> I would be worried about this message. Maybe >>>>>>>>> kernel/firewall is >>>>>>>>> >>>> doing >>>>>>>>> >>>>>>>>> something fishy behind your back and blocking some >>>>>>>>> connections or >>>>>>>>> >>>> so. >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> Petr^2 Spacek >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>>> Aug 18 13:00:01 prod-ipa-master-int systemd[1]: Started >>>>>>>>> Session >>>>>>>>> >>>> 4885 >>>>>>>>> >>>>>> of >>>>>>>>> >>>>>>>>>> user root. >>>>>>>>> >>>>>>>>>> Aug 18 13:00:01 prod-ipa-master-int systemd[1]: >>>>>>>>> Starting Session >>>>>>>>> >>>> 4885 >>>>>>>>> >>>>>> of >>>>>>>>> >>>>>>>>>> user root. >>>>>>>>> >>>>>>>>>> Aug 18 13:01:01 prod-ipa-master-int systemd[1]: Started >>>>>>>>> Session >>>>>>>>> >>>> 4886 >>>>>>>>> >>>>>> of >>>>>>>>> >>>>>>>>>> user root. >>>>>>>>> >>>>>>>>>> Aug 18 13:01:01 prod-ipa-master-int systemd[1]: >>>>>>>>> Starting Session >>>>>>>>> >>>> 4886 >>>>>>>>> >>>>>> of >>>>>>>>> >>>>>>>>>> user root. >>>>>>>>> >>>>>>>>>> Aug 18 13:02:40 prod-ipa-master-int python[28984]: >>>>>>>>> ansible-command >>>>>>>>> >>>>>>>>> Invoked >>>>>>>>> >>>>>>>>>> with creates=None executable=None shell=True args= >>>>>>>>> removes=None >>>>>>>>> >>>>>>>>> warn=True >>>>>>>>> >>>>>>>>>> chdir=None >>>>>>>>> >>>>>>>>>> Aug 18 13:04:37 prod-ipa-master-int sssd_be: GSSAPI >>>>>>>>> Error: >>>>>>>>> >>>> Unspecified >>>>>>>>> >>>>>>>>> GSS >>>>>>>>> >>>>>>>>>> failure. Minor code may provide more information (KDC >>>>>>>>> returned >>>>>>>>> >>>> error >>>>>>>>> >>>>>>>>>> string: PROCESS_TGS) >>>>>>>>> >>>>>>>>>> >>>>>>>>> >>>>>>>>>> Could it be possible that its due to the initial load >>>>>>>>> of adding >>>>>>>>> >>>> the >>>>>>>>> >>>>>>>>> clients >>>>>>>>> >>>>>>>>>> or is there something else that I need to take care of. >>>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>> >>>>>> >>>>>> >>>>> >>>>> >>>>> >>>>> >>>>> -- >>>>> Manage your subscription for the Freeipa-users mailing list: >>>>> https://www.redhat.com/mailman/listinfo/freeipa-users >>>>> Go to http://freeipa.org for more info on the project >>>>> >>>> >>>> >>> >>> >>> >>> >> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From tbordaz at redhat.com Mon Sep 5 12:31:23 2016 From: tbordaz at redhat.com (thierry bordaz) Date: Mon, 5 Sep 2016 14:31:23 +0200 Subject: [Freeipa-users] Freeipa 4.2.0 hangs intermittently In-Reply-To: References: <57BEB0AB.2090506@redhat.com> <57C44AC6.1030402@redhat.com> <57CD1FDC.2010109@redhat.com> <57CD5488.60404@redhat.com> Message-ID: <57CD659B.6050707@redhat.com> Hi Rakesh, Thanks you soooo much for you continuous help on this !! If the hang is still going on, would you dump the DB lock information with: db_stat -CA -N -h /var/lib/dirsrv/slapd-/db >/tmp/db_lock_info It is looking like several threads are waiting for one/several db locks. With the lock info we will know which thread acquired it/them. A guess is that it could be the checkpointing thread but this one is also waiting (on a memory pool). I am copying Ludwig on this thread in case this pstack rings any bell. best regards thierry On 09/05/2016 02:05 PM, Rakesh Rajasekharan wrote: > Hi Thierry, > > I was able to now reporduce the issue.. > > the ns-slapd process just got hung,.. however, no core file was generate.. > > I manually created a coredump using gcore and then took a strace of it > this is strace from the coredump generated > > GNU gdb (GDB) Red Hat Enterprise Linux 7.6.1-80.el7 > Copyright (C) 2013 Free Software Foundation, Inc. > License GPLv3+: GNU GPL version 3 or later > > This is free software: you are free to change and redistribute it. > There is NO WARRANTY, to the extent permitted by law. Type "show copying" > and "show warranty" for details. > This GDB was configured as "x86_64-redhat-linux-gnu". > For bug reporting instructions, please see: > ... > Reading symbols from /usr/sbin/ns-slapd...Reading symbols from > /usr/sbin/ns-slapd...(no debugging symbols found)...done. > (no debugging symbols found)...done. > [New LWP 105159] > [New LWP 105160] > [New LWP 105161] > [New LWP 105162] > [New LWP 105163] > [New LWP 105164] > [New LWP 105165] > [New LWP 105166] > [New LWP 105167] > [New LWP 105168] > [New LWP 105171] > [New LWP 105172] > [New LWP 105173] > [New LWP 105174] > [New LWP 105175] > [New LWP 105176] > [New LWP 105177] > [New LWP 105178] > [New LWP 105179] > [New LWP 105180] > [New LWP 105181] > [New LWP 105182] > [New LWP 105183] > [New LWP 105184] > [New LWP 105185] > [New LWP 105186] > [New LWP 105187] > [New LWP 105188] > [New LWP 105189] > [New LWP 105190] > [New LWP 105191] > [New LWP 105192] > [New LWP 105193] > [New LWP 105194] > [New LWP 105195] > [New LWP 105196] > [New LWP 105197] > [New LWP 105198] > [New LWP 105199] > [New LWP 105200] > [New LWP 105201] > [New LWP 105203] > [New LWP 105215] > [New LWP 105158] > [Thread debugging using libthread_db enabled] > Using host libthread_db library "/lib64/libthread_db.so.1". > Core was generated by `/usr/sbin/ns-slapd'. > #0 0x00007f1b48ae69b3 in select () from /lib64/libc.so.6 > > Thread 44 (Thread 0x7f1b4b6d9840 (LWP 105158)): > #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 () from > /lib64/libpthread.so.0 > No symbol table info available. > #1 0x00007f1b4941b2c3 in PR_EnterMonitor () from /lib64/libnspr4.so > No symbol table info available. > #2 0x00007f1b4b7125ed in setup_pr_read_pds () > No symbol table info available. > #3 0x00007f1b4b715fd9 in slapd_daemon () > No symbol table info available. > #4 0x00007f1b4b707c6c in main () > No symbol table info available. > > Thread 43 (Thread 0x7f1a5df0f700 (LWP 105215)): > #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 () from > /lib64/libpthread.so.0 > No symbol table info available. > #1 0x00007f1b4941b050 in PR_WaitCondVar () from /lib64/libnspr4.so > No symbol table info available. > #2 0x00007f1b4b71df25 in ps_send_results () > No symbol table info available. > #3 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so > No symbol table info available. > #4 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 > No symbol table info available. > #5 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 > No symbol table info available. > > Thread 42 (Thread 0x7f1b03fe7700 (LWP 105203)): > #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 () from > /lib64/libpthread.so.0 > No symbol table info available. > #1 0x00007f1b4941b050 in PR_WaitCondVar () from /lib64/libnspr4.so > No symbol table info available. > #2 0x00007f1b4b71df25 in ps_send_results () > No symbol table info available. > #3 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so > No symbol table info available. > #4 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 > No symbol table info available. > #5 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 > No symbol table info available. > > Thread 41 (Thread 0x7f1b047e8700 (LWP 105201)): > #0 0x00007f1b48ae69b3 in select () from /lib64/libc.so.6 > No symbol table info available. > #1 0x00007f1b4b2710e9 in DS_Sleep () from /usr/lib64/dirsrv/libslapd.so.0 > No symbol table info available. > #2 0x00007f1b4b712325 in time_thread () > No symbol table info available. > #3 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so > No symbol table info available. > #4 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 > No symbol table info available. > #5 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 > No symbol table info available. > > Thread 40 (Thread 0x7f1b04fe9700 (LWP 105200)): > #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 () from > /lib64/libpthread.so.0 > No symbol table info available. > #1 0x00007f1b433212f3 in __db_hybrid_mutex_suspend () from > /lib64/libdb-5.3.so > No symbol table info available. > #2 0x00007f1b43320640 in __db_tas_mutex_lock () from > /lib64/libdb-5.3.so > No symbol table info available. > #3 0x00007f1b4343ae1b in __memp_fget () from /lib64/libdb-5.3.so > > No symbol table info available. > #4 0x00007f1b4333e211 in __bam_search () from /lib64/libdb-5.3.so > > No symbol table info available. > #5 0x00007f1b43329256 in __bamc_search () from /lib64/libdb-5.3.so > > No symbol table info available. > #6 0x00007f1b4332d6c4 in __bamc_put () from /lib64/libdb-5.3.so > > No symbol table info available. > #7 0x00007f1b433e46d5 in __dbc_iput () from /lib64/libdb-5.3.so > > No symbol table info available. > #8 0x00007f1b433dfa0e in __db_put () from /lib64/libdb-5.3.so > > No symbol table info available. > #9 0x00007f1b433f4fa4 in __db_put_pp () from /lib64/libdb-5.3.so > > No symbol table info available. > #10 0x00007f1b3e8cf4cf in idl_new_insert_key () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > No symbol table info available. > #11 0x00007f1b3e8de5b5 in addordel_values_sv.isra.3 () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > No symbol table info available. > #12 0x00007f1b3e8df25a in index_addordel_values_ext_sv () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > No symbol table info available. > #13 0x00007f1b3e8df694 in index_addordel_values_sv () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > No symbol table info available. > #14 0x00007f1b3e8df75a in index_addordel_entry () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > No symbol table info available. > #15 0x00007f1b3e8e25fa in ldbm_back_add () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > No symbol table info available. > #16 0x00007f1b4b1e4cca in op_shared_add () from > /usr/lib64/dirsrv/libslapd.so.0 > No symbol table info available. > #17 0x00007f1b4b1e6020 in do_add () from /usr/lib64/dirsrv/libslapd.so.0 > No symbol table info available. > #18 0x00007f1b4b711303 in connection_threadmain () > No symbol table info available. > #19 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so > No symbol table info available. > #20 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 > No symbol table info available. > #21 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 > No symbol table info available. > > Thread 39 (Thread 0x7f1b057ea700 (LWP 105199)): > #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 () from > /lib64/libpthread.so.0 > No symbol table info available. > #1 0x00007f1b4941b050 in PR_WaitCondVar () from /lib64/libnspr4.so > No symbol table info available. > #2 0x00007f1b4b70efee in connection_wait_for_new_work () > No symbol table info available. > #3 0x00007f1b4b710719 in connection_threadmain () > No symbol table info available. > #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so > No symbol table info available. > #5 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 > No symbol table info available. > #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 > No symbol table info available. > > Thread 38 (Thread 0x7f1b05feb700 (LWP 105198)): > #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 () from > /lib64/libpthread.so.0 > No symbol table info available. > #1 0x00007f1b4941b050 in PR_WaitCondVar () from /lib64/libnspr4.so > No symbol table info available. > #2 0x00007f1b4b70efee in connection_wait_for_new_work () > No symbol table info available. > #3 0x00007f1b4b710719 in connection_threadmain () > No symbol table info available. > #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so > No symbol table info available. > #5 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 > No symbol table info available. > #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 > No symbol table info available. > > Thread 37 (Thread 0x7f1b067ec700 (LWP 105197)): > #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 () from > /lib64/libpthread.so.0 > No symbol table info available. > #1 0x00007f1b4941b050 in PR_WaitCondVar () from /lib64/libnspr4.so > No symbol table info available. > #2 0x00007f1b4b70efee in connection_wait_for_new_work () > No symbol table info available. > #3 0x00007f1b4b710719 in connection_threadmain () > No symbol table info available. > #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so > No symbol table info available. > #5 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 > No symbol table info available. > #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 > No symbol table info available. > > Thread 36 (Thread 0x7f1b06fed700 (LWP 105196)): > #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 () from > /lib64/libpthread.so.0 > No symbol table info available. > #1 0x00007f1b4941b050 in PR_WaitCondVar () from /lib64/libnspr4.so > No symbol table info available. > #2 0x00007f1b4b70efee in connection_wait_for_new_work () > No symbol table info available. > #3 0x00007f1b4b710719 in connection_threadmain () > No symbol table info available. > #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so > No symbol table info available. > #5 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 > No symbol table info available. > #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 > No symbol table info available. > > Thread 35 (Thread 0x7f1b077ee700 (LWP 105195)): > #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 () from > /lib64/libpthread.so.0 > No symbol table info available. > #1 0x00007f1b4941b2c3 in PR_EnterMonitor () from /lib64/libnspr4.so > No symbol table info available. > #2 0x00007f1b3e8c23e6 in dblayer_txn_begin () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > No symbol table info available. > #3 0x00007f1b3e8fe007 in ldbm_back_modify () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > No symbol table info available. > #4 0x00007f1b4b22b541 in op_shared_modify () from > /usr/lib64/dirsrv/libslapd.so.0 > No symbol table info available. > #5 0x00007f1b4b22c004 in modify_internal_pb () from > /usr/lib64/dirsrv/libslapd.so.0 > No symbol table info available. > #6 0x00007f1b3eb48b60 in ipapwd_apply_mods () from > /usr/lib64/dirsrv/plugins/libipa_pwd_extop.so > No symbol table info available. > #7 0x00007f1b3eb4c8e0 in store_new_keys () from > /usr/lib64/dirsrv/plugins/libipa_pwd_extop.so > No symbol table info available. > #8 0x00007f1b3eb4ead0 in ipapwd_extop () from > /usr/lib64/dirsrv/plugins/libipa_pwd_extop.so > No symbol table info available. > #9 0x00007f1b4b23bba2 in plugin_call_exop_plugins () from > /usr/lib64/dirsrv/libslapd.so.0 > No symbol table info available. > #10 0x00007f1b4b717efe in do_extended () > No symbol table info available. > #11 0x00007f1b4b71141a in connection_threadmain () > No symbol table info available. > #12 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so > No symbol table info available. > #13 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 > No symbol table info available. > #14 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 > No symbol table info available. > > Thread 34 (Thread 0x7f1b07fef700 (LWP 105194)): > #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 () from > /lib64/libpthread.so.0 > No symbol table info available. > #1 0x00007f1b4941b050 in PR_WaitCondVar () from /lib64/libnspr4.so > No symbol table info available. > #2 0x00007f1b4b70efee in connection_wait_for_new_work () > No symbol table info available. > #3 0x00007f1b4b710719 in connection_threadmain () > No symbol table info available. > #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so > No symbol table info available. > #5 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 > No symbol table info available. > #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 > No symbol table info available. > > Thread 33 (Thread 0x7f1b087f0700 (LWP 105193)): > #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 () from > /lib64/libpthread.so.0 > No symbol table info available. > #1 0x00007f1b4941b050 in PR_WaitCondVar () from /lib64/libnspr4.so > No symbol table info available. > #2 0x00007f1b4b70efee in connection_wait_for_new_work () > No symbol table info available. > #3 0x00007f1b4b710719 in connection_threadmain () > No symbol table info available. > #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so > No symbol table info available. > #5 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 > No symbol table info available. > #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 > No symbol table info available. > > Thread 32 (Thread 0x7f1b08ff1700 (LWP 105192)): > #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 () from > /lib64/libpthread.so.0 > No symbol table info available. > #1 0x00007f1b4941b050 in PR_WaitCondVar () from /lib64/libnspr4.so > No symbol table info available. > #2 0x00007f1b4b70efee in connection_wait_for_new_work () > No symbol table info available. > #3 0x00007f1b4b710719 in connection_threadmain () > No symbol table info available. > #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so > No symbol table info available. > #5 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 > No symbol table info available. > #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 > No symbol table info available. > > Thread 31 (Thread 0x7f1b097f2700 (LWP 105191)): > #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 () from > /lib64/libpthread.so.0 > No symbol table info available. > #1 0x00007f1b4941b050 in PR_WaitCondVar () from /lib64/libnspr4.so > No symbol table info available. > #2 0x00007f1b4b70efee in connection_wait_for_new_work () > No symbol table info available. > #3 0x00007f1b4b710719 in connection_threadmain () > No symbol table info available. > #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so > No symbol table info available. > #5 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 > No symbol table info available. > #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 > No symbol table info available. > > Thread 30 (Thread 0x7f1b09ff3700 (LWP 105190)): > #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 () from > /lib64/libpthread.so.0 > No symbol table info available. > #1 0x00007f1b433212f3 in __db_hybrid_mutex_suspend () from > /lib64/libdb-5.3.so > No symbol table info available. > #2 0x00007f1b43320640 in __db_tas_mutex_lock () from > /lib64/libdb-5.3.so > No symbol table info available. > #3 0x00007f1b433cad3a in __lock_get_internal () from > /lib64/libdb-5.3.so > No symbol table info available. > #4 0x00007f1b433cb820 in __lock_get () from /lib64/libdb-5.3.so > > No symbol table info available. > #5 0x00007f1b433f7142 in __db_lget () from /lib64/libdb-5.3.so > > No symbol table info available. > #6 0x00007f1b4333e605 in __bam_search () from /lib64/libdb-5.3.so > > No symbol table info available. > #7 0x00007f1b43329256 in __bamc_search () from /lib64/libdb-5.3.so > > No symbol table info available. > #8 0x00007f1b4332ad0f in __bamc_get () from /lib64/libdb-5.3.so > > No symbol table info available. > #9 0x00007f1b433e3ca6 in __dbc_iget () from /lib64/libdb-5.3.so > > No symbol table info available. > #10 0x00007f1b433f2b02 in __dbc_get_pp () from /lib64/libdb-5.3.so > > No symbol table info available. > #11 0x00007f1b3e8ce680 in idl_new_fetch () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > No symbol table info available. > #12 0x00007f1b3e8dcb26 in index_read_ext_allids () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > No symbol table info available. > #13 0x00007f1b3e8c7284 in keys2idl () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > No symbol table info available. > #14 0x00007f1b3e8c79e3 in ava_candidates.isra.0 () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > No symbol table info available. > #15 0x00007f1b3e8c7fd2 in filter_candidates_ext () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > No symbol table info available. > #16 0x00007f1b3e8c9096 in list_candidates () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > No symbol table info available. > #17 0x00007f1b3e8c7f40 in filter_candidates_ext () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > No symbol table info available. > #18 0x00007f1b3e8c9096 in list_candidates () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > No symbol table info available. > #19 0x00007f1b3e8c7f40 in filter_candidates_ext () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > No symbol table info available. > #20 0x00007f1b3e8c9096 in list_candidates () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > No symbol table info available. > #21 0x00007f1b3e8c7f40 in filter_candidates_ext () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > No symbol table info available. > #22 0x00007f1b3e90380c in subtree_candidates () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > No symbol table info available. > #23 0x00007f1b3e904e7f in ldbm_back_search () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > No symbol table info available. > #24 0x00007f1b4b2321d6 in op_shared_search () from > /usr/lib64/dirsrv/libslapd.so.0 > No symbol table info available. > #25 0x00007f1b4b72333e in do_search () > No symbol table info available. > #26 0x00007f1b4b7113c3 in connection_threadmain () > No symbol table info available. > #27 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so > No symbol table info available. > #28 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 > No symbol table info available. > #29 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 > No symbol table info available. > > Thread 29 (Thread 0x7f1b0a7f4700 (LWP 105189)): > #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 () from > /lib64/libpthread.so.0 > No symbol table info available. > #1 0x00007f1b4941b050 in PR_WaitCondVar () from /lib64/libnspr4.so > No symbol table info available. > #2 0x00007f1b4b70efee in connection_wait_for_new_work () > No symbol table info available. > #3 0x00007f1b4b710719 in connection_threadmain () > No symbol table info available. > #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so > No symbol table info available. > #5 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 > No symbol table info available. > #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 > No symbol table info available. > > Thread 28 (Thread 0x7f1b0aff5700 (LWP 105188)): > #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 () from > /lib64/libpthread.so.0 > No symbol table info available. > #1 0x00007f1b4941b050 in PR_WaitCondVar () from /lib64/libnspr4.so > No symbol table info available. > #2 0x00007f1b4b70efee in connection_wait_for_new_work () > No symbol table info available. > #3 0x00007f1b4b710719 in connection_threadmain () > No symbol table info available. > #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so > No symbol table info available. > #5 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 > No symbol table info available. > #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 > No symbol table info available. > > Thread 27 (Thread 0x7f1b0b7f6700 (LWP 105187)): > #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 () from > /lib64/libpthread.so.0 > No symbol table info available. > #1 0x00007f1b4941b050 in PR_WaitCondVar () from /lib64/libnspr4.so > No symbol table info available. > #2 0x00007f1b4b70efee in connection_wait_for_new_work () > No symbol table info available. > #3 0x00007f1b4b710719 in connection_threadmain () > No symbol table info available. > #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so > No symbol table info available. > #5 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 > No symbol table info available. > #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 > No symbol table info available. > > Thread 26 (Thread 0x7f1b0bff7700 (LWP 105186)): > #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 () from > /lib64/libpthread.so.0 > No symbol table info available. > #1 0x00007f1b433212f3 in __db_hybrid_mutex_suspend () from > /lib64/libdb-5.3.so > No symbol table info available. > #2 0x00007f1b43320640 in __db_tas_mutex_lock () from > /lib64/libdb-5.3.so > No symbol table info available. > #3 0x00007f1b433cad3a in __lock_get_internal () from > /lib64/libdb-5.3.so > No symbol table info available. > #4 0x00007f1b433cb820 in __lock_get () from /lib64/libdb-5.3.so > > No symbol table info available. > #5 0x00007f1b433f7142 in __db_lget () from /lib64/libdb-5.3.so > > No symbol table info available. > #6 0x00007f1b4333e605 in __bam_search () from /lib64/libdb-5.3.so > > No symbol table info available. > #7 0x00007f1b43329256 in __bamc_search () from /lib64/libdb-5.3.so > > No symbol table info available. > #8 0x00007f1b4332ad0f in __bamc_get () from /lib64/libdb-5.3.so > > No symbol table info available. > #9 0x00007f1b433e3ca6 in __dbc_iget () from /lib64/libdb-5.3.so > > No symbol table info available. > #10 0x00007f1b433f2b02 in __dbc_get_pp () from /lib64/libdb-5.3.so > > No symbol table info available. > #11 0x00007f1b3e8ce680 in idl_new_fetch () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > No symbol table info available. > #12 0x00007f1b3e8dcb26 in index_read_ext_allids () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > No symbol table info available. > #13 0x00007f1b3e8c7284 in keys2idl () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > No symbol table info available. > #14 0x00007f1b3e8c79e3 in ava_candidates.isra.0 () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > No symbol table info available. > #15 0x00007f1b3e8c7fd2 in filter_candidates_ext () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > No symbol table info available. > #16 0x00007f1b3e8c9096 in list_candidates () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > No symbol table info available. > #17 0x00007f1b3e8c7f40 in filter_candidates_ext () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > No symbol table info available. > #18 0x00007f1b3e90380c in subtree_candidates () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > No symbol table info available. > #19 0x00007f1b3e904e7f in ldbm_back_search () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > No symbol table info available. > #20 0x00007f1b4b2321d6 in op_shared_search () from > /usr/lib64/dirsrv/libslapd.so.0 > No symbol table info available. > #21 0x00007f1b4b24294e in search_internal_callback_pb () from > /usr/lib64/dirsrv/libslapd.so.0 > No symbol table info available. > #22 0x00007f1b4b242be8 in search_internal_pb () from > /usr/lib64/dirsrv/libslapd.so.0 > No symbol table info available. > #23 0x00007f1b4b72149b in ids_sasl_canon_user () > No symbol table info available. > #24 0x00007f1b491e193b in _sasl_canon_user () from /lib64/libsasl2.so.3 > No symbol table info available. > #25 0x00007f1b491e1c4c in _sasl_canon_user_lookup () from > /lib64/libsasl2.so.3 > No symbol table info available. > #26 0x00007f1b430e9fb9 in gssapi_server_mech_step () from > /usr/lib64/sasl2/libgssapiv2.so > No symbol table info available. > #27 0x00007f1b491edb9b in sasl_server_step () from /lib64/libsasl2.so.3 > No symbol table info available. > #28 0x00007f1b4b72283b in ids_sasl_check_bind () > No symbol table info available. > #29 0x00007f1b4b70a61a in do_bind () > No symbol table info available. > #30 0x00007f1b4b7113fd in connection_threadmain () > No symbol table info available. > #31 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so > No symbol table info available. > #32 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 > No symbol table info available. > #33 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 > No symbol table info available. > > Thread 25 (Thread 0x7f1b0c7f8700 (LWP 105185)): > #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 () from > /lib64/libpthread.so.0 > No symbol table info available. > #1 0x00007f1b4941b050 in PR_WaitCondVar () from /lib64/libnspr4.so > No symbol table info available. > #2 0x00007f1b4b70efee in connection_wait_for_new_work () > No symbol table info available. > #3 0x00007f1b4b710719 in connection_threadmain () > No symbol table info available. > #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so > No symbol table info available. > #5 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 > No symbol table info available. > #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 > No symbol table info available. > > Thread 24 (Thread 0x7f1b0cff9700 (LWP 105184)): > #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 () from > /lib64/libpthread.so.0 > No symbol table info available. > #1 0x00007f1b4941b050 in PR_WaitCondVar () from /lib64/libnspr4.so > No symbol table info available. > #2 0x00007f1b4b70efee in connection_wait_for_new_work () > No symbol table info available. > #3 0x00007f1b4b710719 in connection_threadmain () > No symbol table info available. > #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so > No symbol table info available. > #5 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 > No symbol table info available. > #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 > No symbol table info available. > > Thread 23 (Thread 0x7f1b0d7fa700 (LWP 105183)): > #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 () from > /lib64/libpthread.so.0 > No symbol table info available. > #1 0x00007f1b4941b050 in PR_WaitCondVar () from /lib64/libnspr4.so > No symbol table info available. > #2 0x00007f1b4b70efee in connection_wait_for_new_work () > No symbol table info available. > #3 0x00007f1b4b710719 in connection_threadmain () > No symbol table info available. > #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so > No symbol table info available. > #5 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 > No symbol table info available. > #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 > No symbol table info available. > > Thread 22 (Thread 0x7f1b0dffb700 (LWP 105182)): > #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 () from > /lib64/libpthread.so.0 > No symbol table info available. > #1 0x00007f1b4941b050 in PR_WaitCondVar () from /lib64/libnspr4.so > No symbol table info available. > #2 0x00007f1b4b70efee in connection_wait_for_new_work () > No symbol table info available. > #3 0x00007f1b4b710719 in connection_threadmain () > No symbol table info available. > #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so > No symbol table info available. > #5 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 > No symbol table info available. > #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 > No symbol table info available. > > Thread 21 (Thread 0x7f1b0e7fc700 (LWP 105181)): > #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 () from > /lib64/libpthread.so.0 > No symbol table info available. > #1 0x00007f1b4941b050 in PR_WaitCondVar () from /lib64/libnspr4.so > No symbol table info available. > #2 0x00007f1b4b70efee in connection_wait_for_new_work () > No symbol table info available. > #3 0x00007f1b4b710719 in connection_threadmain () > No symbol table info available. > #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so > No symbol table info available. > #5 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 > No symbol table info available. > #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 > No symbol table info available. > > Thread 20 (Thread 0x7f1b0effd700 (LWP 105180)): > #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 () from > /lib64/libpthread.so.0 > No symbol table info available. > #1 0x00007f1b433212f3 in __db_hybrid_mutex_suspend () from > /lib64/libdb-5.3.so > No symbol table info available. > #2 0x00007f1b43320b37 in __db_tas_mutex_readlock () from > /lib64/libdb-5.3.so > No symbol table info available. > #3 0x00007f1b4343ad35 in __memp_fget () from /lib64/libdb-5.3.so > > No symbol table info available. > #4 0x00007f1b433e4333 in __dbc_iget () from /lib64/libdb-5.3.so > > No symbol table info available. > #5 0x00007f1b433f2b02 in __dbc_get_pp () from /lib64/libdb-5.3.so > > No symbol table info available. > #6 0x00007f1b3e8ce822 in idl_new_fetch () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > No symbol table info available. > #7 0x00007f1b3e8dcb26 in index_read_ext_allids () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > No symbol table info available. > #8 0x00007f1b3e8c7284 in keys2idl () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > No symbol table info available. > #9 0x00007f1b3e8c79e3 in ava_candidates.isra.0 () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > No symbol table info available. > #10 0x00007f1b3e8c7fd2 in filter_candidates_ext () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > No symbol table info available. > #11 0x00007f1b3e8c9096 in list_candidates () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > No symbol table info available. > #12 0x00007f1b3e8c7f40 in filter_candidates_ext () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > No symbol table info available. > #13 0x00007f1b3e8c9096 in list_candidates () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > No symbol table info available. > #14 0x00007f1b3e8c7f40 in filter_candidates_ext () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > No symbol table info available. > #15 0x00007f1b3e8c9096 in list_candidates () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > No symbol table info available. > #16 0x00007f1b3e8c7f40 in filter_candidates_ext () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > No symbol table info available. > #17 0x00007f1b3e90380c in subtree_candidates () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > No symbol table info available. > #18 0x00007f1b3e904e7f in ldbm_back_search () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > No symbol table info available. > #19 0x00007f1b4b2321d6 in op_shared_search () from > /usr/lib64/dirsrv/libslapd.so.0 > No symbol table info available. > #20 0x00007f1b4b72333e in do_search () > No symbol table info available. > #21 0x00007f1b4b7113c3 in connection_threadmain () > No symbol table info available. > #22 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so > No symbol table info available. > #23 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 > No symbol table info available. > #24 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 > No symbol table info available. > > Thread 19 (Thread 0x7f1b0f7fe700 (LWP 105179)): > #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 () from > /lib64/libpthread.so.0 > No symbol table info available. > #1 0x00007f1b433212f3 in __db_hybrid_mutex_suspend () from > /lib64/libdb-5.3.so > No symbol table info available. > #2 0x00007f1b43320640 in __db_tas_mutex_lock () from > /lib64/libdb-5.3.so > No symbol table info available. > #3 0x00007f1b433cad3a in __lock_get_internal () from > /lib64/libdb-5.3.so > No symbol table info available. > #4 0x00007f1b433cb820 in __lock_get () from /lib64/libdb-5.3.so > > No symbol table info available. > #5 0x00007f1b433f7142 in __db_lget () from /lib64/libdb-5.3.so > > No symbol table info available. > #6 0x00007f1b4333e605 in __bam_search () from /lib64/libdb-5.3.so > > No symbol table info available. > #7 0x00007f1b43329256 in __bamc_search () from /lib64/libdb-5.3.so > > No symbol table info available. > #8 0x00007f1b4332ad0f in __bamc_get () from /lib64/libdb-5.3.so > > No symbol table info available. > #9 0x00007f1b433e3ca6 in __dbc_iget () from /lib64/libdb-5.3.so > > No symbol table info available. > #10 0x00007f1b433f2b02 in __dbc_get_pp () from /lib64/libdb-5.3.so > > No symbol table info available. > #11 0x00007f1b3e8ce680 in idl_new_fetch () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > No symbol table info available. > #12 0x00007f1b3e8dcb26 in index_read_ext_allids () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > No symbol table info available. > #13 0x00007f1b3e8c7284 in keys2idl () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > No symbol table info available. > #14 0x00007f1b3e8c79e3 in ava_candidates.isra.0 () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > No symbol table info available. > #15 0x00007f1b3e8c7fd2 in filter_candidates_ext () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > No symbol table info available. > #16 0x00007f1b3e8c9096 in list_candidates () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > No symbol table info available. > #17 0x00007f1b3e8c7f40 in filter_candidates_ext () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > No symbol table info available. > #18 0x00007f1b3e8c9096 in list_candidates () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > No symbol table info available. > #19 0x00007f1b3e8c7f40 in filter_candidates_ext () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > No symbol table info available. > #20 0x00007f1b3e8c9096 in list_candidates () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > No symbol table info available. > #21 0x00007f1b3e8c7f40 in filter_candidates_ext () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > No symbol table info available. > #22 0x00007f1b3e90380c in subtree_candidates () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > No symbol table info available. > #23 0x00007f1b3e904e7f in ldbm_back_search () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > No symbol table info available. > #24 0x00007f1b4b2321d6 in op_shared_search () from > /usr/lib64/dirsrv/libslapd.so.0 > No symbol table info available. > #25 0x00007f1b4b72333e in do_search () > No symbol table info available. > #26 0x00007f1b4b7113c3 in connection_threadmain () > No symbol table info available. > #27 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so > No symbol table info available. > #28 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 > No symbol table info available. > #29 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 > No symbol table info available. > > Thread 18 (Thread 0x7f1b0ffff700 (LWP 105178)): > #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 () from > /lib64/libpthread.so.0 > No symbol table info available. > #1 0x00007f1b4941b050 in PR_WaitCondVar () from /lib64/libnspr4.so > No symbol table info available. > #2 0x00007f1b4b70efee in connection_wait_for_new_work () > No symbol table info available. > #3 0x00007f1b4b710719 in connection_threadmain () > No symbol table info available. > #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so > No symbol table info available. > #5 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 > No symbol table info available. > #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 > No symbol table info available. > > Thread 17 (Thread 0x7f1b14d28700 (LWP 105177)): > #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 () from > /lib64/libpthread.so.0 > No symbol table info available. > #1 0x00007f1b4941b050 in PR_WaitCondVar () from /lib64/libnspr4.so > No symbol table info available. > #2 0x00007f1b4b70efee in connection_wait_for_new_work () > No symbol table info available. > #3 0x00007f1b4b710719 in connection_threadmain () > No symbol table info available. > #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so > No symbol table info available. > #5 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 > No symbol table info available. > #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 > No symbol table info available. > > Thread 16 (Thread 0x7f1b15529700 (LWP 105176)): > #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 () from > /lib64/libpthread.so.0 > No symbol table info available. > #1 0x00007f1b4941b050 in PR_WaitCondVar () from /lib64/libnspr4.so > No symbol table info available. > #2 0x00007f1b4b70efee in connection_wait_for_new_work () > No symbol table info available. > #3 0x00007f1b4b710719 in connection_threadmain () > No symbol table info available. > #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so > No symbol table info available. > #5 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 > No symbol table info available. > #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 > No symbol table info available. > > Thread 15 (Thread 0x7f1b15d2a700 (LWP 105175)): > #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 () from > /lib64/libpthread.so.0 > No symbol table info available. > #1 0x00007f1b4941b050 in PR_WaitCondVar () from /lib64/libnspr4.so > No symbol table info available. > #2 0x00007f1b4b70efee in connection_wait_for_new_work () > No symbol table info available. > #3 0x00007f1b4b710719 in connection_threadmain () > No symbol table info available. > #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so > No symbol table info available. > #5 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 > No symbol table info available. > #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 > No symbol table info available. > > Thread 14 (Thread 0x7f1b1652b700 (LWP 105174)): > #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 () from > /lib64/libpthread.so.0 > No symbol table info available. > #1 0x00007f1b4941b050 in PR_WaitCondVar () from /lib64/libnspr4.so > No symbol table info available. > #2 0x00007f1b4b70efee in connection_wait_for_new_work () > No symbol table info available. > #3 0x00007f1b4b710719 in connection_threadmain () > No symbol table info available. > #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so > No symbol table info available. > #5 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 > No symbol table info available. > #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 > No symbol table info available. > > Thread 13 (Thread 0x7f1b16d2c700 (LWP 105173)): > #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 () from > /lib64/libpthread.so.0 > No symbol table info available. > #1 0x00007f1b433212f3 in __db_hybrid_mutex_suspend () from > /lib64/libdb-5.3.so > No symbol table info available. > #2 0x00007f1b43320640 in __db_tas_mutex_lock () from > /lib64/libdb-5.3.so > No symbol table info available. > #3 0x00007f1b433cad3a in __lock_get_internal () from > /lib64/libdb-5.3.so > No symbol table info available. > #4 0x00007f1b433cb820 in __lock_get () from /lib64/libdb-5.3.so > > No symbol table info available. > #5 0x00007f1b433f7142 in __db_lget () from /lib64/libdb-5.3.so > > No symbol table info available. > #6 0x00007f1b4333e605 in __bam_search () from /lib64/libdb-5.3.so > > No symbol table info available. > #7 0x00007f1b43329256 in __bamc_search () from /lib64/libdb-5.3.so > > No symbol table info available. > #8 0x00007f1b4332ad0f in __bamc_get () from /lib64/libdb-5.3.so > > No symbol table info available. > #9 0x00007f1b433e3ca6 in __dbc_iget () from /lib64/libdb-5.3.so > > No symbol table info available. > #10 0x00007f1b433f2b02 in __dbc_get_pp () from /lib64/libdb-5.3.so > > No symbol table info available. > #11 0x00007f1b3e8ce680 in idl_new_fetch () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > No symbol table info available. > #12 0x00007f1b3e8dcb26 in index_read_ext_allids () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > No symbol table info available. > #13 0x00007f1b3e8c7284 in keys2idl () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > No symbol table info available. > #14 0x00007f1b3e8c79e3 in ava_candidates.isra.0 () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > No symbol table info available. > #15 0x00007f1b3e8c7fd2 in filter_candidates_ext () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > No symbol table info available. > #16 0x00007f1b3e8c9096 in list_candidates () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > No symbol table info available. > #17 0x00007f1b3e8c7f40 in filter_candidates_ext () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > No symbol table info available. > #18 0x00007f1b3e8c9096 in list_candidates () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > No symbol table info available. > #19 0x00007f1b3e8c7f40 in filter_candidates_ext () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > No symbol table info available. > #20 0x00007f1b3e8c9096 in list_candidates () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > No symbol table info available. > #21 0x00007f1b3e8c7f40 in filter_candidates_ext () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > No symbol table info available. > #22 0x00007f1b3e90380c in subtree_candidates () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > No symbol table info available. > #23 0x00007f1b3e904e7f in ldbm_back_search () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > No symbol table info available. > #24 0x00007f1b4b2321d6 in op_shared_search () from > /usr/lib64/dirsrv/libslapd.so.0 > No symbol table info available. > #25 0x00007f1b4b72333e in do_search () > No symbol table info available. > #26 0x00007f1b4b7113c3 in connection_threadmain () > No symbol table info available. > #27 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so > No symbol table info available. > #28 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 > No symbol table info available. > #29 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 > No symbol table info available. > > Thread 12 (Thread 0x7f1b1752d700 (LWP 105172)): > #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 () from > /lib64/libpthread.so.0 > No symbol table info available. > #1 0x00007f1b4941b050 in PR_WaitCondVar () from /lib64/libnspr4.so > No symbol table info available. > #2 0x00007f1b4b70efee in connection_wait_for_new_work () > No symbol table info available. > #3 0x00007f1b4b710719 in connection_threadmain () > No symbol table info available. > #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so > No symbol table info available. > #5 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 > No symbol table info available. > #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 > No symbol table info available. > > Thread 11 (Thread 0x7f1b17d2e700 (LWP 105171)): > #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 () from > /lib64/libpthread.so.0 > No symbol table info available. > #1 0x00007f1b4941b050 in PR_WaitCondVar () from /lib64/libnspr4.so > No symbol table info available. > #2 0x00007f1b4b70efee in connection_wait_for_new_work () > No symbol table info available. > #3 0x00007f1b4b710719 in connection_threadmain () > No symbol table info available. > #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so > No symbol table info available. > #5 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 > No symbol table info available. > #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 > No symbol table info available. > > Thread 10 (Thread 0x7f1b2509d700 (LWP 105168)): > #0 0x00007f1b48dc5a82 in pthread_cond_timedwait@@GLIBC_2.3.2 () from > /lib64/libpthread.so.0 > No symbol table info available. > #1 0x00007f1b4941ab07 in pt_TimedWait () from /lib64/libnspr4.so > No symbol table info available. > #2 0x00007f1b4941afce in PR_WaitCondVar () from /lib64/libnspr4.so > No symbol table info available. > #3 0x00007f1b4b2061e8 in eq_loop () from /usr/lib64/dirsrv/libslapd.so.0 > No symbol table info available. > #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so > No symbol table info available. > #5 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 > No symbol table info available. > #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 > No symbol table info available. > > Thread 9 (Thread 0x7f1b2589e700 (LWP 105167)): > #0 0x00007f1b48dc5a82 in pthread_cond_timedwait@@GLIBC_2.3.2 () from > /lib64/libpthread.so.0 > No symbol table info available. > #1 0x00007f1b4941ab07 in pt_TimedWait () from /lib64/libnspr4.so > No symbol table info available. > #2 0x00007f1b4941afce in PR_WaitCondVar () from /lib64/libnspr4.so > No symbol table info available. > #3 0x00007f1b4b719d03 in housecleaning () > No symbol table info available. > #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so > No symbol table info available. > #5 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 > No symbol table info available. > #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 > No symbol table info available. > > Thread 8 (Thread 0x7f1b2609f700 (LWP 105166)): > #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 () from > /lib64/libpthread.so.0 > No symbol table info available. > #1 0x00007f1b4941b050 in PR_WaitCondVar () from /lib64/libnspr4.so > No symbol table info available. > #2 0x00007f1b4b260198 in slapi_wait_condvar () from > /usr/lib64/dirsrv/libslapd.so.0 > No symbol table info available. > #3 0x00007f1b3cf5deed in roles_cache_wait_on_change () from > /usr/lib64/dirsrv/plugins/libroles-plugin.so > No symbol table info available. > #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so > No symbol table info available. > #5 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 > No symbol table info available. > #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 > No symbol table info available. > > Thread 7 (Thread 0x7f1b268a0700 (LWP 105165)): > #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 () from > /lib64/libpthread.so.0 > No symbol table info available. > #1 0x00007f1b4941b050 in PR_WaitCondVar () from /lib64/libnspr4.so > No symbol table info available. > #2 0x00007f1b4b260198 in slapi_wait_condvar () from > /usr/lib64/dirsrv/libslapd.so.0 > No symbol table info available. > #3 0x00007f1b3cf5deed in roles_cache_wait_on_change () from > /usr/lib64/dirsrv/plugins/libroles-plugin.so > No symbol table info available. > #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so > No symbol table info available. > #5 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 > No symbol table info available. > #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 > No symbol table info available. > > Thread 6 (Thread 0x7f1b270a1700 (LWP 105164)): > #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 () from > /lib64/libpthread.so.0 > No symbol table info available. > #1 0x00007f1b4941b050 in PR_WaitCondVar () from /lib64/libnspr4.so > No symbol table info available. > #2 0x00007f1b4b260198 in slapi_wait_condvar () from > /usr/lib64/dirsrv/libslapd.so.0 > No symbol table info available. > #3 0x00007f1b3cf5deed in roles_cache_wait_on_change () from > /usr/lib64/dirsrv/plugins/libroles-plugin.so > No symbol table info available. > #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so > No symbol table info available. > #5 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 > No symbol table info available. > #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 > No symbol table info available. > > Thread 5 (Thread 0x7f1b278a2700 (LWP 105163)): > #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 () from > /lib64/libpthread.so.0 > No symbol table info available. > #1 0x00007f1b4941b050 in PR_WaitCondVar () from /lib64/libnspr4.so > No symbol table info available. > #2 0x00007f1b4b260198 in slapi_wait_condvar () from > /usr/lib64/dirsrv/libslapd.so.0 > No symbol table info available. > #3 0x00007f1b415f262e in cos_cache_wait_on_change () from > /usr/lib64/dirsrv/plugins/libcos-plugin.so > No symbol table info available. > #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so > No symbol table info available. > #5 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 > No symbol table info available. > #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 > No symbol table info available. > > Thread 4 (Thread 0x7f1b282ac700 (LWP 105162)): > #0 0x00007f1b48ae69b3 in select () from /lib64/libc.so.6 > No symbol table info available. > #1 0x00007f1b4b2710e9 in DS_Sleep () from /usr/lib64/dirsrv/libslapd.so.0 > No symbol table info available. > #2 0x00007f1b3e8ba707 in perf_threadmain () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > No symbol table info available. > #3 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so > No symbol table info available. > #4 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 > No symbol table info available. > #5 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 > No symbol table info available. > > Thread 3 (Thread 0x7f1b28aad700 (LWP 105161)): > #0 0x00007f1b48ae69b3 in select () from /lib64/libc.so.6 > No symbol table info available. > #1 0x00007f1b4b2710e9 in DS_Sleep () from /usr/lib64/dirsrv/libslapd.so.0 > No symbol table info available. > #2 0x00007f1b3e8bfb7f in trickle_threadmain () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > No symbol table info available. > #3 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so > No symbol table info available. > #4 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 > No symbol table info available. > #5 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 > No symbol table info available. > > Thread 2 (Thread 0x7f1b292ae700 (LWP 105160)): > #0 0x00007f1b48ae69b3 in select () from /lib64/libc.so.6 > No symbol table info available. > #1 0x00007f1b4344dbad in __os_yield () from /lib64/libdb-5.3.so > > No symbol table info available. > #2 0x00007f1b434492b3 in __memp_sync_int () from /lib64/libdb-5.3.so > > No symbol table info available. > #3 0x00007f1b43459752 in __txn_checkpoint () from /lib64/libdb-5.3.so > > No symbol table info available. > #4 0x00007f1b43459b74 in __txn_checkpoint_pp () from > /lib64/libdb-5.3.so > No symbol table info available. > #5 0x00007f1b3e8c3a87 in checkpoint_threadmain () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > No symbol table info available. > #6 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so > No symbol table info available. > #7 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 > No symbol table info available. > #8 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 > No symbol table info available. > > Thread 1 (Thread 0x7f1b29aaf700 (LWP 105159)): > #0 0x00007f1b48ae69b3 in select () from /lib64/libc.so.6 > No symbol table info available. > #1 0x00007f1b4b2710e9 in DS_Sleep () from /usr/lib64/dirsrv/libslapd.so.0 > No symbol table info available. > #2 0x00007f1b3e8bf907 in deadlock_threadmain () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > No symbol table info available. > #3 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so > No symbol table info available. > #4 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 > No symbol table info available. > #5 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 > No symbol table info available. > > > > Thanks, > > Rakesh > > > On Mon, Sep 5, 2016 at 4:48 PM, thierry bordaz > wrote: > > > > On 09/05/2016 12:05 PM, Rakesh Rajasekharan wrote: >> Hi Thierry, >> >> >> I was getting the hang issue while running ipa-client-install >> simultaneously on few clients.. >> However, today, I am not able to replicate that. >> >> I could not get a gdb . But i will try getting that the next time >> I face this issue. >> >> The CPU does not stay high.. it just momentarily touches a high >> value and then drops down to around 2-7% >> >> One question I have is , is it ok to set it nsslapd-threadnumber >> to a very high value . >> I have around 4000 clients and with nsslapd-maxthreadsperconn set >> to 5..So, can I set nsslapd-threadnumber to around 25000. > > Hello, > > I know some users running in production with several hunderds of > threads (>600) and this without problem. > > I do not recall having suggested to increase that number and for > what reason. > Usually 30 workers is a good enough value. It can create > bootleneck if for some reason each operation is very long to > satisfy and exhaust the number of workers. You can monitor the > work queue: > > ldapsearch -D "cn=directory manager" -w xxx -LLL -b > "cn=monitor" -s base opsinitiated opscompleted > > > If opscompleted-opsinitiated remains close to threadnumber, then > yes it would be valuable to increase it. > > The computation #client * #async_op_per_client sound an overkill. > Even if all clients send at the exact same time all their > requests, it is very likely that some common resource (db page, > log, allocator...) will serialize them. If you monitor a need to > increase the work, you would for example set it to 50, then > monitor, then set it to 100, then monitor... until you find a good > enough value. > Note the increasing the #thread, increases the memory footprint > that will reduce the efficiency of file system cache and can > increase the response time. > > > best regards > thierry > >> >> Thanks >> >> On Mon, Sep 5, 2016 at 1:03 PM, thierry bordaz >> > wrote: >> >> >> Hi Rakesh, >> >> Were you able to get a pstack or full stack with gdb >> (http://www.port389.org/docs/389ds/FAQ/faq.html#debugging-crashes) >> when the server hangs ? >> >> If it happens with 500 threads as well as with 30, using 30 >> threads is a better choice to debug this issue. >> I will try to reproduce using 150 parallel 'ipa user-find >> p-testipa' commands >> >> Something I am unsure is if the CPU consumption stays high >> (you mentioned 340% CPU usage) as long as the hang happens or >> if after a suddent shot up to 340% (that marks the beginning >> of the hang) it drops and stay hanging ? >> >> thanks >> thierry >> >> On 09/04/2016 08:40 PM, Rakesh Rajasekharan wrote: >>> starce on the slapd process actually had this in the output.. >>> FUTEX_WAIT_PRIVATE >>> >>> and checking for the number of threads slapd had.. there >>> were 5015 threads >>> >>> ps -efL|grep slapd|wc -l >>> 5015 >>> >>> strace on most of the threads gave this output >>> >>> strace -p 67411 >>> Process 67411 attached >>> futex(0x7f3f0226b41c, FUTEX_WAIT_PRIVATE, 1, NULL) = -1 >>> EAGAIN (Resource temporarily unavailable) >>> futex(0x7f3f0226b41c, FUTEX_WAIT_PRIVATE, 2, NULL^CProcess >>> 67411 detached >>> >>> >>> >>> >>> >>> On Sun, Sep 4, 2016 at 5:34 PM, Rakesh Rajasekharan >>> >> > wrote: >>> >>> I have again got the issue of IPA hanging.. The issue >>> came up when i tried to run ipa-client-isntall on 142 >>> clients simultaneously >>> >>> >>> None of the IPA commands are responding, and I see this >>> error >>> >>> ipa user-find p-testipa >>> ipa: ERROR: Insufficient access: SASL(-1): generic >>> failure: GSSAPI Error: Unspecified GSS failure. Minor >>> code may provide more information (KDC returned error >>> string: PROCESS_TGS) >>> >>> KRB5_TRACE=/dev/stdout kinit admin >>> [41178] 1472984115.233214: Getting initial credentials >>> for admin at XYZ.COM >>> [41178] 1472984115.235257: Sending request (167 bytes) >>> to XYZ.COM >>> [41178] 1472984115.235419: Initiating TCP connection to >>> stream 10.1.3.36:88 >>> [41178] 1472984115.235685: Sending TCP request to stream >>> 10.1.3.36:88 >>> [41178] 1472984120.238914: Received answer (174 bytes) >>> from stream 10.1.3.36:88 >>> [41178] 1472984120.238925: Terminating TCP connection to >>> stream 10.1.3.36:88 >>> [41178] 1472984120.238993: Response was from master KDC >>> [41 >>> >>> >>> Running an ldapsearch to see the db.. does not give any >>> results and just hangs there >>> >>> ldapsearch -x -D 'cn=Directory Manager' -W -s one -b >>> 'cn=kerberos,dc=xyz,dc=com' >>> Enter LDAP Password: >>> >>> even an ldapsearch -x does not respond >>> At this point, am sure that slapd is the one causing issues >>> >>> Running an strace against the hung slapd itself seems to >>> get stuck does not proceed after saying "attaching to >>> process" >>> >>> From some others posts I read Thierry suggesting to >>> increase the nsslapd-threadnumber value >>> >>> It was set to 30, I think that might be too low. >>> >>> I have raised it to 500 >>> >>> Now after restarting the service .. ldapsearch starts >>> responding. >>> But running the test to add a sudden high number of >>> clients again left ns-slapd to hung state >>> >>> When i attempted adding the clients.. the ns-slapd cpu >>> usage shot up to 340% and after that ns-slapd stopped >>> responding >>> >>> So now, atleast I know what might be causing the issue >>> and I can now easily reproduce it. >>> >>> Is there a way I can make ns-slapd handle a sudden bump >>> in incoming request for ipa-client-install >>> >>> Thanks >>> Rakesh >>> >>> >>> >>> >>> >>> >>> On Mon, Aug 29, 2016 at 11:18 PM, Rich Megginson >>> > wrote: >>> >>> On 08/29/2016 10:53 AM, Rakesh Rajasekharan wrote: >>>> Hi Thierry, >>>> >>>> My machine has 30GB RAM ..and 389-ds version is 1.3.4 >>>> >>>> ldapsearch shows the values for >>>> nsslapd-cachememsize updated to 200MB. >>>> >>>> ldapsearch -LLL -o ldif-wrap=no -D "cn=directory >>>> manager" -w 'mypassword' -b 'cn=userRoot,cn=ldbm >>>> database,cn=plugins,cn=config'|grep >>>> nsslapd-cachememsize >>>> nsslapd-cachememsize: 209715200 >>>> >>>> >>>> So, it seems to have updated though seeing that >>>> warning(WARNING: ipaca: entry cache size 10485760B >>>> is less than db size 11599872B) in the log confuses >>>> me a bit. >>>> >>>> Thers one more entry that I found from the >>>> ldapsearch to be bit low >>>> >>>> nsslapd-dncachememsize: 10485760 >>>> maxdncachesize: 10485760 >>>> >>>> Should I update these as well to a higher value >>>> >>>> At the time when the issue happened, the memory >>>> usage as well as the overall load of the system was >>>> very low . >>>> I will try reproducing the issue atleast in my QA >>>> env..probably by trying to mock simultaneous >>>> parallel logins to a large number of hosts >>> >>> To monitor your cache sizes, please use the dbmon.sh >>> tool provided with your distro. If that is not >>> available with your particular distro, see >>> https://github.com/richm/scripts/wiki/dbmon.sh >>> >>> >>> >>>> >>>> >>>> thanks >>>> Rakesh >>>> >>>> >>>> >>>> >>>> On Mon, Aug 29, 2016 at 8:16 PM, thierry bordaz >>>> > wrote: >>>> >>>> Hi Rakesh, >>>> >>>> Those tuning may depend on the memory available >>>> on your machine. >>>> nsslapd-cachememsize allows the entry cache to >>>> consume up to 200Mb but its memory footprint is >>>> known to go above. >>>> 200Mb both looks pretty good to me. How large >>>> is your machine ? What is your version of 389-ds ? >>>> >>>> Those warnings do not change your settings. It >>>> just raise that entry cache of 'ipaca' and >>>> 'retrocl' are small but it is fine. The size of >>>> the entry cache is important mostly in userRoot. >>>> You may double check the actual values, after >>>> restart, with ldapsearch on >>>> 'cn=userRoot,cn=ldbm >>>> database,cn=plugins,cn=config' and >>>> 'cn=config,cn=ldbm database,cn=plugins,cn=config'. >>>> >>>> A step is to know what will be response time of >>>> DS to know if it is responsible of the hang or not. >>>> The logs and possibly pstack during those >>>> intermittent hangs will help to determine that. >>>> >>>> regards >>>> thierry >>>> >>>> >>>> >>>> >>>> >>>> On 08/29/2016 04:25 PM, Rakesh Rajasekharan wrote: >>>>> I tried increasing the nsslapd-dbcachesize and >>>>> nsslapd-cachememsize in my QA envs to 200MB. >>>>> >>>>> However, in my log files, I still see this message >>>>> [29/Aug/2016:04:34:37 +0000] - WARNING: ipaca: >>>>> entry cache size 10485760B is less than db >>>>> size 11599872B; We recommend to increase the >>>>> entry cache size nsslapd-cachememsize. >>>>> [29/Aug/2016:04:34:37 +0000] - WARNING: >>>>> changelog: entry cache size 2097152B is less >>>>> than db size 441647104B; We recommend to >>>>> increase the entry cache size >>>>> nsslapd-cachememsize. >>>>> >>>>> these are my ldif files that i used to modify >>>>> the values >>>>> modify entry cache size >>>>> cat modify-cache-mem-size.ldif >>>>> dn: cn=userRoot,cn=ldbm >>>>> database,cn=plugins,cn=config >>>>> changetype: modify >>>>> replace: nsslapd-cachememsize >>>>> nsslapd-cachememsize: 209715200 >>>>> >>>>> modify db cache size >>>>> cat modfy-db-cache-size.ldif >>>>> dn: cn=config,cn=ldbm >>>>> database,cn=plugins,cn=config >>>>> changetype: modify >>>>> replace: nsslapd-dbcachesize >>>>> nsslapd-dbcachesize: 209715200 >>>>> >>>>> After modifying , i restarted IPA services >>>>> >>>>> Is there anything else that I need to take >>>>> care of as the logs suggest its still not >>>>> getting the updated values >>>>> >>>>> Thanks >>>>> Rakesh >>>>> >>>>> On Mon, Aug 29, 2016 at 6:07 PM, Rakesh >>>>> Rajasekharan >>>> > wrote: >>>>> >>>>> Hi Thierry, >>>>> >>>>> Coz of the issues we had to revert back to >>>>> earlier running openldap in production. >>>>> >>>>> I have now done a few TCP related changes >>>>> in sysctl.conf and have also increased the >>>>> nsslapd-dbcachesize and >>>>> nsslapd-cachememsize to 200MB >>>>> >>>>> I will again start migrating hosts back to >>>>> IPA and see if I face the earlier issue. >>>>> >>>>> I will update back once I have something >>>>> >>>>> >>>>> Thanks, >>>>> Rakesh >>>>> >>>>> >>>>> >>>>> On Thu, Aug 25, 2016 at 2:17 PM, thierry >>>>> bordaz >>>> > wrote: >>>>> >>>>> >>>>> >>>>> On 08/25/2016 10:15 AM, Rakesh >>>>> Rajasekharan wrote: >>>>>> All of the troubleshooting seems fine. >>>>>> >>>>>> >>>>>> However, Running libconv.pl >>>>>> gives me this output >>>>>> >>>>>> ----- Recommendations ----- >>>>>> >>>>>> 1. You have unindexed components, >>>>>> this can be caused from a search on >>>>>> an unindexed attribute, or your >>>>>> returned results exceeded the >>>>>> allidsthreshold. Unindexed components >>>>>> are not recommended. To refuse >>>>>> unindexed searches, switch >>>>>> 'nsslapd-require-index' to 'on' under >>>>>> your database entry (e.g. >>>>>> cn=UserRoot,cn=ldbm >>>>>> database,cn=plugins,cn=config). >>>>>> >>>>>> 2. You have a significant >>>>>> difference between binds and unbinds. >>>>>> You may want to investigate this >>>>>> difference. >>>>>> >>>>>> >>>>>> I feel, this could be a pointer to >>>>>> things going slow.. and IPA hanging. >>>>>> I think i now have something that I >>>>>> can try and nail down this issue. >>>>>> >>>>>> On a sidenote, I was earlier running >>>>>> openldap and migrated over to Freeipa, >>>>>> >>>>>> Thanks >>>>>> Rakesh >>>>>> >>>>>> >>>>>> >>>>>> On Wed, Aug 24, 2016 at 12:38 PM, >>>>>> Petr Spacek >>>>> > wrote: >>>>>> >>>>>> On 23.8.2016 18:44, Rakesh >>>>>> Rajasekharan wrote: >>>>>> > I think thers something >>>>>> seriously wrong with my system >>>>>> > >>>>>> > not able to run any IPA commands >>>>>> > >>>>>> > klist >>>>>> > Ticket cache: >>>>>> KEYRING:persistent:0:0 >>>>>> > Default principal: >>>>>> admin at XYZ.COM >>>>>> > >>>>>> > Valid starting Expires Service >>>>>> principal >>>>>> > 2016-08-23T16:26:36 >>>>>> 2016-08-24T16:26:22 >>>>>> krbtgt/XYZ.COM at XYZ.COM >>>>>> >>>>>> > >>>>>> > >>>>>> > [root at prod-ipa-master-1a :~] >>>>>> ipactl status >>>>>> > Directory Service: RUNNING >>>>>> > krb5kdc Service: RUNNING >>>>>> > kadmin Service: RUNNING >>>>>> > ipa_memcached Service: RUNNING >>>>>> > httpd Service: RUNNING >>>>>> > pki-tomcatd Service: RUNNING >>>>>> > ipa-otpd Service: RUNNING >>>>>> > ipa: INFO: The ipactl command >>>>>> was successful >>>>>> > >>>>>> > >>>>>> > >>>>>> > [root at prod-ipa-master :~] ipa >>>>>> user-find p-testuser >>>>>> > ipa: ERROR: Kerberos error: >>>>>> ('Unspecified GSS failure. Minor >>>>>> code may >>>>>> > provide more information', >>>>>> 851968)/("Cannot contact any KDC >>>>>> for realm ' >>>>>> > XYZ.COM '", >>>>>> -1765328228) >>>>>> >>>>> >>>>> Hi Rakesh, >>>>> >>>>> Having a reproducible test case >>>>> would you rerun the command above. >>>>> During its processing you may >>>>> monitor DS process load (top). If >>>>> it is high, you may get some >>>>> pstacks of it. >>>>> Also would you attach the part of >>>>> DS access logs taken during the >>>>> command. >>>>> >>>>> regards >>>>> thierry >>>>> >>>>>> > >>>>>> >>>>>> This is weird because the server >>>>>> seems to be up. >>>>>> >>>>>> Please follow >>>>>> http://www.freeipa.org/page/Troubleshooting#Authentication.2FKerberos >>>>>> >>>>>> Petr^2 Spacek >>>>>> >>>>>> > >>>>>> > >>>>>> > Thanks >>>>>> > >>>>>> > Rakesh >>>>>> > >>>>>> > On Tue, Aug 23, 2016 at 10:01 >>>>>> PM, Rakesh Rajasekharan < >>>>>> > rakesh.rajasekharan at gmail.com >>>>>> > >>>>>> wrote: >>>>>> > >>>>>> >> i changed the loggin level to >>>>>> 4 . Modifying nsslapd-accesslog-level >>>>>> >> >>>>>> >> But, the hang is still there. >>>>>> though I dont see the sigfault now >>>>>> >> >>>>>> >> >>>>>> >> >>>>>> >> >>>>>> >> On Tue, Aug 23, 2016 at 9:02 >>>>>> PM, Rakesh Rajasekharan < >>>>>> >> rakesh.rajasekharan at gmail.com >>>>>> > >>>>>> wrote: >>>>>> >> >>>>>> >>> My disk was getting filled >>>>>> too fast >>>>>> >>> >>>>>> >>> logs under /var/log/dirsrv >>>>>> was coming around 5 gb quickly >>>>>> filling up >>>>>> >>> >>>>>> >>> Is there a way to make the >>>>>> logging less verbose >>>>>> >>> >>>>>> >>> >>>>>> >>> >>>>>> >>> On Tue, Aug 23, 2016 at 6:41 >>>>>> PM, Petr Spacek >>>>>> >>>>> > wrote: >>>>>> >>> >>>>>> >>>> On 23.8.2016 15:07, Rakesh >>>>>> Rajasekharan wrote: >>>>>> >>>>> I was able to fix that may >>>>>> be temporarily... when i checked the >>>>>> >>>> network.. >>>>>> >>>>> there was another process >>>>>> that was running and consuming a >>>>>> lot of >>>>>> >>>> network ( >>>>>> >>>>> i have no idea who did >>>>>> that. I need to seriously start >>>>>> restricting >>>>>> >>>> people >>>>>> >>>>> access to this machine ) >>>>>> >>>>> >>>>>> >>>>> after killing that >>>>>> perfomance improved drastically >>>>>> >>>>> >>>>>> >>>>> But now, suddenly I started >>>>>> experiencing the same hang. >>>>>> >>>>> >>>>>> >>>>> This time , I gert the >>>>>> following error when checked dmesg >>>>>> >>>>> >>>>>> >>>>> [ 301.236976] >>>>>> ns-slapd[3124]: segfault at 0 ip >>>>>> 00007f1de416951c sp >>>>>> >>>>> 00007f1dee1dba70 error 4 in >>>>>> libcos-plugin.so[7f1de4166000+b000] >>>>>> >>>>> [ 1116.248431] TCP: >>>>>> request_sock_TCP: Possible SYN >>>>>> flooding on port 88. >>>>>> >>>>> Sending cookies. Check SNMP >>>>>> counters. >>>>>> >>>>> [11831.397037] >>>>>> ns-slapd[22550]: segfault at 0 ip >>>>>> 00007f533d82251c sp >>>>>> >>>>> 00007f5347894a70 error 4 in >>>>>> libcos-plugin.so[7f533d81f000+b000] >>>>>> >>>>> [11832.727989] >>>>>> ns-slapd[22606]: segfault at 0 ip >>>>>> 00007f6231eb951c sp >>>>>> >>>>> 00007f623bf2ba70 error 4 in >>>>>> libcos-plugin.so[7f6231eb6000+b00 >>>>>> >>>> >>>>>> >>>> Okay, this one is serious. >>>>>> The LDAP server crashed. >>>>>> >>>> >>>>>> >>>> 1. Make sure all your >>>>>> packages are up-to-date. >>>>>> >>>> >>>>>> >>>> Please see >>>>>> >>>> >>>>>> http://directory.fedoraproject.org/docs/389ds/FAQ/faq.html#d >>>>>> >>>> ebugging-crashes >>>>>> >>>> for further instructions how >>>>>> to debug this. >>>>>> >>>> >>>>>> >>>> Petr^2 Spacek >>>>>> >>>> >>>>>> >>>>> >>>>>> >>>>> and in >>>>>> /var/log/dirsrv/example-com/errors >>>>>> >>>>> >>>>>> >>>>> [23/Aug/2016:12:49:36 >>>>>> +0000] DSRetroclPlugin - >>>>>> delete_changerecord: >>>>>> >>>> could >>>>>> >>>>> not delete change record >>>>>> 3291138 (rc: 32) >>>>>> >>>>> [23/Aug/2016:12:49:36 >>>>>> +0000] DSRetroclPlugin - >>>>>> delete_changerecord: >>>>>> >>>> could >>>>>> >>>>> not delete change record >>>>>> 3291139 (rc: 32) >>>>>> >>>>> [23/Aug/2016:12:49:36 >>>>>> +0000] DSRetroclPlugin - >>>>>> delete_changerecord: >>>>>> >>>> could >>>>>> >>>>> not delete change record >>>>>> 3291140 (rc: 32) >>>>>> >>>>> [23/Aug/2016:12:49:36 >>>>>> +0000] DSRetroclPlugin - >>>>>> delete_changerecord: >>>>>> >>>> could >>>>>> >>>>> not delete change record >>>>>> 3291141 (rc: 32) >>>>>> >>>>> [23/Aug/2016:12:49:36 >>>>>> +0000] DSRetroclPlugin - >>>>>> delete_changerecord: >>>>>> >>>> could >>>>>> >>>>> not delete change record >>>>>> 3291142 (rc: 32) >>>>>> >>>>> [23/Aug/2016:12:49:36 >>>>>> +0000] DSRetroclPlugin - >>>>>> delete_changerecord: >>>>>> >>>> could >>>>>> >>>>> not delete change record >>>>>> 3291143 (rc: 32) >>>>>> >>>>> [23/Aug/2016:12:49:36 >>>>>> +0000] DSRetroclPlugin - >>>>>> delete_changerecord: >>>>>> >>>> could >>>>>> >>>>> not delete change record >>>>>> 3291144 (rc: 32) >>>>>> >>>>> [23/Aug/2016:12:49:36 >>>>>> +0000] DSRetroclPlugin - >>>>>> delete_changerecord: >>>>>> >>>> could >>>>>> >>>>> not delete change record >>>>>> 3291145 (rc: 32) >>>>>> >>>>> [23/Aug/2016:12:49:50 >>>>>> +0000] - Retry count exceeded in >>>>>> delete >>>>>> >>>>> [23/Aug/2016:12:49:50 >>>>>> +0000] DSRetroclPlugin - >>>>>> delete_changerecord: >>>>>> >>>> could >>>>>> >>>>> not delete change record >>>>>> 3292734 (rc: 51) >>>>>> >>>>> >>>>>> >>>>> >>>>>> >>>>> Can i do something about >>>>>> this error.. I treid to restart >>>>>> ipa a couple >>>>>> >>>> of >>>>>> >>>>> time but that did not help >>>>>> >>>>> >>>>>> >>>>> Thanks >>>>>> >>>>> Rakesh >>>>>> >>>>> >>>>>> >>>>> On Mon, Aug 22, 2016 at >>>>>> 2:27 PM, Petr Spacek >>>>>> >>>>> > >>>>>> >>>> wrote: >>>>>> >>>>> >>>>>> >>>>>> On 19.8.2016 19:32, Rakesh >>>>>> Rajasekharan wrote: >>>>>> >>>>>>> I am running my set up on >>>>>> AWS cloud, and entropy is low at >>>>>> around >>>>>> >>>> 180 . >>>>>> >>>>>>> >>>>>> >>>>>>> I plan to increase it bu >>>>>> installing haveged . But, would >>>>>> low entropy >>>>>> >>>> by >>>>>> >>>>>> any >>>>>> >>>>>>> chance cause this issue >>>>>> of intermittent hang . >>>>>> >>>>>>> Also, the hang is mostly >>>>>> observed when registering around >>>>>> 20 clients >>>>>> >>>>>>> together >>>>>> >>>>>> >>>>>> >>>>>> Possibly, I'm not sure. If >>>>>> you want to dig into this, I >>>>>> would do this: >>>>>> >>>>>> 1. look what process hangs >>>>>> on client (using pstree command >>>>>> or so) >>>>>> >>>>>> $ pstree >>>>>> >>>>>> >>>>>> >>>>>> 2. look to what server and >>>>>> port is the hanging client >>>>>> connected to >>>>>> >>>>>> $ lsof -p >>>>> hanging process> >>>>>> >>>>>> >>>>>> >>>>>> 3. jump to server and see >>>>>> what process is bound to the >>>>>> target port >>>>>> >>>>>> $ netstat -pn >>>>>> >>>>>> >>>>>> >>>>>> 4. see where the process >>>>>> if hanging >>>>>> >>>>>> $ strace -p >>>>> hanging process> >>>>>> >>>>>> >>>>>> >>>>>> I hope it helps. >>>>>> >>>>>> >>>>>> >>>>>> Petr^2 Spacek >>>>>> >>>>>> >>>>>> >>>>>>> On Fri, Aug 19, 2016 at >>>>>> 7:24 PM, Rakesh Rajasekharan < >>>>>> >>>>>>> >>>>>> rakesh.rajasekharan at gmail.com >>>>>> > >>>>>> wrote: >>>>>> >>>>>>> >>>>>> >>>>>>>> yes there seems to be >>>>>> something thats worrying.. I have >>>>>> faced this >>>>>> >>>> today >>>>>> >>>>>>>> as well. >>>>>> >>>>>>>> There are few hosts >>>>>> around 280 odd left and when i >>>>>> try adding them >>>>>> >>>> to >>>>>> >>>>>> IPA >>>>>> >>>>>>>> , the slowness begins.. >>>>>> >>>>>>>> >>>>>> >>>>>>>> all the ipa commands >>>>>> like ipa user-find.. etc becomes >>>>>> very slow in >>>>>> >>>>>>>> responding. >>>>>> >>>>>>>> >>>>>> >>>>>>>> the SYNC_RECV are not >>>>>> many though just around 80-90 and >>>>>> today that >>>>>> >>>> was >>>>>> >>>>>>>> around 20 only >>>>>> >>>>>>>> >>>>>> >>>>>>>> >>>>>> >>>>>>>> I have for now increased >>>>>> tcp_max_syn_backlog to 5000. >>>>>> >>>>>>>> For now the slowness >>>>>> seems to have gone.. but I will >>>>>> do a try >>>>>> >>>> adding the >>>>>> >>>>>>>> clients again tomorrow >>>>>> and see how it goes >>>>>> >>>>>>>> >>>>>> >>>>>>>> Thanks >>>>>> >>>>>>>> Rakesh >>>>>> >>>>>>>> >>>>>> >>>>>>>> The issues >>>>>> >>>>>>>> >>>>>> >>>>>>>> On Fri, Aug 19, 2016 at >>>>>> 12:58 PM, Petr Spacek >>>>>> >>>>> > >>>>>> >>>>>> wrote: >>>>>> >>>>>>>> >>>>>> >>>>>>>>> On 18.8.2016 17:23, >>>>>> Rakesh Rajasekharan wrote: >>>>>> >>>>>>>>>> Hi >>>>>> >>>>>>>>>> >>>>>> >>>>>>>>>> I am migrating to >>>>>> freeipa from openldap and have >>>>>> around 4000 >>>>>> >>>> clients >>>>>> >>>>>>>>>> >>>>>> >>>>>>>>>> I had openned a >>>>>> another thread on that, but chose >>>>>> to start a new >>>>>> >>>> one >>>>>> >>>>>>>>> here >>>>>> >>>>>>>>>> as its a separate issue >>>>>> >>>>>>>>>> >>>>>> >>>>>>>>>> I was able to change >>>>>> the nssslapd-maxdescriptors >>>>>> adding an ldif >>>>>> >>>> file >>>>>> >>>>>>>>>> >>>>>> >>>>>>>>>> cat nsslapd-modify.ldif >>>>>> >>>>>>>>>> dn: cn=config >>>>>> >>>>>>>>>> changetype: modify >>>>>> >>>>>>>>>> replace: >>>>>> nsslapd-maxdescriptors >>>>>> >>>>>>>>>> >>>>>> nsslapd-maxdescriptors: 17000 >>>>>> >>>>>>>>>> >>>>>> >>>>>>>>>> and running the >>>>>> ldapmodify command >>>>>> >>>>>>>>>> >>>>>> >>>>>>>>>> I have now started >>>>>> moving clients running an >>>>>> openldap to Freeipa >>>>>> >>>> and >>>>>> >>>>>>>>> have >>>>>> >>>>>>>>>> today moved close to >>>>>> 2000 clients >>>>>> >>>>>>>>>> >>>>>> >>>>>>>>>> However, I have >>>>>> noticed that IPA hangs >>>>>> intermittently. >>>>>> >>>>>>>>>> >>>>>> >>>>>>>>>> running a kinit admin >>>>>> returns the below error >>>>>> >>>>>>>>>> kinit: Generic error >>>>>> (see e-text) while getting initial >>>>>> >>>> credentials >>>>>> >>>>>>>>>> >>>>>> >>>>>>>>>> from the >>>>>> /var/log/messages, I see this entry >>>>>> >>>>>>>>>> >>>>>> >>>>>>>>>> prod-ipa-master-int >>>>>> kernel: [104090.315801] TCP: >>>>>> >>>> request_sock_TCP: >>>>>> >>>>>>>>>> Possible SYN flooding >>>>>> on port 88. Sending cookies. >>>>>> Check SNMP >>>>>> >>>>>> counters. >>>>>> >>>>>>>>> >>>>>> >>>>>>>>> I would be worried >>>>>> about this message. Maybe >>>>>> kernel/firewall is >>>>>> >>>> doing >>>>>> >>>>>>>>> something fishy behind >>>>>> your back and blocking some >>>>>> connections or >>>>>> >>>> so. >>>>>> >>>>>>>>> >>>>>> >>>>>>>>> Petr^2 Spacek >>>>>> >>>>>>>>> >>>>>> >>>>>>>>> >>>>>> >>>>>>>>>> Aug 18 13:00:01 >>>>>> prod-ipa-master-int systemd[1]: >>>>>> Started Session >>>>>> >>>> 4885 >>>>>> >>>>>> of >>>>>> >>>>>>>>>> user root. >>>>>> >>>>>>>>>> Aug 18 13:00:01 >>>>>> prod-ipa-master-int systemd[1]: >>>>>> Starting Session >>>>>> >>>> 4885 >>>>>> >>>>>> of >>>>>> >>>>>>>>>> user root. >>>>>> >>>>>>>>>> Aug 18 13:01:01 >>>>>> prod-ipa-master-int systemd[1]: >>>>>> Started Session >>>>>> >>>> 4886 >>>>>> >>>>>> of >>>>>> >>>>>>>>>> user root. >>>>>> >>>>>>>>>> Aug 18 13:01:01 >>>>>> prod-ipa-master-int systemd[1]: >>>>>> Starting Session >>>>>> >>>> 4886 >>>>>> >>>>>> of >>>>>> >>>>>>>>>> user root. >>>>>> >>>>>>>>>> Aug 18 13:02:40 >>>>>> prod-ipa-master-int >>>>>> python[28984]: ansible-command >>>>>> >>>>>>>>> Invoked >>>>>> >>>>>>>>>> with creates=None >>>>>> executable=None shell=True args= >>>>>> removes=None >>>>>> >>>>>>>>> warn=True >>>>>> >>>>>>>>>> chdir=None >>>>>> >>>>>>>>>> Aug 18 13:04:37 >>>>>> prod-ipa-master-int sssd_be: >>>>>> GSSAPI Error: >>>>>> >>>> Unspecified >>>>>> >>>>>>>>> GSS >>>>>> >>>>>>>>>> failure. Minor code >>>>>> may provide more information (KDC >>>>>> returned >>>>>> >>>> error >>>>>> >>>>>>>>>> string: PROCESS_TGS) >>>>>> >>>>>>>>>> >>>>>> >>>>>>>>>> Could it be possible >>>>>> that its due to the initial load >>>>>> of adding >>>>>> >>>> the >>>>>> >>>>>>>>> clients >>>>>> >>>>>>>>>> or is there something >>>>>> else that I need to take care of. >>>>>> >>>>>> >>>>>> >>>>>> >>>>> >>>>> >>>>> >>>> >>>> >>>> >>>> >>> >>> >>> -- >>> Manage your subscription for the Freeipa-users >>> mailing list: >>> https://www.redhat.com/mailman/listinfo/freeipa-users >>> Go to http://freeipa.org for more info on the project >>> >>> >>> >>> >>> >> >> > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From tbordaz at redhat.com Mon Sep 5 12:37:53 2016 From: tbordaz at redhat.com (thierry bordaz) Date: Mon, 5 Sep 2016 14:37:53 +0200 Subject: [Freeipa-users] Freeipa 4.2.0 hangs intermittently In-Reply-To: References: <57BEB0AB.2090506@redhat.com> <57C44AC6.1030402@redhat.com> <57CD1FDC.2010109@redhat.com> <57CD5488.60404@redhat.com> Message-ID: <57CD6721.7020409@redhat.com> Hi Rakesh, Looking at this monitoring I do not see any reason to increase the number of worker threads. It shows that (before the hang), only one thread (out of 30) was busy. When the hang occurred, 7 threads (out of 30) were busy. regards thierry On 09/05/2016 02:14 PM, Rakesh Rajasekharan wrote: > btw the values for opscompleted and opsinitiated are > > ldapsearch -D "cn=directory manager" -w xxxx -LLL -b "cn=monitor" -s > base opsinitiated opscompleted > dn: cn=monitor > opsinitiated: 25655 > opscompleted: 25654 > > Thanks > > On Mon, Sep 5, 2016 at 5:35 PM, Rakesh Rajasekharan > > > wrote: > > Hi Thierry, > > I was able to now reporduce the issue.. > > the ns-slapd process just got hung,.. however, no core file was > generate.. > > I manually created a coredump using gcore and then took a strace of it > this is strace from the coredump generated > > GNU gdb (GDB) Red Hat Enterprise Linux 7.6.1-80.el7 > Copyright (C) 2013 Free Software Foundation, Inc. > License GPLv3+: GNU GPL version 3 or later > > > This is free software: you are free to change and redistribute it. > There is NO WARRANTY, to the extent permitted by law. Type "show > copying" > and "show warranty" for details. > This GDB was configured as "x86_64-redhat-linux-gnu". > For bug reporting instructions, please see: > >... > Reading symbols from /usr/sbin/ns-slapd...Reading symbols from > /usr/sbin/ns-slapd...(no debugging symbols found)...done. > (no debugging symbols found)...done. > [New LWP 105159] > [New LWP 105160] > [New LWP 105161] > [New LWP 105162] > [New LWP 105163] > [New LWP 105164] > [New LWP 105165] > [New LWP 105166] > [New LWP 105167] > [New LWP 105168] > [New LWP 105171] > [New LWP 105172] > [New LWP 105173] > [New LWP 105174] > [New LWP 105175] > [New LWP 105176] > [New LWP 105177] > [New LWP 105178] > [New LWP 105179] > [New LWP 105180] > [New LWP 105181] > [New LWP 105182] > [New LWP 105183] > [New LWP 105184] > [New LWP 105185] > [New LWP 105186] > [New LWP 105187] > [New LWP 105188] > [New LWP 105189] > [New LWP 105190] > [New LWP 105191] > [New LWP 105192] > [New LWP 105193] > [New LWP 105194] > [New LWP 105195] > [New LWP 105196] > [New LWP 105197] > [New LWP 105198] > [New LWP 105199] > [New LWP 105200] > [New LWP 105201] > [New LWP 105203] > [New LWP 105215] > [New LWP 105158] > [Thread debugging using libthread_db enabled] > Using host libthread_db library "/lib64/libthread_db.so.1". > Core was generated by `/usr/sbin/ns-slapd'. > #0 0x00007f1b48ae69b3 in select () from /lib64/libc.so.6 > > Thread 44 (Thread 0x7f1b4b6d9840 (LWP 105158)): > #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 () from > /lib64/libpthread.so.0 > No symbol table info available. > #1 0x00007f1b4941b2c3 in PR_EnterMonitor () from /lib64/libnspr4.so > No symbol table info available. > #2 0x00007f1b4b7125ed in setup_pr_read_pds () > No symbol table info available. > #3 0x00007f1b4b715fd9 in slapd_daemon () > No symbol table info available. > #4 0x00007f1b4b707c6c in main () > No symbol table info available. > > Thread 43 (Thread 0x7f1a5df0f700 (LWP 105215)): > #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 () from > /lib64/libpthread.so.0 > No symbol table info available. > #1 0x00007f1b4941b050 in PR_WaitCondVar () from /lib64/libnspr4.so > No symbol table info available. > #2 0x00007f1b4b71df25 in ps_send_results () > No symbol table info available. > #3 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so > No symbol table info available. > #4 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 > No symbol table info available. > #5 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 > No symbol table info available. > > Thread 42 (Thread 0x7f1b03fe7700 (LWP 105203)): > #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 () from > /lib64/libpthread.so.0 > No symbol table info available. > #1 0x00007f1b4941b050 in PR_WaitCondVar () from /lib64/libnspr4.so > No symbol table info available. > #2 0x00007f1b4b71df25 in ps_send_results () > No symbol table info available. > #3 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so > No symbol table info available. > #4 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 > No symbol table info available. > #5 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 > No symbol table info available. > > Thread 41 (Thread 0x7f1b047e8700 (LWP 105201)): > #0 0x00007f1b48ae69b3 in select () from /lib64/libc.so.6 > No symbol table info available. > #1 0x00007f1b4b2710e9 in DS_Sleep () from > /usr/lib64/dirsrv/libslapd.so.0 > No symbol table info available. > #2 0x00007f1b4b712325 in time_thread () > No symbol table info available. > #3 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so > No symbol table info available. > #4 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 > No symbol table info available. > #5 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 > No symbol table info available. > > Thread 40 (Thread 0x7f1b04fe9700 (LWP 105200)): > #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 () from > /lib64/libpthread.so.0 > No symbol table info available. > #1 0x00007f1b433212f3 in __db_hybrid_mutex_suspend () from > /lib64/libdb-5.3.so > No symbol table info available. > #2 0x00007f1b43320640 in __db_tas_mutex_lock () from > /lib64/libdb-5.3.so > No symbol table info available. > #3 0x00007f1b4343ae1b in __memp_fget () from /lib64/libdb-5.3.so > > No symbol table info available. > #4 0x00007f1b4333e211 in __bam_search () from /lib64/libdb-5.3.so > > No symbol table info available. > #5 0x00007f1b43329256 in __bamc_search () from > /lib64/libdb-5.3.so > No symbol table info available. > #6 0x00007f1b4332d6c4 in __bamc_put () from /lib64/libdb-5.3.so > > No symbol table info available. > #7 0x00007f1b433e46d5 in __dbc_iput () from /lib64/libdb-5.3.so > > No symbol table info available. > #8 0x00007f1b433dfa0e in __db_put () from /lib64/libdb-5.3.so > > No symbol table info available. > #9 0x00007f1b433f4fa4 in __db_put_pp () from /lib64/libdb-5.3.so > > No symbol table info available. > #10 0x00007f1b3e8cf4cf in idl_new_insert_key () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > No symbol table info available. > #11 0x00007f1b3e8de5b5 in addordel_values_sv.isra.3 () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > No symbol table info available. > #12 0x00007f1b3e8df25a in index_addordel_values_ext_sv () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > No symbol table info available. > #13 0x00007f1b3e8df694 in index_addordel_values_sv () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > No symbol table info available. > #14 0x00007f1b3e8df75a in index_addordel_entry () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > No symbol table info available. > #15 0x00007f1b3e8e25fa in ldbm_back_add () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > No symbol table info available. > #16 0x00007f1b4b1e4cca in op_shared_add () from > /usr/lib64/dirsrv/libslapd.so.0 > No symbol table info available. > #17 0x00007f1b4b1e6020 in do_add () from > /usr/lib64/dirsrv/libslapd.so.0 > No symbol table info available. > #18 0x00007f1b4b711303 in connection_threadmain () > No symbol table info available. > #19 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so > No symbol table info available. > #20 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 > No symbol table info available. > #21 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 > No symbol table info available. > > Thread 39 (Thread 0x7f1b057ea700 (LWP 105199)): > #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 () from > /lib64/libpthread.so.0 > No symbol table info available. > #1 0x00007f1b4941b050 in PR_WaitCondVar () from /lib64/libnspr4.so > No symbol table info available. > #2 0x00007f1b4b70efee in connection_wait_for_new_work () > No symbol table info available. > #3 0x00007f1b4b710719 in connection_threadmain () > No symbol table info available. > #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so > No symbol table info available. > #5 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 > No symbol table info available. > #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 > No symbol table info available. > > Thread 38 (Thread 0x7f1b05feb700 (LWP 105198)): > #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 () from > /lib64/libpthread.so.0 > No symbol table info available. > #1 0x00007f1b4941b050 in PR_WaitCondVar () from /lib64/libnspr4.so > No symbol table info available. > #2 0x00007f1b4b70efee in connection_wait_for_new_work () > No symbol table info available. > #3 0x00007f1b4b710719 in connection_threadmain () > No symbol table info available. > #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so > No symbol table info available. > #5 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 > No symbol table info available. > #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 > No symbol table info available. > > Thread 37 (Thread 0x7f1b067ec700 (LWP 105197)): > #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 () from > /lib64/libpthread.so.0 > No symbol table info available. > #1 0x00007f1b4941b050 in PR_WaitCondVar () from /lib64/libnspr4.so > No symbol table info available. > #2 0x00007f1b4b70efee in connection_wait_for_new_work () > No symbol table info available. > #3 0x00007f1b4b710719 in connection_threadmain () > No symbol table info available. > #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so > No symbol table info available. > #5 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 > No symbol table info available. > #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 > No symbol table info available. > > Thread 36 (Thread 0x7f1b06fed700 (LWP 105196)): > #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 () from > /lib64/libpthread.so.0 > No symbol table info available. > #1 0x00007f1b4941b050 in PR_WaitCondVar () from /lib64/libnspr4.so > No symbol table info available. > #2 0x00007f1b4b70efee in connection_wait_for_new_work () > No symbol table info available. > #3 0x00007f1b4b710719 in connection_threadmain () > No symbol table info available. > #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so > No symbol table info available. > #5 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 > No symbol table info available. > #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 > No symbol table info available. > > Thread 35 (Thread 0x7f1b077ee700 (LWP 105195)): > #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 () from > /lib64/libpthread.so.0 > No symbol table info available. > #1 0x00007f1b4941b2c3 in PR_EnterMonitor () from /lib64/libnspr4.so > No symbol table info available. > #2 0x00007f1b3e8c23e6 in dblayer_txn_begin () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > No symbol table info available. > #3 0x00007f1b3e8fe007 in ldbm_back_modify () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > No symbol table info available. > #4 0x00007f1b4b22b541 in op_shared_modify () from > /usr/lib64/dirsrv/libslapd.so.0 > No symbol table info available. > #5 0x00007f1b4b22c004 in modify_internal_pb () from > /usr/lib64/dirsrv/libslapd.so.0 > No symbol table info available. > #6 0x00007f1b3eb48b60 in ipapwd_apply_mods () from > /usr/lib64/dirsrv/plugins/libipa_pwd_extop.so > No symbol table info available. > #7 0x00007f1b3eb4c8e0 in store_new_keys () from > /usr/lib64/dirsrv/plugins/libipa_pwd_extop.so > No symbol table info available. > #8 0x00007f1b3eb4ead0 in ipapwd_extop () from > /usr/lib64/dirsrv/plugins/libipa_pwd_extop.so > No symbol table info available. > #9 0x00007f1b4b23bba2 in plugin_call_exop_plugins () from > /usr/lib64/dirsrv/libslapd.so.0 > No symbol table info available. > #10 0x00007f1b4b717efe in do_extended () > No symbol table info available. > #11 0x00007f1b4b71141a in connection_threadmain () > No symbol table info available. > #12 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so > No symbol table info available. > #13 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 > No symbol table info available. > #14 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 > No symbol table info available. > > Thread 34 (Thread 0x7f1b07fef700 (LWP 105194)): > #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 () from > /lib64/libpthread.so.0 > No symbol table info available. > #1 0x00007f1b4941b050 in PR_WaitCondVar () from /lib64/libnspr4.so > No symbol table info available. > #2 0x00007f1b4b70efee in connection_wait_for_new_work () > No symbol table info available. > #3 0x00007f1b4b710719 in connection_threadmain () > No symbol table info available. > #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so > No symbol table info available. > #5 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 > No symbol table info available. > #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 > No symbol table info available. > > Thread 33 (Thread 0x7f1b087f0700 (LWP 105193)): > #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 () from > /lib64/libpthread.so.0 > No symbol table info available. > #1 0x00007f1b4941b050 in PR_WaitCondVar () from /lib64/libnspr4.so > No symbol table info available. > #2 0x00007f1b4b70efee in connection_wait_for_new_work () > No symbol table info available. > #3 0x00007f1b4b710719 in connection_threadmain () > No symbol table info available. > #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so > No symbol table info available. > #5 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 > No symbol table info available. > #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 > No symbol table info available. > > Thread 32 (Thread 0x7f1b08ff1700 (LWP 105192)): > #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 () from > /lib64/libpthread.so.0 > No symbol table info available. > #1 0x00007f1b4941b050 in PR_WaitCondVar () from /lib64/libnspr4.so > No symbol table info available. > #2 0x00007f1b4b70efee in connection_wait_for_new_work () > No symbol table info available. > #3 0x00007f1b4b710719 in connection_threadmain () > No symbol table info available. > #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so > No symbol table info available. > #5 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 > No symbol table info available. > #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 > No symbol table info available. > > Thread 31 (Thread 0x7f1b097f2700 (LWP 105191)): > #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 () from > /lib64/libpthread.so.0 > No symbol table info available. > #1 0x00007f1b4941b050 in PR_WaitCondVar () from /lib64/libnspr4.so > No symbol table info available. > #2 0x00007f1b4b70efee in connection_wait_for_new_work () > No symbol table info available. > #3 0x00007f1b4b710719 in connection_threadmain () > No symbol table info available. > #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so > No symbol table info available. > #5 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 > No symbol table info available. > #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 > No symbol table info available. > > Thread 30 (Thread 0x7f1b09ff3700 (LWP 105190)): > #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 () from > /lib64/libpthread.so.0 > No symbol table info available. > #1 0x00007f1b433212f3 in __db_hybrid_mutex_suspend () from > /lib64/libdb-5.3.so > No symbol table info available. > #2 0x00007f1b43320640 in __db_tas_mutex_lock () from > /lib64/libdb-5.3.so > No symbol table info available. > #3 0x00007f1b433cad3a in __lock_get_internal () from > /lib64/libdb-5.3.so > No symbol table info available. > #4 0x00007f1b433cb820 in __lock_get () from /lib64/libdb-5.3.so > > No symbol table info available. > #5 0x00007f1b433f7142 in __db_lget () from /lib64/libdb-5.3.so > > No symbol table info available. > #6 0x00007f1b4333e605 in __bam_search () from /lib64/libdb-5.3.so > > No symbol table info available. > #7 0x00007f1b43329256 in __bamc_search () from > /lib64/libdb-5.3.so > No symbol table info available. > #8 0x00007f1b4332ad0f in __bamc_get () from /lib64/libdb-5.3.so > > No symbol table info available. > #9 0x00007f1b433e3ca6 in __dbc_iget () from /lib64/libdb-5.3.so > > No symbol table info available. > #10 0x00007f1b433f2b02 in __dbc_get_pp () from /lib64/libdb-5.3.so > > No symbol table info available. > #11 0x00007f1b3e8ce680 in idl_new_fetch () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > No symbol table info available. > #12 0x00007f1b3e8dcb26 in index_read_ext_allids () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > No symbol table info available. > #13 0x00007f1b3e8c7284 in keys2idl () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > No symbol table info available. > #14 0x00007f1b3e8c79e3 in ava_candidates.isra.0 () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > No symbol table info available. > #15 0x00007f1b3e8c7fd2 in filter_candidates_ext () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > No symbol table info available. > #16 0x00007f1b3e8c9096 in list_candidates () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > No symbol table info available. > #17 0x00007f1b3e8c7f40 in filter_candidates_ext () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > No symbol table info available. > #18 0x00007f1b3e8c9096 in list_candidates () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > No symbol table info available. > #19 0x00007f1b3e8c7f40 in filter_candidates_ext () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > No symbol table info available. > #20 0x00007f1b3e8c9096 in list_candidates () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > No symbol table info available. > #21 0x00007f1b3e8c7f40 in filter_candidates_ext () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > No symbol table info available. > #22 0x00007f1b3e90380c in subtree_candidates () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > No symbol table info available. > #23 0x00007f1b3e904e7f in ldbm_back_search () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > No symbol table info available. > #24 0x00007f1b4b2321d6 in op_shared_search () from > /usr/lib64/dirsrv/libslapd.so.0 > No symbol table info available. > #25 0x00007f1b4b72333e in do_search () > No symbol table info available. > #26 0x00007f1b4b7113c3 in connection_threadmain () > No symbol table info available. > #27 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so > No symbol table info available. > #28 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 > No symbol table info available. > #29 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 > No symbol table info available. > > Thread 29 (Thread 0x7f1b0a7f4700 (LWP 105189)): > #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 () from > /lib64/libpthread.so.0 > No symbol table info available. > #1 0x00007f1b4941b050 in PR_WaitCondVar () from /lib64/libnspr4.so > No symbol table info available. > #2 0x00007f1b4b70efee in connection_wait_for_new_work () > No symbol table info available. > #3 0x00007f1b4b710719 in connection_threadmain () > No symbol table info available. > #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so > No symbol table info available. > #5 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 > No symbol table info available. > #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 > No symbol table info available. > > Thread 28 (Thread 0x7f1b0aff5700 (LWP 105188)): > #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 () from > /lib64/libpthread.so.0 > No symbol table info available. > #1 0x00007f1b4941b050 in PR_WaitCondVar () from /lib64/libnspr4.so > No symbol table info available. > #2 0x00007f1b4b70efee in connection_wait_for_new_work () > No symbol table info available. > #3 0x00007f1b4b710719 in connection_threadmain () > No symbol table info available. > #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so > No symbol table info available. > #5 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 > No symbol table info available. > #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 > No symbol table info available. > > Thread 27 (Thread 0x7f1b0b7f6700 (LWP 105187)): > #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 () from > /lib64/libpthread.so.0 > No symbol table info available. > #1 0x00007f1b4941b050 in PR_WaitCondVar () from /lib64/libnspr4.so > No symbol table info available. > #2 0x00007f1b4b70efee in connection_wait_for_new_work () > No symbol table info available. > #3 0x00007f1b4b710719 in connection_threadmain () > No symbol table info available. > #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so > No symbol table info available. > #5 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 > No symbol table info available. > #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 > No symbol table info available. > > Thread 26 (Thread 0x7f1b0bff7700 (LWP 105186)): > #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 () from > /lib64/libpthread.so.0 > No symbol table info available. > #1 0x00007f1b433212f3 in __db_hybrid_mutex_suspend () from > /lib64/libdb-5.3.so > No symbol table info available. > #2 0x00007f1b43320640 in __db_tas_mutex_lock () from > /lib64/libdb-5.3.so > No symbol table info available. > #3 0x00007f1b433cad3a in __lock_get_internal () from > /lib64/libdb-5.3.so > No symbol table info available. > #4 0x00007f1b433cb820 in __lock_get () from /lib64/libdb-5.3.so > > No symbol table info available. > #5 0x00007f1b433f7142 in __db_lget () from /lib64/libdb-5.3.so > > No symbol table info available. > #6 0x00007f1b4333e605 in __bam_search () from /lib64/libdb-5.3.so > > No symbol table info available. > #7 0x00007f1b43329256 in __bamc_search () from > /lib64/libdb-5.3.so > No symbol table info available. > #8 0x00007f1b4332ad0f in __bamc_get () from /lib64/libdb-5.3.so > > No symbol table info available. > #9 0x00007f1b433e3ca6 in __dbc_iget () from /lib64/libdb-5.3.so > > No symbol table info available. > #10 0x00007f1b433f2b02 in __dbc_get_pp () from /lib64/libdb-5.3.so > > No symbol table info available. > #11 0x00007f1b3e8ce680 in idl_new_fetch () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > No symbol table info available. > #12 0x00007f1b3e8dcb26 in index_read_ext_allids () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > No symbol table info available. > #13 0x00007f1b3e8c7284 in keys2idl () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > No symbol table info available. > #14 0x00007f1b3e8c79e3 in ava_candidates.isra.0 () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > No symbol table info available. > #15 0x00007f1b3e8c7fd2 in filter_candidates_ext () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > No symbol table info available. > #16 0x00007f1b3e8c9096 in list_candidates () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > No symbol table info available. > #17 0x00007f1b3e8c7f40 in filter_candidates_ext () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > No symbol table info available. > #18 0x00007f1b3e90380c in subtree_candidates () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > No symbol table info available. > #19 0x00007f1b3e904e7f in ldbm_back_search () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > No symbol table info available. > #20 0x00007f1b4b2321d6 in op_shared_search () from > /usr/lib64/dirsrv/libslapd.so.0 > No symbol table info available. > #21 0x00007f1b4b24294e in search_internal_callback_pb () from > /usr/lib64/dirsrv/libslapd.so.0 > No symbol table info available. > #22 0x00007f1b4b242be8 in search_internal_pb () from > /usr/lib64/dirsrv/libslapd.so.0 > No symbol table info available. > #23 0x00007f1b4b72149b in ids_sasl_canon_user () > No symbol table info available. > #24 0x00007f1b491e193b in _sasl_canon_user () from > /lib64/libsasl2.so.3 > No symbol table info available. > #25 0x00007f1b491e1c4c in _sasl_canon_user_lookup () from > /lib64/libsasl2.so.3 > No symbol table info available. > #26 0x00007f1b430e9fb9 in gssapi_server_mech_step () from > /usr/lib64/sasl2/libgssapiv2.so > No symbol table info available. > #27 0x00007f1b491edb9b in sasl_server_step () from > /lib64/libsasl2.so.3 > No symbol table info available. > #28 0x00007f1b4b72283b in ids_sasl_check_bind () > No symbol table info available. > #29 0x00007f1b4b70a61a in do_bind () > No symbol table info available. > #30 0x00007f1b4b7113fd in connection_threadmain () > No symbol table info available. > #31 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so > No symbol table info available. > #32 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 > No symbol table info available. > #33 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 > No symbol table info available. > > Thread 25 (Thread 0x7f1b0c7f8700 (LWP 105185)): > #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 () from > /lib64/libpthread.so.0 > No symbol table info available. > #1 0x00007f1b4941b050 in PR_WaitCondVar () from /lib64/libnspr4.so > No symbol table info available. > #2 0x00007f1b4b70efee in connection_wait_for_new_work () > No symbol table info available. > #3 0x00007f1b4b710719 in connection_threadmain () > No symbol table info available. > #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so > No symbol table info available. > #5 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 > No symbol table info available. > #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 > No symbol table info available. > > Thread 24 (Thread 0x7f1b0cff9700 (LWP 105184)): > #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 () from > /lib64/libpthread.so.0 > No symbol table info available. > #1 0x00007f1b4941b050 in PR_WaitCondVar () from /lib64/libnspr4.so > No symbol table info available. > #2 0x00007f1b4b70efee in connection_wait_for_new_work () > No symbol table info available. > #3 0x00007f1b4b710719 in connection_threadmain () > No symbol table info available. > #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so > No symbol table info available. > #5 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 > No symbol table info available. > #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 > No symbol table info available. > > Thread 23 (Thread 0x7f1b0d7fa700 (LWP 105183)): > #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 () from > /lib64/libpthread.so.0 > No symbol table info available. > #1 0x00007f1b4941b050 in PR_WaitCondVar () from /lib64/libnspr4.so > No symbol table info available. > #2 0x00007f1b4b70efee in connection_wait_for_new_work () > No symbol table info available. > #3 0x00007f1b4b710719 in connection_threadmain () > No symbol table info available. > #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so > No symbol table info available. > #5 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 > No symbol table info available. > #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 > No symbol table info available. > > Thread 22 (Thread 0x7f1b0dffb700 (LWP 105182)): > #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 () from > /lib64/libpthread.so.0 > No symbol table info available. > #1 0x00007f1b4941b050 in PR_WaitCondVar () from /lib64/libnspr4.so > No symbol table info available. > #2 0x00007f1b4b70efee in connection_wait_for_new_work () > No symbol table info available. > #3 0x00007f1b4b710719 in connection_threadmain () > No symbol table info available. > #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so > No symbol table info available. > #5 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 > No symbol table info available. > #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 > No symbol table info available. > > Thread 21 (Thread 0x7f1b0e7fc700 (LWP 105181)): > #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 () from > /lib64/libpthread.so.0 > No symbol table info available. > #1 0x00007f1b4941b050 in PR_WaitCondVar () from /lib64/libnspr4.so > No symbol table info available. > #2 0x00007f1b4b70efee in connection_wait_for_new_work () > No symbol table info available. > #3 0x00007f1b4b710719 in connection_threadmain () > No symbol table info available. > #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so > No symbol table info available. > #5 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 > No symbol table info available. > #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 > No symbol table info available. > > Thread 20 (Thread 0x7f1b0effd700 (LWP 105180)): > #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 () from > /lib64/libpthread.so.0 > No symbol table info available. > #1 0x00007f1b433212f3 in __db_hybrid_mutex_suspend () from > /lib64/libdb-5.3.so > No symbol table info available. > #2 0x00007f1b43320b37 in __db_tas_mutex_readlock () from > /lib64/libdb-5.3.so > No symbol table info available. > #3 0x00007f1b4343ad35 in __memp_fget () from /lib64/libdb-5.3.so > > No symbol table info available. > #4 0x00007f1b433e4333 in __dbc_iget () from /lib64/libdb-5.3.so > > No symbol table info available. > #5 0x00007f1b433f2b02 in __dbc_get_pp () from /lib64/libdb-5.3.so > > No symbol table info available. > #6 0x00007f1b3e8ce822 in idl_new_fetch () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > No symbol table info available. > #7 0x00007f1b3e8dcb26 in index_read_ext_allids () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > No symbol table info available. > #8 0x00007f1b3e8c7284 in keys2idl () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > No symbol table info available. > #9 0x00007f1b3e8c79e3 in ava_candidates.isra.0 () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > No symbol table info available. > #10 0x00007f1b3e8c7fd2 in filter_candidates_ext () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > No symbol table info available. > #11 0x00007f1b3e8c9096 in list_candidates () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > No symbol table info available. > #12 0x00007f1b3e8c7f40 in filter_candidates_ext () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > No symbol table info available. > #13 0x00007f1b3e8c9096 in list_candidates () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > No symbol table info available. > #14 0x00007f1b3e8c7f40 in filter_candidates_ext () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > No symbol table info available. > #15 0x00007f1b3e8c9096 in list_candidates () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > No symbol table info available. > #16 0x00007f1b3e8c7f40 in filter_candidates_ext () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > No symbol table info available. > #17 0x00007f1b3e90380c in subtree_candidates () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > No symbol table info available. > #18 0x00007f1b3e904e7f in ldbm_back_search () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > No symbol table info available. > #19 0x00007f1b4b2321d6 in op_shared_search () from > /usr/lib64/dirsrv/libslapd.so.0 > No symbol table info available. > #20 0x00007f1b4b72333e in do_search () > No symbol table info available. > #21 0x00007f1b4b7113c3 in connection_threadmain () > No symbol table info available. > #22 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so > No symbol table info available. > #23 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 > No symbol table info available. > #24 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 > No symbol table info available. > > Thread 19 (Thread 0x7f1b0f7fe700 (LWP 105179)): > #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 () from > /lib64/libpthread.so.0 > No symbol table info available. > #1 0x00007f1b433212f3 in __db_hybrid_mutex_suspend () from > /lib64/libdb-5.3.so > No symbol table info available. > #2 0x00007f1b43320640 in __db_tas_mutex_lock () from > /lib64/libdb-5.3.so > No symbol table info available. > #3 0x00007f1b433cad3a in __lock_get_internal () from > /lib64/libdb-5.3.so > No symbol table info available. > #4 0x00007f1b433cb820 in __lock_get () from /lib64/libdb-5.3.so > > No symbol table info available. > #5 0x00007f1b433f7142 in __db_lget () from /lib64/libdb-5.3.so > > No symbol table info available. > #6 0x00007f1b4333e605 in __bam_search () from /lib64/libdb-5.3.so > > No symbol table info available. > #7 0x00007f1b43329256 in __bamc_search () from > /lib64/libdb-5.3.so > No symbol table info available. > #8 0x00007f1b4332ad0f in __bamc_get () from /lib64/libdb-5.3.so > > No symbol table info available. > #9 0x00007f1b433e3ca6 in __dbc_iget () from /lib64/libdb-5.3.so > > No symbol table info available. > #10 0x00007f1b433f2b02 in __dbc_get_pp () from /lib64/libdb-5.3.so > > No symbol table info available. > #11 0x00007f1b3e8ce680 in idl_new_fetch () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > No symbol table info available. > #12 0x00007f1b3e8dcb26 in index_read_ext_allids () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > No symbol table info available. > #13 0x00007f1b3e8c7284 in keys2idl () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > No symbol table info available. > #14 0x00007f1b3e8c79e3 in ava_candidates.isra.0 () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > No symbol table info available. > #15 0x00007f1b3e8c7fd2 in filter_candidates_ext () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > No symbol table info available. > #16 0x00007f1b3e8c9096 in list_candidates () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > No symbol table info available. > #17 0x00007f1b3e8c7f40 in filter_candidates_ext () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > No symbol table info available. > #18 0x00007f1b3e8c9096 in list_candidates () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > No symbol table info available. > #19 0x00007f1b3e8c7f40 in filter_candidates_ext () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > No symbol table info available. > #20 0x00007f1b3e8c9096 in list_candidates () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > No symbol table info available. > #21 0x00007f1b3e8c7f40 in filter_candidates_ext () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > No symbol table info available. > #22 0x00007f1b3e90380c in subtree_candidates () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > No symbol table info available. > #23 0x00007f1b3e904e7f in ldbm_back_search () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > No symbol table info available. > #24 0x00007f1b4b2321d6 in op_shared_search () from > /usr/lib64/dirsrv/libslapd.so.0 > No symbol table info available. > #25 0x00007f1b4b72333e in do_search () > No symbol table info available. > #26 0x00007f1b4b7113c3 in connection_threadmain () > No symbol table info available. > #27 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so > No symbol table info available. > #28 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 > No symbol table info available. > #29 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 > No symbol table info available. > > Thread 18 (Thread 0x7f1b0ffff700 (LWP 105178)): > #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 () from > /lib64/libpthread.so.0 > No symbol table info available. > #1 0x00007f1b4941b050 in PR_WaitCondVar () from /lib64/libnspr4.so > No symbol table info available. > #2 0x00007f1b4b70efee in connection_wait_for_new_work () > No symbol table info available. > #3 0x00007f1b4b710719 in connection_threadmain () > No symbol table info available. > #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so > No symbol table info available. > #5 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 > No symbol table info available. > #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 > No symbol table info available. > > Thread 17 (Thread 0x7f1b14d28700 (LWP 105177)): > #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 () from > /lib64/libpthread.so.0 > No symbol table info available. > #1 0x00007f1b4941b050 in PR_WaitCondVar () from /lib64/libnspr4.so > No symbol table info available. > #2 0x00007f1b4b70efee in connection_wait_for_new_work () > No symbol table info available. > #3 0x00007f1b4b710719 in connection_threadmain () > No symbol table info available. > #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so > No symbol table info available. > #5 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 > No symbol table info available. > #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 > No symbol table info available. > > Thread 16 (Thread 0x7f1b15529700 (LWP 105176)): > #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 () from > /lib64/libpthread.so.0 > No symbol table info available. > #1 0x00007f1b4941b050 in PR_WaitCondVar () from /lib64/libnspr4.so > No symbol table info available. > #2 0x00007f1b4b70efee in connection_wait_for_new_work () > No symbol table info available. > #3 0x00007f1b4b710719 in connection_threadmain () > No symbol table info available. > #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so > No symbol table info available. > #5 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 > No symbol table info available. > #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 > No symbol table info available. > > Thread 15 (Thread 0x7f1b15d2a700 (LWP 105175)): > #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 () from > /lib64/libpthread.so.0 > No symbol table info available. > #1 0x00007f1b4941b050 in PR_WaitCondVar () from /lib64/libnspr4.so > No symbol table info available. > #2 0x00007f1b4b70efee in connection_wait_for_new_work () > No symbol table info available. > #3 0x00007f1b4b710719 in connection_threadmain () > No symbol table info available. > #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so > No symbol table info available. > #5 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 > No symbol table info available. > #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 > No symbol table info available. > > Thread 14 (Thread 0x7f1b1652b700 (LWP 105174)): > #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 () from > /lib64/libpthread.so.0 > No symbol table info available. > #1 0x00007f1b4941b050 in PR_WaitCondVar () from /lib64/libnspr4.so > No symbol table info available. > #2 0x00007f1b4b70efee in connection_wait_for_new_work () > No symbol table info available. > #3 0x00007f1b4b710719 in connection_threadmain () > No symbol table info available. > #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so > No symbol table info available. > #5 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 > No symbol table info available. > #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 > No symbol table info available. > > Thread 13 (Thread 0x7f1b16d2c700 (LWP 105173)): > #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 () from > /lib64/libpthread.so.0 > No symbol table info available. > #1 0x00007f1b433212f3 in __db_hybrid_mutex_suspend () from > /lib64/libdb-5.3.so > No symbol table info available. > #2 0x00007f1b43320640 in __db_tas_mutex_lock () from > /lib64/libdb-5.3.so > No symbol table info available. > #3 0x00007f1b433cad3a in __lock_get_internal () from > /lib64/libdb-5.3.so > No symbol table info available. > #4 0x00007f1b433cb820 in __lock_get () from /lib64/libdb-5.3.so > > No symbol table info available. > #5 0x00007f1b433f7142 in __db_lget () from /lib64/libdb-5.3.so > > No symbol table info available. > #6 0x00007f1b4333e605 in __bam_search () from /lib64/libdb-5.3.so > > No symbol table info available. > #7 0x00007f1b43329256 in __bamc_search () from > /lib64/libdb-5.3.so > No symbol table info available. > #8 0x00007f1b4332ad0f in __bamc_get () from /lib64/libdb-5.3.so > > No symbol table info available. > #9 0x00007f1b433e3ca6 in __dbc_iget () from /lib64/libdb-5.3.so > > No symbol table info available. > #10 0x00007f1b433f2b02 in __dbc_get_pp () from /lib64/libdb-5.3.so > > No symbol table info available. > #11 0x00007f1b3e8ce680 in idl_new_fetch () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > No symbol table info available. > #12 0x00007f1b3e8dcb26 in index_read_ext_allids () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > No symbol table info available. > #13 0x00007f1b3e8c7284 in keys2idl () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > No symbol table info available. > #14 0x00007f1b3e8c79e3 in ava_candidates.isra.0 () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > No symbol table info available. > #15 0x00007f1b3e8c7fd2 in filter_candidates_ext () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > No symbol table info available. > #16 0x00007f1b3e8c9096 in list_candidates () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > No symbol table info available. > #17 0x00007f1b3e8c7f40 in filter_candidates_ext () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > No symbol table info available. > #18 0x00007f1b3e8c9096 in list_candidates () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > No symbol table info available. > #19 0x00007f1b3e8c7f40 in filter_candidates_ext () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > No symbol table info available. > #20 0x00007f1b3e8c9096 in list_candidates () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > No symbol table info available. > #21 0x00007f1b3e8c7f40 in filter_candidates_ext () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > No symbol table info available. > #22 0x00007f1b3e90380c in subtree_candidates () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > No symbol table info available. > #23 0x00007f1b3e904e7f in ldbm_back_search () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > No symbol table info available. > #24 0x00007f1b4b2321d6 in op_shared_search () from > /usr/lib64/dirsrv/libslapd.so.0 > No symbol table info available. > #25 0x00007f1b4b72333e in do_search () > No symbol table info available. > #26 0x00007f1b4b7113c3 in connection_threadmain () > No symbol table info available. > #27 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so > No symbol table info available. > #28 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 > No symbol table info available. > #29 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 > No symbol table info available. > > Thread 12 (Thread 0x7f1b1752d700 (LWP 105172)): > #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 () from > /lib64/libpthread.so.0 > No symbol table info available. > #1 0x00007f1b4941b050 in PR_WaitCondVar () from /lib64/libnspr4.so > No symbol table info available. > #2 0x00007f1b4b70efee in connection_wait_for_new_work () > No symbol table info available. > #3 0x00007f1b4b710719 in connection_threadmain () > No symbol table info available. > #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so > No symbol table info available. > #5 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 > No symbol table info available. > #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 > No symbol table info available. > > Thread 11 (Thread 0x7f1b17d2e700 (LWP 105171)): > #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 () from > /lib64/libpthread.so.0 > No symbol table info available. > #1 0x00007f1b4941b050 in PR_WaitCondVar () from /lib64/libnspr4.so > No symbol table info available. > #2 0x00007f1b4b70efee in connection_wait_for_new_work () > No symbol table info available. > #3 0x00007f1b4b710719 in connection_threadmain () > No symbol table info available. > #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so > No symbol table info available. > #5 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 > No symbol table info available. > #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 > No symbol table info available. > > Thread 10 (Thread 0x7f1b2509d700 (LWP 105168)): > #0 0x00007f1b48dc5a82 in pthread_cond_timedwait@@GLIBC_2.3.2 () > from /lib64/libpthread.so.0 > No symbol table info available. > #1 0x00007f1b4941ab07 in pt_TimedWait () from /lib64/libnspr4.so > No symbol table info available. > #2 0x00007f1b4941afce in PR_WaitCondVar () from /lib64/libnspr4.so > No symbol table info available. > #3 0x00007f1b4b2061e8 in eq_loop () from > /usr/lib64/dirsrv/libslapd.so.0 > No symbol table info available. > #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so > No symbol table info available. > #5 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 > No symbol table info available. > #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 > No symbol table info available. > > Thread 9 (Thread 0x7f1b2589e700 (LWP 105167)): > #0 0x00007f1b48dc5a82 in pthread_cond_timedwait@@GLIBC_2.3.2 () > from /lib64/libpthread.so.0 > No symbol table info available. > #1 0x00007f1b4941ab07 in pt_TimedWait () from /lib64/libnspr4.so > No symbol table info available. > #2 0x00007f1b4941afce in PR_WaitCondVar () from /lib64/libnspr4.so > No symbol table info available. > #3 0x00007f1b4b719d03 in housecleaning () > No symbol table info available. > #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so > No symbol table info available. > #5 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 > No symbol table info available. > #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 > No symbol table info available. > > Thread 8 (Thread 0x7f1b2609f700 (LWP 105166)): > #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 () from > /lib64/libpthread.so.0 > No symbol table info available. > #1 0x00007f1b4941b050 in PR_WaitCondVar () from /lib64/libnspr4.so > No symbol table info available. > #2 0x00007f1b4b260198 in slapi_wait_condvar () from > /usr/lib64/dirsrv/libslapd.so.0 > No symbol table info available. > #3 0x00007f1b3cf5deed in roles_cache_wait_on_change () from > /usr/lib64/dirsrv/plugins/libroles-plugin.so > No symbol table info available. > #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so > No symbol table info available. > #5 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 > No symbol table info available. > #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 > No symbol table info available. > > Thread 7 (Thread 0x7f1b268a0700 (LWP 105165)): > #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 () from > /lib64/libpthread.so.0 > No symbol table info available. > #1 0x00007f1b4941b050 in PR_WaitCondVar () from /lib64/libnspr4.so > No symbol table info available. > #2 0x00007f1b4b260198 in slapi_wait_condvar () from > /usr/lib64/dirsrv/libslapd.so.0 > No symbol table info available. > #3 0x00007f1b3cf5deed in roles_cache_wait_on_change () from > /usr/lib64/dirsrv/plugins/libroles-plugin.so > No symbol table info available. > #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so > No symbol table info available. > #5 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 > No symbol table info available. > #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 > No symbol table info available. > > Thread 6 (Thread 0x7f1b270a1700 (LWP 105164)): > #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 () from > /lib64/libpthread.so.0 > No symbol table info available. > #1 0x00007f1b4941b050 in PR_WaitCondVar () from /lib64/libnspr4.so > No symbol table info available. > #2 0x00007f1b4b260198 in slapi_wait_condvar () from > /usr/lib64/dirsrv/libslapd.so.0 > No symbol table info available. > #3 0x00007f1b3cf5deed in roles_cache_wait_on_change () from > /usr/lib64/dirsrv/plugins/libroles-plugin.so > No symbol table info available. > #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so > No symbol table info available. > #5 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 > No symbol table info available. > #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 > No symbol table info available. > > Thread 5 (Thread 0x7f1b278a2700 (LWP 105163)): > #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 () from > /lib64/libpthread.so.0 > No symbol table info available. > #1 0x00007f1b4941b050 in PR_WaitCondVar () from /lib64/libnspr4.so > No symbol table info available. > #2 0x00007f1b4b260198 in slapi_wait_condvar () from > /usr/lib64/dirsrv/libslapd.so.0 > No symbol table info available. > #3 0x00007f1b415f262e in cos_cache_wait_on_change () from > /usr/lib64/dirsrv/plugins/libcos-plugin.so > No symbol table info available. > #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so > No symbol table info available. > #5 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 > No symbol table info available. > #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 > No symbol table info available. > > Thread 4 (Thread 0x7f1b282ac700 (LWP 105162)): > #0 0x00007f1b48ae69b3 in select () from /lib64/libc.so.6 > No symbol table info available. > #1 0x00007f1b4b2710e9 in DS_Sleep () from > /usr/lib64/dirsrv/libslapd.so.0 > No symbol table info available. > #2 0x00007f1b3e8ba707 in perf_threadmain () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > No symbol table info available. > #3 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so > No symbol table info available. > #4 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 > No symbol table info available. > #5 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 > No symbol table info available. > > Thread 3 (Thread 0x7f1b28aad700 (LWP 105161)): > #0 0x00007f1b48ae69b3 in select () from /lib64/libc.so.6 > No symbol table info available. > #1 0x00007f1b4b2710e9 in DS_Sleep () from > /usr/lib64/dirsrv/libslapd.so.0 > No symbol table info available. > #2 0x00007f1b3e8bfb7f in trickle_threadmain () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > No symbol table info available. > #3 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so > No symbol table info available. > #4 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 > No symbol table info available. > #5 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 > No symbol table info available. > > Thread 2 (Thread 0x7f1b292ae700 (LWP 105160)): > #0 0x00007f1b48ae69b3 in select () from /lib64/libc.so.6 > No symbol table info available. > #1 0x00007f1b4344dbad in __os_yield () from /lib64/libdb-5.3.so > > No symbol table info available. > #2 0x00007f1b434492b3 in __memp_sync_int () from > /lib64/libdb-5.3.so > No symbol table info available. > #3 0x00007f1b43459752 in __txn_checkpoint () from > /lib64/libdb-5.3.so > No symbol table info available. > #4 0x00007f1b43459b74 in __txn_checkpoint_pp () from > /lib64/libdb-5.3.so > No symbol table info available. > #5 0x00007f1b3e8c3a87 in checkpoint_threadmain () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > No symbol table info available. > #6 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so > No symbol table info available. > #7 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 > No symbol table info available. > #8 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 > No symbol table info available. > > Thread 1 (Thread 0x7f1b29aaf700 (LWP 105159)): > #0 0x00007f1b48ae69b3 in select () from /lib64/libc.so.6 > No symbol table info available. > #1 0x00007f1b4b2710e9 in DS_Sleep () from > /usr/lib64/dirsrv/libslapd.so.0 > No symbol table info available. > #2 0x00007f1b3e8bf907 in deadlock_threadmain () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > No symbol table info available. > #3 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so > No symbol table info available. > #4 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 > No symbol table info available. > #5 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 > No symbol table info available. > > > > Thanks, > > Rakesh > > > On Mon, Sep 5, 2016 at 4:48 PM, thierry bordaz > wrote: > > > > On 09/05/2016 12:05 PM, Rakesh Rajasekharan wrote: >> Hi Thierry, >> >> >> I was getting the hang issue while running ipa-client-install >> simultaneously on few clients.. >> However, today, I am not able to replicate that. >> >> I could not get a gdb . But i will try getting that the next >> time I face this issue. >> >> The CPU does not stay high.. it just momentarily touches a >> high value and then drops down to around 2-7% >> >> One question I have is , is it ok to set it >> nsslapd-threadnumber to a very high value . >> I have around 4000 clients and with nsslapd-maxthreadsperconn >> set to 5..So, can I set nsslapd-threadnumber to around 25000. > > Hello, > > I know some users running in production with several hunderds > of threads (>600) and this without problem. > > I do not recall having suggested to increase that number and > for what reason. > Usually 30 workers is a good enough value. It can create > bootleneck if for some reason each operation is very long to > satisfy and exhaust the number of workers. You can monitor the > work queue: > > ldapsearch -D "cn=directory manager" -w xxx -LLL -b > "cn=monitor" -s base opsinitiated opscompleted > > > If opscompleted-opsinitiated remains close to threadnumber, > then yes it would be valuable to increase it. > > The computation #client * #async_op_per_client sound an > overkill. Even if all clients send at the exact same time all > their requests, it is very likely that some common resource > (db page, log, allocator...) will serialize them. If you > monitor a need to increase the work, you would for example set > it to 50, then monitor, then set it to 100, then monitor... > until you find a good enough value. > Note the increasing the #thread, increases the memory > footprint that will reduce the efficiency of file system cache > and can increase the response time. > > > best regards > thierry > >> >> Thanks >> >> On Mon, Sep 5, 2016 at 1:03 PM, thierry bordaz >> > wrote: >> >> >> Hi Rakesh, >> >> Were you able to get a pstack or full stack with gdb >> (http://www.port389.org/docs/389ds/FAQ/faq.html#debugging-crashes) >> when the server hangs ? >> >> If it happens with 500 threads as well as with 30, using >> 30 threads is a better choice to debug this issue. >> I will try to reproduce using 150 parallel 'ipa user-find >> p-testipa' commands >> >> Something I am unsure is if the CPU consumption stays >> high (you mentioned 340% CPU usage) as long as the hang >> happens or if after a suddent shot up to 340% (that marks >> the beginning of the hang) it drops and stay hanging ? >> >> thanks >> thierry >> >> On 09/04/2016 08:40 PM, Rakesh Rajasekharan wrote: >>> starce on the slapd process actually had this in the >>> output.. >>> FUTEX_WAIT_PRIVATE >>> >>> and checking for the number of threads slapd had.. there >>> were 5015 threads >>> >>> ps -efL|grep slapd|wc -l >>> 5015 >>> >>> strace on most of the threads gave this output >>> >>> strace -p 67411 >>> Process 67411 attached >>> futex(0x7f3f0226b41c, FUTEX_WAIT_PRIVATE, 1, NULL) = -1 >>> EAGAIN (Resource temporarily unavailable) >>> futex(0x7f3f0226b41c, FUTEX_WAIT_PRIVATE, 2, >>> NULL^CProcess 67411 detached >>> >>> >>> >>> >>> >>> On Sun, Sep 4, 2016 at 5:34 PM, Rakesh Rajasekharan >>> >> > wrote: >>> >>> I have again got the issue of IPA hanging.. The >>> issue came up when i tried to run ipa-client-isntall >>> on 142 clients simultaneously >>> >>> >>> None of the IPA commands are responding, and I see >>> this error >>> >>> ipa user-find p-testipa >>> ipa: ERROR: Insufficient access: SASL(-1): generic >>> failure: GSSAPI Error: Unspecified GSS failure. >>> Minor code may provide more information (KDC >>> returned error string: PROCESS_TGS) >>> >>> KRB5_TRACE=/dev/stdout kinit admin >>> [41178] 1472984115.233214: Getting initial >>> credentials for admin at XYZ.COM >>> [41178] 1472984115.235257: Sending request (167 >>> bytes) to XYZ.COM >>> [41178] 1472984115.235419: Initiating TCP connection >>> to stream 10.1.3.36:88 >>> [41178] 1472984115.235685: Sending TCP request to >>> stream 10.1.3.36:88 >>> [41178] 1472984120.238914: Received answer (174 >>> bytes) from stream 10.1.3.36:88 >>> [41178] 1472984120.238925: Terminating TCP >>> connection to stream 10.1.3.36:88 >>> [41178] 1472984120.238993: Response was from master KDC >>> [41 >>> >>> >>> Running an ldapsearch to see the db.. does not give >>> any results and just hangs there >>> >>> ldapsearch -x -D 'cn=Directory Manager' -W -s one -b >>> 'cn=kerberos,dc=xyz,dc=com' >>> Enter LDAP Password: >>> >>> even an ldapsearch -x does not respond >>> At this point, am sure that slapd is the one causing >>> issues >>> >>> Running an strace against the hung slapd itself >>> seems to get stuck does not proceed after saying >>> "attaching to process" >>> >>> From some others posts I read Thierry suggesting to >>> increase the nsslapd-threadnumber value >>> >>> It was set to 30, I think that might be too low. >>> >>> I have raised it to 500 >>> >>> Now after restarting the service .. ldapsearch >>> starts responding. >>> But running the test to add a sudden high number of >>> clients again left ns-slapd to hung state >>> >>> When i attempted adding the clients.. the ns-slapd >>> cpu usage shot up to 340% and after that ns-slapd >>> stopped responding >>> >>> So now, atleast I know what might be causing the >>> issue and I can now easily reproduce it. >>> >>> Is there a way I can make ns-slapd handle a sudden >>> bump in incoming request for ipa-client-install >>> >>> Thanks >>> Rakesh >>> >>> >>> >>> >>> >>> >>> On Mon, Aug 29, 2016 at 11:18 PM, Rich Megginson >>> > >>> wrote: >>> >>> On 08/29/2016 10:53 AM, Rakesh Rajasekharan wrote: >>>> Hi Thierry, >>>> >>>> My machine has 30GB RAM ..and 389-ds version >>>> is 1.3.4 >>>> >>>> ldapsearch shows the values for >>>> nsslapd-cachememsize updated to 200MB. >>>> >>>> ldapsearch -LLL -o ldif-wrap=no -D >>>> "cn=directory manager" -w 'mypassword' -b >>>> 'cn=userRoot,cn=ldbm >>>> database,cn=plugins,cn=config'|grep >>>> nsslapd-cachememsize >>>> nsslapd-cachememsize: 209715200 >>>> >>>> >>>> So, it seems to have updated though seeing that >>>> warning(WARNING: ipaca: entry cache size >>>> 10485760B is less than db size 11599872B) in >>>> the log confuses me a bit. >>>> >>>> Thers one more entry that I found from the >>>> ldapsearch to be bit low >>>> >>>> nsslapd-dncachememsize: 10485760 >>>> maxdncachesize: 10485760 >>>> >>>> Should I update these as well to a higher value >>>> >>>> At the time when the issue happened, the memory >>>> usage as well as the overall load of the system >>>> was very low . >>>> I will try reproducing the issue atleast in my >>>> QA env..probably by trying to mock simultaneous >>>> parallel logins to a large number of hosts >>> >>> To monitor your cache sizes, please use the >>> dbmon.sh tool provided with your distro. If >>> that is not available with your particular >>> distro, see >>> https://github.com/richm/scripts/wiki/dbmon.sh >>> >>> >>> >>>> >>>> >>>> thanks >>>> Rakesh >>>> >>>> >>>> >>>> >>>> On Mon, Aug 29, 2016 at 8:16 PM, thierry bordaz >>>> >>> > wrote: >>>> >>>> Hi Rakesh, >>>> >>>> Those tuning may depend on the memory >>>> available on your machine. >>>> nsslapd-cachememsize allows the entry cache >>>> to consume up to 200Mb but its memory >>>> footprint is known to go above. >>>> 200Mb both looks pretty good to me. How >>>> large is your machine ? What is your >>>> version of 389-ds ? >>>> >>>> Those warnings do not change your settings. >>>> It just raise that entry cache of 'ipaca' >>>> and 'retrocl' are small but it is fine. The >>>> size of the entry cache is important mostly >>>> in userRoot. >>>> You may double check the actual values, >>>> after restart, with ldapsearch on >>>> 'cn=userRoot,cn=ldbm >>>> database,cn=plugins,cn=config' and >>>> 'cn=config,cn=ldbm >>>> database,cn=plugins,cn=config'. >>>> >>>> A step is to know what will be response >>>> time of DS to know if it is responsible of >>>> the hang or not. >>>> The logs and possibly pstack during those >>>> intermittent hangs will help to determine that. >>>> >>>> regards >>>> thierry >>>> >>>> >>>> >>>> >>>> >>>> On 08/29/2016 04:25 PM, Rakesh Rajasekharan >>>> wrote: >>>>> I tried increasing the nsslapd-dbcachesize >>>>> and nsslapd-cachememsize in my QA envs to >>>>> 200MB. >>>>> >>>>> However, in my log files, I still see this >>>>> message >>>>> [29/Aug/2016:04:34:37 +0000] - WARNING: >>>>> ipaca: entry cache size 10485760B is less >>>>> than db size 11599872B; We recommend to >>>>> increase the entry cache size >>>>> nsslapd-cachememsize. >>>>> [29/Aug/2016:04:34:37 +0000] - WARNING: >>>>> changelog: entry cache size 2097152B is >>>>> less than db size 441647104B; We recommend >>>>> to increase the entry cache size >>>>> nsslapd-cachememsize. >>>>> >>>>> these are my ldif files that i used to >>>>> modify the values >>>>> modify entry cache size >>>>> cat modify-cache-mem-size.ldif >>>>> dn: cn=userRoot,cn=ldbm >>>>> database,cn=plugins,cn=config >>>>> changetype: modify >>>>> replace: nsslapd-cachememsize >>>>> nsslapd-cachememsize: 209715200 >>>>> >>>>> modify db cache size >>>>> cat modfy-db-cache-size.ldif >>>>> dn: cn=config,cn=ldbm >>>>> database,cn=plugins,cn=config >>>>> changetype: modify >>>>> replace: nsslapd-dbcachesize >>>>> nsslapd-dbcachesize: 209715200 >>>>> >>>>> After modifying , i restarted IPA services >>>>> >>>>> Is there anything else that I need to >>>>> take care of as the logs suggest its still >>>>> not getting the updated values >>>>> >>>>> Thanks >>>>> Rakesh >>>>> >>>>> On Mon, Aug 29, 2016 at 6:07 PM, Rakesh >>>>> Rajasekharan >>>>> >>>> > wrote: >>>>> >>>>> Hi Thierry, >>>>> >>>>> Coz of the issues we had to revert >>>>> back to earlier running openldap in >>>>> production. >>>>> >>>>> I have now done a few TCP related >>>>> changes in sysctl.conf and have also >>>>> increased the nsslapd-dbcachesize and >>>>> nsslapd-cachememsize to 200MB >>>>> >>>>> I will again start migrating hosts >>>>> back to IPA and see if I face the >>>>> earlier issue. >>>>> >>>>> I will update back once I have something >>>>> >>>>> >>>>> Thanks, >>>>> Rakesh >>>>> >>>>> >>>>> >>>>> On Thu, Aug 25, 2016 at 2:17 PM, >>>>> thierry bordaz >>>> > wrote: >>>>> >>>>> >>>>> >>>>> On 08/25/2016 10:15 AM, Rakesh >>>>> Rajasekharan wrote: >>>>>> All of the troubleshooting seems >>>>>> fine. >>>>>> >>>>>> >>>>>> However, Running libconv.pl >>>>>> gives me this >>>>>> output >>>>>> >>>>>> ----- Recommendations ----- >>>>>> >>>>>> 1. You have unindexed >>>>>> components, this can be caused >>>>>> from a search on an unindexed >>>>>> attribute, or your returned >>>>>> results exceeded the >>>>>> allidsthreshold. Unindexed >>>>>> components are not recommended. >>>>>> To refuse unindexed searches, >>>>>> switch 'nsslapd-require-index' to >>>>>> 'on' under your database entry >>>>>> (e.g. cn=UserRoot,cn=ldbm >>>>>> database,cn=plugins,cn=config). >>>>>> >>>>>> 2. You have a significant >>>>>> difference between binds and >>>>>> unbinds. You may want to >>>>>> investigate this difference. >>>>>> >>>>>> >>>>>> I feel, this could be a pointer >>>>>> to things going slow.. and IPA >>>>>> hanging. I think i now have >>>>>> something that I can try and nail >>>>>> down this issue. >>>>>> >>>>>> On a sidenote, I was earlier >>>>>> running openldap and migrated >>>>>> over to Freeipa, >>>>>> >>>>>> Thanks >>>>>> Rakesh >>>>>> >>>>>> >>>>>> >>>>>> On Wed, Aug 24, 2016 at 12:38 PM, >>>>>> Petr Spacek >>>>> > wrote: >>>>>> >>>>>> On 23.8.2016 18:44, Rakesh >>>>>> Rajasekharan wrote: >>>>>> > I think thers something >>>>>> seriously wrong with my system >>>>>> > >>>>>> > not able to run any IPA >>>>>> commands >>>>>> > >>>>>> > klist >>>>>> > Ticket cache: >>>>>> KEYRING:persistent:0:0 >>>>>> > Default principal: >>>>>> admin at XYZ.COM >>>>>> >>>>>> > >>>>>> > Valid starting Expires >>>>>> Service principal >>>>>> > 2016-08-23T16:26:36 >>>>>> 2016-08-24T16:26:22 >>>>>> krbtgt/XYZ.COM at XYZ.COM >>>>>> >>>>>> > >>>>>> > >>>>>> > [root at prod-ipa-master-1a >>>>>> :~] ipactl status >>>>>> > Directory Service: RUNNING >>>>>> > krb5kdc Service: RUNNING >>>>>> > kadmin Service: RUNNING >>>>>> > ipa_memcached Service: RUNNING >>>>>> > httpd Service: RUNNING >>>>>> > pki-tomcatd Service: RUNNING >>>>>> > ipa-otpd Service: RUNNING >>>>>> > ipa: INFO: The ipactl >>>>>> command was successful >>>>>> > >>>>>> > >>>>>> > >>>>>> > [root at prod-ipa-master :~] >>>>>> ipa user-find p-testuser >>>>>> > ipa: ERROR: Kerberos error: >>>>>> ('Unspecified GSS failure. >>>>>> Minor code may >>>>>> > provide more information', >>>>>> 851968)/("Cannot contact any >>>>>> KDC for realm ' >>>>>> > XYZ.COM '", >>>>>> -1765328228) >>>>>> >>>>> >>>>> Hi Rakesh, >>>>> >>>>> Having a reproducible test >>>>> case would you rerun the >>>>> command above. >>>>> During its processing you may >>>>> monitor DS process load (top). >>>>> If it is high, you may get >>>>> some pstacks of it. >>>>> Also would you attach the part >>>>> of DS access logs taken during >>>>> the command. >>>>> >>>>> regards >>>>> thierry >>>>> >>>>>> > >>>>>> >>>>>> This is weird because the >>>>>> server seems to be up. >>>>>> >>>>>> Please follow >>>>>> http://www.freeipa.org/page/Troubleshooting#Authentication.2FKerberos >>>>>> >>>>>> Petr^2 Spacek >>>>>> >>>>>> > >>>>>> > >>>>>> > Thanks >>>>>> > >>>>>> > Rakesh >>>>>> > >>>>>> > On Tue, Aug 23, 2016 at >>>>>> 10:01 PM, Rakesh Rajasekharan < >>>>>> > >>>>>> rakesh.rajasekharan at gmail.com >>>>>> > >>>>>> wrote: >>>>>> > >>>>>> >> i changed the loggin level >>>>>> to 4 . Modifying >>>>>> nsslapd-accesslog-level >>>>>> >> >>>>>> >> But, the hang is still >>>>>> there. though I dont see the >>>>>> sigfault now >>>>>> >> >>>>>> >> >>>>>> >> >>>>>> >> >>>>>> >> On Tue, Aug 23, 2016 at >>>>>> 9:02 PM, Rakesh Rajasekharan < >>>>>> >> >>>>>> rakesh.rajasekharan at gmail.com >>>>>> > >>>>>> wrote: >>>>>> >> >>>>>> >>> My disk was getting >>>>>> filled too fast >>>>>> >>> >>>>>> >>> logs under >>>>>> /var/log/dirsrv was coming >>>>>> around 5 gb quickly filling up >>>>>> >>> >>>>>> >>> Is there a way to make >>>>>> the logging less verbose >>>>>> >>> >>>>>> >>> >>>>>> >>> >>>>>> >>> On Tue, Aug 23, 2016 at >>>>>> 6:41 PM, Petr Spacek >>>>>> >>>>> > >>>>>> wrote: >>>>>> >>> >>>>>> >>>> On 23.8.2016 15:07, >>>>>> Rakesh Rajasekharan wrote: >>>>>> >>>>> I was able to fix that >>>>>> may be temporarily... when i >>>>>> checked the >>>>>> >>>> network.. >>>>>> >>>>> there was another >>>>>> process that was running and >>>>>> consuming a lot of >>>>>> >>>> network ( >>>>>> >>>>> i have no idea who did >>>>>> that. I need to seriously >>>>>> start restricting >>>>>> >>>> people >>>>>> >>>>> access to this machine ) >>>>>> >>>>> >>>>>> >>>>> after killing that >>>>>> perfomance improved drastically >>>>>> >>>>> >>>>>> >>>>> But now, suddenly I >>>>>> started experiencing the same >>>>>> hang. >>>>>> >>>>> >>>>>> >>>>> This time , I gert the >>>>>> following error when checked >>>>>> dmesg >>>>>> >>>>> >>>>>> >>>>> [ 301.236976] >>>>>> ns-slapd[3124]: segfault at 0 >>>>>> ip 00007f1de416951c sp >>>>>> >>>>> 00007f1dee1dba70 error >>>>>> 4 in >>>>>> libcos-plugin.so[7f1de4166000+b000] >>>>>> >>>>> [ 1116.248431] TCP: >>>>>> request_sock_TCP: Possible >>>>>> SYN flooding on port 88. >>>>>> >>>>> Sending cookies. Check >>>>>> SNMP counters. >>>>>> >>>>> [11831.397037] >>>>>> ns-slapd[22550]: segfault at >>>>>> 0 ip 00007f533d82251c sp >>>>>> >>>>> 00007f5347894a70 error >>>>>> 4 in >>>>>> libcos-plugin.so[7f533d81f000+b000] >>>>>> >>>>> [11832.727989] >>>>>> ns-slapd[22606]: segfault at >>>>>> 0 ip 00007f6231eb951c sp >>>>>> >>>>> 00007f623bf2ba70 error >>>>>> 4 in >>>>>> libcos-plugin.so[7f6231eb6000+b00 >>>>>> >>>> >>>>>> >>>> Okay, this one is >>>>>> serious. The LDAP server crashed. >>>>>> >>>> >>>>>> >>>> 1. Make sure all your >>>>>> packages are up-to-date. >>>>>> >>>> >>>>>> >>>> Please see >>>>>> >>>> >>>>>> http://directory.fedoraproject.org/docs/389ds/FAQ/faq.html#d >>>>>> >>>> ebugging-crashes >>>>>> >>>> for further instructions >>>>>> how to debug this. >>>>>> >>>> >>>>>> >>>> Petr^2 Spacek >>>>>> >>>> >>>>>> >>>>> >>>>>> >>>>> and in >>>>>> /var/log/dirsrv/example-com/errors >>>>>> >>>>> >>>>>> >>>>> [23/Aug/2016:12:49:36 >>>>>> +0000] DSRetroclPlugin - >>>>>> delete_changerecord: >>>>>> >>>> could >>>>>> >>>>> not delete change >>>>>> record 3291138 (rc: 32) >>>>>> >>>>> [23/Aug/2016:12:49:36 >>>>>> +0000] DSRetroclPlugin - >>>>>> delete_changerecord: >>>>>> >>>> could >>>>>> >>>>> not delete change >>>>>> record 3291139 (rc: 32) >>>>>> >>>>> [23/Aug/2016:12:49:36 >>>>>> +0000] DSRetroclPlugin - >>>>>> delete_changerecord: >>>>>> >>>> could >>>>>> >>>>> not delete change >>>>>> record 3291140 (rc: 32) >>>>>> >>>>> [23/Aug/2016:12:49:36 >>>>>> +0000] DSRetroclPlugin - >>>>>> delete_changerecord: >>>>>> >>>> could >>>>>> >>>>> not delete change >>>>>> record 3291141 (rc: 32) >>>>>> >>>>> [23/Aug/2016:12:49:36 >>>>>> +0000] DSRetroclPlugin - >>>>>> delete_changerecord: >>>>>> >>>> could >>>>>> >>>>> not delete change >>>>>> record 3291142 (rc: 32) >>>>>> >>>>> [23/Aug/2016:12:49:36 >>>>>> +0000] DSRetroclPlugin - >>>>>> delete_changerecord: >>>>>> >>>> could >>>>>> >>>>> not delete change >>>>>> record 3291143 (rc: 32) >>>>>> >>>>> [23/Aug/2016:12:49:36 >>>>>> +0000] DSRetroclPlugin - >>>>>> delete_changerecord: >>>>>> >>>> could >>>>>> >>>>> not delete change >>>>>> record 3291144 (rc: 32) >>>>>> >>>>> [23/Aug/2016:12:49:36 >>>>>> +0000] DSRetroclPlugin - >>>>>> delete_changerecord: >>>>>> >>>> could >>>>>> >>>>> not delete change >>>>>> record 3291145 (rc: 32) >>>>>> >>>>> [23/Aug/2016:12:49:50 >>>>>> +0000] - Retry count exceeded >>>>>> in delete >>>>>> >>>>> [23/Aug/2016:12:49:50 >>>>>> +0000] DSRetroclPlugin - >>>>>> delete_changerecord: >>>>>> >>>> could >>>>>> >>>>> not delete change >>>>>> record 3292734 (rc: 51) >>>>>> >>>>> >>>>>> >>>>> >>>>>> >>>>> Can i do something >>>>>> about this error.. I treid to >>>>>> restart ipa a couple >>>>>> >>>> of >>>>>> >>>>> time but that did not help >>>>>> >>>>> >>>>>> >>>>> Thanks >>>>>> >>>>> Rakesh >>>>>> >>>>> >>>>>> >>>>> On Mon, Aug 22, 2016 at >>>>>> 2:27 PM, Petr Spacek >>>>>> >>>>> > >>>>>> >>>> wrote: >>>>>> >>>>> >>>>>> >>>>>> On 19.8.2016 19:32, >>>>>> Rakesh Rajasekharan wrote: >>>>>> >>>>>>> I am running my set >>>>>> up on AWS cloud, and entropy >>>>>> is low at around >>>>>> >>>> 180 . >>>>>> >>>>>>> >>>>>> >>>>>>> I plan to increase it >>>>>> bu installing haveged . But, >>>>>> would low entropy >>>>>> >>>> by >>>>>> >>>>>> any >>>>>> >>>>>>> chance cause this >>>>>> issue of intermittent hang . >>>>>> >>>>>>> Also, the hang is >>>>>> mostly observed when >>>>>> registering around 20 clients >>>>>> >>>>>>> together >>>>>> >>>>>> >>>>>> >>>>>> Possibly, I'm not >>>>>> sure. If you want to dig into >>>>>> this, I would do this: >>>>>> >>>>>> 1. look what process >>>>>> hangs on client (using pstree >>>>>> command or so) >>>>>> >>>>>> $ pstree >>>>>> >>>>>> >>>>>> >>>>>> 2. look to what server >>>>>> and port is the hanging >>>>>> client connected to >>>>>> >>>>>> $ lsof -p >>>>> hanging process> >>>>>> >>>>>> >>>>>> >>>>>> 3. jump to server and >>>>>> see what process is bound to >>>>>> the target port >>>>>> >>>>>> $ netstat -pn >>>>>> >>>>>> >>>>>> >>>>>> 4. see where the >>>>>> process if hanging >>>>>> >>>>>> $ strace -p >>>>> the hanging process> >>>>>> >>>>>> >>>>>> >>>>>> I hope it helps. >>>>>> >>>>>> >>>>>> >>>>>> Petr^2 Spacek >>>>>> >>>>>> >>>>>> >>>>>>> On Fri, Aug 19, 2016 >>>>>> at 7:24 PM, Rakesh Rajasekharan < >>>>>> >>>>>>> >>>>>> rakesh.rajasekharan at gmail.com >>>>>> > >>>>>> wrote: >>>>>> >>>>>>> >>>>>> >>>>>>>> yes there seems to >>>>>> be something thats worrying.. >>>>>> I have faced this >>>>>> >>>> today >>>>>> >>>>>>>> as well. >>>>>> >>>>>>>> There are few hosts >>>>>> around 280 odd left and when >>>>>> i try adding them >>>>>> >>>> to >>>>>> >>>>>> IPA >>>>>> >>>>>>>> , the slowness begins.. >>>>>> >>>>>>>> >>>>>> >>>>>>>> all the ipa commands >>>>>> like ipa user-find.. etc >>>>>> becomes very slow in >>>>>> >>>>>>>> responding. >>>>>> >>>>>>>> >>>>>> >>>>>>>> the SYNC_RECV are >>>>>> not many though just around >>>>>> 80-90 and today that >>>>>> >>>> was >>>>>> >>>>>>>> around 20 only >>>>>> >>>>>>>> >>>>>> >>>>>>>> >>>>>> >>>>>>>> I have for now >>>>>> increased tcp_max_syn_backlog >>>>>> to 5000. >>>>>> >>>>>>>> For now the slowness >>>>>> seems to have gone.. but I >>>>>> will do a try >>>>>> >>>> adding the >>>>>> >>>>>>>> clients again >>>>>> tomorrow and see how it goes >>>>>> >>>>>>>> >>>>>> >>>>>>>> Thanks >>>>>> >>>>>>>> Rakesh >>>>>> >>>>>>>> >>>>>> >>>>>>>> The issues >>>>>> >>>>>>>> >>>>>> >>>>>>>> On Fri, Aug 19, 2016 >>>>>> at 12:58 PM, Petr Spacek >>>>>> >>>>> > >>>>>> >>>>>> wrote: >>>>>> >>>>>>>> >>>>>> >>>>>>>>> On 18.8.2016 17:23, >>>>>> Rakesh Rajasekharan wrote: >>>>>> >>>>>>>>>> Hi >>>>>> >>>>>>>>>> >>>>>> >>>>>>>>>> I am migrating to >>>>>> freeipa from openldap and >>>>>> have around 4000 >>>>>> >>>> clients >>>>>> >>>>>>>>>> >>>>>> >>>>>>>>>> I had openned a >>>>>> another thread on that, but >>>>>> chose to start a new >>>>>> >>>> one >>>>>> >>>>>>>>> here >>>>>> >>>>>>>>>> as its a separate >>>>>> issue >>>>>> >>>>>>>>>> >>>>>> >>>>>>>>>> I was able to >>>>>> change the >>>>>> nssslapd-maxdescriptors >>>>>> adding an ldif >>>>>> >>>> file >>>>>> >>>>>>>>>> >>>>>> >>>>>>>>>> cat >>>>>> nsslapd-modify.ldif >>>>>> >>>>>>>>>> dn: cn=config >>>>>> >>>>>>>>>> changetype: modify >>>>>> >>>>>>>>>> replace: >>>>>> nsslapd-maxdescriptors >>>>>> >>>>>>>>>> >>>>>> nsslapd-maxdescriptors: 17000 >>>>>> >>>>>>>>>> >>>>>> >>>>>>>>>> and running the >>>>>> ldapmodify command >>>>>> >>>>>>>>>> >>>>>> >>>>>>>>>> I have now started >>>>>> moving clients running an >>>>>> openldap to Freeipa >>>>>> >>>> and >>>>>> >>>>>>>>> have >>>>>> >>>>>>>>>> today moved close >>>>>> to 2000 clients >>>>>> >>>>>>>>>> >>>>>> >>>>>>>>>> However, I have >>>>>> noticed that IPA hangs >>>>>> intermittently. >>>>>> >>>>>>>>>> >>>>>> >>>>>>>>>> running a kinit >>>>>> admin returns the below error >>>>>> >>>>>>>>>> kinit: Generic >>>>>> error (see e-text) while >>>>>> getting initial >>>>>> >>>> credentials >>>>>> >>>>>>>>>> >>>>>> >>>>>>>>>> from the >>>>>> /var/log/messages, I see this >>>>>> entry >>>>>> >>>>>>>>>> >>>>>> >>>>>>>>>> >>>>>> prod-ipa-master-int kernel: >>>>>> [104090.315801] TCP: >>>>>> >>>> request_sock_TCP: >>>>>> >>>>>>>>>> Possible SYN >>>>>> flooding on port 88. Sending >>>>>> cookies. Check SNMP >>>>>> >>>>>> counters. >>>>>> >>>>>>>>> >>>>>> >>>>>>>>> I would be worried >>>>>> about this message. Maybe >>>>>> kernel/firewall is >>>>>> >>>> doing >>>>>> >>>>>>>>> something fishy >>>>>> behind your back and blocking >>>>>> some connections or >>>>>> >>>> so. >>>>>> >>>>>>>>> >>>>>> >>>>>>>>> Petr^2 Spacek >>>>>> >>>>>>>>> >>>>>> >>>>>>>>> >>>>>> >>>>>>>>>> Aug 18 13:00:01 >>>>>> prod-ipa-master-int >>>>>> systemd[1]: Started Session >>>>>> >>>> 4885 >>>>>> >>>>>> of >>>>>> >>>>>>>>>> user root. >>>>>> >>>>>>>>>> Aug 18 13:00:01 >>>>>> prod-ipa-master-int >>>>>> systemd[1]: Starting Session >>>>>> >>>> 4885 >>>>>> >>>>>> of >>>>>> >>>>>>>>>> user root. >>>>>> >>>>>>>>>> Aug 18 13:01:01 >>>>>> prod-ipa-master-int >>>>>> systemd[1]: Started Session >>>>>> >>>> 4886 >>>>>> >>>>>> of >>>>>> >>>>>>>>>> user root. >>>>>> >>>>>>>>>> Aug 18 13:01:01 >>>>>> prod-ipa-master-int >>>>>> systemd[1]: Starting Session >>>>>> >>>> 4886 >>>>>> >>>>>> of >>>>>> >>>>>>>>>> user root. >>>>>> >>>>>>>>>> Aug 18 13:02:40 >>>>>> prod-ipa-master-int >>>>>> python[28984]: ansible-command >>>>>> >>>>>>>>> Invoked >>>>>> >>>>>>>>>> with creates=None >>>>>> executable=None shell=True >>>>>> args= removes=None >>>>>> >>>>>>>>> warn=True >>>>>> >>>>>>>>>> chdir=None >>>>>> >>>>>>>>>> Aug 18 13:04:37 >>>>>> prod-ipa-master-int sssd_be: >>>>>> GSSAPI Error: >>>>>> >>>> Unspecified >>>>>> >>>>>>>>> GSS >>>>>> >>>>>>>>>> failure. Minor >>>>>> code may provide more >>>>>> information (KDC returned >>>>>> >>>> error >>>>>> >>>>>>>>>> string: PROCESS_TGS) >>>>>> >>>>>>>>>> >>>>>> >>>>>>>>>> Could it be >>>>>> possible that its due to the >>>>>> initial load of adding >>>>>> >>>> the >>>>>> >>>>>>>>> clients >>>>>> >>>>>>>>>> or is there >>>>>> something else that I need to >>>>>> take care of. >>>>>> >>>>>> >>>>>> >>>>>> >>>>> >>>>> >>>>> >>>> >>>> >>>> >>>> >>> >>> >>> -- >>> Manage your subscription for the Freeipa-users >>> mailing list: >>> https://www.redhat.com/mailman/listinfo/freeipa-users >>> >>> Go to http://freeipa.org for more info on the >>> project >>> >>> >>> >>> >>> >> >> > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From lkrispen at redhat.com Mon Sep 5 12:44:45 2016 From: lkrispen at redhat.com (Ludwig Krispenz) Date: Mon, 05 Sep 2016 14:44:45 +0200 Subject: [Freeipa-users] Freeipa 4.2.0 hangs intermittently In-Reply-To: <57CD659B.6050707@redhat.com> References: <57BEB0AB.2090506@redhat.com> <57C44AC6.1030402@redhat.com> <57CD1FDC.2010109@redhat.com> <57CD5488.60404@redhat.com> <57CD659B.6050707@redhat.com> Message-ID: <57CD68BD.9030802@redhat.com> On 09/05/2016 02:31 PM, thierry bordaz wrote: > Hi Rakesh, > > Thanks you soooo much for you continuous help on this !! > > If the hang is still going on, would you dump the DB lock information > with: > > db_stat -CA -N -h /var/lib/dirsrv/slapd-/db > >/tmp/db_lock_info > > It is looking like several threads are waiting for one/several db > locks. With the lock info we will know which thread acquired it/them. > A guess is that it could be the checkpointing thread but this one is > also waiting (on a memory pool). > > I am copying Ludwig on this thread in case this pstack rings any bell. the frequent presence of __db_tas_mutex_lock() looks like the NUMA problem of libdb, although the threads already have stopped spinning (maybe this was an effect of gcore). If the hang aoccurs again, could you check cpu consumption top -H -p and get a pstack pstack > > best regards > thierry > > On 09/05/2016 02:05 PM, Rakesh Rajasekharan wrote: >> Hi Thierry, >> >> I was able to now reporduce the issue.. >> >> the ns-slapd process just got hung,.. however, no core file was >> generate.. >> >> I manually created a coredump using gcore and then took a strace of it >> this is strace from the coredump generated >> >> GNU gdb (GDB) Red Hat Enterprise Linux 7.6.1-80.el7 >> Copyright (C) 2013 Free Software Foundation, Inc. >> License GPLv3+: GNU GPL version 3 or later >> >> This is free software: you are free to change and redistribute it. >> There is NO WARRANTY, to the extent permitted by law. Type "show copying" >> and "show warranty" for details. >> This GDB was configured as "x86_64-redhat-linux-gnu". >> For bug reporting instructions, please see: >> ... >> Reading symbols from /usr/sbin/ns-slapd...Reading symbols from >> /usr/sbin/ns-slapd...(no debugging symbols found)...done. >> (no debugging symbols found)...done. >> [New LWP 105159] >> [New LWP 105160] >> [New LWP 105161] >> [New LWP 105162] >> [New LWP 105163] >> [New LWP 105164] >> [New LWP 105165] >> [New LWP 105166] >> [New LWP 105167] >> [New LWP 105168] >> [New LWP 105171] >> [New LWP 105172] >> [New LWP 105173] >> [New LWP 105174] >> [New LWP 105175] >> [New LWP 105176] >> [New LWP 105177] >> [New LWP 105178] >> [New LWP 105179] >> [New LWP 105180] >> [New LWP 105181] >> [New LWP 105182] >> [New LWP 105183] >> [New LWP 105184] >> [New LWP 105185] >> [New LWP 105186] >> [New LWP 105187] >> [New LWP 105188] >> [New LWP 105189] >> [New LWP 105190] >> [New LWP 105191] >> [New LWP 105192] >> [New LWP 105193] >> [New LWP 105194] >> [New LWP 105195] >> [New LWP 105196] >> [New LWP 105197] >> [New LWP 105198] >> [New LWP 105199] >> [New LWP 105200] >> [New LWP 105201] >> [New LWP 105203] >> [New LWP 105215] >> [New LWP 105158] >> [Thread debugging using libthread_db enabled] >> Using host libthread_db library "/lib64/libthread_db.so.1". >> Core was generated by `/usr/sbin/ns-slapd'. >> #0 0x00007f1b48ae69b3 in select () from /lib64/libc.so.6 >> >> Thread 44 (Thread 0x7f1b4b6d9840 (LWP 105158)): >> #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 () from >> /lib64/libpthread.so.0 >> No symbol table info available. >> #1 0x00007f1b4941b2c3 in PR_EnterMonitor () from /lib64/libnspr4.so >> No symbol table info available. >> #2 0x00007f1b4b7125ed in setup_pr_read_pds () >> No symbol table info available. >> #3 0x00007f1b4b715fd9 in slapd_daemon () >> No symbol table info available. >> #4 0x00007f1b4b707c6c in main () >> No symbol table info available. >> >> Thread 43 (Thread 0x7f1a5df0f700 (LWP 105215)): >> #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 () from >> /lib64/libpthread.so.0 >> No symbol table info available. >> #1 0x00007f1b4941b050 in PR_WaitCondVar () from /lib64/libnspr4.so >> No symbol table info available. >> #2 0x00007f1b4b71df25 in ps_send_results () >> No symbol table info available. >> #3 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so >> No symbol table info available. >> #4 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 >> No symbol table info available. >> #5 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 >> No symbol table info available. >> >> Thread 42 (Thread 0x7f1b03fe7700 (LWP 105203)): >> #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 () from >> /lib64/libpthread.so.0 >> No symbol table info available. >> #1 0x00007f1b4941b050 in PR_WaitCondVar () from /lib64/libnspr4.so >> No symbol table info available. >> #2 0x00007f1b4b71df25 in ps_send_results () >> No symbol table info available. >> #3 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so >> No symbol table info available. >> #4 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 >> No symbol table info available. >> #5 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 >> No symbol table info available. >> >> Thread 41 (Thread 0x7f1b047e8700 (LWP 105201)): >> #0 0x00007f1b48ae69b3 in select () from /lib64/libc.so.6 >> No symbol table info available. >> #1 0x00007f1b4b2710e9 in DS_Sleep () from >> /usr/lib64/dirsrv/libslapd.so.0 >> No symbol table info available. >> #2 0x00007f1b4b712325 in time_thread () >> No symbol table info available. >> #3 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so >> No symbol table info available. >> #4 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 >> No symbol table info available. >> #5 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 >> No symbol table info available. >> >> Thread 40 (Thread 0x7f1b04fe9700 (LWP 105200)): >> #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 () from >> /lib64/libpthread.so.0 >> No symbol table info available. >> #1 0x00007f1b433212f3 in __db_hybrid_mutex_suspend () from >> /lib64/libdb-5.3.so >> No symbol table info available. >> #2 0x00007f1b43320640 in __db_tas_mutex_lock () from >> /lib64/libdb-5.3.so >> No symbol table info available. >> #3 0x00007f1b4343ae1b in __memp_fget () from /lib64/libdb-5.3.so >> >> No symbol table info available. >> #4 0x00007f1b4333e211 in __bam_search () from /lib64/libdb-5.3.so >> >> No symbol table info available. >> #5 0x00007f1b43329256 in __bamc_search () from /lib64/libdb-5.3.so >> >> No symbol table info available. >> #6 0x00007f1b4332d6c4 in __bamc_put () from /lib64/libdb-5.3.so >> >> No symbol table info available. >> #7 0x00007f1b433e46d5 in __dbc_iput () from /lib64/libdb-5.3.so >> >> No symbol table info available. >> #8 0x00007f1b433dfa0e in __db_put () from /lib64/libdb-5.3.so >> >> No symbol table info available. >> #9 0x00007f1b433f4fa4 in __db_put_pp () from /lib64/libdb-5.3.so >> >> No symbol table info available. >> #10 0x00007f1b3e8cf4cf in idl_new_insert_key () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> No symbol table info available. >> #11 0x00007f1b3e8de5b5 in addordel_values_sv.isra.3 () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> No symbol table info available. >> #12 0x00007f1b3e8df25a in index_addordel_values_ext_sv () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> No symbol table info available. >> #13 0x00007f1b3e8df694 in index_addordel_values_sv () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> No symbol table info available. >> #14 0x00007f1b3e8df75a in index_addordel_entry () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> No symbol table info available. >> #15 0x00007f1b3e8e25fa in ldbm_back_add () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> No symbol table info available. >> #16 0x00007f1b4b1e4cca in op_shared_add () from >> /usr/lib64/dirsrv/libslapd.so.0 >> No symbol table info available. >> #17 0x00007f1b4b1e6020 in do_add () from /usr/lib64/dirsrv/libslapd.so.0 >> No symbol table info available. >> #18 0x00007f1b4b711303 in connection_threadmain () >> No symbol table info available. >> #19 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so >> No symbol table info available. >> #20 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 >> No symbol table info available. >> #21 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 >> No symbol table info available. >> >> Thread 39 (Thread 0x7f1b057ea700 (LWP 105199)): >> #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 () from >> /lib64/libpthread.so.0 >> No symbol table info available. >> #1 0x00007f1b4941b050 in PR_WaitCondVar () from /lib64/libnspr4.so >> No symbol table info available. >> #2 0x00007f1b4b70efee in connection_wait_for_new_work () >> No symbol table info available. >> #3 0x00007f1b4b710719 in connection_threadmain () >> No symbol table info available. >> #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so >> No symbol table info available. >> #5 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 >> No symbol table info available. >> #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 >> No symbol table info available. >> >> Thread 38 (Thread 0x7f1b05feb700 (LWP 105198)): >> #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 () from >> /lib64/libpthread.so.0 >> No symbol table info available. >> #1 0x00007f1b4941b050 in PR_WaitCondVar () from /lib64/libnspr4.so >> No symbol table info available. >> #2 0x00007f1b4b70efee in connection_wait_for_new_work () >> No symbol table info available. >> #3 0x00007f1b4b710719 in connection_threadmain () >> No symbol table info available. >> #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so >> No symbol table info available. >> #5 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 >> No symbol table info available. >> #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 >> No symbol table info available. >> >> Thread 37 (Thread 0x7f1b067ec700 (LWP 105197)): >> #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 () from >> /lib64/libpthread.so.0 >> No symbol table info available. >> #1 0x00007f1b4941b050 in PR_WaitCondVar () from /lib64/libnspr4.so >> No symbol table info available. >> #2 0x00007f1b4b70efee in connection_wait_for_new_work () >> No symbol table info available. >> #3 0x00007f1b4b710719 in connection_threadmain () >> No symbol table info available. >> #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so >> No symbol table info available. >> #5 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 >> No symbol table info available. >> #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 >> No symbol table info available. >> >> Thread 36 (Thread 0x7f1b06fed700 (LWP 105196)): >> #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 () from >> /lib64/libpthread.so.0 >> No symbol table info available. >> #1 0x00007f1b4941b050 in PR_WaitCondVar () from /lib64/libnspr4.so >> No symbol table info available. >> #2 0x00007f1b4b70efee in connection_wait_for_new_work () >> No symbol table info available. >> #3 0x00007f1b4b710719 in connection_threadmain () >> No symbol table info available. >> #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so >> No symbol table info available. >> #5 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 >> No symbol table info available. >> #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 >> No symbol table info available. >> >> Thread 35 (Thread 0x7f1b077ee700 (LWP 105195)): >> #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 () from >> /lib64/libpthread.so.0 >> No symbol table info available. >> #1 0x00007f1b4941b2c3 in PR_EnterMonitor () from /lib64/libnspr4.so >> No symbol table info available. >> #2 0x00007f1b3e8c23e6 in dblayer_txn_begin () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> No symbol table info available. >> #3 0x00007f1b3e8fe007 in ldbm_back_modify () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> No symbol table info available. >> #4 0x00007f1b4b22b541 in op_shared_modify () from >> /usr/lib64/dirsrv/libslapd.so.0 >> No symbol table info available. >> #5 0x00007f1b4b22c004 in modify_internal_pb () from >> /usr/lib64/dirsrv/libslapd.so.0 >> No symbol table info available. >> #6 0x00007f1b3eb48b60 in ipapwd_apply_mods () from >> /usr/lib64/dirsrv/plugins/libipa_pwd_extop.so >> No symbol table info available. >> #7 0x00007f1b3eb4c8e0 in store_new_keys () from >> /usr/lib64/dirsrv/plugins/libipa_pwd_extop.so >> No symbol table info available. >> #8 0x00007f1b3eb4ead0 in ipapwd_extop () from >> /usr/lib64/dirsrv/plugins/libipa_pwd_extop.so >> No symbol table info available. >> #9 0x00007f1b4b23bba2 in plugin_call_exop_plugins () from >> /usr/lib64/dirsrv/libslapd.so.0 >> No symbol table info available. >> #10 0x00007f1b4b717efe in do_extended () >> No symbol table info available. >> #11 0x00007f1b4b71141a in connection_threadmain () >> No symbol table info available. >> #12 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so >> No symbol table info available. >> #13 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 >> No symbol table info available. >> #14 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 >> No symbol table info available. >> >> Thread 34 (Thread 0x7f1b07fef700 (LWP 105194)): >> #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 () from >> /lib64/libpthread.so.0 >> No symbol table info available. >> #1 0x00007f1b4941b050 in PR_WaitCondVar () from /lib64/libnspr4.so >> No symbol table info available. >> #2 0x00007f1b4b70efee in connection_wait_for_new_work () >> No symbol table info available. >> #3 0x00007f1b4b710719 in connection_threadmain () >> No symbol table info available. >> #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so >> No symbol table info available. >> #5 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 >> No symbol table info available. >> #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 >> No symbol table info available. >> >> Thread 33 (Thread 0x7f1b087f0700 (LWP 105193)): >> #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 () from >> /lib64/libpthread.so.0 >> No symbol table info available. >> #1 0x00007f1b4941b050 in PR_WaitCondVar () from /lib64/libnspr4.so >> No symbol table info available. >> #2 0x00007f1b4b70efee in connection_wait_for_new_work () >> No symbol table info available. >> #3 0x00007f1b4b710719 in connection_threadmain () >> No symbol table info available. >> #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so >> No symbol table info available. >> #5 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 >> No symbol table info available. >> #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 >> No symbol table info available. >> >> Thread 32 (Thread 0x7f1b08ff1700 (LWP 105192)): >> #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 () from >> /lib64/libpthread.so.0 >> No symbol table info available. >> #1 0x00007f1b4941b050 in PR_WaitCondVar () from /lib64/libnspr4.so >> No symbol table info available. >> #2 0x00007f1b4b70efee in connection_wait_for_new_work () >> No symbol table info available. >> #3 0x00007f1b4b710719 in connection_threadmain () >> No symbol table info available. >> #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so >> No symbol table info available. >> #5 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 >> No symbol table info available. >> #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 >> No symbol table info available. >> >> Thread 31 (Thread 0x7f1b097f2700 (LWP 105191)): >> #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 () from >> /lib64/libpthread.so.0 >> No symbol table info available. >> #1 0x00007f1b4941b050 in PR_WaitCondVar () from /lib64/libnspr4.so >> No symbol table info available. >> #2 0x00007f1b4b70efee in connection_wait_for_new_work () >> No symbol table info available. >> #3 0x00007f1b4b710719 in connection_threadmain () >> No symbol table info available. >> #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so >> No symbol table info available. >> #5 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 >> No symbol table info available. >> #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 >> No symbol table info available. >> >> Thread 30 (Thread 0x7f1b09ff3700 (LWP 105190)): >> #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 () from >> /lib64/libpthread.so.0 >> No symbol table info available. >> #1 0x00007f1b433212f3 in __db_hybrid_mutex_suspend () from >> /lib64/libdb-5.3.so >> No symbol table info available. >> #2 0x00007f1b43320640 in __db_tas_mutex_lock () from >> /lib64/libdb-5.3.so >> No symbol table info available. >> #3 0x00007f1b433cad3a in __lock_get_internal () from >> /lib64/libdb-5.3.so >> No symbol table info available. >> #4 0x00007f1b433cb820 in __lock_get () from /lib64/libdb-5.3.so >> >> No symbol table info available. >> #5 0x00007f1b433f7142 in __db_lget () from /lib64/libdb-5.3.so >> >> No symbol table info available. >> #6 0x00007f1b4333e605 in __bam_search () from /lib64/libdb-5.3.so >> >> No symbol table info available. >> #7 0x00007f1b43329256 in __bamc_search () from /lib64/libdb-5.3.so >> >> No symbol table info available. >> #8 0x00007f1b4332ad0f in __bamc_get () from /lib64/libdb-5.3.so >> >> No symbol table info available. >> #9 0x00007f1b433e3ca6 in __dbc_iget () from /lib64/libdb-5.3.so >> >> No symbol table info available. >> #10 0x00007f1b433f2b02 in __dbc_get_pp () from /lib64/libdb-5.3.so >> >> No symbol table info available. >> #11 0x00007f1b3e8ce680 in idl_new_fetch () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> No symbol table info available. >> #12 0x00007f1b3e8dcb26 in index_read_ext_allids () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> No symbol table info available. >> #13 0x00007f1b3e8c7284 in keys2idl () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> No symbol table info available. >> #14 0x00007f1b3e8c79e3 in ava_candidates.isra.0 () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> No symbol table info available. >> #15 0x00007f1b3e8c7fd2 in filter_candidates_ext () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> No symbol table info available. >> #16 0x00007f1b3e8c9096 in list_candidates () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> No symbol table info available. >> #17 0x00007f1b3e8c7f40 in filter_candidates_ext () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> No symbol table info available. >> #18 0x00007f1b3e8c9096 in list_candidates () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> No symbol table info available. >> #19 0x00007f1b3e8c7f40 in filter_candidates_ext () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> No symbol table info available. >> #20 0x00007f1b3e8c9096 in list_candidates () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> No symbol table info available. >> #21 0x00007f1b3e8c7f40 in filter_candidates_ext () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> No symbol table info available. >> #22 0x00007f1b3e90380c in subtree_candidates () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> No symbol table info available. >> #23 0x00007f1b3e904e7f in ldbm_back_search () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> No symbol table info available. >> #24 0x00007f1b4b2321d6 in op_shared_search () from >> /usr/lib64/dirsrv/libslapd.so.0 >> No symbol table info available. >> #25 0x00007f1b4b72333e in do_search () >> No symbol table info available. >> #26 0x00007f1b4b7113c3 in connection_threadmain () >> No symbol table info available. >> #27 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so >> No symbol table info available. >> #28 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 >> No symbol table info available. >> #29 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 >> No symbol table info available. >> >> Thread 29 (Thread 0x7f1b0a7f4700 (LWP 105189)): >> #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 () from >> /lib64/libpthread.so.0 >> No symbol table info available. >> #1 0x00007f1b4941b050 in PR_WaitCondVar () from /lib64/libnspr4.so >> No symbol table info available. >> #2 0x00007f1b4b70efee in connection_wait_for_new_work () >> No symbol table info available. >> #3 0x00007f1b4b710719 in connection_threadmain () >> No symbol table info available. >> #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so >> No symbol table info available. >> #5 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 >> No symbol table info available. >> #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 >> No symbol table info available. >> >> Thread 28 (Thread 0x7f1b0aff5700 (LWP 105188)): >> #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 () from >> /lib64/libpthread.so.0 >> No symbol table info available. >> #1 0x00007f1b4941b050 in PR_WaitCondVar () from /lib64/libnspr4.so >> No symbol table info available. >> #2 0x00007f1b4b70efee in connection_wait_for_new_work () >> No symbol table info available. >> #3 0x00007f1b4b710719 in connection_threadmain () >> No symbol table info available. >> #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so >> No symbol table info available. >> #5 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 >> No symbol table info available. >> #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 >> No symbol table info available. >> >> Thread 27 (Thread 0x7f1b0b7f6700 (LWP 105187)): >> #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 () from >> /lib64/libpthread.so.0 >> No symbol table info available. >> #1 0x00007f1b4941b050 in PR_WaitCondVar () from /lib64/libnspr4.so >> No symbol table info available. >> #2 0x00007f1b4b70efee in connection_wait_for_new_work () >> No symbol table info available. >> #3 0x00007f1b4b710719 in connection_threadmain () >> No symbol table info available. >> #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so >> No symbol table info available. >> #5 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 >> No symbol table info available. >> #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 >> No symbol table info available. >> >> Thread 26 (Thread 0x7f1b0bff7700 (LWP 105186)): >> #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 () from >> /lib64/libpthread.so.0 >> No symbol table info available. >> #1 0x00007f1b433212f3 in __db_hybrid_mutex_suspend () from >> /lib64/libdb-5.3.so >> No symbol table info available. >> #2 0x00007f1b43320640 in __db_tas_mutex_lock () from >> /lib64/libdb-5.3.so >> No symbol table info available. >> #3 0x00007f1b433cad3a in __lock_get_internal () from >> /lib64/libdb-5.3.so >> No symbol table info available. >> #4 0x00007f1b433cb820 in __lock_get () from /lib64/libdb-5.3.so >> >> No symbol table info available. >> #5 0x00007f1b433f7142 in __db_lget () from /lib64/libdb-5.3.so >> >> No symbol table info available. >> #6 0x00007f1b4333e605 in __bam_search () from /lib64/libdb-5.3.so >> >> No symbol table info available. >> #7 0x00007f1b43329256 in __bamc_search () from /lib64/libdb-5.3.so >> >> No symbol table info available. >> #8 0x00007f1b4332ad0f in __bamc_get () from /lib64/libdb-5.3.so >> >> No symbol table info available. >> #9 0x00007f1b433e3ca6 in __dbc_iget () from /lib64/libdb-5.3.so >> >> No symbol table info available. >> #10 0x00007f1b433f2b02 in __dbc_get_pp () from /lib64/libdb-5.3.so >> >> No symbol table info available. >> #11 0x00007f1b3e8ce680 in idl_new_fetch () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> No symbol table info available. >> #12 0x00007f1b3e8dcb26 in index_read_ext_allids () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> No symbol table info available. >> #13 0x00007f1b3e8c7284 in keys2idl () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> No symbol table info available. >> #14 0x00007f1b3e8c79e3 in ava_candidates.isra.0 () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> No symbol table info available. >> #15 0x00007f1b3e8c7fd2 in filter_candidates_ext () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> No symbol table info available. >> #16 0x00007f1b3e8c9096 in list_candidates () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> No symbol table info available. >> #17 0x00007f1b3e8c7f40 in filter_candidates_ext () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> No symbol table info available. >> #18 0x00007f1b3e90380c in subtree_candidates () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> No symbol table info available. >> #19 0x00007f1b3e904e7f in ldbm_back_search () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> No symbol table info available. >> #20 0x00007f1b4b2321d6 in op_shared_search () from >> /usr/lib64/dirsrv/libslapd.so.0 >> No symbol table info available. >> #21 0x00007f1b4b24294e in search_internal_callback_pb () from >> /usr/lib64/dirsrv/libslapd.so.0 >> No symbol table info available. >> #22 0x00007f1b4b242be8 in search_internal_pb () from >> /usr/lib64/dirsrv/libslapd.so.0 >> No symbol table info available. >> #23 0x00007f1b4b72149b in ids_sasl_canon_user () >> No symbol table info available. >> #24 0x00007f1b491e193b in _sasl_canon_user () from /lib64/libsasl2.so.3 >> No symbol table info available. >> #25 0x00007f1b491e1c4c in _sasl_canon_user_lookup () from >> /lib64/libsasl2.so.3 >> No symbol table info available. >> #26 0x00007f1b430e9fb9 in gssapi_server_mech_step () from >> /usr/lib64/sasl2/libgssapiv2.so >> No symbol table info available. >> #27 0x00007f1b491edb9b in sasl_server_step () from /lib64/libsasl2.so.3 >> No symbol table info available. >> #28 0x00007f1b4b72283b in ids_sasl_check_bind () >> No symbol table info available. >> #29 0x00007f1b4b70a61a in do_bind () >> No symbol table info available. >> #30 0x00007f1b4b7113fd in connection_threadmain () >> No symbol table info available. >> #31 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so >> No symbol table info available. >> #32 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 >> No symbol table info available. >> #33 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 >> No symbol table info available. >> >> Thread 25 (Thread 0x7f1b0c7f8700 (LWP 105185)): >> #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 () from >> /lib64/libpthread.so.0 >> No symbol table info available. >> #1 0x00007f1b4941b050 in PR_WaitCondVar () from /lib64/libnspr4.so >> No symbol table info available. >> #2 0x00007f1b4b70efee in connection_wait_for_new_work () >> No symbol table info available. >> #3 0x00007f1b4b710719 in connection_threadmain () >> No symbol table info available. >> #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so >> No symbol table info available. >> #5 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 >> No symbol table info available. >> #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 >> No symbol table info available. >> >> Thread 24 (Thread 0x7f1b0cff9700 (LWP 105184)): >> #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 () from >> /lib64/libpthread.so.0 >> No symbol table info available. >> #1 0x00007f1b4941b050 in PR_WaitCondVar () from /lib64/libnspr4.so >> No symbol table info available. >> #2 0x00007f1b4b70efee in connection_wait_for_new_work () >> No symbol table info available. >> #3 0x00007f1b4b710719 in connection_threadmain () >> No symbol table info available. >> #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so >> No symbol table info available. >> #5 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 >> No symbol table info available. >> #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 >> No symbol table info available. >> >> Thread 23 (Thread 0x7f1b0d7fa700 (LWP 105183)): >> #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 () from >> /lib64/libpthread.so.0 >> No symbol table info available. >> #1 0x00007f1b4941b050 in PR_WaitCondVar () from /lib64/libnspr4.so >> No symbol table info available. >> #2 0x00007f1b4b70efee in connection_wait_for_new_work () >> No symbol table info available. >> #3 0x00007f1b4b710719 in connection_threadmain () >> No symbol table info available. >> #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so >> No symbol table info available. >> #5 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 >> No symbol table info available. >> #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 >> No symbol table info available. >> >> Thread 22 (Thread 0x7f1b0dffb700 (LWP 105182)): >> #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 () from >> /lib64/libpthread.so.0 >> No symbol table info available. >> #1 0x00007f1b4941b050 in PR_WaitCondVar () from /lib64/libnspr4.so >> No symbol table info available. >> #2 0x00007f1b4b70efee in connection_wait_for_new_work () >> No symbol table info available. >> #3 0x00007f1b4b710719 in connection_threadmain () >> No symbol table info available. >> #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so >> No symbol table info available. >> #5 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 >> No symbol table info available. >> #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 >> No symbol table info available. >> >> Thread 21 (Thread 0x7f1b0e7fc700 (LWP 105181)): >> #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 () from >> /lib64/libpthread.so.0 >> No symbol table info available. >> #1 0x00007f1b4941b050 in PR_WaitCondVar () from /lib64/libnspr4.so >> No symbol table info available. >> #2 0x00007f1b4b70efee in connection_wait_for_new_work () >> No symbol table info available. >> #3 0x00007f1b4b710719 in connection_threadmain () >> No symbol table info available. >> #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so >> No symbol table info available. >> #5 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 >> No symbol table info available. >> #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 >> No symbol table info available. >> >> Thread 20 (Thread 0x7f1b0effd700 (LWP 105180)): >> #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 () from >> /lib64/libpthread.so.0 >> No symbol table info available. >> #1 0x00007f1b433212f3 in __db_hybrid_mutex_suspend () from >> /lib64/libdb-5.3.so >> No symbol table info available. >> #2 0x00007f1b43320b37 in __db_tas_mutex_readlock () from >> /lib64/libdb-5.3.so >> No symbol table info available. >> #3 0x00007f1b4343ad35 in __memp_fget () from /lib64/libdb-5.3.so >> >> No symbol table info available. >> #4 0x00007f1b433e4333 in __dbc_iget () from /lib64/libdb-5.3.so >> >> No symbol table info available. >> #5 0x00007f1b433f2b02 in __dbc_get_pp () from /lib64/libdb-5.3.so >> >> No symbol table info available. >> #6 0x00007f1b3e8ce822 in idl_new_fetch () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> No symbol table info available. >> #7 0x00007f1b3e8dcb26 in index_read_ext_allids () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> No symbol table info available. >> #8 0x00007f1b3e8c7284 in keys2idl () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> No symbol table info available. >> #9 0x00007f1b3e8c79e3 in ava_candidates.isra.0 () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> No symbol table info available. >> #10 0x00007f1b3e8c7fd2 in filter_candidates_ext () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> No symbol table info available. >> #11 0x00007f1b3e8c9096 in list_candidates () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> No symbol table info available. >> #12 0x00007f1b3e8c7f40 in filter_candidates_ext () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> No symbol table info available. >> #13 0x00007f1b3e8c9096 in list_candidates () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> No symbol table info available. >> #14 0x00007f1b3e8c7f40 in filter_candidates_ext () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> No symbol table info available. >> #15 0x00007f1b3e8c9096 in list_candidates () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> No symbol table info available. >> #16 0x00007f1b3e8c7f40 in filter_candidates_ext () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> No symbol table info available. >> #17 0x00007f1b3e90380c in subtree_candidates () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> No symbol table info available. >> #18 0x00007f1b3e904e7f in ldbm_back_search () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> No symbol table info available. >> #19 0x00007f1b4b2321d6 in op_shared_search () from >> /usr/lib64/dirsrv/libslapd.so.0 >> No symbol table info available. >> #20 0x00007f1b4b72333e in do_search () >> No symbol table info available. >> #21 0x00007f1b4b7113c3 in connection_threadmain () >> No symbol table info available. >> #22 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so >> No symbol table info available. >> #23 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 >> No symbol table info available. >> #24 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 >> No symbol table info available. >> >> Thread 19 (Thread 0x7f1b0f7fe700 (LWP 105179)): >> #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 () from >> /lib64/libpthread.so.0 >> No symbol table info available. >> #1 0x00007f1b433212f3 in __db_hybrid_mutex_suspend () from >> /lib64/libdb-5.3.so >> No symbol table info available. >> #2 0x00007f1b43320640 in __db_tas_mutex_lock () from >> /lib64/libdb-5.3.so >> No symbol table info available. >> #3 0x00007f1b433cad3a in __lock_get_internal () from >> /lib64/libdb-5.3.so >> No symbol table info available. >> #4 0x00007f1b433cb820 in __lock_get () from /lib64/libdb-5.3.so >> >> No symbol table info available. >> #5 0x00007f1b433f7142 in __db_lget () from /lib64/libdb-5.3.so >> >> No symbol table info available. >> #6 0x00007f1b4333e605 in __bam_search () from /lib64/libdb-5.3.so >> >> No symbol table info available. >> #7 0x00007f1b43329256 in __bamc_search () from /lib64/libdb-5.3.so >> >> No symbol table info available. >> #8 0x00007f1b4332ad0f in __bamc_get () from /lib64/libdb-5.3.so >> >> No symbol table info available. >> #9 0x00007f1b433e3ca6 in __dbc_iget () from /lib64/libdb-5.3.so >> >> No symbol table info available. >> #10 0x00007f1b433f2b02 in __dbc_get_pp () from /lib64/libdb-5.3.so >> >> No symbol table info available. >> #11 0x00007f1b3e8ce680 in idl_new_fetch () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> No symbol table info available. >> #12 0x00007f1b3e8dcb26 in index_read_ext_allids () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> No symbol table info available. >> #13 0x00007f1b3e8c7284 in keys2idl () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> No symbol table info available. >> #14 0x00007f1b3e8c79e3 in ava_candidates.isra.0 () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> No symbol table info available. >> #15 0x00007f1b3e8c7fd2 in filter_candidates_ext () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> No symbol table info available. >> #16 0x00007f1b3e8c9096 in list_candidates () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> No symbol table info available. >> #17 0x00007f1b3e8c7f40 in filter_candidates_ext () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> No symbol table info available. >> #18 0x00007f1b3e8c9096 in list_candidates () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> No symbol table info available. >> #19 0x00007f1b3e8c7f40 in filter_candidates_ext () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> No symbol table info available. >> #20 0x00007f1b3e8c9096 in list_candidates () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> No symbol table info available. >> #21 0x00007f1b3e8c7f40 in filter_candidates_ext () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> No symbol table info available. >> #22 0x00007f1b3e90380c in subtree_candidates () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> No symbol table info available. >> #23 0x00007f1b3e904e7f in ldbm_back_search () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> No symbol table info available. >> #24 0x00007f1b4b2321d6 in op_shared_search () from >> /usr/lib64/dirsrv/libslapd.so.0 >> No symbol table info available. >> #25 0x00007f1b4b72333e in do_search () >> No symbol table info available. >> #26 0x00007f1b4b7113c3 in connection_threadmain () >> No symbol table info available. >> #27 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so >> No symbol table info available. >> #28 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 >> No symbol table info available. >> #29 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 >> No symbol table info available. >> >> Thread 18 (Thread 0x7f1b0ffff700 (LWP 105178)): >> #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 () from >> /lib64/libpthread.so.0 >> No symbol table info available. >> #1 0x00007f1b4941b050 in PR_WaitCondVar () from /lib64/libnspr4.so >> No symbol table info available. >> #2 0x00007f1b4b70efee in connection_wait_for_new_work () >> No symbol table info available. >> #3 0x00007f1b4b710719 in connection_threadmain () >> No symbol table info available. >> #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so >> No symbol table info available. >> #5 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 >> No symbol table info available. >> #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 >> No symbol table info available. >> >> Thread 17 (Thread 0x7f1b14d28700 (LWP 105177)): >> #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 () from >> /lib64/libpthread.so.0 >> No symbol table info available. >> #1 0x00007f1b4941b050 in PR_WaitCondVar () from /lib64/libnspr4.so >> No symbol table info available. >> #2 0x00007f1b4b70efee in connection_wait_for_new_work () >> No symbol table info available. >> #3 0x00007f1b4b710719 in connection_threadmain () >> No symbol table info available. >> #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so >> No symbol table info available. >> #5 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 >> No symbol table info available. >> #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 >> No symbol table info available. >> >> Thread 16 (Thread 0x7f1b15529700 (LWP 105176)): >> #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 () from >> /lib64/libpthread.so.0 >> No symbol table info available. >> #1 0x00007f1b4941b050 in PR_WaitCondVar () from /lib64/libnspr4.so >> No symbol table info available. >> #2 0x00007f1b4b70efee in connection_wait_for_new_work () >> No symbol table info available. >> #3 0x00007f1b4b710719 in connection_threadmain () >> No symbol table info available. >> #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so >> No symbol table info available. >> #5 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 >> No symbol table info available. >> #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 >> No symbol table info available. >> >> Thread 15 (Thread 0x7f1b15d2a700 (LWP 105175)): >> #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 () from >> /lib64/libpthread.so.0 >> No symbol table info available. >> #1 0x00007f1b4941b050 in PR_WaitCondVar () from /lib64/libnspr4.so >> No symbol table info available. >> #2 0x00007f1b4b70efee in connection_wait_for_new_work () >> No symbol table info available. >> #3 0x00007f1b4b710719 in connection_threadmain () >> No symbol table info available. >> #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so >> No symbol table info available. >> #5 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 >> No symbol table info available. >> #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 >> No symbol table info available. >> >> Thread 14 (Thread 0x7f1b1652b700 (LWP 105174)): >> #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 () from >> /lib64/libpthread.so.0 >> No symbol table info available. >> #1 0x00007f1b4941b050 in PR_WaitCondVar () from /lib64/libnspr4.so >> No symbol table info available. >> #2 0x00007f1b4b70efee in connection_wait_for_new_work () >> No symbol table info available. >> #3 0x00007f1b4b710719 in connection_threadmain () >> No symbol table info available. >> #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so >> No symbol table info available. >> #5 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 >> No symbol table info available. >> #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 >> No symbol table info available. >> >> Thread 13 (Thread 0x7f1b16d2c700 (LWP 105173)): >> #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 () from >> /lib64/libpthread.so.0 >> No symbol table info available. >> #1 0x00007f1b433212f3 in __db_hybrid_mutex_suspend () from >> /lib64/libdb-5.3.so >> No symbol table info available. >> #2 0x00007f1b43320640 in __db_tas_mutex_lock () from >> /lib64/libdb-5.3.so >> No symbol table info available. >> #3 0x00007f1b433cad3a in __lock_get_internal () from >> /lib64/libdb-5.3.so >> No symbol table info available. >> #4 0x00007f1b433cb820 in __lock_get () from /lib64/libdb-5.3.so >> >> No symbol table info available. >> #5 0x00007f1b433f7142 in __db_lget () from /lib64/libdb-5.3.so >> >> No symbol table info available. >> #6 0x00007f1b4333e605 in __bam_search () from /lib64/libdb-5.3.so >> >> No symbol table info available. >> #7 0x00007f1b43329256 in __bamc_search () from /lib64/libdb-5.3.so >> >> No symbol table info available. >> #8 0x00007f1b4332ad0f in __bamc_get () from /lib64/libdb-5.3.so >> >> No symbol table info available. >> #9 0x00007f1b433e3ca6 in __dbc_iget () from /lib64/libdb-5.3.so >> >> No symbol table info available. >> #10 0x00007f1b433f2b02 in __dbc_get_pp () from /lib64/libdb-5.3.so >> >> No symbol table info available. >> #11 0x00007f1b3e8ce680 in idl_new_fetch () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> No symbol table info available. >> #12 0x00007f1b3e8dcb26 in index_read_ext_allids () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> No symbol table info available. >> #13 0x00007f1b3e8c7284 in keys2idl () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> No symbol table info available. >> #14 0x00007f1b3e8c79e3 in ava_candidates.isra.0 () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> No symbol table info available. >> #15 0x00007f1b3e8c7fd2 in filter_candidates_ext () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> No symbol table info available. >> #16 0x00007f1b3e8c9096 in list_candidates () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> No symbol table info available. >> #17 0x00007f1b3e8c7f40 in filter_candidates_ext () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> No symbol table info available. >> #18 0x00007f1b3e8c9096 in list_candidates () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> No symbol table info available. >> #19 0x00007f1b3e8c7f40 in filter_candidates_ext () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> No symbol table info available. >> #20 0x00007f1b3e8c9096 in list_candidates () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> No symbol table info available. >> #21 0x00007f1b3e8c7f40 in filter_candidates_ext () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> No symbol table info available. >> #22 0x00007f1b3e90380c in subtree_candidates () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> No symbol table info available. >> #23 0x00007f1b3e904e7f in ldbm_back_search () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> No symbol table info available. >> #24 0x00007f1b4b2321d6 in op_shared_search () from >> /usr/lib64/dirsrv/libslapd.so.0 >> No symbol table info available. >> #25 0x00007f1b4b72333e in do_search () >> No symbol table info available. >> #26 0x00007f1b4b7113c3 in connection_threadmain () >> No symbol table info available. >> #27 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so >> No symbol table info available. >> #28 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 >> No symbol table info available. >> #29 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 >> No symbol table info available. >> >> Thread 12 (Thread 0x7f1b1752d700 (LWP 105172)): >> #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 () from >> /lib64/libpthread.so.0 >> No symbol table info available. >> #1 0x00007f1b4941b050 in PR_WaitCondVar () from /lib64/libnspr4.so >> No symbol table info available. >> #2 0x00007f1b4b70efee in connection_wait_for_new_work () >> No symbol table info available. >> #3 0x00007f1b4b710719 in connection_threadmain () >> No symbol table info available. >> #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so >> No symbol table info available. >> #5 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 >> No symbol table info available. >> #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 >> No symbol table info available. >> >> Thread 11 (Thread 0x7f1b17d2e700 (LWP 105171)): >> #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 () from >> /lib64/libpthread.so.0 >> No symbol table info available. >> #1 0x00007f1b4941b050 in PR_WaitCondVar () from /lib64/libnspr4.so >> No symbol table info available. >> #2 0x00007f1b4b70efee in connection_wait_for_new_work () >> No symbol table info available. >> #3 0x00007f1b4b710719 in connection_threadmain () >> No symbol table info available. >> #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so >> No symbol table info available. >> #5 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 >> No symbol table info available. >> #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 >> No symbol table info available. >> >> Thread 10 (Thread 0x7f1b2509d700 (LWP 105168)): >> #0 0x00007f1b48dc5a82 in pthread_cond_timedwait@@GLIBC_2.3.2 () from >> /lib64/libpthread.so.0 >> No symbol table info available. >> #1 0x00007f1b4941ab07 in pt_TimedWait () from /lib64/libnspr4.so >> No symbol table info available. >> #2 0x00007f1b4941afce in PR_WaitCondVar () from /lib64/libnspr4.so >> No symbol table info available. >> #3 0x00007f1b4b2061e8 in eq_loop () from /usr/lib64/dirsrv/libslapd.so.0 >> No symbol table info available. >> #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so >> No symbol table info available. >> #5 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 >> No symbol table info available. >> #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 >> No symbol table info available. >> >> Thread 9 (Thread 0x7f1b2589e700 (LWP 105167)): >> #0 0x00007f1b48dc5a82 in pthread_cond_timedwait@@GLIBC_2.3.2 () from >> /lib64/libpthread.so.0 >> No symbol table info available. >> #1 0x00007f1b4941ab07 in pt_TimedWait () from /lib64/libnspr4.so >> No symbol table info available. >> #2 0x00007f1b4941afce in PR_WaitCondVar () from /lib64/libnspr4.so >> No symbol table info available. >> #3 0x00007f1b4b719d03 in housecleaning () >> No symbol table info available. >> #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so >> No symbol table info available. >> #5 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 >> No symbol table info available. >> #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 >> No symbol table info available. >> >> Thread 8 (Thread 0x7f1b2609f700 (LWP 105166)): >> #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 () from >> /lib64/libpthread.so.0 >> No symbol table info available. >> #1 0x00007f1b4941b050 in PR_WaitCondVar () from /lib64/libnspr4.so >> No symbol table info available. >> #2 0x00007f1b4b260198 in slapi_wait_condvar () from >> /usr/lib64/dirsrv/libslapd.so.0 >> No symbol table info available. >> #3 0x00007f1b3cf5deed in roles_cache_wait_on_change () from >> /usr/lib64/dirsrv/plugins/libroles-plugin.so >> No symbol table info available. >> #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so >> No symbol table info available. >> #5 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 >> No symbol table info available. >> #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 >> No symbol table info available. >> >> Thread 7 (Thread 0x7f1b268a0700 (LWP 105165)): >> #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 () from >> /lib64/libpthread.so.0 >> No symbol table info available. >> #1 0x00007f1b4941b050 in PR_WaitCondVar () from /lib64/libnspr4.so >> No symbol table info available. >> #2 0x00007f1b4b260198 in slapi_wait_condvar () from >> /usr/lib64/dirsrv/libslapd.so.0 >> No symbol table info available. >> #3 0x00007f1b3cf5deed in roles_cache_wait_on_change () from >> /usr/lib64/dirsrv/plugins/libroles-plugin.so >> No symbol table info available. >> #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so >> No symbol table info available. >> #5 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 >> No symbol table info available. >> #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 >> No symbol table info available. >> >> Thread 6 (Thread 0x7f1b270a1700 (LWP 105164)): >> #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 () from >> /lib64/libpthread.so.0 >> No symbol table info available. >> #1 0x00007f1b4941b050 in PR_WaitCondVar () from /lib64/libnspr4.so >> No symbol table info available. >> #2 0x00007f1b4b260198 in slapi_wait_condvar () from >> /usr/lib64/dirsrv/libslapd.so.0 >> No symbol table info available. >> #3 0x00007f1b3cf5deed in roles_cache_wait_on_change () from >> /usr/lib64/dirsrv/plugins/libroles-plugin.so >> No symbol table info available. >> #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so >> No symbol table info available. >> #5 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 >> No symbol table info available. >> #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 >> No symbol table info available. >> >> Thread 5 (Thread 0x7f1b278a2700 (LWP 105163)): >> #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 () from >> /lib64/libpthread.so.0 >> No symbol table info available. >> #1 0x00007f1b4941b050 in PR_WaitCondVar () from /lib64/libnspr4.so >> No symbol table info available. >> #2 0x00007f1b4b260198 in slapi_wait_condvar () from >> /usr/lib64/dirsrv/libslapd.so.0 >> No symbol table info available. >> #3 0x00007f1b415f262e in cos_cache_wait_on_change () from >> /usr/lib64/dirsrv/plugins/libcos-plugin.so >> No symbol table info available. >> #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so >> No symbol table info available. >> #5 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 >> No symbol table info available. >> #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 >> No symbol table info available. >> >> Thread 4 (Thread 0x7f1b282ac700 (LWP 105162)): >> #0 0x00007f1b48ae69b3 in select () from /lib64/libc.so.6 >> No symbol table info available. >> #1 0x00007f1b4b2710e9 in DS_Sleep () from >> /usr/lib64/dirsrv/libslapd.so.0 >> No symbol table info available. >> #2 0x00007f1b3e8ba707 in perf_threadmain () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> No symbol table info available. >> #3 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so >> No symbol table info available. >> #4 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 >> No symbol table info available. >> #5 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 >> No symbol table info available. >> >> Thread 3 (Thread 0x7f1b28aad700 (LWP 105161)): >> #0 0x00007f1b48ae69b3 in select () from /lib64/libc.so.6 >> No symbol table info available. >> #1 0x00007f1b4b2710e9 in DS_Sleep () from >> /usr/lib64/dirsrv/libslapd.so.0 >> No symbol table info available. >> #2 0x00007f1b3e8bfb7f in trickle_threadmain () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> No symbol table info available. >> #3 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so >> No symbol table info available. >> #4 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 >> No symbol table info available. >> #5 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 >> No symbol table info available. >> >> Thread 2 (Thread 0x7f1b292ae700 (LWP 105160)): >> #0 0x00007f1b48ae69b3 in select () from /lib64/libc.so.6 >> No symbol table info available. >> #1 0x00007f1b4344dbad in __os_yield () from /lib64/libdb-5.3.so >> >> No symbol table info available. >> #2 0x00007f1b434492b3 in __memp_sync_int () from /lib64/libdb-5.3.so >> >> No symbol table info available. >> #3 0x00007f1b43459752 in __txn_checkpoint () from >> /lib64/libdb-5.3.so >> No symbol table info available. >> #4 0x00007f1b43459b74 in __txn_checkpoint_pp () from >> /lib64/libdb-5.3.so >> No symbol table info available. >> #5 0x00007f1b3e8c3a87 in checkpoint_threadmain () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> No symbol table info available. >> #6 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so >> No symbol table info available. >> #7 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 >> No symbol table info available. >> #8 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 >> No symbol table info available. >> >> Thread 1 (Thread 0x7f1b29aaf700 (LWP 105159)): >> #0 0x00007f1b48ae69b3 in select () from /lib64/libc.so.6 >> No symbol table info available. >> #1 0x00007f1b4b2710e9 in DS_Sleep () from >> /usr/lib64/dirsrv/libslapd.so.0 >> No symbol table info available. >> #2 0x00007f1b3e8bf907 in deadlock_threadmain () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> No symbol table info available. >> #3 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so >> No symbol table info available. >> #4 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 >> No symbol table info available. >> #5 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 >> No symbol table info available. >> >> >> >> Thanks, >> >> Rakesh >> >> >> On Mon, Sep 5, 2016 at 4:48 PM, thierry bordaz > > wrote: >> >> >> >> On 09/05/2016 12:05 PM, Rakesh Rajasekharan wrote: >>> Hi Thierry, >>> >>> >>> I was getting the hang issue while running ipa-client-install >>> simultaneously on few clients.. >>> However, today, I am not able to replicate that. >>> >>> I could not get a gdb . But i will try getting that the next >>> time I face this issue. >>> >>> The CPU does not stay high.. it just momentarily touches a high >>> value and then drops down to around 2-7% >>> >>> One question I have is , is it ok to set it nsslapd-threadnumber >>> to a very high value . >>> I have around 4000 clients and with nsslapd-maxthreadsperconn >>> set to 5..So, can I set nsslapd-threadnumber to around 25000. >> >> Hello, >> >> I know some users running in production with several hunderds of >> threads (>600) and this without problem. >> >> I do not recall having suggested to increase that number and for >> what reason. >> Usually 30 workers is a good enough value. It can create >> bootleneck if for some reason each operation is very long to >> satisfy and exhaust the number of workers. You can monitor the >> work queue: >> >> ldapsearch -D "cn=directory manager" -w xxx -LLL -b >> "cn=monitor" -s base opsinitiated opscompleted >> >> >> If opscompleted-opsinitiated remains close to threadnumber, then >> yes it would be valuable to increase it. >> >> The computation #client * #async_op_per_client sound an overkill. >> Even if all clients send at the exact same time all their >> requests, it is very likely that some common resource (db page, >> log, allocator...) will serialize them. If you monitor a need to >> increase the work, you would for example set it to 50, then >> monitor, then set it to 100, then monitor... until you find a >> good enough value. >> Note the increasing the #thread, increases the memory footprint >> that will reduce the efficiency of file system cache and can >> increase the response time. >> >> >> best regards >> thierry >> >>> >>> Thanks >>> >>> On Mon, Sep 5, 2016 at 1:03 PM, thierry bordaz >>> wrote: >>> >>> >>> Hi Rakesh, >>> >>> Were you able to get a pstack or full stack with gdb >>> (http://www.port389.org/docs/389ds/FAQ/faq.html#debugging-crashes) >>> when the server hangs ? >>> >>> If it happens with 500 threads as well as with 30, using 30 >>> threads is a better choice to debug this issue. >>> I will try to reproduce using 150 parallel 'ipa user-find >>> p-testipa' commands >>> >>> Something I am unsure is if the CPU consumption stays high >>> (you mentioned 340% CPU usage) as long as the hang happens >>> or if after a suddent shot up to 340% (that marks the >>> beginning of the hang) it drops and stay hanging ? >>> >>> thanks >>> thierry >>> >>> On 09/04/2016 08:40 PM, Rakesh Rajasekharan wrote: >>>> starce on the slapd process actually had this in the output.. >>>> FUTEX_WAIT_PRIVATE >>>> >>>> and checking for the number of threads slapd had.. there >>>> were 5015 threads >>>> >>>> ps -efL|grep slapd|wc -l >>>> 5015 >>>> >>>> strace on most of the threads gave this output >>>> >>>> strace -p 67411 >>>> Process 67411 attached >>>> futex(0x7f3f0226b41c, FUTEX_WAIT_PRIVATE, 1, NULL) = -1 >>>> EAGAIN (Resource temporarily unavailable) >>>> futex(0x7f3f0226b41c, FUTEX_WAIT_PRIVATE, 2, NULL^CProcess >>>> 67411 detached >>>> >>>> >>>> >>>> >>>> >>>> On Sun, Sep 4, 2016 at 5:34 PM, Rakesh Rajasekharan >>>> >>> > wrote: >>>> >>>> I have again got the issue of IPA hanging.. The issue >>>> came up when i tried to run ipa-client-isntall on 142 >>>> clients simultaneously >>>> >>>> >>>> None of the IPA commands are responding, and I see this >>>> error >>>> >>>> ipa user-find p-testipa >>>> ipa: ERROR: Insufficient access: SASL(-1): generic >>>> failure: GSSAPI Error: Unspecified GSS failure. Minor >>>> code may provide more information (KDC returned error >>>> string: PROCESS_TGS) >>>> >>>> KRB5_TRACE=/dev/stdout kinit admin >>>> [41178] 1472984115.233214: Getting initial credentials >>>> for admin at XYZ.COM >>>> [41178] 1472984115.235257: Sending request (167 bytes) >>>> to XYZ.COM >>>> [41178] 1472984115.235419: Initiating TCP connection to >>>> stream 10.1.3.36:88 >>>> [41178] 1472984115.235685: Sending TCP request to >>>> stream 10.1.3.36:88 >>>> [41178] 1472984120.238914: Received answer (174 bytes) >>>> from stream 10.1.3.36:88 >>>> [41178] 1472984120.238925: Terminating TCP connection >>>> to stream 10.1.3.36:88 >>>> [41178] 1472984120.238993: Response was from master KDC >>>> [41 >>>> >>>> >>>> Running an ldapsearch to see the db.. does not give any >>>> results and just hangs there >>>> >>>> ldapsearch -x -D 'cn=Directory Manager' -W -s one -b >>>> 'cn=kerberos,dc=xyz,dc=com' >>>> Enter LDAP Password: >>>> >>>> even an ldapsearch -x does not respond >>>> At this point, am sure that slapd is the one causing issues >>>> >>>> Running an strace against the hung slapd itself seems >>>> to get stuck does not proceed after saying "attaching >>>> to process" >>>> >>>> From some others posts I read Thierry suggesting to >>>> increase the nsslapd-threadnumber value >>>> >>>> It was set to 30, I think that might be too low. >>>> >>>> I have raised it to 500 >>>> >>>> Now after restarting the service .. ldapsearch starts >>>> responding. >>>> But running the test to add a sudden high number of >>>> clients again left ns-slapd to hung state >>>> >>>> When i attempted adding the clients.. the ns-slapd cpu >>>> usage shot up to 340% and after that ns-slapd stopped >>>> responding >>>> >>>> So now, atleast I know what might be causing the issue >>>> and I can now easily reproduce it. >>>> >>>> Is there a way I can make ns-slapd handle a sudden bump >>>> in incoming request for ipa-client-install >>>> >>>> Thanks >>>> Rakesh >>>> >>>> >>>> >>>> >>>> >>>> >>>> On Mon, Aug 29, 2016 at 11:18 PM, Rich Megginson >>>> wrote: >>>> >>>> On 08/29/2016 10:53 AM, Rakesh Rajasekharan wrote: >>>>> Hi Thierry, >>>>> >>>>> My machine has 30GB RAM ..and 389-ds version is 1.3.4 >>>>> >>>>> ldapsearch shows the values for >>>>> nsslapd-cachememsize updated to 200MB. >>>>> >>>>> ldapsearch -LLL -o ldif-wrap=no -D "cn=directory >>>>> manager" -w 'mypassword' -b 'cn=userRoot,cn=ldbm >>>>> database,cn=plugins,cn=config'|grep >>>>> nsslapd-cachememsize >>>>> nsslapd-cachememsize: 209715200 >>>>> >>>>> >>>>> So, it seems to have updated though seeing that >>>>> warning(WARNING: ipaca: entry cache size 10485760B >>>>> is less than db size 11599872B) in the log >>>>> confuses me a bit. >>>>> >>>>> Thers one more entry that I found from the >>>>> ldapsearch to be bit low >>>>> >>>>> nsslapd-dncachememsize: 10485760 >>>>> maxdncachesize: 10485760 >>>>> >>>>> Should I update these as well to a higher value >>>>> >>>>> At the time when the issue happened, the memory >>>>> usage as well as the overall load of the system >>>>> was very low . >>>>> I will try reproducing the issue atleast in my QA >>>>> env..probably by trying to mock simultaneous >>>>> parallel logins to a large number of hosts >>>> >>>> To monitor your cache sizes, please use the >>>> dbmon.sh tool provided with your distro. If that >>>> is not available with your particular distro, see >>>> https://github.com/richm/scripts/wiki/dbmon.sh >>>> >>>> >>>>> >>>>> >>>>> thanks >>>>> Rakesh >>>>> >>>>> >>>>> >>>>> >>>>> On Mon, Aug 29, 2016 at 8:16 PM, thierry bordaz >>>>> wrote: >>>>> >>>>> Hi Rakesh, >>>>> >>>>> Those tuning may depend on the memory >>>>> available on your machine. >>>>> nsslapd-cachememsize allows the entry cache to >>>>> consume up to 200Mb but its memory footprint >>>>> is known to go above. >>>>> 200Mb both looks pretty good to me. How large >>>>> is your machine ? What is your version of 389-ds ? >>>>> >>>>> Those warnings do not change your settings. It >>>>> just raise that entry cache of 'ipaca' and >>>>> 'retrocl' are small but it is fine. The size >>>>> of the entry cache is important mostly in >>>>> userRoot. >>>>> You may double check the actual values, after >>>>> restart, with ldapsearch on >>>>> 'cn=userRoot,cn=ldbm >>>>> database,cn=plugins,cn=config' and >>>>> 'cn=config,cn=ldbm database,cn=plugins,cn=config'. >>>>> >>>>> A step is to know what will be response time >>>>> of DS to know if it is responsible of the hang >>>>> or not. >>>>> The logs and possibly pstack during those >>>>> intermittent hangs will help to determine that. >>>>> >>>>> regards >>>>> thierry >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> On 08/29/2016 04:25 PM, Rakesh Rajasekharan wrote: >>>>>> I tried increasing the nsslapd-dbcachesize >>>>>> and nsslapd-cachememsize in my QA envs to 200MB. >>>>>> >>>>>> However, in my log files, I still see this >>>>>> message >>>>>> [29/Aug/2016:04:34:37 +0000] - WARNING: >>>>>> ipaca: entry cache size 10485760B is less >>>>>> than db size 11599872B; We recommend to >>>>>> increase the entry cache size >>>>>> nsslapd-cachememsize. >>>>>> [29/Aug/2016:04:34:37 +0000] - WARNING: >>>>>> changelog: entry cache size 2097152B is less >>>>>> than db size 441647104B; We recommend to >>>>>> increase the entry cache size >>>>>> nsslapd-cachememsize. >>>>>> >>>>>> these are my ldif files that i used to modify >>>>>> the values >>>>>> modify entry cache size >>>>>> cat modify-cache-mem-size.ldif >>>>>> dn: cn=userRoot,cn=ldbm >>>>>> database,cn=plugins,cn=config >>>>>> changetype: modify >>>>>> replace: nsslapd-cachememsize >>>>>> nsslapd-cachememsize: 209715200 >>>>>> >>>>>> modify db cache size >>>>>> cat modfy-db-cache-size.ldif >>>>>> dn: cn=config,cn=ldbm >>>>>> database,cn=plugins,cn=config >>>>>> changetype: modify >>>>>> replace: nsslapd-dbcachesize >>>>>> nsslapd-dbcachesize: 209715200 >>>>>> >>>>>> After modifying , i restarted IPA services >>>>>> >>>>>> Is there anything else that I need to take >>>>>> care of as the logs suggest its still not >>>>>> getting the updated values >>>>>> >>>>>> Thanks >>>>>> Rakesh >>>>>> >>>>>> On Mon, Aug 29, 2016 at 6:07 PM, Rakesh >>>>>> Rajasekharan >>>>>> wrote: >>>>>> >>>>>> Hi Thierry, >>>>>> >>>>>> Coz of the issues we had to revert back >>>>>> to earlier running openldap in production. >>>>>> >>>>>> I have now done a few TCP related changes >>>>>> in sysctl.conf and have also increased >>>>>> the nsslapd-dbcachesize and >>>>>> nsslapd-cachememsize to 200MB >>>>>> >>>>>> I will again start migrating hosts back >>>>>> to IPA and see if I face the earlier issue. >>>>>> >>>>>> I will update back once I have something >>>>>> >>>>>> >>>>>> Thanks, >>>>>> Rakesh >>>>>> >>>>>> >>>>>> >>>>>> On Thu, Aug 25, 2016 at 2:17 PM, thierry >>>>>> bordaz wrote: >>>>>> >>>>>> >>>>>> >>>>>> On 08/25/2016 10:15 AM, Rakesh >>>>>> Rajasekharan wrote: >>>>>>> All of the troubleshooting seems fine. >>>>>>> >>>>>>> >>>>>>> However, Running libconv.pl >>>>>>> gives me this output >>>>>>> >>>>>>> ----- Recommendations ----- >>>>>>> >>>>>>> 1. You have unindexed components, >>>>>>> this can be caused from a search on >>>>>>> an unindexed attribute, or your >>>>>>> returned results exceeded the >>>>>>> allidsthreshold. Unindexed >>>>>>> components are not recommended. To >>>>>>> refuse unindexed searches, switch >>>>>>> 'nsslapd-require-index' to 'on' >>>>>>> under your database entry (e.g. >>>>>>> cn=UserRoot,cn=ldbm >>>>>>> database,cn=plugins,cn=config). >>>>>>> >>>>>>> 2. You have a significant >>>>>>> difference between binds and >>>>>>> unbinds. You may want to investigate >>>>>>> this difference. >>>>>>> >>>>>>> >>>>>>> I feel, this could be a pointer to >>>>>>> things going slow.. and IPA hanging. >>>>>>> I think i now have something that I >>>>>>> can try and nail down this issue. >>>>>>> >>>>>>> On a sidenote, I was earlier running >>>>>>> openldap and migrated over to Freeipa, >>>>>>> >>>>>>> Thanks >>>>>>> Rakesh >>>>>>> >>>>>>> >>>>>>> >>>>>>> On Wed, Aug 24, 2016 at 12:38 PM, >>>>>>> Petr Spacek wrote: >>>>>>> >>>>>>> On 23.8.2016 18:44, Rakesh >>>>>>> Rajasekharan wrote: >>>>>>> > I think thers something >>>>>>> seriously wrong with my system >>>>>>> > >>>>>>> > not able to run any IPA commands >>>>>>> > >>>>>>> > klist >>>>>>> > Ticket cache: >>>>>>> KEYRING:persistent:0:0 >>>>>>> > Default principal: admin at XYZ.COM >>>>>>> > >>>>>>> > Valid starting Expires >>>>>>> Service principal >>>>>>> > 2016-08-23T16:26:36 >>>>>>> 2016-08-24T16:26:22 >>>>>>> krbtgt/XYZ.COM at XYZ.COM >>>>>>> > >>>>>>> > >>>>>>> > [root at prod-ipa-master-1a :~] >>>>>>> ipactl status >>>>>>> > Directory Service: RUNNING >>>>>>> > krb5kdc Service: RUNNING >>>>>>> > kadmin Service: RUNNING >>>>>>> > ipa_memcached Service: RUNNING >>>>>>> > httpd Service: RUNNING >>>>>>> > pki-tomcatd Service: RUNNING >>>>>>> > ipa-otpd Service: RUNNING >>>>>>> > ipa: INFO: The ipactl command >>>>>>> was successful >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > [root at prod-ipa-master :~] ipa >>>>>>> user-find p-testuser >>>>>>> > ipa: ERROR: Kerberos error: >>>>>>> ('Unspecified GSS failure. Minor >>>>>>> code may >>>>>>> > provide more information', >>>>>>> 851968)/("Cannot contact any KDC >>>>>>> for realm ' >>>>>>> > XYZ.COM '", >>>>>>> -1765328228) >>>>>>> >>>>>> >>>>>> Hi Rakesh, >>>>>> >>>>>> Having a reproducible test case >>>>>> would you rerun the command above. >>>>>> During its processing you may >>>>>> monitor DS process load (top). If >>>>>> it is high, you may get some >>>>>> pstacks of it. >>>>>> Also would you attach the part of >>>>>> DS access logs taken during the >>>>>> command. >>>>>> >>>>>> regards >>>>>> thierry >>>>>> >>>>>>> > >>>>>>> >>>>>>> This is weird because the server >>>>>>> seems to be up. >>>>>>> >>>>>>> Please follow >>>>>>> http://www.freeipa.org/page/Troubleshooting#Authentication.2FKerberos >>>>>>> >>>>>>> Petr^2 Spacek >>>>>>> >>>>>>> > >>>>>>> > >>>>>>> > Thanks >>>>>>> > >>>>>>> > Rakesh >>>>>>> > >>>>>>> > On Tue, Aug 23, 2016 at 10:01 >>>>>>> PM, Rakesh Rajasekharan < >>>>>>> > rakesh.rajasekharan at gmail.com> >>>>>>> wrote: >>>>>>> > >>>>>>> >> i changed the loggin level to >>>>>>> 4 . Modifying >>>>>>> nsslapd-accesslog-level >>>>>>> >> >>>>>>> >> But, the hang is still there. >>>>>>> though I dont see the sigfault now >>>>>>> >> >>>>>>> >> >>>>>>> >> >>>>>>> >> >>>>>>> >> On Tue, Aug 23, 2016 at 9:02 >>>>>>> PM, Rakesh Rajasekharan < >>>>>>> >> >>>>>>> rakesh.rajasekharan at gmail.com> >>>>>>> wrote: >>>>>>> >> >>>>>>> >>> My disk was getting filled >>>>>>> too fast >>>>>>> >>> >>>>>>> >>> logs under /var/log/dirsrv >>>>>>> was coming around 5 gb quickly >>>>>>> filling up >>>>>>> >>> >>>>>>> >>> Is there a way to make the >>>>>>> logging less verbose >>>>>>> >>> >>>>>>> >>> >>>>>>> >>> >>>>>>> >>> On Tue, Aug 23, 2016 at 6:41 >>>>>>> PM, Petr Spacek >>>>>>> wrote: >>>>>>> >>> >>>>>>> >>>> On 23.8.2016 15:07, Rakesh >>>>>>> Rajasekharan wrote: >>>>>>> >>>>> I was able to fix that may >>>>>>> be temporarily... when i checked the >>>>>>> >>>> network.. >>>>>>> >>>>> there was another process >>>>>>> that was running and consuming a >>>>>>> lot of >>>>>>> >>>> network ( >>>>>>> >>>>> i have no idea who did >>>>>>> that. I need to seriously start >>>>>>> restricting >>>>>>> >>>> people >>>>>>> >>>>> access to this machine ) >>>>>>> >>>>> >>>>>>> >>>>> after killing that >>>>>>> perfomance improved drastically >>>>>>> >>>>> >>>>>>> >>>>> But now, suddenly I >>>>>>> started experiencing the same hang. >>>>>>> >>>>> >>>>>>> >>>>> This time , I gert the >>>>>>> following error when checked dmesg >>>>>>> >>>>> >>>>>>> >>>>> [ 301.236976] >>>>>>> ns-slapd[3124]: segfault at 0 ip >>>>>>> 00007f1de416951c sp >>>>>>> >>>>> 00007f1dee1dba70 error 4 >>>>>>> in >>>>>>> libcos-plugin.so[7f1de4166000+b000] >>>>>>> >>>>> [ 1116.248431] TCP: >>>>>>> request_sock_TCP: Possible SYN >>>>>>> flooding on port 88. >>>>>>> >>>>> Sending cookies. Check >>>>>>> SNMP counters. >>>>>>> >>>>> [11831.397037] >>>>>>> ns-slapd[22550]: segfault at 0 >>>>>>> ip 00007f533d82251c sp >>>>>>> >>>>> 00007f5347894a70 error 4 >>>>>>> in >>>>>>> libcos-plugin.so[7f533d81f000+b000] >>>>>>> >>>>> [11832.727989] >>>>>>> ns-slapd[22606]: segfault at 0 >>>>>>> ip 00007f6231eb951c sp >>>>>>> >>>>> 00007f623bf2ba70 error 4 >>>>>>> in libcos-plugin.so[7f6231eb6000+b00 >>>>>>> >>>> >>>>>>> >>>> Okay, this one is serious. >>>>>>> The LDAP server crashed. >>>>>>> >>>> >>>>>>> >>>> 1. Make sure all your >>>>>>> packages are up-to-date. >>>>>>> >>>> >>>>>>> >>>> Please see >>>>>>> >>>> >>>>>>> http://directory.fedoraproject.org/docs/389ds/FAQ/faq.html#d >>>>>>> >>>> ebugging-crashes >>>>>>> >>>> for further instructions >>>>>>> how to debug this. >>>>>>> >>>> >>>>>>> >>>> Petr^2 Spacek >>>>>>> >>>> >>>>>>> >>>>> >>>>>>> >>>>> and in >>>>>>> /var/log/dirsrv/example-com/errors >>>>>>> >>>>> >>>>>>> >>>>> [23/Aug/2016:12:49:36 >>>>>>> +0000] DSRetroclPlugin - >>>>>>> delete_changerecord: >>>>>>> >>>> could >>>>>>> >>>>> not delete change record >>>>>>> 3291138 (rc: 32) >>>>>>> >>>>> [23/Aug/2016:12:49:36 >>>>>>> +0000] DSRetroclPlugin - >>>>>>> delete_changerecord: >>>>>>> >>>> could >>>>>>> >>>>> not delete change record >>>>>>> 3291139 (rc: 32) >>>>>>> >>>>> [23/Aug/2016:12:49:36 >>>>>>> +0000] DSRetroclPlugin - >>>>>>> delete_changerecord: >>>>>>> >>>> could >>>>>>> >>>>> not delete change record >>>>>>> 3291140 (rc: 32) >>>>>>> >>>>> [23/Aug/2016:12:49:36 >>>>>>> +0000] DSRetroclPlugin - >>>>>>> delete_changerecord: >>>>>>> >>>> could >>>>>>> >>>>> not delete change record >>>>>>> 3291141 (rc: 32) >>>>>>> >>>>> [23/Aug/2016:12:49:36 >>>>>>> +0000] DSRetroclPlugin - >>>>>>> delete_changerecord: >>>>>>> >>>> could >>>>>>> >>>>> not delete change record >>>>>>> 3291142 (rc: 32) >>>>>>> >>>>> [23/Aug/2016:12:49:36 >>>>>>> +0000] DSRetroclPlugin - >>>>>>> delete_changerecord: >>>>>>> >>>> could >>>>>>> >>>>> not delete change record >>>>>>> 3291143 (rc: 32) >>>>>>> >>>>> [23/Aug/2016:12:49:36 >>>>>>> +0000] DSRetroclPlugin - >>>>>>> delete_changerecord: >>>>>>> >>>> could >>>>>>> >>>>> not delete change record >>>>>>> 3291144 (rc: 32) >>>>>>> >>>>> [23/Aug/2016:12:49:36 >>>>>>> +0000] DSRetroclPlugin - >>>>>>> delete_changerecord: >>>>>>> >>>> could >>>>>>> >>>>> not delete change record >>>>>>> 3291145 (rc: 32) >>>>>>> >>>>> [23/Aug/2016:12:49:50 >>>>>>> +0000] - Retry count exceeded in >>>>>>> delete >>>>>>> >>>>> [23/Aug/2016:12:49:50 >>>>>>> +0000] DSRetroclPlugin - >>>>>>> delete_changerecord: >>>>>>> >>>> could >>>>>>> >>>>> not delete change record >>>>>>> 3292734 (rc: 51) >>>>>>> >>>>> >>>>>>> >>>>> >>>>>>> >>>>> Can i do something about >>>>>>> this error.. I treid to restart >>>>>>> ipa a couple >>>>>>> >>>> of >>>>>>> >>>>> time but that did not help >>>>>>> >>>>> >>>>>>> >>>>> Thanks >>>>>>> >>>>> Rakesh >>>>>>> >>>>> >>>>>>> >>>>> On Mon, Aug 22, 2016 at >>>>>>> 2:27 PM, Petr Spacek >>>>>>> >>>>>>> >>>> wrote: >>>>>>> >>>>> >>>>>>> >>>>>> On 19.8.2016 19:32, >>>>>>> Rakesh Rajasekharan wrote: >>>>>>> >>>>>>> I am running my set up >>>>>>> on AWS cloud, and entropy is low >>>>>>> at around >>>>>>> >>>> 180 . >>>>>>> >>>>>>> >>>>>>> >>>>>>> I plan to increase it bu >>>>>>> installing haveged . But, would >>>>>>> low entropy >>>>>>> >>>> by >>>>>>> >>>>>> any >>>>>>> >>>>>>> chance cause this issue >>>>>>> of intermittent hang . >>>>>>> >>>>>>> Also, the hang is mostly >>>>>>> observed when registering around >>>>>>> 20 clients >>>>>>> >>>>>>> together >>>>>>> >>>>>> >>>>>>> >>>>>> Possibly, I'm not sure. >>>>>>> If you want to dig into this, I >>>>>>> would do this: >>>>>>> >>>>>> 1. look what process >>>>>>> hangs on client (using pstree >>>>>>> command or so) >>>>>>> >>>>>> $ pstree >>>>>>> >>>>>> >>>>>>> >>>>>> 2. look to what server >>>>>>> and port is the hanging client >>>>>>> connected to >>>>>>> >>>>>> $ lsof -p >>>>>> hanging process> >>>>>>> >>>>>> >>>>>>> >>>>>> 3. jump to server and see >>>>>>> what process is bound to the >>>>>>> target port >>>>>>> >>>>>> $ netstat -pn >>>>>>> >>>>>> >>>>>>> >>>>>> 4. see where the process >>>>>>> if hanging >>>>>>> >>>>>> $ strace -p >>>>>> hanging process> >>>>>>> >>>>>> >>>>>>> >>>>>> I hope it helps. >>>>>>> >>>>>> >>>>>>> >>>>>> Petr^2 Spacek >>>>>>> >>>>>> >>>>>>> >>>>>>> On Fri, Aug 19, 2016 at >>>>>>> 7:24 PM, Rakesh Rajasekharan < >>>>>>> >>>>>>> >>>>>>> rakesh.rajasekharan at gmail.com> >>>>>>> wrote: >>>>>>> >>>>>>> >>>>>>> >>>>>>>> yes there seems to be >>>>>>> something thats worrying.. I >>>>>>> have faced this >>>>>>> >>>> today >>>>>>> >>>>>>>> as well. >>>>>>> >>>>>>>> There are few hosts >>>>>>> around 280 odd left and when i >>>>>>> try adding them >>>>>>> >>>> to >>>>>>> >>>>>> IPA >>>>>>> >>>>>>>> , the slowness begins.. >>>>>>> >>>>>>>> >>>>>>> >>>>>>>> all the ipa commands >>>>>>> like ipa user-find.. etc becomes >>>>>>> very slow in >>>>>>> >>>>>>>> responding. >>>>>>> >>>>>>>> >>>>>>> >>>>>>>> the SYNC_RECV are not >>>>>>> many though just around 80-90 >>>>>>> and today that >>>>>>> >>>> was >>>>>>> >>>>>>>> around 20 only >>>>>>> >>>>>>>> >>>>>>> >>>>>>>> >>>>>>> >>>>>>>> I have for now >>>>>>> increased tcp_max_syn_backlog to >>>>>>> 5000. >>>>>>> >>>>>>>> For now the slowness >>>>>>> seems to have gone.. but I will >>>>>>> do a try >>>>>>> >>>> adding the >>>>>>> >>>>>>>> clients again tomorrow >>>>>>> and see how it goes >>>>>>> >>>>>>>> >>>>>>> >>>>>>>> Thanks >>>>>>> >>>>>>>> Rakesh >>>>>>> >>>>>>>> >>>>>>> >>>>>>>> The issues >>>>>>> >>>>>>>> >>>>>>> >>>>>>>> On Fri, Aug 19, 2016 at >>>>>>> 12:58 PM, Petr Spacek >>>>>>> >>>>>>> >>>>>> wrote: >>>>>>> >>>>>>>> >>>>>>> >>>>>>>>> On 18.8.2016 17:23, >>>>>>> Rakesh Rajasekharan wrote: >>>>>>> >>>>>>>>>> Hi >>>>>>> >>>>>>>>>> >>>>>>> >>>>>>>>>> I am migrating to >>>>>>> freeipa from openldap and have >>>>>>> around 4000 >>>>>>> >>>> clients >>>>>>> >>>>>>>>>> >>>>>>> >>>>>>>>>> I had openned a >>>>>>> another thread on that, but >>>>>>> chose to start a new >>>>>>> >>>> one >>>>>>> >>>>>>>>> here >>>>>>> >>>>>>>>>> as its a separate issue >>>>>>> >>>>>>>>>> >>>>>>> >>>>>>>>>> I was able to change >>>>>>> the nssslapd-maxdescriptors >>>>>>> adding an ldif >>>>>>> >>>> file >>>>>>> >>>>>>>>>> >>>>>>> >>>>>>>>>> cat nsslapd-modify.ldif >>>>>>> >>>>>>>>>> dn: cn=config >>>>>>> >>>>>>>>>> changetype: modify >>>>>>> >>>>>>>>>> replace: >>>>>>> nsslapd-maxdescriptors >>>>>>> >>>>>>>>>> >>>>>>> nsslapd-maxdescriptors: 17000 >>>>>>> >>>>>>>>>> >>>>>>> >>>>>>>>>> and running the >>>>>>> ldapmodify command >>>>>>> >>>>>>>>>> >>>>>>> >>>>>>>>>> I have now started >>>>>>> moving clients running an >>>>>>> openldap to Freeipa >>>>>>> >>>> and >>>>>>> >>>>>>>>> have >>>>>>> >>>>>>>>>> today moved close to >>>>>>> 2000 clients >>>>>>> >>>>>>>>>> >>>>>>> >>>>>>>>>> However, I have >>>>>>> noticed that IPA hangs >>>>>>> intermittently. >>>>>>> >>>>>>>>>> >>>>>>> >>>>>>>>>> running a kinit admin >>>>>>> returns the below error >>>>>>> >>>>>>>>>> kinit: Generic error >>>>>>> (see e-text) while getting initial >>>>>>> >>>> credentials >>>>>>> >>>>>>>>>> >>>>>>> >>>>>>>>>> from the >>>>>>> /var/log/messages, I see this entry >>>>>>> >>>>>>>>>> >>>>>>> >>>>>>>>>> prod-ipa-master-int >>>>>>> kernel: [104090.315801] TCP: >>>>>>> >>>> request_sock_TCP: >>>>>>> >>>>>>>>>> Possible SYN flooding >>>>>>> on port 88. Sending cookies. >>>>>>> Check SNMP >>>>>>> >>>>>> counters. >>>>>>> >>>>>>>>> >>>>>>> >>>>>>>>> I would be worried >>>>>>> about this message. Maybe >>>>>>> kernel/firewall is >>>>>>> >>>> doing >>>>>>> >>>>>>>>> something fishy behind >>>>>>> your back and blocking some >>>>>>> connections or >>>>>>> >>>> so. >>>>>>> >>>>>>>>> >>>>>>> >>>>>>>>> Petr^2 Spacek >>>>>>> >>>>>>>>> >>>>>>> >>>>>>>>> >>>>>>> >>>>>>>>>> Aug 18 13:00:01 >>>>>>> prod-ipa-master-int systemd[1]: >>>>>>> Started Session >>>>>>> >>>> 4885 >>>>>>> >>>>>> of >>>>>>> >>>>>>>>>> user root. >>>>>>> >>>>>>>>>> Aug 18 13:00:01 >>>>>>> prod-ipa-master-int systemd[1]: >>>>>>> Starting Session >>>>>>> >>>> 4885 >>>>>>> >>>>>> of >>>>>>> >>>>>>>>>> user root. >>>>>>> >>>>>>>>>> Aug 18 13:01:01 >>>>>>> prod-ipa-master-int systemd[1]: >>>>>>> Started Session >>>>>>> >>>> 4886 >>>>>>> >>>>>> of >>>>>>> >>>>>>>>>> user root. >>>>>>> >>>>>>>>>> Aug 18 13:01:01 >>>>>>> prod-ipa-master-int systemd[1]: >>>>>>> Starting Session >>>>>>> >>>> 4886 >>>>>>> >>>>>> of >>>>>>> >>>>>>>>>> user root. >>>>>>> >>>>>>>>>> Aug 18 13:02:40 >>>>>>> prod-ipa-master-int >>>>>>> python[28984]: ansible-command >>>>>>> >>>>>>>>> Invoked >>>>>>> >>>>>>>>>> with creates=None >>>>>>> executable=None shell=True args= >>>>>>> removes=None >>>>>>> >>>>>>>>> warn=True >>>>>>> >>>>>>>>>> chdir=None >>>>>>> >>>>>>>>>> Aug 18 13:04:37 >>>>>>> prod-ipa-master-int sssd_be: >>>>>>> GSSAPI Error: >>>>>>> >>>> Unspecified >>>>>>> >>>>>>>>> GSS >>>>>>> >>>>>>>>>> failure. Minor code >>>>>>> may provide more information >>>>>>> (KDC returned >>>>>>> >>>> error >>>>>>> >>>>>>>>>> string: PROCESS_TGS) >>>>>>> >>>>>>>>>> >>>>>>> >>>>>>>>>> Could it be possible >>>>>>> that its due to the initial load >>>>>>> of adding >>>>>>> >>>> the >>>>>>> >>>>>>>>> clients >>>>>>> >>>>>>>>>> or is there something >>>>>>> else that I need to take care of. >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>> >>>>>> >>>>>> >>>>> >>>>> >>>>> >>>>> >>>> >>>> >>>> -- >>>> Manage your subscription for the Freeipa-users >>>> mailing list: >>>> https://www.redhat.com/mailman/listinfo/freeipa-users >>>> Go to http://freeipa.org for more info on the project >>>> >>>> >>>> >>>> >>>> >>> >>> >> >> > -- Red Hat GmbH, http://www.de.redhat.com/, Registered seat: Grasbrunn, Commercial register: Amtsgericht Muenchen, HRB 153243, Managing Directors: Charles Cachera, Michael Cunningham, Michael O'Neill, Eric Shander -------------- next part -------------- An HTML attachment was scrubbed... URL: From gjn at gjn.priv.at Mon Sep 5 14:53:39 2016 From: gjn at gjn.priv.at (=?ISO-8859-1?Q?G=FCnther_J=2E?= Niederwimmer) Date: Mon, 05 Sep 2016 16:53:39 +0200 Subject: [Freeipa-users] Error by creating a services Message-ID: <38768539.99melkodZ8@techz> Hello, CentOs 7.2 FreeIPA: 4.2.0-15 Why is this Error only on one Server ? IPA Error 4019: DNSNotARecordError Host does not have corresponding DNS A/AAAA record When I create on a other Server (KVM) a service, this is no Problem, but on my new "mx03.example.com" I have this Errors ? The DNS is in the Moment not configured with IPA, it is a KVM Client (named) and it works correct? Is this a Name Problem (mx03) or ...... Thanks for a answer Reverse Zone is correct set to mx03."example.com" -- mit freundlichen Gr??en / best regards, G?nther J. Niederwimmer From mbasti at redhat.com Mon Sep 5 15:09:03 2016 From: mbasti at redhat.com (Martin Basti) Date: Mon, 5 Sep 2016 17:09:03 +0200 Subject: [Freeipa-users] Error by creating a services In-Reply-To: <38768539.99melkodZ8@techz> References: <38768539.99melkodZ8@techz> Message-ID: On 05.09.2016 16:53, G?nther J. Niederwimmer wrote: > Hello, > > CentOs 7.2 > FreeIPA: 4.2.0-15 > > Why is this Error only on one Server ? Hello, probably you have something DNS related misconfigured on that particular server. Can you resolve hostname manually from server? (host, dig A commands) Martin > > IPA Error 4019: DNSNotARecordError > Host does not have corresponding DNS A/AAAA record > > When I create on a other Server (KVM) a service, this is no Problem, but on my > new "mx03.example.com" I have this Errors ? > > The DNS is in the Moment not configured with IPA, it is a KVM Client (named) > and it works correct? > > Is this a Name Problem (mx03) or ...... > > Thanks for a answer > > Reverse Zone is correct set to mx03."example.com" > From gjn at gjn.priv.at Mon Sep 5 15:43:26 2016 From: gjn at gjn.priv.at (=?ISO-8859-1?Q?G=FCnther_J=2E?= Niederwimmer) Date: Mon, 05 Sep 2016 17:43:26 +0200 Subject: [Freeipa-users] Error by creating a services In-Reply-To: References: <38768539.99melkodZ8@techz> Message-ID: <24970716.1YqJe0nuC0@techz> Hello, Am Montag, 5. September 2016, 17:09:03 schrieb Martin Basti: > On 05.09.2016 16:53, G?nther J. Niederwimmer wrote: > > Hello, > > > > CentOs 7.2 > > FreeIPA: 4.2.0-15 > > > > Why is this Error only on one Server ? > > Hello, > > probably you have something DNS related misconfigured on that particular > server. > > Can you resolve hostname manually from server? (host, dig A commands) This was my first check ;-) , yes I all is correct host and dig is working, this is a external server and I can connect remote with mx03.example.com I check also the reverse Zone, this is also correct ? But it is not possible to create a service, the enrollment is working with no errors ? OK, now I make ipa-client-install --uninstall and a ipa-client-install again and test it ? > > IPA Error 4019: DNSNotARecordError > > Host does not have corresponding DNS A/AAAA record > > > > When I create on a other Server (KVM) a service, this is no Problem, but > > on my new "mx03.example.com" I have this Errors ? > > > > The DNS is in the Moment not configured with IPA, it is a KVM Client > > (named) and it works correct? > > Is this a Name Problem (mx03) or ...... > > > > Thanks for a answer > > > > Reverse Zone is correct set to mx03."example.com" -- mit freundlichen Gr??en / best regards, G?nther J. Niederwimmer From rakesh.rajasekharan at gmail.com Mon Sep 5 18:39:19 2016 From: rakesh.rajasekharan at gmail.com (Rakesh Rajasekharan) Date: Tue, 6 Sep 2016 00:09:19 +0530 Subject: [Freeipa-users] Freeipa 4.2.0 hangs intermittently In-Reply-To: <57CD68BD.9030802@redhat.com> References: <57BEB0AB.2090506@redhat.com> <57C44AC6.1030402@redhat.com> <57CD1FDC.2010109@redhat.com> <57CD5488.60404@redhat.com> <57CD659B.6050707@redhat.com> <57CD68BD.9030802@redhat.com> Message-ID: Hi Thierry/Ludwig, Unfortunately I had by then restarted the service and since then have been trying to hit the hang issue again but no luck this time It has been running fine for the last few hours now in-spite of trying all that I believe could cause the hang like making simultaneous client addition requests. I will continue testing this and next time I get it I will make sure I get the pstack and also get the dump of db lock info. Thanks, Rakesh On Mon, Sep 5, 2016 at 6:14 PM, Ludwig Krispenz wrote: > > On 09/05/2016 02:31 PM, thierry bordaz wrote: > > Hi Rakesh, > > Thanks you soooo much for you continuous help on this !! > > If the hang is still going on, would you dump the DB lock information with: > > db_stat -CA -N -h /var/lib/dirsrv/slapd-/db >/tmp/db_lock_info > > It is looking like several threads are waiting for one/several db locks. > With the lock info we will know which thread acquired it/them. > A guess is that it could be the checkpointing thread but this one is also > waiting (on a memory pool). > > I am copying Ludwig on this thread in case this pstack rings any bell. > > the frequent presence of __db_tas_mutex_lock() looks like the NUMA problem > of libdb, although the threads already have stopped spinning (maybe this > was an effect of gcore). > If the hang aoccurs again, could you check cpu consumption > top -H -p > and get a pstack > pstack > > > best regards > thierry > > On 09/05/2016 02:05 PM, Rakesh Rajasekharan wrote: > > Hi Thierry, > > I was able to now reporduce the issue.. > > the ns-slapd process just got hung,.. however, no core file was generate.. > > I manually created a coredump using gcore and then took a strace of it > this is strace from the coredump generated > > GNU gdb (GDB) Red Hat Enterprise Linux 7.6.1-80.el7 > Copyright (C) 2013 Free Software Foundation, Inc. > License GPLv3+: GNU GPL version 3 or later html> > This is free software: you are free to change and redistribute it. > There is NO WARRANTY, to the extent permitted by law. Type "show copying" > and "show warranty" for details. > This GDB was configured as "x86_64-redhat-linux-gnu". > For bug reporting instructions, please see: > ... > Reading symbols from /usr/sbin/ns-slapd...Reading symbols from > /usr/sbin/ns-slapd...(no debugging symbols found)...done. > (no debugging symbols found)...done. > [New LWP 105159] > [New LWP 105160] > [New LWP 105161] > [New LWP 105162] > [New LWP 105163] > [New LWP 105164] > [New LWP 105165] > [New LWP 105166] > [New LWP 105167] > [New LWP 105168] > [New LWP 105171] > [New LWP 105172] > [New LWP 105173] > [New LWP 105174] > [New LWP 105175] > [New LWP 105176] > [New LWP 105177] > [New LWP 105178] > [New LWP 105179] > [New LWP 105180] > [New LWP 105181] > [New LWP 105182] > [New LWP 105183] > [New LWP 105184] > [New LWP 105185] > [New LWP 105186] > [New LWP 105187] > [New LWP 105188] > [New LWP 105189] > [New LWP 105190] > [New LWP 105191] > [New LWP 105192] > [New LWP 105193] > [New LWP 105194] > [New LWP 105195] > [New LWP 105196] > [New LWP 105197] > [New LWP 105198] > [New LWP 105199] > [New LWP 105200] > [New LWP 105201] > [New LWP 105203] > [New LWP 105215] > [New LWP 105158] > [Thread debugging using libthread_db enabled] > Using host libthread_db library "/lib64/libthread_db.so.1". > Core was generated by `/usr/sbin/ns-slapd'. > #0 0x00007f1b48ae69b3 in select () from /lib64/libc.so.6 > > Thread 44 (Thread 0x7f1b4b6d9840 (LWP 105158)): > #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 () from > /lib64/libpthread.so.0 > No symbol table info available. > #1 0x00007f1b4941b2c3 in PR_EnterMonitor () from /lib64/libnspr4.so > No symbol table info available. > #2 0x00007f1b4b7125ed in setup_pr_read_pds () > No symbol table info available. > #3 0x00007f1b4b715fd9 in slapd_daemon () > No symbol table info available. > #4 0x00007f1b4b707c6c in main () > No symbol table info available. > > Thread 43 (Thread 0x7f1a5df0f700 (LWP 105215)): > #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 () from > /lib64/libpthread.so.0 > No symbol table info available. > #1 0x00007f1b4941b050 in PR_WaitCondVar () from /lib64/libnspr4.so > No symbol table info available. > #2 0x00007f1b4b71df25 in ps_send_results () > No symbol table info available. > #3 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so > No symbol table info available. > #4 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 > No symbol table info available. > #5 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 > No symbol table info available. > > Thread 42 (Thread 0x7f1b03fe7700 (LWP 105203)): > #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 () from > /lib64/libpthread.so.0 > No symbol table info available. > #1 0x00007f1b4941b050 in PR_WaitCondVar () from /lib64/libnspr4.so > No symbol table info available. > #2 0x00007f1b4b71df25 in ps_send_results () > No symbol table info available. > #3 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so > No symbol table info available. > #4 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 > No symbol table info available. > #5 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 > No symbol table info available. > > Thread 41 (Thread 0x7f1b047e8700 (LWP 105201)): > #0 0x00007f1b48ae69b3 in select () from /lib64/libc.so.6 > No symbol table info available. > #1 0x00007f1b4b2710e9 in DS_Sleep () from /usr/lib64/dirsrv/libslapd.so.0 > No symbol table info available. > #2 0x00007f1b4b712325 in time_thread () > No symbol table info available. > #3 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so > No symbol table info available. > #4 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 > No symbol table info available. > #5 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 > No symbol table info available. > > Thread 40 (Thread 0x7f1b04fe9700 (LWP 105200)): > #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 () from > /lib64/libpthread.so.0 > No symbol table info available. > #1 0x00007f1b433212f3 in __db_hybrid_mutex_suspend () from /lib64/ > libdb-5.3.so > No symbol table info available. > #2 0x00007f1b43320640 in __db_tas_mutex_lock () from /lib64/libdb-5.3.so > No symbol table info available. > #3 0x00007f1b4343ae1b in __memp_fget () from /lib64/libdb-5.3.so > No symbol table info available. > #4 0x00007f1b4333e211 in __bam_search () from /lib64/libdb-5.3.so > No symbol table info available. > #5 0x00007f1b43329256 in __bamc_search () from /lib64/libdb-5.3.so > No symbol table info available. > #6 0x00007f1b4332d6c4 in __bamc_put () from /lib64/libdb-5.3.so > No symbol table info available. > #7 0x00007f1b433e46d5 in __dbc_iput () from /lib64/libdb-5.3.so > No symbol table info available. > #8 0x00007f1b433dfa0e in __db_put () from /lib64/libdb-5.3.so > No symbol table info available. > #9 0x00007f1b433f4fa4 in __db_put_pp () from /lib64/libdb-5.3.so > No symbol table info available. > #10 0x00007f1b3e8cf4cf in idl_new_insert_key () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > No symbol table info available. > #11 0x00007f1b3e8de5b5 in addordel_values_sv.isra.3 () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > No symbol table info available. > #12 0x00007f1b3e8df25a in index_addordel_values_ext_sv () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > No symbol table info available. > #13 0x00007f1b3e8df694 in index_addordel_values_sv () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > No symbol table info available. > #14 0x00007f1b3e8df75a in index_addordel_entry () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > No symbol table info available. > #15 0x00007f1b3e8e25fa in ldbm_back_add () from /usr/lib64/dirsrv/plugins/ > libback-ldbm.so > No symbol table info available. > #16 0x00007f1b4b1e4cca in op_shared_add () from > /usr/lib64/dirsrv/libslapd.so.0 > No symbol table info available. > #17 0x00007f1b4b1e6020 in do_add () from /usr/lib64/dirsrv/libslapd.so.0 > No symbol table info available. > #18 0x00007f1b4b711303 in connection_threadmain () > No symbol table info available. > #19 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so > No symbol table info available. > #20 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 > No symbol table info available. > #21 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 > No symbol table info available. > > Thread 39 (Thread 0x7f1b057ea700 (LWP 105199)): > #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 () from > /lib64/libpthread.so.0 > No symbol table info available. > #1 0x00007f1b4941b050 in PR_WaitCondVar () from /lib64/libnspr4.so > No symbol table info available. > #2 0x00007f1b4b70efee in connection_wait_for_new_work () > No symbol table info available. > #3 0x00007f1b4b710719 in connection_threadmain () > No symbol table info available. > #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so > No symbol table info available. > #5 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 > No symbol table info available. > #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 > No symbol table info available. > > Thread 38 (Thread 0x7f1b05feb700 (LWP 105198)): > #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 () from > /lib64/libpthread.so.0 > No symbol table info available. > #1 0x00007f1b4941b050 in PR_WaitCondVar () from /lib64/libnspr4.so > No symbol table info available. > #2 0x00007f1b4b70efee in connection_wait_for_new_work () > No symbol table info available. > #3 0x00007f1b4b710719 in connection_threadmain () > No symbol table info available. > #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so > No symbol table info available. > #5 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 > No symbol table info available. > #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 > No symbol table info available. > > Thread 37 (Thread 0x7f1b067ec700 (LWP 105197)): > #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 () from > /lib64/libpthread.so.0 > No symbol table info available. > #1 0x00007f1b4941b050 in PR_WaitCondVar () from /lib64/libnspr4.so > No symbol table info available. > #2 0x00007f1b4b70efee in connection_wait_for_new_work () > No symbol table info available. > #3 0x00007f1b4b710719 in connection_threadmain () > No symbol table info available. > #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so > No symbol table info available. > #5 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 > No symbol table info available. > #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 > No symbol table info available. > > Thread 36 (Thread 0x7f1b06fed700 (LWP 105196)): > #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 () from > /lib64/libpthread.so.0 > No symbol table info available. > #1 0x00007f1b4941b050 in PR_WaitCondVar () from /lib64/libnspr4.so > No symbol table info available. > #2 0x00007f1b4b70efee in connection_wait_for_new_work () > No symbol table info available. > #3 0x00007f1b4b710719 in connection_threadmain () > No symbol table info available. > #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so > No symbol table info available. > #5 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 > No symbol table info available. > #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 > No symbol table info available. > > Thread 35 (Thread 0x7f1b077ee700 (LWP 105195)): > #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 () from > /lib64/libpthread.so.0 > No symbol table info available. > #1 0x00007f1b4941b2c3 in PR_EnterMonitor () from /lib64/libnspr4.so > No symbol table info available. > #2 0x00007f1b3e8c23e6 in dblayer_txn_begin () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > No symbol table info available. > #3 0x00007f1b3e8fe007 in ldbm_back_modify () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > No symbol table info available. > #4 0x00007f1b4b22b541 in op_shared_modify () from > /usr/lib64/dirsrv/libslapd.so.0 > No symbol table info available. > #5 0x00007f1b4b22c004 in modify_internal_pb () from > /usr/lib64/dirsrv/libslapd.so.0 > No symbol table info available. > #6 0x00007f1b3eb48b60 in ipapwd_apply_mods () from > /usr/lib64/dirsrv/plugins/libipa_pwd_extop.so > No symbol table info available. > #7 0x00007f1b3eb4c8e0 in store_new_keys () from /usr/lib64/dirsrv/plugins/ > libipa_pwd_extop.so > No symbol table info available. > #8 0x00007f1b3eb4ead0 in ipapwd_extop () from /usr/lib64/dirsrv/plugins/ > libipa_pwd_extop.so > No symbol table info available. > #9 0x00007f1b4b23bba2 in plugin_call_exop_plugins () from > /usr/lib64/dirsrv/libslapd.so.0 > No symbol table info available. > #10 0x00007f1b4b717efe in do_extended () > No symbol table info available. > #11 0x00007f1b4b71141a in connection_threadmain () > No symbol table info available. > #12 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so > No symbol table info available. > #13 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 > No symbol table info available. > #14 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 > No symbol table info available. > > Thread 34 (Thread 0x7f1b07fef700 (LWP 105194)): > #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 () from > /lib64/libpthread.so.0 > No symbol table info available. > #1 0x00007f1b4941b050 in PR_WaitCondVar () from /lib64/libnspr4.so > No symbol table info available. > #2 0x00007f1b4b70efee in connection_wait_for_new_work () > No symbol table info available. > #3 0x00007f1b4b710719 in connection_threadmain () > No symbol table info available. > #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so > No symbol table info available. > #5 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 > No symbol table info available. > #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 > No symbol table info available. > > Thread 33 (Thread 0x7f1b087f0700 (LWP 105193)): > #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 () from > /lib64/libpthread.so.0 > No symbol table info available. > #1 0x00007f1b4941b050 in PR_WaitCondVar () from /lib64/libnspr4.so > No symbol table info available. > #2 0x00007f1b4b70efee in connection_wait_for_new_work () > No symbol table info available. > #3 0x00007f1b4b710719 in connection_threadmain () > No symbol table info available. > #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so > No symbol table info available. > #5 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 > No symbol table info available. > #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 > No symbol table info available. > > Thread 32 (Thread 0x7f1b08ff1700 (LWP 105192)): > #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 () from > /lib64/libpthread.so.0 > No symbol table info available. > #1 0x00007f1b4941b050 in PR_WaitCondVar () from /lib64/libnspr4.so > No symbol table info available. > #2 0x00007f1b4b70efee in connection_wait_for_new_work () > No symbol table info available. > #3 0x00007f1b4b710719 in connection_threadmain () > No symbol table info available. > #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so > No symbol table info available. > #5 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 > No symbol table info available. > #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 > No symbol table info available. > > Thread 31 (Thread 0x7f1b097f2700 (LWP 105191)): > #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 () from > /lib64/libpthread.so.0 > No symbol table info available. > #1 0x00007f1b4941b050 in PR_WaitCondVar () from /lib64/libnspr4.so > No symbol table info available. > #2 0x00007f1b4b70efee in connection_wait_for_new_work () > No symbol table info available. > #3 0x00007f1b4b710719 in connection_threadmain () > No symbol table info available. > #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so > No symbol table info available. > #5 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 > No symbol table info available. > #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 > No symbol table info available. > > Thread 30 (Thread 0x7f1b09ff3700 (LWP 105190)): > #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 () from > /lib64/libpthread.so.0 > No symbol table info available. > #1 0x00007f1b433212f3 in __db_hybrid_mutex_suspend () from /lib64/ > libdb-5.3.so > No symbol table info available. > #2 0x00007f1b43320640 in __db_tas_mutex_lock () from /lib64/libdb-5.3.so > No symbol table info available. > #3 0x00007f1b433cad3a in __lock_get_internal () from /lib64/libdb-5.3.so > No symbol table info available. > #4 0x00007f1b433cb820 in __lock_get () from /lib64/libdb-5.3.so > No symbol table info available. > #5 0x00007f1b433f7142 in __db_lget () from /lib64/libdb-5.3.so > No symbol table info available. > #6 0x00007f1b4333e605 in __bam_search () from /lib64/libdb-5.3.so > No symbol table info available. > #7 0x00007f1b43329256 in __bamc_search () from /lib64/libdb-5.3.so > No symbol table info available. > #8 0x00007f1b4332ad0f in __bamc_get () from /lib64/libdb-5.3.so > No symbol table info available. > #9 0x00007f1b433e3ca6 in __dbc_iget () from /lib64/libdb-5.3.so > No symbol table info available. > #10 0x00007f1b433f2b02 in __dbc_get_pp () from /lib64/libdb-5.3.so > No symbol table info available. > #11 0x00007f1b3e8ce680 in idl_new_fetch () from /usr/lib64/dirsrv/plugins/ > libback-ldbm.so > No symbol table info available. > #12 0x00007f1b3e8dcb26 in index_read_ext_allids () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > No symbol table info available. > #13 0x00007f1b3e8c7284 in keys2idl () from /usr/lib64/dirsrv/plugins/ > libback-ldbm.so > No symbol table info available. > #14 0x00007f1b3e8c79e3 in ava_candidates.isra.0 () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > No symbol table info available. > #15 0x00007f1b3e8c7fd2 in filter_candidates_ext () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > No symbol table info available. > #16 0x00007f1b3e8c9096 in list_candidates () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > No symbol table info available. > #17 0x00007f1b3e8c7f40 in filter_candidates_ext () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > No symbol table info available. > #18 0x00007f1b3e8c9096 in list_candidates () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > No symbol table info available. > #19 0x00007f1b3e8c7f40 in filter_candidates_ext () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > No symbol table info available. > #20 0x00007f1b3e8c9096 in list_candidates () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > No symbol table info available. > #21 0x00007f1b3e8c7f40 in filter_candidates_ext () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > No symbol table info available. > #22 0x00007f1b3e90380c in subtree_candidates () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > No symbol table info available. > #23 0x00007f1b3e904e7f in ldbm_back_search () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > No symbol table info available. > #24 0x00007f1b4b2321d6 in op_shared_search () from > /usr/lib64/dirsrv/libslapd.so.0 > No symbol table info available. > #25 0x00007f1b4b72333e in do_search () > No symbol table info available. > #26 0x00007f1b4b7113c3 in connection_threadmain () > No symbol table info available. > #27 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so > No symbol table info available. > #28 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 > No symbol table info available. > #29 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 > No symbol table info available. > > Thread 29 (Thread 0x7f1b0a7f4700 (LWP 105189)): > #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 () from > /lib64/libpthread.so.0 > No symbol table info available. > #1 0x00007f1b4941b050 in PR_WaitCondVar () from /lib64/libnspr4.so > No symbol table info available. > #2 0x00007f1b4b70efee in connection_wait_for_new_work () > No symbol table info available. > #3 0x00007f1b4b710719 in connection_threadmain () > No symbol table info available. > #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so > No symbol table info available. > #5 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 > No symbol table info available. > #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 > No symbol table info available. > > Thread 28 (Thread 0x7f1b0aff5700 (LWP 105188)): > #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 () from > /lib64/libpthread.so.0 > No symbol table info available. > #1 0x00007f1b4941b050 in PR_WaitCondVar () from /lib64/libnspr4.so > No symbol table info available. > #2 0x00007f1b4b70efee in connection_wait_for_new_work () > No symbol table info available. > #3 0x00007f1b4b710719 in connection_threadmain () > No symbol table info available. > #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so > No symbol table info available. > #5 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 > No symbol table info available. > #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 > No symbol table info available. > > Thread 27 (Thread 0x7f1b0b7f6700 (LWP 105187)): > #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 () from > /lib64/libpthread.so.0 > No symbol table info available. > #1 0x00007f1b4941b050 in PR_WaitCondVar () from /lib64/libnspr4.so > No symbol table info available. > #2 0x00007f1b4b70efee in connection_wait_for_new_work () > No symbol table info available. > #3 0x00007f1b4b710719 in connection_threadmain () > No symbol table info available. > #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so > No symbol table info available. > #5 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 > No symbol table info available. > #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 > No symbol table info available. > > Thread 26 (Thread 0x7f1b0bff7700 (LWP 105186)): > #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 () from > /lib64/libpthread.so.0 > No symbol table info available. > #1 0x00007f1b433212f3 in __db_hybrid_mutex_suspend () from /lib64/ > libdb-5.3.so > No symbol table info available. > #2 0x00007f1b43320640 in __db_tas_mutex_lock () from /lib64/libdb-5.3.so > No symbol table info available. > #3 0x00007f1b433cad3a in __lock_get_internal () from /lib64/libdb-5.3.so > No symbol table info available. > #4 0x00007f1b433cb820 in __lock_get () from /lib64/libdb-5.3.so > No symbol table info available. > #5 0x00007f1b433f7142 in __db_lget () from /lib64/libdb-5.3.so > No symbol table info available. > #6 0x00007f1b4333e605 in __bam_search () from /lib64/libdb-5.3.so > No symbol table info available. > #7 0x00007f1b43329256 in __bamc_search () from /lib64/libdb-5.3.so > No symbol table info available. > #8 0x00007f1b4332ad0f in __bamc_get () from /lib64/libdb-5.3.so > No symbol table info available. > #9 0x00007f1b433e3ca6 in __dbc_iget () from /lib64/libdb-5.3.so > No symbol table info available. > #10 0x00007f1b433f2b02 in __dbc_get_pp () from /lib64/libdb-5.3.so > No symbol table info available. > #11 0x00007f1b3e8ce680 in idl_new_fetch () from /usr/lib64/dirsrv/plugins/ > libback-ldbm.so > No symbol table info available. > #12 0x00007f1b3e8dcb26 in index_read_ext_allids () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > No symbol table info available. > #13 0x00007f1b3e8c7284 in keys2idl () from /usr/lib64/dirsrv/plugins/ > libback-ldbm.so > No symbol table info available. > #14 0x00007f1b3e8c79e3 in ava_candidates.isra.0 () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > No symbol table info available. > #15 0x00007f1b3e8c7fd2 in filter_candidates_ext () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > No symbol table info available. > #16 0x00007f1b3e8c9096 in list_candidates () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > No symbol table info available. > #17 0x00007f1b3e8c7f40 in filter_candidates_ext () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > No symbol table info available. > #18 0x00007f1b3e90380c in subtree_candidates () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > No symbol table info available. > #19 0x00007f1b3e904e7f in ldbm_back_search () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > No symbol table info available. > #20 0x00007f1b4b2321d6 in op_shared_search () from > /usr/lib64/dirsrv/libslapd.so.0 > No symbol table info available. > #21 0x00007f1b4b24294e in search_internal_callback_pb () from > /usr/lib64/dirsrv/libslapd.so.0 > No symbol table info available. > #22 0x00007f1b4b242be8 in search_internal_pb () from > /usr/lib64/dirsrv/libslapd.so.0 > No symbol table info available. > #23 0x00007f1b4b72149b in ids_sasl_canon_user () > No symbol table info available. > #24 0x00007f1b491e193b in _sasl_canon_user () from /lib64/libsasl2.so.3 > No symbol table info available. > #25 0x00007f1b491e1c4c in _sasl_canon_user_lookup () from > /lib64/libsasl2.so.3 > No symbol table info available. > #26 0x00007f1b430e9fb9 in gssapi_server_mech_step () from > /usr/lib64/sasl2/libgssapiv2.so > No symbol table info available. > #27 0x00007f1b491edb9b in sasl_server_step () from /lib64/libsasl2.so.3 > No symbol table info available. > #28 0x00007f1b4b72283b in ids_sasl_check_bind () > No symbol table info available. > #29 0x00007f1b4b70a61a in do_bind () > No symbol table info available. > #30 0x00007f1b4b7113fd in connection_threadmain () > No symbol table info available. > #31 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so > No symbol table info available. > #32 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 > No symbol table info available. > #33 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 > No symbol table info available. > > Thread 25 (Thread 0x7f1b0c7f8700 (LWP 105185)): > #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 () from > /lib64/libpthread.so.0 > No symbol table info available. > #1 0x00007f1b4941b050 in PR_WaitCondVar () from /lib64/libnspr4.so > No symbol table info available. > #2 0x00007f1b4b70efee in connection_wait_for_new_work () > No symbol table info available. > #3 0x00007f1b4b710719 in connection_threadmain () > No symbol table info available. > #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so > No symbol table info available. > #5 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 > No symbol table info available. > #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 > No symbol table info available. > > Thread 24 (Thread 0x7f1b0cff9700 (LWP 105184)): > #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 () from > /lib64/libpthread.so.0 > No symbol table info available. > #1 0x00007f1b4941b050 in PR_WaitCondVar () from /lib64/libnspr4.so > No symbol table info available. > #2 0x00007f1b4b70efee in connection_wait_for_new_work () > No symbol table info available. > #3 0x00007f1b4b710719 in connection_threadmain () > No symbol table info available. > #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so > No symbol table info available. > #5 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 > No symbol table info available. > #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 > No symbol table info available. > > Thread 23 (Thread 0x7f1b0d7fa700 (LWP 105183)): > #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 () from > /lib64/libpthread.so.0 > No symbol table info available. > #1 0x00007f1b4941b050 in PR_WaitCondVar () from /lib64/libnspr4.so > No symbol table info available. > #2 0x00007f1b4b70efee in connection_wait_for_new_work () > No symbol table info available. > #3 0x00007f1b4b710719 in connection_threadmain () > No symbol table info available. > #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so > No symbol table info available. > #5 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 > No symbol table info available. > #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 > No symbol table info available. > > Thread 22 (Thread 0x7f1b0dffb700 (LWP 105182)): > #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 () from > /lib64/libpthread.so.0 > No symbol table info available. > #1 0x00007f1b4941b050 in PR_WaitCondVar () from /lib64/libnspr4.so > No symbol table info available. > #2 0x00007f1b4b70efee in connection_wait_for_new_work () > No symbol table info available. > #3 0x00007f1b4b710719 in connection_threadmain () > No symbol table info available. > #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so > No symbol table info available. > #5 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 > No symbol table info available. > #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 > No symbol table info available. > > Thread 21 (Thread 0x7f1b0e7fc700 (LWP 105181)): > #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 () from > /lib64/libpthread.so.0 > No symbol table info available. > #1 0x00007f1b4941b050 in PR_WaitCondVar () from /lib64/libnspr4.so > No symbol table info available. > #2 0x00007f1b4b70efee in connection_wait_for_new_work () > No symbol table info available. > #3 0x00007f1b4b710719 in connection_threadmain () > No symbol table info available. > #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so > No symbol table info available. > #5 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 > No symbol table info available. > #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 > No symbol table info available. > > Thread 20 (Thread 0x7f1b0effd700 (LWP 105180)): > #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 () from > /lib64/libpthread.so.0 > No symbol table info available. > #1 0x00007f1b433212f3 in __db_hybrid_mutex_suspend () from /lib64/ > libdb-5.3.so > No symbol table info available. > #2 0x00007f1b43320b37 in __db_tas_mutex_readlock () from /lib64/ > libdb-5.3.so > No symbol table info available. > #3 0x00007f1b4343ad35 in __memp_fget () from /lib64/libdb-5.3.so > No symbol table info available. > #4 0x00007f1b433e4333 in __dbc_iget () from /lib64/libdb-5.3.so > No symbol table info available. > #5 0x00007f1b433f2b02 in __dbc_get_pp () from /lib64/libdb-5.3.so > No symbol table info available. > #6 0x00007f1b3e8ce822 in idl_new_fetch () from /usr/lib64/dirsrv/plugins/ > libback-ldbm.so > No symbol table info available. > #7 0x00007f1b3e8dcb26 in index_read_ext_allids () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > No symbol table info available. > #8 0x00007f1b3e8c7284 in keys2idl () from /usr/lib64/dirsrv/plugins/ > libback-ldbm.so > No symbol table info available. > #9 0x00007f1b3e8c79e3 in ava_candidates.isra.0 () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > No symbol table info available. > #10 0x00007f1b3e8c7fd2 in filter_candidates_ext () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > No symbol table info available. > #11 0x00007f1b3e8c9096 in list_candidates () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > No symbol table info available. > #12 0x00007f1b3e8c7f40 in filter_candidates_ext () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > No symbol table info available. > #13 0x00007f1b3e8c9096 in list_candidates () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > No symbol table info available. > #14 0x00007f1b3e8c7f40 in filter_candidates_ext () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > No symbol table info available. > #15 0x00007f1b3e8c9096 in list_candidates () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > No symbol table info available. > #16 0x00007f1b3e8c7f40 in filter_candidates_ext () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > No symbol table info available. > #17 0x00007f1b3e90380c in subtree_candidates () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > No symbol table info available. > #18 0x00007f1b3e904e7f in ldbm_back_search () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > No symbol table info available. > #19 0x00007f1b4b2321d6 in op_shared_search () from > /usr/lib64/dirsrv/libslapd.so.0 > No symbol table info available. > #20 0x00007f1b4b72333e in do_search () > No symbol table info available. > #21 0x00007f1b4b7113c3 in connection_threadmain () > No symbol table info available. > #22 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so > No symbol table info available. > #23 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 > No symbol table info available. > #24 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 > No symbol table info available. > > Thread 19 (Thread 0x7f1b0f7fe700 (LWP 105179)): > #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 () from > /lib64/libpthread.so.0 > No symbol table info available. > #1 0x00007f1b433212f3 in __db_hybrid_mutex_suspend () from /lib64/ > libdb-5.3.so > No symbol table info available. > #2 0x00007f1b43320640 in __db_tas_mutex_lock () from /lib64/libdb-5.3.so > No symbol table info available. > #3 0x00007f1b433cad3a in __lock_get_internal () from /lib64/libdb-5.3.so > No symbol table info available. > #4 0x00007f1b433cb820 in __lock_get () from /lib64/libdb-5.3.so > No symbol table info available. > #5 0x00007f1b433f7142 in __db_lget () from /lib64/libdb-5.3.so > No symbol table info available. > #6 0x00007f1b4333e605 in __bam_search () from /lib64/libdb-5.3.so > No symbol table info available. > #7 0x00007f1b43329256 in __bamc_search () from /lib64/libdb-5.3.so > No symbol table info available. > #8 0x00007f1b4332ad0f in __bamc_get () from /lib64/libdb-5.3.so > No symbol table info available. > #9 0x00007f1b433e3ca6 in __dbc_iget () from /lib64/libdb-5.3.so > No symbol table info available. > #10 0x00007f1b433f2b02 in __dbc_get_pp () from /lib64/libdb-5.3.so > No symbol table info available. > #11 0x00007f1b3e8ce680 in idl_new_fetch () from /usr/lib64/dirsrv/plugins/ > libback-ldbm.so > No symbol table info available. > #12 0x00007f1b3e8dcb26 in index_read_ext_allids () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > No symbol table info available. > #13 0x00007f1b3e8c7284 in keys2idl () from /usr/lib64/dirsrv/plugins/ > libback-ldbm.so > No symbol table info available. > #14 0x00007f1b3e8c79e3 in ava_candidates.isra.0 () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > No symbol table info available. > #15 0x00007f1b3e8c7fd2 in filter_candidates_ext () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > No symbol table info available. > #16 0x00007f1b3e8c9096 in list_candidates () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > No symbol table info available. > #17 0x00007f1b3e8c7f40 in filter_candidates_ext () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > No symbol table info available. > #18 0x00007f1b3e8c9096 in list_candidates () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > No symbol table info available. > #19 0x00007f1b3e8c7f40 in filter_candidates_ext () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > No symbol table info available. > #20 0x00007f1b3e8c9096 in list_candidates () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > No symbol table info available. > #21 0x00007f1b3e8c7f40 in filter_candidates_ext () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > No symbol table info available. > #22 0x00007f1b3e90380c in subtree_candidates () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > No symbol table info available. > #23 0x00007f1b3e904e7f in ldbm_back_search () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > No symbol table info available. > #24 0x00007f1b4b2321d6 in op_shared_search () from > /usr/lib64/dirsrv/libslapd.so.0 > No symbol table info available. > #25 0x00007f1b4b72333e in do_search () > No symbol table info available. > #26 0x00007f1b4b7113c3 in connection_threadmain () > No symbol table info available. > #27 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so > No symbol table info available. > #28 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 > No symbol table info available. > #29 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 > No symbol table info available. > > Thread 18 (Thread 0x7f1b0ffff700 (LWP 105178)): > #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 () from > /lib64/libpthread.so.0 > No symbol table info available. > #1 0x00007f1b4941b050 in PR_WaitCondVar () from /lib64/libnspr4.so > No symbol table info available. > #2 0x00007f1b4b70efee in connection_wait_for_new_work () > No symbol table info available. > #3 0x00007f1b4b710719 in connection_threadmain () > No symbol table info available. > #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so > No symbol table info available. > #5 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 > No symbol table info available. > #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 > No symbol table info available. > > Thread 17 (Thread 0x7f1b14d28700 (LWP 105177)): > #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 () from > /lib64/libpthread.so.0 > No symbol table info available. > #1 0x00007f1b4941b050 in PR_WaitCondVar () from /lib64/libnspr4.so > No symbol table info available. > #2 0x00007f1b4b70efee in connection_wait_for_new_work () > No symbol table info available. > #3 0x00007f1b4b710719 in connection_threadmain () > No symbol table info available. > #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so > No symbol table info available. > #5 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 > No symbol table info available. > #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 > No symbol table info available. > > Thread 16 (Thread 0x7f1b15529700 (LWP 105176)): > #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 () from > /lib64/libpthread.so.0 > No symbol table info available. > #1 0x00007f1b4941b050 in PR_WaitCondVar () from /lib64/libnspr4.so > No symbol table info available. > #2 0x00007f1b4b70efee in connection_wait_for_new_work () > No symbol table info available. > #3 0x00007f1b4b710719 in connection_threadmain () > No symbol table info available. > #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so > No symbol table info available. > #5 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 > No symbol table info available. > #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 > No symbol table info available. > > Thread 15 (Thread 0x7f1b15d2a700 (LWP 105175)): > #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 () from > /lib64/libpthread.so.0 > No symbol table info available. > #1 0x00007f1b4941b050 in PR_WaitCondVar () from /lib64/libnspr4.so > No symbol table info available. > #2 0x00007f1b4b70efee in connection_wait_for_new_work () > No symbol table info available. > #3 0x00007f1b4b710719 in connection_threadmain () > No symbol table info available. > #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so > No symbol table info available. > #5 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 > No symbol table info available. > #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 > No symbol table info available. > > Thread 14 (Thread 0x7f1b1652b700 (LWP 105174)): > #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 () from > /lib64/libpthread.so.0 > No symbol table info available. > #1 0x00007f1b4941b050 in PR_WaitCondVar () from /lib64/libnspr4.so > No symbol table info available. > #2 0x00007f1b4b70efee in connection_wait_for_new_work () > No symbol table info available. > #3 0x00007f1b4b710719 in connection_threadmain () > No symbol table info available. > #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so > No symbol table info available. > #5 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 > No symbol table info available. > #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 > No symbol table info available. > > Thread 13 (Thread 0x7f1b16d2c700 (LWP 105173)): > #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 () from > /lib64/libpthread.so.0 > No symbol table info available. > #1 0x00007f1b433212f3 in __db_hybrid_mutex_suspend () from /lib64/ > libdb-5.3.so > No symbol table info available. > #2 0x00007f1b43320640 in __db_tas_mutex_lock () from /lib64/libdb-5.3.so > No symbol table info available. > #3 0x00007f1b433cad3a in __lock_get_internal () from /lib64/libdb-5.3.so > No symbol table info available. > #4 0x00007f1b433cb820 in __lock_get () from /lib64/libdb-5.3.so > No symbol table info available. > #5 0x00007f1b433f7142 in __db_lget () from /lib64/libdb-5.3.so > No symbol table info available. > #6 0x00007f1b4333e605 in __bam_search () from /lib64/libdb-5.3.so > No symbol table info available. > #7 0x00007f1b43329256 in __bamc_search () from /lib64/libdb-5.3.so > No symbol table info available. > #8 0x00007f1b4332ad0f in __bamc_get () from /lib64/libdb-5.3.so > No symbol table info available. > #9 0x00007f1b433e3ca6 in __dbc_iget () from /lib64/libdb-5.3.so > No symbol table info available. > #10 0x00007f1b433f2b02 in __dbc_get_pp () from /lib64/libdb-5.3.so > No symbol table info available. > #11 0x00007f1b3e8ce680 in idl_new_fetch () from /usr/lib64/dirsrv/plugins/ > libback-ldbm.so > No symbol table info available. > #12 0x00007f1b3e8dcb26 in index_read_ext_allids () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > No symbol table info available. > #13 0x00007f1b3e8c7284 in keys2idl () from /usr/lib64/dirsrv/plugins/ > libback-ldbm.so > No symbol table info available. > #14 0x00007f1b3e8c79e3 in ava_candidates.isra.0 () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > No symbol table info available. > #15 0x00007f1b3e8c7fd2 in filter_candidates_ext () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > No symbol table info available. > #16 0x00007f1b3e8c9096 in list_candidates () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > No symbol table info available. > #17 0x00007f1b3e8c7f40 in filter_candidates_ext () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > No symbol table info available. > #18 0x00007f1b3e8c9096 in list_candidates () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > No symbol table info available. > #19 0x00007f1b3e8c7f40 in filter_candidates_ext () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > No symbol table info available. > #20 0x00007f1b3e8c9096 in list_candidates () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > No symbol table info available. > #21 0x00007f1b3e8c7f40 in filter_candidates_ext () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > No symbol table info available. > #22 0x00007f1b3e90380c in subtree_candidates () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > No symbol table info available. > #23 0x00007f1b3e904e7f in ldbm_back_search () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > No symbol table info available. > #24 0x00007f1b4b2321d6 in op_shared_search () from > /usr/lib64/dirsrv/libslapd.so.0 > No symbol table info available. > #25 0x00007f1b4b72333e in do_search () > No symbol table info available. > #26 0x00007f1b4b7113c3 in connection_threadmain () > No symbol table info available. > #27 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so > No symbol table info available. > #28 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 > No symbol table info available. > #29 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 > No symbol table info available. > > Thread 12 (Thread 0x7f1b1752d700 (LWP 105172)): > #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 () from > /lib64/libpthread.so.0 > No symbol table info available. > #1 0x00007f1b4941b050 in PR_WaitCondVar () from /lib64/libnspr4.so > No symbol table info available. > #2 0x00007f1b4b70efee in connection_wait_for_new_work () > No symbol table info available. > #3 0x00007f1b4b710719 in connection_threadmain () > No symbol table info available. > #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so > No symbol table info available. > #5 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 > No symbol table info available. > #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 > No symbol table info available. > > Thread 11 (Thread 0x7f1b17d2e700 (LWP 105171)): > #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 () from > /lib64/libpthread.so.0 > No symbol table info available. > #1 0x00007f1b4941b050 in PR_WaitCondVar () from /lib64/libnspr4.so > No symbol table info available. > #2 0x00007f1b4b70efee in connection_wait_for_new_work () > No symbol table info available. > #3 0x00007f1b4b710719 in connection_threadmain () > No symbol table info available. > #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so > No symbol table info available. > #5 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 > No symbol table info available. > #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 > No symbol table info available. > > Thread 10 (Thread 0x7f1b2509d700 (LWP 105168)): > #0 0x00007f1b48dc5a82 in pthread_cond_timedwait@@GLIBC_2.3.2 () from > /lib64/libpthread.so.0 > No symbol table info available. > #1 0x00007f1b4941ab07 in pt_TimedWait () from /lib64/libnspr4.so > No symbol table info available. > #2 0x00007f1b4941afce in PR_WaitCondVar () from /lib64/libnspr4.so > No symbol table info available. > #3 0x00007f1b4b2061e8 in eq_loop () from /usr/lib64/dirsrv/libslapd.so.0 > No symbol table info available. > #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so > No symbol table info available. > #5 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 > No symbol table info available. > #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 > No symbol table info available. > > Thread 9 (Thread 0x7f1b2589e700 (LWP 105167)): > #0 0x00007f1b48dc5a82 in pthread_cond_timedwait@@GLIBC_2.3.2 () from > /lib64/libpthread.so.0 > No symbol table info available. > #1 0x00007f1b4941ab07 in pt_TimedWait () from /lib64/libnspr4.so > No symbol table info available. > #2 0x00007f1b4941afce in PR_WaitCondVar () from /lib64/libnspr4.so > No symbol table info available. > #3 0x00007f1b4b719d03 in housecleaning () > No symbol table info available. > #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so > No symbol table info available. > #5 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 > No symbol table info available. > #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 > No symbol table info available. > > Thread 8 (Thread 0x7f1b2609f700 (LWP 105166)): > #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 () from > /lib64/libpthread.so.0 > No symbol table info available. > #1 0x00007f1b4941b050 in PR_WaitCondVar () from /lib64/libnspr4.so > No symbol table info available. > #2 0x00007f1b4b260198 in slapi_wait_condvar () from > /usr/lib64/dirsrv/libslapd.so.0 > No symbol table info available. > #3 0x00007f1b3cf5deed in roles_cache_wait_on_change () from > /usr/lib64/dirsrv/plugins/libroles-plugin.so > No symbol table info available. > #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so > No symbol table info available. > #5 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 > No symbol table info available. > #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 > No symbol table info available. > > Thread 7 (Thread 0x7f1b268a0700 (LWP 105165)): > #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 () from > /lib64/libpthread.so.0 > No symbol table info available. > #1 0x00007f1b4941b050 in PR_WaitCondVar () from /lib64/libnspr4.so > No symbol table info available. > #2 0x00007f1b4b260198 in slapi_wait_condvar () from > /usr/lib64/dirsrv/libslapd.so.0 > No symbol table info available. > #3 0x00007f1b3cf5deed in roles_cache_wait_on_change () from > /usr/lib64/dirsrv/plugins/libroles-plugin.so > No symbol table info available. > #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so > No symbol table info available. > #5 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 > No symbol table info available. > #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 > No symbol table info available. > > Thread 6 (Thread 0x7f1b270a1700 (LWP 105164)): > #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 () from > /lib64/libpthread.so.0 > No symbol table info available. > #1 0x00007f1b4941b050 in PR_WaitCondVar () from /lib64/libnspr4.so > No symbol table info available. > #2 0x00007f1b4b260198 in slapi_wait_condvar () from > /usr/lib64/dirsrv/libslapd.so.0 > No symbol table info available. > #3 0x00007f1b3cf5deed in roles_cache_wait_on_change () from > /usr/lib64/dirsrv/plugins/libroles-plugin.so > No symbol table info available. > #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so > No symbol table info available. > #5 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 > No symbol table info available. > #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 > No symbol table info available. > > Thread 5 (Thread 0x7f1b278a2700 (LWP 105163)): > #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 () from > /lib64/libpthread.so.0 > No symbol table info available. > #1 0x00007f1b4941b050 in PR_WaitCondVar () from /lib64/libnspr4.so > No symbol table info available. > #2 0x00007f1b4b260198 in slapi_wait_condvar () from > /usr/lib64/dirsrv/libslapd.so.0 > No symbol table info available. > #3 0x00007f1b415f262e in cos_cache_wait_on_change () from > /usr/lib64/dirsrv/plugins/libcos-plugin.so > No symbol table info available. > #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so > No symbol table info available. > #5 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 > No symbol table info available. > #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 > No symbol table info available. > > Thread 4 (Thread 0x7f1b282ac700 (LWP 105162)): > #0 0x00007f1b48ae69b3 in select () from /lib64/libc.so.6 > No symbol table info available. > #1 0x00007f1b4b2710e9 in DS_Sleep () from /usr/lib64/dirsrv/libslapd.so.0 > No symbol table info available. > #2 0x00007f1b3e8ba707 in perf_threadmain () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > No symbol table info available. > #3 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so > No symbol table info available. > #4 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 > No symbol table info available. > #5 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 > No symbol table info available. > > Thread 3 (Thread 0x7f1b28aad700 (LWP 105161)): > #0 0x00007f1b48ae69b3 in select () from /lib64/libc.so.6 > No symbol table info available. > #1 0x00007f1b4b2710e9 in DS_Sleep () from /usr/lib64/dirsrv/libslapd.so.0 > No symbol table info available. > #2 0x00007f1b3e8bfb7f in trickle_threadmain () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > No symbol table info available. > #3 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so > No symbol table info available. > #4 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 > No symbol table info available. > #5 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 > No symbol table info available. > > Thread 2 (Thread 0x7f1b292ae700 (LWP 105160)): > #0 0x00007f1b48ae69b3 in select () from /lib64/libc.so.6 > No symbol table info available. > #1 0x00007f1b4344dbad in __os_yield () from /lib64/libdb-5.3.so > No symbol table info available. > #2 0x00007f1b434492b3 in __memp_sync_int () from /lib64/libdb-5.3.so > No symbol table info available. > #3 0x00007f1b43459752 in __txn_checkpoint () from /lib64/libdb-5.3.so > No symbol table info available. > #4 0x00007f1b43459b74 in __txn_checkpoint_pp () from /lib64/libdb-5.3.so > No symbol table info available. > #5 0x00007f1b3e8c3a87 in checkpoint_threadmain () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > No symbol table info available. > #6 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so > No symbol table info available. > #7 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 > No symbol table info available. > #8 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 > No symbol table info available. > > Thread 1 (Thread 0x7f1b29aaf700 (LWP 105159)): > #0 0x00007f1b48ae69b3 in select () from /lib64/libc.so.6 > No symbol table info available. > #1 0x00007f1b4b2710e9 in DS_Sleep () from /usr/lib64/dirsrv/libslapd.so.0 > No symbol table info available. > #2 0x00007f1b3e8bf907 in deadlock_threadmain () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > No symbol table info available. > #3 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so > No symbol table info available. > #4 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 > No symbol table info available. > #5 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 > No symbol table info available. > > > > Thanks, > > Rakesh > > > On Mon, Sep 5, 2016 at 4:48 PM, thierry bordaz wrote: > >> >> >> On 09/05/2016 12:05 PM, Rakesh Rajasekharan wrote: >> >> Hi Thierry, >> >> >> I was getting the hang issue while running ipa-client-install >> simultaneously on few clients.. >> However, today, I am not able to replicate that. >> >> I could not get a gdb . But i will try getting that the next time I face >> this issue. >> >> The CPU does not stay high.. it just momentarily touches a high value and >> then drops down to around 2-7% >> >> One question I have is , is it ok to set it nsslapd-threadnumber to a >> very high value . >> I have around 4000 clients and with nsslapd-maxthreadsperconn set to >> 5..So, can I set nsslapd-threadnumber to around 25000. >> >> >> Hello, >> >> I know some users running in production with several hunderds of threads >> (>600) and this without problem. >> >> I do not recall having suggested to increase that number and for what >> reason. >> Usually 30 workers is a good enough value. It can create bootleneck if >> for some reason each operation is very long to satisfy and exhaust the >> number of workers. You can monitor the work queue: >> >> ldapsearch -D "cn=directory manager" -w xxx -LLL -b "cn=monitor" -s base >> opsinitiated opscompleted >> >> >> If opscompleted-opsinitiated remains close to threadnumber, then yes it >> would be valuable to increase it. >> >> The computation #client * #async_op_per_client sound an overkill. Even if >> all clients send at the exact same time all their requests, it is very >> likely that some common resource (db page, log, allocator...) will >> serialize them. If you monitor a need to increase the work, you would for >> example set it to 50, then monitor, then set it to 100, then monitor... >> until you find a good enough value. >> Note the increasing the #thread, increases the memory footprint that will >> reduce the efficiency of file system cache and can increase the response >> time. >> >> >> best regards >> thierry >> >> >> Thanks >> >> On Mon, Sep 5, 2016 at 1:03 PM, thierry bordaz >> wrote: >> >>> >>> Hi Rakesh, >>> >>> Were you able to get a pstack or full stack with gdb ( >>> http://www.port389.org/docs/389ds/FAQ/faq.html#debugging-crashes) when >>> the server hangs ? >>> >>> If it happens with 500 threads as well as with 30, using 30 threads is a >>> better choice to debug this issue. >>> I will try to reproduce using 150 parallel 'ipa user-find p-testipa' >>> commands >>> >>> Something I am unsure is if the CPU consumption stays high (you >>> mentioned 340% CPU usage) as long as the hang happens or if after a suddent >>> shot up to 340% (that marks the beginning of the hang) it drops and stay >>> hanging ? >>> >>> thanks >>> thierry >>> >>> On 09/04/2016 08:40 PM, Rakesh Rajasekharan wrote: >>> >>> starce on the slapd process actually had this in the output.. >>> FUTEX_WAIT_PRIVATE >>> >>> and checking for the number of threads slapd had.. there were 5015 >>> threads >>> >>> ps -efL|grep slapd|wc -l >>> 5015 >>> >>> strace on most of the threads gave this output >>> >>> strace -p 67411 >>> Process 67411 attached >>> futex(0x7f3f0226b41c, FUTEX_WAIT_PRIVATE, 1, NULL) = -1 EAGAIN (Resource >>> temporarily unavailable) >>> futex(0x7f3f0226b41c, FUTEX_WAIT_PRIVATE, 2, NULL^CProcess 67411 detached >>> >>> >>> >>> >>> >>> On Sun, Sep 4, 2016 at 5:34 PM, Rakesh Rajasekharan < >>> rakesh.rajasekharan at gmail.com> wrote: >>> >>>> I have again got the issue of IPA hanging.. The issue came up when i >>>> tried to run ipa-client-isntall on 142 clients simultaneously >>>> >>>> >>>> None of the IPA commands are responding, and I see this error >>>> >>>> ipa user-find p-testipa >>>> ipa: ERROR: Insufficient access: SASL(-1): generic failure: GSSAPI >>>> Error: Unspecified GSS failure. Minor code may provide more information >>>> (KDC returned error string: PROCESS_TGS) >>>> >>>> KRB5_TRACE=/dev/stdout kinit admin >>>> [41178] 1472984115.233214: Getting initial credentials for >>>> admin at XYZ.COM >>>> [41178] 1472984115.235257: Sending request (167 bytes) to XYZ.COM >>>> [41178] 1472984115.235419: Initiating TCP connection to stream >>>> 10.1.3.36:88 >>>> [41178] 1472984115.235685: Sending TCP request to stream 10.1.3.36:88 >>>> [41178] 1472984120.238914: Received answer (174 bytes) from stream >>>> 10.1.3.36:88 >>>> [41178] 1472984120.238925: Terminating TCP connection to stream >>>> 10.1.3.36:88 >>>> [41178] 1472984120.238993: Response was from master KDC >>>> [41 >>>> >>>> >>>> Running an ldapsearch to see the db.. does not give any results and >>>> just hangs there >>>> >>>> ldapsearch -x -D 'cn=Directory Manager' -W -s one -b >>>> 'cn=kerberos,dc=xyz,dc=com' >>>> Enter LDAP Password: >>>> >>>> even an ldapsearch -x does not respond >>>> At this point, am sure that slapd is the one causing issues >>>> >>>> Running an strace against the hung slapd itself seems to get stuck does >>>> not proceed after saying "attaching to process" >>>> >>>> From some others posts I read Thierry suggesting to increase the >>>> nsslapd-threadnumber value >>>> >>>> It was set to 30, I think that might be too low. >>>> >>> ... > > [Message clipped] -------------- next part -------------- An HTML attachment was scrubbed... URL: From orion at cora.nwra.com Tue Sep 6 02:27:03 2016 From: orion at cora.nwra.com (Orion Poplawski) Date: Mon, 5 Sep 2016 20:27:03 -0600 Subject: [Freeipa-users] Default gid for AD trust users In-Reply-To: <20160902211536.GA29320@10.4.128.1> References: <20160902211536.GA29320@10.4.128.1> Message-ID: <5d362edf-6126-e63b-813f-a6fc90776f27@cora.nwra.com> On 09/02/2016 03:15 PM, Lukas Slebodnik wrote: > On (24/08/16 11:42), Orion Poplawski wrote: >> While that is definitely *a* convention, it's not the one we've used which >> puts users by default in shared groups (nwra, visitors, etc). For example: >> >> uid=2941(user) gid=1991(nwra) >> > The user "user" should be a member "nwra" group. > If no then you have other issues. > > Why does it matter whether it is a primary group or no? > > LS > Because that is the default group ownership of files created by the user. Yes, they can change it, and yes you can use setgid directories, but it is the default. -- Orion Poplawski Technical Manager 303-415-9701 x222 NWRA/CoRA Division FAX: 303-415-9702 3380 Mitchell Lane orion at cora.nwra.com Boulder, CO 80301 http://www.cora.nwra.com From jrichard at placeiq.com Tue Sep 6 05:02:34 2016 From: jrichard at placeiq.com (Jim Richard) Date: Tue, 6 Sep 2016 01:02:34 -0400 Subject: [Freeipa-users] Active directory trust and SSH Message-ID: <580F08AF-318A-460A-8441-89BAD1F0196B@placeiq.com> So I have two-way trust setup and it seems to work. And as described here: https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/7/html/Windows_Integration_Guide/trust-ssh.html SSSD allows user names in the format user at AD.DOMAIN, ad.domain\user and AD\user That works just as described. I have two domains/realms - idm.placeiq.net and idm-ad.placeiq.net, the second being the Active Directory domain. My desire is to have AD be the source for all user/authentication - the AD users will use their creds to ssh in to all of the Centos hosts in the idm.placeiq.net domain. The hosts that live in IDM are a combination of Centos 6.8 and 7.X hosts. How can I make it so a user does not have to: ssh 'IDM-AD\Administrator?@hostname or ssh Administrator at idm-ad.placeiq.net@hostname Instead when I say Administrator at hostname it auto-magically knows I mean "ssh Administrator at idm-ad.placeiq.net@10.1.41.202 I?ve tried modifiying krb5.conf as such but it seems like I?m missing a step. [libdefaults] #default_realm = IDM.PLACEIQ.NET default_realm = IDM-AD.PLACEIQ.NET I think my clients use the localauth plugin but I?m not entirely sure. If so, how can I configure its behavior? Jim Richard SYSTEM ADMINISTRATOR III (646) 338-8905 -------------- next part -------------- An HTML attachment was scrubbed... URL: From tkrizek at redhat.com Tue Sep 6 06:30:31 2016 From: tkrizek at redhat.com (Tomas Krizek) Date: Tue, 6 Sep 2016 08:30:31 +0200 Subject: [Freeipa-users] Active directory trust and SSH In-Reply-To: <580F08AF-318A-460A-8441-89BAD1F0196B@placeiq.com> References: <580F08AF-318A-460A-8441-89BAD1F0196B@placeiq.com> Message-ID: <16bef410-e295-a593-4a16-9290f5308fb6@redhat.com> On 09/06/2016 07:02 AM, Jim Richard wrote: > So I have two-way trust setup and it seems to work. > > And as described here: > https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/7/html/Windows_Integration_Guide/trust-ssh.html > > SSSD allows user names in the > format user at AD.DOMAIN, ad.domain\user and AD\user > > That works just as described. > > I have two domains/realms - idm.placeiq.net and idm-ad.placeiq.net, > the second being the Active Directory domain. > > My desire is to have AD be the source for all user/authentication - > the AD users will use their creds to ssh in to all of the Centos hosts > in the idm.placeiq.net domain. > > The hosts that live in IDM are a combination of Centos 6.8 and 7.X hosts. > > How can I make it so a user does not have to: > > ssh 'IDM-AD\Administrator?@hostname or ssh > Administrator at idm-ad.placeiq.net@hostname > > Instead when I say Administrator at hostname it auto-magically knows I > mean "ssh Administrator at idm-ad.placeiq.net@10.1.41.202 > > I?ve tried modifiying krb5.conf as such but it seems like I?m missing > a step. > > [libdefaults] > #default_realm = IDM.PLACEIQ.NET > default_realm = IDM-AD.PLACEIQ.NET > > > I think my clients use the localauth plugin but I?m not entirely sure. > If so, how can I configure its behavior? > > > > > Jim Richard > SYSTEM ADMINISTRATOR III > (646) 338-8905 > > PlaceIQ:Location Data Accuracy > > > > > I don't think what you're asking for is possible to do as a FreeIPA configuration. The documentation describes how to login without prompting for passwords, but I think it is still necessary to provide the username with AD realm when logging in. If you're always logging in as the same user to certain machines, you could configure a default user in the ssh_config. Perhaps someone else will have a better answer. -- Tomas Krizek -------------- next part -------------- An HTML attachment was scrubbed... URL: From jhrozek at redhat.com Tue Sep 6 06:57:06 2016 From: jhrozek at redhat.com (Jakub Hrozek) Date: Tue, 6 Sep 2016 08:57:06 +0200 Subject: [Freeipa-users] Active directory trust and SSH In-Reply-To: <580F08AF-318A-460A-8441-89BAD1F0196B@placeiq.com> References: <580F08AF-318A-460A-8441-89BAD1F0196B@placeiq.com> Message-ID: <20160906065706.p5begrxlye6zurox@hendrix> On Tue, Sep 06, 2016 at 01:02:34AM -0400, Jim Richard wrote: > So I have two-way trust setup and it seems to work. > > And as described here: https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/7/html/Windows_Integration_Guide/trust-ssh.html > > SSSD allows user names in the format user at AD.DOMAIN, ad.domain\user and AD\user > > That works just as described. > > I have two domains/realms - idm.placeiq.net and idm-ad.placeiq.net, the second being the Active Directory domain. > > My desire is to have AD be the source for all user/authentication - the AD users will use their creds to ssh in to all of the Centos hosts in the idm.placeiq.net domain. > > The hosts that live in IDM are a combination of Centos 6.8 and 7.X hosts. > > How can I make it so a user does not have to: > > ssh 'IDM-AD\Administrator?@hostname or ssh Administrator at idm-ad.placeiq.net@hostname > > Instead when I say Administrator at hostname it auto-magically knows I mean "ssh Administrator at idm-ad.placeiq.net@10.1.41.202 > > I?ve tried modifiying krb5.conf as such but it seems like I?m missing a step. > > [libdefaults] > #default_realm = IDM.PLACEIQ.NET > default_realm = IDM-AD.PLACEIQ.NET > > > I think my clients use the localauth plugin but I?m not entirely sure. If so, how can I configure its behavior? Put: default_domain_suffix = AD.DOMAIN into the [sssd] section of your sssd.conf. This setting auto-qualifies any user or group queries unless you qualify them yourself (so you need to qualify any IPA user/group lookups..). From rakesh.rajasekharan at gmail.com Tue Sep 6 08:59:28 2016 From: rakesh.rajasekharan at gmail.com (Rakesh Rajasekharan) Date: Tue, 6 Sep 2016 14:29:28 +0530 Subject: [Freeipa-users] Freeipa 4.2.0 hangs intermittently In-Reply-To: References: <57BEB0AB.2090506@redhat.com> <57C44AC6.1030402@redhat.com> <57CD1FDC.2010109@redhat.com> <57CD5488.60404@redhat.com> <57CD659B.6050707@redhat.com> <57CD68BD.9030802@redhat.com> Message-ID: Hi Thierry/Ludwig, I again hit the hang issue. This is the pstack of the ns-slapd So was able to this time get the pstack and db locks db locks details Default locking region information: 451 Last allocated locker ID 0x7fffffff Current maximum unused locker ID 9 Number of lock modes 800 Initial number of locks allocated 0 Initial number of lockers allocated 800 Initial number of lock objects allocated 10000 Maximum number of locks possible 10000 Maximum number of lockers possible 10000 Maximum number of lock objects possible 800 Current number of locks allocated 366 Current number of lockers allocated 800 Current number of lock objects allocated 160 Number of lock object partitions 8191 Size of object hash table 126 Number of current locks 1005 Maximum number of locks at any one time 33 Maximum number of locks in any one bucket 3987 Maximum number of locks stolen by for an empty partition 75 Maximum number of locks stolen for any one partition 309 Number of current lockers 313 Maximum number of lockers at any one time 121 Number of current lock objects 676 Maximum number of lock objects at any one time 3 Maximum number of lock objects in any one bucket 69 Maximum number of objects stolen by for an empty partition 5 Maximum number of objects stolen for any one partition 22M Total number of locks requested (22009642) 21M Total number of locks released (21992878) 0 Total number of locks upgraded 95 Total number of locks downgraded 3348 Lock requests not available due to conflicts, for which we waited 12646 Lock requests not available due to conflicts, for which we did not wait 0 Number of deadlocks 0 Lock timeout value 0 Number of locks that have timed out 0 Transaction timeout value 0 Number of transactions that have timed out 2MB 560KB Region size 71719 The number of partition locks that required waiting (0%) 23105 The maximum number of times any partition lock was waited for (0%) 0 The number of object queue operations that required waiting (0%) 163 The number of locker allocations that required waiting (0%) 3172 The number of region locks that required waiting (0%) 5 Maximum hash bucket length =-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-= Lock REGINFO information: Environment Region type 1 Region ID /var/lib/dirsrv/slapd-SPRINKLR-COM/db/__db.001 Region name 0x7f67ef869000 Region address 0x7f67ef8690a0 Region allocation head 0x7f67ef8712b0 Region primary address 0 Region maximum allocation 0 Region allocated Region allocations: 1581345 allocations, 0 failures, 1580794 frees, 6 longest Allocations by power-of-two sizes: 1KB 1581329 2KB 3 4KB 4 8KB 3 16KB 3 32KB 1 64KB 0 128KB 0 256KB 2 512KB 0 1024KB 1 REGION_SHARED Region flags =-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-= Lock region parameters: 2 Lock region region mutex [3172/14M 0% !Own] 16381 locker table size 8191 object table size 34128 obj_off 900216 locker_off 0 need_dd =-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-= Lock conflict matrix: =-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-= Locks grouped by lockers: Locker Mode Count Status ----------------- Object --------------- 2 dd=308 locks held 1 write locks 0 pid/thread 4845/140085720139840 flags 10 priority 100 2 READ 1 HELD userRoot/id2entry.db handle 0 3 dd=307 locks held 0 write locks 0 pid/thread 4845/140084829472512 flags 0 priority 100 4 dd=306 locks held 1 write locks 0 pid/thread 4845/140085720139840 flags 10 priority 100 4 READ 1 HELD ipaca/id2entry.db handle 0 5 dd=305 locks held 0 write locks 0 pid/thread 4845/140084837865216 flags 0 priority 100 6 dd=304 locks held 1 write locks 0 pid/thread 4845/140085720139840 flags 10 priority 100 6 READ 1 HELD ipaca/entryrdn.db handle 0 7 dd=303 locks held 0 write locks 0 pid/thread 4845/140084804294400 flags 0 priority 100 8 dd=302 locks held 0 write locks 0 pid/thread 4845/140084720432896 flags 0 priority 100 9 dd=301 locks held 0 write locks 0 pid/thread 4845/140084837865216 flags 0 priority 100 a dd=300 locks held 1 write locks 0 pid/thread 4845/140085720139840 flags 10 priority 100 a READ 1 HELD ipaca/vlv#allcertspkitomcatindex.db handle 0 e dd=299 locks held 1 write locks 0 pid/thread 4845/140085720139840 flags 10 priority 100 e READ 1 HELD ipaca/vlv#allnonrevokedcertspkitomcatindex.db handle 0 10 dd=298 locks held 1 write locks 0 pid/thread 4845/140085720139840 flags 10 priority 100 10 READ 1 HELD ipaca/vlv#allrevokedcertspkitomcatindex.db handle 0 11 dd=297 locks held 1 write locks 0 pid/thread 4845/140085720139840 flags 10 priority 100 11 READ 1 HELD ipaca/vlv#allrevokedcertsnotafterpkitomcatindex.db handle 0 14 dd=296 locks held 1 write locks 0 pid/thread 4845/140085720139840 flags 10 priority 100 14 READ 1 HELD ipaca/vlv#allrevokedorrevokedexpiredcertspkitomcatindex.db handle 0 15 dd=295 locks held 1 write locks 0 pid/thread 4845/140085720139840 flags 10 priority 100 15 READ 1 HELD ipaca/vlv#allvalidcertspkitomcatindex.db handle 0 16 dd=294 locks held 1 write locks 0 pid/thread 4845/140085720139840 flags 10 priority 100 16 READ 1 HELD ipaca/vlv#allvalidcertsnotafterpkitomcatindex.db handle 0 17 dd=293 locks held 1 write locks 0 pid/thread 4845/140085720139840 flags 10 priority 100 17 READ 1 HELD ipaca/vlv#allvalidorrevokedcertspkitomcatindex.db handle 0 18 dd=292 locks held 1 write locks 0 pid/thread 4845/140085720139840 flags 10 priority 100 18 READ 1 HELD ipaca/vlv#caallpkitomcatindex.db handle 0 1d dd=291 locks held 1 write locks 0 pid/thread 4845/140085720139840 flags 10 priority 100 1d READ 1 HELD ipaca/vlv#cacompletepkitomcatindex.db handle 0 1e dd=290 locks held 1 write locks 0 pid/thread 4845/140085720139840 flags 10 priority 100 1e READ 1 HELD ipaca/vlv#cacompleteenrollmentpkitomcatindex.db handle 0 20 dd=289 locks held 1 write locks 0 pid/thread 4845/140085720139840 flags 10 priority 100 20 READ 1 HELD ipaca/vlv#cacompleterevocationpkitomcatindex.db handle 0 21 dd=288 locks held 1 write locks 0 pid/thread 4845/140085720139840 flags 10 priority 100 21 READ 1 HELD ipaca/vlv#caenrollmentpkitomcatindex.db handle 0 2b dd=287 locks held 1 write locks 0 pid/thread 4845/140085720139840 flags 10 priority 100 2b READ 1 HELD ipaca/vlv#carevocationpkitomcatindex.db handle 0 2c dd=286 locks held 1 write locks 0 pid/thread 4845/140085720139840 flags 10 priority 100 2c READ 1 HELD changelog/id2entry.db handle 0 2d dd=285 locks held 0 write locks 0 pid/thread 4845/140085719607040 flags 0 priority 100 2e dd=284 locks held 1 write locks 0 pid/thread 4845/140085720139840 flags 10 priority 100 2e READ 1 HELD changelog/entryusn.db handle 0 2f dd=283 locks held 0 write locks 0 pid/thread 4845/140085720139840 flags 0 priority 100 30 dd=282 locks held 1 write locks 0 pid/thread 4845/140085720139840 flags 10 priority 100 30 READ 1 HELD userRoot/entryusn.db handle 0 31 dd=281 locks held 0 write locks 0 pid/thread 4845/140085720139840 flags 0 priority 100 32 dd=280 locks held 1 write locks 0 pid/thread 4845/140085720139840 flags 10 priority 100 32 READ 1 HELD ipaca/entryusn.db handle 0 33 dd=279 locks held 0 write locks 0 pid/thread 4845/140085720139840 flags 0 priority 100 34 dd=278 locks held 1 write locks 0 pid/thread 4845/140085720139840 flags 10 priority 100 34 READ 1 HELD userRoot/entryrdn.db handle 0 35 dd=277 locks held 0 write locks 0 pid/thread 4845/140084544186112 flags 0 priority 100 36 dd=276 locks held 0 write locks 0 pid/thread 4845/140084636505856 flags 0 priority 100 37 dd=275 locks held 0 write locks 0 pid/thread 4845/140084854650624 flags 0 priority 100 38 dd=274 locks held 1 write locks 0 pid/thread 4845/140085720139840 flags 10 priority 100 38 READ 1 HELD userRoot/objectclass.db handle 0 39 dd=273 locks held 0 write locks 0 pid/thread 4845/140084569364224 flags 0 priority 100 3a dd=272 locks held 1 write locks 0 pid/thread 4845/140085720139840 flags 10 priority 100 3a READ 1 HELD userRoot/ancestorid.db handle 0 3b dd=271 locks held 0 write locks 0 pid/thread 4845/140084854650624 flags 0 priority 100 3c dd=270 locks held 1 write locks 0 pid/thread 4845/140085720139840 flags 10 priority 100 3c READ 1 HELD changelog/entryrdn.db handle 0 3d dd=269 locks held 0 write locks 0 pid/thread 4845/140085719607040 flags 0 priority 100 3e dd=268 locks held 0 write locks 0 pid/thread 4845/140084602935040 flags 0 priority 100 3f dd=267 locks held 0 write locks 0 pid/thread 4845/140085719607040 flags 0 priority 100 40 dd=266 locks held 1 write locks 0 pid/thread 4845/140085720139840 flags 10 priority 100 40 READ 1 HELD changelog/objectclass.db handle 0 41 dd=265 locks held 0 write locks 0 pid/thread 4845/140085719607040 flags 0 priority 100 42 dd=264 locks held 0 write locks 0 pid/thread 4845/140085719607040 flags 0 priority 100 43 dd=263 locks held 0 write locks 0 pid/thread 4845/140084653291264 flags 0 priority 100 44 dd=262 locks held 1 write locks 0 pid/thread 4845/140085720139840 flags 10 priority 100 44 READ 1 HELD ipaca/objectclass.db handle 0 45 dd=261 locks held 0 write locks 0 pid/thread 4845/140084577756928 flags 0 priority 100 46 dd=260 locks held 0 write locks 0 pid/thread 4845/140084720432896 flags 0 priority 100 47 dd=259 locks held 1 write locks 0 pid/thread 4845/140085720139840 flags 10 priority 100 47 READ 1 HELD changelog/aci.db handle 0 48 dd=258 locks held 0 write locks 0 pid/thread 4845/140085720139840 flags 0 priority 100 49 dd=257 locks held 1 write locks 0 pid/thread 4845/140085720139840 flags 10 priority 100 49 READ 1 HELD userRoot/aci.db handle 0 4a dd=256 locks held 0 write locks 0 pid/thread 4845/140085720139840 flags 0 priority 100 4b dd=255 locks held 0 write locks 0 pid/thread 4845/140084535793408 flags 0 priority 100 4c dd=254 locks held 1 write locks 0 pid/thread 4845/140085720139840 flags 10 priority 100 4c READ 1 HELD ipaca/aci.db handle 0 4d dd=253 locks held 0 write locks 0 pid/thread 4845/140085720139840 flags 0 priority 100 4e dd=252 locks held 1 write locks 0 pid/thread 4845/140085720139840 flags 10 priority 100 4e READ 1 HELD userRoot/parentid.db handle 0 4f dd=251 locks held 0 write locks 0 pid/thread 4845/140085720139840 flags 0 priority 100 50 dd=250 locks held 0 write locks 0 pid/thread 4845/140085719607040 flags 0 priority 100 51 dd=249 locks held 1 write locks 0 pid/thread 4845/140085720139840 flags 10 priority 100 51 READ 1 HELD changelog/nsuniqueid.db handle 0 52 dd=248 locks held 1 write locks 0 pid/thread 4845/140085720139840 flags 10 priority 100 52 READ 1 HELD changelog/changenumber.db handle 0 53 dd=247 locks held 0 write locks 0 pid/thread 4845/140084602935040 flags 0 priority 100 54 dd=246 locks held 0 write locks 0 pid/thread 4845/140085719607040 flags 0 priority 100 55 dd=245 locks held 1 write locks 0 pid/thread 4845/140085719607040 flags 10 priority 100 55 READ 1 HELD changelog/targetuniqueid.db handle 0 56 dd=244 locks held 1 write locks 0 pid/thread 4845/140085719607040 flags 10 priority 100 56 READ 1 HELD changelog/parentid.db handle 0 57 dd=243 locks held 1 write locks 0 pid/thread 4845/140085719607040 flags 10 priority 100 57 READ 1 HELD changelog/ancestorid.db handle 0 58 dd=242 locks held 1 write locks 0 pid/thread 4845/140085719607040 flags 10 priority 100 58 READ 1 HELD changelog/numsubordinates.db handle 0 59 dd=241 locks held 0 write locks 0 pid/thread 4845/140085719607040 flags 0 priority 100 5a dd=240 locks held 0 write locks 0 pid/thread 4845/140084544186112 flags 0 priority 100 5b dd=239 locks held 1 write locks 0 pid/thread 4845/140084577756928 flags 10 priority 100 5b READ 1 HELD userRoot/krbPrincipalName.db handle 0 5c dd=238 locks held 0 write locks 0 pid/thread 4845/140084636505856 flags 0 priority 100 5d dd=237 locks held 0 write locks 0 pid/thread 4845/140084695254784 flags 0 priority 100 5e dd=236 locks held 1 write locks 0 pid/thread 4845/140084560971520 flags 10 priority 100 5e READ 1 HELD userRoot/ipakrbprincipalalias.db handle 0 5f dd=235 locks held 0 write locks 0 pid/thread 4845/140084703647488 flags 0 priority 100 60 dd=234 locks held 0 write locks 0 pid/thread 4845/140084653291264 flags 0 priority 100 61 dd=233 locks held 0 write locks 0 pid/thread 4845/140084577756928 flags 0 priority 100 62 dd=232 locks held 0 write locks 0 pid/thread 4845/140084686862080 flags 0 priority 100 63 dd=231 locks held 1 write locks 0 pid/thread 4845/140084611327744 flags 10 priority 100 63 READ 1 HELD changelog/seeAlso.db handle 0 64 dd=230 locks held 0 write locks 0 pid/thread 4845/140084686862080 flags 0 priority 100 65 dd=229 locks held 1 write locks 0 pid/thread 4845/140084611327744 flags 10 priority 100 65 READ 1 HELD userRoot/seeAlso.db handle 0 66 dd=228 locks held 0 write locks 0 pid/thread 4845/140084636505856 flags 0 priority 100 67 dd=227 locks held 1 write locks 0 pid/thread 4845/140084611327744 flags 10 priority 100 67 READ 1 HELD ipaca/seeAlso.db handle 0 68 dd=226 locks held 0 write locks 0 pid/thread 4845/140084686862080 flags 0 priority 100 69 dd=225 locks held 1 write locks 0 pid/thread 4845/140084678469376 flags 10 priority 100 69 READ 1 HELD ipaca/parentid.db handle 0 6a dd=224 locks held 0 write locks 0 pid/thread 4845/140084678469376 flags 0 priority 100 6b dd=223 locks held 0 write locks 0 pid/thread 4845/140084829472512 flags 0 priority 100 6c dd=222 locks held 0 write locks 0 pid/thread 4845/140084653291264 flags 0 priority 100 6d dd=221 locks held 0 write locks 0 pid/thread 4845/140084569364224 flags 0 priority 100 6e dd=220 locks held 0 write locks 0 pid/thread 4845/140084636505856 flags 0 priority 100 6e READ 1 WAIT userRoot/objectclass.db page 18 6f dd=219 locks held 0 write locks 0 pid/thread 4845/140084544186112 flags 0 priority 100 70 dd=218 locks held 0 write locks 0 pid/thread 4845/140084569364224 flags 0 priority 100 71 dd=217 locks held 0 write locks 0 pid/thread 4845/140084619720448 flags 0 priority 100 72 dd=216 locks held 0 write locks 0 pid/thread 4845/140084552578816 flags 0 priority 100 73 dd=215 locks held 0 write locks 0 pid/thread 4845/140084829472512 flags 0 priority 100 74 dd=214 locks held 0 write locks 0 pid/thread 4845/140084628113152 flags 0 priority 100 75 dd=213 locks held 0 write locks 0 pid/thread 4845/140084686862080 flags 0 priority 100 76 dd=212 locks held 1 write locks 0 pid/thread 4845/140084560971520 flags 10 priority 100 76 READ 1 HELD userRoot/ipauniqueid.db handle 0 77 dd=211 locks held 0 write locks 0 pid/thread 4845/140084661683968 flags 0 priority 100 78 dd=210 locks held 0 write locks 0 pid/thread 4845/140084846257920 flags 0 priority 100 79 dd=209 locks held 0 write locks 0 pid/thread 4845/140084628113152 flags 0 priority 100 7a dd=208 locks held 1 write locks 0 pid/thread 4845/140084560971520 flags 10 priority 100 7a READ 1 HELD userRoot/nsuniqueid.db handle 0 7b dd=207 locks held 1 write locks 0 pid/thread 4845/140084560971520 flags 10 priority 100 7b READ 1 HELD userRoot/cn.db handle 0 7c dd=206 locks held 1 write locks 0 pid/thread 4845/140084560971520 flags 10 priority 100 7c READ 1 HELD userRoot/fqdn.db handle 0 7d dd=205 locks held 1 write locks 0 pid/thread 4845/140084560971520 flags 10 priority 100 7d READ 1 HELD userRoot/managedby.db handle 0 7e dd=204 locks held 1 write locks 0 pid/thread 4845/140084560971520 flags 10 priority 100 7e READ 1 HELD userRoot/numsubordinates.db handle 0 7f dd=203 locks held 1 write locks 0 pid/thread 4845/140084678469376 flags 10 priority 100 7f READ 1 HELD userRoot/member.db handle 0 80 dd=202 locks held 0 write locks 0 pid/thread 4845/140084712040192 flags 0 priority 100 81 dd=201 locks held 1 write locks 0 pid/thread 4845/140084678469376 flags 10 priority 100 81 READ 1 HELD userRoot/memberUser.db handle 0 82 dd=200 locks held 0 write locks 0 pid/thread 4845/140084577756928 flags 0 priority 100 83 dd=199 locks held 1 write locks 0 pid/thread 4845/140084678469376 flags 10 priority 100 83 READ 1 HELD userRoot/memberHost.db handle 0 84 dd=198 locks held 0 write locks 0 pid/thread 4845/140084712040192 flags 0 priority 100 85 dd=197 locks held 0 write locks 0 pid/thread 4845/140084577756928 flags 0 priority 100 86 dd=196 locks held 0 write locks 0 pid/thread 4845/140084670076672 flags 0 priority 100 87 dd=195 locks held 0 write locks 0 pid/thread 4845/140084577756928 flags 0 priority 100 88 dd=194 locks held 0 write locks 0 pid/thread 4845/140084846257920 flags 0 priority 100 89 dd=193 locks held 0 write locks 0 pid/thread 4845/140084552578816 flags 0 priority 100 8a dd=192 locks held 0 write locks 0 pid/thread 4845/140084703647488 flags 0 priority 100 8b dd=191 locks held 0 write locks 0 pid/thread 4845/140084544186112 flags 0 priority 100 8c dd=190 locks held 0 write locks 0 pid/thread 4845/140084846257920 flags 0 priority 100 8d dd=189 locks held 0 write locks 0 pid/thread 4845/140084661683968 flags 0 priority 100 8e dd=188 locks held 1 write locks 0 pid/thread 4845/140085067675392 flags 10 priority 100 8e READ 1 HELD userRoot/macAddress.db handle 0 8f dd=187 locks held 0 write locks 0 pid/thread 4845/140085067675392 flags 0 priority 100 90 dd=186 locks held 0 write locks 0 pid/thread 4845/140084703647488 flags 0 priority 100 91 dd=185 locks held 0 write locks 0 pid/thread 4845/140084552578816 flags 0 priority 100 92 dd=184 locks held 0 write locks 0 pid/thread 4845/140084560971520 flags 0 priority 100 92 READ 1 WAIT userRoot/objectclass.db page 9 93 dd=183 locks held 0 write locks 0 pid/thread 4845/140084577756928 flags 0 priority 100 94 dd=182 locks held 0 write locks 0 pid/thread 4845/140084821079808 flags 0 priority 100 94 READ 1 WAIT userRoot/objectclass.db page 18 95 dd=181 locks held 0 write locks 0 pid/thread 4845/140084544186112 flags 0 priority 100 96 dd=180 locks held 0 write locks 0 pid/thread 4845/140084804294400 flags 0 priority 100 97 dd=179 locks held 1 write locks 0 pid/thread 4845/140084712040192 flags 10 priority 100 97 READ 1 HELD userRoot/uid.db handle 0 98 dd=178 locks held 0 write locks 0 pid/thread 4845/140084552578816 flags 0 priority 100 99 dd=177 locks held 0 write locks 0 pid/thread 4845/140084577756928 flags 0 priority 100 9a dd=176 locks held 0 write locks 0 pid/thread 4845/140084602935040 flags 0 priority 100 9b dd=175 locks held 1 write locks 0 pid/thread 4845/140085076068096 flags 10 priority 100 9b READ 1 HELD userRoot/uniquemember.db handle 0 9c dd=174 locks held 1 write locks 0 pid/thread 4845/140085076068096 flags 10 priority 100 9c READ 1 HELD userRoot/owner.db handle 0 9d dd=173 locks held 1 write locks 0 pid/thread 4845/140085076068096 flags 10 priority 100 9d READ 1 HELD userRoot/ipaallowedtarget.db handle 0 9e dd=172 locks held 1 write locks 0 pid/thread 4845/140085076068096 flags 10 priority 100 9e READ 1 HELD userRoot/ipaMemberCertProfile.db handle 0 9f dd=171 locks held 1 write locks 0 pid/thread 4845/140085076068096 flags 10 priority 100 9f READ 1 HELD userRoot/ipaassignedidview.db handle 0 a0 dd=170 locks held 1 write locks 0 pid/thread 4845/140085076068096 flags 10 priority 100 a0 READ 1 HELD userRoot/secretary.db handle 0 a1 dd=169 locks held 1 write locks 0 pid/thread 4845/140085076068096 flags 10 priority 100 a1 READ 1 HELD userRoot/memberdenycmd.db handle 0 a2 dd=168 locks held 1 write locks 0 pid/thread 4845/140085076068096 flags 10 priority 100 a2 READ 1 HELD userRoot/memberallowcmd.db handle 0 a3 dd=167 locks held 1 write locks 0 pid/thread 4845/140085076068096 flags 10 priority 100 a3 READ 1 HELD userRoot/manager.db handle 0 a4 dd=166 locks held 1 write locks 0 pid/thread 4845/140085076068096 flags 10 priority 100 a4 READ 1 HELD userRoot/ipasudorunas.db handle 0 a5 dd=165 locks held 1 write locks 0 pid/thread 4845/140085076068096 flags 10 priority 100 a5 READ 1 HELD userRoot/ipaMemberCa.db handle 0 a6 dd=164 locks held 1 write locks 0 pid/thread 4845/140085076068096 flags 10 priority 100 a6 READ 1 HELD userRoot/ipasudorunasgroup.db handle 0 a7 dd=163 locks held 1 write locks 0 pid/thread 4845/140085076068096 flags 10 priority 100 a7 READ 1 HELD userRoot/sourcehost.db handle 0 a8 dd=162 locks held 1 write locks 0 pid/thread 4845/140085076068096 flags 10 priority 100 a8 READ 1 HELD userRoot/memberservice.db handle 0 a9 dd=161 locks held 1 write locks 0 pid/thread 4845/140085076068096 flags 10 priority 100 a9 READ 1 HELD userRoot/ipatokenradiusconfiglink.db handle 0 aa dd=160 locks held 0 write locks 0 pid/thread 4845/140084703647488 flags 0 priority 100 ab dd=159 locks held 0 write locks 0 pid/thread 4845/140084821079808 flags 0 priority 100 ac dd=158 locks held 0 write locks 0 pid/thread 4845/140084628113152 flags 0 priority 100 ad dd=157 locks held 0 write locks 0 pid/thread 4845/140084829472512 flags 0 priority 100 ae dd=156 locks held 0 write locks 0 pid/thread 4845/140084829472512 flags 0 priority 100 af dd=155 locks held 0 write locks 0 pid/thread 4845/140084854650624 flags 0 priority 100 b0 dd=154 locks held 0 write locks 0 pid/thread 4845/140084854650624 flags 0 priority 100 b1 dd=153 locks held 0 write locks 0 pid/thread 4845/140084837865216 flags 0 priority 100 b2 dd=152 locks held 0 write locks 0 pid/thread 4845/140084804294400 flags 0 priority 100 b3 dd=151 locks held 0 write locks 0 pid/thread 4845/140084812687104 flags 0 priority 100 b4 dd=150 locks held 0 write locks 0 pid/thread 4845/140084804294400 flags 0 priority 100 b5 dd=149 locks held 0 write locks 0 pid/thread 4845/140084837865216 flags 0 priority 100 b6 dd=148 locks held 0 write locks 0 pid/thread 4845/140084569364224 flags 0 priority 100 b7 dd=147 locks held 0 write locks 0 pid/thread 4845/140084829472512 flags 0 priority 100 b8 dd=146 locks held 0 write locks 0 pid/thread 4845/140084586149632 flags 0 priority 100 b9 dd=145 locks held 0 write locks 0 pid/thread 4845/140084653291264 flags 0 priority 100 ba dd=144 locks held 0 write locks 0 pid/thread 4845/140084636505856 flags 0 priority 100 bb dd=143 locks held 0 write locks 0 pid/thread 4845/140084804294400 flags 0 priority 100 bc dd=142 locks held 0 write locks 0 pid/thread 4845/140084804294400 flags 0 priority 100 bd dd=141 locks held 0 write locks 0 pid/thread 4845/140084695254784 flags 0 priority 100 be dd=140 locks held 0 write locks 0 pid/thread 4845/140084670076672 flags 0 priority 100 bf dd=139 locks held 0 write locks 0 pid/thread 4845/140084628113152 flags 0 priority 100 c0 dd=138 locks held 0 write locks 0 pid/thread 4845/140084661683968 flags 0 priority 100 c1 dd=137 locks held 0 write locks 0 pid/thread 4845/140084712040192 flags 0 priority 100 c2 dd=136 locks held 0 write locks 0 pid/thread 4845/140084661683968 flags 0 priority 100 c3 dd=135 locks held 0 write locks 0 pid/thread 4845/140084577756928 flags 0 priority 100 c4 dd=134 locks held 0 write locks 0 pid/thread 4845/140084854650624 flags 0 priority 100 c5 dd=133 locks held 1 write locks 0 pid/thread 4845/140084644898560 flags 10 priority 100 c5 READ 1 HELD userRoot/uidnumber.db handle 0 c6 dd=132 locks held 0 write locks 0 pid/thread 4845/140084594542336 flags 0 priority 100 c8 dd=131 locks held 1 write locks 0 pid/thread 4845/140084678469376 flags 10 priority 100 c8 READ 1 HELD ipaca/certstatus.db handle 0 c9 dd=130 locks held 0 write locks 0 pid/thread 4845/140084720432896 flags 0 priority 100 ca dd=129 locks held 0 write locks 0 pid/thread 4845/140084619720448 flags 0 priority 100 cb dd=128 locks held 0 write locks 0 pid/thread 4845/140084821079808 flags 0 priority 100 cc dd=127 locks held 0 write locks 0 pid/thread 4845/140084594542336 flags 0 priority 100 cd dd=126 locks held 0 write locks 0 pid/thread 4845/140084560971520 flags 0 priority 100 ce dd=125 locks held 1 write locks 0 pid/thread 4845/140084837865216 flags 10 priority 100 ce READ 1 HELD userRoot/gidnumber.db handle 0 cf dd=124 locks held 0 write locks 0 pid/thread 4845/140084720432896 flags 0 priority 100 d0 dd=123 locks held 0 write locks 0 pid/thread 4845/140084569364224 flags 0 priority 100 d1 dd=122 locks held 1 write locks 0 pid/thread 4845/140084678469376 flags 10 priority 100 d1 READ 1 HELD userRoot/nscpEntryDN.db handle 0 d2 dd=121 locks held 0 write locks 0 pid/thread 4845/140084695254784 flags 0 priority 100 d3 dd=120 locks held 0 write locks 0 pid/thread 4845/140084569364224 flags 0 priority 100 d4 dd=119 locks held 0 write locks 0 pid/thread 4845/140084653291264 flags 0 priority 100 d5 dd=118 locks held 0 write locks 0 pid/thread 4845/140084829472512 flags 0 priority 100 d6 dd=117 locks held 0 write locks 0 pid/thread 4845/140084661683968 flags 0 priority 100 d7 dd=116 locks held 0 write locks 0 pid/thread 4845/140084804294400 flags 0 priority 100 d8 dd=115 locks held 1 write locks 0 pid/thread 4845/140084821079808 flags 10 priority 100 d8 READ 1 HELD ipaca/description.db handle 0 d9 dd=114 locks held 0 write locks 0 pid/thread 4845/140084661683968 flags 0 priority 100 da dd=113 locks held 0 write locks 0 pid/thread 4845/140084837865216 flags 0 priority 100 db dd=112 locks held 1 write locks 0 pid/thread 4845/140084678469376 flags 10 priority 100 db READ 1 HELD ipaca/cn.db handle 0 dc dd=111 locks held 0 write locks 0 pid/thread 4845/140084577756928 flags 0 priority 100 dd dd=110 locks held 1 write locks 0 pid/thread 4845/140084678469376 flags 10 priority 100 dd READ 1 HELD ipaca/ancestorid.db handle 0 de dd=109 locks held 0 write locks 0 pid/thread 4845/140084577756928 flags 0 priority 100 df dd=108 locks held 0 write locks 0 pid/thread 4845/140084703647488 flags 0 priority 100 e0 dd=107 locks held 0 write locks 0 pid/thread 4845/140084569364224 flags 0 priority 100 e1 dd=106 locks held 0 write locks 0 pid/thread 4845/140084569364224 flags 0 priority 100 e2 dd=105 locks held 0 write locks 0 pid/thread 4845/140084821079808 flags 0 priority 100 e3 dd=104 locks held 1 write locks 0 pid/thread 4845/140084686862080 flags 10 priority 100 e3 READ 1 HELD ipaca/serialno.db handle 0 e4 dd=103 locks held 0 write locks 0 pid/thread 4845/140084837865216 flags 0 priority 100 e5 dd=102 locks held 1 write locks 0 pid/thread 4845/140084695254784 flags 10 priority 100 e5 READ 1 HELD ipaca/revInfo.db handle 0 e6 dd=101 locks held 1 write locks 0 pid/thread 4845/140084695254784 flags 10 priority 100 e6 READ 1 HELD ipaca/revokedby.db handle 0 e7 dd=100 locks held 1 write locks 0 pid/thread 4845/140084695254784 flags 10 priority 100 e7 READ 1 HELD ipaca/revokedOn.db handle 0 e8 dd=99 locks held 0 write locks 0 pid/thread 4845/140084829472512 flags 0 priority 100 e9 dd=98 locks held 0 write locks 0 pid/thread 4845/140084846257920 flags 0 priority 100 ea dd=97 locks held 1 write locks 0 pid/thread 4845/140084703647488 flags 10 priority 100 ea READ 1 HELD ipaca/nsuniqueid.db handle 0 eb dd=96 locks held 1 write locks 0 pid/thread 4845/140084703647488 flags 10 priority 100 eb READ 1 HELD ipaca/requestid.db handle 0 ec dd=95 locks held 1 write locks 0 pid/thread 4845/140084703647488 flags 10 priority 100 ec READ 1 HELD ipaca/requeststate.db handle 0 ed dd=94 locks held 1 write locks 0 pid/thread 4845/140084703647488 flags 10 priority 100 ed READ 1 HELD ipaca/dateOfCreate.db handle 0 ee dd=93 locks held 1 write locks 0 pid/thread 4845/140084703647488 flags 10 priority 100 ee READ 1 HELD ipaca/requesttype.db handle 0 ef dd=92 locks held 1 write locks 0 pid/thread 4845/140084703647488 flags 10 priority 100 ef READ 1 HELD ipaca/numsubordinates.db handle 0 f0 dd=91 locks held 1 write locks 0 pid/thread 4845/140084611327744 flags 10 priority 100 f0 READ 1 HELD userRoot/userCertificate.db handle 0 f1 dd=90 locks held 0 write locks 0 pid/thread 4845/140084821079808 flags 0 priority 100 f2 dd=89 locks held 0 write locks 0 pid/thread 4845/140084569364224 flags 0 priority 100 f3 dd=88 locks held 0 write locks 0 pid/thread 4845/140084602935040 flags 0 priority 100 f4 dd=87 locks held 0 write locks 0 pid/thread 4845/140084720432896 flags 0 priority 100 f5 dd=86 locks held 0 write locks 0 pid/thread 4845/140084628113152 flags 0 priority 100 f6 dd=85 locks held 0 write locks 0 pid/thread 4845/140084703647488 flags 0 priority 100 f7 dd=84 locks held 0 write locks 0 pid/thread 4845/140084611327744 flags 0 priority 100 f8 dd=83 locks held 0 write locks 0 pid/thread 4845/140084661683968 flags 0 priority 100 f9 dd=82 locks held 0 write locks 0 pid/thread 4845/140084837865216 flags 0 priority 100 fa dd=81 locks held 0 write locks 0 pid/thread 4845/140084544186112 flags 0 priority 100 fb dd=80 locks held 0 write locks 0 pid/thread 4845/140084703647488 flags 0 priority 100 fc dd=79 locks held 0 write locks 0 pid/thread 4845/140084544186112 flags 0 priority 100 fd dd=78 locks held 0 write locks 0 pid/thread 4845/140084535793408 flags 0 priority 100 fe dd=77 locks held 0 write locks 0 pid/thread 4845/140084586149632 flags 0 priority 100 ff dd=76 locks held 0 write locks 0 pid/thread 4845/140084569364224 flags 0 priority 100 100 dd=75 locks held 0 write locks 0 pid/thread 4845/140084804294400 flags 0 priority 100 101 dd=74 locks held 0 write locks 0 pid/thread 4845/140084703647488 flags 0 priority 100 102 dd=73 locks held 0 write locks 0 pid/thread 4845/140084703647488 flags 0 priority 100 103 dd=72 locks held 0 write locks 0 pid/thread 4845/140084821079808 flags 0 priority 100 104 dd=71 locks held 0 write locks 0 pid/thread 4845/140084695254784 flags 0 priority 100 105 dd=70 locks held 0 write locks 0 pid/thread 4845/140084544186112 flags 0 priority 100 106 dd=69 locks held 0 write locks 0 pid/thread 4845/140084854650624 flags 0 priority 100 107 dd=68 locks held 0 write locks 0 pid/thread 4845/140084628113152 flags 0 priority 100 108 dd=67 locks held 0 write locks 0 pid/thread 4845/140084569364224 flags 0 priority 100 109 dd=66 locks held 0 write locks 0 pid/thread 4845/140084594542336 flags 0 priority 100 10a dd=65 locks held 0 write locks 0 pid/thread 4845/140084661683968 flags 0 priority 100 10b dd=64 locks held 0 write locks 0 pid/thread 4845/140084653291264 flags 0 priority 100 10c dd=63 locks held 0 write locks 0 pid/thread 4845/140084695254784 flags 0 priority 100 10d dd=62 locks held 0 write locks 0 pid/thread 4845/140084720432896 flags 0 priority 100 10e dd=61 locks held 0 write locks 0 pid/thread 4845/140084619720448 flags 0 priority 100 10f dd=60 locks held 0 write locks 0 pid/thread 4845/140084720432896 flags 0 priority 100 110 dd=59 locks held 0 write locks 0 pid/thread 4845/140084569364224 flags 0 priority 100 111 dd=58 locks held 0 write locks 0 pid/thread 4845/140084695254784 flags 0 priority 100 112 dd=57 locks held 0 write locks 0 pid/thread 4845/140084653291264 flags 0 priority 100 113 dd=56 locks held 0 write locks 0 pid/thread 4845/140084712040192 flags 0 priority 100 114 dd=55 locks held 0 write locks 0 pid/thread 4845/140084686862080 flags 0 priority 100 115 dd=54 locks held 0 write locks 0 pid/thread 4845/140084577756928 flags 0 priority 100 115 READ 1 WAIT userRoot/objectclass.db page 9 116 dd=53 locks held 0 write locks 0 pid/thread 4845/140084837865216 flags 0 priority 100 116 READ 1 WAIT userRoot/objectclass.db page 9 117 dd=52 locks held 0 write locks 0 pid/thread 4845/140084577756928 flags 0 priority 100 118 dd=51 locks held 0 write locks 0 pid/thread 4845/140084569364224 flags 0 priority 100 119 dd=50 locks held 0 write locks 0 pid/thread 4845/140084560971520 flags 0 priority 100 11a dd=49 locks held 1 write locks 0 pid/thread 4845/140084695254784 flags 10 priority 100 11a READ 1 HELD ipaca/metaInfo.db handle 0 11b dd=48 locks held 1 write locks 0 pid/thread 4845/140084695254784 flags 10 priority 100 11b READ 1 HELD ipaca/notbefore.db handle 0 11c dd=47 locks held 1 write locks 0 pid/thread 4845/140084695254784 flags 10 priority 100 11c READ 1 HELD ipaca/notafter.db handle 0 11d dd=46 locks held 1 write locks 0 pid/thread 4845/140084695254784 flags 10 priority 100 11d READ 1 HELD ipaca/duration.db handle 0 11e dd=45 locks held 1 write locks 0 pid/thread 4845/140084695254784 flags 10 priority 100 11e READ 1 HELD ipaca/subjectname.db handle 0 11f dd=44 locks held 1 write locks 0 pid/thread 4845/140084695254784 flags 10 priority 100 11f READ 1 HELD ipaca/publicKeyData.db handle 0 120 dd=43 locks held 1 write locks 0 pid/thread 4845/140084695254784 flags 10 priority 100 120 READ 1 HELD ipaca/extension.db handle 0 121 dd=42 locks held 1 write locks 0 pid/thread 4845/140084695254784 flags 10 priority 100 121 READ 1 HELD ipaca/issuedby.db handle 0 122 dd=41 locks held 0 write locks 0 pid/thread 4845/140084712040192 flags 0 priority 100 123 dd=40 locks held 0 write locks 0 pid/thread 4845/140084812687104 flags 0 priority 100 124 dd=39 locks held 0 write locks 0 pid/thread 4845/140084544186112 flags 0 priority 100 125 dd=38 locks held 0 write locks 0 pid/thread 4845/140084686862080 flags 0 priority 100 126 dd=37 locks held 0 write locks 0 pid/thread 4845/140084552578816 flags 0 priority 100 128 dd=36 locks held 0 write locks 0 pid/thread 4845/140084686862080 flags 0 priority 100 12a dd=35 locks held 0 write locks 0 pid/thread 4845/140084720432896 flags 0 priority 100 12b dd=34 locks held 0 write locks 0 pid/thread 4845/140084552578816 flags 0 priority 100 12c dd=33 locks held 0 write locks 0 pid/thread 4845/140084670076672 flags 0 priority 100 12d dd=32 locks held 0 write locks 0 pid/thread 4845/140084577756928 flags 0 priority 100 12e dd=31 locks held 0 write locks 0 pid/thread 4845/140084712040192 flags 0 priority 100 12f dd=30 locks held 0 write locks 0 pid/thread 4845/140084670076672 flags 0 priority 100 130 dd=29 locks held 0 write locks 0 pid/thread 4845/140084577756928 flags 0 priority 100 131 dd=28 locks held 0 write locks 0 pid/thread 4845/140084560971520 flags 0 priority 100 132 dd=27 locks held 0 write locks 0 pid/thread 4845/140084670076672 flags 0 priority 100 133 dd=26 locks held 0 write locks 0 pid/thread 4845/140084602935040 flags 0 priority 100 134 dd=25 locks held 0 write locks 0 pid/thread 4845/140084552578816 flags 0 priority 100 135 dd=24 locks held 0 write locks 0 pid/thread 4845/140084678469376 flags 0 priority 100 136 dd=23 locks held 0 write locks 0 pid/thread 4845/140084544186112 flags 0 priority 100 137 dd=22 locks held 0 write locks 0 pid/thread 4845/140084577756928 flags 0 priority 100 138 dd=21 locks held 0 write locks 0 pid/thread 4845/140084703647488 flags 0 priority 100 139 dd=20 locks held 0 write locks 0 pid/thread 4845/140084678469376 flags 0 priority 100 13b dd=19 locks held 0 write locks 0 pid/thread 4845/140084812687104 flags 0 priority 100 144 dd=18 locks held 0 write locks 0 pid/thread 4845/140084670076672 flags 0 priority 100 156 dd=17 locks held 0 write locks 0 pid/thread 4845/140084594542336 flags 0 priority 100 15a dd=16 locks held 0 write locks 0 pid/thread 4845/140084577756928 flags 0 priority 100 168 dd=15 locks held 0 write locks 0 pid/thread 4845/140084653291264 flags 0 priority 100 16a dd=14 locks held 0 write locks 0 pid/thread 4845/140084829472512 flags 0 priority 100 17d dd=13 locks held 0 write locks 0 pid/thread 4845/140084829472512 flags 0 priority 100 17e dd=12 locks held 0 write locks 0 pid/thread 4845/140084829472512 flags 0 priority 100 17f dd=11 locks held 0 write locks 0 pid/thread 4845/140084586149632 flags 0 priority 100 180 dd=10 locks held 0 write locks 0 pid/thread 4845/140084594542336 flags 0 priority 100 181 dd= 9 locks held 0 write locks 0 pid/thread 4845/140084577756928 flags 0 priority 100 193 dd= 8 locks held 0 write locks 0 pid/thread 4845/140084812687104 flags 0 priority 100 196 dd= 7 locks held 0 write locks 0 pid/thread 4845/140084695254784 flags 0 priority 100 1a1 dd= 6 locks held 0 write locks 0 pid/thread 4845/140084720432896 flags 0 priority 100 1a2 dd= 5 locks held 0 write locks 0 pid/thread 4845/140084577756928 flags 0 priority 100 1a3 dd= 4 locks held 0 write locks 0 pid/thread 4845/140084812687104 flags 0 priority 100 1a4 dd= 3 locks held 0 write locks 0 pid/thread 4845/140084812687104 flags 0 priority 100 1a5 dd= 2 locks held 0 write locks 0 pid/thread 4845/140084577756928 flags 0 priority 100 1a6 dd= 1 locks held 0 write locks 0 pid/thread 4845/140084594542336 flags 0 priority 100 800b3a33 dd= 0 locks held 26 write locks 22 pid/thread 4845/140084661683968 flags 0 priority 100 800b3a33 WRITE 1 HELD userRoot/objectclass.db page 9 800b3a33 WRITE 1 HELD userRoot/objectclass.db page 8 800b3a33 WRITE 1 HELD userRoot/objectclass.db page 18 800b3a33 WRITE 1 HELD userRoot/objectclass.db page 7 800b3a33 WRITE 1 HELD userRoot/objectclass.db page 14 800b3a33 WRITE 1 HELD userRoot/objectclass.db page 11 800b3a33 WRITE 2 HELD userRoot/objectclass.db page 3 800b3a33 WRITE 1 HELD userRoot/cn.db page 150 800b3a33 WRITE 1 HELD userRoot/cn.db page 89 800b3a33 WRITE 1 HELD userRoot/cn.db page 53 800b3a33 WRITE 1 HELD userRoot/cn.db page 131 800b3a33 WRITE 1 HELD userRoot/cn.db page 117 800b3a33 WRITE 1 HELD userRoot/cn.db page 11 800b3a33 WRITE 2 HELD userRoot/cn.db page 141 800b3a33 WRITE 1 HELD userRoot/cn.db page 41 800b3a33 WRITE 1 HELD userRoot/cn.db page 40 800b3a33 WRITE 1 HELD userRoot/cn.db page 143 800b3a33 WRITE 1 HELD userRoot/cn.db page 30 800b3a33 WRITE 2 HELD userRoot/id2entry.db page 8184 800b3a33 WRITE 1 HELD userRoot/id2entry.db page 8183 800b3a33 WRITE 1 HELD userRoot/id2entry.db page 7779 800b3a33 WRITE 1 HELD userRoot/id2entry.db page 0 800b3a33 READ 3 HELD userRoot/entryrdn.db page 3 800b3a33 READ 1 HELD userRoot/entryrdn.db page 36 800b3a33 READ 1 HELD userRoot/entryrdn.db page 4 800b3a33 READ 1 HELD userRoot/nsuniqueid.db page 86 =-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-= Locks grouped by object: Locker Mode Count Status ----------------- Object --------------- a4 READ 1 HELD userRoot/ipasudorunas.db handle 0 9e READ 1 HELD userRoot/ipaMemberCertProfile.db handle 0 10 READ 1 HELD ipaca/vlv#allrevokedcertspkitomcatindex.db handle 0 800b3a33 WRITE 2 HELD userRoot/id2entry.db page 8184 800b3a33 WRITE 1 HELD userRoot/id2entry.db page 8183 800b3a33 WRITE 1 HELD userRoot/id2entry.db page 7779 a1 READ 1 HELD userRoot/memberdenycmd.db handle 0 a READ 1 HELD ipaca/vlv#allcertspkitomcatindex.db handle 0 a7 READ 1 HELD userRoot/sourcehost.db handle 0 57 READ 1 HELD changelog/ancestorid.db handle 0 e5 READ 1 HELD ipaca/revInfo.db handle 0 e6 READ 1 HELD ipaca/revokedby.db handle 0 11 READ 1 HELD ipaca/vlv#allrevokedcertsnotafterpkitomcatindex.db handle 0 4e READ 1 HELD userRoot/parentid.db handle 0 4 READ 1 HELD ipaca/id2entry.db handle 0 58 READ 1 HELD changelog/numsubordinates.db handle 0 67 READ 1 HELD ipaca/seeAlso.db handle 0 2b READ 1 HELD ipaca/vlv#carevocationpkitomcatindex.db handle 0 ec READ 1 HELD ipaca/requeststate.db handle 0 11b READ 1 HELD ipaca/notbefore.db handle 0 7e READ 1 HELD userRoot/numsubordinates.db handle 0 a2 READ 1 HELD userRoot/memberallowcmd.db handle 0 9c READ 1 HELD userRoot/owner.db handle 0 1e READ 1 HELD ipaca/vlv#cacompleteenrollmentpkitomcatindex.db handle 0 11c READ 1 HELD ipaca/notafter.db handle 0 40 READ 1 HELD changelog/objectclass.db handle 0 2e READ 1 HELD changelog/entryusn.db handle 0 16 READ 1 HELD ipaca/vlv#allvalidcertsnotafterpkitomcatindex.db handle 0 15 READ 1 HELD ipaca/vlv#allvalidcertspkitomcatindex.db handle 0 a5 READ 1 HELD userRoot/ipaMemberCa.db handle 0 18 READ 1 HELD ipaca/vlv#caallpkitomcatindex.db handle 0 a0 READ 1 HELD userRoot/secretary.db handle 0 81 READ 1 HELD userRoot/memberUser.db handle 0 9f READ 1 HELD userRoot/ipaassignedidview.db handle 0 a3 READ 1 HELD userRoot/manager.db handle 0 d8 READ 1 HELD ipaca/description.db handle 0 a8 READ 1 HELD userRoot/memberservice.db handle 0 44 READ 1 HELD ipaca/objectclass.db handle 0 ed READ 1 HELD ipaca/dateOfCreate.db handle 0 800b3a33 WRITE 1 HELD userRoot/id2entry.db page 0 2 READ 1 HELD userRoot/id2entry.db handle 0 5b READ 1 HELD userRoot/krbPrincipalName.db handle 0 ee READ 1 HELD ipaca/requesttype.db handle 0 c8 READ 1 HELD ipaca/certstatus.db handle 0 56 READ 1 HELD changelog/parentid.db handle 0 83 READ 1 HELD userRoot/memberHost.db handle 0 5e READ 1 HELD userRoot/ipakrbprincipalalias.db handle 0 121 READ 1 HELD ipaca/issuedby.db handle 0 6 READ 1 HELD ipaca/entryrdn.db handle 0 e7 READ 1 HELD ipaca/revokedOn.db handle 0 4c READ 1 HELD ipaca/aci.db handle 0 11d READ 1 HELD ipaca/duration.db handle 0 14 READ 1 HELD ipaca/vlv#allrevokedorrevokedexpiredcertspkitomcatindex.db handle 0 21 READ 1 HELD ipaca/vlv#caenrollmentpkitomcatindex.db handle 0 55 READ 1 HELD changelog/targetuniqueid.db handle 0 800b3a33 READ 1 HELD userRoot/entryrdn.db page 36 2c READ 1 HELD changelog/id2entry.db handle 0 800b3a33 READ 1 HELD userRoot/entryrdn.db page 4 34 READ 1 HELD userRoot/entryrdn.db handle 0 800b3a33 READ 3 HELD userRoot/entryrdn.db page 3 eb READ 1 HELD ipaca/requestid.db handle 0 49 READ 1 HELD userRoot/aci.db handle 0 76 READ 1 HELD userRoot/ipauniqueid.db handle 0 63 READ 1 HELD changelog/seeAlso.db handle 0 7c READ 1 HELD userRoot/fqdn.db handle 0 db READ 1 HELD ipaca/cn.db handle 0 ef READ 1 HELD ipaca/numsubordinates.db handle 0 32 READ 1 HELD ipaca/entryusn.db handle 0 1d READ 1 HELD ipaca/vlv#cacompletepkitomcatindex.db handle 0 dd READ 1 HELD ipaca/ancestorid.db handle 0 a9 READ 1 HELD userRoot/ipatokenradiusconfiglink.db handle 0 f0 READ 1 HELD userRoot/userCertificate.db handle 0 11a READ 1 HELD ipaca/metaInfo.db handle 0 7d READ 1 HELD userRoot/managedby.db handle 0 ea READ 1 HELD ipaca/nsuniqueid.db handle 0 c5 READ 1 HELD userRoot/uidnumber.db handle 0 d1 READ 1 HELD userRoot/nscpEntryDN.db handle 0 120 READ 1 HELD ipaca/extension.db handle 0 800b3a33 WRITE 1 HELD userRoot/cn.db page 150 69 READ 1 HELD ipaca/parentid.db handle 0 800b3a33 WRITE 1 HELD userRoot/cn.db page 131 800b3a33 WRITE 1 HELD userRoot/cn.db page 143 800b3a33 WRITE 2 HELD userRoot/cn.db page 141 20 READ 1 HELD ipaca/vlv#cacompleterevocationpkitomcatindex.db handle 0 8e READ 1 HELD userRoot/macAddress.db handle 0 e READ 1 HELD ipaca/vlv#allnonrevokedcertspkitomcatindex.db handle 0 17 READ 1 HELD ipaca/vlv#allvalidorrevokedcertspkitomcatindex.db handle 0 800b3a33 WRITE 1 HELD userRoot/cn.db page 30 800b3a33 WRITE 1 HELD userRoot/objectclass.db page 14 800b3a33 WRITE 1 HELD userRoot/objectclass.db page 9 116 READ 1 WAIT userRoot/objectclass.db page 9 115 READ 1 WAIT userRoot/objectclass.db page 9 92 READ 1 WAIT userRoot/objectclass.db page 9 800b3a33 WRITE 1 HELD userRoot/objectclass.db page 8 7b READ 1 HELD userRoot/cn.db handle 0 800b3a33 WRITE 1 HELD userRoot/objectclass.db page 11 800b3a33 WRITE 1 HELD userRoot/objectclass.db page 7 800b3a33 WRITE 1 HELD userRoot/cn.db page 11 38 READ 1 HELD userRoot/objectclass.db handle 0 800b3a33 WRITE 2 HELD userRoot/objectclass.db page 3 800b3a33 WRITE 1 HELD userRoot/cn.db page 53 800b3a33 WRITE 1 HELD userRoot/objectclass.db page 18 6e READ 1 WAIT userRoot/objectclass.db page 18 94 READ 1 WAIT userRoot/objectclass.db page 18 800b3a33 WRITE 1 HELD userRoot/cn.db page 41 800b3a33 WRITE 1 HELD userRoot/cn.db page 40 800b3a33 WRITE 1 HELD userRoot/cn.db page 89 800b3a33 WRITE 1 HELD userRoot/cn.db page 117 30 READ 1 HELD userRoot/entryusn.db handle 0 3c READ 1 HELD changelog/entryrdn.db handle 0 97 READ 1 HELD userRoot/uid.db handle 0 47 READ 1 HELD changelog/aci.db handle 0 e3 READ 1 HELD ipaca/serialno.db handle 0 7a READ 1 HELD userRoot/nsuniqueid.db handle 0 800b3a33 READ 1 HELD userRoot/nsuniqueid.db page 86 51 READ 1 HELD changelog/nsuniqueid.db handle 0 11e READ 1 HELD ipaca/subjectname.db handle 0 7f READ 1 HELD userRoot/member.db handle 0 11f READ 1 HELD ipaca/publicKeyData.db handle 0 9b READ 1 HELD userRoot/uniquemember.db handle 0 52 READ 1 HELD changelog/changenumber.db handle 0 ce READ 1 HELD userRoot/gidnumber.db handle 0 65 READ 1 HELD userRoot/seeAlso.db handle 0 9d READ 1 HELD userRoot/ipaallowedtarget.db handle 0 a6 READ 1 HELD userRoot/ipasudorunasgroup.db handle 0 3a READ 1 HELD userRoot/ancestorid.db handle 0 pstack output root at ipa-master ]# pstack 4845 Thread 44 (Thread 0x7f681dd3a700 (LWP 4847)): #0 0x00007f683cd719b3 in select () from /lib64/libc.so.6 #1 0x00007f683f4fc0e9 in DS_Sleep () from /usr/lib64/dirsrv/libslapd.so.0 #2 0x00007f6832b4a907 in deadlock_threadmain () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #3 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so #4 0x00007f683d04cdc5 in start_thread () from /lib64/libpthread.so.0 #5 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 Thread 43 (Thread 0x7f681d539700 (LWP 4848)): #0 0x00007f683cd719b3 in select () from /lib64/libc.so.6 #1 0x00007f68376d8bad in __os_yield () from /lib64/libdb-5.3.so #2 0x00007f68376d42b3 in __memp_sync_int () from /lib64/libdb-5.3.so #3 0x00007f68376e4752 in __txn_checkpoint () from /lib64/libdb-5.3.so #4 0x00007f68376e4b74 in __txn_checkpoint_pp () from /lib64/libdb-5.3.so #5 0x00007f6832b4ea87 in checkpoint_threadmain () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #6 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so #7 0x00007f683d04cdc5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 Thread 42 (Thread 0x7f681cd38700 (LWP 4849)): #0 0x00007f683cd719b3 in select () from /lib64/libc.so.6 #1 0x00007f683f4fc0e9 in DS_Sleep () from /usr/lib64/dirsrv/libslapd.so.0 #2 0x00007f6832b4ab7f in trickle_threadmain () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #3 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so #4 0x00007f683d04cdc5 in start_thread () from /lib64/libpthread.so.0 #5 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 Thread 41 (Thread 0x7f681c537700 (LWP 4850)): #0 0x00007f683cd719b3 in select () from /lib64/libc.so.6 #1 0x00007f683f4fc0e9 in DS_Sleep () from /usr/lib64/dirsrv/libslapd.so.0 #2 0x00007f6832b45707 in perf_threadmain () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #3 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so #4 0x00007f683d04cdc5 in start_thread () from /lib64/libpthread.so.0 #5 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 Thread 40 (Thread 0x7f681bb2d700 (LWP 4851)): #0 0x00007f683d0506d5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f683d6a6050 in PR_WaitCondVar () from /lib64/libnspr4.so #2 0x00007f683f4eb198 in slapi_wait_condvar () from /usr/lib64/dirsrv/libslapd.so.0 #3 0x00007f683587d62e in cos_cache_wait_on_change () from /usr/lib64/dirsrv/plugins/libcos-plugin.so #4 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so #5 0x00007f683d04cdc5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 Thread 39 (Thread 0x7f681b32c700 (LWP 4852)): #0 0x00007f683d0506d5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f683d6a6050 in PR_WaitCondVar () from /lib64/libnspr4.so #2 0x00007f683f4eb198 in slapi_wait_condvar () from /usr/lib64/dirsrv/libslapd.so.0 #3 0x00007f68311e8eed in roles_cache_wait_on_change () from /usr/lib64/dirsrv/plugins/libroles-plugin.so #4 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so #5 0x00007f683d04cdc5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 Thread 38 (Thread 0x7f681ab2b700 (LWP 4853)): #0 0x00007f683d0506d5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f683d6a6050 in PR_WaitCondVar () from /lib64/libnspr4.so #2 0x00007f683f4eb198 in slapi_wait_condvar () from /usr/lib64/dirsrv/libslapd.so.0 #3 0x00007f68311e8eed in roles_cache_wait_on_change () from /usr/lib64/dirsrv/plugins/libroles-plugin.so #4 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so #5 0x00007f683d04cdc5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 Thread 37 (Thread 0x7f681a32a700 (LWP 4854)): #0 0x00007f683d0506d5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f683d6a6050 in PR_WaitCondVar () from /lib64/libnspr4.so #2 0x00007f683f4eb198 in slapi_wait_condvar () from /usr/lib64/dirsrv/libslapd.so.0 #3 0x00007f68311e8eed in roles_cache_wait_on_change () from /usr/lib64/dirsrv/plugins/libroles-plugin.so #4 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so #5 0x00007f683d04cdc5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 Thread 36 (Thread 0x7f6819b29700 (LWP 4855)): #0 0x00007f683d050a82 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f683d6a5b07 in pt_TimedWait () from /lib64/libnspr4.so #2 0x00007f683d6a5fce in PR_WaitCondVar () from /lib64/libnspr4.so #3 0x00007f683f9a4d03 in housecleaning () #4 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so #5 0x00007f683d04cdc5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 Thread 35 (Thread 0x7f6819328700 (LWP 4856)): #0 0x00007f683d050a82 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f683d6a5b07 in pt_TimedWait () from /lib64/libnspr4.so #2 0x00007f683d6a5fce in PR_WaitCondVar () from /lib64/libnspr4.so #3 0x00007f683f4911e8 in eq_loop () from /usr/lib64/dirsrv/libslapd.so.0 #4 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so #5 0x00007f683d04cdc5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 Thread 34 (Thread 0x7f680bfff700 (LWP 4859)): #0 0x00007f683d0506d5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f683d6a6050 in PR_WaitCondVar () from /lib64/libnspr4.so #2 0x00007f683f999fee in connection_wait_for_new_work () #3 0x00007f683f99b719 in connection_threadmain () #4 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so #5 0x00007f683d04cdc5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 Thread 33 (Thread 0x7f680b7fe700 (LWP 4860)): #0 0x00007f683d0506d5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f683d6a6050 in PR_WaitCondVar () from /lib64/libnspr4.so #2 0x00007f683f999fee in connection_wait_for_new_work () #3 0x00007f683f99b719 in connection_threadmain () #4 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so #5 0x00007f683d04cdc5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 Thread 32 (Thread 0x7f680affd700 (LWP 4861)): #0 0x00007f683d0506d5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f68375ac2f3 in __db_hybrid_mutex_suspend () from /lib64/ libdb-5.3.so #2 0x00007f68375ab640 in __db_tas_mutex_lock () from /lib64/libdb-5.3.so #3 0x00007f6837655d3a in __lock_get_internal () from /lib64/libdb-5.3.so #4 0x00007f6837656820 in __lock_get () from /lib64/libdb-5.3.so #5 0x00007f6837682142 in __db_lget () from /lib64/libdb-5.3.so #6 0x00007f68375c9605 in __bam_search () from /lib64/libdb-5.3.so #7 0x00007f68375b4256 in __bamc_search () from /lib64/libdb-5.3.so #8 0x00007f68375b5d0f in __bamc_get () from /lib64/libdb-5.3.so #9 0x00007f683766eca6 in __dbc_iget () from /lib64/libdb-5.3.so #10 0x00007f683767db02 in __dbc_get_pp () from /lib64/libdb-5.3.so #11 0x00007f6832b59680 in idl_new_fetch () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #12 0x00007f6832b67b26 in index_read_ext_allids () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #13 0x00007f6832b52284 in keys2idl () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #14 0x00007f6832b529e3 in ava_candidates.isra.0 () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #15 0x00007f6832b52fd2 in filter_candidates_ext () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #16 0x00007f6832b54096 in list_candidates () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #17 0x00007f6832b52f40 in filter_candidates_ext () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #18 0x00007f6832b54096 in list_candidates () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #19 0x00007f6832b52f40 in filter_candidates_ext () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #20 0x00007f6832b54096 in list_candidates () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #21 0x00007f6832b52f40 in filter_candidates_ext () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #22 0x00007f6832b8e80c in subtree_candidates () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #23 0x00007f6832b8fe7f in ldbm_back_search () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #24 0x00007f683f4bd1d6 in op_shared_search () from /usr/lib64/dirsrv/libslapd.so.0 #25 0x00007f683f9ae33e in do_search () #26 0x00007f683f99c3c3 in connection_threadmain () #27 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so #28 0x00007f683d04cdc5 in start_thread () from /lib64/libpthread.so.0 #29 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 Thread 31 (Thread 0x7f680a7fc700 (LWP 4862)): #0 0x00007f683d0506d5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f683d6a62c3 in PR_EnterMonitor () from /lib64/libnspr4.so #2 0x00007f6832b4d3e6 in dblayer_txn_begin () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #3 0x00007f6832b89007 in ldbm_back_modify () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #4 0x00007f683f4b6541 in op_shared_modify () from /usr/lib64/dirsrv/libslapd.so.0 #5 0x00007f683f4b7004 in modify_internal_pb () from /usr/lib64/dirsrv/libslapd.so.0 #6 0x00007f683483bbbb in ipalockout_postop () from /usr/lib64/dirsrv/plugins/libipa_lockout.so #7 0x00007f683f4c8848 in plugin_call_func () from /usr/lib64/dirsrv/libslapd.so.0 #8 0x00007f683f4c8b13 in plugin_call_plugins () from /usr/lib64/dirsrv/libslapd.so.0 #9 0x00007f683f9954d8 in do_bind () #10 0x00007f683f99c3fd in connection_threadmain () #11 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so #12 0x00007f683d04cdc5 in start_thread () from /lib64/libpthread.so.0 #13 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 Thread 30 (Thread 0x7f6809ffb700 (LWP 4863)): #0 0x00007f683d0506d5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f68375ac2f3 in __db_hybrid_mutex_suspend () from /lib64/ libdb-5.3.so #2 0x00007f68375ab640 in __db_tas_mutex_lock () from /lib64/libdb-5.3.so #3 0x00007f6837655d3a in __lock_get_internal () from /lib64/libdb-5.3.so #4 0x00007f6837656820 in __lock_get () from /lib64/libdb-5.3.so #5 0x00007f6837682142 in __db_lget () from /lib64/libdb-5.3.so #6 0x00007f68375c9605 in __bam_search () from /lib64/libdb-5.3.so #7 0x00007f68375b4256 in __bamc_search () from /lib64/libdb-5.3.so #8 0x00007f68375b5d0f in __bamc_get () from /lib64/libdb-5.3.so #9 0x00007f683766eca6 in __dbc_iget () from /lib64/libdb-5.3.so #10 0x00007f683767db02 in __dbc_get_pp () from /lib64/libdb-5.3.so #11 0x00007f6832b59680 in idl_new_fetch () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #12 0x00007f6832b67b26 in index_read_ext_allids () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #13 0x00007f6832b52284 in keys2idl () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #14 0x00007f6832b529e3 in ava_candidates.isra.0 () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #15 0x00007f6832b52fd2 in filter_candidates_ext () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #16 0x00007f6832b54096 in list_candidates () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #17 0x00007f6832b52f40 in filter_candidates_ext () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #18 0x00007f6832b8e80c in subtree_candidates () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #19 0x00007f6832b8fe7f in ldbm_back_search () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #20 0x00007f683f4bd1d6 in op_shared_search () from /usr/lib64/dirsrv/libslapd.so.0 #21 0x00007f683f4cd94e in search_internal_callback_pb () from /usr/lib64/dirsrv/libslapd.so.0 #22 0x00007f683f4cdbe8 in search_internal_pb () from /usr/lib64/dirsrv/libslapd.so.0 #23 0x00007f683f9ac49b in ids_sasl_canon_user () #24 0x00007f683d46c93b in _sasl_canon_user () from /lib64/libsasl2.so.3 #25 0x00007f683d46cc4c in _sasl_canon_user_lookup () from /lib64/libsasl2.so.3 #26 0x00007f6837374fb9 in gssapi_server_mech_step () from /usr/lib64/sasl2/libgssapiv2.so #27 0x00007f683d478b9b in sasl_server_step () from /lib64/libsasl2.so.3 #28 0x00007f683f9ad83b in ids_sasl_check_bind () #29 0x00007f683f99561a in do_bind () #30 0x00007f683f99c3fd in connection_threadmain () #31 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so #32 0x00007f683d04cdc5 in start_thread () from /lib64/libpthread.so.0 #33 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 Thread 29 (Thread 0x7f68097fa700 (LWP 4864)): #0 0x00007f683d0506d5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f683d6a6050 in PR_WaitCondVar () from /lib64/libnspr4.so #2 0x00007f683f999fee in connection_wait_for_new_work () #3 0x00007f683f99b719 in connection_threadmain () #4 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so #5 0x00007f683d04cdc5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 Thread 28 (Thread 0x7f6808ff9700 (LWP 4865)): #0 0x00007f683d0506d5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f683d6a6050 in PR_WaitCondVar () from /lib64/libnspr4.so #2 0x00007f683f999fee in connection_wait_for_new_work () #3 0x00007f683f99b719 in connection_threadmain () #4 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so #5 0x00007f683d04cdc5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 Thread 27 (Thread 0x7f6803fff700 (LWP 4866)): #0 0x00007f683d0506d5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f683d6a6050 in PR_WaitCondVar () from /lib64/libnspr4.so #2 0x00007f683f999fee in connection_wait_for_new_work () #3 0x00007f683f99b719 in connection_threadmain () #4 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so #5 0x00007f683d04cdc5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 Thread 26 (Thread 0x7f68037fe700 (LWP 4867)): #0 0x00007f683d0506d5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f683d6a6050 in PR_WaitCondVar () from /lib64/libnspr4.so #2 0x00007f683f999fee in connection_wait_for_new_work () #3 0x00007f683f99b719 in connection_threadmain () #4 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so #5 0x00007f683d04cdc5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 Thread 25 (Thread 0x7f6802ffd700 (LWP 4868)): #0 0x00007f683d0506d5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f683d6a6050 in PR_WaitCondVar () from /lib64/libnspr4.so #2 0x00007f683f999fee in connection_wait_for_new_work () #3 0x00007f683f99b719 in connection_threadmain () #4 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so #5 0x00007f683d04cdc5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 Thread 24 (Thread 0x7f68027fc700 (LWP 4869)): #0 0x00007f683d0506d5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f68375ac2f3 in __db_hybrid_mutex_suspend () from /lib64/ libdb-5.3.so #2 0x00007f68375abb37 in __db_tas_mutex_readlock () from /lib64/ libdb-5.3.so #3 0x00007f68376c5d35 in __memp_fget () from /lib64/libdb-5.3.so #4 0x00007f683766f333 in __dbc_iget () from /lib64/libdb-5.3.so #5 0x00007f683767db02 in __dbc_get_pp () from /lib64/libdb-5.3.so #6 0x00007f6832b59822 in idl_new_fetch () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #7 0x00007f6832b67b26 in index_read_ext_allids () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #8 0x00007f6832b52284 in keys2idl () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #9 0x00007f6832b529e3 in ava_candidates.isra.0 () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #10 0x00007f6832b52fd2 in filter_candidates_ext () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #11 0x00007f6832b54096 in list_candidates () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #12 0x00007f6832b52f40 in filter_candidates_ext () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #13 0x00007f6832b54096 in list_candidates () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #14 0x00007f6832b52f40 in filter_candidates_ext () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #15 0x00007f6832b54096 in list_candidates () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #16 0x00007f6832b52f40 in filter_candidates_ext () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #17 0x00007f6832b8e80c in subtree_candidates () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #18 0x00007f6832b8fe7f in ldbm_back_search () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #19 0x00007f683f4bd1d6 in op_shared_search () from /usr/lib64/dirsrv/libslapd.so.0 #20 0x00007f683f9ae33e in do_search () #21 0x00007f683f99c3c3 in connection_threadmain () #22 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so #23 0x00007f683d04cdc5 in start_thread () from /lib64/libpthread.so.0 #24 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 Thread 23 (Thread 0x7f6801ffb700 (LWP 4870)): #0 0x00007f683d0506d5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f683d6a6050 in PR_WaitCondVar () from /lib64/libnspr4.so #2 0x00007f683f999fee in connection_wait_for_new_work () #3 0x00007f683f99b719 in connection_threadmain () #4 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so #5 0x00007f683d04cdc5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 Thread 22 (Thread 0x7f68017fa700 (LWP 4871)): #0 0x00007f683d0506d5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f683d6a6050 in PR_WaitCondVar () from /lib64/libnspr4.so #2 0x00007f683f999fee in connection_wait_for_new_work () #3 0x00007f683f99b719 in connection_threadmain () #4 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so #5 0x00007f683d04cdc5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 Thread 21 (Thread 0x7f6800ff9700 (LWP 4872)): #0 0x00007f683d0506d5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f683d6a6050 in PR_WaitCondVar () from /lib64/libnspr4.so #2 0x00007f683f999fee in connection_wait_for_new_work () #3 0x00007f683f99b719 in connection_threadmain () #4 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so #5 0x00007f683d04cdc5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 Thread 20 (Thread 0x7f68007f8700 (LWP 4873)): #0 0x00007f683d0506d5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f68375ac2f3 in __db_hybrid_mutex_suspend () from /lib64/ libdb-5.3.so #2 0x00007f68375ab640 in __db_tas_mutex_lock () from /lib64/libdb-5.3.so #3 0x00007f68376c5e1b in __memp_fget () from /lib64/libdb-5.3.so #4 0x00007f68375c9211 in __bam_search () from /lib64/libdb-5.3.so #5 0x00007f68375b4256 in __bamc_search () from /lib64/libdb-5.3.so #6 0x00007f68375b86c4 in __bamc_put () from /lib64/libdb-5.3.so #7 0x00007f683766f6d5 in __dbc_iput () from /lib64/libdb-5.3.so #8 0x00007f683766aa0e in __db_put () from /lib64/libdb-5.3.so #9 0x00007f683767ffa4 in __db_put_pp () from /lib64/libdb-5.3.so #10 0x00007f6832b5a4cf in idl_new_insert_key () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #11 0x00007f6832b695b5 in addordel_values_sv.isra.3 () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #12 0x00007f6832b6a25a in index_addordel_values_ext_sv () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #13 0x00007f6832b6a694 in index_addordel_values_sv () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #14 0x00007f6832b6a75a in index_addordel_entry () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #15 0x00007f6832b6d5fa in ldbm_back_add () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #16 0x00007f683f46fcca in op_shared_add () from /usr/lib64/dirsrv/libslapd.so.0 #17 0x00007f683f471020 in do_add () from /usr/lib64/dirsrv/libslapd.so.0 #18 0x00007f683f99c303 in connection_threadmain () #19 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so #20 0x00007f683d04cdc5 in start_thread () from /lib64/libpthread.so.0 #21 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 Thread 19 (Thread 0x7f67ffff7700 (LWP 4874)): #0 0x00007f683d0506d5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f683d6a62c3 in PR_EnterMonitor () from /lib64/libnspr4.so #2 0x00007f6832b4d3e6 in dblayer_txn_begin () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #3 0x00007f6832b89007 in ldbm_back_modify () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #4 0x00007f683f4b6541 in op_shared_modify () from /usr/lib64/dirsrv/libslapd.so.0 #5 0x00007f683f4b7004 in modify_internal_pb () from /usr/lib64/dirsrv/libslapd.so.0 #6 0x00007f6832dd3b60 in ipapwd_apply_mods () from /usr/lib64/dirsrv/plugins/libipa_pwd_extop.so #7 0x00007f6832dd78e0 in store_new_keys () from /usr/lib64/dirsrv/plugins/libipa_pwd_extop.so #8 0x00007f6832dd9ad0 in ipapwd_extop () from /usr/lib64/dirsrv/plugins/libipa_pwd_extop.so #9 0x00007f683f4c6ba2 in plugin_call_exop_plugins () from /usr/lib64/dirsrv/libslapd.so.0 #10 0x00007f683f9a2efe in do_extended () #11 0x00007f683f99c41a in connection_threadmain () #12 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so #13 0x00007f683d04cdc5 in start_thread () from /lib64/libpthread.so.0 #14 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 Thread 18 (Thread 0x7f67ff7f6700 (LWP 4875)): #0 0x00007f683d0506d5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f683d6a6050 in PR_WaitCondVar () from /lib64/libnspr4.so #2 0x00007f683f999fee in connection_wait_for_new_work () #3 0x00007f683f99b719 in connection_threadmain () #4 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so #5 0x00007f683d04cdc5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 Thread 17 (Thread 0x7f67feff5700 (LWP 4876)): #0 0x00007f683d0506d5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f68375ac2f3 in __db_hybrid_mutex_suspend () from /lib64/ libdb-5.3.so #2 0x00007f68375ab640 in __db_tas_mutex_lock () from /lib64/libdb-5.3.so #3 0x00007f6837655d3a in __lock_get_internal () from /lib64/libdb-5.3.so #4 0x00007f6837656820 in __lock_get () from /lib64/libdb-5.3.so #5 0x00007f6837682142 in __db_lget () from /lib64/libdb-5.3.so #6 0x00007f68375c9605 in __bam_search () from /lib64/libdb-5.3.so #7 0x00007f68375b4256 in __bamc_search () from /lib64/libdb-5.3.so #8 0x00007f68375b5d0f in __bamc_get () from /lib64/libdb-5.3.so #9 0x00007f683766eca6 in __dbc_iget () from /lib64/libdb-5.3.so #10 0x00007f683767db02 in __dbc_get_pp () from /lib64/libdb-5.3.so #11 0x00007f6832b59680 in idl_new_fetch () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #12 0x00007f6832b67b26 in index_read_ext_allids () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #13 0x00007f6832b52284 in keys2idl () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #14 0x00007f6832b529e3 in ava_candidates.isra.0 () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #15 0x00007f6832b52fd2 in filter_candidates_ext () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #16 0x00007f6832b54096 in list_candidates () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #17 0x00007f6832b52f40 in filter_candidates_ext () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #18 0x00007f6832b8e80c in subtree_candidates () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #19 0x00007f6832b8fe7f in ldbm_back_search () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #20 0x00007f683f4bd1d6 in op_shared_search () from /usr/lib64/dirsrv/libslapd.so.0 #21 0x00007f683f4cd94e in search_internal_callback_pb () from /usr/lib64/dirsrv/libslapd.so.0 #22 0x00007f683f4cdbe8 in search_internal_pb () from /usr/lib64/dirsrv/libslapd.so.0 #23 0x00007f6832dd75dc in get_entry_by_principal () from /usr/lib64/dirsrv/plugins/libipa_pwd_extop.so #24 0x00007f6832dd8eb1 in ipapwd_extop () from /usr/lib64/dirsrv/plugins/libipa_pwd_extop.so #25 0x00007f683f4c6ba2 in plugin_call_exop_plugins () from /usr/lib64/dirsrv/libslapd.so.0 #26 0x00007f683f9a2efe in do_extended () #27 0x00007f683f99c41a in connection_threadmain () #28 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so #29 0x00007f683d04cdc5 in start_thread () from /lib64/libpthread.so.0 #30 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 Thread 16 (Thread 0x7f67fe7f4700 (LWP 4877)): #0 0x00007f683d0506d5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f683d6a6050 in PR_WaitCondVar () from /lib64/libnspr4.so #2 0x00007f683f999fee in connection_wait_for_new_work () #3 0x00007f683f99b719 in connection_threadmain () #4 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so #5 0x00007f683d04cdc5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 Thread 15 (Thread 0x7f67fdff3700 (LWP 4878)): #0 0x00007f683d0506d5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f683d6a6050 in PR_WaitCondVar () from /lib64/libnspr4.so #2 0x00007f683f999fee in connection_wait_for_new_work () #3 0x00007f683f99b719 in connection_threadmain () #4 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so #5 0x00007f683d04cdc5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 Thread 14 (Thread 0x7f67fd7f2700 (LWP 4879)): #0 0x00007f683d0506d5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f683d6a6050 in PR_WaitCondVar () from /lib64/libnspr4.so #2 0x00007f683f999fee in connection_wait_for_new_work () #3 0x00007f683f99b719 in connection_threadmain () #4 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so #5 0x00007f683d04cdc5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 Thread 13 (Thread 0x7f67fcff1700 (LWP 4880)): #0 0x00007f683d0506d5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f683d6a6050 in PR_WaitCondVar () from /lib64/libnspr4.so #2 0x00007f683f999fee in connection_wait_for_new_work () #3 0x00007f683f99b719 in connection_threadmain () #4 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so #5 0x00007f683d04cdc5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 Thread 12 (Thread 0x7f67fc7f0700 (LWP 4881)): #0 0x00007f683d0506d5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f683d6a6050 in PR_WaitCondVar () from /lib64/libnspr4.so #2 0x00007f683f999fee in connection_wait_for_new_work () #3 0x00007f683f99b719 in connection_threadmain () #4 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so #5 0x00007f683d04cdc5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 Thread 11 (Thread 0x7f67fbfef700 (LWP 4882)): #0 0x00007f683d0506d5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f683d6a6050 in PR_WaitCondVar () from /lib64/libnspr4.so #2 0x00007f683f999fee in connection_wait_for_new_work () #3 0x00007f683f99b719 in connection_threadmain () #4 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so #5 0x00007f683d04cdc5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 Thread 10 (Thread 0x7f67fb7ee700 (LWP 4883)): #0 0x00007f683d0506d5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f68375ac2f3 in __db_hybrid_mutex_suspend () from /lib64/ libdb-5.3.so #2 0x00007f68375ab640 in __db_tas_mutex_lock () from /lib64/libdb-5.3.so #3 0x00007f6837655d3a in __lock_get_internal () from /lib64/libdb-5.3.so #4 0x00007f6837656820 in __lock_get () from /lib64/libdb-5.3.so #5 0x00007f6837682142 in __db_lget () from /lib64/libdb-5.3.so #6 0x00007f68375c9605 in __bam_search () from /lib64/libdb-5.3.so #7 0x00007f68375b4256 in __bamc_search () from /lib64/libdb-5.3.so #8 0x00007f68375b5d0f in __bamc_get () from /lib64/libdb-5.3.so #9 0x00007f683766eca6 in __dbc_iget () from /lib64/libdb-5.3.so #10 0x00007f683767db02 in __dbc_get_pp () from /lib64/libdb-5.3.so #11 0x00007f6832b59680 in idl_new_fetch () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #12 0x00007f6832b67b26 in index_read_ext_allids () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #13 0x00007f6832b52284 in keys2idl () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #14 0x00007f6832b529e3 in ava_candidates.isra.0 () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #15 0x00007f6832b52fd2 in filter_candidates_ext () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #16 0x00007f6832b54096 in list_candidates () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #17 0x00007f6832b52f40 in filter_candidates_ext () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #18 0x00007f6832b54096 in list_candidates () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #19 0x00007f6832b52f40 in filter_candidates_ext () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #20 0x00007f6832b54096 in list_candidates () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #21 0x00007f6832b52f40 in filter_candidates_ext () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #22 0x00007f6832b8e80c in subtree_candidates () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #23 0x00007f6832b8fe7f in ldbm_back_search () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #24 0x00007f683f4bd1d6 in op_shared_search () from /usr/lib64/dirsrv/libslapd.so.0 #25 0x00007f683f9ae33e in do_search () #26 0x00007f683f99c3c3 in connection_threadmain () #27 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so #28 0x00007f683d04cdc5 in start_thread () from /lib64/libpthread.so.0 #29 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 Thread 9 (Thread 0x7f67fafed700 (LWP 4884)): #0 0x00007f683d0506d5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f683d6a6050 in PR_WaitCondVar () from /lib64/libnspr4.so #2 0x00007f683f999fee in connection_wait_for_new_work () #3 0x00007f683f99b719 in connection_threadmain () #4 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so #5 0x00007f683d04cdc5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 Thread 8 (Thread 0x7f67fa7ec700 (LWP 4885)): #0 0x00007f683d0506d5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f68375ac2f3 in __db_hybrid_mutex_suspend () from /lib64/ libdb-5.3.so #2 0x00007f68375ab640 in __db_tas_mutex_lock () from /lib64/libdb-5.3.so #3 0x00007f6837655d3a in __lock_get_internal () from /lib64/libdb-5.3.so #4 0x00007f6837656820 in __lock_get () from /lib64/libdb-5.3.so #5 0x00007f6837682142 in __db_lget () from /lib64/libdb-5.3.so #6 0x00007f68375c9605 in __bam_search () from /lib64/libdb-5.3.so #7 0x00007f68375b4256 in __bamc_search () from /lib64/libdb-5.3.so #8 0x00007f68375b5d0f in __bamc_get () from /lib64/libdb-5.3.so #9 0x00007f683766eca6 in __dbc_iget () from /lib64/libdb-5.3.so #10 0x00007f683767db02 in __dbc_get_pp () from /lib64/libdb-5.3.so #11 0x00007f6832b59680 in idl_new_fetch () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #12 0x00007f6832b67b26 in index_read_ext_allids () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #13 0x00007f6832b52284 in keys2idl () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #14 0x00007f6832b529e3 in ava_candidates.isra.0 () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #15 0x00007f6832b52fd2 in filter_candidates_ext () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #16 0x00007f6832b54096 in list_candidates () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #17 0x00007f6832b52f40 in filter_candidates_ext () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #18 0x00007f6832b54096 in list_candidates () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #19 0x00007f6832b52f40 in filter_candidates_ext () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #20 0x00007f6832b54096 in list_candidates () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #21 0x00007f6832b52f40 in filter_candidates_ext () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #22 0x00007f6832b8e80c in subtree_candidates () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #23 0x00007f6832b8fe7f in ldbm_back_search () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #24 0x00007f683f4bd1d6 in op_shared_search () from /usr/lib64/dirsrv/libslapd.so.0 #25 0x00007f683f9ae33e in do_search () #26 0x00007f683f99c3c3 in connection_threadmain () #27 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so #28 0x00007f683d04cdc5 in start_thread () from /lib64/libpthread.so.0 #29 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 Thread 7 (Thread 0x7f67f9feb700 (LWP 4886)): #0 0x00007f683d0506d5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f683d6a6050 in PR_WaitCondVar () from /lib64/libnspr4.so #2 0x00007f683f999fee in connection_wait_for_new_work () #3 0x00007f683f99b719 in connection_threadmain () #4 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so #5 0x00007f683d04cdc5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 Thread 6 (Thread 0x7f67f97ea700 (LWP 4887)): #0 0x00007f683d0506d5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f683d6a6050 in PR_WaitCondVar () from /lib64/libnspr4.so #2 0x00007f683f999fee in connection_wait_for_new_work () #3 0x00007f683f99b719 in connection_threadmain () #4 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so #5 0x00007f683d04cdc5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 Thread 5 (Thread 0x7f67f8fe9700 (LWP 4888)): #0 0x00007f683d0506d5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f683d6a6050 in PR_WaitCondVar () from /lib64/libnspr4.so #2 0x00007f683f999fee in connection_wait_for_new_work () #3 0x00007f683f99b719 in connection_threadmain () #4 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so #5 0x00007f683d04cdc5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 Thread 4 (Thread 0x7f67f87e8700 (LWP 4889)): #0 0x00007f683cd719b3 in select () from /lib64/libc.so.6 #1 0x00007f683f4fc0e9 in DS_Sleep () from /usr/lib64/dirsrv/libslapd.so.0 #2 0x00007f683f99d325 in time_thread () #3 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so #4 0x00007f683d04cdc5 in start_thread () from /lib64/libpthread.so.0 #5 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 Thread 3 (Thread 0x7f67f7de6700 (LWP 4892)): #0 0x00007f683d0506d5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f683d6a6050 in PR_WaitCondVar () from /lib64/libnspr4.so #2 0x00007f683f9a8f25 in ps_send_results () #3 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so #4 0x00007f683d04cdc5 in start_thread () from /lib64/libpthread.so.0 #5 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 Thread 2 (Thread 0x7f67f75e5700 (LWP 5084)): #0 0x00007f683d0506d5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f683d6a6050 in PR_WaitCondVar () from /lib64/libnspr4.so #2 0x00007f683f9a8f25 in ps_send_results () #3 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so #4 0x00007f683d04cdc5 in start_thread () from /lib64/libpthread.so.0 #5 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 Thread 1 (Thread 0x7f683f964840 (LWP 4845)): #0 0x00007f683d0506d5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f683d6a62c3 in PR_EnterMonitor () from /lib64/libnspr4.so #2 0x00007f683f99d5ed in setup_pr_read_pds () #3 0x00007f683f9a0fd9 in slapd_daemon () #4 0x00007f683f992c6c in main () the top output seems ok, there was just a sudden spike to around 340% cpu then it comes down top - 08:53:12 up 6 days, 23:20, 1 user, load average: 0.00, 0.01, 0.12 Threads: 44 total, 0 running, 44 sleeping, 0 stopped, 0 zombie %Cpu(s): 0.0 us, 0.0 sy, 0.0 ni,100.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st KiB Mem : 65533176 total, 52165676 free, 6135088 used, 7232412 buff/cache KiB Swap: 3999996 total, 3999996 free, 0 used. 58452040 avail Mem PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 4845 dirsrv 20 0 5133196 3.150g 281272 S 0.0 5.0 31:34.51 ns-slapd 4847 dirsrv 20 0 5133196 3.150g 281272 S 0.0 5.0 0:13.53 ns-slapd 4848 dirsrv 20 0 5133196 3.150g 281272 S 0.0 5.0 0:13.16 ns-slapd 4849 dirsrv 20 0 5133196 3.150g 281272 S 0.0 5.0 0:45.27 ns-slapd 4850 dirsrv 20 0 5133196 3.150g 281272 S 0.0 5.0 0:01.60 ns-slapd 4851 dirsrv 20 0 5133196 3.150g 281272 S 0.0 5.0 0:00.00 ns-slapd 4852 dirsrv 20 0 5133196 3.150g 281272 S 0.0 5.0 0:00.00 ns-slapd 4853 dirsrv 20 0 5133196 3.150g 281272 S 0.0 5.0 0:00.00 ns-slapd 4854 dirsrv 20 0 5133196 3.150g 281272 S 0.0 5.0 0:00.00 ns-slapd 4855 dirsrv 20 0 5133196 3.150g 281272 S 0.0 5.0 0:03.02 ns-slapd 4856 dirsrv 20 0 5133196 3.150g 281272 S 0.0 5.0 0:00.26 ns-slapd 4859 dirsrv 20 0 5133196 3.150g 281272 S 0.0 5.0 3:18.67 ns-slapd 4860 dirsrv 20 0 5133196 3.150g 281272 S 0.0 5.0 3:19.82 ns-slapd 4861 dirsrv 20 0 5133196 3.150g 281272 S 0.0 5.0 3:20.23 ns-slapd 4862 dirsrv 20 0 5133196 3.150g 281272 S 0.0 5.0 3:22.65 ns-slapd 4863 dirsrv 20 0 5133196 3.150g 281272 S 0.0 5.0 3:16.64 ns-slapd 4864 dirsrv 20 0 5133196 3.150g 281272 S 0.0 5.0 3:17.39 ns-slapd 4865 dirsrv 20 0 5133196 3.150g 281272 S 0.0 5.0 3:17.80 ns-slapd Thanks Rakesh On Tue, Sep 6, 2016 at 12:09 AM, Rakesh Rajasekharan < rakesh.rajasekharan at gmail.com> wrote: > Hi Thierry/Ludwig, > > Unfortunately I had by then restarted the service and since then have been > trying to hit the hang issue again but no luck this time > > It has been running fine for the last few hours now in-spite of trying all > that I believe could cause the hang like making simultaneous client > addition requests. > > I will continue testing this and next time I get it I will make sure I get > the pstack and also get the dump of db lock info. > > Thanks, > Rakesh > > > > On Mon, Sep 5, 2016 at 6:14 PM, Ludwig Krispenz > wrote: > >> >> On 09/05/2016 02:31 PM, thierry bordaz wrote: >> >> Hi Rakesh, >> >> Thanks you soooo much for you continuous help on this !! >> >> If the hang is still going on, would you dump the DB lock information >> with: >> >> db_stat -CA -N -h /var/lib/dirsrv/slapd-/db >/tmp/db_lock_info >> >> It is looking like several threads are waiting for one/several db locks. >> With the lock info we will know which thread acquired it/them. >> A guess is that it could be the checkpointing thread but this one is also >> waiting (on a memory pool). >> >> I am copying Ludwig on this thread in case this pstack rings any bell. >> >> the frequent presence of __db_tas_mutex_lock() looks like the NUMA >> problem of libdb, although the threads already have stopped spinning (maybe >> this was an effect of gcore). >> If the hang aoccurs again, could you check cpu consumption >> top -H -p >> and get a pstack >> pstack >> >> >> best regards >> thierry >> >> On 09/05/2016 02:05 PM, Rakesh Rajasekharan wrote: >> >> Hi Thierry, >> >> I was able to now reporduce the issue.. >> >> the ns-slapd process just got hung,.. however, no core file was generate.. >> >> I manually created a coredump using gcore and then took a strace of it >> this is strace from the coredump generated >> >> GNU gdb (GDB) Red Hat Enterprise Linux 7.6.1-80.el7 >> Copyright (C) 2013 Free Software Foundation, Inc. >> License GPLv3+: GNU GPL version 3 or later > tml> >> This is free software: you are free to change and redistribute it. >> There is NO WARRANTY, to the extent permitted by law. Type "show copying" >> and "show warranty" for details. >> This GDB was configured as "x86_64-redhat-linux-gnu". >> For bug reporting instructions, please see: >> ... >> Reading symbols from /usr/sbin/ns-slapd...Reading symbols from >> /usr/sbin/ns-slapd...(no debugging symbols found)...done. >> (no debugging symbols found)...done. >> [New LWP 105159] >> [New LWP 105160] >> [New LWP 105161] >> [New LWP 105162] >> [New LWP 105163] >> [New LWP 105164] >> [New LWP 105165] >> [New LWP 105166] >> [New LWP 105167] >> [New LWP 105168] >> [New LWP 105171] >> [New LWP 105172] >> [New LWP 105173] >> [New LWP 105174] >> [New LWP 105175] >> [New LWP 105176] >> [New LWP 105177] >> [New LWP 105178] >> [New LWP 105179] >> [New LWP 105180] >> [New LWP 105181] >> [New LWP 105182] >> [New LWP 105183] >> [New LWP 105184] >> [New LWP 105185] >> [New LWP 105186] >> [New LWP 105187] >> [New LWP 105188] >> [New LWP 105189] >> [New LWP 105190] >> [New LWP 105191] >> [New LWP 105192] >> [New LWP 105193] >> [New LWP 105194] >> [New LWP 105195] >> [New LWP 105196] >> [New LWP 105197] >> [New LWP 105198] >> [New LWP 105199] >> [New LWP 105200] >> [New LWP 105201] >> [New LWP 105203] >> [New LWP 105215] >> [New LWP 105158] >> [Thread debugging using libthread_db enabled] >> Using host libthread_db library "/lib64/libthread_db.so.1". >> Core was generated by `/usr/sbin/ns-slapd'. >> #0 0x00007f1b48ae69b3 in select () from /lib64/libc.so.6 >> >> Thread 44 (Thread 0x7f1b4b6d9840 (LWP 105158)): >> #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 () from >> /lib64/libpthread.so.0 >> No symbol table info available. >> #1 0x00007f1b4941b2c3 in PR_EnterMonitor () from /lib64/libnspr4.so >> No symbol table info available. >> #2 0x00007f1b4b7125ed in setup_pr_read_pds () >> No symbol table info available. >> #3 0x00007f1b4b715fd9 in slapd_daemon () >> No symbol table info available. >> #4 0x00007f1b4b707c6c in main () >> No symbol table info available. >> >> Thread 43 (Thread 0x7f1a5df0f700 (LWP 105215)): >> #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 () from >> /lib64/libpthread.so.0 >> No symbol table info available. >> #1 0x00007f1b4941b050 in PR_WaitCondVar () from /lib64/libnspr4.so >> No symbol table info available. >> #2 0x00007f1b4b71df25 in ps_send_results () >> No symbol table info available. >> #3 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so >> No symbol table info available. >> #4 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 >> No symbol table info available. >> #5 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 >> No symbol table info available. >> >> Thread 42 (Thread 0x7f1b03fe7700 (LWP 105203)): >> #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 () from >> /lib64/libpthread.so.0 >> No symbol table info available. >> #1 0x00007f1b4941b050 in PR_WaitCondVar () from /lib64/libnspr4.so >> No symbol table info available. >> #2 0x00007f1b4b71df25 in ps_send_results () >> No symbol table info available. >> #3 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so >> No symbol table info available. >> #4 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 >> No symbol table info available. >> #5 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 >> No symbol table info available. >> >> Thread 41 (Thread 0x7f1b047e8700 (LWP 105201)): >> #0 0x00007f1b48ae69b3 in select () from /lib64/libc.so.6 >> No symbol table info available. >> #1 0x00007f1b4b2710e9 in DS_Sleep () from /usr/lib64/dirsrv/libslapd.so. >> 0 >> No symbol table info available. >> #2 0x00007f1b4b712325 in time_thread () >> No symbol table info available. >> #3 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so >> No symbol table info available. >> #4 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 >> No symbol table info available. >> #5 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 >> No symbol table info available. >> >> Thread 40 (Thread 0x7f1b04fe9700 (LWP 105200)): >> #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 () from >> /lib64/libpthread.so.0 >> No symbol table info available. >> #1 0x00007f1b433212f3 in __db_hybrid_mutex_suspend () from /lib64/ >> libdb-5.3.so >> No symbol table info available. >> #2 0x00007f1b43320640 in __db_tas_mutex_lock () from /lib64/libdb-5.3.so >> No symbol table info available. >> #3 0x00007f1b4343ae1b in __memp_fget () from /lib64/libdb-5.3.so >> No symbol table info available. >> #4 0x00007f1b4333e211 in __bam_search () from /lib64/libdb-5.3.so >> No symbol table info available. >> #5 0x00007f1b43329256 in __bamc_search () from /lib64/libdb-5.3.so >> No symbol table info available. >> #6 0x00007f1b4332d6c4 in __bamc_put () from /lib64/libdb-5.3.so >> No symbol table info available. >> #7 0x00007f1b433e46d5 in __dbc_iput () from /lib64/libdb-5.3.so >> No symbol table info available. >> #8 0x00007f1b433dfa0e in __db_put () from /lib64/libdb-5.3.so >> No symbol table info available. >> #9 0x00007f1b433f4fa4 in __db_put_pp () from /lib64/libdb-5.3.so >> No symbol table info available. >> #10 0x00007f1b3e8cf4cf in idl_new_insert_key () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> No symbol table info available. >> #11 0x00007f1b3e8de5b5 in addordel_values_sv.isra.3 () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> No symbol table info available. >> #12 0x00007f1b3e8df25a in index_addordel_values_ext_sv () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> No symbol table info available. >> #13 0x00007f1b3e8df694 in index_addordel_values_sv () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> No symbol table info available. >> #14 0x00007f1b3e8df75a in index_addordel_entry () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> No symbol table info available. >> #15 0x00007f1b3e8e25fa in ldbm_back_add () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> No symbol table info available. >> #16 0x00007f1b4b1e4cca in op_shared_add () from >> /usr/lib64/dirsrv/libslapd.so.0 >> No symbol table info available. >> #17 0x00007f1b4b1e6020 in do_add () from /usr/lib64/dirsrv/libslapd.so.0 >> No symbol table info available. >> #18 0x00007f1b4b711303 in connection_threadmain () >> No symbol table info available. >> #19 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so >> No symbol table info available. >> #20 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 >> No symbol table info available. >> #21 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 >> No symbol table info available. >> >> Thread 39 (Thread 0x7f1b057ea700 (LWP 105199)): >> #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 () from >> /lib64/libpthread.so.0 >> No symbol table info available. >> #1 0x00007f1b4941b050 in PR_WaitCondVar () from /lib64/libnspr4.so >> No symbol table info available. >> #2 0x00007f1b4b70efee in connection_wait_for_new_work () >> No symbol table info available. >> #3 0x00007f1b4b710719 in connection_threadmain () >> No symbol table info available. >> #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so >> No symbol table info available. >> #5 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 >> No symbol table info available. >> #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 >> No symbol table info available. >> >> Thread 38 (Thread 0x7f1b05feb700 (LWP 105198)): >> #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 () from >> /lib64/libpthread.so.0 >> No symbol table info available. >> #1 0x00007f1b4941b050 in PR_WaitCondVar () from /lib64/libnspr4.so >> No symbol table info available. >> #2 0x00007f1b4b70efee in connection_wait_for_new_work () >> No symbol table info available. >> #3 0x00007f1b4b710719 in connection_threadmain () >> No symbol table info available. >> #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so >> No symbol table info available. >> #5 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 >> No symbol table info available. >> #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 >> No symbol table info available. >> >> Thread 37 (Thread 0x7f1b067ec700 (LWP 105197)): >> #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 () from >> /lib64/libpthread.so.0 >> No symbol table info available. >> #1 0x00007f1b4941b050 in PR_WaitCondVar () from /lib64/libnspr4.so >> No symbol table info available. >> #2 0x00007f1b4b70efee in connection_wait_for_new_work () >> No symbol table info available. >> #3 0x00007f1b4b710719 in connection_threadmain () >> No symbol table info available. >> #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so >> No symbol table info available. >> #5 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 >> No symbol table info available. >> #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 >> No symbol table info available. >> >> Thread 36 (Thread 0x7f1b06fed700 (LWP 105196)): >> #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 () from >> /lib64/libpthread.so.0 >> No symbol table info available. >> #1 0x00007f1b4941b050 in PR_WaitCondVar () from /lib64/libnspr4.so >> No symbol table info available. >> #2 0x00007f1b4b70efee in connection_wait_for_new_work () >> No symbol table info available. >> #3 0x00007f1b4b710719 in connection_threadmain () >> No symbol table info available. >> #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so >> No symbol table info available. >> #5 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 >> No symbol table info available. >> #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 >> No symbol table info available. >> >> Thread 35 (Thread 0x7f1b077ee700 (LWP 105195)): >> #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 () from >> /lib64/libpthread.so.0 >> No symbol table info available. >> #1 0x00007f1b4941b2c3 in PR_EnterMonitor () from /lib64/libnspr4.so >> No symbol table info available. >> #2 0x00007f1b3e8c23e6 in dblayer_txn_begin () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> No symbol table info available. >> #3 0x00007f1b3e8fe007 in ldbm_back_modify () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> No symbol table info available. >> #4 0x00007f1b4b22b541 in op_shared_modify () from >> /usr/lib64/dirsrv/libslapd.so.0 >> No symbol table info available. >> #5 0x00007f1b4b22c004 in modify_internal_pb () from >> /usr/lib64/dirsrv/libslapd.so.0 >> No symbol table info available. >> #6 0x00007f1b3eb48b60 in ipapwd_apply_mods () from >> /usr/lib64/dirsrv/plugins/libipa_pwd_extop.so >> No symbol table info available. >> #7 0x00007f1b3eb4c8e0 in store_new_keys () from >> /usr/lib64/dirsrv/plugins/libipa_pwd_extop.so >> No symbol table info available. >> #8 0x00007f1b3eb4ead0 in ipapwd_extop () from >> /usr/lib64/dirsrv/plugins/libipa_pwd_extop.so >> No symbol table info available. >> #9 0x00007f1b4b23bba2 in plugin_call_exop_plugins () from >> /usr/lib64/dirsrv/libslapd.so.0 >> No symbol table info available. >> #10 0x00007f1b4b717efe in do_extended () >> No symbol table info available. >> #11 0x00007f1b4b71141a in connection_threadmain () >> No symbol table info available. >> #12 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so >> No symbol table info available. >> #13 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 >> No symbol table info available. >> #14 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 >> No symbol table info available. >> >> Thread 34 (Thread 0x7f1b07fef700 (LWP 105194)): >> #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 () from >> /lib64/libpthread.so.0 >> No symbol table info available. >> #1 0x00007f1b4941b050 in PR_WaitCondVar () from /lib64/libnspr4.so >> No symbol table info available. >> #2 0x00007f1b4b70efee in connection_wait_for_new_work () >> No symbol table info available. >> #3 0x00007f1b4b710719 in connection_threadmain () >> No symbol table info available. >> #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so >> No symbol table info available. >> #5 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 >> No symbol table info available. >> #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 >> No symbol table info available. >> >> Thread 33 (Thread 0x7f1b087f0700 (LWP 105193)): >> #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 () from >> /lib64/libpthread.so.0 >> No symbol table info available. >> #1 0x00007f1b4941b050 in PR_WaitCondVar () from /lib64/libnspr4.so >> No symbol table info available. >> #2 0x00007f1b4b70efee in connection_wait_for_new_work () >> No symbol table info available. >> #3 0x00007f1b4b710719 in connection_threadmain () >> No symbol table info available. >> #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so >> No symbol table info available. >> #5 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 >> No symbol table info available. >> #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 >> No symbol table info available. >> >> Thread 32 (Thread 0x7f1b08ff1700 (LWP 105192)): >> #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 () from >> /lib64/libpthread.so.0 >> No symbol table info available. >> #1 0x00007f1b4941b050 in PR_WaitCondVar () from /lib64/libnspr4.so >> No symbol table info available. >> #2 0x00007f1b4b70efee in connection_wait_for_new_work () >> No symbol table info available. >> #3 0x00007f1b4b710719 in connection_threadmain () >> No symbol table info available. >> #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so >> No symbol table info available. >> #5 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 >> No symbol table info available. >> #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 >> No symbol table info available. >> >> Thread 31 (Thread 0x7f1b097f2700 (LWP 105191)): >> #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 () from >> /lib64/libpthread.so.0 >> No symbol table info available. >> #1 0x00007f1b4941b050 in PR_WaitCondVar () from /lib64/libnspr4.so >> No symbol table info available. >> #2 0x00007f1b4b70efee in connection_wait_for_new_work () >> No symbol table info available. >> #3 0x00007f1b4b710719 in connection_threadmain () >> No symbol table info available. >> #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so >> No symbol table info available. >> #5 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 >> No symbol table info available. >> #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 >> No symbol table info available. >> >> Thread 30 (Thread 0x7f1b09ff3700 (LWP 105190)): >> #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 () from >> /lib64/libpthread.so.0 >> No symbol table info available. >> #1 0x00007f1b433212f3 in __db_hybrid_mutex_suspend () from /lib64/ >> libdb-5.3.so >> No symbol table info available. >> #2 0x00007f1b43320640 in __db_tas_mutex_lock () from /lib64/libdb-5.3.so >> No symbol table info available. >> #3 0x00007f1b433cad3a in __lock_get_internal () from /lib64/libdb-5.3.so >> No symbol table info available. >> #4 0x00007f1b433cb820 in __lock_get () from /lib64/libdb-5.3.so >> No symbol table info available. >> #5 0x00007f1b433f7142 in __db_lget () from /lib64/libdb-5.3.so >> No symbol table info available. >> #6 0x00007f1b4333e605 in __bam_search () from /lib64/libdb-5.3.so >> No symbol table info available. >> #7 0x00007f1b43329256 in __bamc_search () from /lib64/libdb-5.3.so >> No symbol table info available. >> #8 0x00007f1b4332ad0f in __bamc_get () from /lib64/libdb-5.3.so >> No symbol table info available. >> #9 0x00007f1b433e3ca6 in __dbc_iget () from /lib64/libdb-5.3.so >> No symbol table info available. >> #10 0x00007f1b433f2b02 in __dbc_get_pp () from /lib64/libdb-5.3.so >> No symbol table info available. >> #11 0x00007f1b3e8ce680 in idl_new_fetch () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> No symbol table info available. >> #12 0x00007f1b3e8dcb26 in index_read_ext_allids () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> No symbol table info available. >> #13 0x00007f1b3e8c7284 in keys2idl () from /usr/lib64/dirsrv/plugins/libb >> ack-ldbm.so >> No symbol table info available. >> #14 0x00007f1b3e8c79e3 in ava_candidates.isra.0 () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> No symbol table info available. >> #15 0x00007f1b3e8c7fd2 in filter_candidates_ext () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> No symbol table info available. >> #16 0x00007f1b3e8c9096 in list_candidates () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> No symbol table info available. >> #17 0x00007f1b3e8c7f40 in filter_candidates_ext () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> No symbol table info available. >> #18 0x00007f1b3e8c9096 in list_candidates () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> No symbol table info available. >> #19 0x00007f1b3e8c7f40 in filter_candidates_ext () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> No symbol table info available. >> #20 0x00007f1b3e8c9096 in list_candidates () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> No symbol table info available. >> #21 0x00007f1b3e8c7f40 in filter_candidates_ext () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> No symbol table info available. >> #22 0x00007f1b3e90380c in subtree_candidates () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> No symbol table info available. >> #23 0x00007f1b3e904e7f in ldbm_back_search () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> No symbol table info available. >> #24 0x00007f1b4b2321d6 in op_shared_search () from >> /usr/lib64/dirsrv/libslapd.so.0 >> No symbol table info available. >> #25 0x00007f1b4b72333e in do_search () >> No symbol table info available. >> #26 0x00007f1b4b7113c3 in connection_threadmain () >> No symbol table info available. >> #27 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so >> No symbol table info available. >> #28 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 >> No symbol table info available. >> #29 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 >> No symbol table info available. >> >> Thread 29 (Thread 0x7f1b0a7f4700 (LWP 105189)): >> #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 () from >> /lib64/libpthread.so.0 >> No symbol table info available. >> #1 0x00007f1b4941b050 in PR_WaitCondVar () from /lib64/libnspr4.so >> No symbol table info available. >> #2 0x00007f1b4b70efee in connection_wait_for_new_work () >> No symbol table info available. >> #3 0x00007f1b4b710719 in connection_threadmain () >> No symbol table info available. >> #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so >> No symbol table info available. >> #5 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 >> No symbol table info available. >> #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 >> No symbol table info available. >> >> Thread 28 (Thread 0x7f1b0aff5700 (LWP 105188)): >> #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 () from >> /lib64/libpthread.so.0 >> No symbol table info available. >> #1 0x00007f1b4941b050 in PR_WaitCondVar () from /lib64/libnspr4.so >> No symbol table info available. >> #2 0x00007f1b4b70efee in connection_wait_for_new_work () >> No symbol table info available. >> #3 0x00007f1b4b710719 in connection_threadmain () >> No symbol table info available. >> #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so >> No symbol table info available. >> #5 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 >> No symbol table info available. >> #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 >> No symbol table info available. >> >> Thread 27 (Thread 0x7f1b0b7f6700 (LWP 105187)): >> #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 () from >> /lib64/libpthread.so.0 >> No symbol table info available. >> #1 0x00007f1b4941b050 in PR_WaitCondVar () from /lib64/libnspr4.so >> No symbol table info available. >> #2 0x00007f1b4b70efee in connection_wait_for_new_work () >> No symbol table info available. >> #3 0x00007f1b4b710719 in connection_threadmain () >> No symbol table info available. >> #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so >> No symbol table info available. >> #5 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 >> No symbol table info available. >> #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 >> No symbol table info available. >> >> Thread 26 (Thread 0x7f1b0bff7700 (LWP 105186)): >> #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 () from >> /lib64/libpthread.so.0 >> No symbol table info available. >> #1 0x00007f1b433212f3 in __db_hybrid_mutex_suspend () from /lib64/ >> libdb-5.3.so >> No symbol table info available. >> #2 0x00007f1b43320640 in __db_tas_mutex_lock () from /lib64/libdb-5.3.so >> No symbol table info available. >> #3 0x00007f1b433cad3a in __lock_get_internal () from /lib64/libdb-5.3.so >> No symbol table info available. >> #4 0x00007f1b433cb820 in __lock_get () from /lib64/libdb-5.3.so >> No symbol table info available. >> #5 0x00007f1b433f7142 in __db_lget () from /lib64/libdb-5.3.so >> No symbol table info available. >> #6 0x00007f1b4333e605 in __bam_search () from /lib64/libdb-5.3.so >> No symbol table info available. >> #7 0x00007f1b43329256 in __bamc_search () from /lib64/libdb-5.3.so >> No symbol table info available. >> #8 0x00007f1b4332ad0f in __bamc_get () from /lib64/libdb-5.3.so >> No symbol table info available. >> #9 0x00007f1b433e3ca6 in __dbc_iget () from /lib64/libdb-5.3.so >> No symbol table info available. >> #10 0x00007f1b433f2b02 in __dbc_get_pp () from /lib64/libdb-5.3.so >> No symbol table info available. >> #11 0x00007f1b3e8ce680 in idl_new_fetch () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> No symbol table info available. >> #12 0x00007f1b3e8dcb26 in index_read_ext_allids () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> No symbol table info available. >> #13 0x00007f1b3e8c7284 in keys2idl () from /usr/lib64/dirsrv/plugins/libb >> ack-ldbm.so >> No symbol table info available. >> #14 0x00007f1b3e8c79e3 in ava_candidates.isra.0 () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> No symbol table info available. >> #15 0x00007f1b3e8c7fd2 in filter_candidates_ext () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> No symbol table info available. >> #16 0x00007f1b3e8c9096 in list_candidates () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> No symbol table info available. >> #17 0x00007f1b3e8c7f40 in filter_candidates_ext () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> No symbol table info available. >> #18 0x00007f1b3e90380c in subtree_candidates () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> No symbol table info available. >> #19 0x00007f1b3e904e7f in ldbm_back_search () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> No symbol table info available. >> #20 0x00007f1b4b2321d6 in op_shared_search () from >> /usr/lib64/dirsrv/libslapd.so.0 >> No symbol table info available. >> #21 0x00007f1b4b24294e in search_internal_callback_pb () from >> /usr/lib64/dirsrv/libslapd.so.0 >> No symbol table info available. >> #22 0x00007f1b4b242be8 in search_internal_pb () from >> /usr/lib64/dirsrv/libslapd.so.0 >> No symbol table info available. >> #23 0x00007f1b4b72149b in ids_sasl_canon_user () >> No symbol table info available. >> #24 0x00007f1b491e193b in _sasl_canon_user () from /lib64/libsasl2.so.3 >> No symbol table info available. >> #25 0x00007f1b491e1c4c in _sasl_canon_user_lookup () from >> /lib64/libsasl2.so.3 >> No symbol table info available. >> #26 0x00007f1b430e9fb9 in gssapi_server_mech_step () from >> /usr/lib64/sasl2/libgssapiv2.so >> No symbol table info available. >> #27 0x00007f1b491edb9b in sasl_server_step () from /lib64/libsasl2.so.3 >> No symbol table info available. >> #28 0x00007f1b4b72283b in ids_sasl_check_bind () >> No symbol table info available. >> #29 0x00007f1b4b70a61a in do_bind () >> No symbol table info available. >> #30 0x00007f1b4b7113fd in connection_threadmain () >> No symbol table info available. >> #31 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so >> No symbol table info available. >> #32 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 >> No symbol table info available. >> #33 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 >> No symbol table info available. >> >> Thread 25 (Thread 0x7f1b0c7f8700 (LWP 105185)): >> #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 () from >> /lib64/libpthread.so.0 >> No symbol table info available. >> #1 0x00007f1b4941b050 in PR_WaitCondVar () from /lib64/libnspr4.so >> No symbol table info available. >> #2 0x00007f1b4b70efee in connection_wait_for_new_work () >> No symbol table info available. >> #3 0x00007f1b4b710719 in connection_threadmain () >> No symbol table info available. >> #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so >> No symbol table info available. >> #5 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 >> No symbol table info available. >> #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 >> No symbol table info available. >> >> Thread 24 (Thread 0x7f1b0cff9700 (LWP 105184)): >> #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 () from >> /lib64/libpthread.so.0 >> No symbol table info available. >> #1 0x00007f1b4941b050 in PR_WaitCondVar () from /lib64/libnspr4.so >> No symbol table info available. >> #2 0x00007f1b4b70efee in connection_wait_for_new_work () >> No symbol table info available. >> #3 0x00007f1b4b710719 in connection_threadmain () >> No symbol table info available. >> #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so >> No symbol table info available. >> #5 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 >> No symbol table info available. >> #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 >> No symbol table info available. >> >> Thread 23 (Thread 0x7f1b0d7fa700 (LWP 105183)): >> #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 () from >> /lib64/libpthread.so.0 >> No symbol table info available. >> #1 0x00007f1b4941b050 in PR_WaitCondVar () from /lib64/libnspr4.so >> No symbol table info available. >> #2 0x00007f1b4b70efee in connection_wait_for_new_work () >> No symbol table info available. >> #3 0x00007f1b4b710719 in connection_threadmain () >> No symbol table info available. >> #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so >> No symbol table info available. >> #5 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 >> No symbol table info available. >> #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 >> No symbol table info available. >> >> Thread 22 (Thread 0x7f1b0dffb700 (LWP 105182)): >> #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 () from >> /lib64/libpthread.so.0 >> No symbol table info available. >> #1 0x00007f1b4941b050 in PR_WaitCondVar () from /lib64/libnspr4.so >> No symbol table info available. >> #2 0x00007f1b4b70efee in connection_wait_for_new_work () >> No symbol table info available. >> #3 0x00007f1b4b710719 in connection_threadmain () >> No symbol table info available. >> #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so >> No symbol table info available. >> #5 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 >> No symbol table info available. >> #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 >> No symbol table info available. >> >> Thread 21 (Thread 0x7f1b0e7fc700 (LWP 105181)): >> #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 () from >> /lib64/libpthread.so.0 >> No symbol table info available. >> #1 0x00007f1b4941b050 in PR_WaitCondVar () from /lib64/libnspr4.so >> No symbol table info available. >> #2 0x00007f1b4b70efee in connection_wait_for_new_work () >> No symbol table info available. >> #3 0x00007f1b4b710719 in connection_threadmain () >> No symbol table info available. >> #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so >> No symbol table info available. >> #5 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 >> No symbol table info available. >> #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 >> No symbol table info available. >> >> Thread 20 (Thread 0x7f1b0effd700 (LWP 105180)): >> #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 () from >> /lib64/libpthread.so.0 >> No symbol table info available. >> #1 0x00007f1b433212f3 in __db_hybrid_mutex_suspend () from /lib64/ >> libdb-5.3.so >> No symbol table info available. >> #2 0x00007f1b43320b37 in __db_tas_mutex_readlock () from /lib64/ >> libdb-5.3.so >> No symbol table info available. >> #3 0x00007f1b4343ad35 in __memp_fget () from /lib64/libdb-5.3.so >> No symbol table info available. >> #4 0x00007f1b433e4333 in __dbc_iget () from /lib64/libdb-5.3.so >> No symbol table info available. >> #5 0x00007f1b433f2b02 in __dbc_get_pp () from /lib64/libdb-5.3.so >> No symbol table info available. >> #6 0x00007f1b3e8ce822 in idl_new_fetch () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> No symbol table info available. >> #7 0x00007f1b3e8dcb26 in index_read_ext_allids () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> No symbol table info available. >> #8 0x00007f1b3e8c7284 in keys2idl () from /usr/lib64/dirsrv/plugins/libb >> ack-ldbm.so >> No symbol table info available. >> #9 0x00007f1b3e8c79e3 in ava_candidates.isra.0 () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> No symbol table info available. >> #10 0x00007f1b3e8c7fd2 in filter_candidates_ext () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> No symbol table info available. >> #11 0x00007f1b3e8c9096 in list_candidates () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> No symbol table info available. >> #12 0x00007f1b3e8c7f40 in filter_candidates_ext () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> No symbol table info available. >> #13 0x00007f1b3e8c9096 in list_candidates () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> No symbol table info available. >> #14 0x00007f1b3e8c7f40 in filter_candidates_ext () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> No symbol table info available. >> #15 0x00007f1b3e8c9096 in list_candidates () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> No symbol table info available. >> #16 0x00007f1b3e8c7f40 in filter_candidates_ext () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> No symbol table info available. >> #17 0x00007f1b3e90380c in subtree_candidates () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> No symbol table info available. >> #18 0x00007f1b3e904e7f in ldbm_back_search () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> No symbol table info available. >> #19 0x00007f1b4b2321d6 in op_shared_search () from >> /usr/lib64/dirsrv/libslapd.so.0 >> No symbol table info available. >> #20 0x00007f1b4b72333e in do_search () >> No symbol table info available. >> #21 0x00007f1b4b7113c3 in connection_threadmain () >> No symbol table info available. >> #22 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so >> No symbol table info available. >> #23 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 >> No symbol table info available. >> #24 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 >> No symbol table info available. >> >> Thread 19 (Thread 0x7f1b0f7fe700 (LWP 105179)): >> #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 () from >> /lib64/libpthread.so.0 >> No symbol table info available. >> #1 0x00007f1b433212f3 in __db_hybrid_mutex_suspend () from /lib64/ >> libdb-5.3.so >> No symbol table info available. >> #2 0x00007f1b43320640 in __db_tas_mutex_lock () from /lib64/libdb-5.3.so >> No symbol table info available. >> #3 0x00007f1b433cad3a in __lock_get_internal () from /lib64/libdb-5.3.so >> No symbol table info available. >> #4 0x00007f1b433cb820 in __lock_get () from /lib64/libdb-5.3.so >> No symbol table info available. >> #5 0x00007f1b433f7142 in __db_lget () from /lib64/libdb-5.3.so >> No symbol table info available. >> #6 0x00007f1b4333e605 in __bam_search () from /lib64/libdb-5.3.so >> No symbol table info available. >> #7 0x00007f1b43329256 in __bamc_search () from /lib64/libdb-5.3.so >> No symbol table info available. >> #8 0x00007f1b4332ad0f in __bamc_get () from /lib64/libdb-5.3.so >> No symbol table info available. >> #9 0x00007f1b433e3ca6 in __dbc_iget () from /lib64/libdb-5.3.so >> No symbol table info available. >> #10 0x00007f1b433f2b02 in __dbc_get_pp () from /lib64/libdb-5.3.so >> No symbol table info available. >> #11 0x00007f1b3e8ce680 in idl_new_fetch () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> No symbol table info available. >> #12 0x00007f1b3e8dcb26 in index_read_ext_allids () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> No symbol table info available. >> #13 0x00007f1b3e8c7284 in keys2idl () from /usr/lib64/dirsrv/plugins/libb >> ack-ldbm.so >> No symbol table info available. >> #14 0x00007f1b3e8c79e3 in ava_candidates.isra.0 () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> No symbol table info available. >> #15 0x00007f1b3e8c7fd2 in filter_candidates_ext () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> No symbol table info available. >> #16 0x00007f1b3e8c9096 in list_candidates () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> No symbol table info available. >> #17 0x00007f1b3e8c7f40 in filter_candidates_ext () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> No symbol table info available. >> #18 0x00007f1b3e8c9096 in list_candidates () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> No symbol table info available. >> #19 0x00007f1b3e8c7f40 in filter_candidates_ext () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> No symbol table info available. >> #20 0x00007f1b3e8c9096 in list_candidates () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> No symbol table info available. >> #21 0x00007f1b3e8c7f40 in filter_candidates_ext () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> No symbol table info available. >> #22 0x00007f1b3e90380c in subtree_candidates () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> No symbol table info available. >> #23 0x00007f1b3e904e7f in ldbm_back_search () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> No symbol table info available. >> #24 0x00007f1b4b2321d6 in op_shared_search () from >> /usr/lib64/dirsrv/libslapd.so.0 >> No symbol table info available. >> #25 0x00007f1b4b72333e in do_search () >> No symbol table info available. >> #26 0x00007f1b4b7113c3 in connection_threadmain () >> No symbol table info available. >> #27 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so >> No symbol table info available. >> #28 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 >> No symbol table info available. >> #29 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 >> No symbol table info available. >> >> Thread 18 (Thread 0x7f1b0ffff700 (LWP 105178)): >> #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 () from >> /lib64/libpthread.so.0 >> No symbol table info available. >> #1 0x00007f1b4941b050 in PR_WaitCondVar () from /lib64/libnspr4.so >> No symbol table info available. >> #2 0x00007f1b4b70efee in connection_wait_for_new_work () >> No symbol table info available. >> #3 0x00007f1b4b710719 in connection_threadmain () >> No symbol table info available. >> #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so >> No symbol table info available. >> #5 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 >> No symbol table info available. >> #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 >> No symbol table info available. >> >> Thread 17 (Thread 0x7f1b14d28700 (LWP 105177)): >> #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 () from >> /lib64/libpthread.so.0 >> No symbol table info available. >> #1 0x00007f1b4941b050 in PR_WaitCondVar () from /lib64/libnspr4.so >> No symbol table info available. >> #2 0x00007f1b4b70efee in connection_wait_for_new_work () >> No symbol table info available. >> #3 0x00007f1b4b710719 in connection_threadmain () >> No symbol table info available. >> #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so >> No symbol table info available. >> #5 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 >> No symbol table info available. >> #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 >> No symbol table info available. >> >> Thread 16 (Thread 0x7f1b15529700 (LWP 105176)): >> #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 () from >> /lib64/libpthread.so.0 >> No symbol table info available. >> #1 0x00007f1b4941b050 in PR_WaitCondVar () from /lib64/libnspr4.so >> No symbol table info available. >> #2 0x00007f1b4b70efee in connection_wait_for_new_work () >> No symbol table info available. >> #3 0x00007f1b4b710719 in connection_threadmain () >> No symbol table info available. >> #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so >> No symbol table info available. >> #5 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 >> No symbol table info available. >> #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 >> No symbol table info available. >> >> Thread 15 (Thread 0x7f1b15d2a700 (LWP 105175)): >> #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 () from >> /lib64/libpthread.so.0 >> No symbol table info available. >> #1 0x00007f1b4941b050 in PR_WaitCondVar () from /lib64/libnspr4.so >> No symbol table info available. >> #2 0x00007f1b4b70efee in connection_wait_for_new_work () >> No symbol table info available. >> #3 0x00007f1b4b710719 in connection_threadmain () >> No symbol table info available. >> #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so >> No symbol table info available. >> #5 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 >> No symbol table info available. >> #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 >> No symbol table info available. >> >> Thread 14 (Thread 0x7f1b1652b700 (LWP 105174)): >> #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 () from >> /lib64/libpthread.so.0 >> No symbol table info available. >> #1 0x00007f1b4941b050 in PR_WaitCondVar () from /lib64/libnspr4.so >> No symbol table info available. >> #2 0x00007f1b4b70efee in connection_wait_for_new_work () >> No symbol table info available. >> #3 0x00007f1b4b710719 in connection_threadmain () >> No symbol table info available. >> #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so >> No symbol table info available. >> #5 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 >> No symbol table info available. >> #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 >> No symbol table info available. >> >> Thread 13 (Thread 0x7f1b16d2c700 (LWP 105173)): >> #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 () from >> /lib64/libpthread.so.0 >> No symbol table info available. >> #1 0x00007f1b433212f3 in __db_hybrid_mutex_suspend () from /lib64/ >> libdb-5.3.so >> No symbol table info available. >> #2 0x00007f1b43320640 in __db_tas_mutex_lock () from /lib64/libdb-5.3.so >> No symbol table info available. >> #3 0x00007f1b433cad3a in __lock_get_internal () from /lib64/libdb-5.3.so >> No symbol table info available. >> #4 0x00007f1b433cb820 in __lock_get () from /lib64/libdb-5.3.so >> No symbol table info available. >> #5 0x00007f1b433f7142 in __db_lget () from /lib64/libdb-5.3.so >> No symbol table info available. >> #6 0x00007f1b4333e605 in __bam_search () from /lib64/libdb-5.3.so >> No symbol table info available. >> #7 0x00007f1b43329256 in __bamc_search () from /lib64/libdb-5.3.so >> No symbol table info available. >> #8 0x00007f1b4332ad0f in __bamc_get () from /lib64/libdb-5.3.so >> No symbol table info available. >> #9 0x00007f1b433e3ca6 in __dbc_iget () from /lib64/libdb-5.3.so >> No symbol table info available. >> #10 0x00007f1b433f2b02 in __dbc_get_pp () from /lib64/libdb-5.3.so >> No symbol table info available. >> #11 0x00007f1b3e8ce680 in idl_new_fetch () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> No symbol table info available. >> #12 0x00007f1b3e8dcb26 in index_read_ext_allids () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> No symbol table info available. >> #13 0x00007f1b3e8c7284 in keys2idl () from /usr/lib64/dirsrv/plugins/libb >> ack-ldbm.so >> No symbol table info available. >> #14 0x00007f1b3e8c79e3 in ava_candidates.isra.0 () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> No symbol table info available. >> #15 0x00007f1b3e8c7fd2 in filter_candidates_ext () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> No symbol table info available. >> #16 0x00007f1b3e8c9096 in list_candidates () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> No symbol table info available. >> #17 0x00007f1b3e8c7f40 in filter_candidates_ext () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> No symbol table info available. >> #18 0x00007f1b3e8c9096 in list_candidates () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> No symbol table info available. >> #19 0x00007f1b3e8c7f40 in filter_candidates_ext () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> No symbol table info available. >> #20 0x00007f1b3e8c9096 in list_candidates () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> No symbol table info available. >> #21 0x00007f1b3e8c7f40 in filter_candidates_ext () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> No symbol table info available. >> #22 0x00007f1b3e90380c in subtree_candidates () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> No symbol table info available. >> #23 0x00007f1b3e904e7f in ldbm_back_search () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> No symbol table info available. >> #24 0x00007f1b4b2321d6 in op_shared_search () from >> /usr/lib64/dirsrv/libslapd.so.0 >> No symbol table info available. >> #25 0x00007f1b4b72333e in do_search () >> No symbol table info available. >> #26 0x00007f1b4b7113c3 in connection_threadmain () >> No symbol table info available. >> #27 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so >> No symbol table info available. >> #28 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 >> No symbol table info available. >> #29 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 >> No symbol table info available. >> >> Thread 12 (Thread 0x7f1b1752d700 (LWP 105172)): >> #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 () from >> /lib64/libpthread.so.0 >> No symbol table info available. >> #1 0x00007f1b4941b050 in PR_WaitCondVar () from /lib64/libnspr4.so >> No symbol table info available. >> #2 0x00007f1b4b70efee in connection_wait_for_new_work () >> No symbol table info available. >> #3 0x00007f1b4b710719 in connection_threadmain () >> No symbol table info available. >> #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so >> No symbol table info available. >> #5 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 >> No symbol table info available. >> #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 >> No symbol table info available. >> >> Thread 11 (Thread 0x7f1b17d2e700 (LWP 105171)): >> #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 () from >> /lib64/libpthread.so.0 >> No symbol table info available. >> #1 0x00007f1b4941b050 in PR_WaitCondVar () from /lib64/libnspr4.so >> No symbol table info available. >> #2 0x00007f1b4b70efee in connection_wait_for_new_work () >> No symbol table info available. >> #3 0x00007f1b4b710719 in connection_threadmain () >> No symbol table info available. >> #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so >> No symbol table info available. >> #5 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 >> No symbol table info available. >> #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 >> No symbol table info available. >> >> Thread 10 (Thread 0x7f1b2509d700 (LWP 105168)): >> #0 0x00007f1b48dc5a82 in pthread_cond_timedwait@@GLIBC_2.3.2 () from >> /lib64/libpthread.so.0 >> No symbol table info available. >> #1 0x00007f1b4941ab07 in pt_TimedWait () from /lib64/libnspr4.so >> No symbol table info available. >> #2 0x00007f1b4941afce in PR_WaitCondVar () from /lib64/libnspr4.so >> No symbol table info available. >> #3 0x00007f1b4b2061e8 in eq_loop () from /usr/lib64/dirsrv/libslapd.so.0 >> No symbol table info available. >> #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so >> No symbol table info available. >> #5 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 >> No symbol table info available. >> #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 >> No symbol table info available. >> >> Thread 9 (Thread 0x7f1b2589e700 (LWP 105167)): >> #0 0x00007f1b48dc5a82 in pthread_cond_timedwait@@GLIBC_2.3.2 () from >> /lib64/libpthread.so.0 >> No symbol table info available. >> #1 0x00007f1b4941ab07 in pt_TimedWait () from /lib64/libnspr4.so >> No symbol table info available. >> #2 0x00007f1b4941afce in PR_WaitCondVar () from /lib64/libnspr4.so >> No symbol table info available. >> #3 0x00007f1b4b719d03 in housecleaning () >> No symbol table info available. >> #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so >> No symbol table info available. >> #5 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 >> No symbol table info available. >> #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 >> No symbol table info available. >> >> Thread 8 (Thread 0x7f1b2609f700 (LWP 105166)): >> #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 () from >> /lib64/libpthread.so.0 >> No symbol table info available. >> #1 0x00007f1b4941b050 in PR_WaitCondVar () from /lib64/libnspr4.so >> No symbol table info available. >> #2 0x00007f1b4b260198 in slapi_wait_condvar () from >> /usr/lib64/dirsrv/libslapd.so.0 >> No symbol table info available. >> #3 0x00007f1b3cf5deed in roles_cache_wait_on_change () from >> /usr/lib64/dirsrv/plugins/libroles-plugin.so >> No symbol table info available. >> #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so >> No symbol table info available. >> #5 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 >> No symbol table info available. >> #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 >> No symbol table info available. >> >> Thread 7 (Thread 0x7f1b268a0700 (LWP 105165)): >> #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 () from >> /lib64/libpthread.so.0 >> No symbol table info available. >> #1 0x00007f1b4941b050 in PR_WaitCondVar () from /lib64/libnspr4.so >> No symbol table info available. >> #2 0x00007f1b4b260198 in slapi_wait_condvar () from >> /usr/lib64/dirsrv/libslapd.so.0 >> No symbol table info available. >> #3 0x00007f1b3cf5deed in roles_cache_wait_on_change () from >> /usr/lib64/dirsrv/plugins/libroles-plugin.so >> No symbol table info available. >> #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so >> No symbol table info available. >> #5 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 >> No symbol table info available. >> #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 >> No symbol table info available. >> >> Thread 6 (Thread 0x7f1b270a1700 (LWP 105164)): >> #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 () from >> /lib64/libpthread.so.0 >> No symbol table info available. >> #1 0x00007f1b4941b050 in PR_WaitCondVar () from /lib64/libnspr4.so >> No symbol table info available. >> #2 0x00007f1b4b260198 in slapi_wait_condvar () from >> /usr/lib64/dirsrv/libslapd.so.0 >> No symbol table info available. >> #3 0x00007f1b3cf5deed in roles_cache_wait_on_change () from >> /usr/lib64/dirsrv/plugins/libroles-plugin.so >> No symbol table info available. >> #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so >> No symbol table info available. >> #5 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 >> No symbol table info available. >> #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 >> No symbol table info available. >> >> Thread 5 (Thread 0x7f1b278a2700 (LWP 105163)): >> #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 () from >> /lib64/libpthread.so.0 >> No symbol table info available. >> #1 0x00007f1b4941b050 in PR_WaitCondVar () from /lib64/libnspr4.so >> No symbol table info available. >> #2 0x00007f1b4b260198 in slapi_wait_condvar () from >> /usr/lib64/dirsrv/libslapd.so.0 >> No symbol table info available. >> #3 0x00007f1b415f262e in cos_cache_wait_on_change () from >> /usr/lib64/dirsrv/plugins/libcos-plugin.so >> No symbol table info available. >> #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so >> No symbol table info available. >> #5 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 >> No symbol table info available. >> #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 >> No symbol table info available. >> >> Thread 4 (Thread 0x7f1b282ac700 (LWP 105162)): >> #0 0x00007f1b48ae69b3 in select () from /lib64/libc.so.6 >> No symbol table info available. >> #1 0x00007f1b4b2710e9 in DS_Sleep () from /usr/lib64/dirsrv/libslapd.so. >> 0 >> No symbol table info available. >> #2 0x00007f1b3e8ba707 in perf_threadmain () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> No symbol table info available. >> #3 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so >> No symbol table info available. >> #4 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 >> No symbol table info available. >> #5 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 >> No symbol table info available. >> >> Thread 3 (Thread 0x7f1b28aad700 (LWP 105161)): >> #0 0x00007f1b48ae69b3 in select () from /lib64/libc.so.6 >> No symbol table info available. >> #1 0x00007f1b4b2710e9 in DS_Sleep () from /usr/lib64/dirsrv/libslapd.so. >> 0 >> No symbol table info available. >> #2 0x00007f1b3e8bfb7f in trickle_threadmain () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> No symbol table info available. >> #3 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so >> No symbol table info available. >> #4 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 >> No symbol table info available. >> #5 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 >> No symbol table info available. >> >> Thread 2 (Thread 0x7f1b292ae700 (LWP 105160)): >> #0 0x00007f1b48ae69b3 in select () from /lib64/libc.so.6 >> No symbol table info available. >> #1 0x00007f1b4344dbad in __os_yield () from /lib64/libdb-5.3.so >> No symbol table info available. >> #2 0x00007f1b434492b3 in __memp_sync_int () from /lib64/libdb-5.3.so >> No symbol table info available. >> #3 0x00007f1b43459752 in __txn_checkpoint () from /lib64/libdb-5.3.so >> No symbol table info available. >> #4 0x00007f1b43459b74 in __txn_checkpoint_pp () from /lib64/libdb-5.3.so >> No symbol table info available. >> #5 0x00007f1b3e8c3a87 in checkpoint_threadmain () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> No symbol table info available. >> #6 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so >> No symbol table info available. >> #7 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 >> No symbol table info available. >> #8 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 >> No symbol table info available. >> >> Thread 1 (Thread 0x7f1b29aaf700 (LWP 105159)): >> #0 0x00007f1b48ae69b3 in select () from /lib64/libc.so.6 >> No symbol table info available. >> #1 0x00007f1b4b2710e9 in DS_Sleep () from /usr/lib64/dirsrv/libslapd.so. >> 0 >> No symbol table info available. >> #2 0x00007f1b3e8bf907 in deadlock_threadmain () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> No symbol table info available. >> #3 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so >> No symbol table info available. >> #4 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 >> No symbol table info available. >> #5 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 >> No symbol table info available. >> >> >> >> Thanks, >> >> Rakesh >> >> >> On Mon, Sep 5, 2016 at 4:48 PM, thierry bordaz >> wrote: >> >>> >>> >>> On 09/05/2016 12:05 PM, Rakesh Rajasekharan wrote: >>> >>> Hi Thierry, >>> >>> >>> I was getting the hang issue while running ipa-client-install >>> simultaneously on few clients.. >>> However, today, I am not able to replicate that. >>> >>> I could not get a gdb . But i will try getting that the next time I face >>> this issue. >>> >>> The CPU does not stay high.. it just momentarily touches a high value >>> and then drops down to around 2-7% >>> >>> One question I have is , is it ok to set it nsslapd-threadnumber to a >>> very high value . >>> I have around 4000 clients and with nsslapd-maxthreadsperconn set to >>> 5..So, can I set nsslapd-threadnumber to around 25000. >>> >>> >>> Hello, >>> >>> I know some users running in production with several hunderds of threads >>> (>600) and this without problem. >>> >>> I do not recall having suggested to increase that number and for what >>> reason. >>> Usually 30 workers is a good enough value. It can create bootleneck if >>> for some reason each operation is very long to satisfy and exhaust the >>> number of workers. You can monitor the work queue: >>> >>> ldapsearch -D "cn=directory manager" -w xxx -LLL -b "cn=monitor" -s >>> base opsinitiated opscompleted >>> >>> >>> If opscompleted-opsinitiated remains close to threadnumber, then yes it >>> would be valuable to increase it. >>> >>> The computation #client * #async_op_per_client sound an overkill. Even >>> if all clients send at the exact same time all their requests, it is very >>> likely that some common resource (db page, log, allocator...) will >>> serialize them. If you monitor a need to increase the work, you would for >>> example set it to 50, then monitor, then set it to 100, then monitor... >>> until you find a good enough value. >>> Note the increasing the #thread, increases the memory footprint that >>> will reduce the efficiency of file system cache and can increase the >>> response time. >>> >>> >>> best regards >>> thierry >>> >>> >>> Thanks >>> >>> On Mon, Sep 5, 2016 at 1:03 PM, thierry bordaz >>> wrote: >>> >>>> >>>> Hi Rakesh, >>>> >>>> Were you able to get a pstack or full stack with gdb ( >>>> http://www.port389.org/docs/389ds/FAQ/faq.html#debugging-crashes) when >>>> the server hangs ? >>>> >>>> If it happens with 500 threads as well as with 30, using 30 threads is >>>> a better choice to debug this issue. >>>> I will try to reproduce using 150 parallel 'ipa user-find p-testipa' >>>> commands >>>> >>>> Something I am unsure is if the CPU consumption stays high (you >>>> mentioned 340% CPU usage) as long as the hang happens or if after a suddent >>>> shot up to 340% (that marks the beginning of the hang) it drops and stay >>>> hanging ? >>>> >>>> thanks >>>> thierry >>>> >>>> On 09/04/2016 08:40 PM, Rakesh Rajasekharan wrote: >>>> >>>> starce on the slapd process actually had this in the output.. >>>> FUTEX_WAIT_PRIVATE >>>> >>>> and checking for the number of threads slapd had.. there were 5015 >>>> threads >>>> >>>> ps -efL|grep slapd|wc -l >>>> 5015 >>>> >>>> strace on most of the threads gave this output >>>> >>>> strace -p 67411 >>>> Process 67411 attached >>>> futex(0x7f3f0226b41c, FUTEX_WAIT_PRIVATE, 1, NULL) = -1 EAGAIN >>>> (Resource temporarily unavailable) >>>> futex(0x7f3f0226b41c, FUTEX_WAIT_PRIVATE, 2, NULL^CProcess 67411 >>>> detached >>>> >>>> >>>> >>>> >>>> >>>> On Sun, Sep 4, 2016 at 5:34 PM, Rakesh Rajasekharan < >>>> rakesh.rajasekharan at gmail.com> wrote: >>>> >>>>> I have again got the issue of IPA hanging.. The issue came up when i >>>>> tried to run ipa-client-isntall on 142 clients simultaneously >>>>> >>>>> >>>>> None of the IPA commands are responding, and I see this error >>>>> >>>>> ipa user-find p-testipa >>>>> ipa: ERROR: Insufficient access: SASL(-1): generic failure: GSSAPI >>>>> Error: Unspecified GSS failure. Minor code may provide more information >>>>> (KDC returned error string: PROCESS_TGS) >>>>> >>>>> KRB5_TRACE=/dev/stdout kinit admin >>>>> [41178] 1472984115.233214: Getting initial credentials for >>>>> admin at XYZ.COM >>>>> [41178] 1472984115.235257: Sending request (167 bytes) to XYZ.COM >>>>> [41178] 1472984115.235419: Initiating TCP connection to stream >>>>> 10.1.3.36:88 >>>>> [41178] 1472984115.235685: Sending TCP request to stream 10.1.3.36:88 >>>>> [41178] 1472984120.238914: Received answer (174 bytes) from stream >>>>> 10.1.3.36:88 >>>>> [41178] 1472984120.238925: Terminating TCP connection to stream >>>>> 10.1.3.36:88 >>>>> [41178] 1472984120.238993: Response was from master KDC >>>>> [41 >>>>> >>>>> >>>>> Running an ldapsearch to see the db.. does not give any results and >>>>> just hangs there >>>>> >>>>> ldapsearch -x -D 'cn=Directory Manager' -W -s one -b >>>>> 'cn=kerberos,dc=xyz,dc=com' >>>>> Enter LDAP Password: >>>>> >>>>> even an ldapsearch -x does not respond >>>>> At this point, am sure that slapd is the one causing issues >>>>> >>>>> Running an strace against the hung slapd itself seems to get stuck >>>>> does not proceed after saying "attaching to process" >>>>> >>>>> From some others posts I read Thierry suggesting to increase the >>>>> nsslapd-threadnumber value >>>>> >>>>> It was set to 30, I think that might be too low. >>>>> >>>> ... >> >> [Message clipped] > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jdieter at lesbg.com Tue Sep 6 09:07:06 2016 From: jdieter at lesbg.com (Jonathan Dieter) Date: Tue, 06 Sep 2016 12:07:06 +0300 Subject: [Freeipa-users] Unable to setup replica (kadmin.local: No such entry in the database) Message-ID: <1473152826.2014.17.camel@lesbg.com> I have a FreeIPA install with three server replicas that was originally setup a couple years ago and, over time, has been upgraded to 4.2.4 on Fedora 23. I'm trying to add a fourth replica and it's failing in two places (depending on whether I enable the CA or not). ?I'm assuming the problem is that one of the upgrades didn't quite go right, and my install is now missing something required for replication, but that's just a guess. Without --setup-ca: Configuring Kerberos KDC (krb5kdc). Estimated time: 30 seconds ? [1/8]: adding sasl mappings to the directory ? [2/8]: configuring KDC ? [3/8]: creating a keytab for the directory ? [error] CalledProcessError: Command ''kadmin.local' '-q' 'addprinc -randkey ldap/ipa4.lesbg.com at LOCAL.LESBG.COM' '-x' 'ipa-setup-override- restrictions'' returned non-zero exit status 1 Looking at the logs, the error is: 2016-09-06T08:46:47Z DEBUG Process finished, return code=1 2016-09-06T08:46:47Z DEBUG stdout=Authenticating as principal root/admi n at LOCAL.LESBG.COM with password. 2016-09-06T08:46:47Z DEBUG stderr=kadmin.local: No such entry in the database while initializing kadmin.local interface Full logs at: http://lesloueizeh.com/jdieter/ipareplica-install-1.log With --setup-ca: Configuring certificate server (pki-tomcatd). Estimated time: 3 minutes 30 seconds ? [1/23]: creating certificate server user ? ... ? [15/23]: authorizing RA to modify profiles ? [error] EmptyResult: no matching entry found Full logs at: http://lesloueizeh.com/jdieter/ipareplica-install-2.log From tbordaz at redhat.com Tue Sep 6 09:31:07 2016 From: tbordaz at redhat.com (thierry bordaz) Date: Tue, 6 Sep 2016 11:31:07 +0200 Subject: [Freeipa-users] Freeipa 4.2.0 hangs intermittently In-Reply-To: References: <57BEB0AB.2090506@redhat.com> <57C44AC6.1030402@redhat.com> <57CD1FDC.2010109@redhat.com> <57CD5488.60404@redhat.com> <57CD659B.6050707@redhat.com> <57CD68BD.9030802@redhat.com> Message-ID: <57CE8CDB.6000809@redhat.com> Hi, The following thread is hanging most of the others threads Thread 20 (Thread 0x7f68007f8700 (LWP 4873)): #0 0x00007f683d0506d5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f68375ac2f3 in __db_hybrid_mutex_suspend () from /lib64/libdb-5.3.so #2 0x00007f68375ab640 in __db_tas_mutex_lock () from /lib64/libdb-5.3.so #3 0x00007f68376c5e1b in __memp_fget () from /lib64/libdb-5.3.so #4 0x00007f68375c9211 in __bam_search () from /lib64/libdb-5.3.so #5 0x00007f68375b4256 in __bamc_search () from /lib64/libdb-5.3.so #6 0x00007f68375b86c4 in __bamc_put () from /lib64/libdb-5.3.so #7 0x00007f683766f6d5 in __dbc_iput () from /lib64/libdb-5.3.so #8 0x00007f683766aa0e in __db_put () from /lib64/libdb-5.3.so #9 0x00007f683767ffa4 in __db_put_pp () from /lib64/libdb-5.3.so #10 0x00007f6832b5a4cf in idl_new_insert_key () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #11 0x00007f6832b695b5 in addordel_values_sv.isra.3 () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #12 0x00007f6832b6a25a in index_addordel_values_ext_sv () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #13 0x00007f6832b6a694 in index_addordel_values_sv () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #14 0x00007f6832b6a75a in index_addordel_entry () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #15 0x00007f6832b6d5fa in ldbm_back_add () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #16 0x00007f683f46fcca in op_shared_add () from /usr/lib64/dirsrv/libslapd.so.0 #17 0x00007f683f471020 in do_add () from /usr/lib64/dirsrv/libslapd.so.0 #18 0x00007f683f99c303 in connection_threadmain () #19 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so #20 0x00007f683d04cdc5 in start_thread () from /lib64/libpthread.so.0 #21 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 This thread is likely not hang on 'db_tas_mutex_lock' as the db_lock does not show it waiting for something. Unfortunately it is not shown in the output of top -H, did you get the complete output ? Ludwig mentioned a known issue that DS is not scaling well on NUMA arch, do you know if the machine is NUMA ? A possibility is that this thread get severely slowed down by a NUMA remote lock and showing DS has hung. This is why it would be interesting to know if it consums CPU. thanks theirry On 09/06/2016 10:59 AM, Rakesh Rajasekharan wrote: > Hi Thierry/Ludwig, > > I again hit the hang issue. This is the pstack of the ns-slapd > > So was able to this time get the pstack and db locks > > db locks details > Default locking region information: > 451 Last allocated locker ID > 0x7fffffff Current maximum unused locker ID > 9 Number of lock modes > 800 Initial number of locks allocated > 0 Initial number of lockers allocated > 800 Initial number of lock objects allocated > 10000 Maximum number of locks possible > 10000 Maximum number of lockers possible > 10000 Maximum number of lock objects possible > 800 Current number of locks allocated > 366 Current number of lockers allocated > 800 Current number of lock objects allocated > 160 Number of lock object partitions > 8191 Size of object hash table > 126 Number of current locks > 1005 Maximum number of locks at any one time > 33 Maximum number of locks in any one bucket > 3987 Maximum number of locks stolen by for an empty partition > 75 Maximum number of locks stolen for any one partition > 309 Number of current lockers > 313 Maximum number of lockers at any one time > 121 Number of current lock objects > 676 Maximum number of lock objects at any one time > 3 Maximum number of lock objects in any one bucket > 69 Maximum number of objects stolen by for an empty partition > 5 Maximum number of objects stolen for any one partition > 22M Total number of locks requested (22009642) > 21M Total number of locks released (21992878) > 0 Total number of locks upgraded > 95 Total number of locks downgraded > 3348 Lock requests not available due to conflicts, for which we > waited > 12646 Lock requests not available due to conflicts, for which we > did not wait > 0 Number of deadlocks > 0 Lock timeout value > 0 Number of locks that have timed out > 0 Transaction timeout value > 0 Number of transactions that have timed out > 2MB 560KB Region size > 71719 The number of partition locks that required waiting (0%) > 23105 The maximum number of times any partition lock was waited > for (0%) > 0 The number of object queue operations that required waiting > (0%) > 163 The number of locker allocations that required waiting (0%) > 3172 The number of region locks that required waiting (0%) > 5 Maximum hash bucket length > =-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-= > Lock REGINFO information: > Environment Region type > 1 Region ID > /var/lib/dirsrv/slapd-SPRINKLR-COM/db/__db.001 Region name > 0x7f67ef869000 Region address > 0x7f67ef8690a0 Region allocation head > 0x7f67ef8712b0 Region primary address > 0 Region maximum allocation > 0 Region allocated > Region allocations: 1581345 allocations, 0 failures, 1580794 frees, 6 > longest > Allocations by power-of-two sizes: > 1KB 1581329 > 2KB 3 > 4KB 4 > 8KB 3 > 16KB 3 > 32KB 1 > 64KB 0 > 128KB 0 > 256KB 2 > 512KB 0 > 1024KB 1 > REGION_SHARED Region flags > =-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-= > Lock region parameters: > 2 Lock region region mutex [3172/14M 0% !Own] > 16381 locker table size > 8191 object table size > 34128 obj_off > 900216 locker_off > 0 need_dd > =-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-= > Lock conflict matrix: > =-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-= > Locks grouped by lockers: > Locker Mode Count Status ----------------- Object --------------- > 2 dd=308 locks held 1 write locks 0 pid/thread > 4845/140085720139840 flags 10 priority 100 > 2 READ 1 HELD userRoot/id2entry.db handle 0 > 3 dd=307 locks held 0 write locks 0 pid/thread > 4845/140084829472512 flags 0 priority 100 > 4 dd=306 locks held 1 write locks 0 pid/thread > 4845/140085720139840 flags 10 priority 100 > 4 READ 1 HELD ipaca/id2entry.db handle 0 > 5 dd=305 locks held 0 write locks 0 pid/thread > 4845/140084837865216 flags 0 priority 100 > 6 dd=304 locks held 1 write locks 0 pid/thread > 4845/140085720139840 flags 10 priority 100 > 6 READ 1 HELD ipaca/entryrdn.db handle 0 > 7 dd=303 locks held 0 write locks 0 pid/thread > 4845/140084804294400 flags 0 priority 100 > 8 dd=302 locks held 0 write locks 0 pid/thread > 4845/140084720432896 flags 0 priority 100 > 9 dd=301 locks held 0 write locks 0 pid/thread > 4845/140084837865216 flags 0 priority 100 > a dd=300 locks held 1 write locks 0 pid/thread > 4845/140085720139840 flags 10 priority 100 > a READ 1 HELD ipaca/vlv#allcertspkitomcatindex.db > handle 0 > e dd=299 locks held 1 write locks 0 pid/thread > 4845/140085720139840 flags 10 priority 100 > e READ 1 HELD > ipaca/vlv#allnonrevokedcertspkitomcatindex.db handle 0 > 10 dd=298 locks held 1 write locks 0 pid/thread > 4845/140085720139840 flags 10 priority 100 > 10 READ 1 HELD > ipaca/vlv#allrevokedcertspkitomcatindex.db handle 0 > 11 dd=297 locks held 1 write locks 0 pid/thread > 4845/140085720139840 flags 10 priority 100 > 11 READ 1 HELD > ipaca/vlv#allrevokedcertsnotafterpkitomcatindex.db handle 0 > 14 dd=296 locks held 1 write locks 0 pid/thread > 4845/140085720139840 flags 10 priority 100 > 14 READ 1 HELD > ipaca/vlv#allrevokedorrevokedexpiredcertspkitomcatindex.db handle 0 > 15 dd=295 locks held 1 write locks 0 pid/thread > 4845/140085720139840 flags 10 priority 100 > 15 READ 1 HELD ipaca/vlv#allvalidcertspkitomcatindex.db > handle 0 > 16 dd=294 locks held 1 write locks 0 pid/thread > 4845/140085720139840 flags 10 priority 100 > 16 READ 1 HELD > ipaca/vlv#allvalidcertsnotafterpkitomcatindex.db handle 0 > 17 dd=293 locks held 1 write locks 0 pid/thread > 4845/140085720139840 flags 10 priority 100 > 17 READ 1 HELD > ipaca/vlv#allvalidorrevokedcertspkitomcatindex.db handle 0 > 18 dd=292 locks held 1 write locks 0 pid/thread > 4845/140085720139840 flags 10 priority 100 > 18 READ 1 HELD ipaca/vlv#caallpkitomcatindex.db > handle 0 > 1d dd=291 locks held 1 write locks 0 pid/thread > 4845/140085720139840 flags 10 priority 100 > 1d READ 1 HELD ipaca/vlv#cacompletepkitomcatindex.db > handle 0 > 1e dd=290 locks held 1 write locks 0 pid/thread > 4845/140085720139840 flags 10 priority 100 > 1e READ 1 HELD > ipaca/vlv#cacompleteenrollmentpkitomcatindex.db handle 0 > 20 dd=289 locks held 1 write locks 0 pid/thread > 4845/140085720139840 flags 10 priority 100 > 20 READ 1 HELD > ipaca/vlv#cacompleterevocationpkitomcatindex.db handle 0 > 21 dd=288 locks held 1 write locks 0 pid/thread > 4845/140085720139840 flags 10 priority 100 > 21 READ 1 HELD ipaca/vlv#caenrollmentpkitomcatindex.db > handle 0 > 2b dd=287 locks held 1 write locks 0 pid/thread > 4845/140085720139840 flags 10 priority 100 > 2b READ 1 HELD ipaca/vlv#carevocationpkitomcatindex.db > handle 0 > 2c dd=286 locks held 1 write locks 0 pid/thread > 4845/140085720139840 flags 10 priority 100 > 2c READ 1 HELD changelog/id2entry.db handle 0 > 2d dd=285 locks held 0 write locks 0 pid/thread > 4845/140085719607040 flags 0 priority 100 > 2e dd=284 locks held 1 write locks 0 pid/thread > 4845/140085720139840 flags 10 priority 100 > 2e READ 1 HELD changelog/entryusn.db handle 0 > 2f dd=283 locks held 0 write locks 0 pid/thread > 4845/140085720139840 flags 0 priority 100 > 30 dd=282 locks held 1 write locks 0 pid/thread > 4845/140085720139840 flags 10 priority 100 > 30 READ 1 HELD userRoot/entryusn.db handle 0 > 31 dd=281 locks held 0 write locks 0 pid/thread > 4845/140085720139840 flags 0 priority 100 > 32 dd=280 locks held 1 write locks 0 pid/thread > 4845/140085720139840 flags 10 priority 100 > 32 READ 1 HELD ipaca/entryusn.db handle 0 > 33 dd=279 locks held 0 write locks 0 pid/thread > 4845/140085720139840 flags 0 priority 100 > 34 dd=278 locks held 1 write locks 0 pid/thread > 4845/140085720139840 flags 10 priority 100 > 34 READ 1 HELD userRoot/entryrdn.db handle 0 > 35 dd=277 locks held 0 write locks 0 pid/thread > 4845/140084544186112 flags 0 priority 100 > 36 dd=276 locks held 0 write locks 0 pid/thread > 4845/140084636505856 flags 0 priority 100 > 37 dd=275 locks held 0 write locks 0 pid/thread > 4845/140084854650624 flags 0 priority 100 > 38 dd=274 locks held 1 write locks 0 pid/thread > 4845/140085720139840 flags 10 priority 100 > 38 READ 1 HELD userRoot/objectclass.db handle 0 > 39 dd=273 locks held 0 write locks 0 pid/thread > 4845/140084569364224 flags 0 priority 100 > 3a dd=272 locks held 1 write locks 0 pid/thread > 4845/140085720139840 flags 10 priority 100 > 3a READ 1 HELD userRoot/ancestorid.db handle 0 > 3b dd=271 locks held 0 write locks 0 pid/thread > 4845/140084854650624 flags 0 priority 100 > 3c dd=270 locks held 1 write locks 0 pid/thread > 4845/140085720139840 flags 10 priority 100 > 3c READ 1 HELD changelog/entryrdn.db handle 0 > 3d dd=269 locks held 0 write locks 0 pid/thread > 4845/140085719607040 flags 0 priority 100 > 3e dd=268 locks held 0 write locks 0 pid/thread > 4845/140084602935040 flags 0 priority 100 > 3f dd=267 locks held 0 write locks 0 pid/thread > 4845/140085719607040 flags 0 priority 100 > 40 dd=266 locks held 1 write locks 0 pid/thread > 4845/140085720139840 flags 10 priority 100 > 40 READ 1 HELD changelog/objectclass.db handle 0 > 41 dd=265 locks held 0 write locks 0 pid/thread > 4845/140085719607040 flags 0 priority 100 > 42 dd=264 locks held 0 write locks 0 pid/thread > 4845/140085719607040 flags 0 priority 100 > 43 dd=263 locks held 0 write locks 0 pid/thread > 4845/140084653291264 flags 0 priority 100 > 44 dd=262 locks held 1 write locks 0 pid/thread > 4845/140085720139840 flags 10 priority 100 > 44 READ 1 HELD ipaca/objectclass.db handle 0 > 45 dd=261 locks held 0 write locks 0 pid/thread > 4845/140084577756928 flags 0 priority 100 > 46 dd=260 locks held 0 write locks 0 pid/thread > 4845/140084720432896 flags 0 priority 100 > 47 dd=259 locks held 1 write locks 0 pid/thread > 4845/140085720139840 flags 10 priority 100 > 47 READ 1 HELD changelog/aci.db handle 0 > 48 dd=258 locks held 0 write locks 0 pid/thread > 4845/140085720139840 flags 0 priority 100 > 49 dd=257 locks held 1 write locks 0 pid/thread > 4845/140085720139840 flags 10 priority 100 > 49 READ 1 HELD userRoot/aci.db handle 0 > 4a dd=256 locks held 0 write locks 0 pid/thread > 4845/140085720139840 flags 0 priority 100 > 4b dd=255 locks held 0 write locks 0 pid/thread > 4845/140084535793408 flags 0 priority 100 > 4c dd=254 locks held 1 write locks 0 pid/thread > 4845/140085720139840 flags 10 priority 100 > 4c READ 1 HELD ipaca/aci.db handle 0 > 4d dd=253 locks held 0 write locks 0 pid/thread > 4845/140085720139840 flags 0 priority 100 > 4e dd=252 locks held 1 write locks 0 pid/thread > 4845/140085720139840 flags 10 priority 100 > 4e READ 1 HELD userRoot/parentid.db handle 0 > 4f dd=251 locks held 0 write locks 0 pid/thread > 4845/140085720139840 flags 0 priority 100 > 50 dd=250 locks held 0 write locks 0 pid/thread > 4845/140085719607040 flags 0 priority 100 > 51 dd=249 locks held 1 write locks 0 pid/thread > 4845/140085720139840 flags 10 priority 100 > 51 READ 1 HELD changelog/nsuniqueid.db handle 0 > 52 dd=248 locks held 1 write locks 0 pid/thread > 4845/140085720139840 flags 10 priority 100 > 52 READ 1 HELD changelog/changenumber.db handle 0 > 53 dd=247 locks held 0 write locks 0 pid/thread > 4845/140084602935040 flags 0 priority 100 > 54 dd=246 locks held 0 write locks 0 pid/thread > 4845/140085719607040 flags 0 priority 100 > 55 dd=245 locks held 1 write locks 0 pid/thread > 4845/140085719607040 flags 10 priority 100 > 55 READ 1 HELD changelog/targetuniqueid.db handle 0 > 56 dd=244 locks held 1 write locks 0 pid/thread > 4845/140085719607040 flags 10 priority 100 > 56 READ 1 HELD changelog/parentid.db handle 0 > 57 dd=243 locks held 1 write locks 0 pid/thread > 4845/140085719607040 flags 10 priority 100 > 57 READ 1 HELD changelog/ancestorid.db handle 0 > 58 dd=242 locks held 1 write locks 0 pid/thread > 4845/140085719607040 flags 10 priority 100 > 58 READ 1 HELD changelog/numsubordinates.db handle 0 > 59 dd=241 locks held 0 write locks 0 pid/thread > 4845/140085719607040 flags 0 priority 100 > 5a dd=240 locks held 0 write locks 0 pid/thread > 4845/140084544186112 flags 0 priority 100 > 5b dd=239 locks held 1 write locks 0 pid/thread > 4845/140084577756928 flags 10 priority 100 > 5b READ 1 HELD userRoot/krbPrincipalName.db handle 0 > 5c dd=238 locks held 0 write locks 0 pid/thread > 4845/140084636505856 flags 0 priority 100 > 5d dd=237 locks held 0 write locks 0 pid/thread > 4845/140084695254784 flags 0 priority 100 > 5e dd=236 locks held 1 write locks 0 pid/thread > 4845/140084560971520 flags 10 priority 100 > 5e READ 1 HELD userRoot/ipakrbprincipalalias.db > handle 0 > 5f dd=235 locks held 0 write locks 0 pid/thread > 4845/140084703647488 flags 0 priority 100 > 60 dd=234 locks held 0 write locks 0 pid/thread > 4845/140084653291264 flags 0 priority 100 > 61 dd=233 locks held 0 write locks 0 pid/thread > 4845/140084577756928 flags 0 priority 100 > 62 dd=232 locks held 0 write locks 0 pid/thread > 4845/140084686862080 flags 0 priority 100 > 63 dd=231 locks held 1 write locks 0 pid/thread > 4845/140084611327744 flags 10 priority 100 > 63 READ 1 HELD changelog/seeAlso.db handle 0 > 64 dd=230 locks held 0 write locks 0 pid/thread > 4845/140084686862080 flags 0 priority 100 > 65 dd=229 locks held 1 write locks 0 pid/thread > 4845/140084611327744 flags 10 priority 100 > 65 READ 1 HELD userRoot/seeAlso.db handle 0 > 66 dd=228 locks held 0 write locks 0 pid/thread > 4845/140084636505856 flags 0 priority 100 > 67 dd=227 locks held 1 write locks 0 pid/thread > 4845/140084611327744 flags 10 priority 100 > 67 READ 1 HELD ipaca/seeAlso.db handle 0 > 68 dd=226 locks held 0 write locks 0 pid/thread > 4845/140084686862080 flags 0 priority 100 > 69 dd=225 locks held 1 write locks 0 pid/thread > 4845/140084678469376 flags 10 priority 100 > 69 READ 1 HELD ipaca/parentid.db handle 0 > 6a dd=224 locks held 0 write locks 0 pid/thread > 4845/140084678469376 flags 0 priority 100 > 6b dd=223 locks held 0 write locks 0 pid/thread > 4845/140084829472512 flags 0 priority 100 > 6c dd=222 locks held 0 write locks 0 pid/thread > 4845/140084653291264 flags 0 priority 100 > 6d dd=221 locks held 0 write locks 0 pid/thread > 4845/140084569364224 flags 0 priority 100 > 6e dd=220 locks held 0 write locks 0 pid/thread > 4845/140084636505856 flags 0 priority 100 > 6e READ 1 WAIT userRoot/objectclass.db page 18 > 6f dd=219 locks held 0 write locks 0 pid/thread > 4845/140084544186112 flags 0 priority 100 > 70 dd=218 locks held 0 write locks 0 pid/thread > 4845/140084569364224 flags 0 priority 100 > 71 dd=217 locks held 0 write locks 0 pid/thread > 4845/140084619720448 flags 0 priority 100 > 72 dd=216 locks held 0 write locks 0 pid/thread > 4845/140084552578816 flags 0 priority 100 > 73 dd=215 locks held 0 write locks 0 pid/thread > 4845/140084829472512 flags 0 priority 100 > 74 dd=214 locks held 0 write locks 0 pid/thread > 4845/140084628113152 flags 0 priority 100 > 75 dd=213 locks held 0 write locks 0 pid/thread > 4845/140084686862080 flags 0 priority 100 > 76 dd=212 locks held 1 write locks 0 pid/thread > 4845/140084560971520 flags 10 priority 100 > 76 READ 1 HELD userRoot/ipauniqueid.db handle 0 > 77 dd=211 locks held 0 write locks 0 pid/thread > 4845/140084661683968 flags 0 priority 100 > 78 dd=210 locks held 0 write locks 0 pid/thread > 4845/140084846257920 flags 0 priority 100 > 79 dd=209 locks held 0 write locks 0 pid/thread > 4845/140084628113152 flags 0 priority 100 > 7a dd=208 locks held 1 write locks 0 pid/thread > 4845/140084560971520 flags 10 priority 100 > 7a READ 1 HELD userRoot/nsuniqueid.db handle 0 > 7b dd=207 locks held 1 write locks 0 pid/thread > 4845/140084560971520 flags 10 priority 100 > 7b READ 1 HELD userRoot/cn.db handle 0 > 7c dd=206 locks held 1 write locks 0 pid/thread > 4845/140084560971520 flags 10 priority 100 > 7c READ 1 HELD userRoot/fqdn.db handle 0 > 7d dd=205 locks held 1 write locks 0 pid/thread > 4845/140084560971520 flags 10 priority 100 > 7d READ 1 HELD userRoot/managedby.db handle 0 > 7e dd=204 locks held 1 write locks 0 pid/thread > 4845/140084560971520 flags 10 priority 100 > 7e READ 1 HELD userRoot/numsubordinates.db handle 0 > 7f dd=203 locks held 1 write locks 0 pid/thread > 4845/140084678469376 flags 10 priority 100 > 7f READ 1 HELD userRoot/member.db handle 0 > 80 dd=202 locks held 0 write locks 0 pid/thread > 4845/140084712040192 flags 0 priority 100 > 81 dd=201 locks held 1 write locks 0 pid/thread > 4845/140084678469376 flags 10 priority 100 > 81 READ 1 HELD userRoot/memberUser.db handle 0 > 82 dd=200 locks held 0 write locks 0 pid/thread > 4845/140084577756928 flags 0 priority 100 > 83 dd=199 locks held 1 write locks 0 pid/thread > 4845/140084678469376 flags 10 priority 100 > 83 READ 1 HELD userRoot/memberHost.db handle 0 > 84 dd=198 locks held 0 write locks 0 pid/thread > 4845/140084712040192 flags 0 priority 100 > 85 dd=197 locks held 0 write locks 0 pid/thread > 4845/140084577756928 flags 0 priority 100 > 86 dd=196 locks held 0 write locks 0 pid/thread > 4845/140084670076672 flags 0 priority 100 > 87 dd=195 locks held 0 write locks 0 pid/thread > 4845/140084577756928 flags 0 priority 100 > 88 dd=194 locks held 0 write locks 0 pid/thread > 4845/140084846257920 flags 0 priority 100 > 89 dd=193 locks held 0 write locks 0 pid/thread > 4845/140084552578816 flags 0 priority 100 > 8a dd=192 locks held 0 write locks 0 pid/thread > 4845/140084703647488 flags 0 priority 100 > 8b dd=191 locks held 0 write locks 0 pid/thread > 4845/140084544186112 flags 0 priority 100 > 8c dd=190 locks held 0 write locks 0 pid/thread > 4845/140084846257920 flags 0 priority 100 > 8d dd=189 locks held 0 write locks 0 pid/thread > 4845/140084661683968 flags 0 priority 100 > 8e dd=188 locks held 1 write locks 0 pid/thread > 4845/140085067675392 flags 10 priority 100 > 8e READ 1 HELD userRoot/macAddress.db handle 0 > 8f dd=187 locks held 0 write locks 0 pid/thread > 4845/140085067675392 flags 0 priority 100 > 90 dd=186 locks held 0 write locks 0 pid/thread > 4845/140084703647488 flags 0 priority 100 > 91 dd=185 locks held 0 write locks 0 pid/thread > 4845/140084552578816 flags 0 priority 100 > 92 dd=184 locks held 0 write locks 0 pid/thread > 4845/140084560971520 flags 0 priority 100 > 92 READ 1 WAIT userRoot/objectclass.db page 9 > 93 dd=183 locks held 0 write locks 0 pid/thread > 4845/140084577756928 flags 0 priority 100 > 94 dd=182 locks held 0 write locks 0 pid/thread > 4845/140084821079808 flags 0 priority 100 > 94 READ 1 WAIT userRoot/objectclass.db page 18 > 95 dd=181 locks held 0 write locks 0 pid/thread > 4845/140084544186112 flags 0 priority 100 > 96 dd=180 locks held 0 write locks 0 pid/thread > 4845/140084804294400 flags 0 priority 100 > 97 dd=179 locks held 1 write locks 0 pid/thread > 4845/140084712040192 flags 10 priority 100 > 97 READ 1 HELD userRoot/uid.db handle 0 > 98 dd=178 locks held 0 write locks 0 pid/thread > 4845/140084552578816 flags 0 priority 100 > 99 dd=177 locks held 0 write locks 0 pid/thread > 4845/140084577756928 flags 0 priority 100 > 9a dd=176 locks held 0 write locks 0 pid/thread > 4845/140084602935040 flags 0 priority 100 > 9b dd=175 locks held 1 write locks 0 pid/thread > 4845/140085076068096 flags 10 priority 100 > 9b READ 1 HELD userRoot/uniquemember.db handle 0 > 9c dd=174 locks held 1 write locks 0 pid/thread > 4845/140085076068096 flags 10 priority 100 > 9c READ 1 HELD userRoot/owner.db handle 0 > 9d dd=173 locks held 1 write locks 0 pid/thread > 4845/140085076068096 flags 10 priority 100 > 9d READ 1 HELD userRoot/ipaallowedtarget.db handle 0 > 9e dd=172 locks held 1 write locks 0 pid/thread > 4845/140085076068096 flags 10 priority 100 > 9e READ 1 HELD userRoot/ipaMemberCertProfile.db > handle 0 > 9f dd=171 locks held 1 write locks 0 pid/thread > 4845/140085076068096 flags 10 priority 100 > 9f READ 1 HELD userRoot/ipaassignedidview.db > handle 0 > a0 dd=170 locks held 1 write locks 0 pid/thread > 4845/140085076068096 flags 10 priority 100 > a0 READ 1 HELD userRoot/secretary.db handle 0 > a1 dd=169 locks held 1 write locks 0 pid/thread > 4845/140085076068096 flags 10 priority 100 > a1 READ 1 HELD userRoot/memberdenycmd.db handle 0 > a2 dd=168 locks held 1 write locks 0 pid/thread > 4845/140085076068096 flags 10 priority 100 > a2 READ 1 HELD userRoot/memberallowcmd.db handle 0 > a3 dd=167 locks held 1 write locks 0 pid/thread > 4845/140085076068096 flags 10 priority 100 > a3 READ 1 HELD userRoot/manager.db handle 0 > a4 dd=166 locks held 1 write locks 0 pid/thread > 4845/140085076068096 flags 10 priority 100 > a4 READ 1 HELD userRoot/ipasudorunas.db handle 0 > a5 dd=165 locks held 1 write locks 0 pid/thread > 4845/140085076068096 flags 10 priority 100 > a5 READ 1 HELD userRoot/ipaMemberCa.db handle 0 > a6 dd=164 locks held 1 write locks 0 pid/thread > 4845/140085076068096 flags 10 priority 100 > a6 READ 1 HELD userRoot/ipasudorunasgroup.db > handle 0 > a7 dd=163 locks held 1 write locks 0 pid/thread > 4845/140085076068096 flags 10 priority 100 > a7 READ 1 HELD userRoot/sourcehost.db handle 0 > a8 dd=162 locks held 1 write locks 0 pid/thread > 4845/140085076068096 flags 10 priority 100 > a8 READ 1 HELD userRoot/memberservice.db handle 0 > a9 dd=161 locks held 1 write locks 0 pid/thread > 4845/140085076068096 flags 10 priority 100 > a9 READ 1 HELD userRoot/ipatokenradiusconfiglink.db > handle 0 > aa dd=160 locks held 0 write locks 0 pid/thread > 4845/140084703647488 flags 0 priority 100 > ab dd=159 locks held 0 write locks 0 pid/thread > 4845/140084821079808 flags 0 priority 100 > ac dd=158 locks held 0 write locks 0 pid/thread > 4845/140084628113152 flags 0 priority 100 > ad dd=157 locks held 0 write locks 0 pid/thread > 4845/140084829472512 flags 0 priority 100 > ae dd=156 locks held 0 write locks 0 pid/thread > 4845/140084829472512 flags 0 priority 100 > af dd=155 locks held 0 write locks 0 pid/thread > 4845/140084854650624 flags 0 priority 100 > b0 dd=154 locks held 0 write locks 0 pid/thread > 4845/140084854650624 flags 0 priority 100 > b1 dd=153 locks held 0 write locks 0 pid/thread > 4845/140084837865216 flags 0 priority 100 > b2 dd=152 locks held 0 write locks 0 pid/thread > 4845/140084804294400 flags 0 priority 100 > b3 dd=151 locks held 0 write locks 0 pid/thread > 4845/140084812687104 flags 0 priority 100 > b4 dd=150 locks held 0 write locks 0 pid/thread > 4845/140084804294400 flags 0 priority 100 > b5 dd=149 locks held 0 write locks 0 pid/thread > 4845/140084837865216 flags 0 priority 100 > b6 dd=148 locks held 0 write locks 0 pid/thread > 4845/140084569364224 flags 0 priority 100 > b7 dd=147 locks held 0 write locks 0 pid/thread > 4845/140084829472512 flags 0 priority 100 > b8 dd=146 locks held 0 write locks 0 pid/thread > 4845/140084586149632 flags 0 priority 100 > b9 dd=145 locks held 0 write locks 0 pid/thread > 4845/140084653291264 flags 0 priority 100 > ba dd=144 locks held 0 write locks 0 pid/thread > 4845/140084636505856 flags 0 priority 100 > bb dd=143 locks held 0 write locks 0 pid/thread > 4845/140084804294400 flags 0 priority 100 > bc dd=142 locks held 0 write locks 0 pid/thread > 4845/140084804294400 flags 0 priority 100 > bd dd=141 locks held 0 write locks 0 pid/thread > 4845/140084695254784 flags 0 priority 100 > be dd=140 locks held 0 write locks 0 pid/thread > 4845/140084670076672 flags 0 priority 100 > bf dd=139 locks held 0 write locks 0 pid/thread > 4845/140084628113152 flags 0 priority 100 > c0 dd=138 locks held 0 write locks 0 pid/thread > 4845/140084661683968 flags 0 priority 100 > c1 dd=137 locks held 0 write locks 0 pid/thread > 4845/140084712040192 flags 0 priority 100 > c2 dd=136 locks held 0 write locks 0 pid/thread > 4845/140084661683968 flags 0 priority 100 > c3 dd=135 locks held 0 write locks 0 pid/thread > 4845/140084577756928 flags 0 priority 100 > c4 dd=134 locks held 0 write locks 0 pid/thread > 4845/140084854650624 flags 0 priority 100 > c5 dd=133 locks held 1 write locks 0 pid/thread > 4845/140084644898560 flags 10 priority 100 > c5 READ 1 HELD userRoot/uidnumber.db handle 0 > c6 dd=132 locks held 0 write locks 0 pid/thread > 4845/140084594542336 flags 0 priority 100 > c8 dd=131 locks held 1 write locks 0 pid/thread > 4845/140084678469376 flags 10 priority 100 > c8 READ 1 HELD ipaca/certstatus.db handle 0 > c9 dd=130 locks held 0 write locks 0 pid/thread > 4845/140084720432896 flags 0 priority 100 > ca dd=129 locks held 0 write locks 0 pid/thread > 4845/140084619720448 flags 0 priority 100 > cb dd=128 locks held 0 write locks 0 pid/thread > 4845/140084821079808 flags 0 priority 100 > cc dd=127 locks held 0 write locks 0 pid/thread > 4845/140084594542336 flags 0 priority 100 > cd dd=126 locks held 0 write locks 0 pid/thread > 4845/140084560971520 flags 0 priority 100 > ce dd=125 locks held 1 write locks 0 pid/thread > 4845/140084837865216 flags 10 priority 100 > ce READ 1 HELD userRoot/gidnumber.db handle 0 > cf dd=124 locks held 0 write locks 0 pid/thread > 4845/140084720432896 flags 0 priority 100 > d0 dd=123 locks held 0 write locks 0 pid/thread > 4845/140084569364224 flags 0 priority 100 > d1 dd=122 locks held 1 write locks 0 pid/thread > 4845/140084678469376 flags 10 priority 100 > d1 READ 1 HELD userRoot/nscpEntryDN.db handle 0 > d2 dd=121 locks held 0 write locks 0 pid/thread > 4845/140084695254784 flags 0 priority 100 > d3 dd=120 locks held 0 write locks 0 pid/thread > 4845/140084569364224 flags 0 priority 100 > d4 dd=119 locks held 0 write locks 0 pid/thread > 4845/140084653291264 flags 0 priority 100 > d5 dd=118 locks held 0 write locks 0 pid/thread > 4845/140084829472512 flags 0 priority 100 > d6 dd=117 locks held 0 write locks 0 pid/thread > 4845/140084661683968 flags 0 priority 100 > d7 dd=116 locks held 0 write locks 0 pid/thread > 4845/140084804294400 flags 0 priority 100 > d8 dd=115 locks held 1 write locks 0 pid/thread > 4845/140084821079808 flags 10 priority 100 > d8 READ 1 HELD ipaca/description.db handle 0 > d9 dd=114 locks held 0 write locks 0 pid/thread > 4845/140084661683968 flags 0 priority 100 > da dd=113 locks held 0 write locks 0 pid/thread > 4845/140084837865216 flags 0 priority 100 > db dd=112 locks held 1 write locks 0 pid/thread > 4845/140084678469376 flags 10 priority 100 > db READ 1 HELD ipaca/cn.db handle 0 > dc dd=111 locks held 0 write locks 0 pid/thread > 4845/140084577756928 flags 0 priority 100 > dd dd=110 locks held 1 write locks 0 pid/thread > 4845/140084678469376 flags 10 priority 100 > dd READ 1 HELD ipaca/ancestorid.db handle 0 > de dd=109 locks held 0 write locks 0 pid/thread > 4845/140084577756928 flags 0 priority 100 > df dd=108 locks held 0 write locks 0 pid/thread > 4845/140084703647488 flags 0 priority 100 > e0 dd=107 locks held 0 write locks 0 pid/thread > 4845/140084569364224 flags 0 priority 100 > e1 dd=106 locks held 0 write locks 0 pid/thread > 4845/140084569364224 flags 0 priority 100 > e2 dd=105 locks held 0 write locks 0 pid/thread > 4845/140084821079808 flags 0 priority 100 > e3 dd=104 locks held 1 write locks 0 pid/thread > 4845/140084686862080 flags 10 priority 100 > e3 READ 1 HELD ipaca/serialno.db handle 0 > e4 dd=103 locks held 0 write locks 0 pid/thread > 4845/140084837865216 flags 0 priority 100 > e5 dd=102 locks held 1 write locks 0 pid/thread > 4845/140084695254784 flags 10 priority 100 > e5 READ 1 HELD ipaca/revInfo.db handle 0 > e6 dd=101 locks held 1 write locks 0 pid/thread > 4845/140084695254784 flags 10 priority 100 > e6 READ 1 HELD ipaca/revokedby.db handle 0 > e7 dd=100 locks held 1 write locks 0 pid/thread > 4845/140084695254784 flags 10 priority 100 > e7 READ 1 HELD ipaca/revokedOn.db handle 0 > e8 dd=99 locks held 0 write locks 0 pid/thread > 4845/140084829472512 flags 0 priority 100 > e9 dd=98 locks held 0 write locks 0 pid/thread > 4845/140084846257920 flags 0 priority 100 > ea dd=97 locks held 1 write locks 0 pid/thread > 4845/140084703647488 flags 10 priority 100 > ea READ 1 HELD ipaca/nsuniqueid.db handle 0 > eb dd=96 locks held 1 write locks 0 pid/thread > 4845/140084703647488 flags 10 priority 100 > eb READ 1 HELD ipaca/requestid.db handle 0 > ec dd=95 locks held 1 write locks 0 pid/thread > 4845/140084703647488 flags 10 priority 100 > ec READ 1 HELD ipaca/requeststate.db handle 0 > ed dd=94 locks held 1 write locks 0 pid/thread > 4845/140084703647488 flags 10 priority 100 > ed READ 1 HELD ipaca/dateOfCreate.db handle 0 > ee dd=93 locks held 1 write locks 0 pid/thread > 4845/140084703647488 flags 10 priority 100 > ee READ 1 HELD ipaca/requesttype.db handle 0 > ef dd=92 locks held 1 write locks 0 pid/thread > 4845/140084703647488 flags 10 priority 100 > ef READ 1 HELD ipaca/numsubordinates.db handle 0 > f0 dd=91 locks held 1 write locks 0 pid/thread > 4845/140084611327744 flags 10 priority 100 > f0 READ 1 HELD userRoot/userCertificate.db handle 0 > f1 dd=90 locks held 0 write locks 0 pid/thread > 4845/140084821079808 flags 0 priority 100 > f2 dd=89 locks held 0 write locks 0 pid/thread > 4845/140084569364224 flags 0 priority 100 > f3 dd=88 locks held 0 write locks 0 pid/thread > 4845/140084602935040 flags 0 priority 100 > f4 dd=87 locks held 0 write locks 0 pid/thread > 4845/140084720432896 flags 0 priority 100 > f5 dd=86 locks held 0 write locks 0 pid/thread > 4845/140084628113152 flags 0 priority 100 > f6 dd=85 locks held 0 write locks 0 pid/thread > 4845/140084703647488 flags 0 priority 100 > f7 dd=84 locks held 0 write locks 0 pid/thread > 4845/140084611327744 flags 0 priority 100 > f8 dd=83 locks held 0 write locks 0 pid/thread > 4845/140084661683968 flags 0 priority 100 > f9 dd=82 locks held 0 write locks 0 pid/thread > 4845/140084837865216 flags 0 priority 100 > fa dd=81 locks held 0 write locks 0 pid/thread > 4845/140084544186112 flags 0 priority 100 > fb dd=80 locks held 0 write locks 0 pid/thread > 4845/140084703647488 flags 0 priority 100 > fc dd=79 locks held 0 write locks 0 pid/thread > 4845/140084544186112 flags 0 priority 100 > fd dd=78 locks held 0 write locks 0 pid/thread > 4845/140084535793408 flags 0 priority 100 > fe dd=77 locks held 0 write locks 0 pid/thread > 4845/140084586149632 flags 0 priority 100 > ff dd=76 locks held 0 write locks 0 pid/thread > 4845/140084569364224 flags 0 priority 100 > 100 dd=75 locks held 0 write locks 0 pid/thread > 4845/140084804294400 flags 0 priority 100 > 101 dd=74 locks held 0 write locks 0 pid/thread > 4845/140084703647488 flags 0 priority 100 > 102 dd=73 locks held 0 write locks 0 pid/thread > 4845/140084703647488 flags 0 priority 100 > 103 dd=72 locks held 0 write locks 0 pid/thread > 4845/140084821079808 flags 0 priority 100 > 104 dd=71 locks held 0 write locks 0 pid/thread > 4845/140084695254784 flags 0 priority 100 > 105 dd=70 locks held 0 write locks 0 pid/thread > 4845/140084544186112 flags 0 priority 100 > 106 dd=69 locks held 0 write locks 0 pid/thread > 4845/140084854650624 flags 0 priority 100 > 107 dd=68 locks held 0 write locks 0 pid/thread > 4845/140084628113152 flags 0 priority 100 > 108 dd=67 locks held 0 write locks 0 pid/thread > 4845/140084569364224 flags 0 priority 100 > 109 dd=66 locks held 0 write locks 0 pid/thread > 4845/140084594542336 flags 0 priority 100 > 10a dd=65 locks held 0 write locks 0 pid/thread > 4845/140084661683968 flags 0 priority 100 > 10b dd=64 locks held 0 write locks 0 pid/thread > 4845/140084653291264 flags 0 priority 100 > 10c dd=63 locks held 0 write locks 0 pid/thread > 4845/140084695254784 flags 0 priority 100 > 10d dd=62 locks held 0 write locks 0 pid/thread > 4845/140084720432896 flags 0 priority 100 > 10e dd=61 locks held 0 write locks 0 pid/thread > 4845/140084619720448 flags 0 priority 100 > 10f dd=60 locks held 0 write locks 0 pid/thread > 4845/140084720432896 flags 0 priority 100 > 110 dd=59 locks held 0 write locks 0 pid/thread > 4845/140084569364224 flags 0 priority 100 > 111 dd=58 locks held 0 write locks 0 pid/thread > 4845/140084695254784 flags 0 priority 100 > 112 dd=57 locks held 0 write locks 0 pid/thread > 4845/140084653291264 flags 0 priority 100 > 113 dd=56 locks held 0 write locks 0 pid/thread > 4845/140084712040192 flags 0 priority 100 > 114 dd=55 locks held 0 write locks 0 pid/thread > 4845/140084686862080 flags 0 priority 100 > 115 dd=54 locks held 0 write locks 0 pid/thread > 4845/140084577756928 flags 0 priority 100 > 115 READ 1 WAIT userRoot/objectclass.db page 9 > 116 dd=53 locks held 0 write locks 0 pid/thread > 4845/140084837865216 flags 0 priority 100 > 116 READ 1 WAIT userRoot/objectclass.db page 9 > 117 dd=52 locks held 0 write locks 0 pid/thread > 4845/140084577756928 flags 0 priority 100 > 118 dd=51 locks held 0 write locks 0 pid/thread > 4845/140084569364224 flags 0 priority 100 > 119 dd=50 locks held 0 write locks 0 pid/thread > 4845/140084560971520 flags 0 priority 100 > 11a dd=49 locks held 1 write locks 0 pid/thread > 4845/140084695254784 flags 10 priority 100 > 11a READ 1 HELD ipaca/metaInfo.db handle 0 > 11b dd=48 locks held 1 write locks 0 pid/thread > 4845/140084695254784 flags 10 priority 100 > 11b READ 1 HELD ipaca/notbefore.db handle 0 > 11c dd=47 locks held 1 write locks 0 pid/thread > 4845/140084695254784 flags 10 priority 100 > 11c READ 1 HELD ipaca/notafter.db handle 0 > 11d dd=46 locks held 1 write locks 0 pid/thread > 4845/140084695254784 flags 10 priority 100 > 11d READ 1 HELD ipaca/duration.db handle 0 > 11e dd=45 locks held 1 write locks 0 pid/thread > 4845/140084695254784 flags 10 priority 100 > 11e READ 1 HELD ipaca/subjectname.db handle 0 > 11f dd=44 locks held 1 write locks 0 pid/thread > 4845/140084695254784 flags 10 priority 100 > 11f READ 1 HELD ipaca/publicKeyData.db handle 0 > 120 dd=43 locks held 1 write locks 0 pid/thread > 4845/140084695254784 flags 10 priority 100 > 120 READ 1 HELD ipaca/extension.db handle 0 > 121 dd=42 locks held 1 write locks 0 pid/thread > 4845/140084695254784 flags 10 priority 100 > 121 READ 1 HELD ipaca/issuedby.db handle 0 > 122 dd=41 locks held 0 write locks 0 pid/thread > 4845/140084712040192 flags 0 priority 100 > 123 dd=40 locks held 0 write locks 0 pid/thread > 4845/140084812687104 flags 0 priority 100 > 124 dd=39 locks held 0 write locks 0 pid/thread > 4845/140084544186112 flags 0 priority 100 > 125 dd=38 locks held 0 write locks 0 pid/thread > 4845/140084686862080 flags 0 priority 100 > 126 dd=37 locks held 0 write locks 0 pid/thread > 4845/140084552578816 flags 0 priority 100 > 128 dd=36 locks held 0 write locks 0 pid/thread > 4845/140084686862080 flags 0 priority 100 > 12a dd=35 locks held 0 write locks 0 pid/thread > 4845/140084720432896 flags 0 priority 100 > 12b dd=34 locks held 0 write locks 0 pid/thread > 4845/140084552578816 flags 0 priority 100 > 12c dd=33 locks held 0 write locks 0 pid/thread > 4845/140084670076672 flags 0 priority 100 > 12d dd=32 locks held 0 write locks 0 pid/thread > 4845/140084577756928 flags 0 priority 100 > 12e dd=31 locks held 0 write locks 0 pid/thread > 4845/140084712040192 flags 0 priority 100 > 12f dd=30 locks held 0 write locks 0 pid/thread > 4845/140084670076672 flags 0 priority 100 > 130 dd=29 locks held 0 write locks 0 pid/thread > 4845/140084577756928 flags 0 priority 100 > 131 dd=28 locks held 0 write locks 0 pid/thread > 4845/140084560971520 flags 0 priority 100 > 132 dd=27 locks held 0 write locks 0 pid/thread > 4845/140084670076672 flags 0 priority 100 > 133 dd=26 locks held 0 write locks 0 pid/thread > 4845/140084602935040 flags 0 priority 100 > 134 dd=25 locks held 0 write locks 0 pid/thread > 4845/140084552578816 flags 0 priority 100 > 135 dd=24 locks held 0 write locks 0 pid/thread > 4845/140084678469376 flags 0 priority 100 > 136 dd=23 locks held 0 write locks 0 pid/thread > 4845/140084544186112 flags 0 priority 100 > 137 dd=22 locks held 0 write locks 0 pid/thread > 4845/140084577756928 flags 0 priority 100 > 138 dd=21 locks held 0 write locks 0 pid/thread > 4845/140084703647488 flags 0 priority 100 > 139 dd=20 locks held 0 write locks 0 pid/thread > 4845/140084678469376 flags 0 priority 100 > 13b dd=19 locks held 0 write locks 0 pid/thread > 4845/140084812687104 flags 0 priority 100 > 144 dd=18 locks held 0 write locks 0 pid/thread > 4845/140084670076672 flags 0 priority 100 > 156 dd=17 locks held 0 write locks 0 pid/thread > 4845/140084594542336 flags 0 priority 100 > 15a dd=16 locks held 0 write locks 0 pid/thread > 4845/140084577756928 flags 0 priority 100 > 168 dd=15 locks held 0 write locks 0 pid/thread > 4845/140084653291264 flags 0 priority 100 > 16a dd=14 locks held 0 write locks 0 pid/thread > 4845/140084829472512 flags 0 priority 100 > 17d dd=13 locks held 0 write locks 0 pid/thread > 4845/140084829472512 flags 0 priority 100 > 17e dd=12 locks held 0 write locks 0 pid/thread > 4845/140084829472512 flags 0 priority 100 > 17f dd=11 locks held 0 write locks 0 pid/thread > 4845/140084586149632 flags 0 priority 100 > 180 dd=10 locks held 0 write locks 0 pid/thread > 4845/140084594542336 flags 0 priority 100 > 181 dd= 9 locks held 0 write locks 0 pid/thread > 4845/140084577756928 flags 0 priority 100 > 193 dd= 8 locks held 0 write locks 0 pid/thread > 4845/140084812687104 flags 0 priority 100 > 196 dd= 7 locks held 0 write locks 0 pid/thread > 4845/140084695254784 flags 0 priority 100 > 1a1 dd= 6 locks held 0 write locks 0 pid/thread > 4845/140084720432896 flags 0 priority 100 > 1a2 dd= 5 locks held 0 write locks 0 pid/thread > 4845/140084577756928 flags 0 priority 100 > 1a3 dd= 4 locks held 0 write locks 0 pid/thread > 4845/140084812687104 flags 0 priority 100 > 1a4 dd= 3 locks held 0 write locks 0 pid/thread > 4845/140084812687104 flags 0 priority 100 > 1a5 dd= 2 locks held 0 write locks 0 pid/thread > 4845/140084577756928 flags 0 priority 100 > 1a6 dd= 1 locks held 0 write locks 0 pid/thread > 4845/140084594542336 flags 0 priority 100 > 800b3a33 dd= 0 locks held 26 write locks 22 pid/thread > 4845/140084661683968 flags 0 priority 100 > 800b3a33 WRITE 1 HELD userRoot/objectclass.db page 9 > 800b3a33 WRITE 1 HELD userRoot/objectclass.db page 8 > 800b3a33 WRITE 1 HELD userRoot/objectclass.db page 18 > 800b3a33 WRITE 1 HELD userRoot/objectclass.db page 7 > 800b3a33 WRITE 1 HELD userRoot/objectclass.db page 14 > 800b3a33 WRITE 1 HELD userRoot/objectclass.db page 11 > 800b3a33 WRITE 2 HELD userRoot/objectclass.db page 3 > 800b3a33 WRITE 1 HELD userRoot/cn.db page 150 > 800b3a33 WRITE 1 HELD userRoot/cn.db page 89 > 800b3a33 WRITE 1 HELD userRoot/cn.db page 53 > 800b3a33 WRITE 1 HELD userRoot/cn.db page 131 > 800b3a33 WRITE 1 HELD userRoot/cn.db page 117 > 800b3a33 WRITE 1 HELD userRoot/cn.db page 11 > 800b3a33 WRITE 2 HELD userRoot/cn.db page 141 > 800b3a33 WRITE 1 HELD userRoot/cn.db page 41 > 800b3a33 WRITE 1 HELD userRoot/cn.db page 40 > 800b3a33 WRITE 1 HELD userRoot/cn.db page 143 > 800b3a33 WRITE 1 HELD userRoot/cn.db page 30 > 800b3a33 WRITE 2 HELD userRoot/id2entry.db page 8184 > 800b3a33 WRITE 1 HELD userRoot/id2entry.db page 8183 > 800b3a33 WRITE 1 HELD userRoot/id2entry.db page 7779 > 800b3a33 WRITE 1 HELD userRoot/id2entry.db page 0 > 800b3a33 READ 3 HELD userRoot/entryrdn.db page 3 > 800b3a33 READ 1 HELD userRoot/entryrdn.db page 36 > 800b3a33 READ 1 HELD userRoot/entryrdn.db page 4 > 800b3a33 READ 1 HELD userRoot/nsuniqueid.db page 86 > =-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-= > Locks grouped by object: > Locker Mode Count Status ----------------- Object --------------- > a4 READ 1 HELD userRoot/ipasudorunas.db handle 0 > > 9e READ 1 HELD userRoot/ipaMemberCertProfile.db > handle 0 > > 10 READ 1 HELD > ipaca/vlv#allrevokedcertspkitomcatindex.db handle 0 > > 800b3a33 WRITE 2 HELD userRoot/id2entry.db page 8184 > > 800b3a33 WRITE 1 HELD userRoot/id2entry.db page 8183 > > 800b3a33 WRITE 1 HELD userRoot/id2entry.db page 7779 > > a1 READ 1 HELD userRoot/memberdenycmd.db handle 0 > > a READ 1 HELD ipaca/vlv#allcertspkitomcatindex.db > handle 0 > > a7 READ 1 HELD userRoot/sourcehost.db handle 0 > > 57 READ 1 HELD changelog/ancestorid.db handle 0 > > e5 READ 1 HELD ipaca/revInfo.db handle 0 > > e6 READ 1 HELD ipaca/revokedby.db handle 0 > > 11 READ 1 HELD > ipaca/vlv#allrevokedcertsnotafterpkitomcatindex.db handle 0 > > 4e READ 1 HELD userRoot/parentid.db handle 0 > > 4 READ 1 HELD ipaca/id2entry.db handle 0 > > 58 READ 1 HELD changelog/numsubordinates.db handle 0 > > 67 READ 1 HELD ipaca/seeAlso.db handle 0 > > 2b READ 1 HELD ipaca/vlv#carevocationpkitomcatindex.db > handle 0 > > ec READ 1 HELD ipaca/requeststate.db handle 0 > > 11b READ 1 HELD ipaca/notbefore.db handle 0 > > 7e READ 1 HELD userRoot/numsubordinates.db handle 0 > > a2 READ 1 HELD userRoot/memberallowcmd.db handle 0 > > 9c READ 1 HELD userRoot/owner.db handle 0 > > 1e READ 1 HELD > ipaca/vlv#cacompleteenrollmentpkitomcatindex.db handle 0 > > 11c READ 1 HELD ipaca/notafter.db handle 0 > > 40 READ 1 HELD changelog/objectclass.db handle 0 > > 2e READ 1 HELD changelog/entryusn.db handle 0 > > 16 READ 1 HELD > ipaca/vlv#allvalidcertsnotafterpkitomcatindex.db handle 0 > > 15 READ 1 HELD ipaca/vlv#allvalidcertspkitomcatindex.db > handle 0 > > a5 READ 1 HELD userRoot/ipaMemberCa.db handle 0 > > 18 READ 1 HELD ipaca/vlv#caallpkitomcatindex.db > handle 0 > > a0 READ 1 HELD userRoot/secretary.db handle 0 > > 81 READ 1 HELD userRoot/memberUser.db handle 0 > > 9f READ 1 HELD userRoot/ipaassignedidview.db > handle 0 > > a3 READ 1 HELD userRoot/manager.db handle 0 > > d8 READ 1 HELD ipaca/description.db handle 0 > > a8 READ 1 HELD userRoot/memberservice.db handle 0 > > 44 READ 1 HELD ipaca/objectclass.db handle 0 > > ed READ 1 HELD ipaca/dateOfCreate.db handle 0 > > 800b3a33 WRITE 1 HELD userRoot/id2entry.db page 0 > > 2 READ 1 HELD userRoot/id2entry.db handle 0 > > 5b READ 1 HELD userRoot/krbPrincipalName.db handle 0 > > ee READ 1 HELD ipaca/requesttype.db handle 0 > > c8 READ 1 HELD ipaca/certstatus.db handle 0 > > 56 READ 1 HELD changelog/parentid.db handle 0 > > 83 READ 1 HELD userRoot/memberHost.db handle 0 > > 5e READ 1 HELD userRoot/ipakrbprincipalalias.db > handle 0 > > 121 READ 1 HELD ipaca/issuedby.db handle 0 > > 6 READ 1 HELD ipaca/entryrdn.db handle 0 > > e7 READ 1 HELD ipaca/revokedOn.db handle 0 > > 4c READ 1 HELD ipaca/aci.db handle 0 > > 11d READ 1 HELD ipaca/duration.db handle 0 > > 14 READ 1 HELD > ipaca/vlv#allrevokedorrevokedexpiredcertspkitomcatindex.db handle 0 > > 21 READ 1 HELD ipaca/vlv#caenrollmentpkitomcatindex.db > handle 0 > > 55 READ 1 HELD changelog/targetuniqueid.db handle 0 > > 800b3a33 READ 1 HELD userRoot/entryrdn.db page 36 > > 2c READ 1 HELD changelog/id2entry.db handle 0 > > 800b3a33 READ 1 HELD userRoot/entryrdn.db page 4 > > 34 READ 1 HELD userRoot/entryrdn.db handle 0 > > 800b3a33 READ 3 HELD userRoot/entryrdn.db page 3 > > eb READ 1 HELD ipaca/requestid.db handle 0 > > 49 READ 1 HELD userRoot/aci.db handle 0 > > 76 READ 1 HELD userRoot/ipauniqueid.db handle 0 > > 63 READ 1 HELD changelog/seeAlso.db handle 0 > > 7c READ 1 HELD userRoot/fqdn.db handle 0 > > db READ 1 HELD ipaca/cn.db handle 0 > > ef READ 1 HELD ipaca/numsubordinates.db handle 0 > > 32 READ 1 HELD ipaca/entryusn.db handle 0 > > 1d READ 1 HELD ipaca/vlv#cacompletepkitomcatindex.db > handle 0 > > dd READ 1 HELD ipaca/ancestorid.db handle 0 > > a9 READ 1 HELD userRoot/ipatokenradiusconfiglink.db > handle 0 > > f0 READ 1 HELD userRoot/userCertificate.db handle 0 > > 11a READ 1 HELD ipaca/metaInfo.db handle 0 > > 7d READ 1 HELD userRoot/managedby.db handle 0 > > ea READ 1 HELD ipaca/nsuniqueid.db handle 0 > > c5 READ 1 HELD userRoot/uidnumber.db handle 0 > > d1 READ 1 HELD userRoot/nscpEntryDN.db handle 0 > > 120 READ 1 HELD ipaca/extension.db handle 0 > > 800b3a33 WRITE 1 HELD userRoot/cn.db page 150 > > 69 READ 1 HELD ipaca/parentid.db handle 0 > > 800b3a33 WRITE 1 HELD userRoot/cn.db page 131 > > 800b3a33 WRITE 1 HELD userRoot/cn.db page 143 > > 800b3a33 WRITE 2 HELD userRoot/cn.db page 141 > > 20 READ 1 HELD > ipaca/vlv#cacompleterevocationpkitomcatindex.db handle 0 > > 8e READ 1 HELD userRoot/macAddress.db handle 0 > > e READ 1 HELD > ipaca/vlv#allnonrevokedcertspkitomcatindex.db handle 0 > > 17 READ 1 HELD > ipaca/vlv#allvalidorrevokedcertspkitomcatindex.db handle 0 > > 800b3a33 WRITE 1 HELD userRoot/cn.db page 30 > > 800b3a33 WRITE 1 HELD userRoot/objectclass.db page 14 > > 800b3a33 WRITE 1 HELD userRoot/objectclass.db page 9 > 116 READ 1 WAIT userRoot/objectclass.db page 9 > 115 READ 1 WAIT userRoot/objectclass.db page 9 > 92 READ 1 WAIT userRoot/objectclass.db page 9 > > 800b3a33 WRITE 1 HELD userRoot/objectclass.db page 8 > > 7b READ 1 HELD userRoot/cn.db handle 0 > > 800b3a33 WRITE 1 HELD userRoot/objectclass.db page 11 > > 800b3a33 WRITE 1 HELD userRoot/objectclass.db page 7 > > 800b3a33 WRITE 1 HELD userRoot/cn.db page 11 > > 38 READ 1 HELD userRoot/objectclass.db handle 0 > > 800b3a33 WRITE 2 HELD userRoot/objectclass.db page 3 > > 800b3a33 WRITE 1 HELD userRoot/cn.db page 53 > > 800b3a33 WRITE 1 HELD userRoot/objectclass.db page 18 > 6e READ 1 WAIT userRoot/objectclass.db page 18 > 94 READ 1 WAIT userRoot/objectclass.db page 18 > > 800b3a33 WRITE 1 HELD userRoot/cn.db page 41 > > 800b3a33 WRITE 1 HELD userRoot/cn.db page 40 > > 800b3a33 WRITE 1 HELD userRoot/cn.db page 89 > > 800b3a33 WRITE 1 HELD userRoot/cn.db page 117 > > 30 READ 1 HELD userRoot/entryusn.db handle 0 > > 3c READ 1 HELD changelog/entryrdn.db handle 0 > > 97 READ 1 HELD userRoot/uid.db handle 0 > > 47 READ 1 HELD changelog/aci.db handle 0 > > e3 READ 1 HELD ipaca/serialno.db handle 0 > > 7a READ 1 HELD userRoot/nsuniqueid.db handle 0 > > 800b3a33 READ 1 HELD userRoot/nsuniqueid.db page 86 > > 51 READ 1 HELD changelog/nsuniqueid.db handle 0 > > 11e READ 1 HELD ipaca/subjectname.db handle 0 > > 7f READ 1 HELD userRoot/member.db handle 0 > > 11f READ 1 HELD ipaca/publicKeyData.db handle 0 > > 9b READ 1 HELD userRoot/uniquemember.db handle 0 > > 52 READ 1 HELD changelog/changenumber.db handle 0 > > ce READ 1 HELD userRoot/gidnumber.db handle 0 > > 65 READ 1 HELD userRoot/seeAlso.db handle 0 > > 9d READ 1 HELD userRoot/ipaallowedtarget.db handle 0 > > a6 READ 1 HELD userRoot/ipasudorunasgroup.db > handle 0 > > 3a READ 1 HELD userRoot/ancestorid.db handle 0 > > > > > pstack output > > root at ipa-master ]# pstack 4845 > Thread 44 (Thread 0x7f681dd3a700 (LWP 4847)): > #0 0x00007f683cd719b3 in select () from /lib64/libc.so.6 > #1 0x00007f683f4fc0e9 in DS_Sleep () from /usr/lib64/dirsrv/libslapd.so.0 > #2 0x00007f6832b4a907 in deadlock_threadmain () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #3 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so > #4 0x00007f683d04cdc5 in start_thread () from /lib64/libpthread.so.0 > #5 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 > Thread 43 (Thread 0x7f681d539700 (LWP 4848)): > #0 0x00007f683cd719b3 in select () from /lib64/libc.so.6 > #1 0x00007f68376d8bad in __os_yield () from /lib64/libdb-5.3.so > > #2 0x00007f68376d42b3 in __memp_sync_int () from /lib64/libdb-5.3.so > > #3 0x00007f68376e4752 in __txn_checkpoint () from /lib64/libdb-5.3.so > > #4 0x00007f68376e4b74 in __txn_checkpoint_pp () from > /lib64/libdb-5.3.so > #5 0x00007f6832b4ea87 in checkpoint_threadmain () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #6 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so > #7 0x00007f683d04cdc5 in start_thread () from /lib64/libpthread.so.0 > #8 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 > Thread 42 (Thread 0x7f681cd38700 (LWP 4849)): > #0 0x00007f683cd719b3 in select () from /lib64/libc.so.6 > #1 0x00007f683f4fc0e9 in DS_Sleep () from /usr/lib64/dirsrv/libslapd.so.0 > #2 0x00007f6832b4ab7f in trickle_threadmain () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #3 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so > #4 0x00007f683d04cdc5 in start_thread () from /lib64/libpthread.so.0 > #5 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 > Thread 41 (Thread 0x7f681c537700 (LWP 4850)): > #0 0x00007f683cd719b3 in select () from /lib64/libc.so.6 > #1 0x00007f683f4fc0e9 in DS_Sleep () from /usr/lib64/dirsrv/libslapd.so.0 > #2 0x00007f6832b45707 in perf_threadmain () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #3 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so > #4 0x00007f683d04cdc5 in start_thread () from /lib64/libpthread.so.0 > #5 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 > Thread 40 (Thread 0x7f681bb2d700 (LWP 4851)): > #0 0x00007f683d0506d5 in pthread_cond_wait@@GLIBC_2.3.2 () from > /lib64/libpthread.so.0 > #1 0x00007f683d6a6050 in PR_WaitCondVar () from /lib64/libnspr4.so > #2 0x00007f683f4eb198 in slapi_wait_condvar () from > /usr/lib64/dirsrv/libslapd.so.0 > #3 0x00007f683587d62e in cos_cache_wait_on_change () from > /usr/lib64/dirsrv/plugins/libcos-plugin.so > #4 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so > #5 0x00007f683d04cdc5 in start_thread () from /lib64/libpthread.so.0 > #6 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 > Thread 39 (Thread 0x7f681b32c700 (LWP 4852)): > #0 0x00007f683d0506d5 in pthread_cond_wait@@GLIBC_2.3.2 () from > /lib64/libpthread.so.0 > #1 0x00007f683d6a6050 in PR_WaitCondVar () from /lib64/libnspr4.so > #2 0x00007f683f4eb198 in slapi_wait_condvar () from > /usr/lib64/dirsrv/libslapd.so.0 > #3 0x00007f68311e8eed in roles_cache_wait_on_change () from > /usr/lib64/dirsrv/plugins/libroles-plugin.so > #4 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so > #5 0x00007f683d04cdc5 in start_thread () from /lib64/libpthread.so.0 > #6 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 > Thread 38 (Thread 0x7f681ab2b700 (LWP 4853)): > #0 0x00007f683d0506d5 in pthread_cond_wait@@GLIBC_2.3.2 () from > /lib64/libpthread.so.0 > #1 0x00007f683d6a6050 in PR_WaitCondVar () from /lib64/libnspr4.so > #2 0x00007f683f4eb198 in slapi_wait_condvar () from > /usr/lib64/dirsrv/libslapd.so.0 > #3 0x00007f68311e8eed in roles_cache_wait_on_change () from > /usr/lib64/dirsrv/plugins/libroles-plugin.so > #4 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so > #5 0x00007f683d04cdc5 in start_thread () from /lib64/libpthread.so.0 > #6 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 > Thread 37 (Thread 0x7f681a32a700 (LWP 4854)): > #0 0x00007f683d0506d5 in pthread_cond_wait@@GLIBC_2.3.2 () from > /lib64/libpthread.so.0 > #1 0x00007f683d6a6050 in PR_WaitCondVar () from /lib64/libnspr4.so > #2 0x00007f683f4eb198 in slapi_wait_condvar () from > /usr/lib64/dirsrv/libslapd.so.0 > #3 0x00007f68311e8eed in roles_cache_wait_on_change () from > /usr/lib64/dirsrv/plugins/libroles-plugin.so > #4 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so > #5 0x00007f683d04cdc5 in start_thread () from /lib64/libpthread.so.0 > #6 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 > Thread 36 (Thread 0x7f6819b29700 (LWP 4855)): > #0 0x00007f683d050a82 in pthread_cond_timedwait@@GLIBC_2.3.2 () from > /lib64/libpthread.so.0 > #1 0x00007f683d6a5b07 in pt_TimedWait () from /lib64/libnspr4.so > #2 0x00007f683d6a5fce in PR_WaitCondVar () from /lib64/libnspr4.so > #3 0x00007f683f9a4d03 in housecleaning () > #4 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so > #5 0x00007f683d04cdc5 in start_thread () from /lib64/libpthread.so.0 > #6 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 > Thread 35 (Thread 0x7f6819328700 (LWP 4856)): > #0 0x00007f683d050a82 in pthread_cond_timedwait@@GLIBC_2.3.2 () from > /lib64/libpthread.so.0 > #1 0x00007f683d6a5b07 in pt_TimedWait () from /lib64/libnspr4.so > #2 0x00007f683d6a5fce in PR_WaitCondVar () from /lib64/libnspr4.so > #3 0x00007f683f4911e8 in eq_loop () from /usr/lib64/dirsrv/libslapd.so.0 > #4 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so > #5 0x00007f683d04cdc5 in start_thread () from /lib64/libpthread.so.0 > #6 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 > Thread 34 (Thread 0x7f680bfff700 (LWP 4859)): > #0 0x00007f683d0506d5 in pthread_cond_wait@@GLIBC_2.3.2 () from > /lib64/libpthread.so.0 > #1 0x00007f683d6a6050 in PR_WaitCondVar () from /lib64/libnspr4.so > #2 0x00007f683f999fee in connection_wait_for_new_work () > #3 0x00007f683f99b719 in connection_threadmain () > #4 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so > #5 0x00007f683d04cdc5 in start_thread () from /lib64/libpthread.so.0 > #6 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 > Thread 33 (Thread 0x7f680b7fe700 (LWP 4860)): > #0 0x00007f683d0506d5 in pthread_cond_wait@@GLIBC_2.3.2 () from > /lib64/libpthread.so.0 > #1 0x00007f683d6a6050 in PR_WaitCondVar () from /lib64/libnspr4.so > #2 0x00007f683f999fee in connection_wait_for_new_work () > #3 0x00007f683f99b719 in connection_threadmain () > #4 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so > #5 0x00007f683d04cdc5 in start_thread () from /lib64/libpthread.so.0 > #6 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 > Thread 32 (Thread 0x7f680affd700 (LWP 4861)): > #0 0x00007f683d0506d5 in pthread_cond_wait@@GLIBC_2.3.2 () from > /lib64/libpthread.so.0 > #1 0x00007f68375ac2f3 in __db_hybrid_mutex_suspend () from > /lib64/libdb-5.3.so > #2 0x00007f68375ab640 in __db_tas_mutex_lock () from > /lib64/libdb-5.3.so > #3 0x00007f6837655d3a in __lock_get_internal () from > /lib64/libdb-5.3.so > #4 0x00007f6837656820 in __lock_get () from /lib64/libdb-5.3.so > > #5 0x00007f6837682142 in __db_lget () from /lib64/libdb-5.3.so > > #6 0x00007f68375c9605 in __bam_search () from /lib64/libdb-5.3.so > > #7 0x00007f68375b4256 in __bamc_search () from /lib64/libdb-5.3.so > > #8 0x00007f68375b5d0f in __bamc_get () from /lib64/libdb-5.3.so > > #9 0x00007f683766eca6 in __dbc_iget () from /lib64/libdb-5.3.so > > #10 0x00007f683767db02 in __dbc_get_pp () from /lib64/libdb-5.3.so > > #11 0x00007f6832b59680 in idl_new_fetch () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #12 0x00007f6832b67b26 in index_read_ext_allids () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #13 0x00007f6832b52284 in keys2idl () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #14 0x00007f6832b529e3 in ava_candidates.isra.0 () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #15 0x00007f6832b52fd2 in filter_candidates_ext () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #16 0x00007f6832b54096 in list_candidates () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #17 0x00007f6832b52f40 in filter_candidates_ext () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #18 0x00007f6832b54096 in list_candidates () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #19 0x00007f6832b52f40 in filter_candidates_ext () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #20 0x00007f6832b54096 in list_candidates () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #21 0x00007f6832b52f40 in filter_candidates_ext () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #22 0x00007f6832b8e80c in subtree_candidates () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #23 0x00007f6832b8fe7f in ldbm_back_search () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #24 0x00007f683f4bd1d6 in op_shared_search () from > /usr/lib64/dirsrv/libslapd.so.0 > #25 0x00007f683f9ae33e in do_search () > #26 0x00007f683f99c3c3 in connection_threadmain () > #27 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so > #28 0x00007f683d04cdc5 in start_thread () from /lib64/libpthread.so.0 > #29 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 > Thread 31 (Thread 0x7f680a7fc700 (LWP 4862)): > #0 0x00007f683d0506d5 in pthread_cond_wait@@GLIBC_2.3.2 () from > /lib64/libpthread.so.0 > #1 0x00007f683d6a62c3 in PR_EnterMonitor () from /lib64/libnspr4.so > #2 0x00007f6832b4d3e6 in dblayer_txn_begin () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #3 0x00007f6832b89007 in ldbm_back_modify () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #4 0x00007f683f4b6541 in op_shared_modify () from > /usr/lib64/dirsrv/libslapd.so.0 > #5 0x00007f683f4b7004 in modify_internal_pb () from > /usr/lib64/dirsrv/libslapd.so.0 > #6 0x00007f683483bbbb in ipalockout_postop () from > /usr/lib64/dirsrv/plugins/libipa_lockout.so > #7 0x00007f683f4c8848 in plugin_call_func () from > /usr/lib64/dirsrv/libslapd.so.0 > #8 0x00007f683f4c8b13 in plugin_call_plugins () from > /usr/lib64/dirsrv/libslapd.so.0 > #9 0x00007f683f9954d8 in do_bind () > #10 0x00007f683f99c3fd in connection_threadmain () > #11 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so > #12 0x00007f683d04cdc5 in start_thread () from /lib64/libpthread.so.0 > #13 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 > Thread 30 (Thread 0x7f6809ffb700 (LWP 4863)): > #0 0x00007f683d0506d5 in pthread_cond_wait@@GLIBC_2.3.2 () from > /lib64/libpthread.so.0 > #1 0x00007f68375ac2f3 in __db_hybrid_mutex_suspend () from > /lib64/libdb-5.3.so > #2 0x00007f68375ab640 in __db_tas_mutex_lock () from > /lib64/libdb-5.3.so > #3 0x00007f6837655d3a in __lock_get_internal () from > /lib64/libdb-5.3.so > #4 0x00007f6837656820 in __lock_get () from /lib64/libdb-5.3.so > > #5 0x00007f6837682142 in __db_lget () from /lib64/libdb-5.3.so > > #6 0x00007f68375c9605 in __bam_search () from /lib64/libdb-5.3.so > > #7 0x00007f68375b4256 in __bamc_search () from /lib64/libdb-5.3.so > > #8 0x00007f68375b5d0f in __bamc_get () from /lib64/libdb-5.3.so > > #9 0x00007f683766eca6 in __dbc_iget () from /lib64/libdb-5.3.so > > #10 0x00007f683767db02 in __dbc_get_pp () from /lib64/libdb-5.3.so > > #11 0x00007f6832b59680 in idl_new_fetch () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #12 0x00007f6832b67b26 in index_read_ext_allids () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #13 0x00007f6832b52284 in keys2idl () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #14 0x00007f6832b529e3 in ava_candidates.isra.0 () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #15 0x00007f6832b52fd2 in filter_candidates_ext () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #16 0x00007f6832b54096 in list_candidates () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #17 0x00007f6832b52f40 in filter_candidates_ext () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #18 0x00007f6832b8e80c in subtree_candidates () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #19 0x00007f6832b8fe7f in ldbm_back_search () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #20 0x00007f683f4bd1d6 in op_shared_search () from > /usr/lib64/dirsrv/libslapd.so.0 > #21 0x00007f683f4cd94e in search_internal_callback_pb () from > /usr/lib64/dirsrv/libslapd.so.0 > #22 0x00007f683f4cdbe8 in search_internal_pb () from > /usr/lib64/dirsrv/libslapd.so.0 > #23 0x00007f683f9ac49b in ids_sasl_canon_user () > #24 0x00007f683d46c93b in _sasl_canon_user () from /lib64/libsasl2.so.3 > #25 0x00007f683d46cc4c in _sasl_canon_user_lookup () from > /lib64/libsasl2.so.3 > #26 0x00007f6837374fb9 in gssapi_server_mech_step () from > /usr/lib64/sasl2/libgssapiv2.so > #27 0x00007f683d478b9b in sasl_server_step () from /lib64/libsasl2.so.3 > #28 0x00007f683f9ad83b in ids_sasl_check_bind () > #29 0x00007f683f99561a in do_bind () > #30 0x00007f683f99c3fd in connection_threadmain () > #31 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so > #32 0x00007f683d04cdc5 in start_thread () from /lib64/libpthread.so.0 > #33 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 > Thread 29 (Thread 0x7f68097fa700 (LWP 4864)): > #0 0x00007f683d0506d5 in pthread_cond_wait@@GLIBC_2.3.2 () from > /lib64/libpthread.so.0 > #1 0x00007f683d6a6050 in PR_WaitCondVar () from /lib64/libnspr4.so > #2 0x00007f683f999fee in connection_wait_for_new_work () > #3 0x00007f683f99b719 in connection_threadmain () > #4 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so > #5 0x00007f683d04cdc5 in start_thread () from /lib64/libpthread.so.0 > #6 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 > Thread 28 (Thread 0x7f6808ff9700 (LWP 4865)): > #0 0x00007f683d0506d5 in pthread_cond_wait@@GLIBC_2.3.2 () from > /lib64/libpthread.so.0 > #1 0x00007f683d6a6050 in PR_WaitCondVar () from /lib64/libnspr4.so > #2 0x00007f683f999fee in connection_wait_for_new_work () > #3 0x00007f683f99b719 in connection_threadmain () > #4 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so > #5 0x00007f683d04cdc5 in start_thread () from /lib64/libpthread.so.0 > #6 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 > Thread 27 (Thread 0x7f6803fff700 (LWP 4866)): > #0 0x00007f683d0506d5 in pthread_cond_wait@@GLIBC_2.3.2 () from > /lib64/libpthread.so.0 > #1 0x00007f683d6a6050 in PR_WaitCondVar () from /lib64/libnspr4.so > #2 0x00007f683f999fee in connection_wait_for_new_work () > #3 0x00007f683f99b719 in connection_threadmain () > #4 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so > #5 0x00007f683d04cdc5 in start_thread () from /lib64/libpthread.so.0 > #6 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 > Thread 26 (Thread 0x7f68037fe700 (LWP 4867)): > #0 0x00007f683d0506d5 in pthread_cond_wait@@GLIBC_2.3.2 () from > /lib64/libpthread.so.0 > #1 0x00007f683d6a6050 in PR_WaitCondVar () from /lib64/libnspr4.so > #2 0x00007f683f999fee in connection_wait_for_new_work () > #3 0x00007f683f99b719 in connection_threadmain () > #4 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so > #5 0x00007f683d04cdc5 in start_thread () from /lib64/libpthread.so.0 > #6 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 > Thread 25 (Thread 0x7f6802ffd700 (LWP 4868)): > #0 0x00007f683d0506d5 in pthread_cond_wait@@GLIBC_2.3.2 () from > /lib64/libpthread.so.0 > #1 0x00007f683d6a6050 in PR_WaitCondVar () from /lib64/libnspr4.so > #2 0x00007f683f999fee in connection_wait_for_new_work () > #3 0x00007f683f99b719 in connection_threadmain () > #4 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so > #5 0x00007f683d04cdc5 in start_thread () from /lib64/libpthread.so.0 > #6 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 > Thread 24 (Thread 0x7f68027fc700 (LWP 4869)): > #0 0x00007f683d0506d5 in pthread_cond_wait@@GLIBC_2.3.2 () from > /lib64/libpthread.so.0 > #1 0x00007f68375ac2f3 in __db_hybrid_mutex_suspend () from > /lib64/libdb-5.3.so > #2 0x00007f68375abb37 in __db_tas_mutex_readlock () from > /lib64/libdb-5.3.so > #3 0x00007f68376c5d35 in __memp_fget () from /lib64/libdb-5.3.so > > #4 0x00007f683766f333 in __dbc_iget () from /lib64/libdb-5.3.so > > #5 0x00007f683767db02 in __dbc_get_pp () from /lib64/libdb-5.3.so > > #6 0x00007f6832b59822 in idl_new_fetch () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #7 0x00007f6832b67b26 in index_read_ext_allids () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #8 0x00007f6832b52284 in keys2idl () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #9 0x00007f6832b529e3 in ava_candidates.isra.0 () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #10 0x00007f6832b52fd2 in filter_candidates_ext () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #11 0x00007f6832b54096 in list_candidates () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #12 0x00007f6832b52f40 in filter_candidates_ext () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #13 0x00007f6832b54096 in list_candidates () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #14 0x00007f6832b52f40 in filter_candidates_ext () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #15 0x00007f6832b54096 in list_candidates () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #16 0x00007f6832b52f40 in filter_candidates_ext () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #17 0x00007f6832b8e80c in subtree_candidates () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #18 0x00007f6832b8fe7f in ldbm_back_search () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #19 0x00007f683f4bd1d6 in op_shared_search () from > /usr/lib64/dirsrv/libslapd.so.0 > #20 0x00007f683f9ae33e in do_search () > #21 0x00007f683f99c3c3 in connection_threadmain () > #22 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so > #23 0x00007f683d04cdc5 in start_thread () from /lib64/libpthread.so.0 > #24 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 > Thread 23 (Thread 0x7f6801ffb700 (LWP 4870)): > #0 0x00007f683d0506d5 in pthread_cond_wait@@GLIBC_2.3.2 () from > /lib64/libpthread.so.0 > #1 0x00007f683d6a6050 in PR_WaitCondVar () from /lib64/libnspr4.so > #2 0x00007f683f999fee in connection_wait_for_new_work () > #3 0x00007f683f99b719 in connection_threadmain () > #4 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so > #5 0x00007f683d04cdc5 in start_thread () from /lib64/libpthread.so.0 > #6 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 > Thread 22 (Thread 0x7f68017fa700 (LWP 4871)): > #0 0x00007f683d0506d5 in pthread_cond_wait@@GLIBC_2.3.2 () from > /lib64/libpthread.so.0 > #1 0x00007f683d6a6050 in PR_WaitCondVar () from /lib64/libnspr4.so > #2 0x00007f683f999fee in connection_wait_for_new_work () > #3 0x00007f683f99b719 in connection_threadmain () > #4 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so > #5 0x00007f683d04cdc5 in start_thread () from /lib64/libpthread.so.0 > #6 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 > Thread 21 (Thread 0x7f6800ff9700 (LWP 4872)): > #0 0x00007f683d0506d5 in pthread_cond_wait@@GLIBC_2.3.2 () from > /lib64/libpthread.so.0 > #1 0x00007f683d6a6050 in PR_WaitCondVar () from /lib64/libnspr4.so > #2 0x00007f683f999fee in connection_wait_for_new_work () > #3 0x00007f683f99b719 in connection_threadmain () > #4 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so > #5 0x00007f683d04cdc5 in start_thread () from /lib64/libpthread.so.0 > #6 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 > Thread 20 (Thread 0x7f68007f8700 (LWP 4873)): > #0 0x00007f683d0506d5 in pthread_cond_wait@@GLIBC_2.3.2 () from > /lib64/libpthread.so.0 > #1 0x00007f68375ac2f3 in __db_hybrid_mutex_suspend () from > /lib64/libdb-5.3.so > #2 0x00007f68375ab640 in __db_tas_mutex_lock () from > /lib64/libdb-5.3.so > #3 0x00007f68376c5e1b in __memp_fget () from /lib64/libdb-5.3.so > > #4 0x00007f68375c9211 in __bam_search () from /lib64/libdb-5.3.so > > #5 0x00007f68375b4256 in __bamc_search () from /lib64/libdb-5.3.so > > #6 0x00007f68375b86c4 in __bamc_put () from /lib64/libdb-5.3.so > > #7 0x00007f683766f6d5 in __dbc_iput () from /lib64/libdb-5.3.so > > #8 0x00007f683766aa0e in __db_put () from /lib64/libdb-5.3.so > > #9 0x00007f683767ffa4 in __db_put_pp () from /lib64/libdb-5.3.so > > #10 0x00007f6832b5a4cf in idl_new_insert_key () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #11 0x00007f6832b695b5 in addordel_values_sv.isra.3 () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #12 0x00007f6832b6a25a in index_addordel_values_ext_sv () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #13 0x00007f6832b6a694 in index_addordel_values_sv () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #14 0x00007f6832b6a75a in index_addordel_entry () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #15 0x00007f6832b6d5fa in ldbm_back_add () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #16 0x00007f683f46fcca in op_shared_add () from > /usr/lib64/dirsrv/libslapd.so.0 > #17 0x00007f683f471020 in do_add () from /usr/lib64/dirsrv/libslapd.so.0 > #18 0x00007f683f99c303 in connection_threadmain () > #19 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so > #20 0x00007f683d04cdc5 in start_thread () from /lib64/libpthread.so.0 > #21 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 > Thread 19 (Thread 0x7f67ffff7700 (LWP 4874)): > #0 0x00007f683d0506d5 in pthread_cond_wait@@GLIBC_2.3.2 () from > /lib64/libpthread.so.0 > #1 0x00007f683d6a62c3 in PR_EnterMonitor () from /lib64/libnspr4.so > #2 0x00007f6832b4d3e6 in dblayer_txn_begin () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #3 0x00007f6832b89007 in ldbm_back_modify () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #4 0x00007f683f4b6541 in op_shared_modify () from > /usr/lib64/dirsrv/libslapd.so.0 > #5 0x00007f683f4b7004 in modify_internal_pb () from > /usr/lib64/dirsrv/libslapd.so.0 > #6 0x00007f6832dd3b60 in ipapwd_apply_mods () from > /usr/lib64/dirsrv/plugins/libipa_pwd_extop.so > #7 0x00007f6832dd78e0 in store_new_keys () from > /usr/lib64/dirsrv/plugins/libipa_pwd_extop.so > #8 0x00007f6832dd9ad0 in ipapwd_extop () from > /usr/lib64/dirsrv/plugins/libipa_pwd_extop.so > #9 0x00007f683f4c6ba2 in plugin_call_exop_plugins () from > /usr/lib64/dirsrv/libslapd.so.0 > #10 0x00007f683f9a2efe in do_extended () > #11 0x00007f683f99c41a in connection_threadmain () > #12 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so > #13 0x00007f683d04cdc5 in start_thread () from /lib64/libpthread.so.0 > #14 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 > Thread 18 (Thread 0x7f67ff7f6700 (LWP 4875)): > #0 0x00007f683d0506d5 in pthread_cond_wait@@GLIBC_2.3.2 () from > /lib64/libpthread.so.0 > #1 0x00007f683d6a6050 in PR_WaitCondVar () from /lib64/libnspr4.so > #2 0x00007f683f999fee in connection_wait_for_new_work () > #3 0x00007f683f99b719 in connection_threadmain () > #4 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so > #5 0x00007f683d04cdc5 in start_thread () from /lib64/libpthread.so.0 > #6 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 > Thread 17 (Thread 0x7f67feff5700 (LWP 4876)): > #0 0x00007f683d0506d5 in pthread_cond_wait@@GLIBC_2.3.2 () from > /lib64/libpthread.so.0 > #1 0x00007f68375ac2f3 in __db_hybrid_mutex_suspend () from > /lib64/libdb-5.3.so > #2 0x00007f68375ab640 in __db_tas_mutex_lock () from > /lib64/libdb-5.3.so > #3 0x00007f6837655d3a in __lock_get_internal () from > /lib64/libdb-5.3.so > #4 0x00007f6837656820 in __lock_get () from /lib64/libdb-5.3.so > > #5 0x00007f6837682142 in __db_lget () from /lib64/libdb-5.3.so > > #6 0x00007f68375c9605 in __bam_search () from /lib64/libdb-5.3.so > > #7 0x00007f68375b4256 in __bamc_search () from /lib64/libdb-5.3.so > > #8 0x00007f68375b5d0f in __bamc_get () from /lib64/libdb-5.3.so > > #9 0x00007f683766eca6 in __dbc_iget () from /lib64/libdb-5.3.so > > #10 0x00007f683767db02 in __dbc_get_pp () from /lib64/libdb-5.3.so > > #11 0x00007f6832b59680 in idl_new_fetch () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #12 0x00007f6832b67b26 in index_read_ext_allids () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #13 0x00007f6832b52284 in keys2idl () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #14 0x00007f6832b529e3 in ava_candidates.isra.0 () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #15 0x00007f6832b52fd2 in filter_candidates_ext () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #16 0x00007f6832b54096 in list_candidates () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #17 0x00007f6832b52f40 in filter_candidates_ext () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #18 0x00007f6832b8e80c in subtree_candidates () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #19 0x00007f6832b8fe7f in ldbm_back_search () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #20 0x00007f683f4bd1d6 in op_shared_search () from > /usr/lib64/dirsrv/libslapd.so.0 > #21 0x00007f683f4cd94e in search_internal_callback_pb () from > /usr/lib64/dirsrv/libslapd.so.0 > #22 0x00007f683f4cdbe8 in search_internal_pb () from > /usr/lib64/dirsrv/libslapd.so.0 > #23 0x00007f6832dd75dc in get_entry_by_principal () from > /usr/lib64/dirsrv/plugins/libipa_pwd_extop.so > #24 0x00007f6832dd8eb1 in ipapwd_extop () from > /usr/lib64/dirsrv/plugins/libipa_pwd_extop.so > #25 0x00007f683f4c6ba2 in plugin_call_exop_plugins () from > /usr/lib64/dirsrv/libslapd.so.0 > #26 0x00007f683f9a2efe in do_extended () > #27 0x00007f683f99c41a in connection_threadmain () > #28 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so > #29 0x00007f683d04cdc5 in start_thread () from /lib64/libpthread.so.0 > #30 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 > Thread 16 (Thread 0x7f67fe7f4700 (LWP 4877)): > #0 0x00007f683d0506d5 in pthread_cond_wait@@GLIBC_2.3.2 () from > /lib64/libpthread.so.0 > #1 0x00007f683d6a6050 in PR_WaitCondVar () from /lib64/libnspr4.so > #2 0x00007f683f999fee in connection_wait_for_new_work () > #3 0x00007f683f99b719 in connection_threadmain () > #4 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so > #5 0x00007f683d04cdc5 in start_thread () from /lib64/libpthread.so.0 > #6 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 > Thread 15 (Thread 0x7f67fdff3700 (LWP 4878)): > #0 0x00007f683d0506d5 in pthread_cond_wait@@GLIBC_2.3.2 () from > /lib64/libpthread.so.0 > #1 0x00007f683d6a6050 in PR_WaitCondVar () from /lib64/libnspr4.so > #2 0x00007f683f999fee in connection_wait_for_new_work () > #3 0x00007f683f99b719 in connection_threadmain () > #4 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so > #5 0x00007f683d04cdc5 in start_thread () from /lib64/libpthread.so.0 > #6 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 > Thread 14 (Thread 0x7f67fd7f2700 (LWP 4879)): > #0 0x00007f683d0506d5 in pthread_cond_wait@@GLIBC_2.3.2 () from > /lib64/libpthread.so.0 > #1 0x00007f683d6a6050 in PR_WaitCondVar () from /lib64/libnspr4.so > #2 0x00007f683f999fee in connection_wait_for_new_work () > #3 0x00007f683f99b719 in connection_threadmain () > #4 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so > #5 0x00007f683d04cdc5 in start_thread () from /lib64/libpthread.so.0 > #6 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 > Thread 13 (Thread 0x7f67fcff1700 (LWP 4880)): > #0 0x00007f683d0506d5 in pthread_cond_wait@@GLIBC_2.3.2 () from > /lib64/libpthread.so.0 > #1 0x00007f683d6a6050 in PR_WaitCondVar () from /lib64/libnspr4.so > #2 0x00007f683f999fee in connection_wait_for_new_work () > #3 0x00007f683f99b719 in connection_threadmain () > #4 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so > #5 0x00007f683d04cdc5 in start_thread () from /lib64/libpthread.so.0 > #6 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 > Thread 12 (Thread 0x7f67fc7f0700 (LWP 4881)): > #0 0x00007f683d0506d5 in pthread_cond_wait@@GLIBC_2.3.2 () from > /lib64/libpthread.so.0 > #1 0x00007f683d6a6050 in PR_WaitCondVar () from /lib64/libnspr4.so > #2 0x00007f683f999fee in connection_wait_for_new_work () > #3 0x00007f683f99b719 in connection_threadmain () > #4 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so > #5 0x00007f683d04cdc5 in start_thread () from /lib64/libpthread.so.0 > #6 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 > Thread 11 (Thread 0x7f67fbfef700 (LWP 4882)): > #0 0x00007f683d0506d5 in pthread_cond_wait@@GLIBC_2.3.2 () from > /lib64/libpthread.so.0 > #1 0x00007f683d6a6050 in PR_WaitCondVar () from /lib64/libnspr4.so > #2 0x00007f683f999fee in connection_wait_for_new_work () > #3 0x00007f683f99b719 in connection_threadmain () > #4 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so > #5 0x00007f683d04cdc5 in start_thread () from /lib64/libpthread.so.0 > #6 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 > Thread 10 (Thread 0x7f67fb7ee700 (LWP 4883)): > #0 0x00007f683d0506d5 in pthread_cond_wait@@GLIBC_2.3.2 () from > /lib64/libpthread.so.0 > #1 0x00007f68375ac2f3 in __db_hybrid_mutex_suspend () from > /lib64/libdb-5.3.so > #2 0x00007f68375ab640 in __db_tas_mutex_lock () from > /lib64/libdb-5.3.so > #3 0x00007f6837655d3a in __lock_get_internal () from > /lib64/libdb-5.3.so > #4 0x00007f6837656820 in __lock_get () from /lib64/libdb-5.3.so > > #5 0x00007f6837682142 in __db_lget () from /lib64/libdb-5.3.so > > #6 0x00007f68375c9605 in __bam_search () from /lib64/libdb-5.3.so > > #7 0x00007f68375b4256 in __bamc_search () from /lib64/libdb-5.3.so > > #8 0x00007f68375b5d0f in __bamc_get () from /lib64/libdb-5.3.so > > #9 0x00007f683766eca6 in __dbc_iget () from /lib64/libdb-5.3.so > > #10 0x00007f683767db02 in __dbc_get_pp () from /lib64/libdb-5.3.so > > #11 0x00007f6832b59680 in idl_new_fetch () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #12 0x00007f6832b67b26 in index_read_ext_allids () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #13 0x00007f6832b52284 in keys2idl () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #14 0x00007f6832b529e3 in ava_candidates.isra.0 () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #15 0x00007f6832b52fd2 in filter_candidates_ext () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #16 0x00007f6832b54096 in list_candidates () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #17 0x00007f6832b52f40 in filter_candidates_ext () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #18 0x00007f6832b54096 in list_candidates () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #19 0x00007f6832b52f40 in filter_candidates_ext () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #20 0x00007f6832b54096 in list_candidates () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #21 0x00007f6832b52f40 in filter_candidates_ext () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #22 0x00007f6832b8e80c in subtree_candidates () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #23 0x00007f6832b8fe7f in ldbm_back_search () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #24 0x00007f683f4bd1d6 in op_shared_search () from > /usr/lib64/dirsrv/libslapd.so.0 > #25 0x00007f683f9ae33e in do_search () > #26 0x00007f683f99c3c3 in connection_threadmain () > #27 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so > #28 0x00007f683d04cdc5 in start_thread () from /lib64/libpthread.so.0 > #29 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 > Thread 9 (Thread 0x7f67fafed700 (LWP 4884)): > #0 0x00007f683d0506d5 in pthread_cond_wait@@GLIBC_2.3.2 () from > /lib64/libpthread.so.0 > #1 0x00007f683d6a6050 in PR_WaitCondVar () from /lib64/libnspr4.so > #2 0x00007f683f999fee in connection_wait_for_new_work () > #3 0x00007f683f99b719 in connection_threadmain () > #4 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so > #5 0x00007f683d04cdc5 in start_thread () from /lib64/libpthread.so.0 > #6 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 > Thread 8 (Thread 0x7f67fa7ec700 (LWP 4885)): > #0 0x00007f683d0506d5 in pthread_cond_wait@@GLIBC_2.3.2 () from > /lib64/libpthread.so.0 > #1 0x00007f68375ac2f3 in __db_hybrid_mutex_suspend () from > /lib64/libdb-5.3.so > #2 0x00007f68375ab640 in __db_tas_mutex_lock () from > /lib64/libdb-5.3.so > #3 0x00007f6837655d3a in __lock_get_internal () from > /lib64/libdb-5.3.so > #4 0x00007f6837656820 in __lock_get () from /lib64/libdb-5.3.so > > #5 0x00007f6837682142 in __db_lget () from /lib64/libdb-5.3.so > > #6 0x00007f68375c9605 in __bam_search () from /lib64/libdb-5.3.so > > #7 0x00007f68375b4256 in __bamc_search () from /lib64/libdb-5.3.so > > #8 0x00007f68375b5d0f in __bamc_get () from /lib64/libdb-5.3.so > > #9 0x00007f683766eca6 in __dbc_iget () from /lib64/libdb-5.3.so > > #10 0x00007f683767db02 in __dbc_get_pp () from /lib64/libdb-5.3.so > > #11 0x00007f6832b59680 in idl_new_fetch () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #12 0x00007f6832b67b26 in index_read_ext_allids () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #13 0x00007f6832b52284 in keys2idl () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #14 0x00007f6832b529e3 in ava_candidates.isra.0 () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #15 0x00007f6832b52fd2 in filter_candidates_ext () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #16 0x00007f6832b54096 in list_candidates () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #17 0x00007f6832b52f40 in filter_candidates_ext () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #18 0x00007f6832b54096 in list_candidates () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #19 0x00007f6832b52f40 in filter_candidates_ext () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #20 0x00007f6832b54096 in list_candidates () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #21 0x00007f6832b52f40 in filter_candidates_ext () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #22 0x00007f6832b8e80c in subtree_candidates () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #23 0x00007f6832b8fe7f in ldbm_back_search () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #24 0x00007f683f4bd1d6 in op_shared_search () from > /usr/lib64/dirsrv/libslapd.so.0 > #25 0x00007f683f9ae33e in do_search () > #26 0x00007f683f99c3c3 in connection_threadmain () > #27 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so > #28 0x00007f683d04cdc5 in start_thread () from /lib64/libpthread.so.0 > #29 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 > Thread 7 (Thread 0x7f67f9feb700 (LWP 4886)): > #0 0x00007f683d0506d5 in pthread_cond_wait@@GLIBC_2.3.2 () from > /lib64/libpthread.so.0 > #1 0x00007f683d6a6050 in PR_WaitCondVar () from /lib64/libnspr4.so > #2 0x00007f683f999fee in connection_wait_for_new_work () > #3 0x00007f683f99b719 in connection_threadmain () > #4 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so > #5 0x00007f683d04cdc5 in start_thread () from /lib64/libpthread.so.0 > #6 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 > Thread 6 (Thread 0x7f67f97ea700 (LWP 4887)): > #0 0x00007f683d0506d5 in pthread_cond_wait@@GLIBC_2.3.2 () from > /lib64/libpthread.so.0 > #1 0x00007f683d6a6050 in PR_WaitCondVar () from /lib64/libnspr4.so > #2 0x00007f683f999fee in connection_wait_for_new_work () > #3 0x00007f683f99b719 in connection_threadmain () > #4 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so > #5 0x00007f683d04cdc5 in start_thread () from /lib64/libpthread.so.0 > #6 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 > Thread 5 (Thread 0x7f67f8fe9700 (LWP 4888)): > #0 0x00007f683d0506d5 in pthread_cond_wait@@GLIBC_2.3.2 () from > /lib64/libpthread.so.0 > #1 0x00007f683d6a6050 in PR_WaitCondVar () from /lib64/libnspr4.so > #2 0x00007f683f999fee in connection_wait_for_new_work () > #3 0x00007f683f99b719 in connection_threadmain () > #4 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so > #5 0x00007f683d04cdc5 in start_thread () from /lib64/libpthread.so.0 > #6 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 > Thread 4 (Thread 0x7f67f87e8700 (LWP 4889)): > #0 0x00007f683cd719b3 in select () from /lib64/libc.so.6 > #1 0x00007f683f4fc0e9 in DS_Sleep () from /usr/lib64/dirsrv/libslapd.so.0 > #2 0x00007f683f99d325 in time_thread () > #3 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so > #4 0x00007f683d04cdc5 in start_thread () from /lib64/libpthread.so.0 > #5 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 > Thread 3 (Thread 0x7f67f7de6700 (LWP 4892)): > #0 0x00007f683d0506d5 in pthread_cond_wait@@GLIBC_2.3.2 () from > /lib64/libpthread.so.0 > #1 0x00007f683d6a6050 in PR_WaitCondVar () from /lib64/libnspr4.so > #2 0x00007f683f9a8f25 in ps_send_results () > #3 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so > #4 0x00007f683d04cdc5 in start_thread () from /lib64/libpthread.so.0 > #5 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 > Thread 2 (Thread 0x7f67f75e5700 (LWP 5084)): > #0 0x00007f683d0506d5 in pthread_cond_wait@@GLIBC_2.3.2 () from > /lib64/libpthread.so.0 > #1 0x00007f683d6a6050 in PR_WaitCondVar () from /lib64/libnspr4.so > #2 0x00007f683f9a8f25 in ps_send_results () > #3 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so > #4 0x00007f683d04cdc5 in start_thread () from /lib64/libpthread.so.0 > #5 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 > Thread 1 (Thread 0x7f683f964840 (LWP 4845)): > #0 0x00007f683d0506d5 in pthread_cond_wait@@GLIBC_2.3.2 () from > /lib64/libpthread.so.0 > #1 0x00007f683d6a62c3 in PR_EnterMonitor () from /lib64/libnspr4.so > #2 0x00007f683f99d5ed in setup_pr_read_pds () > #3 0x00007f683f9a0fd9 in slapd_daemon () > #4 0x00007f683f992c6c in main () > > > > the top output seems ok, there was just a sudden spike to around 340% > cpu then it comes down > > top - 08:53:12 up 6 days, 23:20, 1 user, load average: 0.00, 0.01, 0.12 > Threads: 44 total, 0 running, 44 sleeping, 0 stopped, 0 zombie > %Cpu(s): 0.0 us, 0.0 sy, 0.0 ni,100.0 id, 0.0 wa, 0.0 hi, 0.0 > si, 0.0 st > KiB Mem : 65533176 total, 52165676 free, 6135088 used, 7232412 buff/cache > KiB Swap: 3999996 total, 3999996 free, 0 used. 58452040 avail Mem > > PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ > COMMAND > 4845 dirsrv 20 0 5133196 3.150g 281272 S 0.0 5.0 31:34.51 > ns-slapd > 4847 dirsrv 20 0 5133196 3.150g 281272 S 0.0 5.0 0:13.53 ns-slapd > 4848 dirsrv 20 0 5133196 3.150g 281272 S 0.0 5.0 0:13.16 ns-slapd > 4849 dirsrv 20 0 5133196 3.150g 281272 S 0.0 5.0 0:45.27 ns-slapd > 4850 dirsrv 20 0 5133196 3.150g 281272 S 0.0 5.0 0:01.60 ns-slapd > 4851 dirsrv 20 0 5133196 3.150g 281272 S 0.0 5.0 0:00.00 ns-slapd > 4852 dirsrv 20 0 5133196 3.150g 281272 S 0.0 5.0 0:00.00 ns-slapd > 4853 dirsrv 20 0 5133196 3.150g 281272 S 0.0 5.0 0:00.00 ns-slapd > 4854 dirsrv 20 0 5133196 3.150g 281272 S 0.0 5.0 0:00.00 ns-slapd > 4855 dirsrv 20 0 5133196 3.150g 281272 S 0.0 5.0 0:03.02 ns-slapd > 4856 dirsrv 20 0 5133196 3.150g 281272 S 0.0 5.0 0:00.26 ns-slapd > 4859 dirsrv 20 0 5133196 3.150g 281272 S 0.0 5.0 3:18.67 ns-slapd > 4860 dirsrv 20 0 5133196 3.150g 281272 S 0.0 5.0 3:19.82 ns-slapd > 4861 dirsrv 20 0 5133196 3.150g 281272 S 0.0 5.0 3:20.23 ns-slapd > 4862 dirsrv 20 0 5133196 3.150g 281272 S 0.0 5.0 3:22.65 ns-slapd > 4863 dirsrv 20 0 5133196 3.150g 281272 S 0.0 5.0 3:16.64 ns-slapd > 4864 dirsrv 20 0 5133196 3.150g 281272 S 0.0 5.0 3:17.39 ns-slapd > 4865 dirsrv 20 0 5133196 3.150g 281272 S 0.0 5.0 3:17.80 ns-slapd > > > Thanks > Rakesh > > On Tue, Sep 6, 2016 at 12:09 AM, Rakesh Rajasekharan > > > wrote: > > Hi Thierry/Ludwig, > > Unfortunately I had by then restarted the service and since then > have been trying to hit the hang issue again but no luck this time > > It has been running fine for the last few hours now in-spite of > trying all that I believe could cause the hang like making > simultaneous client addition requests. > > I will continue testing this and next time I get it I will make > sure I get the pstack and also get the dump of db lock info. > > Thanks, > Rakesh > > > > On Mon, Sep 5, 2016 at 6:14 PM, Ludwig Krispenz > > wrote: > > > On 09/05/2016 02:31 PM, thierry bordaz wrote: >> Hi Rakesh, >> >> Thanks you soooo much for you continuous help on this !! >> >> If the hang is still going on, would you dump the DB lock >> information with: >> >> db_stat -CA -N -h /var/lib/dirsrv/slapd-/db >> >/tmp/db_lock_info >> >> It is looking like several threads are waiting for >> one/several db locks. With the lock info we will know which >> thread acquired it/them. >> A guess is that it could be the checkpointing thread but this >> one is also waiting (on a memory pool). >> >> I am copying Ludwig on this thread in case this pstack rings >> any bell. > the frequent presence of __db_tas_mutex_lock() looks like the > NUMA problem of libdb, although the threads already have > stopped spinning (maybe this was an effect of gcore). > If the hang aoccurs again, could you check cpu consumption > top -H -p > and get a pstack > pstack >> >> best regards >> thierry >> >> On 09/05/2016 02:05 PM, Rakesh Rajasekharan wrote: >>> Hi Thierry, >>> >>> I was able to now reporduce the issue.. >>> >>> the ns-slapd process just got hung,.. however, no core file >>> was generate.. >>> >>> I manually created a coredump using gcore and then took a >>> strace of it >>> this is strace from the coredump generated >>> >>> GNU gdb (GDB) Red Hat Enterprise Linux 7.6.1-80.el7 >>> Copyright (C) 2013 Free Software Foundation, Inc. >>> License GPLv3+: GNU GPL version 3 or later >>> >> > >>> This is free software: you are free to change and >>> redistribute it. >>> There is NO WARRANTY, to the extent permitted by law. Type >>> "show copying" >>> and "show warranty" for details. >>> This GDB was configured as "x86_64-redhat-linux-gnu". >>> For bug reporting instructions, please see: >>> >> >... >>> Reading symbols from /usr/sbin/ns-slapd...Reading symbols >>> from /usr/sbin/ns-slapd...(no debugging symbols found)...done. >>> (no debugging symbols found)...done. >>> [New LWP 105159] >>> [New LWP 105160] >>> [New LWP 105161] >>> [New LWP 105162] >>> [New LWP 105163] >>> [New LWP 105164] >>> [New LWP 105165] >>> [New LWP 105166] >>> [New LWP 105167] >>> [New LWP 105168] >>> [New LWP 105171] >>> [New LWP 105172] >>> [New LWP 105173] >>> [New LWP 105174] >>> [New LWP 105175] >>> [New LWP 105176] >>> [New LWP 105177] >>> [New LWP 105178] >>> [New LWP 105179] >>> [New LWP 105180] >>> [New LWP 105181] >>> [New LWP 105182] >>> [New LWP 105183] >>> [New LWP 105184] >>> [New LWP 105185] >>> [New LWP 105186] >>> [New LWP 105187] >>> [New LWP 105188] >>> [New LWP 105189] >>> [New LWP 105190] >>> [New LWP 105191] >>> [New LWP 105192] >>> [New LWP 105193] >>> [New LWP 105194] >>> [New LWP 105195] >>> [New LWP 105196] >>> [New LWP 105197] >>> [New LWP 105198] >>> [New LWP 105199] >>> [New LWP 105200] >>> [New LWP 105201] >>> [New LWP 105203] >>> [New LWP 105215] >>> [New LWP 105158] >>> [Thread debugging using libthread_db enabled] >>> Using host libthread_db library "/lib64/libthread_db.so.1". >>> Core was generated by `/usr/sbin/ns-slapd'. >>> #0 0x00007f1b48ae69b3 in select () from /lib64/libc.so.6 >>> >>> Thread 44 (Thread 0x7f1b4b6d9840 (LWP 105158)): >>> #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 >>> () from >>> /lib64/libpthread.so.0 >>> No symbol table info available. >>> #1 0x00007f1b4941b2c3 in PR_EnterMonitor () from >>> /lib64/libnspr4.so >>> No symbol table info available. >>> #2 0x00007f1b4b7125ed in setup_pr_read_pds () >>> No symbol table info available. >>> #3 0x00007f1b4b715fd9 in slapd_daemon () >>> No symbol table info available. >>> #4 0x00007f1b4b707c6c in main () >>> No symbol table info available. >>> >>> Thread 43 (Thread 0x7f1a5df0f700 (LWP 105215)): >>> #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 >>> () from >>> /lib64/libpthread.so.0 >>> No symbol table info available. >>> #1 0x00007f1b4941b050 in PR_WaitCondVar () from >>> /lib64/libnspr4.so >>> No symbol table info available. >>> #2 0x00007f1b4b71df25 in ps_send_results () >>> No symbol table info available. >>> #3 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so >>> No symbol table info available. >>> #4 0x00007f1b48dc1dc5 in start_thread () from >>> /lib64/libpthread.so.0 >>> No symbol table info available. >>> #5 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 >>> No symbol table info available. >>> >>> Thread 42 (Thread 0x7f1b03fe7700 (LWP 105203)): >>> #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 >>> () from >>> /lib64/libpthread.so.0 >>> No symbol table info available. >>> #1 0x00007f1b4941b050 in PR_WaitCondVar () from >>> /lib64/libnspr4.so >>> No symbol table info available. >>> #2 0x00007f1b4b71df25 in ps_send_results () >>> No symbol table info available. >>> #3 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so >>> No symbol table info available. >>> #4 0x00007f1b48dc1dc5 in start_thread () from >>> /lib64/libpthread.so.0 >>> No symbol table info available. >>> #5 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 >>> No symbol table info available. >>> >>> Thread 41 (Thread 0x7f1b047e8700 (LWP 105201)): >>> #0 0x00007f1b48ae69b3 in select () from /lib64/libc.so.6 >>> No symbol table info available. >>> #1 0x00007f1b4b2710e9 in DS_Sleep () from >>> /usr/lib64/dirsrv/libslapd.so.0 >>> No symbol table info available. >>> #2 0x00007f1b4b712325 in time_thread () >>> No symbol table info available. >>> #3 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so >>> No symbol table info available. >>> #4 0x00007f1b48dc1dc5 in start_thread () from >>> /lib64/libpthread.so.0 >>> No symbol table info available. >>> #5 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 >>> No symbol table info available. >>> >>> Thread 40 (Thread 0x7f1b04fe9700 (LWP 105200)): >>> #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 >>> () from >>> /lib64/libpthread.so.0 >>> No symbol table info available. >>> #1 0x00007f1b433212f3 in __db_hybrid_mutex_suspend () from >>> /lib64/libdb-5.3.so >>> No symbol table info available. >>> #2 0x00007f1b43320640 in __db_tas_mutex_lock () from >>> /lib64/libdb-5.3.so >>> No symbol table info available. >>> #3 0x00007f1b4343ae1b in __memp_fget () from >>> /lib64/libdb-5.3.so >>> No symbol table info available. >>> #4 0x00007f1b4333e211 in __bam_search () from >>> /lib64/libdb-5.3.so >>> No symbol table info available. >>> #5 0x00007f1b43329256 in __bamc_search () from >>> /lib64/libdb-5.3.so >>> No symbol table info available. >>> #6 0x00007f1b4332d6c4 in __bamc_put () from >>> /lib64/libdb-5.3.so >>> No symbol table info available. >>> #7 0x00007f1b433e46d5 in __dbc_iput () from >>> /lib64/libdb-5.3.so >>> No symbol table info available. >>> #8 0x00007f1b433dfa0e in __db_put () from >>> /lib64/libdb-5.3.so >>> No symbol table info available. >>> #9 0x00007f1b433f4fa4 in __db_put_pp () from >>> /lib64/libdb-5.3.so >>> No symbol table info available. >>> #10 0x00007f1b3e8cf4cf in idl_new_insert_key () from >>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>> No symbol table info available. >>> #11 0x00007f1b3e8de5b5 in addordel_values_sv.isra.3 () from >>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>> No symbol table info available. >>> #12 0x00007f1b3e8df25a in index_addordel_values_ext_sv () >>> from /usr/lib64/dirsrv/plugins/libback-ldbm.so >>> No symbol table info available. >>> #13 0x00007f1b3e8df694 in index_addordel_values_sv () from >>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>> No symbol table info available. >>> #14 0x00007f1b3e8df75a in index_addordel_entry () from >>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>> No symbol table info available. >>> #15 0x00007f1b3e8e25fa in ldbm_back_add () from >>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>> No symbol table info available. >>> #16 0x00007f1b4b1e4cca in op_shared_add () from >>> /usr/lib64/dirsrv/libslapd.so.0 >>> No symbol table info available. >>> #17 0x00007f1b4b1e6020 in do_add () from >>> /usr/lib64/dirsrv/libslapd.so.0 >>> No symbol table info available. >>> #18 0x00007f1b4b711303 in connection_threadmain () >>> No symbol table info available. >>> #19 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so >>> No symbol table info available. >>> #20 0x00007f1b48dc1dc5 in start_thread () from >>> /lib64/libpthread.so.0 >>> No symbol table info available. >>> #21 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 >>> No symbol table info available. >>> >>> Thread 39 (Thread 0x7f1b057ea700 (LWP 105199)): >>> #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 >>> () from >>> /lib64/libpthread.so.0 >>> No symbol table info available. >>> #1 0x00007f1b4941b050 in PR_WaitCondVar () from >>> /lib64/libnspr4.so >>> No symbol table info available. >>> #2 0x00007f1b4b70efee in connection_wait_for_new_work () >>> No symbol table info available. >>> #3 0x00007f1b4b710719 in connection_threadmain () >>> No symbol table info available. >>> #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so >>> No symbol table info available. >>> #5 0x00007f1b48dc1dc5 in start_thread () from >>> /lib64/libpthread.so.0 >>> No symbol table info available. >>> #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 >>> No symbol table info available. >>> >>> Thread 38 (Thread 0x7f1b05feb700 (LWP 105198)): >>> #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 >>> () from >>> /lib64/libpthread.so.0 >>> No symbol table info available. >>> #1 0x00007f1b4941b050 in PR_WaitCondVar () from >>> /lib64/libnspr4.so >>> No symbol table info available. >>> #2 0x00007f1b4b70efee in connection_wait_for_new_work () >>> No symbol table info available. >>> #3 0x00007f1b4b710719 in connection_threadmain () >>> No symbol table info available. >>> #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so >>> No symbol table info available. >>> #5 0x00007f1b48dc1dc5 in start_thread () from >>> /lib64/libpthread.so.0 >>> No symbol table info available. >>> #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 >>> No symbol table info available. >>> >>> Thread 37 (Thread 0x7f1b067ec700 (LWP 105197)): >>> #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 >>> () from >>> /lib64/libpthread.so.0 >>> No symbol table info available. >>> #1 0x00007f1b4941b050 in PR_WaitCondVar () from >>> /lib64/libnspr4.so >>> No symbol table info available. >>> #2 0x00007f1b4b70efee in connection_wait_for_new_work () >>> No symbol table info available. >>> #3 0x00007f1b4b710719 in connection_threadmain () >>> No symbol table info available. >>> #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so >>> No symbol table info available. >>> #5 0x00007f1b48dc1dc5 in start_thread () from >>> /lib64/libpthread.so.0 >>> No symbol table info available. >>> #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 >>> No symbol table info available. >>> >>> Thread 36 (Thread 0x7f1b06fed700 (LWP 105196)): >>> #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 >>> () from >>> /lib64/libpthread.so.0 >>> No symbol table info available. >>> #1 0x00007f1b4941b050 in PR_WaitCondVar () from >>> /lib64/libnspr4.so >>> No symbol table info available. >>> #2 0x00007f1b4b70efee in connection_wait_for_new_work () >>> No symbol table info available. >>> #3 0x00007f1b4b710719 in connection_threadmain () >>> No symbol table info available. >>> #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so >>> No symbol table info available. >>> #5 0x00007f1b48dc1dc5 in start_thread () from >>> /lib64/libpthread.so.0 >>> No symbol table info available. >>> #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 >>> No symbol table info available. >>> >>> Thread 35 (Thread 0x7f1b077ee700 (LWP 105195)): >>> #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 >>> () from >>> /lib64/libpthread.so.0 >>> No symbol table info available. >>> #1 0x00007f1b4941b2c3 in PR_EnterMonitor () from >>> /lib64/libnspr4.so >>> No symbol table info available. >>> #2 0x00007f1b3e8c23e6 in dblayer_txn_begin () from >>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>> No symbol table info available. >>> #3 0x00007f1b3e8fe007 in ldbm_back_modify () from >>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>> No symbol table info available. >>> #4 0x00007f1b4b22b541 in op_shared_modify () from >>> /usr/lib64/dirsrv/libslapd.so.0 >>> No symbol table info available. >>> #5 0x00007f1b4b22c004 in modify_internal_pb () from >>> /usr/lib64/dirsrv/libslapd.so.0 >>> No symbol table info available. >>> #6 0x00007f1b3eb48b60 in ipapwd_apply_mods () from >>> /usr/lib64/dirsrv/plugins/libipa_pwd_extop.so >>> No symbol table info available. >>> #7 0x00007f1b3eb4c8e0 in store_new_keys () from >>> /usr/lib64/dirsrv/plugins/libipa_pwd_extop.so >>> No symbol table info available. >>> #8 0x00007f1b3eb4ead0 in ipapwd_extop () from >>> /usr/lib64/dirsrv/plugins/libipa_pwd_extop.so >>> No symbol table info available. >>> #9 0x00007f1b4b23bba2 in plugin_call_exop_plugins () from >>> /usr/lib64/dirsrv/libslapd.so.0 >>> No symbol table info available. >>> #10 0x00007f1b4b717efe in do_extended () >>> No symbol table info available. >>> #11 0x00007f1b4b71141a in connection_threadmain () >>> No symbol table info available. >>> #12 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so >>> No symbol table info available. >>> #13 0x00007f1b48dc1dc5 in start_thread () from >>> /lib64/libpthread.so.0 >>> No symbol table info available. >>> #14 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 >>> No symbol table info available. >>> >>> Thread 34 (Thread 0x7f1b07fef700 (LWP 105194)): >>> #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 >>> () from >>> /lib64/libpthread.so.0 >>> No symbol table info available. >>> #1 0x00007f1b4941b050 in PR_WaitCondVar () from >>> /lib64/libnspr4.so >>> No symbol table info available. >>> #2 0x00007f1b4b70efee in connection_wait_for_new_work () >>> No symbol table info available. >>> #3 0x00007f1b4b710719 in connection_threadmain () >>> No symbol table info available. >>> #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so >>> No symbol table info available. >>> #5 0x00007f1b48dc1dc5 in start_thread () from >>> /lib64/libpthread.so.0 >>> No symbol table info available. >>> #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 >>> No symbol table info available. >>> >>> Thread 33 (Thread 0x7f1b087f0700 (LWP 105193)): >>> #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 >>> () from >>> /lib64/libpthread.so.0 >>> No symbol table info available. >>> #1 0x00007f1b4941b050 in PR_WaitCondVar () from >>> /lib64/libnspr4.so >>> No symbol table info available. >>> #2 0x00007f1b4b70efee in connection_wait_for_new_work () >>> No symbol table info available. >>> #3 0x00007f1b4b710719 in connection_threadmain () >>> No symbol table info available. >>> #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so >>> No symbol table info available. >>> #5 0x00007f1b48dc1dc5 in start_thread () from >>> /lib64/libpthread.so.0 >>> No symbol table info available. >>> #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 >>> No symbol table info available. >>> >>> Thread 32 (Thread 0x7f1b08ff1700 (LWP 105192)): >>> #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 >>> () from >>> /lib64/libpthread.so.0 >>> No symbol table info available. >>> #1 0x00007f1b4941b050 in PR_WaitCondVar () from >>> /lib64/libnspr4.so >>> No symbol table info available. >>> #2 0x00007f1b4b70efee in connection_wait_for_new_work () >>> No symbol table info available. >>> #3 0x00007f1b4b710719 in connection_threadmain () >>> No symbol table info available. >>> #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so >>> No symbol table info available. >>> #5 0x00007f1b48dc1dc5 in start_thread () from >>> /lib64/libpthread.so.0 >>> No symbol table info available. >>> #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 >>> No symbol table info available. >>> >>> Thread 31 (Thread 0x7f1b097f2700 (LWP 105191)): >>> #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 >>> () from >>> /lib64/libpthread.so.0 >>> No symbol table info available. >>> #1 0x00007f1b4941b050 in PR_WaitCondVar () from >>> /lib64/libnspr4.so >>> No symbol table info available. >>> #2 0x00007f1b4b70efee in connection_wait_for_new_work () >>> No symbol table info available. >>> #3 0x00007f1b4b710719 in connection_threadmain () >>> No symbol table info available. >>> #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so >>> No symbol table info available. >>> #5 0x00007f1b48dc1dc5 in start_thread () from >>> /lib64/libpthread.so.0 >>> No symbol table info available. >>> #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 >>> No symbol table info available. >>> >>> Thread 30 (Thread 0x7f1b09ff3700 (LWP 105190)): >>> #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 >>> () from >>> /lib64/libpthread.so.0 >>> No symbol table info available. >>> #1 0x00007f1b433212f3 in __db_hybrid_mutex_suspend () from >>> /lib64/libdb-5.3.so >>> No symbol table info available. >>> #2 0x00007f1b43320640 in __db_tas_mutex_lock () from >>> /lib64/libdb-5.3.so >>> No symbol table info available. >>> #3 0x00007f1b433cad3a in __lock_get_internal () from >>> /lib64/libdb-5.3.so >>> No symbol table info available. >>> #4 0x00007f1b433cb820 in __lock_get () from >>> /lib64/libdb-5.3.so >>> No symbol table info available. >>> #5 0x00007f1b433f7142 in __db_lget () from >>> /lib64/libdb-5.3.so >>> No symbol table info available. >>> #6 0x00007f1b4333e605 in __bam_search () from >>> /lib64/libdb-5.3.so >>> No symbol table info available. >>> #7 0x00007f1b43329256 in __bamc_search () from >>> /lib64/libdb-5.3.so >>> No symbol table info available. >>> #8 0x00007f1b4332ad0f in __bamc_get () from >>> /lib64/libdb-5.3.so >>> No symbol table info available. >>> #9 0x00007f1b433e3ca6 in __dbc_iget () from >>> /lib64/libdb-5.3.so >>> No symbol table info available. >>> #10 0x00007f1b433f2b02 in __dbc_get_pp () from >>> /lib64/libdb-5.3.so >>> No symbol table info available. >>> #11 0x00007f1b3e8ce680 in idl_new_fetch () from >>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>> No symbol table info available. >>> #12 0x00007f1b3e8dcb26 in index_read_ext_allids () from >>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>> No symbol table info available. >>> #13 0x00007f1b3e8c7284 in keys2idl () from >>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>> No symbol table info available. >>> #14 0x00007f1b3e8c79e3 in ava_candidates.isra.0 () from >>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>> No symbol table info available. >>> #15 0x00007f1b3e8c7fd2 in filter_candidates_ext () from >>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>> No symbol table info available. >>> #16 0x00007f1b3e8c9096 in list_candidates () from >>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>> No symbol table info available. >>> #17 0x00007f1b3e8c7f40 in filter_candidates_ext () from >>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>> No symbol table info available. >>> #18 0x00007f1b3e8c9096 in list_candidates () from >>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>> No symbol table info available. >>> #19 0x00007f1b3e8c7f40 in filter_candidates_ext () from >>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>> No symbol table info available. >>> #20 0x00007f1b3e8c9096 in list_candidates () from >>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>> No symbol table info available. >>> #21 0x00007f1b3e8c7f40 in filter_candidates_ext () from >>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>> No symbol table info available. >>> #22 0x00007f1b3e90380c in subtree_candidates () from >>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>> No symbol table info available. >>> #23 0x00007f1b3e904e7f in ldbm_back_search () from >>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>> No symbol table info available. >>> #24 0x00007f1b4b2321d6 in op_shared_search () from >>> /usr/lib64/dirsrv/libslapd.so.0 >>> No symbol table info available. >>> #25 0x00007f1b4b72333e in do_search () >>> No symbol table info available. >>> #26 0x00007f1b4b7113c3 in connection_threadmain () >>> No symbol table info available. >>> #27 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so >>> No symbol table info available. >>> #28 0x00007f1b48dc1dc5 in start_thread () from >>> /lib64/libpthread.so.0 >>> No symbol table info available. >>> #29 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 >>> No symbol table info available. >>> >>> Thread 29 (Thread 0x7f1b0a7f4700 (LWP 105189)): >>> #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 >>> () from >>> /lib64/libpthread.so.0 >>> No symbol table info available. >>> #1 0x00007f1b4941b050 in PR_WaitCondVar () from >>> /lib64/libnspr4.so >>> No symbol table info available. >>> #2 0x00007f1b4b70efee in connection_wait_for_new_work () >>> No symbol table info available. >>> #3 0x00007f1b4b710719 in connection_threadmain () >>> No symbol table info available. >>> #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so >>> No symbol table info available. >>> #5 0x00007f1b48dc1dc5 in start_thread () from >>> /lib64/libpthread.so.0 >>> No symbol table info available. >>> #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 >>> No symbol table info available. >>> >>> Thread 28 (Thread 0x7f1b0aff5700 (LWP 105188)): >>> #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 >>> () from >>> /lib64/libpthread.so.0 >>> No symbol table info available. >>> #1 0x00007f1b4941b050 in PR_WaitCondVar () from >>> /lib64/libnspr4.so >>> No symbol table info available. >>> #2 0x00007f1b4b70efee in connection_wait_for_new_work () >>> No symbol table info available. >>> #3 0x00007f1b4b710719 in connection_threadmain () >>> No symbol table info available. >>> #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so >>> No symbol table info available. >>> #5 0x00007f1b48dc1dc5 in start_thread () from >>> /lib64/libpthread.so.0 >>> No symbol table info available. >>> #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 >>> No symbol table info available. >>> >>> Thread 27 (Thread 0x7f1b0b7f6700 (LWP 105187)): >>> #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 >>> () from >>> /lib64/libpthread.so.0 >>> No symbol table info available. >>> #1 0x00007f1b4941b050 in PR_WaitCondVar () from >>> /lib64/libnspr4.so >>> No symbol table info available. >>> #2 0x00007f1b4b70efee in connection_wait_for_new_work () >>> No symbol table info available. >>> #3 0x00007f1b4b710719 in connection_threadmain () >>> No symbol table info available. >>> #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so >>> No symbol table info available. >>> #5 0x00007f1b48dc1dc5 in start_thread () from >>> /lib64/libpthread.so.0 >>> No symbol table info available. >>> #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 >>> No symbol table info available. >>> >>> Thread 26 (Thread 0x7f1b0bff7700 (LWP 105186)): >>> #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 >>> () from >>> /lib64/libpthread.so.0 >>> No symbol table info available. >>> #1 0x00007f1b433212f3 in __db_hybrid_mutex_suspend () from >>> /lib64/libdb-5.3.so >>> No symbol table info available. >>> #2 0x00007f1b43320640 in __db_tas_mutex_lock () from >>> /lib64/libdb-5.3.so >>> No symbol table info available. >>> #3 0x00007f1b433cad3a in __lock_get_internal () from >>> /lib64/libdb-5.3.so >>> No symbol table info available. >>> #4 0x00007f1b433cb820 in __lock_get () from >>> /lib64/libdb-5.3.so >>> No symbol table info available. >>> #5 0x00007f1b433f7142 in __db_lget () from >>> /lib64/libdb-5.3.so >>> No symbol table info available. >>> #6 0x00007f1b4333e605 in __bam_search () from >>> /lib64/libdb-5.3.so >>> No symbol table info available. >>> #7 0x00007f1b43329256 in __bamc_search () from >>> /lib64/libdb-5.3.so >>> No symbol table info available. >>> #8 0x00007f1b4332ad0f in __bamc_get () from >>> /lib64/libdb-5.3.so >>> No symbol table info available. >>> #9 0x00007f1b433e3ca6 in __dbc_iget () from >>> /lib64/libdb-5.3.so >>> No symbol table info available. >>> #10 0x00007f1b433f2b02 in __dbc_get_pp () from >>> /lib64/libdb-5.3.so >>> No symbol table info available. >>> #11 0x00007f1b3e8ce680 in idl_new_fetch () from >>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>> No symbol table info available. >>> #12 0x00007f1b3e8dcb26 in index_read_ext_allids () from >>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>> No symbol table info available. >>> #13 0x00007f1b3e8c7284 in keys2idl () from >>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>> No symbol table info available. >>> #14 0x00007f1b3e8c79e3 in ava_candidates.isra.0 () from >>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>> No symbol table info available. >>> #15 0x00007f1b3e8c7fd2 in filter_candidates_ext () from >>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>> No symbol table info available. >>> #16 0x00007f1b3e8c9096 in list_candidates () from >>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>> No symbol table info available. >>> #17 0x00007f1b3e8c7f40 in filter_candidates_ext () from >>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>> No symbol table info available. >>> #18 0x00007f1b3e90380c in subtree_candidates () from >>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>> No symbol table info available. >>> #19 0x00007f1b3e904e7f in ldbm_back_search () from >>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>> No symbol table info available. >>> #20 0x00007f1b4b2321d6 in op_shared_search () from >>> /usr/lib64/dirsrv/libslapd.so.0 >>> No symbol table info available. >>> #21 0x00007f1b4b24294e in search_internal_callback_pb () >>> from /usr/lib64/dirsrv/libslapd.so.0 >>> No symbol table info available. >>> #22 0x00007f1b4b242be8 in search_internal_pb () from >>> /usr/lib64/dirsrv/libslapd.so.0 >>> No symbol table info available. >>> #23 0x00007f1b4b72149b in ids_sasl_canon_user () >>> No symbol table info available. >>> #24 0x00007f1b491e193b in _sasl_canon_user () from >>> /lib64/libsasl2.so.3 >>> No symbol table info available. >>> #25 0x00007f1b491e1c4c in _sasl_canon_user_lookup () from >>> /lib64/libsasl2.so.3 >>> No symbol table info available. >>> #26 0x00007f1b430e9fb9 in gssapi_server_mech_step () from >>> /usr/lib64/sasl2/libgssapiv2.so >>> No symbol table info available. >>> #27 0x00007f1b491edb9b in sasl_server_step () from >>> /lib64/libsasl2.so.3 >>> No symbol table info available. >>> #28 0x00007f1b4b72283b in ids_sasl_check_bind () >>> No symbol table info available. >>> #29 0x00007f1b4b70a61a in do_bind () >>> No symbol table info available. >>> #30 0x00007f1b4b7113fd in connection_threadmain () >>> No symbol table info available. >>> #31 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so >>> No symbol table info available. >>> #32 0x00007f1b48dc1dc5 in start_thread () from >>> /lib64/libpthread.so.0 >>> No symbol table info available. >>> #33 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 >>> No symbol table info available. >>> >>> Thread 25 (Thread 0x7f1b0c7f8700 (LWP 105185)): >>> #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 >>> () from >>> /lib64/libpthread.so.0 >>> No symbol table info available. >>> #1 0x00007f1b4941b050 in PR_WaitCondVar () from >>> /lib64/libnspr4.so >>> No symbol table info available. >>> #2 0x00007f1b4b70efee in connection_wait_for_new_work () >>> No symbol table info available. >>> #3 0x00007f1b4b710719 in connection_threadmain () >>> No symbol table info available. >>> #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so >>> No symbol table info available. >>> #5 0x00007f1b48dc1dc5 in start_thread () from >>> /lib64/libpthread.so.0 >>> No symbol table info available. >>> #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 >>> No symbol table info available. >>> >>> Thread 24 (Thread 0x7f1b0cff9700 (LWP 105184)): >>> #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 >>> () from >>> /lib64/libpthread.so.0 >>> No symbol table info available. >>> #1 0x00007f1b4941b050 in PR_WaitCondVar () from >>> /lib64/libnspr4.so >>> No symbol table info available. >>> #2 0x00007f1b4b70efee in connection_wait_for_new_work () >>> No symbol table info available. >>> #3 0x00007f1b4b710719 in connection_threadmain () >>> No symbol table info available. >>> #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so >>> No symbol table info available. >>> #5 0x00007f1b48dc1dc5 in start_thread () from >>> /lib64/libpthread.so.0 >>> No symbol table info available. >>> #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 >>> No symbol table info available. >>> >>> Thread 23 (Thread 0x7f1b0d7fa700 (LWP 105183)): >>> #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 >>> () from >>> /lib64/libpthread.so.0 >>> No symbol table info available. >>> #1 0x00007f1b4941b050 in PR_WaitCondVar () from >>> /lib64/libnspr4.so >>> No symbol table info available. >>> #2 0x00007f1b4b70efee in connection_wait_for_new_work () >>> No symbol table info available. >>> #3 0x00007f1b4b710719 in connection_threadmain () >>> No symbol table info available. >>> #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so >>> No symbol table info available. >>> #5 0x00007f1b48dc1dc5 in start_thread () from >>> /lib64/libpthread.so.0 >>> No symbol table info available. >>> #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 >>> No symbol table info available. >>> >>> Thread 22 (Thread 0x7f1b0dffb700 (LWP 105182)): >>> #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 >>> () from >>> /lib64/libpthread.so.0 >>> No symbol table info available. >>> #1 0x00007f1b4941b050 in PR_WaitCondVar () from >>> /lib64/libnspr4.so >>> No symbol table info available. >>> #2 0x00007f1b4b70efee in connection_wait_for_new_work () >>> No symbol table info available. >>> #3 0x00007f1b4b710719 in connection_threadmain () >>> No symbol table info available. >>> #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so >>> No symbol table info available. >>> #5 0x00007f1b48dc1dc5 in start_thread () from >>> /lib64/libpthread.so.0 >>> No symbol table info available. >>> #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 >>> No symbol table info available. >>> >>> Thread 21 (Thread 0x7f1b0e7fc700 (LWP 105181)): >>> #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 >>> () from >>> /lib64/libpthread.so.0 >>> No symbol table info available. >>> #1 0x00007f1b4941b050 in PR_WaitCondVar () from >>> /lib64/libnspr4.so >>> No symbol table info available. >>> #2 0x00007f1b4b70efee in connection_wait_for_new_work () >>> No symbol table info available. >>> #3 0x00007f1b4b710719 in connection_threadmain () >>> No symbol table info available. >>> #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so >>> No symbol table info available. >>> #5 0x00007f1b48dc1dc5 in start_thread () from >>> /lib64/libpthread.so.0 >>> No symbol table info available. >>> #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 >>> No symbol table info available. >>> >>> Thread 20 (Thread 0x7f1b0effd700 (LWP 105180)): >>> #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 >>> () from >>> /lib64/libpthread.so.0 >>> No symbol table info available. >>> #1 0x00007f1b433212f3 in __db_hybrid_mutex_suspend () from >>> /lib64/libdb-5.3.so >>> No symbol table info available. >>> #2 0x00007f1b43320b37 in __db_tas_mutex_readlock () from >>> /lib64/libdb-5.3.so >>> No symbol table info available. >>> #3 0x00007f1b4343ad35 in __memp_fget () from >>> /lib64/libdb-5.3.so >>> No symbol table info available. >>> #4 0x00007f1b433e4333 in __dbc_iget () from >>> /lib64/libdb-5.3.so >>> No symbol table info available. >>> #5 0x00007f1b433f2b02 in __dbc_get_pp () from >>> /lib64/libdb-5.3.so >>> No symbol table info available. >>> #6 0x00007f1b3e8ce822 in idl_new_fetch () from >>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>> No symbol table info available. >>> #7 0x00007f1b3e8dcb26 in index_read_ext_allids () from >>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>> No symbol table info available. >>> #8 0x00007f1b3e8c7284 in keys2idl () from >>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>> No symbol table info available. >>> #9 0x00007f1b3e8c79e3 in ava_candidates.isra.0 () from >>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>> No symbol table info available. >>> #10 0x00007f1b3e8c7fd2 in filter_candidates_ext () from >>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>> No symbol table info available. >>> #11 0x00007f1b3e8c9096 in list_candidates () from >>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>> No symbol table info available. >>> #12 0x00007f1b3e8c7f40 in filter_candidates_ext () from >>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>> No symbol table info available. >>> #13 0x00007f1b3e8c9096 in list_candidates () from >>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>> No symbol table info available. >>> #14 0x00007f1b3e8c7f40 in filter_candidates_ext () from >>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>> No symbol table info available. >>> #15 0x00007f1b3e8c9096 in list_candidates () from >>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>> No symbol table info available. >>> #16 0x00007f1b3e8c7f40 in filter_candidates_ext () from >>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>> No symbol table info available. >>> #17 0x00007f1b3e90380c in subtree_candidates () from >>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>> No symbol table info available. >>> #18 0x00007f1b3e904e7f in ldbm_back_search () from >>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>> No symbol table info available. >>> #19 0x00007f1b4b2321d6 in op_shared_search () from >>> /usr/lib64/dirsrv/libslapd.so.0 >>> No symbol table info available. >>> #20 0x00007f1b4b72333e in do_search () >>> No symbol table info available. >>> #21 0x00007f1b4b7113c3 in connection_threadmain () >>> No symbol table info available. >>> #22 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so >>> No symbol table info available. >>> #23 0x00007f1b48dc1dc5 in start_thread () from >>> /lib64/libpthread.so.0 >>> No symbol table info available. >>> #24 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 >>> No symbol table info available. >>> >>> Thread 19 (Thread 0x7f1b0f7fe700 (LWP 105179)): >>> #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 >>> () from >>> /lib64/libpthread.so.0 >>> No symbol table info available. >>> #1 0x00007f1b433212f3 in __db_hybrid_mutex_suspend () from >>> /lib64/libdb-5.3.so >>> No symbol table info available. >>> #2 0x00007f1b43320640 in __db_tas_mutex_lock () from >>> /lib64/libdb-5.3.so >>> No symbol table info available. >>> #3 0x00007f1b433cad3a in __lock_get_internal () from >>> /lib64/libdb-5.3.so >>> No symbol table info available. >>> #4 0x00007f1b433cb820 in __lock_get () from >>> /lib64/libdb-5.3.so >>> No symbol table info available. >>> #5 0x00007f1b433f7142 in __db_lget () from >>> /lib64/libdb-5.3.so >>> No symbol table info available. >>> #6 0x00007f1b4333e605 in __bam_search () from >>> /lib64/libdb-5.3.so >>> No symbol table info available. >>> #7 0x00007f1b43329256 in __bamc_search () from >>> /lib64/libdb-5.3.so >>> No symbol table info available. >>> #8 0x00007f1b4332ad0f in __bamc_get () from >>> /lib64/libdb-5.3.so >>> No symbol table info available. >>> #9 0x00007f1b433e3ca6 in __dbc_iget () from >>> /lib64/libdb-5.3.so >>> No symbol table info available. >>> #10 0x00007f1b433f2b02 in __dbc_get_pp () from >>> /lib64/libdb-5.3.so >>> No symbol table info available. >>> #11 0x00007f1b3e8ce680 in idl_new_fetch () from >>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>> No symbol table info available. >>> #12 0x00007f1b3e8dcb26 in index_read_ext_allids () from >>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>> No symbol table info available. >>> #13 0x00007f1b3e8c7284 in keys2idl () from >>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>> No symbol table info available. >>> #14 0x00007f1b3e8c79e3 in ava_candidates.isra.0 () from >>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>> No symbol table info available. >>> #15 0x00007f1b3e8c7fd2 in filter_candidates_ext () from >>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>> No symbol table info available. >>> #16 0x00007f1b3e8c9096 in list_candidates () from >>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>> No symbol table info available. >>> #17 0x00007f1b3e8c7f40 in filter_candidates_ext () from >>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>> No symbol table info available. >>> #18 0x00007f1b3e8c9096 in list_candidates () from >>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>> No symbol table info available. >>> #19 0x00007f1b3e8c7f40 in filter_candidates_ext () from >>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>> No symbol table info available. >>> #20 0x00007f1b3e8c9096 in list_candidates () from >>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>> No symbol table info available. >>> #21 0x00007f1b3e8c7f40 in filter_candidates_ext () from >>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>> No symbol table info available. >>> #22 0x00007f1b3e90380c in subtree_candidates () from >>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>> No symbol table info available. >>> #23 0x00007f1b3e904e7f in ldbm_back_search () from >>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>> No symbol table info available. >>> #24 0x00007f1b4b2321d6 in op_shared_search () from >>> /usr/lib64/dirsrv/libslapd.so.0 >>> No symbol table info available. >>> #25 0x00007f1b4b72333e in do_search () >>> No symbol table info available. >>> #26 0x00007f1b4b7113c3 in connection_threadmain () >>> No symbol table info available. >>> #27 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so >>> No symbol table info available. >>> #28 0x00007f1b48dc1dc5 in start_thread () from >>> /lib64/libpthread.so.0 >>> No symbol table info available. >>> #29 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 >>> No symbol table info available. >>> >>> Thread 18 (Thread 0x7f1b0ffff700 (LWP 105178)): >>> #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 >>> () from >>> /lib64/libpthread.so.0 >>> No symbol table info available. >>> #1 0x00007f1b4941b050 in PR_WaitCondVar () from >>> /lib64/libnspr4.so >>> No symbol table info available. >>> #2 0x00007f1b4b70efee in connection_wait_for_new_work () >>> No symbol table info available. >>> #3 0x00007f1b4b710719 in connection_threadmain () >>> No symbol table info available. >>> #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so >>> No symbol table info available. >>> #5 0x00007f1b48dc1dc5 in start_thread () from >>> /lib64/libpthread.so.0 >>> No symbol table info available. >>> #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 >>> No symbol table info available. >>> >>> Thread 17 (Thread 0x7f1b14d28700 (LWP 105177)): >>> #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 >>> () from >>> /lib64/libpthread.so.0 >>> No symbol table info available. >>> #1 0x00007f1b4941b050 in PR_WaitCondVar () from >>> /lib64/libnspr4.so >>> No symbol table info available. >>> #2 0x00007f1b4b70efee in connection_wait_for_new_work () >>> No symbol table info available. >>> #3 0x00007f1b4b710719 in connection_threadmain () >>> No symbol table info available. >>> #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so >>> No symbol table info available. >>> #5 0x00007f1b48dc1dc5 in start_thread () from >>> /lib64/libpthread.so.0 >>> No symbol table info available. >>> #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 >>> No symbol table info available. >>> >>> Thread 16 (Thread 0x7f1b15529700 (LWP 105176)): >>> #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 >>> () from >>> /lib64/libpthread.so.0 >>> No symbol table info available. >>> #1 0x00007f1b4941b050 in PR_WaitCondVar () from >>> /lib64/libnspr4.so >>> No symbol table info available. >>> #2 0x00007f1b4b70efee in connection_wait_for_new_work () >>> No symbol table info available. >>> #3 0x00007f1b4b710719 in connection_threadmain () >>> No symbol table info available. >>> #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so >>> No symbol table info available. >>> #5 0x00007f1b48dc1dc5 in start_thread () from >>> /lib64/libpthread.so.0 >>> No symbol table info available. >>> #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 >>> No symbol table info available. >>> >>> Thread 15 (Thread 0x7f1b15d2a700 (LWP 105175)): >>> #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 >>> () from >>> /lib64/libpthread.so.0 >>> No symbol table info available. >>> #1 0x00007f1b4941b050 in PR_WaitCondVar () from >>> /lib64/libnspr4.so >>> No symbol table info available. >>> #2 0x00007f1b4b70efee in connection_wait_for_new_work () >>> No symbol table info available. >>> #3 0x00007f1b4b710719 in connection_threadmain () >>> No symbol table info available. >>> #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so >>> No symbol table info available. >>> #5 0x00007f1b48dc1dc5 in start_thread () from >>> /lib64/libpthread.so.0 >>> No symbol table info available. >>> #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 >>> No symbol table info available. >>> >>> Thread 14 (Thread 0x7f1b1652b700 (LWP 105174)): >>> #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 >>> () from >>> /lib64/libpthread.so.0 >>> No symbol table info available. >>> #1 0x00007f1b4941b050 in PR_WaitCondVar () from >>> /lib64/libnspr4.so >>> No symbol table info available. >>> #2 0x00007f1b4b70efee in connection_wait_for_new_work () >>> No symbol table info available. >>> #3 0x00007f1b4b710719 in connection_threadmain () >>> No symbol table info available. >>> #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so >>> No symbol table info available. >>> #5 0x00007f1b48dc1dc5 in start_thread () from >>> /lib64/libpthread.so.0 >>> No symbol table info available. >>> #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 >>> No symbol table info available. >>> >>> Thread 13 (Thread 0x7f1b16d2c700 (LWP 105173)): >>> #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 >>> () from >>> /lib64/libpthread.so.0 >>> No symbol table info available. >>> #1 0x00007f1b433212f3 in __db_hybrid_mutex_suspend () from >>> /lib64/libdb-5.3.so >>> No symbol table info available. >>> #2 0x00007f1b43320640 in __db_tas_mutex_lock () from >>> /lib64/libdb-5.3.so >>> No symbol table info available. >>> #3 0x00007f1b433cad3a in __lock_get_internal () from >>> /lib64/libdb-5.3.so >>> No symbol table info available. >>> #4 0x00007f1b433cb820 in __lock_get () from >>> /lib64/libdb-5.3.so >>> No symbol table info available. >>> #5 0x00007f1b433f7142 in __db_lget () from >>> /lib64/libdb-5.3.so >>> No symbol table info available. >>> #6 0x00007f1b4333e605 in __bam_search () from >>> /lib64/libdb-5.3.so >>> No symbol table info available. >>> #7 0x00007f1b43329256 in __bamc_search () from >>> /lib64/libdb-5.3.so >>> No symbol table info available. >>> #8 0x00007f1b4332ad0f in __bamc_get () from >>> /lib64/libdb-5.3.so >>> No symbol table info available. >>> #9 0x00007f1b433e3ca6 in __dbc_iget () from >>> /lib64/libdb-5.3.so >>> No symbol table info available. >>> #10 0x00007f1b433f2b02 in __dbc_get_pp () from >>> /lib64/libdb-5.3.so >>> No symbol table info available. >>> #11 0x00007f1b3e8ce680 in idl_new_fetch () from >>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>> No symbol table info available. >>> #12 0x00007f1b3e8dcb26 in index_read_ext_allids () from >>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>> No symbol table info available. >>> #13 0x00007f1b3e8c7284 in keys2idl () from >>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>> No symbol table info available. >>> #14 0x00007f1b3e8c79e3 in ava_candidates.isra.0 () from >>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>> No symbol table info available. >>> #15 0x00007f1b3e8c7fd2 in filter_candidates_ext () from >>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>> No symbol table info available. >>> #16 0x00007f1b3e8c9096 in list_candidates () from >>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>> No symbol table info available. >>> #17 0x00007f1b3e8c7f40 in filter_candidates_ext () from >>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>> No symbol table info available. >>> #18 0x00007f1b3e8c9096 in list_candidates () from >>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>> No symbol table info available. >>> #19 0x00007f1b3e8c7f40 in filter_candidates_ext () from >>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>> No symbol table info available. >>> #20 0x00007f1b3e8c9096 in list_candidates () from >>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>> No symbol table info available. >>> #21 0x00007f1b3e8c7f40 in filter_candidates_ext () from >>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>> No symbol table info available. >>> #22 0x00007f1b3e90380c in subtree_candidates () from >>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>> No symbol table info available. >>> #23 0x00007f1b3e904e7f in ldbm_back_search () from >>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>> No symbol table info available. >>> #24 0x00007f1b4b2321d6 in op_shared_search () from >>> /usr/lib64/dirsrv/libslapd.so.0 >>> No symbol table info available. >>> #25 0x00007f1b4b72333e in do_search () >>> No symbol table info available. >>> #26 0x00007f1b4b7113c3 in connection_threadmain () >>> No symbol table info available. >>> #27 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so >>> No symbol table info available. >>> #28 0x00007f1b48dc1dc5 in start_thread () from >>> /lib64/libpthread.so.0 >>> No symbol table info available. >>> #29 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 >>> No symbol table info available. >>> >>> Thread 12 (Thread 0x7f1b1752d700 (LWP 105172)): >>> #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 >>> () from >>> /lib64/libpthread.so.0 >>> No symbol table info available. >>> #1 0x00007f1b4941b050 in PR_WaitCondVar () from >>> /lib64/libnspr4.so >>> No symbol table info available. >>> #2 0x00007f1b4b70efee in connection_wait_for_new_work () >>> No symbol table info available. >>> #3 0x00007f1b4b710719 in connection_threadmain () >>> No symbol table info available. >>> #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so >>> No symbol table info available. >>> #5 0x00007f1b48dc1dc5 in start_thread () from >>> /lib64/libpthread.so.0 >>> No symbol table info available. >>> #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 >>> No symbol table info available. >>> >>> Thread 11 (Thread 0x7f1b17d2e700 (LWP 105171)): >>> #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 >>> () from >>> /lib64/libpthread.so.0 >>> No symbol table info available. >>> #1 0x00007f1b4941b050 in PR_WaitCondVar () from >>> /lib64/libnspr4.so >>> No symbol table info available. >>> #2 0x00007f1b4b70efee in connection_wait_for_new_work () >>> No symbol table info available. >>> #3 0x00007f1b4b710719 in connection_threadmain () >>> No symbol table info available. >>> #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so >>> No symbol table info available. >>> #5 0x00007f1b48dc1dc5 in start_thread () from >>> /lib64/libpthread.so.0 >>> No symbol table info available. >>> #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 >>> No symbol table info available. >>> >>> Thread 10 (Thread 0x7f1b2509d700 (LWP 105168)): >>> #0 0x00007f1b48dc5a82 in >>> pthread_cond_timedwait@@GLIBC_2.3.2 >>> () from >>> /lib64/libpthread.so.0 >>> No symbol table info available. >>> #1 0x00007f1b4941ab07 in pt_TimedWait () from >>> /lib64/libnspr4.so >>> No symbol table info available. >>> #2 0x00007f1b4941afce in PR_WaitCondVar () from >>> /lib64/libnspr4.so >>> No symbol table info available. >>> #3 0x00007f1b4b2061e8 in eq_loop () from >>> /usr/lib64/dirsrv/libslapd.so.0 >>> No symbol table info available. >>> #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so >>> No symbol table info available. >>> #5 0x00007f1b48dc1dc5 in start_thread () from >>> /lib64/libpthread.so.0 >>> No symbol table info available. >>> #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 >>> No symbol table info available. >>> >>> Thread 9 (Thread 0x7f1b2589e700 (LWP 105167)): >>> #0 0x00007f1b48dc5a82 in >>> pthread_cond_timedwait@@GLIBC_2.3.2 >>> () from >>> /lib64/libpthread.so.0 >>> No symbol table info available. >>> #1 0x00007f1b4941ab07 in pt_TimedWait () from >>> /lib64/libnspr4.so >>> No symbol table info available. >>> #2 0x00007f1b4941afce in PR_WaitCondVar () from >>> /lib64/libnspr4.so >>> No symbol table info available. >>> #3 0x00007f1b4b719d03 in housecleaning () >>> No symbol table info available. >>> #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so >>> No symbol table info available. >>> #5 0x00007f1b48dc1dc5 in start_thread () from >>> /lib64/libpthread.so.0 >>> No symbol table info available. >>> #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 >>> No symbol table info available. >>> >>> Thread 8 (Thread 0x7f1b2609f700 (LWP 105166)): >>> #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 >>> () from >>> /lib64/libpthread.so.0 >>> No symbol table info available. >>> #1 0x00007f1b4941b050 in PR_WaitCondVar () from >>> /lib64/libnspr4.so >>> No symbol table info available. >>> #2 0x00007f1b4b260198 in slapi_wait_condvar () from >>> /usr/lib64/dirsrv/libslapd.so.0 >>> No symbol table info available. >>> #3 0x00007f1b3cf5deed in roles_cache_wait_on_change () from >>> /usr/lib64/dirsrv/plugins/libroles-plugin.so >>> No symbol table info available. >>> #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so >>> No symbol table info available. >>> #5 0x00007f1b48dc1dc5 in start_thread () from >>> /lib64/libpthread.so.0 >>> No symbol table info available. >>> #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 >>> No symbol table info available. >>> >>> Thread 7 (Thread 0x7f1b268a0700 (LWP 105165)): >>> #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 >>> () from >>> /lib64/libpthread.so.0 >>> No symbol table info available. >>> #1 0x00007f1b4941b050 in PR_WaitCondVar () from >>> /lib64/libnspr4.so >>> No symbol table info available. >>> #2 0x00007f1b4b260198 in slapi_wait_condvar () from >>> /usr/lib64/dirsrv/libslapd.so.0 >>> No symbol table info available. >>> #3 0x00007f1b3cf5deed in roles_cache_wait_on_change () from >>> /usr/lib64/dirsrv/plugins/libroles-plugin.so >>> No symbol table info available. >>> #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so >>> No symbol table info available. >>> #5 0x00007f1b48dc1dc5 in start_thread () from >>> /lib64/libpthread.so.0 >>> No symbol table info available. >>> #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 >>> No symbol table info available. >>> >>> Thread 6 (Thread 0x7f1b270a1700 (LWP 105164)): >>> #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 >>> () from >>> /lib64/libpthread.so.0 >>> No symbol table info available. >>> #1 0x00007f1b4941b050 in PR_WaitCondVar () from >>> /lib64/libnspr4.so >>> No symbol table info available. >>> #2 0x00007f1b4b260198 in slapi_wait_condvar () from >>> /usr/lib64/dirsrv/libslapd.so.0 >>> No symbol table info available. >>> #3 0x00007f1b3cf5deed in roles_cache_wait_on_change () from >>> /usr/lib64/dirsrv/plugins/libroles-plugin.so >>> No symbol table info available. >>> #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so >>> No symbol table info available. >>> #5 0x00007f1b48dc1dc5 in start_thread () from >>> /lib64/libpthread.so.0 >>> No symbol table info available. >>> #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 >>> No symbol table info available. >>> >>> Thread 5 (Thread 0x7f1b278a2700 (LWP 105163)): >>> #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 >>> () from >>> /lib64/libpthread.so.0 >>> No symbol table info available. >>> #1 0x00007f1b4941b050 in PR_WaitCondVar () from >>> /lib64/libnspr4.so >>> No symbol table info available. >>> #2 0x00007f1b4b260198 in slapi_wait_condvar () from >>> /usr/lib64/dirsrv/libslapd.so.0 >>> No symbol table info available. >>> #3 0x00007f1b415f262e in cos_cache_wait_on_change () from >>> /usr/lib64/dirsrv/plugins/libcos-plugin.so >>> No symbol table info available. >>> #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so >>> No symbol table info available. >>> #5 0x00007f1b48dc1dc5 in start_thread () from >>> /lib64/libpthread.so.0 >>> No symbol table info available. >>> #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 >>> No symbol table info available. >>> >>> Thread 4 (Thread 0x7f1b282ac700 (LWP 105162)): >>> #0 0x00007f1b48ae69b3 in select () from /lib64/libc.so.6 >>> No symbol table info available. >>> #1 0x00007f1b4b2710e9 in DS_Sleep () from >>> /usr/lib64/dirsrv/libslapd.so.0 >>> No symbol table info available. >>> #2 0x00007f1b3e8ba707 in perf_threadmain () from >>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>> No symbol table info available. >>> #3 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so >>> No symbol table info available. >>> #4 0x00007f1b48dc1dc5 in start_thread () from >>> /lib64/libpthread.so.0 >>> No symbol table info available. >>> #5 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 >>> No symbol table info available. >>> >>> Thread 3 (Thread 0x7f1b28aad700 (LWP 105161)): >>> #0 0x00007f1b48ae69b3 in select () from /lib64/libc.so.6 >>> No symbol table info available. >>> #1 0x00007f1b4b2710e9 in DS_Sleep () from >>> /usr/lib64/dirsrv/libslapd.so.0 >>> No symbol table info available. >>> #2 0x00007f1b3e8bfb7f in trickle_threadmain () from >>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>> No symbol table info available. >>> #3 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so >>> No symbol table info available. >>> #4 0x00007f1b48dc1dc5 in start_thread () from >>> /lib64/libpthread.so.0 >>> No symbol table info available. >>> #5 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 >>> No symbol table info available. >>> >>> Thread 2 (Thread 0x7f1b292ae700 (LWP 105160)): >>> #0 0x00007f1b48ae69b3 in select () from /lib64/libc.so.6 >>> No symbol table info available. >>> #1 0x00007f1b4344dbad in __os_yield () from >>> /lib64/libdb-5.3.so >>> No symbol table info available. >>> #2 0x00007f1b434492b3 in __memp_sync_int () from >>> /lib64/libdb-5.3.so >>> No symbol table info available. >>> #3 0x00007f1b43459752 in __txn_checkpoint () from >>> /lib64/libdb-5.3.so >>> No symbol table info available. >>> #4 0x00007f1b43459b74 in __txn_checkpoint_pp () from >>> /lib64/libdb-5.3.so >>> No symbol table info available. >>> #5 0x00007f1b3e8c3a87 in checkpoint_threadmain () from >>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>> No symbol table info available. >>> #6 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so >>> No symbol table info available. >>> #7 0x00007f1b48dc1dc5 in start_thread () from >>> /lib64/libpthread.so.0 >>> No symbol table info available. >>> #8 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 >>> No symbol table info available. >>> >>> Thread 1 (Thread 0x7f1b29aaf700 (LWP 105159)): >>> #0 0x00007f1b48ae69b3 in select () from /lib64/libc.so.6 >>> No symbol table info available. >>> #1 0x00007f1b4b2710e9 in DS_Sleep () from >>> /usr/lib64/dirsrv/libslapd.so.0 >>> No symbol table info available. >>> #2 0x00007f1b3e8bf907 in deadlock_threadmain () from >>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>> No symbol table info available. >>> #3 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so >>> No symbol table info available. >>> #4 0x00007f1b48dc1dc5 in start_thread () from >>> /lib64/libpthread.so.0 >>> No symbol table info available. >>> #5 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 >>> No symbol table info available. >>> >>> >>> >>> Thanks, >>> >>> Rakesh >>> >>> >>> On Mon, Sep 5, 2016 at 4:48 PM, thierry bordaz >>> > wrote: >>> >>> >>> >>> On 09/05/2016 12:05 PM, Rakesh Rajasekharan wrote: >>>> Hi Thierry, >>>> >>>> >>>> I was getting the hang issue while running >>>> ipa-client-install simultaneously on few clients.. >>>> However, today, I am not able to replicate that. >>>> >>>> I could not get a gdb . But i will try getting that the >>>> next time I face this issue. >>>> >>>> The CPU does not stay high.. it just momentarily >>>> touches a high value and then drops down to around 2-7% >>>> >>>> One question I have is , is it ok to set it >>>> nsslapd-threadnumber to a very high value . >>>> I have around 4000 clients and with >>>> nsslapd-maxthreadsperconn set to 5..So, can I set >>>> nsslapd-threadnumber to around 25000. >>> >>> Hello, >>> >>> I know some users running in production with several >>> hunderds of threads (>600) and this without problem. >>> >>> I do not recall having suggested to increase that number >>> and for what reason. >>> Usually 30 workers is a good enough value. It can create >>> bootleneck if for some reason each operation is very >>> long to satisfy and exhaust the number of workers. You >>> can monitor the work queue: >>> >>> ldapsearch -D "cn=directory manager" -w xxx -LLL -b >>> "cn=monitor" -s base opsinitiated opscompleted >>> >>> >>> If opscompleted-opsinitiated remains close to >>> threadnumber, then yes it would be valuable to increase it. >>> >>> The computation #client * #async_op_per_client sound an >>> overkill. Even if all clients send at the exact same >>> time all their requests, it is very likely that some >>> common resource (db page, log, allocator...) will >>> serialize them. If you monitor a need to increase the >>> work, you would for example set it to 50, then monitor, >>> then set it to 100, then monitor... until you find a >>> good enough value. >>> Note the increasing the #thread, increases the memory >>> footprint that will reduce the efficiency of file system >>> cache and can increase the response time. >>> >>> >>> best regards >>> thierry >>> >>>> >>>> Thanks >>>> >>>> On Mon, Sep 5, 2016 at 1:03 PM, thierry bordaz >>>> > wrote: >>>> >>>> >>>> Hi Rakesh, >>>> >>>> Were you able to get a pstack or full stack with >>>> gdb >>>> (http://www.port389.org/docs/389ds/FAQ/faq.html#debugging-crashes) >>>> when the server hangs ? >>>> >>>> If it happens with 500 threads as well as with 30, >>>> using 30 threads is a better choice to debug this >>>> issue. >>>> I will try to reproduce using 150 parallel 'ipa >>>> user-find p-testipa' commands >>>> >>>> Something I am unsure is if the CPU consumption >>>> stays high (you mentioned 340% CPU usage) as long >>>> as the hang happens or if after a suddent shot up >>>> to 340% (that marks the beginning of the hang) it >>>> drops and stay hanging ? >>>> >>>> thanks >>>> thierry >>>> >>>> On 09/04/2016 08:40 PM, Rakesh Rajasekharan wrote: >>>>> starce on the slapd process actually had this in >>>>> the output.. >>>>> FUTEX_WAIT_PRIVATE >>>>> >>>>> and checking for the number of threads slapd had.. >>>>> there were 5015 threads >>>>> >>>>> ps -efL|grep slapd|wc -l >>>>> 5015 >>>>> >>>>> strace on most of the threads gave this output >>>>> >>>>> strace -p 67411 >>>>> Process 67411 attached >>>>> futex(0x7f3f0226b41c, FUTEX_WAIT_PRIVATE, 1, NULL) >>>>> = -1 EAGAIN (Resource temporarily unavailable) >>>>> futex(0x7f3f0226b41c, FUTEX_WAIT_PRIVATE, 2, >>>>> NULL^CProcess 67411 detached >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> On Sun, Sep 4, 2016 at 5:34 PM, Rakesh >>>>> Rajasekharan >>>> > wrote: >>>>> >>>>> I have again got the issue of IPA hanging.. >>>>> The issue came up when i tried to run >>>>> ipa-client-isntall on 142 clients simultaneously >>>>> >>>>> >>>>> None of the IPA commands are responding, and I >>>>> see this error >>>>> >>>>> ipa user-find p-testipa >>>>> ipa: ERROR: Insufficient access: SASL(-1): >>>>> generic failure: GSSAPI Error: Unspecified GSS >>>>> failure. Minor code may provide more >>>>> information (KDC returned error string: >>>>> PROCESS_TGS) >>>>> >>>>> KRB5_TRACE=/dev/stdout kinit admin >>>>> [41178] 1472984115.233214: Getting initial >>>>> credentials for admin at XYZ.COM >>>>> >>>>> [41178] 1472984115.235257: Sending request >>>>> (167 bytes) to XYZ.COM >>>>> [41178] 1472984115.235419: Initiating TCP >>>>> connection to stream 10.1.3.36:88 >>>>> >>>>> [41178] 1472984115.235685: Sending TCP request >>>>> to stream 10.1.3.36:88 >>>>> [41178] 1472984120.238914: Received answer >>>>> (174 bytes) from stream 10.1.3.36:88 >>>>> >>>>> [41178] 1472984120.238925: Terminating TCP >>>>> connection to stream 10.1.3.36:88 >>>>> >>>>> [41178] 1472984120.238993: Response was from >>>>> master KDC >>>>> [41 >>>>> >>>>> >>>>> Running an ldapsearch to see the db.. does not >>>>> give any results and just hangs there >>>>> >>>>> ldapsearch -x -D 'cn=Directory Manager' -W -s >>>>> one -b 'cn=kerberos,dc=xyz,dc=com' >>>>> Enter LDAP Password: >>>>> >>>>> even an ldapsearch -x does not respond >>>>> At this point, am sure that slapd is the one >>>>> causing issues >>>>> >>>>> Running an strace against the hung slapd >>>>> itself seems to get stuck does not proceed >>>>> after saying "attaching to process" >>>>> >>>>> From some others posts I read Thierry >>>>> suggesting to increase the >>>>> nsslapd-threadnumber value >>>>> >>>>> It was set to 30, I think that might be too low. >>>>> > ... > > [Message clipped] > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From rakesh.rajasekharan at gmail.com Tue Sep 6 09:49:25 2016 From: rakesh.rajasekharan at gmail.com (Rakesh Rajasekharan) Date: Tue, 6 Sep 2016 15:19:25 +0530 Subject: [Freeipa-users] Freeipa 4.2.0 hangs intermittently In-Reply-To: <57CE8CDB.6000809@redhat.com> References: <57BEB0AB.2090506@redhat.com> <57C44AC6.1030402@redhat.com> <57CD1FDC.2010109@redhat.com> <57CD5488.60404@redhat.com> <57CD659B.6050707@redhat.com> <57CD68BD.9030802@redhat.com> <57CE8CDB.6000809@redhat.com> Message-ID: Hi Thierry, sorry I missed the entire top output here it is now Threads: 44 total, 0 running, 44 sleeping, 0 stopped, 0 zombie %Cpu(s): 0.2 us, 0.1 sy, 0.0 ni, 99.7 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st KiB Mem : 65533176 total, 52071076 free, 6230456 used, 7231644 buff/cache KiB Swap: 3999996 total, 3999996 free, 0 used. 58357472 avail Mem PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 4845 dirsrv 20 0 5133196 3.243g 281272 S 0.0 5.2 31:34.51 ns-slapd 4847 dirsrv 20 0 5133196 3.243g 281272 S 0.0 5.2 0:14.05 ns-slapd 4848 dirsrv 20 0 5133196 3.243g 281272 S 0.0 5.2 0:13.24 ns-slapd 4849 dirsrv 20 0 5133196 3.243g 281272 S 0.0 5.2 0:46.96 ns-slapd 4850 dirsrv 20 0 5133196 3.243g 281272 S 0.0 5.2 0:01.66 ns-slapd 4851 dirsrv 20 0 5133196 3.243g 281272 S 0.0 5.2 0:00.00 ns-slapd 4852 dirsrv 20 0 5133196 3.243g 281272 S 0.0 5.2 0:00.00 ns-slapd 4853 dirsrv 20 0 5133196 3.243g 281272 S 0.0 5.2 0:00.00 ns-slapd 4854 dirsrv 20 0 5133196 3.243g 281272 S 0.0 5.2 0:00.00 ns-slapd 4855 dirsrv 20 0 5133196 3.243g 281272 S 0.0 5.2 0:03.02 ns-slapd 4856 dirsrv 20 0 5133196 3.243g 281272 S 0.0 5.2 0:00.28 ns-slapd 4859 dirsrv 20 0 5133196 3.243g 281272 S 0.0 5.2 3:18.67 ns-slapd 4860 dirsrv 20 0 5133196 3.243g 281272 S 0.0 5.2 3:19.82 ns-slapd 4861 dirsrv 20 0 5133196 3.243g 281272 S 0.0 5.2 3:20.23 ns-slapd 4862 dirsrv 20 0 5133196 3.243g 281272 S 0.0 5.2 3:22.65 ns-slapd 4863 dirsrv 20 0 5133196 3.243g 281272 S 0.0 5.2 3:16.64 ns-slapd 4864 dirsrv 20 0 5133196 3.243g 281272 S 0.0 5.2 3:17.39 ns-slapd 4865 dirsrv 20 0 5133196 3.243g 281272 S 0.0 5.2 3:17.80 ns-slapd 4866 dirsrv 20 0 5133196 3.243g 281272 S 0.0 5.2 3:17.57 ns-slapd 4867 dirsrv 20 0 5133196 3.243g 281272 S 0.0 5.2 3:19.44 ns-slapd 4868 dirsrv 20 0 5133196 3.243g 281272 S 0.0 5.2 3:16.28 ns-slapd 4869 dirsrv 20 0 5133196 3.243g 281272 S 0.0 5.2 3:16.17 ns-slapd 4870 dirsrv 20 0 5133196 3.243g 281272 S 0.0 5.2 3:19.07 ns-slapd 4871 dirsrv 20 0 5133196 3.243g 281272 S 0.0 5.2 3:20.04 ns-slapd 4872 dirsrv 20 0 5133196 3.243g 281272 S 0.0 5.2 3:20.06 ns-slapd 4873 dirsrv 20 0 5133196 3.243g 281272 S 0.0 5.2 3:18.41 ns-slapd 4874 dirsrv 20 0 5133196 3.243g 281272 S 0.0 5.2 3:20.25 ns-slapd 4875 dirsrv 20 0 5133196 3.243g 281272 S 0.0 5.2 3:19.58 ns-slapd 4876 dirsrv 20 0 5133196 3.243g 281272 S 0.0 5.2 3:16.54 ns-slapd 4877 dirsrv 20 0 5133196 3.243g 281272 S 0.0 5.2 3:14.37 ns-slapd 4878 dirsrv 20 0 5133196 3.243g 281272 S 0.0 5.2 3:18.80 ns-slapd 4879 dirsrv 20 0 5133196 3.243g 281272 S 0.0 5.2 3:18.00 ns-slapd 4880 dirsrv 20 0 5133196 3.243g 281272 S 0.0 5.2 3:15.73 ns-slapd 4881 dirsrv 20 0 5133196 3.243g 281272 S 0.0 5.2 3:18.07 ns-slapd 4882 dirsrv 20 0 5133196 3.243g 281272 S 0.0 5.2 3:16.45 ns-slapd 4883 dirsrv 20 0 5133196 3.243g 281272 S 0.0 5.2 3:17.93 ns-slapd 4884 dirsrv 20 0 5133196 3.243g 281272 S 0.0 5.2 3:14.65 ns-slapd 4885 dirsrv 20 0 5133196 3.243g 281272 S 0.0 5.2 3:19.38 ns-slapd I am not sure if the machine is NUMA, doing few checks dmesg | grep -i numa [ 0.000000] No NUMA configuration found grep NUMA=y /boot/config-`uname -r` CONFIG_NUMA=y CONFIG_AMD_NUMA=y CONFIG_X86_64_ACPI_NUMA=y CONFIG_ACPI_NUMA=y find /proc -name numa_maps /proc/4845/task/4845/numa_maps /proc/4845/task/4847/numa_maps /proc/4845/task/4848/numa_maps /proc/4845/task/4849/numa_maps /proc/4845/task/4850/numa_maps /proc/4845/task/4851/numa_maps /proc/4845/task/4852/numa_maps /proc/4845/task/4853/numa_maps /proc/4845/task/4854/numa_maps /proc/4845/task/4855/numa_maps /proc/4845/task/4856/numa_maps /proc/4845/task/4859/numa_maps /proc/4845/task/4860/numa_maps /proc/4845/task/4861/numa_maps /proc/4845/task/4862/numa_maps 4845 is the pid of my slapd process Thanks Rakesh On Tue, Sep 6, 2016 at 3:01 PM, thierry bordaz wrote: > Hi, > > The following thread is hanging most of the others threads > > Thread 20 (Thread 0x7f68007f8700 (LWP 4873)): > > #0 0x00007f683d0506d5 in pthread_cond_wait@@GLIBC_2.3.2 () from > /lib64/libpthread.so.0 > #1 0x00007f68375ac2f3 in __db_hybrid_mutex_suspend () from /lib64/ > libdb-5.3.so > #2 0x00007f68375ab640 in __db_tas_mutex_lock () from /lib64/libdb-5.3.so > #3 0x00007f68376c5e1b in __memp_fget () from /lib64/libdb-5.3.so > #4 0x00007f68375c9211 in __bam_search () from /lib64/libdb-5.3.so > #5 0x00007f68375b4256 in __bamc_search () from /lib64/libdb-5.3.so > #6 0x00007f68375b86c4 in __bamc_put () from /lib64/libdb-5.3.so > #7 0x00007f683766f6d5 in __dbc_iput () from /lib64/libdb-5.3.so > #8 0x00007f683766aa0e in __db_put () from /lib64/libdb-5.3.so > #9 0x00007f683767ffa4 in __db_put_pp () from /lib64/libdb-5.3.so > #10 0x00007f6832b5a4cf in idl_new_insert_key () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #11 0x00007f6832b695b5 in addordel_values_sv.isra.3 () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #12 0x00007f6832b6a25a in index_addordel_values_ext_sv () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #13 0x00007f6832b6a694 in index_addordel_values_sv () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #14 0x00007f6832b6a75a in index_addordel_entry () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #15 0x00007f6832b6d5fa in ldbm_back_add () from /usr/lib64/dirsrv/plugins/ > libback-ldbm.so > #16 0x00007f683f46fcca in op_shared_add () from > /usr/lib64/dirsrv/libslapd.so.0 > #17 0x00007f683f471020 in do_add () from /usr/lib64/dirsrv/libslapd.so.0 > #18 0x00007f683f99c303 in connection_threadmain () > #19 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so > #20 0x00007f683d04cdc5 in start_thread () from /lib64/libpthread.so.0 > #21 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 > > This thread is likely not hang on 'db_tas_mutex_lock' as the db_lock does > not show it waiting for something. > Unfortunately it is not shown in the output of top -H, did you get the > complete output ? > Ludwig mentioned a known issue that DS is not scaling well on NUMA arch, > do you know if the machine is NUMA ? > A possibility is that this thread get severely slowed down by a NUMA > remote lock and showing DS has hung. > This is why it would be interesting to know if it consums CPU. > > thanks > theirry > > On 09/06/2016 10:59 AM, Rakesh Rajasekharan wrote: > > Hi Thierry/Ludwig, > > I again hit the hang issue. This is the pstack of the ns-slapd > > So was able to this time get the pstack and db locks > > db locks details > Default locking region information: > 451 Last allocated locker ID > 0x7fffffff Current maximum unused locker ID > 9 Number of lock modes > 800 Initial number of locks allocated > 0 Initial number of lockers allocated > 800 Initial number of lock objects allocated > 10000 Maximum number of locks possible > 10000 Maximum number of lockers possible > 10000 Maximum number of lock objects possible > 800 Current number of locks allocated > 366 Current number of lockers allocated > 800 Current number of lock objects allocated > 160 Number of lock object partitions > 8191 Size of object hash table > 126 Number of current locks > 1005 Maximum number of locks at any one time > 33 Maximum number of locks in any one bucket > 3987 Maximum number of locks stolen by for an empty partition > 75 Maximum number of locks stolen for any one partition > 309 Number of current lockers > 313 Maximum number of lockers at any one time > 121 Number of current lock objects > 676 Maximum number of lock objects at any one time > 3 Maximum number of lock objects in any one bucket > 69 Maximum number of objects stolen by for an empty partition > 5 Maximum number of objects stolen for any one partition > 22M Total number of locks requested (22009642) > 21M Total number of locks released (21992878) > 0 Total number of locks upgraded > 95 Total number of locks downgraded > 3348 Lock requests not available due to conflicts, for which we > waited > 12646 Lock requests not available due to conflicts, for which we did > not wait > 0 Number of deadlocks > 0 Lock timeout value > 0 Number of locks that have timed out > 0 Transaction timeout value > 0 Number of transactions that have timed out > 2MB 560KB Region size > 71719 The number of partition locks that required waiting (0%) > 23105 The maximum number of times any partition lock was waited for > (0%) > 0 The number of object queue operations that required waiting (0%) > 163 The number of locker allocations that required waiting (0%) > 3172 The number of region locks that required waiting (0%) > 5 Maximum hash bucket length > =-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-= > Lock REGINFO information: > Environment Region type > 1 Region ID > /var/lib/dirsrv/slapd-SPRINKLR-COM/db/__db.001 Region name > 0x7f67ef869000 Region address > 0x7f67ef8690a0 Region allocation head > 0x7f67ef8712b0 Region primary address > 0 Region maximum allocation > 0 Region allocated > Region allocations: 1581345 allocations, 0 failures, 1580794 frees, 6 > longest > Allocations by power-of-two sizes: > 1KB 1581329 > 2KB 3 > 4KB 4 > 8KB 3 > 16KB 3 > 32KB 1 > 64KB 0 > 128KB 0 > 256KB 2 > 512KB 0 > 1024KB 1 > REGION_SHARED Region flags > =-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-= > Lock region parameters: > 2 Lock region region mutex [3172/14M 0% !Own] > 16381 locker table size > 8191 object table size > 34128 obj_off > 900216 locker_off > 0 need_dd > =-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-= > Lock conflict matrix: > =-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-= > Locks grouped by lockers: > Locker Mode Count Status ----------------- Object --------------- > 2 dd=308 locks held 1 write locks 0 pid/thread > 4845/140085720139840 flags 10 priority 100 > 2 READ 1 HELD userRoot/id2entry.db handle 0 > 3 dd=307 locks held 0 write locks 0 pid/thread > 4845/140084829472512 flags 0 priority 100 > 4 dd=306 locks held 1 write locks 0 pid/thread > 4845/140085720139840 flags 10 priority 100 > 4 READ 1 HELD ipaca/id2entry.db handle 0 > 5 dd=305 locks held 0 write locks 0 pid/thread > 4845/140084837865216 flags 0 priority 100 > 6 dd=304 locks held 1 write locks 0 pid/thread > 4845/140085720139840 flags 10 priority 100 > 6 READ 1 HELD ipaca/entryrdn.db handle 0 > 7 dd=303 locks held 0 write locks 0 pid/thread > 4845/140084804294400 flags 0 priority 100 > 8 dd=302 locks held 0 write locks 0 pid/thread > 4845/140084720432896 flags 0 priority 100 > 9 dd=301 locks held 0 write locks 0 pid/thread > 4845/140084837865216 flags 0 priority 100 > a dd=300 locks held 1 write locks 0 pid/thread > 4845/140085720139840 flags 10 priority 100 > a READ 1 HELD ipaca/vlv#allcertspkitomcatindex.db > handle 0 > e dd=299 locks held 1 write locks 0 pid/thread > 4845/140085720139840 flags 10 priority 100 > e READ 1 HELD ipaca/vlv#allnonrevokedcertspkitomcatindex.db > handle 0 > 10 dd=298 locks held 1 write locks 0 pid/thread > 4845/140085720139840 flags 10 priority 100 > 10 READ 1 HELD ipaca/vlv#allrevokedcertspkitomcatindex.db > handle 0 > 11 dd=297 locks held 1 write locks 0 pid/thread > 4845/140085720139840 flags 10 priority 100 > 11 READ 1 HELD ipaca/vlv#allrevokedcertsnotafterpkitomcatindex.db > handle 0 > 14 dd=296 locks held 1 write locks 0 pid/thread > 4845/140085720139840 flags 10 priority 100 > 14 READ 1 HELD ipaca/vlv#allrevokedorrevokedexpiredcertspkitomcatindex.db > handle 0 > 15 dd=295 locks held 1 write locks 0 pid/thread > 4845/140085720139840 flags 10 priority 100 > 15 READ 1 HELD ipaca/vlv#allvalidcertspkitomcatindex.db > handle 0 > 16 dd=294 locks held 1 write locks 0 pid/thread > 4845/140085720139840 flags 10 priority 100 > 16 READ 1 HELD ipaca/vlv#allvalidcertsnotafterpkitomcatindex.db > handle 0 > 17 dd=293 locks held 1 write locks 0 pid/thread > 4845/140085720139840 flags 10 priority 100 > 17 READ 1 HELD ipaca/vlv#allvalidorrevokedcertspkitomcatindex.db > handle 0 > 18 dd=292 locks held 1 write locks 0 pid/thread > 4845/140085720139840 flags 10 priority 100 > 18 READ 1 HELD ipaca/vlv#caallpkitomcatindex.db > handle 0 > 1d dd=291 locks held 1 write locks 0 pid/thread > 4845/140085720139840 flags 10 priority 100 > 1d READ 1 HELD ipaca/vlv#cacompletepkitomcatindex.db > handle 0 > 1e dd=290 locks held 1 write locks 0 pid/thread > 4845/140085720139840 flags 10 priority 100 > 1e READ 1 HELD ipaca/vlv#cacompleteenrollmentpkitomcatindex.db > handle 0 > 20 dd=289 locks held 1 write locks 0 pid/thread > 4845/140085720139840 flags 10 priority 100 > 20 READ 1 HELD ipaca/vlv#cacompleterevocationpkitomcatindex.db > handle 0 > 21 dd=288 locks held 1 write locks 0 pid/thread > 4845/140085720139840 flags 10 priority 100 > 21 READ 1 HELD ipaca/vlv#caenrollmentpkitomcatindex.db > handle 0 > 2b dd=287 locks held 1 write locks 0 pid/thread > 4845/140085720139840 flags 10 priority 100 > 2b READ 1 HELD ipaca/vlv#carevocationpkitomcatindex.db > handle 0 > 2c dd=286 locks held 1 write locks 0 pid/thread > 4845/140085720139840 flags 10 priority 100 > 2c READ 1 HELD changelog/id2entry.db handle 0 > 2d dd=285 locks held 0 write locks 0 pid/thread > 4845/140085719607040 flags 0 priority 100 > 2e dd=284 locks held 1 write locks 0 pid/thread > 4845/140085720139840 flags 10 priority 100 > 2e READ 1 HELD changelog/entryusn.db handle 0 > 2f dd=283 locks held 0 write locks 0 pid/thread > 4845/140085720139840 flags 0 priority 100 > 30 dd=282 locks held 1 write locks 0 pid/thread > 4845/140085720139840 flags 10 priority 100 > 30 READ 1 HELD userRoot/entryusn.db handle 0 > 31 dd=281 locks held 0 write locks 0 pid/thread > 4845/140085720139840 flags 0 priority 100 > 32 dd=280 locks held 1 write locks 0 pid/thread > 4845/140085720139840 flags 10 priority 100 > 32 READ 1 HELD ipaca/entryusn.db handle 0 > 33 dd=279 locks held 0 write locks 0 pid/thread > 4845/140085720139840 flags 0 priority 100 > 34 dd=278 locks held 1 write locks 0 pid/thread > 4845/140085720139840 flags 10 priority 100 > 34 READ 1 HELD userRoot/entryrdn.db handle 0 > 35 dd=277 locks held 0 write locks 0 pid/thread > 4845/140084544186112 flags 0 priority 100 > 36 dd=276 locks held 0 write locks 0 pid/thread > 4845/140084636505856 flags 0 priority 100 > 37 dd=275 locks held 0 write locks 0 pid/thread > 4845/140084854650624 flags 0 priority 100 > 38 dd=274 locks held 1 write locks 0 pid/thread > 4845/140085720139840 flags 10 priority 100 > 38 READ 1 HELD userRoot/objectclass.db handle 0 > 39 dd=273 locks held 0 write locks 0 pid/thread > 4845/140084569364224 flags 0 priority 100 > 3a dd=272 locks held 1 write locks 0 pid/thread > 4845/140085720139840 flags 10 priority 100 > 3a READ 1 HELD userRoot/ancestorid.db handle 0 > 3b dd=271 locks held 0 write locks 0 pid/thread > 4845/140084854650624 flags 0 priority 100 > 3c dd=270 locks held 1 write locks 0 pid/thread > 4845/140085720139840 flags 10 priority 100 > 3c READ 1 HELD changelog/entryrdn.db handle 0 > 3d dd=269 locks held 0 write locks 0 pid/thread > 4845/140085719607040 flags 0 priority 100 > 3e dd=268 locks held 0 write locks 0 pid/thread > 4845/140084602935040 flags 0 priority 100 > 3f dd=267 locks held 0 write locks 0 pid/thread > 4845/140085719607040 flags 0 priority 100 > 40 dd=266 locks held 1 write locks 0 pid/thread > 4845/140085720139840 flags 10 priority 100 > 40 READ 1 HELD changelog/objectclass.db handle 0 > 41 dd=265 locks held 0 write locks 0 pid/thread > 4845/140085719607040 flags 0 priority 100 > 42 dd=264 locks held 0 write locks 0 pid/thread > 4845/140085719607040 flags 0 priority 100 > 43 dd=263 locks held 0 write locks 0 pid/thread > 4845/140084653291264 flags 0 priority 100 > 44 dd=262 locks held 1 write locks 0 pid/thread > 4845/140085720139840 flags 10 priority 100 > 44 READ 1 HELD ipaca/objectclass.db handle 0 > 45 dd=261 locks held 0 write locks 0 pid/thread > 4845/140084577756928 flags 0 priority 100 > 46 dd=260 locks held 0 write locks 0 pid/thread > 4845/140084720432896 flags 0 priority 100 > 47 dd=259 locks held 1 write locks 0 pid/thread > 4845/140085720139840 flags 10 priority 100 > 47 READ 1 HELD changelog/aci.db handle 0 > 48 dd=258 locks held 0 write locks 0 pid/thread > 4845/140085720139840 flags 0 priority 100 > 49 dd=257 locks held 1 write locks 0 pid/thread > 4845/140085720139840 flags 10 priority 100 > 49 READ 1 HELD userRoot/aci.db handle 0 > 4a dd=256 locks held 0 write locks 0 pid/thread > 4845/140085720139840 flags 0 priority 100 > 4b dd=255 locks held 0 write locks 0 pid/thread > 4845/140084535793408 flags 0 priority 100 > 4c dd=254 locks held 1 write locks 0 pid/thread > 4845/140085720139840 flags 10 priority 100 > 4c READ 1 HELD ipaca/aci.db handle 0 > 4d dd=253 locks held 0 write locks 0 pid/thread > 4845/140085720139840 flags 0 priority 100 > 4e dd=252 locks held 1 write locks 0 pid/thread > 4845/140085720139840 flags 10 priority 100 > 4e READ 1 HELD userRoot/parentid.db handle 0 > 4f dd=251 locks held 0 write locks 0 pid/thread > 4845/140085720139840 flags 0 priority 100 > 50 dd=250 locks held 0 write locks 0 pid/thread > 4845/140085719607040 flags 0 priority 100 > 51 dd=249 locks held 1 write locks 0 pid/thread > 4845/140085720139840 flags 10 priority 100 > 51 READ 1 HELD changelog/nsuniqueid.db handle 0 > 52 dd=248 locks held 1 write locks 0 pid/thread > 4845/140085720139840 flags 10 priority 100 > 52 READ 1 HELD changelog/changenumber.db handle 0 > 53 dd=247 locks held 0 write locks 0 pid/thread > 4845/140084602935040 flags 0 priority 100 > 54 dd=246 locks held 0 write locks 0 pid/thread > 4845/140085719607040 flags 0 priority 100 > 55 dd=245 locks held 1 write locks 0 pid/thread > 4845/140085719607040 flags 10 priority 100 > 55 READ 1 HELD changelog/targetuniqueid.db handle > 0 > 56 dd=244 locks held 1 write locks 0 pid/thread > 4845/140085719607040 flags 10 priority 100 > 56 READ 1 HELD changelog/parentid.db handle 0 > 57 dd=243 locks held 1 write locks 0 pid/thread > 4845/140085719607040 flags 10 priority 100 > 57 READ 1 HELD changelog/ancestorid.db handle 0 > 58 dd=242 locks held 1 write locks 0 pid/thread > 4845/140085719607040 flags 10 priority 100 > 58 READ 1 HELD changelog/numsubordinates.db > handle 0 > 59 dd=241 locks held 0 write locks 0 pid/thread > 4845/140085719607040 flags 0 priority 100 > 5a dd=240 locks held 0 write locks 0 pid/thread > 4845/140084544186112 flags 0 priority 100 > 5b dd=239 locks held 1 write locks 0 pid/thread > 4845/140084577756928 flags 10 priority 100 > 5b READ 1 HELD userRoot/krbPrincipalName.db > handle 0 > 5c dd=238 locks held 0 write locks 0 pid/thread > 4845/140084636505856 flags 0 priority 100 > 5d dd=237 locks held 0 write locks 0 pid/thread > 4845/140084695254784 flags 0 priority 100 > 5e dd=236 locks held 1 write locks 0 pid/thread > 4845/140084560971520 flags 10 priority 100 > 5e READ 1 HELD userRoot/ipakrbprincipalalias.db > handle 0 > 5f dd=235 locks held 0 write locks 0 pid/thread > 4845/140084703647488 flags 0 priority 100 > 60 dd=234 locks held 0 write locks 0 pid/thread > 4845/140084653291264 flags 0 priority 100 > 61 dd=233 locks held 0 write locks 0 pid/thread > 4845/140084577756928 flags 0 priority 100 > 62 dd=232 locks held 0 write locks 0 pid/thread > 4845/140084686862080 flags 0 priority 100 > 63 dd=231 locks held 1 write locks 0 pid/thread > 4845/140084611327744 flags 10 priority 100 > 63 READ 1 HELD changelog/seeAlso.db handle 0 > 64 dd=230 locks held 0 write locks 0 pid/thread > 4845/140084686862080 flags 0 priority 100 > 65 dd=229 locks held 1 write locks 0 pid/thread > 4845/140084611327744 flags 10 priority 100 > 65 READ 1 HELD userRoot/seeAlso.db handle 0 > 66 dd=228 locks held 0 write locks 0 pid/thread > 4845/140084636505856 flags 0 priority 100 > 67 dd=227 locks held 1 write locks 0 pid/thread > 4845/140084611327744 flags 10 priority 100 > 67 READ 1 HELD ipaca/seeAlso.db handle 0 > 68 dd=226 locks held 0 write locks 0 pid/thread > 4845/140084686862080 flags 0 priority 100 > 69 dd=225 locks held 1 write locks 0 pid/thread > 4845/140084678469376 flags 10 priority 100 > 69 READ 1 HELD ipaca/parentid.db handle 0 > 6a dd=224 locks held 0 write locks 0 pid/thread > 4845/140084678469376 flags 0 priority 100 > 6b dd=223 locks held 0 write locks 0 pid/thread > 4845/140084829472512 flags 0 priority 100 > 6c dd=222 locks held 0 write locks 0 pid/thread > 4845/140084653291264 flags 0 priority 100 > 6d dd=221 locks held 0 write locks 0 pid/thread > 4845/140084569364224 flags 0 priority 100 > 6e dd=220 locks held 0 write locks 0 pid/thread > 4845/140084636505856 flags 0 priority 100 > 6e READ 1 WAIT userRoot/objectclass.db page 18 > 6f dd=219 locks held 0 write locks 0 pid/thread > 4845/140084544186112 flags 0 priority 100 > 70 dd=218 locks held 0 write locks 0 pid/thread > 4845/140084569364224 flags 0 priority 100 > 71 dd=217 locks held 0 write locks 0 pid/thread > 4845/140084619720448 flags 0 priority 100 > 72 dd=216 locks held 0 write locks 0 pid/thread > 4845/140084552578816 flags 0 priority 100 > 73 dd=215 locks held 0 write locks 0 pid/thread > 4845/140084829472512 flags 0 priority 100 > 74 dd=214 locks held 0 write locks 0 pid/thread > 4845/140084628113152 flags 0 priority 100 > 75 dd=213 locks held 0 write locks 0 pid/thread > 4845/140084686862080 flags 0 priority 100 > 76 dd=212 locks held 1 write locks 0 pid/thread > 4845/140084560971520 flags 10 priority 100 > 76 READ 1 HELD userRoot/ipauniqueid.db handle 0 > 77 dd=211 locks held 0 write locks 0 pid/thread > 4845/140084661683968 flags 0 priority 100 > 78 dd=210 locks held 0 write locks 0 pid/thread > 4845/140084846257920 flags 0 priority 100 > 79 dd=209 locks held 0 write locks 0 pid/thread > 4845/140084628113152 flags 0 priority 100 > 7a dd=208 locks held 1 write locks 0 pid/thread > 4845/140084560971520 flags 10 priority 100 > 7a READ 1 HELD userRoot/nsuniqueid.db handle 0 > 7b dd=207 locks held 1 write locks 0 pid/thread > 4845/140084560971520 flags 10 priority 100 > 7b READ 1 HELD userRoot/cn.db handle 0 > 7c dd=206 locks held 1 write locks 0 pid/thread > 4845/140084560971520 flags 10 priority 100 > 7c READ 1 HELD userRoot/fqdn.db handle 0 > 7d dd=205 locks held 1 write locks 0 pid/thread > 4845/140084560971520 flags 10 priority 100 > 7d READ 1 HELD userRoot/managedby.db handle 0 > 7e dd=204 locks held 1 write locks 0 pid/thread > 4845/140084560971520 flags 10 priority 100 > 7e READ 1 HELD userRoot/numsubordinates.db handle > 0 > 7f dd=203 locks held 1 write locks 0 pid/thread > 4845/140084678469376 flags 10 priority 100 > 7f READ 1 HELD userRoot/member.db handle 0 > 80 dd=202 locks held 0 write locks 0 pid/thread > 4845/140084712040192 flags 0 priority 100 > 81 dd=201 locks held 1 write locks 0 pid/thread > 4845/140084678469376 flags 10 priority 100 > 81 READ 1 HELD userRoot/memberUser.db handle 0 > 82 dd=200 locks held 0 write locks 0 pid/thread > 4845/140084577756928 flags 0 priority 100 > 83 dd=199 locks held 1 write locks 0 pid/thread > 4845/140084678469376 flags 10 priority 100 > 83 READ 1 HELD userRoot/memberHost.db handle 0 > 84 dd=198 locks held 0 write locks 0 pid/thread > 4845/140084712040192 flags 0 priority 100 > 85 dd=197 locks held 0 write locks 0 pid/thread > 4845/140084577756928 flags 0 priority 100 > 86 dd=196 locks held 0 write locks 0 pid/thread > 4845/140084670076672 flags 0 priority 100 > 87 dd=195 locks held 0 write locks 0 pid/thread > 4845/140084577756928 flags 0 priority 100 > 88 dd=194 locks held 0 write locks 0 pid/thread > 4845/140084846257920 flags 0 priority 100 > 89 dd=193 locks held 0 write locks 0 pid/thread > 4845/140084552578816 flags 0 priority 100 > 8a dd=192 locks held 0 write locks 0 pid/thread > 4845/140084703647488 flags 0 priority 100 > 8b dd=191 locks held 0 write locks 0 pid/thread > 4845/140084544186112 flags 0 priority 100 > 8c dd=190 locks held 0 write locks 0 pid/thread > 4845/140084846257920 flags 0 priority 100 > 8d dd=189 locks held 0 write locks 0 pid/thread > 4845/140084661683968 flags 0 priority 100 > 8e dd=188 locks held 1 write locks 0 pid/thread > 4845/140085067675392 flags 10 priority 100 > 8e READ 1 HELD userRoot/macAddress.db handle 0 > 8f dd=187 locks held 0 write locks 0 pid/thread > 4845/140085067675392 flags 0 priority 100 > 90 dd=186 locks held 0 write locks 0 pid/thread > 4845/140084703647488 flags 0 priority 100 > 91 dd=185 locks held 0 write locks 0 pid/thread > 4845/140084552578816 flags 0 priority 100 > 92 dd=184 locks held 0 write locks 0 pid/thread > 4845/140084560971520 flags 0 priority 100 > 92 READ 1 WAIT userRoot/objectclass.db page 9 > 93 dd=183 locks held 0 write locks 0 pid/thread > 4845/140084577756928 flags 0 priority 100 > 94 dd=182 locks held 0 write locks 0 pid/thread > 4845/140084821079808 flags 0 priority 100 > 94 READ 1 WAIT userRoot/objectclass.db page 18 > 95 dd=181 locks held 0 write locks 0 pid/thread > 4845/140084544186112 flags 0 priority 100 > 96 dd=180 locks held 0 write locks 0 pid/thread > 4845/140084804294400 flags 0 priority 100 > 97 dd=179 locks held 1 write locks 0 pid/thread > 4845/140084712040192 flags 10 priority 100 > 97 READ 1 HELD userRoot/uid.db handle 0 > 98 dd=178 locks held 0 write locks 0 pid/thread > 4845/140084552578816 flags 0 priority 100 > 99 dd=177 locks held 0 write locks 0 pid/thread > 4845/140084577756928 flags 0 priority 100 > 9a dd=176 locks held 0 write locks 0 pid/thread > 4845/140084602935040 flags 0 priority 100 > 9b dd=175 locks held 1 write locks 0 pid/thread > 4845/140085076068096 flags 10 priority 100 > 9b READ 1 HELD userRoot/uniquemember.db handle 0 > 9c dd=174 locks held 1 write locks 0 pid/thread > 4845/140085076068096 flags 10 priority 100 > 9c READ 1 HELD userRoot/owner.db handle 0 > 9d dd=173 locks held 1 write locks 0 pid/thread > 4845/140085076068096 flags 10 priority 100 > 9d READ 1 HELD userRoot/ipaallowedtarget.db > handle 0 > 9e dd=172 locks held 1 write locks 0 pid/thread > 4845/140085076068096 flags 10 priority 100 > 9e READ 1 HELD userRoot/ipaMemberCertProfile.db > handle 0 > 9f dd=171 locks held 1 write locks 0 pid/thread > 4845/140085076068096 flags 10 priority 100 > 9f READ 1 HELD userRoot/ipaassignedidview.db > handle 0 > a0 dd=170 locks held 1 write locks 0 pid/thread > 4845/140085076068096 flags 10 priority 100 > a0 READ 1 HELD userRoot/secretary.db handle 0 > a1 dd=169 locks held 1 write locks 0 pid/thread > 4845/140085076068096 flags 10 priority 100 > a1 READ 1 HELD userRoot/memberdenycmd.db handle 0 > a2 dd=168 locks held 1 write locks 0 pid/thread > 4845/140085076068096 flags 10 priority 100 > a2 READ 1 HELD userRoot/memberallowcmd.db handle 0 > a3 dd=167 locks held 1 write locks 0 pid/thread > 4845/140085076068096 flags 10 priority 100 > a3 READ 1 HELD userRoot/manager.db handle 0 > a4 dd=166 locks held 1 write locks 0 pid/thread > 4845/140085076068096 flags 10 priority 100 > a4 READ 1 HELD userRoot/ipasudorunas.db handle 0 > a5 dd=165 locks held 1 write locks 0 pid/thread > 4845/140085076068096 flags 10 priority 100 > a5 READ 1 HELD userRoot/ipaMemberCa.db handle 0 > a6 dd=164 locks held 1 write locks 0 pid/thread > 4845/140085076068096 flags 10 priority 100 > a6 READ 1 HELD userRoot/ipasudorunasgroup.db > handle 0 > a7 dd=163 locks held 1 write locks 0 pid/thread > 4845/140085076068096 flags 10 priority 100 > a7 READ 1 HELD userRoot/sourcehost.db handle 0 > a8 dd=162 locks held 1 write locks 0 pid/thread > 4845/140085076068096 flags 10 priority 100 > a8 READ 1 HELD userRoot/memberservice.db handle 0 > a9 dd=161 locks held 1 write locks 0 pid/thread > 4845/140085076068096 flags 10 priority 100 > a9 READ 1 HELD userRoot/ipatokenradiusconfiglink.db > handle 0 > aa dd=160 locks held 0 write locks 0 pid/thread > 4845/140084703647488 flags 0 priority 100 > ab dd=159 locks held 0 write locks 0 pid/thread > 4845/140084821079808 flags 0 priority 100 > ac dd=158 locks held 0 write locks 0 pid/thread > 4845/140084628113152 flags 0 priority 100 > ad dd=157 locks held 0 write locks 0 pid/thread > 4845/140084829472512 flags 0 priority 100 > ae dd=156 locks held 0 write locks 0 pid/thread > 4845/140084829472512 flags 0 priority 100 > af dd=155 locks held 0 write locks 0 pid/thread > 4845/140084854650624 flags 0 priority 100 > b0 dd=154 locks held 0 write locks 0 pid/thread > 4845/140084854650624 flags 0 priority 100 > b1 dd=153 locks held 0 write locks 0 pid/thread > 4845/140084837865216 flags 0 priority 100 > b2 dd=152 locks held 0 write locks 0 pid/thread > 4845/140084804294400 flags 0 priority 100 > b3 dd=151 locks held 0 write locks 0 pid/thread > 4845/140084812687104 flags 0 priority 100 > b4 dd=150 locks held 0 write locks 0 pid/thread > 4845/140084804294400 flags 0 priority 100 > b5 dd=149 locks held 0 write locks 0 pid/thread > 4845/140084837865216 flags 0 priority 100 > b6 dd=148 locks held 0 write locks 0 pid/thread > 4845/140084569364224 flags 0 priority 100 > b7 dd=147 locks held 0 write locks 0 pid/thread > 4845/140084829472512 flags 0 priority 100 > b8 dd=146 locks held 0 write locks 0 pid/thread > 4845/140084586149632 flags 0 priority 100 > b9 dd=145 locks held 0 write locks 0 pid/thread > 4845/140084653291264 flags 0 priority 100 > ba dd=144 locks held 0 write locks 0 pid/thread > 4845/140084636505856 flags 0 priority 100 > bb dd=143 locks held 0 write locks 0 pid/thread > 4845/140084804294400 flags 0 priority 100 > bc dd=142 locks held 0 write locks 0 pid/thread > 4845/140084804294400 flags 0 priority 100 > bd dd=141 locks held 0 write locks 0 pid/thread > 4845/140084695254784 flags 0 priority 100 > be dd=140 locks held 0 write locks 0 pid/thread > 4845/140084670076672 flags 0 priority 100 > bf dd=139 locks held 0 write locks 0 pid/thread > 4845/140084628113152 flags 0 priority 100 > c0 dd=138 locks held 0 write locks 0 pid/thread > 4845/140084661683968 flags 0 priority 100 > c1 dd=137 locks held 0 write locks 0 pid/thread > 4845/140084712040192 flags 0 priority 100 > c2 dd=136 locks held 0 write locks 0 pid/thread > 4845/140084661683968 flags 0 priority 100 > c3 dd=135 locks held 0 write locks 0 pid/thread > 4845/140084577756928 flags 0 priority 100 > c4 dd=134 locks held 0 write locks 0 pid/thread > 4845/140084854650624 flags 0 priority 100 > c5 dd=133 locks held 1 write locks 0 pid/thread > 4845/140084644898560 flags 10 priority 100 > c5 READ 1 HELD userRoot/uidnumber.db handle 0 > c6 dd=132 locks held 0 write locks 0 pid/thread > 4845/140084594542336 flags 0 priority 100 > c8 dd=131 locks held 1 write locks 0 pid/thread > 4845/140084678469376 flags 10 priority 100 > c8 READ 1 HELD ipaca/certstatus.db handle 0 > c9 dd=130 locks held 0 write locks 0 pid/thread > 4845/140084720432896 flags 0 priority 100 > ca dd=129 locks held 0 write locks 0 pid/thread > 4845/140084619720448 flags 0 priority 100 > cb dd=128 locks held 0 write locks 0 pid/thread > 4845/140084821079808 flags 0 priority 100 > cc dd=127 locks held 0 write locks 0 pid/thread > 4845/140084594542336 flags 0 priority 100 > cd dd=126 locks held 0 write locks 0 pid/thread > 4845/140084560971520 flags 0 priority 100 > ce dd=125 locks held 1 write locks 0 pid/thread > 4845/140084837865216 flags 10 priority 100 > ce READ 1 HELD userRoot/gidnumber.db handle 0 > cf dd=124 locks held 0 write locks 0 pid/thread > 4845/140084720432896 flags 0 priority 100 > d0 dd=123 locks held 0 write locks 0 pid/thread > 4845/140084569364224 flags 0 priority 100 > d1 dd=122 locks held 1 write locks 0 pid/thread > 4845/140084678469376 flags 10 priority 100 > d1 READ 1 HELD userRoot/nscpEntryDN.db handle 0 > d2 dd=121 locks held 0 write locks 0 pid/thread > 4845/140084695254784 flags 0 priority 100 > d3 dd=120 locks held 0 write locks 0 pid/thread > 4845/140084569364224 flags 0 priority 100 > d4 dd=119 locks held 0 write locks 0 pid/thread > 4845/140084653291264 flags 0 priority 100 > d5 dd=118 locks held 0 write locks 0 pid/thread > 4845/140084829472512 flags 0 priority 100 > d6 dd=117 locks held 0 write locks 0 pid/thread > 4845/140084661683968 flags 0 priority 100 > d7 dd=116 locks held 0 write locks 0 pid/thread > 4845/140084804294400 flags 0 priority 100 > d8 dd=115 locks held 1 write locks 0 pid/thread > 4845/140084821079808 flags 10 priority 100 > d8 READ 1 HELD ipaca/description.db handle 0 > d9 dd=114 locks held 0 write locks 0 pid/thread > 4845/140084661683968 flags 0 priority 100 > da dd=113 locks held 0 write locks 0 pid/thread > 4845/140084837865216 flags 0 priority 100 > db dd=112 locks held 1 write locks 0 pid/thread > 4845/140084678469376 flags 10 priority 100 > db READ 1 HELD ipaca/cn.db handle 0 > dc dd=111 locks held 0 write locks 0 pid/thread > 4845/140084577756928 flags 0 priority 100 > dd dd=110 locks held 1 write locks 0 pid/thread > 4845/140084678469376 flags 10 priority 100 > dd READ 1 HELD ipaca/ancestorid.db handle 0 > de dd=109 locks held 0 write locks 0 pid/thread > 4845/140084577756928 flags 0 priority 100 > df dd=108 locks held 0 write locks 0 pid/thread > 4845/140084703647488 flags 0 priority 100 > e0 dd=107 locks held 0 write locks 0 pid/thread > 4845/140084569364224 flags 0 priority 100 > e1 dd=106 locks held 0 write locks 0 pid/thread > 4845/140084569364224 flags 0 priority 100 > e2 dd=105 locks held 0 write locks 0 pid/thread > 4845/140084821079808 flags 0 priority 100 > e3 dd=104 locks held 1 write locks 0 pid/thread > 4845/140084686862080 flags 10 priority 100 > e3 READ 1 HELD ipaca/serialno.db handle 0 > e4 dd=103 locks held 0 write locks 0 pid/thread > 4845/140084837865216 flags 0 priority 100 > e5 dd=102 locks held 1 write locks 0 pid/thread > 4845/140084695254784 flags 10 priority 100 > e5 READ 1 HELD ipaca/revInfo.db handle 0 > e6 dd=101 locks held 1 write locks 0 pid/thread > 4845/140084695254784 flags 10 priority 100 > e6 READ 1 HELD ipaca/revokedby.db handle 0 > e7 dd=100 locks held 1 write locks 0 pid/thread > 4845/140084695254784 flags 10 priority 100 > e7 READ 1 HELD ipaca/revokedOn.db handle 0 > e8 dd=99 locks held 0 write locks 0 pid/thread > 4845/140084829472512 flags 0 priority 100 > e9 dd=98 locks held 0 write locks 0 pid/thread > 4845/140084846257920 flags 0 priority 100 > ea dd=97 locks held 1 write locks 0 pid/thread > 4845/140084703647488 flags 10 priority 100 > ea READ 1 HELD ipaca/nsuniqueid.db handle 0 > eb dd=96 locks held 1 write locks 0 pid/thread > 4845/140084703647488 flags 10 priority 100 > eb READ 1 HELD ipaca/requestid.db handle 0 > ec dd=95 locks held 1 write locks 0 pid/thread > 4845/140084703647488 flags 10 priority 100 > ec READ 1 HELD ipaca/requeststate.db handle 0 > ed dd=94 locks held 1 write locks 0 pid/thread > 4845/140084703647488 flags 10 priority 100 > ed READ 1 HELD ipaca/dateOfCreate.db handle 0 > ee dd=93 locks held 1 write locks 0 pid/thread > 4845/140084703647488 flags 10 priority 100 > ee READ 1 HELD ipaca/requesttype.db handle 0 > ef dd=92 locks held 1 write locks 0 pid/thread > 4845/140084703647488 flags 10 priority 100 > ef READ 1 HELD ipaca/numsubordinates.db handle 0 > f0 dd=91 locks held 1 write locks 0 pid/thread > 4845/140084611327744 flags 10 priority 100 > f0 READ 1 HELD userRoot/userCertificate.db handle > 0 > f1 dd=90 locks held 0 write locks 0 pid/thread > 4845/140084821079808 flags 0 priority 100 > f2 dd=89 locks held 0 write locks 0 pid/thread > 4845/140084569364224 flags 0 priority 100 > f3 dd=88 locks held 0 write locks 0 pid/thread > 4845/140084602935040 flags 0 priority 100 > f4 dd=87 locks held 0 write locks 0 pid/thread > 4845/140084720432896 flags 0 priority 100 > f5 dd=86 locks held 0 write locks 0 pid/thread > 4845/140084628113152 flags 0 priority 100 > f6 dd=85 locks held 0 write locks 0 pid/thread > 4845/140084703647488 flags 0 priority 100 > f7 dd=84 locks held 0 write locks 0 pid/thread > 4845/140084611327744 flags 0 priority 100 > f8 dd=83 locks held 0 write locks 0 pid/thread > 4845/140084661683968 flags 0 priority 100 > f9 dd=82 locks held 0 write locks 0 pid/thread > 4845/140084837865216 flags 0 priority 100 > fa dd=81 locks held 0 write locks 0 pid/thread > 4845/140084544186112 flags 0 priority 100 > fb dd=80 locks held 0 write locks 0 pid/thread > 4845/140084703647488 flags 0 priority 100 > fc dd=79 locks held 0 write locks 0 pid/thread > 4845/140084544186112 flags 0 priority 100 > fd dd=78 locks held 0 write locks 0 pid/thread > 4845/140084535793408 flags 0 priority 100 > fe dd=77 locks held 0 write locks 0 pid/thread > 4845/140084586149632 flags 0 priority 100 > ff dd=76 locks held 0 write locks 0 pid/thread > 4845/140084569364224 flags 0 priority 100 > 100 dd=75 locks held 0 write locks 0 pid/thread > 4845/140084804294400 flags 0 priority 100 > 101 dd=74 locks held 0 write locks 0 pid/thread > 4845/140084703647488 flags 0 priority 100 > 102 dd=73 locks held 0 write locks 0 pid/thread > 4845/140084703647488 flags 0 priority 100 > 103 dd=72 locks held 0 write locks 0 pid/thread > 4845/140084821079808 flags 0 priority 100 > 104 dd=71 locks held 0 write locks 0 pid/thread > 4845/140084695254784 flags 0 priority 100 > 105 dd=70 locks held 0 write locks 0 pid/thread > 4845/140084544186112 flags 0 priority 100 > 106 dd=69 locks held 0 write locks 0 pid/thread > 4845/140084854650624 flags 0 priority 100 > 107 dd=68 locks held 0 write locks 0 pid/thread > 4845/140084628113152 flags 0 priority 100 > 108 dd=67 locks held 0 write locks 0 pid/thread > 4845/140084569364224 flags 0 priority 100 > 109 dd=66 locks held 0 write locks 0 pid/thread > 4845/140084594542336 flags 0 priority 100 > 10a dd=65 locks held 0 write locks 0 pid/thread > 4845/140084661683968 flags 0 priority 100 > 10b dd=64 locks held 0 write locks 0 pid/thread > 4845/140084653291264 flags 0 priority 100 > 10c dd=63 locks held 0 write locks 0 pid/thread > 4845/140084695254784 flags 0 priority 100 > 10d dd=62 locks held 0 write locks 0 pid/thread > 4845/140084720432896 flags 0 priority 100 > 10e dd=61 locks held 0 write locks 0 pid/thread > 4845/140084619720448 flags 0 priority 100 > 10f dd=60 locks held 0 write locks 0 pid/thread > 4845/140084720432896 flags 0 priority 100 > 110 dd=59 locks held 0 write locks 0 pid/thread > 4845/140084569364224 flags 0 priority 100 > 111 dd=58 locks held 0 write locks 0 pid/thread > 4845/140084695254784 flags 0 priority 100 > 112 dd=57 locks held 0 write locks 0 pid/thread > 4845/140084653291264 flags 0 priority 100 > 113 dd=56 locks held 0 write locks 0 pid/thread > 4845/140084712040192 flags 0 priority 100 > 114 dd=55 locks held 0 write locks 0 pid/thread > 4845/140084686862080 flags 0 priority 100 > 115 dd=54 locks held 0 write locks 0 pid/thread > 4845/140084577756928 flags 0 priority 100 > 115 READ 1 WAIT userRoot/objectclass.db page 9 > 116 dd=53 locks held 0 write locks 0 pid/thread > 4845/140084837865216 flags 0 priority 100 > 116 READ 1 WAIT userRoot/objectclass.db page 9 > 117 dd=52 locks held 0 write locks 0 pid/thread > 4845/140084577756928 flags 0 priority 100 > 118 dd=51 locks held 0 write locks 0 pid/thread > 4845/140084569364224 flags 0 priority 100 > 119 dd=50 locks held 0 write locks 0 pid/thread > 4845/140084560971520 flags 0 priority 100 > 11a dd=49 locks held 1 write locks 0 pid/thread > 4845/140084695254784 flags 10 priority 100 > 11a READ 1 HELD ipaca/metaInfo.db handle 0 > 11b dd=48 locks held 1 write locks 0 pid/thread > 4845/140084695254784 flags 10 priority 100 > 11b READ 1 HELD ipaca/notbefore.db handle 0 > 11c dd=47 locks held 1 write locks 0 pid/thread > 4845/140084695254784 flags 10 priority 100 > 11c READ 1 HELD ipaca/notafter.db handle 0 > 11d dd=46 locks held 1 write locks 0 pid/thread > 4845/140084695254784 flags 10 priority 100 > 11d READ 1 HELD ipaca/duration.db handle 0 > 11e dd=45 locks held 1 write locks 0 pid/thread > 4845/140084695254784 flags 10 priority 100 > 11e READ 1 HELD ipaca/subjectname.db handle 0 > 11f dd=44 locks held 1 write locks 0 pid/thread > 4845/140084695254784 flags 10 priority 100 > 11f READ 1 HELD ipaca/publicKeyData.db handle 0 > 120 dd=43 locks held 1 write locks 0 pid/thread > 4845/140084695254784 flags 10 priority 100 > 120 READ 1 HELD ipaca/extension.db handle 0 > 121 dd=42 locks held 1 write locks 0 pid/thread > 4845/140084695254784 flags 10 priority 100 > 121 READ 1 HELD ipaca/issuedby.db handle 0 > 122 dd=41 locks held 0 write locks 0 pid/thread > 4845/140084712040192 flags 0 priority 100 > 123 dd=40 locks held 0 write locks 0 pid/thread > 4845/140084812687104 flags 0 priority 100 > 124 dd=39 locks held 0 write locks 0 pid/thread > 4845/140084544186112 flags 0 priority 100 > 125 dd=38 locks held 0 write locks 0 pid/thread > 4845/140084686862080 flags 0 priority 100 > 126 dd=37 locks held 0 write locks 0 pid/thread > 4845/140084552578816 flags 0 priority 100 > 128 dd=36 locks held 0 write locks 0 pid/thread > 4845/140084686862080 flags 0 priority 100 > 12a dd=35 locks held 0 write locks 0 pid/thread > 4845/140084720432896 flags 0 priority 100 > 12b dd=34 locks held 0 write locks 0 pid/thread > 4845/140084552578816 flags 0 priority 100 > 12c dd=33 locks held 0 write locks 0 pid/thread > 4845/140084670076672 flags 0 priority 100 > 12d dd=32 locks held 0 write locks 0 pid/thread > 4845/140084577756928 flags 0 priority 100 > 12e dd=31 locks held 0 write locks 0 pid/thread > 4845/140084712040192 flags 0 priority 100 > 12f dd=30 locks held 0 write locks 0 pid/thread > 4845/140084670076672 flags 0 priority 100 > 130 dd=29 locks held 0 write locks 0 pid/thread > 4845/140084577756928 flags 0 priority 100 > 131 dd=28 locks held 0 write locks 0 pid/thread > 4845/140084560971520 flags 0 priority 100 > 132 dd=27 locks held 0 write locks 0 pid/thread > 4845/140084670076672 flags 0 priority 100 > 133 dd=26 locks held 0 write locks 0 pid/thread > 4845/140084602935040 flags 0 priority 100 > 134 dd=25 locks held 0 write locks 0 pid/thread > 4845/140084552578816 flags 0 priority 100 > 135 dd=24 locks held 0 write locks 0 pid/thread > 4845/140084678469376 flags 0 priority 100 > 136 dd=23 locks held 0 write locks 0 pid/thread > 4845/140084544186112 flags 0 priority 100 > 137 dd=22 locks held 0 write locks 0 pid/thread > 4845/140084577756928 flags 0 priority 100 > 138 dd=21 locks held 0 write locks 0 pid/thread > 4845/140084703647488 flags 0 priority 100 > 139 dd=20 locks held 0 write locks 0 pid/thread > 4845/140084678469376 flags 0 priority 100 > 13b dd=19 locks held 0 write locks 0 pid/thread > 4845/140084812687104 flags 0 priority 100 > 144 dd=18 locks held 0 write locks 0 pid/thread > 4845/140084670076672 flags 0 priority 100 > 156 dd=17 locks held 0 write locks 0 pid/thread > 4845/140084594542336 flags 0 priority 100 > 15a dd=16 locks held 0 write locks 0 pid/thread > 4845/140084577756928 flags 0 priority 100 > 168 dd=15 locks held 0 write locks 0 pid/thread > 4845/140084653291264 flags 0 priority 100 > 16a dd=14 locks held 0 write locks 0 pid/thread > 4845/140084829472512 flags 0 priority 100 > 17d dd=13 locks held 0 write locks 0 pid/thread > 4845/140084829472512 flags 0 priority 100 > 17e dd=12 locks held 0 write locks 0 pid/thread > 4845/140084829472512 flags 0 priority 100 > 17f dd=11 locks held 0 write locks 0 pid/thread > 4845/140084586149632 flags 0 priority 100 > 180 dd=10 locks held 0 write locks 0 pid/thread > 4845/140084594542336 flags 0 priority 100 > 181 dd= 9 locks held 0 write locks 0 pid/thread > 4845/140084577756928 flags 0 priority 100 > 193 dd= 8 locks held 0 write locks 0 pid/thread > 4845/140084812687104 flags 0 priority 100 > 196 dd= 7 locks held 0 write locks 0 pid/thread > 4845/140084695254784 flags 0 priority 100 > 1a1 dd= 6 locks held 0 write locks 0 pid/thread > 4845/140084720432896 flags 0 priority 100 > 1a2 dd= 5 locks held 0 write locks 0 pid/thread > 4845/140084577756928 flags 0 priority 100 > 1a3 dd= 4 locks held 0 write locks 0 pid/thread > 4845/140084812687104 flags 0 priority 100 > 1a4 dd= 3 locks held 0 write locks 0 pid/thread > 4845/140084812687104 flags 0 priority 100 > 1a5 dd= 2 locks held 0 write locks 0 pid/thread > 4845/140084577756928 flags 0 priority 100 > 1a6 dd= 1 locks held 0 write locks 0 pid/thread > 4845/140084594542336 flags 0 priority 100 > 800b3a33 dd= 0 locks held 26 write locks 22 pid/thread > 4845/140084661683968 flags 0 priority 100 > 800b3a33 WRITE 1 HELD userRoot/objectclass.db page 9 > 800b3a33 WRITE 1 HELD userRoot/objectclass.db page 8 > 800b3a33 WRITE 1 HELD userRoot/objectclass.db page 18 > 800b3a33 WRITE 1 HELD userRoot/objectclass.db page 7 > 800b3a33 WRITE 1 HELD userRoot/objectclass.db page 14 > 800b3a33 WRITE 1 HELD userRoot/objectclass.db page 11 > 800b3a33 WRITE 2 HELD userRoot/objectclass.db page 3 > 800b3a33 WRITE 1 HELD userRoot/cn.db page 150 > 800b3a33 WRITE 1 HELD userRoot/cn.db page 89 > 800b3a33 WRITE 1 HELD userRoot/cn.db page 53 > 800b3a33 WRITE 1 HELD userRoot/cn.db page 131 > 800b3a33 WRITE 1 HELD userRoot/cn.db page 117 > 800b3a33 WRITE 1 HELD userRoot/cn.db page 11 > 800b3a33 WRITE 2 HELD userRoot/cn.db page 141 > 800b3a33 WRITE 1 HELD userRoot/cn.db page 41 > 800b3a33 WRITE 1 HELD userRoot/cn.db page 40 > 800b3a33 WRITE 1 HELD userRoot/cn.db page 143 > 800b3a33 WRITE 1 HELD userRoot/cn.db page 30 > 800b3a33 WRITE 2 HELD userRoot/id2entry.db page 8184 > 800b3a33 WRITE 1 HELD userRoot/id2entry.db page 8183 > 800b3a33 WRITE 1 HELD userRoot/id2entry.db page 7779 > 800b3a33 WRITE 1 HELD userRoot/id2entry.db page 0 > 800b3a33 READ 3 HELD userRoot/entryrdn.db page 3 > 800b3a33 READ 1 HELD userRoot/entryrdn.db page 36 > 800b3a33 READ 1 HELD userRoot/entryrdn.db page 4 > 800b3a33 READ 1 HELD userRoot/nsuniqueid.db page 86 > =-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-= > Locks grouped by object: > Locker Mode Count Status ----------------- Object --------------- > a4 READ 1 HELD userRoot/ipasudorunas.db handle 0 > > 9e READ 1 HELD userRoot/ipaMemberCertProfile.db > handle 0 > > 10 READ 1 HELD ipaca/vlv#allrevokedcertspkitomcatindex.db > handle 0 > > 800b3a33 WRITE 2 HELD userRoot/id2entry.db page 8184 > > 800b3a33 WRITE 1 HELD userRoot/id2entry.db page 8183 > > 800b3a33 WRITE 1 HELD userRoot/id2entry.db page 7779 > > a1 READ 1 HELD userRoot/memberdenycmd.db handle 0 > > a READ 1 HELD ipaca/vlv#allcertspkitomcatindex.db > handle 0 > > a7 READ 1 HELD userRoot/sourcehost.db handle 0 > > 57 READ 1 HELD changelog/ancestorid.db handle 0 > > e5 READ 1 HELD ipaca/revInfo.db handle 0 > > e6 READ 1 HELD ipaca/revokedby.db handle 0 > > 11 READ 1 HELD ipaca/vlv#allrevokedcertsnotafterpkitomcatindex.db > handle 0 > > 4e READ 1 HELD userRoot/parentid.db handle 0 > > 4 READ 1 HELD ipaca/id2entry.db handle 0 > > 58 READ 1 HELD changelog/numsubordinates.db > handle 0 > > 67 READ 1 HELD ipaca/seeAlso.db handle 0 > > 2b READ 1 HELD ipaca/vlv#carevocationpkitomcatindex.db > handle 0 > > ec READ 1 HELD ipaca/requeststate.db handle 0 > > 11b READ 1 HELD ipaca/notbefore.db handle 0 > > 7e READ 1 HELD userRoot/numsubordinates.db handle > 0 > > a2 READ 1 HELD userRoot/memberallowcmd.db handle 0 > > 9c READ 1 HELD userRoot/owner.db handle 0 > > 1e READ 1 HELD ipaca/vlv#cacompleteenrollmentpkitomcatindex.db > handle 0 > > 11c READ 1 HELD ipaca/notafter.db handle 0 > > 40 READ 1 HELD changelog/objectclass.db handle 0 > > 2e READ 1 HELD changelog/entryusn.db handle 0 > > 16 READ 1 HELD ipaca/vlv#allvalidcertsnotafterpkitomcatindex.db > handle 0 > > 15 READ 1 HELD ipaca/vlv#allvalidcertspkitomcatindex.db > handle 0 > > a5 READ 1 HELD userRoot/ipaMemberCa.db handle 0 > > 18 READ 1 HELD ipaca/vlv#caallpkitomcatindex.db > handle 0 > > a0 READ 1 HELD userRoot/secretary.db handle 0 > > 81 READ 1 HELD userRoot/memberUser.db handle 0 > > 9f READ 1 HELD userRoot/ipaassignedidview.db > handle 0 > > a3 READ 1 HELD userRoot/manager.db handle 0 > > d8 READ 1 HELD ipaca/description.db handle 0 > > a8 READ 1 HELD userRoot/memberservice.db handle 0 > > 44 READ 1 HELD ipaca/objectclass.db handle 0 > > ed READ 1 HELD ipaca/dateOfCreate.db handle 0 > > 800b3a33 WRITE 1 HELD userRoot/id2entry.db page 0 > > 2 READ 1 HELD userRoot/id2entry.db handle 0 > > 5b READ 1 HELD userRoot/krbPrincipalName.db > handle 0 > > ee READ 1 HELD ipaca/requesttype.db handle 0 > > c8 READ 1 HELD ipaca/certstatus.db handle 0 > > 56 READ 1 HELD changelog/parentid.db handle 0 > > 83 READ 1 HELD userRoot/memberHost.db handle 0 > > 5e READ 1 HELD userRoot/ipakrbprincipalalias.db > handle 0 > > 121 READ 1 HELD ipaca/issuedby.db handle 0 > > 6 READ 1 HELD ipaca/entryrdn.db handle 0 > > e7 READ 1 HELD ipaca/revokedOn.db handle 0 > > 4c READ 1 HELD ipaca/aci.db handle 0 > > 11d READ 1 HELD ipaca/duration.db handle 0 > > 14 READ 1 HELD ipaca/vlv#allrevokedorrevokedexpiredcertspkitomcatindex.db > handle 0 > > 21 READ 1 HELD ipaca/vlv#caenrollmentpkitomcatindex.db > handle 0 > > 55 READ 1 HELD changelog/targetuniqueid.db handle > 0 > > 800b3a33 READ 1 HELD userRoot/entryrdn.db page 36 > > 2c READ 1 HELD changelog/id2entry.db handle 0 > > 800b3a33 READ 1 HELD userRoot/entryrdn.db page 4 > > 34 READ 1 HELD userRoot/entryrdn.db handle 0 > > 800b3a33 READ 3 HELD userRoot/entryrdn.db page 3 > > eb READ 1 HELD ipaca/requestid.db handle 0 > > 49 READ 1 HELD userRoot/aci.db handle 0 > > 76 READ 1 HELD userRoot/ipauniqueid.db handle 0 > > 63 READ 1 HELD changelog/seeAlso.db handle 0 > > 7c READ 1 HELD userRoot/fqdn.db handle 0 > > db READ 1 HELD ipaca/cn.db handle 0 > > ef READ 1 HELD ipaca/numsubordinates.db handle 0 > > 32 READ 1 HELD ipaca/entryusn.db handle 0 > > 1d READ 1 HELD ipaca/vlv#cacompletepkitomcatindex.db > handle 0 > > dd READ 1 HELD ipaca/ancestorid.db handle 0 > > a9 READ 1 HELD userRoot/ipatokenradiusconfiglink.db > handle 0 > > f0 READ 1 HELD userRoot/userCertificate.db handle > 0 > > 11a READ 1 HELD ipaca/metaInfo.db handle 0 > > 7d READ 1 HELD userRoot/managedby.db handle 0 > > ea READ 1 HELD ipaca/nsuniqueid.db handle 0 > > c5 READ 1 HELD userRoot/uidnumber.db handle 0 > > d1 READ 1 HELD userRoot/nscpEntryDN.db handle 0 > > 120 READ 1 HELD ipaca/extension.db handle 0 > > 800b3a33 WRITE 1 HELD userRoot/cn.db page 150 > > 69 READ 1 HELD ipaca/parentid.db handle 0 > > 800b3a33 WRITE 1 HELD userRoot/cn.db page 131 > > 800b3a33 WRITE 1 HELD userRoot/cn.db page 143 > > 800b3a33 WRITE 2 HELD userRoot/cn.db page 141 > > 20 READ 1 HELD ipaca/vlv#cacompleterevocationpkitomcatindex.db > handle 0 > > 8e READ 1 HELD userRoot/macAddress.db handle 0 > > e READ 1 HELD ipaca/vlv#allnonrevokedcertspkitomcatindex.db > handle 0 > > 17 READ 1 HELD ipaca/vlv#allvalidorrevokedcertspkitomcatindex.db > handle 0 > > 800b3a33 WRITE 1 HELD userRoot/cn.db page 30 > > 800b3a33 WRITE 1 HELD userRoot/objectclass.db page 14 > > 800b3a33 WRITE 1 HELD userRoot/objectclass.db page 9 > 116 READ 1 WAIT userRoot/objectclass.db page 9 > 115 READ 1 WAIT userRoot/objectclass.db page 9 > 92 READ 1 WAIT userRoot/objectclass.db page 9 > > 800b3a33 WRITE 1 HELD userRoot/objectclass.db page 8 > > 7b READ 1 HELD userRoot/cn.db handle 0 > > 800b3a33 WRITE 1 HELD userRoot/objectclass.db page 11 > > 800b3a33 WRITE 1 HELD userRoot/objectclass.db page 7 > > 800b3a33 WRITE 1 HELD userRoot/cn.db page 11 > > 38 READ 1 HELD userRoot/objectclass.db handle 0 > > 800b3a33 WRITE 2 HELD userRoot/objectclass.db page 3 > > 800b3a33 WRITE 1 HELD userRoot/cn.db page 53 > > 800b3a33 WRITE 1 HELD userRoot/objectclass.db page 18 > 6e READ 1 WAIT userRoot/objectclass.db page 18 > 94 READ 1 WAIT userRoot/objectclass.db page 18 > > 800b3a33 WRITE 1 HELD userRoot/cn.db page 41 > > 800b3a33 WRITE 1 HELD userRoot/cn.db page 40 > > 800b3a33 WRITE 1 HELD userRoot/cn.db page 89 > > 800b3a33 WRITE 1 HELD userRoot/cn.db page 117 > > 30 READ 1 HELD userRoot/entryusn.db handle 0 > > 3c READ 1 HELD changelog/entryrdn.db handle 0 > > 97 READ 1 HELD userRoot/uid.db handle 0 > > 47 READ 1 HELD changelog/aci.db handle 0 > > e3 READ 1 HELD ipaca/serialno.db handle 0 > > 7a READ 1 HELD userRoot/nsuniqueid.db handle 0 > > 800b3a33 READ 1 HELD userRoot/nsuniqueid.db page 86 > > 51 READ 1 HELD changelog/nsuniqueid.db handle 0 > > 11e READ 1 HELD ipaca/subjectname.db handle 0 > > 7f READ 1 HELD userRoot/member.db handle 0 > > 11f READ 1 HELD ipaca/publicKeyData.db handle 0 > > 9b READ 1 HELD userRoot/uniquemember.db handle 0 > > 52 READ 1 HELD changelog/changenumber.db handle 0 > > ce READ 1 HELD userRoot/gidnumber.db handle 0 > > 65 READ 1 HELD userRoot/seeAlso.db handle 0 > > 9d READ 1 HELD userRoot/ipaallowedtarget.db > handle 0 > > a6 READ 1 HELD userRoot/ipasudorunasgroup.db > handle 0 > > 3a READ 1 HELD userRoot/ancestorid.db handle 0 > > > > > pstack output > > root at ipa-master ]# pstack 4845 > Thread 44 (Thread 0x7f681dd3a700 (LWP 4847)): > #0 0x00007f683cd719b3 in select () from /lib64/libc.so.6 > #1 0x00007f683f4fc0e9 in DS_Sleep () from /usr/lib64/dirsrv/libslapd.so.0 > #2 0x00007f6832b4a907 in deadlock_threadmain () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #3 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so > #4 0x00007f683d04cdc5 in start_thread () from /lib64/libpthread.so.0 > #5 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 > Thread 43 (Thread 0x7f681d539700 (LWP 4848)): > #0 0x00007f683cd719b3 in select () from /lib64/libc.so.6 > #1 0x00007f68376d8bad in __os_yield () from /lib64/libdb-5.3.so > #2 0x00007f68376d42b3 in __memp_sync_int () from /lib64/libdb-5.3.so > #3 0x00007f68376e4752 in __txn_checkpoint () from /lib64/libdb-5.3.so > #4 0x00007f68376e4b74 in __txn_checkpoint_pp () from /lib64/libdb-5.3.so > #5 0x00007f6832b4ea87 in checkpoint_threadmain () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #6 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so > #7 0x00007f683d04cdc5 in start_thread () from /lib64/libpthread.so.0 > #8 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 > Thread 42 (Thread 0x7f681cd38700 (LWP 4849)): > #0 0x00007f683cd719b3 in select () from /lib64/libc.so.6 > #1 0x00007f683f4fc0e9 in DS_Sleep () from /usr/lib64/dirsrv/libslapd.so.0 > #2 0x00007f6832b4ab7f in trickle_threadmain () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #3 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so > #4 0x00007f683d04cdc5 in start_thread () from /lib64/libpthread.so.0 > #5 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 > Thread 41 (Thread 0x7f681c537700 (LWP 4850)): > #0 0x00007f683cd719b3 in select () from /lib64/libc.so.6 > #1 0x00007f683f4fc0e9 in DS_Sleep () from /usr/lib64/dirsrv/libslapd.so.0 > #2 0x00007f6832b45707 in perf_threadmain () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #3 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so > #4 0x00007f683d04cdc5 in start_thread () from /lib64/libpthread.so.0 > #5 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 > Thread 40 (Thread 0x7f681bb2d700 (LWP 4851)): > #0 0x00007f683d0506d5 in pthread_cond_wait@@GLIBC_2.3.2 () from > /lib64/libpthread.so.0 > #1 0x00007f683d6a6050 in PR_WaitCondVar () from /lib64/libnspr4.so > #2 0x00007f683f4eb198 in slapi_wait_condvar () from > /usr/lib64/dirsrv/libslapd.so.0 > #3 0x00007f683587d62e in cos_cache_wait_on_change () from > /usr/lib64/dirsrv/plugins/libcos-plugin.so > #4 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so > #5 0x00007f683d04cdc5 in start_thread () from /lib64/libpthread.so.0 > #6 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 > Thread 39 (Thread 0x7f681b32c700 (LWP 4852)): > #0 0x00007f683d0506d5 in pthread_cond_wait@@GLIBC_2.3.2 () from > /lib64/libpthread.so.0 > #1 0x00007f683d6a6050 in PR_WaitCondVar () from /lib64/libnspr4.so > #2 0x00007f683f4eb198 in slapi_wait_condvar () from > /usr/lib64/dirsrv/libslapd.so.0 > #3 0x00007f68311e8eed in roles_cache_wait_on_change () from > /usr/lib64/dirsrv/plugins/libroles-plugin.so > #4 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so > #5 0x00007f683d04cdc5 in start_thread () from /lib64/libpthread.so.0 > #6 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 > Thread 38 (Thread 0x7f681ab2b700 (LWP 4853)): > #0 0x00007f683d0506d5 in pthread_cond_wait@@GLIBC_2.3.2 () from > /lib64/libpthread.so.0 > #1 0x00007f683d6a6050 in PR_WaitCondVar () from /lib64/libnspr4.so > #2 0x00007f683f4eb198 in slapi_wait_condvar () from > /usr/lib64/dirsrv/libslapd.so.0 > #3 0x00007f68311e8eed in roles_cache_wait_on_change () from > /usr/lib64/dirsrv/plugins/libroles-plugin.so > #4 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so > #5 0x00007f683d04cdc5 in start_thread () from /lib64/libpthread.so.0 > #6 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 > Thread 37 (Thread 0x7f681a32a700 (LWP 4854)): > #0 0x00007f683d0506d5 in pthread_cond_wait@@GLIBC_2.3.2 () from > /lib64/libpthread.so.0 > #1 0x00007f683d6a6050 in PR_WaitCondVar () from /lib64/libnspr4.so > #2 0x00007f683f4eb198 in slapi_wait_condvar () from > /usr/lib64/dirsrv/libslapd.so.0 > #3 0x00007f68311e8eed in roles_cache_wait_on_change () from > /usr/lib64/dirsrv/plugins/libroles-plugin.so > #4 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so > #5 0x00007f683d04cdc5 in start_thread () from /lib64/libpthread.so.0 > #6 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 > Thread 36 (Thread 0x7f6819b29700 (LWP 4855)): > #0 0x00007f683d050a82 in pthread_cond_timedwait@@GLIBC_2.3.2 () from > /lib64/libpthread.so.0 > #1 0x00007f683d6a5b07 in pt_TimedWait () from /lib64/libnspr4.so > #2 0x00007f683d6a5fce in PR_WaitCondVar () from /lib64/libnspr4.so > #3 0x00007f683f9a4d03 in housecleaning () > #4 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so > #5 0x00007f683d04cdc5 in start_thread () from /lib64/libpthread.so.0 > #6 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 > Thread 35 (Thread 0x7f6819328700 (LWP 4856)): > #0 0x00007f683d050a82 in pthread_cond_timedwait@@GLIBC_2.3.2 () from > /lib64/libpthread.so.0 > #1 0x00007f683d6a5b07 in pt_TimedWait () from /lib64/libnspr4.so > #2 0x00007f683d6a5fce in PR_WaitCondVar () from /lib64/libnspr4.so > #3 0x00007f683f4911e8 in eq_loop () from /usr/lib64/dirsrv/libslapd.so.0 > #4 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so > #5 0x00007f683d04cdc5 in start_thread () from /lib64/libpthread.so.0 > #6 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 > Thread 34 (Thread 0x7f680bfff700 (LWP 4859)): > #0 0x00007f683d0506d5 in pthread_cond_wait@@GLIBC_2.3.2 () from > /lib64/libpthread.so.0 > #1 0x00007f683d6a6050 in PR_WaitCondVar () from /lib64/libnspr4.so > #2 0x00007f683f999fee in connection_wait_for_new_work () > #3 0x00007f683f99b719 in connection_threadmain () > #4 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so > #5 0x00007f683d04cdc5 in start_thread () from /lib64/libpthread.so.0 > #6 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 > Thread 33 (Thread 0x7f680b7fe700 (LWP 4860)): > #0 0x00007f683d0506d5 in pthread_cond_wait@@GLIBC_2.3.2 () from > /lib64/libpthread.so.0 > #1 0x00007f683d6a6050 in PR_WaitCondVar () from /lib64/libnspr4.so > #2 0x00007f683f999fee in connection_wait_for_new_work () > #3 0x00007f683f99b719 in connection_threadmain () > #4 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so > #5 0x00007f683d04cdc5 in start_thread () from /lib64/libpthread.so.0 > #6 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 > Thread 32 (Thread 0x7f680affd700 (LWP 4861)): > #0 0x00007f683d0506d5 in pthread_cond_wait@@GLIBC_2.3.2 () from > /lib64/libpthread.so.0 > #1 0x00007f68375ac2f3 in __db_hybrid_mutex_suspend () from /lib64/ > libdb-5.3.so > #2 0x00007f68375ab640 in __db_tas_mutex_lock () from /lib64/libdb-5.3.so > #3 0x00007f6837655d3a in __lock_get_internal () from /lib64/libdb-5.3.so > #4 0x00007f6837656820 in __lock_get () from /lib64/libdb-5.3.so > #5 0x00007f6837682142 in __db_lget () from /lib64/libdb-5.3.so > #6 0x00007f68375c9605 in __bam_search () from /lib64/libdb-5.3.so > #7 0x00007f68375b4256 in __bamc_search () from /lib64/libdb-5.3.so > #8 0x00007f68375b5d0f in __bamc_get () from /lib64/libdb-5.3.so > #9 0x00007f683766eca6 in __dbc_iget () from /lib64/libdb-5.3.so > #10 0x00007f683767db02 in __dbc_get_pp () from /lib64/libdb-5.3.so > #11 0x00007f6832b59680 in idl_new_fetch () from /usr/lib64/dirsrv/plugins/ > libback-ldbm.so > #12 0x00007f6832b67b26 in index_read_ext_allids () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #13 0x00007f6832b52284 in keys2idl () from /usr/lib64/dirsrv/plugins/ > libback-ldbm.so > #14 0x00007f6832b529e3 in ava_candidates.isra.0 () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #15 0x00007f6832b52fd2 in filter_candidates_ext () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #16 0x00007f6832b54096 in list_candidates () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #17 0x00007f6832b52f40 in filter_candidates_ext () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #18 0x00007f6832b54096 in list_candidates () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #19 0x00007f6832b52f40 in filter_candidates_ext () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #20 0x00007f6832b54096 in list_candidates () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #21 0x00007f6832b52f40 in filter_candidates_ext () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #22 0x00007f6832b8e80c in subtree_candidates () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #23 0x00007f6832b8fe7f in ldbm_back_search () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #24 0x00007f683f4bd1d6 in op_shared_search () from > /usr/lib64/dirsrv/libslapd.so.0 > #25 0x00007f683f9ae33e in do_search () > #26 0x00007f683f99c3c3 in connection_threadmain () > #27 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so > #28 0x00007f683d04cdc5 in start_thread () from /lib64/libpthread.so.0 > #29 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 > Thread 31 (Thread 0x7f680a7fc700 (LWP 4862)): > #0 0x00007f683d0506d5 in pthread_cond_wait@@GLIBC_2.3.2 () from > /lib64/libpthread.so.0 > #1 0x00007f683d6a62c3 in PR_EnterMonitor () from /lib64/libnspr4.so > #2 0x00007f6832b4d3e6 in dblayer_txn_begin () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #3 0x00007f6832b89007 in ldbm_back_modify () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #4 0x00007f683f4b6541 in op_shared_modify () from > /usr/lib64/dirsrv/libslapd.so.0 > #5 0x00007f683f4b7004 in modify_internal_pb () from > /usr/lib64/dirsrv/libslapd.so.0 > #6 0x00007f683483bbbb in ipalockout_postop () from > /usr/lib64/dirsrv/plugins/libipa_lockout.so > #7 0x00007f683f4c8848 in plugin_call_func () from > /usr/lib64/dirsrv/libslapd.so.0 > #8 0x00007f683f4c8b13 in plugin_call_plugins () from > /usr/lib64/dirsrv/libslapd.so.0 > #9 0x00007f683f9954d8 in do_bind () > #10 0x00007f683f99c3fd in connection_threadmain () > #11 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so > #12 0x00007f683d04cdc5 in start_thread () from /lib64/libpthread.so.0 > #13 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 > Thread 30 (Thread 0x7f6809ffb700 (LWP 4863)): > #0 0x00007f683d0506d5 in pthread_cond_wait@@GLIBC_2.3.2 () from > /lib64/libpthread.so.0 > #1 0x00007f68375ac2f3 in __db_hybrid_mutex_suspend () from /lib64/ > libdb-5.3.so > #2 0x00007f68375ab640 in __db_tas_mutex_lock () from /lib64/libdb-5.3.so > #3 0x00007f6837655d3a in __lock_get_internal () from /lib64/libdb-5.3.so > #4 0x00007f6837656820 in __lock_get () from /lib64/libdb-5.3.so > #5 0x00007f6837682142 in __db_lget () from /lib64/libdb-5.3.so > #6 0x00007f68375c9605 in __bam_search () from /lib64/libdb-5.3.so > #7 0x00007f68375b4256 in __bamc_search () from /lib64/libdb-5.3.so > #8 0x00007f68375b5d0f in __bamc_get () from /lib64/libdb-5.3.so > #9 0x00007f683766eca6 in __dbc_iget () from /lib64/libdb-5.3.so > #10 0x00007f683767db02 in __dbc_get_pp () from /lib64/libdb-5.3.so > #11 0x00007f6832b59680 in idl_new_fetch () from /usr/lib64/dirsrv/plugins/ > libback-ldbm.so > #12 0x00007f6832b67b26 in index_read_ext_allids () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #13 0x00007f6832b52284 in keys2idl () from /usr/lib64/dirsrv/plugins/ > libback-ldbm.so > #14 0x00007f6832b529e3 in ava_candidates.isra.0 () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #15 0x00007f6832b52fd2 in filter_candidates_ext () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #16 0x00007f6832b54096 in list_candidates () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #17 0x00007f6832b52f40 in filter_candidates_ext () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #18 0x00007f6832b8e80c in subtree_candidates () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #19 0x00007f6832b8fe7f in ldbm_back_search () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #20 0x00007f683f4bd1d6 in op_shared_search () from > /usr/lib64/dirsrv/libslapd.so.0 > #21 0x00007f683f4cd94e in search_internal_callback_pb () from > /usr/lib64/dirsrv/libslapd.so.0 > #22 0x00007f683f4cdbe8 in search_internal_pb () from > /usr/lib64/dirsrv/libslapd.so.0 > #23 0x00007f683f9ac49b in ids_sasl_canon_user () > #24 0x00007f683d46c93b in _sasl_canon_user () from /lib64/libsasl2.so.3 > #25 0x00007f683d46cc4c in _sasl_canon_user_lookup () from > /lib64/libsasl2.so.3 > #26 0x00007f6837374fb9 in gssapi_server_mech_step () from > /usr/lib64/sasl2/libgssapiv2.so > #27 0x00007f683d478b9b in sasl_server_step () from /lib64/libsasl2.so.3 > #28 0x00007f683f9ad83b in ids_sasl_check_bind () > #29 0x00007f683f99561a in do_bind () > #30 0x00007f683f99c3fd in connection_threadmain () > #31 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so > #32 0x00007f683d04cdc5 in start_thread () from /lib64/libpthread.so.0 > #33 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 > Thread 29 (Thread 0x7f68097fa700 (LWP 4864)): > #0 0x00007f683d0506d5 in pthread_cond_wait@@GLIBC_2.3.2 () from > /lib64/libpthread.so.0 > #1 0x00007f683d6a6050 in PR_WaitCondVar () from /lib64/libnspr4.so > #2 0x00007f683f999fee in connection_wait_for_new_work () > #3 0x00007f683f99b719 in connection_threadmain () > #4 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so > #5 0x00007f683d04cdc5 in start_thread () from /lib64/libpthread.so.0 > #6 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 > Thread 28 (Thread 0x7f6808ff9700 (LWP 4865)): > #0 0x00007f683d0506d5 in pthread_cond_wait@@GLIBC_2.3.2 () from > /lib64/libpthread.so.0 > #1 0x00007f683d6a6050 in PR_WaitCondVar () from /lib64/libnspr4.so > #2 0x00007f683f999fee in connection_wait_for_new_work () > #3 0x00007f683f99b719 in connection_threadmain () > #4 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so > #5 0x00007f683d04cdc5 in start_thread () from /lib64/libpthread.so.0 > #6 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 > Thread 27 (Thread 0x7f6803fff700 (LWP 4866)): > #0 0x00007f683d0506d5 in pthread_cond_wait@@GLIBC_2.3.2 () from > /lib64/libpthread.so.0 > #1 0x00007f683d6a6050 in PR_WaitCondVar () from /lib64/libnspr4.so > #2 0x00007f683f999fee in connection_wait_for_new_work () > #3 0x00007f683f99b719 in connection_threadmain () > #4 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so > #5 0x00007f683d04cdc5 in start_thread () from /lib64/libpthread.so.0 > #6 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 > Thread 26 (Thread 0x7f68037fe700 (LWP 4867)): > #0 0x00007f683d0506d5 in pthread_cond_wait@@GLIBC_2.3.2 () from > /lib64/libpthread.so.0 > #1 0x00007f683d6a6050 in PR_WaitCondVar () from /lib64/libnspr4.so > #2 0x00007f683f999fee in connection_wait_for_new_work () > #3 0x00007f683f99b719 in connection_threadmain () > #4 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so > #5 0x00007f683d04cdc5 in start_thread () from /lib64/libpthread.so.0 > #6 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 > Thread 25 (Thread 0x7f6802ffd700 (LWP 4868)): > #0 0x00007f683d0506d5 in pthread_cond_wait@@GLIBC_2.3.2 () from > /lib64/libpthread.so.0 > #1 0x00007f683d6a6050 in PR_WaitCondVar () from /lib64/libnspr4.so > #2 0x00007f683f999fee in connection_wait_for_new_work () > #3 0x00007f683f99b719 in connection_threadmain () > #4 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so > #5 0x00007f683d04cdc5 in start_thread () from /lib64/libpthread.so.0 > #6 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 > Thread 24 (Thread 0x7f68027fc700 (LWP 4869)): > #0 0x00007f683d0506d5 in pthread_cond_wait@@GLIBC_2.3.2 () from > /lib64/libpthread.so.0 > #1 0x00007f68375ac2f3 in __db_hybrid_mutex_suspend () from /lib64/ > libdb-5.3.so > #2 0x00007f68375abb37 in __db_tas_mutex_readlock () from /lib64/ > libdb-5.3.so > #3 0x00007f68376c5d35 in __memp_fget () from /lib64/libdb-5.3.so > #4 0x00007f683766f333 in __dbc_iget () from /lib64/libdb-5.3.so > #5 0x00007f683767db02 in __dbc_get_pp () from /lib64/libdb-5.3.so > #6 0x00007f6832b59822 in idl_new_fetch () from /usr/lib64/dirsrv/plugins/ > libback-ldbm.so > #7 0x00007f6832b67b26 in index_read_ext_allids () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #8 0x00007f6832b52284 in keys2idl () from /usr/lib64/dirsrv/plugins/ > libback-ldbm.so > #9 0x00007f6832b529e3 in ava_candidates.isra.0 () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #10 0x00007f6832b52fd2 in filter_candidates_ext () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #11 0x00007f6832b54096 in list_candidates () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #12 0x00007f6832b52f40 in filter_candidates_ext () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #13 0x00007f6832b54096 in list_candidates () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #14 0x00007f6832b52f40 in filter_candidates_ext () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #15 0x00007f6832b54096 in list_candidates () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #16 0x00007f6832b52f40 in filter_candidates_ext () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #17 0x00007f6832b8e80c in subtree_candidates () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #18 0x00007f6832b8fe7f in ldbm_back_search () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #19 0x00007f683f4bd1d6 in op_shared_search () from > /usr/lib64/dirsrv/libslapd.so.0 > #20 0x00007f683f9ae33e in do_search () > #21 0x00007f683f99c3c3 in connection_threadmain () > #22 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so > #23 0x00007f683d04cdc5 in start_thread () from /lib64/libpthread.so.0 > #24 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 > Thread 23 (Thread 0x7f6801ffb700 (LWP 4870)): > #0 0x00007f683d0506d5 in pthread_cond_wait@@GLIBC_2.3.2 () from > /lib64/libpthread.so.0 > #1 0x00007f683d6a6050 in PR_WaitCondVar () from /lib64/libnspr4.so > #2 0x00007f683f999fee in connection_wait_for_new_work () > #3 0x00007f683f99b719 in connection_threadmain () > #4 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so > #5 0x00007f683d04cdc5 in start_thread () from /lib64/libpthread.so.0 > #6 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 > Thread 22 (Thread 0x7f68017fa700 (LWP 4871)): > #0 0x00007f683d0506d5 in pthread_cond_wait@@GLIBC_2.3.2 () from > /lib64/libpthread.so.0 > #1 0x00007f683d6a6050 in PR_WaitCondVar () from /lib64/libnspr4.so > #2 0x00007f683f999fee in connection_wait_for_new_work () > #3 0x00007f683f99b719 in connection_threadmain () > #4 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so > #5 0x00007f683d04cdc5 in start_thread () from /lib64/libpthread.so.0 > #6 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 > Thread 21 (Thread 0x7f6800ff9700 (LWP 4872)): > #0 0x00007f683d0506d5 in pthread_cond_wait@@GLIBC_2.3.2 () from > /lib64/libpthread.so.0 > #1 0x00007f683d6a6050 in PR_WaitCondVar () from /lib64/libnspr4.so > #2 0x00007f683f999fee in connection_wait_for_new_work () > #3 0x00007f683f99b719 in connection_threadmain () > #4 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so > #5 0x00007f683d04cdc5 in start_thread () from /lib64/libpthread.so.0 > #6 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 > Thread 20 (Thread 0x7f68007f8700 (LWP 4873)): > #0 0x00007f683d0506d5 in pthread_cond_wait@@GLIBC_2.3.2 () from > /lib64/libpthread.so.0 > #1 0x00007f68375ac2f3 in __db_hybrid_mutex_suspend () from /lib64/ > libdb-5.3.so > #2 0x00007f68375ab640 in __db_tas_mutex_lock () from /lib64/libdb-5.3.so > #3 0x00007f68376c5e1b in __memp_fget () from /lib64/libdb-5.3.so > #4 0x00007f68375c9211 in __bam_search () from /lib64/libdb-5.3.so > #5 0x00007f68375b4256 in __bamc_search () from /lib64/libdb-5.3.so > #6 0x00007f68375b86c4 in __bamc_put () from /lib64/libdb-5.3.so > #7 0x00007f683766f6d5 in __dbc_iput () from /lib64/libdb-5.3.so > #8 0x00007f683766aa0e in __db_put () from /lib64/libdb-5.3.so > #9 0x00007f683767ffa4 in __db_put_pp () from /lib64/libdb-5.3.so > #10 0x00007f6832b5a4cf in idl_new_insert_key () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #11 0x00007f6832b695b5 in addordel_values_sv.isra.3 () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #12 0x00007f6832b6a25a in index_addordel_values_ext_sv () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #13 0x00007f6832b6a694 in index_addordel_values_sv () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #14 0x00007f6832b6a75a in index_addordel_entry () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #15 0x00007f6832b6d5fa in ldbm_back_add () from /usr/lib64/dirsrv/plugins/ > libback-ldbm.so > #16 0x00007f683f46fcca in op_shared_add () from > /usr/lib64/dirsrv/libslapd.so.0 > #17 0x00007f683f471020 in do_add () from /usr/lib64/dirsrv/libslapd.so.0 > #18 0x00007f683f99c303 in connection_threadmain () > #19 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so > #20 0x00007f683d04cdc5 in start_thread () from /lib64/libpthread.so.0 > #21 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 > Thread 19 (Thread 0x7f67ffff7700 (LWP 4874)): > #0 0x00007f683d0506d5 in pthread_cond_wait@@GLIBC_2.3.2 () from > /lib64/libpthread.so.0 > #1 0x00007f683d6a62c3 in PR_EnterMonitor () from /lib64/libnspr4.so > #2 0x00007f6832b4d3e6 in dblayer_txn_begin () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #3 0x00007f6832b89007 in ldbm_back_modify () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #4 0x00007f683f4b6541 in op_shared_modify () from > /usr/lib64/dirsrv/libslapd.so.0 > #5 0x00007f683f4b7004 in modify_internal_pb () from > /usr/lib64/dirsrv/libslapd.so.0 > #6 0x00007f6832dd3b60 in ipapwd_apply_mods () from > /usr/lib64/dirsrv/plugins/libipa_pwd_extop.so > #7 0x00007f6832dd78e0 in store_new_keys () from /usr/lib64/dirsrv/plugins/ > libipa_pwd_extop.so > #8 0x00007f6832dd9ad0 in ipapwd_extop () from /usr/lib64/dirsrv/plugins/ > libipa_pwd_extop.so > #9 0x00007f683f4c6ba2 in plugin_call_exop_plugins () from > /usr/lib64/dirsrv/libslapd.so.0 > #10 0x00007f683f9a2efe in do_extended () > #11 0x00007f683f99c41a in connection_threadmain () > #12 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so > #13 0x00007f683d04cdc5 in start_thread () from /lib64/libpthread.so.0 > #14 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 > Thread 18 (Thread 0x7f67ff7f6700 (LWP 4875)): > #0 0x00007f683d0506d5 in pthread_cond_wait@@GLIBC_2.3.2 () from > /lib64/libpthread.so.0 > #1 0x00007f683d6a6050 in PR_WaitCondVar () from /lib64/libnspr4.so > #2 0x00007f683f999fee in connection_wait_for_new_work () > #3 0x00007f683f99b719 in connection_threadmain () > #4 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so > #5 0x00007f683d04cdc5 in start_thread () from /lib64/libpthread.so.0 > #6 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 > Thread 17 (Thread 0x7f67feff5700 (LWP 4876)): > #0 0x00007f683d0506d5 in pthread_cond_wait@@GLIBC_2.3.2 () from > /lib64/libpthread.so.0 > #1 0x00007f68375ac2f3 in __db_hybrid_mutex_suspend () from /lib64/ > libdb-5.3.so > #2 0x00007f68375ab640 in __db_tas_mutex_lock () from /lib64/libdb-5.3.so > #3 0x00007f6837655d3a in __lock_get_internal () from /lib64/libdb-5.3.so > #4 0x00007f6837656820 in __lock_get () from /lib64/libdb-5.3.so > #5 0x00007f6837682142 in __db_lget () from /lib64/libdb-5.3.so > #6 0x00007f68375c9605 in __bam_search () from /lib64/libdb-5.3.so > #7 0x00007f68375b4256 in __bamc_search () from /lib64/libdb-5.3.so > #8 0x00007f68375b5d0f in __bamc_get () from /lib64/libdb-5.3.so > #9 0x00007f683766eca6 in __dbc_iget () from /lib64/libdb-5.3.so > #10 0x00007f683767db02 in __dbc_get_pp () from /lib64/libdb-5.3.so > #11 0x00007f6832b59680 in idl_new_fetch () from /usr/lib64/dirsrv/plugins/ > libback-ldbm.so > #12 0x00007f6832b67b26 in index_read_ext_allids () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #13 0x00007f6832b52284 in keys2idl () from /usr/lib64/dirsrv/plugins/ > libback-ldbm.so > #14 0x00007f6832b529e3 in ava_candidates.isra.0 () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #15 0x00007f6832b52fd2 in filter_candidates_ext () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #16 0x00007f6832b54096 in list_candidates () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #17 0x00007f6832b52f40 in filter_candidates_ext () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #18 0x00007f6832b8e80c in subtree_candidates () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #19 0x00007f6832b8fe7f in ldbm_back_search () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #20 0x00007f683f4bd1d6 in op_shared_search () from > /usr/lib64/dirsrv/libslapd.so.0 > #21 0x00007f683f4cd94e in search_internal_callback_pb () from > /usr/lib64/dirsrv/libslapd.so.0 > #22 0x00007f683f4cdbe8 in search_internal_pb () from > /usr/lib64/dirsrv/libslapd.so.0 > #23 0x00007f6832dd75dc in get_entry_by_principal () from > /usr/lib64/dirsrv/plugins/libipa_pwd_extop.so > #24 0x00007f6832dd8eb1 in ipapwd_extop () from /usr/lib64/dirsrv/plugins/ > libipa_pwd_extop.so > #25 0x00007f683f4c6ba2 in plugin_call_exop_plugins () from > /usr/lib64/dirsrv/libslapd.so.0 > #26 0x00007f683f9a2efe in do_extended () > #27 0x00007f683f99c41a in connection_threadmain () > #28 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so > #29 0x00007f683d04cdc5 in start_thread () from /lib64/libpthread.so.0 > #30 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 > Thread 16 (Thread 0x7f67fe7f4700 (LWP 4877)): > #0 0x00007f683d0506d5 in pthread_cond_wait@@GLIBC_2.3.2 () from > /lib64/libpthread.so.0 > #1 0x00007f683d6a6050 in PR_WaitCondVar () from /lib64/libnspr4.so > #2 0x00007f683f999fee in connection_wait_for_new_work () > #3 0x00007f683f99b719 in connection_threadmain () > #4 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so > #5 0x00007f683d04cdc5 in start_thread () from /lib64/libpthread.so.0 > #6 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 > Thread 15 (Thread 0x7f67fdff3700 (LWP 4878)): > #0 0x00007f683d0506d5 in pthread_cond_wait@@GLIBC_2.3.2 () from > /lib64/libpthread.so.0 > #1 0x00007f683d6a6050 in PR_WaitCondVar () from /lib64/libnspr4.so > #2 0x00007f683f999fee in connection_wait_for_new_work () > #3 0x00007f683f99b719 in connection_threadmain () > #4 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so > #5 0x00007f683d04cdc5 in start_thread () from /lib64/libpthread.so.0 > #6 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 > Thread 14 (Thread 0x7f67fd7f2700 (LWP 4879)): > #0 0x00007f683d0506d5 in pthread_cond_wait@@GLIBC_2.3.2 () from > /lib64/libpthread.so.0 > #1 0x00007f683d6a6050 in PR_WaitCondVar () from /lib64/libnspr4.so > #2 0x00007f683f999fee in connection_wait_for_new_work () > #3 0x00007f683f99b719 in connection_threadmain () > #4 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so > #5 0x00007f683d04cdc5 in start_thread () from /lib64/libpthread.so.0 > #6 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 > Thread 13 (Thread 0x7f67fcff1700 (LWP 4880)): > #0 0x00007f683d0506d5 in pthread_cond_wait@@GLIBC_2.3.2 () from > /lib64/libpthread.so.0 > #1 0x00007f683d6a6050 in PR_WaitCondVar () from /lib64/libnspr4.so > #2 0x00007f683f999fee in connection_wait_for_new_work () > #3 0x00007f683f99b719 in connection_threadmain () > #4 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so > #5 0x00007f683d04cdc5 in start_thread () from /lib64/libpthread.so.0 > #6 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 > Thread 12 (Thread 0x7f67fc7f0700 (LWP 4881)): > #0 0x00007f683d0506d5 in pthread_cond_wait@@GLIBC_2.3.2 () from > /lib64/libpthread.so.0 > #1 0x00007f683d6a6050 in PR_WaitCondVar () from /lib64/libnspr4.so > #2 0x00007f683f999fee in connection_wait_for_new_work () > #3 0x00007f683f99b719 in connection_threadmain () > #4 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so > #5 0x00007f683d04cdc5 in start_thread () from /lib64/libpthread.so.0 > #6 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 > Thread 11 (Thread 0x7f67fbfef700 (LWP 4882)): > #0 0x00007f683d0506d5 in pthread_cond_wait@@GLIBC_2.3.2 () from > /lib64/libpthread.so.0 > #1 0x00007f683d6a6050 in PR_WaitCondVar () from /lib64/libnspr4.so > #2 0x00007f683f999fee in connection_wait_for_new_work () > #3 0x00007f683f99b719 in connection_threadmain () > #4 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so > #5 0x00007f683d04cdc5 in start_thread () from /lib64/libpthread.so.0 > #6 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 > Thread 10 (Thread 0x7f67fb7ee700 (LWP 4883)): > #0 0x00007f683d0506d5 in pthread_cond_wait@@GLIBC_2.3.2 () from > /lib64/libpthread.so.0 > #1 0x00007f68375ac2f3 in __db_hybrid_mutex_suspend () from /lib64/ > libdb-5.3.so > #2 0x00007f68375ab640 in __db_tas_mutex_lock () from /lib64/libdb-5.3.so > #3 0x00007f6837655d3a in __lock_get_internal () from /lib64/libdb-5.3.so > #4 0x00007f6837656820 in __lock_get () from /lib64/libdb-5.3.so > #5 0x00007f6837682142 in __db_lget () from /lib64/libdb-5.3.so > #6 0x00007f68375c9605 in __bam_search () from /lib64/libdb-5.3.so > #7 0x00007f68375b4256 in __bamc_search () from /lib64/libdb-5.3.so > #8 0x00007f68375b5d0f in __bamc_get () from /lib64/libdb-5.3.so > #9 0x00007f683766eca6 in __dbc_iget () from /lib64/libdb-5.3.so > #10 0x00007f683767db02 in __dbc_get_pp () from /lib64/libdb-5.3.so > #11 0x00007f6832b59680 in idl_new_fetch () from /usr/lib64/dirsrv/plugins/ > libback-ldbm.so > #12 0x00007f6832b67b26 in index_read_ext_allids () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #13 0x00007f6832b52284 in keys2idl () from /usr/lib64/dirsrv/plugins/ > libback-ldbm.so > #14 0x00007f6832b529e3 in ava_candidates.isra.0 () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #15 0x00007f6832b52fd2 in filter_candidates_ext () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #16 0x00007f6832b54096 in list_candidates () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #17 0x00007f6832b52f40 in filter_candidates_ext () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #18 0x00007f6832b54096 in list_candidates () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #19 0x00007f6832b52f40 in filter_candidates_ext () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #20 0x00007f6832b54096 in list_candidates () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #21 0x00007f6832b52f40 in filter_candidates_ext () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #22 0x00007f6832b8e80c in subtree_candidates () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #23 0x00007f6832b8fe7f in ldbm_back_search () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #24 0x00007f683f4bd1d6 in op_shared_search () from > /usr/lib64/dirsrv/libslapd.so.0 > #25 0x00007f683f9ae33e in do_search () > #26 0x00007f683f99c3c3 in connection_threadmain () > #27 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so > #28 0x00007f683d04cdc5 in start_thread () from /lib64/libpthread.so.0 > #29 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 > Thread 9 (Thread 0x7f67fafed700 (LWP 4884)): > #0 0x00007f683d0506d5 in pthread_cond_wait@@GLIBC_2.3.2 () from > /lib64/libpthread.so.0 > #1 0x00007f683d6a6050 in PR_WaitCondVar () from /lib64/libnspr4.so > #2 0x00007f683f999fee in connection_wait_for_new_work () > #3 0x00007f683f99b719 in connection_threadmain () > #4 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so > #5 0x00007f683d04cdc5 in start_thread () from /lib64/libpthread.so.0 > #6 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 > Thread 8 (Thread 0x7f67fa7ec700 (LWP 4885)): > #0 0x00007f683d0506d5 in pthread_cond_wait@@GLIBC_2.3.2 () from > /lib64/libpthread.so.0 > #1 0x00007f68375ac2f3 in __db_hybrid_mutex_suspend () from /lib64/ > libdb-5.3.so > #2 0x00007f68375ab640 in __db_tas_mutex_lock () from /lib64/libdb-5.3.so > #3 0x00007f6837655d3a in __lock_get_internal () from /lib64/libdb-5.3.so > #4 0x00007f6837656820 in __lock_get () from /lib64/libdb-5.3.so > #5 0x00007f6837682142 in __db_lget () from /lib64/libdb-5.3.so > #6 0x00007f68375c9605 in __bam_search () from /lib64/libdb-5.3.so > #7 0x00007f68375b4256 in __bamc_search () from /lib64/libdb-5.3.so > #8 0x00007f68375b5d0f in __bamc_get () from /lib64/libdb-5.3.so > #9 0x00007f683766eca6 in __dbc_iget () from /lib64/libdb-5.3.so > #10 0x00007f683767db02 in __dbc_get_pp () from /lib64/libdb-5.3.so > #11 0x00007f6832b59680 in idl_new_fetch () from /usr/lib64/dirsrv/plugins/ > libback-ldbm.so > #12 0x00007f6832b67b26 in index_read_ext_allids () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #13 0x00007f6832b52284 in keys2idl () from /usr/lib64/dirsrv/plugins/ > libback-ldbm.so > #14 0x00007f6832b529e3 in ava_candidates.isra.0 () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #15 0x00007f6832b52fd2 in filter_candidates_ext () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #16 0x00007f6832b54096 in list_candidates () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #17 0x00007f6832b52f40 in filter_candidates_ext () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #18 0x00007f6832b54096 in list_candidates () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #19 0x00007f6832b52f40 in filter_candidates_ext () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #20 0x00007f6832b54096 in list_candidates () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #21 0x00007f6832b52f40 in filter_candidates_ext () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #22 0x00007f6832b8e80c in subtree_candidates () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #23 0x00007f6832b8fe7f in ldbm_back_search () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #24 0x00007f683f4bd1d6 in op_shared_search () from > /usr/lib64/dirsrv/libslapd.so.0 > #25 0x00007f683f9ae33e in do_search () > #26 0x00007f683f99c3c3 in connection_threadmain () > #27 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so > #28 0x00007f683d04cdc5 in start_thread () from /lib64/libpthread.so.0 > #29 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 > Thread 7 (Thread 0x7f67f9feb700 (LWP 4886)): > #0 0x00007f683d0506d5 in pthread_cond_wait@@GLIBC_2.3.2 () from > /lib64/libpthread.so.0 > #1 0x00007f683d6a6050 in PR_WaitCondVar () from /lib64/libnspr4.so > #2 0x00007f683f999fee in connection_wait_for_new_work () > #3 0x00007f683f99b719 in connection_threadmain () > #4 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so > #5 0x00007f683d04cdc5 in start_thread () from /lib64/libpthread.so.0 > #6 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 > Thread 6 (Thread 0x7f67f97ea700 (LWP 4887)): > #0 0x00007f683d0506d5 in pthread_cond_wait@@GLIBC_2.3.2 () from > /lib64/libpthread.so.0 > #1 0x00007f683d6a6050 in PR_WaitCondVar () from /lib64/libnspr4.so > #2 0x00007f683f999fee in connection_wait_for_new_work () > #3 0x00007f683f99b719 in connection_threadmain () > #4 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so > #5 0x00007f683d04cdc5 in start_thread () from /lib64/libpthread.so.0 > #6 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 > Thread 5 (Thread 0x7f67f8fe9700 (LWP 4888)): > #0 0x00007f683d0506d5 in pthread_cond_wait@@GLIBC_2.3.2 () from > /lib64/libpthread.so.0 > #1 0x00007f683d6a6050 in PR_WaitCondVar () from /lib64/libnspr4.so > #2 0x00007f683f999fee in connection_wait_for_new_work () > #3 0x00007f683f99b719 in connection_threadmain () > #4 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so > #5 0x00007f683d04cdc5 in start_thread () from /lib64/libpthread.so.0 > #6 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 > Thread 4 (Thread 0x7f67f87e8700 (LWP 4889)): > #0 0x00007f683cd719b3 in select () from /lib64/libc.so.6 > #1 0x00007f683f4fc0e9 in DS_Sleep () from /usr/lib64/dirsrv/libslapd.so.0 > #2 0x00007f683f99d325 in time_thread () > #3 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so > #4 0x00007f683d04cdc5 in start_thread () from /lib64/libpthread.so.0 > #5 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 > Thread 3 (Thread 0x7f67f7de6700 (LWP 4892)): > #0 0x00007f683d0506d5 in pthread_cond_wait@@GLIBC_2.3.2 () from > /lib64/libpthread.so.0 > #1 0x00007f683d6a6050 in PR_WaitCondVar () from /lib64/libnspr4.so > #2 0x00007f683f9a8f25 in ps_send_results () > #3 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so > #4 0x00007f683d04cdc5 in start_thread () from /lib64/libpthread.so.0 > #5 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 > Thread 2 (Thread 0x7f67f75e5700 (LWP 5084)): > #0 0x00007f683d0506d5 in pthread_cond_wait@@GLIBC_2.3.2 () from > /lib64/libpthread.so.0 > #1 0x00007f683d6a6050 in PR_WaitCondVar () from /lib64/libnspr4.so > #2 0x00007f683f9a8f25 in ps_send_results () > #3 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so > #4 0x00007f683d04cdc5 in start_thread () from /lib64/libpthread.so.0 > #5 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 > Thread 1 (Thread 0x7f683f964840 (LWP 4845)): > #0 0x00007f683d0506d5 in pthread_cond_wait@@GLIBC_2.3.2 () from > /lib64/libpthread.so.0 > #1 0x00007f683d6a62c3 in PR_EnterMonitor () from /lib64/libnspr4.so > #2 0x00007f683f99d5ed in setup_pr_read_pds () > #3 0x00007f683f9a0fd9 in slapd_daemon () > #4 0x00007f683f992c6c in main () > > > > the top output seems ok, there was just a sudden spike to around 340% cpu > then it comes down > > top - 08:53:12 up 6 days, 23:20, 1 user, load average: 0.00, 0.01, 0.12 > Threads: 44 total, 0 running, 44 sleeping, 0 stopped, 0 zombie > %Cpu(s): 0.0 us, 0.0 sy, 0.0 ni,100.0 id, 0.0 wa, 0.0 hi, 0.0 si, > 0.0 st > KiB Mem : 65533176 total, 52165676 free, 6135088 used, 7232412 buff/cache > KiB Swap: 3999996 total, 3999996 free, 0 used. 58452040 avail Mem > > PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND > 4845 dirsrv 20 0 5133196 3.150g 281272 S 0.0 5.0 31:34.51 > ns-slapd > 4847 dirsrv 20 0 5133196 3.150g 281272 S 0.0 5.0 0:13.53 > ns-slapd > 4848 dirsrv 20 0 5133196 3.150g 281272 S 0.0 5.0 0:13.16 > ns-slapd > 4849 dirsrv 20 0 5133196 3.150g 281272 S 0.0 5.0 0:45.27 > ns-slapd > 4850 dirsrv 20 0 5133196 3.150g 281272 S 0.0 5.0 0:01.60 > ns-slapd > 4851 dirsrv 20 0 5133196 3.150g 281272 S 0.0 5.0 0:00.00 > ns-slapd > 4852 dirsrv 20 0 5133196 3.150g 281272 S 0.0 5.0 0:00.00 > ns-slapd > 4853 dirsrv 20 0 5133196 3.150g 281272 S 0.0 5.0 0:00.00 > ns-slapd > 4854 dirsrv 20 0 5133196 3.150g 281272 S 0.0 5.0 0:00.00 > ns-slapd > 4855 dirsrv 20 0 5133196 3.150g 281272 S 0.0 5.0 0:03.02 > ns-slapd > 4856 dirsrv 20 0 5133196 3.150g 281272 S 0.0 5.0 0:00.26 > ns-slapd > 4859 dirsrv 20 0 5133196 3.150g 281272 S 0.0 5.0 3:18.67 > ns-slapd > 4860 dirsrv 20 0 5133196 3.150g 281272 S 0.0 5.0 3:19.82 > ns-slapd > 4861 dirsrv 20 0 5133196 3.150g 281272 S 0.0 5.0 3:20.23 > ns-slapd > 4862 dirsrv 20 0 5133196 3.150g 281272 S 0.0 5.0 3:22.65 > ns-slapd > 4863 dirsrv 20 0 5133196 3.150g 281272 S 0.0 5.0 3:16.64 > ns-slapd > 4864 dirsrv 20 0 5133196 3.150g 281272 S 0.0 5.0 3:17.39 > ns-slapd > 4865 dirsrv 20 0 5133196 3.150g 281272 S 0.0 5.0 3:17.80 > ns-slapd > > > Thanks > Rakesh > > On Tue, Sep 6, 2016 at 12:09 AM, Rakesh Rajasekharan < > rakesh.rajasekharan at gmail.com> wrote: > >> Hi Thierry/Ludwig, >> >> Unfortunately I had by then restarted the service and since then have >> been trying to hit the hang issue again but no luck this time >> >> It has been running fine for the last few hours now in-spite of trying >> all that I believe could cause the hang like making simultaneous client >> addition requests. >> >> I will continue testing this and next time I get it I will make sure I >> get the pstack and also get the dump of db lock info. >> >> Thanks, >> Rakesh >> >> >> >> On Mon, Sep 5, 2016 at 6:14 PM, Ludwig Krispenz < >> lkrispen at redhat.com> wrote: >> >>> >>> On 09/05/2016 02:31 PM, thierry bordaz wrote: >>> >>> Hi Rakesh, >>> >>> Thanks you soooo much for you continuous help on this !! >>> >>> If the hang is still going on, would you dump the DB lock information >>> with: >>> >>> db_stat -CA -N -h /var/lib/dirsrv/slapd-/db >>> >/tmp/db_lock_info >>> >>> It is looking like several threads are waiting for one/several db locks. >>> With the lock info we will know which thread acquired it/them. >>> A guess is that it could be the checkpointing thread but this one is >>> also waiting (on a memory pool). >>> >>> I am copying Ludwig on this thread in case this pstack rings any bell. >>> >>> the frequent presence of __db_tas_mutex_lock() looks like the NUMA >>> problem of libdb, although the threads already have stopped spinning (maybe >>> this was an effect of gcore). >>> If the hang aoccurs again, could you check cpu consumption >>> top -H -p >>> and get a pstack >>> pstack >>> >>> >>> best regards >>> thierry >>> >>> On 09/05/2016 02:05 PM, Rakesh Rajasekharan wrote: >>> >>> Hi Thierry, >>> >>> I was able to now reporduce the issue.. >>> >>> the ns-slapd process just got hung,.. however, no core file was >>> generate.. >>> >>> I manually created a coredump using gcore and then took a strace of it >>> this is strace from the coredump generated >>> >>> GNU gdb (GDB) Red Hat Enterprise Linux 7.6.1-80.el7 >>> Copyright (C) 2013 Free Software Foundation, Inc. >>> License GPLv3+: GNU GPL version 3 or later < >>> http://gnu.org/licenses/gpl.html> >>> This is free software: you are free to change and redistribute it. >>> There is NO WARRANTY, to the extent permitted by law. Type "show >>> copying" >>> and "show warranty" for details. >>> This GDB was configured as "x86_64-redhat-linux-gnu". >>> For bug reporting instructions, please see: >>> ... >>> Reading symbols from /usr/sbin/ns-slapd...Reading symbols from >>> /usr/sbin/ns-slapd...(no debugging symbols found)...done. >>> (no debugging symbols found)...done. >>> [New LWP 105159] >>> [New LWP 105160] >>> [New LWP 105161] >>> [New LWP 105162] >>> [New LWP 105163] >>> [New LWP 105164] >>> [New LWP 105165] >>> [New LWP 105166] >>> [New LWP 105167] >>> [New LWP 105168] >>> [New LWP 105171] >>> [New LWP 105172] >>> [New LWP 105173] >>> [New LWP 105174] >>> [New LWP 105175] >>> [New LWP 105176] >>> [New LWP 105177] >>> [New LWP 105178] >>> [New LWP 105179] >>> [New LWP 105180] >>> [New LWP 105181] >>> [New LWP 105182] >>> [New LWP 105183] >>> [New LWP 105184] >>> [New LWP 105185] >>> [New LWP 105186] >>> [New LWP 105187] >>> [New LWP 105188] >>> [New LWP 105189] >>> [New LWP 105190] >>> [New LWP 105191] >>> [New LWP 105192] >>> [New LWP 105193] >>> [New LWP 105194] >>> [New LWP 105195] >>> [New LWP 105196] >>> [New LWP 105197] >>> [New LWP 105198] >>> [New LWP 105199] >>> [New LWP 105200] >>> [New LWP 105201] >>> [New LWP 105203] >>> [New LWP 105215] >>> [New LWP 105158] >>> [Thread debugging using libthread_db enabled] >>> Using host libthread_db library "/lib64/libthread_db.so.1". >>> Core was generated by `/usr/sbin/ns-slapd'. >>> #0 0x00007f1b48ae69b3 in select () from /lib64/libc.so.6 >>> >>> Thread 44 (Thread 0x7f1b4b6d9840 (LWP 105158)): >>> #0 0x00007f1b48dc56d5 in >>> pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 >>> No symbol table info available. >>> #1 0x00007f1b4941b2c3 in PR_EnterMonitor () from /lib64/libnspr4.so >>> No symbol table info available. >>> #2 0x00007f1b4b7125ed in setup_pr_read_pds () >>> No symbol table info available. >>> #3 0x00007f1b4b715fd9 in slapd_daemon () >>> No symbol table info available. >>> #4 0x00007f1b4b707c6c in main () >>> No symbol table info available. >>> >>> Thread 43 (Thread 0x7f1a5df0f700 (LWP 105215)): >>> #0 0x00007f1b48dc56d5 in >>> pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 >>> No symbol table info available. >>> #1 0x00007f1b4941b050 in PR_WaitCondVar () from /lib64/libnspr4.so >>> No symbol table info available. >>> #2 0x00007f1b4b71df25 in ps_send_results () >>> No symbol table info available. >>> #3 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so >>> No symbol table info available. >>> #4 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 >>> No symbol table info available. >>> #5 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 >>> No symbol table info available. >>> >>> Thread 42 (Thread 0x7f1b03fe7700 (LWP 105203)): >>> #0 0x00007f1b48dc56d5 in >>> pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 >>> No symbol table info available. >>> #1 0x00007f1b4941b050 in PR_WaitCondVar () from /lib64/libnspr4.so >>> No symbol table info available. >>> #2 0x00007f1b4b71df25 in ps_send_results () >>> No symbol table info available. >>> #3 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so >>> No symbol table info available. >>> #4 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 >>> No symbol table info available. >>> #5 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 >>> No symbol table info available. >>> >>> Thread 41 (Thread 0x7f1b047e8700 (LWP 105201)): >>> #0 0x00007f1b48ae69b3 in select () from /lib64/libc.so.6 >>> No symbol table info available. >>> #1 0x00007f1b4b2710e9 in DS_Sleep () from /usr/lib64/dirsrv/libslapd.so. >>> 0 >>> No symbol table info available. >>> #2 0x00007f1b4b712325 in time_thread () >>> No symbol table info available. >>> #3 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so >>> No symbol table info available. >>> #4 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 >>> No symbol table info available. >>> #5 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 >>> No symbol table info available. >>> >>> Thread 40 (Thread 0x7f1b04fe9700 (LWP 105200)): >>> #0 0x00007f1b48dc56d5 in >>> pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 >>> No symbol table info available. >>> #1 0x00007f1b433212f3 in __db_hybrid_mutex_suspend () from /lib64/ >>> libdb-5.3.so >>> No symbol table info available. >>> #2 0x00007f1b43320640 in __db_tas_mutex_lock () from /lib64/ >>> libdb-5.3.so >>> No symbol table info available. >>> #3 0x00007f1b4343ae1b in __memp_fget () from /lib64/libdb-5.3.so >>> No symbol table info available. >>> #4 0x00007f1b4333e211 in __bam_search () from /lib64/libdb-5.3.so >>> No symbol table info available. >>> #5 0x00007f1b43329256 in __bamc_search () from /lib64/libdb-5.3.so >>> No symbol table info available. >>> #6 0x00007f1b4332d6c4 in __bamc_put () from /lib64/libdb-5.3.so >>> No symbol table info available. >>> #7 0x00007f1b433e46d5 in __dbc_iput () from /lib64/libdb-5.3.so >>> No symbol table info available. >>> #8 0x00007f1b433dfa0e in __db_put () from /lib64/libdb-5.3.so >>> No symbol table info available. >>> #9 0x00007f1b433f4fa4 in __db_put_pp () from /lib64/libdb-5.3.so >>> No symbol table info available. >>> #10 0x00007f1b3e8cf4cf in idl_new_insert_key () from >>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>> No symbol table info available. >>> #11 0x00007f1b3e8de5b5 in addordel_values_sv.isra.3 () from >>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>> No symbol table info available. >>> #12 0x00007f1b3e8df25a in index_addordel_values_ext_sv () from >>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>> No symbol table info available. >>> #13 0x00007f1b3e8df694 in index_addordel_values_sv () from >>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>> No symbol table info available. >>> #14 0x00007f1b3e8df75a in index_addordel_entry () from >>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>> No symbol table info available. >>> #15 0x00007f1b3e8e25fa in ldbm_back_add () from >>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>> No symbol table info available. >>> #16 0x00007f1b4b1e4cca in op_shared_add () from >>> /usr/lib64/dirsrv/libslapd.so.0 >>> No symbol table info available. >>> #17 0x00007f1b4b1e6020 in do_add () from /usr/lib64/dirsrv/libslapd.so.0 >>> No symbol table info available. >>> #18 0x00007f1b4b711303 in connection_threadmain () >>> No symbol table info available. >>> #19 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so >>> No symbol table info available. >>> #20 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 >>> No symbol table info available. >>> #21 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 >>> No symbol table info available. >>> >>> Thread 39 (Thread 0x7f1b057ea700 (LWP 105199)): >>> #0 0x00007f1b48dc56d5 in >>> pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 >>> No symbol table info available. >>> #1 0x00007f1b4941b050 in PR_WaitCondVar () from /lib64/libnspr4.so >>> No symbol table info available. >>> #2 0x00007f1b4b70efee in connection_wait_for_new_work () >>> No symbol table info available. >>> #3 0x00007f1b4b710719 in connection_threadmain () >>> No symbol table info available. >>> #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so >>> No symbol table info available. >>> #5 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 >>> No symbol table info available. >>> #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 >>> No symbol table info available. >>> >>> Thread 38 (Thread 0x7f1b05feb700 (LWP 105198)): >>> #0 0x00007f1b48dc56d5 in >>> pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 >>> No symbol table info available. >>> #1 0x00007f1b4941b050 in PR_WaitCondVar () from /lib64/libnspr4.so >>> No symbol table info available. >>> #2 0x00007f1b4b70efee in connection_wait_for_new_work () >>> No symbol table info available. >>> #3 0x00007f1b4b710719 in connection_threadmain () >>> No symbol table info available. >>> #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so >>> No symbol table info available. >>> #5 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 >>> No symbol table info available. >>> #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 >>> No symbol table info available. >>> >>> Thread 37 (Thread 0x7f1b067ec700 (LWP 105197)): >>> #0 0x00007f1b48dc56d5 in >>> pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 >>> No symbol table info available. >>> #1 0x00007f1b4941b050 in PR_WaitCondVar () from /lib64/libnspr4.so >>> No symbol table info available. >>> #2 0x00007f1b4b70efee in connection_wait_for_new_work () >>> No symbol table info available. >>> #3 0x00007f1b4b710719 in connection_threadmain () >>> No symbol table info available. >>> #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so >>> No symbol table info available. >>> #5 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 >>> No symbol table info available. >>> #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 >>> No symbol table info available. >>> >>> Thread 36 (Thread 0x7f1b06fed700 (LWP 105196)): >>> #0 0x00007f1b48dc56d5 in >>> pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 >>> No symbol table info available. >>> #1 0x00007f1b4941b050 in PR_WaitCondVar () from /lib64/libnspr4.so >>> No symbol table info available. >>> #2 0x00007f1b4b70efee in connection_wait_for_new_work () >>> No symbol table info available. >>> #3 0x00007f1b4b710719 in connection_threadmain () >>> No symbol table info available. >>> #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so >>> No symbol table info available. >>> #5 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 >>> No symbol table info available. >>> #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 >>> No symbol table info available. >>> >>> Thread 35 (Thread 0x7f1b077ee700 (LWP 105195)): >>> #0 0x00007f1b48dc56d5 in >>> pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 >>> No symbol table info available. >>> #1 0x00007f1b4941b2c3 in PR_EnterMonitor () from /lib64/libnspr4.so >>> No symbol table info available. >>> #2 0x00007f1b3e8c23e6 in dblayer_txn_begin () from >>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>> No symbol table info available. >>> #3 0x00007f1b3e8fe007 in ldbm_back_modify () from >>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>> No symbol table info available. >>> #4 0x00007f1b4b22b541 in op_shared_modify () from >>> /usr/lib64/dirsrv/libslapd.so.0 >>> No symbol table info available. >>> #5 0x00007f1b4b22c004 in modify_internal_pb () from >>> /usr/lib64/dirsrv/libslapd.so.0 >>> No symbol table info available. >>> #6 0x00007f1b3eb48b60 in ipapwd_apply_mods () from >>> /usr/lib64/dirsrv/plugins/libipa_pwd_extop.so >>> No symbol table info available. >>> #7 0x00007f1b3eb4c8e0 in store_new_keys () from >>> /usr/lib64/dirsrv/plugins/libipa_pwd_extop.so >>> No symbol table info available. >>> #8 0x00007f1b3eb4ead0 in ipapwd_extop () from >>> /usr/lib64/dirsrv/plugins/libipa_pwd_extop.so >>> No symbol table info available. >>> #9 0x00007f1b4b23bba2 in plugin_call_exop_plugins () from >>> /usr/lib64/dirsrv/libslapd.so.0 >>> No symbol table info available. >>> #10 0x00007f1b4b717efe in do_extended () >>> No symbol table info available. >>> #11 0x00007f1b4b71141a in connection_threadmain () >>> No symbol table info available. >>> #12 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so >>> No symbol table info available. >>> #13 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 >>> No symbol table info available. >>> #14 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 >>> No symbol table info available. >>> >>> Thread 34 (Thread 0x7f1b07fef700 (LWP 105194)): >>> #0 0x00007f1b48dc56d5 in >>> pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 >>> No symbol table info available. >>> #1 0x00007f1b4941b050 in PR_WaitCondVar () from /lib64/libnspr4.so >>> No symbol table info available. >>> #2 0x00007f1b4b70efee in connection_wait_for_new_work () >>> No symbol table info available. >>> #3 0x00007f1b4b710719 in connection_threadmain () >>> No symbol table info available. >>> #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so >>> No symbol table info available. >>> #5 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 >>> No symbol table info available. >>> #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 >>> No symbol table info available. >>> >>> Thread 33 (Thread 0x7f1b087f0700 (LWP 105193)): >>> #0 0x00007f1b48dc56d5 in >>> pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 >>> No symbol table info available. >>> #1 0x00007f1b4941b050 in PR_WaitCondVar () from /lib64/libnspr4.so >>> No symbol table info available. >>> #2 0x00007f1b4b70efee in connection_wait_for_new_work () >>> No symbol table info available. >>> #3 0x00007f1b4b710719 in connection_threadmain () >>> No symbol table info available. >>> #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so >>> No symbol table info available. >>> #5 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 >>> No symbol table info available. >>> #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 >>> No symbol table info available. >>> >>> Thread 32 (Thread 0x7f1b08ff1700 (LWP 105192)): >>> #0 0x00007f1b48dc56d5 in >>> pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 >>> No symbol table info available. >>> #1 0x00007f1b4941b050 in PR_WaitCondVar () from /lib64/libnspr4.so >>> No symbol table info available. >>> #2 0x00007f1b4b70efee in connection_wait_for_new_work () >>> No symbol table info available. >>> #3 0x00007f1b4b710719 in connection_threadmain () >>> No symbol table info available. >>> #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so >>> No symbol table info available. >>> #5 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 >>> No symbol table info available. >>> #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 >>> No symbol table info available. >>> >>> Thread 31 (Thread 0x7f1b097f2700 (LWP 105191)): >>> #0 0x00007f1b48dc56d5 in >>> pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 >>> No symbol table info available. >>> #1 0x00007f1b4941b050 in PR_WaitCondVar () from /lib64/libnspr4.so >>> No symbol table info available. >>> #2 0x00007f1b4b70efee in connection_wait_for_new_work () >>> No symbol table info available. >>> #3 0x00007f1b4b710719 in connection_threadmain () >>> No symbol table info available. >>> #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so >>> No symbol table info available. >>> #5 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 >>> No symbol table info available. >>> #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 >>> No symbol table info available. >>> >>> Thread 30 (Thread 0x7f1b09ff3700 (LWP 105190)): >>> #0 0x00007f1b48dc56d5 in >>> pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 >>> No symbol table info available. >>> #1 0x00007f1b433212f3 in __db_hybrid_mutex_suspend () from /lib64/ >>> libdb-5.3.so >>> No symbol table info available. >>> #2 0x00007f1b43320640 in __db_tas_mutex_lock () from /lib64/ >>> libdb-5.3.so >>> No symbol table info available. >>> #3 0x00007f1b433cad3a in __lock_get_internal () from /lib64/ >>> libdb-5.3.so >>> No symbol table info available. >>> #4 0x00007f1b433cb820 in __lock_get () from /lib64/libdb-5.3.so >>> No symbol table info available. >>> #5 0x00007f1b433f7142 in __db_lget () from /lib64/libdb-5.3.so >>> No symbol table info available. >>> #6 0x00007f1b4333e605 in __bam_search () from /lib64/libdb-5.3.so >>> No symbol table info available. >>> #7 0x00007f1b43329256 in __bamc_search () from /lib64/libdb-5.3.so >>> No symbol table info available. >>> #8 0x00007f1b4332ad0f in __bamc_get () from /lib64/libdb-5.3.so >>> No symbol table info available. >>> #9 0x00007f1b433e3ca6 in __dbc_iget () from /lib64/libdb-5.3.so >>> No symbol table info available. >>> #10 0x00007f1b433f2b02 in __dbc_get_pp () from /lib64/libdb-5.3.so >>> No symbol table info available. >>> #11 0x00007f1b3e8ce680 in idl_new_fetch () from >>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>> No symbol table info available. >>> #12 0x00007f1b3e8dcb26 in index_read_ext_allids () from >>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>> No symbol table info available. >>> #13 0x00007f1b3e8c7284 in keys2idl () from /usr/lib64/dirsrv/plugins/libb >>> ack-ldbm.so >>> No symbol table info available. >>> #14 0x00007f1b3e8c79e3 in ava_candidates.isra.0 () from >>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>> No symbol table info available. >>> #15 0x00007f1b3e8c7fd2 in filter_candidates_ext () from >>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>> No symbol table info available. >>> #16 0x00007f1b3e8c9096 in list_candidates () from >>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>> No symbol table info available. >>> #17 0x00007f1b3e8c7f40 in filter_candidates_ext () from >>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>> No symbol table info available. >>> #18 0x00007f1b3e8c9096 in list_candidates () from >>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>> No symbol table info available. >>> #19 0x00007f1b3e8c7f40 in filter_candidates_ext () from >>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>> No symbol table info available. >>> #20 0x00007f1b3e8c9096 in list_candidates () from >>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>> No symbol table info available. >>> #21 0x00007f1b3e8c7f40 in filter_candidates_ext () from >>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>> No symbol table info available. >>> #22 0x00007f1b3e90380c in subtree_candidates () from >>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>> No symbol table info available. >>> #23 0x00007f1b3e904e7f in ldbm_back_search () from >>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>> No symbol table info available. >>> #24 0x00007f1b4b2321d6 in op_shared_search () from >>> /usr/lib64/dirsrv/libslapd.so.0 >>> No symbol table info available. >>> #25 0x00007f1b4b72333e in do_search () >>> No symbol table info available. >>> #26 0x00007f1b4b7113c3 in connection_threadmain () >>> No symbol table info available. >>> #27 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so >>> No symbol table info available. >>> #28 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 >>> No symbol table info available. >>> #29 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 >>> No symbol table info available. >>> >>> Thread 29 (Thread 0x7f1b0a7f4700 (LWP 105189)): >>> #0 0x00007f1b48dc56d5 in >>> pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 >>> No symbol table info available. >>> #1 0x00007f1b4941b050 in PR_WaitCondVar () from /lib64/libnspr4.so >>> No symbol table info available. >>> #2 0x00007f1b4b70efee in connection_wait_for_new_work () >>> No symbol table info available. >>> #3 0x00007f1b4b710719 in connection_threadmain () >>> No symbol table info available. >>> #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so >>> No symbol table info available. >>> #5 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 >>> No symbol table info available. >>> #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 >>> No symbol table info available. >>> >>> Thread 28 (Thread 0x7f1b0aff5700 (LWP 105188)): >>> #0 0x00007f1b48dc56d5 in >>> pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 >>> No symbol table info available. >>> #1 0x00007f1b4941b050 in PR_WaitCondVar () from /lib64/libnspr4.so >>> No symbol table info available. >>> #2 0x00007f1b4b70efee in connection_wait_for_new_work () >>> No symbol table info available. >>> #3 0x00007f1b4b710719 in connection_threadmain () >>> No symbol table info available. >>> #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so >>> No symbol table info available. >>> #5 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 >>> No symbol table info available. >>> #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 >>> No symbol table info available. >>> >>> Thread 27 (Thread 0x7f1b0b7f6700 (LWP 105187)): >>> #0 0x00007f1b48dc56d5 in >>> pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 >>> No symbol table info available. >>> #1 0x00007f1b4941b050 in PR_WaitCondVar () from /lib64/libnspr4.so >>> No symbol table info available. >>> #2 0x00007f1b4b70efee in connection_wait_for_new_work () >>> No symbol table info available. >>> #3 0x00007f1b4b710719 in connection_threadmain () >>> No symbol table info available. >>> #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so >>> No symbol table info available. >>> #5 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 >>> No symbol table info available. >>> #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 >>> No symbol table info available. >>> >>> Thread 26 (Thread 0x7f1b0bff7700 (LWP 105186)): >>> #0 0x00007f1b48dc56d5 in >>> pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 >>> No symbol table info available. >>> #1 0x00007f1b433212f3 in __db_hybrid_mutex_suspend () from /lib64/ >>> libdb-5.3.so >>> No symbol table info available. >>> #2 0x00007f1b43320640 in __db_tas_mutex_lock () from /lib64/ >>> libdb-5.3.so >>> No symbol table info available. >>> #3 0x00007f1b433cad3a in __lock_get_internal () from /lib64/ >>> libdb-5.3.so >>> No symbol table info available. >>> #4 0x00007f1b433cb820 in __lock_get () from /lib64/libdb-5.3.so >>> No symbol table info available. >>> #5 0x00007f1b433f7142 in __db_lget () from /lib64/libdb-5.3.so >>> No symbol table info available. >>> #6 0x00007f1b4333e605 in __bam_search () from /lib64/libdb-5.3.so >>> No symbol table info available. >>> #7 0x00007f1b43329256 in __bamc_search () from /lib64/libdb-5.3.so >>> No symbol table info available. >>> #8 0x00007f1b4332ad0f in __bamc_get () from /lib64/libdb-5.3.so >>> No symbol table info available. >>> #9 0x00007f1b433e3ca6 in __dbc_iget () from /lib64/libdb-5.3.so >>> No symbol table info available. >>> #10 0x00007f1b433f2b02 in __dbc_get_pp () from /lib64/libdb-5.3.so >>> No symbol table info available. >>> #11 0x00007f1b3e8ce680 in idl_new_fetch () from >>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>> No symbol table info available. >>> #12 0x00007f1b3e8dcb26 in index_read_ext_allids () from >>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>> No symbol table info available. >>> #13 0x00007f1b3e8c7284 in keys2idl () from /usr/lib64/dirsrv/plugins/libb >>> ack-ldbm.so >>> No symbol table info available. >>> #14 0x00007f1b3e8c79e3 in ava_candidates.isra.0 () from >>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>> No symbol table info available. >>> #15 0x00007f1b3e8c7fd2 in filter_candidates_ext () from >>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>> No symbol table info available. >>> #16 0x00007f1b3e8c9096 in list_candidates () from >>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>> No symbol table info available. >>> #17 0x00007f1b3e8c7f40 in filter_candidates_ext () from >>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>> No symbol table info available. >>> #18 0x00007f1b3e90380c in subtree_candidates () from >>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>> No symbol table info available. >>> #19 0x00007f1b3e904e7f in ldbm_back_search () from >>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>> No symbol table info available. >>> #20 0x00007f1b4b2321d6 in op_shared_search () from >>> /usr/lib64/dirsrv/libslapd.so.0 >>> No symbol table info available. >>> #21 0x00007f1b4b24294e in search_internal_callback_pb () from >>> /usr/lib64/dirsrv/libslapd.so.0 >>> No symbol table info available. >>> #22 0x00007f1b4b242be8 in search_internal_pb () from >>> /usr/lib64/dirsrv/libslapd.so.0 >>> No symbol table info available. >>> #23 0x00007f1b4b72149b in ids_sasl_canon_user () >>> No symbol table info available. >>> #24 0x00007f1b491e193b in _sasl_canon_user () from /lib64/libsasl2.so.3 >>> No symbol table info available. >>> #25 0x00007f1b491e1c4c in _sasl_canon_user_lookup () from >>> /lib64/libsasl2.so.3 >>> No symbol table info available. >>> #26 0x00007f1b430e9fb9 in gssapi_server_mech_step () from >>> /usr/lib64/sasl2/libgssapiv2.so >>> No symbol table info available. >>> #27 0x00007f1b491edb9b in sasl_server_step () from /lib64/libsasl2.so.3 >>> No symbol table info available. >>> #28 0x00007f1b4b72283b in ids_sasl_check_bind () >>> No symbol table info available. >>> #29 0x00007f1b4b70a61a in do_bind () >>> No symbol table info available. >>> #30 0x00007f1b4b7113fd in connection_threadmain () >>> No symbol table info available. >>> #31 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so >>> No symbol table info available. >>> #32 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 >>> No symbol table info available. >>> #33 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 >>> No symbol table info available. >>> >>> Thread 25 (Thread 0x7f1b0c7f8700 (LWP 105185)): >>> #0 0x00007f1b48dc56d5 in >>> pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 >>> No symbol table info available. >>> #1 0x00007f1b4941b050 in PR_WaitCondVar () from /lib64/libnspr4.so >>> No symbol table info available. >>> #2 0x00007f1b4b70efee in connection_wait_for_new_work () >>> No symbol table info available. >>> #3 0x00007f1b4b710719 in connection_threadmain () >>> No symbol table info available. >>> #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so >>> No symbol table info available. >>> #5 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 >>> No symbol table info available. >>> #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 >>> No symbol table info available. >>> >>> Thread 24 (Thread 0x7f1b0cff9700 (LWP 105184)): >>> #0 0x00007f1b48dc56d5 in >>> pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 >>> No symbol table info available. >>> #1 0x00007f1b4941b050 in PR_WaitCondVar () from /lib64/libnspr4.so >>> No symbol table info available. >>> #2 0x00007f1b4b70efee in connection_wait_for_new_work () >>> No symbol table info available. >>> #3 0x00007f1b4b710719 in connection_threadmain () >>> No symbol table info available. >>> #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so >>> No symbol table info available. >>> #5 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 >>> No symbol table info available. >>> #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 >>> No symbol table info available. >>> >>> Thread 23 (Thread 0x7f1b0d7fa700 (LWP 105183)): >>> #0 0x00007f1b48dc56d5 in >>> pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 >>> No symbol table info available. >>> #1 0x00007f1b4941b050 in PR_WaitCondVar () from /lib64/libnspr4.so >>> No symbol table info available. >>> #2 0x00007f1b4b70efee in connection_wait_for_new_work () >>> No symbol table info available. >>> #3 0x00007f1b4b710719 in connection_threadmain () >>> No symbol table info available. >>> #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so >>> No symbol table info available. >>> #5 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 >>> No symbol table info available. >>> #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 >>> No symbol table info available. >>> >>> Thread 22 (Thread 0x7f1b0dffb700 (LWP 105182)): >>> #0 0x00007f1b48dc56d5 in >>> pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 >>> No symbol table info available. >>> #1 0x00007f1b4941b050 in PR_WaitCondVar () from /lib64/libnspr4.so >>> No symbol table info available. >>> #2 0x00007f1b4b70efee in connection_wait_for_new_work () >>> No symbol table info available. >>> #3 0x00007f1b4b710719 in connection_threadmain () >>> No symbol table info available. >>> #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so >>> No symbol table info available. >>> #5 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 >>> No symbol table info available. >>> #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 >>> No symbol table info available. >>> >>> Thread 21 (Thread 0x7f1b0e7fc700 (LWP 105181)): >>> #0 0x00007f1b48dc56d5 in >>> pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 >>> No symbol table info available. >>> #1 0x00007f1b4941b050 in PR_WaitCondVar () from /lib64/libnspr4.so >>> No symbol table info available. >>> #2 0x00007f1b4b70efee in connection_wait_for_new_work () >>> No symbol table info available. >>> #3 0x00007f1b4b710719 in connection_threadmain () >>> No symbol table info available. >>> #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so >>> No symbol table info available. >>> #5 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 >>> No symbol table info available. >>> #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 >>> No symbol table info available. >>> >>> Thread 20 (Thread 0x7f1b0effd700 (LWP 105180)): >>> #0 0x00007f1b48dc56d5 in >>> pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 >>> No symbol table info available. >>> #1 0x00007f1b433212f3 in __db_hybrid_mutex_suspend () from /lib64/ >>> libdb-5.3.so >>> No symbol table info available. >>> #2 0x00007f1b43320b37 in __db_tas_mutex_readlock () from /lib64/ >>> libdb-5.3.so >>> No symbol table info available. >>> #3 0x00007f1b4343ad35 in __memp_fget () from /lib64/libdb-5.3.so >>> No symbol table info available. >>> #4 0x00007f1b433e4333 in __dbc_iget () from /lib64/libdb-5.3.so >>> No symbol table info available. >>> #5 0x00007f1b433f2b02 in __dbc_get_pp () from /lib64/libdb-5.3.so >>> No symbol table info available. >>> #6 0x00007f1b3e8ce822 in idl_new_fetch () from >>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>> No symbol table info available. >>> #7 0x00007f1b3e8dcb26 in index_read_ext_allids () from >>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>> No symbol table info available. >>> #8 0x00007f1b3e8c7284 in keys2idl () from /usr/lib64/dirsrv/plugins/libb >>> ack-ldbm.so >>> No symbol table info available. >>> #9 0x00007f1b3e8c79e3 in ava_candidates.isra.0 () from >>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>> No symbol table info available. >>> #10 0x00007f1b3e8c7fd2 in filter_candidates_ext () from >>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>> No symbol table info available. >>> #11 0x00007f1b3e8c9096 in list_candidates () from >>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>> No symbol table info available. >>> #12 0x00007f1b3e8c7f40 in filter_candidates_ext () from >>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>> No symbol table info available. >>> #13 0x00007f1b3e8c9096 in list_candidates () from >>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>> No symbol table info available. >>> #14 0x00007f1b3e8c7f40 in filter_candidates_ext () from >>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>> No symbol table info available. >>> #15 0x00007f1b3e8c9096 in list_candidates () from >>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>> No symbol table info available. >>> #16 0x00007f1b3e8c7f40 in filter_candidates_ext () from >>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>> No symbol table info available. >>> #17 0x00007f1b3e90380c in subtree_candidates () from >>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>> No symbol table info available. >>> #18 0x00007f1b3e904e7f in ldbm_back_search () from >>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>> No symbol table info available. >>> #19 0x00007f1b4b2321d6 in op_shared_search () from >>> /usr/lib64/dirsrv/libslapd.so.0 >>> No symbol table info available. >>> #20 0x00007f1b4b72333e in do_search () >>> No symbol table info available. >>> #21 0x00007f1b4b7113c3 in connection_threadmain () >>> No symbol table info available. >>> #22 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so >>> No symbol table info available. >>> #23 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 >>> No symbol table info available. >>> #24 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 >>> No symbol table info available. >>> >>> Thread 19 (Thread 0x7f1b0f7fe700 (LWP 105179)): >>> #0 0x00007f1b48dc56d5 in >>> pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 >>> No symbol table info available. >>> #1 0x00007f1b433212f3 in __db_hybrid_mutex_suspend () from /lib64/ >>> libdb-5.3.so >>> No symbol table info available. >>> #2 0x00007f1b43320640 in __db_tas_mutex_lock () from /lib64/ >>> libdb-5.3.so >>> No symbol table info available. >>> #3 0x00007f1b433cad3a in __lock_get_internal () from /lib64/ >>> libdb-5.3.so >>> No symbol table info available. >>> #4 0x00007f1b433cb820 in __lock_get () from /lib64/libdb-5.3.so >>> No symbol table info available. >>> #5 0x00007f1b433f7142 in __db_lget () from /lib64/libdb-5.3.so >>> No symbol table info available. >>> #6 0x00007f1b4333e605 in __bam_search () from /lib64/libdb-5.3.so >>> No symbol table info available. >>> #7 0x00007f1b43329256 in __bamc_search () from /lib64/libdb-5.3.so >>> No symbol table info available. >>> #8 0x00007f1b4332ad0f in __bamc_get () from /lib64/libdb-5.3.so >>> No symbol table info available. >>> #9 0x00007f1b433e3ca6 in __dbc_iget () from /lib64/libdb-5.3.so >>> No symbol table info available. >>> #10 0x00007f1b433f2b02 in __dbc_get_pp () from /lib64/libdb-5.3.so >>> No symbol table info available. >>> #11 0x00007f1b3e8ce680 in idl_new_fetch () from >>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>> No symbol table info available. >>> #12 0x00007f1b3e8dcb26 in index_read_ext_allids () from >>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>> No symbol table info available. >>> #13 0x00007f1b3e8c7284 in keys2idl () from /usr/lib64/dirsrv/plugins/libb >>> ack-ldbm.so >>> No symbol table info available. >>> #14 0x00007f1b3e8c79e3 in ava_candidates.isra.0 () from >>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>> No symbol table info available. >>> #15 0x00007f1b3e8c7fd2 in filter_candidates_ext () from >>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>> No symbol table info available. >>> #16 0x00007f1b3e8c9096 in list_candidates () from >>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>> No symbol table info available. >>> #17 0x00007f1b3e8c7f40 in filter_candidates_ext () from >>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>> No symbol table info available. >>> #18 0x00007f1b3e8c9096 in list_candidates () from >>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>> No symbol table info available. >>> #19 0x00007f1b3e8c7f40 in filter_candidates_ext () from >>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>> No symbol table info available. >>> #20 0x00007f1b3e8c9096 in list_candidates () from >>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>> No symbol table info available. >>> #21 0x00007f1b3e8c7f40 in filter_candidates_ext () from >>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>> No symbol table info available. >>> #22 0x00007f1b3e90380c in subtree_candidates () from >>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>> No symbol table info available. >>> #23 0x00007f1b3e904e7f in ldbm_back_search () from >>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>> No symbol table info available. >>> #24 0x00007f1b4b2321d6 in op_shared_search () from >>> /usr/lib64/dirsrv/libslapd.so.0 >>> No symbol table info available. >>> #25 0x00007f1b4b72333e in do_search () >>> No symbol table info available. >>> #26 0x00007f1b4b7113c3 in connection_threadmain () >>> No symbol table info available. >>> #27 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so >>> No symbol table info available. >>> #28 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 >>> No symbol table info available. >>> #29 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 >>> No symbol table info available. >>> >>> Thread 18 (Thread 0x7f1b0ffff700 (LWP 105178)): >>> #0 0x00007f1b48dc56d5 in >>> pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 >>> No symbol table info available. >>> #1 0x00007f1b4941b050 in PR_WaitCondVar () from /lib64/libnspr4.so >>> No symbol table info available. >>> #2 0x00007f1b4b70efee in connection_wait_for_new_work () >>> No symbol table info available. >>> #3 0x00007f1b4b710719 in connection_threadmain () >>> No symbol table info available. >>> #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so >>> No symbol table info available. >>> #5 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 >>> No symbol table info available. >>> #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 >>> No symbol table info available. >>> >>> Thread 17 (Thread 0x7f1b14d28700 (LWP 105177)): >>> #0 0x00007f1b48dc56d5 in >>> pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 >>> No symbol table info available. >>> #1 0x00007f1b4941b050 in PR_WaitCondVar () from /lib64/libnspr4.so >>> No symbol table info available. >>> #2 0x00007f1b4b70efee in connection_wait_for_new_work () >>> No symbol table info available. >>> #3 0x00007f1b4b710719 in connection_threadmain () >>> No symbol table info available. >>> #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so >>> No symbol table info available. >>> #5 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 >>> No symbol table info available. >>> #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 >>> No symbol table info available. >>> >>> Thread 16 (Thread 0x7f1b15529700 (LWP 105176)): >>> #0 0x00007f1b48dc56d5 in >>> pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 >>> No symbol table info available. >>> #1 0x00007f1b4941b050 in PR_WaitCondVar () from /lib64/libnspr4.so >>> No symbol table info available. >>> #2 0x00007f1b4b70efee in connection_wait_for_new_work () >>> No symbol table info available. >>> #3 0x00007f1b4b710719 in connection_threadmain () >>> No symbol table info available. >>> #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so >>> No symbol table info available. >>> #5 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 >>> No symbol table info available. >>> #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 >>> No symbol table info available. >>> >>> Thread 15 (Thread 0x7f1b15d2a700 (LWP 105175)): >>> #0 0x00007f1b48dc56d5 in >>> pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 >>> No symbol table info available. >>> #1 0x00007f1b4941b050 in PR_WaitCondVar () from /lib64/libnspr4.so >>> No symbol table info available. >>> #2 0x00007f1b4b70efee in connection_wait_for_new_work () >>> No symbol table info available. >>> #3 0x00007f1b4b710719 in connection_threadmain () >>> No symbol table info available. >>> #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so >>> No symbol table info available. >>> #5 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 >>> No symbol table info available. >>> #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 >>> No symbol table info available. >>> >>> Thread 14 (Thread 0x7f1b1652b700 (LWP 105174)): >>> #0 0x00007f1b48dc56d5 in >>> pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 >>> No symbol table info available. >>> #1 0x00007f1b4941b050 in PR_WaitCondVar () from /lib64/libnspr4.so >>> No symbol table info available. >>> #2 0x00007f1b4b70efee in connection_wait_for_new_work () >>> No symbol table info available. >>> #3 0x00007f1b4b710719 in connection_threadmain () >>> No symbol table info available. >>> #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so >>> No symbol table info available. >>> #5 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 >>> No symbol table info available. >>> #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 >>> No symbol table info available. >>> >>> Thread 13 (Thread 0x7f1b16d2c700 (LWP 105173)): >>> #0 0x00007f1b48dc56d5 in >>> pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 >>> No symbol table info available. >>> #1 0x00007f1b433212f3 in __db_hybrid_mutex_suspend () from /lib64/ >>> libdb-5.3.so >>> No symbol table info available. >>> #2 0x00007f1b43320640 in __db_tas_mutex_lock () from /lib64/ >>> libdb-5.3.so >>> No symbol table info available. >>> #3 0x00007f1b433cad3a in __lock_get_internal () from /lib64/ >>> libdb-5.3.so >>> No symbol table info available. >>> #4 0x00007f1b433cb820 in __lock_get () from /lib64/libdb-5.3.so >>> No symbol table info available. >>> #5 0x00007f1b433f7142 in __db_lget () from /lib64/libdb-5.3.so >>> No symbol table info available. >>> #6 0x00007f1b4333e605 in __bam_search () from /lib64/libdb-5.3.so >>> No symbol table info available. >>> #7 0x00007f1b43329256 in __bamc_search () from /lib64/libdb-5.3.so >>> No symbol table info available. >>> #8 0x00007f1b4332ad0f in __bamc_get () from /lib64/libdb-5.3.so >>> No symbol table info available. >>> #9 0x00007f1b433e3ca6 in __dbc_iget () from /lib64/libdb-5.3.so >>> No symbol table info available. >>> #10 0x00007f1b433f2b02 in __dbc_get_pp () from /lib64/libdb-5.3.so >>> No symbol table info available. >>> #11 0x00007f1b3e8ce680 in idl_new_fetch () from >>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>> No symbol table info available. >>> #12 0x00007f1b3e8dcb26 in index_read_ext_allids () from >>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>> No symbol table info available. >>> #13 0x00007f1b3e8c7284 in keys2idl () from /usr/lib64/dirsrv/plugins/libb >>> ack-ldbm.so >>> No symbol table info available. >>> #14 0x00007f1b3e8c79e3 in ava_candidates.isra.0 () from >>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>> No symbol table info available. >>> #15 0x00007f1b3e8c7fd2 in filter_candidates_ext () from >>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>> No symbol table info available. >>> #16 0x00007f1b3e8c9096 in list_candidates () from >>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>> No symbol table info available. >>> #17 0x00007f1b3e8c7f40 in filter_candidates_ext () from >>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>> No symbol table info available. >>> #18 0x00007f1b3e8c9096 in list_candidates () from >>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>> No symbol table info available. >>> #19 0x00007f1b3e8c7f40 in filter_candidates_ext () from >>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>> No symbol table info available. >>> #20 0x00007f1b3e8c9096 in list_candidates () from >>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>> No symbol table info available. >>> #21 0x00007f1b3e8c7f40 in filter_candidates_ext () from >>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>> No symbol table info available. >>> #22 0x00007f1b3e90380c in subtree_candidates () from >>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>> No symbol table info available. >>> #23 0x00007f1b3e904e7f in ldbm_back_search () from >>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>> No symbol table info available. >>> #24 0x00007f1b4b2321d6 in op_shared_search () from >>> /usr/lib64/dirsrv/libslapd.so.0 >>> No symbol table info available. >>> #25 0x00007f1b4b72333e in do_search () >>> No symbol table info available. >>> #26 0x00007f1b4b7113c3 in connection_threadmain () >>> No symbol table info available. >>> #27 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so >>> No symbol table info available. >>> #28 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 >>> No symbol table info available. >>> #29 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 >>> No symbol table info available. >>> >>> Thread 12 (Thread 0x7f1b1752d700 (LWP 105172)): >>> #0 0x00007f1b48dc56d5 in >>> pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 >>> No symbol table info available. >>> #1 0x00007f1b4941b050 in PR_WaitCondVar () from /lib64/libnspr4.so >>> No symbol table info available. >>> #2 0x00007f1b4b70efee in connection_wait_for_new_work () >>> No symbol table info available. >>> #3 0x00007f1b4b710719 in connection_threadmain () >>> No symbol table info available. >>> #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so >>> No symbol table info available. >>> #5 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 >>> No symbol table info available. >>> #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 >>> No symbol table info available. >>> >>> Thread 11 (Thread 0x7f1b17d2e700 (LWP 105171)): >>> #0 0x00007f1b48dc56d5 in >>> pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 >>> No symbol table info available. >>> #1 0x00007f1b4941b050 in PR_WaitCondVar () from /lib64/libnspr4.so >>> No symbol table info available. >>> #2 0x00007f1b4b70efee in connection_wait_for_new_work () >>> No symbol table info available. >>> #3 0x00007f1b4b710719 in connection_threadmain () >>> No symbol table info available. >>> #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so >>> No symbol table info available. >>> #5 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 >>> No symbol table info available. >>> #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 >>> No symbol table info available. >>> >>> Thread 10 (Thread 0x7f1b2509d700 (LWP 105168)): >>> #0 0x00007f1b48dc5a82 in pthread_cond_timedwait@@GLIBC_2.3.2 () from >>> /lib64/libpthread.so.0 >>> No symbol table info available. >>> #1 0x00007f1b4941ab07 in pt_TimedWait () from /lib64/libnspr4.so >>> No symbol table info available. >>> #2 0x00007f1b4941afce in PR_WaitCondVar () from /lib64/libnspr4.so >>> No symbol table info available. >>> #3 0x00007f1b4b2061e8 in eq_loop () from /usr/lib64/dirsrv/libslapd.so. >>> 0 >>> No symbol table info available. >>> #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so >>> No symbol table info available. >>> #5 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 >>> No symbol table info available. >>> #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 >>> No symbol table info available. >>> >>> Thread 9 (Thread 0x7f1b2589e700 (LWP 105167)): >>> #0 0x00007f1b48dc5a82 in pthread_cond_timedwait@@GLIBC_2.3.2 () from >>> /lib64/libpthread.so.0 >>> No symbol table info available. >>> #1 0x00007f1b4941ab07 in pt_TimedWait () from /lib64/libnspr4.so >>> No symbol table info available. >>> #2 0x00007f1b4941afce in PR_WaitCondVar () from /lib64/libnspr4.so >>> No symbol table info available. >>> #3 0x00007f1b4b719d03 in housecleaning () >>> No symbol table info available. >>> #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so >>> No symbol table info available. >>> #5 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 >>> No symbol table info available. >>> #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 >>> No symbol table info available. >>> >>> Thread 8 (Thread 0x7f1b2609f700 (LWP 105166)): >>> #0 0x00007f1b48dc56d5 in >>> pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 >>> No symbol table info available. >>> #1 0x00007f1b4941b050 in PR_WaitCondVar () from /lib64/libnspr4.so >>> No symbol table info available. >>> #2 0x00007f1b4b260198 in slapi_wait_condvar () from >>> /usr/lib64/dirsrv/libslapd.so.0 >>> No symbol table info available. >>> #3 0x00007f1b3cf5deed in roles_cache_wait_on_change () from >>> /usr/lib64/dirsrv/plugins/libroles-plugin.so >>> No symbol table info available. >>> #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so >>> No symbol table info available. >>> #5 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 >>> No symbol table info available. >>> #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 >>> No symbol table info available. >>> >>> Thread 7 (Thread 0x7f1b268a0700 (LWP 105165)): >>> #0 0x00007f1b48dc56d5 in >>> pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 >>> No symbol table info available. >>> #1 0x00007f1b4941b050 in PR_WaitCondVar () from /lib64/libnspr4.so >>> No symbol table info available. >>> #2 0x00007f1b4b260198 in slapi_wait_condvar () from >>> /usr/lib64/dirsrv/libslapd.so.0 >>> No symbol table info available. >>> #3 0x00007f1b3cf5deed in roles_cache_wait_on_change () from >>> /usr/lib64/dirsrv/plugins/libroles-plugin.so >>> No symbol table info available. >>> #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so >>> No symbol table info available. >>> #5 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 >>> No symbol table info available. >>> #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 >>> No symbol table info available. >>> >>> Thread 6 (Thread 0x7f1b270a1700 (LWP 105164)): >>> #0 0x00007f1b48dc56d5 in >>> pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 >>> No symbol table info available. >>> #1 0x00007f1b4941b050 in PR_WaitCondVar () from /lib64/libnspr4.so >>> No symbol table info available. >>> #2 0x00007f1b4b260198 in slapi_wait_condvar () from >>> /usr/lib64/dirsrv/libslapd.so.0 >>> No symbol table info available. >>> #3 0x00007f1b3cf5deed in roles_cache_wait_on_change () from >>> /usr/lib64/dirsrv/plugins/libroles-plugin.so >>> No symbol table info available. >>> #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so >>> No symbol table info available. >>> #5 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 >>> No symbol table info available. >>> #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 >>> No symbol table info available. >>> >>> Thread 5 (Thread 0x7f1b278a2700 (LWP 105163)): >>> #0 0x00007f1b48dc56d5 in >>> pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 >>> No symbol table info available. >>> #1 0x00007f1b4941b050 in PR_WaitCondVar () from /lib64/libnspr4.so >>> No symbol table info available. >>> #2 0x00007f1b4b260198 in slapi_wait_condvar () from >>> /usr/lib64/dirsrv/libslapd.so.0 >>> No symbol table info available. >>> #3 0x00007f1b415f262e in cos_cache_wait_on_change () from >>> /usr/lib64/dirsrv/plugins/libcos-plugin.so >>> No symbol table info available. >>> #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so >>> No symbol table info available. >>> #5 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 >>> No symbol table info available. >>> #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 >>> No symbol table info available. >>> >>> Thread 4 (Thread 0x7f1b282ac700 (LWP 105162)): >>> #0 0x00007f1b48ae69b3 in select () from /lib64/libc.so.6 >>> No symbol table info available. >>> #1 0x00007f1b4b2710e9 in DS_Sleep () from /usr/lib64/dirsrv/libslapd.so. >>> 0 >>> No symbol table info available. >>> #2 0x00007f1b3e8ba707 in perf_threadmain () from >>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>> No symbol table info available. >>> #3 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so >>> No symbol table info available. >>> #4 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 >>> No symbol table info available. >>> #5 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 >>> No symbol table info available. >>> >>> Thread 3 (Thread 0x7f1b28aad700 (LWP 105161)): >>> #0 0x00007f1b48ae69b3 in select () from /lib64/libc.so.6 >>> No symbol table info available. >>> #1 0x00007f1b4b2710e9 in DS_Sleep () from /usr/lib64/dirsrv/libslapd.so. >>> 0 >>> No symbol table info available. >>> #2 0x00007f1b3e8bfb7f in trickle_threadmain () from >>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>> No symbol table info available. >>> #3 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so >>> No symbol table info available. >>> #4 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 >>> No symbol table info available. >>> #5 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 >>> No symbol table info available. >>> >>> Thread 2 (Thread 0x7f1b292ae700 (LWP 105160)): >>> #0 0x00007f1b48ae69b3 in select () from /lib64/libc.so.6 >>> No symbol table info available. >>> #1 0x00007f1b4344dbad in __os_yield () from /lib64/libdb-5.3.so >>> No symbol table info available. >>> #2 0x00007f1b434492b3 in __memp_sync_int () from /lib64/libdb-5.3.so >>> No symbol table info available. >>> #3 0x00007f1b43459752 in __txn_checkpoint () from /lib64/libdb-5.3.so >>> No symbol table info available. >>> #4 0x00007f1b43459b74 in __txn_checkpoint_pp () from /lib64/ >>> libdb-5.3.so >>> No symbol table info available. >>> #5 0x00007f1b3e8c3a87 in checkpoint_threadmain () from >>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>> No symbol table info available. >>> #6 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so >>> No symbol table info available. >>> #7 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 >>> No symbol table info available. >>> #8 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 >>> No symbol table info available. >>> >>> Thread 1 (Thread 0x7f1b29aaf700 (LWP 105159)): >>> #0 0x00007f1b48ae69b3 in select () from /lib64/libc.so.6 >>> No symbol table info available. >>> #1 0x00007f1b4b2710e9 in DS_Sleep () from /usr/lib64/dirsrv/libslapd.so. >>> 0 >>> No symbol table info available. >>> #2 0x00007f1b3e8bf907 in deadlock_threadmain () from >>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>> No symbol table info available. >>> #3 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so >>> No symbol table info available. >>> #4 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 >>> No symbol table info available. >>> #5 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 >>> No symbol table info available. >>> >>> >>> >>> Thanks, >>> >>> Rakesh >>> >>> >>> On Mon, Sep 5, 2016 at 4:48 PM, thierry bordaz < >>> tbordaz at redhat.com> wrote: >>> >>>> >>>> >>>> On 09/05/2016 12:05 PM, Rakesh Rajasekharan wrote: >>>> >>>> Hi Thierry, >>>> >>>> >>>> I was getting the hang issue while running ipa-client-install >>>> simultaneously on few clients.. >>>> However, today, I am not able to replicate that. >>>> >>>> I could not get a gdb . But i will try getting that the next time I >>>> face this issue. >>>> >>>> The CPU does not stay high.. it just momentarily touches a high value >>>> and then drops down to around 2-7% >>>> >>>> One question I have is , is it ok to set it nsslapd-threadnumber to a >>>> very high value . >>>> I have around 4000 clients and with nsslapd-maxthreadsperconn set to >>>> 5..So, can I set nsslapd-threadnumber to around 25000. >>>> >>>> >>>> Hello, >>>> >>>> I know some users running in production with several hunderds of >>>> threads (>600) and this without problem. >>>> >>>> I do not recall having suggested to increase that number and for what >>>> reason. >>>> Usually 30 workers is a good enough value. It can create bootleneck if >>>> for some reason each operation is very long to satisfy and exhaust the >>>> number of workers. You can monitor the work queue: >>>> >>>> ldapsearch -D "cn=directory manager" -w xxx -LLL -b "cn=monitor" -s >>>> base opsinitiated opscompleted >>>> >>>> >>>> If opscompleted-opsinitiated remains close to threadnumber, then yes it >>>> would be valuable to increase it. >>>> >>>> The computation #client * #async_op_per_client sound an overkill. Even >>>> if all clients send at the exact same time all their requests, it is very >>>> likely that some common resource (db page, log, allocator...) will >>>> serialize them. If you monitor a need to increase the work, you would for >>>> example set it to 50, then monitor, then set it to 100, then monitor... >>>> until you find a good enough value. >>>> Note the increasing the #thread, increases the memory footprint that >>>> will reduce the efficiency of file system cache and can increase the >>>> response time. >>>> >>>> >>>> best regards >>>> thierry >>>> >>>> >>>> Thanks >>>> >>>> On Mon, Sep 5, 2016 at 1:03 PM, thierry bordaz < >>>> tbordaz at redhat.com> wrote: >>>> >>>>> >>>>> Hi Rakesh, >>>>> >>>>> Were you able to get a pstack or full stack with gdb ( >>>>> http://www.port389.org/docs/3 >>>>> 89ds/FAQ/faq.html#debugging-crashes) when the server hangs ? >>>>> >>>>> If it happens with 500 threads as well as with 30, using 30 threads is >>>>> a better choice to debug this issue. >>>>> I will try to reproduce using 150 parallel 'ipa user-find p-testipa' >>>>> commands >>>>> >>>>> Something I am unsure is if the CPU consumption stays high (you >>>>> mentioned 340% CPU usage) as long as the hang happens or if after a suddent >>>>> shot up to 340% (that marks the beginning of the hang) it drops and stay >>>>> hanging ? >>>>> >>>>> thanks >>>>> thierry >>>>> >>>>> On 09/04/2016 08:40 PM, Rakesh Rajasekharan wrote: >>>>> >>>>> starce on the slapd process actually had this in the output.. >>>>> FUTEX_WAIT_PRIVATE >>>>> >>>>> and checking for the number of threads slapd had.. there were 5015 >>>>> threads >>>>> >>>>> ps -efL|grep slapd|wc -l >>>>> 5015 >>>>> >>>>> strace on most of the threads gave this output >>>>> >>>>> strace -p 67411 >>>>> Process 67411 attached >>>>> futex(0x7f3f0226b41c, FUTEX_WAIT_PRIVATE, 1, NULL) = -1 EAGAIN >>>>> (Resource temporarily unavailable) >>>>> futex(0x7f3f0226b41c, FUTEX_WAIT_PRIVATE, 2, NULL^CProcess 67411 >>>>> detached >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> On Sun, Sep 4, 2016 at 5:34 PM, Rakesh Rajasekharan < >>>>> rakesh.rajasekharan at gmail.com> wrote: >>>>> >>>>>> I have again got the issue of IPA hanging.. The issue came up when i >>>>>> tried to run ipa-client-isntall on 142 clients simultaneously >>>>>> >>>>>> >>>>>> None of the IPA commands are responding, and I see this error >>>>>> >>>>>> ipa user-find p-testipa >>>>>> ipa: ERROR: Insufficient access: SASL(-1): generic failure: GSSAPI >>>>>> Error: Unspecified GSS failure. Minor code may provide more information >>>>>> (KDC returned error string: PROCESS_TGS) >>>>>> >>>>>> KRB5_TRACE=/dev/stdout kinit admin >>>>>> [41178] 1472984115.233214: Getting initial credentials for >>>>>> admin at XYZ.COM >>>>>> [41178] 1472984115.235257: Sending request (167 bytes) to XYZ.COM >>>>>> [41178] 1472984115.235419: Initiating TCP connection to stream >>>>>> 10.1.3.36:88 >>>>>> [41178] 1472984115.235685: Sending TCP request to stream 10.1.3.36:88 >>>>>> [41178] 1472984120.238914: Received answer (174 bytes) from stream >>>>>> 10.1.3.36:88 >>>>>> [41178] 1472984120.238925: Terminating TCP connection to stream >>>>>> 10.1.3.36:88 >>>>>> [41178] 1472984120.238993: Response was from master KDC >>>>>> [41 >>>>>> >>>>>> >>>>>> Running an ldapsearch to see the db.. does not give any results and >>>>>> just hangs there >>>>>> >>>>>> ldapsearch -x -D 'cn=Directory Manager' -W -s one -b >>>>>> 'cn=kerberos,dc=xyz,dc=com' >>>>>> Enter LDAP Password: >>>>>> >>>>>> even an ldapsearch -x does not respond >>>>>> At this point, am sure that slapd is the one causing issues >>>>>> >>>>>> Running an strace against the hung slapd itself seems to get stuck >>>>>> does not proceed after saying "attaching to process" >>>>>> >>>>>> From some others posts I read Thierry suggesting to increase the >>>>>> nsslapd-threadnumber value >>>>>> >>>>>> It was set to 30, I think that might be too low. >>>>>> >>>>> ... >>> >>> [Message clipped] >> >> >> > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From lkrispen at redhat.com Tue Sep 6 10:43:50 2016 From: lkrispen at redhat.com (Ludwig Krispenz) Date: Tue, 06 Sep 2016 12:43:50 +0200 Subject: [Freeipa-users] Freeipa 4.2.0 hangs intermittently In-Reply-To: <57CE8CDB.6000809@redhat.com> References: <57BEB0AB.2090506@redhat.com> <57C44AC6.1030402@redhat.com> <57CD1FDC.2010109@redhat.com> <57CD5488.60404@redhat.com> <57CD659B.6050707@redhat.com> <57CD68BD.9030802@redhat.com> <57CE8CDB.6000809@redhat.com> Message-ID: <57CE9DE5.6020301@redhat.com> Hi, I think you are running into this bug: https://bugzilla.redhat.com/show_bug.cgi?id=1349779 it was the outcome of an analysis started in this one: https://bugzilla.redhat.com/show_bug.cgi?id=1347800 the problem is that it looks like the checkpoint thread is holding a mutex and otehr threads are blocked on it. It looks like an issue in BerkeleyDB although there is no explanation from the BDB code why this could happen, we also had a support case with Oracle to investigate this, with no solution. Oracle at the end was providing a diagnostic patch, which we did not test since the customer with the original problem had worked around this by changing their configuration. I don't know the status of this patch and if we can make it available, probably depends on your support status, you need to contact support for this. About the workarounds: you can try to change timing for the database update and checkpointing eg by reducing the dbcache size, changing the checkpoint interval. In the customer case they identified one substring index, which was frequently updated but not really required - and removing this index helped. Regards, Ludwig On 09/06/2016 11:31 AM, thierry bordaz wrote: > Hi, > > The following thread is hanging most of the others threads > > Thread 20 (Thread 0x7f68007f8700 (LWP 4873)): > #0 0x00007f683d0506d5 in pthread_cond_wait@@GLIBC_2.3.2 () from > /lib64/libpthread.so.0 > #1 0x00007f68375ac2f3 in __db_hybrid_mutex_suspend () from > /lib64/libdb-5.3.so > #2 0x00007f68375ab640 in __db_tas_mutex_lock () from > /lib64/libdb-5.3.so > #3 0x00007f68376c5e1b in __memp_fget () from /lib64/libdb-5.3.so > #4 0x00007f68375c9211 in __bam_search () from /lib64/libdb-5.3.so > #5 0x00007f68375b4256 in __bamc_search () from /lib64/libdb-5.3.so > #6 0x00007f68375b86c4 in __bamc_put () from /lib64/libdb-5.3.so > #7 0x00007f683766f6d5 in __dbc_iput () from /lib64/libdb-5.3.so > #8 0x00007f683766aa0e in __db_put () from /lib64/libdb-5.3.so > #9 0x00007f683767ffa4 in __db_put_pp () from /lib64/libdb-5.3.so > #10 0x00007f6832b5a4cf in idl_new_insert_key () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #11 0x00007f6832b695b5 in addordel_values_sv.isra.3 () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #12 0x00007f6832b6a25a in index_addordel_values_ext_sv () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #13 0x00007f6832b6a694 in index_addordel_values_sv () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #14 0x00007f6832b6a75a in index_addordel_entry () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #15 0x00007f6832b6d5fa in ldbm_back_add () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #16 0x00007f683f46fcca in op_shared_add () from > /usr/lib64/dirsrv/libslapd.so.0 > #17 0x00007f683f471020 in do_add () from > /usr/lib64/dirsrv/libslapd.so.0 > #18 0x00007f683f99c303 in connection_threadmain () > #19 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so > #20 0x00007f683d04cdc5 in start_thread () from /lib64/libpthread.so.0 > #21 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 > > This thread is likely not hang on 'db_tas_mutex_lock' as the db_lock > does not show it waiting for something. > Unfortunately it is not shown in the output of top -H, did you get the > complete output ? > Ludwig mentioned a known issue that DS is not scaling well on NUMA > arch, do you know if the machine is NUMA ? > A possibility is that this thread get severely slowed down by a NUMA > remote lock and showing DS has hung. > This is why it would be interesting to know if it consums CPU. > > thanks > theirry > > On 09/06/2016 10:59 AM, Rakesh Rajasekharan wrote: >> Hi Thierry/Ludwig, >> >> I again hit the hang issue. This is the pstack of the ns-slapd >> >> So was able to this time get the pstack and db locks >> >> db locks details >> Default locking region information: >> 451 Last allocated locker ID >> 0x7fffffff Current maximum unused locker ID >> 9 Number of lock modes >> 800 Initial number of locks allocated >> 0 Initial number of lockers allocated >> 800 Initial number of lock objects allocated >> 10000 Maximum number of locks possible >> 10000 Maximum number of lockers possible >> 10000 Maximum number of lock objects possible >> 800 Current number of locks allocated >> 366 Current number of lockers allocated >> 800 Current number of lock objects allocated >> 160 Number of lock object partitions >> 8191 Size of object hash table >> 126 Number of current locks >> 1005 Maximum number of locks at any one time >> 33 Maximum number of locks in any one bucket >> 3987 Maximum number of locks stolen by for an empty partition >> 75 Maximum number of locks stolen for any one partition >> 309 Number of current lockers >> 313 Maximum number of lockers at any one time >> 121 Number of current lock objects >> 676 Maximum number of lock objects at any one time >> 3 Maximum number of lock objects in any one bucket >> 69 Maximum number of objects stolen by for an empty partition >> 5 Maximum number of objects stolen for any one partition >> 22M Total number of locks requested (22009642) >> 21M Total number of locks released (21992878) >> 0 Total number of locks upgraded >> 95 Total number of locks downgraded >> 3348 Lock requests not available due to conflicts, for which we >> waited >> 12646 Lock requests not available due to conflicts, for which we >> did not wait >> 0 Number of deadlocks >> 0 Lock timeout value >> 0 Number of locks that have timed out >> 0 Transaction timeout value >> 0 Number of transactions that have timed out >> 2MB 560KB Region size >> 71719 The number of partition locks that required waiting (0%) >> 23105 The maximum number of times any partition lock was waited >> for (0%) >> 0 The number of object queue operations that required >> waiting (0%) >> 163 The number of locker allocations that required waiting (0%) >> 3172 The number of region locks that required waiting (0%) >> 5 Maximum hash bucket length >> =-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-= >> Lock REGINFO information: >> Environment Region type >> 1 Region ID >> /var/lib/dirsrv/slapd-SPRINKLR-COM/db/__db.001 Region name >> 0x7f67ef869000 Region address >> 0x7f67ef8690a0 Region allocation head >> 0x7f67ef8712b0 Region primary address >> 0 Region maximum allocation >> 0 Region allocated >> Region allocations: 1581345 allocations, 0 failures, 1580794 frees, 6 >> longest >> Allocations by power-of-two sizes: >> 1KB 1581329 >> 2KB 3 >> 4KB 4 >> 8KB 3 >> 16KB 3 >> 32KB 1 >> 64KB 0 >> 128KB 0 >> 256KB 2 >> 512KB 0 >> 1024KB 1 >> REGION_SHARED Region flags >> =-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-= >> Lock region parameters: >> 2 Lock region region mutex [3172/14M 0% !Own] >> 16381 locker table size >> 8191 object table size >> 34128 obj_off >> 900216 locker_off >> 0 need_dd >> =-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-= >> Lock conflict matrix: >> =-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-= >> Locks grouped by lockers: >> Locker Mode Count Status ----------------- Object --------------- >> 2 dd=308 locks held 1 write locks 0 pid/thread >> 4845/140085720139840 flags 10 priority 100 >> 2 READ 1 HELD userRoot/id2entry.db handle 0 >> 3 dd=307 locks held 0 write locks 0 pid/thread >> 4845/140084829472512 flags 0 priority 100 >> 4 dd=306 locks held 1 write locks 0 pid/thread >> 4845/140085720139840 flags 10 priority 100 >> 4 READ 1 HELD ipaca/id2entry.db handle 0 >> 5 dd=305 locks held 0 write locks 0 pid/thread >> 4845/140084837865216 flags 0 priority 100 >> 6 dd=304 locks held 1 write locks 0 pid/thread >> 4845/140085720139840 flags 10 priority 100 >> 6 READ 1 HELD ipaca/entryrdn.db handle 0 >> 7 dd=303 locks held 0 write locks 0 pid/thread >> 4845/140084804294400 flags 0 priority 100 >> 8 dd=302 locks held 0 write locks 0 pid/thread >> 4845/140084720432896 flags 0 priority 100 >> 9 dd=301 locks held 0 write locks 0 pid/thread >> 4845/140084837865216 flags 0 priority 100 >> a dd=300 locks held 1 write locks 0 pid/thread >> 4845/140085720139840 flags 10 priority 100 >> a READ 1 HELD ipaca/vlv#allcertspkitomcatindex.db >> handle 0 >> e dd=299 locks held 1 write locks 0 pid/thread >> 4845/140085720139840 flags 10 priority 100 >> e READ 1 HELD >> ipaca/vlv#allnonrevokedcertspkitomcatindex.db handle 0 >> 10 dd=298 locks held 1 write locks 0 pid/thread >> 4845/140085720139840 flags 10 priority 100 >> 10 READ 1 HELD >> ipaca/vlv#allrevokedcertspkitomcatindex.db handle 0 >> 11 dd=297 locks held 1 write locks 0 pid/thread >> 4845/140085720139840 flags 10 priority 100 >> 11 READ 1 HELD >> ipaca/vlv#allrevokedcertsnotafterpkitomcatindex.db handle 0 >> 14 dd=296 locks held 1 write locks 0 pid/thread >> 4845/140085720139840 flags 10 priority 100 >> 14 READ 1 HELD >> ipaca/vlv#allrevokedorrevokedexpiredcertspkitomcatindex.db >> handle 0 >> 15 dd=295 locks held 1 write locks 0 pid/thread >> 4845/140085720139840 flags 10 priority 100 >> 15 READ 1 HELD >> ipaca/vlv#allvalidcertspkitomcatindex.db handle 0 >> 16 dd=294 locks held 1 write locks 0 pid/thread >> 4845/140085720139840 flags 10 priority 100 >> 16 READ 1 HELD >> ipaca/vlv#allvalidcertsnotafterpkitomcatindex.db handle 0 >> 17 dd=293 locks held 1 write locks 0 pid/thread >> 4845/140085720139840 flags 10 priority 100 >> 17 READ 1 HELD >> ipaca/vlv#allvalidorrevokedcertspkitomcatindex.db handle 0 >> 18 dd=292 locks held 1 write locks 0 pid/thread >> 4845/140085720139840 flags 10 priority 100 >> 18 READ 1 HELD ipaca/vlv#caallpkitomcatindex.db >> handle 0 >> 1d dd=291 locks held 1 write locks 0 pid/thread >> 4845/140085720139840 flags 10 priority 100 >> 1d READ 1 HELD ipaca/vlv#cacompletepkitomcatindex.db >> handle 0 >> 1e dd=290 locks held 1 write locks 0 pid/thread >> 4845/140085720139840 flags 10 priority 100 >> 1e READ 1 HELD >> ipaca/vlv#cacompleteenrollmentpkitomcatindex.db handle 0 >> 20 dd=289 locks held 1 write locks 0 pid/thread >> 4845/140085720139840 flags 10 priority 100 >> 20 READ 1 HELD >> ipaca/vlv#cacompleterevocationpkitomcatindex.db handle 0 >> 21 dd=288 locks held 1 write locks 0 pid/thread >> 4845/140085720139840 flags 10 priority 100 >> 21 READ 1 HELD ipaca/vlv#caenrollmentpkitomcatindex.db >> handle 0 >> 2b dd=287 locks held 1 write locks 0 pid/thread >> 4845/140085720139840 flags 10 priority 100 >> 2b READ 1 HELD ipaca/vlv#carevocationpkitomcatindex.db >> handle 0 >> 2c dd=286 locks held 1 write locks 0 pid/thread >> 4845/140085720139840 flags 10 priority 100 >> 2c READ 1 HELD changelog/id2entry.db handle 0 >> 2d dd=285 locks held 0 write locks 0 pid/thread >> 4845/140085719607040 flags 0 priority 100 >> 2e dd=284 locks held 1 write locks 0 pid/thread >> 4845/140085720139840 flags 10 priority 100 >> 2e READ 1 HELD changelog/entryusn.db handle 0 >> 2f dd=283 locks held 0 write locks 0 pid/thread >> 4845/140085720139840 flags 0 priority 100 >> 30 dd=282 locks held 1 write locks 0 pid/thread >> 4845/140085720139840 flags 10 priority 100 >> 30 READ 1 HELD userRoot/entryusn.db handle 0 >> 31 dd=281 locks held 0 write locks 0 pid/thread >> 4845/140085720139840 flags 0 priority 100 >> 32 dd=280 locks held 1 write locks 0 pid/thread >> 4845/140085720139840 flags 10 priority 100 >> 32 READ 1 HELD ipaca/entryusn.db handle 0 >> 33 dd=279 locks held 0 write locks 0 pid/thread >> 4845/140085720139840 flags 0 priority 100 >> 34 dd=278 locks held 1 write locks 0 pid/thread >> 4845/140085720139840 flags 10 priority 100 >> 34 READ 1 HELD userRoot/entryrdn.db handle 0 >> 35 dd=277 locks held 0 write locks 0 pid/thread >> 4845/140084544186112 flags 0 priority 100 >> 36 dd=276 locks held 0 write locks 0 pid/thread >> 4845/140084636505856 flags 0 priority 100 >> 37 dd=275 locks held 0 write locks 0 pid/thread >> 4845/140084854650624 flags 0 priority 100 >> 38 dd=274 locks held 1 write locks 0 pid/thread >> 4845/140085720139840 flags 10 priority 100 >> 38 READ 1 HELD userRoot/objectclass.db handle 0 >> 39 dd=273 locks held 0 write locks 0 pid/thread >> 4845/140084569364224 flags 0 priority 100 >> 3a dd=272 locks held 1 write locks 0 pid/thread >> 4845/140085720139840 flags 10 priority 100 >> 3a READ 1 HELD userRoot/ancestorid.db handle 0 >> 3b dd=271 locks held 0 write locks 0 pid/thread >> 4845/140084854650624 flags 0 priority 100 >> 3c dd=270 locks held 1 write locks 0 pid/thread >> 4845/140085720139840 flags 10 priority 100 >> 3c READ 1 HELD changelog/entryrdn.db handle 0 >> 3d dd=269 locks held 0 write locks 0 pid/thread >> 4845/140085719607040 flags 0 priority 100 >> 3e dd=268 locks held 0 write locks 0 pid/thread >> 4845/140084602935040 flags 0 priority 100 >> 3f dd=267 locks held 0 write locks 0 pid/thread >> 4845/140085719607040 flags 0 priority 100 >> 40 dd=266 locks held 1 write locks 0 pid/thread >> 4845/140085720139840 flags 10 priority 100 >> 40 READ 1 HELD changelog/objectclass.db handle 0 >> 41 dd=265 locks held 0 write locks 0 pid/thread >> 4845/140085719607040 flags 0 priority 100 >> 42 dd=264 locks held 0 write locks 0 pid/thread >> 4845/140085719607040 flags 0 priority 100 >> 43 dd=263 locks held 0 write locks 0 pid/thread >> 4845/140084653291264 flags 0 priority 100 >> 44 dd=262 locks held 1 write locks 0 pid/thread >> 4845/140085720139840 flags 10 priority 100 >> 44 READ 1 HELD ipaca/objectclass.db handle 0 >> 45 dd=261 locks held 0 write locks 0 pid/thread >> 4845/140084577756928 flags 0 priority 100 >> 46 dd=260 locks held 0 write locks 0 pid/thread >> 4845/140084720432896 flags 0 priority 100 >> 47 dd=259 locks held 1 write locks 0 pid/thread >> 4845/140085720139840 flags 10 priority 100 >> 47 READ 1 HELD changelog/aci.db handle 0 >> 48 dd=258 locks held 0 write locks 0 pid/thread >> 4845/140085720139840 flags 0 priority 100 >> 49 dd=257 locks held 1 write locks 0 pid/thread >> 4845/140085720139840 flags 10 priority 100 >> 49 READ 1 HELD userRoot/aci.db handle 0 >> 4a dd=256 locks held 0 write locks 0 pid/thread >> 4845/140085720139840 flags 0 priority 100 >> 4b dd=255 locks held 0 write locks 0 pid/thread >> 4845/140084535793408 flags 0 priority 100 >> 4c dd=254 locks held 1 write locks 0 pid/thread >> 4845/140085720139840 flags 10 priority 100 >> 4c READ 1 HELD ipaca/aci.db handle 0 >> 4d dd=253 locks held 0 write locks 0 pid/thread >> 4845/140085720139840 flags 0 priority 100 >> 4e dd=252 locks held 1 write locks 0 pid/thread >> 4845/140085720139840 flags 10 priority 100 >> 4e READ 1 HELD userRoot/parentid.db handle 0 >> 4f dd=251 locks held 0 write locks 0 pid/thread >> 4845/140085720139840 flags 0 priority 100 >> 50 dd=250 locks held 0 write locks 0 pid/thread >> 4845/140085719607040 flags 0 priority 100 >> 51 dd=249 locks held 1 write locks 0 pid/thread >> 4845/140085720139840 flags 10 priority 100 >> 51 READ 1 HELD changelog/nsuniqueid.db handle 0 >> 52 dd=248 locks held 1 write locks 0 pid/thread >> 4845/140085720139840 flags 10 priority 100 >> 52 READ 1 HELD changelog/changenumber.db handle 0 >> 53 dd=247 locks held 0 write locks 0 pid/thread >> 4845/140084602935040 flags 0 priority 100 >> 54 dd=246 locks held 0 write locks 0 pid/thread >> 4845/140085719607040 flags 0 priority 100 >> 55 dd=245 locks held 1 write locks 0 pid/thread >> 4845/140085719607040 flags 10 priority 100 >> 55 READ 1 HELD changelog/targetuniqueid.db handle 0 >> 56 dd=244 locks held 1 write locks 0 pid/thread >> 4845/140085719607040 flags 10 priority 100 >> 56 READ 1 HELD changelog/parentid.db handle 0 >> 57 dd=243 locks held 1 write locks 0 pid/thread >> 4845/140085719607040 flags 10 priority 100 >> 57 READ 1 HELD changelog/ancestorid.db handle 0 >> 58 dd=242 locks held 1 write locks 0 pid/thread >> 4845/140085719607040 flags 10 priority 100 >> 58 READ 1 HELD changelog/numsubordinates.db >> handle 0 >> 59 dd=241 locks held 0 write locks 0 pid/thread >> 4845/140085719607040 flags 0 priority 100 >> 5a dd=240 locks held 0 write locks 0 pid/thread >> 4845/140084544186112 flags 0 priority 100 >> 5b dd=239 locks held 1 write locks 0 pid/thread >> 4845/140084577756928 flags 10 priority 100 >> 5b READ 1 HELD userRoot/krbPrincipalName.db >> handle 0 >> 5c dd=238 locks held 0 write locks 0 pid/thread >> 4845/140084636505856 flags 0 priority 100 >> 5d dd=237 locks held 0 write locks 0 pid/thread >> 4845/140084695254784 flags 0 priority 100 >> 5e dd=236 locks held 1 write locks 0 pid/thread >> 4845/140084560971520 flags 10 priority 100 >> 5e READ 1 HELD userRoot/ipakrbprincipalalias.db >> handle 0 >> 5f dd=235 locks held 0 write locks 0 pid/thread >> 4845/140084703647488 flags 0 priority 100 >> 60 dd=234 locks held 0 write locks 0 pid/thread >> 4845/140084653291264 flags 0 priority 100 >> 61 dd=233 locks held 0 write locks 0 pid/thread >> 4845/140084577756928 flags 0 priority 100 >> 62 dd=232 locks held 0 write locks 0 pid/thread >> 4845/140084686862080 flags 0 priority 100 >> 63 dd=231 locks held 1 write locks 0 pid/thread >> 4845/140084611327744 flags 10 priority 100 >> 63 READ 1 HELD changelog/seeAlso.db handle 0 >> 64 dd=230 locks held 0 write locks 0 pid/thread >> 4845/140084686862080 flags 0 priority 100 >> 65 dd=229 locks held 1 write locks 0 pid/thread >> 4845/140084611327744 flags 10 priority 100 >> 65 READ 1 HELD userRoot/seeAlso.db handle 0 >> 66 dd=228 locks held 0 write locks 0 pid/thread >> 4845/140084636505856 flags 0 priority 100 >> 67 dd=227 locks held 1 write locks 0 pid/thread >> 4845/140084611327744 flags 10 priority 100 >> 67 READ 1 HELD ipaca/seeAlso.db handle 0 >> 68 dd=226 locks held 0 write locks 0 pid/thread >> 4845/140084686862080 flags 0 priority 100 >> 69 dd=225 locks held 1 write locks 0 pid/thread >> 4845/140084678469376 flags 10 priority 100 >> 69 READ 1 HELD ipaca/parentid.db handle 0 >> 6a dd=224 locks held 0 write locks 0 pid/thread >> 4845/140084678469376 flags 0 priority 100 >> 6b dd=223 locks held 0 write locks 0 pid/thread >> 4845/140084829472512 flags 0 priority 100 >> 6c dd=222 locks held 0 write locks 0 pid/thread >> 4845/140084653291264 flags 0 priority 100 >> 6d dd=221 locks held 0 write locks 0 pid/thread >> 4845/140084569364224 flags 0 priority 100 >> 6e dd=220 locks held 0 write locks 0 pid/thread >> 4845/140084636505856 flags 0 priority 100 >> 6e READ 1 WAIT userRoot/objectclass.db page 18 >> 6f dd=219 locks held 0 write locks 0 pid/thread >> 4845/140084544186112 flags 0 priority 100 >> 70 dd=218 locks held 0 write locks 0 pid/thread >> 4845/140084569364224 flags 0 priority 100 >> 71 dd=217 locks held 0 write locks 0 pid/thread >> 4845/140084619720448 flags 0 priority 100 >> 72 dd=216 locks held 0 write locks 0 pid/thread >> 4845/140084552578816 flags 0 priority 100 >> 73 dd=215 locks held 0 write locks 0 pid/thread >> 4845/140084829472512 flags 0 priority 100 >> 74 dd=214 locks held 0 write locks 0 pid/thread >> 4845/140084628113152 flags 0 priority 100 >> 75 dd=213 locks held 0 write locks 0 pid/thread >> 4845/140084686862080 flags 0 priority 100 >> 76 dd=212 locks held 1 write locks 0 pid/thread >> 4845/140084560971520 flags 10 priority 100 >> 76 READ 1 HELD userRoot/ipauniqueid.db handle 0 >> 77 dd=211 locks held 0 write locks 0 pid/thread >> 4845/140084661683968 flags 0 priority 100 >> 78 dd=210 locks held 0 write locks 0 pid/thread >> 4845/140084846257920 flags 0 priority 100 >> 79 dd=209 locks held 0 write locks 0 pid/thread >> 4845/140084628113152 flags 0 priority 100 >> 7a dd=208 locks held 1 write locks 0 pid/thread >> 4845/140084560971520 flags 10 priority 100 >> 7a READ 1 HELD userRoot/nsuniqueid.db handle 0 >> 7b dd=207 locks held 1 write locks 0 pid/thread >> 4845/140084560971520 flags 10 priority 100 >> 7b READ 1 HELD userRoot/cn.db handle 0 >> 7c dd=206 locks held 1 write locks 0 pid/thread >> 4845/140084560971520 flags 10 priority 100 >> 7c READ 1 HELD userRoot/fqdn.db handle 0 >> 7d dd=205 locks held 1 write locks 0 pid/thread >> 4845/140084560971520 flags 10 priority 100 >> 7d READ 1 HELD userRoot/managedby.db handle 0 >> 7e dd=204 locks held 1 write locks 0 pid/thread >> 4845/140084560971520 flags 10 priority 100 >> 7e READ 1 HELD userRoot/numsubordinates.db handle 0 >> 7f dd=203 locks held 1 write locks 0 pid/thread >> 4845/140084678469376 flags 10 priority 100 >> 7f READ 1 HELD userRoot/member.db handle 0 >> 80 dd=202 locks held 0 write locks 0 pid/thread >> 4845/140084712040192 flags 0 priority 100 >> 81 dd=201 locks held 1 write locks 0 pid/thread >> 4845/140084678469376 flags 10 priority 100 >> 81 READ 1 HELD userRoot/memberUser.db handle 0 >> 82 dd=200 locks held 0 write locks 0 pid/thread >> 4845/140084577756928 flags 0 priority 100 >> 83 dd=199 locks held 1 write locks 0 pid/thread >> 4845/140084678469376 flags 10 priority 100 >> 83 READ 1 HELD userRoot/memberHost.db handle 0 >> 84 dd=198 locks held 0 write locks 0 pid/thread >> 4845/140084712040192 flags 0 priority 100 >> 85 dd=197 locks held 0 write locks 0 pid/thread >> 4845/140084577756928 flags 0 priority 100 >> 86 dd=196 locks held 0 write locks 0 pid/thread >> 4845/140084670076672 flags 0 priority 100 >> 87 dd=195 locks held 0 write locks 0 pid/thread >> 4845/140084577756928 flags 0 priority 100 >> 88 dd=194 locks held 0 write locks 0 pid/thread >> 4845/140084846257920 flags 0 priority 100 >> 89 dd=193 locks held 0 write locks 0 pid/thread >> 4845/140084552578816 flags 0 priority 100 >> 8a dd=192 locks held 0 write locks 0 pid/thread >> 4845/140084703647488 flags 0 priority 100 >> 8b dd=191 locks held 0 write locks 0 pid/thread >> 4845/140084544186112 flags 0 priority 100 >> 8c dd=190 locks held 0 write locks 0 pid/thread >> 4845/140084846257920 flags 0 priority 100 >> 8d dd=189 locks held 0 write locks 0 pid/thread >> 4845/140084661683968 flags 0 priority 100 >> 8e dd=188 locks held 1 write locks 0 pid/thread >> 4845/140085067675392 flags 10 priority 100 >> 8e READ 1 HELD userRoot/macAddress.db handle 0 >> 8f dd=187 locks held 0 write locks 0 pid/thread >> 4845/140085067675392 flags 0 priority 100 >> 90 dd=186 locks held 0 write locks 0 pid/thread >> 4845/140084703647488 flags 0 priority 100 >> 91 dd=185 locks held 0 write locks 0 pid/thread >> 4845/140084552578816 flags 0 priority 100 >> 92 dd=184 locks held 0 write locks 0 pid/thread >> 4845/140084560971520 flags 0 priority 100 >> 92 READ 1 WAIT userRoot/objectclass.db page 9 >> 93 dd=183 locks held 0 write locks 0 pid/thread >> 4845/140084577756928 flags 0 priority 100 >> 94 dd=182 locks held 0 write locks 0 pid/thread >> 4845/140084821079808 flags 0 priority 100 >> 94 READ 1 WAIT userRoot/objectclass.db page 18 >> 95 dd=181 locks held 0 write locks 0 pid/thread >> 4845/140084544186112 flags 0 priority 100 >> 96 dd=180 locks held 0 write locks 0 pid/thread >> 4845/140084804294400 flags 0 priority 100 >> 97 dd=179 locks held 1 write locks 0 pid/thread >> 4845/140084712040192 flags 10 priority 100 >> 97 READ 1 HELD userRoot/uid.db handle 0 >> 98 dd=178 locks held 0 write locks 0 pid/thread >> 4845/140084552578816 flags 0 priority 100 >> 99 dd=177 locks held 0 write locks 0 pid/thread >> 4845/140084577756928 flags 0 priority 100 >> 9a dd=176 locks held 0 write locks 0 pid/thread >> 4845/140084602935040 flags 0 priority 100 >> 9b dd=175 locks held 1 write locks 0 pid/thread >> 4845/140085076068096 flags 10 priority 100 >> 9b READ 1 HELD userRoot/uniquemember.db handle 0 >> 9c dd=174 locks held 1 write locks 0 pid/thread >> 4845/140085076068096 flags 10 priority 100 >> 9c READ 1 HELD userRoot/owner.db handle 0 >> 9d dd=173 locks held 1 write locks 0 pid/thread >> 4845/140085076068096 flags 10 priority 100 >> 9d READ 1 HELD userRoot/ipaallowedtarget.db >> handle 0 >> 9e dd=172 locks held 1 write locks 0 pid/thread >> 4845/140085076068096 flags 10 priority 100 >> 9e READ 1 HELD userRoot/ipaMemberCertProfile.db >> handle 0 >> 9f dd=171 locks held 1 write locks 0 pid/thread >> 4845/140085076068096 flags 10 priority 100 >> 9f READ 1 HELD userRoot/ipaassignedidview.db >> handle 0 >> a0 dd=170 locks held 1 write locks 0 pid/thread >> 4845/140085076068096 flags 10 priority 100 >> a0 READ 1 HELD userRoot/secretary.db handle 0 >> a1 dd=169 locks held 1 write locks 0 pid/thread >> 4845/140085076068096 flags 10 priority 100 >> a1 READ 1 HELD userRoot/memberdenycmd.db handle 0 >> a2 dd=168 locks held 1 write locks 0 pid/thread >> 4845/140085076068096 flags 10 priority 100 >> a2 READ 1 HELD userRoot/memberallowcmd.db handle 0 >> a3 dd=167 locks held 1 write locks 0 pid/thread >> 4845/140085076068096 flags 10 priority 100 >> a3 READ 1 HELD userRoot/manager.db handle 0 >> a4 dd=166 locks held 1 write locks 0 pid/thread >> 4845/140085076068096 flags 10 priority 100 >> a4 READ 1 HELD userRoot/ipasudorunas.db handle 0 >> a5 dd=165 locks held 1 write locks 0 pid/thread >> 4845/140085076068096 flags 10 priority 100 >> a5 READ 1 HELD userRoot/ipaMemberCa.db handle 0 >> a6 dd=164 locks held 1 write locks 0 pid/thread >> 4845/140085076068096 flags 10 priority 100 >> a6 READ 1 HELD userRoot/ipasudorunasgroup.db >> handle 0 >> a7 dd=163 locks held 1 write locks 0 pid/thread >> 4845/140085076068096 flags 10 priority 100 >> a7 READ 1 HELD userRoot/sourcehost.db handle 0 >> a8 dd=162 locks held 1 write locks 0 pid/thread >> 4845/140085076068096 flags 10 priority 100 >> a8 READ 1 HELD userRoot/memberservice.db handle 0 >> a9 dd=161 locks held 1 write locks 0 pid/thread >> 4845/140085076068096 flags 10 priority 100 >> a9 READ 1 HELD userRoot/ipatokenradiusconfiglink.db >> handle 0 >> aa dd=160 locks held 0 write locks 0 pid/thread >> 4845/140084703647488 flags 0 priority 100 >> ab dd=159 locks held 0 write locks 0 pid/thread >> 4845/140084821079808 flags 0 priority 100 >> ac dd=158 locks held 0 write locks 0 pid/thread >> 4845/140084628113152 flags 0 priority 100 >> ad dd=157 locks held 0 write locks 0 pid/thread >> 4845/140084829472512 flags 0 priority 100 >> ae dd=156 locks held 0 write locks 0 pid/thread >> 4845/140084829472512 flags 0 priority 100 >> af dd=155 locks held 0 write locks 0 pid/thread >> 4845/140084854650624 flags 0 priority 100 >> b0 dd=154 locks held 0 write locks 0 pid/thread >> 4845/140084854650624 flags 0 priority 100 >> b1 dd=153 locks held 0 write locks 0 pid/thread >> 4845/140084837865216 flags 0 priority 100 >> b2 dd=152 locks held 0 write locks 0 pid/thread >> 4845/140084804294400 flags 0 priority 100 >> b3 dd=151 locks held 0 write locks 0 pid/thread >> 4845/140084812687104 flags 0 priority 100 >> b4 dd=150 locks held 0 write locks 0 pid/thread >> 4845/140084804294400 flags 0 priority 100 >> b5 dd=149 locks held 0 write locks 0 pid/thread >> 4845/140084837865216 flags 0 priority 100 >> b6 dd=148 locks held 0 write locks 0 pid/thread >> 4845/140084569364224 flags 0 priority 100 >> b7 dd=147 locks held 0 write locks 0 pid/thread >> 4845/140084829472512 flags 0 priority 100 >> b8 dd=146 locks held 0 write locks 0 pid/thread >> 4845/140084586149632 flags 0 priority 100 >> b9 dd=145 locks held 0 write locks 0 pid/thread >> 4845/140084653291264 flags 0 priority 100 >> ba dd=144 locks held 0 write locks 0 pid/thread >> 4845/140084636505856 flags 0 priority 100 >> bb dd=143 locks held 0 write locks 0 pid/thread >> 4845/140084804294400 flags 0 priority 100 >> bc dd=142 locks held 0 write locks 0 pid/thread >> 4845/140084804294400 flags 0 priority 100 >> bd dd=141 locks held 0 write locks 0 pid/thread >> 4845/140084695254784 flags 0 priority 100 >> be dd=140 locks held 0 write locks 0 pid/thread >> 4845/140084670076672 flags 0 priority 100 >> bf dd=139 locks held 0 write locks 0 pid/thread >> 4845/140084628113152 flags 0 priority 100 >> c0 dd=138 locks held 0 write locks 0 pid/thread >> 4845/140084661683968 flags 0 priority 100 >> c1 dd=137 locks held 0 write locks 0 pid/thread >> 4845/140084712040192 flags 0 priority 100 >> c2 dd=136 locks held 0 write locks 0 pid/thread >> 4845/140084661683968 flags 0 priority 100 >> c3 dd=135 locks held 0 write locks 0 pid/thread >> 4845/140084577756928 flags 0 priority 100 >> c4 dd=134 locks held 0 write locks 0 pid/thread >> 4845/140084854650624 flags 0 priority 100 >> c5 dd=133 locks held 1 write locks 0 pid/thread >> 4845/140084644898560 flags 10 priority 100 >> c5 READ 1 HELD userRoot/uidnumber.db handle 0 >> c6 dd=132 locks held 0 write locks 0 pid/thread >> 4845/140084594542336 flags 0 priority 100 >> c8 dd=131 locks held 1 write locks 0 pid/thread >> 4845/140084678469376 flags 10 priority 100 >> c8 READ 1 HELD ipaca/certstatus.db handle 0 >> c9 dd=130 locks held 0 write locks 0 pid/thread >> 4845/140084720432896 flags 0 priority 100 >> ca dd=129 locks held 0 write locks 0 pid/thread >> 4845/140084619720448 flags 0 priority 100 >> cb dd=128 locks held 0 write locks 0 pid/thread >> 4845/140084821079808 flags 0 priority 100 >> cc dd=127 locks held 0 write locks 0 pid/thread >> 4845/140084594542336 flags 0 priority 100 >> cd dd=126 locks held 0 write locks 0 pid/thread >> 4845/140084560971520 flags 0 priority 100 >> ce dd=125 locks held 1 write locks 0 pid/thread >> 4845/140084837865216 flags 10 priority 100 >> ce READ 1 HELD userRoot/gidnumber.db handle 0 >> cf dd=124 locks held 0 write locks 0 pid/thread >> 4845/140084720432896 flags 0 priority 100 >> d0 dd=123 locks held 0 write locks 0 pid/thread >> 4845/140084569364224 flags 0 priority 100 >> d1 dd=122 locks held 1 write locks 0 pid/thread >> 4845/140084678469376 flags 10 priority 100 >> d1 READ 1 HELD userRoot/nscpEntryDN.db handle 0 >> d2 dd=121 locks held 0 write locks 0 pid/thread >> 4845/140084695254784 flags 0 priority 100 >> d3 dd=120 locks held 0 write locks 0 pid/thread >> 4845/140084569364224 flags 0 priority 100 >> d4 dd=119 locks held 0 write locks 0 pid/thread >> 4845/140084653291264 flags 0 priority 100 >> d5 dd=118 locks held 0 write locks 0 pid/thread >> 4845/140084829472512 flags 0 priority 100 >> d6 dd=117 locks held 0 write locks 0 pid/thread >> 4845/140084661683968 flags 0 priority 100 >> d7 dd=116 locks held 0 write locks 0 pid/thread >> 4845/140084804294400 flags 0 priority 100 >> d8 dd=115 locks held 1 write locks 0 pid/thread >> 4845/140084821079808 flags 10 priority 100 >> d8 READ 1 HELD ipaca/description.db handle 0 >> d9 dd=114 locks held 0 write locks 0 pid/thread >> 4845/140084661683968 flags 0 priority 100 >> da dd=113 locks held 0 write locks 0 pid/thread >> 4845/140084837865216 flags 0 priority 100 >> db dd=112 locks held 1 write locks 0 pid/thread >> 4845/140084678469376 flags 10 priority 100 >> db READ 1 HELD ipaca/cn.db handle 0 >> dc dd=111 locks held 0 write locks 0 pid/thread >> 4845/140084577756928 flags 0 priority 100 >> dd dd=110 locks held 1 write locks 0 pid/thread >> 4845/140084678469376 flags 10 priority 100 >> dd READ 1 HELD ipaca/ancestorid.db handle 0 >> de dd=109 locks held 0 write locks 0 pid/thread >> 4845/140084577756928 flags 0 priority 100 >> df dd=108 locks held 0 write locks 0 pid/thread >> 4845/140084703647488 flags 0 priority 100 >> e0 dd=107 locks held 0 write locks 0 pid/thread >> 4845/140084569364224 flags 0 priority 100 >> e1 dd=106 locks held 0 write locks 0 pid/thread >> 4845/140084569364224 flags 0 priority 100 >> e2 dd=105 locks held 0 write locks 0 pid/thread >> 4845/140084821079808 flags 0 priority 100 >> e3 dd=104 locks held 1 write locks 0 pid/thread >> 4845/140084686862080 flags 10 priority 100 >> e3 READ 1 HELD ipaca/serialno.db handle 0 >> e4 dd=103 locks held 0 write locks 0 pid/thread >> 4845/140084837865216 flags 0 priority 100 >> e5 dd=102 locks held 1 write locks 0 pid/thread >> 4845/140084695254784 flags 10 priority 100 >> e5 READ 1 HELD ipaca/revInfo.db handle 0 >> e6 dd=101 locks held 1 write locks 0 pid/thread >> 4845/140084695254784 flags 10 priority 100 >> e6 READ 1 HELD ipaca/revokedby.db handle 0 >> e7 dd=100 locks held 1 write locks 0 pid/thread >> 4845/140084695254784 flags 10 priority 100 >> e7 READ 1 HELD ipaca/revokedOn.db handle 0 >> e8 dd=99 locks held 0 write locks 0 pid/thread >> 4845/140084829472512 flags 0 priority 100 >> e9 dd=98 locks held 0 write locks 0 pid/thread >> 4845/140084846257920 flags 0 priority 100 >> ea dd=97 locks held 1 write locks 0 pid/thread >> 4845/140084703647488 flags 10 priority 100 >> ea READ 1 HELD ipaca/nsuniqueid.db handle 0 >> eb dd=96 locks held 1 write locks 0 pid/thread >> 4845/140084703647488 flags 10 priority 100 >> eb READ 1 HELD ipaca/requestid.db handle 0 >> ec dd=95 locks held 1 write locks 0 pid/thread >> 4845/140084703647488 flags 10 priority 100 >> ec READ 1 HELD ipaca/requeststate.db handle 0 >> ed dd=94 locks held 1 write locks 0 pid/thread >> 4845/140084703647488 flags 10 priority 100 >> ed READ 1 HELD ipaca/dateOfCreate.db handle 0 >> ee dd=93 locks held 1 write locks 0 pid/thread >> 4845/140084703647488 flags 10 priority 100 >> ee READ 1 HELD ipaca/requesttype.db handle 0 >> ef dd=92 locks held 1 write locks 0 pid/thread >> 4845/140084703647488 flags 10 priority 100 >> ef READ 1 HELD ipaca/numsubordinates.db handle 0 >> f0 dd=91 locks held 1 write locks 0 pid/thread >> 4845/140084611327744 flags 10 priority 100 >> f0 READ 1 HELD userRoot/userCertificate.db handle 0 >> f1 dd=90 locks held 0 write locks 0 pid/thread >> 4845/140084821079808 flags 0 priority 100 >> f2 dd=89 locks held 0 write locks 0 pid/thread >> 4845/140084569364224 flags 0 priority 100 >> f3 dd=88 locks held 0 write locks 0 pid/thread >> 4845/140084602935040 flags 0 priority 100 >> f4 dd=87 locks held 0 write locks 0 pid/thread >> 4845/140084720432896 flags 0 priority 100 >> f5 dd=86 locks held 0 write locks 0 pid/thread >> 4845/140084628113152 flags 0 priority 100 >> f6 dd=85 locks held 0 write locks 0 pid/thread >> 4845/140084703647488 flags 0 priority 100 >> f7 dd=84 locks held 0 write locks 0 pid/thread >> 4845/140084611327744 flags 0 priority 100 >> f8 dd=83 locks held 0 write locks 0 pid/thread >> 4845/140084661683968 flags 0 priority 100 >> f9 dd=82 locks held 0 write locks 0 pid/thread >> 4845/140084837865216 flags 0 priority 100 >> fa dd=81 locks held 0 write locks 0 pid/thread >> 4845/140084544186112 flags 0 priority 100 >> fb dd=80 locks held 0 write locks 0 pid/thread >> 4845/140084703647488 flags 0 priority 100 >> fc dd=79 locks held 0 write locks 0 pid/thread >> 4845/140084544186112 flags 0 priority 100 >> fd dd=78 locks held 0 write locks 0 pid/thread >> 4845/140084535793408 flags 0 priority 100 >> fe dd=77 locks held 0 write locks 0 pid/thread >> 4845/140084586149632 flags 0 priority 100 >> ff dd=76 locks held 0 write locks 0 pid/thread >> 4845/140084569364224 flags 0 priority 100 >> 100 dd=75 locks held 0 write locks 0 pid/thread >> 4845/140084804294400 flags 0 priority 100 >> 101 dd=74 locks held 0 write locks 0 pid/thread >> 4845/140084703647488 flags 0 priority 100 >> 102 dd=73 locks held 0 write locks 0 pid/thread >> 4845/140084703647488 flags 0 priority 100 >> 103 dd=72 locks held 0 write locks 0 pid/thread >> 4845/140084821079808 flags 0 priority 100 >> 104 dd=71 locks held 0 write locks 0 pid/thread >> 4845/140084695254784 flags 0 priority 100 >> 105 dd=70 locks held 0 write locks 0 pid/thread >> 4845/140084544186112 flags 0 priority 100 >> 106 dd=69 locks held 0 write locks 0 pid/thread >> 4845/140084854650624 flags 0 priority 100 >> 107 dd=68 locks held 0 write locks 0 pid/thread >> 4845/140084628113152 flags 0 priority 100 >> 108 dd=67 locks held 0 write locks 0 pid/thread >> 4845/140084569364224 flags 0 priority 100 >> 109 dd=66 locks held 0 write locks 0 pid/thread >> 4845/140084594542336 flags 0 priority 100 >> 10a dd=65 locks held 0 write locks 0 pid/thread >> 4845/140084661683968 flags 0 priority 100 >> 10b dd=64 locks held 0 write locks 0 pid/thread >> 4845/140084653291264 flags 0 priority 100 >> 10c dd=63 locks held 0 write locks 0 pid/thread >> 4845/140084695254784 flags 0 priority 100 >> 10d dd=62 locks held 0 write locks 0 pid/thread >> 4845/140084720432896 flags 0 priority 100 >> 10e dd=61 locks held 0 write locks 0 pid/thread >> 4845/140084619720448 flags 0 priority 100 >> 10f dd=60 locks held 0 write locks 0 pid/thread >> 4845/140084720432896 flags 0 priority 100 >> 110 dd=59 locks held 0 write locks 0 pid/thread >> 4845/140084569364224 flags 0 priority 100 >> 111 dd=58 locks held 0 write locks 0 pid/thread >> 4845/140084695254784 flags 0 priority 100 >> 112 dd=57 locks held 0 write locks 0 pid/thread >> 4845/140084653291264 flags 0 priority 100 >> 113 dd=56 locks held 0 write locks 0 pid/thread >> 4845/140084712040192 flags 0 priority 100 >> 114 dd=55 locks held 0 write locks 0 pid/thread >> 4845/140084686862080 flags 0 priority 100 >> 115 dd=54 locks held 0 write locks 0 pid/thread >> 4845/140084577756928 flags 0 priority 100 >> 115 READ 1 WAIT userRoot/objectclass.db page 9 >> 116 dd=53 locks held 0 write locks 0 pid/thread >> 4845/140084837865216 flags 0 priority 100 >> 116 READ 1 WAIT userRoot/objectclass.db page 9 >> 117 dd=52 locks held 0 write locks 0 pid/thread >> 4845/140084577756928 flags 0 priority 100 >> 118 dd=51 locks held 0 write locks 0 pid/thread >> 4845/140084569364224 flags 0 priority 100 >> 119 dd=50 locks held 0 write locks 0 pid/thread >> 4845/140084560971520 flags 0 priority 100 >> 11a dd=49 locks held 1 write locks 0 pid/thread >> 4845/140084695254784 flags 10 priority 100 >> 11a READ 1 HELD ipaca/metaInfo.db handle 0 >> 11b dd=48 locks held 1 write locks 0 pid/thread >> 4845/140084695254784 flags 10 priority 100 >> 11b READ 1 HELD ipaca/notbefore.db handle 0 >> 11c dd=47 locks held 1 write locks 0 pid/thread >> 4845/140084695254784 flags 10 priority 100 >> 11c READ 1 HELD ipaca/notafter.db handle 0 >> 11d dd=46 locks held 1 write locks 0 pid/thread >> 4845/140084695254784 flags 10 priority 100 >> 11d READ 1 HELD ipaca/duration.db handle 0 >> 11e dd=45 locks held 1 write locks 0 pid/thread >> 4845/140084695254784 flags 10 priority 100 >> 11e READ 1 HELD ipaca/subjectname.db handle 0 >> 11f dd=44 locks held 1 write locks 0 pid/thread >> 4845/140084695254784 flags 10 priority 100 >> 11f READ 1 HELD ipaca/publicKeyData.db handle 0 >> 120 dd=43 locks held 1 write locks 0 pid/thread >> 4845/140084695254784 flags 10 priority 100 >> 120 READ 1 HELD ipaca/extension.db handle 0 >> 121 dd=42 locks held 1 write locks 0 pid/thread >> 4845/140084695254784 flags 10 priority 100 >> 121 READ 1 HELD ipaca/issuedby.db handle 0 >> 122 dd=41 locks held 0 write locks 0 pid/thread >> 4845/140084712040192 flags 0 priority 100 >> 123 dd=40 locks held 0 write locks 0 pid/thread >> 4845/140084812687104 flags 0 priority 100 >> 124 dd=39 locks held 0 write locks 0 pid/thread >> 4845/140084544186112 flags 0 priority 100 >> 125 dd=38 locks held 0 write locks 0 pid/thread >> 4845/140084686862080 flags 0 priority 100 >> 126 dd=37 locks held 0 write locks 0 pid/thread >> 4845/140084552578816 flags 0 priority 100 >> 128 dd=36 locks held 0 write locks 0 pid/thread >> 4845/140084686862080 flags 0 priority 100 >> 12a dd=35 locks held 0 write locks 0 pid/thread >> 4845/140084720432896 flags 0 priority 100 >> 12b dd=34 locks held 0 write locks 0 pid/thread >> 4845/140084552578816 flags 0 priority 100 >> 12c dd=33 locks held 0 write locks 0 pid/thread >> 4845/140084670076672 flags 0 priority 100 >> 12d dd=32 locks held 0 write locks 0 pid/thread >> 4845/140084577756928 flags 0 priority 100 >> 12e dd=31 locks held 0 write locks 0 pid/thread >> 4845/140084712040192 flags 0 priority 100 >> 12f dd=30 locks held 0 write locks 0 pid/thread >> 4845/140084670076672 flags 0 priority 100 >> 130 dd=29 locks held 0 write locks 0 pid/thread >> 4845/140084577756928 flags 0 priority 100 >> 131 dd=28 locks held 0 write locks 0 pid/thread >> 4845/140084560971520 flags 0 priority 100 >> 132 dd=27 locks held 0 write locks 0 pid/thread >> 4845/140084670076672 flags 0 priority 100 >> 133 dd=26 locks held 0 write locks 0 pid/thread >> 4845/140084602935040 flags 0 priority 100 >> 134 dd=25 locks held 0 write locks 0 pid/thread >> 4845/140084552578816 flags 0 priority 100 >> 135 dd=24 locks held 0 write locks 0 pid/thread >> 4845/140084678469376 flags 0 priority 100 >> 136 dd=23 locks held 0 write locks 0 pid/thread >> 4845/140084544186112 flags 0 priority 100 >> 137 dd=22 locks held 0 write locks 0 pid/thread >> 4845/140084577756928 flags 0 priority 100 >> 138 dd=21 locks held 0 write locks 0 pid/thread >> 4845/140084703647488 flags 0 priority 100 >> 139 dd=20 locks held 0 write locks 0 pid/thread >> 4845/140084678469376 flags 0 priority 100 >> 13b dd=19 locks held 0 write locks 0 pid/thread >> 4845/140084812687104 flags 0 priority 100 >> 144 dd=18 locks held 0 write locks 0 pid/thread >> 4845/140084670076672 flags 0 priority 100 >> 156 dd=17 locks held 0 write locks 0 pid/thread >> 4845/140084594542336 flags 0 priority 100 >> 15a dd=16 locks held 0 write locks 0 pid/thread >> 4845/140084577756928 flags 0 priority 100 >> 168 dd=15 locks held 0 write locks 0 pid/thread >> 4845/140084653291264 flags 0 priority 100 >> 16a dd=14 locks held 0 write locks 0 pid/thread >> 4845/140084829472512 flags 0 priority 100 >> 17d dd=13 locks held 0 write locks 0 pid/thread >> 4845/140084829472512 flags 0 priority 100 >> 17e dd=12 locks held 0 write locks 0 pid/thread >> 4845/140084829472512 flags 0 priority 100 >> 17f dd=11 locks held 0 write locks 0 pid/thread >> 4845/140084586149632 flags 0 priority 100 >> 180 dd=10 locks held 0 write locks 0 pid/thread >> 4845/140084594542336 flags 0 priority 100 >> 181 dd= 9 locks held 0 write locks 0 pid/thread >> 4845/140084577756928 flags 0 priority 100 >> 193 dd= 8 locks held 0 write locks 0 pid/thread >> 4845/140084812687104 flags 0 priority 100 >> 196 dd= 7 locks held 0 write locks 0 pid/thread >> 4845/140084695254784 flags 0 priority 100 >> 1a1 dd= 6 locks held 0 write locks 0 pid/thread >> 4845/140084720432896 flags 0 priority 100 >> 1a2 dd= 5 locks held 0 write locks 0 pid/thread >> 4845/140084577756928 flags 0 priority 100 >> 1a3 dd= 4 locks held 0 write locks 0 pid/thread >> 4845/140084812687104 flags 0 priority 100 >> 1a4 dd= 3 locks held 0 write locks 0 pid/thread >> 4845/140084812687104 flags 0 priority 100 >> 1a5 dd= 2 locks held 0 write locks 0 pid/thread >> 4845/140084577756928 flags 0 priority 100 >> 1a6 dd= 1 locks held 0 write locks 0 pid/thread >> 4845/140084594542336 flags 0 priority 100 >> 800b3a33 dd= 0 locks held 26 write locks 22 pid/thread >> 4845/140084661683968 flags 0 priority 100 >> 800b3a33 WRITE 1 HELD userRoot/objectclass.db page 9 >> 800b3a33 WRITE 1 HELD userRoot/objectclass.db page 8 >> 800b3a33 WRITE 1 HELD userRoot/objectclass.db page 18 >> 800b3a33 WRITE 1 HELD userRoot/objectclass.db page 7 >> 800b3a33 WRITE 1 HELD userRoot/objectclass.db page 14 >> 800b3a33 WRITE 1 HELD userRoot/objectclass.db page 11 >> 800b3a33 WRITE 2 HELD userRoot/objectclass.db page 3 >> 800b3a33 WRITE 1 HELD userRoot/cn.db page 150 >> 800b3a33 WRITE 1 HELD userRoot/cn.db page 89 >> 800b3a33 WRITE 1 HELD userRoot/cn.db page 53 >> 800b3a33 WRITE 1 HELD userRoot/cn.db page 131 >> 800b3a33 WRITE 1 HELD userRoot/cn.db page 117 >> 800b3a33 WRITE 1 HELD userRoot/cn.db page 11 >> 800b3a33 WRITE 2 HELD userRoot/cn.db page 141 >> 800b3a33 WRITE 1 HELD userRoot/cn.db page 41 >> 800b3a33 WRITE 1 HELD userRoot/cn.db page 40 >> 800b3a33 WRITE 1 HELD userRoot/cn.db page 143 >> 800b3a33 WRITE 1 HELD userRoot/cn.db page 30 >> 800b3a33 WRITE 2 HELD userRoot/id2entry.db page 8184 >> 800b3a33 WRITE 1 HELD userRoot/id2entry.db page 8183 >> 800b3a33 WRITE 1 HELD userRoot/id2entry.db page 7779 >> 800b3a33 WRITE 1 HELD userRoot/id2entry.db page 0 >> 800b3a33 READ 3 HELD userRoot/entryrdn.db page 3 >> 800b3a33 READ 1 HELD userRoot/entryrdn.db page 36 >> 800b3a33 READ 1 HELD userRoot/entryrdn.db page 4 >> 800b3a33 READ 1 HELD userRoot/nsuniqueid.db page 86 >> =-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-= >> Locks grouped by object: >> Locker Mode Count Status ----------------- Object --------------- >> a4 READ 1 HELD userRoot/ipasudorunas.db handle 0 >> >> 9e READ 1 HELD userRoot/ipaMemberCertProfile.db >> handle 0 >> >> 10 READ 1 HELD >> ipaca/vlv#allrevokedcertspkitomcatindex.db handle 0 >> >> 800b3a33 WRITE 2 HELD userRoot/id2entry.db page 8184 >> >> 800b3a33 WRITE 1 HELD userRoot/id2entry.db page 8183 >> >> 800b3a33 WRITE 1 HELD userRoot/id2entry.db page 7779 >> >> a1 READ 1 HELD userRoot/memberdenycmd.db handle 0 >> >> a READ 1 HELD ipaca/vlv#allcertspkitomcatindex.db >> handle 0 >> >> a7 READ 1 HELD userRoot/sourcehost.db handle 0 >> >> 57 READ 1 HELD changelog/ancestorid.db handle 0 >> >> e5 READ 1 HELD ipaca/revInfo.db handle 0 >> >> e6 READ 1 HELD ipaca/revokedby.db handle 0 >> >> 11 READ 1 HELD >> ipaca/vlv#allrevokedcertsnotafterpkitomcatindex.db handle 0 >> >> 4e READ 1 HELD userRoot/parentid.db handle 0 >> >> 4 READ 1 HELD ipaca/id2entry.db handle 0 >> >> 58 READ 1 HELD changelog/numsubordinates.db >> handle 0 >> >> 67 READ 1 HELD ipaca/seeAlso.db handle 0 >> >> 2b READ 1 HELD ipaca/vlv#carevocationpkitomcatindex.db >> handle 0 >> >> ec READ 1 HELD ipaca/requeststate.db handle 0 >> >> 11b READ 1 HELD ipaca/notbefore.db handle 0 >> >> 7e READ 1 HELD userRoot/numsubordinates.db handle 0 >> >> a2 READ 1 HELD userRoot/memberallowcmd.db handle 0 >> >> 9c READ 1 HELD userRoot/owner.db handle 0 >> >> 1e READ 1 HELD >> ipaca/vlv#cacompleteenrollmentpkitomcatindex.db handle 0 >> >> 11c READ 1 HELD ipaca/notafter.db handle 0 >> >> 40 READ 1 HELD changelog/objectclass.db handle 0 >> >> 2e READ 1 HELD changelog/entryusn.db handle 0 >> >> 16 READ 1 HELD >> ipaca/vlv#allvalidcertsnotafterpkitomcatindex.db handle 0 >> >> 15 READ 1 HELD >> ipaca/vlv#allvalidcertspkitomcatindex.db handle 0 >> >> a5 READ 1 HELD userRoot/ipaMemberCa.db handle 0 >> >> 18 READ 1 HELD ipaca/vlv#caallpkitomcatindex.db >> handle 0 >> >> a0 READ 1 HELD userRoot/secretary.db handle 0 >> >> 81 READ 1 HELD userRoot/memberUser.db handle 0 >> >> 9f READ 1 HELD userRoot/ipaassignedidview.db >> handle 0 >> >> a3 READ 1 HELD userRoot/manager.db handle 0 >> >> d8 READ 1 HELD ipaca/description.db handle 0 >> >> a8 READ 1 HELD userRoot/memberservice.db handle 0 >> >> 44 READ 1 HELD ipaca/objectclass.db handle 0 >> >> ed READ 1 HELD ipaca/dateOfCreate.db handle 0 >> >> 800b3a33 WRITE 1 HELD userRoot/id2entry.db page 0 >> >> 2 READ 1 HELD userRoot/id2entry.db handle 0 >> >> 5b READ 1 HELD userRoot/krbPrincipalName.db >> handle 0 >> >> ee READ 1 HELD ipaca/requesttype.db handle 0 >> >> c8 READ 1 HELD ipaca/certstatus.db handle 0 >> >> 56 READ 1 HELD changelog/parentid.db handle 0 >> >> 83 READ 1 HELD userRoot/memberHost.db handle 0 >> >> 5e READ 1 HELD userRoot/ipakrbprincipalalias.db >> handle 0 >> >> 121 READ 1 HELD ipaca/issuedby.db handle 0 >> >> 6 READ 1 HELD ipaca/entryrdn.db handle 0 >> >> e7 READ 1 HELD ipaca/revokedOn.db handle 0 >> >> 4c READ 1 HELD ipaca/aci.db handle 0 >> >> 11d READ 1 HELD ipaca/duration.db handle 0 >> >> 14 READ 1 HELD >> ipaca/vlv#allrevokedorrevokedexpiredcertspkitomcatindex.db >> handle 0 >> >> 21 READ 1 HELD ipaca/vlv#caenrollmentpkitomcatindex.db >> handle 0 >> >> 55 READ 1 HELD changelog/targetuniqueid.db handle 0 >> >> 800b3a33 READ 1 HELD userRoot/entryrdn.db page 36 >> >> 2c READ 1 HELD changelog/id2entry.db handle 0 >> >> 800b3a33 READ 1 HELD userRoot/entryrdn.db page 4 >> >> 34 READ 1 HELD userRoot/entryrdn.db handle 0 >> >> 800b3a33 READ 3 HELD userRoot/entryrdn.db page 3 >> >> eb READ 1 HELD ipaca/requestid.db handle 0 >> >> 49 READ 1 HELD userRoot/aci.db handle 0 >> >> 76 READ 1 HELD userRoot/ipauniqueid.db handle 0 >> >> 63 READ 1 HELD changelog/seeAlso.db handle 0 >> >> 7c READ 1 HELD userRoot/fqdn.db handle 0 >> >> db READ 1 HELD ipaca/cn.db handle 0 >> >> ef READ 1 HELD ipaca/numsubordinates.db handle 0 >> >> 32 READ 1 HELD ipaca/entryusn.db handle 0 >> >> 1d READ 1 HELD ipaca/vlv#cacompletepkitomcatindex.db >> handle 0 >> >> dd READ 1 HELD ipaca/ancestorid.db handle 0 >> >> a9 READ 1 HELD userRoot/ipatokenradiusconfiglink.db >> handle 0 >> >> f0 READ 1 HELD userRoot/userCertificate.db handle 0 >> >> 11a READ 1 HELD ipaca/metaInfo.db handle 0 >> >> 7d READ 1 HELD userRoot/managedby.db handle 0 >> >> ea READ 1 HELD ipaca/nsuniqueid.db handle 0 >> >> c5 READ 1 HELD userRoot/uidnumber.db handle 0 >> >> d1 READ 1 HELD userRoot/nscpEntryDN.db handle 0 >> >> 120 READ 1 HELD ipaca/extension.db handle 0 >> >> 800b3a33 WRITE 1 HELD userRoot/cn.db page 150 >> >> 69 READ 1 HELD ipaca/parentid.db handle 0 >> >> 800b3a33 WRITE 1 HELD userRoot/cn.db page 131 >> >> 800b3a33 WRITE 1 HELD userRoot/cn.db page 143 >> >> 800b3a33 WRITE 2 HELD userRoot/cn.db page 141 >> >> 20 READ 1 HELD >> ipaca/vlv#cacompleterevocationpkitomcatindex.db handle 0 >> >> 8e READ 1 HELD userRoot/macAddress.db handle 0 >> >> e READ 1 HELD >> ipaca/vlv#allnonrevokedcertspkitomcatindex.db handle 0 >> >> 17 READ 1 HELD >> ipaca/vlv#allvalidorrevokedcertspkitomcatindex.db handle 0 >> >> 800b3a33 WRITE 1 HELD userRoot/cn.db page 30 >> >> 800b3a33 WRITE 1 HELD userRoot/objectclass.db page 14 >> >> 800b3a33 WRITE 1 HELD userRoot/objectclass.db page 9 >> 116 READ 1 WAIT userRoot/objectclass.db page 9 >> 115 READ 1 WAIT userRoot/objectclass.db page 9 >> 92 READ 1 WAIT userRoot/objectclass.db page 9 >> >> 800b3a33 WRITE 1 HELD userRoot/objectclass.db page 8 >> >> 7b READ 1 HELD userRoot/cn.db handle 0 >> >> 800b3a33 WRITE 1 HELD userRoot/objectclass.db page 11 >> >> 800b3a33 WRITE 1 HELD userRoot/objectclass.db page 7 >> >> 800b3a33 WRITE 1 HELD userRoot/cn.db page 11 >> >> 38 READ 1 HELD userRoot/objectclass.db handle 0 >> >> 800b3a33 WRITE 2 HELD userRoot/objectclass.db page 3 >> >> 800b3a33 WRITE 1 HELD userRoot/cn.db page 53 >> >> 800b3a33 WRITE 1 HELD userRoot/objectclass.db page 18 >> 6e READ 1 WAIT userRoot/objectclass.db page 18 >> 94 READ 1 WAIT userRoot/objectclass.db page 18 >> >> 800b3a33 WRITE 1 HELD userRoot/cn.db page 41 >> >> 800b3a33 WRITE 1 HELD userRoot/cn.db page 40 >> >> 800b3a33 WRITE 1 HELD userRoot/cn.db page 89 >> >> 800b3a33 WRITE 1 HELD userRoot/cn.db page 117 >> >> 30 READ 1 HELD userRoot/entryusn.db handle 0 >> >> 3c READ 1 HELD changelog/entryrdn.db handle 0 >> >> 97 READ 1 HELD userRoot/uid.db handle 0 >> >> 47 READ 1 HELD changelog/aci.db handle 0 >> >> e3 READ 1 HELD ipaca/serialno.db handle 0 >> >> 7a READ 1 HELD userRoot/nsuniqueid.db handle 0 >> >> 800b3a33 READ 1 HELD userRoot/nsuniqueid.db page 86 >> >> 51 READ 1 HELD changelog/nsuniqueid.db handle 0 >> >> 11e READ 1 HELD ipaca/subjectname.db handle 0 >> >> 7f READ 1 HELD userRoot/member.db handle 0 >> >> 11f READ 1 HELD ipaca/publicKeyData.db handle 0 >> >> 9b READ 1 HELD userRoot/uniquemember.db handle 0 >> >> 52 READ 1 HELD changelog/changenumber.db handle 0 >> >> ce READ 1 HELD userRoot/gidnumber.db handle 0 >> >> 65 READ 1 HELD userRoot/seeAlso.db handle 0 >> >> 9d READ 1 HELD userRoot/ipaallowedtarget.db >> handle 0 >> >> a6 READ 1 HELD userRoot/ipasudorunasgroup.db >> handle 0 >> >> 3a READ 1 HELD userRoot/ancestorid.db handle 0 >> >> >> >> >> pstack output >> >> root at ipa-master ]# pstack 4845 >> Thread 44 (Thread 0x7f681dd3a700 (LWP 4847)): >> #0 0x00007f683cd719b3 in select () from /lib64/libc.so.6 >> #1 0x00007f683f4fc0e9 in DS_Sleep () from >> /usr/lib64/dirsrv/libslapd.so.0 >> #2 0x00007f6832b4a907 in deadlock_threadmain () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #3 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so >> #4 0x00007f683d04cdc5 in start_thread () from /lib64/libpthread.so.0 >> #5 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 >> Thread 43 (Thread 0x7f681d539700 (LWP 4848)): >> #0 0x00007f683cd719b3 in select () from /lib64/libc.so.6 >> #1 0x00007f68376d8bad in __os_yield () from /lib64/libdb-5.3.so >> >> #2 0x00007f68376d42b3 in __memp_sync_int () from /lib64/libdb-5.3.so >> >> #3 0x00007f68376e4752 in __txn_checkpoint () from >> /lib64/libdb-5.3.so >> #4 0x00007f68376e4b74 in __txn_checkpoint_pp () from >> /lib64/libdb-5.3.so >> #5 0x00007f6832b4ea87 in checkpoint_threadmain () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #6 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so >> #7 0x00007f683d04cdc5 in start_thread () from /lib64/libpthread.so.0 >> #8 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 >> Thread 42 (Thread 0x7f681cd38700 (LWP 4849)): >> #0 0x00007f683cd719b3 in select () from /lib64/libc.so.6 >> #1 0x00007f683f4fc0e9 in DS_Sleep () from >> /usr/lib64/dirsrv/libslapd.so.0 >> #2 0x00007f6832b4ab7f in trickle_threadmain () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #3 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so >> #4 0x00007f683d04cdc5 in start_thread () from /lib64/libpthread.so.0 >> #5 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 >> Thread 41 (Thread 0x7f681c537700 (LWP 4850)): >> #0 0x00007f683cd719b3 in select () from /lib64/libc.so.6 >> #1 0x00007f683f4fc0e9 in DS_Sleep () from >> /usr/lib64/dirsrv/libslapd.so.0 >> #2 0x00007f6832b45707 in perf_threadmain () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #3 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so >> #4 0x00007f683d04cdc5 in start_thread () from /lib64/libpthread.so.0 >> #5 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 >> Thread 40 (Thread 0x7f681bb2d700 (LWP 4851)): >> #0 0x00007f683d0506d5 in pthread_cond_wait@@GLIBC_2.3.2 () from >> /lib64/libpthread.so.0 >> #1 0x00007f683d6a6050 in PR_WaitCondVar () from /lib64/libnspr4.so >> #2 0x00007f683f4eb198 in slapi_wait_condvar () from >> /usr/lib64/dirsrv/libslapd.so.0 >> #3 0x00007f683587d62e in cos_cache_wait_on_change () from >> /usr/lib64/dirsrv/plugins/libcos-plugin.so >> #4 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so >> #5 0x00007f683d04cdc5 in start_thread () from /lib64/libpthread.so.0 >> #6 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 >> Thread 39 (Thread 0x7f681b32c700 (LWP 4852)): >> #0 0x00007f683d0506d5 in pthread_cond_wait@@GLIBC_2.3.2 () from >> /lib64/libpthread.so.0 >> #1 0x00007f683d6a6050 in PR_WaitCondVar () from /lib64/libnspr4.so >> #2 0x00007f683f4eb198 in slapi_wait_condvar () from >> /usr/lib64/dirsrv/libslapd.so.0 >> #3 0x00007f68311e8eed in roles_cache_wait_on_change () from >> /usr/lib64/dirsrv/plugins/libroles-plugin.so >> #4 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so >> #5 0x00007f683d04cdc5 in start_thread () from /lib64/libpthread.so.0 >> #6 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 >> Thread 38 (Thread 0x7f681ab2b700 (LWP 4853)): >> #0 0x00007f683d0506d5 in pthread_cond_wait@@GLIBC_2.3.2 () from >> /lib64/libpthread.so.0 >> #1 0x00007f683d6a6050 in PR_WaitCondVar () from /lib64/libnspr4.so >> #2 0x00007f683f4eb198 in slapi_wait_condvar () from >> /usr/lib64/dirsrv/libslapd.so.0 >> #3 0x00007f68311e8eed in roles_cache_wait_on_change () from >> /usr/lib64/dirsrv/plugins/libroles-plugin.so >> #4 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so >> #5 0x00007f683d04cdc5 in start_thread () from /lib64/libpthread.so.0 >> #6 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 >> Thread 37 (Thread 0x7f681a32a700 (LWP 4854)): >> #0 0x00007f683d0506d5 in pthread_cond_wait@@GLIBC_2.3.2 () from >> /lib64/libpthread.so.0 >> #1 0x00007f683d6a6050 in PR_WaitCondVar () from /lib64/libnspr4.so >> #2 0x00007f683f4eb198 in slapi_wait_condvar () from >> /usr/lib64/dirsrv/libslapd.so.0 >> #3 0x00007f68311e8eed in roles_cache_wait_on_change () from >> /usr/lib64/dirsrv/plugins/libroles-plugin.so >> #4 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so >> #5 0x00007f683d04cdc5 in start_thread () from /lib64/libpthread.so.0 >> #6 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 >> Thread 36 (Thread 0x7f6819b29700 (LWP 4855)): >> #0 0x00007f683d050a82 in pthread_cond_timedwait@@GLIBC_2.3.2 () from >> /lib64/libpthread.so.0 >> #1 0x00007f683d6a5b07 in pt_TimedWait () from /lib64/libnspr4.so >> #2 0x00007f683d6a5fce in PR_WaitCondVar () from /lib64/libnspr4.so >> #3 0x00007f683f9a4d03 in housecleaning () >> #4 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so >> #5 0x00007f683d04cdc5 in start_thread () from /lib64/libpthread.so.0 >> #6 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 >> Thread 35 (Thread 0x7f6819328700 (LWP 4856)): >> #0 0x00007f683d050a82 in pthread_cond_timedwait@@GLIBC_2.3.2 () from >> /lib64/libpthread.so.0 >> #1 0x00007f683d6a5b07 in pt_TimedWait () from /lib64/libnspr4.so >> #2 0x00007f683d6a5fce in PR_WaitCondVar () from /lib64/libnspr4.so >> #3 0x00007f683f4911e8 in eq_loop () from /usr/lib64/dirsrv/libslapd.so.0 >> #4 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so >> #5 0x00007f683d04cdc5 in start_thread () from /lib64/libpthread.so.0 >> #6 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 >> Thread 34 (Thread 0x7f680bfff700 (LWP 4859)): >> #0 0x00007f683d0506d5 in pthread_cond_wait@@GLIBC_2.3.2 () from >> /lib64/libpthread.so.0 >> #1 0x00007f683d6a6050 in PR_WaitCondVar () from /lib64/libnspr4.so >> #2 0x00007f683f999fee in connection_wait_for_new_work () >> #3 0x00007f683f99b719 in connection_threadmain () >> #4 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so >> #5 0x00007f683d04cdc5 in start_thread () from /lib64/libpthread.so.0 >> #6 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 >> Thread 33 (Thread 0x7f680b7fe700 (LWP 4860)): >> #0 0x00007f683d0506d5 in pthread_cond_wait@@GLIBC_2.3.2 () from >> /lib64/libpthread.so.0 >> #1 0x00007f683d6a6050 in PR_WaitCondVar () from /lib64/libnspr4.so >> #2 0x00007f683f999fee in connection_wait_for_new_work () >> #3 0x00007f683f99b719 in connection_threadmain () >> #4 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so >> #5 0x00007f683d04cdc5 in start_thread () from /lib64/libpthread.so.0 >> #6 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 >> Thread 32 (Thread 0x7f680affd700 (LWP 4861)): >> #0 0x00007f683d0506d5 in pthread_cond_wait@@GLIBC_2.3.2 () from >> /lib64/libpthread.so.0 >> #1 0x00007f68375ac2f3 in __db_hybrid_mutex_suspend () from >> /lib64/libdb-5.3.so >> #2 0x00007f68375ab640 in __db_tas_mutex_lock () from >> /lib64/libdb-5.3.so >> #3 0x00007f6837655d3a in __lock_get_internal () from >> /lib64/libdb-5.3.so >> #4 0x00007f6837656820 in __lock_get () from /lib64/libdb-5.3.so >> >> #5 0x00007f6837682142 in __db_lget () from /lib64/libdb-5.3.so >> >> #6 0x00007f68375c9605 in __bam_search () from /lib64/libdb-5.3.so >> >> #7 0x00007f68375b4256 in __bamc_search () from /lib64/libdb-5.3.so >> >> #8 0x00007f68375b5d0f in __bamc_get () from /lib64/libdb-5.3.so >> >> #9 0x00007f683766eca6 in __dbc_iget () from /lib64/libdb-5.3.so >> >> #10 0x00007f683767db02 in __dbc_get_pp () from /lib64/libdb-5.3.so >> >> #11 0x00007f6832b59680 in idl_new_fetch () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #12 0x00007f6832b67b26 in index_read_ext_allids () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #13 0x00007f6832b52284 in keys2idl () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #14 0x00007f6832b529e3 in ava_candidates.isra.0 () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #15 0x00007f6832b52fd2 in filter_candidates_ext () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #16 0x00007f6832b54096 in list_candidates () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #17 0x00007f6832b52f40 in filter_candidates_ext () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #18 0x00007f6832b54096 in list_candidates () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #19 0x00007f6832b52f40 in filter_candidates_ext () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #20 0x00007f6832b54096 in list_candidates () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #21 0x00007f6832b52f40 in filter_candidates_ext () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #22 0x00007f6832b8e80c in subtree_candidates () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #23 0x00007f6832b8fe7f in ldbm_back_search () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #24 0x00007f683f4bd1d6 in op_shared_search () from >> /usr/lib64/dirsrv/libslapd.so.0 >> #25 0x00007f683f9ae33e in do_search () >> #26 0x00007f683f99c3c3 in connection_threadmain () >> #27 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so >> #28 0x00007f683d04cdc5 in start_thread () from /lib64/libpthread.so.0 >> #29 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 >> Thread 31 (Thread 0x7f680a7fc700 (LWP 4862)): >> #0 0x00007f683d0506d5 in pthread_cond_wait@@GLIBC_2.3.2 () from >> /lib64/libpthread.so.0 >> #1 0x00007f683d6a62c3 in PR_EnterMonitor () from /lib64/libnspr4.so >> #2 0x00007f6832b4d3e6 in dblayer_txn_begin () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #3 0x00007f6832b89007 in ldbm_back_modify () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #4 0x00007f683f4b6541 in op_shared_modify () from >> /usr/lib64/dirsrv/libslapd.so.0 >> #5 0x00007f683f4b7004 in modify_internal_pb () from >> /usr/lib64/dirsrv/libslapd.so.0 >> #6 0x00007f683483bbbb in ipalockout_postop () from >> /usr/lib64/dirsrv/plugins/libipa_lockout.so >> #7 0x00007f683f4c8848 in plugin_call_func () from >> /usr/lib64/dirsrv/libslapd.so.0 >> #8 0x00007f683f4c8b13 in plugin_call_plugins () from >> /usr/lib64/dirsrv/libslapd.so.0 >> #9 0x00007f683f9954d8 in do_bind () >> #10 0x00007f683f99c3fd in connection_threadmain () >> #11 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so >> #12 0x00007f683d04cdc5 in start_thread () from /lib64/libpthread.so.0 >> #13 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 >> Thread 30 (Thread 0x7f6809ffb700 (LWP 4863)): >> #0 0x00007f683d0506d5 in pthread_cond_wait@@GLIBC_2.3.2 () from >> /lib64/libpthread.so.0 >> #1 0x00007f68375ac2f3 in __db_hybrid_mutex_suspend () from >> /lib64/libdb-5.3.so >> #2 0x00007f68375ab640 in __db_tas_mutex_lock () from >> /lib64/libdb-5.3.so >> #3 0x00007f6837655d3a in __lock_get_internal () from >> /lib64/libdb-5.3.so >> #4 0x00007f6837656820 in __lock_get () from /lib64/libdb-5.3.so >> >> #5 0x00007f6837682142 in __db_lget () from /lib64/libdb-5.3.so >> >> #6 0x00007f68375c9605 in __bam_search () from /lib64/libdb-5.3.so >> >> #7 0x00007f68375b4256 in __bamc_search () from /lib64/libdb-5.3.so >> >> #8 0x00007f68375b5d0f in __bamc_get () from /lib64/libdb-5.3.so >> >> #9 0x00007f683766eca6 in __dbc_iget () from /lib64/libdb-5.3.so >> >> #10 0x00007f683767db02 in __dbc_get_pp () from /lib64/libdb-5.3.so >> >> #11 0x00007f6832b59680 in idl_new_fetch () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #12 0x00007f6832b67b26 in index_read_ext_allids () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #13 0x00007f6832b52284 in keys2idl () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #14 0x00007f6832b529e3 in ava_candidates.isra.0 () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #15 0x00007f6832b52fd2 in filter_candidates_ext () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #16 0x00007f6832b54096 in list_candidates () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #17 0x00007f6832b52f40 in filter_candidates_ext () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #18 0x00007f6832b8e80c in subtree_candidates () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #19 0x00007f6832b8fe7f in ldbm_back_search () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #20 0x00007f683f4bd1d6 in op_shared_search () from >> /usr/lib64/dirsrv/libslapd.so.0 >> #21 0x00007f683f4cd94e in search_internal_callback_pb () from >> /usr/lib64/dirsrv/libslapd.so.0 >> #22 0x00007f683f4cdbe8 in search_internal_pb () from >> /usr/lib64/dirsrv/libslapd.so.0 >> #23 0x00007f683f9ac49b in ids_sasl_canon_user () >> #24 0x00007f683d46c93b in _sasl_canon_user () from /lib64/libsasl2.so.3 >> #25 0x00007f683d46cc4c in _sasl_canon_user_lookup () from >> /lib64/libsasl2.so.3 >> #26 0x00007f6837374fb9 in gssapi_server_mech_step () from >> /usr/lib64/sasl2/libgssapiv2.so >> #27 0x00007f683d478b9b in sasl_server_step () from /lib64/libsasl2.so.3 >> #28 0x00007f683f9ad83b in ids_sasl_check_bind () >> #29 0x00007f683f99561a in do_bind () >> #30 0x00007f683f99c3fd in connection_threadmain () >> #31 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so >> #32 0x00007f683d04cdc5 in start_thread () from /lib64/libpthread.so.0 >> #33 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 >> Thread 29 (Thread 0x7f68097fa700 (LWP 4864)): >> #0 0x00007f683d0506d5 in pthread_cond_wait@@GLIBC_2.3.2 () from >> /lib64/libpthread.so.0 >> #1 0x00007f683d6a6050 in PR_WaitCondVar () from /lib64/libnspr4.so >> #2 0x00007f683f999fee in connection_wait_for_new_work () >> #3 0x00007f683f99b719 in connection_threadmain () >> #4 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so >> #5 0x00007f683d04cdc5 in start_thread () from /lib64/libpthread.so.0 >> #6 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 >> Thread 28 (Thread 0x7f6808ff9700 (LWP 4865)): >> #0 0x00007f683d0506d5 in pthread_cond_wait@@GLIBC_2.3.2 () from >> /lib64/libpthread.so.0 >> #1 0x00007f683d6a6050 in PR_WaitCondVar () from /lib64/libnspr4.so >> #2 0x00007f683f999fee in connection_wait_for_new_work () >> #3 0x00007f683f99b719 in connection_threadmain () >> #4 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so >> #5 0x00007f683d04cdc5 in start_thread () from /lib64/libpthread.so.0 >> #6 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 >> Thread 27 (Thread 0x7f6803fff700 (LWP 4866)): >> #0 0x00007f683d0506d5 in pthread_cond_wait@@GLIBC_2.3.2 () from >> /lib64/libpthread.so.0 >> #1 0x00007f683d6a6050 in PR_WaitCondVar () from /lib64/libnspr4.so >> #2 0x00007f683f999fee in connection_wait_for_new_work () >> #3 0x00007f683f99b719 in connection_threadmain () >> #4 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so >> #5 0x00007f683d04cdc5 in start_thread () from /lib64/libpthread.so.0 >> #6 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 >> Thread 26 (Thread 0x7f68037fe700 (LWP 4867)): >> #0 0x00007f683d0506d5 in pthread_cond_wait@@GLIBC_2.3.2 () from >> /lib64/libpthread.so.0 >> #1 0x00007f683d6a6050 in PR_WaitCondVar () from /lib64/libnspr4.so >> #2 0x00007f683f999fee in connection_wait_for_new_work () >> #3 0x00007f683f99b719 in connection_threadmain () >> #4 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so >> #5 0x00007f683d04cdc5 in start_thread () from /lib64/libpthread.so.0 >> #6 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 >> Thread 25 (Thread 0x7f6802ffd700 (LWP 4868)): >> #0 0x00007f683d0506d5 in pthread_cond_wait@@GLIBC_2.3.2 () from >> /lib64/libpthread.so.0 >> #1 0x00007f683d6a6050 in PR_WaitCondVar () from /lib64/libnspr4.so >> #2 0x00007f683f999fee in connection_wait_for_new_work () >> #3 0x00007f683f99b719 in connection_threadmain () >> #4 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so >> #5 0x00007f683d04cdc5 in start_thread () from /lib64/libpthread.so.0 >> #6 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 >> Thread 24 (Thread 0x7f68027fc700 (LWP 4869)): >> #0 0x00007f683d0506d5 in pthread_cond_wait@@GLIBC_2.3.2 () from >> /lib64/libpthread.so.0 >> #1 0x00007f68375ac2f3 in __db_hybrid_mutex_suspend () from >> /lib64/libdb-5.3.so >> #2 0x00007f68375abb37 in __db_tas_mutex_readlock () from >> /lib64/libdb-5.3.so >> #3 0x00007f68376c5d35 in __memp_fget () from /lib64/libdb-5.3.so >> >> #4 0x00007f683766f333 in __dbc_iget () from /lib64/libdb-5.3.so >> >> #5 0x00007f683767db02 in __dbc_get_pp () from /lib64/libdb-5.3.so >> >> #6 0x00007f6832b59822 in idl_new_fetch () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #7 0x00007f6832b67b26 in index_read_ext_allids () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #8 0x00007f6832b52284 in keys2idl () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #9 0x00007f6832b529e3 in ava_candidates.isra.0 () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #10 0x00007f6832b52fd2 in filter_candidates_ext () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #11 0x00007f6832b54096 in list_candidates () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #12 0x00007f6832b52f40 in filter_candidates_ext () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #13 0x00007f6832b54096 in list_candidates () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #14 0x00007f6832b52f40 in filter_candidates_ext () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #15 0x00007f6832b54096 in list_candidates () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #16 0x00007f6832b52f40 in filter_candidates_ext () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #17 0x00007f6832b8e80c in subtree_candidates () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #18 0x00007f6832b8fe7f in ldbm_back_search () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #19 0x00007f683f4bd1d6 in op_shared_search () from >> /usr/lib64/dirsrv/libslapd.so.0 >> #20 0x00007f683f9ae33e in do_search () >> #21 0x00007f683f99c3c3 in connection_threadmain () >> #22 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so >> #23 0x00007f683d04cdc5 in start_thread () from /lib64/libpthread.so.0 >> #24 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 >> Thread 23 (Thread 0x7f6801ffb700 (LWP 4870)): >> #0 0x00007f683d0506d5 in pthread_cond_wait@@GLIBC_2.3.2 () from >> /lib64/libpthread.so.0 >> #1 0x00007f683d6a6050 in PR_WaitCondVar () from /lib64/libnspr4.so >> #2 0x00007f683f999fee in connection_wait_for_new_work () >> #3 0x00007f683f99b719 in connection_threadmain () >> #4 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so >> #5 0x00007f683d04cdc5 in start_thread () from /lib64/libpthread.so.0 >> #6 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 >> Thread 22 (Thread 0x7f68017fa700 (LWP 4871)): >> #0 0x00007f683d0506d5 in pthread_cond_wait@@GLIBC_2.3.2 () from >> /lib64/libpthread.so.0 >> #1 0x00007f683d6a6050 in PR_WaitCondVar () from /lib64/libnspr4.so >> #2 0x00007f683f999fee in connection_wait_for_new_work () >> #3 0x00007f683f99b719 in connection_threadmain () >> #4 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so >> #5 0x00007f683d04cdc5 in start_thread () from /lib64/libpthread.so.0 >> #6 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 >> Thread 21 (Thread 0x7f6800ff9700 (LWP 4872)): >> #0 0x00007f683d0506d5 in pthread_cond_wait@@GLIBC_2.3.2 () from >> /lib64/libpthread.so.0 >> #1 0x00007f683d6a6050 in PR_WaitCondVar () from /lib64/libnspr4.so >> #2 0x00007f683f999fee in connection_wait_for_new_work () >> #3 0x00007f683f99b719 in connection_threadmain () >> #4 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so >> #5 0x00007f683d04cdc5 in start_thread () from /lib64/libpthread.so.0 >> #6 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 >> Thread 20 (Thread 0x7f68007f8700 (LWP 4873)): >> #0 0x00007f683d0506d5 in pthread_cond_wait@@GLIBC_2.3.2 () from >> /lib64/libpthread.so.0 >> #1 0x00007f68375ac2f3 in __db_hybrid_mutex_suspend () from >> /lib64/libdb-5.3.so >> #2 0x00007f68375ab640 in __db_tas_mutex_lock () from >> /lib64/libdb-5.3.so >> #3 0x00007f68376c5e1b in __memp_fget () from /lib64/libdb-5.3.so >> >> #4 0x00007f68375c9211 in __bam_search () from /lib64/libdb-5.3.so >> >> #5 0x00007f68375b4256 in __bamc_search () from /lib64/libdb-5.3.so >> >> #6 0x00007f68375b86c4 in __bamc_put () from /lib64/libdb-5.3.so >> >> #7 0x00007f683766f6d5 in __dbc_iput () from /lib64/libdb-5.3.so >> >> #8 0x00007f683766aa0e in __db_put () from /lib64/libdb-5.3.so >> >> #9 0x00007f683767ffa4 in __db_put_pp () from /lib64/libdb-5.3.so >> >> #10 0x00007f6832b5a4cf in idl_new_insert_key () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #11 0x00007f6832b695b5 in addordel_values_sv.isra.3 () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #12 0x00007f6832b6a25a in index_addordel_values_ext_sv () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #13 0x00007f6832b6a694 in index_addordel_values_sv () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #14 0x00007f6832b6a75a in index_addordel_entry () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #15 0x00007f6832b6d5fa in ldbm_back_add () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #16 0x00007f683f46fcca in op_shared_add () from >> /usr/lib64/dirsrv/libslapd.so.0 >> #17 0x00007f683f471020 in do_add () from /usr/lib64/dirsrv/libslapd.so.0 >> #18 0x00007f683f99c303 in connection_threadmain () >> #19 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so >> #20 0x00007f683d04cdc5 in start_thread () from /lib64/libpthread.so.0 >> #21 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 >> Thread 19 (Thread 0x7f67ffff7700 (LWP 4874)): >> #0 0x00007f683d0506d5 in pthread_cond_wait@@GLIBC_2.3.2 () from >> /lib64/libpthread.so.0 >> #1 0x00007f683d6a62c3 in PR_EnterMonitor () from /lib64/libnspr4.so >> #2 0x00007f6832b4d3e6 in dblayer_txn_begin () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #3 0x00007f6832b89007 in ldbm_back_modify () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #4 0x00007f683f4b6541 in op_shared_modify () from >> /usr/lib64/dirsrv/libslapd.so.0 >> #5 0x00007f683f4b7004 in modify_internal_pb () from >> /usr/lib64/dirsrv/libslapd.so.0 >> #6 0x00007f6832dd3b60 in ipapwd_apply_mods () from >> /usr/lib64/dirsrv/plugins/libipa_pwd_extop.so >> #7 0x00007f6832dd78e0 in store_new_keys () from >> /usr/lib64/dirsrv/plugins/libipa_pwd_extop.so >> #8 0x00007f6832dd9ad0 in ipapwd_extop () from >> /usr/lib64/dirsrv/plugins/libipa_pwd_extop.so >> #9 0x00007f683f4c6ba2 in plugin_call_exop_plugins () from >> /usr/lib64/dirsrv/libslapd.so.0 >> #10 0x00007f683f9a2efe in do_extended () >> #11 0x00007f683f99c41a in connection_threadmain () >> #12 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so >> #13 0x00007f683d04cdc5 in start_thread () from /lib64/libpthread.so.0 >> #14 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 >> Thread 18 (Thread 0x7f67ff7f6700 (LWP 4875)): >> #0 0x00007f683d0506d5 in pthread_cond_wait@@GLIBC_2.3.2 () from >> /lib64/libpthread.so.0 >> #1 0x00007f683d6a6050 in PR_WaitCondVar () from /lib64/libnspr4.so >> #2 0x00007f683f999fee in connection_wait_for_new_work () >> #3 0x00007f683f99b719 in connection_threadmain () >> #4 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so >> #5 0x00007f683d04cdc5 in start_thread () from /lib64/libpthread.so.0 >> #6 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 >> Thread 17 (Thread 0x7f67feff5700 (LWP 4876)): >> #0 0x00007f683d0506d5 in pthread_cond_wait@@GLIBC_2.3.2 () from >> /lib64/libpthread.so.0 >> #1 0x00007f68375ac2f3 in __db_hybrid_mutex_suspend () from >> /lib64/libdb-5.3.so >> #2 0x00007f68375ab640 in __db_tas_mutex_lock () from >> /lib64/libdb-5.3.so >> #3 0x00007f6837655d3a in __lock_get_internal () from >> /lib64/libdb-5.3.so >> #4 0x00007f6837656820 in __lock_get () from /lib64/libdb-5.3.so >> >> #5 0x00007f6837682142 in __db_lget () from /lib64/libdb-5.3.so >> >> #6 0x00007f68375c9605 in __bam_search () from /lib64/libdb-5.3.so >> >> #7 0x00007f68375b4256 in __bamc_search () from /lib64/libdb-5.3.so >> >> #8 0x00007f68375b5d0f in __bamc_get () from /lib64/libdb-5.3.so >> >> #9 0x00007f683766eca6 in __dbc_iget () from /lib64/libdb-5.3.so >> >> #10 0x00007f683767db02 in __dbc_get_pp () from /lib64/libdb-5.3.so >> >> #11 0x00007f6832b59680 in idl_new_fetch () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #12 0x00007f6832b67b26 in index_read_ext_allids () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #13 0x00007f6832b52284 in keys2idl () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #14 0x00007f6832b529e3 in ava_candidates.isra.0 () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #15 0x00007f6832b52fd2 in filter_candidates_ext () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #16 0x00007f6832b54096 in list_candidates () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #17 0x00007f6832b52f40 in filter_candidates_ext () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #18 0x00007f6832b8e80c in subtree_candidates () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #19 0x00007f6832b8fe7f in ldbm_back_search () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #20 0x00007f683f4bd1d6 in op_shared_search () from >> /usr/lib64/dirsrv/libslapd.so.0 >> #21 0x00007f683f4cd94e in search_internal_callback_pb () from >> /usr/lib64/dirsrv/libslapd.so.0 >> #22 0x00007f683f4cdbe8 in search_internal_pb () from >> /usr/lib64/dirsrv/libslapd.so.0 >> #23 0x00007f6832dd75dc in get_entry_by_principal () from >> /usr/lib64/dirsrv/plugins/libipa_pwd_extop.so >> #24 0x00007f6832dd8eb1 in ipapwd_extop () from >> /usr/lib64/dirsrv/plugins/libipa_pwd_extop.so >> #25 0x00007f683f4c6ba2 in plugin_call_exop_plugins () from >> /usr/lib64/dirsrv/libslapd.so.0 >> #26 0x00007f683f9a2efe in do_extended () >> #27 0x00007f683f99c41a in connection_threadmain () >> #28 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so >> #29 0x00007f683d04cdc5 in start_thread () from /lib64/libpthread.so.0 >> #30 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 >> Thread 16 (Thread 0x7f67fe7f4700 (LWP 4877)): >> #0 0x00007f683d0506d5 in pthread_cond_wait@@GLIBC_2.3.2 () from >> /lib64/libpthread.so.0 >> #1 0x00007f683d6a6050 in PR_WaitCondVar () from /lib64/libnspr4.so >> #2 0x00007f683f999fee in connection_wait_for_new_work () >> #3 0x00007f683f99b719 in connection_threadmain () >> #4 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so >> #5 0x00007f683d04cdc5 in start_thread () from /lib64/libpthread.so.0 >> #6 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 >> Thread 15 (Thread 0x7f67fdff3700 (LWP 4878)): >> #0 0x00007f683d0506d5 in pthread_cond_wait@@GLIBC_2.3.2 () from >> /lib64/libpthread.so.0 >> #1 0x00007f683d6a6050 in PR_WaitCondVar () from /lib64/libnspr4.so >> #2 0x00007f683f999fee in connection_wait_for_new_work () >> #3 0x00007f683f99b719 in connection_threadmain () >> #4 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so >> #5 0x00007f683d04cdc5 in start_thread () from /lib64/libpthread.so.0 >> #6 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 >> Thread 14 (Thread 0x7f67fd7f2700 (LWP 4879)): >> #0 0x00007f683d0506d5 in pthread_cond_wait@@GLIBC_2.3.2 () from >> /lib64/libpthread.so.0 >> #1 0x00007f683d6a6050 in PR_WaitCondVar () from /lib64/libnspr4.so >> #2 0x00007f683f999fee in connection_wait_for_new_work () >> #3 0x00007f683f99b719 in connection_threadmain () >> #4 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so >> #5 0x00007f683d04cdc5 in start_thread () from /lib64/libpthread.so.0 >> #6 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 >> Thread 13 (Thread 0x7f67fcff1700 (LWP 4880)): >> #0 0x00007f683d0506d5 in pthread_cond_wait@@GLIBC_2.3.2 () from >> /lib64/libpthread.so.0 >> #1 0x00007f683d6a6050 in PR_WaitCondVar () from /lib64/libnspr4.so >> #2 0x00007f683f999fee in connection_wait_for_new_work () >> #3 0x00007f683f99b719 in connection_threadmain () >> #4 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so >> #5 0x00007f683d04cdc5 in start_thread () from /lib64/libpthread.so.0 >> #6 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 >> Thread 12 (Thread 0x7f67fc7f0700 (LWP 4881)): >> #0 0x00007f683d0506d5 in pthread_cond_wait@@GLIBC_2.3.2 () from >> /lib64/libpthread.so.0 >> #1 0x00007f683d6a6050 in PR_WaitCondVar () from /lib64/libnspr4.so >> #2 0x00007f683f999fee in connection_wait_for_new_work () >> #3 0x00007f683f99b719 in connection_threadmain () >> #4 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so >> #5 0x00007f683d04cdc5 in start_thread () from /lib64/libpthread.so.0 >> #6 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 >> Thread 11 (Thread 0x7f67fbfef700 (LWP 4882)): >> #0 0x00007f683d0506d5 in pthread_cond_wait@@GLIBC_2.3.2 () from >> /lib64/libpthread.so.0 >> #1 0x00007f683d6a6050 in PR_WaitCondVar () from /lib64/libnspr4.so >> #2 0x00007f683f999fee in connection_wait_for_new_work () >> #3 0x00007f683f99b719 in connection_threadmain () >> #4 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so >> #5 0x00007f683d04cdc5 in start_thread () from /lib64/libpthread.so.0 >> #6 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 >> Thread 10 (Thread 0x7f67fb7ee700 (LWP 4883)): >> #0 0x00007f683d0506d5 in pthread_cond_wait@@GLIBC_2.3.2 () from >> /lib64/libpthread.so.0 >> #1 0x00007f68375ac2f3 in __db_hybrid_mutex_suspend () from >> /lib64/libdb-5.3.so >> #2 0x00007f68375ab640 in __db_tas_mutex_lock () from >> /lib64/libdb-5.3.so >> #3 0x00007f6837655d3a in __lock_get_internal () from >> /lib64/libdb-5.3.so >> #4 0x00007f6837656820 in __lock_get () from /lib64/libdb-5.3.so >> >> #5 0x00007f6837682142 in __db_lget () from /lib64/libdb-5.3.so >> >> #6 0x00007f68375c9605 in __bam_search () from /lib64/libdb-5.3.so >> >> #7 0x00007f68375b4256 in __bamc_search () from /lib64/libdb-5.3.so >> >> #8 0x00007f68375b5d0f in __bamc_get () from /lib64/libdb-5.3.so >> >> #9 0x00007f683766eca6 in __dbc_iget () from /lib64/libdb-5.3.so >> >> #10 0x00007f683767db02 in __dbc_get_pp () from /lib64/libdb-5.3.so >> >> #11 0x00007f6832b59680 in idl_new_fetch () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #12 0x00007f6832b67b26 in index_read_ext_allids () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #13 0x00007f6832b52284 in keys2idl () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #14 0x00007f6832b529e3 in ava_candidates.isra.0 () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #15 0x00007f6832b52fd2 in filter_candidates_ext () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #16 0x00007f6832b54096 in list_candidates () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #17 0x00007f6832b52f40 in filter_candidates_ext () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #18 0x00007f6832b54096 in list_candidates () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #19 0x00007f6832b52f40 in filter_candidates_ext () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #20 0x00007f6832b54096 in list_candidates () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #21 0x00007f6832b52f40 in filter_candidates_ext () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #22 0x00007f6832b8e80c in subtree_candidates () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #23 0x00007f6832b8fe7f in ldbm_back_search () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #24 0x00007f683f4bd1d6 in op_shared_search () from >> /usr/lib64/dirsrv/libslapd.so.0 >> #25 0x00007f683f9ae33e in do_search () >> #26 0x00007f683f99c3c3 in connection_threadmain () >> #27 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so >> #28 0x00007f683d04cdc5 in start_thread () from /lib64/libpthread.so.0 >> #29 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 >> Thread 9 (Thread 0x7f67fafed700 (LWP 4884)): >> #0 0x00007f683d0506d5 in pthread_cond_wait@@GLIBC_2.3.2 () from >> /lib64/libpthread.so.0 >> #1 0x00007f683d6a6050 in PR_WaitCondVar () from /lib64/libnspr4.so >> #2 0x00007f683f999fee in connection_wait_for_new_work () >> #3 0x00007f683f99b719 in connection_threadmain () >> #4 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so >> #5 0x00007f683d04cdc5 in start_thread () from /lib64/libpthread.so.0 >> #6 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 >> Thread 8 (Thread 0x7f67fa7ec700 (LWP 4885)): >> #0 0x00007f683d0506d5 in pthread_cond_wait@@GLIBC_2.3.2 () from >> /lib64/libpthread.so.0 >> #1 0x00007f68375ac2f3 in __db_hybrid_mutex_suspend () from >> /lib64/libdb-5.3.so >> #2 0x00007f68375ab640 in __db_tas_mutex_lock () from >> /lib64/libdb-5.3.so >> #3 0x00007f6837655d3a in __lock_get_internal () from >> /lib64/libdb-5.3.so >> #4 0x00007f6837656820 in __lock_get () from /lib64/libdb-5.3.so >> >> #5 0x00007f6837682142 in __db_lget () from /lib64/libdb-5.3.so >> >> #6 0x00007f68375c9605 in __bam_search () from /lib64/libdb-5.3.so >> >> #7 0x00007f68375b4256 in __bamc_search () from /lib64/libdb-5.3.so >> >> #8 0x00007f68375b5d0f in __bamc_get () from /lib64/libdb-5.3.so >> >> #9 0x00007f683766eca6 in __dbc_iget () from /lib64/libdb-5.3.so >> >> #10 0x00007f683767db02 in __dbc_get_pp () from /lib64/libdb-5.3.so >> >> #11 0x00007f6832b59680 in idl_new_fetch () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #12 0x00007f6832b67b26 in index_read_ext_allids () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #13 0x00007f6832b52284 in keys2idl () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #14 0x00007f6832b529e3 in ava_candidates.isra.0 () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #15 0x00007f6832b52fd2 in filter_candidates_ext () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #16 0x00007f6832b54096 in list_candidates () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #17 0x00007f6832b52f40 in filter_candidates_ext () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #18 0x00007f6832b54096 in list_candidates () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #19 0x00007f6832b52f40 in filter_candidates_ext () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #20 0x00007f6832b54096 in list_candidates () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #21 0x00007f6832b52f40 in filter_candidates_ext () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #22 0x00007f6832b8e80c in subtree_candidates () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #23 0x00007f6832b8fe7f in ldbm_back_search () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #24 0x00007f683f4bd1d6 in op_shared_search () from >> /usr/lib64/dirsrv/libslapd.so.0 >> #25 0x00007f683f9ae33e in do_search () >> #26 0x00007f683f99c3c3 in connection_threadmain () >> #27 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so >> #28 0x00007f683d04cdc5 in start_thread () from /lib64/libpthread.so.0 >> #29 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 >> Thread 7 (Thread 0x7f67f9feb700 (LWP 4886)): >> #0 0x00007f683d0506d5 in pthread_cond_wait@@GLIBC_2.3.2 () from >> /lib64/libpthread.so.0 >> #1 0x00007f683d6a6050 in PR_WaitCondVar () from /lib64/libnspr4.so >> #2 0x00007f683f999fee in connection_wait_for_new_work () >> #3 0x00007f683f99b719 in connection_threadmain () >> #4 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so >> #5 0x00007f683d04cdc5 in start_thread () from /lib64/libpthread.so.0 >> #6 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 >> Thread 6 (Thread 0x7f67f97ea700 (LWP 4887)): >> #0 0x00007f683d0506d5 in pthread_cond_wait@@GLIBC_2.3.2 () from >> /lib64/libpthread.so.0 >> #1 0x00007f683d6a6050 in PR_WaitCondVar () from /lib64/libnspr4.so >> #2 0x00007f683f999fee in connection_wait_for_new_work () >> #3 0x00007f683f99b719 in connection_threadmain () >> #4 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so >> #5 0x00007f683d04cdc5 in start_thread () from /lib64/libpthread.so.0 >> #6 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 >> Thread 5 (Thread 0x7f67f8fe9700 (LWP 4888)): >> #0 0x00007f683d0506d5 in pthread_cond_wait@@GLIBC_2.3.2 () from >> /lib64/libpthread.so.0 >> #1 0x00007f683d6a6050 in PR_WaitCondVar () from /lib64/libnspr4.so >> #2 0x00007f683f999fee in connection_wait_for_new_work () >> #3 0x00007f683f99b719 in connection_threadmain () >> #4 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so >> #5 0x00007f683d04cdc5 in start_thread () from /lib64/libpthread.so.0 >> #6 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 >> Thread 4 (Thread 0x7f67f87e8700 (LWP 4889)): >> #0 0x00007f683cd719b3 in select () from /lib64/libc.so.6 >> #1 0x00007f683f4fc0e9 in DS_Sleep () from >> /usr/lib64/dirsrv/libslapd.so.0 >> #2 0x00007f683f99d325 in time_thread () >> #3 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so >> #4 0x00007f683d04cdc5 in start_thread () from /lib64/libpthread.so.0 >> #5 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 >> Thread 3 (Thread 0x7f67f7de6700 (LWP 4892)): >> #0 0x00007f683d0506d5 in pthread_cond_wait@@GLIBC_2.3.2 () from >> /lib64/libpthread.so.0 >> #1 0x00007f683d6a6050 in PR_WaitCondVar () from /lib64/libnspr4.so >> #2 0x00007f683f9a8f25 in ps_send_results () >> #3 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so >> #4 0x00007f683d04cdc5 in start_thread () from /lib64/libpthread.so.0 >> #5 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 >> Thread 2 (Thread 0x7f67f75e5700 (LWP 5084)): >> #0 0x00007f683d0506d5 in pthread_cond_wait@@GLIBC_2.3.2 () from >> /lib64/libpthread.so.0 >> #1 0x00007f683d6a6050 in PR_WaitCondVar () from /lib64/libnspr4.so >> #2 0x00007f683f9a8f25 in ps_send_results () >> #3 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so >> #4 0x00007f683d04cdc5 in start_thread () from /lib64/libpthread.so.0 >> #5 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 >> Thread 1 (Thread 0x7f683f964840 (LWP 4845)): >> #0 0x00007f683d0506d5 in pthread_cond_wait@@GLIBC_2.3.2 () from >> /lib64/libpthread.so.0 >> #1 0x00007f683d6a62c3 in PR_EnterMonitor () from /lib64/libnspr4.so >> #2 0x00007f683f99d5ed in setup_pr_read_pds () >> #3 0x00007f683f9a0fd9 in slapd_daemon () >> #4 0x00007f683f992c6c in main () >> >> >> >> the top output seems ok, there was just a sudden spike to around 340% >> cpu then it comes down >> >> top - 08:53:12 up 6 days, 23:20, 1 user, load average: 0.00, 0.01, 0.12 >> Threads: 44 total, 0 running, 44 sleeping, 0 stopped, 0 zombie >> %Cpu(s): 0.0 us, 0.0 sy, 0.0 ni,100.0 id, 0.0 wa, 0.0 hi, 0.0 >> si, 0.0 st >> KiB Mem : 65533176 total, 52165676 free, 6135088 used, 7232412 >> buff/cache >> KiB Swap: 3999996 total, 3999996 free, 0 used. 58452040 >> avail Mem >> >> PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ >> COMMAND >> 4845 dirsrv 20 0 5133196 3.150g 281272 S 0.0 5.0 31:34.51 >> ns-slapd >> 4847 dirsrv 20 0 5133196 3.150g 281272 S 0.0 5.0 0:13.53 >> ns-slapd >> 4848 dirsrv 20 0 5133196 3.150g 281272 S 0.0 5.0 0:13.16 >> ns-slapd >> 4849 dirsrv 20 0 5133196 3.150g 281272 S 0.0 5.0 0:45.27 >> ns-slapd >> 4850 dirsrv 20 0 5133196 3.150g 281272 S 0.0 5.0 0:01.60 >> ns-slapd >> 4851 dirsrv 20 0 5133196 3.150g 281272 S 0.0 5.0 0:00.00 >> ns-slapd >> 4852 dirsrv 20 0 5133196 3.150g 281272 S 0.0 5.0 0:00.00 >> ns-slapd >> 4853 dirsrv 20 0 5133196 3.150g 281272 S 0.0 5.0 0:00.00 >> ns-slapd >> 4854 dirsrv 20 0 5133196 3.150g 281272 S 0.0 5.0 0:00.00 >> ns-slapd >> 4855 dirsrv 20 0 5133196 3.150g 281272 S 0.0 5.0 0:03.02 >> ns-slapd >> 4856 dirsrv 20 0 5133196 3.150g 281272 S 0.0 5.0 0:00.26 >> ns-slapd >> 4859 dirsrv 20 0 5133196 3.150g 281272 S 0.0 5.0 3:18.67 >> ns-slapd >> 4860 dirsrv 20 0 5133196 3.150g 281272 S 0.0 5.0 3:19.82 >> ns-slapd >> 4861 dirsrv 20 0 5133196 3.150g 281272 S 0.0 5.0 3:20.23 >> ns-slapd >> 4862 dirsrv 20 0 5133196 3.150g 281272 S 0.0 5.0 3:22.65 >> ns-slapd >> 4863 dirsrv 20 0 5133196 3.150g 281272 S 0.0 5.0 3:16.64 >> ns-slapd >> 4864 dirsrv 20 0 5133196 3.150g 281272 S 0.0 5.0 3:17.39 >> ns-slapd >> 4865 dirsrv 20 0 5133196 3.150g 281272 S 0.0 5.0 3:17.80 >> ns-slapd >> >> >> Thanks >> Rakesh >> >> On Tue, Sep 6, 2016 at 12:09 AM, Rakesh Rajasekharan >> > > wrote: >> >> Hi Thierry/Ludwig, >> >> Unfortunately I had by then restarted the service and since then >> have been trying to hit the hang issue again but no luck this time >> >> It has been running fine for the last few hours now in-spite of >> trying all that I believe could cause the hang like making >> simultaneous client addition requests. >> >> I will continue testing this and next time I get it I will make >> sure I get the pstack and also get the dump of db lock info. >> >> Thanks, >> Rakesh >> >> >> >> On Mon, Sep 5, 2016 at 6:14 PM, Ludwig Krispenz >> wrote: >> >> >> On 09/05/2016 02:31 PM, thierry bordaz wrote: >>> Hi Rakesh, >>> >>> Thanks you soooo much for you continuous help on this !! >>> >>> If the hang is still going on, would you dump the DB lock >>> information with: >>> >>> db_stat -CA -N -h /var/lib/dirsrv/slapd-/db >>> >/tmp/db_lock_info >>> >>> It is looking like several threads are waiting for >>> one/several db locks. With the lock info we will know which >>> thread acquired it/them. >>> A guess is that it could be the checkpointing thread but >>> this one is also waiting (on a memory pool). >>> >>> I am copying Ludwig on this thread in case this pstack rings >>> any bell. >> the frequent presence of __db_tas_mutex_lock() looks like the >> NUMA problem of libdb, although the threads already have >> stopped spinning (maybe this was an effect of gcore). >> If the hang aoccurs again, could you check cpu consumption >> top -H -p >> and get a pstack >> pstack >>> >>> best regards >>> thierry >>> >>> On 09/05/2016 02:05 PM, Rakesh Rajasekharan wrote: >>>> Hi Thierry, >>>> >>>> I was able to now reporduce the issue.. >>>> >>>> the ns-slapd process just got hung,.. however, no core file >>>> was generate.. >>>> >>>> I manually created a coredump using gcore and then took a >>>> strace of it >>>> this is strace from the coredump generated >>>> >>>> GNU gdb (GDB) Red Hat Enterprise Linux 7.6.1-80.el7 >>>> Copyright (C) 2013 Free Software Foundation, Inc. >>>> License GPLv3+: GNU GPL version 3 or later >>>> >>>> This is free software: you are free to change and >>>> redistribute it. >>>> There is NO WARRANTY, to the extent permitted by law. Type >>>> "show copying" >>>> and "show warranty" for details. >>>> This GDB was configured as "x86_64-redhat-linux-gnu". >>>> For bug reporting instructions, please see: >>>> >>> >... >>>> Reading symbols from /usr/sbin/ns-slapd...Reading symbols >>>> from /usr/sbin/ns-slapd...(no debugging symbols found)...done. >>>> (no debugging symbols found)...done. >>>> [New LWP 105159] >>>> [New LWP 105160] >>>> [New LWP 105161] >>>> [New LWP 105162] >>>> [New LWP 105163] >>>> [New LWP 105164] >>>> [New LWP 105165] >>>> [New LWP 105166] >>>> [New LWP 105167] >>>> [New LWP 105168] >>>> [New LWP 105171] >>>> [New LWP 105172] >>>> [New LWP 105173] >>>> [New LWP 105174] >>>> [New LWP 105175] >>>> [New LWP 105176] >>>> [New LWP 105177] >>>> [New LWP 105178] >>>> [New LWP 105179] >>>> [New LWP 105180] >>>> [New LWP 105181] >>>> [New LWP 105182] >>>> [New LWP 105183] >>>> [New LWP 105184] >>>> [New LWP 105185] >>>> [New LWP 105186] >>>> [New LWP 105187] >>>> [New LWP 105188] >>>> [New LWP 105189] >>>> [New LWP 105190] >>>> [New LWP 105191] >>>> [New LWP 105192] >>>> [New LWP 105193] >>>> [New LWP 105194] >>>> [New LWP 105195] >>>> [New LWP 105196] >>>> [New LWP 105197] >>>> [New LWP 105198] >>>> [New LWP 105199] >>>> [New LWP 105200] >>>> [New LWP 105201] >>>> [New LWP 105203] >>>> [New LWP 105215] >>>> [New LWP 105158] >>>> [Thread debugging using libthread_db enabled] >>>> Using host libthread_db library "/lib64/libthread_db.so.1". >>>> Core was generated by `/usr/sbin/ns-slapd'. >>>> #0 0x00007f1b48ae69b3 in select () from /lib64/libc.so.6 >>>> >>>> Thread 44 (Thread 0x7f1b4b6d9840 (LWP 105158)): >>>> #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 () >>>> from /lib64/libpthread.so.0 >>>> No symbol table info available. >>>> #1 0x00007f1b4941b2c3 in PR_EnterMonitor () from >>>> /lib64/libnspr4.so >>>> No symbol table info available. >>>> #2 0x00007f1b4b7125ed in setup_pr_read_pds () >>>> No symbol table info available. >>>> #3 0x00007f1b4b715fd9 in slapd_daemon () >>>> No symbol table info available. >>>> #4 0x00007f1b4b707c6c in main () >>>> No symbol table info available. >>>> >>>> Thread 43 (Thread 0x7f1a5df0f700 (LWP 105215)): >>>> #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 () >>>> from /lib64/libpthread.so.0 >>>> No symbol table info available. >>>> #1 0x00007f1b4941b050 in PR_WaitCondVar () from >>>> /lib64/libnspr4.so >>>> No symbol table info available. >>>> #2 0x00007f1b4b71df25 in ps_send_results () >>>> No symbol table info available. >>>> #3 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so >>>> No symbol table info available. >>>> #4 0x00007f1b48dc1dc5 in start_thread () from >>>> /lib64/libpthread.so.0 >>>> No symbol table info available. >>>> #5 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 >>>> No symbol table info available. >>>> >>>> Thread 42 (Thread 0x7f1b03fe7700 (LWP 105203)): >>>> #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 () >>>> from /lib64/libpthread.so.0 >>>> No symbol table info available. >>>> #1 0x00007f1b4941b050 in PR_WaitCondVar () from >>>> /lib64/libnspr4.so >>>> No symbol table info available. >>>> #2 0x00007f1b4b71df25 in ps_send_results () >>>> No symbol table info available. >>>> #3 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so >>>> No symbol table info available. >>>> #4 0x00007f1b48dc1dc5 in start_thread () from >>>> /lib64/libpthread.so.0 >>>> No symbol table info available. >>>> #5 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 >>>> No symbol table info available. >>>> >>>> Thread 41 (Thread 0x7f1b047e8700 (LWP 105201)): >>>> #0 0x00007f1b48ae69b3 in select () from /lib64/libc.so.6 >>>> No symbol table info available. >>>> #1 0x00007f1b4b2710e9 in DS_Sleep () from >>>> /usr/lib64/dirsrv/libslapd.so.0 >>>> No symbol table info available. >>>> #2 0x00007f1b4b712325 in time_thread () >>>> No symbol table info available. >>>> #3 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so >>>> No symbol table info available. >>>> #4 0x00007f1b48dc1dc5 in start_thread () from >>>> /lib64/libpthread.so.0 >>>> No symbol table info available. >>>> #5 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 >>>> No symbol table info available. >>>> >>>> Thread 40 (Thread 0x7f1b04fe9700 (LWP 105200)): >>>> #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 () >>>> from /lib64/libpthread.so.0 >>>> No symbol table info available. >>>> #1 0x00007f1b433212f3 in __db_hybrid_mutex_suspend () from >>>> /lib64/libdb-5.3.so >>>> No symbol table info available. >>>> #2 0x00007f1b43320640 in __db_tas_mutex_lock () from >>>> /lib64/libdb-5.3.so >>>> No symbol table info available. >>>> #3 0x00007f1b4343ae1b in __memp_fget () from >>>> /lib64/libdb-5.3.so >>>> No symbol table info available. >>>> #4 0x00007f1b4333e211 in __bam_search () from >>>> /lib64/libdb-5.3.so >>>> No symbol table info available. >>>> #5 0x00007f1b43329256 in __bamc_search () from >>>> /lib64/libdb-5.3.so >>>> No symbol table info available. >>>> #6 0x00007f1b4332d6c4 in __bamc_put () from >>>> /lib64/libdb-5.3.so >>>> No symbol table info available. >>>> #7 0x00007f1b433e46d5 in __dbc_iput () from >>>> /lib64/libdb-5.3.so >>>> No symbol table info available. >>>> #8 0x00007f1b433dfa0e in __db_put () from >>>> /lib64/libdb-5.3.so >>>> No symbol table info available. >>>> #9 0x00007f1b433f4fa4 in __db_put_pp () from >>>> /lib64/libdb-5.3.so >>>> No symbol table info available. >>>> #10 0x00007f1b3e8cf4cf in idl_new_insert_key () from >>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>> No symbol table info available. >>>> #11 0x00007f1b3e8de5b5 in addordel_values_sv.isra.3 () from >>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>> No symbol table info available. >>>> #12 0x00007f1b3e8df25a in index_addordel_values_ext_sv () >>>> from /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>> No symbol table info available. >>>> #13 0x00007f1b3e8df694 in index_addordel_values_sv () from >>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>> No symbol table info available. >>>> #14 0x00007f1b3e8df75a in index_addordel_entry () from >>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>> No symbol table info available. >>>> #15 0x00007f1b3e8e25fa in ldbm_back_add () from >>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>> No symbol table info available. >>>> #16 0x00007f1b4b1e4cca in op_shared_add () from >>>> /usr/lib64/dirsrv/libslapd.so.0 >>>> No symbol table info available. >>>> #17 0x00007f1b4b1e6020 in do_add () from >>>> /usr/lib64/dirsrv/libslapd.so.0 >>>> No symbol table info available. >>>> #18 0x00007f1b4b711303 in connection_threadmain () >>>> No symbol table info available. >>>> #19 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so >>>> No symbol table info available. >>>> #20 0x00007f1b48dc1dc5 in start_thread () from >>>> /lib64/libpthread.so.0 >>>> No symbol table info available. >>>> #21 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 >>>> No symbol table info available. >>>> >>>> Thread 39 (Thread 0x7f1b057ea700 (LWP 105199)): >>>> #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 () >>>> from /lib64/libpthread.so.0 >>>> No symbol table info available. >>>> #1 0x00007f1b4941b050 in PR_WaitCondVar () from >>>> /lib64/libnspr4.so >>>> No symbol table info available. >>>> #2 0x00007f1b4b70efee in connection_wait_for_new_work () >>>> No symbol table info available. >>>> #3 0x00007f1b4b710719 in connection_threadmain () >>>> No symbol table info available. >>>> #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so >>>> No symbol table info available. >>>> #5 0x00007f1b48dc1dc5 in start_thread () from >>>> /lib64/libpthread.so.0 >>>> No symbol table info available. >>>> #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 >>>> No symbol table info available. >>>> >>>> Thread 38 (Thread 0x7f1b05feb700 (LWP 105198)): >>>> #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 () >>>> from /lib64/libpthread.so.0 >>>> No symbol table info available. >>>> #1 0x00007f1b4941b050 in PR_WaitCondVar () from >>>> /lib64/libnspr4.so >>>> No symbol table info available. >>>> #2 0x00007f1b4b70efee in connection_wait_for_new_work () >>>> No symbol table info available. >>>> #3 0x00007f1b4b710719 in connection_threadmain () >>>> No symbol table info available. >>>> #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so >>>> No symbol table info available. >>>> #5 0x00007f1b48dc1dc5 in start_thread () from >>>> /lib64/libpthread.so.0 >>>> No symbol table info available. >>>> #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 >>>> No symbol table info available. >>>> >>>> Thread 37 (Thread 0x7f1b067ec700 (LWP 105197)): >>>> #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 () >>>> from /lib64/libpthread.so.0 >>>> No symbol table info available. >>>> #1 0x00007f1b4941b050 in PR_WaitCondVar () from >>>> /lib64/libnspr4.so >>>> No symbol table info available. >>>> #2 0x00007f1b4b70efee in connection_wait_for_new_work () >>>> No symbol table info available. >>>> #3 0x00007f1b4b710719 in connection_threadmain () >>>> No symbol table info available. >>>> #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so >>>> No symbol table info available. >>>> #5 0x00007f1b48dc1dc5 in start_thread () from >>>> /lib64/libpthread.so.0 >>>> No symbol table info available. >>>> #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 >>>> No symbol table info available. >>>> >>>> Thread 36 (Thread 0x7f1b06fed700 (LWP 105196)): >>>> #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 () >>>> from /lib64/libpthread.so.0 >>>> No symbol table info available. >>>> #1 0x00007f1b4941b050 in PR_WaitCondVar () from >>>> /lib64/libnspr4.so >>>> No symbol table info available. >>>> #2 0x00007f1b4b70efee in connection_wait_for_new_work () >>>> No symbol table info available. >>>> #3 0x00007f1b4b710719 in connection_threadmain () >>>> No symbol table info available. >>>> #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so >>>> No symbol table info available. >>>> #5 0x00007f1b48dc1dc5 in start_thread () from >>>> /lib64/libpthread.so.0 >>>> No symbol table info available. >>>> #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 >>>> No symbol table info available. >>>> >>>> Thread 35 (Thread 0x7f1b077ee700 (LWP 105195)): >>>> #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 () >>>> from /lib64/libpthread.so.0 >>>> No symbol table info available. >>>> #1 0x00007f1b4941b2c3 in PR_EnterMonitor () from >>>> /lib64/libnspr4.so >>>> No symbol table info available. >>>> #2 0x00007f1b3e8c23e6 in dblayer_txn_begin () from >>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>> No symbol table info available. >>>> #3 0x00007f1b3e8fe007 in ldbm_back_modify () from >>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>> No symbol table info available. >>>> #4 0x00007f1b4b22b541 in op_shared_modify () from >>>> /usr/lib64/dirsrv/libslapd.so.0 >>>> No symbol table info available. >>>> #5 0x00007f1b4b22c004 in modify_internal_pb () from >>>> /usr/lib64/dirsrv/libslapd.so.0 >>>> No symbol table info available. >>>> #6 0x00007f1b3eb48b60 in ipapwd_apply_mods () from >>>> /usr/lib64/dirsrv/plugins/libipa_pwd_extop.so >>>> No symbol table info available. >>>> #7 0x00007f1b3eb4c8e0 in store_new_keys () from >>>> /usr/lib64/dirsrv/plugins/libipa_pwd_extop.so >>>> No symbol table info available. >>>> #8 0x00007f1b3eb4ead0 in ipapwd_extop () from >>>> /usr/lib64/dirsrv/plugins/libipa_pwd_extop.so >>>> No symbol table info available. >>>> #9 0x00007f1b4b23bba2 in plugin_call_exop_plugins () from >>>> /usr/lib64/dirsrv/libslapd.so.0 >>>> No symbol table info available. >>>> #10 0x00007f1b4b717efe in do_extended () >>>> No symbol table info available. >>>> #11 0x00007f1b4b71141a in connection_threadmain () >>>> No symbol table info available. >>>> #12 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so >>>> No symbol table info available. >>>> #13 0x00007f1b48dc1dc5 in start_thread () from >>>> /lib64/libpthread.so.0 >>>> No symbol table info available. >>>> #14 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 >>>> No symbol table info available. >>>> >>>> Thread 34 (Thread 0x7f1b07fef700 (LWP 105194)): >>>> #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 () >>>> from /lib64/libpthread.so.0 >>>> No symbol table info available. >>>> #1 0x00007f1b4941b050 in PR_WaitCondVar () from >>>> /lib64/libnspr4.so >>>> No symbol table info available. >>>> #2 0x00007f1b4b70efee in connection_wait_for_new_work () >>>> No symbol table info available. >>>> #3 0x00007f1b4b710719 in connection_threadmain () >>>> No symbol table info available. >>>> #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so >>>> No symbol table info available. >>>> #5 0x00007f1b48dc1dc5 in start_thread () from >>>> /lib64/libpthread.so.0 >>>> No symbol table info available. >>>> #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 >>>> No symbol table info available. >>>> >>>> Thread 33 (Thread 0x7f1b087f0700 (LWP 105193)): >>>> #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 () >>>> from /lib64/libpthread.so.0 >>>> No symbol table info available. >>>> #1 0x00007f1b4941b050 in PR_WaitCondVar () from >>>> /lib64/libnspr4.so >>>> No symbol table info available. >>>> #2 0x00007f1b4b70efee in connection_wait_for_new_work () >>>> No symbol table info available. >>>> #3 0x00007f1b4b710719 in connection_threadmain () >>>> No symbol table info available. >>>> #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so >>>> No symbol table info available. >>>> #5 0x00007f1b48dc1dc5 in start_thread () from >>>> /lib64/libpthread.so.0 >>>> No symbol table info available. >>>> #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 >>>> No symbol table info available. >>>> >>>> Thread 32 (Thread 0x7f1b08ff1700 (LWP 105192)): >>>> #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 () >>>> from /lib64/libpthread.so.0 >>>> No symbol table info available. >>>> #1 0x00007f1b4941b050 in PR_WaitCondVar () from >>>> /lib64/libnspr4.so >>>> No symbol table info available. >>>> #2 0x00007f1b4b70efee in connection_wait_for_new_work () >>>> No symbol table info available. >>>> #3 0x00007f1b4b710719 in connection_threadmain () >>>> No symbol table info available. >>>> #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so >>>> No symbol table info available. >>>> #5 0x00007f1b48dc1dc5 in start_thread () from >>>> /lib64/libpthread.so.0 >>>> No symbol table info available. >>>> #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 >>>> No symbol table info available. >>>> >>>> Thread 31 (Thread 0x7f1b097f2700 (LWP 105191)): >>>> #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 () >>>> from /lib64/libpthread.so.0 >>>> No symbol table info available. >>>> #1 0x00007f1b4941b050 in PR_WaitCondVar () from >>>> /lib64/libnspr4.so >>>> No symbol table info available. >>>> #2 0x00007f1b4b70efee in connection_wait_for_new_work () >>>> No symbol table info available. >>>> #3 0x00007f1b4b710719 in connection_threadmain () >>>> No symbol table info available. >>>> #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so >>>> No symbol table info available. >>>> #5 0x00007f1b48dc1dc5 in start_thread () from >>>> /lib64/libpthread.so.0 >>>> No symbol table info available. >>>> #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 >>>> No symbol table info available. >>>> >>>> Thread 30 (Thread 0x7f1b09ff3700 (LWP 105190)): >>>> #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 () >>>> from /lib64/libpthread.so.0 >>>> No symbol table info available. >>>> #1 0x00007f1b433212f3 in __db_hybrid_mutex_suspend () from >>>> /lib64/libdb-5.3.so >>>> No symbol table info available. >>>> #2 0x00007f1b43320640 in __db_tas_mutex_lock () from >>>> /lib64/libdb-5.3.so >>>> No symbol table info available. >>>> #3 0x00007f1b433cad3a in __lock_get_internal () from >>>> /lib64/libdb-5.3.so >>>> No symbol table info available. >>>> #4 0x00007f1b433cb820 in __lock_get () from >>>> /lib64/libdb-5.3.so >>>> No symbol table info available. >>>> #5 0x00007f1b433f7142 in __db_lget () from >>>> /lib64/libdb-5.3.so >>>> No symbol table info available. >>>> #6 0x00007f1b4333e605 in __bam_search () from >>>> /lib64/libdb-5.3.so >>>> No symbol table info available. >>>> #7 0x00007f1b43329256 in __bamc_search () from >>>> /lib64/libdb-5.3.so >>>> No symbol table info available. >>>> #8 0x00007f1b4332ad0f in __bamc_get () from >>>> /lib64/libdb-5.3.so >>>> No symbol table info available. >>>> #9 0x00007f1b433e3ca6 in __dbc_iget () from >>>> /lib64/libdb-5.3.so >>>> No symbol table info available. >>>> #10 0x00007f1b433f2b02 in __dbc_get_pp () from >>>> /lib64/libdb-5.3.so >>>> No symbol table info available. >>>> #11 0x00007f1b3e8ce680 in idl_new_fetch () from >>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>> No symbol table info available. >>>> #12 0x00007f1b3e8dcb26 in index_read_ext_allids () from >>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>> No symbol table info available. >>>> #13 0x00007f1b3e8c7284 in keys2idl () from >>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>> No symbol table info available. >>>> #14 0x00007f1b3e8c79e3 in ava_candidates.isra.0 () from >>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>> No symbol table info available. >>>> #15 0x00007f1b3e8c7fd2 in filter_candidates_ext () from >>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>> No symbol table info available. >>>> #16 0x00007f1b3e8c9096 in list_candidates () from >>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>> No symbol table info available. >>>> #17 0x00007f1b3e8c7f40 in filter_candidates_ext () from >>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>> No symbol table info available. >>>> #18 0x00007f1b3e8c9096 in list_candidates () from >>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>> No symbol table info available. >>>> #19 0x00007f1b3e8c7f40 in filter_candidates_ext () from >>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>> No symbol table info available. >>>> #20 0x00007f1b3e8c9096 in list_candidates () from >>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>> No symbol table info available. >>>> #21 0x00007f1b3e8c7f40 in filter_candidates_ext () from >>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>> No symbol table info available. >>>> #22 0x00007f1b3e90380c in subtree_candidates () from >>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>> No symbol table info available. >>>> #23 0x00007f1b3e904e7f in ldbm_back_search () from >>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>> No symbol table info available. >>>> #24 0x00007f1b4b2321d6 in op_shared_search () from >>>> /usr/lib64/dirsrv/libslapd.so.0 >>>> No symbol table info available. >>>> #25 0x00007f1b4b72333e in do_search () >>>> No symbol table info available. >>>> #26 0x00007f1b4b7113c3 in connection_threadmain () >>>> No symbol table info available. >>>> #27 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so >>>> No symbol table info available. >>>> #28 0x00007f1b48dc1dc5 in start_thread () from >>>> /lib64/libpthread.so.0 >>>> No symbol table info available. >>>> #29 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 >>>> No symbol table info available. >>>> >>>> Thread 29 (Thread 0x7f1b0a7f4700 (LWP 105189)): >>>> #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 () >>>> from /lib64/libpthread.so.0 >>>> No symbol table info available. >>>> #1 0x00007f1b4941b050 in PR_WaitCondVar () from >>>> /lib64/libnspr4.so >>>> No symbol table info available. >>>> #2 0x00007f1b4b70efee in connection_wait_for_new_work () >>>> No symbol table info available. >>>> #3 0x00007f1b4b710719 in connection_threadmain () >>>> No symbol table info available. >>>> #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so >>>> No symbol table info available. >>>> #5 0x00007f1b48dc1dc5 in start_thread () from >>>> /lib64/libpthread.so.0 >>>> No symbol table info available. >>>> #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 >>>> No symbol table info available. >>>> >>>> Thread 28 (Thread 0x7f1b0aff5700 (LWP 105188)): >>>> #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 () >>>> from /lib64/libpthread.so.0 >>>> No symbol table info available. >>>> #1 0x00007f1b4941b050 in PR_WaitCondVar () from >>>> /lib64/libnspr4.so >>>> No symbol table info available. >>>> #2 0x00007f1b4b70efee in connection_wait_for_new_work () >>>> No symbol table info available. >>>> #3 0x00007f1b4b710719 in connection_threadmain () >>>> No symbol table info available. >>>> #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so >>>> No symbol table info available. >>>> #5 0x00007f1b48dc1dc5 in start_thread () from >>>> /lib64/libpthread.so.0 >>>> No symbol table info available. >>>> #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 >>>> No symbol table info available. >>>> >>>> Thread 27 (Thread 0x7f1b0b7f6700 (LWP 105187)): >>>> #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 () >>>> from /lib64/libpthread.so.0 >>>> No symbol table info available. >>>> #1 0x00007f1b4941b050 in PR_WaitCondVar () from >>>> /lib64/libnspr4.so >>>> No symbol table info available. >>>> #2 0x00007f1b4b70efee in connection_wait_for_new_work () >>>> No symbol table info available. >>>> #3 0x00007f1b4b710719 in connection_threadmain () >>>> No symbol table info available. >>>> #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so >>>> No symbol table info available. >>>> #5 0x00007f1b48dc1dc5 in start_thread () from >>>> /lib64/libpthread.so.0 >>>> No symbol table info available. >>>> #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 >>>> No symbol table info available. >>>> >>>> Thread 26 (Thread 0x7f1b0bff7700 (LWP 105186)): >>>> #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 () >>>> from /lib64/libpthread.so.0 >>>> No symbol table info available. >>>> #1 0x00007f1b433212f3 in __db_hybrid_mutex_suspend () from >>>> /lib64/libdb-5.3.so >>>> No symbol table info available. >>>> #2 0x00007f1b43320640 in __db_tas_mutex_lock () from >>>> /lib64/libdb-5.3.so >>>> No symbol table info available. >>>> #3 0x00007f1b433cad3a in __lock_get_internal () from >>>> /lib64/libdb-5.3.so >>>> No symbol table info available. >>>> #4 0x00007f1b433cb820 in __lock_get () from >>>> /lib64/libdb-5.3.so >>>> No symbol table info available. >>>> #5 0x00007f1b433f7142 in __db_lget () from >>>> /lib64/libdb-5.3.so >>>> No symbol table info available. >>>> #6 0x00007f1b4333e605 in __bam_search () from >>>> /lib64/libdb-5.3.so >>>> No symbol table info available. >>>> #7 0x00007f1b43329256 in __bamc_search () from >>>> /lib64/libdb-5.3.so >>>> No symbol table info available. >>>> #8 0x00007f1b4332ad0f in __bamc_get () from >>>> /lib64/libdb-5.3.so >>>> No symbol table info available. >>>> #9 0x00007f1b433e3ca6 in __dbc_iget () from >>>> /lib64/libdb-5.3.so >>>> No symbol table info available. >>>> #10 0x00007f1b433f2b02 in __dbc_get_pp () from >>>> /lib64/libdb-5.3.so >>>> No symbol table info available. >>>> #11 0x00007f1b3e8ce680 in idl_new_fetch () from >>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>> No symbol table info available. >>>> #12 0x00007f1b3e8dcb26 in index_read_ext_allids () from >>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>> No symbol table info available. >>>> #13 0x00007f1b3e8c7284 in keys2idl () from >>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>> No symbol table info available. >>>> #14 0x00007f1b3e8c79e3 in ava_candidates.isra.0 () from >>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>> No symbol table info available. >>>> #15 0x00007f1b3e8c7fd2 in filter_candidates_ext () from >>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>> No symbol table info available. >>>> #16 0x00007f1b3e8c9096 in list_candidates () from >>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>> No symbol table info available. >>>> #17 0x00007f1b3e8c7f40 in filter_candidates_ext () from >>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>> No symbol table info available. >>>> #18 0x00007f1b3e90380c in subtree_candidates () from >>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>> No symbol table info available. >>>> #19 0x00007f1b3e904e7f in ldbm_back_search () from >>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>> No symbol table info available. >>>> #20 0x00007f1b4b2321d6 in op_shared_search () from >>>> /usr/lib64/dirsrv/libslapd.so.0 >>>> No symbol table info available. >>>> #21 0x00007f1b4b24294e in search_internal_callback_pb () >>>> from /usr/lib64/dirsrv/libslapd.so.0 >>>> No symbol table info available. >>>> #22 0x00007f1b4b242be8 in search_internal_pb () from >>>> /usr/lib64/dirsrv/libslapd.so.0 >>>> No symbol table info available. >>>> #23 0x00007f1b4b72149b in ids_sasl_canon_user () >>>> No symbol table info available. >>>> #24 0x00007f1b491e193b in _sasl_canon_user () from >>>> /lib64/libsasl2.so.3 >>>> No symbol table info available. >>>> #25 0x00007f1b491e1c4c in _sasl_canon_user_lookup () from >>>> /lib64/libsasl2.so.3 >>>> No symbol table info available. >>>> #26 0x00007f1b430e9fb9 in gssapi_server_mech_step () from >>>> /usr/lib64/sasl2/libgssapiv2.so >>>> No symbol table info available. >>>> #27 0x00007f1b491edb9b in sasl_server_step () from >>>> /lib64/libsasl2.so.3 >>>> No symbol table info available. >>>> #28 0x00007f1b4b72283b in ids_sasl_check_bind () >>>> No symbol table info available. >>>> #29 0x00007f1b4b70a61a in do_bind () >>>> No symbol table info available. >>>> #30 0x00007f1b4b7113fd in connection_threadmain () >>>> No symbol table info available. >>>> #31 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so >>>> No symbol table info available. >>>> #32 0x00007f1b48dc1dc5 in start_thread () from >>>> /lib64/libpthread.so.0 >>>> No symbol table info available. >>>> #33 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 >>>> No symbol table info available. >>>> >>>> Thread 25 (Thread 0x7f1b0c7f8700 (LWP 105185)): >>>> #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 () >>>> from /lib64/libpthread.so.0 >>>> No symbol table info available. >>>> #1 0x00007f1b4941b050 in PR_WaitCondVar () from >>>> /lib64/libnspr4.so >>>> No symbol table info available. >>>> #2 0x00007f1b4b70efee in connection_wait_for_new_work () >>>> No symbol table info available. >>>> #3 0x00007f1b4b710719 in connection_threadmain () >>>> No symbol table info available. >>>> #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so >>>> No symbol table info available. >>>> #5 0x00007f1b48dc1dc5 in start_thread () from >>>> /lib64/libpthread.so.0 >>>> No symbol table info available. >>>> #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 >>>> No symbol table info available. >>>> >>>> Thread 24 (Thread 0x7f1b0cff9700 (LWP 105184)): >>>> #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 () >>>> from /lib64/libpthread.so.0 >>>> No symbol table info available. >>>> #1 0x00007f1b4941b050 in PR_WaitCondVar () from >>>> /lib64/libnspr4.so >>>> No symbol table info available. >>>> #2 0x00007f1b4b70efee in connection_wait_for_new_work () >>>> No symbol table info available. >>>> #3 0x00007f1b4b710719 in connection_threadmain () >>>> No symbol table info available. >>>> #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so >>>> No symbol table info available. >>>> #5 0x00007f1b48dc1dc5 in start_thread () from >>>> /lib64/libpthread.so.0 >>>> No symbol table info available. >>>> #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 >>>> No symbol table info available. >>>> >>>> Thread 23 (Thread 0x7f1b0d7fa700 (LWP 105183)): >>>> #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 () >>>> from /lib64/libpthread.so.0 >>>> No symbol table info available. >>>> #1 0x00007f1b4941b050 in PR_WaitCondVar () from >>>> /lib64/libnspr4.so >>>> No symbol table info available. >>>> #2 0x00007f1b4b70efee in connection_wait_for_new_work () >>>> No symbol table info available. >>>> #3 0x00007f1b4b710719 in connection_threadmain () >>>> No symbol table info available. >>>> #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so >>>> No symbol table info available. >>>> #5 0x00007f1b48dc1dc5 in start_thread () from >>>> /lib64/libpthread.so.0 >>>> No symbol table info available. >>>> #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 >>>> No symbol table info available. >>>> >>>> Thread 22 (Thread 0x7f1b0dffb700 (LWP 105182)): >>>> #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 () >>>> from /lib64/libpthread.so.0 >>>> No symbol table info available. >>>> #1 0x00007f1b4941b050 in PR_WaitCondVar () from >>>> /lib64/libnspr4.so >>>> No symbol table info available. >>>> #2 0x00007f1b4b70efee in connection_wait_for_new_work () >>>> No symbol table info available. >>>> #3 0x00007f1b4b710719 in connection_threadmain () >>>> No symbol table info available. >>>> #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so >>>> No symbol table info available. >>>> #5 0x00007f1b48dc1dc5 in start_thread () from >>>> /lib64/libpthread.so.0 >>>> No symbol table info available. >>>> #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 >>>> No symbol table info available. >>>> >>>> Thread 21 (Thread 0x7f1b0e7fc700 (LWP 105181)): >>>> #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 () >>>> from /lib64/libpthread.so.0 >>>> No symbol table info available. >>>> #1 0x00007f1b4941b050 in PR_WaitCondVar () from >>>> /lib64/libnspr4.so >>>> No symbol table info available. >>>> #2 0x00007f1b4b70efee in connection_wait_for_new_work () >>>> No symbol table info available. >>>> #3 0x00007f1b4b710719 in connection_threadmain () >>>> No symbol table info available. >>>> #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so >>>> No symbol table info available. >>>> #5 0x00007f1b48dc1dc5 in start_thread () from >>>> /lib64/libpthread.so.0 >>>> No symbol table info available. >>>> #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 >>>> No symbol table info available. >>>> >>>> Thread 20 (Thread 0x7f1b0effd700 (LWP 105180)): >>>> #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 () >>>> from /lib64/libpthread.so.0 >>>> No symbol table info available. >>>> #1 0x00007f1b433212f3 in __db_hybrid_mutex_suspend () from >>>> /lib64/libdb-5.3.so >>>> No symbol table info available. >>>> #2 0x00007f1b43320b37 in __db_tas_mutex_readlock () from >>>> /lib64/libdb-5.3.so >>>> No symbol table info available. >>>> #3 0x00007f1b4343ad35 in __memp_fget () from >>>> /lib64/libdb-5.3.so >>>> No symbol table info available. >>>> #4 0x00007f1b433e4333 in __dbc_iget () from >>>> /lib64/libdb-5.3.so >>>> No symbol table info available. >>>> #5 0x00007f1b433f2b02 in __dbc_get_pp () from >>>> /lib64/libdb-5.3.so >>>> No symbol table info available. >>>> #6 0x00007f1b3e8ce822 in idl_new_fetch () from >>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>> No symbol table info available. >>>> #7 0x00007f1b3e8dcb26 in index_read_ext_allids () from >>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>> No symbol table info available. >>>> #8 0x00007f1b3e8c7284 in keys2idl () from >>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>> No symbol table info available. >>>> #9 0x00007f1b3e8c79e3 in ava_candidates.isra.0 () from >>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>> No symbol table info available. >>>> #10 0x00007f1b3e8c7fd2 in filter_candidates_ext () from >>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>> No symbol table info available. >>>> #11 0x00007f1b3e8c9096 in list_candidates () from >>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>> No symbol table info available. >>>> #12 0x00007f1b3e8c7f40 in filter_candidates_ext () from >>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>> No symbol table info available. >>>> #13 0x00007f1b3e8c9096 in list_candidates () from >>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>> No symbol table info available. >>>> #14 0x00007f1b3e8c7f40 in filter_candidates_ext () from >>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>> No symbol table info available. >>>> #15 0x00007f1b3e8c9096 in list_candidates () from >>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>> No symbol table info available. >>>> #16 0x00007f1b3e8c7f40 in filter_candidates_ext () from >>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>> No symbol table info available. >>>> #17 0x00007f1b3e90380c in subtree_candidates () from >>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>> No symbol table info available. >>>> #18 0x00007f1b3e904e7f in ldbm_back_search () from >>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>> No symbol table info available. >>>> #19 0x00007f1b4b2321d6 in op_shared_search () from >>>> /usr/lib64/dirsrv/libslapd.so.0 >>>> No symbol table info available. >>>> #20 0x00007f1b4b72333e in do_search () >>>> No symbol table info available. >>>> #21 0x00007f1b4b7113c3 in connection_threadmain () >>>> No symbol table info available. >>>> #22 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so >>>> No symbol table info available. >>>> #23 0x00007f1b48dc1dc5 in start_thread () from >>>> /lib64/libpthread.so.0 >>>> No symbol table info available. >>>> #24 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 >>>> No symbol table info available. >>>> >>>> Thread 19 (Thread 0x7f1b0f7fe700 (LWP 105179)): >>>> #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 () >>>> from /lib64/libpthread.so.0 >>>> No symbol table info available. >>>> #1 0x00007f1b433212f3 in __db_hybrid_mutex_suspend () from >>>> /lib64/libdb-5.3.so >>>> No symbol table info available. >>>> #2 0x00007f1b43320640 in __db_tas_mutex_lock () from >>>> /lib64/libdb-5.3.so >>>> No symbol table info available. >>>> #3 0x00007f1b433cad3a in __lock_get_internal () from >>>> /lib64/libdb-5.3.so >>>> No symbol table info available. >>>> #4 0x00007f1b433cb820 in __lock_get () from >>>> /lib64/libdb-5.3.so >>>> No symbol table info available. >>>> #5 0x00007f1b433f7142 in __db_lget () from >>>> /lib64/libdb-5.3.so >>>> No symbol table info available. >>>> #6 0x00007f1b4333e605 in __bam_search () from >>>> /lib64/libdb-5.3.so >>>> No symbol table info available. >>>> #7 0x00007f1b43329256 in __bamc_search () from >>>> /lib64/libdb-5.3.so >>>> No symbol table info available. >>>> #8 0x00007f1b4332ad0f in __bamc_get () from >>>> /lib64/libdb-5.3.so >>>> No symbol table info available. >>>> #9 0x00007f1b433e3ca6 in __dbc_iget () from >>>> /lib64/libdb-5.3.so >>>> No symbol table info available. >>>> #10 0x00007f1b433f2b02 in __dbc_get_pp () from >>>> /lib64/libdb-5.3.so >>>> No symbol table info available. >>>> #11 0x00007f1b3e8ce680 in idl_new_fetch () from >>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>> No symbol table info available. >>>> #12 0x00007f1b3e8dcb26 in index_read_ext_allids () from >>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>> No symbol table info available. >>>> #13 0x00007f1b3e8c7284 in keys2idl () from >>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>> No symbol table info available. >>>> #14 0x00007f1b3e8c79e3 in ava_candidates.isra.0 () from >>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>> No symbol table info available. >>>> #15 0x00007f1b3e8c7fd2 in filter_candidates_ext () from >>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>> No symbol table info available. >>>> #16 0x00007f1b3e8c9096 in list_candidates () from >>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>> No symbol table info available. >>>> #17 0x00007f1b3e8c7f40 in filter_candidates_ext () from >>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>> No symbol table info available. >>>> #18 0x00007f1b3e8c9096 in list_candidates () from >>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>> No symbol table info available. >>>> #19 0x00007f1b3e8c7f40 in filter_candidates_ext () from >>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>> No symbol table info available. >>>> #20 0x00007f1b3e8c9096 in list_candidates () from >>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>> No symbol table info available. >>>> #21 0x00007f1b3e8c7f40 in filter_candidates_ext () from >>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>> No symbol table info available. >>>> #22 0x00007f1b3e90380c in subtree_candidates () from >>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>> No symbol table info available. >>>> #23 0x00007f1b3e904e7f in ldbm_back_search () from >>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>> No symbol table info available. >>>> #24 0x00007f1b4b2321d6 in op_shared_search () from >>>> /usr/lib64/dirsrv/libslapd.so.0 >>>> No symbol table info available. >>>> #25 0x00007f1b4b72333e in do_search () >>>> No symbol table info available. >>>> #26 0x00007f1b4b7113c3 in connection_threadmain () >>>> No symbol table info available. >>>> #27 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so >>>> No symbol table info available. >>>> #28 0x00007f1b48dc1dc5 in start_thread () from >>>> /lib64/libpthread.so.0 >>>> No symbol table info available. >>>> #29 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 >>>> No symbol table info available. >>>> >>>> Thread 18 (Thread 0x7f1b0ffff700 (LWP 105178)): >>>> #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 () >>>> from /lib64/libpthread.so.0 >>>> No symbol table info available. >>>> #1 0x00007f1b4941b050 in PR_WaitCondVar () from >>>> /lib64/libnspr4.so >>>> No symbol table info available. >>>> #2 0x00007f1b4b70efee in connection_wait_for_new_work () >>>> No symbol table info available. >>>> #3 0x00007f1b4b710719 in connection_threadmain () >>>> No symbol table info available. >>>> #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so >>>> No symbol table info available. >>>> #5 0x00007f1b48dc1dc5 in start_thread () from >>>> /lib64/libpthread.so.0 >>>> No symbol table info available. >>>> #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 >>>> No symbol table info available. >>>> >>>> Thread 17 (Thread 0x7f1b14d28700 (LWP 105177)): >>>> #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 () >>>> from /lib64/libpthread.so.0 >>>> No symbol table info available. >>>> #1 0x00007f1b4941b050 in PR_WaitCondVar () from >>>> /lib64/libnspr4.so >>>> No symbol table info available. >>>> #2 0x00007f1b4b70efee in connection_wait_for_new_work () >>>> No symbol table info available. >>>> #3 0x00007f1b4b710719 in connection_threadmain () >>>> No symbol table info available. >>>> #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so >>>> No symbol table info available. >>>> #5 0x00007f1b48dc1dc5 in start_thread () from >>>> /lib64/libpthread.so.0 >>>> No symbol table info available. >>>> #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 >>>> No symbol table info available. >>>> >>>> Thread 16 (Thread 0x7f1b15529700 (LWP 105176)): >>>> #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 () >>>> from /lib64/libpthread.so.0 >>>> No symbol table info available. >>>> #1 0x00007f1b4941b050 in PR_WaitCondVar () from >>>> /lib64/libnspr4.so >>>> No symbol table info available. >>>> #2 0x00007f1b4b70efee in connection_wait_for_new_work () >>>> No symbol table info available. >>>> #3 0x00007f1b4b710719 in connection_threadmain () >>>> No symbol table info available. >>>> #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so >>>> No symbol table info available. >>>> #5 0x00007f1b48dc1dc5 in start_thread () from >>>> /lib64/libpthread.so.0 >>>> No symbol table info available. >>>> #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 >>>> No symbol table info available. >>>> >>>> Thread 15 (Thread 0x7f1b15d2a700 (LWP 105175)): >>>> #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 () >>>> from /lib64/libpthread.so.0 >>>> No symbol table info available. >>>> #1 0x00007f1b4941b050 in PR_WaitCondVar () from >>>> /lib64/libnspr4.so >>>> No symbol table info available. >>>> #2 0x00007f1b4b70efee in connection_wait_for_new_work () >>>> No symbol table info available. >>>> #3 0x00007f1b4b710719 in connection_threadmain () >>>> No symbol table info available. >>>> #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so >>>> No symbol table info available. >>>> #5 0x00007f1b48dc1dc5 in start_thread () from >>>> /lib64/libpthread.so.0 >>>> No symbol table info available. >>>> #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 >>>> No symbol table info available. >>>> >>>> Thread 14 (Thread 0x7f1b1652b700 (LWP 105174)): >>>> #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 () >>>> from /lib64/libpthread.so.0 >>>> No symbol table info available. >>>> #1 0x00007f1b4941b050 in PR_WaitCondVar () from >>>> /lib64/libnspr4.so >>>> No symbol table info available. >>>> #2 0x00007f1b4b70efee in connection_wait_for_new_work () >>>> No symbol table info available. >>>> #3 0x00007f1b4b710719 in connection_threadmain () >>>> No symbol table info available. >>>> #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so >>>> No symbol table info available. >>>> #5 0x00007f1b48dc1dc5 in start_thread () from >>>> /lib64/libpthread.so.0 >>>> No symbol table info available. >>>> #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 >>>> No symbol table info available. >>>> >>>> Thread 13 (Thread 0x7f1b16d2c700 (LWP 105173)): >>>> #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 () >>>> from /lib64/libpthread.so.0 >>>> No symbol table info available. >>>> #1 0x00007f1b433212f3 in __db_hybrid_mutex_suspend () from >>>> /lib64/libdb-5.3.so >>>> No symbol table info available. >>>> #2 0x00007f1b43320640 in __db_tas_mutex_lock () from >>>> /lib64/libdb-5.3.so >>>> No symbol table info available. >>>> #3 0x00007f1b433cad3a in __lock_get_internal () from >>>> /lib64/libdb-5.3.so >>>> No symbol table info available. >>>> #4 0x00007f1b433cb820 in __lock_get () from >>>> /lib64/libdb-5.3.so >>>> No symbol table info available. >>>> #5 0x00007f1b433f7142 in __db_lget () from >>>> /lib64/libdb-5.3.so >>>> No symbol table info available. >>>> #6 0x00007f1b4333e605 in __bam_search () from >>>> /lib64/libdb-5.3.so >>>> No symbol table info available. >>>> #7 0x00007f1b43329256 in __bamc_search () from >>>> /lib64/libdb-5.3.so >>>> No symbol table info available. >>>> #8 0x00007f1b4332ad0f in __bamc_get () from >>>> /lib64/libdb-5.3.so >>>> No symbol table info available. >>>> #9 0x00007f1b433e3ca6 in __dbc_iget () from >>>> /lib64/libdb-5.3.so >>>> No symbol table info available. >>>> #10 0x00007f1b433f2b02 in __dbc_get_pp () from >>>> /lib64/libdb-5.3.so >>>> No symbol table info available. >>>> #11 0x00007f1b3e8ce680 in idl_new_fetch () from >>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>> No symbol table info available. >>>> #12 0x00007f1b3e8dcb26 in index_read_ext_allids () from >>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>> No symbol table info available. >>>> #13 0x00007f1b3e8c7284 in keys2idl () from >>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>> No symbol table info available. >>>> #14 0x00007f1b3e8c79e3 in ava_candidates.isra.0 () from >>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>> No symbol table info available. >>>> #15 0x00007f1b3e8c7fd2 in filter_candidates_ext () from >>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>> No symbol table info available. >>>> #16 0x00007f1b3e8c9096 in list_candidates () from >>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>> No symbol table info available. >>>> #17 0x00007f1b3e8c7f40 in filter_candidates_ext () from >>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>> No symbol table info available. >>>> #18 0x00007f1b3e8c9096 in list_candidates () from >>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>> No symbol table info available. >>>> #19 0x00007f1b3e8c7f40 in filter_candidates_ext () from >>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>> No symbol table info available. >>>> #20 0x00007f1b3e8c9096 in list_candidates () from >>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>> No symbol table info available. >>>> #21 0x00007f1b3e8c7f40 in filter_candidates_ext () from >>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>> No symbol table info available. >>>> #22 0x00007f1b3e90380c in subtree_candidates () from >>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>> No symbol table info available. >>>> #23 0x00007f1b3e904e7f in ldbm_back_search () from >>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>> No symbol table info available. >>>> #24 0x00007f1b4b2321d6 in op_shared_search () from >>>> /usr/lib64/dirsrv/libslapd.so.0 >>>> No symbol table info available. >>>> #25 0x00007f1b4b72333e in do_search () >>>> No symbol table info available. >>>> #26 0x00007f1b4b7113c3 in connection_threadmain () >>>> No symbol table info available. >>>> #27 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so >>>> No symbol table info available. >>>> #28 0x00007f1b48dc1dc5 in start_thread () from >>>> /lib64/libpthread.so.0 >>>> No symbol table info available. >>>> #29 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 >>>> No symbol table info available. >>>> >>>> Thread 12 (Thread 0x7f1b1752d700 (LWP 105172)): >>>> #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 () >>>> from /lib64/libpthread.so.0 >>>> No symbol table info available. >>>> #1 0x00007f1b4941b050 in PR_WaitCondVar () from >>>> /lib64/libnspr4.so >>>> No symbol table info available. >>>> #2 0x00007f1b4b70efee in connection_wait_for_new_work () >>>> No symbol table info available. >>>> #3 0x00007f1b4b710719 in connection_threadmain () >>>> No symbol table info available. >>>> #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so >>>> No symbol table info available. >>>> #5 0x00007f1b48dc1dc5 in start_thread () from >>>> /lib64/libpthread.so.0 >>>> No symbol table info available. >>>> #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 >>>> No symbol table info available. >>>> >>>> Thread 11 (Thread 0x7f1b17d2e700 (LWP 105171)): >>>> #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 () >>>> from /lib64/libpthread.so.0 >>>> No symbol table info available. >>>> #1 0x00007f1b4941b050 in PR_WaitCondVar () from >>>> /lib64/libnspr4.so >>>> No symbol table info available. >>>> #2 0x00007f1b4b70efee in connection_wait_for_new_work () >>>> No symbol table info available. >>>> #3 0x00007f1b4b710719 in connection_threadmain () >>>> No symbol table info available. >>>> #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so >>>> No symbol table info available. >>>> #5 0x00007f1b48dc1dc5 in start_thread () from >>>> /lib64/libpthread.so.0 >>>> No symbol table info available. >>>> #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 >>>> No symbol table info available. >>>> >>>> Thread 10 (Thread 0x7f1b2509d700 (LWP 105168)): >>>> #0 0x00007f1b48dc5a82 in >>>> pthread_cond_timedwait@@GLIBC_2.3.2 >>>> () from >>>> /lib64/libpthread.so.0 >>>> No symbol table info available. >>>> #1 0x00007f1b4941ab07 in pt_TimedWait () from >>>> /lib64/libnspr4.so >>>> No symbol table info available. >>>> #2 0x00007f1b4941afce in PR_WaitCondVar () from >>>> /lib64/libnspr4.so >>>> No symbol table info available. >>>> #3 0x00007f1b4b2061e8 in eq_loop () from >>>> /usr/lib64/dirsrv/libslapd.so.0 >>>> No symbol table info available. >>>> #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so >>>> No symbol table info available. >>>> #5 0x00007f1b48dc1dc5 in start_thread () from >>>> /lib64/libpthread.so.0 >>>> No symbol table info available. >>>> #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 >>>> No symbol table info available. >>>> >>>> Thread 9 (Thread 0x7f1b2589e700 (LWP 105167)): >>>> #0 0x00007f1b48dc5a82 in >>>> pthread_cond_timedwait@@GLIBC_2.3.2 >>>> () from >>>> /lib64/libpthread.so.0 >>>> No symbol table info available. >>>> #1 0x00007f1b4941ab07 in pt_TimedWait () from >>>> /lib64/libnspr4.so >>>> No symbol table info available. >>>> #2 0x00007f1b4941afce in PR_WaitCondVar () from >>>> /lib64/libnspr4.so >>>> No symbol table info available. >>>> #3 0x00007f1b4b719d03 in housecleaning () >>>> No symbol table info available. >>>> #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so >>>> No symbol table info available. >>>> #5 0x00007f1b48dc1dc5 in start_thread () from >>>> /lib64/libpthread.so.0 >>>> No symbol table info available. >>>> #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 >>>> No symbol table info available. >>>> >>>> Thread 8 (Thread 0x7f1b2609f700 (LWP 105166)): >>>> #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 () >>>> from /lib64/libpthread.so.0 >>>> No symbol table info available. >>>> #1 0x00007f1b4941b050 in PR_WaitCondVar () from >>>> /lib64/libnspr4.so >>>> No symbol table info available. >>>> #2 0x00007f1b4b260198 in slapi_wait_condvar () from >>>> /usr/lib64/dirsrv/libslapd.so.0 >>>> No symbol table info available. >>>> #3 0x00007f1b3cf5deed in roles_cache_wait_on_change () >>>> from /usr/lib64/dirsrv/plugins/libroles-plugin.so >>>> No symbol table info available. >>>> #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so >>>> No symbol table info available. >>>> #5 0x00007f1b48dc1dc5 in start_thread () from >>>> /lib64/libpthread.so.0 >>>> No symbol table info available. >>>> #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 >>>> No symbol table info available. >>>> >>>> Thread 7 (Thread 0x7f1b268a0700 (LWP 105165)): >>>> #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 () >>>> from /lib64/libpthread.so.0 >>>> No symbol table info available. >>>> #1 0x00007f1b4941b050 in PR_WaitCondVar () from >>>> /lib64/libnspr4.so >>>> No symbol table info available. >>>> #2 0x00007f1b4b260198 in slapi_wait_condvar () from >>>> /usr/lib64/dirsrv/libslapd.so.0 >>>> No symbol table info available. >>>> #3 0x00007f1b3cf5deed in roles_cache_wait_on_change () >>>> from /usr/lib64/dirsrv/plugins/libroles-plugin.so >>>> No symbol table info available. >>>> #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so >>>> No symbol table info available. >>>> #5 0x00007f1b48dc1dc5 in start_thread () from >>>> /lib64/libpthread.so.0 >>>> No symbol table info available. >>>> #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 >>>> No symbol table info available. >>>> >>>> Thread 6 (Thread 0x7f1b270a1700 (LWP 105164)): >>>> #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 () >>>> from /lib64/libpthread.so.0 >>>> No symbol table info available. >>>> #1 0x00007f1b4941b050 in PR_WaitCondVar () from >>>> /lib64/libnspr4.so >>>> No symbol table info available. >>>> #2 0x00007f1b4b260198 in slapi_wait_condvar () from >>>> /usr/lib64/dirsrv/libslapd.so.0 >>>> No symbol table info available. >>>> #3 0x00007f1b3cf5deed in roles_cache_wait_on_change () >>>> from /usr/lib64/dirsrv/plugins/libroles-plugin.so >>>> No symbol table info available. >>>> #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so >>>> No symbol table info available. >>>> #5 0x00007f1b48dc1dc5 in start_thread () from >>>> /lib64/libpthread.so.0 >>>> No symbol table info available. >>>> #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 >>>> No symbol table info available. >>>> >>>> Thread 5 (Thread 0x7f1b278a2700 (LWP 105163)): >>>> #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 () >>>> from /lib64/libpthread.so.0 >>>> No symbol table info available. >>>> #1 0x00007f1b4941b050 in PR_WaitCondVar () from >>>> /lib64/libnspr4.so >>>> No symbol table info available. >>>> #2 0x00007f1b4b260198 in slapi_wait_condvar () from >>>> /usr/lib64/dirsrv/libslapd.so.0 >>>> No symbol table info available. >>>> #3 0x00007f1b415f262e in cos_cache_wait_on_change () from >>>> /usr/lib64/dirsrv/plugins/libcos-plugin.so >>>> No symbol table info available. >>>> #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so >>>> No symbol table info available. >>>> #5 0x00007f1b48dc1dc5 in start_thread () from >>>> /lib64/libpthread.so.0 >>>> No symbol table info available. >>>> #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 >>>> No symbol table info available. >>>> >>>> Thread 4 (Thread 0x7f1b282ac700 (LWP 105162)): >>>> #0 0x00007f1b48ae69b3 in select () from /lib64/libc.so.6 >>>> No symbol table info available. >>>> #1 0x00007f1b4b2710e9 in DS_Sleep () from >>>> /usr/lib64/dirsrv/libslapd.so.0 >>>> No symbol table info available. >>>> #2 0x00007f1b3e8ba707 in perf_threadmain () from >>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>> No symbol table info available. >>>> #3 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so >>>> No symbol table info available. >>>> #4 0x00007f1b48dc1dc5 in start_thread () from >>>> /lib64/libpthread.so.0 >>>> No symbol table info available. >>>> #5 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 >>>> No symbol table info available. >>>> >>>> Thread 3 (Thread 0x7f1b28aad700 (LWP 105161)): >>>> #0 0x00007f1b48ae69b3 in select () from /lib64/libc.so.6 >>>> No symbol table info available. >>>> #1 0x00007f1b4b2710e9 in DS_Sleep () from >>>> /usr/lib64/dirsrv/libslapd.so.0 >>>> No symbol table info available. >>>> #2 0x00007f1b3e8bfb7f in trickle_threadmain () from >>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>> No symbol table info available. >>>> #3 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so >>>> No symbol table info available. >>>> #4 0x00007f1b48dc1dc5 in start_thread () from >>>> /lib64/libpthread.so.0 >>>> No symbol table info available. >>>> #5 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 >>>> No symbol table info available. >>>> >>>> Thread 2 (Thread 0x7f1b292ae700 (LWP 105160)): >>>> #0 0x00007f1b48ae69b3 in select () from /lib64/libc.so.6 >>>> No symbol table info available. >>>> #1 0x00007f1b4344dbad in __os_yield () from >>>> /lib64/libdb-5.3.so >>>> No symbol table info available. >>>> #2 0x00007f1b434492b3 in __memp_sync_int () from >>>> /lib64/libdb-5.3.so >>>> No symbol table info available. >>>> #3 0x00007f1b43459752 in __txn_checkpoint () from >>>> /lib64/libdb-5.3.so >>>> No symbol table info available. >>>> #4 0x00007f1b43459b74 in __txn_checkpoint_pp () from >>>> /lib64/libdb-5.3.so >>>> No symbol table info available. >>>> #5 0x00007f1b3e8c3a87 in checkpoint_threadmain () from >>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>> No symbol table info available. >>>> #6 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so >>>> No symbol table info available. >>>> #7 0x00007f1b48dc1dc5 in start_thread () from >>>> /lib64/libpthread.so.0 >>>> No symbol table info available. >>>> #8 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 >>>> No symbol table info available. >>>> >>>> Thread 1 (Thread 0x7f1b29aaf700 (LWP 105159)): >>>> #0 0x00007f1b48ae69b3 in select () from /lib64/libc.so.6 >>>> No symbol table info available. >>>> #1 0x00007f1b4b2710e9 in DS_Sleep () from >>>> /usr/lib64/dirsrv/libslapd.so.0 >>>> No symbol table info available. >>>> #2 0x00007f1b3e8bf907 in deadlock_threadmain () from >>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>> No symbol table info available. >>>> #3 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so >>>> No symbol table info available. >>>> #4 0x00007f1b48dc1dc5 in start_thread () from >>>> /lib64/libpthread.so.0 >>>> No symbol table info available. >>>> #5 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 >>>> No symbol table info available. >>>> >>>> >>>> >>>> Thanks, >>>> >>>> Rakesh >>>> >>>> >>>> On Mon, Sep 5, 2016 at 4:48 PM, thierry bordaz >>>> wrote: >>>> >>>> >>>> >>>> On 09/05/2016 12:05 PM, Rakesh Rajasekharan wrote: >>>>> Hi Thierry, >>>>> >>>>> >>>>> I was getting the hang issue while running >>>>> ipa-client-install simultaneously on few clients.. >>>>> However, today, I am not able to replicate that. >>>>> >>>>> I could not get a gdb . But i will try getting that >>>>> the next time I face this issue. >>>>> >>>>> The CPU does not stay high.. it just momentarily >>>>> touches a high value and then drops down to around 2-7% >>>>> >>>>> One question I have is , is it ok to set it >>>>> nsslapd-threadnumber to a very high value . >>>>> I have around 4000 clients and with >>>>> nsslapd-maxthreadsperconn set to 5..So, can I set >>>>> nsslapd-threadnumber to around 25000. >>>> >>>> Hello, >>>> >>>> I know some users running in production with several >>>> hunderds of threads (>600) and this without problem. >>>> >>>> I do not recall having suggested to increase that >>>> number and for what reason. >>>> Usually 30 workers is a good enough value. It can >>>> create bootleneck if for some reason each operation is >>>> very long to satisfy and exhaust the number of workers. >>>> You can monitor the work queue: >>>> >>>> ldapsearch -D "cn=directory manager" -w xxx -LLL >>>> -b "cn=monitor" -s base opsinitiated opscompleted >>>> >>>> >>>> If opscompleted-opsinitiated remains close to >>>> threadnumber, then yes it would be valuable to increase it. >>>> >>>> The computation #client * #async_op_per_client sound an >>>> overkill. Even if all clients send at the exact same >>>> time all their requests, it is very likely that some >>>> common resource (db page, log, allocator...) will >>>> serialize them. If you monitor a need to increase the >>>> work, you would for example set it to 50, then monitor, >>>> then set it to 100, then monitor... until you find a >>>> good enough value. >>>> Note the increasing the #thread, increases the memory >>>> footprint that will reduce the efficiency of file >>>> system cache and can increase the response time. >>>> >>>> >>>> best regards >>>> thierry >>>> >>>>> >>>>> Thanks >>>>> >>>>> On Mon, Sep 5, 2016 at 1:03 PM, thierry bordaz >>>>> wrote: >>>>> >>>>> >>>>> Hi Rakesh, >>>>> >>>>> Were you able to get a pstack or full stack with >>>>> gdb >>>>> (http://www.port389.org/docs/389ds/FAQ/faq.html#debugging-crashes) >>>>> when the server hangs ? >>>>> >>>>> If it happens with 500 threads as well as with 30, >>>>> using 30 threads is a better choice to debug this >>>>> issue. >>>>> I will try to reproduce using 150 parallel 'ipa >>>>> user-find p-testipa' commands >>>>> >>>>> Something I am unsure is if the CPU consumption >>>>> stays high (you mentioned 340% CPU usage) as long >>>>> as the hang happens or if after a suddent shot up >>>>> to 340% (that marks the beginning of the hang) it >>>>> drops and stay hanging ? >>>>> >>>>> thanks >>>>> thierry >>>>> >>>>> On 09/04/2016 08:40 PM, Rakesh Rajasekharan wrote: >>>>>> starce on the slapd process actually had this in >>>>>> the output.. >>>>>> FUTEX_WAIT_PRIVATE >>>>>> >>>>>> and checking for the number of threads slapd >>>>>> had.. there were 5015 threads >>>>>> >>>>>> ps -efL|grep slapd|wc -l >>>>>> 5015 >>>>>> >>>>>> strace on most of the threads gave this output >>>>>> >>>>>> strace -p 67411 >>>>>> Process 67411 attached >>>>>> futex(0x7f3f0226b41c, FUTEX_WAIT_PRIVATE, 1, >>>>>> NULL) = -1 EAGAIN (Resource temporarily unavailable) >>>>>> futex(0x7f3f0226b41c, FUTEX_WAIT_PRIVATE, 2, >>>>>> NULL^CProcess 67411 detached >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> On Sun, Sep 4, 2016 at 5:34 PM, Rakesh >>>>>> Rajasekharan wrote: >>>>>> >>>>>> I have again got the issue of IPA hanging.. >>>>>> The issue came up when i tried to run >>>>>> ipa-client-isntall on 142 clients simultaneously >>>>>> >>>>>> >>>>>> None of the IPA commands are responding, and >>>>>> I see this error >>>>>> >>>>>> ipa user-find p-testipa >>>>>> ipa: ERROR: Insufficient access: SASL(-1): >>>>>> generic failure: GSSAPI Error: Unspecified >>>>>> GSS failure. Minor code may provide more >>>>>> information (KDC returned error string: >>>>>> PROCESS_TGS) >>>>>> >>>>>> KRB5_TRACE=/dev/stdout kinit admin >>>>>> [41178] 1472984115.233214: Getting initial >>>>>> credentials for admin at XYZ.COM >>>>>> [41178] 1472984115.235257: Sending request >>>>>> (167 bytes) to XYZ.COM >>>>>> [41178] 1472984115.235419: Initiating TCP >>>>>> connection to stream 10.1.3.36:88 >>>>>> >>>>>> [41178] 1472984115.235685: Sending TCP >>>>>> request to stream 10.1.3.36:88 >>>>>> >>>>>> [41178] 1472984120.238914: Received answer >>>>>> (174 bytes) from stream 10.1.3.36:88 >>>>>> >>>>>> [41178] 1472984120.238925: Terminating TCP >>>>>> connection to stream 10.1.3.36:88 >>>>>> >>>>>> [41178] 1472984120.238993: Response was from >>>>>> master KDC >>>>>> [41 >>>>>> >>>>>> >>>>>> Running an ldapsearch to see the db.. does >>>>>> not give any results and just hangs there >>>>>> >>>>>> ldapsearch -x -D 'cn=Directory Manager' -W -s >>>>>> one -b 'cn=kerberos,dc=xyz,dc=com' >>>>>> Enter LDAP Password: >>>>>> >>>>>> even an ldapsearch -x does not respond >>>>>> At this point, am sure that slapd is the one >>>>>> causing issues >>>>>> >>>>>> Running an strace against the hung slapd >>>>>> itself seems to get stuck does not proceed >>>>>> after saying "attaching to process" >>>>>> >>>>>> From some others posts I read Thierry >>>>>> suggesting to increase the >>>>>> nsslapd-threadnumber value >>>>>> >>>>>> It was set to 30, I think that might be too low. >>>>>> >> ... >> >> [Message clipped] >> >> >> > -- Red Hat GmbH, http://www.de.redhat.com/, Registered seat: Grasbrunn, Commercial register: Amtsgericht Muenchen, HRB 153243, Managing Directors: Charles Cachera, Michael Cunningham, Michael O'Neill, Eric Shander -------------- next part -------------- An HTML attachment was scrubbed... URL: From gjn at gjn.priv.at Tue Sep 6 10:56:17 2016 From: gjn at gjn.priv.at (=?ISO-8859-1?Q?G=FCnther_J=2E?= Niederwimmer) Date: Tue, 06 Sep 2016 12:56:17 +0200 Subject: [Freeipa-users] Error by creating a services SOLVED In-Reply-To: <24970716.1YqJe0nuC0@techz> References: <38768539.99melkodZ8@techz> <24970716.1YqJe0nuC0@techz> Message-ID: <1515469.Fd1tIfKgMR@techz> Hello, Am Montag, 5. September 2016, 17:43:26 schrieb G?nther J. Niederwimmer: I found it with Help.... Thanks for the Help, my mistake was a in a local DNS Entry! > Am Montag, 5. September 2016, 17:09:03 schrieb Martin Basti: > > On 05.09.2016 16:53, G?nther J. Niederwimmer wrote: > > > Hello, > > > > > > CentOs 7.2 > > > FreeIPA: 4.2.0-15 > > > > > > Why is this Error only on one Server ? > > > > Hello, > > > > probably you have something DNS related misconfigured on that particular > > server. > > > > Can you resolve hostname manually from server? (host, dig A commands) > > This was my first check ;-) , yes I all is correct host and dig is working, > this is a external server and I can connect remote with mx03.example.com > > I check also the reverse Zone, this is also correct ? > > But it is not possible to create a service, the enrollment is working with > no errors ? > > OK, now I make ipa-client-install --uninstall and a ipa-client-install again > and test it ? > > > > IPA Error 4019: DNSNotARecordError > > > Host does not have corresponding DNS A/AAAA record > > > > > > When I create on a other Server (KVM) a service, this is no Problem, but > > > on my new "mx03.example.com" I have this Errors ? > > > > > > The DNS is in the Moment not configured with IPA, it is a KVM Client > > > (named) and it works correct? > > > > > > > > > Is this a Name Problem (mx03) or ...... > > > > > > Thanks for a answer > > > > > > Reverse Zone is correct set to mx03."example.com" -- mit freundlichen Gr??en / best regards, G?nther J. Niederwimmer From rakesh.rajasekharan at gmail.com Tue Sep 6 12:21:10 2016 From: rakesh.rajasekharan at gmail.com (Rakesh Rajasekharan) Date: Tue, 6 Sep 2016 17:51:10 +0530 Subject: [Freeipa-users] Freeipa 4.2.0 hangs intermittently In-Reply-To: <57CE9DE5.6020301@redhat.com> References: <57BEB0AB.2090506@redhat.com> <57C44AC6.1030402@redhat.com> <57CD1FDC.2010109@redhat.com> <57CD5488.60404@redhat.com> <57CD659B.6050707@redhat.com> <57CD68BD.9030802@redhat.com> <57CE8CDB.6000809@redhat.com> <57CE9DE5.6020301@redhat.com> Message-ID: Hi, With script provided by Rich, I looked at my db cache usage and it does not look to be high enough # under each db are the list of selected indexes specified with INDEXLIST dbname count free free% size changelog:ent 5 2076333 99.0 4163.8 changelog:dn 15976 9303557 88.7 74.0 userroot:ent 4763 175321074 83.6 7221.1 userroot:dn 4763 9995056 95.3 103.0 ipaca:ent 425 6337866 60.4 9759.8 ipaca:dn 425 10451081 99.7 81.6 Right now I set it to 200MB. So, I am planning to bring it down to 100MB Also, *nsslapd-db-checkpoint-interval *will change it to 80 or 90 from default of 60 I will check this out and the impact it has on the hang issues and get back Thanks Rakesh On Tue, Sep 6, 2016 at 4:13 PM, Ludwig Krispenz wrote: > Hi, > > I think you are running into this bug: https://bugzilla.redhat.com/ > show_bug.cgi?id=1349779 > it was the outcome of an analysis started in this one: > https://bugzilla.redhat.com/show_bug.cgi?id=1347800 > > the problem is that it looks like the checkpoint thread is holding a mutex > and otehr threads are blocked on it. It looks like an issue in BerkeleyDB > although there is no explanation from the BDB code why this could happen, > we also had a support case with Oracle to investigate this, with no > solution. Oracle at the end was providing a diagnostic patch, which we did > not test since the customer with the original problem had worked around > this by changing their configuration. > I don't know the status of this patch and if we can make it available, > probably depends on your support status, you need to contact support for > this. > > About the workarounds: > you can try to change timing for the database update and checkpointing eg > by reducing the dbcache size, changing the checkpoint interval. In the > customer case they identified one substring index, which was frequently > updated but not really required - and removing this index helped. > > Regards, > Ludwig > > On 09/06/2016 11:31 AM, thierry bordaz wrote: > > Hi, > > The following thread is hanging most of the others threads > > Thread 20 (Thread 0x7f68007f8700 (LWP 4873)): > > #0 0x00007f683d0506d5 in pthread_cond_wait@@GLIBC_2.3.2 () from > /lib64/libpthread.so.0 > #1 0x00007f68375ac2f3 in __db_hybrid_mutex_suspend () from /lib64/ > libdb-5.3.so > #2 0x00007f68375ab640 in __db_tas_mutex_lock () from /lib64/libdb-5.3.so > #3 0x00007f68376c5e1b in __memp_fget () from /lib64/libdb-5.3.so > #4 0x00007f68375c9211 in __bam_search () from /lib64/libdb-5.3.so > #5 0x00007f68375b4256 in __bamc_search () from /lib64/libdb-5.3.so > #6 0x00007f68375b86c4 in __bamc_put () from /lib64/libdb-5.3.so > #7 0x00007f683766f6d5 in __dbc_iput () from /lib64/libdb-5.3.so > #8 0x00007f683766aa0e in __db_put () from /lib64/libdb-5.3.so > #9 0x00007f683767ffa4 in __db_put_pp () from /lib64/libdb-5.3.so > #10 0x00007f6832b5a4cf in idl_new_insert_key () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #11 0x00007f6832b695b5 in addordel_values_sv.isra.3 () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #12 0x00007f6832b6a25a in index_addordel_values_ext_sv () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #13 0x00007f6832b6a694 in index_addordel_values_sv () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #14 0x00007f6832b6a75a in index_addordel_entry () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #15 0x00007f6832b6d5fa in ldbm_back_add () from /usr/lib64/dirsrv/plugins/ > libback-ldbm.so > #16 0x00007f683f46fcca in op_shared_add () from > /usr/lib64/dirsrv/libslapd.so.0 > #17 0x00007f683f471020 in do_add () from /usr/lib64/dirsrv/libslapd.so.0 > #18 0x00007f683f99c303 in connection_threadmain () > #19 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so > #20 0x00007f683d04cdc5 in start_thread () from /lib64/libpthread.so.0 > #21 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 > > This thread is likely not hang on 'db_tas_mutex_lock' as the db_lock does > not show it waiting for something. > Unfortunately it is not shown in the output of top -H, did you get the > complete output ? > Ludwig mentioned a known issue that DS is not scaling well on NUMA arch, > do you know if the machine is NUMA ? > A possibility is that this thread get severely slowed down by a NUMA > remote lock and showing DS has hung. > This is why it would be interesting to know if it consums CPU. > > thanks > theirry > > On 09/06/2016 10:59 AM, Rakesh Rajasekharan wrote: > > Hi Thierry/Ludwig, > > I again hit the hang issue. This is the pstack of the ns-slapd > > So was able to this time get the pstack and db locks > > db locks details > Default locking region information: > 451 Last allocated locker ID > 0x7fffffff Current maximum unused locker ID > 9 Number of lock modes > 800 Initial number of locks allocated > 0 Initial number of lockers allocated > 800 Initial number of lock objects allocated > 10000 Maximum number of locks possible > 10000 Maximum number of lockers possible > 10000 Maximum number of lock objects possible > 800 Current number of locks allocated > 366 Current number of lockers allocated > 800 Current number of lock objects allocated > 160 Number of lock object partitions > 8191 Size of object hash table > 126 Number of current locks > 1005 Maximum number of locks at any one time > 33 Maximum number of locks in any one bucket > 3987 Maximum number of locks stolen by for an empty partition > 75 Maximum number of locks stolen for any one partition > 309 Number of current lockers > 313 Maximum number of lockers at any one time > 121 Number of current lock objects > 676 Maximum number of lock objects at any one time > 3 Maximum number of lock objects in any one bucket > 69 Maximum number of objects stolen by for an empty partition > 5 Maximum number of objects stolen for any one partition > 22M Total number of locks requested (22009642) > 21M Total number of locks released (21992878) > 0 Total number of locks upgraded > 95 Total number of locks downgraded > 3348 Lock requests not available due to conflicts, for which we > waited > 12646 Lock requests not available due to conflicts, for which we did > not wait > 0 Number of deadlocks > 0 Lock timeout value > 0 Number of locks that have timed out > 0 Transaction timeout value > 0 Number of transactions that have timed out > 2MB 560KB Region size > 71719 The number of partition locks that required waiting (0%) > 23105 The maximum number of times any partition lock was waited for > (0%) > 0 The number of object queue operations that required waiting (0%) > 163 The number of locker allocations that required waiting (0%) > 3172 The number of region locks that required waiting (0%) > 5 Maximum hash bucket length > =-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-= > Lock REGINFO information: > Environment Region type > 1 Region ID > /var/lib/dirsrv/slapd-SPRINKLR-COM/db/__db.001 Region name > 0x7f67ef869000 Region address > 0x7f67ef8690a0 Region allocation head > 0x7f67ef8712b0 Region primary address > 0 Region maximum allocation > 0 Region allocated > Region allocations: 1581345 allocations, 0 failures, 1580794 frees, 6 > longest > Allocations by power-of-two sizes: > 1KB 1581329 > 2KB 3 > 4KB 4 > 8KB 3 > 16KB 3 > 32KB 1 > 64KB 0 > 128KB 0 > 256KB 2 > 512KB 0 > 1024KB 1 > REGION_SHARED Region flags > =-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-= > Lock region parameters: > 2 Lock region region mutex [3172/14M 0% !Own] > 16381 locker table size > 8191 object table size > 34128 obj_off > 900216 locker_off > 0 need_dd > =-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-= > Lock conflict matrix: > =-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-= > Locks grouped by lockers: > Locker Mode Count Status ----------------- Object --------------- > 2 dd=308 locks held 1 write locks 0 pid/thread > 4845/140085720139840 flags 10 priority 100 > 2 READ 1 HELD userRoot/id2entry.db handle 0 > 3 dd=307 locks held 0 write locks 0 pid/thread > 4845/140084829472512 flags 0 priority 100 > 4 dd=306 locks held 1 write locks 0 pid/thread > 4845/140085720139840 flags 10 priority 100 > 4 READ 1 HELD ipaca/id2entry.db handle 0 > 5 dd=305 locks held 0 write locks 0 pid/thread > 4845/140084837865216 flags 0 priority 100 > 6 dd=304 locks held 1 write locks 0 pid/thread > 4845/140085720139840 flags 10 priority 100 > 6 READ 1 HELD ipaca/entryrdn.db handle 0 > 7 dd=303 locks held 0 write locks 0 pid/thread > 4845/140084804294400 flags 0 priority 100 > 8 dd=302 locks held 0 write locks 0 pid/thread > 4845/140084720432896 flags 0 priority 100 > 9 dd=301 locks held 0 write locks 0 pid/thread > 4845/140084837865216 flags 0 priority 100 > a dd=300 locks held 1 write locks 0 pid/thread > 4845/140085720139840 flags 10 priority 100 > a READ 1 HELD ipaca/vlv#allcertspkitomcatindex.db > handle 0 > e dd=299 locks held 1 write locks 0 pid/thread > 4845/140085720139840 flags 10 priority 100 > e READ 1 HELD ipaca/vlv#allnonrevokedcertspkitomcatindex.db > handle 0 > 10 dd=298 locks held 1 write locks 0 pid/thread > 4845/140085720139840 flags 10 priority 100 > 10 READ 1 HELD ipaca/vlv#allrevokedcertspkitomcatindex.db > handle 0 > 11 dd=297 locks held 1 write locks 0 pid/thread > 4845/140085720139840 flags 10 priority 100 > 11 READ 1 HELD ipaca/vlv#allrevokedcertsnotafterpkitomcatindex.db > handle 0 > 14 dd=296 locks held 1 write locks 0 pid/thread > 4845/140085720139840 flags 10 priority 100 > 14 READ 1 HELD ipaca/vlv#allrevokedorrevokedexpiredcertspkitomcatindex.db > handle 0 > 15 dd=295 locks held 1 write locks 0 pid/thread > 4845/140085720139840 flags 10 priority 100 > 15 READ 1 HELD ipaca/vlv#allvalidcertspkitomcatindex.db > handle 0 > 16 dd=294 locks held 1 write locks 0 pid/thread > 4845/140085720139840 flags 10 priority 100 > 16 READ 1 HELD ipaca/vlv#allvalidcertsnotafterpkitomcatindex.db > handle 0 > 17 dd=293 locks held 1 write locks 0 pid/thread > 4845/140085720139840 flags 10 priority 100 > 17 READ 1 HELD ipaca/vlv#allvalidorrevokedcertspkitomcatindex.db > handle 0 > 18 dd=292 locks held 1 write locks 0 pid/thread > 4845/140085720139840 flags 10 priority 100 > 18 READ 1 HELD ipaca/vlv#caallpkitomcatindex.db > handle 0 > 1d dd=291 locks held 1 write locks 0 pid/thread > 4845/140085720139840 flags 10 priority 100 > 1d READ 1 HELD ipaca/vlv#cacompletepkitomcatindex.db > handle 0 > 1e dd=290 locks held 1 write locks 0 pid/thread > 4845/140085720139840 flags 10 priority 100 > 1e READ 1 HELD ipaca/vlv#cacompleteenrollmentpkitomcatindex.db > handle 0 > 20 dd=289 locks held 1 write locks 0 pid/thread > 4845/140085720139840 flags 10 priority 100 > 20 READ 1 HELD ipaca/vlv#cacompleterevocationpkitomcatindex.db > handle 0 > 21 dd=288 locks held 1 write locks 0 pid/thread > 4845/140085720139840 flags 10 priority 100 > 21 READ 1 HELD ipaca/vlv#caenrollmentpkitomcatindex.db > handle 0 > 2b dd=287 locks held 1 write locks 0 pid/thread > 4845/140085720139840 flags 10 priority 100 > 2b READ 1 HELD ipaca/vlv#carevocationpkitomcatindex.db > handle 0 > 2c dd=286 locks held 1 write locks 0 pid/thread > 4845/140085720139840 flags 10 priority 100 > 2c READ 1 HELD changelog/id2entry.db handle 0 > 2d dd=285 locks held 0 write locks 0 pid/thread > 4845/140085719607040 flags 0 priority 100 > 2e dd=284 locks held 1 write locks 0 pid/thread > 4845/140085720139840 flags 10 priority 100 > 2e READ 1 HELD changelog/entryusn.db handle 0 > 2f dd=283 locks held 0 write locks 0 pid/thread > 4845/140085720139840 flags 0 priority 100 > 30 dd=282 locks held 1 write locks 0 pid/thread > 4845/140085720139840 flags 10 priority 100 > 30 READ 1 HELD userRoot/entryusn.db handle 0 > 31 dd=281 locks held 0 write locks 0 pid/thread > 4845/140085720139840 flags 0 priority 100 > 32 dd=280 locks held 1 write locks 0 pid/thread > 4845/140085720139840 flags 10 priority 100 > 32 READ 1 HELD ipaca/entryusn.db handle 0 > 33 dd=279 locks held 0 write locks 0 pid/thread > 4845/140085720139840 flags 0 priority 100 > 34 dd=278 locks held 1 write locks 0 pid/thread > 4845/140085720139840 flags 10 priority 100 > 34 READ 1 HELD userRoot/entryrdn.db handle 0 > 35 dd=277 locks held 0 write locks 0 pid/thread > 4845/140084544186112 flags 0 priority 100 > 36 dd=276 locks held 0 write locks 0 pid/thread > 4845/140084636505856 flags 0 priority 100 > 37 dd=275 locks held 0 write locks 0 pid/thread > 4845/140084854650624 flags 0 priority 100 > 38 dd=274 locks held 1 write locks 0 pid/thread > 4845/140085720139840 flags 10 priority 100 > 38 READ 1 HELD userRoot/objectclass.db handle 0 > 39 dd=273 locks held 0 write locks 0 pid/thread > 4845/140084569364224 flags 0 priority 100 > 3a dd=272 locks held 1 write locks 0 pid/thread > 4845/140085720139840 flags 10 priority 100 > 3a READ 1 HELD userRoot/ancestorid.db handle 0 > 3b dd=271 locks held 0 write locks 0 pid/thread > 4845/140084854650624 flags 0 priority 100 > 3c dd=270 locks held 1 write locks 0 pid/thread > 4845/140085720139840 flags 10 priority 100 > 3c READ 1 HELD changelog/entryrdn.db handle 0 > 3d dd=269 locks held 0 write locks 0 pid/thread > 4845/140085719607040 flags 0 priority 100 > 3e dd=268 locks held 0 write locks 0 pid/thread > 4845/140084602935040 flags 0 priority 100 > 3f dd=267 locks held 0 write locks 0 pid/thread > 4845/140085719607040 flags 0 priority 100 > 40 dd=266 locks held 1 write locks 0 pid/thread > 4845/140085720139840 flags 10 priority 100 > 40 READ 1 HELD changelog/objectclass.db handle 0 > 41 dd=265 locks held 0 write locks 0 pid/thread > 4845/140085719607040 flags 0 priority 100 > 42 dd=264 locks held 0 write locks 0 pid/thread > 4845/140085719607040 flags 0 priority 100 > 43 dd=263 locks held 0 write locks 0 pid/thread > 4845/140084653291264 flags 0 priority 100 > 44 dd=262 locks held 1 write locks 0 pid/thread > 4845/140085720139840 flags 10 priority 100 > 44 READ 1 HELD ipaca/objectclass.db handle 0 > 45 dd=261 locks held 0 write locks 0 pid/thread > 4845/140084577756928 flags 0 priority 100 > 46 dd=260 locks held 0 write locks 0 pid/thread > 4845/140084720432896 flags 0 priority 100 > 47 dd=259 locks held 1 write locks 0 pid/thread > 4845/140085720139840 flags 10 priority 100 > 47 READ 1 HELD changelog/aci.db handle 0 > 48 dd=258 locks held 0 write locks 0 pid/thread > 4845/140085720139840 flags 0 priority 100 > 49 dd=257 locks held 1 write locks 0 pid/thread > 4845/140085720139840 flags 10 priority 100 > 49 READ 1 HELD userRoot/aci.db handle 0 > 4a dd=256 locks held 0 write locks 0 pid/thread > 4845/140085720139840 flags 0 priority 100 > 4b dd=255 locks held 0 write locks 0 pid/thread > 4845/140084535793408 flags 0 priority 100 > 4c dd=254 locks held 1 write locks 0 pid/thread > 4845/140085720139840 flags 10 priority 100 > 4c READ 1 HELD ipaca/aci.db handle 0 > 4d dd=253 locks held 0 write locks 0 pid/thread > 4845/140085720139840 flags 0 priority 100 > 4e dd=252 locks held 1 write locks 0 pid/thread > 4845/140085720139840 flags 10 priority 100 > 4e READ 1 HELD userRoot/parentid.db handle 0 > 4f dd=251 locks held 0 write locks 0 pid/thread > 4845/140085720139840 flags 0 priority 100 > 50 dd=250 locks held 0 write locks 0 pid/thread > 4845/140085719607040 flags 0 priority 100 > 51 dd=249 locks held 1 write locks 0 pid/thread > 4845/140085720139840 flags 10 priority 100 > 51 READ 1 HELD changelog/nsuniqueid.db handle 0 > 52 dd=248 locks held 1 write locks 0 pid/thread > 4845/140085720139840 flags 10 priority 100 > 52 READ 1 HELD changelog/changenumber.db handle 0 > 53 dd=247 locks held 0 write locks 0 pid/thread > 4845/140084602935040 flags 0 priority 100 > 54 dd=246 locks held 0 write locks 0 pid/thread > 4845/140085719607040 flags 0 priority 100 > 55 dd=245 locks held 1 write locks 0 pid/thread > 4845/140085719607040 flags 10 priority 100 > 55 READ 1 HELD changelog/targetuniqueid.db handle > 0 > 56 dd=244 locks held 1 write locks 0 pid/thread > 4845/140085719607040 flags 10 priority 100 > 56 READ 1 HELD changelog/parentid.db handle 0 > 57 dd=243 locks held 1 write locks 0 pid/thread > 4845/140085719607040 flags 10 priority 100 > 57 READ 1 HELD changelog/ancestorid.db handle 0 > 58 dd=242 locks held 1 write locks 0 pid/thread > 4845/140085719607040 flags 10 priority 100 > 58 READ 1 HELD changelog/numsubordinates.db > handle 0 > 59 dd=241 locks held 0 write locks 0 pid/thread > 4845/140085719607040 flags 0 priority 100 > 5a dd=240 locks held 0 write locks 0 pid/thread > 4845/140084544186112 flags 0 priority 100 > 5b dd=239 locks held 1 write locks 0 pid/thread > 4845/140084577756928 flags 10 priority 100 > 5b READ 1 HELD userRoot/krbPrincipalName.db > handle 0 > 5c dd=238 locks held 0 write locks 0 pid/thread > 4845/140084636505856 flags 0 priority 100 > 5d dd=237 locks held 0 write locks 0 pid/thread > 4845/140084695254784 flags 0 priority 100 > 5e dd=236 locks held 1 write locks 0 pid/thread > 4845/140084560971520 flags 10 priority 100 > 5e READ 1 HELD userRoot/ipakrbprincipalalias.db > handle 0 > 5f dd=235 locks held 0 write locks 0 pid/thread > 4845/140084703647488 flags 0 priority 100 > 60 dd=234 locks held 0 write locks 0 pid/thread > 4845/140084653291264 flags 0 priority 100 > 61 dd=233 locks held 0 write locks 0 pid/thread > 4845/140084577756928 flags 0 priority 100 > 62 dd=232 locks held 0 write locks 0 pid/thread > 4845/140084686862080 flags 0 priority 100 > 63 dd=231 locks held 1 write locks 0 pid/thread > 4845/140084611327744 flags 10 priority 100 > 63 READ 1 HELD changelog/seeAlso.db handle 0 > 64 dd=230 locks held 0 write locks 0 pid/thread > 4845/140084686862080 flags 0 priority 100 > 65 dd=229 locks held 1 write locks 0 pid/thread > 4845/140084611327744 flags 10 priority 100 > 65 READ 1 HELD userRoot/seeAlso.db handle 0 > 66 dd=228 locks held 0 write locks 0 pid/thread > 4845/140084636505856 flags 0 priority 100 > 67 dd=227 locks held 1 write locks 0 pid/thread > 4845/140084611327744 flags 10 priority 100 > 67 READ 1 HELD ipaca/seeAlso.db handle 0 > 68 dd=226 locks held 0 write locks 0 pid/thread > 4845/140084686862080 flags 0 priority 100 > 69 dd=225 locks held 1 write locks 0 pid/thread > 4845/140084678469376 flags 10 priority 100 > 69 READ 1 HELD ipaca/parentid.db handle 0 > 6a dd=224 locks held 0 write locks 0 pid/thread > 4845/140084678469376 flags 0 priority 100 > 6b dd=223 locks held 0 write locks 0 pid/thread > 4845/140084829472512 flags 0 priority 100 > 6c dd=222 locks held 0 write locks 0 pid/thread > 4845/140084653291264 flags 0 priority 100 > 6d dd=221 locks held 0 write locks 0 pid/thread > 4845/140084569364224 flags 0 priority 100 > 6e dd=220 locks held 0 write locks 0 pid/thread > 4845/140084636505856 flags 0 priority 100 > 6e READ 1 WAIT userRoot/objectclass.db page 18 > 6f dd=219 locks held 0 write locks 0 pid/thread > 4845/140084544186112 flags 0 priority 100 > 70 dd=218 locks held 0 write locks 0 pid/thread > 4845/140084569364224 flags 0 priority 100 > 71 dd=217 locks held 0 write locks 0 pid/thread > 4845/140084619720448 flags 0 priority 100 > 72 dd=216 locks held 0 write locks 0 pid/thread > 4845/140084552578816 flags 0 priority 100 > 73 dd=215 locks held 0 write locks 0 pid/thread > 4845/140084829472512 flags 0 priority 100 > 74 dd=214 locks held 0 write locks 0 pid/thread > 4845/140084628113152 flags 0 priority 100 > 75 dd=213 locks held 0 write locks 0 pid/thread > 4845/140084686862080 flags 0 priority 100 > 76 dd=212 locks held 1 write locks 0 pid/thread > 4845/140084560971520 flags 10 priority 100 > 76 READ 1 HELD userRoot/ipauniqueid.db handle 0 > 77 dd=211 locks held 0 write locks 0 pid/thread > 4845/140084661683968 flags 0 priority 100 > 78 dd=210 locks held 0 write locks 0 pid/thread > 4845/140084846257920 flags 0 priority 100 > 79 dd=209 locks held 0 write locks 0 pid/thread > 4845/140084628113152 flags 0 priority 100 > 7a dd=208 locks held 1 write locks 0 pid/thread > 4845/140084560971520 flags 10 priority 100 > 7a READ 1 HELD userRoot/nsuniqueid.db handle 0 > 7b dd=207 locks held 1 write locks 0 pid/thread > 4845/140084560971520 flags 10 priority 100 > 7b READ 1 HELD userRoot/cn.db handle 0 > 7c dd=206 locks held 1 write locks 0 pid/thread > 4845/140084560971520 flags 10 priority 100 > 7c READ 1 HELD userRoot/fqdn.db handle 0 > 7d dd=205 locks held 1 write locks 0 pid/thread > 4845/140084560971520 flags 10 priority 100 > 7d READ 1 HELD userRoot/managedby.db handle 0 > 7e dd=204 locks held 1 write locks 0 pid/thread > 4845/140084560971520 flags 10 priority 100 > 7e READ 1 HELD userRoot/numsubordinates.db handle > 0 > 7f dd=203 locks held 1 write locks 0 pid/thread > 4845/140084678469376 flags 10 priority 100 > 7f READ 1 HELD userRoot/member.db handle 0 > 80 dd=202 locks held 0 write locks 0 pid/thread > 4845/140084712040192 flags 0 priority 100 > 81 dd=201 locks held 1 write locks 0 pid/thread > 4845/140084678469376 flags 10 priority 100 > 81 READ 1 HELD userRoot/memberUser.db handle 0 > 82 dd=200 locks held 0 write locks 0 pid/thread > 4845/140084577756928 flags 0 priority 100 > 83 dd=199 locks held 1 write locks 0 pid/thread > 4845/140084678469376 flags 10 priority 100 > 83 READ 1 HELD userRoot/memberHost.db handle 0 > 84 dd=198 locks held 0 write locks 0 pid/thread > 4845/140084712040192 flags 0 priority 100 > 85 dd=197 locks held 0 write locks 0 pid/thread > 4845/140084577756928 flags 0 priority 100 > 86 dd=196 locks held 0 write locks 0 pid/thread > 4845/140084670076672 flags 0 priority 100 > 87 dd=195 locks held 0 write locks 0 pid/thread > 4845/140084577756928 flags 0 priority 100 > 88 dd=194 locks held 0 write locks 0 pid/thread > 4845/140084846257920 flags 0 priority 100 > 89 dd=193 locks held 0 write locks 0 pid/thread > 4845/140084552578816 flags 0 priority 100 > 8a dd=192 locks held 0 write locks 0 pid/thread > 4845/140084703647488 flags 0 priority 100 > 8b dd=191 locks held 0 write locks 0 pid/thread > 4845/140084544186112 flags 0 priority 100 > 8c dd=190 locks held 0 write locks 0 pid/thread > 4845/140084846257920 flags 0 priority 100 > 8d dd=189 locks held 0 write locks 0 pid/thread > 4845/140084661683968 flags 0 priority 100 > 8e dd=188 locks held 1 write locks 0 pid/thread > 4845/140085067675392 flags 10 priority 100 > 8e READ 1 HELD userRoot/macAddress.db handle 0 > 8f dd=187 locks held 0 write locks 0 pid/thread > 4845/140085067675392 flags 0 priority 100 > 90 dd=186 locks held 0 write locks 0 pid/thread > 4845/140084703647488 flags 0 priority 100 > 91 dd=185 locks held 0 write locks 0 pid/thread > 4845/140084552578816 flags 0 priority 100 > 92 dd=184 locks held 0 write locks 0 pid/thread > 4845/140084560971520 flags 0 priority 100 > 92 READ 1 WAIT userRoot/objectclass.db page 9 > 93 dd=183 locks held 0 write locks 0 pid/thread > 4845/140084577756928 flags 0 priority 100 > 94 dd=182 locks held 0 write locks 0 pid/thread > 4845/140084821079808 flags 0 priority 100 > 94 READ 1 WAIT userRoot/objectclass.db page 18 > 95 dd=181 locks held 0 write locks 0 pid/thread > 4845/140084544186112 flags 0 priority 100 > 96 dd=180 locks held 0 write locks 0 pid/thread > 4845/140084804294400 flags 0 priority 100 > 97 dd=179 locks held 1 write locks 0 pid/thread > 4845/140084712040192 flags 10 priority 100 > 97 READ 1 HELD userRoot/uid.db handle 0 > 98 dd=178 locks held 0 write locks 0 pid/thread > 4845/140084552578816 flags 0 priority 100 > 99 dd=177 locks held 0 write locks 0 pid/thread > 4845/140084577756928 flags 0 priority 100 > 9a dd=176 locks held 0 write locks 0 pid/thread > 4845/140084602935040 flags 0 priority 100 > 9b dd=175 locks held 1 write locks 0 pid/thread > 4845/140085076068096 flags 10 priority 100 > 9b READ 1 HELD userRoot/uniquemember.db handle 0 > 9c dd=174 locks held 1 write locks 0 pid/thread > 4845/140085076068096 flags 10 priority 100 > 9c READ 1 HELD userRoot/owner.db handle 0 > 9d dd=173 locks held 1 write locks 0 pid/thread > 4845/140085076068096 flags 10 priority 100 > 9d READ 1 HELD userRoot/ipaallowedtarget.db > handle 0 > 9e dd=172 locks held 1 write locks 0 pid/thread > 4845/140085076068096 flags 10 priority 100 > 9e READ 1 HELD userRoot/ipaMemberCertProfile.db > handle 0 > 9f dd=171 locks held 1 write locks 0 pid/thread > 4845/140085076068096 flags 10 priority 100 > 9f READ 1 HELD userRoot/ipaassignedidview.db > handle 0 > a0 dd=170 locks held 1 write locks 0 pid/thread > 4845/140085076068096 flags 10 priority 100 > a0 READ 1 HELD userRoot/secretary.db handle 0 > a1 dd=169 locks held 1 write locks 0 pid/thread > 4845/140085076068096 flags 10 priority 100 > a1 READ 1 HELD userRoot/memberdenycmd.db handle 0 > a2 dd=168 locks held 1 write locks 0 pid/thread > 4845/140085076068096 flags 10 priority 100 > a2 READ 1 HELD userRoot/memberallowcmd.db handle 0 > a3 dd=167 locks held 1 write locks 0 pid/thread > 4845/140085076068096 flags 10 priority 100 > a3 READ 1 HELD userRoot/manager.db handle 0 > a4 dd=166 locks held 1 write locks 0 pid/thread > 4845/140085076068096 flags 10 priority 100 > a4 READ 1 HELD userRoot/ipasudorunas.db handle 0 > a5 dd=165 locks held 1 write locks 0 pid/thread > 4845/140085076068096 flags 10 priority 100 > a5 READ 1 HELD userRoot/ipaMemberCa.db handle 0 > a6 dd=164 locks held 1 write locks 0 pid/thread > 4845/140085076068096 flags 10 priority 100 > a6 READ 1 HELD userRoot/ipasudorunasgroup.db > handle 0 > a7 dd=163 locks held 1 write locks 0 pid/thread > 4845/140085076068096 flags 10 priority 100 > a7 READ 1 HELD userRoot/sourcehost.db handle 0 > a8 dd=162 locks held 1 write locks 0 pid/thread > 4845/140085076068096 flags 10 priority 100 > a8 READ 1 HELD userRoot/memberservice.db handle 0 > a9 dd=161 locks held 1 write locks 0 pid/thread > 4845/140085076068096 flags 10 priority 100 > a9 READ 1 HELD userRoot/ipatokenradiusconfiglink.db > handle 0 > aa dd=160 locks held 0 write locks 0 pid/thread > 4845/140084703647488 flags 0 priority 100 > ab dd=159 locks held 0 write locks 0 pid/thread > 4845/140084821079808 flags 0 priority 100 > ac dd=158 locks held 0 write locks 0 pid/thread > 4845/140084628113152 flags 0 priority 100 > ad dd=157 locks held 0 write locks 0 pid/thread > 4845/140084829472512 flags 0 priority 100 > ae dd=156 locks held 0 write locks 0 pid/thread > 4845/140084829472512 flags 0 priority 100 > af dd=155 locks held 0 write locks 0 pid/thread > 4845/140084854650624 flags 0 priority 100 > b0 dd=154 locks held 0 write locks 0 pid/thread > 4845/140084854650624 flags 0 priority 100 > b1 dd=153 locks held 0 write locks 0 pid/thread > 4845/140084837865216 flags 0 priority 100 > b2 dd=152 locks held 0 write locks 0 pid/thread > 4845/140084804294400 flags 0 priority 100 > b3 dd=151 locks held 0 write locks 0 pid/thread > 4845/140084812687104 flags 0 priority 100 > b4 dd=150 locks held 0 write locks 0 pid/thread > 4845/140084804294400 flags 0 priority 100 > b5 dd=149 locks held 0 write locks 0 pid/thread > 4845/140084837865216 flags 0 priority 100 > b6 dd=148 locks held 0 write locks 0 pid/thread > 4845/140084569364224 flags 0 priority 100 > b7 dd=147 locks held 0 write locks 0 pid/thread > 4845/140084829472512 flags 0 priority 100 > b8 dd=146 locks held 0 write locks 0 pid/thread > 4845/140084586149632 flags 0 priority 100 > b9 dd=145 locks held 0 write locks 0 pid/thread > 4845/140084653291264 flags 0 priority 100 > ba dd=144 locks held 0 write locks 0 pid/thread > 4845/140084636505856 flags 0 priority 100 > bb dd=143 locks held 0 write locks 0 pid/thread > 4845/140084804294400 flags 0 priority 100 > bc dd=142 locks held 0 write locks 0 pid/thread > 4845/140084804294400 flags 0 priority 100 > bd dd=141 locks held 0 write locks 0 pid/thread > 4845/140084695254784 flags 0 priority 100 > be dd=140 locks held 0 write locks 0 pid/thread > 4845/140084670076672 flags 0 priority 100 > bf dd=139 locks held 0 write locks 0 pid/thread > 4845/140084628113152 flags 0 priority 100 > c0 dd=138 locks held 0 write locks 0 pid/thread > 4845/140084661683968 flags 0 priority 100 > c1 dd=137 locks held 0 write locks 0 pid/thread > 4845/140084712040192 flags 0 priority 100 > c2 dd=136 locks held 0 write locks 0 pid/thread > 4845/140084661683968 flags 0 priority 100 > c3 dd=135 locks held 0 write locks 0 pid/thread > 4845/140084577756928 flags 0 priority 100 > c4 dd=134 locks held 0 write locks 0 pid/thread > 4845/140084854650624 flags 0 priority 100 > c5 dd=133 locks held 1 write locks 0 pid/thread > 4845/140084644898560 flags 10 priority 100 > c5 READ 1 HELD userRoot/uidnumber.db handle 0 > c6 dd=132 locks held 0 write locks 0 pid/thread > 4845/140084594542336 flags 0 priority 100 > c8 dd=131 locks held 1 write locks 0 pid/thread > 4845/140084678469376 flags 10 priority 100 > c8 READ 1 HELD ipaca/certstatus.db handle 0 > c9 dd=130 locks held 0 write locks 0 pid/thread > 4845/140084720432896 flags 0 priority 100 > ca dd=129 locks held 0 write locks 0 pid/thread > 4845/140084619720448 flags 0 priority 100 > cb dd=128 locks held 0 write locks 0 pid/thread > 4845/140084821079808 flags 0 priority 100 > cc dd=127 locks held 0 write locks 0 pid/thread > 4845/140084594542336 flags 0 priority 100 > cd dd=126 locks held 0 write locks 0 pid/thread > 4845/140084560971520 flags 0 priority 100 > ce dd=125 locks held 1 write locks 0 pid/thread > 4845/140084837865216 flags 10 priority 100 > ce READ 1 HELD userRoot/gidnumber.db handle 0 > cf dd=124 locks held 0 write locks 0 pid/thread > 4845/140084720432896 flags 0 priority 100 > d0 dd=123 locks held 0 write locks 0 pid/thread > 4845/140084569364224 flags 0 priority 100 > d1 dd=122 locks held 1 write locks 0 pid/thread > 4845/140084678469376 flags 10 priority 100 > d1 READ 1 HELD userRoot/nscpEntryDN.db handle 0 > d2 dd=121 locks held 0 write locks 0 pid/thread > 4845/140084695254784 flags 0 priority 100 > d3 dd=120 locks held 0 write locks 0 pid/thread > 4845/140084569364224 flags 0 priority 100 > d4 dd=119 locks held 0 write locks 0 pid/thread > 4845/140084653291264 flags 0 priority 100 > d5 dd=118 locks held 0 write locks 0 pid/thread > 4845/140084829472512 flags 0 priority 100 > d6 dd=117 locks held 0 write locks 0 pid/thread > 4845/140084661683968 flags 0 priority 100 > d7 dd=116 locks held 0 write locks 0 pid/thread > 4845/140084804294400 flags 0 priority 100 > d8 dd=115 locks held 1 write locks 0 pid/thread > 4845/140084821079808 flags 10 priority 100 > d8 READ 1 HELD ipaca/description.db handle 0 > d9 dd=114 locks held 0 write locks 0 pid/thread > 4845/140084661683968 flags 0 priority 100 > da dd=113 locks held 0 write locks 0 pid/thread > 4845/140084837865216 flags 0 priority 100 > db dd=112 locks held 1 write locks 0 pid/thread > 4845/140084678469376 flags 10 priority 100 > db READ 1 HELD ipaca/cn.db handle 0 > dc dd=111 locks held 0 write locks 0 pid/thread > 4845/140084577756928 flags 0 priority 100 > dd dd=110 locks held 1 write locks 0 pid/thread > 4845/140084678469376 flags 10 priority 100 > dd READ 1 HELD ipaca/ancestorid.db handle 0 > de dd=109 locks held 0 write locks 0 pid/thread > 4845/140084577756928 flags 0 priority 100 > df dd=108 locks held 0 write locks 0 pid/thread > 4845/140084703647488 flags 0 priority 100 > e0 dd=107 locks held 0 write locks 0 pid/thread > 4845/140084569364224 flags 0 priority 100 > e1 dd=106 locks held 0 write locks 0 pid/thread > 4845/140084569364224 flags 0 priority 100 > e2 dd=105 locks held 0 write locks 0 pid/thread > 4845/140084821079808 flags 0 priority 100 > e3 dd=104 locks held 1 write locks 0 pid/thread > 4845/140084686862080 flags 10 priority 100 > e3 READ 1 HELD ipaca/serialno.db handle 0 > e4 dd=103 locks held 0 write locks 0 pid/thread > 4845/140084837865216 flags 0 priority 100 > e5 dd=102 locks held 1 write locks 0 pid/thread > 4845/140084695254784 flags 10 priority 100 > e5 READ 1 HELD ipaca/revInfo.db handle 0 > e6 dd=101 locks held 1 write locks 0 pid/thread > 4845/140084695254784 flags 10 priority 100 > e6 READ 1 HELD ipaca/revokedby.db handle 0 > e7 dd=100 locks held 1 write locks 0 pid/thread > 4845/140084695254784 flags 10 priority 100 > e7 READ 1 HELD ipaca/revokedOn.db handle 0 > e8 dd=99 locks held 0 write locks 0 pid/thread > 4845/140084829472512 flags 0 priority 100 > e9 dd=98 locks held 0 write locks 0 pid/thread > 4845/140084846257920 flags 0 priority 100 > ea dd=97 locks held 1 write locks 0 pid/thread > 4845/140084703647488 flags 10 priority 100 > ea READ 1 HELD ipaca/nsuniqueid.db handle 0 > eb dd=96 locks held 1 write locks 0 pid/thread > 4845/140084703647488 flags 10 priority 100 > eb READ 1 HELD ipaca/requestid.db handle 0 > ec dd=95 locks held 1 write locks 0 pid/thread > 4845/140084703647488 flags 10 priority 100 > ec READ 1 HELD ipaca/requeststate.db handle 0 > ed dd=94 locks held 1 write locks 0 pid/thread > 4845/140084703647488 flags 10 priority 100 > ed READ 1 HELD ipaca/dateOfCreate.db handle 0 > ee dd=93 locks held 1 write locks 0 pid/thread > 4845/140084703647488 flags 10 priority 100 > ee READ 1 HELD ipaca/requesttype.db handle 0 > ef dd=92 locks held 1 write locks 0 pid/thread > 4845/140084703647488 flags 10 priority 100 > ef READ 1 HELD ipaca/numsubordinates.db handle 0 > f0 dd=91 locks held 1 write locks 0 pid/thread > 4845/140084611327744 flags 10 priority 100 > f0 READ 1 HELD userRoot/userCertificate.db handle > 0 > f1 dd=90 locks held 0 write locks 0 pid/thread > 4845/140084821079808 flags 0 priority 100 > f2 dd=89 locks held 0 write locks 0 pid/thread > 4845/140084569364224 flags 0 priority 100 > f3 dd=88 locks held 0 write locks 0 pid/thread > 4845/140084602935040 flags 0 priority 100 > f4 dd=87 locks held 0 write locks 0 pid/thread > 4845/140084720432896 flags 0 priority 100 > f5 dd=86 locks held 0 write locks 0 pid/thread > 4845/140084628113152 flags 0 priority 100 > f6 dd=85 locks held 0 write locks 0 pid/thread > 4845/140084703647488 flags 0 priority 100 > f7 dd=84 locks held 0 write locks 0 pid/thread > 4845/140084611327744 flags 0 priority 100 > f8 dd=83 locks held 0 write locks 0 pid/thread > 4845/140084661683968 flags 0 priority 100 > f9 dd=82 locks held 0 write locks 0 pid/thread > 4845/140084837865216 flags 0 priority 100 > fa dd=81 locks held 0 write locks 0 pid/thread > 4845/140084544186112 flags 0 priority 100 > fb dd=80 locks held 0 write locks 0 pid/thread > 4845/140084703647488 flags 0 priority 100 > fc dd=79 locks held 0 write locks 0 pid/thread > 4845/140084544186112 flags 0 priority 100 > fd dd=78 locks held 0 write locks 0 pid/thread > 4845/140084535793408 flags 0 priority 100 > fe dd=77 locks held 0 write locks 0 pid/thread > 4845/140084586149632 flags 0 priority 100 > ff dd=76 locks held 0 write locks 0 pid/thread > 4845/140084569364224 flags 0 priority 100 > 100 dd=75 locks held 0 write locks 0 pid/thread > 4845/140084804294400 flags 0 priority 100 > 101 dd=74 locks held 0 write locks 0 pid/thread > 4845/140084703647488 flags 0 priority 100 > 102 dd=73 locks held 0 write locks 0 pid/thread > 4845/140084703647488 flags 0 priority 100 > 103 dd=72 locks held 0 write locks 0 pid/thread > 4845/140084821079808 flags 0 priority 100 > 104 dd=71 locks held 0 write locks 0 pid/thread > 4845/140084695254784 flags 0 priority 100 > 105 dd=70 locks held 0 write locks 0 pid/thread > 4845/140084544186112 flags 0 priority 100 > 106 dd=69 locks held 0 write locks 0 pid/thread > 4845/140084854650624 flags 0 priority 100 > 107 dd=68 locks held 0 write locks 0 pid/thread > 4845/140084628113152 flags 0 priority 100 > 108 dd=67 locks held 0 write locks 0 pid/thread > 4845/140084569364224 flags 0 priority 100 > 109 dd=66 locks held 0 write locks 0 pid/thread > 4845/140084594542336 flags 0 priority 100 > 10a dd=65 locks held 0 write locks 0 pid/thread > 4845/140084661683968 flags 0 priority 100 > 10b dd=64 locks held 0 write locks 0 pid/thread > 4845/140084653291264 flags 0 priority 100 > 10c dd=63 locks held 0 write locks 0 pid/thread > 4845/140084695254784 flags 0 priority 100 > 10d dd=62 locks held 0 write locks 0 pid/thread > 4845/140084720432896 flags 0 priority 100 > 10e dd=61 locks held 0 write locks 0 pid/thread > 4845/140084619720448 flags 0 priority 100 > 10f dd=60 locks held 0 write locks 0 pid/thread > 4845/140084720432896 flags 0 priority 100 > 110 dd=59 locks held 0 write locks 0 pid/thread > 4845/140084569364224 flags 0 priority 100 > 111 dd=58 locks held 0 write locks 0 pid/thread > 4845/140084695254784 flags 0 priority 100 > 112 dd=57 locks held 0 write locks 0 pid/thread > 4845/140084653291264 flags 0 priority 100 > 113 dd=56 locks held 0 write locks 0 pid/thread > 4845/140084712040192 flags 0 priority 100 > 114 dd=55 locks held 0 write locks 0 pid/thread > 4845/140084686862080 flags 0 priority 100 > 115 dd=54 locks held 0 write locks 0 pid/thread > 4845/140084577756928 flags 0 priority 100 > 115 READ 1 WAIT userRoot/objectclass.db page 9 > 116 dd=53 locks held 0 write locks 0 pid/thread > 4845/140084837865216 flags 0 priority 100 > 116 READ 1 WAIT userRoot/objectclass.db page 9 > 117 dd=52 locks held 0 write locks 0 pid/thread > 4845/140084577756928 flags 0 priority 100 > 118 dd=51 locks held 0 write locks 0 pid/thread > 4845/140084569364224 flags 0 priority 100 > 119 dd=50 locks held 0 write locks 0 pid/thread > 4845/140084560971520 flags 0 priority 100 > 11a dd=49 locks held 1 write locks 0 pid/thread > 4845/140084695254784 flags 10 priority 100 > 11a READ 1 HELD ipaca/metaInfo.db handle 0 > 11b dd=48 locks held 1 write locks 0 pid/thread > 4845/140084695254784 flags 10 priority 100 > 11b READ 1 HELD ipaca/notbefore.db handle 0 > 11c dd=47 locks held 1 write locks 0 pid/thread > 4845/140084695254784 flags 10 priority 100 > 11c READ 1 HELD ipaca/notafter.db handle 0 > 11d dd=46 locks held 1 write locks 0 pid/thread > 4845/140084695254784 flags 10 priority 100 > 11d READ 1 HELD ipaca/duration.db handle 0 > 11e dd=45 locks held 1 write locks 0 pid/thread > 4845/140084695254784 flags 10 priority 100 > 11e READ 1 HELD ipaca/subjectname.db handle 0 > 11f dd=44 locks held 1 write locks 0 pid/thread > 4845/140084695254784 flags 10 priority 100 > 11f READ 1 HELD ipaca/publicKeyData.db handle 0 > 120 dd=43 locks held 1 write locks 0 pid/thread > 4845/140084695254784 flags 10 priority 100 > 120 READ 1 HELD ipaca/extension.db handle 0 > 121 dd=42 locks held 1 write locks 0 pid/thread > 4845/140084695254784 flags 10 priority 100 > 121 READ 1 HELD ipaca/issuedby.db handle 0 > 122 dd=41 locks held 0 write locks 0 pid/thread > 4845/140084712040192 flags 0 priority 100 > 123 dd=40 locks held 0 write locks 0 pid/thread > 4845/140084812687104 flags 0 priority 100 > 124 dd=39 locks held 0 write locks 0 pid/thread > 4845/140084544186112 flags 0 priority 100 > 125 dd=38 locks held 0 write locks 0 pid/thread > 4845/140084686862080 flags 0 priority 100 > 126 dd=37 locks held 0 write locks 0 pid/thread > 4845/140084552578816 flags 0 priority 100 > 128 dd=36 locks held 0 write locks 0 pid/thread > 4845/140084686862080 flags 0 priority 100 > 12a dd=35 locks held 0 write locks 0 pid/thread > 4845/140084720432896 flags 0 priority 100 > 12b dd=34 locks held 0 write locks 0 pid/thread > 4845/140084552578816 flags 0 priority 100 > 12c dd=33 locks held 0 write locks 0 pid/thread > 4845/140084670076672 flags 0 priority 100 > 12d dd=32 locks held 0 write locks 0 pid/thread > 4845/140084577756928 flags 0 priority 100 > 12e dd=31 locks held 0 write locks 0 pid/thread > 4845/140084712040192 flags 0 priority 100 > 12f dd=30 locks held 0 write locks 0 pid/thread > 4845/140084670076672 flags 0 priority 100 > 130 dd=29 locks held 0 write locks 0 pid/thread > 4845/140084577756928 flags 0 priority 100 > 131 dd=28 locks held 0 write locks 0 pid/thread > 4845/140084560971520 flags 0 priority 100 > 132 dd=27 locks held 0 write locks 0 pid/thread > 4845/140084670076672 flags 0 priority 100 > 133 dd=26 locks held 0 write locks 0 pid/thread > 4845/140084602935040 flags 0 priority 100 > 134 dd=25 locks held 0 write locks 0 pid/thread > 4845/140084552578816 flags 0 priority 100 > 135 dd=24 locks held 0 write locks 0 pid/thread > 4845/140084678469376 flags 0 priority 100 > 136 dd=23 locks held 0 write locks 0 pid/thread > 4845/140084544186112 flags 0 priority 100 > 137 dd=22 locks held 0 write locks 0 pid/thread > 4845/140084577756928 flags 0 priority 100 > 138 dd=21 locks held 0 write locks 0 pid/thread > 4845/140084703647488 flags 0 priority 100 > 139 dd=20 locks held 0 write locks 0 pid/thread > 4845/140084678469376 flags 0 priority 100 > 13b dd=19 locks held 0 write locks 0 pid/thread > 4845/140084812687104 flags 0 priority 100 > 144 dd=18 locks held 0 write locks 0 pid/thread > 4845/140084670076672 flags 0 priority 100 > 156 dd=17 locks held 0 write locks 0 pid/thread > 4845/140084594542336 flags 0 priority 100 > 15a dd=16 locks held 0 write locks 0 pid/thread > 4845/140084577756928 flags 0 priority 100 > 168 dd=15 locks held 0 write locks 0 pid/thread > 4845/140084653291264 flags 0 priority 100 > 16a dd=14 locks held 0 write locks 0 pid/thread > 4845/140084829472512 flags 0 priority 100 > 17d dd=13 locks held 0 write locks 0 pid/thread > 4845/140084829472512 flags 0 priority 100 > 17e dd=12 locks held 0 write locks 0 pid/thread > 4845/140084829472512 flags 0 priority 100 > 17f dd=11 locks held 0 write locks 0 pid/thread > 4845/140084586149632 flags 0 priority 100 > 180 dd=10 locks held 0 write locks 0 pid/thread > 4845/140084594542336 flags 0 priority 100 > 181 dd= 9 locks held 0 write locks 0 pid/thread > 4845/140084577756928 flags 0 priority 100 > 193 dd= 8 locks held 0 write locks 0 pid/thread > 4845/140084812687104 flags 0 priority 100 > 196 dd= 7 locks held 0 write locks 0 pid/thread > 4845/140084695254784 flags 0 priority 100 > 1a1 dd= 6 locks held 0 write locks 0 pid/thread > 4845/140084720432896 flags 0 priority 100 > 1a2 dd= 5 locks held 0 write locks 0 pid/thread > 4845/140084577756928 flags 0 priority 100 > 1a3 dd= 4 locks held 0 write locks 0 pid/thread > 4845/140084812687104 flags 0 priority 100 > 1a4 dd= 3 locks held 0 write locks 0 pid/thread > 4845/140084812687104 flags 0 priority 100 > 1a5 dd= 2 locks held 0 write locks 0 pid/thread > 4845/140084577756928 flags 0 priority 100 > 1a6 dd= 1 locks held 0 write locks 0 pid/thread > 4845/140084594542336 flags 0 priority 100 > 800b3a33 dd= 0 locks held 26 write locks 22 pid/thread > 4845/140084661683968 flags 0 priority 100 > 800b3a33 WRITE 1 HELD userRoot/objectclass.db page 9 > 800b3a33 WRITE 1 HELD userRoot/objectclass.db page 8 > 800b3a33 WRITE 1 HELD userRoot/objectclass.db page 18 > 800b3a33 WRITE 1 HELD userRoot/objectclass.db page 7 > 800b3a33 WRITE 1 HELD userRoot/objectclass.db page 14 > 800b3a33 WRITE 1 HELD userRoot/objectclass.db page 11 > 800b3a33 WRITE 2 HELD userRoot/objectclass.db page 3 > 800b3a33 WRITE 1 HELD userRoot/cn.db page 150 > 800b3a33 WRITE 1 HELD userRoot/cn.db page 89 > 800b3a33 WRITE 1 HELD userRoot/cn.db page 53 > 800b3a33 WRITE 1 HELD userRoot/cn.db page 131 > 800b3a33 WRITE 1 HELD userRoot/cn.db page 117 > 800b3a33 WRITE 1 HELD userRoot/cn.db page 11 > 800b3a33 WRITE 2 HELD userRoot/cn.db page 141 > 800b3a33 WRITE 1 HELD userRoot/cn.db page 41 > 800b3a33 WRITE 1 HELD userRoot/cn.db page 40 > 800b3a33 WRITE 1 HELD userRoot/cn.db page 143 > 800b3a33 WRITE 1 HELD userRoot/cn.db page 30 > 800b3a33 WRITE 2 HELD userRoot/id2entry.db page 8184 > 800b3a33 WRITE 1 HELD userRoot/id2entry.db page 8183 > 800b3a33 WRITE 1 HELD userRoot/id2entry.db page 7779 > 800b3a33 WRITE 1 HELD userRoot/id2entry.db page 0 > 800b3a33 READ 3 HELD userRoot/entryrdn.db page 3 > 800b3a33 READ 1 HELD userRoot/entryrdn.db page 36 > 800b3a33 READ 1 HELD userRoot/entryrdn.db page 4 > 800b3a33 READ 1 HELD userRoot/nsuniqueid.db page 86 > =-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-= > Locks grouped by object: > Locker Mode Count Status ----------------- Object --------------- > a4 READ 1 HELD userRoot/ipasudorunas.db handle 0 > > 9e READ 1 HELD userRoot/ipaMemberCertProfile.db > handle 0 > > 10 READ 1 HELD ipaca/vlv#allrevokedcertspkitomcatindex.db > handle 0 > > 800b3a33 WRITE 2 HELD userRoot/id2entry.db page 8184 > > 800b3a33 WRITE 1 HELD userRoot/id2entry.db page 8183 > > 800b3a33 WRITE 1 HELD userRoot/id2entry.db page 7779 > > a1 READ 1 HELD userRoot/memberdenycmd.db handle 0 > > a READ 1 HELD ipaca/vlv#allcertspkitomcatindex.db > handle 0 > > a7 READ 1 HELD userRoot/sourcehost.db handle 0 > > 57 READ 1 HELD changelog/ancestorid.db handle 0 > > e5 READ 1 HELD ipaca/revInfo.db handle 0 > > e6 READ 1 HELD ipaca/revokedby.db handle 0 > > 11 READ 1 HELD ipaca/vlv#allrevokedcertsnotafterpkitomcatindex.db > handle 0 > > 4e READ 1 HELD userRoot/parentid.db handle 0 > > 4 READ 1 HELD ipaca/id2entry.db handle 0 > > 58 READ 1 HELD changelog/numsubordinates.db > handle 0 > > 67 READ 1 HELD ipaca/seeAlso.db handle 0 > > 2b READ 1 HELD ipaca/vlv#carevocationpkitomcatindex.db > handle 0 > > ec READ 1 HELD ipaca/requeststate.db handle 0 > > 11b READ 1 HELD ipaca/notbefore.db handle 0 > > 7e READ 1 HELD userRoot/numsubordinates.db handle > 0 > > a2 READ 1 HELD userRoot/memberallowcmd.db handle 0 > > 9c READ 1 HELD userRoot/owner.db handle 0 > > 1e READ 1 HELD ipaca/vlv#cacompleteenrollmentpkitomcatindex.db > handle 0 > > 11c READ 1 HELD ipaca/notafter.db handle 0 > > 40 READ 1 HELD changelog/objectclass.db handle 0 > > 2e READ 1 HELD changelog/entryusn.db handle 0 > > 16 READ 1 HELD ipaca/vlv#allvalidcertsnotafterpkitomcatindex.db > handle 0 > > 15 READ 1 HELD ipaca/vlv#allvalidcertspkitomcatindex.db > handle 0 > > a5 READ 1 HELD userRoot/ipaMemberCa.db handle 0 > > 18 READ 1 HELD ipaca/vlv#caallpkitomcatindex.db > handle 0 > > a0 READ 1 HELD userRoot/secretary.db handle 0 > > 81 READ 1 HELD userRoot/memberUser.db handle 0 > > 9f READ 1 HELD userRoot/ipaassignedidview.db > handle 0 > > a3 READ 1 HELD userRoot/manager.db handle 0 > > d8 READ 1 HELD ipaca/description.db handle 0 > > a8 READ 1 HELD userRoot/memberservice.db handle 0 > > 44 READ 1 HELD ipaca/objectclass.db handle 0 > > ed READ 1 HELD ipaca/dateOfCreate.db handle 0 > > 800b3a33 WRITE 1 HELD userRoot/id2entry.db page 0 > > 2 READ 1 HELD userRoot/id2entry.db handle 0 > > 5b READ 1 HELD userRoot/krbPrincipalName.db > handle 0 > > ee READ 1 HELD ipaca/requesttype.db handle 0 > > c8 READ 1 HELD ipaca/certstatus.db handle 0 > > 56 READ 1 HELD changelog/parentid.db handle 0 > > 83 READ 1 HELD userRoot/memberHost.db handle 0 > > 5e READ 1 HELD userRoot/ipakrbprincipalalias.db > handle 0 > > 121 READ 1 HELD ipaca/issuedby.db handle 0 > > 6 READ 1 HELD ipaca/entryrdn.db handle 0 > > e7 READ 1 HELD ipaca/revokedOn.db handle 0 > > 4c READ 1 HELD ipaca/aci.db handle 0 > > 11d READ 1 HELD ipaca/duration.db handle 0 > > 14 READ 1 HELD ipaca/vlv#allrevokedorrevokedexpiredcertspkitomcatindex.db > handle 0 > > 21 READ 1 HELD ipaca/vlv#caenrollmentpkitomcatindex.db > handle 0 > > 55 READ 1 HELD changelog/targetuniqueid.db handle > 0 > > 800b3a33 READ 1 HELD userRoot/entryrdn.db page 36 > > 2c READ 1 HELD changelog/id2entry.db handle 0 > > 800b3a33 READ 1 HELD userRoot/entryrdn.db page 4 > > 34 READ 1 HELD userRoot/entryrdn.db handle 0 > > 800b3a33 READ 3 HELD userRoot/entryrdn.db page 3 > > eb READ 1 HELD ipaca/requestid.db handle 0 > > 49 READ 1 HELD userRoot/aci.db handle 0 > > 76 READ 1 HELD userRoot/ipauniqueid.db handle 0 > > 63 READ 1 HELD changelog/seeAlso.db handle 0 > > 7c READ 1 HELD userRoot/fqdn.db handle 0 > > db READ 1 HELD ipaca/cn.db handle 0 > > ef READ 1 HELD ipaca/numsubordinates.db handle 0 > > 32 READ 1 HELD ipaca/entryusn.db handle 0 > > 1d READ 1 HELD ipaca/vlv#cacompletepkitomcatindex.db > handle 0 > > dd READ 1 HELD ipaca/ancestorid.db handle 0 > > a9 READ 1 HELD userRoot/ipatokenradiusconfiglink.db > handle 0 > > f0 READ 1 HELD userRoot/userCertificate.db handle > 0 > > 11a READ 1 HELD ipaca/metaInfo.db handle 0 > > 7d READ 1 HELD userRoot/managedby.db handle 0 > > ea READ 1 HELD ipaca/nsuniqueid.db handle 0 > > c5 READ 1 HELD userRoot/uidnumber.db handle 0 > > d1 READ 1 HELD userRoot/nscpEntryDN.db handle 0 > > 120 READ 1 HELD ipaca/extension.db handle 0 > > 800b3a33 WRITE 1 HELD userRoot/cn.db page 150 > > 69 READ 1 HELD ipaca/parentid.db handle 0 > > 800b3a33 WRITE 1 HELD userRoot/cn.db page 131 > > 800b3a33 WRITE 1 HELD userRoot/cn.db page 143 > > 800b3a33 WRITE 2 HELD userRoot/cn.db page 141 > > 20 READ 1 HELD ipaca/vlv#cacompleterevocationpkitomcatindex.db > handle 0 > > 8e READ 1 HELD userRoot/macAddress.db handle 0 > > e READ 1 HELD ipaca/vlv#allnonrevokedcertspkitomcatindex.db > handle 0 > > 17 READ 1 HELD ipaca/vlv#allvalidorrevokedcertspkitomcatindex.db > handle 0 > > 800b3a33 WRITE 1 HELD userRoot/cn.db page 30 > > 800b3a33 WRITE 1 HELD userRoot/objectclass.db page 14 > > 800b3a33 WRITE 1 HELD userRoot/objectclass.db page 9 > 116 READ 1 WAIT userRoot/objectclass.db page 9 > 115 READ 1 WAIT userRoot/objectclass.db page 9 > 92 READ 1 WAIT userRoot/objectclass.db page 9 > > 800b3a33 WRITE 1 HELD userRoot/objectclass.db page 8 > > 7b READ 1 HELD userRoot/cn.db handle 0 > > 800b3a33 WRITE 1 HELD userRoot/objectclass.db page 11 > > 800b3a33 WRITE 1 HELD userRoot/objectclass.db page 7 > > 800b3a33 WRITE 1 HELD userRoot/cn.db page 11 > > 38 READ 1 HELD userRoot/objectclass.db handle 0 > > 800b3a33 WRITE 2 HELD userRoot/objectclass.db page 3 > > 800b3a33 WRITE 1 HELD userRoot/cn.db page 53 > > 800b3a33 WRITE 1 HELD userRoot/objectclass.db page 18 > 6e READ 1 WAIT userRoot/objectclass.db page 18 > 94 READ 1 WAIT userRoot/objectclass.db page 18 > > 800b3a33 WRITE 1 HELD userRoot/cn.db page 41 > > 800b3a33 WRITE 1 HELD userRoot/cn.db page 40 > > 800b3a33 WRITE 1 HELD userRoot/cn.db page 89 > > 800b3a33 WRITE 1 HELD userRoot/cn.db page 117 > > 30 READ 1 HELD userRoot/entryusn.db handle 0 > > 3c READ 1 HELD changelog/entryrdn.db handle 0 > > 97 READ 1 HELD userRoot/uid.db handle 0 > > 47 READ 1 HELD changelog/aci.db handle 0 > > e3 READ 1 HELD ipaca/serialno.db handle 0 > > 7a READ 1 HELD userRoot/nsuniqueid.db handle 0 > > 800b3a33 READ 1 HELD userRoot/nsuniqueid.db page 86 > > 51 READ 1 HELD changelog/nsuniqueid.db handle 0 > > 11e READ 1 HELD ipaca/subjectname.db handle 0 > > 7f READ 1 HELD userRoot/member.db handle 0 > > 11f READ 1 HELD ipaca/publicKeyData.db handle 0 > > 9b READ 1 HELD userRoot/uniquemember.db handle 0 > > 52 READ 1 HELD changelog/changenumber.db handle 0 > > ce READ 1 HELD userRoot/gidnumber.db handle 0 > > 65 READ 1 HELD userRoot/seeAlso.db handle 0 > > 9d READ 1 HELD userRoot/ipaallowedtarget.db > handle 0 > > a6 READ 1 HELD userRoot/ipasudorunasgroup.db > handle 0 > > 3a READ 1 HELD userRoot/ancestorid.db handle 0 > > > > > pstack output > > root at ipa-master ]# pstack 4845 > Thread 44 (Thread 0x7f681dd3a700 (LWP 4847)): > #0 0x00007f683cd719b3 in select () from /lib64/libc.so.6 > #1 0x00007f683f4fc0e9 in DS_Sleep () from /usr/lib64/dirsrv/libslapd.so.0 > #2 0x00007f6832b4a907 in deadlock_threadmain () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #3 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so > #4 0x00007f683d04cdc5 in start_thread () from /lib64/libpthread.so.0 > #5 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 > Thread 43 (Thread 0x7f681d539700 (LWP 4848)): > #0 0x00007f683cd719b3 in select () from /lib64/libc.so.6 > #1 0x00007f68376d8bad in __os_yield () from /lib64/libdb-5.3.so > #2 0x00007f68376d42b3 in __memp_sync_int () from /lib64/libdb-5.3.so > #3 0x00007f68376e4752 in __txn_checkpoint () from /lib64/libdb-5.3.so > #4 0x00007f68376e4b74 in __txn_checkpoint_pp () from /lib64/libdb-5.3.so > #5 0x00007f6832b4ea87 in checkpoint_threadmain () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #6 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so > #7 0x00007f683d04cdc5 in start_thread () from /lib64/libpthread.so.0 > #8 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 > Thread 42 (Thread 0x7f681cd38700 (LWP 4849)): > #0 0x00007f683cd719b3 in select () from /lib64/libc.so.6 > #1 0x00007f683f4fc0e9 in DS_Sleep () from /usr/lib64/dirsrv/libslapd.so.0 > #2 0x00007f6832b4ab7f in trickle_threadmain () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #3 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so > #4 0x00007f683d04cdc5 in start_thread () from /lib64/libpthread.so.0 > #5 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 > Thread 41 (Thread 0x7f681c537700 (LWP 4850)): > #0 0x00007f683cd719b3 in select () from /lib64/libc.so.6 > #1 0x00007f683f4fc0e9 in DS_Sleep () from /usr/lib64/dirsrv/libslapd.so.0 > #2 0x00007f6832b45707 in perf_threadmain () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #3 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so > #4 0x00007f683d04cdc5 in start_thread () from /lib64/libpthread.so.0 > #5 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 > Thread 40 (Thread 0x7f681bb2d700 (LWP 4851)): > #0 0x00007f683d0506d5 in pthread_cond_wait@@GLIBC_2.3.2 () from > /lib64/libpthread.so.0 > #1 0x00007f683d6a6050 in PR_WaitCondVar () from /lib64/libnspr4.so > #2 0x00007f683f4eb198 in slapi_wait_condvar () from > /usr/lib64/dirsrv/libslapd.so.0 > #3 0x00007f683587d62e in cos_cache_wait_on_change () from > /usr/lib64/dirsrv/plugins/libcos-plugin.so > #4 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so > #5 0x00007f683d04cdc5 in start_thread () from /lib64/libpthread.so.0 > #6 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 > Thread 39 (Thread 0x7f681b32c700 (LWP 4852)): > #0 0x00007f683d0506d5 in pthread_cond_wait@@GLIBC_2.3.2 () from > /lib64/libpthread.so.0 > #1 0x00007f683d6a6050 in PR_WaitCondVar () from /lib64/libnspr4.so > #2 0x00007f683f4eb198 in slapi_wait_condvar () from > /usr/lib64/dirsrv/libslapd.so.0 > #3 0x00007f68311e8eed in roles_cache_wait_on_change () from > /usr/lib64/dirsrv/plugins/libroles-plugin.so > #4 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so > #5 0x00007f683d04cdc5 in start_thread () from /lib64/libpthread.so.0 > #6 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 > Thread 38 (Thread 0x7f681ab2b700 (LWP 4853)): > #0 0x00007f683d0506d5 in pthread_cond_wait@@GLIBC_2.3.2 () from > /lib64/libpthread.so.0 > #1 0x00007f683d6a6050 in PR_WaitCondVar () from /lib64/libnspr4.so > #2 0x00007f683f4eb198 in slapi_wait_condvar () from > /usr/lib64/dirsrv/libslapd.so.0 > #3 0x00007f68311e8eed in roles_cache_wait_on_change () from > /usr/lib64/dirsrv/plugins/libroles-plugin.so > #4 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so > #5 0x00007f683d04cdc5 in start_thread () from /lib64/libpthread.so.0 > #6 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 > Thread 37 (Thread 0x7f681a32a700 (LWP 4854)): > #0 0x00007f683d0506d5 in pthread_cond_wait@@GLIBC_2.3.2 () from > /lib64/libpthread.so.0 > #1 0x00007f683d6a6050 in PR_WaitCondVar () from /lib64/libnspr4.so > #2 0x00007f683f4eb198 in slapi_wait_condvar () from > /usr/lib64/dirsrv/libslapd.so.0 > #3 0x00007f68311e8eed in roles_cache_wait_on_change () from > /usr/lib64/dirsrv/plugins/libroles-plugin.so > #4 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so > #5 0x00007f683d04cdc5 in start_thread () from /lib64/libpthread.so.0 > #6 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 > Thread 36 (Thread 0x7f6819b29700 (LWP 4855)): > #0 0x00007f683d050a82 in pthread_cond_timedwait@@GLIBC_2.3.2 () from > /lib64/libpthread.so.0 > #1 0x00007f683d6a5b07 in pt_TimedWait () from /lib64/libnspr4.so > #2 0x00007f683d6a5fce in PR_WaitCondVar () from /lib64/libnspr4.so > #3 0x00007f683f9a4d03 in housecleaning () > #4 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so > #5 0x00007f683d04cdc5 in start_thread () from /lib64/libpthread.so.0 > #6 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 > Thread 35 (Thread 0x7f6819328700 (LWP 4856)): > #0 0x00007f683d050a82 in pthread_cond_timedwait@@GLIBC_2.3.2 () from > /lib64/libpthread.so.0 > #1 0x00007f683d6a5b07 in pt_TimedWait () from /lib64/libnspr4.so > #2 0x00007f683d6a5fce in PR_WaitCondVar () from /lib64/libnspr4.so > #3 0x00007f683f4911e8 in eq_loop () from /usr/lib64/dirsrv/libslapd.so.0 > #4 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so > #5 0x00007f683d04cdc5 in start_thread () from /lib64/libpthread.so.0 > #6 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 > Thread 34 (Thread 0x7f680bfff700 (LWP 4859)): > #0 0x00007f683d0506d5 in pthread_cond_wait@@GLIBC_2.3.2 () from > /lib64/libpthread.so.0 > #1 0x00007f683d6a6050 in PR_WaitCondVar () from /lib64/libnspr4.so > #2 0x00007f683f999fee in connection_wait_for_new_work () > #3 0x00007f683f99b719 in connection_threadmain () > #4 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so > #5 0x00007f683d04cdc5 in start_thread () from /lib64/libpthread.so.0 > #6 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 > Thread 33 (Thread 0x7f680b7fe700 (LWP 4860)): > #0 0x00007f683d0506d5 in pthread_cond_wait@@GLIBC_2.3.2 () from > /lib64/libpthread.so.0 > #1 0x00007f683d6a6050 in PR_WaitCondVar () from /lib64/libnspr4.so > #2 0x00007f683f999fee in connection_wait_for_new_work () > #3 0x00007f683f99b719 in connection_threadmain () > #4 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so > #5 0x00007f683d04cdc5 in start_thread () from /lib64/libpthread.so.0 > #6 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 > Thread 32 (Thread 0x7f680affd700 (LWP 4861)): > #0 0x00007f683d0506d5 in pthread_cond_wait@@GLIBC_2.3.2 () from > /lib64/libpthread.so.0 > #1 0x00007f68375ac2f3 in __db_hybrid_mutex_suspend () from /lib64/ > libdb-5.3.so > #2 0x00007f68375ab640 in __db_tas_mutex_lock () from /lib64/libdb-5.3.so > #3 0x00007f6837655d3a in __lock_get_internal () from /lib64/libdb-5.3.so > #4 0x00007f6837656820 in __lock_get () from /lib64/libdb-5.3.so > #5 0x00007f6837682142 in __db_lget () from /lib64/libdb-5.3.so > #6 0x00007f68375c9605 in __bam_search () from /lib64/libdb-5.3.so > #7 0x00007f68375b4256 in __bamc_search () from /lib64/libdb-5.3.so > #8 0x00007f68375b5d0f in __bamc_get () from /lib64/libdb-5.3.so > #9 0x00007f683766eca6 in __dbc_iget () from /lib64/libdb-5.3.so > #10 0x00007f683767db02 in __dbc_get_pp () from /lib64/libdb-5.3.so > #11 0x00007f6832b59680 in idl_new_fetch () from /usr/lib64/dirsrv/plugins/ > libback-ldbm.so > #12 0x00007f6832b67b26 in index_read_ext_allids () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #13 0x00007f6832b52284 in keys2idl () from /usr/lib64/dirsrv/plugins/ > libback-ldbm.so > #14 0x00007f6832b529e3 in ava_candidates.isra.0 () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #15 0x00007f6832b52fd2 in filter_candidates_ext () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #16 0x00007f6832b54096 in list_candidates () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #17 0x00007f6832b52f40 in filter_candidates_ext () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #18 0x00007f6832b54096 in list_candidates () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #19 0x00007f6832b52f40 in filter_candidates_ext () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #20 0x00007f6832b54096 in list_candidates () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #21 0x00007f6832b52f40 in filter_candidates_ext () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #22 0x00007f6832b8e80c in subtree_candidates () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #23 0x00007f6832b8fe7f in ldbm_back_search () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #24 0x00007f683f4bd1d6 in op_shared_search () from > /usr/lib64/dirsrv/libslapd.so.0 > #25 0x00007f683f9ae33e in do_search () > #26 0x00007f683f99c3c3 in connection_threadmain () > #27 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so > #28 0x00007f683d04cdc5 in start_thread () from /lib64/libpthread.so.0 > #29 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 > Thread 31 (Thread 0x7f680a7fc700 (LWP 4862)): > #0 0x00007f683d0506d5 in pthread_cond_wait@@GLIBC_2.3.2 () from > /lib64/libpthread.so.0 > #1 0x00007f683d6a62c3 in PR_EnterMonitor () from /lib64/libnspr4.so > #2 0x00007f6832b4d3e6 in dblayer_txn_begin () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #3 0x00007f6832b89007 in ldbm_back_modify () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #4 0x00007f683f4b6541 in op_shared_modify () from > /usr/lib64/dirsrv/libslapd.so.0 > #5 0x00007f683f4b7004 in modify_internal_pb () from > /usr/lib64/dirsrv/libslapd.so.0 > #6 0x00007f683483bbbb in ipalockout_postop () from > /usr/lib64/dirsrv/plugins/libipa_lockout.so > #7 0x00007f683f4c8848 in plugin_call_func () from > /usr/lib64/dirsrv/libslapd.so.0 > #8 0x00007f683f4c8b13 in plugin_call_plugins () from > /usr/lib64/dirsrv/libslapd.so.0 > #9 0x00007f683f9954d8 in do_bind () > #10 0x00007f683f99c3fd in connection_threadmain () > #11 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so > #12 0x00007f683d04cdc5 in start_thread () from /lib64/libpthread.so.0 > #13 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 > Thread 30 (Thread 0x7f6809ffb700 (LWP 4863)): > #0 0x00007f683d0506d5 in pthread_cond_wait@@GLIBC_2.3.2 () from > /lib64/libpthread.so.0 > #1 0x00007f68375ac2f3 in __db_hybrid_mutex_suspend () from /lib64/ > libdb-5.3.so > #2 0x00007f68375ab640 in __db_tas_mutex_lock () from /lib64/libdb-5.3.so > #3 0x00007f6837655d3a in __lock_get_internal () from /lib64/libdb-5.3.so > #4 0x00007f6837656820 in __lock_get () from /lib64/libdb-5.3.so > #5 0x00007f6837682142 in __db_lget () from /lib64/libdb-5.3.so > #6 0x00007f68375c9605 in __bam_search () from /lib64/libdb-5.3.so > #7 0x00007f68375b4256 in __bamc_search () from /lib64/libdb-5.3.so > #8 0x00007f68375b5d0f in __bamc_get () from /lib64/libdb-5.3.so > #9 0x00007f683766eca6 in __dbc_iget () from /lib64/libdb-5.3.so > #10 0x00007f683767db02 in __dbc_get_pp () from /lib64/libdb-5.3.so > #11 0x00007f6832b59680 in idl_new_fetch () from /usr/lib64/dirsrv/plugins/ > libback-ldbm.so > #12 0x00007f6832b67b26 in index_read_ext_allids () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #13 0x00007f6832b52284 in keys2idl () from /usr/lib64/dirsrv/plugins/ > libback-ldbm.so > #14 0x00007f6832b529e3 in ava_candidates.isra.0 () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #15 0x00007f6832b52fd2 in filter_candidates_ext () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #16 0x00007f6832b54096 in list_candidates () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #17 0x00007f6832b52f40 in filter_candidates_ext () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #18 0x00007f6832b8e80c in subtree_candidates () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #19 0x00007f6832b8fe7f in ldbm_back_search () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #20 0x00007f683f4bd1d6 in op_shared_search () from > /usr/lib64/dirsrv/libslapd.so.0 > #21 0x00007f683f4cd94e in search_internal_callback_pb () from > /usr/lib64/dirsrv/libslapd.so.0 > #22 0x00007f683f4cdbe8 in search_internal_pb () from > /usr/lib64/dirsrv/libslapd.so.0 > #23 0x00007f683f9ac49b in ids_sasl_canon_user () > #24 0x00007f683d46c93b in _sasl_canon_user () from /lib64/libsasl2.so.3 > #25 0x00007f683d46cc4c in _sasl_canon_user_lookup () from > /lib64/libsasl2.so.3 > #26 0x00007f6837374fb9 in gssapi_server_mech_step () from > /usr/lib64/sasl2/libgssapiv2.so > #27 0x00007f683d478b9b in sasl_server_step () from /lib64/libsasl2.so.3 > #28 0x00007f683f9ad83b in ids_sasl_check_bind () > #29 0x00007f683f99561a in do_bind () > #30 0x00007f683f99c3fd in connection_threadmain () > #31 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so > #32 0x00007f683d04cdc5 in start_thread () from /lib64/libpthread.so.0 > #33 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 > Thread 29 (Thread 0x7f68097fa700 (LWP 4864)): > #0 0x00007f683d0506d5 in pthread_cond_wait@@GLIBC_2.3.2 () from > /lib64/libpthread.so.0 > #1 0x00007f683d6a6050 in PR_WaitCondVar () from /lib64/libnspr4.so > #2 0x00007f683f999fee in connection_wait_for_new_work () > #3 0x00007f683f99b719 in connection_threadmain () > #4 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so > #5 0x00007f683d04cdc5 in start_thread () from /lib64/libpthread.so.0 > #6 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 > Thread 28 (Thread 0x7f6808ff9700 (LWP 4865)): > #0 0x00007f683d0506d5 in pthread_cond_wait@@GLIBC_2.3.2 () from > /lib64/libpthread.so.0 > #1 0x00007f683d6a6050 in PR_WaitCondVar () from /lib64/libnspr4.so > #2 0x00007f683f999fee in connection_wait_for_new_work () > #3 0x00007f683f99b719 in connection_threadmain () > #4 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so > #5 0x00007f683d04cdc5 in start_thread () from /lib64/libpthread.so.0 > #6 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 > Thread 27 (Thread 0x7f6803fff700 (LWP 4866)): > #0 0x00007f683d0506d5 in pthread_cond_wait@@GLIBC_2.3.2 () from > /lib64/libpthread.so.0 > #1 0x00007f683d6a6050 in PR_WaitCondVar () from /lib64/libnspr4.so > #2 0x00007f683f999fee in connection_wait_for_new_work () > #3 0x00007f683f99b719 in connection_threadmain () > #4 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so > #5 0x00007f683d04cdc5 in start_thread () from /lib64/libpthread.so.0 > #6 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 > Thread 26 (Thread 0x7f68037fe700 (LWP 4867)): > #0 0x00007f683d0506d5 in pthread_cond_wait@@GLIBC_2.3.2 () from > /lib64/libpthread.so.0 > #1 0x00007f683d6a6050 in PR_WaitCondVar () from /lib64/libnspr4.so > #2 0x00007f683f999fee in connection_wait_for_new_work () > #3 0x00007f683f99b719 in connection_threadmain () > #4 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so > #5 0x00007f683d04cdc5 in start_thread () from /lib64/libpthread.so.0 > #6 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 > Thread 25 (Thread 0x7f6802ffd700 (LWP 4868)): > #0 0x00007f683d0506d5 in pthread_cond_wait@@GLIBC_2.3.2 () from > /lib64/libpthread.so.0 > #1 0x00007f683d6a6050 in PR_WaitCondVar () from /lib64/libnspr4.so > #2 0x00007f683f999fee in connection_wait_for_new_work () > #3 0x00007f683f99b719 in connection_threadmain () > #4 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so > #5 0x00007f683d04cdc5 in start_thread () from /lib64/libpthread.so.0 > #6 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 > Thread 24 (Thread 0x7f68027fc700 (LWP 4869)): > #0 0x00007f683d0506d5 in pthread_cond_wait@@GLIBC_2.3.2 () from > /lib64/libpthread.so.0 > #1 0x00007f68375ac2f3 in __db_hybrid_mutex_suspend () from /lib64/ > libdb-5.3.so > #2 0x00007f68375abb37 in __db_tas_mutex_readlock () from /lib64/ > libdb-5.3.so > #3 0x00007f68376c5d35 in __memp_fget () from /lib64/libdb-5.3.so > #4 0x00007f683766f333 in __dbc_iget () from /lib64/libdb-5.3.so > #5 0x00007f683767db02 in __dbc_get_pp () from /lib64/libdb-5.3.so > #6 0x00007f6832b59822 in idl_new_fetch () from /usr/lib64/dirsrv/plugins/ > libback-ldbm.so > #7 0x00007f6832b67b26 in index_read_ext_allids () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #8 0x00007f6832b52284 in keys2idl () from /usr/lib64/dirsrv/plugins/ > libback-ldbm.so > #9 0x00007f6832b529e3 in ava_candidates.isra.0 () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #10 0x00007f6832b52fd2 in filter_candidates_ext () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #11 0x00007f6832b54096 in list_candidates () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #12 0x00007f6832b52f40 in filter_candidates_ext () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #13 0x00007f6832b54096 in list_candidates () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #14 0x00007f6832b52f40 in filter_candidates_ext () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #15 0x00007f6832b54096 in list_candidates () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #16 0x00007f6832b52f40 in filter_candidates_ext () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #17 0x00007f6832b8e80c in subtree_candidates () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #18 0x00007f6832b8fe7f in ldbm_back_search () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #19 0x00007f683f4bd1d6 in op_shared_search () from > /usr/lib64/dirsrv/libslapd.so.0 > #20 0x00007f683f9ae33e in do_search () > #21 0x00007f683f99c3c3 in connection_threadmain () > #22 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so > #23 0x00007f683d04cdc5 in start_thread () from /lib64/libpthread.so.0 > #24 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 > Thread 23 (Thread 0x7f6801ffb700 (LWP 4870)): > #0 0x00007f683d0506d5 in pthread_cond_wait@@GLIBC_2.3.2 () from > /lib64/libpthread.so.0 > #1 0x00007f683d6a6050 in PR_WaitCondVar () from /lib64/libnspr4.so > #2 0x00007f683f999fee in connection_wait_for_new_work () > #3 0x00007f683f99b719 in connection_threadmain () > #4 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so > #5 0x00007f683d04cdc5 in start_thread () from /lib64/libpthread.so.0 > #6 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 > Thread 22 (Thread 0x7f68017fa700 (LWP 4871)): > #0 0x00007f683d0506d5 in pthread_cond_wait@@GLIBC_2.3.2 () from > /lib64/libpthread.so.0 > #1 0x00007f683d6a6050 in PR_WaitCondVar () from /lib64/libnspr4.so > #2 0x00007f683f999fee in connection_wait_for_new_work () > #3 0x00007f683f99b719 in connection_threadmain () > #4 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so > #5 0x00007f683d04cdc5 in start_thread () from /lib64/libpthread.so.0 > #6 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 > Thread 21 (Thread 0x7f6800ff9700 (LWP 4872)): > #0 0x00007f683d0506d5 in pthread_cond_wait@@GLIBC_2.3.2 () from > /lib64/libpthread.so.0 > #1 0x00007f683d6a6050 in PR_WaitCondVar () from /lib64/libnspr4.so > #2 0x00007f683f999fee in connection_wait_for_new_work () > #3 0x00007f683f99b719 in connection_threadmain () > #4 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so > #5 0x00007f683d04cdc5 in start_thread () from /lib64/libpthread.so.0 > #6 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 > Thread 20 (Thread 0x7f68007f8700 (LWP 4873)): > #0 0x00007f683d0506d5 in pthread_cond_wait@@GLIBC_2.3.2 () from > /lib64/libpthread.so.0 > #1 0x00007f68375ac2f3 in __db_hybrid_mutex_suspend () from /lib64/ > libdb-5.3.so > #2 0x00007f68375ab640 in __db_tas_mutex_lock () from /lib64/libdb-5.3.so > #3 0x00007f68376c5e1b in __memp_fget () from /lib64/libdb-5.3.so > #4 0x00007f68375c9211 in __bam_search () from /lib64/libdb-5.3.so > #5 0x00007f68375b4256 in __bamc_search () from /lib64/libdb-5.3.so > #6 0x00007f68375b86c4 in __bamc_put () from /lib64/libdb-5.3.so > #7 0x00007f683766f6d5 in __dbc_iput () from /lib64/libdb-5.3.so > #8 0x00007f683766aa0e in __db_put () from /lib64/libdb-5.3.so > #9 0x00007f683767ffa4 in __db_put_pp () from /lib64/libdb-5.3.so > #10 0x00007f6832b5a4cf in idl_new_insert_key () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #11 0x00007f6832b695b5 in addordel_values_sv.isra.3 () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #12 0x00007f6832b6a25a in index_addordel_values_ext_sv () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #13 0x00007f6832b6a694 in index_addordel_values_sv () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #14 0x00007f6832b6a75a in index_addordel_entry () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #15 0x00007f6832b6d5fa in ldbm_back_add () from /usr/lib64/dirsrv/plugins/ > libback-ldbm.so > #16 0x00007f683f46fcca in op_shared_add () from > /usr/lib64/dirsrv/libslapd.so.0 > #17 0x00007f683f471020 in do_add () from /usr/lib64/dirsrv/libslapd.so.0 > #18 0x00007f683f99c303 in connection_threadmain () > #19 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so > #20 0x00007f683d04cdc5 in start_thread () from /lib64/libpthread.so.0 > #21 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 > Thread 19 (Thread 0x7f67ffff7700 (LWP 4874)): > #0 0x00007f683d0506d5 in pthread_cond_wait@@GLIBC_2.3.2 () from > /lib64/libpthread.so.0 > #1 0x00007f683d6a62c3 in PR_EnterMonitor () from /lib64/libnspr4.so > #2 0x00007f6832b4d3e6 in dblayer_txn_begin () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #3 0x00007f6832b89007 in ldbm_back_modify () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #4 0x00007f683f4b6541 in op_shared_modify () from > /usr/lib64/dirsrv/libslapd.so.0 > #5 0x00007f683f4b7004 in modify_internal_pb () from > /usr/lib64/dirsrv/libslapd.so.0 > #6 0x00007f6832dd3b60 in ipapwd_apply_mods () from > /usr/lib64/dirsrv/plugins/libipa_pwd_extop.so > #7 0x00007f6832dd78e0 in store_new_keys () from /usr/lib64/dirsrv/plugins/ > libipa_pwd_extop.so > #8 0x00007f6832dd9ad0 in ipapwd_extop () from /usr/lib64/dirsrv/plugins/ > libipa_pwd_extop.so > #9 0x00007f683f4c6ba2 in plugin_call_exop_plugins () from > /usr/lib64/dirsrv/libslapd.so.0 > #10 0x00007f683f9a2efe in do_extended () > #11 0x00007f683f99c41a in connection_threadmain () > #12 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so > #13 0x00007f683d04cdc5 in start_thread () from /lib64/libpthread.so.0 > #14 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 > Thread 18 (Thread 0x7f67ff7f6700 (LWP 4875)): > #0 0x00007f683d0506d5 in pthread_cond_wait@@GLIBC_2.3.2 () from > /lib64/libpthread.so.0 > #1 0x00007f683d6a6050 in PR_WaitCondVar () from /lib64/libnspr4.so > #2 0x00007f683f999fee in connection_wait_for_new_work () > #3 0x00007f683f99b719 in connection_threadmain () > #4 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so > #5 0x00007f683d04cdc5 in start_thread () from /lib64/libpthread.so.0 > #6 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 > Thread 17 (Thread 0x7f67feff5700 (LWP 4876)): > #0 0x00007f683d0506d5 in pthread_cond_wait@@GLIBC_2.3.2 () from > /lib64/libpthread.so.0 > #1 0x00007f68375ac2f3 in __db_hybrid_mutex_suspend () from /lib64/ > libdb-5.3.so > #2 0x00007f68375ab640 in __db_tas_mutex_lock () from /lib64/libdb-5.3.so > #3 0x00007f6837655d3a in __lock_get_internal () from /lib64/libdb-5.3.so > #4 0x00007f6837656820 in __lock_get () from /lib64/libdb-5.3.so > #5 0x00007f6837682142 in __db_lget () from /lib64/libdb-5.3.so > #6 0x00007f68375c9605 in __bam_search () from /lib64/libdb-5.3.so > #7 0x00007f68375b4256 in __bamc_search () from /lib64/libdb-5.3.so > #8 0x00007f68375b5d0f in __bamc_get () from /lib64/libdb-5.3.so > #9 0x00007f683766eca6 in __dbc_iget () from /lib64/libdb-5.3.so > #10 0x00007f683767db02 in __dbc_get_pp () from /lib64/libdb-5.3.so > #11 0x00007f6832b59680 in idl_new_fetch () from /usr/lib64/dirsrv/plugins/ > libback-ldbm.so > #12 0x00007f6832b67b26 in index_read_ext_allids () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #13 0x00007f6832b52284 in keys2idl () from /usr/lib64/dirsrv/plugins/ > libback-ldbm.so > #14 0x00007f6832b529e3 in ava_candidates.isra.0 () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #15 0x00007f6832b52fd2 in filter_candidates_ext () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #16 0x00007f6832b54096 in list_candidates () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #17 0x00007f6832b52f40 in filter_candidates_ext () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #18 0x00007f6832b8e80c in subtree_candidates () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #19 0x00007f6832b8fe7f in ldbm_back_search () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #20 0x00007f683f4bd1d6 in op_shared_search () from > /usr/lib64/dirsrv/libslapd.so.0 > #21 0x00007f683f4cd94e in search_internal_callback_pb () from > /usr/lib64/dirsrv/libslapd.so.0 > #22 0x00007f683f4cdbe8 in search_internal_pb () from > /usr/lib64/dirsrv/libslapd.so.0 > #23 0x00007f6832dd75dc in get_entry_by_principal () from > /usr/lib64/dirsrv/plugins/libipa_pwd_extop.so > #24 0x00007f6832dd8eb1 in ipapwd_extop () from /usr/lib64/dirsrv/plugins/ > libipa_pwd_extop.so > #25 0x00007f683f4c6ba2 in plugin_call_exop_plugins () from > /usr/lib64/dirsrv/libslapd.so.0 > #26 0x00007f683f9a2efe in do_extended () > #27 0x00007f683f99c41a in connection_threadmain () > #28 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so > #29 0x00007f683d04cdc5 in start_thread () from /lib64/libpthread.so.0 > #30 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 > Thread 16 (Thread 0x7f67fe7f4700 (LWP 4877)): > #0 0x00007f683d0506d5 in pthread_cond_wait@@GLIBC_2.3.2 () from > /lib64/libpthread.so.0 > #1 0x00007f683d6a6050 in PR_WaitCondVar () from /lib64/libnspr4.so > #2 0x00007f683f999fee in connection_wait_for_new_work () > #3 0x00007f683f99b719 in connection_threadmain () > #4 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so > #5 0x00007f683d04cdc5 in start_thread () from /lib64/libpthread.so.0 > #6 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 > Thread 15 (Thread 0x7f67fdff3700 (LWP 4878)): > #0 0x00007f683d0506d5 in pthread_cond_wait@@GLIBC_2.3.2 () from > /lib64/libpthread.so.0 > #1 0x00007f683d6a6050 in PR_WaitCondVar () from /lib64/libnspr4.so > #2 0x00007f683f999fee in connection_wait_for_new_work () > #3 0x00007f683f99b719 in connection_threadmain () > #4 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so > #5 0x00007f683d04cdc5 in start_thread () from /lib64/libpthread.so.0 > #6 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 > Thread 14 (Thread 0x7f67fd7f2700 (LWP 4879)): > #0 0x00007f683d0506d5 in pthread_cond_wait@@GLIBC_2.3.2 () from > /lib64/libpthread.so.0 > #1 0x00007f683d6a6050 in PR_WaitCondVar () from /lib64/libnspr4.so > #2 0x00007f683f999fee in connection_wait_for_new_work () > #3 0x00007f683f99b719 in connection_threadmain () > #4 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so > #5 0x00007f683d04cdc5 in start_thread () from /lib64/libpthread.so.0 > #6 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 > Thread 13 (Thread 0x7f67fcff1700 (LWP 4880)): > #0 0x00007f683d0506d5 in pthread_cond_wait@@GLIBC_2.3.2 () from > /lib64/libpthread.so.0 > #1 0x00007f683d6a6050 in PR_WaitCondVar () from /lib64/libnspr4.so > #2 0x00007f683f999fee in connection_wait_for_new_work () > #3 0x00007f683f99b719 in connection_threadmain () > #4 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so > #5 0x00007f683d04cdc5 in start_thread () from /lib64/libpthread.so.0 > #6 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 > Thread 12 (Thread 0x7f67fc7f0700 (LWP 4881)): > #0 0x00007f683d0506d5 in pthread_cond_wait@@GLIBC_2.3.2 () from > /lib64/libpthread.so.0 > #1 0x00007f683d6a6050 in PR_WaitCondVar () from /lib64/libnspr4.so > #2 0x00007f683f999fee in connection_wait_for_new_work () > #3 0x00007f683f99b719 in connection_threadmain () > #4 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so > #5 0x00007f683d04cdc5 in start_thread () from /lib64/libpthread.so.0 > #6 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 > Thread 11 (Thread 0x7f67fbfef700 (LWP 4882)): > #0 0x00007f683d0506d5 in pthread_cond_wait@@GLIBC_2.3.2 () from > /lib64/libpthread.so.0 > #1 0x00007f683d6a6050 in PR_WaitCondVar () from /lib64/libnspr4.so > #2 0x00007f683f999fee in connection_wait_for_new_work () > #3 0x00007f683f99b719 in connection_threadmain () > #4 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so > #5 0x00007f683d04cdc5 in start_thread () from /lib64/libpthread.so.0 > #6 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 > Thread 10 (Thread 0x7f67fb7ee700 (LWP 4883)): > #0 0x00007f683d0506d5 in pthread_cond_wait@@GLIBC_2.3.2 () from > /lib64/libpthread.so.0 > #1 0x00007f68375ac2f3 in __db_hybrid_mutex_suspend () from /lib64/ > libdb-5.3.so > #2 0x00007f68375ab640 in __db_tas_mutex_lock () from /lib64/libdb-5.3.so > #3 0x00007f6837655d3a in __lock_get_internal () from /lib64/libdb-5.3.so > #4 0x00007f6837656820 in __lock_get () from /lib64/libdb-5.3.so > #5 0x00007f6837682142 in __db_lget () from /lib64/libdb-5.3.so > #6 0x00007f68375c9605 in __bam_search () from /lib64/libdb-5.3.so > #7 0x00007f68375b4256 in __bamc_search () from /lib64/libdb-5.3.so > #8 0x00007f68375b5d0f in __bamc_get () from /lib64/libdb-5.3.so > #9 0x00007f683766eca6 in __dbc_iget () from /lib64/libdb-5.3.so > #10 0x00007f683767db02 in __dbc_get_pp () from /lib64/libdb-5.3.so > #11 0x00007f6832b59680 in idl_new_fetch () from /usr/lib64/dirsrv/plugins/ > libback-ldbm.so > #12 0x00007f6832b67b26 in index_read_ext_allids () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #13 0x00007f6832b52284 in keys2idl () from /usr/lib64/dirsrv/plugins/ > libback-ldbm.so > #14 0x00007f6832b529e3 in ava_candidates.isra.0 () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #15 0x00007f6832b52fd2 in filter_candidates_ext () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #16 0x00007f6832b54096 in list_candidates () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #17 0x00007f6832b52f40 in filter_candidates_ext () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #18 0x00007f6832b54096 in list_candidates () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #19 0x00007f6832b52f40 in filter_candidates_ext () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #20 0x00007f6832b54096 in list_candidates () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #21 0x00007f6832b52f40 in filter_candidates_ext () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #22 0x00007f6832b8e80c in subtree_candidates () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #23 0x00007f6832b8fe7f in ldbm_back_search () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #24 0x00007f683f4bd1d6 in op_shared_search () from > /usr/lib64/dirsrv/libslapd.so.0 > #25 0x00007f683f9ae33e in do_search () > #26 0x00007f683f99c3c3 in connection_threadmain () > #27 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so > #28 0x00007f683d04cdc5 in start_thread () from /lib64/libpthread.so.0 > #29 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 > Thread 9 (Thread 0x7f67fafed700 (LWP 4884)): > #0 0x00007f683d0506d5 in pthread_cond_wait@@GLIBC_2.3.2 () from > /lib64/libpthread.so.0 > #1 0x00007f683d6a6050 in PR_WaitCondVar () from /lib64/libnspr4.so > #2 0x00007f683f999fee in connection_wait_for_new_work () > #3 0x00007f683f99b719 in connection_threadmain () > #4 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so > #5 0x00007f683d04cdc5 in start_thread () from /lib64/libpthread.so.0 > #6 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 > Thread 8 (Thread 0x7f67fa7ec700 (LWP 4885)): > #0 0x00007f683d0506d5 in pthread_cond_wait@@GLIBC_2.3.2 () from > /lib64/libpthread.so.0 > #1 0x00007f68375ac2f3 in __db_hybrid_mutex_suspend () from /lib64/ > libdb-5.3.so > #2 0x00007f68375ab640 in __db_tas_mutex_lock () from /lib64/libdb-5.3.so > #3 0x00007f6837655d3a in __lock_get_internal () from /lib64/libdb-5.3.so > #4 0x00007f6837656820 in __lock_get () from /lib64/libdb-5.3.so > #5 0x00007f6837682142 in __db_lget () from /lib64/libdb-5.3.so > #6 0x00007f68375c9605 in __bam_search () from /lib64/libdb-5.3.so > #7 0x00007f68375b4256 in __bamc_search () from /lib64/libdb-5.3.so > #8 0x00007f68375b5d0f in __bamc_get () from /lib64/libdb-5.3.so > #9 0x00007f683766eca6 in __dbc_iget () from /lib64/libdb-5.3.so > #10 0x00007f683767db02 in __dbc_get_pp () from /lib64/libdb-5.3.so > #11 0x00007f6832b59680 in idl_new_fetch () from /usr/lib64/dirsrv/plugins/ > libback-ldbm.so > #12 0x00007f6832b67b26 in index_read_ext_allids () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #13 0x00007f6832b52284 in keys2idl () from /usr/lib64/dirsrv/plugins/ > libback-ldbm.so > #14 0x00007f6832b529e3 in ava_candidates.isra.0 () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #15 0x00007f6832b52fd2 in filter_candidates_ext () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #16 0x00007f6832b54096 in list_candidates () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #17 0x00007f6832b52f40 in filter_candidates_ext () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #18 0x00007f6832b54096 in list_candidates () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #19 0x00007f6832b52f40 in filter_candidates_ext () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #20 0x00007f6832b54096 in list_candidates () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #21 0x00007f6832b52f40 in filter_candidates_ext () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #22 0x00007f6832b8e80c in subtree_candidates () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #23 0x00007f6832b8fe7f in ldbm_back_search () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #24 0x00007f683f4bd1d6 in op_shared_search () from > /usr/lib64/dirsrv/libslapd.so.0 > #25 0x00007f683f9ae33e in do_search () > #26 0x00007f683f99c3c3 in connection_threadmain () > #27 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so > #28 0x00007f683d04cdc5 in start_thread () from /lib64/libpthread.so.0 > #29 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 > Thread 7 (Thread 0x7f67f9feb700 (LWP 4886)): > #0 0x00007f683d0506d5 in pthread_cond_wait@@GLIBC_2.3.2 () from > /lib64/libpthread.so.0 > #1 0x00007f683d6a6050 in PR_WaitCondVar () from /lib64/libnspr4.so > #2 0x00007f683f999fee in connection_wait_for_new_work () > #3 0x00007f683f99b719 in connection_threadmain () > #4 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so > #5 0x00007f683d04cdc5 in start_thread () from /lib64/libpthread.so.0 > #6 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 > Thread 6 (Thread 0x7f67f97ea700 (LWP 4887)): > #0 0x00007f683d0506d5 in pthread_cond_wait@@GLIBC_2.3.2 () from > /lib64/libpthread.so.0 > #1 0x00007f683d6a6050 in PR_WaitCondVar () from /lib64/libnspr4.so > #2 0x00007f683f999fee in connection_wait_for_new_work () > #3 0x00007f683f99b719 in connection_threadmain () > #4 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so > #5 0x00007f683d04cdc5 in start_thread () from /lib64/libpthread.so.0 > #6 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 > Thread 5 (Thread 0x7f67f8fe9700 (LWP 4888)): > #0 0x00007f683d0506d5 in pthread_cond_wait@@GLIBC_2.3.2 () from > /lib64/libpthread.so.0 > #1 0x00007f683d6a6050 in PR_WaitCondVar () from /lib64/libnspr4.so > #2 0x00007f683f999fee in connection_wait_for_new_work () > #3 0x00007f683f99b719 in connection_threadmain () > #4 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so > #5 0x00007f683d04cdc5 in start_thread () from /lib64/libpthread.so.0 > #6 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 > Thread 4 (Thread 0x7f67f87e8700 (LWP 4889)): > #0 0x00007f683cd719b3 in select () from /lib64/libc.so.6 > #1 0x00007f683f4fc0e9 in DS_Sleep () from /usr/lib64/dirsrv/libslapd.so.0 > #2 0x00007f683f99d325 in time_thread () > #3 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so > #4 0x00007f683d04cdc5 in start_thread () from /lib64/libpthread.so.0 > #5 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 > Thread 3 (Thread 0x7f67f7de6700 (LWP 4892)): > #0 0x00007f683d0506d5 in pthread_cond_wait@@GLIBC_2.3.2 () from > /lib64/libpthread.so.0 > #1 0x00007f683d6a6050 in PR_WaitCondVar () from /lib64/libnspr4.so > #2 0x00007f683f9a8f25 in ps_send_results () > #3 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so > #4 0x00007f683d04cdc5 in start_thread () from /lib64/libpthread.so.0 > #5 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 > Thread 2 (Thread 0x7f67f75e5700 (LWP 5084)): > #0 0x00007f683d0506d5 in pthread_cond_wait@@GLIBC_2.3.2 () from > /lib64/libpthread.so.0 > #1 0x00007f683d6a6050 in PR_WaitCondVar () from /lib64/libnspr4.so > #2 0x00007f683f9a8f25 in ps_send_results () > #3 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so > #4 0x00007f683d04cdc5 in start_thread () from /lib64/libpthread.so.0 > #5 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 > Thread 1 (Thread 0x7f683f964840 (LWP 4845)): > #0 0x00007f683d0506d5 in pthread_cond_wait@@GLIBC_2.3.2 () from > /lib64/libpthread.so.0 > #1 0x00007f683d6a62c3 in PR_EnterMonitor () from /lib64/libnspr4.so > #2 0x00007f683f99d5ed in setup_pr_read_pds () > #3 0x00007f683f9a0fd9 in slapd_daemon () > #4 0x00007f683f992c6c in main () > > > > the top output seems ok, there was just a sudden spike to around 340% cpu > then it comes down > > top - 08:53:12 up 6 days, 23:20, 1 user, load average: 0.00, 0.01, 0.12 > Threads: 44 total, 0 running, 44 sleeping, 0 stopped, 0 zombie > %Cpu(s): 0.0 us, 0.0 sy, 0.0 ni,100.0 id, 0.0 wa, 0.0 hi, 0.0 si, > 0.0 st > KiB Mem : 65533176 total, 52165676 free, 6135088 used, 7232412 buff/cache > KiB Swap: 3999996 total, 3999996 free, 0 used. 58452040 avail Mem > > PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND > 4845 dirsrv 20 0 5133196 3.150g 281272 S 0.0 5.0 31:34.51 > ns-slapd > 4847 dirsrv 20 0 5133196 3.150g 281272 S 0.0 5.0 0:13.53 > ns-slapd > 4848 dirsrv 20 0 5133196 3.150g 281272 S 0.0 5.0 0:13.16 > ns-slapd > 4849 dirsrv 20 0 5133196 3.150g 281272 S 0.0 5.0 0:45.27 > ns-slapd > 4850 dirsrv 20 0 5133196 3.150g 281272 S 0.0 5.0 0:01.60 > ns-slapd > 4851 dirsrv 20 0 5133196 3.150g 281272 S 0.0 5.0 0:00.00 > ns-slapd > 4852 dirsrv 20 0 5133196 3.150g 281272 S 0.0 5.0 0:00.00 > ns-slapd > 4853 dirsrv 20 0 5133196 3.150g 281272 S 0.0 5.0 0:00.00 > ns-slapd > 4854 dirsrv 20 0 5133196 3.150g 281272 S 0.0 5.0 0:00.00 > ns-slapd > 4855 dirsrv 20 0 5133196 3.150g 281272 S 0.0 5.0 0:03.02 > ns-slapd > 4856 dirsrv 20 0 5133196 3.150g 281272 S 0.0 5.0 0:00.26 > ns-slapd > 4859 dirsrv 20 0 5133196 3.150g 281272 S 0.0 5.0 3:18.67 > ns-slapd > 4860 dirsrv 20 0 5133196 3.150g 281272 S 0.0 5.0 3:19.82 > ns-slapd > 4861 dirsrv 20 0 5133196 3.150g 281272 S 0.0 5.0 3:20.23 > ns-slapd > 4862 dirsrv 20 0 5133196 3.150g 281272 S 0.0 5.0 3:22.65 > ns-slapd > 4863 dirsrv 20 0 5133196 3.150g 281272 S 0.0 5.0 3:16.64 > ns-slapd > 4864 dirsrv 20 0 5133196 3.150g 281272 S 0.0 5.0 3:17.39 > ns-slapd > 4865 dirsrv 20 0 5133196 3.150g 281272 S 0.0 5.0 3:17.80 > ns-slapd > > > Thanks > Rakesh > > On Tue, Sep 6, 2016 at 12:09 AM, Rakesh Rajasekharan < > rakesh.rajasekharan at gmail.com> wrote: > >> Hi Thierry/Ludwig, >> >> Unfortunately I had by then restarted the service and since then have >> been trying to hit the hang issue again but no luck this time >> >> It has been running fine for the last few hours now in-spite of trying >> all that I believe could cause the hang like making simultaneous client >> addition requests. >> >> I will continue testing this and next time I get it I will make sure I >> get the pstack and also get the dump of db lock info. >> >> Thanks, >> Rakesh >> >> >> >> On Mon, Sep 5, 2016 at 6:14 PM, Ludwig Krispenz >> wrote: >> >>> >>> On 09/05/2016 02:31 PM, thierry bordaz wrote: >>> >>> Hi Rakesh, >>> >>> Thanks you soooo much for you continuous help on this !! >>> >>> If the hang is still going on, would you dump the DB lock information >>> with: >>> >>> db_stat -CA -N -h /var/lib/dirsrv/slapd-/db >>> >/tmp/db_lock_info >>> >>> It is looking like several threads are waiting for one/several db locks. >>> With the lock info we will know which thread acquired it/them. >>> A guess is that it could be the checkpointing thread but this one is >>> also waiting (on a memory pool). >>> >>> I am copying Ludwig on this thread in case this pstack rings any bell. >>> >>> the frequent presence of __db_tas_mutex_lock() looks like the NUMA >>> problem of libdb, although the threads already have stopped spinning (maybe >>> this was an effect of gcore). >>> If the hang aoccurs again, could you check cpu consumption >>> top -H -p >>> and get a pstack >>> pstack >>> >>> >>> best regards >>> thierry >>> >>> On 09/05/2016 02:05 PM, Rakesh Rajasekharan wrote: >>> >>> Hi Thierry, >>> >>> I was able to now reporduce the issue.. >>> >>> the ns-slapd process just got hung,.. however, no core file was >>> generate.. >>> >>> I manually created a coredump using gcore and then took a strace of it >>> this is strace from the coredump generated >>> >>> GNU gdb (GDB) Red Hat Enterprise Linux 7.6.1-80.el7 >>> Copyright (C) 2013 Free Software Foundation, Inc. >>> License GPLv3+: GNU GPL version 3 or later < >>> http://gnu.org/licenses/gpl.html> >>> This is free software: you are free to change and redistribute it. >>> There is NO WARRANTY, to the extent permitted by law. Type "show >>> copying" >>> and "show warranty" for details. >>> This GDB was configured as "x86_64-redhat-linux-gnu". >>> For bug reporting instructions, please see: >>> ... >>> Reading symbols from /usr/sbin/ns-slapd...Reading symbols from >>> /usr/sbin/ns-slapd...(no debugging symbols found)...done. >>> (no debugging symbols found)...done. >>> [New LWP 105159] >>> [New LWP 105160] >>> [New LWP 105161] >>> [New LWP 105162] >>> [New LWP 105163] >>> [New LWP 105164] >>> [New LWP 105165] >>> [New LWP 105166] >>> [New LWP 105167] >>> [New LWP 105168] >>> [New LWP 105171] >>> [New LWP 105172] >>> [New LWP 105173] >>> [New LWP 105174] >>> [New LWP 105175] >>> [New LWP 105176] >>> [New LWP 105177] >>> [New LWP 105178] >>> [New LWP 105179] >>> [New LWP 105180] >>> [New LWP 105181] >>> [New LWP 105182] >>> [New LWP 105183] >>> [New LWP 105184] >>> [New LWP 105185] >>> [New LWP 105186] >>> [New LWP 105187] >>> [New LWP 105188] >>> [New LWP 105189] >>> [New LWP 105190] >>> [New LWP 105191] >>> [New LWP 105192] >>> [New LWP 105193] >>> [New LWP 105194] >>> [New LWP 105195] >>> [New LWP 105196] >>> [New LWP 105197] >>> [New LWP 105198] >>> [New LWP 105199] >>> [New LWP 105200] >>> [New LWP 105201] >>> [New LWP 105203] >>> [New LWP 105215] >>> [New LWP 105158] >>> [Thread debugging using libthread_db enabled] >>> Using host libthread_db library "/lib64/libthread_db.so.1". >>> Core was generated by `/usr/sbin/ns-slapd'. >>> #0 0x00007f1b48ae69b3 in select () from /lib64/libc.so.6 >>> >>> Thread 44 (Thread 0x7f1b4b6d9840 (LWP 105158)): >>> #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 () from >>> /lib64/libpthread.so.0 >>> No symbol table info available. >>> #1 0x00007f1b4941b2c3 in PR_EnterMonitor () from /lib64/libnspr4.so >>> No symbol table info available. >>> #2 0x00007f1b4b7125ed in setup_pr_read_pds () >>> No symbol table info available. >>> #3 0x00007f1b4b715fd9 in slapd_daemon () >>> No symbol table info available. >>> #4 0x00007f1b4b707c6c in main () >>> No symbol table info available. >>> >>> Thread 43 (Thread 0x7f1a5df0f700 (LWP 105215)): >>> #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 () from >>> /lib64/libpthread.so.0 >>> No symbol table info available. >>> #1 0x00007f1b4941b050 in PR_WaitCondVar () from /lib64/libnspr4.so >>> No symbol table info available. >>> #2 0x00007f1b4b71df25 in ps_send_results () >>> No symbol table info available. >>> #3 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so >>> No symbol table info available. >>> #4 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 >>> No symbol table info available. >>> #5 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 >>> No symbol table info available. >>> >>> Thread 42 (Thread 0x7f1b03fe7700 (LWP 105203)): >>> #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 () from >>> /lib64/libpthread.so.0 >>> No symbol table info available. >>> #1 0x00007f1b4941b050 in PR_WaitCondVar () from /lib64/libnspr4.so >>> No symbol table info available. >>> #2 0x00007f1b4b71df25 in ps_send_results () >>> No symbol table info available. >>> #3 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so >>> No symbol table info available. >>> #4 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 >>> No symbol table info available. >>> #5 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 >>> No symbol table info available. >>> >>> Thread 41 (Thread 0x7f1b047e8700 (LWP 105201)): >>> #0 0x00007f1b48ae69b3 in select () from /lib64/libc.so.6 >>> No symbol table info available. >>> #1 0x00007f1b4b2710e9 in DS_Sleep () from /usr/lib64/dirsrv/libslapd.so. >>> 0 >>> No symbol table info available. >>> #2 0x00007f1b4b712325 in time_thread () >>> No symbol table info available. >>> #3 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so >>> No symbol table info available. >>> #4 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 >>> No symbol table info available. >>> #5 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 >>> No symbol table info available. >>> >>> Thread 40 (Thread 0x7f1b04fe9700 (LWP 105200)): >>> #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 () from >>> /lib64/libpthread.so.0 >>> No symbol table info available. >>> #1 0x00007f1b433212f3 in __db_hybrid_mutex_suspend () from /lib64/ >>> libdb-5.3.so >>> No symbol table info available. >>> #2 0x00007f1b43320640 in __db_tas_mutex_lock () from /lib64/ >>> libdb-5.3.so >>> No symbol table info available. >>> #3 0x00007f1b4343ae1b in __memp_fget () from /lib64/libdb-5.3.so >>> No symbol table info available. >>> #4 0x00007f1b4333e211 in __bam_search () from /lib64/libdb-5.3.so >>> No symbol table info available. >>> #5 0x00007f1b43329256 in __bamc_search () from /lib64/libdb-5.3.so >>> No symbol table info available. >>> #6 0x00007f1b4332d6c4 in __bamc_put () from /lib64/libdb-5.3.so >>> No symbol table info available. >>> #7 0x00007f1b433e46d5 in __dbc_iput () from /lib64/libdb-5.3.so >>> No symbol table info available. >>> #8 0x00007f1b433dfa0e in __db_put () from /lib64/libdb-5.3.so >>> No symbol table info available. >>> #9 0x00007f1b433f4fa4 in __db_put_pp () from /lib64/libdb-5.3.so >>> No symbol table info available. >>> #10 0x00007f1b3e8cf4cf in idl_new_insert_key () from >>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>> No symbol table info available. >>> #11 0x00007f1b3e8de5b5 in addordel_values_sv.isra.3 () from >>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>> No symbol table info available. >>> #12 0x00007f1b3e8df25a in index_addordel_values_ext_sv () from >>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>> No symbol table info available. >>> #13 0x00007f1b3e8df694 in index_addordel_values_sv () from >>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>> No symbol table info available. >>> #14 0x00007f1b3e8df75a in index_addordel_entry () from >>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>> No symbol table info available. >>> #15 0x00007f1b3e8e25fa in ldbm_back_add () from >>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>> No symbol table info available. >>> #16 0x00007f1b4b1e4cca in op_shared_add () from >>> /usr/lib64/dirsrv/libslapd.so.0 >>> No symbol table info available. >>> #17 0x00007f1b4b1e6020 in do_add () from /usr/lib64/dirsrv/libslapd.so.0 >>> No symbol table info available. >>> #18 0x00007f1b4b711303 in connection_threadmain () >>> No symbol table info available. >>> #19 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so >>> No symbol table info available. >>> #20 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 >>> No symbol table info available. >>> #21 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 >>> No symbol table info available. >>> >>> Thread 39 (Thread 0x7f1b057ea700 (LWP 105199)): >>> #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 () from >>> /lib64/libpthread.so.0 >>> No symbol table info available. >>> #1 0x00007f1b4941b050 in PR_WaitCondVar () from /lib64/libnspr4.so >>> No symbol table info available. >>> #2 0x00007f1b4b70efee in connection_wait_for_new_work () >>> No symbol table info available. >>> #3 0x00007f1b4b710719 in connection_threadmain () >>> No symbol table info available. >>> #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so >>> No symbol table info available. >>> #5 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 >>> No symbol table info available. >>> #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 >>> No symbol table info available. >>> >>> Thread 38 (Thread 0x7f1b05feb700 (LWP 105198)): >>> #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 () from >>> /lib64/libpthread.so.0 >>> No symbol table info available. >>> #1 0x00007f1b4941b050 in PR_WaitCondVar () from /lib64/libnspr4.so >>> No symbol table info available. >>> #2 0x00007f1b4b70efee in connection_wait_for_new_work () >>> No symbol table info available. >>> #3 0x00007f1b4b710719 in connection_threadmain () >>> No symbol table info available. >>> #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so >>> No symbol table info available. >>> #5 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 >>> No symbol table info available. >>> #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 >>> No symbol table info available. >>> >>> Thread 37 (Thread 0x7f1b067ec700 (LWP 105197)): >>> #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 () from >>> /lib64/libpthread.so.0 >>> No symbol table info available. >>> #1 0x00007f1b4941b050 in PR_WaitCondVar () from /lib64/libnspr4.so >>> No symbol table info available. >>> #2 0x00007f1b4b70efee in connection_wait_for_new_work () >>> No symbol table info available. >>> #3 0x00007f1b4b710719 in connection_threadmain () >>> No symbol table info available. >>> #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so >>> No symbol table info available. >>> #5 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 >>> No symbol table info available. >>> #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 >>> No symbol table info available. >>> >>> Thread 36 (Thread 0x7f1b06fed700 (LWP 105196)): >>> #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 () from >>> /lib64/libpthread.so.0 >>> No symbol table info available. >>> #1 0x00007f1b4941b050 in PR_WaitCondVar () from /lib64/libnspr4.so >>> No symbol table info available. >>> #2 0x00007f1b4b70efee in connection_wait_for_new_work () >>> No symbol table info available. >>> #3 0x00007f1b4b710719 in connection_threadmain () >>> No symbol table info available. >>> #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so >>> No symbol table info available. >>> #5 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 >>> No symbol table info available. >>> #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 >>> No symbol table info available. >>> >>> Thread 35 (Thread 0x7f1b077ee700 (LWP 105195)): >>> #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 () from >>> /lib64/libpthread.so.0 >>> No symbol table info available. >>> #1 0x00007f1b4941b2c3 in PR_EnterMonitor () from /lib64/libnspr4.so >>> No symbol table info available. >>> #2 0x00007f1b3e8c23e6 in dblayer_txn_begin () from >>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>> No symbol table info available. >>> #3 0x00007f1b3e8fe007 in ldbm_back_modify () from >>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>> No symbol table info available. >>> #4 0x00007f1b4b22b541 in op_shared_modify () from >>> /usr/lib64/dirsrv/libslapd.so.0 >>> No symbol table info available. >>> #5 0x00007f1b4b22c004 in modify_internal_pb () from >>> /usr/lib64/dirsrv/libslapd.so.0 >>> No symbol table info available. >>> #6 0x00007f1b3eb48b60 in ipapwd_apply_mods () from >>> /usr/lib64/dirsrv/plugins/libipa_pwd_extop.so >>> No symbol table info available. >>> #7 0x00007f1b3eb4c8e0 in store_new_keys () from >>> /usr/lib64/dirsrv/plugins/libipa_pwd_extop.so >>> No symbol table info available. >>> #8 0x00007f1b3eb4ead0 in ipapwd_extop () from >>> /usr/lib64/dirsrv/plugins/libipa_pwd_extop.so >>> No symbol table info available. >>> #9 0x00007f1b4b23bba2 in plugin_call_exop_plugins () from >>> /usr/lib64/dirsrv/libslapd.so.0 >>> No symbol table info available. >>> #10 0x00007f1b4b717efe in do_extended () >>> No symbol table info available. >>> #11 0x00007f1b4b71141a in connection_threadmain () >>> No symbol table info available. >>> #12 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so >>> No symbol table info available. >>> #13 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 >>> No symbol table info available. >>> #14 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 >>> No symbol table info available. >>> >>> Thread 34 (Thread 0x7f1b07fef700 (LWP 105194)): >>> #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 () from >>> /lib64/libpthread.so.0 >>> No symbol table info available. >>> #1 0x00007f1b4941b050 in PR_WaitCondVar () from /lib64/libnspr4.so >>> No symbol table info available. >>> #2 0x00007f1b4b70efee in connection_wait_for_new_work () >>> No symbol table info available. >>> #3 0x00007f1b4b710719 in connection_threadmain () >>> No symbol table info available. >>> #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so >>> No symbol table info available. >>> #5 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 >>> No symbol table info available. >>> #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 >>> No symbol table info available. >>> >>> Thread 33 (Thread 0x7f1b087f0700 (LWP 105193)): >>> #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 () from >>> /lib64/libpthread.so.0 >>> No symbol table info available. >>> #1 0x00007f1b4941b050 in PR_WaitCondVar () from /lib64/libnspr4.so >>> No symbol table info available. >>> #2 0x00007f1b4b70efee in connection_wait_for_new_work () >>> No symbol table info available. >>> #3 0x00007f1b4b710719 in connection_threadmain () >>> No symbol table info available. >>> #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so >>> No symbol table info available. >>> #5 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 >>> No symbol table info available. >>> #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 >>> No symbol table info available. >>> >>> Thread 32 (Thread 0x7f1b08ff1700 (LWP 105192)): >>> #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 () from >>> /lib64/libpthread.so.0 >>> No symbol table info available. >>> #1 0x00007f1b4941b050 in PR_WaitCondVar () from /lib64/libnspr4.so >>> No symbol table info available. >>> #2 0x00007f1b4b70efee in connection_wait_for_new_work () >>> No symbol table info available. >>> #3 0x00007f1b4b710719 in connection_threadmain () >>> No symbol table info available. >>> #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so >>> No symbol table info available. >>> #5 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 >>> No symbol table info available. >>> #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 >>> No symbol table info available. >>> >>> Thread 31 (Thread 0x7f1b097f2700 (LWP 105191)): >>> #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 () from >>> /lib64/libpthread.so.0 >>> No symbol table info available. >>> #1 0x00007f1b4941b050 in PR_WaitCondVar () from /lib64/libnspr4.so >>> No symbol table info available. >>> #2 0x00007f1b4b70efee in connection_wait_for_new_work () >>> No symbol table info available. >>> #3 0x00007f1b4b710719 in connection_threadmain () >>> No symbol table info available. >>> #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so >>> No symbol table info available. >>> #5 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 >>> No symbol table info available. >>> #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 >>> No symbol table info available. >>> >>> Thread 30 (Thread 0x7f1b09ff3700 (LWP 105190)): >>> #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 () from >>> /lib64/libpthread.so.0 >>> No symbol table info available. >>> #1 0x00007f1b433212f3 in __db_hybrid_mutex_suspend () from /lib64/ >>> libdb-5.3.so >>> No symbol table info available. >>> #2 0x00007f1b43320640 in __db_tas_mutex_lock () from /lib64/ >>> libdb-5.3.so >>> No symbol table info available. >>> #3 0x00007f1b433cad3a in __lock_get_internal () from /lib64/ >>> libdb-5.3.so >>> No symbol table info available. >>> #4 0x00007f1b433cb820 in __lock_get () from /lib64/libdb-5.3.so >>> No symbol table info available. >>> #5 0x00007f1b433f7142 in __db_lget () from /lib64/libdb-5.3.so >>> No symbol table info available. >>> #6 0x00007f1b4333e605 in __bam_search () from /lib64/libdb-5.3.so >>> No symbol table info available. >>> #7 0x00007f1b43329256 in __bamc_search () from /lib64/libdb-5.3.so >>> No symbol table info available. >>> #8 0x00007f1b4332ad0f in __bamc_get () from /lib64/libdb-5.3.so >>> No symbol table info available. >>> #9 0x00007f1b433e3ca6 in __dbc_iget () from /lib64/libdb-5.3.so >>> No symbol table info available. >>> #10 0x00007f1b433f2b02 in __dbc_get_pp () from /lib64/libdb-5.3.so >>> No symbol table info available. >>> #11 0x00007f1b3e8ce680 in idl_new_fetch () from >>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>> No symbol table info available. >>> #12 0x00007f1b3e8dcb26 in index_read_ext_allids () from >>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>> No symbol table info available. >>> #13 0x00007f1b3e8c7284 in keys2idl () from /usr/lib64/dirsrv/plugins/libb >>> ack-ldbm.so >>> No symbol table info available. >>> #14 0x00007f1b3e8c79e3 in ava_candidates.isra.0 () from >>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>> No symbol table info available. >>> #15 0x00007f1b3e8c7fd2 in filter_candidates_ext () from >>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>> No symbol table info available. >>> #16 0x00007f1b3e8c9096 in list_candidates () from >>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>> No symbol table info available. >>> #17 0x00007f1b3e8c7f40 in filter_candidates_ext () from >>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>> No symbol table info available. >>> #18 0x00007f1b3e8c9096 in list_candidates () from >>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>> No symbol table info available. >>> #19 0x00007f1b3e8c7f40 in filter_candidates_ext () from >>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>> No symbol table info available. >>> #20 0x00007f1b3e8c9096 in list_candidates () from >>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>> No symbol table info available. >>> #21 0x00007f1b3e8c7f40 in filter_candidates_ext () from >>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>> No symbol table info available. >>> #22 0x00007f1b3e90380c in subtree_candidates () from >>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>> No symbol table info available. >>> #23 0x00007f1b3e904e7f in ldbm_back_search () from >>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>> No symbol table info available. >>> #24 0x00007f1b4b2321d6 in op_shared_search () from >>> /usr/lib64/dirsrv/libslapd.so.0 >>> No symbol table info available. >>> #25 0x00007f1b4b72333e in do_search () >>> No symbol table info available. >>> #26 0x00007f1b4b7113c3 in connection_threadmain () >>> No symbol table info available. >>> #27 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so >>> No symbol table info available. >>> #28 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 >>> No symbol table info available. >>> #29 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 >>> No symbol table info available. >>> >>> Thread 29 (Thread 0x7f1b0a7f4700 (LWP 105189)): >>> #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 () from >>> /lib64/libpthread.so.0 >>> No symbol table info available. >>> #1 0x00007f1b4941b050 in PR_WaitCondVar () from /lib64/libnspr4.so >>> No symbol table info available. >>> #2 0x00007f1b4b70efee in connection_wait_for_new_work () >>> No symbol table info available. >>> #3 0x00007f1b4b710719 in connection_threadmain () >>> No symbol table info available. >>> #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so >>> No symbol table info available. >>> #5 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 >>> No symbol table info available. >>> #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 >>> No symbol table info available. >>> >>> Thread 28 (Thread 0x7f1b0aff5700 (LWP 105188)): >>> #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 () from >>> /lib64/libpthread.so.0 >>> No symbol table info available. >>> #1 0x00007f1b4941b050 in PR_WaitCondVar () from /lib64/libnspr4.so >>> No symbol table info available. >>> #2 0x00007f1b4b70efee in connection_wait_for_new_work () >>> No symbol table info available. >>> #3 0x00007f1b4b710719 in connection_threadmain () >>> No symbol table info available. >>> #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so >>> No symbol table info available. >>> #5 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 >>> No symbol table info available. >>> #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 >>> No symbol table info available. >>> >>> Thread 27 (Thread 0x7f1b0b7f6700 (LWP 105187)): >>> #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 () from >>> /lib64/libpthread.so.0 >>> No symbol table info available. >>> #1 0x00007f1b4941b050 in PR_WaitCondVar () from /lib64/libnspr4.so >>> No symbol table info available. >>> #2 0x00007f1b4b70efee in connection_wait_for_new_work () >>> No symbol table info available. >>> #3 0x00007f1b4b710719 in connection_threadmain () >>> No symbol table info available. >>> #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so >>> No symbol table info available. >>> #5 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 >>> No symbol table info available. >>> #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 >>> No symbol table info available. >>> >>> Thread 26 (Thread 0x7f1b0bff7700 (LWP 105186)): >>> #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 () from >>> /lib64/libpthread.so.0 >>> No symbol table info available. >>> #1 0x00007f1b433212f3 in __db_hybrid_mutex_suspend () from /lib64/ >>> libdb-5.3.so >>> No symbol table info available. >>> #2 0x00007f1b43320640 in __db_tas_mutex_lock () from /lib64/ >>> libdb-5.3.so >>> No symbol table info available. >>> #3 0x00007f1b433cad3a in __lock_get_internal () from /lib64/ >>> libdb-5.3.so >>> No symbol table info available. >>> #4 0x00007f1b433cb820 in __lock_get () from /lib64/libdb-5.3.so >>> No symbol table info available. >>> #5 0x00007f1b433f7142 in __db_lget () from /lib64/libdb-5.3.so >>> No symbol table info available. >>> #6 0x00007f1b4333e605 in __bam_search () from /lib64/libdb-5.3.so >>> No symbol table info available. >>> #7 0x00007f1b43329256 in __bamc_search () from /lib64/libdb-5.3.so >>> No symbol table info available. >>> #8 0x00007f1b4332ad0f in __bamc_get () from /lib64/libdb-5.3.so >>> No symbol table info available. >>> #9 0x00007f1b433e3ca6 in __dbc_iget () from /lib64/libdb-5.3.so >>> No symbol table info available. >>> #10 0x00007f1b433f2b02 in __dbc_get_pp () from /lib64/libdb-5.3.so >>> No symbol table info available. >>> #11 0x00007f1b3e8ce680 in idl_new_fetch () from >>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>> No symbol table info available. >>> #12 0x00007f1b3e8dcb26 in index_read_ext_allids () from >>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>> No symbol table info available. >>> #13 0x00007f1b3e8c7284 in keys2idl () from /usr/lib64/dirsrv/plugins/libb >>> ack-ldbm.so >>> No symbol table info available. >>> #14 0x00007f1b3e8c79e3 in ava_candidates.isra.0 () from >>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>> No symbol table info available. >>> #15 0x00007f1b3e8c7fd2 in filter_candidates_ext () from >>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>> No symbol table info available. >>> #16 0x00007f1b3e8c9096 in list_candidates () from >>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>> No symbol table info available. >>> #17 0x00007f1b3e8c7f40 in filter_candidates_ext () from >>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>> No symbol table info available. >>> #18 0x00007f1b3e90380c in subtree_candidates () from >>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>> No symbol table info available. >>> #19 0x00007f1b3e904e7f in ldbm_back_search () from >>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>> No symbol table info available. >>> #20 0x00007f1b4b2321d6 in op_shared_search () from >>> /usr/lib64/dirsrv/libslapd.so.0 >>> No symbol table info available. >>> #21 0x00007f1b4b24294e in search_internal_callback_pb () from >>> /usr/lib64/dirsrv/libslapd.so.0 >>> No symbol table info available. >>> #22 0x00007f1b4b242be8 in search_internal_pb () from >>> /usr/lib64/dirsrv/libslapd.so.0 >>> No symbol table info available. >>> #23 0x00007f1b4b72149b in ids_sasl_canon_user () >>> No symbol table info available. >>> #24 0x00007f1b491e193b in _sasl_canon_user () from /lib64/libsasl2.so.3 >>> No symbol table info available. >>> #25 0x00007f1b491e1c4c in _sasl_canon_user_lookup () from >>> /lib64/libsasl2.so.3 >>> No symbol table info available. >>> #26 0x00007f1b430e9fb9 in gssapi_server_mech_step () from >>> /usr/lib64/sasl2/libgssapiv2.so >>> No symbol table info available. >>> #27 0x00007f1b491edb9b in sasl_server_step () from /lib64/libsasl2.so.3 >>> No symbol table info available. >>> #28 0x00007f1b4b72283b in ids_sasl_check_bind () >>> No symbol table info available. >>> #29 0x00007f1b4b70a61a in do_bind () >>> No symbol table info available. >>> #30 0x00007f1b4b7113fd in connection_threadmain () >>> No symbol table info available. >>> #31 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so >>> No symbol table info available. >>> #32 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 >>> No symbol table info available. >>> #33 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 >>> No symbol table info available. >>> >>> Thread 25 (Thread 0x7f1b0c7f8700 (LWP 105185)): >>> #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 () from >>> /lib64/libpthread.so.0 >>> No symbol table info available. >>> #1 0x00007f1b4941b050 in PR_WaitCondVar () from /lib64/libnspr4.so >>> No symbol table info available. >>> #2 0x00007f1b4b70efee in connection_wait_for_new_work () >>> No symbol table info available. >>> #3 0x00007f1b4b710719 in connection_threadmain () >>> No symbol table info available. >>> #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so >>> No symbol table info available. >>> #5 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 >>> No symbol table info available. >>> #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 >>> No symbol table info available. >>> >>> Thread 24 (Thread 0x7f1b0cff9700 (LWP 105184)): >>> #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 () from >>> /lib64/libpthread.so.0 >>> No symbol table info available. >>> #1 0x00007f1b4941b050 in PR_WaitCondVar () from /lib64/libnspr4.so >>> No symbol table info available. >>> #2 0x00007f1b4b70efee in connection_wait_for_new_work () >>> No symbol table info available. >>> #3 0x00007f1b4b710719 in connection_threadmain () >>> No symbol table info available. >>> #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so >>> No symbol table info available. >>> #5 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 >>> No symbol table info available. >>> #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 >>> No symbol table info available. >>> >>> Thread 23 (Thread 0x7f1b0d7fa700 (LWP 105183)): >>> #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 () from >>> /lib64/libpthread.so.0 >>> No symbol table info available. >>> #1 0x00007f1b4941b050 in PR_WaitCondVar () from /lib64/libnspr4.so >>> No symbol table info available. >>> #2 0x00007f1b4b70efee in connection_wait_for_new_work () >>> No symbol table info available. >>> #3 0x00007f1b4b710719 in connection_threadmain () >>> No symbol table info available. >>> #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so >>> No symbol table info available. >>> #5 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 >>> No symbol table info available. >>> #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 >>> No symbol table info available. >>> >>> Thread 22 (Thread 0x7f1b0dffb700 (LWP 105182)): >>> #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 () from >>> /lib64/libpthread.so.0 >>> No symbol table info available. >>> #1 0x00007f1b4941b050 in PR_WaitCondVar () from /lib64/libnspr4.so >>> No symbol table info available. >>> #2 0x00007f1b4b70efee in connection_wait_for_new_work () >>> No symbol table info available. >>> #3 0x00007f1b4b710719 in connection_threadmain () >>> No symbol table info available. >>> #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so >>> No symbol table info available. >>> #5 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 >>> No symbol table info available. >>> #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 >>> No symbol table info available. >>> >>> Thread 21 (Thread 0x7f1b0e7fc700 (LWP 105181)): >>> #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 () from >>> /lib64/libpthread.so.0 >>> No symbol table info available. >>> #1 0x00007f1b4941b050 in PR_WaitCondVar () from /lib64/libnspr4.so >>> No symbol table info available. >>> #2 0x00007f1b4b70efee in connection_wait_for_new_work () >>> No symbol table info available. >>> #3 0x00007f1b4b710719 in connection_threadmain () >>> No symbol table info available. >>> #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so >>> No symbol table info available. >>> #5 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 >>> No symbol table info available. >>> #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 >>> No symbol table info available. >>> >>> Thread 20 (Thread 0x7f1b0effd700 (LWP 105180)): >>> #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 () from >>> /lib64/libpthread.so.0 >>> No symbol table info available. >>> #1 0x00007f1b433212f3 in __db_hybrid_mutex_suspend () from /lib64/ >>> libdb-5.3.so >>> No symbol table info available. >>> #2 0x00007f1b43320b37 in __db_tas_mutex_readlock () from /lib64/ >>> libdb-5.3.so >>> No symbol table info available. >>> #3 0x00007f1b4343ad35 in __memp_fget () from /lib64/libdb-5.3.so >>> No symbol table info available. >>> #4 0x00007f1b433e4333 in __dbc_iget () from /lib64/libdb-5.3.so >>> No symbol table info available. >>> #5 0x00007f1b433f2b02 in __dbc_get_pp () from /lib64/libdb-5.3.so >>> No symbol table info available. >>> #6 0x00007f1b3e8ce822 in idl_new_fetch () from >>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>> No symbol table info available. >>> #7 0x00007f1b3e8dcb26 in index_read_ext_allids () from >>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>> No symbol table info available. >>> #8 0x00007f1b3e8c7284 in keys2idl () from /usr/lib64/dirsrv/plugins/libb >>> ack-ldbm.so >>> No symbol table info available. >>> #9 0x00007f1b3e8c79e3 in ava_candidates.isra.0 () from >>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>> No symbol table info available. >>> #10 0x00007f1b3e8c7fd2 in filter_candidates_ext () from >>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>> No symbol table info available. >>> #11 0x00007f1b3e8c9096 in list_candidates () from >>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>> No symbol table info available. >>> #12 0x00007f1b3e8c7f40 in filter_candidates_ext () from >>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>> No symbol table info available. >>> #13 0x00007f1b3e8c9096 in list_candidates () from >>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>> No symbol table info available. >>> #14 0x00007f1b3e8c7f40 in filter_candidates_ext () from >>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>> No symbol table info available. >>> #15 0x00007f1b3e8c9096 in list_candidates () from >>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>> No symbol table info available. >>> #16 0x00007f1b3e8c7f40 in filter_candidates_ext () from >>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>> No symbol table info available. >>> #17 0x00007f1b3e90380c in subtree_candidates () from >>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>> No symbol table info available. >>> #18 0x00007f1b3e904e7f in ldbm_back_search () from >>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>> No symbol table info available. >>> #19 0x00007f1b4b2321d6 in op_shared_search () from >>> /usr/lib64/dirsrv/libslapd.so.0 >>> No symbol table info available. >>> #20 0x00007f1b4b72333e in do_search () >>> No symbol table info available. >>> #21 0x00007f1b4b7113c3 in connection_threadmain () >>> No symbol table info available. >>> #22 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so >>> No symbol table info available. >>> #23 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 >>> No symbol table info available. >>> #24 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 >>> No symbol table info available. >>> >>> Thread 19 (Thread 0x7f1b0f7fe700 (LWP 105179)): >>> #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 () from >>> /lib64/libpthread.so.0 >>> No symbol table info available. >>> #1 0x00007f1b433212f3 in __db_hybrid_mutex_suspend () from /lib64/ >>> libdb-5.3.so >>> No symbol table info available. >>> #2 0x00007f1b43320640 in __db_tas_mutex_lock () from /lib64/ >>> libdb-5.3.so >>> No symbol table info available. >>> #3 0x00007f1b433cad3a in __lock_get_internal () from /lib64/ >>> libdb-5.3.so >>> No symbol table info available. >>> #4 0x00007f1b433cb820 in __lock_get () from /lib64/libdb-5.3.so >>> No symbol table info available. >>> #5 0x00007f1b433f7142 in __db_lget () from /lib64/libdb-5.3.so >>> No symbol table info available. >>> #6 0x00007f1b4333e605 in __bam_search () from /lib64/libdb-5.3.so >>> No symbol table info available. >>> #7 0x00007f1b43329256 in __bamc_search () from /lib64/libdb-5.3.so >>> No symbol table info available. >>> #8 0x00007f1b4332ad0f in __bamc_get () from /lib64/libdb-5.3.so >>> No symbol table info available. >>> #9 0x00007f1b433e3ca6 in __dbc_iget () from /lib64/libdb-5.3.so >>> No symbol table info available. >>> #10 0x00007f1b433f2b02 in __dbc_get_pp () from /lib64/libdb-5.3.so >>> No symbol table info available. >>> #11 0x00007f1b3e8ce680 in idl_new_fetch () from >>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>> No symbol table info available. >>> #12 0x00007f1b3e8dcb26 in index_read_ext_allids () from >>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>> No symbol table info available. >>> #13 0x00007f1b3e8c7284 in keys2idl () from /usr/lib64/dirsrv/plugins/libb >>> ack-ldbm.so >>> No symbol table info available. >>> #14 0x00007f1b3e8c79e3 in ava_candidates.isra.0 () from >>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>> No symbol table info available. >>> #15 0x00007f1b3e8c7fd2 in filter_candidates_ext () from >>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>> No symbol table info available. >>> #16 0x00007f1b3e8c9096 in list_candidates () from >>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>> No symbol table info available. >>> #17 0x00007f1b3e8c7f40 in filter_candidates_ext () from >>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>> No symbol table info available. >>> #18 0x00007f1b3e8c9096 in list_candidates () from >>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>> No symbol table info available. >>> #19 0x00007f1b3e8c7f40 in filter_candidates_ext () from >>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>> No symbol table info available. >>> #20 0x00007f1b3e8c9096 in list_candidates () from >>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>> No symbol table info available. >>> #21 0x00007f1b3e8c7f40 in filter_candidates_ext () from >>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>> No symbol table info available. >>> #22 0x00007f1b3e90380c in subtree_candidates () from >>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>> No symbol table info available. >>> #23 0x00007f1b3e904e7f in ldbm_back_search () from >>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>> No symbol table info available. >>> #24 0x00007f1b4b2321d6 in op_shared_search () from >>> /usr/lib64/dirsrv/libslapd.so.0 >>> No symbol table info available. >>> #25 0x00007f1b4b72333e in do_search () >>> No symbol table info available. >>> #26 0x00007f1b4b7113c3 in connection_threadmain () >>> No symbol table info available. >>> #27 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so >>> No symbol table info available. >>> #28 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 >>> No symbol table info available. >>> #29 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 >>> No symbol table info available. >>> >>> Thread 18 (Thread 0x7f1b0ffff700 (LWP 105178)): >>> #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 () from >>> /lib64/libpthread.so.0 >>> No symbol table info available. >>> #1 0x00007f1b4941b050 in PR_WaitCondVar () from /lib64/libnspr4.so >>> No symbol table info available. >>> #2 0x00007f1b4b70efee in connection_wait_for_new_work () >>> No symbol table info available. >>> #3 0x00007f1b4b710719 in connection_threadmain () >>> No symbol table info available. >>> #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so >>> No symbol table info available. >>> #5 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 >>> No symbol table info available. >>> #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 >>> No symbol table info available. >>> >>> Thread 17 (Thread 0x7f1b14d28700 (LWP 105177)): >>> #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 () from >>> /lib64/libpthread.so.0 >>> No symbol table info available. >>> #1 0x00007f1b4941b050 in PR_WaitCondVar () from /lib64/libnspr4.so >>> No symbol table info available. >>> #2 0x00007f1b4b70efee in connection_wait_for_new_work () >>> No symbol table info available. >>> #3 0x00007f1b4b710719 in connection_threadmain () >>> No symbol table info available. >>> #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so >>> No symbol table info available. >>> #5 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 >>> No symbol table info available. >>> #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 >>> No symbol table info available. >>> >>> Thread 16 (Thread 0x7f1b15529700 (LWP 105176)): >>> #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 () from >>> /lib64/libpthread.so.0 >>> No symbol table info available. >>> #1 0x00007f1b4941b050 in PR_WaitCondVar () from /lib64/libnspr4.so >>> No symbol table info available. >>> #2 0x00007f1b4b70efee in connection_wait_for_new_work () >>> No symbol table info available. >>> #3 0x00007f1b4b710719 in connection_threadmain () >>> No symbol table info available. >>> #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so >>> No symbol table info available. >>> #5 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 >>> No symbol table info available. >>> #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 >>> No symbol table info available. >>> >>> Thread 15 (Thread 0x7f1b15d2a700 (LWP 105175)): >>> #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 () from >>> /lib64/libpthread.so.0 >>> No symbol table info available. >>> #1 0x00007f1b4941b050 in PR_WaitCondVar () from /lib64/libnspr4.so >>> No symbol table info available. >>> #2 0x00007f1b4b70efee in connection_wait_for_new_work () >>> No symbol table info available. >>> #3 0x00007f1b4b710719 in connection_threadmain () >>> No symbol table info available. >>> #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so >>> No symbol table info available. >>> #5 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 >>> No symbol table info available. >>> #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 >>> No symbol table info available. >>> >>> Thread 14 (Thread 0x7f1b1652b700 (LWP 105174)): >>> #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 () from >>> /lib64/libpthread.so.0 >>> No symbol table info available. >>> #1 0x00007f1b4941b050 in PR_WaitCondVar () from /lib64/libnspr4.so >>> No symbol table info available. >>> #2 0x00007f1b4b70efee in connection_wait_for_new_work () >>> No symbol table info available. >>> #3 0x00007f1b4b710719 in connection_threadmain () >>> No symbol table info available. >>> #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so >>> No symbol table info available. >>> #5 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 >>> No symbol table info available. >>> #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 >>> No symbol table info available. >>> >>> Thread 13 (Thread 0x7f1b16d2c700 (LWP 105173)): >>> #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 () from >>> /lib64/libpthread.so.0 >>> No symbol table info available. >>> #1 0x00007f1b433212f3 in __db_hybrid_mutex_suspend () from /lib64/ >>> libdb-5.3.so >>> No symbol table info available. >>> #2 0x00007f1b43320640 in __db_tas_mutex_lock () from /lib64/ >>> libdb-5.3.so >>> No symbol table info available. >>> #3 0x00007f1b433cad3a in __lock_get_internal () from /lib64/ >>> libdb-5.3.so >>> No symbol table info available. >>> #4 0x00007f1b433cb820 in __lock_get () from /lib64/libdb-5.3.so >>> No symbol table info available. >>> #5 0x00007f1b433f7142 in __db_lget () from /lib64/libdb-5.3.so >>> No symbol table info available. >>> #6 0x00007f1b4333e605 in __bam_search () from /lib64/libdb-5.3.so >>> No symbol table info available. >>> #7 0x00007f1b43329256 in __bamc_search () from /lib64/libdb-5.3.so >>> No symbol table info available. >>> #8 0x00007f1b4332ad0f in __bamc_get () from /lib64/libdb-5.3.so >>> No symbol table info available. >>> #9 0x00007f1b433e3ca6 in __dbc_iget () from /lib64/libdb-5.3.so >>> No symbol table info available. >>> #10 0x00007f1b433f2b02 in __dbc_get_pp () from /lib64/libdb-5.3.so >>> No symbol table info available. >>> #11 0x00007f1b3e8ce680 in idl_new_fetch () from >>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>> No symbol table info available. >>> #12 0x00007f1b3e8dcb26 in index_read_ext_allids () from >>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>> No symbol table info available. >>> #13 0x00007f1b3e8c7284 in keys2idl () from /usr/lib64/dirsrv/plugins/libb >>> ack-ldbm.so >>> No symbol table info available. >>> #14 0x00007f1b3e8c79e3 in ava_candidates.isra.0 () from >>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>> No symbol table info available. >>> #15 0x00007f1b3e8c7fd2 in filter_candidates_ext () from >>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>> No symbol table info available. >>> #16 0x00007f1b3e8c9096 in list_candidates () from >>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>> No symbol table info available. >>> #17 0x00007f1b3e8c7f40 in filter_candidates_ext () from >>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>> No symbol table info available. >>> #18 0x00007f1b3e8c9096 in list_candidates () from >>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>> No symbol table info available. >>> #19 0x00007f1b3e8c7f40 in filter_candidates_ext () from >>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>> No symbol table info available. >>> #20 0x00007f1b3e8c9096 in list_candidates () from >>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>> No symbol table info available. >>> #21 0x00007f1b3e8c7f40 in filter_candidates_ext () from >>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>> No symbol table info available. >>> #22 0x00007f1b3e90380c in subtree_candidates () from >>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>> No symbol table info available. >>> #23 0x00007f1b3e904e7f in ldbm_back_search () from >>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>> No symbol table info available. >>> #24 0x00007f1b4b2321d6 in op_shared_search () from >>> /usr/lib64/dirsrv/libslapd.so.0 >>> No symbol table info available. >>> #25 0x00007f1b4b72333e in do_search () >>> No symbol table info available. >>> #26 0x00007f1b4b7113c3 in connection_threadmain () >>> No symbol table info available. >>> #27 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so >>> No symbol table info available. >>> #28 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 >>> No symbol table info available. >>> #29 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 >>> No symbol table info available. >>> >>> Thread 12 (Thread 0x7f1b1752d700 (LWP 105172)): >>> #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 () from >>> /lib64/libpthread.so.0 >>> No symbol table info available. >>> #1 0x00007f1b4941b050 in PR_WaitCondVar () from /lib64/libnspr4.so >>> No symbol table info available. >>> #2 0x00007f1b4b70efee in connection_wait_for_new_work () >>> No symbol table info available. >>> #3 0x00007f1b4b710719 in connection_threadmain () >>> No symbol table info available. >>> #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so >>> No symbol table info available. >>> #5 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 >>> No symbol table info available. >>> #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 >>> No symbol table info available. >>> >>> Thread 11 (Thread 0x7f1b17d2e700 (LWP 105171)): >>> #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 () from >>> /lib64/libpthread.so.0 >>> No symbol table info available. >>> #1 0x00007f1b4941b050 in PR_WaitCondVar () from /lib64/libnspr4.so >>> No symbol table info available. >>> #2 0x00007f1b4b70efee in connection_wait_for_new_work () >>> No symbol table info available. >>> #3 0x00007f1b4b710719 in connection_threadmain () >>> No symbol table info available. >>> #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so >>> No symbol table info available. >>> #5 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 >>> No symbol table info available. >>> #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 >>> No symbol table info available. >>> >>> Thread 10 (Thread 0x7f1b2509d700 (LWP 105168)): >>> #0 0x00007f1b48dc5a82 in pthread_cond_timedwait@@GLIBC_2.3.2 () from >>> /lib64/libpthread.so.0 >>> No symbol table info available. >>> #1 0x00007f1b4941ab07 in pt_TimedWait () from /lib64/libnspr4.so >>> No symbol table info available. >>> #2 0x00007f1b4941afce in PR_WaitCondVar () from /lib64/libnspr4.so >>> No symbol table info available. >>> #3 0x00007f1b4b2061e8 in eq_loop () from /usr/lib64/dirsrv/libslapd.so. >>> 0 >>> No symbol table info available. >>> #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so >>> No symbol table info available. >>> #5 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 >>> No symbol table info available. >>> #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 >>> No symbol table info available. >>> >>> Thread 9 (Thread 0x7f1b2589e700 (LWP 105167)): >>> #0 0x00007f1b48dc5a82 in pthread_cond_timedwait@@GLIBC_2.3.2 () from >>> /lib64/libpthread.so.0 >>> No symbol table info available. >>> #1 0x00007f1b4941ab07 in pt_TimedWait () from /lib64/libnspr4.so >>> No symbol table info available. >>> #2 0x00007f1b4941afce in PR_WaitCondVar () from /lib64/libnspr4.so >>> No symbol table info available. >>> #3 0x00007f1b4b719d03 in housecleaning () >>> No symbol table info available. >>> #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so >>> No symbol table info available. >>> #5 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 >>> No symbol table info available. >>> #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 >>> No symbol table info available. >>> >>> Thread 8 (Thread 0x7f1b2609f700 (LWP 105166)): >>> #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 () from >>> /lib64/libpthread.so.0 >>> No symbol table info available. >>> #1 0x00007f1b4941b050 in PR_WaitCondVar () from /lib64/libnspr4.so >>> No symbol table info available. >>> #2 0x00007f1b4b260198 in slapi_wait_condvar () from >>> /usr/lib64/dirsrv/libslapd.so.0 >>> No symbol table info available. >>> #3 0x00007f1b3cf5deed in roles_cache_wait_on_change () from >>> /usr/lib64/dirsrv/plugins/libroles-plugin.so >>> No symbol table info available. >>> #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so >>> No symbol table info available. >>> #5 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 >>> No symbol table info available. >>> #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 >>> No symbol table info available. >>> >>> Thread 7 (Thread 0x7f1b268a0700 (LWP 105165)): >>> #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 () from >>> /lib64/libpthread.so.0 >>> No symbol table info available. >>> #1 0x00007f1b4941b050 in PR_WaitCondVar () from /lib64/libnspr4.so >>> No symbol table info available. >>> #2 0x00007f1b4b260198 in slapi_wait_condvar () from >>> /usr/lib64/dirsrv/libslapd.so.0 >>> No symbol table info available. >>> #3 0x00007f1b3cf5deed in roles_cache_wait_on_change () from >>> /usr/lib64/dirsrv/plugins/libroles-plugin.so >>> No symbol table info available. >>> #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so >>> No symbol table info available. >>> #5 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 >>> No symbol table info available. >>> #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 >>> No symbol table info available. >>> >>> Thread 6 (Thread 0x7f1b270a1700 (LWP 105164)): >>> #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 () from >>> /lib64/libpthread.so.0 >>> No symbol table info available. >>> #1 0x00007f1b4941b050 in PR_WaitCondVar () from /lib64/libnspr4.so >>> No symbol table info available. >>> #2 0x00007f1b4b260198 in slapi_wait_condvar () from >>> /usr/lib64/dirsrv/libslapd.so.0 >>> No symbol table info available. >>> #3 0x00007f1b3cf5deed in roles_cache_wait_on_change () from >>> /usr/lib64/dirsrv/plugins/libroles-plugin.so >>> No symbol table info available. >>> #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so >>> No symbol table info available. >>> #5 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 >>> No symbol table info available. >>> #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 >>> No symbol table info available. >>> >>> Thread 5 (Thread 0x7f1b278a2700 (LWP 105163)): >>> #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 () from >>> /lib64/libpthread.so.0 >>> No symbol table info available. >>> #1 0x00007f1b4941b050 in PR_WaitCondVar () from /lib64/libnspr4.so >>> No symbol table info available. >>> #2 0x00007f1b4b260198 in slapi_wait_condvar () from >>> /usr/lib64/dirsrv/libslapd.so.0 >>> No symbol table info available. >>> #3 0x00007f1b415f262e in cos_cache_wait_on_change () from >>> /usr/lib64/dirsrv/plugins/libcos-plugin.so >>> No symbol table info available. >>> #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so >>> No symbol table info available. >>> #5 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 >>> No symbol table info available. >>> #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 >>> No symbol table info available. >>> >>> Thread 4 (Thread 0x7f1b282ac700 (LWP 105162)): >>> #0 0x00007f1b48ae69b3 in select () from /lib64/libc.so.6 >>> No symbol table info available. >>> #1 0x00007f1b4b2710e9 in DS_Sleep () from /usr/lib64/dirsrv/libslapd.so. >>> 0 >>> No symbol table info available. >>> #2 0x00007f1b3e8ba707 in perf_threadmain () from >>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>> No symbol table info available. >>> #3 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so >>> No symbol table info available. >>> #4 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 >>> No symbol table info available. >>> #5 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 >>> No symbol table info available. >>> >>> Thread 3 (Thread 0x7f1b28aad700 (LWP 105161)): >>> #0 0x00007f1b48ae69b3 in select () from /lib64/libc.so.6 >>> No symbol table info available. >>> #1 0x00007f1b4b2710e9 in DS_Sleep () from /usr/lib64/dirsrv/libslapd.so. >>> 0 >>> No symbol table info available. >>> #2 0x00007f1b3e8bfb7f in trickle_threadmain () from >>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>> No symbol table info available. >>> #3 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so >>> No symbol table info available. >>> #4 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 >>> No symbol table info available. >>> #5 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 >>> No symbol table info available. >>> >>> Thread 2 (Thread 0x7f1b292ae700 (LWP 105160)): >>> #0 0x00007f1b48ae69b3 in select () from /lib64/libc.so.6 >>> No symbol table info available. >>> #1 0x00007f1b4344dbad in __os_yield () from /lib64/libdb-5.3.so >>> No symbol table info available. >>> #2 0x00007f1b434492b3 in __memp_sync_int () from /lib64/libdb-5.3.so >>> No symbol table info available. >>> #3 0x00007f1b43459752 in __txn_checkpoint () from /lib64/libdb-5.3.so >>> No symbol table info available. >>> #4 0x00007f1b43459b74 in __txn_checkpoint_pp () from /lib64/ >>> libdb-5.3.so >>> No symbol table info available. >>> #5 0x00007f1b3e8c3a87 in checkpoint_threadmain () from >>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>> No symbol table info available. >>> #6 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so >>> No symbol table info available. >>> #7 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 >>> No symbol table info available. >>> #8 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 >>> No symbol table info available. >>> >>> Thread 1 (Thread 0x7f1b29aaf700 (LWP 105159)): >>> #0 0x00007f1b48ae69b3 in select () from /lib64/libc.so.6 >>> No symbol table info available. >>> #1 0x00007f1b4b2710e9 in DS_Sleep () from /usr/lib64/dirsrv/libslapd.so. >>> 0 >>> No symbol table info available. >>> #2 0x00007f1b3e8bf907 in deadlock_threadmain () from >>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>> No symbol table info available. >>> #3 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so >>> No symbol table info available. >>> #4 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 >>> No symbol table info available. >>> #5 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 >>> No symbol table info available. >>> >>> >>> >>> Thanks, >>> >>> Rakesh >>> >>> >>> On Mon, Sep 5, 2016 at 4:48 PM, thierry bordaz >>> wrote: >>> >>>> >>>> >>>> On 09/05/2016 12:05 PM, Rakesh Rajasekharan wrote: >>>> >>>> Hi Thierry, >>>> >>>> >>>> I was getting the hang issue while running ipa-client-install >>>> simultaneously on few clients.. >>>> However, today, I am not able to replicate that. >>>> >>>> I could not get a gdb . But i will try getting that the next time I >>>> face this issue. >>>> >>>> The CPU does not stay high.. it just momentarily touches a high value >>>> and then drops down to around 2-7% >>>> >>>> One question I have is , is it ok to set it nsslapd-threadnumber to a >>>> very high value . >>>> I have around 4000 clients and with nsslapd-maxthreadsperconn set to >>>> 5..So, can I set nsslapd-threadnumber to around 25000. >>>> >>>> >>>> Hello, >>>> >>>> I know some users running in production with several hunderds of >>>> threads (>600) and this without problem. >>>> >>>> I do not recall having suggested to increase that number and for what >>>> reason. >>>> Usually 30 workers is a good enough value. It can create bootleneck if >>>> for some reason each operation is very long to satisfy and exhaust the >>>> number of workers. You can monitor the work queue: >>>> >>>> ldapsearch -D "cn=directory manager" -w xxx -LLL -b "cn=monitor" -s >>>> base opsinitiated opscompleted >>>> >>>> >>>> If opscompleted-opsinitiated remains close to threadnumber, then yes it >>>> would be valuable to increase it. >>>> >>>> The computation #client * #async_op_per_client sound an overkill. Even >>>> if all clients send at the exact same time all their requests, it is very >>>> likely that some common resource (db page, log, allocator...) will >>>> serialize them. If you monitor a need to increase the work, you would for >>>> example set it to 50, then monitor, then set it to 100, then monitor... >>>> until you find a good enough value. >>>> Note the increasing the #thread, increases the memory footprint that >>>> will reduce the efficiency of file system cache and can increase the >>>> response time. >>>> >>>> >>>> best regards >>>> thierry >>>> >>>> >>>> Thanks >>>> >>>> On Mon, Sep 5, 2016 at 1:03 PM, thierry bordaz >>>> wrote: >>>> >>>>> >>>>> Hi Rakesh, >>>>> >>>>> Were you able to get a pstack or full stack with gdb ( >>>>> http://www.port389.org/docs/389ds/FAQ/faq.html#debugging-crashes) >>>>> when the server hangs ? >>>>> >>>>> If it happens with 500 threads as well as with 30, using 30 threads is >>>>> a better choice to debug this issue. >>>>> I will try to reproduce using 150 parallel 'ipa user-find p-testipa' >>>>> commands >>>>> >>>>> Something I am unsure is if the CPU consumption stays high (you >>>>> mentioned 340% CPU usage) as long as the hang happens or if after a suddent >>>>> shot up to 340% (that marks the beginning of the hang) it drops and stay >>>>> hanging ? >>>>> >>>>> thanks >>>>> thierry >>>>> >>>>> On 09/04/2016 08:40 PM, Rakesh Rajasekharan wrote: >>>>> >>>>> starce on the slapd process actually had this in the output.. >>>>> FUTEX_WAIT_PRIVATE >>>>> >>>>> and checking for the number of threads slapd had.. there were 5015 >>>>> threads >>>>> >>>>> ps -efL|grep slapd|wc -l >>>>> 5015 >>>>> >>>>> strace on most of the threads gave this output >>>>> >>>>> strace -p 67411 >>>>> Process 67411 attached >>>>> futex(0x7f3f0226b41c, FUTEX_WAIT_PRIVATE, 1, NULL) = -1 EAGAIN >>>>> (Resource temporarily unavailable) >>>>> futex(0x7f3f0226b41c, FUTEX_WAIT_PRIVATE, 2, NULL^CProcess 67411 >>>>> detached >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> On Sun, Sep 4, 2016 at 5:34 PM, Rakesh Rajasekharan < >>>>> rakesh.rajasekharan at gmail.com> wrote: >>>>> >>>>>> I have again got the issue of IPA hanging.. The issue came up when i >>>>>> tried to run ipa-client-isntall on 142 clients simultaneously >>>>>> >>>>>> >>>>>> None of the IPA commands are responding, and I see this error >>>>>> >>>>>> ipa user-find p-testipa >>>>>> ipa: ERROR: Insufficient access: SASL(-1): generic failure: GSSAPI >>>>>> Error: Unspecified GSS failure. Minor code may provide more information >>>>>> (KDC returned error string: PROCESS_TGS) >>>>>> >>>>>> KRB5_TRACE=/dev/stdout kinit admin >>>>>> [41178] 1472984115.233214: Getting initial credentials for >>>>>> admin at XYZ.COM >>>>>> [41178] 1472984115.235257: Sending request (167 bytes) to XYZ.COM >>>>>> [41178] 1472984115.235419: Initiating TCP connection to stream >>>>>> 10.1.3.36:88 >>>>>> [41178] 1472984115.235685: Sending TCP request to stream 10.1.3.36:88 >>>>>> [41178] 1472984120.238914: Received answer (174 bytes) from stream >>>>>> 10.1.3.36:88 >>>>>> [41178] 1472984120.238925: Terminating TCP connection to stream >>>>>> 10.1.3.36:88 >>>>>> [41178] 1472984120.238993: Response was from master KDC >>>>>> [41 >>>>>> >>>>>> >>>>>> Running an ldapsearch to see the db.. does not give any results and >>>>>> just hangs there >>>>>> >>>>>> ldapsearch -x -D 'cn=Directory Manager' -W -s one -b >>>>>> 'cn=kerberos,dc=xyz,dc=com' >>>>>> Enter LDAP Password: >>>>>> >>>>>> even an ldapsearch -x does not respond >>>>>> At this point, am sure that slapd is the one causing issues >>>>>> >>>>>> Running an strace against the hung slapd itself seems to get stuck >>>>>> does not proceed after saying "attaching to process" >>>>>> >>>>>> From some others posts I read Thierry suggesting to increase the >>>>>> nsslapd-threadnumber value >>>>>> >>>>>> It was set to 30, I think that might be too low. >>>>>> >>>>> ... >>> >>> [Message clipped] >> >> >> > > > -- > Red Hat GmbH, http://www.de.redhat.com/, Registered seat: Grasbrunn, > Commercial register: Amtsgericht Muenchen, HRB 153243, > Managing Directors: Charles Cachera, Michael Cunningham, Michael O'Neill, Eric Shander > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jdieter at lesbg.com Tue Sep 6 14:07:55 2016 From: jdieter at lesbg.com (Jonathan Dieter) Date: Tue, 06 Sep 2016 17:07:55 +0300 Subject: [Freeipa-users] Unable to setup replica (kadmin.local: No such entry in the database) In-Reply-To: <1473152826.2014.17.camel@lesbg.com> References: <1473152826.2014.17.camel@lesbg.com> Message-ID: <1473170875.2014.22.camel@lesbg.com> On Tue, 2016-09-06 at 12:07 +0300, Jonathan Dieter wrote: > I'm trying to add a fourth replica and it's failing in two places > (depending on whether I enable the CA or not). Both problems were tied to the same thing: I had a root user setup in IPA (don't ask me why, I'm sure it seemed like a good idea at the time), and, in both cases, the upgrade script was trying to use the root user via Kerberos rather than the Directory Manager. I've removed root from IPA, and the replica installed perfectly. Jonathan From th at casalogic.dk Wed Sep 7 07:15:21 2016 From: th at casalogic.dk (Troels Hansen) Date: Wed, 7 Sep 2016 09:15:21 +0200 (CEST) Subject: [Freeipa-users] SSH login using putty from Windows to SSSD client in IPA AD trust Message-ID: <1321457441.176184.1473232521372.JavaMail.zimbra@casalogic.dk> Running RHEL 7.2, IPA 4.2 and SSSD 1.13, we have set up a IPA-AD trust and trying to get Putty GSSAPI login to work. In Putty GSSAPI have been enabled, and GSSAPI is enabled in sshd. Logging in using password from Windows to Linux works, and logging in from Linux to Linux using kerberos works. AD trust is a follows: # ipa trust-find ---------------- 2 trusts matched ---------------- Realm name: net.dr.dk Domain NetBIOS name: NET Domain Security Identifier: S-1-5-21-xxxxxxxxx-xxxxxxxx-xxxxxxxx Realm name: place.dr.dk Domain NetBIOS name: PLACE Domain Security Identifier: S-1-5-21-xxxxxx-xxxxxx-xxxxxxx Trust type: Active Directory domain ---------------------------- Number of entries returned 2 ---------------------------- # ipa trust-show place.dr.dk Realm name: place.dr.dk Domain NetBIOS name: PLACE Domain Security Identifier: S-1-5-21-xxxx-xxxx-xxxxx Trust direction: Trusting forest Trust type: Active Directory domain # ipa trust-show net.dr.dk Realm name: net.dr.dk Domain NetBIOS name: NET Domain Security Identifier: S-1-5-21-xxxxxxxxxxxxx-xxxxxxxxxxxx-xxxxxxxxxx users are located in net.dr.dk. >From looking at the doc's this should just work... However, can't get it to work. Am I missing something? -- Med venlig hilsen Troels Hansen Systemkonsulent Casalogic A/S T (+45) 70 20 10 63 M (+45) 22 43 71 57 Red Hat, SUSE, VMware, Citrix, Novell, Yellowfin BI, EnterpriseDB, Sophos og meget mere. -------------- next part -------------- An HTML attachment was scrubbed... URL: From abokovoy at redhat.com Wed Sep 7 07:27:17 2016 From: abokovoy at redhat.com (Alexander Bokovoy) Date: Wed, 7 Sep 2016 10:27:17 +0300 Subject: [Freeipa-users] SSH login using putty from Windows to SSSD client in IPA AD trust In-Reply-To: <1321457441.176184.1473232521372.JavaMail.zimbra@casalogic.dk> References: <1321457441.176184.1473232521372.JavaMail.zimbra@casalogic.dk> Message-ID: <20160907072717.n232wk2vdqdwvpnt@redhat.com> On Wed, 07 Sep 2016, Troels Hansen wrote: >Running RHEL 7.2, IPA 4.2 and SSSD 1.13, we have set up a IPA-AD trust >and trying to get Putty GSSAPI login to work. In Putty GSSAPI have >been enabled, and GSSAPI is enabled in sshd. > >Logging in using password from Windows to Linux works, and logging in >from Linux to Linux using kerberos works. > >AD trust is a follows: > ># ipa trust-find >---------------- >2 trusts matched >---------------- >Realm name: net.dr.dk >Domain NetBIOS name: NET >Domain Security Identifier: S-1-5-21-xxxxxxxxx-xxxxxxxx-xxxxxxxx > >Realm name: place.dr.dk >Domain NetBIOS name: PLACE >Domain Security Identifier: S-1-5-21-xxxxxx-xxxxxx-xxxxxxx >Trust type: Active Directory domain >---------------------------- >Number of entries returned 2 >---------------------------- > ># ipa trust-show place.dr.dk >Realm name: place.dr.dk >Domain NetBIOS name: PLACE >Domain Security Identifier: S-1-5-21-xxxx-xxxx-xxxxx >Trust direction: Trusting forest >Trust type: Active Directory domain > ># ipa trust-show net.dr.dk >Realm name: net.dr.dk >Domain NetBIOS name: NET >Domain Security Identifier: S-1-5-21-xxxxxxxxxxxxx-xxxxxxxxxxxx-xxxxxxxxxx > >users are located in net.dr.dk. > >>From looking at the doc's this should just work... However, can't get >>it to work. Am I missing something? Make screenshots of PuTTY screens showing what you configured and what does not work. You can also ask PuTTY to generate logs. -- / Alexander Bokovoy From sbose at redhat.com Wed Sep 7 07:43:00 2016 From: sbose at redhat.com (Sumit Bose) Date: Wed, 7 Sep 2016 09:43:00 +0200 Subject: [Freeipa-users] SSH login using putty from Windows to SSSD client in IPA AD trust In-Reply-To: <20160907072717.n232wk2vdqdwvpnt@redhat.com> References: <1321457441.176184.1473232521372.JavaMail.zimbra@casalogic.dk> <20160907072717.n232wk2vdqdwvpnt@redhat.com> Message-ID: <20160907074300.GT5463@p.Speedport_W_724V_Typ_A_05011603_00_009> On Wed, Sep 07, 2016 at 10:27:17AM +0300, Alexander Bokovoy wrote: > On Wed, 07 Sep 2016, Troels Hansen wrote: > > Running RHEL 7.2, IPA 4.2 and SSSD 1.13, we have set up a IPA-AD trust > > and trying to get Putty GSSAPI login to work. In Putty GSSAPI have > > been enabled, and GSSAPI is enabled in sshd. > > > > Logging in using password from Windows to Linux works, and logging in > > from Linux to Linux using kerberos works. > > > > AD trust is a follows: > > > > # ipa trust-find > > ---------------- > > 2 trusts matched > > ---------------- > > Realm name: net.dr.dk > > Domain NetBIOS name: NET > > Domain Security Identifier: S-1-5-21-xxxxxxxxx-xxxxxxxx-xxxxxxxx > > > > Realm name: place.dr.dk > > Domain NetBIOS name: PLACE > > Domain Security Identifier: S-1-5-21-xxxxxx-xxxxxx-xxxxxxx > > Trust type: Active Directory domain > > ---------------------------- > > Number of entries returned 2 > > ---------------------------- > > > > # ipa trust-show place.dr.dk > > Realm name: place.dr.dk > > Domain NetBIOS name: PLACE > > Domain Security Identifier: S-1-5-21-xxxx-xxxx-xxxxx > > Trust direction: Trusting forest > > Trust type: Active Directory domain > > > > # ipa trust-show net.dr.dk > > Realm name: net.dr.dk > > Domain NetBIOS name: NET > > Domain Security Identifier: S-1-5-21-xxxxxxxxxxxxx-xxxxxxxxxxxx-xxxxxxxxxx > > > > users are located in net.dr.dk. > > > > > From looking at the doc's this should just work... However, can't get > > > it to work. Am I missing something? > Make screenshots of PuTTY screens showing what you configured and what > does not work. You can also ask PuTTY to generate logs. Additionally please check the klist output on the Windows client. It should show the host principal of the Linux client (host/client.ipa.domain at IPA.DOMAIN). If the principal is there the sshd logs on the Linux client with a high debug level might also have some hints why GSSAPI authentication failed. HTH bye, Sumit > > -- > / Alexander Bokovoy > > -- > Manage your subscription for the Freeipa-users mailing list: > https://www.redhat.com/mailman/listinfo/freeipa-users > Go to http://freeipa.org for more info on the project From th at casalogic.dk Wed Sep 7 07:49:35 2016 From: th at casalogic.dk (Troels Hansen) Date: Wed, 7 Sep 2016 09:49:35 +0200 (CEST) Subject: [Freeipa-users] SSH login using putty from Windows to SSSD client in IPA AD trust In-Reply-To: <20160907072717.n232wk2vdqdwvpnt@redhat.com> References: <1321457441.176184.1473232521372.JavaMail.zimbra@casalogic.dk> <20160907072717.n232wk2vdqdwvpnt@redhat.com> Message-ID: <564306918.176779.1473234575421.JavaMail.zimbra@casalogic.dk> When logging in, putty only shows: Using username "drextrha at net.dr.dk". drextrha at net.dr.dk@rhel02udv.linux.dr.dk's password: Putty log shows its only using SSPI, secur32.dll for GSSAPI, but fails: Event Log: Using SSPI from SECUR32.DLL Event Log: Attempting GSSAPI authentication Outgoing packet #0x6, type 50 / 0x32 (SSH2_MSG_USERAUTH_REQUEST) 00000000 00 00 00 12 64 72 65 78 74 72 68 61 40 6e 65 74 ....drextrha at net 00000010 2e 64 72 2e 64 6b 00 00 00 0e 73 73 68 2d 63 6f .dr.dk....ssh-co 00000020 6e 6e 65 63 74 69 6f 6e 00 00 00 0f 67 73 73 61 nnection....gssa 00000030 70 69 2d 77 69 74 68 2d 6d 69 63 00 00 00 01 00 pi-with-mic..... 00000040 00 00 0b 06 09 2a 86 48 86 f7 12 01 02 02 .....*.H...... Incoming packet #0x6, type 60 / 0x3c (SSH2_MSG_USERAUTH_GSSAPI_RESPONSE) 00000000 00 00 00 0b 06 09 2a 86 48 86 f7 12 01 02 02 ......*.H...... Event Log: GSSAPI authentication initialisation failed Event Log: The target was not recognized. ----- On Sep 7, 2016, at 9:27 AM, Alexander Bokovoy wrote: > On Wed, 07 Sep 2016, Troels Hansen wrote: >> Running RHEL 7.2, IPA 4.2 and SSSD 1.13, we have set up a IPA-AD trust >> and trying to get Putty GSSAPI login to work. In Putty GSSAPI have >> been enabled, and GSSAPI is enabled in sshd. >> Logging in using password from Windows to Linux works, and logging in >> from Linux to Linux using kerberos works. >> AD trust is a follows: >> # ipa trust-find >> ---------------- >> 2 trusts matched >> ---------------- >> Realm name: net.dr.dk >> Domain NetBIOS name: NET >> Domain Security Identifier: S-1-5-21-xxxxxxxxx-xxxxxxxx-xxxxxxxx >> Realm name: place.dr.dk >> Domain NetBIOS name: PLACE >> Domain Security Identifier: S-1-5-21-xxxxxx-xxxxxx-xxxxxxx >> Trust type: Active Directory domain >> ---------------------------- >> Number of entries returned 2 >> ---------------------------- >> # ipa trust-show place.dr.dk >> Realm name: place.dr.dk >> Domain NetBIOS name: PLACE >> Domain Security Identifier: S-1-5-21-xxxx-xxxx-xxxxx >> Trust direction: Trusting forest >> Trust type: Active Directory domain >> # ipa trust-show net.dr.dk >> Realm name: net.dr.dk >> Domain NetBIOS name: NET >> Domain Security Identifier: S-1-5-21-xxxxxxxxxxxxx-xxxxxxxxxxxx-xxxxxxxxxx >> users are located in net.dr.dk. >>> From looking at the doc's this should just work... However, can't get >>> it to work. Am I missing something? > Make screenshots of PuTTY screens showing what you configured and what > does not work. You can also ask PuTTY to generate logs. > -- > / Alexander Bokovoy -- Med venlig hilsen Troels Hansen Systemkonsulent Casalogic A/S T (+45) 70 20 10 63 M (+45) 22 43 71 57 Red Hat, SUSE, VMware, Citrix, Novell, Yellowfin BI, EnterpriseDB, Sophos og meget mere. -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: gssapi.png Type: image/png Size: 45392 bytes Desc: not available URL: From abokovoy at redhat.com Wed Sep 7 07:55:27 2016 From: abokovoy at redhat.com (Alexander Bokovoy) Date: Wed, 7 Sep 2016 10:55:27 +0300 Subject: [Freeipa-users] SSH login using putty from Windows to SSSD client in IPA AD trust In-Reply-To: <564306918.176779.1473234575421.JavaMail.zimbra@casalogic.dk> References: <1321457441.176184.1473232521372.JavaMail.zimbra@casalogic.dk> <20160907072717.n232wk2vdqdwvpnt@redhat.com> <564306918.176779.1473234575421.JavaMail.zimbra@casalogic.dk> Message-ID: <20160907075527.5djfu5bsjgpdwekm@redhat.com> On Wed, 07 Sep 2016, Troels Hansen wrote: >When logging in, putty only shows: >Using username "drextrha at net.dr.dk". >drextrha at net.dr.dk@rhel02udv.linux.dr.dk's password: > >Putty log shows its only using SSPI, secur32.dll for GSSAPI, but fails: > >Event Log: Using SSPI from SECUR32.DLL >Event Log: Attempting GSSAPI authentication >Outgoing packet #0x6, type 50 / 0x32 (SSH2_MSG_USERAUTH_REQUEST) >00000000 00 00 00 12 64 72 65 78 74 72 68 61 40 6e 65 74 ....drextrha at net >00000010 2e 64 72 2e 64 6b 00 00 00 0e 73 73 68 2d 63 6f .dr.dk....ssh-co >00000020 6e 6e 65 63 74 69 6f 6e 00 00 00 0f 67 73 73 61 nnection....gssa >00000030 70 69 2d 77 69 74 68 2d 6d 69 63 00 00 00 01 00 pi-with-mic..... >00000040 00 00 0b 06 09 2a 86 48 86 f7 12 01 02 02 .....*.H...... >Incoming packet #0x6, type 60 / 0x3c (SSH2_MSG_USERAUTH_GSSAPI_RESPONSE) >00000000 00 00 00 0b 06 09 2a 86 48 86 f7 12 01 02 02 ......*.H...... >Event Log: GSSAPI authentication initialisation failed >Event Log: The target was not recognized. "Target was not recognized" means AD DC doesn't know that rhel02edv.linux.dr.dk belongs to LINUX.DR.DK realm and thus has to forward the authentication requests there. What do you have in the trust properties on AD side? Specifically, what does name routing suffixes show there? > >----- On Sep 7, 2016, at 9:27 AM, Alexander Bokovoy wrote: > >> On Wed, 07 Sep 2016, Troels Hansen wrote: > >>> Running RHEL 7.2, IPA 4.2 and SSSD 1.13, we have set up a IPA-AD trust >>> and trying to get Putty GSSAPI login to work. In Putty GSSAPI have >>> been enabled, and GSSAPI is enabled in sshd. > >>> Logging in using password from Windows to Linux works, and logging in >>> from Linux to Linux using kerberos works. > >>> AD trust is a follows: > >>> # ipa trust-find >>> ---------------- >>> 2 trusts matched >>> ---------------- >>> Realm name: net.dr.dk >>> Domain NetBIOS name: NET >>> Domain Security Identifier: S-1-5-21-xxxxxxxxx-xxxxxxxx-xxxxxxxx > >>> Realm name: place.dr.dk >>> Domain NetBIOS name: PLACE >>> Domain Security Identifier: S-1-5-21-xxxxxx-xxxxxx-xxxxxxx >>> Trust type: Active Directory domain >>> ---------------------------- >>> Number of entries returned 2 >>> ---------------------------- > >>> # ipa trust-show place.dr.dk >>> Realm name: place.dr.dk >>> Domain NetBIOS name: PLACE >>> Domain Security Identifier: S-1-5-21-xxxx-xxxx-xxxxx >>> Trust direction: Trusting forest >>> Trust type: Active Directory domain > >>> # ipa trust-show net.dr.dk >>> Realm name: net.dr.dk >>> Domain NetBIOS name: NET >>> Domain Security Identifier: S-1-5-21-xxxxxxxxxxxxx-xxxxxxxxxxxx-xxxxxxxxxx > >>> users are located in net.dr.dk. > >>>> From looking at the doc's this should just work... However, can't get >>>> it to work. Am I missing something? >> Make screenshots of PuTTY screens showing what you configured and what >> does not work. You can also ask PuTTY to generate logs. > >> -- >> / Alexander Bokovoy > >-- > >Med venlig hilsen > >Troels Hansen > >Systemkonsulent > >Casalogic A/S > >T (+45) 70 20 10 63 > >M (+45) 22 43 71 57 >Red Hat, SUSE, VMware, Citrix, Novell, Yellowfin BI, EnterpriseDB, Sophos og meget mere. >-- >Manage your subscription for the Freeipa-users mailing list: >https://www.redhat.com/mailman/listinfo/freeipa-users >Go to http://freeipa.org for more info on the project -- / Alexander Bokovoy From th at casalogic.dk Wed Sep 7 07:55:45 2016 From: th at casalogic.dk (Troels Hansen) Date: Wed, 7 Sep 2016 09:55:45 +0200 (CEST) Subject: [Freeipa-users] SSH login using putty from Windows to SSSD client in IPA AD trust In-Reply-To: <20160907074300.GT5463@p.Speedport_W_724V_Typ_A_05011603_00_009> References: <1321457441.176184.1473232521372.JavaMail.zimbra@casalogic.dk> <20160907072717.n232wk2vdqdwvpnt@redhat.com> <20160907074300.GT5463@p.Speedport_W_724V_Typ_A_05011603_00_009> Message-ID: <753870620.176895.1473234945940.JavaMail.zimbra@casalogic.dk> ----- On Sep 7, 2016, at 9:43 AM, Sumit Bose sbose at redhat.com wrote: > Additionally please check the klist output on the Windows client. It > should show the host principal of the Linux client > (host/client.ipa.domain at IPA.DOMAIN). If the principal is there the sshd > logs on the Linux client with a high debug level might also have some > hints why GSSAPI authentication failed. > Hmm, no host tickets. Only krbtgt for the domain and LDAP and CIFS principal for thc DC's From th at casalogic.dk Wed Sep 7 08:17:54 2016 From: th at casalogic.dk (Troels Hansen) Date: Wed, 7 Sep 2016 10:17:54 +0200 (CEST) Subject: [Freeipa-users] SSH login using putty from Windows to SSSD client in IPA AD trust In-Reply-To: <20160907075527.5djfu5bsjgpdwekm@redhat.com> References: <1321457441.176184.1473232521372.JavaMail.zimbra@casalogic.dk> <20160907072717.n232wk2vdqdwvpnt@redhat.com> <564306918.176779.1473234575421.JavaMail.zimbra@casalogic.dk> <20160907075527.5djfu5bsjgpdwekm@redhat.com> Message-ID: <1617255738.177066.1473236274783.JavaMail.zimbra@casalogic.dk> ----- On Sep 7, 2016, at 9:55 AM, Alexander Bokovoy abokovoy at redhat.com wrote: > "Target was not recognized" means AD DC doesn't know that > rhel02edv.linux.dr.dk belongs to LINUX.DR.DK realm and thus has to > forward the authentication requests there. > > What do you have in the trust properties on AD side? Specifically, what > does name routing suffixes show there? Yes, its correct, there is no routing configured. I can't see to be able to add it manually, and auto refresh doesn't work: https://fedorahosted.org/freeipa/ticket/5683 ?? From th at casalogic.dk Wed Sep 7 08:31:09 2016 From: th at casalogic.dk (Troels Hansen) Date: Wed, 7 Sep 2016 10:31:09 +0200 (CEST) Subject: [Freeipa-users] SSH login using putty from Windows to SSSD client in IPA AD trust In-Reply-To: <1617255738.177066.1473236274783.JavaMail.zimbra@casalogic.dk> References: <1321457441.176184.1473232521372.JavaMail.zimbra@casalogic.dk> <20160907072717.n232wk2vdqdwvpnt@redhat.com> <564306918.176779.1473234575421.JavaMail.zimbra@casalogic.dk> <20160907075527.5djfu5bsjgpdwekm@redhat.com> <1617255738.177066.1473236274783.JavaMail.zimbra@casalogic.dk> Message-ID: <936943241.177115.1473237069674.JavaMail.zimbra@casalogic.dk> ----- On Sep 7, 2016, at 10:17 AM, Troels Hansen th at casalogic.dk wrote: > > Yes, its correct, there is no routing configured. > I can't see to be able to add it manually, and auto refresh doesn't work: > https://fedorahosted.org/freeipa/ticket/5683 > According to the DOC's it should work? https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/7/html/Windows_Integration_Guide/creating-trusts.html#pic.trust-refresh-routing From sbose at redhat.com Wed Sep 7 08:31:33 2016 From: sbose at redhat.com (Sumit Bose) Date: Wed, 7 Sep 2016 10:31:33 +0200 Subject: [Freeipa-users] SSH login using putty from Windows to SSSD client in IPA AD trust In-Reply-To: <753870620.176895.1473234945940.JavaMail.zimbra@casalogic.dk> References: <1321457441.176184.1473232521372.JavaMail.zimbra@casalogic.dk> <20160907072717.n232wk2vdqdwvpnt@redhat.com> <20160907074300.GT5463@p.Speedport_W_724V_Typ_A_05011603_00_009> <753870620.176895.1473234945940.JavaMail.zimbra@casalogic.dk> Message-ID: <20160907083133.GU5463@p.Speedport_W_724V_Typ_A_05011603_00_009> On Wed, Sep 07, 2016 at 09:55:45AM +0200, Troels Hansen wrote: > > > ----- On Sep 7, 2016, at 9:43 AM, Sumit Bose sbose at redhat.com wrote: > > > Additionally please check the klist output on the Windows client. It > > should show the host principal of the Linux client > > (host/client.ipa.domain at IPA.DOMAIN). If the principal is there the sshd > > logs on the Linux client with a high debug level might also have some > > hints why GSSAPI authentication failed. > > > > > Hmm, no host tickets. Only krbtgt for the domain and LDAP and CIFS principal for thc DC's So I guess there is no cross-realm ticket either, i.e. krbtgt/IPA.DOMAIN at AD.DOMAIN. Can you check on AD if the IPA DNS domain is listed in the 'Name Suffix Routing' tab in the trust properties of the IPA domain? Additionally please check if the DNS SRV records like e.g. _kerberos._udp.ipa.domain can be resolved on the AD side. HTH bye, Sumit From abokovoy at redhat.com Wed Sep 7 08:36:03 2016 From: abokovoy at redhat.com (Alexander Bokovoy) Date: Wed, 7 Sep 2016 11:36:03 +0300 Subject: [Freeipa-users] SSH login using putty from Windows to SSSD client in IPA AD trust In-Reply-To: <1617255738.177066.1473236274783.JavaMail.zimbra@casalogic.dk> References: <1321457441.176184.1473232521372.JavaMail.zimbra@casalogic.dk> <20160907072717.n232wk2vdqdwvpnt@redhat.com> <564306918.176779.1473234575421.JavaMail.zimbra@casalogic.dk> <20160907075527.5djfu5bsjgpdwekm@redhat.com> <1617255738.177066.1473236274783.JavaMail.zimbra@casalogic.dk> Message-ID: <20160907083603.sr7skn53flrpvloj@redhat.com> On Wed, 07 Sep 2016, Troels Hansen wrote: > >----- On Sep 7, 2016, at 9:55 AM, Alexander Bokovoy abokovoy at redhat.com wrote: > >> "Target was not recognized" means AD DC doesn't know that >> rhel02edv.linux.dr.dk belongs to LINUX.DR.DK realm and thus has to >> forward the authentication requests there. >> >> What do you have in the trust properties on AD side? Specifically, what >> does name routing suffixes show there? > >Yes, its correct, there is no routing configured. >I can't see to be able to add it manually, and auto refresh doesn't work: >https://fedorahosted.org/freeipa/ticket/5683 How exactly did you establish the trust? I see you have one-way trust but did you establish it with AD admin credentials or using a shared secret? If the latter, it is a known issue that AD does not activate the trust for shared secret one-way case and aforementioned bug prevents us to validate the rust afterwards. -- / Alexander Bokovoy From th at casalogic.dk Wed Sep 7 08:45:07 2016 From: th at casalogic.dk (Troels Hansen) Date: Wed, 7 Sep 2016 10:45:07 +0200 (CEST) Subject: [Freeipa-users] SSH login using putty from Windows to SSSD client in IPA AD trust In-Reply-To: <20160907083603.sr7skn53flrpvloj@redhat.com> References: <1321457441.176184.1473232521372.JavaMail.zimbra@casalogic.dk> <20160907072717.n232wk2vdqdwvpnt@redhat.com> <564306918.176779.1473234575421.JavaMail.zimbra@casalogic.dk> <20160907075527.5djfu5bsjgpdwekm@redhat.com> <1617255738.177066.1473236274783.JavaMail.zimbra@casalogic.dk> <20160907083603.sr7skn53flrpvloj@redhat.com> Message-ID: <702327034.177297.1473237907454.JavaMail.zimbra@casalogic.dk> ----- On Sep 7, 2016, at 10:36 AM, Alexander Bokovoy abokovoy at redhat.com wrote: > How exactly did you establish the trust? I see you have one-way trust > but did you establish it with AD admin credentials or using a shared > secret? If the latter, it is a known issue that AD does not activate the > trust for shared secret one-way case and aforementioned bug prevents us > to validate the rust afterwards. Not quite sure actually. I can remember we tried using shared secret but not sure if we got it to work or if we falled back to user and password (bash history on IPA server expired). From th at casalogic.dk Wed Sep 7 08:47:26 2016 From: th at casalogic.dk (Troels Hansen) Date: Wed, 7 Sep 2016 10:47:26 +0200 (CEST) Subject: [Freeipa-users] SSH login using putty from Windows to SSSD client in IPA AD trust In-Reply-To: <20160907083133.GU5463@p.Speedport_W_724V_Typ_A_05011603_00_009> References: <1321457441.176184.1473232521372.JavaMail.zimbra@casalogic.dk> <20160907072717.n232wk2vdqdwvpnt@redhat.com> <20160907074300.GT5463@p.Speedport_W_724V_Typ_A_05011603_00_009> <753870620.176895.1473234945940.JavaMail.zimbra@casalogic.dk> <20160907083133.GU5463@p.Speedport_W_724V_Typ_A_05011603_00_009> Message-ID: <1109708738.177438.1473238046693.JavaMail.zimbra@casalogic.dk> ----- On Sep 7, 2016, at 10:31 AM, Sumit Bose sbose at redhat.com wrote: > > So I guess there is no cross-realm ticket either, i.e. > krbtgt/IPA.DOMAIN at AD.DOMAIN. Can you check on AD if the IPA DNS domain > is listed in the 'Name Suffix Routing' tab in the trust properties of > the IPA domain? Additionally please check if the DNS SRV records like > e.g. _kerberos._udp.ipa.domain can be resolved on the AD side. > No, no cross realm tickets on Windows client. Its a one-way trust if that makes a difference? DNS is working. DNS config is only done on AD side, so IPA dns config is done there and Linux clients is configured to use AD as DNS. Alexander just wrote that if we had used shared secret to create the trust the routing is missing and can't be fetched afterwards. From abokovoy at redhat.com Wed Sep 7 08:59:44 2016 From: abokovoy at redhat.com (Alexander Bokovoy) Date: Wed, 7 Sep 2016 11:59:44 +0300 Subject: [Freeipa-users] SSH login using putty from Windows to SSSD client in IPA AD trust In-Reply-To: <702327034.177297.1473237907454.JavaMail.zimbra@casalogic.dk> References: <1321457441.176184.1473232521372.JavaMail.zimbra@casalogic.dk> <20160907072717.n232wk2vdqdwvpnt@redhat.com> <564306918.176779.1473234575421.JavaMail.zimbra@casalogic.dk> <20160907075527.5djfu5bsjgpdwekm@redhat.com> <1617255738.177066.1473236274783.JavaMail.zimbra@casalogic.dk> <20160907083603.sr7skn53flrpvloj@redhat.com> <702327034.177297.1473237907454.JavaMail.zimbra@casalogic.dk> Message-ID: <20160907085944.ploxchikfizlv2mu@redhat.com> On Wed, 07 Sep 2016, Troels Hansen wrote: >----- On Sep 7, 2016, at 10:36 AM, Alexander Bokovoy abokovoy at redhat.com wrote: > >> How exactly did you establish the trust? I see you have one-way trust >> but did you establish it with AD admin credentials or using a shared >> secret? If the latter, it is a known issue that AD does not activate the >> trust for shared secret one-way case and aforementioned bug prevents us >> to validate the rust afterwards. > > >Not quite sure actually. >I can remember we tried using shared secret but not sure if we got it >to work or if we falled back to user and password (bash history on IPA >server expired). There are two solutions here: use admin credentials to establish one-way trust or use two-way trust (whether with shared secret or admin credentials). You can re-establish trust. It will drop the trusted domain objects on both sides and re-create them, but the rest will be kept intact on IPA side, so it could be used to repair such cases. -- / Alexander Bokovoy From rakesh.rajasekharan at gmail.com Wed Sep 7 11:46:07 2016 From: rakesh.rajasekharan at gmail.com (Rakesh Rajasekharan) Date: Wed, 7 Sep 2016 17:16:07 +0530 Subject: [Freeipa-users] Freeipa 4.2.0 hangs intermittently In-Reply-To: References: <57BEB0AB.2090506@redhat.com> <57C44AC6.1030402@redhat.com> <57CD1FDC.2010109@redhat.com> <57CD5488.60404@redhat.com> <57CD659B.6050707@redhat.com> <57CD68BD.9030802@redhat.com> <57CE8CDB.6000809@redhat.com> <57CE9DE5.6020301@redhat.com> Message-ID: I changed the nsslapd-db-checkpoint interval first to 80 and increased till 160.. however the hang has not gone away yet I migrated my data from a openldap, can that cause issues like this.. If so, i could try creating users from scratch directly on IPA rather than migrating and test it But before that , have taken a fresh pstack just incase if it helps to nail down the issue Thread 44 (Thread 0x7f53ed52f700 (LWP 128165)): #0 0x00007f54047649b3 in select () from /lib64/libc.so.6 #1 0x00007f5406eef0e9 in DS_Sleep () from /usr/lib64/dirsrv/libslapd.so.0 #2 0x00007f53fa53d907 in deadlock_threadmain () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #3 0x00007f540509e7bb in _pt_root () from /lib64/libnspr4.so #4 0x00007f5404a3fdc5 in start_thread () from /lib64/libpthread.so.0 #5 0x00007f540476d28d in clone () from /lib64/libc.so.6 Thread 43 (Thread 0x7f53ecd2e700 (LWP 128166)): #0 0x00007f54047649b3 in select () from /lib64/libc.so.6 #1 0x00007f53ff0cbbad in __os_yield () from /lib64/libdb-5.3.so #2 0x00007f53ff0c72b3 in __memp_sync_int () from /lib64/libdb-5.3.so #3 0x00007f53ff0d7752 in __txn_checkpoint () from /lib64/libdb-5.3.so #4 0x00007f53ff0d7b74 in __txn_checkpoint_pp () from /lib64/libdb-5.3.so #5 0x00007f53fa541a87 in checkpoint_threadmain () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #6 0x00007f540509e7bb in _pt_root () from /lib64/libnspr4.so #7 0x00007f5404a3fdc5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007f540476d28d in clone () from /lib64/libc.so.6 Thread 42 (Thread 0x7f53ec52d700 (LWP 128167)): #0 0x00007f54047649b3 in select () from /lib64/libc.so.6 #1 0x00007f5406eef0e9 in DS_Sleep () from /usr/lib64/dirsrv/libslapd.so.0 #2 0x00007f53fa53db7f in trickle_threadmain () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #3 0x00007f540509e7bb in _pt_root () from /lib64/libnspr4.so #4 0x00007f5404a3fdc5 in start_thread () from /lib64/libpthread.so.0 #5 0x00007f540476d28d in clone () from /lib64/libc.so.6 Thread 41 (Thread 0x7f53ebd2c700 (LWP 128168)): #0 0x00007f54047649b3 in select () from /lib64/libc.so.6 #1 0x00007f5406eef0e9 in DS_Sleep () from /usr/lib64/dirsrv/libslapd.so.0 #2 0x00007f53fa538707 in perf_threadmain () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #3 0x00007f540509e7bb in _pt_root () from /lib64/libnspr4.so #4 0x00007f5404a3fdc5 in start_thread () from /lib64/libpthread.so.0 #5 0x00007f540476d28d in clone () from /lib64/libc.so.6 Thread 40 (Thread 0x7f53eb322700 (LWP 128204)): #0 0x00007f5404a436d5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f5405099050 in PR_WaitCondVar () from /lib64/libnspr4.so #2 0x00007f5406ede198 in slapi_wait_condvar () from /usr/lib64/dirsrv/libslapd.so.0 #3 0x00007f53fd27062e in cos_cache_wait_on_change () from /usr/lib64/dirsrv/plugins/libcos-plugin.so #4 0x00007f540509e7bb in _pt_root () from /lib64/libnspr4.so #5 0x00007f5404a3fdc5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f540476d28d in clone () from /lib64/libc.so.6 Thread 39 (Thread 0x7f53eab21700 (LWP 128206)): #0 0x00007f5404a436d5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f5405099050 in PR_WaitCondVar () from /lib64/libnspr4.so #2 0x00007f5406ede198 in slapi_wait_condvar () from /usr/lib64/dirsrv/libslapd.so.0 #3 0x00007f53f8bdbeed in roles_cache_wait_on_change () from /usr/lib64/dirsrv/plugins/libroles-plugin.so #4 0x00007f540509e7bb in _pt_root () from /lib64/libnspr4.so #5 0x00007f5404a3fdc5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f540476d28d in clone () from /lib64/libc.so.6 Thread 38 (Thread 0x7f53ea320700 (LWP 128207)): #0 0x00007f5404a436d5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f5405099050 in PR_WaitCondVar () from /lib64/libnspr4.so #2 0x00007f5406ede198 in slapi_wait_condvar () from /usr/lib64/dirsrv/libslapd.so.0 #3 0x00007f53f8bdbeed in roles_cache_wait_on_change () from /usr/lib64/dirsrv/plugins/libroles-plugin.so #4 0x00007f540509e7bb in _pt_root () from /lib64/libnspr4.so #5 0x00007f5404a3fdc5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f540476d28d in clone () from /lib64/libc.so.6 Thread 37 (Thread 0x7f53e9b1f700 (LWP 128208)): #0 0x00007f5404a436d5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f5405099050 in PR_WaitCondVar () from /lib64/libnspr4.so #2 0x00007f5406ede198 in slapi_wait_condvar () from /usr/lib64/dirsrv/libslapd.so.0 #3 0x00007f53f8bdbeed in roles_cache_wait_on_change () from /usr/lib64/dirsrv/plugins/libroles-plugin.so #4 0x00007f540509e7bb in _pt_root () from /lib64/libnspr4.so #5 0x00007f5404a3fdc5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f540476d28d in clone () from /lib64/libc.so.6 Thread 36 (Thread 0x7f53e931e700 (LWP 128209)): #0 0x00007f5404a43a82 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f5405098b07 in pt_TimedWait () from /lib64/libnspr4.so #2 0x00007f5405098fce in PR_WaitCondVar () from /lib64/libnspr4.so #3 0x00007f5407397d03 in housecleaning () #4 0x00007f540509e7bb in _pt_root () from /lib64/libnspr4.so #5 0x00007f5404a3fdc5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f540476d28d in clone () from /lib64/libc.so.6 Thread 35 (Thread 0x7f53e8b1d700 (LWP 128210)): #0 0x00007f5404a43a82 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f5405098b07 in pt_TimedWait () from /lib64/libnspr4.so #2 0x00007f5405098fce in PR_WaitCondVar () from /lib64/libnspr4.so #3 0x00007f5406e841e8 in eq_loop () from /usr/lib64/dirsrv/libslapd.so.0 #4 0x00007f540509e7bb in _pt_root () from /lib64/libnspr4.so #5 0x00007f5404a3fdc5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f540476d28d in clone () from /lib64/libc.so.6 Thread 34 (Thread 0x7f53db7fe700 (LWP 128212)): #0 0x00007f5404a436d5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f53fef9f2f3 in __db_hybrid_mutex_suspend () from /lib64/ libdb-5.3.so #2 0x00007f53fef9e640 in __db_tas_mutex_lock () from /lib64/libdb-5.3.so #3 0x00007f53ff048d3a in __lock_get_internal () from /lib64/libdb-5.3.so #4 0x00007f53ff049820 in __lock_get () from /lib64/libdb-5.3.so #5 0x00007f53ff075142 in __db_lget () from /lib64/libdb-5.3.so #6 0x00007f53fefbc605 in __bam_search () from /lib64/libdb-5.3.so #7 0x00007f53fefa7256 in __bamc_search () from /lib64/libdb-5.3.so #8 0x00007f53fefa8d0f in __bamc_get () from /lib64/libdb-5.3.so #9 0x00007f53ff061ca6 in __dbc_iget () from /lib64/libdb-5.3.so #10 0x00007f53ff070b02 in __dbc_get_pp () from /lib64/libdb-5.3.so #11 0x00007f53fa54c680 in idl_new_fetch () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #12 0x00007f53fa55ab26 in index_read_ext_allids () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #13 0x00007f53fa545284 in keys2idl () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #14 0x00007f53fa5459e3 in ava_candidates.isra.0 () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #15 0x00007f53fa545fd2 in filter_candidates_ext () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #16 0x00007f53fa547096 in list_candidates () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #17 0x00007f53fa545f40 in filter_candidates_ext () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #18 0x00007f53fa58180c in subtree_candidates () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #19 0x00007f53fa582e7f in ldbm_back_search () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #20 0x00007f5406eb01d6 in op_shared_search () from /usr/lib64/dirsrv/libslapd.so.0 #21 0x00007f54073a133e in do_search () #22 0x00007f540738f567 in connection_threadmain () #23 0x00007f540509e7bb in _pt_root () from /lib64/libnspr4.so #24 0x00007f5404a3fdc5 in start_thread () from /lib64/libpthread.so.0 #25 0x00007f540476d28d in clone () from /lib64/libc.so.6 Thread 33 (Thread 0x7f53daffd700 (LWP 128213)): #0 0x00007f5404a436d5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f53fef9f2f3 in __db_hybrid_mutex_suspend () from /lib64/ libdb-5.3.so #2 0x00007f53fef9e640 in __db_tas_mutex_lock () from /lib64/libdb-5.3.so #3 0x00007f53ff048d3a in __lock_get_internal () from /lib64/libdb-5.3.so #4 0x00007f53ff049820 in __lock_get () from /lib64/libdb-5.3.so #5 0x00007f53ff075142 in __db_lget () from /lib64/libdb-5.3.so #6 0x00007f53fefbc605 in __bam_search () from /lib64/libdb-5.3.so #7 0x00007f53fefa7256 in __bamc_search () from /lib64/libdb-5.3.so #8 0x00007f53fefa8d0f in __bamc_get () from /lib64/libdb-5.3.so #9 0x00007f53ff061ca6 in __dbc_iget () from /lib64/libdb-5.3.so #10 0x00007f53ff070b02 in __dbc_get_pp () from /lib64/libdb-5.3.so #11 0x00007f53fa54c680 in idl_new_fetch () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #12 0x00007f53fa55ab26 in index_read_ext_allids () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #13 0x00007f53fa545284 in keys2idl () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #14 0x00007f53fa5459e3 in ava_candidates.isra.0 () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #15 0x00007f53fa545fd2 in filter_candidates_ext () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #16 0x00007f53fa547096 in list_candidates () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #17 0x00007f53fa545f40 in filter_candidates_ext () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #18 0x00007f53fa58180c in subtree_candidates () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #19 0x00007f53fa582e7f in ldbm_back_search () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #20 0x00007f5406eb01d6 in op_shared_search () from /usr/lib64/dirsrv/libslapd.so.0 #21 0x00007f54073a133e in do_search () #22 0x00007f540738f567 in connection_threadmain () #23 0x00007f540509e7bb in _pt_root () from /lib64/libnspr4.so #24 0x00007f5404a3fdc5 in start_thread () from /lib64/libpthread.so.0 #25 0x00007f540476d28d in clone () from /lib64/libc.so.6 Thread 32 (Thread 0x7f53da7fc700 (LWP 128214)): #0 0x00007f5404a436d5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f53fef9f2f3 in __db_hybrid_mutex_suspend () from /lib64/ libdb-5.3.so #2 0x00007f53fef9e640 in __db_tas_mutex_lock () from /lib64/libdb-5.3.so #3 0x00007f53ff048d3a in __lock_get_internal () from /lib64/libdb-5.3.so #4 0x00007f53ff049820 in __lock_get () from /lib64/libdb-5.3.so #5 0x00007f53ff075142 in __db_lget () from /lib64/libdb-5.3.so #6 0x00007f53fefbc605 in __bam_search () from /lib64/libdb-5.3.so #7 0x00007f53fefa7256 in __bamc_search () from /lib64/libdb-5.3.so #8 0x00007f53fefa8d0f in __bamc_get () from /lib64/libdb-5.3.so #9 0x00007f53ff061ca6 in __dbc_iget () from /lib64/libdb-5.3.so #10 0x00007f53ff070b02 in __dbc_get_pp () from /lib64/libdb-5.3.so #11 0x00007f53fa54c680 in idl_new_fetch () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #12 0x00007f53fa55ab26 in index_read_ext_allids () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #13 0x00007f53fa545284 in keys2idl () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #14 0x00007f53fa5459e3 in ava_candidates.isra.0 () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #15 0x00007f53fa545fd2 in filter_candidates_ext () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #16 0x00007f53fa547096 in list_candidates () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #17 0x00007f53fa545f40 in filter_candidates_ext () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #18 0x00007f53fa547096 in list_candidates () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #19 0x00007f53fa545f40 in filter_candidates_ext () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #20 0x00007f53fa547096 in list_candidates () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #21 0x00007f53fa545f40 in filter_candidates_ext () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #22 0x00007f53fa58180c in subtree_candidates () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #23 0x00007f53fa582e7f in ldbm_back_search () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #24 0x00007f5406eb01d6 in op_shared_search () from /usr/lib64/dirsrv/libslapd.so.0 #25 0x00007f54073a133e in do_search () #26 0x00007f540738f3c3 in connection_threadmain () #27 0x00007f540509e7bb in _pt_root () from /lib64/libnspr4.so #28 0x00007f5404a3fdc5 in start_thread () from /lib64/libpthread.so.0 #29 0x00007f540476d28d in clone () from /lib64/libc.so.6 Thread 31 (Thread 0x7f53d9ffb700 (LWP 128215)): #0 0x00007f5404a436d5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f53fef9f2f3 in __db_hybrid_mutex_suspend () from /lib64/ libdb-5.3.so #2 0x00007f53fef9e640 in __db_tas_mutex_lock () from /lib64/libdb-5.3.so #3 0x00007f53ff048d3a in __lock_get_internal () from /lib64/libdb-5.3.so #4 0x00007f53ff049820 in __lock_get () from /lib64/libdb-5.3.so #5 0x00007f53ff075142 in __db_lget () from /lib64/libdb-5.3.so #6 0x00007f53fefbc605 in __bam_search () from /lib64/libdb-5.3.so #7 0x00007f53fefa7256 in __bamc_search () from /lib64/libdb-5.3.so #8 0x00007f53fefa8d0f in __bamc_get () from /lib64/libdb-5.3.so #9 0x00007f53ff061ca6 in __dbc_iget () from /lib64/libdb-5.3.so #10 0x00007f53ff070b02 in __dbc_get_pp () from /lib64/libdb-5.3.so #11 0x00007f53fa54c680 in idl_new_fetch () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #12 0x00007f53fa55ab26 in index_read_ext_allids () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #13 0x00007f53fa545284 in keys2idl () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #14 0x00007f53fa5459e3 in ava_candidates.isra.0 () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #15 0x00007f53fa545fd2 in filter_candidates_ext () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #16 0x00007f53fa547096 in list_candidates () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #17 0x00007f53fa545f40 in filter_candidates_ext () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #18 0x00007f53fa58180c in subtree_candidates () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #19 0x00007f53fa582e7f in ldbm_back_search () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #20 0x00007f5406eb01d6 in op_shared_search () from /usr/lib64/dirsrv/libslapd.so.0 #21 0x00007f54073a133e in do_search () #22 0x00007f540738f567 in connection_threadmain () #23 0x00007f540509e7bb in _pt_root () from /lib64/libnspr4.so #24 0x00007f5404a3fdc5 in start_thread () from /lib64/libpthread.so.0 #25 0x00007f540476d28d in clone () from /lib64/libc.so.6 Thread 30 (Thread 0x7f53d97fa700 (LWP 128216)): #0 0x00007f5404a436d5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f53fef9f2f3 in __db_hybrid_mutex_suspend () from /lib64/ libdb-5.3.so #2 0x00007f53fef9e640 in __db_tas_mutex_lock () from /lib64/libdb-5.3.so #3 0x00007f53ff048d3a in __lock_get_internal () from /lib64/libdb-5.3.so #4 0x00007f53ff049820 in __lock_get () from /lib64/libdb-5.3.so #5 0x00007f53ff075142 in __db_lget () from /lib64/libdb-5.3.so #6 0x00007f53fefbc605 in __bam_search () from /lib64/libdb-5.3.so #7 0x00007f53fefa7256 in __bamc_search () from /lib64/libdb-5.3.so #8 0x00007f53fefa8d0f in __bamc_get () from /lib64/libdb-5.3.so #9 0x00007f53ff061ca6 in __dbc_iget () from /lib64/libdb-5.3.so #10 0x00007f53ff070b02 in __dbc_get_pp () from /lib64/libdb-5.3.so #11 0x00007f53fa54c680 in idl_new_fetch () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #12 0x00007f53fa55ab26 in index_read_ext_allids () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #13 0x00007f53fa545284 in keys2idl () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #14 0x00007f53fa5459e3 in ava_candidates.isra.0 () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #15 0x00007f53fa545fd2 in filter_candidates_ext () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #16 0x00007f53fa547096 in list_candidates () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #17 0x00007f53fa545f40 in filter_candidates_ext () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #18 0x00007f53fa58180c in subtree_candidates () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #19 0x00007f53fa582e7f in ldbm_back_search () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #20 0x00007f5406eb01d6 in op_shared_search () from /usr/lib64/dirsrv/libslapd.so.0 #21 0x00007f54073a133e in do_search () #22 0x00007f540738f567 in connection_threadmain () #23 0x00007f540509e7bb in _pt_root () from /lib64/libnspr4.so #24 0x00007f5404a3fdc5 in start_thread () from /lib64/libpthread.so.0 #25 0x00007f540476d28d in clone () from /lib64/libc.so.6 Thread 29 (Thread 0x7f53d8ff9700 (LWP 128217)): #0 0x00007f5404a436d5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f53fef9f2f3 in __db_hybrid_mutex_suspend () from /lib64/ libdb-5.3.so #2 0x00007f53fef9e640 in __db_tas_mutex_lock () from /lib64/libdb-5.3.so #3 0x00007f53ff048d3a in __lock_get_internal () from /lib64/libdb-5.3.so #4 0x00007f53ff049820 in __lock_get () from /lib64/libdb-5.3.so #5 0x00007f53ff075142 in __db_lget () from /lib64/libdb-5.3.so #6 0x00007f53fefbc605 in __bam_search () from /lib64/libdb-5.3.so #7 0x00007f53fefa7256 in __bamc_search () from /lib64/libdb-5.3.so #8 0x00007f53fefa8d0f in __bamc_get () from /lib64/libdb-5.3.so #9 0x00007f53ff061ca6 in __dbc_iget () from /lib64/libdb-5.3.so #10 0x00007f53ff070b02 in __dbc_get_pp () from /lib64/libdb-5.3.so #11 0x00007f53fa54c680 in idl_new_fetch () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #12 0x00007f53fa55ab26 in index_read_ext_allids () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #13 0x00007f53fa545284 in keys2idl () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #14 0x00007f53fa5459e3 in ava_candidates.isra.0 () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #15 0x00007f53fa545fd2 in filter_candidates_ext () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #16 0x00007f53fa547096 in list_candidates () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #17 0x00007f53fa545f40 in filter_candidates_ext () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #18 0x00007f53fa58180c in subtree_candidates () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #19 0x00007f53fa582e7f in ldbm_back_search () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #20 0x00007f5406eb01d6 in op_shared_search () from /usr/lib64/dirsrv/libslapd.so.0 #21 0x00007f54073a133e in do_search () #22 0x00007f540738f567 in connection_threadmain () #23 0x00007f540509e7bb in _pt_root () from /lib64/libnspr4.so #24 0x00007f5404a3fdc5 in start_thread () from /lib64/libpthread.so.0 #25 0x00007f540476d28d in clone () from /lib64/libc.so.6 Thread 28 (Thread 0x7f53d87f8700 (LWP 128218)): #0 0x00007f5404a436d5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f53fef9f2f3 in __db_hybrid_mutex_suspend () from /lib64/ libdb-5.3.so #2 0x00007f53fef9e640 in __db_tas_mutex_lock () from /lib64/libdb-5.3.so #3 0x00007f53ff048d3a in __lock_get_internal () from /lib64/libdb-5.3.so #4 0x00007f53ff049820 in __lock_get () from /lib64/libdb-5.3.so #5 0x00007f53ff075142 in __db_lget () from /lib64/libdb-5.3.so #6 0x00007f53fefbc605 in __bam_search () from /lib64/libdb-5.3.so #7 0x00007f53fefa7256 in __bamc_search () from /lib64/libdb-5.3.so #8 0x00007f53fefa8d0f in __bamc_get () from /lib64/libdb-5.3.so #9 0x00007f53ff061ca6 in __dbc_iget () from /lib64/libdb-5.3.so #10 0x00007f53ff070b02 in __dbc_get_pp () from /lib64/libdb-5.3.so #11 0x00007f53fa54c680 in idl_new_fetch () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #12 0x00007f53fa55ab26 in index_read_ext_allids () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #13 0x00007f53fa545284 in keys2idl () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #14 0x00007f53fa5459e3 in ava_candidates.isra.0 () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #15 0x00007f53fa545fd2 in filter_candidates_ext () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #16 0x00007f53fa547096 in list_candidates () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #17 0x00007f53fa545f40 in filter_candidates_ext () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #18 0x00007f53fa58180c in subtree_candidates () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #19 0x00007f53fa582e7f in ldbm_back_search () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #20 0x00007f5406eb01d6 in op_shared_search () from /usr/lib64/dirsrv/libslapd.so.0 #21 0x00007f54073a133e in do_search () #22 0x00007f540738f567 in connection_threadmain () #23 0x00007f540509e7bb in _pt_root () from /lib64/libnspr4.so #24 0x00007f5404a3fdc5 in start_thread () from /lib64/libpthread.so.0 #25 0x00007f540476d28d in clone () from /lib64/libc.so.6 Thread 27 (Thread 0x7f53d7ff7700 (LWP 128219)): #0 0x00007f5404a436d5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f53fef9f2f3 in __db_hybrid_mutex_suspend () from /lib64/ libdb-5.3.so #2 0x00007f53fef9e640 in __db_tas_mutex_lock () from /lib64/libdb-5.3.so #3 0x00007f53ff048d3a in __lock_get_internal () from /lib64/libdb-5.3.so #4 0x00007f53ff049820 in __lock_get () from /lib64/libdb-5.3.so #5 0x00007f53ff075142 in __db_lget () from /lib64/libdb-5.3.so #6 0x00007f53fefbc605 in __bam_search () from /lib64/libdb-5.3.so #7 0x00007f53fefa7256 in __bamc_search () from /lib64/libdb-5.3.so #8 0x00007f53fefa8d0f in __bamc_get () from /lib64/libdb-5.3.so #9 0x00007f53ff061ca6 in __dbc_iget () from /lib64/libdb-5.3.so #10 0x00007f53ff070b02 in __dbc_get_pp () from /lib64/libdb-5.3.so #11 0x00007f53fa54c680 in idl_new_fetch () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #12 0x00007f53fa55ab26 in index_read_ext_allids () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #13 0x00007f53fa545284 in keys2idl () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #14 0x00007f53fa5459e3 in ava_candidates.isra.0 () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #15 0x00007f53fa545fd2 in filter_candidates_ext () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #16 0x00007f53fa547096 in list_candidates () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #17 0x00007f53fa545f40 in filter_candidates_ext () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #18 0x00007f53fa58180c in subtree_candidates () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #19 0x00007f53fa582e7f in ldbm_back_search () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #20 0x00007f5406eb01d6 in op_shared_search () from /usr/lib64/dirsrv/libslapd.so.0 #21 0x00007f54073a133e in do_search () #22 0x00007f540738f567 in connection_threadmain () #23 0x00007f540509e7bb in _pt_root () from /lib64/libnspr4.so #24 0x00007f5404a3fdc5 in start_thread () from /lib64/libpthread.so.0 #25 0x00007f540476d28d in clone () from /lib64/libc.so.6 Thread 26 (Thread 0x7f53d77f6700 (LWP 128220)): #0 0x00007f5404a436d5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f53fef9f2f3 in __db_hybrid_mutex_suspend () from /lib64/ libdb-5.3.so #2 0x00007f53fef9e640 in __db_tas_mutex_lock () from /lib64/libdb-5.3.so #3 0x00007f53ff048d3a in __lock_get_internal () from /lib64/libdb-5.3.so #4 0x00007f53ff049820 in __lock_get () from /lib64/libdb-5.3.so #5 0x00007f53ff075142 in __db_lget () from /lib64/libdb-5.3.so #6 0x00007f53fefbc605 in __bam_search () from /lib64/libdb-5.3.so #7 0x00007f53fefa7256 in __bamc_search () from /lib64/libdb-5.3.so #8 0x00007f53fefa8d0f in __bamc_get () from /lib64/libdb-5.3.so #9 0x00007f53ff061ca6 in __dbc_iget () from /lib64/libdb-5.3.so #10 0x00007f53ff070b02 in __dbc_get_pp () from /lib64/libdb-5.3.so #11 0x00007f53fa54c680 in idl_new_fetch () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #12 0x00007f53fa55ab26 in index_read_ext_allids () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #13 0x00007f53fa545284 in keys2idl () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #14 0x00007f53fa5459e3 in ava_candidates.isra.0 () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #15 0x00007f53fa545fd2 in filter_candidates_ext () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #16 0x00007f53fa547096 in list_candidates () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #17 0x00007f53fa545f40 in filter_candidates_ext () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #18 0x00007f53fa58180c in subtree_candidates () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #19 0x00007f53fa582e7f in ldbm_back_search () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #20 0x00007f5406eb01d6 in op_shared_search () from /usr/lib64/dirsrv/libslapd.so.0 #21 0x00007f54073a133e in do_search () #22 0x00007f540738f567 in connection_threadmain () #23 0x00007f540509e7bb in _pt_root () from /lib64/libnspr4.so #24 0x00007f5404a3fdc5 in start_thread () from /lib64/libpthread.so.0 #25 0x00007f540476d28d in clone () from /lib64/libc.so.6 Thread 25 (Thread 0x7f53d6ff5700 (LWP 128221)): #0 0x00007f5404a436d5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f53fef9f2f3 in __db_hybrid_mutex_suspend () from /lib64/ libdb-5.3.so #2 0x00007f53fef9e640 in __db_tas_mutex_lock () from /lib64/libdb-5.3.so #3 0x00007f53ff048d3a in __lock_get_internal () from /lib64/libdb-5.3.so #4 0x00007f53ff049820 in __lock_get () from /lib64/libdb-5.3.so #5 0x00007f53ff075142 in __db_lget () from /lib64/libdb-5.3.so #6 0x00007f53fefbc605 in __bam_search () from /lib64/libdb-5.3.so #7 0x00007f53fefa7256 in __bamc_search () from /lib64/libdb-5.3.so #8 0x00007f53fefa8d0f in __bamc_get () from /lib64/libdb-5.3.so #9 0x00007f53ff061ca6 in __dbc_iget () from /lib64/libdb-5.3.so #10 0x00007f53ff070b02 in __dbc_get_pp () from /lib64/libdb-5.3.so #11 0x00007f53fa54c680 in idl_new_fetch () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #12 0x00007f53fa55ab26 in index_read_ext_allids () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #13 0x00007f53fa545284 in keys2idl () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #14 0x00007f53fa5459e3 in ava_candidates.isra.0 () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #15 0x00007f53fa545fd2 in filter_candidates_ext () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #16 0x00007f53fa547096 in list_candidates () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #17 0x00007f53fa545f40 in filter_candidates_ext () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #18 0x00007f53fa58180c in subtree_candidates () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #19 0x00007f53fa582e7f in ldbm_back_search () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #20 0x00007f5406eb01d6 in op_shared_search () from /usr/lib64/dirsrv/libslapd.so.0 #21 0x00007f54073a133e in do_search () #22 0x00007f540738f567 in connection_threadmain () #23 0x00007f540509e7bb in _pt_root () from /lib64/libnspr4.so #24 0x00007f5404a3fdc5 in start_thread () from /lib64/libpthread.so.0 #25 0x00007f540476d28d in clone () from /lib64/libc.so.6 Thread 24 (Thread 0x7f53d67f4700 (LWP 128222)): #0 0x00007f5404a436d5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f53fef9f2f3 in __db_hybrid_mutex_suspend () from /lib64/ libdb-5.3.so #2 0x00007f53fef9e640 in __db_tas_mutex_lock () from /lib64/libdb-5.3.so #3 0x00007f53ff048d3a in __lock_get_internal () from /lib64/libdb-5.3.so #4 0x00007f53ff049820 in __lock_get () from /lib64/libdb-5.3.so #5 0x00007f53ff075142 in __db_lget () from /lib64/libdb-5.3.so #6 0x00007f53fefbc605 in __bam_search () from /lib64/libdb-5.3.so #7 0x00007f53fefa7256 in __bamc_search () from /lib64/libdb-5.3.so #8 0x00007f53fefa8d0f in __bamc_get () from /lib64/libdb-5.3.so #9 0x00007f53ff061ca6 in __dbc_iget () from /lib64/libdb-5.3.so #10 0x00007f53ff070b02 in __dbc_get_pp () from /lib64/libdb-5.3.so #11 0x00007f53fa54c680 in idl_new_fetch () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #12 0x00007f53fa55ab26 in index_read_ext_allids () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #13 0x00007f53fa545284 in keys2idl () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #14 0x00007f53fa5459e3 in ava_candidates.isra.0 () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #15 0x00007f53fa545fd2 in filter_candidates_ext () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #16 0x00007f53fa547096 in list_candidates () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #17 0x00007f53fa545f40 in filter_candidates_ext () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #18 0x00007f53fa58180c in subtree_candidates () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #19 0x00007f53fa582e7f in ldbm_back_search () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #20 0x00007f5406eb01d6 in op_shared_search () from /usr/lib64/dirsrv/libslapd.so.0 #21 0x00007f54073a133e in do_search () #22 0x00007f540738f567 in connection_threadmain () #23 0x00007f540509e7bb in _pt_root () from /lib64/libnspr4.so #24 0x00007f5404a3fdc5 in start_thread () from /lib64/libpthread.so.0 #25 0x00007f540476d28d in clone () from /lib64/libc.so.6 Thread 23 (Thread 0x7f53d5ff3700 (LWP 128223)): #0 0x00007f5404a436d5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f53fef9f2f3 in __db_hybrid_mutex_suspend () from /lib64/ libdb-5.3.so #2 0x00007f53fef9e640 in __db_tas_mutex_lock () from /lib64/libdb-5.3.so #3 0x00007f53ff0b8e1b in __memp_fget () from /lib64/libdb-5.3.so #4 0x00007f53fefbc211 in __bam_search () from /lib64/libdb-5.3.so #5 0x00007f53fefa7256 in __bamc_search () from /lib64/libdb-5.3.so #6 0x00007f53fefab6c4 in __bamc_put () from /lib64/libdb-5.3.so #7 0x00007f53ff0626d5 in __dbc_iput () from /lib64/libdb-5.3.so #8 0x00007f53ff05da0e in __db_put () from /lib64/libdb-5.3.so #9 0x00007f53ff072fa4 in __db_put_pp () from /lib64/libdb-5.3.so #10 0x00007f53fa54d4cf in idl_new_insert_key () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #11 0x00007f53fa55c5b5 in addordel_values_sv.isra.3 () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #12 0x00007f53fa55d25a in index_addordel_values_ext_sv () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #13 0x00007f53fa55d694 in index_addordel_values_sv () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #14 0x00007f53fa55d75a in index_addordel_entry () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #15 0x00007f53fa5605fa in ldbm_back_add () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #16 0x00007f5406e62cca in op_shared_add () from /usr/lib64/dirsrv/libslapd.so.0 #17 0x00007f5406e64020 in do_add () from /usr/lib64/dirsrv/libslapd.so.0 #18 0x00007f540738f303 in connection_threadmain () #19 0x00007f540509e7bb in _pt_root () from /lib64/libnspr4.so #20 0x00007f5404a3fdc5 in start_thread () from /lib64/libpthread.so.0 #21 0x00007f540476d28d in clone () from /lib64/libc.so.6 Thread 22 (Thread 0x7f53d57f2700 (LWP 128224)): #0 0x00007f5404a436d5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f53fef9f2f3 in __db_hybrid_mutex_suspend () from /lib64/ libdb-5.3.so #2 0x00007f53fef9e640 in __db_tas_mutex_lock () from /lib64/libdb-5.3.so #3 0x00007f53ff048d3a in __lock_get_internal () from /lib64/libdb-5.3.so #4 0x00007f53ff049820 in __lock_get () from /lib64/libdb-5.3.so #5 0x00007f53ff075142 in __db_lget () from /lib64/libdb-5.3.so #6 0x00007f53fefbc605 in __bam_search () from /lib64/libdb-5.3.so #7 0x00007f53fefa7256 in __bamc_search () from /lib64/libdb-5.3.so #8 0x00007f53fefa8d0f in __bamc_get () from /lib64/libdb-5.3.so #9 0x00007f53ff061ca6 in __dbc_iget () from /lib64/libdb-5.3.so #10 0x00007f53ff070b02 in __dbc_get_pp () from /lib64/libdb-5.3.so #11 0x00007f53fa54c680 in idl_new_fetch () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #12 0x00007f53fa55ab26 in index_read_ext_allids () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #13 0x00007f53fa545284 in keys2idl () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #14 0x00007f53fa5459e3 in ava_candidates.isra.0 () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #15 0x00007f53fa545fd2 in filter_candidates_ext () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #16 0x00007f53fa547096 in list_candidates () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #17 0x00007f53fa545f40 in filter_candidates_ext () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #18 0x00007f53fa58180c in subtree_candidates () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #19 0x00007f53fa582e7f in ldbm_back_search () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #20 0x00007f5406eb01d6 in op_shared_search () from /usr/lib64/dirsrv/libslapd.so.0 #21 0x00007f54073a133e in do_search () #22 0x00007f540738f567 in connection_threadmain () #23 0x00007f540509e7bb in _pt_root () from /lib64/libnspr4.so #24 0x00007f5404a3fdc5 in start_thread () from /lib64/libpthread.so.0 #25 0x00007f540476d28d in clone () from /lib64/libc.so.6 Thread 21 (Thread 0x7f53d4ff1700 (LWP 128225)): #0 0x00007f5404a436d5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f53fef9f2f3 in __db_hybrid_mutex_suspend () from /lib64/ libdb-5.3.so #2 0x00007f53fef9e640 in __db_tas_mutex_lock () from /lib64/libdb-5.3.so #3 0x00007f53ff048d3a in __lock_get_internal () from /lib64/libdb-5.3.so #4 0x00007f53ff049820 in __lock_get () from /lib64/libdb-5.3.so #5 0x00007f53ff075142 in __db_lget () from /lib64/libdb-5.3.so #6 0x00007f53fefbc605 in __bam_search () from /lib64/libdb-5.3.so #7 0x00007f53fefa7256 in __bamc_search () from /lib64/libdb-5.3.so #8 0x00007f53fefa8d0f in __bamc_get () from /lib64/libdb-5.3.so #9 0x00007f53ff061ca6 in __dbc_iget () from /lib64/libdb-5.3.so #10 0x00007f53ff070b02 in __dbc_get_pp () from /lib64/libdb-5.3.so #11 0x00007f53fa54c680 in idl_new_fetch () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #12 0x00007f53fa55ab26 in index_read_ext_allids () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #13 0x00007f53fa545284 in keys2idl () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #14 0x00007f53fa5459e3 in ava_candidates.isra.0 () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #15 0x00007f53fa545fd2 in filter_candidates_ext () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #16 0x00007f53fa547096 in list_candidates () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #17 0x00007f53fa545f40 in filter_candidates_ext () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #18 0x00007f53fa58180c in subtree_candidates () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #19 0x00007f53fa582e7f in ldbm_back_search () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #20 0x00007f5406eb01d6 in op_shared_search () from /usr/lib64/dirsrv/libslapd.so.0 #21 0x00007f54073a133e in do_search () #22 0x00007f540738f567 in connection_threadmain () #23 0x00007f540509e7bb in _pt_root () from /lib64/libnspr4.so #24 0x00007f5404a3fdc5 in start_thread () from /lib64/libpthread.so.0 #25 0x00007f540476d28d in clone () from /lib64/libc.so.6 Thread 20 (Thread 0x7f53d47f0700 (LWP 128226)): #0 0x00007f5404a436d5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f53fef9f2f3 in __db_hybrid_mutex_suspend () from /lib64/ libdb-5.3.so #2 0x00007f53fef9e640 in __db_tas_mutex_lock () from /lib64/libdb-5.3.so #3 0x00007f53ff048d3a in __lock_get_internal () from /lib64/libdb-5.3.so #4 0x00007f53ff049820 in __lock_get () from /lib64/libdb-5.3.so #5 0x00007f53ff075142 in __db_lget () from /lib64/libdb-5.3.so #6 0x00007f53fefbc605 in __bam_search () from /lib64/libdb-5.3.so #7 0x00007f53fefa7256 in __bamc_search () from /lib64/libdb-5.3.so #8 0x00007f53fefa8d0f in __bamc_get () from /lib64/libdb-5.3.so #9 0x00007f53ff061ca6 in __dbc_iget () from /lib64/libdb-5.3.so #10 0x00007f53ff070b02 in __dbc_get_pp () from /lib64/libdb-5.3.so #11 0x00007f53fa54c680 in idl_new_fetch () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #12 0x00007f53fa55ab26 in index_read_ext_allids () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #13 0x00007f53fa545284 in keys2idl () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #14 0x00007f53fa5459e3 in ava_candidates.isra.0 () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #15 0x00007f53fa545fd2 in filter_candidates_ext () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #16 0x00007f53fa547096 in list_candidates () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #17 0x00007f53fa545f40 in filter_candidates_ext () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #18 0x00007f53fa58180c in subtree_candidates () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #19 0x00007f53fa582e7f in ldbm_back_search () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #20 0x00007f5406eb01d6 in op_shared_search () from /usr/lib64/dirsrv/libslapd.so.0 #21 0x00007f5406ec094e in search_internal_callback_pb () from /usr/lib64/dirsrv/libslapd.so.0 #22 0x00007f5406ec0be8 in search_internal_pb () from /usr/lib64/dirsrv/libslapd.so.0 #23 0x00007f53fe118e90 in search_one_berval () from /usr/lib64/dirsrv/plugins/libattr-unique-plugin.so #24 0x00007f53fe119242 in search () from /usr/lib64/dirsrv/plugins/libattr-unique-plugin.so #25 0x00007f53fe11954c in searchAllSubtrees () from /usr/lib64/dirsrv/plugins/libattr-unique-plugin.so #26 0x00007f53fe11a015 in preop_add () from /usr/lib64/dirsrv/plugins/libattr-unique-plugin.so #27 0x00007f5406ebb848 in plugin_call_func () from /usr/lib64/dirsrv/libslapd.so.0 #28 0x00007f5406ebbb13 in plugin_call_plugins () from /usr/lib64/dirsrv/libslapd.so.0 #29 0x00007f5406e62c2b in op_shared_add () from /usr/lib64/dirsrv/libslapd.so.0 #30 0x00007f5406e64020 in do_add () from /usr/lib64/dirsrv/libslapd.so.0 #31 0x00007f540738f303 in connection_threadmain () #32 0x00007f540509e7bb in _pt_root () from /lib64/libnspr4.so #33 0x00007f5404a3fdc5 in start_thread () from /lib64/libpthread.so.0 #34 0x00007f540476d28d in clone () from /lib64/libc.so.6 Thread 19 (Thread 0x7f53d3fef700 (LWP 128227)): #0 0x00007f5404a436d5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f53fef9f2f3 in __db_hybrid_mutex_suspend () from /lib64/ libdb-5.3.so #2 0x00007f53fef9e640 in __db_tas_mutex_lock () from /lib64/libdb-5.3.so #3 0x00007f53ff048d3a in __lock_get_internal () from /lib64/libdb-5.3.so #4 0x00007f53ff049820 in __lock_get () from /lib64/libdb-5.3.so #5 0x00007f53ff075142 in __db_lget () from /lib64/libdb-5.3.so #6 0x00007f53fefbc605 in __bam_search () from /lib64/libdb-5.3.so #7 0x00007f53fefa7256 in __bamc_search () from /lib64/libdb-5.3.so #8 0x00007f53fefa8d0f in __bamc_get () from /lib64/libdb-5.3.so #9 0x00007f53ff061ca6 in __dbc_iget () from /lib64/libdb-5.3.so #10 0x00007f53ff070b02 in __dbc_get_pp () from /lib64/libdb-5.3.so #11 0x00007f53fa54c680 in idl_new_fetch () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #12 0x00007f53fa55ab26 in index_read_ext_allids () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #13 0x00007f53fa545284 in keys2idl () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #14 0x00007f53fa5459e3 in ava_candidates.isra.0 () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #15 0x00007f53fa545fd2 in filter_candidates_ext () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #16 0x00007f53fa547096 in list_candidates () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #17 0x00007f53fa545f40 in filter_candidates_ext () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #18 0x00007f53fa58180c in subtree_candidates () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #19 0x00007f53fa582e7f in ldbm_back_search () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #20 0x00007f5406eb01d6 in op_shared_search () from /usr/lib64/dirsrv/libslapd.so.0 #21 0x00007f54073a133e in do_search () #22 0x00007f540738f567 in connection_threadmain () #23 0x00007f540509e7bb in _pt_root () from /lib64/libnspr4.so #24 0x00007f5404a3fdc5 in start_thread () from /lib64/libpthread.so.0 #25 0x00007f540476d28d in clone () from /lib64/libc.so.6 Thread 18 (Thread 0x7f53d37ee700 (LWP 128228)): #0 0x00007f5404a436d5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f53fef9f2f3 in __db_hybrid_mutex_suspend () from /lib64/ libdb-5.3.so #2 0x00007f53fef9e640 in __db_tas_mutex_lock () from /lib64/libdb-5.3.so #3 0x00007f53ff048d3a in __lock_get_internal () from /lib64/libdb-5.3.so #4 0x00007f53ff049820 in __lock_get () from /lib64/libdb-5.3.so #5 0x00007f53ff075142 in __db_lget () from /lib64/libdb-5.3.so #6 0x00007f53fefbc605 in __bam_search () from /lib64/libdb-5.3.so #7 0x00007f53fefa7256 in __bamc_search () from /lib64/libdb-5.3.so #8 0x00007f53fefa8d0f in __bamc_get () from /lib64/libdb-5.3.so #9 0x00007f53ff061ca6 in __dbc_iget () from /lib64/libdb-5.3.so #10 0x00007f53ff070b02 in __dbc_get_pp () from /lib64/libdb-5.3.so #11 0x00007f53fa54c680 in idl_new_fetch () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #12 0x00007f53fa55ab26 in index_read_ext_allids () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #13 0x00007f53fa545284 in keys2idl () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #14 0x00007f53fa5459e3 in ava_candidates.isra.0 () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #15 0x00007f53fa545fd2 in filter_candidates_ext () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #16 0x00007f53fa547096 in list_candidates () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #17 0x00007f53fa545f40 in filter_candidates_ext () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #18 0x00007f53fa58180c in subtree_candidates () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #19 0x00007f53fa582e7f in ldbm_back_search () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #20 0x00007f5406eb01d6 in op_shared_search () from /usr/lib64/dirsrv/libslapd.so.0 #21 0x00007f54073a133e in do_search () #22 0x00007f540738f567 in connection_threadmain () #23 0x00007f540509e7bb in _pt_root () from /lib64/libnspr4.so #24 0x00007f5404a3fdc5 in start_thread () from /lib64/libpthread.so.0 #25 0x00007f540476d28d in clone () from /lib64/libc.so.6 Thread 17 (Thread 0x7f53d2fed700 (LWP 128229)): #0 0x00007f5404a436d5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f53fef9f2f3 in __db_hybrid_mutex_suspend () from /lib64/ libdb-5.3.so #2 0x00007f53fef9e640 in __db_tas_mutex_lock () from /lib64/libdb-5.3.so #3 0x00007f53ff048d3a in __lock_get_internal () from /lib64/libdb-5.3.so #4 0x00007f53ff049820 in __lock_get () from /lib64/libdb-5.3.so #5 0x00007f53ff075142 in __db_lget () from /lib64/libdb-5.3.so #6 0x00007f53fefbc605 in __bam_search () from /lib64/libdb-5.3.so #7 0x00007f53fefa7256 in __bamc_search () from /lib64/libdb-5.3.so #8 0x00007f53fefa8d0f in __bamc_get () from /lib64/libdb-5.3.so #9 0x00007f53ff061ca6 in __dbc_iget () from /lib64/libdb-5.3.so #10 0x00007f53ff070b02 in __dbc_get_pp () from /lib64/libdb-5.3.so #11 0x00007f53fa54c680 in idl_new_fetch () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #12 0x00007f53fa55ab26 in index_read_ext_allids () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #13 0x00007f53fa545284 in keys2idl () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #14 0x00007f53fa5459e3 in ava_candidates.isra.0 () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #15 0x00007f53fa545fd2 in filter_candidates_ext () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #16 0x00007f53fa547096 in list_candidates () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #17 0x00007f53fa545f40 in filter_candidates_ext () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #18 0x00007f53fa58180c in subtree_candidates () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #19 0x00007f53fa582e7f in ldbm_back_search () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #20 0x00007f5406eb01d6 in op_shared_search () from /usr/lib64/dirsrv/libslapd.so.0 #21 0x00007f54073a133e in do_search () #22 0x00007f540738f567 in connection_threadmain () #23 0x00007f540509e7bb in _pt_root () from /lib64/libnspr4.so #24 0x00007f5404a3fdc5 in start_thread () from /lib64/libpthread.so.0 #25 0x00007f540476d28d in clone () from /lib64/libc.so.6 Thread 16 (Thread 0x7f53d27ec700 (LWP 128230)): #0 0x00007f5404a436d5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f53fef9f2f3 in __db_hybrid_mutex_suspend () from /lib64/ libdb-5.3.so #2 0x00007f53fef9e640 in __db_tas_mutex_lock () from /lib64/libdb-5.3.so #3 0x00007f53ff048d3a in __lock_get_internal () from /lib64/libdb-5.3.so #4 0x00007f53ff049820 in __lock_get () from /lib64/libdb-5.3.so #5 0x00007f53ff075142 in __db_lget () from /lib64/libdb-5.3.so #6 0x00007f53fefbc605 in __bam_search () from /lib64/libdb-5.3.so #7 0x00007f53fefa7256 in __bamc_search () from /lib64/libdb-5.3.so #8 0x00007f53fefa8d0f in __bamc_get () from /lib64/libdb-5.3.so #9 0x00007f53ff061ca6 in __dbc_iget () from /lib64/libdb-5.3.so #10 0x00007f53ff070b02 in __dbc_get_pp () from /lib64/libdb-5.3.so #11 0x00007f53fa54c680 in idl_new_fetch () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #12 0x00007f53fa55ab26 in index_read_ext_allids () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #13 0x00007f53fa545284 in keys2idl () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #14 0x00007f53fa5459e3 in ava_candidates.isra.0 () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #15 0x00007f53fa545fd2 in filter_candidates_ext () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #16 0x00007f53fa547096 in list_candidates () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #17 0x00007f53fa545f40 in filter_candidates_ext () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #18 0x00007f53fa58180c in subtree_candidates () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #19 0x00007f53fa582e7f in ldbm_back_search () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #20 0x00007f5406eb01d6 in op_shared_search () from /usr/lib64/dirsrv/libslapd.so.0 #21 0x00007f54073a133e in do_search () #22 0x00007f540738f567 in connection_threadmain () #23 0x00007f540509e7bb in _pt_root () from /lib64/libnspr4.so #24 0x00007f5404a3fdc5 in start_thread () from /lib64/libpthread.so.0 #25 0x00007f540476d28d in clone () from /lib64/libc.so.6 Thread 15 (Thread 0x7f53d1feb700 (LWP 128231)): #0 0x00007f5404a436d5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f53fef9f2f3 in __db_hybrid_mutex_suspend () from /lib64/ libdb-5.3.so #2 0x00007f53fef9e640 in __db_tas_mutex_lock () from /lib64/libdb-5.3.so #3 0x00007f53ff048d3a in __lock_get_internal () from /lib64/libdb-5.3.so #4 0x00007f53ff049820 in __lock_get () from /lib64/libdb-5.3.so #5 0x00007f53ff075142 in __db_lget () from /lib64/libdb-5.3.so #6 0x00007f53fefbc605 in __bam_search () from /lib64/libdb-5.3.so #7 0x00007f53fefa7256 in __bamc_search () from /lib64/libdb-5.3.so #8 0x00007f53fefa8d0f in __bamc_get () from /lib64/libdb-5.3.so #9 0x00007f53ff061ca6 in __dbc_iget () from /lib64/libdb-5.3.so #10 0x00007f53ff070b02 in __dbc_get_pp () from /lib64/libdb-5.3.so #11 0x00007f53fa54c680 in idl_new_fetch () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #12 0x00007f53fa55ab26 in index_read_ext_allids () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #13 0x00007f53fa545284 in keys2idl () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #14 0x00007f53fa5459e3 in ava_candidates.isra.0 () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #15 0x00007f53fa545fd2 in filter_candidates_ext () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #16 0x00007f53fa547096 in list_candidates () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #17 0x00007f53fa545f40 in filter_candidates_ext () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #18 0x00007f53fa58180c in subtree_candidates () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #19 0x00007f53fa582e7f in ldbm_back_search () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #20 0x00007f5406eb01d6 in op_shared_search () from /usr/lib64/dirsrv/libslapd.so.0 #21 0x00007f54073a133e in do_search () #22 0x00007f540738f567 in connection_threadmain () #23 0x00007f540509e7bb in _pt_root () from /lib64/libnspr4.so #24 0x00007f5404a3fdc5 in start_thread () from /lib64/libpthread.so.0 #25 0x00007f540476d28d in clone () from /lib64/libc.so.6 Thread 14 (Thread 0x7f53d17ea700 (LWP 128232)): #0 0x00007f5404a436d5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f53fef9f2f3 in __db_hybrid_mutex_suspend () from /lib64/ libdb-5.3.so #2 0x00007f53fef9e640 in __db_tas_mutex_lock () from /lib64/libdb-5.3.so #3 0x00007f53ff048d3a in __lock_get_internal () from /lib64/libdb-5.3.so #4 0x00007f53ff049820 in __lock_get () from /lib64/libdb-5.3.so #5 0x00007f53ff075142 in __db_lget () from /lib64/libdb-5.3.so #6 0x00007f53fefbc605 in __bam_search () from /lib64/libdb-5.3.so #7 0x00007f53fefa7256 in __bamc_search () from /lib64/libdb-5.3.so #8 0x00007f53fefa8d0f in __bamc_get () from /lib64/libdb-5.3.so #9 0x00007f53ff061ca6 in __dbc_iget () from /lib64/libdb-5.3.so #10 0x00007f53ff070b02 in __dbc_get_pp () from /lib64/libdb-5.3.so #11 0x00007f53fa54c680 in idl_new_fetch () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #12 0x00007f53fa55ab26 in index_read_ext_allids () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #13 0x00007f53fa545284 in keys2idl () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #14 0x00007f53fa5459e3 in ava_candidates.isra.0 () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #15 0x00007f53fa545fd2 in filter_candidates_ext () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #16 0x00007f53fa547096 in list_candidates () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #17 0x00007f53fa545f40 in filter_candidates_ext () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #18 0x00007f53fa547096 in list_candidates () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #19 0x00007f53fa545f40 in filter_candidates_ext () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #20 0x00007f53fa547096 in list_candidates () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #21 0x00007f53fa545f40 in filter_candidates_ext () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #22 0x00007f53fa58180c in subtree_candidates () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #23 0x00007f53fa582e7f in ldbm_back_search () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #24 0x00007f5406eb01d6 in op_shared_search () from /usr/lib64/dirsrv/libslapd.so.0 #25 0x00007f54073a133e in do_search () #26 0x00007f540738f3c3 in connection_threadmain () #27 0x00007f540509e7bb in _pt_root () from /lib64/libnspr4.so #28 0x00007f5404a3fdc5 in start_thread () from /lib64/libpthread.so.0 #29 0x00007f540476d28d in clone () from /lib64/libc.so.6 Thread 13 (Thread 0x7f53d0fe9700 (LWP 128233)): #0 0x00007f5404a436d5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f53fef9f2f3 in __db_hybrid_mutex_suspend () from /lib64/ libdb-5.3.so #2 0x00007f53fef9e640 in __db_tas_mutex_lock () from /lib64/libdb-5.3.so #3 0x00007f53ff048d3a in __lock_get_internal () from /lib64/libdb-5.3.so #4 0x00007f53ff049820 in __lock_get () from /lib64/libdb-5.3.so #5 0x00007f53ff075142 in __db_lget () from /lib64/libdb-5.3.so #6 0x00007f53fefbc605 in __bam_search () from /lib64/libdb-5.3.so #7 0x00007f53fefa7256 in __bamc_search () from /lib64/libdb-5.3.so #8 0x00007f53fefa8d0f in __bamc_get () from /lib64/libdb-5.3.so #9 0x00007f53ff061ca6 in __dbc_iget () from /lib64/libdb-5.3.so #10 0x00007f53ff070b02 in __dbc_get_pp () from /lib64/libdb-5.3.so #11 0x00007f53fa54c680 in idl_new_fetch () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #12 0x00007f53fa55ab26 in index_read_ext_allids () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #13 0x00007f53fa545284 in keys2idl () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #14 0x00007f53fa5459e3 in ava_candidates.isra.0 () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #15 0x00007f53fa545fd2 in filter_candidates_ext () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #16 0x00007f53fa547096 in list_candidates () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #17 0x00007f53fa545f40 in filter_candidates_ext () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #18 0x00007f53fa58180c in subtree_candidates () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #19 0x00007f53fa582e7f in ldbm_back_search () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #20 0x00007f5406eb01d6 in op_shared_search () from /usr/lib64/dirsrv/libslapd.so.0 #21 0x00007f54073a133e in do_search () #22 0x00007f540738f567 in connection_threadmain () #23 0x00007f540509e7bb in _pt_root () from /lib64/libnspr4.so #24 0x00007f5404a3fdc5 in start_thread () from /lib64/libpthread.so.0 #25 0x00007f540476d28d in clone () from /lib64/libc.so.6 Thread 12 (Thread 0x7f53d07e8700 (LWP 128234)): #0 0x00007f5404a436d5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f53fef9f2f3 in __db_hybrid_mutex_suspend () from /lib64/ libdb-5.3.so #2 0x00007f53fef9e640 in __db_tas_mutex_lock () from /lib64/libdb-5.3.so #3 0x00007f53ff048d3a in __lock_get_internal () from /lib64/libdb-5.3.so #4 0x00007f53ff049820 in __lock_get () from /lib64/libdb-5.3.so #5 0x00007f53ff075142 in __db_lget () from /lib64/libdb-5.3.so #6 0x00007f53fefbc605 in __bam_search () from /lib64/libdb-5.3.so #7 0x00007f53fefa7256 in __bamc_search () from /lib64/libdb-5.3.so #8 0x00007f53fefa8d0f in __bamc_get () from /lib64/libdb-5.3.so #9 0x00007f53ff061ca6 in __dbc_iget () from /lib64/libdb-5.3.so #10 0x00007f53ff070b02 in __dbc_get_pp () from /lib64/libdb-5.3.so #11 0x00007f53fa54c680 in idl_new_fetch () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #12 0x00007f53fa55ab26 in index_read_ext_allids () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #13 0x00007f53fa545284 in keys2idl () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #14 0x00007f53fa5459e3 in ava_candidates.isra.0 () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #15 0x00007f53fa545fd2 in filter_candidates_ext () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #16 0x00007f53fa547096 in list_candidates () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #17 0x00007f53fa545f40 in filter_candidates_ext () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #18 0x00007f53fa58180c in subtree_candidates () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #19 0x00007f53fa582e7f in ldbm_back_search () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #20 0x00007f5406eb01d6 in op_shared_search () from /usr/lib64/dirsrv/libslapd.so.0 #21 0x00007f54073a133e in do_search () #22 0x00007f540738f567 in connection_threadmain () #23 0x00007f540509e7bb in _pt_root () from /lib64/libnspr4.so #24 0x00007f5404a3fdc5 in start_thread () from /lib64/libpthread.so.0 #25 0x00007f540476d28d in clone () from /lib64/libc.so.6 Thread 11 (Thread 0x7f53cffe7700 (LWP 128235)): #0 0x00007f5404a436d5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f53fef9f2f3 in __db_hybrid_mutex_suspend () from /lib64/ libdb-5.3.so #2 0x00007f53fef9e640 in __db_tas_mutex_lock () from /lib64/libdb-5.3.so #3 0x00007f53ff048d3a in __lock_get_internal () from /lib64/libdb-5.3.so #4 0x00007f53ff049820 in __lock_get () from /lib64/libdb-5.3.so #5 0x00007f53ff075142 in __db_lget () from /lib64/libdb-5.3.so #6 0x00007f53fefbc605 in __bam_search () from /lib64/libdb-5.3.so #7 0x00007f53fefa7256 in __bamc_search () from /lib64/libdb-5.3.so #8 0x00007f53fefa8d0f in __bamc_get () from /lib64/libdb-5.3.so #9 0x00007f53ff061ca6 in __dbc_iget () from /lib64/libdb-5.3.so #10 0x00007f53ff070b02 in __dbc_get_pp () from /lib64/libdb-5.3.so #11 0x00007f53fa54c680 in idl_new_fetch () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #12 0x00007f53fa55ab26 in index_read_ext_allids () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #13 0x00007f53fa545284 in keys2idl () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #14 0x00007f53fa5459e3 in ava_candidates.isra.0 () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #15 0x00007f53fa545fd2 in filter_candidates_ext () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #16 0x00007f53fa547096 in list_candidates () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #17 0x00007f53fa545f40 in filter_candidates_ext () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #18 0x00007f53fa58180c in subtree_candidates () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #19 0x00007f53fa582e7f in ldbm_back_search () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #20 0x00007f5406eb01d6 in op_shared_search () from /usr/lib64/dirsrv/libslapd.so.0 #21 0x00007f54073a133e in do_search () #22 0x00007f540738f567 in connection_threadmain () #23 0x00007f540509e7bb in _pt_root () from /lib64/libnspr4.so #24 0x00007f5404a3fdc5 in start_thread () from /lib64/libpthread.so.0 #25 0x00007f540476d28d in clone () from /lib64/libc.so.6 Thread 10 (Thread 0x7f53cf7e6700 (LWP 128236)): #0 0x00007f5404a436d5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f53fef9f2f3 in __db_hybrid_mutex_suspend () from /lib64/ libdb-5.3.so #2 0x00007f53fef9e640 in __db_tas_mutex_lock () from /lib64/libdb-5.3.so #3 0x00007f53ff048d3a in __lock_get_internal () from /lib64/libdb-5.3.so #4 0x00007f53ff049820 in __lock_get () from /lib64/libdb-5.3.so #5 0x00007f53ff075142 in __db_lget () from /lib64/libdb-5.3.so #6 0x00007f53fefbc605 in __bam_search () from /lib64/libdb-5.3.so #7 0x00007f53fefa7256 in __bamc_search () from /lib64/libdb-5.3.so #8 0x00007f53fefa8d0f in __bamc_get () from /lib64/libdb-5.3.so #9 0x00007f53ff061ca6 in __dbc_iget () from /lib64/libdb-5.3.so #10 0x00007f53ff070b02 in __dbc_get_pp () from /lib64/libdb-5.3.so #11 0x00007f53fa54c680 in idl_new_fetch () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #12 0x00007f53fa55ab26 in index_read_ext_allids () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #13 0x00007f53fa545284 in keys2idl () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #14 0x00007f53fa5459e3 in ava_candidates.isra.0 () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #15 0x00007f53fa545fd2 in filter_candidates_ext () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #16 0x00007f53fa547096 in list_candidates () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #17 0x00007f53fa545f40 in filter_candidates_ext () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #18 0x00007f53fa58180c in subtree_candidates () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #19 0x00007f53fa582e7f in ldbm_back_search () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #20 0x00007f5406eb01d6 in op_shared_search () from /usr/lib64/dirsrv/libslapd.so.0 #21 0x00007f54073a133e in do_search () #22 0x00007f540738f567 in connection_threadmain () #23 0x00007f540509e7bb in _pt_root () from /lib64/libnspr4.so #24 0x00007f5404a3fdc5 in start_thread () from /lib64/libpthread.so.0 #25 0x00007f540476d28d in clone () from /lib64/libc.so.6 Thread 9 (Thread 0x7f53cefe5700 (LWP 128237)): #0 0x00007f5404a436d5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f53fef9f2f3 in __db_hybrid_mutex_suspend () from /lib64/ libdb-5.3.so #2 0x00007f53fef9e640 in __db_tas_mutex_lock () from /lib64/libdb-5.3.so #3 0x00007f53ff048d3a in __lock_get_internal () from /lib64/libdb-5.3.so #4 0x00007f53ff049820 in __lock_get () from /lib64/libdb-5.3.so #5 0x00007f53ff075142 in __db_lget () from /lib64/libdb-5.3.so #6 0x00007f53fefbc605 in __bam_search () from /lib64/libdb-5.3.so #7 0x00007f53fefa7256 in __bamc_search () from /lib64/libdb-5.3.so #8 0x00007f53fefa8d0f in __bamc_get () from /lib64/libdb-5.3.so #9 0x00007f53ff061ca6 in __dbc_iget () from /lib64/libdb-5.3.so #10 0x00007f53ff070b02 in __dbc_get_pp () from /lib64/libdb-5.3.so #11 0x00007f53fa54c680 in idl_new_fetch () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #12 0x00007f53fa55ab26 in index_read_ext_allids () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #13 0x00007f53fa545284 in keys2idl () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #14 0x00007f53fa5459e3 in ava_candidates.isra.0 () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #15 0x00007f53fa545fd2 in filter_candidates_ext () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #16 0x00007f53fa547096 in list_candidates () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #17 0x00007f53fa545f40 in filter_candidates_ext () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #18 0x00007f53fa58180c in subtree_candidates () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #19 0x00007f53fa582e7f in ldbm_back_search () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #20 0x00007f5406eb01d6 in op_shared_search () from /usr/lib64/dirsrv/libslapd.so.0 #21 0x00007f54073a133e in do_search () #22 0x00007f540738f567 in connection_threadmain () #23 0x00007f540509e7bb in _pt_root () from /lib64/libnspr4.so #24 0x00007f5404a3fdc5 in start_thread () from /lib64/libpthread.so.0 #25 0x00007f540476d28d in clone () from /lib64/libc.so.6 Thread 8 (Thread 0x7f53ce7e4700 (LWP 128238)): #0 0x00007f5404a436d5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f53fef9f2f3 in __db_hybrid_mutex_suspend () from /lib64/ libdb-5.3.so #2 0x00007f53fef9e640 in __db_tas_mutex_lock () from /lib64/libdb-5.3.so #3 0x00007f53ff048d3a in __lock_get_internal () from /lib64/libdb-5.3.so #4 0x00007f53ff049820 in __lock_get () from /lib64/libdb-5.3.so #5 0x00007f53ff075142 in __db_lget () from /lib64/libdb-5.3.so #6 0x00007f53fefbc605 in __bam_search () from /lib64/libdb-5.3.so #7 0x00007f53fefa7256 in __bamc_search () from /lib64/libdb-5.3.so #8 0x00007f53fefa8d0f in __bamc_get () from /lib64/libdb-5.3.so #9 0x00007f53ff061ca6 in __dbc_iget () from /lib64/libdb-5.3.so #10 0x00007f53ff070b02 in __dbc_get_pp () from /lib64/libdb-5.3.so #11 0x00007f53fa54c680 in idl_new_fetch () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #12 0x00007f53fa55ab26 in index_read_ext_allids () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #13 0x00007f53fa545284 in keys2idl () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #14 0x00007f53fa5459e3 in ava_candidates.isra.0 () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #15 0x00007f53fa545fd2 in filter_candidates_ext () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #16 0x00007f53fa547096 in list_candidates () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #17 0x00007f53fa545f40 in filter_candidates_ext () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #18 0x00007f53fa58180c in subtree_candidates () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #19 0x00007f53fa582e7f in ldbm_back_search () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #20 0x00007f5406eb01d6 in op_shared_search () from /usr/lib64/dirsrv/libslapd.so.0 #21 0x00007f54073a133e in do_search () #22 0x00007f540738f567 in connection_threadmain () #23 0x00007f540509e7bb in _pt_root () from /lib64/libnspr4.so #24 0x00007f5404a3fdc5 in start_thread () from /lib64/libpthread.so.0 #25 0x00007f540476d28d in clone () from /lib64/libc.so.6 Thread 7 (Thread 0x7f53cdfe3700 (LWP 128239)): #0 0x00007f5404a436d5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f53fef9f2f3 in __db_hybrid_mutex_suspend () from /lib64/ libdb-5.3.so #2 0x00007f53fef9e640 in __db_tas_mutex_lock () from /lib64/libdb-5.3.so #3 0x00007f53ff048d3a in __lock_get_internal () from /lib64/libdb-5.3.so #4 0x00007f53ff049820 in __lock_get () from /lib64/libdb-5.3.so #5 0x00007f53ff075142 in __db_lget () from /lib64/libdb-5.3.so #6 0x00007f53fefbc605 in __bam_search () from /lib64/libdb-5.3.so #7 0x00007f53fefa7256 in __bamc_search () from /lib64/libdb-5.3.so #8 0x00007f53fefa8d0f in __bamc_get () from /lib64/libdb-5.3.so #9 0x00007f53ff061ca6 in __dbc_iget () from /lib64/libdb-5.3.so #10 0x00007f53ff070b02 in __dbc_get_pp () from /lib64/libdb-5.3.so #11 0x00007f53fa54c680 in idl_new_fetch () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #12 0x00007f53fa55ab26 in index_read_ext_allids () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #13 0x00007f53fa545284 in keys2idl () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #14 0x00007f53fa5459e3 in ava_candidates.isra.0 () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #15 0x00007f53fa545fd2 in filter_candidates_ext () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #16 0x00007f53fa547096 in list_candidates () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #17 0x00007f53fa545f40 in filter_candidates_ext () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #18 0x00007f53fa58180c in subtree_candidates () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #19 0x00007f53fa582e7f in ldbm_back_search () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #20 0x00007f5406eb01d6 in op_shared_search () from /usr/lib64/dirsrv/libslapd.so.0 #21 0x00007f54073a133e in do_search () #22 0x00007f540738f567 in connection_threadmain () #23 0x00007f540509e7bb in _pt_root () from /lib64/libnspr4.so #24 0x00007f5404a3fdc5 in start_thread () from /lib64/libpthread.so.0 #25 0x00007f540476d28d in clone () from /lib64/libc.so.6 Thread 6 (Thread 0x7f53cd7e2700 (LWP 128240)): #0 0x00007f5404a436d5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f53fef9f2f3 in __db_hybrid_mutex_suspend () from /lib64/ libdb-5.3.so #2 0x00007f53fef9eb37 in __db_tas_mutex_readlock () from /lib64/ libdb-5.3.so #3 0x00007f53ff0b8d35 in __memp_fget () from /lib64/libdb-5.3.so #4 0x00007f53ff062333 in __dbc_iget () from /lib64/libdb-5.3.so #5 0x00007f53ff070b02 in __dbc_get_pp () from /lib64/libdb-5.3.so #6 0x00007f53fa54c822 in idl_new_fetch () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #7 0x00007f53fa55ab26 in index_read_ext_allids () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #8 0x00007f53fa545284 in keys2idl () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #9 0x00007f53fa5459e3 in ava_candidates.isra.0 () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #10 0x00007f53fa545fd2 in filter_candidates_ext () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #11 0x00007f53fa547096 in list_candidates () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #12 0x00007f53fa545f40 in filter_candidates_ext () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #13 0x00007f53fa547096 in list_candidates () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #14 0x00007f53fa545f40 in filter_candidates_ext () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #15 0x00007f53fa547096 in list_candidates () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #16 0x00007f53fa545f40 in filter_candidates_ext () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #17 0x00007f53fa58180c in subtree_candidates () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #18 0x00007f53fa582e7f in ldbm_back_search () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #19 0x00007f5406eb01d6 in op_shared_search () from /usr/lib64/dirsrv/libslapd.so.0 #20 0x00007f54073a133e in do_search () #21 0x00007f540738f3c3 in connection_threadmain () #22 0x00007f540509e7bb in _pt_root () from /lib64/libnspr4.so #23 0x00007f5404a3fdc5 in start_thread () from /lib64/libpthread.so.0 #24 0x00007f540476d28d in clone () from /lib64/libc.so.6 Thread 5 (Thread 0x7f53ccfe1700 (LWP 128241)): #0 0x00007f5404a436d5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f53fef9f2f3 in __db_hybrid_mutex_suspend () from /lib64/ libdb-5.3.so #2 0x00007f53fef9e640 in __db_tas_mutex_lock () from /lib64/libdb-5.3.so #3 0x00007f53ff048d3a in __lock_get_internal () from /lib64/libdb-5.3.so #4 0x00007f53ff049820 in __lock_get () from /lib64/libdb-5.3.so #5 0x00007f53ff075142 in __db_lget () from /lib64/libdb-5.3.so #6 0x00007f53fefbc605 in __bam_search () from /lib64/libdb-5.3.so #7 0x00007f53fefa7256 in __bamc_search () from /lib64/libdb-5.3.so #8 0x00007f53fefa8d0f in __bamc_get () from /lib64/libdb-5.3.so #9 0x00007f53ff061ca6 in __dbc_iget () from /lib64/libdb-5.3.so #10 0x00007f53ff070b02 in __dbc_get_pp () from /lib64/libdb-5.3.so #11 0x00007f53fa54c680 in idl_new_fetch () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #12 0x00007f53fa55ab26 in index_read_ext_allids () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #13 0x00007f53fa545284 in keys2idl () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #14 0x00007f53fa5459e3 in ava_candidates.isra.0 () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #15 0x00007f53fa545fd2 in filter_candidates_ext () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #16 0x00007f53fa547096 in list_candidates () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #17 0x00007f53fa545f40 in filter_candidates_ext () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #18 0x00007f53fa547096 in list_candidates () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #19 0x00007f53fa545f40 in filter_candidates_ext () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #20 0x00007f53fa547096 in list_candidates () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #21 0x00007f53fa545f40 in filter_candidates_ext () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #22 0x00007f53fa58180c in subtree_candidates () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #23 0x00007f53fa582e7f in ldbm_back_search () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #24 0x00007f5406eb01d6 in op_shared_search () from /usr/lib64/dirsrv/libslapd.so.0 #25 0x00007f54073a133e in do_search () #26 0x00007f540738f3c3 in connection_threadmain () #27 0x00007f540509e7bb in _pt_root () from /lib64/libnspr4.so #28 0x00007f5404a3fdc5 in start_thread () from /lib64/libpthread.so.0 #29 0x00007f540476d28d in clone () from /lib64/libc.so.6 Thread 4 (Thread 0x7f53cc7e0700 (LWP 128242)): #0 0x00007f54047649b3 in select () from /lib64/libc.so.6 #1 0x00007f5406eef0e9 in DS_Sleep () from /usr/lib64/dirsrv/libslapd.so.0 #2 0x00007f5407390325 in time_thread () #3 0x00007f540509e7bb in _pt_root () from /lib64/libnspr4.so #4 0x00007f5404a3fdc5 in start_thread () from /lib64/libpthread.so.0 #5 0x00007f540476d28d in clone () from /lib64/libc.so.6 Thread 3 (Thread 0x7f53cb7a0700 (LWP 128259)): #0 0x00007f5404a436d5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f5405099050 in PR_WaitCondVar () from /lib64/libnspr4.so #2 0x00007f540739bf25 in ps_send_results () #3 0x00007f540509e7bb in _pt_root () from /lib64/libnspr4.so #4 0x00007f5404a3fdc5 in start_thread () from /lib64/libpthread.so.0 #5 0x00007f540476d28d in clone () from /lib64/libc.so.6 Thread 2 (Thread 0x7f53cab9d700 (LWP 9020)): #0 0x00007f5404a436d5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f5405099050 in PR_WaitCondVar () from /lib64/libnspr4.so #2 0x00007f540739bf25 in ps_send_results () #3 0x00007f540509e7bb in _pt_root () from /lib64/libnspr4.so #4 0x00007f5404a3fdc5 in start_thread () from /lib64/libpthread.so.0 #5 0x00007f540476d28d in clone () from /lib64/libc.so.6 Thread 1 (Thread 0x7f5407357840 (LWP 128124)): #0 0x00007f5404762c3d in poll () from /lib64/libc.so.6 #1 0x00007f540509a967 in _pr_poll_with_poll () from /lib64/libnspr4.so #2 0x00007f5407393ff1 in slapd_daemon () #3 0x00007f5407385c6c in main () Thanks On Tue, Sep 6, 2016 at 5:51 PM, Rakesh Rajasekharan < rakesh.rajasekharan at gmail.com> wrote: > Hi, > > With script provided by Rich, I looked at my db cache usage and it does > not look to be high enough > > # under each db are the list of selected indexes specified with INDEXLIST > dbname count free free% size > changelog:ent 5 2076333 99.0 4163.8 > changelog:dn 15976 9303557 88.7 74.0 > userroot:ent 4763 175321074 83.6 7221.1 > userroot:dn 4763 9995056 95.3 103.0 > ipaca:ent 425 6337866 60.4 9759.8 > ipaca:dn 425 10451081 99.7 81.6 > > Right now I set it to 200MB. So, I am planning to bring it down to 100MB > > Also, *nsslapd-db-checkpoint-interval *will change it to 80 or 90 from > default of 60 > > I will check this out and the impact it has on the hang issues and get back > > Thanks > Rakesh > > On Tue, Sep 6, 2016 at 4:13 PM, Ludwig Krispenz > wrote: > >> Hi, >> >> I think you are running into this bug: https://bugzilla.redhat.com/sh >> ow_bug.cgi?id=1349779 >> it was the outcome of an analysis started in this one: >> https://bugzilla.redhat.com/show_bug.cgi?id=1347800 >> >> the problem is that it looks like the checkpoint thread is holding a >> mutex and otehr threads are blocked on it. It looks like an issue in >> BerkeleyDB although there is no explanation from the BDB code why this >> could happen, we also had a support case with Oracle to investigate this, >> with no solution. Oracle at the end was providing a diagnostic patch, which >> we did not test since the customer with the original problem had worked >> around this by changing their configuration. >> I don't know the status of this patch and if we can make it available, >> probably depends on your support status, you need to contact support for >> this. >> >> About the workarounds: >> you can try to change timing for the database update and checkpointing eg >> by reducing the dbcache size, changing the checkpoint interval. In the >> customer case they identified one substring index, which was frequently >> updated but not really required - and removing this index helped. >> >> Regards, >> Ludwig >> >> On 09/06/2016 11:31 AM, thierry bordaz wrote: >> >> Hi, >> >> The following thread is hanging most of the others threads >> >> Thread 20 (Thread 0x7f68007f8700 (LWP 4873)): >> >> #0 0x00007f683d0506d5 in pthread_cond_wait@@GLIBC_2.3.2 () from >> /lib64/libpthread.so.0 >> #1 0x00007f68375ac2f3 in __db_hybrid_mutex_suspend () from /lib64/ >> libdb-5.3.so >> #2 0x00007f68375ab640 in __db_tas_mutex_lock () from /lib64/libdb-5.3.so >> #3 0x00007f68376c5e1b in __memp_fget () from /lib64/libdb-5.3.so >> #4 0x00007f68375c9211 in __bam_search () from /lib64/libdb-5.3.so >> #5 0x00007f68375b4256 in __bamc_search () from /lib64/libdb-5.3.so >> #6 0x00007f68375b86c4 in __bamc_put () from /lib64/libdb-5.3.so >> #7 0x00007f683766f6d5 in __dbc_iput () from /lib64/libdb-5.3.so >> #8 0x00007f683766aa0e in __db_put () from /lib64/libdb-5.3.so >> #9 0x00007f683767ffa4 in __db_put_pp () from /lib64/libdb-5.3.so >> #10 0x00007f6832b5a4cf in idl_new_insert_key () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #11 0x00007f6832b695b5 in addordel_values_sv.isra.3 () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #12 0x00007f6832b6a25a in index_addordel_values_ext_sv () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #13 0x00007f6832b6a694 in index_addordel_values_sv () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #14 0x00007f6832b6a75a in index_addordel_entry () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #15 0x00007f6832b6d5fa in ldbm_back_add () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #16 0x00007f683f46fcca in op_shared_add () from >> /usr/lib64/dirsrv/libslapd.so.0 >> #17 0x00007f683f471020 in do_add () from /usr/lib64/dirsrv/libslapd.so.0 >> #18 0x00007f683f99c303 in connection_threadmain () >> #19 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so >> #20 0x00007f683d04cdc5 in start_thread () from /lib64/libpthread.so.0 >> #21 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 >> >> This thread is likely not hang on 'db_tas_mutex_lock' as the db_lock does >> not show it waiting for something. >> Unfortunately it is not shown in the output of top -H, did you get the >> complete output ? >> Ludwig mentioned a known issue that DS is not scaling well on NUMA arch, >> do you know if the machine is NUMA ? >> A possibility is that this thread get severely slowed down by a NUMA >> remote lock and showing DS has hung. >> This is why it would be interesting to know if it consums CPU. >> >> thanks >> theirry >> >> On 09/06/2016 10:59 AM, Rakesh Rajasekharan wrote: >> >> Hi Thierry/Ludwig, >> >> I again hit the hang issue. This is the pstack of the ns-slapd >> >> So was able to this time get the pstack and db locks >> >> db locks details >> Default locking region information: >> 451 Last allocated locker ID >> 0x7fffffff Current maximum unused locker ID >> 9 Number of lock modes >> 800 Initial number of locks allocated >> 0 Initial number of lockers allocated >> 800 Initial number of lock objects allocated >> 10000 Maximum number of locks possible >> 10000 Maximum number of lockers possible >> 10000 Maximum number of lock objects possible >> 800 Current number of locks allocated >> 366 Current number of lockers allocated >> 800 Current number of lock objects allocated >> 160 Number of lock object partitions >> 8191 Size of object hash table >> 126 Number of current locks >> 1005 Maximum number of locks at any one time >> 33 Maximum number of locks in any one bucket >> 3987 Maximum number of locks stolen by for an empty partition >> 75 Maximum number of locks stolen for any one partition >> 309 Number of current lockers >> 313 Maximum number of lockers at any one time >> 121 Number of current lock objects >> 676 Maximum number of lock objects at any one time >> 3 Maximum number of lock objects in any one bucket >> 69 Maximum number of objects stolen by for an empty partition >> 5 Maximum number of objects stolen for any one partition >> 22M Total number of locks requested (22009642) >> 21M Total number of locks released (21992878) >> 0 Total number of locks upgraded >> 95 Total number of locks downgraded >> 3348 Lock requests not available due to conflicts, for which we >> waited >> 12646 Lock requests not available due to conflicts, for which we did >> not wait >> 0 Number of deadlocks >> 0 Lock timeout value >> 0 Number of locks that have timed out >> 0 Transaction timeout value >> 0 Number of transactions that have timed out >> 2MB 560KB Region size >> 71719 The number of partition locks that required waiting (0%) >> 23105 The maximum number of times any partition lock was waited for >> (0%) >> 0 The number of object queue operations that required waiting >> (0%) >> 163 The number of locker allocations that required waiting (0%) >> 3172 The number of region locks that required waiting (0%) >> 5 Maximum hash bucket length >> =-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-= >> Lock REGINFO information: >> Environment Region type >> 1 Region ID >> /var/lib/dirsrv/slapd-SPRINKLR-COM/db/__db.001 Region name >> 0x7f67ef869000 Region address >> 0x7f67ef8690a0 Region allocation head >> 0x7f67ef8712b0 Region primary address >> 0 Region maximum allocation >> 0 Region allocated >> Region allocations: 1581345 allocations, 0 failures, 1580794 frees, 6 >> longest >> Allocations by power-of-two sizes: >> 1KB 1581329 >> 2KB 3 >> 4KB 4 >> 8KB 3 >> 16KB 3 >> 32KB 1 >> 64KB 0 >> 128KB 0 >> 256KB 2 >> 512KB 0 >> 1024KB 1 >> REGION_SHARED Region flags >> =-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-= >> Lock region parameters: >> 2 Lock region region mutex [3172/14M 0% !Own] >> 16381 locker table size >> 8191 object table size >> 34128 obj_off >> 900216 locker_off >> 0 need_dd >> =-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-= >> Lock conflict matrix: >> =-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-= >> Locks grouped by lockers: >> Locker Mode Count Status ----------------- Object --------------- >> 2 dd=308 locks held 1 write locks 0 pid/thread >> 4845/140085720139840 flags 10 priority 100 >> 2 READ 1 HELD userRoot/id2entry.db handle 0 >> 3 dd=307 locks held 0 write locks 0 pid/thread >> 4845/140084829472512 flags 0 priority 100 >> 4 dd=306 locks held 1 write locks 0 pid/thread >> 4845/140085720139840 flags 10 priority 100 >> 4 READ 1 HELD ipaca/id2entry.db handle 0 >> 5 dd=305 locks held 0 write locks 0 pid/thread >> 4845/140084837865216 flags 0 priority 100 >> 6 dd=304 locks held 1 write locks 0 pid/thread >> 4845/140085720139840 flags 10 priority 100 >> 6 READ 1 HELD ipaca/entryrdn.db handle 0 >> 7 dd=303 locks held 0 write locks 0 pid/thread >> 4845/140084804294400 flags 0 priority 100 >> 8 dd=302 locks held 0 write locks 0 pid/thread >> 4845/140084720432896 flags 0 priority 100 >> 9 dd=301 locks held 0 write locks 0 pid/thread >> 4845/140084837865216 flags 0 priority 100 >> a dd=300 locks held 1 write locks 0 pid/thread >> 4845/140085720139840 flags 10 priority 100 >> a READ 1 HELD ipaca/vlv#allcertspkitomcatindex.db >> handle 0 >> e dd=299 locks held 1 write locks 0 pid/thread >> 4845/140085720139840 flags 10 priority 100 >> e READ 1 HELD ipaca/vlv#allnonrevokedcertspkitomcatindex.db >> handle 0 >> 10 dd=298 locks held 1 write locks 0 pid/thread >> 4845/140085720139840 flags 10 priority 100 >> 10 READ 1 HELD ipaca/vlv#allrevokedcertspkitomcatindex.db >> handle 0 >> 11 dd=297 locks held 1 write locks 0 pid/thread >> 4845/140085720139840 flags 10 priority 100 >> 11 READ 1 HELD ipaca/vlv#allrevokedcertsnotafterpkitomcatindex.db >> handle 0 >> 14 dd=296 locks held 1 write locks 0 pid/thread >> 4845/140085720139840 flags 10 priority 100 >> 14 READ 1 HELD ipaca/vlv#allrevokedorrevokedexpiredcertspkitomcatindex.db >> handle 0 >> 15 dd=295 locks held 1 write locks 0 pid/thread >> 4845/140085720139840 flags 10 priority 100 >> 15 READ 1 HELD ipaca/vlv#allvalidcertspkitomcatindex.db >> handle 0 >> 16 dd=294 locks held 1 write locks 0 pid/thread >> 4845/140085720139840 flags 10 priority 100 >> 16 READ 1 HELD ipaca/vlv#allvalidcertsnotafterpkitomcatindex.db >> handle 0 >> 17 dd=293 locks held 1 write locks 0 pid/thread >> 4845/140085720139840 flags 10 priority 100 >> 17 READ 1 HELD ipaca/vlv#allvalidorrevokedcertspkitomcatindex.db >> handle 0 >> 18 dd=292 locks held 1 write locks 0 pid/thread >> 4845/140085720139840 flags 10 priority 100 >> 18 READ 1 HELD ipaca/vlv#caallpkitomcatindex.db >> handle 0 >> 1d dd=291 locks held 1 write locks 0 pid/thread >> 4845/140085720139840 flags 10 priority 100 >> 1d READ 1 HELD ipaca/vlv#cacompletepkitomcatindex.db >> handle 0 >> 1e dd=290 locks held 1 write locks 0 pid/thread >> 4845/140085720139840 flags 10 priority 100 >> 1e READ 1 HELD ipaca/vlv#cacompleteenrollmentpkitomcatindex.db >> handle 0 >> 20 dd=289 locks held 1 write locks 0 pid/thread >> 4845/140085720139840 flags 10 priority 100 >> 20 READ 1 HELD ipaca/vlv#cacompleterevocationpkitomcatindex.db >> handle 0 >> 21 dd=288 locks held 1 write locks 0 pid/thread >> 4845/140085720139840 flags 10 priority 100 >> 21 READ 1 HELD ipaca/vlv#caenrollmentpkitomcatindex.db >> handle 0 >> 2b dd=287 locks held 1 write locks 0 pid/thread >> 4845/140085720139840 flags 10 priority 100 >> 2b READ 1 HELD ipaca/vlv#carevocationpkitomcatindex.db >> handle 0 >> 2c dd=286 locks held 1 write locks 0 pid/thread >> 4845/140085720139840 flags 10 priority 100 >> 2c READ 1 HELD changelog/id2entry.db handle 0 >> 2d dd=285 locks held 0 write locks 0 pid/thread >> 4845/140085719607040 flags 0 priority 100 >> 2e dd=284 locks held 1 write locks 0 pid/thread >> 4845/140085720139840 flags 10 priority 100 >> 2e READ 1 HELD changelog/entryusn.db handle 0 >> 2f dd=283 locks held 0 write locks 0 pid/thread >> 4845/140085720139840 flags 0 priority 100 >> 30 dd=282 locks held 1 write locks 0 pid/thread >> 4845/140085720139840 flags 10 priority 100 >> 30 READ 1 HELD userRoot/entryusn.db handle 0 >> 31 dd=281 locks held 0 write locks 0 pid/thread >> 4845/140085720139840 flags 0 priority 100 >> 32 dd=280 locks held 1 write locks 0 pid/thread >> 4845/140085720139840 flags 10 priority 100 >> 32 READ 1 HELD ipaca/entryusn.db handle 0 >> 33 dd=279 locks held 0 write locks 0 pid/thread >> 4845/140085720139840 flags 0 priority 100 >> 34 dd=278 locks held 1 write locks 0 pid/thread >> 4845/140085720139840 flags 10 priority 100 >> 34 READ 1 HELD userRoot/entryrdn.db handle 0 >> 35 dd=277 locks held 0 write locks 0 pid/thread >> 4845/140084544186112 flags 0 priority 100 >> 36 dd=276 locks held 0 write locks 0 pid/thread >> 4845/140084636505856 flags 0 priority 100 >> 37 dd=275 locks held 0 write locks 0 pid/thread >> 4845/140084854650624 flags 0 priority 100 >> 38 dd=274 locks held 1 write locks 0 pid/thread >> 4845/140085720139840 flags 10 priority 100 >> 38 READ 1 HELD userRoot/objectclass.db handle 0 >> 39 dd=273 locks held 0 write locks 0 pid/thread >> 4845/140084569364224 flags 0 priority 100 >> 3a dd=272 locks held 1 write locks 0 pid/thread >> 4845/140085720139840 flags 10 priority 100 >> 3a READ 1 HELD userRoot/ancestorid.db handle 0 >> 3b dd=271 locks held 0 write locks 0 pid/thread >> 4845/140084854650624 flags 0 priority 100 >> 3c dd=270 locks held 1 write locks 0 pid/thread >> 4845/140085720139840 flags 10 priority 100 >> 3c READ 1 HELD changelog/entryrdn.db handle 0 >> 3d dd=269 locks held 0 write locks 0 pid/thread >> 4845/140085719607040 flags 0 priority 100 >> 3e dd=268 locks held 0 write locks 0 pid/thread >> 4845/140084602935040 flags 0 priority 100 >> 3f dd=267 locks held 0 write locks 0 pid/thread >> 4845/140085719607040 flags 0 priority 100 >> 40 dd=266 locks held 1 write locks 0 pid/thread >> 4845/140085720139840 flags 10 priority 100 >> 40 READ 1 HELD changelog/objectclass.db handle 0 >> 41 dd=265 locks held 0 write locks 0 pid/thread >> 4845/140085719607040 flags 0 priority 100 >> 42 dd=264 locks held 0 write locks 0 pid/thread >> 4845/140085719607040 flags 0 priority 100 >> 43 dd=263 locks held 0 write locks 0 pid/thread >> 4845/140084653291264 flags 0 priority 100 >> 44 dd=262 locks held 1 write locks 0 pid/thread >> 4845/140085720139840 flags 10 priority 100 >> 44 READ 1 HELD ipaca/objectclass.db handle 0 >> 45 dd=261 locks held 0 write locks 0 pid/thread >> 4845/140084577756928 flags 0 priority 100 >> 46 dd=260 locks held 0 write locks 0 pid/thread >> 4845/140084720432896 flags 0 priority 100 >> 47 dd=259 locks held 1 write locks 0 pid/thread >> 4845/140085720139840 flags 10 priority 100 >> 47 READ 1 HELD changelog/aci.db handle 0 >> 48 dd=258 locks held 0 write locks 0 pid/thread >> 4845/140085720139840 flags 0 priority 100 >> 49 dd=257 locks held 1 write locks 0 pid/thread >> 4845/140085720139840 flags 10 priority 100 >> 49 READ 1 HELD userRoot/aci.db handle 0 >> 4a dd=256 locks held 0 write locks 0 pid/thread >> 4845/140085720139840 flags 0 priority 100 >> 4b dd=255 locks held 0 write locks 0 pid/thread >> 4845/140084535793408 flags 0 priority 100 >> 4c dd=254 locks held 1 write locks 0 pid/thread >> 4845/140085720139840 flags 10 priority 100 >> 4c READ 1 HELD ipaca/aci.db handle 0 >> 4d dd=253 locks held 0 write locks 0 pid/thread >> 4845/140085720139840 flags 0 priority 100 >> 4e dd=252 locks held 1 write locks 0 pid/thread >> 4845/140085720139840 flags 10 priority 100 >> 4e READ 1 HELD userRoot/parentid.db handle 0 >> 4f dd=251 locks held 0 write locks 0 pid/thread >> 4845/140085720139840 flags 0 priority 100 >> 50 dd=250 locks held 0 write locks 0 pid/thread >> 4845/140085719607040 flags 0 priority 100 >> 51 dd=249 locks held 1 write locks 0 pid/thread >> 4845/140085720139840 flags 10 priority 100 >> 51 READ 1 HELD changelog/nsuniqueid.db handle 0 >> 52 dd=248 locks held 1 write locks 0 pid/thread >> 4845/140085720139840 flags 10 priority 100 >> 52 READ 1 HELD changelog/changenumber.db handle 0 >> 53 dd=247 locks held 0 write locks 0 pid/thread >> 4845/140084602935040 flags 0 priority 100 >> 54 dd=246 locks held 0 write locks 0 pid/thread >> 4845/140085719607040 flags 0 priority 100 >> 55 dd=245 locks held 1 write locks 0 pid/thread >> 4845/140085719607040 flags 10 priority 100 >> 55 READ 1 HELD changelog/targetuniqueid.db >> handle 0 >> 56 dd=244 locks held 1 write locks 0 pid/thread >> 4845/140085719607040 flags 10 priority 100 >> 56 READ 1 HELD changelog/parentid.db handle 0 >> 57 dd=243 locks held 1 write locks 0 pid/thread >> 4845/140085719607040 flags 10 priority 100 >> 57 READ 1 HELD changelog/ancestorid.db handle 0 >> 58 dd=242 locks held 1 write locks 0 pid/thread >> 4845/140085719607040 flags 10 priority 100 >> 58 READ 1 HELD changelog/numsubordinates.db >> handle 0 >> 59 dd=241 locks held 0 write locks 0 pid/thread >> 4845/140085719607040 flags 0 priority 100 >> 5a dd=240 locks held 0 write locks 0 pid/thread >> 4845/140084544186112 flags 0 priority 100 >> 5b dd=239 locks held 1 write locks 0 pid/thread >> 4845/140084577756928 flags 10 priority 100 >> 5b READ 1 HELD userRoot/krbPrincipalName.db >> handle 0 >> 5c dd=238 locks held 0 write locks 0 pid/thread >> 4845/140084636505856 flags 0 priority 100 >> 5d dd=237 locks held 0 write locks 0 pid/thread >> 4845/140084695254784 flags 0 priority 100 >> 5e dd=236 locks held 1 write locks 0 pid/thread >> 4845/140084560971520 flags 10 priority 100 >> 5e READ 1 HELD userRoot/ipakrbprincipalalias.db >> handle 0 >> 5f dd=235 locks held 0 write locks 0 pid/thread >> 4845/140084703647488 flags 0 priority 100 >> 60 dd=234 locks held 0 write locks 0 pid/thread >> 4845/140084653291264 flags 0 priority 100 >> 61 dd=233 locks held 0 write locks 0 pid/thread >> 4845/140084577756928 flags 0 priority 100 >> 62 dd=232 locks held 0 write locks 0 pid/thread >> 4845/140084686862080 flags 0 priority 100 >> 63 dd=231 locks held 1 write locks 0 pid/thread >> 4845/140084611327744 flags 10 priority 100 >> 63 READ 1 HELD changelog/seeAlso.db handle 0 >> 64 dd=230 locks held 0 write locks 0 pid/thread >> 4845/140084686862080 flags 0 priority 100 >> 65 dd=229 locks held 1 write locks 0 pid/thread >> 4845/140084611327744 flags 10 priority 100 >> 65 READ 1 HELD userRoot/seeAlso.db handle 0 >> 66 dd=228 locks held 0 write locks 0 pid/thread >> 4845/140084636505856 flags 0 priority 100 >> 67 dd=227 locks held 1 write locks 0 pid/thread >> 4845/140084611327744 flags 10 priority 100 >> 67 READ 1 HELD ipaca/seeAlso.db handle 0 >> 68 dd=226 locks held 0 write locks 0 pid/thread >> 4845/140084686862080 flags 0 priority 100 >> 69 dd=225 locks held 1 write locks 0 pid/thread >> 4845/140084678469376 flags 10 priority 100 >> 69 READ 1 HELD ipaca/parentid.db handle 0 >> 6a dd=224 locks held 0 write locks 0 pid/thread >> 4845/140084678469376 flags 0 priority 100 >> 6b dd=223 locks held 0 write locks 0 pid/thread >> 4845/140084829472512 flags 0 priority 100 >> 6c dd=222 locks held 0 write locks 0 pid/thread >> 4845/140084653291264 flags 0 priority 100 >> 6d dd=221 locks held 0 write locks 0 pid/thread >> 4845/140084569364224 flags 0 priority 100 >> 6e dd=220 locks held 0 write locks 0 pid/thread >> 4845/140084636505856 flags 0 priority 100 >> 6e READ 1 WAIT userRoot/objectclass.db page 18 >> 6f dd=219 locks held 0 write locks 0 pid/thread >> 4845/140084544186112 flags 0 priority 100 >> 70 dd=218 locks held 0 write locks 0 pid/thread >> 4845/140084569364224 flags 0 priority 100 >> 71 dd=217 locks held 0 write locks 0 pid/thread >> 4845/140084619720448 flags 0 priority 100 >> 72 dd=216 locks held 0 write locks 0 pid/thread >> 4845/140084552578816 flags 0 priority 100 >> 73 dd=215 locks held 0 write locks 0 pid/thread >> 4845/140084829472512 flags 0 priority 100 >> 74 dd=214 locks held 0 write locks 0 pid/thread >> 4845/140084628113152 flags 0 priority 100 >> 75 dd=213 locks held 0 write locks 0 pid/thread >> 4845/140084686862080 flags 0 priority 100 >> 76 dd=212 locks held 1 write locks 0 pid/thread >> 4845/140084560971520 flags 10 priority 100 >> 76 READ 1 HELD userRoot/ipauniqueid.db handle 0 >> 77 dd=211 locks held 0 write locks 0 pid/thread >> 4845/140084661683968 flags 0 priority 100 >> 78 dd=210 locks held 0 write locks 0 pid/thread >> 4845/140084846257920 flags 0 priority 100 >> 79 dd=209 locks held 0 write locks 0 pid/thread >> 4845/140084628113152 flags 0 priority 100 >> 7a dd=208 locks held 1 write locks 0 pid/thread >> 4845/140084560971520 flags 10 priority 100 >> 7a READ 1 HELD userRoot/nsuniqueid.db handle 0 >> 7b dd=207 locks held 1 write locks 0 pid/thread >> 4845/140084560971520 flags 10 priority 100 >> 7b READ 1 HELD userRoot/cn.db handle 0 >> 7c dd=206 locks held 1 write locks 0 pid/thread >> 4845/140084560971520 flags 10 priority 100 >> 7c READ 1 HELD userRoot/fqdn.db handle 0 >> 7d dd=205 locks held 1 write locks 0 pid/thread >> 4845/140084560971520 flags 10 priority 100 >> 7d READ 1 HELD userRoot/managedby.db handle 0 >> 7e dd=204 locks held 1 write locks 0 pid/thread >> 4845/140084560971520 flags 10 priority 100 >> 7e READ 1 HELD userRoot/numsubordinates.db >> handle 0 >> 7f dd=203 locks held 1 write locks 0 pid/thread >> 4845/140084678469376 flags 10 priority 100 >> 7f READ 1 HELD userRoot/member.db handle 0 >> 80 dd=202 locks held 0 write locks 0 pid/thread >> 4845/140084712040192 flags 0 priority 100 >> 81 dd=201 locks held 1 write locks 0 pid/thread >> 4845/140084678469376 flags 10 priority 100 >> 81 READ 1 HELD userRoot/memberUser.db handle 0 >> 82 dd=200 locks held 0 write locks 0 pid/thread >> 4845/140084577756928 flags 0 priority 100 >> 83 dd=199 locks held 1 write locks 0 pid/thread >> 4845/140084678469376 flags 10 priority 100 >> 83 READ 1 HELD userRoot/memberHost.db handle 0 >> 84 dd=198 locks held 0 write locks 0 pid/thread >> 4845/140084712040192 flags 0 priority 100 >> 85 dd=197 locks held 0 write locks 0 pid/thread >> 4845/140084577756928 flags 0 priority 100 >> 86 dd=196 locks held 0 write locks 0 pid/thread >> 4845/140084670076672 flags 0 priority 100 >> 87 dd=195 locks held 0 write locks 0 pid/thread >> 4845/140084577756928 flags 0 priority 100 >> 88 dd=194 locks held 0 write locks 0 pid/thread >> 4845/140084846257920 flags 0 priority 100 >> 89 dd=193 locks held 0 write locks 0 pid/thread >> 4845/140084552578816 flags 0 priority 100 >> 8a dd=192 locks held 0 write locks 0 pid/thread >> 4845/140084703647488 flags 0 priority 100 >> 8b dd=191 locks held 0 write locks 0 pid/thread >> 4845/140084544186112 flags 0 priority 100 >> 8c dd=190 locks held 0 write locks 0 pid/thread >> 4845/140084846257920 flags 0 priority 100 >> 8d dd=189 locks held 0 write locks 0 pid/thread >> 4845/140084661683968 flags 0 priority 100 >> 8e dd=188 locks held 1 write locks 0 pid/thread >> 4845/140085067675392 flags 10 priority 100 >> 8e READ 1 HELD userRoot/macAddress.db handle 0 >> 8f dd=187 locks held 0 write locks 0 pid/thread >> 4845/140085067675392 flags 0 priority 100 >> 90 dd=186 locks held 0 write locks 0 pid/thread >> 4845/140084703647488 flags 0 priority 100 >> 91 dd=185 locks held 0 write locks 0 pid/thread >> 4845/140084552578816 flags 0 priority 100 >> 92 dd=184 locks held 0 write locks 0 pid/thread >> 4845/140084560971520 flags 0 priority 100 >> 92 READ 1 WAIT userRoot/objectclass.db page 9 >> 93 dd=183 locks held 0 write locks 0 pid/thread >> 4845/140084577756928 flags 0 priority 100 >> 94 dd=182 locks held 0 write locks 0 pid/thread >> 4845/140084821079808 flags 0 priority 100 >> 94 READ 1 WAIT userRoot/objectclass.db page 18 >> 95 dd=181 locks held 0 write locks 0 pid/thread >> 4845/140084544186112 flags 0 priority 100 >> 96 dd=180 locks held 0 write locks 0 pid/thread >> 4845/140084804294400 flags 0 priority 100 >> 97 dd=179 locks held 1 write locks 0 pid/thread >> 4845/140084712040192 flags 10 priority 100 >> 97 READ 1 HELD userRoot/uid.db handle 0 >> 98 dd=178 locks held 0 write locks 0 pid/thread >> 4845/140084552578816 flags 0 priority 100 >> 99 dd=177 locks held 0 write locks 0 pid/thread >> 4845/140084577756928 flags 0 priority 100 >> 9a dd=176 locks held 0 write locks 0 pid/thread >> 4845/140084602935040 flags 0 priority 100 >> 9b dd=175 locks held 1 write locks 0 pid/thread >> 4845/140085076068096 flags 10 priority 100 >> 9b READ 1 HELD userRoot/uniquemember.db handle 0 >> 9c dd=174 locks held 1 write locks 0 pid/thread >> 4845/140085076068096 flags 10 priority 100 >> 9c READ 1 HELD userRoot/owner.db handle 0 >> 9d dd=173 locks held 1 write locks 0 pid/thread >> 4845/140085076068096 flags 10 priority 100 >> 9d READ 1 HELD userRoot/ipaallowedtarget.db >> handle 0 >> 9e dd=172 locks held 1 write locks 0 pid/thread >> 4845/140085076068096 flags 10 priority 100 >> 9e READ 1 HELD userRoot/ipaMemberCertProfile.db >> handle 0 >> 9f dd=171 locks held 1 write locks 0 pid/thread >> 4845/140085076068096 flags 10 priority 100 >> 9f READ 1 HELD userRoot/ipaassignedidview.db >> handle 0 >> a0 dd=170 locks held 1 write locks 0 pid/thread >> 4845/140085076068096 flags 10 priority 100 >> a0 READ 1 HELD userRoot/secretary.db handle 0 >> a1 dd=169 locks held 1 write locks 0 pid/thread >> 4845/140085076068096 flags 10 priority 100 >> a1 READ 1 HELD userRoot/memberdenycmd.db handle 0 >> a2 dd=168 locks held 1 write locks 0 pid/thread >> 4845/140085076068096 flags 10 priority 100 >> a2 READ 1 HELD userRoot/memberallowcmd.db handle >> 0 >> a3 dd=167 locks held 1 write locks 0 pid/thread >> 4845/140085076068096 flags 10 priority 100 >> a3 READ 1 HELD userRoot/manager.db handle 0 >> a4 dd=166 locks held 1 write locks 0 pid/thread >> 4845/140085076068096 flags 10 priority 100 >> a4 READ 1 HELD userRoot/ipasudorunas.db handle 0 >> a5 dd=165 locks held 1 write locks 0 pid/thread >> 4845/140085076068096 flags 10 priority 100 >> a5 READ 1 HELD userRoot/ipaMemberCa.db handle 0 >> a6 dd=164 locks held 1 write locks 0 pid/thread >> 4845/140085076068096 flags 10 priority 100 >> a6 READ 1 HELD userRoot/ipasudorunasgroup.db >> handle 0 >> a7 dd=163 locks held 1 write locks 0 pid/thread >> 4845/140085076068096 flags 10 priority 100 >> a7 READ 1 HELD userRoot/sourcehost.db handle 0 >> a8 dd=162 locks held 1 write locks 0 pid/thread >> 4845/140085076068096 flags 10 priority 100 >> a8 READ 1 HELD userRoot/memberservice.db handle 0 >> a9 dd=161 locks held 1 write locks 0 pid/thread >> 4845/140085076068096 flags 10 priority 100 >> a9 READ 1 HELD userRoot/ipatokenradiusconfiglink.db >> handle 0 >> aa dd=160 locks held 0 write locks 0 pid/thread >> 4845/140084703647488 flags 0 priority 100 >> ab dd=159 locks held 0 write locks 0 pid/thread >> 4845/140084821079808 flags 0 priority 100 >> ac dd=158 locks held 0 write locks 0 pid/thread >> 4845/140084628113152 flags 0 priority 100 >> ad dd=157 locks held 0 write locks 0 pid/thread >> 4845/140084829472512 flags 0 priority 100 >> ae dd=156 locks held 0 write locks 0 pid/thread >> 4845/140084829472512 flags 0 priority 100 >> af dd=155 locks held 0 write locks 0 pid/thread >> 4845/140084854650624 flags 0 priority 100 >> b0 dd=154 locks held 0 write locks 0 pid/thread >> 4845/140084854650624 flags 0 priority 100 >> b1 dd=153 locks held 0 write locks 0 pid/thread >> 4845/140084837865216 flags 0 priority 100 >> b2 dd=152 locks held 0 write locks 0 pid/thread >> 4845/140084804294400 flags 0 priority 100 >> b3 dd=151 locks held 0 write locks 0 pid/thread >> 4845/140084812687104 flags 0 priority 100 >> b4 dd=150 locks held 0 write locks 0 pid/thread >> 4845/140084804294400 flags 0 priority 100 >> b5 dd=149 locks held 0 write locks 0 pid/thread >> 4845/140084837865216 flags 0 priority 100 >> b6 dd=148 locks held 0 write locks 0 pid/thread >> 4845/140084569364224 flags 0 priority 100 >> b7 dd=147 locks held 0 write locks 0 pid/thread >> 4845/140084829472512 flags 0 priority 100 >> b8 dd=146 locks held 0 write locks 0 pid/thread >> 4845/140084586149632 flags 0 priority 100 >> b9 dd=145 locks held 0 write locks 0 pid/thread >> 4845/140084653291264 flags 0 priority 100 >> ba dd=144 locks held 0 write locks 0 pid/thread >> 4845/140084636505856 flags 0 priority 100 >> bb dd=143 locks held 0 write locks 0 pid/thread >> 4845/140084804294400 flags 0 priority 100 >> bc dd=142 locks held 0 write locks 0 pid/thread >> 4845/140084804294400 flags 0 priority 100 >> bd dd=141 locks held 0 write locks 0 pid/thread >> 4845/140084695254784 flags 0 priority 100 >> be dd=140 locks held 0 write locks 0 pid/thread >> 4845/140084670076672 flags 0 priority 100 >> bf dd=139 locks held 0 write locks 0 pid/thread >> 4845/140084628113152 flags 0 priority 100 >> c0 dd=138 locks held 0 write locks 0 pid/thread >> 4845/140084661683968 flags 0 priority 100 >> c1 dd=137 locks held 0 write locks 0 pid/thread >> 4845/140084712040192 flags 0 priority 100 >> c2 dd=136 locks held 0 write locks 0 pid/thread >> 4845/140084661683968 flags 0 priority 100 >> c3 dd=135 locks held 0 write locks 0 pid/thread >> 4845/140084577756928 flags 0 priority 100 >> c4 dd=134 locks held 0 write locks 0 pid/thread >> 4845/140084854650624 flags 0 priority 100 >> c5 dd=133 locks held 1 write locks 0 pid/thread >> 4845/140084644898560 flags 10 priority 100 >> c5 READ 1 HELD userRoot/uidnumber.db handle 0 >> c6 dd=132 locks held 0 write locks 0 pid/thread >> 4845/140084594542336 flags 0 priority 100 >> c8 dd=131 locks held 1 write locks 0 pid/thread >> 4845/140084678469376 flags 10 priority 100 >> c8 READ 1 HELD ipaca/certstatus.db handle 0 >> c9 dd=130 locks held 0 write locks 0 pid/thread >> 4845/140084720432896 flags 0 priority 100 >> ca dd=129 locks held 0 write locks 0 pid/thread >> 4845/140084619720448 flags 0 priority 100 >> cb dd=128 locks held 0 write locks 0 pid/thread >> 4845/140084821079808 flags 0 priority 100 >> cc dd=127 locks held 0 write locks 0 pid/thread >> 4845/140084594542336 flags 0 priority 100 >> cd dd=126 locks held 0 write locks 0 pid/thread >> 4845/140084560971520 flags 0 priority 100 >> ce dd=125 locks held 1 write locks 0 pid/thread >> 4845/140084837865216 flags 10 priority 100 >> ce READ 1 HELD userRoot/gidnumber.db handle 0 >> cf dd=124 locks held 0 write locks 0 pid/thread >> 4845/140084720432896 flags 0 priority 100 >> d0 dd=123 locks held 0 write locks 0 pid/thread >> 4845/140084569364224 flags 0 priority 100 >> d1 dd=122 locks held 1 write locks 0 pid/thread >> 4845/140084678469376 flags 10 priority 100 >> d1 READ 1 HELD userRoot/nscpEntryDN.db handle 0 >> d2 dd=121 locks held 0 write locks 0 pid/thread >> 4845/140084695254784 flags 0 priority 100 >> d3 dd=120 locks held 0 write locks 0 pid/thread >> 4845/140084569364224 flags 0 priority 100 >> d4 dd=119 locks held 0 write locks 0 pid/thread >> 4845/140084653291264 flags 0 priority 100 >> d5 dd=118 locks held 0 write locks 0 pid/thread >> 4845/140084829472512 flags 0 priority 100 >> d6 dd=117 locks held 0 write locks 0 pid/thread >> 4845/140084661683968 flags 0 priority 100 >> d7 dd=116 locks held 0 write locks 0 pid/thread >> 4845/140084804294400 flags 0 priority 100 >> d8 dd=115 locks held 1 write locks 0 pid/thread >> 4845/140084821079808 flags 10 priority 100 >> d8 READ 1 HELD ipaca/description.db handle 0 >> d9 dd=114 locks held 0 write locks 0 pid/thread >> 4845/140084661683968 flags 0 priority 100 >> da dd=113 locks held 0 write locks 0 pid/thread >> 4845/140084837865216 flags 0 priority 100 >> db dd=112 locks held 1 write locks 0 pid/thread >> 4845/140084678469376 flags 10 priority 100 >> db READ 1 HELD ipaca/cn.db handle 0 >> dc dd=111 locks held 0 write locks 0 pid/thread >> 4845/140084577756928 flags 0 priority 100 >> dd dd=110 locks held 1 write locks 0 pid/thread >> 4845/140084678469376 flags 10 priority 100 >> dd READ 1 HELD ipaca/ancestorid.db handle 0 >> de dd=109 locks held 0 write locks 0 pid/thread >> 4845/140084577756928 flags 0 priority 100 >> df dd=108 locks held 0 write locks 0 pid/thread >> 4845/140084703647488 flags 0 priority 100 >> e0 dd=107 locks held 0 write locks 0 pid/thread >> 4845/140084569364224 flags 0 priority 100 >> e1 dd=106 locks held 0 write locks 0 pid/thread >> 4845/140084569364224 flags 0 priority 100 >> e2 dd=105 locks held 0 write locks 0 pid/thread >> 4845/140084821079808 flags 0 priority 100 >> e3 dd=104 locks held 1 write locks 0 pid/thread >> 4845/140084686862080 flags 10 priority 100 >> e3 READ 1 HELD ipaca/serialno.db handle 0 >> e4 dd=103 locks held 0 write locks 0 pid/thread >> 4845/140084837865216 flags 0 priority 100 >> e5 dd=102 locks held 1 write locks 0 pid/thread >> 4845/140084695254784 flags 10 priority 100 >> e5 READ 1 HELD ipaca/revInfo.db handle 0 >> e6 dd=101 locks held 1 write locks 0 pid/thread >> 4845/140084695254784 flags 10 priority 100 >> e6 READ 1 HELD ipaca/revokedby.db handle 0 >> e7 dd=100 locks held 1 write locks 0 pid/thread >> 4845/140084695254784 flags 10 priority 100 >> e7 READ 1 HELD ipaca/revokedOn.db handle 0 >> e8 dd=99 locks held 0 write locks 0 pid/thread >> 4845/140084829472512 flags 0 priority 100 >> e9 dd=98 locks held 0 write locks 0 pid/thread >> 4845/140084846257920 flags 0 priority 100 >> ea dd=97 locks held 1 write locks 0 pid/thread >> 4845/140084703647488 flags 10 priority 100 >> ea READ 1 HELD ipaca/nsuniqueid.db handle 0 >> eb dd=96 locks held 1 write locks 0 pid/thread >> 4845/140084703647488 flags 10 priority 100 >> eb READ 1 HELD ipaca/requestid.db handle 0 >> ec dd=95 locks held 1 write locks 0 pid/thread >> 4845/140084703647488 flags 10 priority 100 >> ec READ 1 HELD ipaca/requeststate.db handle 0 >> ed dd=94 locks held 1 write locks 0 pid/thread >> 4845/140084703647488 flags 10 priority 100 >> ed READ 1 HELD ipaca/dateOfCreate.db handle 0 >> ee dd=93 locks held 1 write locks 0 pid/thread >> 4845/140084703647488 flags 10 priority 100 >> ee READ 1 HELD ipaca/requesttype.db handle 0 >> ef dd=92 locks held 1 write locks 0 pid/thread >> 4845/140084703647488 flags 10 priority 100 >> ef READ 1 HELD ipaca/numsubordinates.db handle 0 >> f0 dd=91 locks held 1 write locks 0 pid/thread >> 4845/140084611327744 flags 10 priority 100 >> f0 READ 1 HELD userRoot/userCertificate.db >> handle 0 >> f1 dd=90 locks held 0 write locks 0 pid/thread >> 4845/140084821079808 flags 0 priority 100 >> f2 dd=89 locks held 0 write locks 0 pid/thread >> 4845/140084569364224 flags 0 priority 100 >> f3 dd=88 locks held 0 write locks 0 pid/thread >> 4845/140084602935040 flags 0 priority 100 >> f4 dd=87 locks held 0 write locks 0 pid/thread >> 4845/140084720432896 flags 0 priority 100 >> f5 dd=86 locks held 0 write locks 0 pid/thread >> 4845/140084628113152 flags 0 priority 100 >> f6 dd=85 locks held 0 write locks 0 pid/thread >> 4845/140084703647488 flags 0 priority 100 >> f7 dd=84 locks held 0 write locks 0 pid/thread >> 4845/140084611327744 flags 0 priority 100 >> f8 dd=83 locks held 0 write locks 0 pid/thread >> 4845/140084661683968 flags 0 priority 100 >> f9 dd=82 locks held 0 write locks 0 pid/thread >> 4845/140084837865216 flags 0 priority 100 >> fa dd=81 locks held 0 write locks 0 pid/thread >> 4845/140084544186112 flags 0 priority 100 >> fb dd=80 locks held 0 write locks 0 pid/thread >> 4845/140084703647488 flags 0 priority 100 >> fc dd=79 locks held 0 write locks 0 pid/thread >> 4845/140084544186112 flags 0 priority 100 >> fd dd=78 locks held 0 write locks 0 pid/thread >> 4845/140084535793408 flags 0 priority 100 >> fe dd=77 locks held 0 write locks 0 pid/thread >> 4845/140084586149632 flags 0 priority 100 >> ff dd=76 locks held 0 write locks 0 pid/thread >> 4845/140084569364224 flags 0 priority 100 >> 100 dd=75 locks held 0 write locks 0 pid/thread >> 4845/140084804294400 flags 0 priority 100 >> 101 dd=74 locks held 0 write locks 0 pid/thread >> 4845/140084703647488 flags 0 priority 100 >> 102 dd=73 locks held 0 write locks 0 pid/thread >> 4845/140084703647488 flags 0 priority 100 >> 103 dd=72 locks held 0 write locks 0 pid/thread >> 4845/140084821079808 flags 0 priority 100 >> 104 dd=71 locks held 0 write locks 0 pid/thread >> 4845/140084695254784 flags 0 priority 100 >> 105 dd=70 locks held 0 write locks 0 pid/thread >> 4845/140084544186112 flags 0 priority 100 >> 106 dd=69 locks held 0 write locks 0 pid/thread >> 4845/140084854650624 flags 0 priority 100 >> 107 dd=68 locks held 0 write locks 0 pid/thread >> 4845/140084628113152 flags 0 priority 100 >> 108 dd=67 locks held 0 write locks 0 pid/thread >> 4845/140084569364224 flags 0 priority 100 >> 109 dd=66 locks held 0 write locks 0 pid/thread >> 4845/140084594542336 flags 0 priority 100 >> 10a dd=65 locks held 0 write locks 0 pid/thread >> 4845/140084661683968 flags 0 priority 100 >> 10b dd=64 locks held 0 write locks 0 pid/thread >> 4845/140084653291264 flags 0 priority 100 >> 10c dd=63 locks held 0 write locks 0 pid/thread >> 4845/140084695254784 flags 0 priority 100 >> 10d dd=62 locks held 0 write locks 0 pid/thread >> 4845/140084720432896 flags 0 priority 100 >> 10e dd=61 locks held 0 write locks 0 pid/thread >> 4845/140084619720448 flags 0 priority 100 >> 10f dd=60 locks held 0 write locks 0 pid/thread >> 4845/140084720432896 flags 0 priority 100 >> 110 dd=59 locks held 0 write locks 0 pid/thread >> 4845/140084569364224 flags 0 priority 100 >> 111 dd=58 locks held 0 write locks 0 pid/thread >> 4845/140084695254784 flags 0 priority 100 >> 112 dd=57 locks held 0 write locks 0 pid/thread >> 4845/140084653291264 flags 0 priority 100 >> 113 dd=56 locks held 0 write locks 0 pid/thread >> 4845/140084712040192 flags 0 priority 100 >> 114 dd=55 locks held 0 write locks 0 pid/thread >> 4845/140084686862080 flags 0 priority 100 >> 115 dd=54 locks held 0 write locks 0 pid/thread >> 4845/140084577756928 flags 0 priority 100 >> 115 READ 1 WAIT userRoot/objectclass.db page 9 >> 116 dd=53 locks held 0 write locks 0 pid/thread >> 4845/140084837865216 flags 0 priority 100 >> 116 READ 1 WAIT userRoot/objectclass.db page 9 >> 117 dd=52 locks held 0 write locks 0 pid/thread >> 4845/140084577756928 flags 0 priority 100 >> 118 dd=51 locks held 0 write locks 0 pid/thread >> 4845/140084569364224 flags 0 priority 100 >> 119 dd=50 locks held 0 write locks 0 pid/thread >> 4845/140084560971520 flags 0 priority 100 >> 11a dd=49 locks held 1 write locks 0 pid/thread >> 4845/140084695254784 flags 10 priority 100 >> 11a READ 1 HELD ipaca/metaInfo.db handle 0 >> 11b dd=48 locks held 1 write locks 0 pid/thread >> 4845/140084695254784 flags 10 priority 100 >> 11b READ 1 HELD ipaca/notbefore.db handle 0 >> 11c dd=47 locks held 1 write locks 0 pid/thread >> 4845/140084695254784 flags 10 priority 100 >> 11c READ 1 HELD ipaca/notafter.db handle 0 >> 11d dd=46 locks held 1 write locks 0 pid/thread >> 4845/140084695254784 flags 10 priority 100 >> 11d READ 1 HELD ipaca/duration.db handle 0 >> 11e dd=45 locks held 1 write locks 0 pid/thread >> 4845/140084695254784 flags 10 priority 100 >> 11e READ 1 HELD ipaca/subjectname.db handle 0 >> 11f dd=44 locks held 1 write locks 0 pid/thread >> 4845/140084695254784 flags 10 priority 100 >> 11f READ 1 HELD ipaca/publicKeyData.db handle 0 >> 120 dd=43 locks held 1 write locks 0 pid/thread >> 4845/140084695254784 flags 10 priority 100 >> 120 READ 1 HELD ipaca/extension.db handle 0 >> 121 dd=42 locks held 1 write locks 0 pid/thread >> 4845/140084695254784 flags 10 priority 100 >> 121 READ 1 HELD ipaca/issuedby.db handle 0 >> 122 dd=41 locks held 0 write locks 0 pid/thread >> 4845/140084712040192 flags 0 priority 100 >> 123 dd=40 locks held 0 write locks 0 pid/thread >> 4845/140084812687104 flags 0 priority 100 >> 124 dd=39 locks held 0 write locks 0 pid/thread >> 4845/140084544186112 flags 0 priority 100 >> 125 dd=38 locks held 0 write locks 0 pid/thread >> 4845/140084686862080 flags 0 priority 100 >> 126 dd=37 locks held 0 write locks 0 pid/thread >> 4845/140084552578816 flags 0 priority 100 >> 128 dd=36 locks held 0 write locks 0 pid/thread >> 4845/140084686862080 flags 0 priority 100 >> 12a dd=35 locks held 0 write locks 0 pid/thread >> 4845/140084720432896 flags 0 priority 100 >> 12b dd=34 locks held 0 write locks 0 pid/thread >> 4845/140084552578816 flags 0 priority 100 >> 12c dd=33 locks held 0 write locks 0 pid/thread >> 4845/140084670076672 flags 0 priority 100 >> 12d dd=32 locks held 0 write locks 0 pid/thread >> 4845/140084577756928 flags 0 priority 100 >> 12e dd=31 locks held 0 write locks 0 pid/thread >> 4845/140084712040192 flags 0 priority 100 >> 12f dd=30 locks held 0 write locks 0 pid/thread >> 4845/140084670076672 flags 0 priority 100 >> 130 dd=29 locks held 0 write locks 0 pid/thread >> 4845/140084577756928 flags 0 priority 100 >> 131 dd=28 locks held 0 write locks 0 pid/thread >> 4845/140084560971520 flags 0 priority 100 >> 132 dd=27 locks held 0 write locks 0 pid/thread >> 4845/140084670076672 flags 0 priority 100 >> 133 dd=26 locks held 0 write locks 0 pid/thread >> 4845/140084602935040 flags 0 priority 100 >> 134 dd=25 locks held 0 write locks 0 pid/thread >> 4845/140084552578816 flags 0 priority 100 >> 135 dd=24 locks held 0 write locks 0 pid/thread >> 4845/140084678469376 flags 0 priority 100 >> 136 dd=23 locks held 0 write locks 0 pid/thread >> 4845/140084544186112 flags 0 priority 100 >> 137 dd=22 locks held 0 write locks 0 pid/thread >> 4845/140084577756928 flags 0 priority 100 >> 138 dd=21 locks held 0 write locks 0 pid/thread >> 4845/140084703647488 flags 0 priority 100 >> 139 dd=20 locks held 0 write locks 0 pid/thread >> 4845/140084678469376 flags 0 priority 100 >> 13b dd=19 locks held 0 write locks 0 pid/thread >> 4845/140084812687104 flags 0 priority 100 >> 144 dd=18 locks held 0 write locks 0 pid/thread >> 4845/140084670076672 flags 0 priority 100 >> 156 dd=17 locks held 0 write locks 0 pid/thread >> 4845/140084594542336 flags 0 priority 100 >> 15a dd=16 locks held 0 write locks 0 pid/thread >> 4845/140084577756928 flags 0 priority 100 >> 168 dd=15 locks held 0 write locks 0 pid/thread >> 4845/140084653291264 flags 0 priority 100 >> 16a dd=14 locks held 0 write locks 0 pid/thread >> 4845/140084829472512 flags 0 priority 100 >> 17d dd=13 locks held 0 write locks 0 pid/thread >> 4845/140084829472512 flags 0 priority 100 >> 17e dd=12 locks held 0 write locks 0 pid/thread >> 4845/140084829472512 flags 0 priority 100 >> 17f dd=11 locks held 0 write locks 0 pid/thread >> 4845/140084586149632 flags 0 priority 100 >> 180 dd=10 locks held 0 write locks 0 pid/thread >> 4845/140084594542336 flags 0 priority 100 >> 181 dd= 9 locks held 0 write locks 0 pid/thread >> 4845/140084577756928 flags 0 priority 100 >> 193 dd= 8 locks held 0 write locks 0 pid/thread >> 4845/140084812687104 flags 0 priority 100 >> 196 dd= 7 locks held 0 write locks 0 pid/thread >> 4845/140084695254784 flags 0 priority 100 >> 1a1 dd= 6 locks held 0 write locks 0 pid/thread >> 4845/140084720432896 flags 0 priority 100 >> 1a2 dd= 5 locks held 0 write locks 0 pid/thread >> 4845/140084577756928 flags 0 priority 100 >> 1a3 dd= 4 locks held 0 write locks 0 pid/thread >> 4845/140084812687104 flags 0 priority 100 >> 1a4 dd= 3 locks held 0 write locks 0 pid/thread >> 4845/140084812687104 flags 0 priority 100 >> 1a5 dd= 2 locks held 0 write locks 0 pid/thread >> 4845/140084577756928 flags 0 priority 100 >> 1a6 dd= 1 locks held 0 write locks 0 pid/thread >> 4845/140084594542336 flags 0 priority 100 >> 800b3a33 dd= 0 locks held 26 write locks 22 pid/thread >> 4845/140084661683968 flags 0 priority 100 >> 800b3a33 WRITE 1 HELD userRoot/objectclass.db page 9 >> 800b3a33 WRITE 1 HELD userRoot/objectclass.db page 8 >> 800b3a33 WRITE 1 HELD userRoot/objectclass.db page 18 >> 800b3a33 WRITE 1 HELD userRoot/objectclass.db page 7 >> 800b3a33 WRITE 1 HELD userRoot/objectclass.db page 14 >> 800b3a33 WRITE 1 HELD userRoot/objectclass.db page 11 >> 800b3a33 WRITE 2 HELD userRoot/objectclass.db page 3 >> 800b3a33 WRITE 1 HELD userRoot/cn.db page 150 >> 800b3a33 WRITE 1 HELD userRoot/cn.db page 89 >> 800b3a33 WRITE 1 HELD userRoot/cn.db page 53 >> 800b3a33 WRITE 1 HELD userRoot/cn.db page 131 >> 800b3a33 WRITE 1 HELD userRoot/cn.db page 117 >> 800b3a33 WRITE 1 HELD userRoot/cn.db page 11 >> 800b3a33 WRITE 2 HELD userRoot/cn.db page 141 >> 800b3a33 WRITE 1 HELD userRoot/cn.db page 41 >> 800b3a33 WRITE 1 HELD userRoot/cn.db page 40 >> 800b3a33 WRITE 1 HELD userRoot/cn.db page 143 >> 800b3a33 WRITE 1 HELD userRoot/cn.db page 30 >> 800b3a33 WRITE 2 HELD userRoot/id2entry.db page 8184 >> 800b3a33 WRITE 1 HELD userRoot/id2entry.db page 8183 >> 800b3a33 WRITE 1 HELD userRoot/id2entry.db page 7779 >> 800b3a33 WRITE 1 HELD userRoot/id2entry.db page 0 >> 800b3a33 READ 3 HELD userRoot/entryrdn.db page 3 >> 800b3a33 READ 1 HELD userRoot/entryrdn.db page 36 >> 800b3a33 READ 1 HELD userRoot/entryrdn.db page 4 >> 800b3a33 READ 1 HELD userRoot/nsuniqueid.db page 86 >> =-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-= >> Locks grouped by object: >> Locker Mode Count Status ----------------- Object --------------- >> a4 READ 1 HELD userRoot/ipasudorunas.db handle 0 >> >> 9e READ 1 HELD userRoot/ipaMemberCertProfile.db >> handle 0 >> >> 10 READ 1 HELD ipaca/vlv#allrevokedcertspkitomcatindex.db >> handle 0 >> >> 800b3a33 WRITE 2 HELD userRoot/id2entry.db page 8184 >> >> 800b3a33 WRITE 1 HELD userRoot/id2entry.db page 8183 >> >> 800b3a33 WRITE 1 HELD userRoot/id2entry.db page 7779 >> >> a1 READ 1 HELD userRoot/memberdenycmd.db handle 0 >> >> a READ 1 HELD ipaca/vlv#allcertspkitomcatindex.db >> handle 0 >> >> a7 READ 1 HELD userRoot/sourcehost.db handle 0 >> >> 57 READ 1 HELD changelog/ancestorid.db handle 0 >> >> e5 READ 1 HELD ipaca/revInfo.db handle 0 >> >> e6 READ 1 HELD ipaca/revokedby.db handle 0 >> >> 11 READ 1 HELD ipaca/vlv#allrevokedcertsnotafterpkitomcatindex.db >> handle 0 >> >> 4e READ 1 HELD userRoot/parentid.db handle 0 >> >> 4 READ 1 HELD ipaca/id2entry.db handle 0 >> >> 58 READ 1 HELD changelog/numsubordinates.db >> handle 0 >> >> 67 READ 1 HELD ipaca/seeAlso.db handle 0 >> >> 2b READ 1 HELD ipaca/vlv#carevocationpkitomcatindex.db >> handle 0 >> >> ec READ 1 HELD ipaca/requeststate.db handle 0 >> >> 11b READ 1 HELD ipaca/notbefore.db handle 0 >> >> 7e READ 1 HELD userRoot/numsubordinates.db >> handle 0 >> >> a2 READ 1 HELD userRoot/memberallowcmd.db handle >> 0 >> >> 9c READ 1 HELD userRoot/owner.db handle 0 >> >> 1e READ 1 HELD ipaca/vlv#cacompleteenrollmentpkitomcatindex.db >> handle 0 >> >> 11c READ 1 HELD ipaca/notafter.db handle 0 >> >> 40 READ 1 HELD changelog/objectclass.db handle 0 >> >> 2e READ 1 HELD changelog/entryusn.db handle 0 >> >> 16 READ 1 HELD ipaca/vlv#allvalidcertsnotafterpkitomcatindex.db >> handle 0 >> >> 15 READ 1 HELD ipaca/vlv#allvalidcertspkitomcatindex.db >> handle 0 >> >> a5 READ 1 HELD userRoot/ipaMemberCa.db handle 0 >> >> 18 READ 1 HELD ipaca/vlv#caallpkitomcatindex.db >> handle 0 >> >> a0 READ 1 HELD userRoot/secretary.db handle 0 >> >> 81 READ 1 HELD userRoot/memberUser.db handle 0 >> >> 9f READ 1 HELD userRoot/ipaassignedidview.db >> handle 0 >> >> a3 READ 1 HELD userRoot/manager.db handle 0 >> >> d8 READ 1 HELD ipaca/description.db handle 0 >> >> a8 READ 1 HELD userRoot/memberservice.db handle 0 >> >> 44 READ 1 HELD ipaca/objectclass.db handle 0 >> >> ed READ 1 HELD ipaca/dateOfCreate.db handle 0 >> >> 800b3a33 WRITE 1 HELD userRoot/id2entry.db page 0 >> >> 2 READ 1 HELD userRoot/id2entry.db handle 0 >> >> 5b READ 1 HELD userRoot/krbPrincipalName.db >> handle 0 >> >> ee READ 1 HELD ipaca/requesttype.db handle 0 >> >> c8 READ 1 HELD ipaca/certstatus.db handle 0 >> >> 56 READ 1 HELD changelog/parentid.db handle 0 >> >> 83 READ 1 HELD userRoot/memberHost.db handle 0 >> >> 5e READ 1 HELD userRoot/ipakrbprincipalalias.db >> handle 0 >> >> 121 READ 1 HELD ipaca/issuedby.db handle 0 >> >> 6 READ 1 HELD ipaca/entryrdn.db handle 0 >> >> e7 READ 1 HELD ipaca/revokedOn.db handle 0 >> >> 4c READ 1 HELD ipaca/aci.db handle 0 >> >> 11d READ 1 HELD ipaca/duration.db handle 0 >> >> 14 READ 1 HELD ipaca/vlv#allrevokedorrevokedexpiredcertspkitomcatindex.db >> handle 0 >> >> 21 READ 1 HELD ipaca/vlv#caenrollmentpkitomcatindex.db >> handle 0 >> >> 55 READ 1 HELD changelog/targetuniqueid.db >> handle 0 >> >> 800b3a33 READ 1 HELD userRoot/entryrdn.db page 36 >> >> 2c READ 1 HELD changelog/id2entry.db handle 0 >> >> 800b3a33 READ 1 HELD userRoot/entryrdn.db page 4 >> >> 34 READ 1 HELD userRoot/entryrdn.db handle 0 >> >> 800b3a33 READ 3 HELD userRoot/entryrdn.db page 3 >> >> eb READ 1 HELD ipaca/requestid.db handle 0 >> >> 49 READ 1 HELD userRoot/aci.db handle 0 >> >> 76 READ 1 HELD userRoot/ipauniqueid.db handle 0 >> >> 63 READ 1 HELD changelog/seeAlso.db handle 0 >> >> 7c READ 1 HELD userRoot/fqdn.db handle 0 >> >> db READ 1 HELD ipaca/cn.db handle 0 >> >> ef READ 1 HELD ipaca/numsubordinates.db handle 0 >> >> 32 READ 1 HELD ipaca/entryusn.db handle 0 >> >> 1d READ 1 HELD ipaca/vlv#cacompletepkitomcatindex.db >> handle 0 >> >> dd READ 1 HELD ipaca/ancestorid.db handle 0 >> >> a9 READ 1 HELD userRoot/ipatokenradiusconfiglink.db >> handle 0 >> >> f0 READ 1 HELD userRoot/userCertificate.db >> handle 0 >> >> 11a READ 1 HELD ipaca/metaInfo.db handle 0 >> >> 7d READ 1 HELD userRoot/managedby.db handle 0 >> >> ea READ 1 HELD ipaca/nsuniqueid.db handle 0 >> >> c5 READ 1 HELD userRoot/uidnumber.db handle 0 >> >> d1 READ 1 HELD userRoot/nscpEntryDN.db handle 0 >> >> 120 READ 1 HELD ipaca/extension.db handle 0 >> >> 800b3a33 WRITE 1 HELD userRoot/cn.db page 150 >> >> 69 READ 1 HELD ipaca/parentid.db handle 0 >> >> 800b3a33 WRITE 1 HELD userRoot/cn.db page 131 >> >> 800b3a33 WRITE 1 HELD userRoot/cn.db page 143 >> >> 800b3a33 WRITE 2 HELD userRoot/cn.db page 141 >> >> 20 READ 1 HELD ipaca/vlv#cacompleterevocationpkitomcatindex.db >> handle 0 >> >> 8e READ 1 HELD userRoot/macAddress.db handle 0 >> >> e READ 1 HELD ipaca/vlv#allnonrevokedcertspkitomcatindex.db >> handle 0 >> >> 17 READ 1 HELD ipaca/vlv#allvalidorrevokedcertspkitomcatindex.db >> handle 0 >> >> 800b3a33 WRITE 1 HELD userRoot/cn.db page 30 >> >> 800b3a33 WRITE 1 HELD userRoot/objectclass.db page 14 >> >> 800b3a33 WRITE 1 HELD userRoot/objectclass.db page 9 >> 116 READ 1 WAIT userRoot/objectclass.db page 9 >> 115 READ 1 WAIT userRoot/objectclass.db page 9 >> 92 READ 1 WAIT userRoot/objectclass.db page 9 >> >> 800b3a33 WRITE 1 HELD userRoot/objectclass.db page 8 >> >> 7b READ 1 HELD userRoot/cn.db handle 0 >> >> 800b3a33 WRITE 1 HELD userRoot/objectclass.db page 11 >> >> 800b3a33 WRITE 1 HELD userRoot/objectclass.db page 7 >> >> 800b3a33 WRITE 1 HELD userRoot/cn.db page 11 >> >> 38 READ 1 HELD userRoot/objectclass.db handle 0 >> >> 800b3a33 WRITE 2 HELD userRoot/objectclass.db page 3 >> >> 800b3a33 WRITE 1 HELD userRoot/cn.db page 53 >> >> 800b3a33 WRITE 1 HELD userRoot/objectclass.db page 18 >> 6e READ 1 WAIT userRoot/objectclass.db page 18 >> 94 READ 1 WAIT userRoot/objectclass.db page 18 >> >> 800b3a33 WRITE 1 HELD userRoot/cn.db page 41 >> >> 800b3a33 WRITE 1 HELD userRoot/cn.db page 40 >> >> 800b3a33 WRITE 1 HELD userRoot/cn.db page 89 >> >> 800b3a33 WRITE 1 HELD userRoot/cn.db page 117 >> >> 30 READ 1 HELD userRoot/entryusn.db handle 0 >> >> 3c READ 1 HELD changelog/entryrdn.db handle 0 >> >> 97 READ 1 HELD userRoot/uid.db handle 0 >> >> 47 READ 1 HELD changelog/aci.db handle 0 >> >> e3 READ 1 HELD ipaca/serialno.db handle 0 >> >> 7a READ 1 HELD userRoot/nsuniqueid.db handle 0 >> >> 800b3a33 READ 1 HELD userRoot/nsuniqueid.db page 86 >> >> 51 READ 1 HELD changelog/nsuniqueid.db handle 0 >> >> 11e READ 1 HELD ipaca/subjectname.db handle 0 >> >> 7f READ 1 HELD userRoot/member.db handle 0 >> >> 11f READ 1 HELD ipaca/publicKeyData.db handle 0 >> >> 9b READ 1 HELD userRoot/uniquemember.db handle 0 >> >> 52 READ 1 HELD changelog/changenumber.db handle 0 >> >> ce READ 1 HELD userRoot/gidnumber.db handle 0 >> >> 65 READ 1 HELD userRoot/seeAlso.db handle 0 >> >> 9d READ 1 HELD userRoot/ipaallowedtarget.db >> handle 0 >> >> a6 READ 1 HELD userRoot/ipasudorunasgroup.db >> handle 0 >> >> 3a READ 1 HELD userRoot/ancestorid.db handle 0 >> >> >> >> >> pstack output >> >> root at ipa-master ]# pstack 4845 >> Thread 44 (Thread 0x7f681dd3a700 (LWP 4847)): >> #0 0x00007f683cd719b3 in select () from /lib64/libc.so.6 >> #1 0x00007f683f4fc0e9 in DS_Sleep () from /usr/lib64/dirsrv/libslapd.so. >> 0 >> #2 0x00007f6832b4a907 in deadlock_threadmain () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #3 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so >> #4 0x00007f683d04cdc5 in start_thread () from /lib64/libpthread.so.0 >> #5 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 >> Thread 43 (Thread 0x7f681d539700 (LWP 4848)): >> #0 0x00007f683cd719b3 in select () from /lib64/libc.so.6 >> #1 0x00007f68376d8bad in __os_yield () from /lib64/libdb-5.3.so >> #2 0x00007f68376d42b3 in __memp_sync_int () from /lib64/libdb-5.3.so >> #3 0x00007f68376e4752 in __txn_checkpoint () from /lib64/libdb-5.3.so >> #4 0x00007f68376e4b74 in __txn_checkpoint_pp () from /lib64/libdb-5.3.so >> #5 0x00007f6832b4ea87 in checkpoint_threadmain () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #6 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so >> #7 0x00007f683d04cdc5 in start_thread () from /lib64/libpthread.so.0 >> #8 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 >> Thread 42 (Thread 0x7f681cd38700 (LWP 4849)): >> #0 0x00007f683cd719b3 in select () from /lib64/libc.so.6 >> #1 0x00007f683f4fc0e9 in DS_Sleep () from /usr/lib64/dirsrv/libslapd.so. >> 0 >> #2 0x00007f6832b4ab7f in trickle_threadmain () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #3 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so >> #4 0x00007f683d04cdc5 in start_thread () from /lib64/libpthread.so.0 >> #5 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 >> Thread 41 (Thread 0x7f681c537700 (LWP 4850)): >> #0 0x00007f683cd719b3 in select () from /lib64/libc.so.6 >> #1 0x00007f683f4fc0e9 in DS_Sleep () from /usr/lib64/dirsrv/libslapd.so. >> 0 >> #2 0x00007f6832b45707 in perf_threadmain () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #3 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so >> #4 0x00007f683d04cdc5 in start_thread () from /lib64/libpthread.so.0 >> #5 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 >> Thread 40 (Thread 0x7f681bb2d700 (LWP 4851)): >> #0 0x00007f683d0506d5 in pthread_cond_wait@@GLIBC_2.3.2 () from >> /lib64/libpthread.so.0 >> #1 0x00007f683d6a6050 in PR_WaitCondVar () from /lib64/libnspr4.so >> #2 0x00007f683f4eb198 in slapi_wait_condvar () from >> /usr/lib64/dirsrv/libslapd.so.0 >> #3 0x00007f683587d62e in cos_cache_wait_on_change () from >> /usr/lib64/dirsrv/plugins/libcos-plugin.so >> #4 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so >> #5 0x00007f683d04cdc5 in start_thread () from /lib64/libpthread.so.0 >> #6 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 >> Thread 39 (Thread 0x7f681b32c700 (LWP 4852)): >> #0 0x00007f683d0506d5 in pthread_cond_wait@@GLIBC_2.3.2 () from >> /lib64/libpthread.so.0 >> #1 0x00007f683d6a6050 in PR_WaitCondVar () from /lib64/libnspr4.so >> #2 0x00007f683f4eb198 in slapi_wait_condvar () from >> /usr/lib64/dirsrv/libslapd.so.0 >> #3 0x00007f68311e8eed in roles_cache_wait_on_change () from >> /usr/lib64/dirsrv/plugins/libroles-plugin.so >> #4 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so >> #5 0x00007f683d04cdc5 in start_thread () from /lib64/libpthread.so.0 >> #6 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 >> Thread 38 (Thread 0x7f681ab2b700 (LWP 4853)): >> #0 0x00007f683d0506d5 in pthread_cond_wait@@GLIBC_2.3.2 () from >> /lib64/libpthread.so.0 >> #1 0x00007f683d6a6050 in PR_WaitCondVar () from /lib64/libnspr4.so >> #2 0x00007f683f4eb198 in slapi_wait_condvar () from >> /usr/lib64/dirsrv/libslapd.so.0 >> #3 0x00007f68311e8eed in roles_cache_wait_on_change () from >> /usr/lib64/dirsrv/plugins/libroles-plugin.so >> #4 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so >> #5 0x00007f683d04cdc5 in start_thread () from /lib64/libpthread.so.0 >> #6 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 >> Thread 37 (Thread 0x7f681a32a700 (LWP 4854)): >> #0 0x00007f683d0506d5 in pthread_cond_wait@@GLIBC_2.3.2 () from >> /lib64/libpthread.so.0 >> #1 0x00007f683d6a6050 in PR_WaitCondVar () from /lib64/libnspr4.so >> #2 0x00007f683f4eb198 in slapi_wait_condvar () from >> /usr/lib64/dirsrv/libslapd.so.0 >> #3 0x00007f68311e8eed in roles_cache_wait_on_change () from >> /usr/lib64/dirsrv/plugins/libroles-plugin.so >> #4 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so >> #5 0x00007f683d04cdc5 in start_thread () from /lib64/libpthread.so.0 >> #6 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 >> Thread 36 (Thread 0x7f6819b29700 (LWP 4855)): >> #0 0x00007f683d050a82 in pthread_cond_timedwait@@GLIBC_2.3.2 () from >> /lib64/libpthread.so.0 >> #1 0x00007f683d6a5b07 in pt_TimedWait () from /lib64/libnspr4.so >> #2 0x00007f683d6a5fce in PR_WaitCondVar () from /lib64/libnspr4.so >> #3 0x00007f683f9a4d03 in housecleaning () >> #4 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so >> #5 0x00007f683d04cdc5 in start_thread () from /lib64/libpthread.so.0 >> #6 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 >> Thread 35 (Thread 0x7f6819328700 (LWP 4856)): >> #0 0x00007f683d050a82 in pthread_cond_timedwait@@GLIBC_2.3.2 () from >> /lib64/libpthread.so.0 >> #1 0x00007f683d6a5b07 in pt_TimedWait () from /lib64/libnspr4.so >> #2 0x00007f683d6a5fce in PR_WaitCondVar () from /lib64/libnspr4.so >> #3 0x00007f683f4911e8 in eq_loop () from /usr/lib64/dirsrv/libslapd.so.0 >> #4 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so >> #5 0x00007f683d04cdc5 in start_thread () from /lib64/libpthread.so.0 >> #6 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 >> Thread 34 (Thread 0x7f680bfff700 (LWP 4859)): >> #0 0x00007f683d0506d5 in pthread_cond_wait@@GLIBC_2.3.2 () from >> /lib64/libpthread.so.0 >> #1 0x00007f683d6a6050 in PR_WaitCondVar () from /lib64/libnspr4.so >> #2 0x00007f683f999fee in connection_wait_for_new_work () >> #3 0x00007f683f99b719 in connection_threadmain () >> #4 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so >> #5 0x00007f683d04cdc5 in start_thread () from /lib64/libpthread.so.0 >> #6 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 >> Thread 33 (Thread 0x7f680b7fe700 (LWP 4860)): >> #0 0x00007f683d0506d5 in pthread_cond_wait@@GLIBC_2.3.2 () from >> /lib64/libpthread.so.0 >> #1 0x00007f683d6a6050 in PR_WaitCondVar () from /lib64/libnspr4.so >> #2 0x00007f683f999fee in connection_wait_for_new_work () >> #3 0x00007f683f99b719 in connection_threadmain () >> #4 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so >> #5 0x00007f683d04cdc5 in start_thread () from /lib64/libpthread.so.0 >> #6 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 >> Thread 32 (Thread 0x7f680affd700 (LWP 4861)): >> #0 0x00007f683d0506d5 in pthread_cond_wait@@GLIBC_2.3.2 () from >> /lib64/libpthread.so.0 >> #1 0x00007f68375ac2f3 in __db_hybrid_mutex_suspend () from /lib64/ >> libdb-5.3.so >> #2 0x00007f68375ab640 in __db_tas_mutex_lock () from /lib64/libdb-5.3.so >> #3 0x00007f6837655d3a in __lock_get_internal () from /lib64/libdb-5.3.so >> #4 0x00007f6837656820 in __lock_get () from /lib64/libdb-5.3.so >> #5 0x00007f6837682142 in __db_lget () from /lib64/libdb-5.3.so >> #6 0x00007f68375c9605 in __bam_search () from /lib64/libdb-5.3.so >> #7 0x00007f68375b4256 in __bamc_search () from /lib64/libdb-5.3.so >> #8 0x00007f68375b5d0f in __bamc_get () from /lib64/libdb-5.3.so >> #9 0x00007f683766eca6 in __dbc_iget () from /lib64/libdb-5.3.so >> #10 0x00007f683767db02 in __dbc_get_pp () from /lib64/libdb-5.3.so >> #11 0x00007f6832b59680 in idl_new_fetch () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #12 0x00007f6832b67b26 in index_read_ext_allids () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #13 0x00007f6832b52284 in keys2idl () from /usr/lib64/dirsrv/plugins/libb >> ack-ldbm.so >> #14 0x00007f6832b529e3 in ava_candidates.isra.0 () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #15 0x00007f6832b52fd2 in filter_candidates_ext () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #16 0x00007f6832b54096 in list_candidates () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #17 0x00007f6832b52f40 in filter_candidates_ext () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #18 0x00007f6832b54096 in list_candidates () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #19 0x00007f6832b52f40 in filter_candidates_ext () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #20 0x00007f6832b54096 in list_candidates () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #21 0x00007f6832b52f40 in filter_candidates_ext () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #22 0x00007f6832b8e80c in subtree_candidates () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #23 0x00007f6832b8fe7f in ldbm_back_search () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #24 0x00007f683f4bd1d6 in op_shared_search () from >> /usr/lib64/dirsrv/libslapd.so.0 >> #25 0x00007f683f9ae33e in do_search () >> #26 0x00007f683f99c3c3 in connection_threadmain () >> #27 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so >> #28 0x00007f683d04cdc5 in start_thread () from /lib64/libpthread.so.0 >> #29 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 >> Thread 31 (Thread 0x7f680a7fc700 (LWP 4862)): >> #0 0x00007f683d0506d5 in pthread_cond_wait@@GLIBC_2.3.2 () from >> /lib64/libpthread.so.0 >> #1 0x00007f683d6a62c3 in PR_EnterMonitor () from /lib64/libnspr4.so >> #2 0x00007f6832b4d3e6 in dblayer_txn_begin () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #3 0x00007f6832b89007 in ldbm_back_modify () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #4 0x00007f683f4b6541 in op_shared_modify () from >> /usr/lib64/dirsrv/libslapd.so.0 >> #5 0x00007f683f4b7004 in modify_internal_pb () from >> /usr/lib64/dirsrv/libslapd.so.0 >> #6 0x00007f683483bbbb in ipalockout_postop () from >> /usr/lib64/dirsrv/plugins/libipa_lockout.so >> #7 0x00007f683f4c8848 in plugin_call_func () from >> /usr/lib64/dirsrv/libslapd.so.0 >> #8 0x00007f683f4c8b13 in plugin_call_plugins () from >> /usr/lib64/dirsrv/libslapd.so.0 >> #9 0x00007f683f9954d8 in do_bind () >> #10 0x00007f683f99c3fd in connection_threadmain () >> #11 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so >> #12 0x00007f683d04cdc5 in start_thread () from /lib64/libpthread.so.0 >> #13 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 >> Thread 30 (Thread 0x7f6809ffb700 (LWP 4863)): >> #0 0x00007f683d0506d5 in pthread_cond_wait@@GLIBC_2.3.2 () from >> /lib64/libpthread.so.0 >> #1 0x00007f68375ac2f3 in __db_hybrid_mutex_suspend () from /lib64/ >> libdb-5.3.so >> #2 0x00007f68375ab640 in __db_tas_mutex_lock () from /lib64/libdb-5.3.so >> #3 0x00007f6837655d3a in __lock_get_internal () from /lib64/libdb-5.3.so >> #4 0x00007f6837656820 in __lock_get () from /lib64/libdb-5.3.so >> #5 0x00007f6837682142 in __db_lget () from /lib64/libdb-5.3.so >> #6 0x00007f68375c9605 in __bam_search () from /lib64/libdb-5.3.so >> #7 0x00007f68375b4256 in __bamc_search () from /lib64/libdb-5.3.so >> #8 0x00007f68375b5d0f in __bamc_get () from /lib64/libdb-5.3.so >> #9 0x00007f683766eca6 in __dbc_iget () from /lib64/libdb-5.3.so >> #10 0x00007f683767db02 in __dbc_get_pp () from /lib64/libdb-5.3.so >> #11 0x00007f6832b59680 in idl_new_fetch () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #12 0x00007f6832b67b26 in index_read_ext_allids () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #13 0x00007f6832b52284 in keys2idl () from /usr/lib64/dirsrv/plugins/libb >> ack-ldbm.so >> #14 0x00007f6832b529e3 in ava_candidates.isra.0 () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #15 0x00007f6832b52fd2 in filter_candidates_ext () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #16 0x00007f6832b54096 in list_candidates () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #17 0x00007f6832b52f40 in filter_candidates_ext () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #18 0x00007f6832b8e80c in subtree_candidates () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #19 0x00007f6832b8fe7f in ldbm_back_search () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #20 0x00007f683f4bd1d6 in op_shared_search () from >> /usr/lib64/dirsrv/libslapd.so.0 >> #21 0x00007f683f4cd94e in search_internal_callback_pb () from >> /usr/lib64/dirsrv/libslapd.so.0 >> #22 0x00007f683f4cdbe8 in search_internal_pb () from >> /usr/lib64/dirsrv/libslapd.so.0 >> #23 0x00007f683f9ac49b in ids_sasl_canon_user () >> #24 0x00007f683d46c93b in _sasl_canon_user () from /lib64/libsasl2.so.3 >> #25 0x00007f683d46cc4c in _sasl_canon_user_lookup () from >> /lib64/libsasl2.so.3 >> #26 0x00007f6837374fb9 in gssapi_server_mech_step () from >> /usr/lib64/sasl2/libgssapiv2.so >> #27 0x00007f683d478b9b in sasl_server_step () from /lib64/libsasl2.so.3 >> #28 0x00007f683f9ad83b in ids_sasl_check_bind () >> #29 0x00007f683f99561a in do_bind () >> #30 0x00007f683f99c3fd in connection_threadmain () >> #31 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so >> #32 0x00007f683d04cdc5 in start_thread () from /lib64/libpthread.so.0 >> #33 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 >> Thread 29 (Thread 0x7f68097fa700 (LWP 4864)): >> #0 0x00007f683d0506d5 in pthread_cond_wait@@GLIBC_2.3.2 () from >> /lib64/libpthread.so.0 >> #1 0x00007f683d6a6050 in PR_WaitCondVar () from /lib64/libnspr4.so >> #2 0x00007f683f999fee in connection_wait_for_new_work () >> #3 0x00007f683f99b719 in connection_threadmain () >> #4 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so >> #5 0x00007f683d04cdc5 in start_thread () from /lib64/libpthread.so.0 >> #6 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 >> Thread 28 (Thread 0x7f6808ff9700 (LWP 4865)): >> #0 0x00007f683d0506d5 in pthread_cond_wait@@GLIBC_2.3.2 () from >> /lib64/libpthread.so.0 >> #1 0x00007f683d6a6050 in PR_WaitCondVar () from /lib64/libnspr4.so >> #2 0x00007f683f999fee in connection_wait_for_new_work () >> #3 0x00007f683f99b719 in connection_threadmain () >> #4 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so >> #5 0x00007f683d04cdc5 in start_thread () from /lib64/libpthread.so.0 >> #6 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 >> Thread 27 (Thread 0x7f6803fff700 (LWP 4866)): >> #0 0x00007f683d0506d5 in pthread_cond_wait@@GLIBC_2.3.2 () from >> /lib64/libpthread.so.0 >> #1 0x00007f683d6a6050 in PR_WaitCondVar () from /lib64/libnspr4.so >> #2 0x00007f683f999fee in connection_wait_for_new_work () >> #3 0x00007f683f99b719 in connection_threadmain () >> #4 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so >> #5 0x00007f683d04cdc5 in start_thread () from /lib64/libpthread.so.0 >> #6 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 >> Thread 26 (Thread 0x7f68037fe700 (LWP 4867)): >> #0 0x00007f683d0506d5 in pthread_cond_wait@@GLIBC_2.3.2 () from >> /lib64/libpthread.so.0 >> #1 0x00007f683d6a6050 in PR_WaitCondVar () from /lib64/libnspr4.so >> #2 0x00007f683f999fee in connection_wait_for_new_work () >> #3 0x00007f683f99b719 in connection_threadmain () >> #4 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so >> #5 0x00007f683d04cdc5 in start_thread () from /lib64/libpthread.so.0 >> #6 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 >> Thread 25 (Thread 0x7f6802ffd700 (LWP 4868)): >> #0 0x00007f683d0506d5 in pthread_cond_wait@@GLIBC_2.3.2 () from >> /lib64/libpthread.so.0 >> #1 0x00007f683d6a6050 in PR_WaitCondVar () from /lib64/libnspr4.so >> #2 0x00007f683f999fee in connection_wait_for_new_work () >> #3 0x00007f683f99b719 in connection_threadmain () >> #4 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so >> #5 0x00007f683d04cdc5 in start_thread () from /lib64/libpthread.so.0 >> #6 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 >> Thread 24 (Thread 0x7f68027fc700 (LWP 4869)): >> #0 0x00007f683d0506d5 in pthread_cond_wait@@GLIBC_2.3.2 () from >> /lib64/libpthread.so.0 >> #1 0x00007f68375ac2f3 in __db_hybrid_mutex_suspend () from /lib64/ >> libdb-5.3.so >> #2 0x00007f68375abb37 in __db_tas_mutex_readlock () from /lib64/ >> libdb-5.3.so >> #3 0x00007f68376c5d35 in __memp_fget () from /lib64/libdb-5.3.so >> #4 0x00007f683766f333 in __dbc_iget () from /lib64/libdb-5.3.so >> #5 0x00007f683767db02 in __dbc_get_pp () from /lib64/libdb-5.3.so >> #6 0x00007f6832b59822 in idl_new_fetch () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #7 0x00007f6832b67b26 in index_read_ext_allids () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #8 0x00007f6832b52284 in keys2idl () from /usr/lib64/dirsrv/plugins/libb >> ack-ldbm.so >> #9 0x00007f6832b529e3 in ava_candidates.isra.0 () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #10 0x00007f6832b52fd2 in filter_candidates_ext () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #11 0x00007f6832b54096 in list_candidates () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #12 0x00007f6832b52f40 in filter_candidates_ext () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #13 0x00007f6832b54096 in list_candidates () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #14 0x00007f6832b52f40 in filter_candidates_ext () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #15 0x00007f6832b54096 in list_candidates () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #16 0x00007f6832b52f40 in filter_candidates_ext () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #17 0x00007f6832b8e80c in subtree_candidates () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #18 0x00007f6832b8fe7f in ldbm_back_search () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #19 0x00007f683f4bd1d6 in op_shared_search () from >> /usr/lib64/dirsrv/libslapd.so.0 >> #20 0x00007f683f9ae33e in do_search () >> #21 0x00007f683f99c3c3 in connection_threadmain () >> #22 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so >> #23 0x00007f683d04cdc5 in start_thread () from /lib64/libpthread.so.0 >> #24 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 >> Thread 23 (Thread 0x7f6801ffb700 (LWP 4870)): >> #0 0x00007f683d0506d5 in pthread_cond_wait@@GLIBC_2.3.2 () from >> /lib64/libpthread.so.0 >> #1 0x00007f683d6a6050 in PR_WaitCondVar () from /lib64/libnspr4.so >> #2 0x00007f683f999fee in connection_wait_for_new_work () >> #3 0x00007f683f99b719 in connection_threadmain () >> #4 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so >> #5 0x00007f683d04cdc5 in start_thread () from /lib64/libpthread.so.0 >> #6 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 >> Thread 22 (Thread 0x7f68017fa700 (LWP 4871)): >> #0 0x00007f683d0506d5 in pthread_cond_wait@@GLIBC_2.3.2 () from >> /lib64/libpthread.so.0 >> #1 0x00007f683d6a6050 in PR_WaitCondVar () from /lib64/libnspr4.so >> #2 0x00007f683f999fee in connection_wait_for_new_work () >> #3 0x00007f683f99b719 in connection_threadmain () >> #4 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so >> #5 0x00007f683d04cdc5 in start_thread () from /lib64/libpthread.so.0 >> #6 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 >> Thread 21 (Thread 0x7f6800ff9700 (LWP 4872)): >> #0 0x00007f683d0506d5 in pthread_cond_wait@@GLIBC_2.3.2 () from >> /lib64/libpthread.so.0 >> #1 0x00007f683d6a6050 in PR_WaitCondVar () from /lib64/libnspr4.so >> #2 0x00007f683f999fee in connection_wait_for_new_work () >> #3 0x00007f683f99b719 in connection_threadmain () >> #4 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so >> #5 0x00007f683d04cdc5 in start_thread () from /lib64/libpthread.so.0 >> #6 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 >> Thread 20 (Thread 0x7f68007f8700 (LWP 4873)): >> #0 0x00007f683d0506d5 in pthread_cond_wait@@GLIBC_2.3.2 () from >> /lib64/libpthread.so.0 >> #1 0x00007f68375ac2f3 in __db_hybrid_mutex_suspend () from /lib64/ >> libdb-5.3.so >> #2 0x00007f68375ab640 in __db_tas_mutex_lock () from /lib64/libdb-5.3.so >> #3 0x00007f68376c5e1b in __memp_fget () from /lib64/libdb-5.3.so >> #4 0x00007f68375c9211 in __bam_search () from /lib64/libdb-5.3.so >> #5 0x00007f68375b4256 in __bamc_search () from /lib64/libdb-5.3.so >> #6 0x00007f68375b86c4 in __bamc_put () from /lib64/libdb-5.3.so >> #7 0x00007f683766f6d5 in __dbc_iput () from /lib64/libdb-5.3.so >> #8 0x00007f683766aa0e in __db_put () from /lib64/libdb-5.3.so >> #9 0x00007f683767ffa4 in __db_put_pp () from /lib64/libdb-5.3.so >> #10 0x00007f6832b5a4cf in idl_new_insert_key () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #11 0x00007f6832b695b5 in addordel_values_sv.isra.3 () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #12 0x00007f6832b6a25a in index_addordel_values_ext_sv () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #13 0x00007f6832b6a694 in index_addordel_values_sv () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #14 0x00007f6832b6a75a in index_addordel_entry () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #15 0x00007f6832b6d5fa in ldbm_back_add () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #16 0x00007f683f46fcca in op_shared_add () from >> /usr/lib64/dirsrv/libslapd.so.0 >> #17 0x00007f683f471020 in do_add () from /usr/lib64/dirsrv/libslapd.so.0 >> #18 0x00007f683f99c303 in connection_threadmain () >> #19 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so >> #20 0x00007f683d04cdc5 in start_thread () from /lib64/libpthread.so.0 >> #21 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 >> Thread 19 (Thread 0x7f67ffff7700 (LWP 4874)): >> #0 0x00007f683d0506d5 in pthread_cond_wait@@GLIBC_2.3.2 () from >> /lib64/libpthread.so.0 >> #1 0x00007f683d6a62c3 in PR_EnterMonitor () from /lib64/libnspr4.so >> #2 0x00007f6832b4d3e6 in dblayer_txn_begin () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #3 0x00007f6832b89007 in ldbm_back_modify () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #4 0x00007f683f4b6541 in op_shared_modify () from >> /usr/lib64/dirsrv/libslapd.so.0 >> #5 0x00007f683f4b7004 in modify_internal_pb () from >> /usr/lib64/dirsrv/libslapd.so.0 >> #6 0x00007f6832dd3b60 in ipapwd_apply_mods () from >> /usr/lib64/dirsrv/plugins/libipa_pwd_extop.so >> #7 0x00007f6832dd78e0 in store_new_keys () from >> /usr/lib64/dirsrv/plugins/libipa_pwd_extop.so >> #8 0x00007f6832dd9ad0 in ipapwd_extop () from >> /usr/lib64/dirsrv/plugins/libipa_pwd_extop.so >> #9 0x00007f683f4c6ba2 in plugin_call_exop_plugins () from >> /usr/lib64/dirsrv/libslapd.so.0 >> #10 0x00007f683f9a2efe in do_extended () >> #11 0x00007f683f99c41a in connection_threadmain () >> #12 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so >> #13 0x00007f683d04cdc5 in start_thread () from /lib64/libpthread.so.0 >> #14 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 >> Thread 18 (Thread 0x7f67ff7f6700 (LWP 4875)): >> #0 0x00007f683d0506d5 in pthread_cond_wait@@GLIBC_2.3.2 () from >> /lib64/libpthread.so.0 >> #1 0x00007f683d6a6050 in PR_WaitCondVar () from /lib64/libnspr4.so >> #2 0x00007f683f999fee in connection_wait_for_new_work () >> #3 0x00007f683f99b719 in connection_threadmain () >> #4 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so >> #5 0x00007f683d04cdc5 in start_thread () from /lib64/libpthread.so.0 >> #6 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 >> Thread 17 (Thread 0x7f67feff5700 (LWP 4876)): >> #0 0x00007f683d0506d5 in pthread_cond_wait@@GLIBC_2.3.2 () from >> /lib64/libpthread.so.0 >> #1 0x00007f68375ac2f3 in __db_hybrid_mutex_suspend () from /lib64/ >> libdb-5.3.so >> #2 0x00007f68375ab640 in __db_tas_mutex_lock () from /lib64/libdb-5.3.so >> #3 0x00007f6837655d3a in __lock_get_internal () from /lib64/libdb-5.3.so >> #4 0x00007f6837656820 in __lock_get () from /lib64/libdb-5.3.so >> #5 0x00007f6837682142 in __db_lget () from /lib64/libdb-5.3.so >> #6 0x00007f68375c9605 in __bam_search () from /lib64/libdb-5.3.so >> #7 0x00007f68375b4256 in __bamc_search () from /lib64/libdb-5.3.so >> #8 0x00007f68375b5d0f in __bamc_get () from /lib64/libdb-5.3.so >> #9 0x00007f683766eca6 in __dbc_iget () from /lib64/libdb-5.3.so >> #10 0x00007f683767db02 in __dbc_get_pp () from /lib64/libdb-5.3.so >> #11 0x00007f6832b59680 in idl_new_fetch () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #12 0x00007f6832b67b26 in index_read_ext_allids () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #13 0x00007f6832b52284 in keys2idl () from /usr/lib64/dirsrv/plugins/libb >> ack-ldbm.so >> #14 0x00007f6832b529e3 in ava_candidates.isra.0 () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #15 0x00007f6832b52fd2 in filter_candidates_ext () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #16 0x00007f6832b54096 in list_candidates () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #17 0x00007f6832b52f40 in filter_candidates_ext () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #18 0x00007f6832b8e80c in subtree_candidates () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #19 0x00007f6832b8fe7f in ldbm_back_search () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #20 0x00007f683f4bd1d6 in op_shared_search () from >> /usr/lib64/dirsrv/libslapd.so.0 >> #21 0x00007f683f4cd94e in search_internal_callback_pb () from >> /usr/lib64/dirsrv/libslapd.so.0 >> #22 0x00007f683f4cdbe8 in search_internal_pb () from >> /usr/lib64/dirsrv/libslapd.so.0 >> #23 0x00007f6832dd75dc in get_entry_by_principal () from >> /usr/lib64/dirsrv/plugins/libipa_pwd_extop.so >> #24 0x00007f6832dd8eb1 in ipapwd_extop () from >> /usr/lib64/dirsrv/plugins/libipa_pwd_extop.so >> #25 0x00007f683f4c6ba2 in plugin_call_exop_plugins () from >> /usr/lib64/dirsrv/libslapd.so.0 >> #26 0x00007f683f9a2efe in do_extended () >> #27 0x00007f683f99c41a in connection_threadmain () >> #28 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so >> #29 0x00007f683d04cdc5 in start_thread () from /lib64/libpthread.so.0 >> #30 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 >> Thread 16 (Thread 0x7f67fe7f4700 (LWP 4877)): >> #0 0x00007f683d0506d5 in pthread_cond_wait@@GLIBC_2.3.2 () from >> /lib64/libpthread.so.0 >> #1 0x00007f683d6a6050 in PR_WaitCondVar () from /lib64/libnspr4.so >> #2 0x00007f683f999fee in connection_wait_for_new_work () >> #3 0x00007f683f99b719 in connection_threadmain () >> #4 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so >> #5 0x00007f683d04cdc5 in start_thread () from /lib64/libpthread.so.0 >> #6 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 >> Thread 15 (Thread 0x7f67fdff3700 (LWP 4878)): >> #0 0x00007f683d0506d5 in pthread_cond_wait@@GLIBC_2.3.2 () from >> /lib64/libpthread.so.0 >> #1 0x00007f683d6a6050 in PR_WaitCondVar () from /lib64/libnspr4.so >> #2 0x00007f683f999fee in connection_wait_for_new_work () >> #3 0x00007f683f99b719 in connection_threadmain () >> #4 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so >> #5 0x00007f683d04cdc5 in start_thread () from /lib64/libpthread.so.0 >> #6 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 >> Thread 14 (Thread 0x7f67fd7f2700 (LWP 4879)): >> #0 0x00007f683d0506d5 in pthread_cond_wait@@GLIBC_2.3.2 () from >> /lib64/libpthread.so.0 >> #1 0x00007f683d6a6050 in PR_WaitCondVar () from /lib64/libnspr4.so >> #2 0x00007f683f999fee in connection_wait_for_new_work () >> #3 0x00007f683f99b719 in connection_threadmain () >> #4 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so >> #5 0x00007f683d04cdc5 in start_thread () from /lib64/libpthread.so.0 >> #6 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 >> Thread 13 (Thread 0x7f67fcff1700 (LWP 4880)): >> #0 0x00007f683d0506d5 in pthread_cond_wait@@GLIBC_2.3.2 () from >> /lib64/libpthread.so.0 >> #1 0x00007f683d6a6050 in PR_WaitCondVar () from /lib64/libnspr4.so >> #2 0x00007f683f999fee in connection_wait_for_new_work () >> #3 0x00007f683f99b719 in connection_threadmain () >> #4 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so >> #5 0x00007f683d04cdc5 in start_thread () from /lib64/libpthread.so.0 >> #6 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 >> Thread 12 (Thread 0x7f67fc7f0700 (LWP 4881)): >> #0 0x00007f683d0506d5 in pthread_cond_wait@@GLIBC_2.3.2 () from >> /lib64/libpthread.so.0 >> #1 0x00007f683d6a6050 in PR_WaitCondVar () from /lib64/libnspr4.so >> #2 0x00007f683f999fee in connection_wait_for_new_work () >> #3 0x00007f683f99b719 in connection_threadmain () >> #4 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so >> #5 0x00007f683d04cdc5 in start_thread () from /lib64/libpthread.so.0 >> #6 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 >> Thread 11 (Thread 0x7f67fbfef700 (LWP 4882)): >> #0 0x00007f683d0506d5 in pthread_cond_wait@@GLIBC_2.3.2 () from >> /lib64/libpthread.so.0 >> #1 0x00007f683d6a6050 in PR_WaitCondVar () from /lib64/libnspr4.so >> #2 0x00007f683f999fee in connection_wait_for_new_work () >> #3 0x00007f683f99b719 in connection_threadmain () >> #4 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so >> #5 0x00007f683d04cdc5 in start_thread () from /lib64/libpthread.so.0 >> #6 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 >> Thread 10 (Thread 0x7f67fb7ee700 (LWP 4883)): >> #0 0x00007f683d0506d5 in pthread_cond_wait@@GLIBC_2.3.2 () from >> /lib64/libpthread.so.0 >> #1 0x00007f68375ac2f3 in __db_hybrid_mutex_suspend () from /lib64/ >> libdb-5.3.so >> #2 0x00007f68375ab640 in __db_tas_mutex_lock () from /lib64/libdb-5.3.so >> #3 0x00007f6837655d3a in __lock_get_internal () from /lib64/libdb-5.3.so >> #4 0x00007f6837656820 in __lock_get () from /lib64/libdb-5.3.so >> #5 0x00007f6837682142 in __db_lget () from /lib64/libdb-5.3.so >> #6 0x00007f68375c9605 in __bam_search () from /lib64/libdb-5.3.so >> #7 0x00007f68375b4256 in __bamc_search () from /lib64/libdb-5.3.so >> #8 0x00007f68375b5d0f in __bamc_get () from /lib64/libdb-5.3.so >> #9 0x00007f683766eca6 in __dbc_iget () from /lib64/libdb-5.3.so >> #10 0x00007f683767db02 in __dbc_get_pp () from /lib64/libdb-5.3.so >> #11 0x00007f6832b59680 in idl_new_fetch () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #12 0x00007f6832b67b26 in index_read_ext_allids () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #13 0x00007f6832b52284 in keys2idl () from /usr/lib64/dirsrv/plugins/libb >> ack-ldbm.so >> #14 0x00007f6832b529e3 in ava_candidates.isra.0 () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #15 0x00007f6832b52fd2 in filter_candidates_ext () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #16 0x00007f6832b54096 in list_candidates () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #17 0x00007f6832b52f40 in filter_candidates_ext () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #18 0x00007f6832b54096 in list_candidates () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #19 0x00007f6832b52f40 in filter_candidates_ext () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #20 0x00007f6832b54096 in list_candidates () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #21 0x00007f6832b52f40 in filter_candidates_ext () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #22 0x00007f6832b8e80c in subtree_candidates () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #23 0x00007f6832b8fe7f in ldbm_back_search () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #24 0x00007f683f4bd1d6 in op_shared_search () from >> /usr/lib64/dirsrv/libslapd.so.0 >> #25 0x00007f683f9ae33e in do_search () >> #26 0x00007f683f99c3c3 in connection_threadmain () >> #27 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so >> #28 0x00007f683d04cdc5 in start_thread () from /lib64/libpthread.so.0 >> #29 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 >> Thread 9 (Thread 0x7f67fafed700 (LWP 4884)): >> #0 0x00007f683d0506d5 in pthread_cond_wait@@GLIBC_2.3.2 () from >> /lib64/libpthread.so.0 >> #1 0x00007f683d6a6050 in PR_WaitCondVar () from /lib64/libnspr4.so >> #2 0x00007f683f999fee in connection_wait_for_new_work () >> #3 0x00007f683f99b719 in connection_threadmain () >> #4 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so >> #5 0x00007f683d04cdc5 in start_thread () from /lib64/libpthread.so.0 >> #6 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 >> Thread 8 (Thread 0x7f67fa7ec700 (LWP 4885)): >> #0 0x00007f683d0506d5 in pthread_cond_wait@@GLIBC_2.3.2 () from >> /lib64/libpthread.so.0 >> #1 0x00007f68375ac2f3 in __db_hybrid_mutex_suspend () from /lib64/ >> libdb-5.3.so >> #2 0x00007f68375ab640 in __db_tas_mutex_lock () from /lib64/libdb-5.3.so >> #3 0x00007f6837655d3a in __lock_get_internal () from /lib64/libdb-5.3.so >> #4 0x00007f6837656820 in __lock_get () from /lib64/libdb-5.3.so >> #5 0x00007f6837682142 in __db_lget () from /lib64/libdb-5.3.so >> #6 0x00007f68375c9605 in __bam_search () from /lib64/libdb-5.3.so >> #7 0x00007f68375b4256 in __bamc_search () from /lib64/libdb-5.3.so >> #8 0x00007f68375b5d0f in __bamc_get () from /lib64/libdb-5.3.so >> #9 0x00007f683766eca6 in __dbc_iget () from /lib64/libdb-5.3.so >> #10 0x00007f683767db02 in __dbc_get_pp () from /lib64/libdb-5.3.so >> #11 0x00007f6832b59680 in idl_new_fetch () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #12 0x00007f6832b67b26 in index_read_ext_allids () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #13 0x00007f6832b52284 in keys2idl () from /usr/lib64/dirsrv/plugins/libb >> ack-ldbm.so >> #14 0x00007f6832b529e3 in ava_candidates.isra.0 () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #15 0x00007f6832b52fd2 in filter_candidates_ext () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #16 0x00007f6832b54096 in list_candidates () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #17 0x00007f6832b52f40 in filter_candidates_ext () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #18 0x00007f6832b54096 in list_candidates () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #19 0x00007f6832b52f40 in filter_candidates_ext () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #20 0x00007f6832b54096 in list_candidates () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #21 0x00007f6832b52f40 in filter_candidates_ext () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #22 0x00007f6832b8e80c in subtree_candidates () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #23 0x00007f6832b8fe7f in ldbm_back_search () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #24 0x00007f683f4bd1d6 in op_shared_search () from >> /usr/lib64/dirsrv/libslapd.so.0 >> #25 0x00007f683f9ae33e in do_search () >> #26 0x00007f683f99c3c3 in connection_threadmain () >> #27 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so >> #28 0x00007f683d04cdc5 in start_thread () from /lib64/libpthread.so.0 >> #29 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 >> Thread 7 (Thread 0x7f67f9feb700 (LWP 4886)): >> #0 0x00007f683d0506d5 in pthread_cond_wait@@GLIBC_2.3.2 () from >> /lib64/libpthread.so.0 >> #1 0x00007f683d6a6050 in PR_WaitCondVar () from /lib64/libnspr4.so >> #2 0x00007f683f999fee in connection_wait_for_new_work () >> #3 0x00007f683f99b719 in connection_threadmain () >> #4 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so >> #5 0x00007f683d04cdc5 in start_thread () from /lib64/libpthread.so.0 >> #6 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 >> Thread 6 (Thread 0x7f67f97ea700 (LWP 4887)): >> #0 0x00007f683d0506d5 in pthread_cond_wait@@GLIBC_2.3.2 () from >> /lib64/libpthread.so.0 >> #1 0x00007f683d6a6050 in PR_WaitCondVar () from /lib64/libnspr4.so >> #2 0x00007f683f999fee in connection_wait_for_new_work () >> #3 0x00007f683f99b719 in connection_threadmain () >> #4 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so >> #5 0x00007f683d04cdc5 in start_thread () from /lib64/libpthread.so.0 >> #6 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 >> Thread 5 (Thread 0x7f67f8fe9700 (LWP 4888)): >> #0 0x00007f683d0506d5 in pthread_cond_wait@@GLIBC_2.3.2 () from >> /lib64/libpthread.so.0 >> #1 0x00007f683d6a6050 in PR_WaitCondVar () from /lib64/libnspr4.so >> #2 0x00007f683f999fee in connection_wait_for_new_work () >> #3 0x00007f683f99b719 in connection_threadmain () >> #4 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so >> #5 0x00007f683d04cdc5 in start_thread () from /lib64/libpthread.so.0 >> #6 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 >> Thread 4 (Thread 0x7f67f87e8700 (LWP 4889)): >> #0 0x00007f683cd719b3 in select () from /lib64/libc.so.6 >> #1 0x00007f683f4fc0e9 in DS_Sleep () from /usr/lib64/dirsrv/libslapd.so. >> 0 >> #2 0x00007f683f99d325 in time_thread () >> #3 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so >> #4 0x00007f683d04cdc5 in start_thread () from /lib64/libpthread.so.0 >> #5 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 >> Thread 3 (Thread 0x7f67f7de6700 (LWP 4892)): >> #0 0x00007f683d0506d5 in pthread_cond_wait@@GLIBC_2.3.2 () from >> /lib64/libpthread.so.0 >> #1 0x00007f683d6a6050 in PR_WaitCondVar () from /lib64/libnspr4.so >> #2 0x00007f683f9a8f25 in ps_send_results () >> #3 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so >> #4 0x00007f683d04cdc5 in start_thread () from /lib64/libpthread.so.0 >> #5 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 >> Thread 2 (Thread 0x7f67f75e5700 (LWP 5084)): >> #0 0x00007f683d0506d5 in pthread_cond_wait@@GLIBC_2.3.2 () from >> /lib64/libpthread.so.0 >> #1 0x00007f683d6a6050 in PR_WaitCondVar () from /lib64/libnspr4.so >> #2 0x00007f683f9a8f25 in ps_send_results () >> #3 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so >> #4 0x00007f683d04cdc5 in start_thread () from /lib64/libpthread.so.0 >> #5 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 >> Thread 1 (Thread 0x7f683f964840 (LWP 4845)): >> #0 0x00007f683d0506d5 in pthread_cond_wait@@GLIBC_2.3.2 () from >> /lib64/libpthread.so.0 >> #1 0x00007f683d6a62c3 in PR_EnterMonitor () from /lib64/libnspr4.so >> #2 0x00007f683f99d5ed in setup_pr_read_pds () >> #3 0x00007f683f9a0fd9 in slapd_daemon () >> #4 0x00007f683f992c6c in main () >> >> >> >> the top output seems ok, there was just a sudden spike to around 340% cpu >> then it comes down >> >> top - 08:53:12 up 6 days, 23:20, 1 user, load average: 0.00, 0.01, 0.12 >> Threads: 44 total, 0 running, 44 sleeping, 0 stopped, 0 zombie >> %Cpu(s): 0.0 us, 0.0 sy, 0.0 ni,100.0 id, 0.0 wa, 0.0 hi, 0.0 si, >> 0.0 st >> KiB Mem : 65533176 total, 52165676 free, 6135088 used, 7232412 >> buff/cache >> KiB Swap: 3999996 total, 3999996 free, 0 used. 58452040 avail Mem >> >> PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ >> COMMAND >> 4845 dirsrv 20 0 5133196 3.150g 281272 S 0.0 5.0 31:34.51 >> ns-slapd >> 4847 dirsrv 20 0 5133196 3.150g 281272 S 0.0 5.0 0:13.53 >> ns-slapd >> 4848 dirsrv 20 0 5133196 3.150g 281272 S 0.0 5.0 0:13.16 >> ns-slapd >> 4849 dirsrv 20 0 5133196 3.150g 281272 S 0.0 5.0 0:45.27 >> ns-slapd >> 4850 dirsrv 20 0 5133196 3.150g 281272 S 0.0 5.0 0:01.60 >> ns-slapd >> 4851 dirsrv 20 0 5133196 3.150g 281272 S 0.0 5.0 0:00.00 >> ns-slapd >> 4852 dirsrv 20 0 5133196 3.150g 281272 S 0.0 5.0 0:00.00 >> ns-slapd >> 4853 dirsrv 20 0 5133196 3.150g 281272 S 0.0 5.0 0:00.00 >> ns-slapd >> 4854 dirsrv 20 0 5133196 3.150g 281272 S 0.0 5.0 0:00.00 >> ns-slapd >> 4855 dirsrv 20 0 5133196 3.150g 281272 S 0.0 5.0 0:03.02 >> ns-slapd >> 4856 dirsrv 20 0 5133196 3.150g 281272 S 0.0 5.0 0:00.26 >> ns-slapd >> 4859 dirsrv 20 0 5133196 3.150g 281272 S 0.0 5.0 3:18.67 >> ns-slapd >> 4860 dirsrv 20 0 5133196 3.150g 281272 S 0.0 5.0 3:19.82 >> ns-slapd >> 4861 dirsrv 20 0 5133196 3.150g 281272 S 0.0 5.0 3:20.23 >> ns-slapd >> 4862 dirsrv 20 0 5133196 3.150g 281272 S 0.0 5.0 3:22.65 >> ns-slapd >> 4863 dirsrv 20 0 5133196 3.150g 281272 S 0.0 5.0 3:16.64 >> ns-slapd >> 4864 dirsrv 20 0 5133196 3.150g 281272 S 0.0 5.0 3:17.39 >> ns-slapd >> 4865 dirsrv 20 0 5133196 3.150g 281272 S 0.0 5.0 3:17.80 >> ns-slapd >> >> >> Thanks >> Rakesh >> >> On Tue, Sep 6, 2016 at 12:09 AM, Rakesh Rajasekharan < >> rakesh.rajasekharan at gmail.com> wrote: >> >>> Hi Thierry/Ludwig, >>> >>> Unfortunately I had by then restarted the service and since then have >>> been trying to hit the hang issue again but no luck this time >>> >>> It has been running fine for the last few hours now in-spite of trying >>> all that I believe could cause the hang like making simultaneous client >>> addition requests. >>> >>> I will continue testing this and next time I get it I will make sure I >>> get the pstack and also get the dump of db lock info. >>> >>> Thanks, >>> Rakesh >>> >>> >>> >>> On Mon, Sep 5, 2016 at 6:14 PM, Ludwig Krispenz >>> wrote: >>> >>>> >>>> On 09/05/2016 02:31 PM, thierry bordaz wrote: >>>> >>>> Hi Rakesh, >>>> >>>> Thanks you soooo much for you continuous help on this !! >>>> >>>> If the hang is still going on, would you dump the DB lock information >>>> with: >>>> >>>> db_stat -CA -N -h /var/lib/dirsrv/slapd-/db >>>> >/tmp/db_lock_info >>>> >>>> It is looking like several threads are waiting for one/several db >>>> locks. With the lock info we will know which thread acquired it/them. >>>> A guess is that it could be the checkpointing thread but this one is >>>> also waiting (on a memory pool). >>>> >>>> I am copying Ludwig on this thread in case this pstack rings any bell. >>>> >>>> the frequent presence of __db_tas_mutex_lock() looks like the NUMA >>>> problem of libdb, although the threads already have stopped spinning (maybe >>>> this was an effect of gcore). >>>> If the hang aoccurs again, could you check cpu consumption >>>> top -H -p >>>> and get a pstack >>>> pstack >>>> >>>> >>>> best regards >>>> thierry >>>> >>>> On 09/05/2016 02:05 PM, Rakesh Rajasekharan wrote: >>>> >>>> Hi Thierry, >>>> >>>> I was able to now reporduce the issue.. >>>> >>>> the ns-slapd process just got hung,.. however, no core file was >>>> generate.. >>>> >>>> I manually created a coredump using gcore and then took a strace of it >>>> this is strace from the coredump generated >>>> >>>> GNU gdb (GDB) Red Hat Enterprise Linux 7.6.1-80.el7 >>>> Copyright (C) 2013 Free Software Foundation, Inc. >>>> License GPLv3+: GNU GPL version 3 or later < >>>> http://gnu.org/licenses/gpl.html> >>>> This is free software: you are free to change and redistribute it. >>>> There is NO WARRANTY, to the extent permitted by law. Type "show >>>> copying" >>>> and "show warranty" for details. >>>> This GDB was configured as "x86_64-redhat-linux-gnu". >>>> For bug reporting instructions, please see: >>>> ... >>>> Reading symbols from /usr/sbin/ns-slapd...Reading symbols from >>>> /usr/sbin/ns-slapd...(no debugging symbols found)...done. >>>> (no debugging symbols found)...done. >>>> [New LWP 105159] >>>> [New LWP 105160] >>>> [New LWP 105161] >>>> [New LWP 105162] >>>> [New LWP 105163] >>>> [New LWP 105164] >>>> [New LWP 105165] >>>> [New LWP 105166] >>>> [New LWP 105167] >>>> [New LWP 105168] >>>> [New LWP 105171] >>>> [New LWP 105172] >>>> [New LWP 105173] >>>> [New LWP 105174] >>>> [New LWP 105175] >>>> [New LWP 105176] >>>> [New LWP 105177] >>>> [New LWP 105178] >>>> [New LWP 105179] >>>> [New LWP 105180] >>>> [New LWP 105181] >>>> [New LWP 105182] >>>> [New LWP 105183] >>>> [New LWP 105184] >>>> [New LWP 105185] >>>> [New LWP 105186] >>>> [New LWP 105187] >>>> [New LWP 105188] >>>> [New LWP 105189] >>>> [New LWP 105190] >>>> [New LWP 105191] >>>> [New LWP 105192] >>>> [New LWP 105193] >>>> [New LWP 105194] >>>> [New LWP 105195] >>>> [New LWP 105196] >>>> [New LWP 105197] >>>> [New LWP 105198] >>>> [New LWP 105199] >>>> [New LWP 105200] >>>> [New LWP 105201] >>>> [New LWP 105203] >>>> [New LWP 105215] >>>> [New LWP 105158] >>>> [Thread debugging using libthread_db enabled] >>>> Using host libthread_db library "/lib64/libthread_db.so.1". >>>> Core was generated by `/usr/sbin/ns-slapd'. >>>> #0 0x00007f1b48ae69b3 in select () from /lib64/libc.so.6 >>>> >>>> Thread 44 (Thread 0x7f1b4b6d9840 (LWP 105158)): >>>> #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 () from >>>> /lib64/libpthread.so.0 >>>> No symbol table info available. >>>> #1 0x00007f1b4941b2c3 in PR_EnterMonitor () from /lib64/libnspr4.so >>>> No symbol table info available. >>>> #2 0x00007f1b4b7125ed in setup_pr_read_pds () >>>> No symbol table info available. >>>> #3 0x00007f1b4b715fd9 in slapd_daemon () >>>> No symbol table info available. >>>> #4 0x00007f1b4b707c6c in main () >>>> No symbol table info available. >>>> >>>> Thread 43 (Thread 0x7f1a5df0f700 (LWP 105215)): >>>> #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 () from >>>> /lib64/libpthread.so.0 >>>> No symbol table info available. >>>> #1 0x00007f1b4941b050 in PR_WaitCondVar () from /lib64/libnspr4.so >>>> No symbol table info available. >>>> #2 0x00007f1b4b71df25 in ps_send_results () >>>> No symbol table info available. >>>> #3 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so >>>> No symbol table info available. >>>> #4 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 >>>> No symbol table info available. >>>> #5 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 >>>> No symbol table info available. >>>> >>>> Thread 42 (Thread 0x7f1b03fe7700 (LWP 105203)): >>>> #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 () from >>>> /lib64/libpthread.so.0 >>>> No symbol table info available. >>>> #1 0x00007f1b4941b050 in PR_WaitCondVar () from /lib64/libnspr4.so >>>> No symbol table info available. >>>> #2 0x00007f1b4b71df25 in ps_send_results () >>>> No symbol table info available. >>>> #3 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so >>>> No symbol table info available. >>>> #4 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 >>>> No symbol table info available. >>>> #5 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 >>>> No symbol table info available. >>>> >>>> Thread 41 (Thread 0x7f1b047e8700 (LWP 105201)): >>>> #0 0x00007f1b48ae69b3 in select () from /lib64/libc.so.6 >>>> No symbol table info available. >>>> #1 0x00007f1b4b2710e9 in DS_Sleep () from >>>> /usr/lib64/dirsrv/libslapd.so.0 >>>> No symbol table info available. >>>> #2 0x00007f1b4b712325 in time_thread () >>>> No symbol table info available. >>>> #3 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so >>>> No symbol table info available. >>>> #4 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 >>>> No symbol table info available. >>>> #5 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 >>>> No symbol table info available. >>>> >>>> Thread 40 (Thread 0x7f1b04fe9700 (LWP 105200)): >>>> #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 () from >>>> /lib64/libpthread.so.0 >>>> No symbol table info available. >>>> #1 0x00007f1b433212f3 in __db_hybrid_mutex_suspend () from /lib64/ >>>> libdb-5.3.so >>>> No symbol table info available. >>>> #2 0x00007f1b43320640 in __db_tas_mutex_lock () from /lib64/ >>>> libdb-5.3.so >>>> No symbol table info available. >>>> #3 0x00007f1b4343ae1b in __memp_fget () from /lib64/libdb-5.3.so >>>> No symbol table info available. >>>> #4 0x00007f1b4333e211 in __bam_search () from /lib64/libdb-5.3.so >>>> No symbol table info available. >>>> #5 0x00007f1b43329256 in __bamc_search () from /lib64/libdb-5.3.so >>>> No symbol table info available. >>>> #6 0x00007f1b4332d6c4 in __bamc_put () from /lib64/libdb-5.3.so >>>> No symbol table info available. >>>> #7 0x00007f1b433e46d5 in __dbc_iput () from /lib64/libdb-5.3.so >>>> No symbol table info available. >>>> #8 0x00007f1b433dfa0e in __db_put () from /lib64/libdb-5.3.so >>>> No symbol table info available. >>>> #9 0x00007f1b433f4fa4 in __db_put_pp () from /lib64/libdb-5.3.so >>>> No symbol table info available. >>>> #10 0x00007f1b3e8cf4cf in idl_new_insert_key () from >>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>> No symbol table info available. >>>> #11 0x00007f1b3e8de5b5 in addordel_values_sv.isra.3 () from >>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>> No symbol table info available. >>>> #12 0x00007f1b3e8df25a in index_addordel_values_ext_sv () from >>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>> No symbol table info available. >>>> #13 0x00007f1b3e8df694 in index_addordel_values_sv () from >>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>> No symbol table info available. >>>> #14 0x00007f1b3e8df75a in index_addordel_entry () from >>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>> No symbol table info available. >>>> #15 0x00007f1b3e8e25fa in ldbm_back_add () from >>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>> No symbol table info available. >>>> #16 0x00007f1b4b1e4cca in op_shared_add () from >>>> /usr/lib64/dirsrv/libslapd.so.0 >>>> No symbol table info available. >>>> #17 0x00007f1b4b1e6020 in do_add () from /usr/lib64/dirsrv/libslapd.so. >>>> 0 >>>> No symbol table info available. >>>> #18 0x00007f1b4b711303 in connection_threadmain () >>>> No symbol table info available. >>>> #19 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so >>>> No symbol table info available. >>>> #20 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 >>>> No symbol table info available. >>>> #21 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 >>>> No symbol table info available. >>>> >>>> Thread 39 (Thread 0x7f1b057ea700 (LWP 105199)): >>>> #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 () from >>>> /lib64/libpthread.so.0 >>>> No symbol table info available. >>>> #1 0x00007f1b4941b050 in PR_WaitCondVar () from /lib64/libnspr4.so >>>> No symbol table info available. >>>> #2 0x00007f1b4b70efee in connection_wait_for_new_work () >>>> No symbol table info available. >>>> #3 0x00007f1b4b710719 in connection_threadmain () >>>> No symbol table info available. >>>> #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so >>>> No symbol table info available. >>>> #5 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 >>>> No symbol table info available. >>>> #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 >>>> No symbol table info available. >>>> >>>> Thread 38 (Thread 0x7f1b05feb700 (LWP 105198)): >>>> #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 () from >>>> /lib64/libpthread.so.0 >>>> No symbol table info available. >>>> #1 0x00007f1b4941b050 in PR_WaitCondVar () from /lib64/libnspr4.so >>>> No symbol table info available. >>>> #2 0x00007f1b4b70efee in connection_wait_for_new_work () >>>> No symbol table info available. >>>> #3 0x00007f1b4b710719 in connection_threadmain () >>>> No symbol table info available. >>>> #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so >>>> No symbol table info available. >>>> #5 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 >>>> No symbol table info available. >>>> #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 >>>> No symbol table info available. >>>> >>>> Thread 37 (Thread 0x7f1b067ec700 (LWP 105197)): >>>> #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 () from >>>> /lib64/libpthread.so.0 >>>> No symbol table info available. >>>> #1 0x00007f1b4941b050 in PR_WaitCondVar () from /lib64/libnspr4.so >>>> No symbol table info available. >>>> #2 0x00007f1b4b70efee in connection_wait_for_new_work () >>>> No symbol table info available. >>>> #3 0x00007f1b4b710719 in connection_threadmain () >>>> No symbol table info available. >>>> #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so >>>> No symbol table info available. >>>> #5 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 >>>> No symbol table info available. >>>> #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 >>>> No symbol table info available. >>>> >>>> Thread 36 (Thread 0x7f1b06fed700 (LWP 105196)): >>>> #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 () from >>>> /lib64/libpthread.so.0 >>>> No symbol table info available. >>>> #1 0x00007f1b4941b050 in PR_WaitCondVar () from /lib64/libnspr4.so >>>> No symbol table info available. >>>> #2 0x00007f1b4b70efee in connection_wait_for_new_work () >>>> No symbol table info available. >>>> #3 0x00007f1b4b710719 in connection_threadmain () >>>> No symbol table info available. >>>> #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so >>>> No symbol table info available. >>>> #5 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 >>>> No symbol table info available. >>>> #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 >>>> No symbol table info available. >>>> >>>> Thread 35 (Thread 0x7f1b077ee700 (LWP 105195)): >>>> #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 () from >>>> /lib64/libpthread.so.0 >>>> No symbol table info available. >>>> #1 0x00007f1b4941b2c3 in PR_EnterMonitor () from /lib64/libnspr4.so >>>> No symbol table info available. >>>> #2 0x00007f1b3e8c23e6 in dblayer_txn_begin () from >>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>> No symbol table info available. >>>> #3 0x00007f1b3e8fe007 in ldbm_back_modify () from >>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>> No symbol table info available. >>>> #4 0x00007f1b4b22b541 in op_shared_modify () from >>>> /usr/lib64/dirsrv/libslapd.so.0 >>>> No symbol table info available. >>>> #5 0x00007f1b4b22c004 in modify_internal_pb () from >>>> /usr/lib64/dirsrv/libslapd.so.0 >>>> No symbol table info available. >>>> #6 0x00007f1b3eb48b60 in ipapwd_apply_mods () from >>>> /usr/lib64/dirsrv/plugins/libipa_pwd_extop.so >>>> No symbol table info available. >>>> #7 0x00007f1b3eb4c8e0 in store_new_keys () from >>>> /usr/lib64/dirsrv/plugins/libipa_pwd_extop.so >>>> No symbol table info available. >>>> #8 0x00007f1b3eb4ead0 in ipapwd_extop () from >>>> /usr/lib64/dirsrv/plugins/libipa_pwd_extop.so >>>> No symbol table info available. >>>> #9 0x00007f1b4b23bba2 in plugin_call_exop_plugins () from >>>> /usr/lib64/dirsrv/libslapd.so.0 >>>> No symbol table info available. >>>> #10 0x00007f1b4b717efe in do_extended () >>>> No symbol table info available. >>>> #11 0x00007f1b4b71141a in connection_threadmain () >>>> No symbol table info available. >>>> #12 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so >>>> No symbol table info available. >>>> #13 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 >>>> No symbol table info available. >>>> #14 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 >>>> No symbol table info available. >>>> >>>> Thread 34 (Thread 0x7f1b07fef700 (LWP 105194)): >>>> #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 () from >>>> /lib64/libpthread.so.0 >>>> No symbol table info available. >>>> #1 0x00007f1b4941b050 in PR_WaitCondVar () from /lib64/libnspr4.so >>>> No symbol table info available. >>>> #2 0x00007f1b4b70efee in connection_wait_for_new_work () >>>> No symbol table info available. >>>> #3 0x00007f1b4b710719 in connection_threadmain () >>>> No symbol table info available. >>>> #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so >>>> No symbol table info available. >>>> #5 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 >>>> No symbol table info available. >>>> #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 >>>> No symbol table info available. >>>> >>>> Thread 33 (Thread 0x7f1b087f0700 (LWP 105193)): >>>> #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 () from >>>> /lib64/libpthread.so.0 >>>> No symbol table info available. >>>> #1 0x00007f1b4941b050 in PR_WaitCondVar () from /lib64/libnspr4.so >>>> No symbol table info available. >>>> #2 0x00007f1b4b70efee in connection_wait_for_new_work () >>>> No symbol table info available. >>>> #3 0x00007f1b4b710719 in connection_threadmain () >>>> No symbol table info available. >>>> #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so >>>> No symbol table info available. >>>> #5 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 >>>> No symbol table info available. >>>> #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 >>>> No symbol table info available. >>>> >>>> Thread 32 (Thread 0x7f1b08ff1700 (LWP 105192)): >>>> #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 () from >>>> /lib64/libpthread.so.0 >>>> No symbol table info available. >>>> #1 0x00007f1b4941b050 in PR_WaitCondVar () from /lib64/libnspr4.so >>>> No symbol table info available. >>>> #2 0x00007f1b4b70efee in connection_wait_for_new_work () >>>> No symbol table info available. >>>> #3 0x00007f1b4b710719 in connection_threadmain () >>>> No symbol table info available. >>>> #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so >>>> No symbol table info available. >>>> #5 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 >>>> No symbol table info available. >>>> #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 >>>> No symbol table info available. >>>> >>>> Thread 31 (Thread 0x7f1b097f2700 (LWP 105191)): >>>> #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 () from >>>> /lib64/libpthread.so.0 >>>> No symbol table info available. >>>> #1 0x00007f1b4941b050 in PR_WaitCondVar () from /lib64/libnspr4.so >>>> No symbol table info available. >>>> #2 0x00007f1b4b70efee in connection_wait_for_new_work () >>>> No symbol table info available. >>>> #3 0x00007f1b4b710719 in connection_threadmain () >>>> No symbol table info available. >>>> #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so >>>> No symbol table info available. >>>> #5 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 >>>> No symbol table info available. >>>> #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 >>>> No symbol table info available. >>>> >>>> Thread 30 (Thread 0x7f1b09ff3700 (LWP 105190)): >>>> #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 () from >>>> /lib64/libpthread.so.0 >>>> No symbol table info available. >>>> #1 0x00007f1b433212f3 in __db_hybrid_mutex_suspend () from /lib64/ >>>> libdb-5.3.so >>>> No symbol table info available. >>>> #2 0x00007f1b43320640 in __db_tas_mutex_lock () from /lib64/ >>>> libdb-5.3.so >>>> No symbol table info available. >>>> #3 0x00007f1b433cad3a in __lock_get_internal () from /lib64/ >>>> libdb-5.3.so >>>> No symbol table info available. >>>> #4 0x00007f1b433cb820 in __lock_get () from /lib64/libdb-5.3.so >>>> No symbol table info available. >>>> #5 0x00007f1b433f7142 in __db_lget () from /lib64/libdb-5.3.so >>>> No symbol table info available. >>>> #6 0x00007f1b4333e605 in __bam_search () from /lib64/libdb-5.3.so >>>> No symbol table info available. >>>> #7 0x00007f1b43329256 in __bamc_search () from /lib64/libdb-5.3.so >>>> No symbol table info available. >>>> #8 0x00007f1b4332ad0f in __bamc_get () from /lib64/libdb-5.3.so >>>> No symbol table info available. >>>> #9 0x00007f1b433e3ca6 in __dbc_iget () from /lib64/libdb-5.3.so >>>> No symbol table info available. >>>> #10 0x00007f1b433f2b02 in __dbc_get_pp () from /lib64/libdb-5.3.so >>>> No symbol table info available. >>>> #11 0x00007f1b3e8ce680 in idl_new_fetch () from >>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>> No symbol table info available. >>>> #12 0x00007f1b3e8dcb26 in index_read_ext_allids () from >>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>> No symbol table info available. >>>> #13 0x00007f1b3e8c7284 in keys2idl () from >>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>> No symbol table info available. >>>> #14 0x00007f1b3e8c79e3 in ava_candidates.isra.0 () from >>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>> No symbol table info available. >>>> #15 0x00007f1b3e8c7fd2 in filter_candidates_ext () from >>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>> No symbol table info available. >>>> #16 0x00007f1b3e8c9096 in list_candidates () from >>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>> No symbol table info available. >>>> #17 0x00007f1b3e8c7f40 in filter_candidates_ext () from >>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>> No symbol table info available. >>>> #18 0x00007f1b3e8c9096 in list_candidates () from >>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>> No symbol table info available. >>>> #19 0x00007f1b3e8c7f40 in filter_candidates_ext () from >>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>> No symbol table info available. >>>> #20 0x00007f1b3e8c9096 in list_candidates () from >>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>> No symbol table info available. >>>> #21 0x00007f1b3e8c7f40 in filter_candidates_ext () from >>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>> No symbol table info available. >>>> #22 0x00007f1b3e90380c in subtree_candidates () from >>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>> No symbol table info available. >>>> #23 0x00007f1b3e904e7f in ldbm_back_search () from >>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>> No symbol table info available. >>>> #24 0x00007f1b4b2321d6 in op_shared_search () from >>>> /usr/lib64/dirsrv/libslapd.so.0 >>>> No symbol table info available. >>>> #25 0x00007f1b4b72333e in do_search () >>>> No symbol table info available. >>>> #26 0x00007f1b4b7113c3 in connection_threadmain () >>>> No symbol table info available. >>>> #27 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so >>>> No symbol table info available. >>>> #28 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 >>>> No symbol table info available. >>>> #29 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 >>>> No symbol table info available. >>>> >>>> Thread 29 (Thread 0x7f1b0a7f4700 (LWP 105189)): >>>> #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 () from >>>> /lib64/libpthread.so.0 >>>> No symbol table info available. >>>> #1 0x00007f1b4941b050 in PR_WaitCondVar () from /lib64/libnspr4.so >>>> No symbol table info available. >>>> #2 0x00007f1b4b70efee in connection_wait_for_new_work () >>>> No symbol table info available. >>>> #3 0x00007f1b4b710719 in connection_threadmain () >>>> No symbol table info available. >>>> #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so >>>> No symbol table info available. >>>> #5 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 >>>> No symbol table info available. >>>> #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 >>>> No symbol table info available. >>>> >>>> Thread 28 (Thread 0x7f1b0aff5700 (LWP 105188)): >>>> #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 () from >>>> /lib64/libpthread.so.0 >>>> No symbol table info available. >>>> #1 0x00007f1b4941b050 in PR_WaitCondVar () from /lib64/libnspr4.so >>>> No symbol table info available. >>>> #2 0x00007f1b4b70efee in connection_wait_for_new_work () >>>> No symbol table info available. >>>> #3 0x00007f1b4b710719 in connection_threadmain () >>>> No symbol table info available. >>>> #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so >>>> No symbol table info available. >>>> #5 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 >>>> No symbol table info available. >>>> #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 >>>> No symbol table info available. >>>> >>>> Thread 27 (Thread 0x7f1b0b7f6700 (LWP 105187)): >>>> #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 () from >>>> /lib64/libpthread.so.0 >>>> No symbol table info available. >>>> #1 0x00007f1b4941b050 in PR_WaitCondVar () from /lib64/libnspr4.so >>>> No symbol table info available. >>>> #2 0x00007f1b4b70efee in connection_wait_for_new_work () >>>> No symbol table info available. >>>> #3 0x00007f1b4b710719 in connection_threadmain () >>>> No symbol table info available. >>>> #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so >>>> No symbol table info available. >>>> #5 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 >>>> No symbol table info available. >>>> #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 >>>> No symbol table info available. >>>> >>>> Thread 26 (Thread 0x7f1b0bff7700 (LWP 105186)): >>>> #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 () from >>>> /lib64/libpthread.so.0 >>>> No symbol table info available. >>>> #1 0x00007f1b433212f3 in __db_hybrid_mutex_suspend () from /lib64/ >>>> libdb-5.3.so >>>> No symbol table info available. >>>> #2 0x00007f1b43320640 in __db_tas_mutex_lock () from /lib64/ >>>> libdb-5.3.so >>>> No symbol table info available. >>>> #3 0x00007f1b433cad3a in __lock_get_internal () from /lib64/ >>>> libdb-5.3.so >>>> No symbol table info available. >>>> #4 0x00007f1b433cb820 in __lock_get () from /lib64/libdb-5.3.so >>>> No symbol table info available. >>>> #5 0x00007f1b433f7142 in __db_lget () from /lib64/libdb-5.3.so >>>> No symbol table info available. >>>> #6 0x00007f1b4333e605 in __bam_search () from /lib64/libdb-5.3.so >>>> No symbol table info available. >>>> #7 0x00007f1b43329256 in __bamc_search () from /lib64/libdb-5.3.so >>>> No symbol table info available. >>>> #8 0x00007f1b4332ad0f in __bamc_get () from /lib64/libdb-5.3.so >>>> No symbol table info available. >>>> #9 0x00007f1b433e3ca6 in __dbc_iget () from /lib64/libdb-5.3.so >>>> No symbol table info available. >>>> #10 0x00007f1b433f2b02 in __dbc_get_pp () from /lib64/libdb-5.3.so >>>> No symbol table info available. >>>> #11 0x00007f1b3e8ce680 in idl_new_fetch () from >>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>> No symbol table info available. >>>> #12 0x00007f1b3e8dcb26 in index_read_ext_allids () from >>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>> No symbol table info available. >>>> #13 0x00007f1b3e8c7284 in keys2idl () from >>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>> No symbol table info available. >>>> #14 0x00007f1b3e8c79e3 in ava_candidates.isra.0 () from >>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>> No symbol table info available. >>>> #15 0x00007f1b3e8c7fd2 in filter_candidates_ext () from >>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>> No symbol table info available. >>>> #16 0x00007f1b3e8c9096 in list_candidates () from >>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>> No symbol table info available. >>>> #17 0x00007f1b3e8c7f40 in filter_candidates_ext () from >>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>> No symbol table info available. >>>> #18 0x00007f1b3e90380c in subtree_candidates () from >>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>> No symbol table info available. >>>> #19 0x00007f1b3e904e7f in ldbm_back_search () from >>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>> No symbol table info available. >>>> #20 0x00007f1b4b2321d6 in op_shared_search () from >>>> /usr/lib64/dirsrv/libslapd.so.0 >>>> No symbol table info available. >>>> #21 0x00007f1b4b24294e in search_internal_callback_pb () from >>>> /usr/lib64/dirsrv/libslapd.so.0 >>>> No symbol table info available. >>>> #22 0x00007f1b4b242be8 in search_internal_pb () from >>>> /usr/lib64/dirsrv/libslapd.so.0 >>>> No symbol table info available. >>>> #23 0x00007f1b4b72149b in ids_sasl_canon_user () >>>> No symbol table info available. >>>> #24 0x00007f1b491e193b in _sasl_canon_user () from /lib64/libsasl2.so.3 >>>> No symbol table info available. >>>> #25 0x00007f1b491e1c4c in _sasl_canon_user_lookup () from >>>> /lib64/libsasl2.so.3 >>>> No symbol table info available. >>>> #26 0x00007f1b430e9fb9 in gssapi_server_mech_step () from >>>> /usr/lib64/sasl2/libgssapiv2.so >>>> No symbol table info available. >>>> #27 0x00007f1b491edb9b in sasl_server_step () from /lib64/libsasl2.so.3 >>>> No symbol table info available. >>>> #28 0x00007f1b4b72283b in ids_sasl_check_bind () >>>> No symbol table info available. >>>> #29 0x00007f1b4b70a61a in do_bind () >>>> No symbol table info available. >>>> #30 0x00007f1b4b7113fd in connection_threadmain () >>>> No symbol table info available. >>>> #31 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so >>>> No symbol table info available. >>>> #32 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 >>>> No symbol table info available. >>>> #33 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 >>>> No symbol table info available. >>>> >>>> Thread 25 (Thread 0x7f1b0c7f8700 (LWP 105185)): >>>> #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 () from >>>> /lib64/libpthread.so.0 >>>> No symbol table info available. >>>> #1 0x00007f1b4941b050 in PR_WaitCondVar () from /lib64/libnspr4.so >>>> No symbol table info available. >>>> #2 0x00007f1b4b70efee in connection_wait_for_new_work () >>>> No symbol table info available. >>>> #3 0x00007f1b4b710719 in connection_threadmain () >>>> No symbol table info available. >>>> #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so >>>> No symbol table info available. >>>> #5 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 >>>> No symbol table info available. >>>> #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 >>>> No symbol table info available. >>>> >>>> Thread 24 (Thread 0x7f1b0cff9700 (LWP 105184)): >>>> #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 () from >>>> /lib64/libpthread.so.0 >>>> No symbol table info available. >>>> #1 0x00007f1b4941b050 in PR_WaitCondVar () from /lib64/libnspr4.so >>>> No symbol table info available. >>>> #2 0x00007f1b4b70efee in connection_wait_for_new_work () >>>> No symbol table info available. >>>> #3 0x00007f1b4b710719 in connection_threadmain () >>>> No symbol table info available. >>>> #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so >>>> No symbol table info available. >>>> #5 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 >>>> No symbol table info available. >>>> #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 >>>> No symbol table info available. >>>> >>>> Thread 23 (Thread 0x7f1b0d7fa700 (LWP 105183)): >>>> #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 () from >>>> /lib64/libpthread.so.0 >>>> No symbol table info available. >>>> #1 0x00007f1b4941b050 in PR_WaitCondVar () from /lib64/libnspr4.so >>>> No symbol table info available. >>>> #2 0x00007f1b4b70efee in connection_wait_for_new_work () >>>> No symbol table info available. >>>> #3 0x00007f1b4b710719 in connection_threadmain () >>>> No symbol table info available. >>>> #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so >>>> No symbol table info available. >>>> #5 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 >>>> No symbol table info available. >>>> #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 >>>> No symbol table info available. >>>> >>>> Thread 22 (Thread 0x7f1b0dffb700 (LWP 105182)): >>>> #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 () from >>>> /lib64/libpthread.so.0 >>>> No symbol table info available. >>>> #1 0x00007f1b4941b050 in PR_WaitCondVar () from /lib64/libnspr4.so >>>> No symbol table info available. >>>> #2 0x00007f1b4b70efee in connection_wait_for_new_work () >>>> No symbol table info available. >>>> #3 0x00007f1b4b710719 in connection_threadmain () >>>> No symbol table info available. >>>> #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so >>>> No symbol table info available. >>>> #5 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 >>>> No symbol table info available. >>>> #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 >>>> No symbol table info available. >>>> >>>> Thread 21 (Thread 0x7f1b0e7fc700 (LWP 105181)): >>>> #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 () from >>>> /lib64/libpthread.so.0 >>>> No symbol table info available. >>>> #1 0x00007f1b4941b050 in PR_WaitCondVar () from /lib64/libnspr4.so >>>> No symbol table info available. >>>> #2 0x00007f1b4b70efee in connection_wait_for_new_work () >>>> No symbol table info available. >>>> #3 0x00007f1b4b710719 in connection_threadmain () >>>> No symbol table info available. >>>> #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so >>>> No symbol table info available. >>>> #5 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 >>>> No symbol table info available. >>>> #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 >>>> No symbol table info available. >>>> >>>> Thread 20 (Thread 0x7f1b0effd700 (LWP 105180)): >>>> #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 () from >>>> /lib64/libpthread.so.0 >>>> No symbol table info available. >>>> #1 0x00007f1b433212f3 in __db_hybrid_mutex_suspend () from /lib64/ >>>> libdb-5.3.so >>>> No symbol table info available. >>>> #2 0x00007f1b43320b37 in __db_tas_mutex_readlock () from /lib64/ >>>> libdb-5.3.so >>>> No symbol table info available. >>>> #3 0x00007f1b4343ad35 in __memp_fget () from /lib64/libdb-5.3.so >>>> No symbol table info available. >>>> #4 0x00007f1b433e4333 in __dbc_iget () from /lib64/libdb-5.3.so >>>> No symbol table info available. >>>> #5 0x00007f1b433f2b02 in __dbc_get_pp () from /lib64/libdb-5.3.so >>>> No symbol table info available. >>>> #6 0x00007f1b3e8ce822 in idl_new_fetch () from >>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>> No symbol table info available. >>>> #7 0x00007f1b3e8dcb26 in index_read_ext_allids () from >>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>> No symbol table info available. >>>> #8 0x00007f1b3e8c7284 in keys2idl () from >>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>> No symbol table info available. >>>> #9 0x00007f1b3e8c79e3 in ava_candidates.isra.0 () from >>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>> No symbol table info available. >>>> #10 0x00007f1b3e8c7fd2 in filter_candidates_ext () from >>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>> No symbol table info available. >>>> #11 0x00007f1b3e8c9096 in list_candidates () from >>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>> No symbol table info available. >>>> #12 0x00007f1b3e8c7f40 in filter_candidates_ext () from >>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>> No symbol table info available. >>>> #13 0x00007f1b3e8c9096 in list_candidates () from >>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>> No symbol table info available. >>>> #14 0x00007f1b3e8c7f40 in filter_candidates_ext () from >>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>> No symbol table info available. >>>> #15 0x00007f1b3e8c9096 in list_candidates () from >>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>> No symbol table info available. >>>> #16 0x00007f1b3e8c7f40 in filter_candidates_ext () from >>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>> No symbol table info available. >>>> #17 0x00007f1b3e90380c in subtree_candidates () from >>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>> No symbol table info available. >>>> #18 0x00007f1b3e904e7f in ldbm_back_search () from >>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>> No symbol table info available. >>>> #19 0x00007f1b4b2321d6 in op_shared_search () from >>>> /usr/lib64/dirsrv/libslapd.so.0 >>>> No symbol table info available. >>>> #20 0x00007f1b4b72333e in do_search () >>>> No symbol table info available. >>>> #21 0x00007f1b4b7113c3 in connection_threadmain () >>>> No symbol table info available. >>>> #22 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so >>>> No symbol table info available. >>>> #23 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 >>>> No symbol table info available. >>>> #24 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 >>>> No symbol table info available. >>>> >>>> Thread 19 (Thread 0x7f1b0f7fe700 (LWP 105179)): >>>> #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 () from >>>> /lib64/libpthread.so.0 >>>> No symbol table info available. >>>> #1 0x00007f1b433212f3 in __db_hybrid_mutex_suspend () from /lib64/ >>>> libdb-5.3.so >>>> No symbol table info available. >>>> #2 0x00007f1b43320640 in __db_tas_mutex_lock () from /lib64/ >>>> libdb-5.3.so >>>> No symbol table info available. >>>> #3 0x00007f1b433cad3a in __lock_get_internal () from /lib64/ >>>> libdb-5.3.so >>>> No symbol table info available. >>>> #4 0x00007f1b433cb820 in __lock_get () from /lib64/libdb-5.3.so >>>> No symbol table info available. >>>> #5 0x00007f1b433f7142 in __db_lget () from /lib64/libdb-5.3.so >>>> No symbol table info available. >>>> #6 0x00007f1b4333e605 in __bam_search () from /lib64/libdb-5.3.so >>>> No symbol table info available. >>>> #7 0x00007f1b43329256 in __bamc_search () from /lib64/libdb-5.3.so >>>> No symbol table info available. >>>> #8 0x00007f1b4332ad0f in __bamc_get () from /lib64/libdb-5.3.so >>>> No symbol table info available. >>>> #9 0x00007f1b433e3ca6 in __dbc_iget () from /lib64/libdb-5.3.so >>>> No symbol table info available. >>>> #10 0x00007f1b433f2b02 in __dbc_get_pp () from /lib64/libdb-5.3.so >>>> No symbol table info available. >>>> #11 0x00007f1b3e8ce680 in idl_new_fetch () from >>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>> No symbol table info available. >>>> #12 0x00007f1b3e8dcb26 in index_read_ext_allids () from >>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>> No symbol table info available. >>>> #13 0x00007f1b3e8c7284 in keys2idl () from >>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>> No symbol table info available. >>>> #14 0x00007f1b3e8c79e3 in ava_candidates.isra.0 () from >>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>> No symbol table info available. >>>> #15 0x00007f1b3e8c7fd2 in filter_candidates_ext () from >>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>> No symbol table info available. >>>> #16 0x00007f1b3e8c9096 in list_candidates () from >>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>> No symbol table info available. >>>> #17 0x00007f1b3e8c7f40 in filter_candidates_ext () from >>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>> No symbol table info available. >>>> #18 0x00007f1b3e8c9096 in list_candidates () from >>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>> No symbol table info available. >>>> #19 0x00007f1b3e8c7f40 in filter_candidates_ext () from >>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>> No symbol table info available. >>>> #20 0x00007f1b3e8c9096 in list_candidates () from >>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>> No symbol table info available. >>>> #21 0x00007f1b3e8c7f40 in filter_candidates_ext () from >>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>> No symbol table info available. >>>> #22 0x00007f1b3e90380c in subtree_candidates () from >>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>> No symbol table info available. >>>> #23 0x00007f1b3e904e7f in ldbm_back_search () from >>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>> No symbol table info available. >>>> #24 0x00007f1b4b2321d6 in op_shared_search () from >>>> /usr/lib64/dirsrv/libslapd.so.0 >>>> No symbol table info available. >>>> #25 0x00007f1b4b72333e in do_search () >>>> No symbol table info available. >>>> #26 0x00007f1b4b7113c3 in connection_threadmain () >>>> No symbol table info available. >>>> #27 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so >>>> No symbol table info available. >>>> #28 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 >>>> No symbol table info available. >>>> #29 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 >>>> No symbol table info available. >>>> >>>> Thread 18 (Thread 0x7f1b0ffff700 (LWP 105178)): >>>> #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 () from >>>> /lib64/libpthread.so.0 >>>> No symbol table info available. >>>> #1 0x00007f1b4941b050 in PR_WaitCondVar () from /lib64/libnspr4.so >>>> No symbol table info available. >>>> #2 0x00007f1b4b70efee in connection_wait_for_new_work () >>>> No symbol table info available. >>>> #3 0x00007f1b4b710719 in connection_threadmain () >>>> No symbol table info available. >>>> #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so >>>> No symbol table info available. >>>> #5 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 >>>> No symbol table info available. >>>> #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 >>>> No symbol table info available. >>>> >>>> Thread 17 (Thread 0x7f1b14d28700 (LWP 105177)): >>>> #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 () from >>>> /lib64/libpthread.so.0 >>>> No symbol table info available. >>>> #1 0x00007f1b4941b050 in PR_WaitCondVar () from /lib64/libnspr4.so >>>> No symbol table info available. >>>> #2 0x00007f1b4b70efee in connection_wait_for_new_work () >>>> No symbol table info available. >>>> #3 0x00007f1b4b710719 in connection_threadmain () >>>> No symbol table info available. >>>> #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so >>>> No symbol table info available. >>>> #5 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 >>>> No symbol table info available. >>>> #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 >>>> No symbol table info available. >>>> >>>> Thread 16 (Thread 0x7f1b15529700 (LWP 105176)): >>>> #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 () from >>>> /lib64/libpthread.so.0 >>>> No symbol table info available. >>>> #1 0x00007f1b4941b050 in PR_WaitCondVar () from /lib64/libnspr4.so >>>> No symbol table info available. >>>> #2 0x00007f1b4b70efee in connection_wait_for_new_work () >>>> No symbol table info available. >>>> #3 0x00007f1b4b710719 in connection_threadmain () >>>> No symbol table info available. >>>> #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so >>>> No symbol table info available. >>>> #5 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 >>>> No symbol table info available. >>>> #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 >>>> No symbol table info available. >>>> >>>> Thread 15 (Thread 0x7f1b15d2a700 (LWP 105175)): >>>> #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 () from >>>> /lib64/libpthread.so.0 >>>> No symbol table info available. >>>> #1 0x00007f1b4941b050 in PR_WaitCondVar () from /lib64/libnspr4.so >>>> No symbol table info available. >>>> #2 0x00007f1b4b70efee in connection_wait_for_new_work () >>>> No symbol table info available. >>>> #3 0x00007f1b4b710719 in connection_threadmain () >>>> No symbol table info available. >>>> #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so >>>> No symbol table info available. >>>> #5 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 >>>> No symbol table info available. >>>> #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 >>>> No symbol table info available. >>>> >>>> Thread 14 (Thread 0x7f1b1652b700 (LWP 105174)): >>>> #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 () from >>>> /lib64/libpthread.so.0 >>>> No symbol table info available. >>>> #1 0x00007f1b4941b050 in PR_WaitCondVar () from /lib64/libnspr4.so >>>> No symbol table info available. >>>> #2 0x00007f1b4b70efee in connection_wait_for_new_work () >>>> No symbol table info available. >>>> #3 0x00007f1b4b710719 in connection_threadmain () >>>> No symbol table info available. >>>> #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so >>>> No symbol table info available. >>>> #5 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 >>>> No symbol table info available. >>>> #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 >>>> No symbol table info available. >>>> >>>> Thread 13 (Thread 0x7f1b16d2c700 (LWP 105173)): >>>> #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 () from >>>> /lib64/libpthread.so.0 >>>> No symbol table info available. >>>> #1 0x00007f1b433212f3 in __db_hybrid_mutex_suspend () from /lib64/ >>>> libdb-5.3.so >>>> No symbol table info available. >>>> #2 0x00007f1b43320640 in __db_tas_mutex_lock () from /lib64/ >>>> libdb-5.3.so >>>> No symbol table info available. >>>> #3 0x00007f1b433cad3a in __lock_get_internal () from /lib64/ >>>> libdb-5.3.so >>>> No symbol table info available. >>>> #4 0x00007f1b433cb820 in __lock_get () from /lib64/libdb-5.3.so >>>> No symbol table info available. >>>> #5 0x00007f1b433f7142 in __db_lget () from /lib64/libdb-5.3.so >>>> No symbol table info available. >>>> #6 0x00007f1b4333e605 in __bam_search () from /lib64/libdb-5.3.so >>>> No symbol table info available. >>>> #7 0x00007f1b43329256 in __bamc_search () from /lib64/libdb-5.3.so >>>> No symbol table info available. >>>> #8 0x00007f1b4332ad0f in __bamc_get () from /lib64/libdb-5.3.so >>>> No symbol table info available. >>>> #9 0x00007f1b433e3ca6 in __dbc_iget () from /lib64/libdb-5.3.so >>>> No symbol table info available. >>>> #10 0x00007f1b433f2b02 in __dbc_get_pp () from /lib64/libdb-5.3.so >>>> No symbol table info available. >>>> #11 0x00007f1b3e8ce680 in idl_new_fetch () from >>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>> No symbol table info available. >>>> #12 0x00007f1b3e8dcb26 in index_read_ext_allids () from >>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>> No symbol table info available. >>>> #13 0x00007f1b3e8c7284 in keys2idl () from >>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>> No symbol table info available. >>>> #14 0x00007f1b3e8c79e3 in ava_candidates.isra.0 () from >>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>> No symbol table info available. >>>> #15 0x00007f1b3e8c7fd2 in filter_candidates_ext () from >>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>> No symbol table info available. >>>> #16 0x00007f1b3e8c9096 in list_candidates () from >>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>> No symbol table info available. >>>> #17 0x00007f1b3e8c7f40 in filter_candidates_ext () from >>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>> No symbol table info available. >>>> #18 0x00007f1b3e8c9096 in list_candidates () from >>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>> No symbol table info available. >>>> #19 0x00007f1b3e8c7f40 in filter_candidates_ext () from >>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>> No symbol table info available. >>>> #20 0x00007f1b3e8c9096 in list_candidates () from >>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>> No symbol table info available. >>>> #21 0x00007f1b3e8c7f40 in filter_candidates_ext () from >>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>> No symbol table info available. >>>> #22 0x00007f1b3e90380c in subtree_candidates () from >>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>> No symbol table info available. >>>> #23 0x00007f1b3e904e7f in ldbm_back_search () from >>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>> No symbol table info available. >>>> #24 0x00007f1b4b2321d6 in op_shared_search () from >>>> /usr/lib64/dirsrv/libslapd.so.0 >>>> No symbol table info available. >>>> #25 0x00007f1b4b72333e in do_search () >>>> No symbol table info available. >>>> #26 0x00007f1b4b7113c3 in connection_threadmain () >>>> No symbol table info available. >>>> #27 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so >>>> No symbol table info available. >>>> #28 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 >>>> No symbol table info available. >>>> #29 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 >>>> No symbol table info available. >>>> >>>> Thread 12 (Thread 0x7f1b1752d700 (LWP 105172)): >>>> #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 () from >>>> /lib64/libpthread.so.0 >>>> No symbol table info available. >>>> #1 0x00007f1b4941b050 in PR_WaitCondVar () from /lib64/libnspr4.so >>>> No symbol table info available. >>>> #2 0x00007f1b4b70efee in connection_wait_for_new_work () >>>> No symbol table info available. >>>> #3 0x00007f1b4b710719 in connection_threadmain () >>>> No symbol table info available. >>>> #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so >>>> No symbol table info available. >>>> #5 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 >>>> No symbol table info available. >>>> #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 >>>> No symbol table info available. >>>> >>>> Thread 11 (Thread 0x7f1b17d2e700 (LWP 105171)): >>>> #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 () from >>>> /lib64/libpthread.so.0 >>>> No symbol table info available. >>>> #1 0x00007f1b4941b050 in PR_WaitCondVar () from /lib64/libnspr4.so >>>> No symbol table info available. >>>> #2 0x00007f1b4b70efee in connection_wait_for_new_work () >>>> No symbol table info available. >>>> #3 0x00007f1b4b710719 in connection_threadmain () >>>> No symbol table info available. >>>> #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so >>>> No symbol table info available. >>>> #5 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 >>>> No symbol table info available. >>>> #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 >>>> No symbol table info available. >>>> >>>> Thread 10 (Thread 0x7f1b2509d700 (LWP 105168)): >>>> #0 0x00007f1b48dc5a82 in pthread_cond_timedwait@@GLIBC_2.3.2 () from >>>> /lib64/libpthread.so.0 >>>> No symbol table info available. >>>> #1 0x00007f1b4941ab07 in pt_TimedWait () from /lib64/libnspr4.so >>>> No symbol table info available. >>>> #2 0x00007f1b4941afce in PR_WaitCondVar () from /lib64/libnspr4.so >>>> No symbol table info available. >>>> #3 0x00007f1b4b2061e8 in eq_loop () from /usr/lib64/dirsrv/libslapd.so. >>>> 0 >>>> No symbol table info available. >>>> #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so >>>> No symbol table info available. >>>> #5 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 >>>> No symbol table info available. >>>> #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 >>>> No symbol table info available. >>>> >>>> Thread 9 (Thread 0x7f1b2589e700 (LWP 105167)): >>>> #0 0x00007f1b48dc5a82 in pthread_cond_timedwait@@GLIBC_2.3.2 () from >>>> /lib64/libpthread.so.0 >>>> No symbol table info available. >>>> #1 0x00007f1b4941ab07 in pt_TimedWait () from /lib64/libnspr4.so >>>> No symbol table info available. >>>> #2 0x00007f1b4941afce in PR_WaitCondVar () from /lib64/libnspr4.so >>>> No symbol table info available. >>>> #3 0x00007f1b4b719d03 in housecleaning () >>>> No symbol table info available. >>>> #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so >>>> No symbol table info available. >>>> #5 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 >>>> No symbol table info available. >>>> #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 >>>> No symbol table info available. >>>> >>>> Thread 8 (Thread 0x7f1b2609f700 (LWP 105166)): >>>> #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 () from >>>> /lib64/libpthread.so.0 >>>> No symbol table info available. >>>> #1 0x00007f1b4941b050 in PR_WaitCondVar () from /lib64/libnspr4.so >>>> No symbol table info available. >>>> #2 0x00007f1b4b260198 in slapi_wait_condvar () from >>>> /usr/lib64/dirsrv/libslapd.so.0 >>>> No symbol table info available. >>>> #3 0x00007f1b3cf5deed in roles_cache_wait_on_change () from >>>> /usr/lib64/dirsrv/plugins/libroles-plugin.so >>>> No symbol table info available. >>>> #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so >>>> No symbol table info available. >>>> #5 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 >>>> No symbol table info available. >>>> #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 >>>> No symbol table info available. >>>> >>>> Thread 7 (Thread 0x7f1b268a0700 (LWP 105165)): >>>> #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 () from >>>> /lib64/libpthread.so.0 >>>> No symbol table info available. >>>> #1 0x00007f1b4941b050 in PR_WaitCondVar () from /lib64/libnspr4.so >>>> No symbol table info available. >>>> #2 0x00007f1b4b260198 in slapi_wait_condvar () from >>>> /usr/lib64/dirsrv/libslapd.so.0 >>>> No symbol table info available. >>>> #3 0x00007f1b3cf5deed in roles_cache_wait_on_change () from >>>> /usr/lib64/dirsrv/plugins/libroles-plugin.so >>>> No symbol table info available. >>>> #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so >>>> No symbol table info available. >>>> #5 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 >>>> No symbol table info available. >>>> #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 >>>> No symbol table info available. >>>> >>>> Thread 6 (Thread 0x7f1b270a1700 (LWP 105164)): >>>> #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 () from >>>> /lib64/libpthread.so.0 >>>> No symbol table info available. >>>> #1 0x00007f1b4941b050 in PR_WaitCondVar () from /lib64/libnspr4.so >>>> No symbol table info available. >>>> #2 0x00007f1b4b260198 in slapi_wait_condvar () from >>>> /usr/lib64/dirsrv/libslapd.so.0 >>>> No symbol table info available. >>>> #3 0x00007f1b3cf5deed in roles_cache_wait_on_change () from >>>> /usr/lib64/dirsrv/plugins/libroles-plugin.so >>>> No symbol table info available. >>>> #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so >>>> No symbol table info available. >>>> #5 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 >>>> No symbol table info available. >>>> #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 >>>> No symbol table info available. >>>> >>>> Thread 5 (Thread 0x7f1b278a2700 (LWP 105163)): >>>> #0 0x00007f1b48dc56d5 in pthread_cond_wait@@GLIBC_2.3.2 () from >>>> /lib64/libpthread.so.0 >>>> No symbol table info available. >>>> #1 0x00007f1b4941b050 in PR_WaitCondVar () from /lib64/libnspr4.so >>>> No symbol table info available. >>>> #2 0x00007f1b4b260198 in slapi_wait_condvar () from >>>> /usr/lib64/dirsrv/libslapd.so.0 >>>> No symbol table info available. >>>> #3 0x00007f1b415f262e in cos_cache_wait_on_change () from >>>> /usr/lib64/dirsrv/plugins/libcos-plugin.so >>>> No symbol table info available. >>>> #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so >>>> No symbol table info available. >>>> #5 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 >>>> No symbol table info available. >>>> #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 >>>> No symbol table info available. >>>> >>>> Thread 4 (Thread 0x7f1b282ac700 (LWP 105162)): >>>> #0 0x00007f1b48ae69b3 in select () from /lib64/libc.so.6 >>>> No symbol table info available. >>>> #1 0x00007f1b4b2710e9 in DS_Sleep () from >>>> /usr/lib64/dirsrv/libslapd.so.0 >>>> No symbol table info available. >>>> #2 0x00007f1b3e8ba707 in perf_threadmain () from >>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>> No symbol table info available. >>>> #3 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so >>>> No symbol table info available. >>>> #4 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 >>>> No symbol table info available. >>>> #5 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 >>>> No symbol table info available. >>>> >>>> Thread 3 (Thread 0x7f1b28aad700 (LWP 105161)): >>>> #0 0x00007f1b48ae69b3 in select () from /lib64/libc.so.6 >>>> No symbol table info available. >>>> #1 0x00007f1b4b2710e9 in DS_Sleep () from >>>> /usr/lib64/dirsrv/libslapd.so.0 >>>> No symbol table info available. >>>> #2 0x00007f1b3e8bfb7f in trickle_threadmain () from >>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>> No symbol table info available. >>>> #3 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so >>>> No symbol table info available. >>>> #4 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 >>>> No symbol table info available. >>>> #5 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 >>>> No symbol table info available. >>>> >>>> Thread 2 (Thread 0x7f1b292ae700 (LWP 105160)): >>>> #0 0x00007f1b48ae69b3 in select () from /lib64/libc.so.6 >>>> No symbol table info available. >>>> #1 0x00007f1b4344dbad in __os_yield () from /lib64/libdb-5.3.so >>>> No symbol table info available. >>>> #2 0x00007f1b434492b3 in __memp_sync_int () from /lib64/libdb-5.3.so >>>> No symbol table info available. >>>> #3 0x00007f1b43459752 in __txn_checkpoint () from /lib64/libdb-5.3.so >>>> No symbol table info available. >>>> #4 0x00007f1b43459b74 in __txn_checkpoint_pp () from /lib64/ >>>> libdb-5.3.so >>>> No symbol table info available. >>>> #5 0x00007f1b3e8c3a87 in checkpoint_threadmain () from >>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>> No symbol table info available. >>>> #6 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so >>>> No symbol table info available. >>>> #7 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 >>>> No symbol table info available. >>>> #8 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 >>>> No symbol table info available. >>>> >>>> Thread 1 (Thread 0x7f1b29aaf700 (LWP 105159)): >>>> #0 0x00007f1b48ae69b3 in select () from /lib64/libc.so.6 >>>> No symbol table info available. >>>> #1 0x00007f1b4b2710e9 in DS_Sleep () from >>>> /usr/lib64/dirsrv/libslapd.so.0 >>>> No symbol table info available. >>>> #2 0x00007f1b3e8bf907 in deadlock_threadmain () from >>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>> No symbol table info available. >>>> #3 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so >>>> No symbol table info available. >>>> #4 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 >>>> No symbol table info available. >>>> #5 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 >>>> No symbol table info available. >>>> >>>> >>>> >>>> Thanks, >>>> >>>> Rakesh >>>> >>>> >>>> On Mon, Sep 5, 2016 at 4:48 PM, thierry bordaz >>>> wrote: >>>> >>>>> >>>>> >>>>> On 09/05/2016 12:05 PM, Rakesh Rajasekharan wrote: >>>>> >>>>> Hi Thierry, >>>>> >>>>> >>>>> I was getting the hang issue while running ipa-client-install >>>>> simultaneously on few clients.. >>>>> However, today, I am not able to replicate that. >>>>> >>>>> I could not get a gdb . But i will try getting that the next time I >>>>> face this issue. >>>>> >>>>> The CPU does not stay high.. it just momentarily touches a high value >>>>> and then drops down to around 2-7% >>>>> >>>>> One question I have is , is it ok to set it nsslapd-threadnumber to a >>>>> very high value . >>>>> I have around 4000 clients and with nsslapd-maxthreadsperconn set to >>>>> 5..So, can I set nsslapd-threadnumber to around 25000. >>>>> >>>>> >>>>> Hello, >>>>> >>>>> I know some users running in production with several hunderds of >>>>> threads (>600) and this without problem. >>>>> >>>>> I do not recall having suggested to increase that number and for what >>>>> reason. >>>>> Usually 30 workers is a good enough value. It can create bootleneck if >>>>> for some reason each operation is very long to satisfy and exhaust the >>>>> number of workers. You can monitor the work queue: >>>>> >>>>> ldapsearch -D "cn=directory manager" -w xxx -LLL -b "cn=monitor" -s >>>>> base opsinitiated opscompleted >>>>> >>>>> >>>>> If opscompleted-opsinitiated remains close to threadnumber, then yes >>>>> it would be valuable to increase it. >>>>> >>>>> The computation #client * #async_op_per_client sound an overkill. Even >>>>> if all clients send at the exact same time all their requests, it is very >>>>> likely that some common resource (db page, log, allocator...) will >>>>> serialize them. If you monitor a need to increase the work, you would for >>>>> example set it to 50, then monitor, then set it to 100, then monitor... >>>>> until you find a good enough value. >>>>> Note the increasing the #thread, increases the memory footprint that >>>>> will reduce the efficiency of file system cache and can increase the >>>>> response time. >>>>> >>>>> >>>>> best regards >>>>> thierry >>>>> >>>>> >>>>> Thanks >>>>> >>>>> On Mon, Sep 5, 2016 at 1:03 PM, thierry bordaz >>>>> wrote: >>>>> >>>>>> >>>>>> Hi Rakesh, >>>>>> >>>>>> Were you able to get a pstack or full stack with gdb ( >>>>>> http://www.port389.org/docs/389ds/FAQ/faq.html#debugging-crashes) >>>>>> when the server hangs ? >>>>>> >>>>>> If it happens with 500 threads as well as with 30, using 30 threads >>>>>> is a better choice to debug this issue. >>>>>> I will try to reproduce using 150 parallel 'ipa user-find p-testipa' >>>>>> commands >>>>>> >>>>>> Something I am unsure is if the CPU consumption stays high (you >>>>>> mentioned 340% CPU usage) as long as the hang happens or if after a suddent >>>>>> shot up to 340% (that marks the beginning of the hang) it drops and stay >>>>>> hanging ? >>>>>> >>>>>> thanks >>>>>> thierry >>>>>> >>>>>> On 09/04/2016 08:40 PM, Rakesh Rajasekharan wrote: >>>>>> >>>>>> starce on the slapd process actually had this in the output.. >>>>>> FUTEX_WAIT_PRIVATE >>>>>> >>>>>> and checking for the number of threads slapd had.. there were 5015 >>>>>> threads >>>>>> >>>>>> ps -efL|grep slapd|wc -l >>>>>> 5015 >>>>>> >>>>>> strace on most of the threads gave this output >>>>>> >>>>>> strace -p 67411 >>>>>> Process 67411 attached >>>>>> futex(0x7f3f0226b41c, FUTEX_WAIT_PRIVATE, 1, NULL) = -1 EAGAIN >>>>>> (Resource temporarily unavailable) >>>>>> futex(0x7f3f0226b41c, FUTEX_WAIT_PRIVATE, 2, NULL^CProcess 67411 >>>>>> detached >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> On Sun, Sep 4, 2016 at 5:34 PM, Rakesh Rajasekharan < >>>>>> rakesh.rajasekharan at gmail.com> wrote: >>>>>> >>>>>>> I have again got the issue of IPA hanging.. The issue came up when i >>>>>>> tried to run ipa-client-isntall on 142 clients simultaneously >>>>>>> >>>>>>> >>>>>>> None of the IPA commands are responding, and I see this error >>>>>>> >>>>>>> ipa user-find p-testipa >>>>>>> ipa: ERROR: Insufficient access: SASL(-1): generic failure: GSSAPI >>>>>>> Error: Unspecified GSS failure. Minor code may provide more information >>>>>>> (KDC returned error string: PROCESS_TGS) >>>>>>> >>>>>>> KRB5_TRACE=/dev/stdout kinit admin >>>>>>> [41178] 1472984115.233214: Getting initial credentials for >>>>>>> admin at XYZ.COM >>>>>>> [41178] 1472984115.235257: Sending request (167 bytes) to XYZ.COM >>>>>>> [41178] 1472984115.235419: Initiating TCP connection to stream >>>>>>> 10.1.3.36:88 >>>>>>> [41178] 1472984115.235685: Sending TCP request to stream >>>>>>> 10.1.3.36:88 >>>>>>> [41178] 1472984120.238914: Received answer (174 bytes) from stream >>>>>>> 10.1.3.36:88 >>>>>>> [41178] 1472984120.238925: Terminating TCP connection to stream >>>>>>> 10.1.3.36:88 >>>>>>> [41178] 1472984120.238993: Response was from master KDC >>>>>>> [41 >>>>>>> >>>>>>> >>>>>>> Running an ldapsearch to see the db.. does not give any results and >>>>>>> just hangs there >>>>>>> >>>>>>> ldapsearch -x -D 'cn=Directory Manager' -W -s one -b >>>>>>> 'cn=kerberos,dc=xyz,dc=com' >>>>>>> Enter LDAP Password: >>>>>>> >>>>>>> even an ldapsearch -x does not respond >>>>>>> At this point, am sure that slapd is the one causing issues >>>>>>> >>>>>>> Running an strace against the hung slapd itself seems to get stuck >>>>>>> does not proceed after saying "attaching to process" >>>>>>> >>>>>>> From some others posts I read Thierry suggesting to increase the >>>>>>> nsslapd-threadnumber value >>>>>>> >>>>>>> It was set to 30, I think that might be too low. >>>>>>> >>>>>> ... >>>> >>>> [Message clipped] >>> >>> >>> >> >> >> -- >> Red Hat GmbH, http://www.de.redhat.com/, Registered seat: Grasbrunn, >> Commercial register: Amtsgericht Muenchen, HRB 153243, >> Managing Directors: Charles Cachera, Michael Cunningham, Michael O'Neill, Eric Shander >> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From amit.active2008 at gmail.com Wed Sep 7 09:32:26 2016 From: amit.active2008 at gmail.com (Amit Gupta) Date: Wed, 7 Sep 2016 15:02:26 +0530 Subject: [Freeipa-users] FreeIPA server install in Ubuntu 14.04 box. Message-ID: Hi, I am looking for any document to install and config freeipa on ubuntu 14.04 server. Kindly share some reference document. Thanks, Amit Kumar -------------- next part -------------- An HTML attachment was scrubbed... URL: From amit.active2008 at gmail.com Wed Sep 7 09:34:00 2016 From: amit.active2008 at gmail.com (Amit Gupta) Date: Wed, 7 Sep 2016 15:04:00 +0530 Subject: [Freeipa-users] Zabbix intregration with FreeIPA Message-ID: Hi, I am looking for zabbix integration with FreeIPA. Please share some reference documents. Thanks, Amit Kumar -------------- next part -------------- An HTML attachment was scrubbed... URL: From natxo.asenjo at gmail.com Wed Sep 7 12:10:05 2016 From: natxo.asenjo at gmail.com (Natxo Asenjo) Date: Wed, 7 Sep 2016 14:10:05 +0200 Subject: [Freeipa-users] ipa: ERROR: Certificate format error: (SEC_ERROR_LEGACY_DATABASE) The certificate/key database is in an old, unsupported format. Message-ID: hi, using centos 6.8 (server and client), when trying to view some hosts we get this error: $ ipa host-find host-1920.sub.domain.tld ipa: ERROR: Certificate format error: (SEC_ERROR_LEGACY_DATABASE) The certificate/key database is in an old, unsupported format. I saw a thread last year about this, but no solution. Any clues? Thanks -- Groeten, natxo -------------- next part -------------- An HTML attachment was scrubbed... URL: From natxo.asenjo at gmail.com Wed Sep 7 12:15:51 2016 From: natxo.asenjo at gmail.com (Natxo Asenjo) Date: Wed, 7 Sep 2016 14:15:51 +0200 Subject: [Freeipa-users] ipa: ERROR: Certificate format error: (SEC_ERROR_LEGACY_DATABASE) The certificate/key database is in an old, unsupported format. In-Reply-To: References: Message-ID: On Wed, Sep 7, 2016 at 2:10 PM, Natxo Asenjo wrote: > hi, > > using centos 6.8 (server and client), when trying to view some hosts we > get this error: > > > $ ipa host-find host-1920.sub.domain.tld > ipa: ERROR: Certificate format error: (SEC_ERROR_LEGACY_DATABASE) The > certificate/key database is in an old, unsupported format. > this is happening wih all the host objects (all enrolled with certificates). I don't usually look at the hosts objects that much, but yesterday patches were applied and the ipa-server package was updated (among other things): Updated ipa-server-3.0.0-50.el6.centos.1.i686 Update 3.0.0-50.el6.centos.2.i686 So it looks like this is the culprit? -- Groeten, natxo -------------- next part -------------- An HTML attachment was scrubbed... URL: From fujisan43 at gmail.com Wed Sep 7 12:34:52 2016 From: fujisan43 at gmail.com (Fujisan) Date: Wed, 7 Sep 2016 14:34:52 +0200 Subject: [Freeipa-users] Freeipa not working after upgrade to F24 Message-ID: Hello, I just upgraded my server to F24 but ipa is not starting. No DNS, no access to the Freeipa web page. # dnf list freeipa* Last metadata expiration check: 4:15:15 ago on Wed Sep 7 10:08:23 2016. Installed Packages freeipa-admintools.noarch 4.3.2-2.fc24 @@commandline freeipa-client.x86_64 4.3.2-2.fc24 @@commandline freeipa-client-common.noarch 4.3.2-2.fc24 @@commandline freeipa-common.noarch 4.3.2-2.fc24 @@commandline freeipa-python-compat.noarch 4.3.2-2.fc24 @@commandline freeipa-server.x86_64 4.3.2-2.fc24 @@commandline freeipa-server-common.noarch 4.3.2-2.fc24 @@commandline freeipa-server-trust-ad.x86_64 4.3.2-2.fc24 @@commandline Available Packages freeipa-server-dns.noarch I see that freeipa-server-dns is not installed. I guess I have to install it. Is there anything I have to run to make ipa work again? When I run the following, i get the log below. # systemctl restart ipa Sep 7 14:11:04 myserver systemd: Stopped Identity, Policy, Audit. Sep 7 14:11:04 myserver systemd: Starting Identity, Policy, Audit... Sep 7 14:11:05 myserver ipactl: Existing service file detected! Sep 7 14:11:05 myserver ipactl: Assuming stale, cleaning and proceeding Sep 7 14:11:05 myserver systemd: Starting 389 Directory Server MYDOMAIN.... Sep 7 14:11:05 myserver ns-slapd: [07/Sep/2016:14:11:05.516376125 +0200] SSL alert: Sending pin request to SVRCore. You may need to run systemd-tty-ask-password-agent to provide the password. Sep 7 14:11:05 myserver ns-slapd: [07/Sep/2016:14:11:05.526228037 +0200] SSL alert: Security Initialization: Enabling default cipher set. Sep 7 14:11:05 myserver ns-slapd: [07/Sep/2016:14:11:05.532419554 +0200] SSL alert: Configured NSS Ciphers Sep 7 14:11:05 myserver ns-slapd: [07/Sep/2016:14:11:05.540442141 +0200] SSL alert: #011TLS_ECDHE_PSK_WITH_AES_128_GCM_SHA256: enabled Sep 7 14:11:05 myserver ns-slapd: [07/Sep/2016:14:11:05.548496058 +0200] SSL alert: #011TLS_ECDHE_PSK_WITH_CHACHA20_POLY1305_SHA256: enabled Sep 7 14:11:05 myserver ns-slapd: [07/Sep/2016:14:11:05.556478226 +0200] SSL alert: #011TLS_ECDHE_PSK_WITH_AES_256_GCM_SHA384: enabled Sep 7 14:11:05 myserver ns-slapd: [07/Sep/2016:14:11:05.564492649 +0200] SSL alert: #011TLS_DHE_PSK_WITH_AES_128_GCM_SHA256: enabled Sep 7 14:11:05 myserver ns-slapd: [07/Sep/2016:14:11:05.572505526 +0200] SSL alert: #011TLS_DHE_PSK_WITH_CHACHA20_POLY1305_SHA256: enabled Sep 7 14:11:05 myserver ns-slapd: [07/Sep/2016:14:11:05.580465223 +0200] SSL alert: #011TLS_DHE_PSK_WITH_AES_256_GCM_SHA384: enabled Sep 7 14:11:05 myserver ns-slapd: [07/Sep/2016:14:11:05.588482130 +0200] SSL alert: #011TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256: enabled Sep 7 14:11:05 myserver ns-slapd: [07/Sep/2016:14:11:05.596482461 +0200] SSL alert: #011TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256: enabled Sep 7 14:11:05 myserver ns-slapd: [07/Sep/2016:14:11:05.604436371 +0200] SSL alert: #011TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305_SHA256: enabled Sep 7 14:11:05 myserver ns-slapd: [07/Sep/2016:14:11:05.612490300 +0200] SSL alert: #011TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305_SHA256: enabled Sep 7 14:11:05 myserver ns-slapd: [07/Sep/2016:14:11:05.620467469 +0200] SSL alert: #011TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA: enabled Sep 7 14:11:05 myserver ns-slapd: [07/Sep/2016:14:11:05.628487228 +0200] SSL alert: #011TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA: enabled Sep 7 14:11:05 myserver ns-slapd: [07/Sep/2016:14:11:05.636499996 +0200] SSL alert: #011TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA: enabled Sep 7 14:11:05 myserver ns-slapd: [07/Sep/2016:14:11:05.644487569 +0200] SSL alert: #011TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256: enabled Sep 7 14:11:05 myserver ns-slapd: [07/Sep/2016:14:11:05.652491137 +0200] SSL alert: #011TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256: enabled Sep 7 14:11:05 myserver ns-slapd: [07/Sep/2016:14:11:05.660485110 +0200] SSL alert: #011TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA: enabled Sep 7 14:11:05 myserver ns-slapd: [07/Sep/2016:14:11:05.668493371 +0200] SSL alert: #011TLS_DHE_RSA_WITH_AES_128_GCM_SHA256: enabled Sep 7 14:11:05 myserver ns-slapd: [07/Sep/2016:14:11:05.676495739 +0200] SSL alert: #011TLS_DHE_RSA_WITH_CHACHA20_POLY1305_SHA256: enabled Sep 7 14:11:05 myserver ns-slapd: [07/Sep/2016:14:11:05.684488502 +0200] SSL alert: #011TLS_DHE_RSA_WITH_AES_128_CBC_SHA: enabled Sep 7 14:11:05 myserver ns-slapd: [07/Sep/2016:14:11:05.692488455 +0200] SSL alert: #011TLS_DHE_DSS_WITH_AES_128_CBC_SHA: enabled Sep 7 14:11:05 myserver ns-slapd: [07/Sep/2016:14:11:05.700492250 +0200] SSL alert: #011TLS_DHE_RSA_WITH_AES_128_CBC_SHA256: enabled Sep 7 14:11:05 myserver ns-slapd: [07/Sep/2016:14:11:05.708443758 +0200] SSL alert: #011TLS_DHE_RSA_WITH_AES_256_CBC_SHA: enabled Sep 7 14:11:05 myserver ns-slapd: [07/Sep/2016:14:11:05.716451343 +0200] SSL alert: #011TLS_DHE_DSS_WITH_AES_256_CBC_SHA: enabled Sep 7 14:11:05 myserver ns-slapd: [07/Sep/2016:14:11:05.724472692 +0200] SSL alert: #011TLS_DHE_RSA_WITH_AES_256_CBC_SHA256: enabled Sep 7 14:11:05 myserver ns-slapd: [07/Sep/2016:14:11:05.732490857 +0200] SSL alert: #011TLS_RSA_WITH_AES_128_GCM_SHA256: enabled Sep 7 14:11:05 myserver ns-slapd: [07/Sep/2016:14:11:05.740480319 +0200] SSL alert: #011TLS_RSA_WITH_AES_128_CBC_SHA: enabled Sep 7 14:11:05 myserver ns-slapd: [07/Sep/2016:14:11:05.748404253 +0200] SSL alert: #011TLS_RSA_WITH_AES_128_CBC_SHA256: enabled Sep 7 14:11:05 myserver ns-slapd: [07/Sep/2016:14:11:05.756448372 +0200] SSL alert: #011TLS_RSA_WITH_AES_256_CBC_SHA: enabled Sep 7 14:11:05 myserver ns-slapd: [07/Sep/2016:14:11:05.764357312 +0200] SSL alert: #011TLS_RSA_WITH_AES_256_CBC_SHA256: enabled Sep 7 14:11:05 myserver ns-slapd: [07/Sep/2016:14:11:05.838482026 +0200] SSL Initialization - Configured SSL version range: min: TLS1.0, max: TLS1.2 Sep 7 14:11:05 myserver ns-slapd: [07/Sep/2016:14:11:05.838879137 +0200] 389-Directory/1.3.5.13 B2016.221.207 starting up Sep 7 14:11:05 myserver ns-slapd: [07/Sep/2016:14:11:05.886627162 +0200] WARNING: userRoot: entry cache size 512000 B is less than db size 1220608 B; We recommend to increase the entry cache size nsslapd-cachememsize. Sep 7 14:11:05 myserver ns-slapd: [07/Sep/2016:14:11:05.886950543 +0200] WARNING: ipaca: entry cache size 512000 B is less than db size 1097728 B; We recommend to increase the entry cache size nsslapd-cachememsize. Sep 7 14:11:05 myserver ns-slapd: [07/Sep/2016:14:11:05.894438452 +0200] WARNING: changelog: entry cache size 512000 B is less than db size 20570112 B; We recommend to increase the entry cache size nsslapd-cachememsize. Sep 7 14:11:05 myserver ns-slapd: [07/Sep/2016:14:11:05.953423134 +0200] schema-compat-plugin - scheduled schema-compat-plugin tree scan in about 5 seconds after the server startup! Sep 7 14:11:05 myserver ns-slapd: [07/Sep/2016:14:11:05.975290858 +0200] NSACLPlugin - The ACL target cn=groups,cn=compat,dc=mydomain does not exist Sep 7 14:11:05 myserver ns-slapd: [07/Sep/2016:14:11:05.975886392 +0200] NSACLPlugin - The ACL target cn=computers,cn=compat,dc=mydomain does not exist Sep 7 14:11:05 myserver ns-slapd: [07/Sep/2016:14:11:05.983606011 +0200] NSACLPlugin - The ACL target cn=ng,cn=compat,dc=mydomain does not exist Sep 7 14:11:05 myserver ns-slapd: [07/Sep/2016:14:11:05.991657947 +0200] NSACLPlugin - The ACL target ou=sudoers,dc=mydomain does not exist Sep 7 14:11:05 myserver ns-slapd: [07/Sep/2016:14:11:05.999610507 +0200] NSACLPlugin - The ACL target cn=users,cn=compat,dc=mydomain does not exist Sep 7 14:11:06 myserver ns-slapd: [07/Sep/2016:14:11:06.007953922 +0200] NSACLPlugin - The ACL target cn=vaults,cn=kra,dc=mydomain does not exist Sep 7 14:11:06 myserver ns-slapd: [07/Sep/2016:14:11:06.015481701 +0200] NSACLPlugin - The ACL target cn=vaults,cn=kra,dc=mydomain does not exist Sep 7 14:11:06 myserver ns-slapd: [07/Sep/2016:14:11:06.023534205 +0200] NSACLPlugin - The ACL target cn=vaults,cn=kra,dc=mydomain does not exist Sep 7 14:11:06 myserver ns-slapd: [07/Sep/2016:14:11:06.031613942 +0200] NSACLPlugin - The ACL target cn=vaults,cn=kra,dc=mydomain does not exist Sep 7 14:11:06 myserver ns-slapd: [07/Sep/2016:14:11:06.039516804 +0200] NSACLPlugin - The ACL target cn=vaults,cn=kra,dc=mydomain does not exist Sep 7 14:11:06 myserver ns-slapd: [07/Sep/2016:14:11:06.047545327 +0200] NSACLPlugin - The ACL target cn=vaults,cn=kra,dc=mydomain does not exist Sep 7 14:11:06 myserver ns-slapd: [07/Sep/2016:14:11:06.055570094 +0200] NSACLPlugin - The ACL target cn=vaults,cn=kra,dc=mydomain does not exist Sep 7 14:11:06 myserver ns-slapd: [07/Sep/2016:14:11:06.063625432 +0200] NSACLPlugin - The ACL target cn=vaults,cn=kra,dc=mydomain does not exist Sep 7 14:11:06 myserver ns-slapd: [07/Sep/2016:14:11:06.071671997 +0200] NSACLPlugin - The ACL target cn=vaults,cn=kra,dc=mydomain does not exist Sep 7 14:11:06 myserver ns-slapd: [07/Sep/2016:14:11:06.079663479 +0200] NSACLPlugin - The ACL target cn=vaults,cn=kra,dc=mydomain does not exist Sep 7 14:11:06 myserver ns-slapd: [07/Sep/2016:14:11:06.087662618 +0200] NSACLPlugin - The ACL target cn=vaults,cn=kra,dc=mydomain does not exist Sep 7 14:11:06 myserver ns-slapd: [07/Sep/2016:14:11:06.104747969 +0200] NSACLPlugin - The ACL target cn=casigningcert cert-pki-ca,cn=ca_renewal,cn=ipa,cn=etc,dc=mydomain does not exist Sep 7 14:11:06 myserver ns-slapd: [07/Sep/2016:14:11:06.105365122 +0200] NSACLPlugin - The ACL target cn=casigningcert cert-pki-ca,cn=ca_renewal,cn=ipa,cn=etc,dc=mydomain does not exist Sep 7 14:11:06 myserver ns-slapd: [07/Sep/2016:14:11:06.251172314 +0200] NSACLPlugin - The ACL target cn=automember rebuild membership,cn=tasks,cn=config does not exist Sep 7 14:11:06 myserver ns-slapd: [07/Sep/2016:14:11:06.258681034 +0200] Skipping CoS Definition cn=Password Policy,cn=accounts,dc=mydomain--no CoS Templates found, which should be added before the CoS Definition. Sep 7 14:11:06 myserver ns-slapd: [07/Sep/2016:14:11:06.301182019 +0200] set_krb5_creds - Could not get initial credentials for principal [ldap/myserver.mydomain at MYDOMAIN] in keytab [FILE:/etc/dirsrv/ds.keytab]: -1765328228 (Cannot contact any KDC for requested realm) Sep 7 14:11:06 myserver ns-slapd: [07/Sep/2016:14:11:06.304367111 +0200] NSMMReplicationPlugin - agmt="cn=meTozaira.mydomain" (zaira:389): Replication bind with GSSAPI auth failed: LDAP error -1 (Can't contact LDAP server) () Sep 7 14:11:06 myserver ns-slapd: [07/Sep/2016:14:11:06.316611911 +0200] schema-compat-plugin - schema-compat-plugin tree scan will start in about 5 seconds! Sep 7 14:11:06 myserver ns-slapd: [07/Sep/2016:14:11:06.358519964 +0200] slapd started. Listening on All Interfaces port 389 for LDAP requests Sep 7 14:11:06 myserver ns-slapd: [07/Sep/2016:14:11:06.370285321 +0200] Listening on All Interfaces port 636 for LDAPS requests Sep 7 14:11:06 myserver ns-slapd: [07/Sep/2016:14:11:06.378403244 +0200] Listening on /var/run/slapd-MYDOMAIN.socket for LDAPI requests Sep 7 14:11:06 myserver systemd: Started 389 Directory Server MYDOMAIN.. Sep 7 14:11:06 myserver audit: SERVICE_START pid=1 uid=0 auid=4294967295 ses=4294967295 msg='unit=dirsrv at MYDOMAIN comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success' Sep 7 14:11:06 myserver systemd: Starting Kerberos 5 KDC... Sep 7 14:11:06 myserver systemd: Started Kerberos 5 KDC. Sep 7 14:11:06 myserver audit: SERVICE_START pid=1 uid=0 auid=4294967295 ses=4294967295 msg='unit=krb5kdc comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success' Sep 7 14:11:06 myserver systemd: Starting Kerberos 5 Password-changing and Administration... Sep 7 14:11:06 myserver systemd: Started Kerberos 5 Password-changing and Administration. Sep 7 14:11:06 myserver audit: SERVICE_START pid=1 uid=0 auid=4294967295 ses=4294967295 msg='unit=kadmin comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success' Sep 7 14:11:06 myserver systemd: Starting Generate rndc key for BIND (DNS)... Sep 7 14:11:06 myserver systemd: Started Generate rndc key for BIND (DNS). Sep 7 14:11:06 myserver audit: SERVICE_START pid=1 uid=0 auid=4294967295 ses=4294967295 msg='unit=named-setup-rndc comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success' Sep 7 14:11:06 myserver audit: SERVICE_STOP pid=1 uid=0 auid=4294967295 ses=4294967295 msg='unit=named-setup-rndc comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success' Sep 7 14:11:06 myserver systemd: Starting Berkeley Internet Name Domain (DNS) with native PKCS#11... Sep 7 14:11:06 myserver bash: zone localhost.localdomain/IN: loaded serial 0 Sep 7 14:11:06 myserver bash: zone localhost/IN: loaded serial 0 Sep 7 14:11:06 myserver bash: zone 1.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.ip6.arpa/IN: loaded serial 0 Sep 7 14:11:06 myserver bash: zone 1.0.0.127.in-addr.arpa/IN: loaded serial 0 Sep 7 14:11:06 myserver bash: zone 0.in-addr.arpa/IN: loaded serial 0 Sep 7 14:11:06 myserver named-pkcs11[9425]: starting BIND 9.10.4-P2-RedHat-9.10.4-1.P2.fc24 -u named Sep 7 14:11:06 myserver named-pkcs11[9425]: running on Linux x86_64 4.7.2-201.fc24.x86_64 #1 SMP Fri Aug 26 15:58:40 UTC 2016 Sep 7 14:11:06 myserver named-pkcs11[9425]: built with '--build=x86_64-redhat-linux-gnu' '--host=x86_64-redhat-linux-gnu' '--program-prefix=' '--disable-dependency-tracking' '--prefix=/usr' '--exec-prefix=/usr' '--bindir=/usr/bin' '--sbindir=/usr/sbin' '--sysconfdir=/etc' '--datadir=/usr/share' '--includedir=/usr/include' '--libdir=/usr/lib64' '--libexecdir=/usr/libexec' '--sharedstatedir=/var/lib' '--mandir=/usr/share/man' '--infodir=/usr/share/info' '--with-python=/usr/bin/python3' '--with-libtool' '--localstatedir=/var' '--enable-threads' '--enable-ipv6' '--enable-filter-aaaa' '--with-pic' '--disable-static' '--disable-openssl-version-check' '--includedir=/usr/include/bind9' '--with-tuning=large' '--with-geoip' '--enable-native-pkcs11' '--with-pkcs11=/usr/lib64/pkcs11/libsofthsm2.so' '--with-dlopen=yes' '--with-dlz-ldap=yes' '--with-dlz-postgres=yes' '--with-dlz-mysql=yes' '--with-dlz-filesystem=yes' '--with-dlz-bdb=yes' '--with-gssapi=yes' '--disable-isc-spnego' '--enable-fixed-rrset' '--with-docbook-xsl=/usr/share/sgml/docbook/xsl-stylesheets' '--enable-full-report' 'build_alias=x86_64-redhat-linux-gnu' 'host_alias=x86_64-redhat-linux-gnu' 'CFLAGS= -O2 -g -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong --param=ssp-buffer-size=4 -grecord-gcc-switches -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -m64 -mtune=generic' 'LDFLAGS=-Wl,-z,relro -specs=/usr/lib/rpm/redhat/redhat-hardened-ld' 'CPPFLAGS= -DDIG_SIGCHASE' Sep 7 14:11:06 myserver named-pkcs11[9425]: ---------------------------------------------------- Sep 7 14:11:06 myserver named-pkcs11[9425]: BIND 9 is maintained by Internet Systems Consortium, Sep 7 14:11:06 myserver named-pkcs11[9425]: Inc. (ISC), a non-profit 501(c)(3) public-benefit Sep 7 14:11:06 myserver named-pkcs11[9425]: corporation. Support and training for BIND 9 are Sep 7 14:11:06 myserver named-pkcs11[9425]: available at https://www.isc.org/support Sep 7 14:11:06 myserver named-pkcs11[9425]: ---------------------------------------------------- Sep 7 14:11:06 myserver named-pkcs11[9425]: adjusted limit on open files from 4096 to 1048576 Sep 7 14:11:06 myserver named-pkcs11[9425]: found 12 CPUs, using 12 worker threads Sep 7 14:11:06 myserver named-pkcs11[9425]: using 11 UDP listeners per interface Sep 7 14:11:06 myserver named-pkcs11[9425]: using up to 21000 sockets Sep 7 14:11:06 myserver named-pkcs11[9425]: loading configuration from '/etc/named.conf' Sep 7 14:11:06 myserver named-pkcs11[9425]: reading built-in trusted keys from file '/etc/named.iscdlv.key' Sep 7 14:11:06 myserver named-pkcs11[9425]: initializing GeoIP Country (IPv4) (type 1) DB Sep 7 14:11:06 myserver named-pkcs11[9425]: GEO-106FREE 20160705 Build 1 Copy Sep 7 14:11:06 myserver named-pkcs11[9425]: GeoIP Country (IPv6) (type 12) DB not available Sep 7 14:11:06 myserver named-pkcs11[9425]: GeoIP City (IPv4) (type 2) DB not available Sep 7 14:11:06 myserver named-pkcs11[9425]: GeoIP City (IPv4) (type 6) DB not available Sep 7 14:11:06 myserver named-pkcs11[9425]: GeoIP City (IPv6) (type 30) DB not available Sep 7 14:11:06 myserver named-pkcs11[9425]: GeoIP City (IPv6) (type 31) DB not available Sep 7 14:11:06 myserver named-pkcs11[9425]: GeoIP Region (type 3) DB not available Sep 7 14:11:06 myserver named-pkcs11[9425]: GeoIP Region (type 7) DB not available Sep 7 14:11:06 myserver named-pkcs11[9425]: GeoIP ISP (type 4) DB not available Sep 7 14:11:06 myserver named-pkcs11[9425]: GeoIP Org (type 5) DB not available Sep 7 14:11:06 myserver named-pkcs11[9425]: GeoIP AS (type 9) DB not available Sep 7 14:11:06 myserver named-pkcs11[9425]: GeoIP Domain (type 11) DB not available Sep 7 14:11:06 myserver named-pkcs11[9425]: GeoIP NetSpeed (type 10) DB not available Sep 7 14:11:06 myserver named-pkcs11[9425]: using default UDP/IPv4 port range: [32768, 60999] Sep 7 14:11:06 myserver named-pkcs11[9425]: using default UDP/IPv6 port range: [32768, 60999] Sep 7 14:11:06 myserver named-pkcs11[9425]: listening on IPv6 interfaces, port 53 Sep 7 14:11:06 myserver named-pkcs11[9425]: listening on IPv4 interface lo, 127.0.0.1#53 Sep 7 14:11:06 myserver named-pkcs11[9425]: listening on IPv4 interface eno1, 10.0.21.18#53 Sep 7 14:11:06 myserver named-pkcs11[9425]: generating session key for dynamic DNS Sep 7 14:11:06 myserver named-pkcs11[9425]: sizing zone task pool based on 6 zones Sep 7 14:11:07 myserver named-pkcs11[9425]: set up managed keys zone for view _default, file '/var/named/dynamic/managed-keys.bind' Sep 7 14:11:07 myserver named-pkcs11[9425]: bind-dyndb-ldap version 9.0 compiled at 09:01:48 May 27 2016, compiler 6.1.1 20160510 (Red Hat 6.1.1-2) Sep 7 14:11:07 myserver named-pkcs11[9425]: option 'serial_autoincrement' is not supported, ignoring Sep 7 14:11:07 myserver named-pkcs11[9425]: LDAP instance 'ipa' is being synchronized, please ignore message 'all zones loaded' Sep 7 14:11:07 myserver named-pkcs11[9425]: automatic empty zone: 10.IN-ADDR.ARPA Sep 7 14:11:07 myserver named-pkcs11[9425]: automatic empty zone: 16.172.IN-ADDR.ARPA Sep 7 14:11:07 myserver named-pkcs11[9425]: automatic empty zone: 17.172.IN-ADDR.ARPA Sep 7 14:11:07 myserver named-pkcs11[9425]: automatic empty zone: 18.172.IN-ADDR.ARPA Sep 7 14:11:07 myserver named-pkcs11[9425]: automatic empty zone: 19.172.IN-ADDR.ARPA Sep 7 14:11:07 myserver named-pkcs11[9425]: automatic empty zone: 20.172.IN-ADDR.ARPA Sep 7 14:11:07 myserver named-pkcs11[9425]: automatic empty zone: 21.172.IN-ADDR.ARPA Sep 7 14:11:07 myserver named-pkcs11[9425]: automatic empty zone: 22.172.IN-ADDR.ARPA Sep 7 14:11:07 myserver named-pkcs11[9425]: automatic empty zone: 23.172.IN-ADDR.ARPA Sep 7 14:11:07 myserver named-pkcs11[9425]: automatic empty zone: 24.172.IN-ADDR.ARPA Sep 7 14:11:07 myserver named-pkcs11[9425]: automatic empty zone: 25.172.IN-ADDR.ARPA Sep 7 14:11:07 myserver named-pkcs11[9425]: automatic empty zone: 26.172.IN-ADDR.ARPA Sep 7 14:11:07 myserver named-pkcs11[9425]: automatic empty zone: 27.172.IN-ADDR.ARPA Sep 7 14:11:07 myserver named-pkcs11[9425]: automatic empty zone: 28.172.IN-ADDR.ARPA Sep 7 14:11:07 myserver named-pkcs11[9425]: automatic empty zone: 29.172.IN-ADDR.ARPA Sep 7 14:11:07 myserver named-pkcs11[9425]: automatic empty zone: 30.172.IN-ADDR.ARPA Sep 7 14:11:07 myserver named-pkcs11[9425]: automatic empty zone: 31.172.IN-ADDR.ARPA Sep 7 14:11:07 myserver named-pkcs11[9425]: automatic empty zone: 168.192.IN-ADDR.ARPA Sep 7 14:11:07 myserver named-pkcs11[9425]: automatic empty zone: 64.100.IN-ADDR.ARPA Sep 7 14:11:07 myserver named-pkcs11[9425]: automatic empty zone: 65.100.IN-ADDR.ARPA Sep 7 14:11:07 myserver named-pkcs11[9425]: automatic empty zone: 66.100.IN-ADDR.ARPA Sep 7 14:11:07 myserver named-pkcs11[9425]: automatic empty zone: 67.100.IN-ADDR.ARPA Sep 7 14:11:07 myserver named-pkcs11[9425]: automatic empty zone: 68.100.IN-ADDR.ARPA Sep 7 14:11:07 myserver named-pkcs11[9425]: automatic empty zone: 69.100.IN-ADDR.ARPA Sep 7 14:11:07 myserver named-pkcs11[9425]: automatic empty zone: 70.100.IN-ADDR.ARPA Sep 7 14:11:07 myserver named-pkcs11[9425]: automatic empty zone: 71.100.IN-ADDR.ARPA Sep 7 14:11:07 myserver named-pkcs11[9425]: automatic empty zone: 72.100.IN-ADDR.ARPA Sep 7 14:11:07 myserver named-pkcs11[9425]: automatic empty zone: 73.100.IN-ADDR.ARPA Sep 7 14:11:07 myserver named-pkcs11[9425]: automatic empty zone: 74.100.IN-ADDR.ARPA Sep 7 14:11:07 myserver named-pkcs11[9425]: automatic empty zone: 75.100.IN-ADDR.ARPA Sep 7 14:11:07 myserver named-pkcs11[9425]: automatic empty zone: 76.100.IN-ADDR.ARPA Sep 7 14:11:07 myserver named-pkcs11[9425]: automatic empty zone: 77.100.IN-ADDR.ARPA Sep 7 14:11:07 myserver named-pkcs11[9425]: automatic empty zone: 78.100.IN-ADDR.ARPA Sep 7 14:11:07 myserver named-pkcs11[9425]: automatic empty zone: 79.100.IN-ADDR.ARPA Sep 7 14:11:07 myserver named-pkcs11[9425]: automatic empty zone: 80.100.IN-ADDR.ARPA Sep 7 14:11:07 myserver named-pkcs11[9425]: automatic empty zone: 81.100.IN-ADDR.ARPA Sep 7 14:11:07 myserver named-pkcs11[9425]: automatic empty zone: 82.100.IN-ADDR.ARPA Sep 7 14:11:07 myserver named-pkcs11[9425]: automatic empty zone: 83.100.IN-ADDR.ARPA Sep 7 14:11:07 myserver named-pkcs11[9425]: automatic empty zone: 84.100.IN-ADDR.ARPA Sep 7 14:11:07 myserver named-pkcs11[9425]: automatic empty zone: 85.100.IN-ADDR.ARPA Sep 7 14:11:07 myserver named-pkcs11[9425]: automatic empty zone: 86.100.IN-ADDR.ARPA Sep 7 14:11:07 myserver named-pkcs11[9425]: automatic empty zone: 87.100.IN-ADDR.ARPA Sep 7 14:11:07 myserver named-pkcs11[9425]: automatic empty zone: 88.100.IN-ADDR.ARPA Sep 7 14:11:07 myserver named-pkcs11[9425]: automatic empty zone: 89.100.IN-ADDR.ARPA Sep 7 14:11:07 myserver named-pkcs11[9425]: automatic empty zone: 90.100.IN-ADDR.ARPA Sep 7 14:11:07 myserver named-pkcs11[9425]: automatic empty zone: 91.100.IN-ADDR.ARPA Sep 7 14:11:07 myserver named-pkcs11[9425]: automatic empty zone: 92.100.IN-ADDR.ARPA Sep 7 14:11:07 myserver named-pkcs11[9425]: automatic empty zone: 93.100.IN-ADDR.ARPA Sep 7 14:11:07 myserver named-pkcs11[9425]: automatic empty zone: 94.100.IN-ADDR.ARPA Sep 7 14:11:07 myserver named-pkcs11[9425]: automatic empty zone: 95.100.IN-ADDR.ARPA Sep 7 14:11:07 myserver named-pkcs11[9425]: automatic empty zone: 96.100.IN-ADDR.ARPA Sep 7 14:11:07 myserver named-pkcs11[9425]: automatic empty zone: 97.100.IN-ADDR.ARPA Sep 7 14:11:07 myserver named-pkcs11[9425]: automatic empty zone: 98.100.IN-ADDR.ARPA Sep 7 14:11:07 myserver named-pkcs11[9425]: automatic empty zone: 99.100.IN-ADDR.ARPA Sep 7 14:11:07 myserver named-pkcs11[9425]: automatic empty zone: 100.100.IN-ADDR.ARPA Sep 7 14:11:07 myserver named-pkcs11[9425]: automatic empty zone: 101.100.IN-ADDR.ARPA Sep 7 14:11:07 myserver named-pkcs11[9425]: automatic empty zone: 102.100.IN-ADDR.ARPA Sep 7 14:11:07 myserver named-pkcs11[9425]: automatic empty zone: 103.100.IN-ADDR.ARPA Sep 7 14:11:07 myserver named-pkcs11[9425]: automatic empty zone: 104.100.IN-ADDR.ARPA Sep 7 14:11:07 myserver named-pkcs11[9425]: automatic empty zone: 105.100.IN-ADDR.ARPA Sep 7 14:11:07 myserver named-pkcs11[9425]: automatic empty zone: 106.100.IN-ADDR.ARPA Sep 7 14:11:07 myserver named-pkcs11[9425]: automatic empty zone: 107.100.IN-ADDR.ARPA Sep 7 14:11:07 myserver named-pkcs11[9425]: automatic empty zone: 108.100.IN-ADDR.ARPA Sep 7 14:11:07 myserver named-pkcs11[9425]: automatic empty zone: 109.100.IN-ADDR.ARPA Sep 7 14:11:07 myserver named-pkcs11[9425]: automatic empty zone: 110.100.IN-ADDR.ARPA Sep 7 14:11:07 myserver named-pkcs11[9425]: automatic empty zone: 111.100.IN-ADDR.ARPA Sep 7 14:11:07 myserver named-pkcs11[9425]: automatic empty zone: 112.100.IN-ADDR.ARPA Sep 7 14:11:07 myserver named-pkcs11[9425]: automatic empty zone: 113.100.IN-ADDR.ARPA Sep 7 14:11:07 myserver named-pkcs11[9425]: automatic empty zone: 114.100.IN-ADDR.ARPA Sep 7 14:11:07 myserver named-pkcs11[9425]: automatic empty zone: 115.100.IN-ADDR.ARPA Sep 7 14:11:07 myserver named-pkcs11[9425]: automatic empty zone: 116.100.IN-ADDR.ARPA Sep 7 14:11:07 myserver named-pkcs11[9425]: automatic empty zone: 117.100.IN-ADDR.ARPA Sep 7 14:11:07 myserver named-pkcs11[9425]: automatic empty zone: 118.100.IN-ADDR.ARPA Sep 7 14:11:07 myserver named-pkcs11[9425]: automatic empty zone: 119.100.IN-ADDR.ARPA Sep 7 14:11:07 myserver named-pkcs11[9425]: automatic empty zone: 120.100.IN-ADDR.ARPA Sep 7 14:11:07 myserver named-pkcs11[9425]: automatic empty zone: 121.100.IN-ADDR.ARPA Sep 7 14:11:07 myserver named-pkcs11[9425]: automatic empty zone: 122.100.IN-ADDR.ARPA Sep 7 14:11:07 myserver named-pkcs11[9425]: automatic empty zone: 123.100.IN-ADDR.ARPA Sep 7 14:11:07 myserver named-pkcs11[9425]: automatic empty zone: 124.100.IN-ADDR.ARPA Sep 7 14:11:07 myserver named-pkcs11[9425]: automatic empty zone: 125.100.IN-ADDR.ARPA Sep 7 14:11:07 myserver named-pkcs11[9425]: automatic empty zone: 126.100.IN-ADDR.ARPA Sep 7 14:11:07 myserver named-pkcs11[9425]: automatic empty zone: 127.100.IN-ADDR.ARPA Sep 7 14:11:07 myserver named-pkcs11[9425]: automatic empty zone: 127.IN-ADDR.ARPA Sep 7 14:11:07 myserver named-pkcs11[9425]: automatic empty zone: 254.169.IN-ADDR.ARPA Sep 7 14:11:07 myserver named-pkcs11[9425]: automatic empty zone: 2.0.192.IN-ADDR.ARPA Sep 7 14:11:07 myserver named-pkcs11[9425]: automatic empty zone: 100.51.198.IN-ADDR.ARPA Sep 7 14:11:07 myserver named-pkcs11[9425]: automatic empty zone: 113.0.203.IN-ADDR.ARPA Sep 7 14:11:07 myserver named-pkcs11[9425]: automatic empty zone: 255.255.255.255.IN-ADDR.ARPA Sep 7 14:11:07 myserver named-pkcs11[9425]: automatic empty zone: 0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.IP6.ARPA Sep 7 14:11:07 myserver named-pkcs11[9425]: automatic empty zone: D.F.IP6.ARPA Sep 7 14:11:07 myserver named-pkcs11[9425]: automatic empty zone: 8.E.F.IP6.ARPA Sep 7 14:11:07 myserver named-pkcs11[9425]: automatic empty zone: 9.E.F.IP6.ARPA Sep 7 14:11:07 myserver named-pkcs11[9425]: automatic empty zone: A.E.F.IP6.ARPA Sep 7 14:11:07 myserver named-pkcs11[9425]: automatic empty zone: B.E.F.IP6.ARPA Sep 7 14:11:07 myserver named-pkcs11[9425]: automatic empty zone: 8.B.D.0.1.0.0.2.IP6.ARPA Sep 7 14:11:07 myserver named-pkcs11[9425]: automatic empty zone: EMPTY.AS112.ARPA Sep 7 14:11:07 myserver named-pkcs11[9425]: configuring command channel from '/etc/rndc.key' Sep 7 14:11:07 myserver named-pkcs11[9425]: command channel listening on 127.0.0.1#953 Sep 7 14:11:07 myserver named-pkcs11[9425]: configuring command channel from '/etc/rndc.key' Sep 7 14:11:07 myserver named-pkcs11[9425]: command channel listening on ::1#953 Sep 7 14:11:07 myserver named-pkcs11[9425]: managed-keys-zone: journal file is out of date: removing journal file Sep 7 14:11:07 myserver named-pkcs11[9425]: managed-keys-zone: loaded serial 473 Sep 7 14:11:07 myserver named-pkcs11[9425]: ignoring inherited 'forward first;' for zone '.' - did you want 'forward only;' to override automatic empty zone '10.IN-ADDR.ARPA'? Sep 7 14:11:07 myserver named-pkcs11[9425]: ignoring inherited 'forward first;' for zone '.' - did you want 'forward only;' to override automatic empty zone '16.172.IN-ADDR.ARPA'? Sep 7 14:11:07 myserver named-pkcs11[9425]: ignoring inherited 'forward first;' for zone '.' - did you want 'forward only;' to override automatic empty zone '17.172.IN-ADDR.ARPA'? Sep 7 14:11:07 myserver named-pkcs11[9425]: ignoring inherited 'forward first;' for zone '.' - did you want 'forward only;' to override automatic empty zone '18.172.IN-ADDR.ARPA'? Sep 7 14:11:07 myserver named-pkcs11[9425]: ignoring inherited 'forward first;' for zone '.' - did you want 'forward only;' to override automatic empty zone '19.172.IN-ADDR.ARPA'? Sep 7 14:11:07 myserver named-pkcs11[9425]: ignoring inherited 'forward first;' for zone '.' - did you want 'forward only;' to override automatic empty zone '20.172.IN-ADDR.ARPA'? Sep 7 14:11:07 myserver named-pkcs11[9425]: ignoring inherited 'forward first;' for zone '.' - did you want 'forward only;' to override automatic empty zone '21.172.IN-ADDR.ARPA'? Sep 7 14:11:07 myserver named-pkcs11[9425]: ignoring inherited 'forward first;' for zone '.' - did you want 'forward only;' to override automatic empty zone '22.172.IN-ADDR.ARPA'? Sep 7 14:11:07 myserver named-pkcs11[9425]: ignoring inherited 'forward first;' for zone '.' - did you want 'forward only;' to override automatic empty zone '23.172.IN-ADDR.ARPA'? Sep 7 14:11:07 myserver named-pkcs11[9425]: ignoring inherited 'forward first;' for zone '.' - did you want 'forward only;' to override automatic empty zone '24.172.IN-ADDR.ARPA'? Sep 7 14:11:07 myserver named-pkcs11[9425]: ignoring inherited 'forward first;' for zone '.' - did you want 'forward only;' to override automatic empty zone '25.172.IN-ADDR.ARPA'? Sep 7 14:11:07 myserver named-pkcs11[9425]: ignoring inherited 'forward first;' for zone '.' - did you want 'forward only;' to override automatic empty zone '26.172.IN-ADDR.ARPA'? Sep 7 14:11:07 myserver named-pkcs11[9425]: ignoring inherited 'forward first;' for zone '.' - did you want 'forward only;' to override automatic empty zone '27.172.IN-ADDR.ARPA'? Sep 7 14:11:07 myserver named-pkcs11[9425]: ignoring inherited 'forward first;' for zone '.' - did you want 'forward only;' to override automatic empty zone '28.172.IN-ADDR.ARPA'? Sep 7 14:11:07 myserver named-pkcs11[9425]: ignoring inherited 'forward first;' for zone '.' - did you want 'forward only;' to override automatic empty zone '29.172.IN-ADDR.ARPA'? Sep 7 14:11:07 myserver named-pkcs11[9425]: ignoring inherited 'forward first;' for zone '.' - did you want 'forward only;' to override automatic empty zone '30.172.IN-ADDR.ARPA'? Sep 7 14:11:07 myserver named-pkcs11[9425]: zone 0.in-addr.arpa/IN: loaded serial 0 Sep 7 14:11:07 myserver named-pkcs11[9425]: ignoring inherited 'forward first;' for zone '.' - did you want 'forward only;' to override automatic empty zone '31.172.IN-ADDR.ARPA'? Sep 7 14:11:07 myserver named-pkcs11[9425]: ignoring inherited 'forward first;' for zone '.' - did you want 'forward only;' to override automatic empty zone '168.192.IN-ADDR.ARPA'? Sep 7 14:11:07 myserver named-pkcs11[9425]: ignoring inherited 'forward first;' for zone '.' - did you want 'forward only;' to override automatic empty zone '64.100.IN-ADDR.ARPA'? Sep 7 14:11:07 myserver named-pkcs11[9425]: ignoring inherited 'forward first;' for zone '.' - did you want 'forward only;' to override automatic empty zone '65.100.IN-ADDR.ARPA'? Sep 7 14:11:07 myserver named-pkcs11[9425]: ignoring inherited 'forward first;' for zone '.' - did you want 'forward only;' to override automatic empty zone '66.100.IN-ADDR.ARPA'? Sep 7 14:11:07 myserver named-pkcs11[9425]: ignoring inherited 'forward first;' for zone '.' - did you want 'forward only;' to override automatic empty zone '67.100.IN-ADDR.ARPA'? Sep 7 14:11:07 myserver named-pkcs11[9425]: ignoring inherited 'forward first;' for zone '.' - did you want 'forward only;' to override automatic empty zone '68.100.IN-ADDR.ARPA'? Sep 7 14:11:07 myserver named-pkcs11[9425]: ignoring inherited 'forward first;' for zone '.' - did you want 'forward only;' to override automatic empty zone '69.100.IN-ADDR.ARPA'? Sep 7 14:11:07 myserver named-pkcs11[9425]: ignoring inherited 'forward first;' for zone '.' - did you want 'forward only;' to override automatic empty zone '70.100.IN-ADDR.ARPA'? Sep 7 14:11:07 myserver named-pkcs11[9425]: ignoring inherited 'forward first;' for zone '.' - did you want 'forward only;' to override automatic empty zone '71.100.IN-ADDR.ARPA'? Sep 7 14:11:07 myserver named-pkcs11[9425]: ignoring inherited 'forward first;' for zone '.' - did you want 'forward only;' to override automatic empty zone '72.100.IN-ADDR.ARPA'? Sep 7 14:11:07 myserver named-pkcs11[9425]: ignoring inherited 'forward first;' for zone '.' - did you want 'forward only;' to override automatic empty zone '73.100.IN-ADDR.ARPA'? Sep 7 14:11:07 myserver named-pkcs11[9425]: ignoring inherited 'forward first;' for zone '.' - did you want 'forward only;' to override automatic empty zone '74.100.IN-ADDR.ARPA'? Sep 7 14:11:07 myserver named-pkcs11[9425]: ignoring inherited 'forward first;' for zone '.' - did you want 'forward only;' to override automatic empty zone '75.100.IN-ADDR.ARPA'? Sep 7 14:11:07 myserver named-pkcs11[9425]: ignoring inherited 'forward first;' for zone '.' - did you want 'forward only;' to override automatic empty zone '76.100.IN-ADDR.ARPA'? Sep 7 14:11:07 myserver named-pkcs11[9425]: ignoring inherited 'forward first;' for zone '.' - did you want 'forward only;' to override automatic empty zone '77.100.IN-ADDR.ARPA'? Sep 7 14:11:07 myserver named-pkcs11[9425]: ignoring inherited 'forward first;' for zone '.' - did you want 'forward only;' to override automatic empty zone '78.100.IN-ADDR.ARPA'? Sep 7 14:11:07 myserver named-pkcs11[9425]: ignoring inherited 'forward first;' for zone '.' - did you want 'forward only;' to override automatic empty zone '79.100.IN-ADDR.ARPA'? Sep 7 14:11:07 myserver named-pkcs11[9425]: ignoring inherited 'forward first;' for zone '.' - did you want 'forward only;' to override automatic empty zone '80.100.IN-ADDR.ARPA'? Sep 7 14:11:07 myserver named-pkcs11[9425]: ignoring inherited 'forward first;' for zone '.' - did you want 'forward only;' to override automatic empty zone '81.100.IN-ADDR.ARPA'? Sep 7 14:11:07 myserver named-pkcs11[9425]: ignoring inherited 'forward first;' for zone '.' - did you want 'forward only;' to override automatic empty zone '82.100.IN-ADDR.ARPA'? Sep 7 14:11:07 myserver named-pkcs11[9425]: ignoring inherited 'forward first;' for zone '.' - did you want 'forward only;' to override automatic empty zone '83.100.IN-ADDR.ARPA'? Sep 7 14:11:07 myserver named-pkcs11[9425]: ignoring inherited 'forward first;' for zone '.' - did you want 'forward only;' to override automatic empty zone '84.100.IN-ADDR.ARPA'? Sep 7 14:11:07 myserver named-pkcs11[9425]: ignoring inherited 'forward first;' for zone '.' - did you want 'forward only;' to override automatic empty zone '85.100.IN-ADDR.ARPA'? Sep 7 14:11:07 myserver named-pkcs11[9425]: ignoring inherited 'forward first;' for zone '.' - did you want 'forward only;' to override automatic empty zone '86.100.IN-ADDR.ARPA'? Sep 7 14:11:07 myserver named-pkcs11[9425]: ignoring inherited 'forward first;' for zone '.' - did you want 'forward only;' to override automatic empty zone '87.100.IN-ADDR.ARPA'? Sep 7 14:11:07 myserver named-pkcs11[9425]: ignoring inherited 'forward first;' for zone '.' - did you want 'forward only;' to override automatic empty zone '88.100.IN-ADDR.ARPA'? Sep 7 14:11:07 myserver named-pkcs11[9425]: ignoring inherited 'forward first;' for zone '.' - did you want 'forward only;' to override automatic empty zone '89.100.IN-ADDR.ARPA'? Sep 7 14:11:07 myserver named-pkcs11[9425]: ignoring inherited 'forward first;' for zone '.' - did you want 'forward only;' to override automatic empty zone '90.100.IN-ADDR.ARPA'? Sep 7 14:11:07 myserver named-pkcs11[9425]: ignoring inherited 'forward first;' for zone '.' - did you want 'forward only;' to override automatic empty zone '91.100.IN-ADDR.ARPA'? Sep 7 14:11:07 myserver named-pkcs11[9425]: ignoring inherited 'forward first;' for zone '.' - did you want 'forward only;' to override automatic empty zone '92.100.IN-ADDR.ARPA'? Sep 7 14:11:07 myserver named-pkcs11[9425]: ignoring inherited 'forward first;' for zone '.' - did you want 'forward only;' to override automatic empty zone '93.100.IN-ADDR.ARPA'? Sep 7 14:11:07 myserver named-pkcs11[9425]: ignoring inherited 'forward first;' for zone '.' - did you want 'forward only;' to override automatic empty zone '94.100.IN-ADDR.ARPA'? Sep 7 14:11:07 myserver named-pkcs11[9425]: ignoring inherited 'forward first;' for zone '.' - did you want 'forward only;' to override automatic empty zone '95.100.IN-ADDR.ARPA'? Sep 7 14:11:07 myserver named-pkcs11[9425]: ignoring inherited 'forward first;' for zone '.' - did you want 'forward only;' to override automatic empty zone '96.100.IN-ADDR.ARPA'? Sep 7 14:11:07 myserver named-pkcs11[9425]: ignoring inherited 'forward first;' for zone '.' - did you want 'forward only;' to override automatic empty zone '97.100.IN-ADDR.ARPA'? Sep 7 14:11:07 myserver named-pkcs11[9425]: ignoring inherited 'forward first;' for zone '.' - did you want 'forward only;' to override automatic empty zone '98.100.IN-ADDR.ARPA'? Sep 7 14:11:07 myserver named-pkcs11[9425]: ignoring inherited 'forward first;' for zone '.' - did you want 'forward only;' to override automatic empty zone '99.100.IN-ADDR.ARPA'? Sep 7 14:11:07 myserver named-pkcs11[9425]: ignoring inherited 'forward first;' for zone '.' - did you want 'forward only;' to override automatic empty zone '100.100.IN-ADDR.ARPA'? Sep 7 14:11:07 myserver named-pkcs11[9425]: ignoring inherited 'forward first;' for zone '.' - did you want 'forward only;' to override automatic empty zone '101.100.IN-ADDR.ARPA'? Sep 7 14:11:07 myserver named-pkcs11[9425]: ignoring inherited 'forward first;' for zone '.' - did you want 'forward only;' to override automatic empty zone '102.100.IN-ADDR.ARPA'? Sep 7 14:11:07 myserver named-pkcs11[9425]: ignoring inherited 'forward first;' for zone '.' - did you want 'forward only;' to override automatic empty zone '103.100.IN-ADDR.ARPA'? Sep 7 14:11:07 myserver named-pkcs11[9425]: ignoring inherited 'forward first;' for zone '.' - did you want 'forward only;' to override automatic empty zone '104.100.IN-ADDR.ARPA'? Sep 7 14:11:07 myserver named-pkcs11[9425]: ignoring inherited 'forward first;' for zone '.' - did you want 'forward only;' to override automatic empty zone '105.100.IN-ADDR.ARPA'? Sep 7 14:11:07 myserver named-pkcs11[9425]: ignoring inherited 'forward first;' for zone '.' - did you want 'forward only;' to override automatic empty zone '106.100.IN-ADDR.ARPA'? Sep 7 14:11:07 myserver named-pkcs11[9425]: ignoring inherited 'forward first;' for zone '.' - did you want 'forward only;' to override automatic empty zone '107.100.IN-ADDR.ARPA'? Sep 7 14:11:07 myserver named-pkcs11[9425]: ignoring inherited 'forward first;' for zone '.' - did you want 'forward only;' to override automatic empty zone '108.100.IN-ADDR.ARPA'? Sep 7 14:11:07 myserver named-pkcs11[9425]: ignoring inherited 'forward first;' for zone '.' - did you want 'forward only;' to override automatic empty zone '109.100.IN-ADDR.ARPA'? Sep 7 14:11:07 myserver named-pkcs11[9425]: zone localhost.localdomain/IN: loaded serial 0 Sep 7 14:11:07 myserver named-pkcs11[9425]: ignoring inherited 'forward first;' for zone '.' - did you want 'forward only;' to override automatic empty zone '110.100.IN-ADDR.ARPA'? Sep 7 14:11:07 myserver named-pkcs11[9425]: ignoring inherited 'forward first;' for zone '.' - did you want 'forward only;' to override automatic empty zone '111.100.IN-ADDR.ARPA'? Sep 7 14:11:07 myserver named-pkcs11[9425]: ignoring inherited 'forward first;' for zone '.' - did you want 'forward only;' to override automatic empty zone '112.100.IN-ADDR.ARPA'? Sep 7 14:11:07 myserver named-pkcs11[9425]: ignoring inherited 'forward first;' for zone '.' - did you want 'forward only;' to override automatic empty zone '113.100.IN-ADDR.ARPA'? Sep 7 14:11:07 myserver named-pkcs11[9425]: ignoring inherited 'forward first;' for zone '.' - did you want 'forward only;' to override automatic empty zone '114.100.IN-ADDR.ARPA'? Sep 7 14:11:07 myserver named-pkcs11[9425]: ignoring inherited 'forward first;' for zone '.' - did you want 'forward only;' to override automatic empty zone '115.100.IN-ADDR.ARPA'? Sep 7 14:11:07 myserver systemd: Started Berkeley Internet Name Domain (DNS) with native PKCS#11. Sep 7 14:11:07 myserver audit: SERVICE_START pid=1 uid=0 auid=4294967295 ses=4294967295 msg='unit=named-pkcs11 comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success' Sep 7 14:11:07 myserver named-pkcs11[9425]: ignoring inherited 'forward first;' for zone '.' - did you want 'forward only;' to override automatic empty zone '116.100.IN-ADDR.ARPA'? Sep 7 14:11:07 myserver named-pkcs11[9425]: ignoring inherited 'forward first;' for zone '.' - did you want 'forward only;' to override automatic empty zone '117.100.IN-ADDR.ARPA'? Sep 7 14:11:07 myserver named-pkcs11[9425]: ignoring inherited 'forward first;' for zone '.' - did you want 'forward only;' to override automatic empty zone '118.100.IN-ADDR.ARPA'? Sep 7 14:11:07 myserver named-pkcs11[9425]: ignoring inherited 'forward first;' for zone '.' - did you want 'forward only;' to override automatic empty zone '119.100.IN-ADDR.ARPA'? Sep 7 14:11:07 myserver named-pkcs11[9425]: ignoring inherited 'forward first;' for zone '.' - did you want 'forward only;' to override automatic empty zone '120.100.IN-ADDR.ARPA'? Sep 7 14:11:07 myserver named-pkcs11[9425]: ignoring inherited 'forward first;' for zone '.' - did you want 'forward only;' to override automatic empty zone '121.100.IN-ADDR.ARPA'? Sep 7 14:11:07 myserver named-pkcs11[9425]: ignoring inherited 'forward first;' for zone '.' - did you want 'forward only;' to override automatic empty zone '122.100.IN-ADDR.ARPA'? Sep 7 14:11:07 myserver named-pkcs11[9425]: ignoring inherited 'forward first;' for zone '.' - did you want 'forward only;' to override automatic empty zone '123.100.IN-ADDR.ARPA'? Sep 7 14:11:07 myserver named-pkcs11[9425]: ignoring inherited 'forward first;' for zone '.' - did you want 'forward only;' to override automatic empty zone '124.100.IN-ADDR.ARPA'? Sep 7 14:11:07 myserver named-pkcs11[9425]: ignoring inherited 'forward first;' for zone '.' - did you want 'forward only;' to override automatic empty zone '125.100.IN-ADDR.ARPA'? Sep 7 14:11:07 myserver named-pkcs11[9425]: ignoring inherited 'forward first;' for zone '.' - did you want 'forward only;' to override automatic empty zone '126.100.IN-ADDR.ARPA'? Sep 7 14:11:07 myserver named-pkcs11[9425]: ignoring inherited 'forward first;' for zone '.' - did you want 'forward only;' to override automatic empty zone '127.100.IN-ADDR.ARPA'? Sep 7 14:11:07 myserver named-pkcs11[9425]: ignoring inherited 'forward first;' for zone '.' - did you want 'forward only;' to override automatic empty zone '254.169.IN-ADDR.ARPA'? Sep 7 14:11:07 myserver named-pkcs11[9425]: ignoring inherited 'forward first;' for zone '.' - did you want 'forward only;' to override automatic empty zone '2.0.192.IN-ADDR.ARPA'? Sep 7 14:11:07 myserver named-pkcs11[9425]: zone 1.0.0.127.in-addr.arpa/IN: loaded serial 0 Sep 7 14:11:07 myserver named-pkcs11[9425]: ignoring inherited 'forward first;' for zone '.' - did you want 'forward only;' to override automatic empty zone '100.51.198.IN-ADDR.ARPA'? Sep 7 14:11:07 myserver named-pkcs11[9425]: ignoring inherited 'forward first;' for zone '.' - did you want 'forward only;' to override automatic empty zone '113.0.203.IN-ADDR.ARPA'? Sep 7 14:11:07 myserver named-pkcs11[9425]: ignoring inherited 'forward first;' for zone '.' - did you want 'forward only;' to override automatic empty zone '255.255.255.255.IN-ADDR.ARPA'? Sep 7 14:11:07 myserver named-pkcs11[9425]: ignoring inherited 'forward first;' for zone '.' - did you want 'forward only;' to override automatic empty zone '0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.IP6.ARPA'? Sep 7 14:11:07 myserver named-pkcs11[9425]: ignoring inherited 'forward first;' for zone '.' - did you want 'forward only;' to override automatic empty zone 'D.F.IP6.ARPA'? Sep 7 14:11:07 myserver named-pkcs11[9425]: ignoring inherited 'forward first;' for zone '.' - did you want 'forward only;' to override automatic empty zone '8.E.F.IP6.ARPA'? Sep 7 14:11:07 myserver named-pkcs11[9425]: ignoring inherited 'forward first;' for zone '.' - did you want 'forward only;' to override automatic empty zone '9.E.F.IP6.ARPA'? Sep 7 14:11:07 myserver named-pkcs11[9425]: ignoring inherited 'forward first;' for zone '.' - did you want 'forward only;' to override automatic empty zone 'A.E.F.IP6.ARPA'? Sep 7 14:11:07 myserver named-pkcs11[9425]: ignoring inherited 'forward first;' for zone '.' - did you want 'forward only;' to override automatic empty zone 'B.E.F.IP6.ARPA'? Sep 7 14:11:07 myserver named-pkcs11[9425]: ignoring inherited 'forward first;' for zone '.' - did you want 'forward only;' to override automatic empty zone '8.B.D.0.1.0.0.2.IP6.ARPA'? Sep 7 14:11:07 myserver named-pkcs11[9425]: ignoring inherited 'forward first;' for zone '.' - did you want 'forward only;' to override automatic empty zone 'EMPTY.AS112.ARPA'? Sep 7 14:11:07 myserver named-pkcs11[9425]: zone localhost/IN: loaded serial 0 Sep 7 14:11:07 myserver named-pkcs11[9425]: zone 1.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.ip6.arpa/IN: loaded serial 0 Sep 7 14:11:07 myserver named-pkcs11[9425]: all zones loaded Sep 7 14:11:07 myserver named-pkcs11[9425]: running Sep 7 14:11:07 myserver systemd: Starting IPA memcached daemon, increases IPA server performance... Sep 7 14:11:07 myserver systemd: ipa_memcached.service: PID file /var/run/ipa_memcached/ipa_memcached.pid not readable (yet?) after start: No such file or directory Sep 7 14:11:07 myserver systemd: Started IPA memcached daemon, increases IPA server performance. Sep 7 14:11:07 myserver audit: SERVICE_START pid=1 uid=0 auid=4294967295 ses=4294967295 msg='unit=ipa_memcached comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success' Sep 7 14:11:07 myserver systemd: Starting The Apache HTTP Server... Sep 7 14:11:07 myserver named-pkcs11[9425]: checkhints: unable to get root NS rrset from cache: not found Sep 7 14:11:07 myserver named-pkcs11[9425]: zone 21.0.10.in-addr.arpa/IN: loaded serial 1473250267 Sep 7 14:11:07 myserver named-pkcs11[9425]: zone 21.0.10.in-addr.arpa/IN: sending notifies (serial 1473250267) Sep 7 14:11:07 myserver named-pkcs11[9425]: zone 1.168.192.in-addr.arpa/IN: loaded serial 1473250267 Sep 7 14:11:07 myserver named-pkcs11[9425]: zone 1.168.192.in-addr.arpa/IN: sending notifies (serial 1473250267) Sep 7 14:11:07 myserver named-pkcs11[9425]: checkhints: unable to get root NS rrset from cache: not found Sep 7 14:11:07 myserver named-pkcs11[9425]: zone chloe/IN: loaded serial 1473250267 Sep 7 14:11:07 myserver named-pkcs11[9425]: zone chloe/IN: sending notifies (serial 1473250267) Sep 7 14:11:07 myserver named-pkcs11[9425]: zone mydomain/IN: loaded serial 1473250267 Sep 7 14:11:07 myserver named-pkcs11[9425]: zone mydomain/IN: sending notifies (serial 1473250267) Sep 7 14:11:07 myserver named-pkcs11[9425]: 4 master zones from LDAP instance 'ipa' loaded (4 zones defined, 0 inactive, 0 failed to load) Sep 7 14:11:07 myserver named-pkcs11[9425]: checkhints: unable to get root NS rrset from cache: not found Sep 7 14:11:07 myserver ipa-httpd-kdcproxy: ipa : INFO KDC proxy enabled Sep 7 14:11:07 myserver systemd: Started The Apache HTTP Server. Sep 7 14:11:07 myserver audit: SERVICE_START pid=1 uid=0 auid=4294967295 ses=4294967295 msg='unit=httpd comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success' Sep 7 14:11:07 myserver systemd: Starting PKI Tomcat Server pki-tomcat... Sep 7 14:11:07 myserver systemd: Started PKI Tomcat Server pki-tomcat. Sep 7 14:11:07 myserver audit: SERVICE_START pid=1 uid=0 auid=4294967295 ses=4294967295 msg='unit=pki-tomcatd at pki-tomcat comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success' Sep 7 14:11:07 myserver systemd: Reached target PKI Tomcat Server. Sep 7 14:11:07 myserver server: Java virtual machine used: /usr/share/java-utils/java-wrapper Sep 7 14:11:07 myserver server: classpath used: /usr/share/tomcat/bin/bootstrap.jar:/usr/share/tomcat/bin/tomcat-juli.jar:/usr/lib/java/commons-daemon.jar Sep 7 14:11:07 myserver server: main class used: org.apache.catalina.startup.Bootstrap Sep 7 14:11:07 myserver server: flags used: -DRESTEASY_LIB=/usr/share/java/resteasy -Djava.library.path=/usr/lib64/nuxwdog-jni Sep 7 14:11:07 myserver server: options used: -Dcatalina.base=/var/lib/pki/pki-tomcat -Dcatalina.home=/usr/share/tomcat -Djava.endorsed.dirs= -Djava.io.tmpdir=/var/lib/pki/pki-tomcat/temp -Djava.util.logging.config.file=/var/lib/pki/pki-tomcat/conf/logging.properties -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager -Djava.security.manager -Djava.security.policy==/var/lib/pki/pki-tomcat/conf/catalina.policy Sep 7 14:11:07 myserver server: arguments used: start Sep 7 14:11:08 myserver server: Sep 07, 2016 2:11:08 PM org.apache.catalina.startup.ClassLoaderFactory validateFile Sep 7 14:11:08 myserver server: WARNING: Problem with JAR file [/var/lib/pki/pki-tomcat/common/lib/jaxrs-api.jar], exists: [false], canRead: [false] Sep 7 14:11:08 myserver server: Sep 07, 2016 2:11:08 PM org.apache.catalina.startup.SetAllPropertiesRule begin Sep 7 14:11:08 myserver server: WARNING: [SetAllPropertiesRule]{Server/Service/Connector} Setting property 'enableOCSP' to 'false' did not find a matching property. Sep 7 14:11:08 myserver server: Sep 07, 2016 2:11:08 PM org.apache.catalina.startup.SetAllPropertiesRule begin Sep 7 14:11:08 myserver server: WARNING: [SetAllPropertiesRule]{Server/Service/Connector} Setting property 'ocspResponderURL' to 'http://myserver.mydomain:9080/ca/ocsp' did not find a matching property. Sep 7 14:11:08 myserver server: Sep 07, 2016 2:11:08 PM org.apache.catalina.startup.SetAllPropertiesRule begin Sep 7 14:11:08 myserver server: WARNING: [SetAllPropertiesRule]{Server/Service/Connector} Setting property 'ocspResponderCertNickname' to 'ocspSigningCert cert-pki-ca' did not find a matching property. Sep 7 14:11:08 myserver server: Sep 07, 2016 2:11:08 PM org.apache.catalina.startup.SetAllPropertiesRule begin Sep 7 14:11:08 myserver server: WARNING: [SetAllPropertiesRule]{Server/Service/Connector} Setting property 'ocspCacheSize' to '1000' did not find a matching property. Sep 7 14:11:08 myserver server: Sep 07, 2016 2:11:08 PM org.apache.catalina.startup.SetAllPropertiesRule begin Sep 7 14:11:08 myserver server: WARNING: [SetAllPropertiesRule]{Server/Service/Connector} Setting property 'ocspMinCacheEntryDuration' to '60' did not find a matching property. Sep 7 14:11:08 myserver server: Sep 07, 2016 2:11:08 PM org.apache.catalina.startup.SetAllPropertiesRule begin Sep 7 14:11:08 myserver server: WARNING: [SetAllPropertiesRule]{Server/Service/Connector} Setting property 'ocspMaxCacheEntryDuration' to '120' did not find a matching property. Sep 7 14:11:08 myserver server: Sep 07, 2016 2:11:08 PM org.apache.catalina.startup.SetAllPropertiesRule begin Sep 7 14:11:08 myserver server: WARNING: [SetAllPropertiesRule]{Server/Service/Connector} Setting property 'ocspTimeout' to '10' did not find a matching property. Sep 7 14:11:08 myserver server: Sep 07, 2016 2:11:08 PM org.apache.catalina.startup.SetAllPropertiesRule begin Sep 7 14:11:08 myserver server: WARNING: [SetAllPropertiesRule]{Server/Service/Connector} Setting property 'strictCiphers' to 'true' did not find a matching property. Sep 7 14:11:08 myserver server: Sep 07, 2016 2:11:08 PM org.apache.catalina.startup.SetAllPropertiesRule begin Sep 7 14:11:08 myserver server: WARNING: [SetAllPropertiesRule]{Server/Service/Connector} Setting property 'sslOptions' to 'ssl2=false,ssl3=false,tls=true' did not find a matching property. Sep 7 14:11:08 myserver server: Sep 07, 2016 2:11:08 PM org.apache.catalina.startup.SetAllPropertiesRule begin Sep 7 14:11:08 myserver server: WARNING: [SetAllPropertiesRule]{Server/Service/Connector} Setting property 'ssl2Ciphers' to '-SSL2_RC4_128_WITH_MD5,-SSL2_RC4_128_EXPORT40_WITH_MD5,-SSL2_RC2_128_CBC_WITH_MD5,-SSL2_RC2_128_CBC_EXPORT40_WITH_MD5,-SSL2_DES_64_CBC_WITH_MD5,-SSL2_DES_192_EDE3_CBC_WITH_MD5' did not find a matching property. Sep 7 14:11:08 myserver server: Sep 07, 2016 2:11:08 PM org.apache.catalina.startup.SetAllPropertiesRule begin Sep 7 14:11:08 myserver server: WARNING: [SetAllPropertiesRule]{Server/Service/Connector} Setting property 'ssl3Ciphers' to '-SSL3_FORTEZZA_DMS_WITH_NULL_SHA,-SSL3_FORTEZZA_DMS_WITH_RC4_128_SHA,+SSL3_RSA_WITH_RC4_128_SHA,-SSL3_RSA_EXPORT_WITH_RC4_40_MD5,+SSL3_RSA_WITH_3DES_EDE_CBC_SHA,-SSL3_RSA_WITH_DES_CBC_SHA,-SSL3_RSA_EXPORT_WITH_RC2_CBC_40_MD5,-SSL3_FORTEZZA_DMS_WITH_FORTEZZA_CBC_SHA,-SSL_RSA_FIPS_WITH_DES_CBC_SHA,+SSL_RSA_FIPS_WITH_3DES_EDE_CBC_SHA,-SSL3_RSA_WITH_NULL_MD5,-TLS_RSA_EXPORT1024_WITH_RC4_56_SHA,-TLS_ECDH_ECDSA_WITH_AES_256_CBC_SHA,+TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA' did not find a matching property. Sep 7 14:11:08 myserver server: Sep 07, 2016 2:11:08 PM org.apache.catalina.startup.SetAllPropertiesRule begin Sep 7 14:11:08 myserver server: WARNING: [SetAllPropertiesRule]{Server/Service/Connector} Setting property 'tlsCiphers' to '-TLS_ECDH_ECDSA_WITH_AES_128_CBC_SHA,-TLS_ECDH_ECDSA_WITH_3DES_EDE_CBC_SHA,+TLS_ECDH_RSA_WITH_3DES_EDE_CBC_SHA,+TLS_ECDH_RSA_WITH_AES_128_CBC_SHA,+TLS_ECDH_RSA_WITH_AES_256_CBC_SHA,-TLS_ECDH_ECDSA_WITH_AES_256_CBC_SHA,+TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA,+TLS_RSA_WITH_3DES_EDE_CBC_SHA,+TLS_RSA_WITH_AES_128_CBC_SHA,+TLS_RSA_WITH_AES_256_CBC_SHA,+TLS_ECDHE_ECDSA_WITH_3DES_EDE_CBC_SHA,+TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA,-TLS_ECDHE_RSA_WITH_3DES_EDE_CBC_SHA,-TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA,-TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA,+TLS_DHE_DSS_WITH_3DES_EDE_CBC_SHA,+TLS_DHE_DSS_WITH_AES_128_CBC_SHA,+TLS_DHE_DSS_WITH_AES_256_CBC_SHA,+TLS_DHE_RSA_WITH_3DES_EDE_CBC_SHA,+TLS_DHE_RSA_WITH_AES_128_CBC_SHA,+TLS_DHE_RSA_WITH_AES_256_CBC_SHA' did not find a matching property. Sep 7 14:11:08 myserver server: Sep 07, 2016 2:11:08 PM org.apache.catalina.startup.SetAllPropertiesRule begin Sep 7 14:11:08 myserver server: WARNING: [SetAllPropertiesRule]{Server/Service/Connector} Setting property 'sslVersionRangeStream' to 'tls1_0:tls1_2' did not find a matching property. Sep 7 14:11:08 myserver server: Sep 07, 2016 2:11:08 PM org.apache.catalina.startup.SetAllPropertiesRule begin Sep 7 14:11:08 myserver server: WARNING: [SetAllPropertiesRule]{Server/Service/Connector} Setting property 'sslVersionRangeDatagram' to 'tls1_1:tls1_2' did not find a matching property. Sep 7 14:11:08 myserver server: Sep 07, 2016 2:11:08 PM org.apache.catalina.startup.SetAllPropertiesRule begin Sep 7 14:11:08 myserver server: WARNING: [SetAllPropertiesRule]{Server/Service/Connector} Setting property 'sslRangeCiphers' to '-TLS_ECDH_ECDSA_WITH_AES_128_CBC_SHA,-TLS_ECDH_ECDSA_WITH_3DES_EDE_CBC_SHA,+TLS_ECDH_RSA_WITH_3DES_EDE_CBC_SHA,+TLS_ECDH_RSA_WITH_AES_128_CBC_SHA,+TLS_ECDH_RSA_WITH_AES_256_CBC_SHA,-TLS_ECDH_ECDSA_WITH_AES_256_CBC_SHA,+TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA,+TLS_RSA_WITH_3DES_EDE_CBC_SHA,+TLS_RSA_WITH_AES_128_CBC_SHA,+TLS_RSA_WITH_AES_256_CBC_SHA,+TLS_ECDHE_ECDSA_WITH_3DES_EDE_CBC_SHA,+TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA,-TLS_ECDHE_RSA_WITH_3DES_EDE_CBC_SHA,-TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA,-TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA,+TLS_DHE_DSS_WITH_3DES_EDE_CBC_SHA,+TLS_DHE_DSS_WITH_AES_128_CBC_SHA,+TLS_DHE_DSS_WITH_AES_256_CBC_SHA,+TLS_DHE_RSA_WITH_3DES_EDE_CBC_SHA,+TLS_DHE_RSA_WITH_AES_128_CBC_SHA,+TLS_DHE_RSA_WITH_AES_256_CBC_SHA,+TLS_DHE_RSA_WITH_AES_128_CBC_SHA256,+TLS_DHE_RSA_WITH_AES_256_CBC_SHA256,+TLS_RSA_WITH_AES_128_CBC_SHA256,+TLS_RSA_WITH_AES_256_CBC_SHA256,+TLS_RSA_WITH_AES_128_GCM_SHA256,+TLS_DHE_RSA_WITH_AES_128_GCM_SHA256,+TLS_DHE_DSS_WITH_AES_128_GCM_SHA256,+TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256,+TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256,+TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,+TLS_ECDH_ECDSA_WITH_AES_128_GCM_SHA256,+TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,+TLS_ECDH_RSA_WITH_AES_128_GCM_SHA256' did not find a matching property. Sep 7 14:11:08 myserver server: Sep 07, 2016 2:11:08 PM org.apache.catalina.startup.SetAllPropertiesRule begin Sep 7 14:11:08 myserver server: WARNING: [SetAllPropertiesRule]{Server/Service/Connector} Setting property 'serverCertNickFile' to '/var/lib/pki/pki-tomcat/conf/serverCertNick.conf' did not find a matching property. Sep 7 14:11:08 myserver server: Sep 07, 2016 2:11:08 PM org.apache.catalina.startup.SetAllPropertiesRule begin Sep 7 14:11:08 myserver server: WARNING: [SetAllPropertiesRule]{Server/Service/Connector} Setting property 'passwordFile' to '/var/lib/pki/pki-tomcat/conf/password.conf' did not find a matching property. Sep 7 14:11:08 myserver server: Sep 07, 2016 2:11:08 PM org.apache.catalina.startup.SetAllPropertiesRule begin Sep 7 14:11:08 myserver server: WARNING: [SetAllPropertiesRule]{Server/Service/Connector} Setting property 'passwordClass' to 'org.apache.tomcat.util.net.jss.PlainPasswordFile' did not find a matching property. Sep 7 14:11:08 myserver server: Sep 07, 2016 2:11:08 PM org.apache.catalina.startup.SetAllPropertiesRule begin Sep 7 14:11:08 myserver server: WARNING: [SetAllPropertiesRule]{Server/Service/Connector} Setting property 'certdbDir' to '/var/lib/pki/pki-tomcat/alias' did not find a matching property. Sep 7 14:11:08 myserver server: Sep 07, 2016 2:11:08 PM org.apache.tomcat.util.digester.SetPropertiesRule begin Sep 7 14:11:08 myserver server: WARNING: [SetPropertiesRule]{Server/Service/Engine/Host} Setting property 'xmlValidation' to 'false' did not find a matching property. Sep 7 14:11:08 myserver server: Sep 07, 2016 2:11:08 PM org.apache.tomcat.util.digester.SetPropertiesRule begin Sep 7 14:11:08 myserver server: WARNING: [SetPropertiesRule]{Server/Service/Engine/Host} Setting property 'xmlNamespaceAware' to 'false' did not find a matching property. Sep 7 14:11:08 myserver server: Sep 07, 2016 2:11:08 PM org.apache.tomcat.util.digester.SetPropertiesRule begin Sep 7 14:11:08 myserver server: WARNING: [SetPropertiesRule]{Server/Service/Engine/Host/Valve} Setting property 'resolveHosts' to 'false' did not find a matching property. Sep 7 14:11:08 myserver server: Sep 07, 2016 2:11:08 PM org.apache.catalina.startup.VersionLoggerListener log Sep 7 14:11:08 myserver server: INFO: Server version: Apache Tomcat/8.0.36 Sep 7 14:11:08 myserver server: Sep 07, 2016 2:11:08 PM org.apache.catalina.startup.VersionLoggerListener log Sep 7 14:11:08 myserver server: INFO: Server built: Aug 11 2016 18:10:25 UTC Sep 7 14:11:08 myserver server: Sep 07, 2016 2:11:08 PM org.apache.catalina.startup.VersionLoggerListener log Sep 7 14:11:08 myserver server: INFO: Server number: 8.0.36.0 Sep 7 14:11:08 myserver server: Sep 07, 2016 2:11:08 PM org.apache.catalina.startup.VersionLoggerListener log Sep 7 14:11:08 myserver server: INFO: OS Name: Linux Sep 7 14:11:08 myserver server: Sep 07, 2016 2:11:08 PM org.apache.catalina.startup.VersionLoggerListener log Sep 7 14:11:08 myserver server: INFO: OS Version: 4.7.2-201.fc24.x86_64 Sep 7 14:11:08 myserver server: Sep 07, 2016 2:11:08 PM org.apache.catalina.startup.VersionLoggerListener log Sep 7 14:11:08 myserver server: INFO: Architecture: amd64 Sep 7 14:11:08 myserver server: Sep 07, 2016 2:11:08 PM org.apache.catalina.startup.VersionLoggerListener log Sep 7 14:11:08 myserver server: INFO: Java Home: /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.102-1.b14.fc24.x86_64/jre Sep 7 14:11:08 myserver server: Sep 07, 2016 2:11:08 PM org.apache.catalina.startup.VersionLoggerListener log Sep 7 14:11:08 myserver server: INFO: JVM Version: 1.8.0_102-b14 Sep 7 14:11:08 myserver server: Sep 07, 2016 2:11:08 PM org.apache.catalina.startup.VersionLoggerListener log Sep 7 14:11:08 myserver server: INFO: JVM Vendor: Oracle Corporation Sep 7 14:11:08 myserver server: Sep 07, 2016 2:11:08 PM org.apache.catalina.startup.VersionLoggerListener log Sep 7 14:11:08 myserver server: INFO: CATALINA_BASE: /var/lib/pki/pki-tomcat Sep 7 14:11:08 myserver server: Sep 07, 2016 2:11:08 PM org.apache.catalina.startup.VersionLoggerListener log Sep 7 14:11:08 myserver server: INFO: CATALINA_HOME: /usr/share/tomcat Sep 7 14:11:08 myserver server: Sep 07, 2016 2:11:08 PM org.apache.catalina.startup.VersionLoggerListener log Sep 7 14:11:08 myserver server: INFO: Command line argument: -agentpath:/usr/lib/abrt-java-connector/libabrt-java-connector.so=abrt=on Sep 7 14:11:08 myserver server: Sep 07, 2016 2:11:08 PM org.apache.catalina.startup.VersionLoggerListener log Sep 7 14:11:08 myserver server: INFO: Command line argument: -DRESTEASY_LIB=/usr/share/java/resteasy Sep 7 14:11:08 myserver server: Sep 07, 2016 2:11:08 PM org.apache.catalina.startup.VersionLoggerListener log Sep 7 14:11:08 myserver server: INFO: Command line argument: -Djava.library.path=/usr/lib64/nuxwdog-jni Sep 7 14:11:08 myserver server: Sep 07, 2016 2:11:08 PM org.apache.catalina.startup.VersionLoggerListener log Sep 7 14:11:08 myserver server: INFO: Command line argument: -Dcatalina.base=/var/lib/pki/pki-tomcat Sep 7 14:11:08 myserver server: Sep 07, 2016 2:11:08 PM org.apache.catalina.startup.VersionLoggerListener log Sep 7 14:11:08 myserver server: INFO: Command line argument: -Dcatalina.home=/usr/share/tomcat Sep 7 14:11:08 myserver server: Sep 07, 2016 2:11:08 PM org.apache.catalina.startup.VersionLoggerListener log Sep 7 14:11:08 myserver server: INFO: Command line argument: -Djava.endorsed.dirs= Sep 7 14:11:08 myserver server: Sep 07, 2016 2:11:08 PM org.apache.catalina.startup.VersionLoggerListener log Sep 7 14:11:08 myserver server: INFO: Command line argument: -Djava.io.tmpdir=/var/lib/pki/pki-tomcat/temp Sep 7 14:11:08 myserver server: Sep 07, 2016 2:11:08 PM org.apache.catalina.startup.VersionLoggerListener log Sep 7 14:11:08 myserver server: INFO: Command line argument: -Djava.util.logging.config.file=/var/lib/pki/pki-tomcat/conf/logging.properties Sep 7 14:11:08 myserver server: Sep 07, 2016 2:11:08 PM org.apache.catalina.startup.VersionLoggerListener log Sep 7 14:11:08 myserver server: INFO: Command line argument: -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager Sep 7 14:11:08 myserver server: Sep 07, 2016 2:11:08 PM org.apache.catalina.startup.VersionLoggerListener log Sep 7 14:11:08 myserver server: INFO: Command line argument: -Djava.security.manager Sep 7 14:11:08 myserver server: Sep 07, 2016 2:11:08 PM org.apache.catalina.startup.VersionLoggerListener log Sep 7 14:11:08 myserver server: INFO: Command line argument: -Djava.security.policy==/var/lib/pki/pki-tomcat/conf/catalina.policy Sep 7 14:11:08 myserver server: PKIListener: org.apache.catalina.core.StandardServer[before_init] Sep 7 14:11:08 myserver server: Sep 07, 2016 2:11:08 PM org.apache.coyote.AbstractProtocol init Sep 7 14:11:08 myserver server: INFO: Initializing ProtocolHandler ["http-nio-8080"] Sep 7 14:11:08 myserver server: Sep 07, 2016 2:11:08 PM org.apache.tomcat.util.net.NioSelectorPool getSharedSelector Sep 7 14:11:08 myserver server: INFO: Using a shared selector for servlet write/read Sep 7 14:11:08 myserver server: Sep 07, 2016 2:11:08 PM org.apache.coyote.AbstractProtocol init Sep 7 14:11:08 myserver server: INFO: Initializing ProtocolHandler ["http-bio-8443"] Sep 7 14:11:08 myserver server: Error: SSL cipher "TLS_ECDH_RSA_WITH_3DES_EDE_CBC_SHA" not recognized by tomcatjss Sep 7 14:11:08 myserver server: Error: SSL cipher "TLS_ECDH_RSA_WITH_AES_128_CBC_SHA" not recognized by tomcatjss Sep 7 14:11:08 myserver server: Error: SSL cipher "TLS_ECDH_RSA_WITH_AES_256_CBC_SHA" not recognized by tomcatjss Sep 7 14:11:08 myserver server: Error: SSL cipher "TLS_RSA_WITH_3DES_EDE_CBC_SHA" not recognized by tomcatjss Sep 7 14:11:08 myserver server: Error: SSL cipher "TLS_DHE_DSS_WITH_3DES_EDE_CBC_SHA" not recognized by tomcatjss Sep 7 14:11:08 myserver server: Error: SSL cipher "TLS_DHE_RSA_WITH_3DES_EDE_CBC_SHA" not recognized by tomcatjss Sep 7 14:11:08 myserver server: Error: SSL cipher "TLS_ECDH_ECDSA_WITH_AES_128_GCM_SHA256" unsupported by NSS Sep 7 14:11:08 myserver server: Error: SSL cipher "TLS_ECDH_RSA_WITH_AES_128_GCM_SHA256" unsupported by NSS Sep 7 14:11:08 myserver server: Sep 07, 2016 2:11:08 PM org.apache.coyote.AbstractProtocol init Sep 7 14:11:08 myserver server: INFO: Initializing ProtocolHandler ["ajp-nio-127.0.0.1-8009"] Sep 7 14:11:08 myserver server: Sep 07, 2016 2:11:08 PM org.apache.tomcat.util.net.NioSelectorPool getSharedSelector Sep 7 14:11:08 myserver server: INFO: Using a shared selector for servlet write/read Sep 7 14:11:08 myserver server: PKIListener: org.apache.catalina.core.StandardServer[after_init] Sep 7 14:11:08 myserver server: Sep 07, 2016 2:11:08 PM org.apache.catalina.startup.Catalina load Sep 7 14:11:08 myserver server: INFO: Initialization processed in 637 ms Sep 7 14:11:08 myserver server: PKIListener: org.apache.catalina.core.StandardServer[before_start] Sep 7 14:11:08 myserver server: PKIListener: org.apache.catalina.core.StandardServer[configure_start] Sep 7 14:11:08 myserver server: PKIListener: org.apache.catalina.core.StandardServer[start] Sep 7 14:11:08 myserver server: Sep 07, 2016 2:11:08 PM org.apache.catalina.core.StandardService startInternal Sep 7 14:11:08 myserver server: INFO: Starting service Catalina Sep 7 14:11:08 myserver server: Sep 07, 2016 2:11:08 PM org.apache.catalina.core.StandardEngine startInternal Sep 7 14:11:08 myserver server: INFO: Starting Servlet Engine: Apache Tomcat/8.0.36 Sep 7 14:11:08 myserver server: Sep 07, 2016 2:11:08 PM org.apache.catalina.startup.HostConfig deployDescriptor Sep 7 14:11:08 myserver server: INFO: Deploying configuration descriptor /etc/pki/pki-tomcat/Catalina/localhost/ROOT.xml Sep 7 14:11:10 myserver server: Sep 07, 2016 2:11:10 PM org.apache.jasper.servlet.TldScanner scanJars Sep 7 14:11:10 myserver server: INFO: At least one JAR was scanned for TLDs yet contained no TLDs. Enable debug logging for this logger for a complete list of JARs that were scanned but no TLDs were found in them. Skipping unneeded JARs during scanning can improve startup time and JSP compilation time. Sep 7 14:11:10 myserver ns-slapd: [07/Sep/2016:14:11:10.369436624 +0200] NSMMReplicationPlugin - agmt="cn=meTozaira.mydomain" (zaira:389): Replication bind with GSSAPI auth resumed Sep 7 14:11:10 myserver server: Sep 07, 2016 2:11:10 PM org.apache.catalina.startup.HostConfig deployDescriptor Sep 7 14:11:10 myserver server: INFO: Deployment of configuration descriptor /etc/pki/pki-tomcat/Catalina/localhost/ROOT.xml has finished in 1,430 ms Sep 7 14:11:10 myserver server: Sep 07, 2016 2:11:10 PM org.apache.catalina.startup.HostConfig deployDescriptor Sep 7 14:11:10 myserver server: INFO: Deploying configuration descriptor /etc/pki/pki-tomcat/Catalina/localhost/ca.xml Sep 7 14:11:10 myserver server: SSLAuthenticatorWithFallback: Creating SSL authenticator with fallback Sep 7 14:11:10 myserver server: SSLAuthenticatorWithFallback: Setting container Sep 7 14:11:10 myserver server: Sep 07, 2016 2:11:10 PM org.apache.tomcat.util.digester.Digester startElement Sep 7 14:11:10 myserver server: SEVERE: Begin event threw error Sep 7 14:11:10 myserver server: java.lang.NoClassDefFoundError: javax/ws/rs/ServiceUnavailableException Sep 7 14:11:10 myserver server: #011at java.lang.Class.getDeclaredConstructors0(Native Method) Sep 7 14:11:10 myserver server: #011at java.lang.Class.privateGetDeclaredConstructors(Class.java:2671) Sep 7 14:11:10 myserver server: #011at java.lang.Class.getConstructor0(Class.java:3075) Sep 7 14:11:10 myserver server: #011at java.lang.Class.newInstance(Class.java:412) Sep 7 14:11:10 myserver server: #011at org.apache.tomcat.util.digester.ObjectCreateRule.begin(ObjectCreateRule.java:117) Sep 7 14:11:10 myserver server: #011at org.apache.tomcat.util.digester.Digester.startElement(Digester.java:1178) Sep 7 14:11:10 myserver server: #011at org.apache.xerces.parsers.AbstractSAXParser.startElement(Unknown Source) Sep 7 14:11:10 myserver server: #011at org.apache.xerces.parsers.AbstractXMLDocumentParser.emptyElement(Unknown Source) Sep 7 14:11:10 myserver server: #011at org.apache.xerces.impl.XMLDocumentFragmentScannerImpl.scanStartElement(Unknown Source) Sep 7 14:11:10 myserver server: #011at org.apache.xerces.impl.XMLDocumentFragmentScannerImpl$FragmentContentDispatcher.dispatch(Unknown Source) Sep 7 14:11:10 myserver server: #011at org.apache.xerces.impl.XMLDocumentFragmentScannerImpl.scanDocument(Unknown Source) Sep 7 14:11:10 myserver server: #011at org.apache.xerces.parsers.XML11Configuration.parse(Unknown Source) Sep 7 14:11:10 myserver server: #011at org.apache.xerces.parsers.XML11Configuration.parse(Unknown Source) Sep 7 14:11:10 myserver server: #011at org.apache.xerces.parsers.XMLParser.parse(Unknown Source) Sep 7 14:11:10 myserver server: #011at org.apache.xerces.parsers.AbstractSAXParser.parse(Unknown Source) Sep 7 14:11:10 myserver server: #011at org.apache.xerces.jaxp.SAXParserImpl$JAXPSAXParser.parse(Unknown Source) Sep 7 14:11:10 myserver server: #011at org.apache.tomcat.util.digester.Digester.parse(Digester.java:1451) Sep 7 14:11:10 myserver server: #011at org.apache.catalina.startup.ContextConfig.processContextConfig(ContextConfig.java:542) Sep 7 14:11:10 myserver server: #011at org.apache.catalina.startup.ContextConfig.contextConfig(ContextConfig.java:501) Sep 7 14:11:10 myserver server: #011at org.apache.catalina.startup.ContextConfig.init(ContextConfig.java:739) Sep 7 14:11:10 myserver server: #011at org.apache.catalina.startup.ContextConfig.lifecycleEvent(ContextConfig.java:317) Sep 7 14:11:10 myserver server: #011at org.apache.catalina.util.LifecycleSupport.fireLifecycleEvent(LifecycleSupport.java:95) Sep 7 14:11:10 myserver server: #011at org.apache.catalina.util.LifecycleBase.fireLifecycleEvent(LifecycleBase.java:90) Sep 7 14:11:10 myserver server: #011at org.apache.catalina.util.LifecycleBase.setStateInternal(LifecycleBase.java:394) Sep 7 14:11:10 myserver server: #011at org.apache.catalina.util.LifecycleBase.init(LifecycleBase.java:110) Sep 7 14:11:10 myserver server: #011at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:136) Sep 7 14:11:10 myserver server: #011at org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:725) Sep 7 14:11:10 myserver server: #011at org.apache.catalina.core.ContainerBase.access$000(ContainerBase.java:131) Sep 7 14:11:10 myserver server: #011at org.apache.catalina.core.ContainerBase$PrivilegedAddChild.run(ContainerBase.java:153) Sep 7 14:11:10 myserver server: #011at org.apache.catalina.core.ContainerBase$PrivilegedAddChild.run(ContainerBase.java:143) Sep 7 14:11:10 myserver server: #011at java.security.AccessController.doPrivileged(Native Method) Sep 7 14:11:10 myserver server: #011at org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:699) Sep 7 14:11:10 myserver server: #011at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:717) Sep 7 14:11:10 myserver server: #011at org.apache.catalina.startup.HostConfig.deployDescriptor(HostConfig.java:587) Sep 7 14:11:10 myserver server: #011at org.apache.catalina.startup.HostConfig$DeployDescriptor.run(HostConfig.java:1798) Sep 7 14:11:10 myserver server: #011at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) Sep 7 14:11:10 myserver server: #011at java.util.concurrent.FutureTask.run(FutureTask.java:266) Sep 7 14:11:10 myserver server: #011at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) Sep 7 14:11:10 myserver server: #011at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) Sep 7 14:11:10 myserver server: #011at java.lang.Thread.run(Thread.java:745) Sep 7 14:11:10 myserver server: Caused by: java.lang.ClassNotFoundException: javax.ws.rs.ServiceUnavailableException Sep 7 14:11:10 myserver server: #011at java.net.URLClassLoader.findClass(URLClassLoader.java:381) Sep 7 14:11:10 myserver server: #011at java.lang.ClassLoader.loadClass(ClassLoader.java:424) Sep 7 14:11:10 myserver server: #011at java.lang.ClassLoader.loadClass(ClassLoader.java:357) Sep 7 14:11:10 myserver server: #011... 40 more Sep 7 14:11:10 myserver server: Sep 07, 2016 2:11:10 PM org.apache.catalina.startup.HostConfig deployDescriptor Sep 7 14:11:10 myserver server: SEVERE: Error deploying configuration descriptor /etc/pki/pki-tomcat/Catalina/localhost/ca.xml Sep 7 14:11:10 myserver server: java.lang.NoClassDefFoundError: javax/ws/rs/ServiceUnavailableException Sep 7 14:11:10 myserver server: #011at java.lang.Class.getDeclaredConstructors0(Native Method) Sep 7 14:11:10 myserver server: #011at java.lang.Class.privateGetDeclaredConstructors(Class.java:2671) Sep 7 14:11:10 myserver server: #011at java.lang.Class.getConstructor0(Class.java:3075) Sep 7 14:11:10 myserver server: #011at java.lang.Class.newInstance(Class.java:412) Sep 7 14:11:10 myserver server: #011at org.apache.tomcat.util.digester.ObjectCreateRule.begin(ObjectCreateRule.java:117) Sep 7 14:11:10 myserver server: #011at org.apache.tomcat.util.digester.Digester.startElement(Digester.java:1178) Sep 7 14:11:10 myserver server: #011at org.apache.xerces.parsers.AbstractSAXParser.startElement(Unknown Source) Sep 7 14:11:10 myserver server: #011at org.apache.xerces.parsers.AbstractXMLDocumentParser.emptyElement(Unknown Source) Sep 7 14:11:10 myserver server: #011at org.apache.xerces.impl.XMLDocumentFragmentScannerImpl.scanStartElement(Unknown Source) Sep 7 14:11:10 myserver server: #011at org.apache.xerces.impl.XMLDocumentFragmentScannerImpl$FragmentContentDispatcher.dispatch(Unknown Source) Sep 7 14:11:10 myserver server: #011at org.apache.xerces.impl.XMLDocumentFragmentScannerImpl.scanDocument(Unknown Source) Sep 7 14:11:10 myserver server: #011at org.apache.xerces.parsers.XML11Configuration.parse(Unknown Source) Sep 7 14:11:10 myserver server: #011at org.apache.xerces.parsers.XML11Configuration.parse(Unknown Source) Sep 7 14:11:10 myserver server: #011at org.apache.xerces.parsers.XMLParser.parse(Unknown Source) Sep 7 14:11:10 myserver server: #011at org.apache.xerces.parsers.AbstractSAXParser.parse(Unknown Source) Sep 7 14:11:10 myserver server: #011at org.apache.xerces.jaxp.SAXParserImpl$JAXPSAXParser.parse(Unknown Source) Sep 7 14:11:10 myserver server: #011at org.apache.tomcat.util.digester.Digester.parse(Digester.java:1451) Sep 7 14:11:10 myserver server: #011at org.apache.catalina.startup.ContextConfig.processContextConfig(ContextConfig.java:542) Sep 7 14:11:10 myserver server: #011at org.apache.catalina.startup.ContextConfig.contextConfig(ContextConfig.java:501) Sep 7 14:11:10 myserver server: #011at org.apache.catalina.startup.ContextConfig.init(ContextConfig.java:739) Sep 7 14:11:10 myserver server: #011at org.apache.catalina.startup.ContextConfig.lifecycleEvent(ContextConfig.java:317) Sep 7 14:11:10 myserver server: #011at org.apache.catalina.util.LifecycleSupport.fireLifecycleEvent(LifecycleSupport.java:95) Sep 7 14:11:10 myserver server: #011at org.apache.catalina.util.LifecycleBase.fireLifecycleEvent(LifecycleBase.java:90) Sep 7 14:11:10 myserver server: #011at org.apache.catalina.util.LifecycleBase.setStateInternal(LifecycleBase.java:394) Sep 7 14:11:10 myserver server: #011at org.apache.catalina.util.LifecycleBase.init(LifecycleBase.java:110) Sep 7 14:11:10 myserver server: #011at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:136) Sep 7 14:11:10 myserver server: #011at org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:725) Sep 7 14:11:10 myserver server: #011at org.apache.catalina.core.ContainerBase.access$000(ContainerBase.java:131) Sep 7 14:11:10 myserver server: #011at org.apache.catalina.core.ContainerBase$PrivilegedAddChild.run(ContainerBase.java:153) Sep 7 14:11:10 myserver server: #011at org.apache.catalina.core.ContainerBase$PrivilegedAddChild.run(ContainerBase.java:143) Sep 7 14:11:10 myserver server: #011at java.security.AccessController.doPrivileged(Native Method) Sep 7 14:11:10 myserver server: #011at org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:699) Sep 7 14:11:10 myserver server: #011at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:717) Sep 7 14:11:10 myserver server: #011at org.apache.catalina.startup.HostConfig.deployDescriptor(HostConfig.java:587) Sep 7 14:11:10 myserver server: #011at org.apache.catalina.startup.HostConfig$DeployDescriptor.run(HostConfig.java:1798) Sep 7 14:11:10 myserver server: #011at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) Sep 7 14:11:10 myserver server: #011at java.util.concurrent.FutureTask.run(FutureTask.java:266) Sep 7 14:11:10 myserver server: #011at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) Sep 7 14:11:10 myserver server: #011at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) Sep 7 14:11:10 myserver server: #011at java.lang.Thread.run(Thread.java:745) Sep 7 14:11:10 myserver server: Caused by: java.lang.ClassNotFoundException: javax.ws.rs.ServiceUnavailableException Sep 7 14:11:10 myserver server: #011at java.net.URLClassLoader.findClass(URLClassLoader.java:381) Sep 7 14:11:10 myserver server: #011at java.lang.ClassLoader.loadClass(ClassLoader.java:424) Sep 7 14:11:10 myserver server: #011at java.lang.ClassLoader.loadClass(ClassLoader.java:357) Sep 7 14:11:10 myserver server: #011... 40 more Sep 7 14:11:10 myserver server: Sep 07, 2016 2:11:10 PM org.apache.catalina.startup.HostConfig deployDescriptor Sep 7 14:11:10 myserver server: INFO: Deployment of configuration descriptor /etc/pki/pki-tomcat/Catalina/localhost/ca.xml has finished in 11 ms Sep 7 14:11:10 myserver server: Sep 07, 2016 2:11:10 PM org.apache.catalina.startup.HostConfig deployDescriptor Sep 7 14:11:10 myserver server: INFO: Deploying configuration descriptor /etc/pki/pki-tomcat/Catalina/localhost/pki#js.xml Sep 7 14:11:10 myserver server: Sep 07, 2016 2:11:10 PM org.apache.catalina.startup.SetContextPropertiesRule begin Sep 7 14:11:10 myserver server: WARNING: [SetContextPropertiesRule]{Context} Setting property 'allowLinking' to 'true' did not find a matching property. Sep 7 14:11:11 myserver server: Sep 07, 2016 2:11:11 PM org.apache.jasper.servlet.TldScanner scanJars Sep 7 14:11:11 myserver server: INFO: At least one JAR was scanned for TLDs yet contained no TLDs. Enable debug logging for this logger for a complete list of JARs that were scanned but no TLDs were found in them. Skipping unneeded JARs during scanning can improve startup time and JSP compilation time. Sep 7 14:11:11 myserver server: Sep 07, 2016 2:11:11 PM org.apache.catalina.startup.HostConfig deployDescriptor Sep 7 14:11:11 myserver server: INFO: Deployment of configuration descriptor /etc/pki/pki-tomcat/Catalina/localhost/pki#js.xml has finished in 865 ms Sep 7 14:11:11 myserver server: Sep 07, 2016 2:11:11 PM org.apache.catalina.startup.HostConfig deployDescriptor Sep 7 14:11:11 myserver server: INFO: Deploying configuration descriptor /etc/pki/pki-tomcat/Catalina/localhost/pki#admin.xml Sep 7 14:11:11 myserver server: Sep 07, 2016 2:11:11 PM org.apache.catalina.startup.SetContextPropertiesRule begin Sep 7 14:11:11 myserver server: WARNING: [SetContextPropertiesRule]{Context} Setting property 'allowLinking' to 'true' did not find a matching property. Sep 7 14:11:11 myserver ns-slapd: [07/Sep/2016:14:11:11.366262264 +0200] schema-compat-plugin - Finished plugin initialization. Sep 7 14:11:12 myserver server: Sep 07, 2016 2:11:12 PM org.apache.jasper.servlet.TldScanner scanJars Sep 7 14:11:12 myserver server: INFO: At least one JAR was scanned for TLDs yet contained no TLDs. Enable debug logging for this logger for a complete list of JARs that were scanned but no TLDs were found in them. Skipping unneeded JARs during scanning can improve startup time and JSP compilation time. Sep 7 14:11:12 myserver server: Sep 07, 2016 2:11:12 PM org.apache.catalina.startup.HostConfig deployDescriptor Sep 7 14:11:12 myserver server: INFO: Deployment of configuration descriptor /etc/pki/pki-tomcat/Catalina/localhost/pki#admin.xml has finished in 854 ms Sep 7 14:11:12 myserver server: Sep 07, 2016 2:11:12 PM org.apache.coyote.AbstractProtocol start Sep 7 14:11:12 myserver server: INFO: Starting ProtocolHandler ["http-nio-8080"] Sep 7 14:11:12 myserver server: Sep 07, 2016 2:11:12 PM org.apache.coyote.AbstractProtocol start Sep 7 14:11:12 myserver server: INFO: Starting ProtocolHandler ["http-bio-8443"] Sep 7 14:11:12 myserver server: Sep 07, 2016 2:11:12 PM org.apache.coyote.AbstractProtocol start Sep 7 14:11:12 myserver server: INFO: Starting ProtocolHandler ["ajp-nio-127.0.0.1-8009"] Sep 7 14:11:12 myserver server: PKIListener: org.apache.catalina.core.StandardServer[after_start] Sep 7 14:11:12 myserver server: PKIListener: Subsystem CA is running. Sep 7 14:11:12 myserver server: Sep 07, 2016 2:11:12 PM org.apache.catalina.startup.Catalina start Sep 7 14:11:12 myserver server: INFO: Server startup in 3242 ms Sep 7 14:11:12 myserver named-pkcs11[9425]: zone 21.0.10.in-addr.arpa/IN: sending notifies (serial 1473250267) Sep 7 14:11:12 myserver named-pkcs11[9425]: zone 1.168.192.in-addr.arpa/IN: sending notifies (serial 1473250267) Sep 7 14:11:12 myserver named-pkcs11[9425]: zone chloe/IN: sending notifies (serial 1473250267) Sep 7 14:11:12 myserver named-pkcs11[9425]: zone mydomain/IN: sending notifies (serial 1473250267) Regards F. -------------- next part -------------- An HTML attachment was scrubbed... URL: From abokovoy at redhat.com Wed Sep 7 12:53:16 2016 From: abokovoy at redhat.com (Alexander Bokovoy) Date: Wed, 7 Sep 2016 15:53:16 +0300 Subject: [Freeipa-users] Freeipa not working after upgrade to F24 In-Reply-To: References: Message-ID: <20160907125316.yketccvmq5rmoe4t@redhat.com> On Wed, 07 Sep 2016, Fujisan wrote: >Hello, > >I just upgraded my server to F24 but ipa is not starting. > >No DNS, no access to the Freeipa web page. It is an issue with Java jar classes being renamed. Run 'pki-server-upgrade -v' to re-apply changes. There is an issue with upgrade script that fails to import some Python code for SELinux during rpm upgrade and that stops pki-server-upgrade from completing its job. See https://fedorahosted.org/pki/ticket/2452 for more details. >Sep 7 14:11:10 myserver server: Sep 07, 2016 2:11:10 PM >org.apache.tomcat.util.digester.Digester startElement >Sep 7 14:11:10 myserver server: SEVERE: Begin event threw error >Sep 7 14:11:10 myserver server: java.lang.NoClassDefFoundError: >javax/ws/rs/ServiceUnavailableException ^^^ this is it. -- / Alexander Bokovoy From fujisan43 at gmail.com Wed Sep 7 12:58:31 2016 From: fujisan43 at gmail.com (Fujisan) Date: Wed, 7 Sep 2016 14:58:31 +0200 Subject: [Freeipa-users] Freeipa not working after upgrade to F24 In-Reply-To: <20160907125316.yketccvmq5rmoe4t@redhat.com> References: <20160907125316.yketccvmq5rmoe4t@redhat.com> Message-ID: Selinux is disabled on my server. Do I still need to run 'pki-server-upgrade -v'? Regards, F. On Wed, Sep 7, 2016 at 2:53 PM, Alexander Bokovoy wrote: > On Wed, 07 Sep 2016, Fujisan wrote: > >> Hello, >> >> I just upgraded my server to F24 but ipa is not starting. >> >> No DNS, no access to the Freeipa web page. >> > It is an issue with Java jar classes being renamed. Run > 'pki-server-upgrade -v' to re-apply changes. There is an issue with > upgrade script that fails to import some Python code for SELinux during > rpm upgrade and that stops pki-server-upgrade from completing its job. > > See https://fedorahosted.org/pki/ticket/2452 for more details. > > Sep 7 14:11:10 myserver server: Sep 07, 2016 2:11:10 PM >> org.apache.tomcat.util.digester.Digester startElement >> Sep 7 14:11:10 myserver server: SEVERE: Begin event threw error >> Sep 7 14:11:10 myserver server: java.lang.NoClassDefFoundError: >> javax/ws/rs/ServiceUnavailableException >> > ^^^ this is it. > > > -- > / Alexander Bokovoy > -------------- next part -------------- An HTML attachment was scrubbed... URL: From fujisan43 at gmail.com Wed Sep 7 13:01:16 2016 From: fujisan43 at gmail.com (Fujisan) Date: Wed, 7 Sep 2016 15:01:16 +0200 Subject: [Freeipa-users] Freeipa not working after upgrade to F24 In-Reply-To: References: <20160907125316.yketccvmq5rmoe4t@redhat.com> Message-ID: Ok I just tried and you are right. It works! Pfiuuu! You saved my a$$. Thanks. On Wed, Sep 7, 2016 at 2:58 PM, Fujisan wrote: > Selinux is disabled on my server. > Do I still need to run 'pki-server-upgrade -v'? > > Regards, > F. > > On Wed, Sep 7, 2016 at 2:53 PM, Alexander Bokovoy > wrote: > >> On Wed, 07 Sep 2016, Fujisan wrote: >> >>> Hello, >>> >>> I just upgraded my server to F24 but ipa is not starting. >>> >>> No DNS, no access to the Freeipa web page. >>> >> It is an issue with Java jar classes being renamed. Run >> 'pki-server-upgrade -v' to re-apply changes. There is an issue with >> upgrade script that fails to import some Python code for SELinux during >> rpm upgrade and that stops pki-server-upgrade from completing its job. >> >> See https://fedorahosted.org/pki/ticket/2452 for more details. >> >> Sep 7 14:11:10 myserver server: Sep 07, 2016 2:11:10 PM >>> org.apache.tomcat.util.digester.Digester startElement >>> Sep 7 14:11:10 myserver server: SEVERE: Begin event threw error >>> Sep 7 14:11:10 myserver server: java.lang.NoClassDefFoundError: >>> javax/ws/rs/ServiceUnavailableException >>> >> ^^^ this is it. >> >> >> -- >> / Alexander Bokovoy >> > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From abokovoy at redhat.com Wed Sep 7 13:03:24 2016 From: abokovoy at redhat.com (Alexander Bokovoy) Date: Wed, 7 Sep 2016 16:03:24 +0300 Subject: [Freeipa-users] Freeipa not working after upgrade to F24 In-Reply-To: References: <20160907125316.yketccvmq5rmoe4t@redhat.com> Message-ID: <20160907130324.jx23jkfqodgfdk3i@redhat.com> On Wed, 07 Sep 2016, Fujisan wrote: >Selinux is disabled on my server. >Do I still need to run 'pki-server-upgrade -v'? You need to read the ticket I pointed to and look at the files mentioned there. You'll see the issue and it is fixed by running pki-server-upgrade. The issue is not about SELinux itself. > >Regards, >F. > >On Wed, Sep 7, 2016 at 2:53 PM, Alexander Bokovoy >wrote: > >> On Wed, 07 Sep 2016, Fujisan wrote: >> >>> Hello, >>> >>> I just upgraded my server to F24 but ipa is not starting. >>> >>> No DNS, no access to the Freeipa web page. >>> >> It is an issue with Java jar classes being renamed. Run >> 'pki-server-upgrade -v' to re-apply changes. There is an issue with >> upgrade script that fails to import some Python code for SELinux during >> rpm upgrade and that stops pki-server-upgrade from completing its job. >> >> See https://fedorahosted.org/pki/ticket/2452 for more details. >> >> Sep 7 14:11:10 myserver server: Sep 07, 2016 2:11:10 PM >>> org.apache.tomcat.util.digester.Digester startElement >>> Sep 7 14:11:10 myserver server: SEVERE: Begin event threw error >>> Sep 7 14:11:10 myserver server: java.lang.NoClassDefFoundError: >>> javax/ws/rs/ServiceUnavailableException >>> >> ^^^ this is it. >> >> >> -- >> / Alexander Bokovoy >> >-- >Manage your subscription for the Freeipa-users mailing list: >https://www.redhat.com/mailman/listinfo/freeipa-users >Go to http://freeipa.org for more info on the project -- / Alexander Bokovoy From fujisan43 at gmail.com Wed Sep 7 13:09:17 2016 From: fujisan43 at gmail.com (Fujisan) Date: Wed, 7 Sep 2016 15:09:17 +0200 Subject: [Freeipa-users] Freeipa not working after upgrade to F24 In-Reply-To: <20160907130324.jx23jkfqodgfdk3i@redhat.com> References: <20160907125316.yketccvmq5rmoe4t@redhat.com> <20160907130324.jx23jkfqodgfdk3i@redhat.com> Message-ID: Is this issue documented somewhere? On Wed, Sep 7, 2016 at 3:03 PM, Alexander Bokovoy wrote: > On Wed, 07 Sep 2016, Fujisan wrote: > >> Selinux is disabled on my server. >> Do I still need to run 'pki-server-upgrade -v'? >> > You need to read the ticket I pointed to and look at the files mentioned > there. You'll see the issue and it is fixed by running > pki-server-upgrade. > > The issue is not about SELinux itself. > > > >> Regards, >> F. >> >> On Wed, Sep 7, 2016 at 2:53 PM, Alexander Bokovoy >> wrote: >> >> On Wed, 07 Sep 2016, Fujisan wrote: >>> >>> Hello, >>>> >>>> I just upgraded my server to F24 but ipa is not starting. >>>> >>>> No DNS, no access to the Freeipa web page. >>>> >>>> It is an issue with Java jar classes being renamed. Run >>> 'pki-server-upgrade -v' to re-apply changes. There is an issue with >>> upgrade script that fails to import some Python code for SELinux during >>> rpm upgrade and that stops pki-server-upgrade from completing its job. >>> >>> See https://fedorahosted.org/pki/ticket/2452 for more details. >>> >>> Sep 7 14:11:10 myserver server: Sep 07, 2016 2:11:10 PM >>> >>>> org.apache.tomcat.util.digester.Digester startElement >>>> Sep 7 14:11:10 myserver server: SEVERE: Begin event threw error >>>> Sep 7 14:11:10 myserver server: java.lang.NoClassDefFoundError: >>>> javax/ws/rs/ServiceUnavailableException >>>> >>>> ^^^ this is it. >>> >>> >>> -- >>> / Alexander Bokovoy >>> >>> > -- >> Manage your subscription for the Freeipa-users mailing list: >> https://www.redhat.com/mailman/listinfo/freeipa-users >> Go to http://freeipa.org for more info on the project >> > > > -- > / Alexander Bokovoy > -------------- next part -------------- An HTML attachment was scrubbed... URL: From rcritten at redhat.com Wed Sep 7 13:26:59 2016 From: rcritten at redhat.com (Rob Crittenden) Date: Wed, 7 Sep 2016 09:26:59 -0400 Subject: [Freeipa-users] FreeIPA server install in Ubuntu 14.04 box. In-Reply-To: References: Message-ID: <57D015A3.1020901@redhat.com> Amit Gupta wrote: > Hi, > > I am looking for any document to install and config freeipa on ubuntu > 14.04 server. I don't believe there is any Ubuntu-specific documentation on it. The documentation at redhat.com should still generally apply. I think with 14.x though you won't be able to create replicas due to the openldap library linking with gnutls. I believe 16.x is required (mostly for newer versions of IPA that install replicas differently). rob From rcritten at redhat.com Wed Sep 7 13:27:52 2016 From: rcritten at redhat.com (Rob Crittenden) Date: Wed, 7 Sep 2016 09:27:52 -0400 Subject: [Freeipa-users] ipa: ERROR: Certificate format error: (SEC_ERROR_LEGACY_DATABASE) The certificate/key database is in an old, unsupported format. In-Reply-To: References: Message-ID: <57D015D8.7080507@redhat.com> Natxo Asenjo wrote: > hi, > > using centos 6.8 (server and client), when trying to view some hosts we > get this error: > > > $ ipa host-find host-1920.sub.domain.tld > ipa: ERROR: Certificate format error: (SEC_ERROR_LEGACY_DATABASE) The > certificate/key database is in an old, unsupported format. > > > I saw a thread last year about this, but no solution. > > Any clues? /var/log/httpd/error_log may contain a traceback. rob From abokovoy at redhat.com Wed Sep 7 13:43:06 2016 From: abokovoy at redhat.com (Alexander Bokovoy) Date: Wed, 7 Sep 2016 16:43:06 +0300 Subject: [Freeipa-users] Freeipa not working after upgrade to F24 In-Reply-To: References: <20160907125316.yketccvmq5rmoe4t@redhat.com> <20160907130324.jx23jkfqodgfdk3i@redhat.com> Message-ID: <20160907134306.de6dvvqf7exflqir@redhat.com> On Wed, 07 Sep 2016, Fujisan wrote: >Is this issue documented somewhere? You have the link already. I think dogtag guys will do an update to fix the issue soon. > >On Wed, Sep 7, 2016 at 3:03 PM, Alexander Bokovoy >wrote: > >> On Wed, 07 Sep 2016, Fujisan wrote: >> >>> Selinux is disabled on my server. >>> Do I still need to run 'pki-server-upgrade -v'? >>> >> You need to read the ticket I pointed to and look at the files mentioned >> there. You'll see the issue and it is fixed by running >> pki-server-upgrade. >> >> The issue is not about SELinux itself. >> >> >> >>> Regards, >>> F. >>> >>> On Wed, Sep 7, 2016 at 2:53 PM, Alexander Bokovoy >>> wrote: >>> >>> On Wed, 07 Sep 2016, Fujisan wrote: >>>> >>>> Hello, >>>>> >>>>> I just upgraded my server to F24 but ipa is not starting. >>>>> >>>>> No DNS, no access to the Freeipa web page. >>>>> >>>>> It is an issue with Java jar classes being renamed. Run >>>> 'pki-server-upgrade -v' to re-apply changes. There is an issue with >>>> upgrade script that fails to import some Python code for SELinux during >>>> rpm upgrade and that stops pki-server-upgrade from completing its job. >>>> >>>> See https://fedorahosted.org/pki/ticket/2452 for more details. >>>> >>>> Sep 7 14:11:10 myserver server: Sep 07, 2016 2:11:10 PM >>>> >>>>> org.apache.tomcat.util.digester.Digester startElement >>>>> Sep 7 14:11:10 myserver server: SEVERE: Begin event threw error >>>>> Sep 7 14:11:10 myserver server: java.lang.NoClassDefFoundError: >>>>> javax/ws/rs/ServiceUnavailableException >>>>> >>>>> ^^^ this is it. >>>> >>>> >>>> -- >>>> / Alexander Bokovoy >>>> >>>> >> -- >>> Manage your subscription for the Freeipa-users mailing list: >>> https://www.redhat.com/mailman/listinfo/freeipa-users >>> Go to http://freeipa.org for more info on the project >>> >> >> >> -- >> / Alexander Bokovoy >> -- / Alexander Bokovoy From natxo.asenjo at gmail.com Wed Sep 7 13:45:59 2016 From: natxo.asenjo at gmail.com (Natxo Asenjo) Date: Wed, 7 Sep 2016 15:45:59 +0200 Subject: [Freeipa-users] ipa: ERROR: Certificate format error: (SEC_ERROR_LEGACY_DATABASE) The certificate/key database is in an old, unsupported format. In-Reply-To: <57D015D8.7080507@redhat.com> References: <57D015D8.7080507@redhat.com> Message-ID: On Wed, Sep 7, 2016 at 3:27 PM, Rob Crittenden wrote: > Natxo Asenjo wrote: > >> hi, >> >> using centos 6.8 (server and client), when trying to view some hosts we >> get this error: >> >> >> $ ipa host-find host-1920.sub.domain.tld >> ipa: ERROR: Certificate format error: (SEC_ERROR_LEGACY_DATABASE) The >> certificate/key database is in an old, unsupported format. >> >> >> I saw a thread last year about this, but no solution. >> >> Any clues? >> > > /var/log/httpd/error_log may contain a traceback This made me take a look at a replica and there I could not replicate the error, I got the info I requested. In the apache error file I saw indeed a traceback: [Sun Sep 04 03:21:31 2016] [error] ipa: ERROR: non-public: XMLSyntaxError: None [Sun Sep 04 03:21:31 2016] [error] Traceback (most recent call last): [Sun Sep 04 03:21:31 2016] [error] File "/usr/lib/python2.6/site-packages/ipaserver/rpcserver.py", line 334, in wsgi_execute [Sun Sep 04 03:21:31 2016] [error] result = self.Command[name](*args, **options) [Sun Sep 04 03:21:31 2016] [error] File "/usr/lib/python2.6/site-packages/ipalib/frontend.py", line 438, in __call__ [Sun Sep 04 03:21:31 2016] [error] ret = self.run(*args, **options) [Sun Sep 04 03:21:31 2016] [error] File "/usr/lib/python2.6/site-packages/ipalib/frontend.py", line 750, in run [Sun Sep 04 03:21:31 2016] [error] return self.execute(*args, **options) [Sun Sep 04 03:21:31 2016] [error] File "/usr/lib/python2.6/site-packages/ipalib/plugins/cert.py", line 362, in execute [Sun Sep 04 03:21:31 2016] [error] result = api.Command['cert_show'](unicode(serial))['result'] [Sun Sep 04 03:21:31 2016] [error] File "/usr/lib/python2.6/site-packages/ipalib/frontend.py", line 438, in __call__ [Sun Sep 04 03:21:31 2016] [error] ret = self.run(*args, **options) [Sun Sep 04 03:21:31 2016] [error] File "/usr/lib/python2.6/site-packages/ipalib/frontend.py", line 750, in run [Sun Sep 04 03:21:31 2016] [error] return self.execute(*args, **options) [Sun Sep 04 03:21:31 2016] [error] File "/usr/lib/python2.6/site-packages/ipalib/plugins/cert.py", line 493, in execute [Sun Sep 04 03:21:31 2016] [error] result=self.Backend.ra.get_certificate(serial_number) [Sun Sep 04 03:21:31 2016] [error] File "/usr/lib/python2.6/site-packages/ipaserver/plugins/dogtag.py", line 1489, in get_certificate [Sun Sep 04 03:21:31 2016] [error] parse_result = self.get_parse_result_xml(http_body, parse_display_cert_xml) [Sun Sep 04 03:21:31 2016] [error] File "/usr/lib/python2.6/site-packages/ipaserver/plugins/dogtag.py", line 1350, in get_parse_result_xml [Sun Sep 04 03:21:31 2016] [error] doc = etree.fromstring(xml_text, parser) [Sun Sep 04 03:21:31 2016] [error] File "lxml.etree.pyx", line 2532, in lxml.etree.fromstring (src/lxml/lxml.etree.c:48270) [Sun Sep 04 03:21:31 2016] [error] File "parser.pxi", line 1545, in lxml.etree._parseMemoryDocument (src/lxml/lxml.etree.c:71812) [Sun Sep 04 03:21:31 2016] [error] File "parser.pxi", line 1424, in lxml.etree._parseDoc (src/lxml/lxml.etree.c:70673) [Sun Sep 04 03:21:31 2016] [error] File "parser.pxi", line 938, in lxml.etree._BaseParser._parseDoc (src/lxml/lxml.etree.c:67442) [Sun Sep 04 03:21:31 2016] [error] File "parser.pxi", line 539, in lxml.etree._ParserContext._handleParseResultDoc (src/lxml/lxml.etree.c:63824) [Sun Sep 04 03:21:31 2016] [error] File "parser.pxi", line 625, in lxml.etree._handleParseResult (src/lxml/lxml.etree.c:64745) [Sun Sep 04 03:21:31 2016] [error] File "parser.pxi", line 576, in lxml.etree._raiseParseError (src/lxml/lxml.etree.c:64260) [Sun Sep 04 03:21:31 2016] [error] XMLSyntaxError: None restarting httpd fixed the issue. Thanks! Looking into apache never occurred to me, freeipa really is a web service although it provides infrastructure services. -- Groeten, natxo -------------- next part -------------- An HTML attachment was scrubbed... URL: From rcritten at redhat.com Wed Sep 7 13:59:17 2016 From: rcritten at redhat.com (Rob Crittenden) Date: Wed, 7 Sep 2016 09:59:17 -0400 Subject: [Freeipa-users] ipa: ERROR: Certificate format error: (SEC_ERROR_LEGACY_DATABASE) The certificate/key database is in an old, unsupported format. In-Reply-To: References: <57D015D8.7080507@redhat.com> Message-ID: <57D01D35.1050309@redhat.com> Natxo Asenjo wrote: > > > On Wed, Sep 7, 2016 at 3:27 PM, Rob Crittenden > wrote: > > Natxo Asenjo wrote: > > hi, > > using centos 6.8 (server and client), when trying to view some > hosts we > get this error: > > > $ ipa host-find host-1920.sub.domain.tld > ipa: ERROR: Certificate format error: > (SEC_ERROR_LEGACY_DATABASE) The > certificate/key database is in an old, unsupported format. > > > I saw a thread last year about this, but no solution. > > Any clues? > > > /var/log/httpd/error_log may contain a traceback > > > This made me take a look at a replica and there I could not replicate > the error, I got the info I requested. > > In the apache error file I saw indeed a traceback: > > [Sun Sep 04 03:21:31 2016] [error] ipa: ERROR: non-public: > XMLSyntaxError: None > [Sun Sep 04 03:21:31 2016] [error] Traceback (most recent call last): > [Sun Sep 04 03:21:31 2016] [error] File > "/usr/lib/python2.6/site-packages/ipaserver/rpcserver.py", line 334, in > wsgi_execute > [Sun Sep 04 03:21:31 2016] [error] result = > self.Command[name](*args, **options) > [Sun Sep 04 03:21:31 2016] [error] File > "/usr/lib/python2.6/site-packages/ipalib/frontend.py", line 438, in __call__ > [Sun Sep 04 03:21:31 2016] [error] ret = self.run(*args, **options) > [Sun Sep 04 03:21:31 2016] [error] File > "/usr/lib/python2.6/site-packages/ipalib/frontend.py", line 750, in run > [Sun Sep 04 03:21:31 2016] [error] return self.execute(*args, **options) > [Sun Sep 04 03:21:31 2016] [error] File > "/usr/lib/python2.6/site-packages/ipalib/plugins/cert.py", line 362, in > execute > [Sun Sep 04 03:21:31 2016] [error] result = > api.Command['cert_show'](unicode(serial))['result'] > [Sun Sep 04 03:21:31 2016] [error] File > "/usr/lib/python2.6/site-packages/ipalib/frontend.py", line 438, in __call__ > [Sun Sep 04 03:21:31 2016] [error] ret = self.run(*args, **options) > [Sun Sep 04 03:21:31 2016] [error] File > "/usr/lib/python2.6/site-packages/ipalib/frontend.py", line 750, in run > [Sun Sep 04 03:21:31 2016] [error] return self.execute(*args, **options) > [Sun Sep 04 03:21:31 2016] [error] File > "/usr/lib/python2.6/site-packages/ipalib/plugins/cert.py", line 493, in > execute > [Sun Sep 04 03:21:31 2016] [error] > result=self.Backend.ra.get_certificate(serial_number) > [Sun Sep 04 03:21:31 2016] [error] File > "/usr/lib/python2.6/site-packages/ipaserver/plugins/dogtag.py", line > 1489, in get_certificate > [Sun Sep 04 03:21:31 2016] [error] parse_result = > self.get_parse_result_xml(http_body, parse_display_cert_xml) > [Sun Sep 04 03:21:31 2016] [error] File > "/usr/lib/python2.6/site-packages/ipaserver/plugins/dogtag.py", line > 1350, in get_parse_result_xml > [Sun Sep 04 03:21:31 2016] [error] doc = etree.fromstring(xml_text, > parser) > [Sun Sep 04 03:21:31 2016] [error] File "lxml.etree.pyx", line 2532, > in lxml.etree.fromstring (src/lxml/lxml.etree.c:48270) > [Sun Sep 04 03:21:31 2016] [error] File "parser.pxi", line 1545, in > lxml.etree._parseMemoryDocument (src/lxml/lxml.etree.c:71812) > [Sun Sep 04 03:21:31 2016] [error] File "parser.pxi", line 1424, in > lxml.etree._parseDoc (src/lxml/lxml.etree.c:70673) > [Sun Sep 04 03:21:31 2016] [error] File "parser.pxi", line 938, in > lxml.etree._BaseParser._parseDoc (src/lxml/lxml.etree.c:67442) > [Sun Sep 04 03:21:31 2016] [error] File "parser.pxi", line 539, in > lxml.etree._ParserContext._handleParseResultDoc > (src/lxml/lxml.etree.c:63824) > [Sun Sep 04 03:21:31 2016] [error] File "parser.pxi", line 625, in > lxml.etree._handleParseResult (src/lxml/lxml.etree.c:64745) > [Sun Sep 04 03:21:31 2016] [error] File "parser.pxi", line 576, in > lxml.etree._raiseParseError (src/lxml/lxml.etree.c:64260) > [Sun Sep 04 03:21:31 2016] [error] XMLSyntaxError: None > > > restarting httpd fixed the issue. Thanks! > > Looking into apache never occurred to me, freeipa really is a web > service although it provides infrastructure services. Yeah, there are a lot of moving parts, that's for sure. Makes me wonder if httpd should be restarted as part of the upgrade. rob From natxo.asenjo at gmail.com Wed Sep 7 14:01:02 2016 From: natxo.asenjo at gmail.com (Natxo Asenjo) Date: Wed, 7 Sep 2016 16:01:02 +0200 Subject: [Freeipa-users] ipa: ERROR: Certificate format error: (SEC_ERROR_LEGACY_DATABASE) The certificate/key database is in an old, unsupported format. In-Reply-To: References: <57D015D8.7080507@redhat.com> Message-ID: alas, not woriking again. On the one kdc $ ipa host-find tftp-1801 ipa: ERROR: Certificate format error: (SEC_ERROR_LEGACY_DATABASE) The certificate/key database is in an old, unsupported format. On the other: $ ipa host-find tftp-1801 -------------- 1 host matched -------------- Host name: tftp-1801.sub.domain.tld ..... After rebooting the kdc with the error, no new tracebacks in the error_log Strange -- -- Groeten, natxo -------------- next part -------------- An HTML attachment was scrubbed... URL: From natxo.asenjo at gmail.com Wed Sep 7 14:03:29 2016 From: natxo.asenjo at gmail.com (Natxo Asenjo) Date: Wed, 7 Sep 2016 16:03:29 +0200 Subject: [Freeipa-users] ipa: ERROR: Certificate format error: (SEC_ERROR_LEGACY_DATABASE) The certificate/key database is in an old, unsupported format. In-Reply-To: References: <57D015D8.7080507@redhat.com> Message-ID: I do see these errors: [Wed Sep 07 15:56:13 2016] [error] ipa: INFO:: ping(): SUCCESS [Wed Sep 07 15:56:13 2016] [error] ipa: INFO: : host_find(u'tftp-1801', all=False, raw=False, version=u'2.49', no_members=False, pkey_only=False): CertificateFormatError [Wed Sep 07 15:56:44 2016] [error] ipa: INFO: : ping(): SUCCESS [Wed Sep 07 15:56:44 2016] [error] ipa: INFO: : host_find(u'tftp-1801', all=False, raw=False, version=u'2.49', no_members=False, pkey_only=False): CertificateFormatError [Wed Sep 07 15:57:57 2016] [error] ipa: INFO: : ping(): SUCCESS [Wed Sep 07 15:57:58 2016] [error] ipa: INFO: : host_find(u'tftp-1801', all=False, raw=False, version=u'2.49', no_members=False, pkey_only=False): CertificateFormatErro On Wed, Sep 7, 2016 at 4:01 PM, Natxo Asenjo wrote: > > alas, not woriking again. > > On the one kdc > > $ ipa host-find tftp-1801 > ipa: ERROR: Certificate format error: (SEC_ERROR_LEGACY_DATABASE) The > certificate/key database is in an old, unsupported format. > > On the other: > > $ ipa host-find tftp-1801 > -------------- > 1 host matched > -------------- > Host name: tftp-1801.sub.domain.tld > ..... > > After rebooting the kdc with the error, no new tracebacks in the error_log > > Strange > > -- > -- > Groeten, > natxo > -- -- Groeten, natxo -------------- next part -------------- An HTML attachment was scrubbed... URL: From venkataramana.kintali at gmail.com Wed Sep 7 14:39:50 2016 From: venkataramana.kintali at gmail.com (Venkataramana Kintali) Date: Wed, 7 Sep 2016 17:39:50 +0300 Subject: [Freeipa-users] Issues with FreeIPA SSH Key authentication Message-ID: Hi, Of late, I am learning FreeIPA . I have installed IPA server and few clients (Version 3.0.0) I am facing an issue with ssh key authentication in my setup. I generated a putty ssh private key (using putty keygen) ,and uploaded it under a user through IPA GUI. I am able to login to some IPA clients but not able to login to other IPA clients with putty using private key and passphrase. Public Key Authentication is enabled on all clients. I am able to from one client to other clients successfully (after doing kinit) without promting password. Can someone please throw some light on this as to what the issue could be here and what else I can check to understand where the problem is ? I searched this online but couldn't find any solution in the context of IPA. Thanks Venkataramana -------------- next part -------------- An HTML attachment was scrubbed... URL: From venkataramana.kintali at gmail.com Wed Sep 7 14:46:09 2016 From: venkataramana.kintali at gmail.com (Venkataramana Kintali) Date: Wed, 7 Sep 2016 17:46:09 +0300 Subject: [Freeipa-users] Issues with FreeIPA SSH Key authentication In-Reply-To: References: Message-ID: On Sep 7, 2016 8:09 PM, "Venkataramana Kintali" < venkataramana.kintali at gmail.com> wrote: > > Hi, > Of late, I am learning FreeIPA . I have installed IPA server and few clients (Version 3.0.0) > I am facing an issue with ssh key authentication in my setup. > I generated a putty ssh private key (using putty keygen) ,and uploaded it under a user through IPA GUI. > I am able to login to some IPA clients but not able to login to other IPA clients with putty using private key and passphrase. I forgot to mention the error . I am getting "server refused our key" for the servers I am unable to login to. > > Public Key Authentication is enabled on all clients. > I am able to from one client to other clients successfully (after doing kinit) without promting password. > > Can someone please throw some light on this as to what the issue could be here and what else I can check to understand where the problem is ? > > I searched this online but couldn't find any solution in the context of IPA. > > > Thanks > Venkataramana > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From APtashnik at cccis.com Wed Sep 7 18:36:57 2016 From: APtashnik at cccis.com (Andrey Ptashnik) Date: Wed, 7 Sep 2016 18:36:57 +0000 Subject: [Freeipa-users] Higher client versions joining 4.2.0 IPA cluster Message-ID: <5AB78C74-18A4-477B-B8A6-B4805D825E61@cccis.com> Hello FreeIPA team, Our FreeIPA server cluster is at version 4.2.0 and expecting Ubuntu 16 machines with FreeIPA client software 4.3.1 soon to join our IPA domain. Are there any compatibility issues that we may encounter? Regards, Andrey -------------- next part -------------- An HTML attachment was scrubbed... URL: From APtashnik at cccis.com Wed Sep 7 18:42:44 2016 From: APtashnik at cccis.com (Andrey Ptashnik) Date: Wed, 7 Sep 2016 18:42:44 +0000 Subject: [Freeipa-users] Higher client versions joining 4.2.0 IPA cluster In-Reply-To: <5AB78C74-18A4-477B-B8A6-B4805D825E61@cccis.com> References: <5AB78C74-18A4-477B-B8A6-B4805D825E61@cccis.com> Message-ID: Hello FreeIPA team, Our FreeIPA server cluster is at version 4.2.0 and expecting Ubuntu 16 machines with FreeIPA client software 4.3.1 soon to join our IPA domain. Are there any compatibility issues that we may encounter? Regards, Andrey From rcritten at redhat.com Wed Sep 7 18:48:01 2016 From: rcritten at redhat.com (Rob Crittenden) Date: Wed, 7 Sep 2016 14:48:01 -0400 Subject: [Freeipa-users] Higher client versions joining 4.2.0 IPA cluster In-Reply-To: <5AB78C74-18A4-477B-B8A6-B4805D825E61@cccis.com> References: <5AB78C74-18A4-477B-B8A6-B4805D825E61@cccis.com> Message-ID: <57D060E1.40700@redhat.com> Andrey Ptashnik wrote: > Hello FreeIPA team, > > Our FreeIPA server cluster is at version 4.2.0 and expecting Ubuntu 16 > machines with FreeIPA client software 4.3.1 soon to join our IPA domain. > Are there any compatibility issues that we may encounter? No, ipa-client-install should work around the API version differences. Using the ipa tool directly will have problems though. This is fixed in the recently released freeIPA 4.4.1. rob From APtashnik at cccis.com Wed Sep 7 19:01:52 2016 From: APtashnik at cccis.com (Andrey Ptashnik) Date: Wed, 7 Sep 2016 19:01:52 +0000 Subject: [Freeipa-users] Higher client versions joining 4.2.0 IPA cluster In-Reply-To: <57D060E1.40700@redhat.com> References: <5AB78C74-18A4-477B-B8A6-B4805D825E61@cccis.com> <57D060E1.40700@redhat.com> Message-ID: <8A969F2B-0274-480C-B20E-BBA8B0EC4FC7@cccis.com> Rob, Thank you for confirmation. We will be relying on LDAP/DNS functionality and pulling policies like Sudo and HBA from client machines. We are not looking to make domain wide changes using ipa tools from those Ubuntu client machines. I guess that should work, right? Regards, Andrey On 9/7/16, 1:48 PM, "Rob Crittenden" wrote: >Andrey Ptashnik wrote: >> Hello FreeIPA team, >> >> Our FreeIPA server cluster is at version 4.2.0 and expecting Ubuntu 16 >> machines with FreeIPA client software 4.3.1 soon to join our IPA domain. >> Are there any compatibility issues that we may encounter? > >No, ipa-client-install should work around the API version differences. >Using the ipa tool directly will have problems though. > >This is fixed in the recently released freeIPA 4.4.1. > >rob From rcritten at redhat.com Wed Sep 7 19:09:11 2016 From: rcritten at redhat.com (Rob Crittenden) Date: Wed, 7 Sep 2016 15:09:11 -0400 Subject: [Freeipa-users] Higher client versions joining 4.2.0 IPA cluster In-Reply-To: <8A969F2B-0274-480C-B20E-BBA8B0EC4FC7@cccis.com> References: <5AB78C74-18A4-477B-B8A6-B4805D825E61@cccis.com> <57D060E1.40700@redhat.com> <8A969F2B-0274-480C-B20E-BBA8B0EC4FC7@cccis.com> Message-ID: <57D065D7.9070401@redhat.com> Andrey Ptashnik wrote: > Rob, > > Thank you for confirmation. > We will be relying on LDAP/DNS functionality and pulling policies like Sudo and HBA from client machines. We are not looking to make domain wide changes using ipa tools from those Ubuntu client machines. > I guess that should work, right? Yeah, ipa-client-* and sssd should work fine, you probably won't notice any issues. Just the ipa tool would notice that the API versions are off. rob > > Regards, > > Andrey > > > > > > > On 9/7/16, 1:48 PM, "Rob Crittenden" wrote: > >> Andrey Ptashnik wrote: >>> Hello FreeIPA team, >>> >>> Our FreeIPA server cluster is at version 4.2.0 and expecting Ubuntu 16 >>> machines with FreeIPA client software 4.3.1 soon to join our IPA domain. >>> Are there any compatibility issues that we may encounter? >> >> No, ipa-client-install should work around the API version differences. >> Using the ipa tool directly will have problems though. >> >> This is fixed in the recently released freeIPA 4.4.1. >> >> rob From rakesh.rajasekharan at gmail.com Wed Sep 7 19:34:28 2016 From: rakesh.rajasekharan at gmail.com (Rakesh Rajasekharan) Date: Thu, 8 Sep 2016 01:04:28 +0530 Subject: [Freeipa-users] Freeipa 4.2.0 hangs intermittently In-Reply-To: References: <57BEB0AB.2090506@redhat.com> <57C44AC6.1030402@redhat.com> <57CD1FDC.2010109@redhat.com> <57CD5488.60404@redhat.com> <57CD659B.6050707@redhat.com> <57CD68BD.9030802@redhat.com> <57CE8CDB.6000809@redhat.com> <57CE9DE5.6020301@redhat.com> Message-ID: I feel since I migrated from my earlier openldap.. There might be some issues with the indexes.. So may be creating fresh users might help.. Is there a way to read clear text user password.. That way I could have just recreated the users and set them the older password... Just trying what ever I could think of to fix this :) On Sep 7, 2016 5:16 PM, "Rakesh Rajasekharan" wrote: I changed the nsslapd-db-checkpoint interval first to 80 and increased till 160.. however the hang has not gone away yet I migrated my data from a openldap, can that cause issues like this.. If so, i could try creating users from scratch directly on IPA rather than migrating and test it But before that , have taken a fresh pstack just incase if it helps to nail down the issue Thread 44 (Thread 0x7f53ed52f700 (LWP 128165)): #0 0x00007f54047649b3 in select () from /lib64/libc.so.6 #1 0x00007f5406eef0e9 in DS_Sleep () from /usr/lib64/dirsrv/libslapd.so.0 #2 0x00007f53fa53d907 in deadlock_threadmain () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #3 0x00007f540509e7bb in _pt_root () from /lib64/libnspr4.so #4 0x00007f5404a3fdc5 in start_thread () from /lib64/libpthread.so.0 #5 0x00007f540476d28d in clone () from /lib64/libc.so.6 Thread 43 (Thread 0x7f53ecd2e700 (LWP 128166)): #0 0x00007f54047649b3 in select () from /lib64/libc.so.6 #1 0x00007f53ff0cbbad in __os_yield () from /lib64/libdb-5.3.so #2 0x00007f53ff0c72b3 in __memp_sync_int () from /lib64/libdb-5.3.so #3 0x00007f53ff0d7752 in __txn_checkpoint () from /lib64/libdb-5.3.so #4 0x00007f53ff0d7b74 in __txn_checkpoint_pp () from /lib64/libdb-5.3.so #5 0x00007f53fa541a87 in checkpoint_threadmain () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #6 0x00007f540509e7bb in _pt_root () from /lib64/libnspr4.so #7 0x00007f5404a3fdc5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007f540476d28d in clone () from /lib64/libc.so.6 Thread 42 (Thread 0x7f53ec52d700 (LWP 128167)): #0 0x00007f54047649b3 in select () from /lib64/libc.so.6 #1 0x00007f5406eef0e9 in DS_Sleep () from /usr/lib64/dirsrv/libslapd.so.0 #2 0x00007f53fa53db7f in trickle_threadmain () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #3 0x00007f540509e7bb in _pt_root () from /lib64/libnspr4.so #4 0x00007f5404a3fdc5 in start_thread () from /lib64/libpthread.so.0 #5 0x00007f540476d28d in clone () from /lib64/libc.so.6 Thread 41 (Thread 0x7f53ebd2c700 (LWP 128168)): #0 0x00007f54047649b3 in select () from /lib64/libc.so.6 #1 0x00007f5406eef0e9 in DS_Sleep () from /usr/lib64/dirsrv/libslapd.so.0 #2 0x00007f53fa538707 in perf_threadmain () from /usr/lib64/dirsrv/plugins/ libback-ldbm.so #3 0x00007f540509e7bb in _pt_root () from /lib64/libnspr4.so #4 0x00007f5404a3fdc5 in start_thread () from /lib64/libpthread.so.0 #5 0x00007f540476d28d in clone () from /lib64/libc.so.6 Thread 40 (Thread 0x7f53eb322700 (LWP 128204)): #0 0x00007f5404a436d5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f5405099050 in PR_WaitCondVar () from /lib64/libnspr4.so #2 0x00007f5406ede198 in slapi_wait_condvar () from /usr/lib64/dirsrv/libslapd.so.0 #3 0x00007f53fd27062e in cos_cache_wait_on_change () from /usr/lib64/dirsrv/plugins/libcos-plugin.so #4 0x00007f540509e7bb in _pt_root () from /lib64/libnspr4.so #5 0x00007f5404a3fdc5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f540476d28d in clone () from /lib64/libc.so.6 Thread 39 (Thread 0x7f53eab21700 (LWP 128206)): #0 0x00007f5404a436d5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f5405099050 in PR_WaitCondVar () from /lib64/libnspr4.so #2 0x00007f5406ede198 in slapi_wait_condvar () from /usr/lib64/dirsrv/libslapd.so.0 #3 0x00007f53f8bdbeed in roles_cache_wait_on_change () from /usr/lib64/dirsrv/plugins/libroles-plugin.so #4 0x00007f540509e7bb in _pt_root () from /lib64/libnspr4.so #5 0x00007f5404a3fdc5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f540476d28d in clone () from /lib64/libc.so.6 Thread 38 (Thread 0x7f53ea320700 (LWP 128207)): #0 0x00007f5404a436d5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f5405099050 in PR_WaitCondVar () from /lib64/libnspr4.so #2 0x00007f5406ede198 in slapi_wait_condvar () from /usr/lib64/dirsrv/libslapd.so.0 #3 0x00007f53f8bdbeed in roles_cache_wait_on_change () from /usr/lib64/dirsrv/plugins/libroles-plugin.so #4 0x00007f540509e7bb in _pt_root () from /lib64/libnspr4.so #5 0x00007f5404a3fdc5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f540476d28d in clone () from /lib64/libc.so.6 Thread 37 (Thread 0x7f53e9b1f700 (LWP 128208)): #0 0x00007f5404a436d5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f5405099050 in PR_WaitCondVar () from /lib64/libnspr4.so #2 0x00007f5406ede198 in slapi_wait_condvar () from /usr/lib64/dirsrv/libslapd.so.0 #3 0x00007f53f8bdbeed in roles_cache_wait_on_change () from /usr/lib64/dirsrv/plugins/libroles-plugin.so #4 0x00007f540509e7bb in _pt_root () from /lib64/libnspr4.so #5 0x00007f5404a3fdc5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f540476d28d in clone () from /lib64/libc.so.6 Thread 36 (Thread 0x7f53e931e700 (LWP 128209)): #0 0x00007f5404a43a82 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f5405098b07 in pt_TimedWait () from /lib64/libnspr4.so #2 0x00007f5405098fce in PR_WaitCondVar () from /lib64/libnspr4.so #3 0x00007f5407397d03 in housecleaning () #4 0x00007f540509e7bb in _pt_root () from /lib64/libnspr4.so #5 0x00007f5404a3fdc5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f540476d28d in clone () from /lib64/libc.so.6 Thread 35 (Thread 0x7f53e8b1d700 (LWP 128210)): #0 0x00007f5404a43a82 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f5405098b07 in pt_TimedWait () from /lib64/libnspr4.so #2 0x00007f5405098fce in PR_WaitCondVar () from /lib64/libnspr4.so #3 0x00007f5406e841e8 in eq_loop () from /usr/lib64/dirsrv/libslapd.so.0 #4 0x00007f540509e7bb in _pt_root () from /lib64/libnspr4.so #5 0x00007f5404a3fdc5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f540476d28d in clone () from /lib64/libc.so.6 Thread 34 (Thread 0x7f53db7fe700 (LWP 128212)): #0 0x00007f5404a436d5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f53fef9f2f3 in __db_hybrid_mutex_suspend () from /lib64/ libdb-5.3.so #2 0x00007f53fef9e640 in __db_tas_mutex_lock () from /lib64/libdb-5.3.so #3 0x00007f53ff048d3a in __lock_get_internal () from /lib64/libdb-5.3.so #4 0x00007f53ff049820 in __lock_get () from /lib64/libdb-5.3.so #5 0x00007f53ff075142 in __db_lget () from /lib64/libdb-5.3.so #6 0x00007f53fefbc605 in __bam_search () from /lib64/libdb-5.3.so #7 0x00007f53fefa7256 in __bamc_search () from /lib64/libdb-5.3.so #8 0x00007f53fefa8d0f in __bamc_get () from /lib64/libdb-5.3.so #9 0x00007f53ff061ca6 in __dbc_iget () from /lib64/libdb-5.3.so #10 0x00007f53ff070b02 in __dbc_get_pp () from /lib64/libdb-5.3.so #11 0x00007f53fa54c680 in idl_new_fetch () from /usr/lib64/dirsrv/plugins/ libback-ldbm.so #12 0x00007f53fa55ab26 in index_read_ext_allids () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #13 0x00007f53fa545284 in keys2idl () from /usr/lib64/dirsrv/plugins/ libback-ldbm.so #14 0x00007f53fa5459e3 in ava_candidates.isra.0 () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #15 0x00007f53fa545fd2 in filter_candidates_ext () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #16 0x00007f53fa547096 in list_candidates () from /usr/lib64/dirsrv/plugins/ libback-ldbm.so #17 0x00007f53fa545f40 in filter_candidates_ext () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #18 0x00007f53fa58180c in subtree_candidates () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #19 0x00007f53fa582e7f in ldbm_back_search () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #20 0x00007f5406eb01d6 in op_shared_search () from /usr/lib64/dirsrv/libslapd.so.0 #21 0x00007f54073a133e in do_search () #22 0x00007f540738f567 in connection_threadmain () #23 0x00007f540509e7bb in _pt_root () from /lib64/libnspr4.so #24 0x00007f5404a3fdc5 in start_thread () from /lib64/libpthread.so.0 #25 0x00007f540476d28d in clone () from /lib64/libc.so.6 Thread 33 (Thread 0x7f53daffd700 (LWP 128213)): #0 0x00007f5404a436d5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f53fef9f2f3 in __db_hybrid_mutex_suspend () from /lib64/ libdb-5.3.so #2 0x00007f53fef9e640 in __db_tas_mutex_lock () from /lib64/libdb-5.3.so #3 0x00007f53ff048d3a in __lock_get_internal () from /lib64/libdb-5.3.so #4 0x00007f53ff049820 in __lock_get () from /lib64/libdb-5.3.so #5 0x00007f53ff075142 in __db_lget () from /lib64/libdb-5.3.so #6 0x00007f53fefbc605 in __bam_search () from /lib64/libdb-5.3.so #7 0x00007f53fefa7256 in __bamc_search () from /lib64/libdb-5.3.so #8 0x00007f53fefa8d0f in __bamc_get () from /lib64/libdb-5.3.so #9 0x00007f53ff061ca6 in __dbc_iget () from /lib64/libdb-5.3.so #10 0x00007f53ff070b02 in __dbc_get_pp () from /lib64/libdb-5.3.so #11 0x00007f53fa54c680 in idl_new_fetch () from /usr/lib64/dirsrv/plugins/ libback-ldbm.so #12 0x00007f53fa55ab26 in index_read_ext_allids () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #13 0x00007f53fa545284 in keys2idl () from /usr/lib64/dirsrv/plugins/ libback-ldbm.so #14 0x00007f53fa5459e3 in ava_candidates.isra.0 () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #15 0x00007f53fa545fd2 in filter_candidates_ext () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #16 0x00007f53fa547096 in list_candidates () from /usr/lib64/dirsrv/plugins/ libback-ldbm.so #17 0x00007f53fa545f40 in filter_candidates_ext () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #18 0x00007f53fa58180c in subtree_candidates () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #19 0x00007f53fa582e7f in ldbm_back_search () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #20 0x00007f5406eb01d6 in op_shared_search () from /usr/lib64/dirsrv/libslapd.so.0 #21 0x00007f54073a133e in do_search () #22 0x00007f540738f567 in connection_threadmain () #23 0x00007f540509e7bb in _pt_root () from /lib64/libnspr4.so #24 0x00007f5404a3fdc5 in start_thread () from /lib64/libpthread.so.0 #25 0x00007f540476d28d in clone () from /lib64/libc.so.6 Thread 32 (Thread 0x7f53da7fc700 (LWP 128214)): #0 0x00007f5404a436d5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f53fef9f2f3 in __db_hybrid_mutex_suspend () from /lib64/ libdb-5.3.so #2 0x00007f53fef9e640 in __db_tas_mutex_lock () from /lib64/libdb-5.3.so #3 0x00007f53ff048d3a in __lock_get_internal () from /lib64/libdb-5.3.so #4 0x00007f53ff049820 in __lock_get () from /lib64/libdb-5.3.so #5 0x00007f53ff075142 in __db_lget () from /lib64/libdb-5.3.so #6 0x00007f53fefbc605 in __bam_search () from /lib64/libdb-5.3.so #7 0x00007f53fefa7256 in __bamc_search () from /lib64/libdb-5.3.so #8 0x00007f53fefa8d0f in __bamc_get () from /lib64/libdb-5.3.so #9 0x00007f53ff061ca6 in __dbc_iget () from /lib64/libdb-5.3.so #10 0x00007f53ff070b02 in __dbc_get_pp () from /lib64/libdb-5.3.so #11 0x00007f53fa54c680 in idl_new_fetch () from /usr/lib64/dirsrv/plugins/ libback-ldbm.so #12 0x00007f53fa55ab26 in index_read_ext_allids () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #13 0x00007f53fa545284 in keys2idl () from /usr/lib64/dirsrv/plugins/ libback-ldbm.so #14 0x00007f53fa5459e3 in ava_candidates.isra.0 () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #15 0x00007f53fa545fd2 in filter_candidates_ext () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #16 0x00007f53fa547096 in list_candidates () from /usr/lib64/dirsrv/plugins/ libback-ldbm.so #17 0x00007f53fa545f40 in filter_candidates_ext () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #18 0x00007f53fa547096 in list_candidates () from /usr/lib64/dirsrv/plugins/ libback-ldbm.so #19 0x00007f53fa545f40 in filter_candidates_ext () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #20 0x00007f53fa547096 in list_candidates () from /usr/lib64/dirsrv/plugins/ libback-ldbm.so #21 0x00007f53fa545f40 in filter_candidates_ext () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #22 0x00007f53fa58180c in subtree_candidates () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #23 0x00007f53fa582e7f in ldbm_back_search () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #24 0x00007f5406eb01d6 in op_shared_search () from /usr/lib64/dirsrv/libslapd.so.0 #25 0x00007f54073a133e in do_search () #26 0x00007f540738f3c3 in connection_threadmain () #27 0x00007f540509e7bb in _pt_root () from /lib64/libnspr4.so #28 0x00007f5404a3fdc5 in start_thread () from /lib64/libpthread.so.0 #29 0x00007f540476d28d in clone () from /lib64/libc.so.6 Thread 31 (Thread 0x7f53d9ffb700 (LWP 128215)): #0 0x00007f5404a436d5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f53fef9f2f3 in __db_hybrid_mutex_suspend () from /lib64/ libdb-5.3.so #2 0x00007f53fef9e640 in __db_tas_mutex_lock () from /lib64/libdb-5.3.so #3 0x00007f53ff048d3a in __lock_get_internal () from /lib64/libdb-5.3.so #4 0x00007f53ff049820 in __lock_get () from /lib64/libdb-5.3.so #5 0x00007f53ff075142 in __db_lget () from /lib64/libdb-5.3.so #6 0x00007f53fefbc605 in __bam_search () from /lib64/libdb-5.3.so #7 0x00007f53fefa7256 in __bamc_search () from /lib64/libdb-5.3.so #8 0x00007f53fefa8d0f in __bamc_get () from /lib64/libdb-5.3.so #9 0x00007f53ff061ca6 in __dbc_iget () from /lib64/libdb-5.3.so #10 0x00007f53ff070b02 in __dbc_get_pp () from /lib64/libdb-5.3.so #11 0x00007f53fa54c680 in idl_new_fetch () from /usr/lib64/dirsrv/plugins/ libback-ldbm.so #12 0x00007f53fa55ab26 in index_read_ext_allids () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #13 0x00007f53fa545284 in keys2idl () from /usr/lib64/dirsrv/plugins/ libback-ldbm.so #14 0x00007f53fa5459e3 in ava_candidates.isra.0 () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #15 0x00007f53fa545fd2 in filter_candidates_ext () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #16 0x00007f53fa547096 in list_candidates () from /usr/lib64/dirsrv/plugins/ libback-ldbm.so #17 0x00007f53fa545f40 in filter_candidates_ext () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #18 0x00007f53fa58180c in subtree_candidates () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #19 0x00007f53fa582e7f in ldbm_back_search () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #20 0x00007f5406eb01d6 in op_shared_search () from /usr/lib64/dirsrv/libslapd.so.0 #21 0x00007f54073a133e in do_search () #22 0x00007f540738f567 in connection_threadmain () #23 0x00007f540509e7bb in _pt_root () from /lib64/libnspr4.so #24 0x00007f5404a3fdc5 in start_thread () from /lib64/libpthread.so.0 #25 0x00007f540476d28d in clone () from /lib64/libc.so.6 Thread 30 (Thread 0x7f53d97fa700 (LWP 128216)): #0 0x00007f5404a436d5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f53fef9f2f3 in __db_hybrid_mutex_suspend () from /lib64/ libdb-5.3.so #2 0x00007f53fef9e640 in __db_tas_mutex_lock () from /lib64/libdb-5.3.so #3 0x00007f53ff048d3a in __lock_get_internal () from /lib64/libdb-5.3.so #4 0x00007f53ff049820 in __lock_get () from /lib64/libdb-5.3.so #5 0x00007f53ff075142 in __db_lget () from /lib64/libdb-5.3.so #6 0x00007f53fefbc605 in __bam_search () from /lib64/libdb-5.3.so #7 0x00007f53fefa7256 in __bamc_search () from /lib64/libdb-5.3.so #8 0x00007f53fefa8d0f in __bamc_get () from /lib64/libdb-5.3.so #9 0x00007f53ff061ca6 in __dbc_iget () from /lib64/libdb-5.3.so #10 0x00007f53ff070b02 in __dbc_get_pp () from /lib64/libdb-5.3.so #11 0x00007f53fa54c680 in idl_new_fetch () from /usr/lib64/dirsrv/plugins/ libback-ldbm.so #12 0x00007f53fa55ab26 in index_read_ext_allids () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #13 0x00007f53fa545284 in keys2idl () from /usr/lib64/dirsrv/plugins/ libback-ldbm.so #14 0x00007f53fa5459e3 in ava_candidates.isra.0 () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #15 0x00007f53fa545fd2 in filter_candidates_ext () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #16 0x00007f53fa547096 in list_candidates () from /usr/lib64/dirsrv/plugins/ libback-ldbm.so #17 0x00007f53fa545f40 in filter_candidates_ext () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #18 0x00007f53fa58180c in subtree_candidates () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #19 0x00007f53fa582e7f in ldbm_back_search () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #20 0x00007f5406eb01d6 in op_shared_search () from /usr/lib64/dirsrv/libslapd.so.0 #21 0x00007f54073a133e in do_search () #22 0x00007f540738f567 in connection_threadmain () #23 0x00007f540509e7bb in _pt_root () from /lib64/libnspr4.so #24 0x00007f5404a3fdc5 in start_thread () from /lib64/libpthread.so.0 #25 0x00007f540476d28d in clone () from /lib64/libc.so.6 Thread 29 (Thread 0x7f53d8ff9700 (LWP 128217)): #0 0x00007f5404a436d5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f53fef9f2f3 in __db_hybrid_mutex_suspend () from /lib64/ libdb-5.3.so #2 0x00007f53fef9e640 in __db_tas_mutex_lock () from /lib64/libdb-5.3.so #3 0x00007f53ff048d3a in __lock_get_internal () from /lib64/libdb-5.3.so #4 0x00007f53ff049820 in __lock_get () from /lib64/libdb-5.3.so #5 0x00007f53ff075142 in __db_lget () from /lib64/libdb-5.3.so #6 0x00007f53fefbc605 in __bam_search () from /lib64/libdb-5.3.so #7 0x00007f53fefa7256 in __bamc_search () from /lib64/libdb-5.3.so #8 0x00007f53fefa8d0f in __bamc_get () from /lib64/libdb-5.3.so #9 0x00007f53ff061ca6 in __dbc_iget () from /lib64/libdb-5.3.so #10 0x00007f53ff070b02 in __dbc_get_pp () from /lib64/libdb-5.3.so #11 0x00007f53fa54c680 in idl_new_fetch () from /usr/lib64/dirsrv/plugins/ libback-ldbm.so #12 0x00007f53fa55ab26 in index_read_ext_allids () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #13 0x00007f53fa545284 in keys2idl () from /usr/lib64/dirsrv/plugins/ libback-ldbm.so #14 0x00007f53fa5459e3 in ava_candidates.isra.0 () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #15 0x00007f53fa545fd2 in filter_candidates_ext () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #16 0x00007f53fa547096 in list_candidates () from /usr/lib64/dirsrv/plugins/ libback-ldbm.so #17 0x00007f53fa545f40 in filter_candidates_ext () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #18 0x00007f53fa58180c in subtree_candidates () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #19 0x00007f53fa582e7f in ldbm_back_search () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #20 0x00007f5406eb01d6 in op_shared_search () from /usr/lib64/dirsrv/libslapd.so.0 #21 0x00007f54073a133e in do_search () #22 0x00007f540738f567 in connection_threadmain () #23 0x00007f540509e7bb in _pt_root () from /lib64/libnspr4.so #24 0x00007f5404a3fdc5 in start_thread () from /lib64/libpthread.so.0 #25 0x00007f540476d28d in clone () from /lib64/libc.so.6 Thread 28 (Thread 0x7f53d87f8700 (LWP 128218)): #0 0x00007f5404a436d5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f53fef9f2f3 in __db_hybrid_mutex_suspend () from /lib64/ libdb-5.3.so #2 0x00007f53fef9e640 in __db_tas_mutex_lock () from /lib64/libdb-5.3.so #3 0x00007f53ff048d3a in __lock_get_internal () from /lib64/libdb-5.3.so #4 0x00007f53ff049820 in __lock_get () from /lib64/libdb-5.3.so #5 0x00007f53ff075142 in __db_lget () from /lib64/libdb-5.3.so #6 0x00007f53fefbc605 in __bam_search () from /lib64/libdb-5.3.so #7 0x00007f53fefa7256 in __bamc_search () from /lib64/libdb-5.3.so #8 0x00007f53fefa8d0f in __bamc_get () from /lib64/libdb-5.3.so #9 0x00007f53ff061ca6 in __dbc_iget () from /lib64/libdb-5.3.so #10 0x00007f53ff070b02 in __dbc_get_pp () from /lib64/libdb-5.3.so #11 0x00007f53fa54c680 in idl_new_fetch () from /usr/lib64/dirsrv/plugins/ libback-ldbm.so #12 0x00007f53fa55ab26 in index_read_ext_allids () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #13 0x00007f53fa545284 in keys2idl () from /usr/lib64/dirsrv/plugins/ libback-ldbm.so #14 0x00007f53fa5459e3 in ava_candidates.isra.0 () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #15 0x00007f53fa545fd2 in filter_candidates_ext () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #16 0x00007f53fa547096 in list_candidates () from /usr/lib64/dirsrv/plugins/ libback-ldbm.so #17 0x00007f53fa545f40 in filter_candidates_ext () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #18 0x00007f53fa58180c in subtree_candidates () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #19 0x00007f53fa582e7f in ldbm_back_search () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #20 0x00007f5406eb01d6 in op_shared_search () from /usr/lib64/dirsrv/libslapd.so.0 #21 0x00007f54073a133e in do_search () #22 0x00007f540738f567 in connection_threadmain () #23 0x00007f540509e7bb in _pt_root () from /lib64/libnspr4.so #24 0x00007f5404a3fdc5 in start_thread () from /lib64/libpthread.so.0 #25 0x00007f540476d28d in clone () from /lib64/libc.so.6 Thread 27 (Thread 0x7f53d7ff7700 (LWP 128219)): #0 0x00007f5404a436d5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f53fef9f2f3 in __db_hybrid_mutex_suspend () from /lib64/ libdb-5.3.so #2 0x00007f53fef9e640 in __db_tas_mutex_lock () from /lib64/libdb-5.3.so #3 0x00007f53ff048d3a in __lock_get_internal () from /lib64/libdb-5.3.so #4 0x00007f53ff049820 in __lock_get () from /lib64/libdb-5.3.so #5 0x00007f53ff075142 in __db_lget () from /lib64/libdb-5.3.so #6 0x00007f53fefbc605 in __bam_search () from /lib64/libdb-5.3.so #7 0x00007f53fefa7256 in __bamc_search () from /lib64/libdb-5.3.so #8 0x00007f53fefa8d0f in __bamc_get () from /lib64/libdb-5.3.so #9 0x00007f53ff061ca6 in __dbc_iget () from /lib64/libdb-5.3.so #10 0x00007f53ff070b02 in __dbc_get_pp () from /lib64/libdb-5.3.so #11 0x00007f53fa54c680 in idl_new_fetch () from /usr/lib64/dirsrv/plugins/ libback-ldbm.so #12 0x00007f53fa55ab26 in index_read_ext_allids () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #13 0x00007f53fa545284 in keys2idl () from /usr/lib64/dirsrv/plugins/ libback-ldbm.so #14 0x00007f53fa5459e3 in ava_candidates.isra.0 () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #15 0x00007f53fa545fd2 in filter_candidates_ext () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #16 0x00007f53fa547096 in list_candidates () from /usr/lib64/dirsrv/plugins/ libback-ldbm.so #17 0x00007f53fa545f40 in filter_candidates_ext () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #18 0x00007f53fa58180c in subtree_candidates () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #19 0x00007f53fa582e7f in ldbm_back_search () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #20 0x00007f5406eb01d6 in op_shared_search () from /usr/lib64/dirsrv/libslapd.so.0 #21 0x00007f54073a133e in do_search () #22 0x00007f540738f567 in connection_threadmain () #23 0x00007f540509e7bb in _pt_root () from /lib64/libnspr4.so #24 0x00007f5404a3fdc5 in start_thread () from /lib64/libpthread.so.0 #25 0x00007f540476d28d in clone () from /lib64/libc.so.6 Thread 26 (Thread 0x7f53d77f6700 (LWP 128220)): #0 0x00007f5404a436d5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f53fef9f2f3 in __db_hybrid_mutex_suspend () from /lib64/ libdb-5.3.so #2 0x00007f53fef9e640 in __db_tas_mutex_lock () from /lib64/libdb-5.3.so #3 0x00007f53ff048d3a in __lock_get_internal () from /lib64/libdb-5.3.so #4 0x00007f53ff049820 in __lock_get () from /lib64/libdb-5.3.so #5 0x00007f53ff075142 in __db_lget () from /lib64/libdb-5.3.so #6 0x00007f53fefbc605 in __bam_search () from /lib64/libdb-5.3.so #7 0x00007f53fefa7256 in __bamc_search () from /lib64/libdb-5.3.so #8 0x00007f53fefa8d0f in __bamc_get () from /lib64/libdb-5.3.so #9 0x00007f53ff061ca6 in __dbc_iget () from /lib64/libdb-5.3.so #10 0x00007f53ff070b02 in __dbc_get_pp () from /lib64/libdb-5.3.so #11 0x00007f53fa54c680 in idl_new_fetch () from /usr/lib64/dirsrv/plugins/ libback-ldbm.so #12 0x00007f53fa55ab26 in index_read_ext_allids () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #13 0x00007f53fa545284 in keys2idl () from /usr/lib64/dirsrv/plugins/ libback-ldbm.so #14 0x00007f53fa5459e3 in ava_candidates.isra.0 () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #15 0x00007f53fa545fd2 in filter_candidates_ext () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #16 0x00007f53fa547096 in list_candidates () from /usr/lib64/dirsrv/plugins/ libback-ldbm.so #17 0x00007f53fa545f40 in filter_candidates_ext () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #18 0x00007f53fa58180c in subtree_candidates () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #19 0x00007f53fa582e7f in ldbm_back_search () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #20 0x00007f5406eb01d6 in op_shared_search () from /usr/lib64/dirsrv/libslapd.so.0 #21 0x00007f54073a133e in do_search () #22 0x00007f540738f567 in connection_threadmain () #23 0x00007f540509e7bb in _pt_root () from /lib64/libnspr4.so #24 0x00007f5404a3fdc5 in start_thread () from /lib64/libpthread.so.0 #25 0x00007f540476d28d in clone () from /lib64/libc.so.6 Thread 25 (Thread 0x7f53d6ff5700 (LWP 128221)): #0 0x00007f5404a436d5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f53fef9f2f3 in __db_hybrid_mutex_suspend () from /lib64/ libdb-5.3.so #2 0x00007f53fef9e640 in __db_tas_mutex_lock () from /lib64/libdb-5.3.so #3 0x00007f53ff048d3a in __lock_get_internal () from /lib64/libdb-5.3.so #4 0x00007f53ff049820 in __lock_get () from /lib64/libdb-5.3.so #5 0x00007f53ff075142 in __db_lget () from /lib64/libdb-5.3.so #6 0x00007f53fefbc605 in __bam_search () from /lib64/libdb-5.3.so #7 0x00007f53fefa7256 in __bamc_search () from /lib64/libdb-5.3.so #8 0x00007f53fefa8d0f in __bamc_get () from /lib64/libdb-5.3.so #9 0x00007f53ff061ca6 in __dbc_iget () from /lib64/libdb-5.3.so #10 0x00007f53ff070b02 in __dbc_get_pp () from /lib64/libdb-5.3.so #11 0x00007f53fa54c680 in idl_new_fetch () from /usr/lib64/dirsrv/plugins/ libback-ldbm.so #12 0x00007f53fa55ab26 in index_read_ext_allids () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #13 0x00007f53fa545284 in keys2idl () from /usr/lib64/dirsrv/plugins/ libback-ldbm.so #14 0x00007f53fa5459e3 in ava_candidates.isra.0 () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #15 0x00007f53fa545fd2 in filter_candidates_ext () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #16 0x00007f53fa547096 in list_candidates () from /usr/lib64/dirsrv/plugins/ libback-ldbm.so #17 0x00007f53fa545f40 in filter_candidates_ext () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #18 0x00007f53fa58180c in subtree_candidates () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #19 0x00007f53fa582e7f in ldbm_back_search () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #20 0x00007f5406eb01d6 in op_shared_search () from /usr/lib64/dirsrv/libslapd.so.0 #21 0x00007f54073a133e in do_search () #22 0x00007f540738f567 in connection_threadmain () #23 0x00007f540509e7bb in _pt_root () from /lib64/libnspr4.so #24 0x00007f5404a3fdc5 in start_thread () from /lib64/libpthread.so.0 #25 0x00007f540476d28d in clone () from /lib64/libc.so.6 Thread 24 (Thread 0x7f53d67f4700 (LWP 128222)): #0 0x00007f5404a436d5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f53fef9f2f3 in __db_hybrid_mutex_suspend () from /lib64/ libdb-5.3.so #2 0x00007f53fef9e640 in __db_tas_mutex_lock () from /lib64/libdb-5.3.so #3 0x00007f53ff048d3a in __lock_get_internal () from /lib64/libdb-5.3.so #4 0x00007f53ff049820 in __lock_get () from /lib64/libdb-5.3.so #5 0x00007f53ff075142 in __db_lget () from /lib64/libdb-5.3.so #6 0x00007f53fefbc605 in __bam_search () from /lib64/libdb-5.3.so #7 0x00007f53fefa7256 in __bamc_search () from /lib64/libdb-5.3.so #8 0x00007f53fefa8d0f in __bamc_get () from /lib64/libdb-5.3.so #9 0x00007f53ff061ca6 in __dbc_iget () from /lib64/libdb-5.3.so #10 0x00007f53ff070b02 in __dbc_get_pp () from /lib64/libdb-5.3.so #11 0x00007f53fa54c680 in idl_new_fetch () from /usr/lib64/dirsrv/plugins/ libback-ldbm.so #12 0x00007f53fa55ab26 in index_read_ext_allids () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #13 0x00007f53fa545284 in keys2idl () from /usr/lib64/dirsrv/plugins/ libback-ldbm.so #14 0x00007f53fa5459e3 in ava_candidates.isra.0 () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #15 0x00007f53fa545fd2 in filter_candidates_ext () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #16 0x00007f53fa547096 in list_candidates () from /usr/lib64/dirsrv/plugins/ libback-ldbm.so #17 0x00007f53fa545f40 in filter_candidates_ext () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #18 0x00007f53fa58180c in subtree_candidates () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #19 0x00007f53fa582e7f in ldbm_back_search () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #20 0x00007f5406eb01d6 in op_shared_search () from /usr/lib64/dirsrv/libslapd.so.0 #21 0x00007f54073a133e in do_search () #22 0x00007f540738f567 in connection_threadmain () #23 0x00007f540509e7bb in _pt_root () from /lib64/libnspr4.so #24 0x00007f5404a3fdc5 in start_thread () from /lib64/libpthread.so.0 #25 0x00007f540476d28d in clone () from /lib64/libc.so.6 Thread 23 (Thread 0x7f53d5ff3700 (LWP 128223)): #0 0x00007f5404a436d5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f53fef9f2f3 in __db_hybrid_mutex_suspend () from /lib64/ libdb-5.3.so #2 0x00007f53fef9e640 in __db_tas_mutex_lock () from /lib64/libdb-5.3.so #3 0x00007f53ff0b8e1b in __memp_fget () from /lib64/libdb-5.3.so #4 0x00007f53fefbc211 in __bam_search () from /lib64/libdb-5.3.so #5 0x00007f53fefa7256 in __bamc_search () from /lib64/libdb-5.3.so #6 0x00007f53fefab6c4 in __bamc_put () from /lib64/libdb-5.3.so #7 0x00007f53ff0626d5 in __dbc_iput () from /lib64/libdb-5.3.so #8 0x00007f53ff05da0e in __db_put () from /lib64/libdb-5.3.so #9 0x00007f53ff072fa4 in __db_put_pp () from /lib64/libdb-5.3.so #10 0x00007f53fa54d4cf in idl_new_insert_key () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #11 0x00007f53fa55c5b5 in addordel_values_sv.isra.3 () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #12 0x00007f53fa55d25a in index_addordel_values_ext_sv () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #13 0x00007f53fa55d694 in index_addordel_values_sv () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #14 0x00007f53fa55d75a in index_addordel_entry () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #15 0x00007f53fa5605fa in ldbm_back_add () from /usr/lib64/dirsrv/plugins/ libback-ldbm.so #16 0x00007f5406e62cca in op_shared_add () from /usr/lib64/dirsrv/libslapd.so.0 #17 0x00007f5406e64020 in do_add () from /usr/lib64/dirsrv/libslapd.so.0 #18 0x00007f540738f303 in connection_threadmain () #19 0x00007f540509e7bb in _pt_root () from /lib64/libnspr4.so #20 0x00007f5404a3fdc5 in start_thread () from /lib64/libpthread.so.0 #21 0x00007f540476d28d in clone () from /lib64/libc.so.6 Thread 22 (Thread 0x7f53d57f2700 (LWP 128224)): #0 0x00007f5404a436d5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f53fef9f2f3 in __db_hybrid_mutex_suspend () from /lib64/ libdb-5.3.so #2 0x00007f53fef9e640 in __db_tas_mutex_lock () from /lib64/libdb-5.3.so #3 0x00007f53ff048d3a in __lock_get_internal () from /lib64/libdb-5.3.so #4 0x00007f53ff049820 in __lock_get () from /lib64/libdb-5.3.so #5 0x00007f53ff075142 in __db_lget () from /lib64/libdb-5.3.so #6 0x00007f53fefbc605 in __bam_search () from /lib64/libdb-5.3.so #7 0x00007f53fefa7256 in __bamc_search () from /lib64/libdb-5.3.so #8 0x00007f53fefa8d0f in __bamc_get () from /lib64/libdb-5.3.so #9 0x00007f53ff061ca6 in __dbc_iget () from /lib64/libdb-5.3.so #10 0x00007f53ff070b02 in __dbc_get_pp () from /lib64/libdb-5.3.so #11 0x00007f53fa54c680 in idl_new_fetch () from /usr/lib64/dirsrv/plugins/ libback-ldbm.so #12 0x00007f53fa55ab26 in index_read_ext_allids () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #13 0x00007f53fa545284 in keys2idl () from /usr/lib64/dirsrv/plugins/ libback-ldbm.so #14 0x00007f53fa5459e3 in ava_candidates.isra.0 () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #15 0x00007f53fa545fd2 in filter_candidates_ext () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #16 0x00007f53fa547096 in list_candidates () from /usr/lib64/dirsrv/plugins/ libback-ldbm.so #17 0x00007f53fa545f40 in filter_candidates_ext () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #18 0x00007f53fa58180c in subtree_candidates () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #19 0x00007f53fa582e7f in ldbm_back_search () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #20 0x00007f5406eb01d6 in op_shared_search () from /usr/lib64/dirsrv/libslapd.so.0 #21 0x00007f54073a133e in do_search () #22 0x00007f540738f567 in connection_threadmain () #23 0x00007f540509e7bb in _pt_root () from /lib64/libnspr4.so #24 0x00007f5404a3fdc5 in start_thread () from /lib64/libpthread.so.0 #25 0x00007f540476d28d in clone () from /lib64/libc.so.6 Thread 21 (Thread 0x7f53d4ff1700 (LWP 128225)): #0 0x00007f5404a436d5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f53fef9f2f3 in __db_hybrid_mutex_suspend () from /lib64/ libdb-5.3.so #2 0x00007f53fef9e640 in __db_tas_mutex_lock () from /lib64/libdb-5.3.so #3 0x00007f53ff048d3a in __lock_get_internal () from /lib64/libdb-5.3.so #4 0x00007f53ff049820 in __lock_get () from /lib64/libdb-5.3.so #5 0x00007f53ff075142 in __db_lget () from /lib64/libdb-5.3.so #6 0x00007f53fefbc605 in __bam_search () from /lib64/libdb-5.3.so #7 0x00007f53fefa7256 in __bamc_search () from /lib64/libdb-5.3.so #8 0x00007f53fefa8d0f in __bamc_get () from /lib64/libdb-5.3.so #9 0x00007f53ff061ca6 in __dbc_iget () from /lib64/libdb-5.3.so #10 0x00007f53ff070b02 in __dbc_get_pp () from /lib64/libdb-5.3.so #11 0x00007f53fa54c680 in idl_new_fetch () from /usr/lib64/dirsrv/plugins/ libback-ldbm.so #12 0x00007f53fa55ab26 in index_read_ext_allids () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #13 0x00007f53fa545284 in keys2idl () from /usr/lib64/dirsrv/plugins/ libback-ldbm.so #14 0x00007f53fa5459e3 in ava_candidates.isra.0 () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #15 0x00007f53fa545fd2 in filter_candidates_ext () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #16 0x00007f53fa547096 in list_candidates () from /usr/lib64/dirsrv/plugins/ libback-ldbm.so #17 0x00007f53fa545f40 in filter_candidates_ext () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #18 0x00007f53fa58180c in subtree_candidates () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #19 0x00007f53fa582e7f in ldbm_back_search () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #20 0x00007f5406eb01d6 in op_shared_search () from /usr/lib64/dirsrv/libslapd.so.0 #21 0x00007f54073a133e in do_search () #22 0x00007f540738f567 in connection_threadmain () #23 0x00007f540509e7bb in _pt_root () from /lib64/libnspr4.so #24 0x00007f5404a3fdc5 in start_thread () from /lib64/libpthread.so.0 #25 0x00007f540476d28d in clone () from /lib64/libc.so.6 Thread 20 (Thread 0x7f53d47f0700 (LWP 128226)): #0 0x00007f5404a436d5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f53fef9f2f3 in __db_hybrid_mutex_suspend () from /lib64/ libdb-5.3.so #2 0x00007f53fef9e640 in __db_tas_mutex_lock () from /lib64/libdb-5.3.so #3 0x00007f53ff048d3a in __lock_get_internal () from /lib64/libdb-5.3.so #4 0x00007f53ff049820 in __lock_get () from /lib64/libdb-5.3.so #5 0x00007f53ff075142 in __db_lget () from /lib64/libdb-5.3.so #6 0x00007f53fefbc605 in __bam_search () from /lib64/libdb-5.3.so #7 0x00007f53fefa7256 in __bamc_search () from /lib64/libdb-5.3.so #8 0x00007f53fefa8d0f in __bamc_get () from /lib64/libdb-5.3.so #9 0x00007f53ff061ca6 in __dbc_iget () from /lib64/libdb-5.3.so #10 0x00007f53ff070b02 in __dbc_get_pp () from /lib64/libdb-5.3.so #11 0x00007f53fa54c680 in idl_new_fetch () from /usr/lib64/dirsrv/plugins/ libback-ldbm.so #12 0x00007f53fa55ab26 in index_read_ext_allids () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #13 0x00007f53fa545284 in keys2idl () from /usr/lib64/dirsrv/plugins/ libback-ldbm.so #14 0x00007f53fa5459e3 in ava_candidates.isra.0 () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #15 0x00007f53fa545fd2 in filter_candidates_ext () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #16 0x00007f53fa547096 in list_candidates () from /usr/lib64/dirsrv/plugins/ libback-ldbm.so #17 0x00007f53fa545f40 in filter_candidates_ext () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #18 0x00007f53fa58180c in subtree_candidates () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #19 0x00007f53fa582e7f in ldbm_back_search () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #20 0x00007f5406eb01d6 in op_shared_search () from /usr/lib64/dirsrv/libslapd.so.0 #21 0x00007f5406ec094e in search_internal_callback_pb () from /usr/lib64/dirsrv/libslapd.so.0 #22 0x00007f5406ec0be8 in search_internal_pb () from /usr/lib64/dirsrv/libslapd.so.0 #23 0x00007f53fe118e90 in search_one_berval () from /usr/lib64/dirsrv/plugins/libattr-unique-plugin.so #24 0x00007f53fe119242 in search () from /usr/lib64/dirsrv/plugins/ libattr-unique-plugin.so #25 0x00007f53fe11954c in searchAllSubtrees () from /usr/lib64/dirsrv/plugins/libattr-unique-plugin.so #26 0x00007f53fe11a015 in preop_add () from /usr/lib64/dirsrv/plugins/ libattr-unique-plugin.so #27 0x00007f5406ebb848 in plugin_call_func () from /usr/lib64/dirsrv/libslapd.so.0 #28 0x00007f5406ebbb13 in plugin_call_plugins () from /usr/lib64/dirsrv/libslapd.so.0 #29 0x00007f5406e62c2b in op_shared_add () from /usr/lib64/dirsrv/libslapd.so.0 #30 0x00007f5406e64020 in do_add () from /usr/lib64/dirsrv/libslapd.so.0 #31 0x00007f540738f303 in connection_threadmain () #32 0x00007f540509e7bb in _pt_root () from /lib64/libnspr4.so #33 0x00007f5404a3fdc5 in start_thread () from /lib64/libpthread.so.0 #34 0x00007f540476d28d in clone () from /lib64/libc.so.6 Thread 19 (Thread 0x7f53d3fef700 (LWP 128227)): #0 0x00007f5404a436d5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f53fef9f2f3 in __db_hybrid_mutex_suspend () from /lib64/ libdb-5.3.so #2 0x00007f53fef9e640 in __db_tas_mutex_lock () from /lib64/libdb-5.3.so #3 0x00007f53ff048d3a in __lock_get_internal () from /lib64/libdb-5.3.so #4 0x00007f53ff049820 in __lock_get () from /lib64/libdb-5.3.so #5 0x00007f53ff075142 in __db_lget () from /lib64/libdb-5.3.so #6 0x00007f53fefbc605 in __bam_search () from /lib64/libdb-5.3.so #7 0x00007f53fefa7256 in __bamc_search () from /lib64/libdb-5.3.so #8 0x00007f53fefa8d0f in __bamc_get () from /lib64/libdb-5.3.so #9 0x00007f53ff061ca6 in __dbc_iget () from /lib64/libdb-5.3.so #10 0x00007f53ff070b02 in __dbc_get_pp () from /lib64/libdb-5.3.so #11 0x00007f53fa54c680 in idl_new_fetch () from /usr/lib64/dirsrv/plugins/ libback-ldbm.so #12 0x00007f53fa55ab26 in index_read_ext_allids () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #13 0x00007f53fa545284 in keys2idl () from /usr/lib64/dirsrv/plugins/ libback-ldbm.so #14 0x00007f53fa5459e3 in ava_candidates.isra.0 () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #15 0x00007f53fa545fd2 in filter_candidates_ext () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #16 0x00007f53fa547096 in list_candidates () from /usr/lib64/dirsrv/plugins/ libback-ldbm.so #17 0x00007f53fa545f40 in filter_candidates_ext () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #18 0x00007f53fa58180c in subtree_candidates () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #19 0x00007f53fa582e7f in ldbm_back_search () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #20 0x00007f5406eb01d6 in op_shared_search () from /usr/lib64/dirsrv/libslapd.so.0 #21 0x00007f54073a133e in do_search () #22 0x00007f540738f567 in connection_threadmain () #23 0x00007f540509e7bb in _pt_root () from /lib64/libnspr4.so #24 0x00007f5404a3fdc5 in start_thread () from /lib64/libpthread.so.0 #25 0x00007f540476d28d in clone () from /lib64/libc.so.6 Thread 18 (Thread 0x7f53d37ee700 (LWP 128228)): #0 0x00007f5404a436d5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f53fef9f2f3 in __db_hybrid_mutex_suspend () from /lib64/ libdb-5.3.so #2 0x00007f53fef9e640 in __db_tas_mutex_lock () from /lib64/libdb-5.3.so #3 0x00007f53ff048d3a in __lock_get_internal () from /lib64/libdb-5.3.so #4 0x00007f53ff049820 in __lock_get () from /lib64/libdb-5.3.so #5 0x00007f53ff075142 in __db_lget () from /lib64/libdb-5.3.so #6 0x00007f53fefbc605 in __bam_search () from /lib64/libdb-5.3.so #7 0x00007f53fefa7256 in __bamc_search () from /lib64/libdb-5.3.so #8 0x00007f53fefa8d0f in __bamc_get () from /lib64/libdb-5.3.so #9 0x00007f53ff061ca6 in __dbc_iget () from /lib64/libdb-5.3.so #10 0x00007f53ff070b02 in __dbc_get_pp () from /lib64/libdb-5.3.so #11 0x00007f53fa54c680 in idl_new_fetch () from /usr/lib64/dirsrv/plugins/ libback-ldbm.so #12 0x00007f53fa55ab26 in index_read_ext_allids () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #13 0x00007f53fa545284 in keys2idl () from /usr/lib64/dirsrv/plugins/ libback-ldbm.so #14 0x00007f53fa5459e3 in ava_candidates.isra.0 () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #15 0x00007f53fa545fd2 in filter_candidates_ext () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #16 0x00007f53fa547096 in list_candidates () from /usr/lib64/dirsrv/plugins/ libback-ldbm.so #17 0x00007f53fa545f40 in filter_candidates_ext () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #18 0x00007f53fa58180c in subtree_candidates () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #19 0x00007f53fa582e7f in ldbm_back_search () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #20 0x00007f5406eb01d6 in op_shared_search () from /usr/lib64/dirsrv/libslapd.so.0 #21 0x00007f54073a133e in do_search () #22 0x00007f540738f567 in connection_threadmain () #23 0x00007f540509e7bb in _pt_root () from /lib64/libnspr4.so #24 0x00007f5404a3fdc5 in start_thread () from /lib64/libpthread.so.0 #25 0x00007f540476d28d in clone () from /lib64/libc.so.6 Thread 17 (Thread 0x7f53d2fed700 (LWP 128229)): #0 0x00007f5404a436d5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f53fef9f2f3 in __db_hybrid_mutex_suspend () from /lib64/ libdb-5.3.so #2 0x00007f53fef9e640 in __db_tas_mutex_lock () from /lib64/libdb-5.3.so #3 0x00007f53ff048d3a in __lock_get_internal () from /lib64/libdb-5.3.so #4 0x00007f53ff049820 in __lock_get () from /lib64/libdb-5.3.so #5 0x00007f53ff075142 in __db_lget () from /lib64/libdb-5.3.so #6 0x00007f53fefbc605 in __bam_search () from /lib64/libdb-5.3.so #7 0x00007f53fefa7256 in __bamc_search () from /lib64/libdb-5.3.so #8 0x00007f53fefa8d0f in __bamc_get () from /lib64/libdb-5.3.so #9 0x00007f53ff061ca6 in __dbc_iget () from /lib64/libdb-5.3.so #10 0x00007f53ff070b02 in __dbc_get_pp () from /lib64/libdb-5.3.so #11 0x00007f53fa54c680 in idl_new_fetch () from /usr/lib64/dirsrv/plugins/ libback-ldbm.so #12 0x00007f53fa55ab26 in index_read_ext_allids () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #13 0x00007f53fa545284 in keys2idl () from /usr/lib64/dirsrv/plugins/ libback-ldbm.so #14 0x00007f53fa5459e3 in ava_candidates.isra.0 () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #15 0x00007f53fa545fd2 in filter_candidates_ext () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #16 0x00007f53fa547096 in list_candidates () from /usr/lib64/dirsrv/plugins/ libback-ldbm.so #17 0x00007f53fa545f40 in filter_candidates_ext () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #18 0x00007f53fa58180c in subtree_candidates () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #19 0x00007f53fa582e7f in ldbm_back_search () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #20 0x00007f5406eb01d6 in op_shared_search () from /usr/lib64/dirsrv/libslapd.so.0 #21 0x00007f54073a133e in do_search () #22 0x00007f540738f567 in connection_threadmain () #23 0x00007f540509e7bb in _pt_root () from /lib64/libnspr4.so #24 0x00007f5404a3fdc5 in start_thread () from /lib64/libpthread.so.0 #25 0x00007f540476d28d in clone () from /lib64/libc.so.6 Thread 16 (Thread 0x7f53d27ec700 (LWP 128230)): #0 0x00007f5404a436d5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f53fef9f2f3 in __db_hybrid_mutex_suspend () from /lib64/ libdb-5.3.so #2 0x00007f53fef9e640 in __db_tas_mutex_lock () from /lib64/libdb-5.3.so #3 0x00007f53ff048d3a in __lock_get_internal () from /lib64/libdb-5.3.so #4 0x00007f53ff049820 in __lock_get () from /lib64/libdb-5.3.so #5 0x00007f53ff075142 in __db_lget () from /lib64/libdb-5.3.so #6 0x00007f53fefbc605 in __bam_search () from /lib64/libdb-5.3.so #7 0x00007f53fefa7256 in __bamc_search () from /lib64/libdb-5.3.so #8 0x00007f53fefa8d0f in __bamc_get () from /lib64/libdb-5.3.so #9 0x00007f53ff061ca6 in __dbc_iget () from /lib64/libdb-5.3.so #10 0x00007f53ff070b02 in __dbc_get_pp () from /lib64/libdb-5.3.so #11 0x00007f53fa54c680 in idl_new_fetch () from /usr/lib64/dirsrv/plugins/ libback-ldbm.so #12 0x00007f53fa55ab26 in index_read_ext_allids () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #13 0x00007f53fa545284 in keys2idl () from /usr/lib64/dirsrv/plugins/ libback-ldbm.so #14 0x00007f53fa5459e3 in ava_candidates.isra.0 () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #15 0x00007f53fa545fd2 in filter_candidates_ext () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #16 0x00007f53fa547096 in list_candidates () from /usr/lib64/dirsrv/plugins/ libback-ldbm.so #17 0x00007f53fa545f40 in filter_candidates_ext () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #18 0x00007f53fa58180c in subtree_candidates () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #19 0x00007f53fa582e7f in ldbm_back_search () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #20 0x00007f5406eb01d6 in op_shared_search () from /usr/lib64/dirsrv/libslapd.so.0 #21 0x00007f54073a133e in do_search () #22 0x00007f540738f567 in connection_threadmain () #23 0x00007f540509e7bb in _pt_root () from /lib64/libnspr4.so #24 0x00007f5404a3fdc5 in start_thread () from /lib64/libpthread.so.0 #25 0x00007f540476d28d in clone () from /lib64/libc.so.6 Thread 15 (Thread 0x7f53d1feb700 (LWP 128231)): #0 0x00007f5404a436d5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f53fef9f2f3 in __db_hybrid_mutex_suspend () from /lib64/ libdb-5.3.so #2 0x00007f53fef9e640 in __db_tas_mutex_lock () from /lib64/libdb-5.3.so #3 0x00007f53ff048d3a in __lock_get_internal () from /lib64/libdb-5.3.so #4 0x00007f53ff049820 in __lock_get () from /lib64/libdb-5.3.so #5 0x00007f53ff075142 in __db_lget () from /lib64/libdb-5.3.so #6 0x00007f53fefbc605 in __bam_search () from /lib64/libdb-5.3.so #7 0x00007f53fefa7256 in __bamc_search () from /lib64/libdb-5.3.so #8 0x00007f53fefa8d0f in __bamc_get () from /lib64/libdb-5.3.so #9 0x00007f53ff061ca6 in __dbc_iget () from /lib64/libdb-5.3.so #10 0x00007f53ff070b02 in __dbc_get_pp () from /lib64/libdb-5.3.so #11 0x00007f53fa54c680 in idl_new_fetch () from /usr/lib64/dirsrv/plugins/ libback-ldbm.so #12 0x00007f53fa55ab26 in index_read_ext_allids () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #13 0x00007f53fa545284 in keys2idl () from /usr/lib64/dirsrv/plugins/ libback-ldbm.so #14 0x00007f53fa5459e3 in ava_candidates.isra.0 () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #15 0x00007f53fa545fd2 in filter_candidates_ext () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #16 0x00007f53fa547096 in list_candidates () from /usr/lib64/dirsrv/plugins/ libback-ldbm.so #17 0x00007f53fa545f40 in filter_candidates_ext () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #18 0x00007f53fa58180c in subtree_candidates () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #19 0x00007f53fa582e7f in ldbm_back_search () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #20 0x00007f5406eb01d6 in op_shared_search () from /usr/lib64/dirsrv/libslapd.so.0 #21 0x00007f54073a133e in do_search () #22 0x00007f540738f567 in connection_threadmain () #23 0x00007f540509e7bb in _pt_root () from /lib64/libnspr4.so #24 0x00007f5404a3fdc5 in start_thread () from /lib64/libpthread.so.0 #25 0x00007f540476d28d in clone () from /lib64/libc.so.6 Thread 14 (Thread 0x7f53d17ea700 (LWP 128232)): #0 0x00007f5404a436d5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f53fef9f2f3 in __db_hybrid_mutex_suspend () from /lib64/ libdb-5.3.so #2 0x00007f53fef9e640 in __db_tas_mutex_lock () from /lib64/libdb-5.3.so #3 0x00007f53ff048d3a in __lock_get_internal () from /lib64/libdb-5.3.so #4 0x00007f53ff049820 in __lock_get () from /lib64/libdb-5.3.so #5 0x00007f53ff075142 in __db_lget () from /lib64/libdb-5.3.so #6 0x00007f53fefbc605 in __bam_search () from /lib64/libdb-5.3.so #7 0x00007f53fefa7256 in __bamc_search () from /lib64/libdb-5.3.so #8 0x00007f53fefa8d0f in __bamc_get () from /lib64/libdb-5.3.so #9 0x00007f53ff061ca6 in __dbc_iget () from /lib64/libdb-5.3.so #10 0x00007f53ff070b02 in __dbc_get_pp () from /lib64/libdb-5.3.so #11 0x00007f53fa54c680 in idl_new_fetch () from /usr/lib64/dirsrv/plugins/ libback-ldbm.so #12 0x00007f53fa55ab26 in index_read_ext_allids () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #13 0x00007f53fa545284 in keys2idl () from /usr/lib64/dirsrv/plugins/ libback-ldbm.so #14 0x00007f53fa5459e3 in ava_candidates.isra.0 () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #15 0x00007f53fa545fd2 in filter_candidates_ext () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #16 0x00007f53fa547096 in list_candidates () from /usr/lib64/dirsrv/plugins/ libback-ldbm.so #17 0x00007f53fa545f40 in filter_candidates_ext () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #18 0x00007f53fa547096 in list_candidates () from /usr/lib64/dirsrv/plugins/ libback-ldbm.so #19 0x00007f53fa545f40 in filter_candidates_ext () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #20 0x00007f53fa547096 in list_candidates () from /usr/lib64/dirsrv/plugins/ libback-ldbm.so #21 0x00007f53fa545f40 in filter_candidates_ext () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #22 0x00007f53fa58180c in subtree_candidates () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #23 0x00007f53fa582e7f in ldbm_back_search () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #24 0x00007f5406eb01d6 in op_shared_search () from /usr/lib64/dirsrv/libslapd.so.0 #25 0x00007f54073a133e in do_search () #26 0x00007f540738f3c3 in connection_threadmain () #27 0x00007f540509e7bb in _pt_root () from /lib64/libnspr4.so #28 0x00007f5404a3fdc5 in start_thread () from /lib64/libpthread.so.0 #29 0x00007f540476d28d in clone () from /lib64/libc.so.6 Thread 13 (Thread 0x7f53d0fe9700 (LWP 128233)): #0 0x00007f5404a436d5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f53fef9f2f3 in __db_hybrid_mutex_suspend () from /lib64/ libdb-5.3.so #2 0x00007f53fef9e640 in __db_tas_mutex_lock () from /lib64/libdb-5.3.so #3 0x00007f53ff048d3a in __lock_get_internal () from /lib64/libdb-5.3.so #4 0x00007f53ff049820 in __lock_get () from /lib64/libdb-5.3.so #5 0x00007f53ff075142 in __db_lget () from /lib64/libdb-5.3.so #6 0x00007f53fefbc605 in __bam_search () from /lib64/libdb-5.3.so #7 0x00007f53fefa7256 in __bamc_search () from /lib64/libdb-5.3.so #8 0x00007f53fefa8d0f in __bamc_get () from /lib64/libdb-5.3.so #9 0x00007f53ff061ca6 in __dbc_iget () from /lib64/libdb-5.3.so #10 0x00007f53ff070b02 in __dbc_get_pp () from /lib64/libdb-5.3.so #11 0x00007f53fa54c680 in idl_new_fetch () from /usr/lib64/dirsrv/plugins/ libback-ldbm.so #12 0x00007f53fa55ab26 in index_read_ext_allids () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #13 0x00007f53fa545284 in keys2idl () from /usr/lib64/dirsrv/plugins/ libback-ldbm.so #14 0x00007f53fa5459e3 in ava_candidates.isra.0 () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #15 0x00007f53fa545fd2 in filter_candidates_ext () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #16 0x00007f53fa547096 in list_candidates () from /usr/lib64/dirsrv/plugins/ libback-ldbm.so #17 0x00007f53fa545f40 in filter_candidates_ext () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #18 0x00007f53fa58180c in subtree_candidates () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #19 0x00007f53fa582e7f in ldbm_back_search () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #20 0x00007f5406eb01d6 in op_shared_search () from /usr/lib64/dirsrv/libslapd.so.0 #21 0x00007f54073a133e in do_search () #22 0x00007f540738f567 in connection_threadmain () #23 0x00007f540509e7bb in _pt_root () from /lib64/libnspr4.so #24 0x00007f5404a3fdc5 in start_thread () from /lib64/libpthread.so.0 #25 0x00007f540476d28d in clone () from /lib64/libc.so.6 Thread 12 (Thread 0x7f53d07e8700 (LWP 128234)): #0 0x00007f5404a436d5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f53fef9f2f3 in __db_hybrid_mutex_suspend () from /lib64/ libdb-5.3.so #2 0x00007f53fef9e640 in __db_tas_mutex_lock () from /lib64/libdb-5.3.so #3 0x00007f53ff048d3a in __lock_get_internal () from /lib64/libdb-5.3.so #4 0x00007f53ff049820 in __lock_get () from /lib64/libdb-5.3.so #5 0x00007f53ff075142 in __db_lget () from /lib64/libdb-5.3.so #6 0x00007f53fefbc605 in __bam_search () from /lib64/libdb-5.3.so #7 0x00007f53fefa7256 in __bamc_search () from /lib64/libdb-5.3.so #8 0x00007f53fefa8d0f in __bamc_get () from /lib64/libdb-5.3.so #9 0x00007f53ff061ca6 in __dbc_iget () from /lib64/libdb-5.3.so #10 0x00007f53ff070b02 in __dbc_get_pp () from /lib64/libdb-5.3.so #11 0x00007f53fa54c680 in idl_new_fetch () from /usr/lib64/dirsrv/plugins/ libback-ldbm.so #12 0x00007f53fa55ab26 in index_read_ext_allids () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #13 0x00007f53fa545284 in keys2idl () from /usr/lib64/dirsrv/plugins/ libback-ldbm.so #14 0x00007f53fa5459e3 in ava_candidates.isra.0 () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #15 0x00007f53fa545fd2 in filter_candidates_ext () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #16 0x00007f53fa547096 in list_candidates () from /usr/lib64/dirsrv/plugins/ libback-ldbm.so #17 0x00007f53fa545f40 in filter_candidates_ext () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #18 0x00007f53fa58180c in subtree_candidates () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #19 0x00007f53fa582e7f in ldbm_back_search () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #20 0x00007f5406eb01d6 in op_shared_search () from /usr/lib64/dirsrv/libslapd.so.0 #21 0x00007f54073a133e in do_search () #22 0x00007f540738f567 in connection_threadmain () #23 0x00007f540509e7bb in _pt_root () from /lib64/libnspr4.so #24 0x00007f5404a3fdc5 in start_thread () from /lib64/libpthread.so.0 #25 0x00007f540476d28d in clone () from /lib64/libc.so.6 Thread 11 (Thread 0x7f53cffe7700 (LWP 128235)): #0 0x00007f5404a436d5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f53fef9f2f3 in __db_hybrid_mutex_suspend () from /lib64/ libdb-5.3.so #2 0x00007f53fef9e640 in __db_tas_mutex_lock () from /lib64/libdb-5.3.so #3 0x00007f53ff048d3a in __lock_get_internal () from /lib64/libdb-5.3.so #4 0x00007f53ff049820 in __lock_get () from /lib64/libdb-5.3.so #5 0x00007f53ff075142 in __db_lget () from /lib64/libdb-5.3.so #6 0x00007f53fefbc605 in __bam_search () from /lib64/libdb-5.3.so #7 0x00007f53fefa7256 in __bamc_search () from /lib64/libdb-5.3.so #8 0x00007f53fefa8d0f in __bamc_get () from /lib64/libdb-5.3.so #9 0x00007f53ff061ca6 in __dbc_iget () from /lib64/libdb-5.3.so #10 0x00007f53ff070b02 in __dbc_get_pp () from /lib64/libdb-5.3.so #11 0x00007f53fa54c680 in idl_new_fetch () from /usr/lib64/dirsrv/plugins/ libback-ldbm.so #12 0x00007f53fa55ab26 in index_read_ext_allids () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #13 0x00007f53fa545284 in keys2idl () from /usr/lib64/dirsrv/plugins/ libback-ldbm.so #14 0x00007f53fa5459e3 in ava_candidates.isra.0 () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #15 0x00007f53fa545fd2 in filter_candidates_ext () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #16 0x00007f53fa547096 in list_candidates () from /usr/lib64/dirsrv/plugins/ libback-ldbm.so #17 0x00007f53fa545f40 in filter_candidates_ext () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #18 0x00007f53fa58180c in subtree_candidates () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #19 0x00007f53fa582e7f in ldbm_back_search () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #20 0x00007f5406eb01d6 in op_shared_search () from /usr/lib64/dirsrv/libslapd.so.0 #21 0x00007f54073a133e in do_search () #22 0x00007f540738f567 in connection_threadmain () #23 0x00007f540509e7bb in _pt_root () from /lib64/libnspr4.so #24 0x00007f5404a3fdc5 in start_thread () from /lib64/libpthread.so.0 #25 0x00007f540476d28d in clone () from /lib64/libc.so.6 Thread 10 (Thread 0x7f53cf7e6700 (LWP 128236)): #0 0x00007f5404a436d5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f53fef9f2f3 in __db_hybrid_mutex_suspend () from /lib64/ libdb-5.3.so #2 0x00007f53fef9e640 in __db_tas_mutex_lock () from /lib64/libdb-5.3.so #3 0x00007f53ff048d3a in __lock_get_internal () from /lib64/libdb-5.3.so #4 0x00007f53ff049820 in __lock_get () from /lib64/libdb-5.3.so #5 0x00007f53ff075142 in __db_lget () from /lib64/libdb-5.3.so #6 0x00007f53fefbc605 in __bam_search () from /lib64/libdb-5.3.so #7 0x00007f53fefa7256 in __bamc_search () from /lib64/libdb-5.3.so #8 0x00007f53fefa8d0f in __bamc_get () from /lib64/libdb-5.3.so #9 0x00007f53ff061ca6 in __dbc_iget () from /lib64/libdb-5.3.so #10 0x00007f53ff070b02 in __dbc_get_pp () from /lib64/libdb-5.3.so #11 0x00007f53fa54c680 in idl_new_fetch () from /usr/lib64/dirsrv/plugins/ libback-ldbm.so #12 0x00007f53fa55ab26 in index_read_ext_allids () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #13 0x00007f53fa545284 in keys2idl () from /usr/lib64/dirsrv/plugins/ libback-ldbm.so #14 0x00007f53fa5459e3 in ava_candidates.isra.0 () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #15 0x00007f53fa545fd2 in filter_candidates_ext () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #16 0x00007f53fa547096 in list_candidates () from /usr/lib64/dirsrv/plugins/ libback-ldbm.so #17 0x00007f53fa545f40 in filter_candidates_ext () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #18 0x00007f53fa58180c in subtree_candidates () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #19 0x00007f53fa582e7f in ldbm_back_search () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #20 0x00007f5406eb01d6 in op_shared_search () from /usr/lib64/dirsrv/libslapd.so.0 #21 0x00007f54073a133e in do_search () #22 0x00007f540738f567 in connection_threadmain () #23 0x00007f540509e7bb in _pt_root () from /lib64/libnspr4.so #24 0x00007f5404a3fdc5 in start_thread () from /lib64/libpthread.so.0 #25 0x00007f540476d28d in clone () from /lib64/libc.so.6 Thread 9 (Thread 0x7f53cefe5700 (LWP 128237)): #0 0x00007f5404a436d5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f53fef9f2f3 in __db_hybrid_mutex_suspend () from /lib64/ libdb-5.3.so #2 0x00007f53fef9e640 in __db_tas_mutex_lock () from /lib64/libdb-5.3.so #3 0x00007f53ff048d3a in __lock_get_internal () from /lib64/libdb-5.3.so #4 0x00007f53ff049820 in __lock_get () from /lib64/libdb-5.3.so #5 0x00007f53ff075142 in __db_lget () from /lib64/libdb-5.3.so #6 0x00007f53fefbc605 in __bam_search () from /lib64/libdb-5.3.so #7 0x00007f53fefa7256 in __bamc_search () from /lib64/libdb-5.3.so #8 0x00007f53fefa8d0f in __bamc_get () from /lib64/libdb-5.3.so #9 0x00007f53ff061ca6 in __dbc_iget () from /lib64/libdb-5.3.so #10 0x00007f53ff070b02 in __dbc_get_pp () from /lib64/libdb-5.3.so #11 0x00007f53fa54c680 in idl_new_fetch () from /usr/lib64/dirsrv/plugins/ libback-ldbm.so #12 0x00007f53fa55ab26 in index_read_ext_allids () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #13 0x00007f53fa545284 in keys2idl () from /usr/lib64/dirsrv/plugins/ libback-ldbm.so #14 0x00007f53fa5459e3 in ava_candidates.isra.0 () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #15 0x00007f53fa545fd2 in filter_candidates_ext () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #16 0x00007f53fa547096 in list_candidates () from /usr/lib64/dirsrv/plugins/ libback-ldbm.so #17 0x00007f53fa545f40 in filter_candidates_ext () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #18 0x00007f53fa58180c in subtree_candidates () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #19 0x00007f53fa582e7f in ldbm_back_search () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #20 0x00007f5406eb01d6 in op_shared_search () from /usr/lib64/dirsrv/libslapd.so.0 #21 0x00007f54073a133e in do_search () #22 0x00007f540738f567 in connection_threadmain () #23 0x00007f540509e7bb in _pt_root () from /lib64/libnspr4.so #24 0x00007f5404a3fdc5 in start_thread () from /lib64/libpthread.so.0 #25 0x00007f540476d28d in clone () from /lib64/libc.so.6 Thread 8 (Thread 0x7f53ce7e4700 (LWP 128238)): #0 0x00007f5404a436d5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f53fef9f2f3 in __db_hybrid_mutex_suspend () from /lib64/ libdb-5.3.so #2 0x00007f53fef9e640 in __db_tas_mutex_lock () from /lib64/libdb-5.3.so #3 0x00007f53ff048d3a in __lock_get_internal () from /lib64/libdb-5.3.so #4 0x00007f53ff049820 in __lock_get () from /lib64/libdb-5.3.so #5 0x00007f53ff075142 in __db_lget () from /lib64/libdb-5.3.so #6 0x00007f53fefbc605 in __bam_search () from /lib64/libdb-5.3.so #7 0x00007f53fefa7256 in __bamc_search () from /lib64/libdb-5.3.so #8 0x00007f53fefa8d0f in __bamc_get () from /lib64/libdb-5.3.so #9 0x00007f53ff061ca6 in __dbc_iget () from /lib64/libdb-5.3.so #10 0x00007f53ff070b02 in __dbc_get_pp () from /lib64/libdb-5.3.so #11 0x00007f53fa54c680 in idl_new_fetch () from /usr/lib64/dirsrv/plugins/ libback-ldbm.so #12 0x00007f53fa55ab26 in index_read_ext_allids () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #13 0x00007f53fa545284 in keys2idl () from /usr/lib64/dirsrv/plugins/ libback-ldbm.so #14 0x00007f53fa5459e3 in ava_candidates.isra.0 () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #15 0x00007f53fa545fd2 in filter_candidates_ext () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #16 0x00007f53fa547096 in list_candidates () from /usr/lib64/dirsrv/plugins/ libback-ldbm.so #17 0x00007f53fa545f40 in filter_candidates_ext () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #18 0x00007f53fa58180c in subtree_candidates () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #19 0x00007f53fa582e7f in ldbm_back_search () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #20 0x00007f5406eb01d6 in op_shared_search () from /usr/lib64/dirsrv/libslapd.so.0 #21 0x00007f54073a133e in do_search () #22 0x00007f540738f567 in connection_threadmain () #23 0x00007f540509e7bb in _pt_root () from /lib64/libnspr4.so #24 0x00007f5404a3fdc5 in start_thread () from /lib64/libpthread.so.0 #25 0x00007f540476d28d in clone () from /lib64/libc.so.6 Thread 7 (Thread 0x7f53cdfe3700 (LWP 128239)): #0 0x00007f5404a436d5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f53fef9f2f3 in __db_hybrid_mutex_suspend () from /lib64/ libdb-5.3.so #2 0x00007f53fef9e640 in __db_tas_mutex_lock () from /lib64/libdb-5.3.so #3 0x00007f53ff048d3a in __lock_get_internal () from /lib64/libdb-5.3.so #4 0x00007f53ff049820 in __lock_get () from /lib64/libdb-5.3.so #5 0x00007f53ff075142 in __db_lget () from /lib64/libdb-5.3.so #6 0x00007f53fefbc605 in __bam_search () from /lib64/libdb-5.3.so #7 0x00007f53fefa7256 in __bamc_search () from /lib64/libdb-5.3.so #8 0x00007f53fefa8d0f in __bamc_get () from /lib64/libdb-5.3.so #9 0x00007f53ff061ca6 in __dbc_iget () from /lib64/libdb-5.3.so #10 0x00007f53ff070b02 in __dbc_get_pp () from /lib64/libdb-5.3.so #11 0x00007f53fa54c680 in idl_new_fetch () from /usr/lib64/dirsrv/plugins/ libback-ldbm.so #12 0x00007f53fa55ab26 in index_read_ext_allids () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #13 0x00007f53fa545284 in keys2idl () from /usr/lib64/dirsrv/plugins/ libback-ldbm.so #14 0x00007f53fa5459e3 in ava_candidates.isra.0 () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #15 0x00007f53fa545fd2 in filter_candidates_ext () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #16 0x00007f53fa547096 in list_candidates () from /usr/lib64/dirsrv/plugins/ libback-ldbm.so #17 0x00007f53fa545f40 in filter_candidates_ext () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #18 0x00007f53fa58180c in subtree_candidates () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #19 0x00007f53fa582e7f in ldbm_back_search () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #20 0x00007f5406eb01d6 in op_shared_search () from /usr/lib64/dirsrv/libslapd.so.0 #21 0x00007f54073a133e in do_search () #22 0x00007f540738f567 in connection_threadmain () #23 0x00007f540509e7bb in _pt_root () from /lib64/libnspr4.so #24 0x00007f5404a3fdc5 in start_thread () from /lib64/libpthread.so.0 #25 0x00007f540476d28d in clone () from /lib64/libc.so.6 Thread 6 (Thread 0x7f53cd7e2700 (LWP 128240)): #0 0x00007f5404a436d5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f53fef9f2f3 in __db_hybrid_mutex_suspend () from /lib64/ libdb-5.3.so #2 0x00007f53fef9eb37 in __db_tas_mutex_readlock () from /lib64/ libdb-5.3.so #3 0x00007f53ff0b8d35 in __memp_fget () from /lib64/libdb-5.3.so #4 0x00007f53ff062333 in __dbc_iget () from /lib64/libdb-5.3.so #5 0x00007f53ff070b02 in __dbc_get_pp () from /lib64/libdb-5.3.so #6 0x00007f53fa54c822 in idl_new_fetch () from /usr/lib64/dirsrv/plugins/ libback-ldbm.so #7 0x00007f53fa55ab26 in index_read_ext_allids () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #8 0x00007f53fa545284 in keys2idl () from /usr/lib64/dirsrv/plugins/ libback-ldbm.so #9 0x00007f53fa5459e3 in ava_candidates.isra.0 () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #10 0x00007f53fa545fd2 in filter_candidates_ext () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #11 0x00007f53fa547096 in list_candidates () from /usr/lib64/dirsrv/plugins/ libback-ldbm.so #12 0x00007f53fa545f40 in filter_candidates_ext () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #13 0x00007f53fa547096 in list_candidates () from /usr/lib64/dirsrv/plugins/ libback-ldbm.so #14 0x00007f53fa545f40 in filter_candidates_ext () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #15 0x00007f53fa547096 in list_candidates () from /usr/lib64/dirsrv/plugins/ libback-ldbm.so #16 0x00007f53fa545f40 in filter_candidates_ext () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #17 0x00007f53fa58180c in subtree_candidates () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #18 0x00007f53fa582e7f in ldbm_back_search () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #19 0x00007f5406eb01d6 in op_shared_search () from /usr/lib64/dirsrv/libslapd.so.0 #20 0x00007f54073a133e in do_search () #21 0x00007f540738f3c3 in connection_threadmain () #22 0x00007f540509e7bb in _pt_root () from /lib64/libnspr4.so #23 0x00007f5404a3fdc5 in start_thread () from /lib64/libpthread.so.0 #24 0x00007f540476d28d in clone () from /lib64/libc.so.6 Thread 5 (Thread 0x7f53ccfe1700 (LWP 128241)): #0 0x00007f5404a436d5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f53fef9f2f3 in __db_hybrid_mutex_suspend () from /lib64/ libdb-5.3.so #2 0x00007f53fef9e640 in __db_tas_mutex_lock () from /lib64/libdb-5.3.so #3 0x00007f53ff048d3a in __lock_get_internal () from /lib64/libdb-5.3.so #4 0x00007f53ff049820 in __lock_get () from /lib64/libdb-5.3.so #5 0x00007f53ff075142 in __db_lget () from /lib64/libdb-5.3.so #6 0x00007f53fefbc605 in __bam_search () from /lib64/libdb-5.3.so #7 0x00007f53fefa7256 in __bamc_search () from /lib64/libdb-5.3.so #8 0x00007f53fefa8d0f in __bamc_get () from /lib64/libdb-5.3.so #9 0x00007f53ff061ca6 in __dbc_iget () from /lib64/libdb-5.3.so #10 0x00007f53ff070b02 in __dbc_get_pp () from /lib64/libdb-5.3.so #11 0x00007f53fa54c680 in idl_new_fetch () from /usr/lib64/dirsrv/plugins/ libback-ldbm.so #12 0x00007f53fa55ab26 in index_read_ext_allids () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #13 0x00007f53fa545284 in keys2idl () from /usr/lib64/dirsrv/plugins/ libback-ldbm.so #14 0x00007f53fa5459e3 in ava_candidates.isra.0 () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #15 0x00007f53fa545fd2 in filter_candidates_ext () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #16 0x00007f53fa547096 in list_candidates () from /usr/lib64/dirsrv/plugins/ libback-ldbm.so #17 0x00007f53fa545f40 in filter_candidates_ext () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #18 0x00007f53fa547096 in list_candidates () from /usr/lib64/dirsrv/plugins/ libback-ldbm.so #19 0x00007f53fa545f40 in filter_candidates_ext () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #20 0x00007f53fa547096 in list_candidates () from /usr/lib64/dirsrv/plugins/ libback-ldbm.so #21 0x00007f53fa545f40 in filter_candidates_ext () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #22 0x00007f53fa58180c in subtree_candidates () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #23 0x00007f53fa582e7f in ldbm_back_search () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #24 0x00007f5406eb01d6 in op_shared_search () from /usr/lib64/dirsrv/libslapd.so.0 #25 0x00007f54073a133e in do_search () #26 0x00007f540738f3c3 in connection_threadmain () #27 0x00007f540509e7bb in _pt_root () from /lib64/libnspr4.so #28 0x00007f5404a3fdc5 in start_thread () from /lib64/libpthread.so.0 #29 0x00007f540476d28d in clone () from /lib64/libc.so.6 Thread 4 (Thread 0x7f53cc7e0700 (LWP 128242)): #0 0x00007f54047649b3 in select () from /lib64/libc.so.6 #1 0x00007f5406eef0e9 in DS_Sleep () from /usr/lib64/dirsrv/libslapd.so.0 #2 0x00007f5407390325 in time_thread () #3 0x00007f540509e7bb in _pt_root () from /lib64/libnspr4.so #4 0x00007f5404a3fdc5 in start_thread () from /lib64/libpthread.so.0 #5 0x00007f540476d28d in clone () from /lib64/libc.so.6 Thread 3 (Thread 0x7f53cb7a0700 (LWP 128259)): #0 0x00007f5404a436d5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f5405099050 in PR_WaitCondVar () from /lib64/libnspr4.so #2 0x00007f540739bf25 in ps_send_results () #3 0x00007f540509e7bb in _pt_root () from /lib64/libnspr4.so #4 0x00007f5404a3fdc5 in start_thread () from /lib64/libpthread.so.0 #5 0x00007f540476d28d in clone () from /lib64/libc.so.6 Thread 2 (Thread 0x7f53cab9d700 (LWP 9020)): #0 0x00007f5404a436d5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f5405099050 in PR_WaitCondVar () from /lib64/libnspr4.so #2 0x00007f540739bf25 in ps_send_results () #3 0x00007f540509e7bb in _pt_root () from /lib64/libnspr4.so #4 0x00007f5404a3fdc5 in start_thread () from /lib64/libpthread.so.0 #5 0x00007f540476d28d in clone () from /lib64/libc.so.6 Thread 1 (Thread 0x7f5407357840 (LWP 128124)): #0 0x00007f5404762c3d in poll () from /lib64/libc.so.6 #1 0x00007f540509a967 in _pr_poll_with_poll () from /lib64/libnspr4.so #2 0x00007f5407393ff1 in slapd_daemon () #3 0x00007f5407385c6c in main () Thanks On Tue, Sep 6, 2016 at 5:51 PM, Rakesh Rajasekharan < rakesh.rajasekharan at gmail.com> wrote: > Hi, > > With script provided by Rich, I looked at my db cache usage and it does > not look to be high enough > > # under each db are the list of selected indexes specified with INDEXLIST > dbname count free free% size > changelog:ent 5 2076333 99.0 4163.8 > changelog:dn 15976 9303557 88.7 74.0 > userroot:ent 4763 175321074 83.6 7221.1 > userroot:dn 4763 9995056 95.3 103.0 > ipaca:ent 425 6337866 60.4 9759.8 > ipaca:dn 425 10451081 99.7 81.6 > > Right now I set it to 200MB. So, I am planning to bring it down to 100MB > > Also, *nsslapd-db-checkpoint-interval *will change it to 80 or 90 from > default of 60 > > I will check this out and the impact it has on the hang issues and get back > > Thanks > Rakesh > > On Tue, Sep 6, 2016 at 4:13 PM, Ludwig Krispenz > wrote: > >> Hi, >> >> I think you are running into this bug: https://bugzilla.redhat.com/sh >> ow_bug.cgi?id=1349779 >> it was the outcome of an analysis started in this one: >> https://bugzilla.redhat.com/show_bug.cgi?id=1347800 >> >> the problem is that it looks like the checkpoint thread is holding a >> mutex and otehr threads are blocked on it. It looks like an issue in >> BerkeleyDB although there is no explanation from the BDB code why this >> could happen, we also had a support case with Oracle to investigate this, >> with no solution. Oracle at the end was providing a diagnostic patch, which >> we did not test since the customer with the original problem had worked >> around this by changing their configuration. >> I don't know the status of this patch and if we can make it available, >> probably depends on your support status, you need to contact support for >> this. >> >> About the workarounds: >> you can try to change timing for the database update and checkpointing eg >> by reducing the dbcache size, changing the checkpoint interval. In the >> customer case they identified one substring index, which was frequently >> updated but not really required - and removing this index helped. >> >> Regards, >> Ludwig >> >> On 09/06/2016 11:31 AM, thierry bordaz wrote: >> >> Hi, >> >> The following thread is hanging most of the others threads >> >> Thread 20 (Thread 0x7f68007f8700 (LWP 4873)): >> >> #0 0x00007f683d0506d5 in pthread_cond_wait@@GLIBC_2.3.2 () from >> /lib64/libpthread.so.0 >> #1 0x00007f68375ac2f3 in __db_hybrid_mutex_suspend () from /lib64/ >> libdb-5.3.so >> #2 0x00007f68375ab640 in __db_tas_mutex_lock () from /lib64/libdb-5.3.so >> #3 0x00007f68376c5e1b in __memp_fget () from /lib64/libdb-5.3.so >> #4 0x00007f68375c9211 in __bam_search () from /lib64/libdb-5.3.so >> #5 0x00007f68375b4256 in __bamc_search () from /lib64/libdb-5.3.so >> #6 0x00007f68375b86c4 in __bamc_put () from /lib64/libdb-5.3.so >> #7 0x00007f683766f6d5 in __dbc_iput () from /lib64/libdb-5.3.so >> #8 0x00007f683766aa0e in __db_put () from /lib64/libdb-5.3.so >> #9 0x00007f683767ffa4 in __db_put_pp () from /lib64/libdb-5.3.so >> #10 0x00007f6832b5a4cf in idl_new_insert_key () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #11 0x00007f6832b695b5 in addordel_values_sv.isra.3 () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #12 0x00007f6832b6a25a in index_addordel_values_ext_sv () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #13 0x00007f6832b6a694 in index_addordel_values_sv () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #14 0x00007f6832b6a75a in index_addordel_entry () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #15 0x00007f6832b6d5fa in ldbm_back_add () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #16 0x00007f683f46fcca in op_shared_add () from >> /usr/lib64/dirsrv/libslapd.so.0 >> #17 0x00007f683f471020 in do_add () from /usr/lib64/dirsrv/libslapd.so.0 >> #18 0x00007f683f99c303 in connection_threadmain () >> #19 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so >> #20 0x00007f683d04cdc5 in start_thread () from /lib64/libpthread.so.0 >> #21 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 >> >> This thread is likely not hang on 'db_tas_mutex_lock' as the db_lock does >> not show it waiting for something. >> Unfortunately it is not shown in the output of top -H, did you get the >> complete output ? >> Ludwig mentioned a known issue that DS is not scaling well on NUMA arch, >> do you know if the machine is NUMA ? >> A possibility is that this thread get severely slowed down by a NUMA >> remote lock and showing DS has hung. >> This is why it would be interesting to know if it consums CPU. >> >> thanks >> theirry >> >> On 09/06/2016 10:59 AM, Rakesh Rajasekharan wrote: >> >> Hi Thierry/Ludwig, >> >> I again hit the hang issue. This is the pstack of the ns-slapd >> >> So was able to this time get the pstack and db locks >> >> db locks details >> Default locking region information: >> 451 Last allocated locker ID >> 0x7fffffff Current maximum unused locker ID >> 9 Number of lock modes >> 800 Initial number of locks allocated >> 0 Initial number of lockers allocated >> 800 Initial number of lock objects allocated >> 10000 Maximum number of locks possible >> 10000 Maximum number of lockers possible >> 10000 Maximum number of lock objects possible >> 800 Current number of locks allocated >> 366 Current number of lockers allocated >> 800 Current number of lock objects allocated >> 160 Number of lock object partitions >> 8191 Size of object hash table >> 126 Number of current locks >> 1005 Maximum number of locks at any one time >> 33 Maximum number of locks in any one bucket >> 3987 Maximum number of locks stolen by for an empty partition >> 75 Maximum number of locks stolen for any one partition >> 309 Number of current lockers >> 313 Maximum number of lockers at any one time >> 121 Number of current lock objects >> 676 Maximum number of lock objects at any one time >> 3 Maximum number of lock objects in any one bucket >> 69 Maximum number of objects stolen by for an empty partition >> 5 Maximum number of objects stolen for any one partition >> 22M Total number of locks requested (22009642) >> 21M Total number of locks released (21992878) >> 0 Total number of locks upgraded >> 95 Total number of locks downgraded >> 3348 Lock requests not available due to conflicts, for which we >> waited >> 12646 Lock requests not available due to conflicts, for which we did >> not wait >> 0 Number of deadlocks >> 0 Lock timeout value >> 0 Number of locks that have timed out >> 0 Transaction timeout value >> 0 Number of transactions that have timed out >> 2MB 560KB Region size >> 71719 The number of partition locks that required waiting (0%) >> 23105 The maximum number of times any partition lock was waited for >> (0%) >> 0 The number of object queue operations that required waiting >> (0%) >> 163 The number of locker allocations that required waiting (0%) >> 3172 The number of region locks that required waiting (0%) >> 5 Maximum hash bucket length >> =-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-= >> Lock REGINFO information: >> Environment Region type >> 1 Region ID >> /var/lib/dirsrv/slapd-SPRINKLR-COM/db/__db.001 Region name >> 0x7f67ef869000 Region address >> 0x7f67ef8690a0 Region allocation head >> 0x7f67ef8712b0 Region primary address >> 0 Region maximum allocation >> 0 Region allocated >> Region allocations: 1581345 allocations, 0 failures, 1580794 frees, 6 >> longest >> Allocations by power-of-two sizes: >> 1KB 1581329 >> 2KB 3 >> 4KB 4 >> 8KB 3 >> 16KB 3 >> 32KB 1 >> 64KB 0 >> 128KB 0 >> 256KB 2 >> 512KB 0 >> 1024KB 1 >> REGION_SHARED Region flags >> =-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-= >> Lock region parameters: >> 2 Lock region region mutex [3172/14M 0% !Own] >> 16381 locker table size >> 8191 object table size >> 34128 obj_off >> 900216 locker_off >> 0 need_dd >> =-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-= >> Lock conflict matrix: >> =-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-= >> Locks grouped by lockers: >> Locker Mode Count Status ----------------- Object --------------- >> 2 dd=308 locks held 1 write locks 0 pid/thread >> 4845/140085720139840 flags 10 priority 100 >> 2 READ 1 HELD userRoot/id2entry.db handle 0 >> 3 dd=307 locks held 0 write locks 0 pid/thread >> 4845/140084829472512 flags 0 priority 100 >> 4 dd=306 locks held 1 write locks 0 pid/thread >> 4845/140085720139840 flags 10 priority 100 >> 4 READ 1 HELD ipaca/id2entry.db handle 0 >> 5 dd=305 locks held 0 write locks 0 pid/thread >> 4845/140084837865216 flags 0 priority 100 >> 6 dd=304 locks held 1 write locks 0 pid/thread >> 4845/140085720139840 flags 10 priority 100 >> 6 READ 1 HELD ipaca/entryrdn.db handle 0 >> 7 dd=303 locks held 0 write locks 0 pid/thread >> 4845/140084804294400 flags 0 priority 100 >> 8 dd=302 locks held 0 write locks 0 pid/thread >> 4845/140084720432896 flags 0 priority 100 >> 9 dd=301 locks held 0 write locks 0 pid/thread >> 4845/140084837865216 flags 0 priority 100 >> a dd=300 locks held 1 write locks 0 pid/thread >> 4845/140085720139840 flags 10 priority 100 >> a READ 1 HELD ipaca/vlv#allcertspkitomcatindex.db >> handle 0 >> e dd=299 locks held 1 write locks 0 pid/thread >> 4845/140085720139840 flags 10 priority 100 >> e READ 1 HELD ipaca/vlv#allnonrevokedcertspkitomcatindex.db >> handle 0 >> 10 dd=298 locks held 1 write locks 0 pid/thread >> 4845/140085720139840 flags 10 priority 100 >> 10 READ 1 HELD ipaca/vlv#allrevokedcertspkitomcatindex.db >> handle 0 >> 11 dd=297 locks held 1 write locks 0 pid/thread >> 4845/140085720139840 flags 10 priority 100 >> 11 READ 1 HELD ipaca/vlv#allrevokedcertsnotafterpkitomcatindex.db >> handle 0 >> 14 dd=296 locks held 1 write locks 0 pid/thread >> 4845/140085720139840 flags 10 priority 100 >> 14 READ 1 HELD ipaca/vlv#allrevokedorrevokedexpiredcertspkitomcatindex.db >> handle 0 >> 15 dd=295 locks held 1 write locks 0 pid/thread >> 4845/140085720139840 flags 10 priority 100 >> 15 READ 1 HELD ipaca/vlv#allvalidcertspkitomcatindex.db >> handle 0 >> 16 dd=294 locks held 1 write locks 0 pid/thread >> 4845/140085720139840 flags 10 priority 100 >> 16 READ 1 HELD ipaca/vlv#allvalidcertsnotafterpkitomcatindex.db >> handle 0 >> 17 dd=293 locks held 1 write locks 0 pid/thread >> 4845/140085720139840 flags 10 priority 100 >> 17 READ 1 HELD ipaca/vlv#allvalidorrevokedcertspkitomcatindex.db >> handle 0 >> 18 dd=292 locks held 1 write locks 0 pid/thread >> 4845/140085720139840 flags 10 priority 100 >> 18 READ 1 HELD ipaca/vlv#caallpkitomcatindex.db >> handle 0 >> 1d dd=291 locks held 1 write locks 0 pid/thread >> 4845/140085720139840 flags 10 priority 100 >> 1d READ 1 HELD ipaca/vlv#cacompletepkitomcatindex.db >> handle 0 >> 1e dd=290 locks held 1 write locks 0 >> >> ... -------------- next part -------------- An HTML attachment was scrubbed... URL: From kashmancy at gmail.com Wed Sep 7 20:12:16 2016 From: kashmancy at gmail.com (Harry Kashouli) Date: Wed, 7 Sep 2016 13:12:16 -0700 Subject: [Freeipa-users] FreeIPA and Tomcat service cannot work at the same time Message-ID: Hi all, System details: Fedora 24 FreeIPA 4.3.2, and working fine Desired outcome: To have pwm running on the same server, for password self-service - https://github.com/pwm-project/pwm My FreeIPA server is running fine, but when I attempt to start Tomcat for pwm, that service will not work. Systemctl output below, let me know if you need anything extra, and where to grab it from. If I enable Tomcat at boot, then both that and the ipa/kerberos services fail to start. It seems I cannot have both Apache Tomcat and FreeIPA running at the same time. I even tried changing Tomcat's ports by increasing the port number, by 1. [root@~]# systemctl status tomcat ? tomcat.service - Apache Tomcat Web Application Container Loaded: loaded (/usr/lib/systemd/system/tomcat.service; disabled; vendor preset: disabled) Active: active (running) since Wed 2016-09-07 13:00:26 PDT; 21s ago Main PID: 13925 (java) Tasks: 18 (limit: 512) CGroup: /system.slice/tomcat.service ??13925 /usr/lib/jvm/jre/bin/java -Djavax.sql.DataSource.Factory=org.apache.commons.dbcp.BasicDataSourceFactory -classpath /usr/share/tomcat/bin/bootstrap.jar:/usr/share/tomcat/bin/tomcat-juli.jar:/usr/lib/java/commons-daemon.jar -Dcatalina.base=/usr/share/t Sep 07 13:00:43 server[13925]: 2016-09-07T13:00:43Z, DEBUG, pwm.PwmEnvironment, released file lock on file /var/lib/tomcat/webapps/pwm/WEB-INF/applicationPath.lock Sep 07 13:00:43 server[13925]: 2016-09-07T13:00:43Z, INFO , pwm.PwmApplication, PWM v1.8.0-SNAPSHOT b18063275 rb329bec559be6e47c64239474c637b8bd9ed2d93 closed for bidness, cya! Sep 07 13:00:43 server[13925]: 07-Sep-2016 13:00:43.443 WARNING [localhost-startStop-1] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The web application [pwm] appears to have started a thread named [PwmSettingXml s Sep 07 13:00:43 server[13925]: java.lang.Thread.sleep(Native Method) Sep 07 13:00:43 server[13925]: password.pwm.util.Helper.pause(Helper.java:112) Sep 07 13:00:43 server[13925]: password.pwm.config.PwmSettingXml$1.run(PwmSettingXml.java:73) Sep 07 13:00:43 server[13925]: 07-Sep-2016 13:00:43.508 INFO [main] org.apache.coyote.AbstractProtocol.stop Stopping ProtocolHandler ["http-nio-8081"] Sep 07 13:00:43 server[13925]: 07-Sep-2016 13:00:43.510 INFO [main] org.apache.coyote.AbstractProtocol.stop Stopping ProtocolHandler ["ajp-nio-8010"] Sep 07 13:00:43 server[13925]: 07-Sep-2016 13:00:43.548 INFO [main] org.apache.coyote.AbstractProtocol.destroy Destroying ProtocolHandler ["http-nio-8081"] Sep 07 13:00:43 server[13925]: 07-Sep-2016 13:00:43.550 INFO [main] org.apache.coyote.AbstractProtocol.destroy Destroying ProtocolHandler ["ajp-nio-8010"] [root@~]# systemctl status tomcat ? tomcat.service - Apache Tomcat Web Application Container Loaded: loaded (/usr/lib/systemd/system/tomcat.service; disabled; vendor preset: disabled) Active: inactive (dead) Sep 07 13:00:43 server[13925]: 2016-09-07T13:00:43Z, DEBUG, pwm.PwmEnvironment, released file lock on file /var/lib/tomcat/webapps/pwm/WEB-INF/applicationPath.lock Sep 07 13:00:43 server[13925]: 2016-09-07T13:00:43Z, INFO , pwm.PwmApplication, PWM v1.8.0-SNAPSHOT b18063275 rb329bec559be6e47c64239474c637b8bd9ed2d93 closed for bidness, cya! Sep 07 13:00:43 server[13925]: 07-Sep-2016 13:00:43.443 WARNING [localhost-startStop-1] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The web application [pwm] appears to have started a thread named [PwmSettingXml s Sep 07 13:00:43 server[13925]: java.lang.Thread.sleep(Native Method) Sep 07 13:00:43 server[13925]: password.pwm.util.Helper.pause(Helper.java:112) Sep 07 13:00:43 server[13925]: password.pwm.config.PwmSettingXml$1.run(PwmSettingXml.java:73) Sep 07 13:00:43 server[13925]: 07-Sep-2016 13:00:43.508 INFO [main] org.apache.coyote.AbstractProtocol.stop Stopping ProtocolHandler ["http-nio-8081"] Sep 07 13:00:43 server[13925]: 07-Sep-2016 13:00:43.510 INFO [main] org.apache.coyote.AbstractProtocol.stop Stopping ProtocolHandler ["ajp-nio-8010"] Sep 07 13:00:43 server[13925]: 07-Sep-2016 13:00:43.548 INFO [main] org.apache.coyote.AbstractProtocol.destroy Destroying ProtocolHandler ["http-nio-8081"] Sep 07 13:00:43 server[13925]: 07-Sep-2016 13:00:43.550 INFO [main] org.apache.coyote.AbstractProtocol.destroy Destroying ProtocolHandler ["ajp-nio-8010"] Cheers, -Harry -------------- next part -------------- An HTML attachment was scrubbed... URL: From rcritten at redhat.com Wed Sep 7 20:16:14 2016 From: rcritten at redhat.com (Rob Crittenden) Date: Wed, 7 Sep 2016 16:16:14 -0400 Subject: [Freeipa-users] Freeipa 4.2.0 hangs intermittently In-Reply-To: References: <57CD1FDC.2010109@redhat.com> <57CD5488.60404@redhat.com> <57CD659B.6050707@redhat.com> <57CD68BD.9030802@redhat.com> <57CE8CDB.6000809@redhat.com> <57CE9DE5.6020301@redhat.com> Message-ID: <57D0758E.6070406@redhat.com> Rakesh Rajasekharan wrote: > I feel since I migrated from my earlier openldap.. There might be some > issues with the indexes.. So may be creating fresh users might help.. > Is there a way to read clear text user password.. That way I could > have just recreated the users and set them the older password... Migrated users are added one-by-one in pretty much the same way that ipa user-add works so there shouldn't be any issues with the indexes, at least not due to the migration. There is no way to retrieve the cleartext password. rob > > Just trying what ever I could think of to fix this :) > > > On Sep 7, 2016 5:16 PM, "Rakesh Rajasekharan" > > > wrote: > > I changed the nsslapd-db-checkpoint interval first to 80 and > increased till 160.. however the hang has not gone away yet > > > I migrated my data from a openldap, can that cause issues like > this.. If so, i could try creating users from scratch directly on > IPA rather than migrating and test it > > But before that , have taken a fresh pstack just incase if it helps > to nail down the issue > > Thread 44 (Thread 0x7f53ed52f700 (LWP 128165)): > #0 0x00007f54047649b3 in select () from /lib64/libc.so.6 > #1 0x00007f5406eef0e9 in DS_Sleep () from > /usr/lib64/dirsrv/libslapd.so.0 > #2 0x00007f53fa53d907 in deadlock_threadmain () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #3 0x00007f540509e7bb in _pt_root () from /lib64/libnspr4.so > #4 0x00007f5404a3fdc5 in start_thread () from /lib64/libpthread.so.0 > #5 0x00007f540476d28d in clone () from /lib64/libc.so.6 > Thread 43 (Thread 0x7f53ecd2e700 (LWP 128166)): > #0 0x00007f54047649b3 in select () from /lib64/libc.so.6 > #1 0x00007f53ff0cbbad in __os_yield () from /lib64/libdb-5.3.so > > #2 0x00007f53ff0c72b3 in __memp_sync_int () from > /lib64/libdb-5.3.so > #3 0x00007f53ff0d7752 in __txn_checkpoint () from > /lib64/libdb-5.3.so > #4 0x00007f53ff0d7b74 in __txn_checkpoint_pp () from > /lib64/libdb-5.3.so > #5 0x00007f53fa541a87 in checkpoint_threadmain () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #6 0x00007f540509e7bb in _pt_root () from /lib64/libnspr4.so > #7 0x00007f5404a3fdc5 in start_thread () from /lib64/libpthread.so.0 > #8 0x00007f540476d28d in clone () from /lib64/libc.so.6 > Thread 42 (Thread 0x7f53ec52d700 (LWP 128167)): > #0 0x00007f54047649b3 in select () from /lib64/libc.so.6 > #1 0x00007f5406eef0e9 in DS_Sleep () from > /usr/lib64/dirsrv/libslapd.so.0 > #2 0x00007f53fa53db7f in trickle_threadmain () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #3 0x00007f540509e7bb in _pt_root () from /lib64/libnspr4.so > #4 0x00007f5404a3fdc5 in start_thread () from /lib64/libpthread.so.0 > #5 0x00007f540476d28d in clone () from /lib64/libc.so.6 > Thread 41 (Thread 0x7f53ebd2c700 (LWP 128168)): > #0 0x00007f54047649b3 in select () from /lib64/libc.so.6 > #1 0x00007f5406eef0e9 in DS_Sleep () from > /usr/lib64/dirsrv/libslapd.so.0 > #2 0x00007f53fa538707 in perf_threadmain () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #3 0x00007f540509e7bb in _pt_root () from /lib64/libnspr4.so > #4 0x00007f5404a3fdc5 in start_thread () from /lib64/libpthread.so.0 > #5 0x00007f540476d28d in clone () from /lib64/libc.so.6 > Thread 40 (Thread 0x7f53eb322700 (LWP 128204)): > #0 0x00007f5404a436d5 in pthread_cond_wait@@GLIBC_2.3.2 () from > /lib64/libpthread.so.0 > #1 0x00007f5405099050 in PR_WaitCondVar () from /lib64/libnspr4.so > #2 0x00007f5406ede198 in slapi_wait_condvar () from > /usr/lib64/dirsrv/libslapd.so.0 > #3 0x00007f53fd27062e in cos_cache_wait_on_change () from > /usr/lib64/dirsrv/plugins/libcos-plugin.so > #4 0x00007f540509e7bb in _pt_root () from /lib64/libnspr4.so > #5 0x00007f5404a3fdc5 in start_thread () from /lib64/libpthread.so.0 > #6 0x00007f540476d28d in clone () from /lib64/libc.so.6 > Thread 39 (Thread 0x7f53eab21700 (LWP 128206)): > #0 0x00007f5404a436d5 in pthread_cond_wait@@GLIBC_2.3.2 () from > /lib64/libpthread.so.0 > #1 0x00007f5405099050 in PR_WaitCondVar () from /lib64/libnspr4.so > #2 0x00007f5406ede198 in slapi_wait_condvar () from > /usr/lib64/dirsrv/libslapd.so.0 > #3 0x00007f53f8bdbeed in roles_cache_wait_on_change () from > /usr/lib64/dirsrv/plugins/libroles-plugin.so > #4 0x00007f540509e7bb in _pt_root () from /lib64/libnspr4.so > #5 0x00007f5404a3fdc5 in start_thread () from /lib64/libpthread.so.0 > #6 0x00007f540476d28d in clone () from /lib64/libc.so.6 > Thread 38 (Thread 0x7f53ea320700 (LWP 128207)): > #0 0x00007f5404a436d5 in pthread_cond_wait@@GLIBC_2.3.2 () from > /lib64/libpthread.so.0 > #1 0x00007f5405099050 in PR_WaitCondVar () from /lib64/libnspr4.so > #2 0x00007f5406ede198 in slapi_wait_condvar () from > /usr/lib64/dirsrv/libslapd.so.0 > #3 0x00007f53f8bdbeed in roles_cache_wait_on_change () from > /usr/lib64/dirsrv/plugins/libroles-plugin.so > #4 0x00007f540509e7bb in _pt_root () from /lib64/libnspr4.so > #5 0x00007f5404a3fdc5 in start_thread () from /lib64/libpthread.so.0 > #6 0x00007f540476d28d in clone () from /lib64/libc.so.6 > Thread 37 (Thread 0x7f53e9b1f700 (LWP 128208)): > #0 0x00007f5404a436d5 in pthread_cond_wait@@GLIBC_2.3.2 () from > /lib64/libpthread.so.0 > #1 0x00007f5405099050 in PR_WaitCondVar () from /lib64/libnspr4.so > #2 0x00007f5406ede198 in slapi_wait_condvar () from > /usr/lib64/dirsrv/libslapd.so.0 > #3 0x00007f53f8bdbeed in roles_cache_wait_on_change () from > /usr/lib64/dirsrv/plugins/libroles-plugin.so > #4 0x00007f540509e7bb in _pt_root () from /lib64/libnspr4.so > #5 0x00007f5404a3fdc5 in start_thread () from /lib64/libpthread.so.0 > #6 0x00007f540476d28d in clone () from /lib64/libc.so.6 > Thread 36 (Thread 0x7f53e931e700 (LWP 128209)): > #0 0x00007f5404a43a82 in pthread_cond_timedwait@@GLIBC_2.3.2 () > from /lib64/libpthread.so.0 > #1 0x00007f5405098b07 in pt_TimedWait () from /lib64/libnspr4.so > #2 0x00007f5405098fce in PR_WaitCondVar () from /lib64/libnspr4.so > #3 0x00007f5407397d03 in housecleaning () > #4 0x00007f540509e7bb in _pt_root () from /lib64/libnspr4.so > #5 0x00007f5404a3fdc5 in start_thread () from /lib64/libpthread.so.0 > #6 0x00007f540476d28d in clone () from /lib64/libc.so.6 > Thread 35 (Thread 0x7f53e8b1d700 (LWP 128210)): > #0 0x00007f5404a43a82 in pthread_cond_timedwait@@GLIBC_2.3.2 () > from /lib64/libpthread.so.0 > #1 0x00007f5405098b07 in pt_TimedWait () from /lib64/libnspr4.so > #2 0x00007f5405098fce in PR_WaitCondVar () from /lib64/libnspr4.so > #3 0x00007f5406e841e8 in eq_loop () from > /usr/lib64/dirsrv/libslapd.so.0 > #4 0x00007f540509e7bb in _pt_root () from /lib64/libnspr4.so > #5 0x00007f5404a3fdc5 in start_thread () from /lib64/libpthread.so.0 > #6 0x00007f540476d28d in clone () from /lib64/libc.so.6 > Thread 34 (Thread 0x7f53db7fe700 (LWP 128212)): > #0 0x00007f5404a436d5 in pthread_cond_wait@@GLIBC_2.3.2 () from > /lib64/libpthread.so.0 > #1 0x00007f53fef9f2f3 in __db_hybrid_mutex_suspend () from > /lib64/libdb-5.3.so > #2 0x00007f53fef9e640 in __db_tas_mutex_lock () from > /lib64/libdb-5.3.so > #3 0x00007f53ff048d3a in __lock_get_internal () from > /lib64/libdb-5.3.so > #4 0x00007f53ff049820 in __lock_get () from /lib64/libdb-5.3.so > > #5 0x00007f53ff075142 in __db_lget () from /lib64/libdb-5.3.so > > #6 0x00007f53fefbc605 in __bam_search () from /lib64/libdb-5.3.so > > #7 0x00007f53fefa7256 in __bamc_search () from /lib64/libdb-5.3.so > > #8 0x00007f53fefa8d0f in __bamc_get () from /lib64/libdb-5.3.so > > #9 0x00007f53ff061ca6 in __dbc_iget () from /lib64/libdb-5.3.so > > #10 0x00007f53ff070b02 in __dbc_get_pp () from /lib64/libdb-5.3.so > > #11 0x00007f53fa54c680 in idl_new_fetch () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #12 0x00007f53fa55ab26 in index_read_ext_allids () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #13 0x00007f53fa545284 in keys2idl () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #14 0x00007f53fa5459e3 in ava_candidates.isra.0 () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #15 0x00007f53fa545fd2 in filter_candidates_ext () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #16 0x00007f53fa547096 in list_candidates () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #17 0x00007f53fa545f40 in filter_candidates_ext () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #18 0x00007f53fa58180c in subtree_candidates () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #19 0x00007f53fa582e7f in ldbm_back_search () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #20 0x00007f5406eb01d6 in op_shared_search () from > /usr/lib64/dirsrv/libslapd.so.0 > #21 0x00007f54073a133e in do_search () > #22 0x00007f540738f567 in connection_threadmain () > #23 0x00007f540509e7bb in _pt_root () from /lib64/libnspr4.so > #24 0x00007f5404a3fdc5 in start_thread () from /lib64/libpthread.so.0 > #25 0x00007f540476d28d in clone () from /lib64/libc.so.6 > Thread 33 (Thread 0x7f53daffd700 (LWP 128213)): > #0 0x00007f5404a436d5 in pthread_cond_wait@@GLIBC_2.3.2 () from > /lib64/libpthread.so.0 > #1 0x00007f53fef9f2f3 in __db_hybrid_mutex_suspend () from > /lib64/libdb-5.3.so > #2 0x00007f53fef9e640 in __db_tas_mutex_lock () from > /lib64/libdb-5.3.so > #3 0x00007f53ff048d3a in __lock_get_internal () from > /lib64/libdb-5.3.so > #4 0x00007f53ff049820 in __lock_get () from /lib64/libdb-5.3.so > > #5 0x00007f53ff075142 in __db_lget () from /lib64/libdb-5.3.so > > #6 0x00007f53fefbc605 in __bam_search () from /lib64/libdb-5.3.so > > #7 0x00007f53fefa7256 in __bamc_search () from /lib64/libdb-5.3.so > > #8 0x00007f53fefa8d0f in __bamc_get () from /lib64/libdb-5.3.so > > #9 0x00007f53ff061ca6 in __dbc_iget () from /lib64/libdb-5.3.so > > #10 0x00007f53ff070b02 in __dbc_get_pp () from /lib64/libdb-5.3.so > > #11 0x00007f53fa54c680 in idl_new_fetch () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #12 0x00007f53fa55ab26 in index_read_ext_allids () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #13 0x00007f53fa545284 in keys2idl () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #14 0x00007f53fa5459e3 in ava_candidates.isra.0 () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #15 0x00007f53fa545fd2 in filter_candidates_ext () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #16 0x00007f53fa547096 in list_candidates () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #17 0x00007f53fa545f40 in filter_candidates_ext () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #18 0x00007f53fa58180c in subtree_candidates () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #19 0x00007f53fa582e7f in ldbm_back_search () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #20 0x00007f5406eb01d6 in op_shared_search () from > /usr/lib64/dirsrv/libslapd.so.0 > #21 0x00007f54073a133e in do_search () > #22 0x00007f540738f567 in connection_threadmain () > #23 0x00007f540509e7bb in _pt_root () from /lib64/libnspr4.so > #24 0x00007f5404a3fdc5 in start_thread () from /lib64/libpthread.so.0 > #25 0x00007f540476d28d in clone () from /lib64/libc.so.6 > Thread 32 (Thread 0x7f53da7fc700 (LWP 128214)): > #0 0x00007f5404a436d5 in pthread_cond_wait@@GLIBC_2.3.2 () from > /lib64/libpthread.so.0 > #1 0x00007f53fef9f2f3 in __db_hybrid_mutex_suspend () from > /lib64/libdb-5.3.so > #2 0x00007f53fef9e640 in __db_tas_mutex_lock () from > /lib64/libdb-5.3.so > #3 0x00007f53ff048d3a in __lock_get_internal () from > /lib64/libdb-5.3.so > #4 0x00007f53ff049820 in __lock_get () from /lib64/libdb-5.3.so > > #5 0x00007f53ff075142 in __db_lget () from /lib64/libdb-5.3.so > > #6 0x00007f53fefbc605 in __bam_search () from /lib64/libdb-5.3.so > > #7 0x00007f53fefa7256 in __bamc_search () from /lib64/libdb-5.3.so > > #8 0x00007f53fefa8d0f in __bamc_get () from /lib64/libdb-5.3.so > > #9 0x00007f53ff061ca6 in __dbc_iget () from /lib64/libdb-5.3.so > > #10 0x00007f53ff070b02 in __dbc_get_pp () from /lib64/libdb-5.3.so > > #11 0x00007f53fa54c680 in idl_new_fetch () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #12 0x00007f53fa55ab26 in index_read_ext_allids () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #13 0x00007f53fa545284 in keys2idl () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #14 0x00007f53fa5459e3 in ava_candidates.isra.0 () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #15 0x00007f53fa545fd2 in filter_candidates_ext () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #16 0x00007f53fa547096 in list_candidates () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #17 0x00007f53fa545f40 in filter_candidates_ext () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #18 0x00007f53fa547096 in list_candidates () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #19 0x00007f53fa545f40 in filter_candidates_ext () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #20 0x00007f53fa547096 in list_candidates () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #21 0x00007f53fa545f40 in filter_candidates_ext () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #22 0x00007f53fa58180c in subtree_candidates () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #23 0x00007f53fa582e7f in ldbm_back_search () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #24 0x00007f5406eb01d6 in op_shared_search () from > /usr/lib64/dirsrv/libslapd.so.0 > #25 0x00007f54073a133e in do_search () > #26 0x00007f540738f3c3 in connection_threadmain () > #27 0x00007f540509e7bb in _pt_root () from /lib64/libnspr4.so > #28 0x00007f5404a3fdc5 in start_thread () from /lib64/libpthread.so.0 > #29 0x00007f540476d28d in clone () from /lib64/libc.so.6 > Thread 31 (Thread 0x7f53d9ffb700 (LWP 128215)): > #0 0x00007f5404a436d5 in pthread_cond_wait@@GLIBC_2.3.2 () from > /lib64/libpthread.so.0 > #1 0x00007f53fef9f2f3 in __db_hybrid_mutex_suspend () from > /lib64/libdb-5.3.so > #2 0x00007f53fef9e640 in __db_tas_mutex_lock () from > /lib64/libdb-5.3.so > #3 0x00007f53ff048d3a in __lock_get_internal () from > /lib64/libdb-5.3.so > #4 0x00007f53ff049820 in __lock_get () from /lib64/libdb-5.3.so > > #5 0x00007f53ff075142 in __db_lget () from /lib64/libdb-5.3.so > > #6 0x00007f53fefbc605 in __bam_search () from /lib64/libdb-5.3.so > > #7 0x00007f53fefa7256 in __bamc_search () from /lib64/libdb-5.3.so > > #8 0x00007f53fefa8d0f in __bamc_get () from /lib64/libdb-5.3.so > > #9 0x00007f53ff061ca6 in __dbc_iget () from /lib64/libdb-5.3.so > > #10 0x00007f53ff070b02 in __dbc_get_pp () from /lib64/libdb-5.3.so > > #11 0x00007f53fa54c680 in idl_new_fetch () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #12 0x00007f53fa55ab26 in index_read_ext_allids () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #13 0x00007f53fa545284 in keys2idl () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #14 0x00007f53fa5459e3 in ava_candidates.isra.0 () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #15 0x00007f53fa545fd2 in filter_candidates_ext () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #16 0x00007f53fa547096 in list_candidates () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #17 0x00007f53fa545f40 in filter_candidates_ext () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #18 0x00007f53fa58180c in subtree_candidates () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #19 0x00007f53fa582e7f in ldbm_back_search () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #20 0x00007f5406eb01d6 in op_shared_search () from > /usr/lib64/dirsrv/libslapd.so.0 > #21 0x00007f54073a133e in do_search () > #22 0x00007f540738f567 in connection_threadmain () > #23 0x00007f540509e7bb in _pt_root () from /lib64/libnspr4.so > #24 0x00007f5404a3fdc5 in start_thread () from /lib64/libpthread.so.0 > #25 0x00007f540476d28d in clone () from /lib64/libc.so.6 > Thread 30 (Thread 0x7f53d97fa700 (LWP 128216)): > #0 0x00007f5404a436d5 in pthread_cond_wait@@GLIBC_2.3.2 () from > /lib64/libpthread.so.0 > #1 0x00007f53fef9f2f3 in __db_hybrid_mutex_suspend () from > /lib64/libdb-5.3.so > #2 0x00007f53fef9e640 in __db_tas_mutex_lock () from > /lib64/libdb-5.3.so > #3 0x00007f53ff048d3a in __lock_get_internal () from > /lib64/libdb-5.3.so > #4 0x00007f53ff049820 in __lock_get () from /lib64/libdb-5.3.so > > #5 0x00007f53ff075142 in __db_lget () from /lib64/libdb-5.3.so > > #6 0x00007f53fefbc605 in __bam_search () from /lib64/libdb-5.3.so > > #7 0x00007f53fefa7256 in __bamc_search () from /lib64/libdb-5.3.so > > #8 0x00007f53fefa8d0f in __bamc_get () from /lib64/libdb-5.3.so > > #9 0x00007f53ff061ca6 in __dbc_iget () from /lib64/libdb-5.3.so > > #10 0x00007f53ff070b02 in __dbc_get_pp () from /lib64/libdb-5.3.so > > #11 0x00007f53fa54c680 in idl_new_fetch () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #12 0x00007f53fa55ab26 in index_read_ext_allids () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #13 0x00007f53fa545284 in keys2idl () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #14 0x00007f53fa5459e3 in ava_candidates.isra.0 () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #15 0x00007f53fa545fd2 in filter_candidates_ext () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #16 0x00007f53fa547096 in list_candidates () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #17 0x00007f53fa545f40 in filter_candidates_ext () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #18 0x00007f53fa58180c in subtree_candidates () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #19 0x00007f53fa582e7f in ldbm_back_search () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #20 0x00007f5406eb01d6 in op_shared_search () from > /usr/lib64/dirsrv/libslapd.so.0 > #21 0x00007f54073a133e in do_search () > #22 0x00007f540738f567 in connection_threadmain () > #23 0x00007f540509e7bb in _pt_root () from /lib64/libnspr4.so > #24 0x00007f5404a3fdc5 in start_thread () from /lib64/libpthread.so.0 > #25 0x00007f540476d28d in clone () from /lib64/libc.so.6 > Thread 29 (Thread 0x7f53d8ff9700 (LWP 128217)): > #0 0x00007f5404a436d5 in pthread_cond_wait@@GLIBC_2.3.2 () from > /lib64/libpthread.so.0 > #1 0x00007f53fef9f2f3 in __db_hybrid_mutex_suspend () from > /lib64/libdb-5.3.so > #2 0x00007f53fef9e640 in __db_tas_mutex_lock () from > /lib64/libdb-5.3.so > #3 0x00007f53ff048d3a in __lock_get_internal () from > /lib64/libdb-5.3.so > #4 0x00007f53ff049820 in __lock_get () from /lib64/libdb-5.3.so > > #5 0x00007f53ff075142 in __db_lget () from /lib64/libdb-5.3.so > > #6 0x00007f53fefbc605 in __bam_search () from /lib64/libdb-5.3.so > > #7 0x00007f53fefa7256 in __bamc_search () from /lib64/libdb-5.3.so > > #8 0x00007f53fefa8d0f in __bamc_get () from /lib64/libdb-5.3.so > > #9 0x00007f53ff061ca6 in __dbc_iget () from /lib64/libdb-5.3.so > > #10 0x00007f53ff070b02 in __dbc_get_pp () from /lib64/libdb-5.3.so > > #11 0x00007f53fa54c680 in idl_new_fetch () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #12 0x00007f53fa55ab26 in index_read_ext_allids () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #13 0x00007f53fa545284 in keys2idl () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #14 0x00007f53fa5459e3 in ava_candidates.isra.0 () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #15 0x00007f53fa545fd2 in filter_candidates_ext () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #16 0x00007f53fa547096 in list_candidates () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #17 0x00007f53fa545f40 in filter_candidates_ext () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #18 0x00007f53fa58180c in subtree_candidates () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #19 0x00007f53fa582e7f in ldbm_back_search () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #20 0x00007f5406eb01d6 in op_shared_search () from > /usr/lib64/dirsrv/libslapd.so.0 > #21 0x00007f54073a133e in do_search () > #22 0x00007f540738f567 in connection_threadmain () > #23 0x00007f540509e7bb in _pt_root () from /lib64/libnspr4.so > #24 0x00007f5404a3fdc5 in start_thread () from /lib64/libpthread.so.0 > #25 0x00007f540476d28d in clone () from /lib64/libc.so.6 > Thread 28 (Thread 0x7f53d87f8700 (LWP 128218)): > #0 0x00007f5404a436d5 in pthread_cond_wait@@GLIBC_2.3.2 () from > /lib64/libpthread.so.0 > #1 0x00007f53fef9f2f3 in __db_hybrid_mutex_suspend () from > /lib64/libdb-5.3.so > #2 0x00007f53fef9e640 in __db_tas_mutex_lock () from > /lib64/libdb-5.3.so > #3 0x00007f53ff048d3a in __lock_get_internal () from > /lib64/libdb-5.3.so > #4 0x00007f53ff049820 in __lock_get () from /lib64/libdb-5.3.so > > #5 0x00007f53ff075142 in __db_lget () from /lib64/libdb-5.3.so > > #6 0x00007f53fefbc605 in __bam_search () from /lib64/libdb-5.3.so > > #7 0x00007f53fefa7256 in __bamc_search () from /lib64/libdb-5.3.so > > #8 0x00007f53fefa8d0f in __bamc_get () from /lib64/libdb-5.3.so > > #9 0x00007f53ff061ca6 in __dbc_iget () from /lib64/libdb-5.3.so > > #10 0x00007f53ff070b02 in __dbc_get_pp () from /lib64/libdb-5.3.so > > #11 0x00007f53fa54c680 in idl_new_fetch () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #12 0x00007f53fa55ab26 in index_read_ext_allids () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #13 0x00007f53fa545284 in keys2idl () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #14 0x00007f53fa5459e3 in ava_candidates.isra.0 () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #15 0x00007f53fa545fd2 in filter_candidates_ext () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #16 0x00007f53fa547096 in list_candidates () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #17 0x00007f53fa545f40 in filter_candidates_ext () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #18 0x00007f53fa58180c in subtree_candidates () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #19 0x00007f53fa582e7f in ldbm_back_search () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #20 0x00007f5406eb01d6 in op_shared_search () from > /usr/lib64/dirsrv/libslapd.so.0 > #21 0x00007f54073a133e in do_search () > #22 0x00007f540738f567 in connection_threadmain () > #23 0x00007f540509e7bb in _pt_root () from /lib64/libnspr4.so > #24 0x00007f5404a3fdc5 in start_thread () from /lib64/libpthread.so.0 > #25 0x00007f540476d28d in clone () from /lib64/libc.so.6 > Thread 27 (Thread 0x7f53d7ff7700 (LWP 128219)): > #0 0x00007f5404a436d5 in pthread_cond_wait@@GLIBC_2.3.2 () from > /lib64/libpthread.so.0 > #1 0x00007f53fef9f2f3 in __db_hybrid_mutex_suspend () from > /lib64/libdb-5.3.so > #2 0x00007f53fef9e640 in __db_tas_mutex_lock () from > /lib64/libdb-5.3.so > #3 0x00007f53ff048d3a in __lock_get_internal () from > /lib64/libdb-5.3.so > #4 0x00007f53ff049820 in __lock_get () from /lib64/libdb-5.3.so > > #5 0x00007f53ff075142 in __db_lget () from /lib64/libdb-5.3.so > > #6 0x00007f53fefbc605 in __bam_search () from /lib64/libdb-5.3.so > > #7 0x00007f53fefa7256 in __bamc_search () from /lib64/libdb-5.3.so > > #8 0x00007f53fefa8d0f in __bamc_get () from /lib64/libdb-5.3.so > > #9 0x00007f53ff061ca6 in __dbc_iget () from /lib64/libdb-5.3.so > > #10 0x00007f53ff070b02 in __dbc_get_pp () from /lib64/libdb-5.3.so > > #11 0x00007f53fa54c680 in idl_new_fetch () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #12 0x00007f53fa55ab26 in index_read_ext_allids () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #13 0x00007f53fa545284 in keys2idl () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #14 0x00007f53fa5459e3 in ava_candidates.isra.0 () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #15 0x00007f53fa545fd2 in filter_candidates_ext () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #16 0x00007f53fa547096 in list_candidates () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #17 0x00007f53fa545f40 in filter_candidates_ext () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #18 0x00007f53fa58180c in subtree_candidates () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #19 0x00007f53fa582e7f in ldbm_back_search () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #20 0x00007f5406eb01d6 in op_shared_search () from > /usr/lib64/dirsrv/libslapd.so.0 > #21 0x00007f54073a133e in do_search () > #22 0x00007f540738f567 in connection_threadmain () > #23 0x00007f540509e7bb in _pt_root () from /lib64/libnspr4.so > #24 0x00007f5404a3fdc5 in start_thread () from /lib64/libpthread.so.0 > #25 0x00007f540476d28d in clone () from /lib64/libc.so.6 > Thread 26 (Thread 0x7f53d77f6700 (LWP 128220)): > #0 0x00007f5404a436d5 in pthread_cond_wait@@GLIBC_2.3.2 () from > /lib64/libpthread.so.0 > #1 0x00007f53fef9f2f3 in __db_hybrid_mutex_suspend () from > /lib64/libdb-5.3.so > #2 0x00007f53fef9e640 in __db_tas_mutex_lock () from > /lib64/libdb-5.3.so > #3 0x00007f53ff048d3a in __lock_get_internal () from > /lib64/libdb-5.3.so > #4 0x00007f53ff049820 in __lock_get () from /lib64/libdb-5.3.so > > #5 0x00007f53ff075142 in __db_lget () from /lib64/libdb-5.3.so > > #6 0x00007f53fefbc605 in __bam_search () from /lib64/libdb-5.3.so > > #7 0x00007f53fefa7256 in __bamc_search () from /lib64/libdb-5.3.so > > #8 0x00007f53fefa8d0f in __bamc_get () from /lib64/libdb-5.3.so > > #9 0x00007f53ff061ca6 in __dbc_iget () from /lib64/libdb-5.3.so > > #10 0x00007f53ff070b02 in __dbc_get_pp () from /lib64/libdb-5.3.so > > #11 0x00007f53fa54c680 in idl_new_fetch () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #12 0x00007f53fa55ab26 in index_read_ext_allids () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #13 0x00007f53fa545284 in keys2idl () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #14 0x00007f53fa5459e3 in ava_candidates.isra.0 () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #15 0x00007f53fa545fd2 in filter_candidates_ext () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #16 0x00007f53fa547096 in list_candidates () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #17 0x00007f53fa545f40 in filter_candidates_ext () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #18 0x00007f53fa58180c in subtree_candidates () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #19 0x00007f53fa582e7f in ldbm_back_search () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #20 0x00007f5406eb01d6 in op_shared_search () from > /usr/lib64/dirsrv/libslapd.so.0 > #21 0x00007f54073a133e in do_search () > #22 0x00007f540738f567 in connection_threadmain () > #23 0x00007f540509e7bb in _pt_root () from /lib64/libnspr4.so > #24 0x00007f5404a3fdc5 in start_thread () from /lib64/libpthread.so.0 > #25 0x00007f540476d28d in clone () from /lib64/libc.so.6 > Thread 25 (Thread 0x7f53d6ff5700 (LWP 128221)): > #0 0x00007f5404a436d5 in pthread_cond_wait@@GLIBC_2.3.2 () from > /lib64/libpthread.so.0 > #1 0x00007f53fef9f2f3 in __db_hybrid_mutex_suspend () from > /lib64/libdb-5.3.so > #2 0x00007f53fef9e640 in __db_tas_mutex_lock () from > /lib64/libdb-5.3.so > #3 0x00007f53ff048d3a in __lock_get_internal () from > /lib64/libdb-5.3.so > #4 0x00007f53ff049820 in __lock_get () from /lib64/libdb-5.3.so > > #5 0x00007f53ff075142 in __db_lget () from /lib64/libdb-5.3.so > > #6 0x00007f53fefbc605 in __bam_search () from /lib64/libdb-5.3.so > > #7 0x00007f53fefa7256 in __bamc_search () from /lib64/libdb-5.3.so > > #8 0x00007f53fefa8d0f in __bamc_get () from /lib64/libdb-5.3.so > > #9 0x00007f53ff061ca6 in __dbc_iget () from /lib64/libdb-5.3.so > > #10 0x00007f53ff070b02 in __dbc_get_pp () from /lib64/libdb-5.3.so > > #11 0x00007f53fa54c680 in idl_new_fetch () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #12 0x00007f53fa55ab26 in index_read_ext_allids () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #13 0x00007f53fa545284 in keys2idl () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #14 0x00007f53fa5459e3 in ava_candidates.isra.0 () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #15 0x00007f53fa545fd2 in filter_candidates_ext () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #16 0x00007f53fa547096 in list_candidates () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #17 0x00007f53fa545f40 in filter_candidates_ext () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #18 0x00007f53fa58180c in subtree_candidates () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #19 0x00007f53fa582e7f in ldbm_back_search () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #20 0x00007f5406eb01d6 in op_shared_search () from > /usr/lib64/dirsrv/libslapd.so.0 > #21 0x00007f54073a133e in do_search () > #22 0x00007f540738f567 in connection_threadmain () > #23 0x00007f540509e7bb in _pt_root () from /lib64/libnspr4.so > #24 0x00007f5404a3fdc5 in start_thread () from /lib64/libpthread.so.0 > #25 0x00007f540476d28d in clone () from /lib64/libc.so.6 > Thread 24 (Thread 0x7f53d67f4700 (LWP 128222)): > #0 0x00007f5404a436d5 in pthread_cond_wait@@GLIBC_2.3.2 () from > /lib64/libpthread.so.0 > #1 0x00007f53fef9f2f3 in __db_hybrid_mutex_suspend () from > /lib64/libdb-5.3.so > #2 0x00007f53fef9e640 in __db_tas_mutex_lock () from > /lib64/libdb-5.3.so > #3 0x00007f53ff048d3a in __lock_get_internal () from > /lib64/libdb-5.3.so > #4 0x00007f53ff049820 in __lock_get () from /lib64/libdb-5.3.so > > #5 0x00007f53ff075142 in __db_lget () from /lib64/libdb-5.3.so > > #6 0x00007f53fefbc605 in __bam_search () from /lib64/libdb-5.3.so > > #7 0x00007f53fefa7256 in __bamc_search () from /lib64/libdb-5.3.so > > #8 0x00007f53fefa8d0f in __bamc_get () from /lib64/libdb-5.3.so > > #9 0x00007f53ff061ca6 in __dbc_iget () from /lib64/libdb-5.3.so > > #10 0x00007f53ff070b02 in __dbc_get_pp () from /lib64/libdb-5.3.so > > #11 0x00007f53fa54c680 in idl_new_fetch () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #12 0x00007f53fa55ab26 in index_read_ext_allids () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #13 0x00007f53fa545284 in keys2idl () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #14 0x00007f53fa5459e3 in ava_candidates.isra.0 () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #15 0x00007f53fa545fd2 in filter_candidates_ext () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #16 0x00007f53fa547096 in list_candidates () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #17 0x00007f53fa545f40 in filter_candidates_ext () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #18 0x00007f53fa58180c in subtree_candidates () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #19 0x00007f53fa582e7f in ldbm_back_search () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #20 0x00007f5406eb01d6 in op_shared_search () from > /usr/lib64/dirsrv/libslapd.so.0 > #21 0x00007f54073a133e in do_search () > #22 0x00007f540738f567 in connection_threadmain () > #23 0x00007f540509e7bb in _pt_root () from /lib64/libnspr4.so > #24 0x00007f5404a3fdc5 in start_thread () from /lib64/libpthread.so.0 > #25 0x00007f540476d28d in clone () from /lib64/libc.so.6 > Thread 23 (Thread 0x7f53d5ff3700 (LWP 128223)): > #0 0x00007f5404a436d5 in pthread_cond_wait@@GLIBC_2.3.2 () from > /lib64/libpthread.so.0 > #1 0x00007f53fef9f2f3 in __db_hybrid_mutex_suspend () from > /lib64/libdb-5.3.so > #2 0x00007f53fef9e640 in __db_tas_mutex_lock () from > /lib64/libdb-5.3.so > #3 0x00007f53ff0b8e1b in __memp_fget () from /lib64/libdb-5.3.so > > #4 0x00007f53fefbc211 in __bam_search () from /lib64/libdb-5.3.so > > #5 0x00007f53fefa7256 in __bamc_search () from /lib64/libdb-5.3.so > > #6 0x00007f53fefab6c4 in __bamc_put () from /lib64/libdb-5.3.so > > #7 0x00007f53ff0626d5 in __dbc_iput () from /lib64/libdb-5.3.so > > #8 0x00007f53ff05da0e in __db_put () from /lib64/libdb-5.3.so > > #9 0x00007f53ff072fa4 in __db_put_pp () from /lib64/libdb-5.3.so > > #10 0x00007f53fa54d4cf in idl_new_insert_key () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #11 0x00007f53fa55c5b5 in addordel_values_sv.isra.3 () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #12 0x00007f53fa55d25a in index_addordel_values_ext_sv () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #13 0x00007f53fa55d694 in index_addordel_values_sv () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #14 0x00007f53fa55d75a in index_addordel_entry () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #15 0x00007f53fa5605fa in ldbm_back_add () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #16 0x00007f5406e62cca in op_shared_add () from > /usr/lib64/dirsrv/libslapd.so.0 > #17 0x00007f5406e64020 in do_add () from /usr/lib64/dirsrv/libslapd.so.0 > #18 0x00007f540738f303 in connection_threadmain () > #19 0x00007f540509e7bb in _pt_root () from /lib64/libnspr4.so > #20 0x00007f5404a3fdc5 in start_thread () from /lib64/libpthread.so.0 > #21 0x00007f540476d28d in clone () from /lib64/libc.so.6 > Thread 22 (Thread 0x7f53d57f2700 (LWP 128224)): > #0 0x00007f5404a436d5 in pthread_cond_wait@@GLIBC_2.3.2 () from > /lib64/libpthread.so.0 > #1 0x00007f53fef9f2f3 in __db_hybrid_mutex_suspend () from > /lib64/libdb-5.3.so > #2 0x00007f53fef9e640 in __db_tas_mutex_lock () from > /lib64/libdb-5.3.so > #3 0x00007f53ff048d3a in __lock_get_internal () from > /lib64/libdb-5.3.so > #4 0x00007f53ff049820 in __lock_get () from /lib64/libdb-5.3.so > > #5 0x00007f53ff075142 in __db_lget () from /lib64/libdb-5.3.so > > #6 0x00007f53fefbc605 in __bam_search () from /lib64/libdb-5.3.so > > #7 0x00007f53fefa7256 in __bamc_search () from /lib64/libdb-5.3.so > > #8 0x00007f53fefa8d0f in __bamc_get () from /lib64/libdb-5.3.so > > #9 0x00007f53ff061ca6 in __dbc_iget () from /lib64/libdb-5.3.so > > #10 0x00007f53ff070b02 in __dbc_get_pp () from /lib64/libdb-5.3.so > > #11 0x00007f53fa54c680 in idl_new_fetch () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #12 0x00007f53fa55ab26 in index_read_ext_allids () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #13 0x00007f53fa545284 in keys2idl () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #14 0x00007f53fa5459e3 in ava_candidates.isra.0 () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #15 0x00007f53fa545fd2 in filter_candidates_ext () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #16 0x00007f53fa547096 in list_candidates () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #17 0x00007f53fa545f40 in filter_candidates_ext () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #18 0x00007f53fa58180c in subtree_candidates () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #19 0x00007f53fa582e7f in ldbm_back_search () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #20 0x00007f5406eb01d6 in op_shared_search () from > /usr/lib64/dirsrv/libslapd.so.0 > #21 0x00007f54073a133e in do_search () > #22 0x00007f540738f567 in connection_threadmain () > #23 0x00007f540509e7bb in _pt_root () from /lib64/libnspr4.so > #24 0x00007f5404a3fdc5 in start_thread () from /lib64/libpthread.so.0 > #25 0x00007f540476d28d in clone () from /lib64/libc.so.6 > Thread 21 (Thread 0x7f53d4ff1700 (LWP 128225)): > #0 0x00007f5404a436d5 in pthread_cond_wait@@GLIBC_2.3.2 () from > /lib64/libpthread.so.0 > #1 0x00007f53fef9f2f3 in __db_hybrid_mutex_suspend () from > /lib64/libdb-5.3.so > #2 0x00007f53fef9e640 in __db_tas_mutex_lock () from > /lib64/libdb-5.3.so > #3 0x00007f53ff048d3a in __lock_get_internal () from > /lib64/libdb-5.3.so > #4 0x00007f53ff049820 in __lock_get () from /lib64/libdb-5.3.so > > #5 0x00007f53ff075142 in __db_lget () from /lib64/libdb-5.3.so > > #6 0x00007f53fefbc605 in __bam_search () from /lib64/libdb-5.3.so > > #7 0x00007f53fefa7256 in __bamc_search () from /lib64/libdb-5.3.so > > #8 0x00007f53fefa8d0f in __bamc_get () from /lib64/libdb-5.3.so > > #9 0x00007f53ff061ca6 in __dbc_iget () from /lib64/libdb-5.3.so > > #10 0x00007f53ff070b02 in __dbc_get_pp () from /lib64/libdb-5.3.so > > #11 0x00007f53fa54c680 in idl_new_fetch () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #12 0x00007f53fa55ab26 in index_read_ext_allids () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #13 0x00007f53fa545284 in keys2idl () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #14 0x00007f53fa5459e3 in ava_candidates.isra.0 () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #15 0x00007f53fa545fd2 in filter_candidates_ext () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #16 0x00007f53fa547096 in list_candidates () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #17 0x00007f53fa545f40 in filter_candidates_ext () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #18 0x00007f53fa58180c in subtree_candidates () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #19 0x00007f53fa582e7f in ldbm_back_search () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #20 0x00007f5406eb01d6 in op_shared_search () from > /usr/lib64/dirsrv/libslapd.so.0 > #21 0x00007f54073a133e in do_search () > #22 0x00007f540738f567 in connection_threadmain () > #23 0x00007f540509e7bb in _pt_root () from /lib64/libnspr4.so > #24 0x00007f5404a3fdc5 in start_thread () from /lib64/libpthread.so.0 > #25 0x00007f540476d28d in clone () from /lib64/libc.so.6 > Thread 20 (Thread 0x7f53d47f0700 (LWP 128226)): > #0 0x00007f5404a436d5 in pthread_cond_wait@@GLIBC_2.3.2 () from > /lib64/libpthread.so.0 > #1 0x00007f53fef9f2f3 in __db_hybrid_mutex_suspend () from > /lib64/libdb-5.3.so > #2 0x00007f53fef9e640 in __db_tas_mutex_lock () from > /lib64/libdb-5.3.so > #3 0x00007f53ff048d3a in __lock_get_internal () from > /lib64/libdb-5.3.so > #4 0x00007f53ff049820 in __lock_get () from /lib64/libdb-5.3.so > > #5 0x00007f53ff075142 in __db_lget () from /lib64/libdb-5.3.so > > #6 0x00007f53fefbc605 in __bam_search () from /lib64/libdb-5.3.so > > #7 0x00007f53fefa7256 in __bamc_search () from /lib64/libdb-5.3.so > > #8 0x00007f53fefa8d0f in __bamc_get () from /lib64/libdb-5.3.so > > #9 0x00007f53ff061ca6 in __dbc_iget () from /lib64/libdb-5.3.so > > #10 0x00007f53ff070b02 in __dbc_get_pp () from /lib64/libdb-5.3.so > > #11 0x00007f53fa54c680 in idl_new_fetch () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #12 0x00007f53fa55ab26 in index_read_ext_allids () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #13 0x00007f53fa545284 in keys2idl () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #14 0x00007f53fa5459e3 in ava_candidates.isra.0 () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #15 0x00007f53fa545fd2 in filter_candidates_ext () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #16 0x00007f53fa547096 in list_candidates () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #17 0x00007f53fa545f40 in filter_candidates_ext () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #18 0x00007f53fa58180c in subtree_candidates () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #19 0x00007f53fa582e7f in ldbm_back_search () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #20 0x00007f5406eb01d6 in op_shared_search () from > /usr/lib64/dirsrv/libslapd.so.0 > #21 0x00007f5406ec094e in search_internal_callback_pb () from > /usr/lib64/dirsrv/libslapd.so.0 > #22 0x00007f5406ec0be8 in search_internal_pb () from > /usr/lib64/dirsrv/libslapd.so.0 > #23 0x00007f53fe118e90 in search_one_berval () from > /usr/lib64/dirsrv/plugins/libattr-unique-plugin.so > #24 0x00007f53fe119242 in search () from > /usr/lib64/dirsrv/plugins/libattr-unique-plugin.so > #25 0x00007f53fe11954c in searchAllSubtrees () from > /usr/lib64/dirsrv/plugins/libattr-unique-plugin.so > #26 0x00007f53fe11a015 in preop_add () from > /usr/lib64/dirsrv/plugins/libattr-unique-plugin.so > #27 0x00007f5406ebb848 in plugin_call_func () from > /usr/lib64/dirsrv/libslapd.so.0 > #28 0x00007f5406ebbb13 in plugin_call_plugins () from > /usr/lib64/dirsrv/libslapd.so.0 > #29 0x00007f5406e62c2b in op_shared_add () from > /usr/lib64/dirsrv/libslapd.so.0 > #30 0x00007f5406e64020 in do_add () from /usr/lib64/dirsrv/libslapd.so.0 > #31 0x00007f540738f303 in connection_threadmain () > #32 0x00007f540509e7bb in _pt_root () from /lib64/libnspr4.so > #33 0x00007f5404a3fdc5 in start_thread () from /lib64/libpthread.so.0 > #34 0x00007f540476d28d in clone () from /lib64/libc.so.6 > Thread 19 (Thread 0x7f53d3fef700 (LWP 128227)): > #0 0x00007f5404a436d5 in pthread_cond_wait@@GLIBC_2.3.2 () from > /lib64/libpthread.so.0 > #1 0x00007f53fef9f2f3 in __db_hybrid_mutex_suspend () from > /lib64/libdb-5.3.so > #2 0x00007f53fef9e640 in __db_tas_mutex_lock () from > /lib64/libdb-5.3.so > #3 0x00007f53ff048d3a in __lock_get_internal () from > /lib64/libdb-5.3.so > #4 0x00007f53ff049820 in __lock_get () from /lib64/libdb-5.3.so > > #5 0x00007f53ff075142 in __db_lget () from /lib64/libdb-5.3.so > > #6 0x00007f53fefbc605 in __bam_search () from /lib64/libdb-5.3.so > > #7 0x00007f53fefa7256 in __bamc_search () from /lib64/libdb-5.3.so > > #8 0x00007f53fefa8d0f in __bamc_get () from /lib64/libdb-5.3.so > > #9 0x00007f53ff061ca6 in __dbc_iget () from /lib64/libdb-5.3.so > > #10 0x00007f53ff070b02 in __dbc_get_pp () from /lib64/libdb-5.3.so > > #11 0x00007f53fa54c680 in idl_new_fetch () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #12 0x00007f53fa55ab26 in index_read_ext_allids () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #13 0x00007f53fa545284 in keys2idl () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #14 0x00007f53fa5459e3 in ava_candidates.isra.0 () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #15 0x00007f53fa545fd2 in filter_candidates_ext () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #16 0x00007f53fa547096 in list_candidates () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #17 0x00007f53fa545f40 in filter_candidates_ext () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #18 0x00007f53fa58180c in subtree_candidates () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #19 0x00007f53fa582e7f in ldbm_back_search () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #20 0x00007f5406eb01d6 in op_shared_search () from > /usr/lib64/dirsrv/libslapd.so.0 > #21 0x00007f54073a133e in do_search () > #22 0x00007f540738f567 in connection_threadmain () > #23 0x00007f540509e7bb in _pt_root () from /lib64/libnspr4.so > #24 0x00007f5404a3fdc5 in start_thread () from /lib64/libpthread.so.0 > #25 0x00007f540476d28d in clone () from /lib64/libc.so.6 > Thread 18 (Thread 0x7f53d37ee700 (LWP 128228)): > #0 0x00007f5404a436d5 in pthread_cond_wait@@GLIBC_2.3.2 () from > /lib64/libpthread.so.0 > #1 0x00007f53fef9f2f3 in __db_hybrid_mutex_suspend () from > /lib64/libdb-5.3.so > #2 0x00007f53fef9e640 in __db_tas_mutex_lock () from > /lib64/libdb-5.3.so > #3 0x00007f53ff048d3a in __lock_get_internal () from > /lib64/libdb-5.3.so > #4 0x00007f53ff049820 in __lock_get () from /lib64/libdb-5.3.so > > #5 0x00007f53ff075142 in __db_lget () from /lib64/libdb-5.3.so > > #6 0x00007f53fefbc605 in __bam_search () from /lib64/libdb-5.3.so > > #7 0x00007f53fefa7256 in __bamc_search () from /lib64/libdb-5.3.so > > #8 0x00007f53fefa8d0f in __bamc_get () from /lib64/libdb-5.3.so > > #9 0x00007f53ff061ca6 in __dbc_iget () from /lib64/libdb-5.3.so > > #10 0x00007f53ff070b02 in __dbc_get_pp () from /lib64/libdb-5.3.so > > #11 0x00007f53fa54c680 in idl_new_fetch () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #12 0x00007f53fa55ab26 in index_read_ext_allids () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #13 0x00007f53fa545284 in keys2idl () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #14 0x00007f53fa5459e3 in ava_candidates.isra.0 () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #15 0x00007f53fa545fd2 in filter_candidates_ext () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #16 0x00007f53fa547096 in list_candidates () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #17 0x00007f53fa545f40 in filter_candidates_ext () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #18 0x00007f53fa58180c in subtree_candidates () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #19 0x00007f53fa582e7f in ldbm_back_search () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #20 0x00007f5406eb01d6 in op_shared_search () from > /usr/lib64/dirsrv/libslapd.so.0 > #21 0x00007f54073a133e in do_search () > #22 0x00007f540738f567 in connection_threadmain () > #23 0x00007f540509e7bb in _pt_root () from /lib64/libnspr4.so > #24 0x00007f5404a3fdc5 in start_thread () from /lib64/libpthread.so.0 > #25 0x00007f540476d28d in clone () from /lib64/libc.so.6 > Thread 17 (Thread 0x7f53d2fed700 (LWP 128229)): > #0 0x00007f5404a436d5 in pthread_cond_wait@@GLIBC_2.3.2 () from > /lib64/libpthread.so.0 > #1 0x00007f53fef9f2f3 in __db_hybrid_mutex_suspend () from > /lib64/libdb-5.3.so > #2 0x00007f53fef9e640 in __db_tas_mutex_lock () from > /lib64/libdb-5.3.so > #3 0x00007f53ff048d3a in __lock_get_internal () from > /lib64/libdb-5.3.so > #4 0x00007f53ff049820 in __lock_get () from /lib64/libdb-5.3.so > > #5 0x00007f53ff075142 in __db_lget () from /lib64/libdb-5.3.so > > #6 0x00007f53fefbc605 in __bam_search () from /lib64/libdb-5.3.so > > #7 0x00007f53fefa7256 in __bamc_search () from /lib64/libdb-5.3.so > > #8 0x00007f53fefa8d0f in __bamc_get () from /lib64/libdb-5.3.so > > #9 0x00007f53ff061ca6 in __dbc_iget () from /lib64/libdb-5.3.so > > #10 0x00007f53ff070b02 in __dbc_get_pp () from /lib64/libdb-5.3.so > > #11 0x00007f53fa54c680 in idl_new_fetch () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #12 0x00007f53fa55ab26 in index_read_ext_allids () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #13 0x00007f53fa545284 in keys2idl () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #14 0x00007f53fa5459e3 in ava_candidates.isra.0 () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #15 0x00007f53fa545fd2 in filter_candidates_ext () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #16 0x00007f53fa547096 in list_candidates () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #17 0x00007f53fa545f40 in filter_candidates_ext () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #18 0x00007f53fa58180c in subtree_candidates () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #19 0x00007f53fa582e7f in ldbm_back_search () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #20 0x00007f5406eb01d6 in op_shared_search () from > /usr/lib64/dirsrv/libslapd.so.0 > #21 0x00007f54073a133e in do_search () > #22 0x00007f540738f567 in connection_threadmain () > #23 0x00007f540509e7bb in _pt_root () from /lib64/libnspr4.so > #24 0x00007f5404a3fdc5 in start_thread () from /lib64/libpthread.so.0 > #25 0x00007f540476d28d in clone () from /lib64/libc.so.6 > Thread 16 (Thread 0x7f53d27ec700 (LWP 128230)): > #0 0x00007f5404a436d5 in pthread_cond_wait@@GLIBC_2.3.2 () from > /lib64/libpthread.so.0 > #1 0x00007f53fef9f2f3 in __db_hybrid_mutex_suspend () from > /lib64/libdb-5.3.so > #2 0x00007f53fef9e640 in __db_tas_mutex_lock () from > /lib64/libdb-5.3.so > #3 0x00007f53ff048d3a in __lock_get_internal () from > /lib64/libdb-5.3.so > #4 0x00007f53ff049820 in __lock_get () from /lib64/libdb-5.3.so > > #5 0x00007f53ff075142 in __db_lget () from /lib64/libdb-5.3.so > > #6 0x00007f53fefbc605 in __bam_search () from /lib64/libdb-5.3.so > > #7 0x00007f53fefa7256 in __bamc_search () from /lib64/libdb-5.3.so > > #8 0x00007f53fefa8d0f in __bamc_get () from /lib64/libdb-5.3.so > > #9 0x00007f53ff061ca6 in __dbc_iget () from /lib64/libdb-5.3.so > > #10 0x00007f53ff070b02 in __dbc_get_pp () from /lib64/libdb-5.3.so > > #11 0x00007f53fa54c680 in idl_new_fetch () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #12 0x00007f53fa55ab26 in index_read_ext_allids () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #13 0x00007f53fa545284 in keys2idl () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #14 0x00007f53fa5459e3 in ava_candidates.isra.0 () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #15 0x00007f53fa545fd2 in filter_candidates_ext () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #16 0x00007f53fa547096 in list_candidates () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #17 0x00007f53fa545f40 in filter_candidates_ext () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #18 0x00007f53fa58180c in subtree_candidates () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #19 0x00007f53fa582e7f in ldbm_back_search () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #20 0x00007f5406eb01d6 in op_shared_search () from > /usr/lib64/dirsrv/libslapd.so.0 > #21 0x00007f54073a133e in do_search () > #22 0x00007f540738f567 in connection_threadmain () > #23 0x00007f540509e7bb in _pt_root () from /lib64/libnspr4.so > #24 0x00007f5404a3fdc5 in start_thread () from /lib64/libpthread.so.0 > #25 0x00007f540476d28d in clone () from /lib64/libc.so.6 > Thread 15 (Thread 0x7f53d1feb700 (LWP 128231)): > #0 0x00007f5404a436d5 in pthread_cond_wait@@GLIBC_2.3.2 () from > /lib64/libpthread.so.0 > #1 0x00007f53fef9f2f3 in __db_hybrid_mutex_suspend () from > /lib64/libdb-5.3.so > #2 0x00007f53fef9e640 in __db_tas_mutex_lock () from > /lib64/libdb-5.3.so > #3 0x00007f53ff048d3a in __lock_get_internal () from > /lib64/libdb-5.3.so > #4 0x00007f53ff049820 in __lock_get () from /lib64/libdb-5.3.so > > #5 0x00007f53ff075142 in __db_lget () from /lib64/libdb-5.3.so > > #6 0x00007f53fefbc605 in __bam_search () from /lib64/libdb-5.3.so > > #7 0x00007f53fefa7256 in __bamc_search () from /lib64/libdb-5.3.so > > #8 0x00007f53fefa8d0f in __bamc_get () from /lib64/libdb-5.3.so > > #9 0x00007f53ff061ca6 in __dbc_iget () from /lib64/libdb-5.3.so > > #10 0x00007f53ff070b02 in __dbc_get_pp () from /lib64/libdb-5.3.so > > #11 0x00007f53fa54c680 in idl_new_fetch () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #12 0x00007f53fa55ab26 in index_read_ext_allids () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #13 0x00007f53fa545284 in keys2idl () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #14 0x00007f53fa5459e3 in ava_candidates.isra.0 () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #15 0x00007f53fa545fd2 in filter_candidates_ext () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #16 0x00007f53fa547096 in list_candidates () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #17 0x00007f53fa545f40 in filter_candidates_ext () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #18 0x00007f53fa58180c in subtree_candidates () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #19 0x00007f53fa582e7f in ldbm_back_search () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #20 0x00007f5406eb01d6 in op_shared_search () from > /usr/lib64/dirsrv/libslapd.so.0 > #21 0x00007f54073a133e in do_search () > #22 0x00007f540738f567 in connection_threadmain () > #23 0x00007f540509e7bb in _pt_root () from /lib64/libnspr4.so > #24 0x00007f5404a3fdc5 in start_thread () from /lib64/libpthread.so.0 > #25 0x00007f540476d28d in clone () from /lib64/libc.so.6 > Thread 14 (Thread 0x7f53d17ea700 (LWP 128232)): > #0 0x00007f5404a436d5 in pthread_cond_wait@@GLIBC_2.3.2 () from > /lib64/libpthread.so.0 > #1 0x00007f53fef9f2f3 in __db_hybrid_mutex_suspend () from > /lib64/libdb-5.3.so > #2 0x00007f53fef9e640 in __db_tas_mutex_lock () from > /lib64/libdb-5.3.so > #3 0x00007f53ff048d3a in __lock_get_internal () from > /lib64/libdb-5.3.so > #4 0x00007f53ff049820 in __lock_get () from /lib64/libdb-5.3.so > > #5 0x00007f53ff075142 in __db_lget () from /lib64/libdb-5.3.so > > #6 0x00007f53fefbc605 in __bam_search () from /lib64/libdb-5.3.so > > #7 0x00007f53fefa7256 in __bamc_search () from /lib64/libdb-5.3.so > > #8 0x00007f53fefa8d0f in __bamc_get () from /lib64/libdb-5.3.so > > #9 0x00007f53ff061ca6 in __dbc_iget () from /lib64/libdb-5.3.so > > #10 0x00007f53ff070b02 in __dbc_get_pp () from /lib64/libdb-5.3.so > > #11 0x00007f53fa54c680 in idl_new_fetch () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #12 0x00007f53fa55ab26 in index_read_ext_allids () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #13 0x00007f53fa545284 in keys2idl () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #14 0x00007f53fa5459e3 in ava_candidates.isra.0 () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #15 0x00007f53fa545fd2 in filter_candidates_ext () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #16 0x00007f53fa547096 in list_candidates () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #17 0x00007f53fa545f40 in filter_candidates_ext () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #18 0x00007f53fa547096 in list_candidates () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #19 0x00007f53fa545f40 in filter_candidates_ext () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #20 0x00007f53fa547096 in list_candidates () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #21 0x00007f53fa545f40 in filter_candidates_ext () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #22 0x00007f53fa58180c in subtree_candidates () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #23 0x00007f53fa582e7f in ldbm_back_search () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #24 0x00007f5406eb01d6 in op_shared_search () from > /usr/lib64/dirsrv/libslapd.so.0 > #25 0x00007f54073a133e in do_search () > #26 0x00007f540738f3c3 in connection_threadmain () > #27 0x00007f540509e7bb in _pt_root () from /lib64/libnspr4.so > #28 0x00007f5404a3fdc5 in start_thread () from /lib64/libpthread.so.0 > #29 0x00007f540476d28d in clone () from /lib64/libc.so.6 > Thread 13 (Thread 0x7f53d0fe9700 (LWP 128233)): > #0 0x00007f5404a436d5 in pthread_cond_wait@@GLIBC_2.3.2 () from > /lib64/libpthread.so.0 > #1 0x00007f53fef9f2f3 in __db_hybrid_mutex_suspend () from > /lib64/libdb-5.3.so > #2 0x00007f53fef9e640 in __db_tas_mutex_lock () from > /lib64/libdb-5.3.so > #3 0x00007f53ff048d3a in __lock_get_internal () from > /lib64/libdb-5.3.so > #4 0x00007f53ff049820 in __lock_get () from /lib64/libdb-5.3.so > > #5 0x00007f53ff075142 in __db_lget () from /lib64/libdb-5.3.so > > #6 0x00007f53fefbc605 in __bam_search () from /lib64/libdb-5.3.so > > #7 0x00007f53fefa7256 in __bamc_search () from /lib64/libdb-5.3.so > > #8 0x00007f53fefa8d0f in __bamc_get () from /lib64/libdb-5.3.so > > #9 0x00007f53ff061ca6 in __dbc_iget () from /lib64/libdb-5.3.so > > #10 0x00007f53ff070b02 in __dbc_get_pp () from /lib64/libdb-5.3.so > > #11 0x00007f53fa54c680 in idl_new_fetch () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #12 0x00007f53fa55ab26 in index_read_ext_allids () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #13 0x00007f53fa545284 in keys2idl () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #14 0x00007f53fa5459e3 in ava_candidates.isra.0 () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #15 0x00007f53fa545fd2 in filter_candidates_ext () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #16 0x00007f53fa547096 in list_candidates () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #17 0x00007f53fa545f40 in filter_candidates_ext () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #18 0x00007f53fa58180c in subtree_candidates () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #19 0x00007f53fa582e7f in ldbm_back_search () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #20 0x00007f5406eb01d6 in op_shared_search () from > /usr/lib64/dirsrv/libslapd.so.0 > #21 0x00007f54073a133e in do_search () > #22 0x00007f540738f567 in connection_threadmain () > #23 0x00007f540509e7bb in _pt_root () from /lib64/libnspr4.so > #24 0x00007f5404a3fdc5 in start_thread () from /lib64/libpthread.so.0 > #25 0x00007f540476d28d in clone () from /lib64/libc.so.6 > Thread 12 (Thread 0x7f53d07e8700 (LWP 128234)): > #0 0x00007f5404a436d5 in pthread_cond_wait@@GLIBC_2.3.2 () from > /lib64/libpthread.so.0 > #1 0x00007f53fef9f2f3 in __db_hybrid_mutex_suspend () from > /lib64/libdb-5.3.so > #2 0x00007f53fef9e640 in __db_tas_mutex_lock () from > /lib64/libdb-5.3.so > #3 0x00007f53ff048d3a in __lock_get_internal () from > /lib64/libdb-5.3.so > #4 0x00007f53ff049820 in __lock_get () from /lib64/libdb-5.3.so > > #5 0x00007f53ff075142 in __db_lget () from /lib64/libdb-5.3.so > > #6 0x00007f53fefbc605 in __bam_search () from /lib64/libdb-5.3.so > > #7 0x00007f53fefa7256 in __bamc_search () from /lib64/libdb-5.3.so > > #8 0x00007f53fefa8d0f in __bamc_get () from /lib64/libdb-5.3.so > > #9 0x00007f53ff061ca6 in __dbc_iget () from /lib64/libdb-5.3.so > > #10 0x00007f53ff070b02 in __dbc_get_pp () from /lib64/libdb-5.3.so > > #11 0x00007f53fa54c680 in idl_new_fetch () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #12 0x00007f53fa55ab26 in index_read_ext_allids () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #13 0x00007f53fa545284 in keys2idl () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #14 0x00007f53fa5459e3 in ava_candidates.isra.0 () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #15 0x00007f53fa545fd2 in filter_candidates_ext () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #16 0x00007f53fa547096 in list_candidates () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #17 0x00007f53fa545f40 in filter_candidates_ext () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #18 0x00007f53fa58180c in subtree_candidates () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #19 0x00007f53fa582e7f in ldbm_back_search () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #20 0x00007f5406eb01d6 in op_shared_search () from > /usr/lib64/dirsrv/libslapd.so.0 > #21 0x00007f54073a133e in do_search () > #22 0x00007f540738f567 in connection_threadmain () > #23 0x00007f540509e7bb in _pt_root () from /lib64/libnspr4.so > #24 0x00007f5404a3fdc5 in start_thread () from /lib64/libpthread.so.0 > #25 0x00007f540476d28d in clone () from /lib64/libc.so.6 > Thread 11 (Thread 0x7f53cffe7700 (LWP 128235)): > #0 0x00007f5404a436d5 in pthread_cond_wait@@GLIBC_2.3.2 () from > /lib64/libpthread.so.0 > #1 0x00007f53fef9f2f3 in __db_hybrid_mutex_suspend () from > /lib64/libdb-5.3.so > #2 0x00007f53fef9e640 in __db_tas_mutex_lock () from > /lib64/libdb-5.3.so > #3 0x00007f53ff048d3a in __lock_get_internal () from > /lib64/libdb-5.3.so > #4 0x00007f53ff049820 in __lock_get () from /lib64/libdb-5.3.so > > #5 0x00007f53ff075142 in __db_lget () from /lib64/libdb-5.3.so > > #6 0x00007f53fefbc605 in __bam_search () from /lib64/libdb-5.3.so > > #7 0x00007f53fefa7256 in __bamc_search () from /lib64/libdb-5.3.so > > #8 0x00007f53fefa8d0f in __bamc_get () from /lib64/libdb-5.3.so > > #9 0x00007f53ff061ca6 in __dbc_iget () from /lib64/libdb-5.3.so > > #10 0x00007f53ff070b02 in __dbc_get_pp () from /lib64/libdb-5.3.so > > #11 0x00007f53fa54c680 in idl_new_fetch () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #12 0x00007f53fa55ab26 in index_read_ext_allids () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #13 0x00007f53fa545284 in keys2idl () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #14 0x00007f53fa5459e3 in ava_candidates.isra.0 () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #15 0x00007f53fa545fd2 in filter_candidates_ext () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #16 0x00007f53fa547096 in list_candidates () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #17 0x00007f53fa545f40 in filter_candidates_ext () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #18 0x00007f53fa58180c in subtree_candidates () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #19 0x00007f53fa582e7f in ldbm_back_search () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #20 0x00007f5406eb01d6 in op_shared_search () from > /usr/lib64/dirsrv/libslapd.so.0 > #21 0x00007f54073a133e in do_search () > #22 0x00007f540738f567 in connection_threadmain () > #23 0x00007f540509e7bb in _pt_root () from /lib64/libnspr4.so > #24 0x00007f5404a3fdc5 in start_thread () from /lib64/libpthread.so.0 > #25 0x00007f540476d28d in clone () from /lib64/libc.so.6 > Thread 10 (Thread 0x7f53cf7e6700 (LWP 128236)): > #0 0x00007f5404a436d5 in pthread_cond_wait@@GLIBC_2.3.2 () from > /lib64/libpthread.so.0 > #1 0x00007f53fef9f2f3 in __db_hybrid_mutex_suspend () from > /lib64/libdb-5.3.so > #2 0x00007f53fef9e640 in __db_tas_mutex_lock () from > /lib64/libdb-5.3.so > #3 0x00007f53ff048d3a in __lock_get_internal () from > /lib64/libdb-5.3.so > #4 0x00007f53ff049820 in __lock_get () from /lib64/libdb-5.3.so > > #5 0x00007f53ff075142 in __db_lget () from /lib64/libdb-5.3.so > > #6 0x00007f53fefbc605 in __bam_search () from /lib64/libdb-5.3.so > > #7 0x00007f53fefa7256 in __bamc_search () from /lib64/libdb-5.3.so > > #8 0x00007f53fefa8d0f in __bamc_get () from /lib64/libdb-5.3.so > > #9 0x00007f53ff061ca6 in __dbc_iget () from /lib64/libdb-5.3.so > > #10 0x00007f53ff070b02 in __dbc_get_pp () from /lib64/libdb-5.3.so > > #11 0x00007f53fa54c680 in idl_new_fetch () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #12 0x00007f53fa55ab26 in index_read_ext_allids () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #13 0x00007f53fa545284 in keys2idl () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #14 0x00007f53fa5459e3 in ava_candidates.isra.0 () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #15 0x00007f53fa545fd2 in filter_candidates_ext () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #16 0x00007f53fa547096 in list_candidates () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #17 0x00007f53fa545f40 in filter_candidates_ext () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #18 0x00007f53fa58180c in subtree_candidates () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #19 0x00007f53fa582e7f in ldbm_back_search () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #20 0x00007f5406eb01d6 in op_shared_search () from > /usr/lib64/dirsrv/libslapd.so.0 > #21 0x00007f54073a133e in do_search () > #22 0x00007f540738f567 in connection_threadmain () > #23 0x00007f540509e7bb in _pt_root () from /lib64/libnspr4.so > #24 0x00007f5404a3fdc5 in start_thread () from /lib64/libpthread.so.0 > #25 0x00007f540476d28d in clone () from /lib64/libc.so.6 > Thread 9 (Thread 0x7f53cefe5700 (LWP 128237)): > #0 0x00007f5404a436d5 in pthread_cond_wait@@GLIBC_2.3.2 () from > /lib64/libpthread.so.0 > #1 0x00007f53fef9f2f3 in __db_hybrid_mutex_suspend () from > /lib64/libdb-5.3.so > #2 0x00007f53fef9e640 in __db_tas_mutex_lock () from > /lib64/libdb-5.3.so > #3 0x00007f53ff048d3a in __lock_get_internal () from > /lib64/libdb-5.3.so > #4 0x00007f53ff049820 in __lock_get () from /lib64/libdb-5.3.so > > #5 0x00007f53ff075142 in __db_lget () from /lib64/libdb-5.3.so > > #6 0x00007f53fefbc605 in __bam_search () from /lib64/libdb-5.3.so > > #7 0x00007f53fefa7256 in __bamc_search () from /lib64/libdb-5.3.so > > #8 0x00007f53fefa8d0f in __bamc_get () from /lib64/libdb-5.3.so > > #9 0x00007f53ff061ca6 in __dbc_iget () from /lib64/libdb-5.3.so > > #10 0x00007f53ff070b02 in __dbc_get_pp () from /lib64/libdb-5.3.so > > #11 0x00007f53fa54c680 in idl_new_fetch () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #12 0x00007f53fa55ab26 in index_read_ext_allids () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #13 0x00007f53fa545284 in keys2idl () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #14 0x00007f53fa5459e3 in ava_candidates.isra.0 () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #15 0x00007f53fa545fd2 in filter_candidates_ext () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #16 0x00007f53fa547096 in list_candidates () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #17 0x00007f53fa545f40 in filter_candidates_ext () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #18 0x00007f53fa58180c in subtree_candidates () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #19 0x00007f53fa582e7f in ldbm_back_search () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #20 0x00007f5406eb01d6 in op_shared_search () from > /usr/lib64/dirsrv/libslapd.so.0 > #21 0x00007f54073a133e in do_search () > #22 0x00007f540738f567 in connection_threadmain () > #23 0x00007f540509e7bb in _pt_root () from /lib64/libnspr4.so > #24 0x00007f5404a3fdc5 in start_thread () from /lib64/libpthread.so.0 > #25 0x00007f540476d28d in clone () from /lib64/libc.so.6 > Thread 8 (Thread 0x7f53ce7e4700 (LWP 128238)): > #0 0x00007f5404a436d5 in pthread_cond_wait@@GLIBC_2.3.2 () from > /lib64/libpthread.so.0 > #1 0x00007f53fef9f2f3 in __db_hybrid_mutex_suspend () from > /lib64/libdb-5.3.so > #2 0x00007f53fef9e640 in __db_tas_mutex_lock () from > /lib64/libdb-5.3.so > #3 0x00007f53ff048d3a in __lock_get_internal () from > /lib64/libdb-5.3.so > #4 0x00007f53ff049820 in __lock_get () from /lib64/libdb-5.3.so > > #5 0x00007f53ff075142 in __db_lget () from /lib64/libdb-5.3.so > > #6 0x00007f53fefbc605 in __bam_search () from /lib64/libdb-5.3.so > > #7 0x00007f53fefa7256 in __bamc_search () from /lib64/libdb-5.3.so > > #8 0x00007f53fefa8d0f in __bamc_get () from /lib64/libdb-5.3.so > > #9 0x00007f53ff061ca6 in __dbc_iget () from /lib64/libdb-5.3.so > > #10 0x00007f53ff070b02 in __dbc_get_pp () from /lib64/libdb-5.3.so > > #11 0x00007f53fa54c680 in idl_new_fetch () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #12 0x00007f53fa55ab26 in index_read_ext_allids () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #13 0x00007f53fa545284 in keys2idl () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #14 0x00007f53fa5459e3 in ava_candidates.isra.0 () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #15 0x00007f53fa545fd2 in filter_candidates_ext () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #16 0x00007f53fa547096 in list_candidates () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #17 0x00007f53fa545f40 in filter_candidates_ext () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #18 0x00007f53fa58180c in subtree_candidates () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #19 0x00007f53fa582e7f in ldbm_back_search () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #20 0x00007f5406eb01d6 in op_shared_search () from > /usr/lib64/dirsrv/libslapd.so.0 > #21 0x00007f54073a133e in do_search () > #22 0x00007f540738f567 in connection_threadmain () > #23 0x00007f540509e7bb in _pt_root () from /lib64/libnspr4.so > #24 0x00007f5404a3fdc5 in start_thread () from /lib64/libpthread.so.0 > #25 0x00007f540476d28d in clone () from /lib64/libc.so.6 > Thread 7 (Thread 0x7f53cdfe3700 (LWP 128239)): > #0 0x00007f5404a436d5 in pthread_cond_wait@@GLIBC_2.3.2 () from > /lib64/libpthread.so.0 > #1 0x00007f53fef9f2f3 in __db_hybrid_mutex_suspend () from > /lib64/libdb-5.3.so > #2 0x00007f53fef9e640 in __db_tas_mutex_lock () from > /lib64/libdb-5.3.so > #3 0x00007f53ff048d3a in __lock_get_internal () from > /lib64/libdb-5.3.so > #4 0x00007f53ff049820 in __lock_get () from /lib64/libdb-5.3.so > > #5 0x00007f53ff075142 in __db_lget () from /lib64/libdb-5.3.so > > #6 0x00007f53fefbc605 in __bam_search () from /lib64/libdb-5.3.so > > #7 0x00007f53fefa7256 in __bamc_search () from /lib64/libdb-5.3.so > > #8 0x00007f53fefa8d0f in __bamc_get () from /lib64/libdb-5.3.so > > #9 0x00007f53ff061ca6 in __dbc_iget () from /lib64/libdb-5.3.so > > #10 0x00007f53ff070b02 in __dbc_get_pp () from /lib64/libdb-5.3.so > > #11 0x00007f53fa54c680 in idl_new_fetch () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #12 0x00007f53fa55ab26 in index_read_ext_allids () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #13 0x00007f53fa545284 in keys2idl () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #14 0x00007f53fa5459e3 in ava_candidates.isra.0 () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #15 0x00007f53fa545fd2 in filter_candidates_ext () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #16 0x00007f53fa547096 in list_candidates () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #17 0x00007f53fa545f40 in filter_candidates_ext () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #18 0x00007f53fa58180c in subtree_candidates () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #19 0x00007f53fa582e7f in ldbm_back_search () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #20 0x00007f5406eb01d6 in op_shared_search () from > /usr/lib64/dirsrv/libslapd.so.0 > #21 0x00007f54073a133e in do_search () > #22 0x00007f540738f567 in connection_threadmain () > #23 0x00007f540509e7bb in _pt_root () from /lib64/libnspr4.so > #24 0x00007f5404a3fdc5 in start_thread () from /lib64/libpthread.so.0 > #25 0x00007f540476d28d in clone () from /lib64/libc.so.6 > Thread 6 (Thread 0x7f53cd7e2700 (LWP 128240)): > #0 0x00007f5404a436d5 in pthread_cond_wait@@GLIBC_2.3.2 () from > /lib64/libpthread.so.0 > #1 0x00007f53fef9f2f3 in __db_hybrid_mutex_suspend () from > /lib64/libdb-5.3.so > #2 0x00007f53fef9eb37 in __db_tas_mutex_readlock () from > /lib64/libdb-5.3.so > #3 0x00007f53ff0b8d35 in __memp_fget () from /lib64/libdb-5.3.so > > #4 0x00007f53ff062333 in __dbc_iget () from /lib64/libdb-5.3.so > > #5 0x00007f53ff070b02 in __dbc_get_pp () from /lib64/libdb-5.3.so > > #6 0x00007f53fa54c822 in idl_new_fetch () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #7 0x00007f53fa55ab26 in index_read_ext_allids () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #8 0x00007f53fa545284 in keys2idl () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #9 0x00007f53fa5459e3 in ava_candidates.isra.0 () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #10 0x00007f53fa545fd2 in filter_candidates_ext () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #11 0x00007f53fa547096 in list_candidates () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #12 0x00007f53fa545f40 in filter_candidates_ext () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #13 0x00007f53fa547096 in list_candidates () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #14 0x00007f53fa545f40 in filter_candidates_ext () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #15 0x00007f53fa547096 in list_candidates () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #16 0x00007f53fa545f40 in filter_candidates_ext () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #17 0x00007f53fa58180c in subtree_candidates () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #18 0x00007f53fa582e7f in ldbm_back_search () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #19 0x00007f5406eb01d6 in op_shared_search () from > /usr/lib64/dirsrv/libslapd.so.0 > #20 0x00007f54073a133e in do_search () > #21 0x00007f540738f3c3 in connection_threadmain () > #22 0x00007f540509e7bb in _pt_root () from /lib64/libnspr4.so > #23 0x00007f5404a3fdc5 in start_thread () from /lib64/libpthread.so.0 > #24 0x00007f540476d28d in clone () from /lib64/libc.so.6 > Thread 5 (Thread 0x7f53ccfe1700 (LWP 128241)): > #0 0x00007f5404a436d5 in pthread_cond_wait@@GLIBC_2.3.2 () from > /lib64/libpthread.so.0 > #1 0x00007f53fef9f2f3 in __db_hybrid_mutex_suspend () from > /lib64/libdb-5.3.so > #2 0x00007f53fef9e640 in __db_tas_mutex_lock () from > /lib64/libdb-5.3.so > #3 0x00007f53ff048d3a in __lock_get_internal () from > /lib64/libdb-5.3.so > #4 0x00007f53ff049820 in __lock_get () from /lib64/libdb-5.3.so > > #5 0x00007f53ff075142 in __db_lget () from /lib64/libdb-5.3.so > > #6 0x00007f53fefbc605 in __bam_search () from /lib64/libdb-5.3.so > > #7 0x00007f53fefa7256 in __bamc_search () from /lib64/libdb-5.3.so > > #8 0x00007f53fefa8d0f in __bamc_get () from /lib64/libdb-5.3.so > > #9 0x00007f53ff061ca6 in __dbc_iget () from /lib64/libdb-5.3.so > > #10 0x00007f53ff070b02 in __dbc_get_pp () from /lib64/libdb-5.3.so > > #11 0x00007f53fa54c680 in idl_new_fetch () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #12 0x00007f53fa55ab26 in index_read_ext_allids () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #13 0x00007f53fa545284 in keys2idl () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #14 0x00007f53fa5459e3 in ava_candidates.isra.0 () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #15 0x00007f53fa545fd2 in filter_candidates_ext () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #16 0x00007f53fa547096 in list_candidates () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #17 0x00007f53fa545f40 in filter_candidates_ext () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #18 0x00007f53fa547096 in list_candidates () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #19 0x00007f53fa545f40 in filter_candidates_ext () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #20 0x00007f53fa547096 in list_candidates () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #21 0x00007f53fa545f40 in filter_candidates_ext () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #22 0x00007f53fa58180c in subtree_candidates () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #23 0x00007f53fa582e7f in ldbm_back_search () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #24 0x00007f5406eb01d6 in op_shared_search () from > /usr/lib64/dirsrv/libslapd.so.0 > #25 0x00007f54073a133e in do_search () > #26 0x00007f540738f3c3 in connection_threadmain () > #27 0x00007f540509e7bb in _pt_root () from /lib64/libnspr4.so > #28 0x00007f5404a3fdc5 in start_thread () from /lib64/libpthread.so.0 > #29 0x00007f540476d28d in clone () from /lib64/libc.so.6 > Thread 4 (Thread 0x7f53cc7e0700 (LWP 128242)): > #0 0x00007f54047649b3 in select () from /lib64/libc.so.6 > #1 0x00007f5406eef0e9 in DS_Sleep () from > /usr/lib64/dirsrv/libslapd.so.0 > #2 0x00007f5407390325 in time_thread () > #3 0x00007f540509e7bb in _pt_root () from /lib64/libnspr4.so > #4 0x00007f5404a3fdc5 in start_thread () from /lib64/libpthread.so.0 > #5 0x00007f540476d28d in clone () from /lib64/libc.so.6 > Thread 3 (Thread 0x7f53cb7a0700 (LWP 128259)): > #0 0x00007f5404a436d5 in pthread_cond_wait@@GLIBC_2.3.2 () from > /lib64/libpthread.so.0 > #1 0x00007f5405099050 in PR_WaitCondVar () from /lib64/libnspr4.so > #2 0x00007f540739bf25 in ps_send_results () > #3 0x00007f540509e7bb in _pt_root () from /lib64/libnspr4.so > #4 0x00007f5404a3fdc5 in start_thread () from /lib64/libpthread.so.0 > #5 0x00007f540476d28d in clone () from /lib64/libc.so.6 > Thread 2 (Thread 0x7f53cab9d700 (LWP 9020)): > #0 0x00007f5404a436d5 in pthread_cond_wait@@GLIBC_2.3.2 () from > /lib64/libpthread.so.0 > #1 0x00007f5405099050 in PR_WaitCondVar () from /lib64/libnspr4.so > #2 0x00007f540739bf25 in ps_send_results () > #3 0x00007f540509e7bb in _pt_root () from /lib64/libnspr4.so > #4 0x00007f5404a3fdc5 in start_thread () from /lib64/libpthread.so.0 > #5 0x00007f540476d28d in clone () from /lib64/libc.so.6 > Thread 1 (Thread 0x7f5407357840 (LWP 128124)): > #0 0x00007f5404762c3d in poll () from /lib64/libc.so.6 > #1 0x00007f540509a967 in _pr_poll_with_poll () from /lib64/libnspr4.so > #2 0x00007f5407393ff1 in slapd_daemon () > #3 0x00007f5407385c6c in main () > > > Thanks > > On Tue, Sep 6, 2016 at 5:51 PM, Rakesh Rajasekharan > > wrote: > > Hi, > > With script provided by Rich, I looked at my db cache usage and > it does not look to be high enough > > # under each db are the list of selected indexes specified with > INDEXLIST > dbname count free free% size > changelog:ent 5 2076333 99.0 4163.8 > changelog:dn 15976 9303557 88.7 74.0 > userroot:ent 4763 175321074 83.6 7221.1 > userroot:dn 4763 9995056 95.3 103.0 > ipaca:ent 425 6337866 60.4 9759.8 > ipaca:dn 425 10451081 99.7 81.6 > > Right now I set it to 200MB. So, I am planning to bring it down > to 100MB > > Also, /|nsslapd-db-checkpoint-interval |/will change it to 80 or > 90 from default of 60 > > I will check this out and the impact it has on the hang issues > and get back > > Thanks > Rakesh > > On Tue, Sep 6, 2016 at 4:13 PM, Ludwig Krispenz > > wrote: > > Hi, > > I think you are running into this bug: > https://bugzilla.redhat.com/show_bug.cgi?id=1349779 > > it was the outcome of an analysis started in this one: > https://bugzilla.redhat.com/show_bug.cgi?id=1347800 > > > the problem is that it looks like the checkpoint thread is > holding a mutex and otehr threads are blocked on it. It > looks like an issue in BerkeleyDB although there is no > explanation from the BDB code why this could happen, we also > had a support case with Oracle to investigate this, with no > solution. Oracle at the end was providing a diagnostic > patch, which we did not test since the customer with the > original problem had worked around this by changing their > configuration. > I don't know the status of this patch and if we can make it > available, probably depends on your support status, you need > to contact support for this. > > About the workarounds: > you can try to change timing for the database update and > checkpointing eg by reducing the dbcache size, changing the > checkpoint interval. In the customer case they identified > one substring index, which was frequently updated but not > really required - and removing this index helped. > > Regards, > Ludwig > > On 09/06/2016 11:31 AM, thierry bordaz wrote: >> Hi, >> >> The following thread is hanging most of the others threads >> >> Thread 20 (Thread 0x7f68007f8700 (LWP 4873)): >> #0 0x00007f683d0506d5 in >> pthread_cond_wait@@GLIBC_2.3.2 >> () from >> /lib64/libpthread.so.0 >> #1 0x00007f68375ac2f3 in __db_hybrid_mutex_suspend () >> from /lib64/libdb-5.3.so >> #2 0x00007f68375ab640 in __db_tas_mutex_lock () from >> /lib64/libdb-5.3.so >> #3 0x00007f68376c5e1b in __memp_fget () from >> /lib64/libdb-5.3.so >> #4 0x00007f68375c9211 in __bam_search () from >> /lib64/libdb-5.3.so >> #5 0x00007f68375b4256 in __bamc_search () from >> /lib64/libdb-5.3.so >> #6 0x00007f68375b86c4 in __bamc_put () from >> /lib64/libdb-5.3.so >> #7 0x00007f683766f6d5 in __dbc_iput () from >> /lib64/libdb-5.3.so >> #8 0x00007f683766aa0e in __db_put () from >> /lib64/libdb-5.3.so >> #9 0x00007f683767ffa4 in __db_put_pp () from >> /lib64/libdb-5.3.so >> #10 0x00007f6832b5a4cf in idl_new_insert_key () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #11 0x00007f6832b695b5 in addordel_values_sv.isra.3 () >> from /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #12 0x00007f6832b6a25a in index_addordel_values_ext_sv >> () from /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #13 0x00007f6832b6a694 in index_addordel_values_sv () >> from /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #14 0x00007f6832b6a75a in index_addordel_entry () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #15 0x00007f6832b6d5fa in ldbm_back_add () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #16 0x00007f683f46fcca in op_shared_add () from >> /usr/lib64/dirsrv/libslapd.so.0 >> #17 0x00007f683f471020 in do_add () from >> /usr/lib64/dirsrv/libslapd.so.0 >> #18 0x00007f683f99c303 in connection_threadmain () >> #19 0x00007f683d6ab7bb in _pt_root () from >> /lib64/libnspr4.so >> #20 0x00007f683d04cdc5 in start_thread () from >> /lib64/libpthread.so.0 >> #21 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 >> >> This thread is likely not hang on 'db_tas_mutex_lock' as >> the db_lock does not show it waiting for something. >> Unfortunately it is not shown in the output of top -H, did >> you get the complete output ? >> Ludwig mentioned a known issue that DS is not scaling well >> on NUMA arch, do you know if the machine is NUMA ? >> A possibility is that this thread get severely slowed down >> by a NUMA remote lock and showing DS has hung. >> This is why it would be interesting to know if it consums CPU. >> >> thanks >> theirry >> >> On 09/06/2016 10:59 AM, Rakesh Rajasekharan wrote: >>> Hi Thierry/Ludwig, >>> >>> I again hit the hang issue. This is the pstack of the >>> ns-slapd >>> >>> So was able to this time get the pstack and db locks >>> >>> db locks details >>> Default locking region information: >>> 451 Last allocated locker ID >>> 0x7fffffff Current maximum unused locker ID >>> 9 Number of lock modes >>> 800 Initial number of locks allocated >>> 0 Initial number of lockers allocated >>> 800 Initial number of lock objects allocated >>> 10000 Maximum number of locks possible >>> 10000 Maximum number of lockers possible >>> 10000 Maximum number of lock objects possible >>> 800 Current number of locks allocated >>> 366 Current number of lockers allocated >>> 800 Current number of lock objects allocated >>> 160 Number of lock object partitions >>> 8191 Size of object hash table >>> 126 Number of current locks >>> 1005 Maximum number of locks at any one time >>> 33 Maximum number of locks in any one bucket >>> 3987 Maximum number of locks stolen by for an empty >>> partition >>> 75 Maximum number of locks stolen for any one >>> partition >>> 309 Number of current lockers >>> 313 Maximum number of lockers at any one time >>> 121 Number of current lock objects >>> 676 Maximum number of lock objects at any one time >>> 3 Maximum number of lock objects in any one bucket >>> 69 Maximum number of objects stolen by for an >>> empty partition >>> 5 Maximum number of objects stolen for any one >>> partition >>> 22M Total number of locks requested (22009642) >>> 21M Total number of locks released (21992878) >>> 0 Total number of locks upgraded >>> 95 Total number of locks downgraded >>> 3348 Lock requests not available due to conflicts, >>> for which we waited >>> 12646 Lock requests not available due to conflicts, >>> for which we did not wait >>> 0 Number of deadlocks >>> 0 Lock timeout value >>> 0 Number of locks that have timed out >>> 0 Transaction timeout value >>> 0 Number of transactions that have timed out >>> 2MB 560KB Region size >>> 71719 The number of partition locks that required >>> waiting (0%) >>> 23105 The maximum number of times any partition lock >>> was waited for (0%) >>> 0 The number of object queue operations that >>> required waiting (0%) >>> 163 The number of locker allocations that required >>> waiting (0%) >>> 3172 The number of region locks that required >>> waiting (0%) >>> 5 Maximum hash bucket length >>> =-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-= >>> Lock REGINFO information: >>> Environment Region type >>> 1 Region ID >>> /var/lib/dirsrv/slapd-SPRINKLR-COM/db/__db.001 Region name >>> 0x7f67ef869000 Region address >>> 0x7f67ef8690a0 Region allocation head >>> 0x7f67ef8712b0 Region primary address >>> 0 Region maximum allocation >>> 0 Region allocated >>> Region allocations: 1581345 allocations, 0 failures, >>> 1580794 frees, 6 longest >>> Allocations by power-of-two sizes: >>> 1KB 1581329 >>> 2KB 3 >>> 4KB 4 >>> 8KB 3 >>> 16KB 3 >>> 32KB 1 >>> 64KB 0 >>> 128KB 0 >>> 256KB 2 >>> 512KB 0 >>> 1024KB 1 >>> REGION_SHARED Region flags >>> =-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-= >>> Lock region parameters: >>> 2 Lock region region mutex [3172/14M 0% !Own] >>> >>> 16381 locker table size >>> 8191 object table size >>> 34128 obj_off >>> 900216 locker_off >>> 0 need_dd >>> =-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-= >>> Lock conflict matrix: >>> =-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-= >>> Locks grouped by lockers: >>> Locker Mode Count Status ----------------- Object >>> --------------- >>> 2 dd=308 locks held 1 write locks 0 pid/thread >>> 4845/140085720139840 flags 10 priority 100 >>> 2 READ 1 HELD userRoot/id2entry.db >>> handle 0 >>> 3 dd=307 locks held 0 write locks 0 pid/thread >>> 4845/140084829472512 flags 0 priority 100 >>> 4 dd=306 locks held 1 write locks 0 pid/thread >>> 4845/140085720139840 flags 10 priority 100 >>> 4 READ 1 HELD ipaca/id2entry.db >>> handle 0 >>> 5 dd=305 locks held 0 write locks 0 pid/thread >>> 4845/140084837865216 flags 0 priority 100 >>> 6 dd=304 locks held 1 write locks 0 pid/thread >>> 4845/140085720139840 flags 10 priority 100 >>> 6 READ 1 HELD ipaca/entryrdn.db >>> handle 0 >>> 7 dd=303 locks held 0 write locks 0 pid/thread >>> 4845/140084804294400 flags 0 priority 100 >>> 8 dd=302 locks held 0 write locks 0 pid/thread >>> 4845/140084720432896 flags 0 priority 100 >>> 9 dd=301 locks held 0 write locks 0 pid/thread >>> 4845/140084837865216 flags 0 priority 100 >>> a dd=300 locks held 1 write locks 0 pid/thread >>> 4845/140085720139840 flags 10 priority 100 >>> a READ 1 HELD >>> ipaca/vlv#allcertspkitomcatindex.db handle 0 >>> e dd=299 locks held 1 write locks 0 pid/thread >>> 4845/140085720139840 flags 10 priority 100 >>> e READ 1 HELD >>> ipaca/vlv#allnonrevokedcertspkitomcatindex.db handle 0 >>> 10 dd=298 locks held 1 write locks 0 pid/thread >>> 4845/140085720139840 flags 10 priority 100 >>> 10 READ 1 HELD >>> ipaca/vlv#allrevokedcertspkitomcatindex.db handle 0 >>> 11 dd=297 locks held 1 write locks 0 pid/thread >>> 4845/140085720139840 flags 10 priority 100 >>> 11 READ 1 HELD >>> ipaca/vlv#allrevokedcertsnotafterpkitomcatindex.db >>> handle 0 >>> 14 dd=296 locks held 1 write locks 0 pid/thread >>> 4845/140085720139840 flags 10 priority 100 >>> 14 READ 1 HELD >>> ipaca/vlv#allrevokedorrevokedexpiredcertspkitomcatindex.db handle >>> 0 >>> 15 dd=295 locks held 1 write locks 0 pid/thread >>> 4845/140085720139840 flags 10 priority 100 >>> 15 READ 1 HELD >>> ipaca/vlv#allvalidcertspkitomcatindex.db handle 0 >>> 16 dd=294 locks held 1 write locks 0 pid/thread >>> 4845/140085720139840 flags 10 priority 100 >>> 16 READ 1 HELD >>> ipaca/vlv#allvalidcertsnotafterpkitomcatindex.db >>> handle 0 >>> 17 dd=293 locks held 1 write locks 0 pid/thread >>> 4845/140085720139840 flags 10 priority 100 >>> 17 READ 1 HELD >>> ipaca/vlv#allvalidorrevokedcertspkitomcatindex.db >>> handle 0 >>> 18 dd=292 locks held 1 write locks 0 pid/thread >>> 4845/140085720139840 flags 10 priority 100 >>> 18 READ 1 HELD >>> ipaca/vlv#caallpkitomcatindex.db handle 0 >>> 1d dd=291 locks held 1 write locks 0 pid/thread >>> 4845/140085720139840 flags 10 priority 100 >>> 1d READ 1 HELD >>> ipaca/vlv#cacompletepkitomcatindex.db handle 0 >>> 1e dd=290 locks held 1 write locks 0 > > ... > > > > From kashmancy at gmail.com Wed Sep 7 20:27:29 2016 From: kashmancy at gmail.com (Harry Kashouli) Date: Wed, 7 Sep 2016 13:27:29 -0700 Subject: [Freeipa-users] FreeIPA and Tomcat service cannot work at the same time In-Reply-To: References: Message-ID: I'm an idiot... I figured it out. I forgot to change the shutdown port for Apache Tomcat, so pki-tomcat was in conflict. Seems to be running fine now! :D -Harry On 7 September 2016 at 13:12, Harry Kashouli wrote: > Hi all, > > System details: > Fedora 24 > FreeIPA 4.3.2, and working fine > > Desired outcome: > To have pwm running on the same server, for password self-service - > https://github.com/pwm-project/pwm > > My FreeIPA server is running fine, but when I attempt to start Tomcat for > pwm, that service will not work. Systemctl output below, let me know if you > need anything extra, and where to grab it from. > > If I enable Tomcat at boot, then both that and the ipa/kerberos services > fail to start. It seems I cannot have both Apache Tomcat and FreeIPA > running at the same time. I even tried changing Tomcat's ports by > increasing the port number, by 1. > > [root@~]# systemctl status tomcat > ? tomcat.service - Apache Tomcat Web Application Container > Loaded: loaded (/usr/lib/systemd/system/tomcat.service; disabled; > vendor preset: disabled) > Active: active (running) since Wed 2016-09-07 13:00:26 PDT; 21s ago > Main PID: 13925 (java) > Tasks: 18 (limit: 512) > CGroup: /system.slice/tomcat.service > ??13925 /usr/lib/jvm/jre/bin/java -Djavax.sql.DataSource. > Factory=org.apache.commons.dbcp.BasicDataSourceFactory -classpath > /usr/share/tomcat/bin/bootstrap.jar:/usr/share/ > tomcat/bin/tomcat-juli.jar:/usr/lib/java/commons-daemon.jar > -Dcatalina.base=/usr/share/t > > Sep 07 13:00:43 server[13925]: 2016-09-07T13:00:43Z, DEBUG, > pwm.PwmEnvironment, released file lock on file /var/lib/tomcat/webapps/pwm/ > WEB-INF/applicationPath.lock > Sep 07 13:00:43 server[13925]: 2016-09-07T13:00:43Z, INFO , > pwm.PwmApplication, PWM v1.8.0-SNAPSHOT b18063275 > rb329bec559be6e47c64239474c637b8bd9ed2d93 closed for bidness, cya! > Sep 07 13:00:43 server[13925]: 07-Sep-2016 13:00:43.443 WARNING > [localhost-startStop-1] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads > The web application [pwm] appears to have started a thread named > [PwmSettingXml s > Sep 07 13:00:43 server[13925]: java.lang.Thread.sleep(Native Method) > Sep 07 13:00:43 server[13925]: password.pwm.util.Helper. > pause(Helper.java:112) > Sep 07 13:00:43 server[13925]: password.pwm.config.PwmSettingXml$1.run( > PwmSettingXml.java:73) > Sep 07 13:00:43 server[13925]: 07-Sep-2016 13:00:43.508 INFO [main] > org.apache.coyote.AbstractProtocol.stop Stopping ProtocolHandler > ["http-nio-8081"] > Sep 07 13:00:43 server[13925]: 07-Sep-2016 13:00:43.510 INFO [main] > org.apache.coyote.AbstractProtocol.stop Stopping ProtocolHandler > ["ajp-nio-8010"] > Sep 07 13:00:43 server[13925]: 07-Sep-2016 13:00:43.548 INFO [main] > org.apache.coyote.AbstractProtocol.destroy Destroying ProtocolHandler > ["http-nio-8081"] > Sep 07 13:00:43 server[13925]: 07-Sep-2016 13:00:43.550 INFO [main] > org.apache.coyote.AbstractProtocol.destroy Destroying ProtocolHandler > ["ajp-nio-8010"] > [root@~]# systemctl status tomcat > ? tomcat.service - Apache Tomcat Web Application Container > Loaded: loaded (/usr/lib/systemd/system/tomcat.service; disabled; > vendor preset: disabled) > Active: inactive (dead) > > Sep 07 13:00:43 server[13925]: 2016-09-07T13:00:43Z, DEBUG, > pwm.PwmEnvironment, released file lock on file /var/lib/tomcat/webapps/pwm/ > WEB-INF/applicationPath.lock > Sep 07 13:00:43 server[13925]: 2016-09-07T13:00:43Z, INFO , > pwm.PwmApplication, PWM v1.8.0-SNAPSHOT b18063275 > rb329bec559be6e47c64239474c637b8bd9ed2d93 closed for bidness, cya! > Sep 07 13:00:43 server[13925]: 07-Sep-2016 13:00:43.443 WARNING > [localhost-startStop-1] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads > The web application [pwm] appears to have started a thread named > [PwmSettingXml s > Sep 07 13:00:43 server[13925]: java.lang.Thread.sleep(Native Method) > Sep 07 13:00:43 server[13925]: password.pwm.util.Helper. > pause(Helper.java:112) > Sep 07 13:00:43 server[13925]: password.pwm.config.PwmSettingXml$1.run( > PwmSettingXml.java:73) > Sep 07 13:00:43 server[13925]: 07-Sep-2016 13:00:43.508 INFO [main] > org.apache.coyote.AbstractProtocol.stop Stopping ProtocolHandler > ["http-nio-8081"] > Sep 07 13:00:43 server[13925]: 07-Sep-2016 13:00:43.510 INFO [main] > org.apache.coyote.AbstractProtocol.stop Stopping ProtocolHandler > ["ajp-nio-8010"] > Sep 07 13:00:43 server[13925]: 07-Sep-2016 13:00:43.548 INFO [main] > org.apache.coyote.AbstractProtocol.destroy Destroying ProtocolHandler > ["http-nio-8081"] > Sep 07 13:00:43 server[13925]: 07-Sep-2016 13:00:43.550 INFO [main] > org.apache.coyote.AbstractProtocol.destroy Destroying ProtocolHandler > ["ajp-nio-8010"] > > Cheers, > -Harry > -------------- next part -------------- An HTML attachment was scrubbed... URL: From mrorourke at earthlink.net Wed Sep 7 20:45:02 2016 From: mrorourke at earthlink.net (Michael ORourke) Date: Wed, 7 Sep 2016 16:45:02 -0400 (GMT-04:00) Subject: [Freeipa-users] AD integration and transitive trusts Message-ID: <4839910.1473281103714.JavaMail.wam@mswamui-thinleaf.atl.sa.earthlink.net> At my company, we are trying to setup a pilot with FreeIPA and we having some issues. We would like to leverage our corporate AD infrastructure which mainly lives in "somedom2.com", and is a member of "rootdom1.com" forest. Note the different DNS naming between the root domain and the tree. Our FreeIPA domain is lnx.somedom2.com and is joined to rootdom1.com. If we create users in rootdom1.com, we can use those account on servers joined to lnx.somedom2.com, but user accounts under somedom2.com will not work. Could this be a transitive trust issue? Is there something unique we need to setup on the linux servers under lnx.somedom2.com (sssd.conf or krb5.conf) to allow authentication from somedom2.com? rootdom1.com (forest root domain) somedom2.com (main domain tree, users and groups accounts which need access to lnx.somedom2.com) lnx.somedom2.com (freeIPA domain, joined to forest rootdom1.com) -Mike From matt.wells at mosaic451.com Wed Sep 7 20:58:25 2016 From: matt.wells at mosaic451.com (Matt Wells) Date: Wed, 7 Sep 2016 13:58:25 -0700 Subject: [Freeipa-users] Two Factor auth and Windows desktop Message-ID: Hi all! I had a question about something that I'm sure has been covered. I promise that I'm trying to find those articles but thus far I've found some pieces but nothing 100%; however I'm still looking. I have two networks - ad.example.com ( active directory ) - linux.example.com ( IPA ) All of my linux systems Auth against IPA and it's perfect. AD and IPA replicate users ( this currently is how we're doing it and are looking at external users to see if that works better for us ). I'm trying to find how to force my IPA two factor to the desktops that auth to AD. As a whole we're trying to enforce this everywhere and I don't want to go to Duo when we have the capability in house. My IPA is at 4.2 AD is on MS 2012 Any direction would be great and as I mentioned I'm certainly going over the archives to research. -------------- next part -------------- An HTML attachment was scrubbed... URL: From abokovoy at redhat.com Wed Sep 7 21:03:39 2016 From: abokovoy at redhat.com (Alexander Bokovoy) Date: Thu, 8 Sep 2016 00:03:39 +0300 Subject: [Freeipa-users] AD integration and transitive trusts In-Reply-To: <4839910.1473281103714.JavaMail.wam@mswamui-thinleaf.atl.sa.earthlink.net> References: <4839910.1473281103714.JavaMail.wam@mswamui-thinleaf.atl.sa.earthlink.net> Message-ID: <20160907210338.oi7nvgget743ugis@redhat.com> On Wed, 07 Sep 2016, Michael ORourke wrote: >At my company, we are trying to setup a pilot with FreeIPA and we >having some issues. We would like to leverage our corporate AD >infrastructure which mainly lives in "somedom2.com", and is a member of >"rootdom1.com" forest. Note the different DNS naming between the root >domain and the tree. Our FreeIPA domain is lnx.somedom2.com and is >joined to rootdom1.com. If we create users in rootdom1.com, we can use >those account on servers joined to lnx.somedom2.com, but user accounts >under somedom2.com will not work. Could this be a transitive trust >issue? Is there something unique we need to setup on the linux servers >under lnx.somedom2.com (sssd.conf or krb5.conf) to allow authentication >from somedom2.com? > >rootdom1.com (forest root domain) > >somedom2.com (main domain tree, users and groups accounts which need access to lnx.somedom2.com) > >lnx.somedom2.com (freeIPA domain, joined to forest rootdom1.com) This configuration should work. There were some issues in SSSD generating incorrectly CA paths for krb5.conf for some of similar setups in RHEL 7.2, this should be addressed in the latest RHEL 7.2.z release and in Fedora 24. However, in order to debug such cases, you need look into https://fedorahosted.org/sssd/wiki/Troubleshooting and provide logs that demonstrate problems you see. Don't forget also to tell freeipa/sssd versions down to 'rpm -q ...' output. -- / Alexander Bokovoy From deepak_dimri at hotmail.com Thu Sep 8 04:49:53 2016 From: deepak_dimri at hotmail.com (Deepak Dimri) Date: Thu, 8 Sep 2016 00:49:53 -0400 Subject: [Freeipa-users] General query regarding nameserver enrtry In-Reply-To: <4d9223c9-216b-d79d-5c93-ec81466c75c6@redhat.com> References: , <4d9223c9-216b-d79d-5c93-ec81466c75c6@redhat.com> Message-ID: Thanks Martin for your reply. It would be cool if i can have IPA client to resolve IPA server without specifying nameserver in resolv.conf How do i configure zone delegation? is there any document i can refer? Many Thanks,Deepak Subject: Re: [Freeipa-users] General query regarding nameserver enrtry To: deepak_dimri at hotmail.com; freeipa-users at redhat.com From: mbasti at redhat.com Date: Mon, 5 Sep 2016 09:12:08 +0200 On 02.09.2016 20:06, Deepak Dimri wrote: Hi All, My ipa-client-install fails until etc/resolv.conf gets updated with IPA nameserver entry. I want to avoid a task of updating resolve.conf in my automation script. Is there a way i can get my IPA client installation successful without updating resolve.conf? what options do i have? Many Thanks, Deepak Hello, do you have proper zone delegation? With proper zone delegation it should be able to resolve IPA from every nameserver. Martin -------------- next part -------------- An HTML attachment was scrubbed... URL: From rcritten at redhat.com Thu Sep 8 13:25:13 2016 From: rcritten at redhat.com (Rob Crittenden) Date: Thu, 8 Sep 2016 09:25:13 -0400 Subject: [Freeipa-users] ipa: ERROR: Certificate format error: (SEC_ERROR_LEGACY_DATABASE) The certificate/key database is in an old, unsupported format. In-Reply-To: References: <57D015D8.7080507@redhat.com> Message-ID: <57D166B9.3070201@redhat.com> Natxo Asenjo wrote: > I do see these errors: > [Wed Sep 07 15:56:13 2016] [error] ipa: INFO:: ping(): SUCCESS > [Wed Sep 07 15:56:13 2016] [error] ipa: INFO: : host_find(u'tftp-1801', > all=False, raw=False, version=u'2.49', no_members=False, > pkey_only=False): CertificateFormatError > [Wed Sep 07 15:56:44 2016] [error] ipa: INFO: : ping(): SUCCESS > [Wed Sep 07 15:56:44 2016] [error] ipa: INFO: : host_find(u'tftp-1801', > all=False, raw=False, version=u'2.49', no_members=False, > pkey_only=False): CertificateFormatError > [Wed Sep 07 15:57:57 2016] [error] ipa: INFO: : ping(): SUCCESS > [Wed Sep 07 15:57:58 2016] [error] ipa: INFO: : host_find(u'tftp-1801', > all=False, raw=False, version=u'2.49', no_members=False, > pkey_only=False): CertificateFormatErro > > > On Wed, Sep 7, 2016 at 4:01 PM, Natxo Asenjo > wrote: > > > alas, not woriking again. > > On the one kdc > > $ ipa host-find tftp-1801 > ipa: ERROR: Certificate format error: (SEC_ERROR_LEGACY_DATABASE) > The certificate/key database is in an old, unsupported format. > > On the other: > > $ ipa host-find tftp-1801 > -------------- > 1 host matched > -------------- > Host name: tftp-1801.sub.domain.tld > ..... > > After rebooting the kdc with the error, no new tracebacks in the > error_log No new tracebacks but still not working? The CertificateFormatError is the server logging the equivalent of what you're seeing in the client. rob From g.kafataridis at nelios.com Thu Sep 8 14:37:48 2016 From: g.kafataridis at nelios.com (Giorgos Kafa) Date: Thu, 8 Sep 2016 17:37:48 +0300 Subject: [Freeipa-users] CA: Cannot add Centos7.2 replica to Centos6.8 ipa server Message-ID: <45b66871-aa01-3c91-0630-0bbeee488ae7@nelios.com> Hello, I am trying to migrate and upgrade my main freeipa installation, so I decided to replicate it and phase it out of our intranet. I manage to get over some obstacles as I had to recreate my cacert.p12 file, but now I am facing an issue that prevents me from setting up CA on the replicated server. Both of my servers are fully updated. I have tried some solutions and found similar other threads but nothing worked. Without CA the replication finalizes without issues but I do not want that. Is there any other way to perform the migration? * Similar issues:* https://fedorahosted.org/freeipa/ticket/5581 https://www.redhat.com/archives/freeipa-users/2016-February/msg00183.html *Master Server has:* CentOS release 6.8 (Final) ipa-server.x86_64 3.0.0-50 pki-util.noarch 9.0.3-50 *Replication Server has:* CentOS Linux release 7.2.1511 (Core) ipa 4.2.0 *ERROR LOG* ipa : DEBUG Starting external process ipa : DEBUG args='/usr/sbin/pkispawn' '-s' 'CA' '-f' '/tmp/tmpuuJbGt' ipa : DEBUG Process finished, return code=1 ipa : DEBUG stdout=Log file: /var/log/pki/pki-ca-spawn.20160908092308.log Loading deployment configuration from /tmp/tmpuuJbGt. Installing CA into /var/lib/pki/pki-tomcat. Storing deployment configuration into /etc/sysconfig/pki/tomcat/pki-tomcat/ca/deployment.cfg. Installation failed. *ipa : DEBUG stderr=/usr/lib/python2.7/site-packages/urllib3/connectionpool.py:769: InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.org/en/latest/security.html** ** InsecureRequestWarning)** **pkispawn : WARNING ....... unable to validate security domain user/password through REST interface. Interface not available** **pkispawn : ERROR ....... Exception from Java Configuration Servlet: 500 Server Error: Internal Server Error** **pkispawn : ERROR ....... ParseError: not well-formed (invalid token): line 1, column 0: {"Attributes":{"Attribute":[]},"ClassName":"com.netscape.certsrv.base.PKIException","Code":500,"Message":"Failed to obtain installation token from security domain"} * ipa.ipaserver.install.cainstance.CAInstance: CRITICAL Failed to configure CA instance: Command ''/usr/sbin/pkispawn' '-s' 'CA' '-f' '/tmp/tmpuuJbGt'' returned non-zero exit status 1 ipa.ipaserver.install.cainstance.CAInstance: CRITICAL See the installation logs and the following files/directories for more information: ipa.ipaserver.install.cainstance.CAInstance: CRITICAL /var/log/pki-ca-install.log ipa.ipaserver.install.cainstance.CAInstance: CRITICAL /var/log/pki/pki-tomcat ipa : DEBUG Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/ipaserver/install/service.py", line 418, in start_creation run_step(full_msg, method) File "/usr/lib/python2.7/site-packages/ipaserver/install/service.py", line 408, in run_step method() File "/usr/lib/python2.7/site-packages/ipaserver/install/cainstance.py", line 622, in __spawn_instance DogtagInstance.spawn_instance(self, cfg_file) File "/usr/lib/python2.7/site-packages/ipaserver/install/dogtaginstance.py", line 201, in spawn_instance self.handle_setup_error(e) File "/usr/lib/python2.7/site-packages/ipaserver/install/dogtaginstance.py", line 465, in handle_setup_error raise RuntimeError("%s configuration failed." % self.subsystem) RuntimeError: CA configuration failed. ipa : DEBUG [error] RuntimeError: CA configuration failed. [error] RuntimeError: CA configuration failed. Your system may be partly configured. Run /usr/sbin/ipa-server-install --uninstall to clean up. ipa.ipapython.install.cli.install_tool(Replica): DEBUG File "/usr/lib/python2.7/site-packages/ipapython/admintool.py", line 171, in execute return_value = self.run() File "/usr/lib/python2.7/site-packages/ipapython/install/cli.py", line 311, in run cfgr.run() File "/usr/lib/python2.7/site-packages/ipapython/install/core.py", line 281, in run self.execute() File "/usr/lib/python2.7/site-packages/ipapython/install/core.py", line 303, in execute for nothing in self._executor(): File "/usr/lib/python2.7/site-packages/ipapython/install/core.py", line 343, in __runner self._handle_exception(exc_info) File "/usr/lib/python2.7/site-packages/ipapython/install/core.py", line 365, in _handle_exception util.raise_exc_info(exc_info) File "/usr/lib/python2.7/site-packages/ipapython/install/core.py", line 333, in __runner step() File "/usr/lib/python2.7/site-packages/ipapython/install/util.py", line 87, in run_generator_with_yield_from raise_exc_info(exc_info) File "/usr/lib/python2.7/site-packages/ipapython/install/util.py", line 65, in run_generator_with_yield_from value = gen.send(prev_value) File "/usr/lib/python2.7/site-packages/ipapython/install/core.py", line 539, in _configure executor.next() File "/usr/lib/python2.7/site-packages/ipapython/install/core.py", line 343, in __runner self._handle_exception(exc_info) File "/usr/lib/python2.7/site-packages/ipapython/install/core.py", line 421, in _handle_exception self.__parent._handle_exception(exc_info) File "/usr/lib/python2.7/site-packages/ipapython/install/core.py", line 365, in _handle_exception util.raise_exc_info(exc_info) File "/usr/lib/python2.7/site-packages/ipapython/install/core.py", line 418, in _handle_exception super(ComponentBase, self)._handle_exception(exc_info) File "/usr/lib/python2.7/site-packages/ipapython/install/core.py", line 365, in _handle_exception util.raise_exc_info(exc_info) File "/usr/lib/python2.7/site-packages/ipapython/install/core.py", line 333, in __runner step() File "/usr/lib/python2.7/site-packages/ipapython/install/util.py", line 87, in run_generator_with_yield_from raise_exc_info(exc_info) File "/usr/lib/python2.7/site-packages/ipapython/install/util.py", line 65, in run_generator_with_yield_from value = gen.send(prev_value) File "/usr/lib/python2.7/site-packages/ipapython/install/common.py", line 63, in _install for nothing in self._installer(self.parent): File "/usr/lib/python2.7/site-packages/ipaserver/install/server/replicainstall.py", line 901, in main install(self) File "/usr/lib/python2.7/site-packages/ipaserver/install/server/replicainstall.py", line 295, in decorated func(installer) File "/usr/lib/python2.7/site-packages/ipaserver/install/server/replicainstall.py", line 616, in install ca.install(False, config, options) File "/usr/lib/python2.7/site-packages/ipaserver/install/ca.py", line 114, in install install_step_0(standalone, replica_config, options) File "/usr/lib/python2.7/site-packages/ipaserver/install/ca.py", line 138, in install_step_0 ra_p12=getattr(options, 'ra_p12', None)) File "/usr/lib/python2.7/site-packages/ipaserver/install/cainstance.py", line 1545, in install_replica_ca subject_base=config.subject_base) File "/usr/lib/python2.7/site-packages/ipaserver/install/cainstance.py", line 488, in configure_instance self.start_creation(runtime=210) File "/usr/lib/python2.7/site-packages/ipaserver/install/service.py", line 418, in start_creation run_step(full_msg, method) File "/usr/lib/python2.7/site-packages/ipaserver/install/service.py", line 408, in run_step method() File "/usr/lib/python2.7/site-packages/ipaserver/install/cainstance.py", line 622, in __spawn_instance DogtagInstance.spawn_instance(self, cfg_file) File "/usr/lib/python2.7/site-packages/ipaserver/install/dogtaginstance.py", line 201, in spawn_instance self.handle_setup_error(e) File "/usr/lib/python2.7/site-packages/ipaserver/install/dogtaginstance.py", line 465, in handle_setup_error raise RuntimeError("%s configuration failed." % self.subsystem) ipa.ipapython.install.cli.install_tool(Replica): DEBUG The ipa-replica-install command failed, exception: RuntimeError: CA configuration failed. ipa.ipapython.install.cli.install_tool(Replica): ERROR CA configuration failed. -------------- next part -------------- An HTML attachment was scrubbed... URL: From natxo.asenjo at gmail.com Thu Sep 8 16:30:35 2016 From: natxo.asenjo at gmail.com (Natxo Asenjo) Date: Thu, 8 Sep 2016 18:30:35 +0200 Subject: [Freeipa-users] ipa: ERROR: Certificate format error: (SEC_ERROR_LEGACY_DATABASE) The certificate/key database is in an old, unsupported format. In-Reply-To: <57D166B9.3070201@redhat.com> References: <57D015D8.7080507@redhat.com> <57D166B9.3070201@redhat.com> Message-ID: On Thu, Sep 8, 2016 at 3:25 PM, Rob Crittenden wrote: > Natxo Asenjo wrote: > >> I do see these errors: >> [Wed Sep 07 15:56:13 2016] [error] ipa: INFO:: ping(): SUCCESS >> [Wed Sep 07 15:56:13 2016] [error] ipa: INFO: : host_find(u'tftp-1801', >> all=False, raw=False, version=u'2.49', no_members=False, >> pkey_only=False): CertificateFormatError >> [Wed Sep 07 15:56:44 2016] [error] ipa: INFO: : ping(): SUCCESS >> [Wed Sep 07 15:56:44 2016] [error] ipa: INFO: : host_find(u'tftp-1801', >> all=False, raw=False, version=u'2.49', no_members=False, >> pkey_only=False): CertificateFormatError >> [Wed Sep 07 15:57:57 2016] [error] ipa: INFO: : ping(): SUCCESS >> [Wed Sep 07 15:57:58 2016] [error] ipa: INFO: : host_find(u'tftp-1801', >> all=False, raw=False, version=u'2.49', no_members=False, >> pkey_only=False): CertificateFormatErro >> >> >> On Wed, Sep 7, 2016 at 4:01 PM, Natxo Asenjo > > wrote: >> >> >> alas, not woriking again. >> >> On the one kdc >> >> $ ipa host-find tftp-1801 >> ipa: ERROR: Certificate format error: (SEC_ERROR_LEGACY_DATABASE) >> The certificate/key database is in an old, unsupported format. >> >> On the other: >> >> $ ipa host-find tftp-1801 >> -------------- >> 1 host matched >> -------------- >> Host name: tftp-1801.sub.domain.tld >> ..... >> >> After rebooting the kdc with the error, no new tracebacks in the >> error_log >> > > No new tracebacks but still not working? > > The CertificateFormatError is the server logging the equivalent of what > you're seeing in the client. > > rob > that's right. Is there anything else I can look at? -- -- Groeten, natxo -------------- next part -------------- An HTML attachment was scrubbed... URL: From natxo.asenjo at gmail.com Thu Sep 8 17:54:23 2016 From: natxo.asenjo at gmail.com (Natxo Asenjo) Date: Thu, 8 Sep 2016 19:54:23 +0200 Subject: [Freeipa-users] CA: Cannot add Centos7.2 replica to Centos6.8 ipa server In-Reply-To: <45b66871-aa01-3c91-0630-0bbeee488ae7@nelios.com> References: <45b66871-aa01-3c91-0630-0bbeee488ae7@nelios.com> Message-ID: Hi Giorgios, On Thu, Sep 8, 2016 at 4:37 PM, Giorgos Kafa wrote: > Hello, I am trying to migrate and upgrade my main freeipa installation, > so I decided to replicate it and phase it out of our intranet. > I manage to get over some obstacles as I had to recreate my cacert.p12 > file, but now I am facing an issue that prevents me from setting up CA on > the replicated server. > first off, did you follow the instructions in https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/7/html/Linux_Domain_Identity_Authentication_and_Policy_Guide/upgrading.html I have tested migrations several times with PKI and they all went fine. -- regards, natxo -------------- next part -------------- An HTML attachment was scrubbed... URL: From g.kafataridis at nelios.com Fri Sep 9 10:13:48 2016 From: g.kafataridis at nelios.com (Giorgos Kafataridis) Date: Fri, 9 Sep 2016 13:13:48 +0300 Subject: [Freeipa-users] CA: Cannot add Centos7.2 replica to Centos6.8 ipa server In-Reply-To: <45b66871-aa01-3c91-0630-0bbeee488ae7@nelios.com> References: <45b66871-aa01-3c91-0630-0bbeee488ae7@nelios.com> Message-ID: <323b9de1-adb3-47af-d4a7-d6083f88d75c@nelios.com> Yes, I have followed https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/7/html/Linux_Domain_Identity_Authentication_and_Policy_Guide/upgrading.html to the letter. The only reason I had to recreate the cacert.p12 file is because it is not renewed automatically in v3, so the cacert.p12 was outdated and the CA was throwing an "p12 invalid digest" error. * I opened all necessary ports * I checked all certs and they are valid for another year /Run connection check to master// //Check connection from replica to remote master 'ipa-server.nelios':// // Directory Service: Unsecure port (389): OK// // Directory Service: Secure port (636): OK// // Kerberos KDC: TCP (88): OK// // Kerberos Kpasswd: TCP (464): OK// // HTTP Server: Unsecure port (80): OK// // HTTP Server: Secure port (443): OK// // PKI-CA: Directory Service port (7389): OK// // //The following list of ports use UDP protocol and would need to be// //checked manually:// // Kerberos KDC: UDP (88): SKIPPED// // Kerberos Kpasswd: UDP (464): SKIPPED// // //Connection from replica to master is OK.// //Start listening on required ports for remote master check// //Get credentials to log in to remote master// //Check SSH connection to remote master// //Execute check on remote master// //Check connection from master to remote replica 'ipa2-server2.nelios':// // Directory Service: Unsecure port (389): OK// // Directory Service: Secure port (636): OK// // Kerberos KDC: TCP (88): OK// // Kerberos KDC: UDP (88): OK// // Kerberos Kpasswd: TCP (464): OK// // Kerberos Kpasswd: UDP (464): OK// // HTTP Server: Unsecure port (80): OK// // HTTP Server: Secure port (443): OK// // //Connection from master to replica is OK.// // //Connection check OK/ *Even with a fresh install of centos 7 with different hostname and ip and I still get the the error below* Configuring certificate server (pki-tomcatd). Estimated time: 3 minutes 30 seconds [1/24]: creating certificate server user [2/24]: configuring certificate server instance ipa.ipaserver.install.cainstance.CAInstance: CRITICAL Failed to configure CA instance: Command ''/usr/sbin/pkispawn' '-s' 'CA' '-f' '/tmp/tmpbMwmp_'' returned non-zero exit status 1 ipa.ipaserver.install.cainstance.CAInstance: CRITICAL See the installation logs and the following files/directories for more information: ipa.ipaserver.install.cainstance.CAInstance: CRITICAL /var/log/pki-ca-install.log ipa.ipaserver.install.cainstance.CAInstance: CRITICAL /var/log/pki/pki-tomcat [error] RuntimeError: CA configuration failed. Your system may be partly configured. Run /usr/sbin/ipa-server-install --uninstall to clean up. ipa.ipapython.install.cli.install_tool(Replica): ERROR CA configuration failed. * **With debug enabled I get: * pa : DEBUG Starting external process ipa : DEBUG args='/usr/sbin/pkispawn' '-s' 'CA' '-f' '/tmp/tmpwY8XjR' ipa : DEBUG Process finished, return code=1 ipa : DEBUG stdout=Log file: /var/log/pki/pki-ca-spawn.20160909044214.log Loading deployment configuration from /tmp/tmpwY8XjR. Installing CA into /var/lib/pki/pki-tomcat. Storing deployment configuration into /etc/sysconfig/pki/tomcat/pki-tomcat/ca/deployment.cfg. Installation failed. ipa : DEBUG stderr=/usr/lib/python2.7/site-packages/urllib3/connectionpool.py:769: InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.org/en/latest/security.html InsecureRequestWarning) pkispawn : WARNING ....... unable to validate security domain user/password through REST interface. Interface not available pkispawn : ERROR ....... Exception from Java Configuration Servlet: 500 Server Error: Internal Server Error pkispawn : ERROR ....... ParseError: not well-formed (invalid token): line 1, column 0: {"Attributes":{"Attribute":[]},"ClassName":"com.netscape.certsrv.base.PKIException","Code":500,"Message":"Failed to obtain installation token from security domain"} Is there a way to validate the repilca .gpg file from a v3 installation against a v4.2 freeipa installation to check for any errors before going through the ipa-replica-install? The ipa-replica-install completes if I don't include the --setup-ca flag but I don't want that -------------- next part -------------- An HTML attachment was scrubbed... URL: From lslebodn at redhat.com Fri Sep 9 10:22:28 2016 From: lslebodn at redhat.com (Lukas Slebodnik) Date: Fri, 9 Sep 2016 12:22:28 +0200 Subject: [Freeipa-users] Issues with FreeIPA SSH Key authentication In-Reply-To: References: Message-ID: <20160909102227.GD19340@10.4.128.1> On (07/09/16 17:39), Venkataramana Kintali wrote: >Hi, >Of late, I am learning FreeIPA . I have installed IPA server and few >clients (Version 3.0.0) >I am facing an issue with ssh key authentication in my setup. >I generated a putty ssh private key (using putty keygen) ,and uploaded it >under a user through IPA GUI. I assume you uploaded public key to the IPA otherwise you did something wrong and I wonder why it works on some machines. >I am able to login to some IPA clients but not able to login to other IPA >clients with putty using private key and passphrase. > Is sssd_ssh running on all clients? (Is sssd.conf almost the same on all machines) Is sshd configuration the same on all machines? /etc/ssh/ssh_config /etc/ssh/sshd_config >Public Key Authentication is enabled on all clients. >I am able to from one client to other clients successfully (after doing >kinit) without promting password. > >Can someone please throw some light on this as to what the issue could be >here and what else I can check to understand where the problem is ? > >I searched this online but couldn't find any solution in the context of IPA. > LS From pvoborni at redhat.com Fri Sep 9 12:06:55 2016 From: pvoborni at redhat.com (Petr Vobornik) Date: Fri, 9 Sep 2016 14:06:55 +0200 Subject: [Freeipa-users] CA: Cannot add Centos7.2 replica to Centos6.8 ipa server In-Reply-To: <323b9de1-adb3-47af-d4a7-d6083f88d75c@nelios.com> References: <45b66871-aa01-3c91-0630-0bbeee488ae7@nelios.com> <323b9de1-adb3-47af-d4a7-d6083f88d75c@nelios.com> Message-ID: On 09/09/2016 12:13 PM, Giorgos Kafataridis wrote: > Yes, I have followed > https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/7/html/Linux_Domain_Identity_Authentication_and_Policy_Guide/upgrading.html > to the letter. > The only reason I had to recreate the cacert.p12 file is because it is not > renewed automatically in v3, so the cacert.p12 was outdated and the CA was > throwing an "p12 invalid digest" error. > > * I opened all necessary ports > * I checked all certs and they are valid for another year > > > /Run connection check to master// > //Check connection from replica to remote master 'ipa-server.nelios':// > // Directory Service: Unsecure port (389): OK// > // Directory Service: Secure port (636): OK// > // Kerberos KDC: TCP (88): OK// > // Kerberos Kpasswd: TCP (464): OK// > // HTTP Server: Unsecure port (80): OK// > // HTTP Server: Secure port (443): OK// > // PKI-CA: Directory Service port (7389): OK// > // > //The following list of ports use UDP protocol and would need to be// > //checked manually:// > // Kerberos KDC: UDP (88): SKIPPED// > // Kerberos Kpasswd: UDP (464): SKIPPED// > // > //Connection from replica to master is OK.// > //Start listening on required ports for remote master check// > //Get credentials to log in to remote master// > //Check SSH connection to remote master// > //Execute check on remote master// > //Check connection from master to remote replica 'ipa2-server2.nelios':// > // Directory Service: Unsecure port (389): OK// > // Directory Service: Secure port (636): OK// > // Kerberos KDC: TCP (88): OK// > // Kerberos KDC: UDP (88): OK// > // Kerberos Kpasswd: TCP (464): OK// > // Kerberos Kpasswd: UDP (464): OK// > // HTTP Server: Unsecure port (80): OK// > // HTTP Server: Secure port (443): OK// > // > //Connection from master to replica is OK.// > // > //Connection check OK/ > > *Even with a fresh install of centos 7 with different hostname and ip and I > still get the the error below* > > Configuring certificate server (pki-tomcatd). Estimated time: 3 minutes 30 seconds > [1/24]: creating certificate server user > [2/24]: configuring certificate server instance > ipa.ipaserver.install.cainstance.CAInstance: CRITICAL Failed to configure CA > instance: Command ''/usr/sbin/pkispawn' '-s' 'CA' '-f' '/tmp/tmpbMwmp_'' > returned non-zero exit status 1 > ipa.ipaserver.install.cainstance.CAInstance: CRITICAL See the installation logs > and the following files/directories for more information: > ipa.ipaserver.install.cainstance.CAInstance: CRITICAL /var/log/pki-ca-install.log > ipa.ipaserver.install.cainstance.CAInstance: CRITICAL /var/log/pki/pki-tomcat > [error] RuntimeError: CA configuration failed. > Your system may be partly configured. > Run /usr/sbin/ipa-server-install --uninstall to clean up. > > ipa.ipapython.install.cli.install_tool(Replica): ERROR CA configuration failed. > > * > **With debug enabled I get: * > > pa : DEBUG Starting external process > ipa : DEBUG args='/usr/sbin/pkispawn' '-s' 'CA' '-f' '/tmp/tmpwY8XjR' > ipa : DEBUG Process finished, return code=1 > ipa : DEBUG stdout=Log file: /var/log/pki/pki-ca-spawn.20160909044214.log > Loading deployment configuration from /tmp/tmpwY8XjR. > Installing CA into /var/lib/pki/pki-tomcat. > Storing deployment configuration into > /etc/sysconfig/pki/tomcat/pki-tomcat/ca/deployment.cfg. > > Installation failed. > > > ipa : DEBUG > stderr=/usr/lib/python2.7/site-packages/urllib3/connectionpool.py:769: > InsecureRequestWarning: Unverified HTTPS request is being made. Adding > certificate verification is strongly advised. See: > https://urllib3.readthedocs.org/en/latest/security.html > InsecureRequestWarning) > pkispawn : WARNING ....... unable to validate security domain user/password > through REST interface. Interface not available > pkispawn : ERROR ....... Exception from Java Configuration Servlet: 500 > Server Error: Internal Server Error > pkispawn : ERROR ....... ParseError: not well-formed (invalid token): line > 1, column 0: > {"Attributes":{"Attribute":[]},"ClassName":"com.netscape.certsrv.base.PKIException","Code":500,"Message":"Failed > to obtain installation token from security domain"} > > > Is there a way to validate the repilca .gpg file from a v3 installation against > a v4.2 freeipa installation to check for any errors before going through the > ipa-replica-install? > The ipa-replica-install completes if I don't include the --setup-ca flag but I > don't want that > There is no automatic method to verify the replica file. Could you share the stack trace from /var/log/pki/pki-tomcat/ca/debug + couple lines before and after? -- Petr Vobornik From larry.rosen at JDRSolutions.com Thu Sep 8 19:16:29 2016 From: larry.rosen at JDRSolutions.com (Larry Rosen) Date: Thu, 8 Sep 2016 19:16:29 +0000 Subject: [Freeipa-users] automated ftp service only accounts and passwords Message-ID: <79B7CEE400C91A4C9FD8BF082D8226073E3700C7@JDRPDC.JDRSolutions.local> How do I set the password on a chroot jailed sftp id account that is not allowed a shell to not expire its password after setting it? There's no way to change it to the fixed password I want. I have created a service_account password policy that has no expiration (set to Max lifetime (days) = 20000 ). Larry -------------- next part -------------- An HTML attachment was scrubbed... URL: From g.kafataridis at nelios.com Fri Sep 9 12:33:51 2016 From: g.kafataridis at nelios.com (Giorgos Kafataridis) Date: Fri, 9 Sep 2016 15:33:51 +0300 Subject: [Freeipa-users] CA: Cannot add Centos7.2 replica to Centos6.8 ipa server In-Reply-To: References: <45b66871-aa01-3c91-0630-0bbeee488ae7@nelios.com> <323b9de1-adb3-47af-d4a7-d6083f88d75c@nelios.com> Message-ID: >> Yes, I have followed >> https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/7/html/Linux_Domain_Identity_Authentication_and_Policy_Guide/upgrading.html >> to the letter. >> The only reason I had to recreate the cacert.p12 file is because it is not >> renewed automatically in v3, so the cacert.p12 was outdated and the CA was >> throwing an "p12 invalid digest" error. >> >> * I opened all necessary ports >> * I checked all certs and they are valid for another year >> >> >> /Run connection check to master// >> //Check connection from replica to remote master 'ipa-server.nelios':// >> // Directory Service: Unsecure port (389): OK// >> // Directory Service: Secure port (636): OK// >> // Kerberos KDC: TCP (88): OK// >> // Kerberos Kpasswd: TCP (464): OK// >> // HTTP Server: Unsecure port (80): OK// >> // HTTP Server: Secure port (443): OK// >> // PKI-CA: Directory Service port (7389): OK// >> // >> //The following list of ports use UDP protocol and would need to be// >> //checked manually:// >> // Kerberos KDC: UDP (88): SKIPPED// >> // Kerberos Kpasswd: UDP (464): SKIPPED// >> // >> //Connection from replica to master is OK.// >> //Start listening on required ports for remote master check// >> //Get credentials to log in to remote master// >> //Check SSH connection to remote master// >> //Execute check on remote master// >> //Check connection from master to remote replica 'ipa2-server2.nelios':// >> // Directory Service: Unsecure port (389): OK// >> // Directory Service: Secure port (636): OK// >> // Kerberos KDC: TCP (88): OK// >> // Kerberos KDC: UDP (88): OK// >> // Kerberos Kpasswd: TCP (464): OK// >> // Kerberos Kpasswd: UDP (464): OK// >> // HTTP Server: Unsecure port (80): OK// >> // HTTP Server: Secure port (443): OK// >> // >> //Connection from master to replica is OK.// >> // >> //Connection check OK/ >> >> *Even with a fresh install of centos 7 with different hostname and ip and I >> still get the the error below* >> >> Configuring certificate server (pki-tomcatd). Estimated time: 3 minutes 30 seconds >> [1/24]: creating certificate server user >> [2/24]: configuring certificate server instance >> ipa.ipaserver.install.cainstance.CAInstance: CRITICAL Failed to configure CA >> instance: Command ''/usr/sbin/pkispawn' '-s' 'CA' '-f' '/tmp/tmpbMwmp_'' >> returned non-zero exit status 1 >> ipa.ipaserver.install.cainstance.CAInstance: CRITICAL See the installation logs >> and the following files/directories for more information: >> ipa.ipaserver.install.cainstance.CAInstance: CRITICAL /var/log/pki-ca-install.log >> ipa.ipaserver.install.cainstance.CAInstance: CRITICAL /var/log/pki/pki-tomcat >> [error] RuntimeError: CA configuration failed. >> Your system may be partly configured. >> Run /usr/sbin/ipa-server-install --uninstall to clean up. >> >> ipa.ipapython.install.cli.install_tool(Replica): ERROR CA configuration failed. >> >> * >> **With debug enabled I get: * >> >> pa : DEBUG Starting external process >> ipa : DEBUG args='/usr/sbin/pkispawn' '-s' 'CA' '-f' '/tmp/tmpwY8XjR' >> ipa : DEBUG Process finished, return code=1 >> ipa : DEBUG stdout=Log file: /var/log/pki/pki-ca-spawn.20160909044214.log >> Loading deployment configuration from /tmp/tmpwY8XjR. >> Installing CA into /var/lib/pki/pki-tomcat. >> Storing deployment configuration into >> /etc/sysconfig/pki/tomcat/pki-tomcat/ca/deployment.cfg. >> >> Installation failed. >> >> >> ipa : DEBUG >> stderr=/usr/lib/python2.7/site-packages/urllib3/connectionpool.py:769: >> InsecureRequestWarning: Unverified HTTPS request is being made. Adding >> certificate verification is strongly advised. See: >> https://urllib3.readthedocs.org/en/latest/security.html >> InsecureRequestWarning) >> pkispawn : WARNING ....... unable to validate security domain user/password >> through REST interface. Interface not available >> pkispawn : ERROR ....... Exception from Java Configuration Servlet: 500 >> Server Error: Internal Server Error >> pkispawn : ERROR ....... ParseError: not well-formed (invalid token): line >> 1, column 0: >> {"Attributes":{"Attribute":[]},"ClassName":"com.netscape.certsrv.base.PKIException","Code":500,"Message":"Failed >> to obtain installation token from security domain"} >> >> >> Is there a way to validate the repilca .gpg file from a v3 installation against >> a v4.2 freeipa installation to check for any errors before going through the >> ipa-replica-install? >> The ipa-replica-install completes if I don't include the --setup-ca flag but I >> don't want that >> > There is no automatic method to verify the replica file. > > Could you share the stack trace from /var/log/pki/pki-tomcat/ca/debug + > couple lines before and after? > > Contents of /var/log/pki/pki-tomcat/ca/debug: [09/Sep/2016:08:22:51][http-bio-8443-exec-3]: MessageFormatInterceptor: SystemConfigResource.configure() [09/Sep/2016:08:22:51][http-bio-8443-exec-3]: MessageFormatInterceptor: content-type: application/json [09/Sep/2016:08:22:51][http-bio-8443-exec-3]: MessageFormatInterceptor: accept: [application/json] [09/Sep/2016:08:22:51][http-bio-8443-exec-3]: MessageFormatInterceptor: request format: application/json [09/Sep/2016:08:22:51][http-bio-8443-exec-3]: MessageFormatInterceptor: response format: application/json [09/Sep/2016:08:22:51][http-bio-8443-exec-3]: SystemConfigService: configure() [09/Sep/2016:08:22:51][http-bio-8443-exec-3]: SystemConfigService: request: ConfigurationRequest [pin=XXXX, token=Internal Key Storage Token, tokenPassword=XXXX, securityDomainType=existingdomain, securityDomainUri=https://ipa-server.nelios:443, securityDomainName=null, securityDomainUser=admin, securityDomainPassword=XXXX, isClone=true, cloneUri=https://ipa-server.nelios:443, subsystemName=CA ipa2-server2.nelios 8443, p12File=/tmp/ca.p12, p12Password=XXXX, hierarchy=root, dsHost=ipa2-server2.nelios, dsPort=389, baseDN=o=ipaca, bindDN=cn=Directory Manager, bindpwd=XXXX, database=ipaca, secureConn=false, removeData=true, replicateSchema=false, masterReplicationPort=7389, cloneReplicationPort=389, replicationSecurity=TLS, systemCerts=[com.netscape.certsrv.system.SystemCertData at 434a841], issuingCA=https://ipa-server.nelios:443, backupKeys=true, backupPassword=XXXX, backupFile=/etc/pki/pki-tomcat/alias/ca_backup_keys.p12, adminUID=null, adminPassword=XXXX, adminEmail=null, adminCertRequest=null, adminCertRequestType=null, adminSubjectDN=null, adminName=null, adminProfileID=null, adminCert=null, importAdminCert=false, generateServerCert=true, external=false, standAlone=false, stepTwo=false, authdbBaseDN=null, authdbHost=null, authdbPort=null, authdbSecureConn=null, caUri=null, kraUri=null, tksUri=null, enableServerSideKeyGen=null, importSharedSecret=null, generateSubsystemCert=null, sharedDB=false, sharedDBUserDN=null, createNewDB=true, setupReplication=True, subordinateSecurityDomainNamenull] [09/Sep/2016:08:22:51][http-bio-8443-exec-3]: === Token Panel === [09/Sep/2016:08:22:51][http-bio-8443-exec-3]: === Security Domain Panel === [09/Sep/2016:08:22:51][http-bio-8443-exec-3]: Joining existing security domain [09/Sep/2016:08:22:51][http-bio-8443-exec-3]: Resolving security domain URLhttps://ipa-server.nelios:443 [09/Sep/2016:08:22:51][http-bio-8443-exec-3]: Getting security domain cert chain [09/Sep/2016:08:22:52][http-bio-8443-exec-3]: Getting install token [09/Sep/2016:08:22:52][http-bio-8443-exec-3]: Getting install token [09/Sep/2016:08:22:52][http-bio-8443-exec-3]: Getting old cookie [09/Sep/2016:08:22:52][http-bio-8443-exec-3]: Token: null [09/Sep/2016:08:22:52][http-bio-8443-exec-3]: Install token is null [09/Sep/2016:08:22:52][http-bio-8443-exec-3]: Failed to obtain installation token from security domain I assume it is the null token the perpetrator ? if yes what should I fix on master? From pvoborni at redhat.com Fri Sep 9 13:09:25 2016 From: pvoborni at redhat.com (Petr Vobornik) Date: Fri, 9 Sep 2016 15:09:25 +0200 Subject: [Freeipa-users] CA: Cannot add Centos7.2 replica to Centos6.8 ipa server In-Reply-To: References: <45b66871-aa01-3c91-0630-0bbeee488ae7@nelios.com> <323b9de1-adb3-47af-d4a7-d6083f88d75c@nelios.com> Message-ID: <9afdf0f3-97d1-cc6e-5f3d-0e11687a124a@redhat.com> On 09/09/2016 02:33 PM, Giorgos Kafataridis wrote: > >>> Yes, I have followed >>> https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/7/html/Linux_Domain_Identity_Authentication_and_Policy_Guide/upgrading.html >>> >>> to the letter. >>> The only reason I had to recreate the cacert.p12 file is because it >>> is not >>> renewed automatically in v3, so the cacert.p12 was outdated and the >>> CA was >>> throwing an "p12 invalid digest" error. >>> >>> * I opened all necessary ports >>> * I checked all certs and they are valid for another year >>> >>> >>> /Run connection check to master// >>> //Check connection from replica to remote master 'ipa-server.nelios':// >>> // Directory Service: Unsecure port (389): OK// >>> // Directory Service: Secure port (636): OK// >>> // Kerberos KDC: TCP (88): OK// >>> // Kerberos Kpasswd: TCP (464): OK// >>> // HTTP Server: Unsecure port (80): OK// >>> // HTTP Server: Secure port (443): OK// >>> // PKI-CA: Directory Service port (7389): OK// >>> // >>> //The following list of ports use UDP protocol and would need to be// >>> //checked manually:// >>> // Kerberos KDC: UDP (88): SKIPPED// >>> // Kerberos Kpasswd: UDP (464): SKIPPED// >>> // >>> //Connection from replica to master is OK.// >>> //Start listening on required ports for remote master check// >>> //Get credentials to log in to remote master// >>> //Check SSH connection to remote master// >>> //Execute check on remote master// >>> //Check connection from master to remote replica >>> 'ipa2-server2.nelios':// >>> // Directory Service: Unsecure port (389): OK// >>> // Directory Service: Secure port (636): OK// >>> // Kerberos KDC: TCP (88): OK// >>> // Kerberos KDC: UDP (88): OK// >>> // Kerberos Kpasswd: TCP (464): OK// >>> // Kerberos Kpasswd: UDP (464): OK// >>> // HTTP Server: Unsecure port (80): OK// >>> // HTTP Server: Secure port (443): OK// >>> // >>> //Connection from master to replica is OK.// >>> // >>> //Connection check OK/ >>> >>> *Even with a fresh install of centos 7 with different hostname and ip >>> and I >>> still get the the error below* >>> >>> Configuring certificate server (pki-tomcatd). Estimated time: 3 >>> minutes 30 seconds >>> [1/24]: creating certificate server user >>> [2/24]: configuring certificate server instance >>> ipa.ipaserver.install.cainstance.CAInstance: CRITICAL Failed to >>> configure CA >>> instance: Command ''/usr/sbin/pkispawn' '-s' 'CA' '-f' '/tmp/tmpbMwmp_'' >>> returned non-zero exit status 1 >>> ipa.ipaserver.install.cainstance.CAInstance: CRITICAL See the >>> installation logs >>> and the following files/directories for more information: >>> ipa.ipaserver.install.cainstance.CAInstance: CRITICAL >>> /var/log/pki-ca-install.log >>> ipa.ipaserver.install.cainstance.CAInstance: CRITICAL >>> /var/log/pki/pki-tomcat >>> [error] RuntimeError: CA configuration failed. >>> Your system may be partly configured. >>> Run /usr/sbin/ipa-server-install --uninstall to clean up. >>> >>> ipa.ipapython.install.cli.install_tool(Replica): ERROR CA >>> configuration failed. >>> >>> * >>> **With debug enabled I get: * >>> >>> pa : DEBUG Starting external process >>> ipa : DEBUG args='/usr/sbin/pkispawn' '-s' 'CA' '-f' >>> '/tmp/tmpwY8XjR' >>> ipa : DEBUG Process finished, return code=1 >>> ipa : DEBUG stdout=Log file: >>> /var/log/pki/pki-ca-spawn.20160909044214.log >>> Loading deployment configuration from /tmp/tmpwY8XjR. >>> Installing CA into /var/lib/pki/pki-tomcat. >>> Storing deployment configuration into >>> /etc/sysconfig/pki/tomcat/pki-tomcat/ca/deployment.cfg. >>> >>> Installation failed. >>> >>> >>> ipa : DEBUG >>> stderr=/usr/lib/python2.7/site-packages/urllib3/connectionpool.py:769: >>> InsecureRequestWarning: Unverified HTTPS request is being made. Adding >>> certificate verification is strongly advised. See: >>> https://urllib3.readthedocs.org/en/latest/security.html >>> InsecureRequestWarning) >>> pkispawn : WARNING ....... unable to validate security domain >>> user/password >>> through REST interface. Interface not available >>> pkispawn : ERROR ....... Exception from Java Configuration >>> Servlet: 500 >>> Server Error: Internal Server Error >>> pkispawn : ERROR ....... ParseError: not well-formed (invalid >>> token): line >>> 1, column 0: >>> {"Attributes":{"Attribute":[]},"ClassName":"com.netscape.certsrv.base.PKIException","Code":500,"Message":"Failed >>> >>> to obtain installation token from security domain"} >>> >>> >>> Is there a way to validate the repilca .gpg file from a v3 >>> installation against >>> a v4.2 freeipa installation to check for any errors before going >>> through the >>> ipa-replica-install? >>> The ipa-replica-install completes if I don't include the --setup-ca >>> flag but I >>> don't want that >>> >> There is no automatic method to verify the replica file. >> >> Could you share the stack trace from /var/log/pki/pki-tomcat/ca/debug + >> couple lines before and after? >> >> > > Contents of /var/log/pki/pki-tomcat/ca/debug: > > [09/Sep/2016:08:22:51][http-bio-8443-exec-3]: MessageFormatInterceptor: > SystemConfigResource.configure() > [09/Sep/2016:08:22:51][http-bio-8443-exec-3]: MessageFormatInterceptor: > content-type: application/json > [09/Sep/2016:08:22:51][http-bio-8443-exec-3]: MessageFormatInterceptor: > accept: [application/json] > [09/Sep/2016:08:22:51][http-bio-8443-exec-3]: MessageFormatInterceptor: > request format: application/json > [09/Sep/2016:08:22:51][http-bio-8443-exec-3]: MessageFormatInterceptor: > response format: application/json > [09/Sep/2016:08:22:51][http-bio-8443-exec-3]: SystemConfigService: > configure() > [09/Sep/2016:08:22:51][http-bio-8443-exec-3]: SystemConfigService: > request: ConfigurationRequest [pin=XXXX, token=Internal Key Storage > Token, tokenPassword=XXXX, securityDomainType=existingdomain, > securityDomainUri=https://ipa-server.nelios:443, > securityDomainName=null, securityDomainUser=admin, > securityDomainPassword=XXXX, isClone=true, > cloneUri=https://ipa-server.nelios:443, subsystemName=CA > ipa2-server2.nelios 8443, p12File=/tmp/ca.p12, p12Password=XXXX, > hierarchy=root, dsHost=ipa2-server2.nelios, dsPort=389, baseDN=o=ipaca, > bindDN=cn=Directory Manager, bindpwd=XXXX, database=ipaca, > secureConn=false, removeData=true, replicateSchema=false, > masterReplicationPort=7389, cloneReplicationPort=389, > replicationSecurity=TLS, > systemCerts=[com.netscape.certsrv.system.SystemCertData at 434a841], > issuingCA=https://ipa-server.nelios:443, backupKeys=true, > backupPassword=XXXX, > backupFile=/etc/pki/pki-tomcat/alias/ca_backup_keys.p12, adminUID=null, > adminPassword=XXXX, adminEmail=null, adminCertRequest=null, > adminCertRequestType=null, adminSubjectDN=null, adminName=null, > adminProfileID=null, adminCert=null, importAdminCert=false, > generateServerCert=true, external=false, standAlone=false, > stepTwo=false, authdbBaseDN=null, authdbHost=null, authdbPort=null, > authdbSecureConn=null, caUri=null, kraUri=null, tksUri=null, > enableServerSideKeyGen=null, importSharedSecret=null, > generateSubsystemCert=null, sharedDB=false, sharedDBUserDN=null, > createNewDB=true, setupReplication=True, subordinateSecurityDomainNamenull] > [09/Sep/2016:08:22:51][http-bio-8443-exec-3]: === Token Panel === > [09/Sep/2016:08:22:51][http-bio-8443-exec-3]: === Security Domain Panel === > [09/Sep/2016:08:22:51][http-bio-8443-exec-3]: Joining existing security > domain > [09/Sep/2016:08:22:51][http-bio-8443-exec-3]: Resolving security domain > URLhttps://ipa-server.nelios:443 > [09/Sep/2016:08:22:51][http-bio-8443-exec-3]: Getting security domain > cert chain > [09/Sep/2016:08:22:52][http-bio-8443-exec-3]: Getting install token > [09/Sep/2016:08:22:52][http-bio-8443-exec-3]: Getting install token > [09/Sep/2016:08:22:52][http-bio-8443-exec-3]: Getting old cookie > [09/Sep/2016:08:22:52][http-bio-8443-exec-3]: Token: null > [09/Sep/2016:08:22:52][http-bio-8443-exec-3]: Install token is null > [09/Sep/2016:08:22:52][http-bio-8443-exec-3]: Failed to obtain > installation token from security domain > > I assume it is the null token the perpetrator ? if yes what should I fix > on master? > I don't know this part much. Therefore CCing PKI experts - in addition to figure out if there is anything to fix on IPA or PKI side. Endi, Matthew, do I understand it correctly that for obtaining the token, it contacts master server with pki_security_domain_user == admin pki_security_domain_password == whatever provided in ipa-replica-install pki_security_domain_user matches uid=admin,ou=people,o=ipaca which has a password which was set during ipa-server-install(and thus pkisilent) on original 6.x server. Therefore if admin password changed between these two installations then it will fail obtain the cookie? (guessing that wrong credential might be the reason) -- Petr Vobornik From rcritten at redhat.com Fri Sep 9 13:29:32 2016 From: rcritten at redhat.com (Rob Crittenden) Date: Fri, 9 Sep 2016 09:29:32 -0400 Subject: [Freeipa-users] automated ftp service only accounts and passwords In-Reply-To: <79B7CEE400C91A4C9FD8BF082D8226073E3700C7@JDRPDC.JDRSolutions.local> References: <79B7CEE400C91A4C9FD8BF082D8226073E3700C7@JDRPDC.JDRSolutions.local> Message-ID: <57D2B93C.4090707@redhat.com> Larry Rosen wrote: > How do I set the password on a chroot jailed sftp id account that is not > allowed a shell to not expire its password after setting it? There?s no > way to change it to the fixed password I want. > > I have created a service_account password policy that has no expiration > (set to Max lifetime (days) = 20000 ). More details are needed. Did you create a service account user or are you using an IPA user? You created a new password policy, is the sftp account in that group? Why can't you set the password to what you want? rob From g.kafataridis at nelios.com Fri Sep 9 14:24:53 2016 From: g.kafataridis at nelios.com (Giorgos Kafataridis) Date: Fri, 9 Sep 2016 17:24:53 +0300 Subject: [Freeipa-users] CA: Cannot add Centos7.2 replica to Centos6.8 ipa server In-Reply-To: <9afdf0f3-97d1-cc6e-5f3d-0e11687a124a@redhat.com> References: <45b66871-aa01-3c91-0630-0bbeee488ae7@nelios.com> <323b9de1-adb3-47af-d4a7-d6083f88d75c@nelios.com> <9afdf0f3-97d1-cc6e-5f3d-0e11687a124a@redhat.com> Message-ID: <05b8e986-ceef-8e75-c76d-b38c45e2e3a1@nelios.com> On 09/09/2016 04:09 PM, Petr Vobornik wrote: > On 09/09/2016 02:33 PM, Giorgos Kafataridis wrote: >>>> Yes, I have followed >>>> https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/7/html/Linux_Domain_Identity_Authentication_and_Policy_Guide/upgrading.html >>>> >>>> to the letter. >>>> The only reason I had to recreate the cacert.p12 file is because it >>>> is not >>>> renewed automatically in v3, so the cacert.p12 was outdated and the >>>> CA was >>>> throwing an "p12 invalid digest" error. >>>> >>>> * I opened all necessary ports >>>> * I checked all certs and they are valid for another year >>>> >>>> >>>> /Run connection check to master// >>>> //Check connection from replica to remote master 'ipa-server.nelios':// >>>> // Directory Service: Unsecure port (389): OK// >>>> // Directory Service: Secure port (636): OK// >>>> // Kerberos KDC: TCP (88): OK// >>>> // Kerberos Kpasswd: TCP (464): OK// >>>> // HTTP Server: Unsecure port (80): OK// >>>> // HTTP Server: Secure port (443): OK// >>>> // PKI-CA: Directory Service port (7389): OK// >>>> // >>>> //The following list of ports use UDP protocol and would need to be// >>>> //checked manually:// >>>> // Kerberos KDC: UDP (88): SKIPPED// >>>> // Kerberos Kpasswd: UDP (464): SKIPPED// >>>> // >>>> //Connection from replica to master is OK.// >>>> //Start listening on required ports for remote master check// >>>> //Get credentials to log in to remote master// >>>> //Check SSH connection to remote master// >>>> //Execute check on remote master// >>>> //Check connection from master to remote replica >>>> 'ipa2-server2.nelios':// >>>> // Directory Service: Unsecure port (389): OK// >>>> // Directory Service: Secure port (636): OK// >>>> // Kerberos KDC: TCP (88): OK// >>>> // Kerberos KDC: UDP (88): OK// >>>> // Kerberos Kpasswd: TCP (464): OK// >>>> // Kerberos Kpasswd: UDP (464): OK// >>>> // HTTP Server: Unsecure port (80): OK// >>>> // HTTP Server: Secure port (443): OK// >>>> // >>>> //Connection from master to replica is OK.// >>>> // >>>> //Connection check OK/ >>>> >>>> *Even with a fresh install of centos 7 with different hostname and ip >>>> and I >>>> still get the the error below* >>>> >>>> Configuring certificate server (pki-tomcatd). Estimated time: 3 >>>> minutes 30 seconds >>>> [1/24]: creating certificate server user >>>> [2/24]: configuring certificate server instance >>>> ipa.ipaserver.install.cainstance.CAInstance: CRITICAL Failed to >>>> configure CA >>>> instance: Command ''/usr/sbin/pkispawn' '-s' 'CA' '-f' '/tmp/tmpbMwmp_'' >>>> returned non-zero exit status 1 >>>> ipa.ipaserver.install.cainstance.CAInstance: CRITICAL See the >>>> installation logs >>>> and the following files/directories for more information: >>>> ipa.ipaserver.install.cainstance.CAInstance: CRITICAL >>>> /var/log/pki-ca-install.log >>>> ipa.ipaserver.install.cainstance.CAInstance: CRITICAL >>>> /var/log/pki/pki-tomcat >>>> [error] RuntimeError: CA configuration failed. >>>> Your system may be partly configured. >>>> Run /usr/sbin/ipa-server-install --uninstall to clean up. >>>> >>>> ipa.ipapython.install.cli.install_tool(Replica): ERROR CA >>>> configuration failed. >>>> >>>> * >>>> **With debug enabled I get: * >>>> >>>> pa : DEBUG Starting external process >>>> ipa : DEBUG args='/usr/sbin/pkispawn' '-s' 'CA' '-f' >>>> '/tmp/tmpwY8XjR' >>>> ipa : DEBUG Process finished, return code=1 >>>> ipa : DEBUG stdout=Log file: >>>> /var/log/pki/pki-ca-spawn.20160909044214.log >>>> Loading deployment configuration from /tmp/tmpwY8XjR. >>>> Installing CA into /var/lib/pki/pki-tomcat. >>>> Storing deployment configuration into >>>> /etc/sysconfig/pki/tomcat/pki-tomcat/ca/deployment.cfg. >>>> >>>> Installation failed. >>>> >>>> >>>> ipa : DEBUG >>>> stderr=/usr/lib/python2.7/site-packages/urllib3/connectionpool.py:769: >>>> InsecureRequestWarning: Unverified HTTPS request is being made. Adding >>>> certificate verification is strongly advised. See: >>>> https://urllib3.readthedocs.org/en/latest/security.html >>>> InsecureRequestWarning) >>>> pkispawn : WARNING ....... unable to validate security domain >>>> user/password >>>> through REST interface. Interface not available >>>> pkispawn : ERROR ....... Exception from Java Configuration >>>> Servlet: 500 >>>> Server Error: Internal Server Error >>>> pkispawn : ERROR ....... ParseError: not well-formed (invalid >>>> token): line >>>> 1, column 0: >>>> {"Attributes":{"Attribute":[]},"ClassName":"com.netscape.certsrv.base.PKIException","Code":500,"Message":"Failed >>>> >>>> to obtain installation token from security domain"} >>>> >>>> >>>> Is there a way to validate the repilca .gpg file from a v3 >>>> installation against >>>> a v4.2 freeipa installation to check for any errors before going >>>> through the >>>> ipa-replica-install? >>>> The ipa-replica-install completes if I don't include the --setup-ca >>>> flag but I >>>> don't want that >>>> >>> There is no automatic method to verify the replica file. >>> >>> Could you share the stack trace from /var/log/pki/pki-tomcat/ca/debug + >>> couple lines before and after? >>> >>> >> Contents of /var/log/pki/pki-tomcat/ca/debug: >> >> [09/Sep/2016:08:22:51][http-bio-8443-exec-3]: MessageFormatInterceptor: >> SystemConfigResource.configure() >> [09/Sep/2016:08:22:51][http-bio-8443-exec-3]: MessageFormatInterceptor: >> content-type: application/json >> [09/Sep/2016:08:22:51][http-bio-8443-exec-3]: MessageFormatInterceptor: >> accept: [application/json] >> [09/Sep/2016:08:22:51][http-bio-8443-exec-3]: MessageFormatInterceptor: >> request format: application/json >> [09/Sep/2016:08:22:51][http-bio-8443-exec-3]: MessageFormatInterceptor: >> response format: application/json >> [09/Sep/2016:08:22:51][http-bio-8443-exec-3]: SystemConfigService: >> configure() >> [09/Sep/2016:08:22:51][http-bio-8443-exec-3]: SystemConfigService: >> request: ConfigurationRequest [pin=XXXX, token=Internal Key Storage >> Token, tokenPassword=XXXX, securityDomainType=existingdomain, >> securityDomainUri=https://ipa-server.nelios:443, >> securityDomainName=null, securityDomainUser=admin, >> securityDomainPassword=XXXX, isClone=true, >> cloneUri=https://ipa-server.nelios:443, subsystemName=CA >> ipa2-server2.nelios 8443, p12File=/tmp/ca.p12, p12Password=XXXX, >> hierarchy=root, dsHost=ipa2-server2.nelios, dsPort=389, baseDN=o=ipaca, >> bindDN=cn=Directory Manager, bindpwd=XXXX, database=ipaca, >> secureConn=false, removeData=true, replicateSchema=false, >> masterReplicationPort=7389, cloneReplicationPort=389, >> replicationSecurity=TLS, >> systemCerts=[com.netscape.certsrv.system.SystemCertData at 434a841], >> issuingCA=https://ipa-server.nelios:443, backupKeys=true, >> backupPassword=XXXX, >> backupFile=/etc/pki/pki-tomcat/alias/ca_backup_keys.p12, adminUID=null, >> adminPassword=XXXX, adminEmail=null, adminCertRequest=null, >> adminCertRequestType=null, adminSubjectDN=null, adminName=null, >> adminProfileID=null, adminCert=null, importAdminCert=false, >> generateServerCert=true, external=false, standAlone=false, >> stepTwo=false, authdbBaseDN=null, authdbHost=null, authdbPort=null, >> authdbSecureConn=null, caUri=null, kraUri=null, tksUri=null, >> enableServerSideKeyGen=null, importSharedSecret=null, >> generateSubsystemCert=null, sharedDB=false, sharedDBUserDN=null, >> createNewDB=true, setupReplication=True, subordinateSecurityDomainNamenull] >> [09/Sep/2016:08:22:51][http-bio-8443-exec-3]: === Token Panel === >> [09/Sep/2016:08:22:51][http-bio-8443-exec-3]: === Security Domain Panel === >> [09/Sep/2016:08:22:51][http-bio-8443-exec-3]: Joining existing security >> domain >> [09/Sep/2016:08:22:51][http-bio-8443-exec-3]: Resolving security domain >> URLhttps://ipa-server.nelios:443 >> [09/Sep/2016:08:22:51][http-bio-8443-exec-3]: Getting security domain >> cert chain >> [09/Sep/2016:08:22:52][http-bio-8443-exec-3]: Getting install token >> [09/Sep/2016:08:22:52][http-bio-8443-exec-3]: Getting install token >> [09/Sep/2016:08:22:52][http-bio-8443-exec-3]: Getting old cookie >> [09/Sep/2016:08:22:52][http-bio-8443-exec-3]: Token: null >> [09/Sep/2016:08:22:52][http-bio-8443-exec-3]: Install token is null >> [09/Sep/2016:08:22:52][http-bio-8443-exec-3]: Failed to obtain >> installation token from security domain >> >> I assume it is the null token the perpetrator ? if yes what should I fix >> on master? >> > I don't know this part much. Therefore CCing PKI experts - in addition > to figure out if there is anything to fix on IPA or PKI side. > > Endi, Matthew, > > do I understand it correctly that for obtaining the token, it contacts > master server with > pki_security_domain_user == admin > pki_security_domain_password == whatever provided in ipa-replica-install > > pki_security_domain_user matches uid=admin,ou=people,o=ipaca which has a > password which was set during ipa-server-install(and thus pkisilent) on > original 6.x server. > > Therefore if admin password changed between these two installations then > it will fail obtain the cookie? (guessing that wrong credential might be > the reason) If I look for uid=admin,ou=people,o=ipaca on master (ipa v3, centos 6.x) this is what I get: [root at ipa-server ~]# ldapsearch -D "cn=directory manager" -W -p 389 -h localhost -b "uid=admin,ou=people,o=ipaca,dc=nelios" # extended LDIF # # LDAPv3 # base with scope subtree # filter: (objectclass=*) # requesting: ALL # # search result search: 2 result: 32 No such object matchedDN: dc=nelios # numResponses: 1 LDAP manager password seems to be correct as I used it more than once in the last few days to remove the failing replicas. -------------- next part -------------- An HTML attachment was scrubbed... URL: From pvoborni at redhat.com Fri Sep 9 15:04:11 2016 From: pvoborni at redhat.com (Petr Vobornik) Date: Fri, 9 Sep 2016 17:04:11 +0200 Subject: [Freeipa-users] CA: Cannot add Centos7.2 replica to Centos6.8 ipa server In-Reply-To: <05b8e986-ceef-8e75-c76d-b38c45e2e3a1@nelios.com> References: <45b66871-aa01-3c91-0630-0bbeee488ae7@nelios.com> <323b9de1-adb3-47af-d4a7-d6083f88d75c@nelios.com> <9afdf0f3-97d1-cc6e-5f3d-0e11687a124a@redhat.com> <05b8e986-ceef-8e75-c76d-b38c45e2e3a1@nelios.com> Message-ID: <79903d5c-2b80-d492-6b29-59ac5b39c6f5@redhat.com> On 09/09/2016 04:24 PM, Giorgos Kafataridis wrote: > > > On 09/09/2016 04:09 PM, Petr Vobornik wrote: >> On 09/09/2016 02:33 PM, Giorgos Kafataridis wrote: >>>>> Yes, I have followed >>>>> https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/7/html/Linux_Domain_Identity_Authentication_and_Policy_Guide/upgrading.html >>>>> >>>>> to the letter. >>>>> The only reason I had to recreate the cacert.p12 file is because it >>>>> is not >>>>> renewed automatically in v3, so the cacert.p12 was outdated and the >>>>> CA was >>>>> throwing an "p12 invalid digest" error. >>>>> >>>>> * I opened all necessary ports >>>>> * I checked all certs and they are valid for another year >>>>> >>>>> >>>>> /Run connection check to master// >>>>> //Check connection from replica to remote master 'ipa-server.nelios':// >>>>> // Directory Service: Unsecure port (389): OK// >>>>> // Directory Service: Secure port (636): OK// >>>>> // Kerberos KDC: TCP (88): OK// >>>>> // Kerberos Kpasswd: TCP (464): OK// >>>>> // HTTP Server: Unsecure port (80): OK// >>>>> // HTTP Server: Secure port (443): OK// >>>>> // PKI-CA: Directory Service port (7389): OK// >>>>> // >>>>> //The following list of ports use UDP protocol and would need to be// >>>>> //checked manually:// >>>>> // Kerberos KDC: UDP (88): SKIPPED// >>>>> // Kerberos Kpasswd: UDP (464): SKIPPED// >>>>> // >>>>> //Connection from replica to master is OK.// >>>>> //Start listening on required ports for remote master check// >>>>> //Get credentials to log in to remote master// >>>>> //Check SSH connection to remote master// >>>>> //Execute check on remote master// >>>>> //Check connection from master to remote replica >>>>> 'ipa2-server2.nelios':// >>>>> // Directory Service: Unsecure port (389): OK// >>>>> // Directory Service: Secure port (636): OK// >>>>> // Kerberos KDC: TCP (88): OK// >>>>> // Kerberos KDC: UDP (88): OK// >>>>> // Kerberos Kpasswd: TCP (464): OK// >>>>> // Kerberos Kpasswd: UDP (464): OK// >>>>> // HTTP Server: Unsecure port (80): OK// >>>>> // HTTP Server: Secure port (443): OK// >>>>> // >>>>> //Connection from master to replica is OK.// >>>>> // >>>>> //Connection check OK/ >>>>> >>>>> *Even with a fresh install of centos 7 with different hostname and ip >>>>> and I >>>>> still get the the error below* >>>>> >>>>> Configuring certificate server (pki-tomcatd). Estimated time: 3 >>>>> minutes 30 seconds >>>>> [1/24]: creating certificate server user >>>>> [2/24]: configuring certificate server instance >>>>> ipa.ipaserver.install.cainstance.CAInstance: CRITICAL Failed to >>>>> configure CA >>>>> instance: Command ''/usr/sbin/pkispawn' '-s' 'CA' '-f' '/tmp/tmpbMwmp_'' >>>>> returned non-zero exit status 1 >>>>> ipa.ipaserver.install.cainstance.CAInstance: CRITICAL See the >>>>> installation logs >>>>> and the following files/directories for more information: >>>>> ipa.ipaserver.install.cainstance.CAInstance: CRITICAL >>>>> /var/log/pki-ca-install.log >>>>> ipa.ipaserver.install.cainstance.CAInstance: CRITICAL >>>>> /var/log/pki/pki-tomcat >>>>> [error] RuntimeError: CA configuration failed. >>>>> Your system may be partly configured. >>>>> Run /usr/sbin/ipa-server-install --uninstall to clean up. >>>>> >>>>> ipa.ipapython.install.cli.install_tool(Replica): ERROR CA >>>>> configuration failed. >>>>> >>>>> * >>>>> **With debug enabled I get: * >>>>> >>>>> pa : DEBUG Starting external process >>>>> ipa : DEBUG args='/usr/sbin/pkispawn' '-s' 'CA' '-f' >>>>> '/tmp/tmpwY8XjR' >>>>> ipa : DEBUG Process finished, return code=1 >>>>> ipa : DEBUG stdout=Log file: >>>>> /var/log/pki/pki-ca-spawn.20160909044214.log >>>>> Loading deployment configuration from /tmp/tmpwY8XjR. >>>>> Installing CA into /var/lib/pki/pki-tomcat. >>>>> Storing deployment configuration into >>>>> /etc/sysconfig/pki/tomcat/pki-tomcat/ca/deployment.cfg. >>>>> >>>>> Installation failed. >>>>> >>>>> >>>>> ipa : DEBUG >>>>> stderr=/usr/lib/python2.7/site-packages/urllib3/connectionpool.py:769: >>>>> InsecureRequestWarning: Unverified HTTPS request is being made. Adding >>>>> certificate verification is strongly advised. See: >>>>> https://urllib3.readthedocs.org/en/latest/security.html >>>>> InsecureRequestWarning) >>>>> pkispawn : WARNING ....... unable to validate security domain >>>>> user/password >>>>> through REST interface. Interface not available >>>>> pkispawn : ERROR ....... Exception from Java Configuration >>>>> Servlet: 500 >>>>> Server Error: Internal Server Error >>>>> pkispawn : ERROR ....... ParseError: not well-formed (invalid >>>>> token): line >>>>> 1, column 0: >>>>> {"Attributes":{"Attribute":[]},"ClassName":"com.netscape.certsrv.base.PKIException","Code":500,"Message":"Failed >>>>> >>>>> to obtain installation token from security domain"} >>>>> >>>>> >>>>> Is there a way to validate the repilca .gpg file from a v3 >>>>> installation against >>>>> a v4.2 freeipa installation to check for any errors before going >>>>> through the >>>>> ipa-replica-install? >>>>> The ipa-replica-install completes if I don't include the --setup-ca >>>>> flag but I >>>>> don't want that >>>>> >>>> There is no automatic method to verify the replica file. >>>> >>>> Could you share the stack trace from /var/log/pki/pki-tomcat/ca/debug + >>>> couple lines before and after? >>>> >>>> >>> Contents of /var/log/pki/pki-tomcat/ca/debug: >>> >>> [09/Sep/2016:08:22:51][http-bio-8443-exec-3]: MessageFormatInterceptor: >>> SystemConfigResource.configure() >>> [09/Sep/2016:08:22:51][http-bio-8443-exec-3]: MessageFormatInterceptor: >>> content-type: application/json >>> [09/Sep/2016:08:22:51][http-bio-8443-exec-3]: MessageFormatInterceptor: >>> accept: [application/json] >>> [09/Sep/2016:08:22:51][http-bio-8443-exec-3]: MessageFormatInterceptor: >>> request format: application/json >>> [09/Sep/2016:08:22:51][http-bio-8443-exec-3]: MessageFormatInterceptor: >>> response format: application/json >>> [09/Sep/2016:08:22:51][http-bio-8443-exec-3]: SystemConfigService: >>> configure() >>> [09/Sep/2016:08:22:51][http-bio-8443-exec-3]: SystemConfigService: >>> request: ConfigurationRequest [pin=XXXX, token=Internal Key Storage >>> Token, tokenPassword=XXXX, securityDomainType=existingdomain, >>> securityDomainUri=https://ipa-server.nelios:443, >>> securityDomainName=null, securityDomainUser=admin, >>> securityDomainPassword=XXXX, isClone=true, >>> cloneUri=https://ipa-server.nelios:443, subsystemName=CA >>> ipa2-server2.nelios 8443, p12File=/tmp/ca.p12, p12Password=XXXX, >>> hierarchy=root, dsHost=ipa2-server2.nelios, dsPort=389, baseDN=o=ipaca, >>> bindDN=cn=Directory Manager, bindpwd=XXXX, database=ipaca, >>> secureConn=false, removeData=true, replicateSchema=false, >>> masterReplicationPort=7389, cloneReplicationPort=389, >>> replicationSecurity=TLS, >>> systemCerts=[com.netscape.certsrv.system.SystemCertData at 434a841], >>> issuingCA=https://ipa-server.nelios:443, backupKeys=true, >>> backupPassword=XXXX, >>> backupFile=/etc/pki/pki-tomcat/alias/ca_backup_keys.p12, adminUID=null, >>> adminPassword=XXXX, adminEmail=null, adminCertRequest=null, >>> adminCertRequestType=null, adminSubjectDN=null, adminName=null, >>> adminProfileID=null, adminCert=null, importAdminCert=false, >>> generateServerCert=true, external=false, standAlone=false, >>> stepTwo=false, authdbBaseDN=null, authdbHost=null, authdbPort=null, >>> authdbSecureConn=null, caUri=null, kraUri=null, tksUri=null, >>> enableServerSideKeyGen=null, importSharedSecret=null, >>> generateSubsystemCert=null, sharedDB=false, sharedDBUserDN=null, >>> createNewDB=true, setupReplication=True, subordinateSecurityDomainNamenull] >>> [09/Sep/2016:08:22:51][http-bio-8443-exec-3]: === Token Panel === >>> [09/Sep/2016:08:22:51][http-bio-8443-exec-3]: === Security Domain Panel === >>> [09/Sep/2016:08:22:51][http-bio-8443-exec-3]: Joining existing security >>> domain >>> [09/Sep/2016:08:22:51][http-bio-8443-exec-3]: Resolving security domain >>> URLhttps://ipa-server.nelios:443 >>> [09/Sep/2016:08:22:51][http-bio-8443-exec-3]: Getting security domain >>> cert chain >>> [09/Sep/2016:08:22:52][http-bio-8443-exec-3]: Getting install token >>> [09/Sep/2016:08:22:52][http-bio-8443-exec-3]: Getting install token >>> [09/Sep/2016:08:22:52][http-bio-8443-exec-3]: Getting old cookie >>> [09/Sep/2016:08:22:52][http-bio-8443-exec-3]: Token: null >>> [09/Sep/2016:08:22:52][http-bio-8443-exec-3]: Install token is null >>> [09/Sep/2016:08:22:52][http-bio-8443-exec-3]: Failed to obtain >>> installation token from security domain >>> >>> I assume it is the null token the perpetrator ? if yes what should I fix >>> on master? >>> >> I don't know this part much. Therefore CCing PKI experts - in addition >> to figure out if there is anything to fix on IPA or PKI side. >> >> Endi, Matthew, >> >> do I understand it correctly that for obtaining the token, it contacts >> master server with >> pki_security_domain_user == admin >> pki_security_domain_password == whatever provided in ipa-replica-install >> >> pki_security_domain_user matches uid=admin,ou=people,o=ipaca which has a >> password which was set during ipa-server-install(and thus pkisilent) on >> original 6.x server. >> >> Therefore if admin password changed between these two installations then >> it will fail obtain the cookie? (guessing that wrong credential might be >> the reason) > > > If I look for uid=admin,ou=people,o=ipaca on master (ipa v3, centos 6.x) this > is what I get: > > [root at ipa-server ~]# ldapsearch -D "cn=directory manager" -W -p 389 -h localhost > -b "uid=admin,ou=people,o=ipaca,dc=nelios" > > # extended LDIF > # > # LDAPv3 > # base with scope subtree > # filter: (objectclass=*) > # requesting: ALL > # > > # search result > search: 2 > result: 32 No such object > matchedDN: dc=nelios > > # numResponses: 1 > > LDAP manager password seems to be correct as I used it more than once in the > last few days to remove the failing replicas. > You search for wrong dn: uid=admin,ou=people,o=ipaca,dc=nelios instead of: uid=admin,ou=people,o=ipaca -- Petr Vobornik From edewata at redhat.com Fri Sep 9 15:12:13 2016 From: edewata at redhat.com (Endi Sukma Dewata) Date: Fri, 9 Sep 2016 10:12:13 -0500 Subject: [Freeipa-users] CA: Cannot add Centos7.2 replica to Centos6.8 ipa server In-Reply-To: <9afdf0f3-97d1-cc6e-5f3d-0e11687a124a@redhat.com> References: <45b66871-aa01-3c91-0630-0bbeee488ae7@nelios.com> <323b9de1-adb3-47af-d4a7-d6083f88d75c@nelios.com> <9afdf0f3-97d1-cc6e-5f3d-0e11687a124a@redhat.com> Message-ID: <79782dbf-1051-b4de-88c8-b035b2d0c8bb@redhat.com> On 9/9/2016 8:09 AM, Petr Vobornik wrote: > On 09/09/2016 02:33 PM, Giorgos Kafataridis wrote: >> >>>> Yes, I have followed >>>> https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/7/html/Linux_Domain_Identity_Authentication_and_Policy_Guide/upgrading.html >>>> >>>> to the letter. >>>> The only reason I had to recreate the cacert.p12 file is because it >>>> is not >>>> renewed automatically in v3, so the cacert.p12 was outdated and the >>>> CA was >>>> throwing an "p12 invalid digest" error. >>>> >>>> * I opened all necessary ports >>>> * I checked all certs and they are valid for another year >>>> >>>> >>>> /Run connection check to master// >>>> //Check connection from replica to remote master 'ipa-server.nelios':// >>>> // Directory Service: Unsecure port (389): OK// >>>> // Directory Service: Secure port (636): OK// >>>> // Kerberos KDC: TCP (88): OK// >>>> // Kerberos Kpasswd: TCP (464): OK// >>>> // HTTP Server: Unsecure port (80): OK// >>>> // HTTP Server: Secure port (443): OK// >>>> // PKI-CA: Directory Service port (7389): OK// >>>> // >>>> //The following list of ports use UDP protocol and would need to be// >>>> //checked manually:// >>>> // Kerberos KDC: UDP (88): SKIPPED// >>>> // Kerberos Kpasswd: UDP (464): SKIPPED// >>>> // >>>> //Connection from replica to master is OK.// >>>> //Start listening on required ports for remote master check// >>>> //Get credentials to log in to remote master// >>>> //Check SSH connection to remote master// >>>> //Execute check on remote master// >>>> //Check connection from master to remote replica >>>> 'ipa2-server2.nelios':// >>>> // Directory Service: Unsecure port (389): OK// >>>> // Directory Service: Secure port (636): OK// >>>> // Kerberos KDC: TCP (88): OK// >>>> // Kerberos KDC: UDP (88): OK// >>>> // Kerberos Kpasswd: TCP (464): OK// >>>> // Kerberos Kpasswd: UDP (464): OK// >>>> // HTTP Server: Unsecure port (80): OK// >>>> // HTTP Server: Secure port (443): OK// >>>> // >>>> //Connection from master to replica is OK.// >>>> // >>>> //Connection check OK/ >>>> >>>> *Even with a fresh install of centos 7 with different hostname and ip >>>> and I >>>> still get the the error below* >>>> >>>> Configuring certificate server (pki-tomcatd). Estimated time: 3 >>>> minutes 30 seconds >>>> [1/24]: creating certificate server user >>>> [2/24]: configuring certificate server instance >>>> ipa.ipaserver.install.cainstance.CAInstance: CRITICAL Failed to >>>> configure CA >>>> instance: Command ''/usr/sbin/pkispawn' '-s' 'CA' '-f' '/tmp/tmpbMwmp_'' >>>> returned non-zero exit status 1 >>>> ipa.ipaserver.install.cainstance.CAInstance: CRITICAL See the >>>> installation logs >>>> and the following files/directories for more information: >>>> ipa.ipaserver.install.cainstance.CAInstance: CRITICAL >>>> /var/log/pki-ca-install.log >>>> ipa.ipaserver.install.cainstance.CAInstance: CRITICAL >>>> /var/log/pki/pki-tomcat >>>> [error] RuntimeError: CA configuration failed. >>>> Your system may be partly configured. >>>> Run /usr/sbin/ipa-server-install --uninstall to clean up. >>>> >>>> ipa.ipapython.install.cli.install_tool(Replica): ERROR CA >>>> configuration failed. >>>> >>>> * >>>> **With debug enabled I get: * >>>> >>>> pa : DEBUG Starting external process >>>> ipa : DEBUG args='/usr/sbin/pkispawn' '-s' 'CA' '-f' >>>> '/tmp/tmpwY8XjR' >>>> ipa : DEBUG Process finished, return code=1 >>>> ipa : DEBUG stdout=Log file: >>>> /var/log/pki/pki-ca-spawn.20160909044214.log >>>> Loading deployment configuration from /tmp/tmpwY8XjR. >>>> Installing CA into /var/lib/pki/pki-tomcat. >>>> Storing deployment configuration into >>>> /etc/sysconfig/pki/tomcat/pki-tomcat/ca/deployment.cfg. >>>> >>>> Installation failed. >>>> >>>> >>>> ipa : DEBUG >>>> stderr=/usr/lib/python2.7/site-packages/urllib3/connectionpool.py:769: >>>> InsecureRequestWarning: Unverified HTTPS request is being made. Adding >>>> certificate verification is strongly advised. See: >>>> https://urllib3.readthedocs.org/en/latest/security.html >>>> InsecureRequestWarning) >>>> pkispawn : WARNING ....... unable to validate security domain >>>> user/password >>>> through REST interface. Interface not available >>>> pkispawn : ERROR ....... Exception from Java Configuration >>>> Servlet: 500 >>>> Server Error: Internal Server Error >>>> pkispawn : ERROR ....... ParseError: not well-formed (invalid >>>> token): line >>>> 1, column 0: >>>> {"Attributes":{"Attribute":[]},"ClassName":"com.netscape.certsrv.base.PKIException","Code":500,"Message":"Failed >>>> >>>> to obtain installation token from security domain"} >>>> >>>> >>>> Is there a way to validate the repilca .gpg file from a v3 >>>> installation against >>>> a v4.2 freeipa installation to check for any errors before going >>>> through the >>>> ipa-replica-install? >>>> The ipa-replica-install completes if I don't include the --setup-ca >>>> flag but I >>>> don't want that >>>> >>> There is no automatic method to verify the replica file. >>> >>> Could you share the stack trace from /var/log/pki/pki-tomcat/ca/debug + >>> couple lines before and after? >>> >>> >> >> Contents of /var/log/pki/pki-tomcat/ca/debug: >> >> [09/Sep/2016:08:22:51][http-bio-8443-exec-3]: MessageFormatInterceptor: >> SystemConfigResource.configure() >> [09/Sep/2016:08:22:51][http-bio-8443-exec-3]: MessageFormatInterceptor: >> content-type: application/json >> [09/Sep/2016:08:22:51][http-bio-8443-exec-3]: MessageFormatInterceptor: >> accept: [application/json] >> [09/Sep/2016:08:22:51][http-bio-8443-exec-3]: MessageFormatInterceptor: >> request format: application/json >> [09/Sep/2016:08:22:51][http-bio-8443-exec-3]: MessageFormatInterceptor: >> response format: application/json >> [09/Sep/2016:08:22:51][http-bio-8443-exec-3]: SystemConfigService: >> configure() >> [09/Sep/2016:08:22:51][http-bio-8443-exec-3]: SystemConfigService: >> request: ConfigurationRequest [pin=XXXX, token=Internal Key Storage >> Token, tokenPassword=XXXX, securityDomainType=existingdomain, >> securityDomainUri=https://ipa-server.nelios:443, >> securityDomainName=null, securityDomainUser=admin, >> securityDomainPassword=XXXX, isClone=true, >> cloneUri=https://ipa-server.nelios:443, subsystemName=CA >> ipa2-server2.nelios 8443, p12File=/tmp/ca.p12, p12Password=XXXX, >> hierarchy=root, dsHost=ipa2-server2.nelios, dsPort=389, baseDN=o=ipaca, >> bindDN=cn=Directory Manager, bindpwd=XXXX, database=ipaca, >> secureConn=false, removeData=true, replicateSchema=false, >> masterReplicationPort=7389, cloneReplicationPort=389, >> replicationSecurity=TLS, >> systemCerts=[com.netscape.certsrv.system.SystemCertData at 434a841], >> issuingCA=https://ipa-server.nelios:443, backupKeys=true, >> backupPassword=XXXX, >> backupFile=/etc/pki/pki-tomcat/alias/ca_backup_keys.p12, adminUID=null, >> adminPassword=XXXX, adminEmail=null, adminCertRequest=null, >> adminCertRequestType=null, adminSubjectDN=null, adminName=null, >> adminProfileID=null, adminCert=null, importAdminCert=false, >> generateServerCert=true, external=false, standAlone=false, >> stepTwo=false, authdbBaseDN=null, authdbHost=null, authdbPort=null, >> authdbSecureConn=null, caUri=null, kraUri=null, tksUri=null, >> enableServerSideKeyGen=null, importSharedSecret=null, >> generateSubsystemCert=null, sharedDB=false, sharedDBUserDN=null, >> createNewDB=true, setupReplication=True, subordinateSecurityDomainNamenull] >> [09/Sep/2016:08:22:51][http-bio-8443-exec-3]: === Token Panel === >> [09/Sep/2016:08:22:51][http-bio-8443-exec-3]: === Security Domain Panel === >> [09/Sep/2016:08:22:51][http-bio-8443-exec-3]: Joining existing security >> domain >> [09/Sep/2016:08:22:51][http-bio-8443-exec-3]: Resolving security domain >> URLhttps://ipa-server.nelios:443 >> [09/Sep/2016:08:22:51][http-bio-8443-exec-3]: Getting security domain >> cert chain >> [09/Sep/2016:08:22:52][http-bio-8443-exec-3]: Getting install token >> [09/Sep/2016:08:22:52][http-bio-8443-exec-3]: Getting install token >> [09/Sep/2016:08:22:52][http-bio-8443-exec-3]: Getting old cookie >> [09/Sep/2016:08:22:52][http-bio-8443-exec-3]: Token: null >> [09/Sep/2016:08:22:52][http-bio-8443-exec-3]: Install token is null >> [09/Sep/2016:08:22:52][http-bio-8443-exec-3]: Failed to obtain >> installation token from security domain >> >> I assume it is the null token the perpetrator ? if yes what should I fix >> on master? >> > > I don't know this part much. Therefore CCing PKI experts - in addition > to figure out if there is anything to fix on IPA or PKI side. > > Endi, Matthew, > > do I understand it correctly that for obtaining the token, it contacts > master server with > pki_security_domain_user == admin > pki_security_domain_password == whatever provided in ipa-replica-install > > pki_security_domain_user matches uid=admin,ou=people,o=ipaca which has a > password which was set during ipa-server-install(and thus pkisilent) on > original 6.x server. > > Therefore if admin password changed between these two installations then > it will fail obtain the cookie? (guessing that wrong credential might be > the reason) > Could you post the CA debug log, access log, and catalina/Tomcat log from the master? CC'ing Ade as well. -- Endi S. Dewata From peljasz at yahoo.co.uk Fri Sep 9 16:30:17 2016 From: peljasz at yahoo.co.uk (lejeczek) Date: Fri, 9 Sep 2016 17:30:17 +0100 Subject: [Freeipa-users] ERROR CA configuration failed. - again Message-ID: hi everybody, looking at ipareplica-install.log: raise RuntimeError("%s configuration failed." % self.subsystem) RuntimeError: CA configuration failed. 2016-09-09T16:23:17Z DEBUG [error] RuntimeError: CA configuration failed. 2016-09-09T16:23:17Z DEBUG File "/usr/lib/python2.7/site-packages/ipapython/admintool.py", line 171, in execute then at /var/log/pki/pki-tomcat/ca/system 0.localhost-startStop-1 - [09/Sep/2016:16:04:22 BST] [3] [3] Cannot build CA chain. Error java.security.cert.CertificateException: Certificate is not a PKCS #11 certificate 0.localhost-startStop-1 - [09/Sep/2016:16:04:22 BST] [13] [3] authz instance DirAclAuthz initialization failed and skipped, error=Property internaldb.ldapconn.port missing value I cannot find anything more telling in the logs. Does it have anything to do with what's in: /etc/httpd/alias/ ? I yum removed `rpm -qa ipa* 389*` pki-base krb5-pkinit krb5-server krb5-workstation pki-tomcat certmonger rm dirs + reinstalled, yet I cannot find the the root cause of this mess. best regards L. From rcritten at redhat.com Fri Sep 9 17:49:05 2016 From: rcritten at redhat.com (Rob Crittenden) Date: Fri, 9 Sep 2016 13:49:05 -0400 Subject: [Freeipa-users] ERROR CA configuration failed. - again In-Reply-To: References: Message-ID: <57D2F611.6060700@redhat.com> lejeczek wrote: > hi everybody, > > looking at ipareplica-install.log: > > raise RuntimeError("%s configuration failed." % self.subsystem) > RuntimeError: CA configuration failed. > > 2016-09-09T16:23:17Z DEBUG [error] RuntimeError: CA configuration failed. > 2016-09-09T16:23:17Z DEBUG File > "/usr/lib/python2.7/site-packages/ipapython/admintool.py", line 171, in > execute > > then at /var/log/pki/pki-tomcat/ca/system I'd suggest looking at the debug log for more details. > 0.localhost-startStop-1 - [09/Sep/2016:16:04:22 BST] [3] [3] Cannot > build CA chain. Error java.security.cert.CertificateException: > Certificate is not a PKCS #11 certificate > 0.localhost-startStop-1 - [09/Sep/2016:16:04:22 BST] [13] [3] authz > instance DirAclAuthz initialization failed and skipped, error=Property > internaldb.ldapconn.port missing value > > I cannot find anything more telling in the logs. Does it have anything > to do with what's in: > /etc/httpd/alias/ > ? No. > I yum removed > `rpm -qa ipa* 389*` pki-base krb5-pkinit krb5-server krb5-workstation > pki-tomcat certmonger > rm dirs + reinstalled, yet I cannot find the the root cause of this mess. I seriously doubt the problem is local to the box. rob From larry.rosen at JDRSolutions.com Fri Sep 9 16:06:50 2016 From: larry.rosen at JDRSolutions.com (Larry Rosen) Date: Fri, 9 Sep 2016 16:06:50 +0000 Subject: [Freeipa-users] automated ftp service only accounts and passwords In-Reply-To: <57D2B93C.4090707@redhat.com> References: <79B7CEE400C91A4C9FD8BF082D8226073E3700C7@JDRPDC.JDRSolutions.local> <57D2B93C.4090707@redhat.com> Message-ID: <79B7CEE400C91A4C9FD8BF082D8226073E370DC0@JDRPDC.JDRSolutions.local> Why does it (secure log) say: Sep 9 12:04:57 lamp-stor-01 sshd[27950]: pam_sss(sshd:auth): received for user xfseuftp: 13 (User account has expired) User info: [sysadmin at redmine ~]$ ipa pwpolicy-show service_accts Group: service_accts Max lifetime (days): 20000 Min lifetime (hours): 0 History size: 0 Character classes: 2 Min length: 8 Priority: 5 Max failures: 0 Failure reset interval: 0 Lockout duration: 0 [sysadmin at redmine ~]$ date Fri Sep 9 11:35:31 EDT 2016 [sysadmin at redmine ~]$ ipa user-show xfseuftp User login: xfseuftp First name: xfs Last name: eur Home directory: /export/xfseur Login shell: /bin/bash Email address: xfseuftp at ipajdr.local UID: 1333300618 GID: 1333200036 Account disabled: False Password: True Member of groups: service_accts, xfseuftp, uat_info_old, ipausers, info Member of HBAC rule: access_lamp_stor_01_server Kerberos keys available: True [sysadmin at redmine ~]$ ipa hbactest --user=xfseuftp --host=lamp-stor-01.ipajdr.local --service sshd -------------------- Access granted: True -------------------- Matched rules: access_lamp_stor_01_server <------- this is the sftp server attempting to access Not matched rules: access_all_servers Not matched rules: access_il09_app_mufg_server Not matched rules: access_ipa_servers Not matched rules: access_lampuat_server Not matched rules: access_ssh_gate_01_server Not matched rules: access_uat_xfs_il10_server Not matched rules: access_xfs_il10_server Not matched rules: dsiroot_access Not matched rules: il10web_access_xfs_il10_server Not matched rules: xfsroot_access ssh/sftp setup: Match User xfseuftp # Force the connection to use the built-in SFTP support. ForceCommand internal-sftp -u 6 # Chroot the connection into the specified directory. ChrootDirectory /export/xfseur # Disable authentication agent forwarding. AllowAgentForwarding no # Disable TCP connection forwarding. AllowTcpForwarding no # Disable X11 remote desktop forwarding. X11Forwarding no When I attempt to change the account's password (I am sure it's the password I set). I've even tried deleting & re-creating the ID from scratch: [sysadmin at redmine ~]$ ipa passwd xfseuftp New Password: Enter New Password again to verify: -------------------------------------------- Changed password for "xfseuftp at IPAJDR.LOCAL" -------------------------------------------- [sysadmin at redmine ~]$ ssh xfseuftp at 10.120.97.149 xfseuftp at 10.120.97.149's password: Permission denied, please try again. xfseuftp at 10.120.97.149's password: Even if I su to the user [root at lamp-stor-01 export]# ipa passwd xfseuftp New Password: Enter New Password again to verify: -------------------------------------------- Changed password for "xfseuftp at IPAJDR.LOCAL" -------------------------------------------- [root at lamp-stor-01 export]# su - xfseuftp Last login: Fri Sep 9 11:57:24 EDT 2016 on pts/1 -bash-4.2$ passwd Changing password for user xfseuftp. Current Password: Password change failed. Server message: Old password not accepted. passwd: Authentication token manipulation error secure log entries when attempted to change password: Sep 9 11:33:15 lamp-stor-01 sshd[26880]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=10.10.90.138 user=xfseuftp Sep 9 11:33:15 lamp-stor-01 sshd[26880]: pam_sss(sshd:auth): User info message: Permission denied. Sep 9 11:33:15 lamp-stor-01 sshd[26880]: pam_sss(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=10.10.90.138 user=xfseuftp Sep 9 11:33:15 lamp-stor-01 sshd[26880]: pam_sss(sshd:auth): received for user xfseuftp: 13 (User account has expired) Sep 9 11:33:16 lamp-stor-01 sshd[26880]: Failed password for xfseuftp from 10.10.90.138 port 33534 ssh2 ..... Sep 9 11:57:56 lamp-stor-01 su: pam_unix(su-l:session): session closed for user xfseuftp Sep 9 11:58:15 lamp-stor-01 su: pam_unix(su-l:session): session opened for user xfseuftp by root(uid=0) Sep 9 11:58:20 lamp-stor-01 passwd: pam_unix(passwd:chauthtok): user "xfseuftp" does not exist in /etc/passwd Sep 9 11:58:23 lamp-stor-01 passwd: pam_sss(passwd:chauthtok): User info message: Password change failed. Server message: Old password not accepted. Sep 9 11:58:23 lamp-stor-01 passwd: pam_sss(passwd:chauthtok): Authentication failed for user xfseuftp: 4 (System error) Sep 9 11:58:27 lamp-stor-01 su: pam_unix(su-l:session): session closed for user xfseuftp -----Original Message----- From: Rob Crittenden [mailto:rcritten at redhat.com] Sent: Friday, September 09, 2016 9:30 AM To: Larry Rosen ; freeipa-users at redhat.com Subject: Re: [Freeipa-users] automated ftp service only accounts and passwords Larry Rosen wrote: > How do I set the password on a chroot jailed sftp id account that is > not allowed a shell to not expire its password after setting it? > There's no way to change it to the fixed password I want. > > I have created a service_account password policy that has no > expiration (set to Max lifetime (days) = 20000 ). More details are needed. Did you create a service account user or are you using an IPA user? You created a new password policy, is the sftp account in that group? Why can't you set the password to what you want? rob From rcritten at redhat.com Fri Sep 9 17:55:59 2016 From: rcritten at redhat.com (Rob Crittenden) Date: Fri, 9 Sep 2016 13:55:59 -0400 Subject: [Freeipa-users] automated ftp service only accounts and passwords In-Reply-To: <79B7CEE400C91A4C9FD8BF082D8226073E370DC0@JDRPDC.JDRSolutions.local> References: <79B7CEE400C91A4C9FD8BF082D8226073E3700C7@JDRPDC.JDRSolutions.local> <57D2B93C.4090707@redhat.com> <79B7CEE400C91A4C9FD8BF082D8226073E370DC0@JDRPDC.JDRSolutions.local> Message-ID: <57D2F7AF.7090805@redhat.com> Larry Rosen wrote: > Why does it (secure log) say: > Sep 9 12:04:57 lamp-stor-01 sshd[27950]: pam_sss(sshd:auth): received for user xfseuftp: 13 (User account has expired) Administratively set passwords are treated as expired so only the end-user knows the password. http://www.freeipa.org/page/New_Passwords_Expired The _account_ expired is a bit surprising but it may mean the same thing. You could confirm by add --all to the user-show and see if there is a principal expiration date but I'd find that to be quite unusual. > > User info: > > [sysadmin at redmine ~]$ ipa pwpolicy-show service_accts > Group: service_accts > Max lifetime (days): 20000 > Min lifetime (hours): 0 > History size: 0 > Character classes: 2 > Min length: 8 > Priority: 5 > Max failures: 0 > Failure reset interval: 0 > Lockout duration: 0 > > [sysadmin at redmine ~]$ date > Fri Sep 9 11:35:31 EDT 2016 > [sysadmin at redmine ~]$ ipa user-show xfseuftp > User login: xfseuftp > First name: xfs > Last name: eur > Home directory: /export/xfseur > Login shell: /bin/bash > Email address: xfseuftp at ipajdr.local > UID: 1333300618 > GID: 1333200036 > Account disabled: False > Password: True > Member of groups: service_accts, xfseuftp, uat_info_old, ipausers, info > Member of HBAC rule: access_lamp_stor_01_server > Kerberos keys available: True > > [sysadmin at redmine ~]$ ipa hbactest --user=xfseuftp --host=lamp-stor-01.ipajdr.local --service sshd > -------------------- > Access granted: True > -------------------- > Matched rules: access_lamp_stor_01_server <------- this is the sftp server attempting to access > Not matched rules: access_all_servers > Not matched rules: access_il09_app_mufg_server > Not matched rules: access_ipa_servers > Not matched rules: access_lampuat_server > Not matched rules: access_ssh_gate_01_server > Not matched rules: access_uat_xfs_il10_server > Not matched rules: access_xfs_il10_server > Not matched rules: dsiroot_access > Not matched rules: il10web_access_xfs_il10_server > Not matched rules: xfsroot_access > ssh/sftp setup: > > Match User xfseuftp > # Force the connection to use the built-in SFTP support. > ForceCommand internal-sftp -u 6 > # Chroot the connection into the specified directory. > ChrootDirectory /export/xfseur > # Disable authentication agent forwarding. > AllowAgentForwarding no > # Disable TCP connection forwarding. > AllowTcpForwarding no > # Disable X11 remote desktop forwarding. > X11Forwarding no > > When I attempt to change the account's password (I am sure it's the password I set). I've even tried deleting & re-creating the ID from scratch: > > [sysadmin at redmine ~]$ ipa passwd xfseuftp > New Password: > Enter New Password again to verify: > -------------------------------------------- > Changed password for "xfseuftp at IPAJDR.LOCAL" > -------------------------------------------- > > [sysadmin at redmine ~]$ ssh xfseuftp at 10.120.97.149 > xfseuftp at 10.120.97.149's password: > Permission denied, please try again. > xfseuftp at 10.120.97.149's password: > > > Even if I su to the user > > [root at lamp-stor-01 export]# ipa passwd xfseuftp > New Password: > Enter New Password again to verify: > -------------------------------------------- > Changed password for "xfseuftp at IPAJDR.LOCAL" > -------------------------------------------- It depends on what ticket you have, not the user executing the command. > [root at lamp-stor-01 export]# su - xfseuftp > Last login: Fri Sep 9 11:57:24 EDT 2016 on pts/1 > -bash-4.2$ passwd > Changing password for user xfseuftp. > Current Password: > Password change failed. Server message: Old password not accepted. > passwd: Authentication token manipulation error > > > > secure log entries when attempted to change password: > > Sep 9 11:33:15 lamp-stor-01 sshd[26880]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=10.10.90.138 user=xfseuftp > Sep 9 11:33:15 lamp-stor-01 sshd[26880]: pam_sss(sshd:auth): User info message: Permission denied. > Sep 9 11:33:15 lamp-stor-01 sshd[26880]: pam_sss(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=10.10.90.138 user=xfseuftp > Sep 9 11:33:15 lamp-stor-01 sshd[26880]: pam_sss(sshd:auth): received for user xfseuftp: 13 (User account has expired) > Sep 9 11:33:16 lamp-stor-01 sshd[26880]: Failed password for xfseuftp from 10.10.90.138 port 33534 ssh2 > ..... > Sep 9 11:57:56 lamp-stor-01 su: pam_unix(su-l:session): session closed for user xfseuftp > Sep 9 11:58:15 lamp-stor-01 su: pam_unix(su-l:session): session opened for user xfseuftp by root(uid=0) > Sep 9 11:58:20 lamp-stor-01 passwd: pam_unix(passwd:chauthtok): user "xfseuftp" does not exist in /etc/passwd > Sep 9 11:58:23 lamp-stor-01 passwd: pam_sss(passwd:chauthtok): User info message: Password change failed. Server message: Old password not accepted. > Sep 9 11:58:23 lamp-stor-01 passwd: pam_sss(passwd:chauthtok): Authentication failed for user xfseuftp: 4 (System error) > Sep 9 11:58:27 lamp-stor-01 su: pam_unix(su-l:session): session closed for user xfseuftp > > -----Original Message----- > From: Rob Crittenden [mailto:rcritten at redhat.com] > Sent: Friday, September 09, 2016 9:30 AM > To: Larry Rosen ; freeipa-users at redhat.com > Subject: Re: [Freeipa-users] automated ftp service only accounts and passwords > > Larry Rosen wrote: >> How do I set the password on a chroot jailed sftp id account that is >> not allowed a shell to not expire its password after setting it? >> There's no way to change it to the fixed password I want. >> >> I have created a service_account password policy that has no >> expiration (set to Max lifetime (days) = 20000 ). > > More details are needed. Did you create a service account user or are you using an IPA user? > > You created a new password policy, is the sftp account in that group? > > Why can't you set the password to what you want? > > rob > From g.kafataridis at nelios.com Fri Sep 9 19:46:57 2016 From: g.kafataridis at nelios.com (Georgios Kafataridis) Date: Fri, 9 Sep 2016 22:46:57 +0300 Subject: [Freeipa-users] CA: Cannot add Centos7.2 replica to Centos6.8 ipa server In-Reply-To: <79903d5c-2b80-d492-6b29-59ac5b39c6f5@redhat.com> References: <45b66871-aa01-3c91-0630-0bbeee488ae7@nelios.com> <323b9de1-adb3-47af-d4a7-d6083f88d75c@nelios.com> <9afdf0f3-97d1-cc6e-5f3d-0e11687a124a@redhat.com> <05b8e986-ceef-8e75-c76d-b38c45e2e3a1@nelios.com> <79903d5c-2b80-d492-6b29-59ac5b39c6f5@redhat.com> Message-ID: I've tried that but still the same result. [root at ipa-server /]# ldapsearch -D "cn=directory manager" -W -p 389 -h localhost -b "uid=admin,ou=people,o=ipaca" Enter LDAP Password: # extended LDIF # # LDAPv3 # base with scope subtree # filter: (objectclass=*) # requesting: ALL # # search result search: 2 result: 32 No such object On Fri, Sep 9, 2016 at 6:04 PM, Petr Vobornik wrote: > On 09/09/2016 04:24 PM, Giorgos Kafataridis wrote: > > > > > > On 09/09/2016 04:09 PM, Petr Vobornik wrote: > >> On 09/09/2016 02:33 PM, Giorgos Kafataridis wrote: > >>>>> Yes, I have followed > >>>>> https://access.redhat.com/documentation/en-US/Red_Hat_ > Enterprise_Linux/7/html/Linux_Domain_Identity_Authentication_and_Policy_ > Guide/upgrading.html > >>>>> > >>>>> to the letter. > >>>>> The only reason I had to recreate the cacert.p12 file is because it > >>>>> is not > >>>>> renewed automatically in v3, so the cacert.p12 was outdated and the > >>>>> CA was > >>>>> throwing an "p12 invalid digest" error. > >>>>> > >>>>> * I opened all necessary ports > >>>>> * I checked all certs and they are valid for another year > >>>>> > >>>>> > >>>>> /Run connection check to master// > >>>>> //Check connection from replica to remote master > 'ipa-server.nelios':// > >>>>> // Directory Service: Unsecure port (389): OK// > >>>>> // Directory Service: Secure port (636): OK// > >>>>> // Kerberos KDC: TCP (88): OK// > >>>>> // Kerberos Kpasswd: TCP (464): OK// > >>>>> // HTTP Server: Unsecure port (80): OK// > >>>>> // HTTP Server: Secure port (443): OK// > >>>>> // PKI-CA: Directory Service port (7389): OK// > >>>>> // > >>>>> //The following list of ports use UDP protocol and would need to be// > >>>>> //checked manually:// > >>>>> // Kerberos KDC: UDP (88): SKIPPED// > >>>>> // Kerberos Kpasswd: UDP (464): SKIPPED// > >>>>> // > >>>>> //Connection from replica to master is OK.// > >>>>> //Start listening on required ports for remote master check// > >>>>> //Get credentials to log in to remote master// > >>>>> //Check SSH connection to remote master// > >>>>> //Execute check on remote master// > >>>>> //Check connection from master to remote replica > >>>>> 'ipa2-server2.nelios':// > >>>>> // Directory Service: Unsecure port (389): OK// > >>>>> // Directory Service: Secure port (636): OK// > >>>>> // Kerberos KDC: TCP (88): OK// > >>>>> // Kerberos KDC: UDP (88): OK// > >>>>> // Kerberos Kpasswd: TCP (464): OK// > >>>>> // Kerberos Kpasswd: UDP (464): OK// > >>>>> // HTTP Server: Unsecure port (80): OK// > >>>>> // HTTP Server: Secure port (443): OK// > >>>>> // > >>>>> //Connection from master to replica is OK.// > >>>>> // > >>>>> //Connection check OK/ > >>>>> > >>>>> *Even with a fresh install of centos 7 with different hostname and ip > >>>>> and I > >>>>> still get the the error below* > >>>>> > >>>>> Configuring certificate server (pki-tomcatd). Estimated time: 3 > >>>>> minutes 30 seconds > >>>>> [1/24]: creating certificate server user > >>>>> [2/24]: configuring certificate server instance > >>>>> ipa.ipaserver.install.cainstance.CAInstance: CRITICAL Failed to > >>>>> configure CA > >>>>> instance: Command ''/usr/sbin/pkispawn' '-s' 'CA' '-f' > '/tmp/tmpbMwmp_'' > >>>>> returned non-zero exit status 1 > >>>>> ipa.ipaserver.install.cainstance.CAInstance: CRITICAL See the > >>>>> installation logs > >>>>> and the following files/directories for more information: > >>>>> ipa.ipaserver.install.cainstance.CAInstance: CRITICAL > >>>>> /var/log/pki-ca-install.log > >>>>> ipa.ipaserver.install.cainstance.CAInstance: CRITICAL > >>>>> /var/log/pki/pki-tomcat > >>>>> [error] RuntimeError: CA configuration failed. > >>>>> Your system may be partly configured. > >>>>> Run /usr/sbin/ipa-server-install --uninstall to clean up. > >>>>> > >>>>> ipa.ipapython.install.cli.install_tool(Replica): ERROR CA > >>>>> configuration failed. > >>>>> > >>>>> * > >>>>> **With debug enabled I get: * > >>>>> > >>>>> pa : DEBUG Starting external process > >>>>> ipa : DEBUG args='/usr/sbin/pkispawn' '-s' 'CA' '-f' > >>>>> '/tmp/tmpwY8XjR' > >>>>> ipa : DEBUG Process finished, return code=1 > >>>>> ipa : DEBUG stdout=Log file: > >>>>> /var/log/pki/pki-ca-spawn.20160909044214.log > >>>>> Loading deployment configuration from /tmp/tmpwY8XjR. > >>>>> Installing CA into /var/lib/pki/pki-tomcat. > >>>>> Storing deployment configuration into > >>>>> /etc/sysconfig/pki/tomcat/pki-tomcat/ca/deployment.cfg. > >>>>> > >>>>> Installation failed. > >>>>> > >>>>> > >>>>> ipa : DEBUG > >>>>> stderr=/usr/lib/python2.7/site-packages/urllib3/ > connectionpool.py:769: > >>>>> InsecureRequestWarning: Unverified HTTPS request is being made. > Adding > >>>>> certificate verification is strongly advised. See: > >>>>> https://urllib3.readthedocs.org/en/latest/security.html > >>>>> InsecureRequestWarning) > >>>>> pkispawn : WARNING ....... unable to validate security domain > >>>>> user/password > >>>>> through REST interface. Interface not available > >>>>> pkispawn : ERROR ....... Exception from Java Configuration > >>>>> Servlet: 500 > >>>>> Server Error: Internal Server Error > >>>>> pkispawn : ERROR ....... ParseError: not well-formed (invalid > >>>>> token): line > >>>>> 1, column 0: > >>>>> {"Attributes":{"Attribute":[]},"ClassName":"com.netscape. > certsrv.base.PKIException","Code":500,"Message":"Failed > >>>>> > >>>>> to obtain installation token from security domain"} > >>>>> > >>>>> > >>>>> Is there a way to validate the repilca .gpg file from a v3 > >>>>> installation against > >>>>> a v4.2 freeipa installation to check for any errors before going > >>>>> through the > >>>>> ipa-replica-install? > >>>>> The ipa-replica-install completes if I don't include the --setup-ca > >>>>> flag but I > >>>>> don't want that > >>>>> > >>>> There is no automatic method to verify the replica file. > >>>> > >>>> Could you share the stack trace from /var/log/pki/pki-tomcat/ca/debug > + > >>>> couple lines before and after? > >>>> > >>>> > >>> Contents of /var/log/pki/pki-tomcat/ca/debug: > >>> > >>> [09/Sep/2016:08:22:51][http-bio-8443-exec-3]: > MessageFormatInterceptor: > >>> SystemConfigResource.configure() > >>> [09/Sep/2016:08:22:51][http-bio-8443-exec-3]: > MessageFormatInterceptor: > >>> content-type: application/json > >>> [09/Sep/2016:08:22:51][http-bio-8443-exec-3]: > MessageFormatInterceptor: > >>> accept: [application/json] > >>> [09/Sep/2016:08:22:51][http-bio-8443-exec-3]: > MessageFormatInterceptor: > >>> request format: application/json > >>> [09/Sep/2016:08:22:51][http-bio-8443-exec-3]: > MessageFormatInterceptor: > >>> response format: application/json > >>> [09/Sep/2016:08:22:51][http-bio-8443-exec-3]: SystemConfigService: > >>> configure() > >>> [09/Sep/2016:08:22:51][http-bio-8443-exec-3]: SystemConfigService: > >>> request: ConfigurationRequest [pin=XXXX, token=Internal Key Storage > >>> Token, tokenPassword=XXXX, securityDomainType=existingdomain, > >>> securityDomainUri=https://ipa-server.nelios:443, > >>> securityDomainName=null, securityDomainUser=admin, > >>> securityDomainPassword=XXXX, isClone=true, > >>> cloneUri=https://ipa-server.nelios:443, subsystemName=CA > >>> ipa2-server2.nelios 8443, p12File=/tmp/ca.p12, p12Password=XXXX, > >>> hierarchy=root, dsHost=ipa2-server2.nelios, dsPort=389, baseDN=o=ipaca, > >>> bindDN=cn=Directory Manager, bindpwd=XXXX, database=ipaca, > >>> secureConn=false, removeData=true, replicateSchema=false, > >>> masterReplicationPort=7389, cloneReplicationPort=389, > >>> replicationSecurity=TLS, > >>> systemCerts=[com.netscape.certsrv.system.SystemCertData at 434a841], > >>> issuingCA=https://ipa-server.nelios:443, backupKeys=true, > >>> backupPassword=XXXX, > >>> backupFile=/etc/pki/pki-tomcat/alias/ca_backup_keys.p12, > adminUID=null, > >>> adminPassword=XXXX, adminEmail=null, adminCertRequest=null, > >>> adminCertRequestType=null, adminSubjectDN=null, adminName=null, > >>> adminProfileID=null, adminCert=null, importAdminCert=false, > >>> generateServerCert=true, external=false, standAlone=false, > >>> stepTwo=false, authdbBaseDN=null, authdbHost=null, authdbPort=null, > >>> authdbSecureConn=null, caUri=null, kraUri=null, tksUri=null, > >>> enableServerSideKeyGen=null, importSharedSecret=null, > >>> generateSubsystemCert=null, sharedDB=false, sharedDBUserDN=null, > >>> createNewDB=true, setupReplication=True, subordinateSecurityDomainNamen > ull] > >>> [09/Sep/2016:08:22:51][http-bio-8443-exec-3]: === Token Panel === > >>> [09/Sep/2016:08:22:51][http-bio-8443-exec-3]: === Security Domain > Panel === > >>> [09/Sep/2016:08:22:51][http-bio-8443-exec-3]: Joining existing > security > >>> domain > >>> [09/Sep/2016:08:22:51][http-bio-8443-exec-3]: Resolving security > domain > >>> URLhttps://ipa-server.nelios:443 > >>> [09/Sep/2016:08:22:51][http-bio-8443-exec-3]: Getting security domain > >>> cert chain > >>> [09/Sep/2016:08:22:52][http-bio-8443-exec-3]: Getting install token > >>> [09/Sep/2016:08:22:52][http-bio-8443-exec-3]: Getting install token > >>> [09/Sep/2016:08:22:52][http-bio-8443-exec-3]: Getting old cookie > >>> [09/Sep/2016:08:22:52][http-bio-8443-exec-3]: Token: null > >>> [09/Sep/2016:08:22:52][http-bio-8443-exec-3]: Install token is null > >>> [09/Sep/2016:08:22:52][http-bio-8443-exec-3]: Failed to obtain > >>> installation token from security domain > >>> > >>> I assume it is the null token the perpetrator ? if yes what should I > fix > >>> on master? > >>> > >> I don't know this part much. Therefore CCing PKI experts - in addition > >> to figure out if there is anything to fix on IPA or PKI side. > >> > >> Endi, Matthew, > >> > >> do I understand it correctly that for obtaining the token, it contacts > >> master server with > >> pki_security_domain_user == admin > >> pki_security_domain_password == whatever provided in > ipa-replica-install > >> > >> pki_security_domain_user matches uid=admin,ou=people,o=ipaca which has a > >> password which was set during ipa-server-install(and thus pkisilent) on > >> original 6.x server. > >> > >> Therefore if admin password changed between these two installations then > >> it will fail obtain the cookie? (guessing that wrong credential might be > >> the reason) > > > > > > If I look for uid=admin,ou=people,o=ipaca on master (ipa v3, centos > 6.x) this > > is what I get: > > > > [root at ipa-server ~]# ldapsearch -D "cn=directory manager" -W -p 389 -h > localhost > > -b "uid=admin,ou=people,o=ipaca,dc=nelios" > > > > # extended LDIF > > # > > # LDAPv3 > > # base with scope subtree > > # filter: (objectclass=*) > > # requesting: ALL > > # > > > > # search result > > search: 2 > > result: 32 No such object > > matchedDN: dc=nelios > > > > # numResponses: 1 > > > > LDAP manager password seems to be correct as I used it more than once in > the > > last few days to remove the failing replicas. > > > > You search for wrong dn: > uid=admin,ou=people,o=ipaca,dc=nelios > instead of: > uid=admin,ou=people,o=ipaca > > -- > Petr Vobornik > -------------- next part -------------- An HTML attachment was scrubbed... URL: From g.kafataridis at nelios.com Fri Sep 9 20:26:22 2016 From: g.kafataridis at nelios.com (Georgios Kafataridis) Date: Fri, 9 Sep 2016 23:26:22 +0300 Subject: [Freeipa-users] CA: Cannot add Centos7.2 replica to Centos6.8 ipa server In-Reply-To: <79782dbf-1051-b4de-88c8-b035b2d0c8bb@redhat.com> References: <45b66871-aa01-3c91-0630-0bbeee488ae7@nelios.com> <323b9de1-adb3-47af-d4a7-d6083f88d75c@nelios.com> <9afdf0f3-97d1-cc6e-5f3d-0e11687a124a@redhat.com> <79782dbf-1051-b4de-88c8-b035b2d0c8bb@redhat.com> Message-ID: These are fresh logs from a last attempt to create a replica Centos 7 /var/log/pki/pki-tomcat/ca/debug [09/Sep/2016:22:59:40][http-bio-8443-exec-3]: === Token Panel === [09/Sep/2016:22:59:40][http-bio-8443-exec-3]: === Security Domain Panel === [09/Sep/2016:22:59:40][http-bio-8443-exec-3]: Joining existing security domain [09/Sep/2016:22:59:40][http-bio-8443-exec-3]: Resolving security domain URLhttps://ipa-server.nelios:443 [09/Sep/2016:22:59:40][http-bio-8443-exec-3]: Getting security domain cert chain [09/Sep/2016:22:59:40][http-bio-8443-exec-3]: Getting install token [09/Sep/2016:22:59:40][http-bio-8443-exec-3]: Getting install token [09/Sep/2016:22:59:41][http-bio-8443-exec-3]: Getting old cookie [09/Sep/2016:22:59:41][http-bio-8443-exec-3]: Token: null [09/Sep/2016:22:59:41][http-bio-8443-exec-3]: Install token is null [09/Sep/2016:22:59:41][http-bio-8443-exec-3]: Failed to obtain installation token from security domain Centos 6 /var/log/pki-ca/debug [09/Sep/2016:22:59:42][TP-Processor3]: GetCookie before auth, url = https://ipa2-server2.nelios:443/ca/admin/console/config/wizard?p=5&subsystem=CA [09/Sep/2016:22:59:42][TP-Processor3]: IP: 192.168.4.175 [09/Sep/2016:22:59:42][TP-Processor3]: AuthMgrName: passwdUserDBAuthMgr [09/Sep/2016:22:59:42][TP-Processor3]: CMSServlet: no client certificate found [09/Sep/2016:22:59:42][TP-Processor3]: Authentication: UID=admin [09/Sep/2016:22:59:42][TP-Processor3]: In LdapBoundConnFactory::getConn() [09/Sep/2016:22:59:42][TP-Processor3]: masterConn is connected: true [09/Sep/2016:22:59:42][TP-Processor3]: getConn: conn is connected true [09/Sep/2016:22:59:42][TP-Processor3]: getConn: mNumConns now 2 [09/Sep/2016:22:59:42][TP-Processor3]: LdapAnonConnFactory::getConn [09/Sep/2016:22:59:42][TP-Processor3]: LdapAnonConnFactory.getConn(): num avail conns now 2 [09/Sep/2016:22:59:42][TP-Processor3]: returnConn: mNumConns now 3 [09/Sep/2016:22:59:42][TP-Processor3]: returnConn: mNumConns now 2 [09/Sep/2016:22:59:42][TP-Processor3]: SignedAuditEventFactory: create() message=[AuditEvent=AUTH_FAIL][SubjectID=$Unidentified$][Outcome=Failure][AuthMgr=passwdUserDBAuthMgr][AttemptedCred=$Unidentified$] authentication failure [09/Sep/2016:22:59:42][TP-Processor3]: GetCookie authentication failed [09/Sep/2016:22:59:42][TP-Processor3]: mErrorFormPath=/admin/ca/securitydomainlogin.template [09/Sep/2016:22:59:42][TP-Processor3]: CMSServlet: curDate=Fri Sep 09 22:59:42 EEST 2016 id=caGetCookie time=39 /var/log/httpd/access_log 192.168.4.175 - - [09/Sep/2016:22:59:21 +0300] "GET /ca/rest/securityDomain/domainInfo HTTP/1.1" 404 315 192.168.4.175 - - [09/Sep/2016:22:59:22 +0300] "GET /ca/admin/ca/getDomainXML HTTP/1.1" 200 1148 192.168.4.175 - - [09/Sep/2016:22:59:22 +0300] "GET /ca/rest/account/login HTTP/1.1" 404 303 192.168.4.175 - - [09/Sep/2016:22:59:41 +0300] "POST /ca/admin/ca/getCertChain HTTP/1.0" 200 1398 192.168.4.175 - - [09/Sep/2016:22:59:42 +0300] "GET /ca/rest/account/login HTTP/1.1" 404 303 192.168.4.175 - - [09/Sep/2016:22:59:42 +0300] "POST /ca/admin/ca/getCookie HTTP/1.1" 200 5170 /var/log/httpd/error_log [Fri Sep 09 22:59:22 2016] [error] [client 192.168.4.175] File does not exist: /var/www/html/ca [Fri Sep 09 22:59:22 2016] [error] [client 192.168.4.175] File does not exist: /var/www/html/ca [Fri Sep 09 22:59:42 2016] [error] [client 192.168.4.175] File does not exist: /var/www/html/ca /var/log/pki-ca/system 5337.TP-Processor3 - [09/Sep/2016:22:59:42 EEST] [6] [6] Failed to authenticate as admin UID=admin. Error: netscape.ldap.LDAPException: error result (49) 5337.TP-Processor3 - [09/Sep/2016:22:59:42 EEST] [3] [3] Servlet caGetCookie: Error getting servlet output stream when rendering template. Error Invalid Credential.. /var/log/pki-ca/catalina.out Sep 08, 2016 4:17:34 PM org.apache.catalina.startup.HostConfig deployDirectory INFO: Deploying web application directory ROOT Sep 08, 2016 4:17:34 PM org.apache.coyote.http11.Http11Protocol start INFO: Starting Coyote HTTP/1.1 on http-9180 Sep 08, 2016 4:17:34 PM org.apache.coyote.http11.Http11Protocol start INFO: Starting Coyote HTTP/1.1 on http-9443 Sep 08, 2016 4:17:35 PM org.apache.coyote.http11.Http11Protocol start INFO: Starting Coyote HTTP/1.1 on http-9445 Sep 08, 2016 4:17:35 PM org.apache.coyote.http11.Http11Protocol start INFO: Starting Coyote HTTP/1.1 on http-9444 Sep 08, 2016 4:17:35 PM org.apache.coyote.http11.Http11Protocol start INFO: Starting Coyote HTTP/1.1 on http-9446 Sep 08, 2016 4:17:35 PM org.apache.jk.common.ChannelSocket init INFO: JK: ajp13 listening on /0.0.0.0:9447 Sep 08, 2016 4:17:35 PM org.apache.jk.server.JkMain start INFO: Jk running ID=0 time=0/123 config=null Sep 08, 2016 4:17:35 PM org.apache.catalina.startup.Catalina start INFO: Server startup in 8550 ms Catalina seems to not have logged anything from yesterday. -------------- next part -------------- An HTML attachment was scrubbed... URL: From rakesh.rajasekharan at gmail.com Sun Sep 11 17:40:54 2016 From: rakesh.rajasekharan at gmail.com (Rakesh Rajasekharan) Date: Sun, 11 Sep 2016 23:10:54 +0530 Subject: [Freeipa-users] Increase ListenBacklog for httpd Message-ID: Hi, In my Freeipa setup, I frequently see this message request_sock_TCP: Possible SYN flooding on port 88. Sending cookies Is there a way to increase the ListenBacklog so that I can workaround this error as suggested i this doc https://access.redhat.com/solutions/30453 Thanks, Rakesh -------------- next part -------------- An HTML attachment was scrubbed... URL: From richard.harmonson at gmail.com Sun Sep 11 18:15:06 2016 From: richard.harmonson at gmail.com (Richard Harmonson) Date: Sun, 11 Sep 2016 11:15:06 -0700 Subject: [Freeipa-users] Disable DNS checks using ipa-server-intall with FreeIPA 4.3.2 on Fedora 24? Message-ID: Is there an option to disable the various DNS checks using ipa-server-install with FreeIPA 4.3.2? Is there plans to do provide the option in future releases? Reviewing the ipa-server-install man page, I am not seeing it. I want to compliment the team for placing safeguards on the installation of FreeIPA in respect to DNS best practices. I get the need to help some of the installers that may not have a strong grasp of DNS. However, there are scenarios where the checks becomes installation blockers or result in undesirable results. I am not trying to be mysterious, but I have no desire to embarrass third parties on their DNS faux pas. Thank you. -------------- next part -------------- An HTML attachment was scrubbed... URL: From datakid at gmail.com Mon Sep 12 01:09:05 2016 From: datakid at gmail.com (Lachlan Musicman) Date: Mon, 12 Sep 2016 11:09:05 +1000 Subject: [Freeipa-users] sssd stops after nss crashes Message-ID: We saw another sssd crash on the weekend (well, Friday night). Centos 7, sssd 1.14.0 from COPR Everything has worked fine for over a month until Friday. According to the log sssd_nss on the host in question: - at about 16:18, watchdog_handler killed a process for a timer overflow. - there is some flopping about as nss/sssd tries to reconnect - at 16:19:12 we see this: (Fri Sep 9 16:19:12 2016) [sssd[nss]] [sbus_dispatch] (0x0400): SBUS is reconnecting. Deferring. - Which continues until 16:20:56 (Fri Sep 9 16:20:56 2016) [sssd[nss]] [sbus_dispatch] (0x0400): SBUS is reconnecting. Deferring. Note that there are 9,573,091 lines of this, at about 80,000 msgs per second. - nss seems to stumble back to life at this point (there are no logs on the freeipa server unfortunately) - at every 15 min interval we see this (I think this might be zabbix polling sssd): (Fri Sep 9 18:30:01 2016) [sssd[nss]] [get_client_cred] (0x0020): SELINUX_getpeercon failed [-1][Unknown error -1]. (Fri Sep 9 18:30:01 2016) [sssd[nss]] [accept_fd_handler] (0x0400): Client connected! (Fri Sep 9 18:30:01 2016) [sssd[nss]] [sss_cmd_get_version] (0x0200): Received client version [1]. (Fri Sep 9 18:30:01 2016) [sssd[nss]] [sss_cmd_get_version] (0x0200): Offered version [1]. (Fri Sep 9 18:30:01 2016) [sssd[nss]] [nss_cmd_getbynam] (0x0400): Running command [38][SSS_NSS_INITGR] with input [root]. (Fri Sep 9 18:30:01 2016) [sssd[nss]] [sss_parse_name_for_domains] (0x0200): name 'root' matched without domain, user is root (Fri Sep 9 18:30:01 2016) [sssd[nss]] [nss_cmd_getbynam] (0x0100): Requesting info for [root] from [] (Fri Sep 9 18:30:01 2016) [sssd[nss]] [nss_cmd_initgroups_search] (0x0400): User [root at unix.petermac.org.au] does not exist in [ unix.petermac.org.au]! (negative cache) (Fri Sep 9 18:30:01 2016) [sssd[nss]] [nss_cmd_initgroups_search] (0x0080): No matching domain found for [root], fail! (Fri Sep 9 18:30:01 2016) [sssd[nss]] [client_recv] (0x0200): Client disconnected! Interspersed there is the occasional successful connection. - Then at 20:41 everything decides to fall over properly: (Fri Sep 9 20:41:07 2016) [sssd[nss]] [watchdog_handler] (0x0010): Watchdog timer overflow, killing process! (Fri Sep 9 20:41:07 2016) [sssd[nss]] [sss_responder_ctx_destructor] (0x0400): Responder is being shut down (Fri Sep 9 20:41:07 2016) [sssd[nss]] [server_setup] (0x0400): CONFDB: /var/lib/sss/db/config.ldb (Fri Sep 9 20:41:07 2016) [sssd[nss]] [confdb_get_domain_internal] (0x0400): No enumeration for [unix.petermac.org.au]! (Fri Sep 9 20:41:07 2016) [sssd[nss]] [sbus_init_connection] (0x0400): Adding connection 0x7f25be061590 (Fri Sep 9 20:41:07 2016) [sssd[nss]] [sbus_opath_hash_add_iface] (0x0400): Registering interface org.freedesktop.sssd.service with path /org/freedesktop/sssd/service (Fri Sep 9 20:41:07 2016) [sssd[nss]] [sbus_conn_register_path] (0x0400): Registering object path /org/freedesktop/sssd/service with D-Bus connection (Fri Sep 9 20:41:07 2016) [sssd[nss]] [sbus_opath_hash_add_iface] (0x0400): Registering interface org.freedesktop.DBus.Properties with path /org/freedesktop/sssd/service (Fri Sep 9 20:41:07 2016) [sssd[nss]] [sbus_opath_hash_add_iface] (0x0400): Registering interface org.freedesktop.DBus.Introspectable with path /org/freedesktop/sssd/service (Fri Sep 9 20:41:07 2016) [sssd[nss]] [monitor_common_send_id] (0x0100): Sending ID: (nss,1) (Fri Sep 9 20:41:07 2016) [sssd[nss]] [sss_names_init_from_args] (0x0100): Using re [(((?P[^\\]+)\\(?P.+$))|((?P[^@]+)@(?P.+$))|(^(?P[^@\\]+)$))]. (Fri Sep 9 20:41:07 2016) [sssd[nss]] [sss_fqnames_init] (0x0100): Using fq format [%1$s@%2$s]. (Fri Sep 9 20:41:07 2016) [sssd[nss]] [check_file] (0x0400): lstat for [/var/lib/sss/pipes/private/sbus-dp_unix.petermac.org.au] failed: [2][No such file or directory]. (Fri Sep 9 20:41:07 2016) [sssd[nss]] [sbus_client_init] (0x0020): check_file failed for [/var/lib/sss/pipes/private/ sbus-dp_unix.petermac.org.au]. (Fri Sep 9 20:41:07 2016) [sssd[nss]] [sss_dp_init] (0x0010): Failed to connect to monitor services. (Fri Sep 9 20:41:07 2016) [sssd[nss]] [sss_process_init] (0x0010): fatal error setting up backend connector (Fri Sep 9 20:41:07 2016) [sssd[nss]] [sss_responder_ctx_destructor] (0x0400): Responder is being shut down (Fri Sep 9 20:41:07 2016) [sssd[nss]] [nss_process_init] (0x0010): sss_process_init() failed (Fri Sep 9 20:41:09 2016) [sssd[nss]] [server_setup] (0x0400): CONFDB: /var/lib/sss/db/config.ldb (Fri Sep 9 20:41:09 2016) [sssd[nss]] [confdb_get_domain_internal] (0x0400): No enumeration for [unix.petermac.org.au]! (Fri Sep 9 20:41:09 2016) [sssd[nss]] [sbus_init_connection] (0x0400): Adding connection 0x7fd8ac102590 (Fri Sep 9 20:41:09 2016) [sssd[nss]] [sbus_opath_hash_add_iface] (0x0400): Registering interface org.freedesktop.sssd.service with path /org/freedesktop/sssd/service (Fri Sep 9 20:41:09 2016) [sssd[nss]] [sbus_conn_register_path] (0x0400): Registering object path /org/freedesktop/sssd/service with D-Bus connection (Fri Sep 9 20:41:09 2016) [sssd[nss]] [sbus_opath_hash_add_iface] (0x0400): Registering interface org.freedesktop.DBus.Properties with path /org/freedesktop/sssd/service (Fri Sep 9 20:41:09 2016) [sssd[nss]] [sbus_opath_hash_add_iface] (0x0400): Registering interface org.freedesktop.DBus.Introspectable with path /org/freedesktop/sssd/service (Fri Sep 9 20:41:09 2016) [sssd[nss]] [monitor_common_send_id] (0x0100): Sending ID: (nss,1) (Fri Sep 9 20:41:09 2016) [sssd[nss]] [sss_names_init_from_args] (0x0100): Using re [(((?P[^\\]+)\\(?P.+$))|((?P[^@]+)@(?P.+$))|(^(?P[^@\\]+)$))]. (Fri Sep 9 20:41:09 2016) [sssd[nss]] [sss_fqnames_init] (0x0100): Using fq format [%1$s@%2$s]. (Fri Sep 9 20:41:09 2016) [sssd[nss]] [check_file] (0x0400): lstat for [/var/lib/sss/pipes/private/sbus-dp_unix.petermac.org.au] failed: [2][No such file or directory]. (Fri Sep 9 20:41:09 2016) [sssd[nss]] [sbus_client_init] (0x0020): check_file failed for [/var/lib/sss/pipes/private/ sbus-dp_unix.petermac.org.au]. (Fri Sep 9 20:41:09 2016) [sssd[nss]] [sss_dp_init] (0x0010): Failed to connect to monitor services. (Fri Sep 9 20:41:09 2016) [sssd[nss]] [sss_process_init] (0x0010): fatal error setting up backend connector (Fri Sep 9 20:41:09 2016) [sssd[nss]] [sss_responder_ctx_destructor] (0x0400): Responder is being shut down (Fri Sep 9 20:41:09 2016) [sssd[nss]] [nss_process_init] (0x0010): sss_process_init() failed (Fri Sep 9 20:41:13 2016) [sssd[nss]] [server_setup] (0x0400): CONFDB: /var/lib/sss/db/config.ldb (Fri Sep 9 20:41:13 2016) [sssd[nss]] [confdb_get_domain_internal] (0x0400): No enumeration for [unix.petermac.org.au]! (Fri Sep 9 20:41:13 2016) [sssd[nss]] [sbus_init_connection] (0x0400): Adding connection 0x7f93f5c92590 (Fri Sep 9 20:41:13 2016) [sssd[nss]] [sbus_opath_hash_add_iface] (0x0400): Registering interface org.freedesktop.sssd.service with path /org/freedesktop/sssd/service (Fri Sep 9 20:41:13 2016) [sssd[nss]] [sbus_conn_register_path] (0x0400): Registering object path /org/freedesktop/sssd/service with D-Bus connection (Fri Sep 9 20:41:13 2016) [sssd[nss]] [sbus_opath_hash_add_iface] (0x0400): Registering interface org.freedesktop.DBus.Properties with path /org/freedesktop/sssd/service (Fri Sep 9 20:41:13 2016) [sssd[nss]] [sbus_opath_hash_add_iface] (0x0400): Registering interface org.freedesktop.DBus.Introspectable with path /org/freedesktop/sssd/service (Fri Sep 9 20:41:13 2016) [sssd[nss]] [monitor_common_send_id] (0x0100): Sending ID: (nss,1) (Fri Sep 9 20:41:13 2016) [sssd[nss]] [sss_names_init_from_args] (0x0100): Using re [(((?P[^\\]+)\\(?P.+$))|((?P[^@]+)@(?P.+$))|(^(?P[^@\\]+)$))]. (Fri Sep 9 20:41:13 2016) [sssd[nss]] [sss_fqnames_init] (0x0100): Using fq format [%1$s@%2$s]. (Fri Sep 9 20:41:13 2016) [sssd[nss]] [check_file] (0x0400): lstat for [/var/lib/sss/pipes/private/sbus-dp_unix.petermac.org.au] failed: [2][No such file or directory]. (Fri Sep 9 20:41:13 2016) [sssd[nss]] [sbus_client_init] (0x0020): check_file failed for [/var/lib/sss/pipes/private/ sbus-dp_unix.petermac.org.au]. (Fri Sep 9 20:41:13 2016) [sssd[nss]] [sss_dp_init] (0x0010): Failed to connect to monitor services. (Fri Sep 9 20:41:13 2016) [sssd[nss]] [sss_process_init] (0x0010): fatal error setting up backend connector (Fri Sep 9 20:41:13 2016) [sssd[nss]] [sss_responder_ctx_destructor] (0x0400): Responder is being shut down (Fri Sep 9 20:41:13 2016) [sssd[nss]] [nss_process_init] (0x0010): sss_process_init() failed In sssd.log from the same day, we see the start at 16:18, and it repeats for 2 minutes: (Fri Sep 9 16:18:02 2016) [sssd] [svc_child_info] (0x0040): Child [4228] exited with code [1] (Fri Sep 9 16:18:02 2016) [sssd] [sbus_dispatch] (0x0080): Connection is not open for dispatching. (Fri Sep 9 16:18:03 2016) [sssd] [mt_svc_restart] (0x0400): Scheduling service nss for restart 1 (Fri Sep 9 16:18:04 2016) [sssd] [get_kill_config] (0x0100): Time between SIGTERM and SIGKILL for [nss]: [60] (Fri Sep 9 16:18:04 2016) [sssd] [start_service] (0x0100): Queueing service nss for startup (Fri Sep 9 16:18:16 2016) [sssd] [sbus_server_init_new_connection] (0x0200): Entering. (Fri Sep 9 16:18:16 2016) [sssd] [sbus_server_init_new_connection] (0x0200): Adding connection 0x7f3617c3aa30. (Fri Sep 9 16:18:16 2016) [sssd] [sbus_init_connection] (0x0400): Adding connection 0x7f3617c3aa30 (Fri Sep 9 16:18:16 2016) [sssd] [sbus_server_init_new_connection] (0x0200): Got a connection (Fri Sep 9 16:18:16 2016) [sssd] [monitor_service_init] (0x0400): Initializing D-BUS Service (Fri Sep 9 16:18:16 2016) [sssd] [sbus_opath_hash_add_iface] (0x0400): Registering interface org.freedesktop.sssd.monitor with path /org/freedesktop/sssd/monitor (Fri Sep 9 16:18:16 2016) [sssd] [sbus_conn_register_path] (0x0400): Registering object path /org/freedesktop/sssd/monitor with D-Bus connection (Fri Sep 9 16:18:16 2016) [sssd] [sbus_opath_hash_add_iface] (0x0400): Registering interface org.freedesktop.DBus.Properties with path /org/freedesktop/sssd/monitor (Fri Sep 9 16:18:16 2016) [sssd] [sbus_opath_hash_add_iface] (0x0400): Registering interface org.freedesktop.DBus.Introspectable with path /org/freedesktop/sssd/monitor (Fri Sep 9 16:18:21 2016) [sssd] [message_type] (0x0200): netlink Message type: 25 (Fri Sep 9 16:18:26 2016) [sssd] [init_timeout] (0x0040): Client timed out before Identification! (Fri Sep 9 16:18:26 2016) [sssd] [sbus_disconnect] (0x0400): Disconnecting 0x7f3617c3aa30 (Fri Sep 9 16:18:26 2016) [sssd] [connection_destructor] (0x0400): Invoking default destructor on connection 0x7f3617c3aa30 (Fri Sep 9 16:18:26 2016) [sssd] [sbus_disconnect] (0x0400): Disconnected 0x7f3617c3aa30 (Fri Sep 9 16:18:51 2016) [sssd] [message_type] (0x0200): netlink Message type: 24 After which we see this: (Fri Sep 9 16:20:21 2016) [sssd] [service_signal] (0x0020): Could not signal service [unix.petermac.org.au]. (Fri Sep 9 16:20:51 2016) [sssd] [message_type] (0x0200): netlink Message type: 24 Which continues until everything dies: (Fri Sep 9 20:38:21 2016) [sssd] [service_signal] (0x0020): Could not signal service [unix.petermac.org.au]. (Fri Sep 9 20:38:51 2016) [sssd] [message_type] (0x0200): netlink Message type: 24 (Fri Sep 9 20:38:51 2016) [sssd] [service_signal] (0x0020): Could not signal service [unix.petermac.org.au]. (Fri Sep 9 20:39:21 2016) [sssd] [message_type] (0x0200): netlink Message type: 25 (Fri Sep 9 20:39:21 2016) [sssd] [service_signal] (0x0020): Could not signal service [unix.petermac.org.au]. (Fri Sep 9 20:39:51 2016) [sssd] [message_type] (0x0200): netlink Message type: 24 (Fri Sep 9 20:39:51 2016) [sssd] [service_signal] (0x0020): Could not signal service [unix.petermac.org.au]. (Fri Sep 9 20:40:21 2016) [sssd] [message_type] (0x0200): netlink Message type: 25 (Fri Sep 9 20:40:21 2016) [sssd] [service_signal] (0x0020): Could not signal service [unix.petermac.org.au]. (Fri Sep 9 20:40:51 2016) [sssd] [message_type] (0x0200): netlink Message type: 24 (Fri Sep 9 20:40:51 2016) [sssd] [service_signal] (0x0020): Could not signal service [unix.petermac.org.au]. (Fri Sep 9 20:41:07 2016) [sssd] [svc_child_info] (0x0040): Child [37140] exited with code [1] (Fri Sep 9 20:41:07 2016) [sssd] [mt_svc_restart] (0x0400): Scheduling service nss for restart 1 (Fri Sep 9 20:41:07 2016) [sssd] [get_kill_config] (0x0100): Time between SIGTERM and SIGKILL for [nss]: [60] (Fri Sep 9 20:41:07 2016) [sssd] [start_service] (0x0100): Queueing service nss for startup (Fri Sep 9 20:41:07 2016) [sssd] [sbus_server_init_new_connection] (0x0200): Entering. (Fri Sep 9 20:41:07 2016) [sssd] [sbus_server_init_new_connection] (0x0200): Adding connection 0x7f3617c251c0. (Fri Sep 9 20:41:07 2016) [sssd] [sbus_init_connection] (0x0400): Adding connection 0x7f3617c251c0 (Fri Sep 9 20:41:07 2016) [sssd] [sbus_server_init_new_connection] (0x0200): Got a connection (Fri Sep 9 20:41:07 2016) [sssd] [monitor_service_init] (0x0400): Initializing D-BUS Service (Fri Sep 9 20:41:07 2016) [sssd] [sbus_opath_hash_add_iface] (0x0400): Registering interface org.freedesktop.sssd.monitor with path /org/freedesktop/sssd/monitor (Fri Sep 9 20:41:07 2016) [sssd] [sbus_conn_register_path] (0x0400): Registering object path /org/freedesktop/sssd/monitor with D-Bus connection (Fri Sep 9 20:41:07 2016) [sssd] [sbus_opath_hash_add_iface] (0x0400): Registering interface org.freedesktop.DBus.Properties with path /org/freedesktop/sssd/monitor (Fri Sep 9 20:41:07 2016) [sssd] [sbus_opath_hash_add_iface] (0x0400): Registering interface org.freedesktop.DBus.Introspectable with path /org/freedesktop/sssd/monitor (Fri Sep 9 20:41:07 2016) [sssd] [svc_child_info] (0x0040): Child [40794] exited with code [3] (Fri Sep 9 20:41:07 2016) [sssd] [sbus_dispatch] (0x0080): Connection is not open for dispatching. (Fri Sep 9 20:41:09 2016) [sssd] [mt_svc_restart] (0x0400): Scheduling service nss for restart 2 (Fri Sep 9 20:41:09 2016) [sssd] [get_kill_config] (0x0100): Time between SIGTERM and SIGKILL for [nss]: [60] (Fri Sep 9 20:41:09 2016) [sssd] [start_service] (0x0100): Queueing service nss for startup (Fri Sep 9 20:41:09 2016) [sssd] [sbus_server_init_new_connection] (0x0200): Entering. (Fri Sep 9 20:41:09 2016) [sssd] [sbus_server_init_new_connection] (0x0200): Adding connection 0x7f3617c2bbd0. (Fri Sep 9 20:41:09 2016) [sssd] [sbus_init_connection] (0x0400): Adding connection 0x7f3617c2bbd0 (Fri Sep 9 20:41:09 2016) [sssd] [sbus_server_init_new_connection] (0x0200): Got a connection (Fri Sep 9 20:41:09 2016) [sssd] [monitor_service_init] (0x0400): Initializing D-BUS Service (Fri Sep 9 20:41:09 2016) [sssd] [sbus_opath_hash_add_iface] (0x0400): Registering interface org.freedesktop.sssd.monitor with path /org/freedesktop/sssd/monitor (Fri Sep 9 20:41:09 2016) [sssd] [sbus_conn_register_path] (0x0400): Registering object path /org/freedesktop/sssd/monitor with D-Bus connection (Fri Sep 9 20:41:09 2016) [sssd] [sbus_opath_hash_add_iface] (0x0400): Registering interface org.freedesktop.DBus.Properties with path /org/freedesktop/sssd/monitor (Fri Sep 9 20:41:09 2016) [sssd] [sbus_opath_hash_add_iface] (0x0400): Registering interface org.freedesktop.DBus.Introspectable with path /org/freedesktop/sssd/monitor (Fri Sep 9 20:41:09 2016) [sssd] [svc_child_info] (0x0040): Child [40795] exited with code [3] (Fri Sep 9 20:41:09 2016) [sssd] [sbus_dispatch] (0x0080): Connection is not open for dispatching. (Fri Sep 9 20:41:13 2016) [sssd] [mt_svc_restart] (0x0400): Scheduling service nss for restart 3 (Fri Sep 9 20:41:13 2016) [sssd] [get_kill_config] (0x0100): Time between SIGTERM and SIGKILL for [nss]: [60] (Fri Sep 9 20:41:13 2016) [sssd] [start_service] (0x0100): Queueing service nss for startup (Fri Sep 9 20:41:13 2016) [sssd] [sbus_server_init_new_connection] (0x0200): Entering. (Fri Sep 9 20:41:13 2016) [sssd] [sbus_server_init_new_connection] (0x0200): Adding connection 0x7f3617c27690. (Fri Sep 9 20:41:13 2016) [sssd] [sbus_init_connection] (0x0400): Adding connection 0x7f3617c27690 (Fri Sep 9 20:41:13 2016) [sssd] [sbus_server_init_new_connection] (0x0200): Got a connection (Fri Sep 9 20:41:13 2016) [sssd] [monitor_service_init] (0x0400): Initializing D-BUS Service (Fri Sep 9 20:41:13 2016) [sssd] [sbus_opath_hash_add_iface] (0x0400): Registering interface org.freedesktop.sssd.monitor with path /org/freedesktop/sssd/monitor (Fri Sep 9 20:41:13 2016) [sssd] [sbus_conn_register_path] (0x0400): Registering object path /org/freedesktop/sssd/monitor with D-Bus connection (Fri Sep 9 20:41:13 2016) [sssd] [sbus_opath_hash_add_iface] (0x0400): Registering interface org.freedesktop.DBus.Properties with path /org/freedesktop/sssd/monitor (Fri Sep 9 20:41:13 2016) [sssd] [sbus_opath_hash_add_iface] (0x0400): Registering interface org.freedesktop.DBus.Introspectable with path /org/freedesktop/sssd/monitor (Fri Sep 9 20:41:13 2016) [sssd] [svc_child_info] (0x0040): Child [40796] exited with code [3] (Fri Sep 9 20:41:13 2016) [sssd] [monitor_restart_service] (0x0010): Process [nss], definitely stopped! (Fri Sep 9 20:41:13 2016) [sssd] [monitor_quit] (0x0040): Returned with: 1 (Fri Sep 9 20:41:13 2016) [sssd] [monitor_quit] (0x0020): Terminating [ unix.petermac.org.au][37141] (Fri Sep 9 20:41:23 2016) [sssd] [monitor_quit] (0x0020): Child [ unix.petermac.org.au] terminated with a signal (Fri Sep 9 20:41:23 2016) [sssd] [monitor_quit] (0x0020): Terminating [pac][4232] (Fri Sep 9 20:41:23 2016) [sssd] [monitor_quit] (0x0020): Child [pac] exited gracefully (Fri Sep 9 20:41:23 2016) [sssd] [monitor_quit] (0x0020): Terminating [ssh][4231] (Fri Sep 9 20:41:23 2016) [sssd] [monitor_quit] (0x0020): Child [ssh] exited gracefully (Fri Sep 9 20:41:23 2016) [sssd] [monitor_quit] (0x0020): Terminating [pam][4230] (Fri Sep 9 20:41:23 2016) [sssd] [monitor_quit] (0x0020): Child [pam] exited gracefully (Fri Sep 9 20:41:23 2016) [sssd] [monitor_quit] (0x0020): Terminating [sudo][4229] (Fri Sep 9 20:41:23 2016) [sssd] [monitor_quit] (0x0020): Child [sudo] exited gracefully On the FreeIPA server we see this in messages (no sssd logging :( ). I don't know if it's significant, but it's the only error in the rough time frame (or that relates in any way to FreeIPA) Sep 9 20:35:47 vmpr-linuxidm winbindd[32689]: [2016/09/09 20:35:47.780283, 0] ipa_sam.c:4208(bind_callback_cleanup) Sep 9 20:35:47 vmpr-linuxidm winbindd[32689]: kerberos error: code=-1765328228, message=Cannot contact any KDC for realm ' UNIX.PETERMAC.ORG.AU' Sep 9 20:35:47 vmpr-linuxidm winbindd[32689]: [2016/09/09 20:35:47.780674, 0] ../source3/lib/smbldap.c:998(smbldap_connect_system) Sep 9 20:35:47 vmpr-linuxidm winbindd[32689]: failed to bind to server ldapi://%2fvar%2frun%2fslapd-UNIX-PETERMAC-ORG-AU.socket with dn="[Anonymous bind]" Error: Local error Sep 9 20:35:47 vmpr-linuxidm winbindd[32689]: #011(unknown) Sep 9 20:36:30 vmpr-linuxidm winbindd[32689]: [2016/09/09 20:36:30.799458, 0] ipa_sam.c:4208(bind_callback_cleanup) Sep 9 20:36:30 vmpr-linuxidm winbindd[32689]: kerberos error: code=-1765328228, message=Cannot contact any KDC for realm ' UNIX.PETERMAC.ORG.AU' (have found this, although it was recently closed https://bugzilla.redhat.com/show_bug.cgi?id=1071449 ) Let me know if you need any more info. Cheers L. ------ The most dangerous phrase in the language is, "We've always done it this way." - Grace Hopper -------------- next part -------------- An HTML attachment was scrubbed... URL: From rakesh.rajasekharan at gmail.com Mon Sep 12 04:43:17 2016 From: rakesh.rajasekharan at gmail.com (Rakesh Rajasekharan) Date: Mon, 12 Sep 2016 10:13:17 +0530 Subject: [Freeipa-users] Increase ListenBacklog for httpd In-Reply-To: References: Message-ID: sorry I guess I did not put the question correctly.... I wanted to know .. like we have the ListenBacklog for apache to basically define the number of connections it can handle.. do we have some thing similar for our krb5kdc service.. as the SYN floodin at 88 looks like krb5kdc service is not able to handle sudden spurt in connections or the number of connections are more than it could handle.. So, would be great if I could know how many connection it can support at any given time ..most of the times I see this error while i add clients to IPA master.. so if thers a known limit , I could first check netstat to see how many connections I have at any point and if its below the limit only then setup ipa-client-install Thanks, Rakesh On Sun, Sep 11, 2016 at 11:10 PM, Rakesh Rajasekharan < rakesh.rajasekharan at gmail.com> wrote: > Hi, > > In my Freeipa setup, I frequently see this message > > request_sock_TCP: Possible SYN flooding on port 88. Sending cookies > > > Is there a way to increase the ListenBacklog so that I can workaround this > error as suggested i this doc > https://access.redhat.com/solutions/30453 > > Thanks, > Rakesh > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From natxo.asenjo at gmail.com Mon Sep 12 07:09:59 2016 From: natxo.asenjo at gmail.com (Natxo Asenjo) Date: Mon, 12 Sep 2016 09:09:59 +0200 Subject: [Freeipa-users] ipa: ERROR: Certificate format error: (SEC_ERROR_LEGACY_DATABASE) The certificate/key database is in an old, unsupported format. In-Reply-To: References: <57D015D8.7080507@redhat.com> <57D166B9.3070201@redhat.com> Message-ID: hi, I can reproduce this everytime. Restarting httpd fixes it for a while, but then ik stops working: $ ipa cert-show 1 ipa: ERROR: cannot connect to ' https://kdc01.unix.domain.tld:443/ca/agent/ca/displayBySerial': (SEC_ERROR_LEGACY_DATABASE) The certificate/key database is in an old, unsupported format. [jose.admin at kdc01 ~]$ sudo /etc/init.d/httpd restart Stopping httpd: [ OK ] Starting httpd: [ OK ] [jose.admin at kdc01 ~]$ ipa cert-show 1 Certificate: MIIDnDCCAoSgAwIBAgIBATANBgkqhkiG9w0BAQsFADA7MRkwFwYDVQQKExBVTklY LklSSVNaT1JHLk5MMR4wHAYDVQQDExVDZXJ0aWZpY2F0ZSBBdXRob3JpdHkwHhcN MTIxMTA3MjEyNDE1WhcNMjAxMTA3MjEyNDE1WjA7MRkwFwYDVQQKExBVTklYLklS SVNaT1JHLk5MMR4wHAYDVQQDExVDZXJ0aWZpY2F0ZSBBdXRob3JpdHkwggEiMA0G CSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQCy2WVy7QkHiuENW/zkMeD4ILoqOruu YKvb2+rqeuI9iw+zBBt569XSxrgcyeTq0G63RjbXgrAzot4EhYg6MoepDVCn0Bnu rUfgbCf5R0Eboigjboh5MGnPylHefLRGARNUCwcTGA4uR9ZQL/rEUqWktmZjanYE vOP8UBeuq5WP5emaX8U03SzMA+cQT9w/zx0eAOYgZW5yx3aA5Q4Fu8qWqMGGAOA6 yDQWqmIpgxiFHHRa7hQK4AjeHgvaColaU979Lh5jAv/XwrYtok1G+UVEp45INpfx r5dLe03ognPFPZ0/xwbBqtt/2qn6rk4L4ukH4P9g4Rw0o7U1yJVx/SOJAgMBAAGj gaowgacwHwYDVR0jBBgwFoAUo5fkii64zz7qM/K8k9Yj3qmENmgwDwYDVR0TAQH/ BAUwAwEB/zAOBgNVHQ8BAf8EBAMCAcYwHQYDVR0OBBYEFKOX5IouuM8+6jPyvJPW I96phDZoMEQGCCsGAQUFBwEBBDgwNjA0BggrBgEFBQcwAYYoaHR0cDovL2tkYzAx LnVuaXguaXJpc3pvcmcubmw6ODAvY2Evb2NzcDANBgkqhkiG9w0BAQsFAAOCAQEA J28gdozd/ptOM5PTKKwyV+otO/wk3yErslxpNUhRZgSNUwT+t6tfF/j+jJRV5sX+ jy09c9Do+p3Hy9gRnIVJONDScvMV9nDc75C6JGXU+FdNJJ+Dbpep/RsQjHrZ+unw IyAWoOpBol8sGzN5tXbeo/M6mGFxaBTH1GKtgv4CKbzQAotvMaGxzKjScHRsGaer NSCZp/90yRJypC3MOosUFcFl4CoYHB42XDTzjvzZQcaFNcgYXOciujwwYHNzsSqY cIKFSWuWvN++7g4yxQMlu8QW0Ms/PntmTmO2cDdNI1tujVyBKe599y4O/Es/MBGt DtVA85ALksJOU27bjtvbBg== Subject: CN=Certificate Authority,O=UNIX.DOMAIN.TLD Issuer: CN=Certificate Authority,O=UNIX.DOMAIN.TLD Not Before: Wed Nov 07 21:24:15 2012 UTC Not After: Sat Nov 07 21:24:15 2020 UTC Fingerprint (MD5): 28:18:34:9d:03:99:b8:ff:2b:bd:55:0a:65:bf:d4:f2 Fingerprint (SHA1): 6f:e1:a4:4f:47:ec:9c:c4:ad:b9:b9:fc:e8:f4:33:4b:0a:cb:43:3e Serial number (hex): 0x1 Serial number: 1 And a few minutes later (5, maximum 10), then I get the SEC_ERROR_LEGACY_DATABASE error. No traceback in /var/log/httpd/error_log. This is the first CA domain controller. I am leaving this job in a few weeks, so I would like to leave everything working properly. Would it be better to upgrade the domain controllers to centos 7 (right now running centos 6.8, fully patched). Thanks for your input. -- regards, natxo On Thu, Sep 8, 2016 at 6:30 PM, Natxo Asenjo wrote: > > > On Thu, Sep 8, 2016 at 3:25 PM, Rob Crittenden > wrote: > >> Natxo Asenjo wrote: >> >>> I do see these errors: >>> [Wed Sep 07 15:56:13 2016] [error] ipa: INFO:: ping(): SUCCESS >>> [Wed Sep 07 15:56:13 2016] [error] ipa: INFO: : host_find(u'tftp-1801', >>> all=False, raw=False, version=u'2.49', no_members=False, >>> pkey_only=False): CertificateFormatError >>> [Wed Sep 07 15:56:44 2016] [error] ipa: INFO: : ping(): SUCCESS >>> [Wed Sep 07 15:56:44 2016] [error] ipa: INFO: : host_find(u'tftp-1801', >>> all=False, raw=False, version=u'2.49', no_members=False, >>> pkey_only=False): CertificateFormatError >>> [Wed Sep 07 15:57:57 2016] [error] ipa: INFO: : ping(): SUCCESS >>> [Wed Sep 07 15:57:58 2016] [error] ipa: INFO: : host_find(u'tftp-1801', >>> all=False, raw=False, version=u'2.49', no_members=False, >>> pkey_only=False): CertificateFormatErro >>> >>> >>> On Wed, Sep 7, 2016 at 4:01 PM, Natxo Asenjo >> > wrote: >>> >>> >>> alas, not woriking again. >>> >>> On the one kdc >>> >>> $ ipa host-find tftp-1801 >>> ipa: ERROR: Certificate format error: (SEC_ERROR_LEGACY_DATABASE) >>> The certificate/key database is in an old, unsupported format. >>> >>> On the other: >>> >>> $ ipa host-find tftp-1801 >>> -------------- >>> 1 host matched >>> -------------- >>> Host name: tftp-1801.sub.domain.tld >>> ..... >>> >>> After rebooting the kdc with the error, no new tracebacks in the >>> error_log >>> >> >> No new tracebacks but still not working? >> >> The CertificateFormatError is the server logging the equivalent of what >> you're seeing in the client. >> >> rob >> > > > that's right. > > Is there anything else I can look at? > > > -- > -- > Groeten, > natxo > -- -- Groeten, natxo -------------- next part -------------- An HTML attachment was scrubbed... URL: From siology.io at gmail.com Sun Sep 11 21:38:12 2016 From: siology.io at gmail.com (siology.io) Date: Mon, 12 Sep 2016 09:38:12 +1200 Subject: [Freeipa-users] problems with ipa server no longer responding to ldap Message-ID: Hello there. My setup is that i have five ipa servers. 2 in one location (alder, auth-syd2), 2 in anouther location (auth-wlg, auth-wlg2), and one in yet anouther location (waffle) which is reached over a long, mostly-but-possibly-notably-not-entirely reliable vpn connection. I'm having an issue with an IPA server falling over. By 'falling over' what i mean is that it no longer responds to ldap queries (although the tcp port 389 is still open via nmap). When i run 'systemctl ipa stop' the command never seems to return, so up to now the only fix i have it to reboot that server. All machines are centos 7. All are using ipa-server-4.2.0-15.0.1.el7.centos.18.x86_64. Replication occurs between: alder<->auth-wlg, alder<->syd2, auth-wlg<->auth-wlg2, and auth-wlg<->waffle, possibly notably *not* between alder and waffle directly. The problem of ldap being unavailable occurs on alder only; the other ipa servers seem to be reliable. Unfortunately, alder is also our most used server. The error logs off alder look like this: http://pastebin.com/TxCVjWTe with reboot done at around 19:55 I did notice upon investigating / googling the errors in this log - starting with the attr_replace (nsslapd-referral) one, that on my servers this ldap query: ldapsearch -ZZ -h alder.blah.com -D "cn=Directory Manager" -W -b "o=ipaca" "(&(objectclass=nstombstone)(nsUniqueId=ffffffff-ffffffff-ffffffff-ffffffff))" | grep "nsds50ruv\|nsDS5ReplicaId" returns results similar to this: nsDS5ReplicaId: 96 nsds50ruv: {replicageneration} 5733d428000000600000 nsds50ruv: {replica 96 ldap://alder.blah.com:389} 5733d474000000600000 57 nsds50ruv: {replica 91 ldap://auth-syd2.blah.com:389} 576337b90004005b000 nsds50ruv: {replica 97 ldap://auth-wlg.blah.com:389} 5733d49a000000610000 nsds50ruv: {replica 1095 ldap://auth-wlg2.blah.com:389} 574fa5b0000004470 nsds50ruv: {replica 1090 ldap://waffle.bsh.blah.com:389} 576b1add00000442 nsds50ruv: {replica 1085 ldap://waffle.bsh.blah.com:389} 576b22f10000043d i.e: waffle is listed twice. If i run that ldap query on waffle though, i get no results at all (but the command does at least return). - so i dont know waffle's nsDS5ReplicaId at the moment. I understand once i know that i can clean-ruv the other id off the other ipa servers? I don't *think* any of this is related to my original issue above though, but it might be a smoking gun, i don't know - just mentioning it in case. At the moment i've not got a lot to go on. Has anyone else seen errors like those in the paste bin, or might know where to look for more useful info ? Possibly also worth noting that alder, and auth-syd2 are AWS ec2 instances. The rest are vm's on site(s). -------------- next part -------------- An HTML attachment was scrubbed... URL: From fujisan43 at gmail.com Mon Sep 12 07:31:08 2016 From: fujisan43 at gmail.com (Fujisan) Date: Mon, 12 Sep 2016 09:31:08 +0200 Subject: [Freeipa-users] The Web UI is not loading Message-ID: Hello, This morning I noticed I could not reload the Freeipa web ui. Its was working well friday but something must have happend over the weekend. The browser says: -------------------------------------------------- This site can?t be reached The webpage at *https://myserver.mydomain/ipa/ui/ * might be temporarily down or it may have moved permanently to a new web address. -------------------------------------------------- # rpm -qa|grep cockpit cockpit-docker-0.117-1.fc24.x86_64 cockpit-0.117-1.fc24.x86_64 cockpit-sosreport-0.117-1.fc24.noarch cockpit-ws-0.117-1.fc24.x86_64 cockpit-networkmanager-0.117-1.fc24.noarch cockpit-doc-0.117-1.fc24.x86_64 cockpit-shell-0.117-1.fc24.noarch cockpit-storaged-0.117-1.fc24.noarch cockpit-bridge-0.117-1.fc24.x86_64 Below are some logs when I restart the ipa service. Regards, F. # systemctl restart cockpit.service Sep 12 09:12:48 myserver systemd: Stopping Identity, Policy, Audit... Sep 12 09:12:49 myserver systemd: Stopping IPA key daemon... Sep 12 09:12:49 myserver ipa-dnskeysyncd: ipa : INFO Signal 15 received: Shutting down! Sep 12 09:12:49 myserver systemd: Stopped IPA key daemon. Sep 12 09:12:49 myserver audit: SERVICE_STOP pid=1 uid=0 auid=4294967295 ses=4294967295 msg='unit=ipa-dnskeysyncd comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success' Sep 12 09:12:49 myserver systemd: Stopping ipa-otpd socket. Sep 12 09:12:49 myserver systemd: Closed ipa-otpd socket. Sep 12 09:12:49 myserver systemd: Stopping Samba Winbind Daemon... Sep 12 09:12:49 myserver winbindd[144495]: [2016/09/12 09:12:49.556881, 0] ../source3/winbindd/winbindd.c:280(winbindd_sig_term_handler) Sep 12 09:12:49 myserver winbindd[144495]: Got sig[15] terminate (is_parent=0) Sep 12 09:12:49 myserver winbindd[144339]: [2016/09/12 09:12:49.556884, 0] ../source3/winbindd/winbindd.c:280(winbindd_sig_term_handler) Sep 12 09:12:49 myserver winbindd[144339]: Got sig[15] terminate (is_parent=1) Sep 12 09:12:49 myserver winbindd[144340]: [2016/09/12 09:12:49.556898, 0] ../source3/winbindd/winbindd.c:280(winbindd_sig_term_handler) Sep 12 09:12:49 myserver winbindd[144340]: Got sig[15] terminate (is_parent=0) Sep 12 09:12:49 myserver systemd: Stopped Samba Winbind Daemon. Sep 12 09:12:49 myserver audit: SERVICE_STOP pid=1 uid=0 auid=4294967295 ses=4294967295 msg='unit=winbind comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success' Sep 12 09:12:49 myserver systemd: Stopping Samba SMB Daemon... Sep 12 09:12:49 myserver smbd[144330]: [2016/09/12 09:12:49.570609, 0] ../source3/rpc_server/lsasd.c:139(lsasd_sig_term_handler) Sep 12 09:12:49 myserver smbd[144330]: termination signal Sep 12 09:12:49 myserver systemd: Stopped Samba SMB Daemon. Sep 12 09:12:49 myserver audit: SERVICE_STOP pid=1 uid=0 auid=4294967295 ses=4294967295 msg='unit=smb comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success' Sep 12 09:12:49 myserver systemd: Stopped target PKI Tomcat Server. Sep 12 09:12:49 myserver systemd: Stopping PKI Tomcat Server pki-tomcat... Sep 12 09:12:49 myserver systemd: Stopping The Apache HTTP Server... Sep 12 09:12:49 myserver server: Java virtual machine used: /usr/share/java-utils/java-wrapper Sep 12 09:12:49 myserver server: classpath used: /usr/share/tomcat/bin/bootstrap.jar:/usr/share/tomcat/bin/tomcat-juli.jar:/usr/lib/java/commons-daemon.jar Sep 12 09:12:49 myserver server: main class used: org.apache.catalina.startup.Bootstrap Sep 12 09:12:49 myserver server: flags used: -DRESTEASY_LIB=/usr/share/java/resteasy -Djava.library.path=/usr/lib64/nuxwdog-jni Sep 12 09:12:49 myserver server: options used: -Dcatalina.base=/var/lib/pki/pki-tomcat -Dcatalina.home=/usr/share/tomcat -Djava.endorsed.dirs= -Djava.io.tmpdir=/var/lib/pki/pki-tomcat/temp -Djava.util.logging.config.file=/var/lib/pki/pki-to mcat/conf/logging.properties -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager Sep 12 09:12:49 myserver server: arguments used: stop Sep 12 09:12:49 myserver server: Sep 12, 2016 9:12:49 AM org.apache.catalina.startup.ClassLoaderFactory validateFile Sep 12 09:12:49 myserver server: WARNING: Problem with JAR file [/usr/share/pki/server/common/lib/symkey.jar], exists: [false], canRead: [false] Sep 12 09:12:49 myserver server: Sep 12, 2016 9:12:49 AM org.apache.catalina.core.StandardServer await Sep 12 09:12:49 myserver server: INFO: A valid shutdown command was received via the shutdown port. Stopping the Server instance. Sep 12 09:12:49 myserver server: PKIListener: org.apache.catalina.core.StandardServer[before_stop] Sep 12 09:12:49 myserver server: PKIListener: org.apache.catalina.core.StandardServer[stop] Sep 12 09:12:49 myserver server: PKIListener: org.apache.catalina.core.StandardServer[configure_stop] Sep 12 09:12:49 myserver server: Sep 12, 2016 9:12:49 AM org.apache.coyote.AbstractProtocol pause Sep 12 09:12:49 myserver server: INFO: Pausing ProtocolHandler ["http-nio-8080"] Sep 12 09:12:50 myserver systemd: Stopped PKI Tomcat Server pki-tomcat. Sep 12 09:12:50 myserver audit: SERVICE_STOP pid=1 uid=0 auid=4294967295 ses=4294967295 msg='unit=pki-tomcatd at pki-tomcat comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success' Sep 12 09:12:50 myserver systemd: Stopped The Apache HTTP Server. Sep 12 09:12:50 myserver audit: SERVICE_STOP pid=1 uid=0 auid=4294967295 ses=4294967295 msg='unit=httpd comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success' Sep 12 09:12:50 myserver systemd: Stopping IPA memcached daemon, increases IPA server performance... Sep 12 09:12:50 myserver audit: SERVICE_STOP pid=1 uid=0 auid=4294967295 ses=4294967295 msg='unit=ipa_memcached comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success' Sep 12 09:12:50 myserver systemd: Stopping Berkeley Internet Name Domain (DNS) with native PKCS#11... Sep 12 09:12:50 myserver named-pkcs11[143981]: received control channel command 'stop' Sep 12 09:12:50 myserver named-pkcs11[143981]: shutting down: flushing changes Sep 12 09:12:50 myserver named-pkcs11[143981]: stopping command channel on 127.0.0.1#953 Sep 12 09:12:50 myserver named-pkcs11[143981]: stopping command channel on ::1#953 Sep 12 09:12:50 myserver named-pkcs11[143981]: zone 21.0.10.in-addr.arpa/IN: shutting down Sep 12 09:12:50 myserver named-pkcs11[143981]: zone 1.168.192.in-addr.arpa/IN: shutting down Sep 12 09:12:50 myserver named-pkcs11[143981]: zone chloe/IN: shutting down Sep 12 09:12:50 myserver named-pkcs11[143981]: zone mydomain/IN: shutting down Sep 12 09:12:50 myserver named-pkcs11[143981]: no longer listening on ::#53 Sep 12 09:12:50 myserver named-pkcs11[143981]: no longer listening on 127.0.0.1#53 Sep 12 09:12:50 myserver named-pkcs11[143981]: no longer listening on 10.0.21.18#53 Sep 12 09:12:50 myserver named-pkcs11[143981]: exiting Sep 12 09:12:50 myserver systemd: Stopped Berkeley Internet Name Domain (DNS) with native PKCS#11. Sep 12 09:12:50 myserver audit: SERVICE_STOP pid=1 uid=0 auid=4294967295 ses=4294967295 msg='unit=named-pkcs11 comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success' Sep 12 09:12:50 myserver systemd: Stopping Kerberos 5 Password-changing and Administration... Sep 12 09:12:50 myserver systemd: kadmin.service: Main process exited, code=exited, status=2/INVALIDARGUMENT Sep 12 09:12:50 myserver systemd: Stopped Kerberos 5 Password-changing and Administration. Sep 12 09:12:50 myserver audit: SERVICE_STOP pid=1 uid=0 auid=4294967295 ses=4294967295 msg='unit=kadmin comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=failed' Sep 12 09:12:50 myserver systemd: kadmin.service: Unit entered failed state. Sep 12 09:12:50 myserver systemd: kadmin.service: Failed with result 'exit-code'. Sep 12 09:12:50 myserver systemd: Stopping Kerberos 5 KDC... Sep 12 09:12:50 myserver systemd: Stopped Kerberos 5 KDC. Sep 12 09:12:50 myserver audit: SERVICE_STOP pid=1 uid=0 auid=4294967295 ses=4294967295 msg='unit=krb5kdc comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success' Sep 12 09:12:50 myserver systemd: Stopping 389 Directory Server MYDOMAIN.... Sep 12 09:12:51 myserver ns-slapd: [12/Sep/2016:09:12:51.194043091 +0200] slapd shutting down - signaling mydomaintion threads - op stack size 16 max work q size 8 max work q stack size 8 Sep 12 09:12:51 myserver ns-slapd: [12/Sep/2016:09:12:51.194861445 +0200] slapd shutting down - waiting for 17 threads to terminate Sep 12 09:12:51 myserver ns-slapd: [12/Sep/2016:09:12:51.202313116 +0200] slapd shutting down - closing down internal subsystems and plugins Sep 12 09:12:52 myserver ns-slapd: [12/Sep/2016:09:12:52.600526800 +0200] Waiting for 4 database threads to stop Sep 12 09:12:52 myserver ns-slapd: [12/Sep/2016:09:12:52.744983340 +0200] All database threads now stopped Sep 12 09:12:52 myserver ns-slapd: [12/Sep/2016:09:12:52.756016065 +0200] slapd shutting down - freed 8 work q stack objects - freed 16 op stack objects Sep 12 09:12:53 myserver ns-slapd: [12/Sep/2016:09:12:53.183376890 +0200] slapd stopped. Sep 12 09:12:53 myserver systemd: Stopped 389 Directory Server MYDOMAIN.. Sep 12 09:12:53 myserver audit: SERVICE_STOP pid=1 uid=0 auid=4294967295 ses=4294967295 msg='unit=dirsrv at MYDOMAIN comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success' Sep 12 09:12:53 myserver ipactl: ipa: INFO: The ipactl command was successful Sep 12 09:12:53 myserver ipactl: Stopping ipa-dnskeysyncd Service Sep 12 09:12:53 myserver ipactl: Stopping ipa-otpd Service Sep 12 09:12:53 myserver ipactl: Stopping winbind Service Sep 12 09:12:53 myserver ipactl: Stopping smb Service Sep 12 09:12:53 myserver ipactl: Stopping pki-tomcatd Service Sep 12 09:12:53 myserver ipactl: Stopping httpd Service Sep 12 09:12:53 myserver ipactl: Stopping ipa_memcached Service Sep 12 09:12:53 myserver ipactl: Stopping named Service Sep 12 09:12:53 myserver ipactl: Stopping kadmin Service Sep 12 09:12:53 myserver ipactl: Stopping krb5kdc Service Sep 12 09:12:53 myserver ipactl: Stopping Directory Service Sep 12 09:12:53 myserver systemd: Stopped Identity, Policy, Audit. Sep 12 09:12:53 myserver audit: SERVICE_STOP pid=1 uid=0 auid=4294967295 ses=4294967295 msg='unit=ipa comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success' Sep 12 09:12:53 myserver systemd: Starting Identity, Policy, Audit... Sep 12 09:12:54 myserver systemd: Starting 389 Directory Server MYDOMAIN.... Sep 12 09:12:54 myserver ns-slapd: [12/Sep/2016:09:12:54.592757197 +0200] SSL alert: Sending pin request to SVRCore. You may need to run systemd-tty-ask-password-agent to provide the password. Sep 12 09:12:54 myserver ns-slapd: [12/Sep/2016:09:12:54.602519638 +0200] SSL alert: Security Initialization: Enabling default cipher set. Sep 12 09:12:54 myserver ns-slapd: [12/Sep/2016:09:12:54.608847323 +0200] SSL alert: Configured NSS Ciphers Sep 12 09:12:54 myserver ns-slapd: [12/Sep/2016:09:12:54.616426411 +0200] SSL alert: #011TLS_ECDHE_PSK_WITH_AES_128_GCM_SHA256: enabled Sep 12 09:12:54 myserver ns-slapd: [12/Sep/2016:09:12:54.624478653 +0200] SSL alert: #011TLS_ECDHE_PSK_WITH_CHACHA20_POLY1305_SHA256: enabled Sep 12 09:12:54 myserver ns-slapd: [12/Sep/2016:09:12:54.632476241 +0200] SSL alert: #011TLS_ECDHE_PSK_WITH_AES_256_GCM_SHA384: enabled Sep 12 09:12:54 myserver ns-slapd: [12/Sep/2016:09:12:54.640480176 +0200] SSL alert: #011TLS_DHE_PSK_WITH_AES_128_GCM_SHA256: enabled Sep 12 09:12:54 myserver ns-slapd: [12/Sep/2016:09:12:54.648468620 +0200] SSL alert: #011TLS_DHE_PSK_WITH_CHACHA20_POLY1305_SHA256: enabled Sep 12 09:12:54 myserver ns-slapd: [12/Sep/2016:09:12:54.656473213 +0200] SSL alert: #011TLS_DHE_PSK_WITH_AES_256_GCM_SHA384: enabled Sep 12 09:12:54 myserver ns-slapd: [12/Sep/2016:09:12:54.664506071 +0200] SSL alert: #011TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256: enabled Sep 12 09:12:54 myserver ns-slapd: [12/Sep/2016:09:12:54.672472469 +0200] SSL alert: #011TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256: enabled Sep 12 09:12:54 myserver ns-slapd: [12/Sep/2016:09:12:54.680470033 +0200] SSL alert: #011TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305_SHA256: enabled Sep 12 09:12:54 myserver ns-slapd: [12/Sep/2016:09:12:54.688418077 +0200] SSL alert: #011TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305_SHA256: enabled Sep 12 09:12:54 myserver ns-slapd: [12/Sep/2016:09:12:54.696430101 +0200] SSL alert: #011TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA: enabled Sep 12 09:12:54 myserver ns-slapd: [12/Sep/2016:09:12:54.704438403 +0200] SSL alert: #011TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA: enabled Sep 12 09:12:54 myserver ns-slapd: [12/Sep/2016:09:12:54.712465402 +0200] SSL alert: #011TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA: enabled Sep 12 09:12:54 myserver ns-slapd: [12/Sep/2016:09:12:54.718472149 +0200] SSL alert: #011TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256: enabled Sep 12 09:12:54 myserver ns-slapd: [12/Sep/2016:09:12:54.726420679 +0200] SSL alert: #011TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256: enabled Sep 12 09:12:54 myserver ns-slapd: [12/Sep/2016:09:12:54.734335342 +0200] SSL alert: #011TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA: enabled Sep 12 09:12:54 myserver ns-slapd: [12/Sep/2016:09:12:54.742345928 +0200] SSL alert: #011TLS_DHE_RSA_WITH_AES_128_GCM_SHA256: enabled Sep 12 09:12:54 myserver ns-slapd: [12/Sep/2016:09:12:54.750379100 +0200] SSL alert: #011TLS_DHE_RSA_WITH_CHACHA20_POLY1305_SHA256: enabled Sep 12 09:12:54 myserver ns-slapd: [12/Sep/2016:09:12:54.758377248 +0200] SSL alert: #011TLS_DHE_RSA_WITH_AES_128_CBC_SHA: enabled Sep 12 09:12:54 myserver ns-slapd: [12/Sep/2016:09:12:54.766397790 +0200] SSL alert: #011TLS_DHE_DSS_WITH_AES_128_CBC_SHA: enabled Sep 12 09:12:54 myserver ns-slapd: [12/Sep/2016:09:12:54.774407368 +0200] SSL alert: #011TLS_DHE_RSA_WITH_AES_128_CBC_SHA256: enabled Sep 12 09:12:54 myserver ns-slapd: [12/Sep/2016:09:12:54.782279403 +0200] SSL alert: #011TLS_DHE_RSA_WITH_AES_256_CBC_SHA: enabled Sep 12 09:12:54 myserver ns-slapd: [12/Sep/2016:09:12:54.790400099 +0200] SSL alert: #011TLS_DHE_DSS_WITH_AES_256_CBC_SHA: enabled Sep 12 09:12:54 myserver ns-slapd: [12/Sep/2016:09:12:54.798430494 +0200] SSL alert: #011TLS_DHE_RSA_WITH_AES_256_CBC_SHA256: enabled Sep 12 09:12:54 myserver ns-slapd: [12/Sep/2016:09:12:54.806315318 +0200] SSL alert: #011TLS_RSA_WITH_AES_128_GCM_SHA256: enabled Sep 12 09:12:54 myserver ns-slapd: [12/Sep/2016:09:12:54.814414686 +0200] SSL alert: #011TLS_RSA_WITH_AES_128_CBC_SHA: enabled Sep 12 09:12:54 myserver ns-slapd: [12/Sep/2016:09:12:54.822359169 +0200] SSL alert: #011TLS_RSA_WITH_AES_128_CBC_SHA256: enabled Sep 12 09:12:54 myserver ns-slapd: [12/Sep/2016:09:12:54.830486922 +0200] SSL alert: #011TLS_RSA_WITH_AES_256_CBC_SHA: enabled Sep 12 09:12:54 myserver ns-slapd: [12/Sep/2016:09:12:54.838464663 +0200] SSL alert: #011TLS_RSA_WITH_AES_256_CBC_SHA256: enabled Sep 12 09:12:54 myserver ns-slapd: [12/Sep/2016:09:12:54.883946405 +0200] SSL Initialization - Configured SSL version range: min: TLS1.0, max: TLS1.2 Sep 12 09:12:54 myserver ns-slapd: [12/Sep/2016:09:12:54.884472271 +0200] 389-Directory/1.3.5.13 B2016.221.207 starting up Sep 12 09:12:54 myserver ns-slapd: [12/Sep/2016:09:12:54.925427594 +0200] WARNING: userRoot: entry cache size 512000 B is less than db size 1228800 B; We recommend to increase the entry cache size nsslapd-cachememsize. Sep 12 09:12:54 myserver ns-slapd: [12/Sep/2016:09:12:54.925838890 +0200] WARNING: ipaca: entry cache size 512000 B is less than db size 1097728 B; We recommend to increase the entry cache size nsslapd-cachememsize. Sep 12 09:12:54 myserver ns-slapd: [12/Sep/2016:09:12:54.933434740 +0200] WARNING: changelog: entry cache size 512000 B is less than db size 19890176 B; We recommend to increase the entry cache size nsslapd-cachememsize. Sep 12 09:12:54 myserver ns-slapd: [12/Sep/2016:09:12:54.995273968 +0200] schema-compat-plugin - scheduled schema-compat-plugin tree scan in about 5 seconds after the server startup! Sep 12 09:12:55 myserver ns-slapd: [12/Sep/2016:09:12:55.017457582 +0200] NSACLPlugin - The ACL target cn=groups,cn=compat,dc=mydomain does not exist Sep 12 09:12:55 myserver ns-slapd: [12/Sep/2016:09:12:55.018095367 +0200] NSACLPlugin - The ACL target cn=computers,cn=compat,dc=mydomain does not exist Sep 12 09:12:55 myserver ns-slapd: [12/Sep/2016:09:12:55.025580223 +0200] NSACLPlugin - The ACL target cn=ng,cn=compat,dc=mydomain does not exist Sep 12 09:12:55 myserver ns-slapd: [12/Sep/2016:09:12:55.033692923 +0200] NSACLPlugin - The ACL target ou=sudoers,dc=mydomain does not exist Sep 12 09:12:55 myserver ns-slapd: [12/Sep/2016:09:12:55.041664186 +0200] NSACLPlugin - The ACL target cn=users,cn=compat,dc=mydomain does not exist Sep 12 09:12:55 myserver ns-slapd: [12/Sep/2016:09:12:55.050306168 +0200] NSACLPlugin - The ACL target cn=vaults,cn=kra,dc=mydomain does not exist Sep 12 09:12:55 myserver ns-slapd: [12/Sep/2016:09:12:55.057636069 +0200] NSACLPlugin - The ACL target cn=vaults,cn=kra,dc=mydomain does not exist Sep 12 09:12:55 myserver ns-slapd: [12/Sep/2016:09:12:55.065649429 +0200] NSACLPlugin - The ACL target cn=vaults,cn=kra,dc=mydomain does not exist Sep 12 09:12:55 myserver ns-slapd: [12/Sep/2016:09:12:55.073609042 +0200] NSACLPlugin - The ACL target cn=vaults,cn=kra,dc=mydomain does not exist Sep 12 09:12:55 myserver ns-slapd: [12/Sep/2016:09:12:55.081615966 +0200] NSACLPlugin - The ACL target cn=vaults,cn=kra,dc=mydomain does not exist Sep 12 09:12:55 myserver ns-slapd: [12/Sep/2016:09:12:55.089606661 +0200] NSACLPlugin - The ACL target cn=vaults,cn=kra,dc=mydomain does not exist Sep 12 09:12:55 myserver ns-slapd: [12/Sep/2016:09:12:55.097586799 +0200] NSACLPlugin - The ACL target cn=vaults,cn=kra,dc=mydomain does not exist Sep 12 09:12:55 myserver ns-slapd: [12/Sep/2016:09:12:55.105570060 +0200] NSACLPlugin - The ACL target cn=vaults,cn=kra,dc=mydomain does not exist Sep 12 09:12:55 myserver ns-slapd: [12/Sep/2016:09:12:55.113575949 +0200] NSACLPlugin - The ACL target cn=vaults,cn=kra,dc=mydomain does not exist Sep 12 09:12:55 myserver ns-slapd: [12/Sep/2016:09:12:55.121549029 +0200] NSACLPlugin - The ACL target cn=vaults,cn=kra,dc=mydomain does not exist Sep 12 09:12:55 myserver ns-slapd: [12/Sep/2016:09:12:55.129558870 +0200] NSACLPlugin - The ACL target cn=vaults,cn=kra,dc=mydomain does not exist Sep 12 09:12:55 myserver ns-slapd: [12/Sep/2016:09:12:55.146394825 +0200] NSACLPlugin - The ACL target cn=casigningcert cert-pki-ca,cn=ca_renewal,cn=ipa,cn=etc,dc=mydomain does not exist Sep 12 09:12:55 myserver ns-slapd: [12/Sep/2016:09:12:55.146930568 +0200] NSACLPlugin - The ACL target cn=casigningcert cert-pki-ca,cn=ca_renewal,cn=ipa,cn=etc,dc=mydomain does not exist Sep 12 09:12:55 myserver ns-slapd: [12/Sep/2016:09:12:55.280021623 +0200] NSACLPlugin - The ACL target cn=automember rebuild membership,cn=tasks,cn=config does not exist Sep 12 09:12:55 myserver ns-slapd: [12/Sep/2016:09:12:55.287423440 +0200] Skipping CoS Definition cn=Password Policy,cn=accounts,dc=mydomain--no CoS Templates found, which should be added before the CoS Definition. Sep 12 09:12:55 myserver ns-slapd: [12/Sep/2016:09:12:55.339402668 +0200] set_krb5_creds - Could not get initial credentials for principal [ldap/myserver.mydomain at MYDOMAIN] in keytab [FILE:/etc/dirsrv/ds.keytab]: -1765328228 (Cannot contact any KDC for requested realm) Sep 12 09:12:55 myserver ns-slapd: [12/Sep/2016:09:12:55.343018668 +0200] NSMMReplicationPlugin - agmt="cn=meTomyreplica.mydomain" (myreplica:389): Replication bind with GSSAPI auth failed: LDAP error -1 (Can't contact LDAP server) () Sep 12 09:12:55 myserver ns-slapd: [12/Sep/2016:09:12:55.349037259 +0200] schema-compat-plugin - schema-compat-plugin tree scan will start in about 5 seconds! Sep 12 09:12:55 myserver ns-slapd: [12/Sep/2016:09:12:55.383652874 +0200] slapd started. Listening on All Interfaces port 389 for LDAP requests Sep 12 09:12:55 myserver ns-slapd: [12/Sep/2016:09:12:55.388408526 +0200] Listening on All Interfaces port 636 for LDAPS requests Sep 12 09:12:55 myserver ns-slapd: [12/Sep/2016:09:12:55.396415389 +0200] Listening on /var/run/slapd-MYDOMAIN.socket for LDAPI requests Sep 12 09:12:55 myserver systemd: Started 389 Directory Server MYDOMAIN.. Sep 12 09:12:55 myserver audit: SERVICE_START pid=1 uid=0 auid=4294967295 ses=4294967295 msg='unit=dirsrv at MYDOMAIN comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success' Sep 12 09:12:55 myserver systemd: Starting Kerberos 5 KDC... Sep 12 09:12:55 myserver systemd: Started Kerberos 5 KDC. Sep 12 09:12:55 myserver audit: SERVICE_START pid=1 uid=0 auid=4294967295 ses=4294967295 msg='unit=krb5kdc comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success' Sep 12 09:12:55 myserver systemd: Starting Kerberos 5 Password-changing and Administration... Sep 12 09:12:55 myserver systemd: Started Kerberos 5 Password-changing and Administration. Sep 12 09:12:55 myserver audit: SERVICE_START pid=1 uid=0 auid=4294967295 ses=4294967295 msg='unit=kadmin comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success' Sep 12 09:12:55 myserver systemd: Starting Generate rndc key for BIND (DNS)... Sep 12 09:12:55 myserver systemd: Started Generate rndc key for BIND (DNS). Sep 12 09:12:55 myserver audit: SERVICE_START pid=1 uid=0 auid=4294967295 ses=4294967295 msg='unit=named-setup-rndc comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success' Sep 12 09:12:55 myserver audit: SERVICE_STOP pid=1 uid=0 auid=4294967295 ses=4294967295 msg='unit=named-setup-rndc comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success' Sep 12 09:12:55 myserver systemd: Starting Berkeley Internet Name Domain (DNS) with native PKCS#11... Sep 12 09:12:55 myserver bash: zone localhost.localdomain/IN: loaded serial 0 Sep 12 09:12:55 myserver bash: zone localhost/IN: loaded serial 0 Sep 12 09:12:55 myserver bash: zone 1.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.ip6.arpa/IN: loaded serial 0 Sep 12 09:12:55 myserver bash: zone 1.0.0.127.in-addr.arpa/IN: loaded serial 0 Sep 12 09:12:55 myserver bash: zone 0.in-addr.arpa/IN: loaded serial 0 Sep 12 09:12:56 myserver named-pkcs11[145109]: starting BIND 9.10.4-P2-RedHat-9.10.4-1.P2.fc24 -u named Sep 12 09:12:56 myserver named-pkcs11[145109]: running on Linux x86_64 4.7.2-201.fc24.x86_64 #1 SMP Fri Aug 26 15:58:40 UTC 2016 Sep 12 09:12:56 myserver named-pkcs11[145109]: built with '--build=x86_64-redhat-linux-gnu' '--host=x86_64-redhat-linux-gnu' '--program-prefix=' '--disable-dependency-tracking' '--prefix=/usr' '--exec-prefix=/usr' '--bindir=/usr/bin' '--sbindir=/usr/sbin' '--sysconfdir=/etc' '--datadir=/usr/share' '--includedir=/usr/include' '--libdir=/usr/lib64' '--libexecdir=/usr/libexec' '--sharedstatedir=/var/lib' '--mandir=/usr/share/man' '--infodir=/usr/share/info' '--with-python=/usr/bin/python3' '--with-libtool' '--localstatedir=/var' '--enable-threads' '--enable-ipv6' '--enable-filter-aaaa' '--with-pic' '--disable-static' '--disable-openssl-version-check' '--includedir=/usr/include/bind9' '--with-tuning=large' '--with-geoip' '--enable-native-pkcs11' '--with-pkcs11=/usr/lib64/pkcs11/libsofthsm2.so' '--with-dlopen=yes' '--with-dlz-ldap=yes' '--with-dlz-postgres=yes' '--with-dlz-mysql=yes' '--with-dlz-filesystem=yes' '--with-dlz-bdb=yes' '--with-gssapi=yes' '--disable-isc-spnego' '--enable-fixed-rrset' '--with-docbook-xsl=/usr/share/sgml/docbook/xsl-stylesheets' '--enable-full-report' 'build_alias=x86_64-redhat-linux-gnu' 'host_alias=x86_64-redhat-linux-gnu' 'CFLAGS= -O2 -g -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong --param=ssp-buffer-size=4 -grecord-gcc-switches -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -m64 -mtune=generic' 'LDFLAGS=-Wl,-z,relro -specs=/usr/lib/rpm/redhat/redhat-hardened-ld' 'CPPFLAGS= -DDIG_SIGCHASE' Sep 12 09:12:56 myserver named-pkcs11[145109]: ---------------------------------------------------- Sep 12 09:12:56 myserver named-pkcs11[145109]: BIND 9 is maintained by Internet Systems Consortium, Sep 12 09:12:56 myserver named-pkcs11[145109]: Inc. (ISC), a non-profit 501(c)(3) public-benefit Sep 12 09:12:56 myserver named-pkcs11[145109]: corporation. Support and training for BIND 9 are Sep 12 09:12:56 myserver named-pkcs11[145109]: available at https://www.isc.org/support Sep 12 09:12:56 myserver named-pkcs11[145109]: ---------------------------------------------------- Sep 12 09:12:56 myserver named-pkcs11[145109]: adjusted limit on open files from 4096 to 1048576 Sep 12 09:12:56 myserver named-pkcs11[145109]: found 12 CPUs, using 12 worker threads Sep 12 09:12:56 myserver named-pkcs11[145109]: using 11 UDP listeners per interface Sep 12 09:12:56 myserver named-pkcs11[145109]: using up to 21000 sockets Sep 12 09:12:56 myserver named-pkcs11[145109]: loading configuration from '/etc/named.conf' Sep 12 09:12:56 myserver named-pkcs11[145109]: reading built-in trusted keys from file '/etc/named.iscdlv.key' Sep 12 09:12:56 myserver named-pkcs11[145109]: initializing GeoIP Country (IPv4) (type 1) DB Sep 12 09:12:56 myserver named-pkcs11[145109]: GEO-106FREE 20160705 Build 1 Copy Sep 12 09:12:56 myserver named-pkcs11[145109]: GeoIP Country (IPv6) (type 12) DB not available Sep 12 09:12:56 myserver named-pkcs11[145109]: GeoIP City (IPv4) (type 2) DB not available Sep 12 09:12:56 myserver named-pkcs11[145109]: GeoIP City (IPv4) (type 6) DB not available Sep 12 09:12:56 myserver named-pkcs11[145109]: GeoIP City (IPv6) (type 30) DB not available Sep 12 09:12:56 myserver named-pkcs11[145109]: GeoIP City (IPv6) (type 31) DB not available Sep 12 09:12:56 myserver named-pkcs11[145109]: GeoIP Region (type 3) DB not available Sep 12 09:12:56 myserver named-pkcs11[145109]: GeoIP Region (type 7) DB not available Sep 12 09:12:56 myserver named-pkcs11[145109]: GeoIP ISP (type 4) DB not available Sep 12 09:12:56 myserver named-pkcs11[145109]: GeoIP Org (type 5) DB not available Sep 12 09:12:56 myserver named-pkcs11[145109]: GeoIP AS (type 9) DB not available Sep 12 09:12:56 myserver named-pkcs11[145109]: GeoIP Domain (type 11) DB not available Sep 12 09:12:56 myserver named-pkcs11[145109]: GeoIP NetSpeed (type 10) DB not available Sep 12 09:12:56 myserver named-pkcs11[145109]: using default UDP/IPv4 port range: [32768, 60999] Sep 12 09:12:56 myserver named-pkcs11[145109]: using default UDP/IPv6 port range: [32768, 60999] Sep 12 09:12:56 myserver named-pkcs11[145109]: listening on IPv6 interfaces, port 53 Sep 12 09:12:56 myserver named-pkcs11[145109]: listening on IPv4 interface lo, 127.0.0.1#53 Sep 12 09:12:56 myserver named-pkcs11[145109]: listening on IPv4 interface eno1, 10.0.21.18#53 Sep 12 09:12:56 myserver named-pkcs11[145109]: generating session key for dynamic DNS Sep 12 09:12:56 myserver named-pkcs11[145109]: sizing zone task pool based on 6 zones Sep 12 09:12:56 myserver named-pkcs11[145109]: set up managed keys zone for view _default, file '/var/named/dynamic/managed-keys.bind' Sep 12 09:12:56 myserver named-pkcs11[145109]: bind-dyndb-ldap version 9.0 compiled at 09:01:48 May 27 2016, compiler 6.1.1 20160510 (Red Hat 6.1.1-2) Sep 12 09:12:56 myserver named-pkcs11[145109]: option 'serial_autoincrement' is not supported, ignoring Sep 12 09:12:56 myserver named-pkcs11[145109]: LDAP instance 'ipa' is being synchronized, please ignore message 'all zones loaded' Sep 12 09:12:56 myserver named-pkcs11[145109]: automatic empty zone: 10.IN-ADDR.ARPA Sep 12 09:12:56 myserver named-pkcs11[145109]: automatic empty zone: 16.172.IN-ADDR.ARPA Sep 12 09:12:56 myserver named-pkcs11[145109]: automatic empty zone: 109.100.IN-ADDR.ARPA Sep 12 09:12:56 myserver named-pkcs11[145109]: automatic empty zone: 110.100.IN-ADDR.ARPA Sep 12 09:12:56 myserver named-pkcs11[145109]: automatic empty zone: 111.100.IN-ADDR.ARPA Sep 12 09:12:56 myserver named-pkcs11[145109]: automatic empty zone: 112.100.IN-ADDR.ARPA Sep 12 09:12:56 myserver named-pkcs11[145109]: automatic empty zone: 113.100.IN-ADDR.ARPA Sep 12 09:12:56 myserver named-pkcs11[145109]: automatic empty zone: 114.100.IN-ADDR.ARPA Sep 12 09:12:56 myserver named-pkcs11[145109]: automatic empty zone: 115.100.IN-ADDR.ARPA Sep 12 09:12:56 myserver named-pkcs11[145109]: automatic empty zone: 116.100.IN-ADDR.ARPA Sep 12 09:12:56 myserver named-pkcs11[145109]: automatic empty zone: 117.100.IN-ADDR.ARPA Sep 12 09:12:56 myserver named-pkcs11[145109]: automatic empty zone: 118.100.IN-ADDR.ARPA Sep 12 09:12:56 myserver named-pkcs11[145109]: automatic empty zone: 119.100.IN-ADDR.ARPA Sep 12 09:12:56 myserver named-pkcs11[145109]: automatic empty zone: 120.100.IN-ADDR.ARPA Sep 12 09:12:56 myserver named-pkcs11[145109]: automatic empty zone: 121.100.IN-ADDR.ARPA Sep 12 09:12:56 myserver named-pkcs11[145109]: automatic empty zone: 122.100.IN-ADDR.ARPA Sep 12 09:12:56 myserver named-pkcs11[145109]: automatic empty zone: 123.100.IN-ADDR.ARPA Sep 12 09:12:56 myserver named-pkcs11[145109]: automatic empty zone: 124.100.IN-ADDR.ARPA Sep 12 09:12:56 myserver named-pkcs11[145109]: automatic empty zone: 125.100.IN-ADDR.ARPA Sep 12 09:12:56 myserver named-pkcs11[145109]: automatic empty zone: 126.100.IN-ADDR.ARPA Sep 12 09:12:56 myserver named-pkcs11[145109]: automatic empty zone: 127.100.IN-ADDR.ARPA Sep 12 09:12:56 myserver named-pkcs11[145109]: automatic empty zone: 127.IN-ADDR.ARPA Sep 12 09:12:56 myserver named-pkcs11[145109]: automatic empty zone: 254.169.IN-ADDR.ARPA Sep 12 09:12:56 myserver named-pkcs11[145109]: automatic empty zone: 2.0.192.IN-ADDR.ARPA Sep 12 09:12:56 myserver named-pkcs11[145109]: automatic empty zone: 100.51.198.IN-ADDR.ARPA Sep 12 09:12:56 myserver named-pkcs11[145109]: automatic empty zone: 113.0.203.IN-ADDR.ARPA Sep 12 09:12:56 myserver named-pkcs11[145109]: automatic empty zone: 255.255.255.255.IN-ADDR.ARPA Sep 12 09:12:56 myserver named-pkcs11[145109]: automatic empty zone: 0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.IP6.ARPA Sep 12 09:12:56 myserver named-pkcs11[145109]: automatic empty zone: D.F.IP6.ARPA Sep 12 09:12:56 myserver named-pkcs11[145109]: automatic empty zone: 8.E.F.IP6.ARPA Sep 12 09:12:56 myserver named-pkcs11[145109]: automatic empty zone: 9.E.F.IP6.ARPA Sep 12 09:12:56 myserver named-pkcs11[145109]: automatic empty zone: A.E.F.IP6.ARPA Sep 12 09:12:56 myserver named-pkcs11[145109]: automatic empty zone: B.E.F.IP6.ARPA Sep 12 09:12:56 myserver named-pkcs11[145109]: automatic empty zone: 8.B.D.0.1.0.0.2.IP6.ARPA Sep 12 09:12:56 myserver named-pkcs11[145109]: automatic empty zone: EMPTY.AS112.ARPA Sep 12 09:12:56 myserver named-pkcs11[145109]: configuring command channel from '/etc/rndc.key' Sep 12 09:12:56 myserver named-pkcs11[145109]: command channel listening on 127.0.0.1#953 Sep 12 09:12:56 myserver named-pkcs11[145109]: configuring command channel from '/etc/rndc.key' Sep 12 09:12:56 myserver named-pkcs11[145109]: command channel listening on ::1#953 Sep 12 09:12:56 myserver named-pkcs11[145109]: managed-keys-zone: journal file is out of date: removing journal file Sep 12 09:12:56 myserver named-pkcs11[145109]: managed-keys-zone: loaded serial 482 Sep 12 09:12:56 myserver named-pkcs11[145109]: ignoring inherited 'forward first;' for zone '.' - did you want 'forward only;' to override automatic empty zone '10.IN-ADDR.ARPA'? Sep 12 09:12:56 myserver named-pkcs11[145109]: ignoring inherited 'forward first;' for zone '.' - did you want 'forward only;' to override automatic empty zone '16.172.IN-ADDR.ARPA'? Sep 12 09:12:56 myserver named-pkcs11[145109]: ignoring inherited 'forward first;' for zone '.' - did you want 'forward only;' to override automatic empty zone '17.172.IN-ADDR.ARPA'? [...] Sep 12 09:12:56 myserver named-pkcs11[145109]: ignoring inherited 'forward first;' for zone '.' - did you want 'forward only;' to override automatic empty zone '100.51.198.IN-ADDR.ARPA'? Sep 12 09:12:56 myserver named-pkcs11[145109]: ignoring inherited 'forward first;' for zone '.' - did you want 'forward only;' to override automatic empty zone '113.0.203.IN-ADDR.ARPA'? Sep 12 09:12:56 myserver named-pkcs11[145109]: ignoring inherited 'forward first;' for zone '.' - did you want 'forward only;' to override automatic empty zone '255.255.255.255.IN-ADDR.ARPA'? Sep 12 09:12:56 myserver named-pkcs11[145109]: ignoring inherited 'forward first;' for zone '.' - did you want 'forward only;' to override automatic empty zone '0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.IP6.ARPA'? Sep 12 09:12:56 myserver named-pkcs11[145109]: ignoring inherited 'forward first;' for zone '.' - did you want 'forward only;' to override automatic empty zone 'D.F.IP6.ARPA'? Sep 12 09:12:56 myserver named-pkcs11[145109]: ignoring inherited 'forward first;' for zone '.' - did you want 'forward only;' to override automatic empty zone '8.E.F.IP6.ARPA'? Sep 12 09:12:56 myserver named-pkcs11[145109]: ignoring inherited 'forward first;' for zone '.' - did you want 'forward only;' to override automatic empty zone '9.E.F.IP6.ARPA'? Sep 12 09:12:56 myserver named-pkcs11[145109]: ignoring inherited 'forward first;' for zone '.' - did you want 'forward only;' to override automatic empty zone 'A.E.F.IP6.ARPA'? Sep 12 09:12:56 myserver named-pkcs11[145109]: ignoring inherited 'forward first;' for zone '.' - did you want 'forward only;' to override automatic empty zone 'B.E.F.IP6.ARPA'? Sep 12 09:12:56 myserver named-pkcs11[145109]: ignoring inherited 'forward first;' for zone '.' - did you want 'forward only;' to override automatic empty zone '8.B.D.0.1.0.0.2.IP6.ARPA'? Sep 12 09:12:56 myserver named-pkcs11[145109]: ignoring inherited 'forward first;' for zone '.' - did you want 'forward only;' to override automatic empty zone 'EMPTY.AS112.ARPA'? Sep 12 09:12:56 myserver named-pkcs11[145109]: zone localhost/IN: loaded serial 0 Sep 12 09:12:56 myserver named-pkcs11[145109]: zone 1.0.0.127.in-addr.arpa/IN: loaded serial 0 Sep 12 09:12:56 myserver named-pkcs11[145109]: all zones loaded Sep 12 09:12:56 myserver named-pkcs11[145109]: running Sep 12 09:12:56 myserver systemd: Starting IPA memcached daemon, increases IPA server performance... Sep 12 09:12:56 myserver systemd: ipa_memcached.service: PID file /var/run/ipa_memcached/ipa_memcached.pid not readable (yet?) after start: No such file or directory Sep 12 09:12:56 myserver systemd: Started IPA memcached daemon, increases IPA server performance. Sep 12 09:12:56 myserver audit: SERVICE_START pid=1 uid=0 auid=4294967295 ses=4294967295 msg='unit=ipa_memcached comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success' Sep 12 09:12:56 myserver systemd: Starting The Apache HTTP Server... Sep 12 09:12:56 myserver named-pkcs11[145109]: zone 21.0.10.in-addr.arpa/IN: loaded serial 1473664376 Sep 12 09:12:56 myserver named-pkcs11[145109]: zone 21.0.10.in-addr.arpa/IN: sending notifies (serial 1473664376) Sep 12 09:12:56 myserver named-pkcs11[145109]: zone 1.168.192.in-addr.arpa/IN: loaded serial 1473664376 Sep 12 09:12:56 myserver named-pkcs11[145109]: zone 1.168.192.in-addr.arpa/IN: sending notifies (serial 1473664376) Sep 12 09:12:56 myserver named-pkcs11[145109]: zone chloe/IN: loaded serial 1473664376 Sep 12 09:12:56 myserver named-pkcs11[145109]: zone chloe/IN: sending notifies (serial 1473664376) Sep 12 09:12:56 myserver named-pkcs11[145109]: zone mydomain/IN: loaded serial 1473664376 Sep 12 09:12:56 myserver named-pkcs11[145109]: zone mydomain/IN: sending notifies (serial 1473664376) Sep 12 09:12:56 myserver named-pkcs11[145109]: 4 master zones from LDAP instance 'ipa' loaded (4 zones defined, 0 inactive, 0 failed to load) Sep 12 09:12:56 myserver named-pkcs11[145109]: checkhints: unable to get root NS rrset from cache: not found Sep 12 09:12:57 myserver ipa-httpd-kdcproxy: ipa : INFO KDC proxy enabled Sep 12 09:12:57 myserver systemd: Started The Apache HTTP Server. Sep 12 09:12:57 myserver audit: SERVICE_START pid=1 uid=0 auid=4294967295 ses=4294967295 msg='unit=httpd comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success' Sep 12 09:12:57 myserver systemd: Starting PKI Tomcat Server pki-tomcat... Sep 12 09:12:57 myserver systemd: Started PKI Tomcat Server pki-tomcat. Sep 12 09:12:57 myserver audit: SERVICE_START pid=1 uid=0 auid=4294967295 ses=4294967295 msg='unit=pki-tomcatd at pki-tomcat comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success' Sep 12 09:12:57 myserver systemd: Reached target PKI Tomcat Server. Sep 12 09:12:57 myserver server: Java virtual machine used: /usr/share/java-utils/java-wrapper Sep 12 09:12:57 myserver server: classpath used: /usr/share/tomcat/bin/bootstrap.jar:/usr/share/tomcat/bin/tomcat-juli.jar:/usr/lib/java/commons-daemon.jar Sep 12 09:12:57 myserver server: main class used: org.apache.catalina.startup.Bootstrap Sep 12 09:12:57 myserver server: flags used: -DRESTEASY_LIB=/usr/share/java/resteasy -Djava.library.path=/usr/lib64/nuxwdog-jni Sep 12 09:12:57 myserver server: options used: -Dcatalina.base=/var/lib/pki/pki-tomcat -Dcatalina.home=/usr/share/tomcat -Djava.endorsed.dirs= -Djava.io.tmpdir=/var/lib/pki/pki-tomcat/temp -Djava.util.logging.config.file=/var/lib/pki/pki-tomcat/conf/logging.properties -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager -Djava.security.manager -Djava.security.policy==/var/lib/pki/pki-tomcat/conf/catalina.policy Sep 12 09:12:57 myserver server: arguments used: start Sep 12 09:12:57 myserver server: Sep 12, 2016 9:12:57 AM org.apache.catalina.startup.ClassLoaderFactory validateFile Sep 12 09:12:57 myserver server: WARNING: Problem with JAR file [/usr/share/pki/server/common/lib/symkey.jar], exists: [false], canRead: [false] Sep 12 09:12:58 myserver server: Sep 12, 2016 9:12:58 AM org.apache.catalina.startup.SetAllPropertiesRule begin Sep 12 09:12:58 myserver server: WARNING: [SetAllPropertiesRule]{Server/Service/Connector} Setting property 'enableOCSP' to 'false' did not find a matching property. Sep 12 09:12:58 myserver server: Sep 12, 2016 9:12:58 AM org.apache.catalina.startup.SetAllPropertiesRule begin Sep 12 09:12:58 myserver server: WARNING: [SetAllPropertiesRule]{Server/Service/Connector} Setting property 'ocspResponderURL' to 'http://myserver.mydomain:9080/ca/ocsp' did not find a matching property. Sep 12 09:12:58 myserver server: Sep 12, 2016 9:12:58 AM org.apache.catalina.startup.SetAllPropertiesRule begin Sep 12 09:12:58 myserver server: WARNING: [SetAllPropertiesRule]{Server/Service/Connector} Setting property 'ocspResponderCertNickname' to 'ocspSigningCert cert-pki-ca' did not find a matching property. Sep 12 09:12:58 myserver server: Sep 12, 2016 9:12:58 AM org.apache.catalina.startup.SetAllPropertiesRule begin Sep 12 09:12:58 myserver server: WARNING: [SetAllPropertiesRule]{Server/Service/Connector} Setting property 'ocspCacheSize' to '1000' did not find a matching property. Sep 12 09:12:58 myserver server: Sep 12, 2016 9:12:58 AM org.apache.catalina.startup.SetAllPropertiesRule begin Sep 12 09:12:58 myserver server: WARNING: [SetAllPropertiesRule]{Server/Service/Connector} Setting property 'ocspMinCacheEntryDuration' to '60' did not find a matching property. Sep 12 09:12:58 myserver server: Sep 12, 2016 9:12:58 AM org.apache.catalina.startup.SetAllPropertiesRule begin Sep 12 09:12:58 myserver server: WARNING: [SetAllPropertiesRule]{Server/Service/Connector} Setting property 'ocspMaxCacheEntryDuration' to '120' did not find a matching property. Sep 12 09:12:58 myserver server: Sep 12, 2016 9:12:58 AM org.apache.catalina.startup.SetAllPropertiesRule begin Sep 12 09:12:58 myserver server: WARNING: [SetAllPropertiesRule]{Server/Service/Connector} Setting property 'ocspTimeout' to '10' did not find a matching property. Sep 12 09:12:58 myserver server: Sep 12, 2016 9:12:58 AM org.apache.catalina.startup.SetAllPropertiesRule begin Sep 12 09:12:58 myserver server: WARNING: [SetAllPropertiesRule]{Server/Service/Connector} Setting property 'strictCiphers' to 'true' did not find a matching property. Sep 12 09:12:58 myserver server: Sep 12, 2016 9:12:58 AM org.apache.catalina.startup.SetAllPropertiesRule begin Sep 12 09:12:58 myserver server: WARNING: [SetAllPropertiesRule]{Server/Service/Connector} Setting property 'sslOptions' to 'ssl2=false,ssl3=false,tls=true' did not find a matching property. Sep 12 09:12:58 myserver server: Sep 12, 2016 9:12:58 AM org.apache.catalina.startup.SetAllPropertiesRule begin Sep 12 09:12:58 myserver server: WARNING: [SetAllPropertiesRule]{Server/Service/Connector} Setting property 'ssl2Ciphers' to '-SSL2_RC4_128_WITH_MD5,-SSL2_RC4_128_EXPORT40_WITH_MD5,-SSL2_RC2_128_CBC_WITH_MD5,-SSL2_RC2_128_CBC_EXPORT40_WITH_MD5,-SSL2_DES_64_CBC_WITH_MD5,-SSL2_DES_192_EDE3_CBC_WITH_MD5' did not find a matching property. Sep 12 09:12:58 myserver server: Sep 12, 2016 9:12:58 AM org.apache.catalina.startup.SetAllPropertiesRule begin Sep 12 09:12:58 myserver server: WARNING: [SetAllPropertiesRule]{Server/Service/Connector} Setting property 'ssl3Ciphers' to '-SSL3_FORTEZZA_DMS_WITH_NULL_SHA,-SSL3_FORTEZZA_DMS_WITH_RC4_128_SHA,+SSL3_RSA_WITH_RC4_128_SHA,-SSL3_RSA_EXPORT_WITH_RC4_40_MD5,+SSL3_RSA_WITH_3DES_EDE_CBC_SHA,-SSL3_RSA_WITH_DES_CBC_SHA,-SSL3_RSA_EXPORT_WITH_RC2_CBC_40_MD5,-SSL3_FORTEZZA_DMS_WITH_FORTEZZA_CBC_SHA,-SSL_RSA_FIPS_WITH_DES_CBC_SHA,+SSL_RSA_FIPS_WITH_3DES_EDE_CBC_SHA,-SSL3_RSA_WITH_NULL_MD5,-TLS_RSA_EXPORT1024_WITH_RC4_56_SHA,-TLS_ECDH_ECDSA_WITH_AES_256_CBC_SHA,+TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA' did not find a matching property. Sep 12 09:12:58 myserver server: Sep 12, 2016 9:12:58 AM org.apache.catalina.startup.SetAllPropertiesRule begin Sep 12 09:12:58 myserver server: WARNING: [SetAllPropertiesRule]{Server/Service/Connector} Setting property 'tlsCiphers' to '-TLS_ECDH_ECDSA_WITH_AES_128_CBC_SHA,-TLS_ECDH_ECDSA_WITH_3DES_EDE_CBC_SHA,+TLS_ECDH_RSA_WITH_3DES_EDE_CBC_SHA,+TLS_ECDH_RSA_WITH_AES_128_CBC_SHA,+TLS_ECDH_RSA_WITH_AES_256_CBC_SHA,-TLS_ECDH_ECDSA_WITH_AES_256_CBC_SHA,+TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA,+TLS_RSA_WITH_3DES_EDE_CBC_SHA,+TLS_RSA_WITH_AES_128_CBC_SHA,+TLS_RSA_WITH_AES_256_CBC_SHA,+TLS_ECDHE_ECDSA_WITH_3DES_EDE_CBC_SHA,+TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA,-TLS_ECDHE_RSA_WITH_3DES_EDE_CBC_SHA,-TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA,-TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA,+TLS_DHE_DSS_WITH_3DES_EDE_CBC_SHA,+TLS_DHE_DSS_WITH_AES_128_CBC_SHA,+TLS_DHE_DSS_WITH_AES_256_CBC_SHA,+TLS_DHE_RSA_WITH_3DES_EDE_CBC_SHA,+TLS_DHE_RSA_WITH_AES_128_CBC_SHA,+TLS_DHE_RSA_WITH_AES_256_CBC_SHA' did not find a matching property. Sep 12 09:12:58 myserver server: Sep 12, 2016 9:12:58 AM org.apache.catalina.startup.SetAllPropertiesRule begin Sep 12 09:12:58 myserver server: WARNING: [SetAllPropertiesRule]{Server/Service/Connector} Setting property 'sslVersionRangeStream' to 'tls1_0:tls1_2' did not find a matching property. Sep 12 09:12:58 myserver server: Sep 12, 2016 9:12:58 AM org.apache.catalina.startup.SetAllPropertiesRule begin Sep 12 09:12:58 myserver server: WARNING: [SetAllPropertiesRule]{Server/Service/Connector} Setting property 'sslVersionRangeDatagram' to 'tls1_1:tls1_2' did not find a matching property. Sep 12 09:12:58 myserver server: Sep 12, 2016 9:12:58 AM org.apache.catalina.startup.SetAllPropertiesRule begin Sep 12 09:12:58 myserver server: WARNING: [SetAllPropertiesRule]{Server/Service/Connector} Setting property 'sslRangeCiphers' to '-TLS_ECDH_ECDSA_WITH_AES_128_CBC_SHA,-TLS_ECDH_ECDSA_WITH_3DES_EDE_CBC_SHA,+TLS_ECDH_RSA_WITH_3DES_EDE_CBC_SHA,+TLS_ECDH_RSA_WITH_AES_128_CBC_SHA,+TLS_ECDH_RSA_WITH_AES_256_CBC_SHA,-TLS_ECDH_ECDSA_WITH_AES_256_CBC_SHA,+TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA,+TLS_RSA_WITH_3DES_EDE_CBC_SHA,+TLS_RSA_WITH_AES_128_CBC_SHA,+TLS_RSA_WITH_AES_256_CBC_SHA,+TLS_ECDHE_ECDSA_WITH_3DES_EDE_CBC_SHA,+TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA,-TLS_ECDHE_RSA_WITH_3DES_EDE_CBC_SHA,-TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA,-TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA,+TLS_DHE_DSS_WITH_3DES_EDE_CBC_SHA,+TLS_DHE_DSS_WITH_AES_128_CBC_SHA,+TLS_DHE_DSS_WITH_AES_256_CBC_SHA,+TLS_DHE_RSA_WITH_3DES_EDE_CBC_SHA,+TLS_DHE_RSA_WITH_AES_128_CBC_SHA,+TLS_DHE_RSA_WITH_AES_256_CBC_SHA,+TLS_DHE_RSA_WITH_AES_128_CBC_SHA256,+TLS_DHE_RSA_WITH_AES_256_CBC_SHA256,+TLS_RSA_WITH_AES_128_CBC_SHA256,+TLS_RSA_WITH_AES_256_CBC_SHA256,+TLS_RSA_WITH_AES_128_GCM_SHA256,+TLS_DHE_RSA_WITH_AES_128_GCM_SHA256,+TLS_DHE_DSS_WITH_AES_128_GCM_SHA256,+TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256,+TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256,+TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,+TLS_ECDH_ECDSA_WITH_AES_128_GCM_SHA256,+TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,+TLS_ECDH_RSA_WITH_AES_128_GCM_SHA256' did not find a matching property. Sep 12 09:12:58 myserver server: Sep 12, 2016 9:12:58 AM org.apache.catalina.startup.SetAllPropertiesRule begin Sep 12 09:12:58 myserver server: WARNING: [SetAllPropertiesRule]{Server/Service/Connector} Setting property 'serverCertNickFile' to '/var/lib/pki/pki-tomcat/conf/serverCertNick.conf' did not find a matching property. Sep 12 09:12:58 myserver server: Sep 12, 2016 9:12:58 AM org.apache.catalina.startup.SetAllPropertiesRule begin Sep 12 09:12:58 myserver server: WARNING: [SetAllPropertiesRule]{Server/Service/Connector} Setting property 'passwordFile' to '/var/lib/pki/pki-tomcat/conf/password.conf' did not find a matching property. Sep 12 09:12:58 myserver server: Sep 12, 2016 9:12:58 AM org.apache.catalina.startup.SetAllPropertiesRule begin Sep 12 09:12:58 myserver server: WARNING: [SetAllPropertiesRule]{Server/Service/Connector} Setting property 'passwordClass' to 'org.apache.tomcat.util.net.jss.PlainPasswordFile' did not find a matching property. Sep 12 09:12:58 myserver server: Sep 12, 2016 9:12:58 AM org.apache.catalina.startup.SetAllPropertiesRule begin Sep 12 09:12:58 myserver server: WARNING: [SetAllPropertiesRule]{Server/Service/Connector} Setting property 'certdbDir' to '/var/lib/pki/pki-tomcat/alias' did not find a matching property. Sep 12 09:12:58 myserver server: Sep 12, 2016 9:12:58 AM org.apache.tomcat.util.digester.SetPropertiesRule begin Sep 12 09:12:58 myserver server: WARNING: [SetPropertiesRule]{Server/Service/Engine/Host} Setting property 'xmlValidation' to 'false' did not find a matching property. Sep 12 09:12:58 myserver server: Sep 12, 2016 9:12:58 AM org.apache.tomcat.util.digester.SetPropertiesRule begin Sep 12 09:12:58 myserver server: WARNING: [SetPropertiesRule]{Server/Service/Engine/Host} Setting property 'xmlNamespaceAware' to 'false' did not find a matching property. Sep 12 09:12:58 myserver server: Sep 12, 2016 9:12:58 AM org.apache.tomcat.util.digester.SetPropertiesRule begin Sep 12 09:12:58 myserver server: WARNING: [SetPropertiesRule]{Server/Service/Engine/Host/Valve} Setting property 'resolveHosts' to 'false' did not find a matching property. Sep 12 09:12:58 myserver server: Sep 12, 2016 9:12:58 AM org.apache.catalina.startup.VersionLoggerListener log Sep 12 09:12:58 myserver server: INFO: Server version: Apache Tomcat/8.0.36 Sep 12 09:12:58 myserver server: Sep 12, 2016 9:12:58 AM org.apache.catalina.startup.VersionLoggerListener log Sep 12 09:12:58 myserver server: INFO: Server built: Aug 11 2016 18:10:25 UTC Sep 12 09:12:58 myserver server: Sep 12, 2016 9:12:58 AM org.apache.catalina.startup.VersionLoggerListener log Sep 12 09:12:58 myserver server: INFO: Server number: 8.0.36.0 Sep 12 09:12:58 myserver server: Sep 12, 2016 9:12:58 AM org.apache.catalina.startup.VersionLoggerListener log Sep 12 09:12:58 myserver server: INFO: OS Name: Linux Sep 12 09:12:58 myserver server: Sep 12, 2016 9:12:58 AM org.apache.catalina.startup.VersionLoggerListener log Sep 12 09:12:58 myserver server: INFO: OS Version: 4.7.2-201.fc24.x86_64 Sep 12 09:12:58 myserver server: Sep 12, 2016 9:12:58 AM org.apache.catalina.startup.VersionLoggerListener log Sep 12 09:12:58 myserver server: INFO: Architecture: amd64 Sep 12 09:12:58 myserver server: Sep 12, 2016 9:12:58 AM org.apache.catalina.startup.VersionLoggerListener log Sep 12 09:12:58 myserver server: INFO: Java Home: /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.102-1.b14.fc24.x86_64/jre Sep 12 09:12:58 myserver server: Sep 12, 2016 9:12:58 AM org.apache.catalina.startup.VersionLoggerListener log Sep 12 09:12:58 myserver server: INFO: JVM Version: 1.8.0_102-b14 Sep 12 09:12:58 myserver server: Sep 12, 2016 9:12:58 AM org.apache.catalina.startup.VersionLoggerListener log Sep 12 09:12:58 myserver server: INFO: JVM Vendor: Oracle Corporation Sep 12 09:12:58 myserver server: Sep 12, 2016 9:12:58 AM org.apache.catalina.startup.VersionLoggerListener log Sep 12 09:12:58 myserver server: INFO: CATALINA_BASE: /var/lib/pki/pki-tomcat Sep 12 09:12:58 myserver server: Sep 12, 2016 9:12:58 AM org.apache.catalina.startup.VersionLoggerListener log Sep 12 09:12:58 myserver server: INFO: CATALINA_HOME: /usr/share/tomcat Sep 12 09:12:58 myserver server: Sep 12, 2016 9:12:58 AM org.apache.catalina.startup.VersionLoggerListener log Sep 12 09:12:58 myserver server: INFO: Command line argument: -agentpath:/usr/lib/abrt-java-connector/libabrt-java-connector.so=abrt=on Sep 12 09:12:58 myserver server: Sep 12, 2016 9:12:58 AM org.apache.catalina.startup.VersionLoggerListener log Sep 12 09:12:58 myserver server: INFO: Command line argument: -DRESTEASY_LIB=/usr/share/java/resteasy Sep 12 09:12:58 myserver server: Sep 12, 2016 9:12:58 AM org.apache.catalina.startup.VersionLoggerListener log Sep 12 09:12:58 myserver server: INFO: Command line argument: -Djava.library.path=/usr/lib64/nuxwdog-jni Sep 12 09:12:58 myserver server: Sep 12, 2016 9:12:58 AM org.apache.catalina.startup.VersionLoggerListener log Sep 12 09:12:58 myserver server: INFO: Command line argument: -Dcatalina.base=/var/lib/pki/pki-tomcat Sep 12 09:12:58 myserver server: Sep 12, 2016 9:12:58 AM org.apache.catalina.startup.VersionLoggerListener log Sep 12 09:12:58 myserver server: INFO: Command line argument: -Dcatalina.home=/usr/share/tomcat Sep 12 09:12:58 myserver server: Sep 12, 2016 9:12:58 AM org.apache.catalina.startup.VersionLoggerListener log Sep 12 09:12:58 myserver server: INFO: Command line argument: -Djava.endorsed.dirs= Sep 12 09:12:58 myserver server: Sep 12, 2016 9:12:58 AM org.apache.catalina.startup.VersionLoggerListener log Sep 12 09:12:58 myserver server: INFO: Command line argument: -Djava.io.tmpdir=/var/lib/pki/pki-tomcat/temp Sep 12 09:12:58 myserver server: Sep 12, 2016 9:12:58 AM org.apache.catalina.startup.VersionLoggerListener log Sep 12 09:12:58 myserver server: INFO: Command line argument: -Djava.util.logging.config.file=/var/lib/pki/pki-tomcat/conf/logging.properties Sep 12 09:12:58 myserver server: Sep 12, 2016 9:12:58 AM org.apache.catalina.startup.VersionLoggerListener log Sep 12 09:12:58 myserver server: INFO: Command line argument: -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager Sep 12 09:12:58 myserver server: Sep 12, 2016 9:12:58 AM org.apache.catalina.startup.VersionLoggerListener log Sep 12 09:12:58 myserver server: INFO: Command line argument: -Djava.security.manager Sep 12 09:12:58 myserver server: Sep 12, 2016 9:12:58 AM org.apache.catalina.startup.VersionLoggerListener log Sep 12 09:12:58 myserver server: INFO: Command line argument: -Djava.security.policy==/var/lib/pki/pki-tomcat/conf/catalina.policy Sep 12 09:12:58 myserver server: PKIListener: org.apache.catalina.core.StandardServer[before_init] Sep 12 09:12:58 myserver server: Sep 12, 2016 9:12:58 AM org.apache.coyote.AbstractProtocol init Sep 12 09:12:58 myserver server: INFO: Initializing ProtocolHandler ["http-nio-8080"] Sep 12 09:12:58 myserver server: Sep 12, 2016 9:12:58 AM org.apache.tomcat.util.net.NioSelectorPool getSharedSelector Sep 12 09:12:58 myserver server: INFO: Using a shared selector for servlet write/read Sep 12 09:12:58 myserver server: Sep 12, 2016 9:12:58 AM org.apache.coyote.AbstractProtocol init Sep 12 09:12:58 myserver server: INFO: Initializing ProtocolHandler ["http-bio-8443"] Sep 12 09:12:58 myserver server: Error: SSL cipher "TLS_ECDH_RSA_WITH_3DES_EDE_CBC_SHA" not recognized by tomcatjss Sep 12 09:12:58 myserver server: Error: SSL cipher "TLS_ECDH_RSA_WITH_AES_128_CBC_SHA" not recognized by tomcatjss Sep 12 09:12:58 myserver server: Error: SSL cipher "TLS_ECDH_RSA_WITH_AES_256_CBC_SHA" not recognized by tomcatjss Sep 12 09:12:58 myserver server: Error: SSL cipher "TLS_RSA_WITH_3DES_EDE_CBC_SHA" not recognized by tomcatjss Sep 12 09:12:58 myserver server: Error: SSL cipher "TLS_DHE_DSS_WITH_3DES_EDE_CBC_SHA" not recognized by tomcatjss Sep 12 09:12:58 myserver server: Error: SSL cipher "TLS_DHE_RSA_WITH_3DES_EDE_CBC_SHA" not recognized by tomcatjss Sep 12 09:12:58 myserver server: Error: SSL cipher "TLS_ECDH_ECDSA_WITH_AES_128_GCM_SHA256" unsupported by NSS Sep 12 09:12:58 myserver server: Error: SSL cipher "TLS_ECDH_RSA_WITH_AES_128_GCM_SHA256" unsupported by NSS Sep 12 09:12:58 myserver server: Sep 12, 2016 9:12:58 AM org.apache.coyote.AbstractProtocol init Sep 12 09:12:58 myserver server: INFO: Initializing ProtocolHandler ["ajp-nio-127.0.0.1-8009"] Sep 12 09:12:58 myserver server: Sep 12, 2016 9:12:58 AM org.apache.tomcat.util.net.NioSelectorPool getSharedSelector Sep 12 09:12:58 myserver server: INFO: Using a shared selector for servlet write/read Sep 12 09:12:58 myserver server: PKIListener: org.apache.catalina.core.StandardServer[after_init] Sep 12 09:12:58 myserver server: Sep 12, 2016 9:12:58 AM org.apache.catalina.startup.Catalina load Sep 12 09:12:58 myserver server: INFO: Initialization processed in 649 ms Sep 12 09:12:58 myserver server: PKIListener: org.apache.catalina.core.StandardServer[before_start] Sep 12 09:12:58 myserver server: PKIListener: org.apache.catalina.core.StandardServer[configure_start] Sep 12 09:12:58 myserver server: PKIListener: org.apache.catalina.core.StandardServer[start] Sep 12 09:12:58 myserver server: Sep 12, 2016 9:12:58 AM org.apache.catalina.core.StandardService startInternal Sep 12 09:12:58 myserver server: INFO: Starting service Catalina Sep 12 09:12:58 myserver server: Sep 12, 2016 9:12:58 AM org.apache.catalina.core.StandardEngine startInternal Sep 12 09:12:58 myserver server: INFO: Starting Servlet Engine: Apache Tomcat/8.0.36 Sep 12 09:12:58 myserver server: Sep 12, 2016 9:12:58 AM org.apache.catalina.startup.HostConfig deployDescriptor Sep 12 09:12:58 myserver server: INFO: Deploying configuration descriptor /etc/pki/pki-tomcat/Catalina/localhost/ROOT.xml Sep 12 09:12:59 myserver ns-slapd: [12/Sep/2016:09:12:59.411513942 +0200] NSMMReplicationPlugin - agmt="cn=meTomyreplica.mydomain" (myreplica:389): Replication bind with GSSAPI auth resumed Sep 12 09:12:59 myserver server: Sep 12, 2016 9:12:59 AM org.apache.jasper.servlet.TldScanner scanJars Sep 12 09:12:59 myserver server: INFO: At least one JAR was scanned for TLDs yet contained no TLDs. Enable debug logging for this logger for a complete list of JARs that were scanned but no TLDs were found in them. Skipping unneeded JARs during scanning can improve startup time and JSP compilation time. Sep 12 09:12:59 myserver server: Sep 12, 2016 9:12:59 AM org.apache.catalina.startup.HostConfig deployDescriptor Sep 12 09:12:59 myserver server: INFO: Deployment of configuration descriptor /etc/pki/pki-tomcat/Catalina/localhost/ROOT.xml has finished in 1,317 ms Sep 12 09:12:59 myserver server: Sep 12, 2016 9:12:59 AM org.apache.catalina.startup.HostConfig deployDescriptor Sep 12 09:12:59 myserver server: INFO: Deploying configuration descriptor /etc/pki/pki-tomcat/Catalina/localhost/ca.xml Sep 12 09:12:59 myserver server: SSLAuthenticatorWithFallback: Creating SSL authenticator with fallback Sep 12 09:12:59 myserver server: SSLAuthenticatorWithFallback: Setting container Sep 12 09:13:00 myserver ns-slapd: [12/Sep/2016:09:13:00.412184764 +0200] schema-compat-plugin - Finished plugin initialization. Sep 12 09:13:01 myserver server: SSLAuthenticatorWithFallback: Initializing authenticators Sep 12 09:13:01 myserver server: SSLAuthenticatorWithFallback: Starting authenticators Sep 12 09:13:01 myserver server: Sep 12, 2016 9:13:01 AM org.apache.jasper.servlet.TldScanner scanJars Sep 12 09:13:01 myserver server: INFO: At least one JAR was scanned for TLDs yet contained no TLDs. Enable debug logging for this logger for a complete list of JARs that were scanned but no TLDs were found in them. Skipping unneeded JARs during scanning can improve startup time and JSP compilation time. Sep 12 09:13:01 myserver server: CMSEngine.initializePasswordStore() begins Sep 12 09:13:01 myserver server: CMSEngine.initializePasswordStore(): tag=internaldb Sep 12 09:13:01 myserver server: CMSEngine.initializePasswordStore(): tag=replicationdb Sep 12 09:13:01 myserver named-pkcs11[145109]: zone 21.0.10.in-addr.arpa/IN: sending notifies (serial 1473664376) Sep 12 09:13:01 myserver named-pkcs11[145109]: zone 1.168.192.in-addr.arpa/IN: sending notifies (serial 1473664376) Sep 12 09:13:01 myserver named-pkcs11[145109]: zone chloe/IN: sending notifies (serial 1473664376) Sep 12 09:13:01 myserver named-pkcs11[145109]: zone mydomain/IN: sending notifies (serial 1473664376) Sep 12 09:13:02 myserver server: CA is started. Sep 12 09:13:02 myserver server: Sep 12, 2016 9:13:02 AM org.apache.catalina.startup.HostConfig deployDescriptor Sep 12 09:13:02 myserver server: INFO: Deployment of configuration descriptor /etc/pki/pki-tomcat/Catalina/localhost/ca.xml has finished in 2,251 ms Sep 12 09:13:02 myserver server: Sep 12, 2016 9:13:02 AM org.apache.catalina.startup.HostConfig deployDescriptor Sep 12 09:13:02 myserver server: INFO: Deploying configuration descriptor /etc/pki/pki-tomcat/Catalina/localhost/pki#js.xml Sep 12 09:13:02 myserver server: Sep 12, 2016 9:13:02 AM org.apache.catalina.startup.SetContextPropertiesRule begin Sep 12 09:13:02 myserver server: WARNING: [SetContextPropertiesRule]{Context} Setting property 'allowLinking' to 'true' did not find a matching property. Sep 12 09:13:02 myserver server: Sep 12, 2016 9:13:02 AM org.apache.jasper.servlet.TldScanner scanJars Sep 12 09:13:02 myserver server: INFO: At least one JAR was scanned for TLDs yet contained no TLDs. Enable debug logging for this logger for a complete list of JARs that were scanned but no TLDs were found in them. Skipping unneeded JARs during scanning can improve startup time and JSP compilation time. Sep 12 09:13:02 myserver server: Sep 12, 2016 9:13:02 AM org.apache.catalina.startup.HostConfig deployDescriptor Sep 12 09:13:02 myserver server: INFO: Deployment of configuration descriptor /etc/pki/pki-tomcat/Catalina/localhost/pki#js.xml has finished in 699 ms Sep 12 09:13:02 myserver server: Sep 12, 2016 9:13:02 AM org.apache.catalina.startup.HostConfig deployDescriptor Sep 12 09:13:02 myserver server: INFO: Deploying configuration descriptor /etc/pki/pki-tomcat/Catalina/localhost/pki#admin.xml Sep 12 09:13:02 myserver server: Sep 12, 2016 9:13:02 AM org.apache.catalina.startup.SetContextPropertiesRule begin Sep 12 09:13:02 myserver server: WARNING: [SetContextPropertiesRule]{Context} Setting property 'allowLinking' to 'true' did not find a matching property. Sep 12 09:13:03 myserver server: Sep 12, 2016 9:13:03 AM org.apache.jasper.servlet.TldScanner scanJars Sep 12 09:13:03 myserver server: INFO: At least one JAR was scanned for TLDs yet contained no TLDs. Enable debug logging for this logger for a complete list of JARs that were scanned but no TLDs were found in them. Skipping unneeded JARs during scanning can improve startup time and JSP compilation time. Sep 12 09:13:03 myserver server: Sep 12, 2016 9:13:03 AM org.apache.catalina.startup.HostConfig deployDescriptor Sep 12 09:13:03 myserver server: INFO: Deployment of configuration descriptor /etc/pki/pki-tomcat/Catalina/localhost/pki#admin.xml has finished in 792 ms Sep 12 09:13:03 myserver server: Sep 12, 2016 9:13:03 AM org.apache.coyote.AbstractProtocol start Sep 12 09:13:03 myserver server: INFO: Starting ProtocolHandler ["http-nio-8080"] Sep 12 09:13:03 myserver server: Sep 12, 2016 9:13:03 AM org.apache.coyote.AbstractProtocol start Sep 12 09:13:03 myserver server: INFO: Starting ProtocolHandler ["http-bio-8443"] Sep 12 09:13:03 myserver server: Sep 12, 2016 9:13:03 AM org.apache.coyote.AbstractProtocol start Sep 12 09:13:03 myserver server: INFO: Starting ProtocolHandler ["ajp-nio-127.0.0.1-8009"] Sep 12 09:13:03 myserver server: PKIListener: org.apache.catalina.core.StandardServer[after_start] Sep 12 09:13:03 myserver server: PKIListener: Subsystem CA is running. Sep 12 09:13:03 myserver server: Sep 12, 2016 9:13:03 AM org.apache.catalina.startup.Catalina start Sep 12 09:13:03 myserver server: INFO: Server startup in 5125 ms Sep 12 09:13:03 myserver systemd: Starting Samba SMB Daemon... Sep 12 09:13:03 myserver systemd: smb.service: Supervising process 145454 which is not our child. We'll most likely not notice when it exits. Sep 12 09:13:04 myserver smbd[145454]: [2016/09/12 09:13:04.125995, 0] ../lib/util/become_daemon.c:124(daemon_ready) Sep 12 09:13:04 myserver audit: SERVICE_START pid=1 uid=0 auid=4294967295 ses=4294967295 msg='unit=smb comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success' Sep 12 09:13:04 myserver smbd[145454]: STATUS=daemon 'smbd' finished starting up and ready to serve connections Sep 12 09:13:04 myserver systemd: Started Samba SMB Daemon. Sep 12 09:13:04 myserver systemd: Starting Samba Winbind Daemon... Sep 12 09:13:04 myserver systemd: winbind.service: Supervising process 145468 which is not our child. We'll most likely not notice when it exits. Sep 12 09:13:02 myserver server: WARNING: [SetContextPropertiesRule]{Context} Setting property 'allowLinking' to 'true' did not find a matching property. Sep 12 09:13:02 myserver server: Sep 12, 2016 9:13:02 AM org.apache.jasper.servlet.TldScanner scanJars Sep 12 09:13:02 myserver server: INFO: At least one JAR was scanned for TLDs yet contained no TLDs. Enable debug logging for this logger for a complete list of JARs that were scanned but no TLDs were found in them. Skipping unneeded JARs during scanning can improve startup time and JSP compilation time. Sep 12 09:13:02 myserver server: Sep 12, 2016 9:13:02 AM org.apache.catalina.startup.HostConfig deployDescriptor Sep 12 09:13:02 myserver server: INFO: Deployment of configuration descriptor /etc/pki/pki-tomcat/Catalina/localhost/pki#js.xml has finished in 699 ms Sep 12 09:13:02 myserver server: Sep 12, 2016 9:13:02 AM org.apache.catalina.startup.HostConfig deployDescriptor Sep 12 09:13:02 myserver server: INFO: Deploying configuration descriptor /etc/pki/pki-tomcat/Catalina/localhost/pki#admin.xml Sep 12 09:13:02 myserver server: Sep 12, 2016 9:13:02 AM org.apache.catalina.startup.SetContextPropertiesRule begin Sep 12 09:13:02 myserver server: WARNING: [SetContextPropertiesRule]{Context} Setting property 'allowLinking' to 'true' did not find a matching property. Sep 12 09:13:03 myserver server: Sep 12, 2016 9:13:03 AM org.apache.jasper.servlet.TldScanner scanJars Sep 12 09:13:03 myserver server: INFO: At least one JAR was scanned for TLDs yet contained no TLDs. Enable debug logging for this logger for a complete list of JARs that were scanned but no TLDs were found in them. Skipping unneeded JARs during scanning can improve startup time and JSP compilation time. Sep 12 09:13:03 myserver server: Sep 12, 2016 9:13:03 AM org.apache.catalina.startup.HostConfig deployDescriptor Sep 12 09:13:03 myserver server: INFO: Deployment of configuration descriptor /etc/pki/pki-tomcat/Catalina/localhost/pki#admin.xml has finished in 792 ms Sep 12 09:13:03 myserver server: Sep 12, 2016 9:13:03 AM org.apache.coyote.AbstractProtocol start Sep 12 09:13:03 myserver server: INFO: Starting ProtocolHandler ["http-nio-8080"] Sep 12 09:13:03 myserver server: Sep 12, 2016 9:13:03 AM org.apache.coyote.AbstractProtocol start Sep 12 09:13:03 myserver server: INFO: Starting ProtocolHandler ["http-bio-8443"] Sep 12 09:13:03 myserver server: Sep 12, 2016 9:13:03 AM org.apache.coyote.AbstractProtocol start Sep 12 09:13:03 myserver server: INFO: Starting ProtocolHandler ["ajp-nio-127.0.0.1-8009"] Sep 12 09:13:03 myserver server: PKIListener: org.apache.catalina.core.StandardServer[after_start] Sep 12 09:13:03 myserver server: PKIListener: Subsystem CA is running. Sep 12 09:13:03 myserver server: Sep 12, 2016 9:13:03 AM org.apache.catalina.startup.Catalina start Sep 12 09:13:03 myserver server: INFO: Server startup in 5125 ms Sep 12 09:13:03 myserver systemd: Starting Samba SMB Daemon... Sep 12 09:13:03 myserver systemd: smb.service: Supervising process 145454 which is not our child. We'll most likely not notice when it exits. Sep 12 09:13:04 myserver smbd[145454]: [2016/09/12 09:13:04.125995, 0] ../lib/util/become_daemon.c:124(daemon_ready) Sep 12 09:13:04 myserver audit: SERVICE_START pid=1 uid=0 auid=4294967295 ses=4294967295 msg='unit=smb comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success' Sep 12 09:13:04 myserver smbd[145454]: STATUS=daemon 'smbd' finished starting up and ready to serve connections Sep 12 09:13:04 myserver systemd: Started Samba SMB Daemon. Sep 12 09:13:04 myserver systemd: Starting Samba Winbind Daemon... Sep 12 09:13:04 myserver systemd: winbind.service: Supervising process 145468 which is not our child. We'll most likely not notice when it exits. Sep 12 09:13:04 myserver winbindd[145468]: [2016/09/12 09:13:04.207791, 0] ../source3/winbindd/winbindd_cache.c:3245(initialize_winbindd_cache) Sep 12 09:13:04 myserver winbindd[145468]: initialize_winbindd_cache: clearing cache and re-creating with version number 2 Sep 12 09:13:04 myserver winbindd[145468]: [2016/09/12 09:13:04.209067, 0] ../lib/util/become_daemon.c:124(daemon_ready) Sep 12 09:13:04 myserver audit: SERVICE_START pid=1 uid=0 auid=4294967295 ses=4294967295 msg='unit=winbind comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success' Sep 12 09:13:04 myserver winbindd[145468]: STATUS=daemon 'winbindd' finished starting up and ready to serve connections Sep 12 09:13:04 myserver systemd: Started Samba Winbind Daemon. Sep 12 09:13:04 myserver systemd: Listening on ipa-otpd socket. Sep 12 09:13:04 myserver systemd: Started IPA key daemon. Sep 12 09:13:04 myserver audit: SERVICE_START pid=1 uid=0 auid=4294967295 ses=4294967295 msg='unit=ipa-dnskeysyncd comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success' Sep 12 09:13:04 myserver ipactl: ipa: INFO: The ipactl command was successful Sep 12 09:13:04 myserver ipactl: Starting Directory Service Sep 12 09:13:04 myserver ipactl: Starting krb5kdc Service Sep 12 09:13:04 myserver ipactl: Starting kadmin Service Sep 12 09:13:04 myserver ipactl: Starting named Service Sep 12 09:13:04 myserver ipactl: Starting ipa_memcached Service Sep 12 09:13:04 myserver ipactl: Starting httpd Service Sep 12 09:13:04 myserver ipactl: Starting pki-tomcatd Service Sep 12 09:13:04 myserver ipactl: Starting smb Service Sep 12 09:13:04 myserver ipactl: Starting winbind Service Sep 12 09:13:04 myserver ipactl: Starting ipa-otpd Service Sep 12 09:13:04 myserver ipactl: Starting ipa-dnskeysyncd Service Sep 12 09:13:04 myserver systemd: Started Identity, Policy, Audit. Sep 12 09:13:04 myserver audit: SERVICE_START pid=1 uid=0 auid=4294967295 ses=4294967295 msg='unit=ipa comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success' Sep 12 09:13:04 myserver ipa-dnskeysyncd: ipa: WARNING: session memcached servers not running Sep 12 09:13:05 myserver ipa-dnskeysyncd: ipa : INFO LDAP bind... Sep 12 09:13:05 myserver ipa-dnskeysyncd: ipa : INFO Commencing sync process Sep 12 09:13:05 myserver ipa-dnskeysyncd: ipa.ipapython.dnssec.keysyncer.KeySyncer: INFO Initial LDAP dump is done, sychronizing with ODS and BIND Sep 12 09:14:03 myserver named-pkcs11[145109]: FORMERR resolving '216.28.16.104.IN-ADDR.ARPA/PTR/IN': 173.245.58.100#53 -------------- next part -------------- An HTML attachment was scrubbed... URL: From jhrozek at redhat.com Mon Sep 12 07:48:03 2016 From: jhrozek at redhat.com (Jakub Hrozek) Date: Mon, 12 Sep 2016 09:48:03 +0200 Subject: [Freeipa-users] sssd stops after nss crashes In-Reply-To: References: Message-ID: <20160912074803.foza7onie4ocrhop@hendrix> On Mon, Sep 12, 2016 at 11:09:05AM +1000, Lachlan Musicman wrote: > (Fri Sep 9 20:41:13 2016) [sssd[nss]] [sbus_client_init] (0x0020): > check_file failed for [/var/lib/sss/pipes/private/ > sbus-dp_unix.petermac.org.au]. It looks like the domain process died and never recovered. What is in /var/log/sssd/sssd_unix.petermac.org.au.log ? From mbasti at redhat.com Mon Sep 12 07:55:09 2016 From: mbasti at redhat.com (Martin Basti) Date: Mon, 12 Sep 2016 09:55:09 +0200 Subject: [Freeipa-users] General query regarding nameserver enrtry In-Reply-To: References: <4d9223c9-216b-d79d-5c93-ec81466c75c6@redhat.com> Message-ID: On 08.09.2016 06:49, Deepak Dimri wrote: > Thanks Martin for your reply. > > It would be cool if i can have IPA client to resolve IPA server > without specifying nameserver in resolv.conf > > How do i configure zone delegation? is there any document i can refer? http://www.zytrax.com/books/dns/ch9/delegate.html http://serverfault.com/questions/530415/what-is-dns-delegation > > Many Thanks, > Deepak > > ------------------------------------------------------------------------ > Subject: Re: [Freeipa-users] General query regarding nameserver enrtry > To: deepak_dimri at hotmail.com; freeipa-users at redhat.com > From: mbasti at redhat.com > Date: Mon, 5 Sep 2016 09:12:08 +0200 > > > > > On 02.09.2016 20:06, Deepak Dimri wrote: > > Hi All, > > My ipa-client-install fails until etc/resolv.conf gets updated > with IPA nameserver entry. I want to avoid a task of updating > resolve.conf in my automation script. Is there a way i can get my > IPA client installation successful without updating resolve.conf? > what options do i have? > > > Many Thanks, > Deepak > > > Hello, > > do you have proper zone delegation? With proper zone delegation it > should be able to resolve IPA from every nameserver. > > Martin -------------- next part -------------- An HTML attachment was scrubbed... URL: From jochen at winteltosh.de Mon Sep 12 08:31:10 2016 From: jochen at winteltosh.de (Jochen Demmer) Date: Mon, 12 Sep 2016 10:31:10 +0200 Subject: [Freeipa-users] bind crashes on rndc reload Message-ID: <77002552-83b0-7e5d-0950-9d8f78b504de@winteltosh.de> Hi, I have a major issue with my setup: Fedora 24 freeipa-common-4.3.2-2.fc24.noarch freeipa-admintools-4.3.2-2.fc24.noarch freeipa-server-dns-4.3.2-2.fc24.noarch freeipa-client-common-4.3.2-2.fc24.noarch freeipa-server-4.3.2-2.fc24.x86_64 freeipa-server-common-4.3.2-2.fc24.noarch freeipa-client-4.3.2-2.fc24.x86_64 bind-dyndb-ldap-9.0-3.fc24.x86_64 bind-libs-lite-9.10.4-1.P2.fc24.x86_64 bind-pkcs11-libs-9.10.4-1.P2.fc24.x86_64 bind99-libs-9.9.9-1.P2.fc24.x86_64 bind-utils-9.10.4-1.P2.fc24.x86_64 rpcbind-0.2.3-11.rc1.fc24.x86_64 bind-license-9.10.4-1.P2.fc24.noarch bind-pkcs11-9.10.4-1.P2.fc24.x86_64 bind-9.10.4-1.P2.fc24.x86_64 bind-libs-9.10.4-1.P2.fc24.x86_64 bind99-license-9.9.9-1.P2.fc24.noarch bind-pkcs11-utils-9.10.4-1.P2.fc24.x86_64 It seems that there is a regular but not daily "rndc reload" sent to the nameserver that leads to a crash of it. I sent a SIGHUP to the named process, but that didn't lead to a crash. Only "rndc reload" does. It does not crash EVERY time, but most of the times. I need to do an "ipactl restart" in order to get the nameserver up and running again. I found this thread, but this doesn't give me any clues: https://www.redhat.com/archives/freeipa-users/2012-May/msg00340.html This is what the log says: http://paste.debian.net/818354/ Please understand that I obfuscated my IP addresses and domain names This is the strace: http://paste.debian.net/818365/ This is my named.conf: http://paste.debian.net/818368/ Hope someone can help. Jochen From mbasti at redhat.com Mon Sep 12 08:41:27 2016 From: mbasti at redhat.com (Martin Basti) Date: Mon, 12 Sep 2016 10:41:27 +0200 Subject: [Freeipa-users] Disable DNS checks using ipa-server-intall with FreeIPA 4.3.2 on Fedora 24? In-Reply-To: References: Message-ID: <0732e66a-115a-f3f7-610c-0b41dc8666c8@redhat.com> On 11.09.2016 20:15, Richard Harmonson wrote: > Is there an option to disable the various DNS checks using > ipa-server-install with FreeIPA 4.3.2? Is there plans to do provide > the option in future releases? Reviewing the ipa-server-install man > page, I am not seeing it. > > I want to compliment the team for placing safeguards on the > installation of FreeIPA in respect to DNS best practices. I get the > need to help some of the installers that may not have a strong grasp > of DNS. However, there are scenarios where the checks becomes > installation blockers or result in undesirable results. I am not > trying to be mysterious, but I have no desire to embarrass third > parties on their DNS faux pas. Thank you. > > Hello, which check exactly do you mean? For zone overlap there is --allow-zone-overlap option. This was added in 4.3 Martin -------------- next part -------------- An HTML attachment was scrubbed... URL: From abokovoy at redhat.com Mon Sep 12 08:55:38 2016 From: abokovoy at redhat.com (Alexander Bokovoy) Date: Mon, 12 Sep 2016 11:55:38 +0300 Subject: [Freeipa-users] The Web UI is not loading In-Reply-To: References: Message-ID: <20160912085538.xn5mhhhy7woc7zis@redhat.com> On Mon, 12 Sep 2016, Fujisan wrote: >Hello, > >This morning I noticed I could not reload the Freeipa web ui. Its was >working well friday but something must have happend over the weekend. Do you have pki-symkey installed? /usr/share/pki/server/common/lib/symkey.jar points to /usr/lib/java/symkey.jar which is missing in your setup: >Sep 12 09:12:49 myserver server: WARNING: Problem with JAR file >[/usr/share/pki/server/common/lib/symkey.jar], exists: [false], canRead: >[false] -- / Alexander Bokovoy From fujisan43 at gmail.com Mon Sep 12 09:13:41 2016 From: fujisan43 at gmail.com (Fujisan) Date: Mon, 12 Sep 2016 11:13:41 +0200 Subject: [Freeipa-users] The Web UI is not loading In-Reply-To: <20160912085538.xn5mhhhy7woc7zis@redhat.com> References: <20160912085538.xn5mhhhy7woc7zis@redhat.com> Message-ID: No it is missing! On Mon, Sep 12, 2016 at 10:55 AM, Alexander Bokovoy wrote: > On Mon, 12 Sep 2016, Fujisan wrote: > >> Hello, >> >> This morning I noticed I could not reload the Freeipa web ui. Its was >> working well friday but something must have happend over the weekend. >> > Do you have pki-symkey installed? > > /usr/share/pki/server/common/lib/symkey.jar points to > /usr/lib/java/symkey.jar which is missing in your setup: > > > > Sep 12 09:12:49 myserver server: WARNING: Problem with JAR file >> [/usr/share/pki/server/common/lib/symkey.jar], exists: [false], canRead: >> [false] >> > > > -- > / Alexander Bokovoy > -------------- next part -------------- An HTML attachment was scrubbed... URL: From fujisan43 at gmail.com Mon Sep 12 09:18:19 2016 From: fujisan43 at gmail.com (Fujisan) Date: Mon, 12 Sep 2016 11:18:19 +0200 Subject: [Freeipa-users] The Web UI is not loading In-Reply-To: References: <20160912085538.xn5mhhhy7woc7zis@redhat.com> Message-ID: Ok I installed the missing package and restarted ipa but it is still not woking. On Mon, Sep 12, 2016 at 11:13 AM, Fujisan wrote: > No it is missing! > > On Mon, Sep 12, 2016 at 10:55 AM, Alexander Bokovoy > wrote: > >> On Mon, 12 Sep 2016, Fujisan wrote: >> >>> Hello, >>> >>> This morning I noticed I could not reload the Freeipa web ui. Its was >>> working well friday but something must have happend over the weekend. >>> >> Do you have pki-symkey installed? >> >> /usr/share/pki/server/common/lib/symkey.jar points to >> /usr/lib/java/symkey.jar which is missing in your setup: >> >> >> >> Sep 12 09:12:49 myserver server: WARNING: Problem with JAR file >>> [/usr/share/pki/server/common/lib/symkey.jar], exists: [false], canRead: >>> [false] >>> >> >> >> -- >> / Alexander Bokovoy >> > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From abokovoy at redhat.com Mon Sep 12 09:23:30 2016 From: abokovoy at redhat.com (Alexander Bokovoy) Date: Mon, 12 Sep 2016 12:23:30 +0300 Subject: [Freeipa-users] The Web UI is not loading In-Reply-To: References: <20160912085538.xn5mhhhy7woc7zis@redhat.com> Message-ID: <20160912092330.bg53mcg4gyl6d27c@redhat.com> On Mon, 12 Sep 2016, Fujisan wrote: >Ok I installed the missing package and restarted ipa but it is still not >woking. We need logs. -- / Alexander Bokovoy From fujisan43 at gmail.com Mon Sep 12 09:41:48 2016 From: fujisan43 at gmail.com (Fujisan) Date: Mon, 12 Sep 2016 11:41:48 +0200 Subject: [Freeipa-users] The Web UI is not loading In-Reply-To: <20160912092330.bg53mcg4gyl6d27c@redhat.com> References: <20160912085538.xn5mhhhy7woc7zis@redhat.com> <20160912092330.bg53mcg4gyl6d27c@redhat.com> Message-ID: Here is what i get when restarting ipa: # systemctl restart ipa Sep 12 11:32:44 myserver systemd: Stopping Identity, Policy, Audit... Sep 12 11:32:45 myserver systemd: Stopping IPA key daemon... Sep 12 11:32:45 myserver ipa-dnskeysyncd: ipa : INFO Signal 15 received: Shutting down! Sep 12 11:32:45 myserver systemd: Stopped IPA key daemon. Sep 12 11:32:45 myserver audit: SERVICE_STOP pid=1 uid=0 auid=4294967295 ses=4294967295 msg='unit=ipa-dnskeysyncd comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success' Sep 12 11:32:45 myserver systemd: Stopping ipa-otpd socket. Sep 12 11:32:45 myserver systemd: Closed ipa-otpd socket. Sep 12 11:32:45 myserver systemd: Stopping Samba Winbind Daemon... Sep 12 11:32:45 myserver winbindd[1409]: [2016/09/12 11:32:45.346681, 0] ../source3/winbindd/winbindd.c:280(winbindd_sig_term_handler) Sep 12 11:32:45 myserver winbindd[1853]: [2016/09/12 11:32:45.346688, 0] ../source3/winbindd/winbindd.c:280(winbindd_sig_term_handler) Sep 12 11:32:45 myserver winbindd[1410]: [2016/09/12 11:32:45.346688, 0] ../source3/winbindd/winbindd.c:280(winbindd_sig_term_handler) Sep 12 11:32:45 myserver winbindd[1409]: Got sig[15] terminate (is_parent=1) Sep 12 11:32:45 myserver winbindd[1853]: Got sig[15] terminate (is_parent=0) Sep 12 11:32:45 myserver winbindd[1410]: Got sig[15] terminate (is_parent=0) Sep 12 11:32:45 myserver systemd: Stopped Samba Winbind Daemon. Sep 12 11:32:45 myserver audit: SERVICE_STOP pid=1 uid=0 auid=4294967295 ses=4294967295 msg='unit=winbind comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success' Sep 12 11:32:45 myserver systemd: Stopping Samba SMB Daemon... Sep 12 11:32:45 myserver smbd[1398]: [2016/09/12 11:32:45.359870, 0] ../source3/rpc_server/lsasd.c:139(lsasd_sig_term_handler) Sep 12 11:32:45 myserver smbd[1398]: termination signal Sep 12 11:32:45 myserver systemd: Stopped Samba SMB Daemon. Sep 12 11:32:45 myserver audit: SERVICE_STOP pid=1 uid=0 auid=4294967295 ses=4294967295 msg='unit=smb comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success' Sep 12 11:32:45 myserver systemd: Stopped target PKI Tomcat Server. Sep 12 11:32:45 myserver systemd: Stopping PKI Tomcat Server pki-tomcat... Sep 12 11:32:45 myserver systemd: Stopping The Apache HTTP Server... Sep 12 11:32:45 myserver server: Java virtual machine used: /usr/share/java-utils/java-wrapper Sep 12 11:32:45 myserver server: classpath used: /usr/share/tomcat/bin/bootstrap.jar:/usr/share/tomcat/bin/tomcat-juli.jar:/usr/lib/java/commons-daemon.jar Sep 12 11:32:45 myserver server: main class used: org.apache.catalina.startup.Bootstrap Sep 12 11:32:45 myserver server: flags used: -DRESTEASY_LIB=/usr/share/java/resteasy -Djava.library.path=/usr/lib64/nuxwdog-jni Sep 12 11:32:45 myserver server: options used: -Dcatalina.base=/var/lib/pki/pki-tomcat -Dcatalina.home=/usr/share/tomcat -Djava.endorsed.dirs= -Djava.io.tmpdir=/var/lib/pki/pki-tomcat/temp -Djava.util.logging.config.file=/var/lib/pki/pki-tomcat/conf/logging.properties -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager Sep 12 11:32:45 myserver server: arguments used: stop Sep 12 11:32:45 myserver server: Sep 12, 2016 11:32:45 AM org.apache.catalina.core.StandardServer await Sep 12 11:32:45 myserver server: INFO: A valid shutdown command was received via the shutdown port. Stopping the Server instance. Sep 12 11:32:45 myserver server: PKIListener: org.apache.catalina.core.StandardServer[before_stop] Sep 12 11:32:45 myserver server: PKIListener: org.apache.catalina.core.StandardServer[stop] Sep 12 11:32:45 myserver server: PKIListener: org.apache.catalina.core.StandardServer[configure_stop] Sep 12 11:32:45 myserver server: Sep 12, 2016 11:32:45 AM org.apache.coyote.AbstractProtocol pause Sep 12 11:32:45 myserver server: INFO: Pausing ProtocolHandler ["http-nio-8080"] Sep 12 11:32:46 myserver systemd: Stopped PKI Tomcat Server pki-tomcat. Sep 12 11:32:46 myserver audit: SERVICE_STOP pid=1 uid=0 auid=4294967295 ses=4294967295 msg='unit=pki-tomcatd at pki-tomcat comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success' Sep 12 11:32:46 myserver systemd: Stopped The Apache HTTP Server. Sep 12 11:32:46 myserver audit: SERVICE_STOP pid=1 uid=0 auid=4294967295 ses=4294967295 msg='unit=httpd comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success' Sep 12 11:32:46 myserver systemd: Stopping IPA memcached daemon, increases IPA server performance... Sep 12 11:32:46 myserver systemd: Stopped IPA memcached daemon, increases IPA server performance. Sep 12 11:32:46 myserver audit: SERVICE_STOP pid=1 uid=0 auid=4294967295 ses=4294967295 msg='unit=ipa_memcached comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success' Sep 12 11:32:46 myserver systemd: Stopping Berkeley Internet Name Domain (DNS) with native PKCS#11... Sep 12 11:32:46 myserver named-pkcs11[1033]: received control channel command 'stop' Sep 12 11:32:46 myserver named-pkcs11[1033]: shutting down: flushing changes Sep 12 11:32:46 myserver named-pkcs11[1033]: stopping command channel on 127.0.0.1#953 Sep 12 11:32:46 myserver named-pkcs11[1033]: stopping command channel on ::1#953 Sep 12 11:32:46 myserver named-pkcs11[1033]: zone 21.0.10.in-addr.arpa/IN: shutting down Sep 12 11:32:46 myserver named-pkcs11[1033]: zone 1.168.192.in-addr.arpa/IN: shutting down Sep 12 11:32:46 myserver named-pkcs11[1033]: zone mydomain/IN: shutting down Sep 12 11:32:46 myserver named-pkcs11[1033]: no longer listening on ::#53 Sep 12 11:32:46 myserver named-pkcs11[1033]: no longer listening on 127.0.0.1#53 Sep 12 11:32:46 myserver named-pkcs11[1033]: no longer listening on 10.0.21.18#53 Sep 12 11:32:46 myserver named-pkcs11[1033]: exiting Sep 12 11:32:46 myserver systemd: Stopped Berkeley Internet Name Domain (DNS) with native PKCS#11. Sep 12 11:32:46 myserver audit: SERVICE_STOP pid=1 uid=0 auid=4294967295 ses=4294967295 msg='unit=named-pkcs11 comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success' Sep 12 11:32:46 myserver systemd: Stopping Kerberos 5 Password-changing and Administration... Sep 12 11:32:46 myserver systemd: kadmin.service: Main process exited, code=exited, status=2/INVALIDARGUMENT Sep 12 11:32:46 myserver systemd: Stopped Kerberos 5 Password-changing and Administration. Sep 12 11:32:46 myserver audit: SERVICE_STOP pid=1 uid=0 auid=4294967295 ses=4294967295 msg='unit=kadmin comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=failed' Sep 12 11:32:46 myserver systemd: kadmin.service: Unit entered failed state. Sep 12 11:32:46 myserver systemd: kadmin.service: Failed with result 'exit-code'. Sep 12 11:32:46 myserver systemd: Stopping Kerberos 5 KDC... Sep 12 11:32:46 myserver systemd: Stopped Kerberos 5 KDC. Sep 12 11:32:46 myserver audit: SERVICE_STOP pid=1 uid=0 auid=4294967295 ses=4294967295 msg='unit=krb5kdc comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success' Sep 12 11:32:46 myserver systemd: Stopping 389 Directory Server MYDOMAIN.... Sep 12 11:32:46 myserver ns-slapd: [12/Sep/2016:11:32:46.971642497 +0200] slapd shutting down - signaling mydomaintion threads - op stack size 17 max work q size 10 max work q stack size 10 Sep 12 11:32:46 myserver ns-slapd: [12/Sep/2016:11:32:46.972363233 +0200] slapd shutting down - waiting for 16 threads to terminate Sep 12 11:32:46 myserver ns-slapd: [12/Sep/2016:11:32:46.979319605 +0200] slapd shutting down - closing down internal subsystems and plugins Sep 12 11:32:48 myserver ns-slapd: [12/Sep/2016:11:32:48.380474801 +0200] Waiting for 4 database threads to stop Sep 12 11:32:48 myserver ns-slapd: [12/Sep/2016:11:32:48.629302878 +0200] All database threads now stopped Sep 12 11:32:48 myserver ns-slapd: [12/Sep/2016:11:32:48.640619470 +0200] slapd shutting down - freed 10 work q stack objects - freed 17 op stack objects Sep 12 11:32:48 myserver ns-slapd: [12/Sep/2016:11:32:48.876849046 +0200] slapd stopped. Sep 12 11:32:48 myserver systemd: Stopped 389 Directory Server MYDOMAIN.. Sep 12 11:32:48 myserver audit: SERVICE_STOP pid=1 uid=0 auid=4294967295 ses=4294967295 msg='unit=dirsrv at MYDOMAIN comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success' Sep 12 11:32:48 myserver ipactl: ipa: INFO: The ipactl command was successful Sep 12 11:32:48 myserver ipactl: Stopping ipa-dnskeysyncd Service Sep 12 11:32:48 myserver ipactl: Stopping ipa-otpd Service Sep 12 11:32:48 myserver ipactl: Stopping winbind Service Sep 12 11:32:48 myserver ipactl: Stopping smb Service Sep 12 11:32:48 myserver ipactl: Stopping pki-tomcatd Service Sep 12 11:32:48 myserver ipactl: Stopping httpd Service Sep 12 11:32:48 myserver ipactl: Stopping ipa_memcached Service Sep 12 11:32:48 myserver ipactl: Stopping named Service Sep 12 11:32:48 myserver ipactl: Stopping kadmin Service Sep 12 11:32:48 myserver ipactl: Stopping krb5kdc Service Sep 12 11:32:48 myserver ipactl: Stopping Directory Service Sep 12 11:32:48 myserver systemd: Stopped Identity, Policy, Audit. Sep 12 11:32:48 myserver audit: SERVICE_STOP pid=1 uid=0 auid=4294967295 ses=4294967295 msg='unit=ipa comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success' Sep 12 11:32:48 myserver systemd: Starting Identity, Policy, Audit... Sep 12 11:32:50 myserver systemd: Starting 389 Directory Server MYDOMAIN.... Sep 12 11:32:50 myserver ns-slapd: [12/Sep/2016:11:32:50.331212718 +0200] SSL alert: Sending pin request to SVRCore. You may need to run systemd-tty-ask-password-agent to provide the password. Sep 12 11:32:50 myserver ns-slapd: [12/Sep/2016:11:32:50.340174964 +0200] SSL alert: Security Initialization: Enabling default cipher set. Sep 12 11:32:50 myserver ns-slapd: [12/Sep/2016:11:32:50.347397369 +0200] SSL alert: Configured NSS Ciphers Sep 12 11:32:50 myserver ns-slapd: [12/Sep/2016:11:32:50.355451158 +0200] SSL alert: #011TLS_ECDHE_PSK_WITH_AES_128_GCM_SHA256: enabled Sep 12 11:32:50 myserver ns-slapd: [12/Sep/2016:11:32:50.363409136 +0200] SSL alert: #011TLS_ECDHE_PSK_WITH_CHACHA20_POLY1305_SHA256: enabled Sep 12 11:32:50 myserver ns-slapd: [12/Sep/2016:11:32:50.371478206 +0200] SSL alert: #011TLS_ECDHE_PSK_WITH_AES_256_GCM_SHA384: enabled Sep 12 11:32:50 myserver ns-slapd: [12/Sep/2016:11:32:50.379411120 +0200] SSL alert: #011TLS_DHE_PSK_WITH_AES_128_GCM_SHA256: enabled Sep 12 11:32:50 myserver ns-slapd: [12/Sep/2016:11:32:50.387361691 +0200] SSL alert: #011TLS_DHE_PSK_WITH_CHACHA20_POLY1305_SHA256: enabled Sep 12 11:32:50 myserver ns-slapd: [12/Sep/2016:11:32:50.395454004 +0200] SSL alert: #011TLS_DHE_PSK_WITH_AES_256_GCM_SHA384: enabled Sep 12 11:32:50 myserver ns-slapd: [12/Sep/2016:11:32:50.403463526 +0200] SSL alert: #011TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256: enabled Sep 12 11:32:50 myserver ns-slapd: [12/Sep/2016:11:32:50.411389401 +0200] SSL alert: #011TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256: enabled Sep 12 11:32:50 myserver ns-slapd: [12/Sep/2016:11:32:50.419445566 +0200] SSL alert: #011TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305_SHA256: enabled Sep 12 11:32:50 myserver ns-slapd: [12/Sep/2016:11:32:50.427461605 +0200] SSL alert: #011TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305_SHA256: enabled Sep 12 11:32:50 myserver ns-slapd: [12/Sep/2016:11:32:50.435474594 +0200] SSL alert: #011TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA: enabled Sep 12 11:32:50 myserver ns-slapd: [12/Sep/2016:11:32:50.443455042 +0200] SSL alert: #011TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA: enabled Sep 12 11:32:50 myserver ns-slapd: [12/Sep/2016:11:32:50.451449803 +0200] SSL alert: #011TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA: enabled Sep 12 11:32:50 myserver ns-slapd: [12/Sep/2016:11:32:50.459399659 +0200] SSL alert: #011TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256: enabled Sep 12 11:32:50 myserver ns-slapd: [12/Sep/2016:11:32:50.467470842 +0200] SSL alert: #011TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256: enabled Sep 12 11:32:50 myserver ns-slapd: [12/Sep/2016:11:32:50.475391400 +0200] SSL alert: #011TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA: enabled Sep 12 11:32:50 myserver ns-slapd: [12/Sep/2016:11:32:50.483432311 +0200] SSL alert: #011TLS_DHE_RSA_WITH_AES_128_GCM_SHA256: enabled Sep 12 11:32:50 myserver ns-slapd: [12/Sep/2016:11:32:50.491448941 +0200] SSL alert: #011TLS_DHE_RSA_WITH_CHACHA20_POLY1305_SHA256: enabled Sep 12 11:32:50 myserver ns-slapd: [12/Sep/2016:11:32:50.499436788 +0200] SSL alert: #011TLS_DHE_RSA_WITH_AES_128_CBC_SHA: enabled Sep 12 11:32:50 myserver ns-slapd: [12/Sep/2016:11:32:50.507446644 +0200] SSL alert: #011TLS_DHE_DSS_WITH_AES_128_CBC_SHA: enabled Sep 12 11:32:50 myserver ns-slapd: [12/Sep/2016:11:32:50.515457482 +0200] SSL alert: #011TLS_DHE_RSA_WITH_AES_128_CBC_SHA256: enabled Sep 12 11:32:50 myserver ns-slapd: [12/Sep/2016:11:32:50.523483242 +0200] SSL alert: #011TLS_DHE_RSA_WITH_AES_256_CBC_SHA: enabled Sep 12 11:32:50 myserver ns-slapd: [12/Sep/2016:11:32:50.531475845 +0200] SSL alert: #011TLS_DHE_DSS_WITH_AES_256_CBC_SHA: enabled Sep 12 11:32:50 myserver ns-slapd: [12/Sep/2016:11:32:50.539484392 +0200] SSL alert: #011TLS_DHE_RSA_WITH_AES_256_CBC_SHA256: enabled Sep 12 11:32:50 myserver ns-slapd: [12/Sep/2016:11:32:50.547470517 +0200] SSL alert: #011TLS_RSA_WITH_AES_128_GCM_SHA256: enabled Sep 12 11:32:50 myserver ns-slapd: [12/Sep/2016:11:32:50.555492982 +0200] SSL alert: #011TLS_RSA_WITH_AES_128_CBC_SHA: enabled Sep 12 11:32:50 myserver ns-slapd: [12/Sep/2016:11:32:50.563455526 +0200] SSL alert: #011TLS_RSA_WITH_AES_128_CBC_SHA256: enabled Sep 12 11:32:50 myserver ns-slapd: [12/Sep/2016:11:32:50.571439438 +0200] SSL alert: #011TLS_RSA_WITH_AES_256_CBC_SHA: enabled Sep 12 11:32:50 myserver ns-slapd: [12/Sep/2016:11:32:50.579431894 +0200] SSL alert: #011TLS_RSA_WITH_AES_256_CBC_SHA256: enabled Sep 12 11:32:50 myserver ns-slapd: [12/Sep/2016:11:32:50.623200246 +0200] SSL Initialization - Configured SSL version range: min: TLS1.0, max: TLS1.2 Sep 12 11:32:50 myserver ns-slapd: [12/Sep/2016:11:32:50.623613487 +0200] 389-Directory/1.3.5.13 B2016.221.207 starting up Sep 12 11:32:50 myserver ns-slapd: [12/Sep/2016:11:32:50.667039502 +0200] WARNING: userRoot: entry cache size 512000 B is less than db size 1228800 B; We recommend to increase the entry cache size nsslapd-cachememsize. Sep 12 11:32:50 myserver ns-slapd: [12/Sep/2016:11:32:50.667420556 +0200] WARNING: ipaca: entry cache size 512000 B is less than db size 1097728 B; We recommend to increase the entry cache size nsslapd-cachememsize. Sep 12 11:32:50 myserver ns-slapd: [12/Sep/2016:11:32:50.675477692 +0200] WARNING: changelog: entry cache size 512000 B is less than db size 19890176 B; We recommend to increase the entry cache size nsslapd-cachememsize. Sep 12 11:32:50 myserver ns-slapd: [12/Sep/2016:11:32:50.734077227 +0200] schema-compat-plugin - scheduled schema-compat-plugin tree scan in about 5 seconds after the server startup! Sep 12 11:32:50 myserver ns-slapd: [12/Sep/2016:11:32:50.756521237 +0200] NSACLPlugin - The ACL target cn=groups,cn=compat,dc=mydomain does not exist Sep 12 11:32:50 myserver ns-slapd: [12/Sep/2016:11:32:50.757154109 +0200] NSACLPlugin - The ACL target cn=computers,cn=compat,dc=mydomain does not exist Sep 12 11:32:50 myserver ns-slapd: [12/Sep/2016:11:32:50.764648124 +0200] NSACLPlugin - The ACL target cn=ng,cn=compat,dc=mydomain does not exist Sep 12 11:32:50 myserver ns-slapd: [12/Sep/2016:11:32:50.772651629 +0200] NSACLPlugin - The ACL target ou=sudoers,dc=mydomain does not exist Sep 12 11:32:50 myserver ns-slapd: [12/Sep/2016:11:32:50.780605122 +0200] NSACLPlugin - The ACL target cn=users,cn=compat,dc=mydomain does not exist Sep 12 11:32:50 myserver ns-slapd: [12/Sep/2016:11:32:50.789342444 +0200] NSACLPlugin - The ACL target cn=vaults,cn=kra,dc=mydomain does not exist Sep 12 11:32:50 myserver ns-slapd: [12/Sep/2016:11:32:50.796526601 +0200] NSACLPlugin - The ACL target cn=vaults,cn=kra,dc=mydomain does not exist Sep 12 11:32:50 myserver ns-slapd: [12/Sep/2016:11:32:50.804457798 +0200] NSACLPlugin - The ACL target cn=vaults,cn=kra,dc=mydomain does not exist Sep 12 11:32:50 myserver ns-slapd: [12/Sep/2016:11:32:50.812594979 +0200] NSACLPlugin - The ACL target cn=vaults,cn=kra,dc=mydomain does not exist Sep 12 11:32:50 myserver ns-slapd: [12/Sep/2016:11:32:50.820548383 +0200] NSACLPlugin - The ACL target cn=vaults,cn=kra,dc=mydomain does not exist Sep 12 11:32:50 myserver ns-slapd: [12/Sep/2016:11:32:50.828658086 +0200] NSACLPlugin - The ACL target cn=vaults,cn=kra,dc=mydomain does not exist Sep 12 11:32:50 myserver ns-slapd: [12/Sep/2016:11:32:50.836663149 +0200] NSACLPlugin - The ACL target cn=vaults,cn=kra,dc=mydomain does not exist Sep 12 11:32:50 myserver ns-slapd: [12/Sep/2016:11:32:50.844641172 +0200] NSACLPlugin - The ACL target cn=vaults,cn=kra,dc=mydomain does not exist Sep 12 11:32:50 myserver ns-slapd: [12/Sep/2016:11:32:50.852669595 +0200] NSACLPlugin - The ACL target cn=vaults,cn=kra,dc=mydomain does not exist Sep 12 11:32:50 myserver ns-slapd: [12/Sep/2016:11:32:50.860645511 +0200] NSACLPlugin - The ACL target cn=vaults,cn=kra,dc=mydomain does not exist Sep 12 11:32:50 myserver ns-slapd: [12/Sep/2016:11:32:50.868663579 +0200] NSACLPlugin - The ACL target cn=vaults,cn=kra,dc=mydomain does not exist Sep 12 11:32:50 myserver ns-slapd: [12/Sep/2016:11:32:50.885473485 +0200] NSACLPlugin - The ACL target cn=casigningcert cert-pki-ca,cn=ca_renewal,cn=ipa,cn=etc,dc=mydomain does not exist Sep 12 11:32:50 myserver ns-slapd: [12/Sep/2016:11:32:50.885984263 +0200] NSACLPlugin - The ACL target cn=casigningcert cert-pki-ca,cn=ca_renewal,cn=ipa,cn=etc,dc=mydomain does not exist Sep 12 11:32:51 myserver ns-slapd: [12/Sep/2016:11:32:51.031101748 +0200] NSACLPlugin - The ACL target cn=automember rebuild membership,cn=tasks,cn=config does not exist Sep 12 11:32:51 myserver ns-slapd: [12/Sep/2016:11:32:51.035898003 +0200] Skipping CoS Definition cn=Password Policy,cn=accounts,dc=mydomain--no CoS Templates found, which should be added before the CoS Definition. Sep 12 11:32:51 myserver ns-slapd: [12/Sep/2016:11:32:51.083016886 +0200] set_krb5_creds - Could not get initial credentials for principal [ldap/myserver.mydomain at MYDOMAIN] in keytab [FILE:/etc/dirsrv/ds.keytab]: -1765328228 (Cannot contact any KDC for requested realm) Sep 12 11:32:51 myserver ns-slapd: [12/Sep/2016:11:32:51.086621794 +0200] schema-compat-plugin - schema-compat-plugin tree scan will start in about 5 seconds! Sep 12 11:32:51 myserver ns-slapd: [12/Sep/2016:11:32:51.091356688 +0200] NSMMReplicationPlugin - agmt="cn=meTomyreplica.mydomain" (myreplica:389): Replication bind with GSSAPI auth failed: LDAP error -1 (Can't contact LDAP server) () Sep 12 11:32:51 myserver ns-slapd: [12/Sep/2016:11:32:51.110846862 +0200] slapd started. Listening on All Interfaces port 389 for LDAP requests Sep 12 11:32:51 myserver ns-slapd: [12/Sep/2016:11:32:51.115284748 +0200] Listening on All Interfaces port 636 for LDAPS requests Sep 12 11:32:51 myserver ns-slapd: [12/Sep/2016:11:32:51.123400775 +0200] Listening on /var/run/slapd-MYDOMAIN.socket for LDAPI requests Sep 12 11:32:51 myserver systemd: Started 389 Directory Server MYDOMAIN.. Sep 12 11:32:51 myserver audit: SERVICE_START pid=1 uid=0 auid=4294967295 ses=4294967295 msg='unit=dirsrv at MYDOMAIN comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success' Sep 12 11:32:51 myserver systemd: Starting Kerberos 5 KDC... Sep 12 11:32:51 myserver systemd: Started Kerberos 5 KDC. Sep 12 11:32:51 myserver audit: SERVICE_START pid=1 uid=0 auid=4294967295 ses=4294967295 msg='unit=krb5kdc comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success' Sep 12 11:32:51 myserver systemd: Starting Kerberos 5 Password-changing and Administration... Sep 12 11:32:51 myserver systemd: Started Kerberos 5 Password-changing and Administration. Sep 12 11:32:51 myserver audit: SERVICE_START pid=1 uid=0 auid=4294967295 ses=4294967295 msg='unit=kadmin comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success' Sep 12 11:32:51 myserver systemd: Starting Generate rndc key for BIND (DNS)... Sep 12 11:32:51 myserver systemd: Started Generate rndc key for BIND (DNS). Sep 12 11:32:51 myserver audit: SERVICE_START pid=1 uid=0 auid=4294967295 ses=4294967295 msg='unit=named-setup-rndc comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success' Sep 12 11:32:51 myserver audit: SERVICE_STOP pid=1 uid=0 auid=4294967295 ses=4294967295 msg='unit=named-setup-rndc comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success' Sep 12 11:32:51 myserver systemd: Starting Berkeley Internet Name Domain (DNS) with native PKCS#11... Sep 12 11:32:51 myserver bash: zone localhost.localdomain/IN: loaded serial 0 Sep 12 11:32:51 myserver bash: zone localhost/IN: loaded serial 0 Sep 12 11:32:51 myserver bash: zone 1.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.ip6.arpa/IN: loaded serial 0 Sep 12 11:32:51 myserver bash: zone 1.0.0.127.in-addr.arpa/IN: loaded serial 0 Sep 12 11:32:51 myserver bash: zone 0.in-addr.arpa/IN: loaded serial 0 Sep 12 11:32:52 myserver named-pkcs11[2120]: starting BIND 9.10.4-P2-RedHat-9.10.4-1.P2.fc24 -u named Sep 12 11:32:52 myserver named-pkcs11[2120]: running on Linux x86_64 4.7.2-201.fc24.x86_64 #1 SMP Fri Aug 26 15:58:40 UTC 2016 Sep 12 11:32:52 myserver named-pkcs11[2120]: built with '--build=x86_64-redhat-linux-gnu' '--host=x86_64-redhat-linux-gnu' '--program-prefix=' '--disable-dependency-tracking' '--prefix=/usr' '--exec-prefix=/usr' '--bindir=/usr/bin' '--sbindir=/usr/sbin' '--sysconfdir=/etc' '--datadir=/usr/share' '--includedir=/usr/include' '--libdir=/usr/lib64' '--libexecdir=/usr/libexec' '--sharedstatedir=/var/lib' '--mandir=/usr/share/man' '--infodir=/usr/share/info' '--with-python=/usr/bin/python3' '--with-libtool' '--localstatedir=/var' '--enable-threads' '--enable-ipv6' '--enable-filter-aaaa' '--with-pic' '--disable-static' '--disable-openssl-version-check' '--includedir=/usr/include/bind9' '--with-tuning=large' '--with-geoip' '--enable-native-pkcs11' '--with-pkcs11=/usr/lib64/pkcs11/libsofthsm2.so' '--with-dlopen=yes' '--with-dlz-ldap=yes' '--with-dlz-postgres=yes' '--with-dlz-mysql=yes' '--with-dlz-filesystem=yes' '--with-dlz-bdb=yes' '--with-gssapi=yes' '--disable-isc-spnego' '--enable-fixed-rrset' '--with-docbook-xsl=/usr/share/sgml/docbook/xsl-stylesheets' '--enable-full-report' 'build_alias=x86_64-redhat-linux-gnu' 'host_alias=x86_64-redhat-linux-gnu' 'CFLAGS= -O2 -g -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong --param=ssp-buffer-size=4 -grecord-gcc-switches -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -m64 -mtune=generic' 'LDFLAGS=-Wl,-z,relro -specs=/usr/lib/rpm/redhat/redhat-hardened-ld' 'CPPFLAGS= -DDIG_SIGCHASE' Sep 12 11:32:52 myserver named-pkcs11[2120]: ---------------------------------------------------- Sep 12 11:32:52 myserver named-pkcs11[2120]: BIND 9 is maintained by Internet Systems Consortium, Sep 12 11:32:52 myserver named-pkcs11[2120]: Inc. (ISC), a non-profit 501(c)(3) public-benefit Sep 12 11:32:52 myserver named-pkcs11[2120]: corporation. Support and training for BIND 9 are Sep 12 11:32:52 myserver named-pkcs11[2120]: available at https://www.isc.org/support Sep 12 11:32:52 myserver named-pkcs11[2120]: ---------------------------------------------------- Sep 12 11:32:52 myserver named-pkcs11[2120]: adjusted limit on open files from 4096 to 1048576 Sep 12 11:32:52 myserver named-pkcs11[2120]: found 12 CPUs, using 12 worker threads Sep 12 11:32:52 myserver named-pkcs11[2120]: using 11 UDP listeners per interface Sep 12 11:32:52 myserver named-pkcs11[2120]: using up to 21000 sockets Sep 12 11:32:52 myserver named-pkcs11[2120]: loading configuration from '/etc/named.conf' Sep 12 11:32:52 myserver named-pkcs11[2120]: reading built-in trusted keys from file '/etc/named.iscdlv.key' Sep 12 11:32:52 myserver named-pkcs11[2120]: initializing GeoIP Country (IPv4) (type 1) DB Sep 12 11:32:52 myserver named-pkcs11[2120]: GEO-106FREE 20160705 Build 1 Copy Sep 12 11:32:52 myserver named-pkcs11[2120]: GeoIP Country (IPv6) (type 12) DB not available Sep 12 11:32:52 myserver named-pkcs11[2120]: GeoIP City (IPv4) (type 2) DB not available Sep 12 11:32:52 myserver named-pkcs11[2120]: GeoIP City (IPv4) (type 6) DB not available Sep 12 11:32:52 myserver named-pkcs11[2120]: GeoIP City (IPv6) (type 30) DB not available Sep 12 11:32:52 myserver named-pkcs11[2120]: GeoIP City (IPv6) (type 31) DB not available Sep 12 11:32:52 myserver named-pkcs11[2120]: GeoIP Region (type 3) DB not available Sep 12 11:32:52 myserver named-pkcs11[2120]: GeoIP Region (type 7) DB not available Sep 12 11:32:52 myserver named-pkcs11[2120]: GeoIP ISP (type 4) DB not available Sep 12 11:32:52 myserver named-pkcs11[2120]: GeoIP Org (type 5) DB not available Sep 12 11:32:52 myserver named-pkcs11[2120]: GeoIP AS (type 9) DB not available Sep 12 11:32:52 myserver named-pkcs11[2120]: GeoIP Domain (type 11) DB not available Sep 12 11:32:52 myserver named-pkcs11[2120]: GeoIP NetSpeed (type 10) DB not available Sep 12 11:32:52 myserver named-pkcs11[2120]: using default UDP/IPv4 port range: [32768, 60999] Sep 12 11:32:52 myserver named-pkcs11[2120]: using default UDP/IPv6 port range: [32768, 60999] Sep 12 11:32:52 myserver named-pkcs11[2120]: listening on IPv6 interfaces, port 53 Sep 12 11:32:52 myserver named-pkcs11[2120]: listening on IPv4 interface lo, 127.0.0.1#53 Sep 12 11:32:52 myserver named-pkcs11[2120]: listening on IPv4 interface eno1, 10.0.21.18#53 Sep 12 11:32:52 myserver named-pkcs11[2120]: generating session key for dynamic DNS Sep 12 11:32:52 myserver named-pkcs11[2120]: sizing zone task pool based on 6 zones Sep 12 11:32:52 myserver named-pkcs11[2120]: set up managed keys zone for view _default, file '/var/named/dynamic/managed-keys.bind' Sep 12 11:32:52 myserver named-pkcs11[2120]: bind-dyndb-ldap version 9.0 compiled at 09:01:48 May 27 2016, compiler 6.1.1 20160510 (Red Hat 6.1.1-2) Sep 12 11:32:52 myserver named-pkcs11[2120]: option 'serial_autoincrement' is not supported, ignoring Sep 12 11:32:52 myserver named-pkcs11[2120]: LDAP instance 'ipa' is being synchronized, please ignore message 'all zones loaded' Sep 12 11:32:52 myserver named-pkcs11[2120]: automatic empty zone: 10.IN-ADDR.ARPA Sep 12 11:32:52 myserver named-pkcs11[2120]: automatic empty zone: 16.172.IN-ADDR.ARPA Sep 12 11:32:52 myserver named-pkcs11[2120]: automatic empty zone: 17.172.IN-ADDR.ARPA [...] Sep 12 11:32:52 myserver named-pkcs11[2120]: automatic empty zone: 124.100.IN-ADDR.ARPA Sep 12 11:32:52 myserver named-pkcs11[2120]: automatic empty zone: 125.100.IN-ADDR.ARPA Sep 12 11:32:52 myserver named-pkcs11[2120]: automatic empty zone: 126.100.IN-ADDR.ARPA Sep 12 11:32:52 myserver named-pkcs11[2120]: automatic empty zone: 127.100.IN-ADDR.ARPA Sep 12 11:32:52 myserver named-pkcs11[2120]: automatic empty zone: 127.IN-ADDR.ARPA Sep 12 11:32:52 myserver named-pkcs11[2120]: automatic empty zone: 254.169.IN-ADDR.ARPA Sep 12 11:32:52 myserver named-pkcs11[2120]: automatic empty zone: 2.0.192.IN-ADDR.ARPA Sep 12 11:32:52 myserver named-pkcs11[2120]: automatic empty zone: 100.51.198.IN-ADDR.ARPA Sep 12 11:32:52 myserver named-pkcs11[2120]: automatic empty zone: 113.0.203.IN-ADDR.ARPA Sep 12 11:32:52 myserver named-pkcs11[2120]: automatic empty zone: 255.255.255.255.IN-ADDR.ARPA Sep 12 11:32:52 myserver named-pkcs11[2120]: automatic empty zone: 0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.IP6.ARPA Sep 12 11:32:52 myserver named-pkcs11[2120]: automatic empty zone: D.F.IP6.ARPA Sep 12 11:32:52 myserver named-pkcs11[2120]: automatic empty zone: 8.E.F.IP6.ARPA Sep 12 11:32:52 myserver named-pkcs11[2120]: automatic empty zone: 9.E.F.IP6.ARPA Sep 12 11:32:52 myserver named-pkcs11[2120]: automatic empty zone: A.E.F.IP6.ARPA Sep 12 11:32:52 myserver named-pkcs11[2120]: automatic empty zone: B.E.F.IP6.ARPA Sep 12 11:32:52 myserver named-pkcs11[2120]: automatic empty zone: 8.B.D.0.1.0.0.2.IP6.ARPA Sep 12 11:32:52 myserver named-pkcs11[2120]: automatic empty zone: EMPTY.AS112.ARPA Sep 12 11:32:52 myserver named-pkcs11[2120]: configuring command channel from '/etc/rndc.key' Sep 12 11:32:52 myserver named-pkcs11[2120]: command channel listening on 127.0.0.1#953 Sep 12 11:32:52 myserver named-pkcs11[2120]: configuring command channel from '/etc/rndc.key' Sep 12 11:32:52 myserver named-pkcs11[2120]: command channel listening on ::1#953 Sep 12 11:32:52 myserver named-pkcs11[2120]: managed-keys-zone: journal file is out of date: removing journal file Sep 12 11:32:52 myserver named-pkcs11[2120]: managed-keys-zone: loaded serial 484 Sep 12 11:32:52 myserver named-pkcs11[2120]: ignoring inherited 'forward first;' for zone '.' - did you want 'forward only;' to override automatic empty zone '10.IN-ADDR.ARPA'? Sep 12 11:32:52 myserver named-pkcs11[2120]: ignoring inherited 'forward first;' for zone '.' - did you want 'forward only;' to override automatic empty zone '16.172.IN-ADDR.ARPA'? Sep 12 11:32:52 myserver named-pkcs11[2120]: ignoring inherited 'forward first;' for zone '.' - did you want 'forward only;' to override automatic empty zone '17.172.IN-ADDR.ARPA'? [...] Sep 12 11:32:52 myserver named-pkcs11[2120]: ignoring inherited 'forward first;' for zone '.' - did you want 'forward only;' to override automatic empty zone '110.100.IN-ADDR.ARPA'? Sep 12 11:32:52 myserver named-pkcs11[2120]: ignoring inherited 'forward first;' for zone '.' - did you want 'forward only;' to override automatic empty zone '111.100.IN-ADDR.ARPA'? Sep 12 11:32:52 myserver named-pkcs11[2120]: ignoring inherited 'forward first;' for zone '.' - did you want 'forward only;' to override automatic empty zone '112.100.IN-ADDR.ARPA'? Sep 12 11:32:52 myserver named-pkcs11[2120]: ignoring inherited 'forward first;' for zone '.' - did you want 'forward only;' to override automatic empty zone '113.100.IN-ADDR.ARPA'? Sep 12 11:32:52 myserver audit: SERVICE_START pid=1 uid=0 auid=4294967295 ses=4294967295 msg='unit=named-pkcs11 comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success' Sep 12 11:32:52 myserver named-pkcs11[2120]: ignoring inherited 'forward first;' for zone '.' - did you want 'forward only;' to override automatic empty zone '114.100.IN-ADDR.ARPA'? Sep 12 11:32:52 myserver systemd: Started Berkeley Internet Name Domain (DNS) with native PKCS#11. Sep 12 11:32:52 myserver named-pkcs11[2120]: ignoring inherited 'forward first;' for zone '.' - did you want 'forward only;' to override automatic empty zone '115.100.IN-ADDR.ARPA'? Sep 12 11:32:52 myserver named-pkcs11[2120]: ignoring inherited 'forward first;' for zone '.' - did you want 'forward only;' to override automatic empty zone '116.100.IN-ADDR.ARPA'? Sep 12 11:32:52 myserver named-pkcs11[2120]: ignoring inherited 'forward first;' for zone '.' - did you want 'forward only;' to override automatic empty zone '117.100.IN-ADDR.ARPA'? [...] Sep 12 11:32:52 myserver named-pkcs11[2120]: ignoring inherited 'forward first;' for zone '.' - did you want 'forward only;' to override automatic empty zone '8.B.D.0.1.0.0.2.IP6.ARPA'? Sep 12 11:32:52 myserver named-pkcs11[2120]: ignoring inherited 'forward first;' for zone '.' - did you want 'forward only;' to override automatic empty zone 'EMPTY.AS112.ARPA'? Sep 12 11:32:52 myserver named-pkcs11[2120]: zone localhost.localdomain/IN: loaded serial 0 Sep 12 11:32:52 myserver named-pkcs11[2120]: zone localhost/IN: loaded serial 0 Sep 12 11:32:52 myserver named-pkcs11[2120]: zone 1.0.0.127.in-addr.arpa/IN: loaded serial 0 Sep 12 11:32:52 myserver named-pkcs11[2120]: zone 1.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.ip6.arpa/IN: loaded serial 0 Sep 12 11:32:52 myserver named-pkcs11[2120]: all zones loaded Sep 12 11:32:52 myserver named-pkcs11[2120]: running Sep 12 11:32:52 myserver named-pkcs11[2120]: checkhints: unable to get root NS rrset from cache: not found Sep 12 11:32:52 myserver systemd: Starting IPA memcached daemon, increases IPA server performance... Sep 12 11:32:52 myserver systemd: ipa_memcached.service: PID file /var/run/ipa_memcached/ipa_memcached.pid not readable (yet?) after start: No such file or directory Sep 12 11:32:52 myserver systemd: Started IPA memcached daemon, increases IPA server performance. Sep 12 11:32:52 myserver audit: SERVICE_START pid=1 uid=0 auid=4294967295 ses=4294967295 msg='unit=ipa_memcached comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success' Sep 12 11:32:52 myserver systemd: Starting The Apache HTTP Server... Sep 12 11:32:52 myserver named-pkcs11[2120]: zone 21.0.10.in-addr.arpa/IN: loaded serial 1473672772 Sep 12 11:32:52 myserver named-pkcs11[2120]: zone 21.0.10.in-addr.arpa/IN: sending notifies (serial 1473672772) Sep 12 11:32:52 myserver named-pkcs11[2120]: zone 1.168.192.in-addr.arpa/IN: loaded serial 1473672772 Sep 12 11:32:52 myserver named-pkcs11[2120]: zone 1.168.192.in-addr.arpa/IN: sending notifies (serial 1473672772) Sep 12 11:32:52 myserver named-pkcs11[2120]: zone mydomain/IN: loaded serial 1473672772 Sep 12 11:32:52 myserver named-pkcs11[2120]: 4 master zones from LDAP instance 'ipa' loaded (4 zones defined, 0 inactive, 0 failed to load) Sep 12 11:32:52 myserver named-pkcs11[2120]: zone mydomain/IN: sending notifies (serial 1473672772) Sep 12 11:32:52 myserver named-pkcs11[2120]: checkhints: unable to get root NS rrset from cache: not found Sep 12 11:32:52 myserver ipa-httpd-kdcproxy: ipa : INFO KDC proxy enabled Sep 12 11:32:52 myserver systemd: Started The Apache HTTP Server. Sep 12 11:32:52 myserver audit: SERVICE_START pid=1 uid=0 auid=4294967295 ses=4294967295 msg='unit=httpd comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success' Sep 12 11:32:52 myserver systemd: Starting PKI Tomcat Server pki-tomcat... Sep 12 11:32:53 myserver systemd: Started PKI Tomcat Server pki-tomcat. Sep 12 11:32:53 myserver audit: SERVICE_START pid=1 uid=0 auid=4294967295 ses=4294967295 msg='unit=pki-tomcatd at pki-tomcat comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success' Sep 12 11:32:53 myserver systemd: Reached target PKI Tomcat Server. Sep 12 11:32:53 myserver server: Java virtual machine used: /usr/share/java-utils/java-wrapper Sep 12 11:32:53 myserver server: classpath used: /usr/share/tomcat/bin/bootstrap.jar:/usr/share/tomcat/bin/tomcat-juli.jar:/usr/lib/java/commons-daemon.jar Sep 12 11:32:53 myserver server: main class used: org.apache.catalina.startup.Bootstrap Sep 12 11:32:53 myserver server: flags used: -DRESTEASY_LIB=/usr/share/java/resteasy -Djava.library.path=/usr/lib64/nuxwdog-jni Sep 12 11:32:53 myserver server: options used: -Dcatalina.base=/var/lib/pki/pki-tomcat -Dcatalina.home=/usr/share/tomcat -Djava.endorsed.dirs= -Djava.io.tmpdir=/var/lib/pki/pki-tomcat/temp -Djava.util.logging.config.file=/var/lib/pki/pki-tomcat/conf/logging.properties -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager -Djava.security.manager -Djava.security.policy==/var/lib/pki/pki-tomcat/conf/catalina.policy Sep 12 11:32:53 myserver server: arguments used: start Sep 12 11:32:53 myserver server: Sep 12, 2016 11:32:53 AM org.apache.catalina.startup.SetAllPropertiesRule begin Sep 12 11:32:53 myserver server: WARNING: [SetAllPropertiesRule]{Server/Service/Connector} Setting property 'enableOCSP' to 'false' did not find a matching property. Sep 12 11:32:53 myserver server: Sep 12, 2016 11:32:53 AM org.apache.catalina.startup.SetAllPropertiesRule begin Sep 12 11:32:53 myserver server: WARNING: [SetAllPropertiesRule]{Server/Service/Connector} Setting property 'ocspResponderURL' to 'http://myserver.mydomain:9080/ca/ocsp' did not find a matching property. Sep 12 11:32:53 myserver server: Sep 12, 2016 11:32:53 AM org.apache.catalina.startup.SetAllPropertiesRule begin Sep 12 11:32:53 myserver server: WARNING: [SetAllPropertiesRule]{Server/Service/Connector} Setting property 'ocspResponderCertNickname' to 'ocspSigningCert cert-pki-ca' did not find a matching property. Sep 12 11:32:53 myserver server: Sep 12, 2016 11:32:53 AM org.apache.catalina.startup.SetAllPropertiesRule begin Sep 12 11:32:53 myserver server: WARNING: [SetAllPropertiesRule]{Server/Service/Connector} Setting property 'ocspCacheSize' to '1000' did not find a matching property. Sep 12 11:32:53 myserver server: Sep 12, 2016 11:32:53 AM org.apache.catalina.startup.SetAllPropertiesRule begin Sep 12 11:32:53 myserver server: WARNING: [SetAllPropertiesRule]{Server/Service/Connector} Setting property 'ocspMinCacheEntryDuration' to '60' did not find a matching property. Sep 12 11:32:53 myserver server: Sep 12, 2016 11:32:53 AM org.apache.catalina.startup.SetAllPropertiesRule begin Sep 12 11:32:53 myserver server: WARNING: [SetAllPropertiesRule]{Server/Service/Connector} Setting property 'ocspMaxCacheEntryDuration' to '120' did not find a matching property. Sep 12 11:32:53 myserver server: Sep 12, 2016 11:32:53 AM org.apache.catalina.startup.SetAllPropertiesRule begin Sep 12 11:32:53 myserver server: WARNING: [SetAllPropertiesRule]{Server/Service/Connector} Setting property 'ocspTimeout' to '10' did not find a matching property. Sep 12 11:32:53 myserver server: Sep 12, 2016 11:32:53 AM org.apache.catalina.startup.SetAllPropertiesRule begin Sep 12 11:32:53 myserver server: WARNING: [SetAllPropertiesRule]{Server/Service/Connector} Setting property 'strictCiphers' to 'true' did not find a matching property. Sep 12 11:32:53 myserver server: Sep 12, 2016 11:32:53 AM org.apache.catalina.startup.SetAllPropertiesRule begin Sep 12 11:32:53 myserver server: WARNING: [SetAllPropertiesRule]{Server/Service/Connector} Setting property 'sslOptions' to 'ssl2=false,ssl3=false,tls=true' did not find a matching property. Sep 12 11:32:53 myserver server: Sep 12, 2016 11:32:53 AM org.apache.catalina.startup.SetAllPropertiesRule begin Sep 12 11:32:53 myserver server: WARNING: [SetAllPropertiesRule]{Server/Service/Connector} Setting property 'ssl2Ciphers' to '-SSL2_RC4_128_WITH_MD5,-SSL2_RC4_128_EXPORT40_WITH_MD5,-SSL2_RC2_128_CBC_WITH_MD5,-SSL2_RC2_128_CBC_EXPORT40_WITH_MD5,-SSL2_DES_64_CBC_WITH_MD5,-SSL2_DES_192_EDE3_CBC_WITH_MD5' did not find a matching property. Sep 12 11:32:53 myserver server: Sep 12, 2016 11:32:53 AM org.apache.catalina.startup.SetAllPropertiesRule begin Sep 12 11:32:53 myserver server: WARNING: [SetAllPropertiesRule]{Server/Service/Connector} Setting property 'ssl3Ciphers' to '-SSL3_FORTEZZA_DMS_WITH_NULL_SHA,-SSL3_FORTEZZA_DMS_WITH_RC4_128_SHA,+SSL3_RSA_WITH_RC4_128_SHA,-SSL3_RSA_EXPORT_WITH_RC4_40_MD5,+SSL3_RSA_WITH_3DES_EDE_CBC_SHA,-SSL3_RSA_WITH_DES_CBC_SHA,-SSL3_RSA_EXPORT_WITH_RC2_CBC_40_MD5,-SSL3_FORTEZZA_DMS_WITH_FORTEZZA_CBC_SHA,-SSL_RSA_FIPS_WITH_DES_CBC_SHA,+SSL_RSA_FIPS_WITH_3DES_EDE_CBC_SHA,-SSL3_RSA_WITH_NULL_MD5,-TLS_RSA_EXPORT1024_WITH_RC4_56_SHA,-TLS_ECDH_ECDSA_WITH_AES_256_CBC_SHA,+TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA' did not find a matching property. Sep 12 11:32:53 myserver server: Sep 12, 2016 11:32:53 AM org.apache.catalina.startup.SetAllPropertiesRule begin Sep 12 11:32:53 myserver server: WARNING: [SetAllPropertiesRule]{Server/Service/Connector} Setting property 'tlsCiphers' to '-TLS_ECDH_ECDSA_WITH_AES_128_CBC_SHA,-TLS_ECDH_ECDSA_WITH_3DES_EDE_CBC_SHA,+TLS_ECDH_RSA_WITH_3DES_EDE_CBC_SHA,+TLS_ECDH_RSA_WITH_AES_128_CBC_SHA,+TLS_ECDH_RSA_WITH_AES_256_CBC_SHA,-TLS_ECDH_ECDSA_WITH_AES_256_CBC_SHA,+TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA,+TLS_RSA_WITH_3DES_EDE_CBC_SHA,+TLS_RSA_WITH_AES_128_CBC_SHA,+TLS_RSA_WITH_AES_256_CBC_SHA,+TLS_ECDHE_ECDSA_WITH_3DES_EDE_CBC_SHA,+TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA,-TLS_ECDHE_RSA_WITH_3DES_EDE_CBC_SHA,-TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA,-TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA,+TLS_DHE_DSS_WITH_3DES_EDE_CBC_SHA,+TLS_DHE_DSS_WITH_AES_128_CBC_SHA,+TLS_DHE_DSS_WITH_AES_256_CBC_SHA,+TLS_DHE_RSA_WITH_3DES_EDE_CBC_SHA,+TLS_DHE_RSA_WITH_AES_128_CBC_SHA,+TLS_DHE_RSA_WITH_AES_256_CBC_SHA' did not find a matching property. Sep 12 11:32:53 myserver server: Sep 12, 2016 11:32:53 AM org.apache.catalina.startup.SetAllPropertiesRule begin Sep 12 11:32:53 myserver server: WARNING: [SetAllPropertiesRule]{Server/Service/Connector} Setting property 'sslVersionRangeStream' to 'tls1_0:tls1_2' did not find a matching property. Sep 12 11:32:53 myserver server: Sep 12, 2016 11:32:53 AM org.apache.catalina.startup.SetAllPropertiesRule begin Sep 12 11:32:53 myserver server: WARNING: [SetAllPropertiesRule]{Server/Service/Connector} Setting property 'sslVersionRangeDatagram' to 'tls1_1:tls1_2' did not find a matching property. Sep 12 11:32:53 myserver server: Sep 12, 2016 11:32:53 AM org.apache.catalina.startup.SetAllPropertiesRule begin Sep 12 11:32:53 myserver server: WARNING: [SetAllPropertiesRule]{Server/Service/Connector} Setting property 'sslRangeCiphers' to '-TLS_ECDH_ECDSA_WITH_AES_128_CBC_SHA,-TLS_ECDH_ECDSA_WITH_3DES_EDE_CBC_SHA,+TLS_ECDH_RSA_WITH_3DES_EDE_CBC_SHA,+TLS_ECDH_RSA_WITH_AES_128_CBC_SHA,+TLS_ECDH_RSA_WITH_AES_256_CBC_SHA,-TLS_ECDH_ECDSA_WITH_AES_256_CBC_SHA,+TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA,+TLS_RSA_WITH_3DES_EDE_CBC_SHA,+TLS_RSA_WITH_AES_128_CBC_SHA,+TLS_RSA_WITH_AES_256_CBC_SHA,+TLS_ECDHE_ECDSA_WITH_3DES_EDE_CBC_SHA,+TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA,-TLS_ECDHE_RSA_WITH_3DES_EDE_CBC_SHA,-TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA,-TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA,+TLS_DHE_DSS_WITH_3DES_EDE_CBC_SHA,+TLS_DHE_DSS_WITH_AES_128_CBC_SHA,+TLS_DHE_DSS_WITH_AES_256_CBC_SHA,+TLS_DHE_RSA_WITH_3DES_EDE_CBC_SHA,+TLS_DHE_RSA_WITH_AES_128_CBC_SHA,+TLS_DHE_RSA_WITH_AES_256_CBC_SHA,+TLS_DHE_RSA_WITH_AES_128_CBC_SHA256,+TLS_DHE_RSA_WITH_AES_256_CBC_SHA256,+TLS_RSA_WITH_AES_128_CBC_SHA256,+TLS_RSA_WITH_AES_256_CBC_SHA256,+TLS_RSA_WITH_AES_128_GCM_SHA256,+TLS_DHE_RSA_WITH_AES_128_GCM_SHA256,+TLS_DHE_DSS_WITH_AES_128_GCM_SHA256,+TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256,+TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256,+TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,+TLS_ECDH_ECDSA_WITH_AES_128_GCM_SHA256,+TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,+TLS_ECDH_RSA_WITH_AES_128_GCM_SHA256' did not find a matching property. Sep 12 11:32:53 myserver server: Sep 12, 2016 11:32:53 AM org.apache.catalina.startup.SetAllPropertiesRule begin Sep 12 11:32:53 myserver server: WARNING: [SetAllPropertiesRule]{Server/Service/Connector} Setting property 'serverCertNickFile' to '/var/lib/pki/pki-tomcat/conf/serverCertNick.conf' did not find a matching property. Sep 12 11:32:53 myserver server: Sep 12, 2016 11:32:53 AM org.apache.catalina.startup.SetAllPropertiesRule begin Sep 12 11:32:53 myserver server: WARNING: [SetAllPropertiesRule]{Server/Service/Connector} Setting property 'passwordFile' to '/var/lib/pki/pki-tomcat/conf/password.conf' did not find a matching property. Sep 12 11:32:53 myserver server: Sep 12, 2016 11:32:53 AM org.apache.catalina.startup.SetAllPropertiesRule begin Sep 12 11:32:53 myserver server: WARNING: [SetAllPropertiesRule]{Server/Service/Connector} Setting property 'passwordClass' to 'org.apache.tomcat.util.net.jss.PlainPasswordFile' did not find a matching property. Sep 12 11:32:53 myserver server: Sep 12, 2016 11:32:53 AM org.apache.catalina.startup.SetAllPropertiesRule begin Sep 12 11:32:53 myserver server: WARNING: [SetAllPropertiesRule]{Server/Service/Connector} Setting property 'certdbDir' to '/var/lib/pki/pki-tomcat/alias' did not find a matching property. Sep 12 11:32:53 myserver server: Sep 12, 2016 11:32:53 AM org.apache.tomcat.util.digester.SetPropertiesRule begin Sep 12 11:32:53 myserver server: WARNING: [SetPropertiesRule]{Server/Service/Engine/Host} Setting property 'xmlValidation' to 'false' did not find a matching property. Sep 12 11:32:53 myserver server: Sep 12, 2016 11:32:53 AM org.apache.tomcat.util.digester.SetPropertiesRule begin Sep 12 11:32:53 myserver server: WARNING: [SetPropertiesRule]{Server/Service/Engine/Host} Setting property 'xmlNamespaceAware' to 'false' did not find a matching property. Sep 12 11:32:53 myserver server: Sep 12, 2016 11:32:53 AM org.apache.tomcat.util.digester.SetPropertiesRule begin Sep 12 11:32:53 myserver server: WARNING: [SetPropertiesRule]{Server/Service/Engine/Host/Valve} Setting property 'resolveHosts' to 'false' did not find a matching property. Sep 12 11:32:53 myserver server: Sep 12, 2016 11:32:53 AM org.apache.catalina.startup.VersionLoggerListener log Sep 12 11:32:53 myserver server: INFO: Server version: Apache Tomcat/8.0.36 Sep 12 11:32:53 myserver server: Sep 12, 2016 11:32:53 AM org.apache.catalina.startup.VersionLoggerListener log Sep 12 11:32:53 myserver server: INFO: Server built: Aug 11 2016 18:10:25 UTC Sep 12 11:32:53 myserver server: Sep 12, 2016 11:32:53 AM org.apache.catalina.startup.VersionLoggerListener log Sep 12 11:32:53 myserver server: INFO: Server number: 8.0.36.0 Sep 12 11:32:53 myserver server: Sep 12, 2016 11:32:53 AM org.apache.catalina.startup.VersionLoggerListener log Sep 12 11:32:53 myserver server: INFO: OS Name: Linux Sep 12 11:32:53 myserver server: Sep 12, 2016 11:32:53 AM org.apache.catalina.startup.VersionLoggerListener log Sep 12 11:32:53 myserver server: INFO: OS Version: 4.7.2-201.fc24.x86_64 Sep 12 11:32:53 myserver server: Sep 12, 2016 11:32:53 AM org.apache.catalina.startup.VersionLoggerListener log Sep 12 11:32:53 myserver server: INFO: Architecture: amd64 Sep 12 11:32:53 myserver server: Sep 12, 2016 11:32:53 AM org.apache.catalina.startup.VersionLoggerListener log Sep 12 11:32:53 myserver server: INFO: Java Home: /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.102-1.b14.fc24.x86_64/jre Sep 12 11:32:53 myserver server: Sep 12, 2016 11:32:53 AM org.apache.catalina.startup.VersionLoggerListener log Sep 12 11:32:53 myserver server: INFO: JVM Version: 1.8.0_102-b14 Sep 12 11:32:53 myserver server: Sep 12, 2016 11:32:53 AM org.apache.catalina.startup.VersionLoggerListener log Sep 12 11:32:53 myserver server: INFO: JVM Vendor: Oracle Corporation Sep 12 11:32:53 myserver server: Sep 12, 2016 11:32:53 AM org.apache.catalina.startup.VersionLoggerListener log Sep 12 11:32:53 myserver server: INFO: CATALINA_BASE: /var/lib/pki/pki-tomcat Sep 12 11:32:53 myserver server: Sep 12, 2016 11:32:53 AM org.apache.catalina.startup.VersionLoggerListener log Sep 12 11:32:53 myserver server: INFO: CATALINA_HOME: /usr/share/tomcat Sep 12 11:32:53 myserver server: Sep 12, 2016 11:32:53 AM org.apache.catalina.startup.VersionLoggerListener log Sep 12 11:32:53 myserver server: INFO: Command line argument: -agentpath:/usr/lib/abrt-java-connector/libabrt-java-connector.so=abrt=on Sep 12 11:32:53 myserver server: Sep 12, 2016 11:32:53 AM org.apache.catalina.startup.VersionLoggerListener log Sep 12 11:32:53 myserver server: INFO: Command line argument: -DRESTEASY_LIB=/usr/share/java/resteasy Sep 12 11:32:53 myserver server: Sep 12, 2016 11:32:53 AM org.apache.catalina.startup.VersionLoggerListener log Sep 12 11:32:53 myserver server: INFO: Command line argument: -Djava.library.path=/usr/lib64/nuxwdog-jni Sep 12 11:32:53 myserver server: Sep 12, 2016 11:32:53 AM org.apache.catalina.startup.VersionLoggerListener log Sep 12 11:32:53 myserver server: INFO: Command line argument: -Dcatalina.base=/var/lib/pki/pki-tomcat Sep 12 11:32:53 myserver server: Sep 12, 2016 11:32:53 AM org.apache.catalina.startup.VersionLoggerListener log Sep 12 11:32:53 myserver server: INFO: Command line argument: -Dcatalina.home=/usr/share/tomcat Sep 12 11:32:53 myserver server: Sep 12, 2016 11:32:53 AM org.apache.catalina.startup.VersionLoggerListener log Sep 12 11:32:53 myserver server: INFO: Command line argument: -Djava.endorsed.dirs= Sep 12 11:32:53 myserver server: Sep 12, 2016 11:32:53 AM org.apache.catalina.startup.VersionLoggerListener log Sep 12 11:32:53 myserver server: INFO: Command line argument: -Djava.io.tmpdir=/var/lib/pki/pki-tomcat/temp Sep 12 11:32:53 myserver server: Sep 12, 2016 11:32:53 AM org.apache.catalina.startup.VersionLoggerListener log Sep 12 11:32:53 myserver server: INFO: Command line argument: -Djava.util.logging.config.file=/var/lib/pki/pki-tomcat/conf/logging.properties Sep 12 11:32:53 myserver server: Sep 12, 2016 11:32:53 AM org.apache.catalina.startup.VersionLoggerListener log Sep 12 11:32:53 myserver server: INFO: Command line argument: -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager Sep 12 11:32:53 myserver server: Sep 12, 2016 11:32:53 AM org.apache.catalina.startup.VersionLoggerListener log Sep 12 11:32:53 myserver server: INFO: Command line argument: -Djava.security.manager Sep 12 11:32:53 myserver server: Sep 12, 2016 11:32:53 AM org.apache.catalina.startup.VersionLoggerListener log Sep 12 11:32:53 myserver server: INFO: Command line argument: -Djava.security.policy==/var/lib/pki/pki-tomcat/conf/catalina.policy Sep 12 11:32:53 myserver server: PKIListener: org.apache.catalina.core.StandardServer[before_init] Sep 12 11:32:53 myserver server: Sep 12, 2016 11:32:53 AM org.apache.coyote.AbstractProtocol init Sep 12 11:32:53 myserver server: INFO: Initializing ProtocolHandler ["http-nio-8080"] Sep 12 11:32:53 myserver server: Sep 12, 2016 11:32:53 AM org.apache.tomcat.util.net.NioSelectorPool getSharedSelector Sep 12 11:32:53 myserver server: INFO: Using a shared selector for servlet write/read Sep 12 11:32:53 myserver server: Sep 12, 2016 11:32:53 AM org.apache.coyote.AbstractProtocol init Sep 12 11:32:53 myserver server: INFO: Initializing ProtocolHandler ["http-bio-8443"] Sep 12 11:32:53 myserver server: Error: SSL cipher "TLS_ECDH_RSA_WITH_3DES_EDE_CBC_SHA" not recognized by tomcatjss Sep 12 11:32:53 myserver server: Error: SSL cipher "TLS_ECDH_RSA_WITH_AES_128_CBC_SHA" not recognized by tomcatjss Sep 12 11:32:53 myserver server: Error: SSL cipher "TLS_ECDH_RSA_WITH_AES_256_CBC_SHA" not recognized by tomcatjss Sep 12 11:32:53 myserver server: Error: SSL cipher "TLS_RSA_WITH_3DES_EDE_CBC_SHA" not recognized by tomcatjss Sep 12 11:32:53 myserver server: Error: SSL cipher "TLS_DHE_DSS_WITH_3DES_EDE_CBC_SHA" not recognized by tomcatjss Sep 12 11:32:53 myserver server: Error: SSL cipher "TLS_DHE_RSA_WITH_3DES_EDE_CBC_SHA" not recognized by tomcatjss Sep 12 11:32:53 myserver server: Error: SSL cipher "TLS_ECDH_ECDSA_WITH_AES_128_GCM_SHA256" unsupported by NSS Sep 12 11:32:53 myserver server: Error: SSL cipher "TLS_ECDH_RSA_WITH_AES_128_GCM_SHA256" unsupported by NSS Sep 12 11:32:54 myserver server: Sep 12, 2016 11:32:54 AM org.apache.coyote.AbstractProtocol init Sep 12 11:32:54 myserver server: INFO: Initializing ProtocolHandler ["ajp-nio-127.0.0.1-8009"] Sep 12 11:32:54 myserver server: Sep 12, 2016 11:32:54 AM org.apache.tomcat.util.net.NioSelectorPool getSharedSelector Sep 12 11:32:54 myserver server: INFO: Using a shared selector for servlet write/read Sep 12 11:32:54 myserver server: PKIListener: org.apache.catalina.core.StandardServer[after_init] Sep 12 11:32:54 myserver server: Sep 12, 2016 11:32:54 AM org.apache.catalina.startup.Catalina load Sep 12 11:32:54 myserver server: INFO: Initialization processed in 646 ms Sep 12 11:32:54 myserver server: PKIListener: org.apache.catalina.core.StandardServer[before_start] Sep 12 11:32:54 myserver server: PKIListener: org.apache.catalina.core.StandardServer[configure_start] Sep 12 11:32:54 myserver server: PKIListener: org.apache.catalina.core.StandardServer[start] Sep 12 11:32:54 myserver server: Sep 12, 2016 11:32:54 AM org.apache.catalina.core.StandardService startInternal Sep 12 11:32:54 myserver server: INFO: Starting service Catalina Sep 12 11:32:54 myserver server: Sep 12, 2016 11:32:54 AM org.apache.catalina.core.StandardEngine startInternal Sep 12 11:32:54 myserver server: INFO: Starting Servlet Engine: Apache Tomcat/8.0.36 Sep 12 11:32:54 myserver server: Sep 12, 2016 11:32:54 AM org.apache.catalina.startup.HostConfig deployDescriptor Sep 12 11:32:54 myserver server: INFO: Deploying configuration descriptor /etc/pki/pki-tomcat/Catalina/localhost/ROOT.xml Sep 12 11:32:54 myserver ns-slapd: [12/Sep/2016:11:32:54.152571540 +0200] NSMMReplicationPlugin - agmt="cn=meTomyreplica.mydomain" (myreplica:389): Replication bind with GSSAPI auth resumed Sep 12 11:32:55 myserver server: Sep 12, 2016 11:32:55 AM org.apache.jasper.servlet.TldScanner scanJars Sep 12 11:32:55 myserver server: INFO: At least one JAR was scanned for TLDs yet contained no TLDs. Enable debug logging for this logger for a complete list of JARs that were scanned but no TLDs were found in them. Skipping unneeded JARs during scanning can improve startup time and JSP compilation time. Sep 12 11:32:55 myserver server: Sep 12, 2016 11:32:55 AM org.apache.catalina.startup.HostConfig deployDescriptor Sep 12 11:32:55 myserver server: INFO: Deployment of configuration descriptor /etc/pki/pki-tomcat/Catalina/localhost/ROOT.xml has finished in 1,361 ms Sep 12 11:32:55 myserver server: Sep 12, 2016 11:32:55 AM org.apache.catalina.startup.HostConfig deployDescriptor Sep 12 11:32:55 myserver server: INFO: Deploying configuration descriptor /etc/pki/pki-tomcat/Catalina/localhost/ca.xml Sep 12 11:32:55 myserver server: SSLAuthenticatorWithFallback: Creating SSL authenticator with fallback Sep 12 11:32:55 myserver server: SSLAuthenticatorWithFallback: Setting container Sep 12 11:32:56 myserver ns-slapd: [12/Sep/2016:11:32:56.173310963 +0200] schema-compat-plugin - Finished plugin initialization. Sep 12 11:32:56 myserver server: SSLAuthenticatorWithFallback: Initializing authenticators Sep 12 11:32:56 myserver server: SSLAuthenticatorWithFallback: Starting authenticators Sep 12 11:32:56 myserver server: Sep 12, 2016 11:32:56 AM org.apache.jasper.servlet.TldScanner scanJars Sep 12 11:32:56 myserver server: INFO: At least one JAR was scanned for TLDs yet contained no TLDs. Enable debug logging for this logger for a complete list of JARs that were scanned but no TLDs were found in them. Skipping unneeded JARs during scanning can improve startup time and JSP compilation time. Sep 12 11:32:56 myserver server: CMSEngine.initializePasswordStore() begins Sep 12 11:32:56 myserver server: CMSEngine.initializePasswordStore(): tag=internaldb Sep 12 11:32:56 myserver server: CMSEngine.initializePasswordStore(): tag=replicationdb Sep 12 11:32:57 myserver named-pkcs11[2120]: zone 21.0.10.in-addr.arpa/IN: sending notifies (serial 1473672772) Sep 12 11:32:57 myserver named-pkcs11[2120]: zone 1.168.192.in-addr.arpa/IN: sending notifies (serial 1473672772) Sep 12 11:32:57 myserver named-pkcs11[2120]: zone mydomain/IN: sending notifies (serial 1473672772) Sep 12 11:32:57 myserver server: CA is started. Sep 12 11:32:57 myserver server: Sep 12, 2016 11:32:57 AM org.apache.catalina.startup.HostConfig deployDescriptor Sep 12 11:32:57 myserver server: INFO: Deployment of configuration descriptor /etc/pki/pki-tomcat/Catalina/localhost/ca.xml has finished in 2,033 ms Sep 12 11:32:57 myserver server: Sep 12, 2016 11:32:57 AM org.apache.catalina.startup.HostConfig deployDescriptor Sep 12 11:32:57 myserver server: INFO: Deploying configuration descriptor /etc/pki/pki-tomcat/Catalina/localhost/pki#js.xml Sep 12 11:32:57 myserver server: Sep 12, 2016 11:32:57 AM org.apache.catalina.startup.SetContextPropertiesRule begin Sep 12 11:32:57 myserver server: WARNING: [SetContextPropertiesRule]{Context} Setting property 'allowLinking' to 'true' did not find a matching property. Sep 12 11:32:58 myserver server: Sep 12, 2016 11:32:58 AM org.apache.jasper.servlet.TldScanner scanJars Sep 12 11:32:58 myserver server: INFO: At least one JAR was scanned for TLDs yet contained no TLDs. Enable debug logging for this logger for a complete list of JARs that were scanned but no TLDs were found in them. Skipping unneeded JARs during scanning can improve startup time and JSP compilation time. Sep 12 11:32:58 myserver server: Sep 12, 2016 11:32:58 AM org.apache.catalina.startup.HostConfig deployDescriptor Sep 12 11:32:58 myserver server: INFO: Deployment of configuration descriptor /etc/pki/pki-tomcat/Catalina/localhost/pki#js.xml has finished in 703 ms Sep 12 11:32:58 myserver server: Sep 12, 2016 11:32:58 AM org.apache.catalina.startup.HostConfig deployDescriptor Sep 12 11:32:58 myserver server: INFO: Deploying configuration descriptor /etc/pki/pki-tomcat/Catalina/localhost/pki#admin.xml Sep 12 11:32:58 myserver server: Sep 12, 2016 11:32:58 AM org.apache.catalina.startup.SetContextPropertiesRule begin Sep 12 11:32:58 myserver server: WARNING: [SetContextPropertiesRule]{Context} Setting property 'allowLinking' to 'true' did not find a matching property. Sep 12 11:32:58 myserver server: Sep 12, 2016 11:32:58 AM org.apache.jasper.servlet.TldScanner scanJars Sep 12 11:32:58 myserver server: INFO: At least one JAR was scanned for TLDs yet contained no TLDs. Enable debug logging for this logger for a complete list of JARs that were scanned but no TLDs were found in them. Skipping unneeded JARs during scanning can improve startup time and JSP compilation time. Sep 12 11:32:57 myserver server: WARNING: [SetContextPropertiesRule]{Context} Setting property 'allowLinking' to 'true' did not find a matching property. Sep 12 11:32:58 myserver server: Sep 12, 2016 11:32:58 AM org.apache.jasper.servlet.TldScanner scanJars Sep 12 11:32:58 myserver server: INFO: At least one JAR was scanned for TLDs yet contained no TLDs. Enable debug logging for this logger for a complete list of JARs that were scanned but no TLDs were found in them. Skipping unneeded JARs during scanning can improve startup time and JSP compilation time. Sep 12 11:32:58 myserver server: Sep 12, 2016 11:32:58 AM org.apache.catalina.startup.HostConfig deployDescriptor Sep 12 11:32:58 myserver server: INFO: Deployment of configuration descriptor /etc/pki/pki-tomcat/Catalina/localhost/pki#js.xml has finished in 703 ms Sep 12 11:32:58 myserver server: Sep 12, 2016 11:32:58 AM org.apache.catalina.startup.HostConfig deployDescriptor Sep 12 11:32:58 myserver server: INFO: Deploying configuration descriptor /etc/pki/pki-tomcat/Catalina/localhost/pki#admin.xml Sep 12 11:32:58 myserver server: Sep 12, 2016 11:32:58 AM org.apache.catalina.startup.SetContextPropertiesRule begin Sep 12 11:32:58 myserver server: WARNING: [SetContextPropertiesRule]{Context} Setting property 'allowLinking' to 'true' did not find a matching property. Sep 12 11:32:58 myserver server: Sep 12, 2016 11:32:58 AM org.apache.jasper.servlet.TldScanner scanJars Sep 12 11:32:58 myserver server: INFO: At least one JAR was scanned for TLDs yet contained no TLDs. Enable debug logging for this logger for a complete list of JARs that were scanned but no TLDs were found in them. Skipping unneeded JARs during scanning can improve startup time and JSP compilation time. Sep 12 11:32:58 myserver server: Sep 12, 2016 11:32:58 AM org.apache.catalina.startup.HostConfig deployDescriptor Sep 12 11:32:58 myserver server: INFO: Deployment of configuration descriptor /etc/pki/pki-tomcat/Catalina/localhost/pki#admin.xml has finished in 779 ms Sep 12 11:32:58 myserver server: Sep 12, 2016 11:32:58 AM org.apache.coyote.AbstractProtocol start Sep 12 11:32:58 myserver server: INFO: Starting ProtocolHandler ["http-nio-8080"] Sep 12 11:32:58 myserver server: Sep 12, 2016 11:32:58 AM org.apache.coyote.AbstractProtocol start Sep 12 11:32:58 myserver server: INFO: Starting ProtocolHandler ["http-bio-8443"] Sep 12 11:32:58 myserver server: Sep 12, 2016 11:32:58 AM org.apache.coyote.AbstractProtocol start Sep 12 11:32:58 myserver server: INFO: Starting ProtocolHandler ["ajp-nio-127.0.0.1-8009"] Sep 12 11:32:58 myserver server: PKIListener: org.apache.catalina.core.StandardServer[after_start] Sep 12 11:32:58 myserver server: PKIListener: Subsystem CA is running. Sep 12 11:32:58 myserver server: Sep 12, 2016 11:32:58 AM org.apache.catalina.startup.Catalina start Sep 12 11:32:58 myserver server: INFO: Server startup in 4969 ms Sep 12 11:32:59 myserver systemd: Starting Samba SMB Daemon... Sep 12 11:32:59 myserver systemd: smb.service: Supervising process 2520 which is not our child. We'll most likely not notice when it exits. Sep 12 11:32:59 myserver smbd[2520]: [2016/09/12 11:32:59.521677, 0] ../lib/util/become_daemon.c:124(daemon_ready) Sep 12 11:32:59 myserver audit: SERVICE_START pid=1 uid=0 auid=4294967295 ses=4294967295 msg='unit=smb comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success' Sep 12 11:32:59 myserver smbd[2520]: STATUS=daemon 'smbd' finished starting up and ready to serve connections Sep 12 11:32:59 myserver systemd: Started Samba SMB Daemon. Sep 12 11:32:59 myserver systemd: Starting Samba Winbind Daemon... Sep 12 11:32:59 myserver systemd: winbind.service: Supervising process 2534 which is not our child. We'll most likely not notice when it exits. Sep 12 11:32:59 myserver winbindd[2534]: [2016/09/12 11:32:59.610689, 0] ../source3/winbindd/winbindd_cache.c:3245(initialize_winbindd_cache) Sep 12 11:32:59 myserver winbindd[2534]: initialize_winbindd_cache: clearing cache and re-creating with version number 2 Sep 12 11:32:59 myserver winbindd[2534]: [2016/09/12 11:32:59.612011, 0] ../lib/util/become_daemon.c:124(daemon_ready) Sep 12 11:32:59 myserver audit: SERVICE_START pid=1 uid=0 auid=4294967295 ses=4294967295 msg='unit=winbind comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success' Sep 12 11:32:59 myserver winbindd[2534]: STATUS=daemon 'winbindd' finished starting up and ready to serve connections Sep 12 11:32:59 myserver systemd: Started Samba Winbind Daemon. Sep 12 11:32:59 myserver systemd: Listening on ipa-otpd socket. Sep 12 11:32:59 myserver systemd: Started IPA key daemon. Sep 12 11:32:59 myserver audit: SERVICE_START pid=1 uid=0 auid=4294967295 ses=4294967295 msg='unit=ipa-dnskeysyncd comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success' Sep 12 11:32:59 myserver ipactl: ipa: INFO: The ipactl command was successful Sep 12 11:32:59 myserver ipactl: Starting Directory Service Sep 12 11:32:59 myserver ipactl: Starting krb5kdc Service Sep 12 11:32:59 myserver ipactl: Starting kadmin Service Sep 12 11:32:59 myserver ipactl: Starting named Service Sep 12 11:32:59 myserver ipactl: Starting ipa_memcached Service Sep 12 11:32:59 myserver ipactl: Starting httpd Service Sep 12 11:32:59 myserver ipactl: Starting pki-tomcatd Service Sep 12 11:32:59 myserver ipactl: Starting smb Service Sep 12 11:32:59 myserver ipactl: Starting winbind Service Sep 12 11:32:59 myserver ipactl: Starting ipa-otpd Service Sep 12 11:32:59 myserver ipactl: Starting ipa-dnskeysyncd Service Sep 12 11:32:59 myserver systemd: Started Identity, Policy, Audit. Sep 12 11:32:59 myserver audit: SERVICE_START pid=1 uid=0 auid=4294967295 ses=4294967295 msg='unit=ipa comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success' Sep 12 11:32:59 myserver ipa-dnskeysyncd: ipa: WARNING: session memcached servers not running Sep 12 11:33:00 myserver ipa-dnskeysyncd: ipa : INFO LDAP bind... Sep 12 11:33:00 myserver ipa-dnskeysyncd: ipa : INFO Commencing sync process Sep 12 11:33:00 myserver ipa-dnskeysyncd: ipa.ipapython.dnssec.keysyncer.KeySyncer: INFO Initial LDAP dump is done, sychronizing with ODS and BIND Sep 12 11:33:11 myserver systemd: Starting dnf makecache... On Mon, Sep 12, 2016 at 11:23 AM, Alexander Bokovoy wrote: > On Mon, 12 Sep 2016, Fujisan wrote: > >> Ok I installed the missing package and restarted ipa but it is still not >> woking. >> > We need logs. > > -- > / Alexander Bokovoy > -------------- next part -------------- An HTML attachment was scrubbed... URL: From amessina at messinet.com Mon Sep 12 09:55:24 2016 From: amessina at messinet.com (Anthony Joseph Messina) Date: Mon, 12 Sep 2016 04:55:24 -0500 Subject: [Freeipa-users] bind crashes on rndc reload In-Reply-To: <77002552-83b0-7e5d-0950-9d8f78b504de@winteltosh.de> References: <77002552-83b0-7e5d-0950-9d8f78b504de@winteltosh.de> Message-ID: <2892489.MTAJ5pYV2v@ws1.m.messinet.com> On Monday, September 12, 2016 10:31:10 AM CDT Jochen Demmer wrote: > Hi, > > I have a major issue with my setup: > Fedora 24 > freeipa-common-4.3.2-2.fc24.noarch > freeipa-admintools-4.3.2-2.fc24.noarch > freeipa-server-dns-4.3.2-2.fc24.noarch > freeipa-client-common-4.3.2-2.fc24.noarch > freeipa-server-4.3.2-2.fc24.x86_64 > freeipa-server-common-4.3.2-2.fc24.noarch > freeipa-client-4.3.2-2.fc24.x86_64 > bind-dyndb-ldap-9.0-3.fc24.x86_64 > bind-libs-lite-9.10.4-1.P2.fc24.x86_64 > bind-pkcs11-libs-9.10.4-1.P2.fc24.x86_64 > bind99-libs-9.9.9-1.P2.fc24.x86_64 > bind-utils-9.10.4-1.P2.fc24.x86_64 > rpcbind-0.2.3-11.rc1.fc24.x86_64 > bind-license-9.10.4-1.P2.fc24.noarch > bind-pkcs11-9.10.4-1.P2.fc24.x86_64 > bind-9.10.4-1.P2.fc24.x86_64 > bind-libs-9.10.4-1.P2.fc24.x86_64 > bind99-license-9.9.9-1.P2.fc24.noarch > bind-pkcs11-utils-9.10.4-1.P2.fc24.x86_64 > > It seems that there is a regular but not daily "rndc reload" sent to the > nameserver that leads to a crash of it. I sent a SIGHUP to the named > process, but that didn't lead to a crash. Only "rndc reload" does. It > does not crash EVERY time, but most of the times. I need to do an > "ipactl restart" in order to get the nameserver up and running again. > > I found this thread, but this doesn't give me any clues: > https://www.redhat.com/archives/freeipa-users/2012-May/msg00340.html > > This is what the log says: > http://paste.debian.net/818354/ > Please understand that I obfuscated my IP addresses and domain names > > This is the strace: > http://paste.debian.net/818365/ > > This is my named.conf: > http://paste.debian.net/818368/ > > Hope someone can help. > Jochen I wish I could, as I have the same issue, usually early Sunday morning after some cron/timer job that reloads: https://bugzilla.redhat.com/show_bug.cgi?id=1362162 -- Anthony - https://messinet.com/ - https://messinet.com/~amessina/gallery F9B6 560E 68EA 037D 8C3D D1C9 FF31 3BDB D9D8 99B6 -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 819 bytes Desc: This is a digitally signed message part. URL: From abokovoy at redhat.com Mon Sep 12 10:07:38 2016 From: abokovoy at redhat.com (Alexander Bokovoy) Date: Mon, 12 Sep 2016 13:07:38 +0300 Subject: [Freeipa-users] The Web UI is not loading In-Reply-To: References: <20160912085538.xn5mhhhy7woc7zis@redhat.com> <20160912092330.bg53mcg4gyl6d27c@redhat.com> Message-ID: <20160912100738.s7jyaptfhsmkissw@redhat.com> On Mon, 12 Sep 2016, Fujisan wrote: >Here is what i get when restarting ipa: > ># systemctl restart ipa [....] >Sep 12 11:32:59 myserver ipactl: ipa: INFO: The ipactl command was >successful >Sep 12 11:32:59 myserver ipactl: Starting Directory Service >Sep 12 11:32:59 myserver ipactl: Starting krb5kdc Service >Sep 12 11:32:59 myserver ipactl: Starting kadmin Service >Sep 12 11:32:59 myserver ipactl: Starting named Service >Sep 12 11:32:59 myserver ipactl: Starting ipa_memcached Service >Sep 12 11:32:59 myserver ipactl: Starting httpd Service >Sep 12 11:32:59 myserver ipactl: Starting pki-tomcatd Service >Sep 12 11:32:59 myserver ipactl: Starting smb Service >Sep 12 11:32:59 myserver ipactl: Starting winbind Service >Sep 12 11:32:59 myserver ipactl: Starting ipa-otpd Service >Sep 12 11:32:59 myserver ipactl: Starting ipa-dnskeysyncd Service >Sep 12 11:32:59 myserver systemd: Started Identity, Policy, Audit. So everything has started successfully. Any other problem? -- / Alexander Bokovoy From lslebodn at redhat.com Mon Sep 12 10:28:50 2016 From: lslebodn at redhat.com (Lukas Slebodnik) Date: Mon, 12 Sep 2016 12:28:50 +0200 Subject: [Freeipa-users] sssd stops after nss crashes In-Reply-To: References: Message-ID: <20160912102849.GE17138@10.4.128.1> On (12/09/16 11:09), Lachlan Musicman wrote: >We saw another sssd crash on the weekend (well, Friday night). > >Centos 7, sssd 1.14.0 from COPR > Please upgrade to 1.14.1 from copr. >Everything has worked fine for over a month until Friday. > >According to the log sssd_nss on the host in question: > > - at about 16:18, watchdog_handler killed a process for a timer overflow. > - there is some flopping about as nss/sssd tries to reconnect > - at 16:19:12 we see this: > >(Fri Sep 9 16:19:12 2016) [sssd[nss]] [sbus_dispatch] (0x0400): SBUS is >reconnecting. Deferring. > > - Which continues until 16:20:56 > >(Fri Sep 9 16:20:56 2016) [sssd[nss]] [sbus_dispatch] (0x0400): SBUS is >reconnecting. Deferring. > >Note that there are 9,573,091 lines of this, at about 80,000 msgs per >second. > > - nss seems to stumble back to life at this point (there are no logs on >the freeipa server unfortunately) > > - at every 15 min interval we see this (I think this might be zabbix >polling sssd): > >(Fri Sep 9 18:30:01 2016) [sssd[nss]] [get_client_cred] (0x0020): >SELINUX_getpeercon failed [-1][Unknown error -1]. What is a state of SELinux on your machine? Please share output of "sestatus" LS From fujisan43 at gmail.com Mon Sep 12 11:13:18 2016 From: fujisan43 at gmail.com (Fujisan) Date: Mon, 12 Sep 2016 13:13:18 +0200 Subject: [Freeipa-users] The Web UI is not loading In-Reply-To: <20160912100738.s7jyaptfhsmkissw@redhat.com> References: <20160912085538.xn5mhhhy7woc7zis@redhat.com> <20160912092330.bg53mcg4gyl6d27c@redhat.com> <20160912100738.s7jyaptfhsmkissw@redhat.com> Message-ID: Yes. I had to restart the browser. Now everything is working again. Thank you. On Mon, Sep 12, 2016 at 12:07 PM, Alexander Bokovoy wrote: > On Mon, 12 Sep 2016, Fujisan wrote: > >> Here is what i get when restarting ipa: >> >> # systemctl restart ipa >> > [....] > > Sep 12 11:32:59 myserver ipactl: ipa: INFO: The ipactl command was >> successful >> Sep 12 11:32:59 myserver ipactl: Starting Directory Service >> Sep 12 11:32:59 myserver ipactl: Starting krb5kdc Service >> Sep 12 11:32:59 myserver ipactl: Starting kadmin Service >> Sep 12 11:32:59 myserver ipactl: Starting named Service >> Sep 12 11:32:59 myserver ipactl: Starting ipa_memcached Service >> Sep 12 11:32:59 myserver ipactl: Starting httpd Service >> Sep 12 11:32:59 myserver ipactl: Starting pki-tomcatd Service >> Sep 12 11:32:59 myserver ipactl: Starting smb Service >> Sep 12 11:32:59 myserver ipactl: Starting winbind Service >> Sep 12 11:32:59 myserver ipactl: Starting ipa-otpd Service >> Sep 12 11:32:59 myserver ipactl: Starting ipa-dnskeysyncd Service >> Sep 12 11:32:59 myserver systemd: Started Identity, Policy, Audit. >> > So everything has started successfully. > > Any other problem? > -- > / Alexander Bokovoy > -------------- next part -------------- An HTML attachment was scrubbed... URL: From datakid at gmail.com Mon Sep 12 11:47:48 2016 From: datakid at gmail.com (Lachlan Musicman) Date: Mon, 12 Sep 2016 21:47:48 +1000 Subject: [Freeipa-users] sssd stops after nss crashes In-Reply-To: <20160912102849.GE17138@10.4.128.1> References: <20160912102849.GE17138@10.4.128.1> Message-ID: SELinux is disabled, updated to 1.14.1 today. This is the first crash in weeks, so we aren't that phased, although we'd love to know it wont happen again - the servers are part of a cluster that executes automated tasks as the data comes off genome sequencing machines - clinical medical analyses that is important for the patients & etc. Cheers L. ------ The most dangerous phrase in the language is, "We've always done it this way." - Grace Hopper On 12 September 2016 at 20:28, Lukas Slebodnik wrote: > On (12/09/16 11:09), Lachlan Musicman wrote: > >We saw another sssd crash on the weekend (well, Friday night). > > > >Centos 7, sssd 1.14.0 from COPR > > > Please upgrade to 1.14.1 from copr. > > >Everything has worked fine for over a month until Friday. > > > >According to the log sssd_nss on the host in question: > > > > - at about 16:18, watchdog_handler killed a process for a timer overflow. > > - there is some flopping about as nss/sssd tries to reconnect > > - at 16:19:12 we see this: > > > >(Fri Sep 9 16:19:12 2016) [sssd[nss]] [sbus_dispatch] (0x0400): SBUS is > >reconnecting. Deferring. > > > > - Which continues until 16:20:56 > > > >(Fri Sep 9 16:20:56 2016) [sssd[nss]] [sbus_dispatch] (0x0400): SBUS is > >reconnecting. Deferring. > > > >Note that there are 9,573,091 lines of this, at about 80,000 msgs per > >second. > > > > - nss seems to stumble back to life at this point (there are no logs on > >the freeipa server unfortunately) > > > > - at every 15 min interval we see this (I think this might be zabbix > >polling sssd): > > > >(Fri Sep 9 18:30:01 2016) [sssd[nss]] [get_client_cred] (0x0020): > >SELINUX_getpeercon failed [-1][Unknown error -1]. > What is a state of SELinux on your machine? > Please share output of "sestatus" > > LS > -------------- next part -------------- An HTML attachment was scrubbed... URL: From th at casalogic.dk Mon Sep 12 12:00:24 2016 From: th at casalogic.dk (Troels Hansen) Date: Mon, 12 Sep 2016 14:00:24 +0200 (CEST) Subject: [Freeipa-users] ipa-client requires ntp Message-ID: <1703960641.230333.1473681624605.JavaMail.zimbra@casalogic.dk> ipa-client (and ipa-server) RPM requires ntp. Shouldn't it be sufficient to req -- Med venlig hilsen Troels Hansen Systemkonsulent Casalogic A/S T (+45) 70 20 10 63 M (+45) 22 43 71 57 Red Hat, SUSE, VMware, Citrix, Novell, Yellowfin BI, EnterpriseDB, Sophos og meget mere. -------------- next part -------------- An HTML attachment was scrubbed... URL: From th at casalogic.dk Mon Sep 12 12:03:16 2016 From: th at casalogic.dk (Troels Hansen) Date: Mon, 12 Sep 2016 14:03:16 +0200 (CEST) Subject: [Freeipa-users] ipa-client requires ntp Message-ID: <1760702069.230482.1473681796930.JavaMail.zimbra@casalogic.dk> Not sure if this should actually go here? ipa-client (and ipa-server) RPM requires ntp. Shouldn't it be sufficient to require any tools that provides ntp functionality (at least ntp and chrony exists in RHEL) ? -- Med venlig hilsen Troels Hansen Systemkonsulent Casalogic A/S T (+45) 70 20 10 63 M (+45) 22 43 71 57 Red Hat, SUSE, VMware, Citrix, Novell, Yellowfin BI, EnterpriseDB, Sophos og meget mere. -------------- next part -------------- An HTML attachment was scrubbed... URL: From th at casalogic.dk Mon Sep 12 12:15:30 2016 From: th at casalogic.dk (Troels Hansen) Date: Mon, 12 Sep 2016 14:15:30 +0200 (CEST) Subject: [Freeipa-users] ipa-client requires ntp In-Reply-To: <1703960641.230333.1473681624605.JavaMail.zimbra@casalogic.dk> References: <1703960641.230333.1473681624605.JavaMail.zimbra@casalogic.dk> Message-ID: <924053852.230835.1473682530725.JavaMail.zimbra@casalogic.dk> Sorry for this half written email...... ----- On Sep 12, 2016, at 2:00 PM, Troels Hansen wrote: > ipa-client (and ipa-server) RPM requires ntp. > Shouldn't it be sufficient to req > -- > Med venlig hilsen > Troels Hansen > Systemkonsulent > Casalogic A/S > T (+45) 70 20 10 63 > M (+45) 22 43 71 57 > Red Hat, SUSE, VMware, Citrix, Novell, Yellowfin BI, EnterpriseDB, Sophos og > meget mere. > -- > Manage your subscription for the Freeipa-users mailing list: > https://www.redhat.com/mailman/listinfo/freeipa-users > Go to http://freeipa.org for more info on the project -- Med venlig hilsen Troels Hansen Systemkonsulent Casalogic A/S T (+45) 70 20 10 63 M (+45) 22 43 71 57 Red Hat, SUSE, VMware, Citrix, Novell, Yellowfin BI, EnterpriseDB, Sophos og meget mere. -------------- next part -------------- An HTML attachment was scrubbed... URL: From rcritten at redhat.com Mon Sep 12 12:54:39 2016 From: rcritten at redhat.com (Rob Crittenden) Date: Mon, 12 Sep 2016 08:54:39 -0400 Subject: [Freeipa-users] ipa-client requires ntp In-Reply-To: <1760702069.230482.1473681796930.JavaMail.zimbra@casalogic.dk> References: <1760702069.230482.1473681796930.JavaMail.zimbra@casalogic.dk> Message-ID: <57D6A58F.9090305@redhat.com> Troels Hansen wrote: > Not sure if this should actually go here? > > ipa-client (and ipa-server) RPM requires ntp. > Shouldn't it be sufficient to require any tools that provides ntp > functionality (at least ntp and chrony exists in RHEL) ? AFAIU there is no way to dynamically prefer one package or another at install time so one was picked. There is a ticket to use chrony if it is available, https://fedorahosted.org/freeipa/ticket/4669 rob > > > -- > > Med venlig hilsen > > *Troels Hansen* > > Systemkonsulent > > Casalogic A/S > > T (+45) 70 20 10 63 > > M (+45) 22 43 71 57 > > > > Red Hat, SUSE, VMware, Citrix, Novell, Yellowfin BI, EnterpriseDB, > Sophos og meget mere. > > > From rcritten at redhat.com Mon Sep 12 13:01:17 2016 From: rcritten at redhat.com (Rob Crittenden) Date: Mon, 12 Sep 2016 09:01:17 -0400 Subject: [Freeipa-users] Disable DNS checks using ipa-server-intall with FreeIPA 4.3.2 on Fedora 24? In-Reply-To: References: Message-ID: <57D6A71D.5000701@redhat.com> Richard Harmonson wrote: > Is there an option to disable the various DNS checks using > ipa-server-install with FreeIPA 4.3.2? Is there plans to do provide the > option in future releases? Reviewing the ipa-server-install man page, I > am not seeing it. > > I want to compliment the team for placing safeguards on the installation > of FreeIPA in respect to DNS best practices. I get the need to help some > of the installers that may not have a strong grasp of DNS. However, > there are scenarios where the checks becomes installation blockers or > result in undesirable results. I am not trying to be mysterious, but I > have no desire to embarrass third parties on their DNS faux pas. Thank you. > > Try --no-host-dns rob From rcritten at redhat.com Mon Sep 12 13:04:04 2016 From: rcritten at redhat.com (Rob Crittenden) Date: Mon, 12 Sep 2016 09:04:04 -0400 Subject: [Freeipa-users] problems with ipa server no longer responding to ldap In-Reply-To: References: Message-ID: <57D6A7C4.5060506@redhat.com> siology.io wrote: > Hello there. > > My setup is that i have five ipa servers. 2 in one location (alder, > auth-syd2), 2 in anouther location (auth-wlg, auth-wlg2), and one in yet > anouther location (waffle) which is reached over a long, > mostly-but-possibly-notably-not-entirely reliable vpn connection. > > I'm having an issue with an IPA server falling over. By 'falling over' > what i mean is that it no longer responds to ldap queries (although the > tcp port 389 is still open via nmap). When i run 'systemctl ipa stop' > the command never seems to return, so up to now the only fix i have it > to reboot that server. If 389-ds is hanging you'll want to follow these instructions to find out why, http://directory.fedoraproject.org/docs/389ds/FAQ/faq.html#debugging-hangs rob > > All machines are centos 7. All are using > ipa-server-4.2.0-15.0.1.el7.centos.18.x86_64. Replication occurs > between: alder<->auth-wlg, alder<->syd2, auth-wlg<->auth-wlg2, and > auth-wlg<->waffle, possibly notably *not* between alder and waffle directly. > > The problem of ldap being unavailable occurs on alder only; the other > ipa servers seem to be reliable. Unfortunately, alder is also our most > used server. > > The error logs off alder look like this: http://pastebin.com/TxCVjWTe > with reboot done at around 19:55 > > I did notice upon investigating / googling the errors in this log - > starting with the attr_replace (nsslapd-referral) one, that on my > servers this ldap query: > > ldapsearch -ZZ -h alder.blah.com -D > "cn=Directory Manager" -W -b "o=ipaca" > "(&(objectclass=nstombstone)(nsUniqueId=ffffffff-ffffffff-ffffffff-ffffffff))" > | grep "nsds50ruv\|nsDS5ReplicaId" > > returns results similar to this: > > nsDS5ReplicaId: 96 > nsds50ruv: {replicageneration} 5733d428000000600000 > nsds50ruv: {replica 96 ldap://alder.blah.com:389 > } 5733d474000000600000 57 > nsds50ruv: {replica 91 ldap://auth-syd2.blah.com:389 > } 576337b90004005b000 > nsds50ruv: {replica 97 ldap://auth-wlg.blah.com:389 > } 5733d49a000000610000 > nsds50ruv: {replica 1095 ldap://auth-wlg2.blah.com:389 > } 574fa5b0000004470 > nsds50ruv: {replica 1090 ldap://waffle.bsh.blah.com:389 > } 576b1add00000442 > nsds50ruv: {replica 1085 ldap://waffle.bsh.blah.com:389 > } 576b22f10000043d > > i.e: waffle is listed twice. If i run that ldap query on waffle though, > i get no results at all (but the command does at least return). - so i > dont know waffle's nsDS5ReplicaId at the moment. I understand once i > know that i can clean-ruv the other id off the other ipa servers? I > don't *think* any of this is related to my original issue above though, > but it might be a smoking gun, i don't know - just mentioning it in case. > > At the moment i've not got a lot to go on. Has anyone else seen errors > like those in the paste bin, or might know where to look for more useful > info ? Possibly also worth noting that alder, and auth-syd2 are AWS ec2 > instances. The rest are vm's on site(s). > > > From th at casalogic.dk Mon Sep 12 13:11:47 2016 From: th at casalogic.dk (Troels Hansen) Date: Mon, 12 Sep 2016 15:11:47 +0200 (CEST) Subject: [Freeipa-users] ipa-client requires ntp In-Reply-To: <57D6A58F.9090305@redhat.com> References: <1760702069.230482.1473681796930.JavaMail.zimbra@casalogic.dk> <57D6A58F.9090305@redhat.com> Message-ID: <917323824.232849.1473685907550.JavaMail.zimbra@casalogic.dk> ----- On Sep 12, 2016, at 2:54 PM, Rob Crittenden rcritten at redhat.com wrote: > Troels Hansen wrote: >> Not sure if this should actually go here? >> >> ipa-client (and ipa-server) RPM requires ntp. >> Shouldn't it be sufficient to require any tools that provides ntp >> functionality (at least ntp and chrony exists in RHEL) ? > > AFAIU there is no way to dynamically prefer one package or another at > install time so one was picked. > > There is a ticket to use chrony if it is available, > https://fedorahosted.org/freeipa/ticket/4669 > Well, you got me there. I just figured that RPM also had boolean dependency resolution, but I can see that it first appeared in RPM 4.13 (RHEL 7 running 4.11). That makes things a bit harder. Anyway, crony support would defenently be a good thing, or at least not require ntp. From richard.harmonson at gmail.com Mon Sep 12 13:49:14 2016 From: richard.harmonson at gmail.com (Richard Harmonson) Date: Mon, 12 Sep 2016 06:49:14 -0700 Subject: [Freeipa-users] Disable DNS checks using ipa-server-intall with FreeIPA 4.3.2 on Fedora 24? In-Reply-To: <0732e66a-115a-f3f7-610c-0b41dc8666c8@redhat.com> References: <0732e66a-115a-f3f7-610c-0b41dc8666c8@redhat.com> Message-ID: Thank you, Martin. '--allow-zone-overlap' may indeed fix one of the challenges. I will give it a try. Another check that is not a blocker but undesirable is the reverse zone lookup. The installer does a check and some turkey upstream of my infrastructure has a zone for 192.168.101.0 in a public DNS hosts. Bizarre! Not exactly a blocker for I assume I can add it after the installation. On Mon, Sep 12, 2016 at 1:41 AM, Martin Basti wrote: > > > On 11.09.2016 20:15, Richard Harmonson wrote: > > Is there an option to disable the various DNS checks using > ipa-server-install with FreeIPA 4.3.2? Is there plans to do provide the > option in future releases? Reviewing the ipa-server-install man page, I am > not seeing it. > > I want to compliment the team for placing safeguards on the installation > of FreeIPA in respect to DNS best practices. I get the need to help some of > the installers that may not have a strong grasp of DNS. However, there are > scenarios where the checks becomes installation blockers or result in > undesirable results. I am not trying to be mysterious, but I have no desire > to embarrass third parties on their DNS faux pas. Thank you. > > > > Hello, which check exactly do you mean? > > For zone overlap there is --allow-zone-overlap option. This was added in > 4.3 > > Martin > -------------- next part -------------- An HTML attachment was scrubbed... URL: From richard.harmonson at gmail.com Mon Sep 12 13:51:06 2016 From: richard.harmonson at gmail.com (Richard Harmonson) Date: Mon, 12 Sep 2016 06:51:06 -0700 Subject: [Freeipa-users] Disable DNS checks using ipa-server-intall with FreeIPA 4.3.2 on Fedora 24? In-Reply-To: <57D6A71D.5000701@redhat.com> References: <57D6A71D.5000701@redhat.com> Message-ID: On Mon, Sep 12, 2016 at 6:01 AM, Rob Crittenden wrote: > Richard Harmonson wrote: > >> Is there an option to disable the various DNS checks using >> ipa-server-install with FreeIPA 4.3.2? Is there plans to do provide the >> option in future releases? Reviewing the ipa-server-install man page, I >> am not seeing it. >> >> I want to compliment the team for placing safeguards on the installation >> of FreeIPA in respect to DNS best practices. I get the need to help some >> of the installers that may not have a strong grasp of DNS. However, >> there are scenarios where the checks becomes installation blockers or >> result in undesirable results. I am not trying to be mysterious, but I >> have no desire to embarrass third parties on their DNS faux pas. Thank >> you. >> >> >> > Try --no-host-dns > > rob > Saw the option, but didn't think it was applicable. No harm in trying it. Thank you, Rob. -------------- next part -------------- An HTML attachment was scrubbed... URL: From rakesh.rajasekharan at gmail.com Mon Sep 12 14:30:44 2016 From: rakesh.rajasekharan at gmail.com (Rakesh Rajasekharan) Date: Mon, 12 Sep 2016 20:00:44 +0530 Subject: [Freeipa-users] Increase ListenBacklog for httpd In-Reply-To: References: Message-ID: can anyone provide some insight on this please.. I have been trying to debug a hang issues for past few weeks.. and finally foudn that it starts with this issue when I see a lot of connections in SYN_RECV state. as it is happening now netstat shows around 14-16 connectiosn in SYNC_RECV If I could get some inputs on this , I could have some workaround to mitigate the issues Thanks On Mon, Sep 12, 2016 at 10:13 AM, Rakesh Rajasekharan < rakesh.rajasekharan at gmail.com> wrote: > sorry I guess I did not put the question correctly.... > > I wanted to know .. like we have the ListenBacklog for apache to basically > define the number of connections it can handle.. do we have some thing > similar for our krb5kdc service.. as the SYN floodin at 88 looks like > krb5kdc service is not able to handle sudden spurt in connections or the > number of connections are more than it could handle.. > > So, would be great if I could know how many connection it can support at > any given time ..most of the times I see this error while i add clients to > IPA master.. so if thers a known limit , I could first check netstat to see > how many connections I have at any point and if its below the limit only > then setup ipa-client-install > > Thanks, > > Rakesh > > On Sun, Sep 11, 2016 at 11:10 PM, Rakesh Rajasekharan < > rakesh.rajasekharan at gmail.com> wrote: > >> Hi, >> >> In my Freeipa setup, I frequently see this message >> >> request_sock_TCP: Possible SYN flooding on port 88. Sending cookies >> >> >> Is there a way to increase the ListenBacklog so that I can workaround >> this error as suggested i this doc >> https://access.redhat.com/solutions/30453 >> >> Thanks, >> Rakesh >> >> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From lslebodn at redhat.com Mon Sep 12 15:41:33 2016 From: lslebodn at redhat.com (Lukas Slebodnik) Date: Mon, 12 Sep 2016 17:41:33 +0200 Subject: [Freeipa-users] sssd stops after nss crashes In-Reply-To: References: <20160912102849.GE17138@10.4.128.1> Message-ID: <20160912154133.GP17138@10.4.128.1> On (12/09/16 21:47), Lachlan Musicman wrote: >SELinux is disabled, updated to 1.14.1 today. > >This is the first crash in weeks, so we aren't that phased, although we'd >love to know it wont happen again BTW Did it really crashed? Do you have a coredump We fixed few bad bugs(regressions) in 1.14.1 and in upstream. e.g. https://fedorahosted.org/sssd/ticket/3154 https://fedorahosted.org/sssd/ticket/3163 >- the servers are part of a cluster that >executes automated tasks as the data comes off genome sequencing machines - >clinical medical analyses that is important for the patients & etc. > I though it's prefered to use stable official release in such enviroment :-) Copr repo[1] is not an official release. Anyway, this copr repo is backported version from fedora and I do not know about any crash report or bug from fedora users. 1.14.1-2 should work well. LS [1] https://copr.fedorainfracloud.org/coprs/g/sssd/sssd-1-14/ From jalvarez at cyberfuel.com Mon Sep 12 16:00:57 2016 From: jalvarez at cyberfuel.com (Jose Alvarez R.) Date: Mon, 12 Sep 2016 10:00:57 -0600 Subject: [Freeipa-users] About AllowGroups with sshd Message-ID: <0afb01d20d0e$d99b9100$8cd2b300$@cyberfuel.com> Hello I have an question I have an FreeIPA 3.0 server(CentOS 6) with some clients servers(CentOS 6). I wants enable root a two servers this servers, because they are backup servers. I add theses lines in /etc/ssh/sshd_config of a client server. AllowUsers root at 192.168.20.2 AllowUsers root at 192.168.20.90 PermitRootLogin yes This working, but when try login with my user IPA, I can't login. I add the line "AllowGroups" with my group of users_IPA AllowGroups But not working, Can you help me ? Thanks, Regards Jose Alvarez. -------------- next part -------------- An HTML attachment was scrubbed... URL: From g.kafataridis at nelios.com Mon Sep 12 17:19:17 2016 From: g.kafataridis at nelios.com (Georgios Kafataridis) Date: Mon, 12 Sep 2016 20:19:17 +0300 Subject: [Freeipa-users] CA: Cannot add Centos7.2 replica to Centos6.8 ipa server In-Reply-To: References: <45b66871-aa01-3c91-0630-0bbeee488ae7@nelios.com> <323b9de1-adb3-47af-d4a7-d6083f88d75c@nelios.com> <9afdf0f3-97d1-cc6e-5f3d-0e11687a124a@redhat.com> <79782dbf-1051-b4de-88c8-b035b2d0c8bb@redhat.com> Message-ID: So, does anyone understand something more than me from the logs ? Can I search for something that can help me solve it ? On 9/9/2016 11:26 ??, Georgios Kafataridis wrote: > These are fresh logs from a last attempt to create a replica > > Centos 7 > > /var/log/pki/pki-tomcat/ca/debug > > > [09/Sep/2016:22:59:40][http-bio-8443-exec-3]: === Token Panel === > [09/Sep/2016:22:59:40][http-bio-8443-exec-3]: === Security Domain > Panel === > [09/Sep/2016:22:59:40][http-bio-8443-exec-3]: Joining existing > security domain > [09/Sep/2016:22:59:40][http-bio-8443-exec-3]: Resolving security > domain URLhttps://ipa-server.nelios:443 > [09/Sep/2016:22:59:40][http-bio-8443-exec-3]: Getting security domain > cert chain > [09/Sep/2016:22:59:40][http-bio-8443-exec-3]: Getting install token > [09/Sep/2016:22:59:40][http-bio-8443-exec-3]: Getting install token > [09/Sep/2016:22:59:41][http-bio-8443-exec-3]: Getting old cookie > [09/Sep/2016:22:59:41][http-bio-8443-exec-3]: Token: null > [09/Sep/2016:22:59:41][http-bio-8443-exec-3]: Install token is null > [09/Sep/2016:22:59:41][http-bio-8443-exec-3]: Failed to obtain > installation token from security domain > > > Centos 6 > > /var/log/pki-ca/debug > > [09/Sep/2016:22:59:42][TP-Processor3]: GetCookie before auth, url > =https://ipa2-server2.nelios:443/ca/admin/console/config/wizard?p=5&subsystem=CA > [09/Sep/2016:22:59:42][TP-Processor3]: IP: 192.168.4.175 > [09/Sep/2016:22:59:42][TP-Processor3]: AuthMgrName: passwdUserDBAuthMgr > [09/Sep/2016:22:59:42][TP-Processor3]: CMSServlet: no client > certificate found > [09/Sep/2016:22:59:42][TP-Processor3]: Authentication: UID=admin > [09/Sep/2016:22:59:42][TP-Processor3]: In LdapBoundConnFactory::getConn() > [09/Sep/2016:22:59:42][TP-Processor3]: masterConn is connected: true > [09/Sep/2016:22:59:42][TP-Processor3]: getConn: conn is connected true > [09/Sep/2016:22:59:42][TP-Processor3]: getConn: mNumConns now 2 > [09/Sep/2016:22:59:42][TP-Processor3]: LdapAnonConnFactory::getConn > [09/Sep/2016:22:59:42][TP-Processor3]: LdapAnonConnFactory.getConn(): > num avail conns now 2 > [09/Sep/2016:22:59:42][TP-Processor3]: returnConn: mNumConns now 3 > [09/Sep/2016:22:59:42][TP-Processor3]: returnConn: mNumConns now 2 > [09/Sep/2016:22:59:42][TP-Processor3]: SignedAuditEventFactory: > create() > message=[AuditEvent=AUTH_FAIL][SubjectID=$Unidentified$][Outcome=Failure][AuthMgr=passwdUserDBAuthMgr][AttemptedCred=$Unidentified$] > authentication failure > > [09/Sep/2016:22:59:42][TP-Processor3]: GetCookie authentication failed > [09/Sep/2016:22:59:42][TP-Processor3]: > mErrorFormPath=/admin/ca/securitydomainlogin.template > [09/Sep/2016:22:59:42][TP-Processor3]: CMSServlet: curDate=Fri Sep 09 > 22:59:42 EEST 2016 id=caGetCookie time=39 > > /var/log/httpd/access_log > > 192.168.4.175 - - [09/Sep/2016:22:59:21 +0300] "GET > /ca/rest/securityDomain/domainInfo HTTP/1.1" 404 315 > 192.168.4.175 - - [09/Sep/2016:22:59:22 +0300] "GET > /ca/admin/ca/getDomainXML HTTP/1.1" 200 1148 > 192.168.4.175 - - [09/Sep/2016:22:59:22 +0300] "GET > /ca/rest/account/login HTTP/1.1" 404 303 > 192.168.4.175 - - [09/Sep/2016:22:59:41 +0300] "POST > /ca/admin/ca/getCertChain HTTP/1.0" 200 1398 > 192.168.4.175 - - [09/Sep/2016:22:59:42 +0300] "GET > /ca/rest/account/login HTTP/1.1" 404 303 > 192.168.4.175 - - [09/Sep/2016:22:59:42 +0300] "POST > /ca/admin/ca/getCookie HTTP/1.1" 200 5170 > > /var/log/httpd/error_log > > [Fri Sep 09 22:59:22 2016] [error] [client 192.168.4.175] File does > not exist: /var/www/html/ca > [Fri Sep 09 22:59:22 2016] [error] [client 192.168.4.175] File does > not exist: /var/www/html/ca > [Fri Sep 09 22:59:42 2016] [error] [client 192.168.4.175] File does > not exist: /var/www/html/ca > > > /var/log/pki-ca/system > > 5337.TP-Processor3 - [09/Sep/2016:22:59:42 EEST] [6] [6] Failed to > authenticate as admin UID=admin. Error: netscape.ldap.LDAPException: > error result (49) > 5337.TP-Processor3 - [09/Sep/2016:22:59:42 EEST] [3] [3] Servlet > caGetCookie: Error getting servlet output stream when rendering > template. Error Invalid Credential.. > > /var/log/pki-ca/catalina.out > > Sep 08, 2016 4:17:34 PM org.apache.catalina.startup.HostConfig > deployDirectory > INFO: Deploying web application directory ROOT > Sep 08, 2016 4:17:34 PM org.apache.coyote.http11.Http11Protocol start > INFO: Starting Coyote HTTP/1.1 on http-9180 > Sep 08, 2016 4:17:34 PM org.apache.coyote.http11.Http11Protocol start > INFO: Starting Coyote HTTP/1.1 on http-9443 > Sep 08, 2016 4:17:35 PM org.apache.coyote.http11.Http11Protocol start > INFO: Starting Coyote HTTP/1.1 on http-9445 > Sep 08, 2016 4:17:35 PM org.apache.coyote.http11.Http11Protocol start > INFO: Starting Coyote HTTP/1.1 on http-9444 > Sep 08, 2016 4:17:35 PM org.apache.coyote.http11.Http11Protocol start > INFO: Starting Coyote HTTP/1.1 on http-9446 > Sep 08, 2016 4:17:35 PM org.apache.jk.common.ChannelSocket init > INFO: JK: ajp13 listening on /0.0.0.0:9447 > Sep 08, 2016 4:17:35 PM org.apache.jk.server.JkMain start > INFO: Jk running ID=0 time=0/123 config=null > Sep 08, 2016 4:17:35 PM org.apache.catalina.startup.Catalina start > INFO: Server startup in 8550 ms > > Catalina seems to not have logged anything from yesterday. > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From rakesh.rajasekharan at gmail.com Mon Sep 12 17:53:12 2016 From: rakesh.rajasekharan at gmail.com (Rakesh Rajasekharan) Date: Mon, 12 Sep 2016 23:23:12 +0530 Subject: [Freeipa-users] Freeipa 4.2.0 slow response Message-ID: Hi, I am experiencing a very slow response from freeipa.. the new passwords that I am resetting are never working for the users and its takes a lot of time for an existing user to login around 25 secs. doing a kinit admin itself is very slowKRB5_TRACE=/dev/stderr kinit admin [11298] 1473702491.60880: Getting initial credentials for admin at XYZ.COM [11298] 1473702491.62981: Sending request (167 bytes) to XYZ.COM [11298] 1473702491.63119: Initiating TCP connection to stream 10.1.3.35:88 [11298] 1473702491.63359: Sending TCP request to stream 10.1.3.35:88 [11298] 1473702493.797835: Received answer (341 bytes) from stream 10.1.3.35:88 [11298] 1473702493.797848: Terminating TCP connection to stream 10.1.3.35:88 [11298] 1473702493.797911: Response was from master KDC [11298] 1473702493.797956: Received error from KDC: -1765328359/Additional pre-authentication required [11298] 1473702493.797993: Processing preauth types: 136, 19, 2, 133 [11298] 1473702493.798005: Selected etype info: etype aes256-cts, salt "V at Cbu147E#1;R0WD", params "" [11298] 1473702493.798009: Received cookie: MIT Password for admin at XYZ.COM: [11298] 1473702498.190064: AS key obtained for encrypted timestamp: aes256-cts/2C9D [11298] 1473702498.190109: Encrypted timestamp (for 1473702498.184527): plain 301AA011180F32303136303931323137343831385AA105020302D0CF, encrypted 25FC8D37EFB6B7837C8D5C6649DFB9972010D40EE29D1222FBA45CAA98428E42C7FCC9B7FE881A04BD3390A6A9EDE9D2D93729FDF3E47B6D [11298] 1473702498.190129: Preauth module encrypted_timestamp (2) (real) returned: 0/Success [11298] 1473702498.190133: Produced preauth for next request: 133, 2 [11298] 1473702498.190148: Sending request (261 bytes) to XYZ.COM [11298] 1473702498.190246: Initiating TCP connection to stream 10.1.3.35:88 [11298] 1473702499.191933: Sending initial UDP request to dgram 10.1.3.35:88 [11298] 1473702502.195157: Sending retry UDP request to dgram 10.1.3.35:88 [11298] 1473702507.200405: Sending retry UDP request to dgram 10.1.3.35:88 [11298] 1473702513.226371: Sending TCP request to stream 10.1.3.35:88 [11298] 1473702515.797243: Received answer (730 bytes) from stream 10.1.3.35:88 [11298] 1473702515.797271: Terminating TCP connection to stream 10.1.3.35:88 [11298] 1473702515.797326: Response was from master KDC [11298] 1473702515.797353: Processing preauth types: 19 [11298] 1473702515.797360: Selected etype info: etype aes256-cts, salt "V at Cbu147E#1;R0WD", params "" [11298] 1473702515.797394: Produced preauth for next request: (empty) [11298] 1473702515.797401: AS key determined by preauth: aes256-cts/2C9D [11298] 1473702515.797445: Decrypted AS reply; session key is: aes256-cts/702E [11298] 1473702515.797460: FAST negotiation: available [11298] 1473702515.797478: Initializing KEYRING:persistent:0:0 with default princ admin at XYZ.COM [11298] 1473702515.797534: Storing admin at XYZ.COM -> krbtgt/XYZ.COM at XYZ.COM in KEYRING:persistent:0:0 [11298] 1473702515.797572: Storing config in KEYRING:persistent:0:0 for krbtgt/XYZ.COM at XYZ.COM: fast_avail: yes [11298] 1473702515.797585: Storing admin at XYZ.COM -> krb5_ccache_conf_data/fast_avail/krbtgt\/XYZ.COM\@XYZ.COM at X-CACHECONF: in KEYRING:persistent:0:0 [11298] 1473702515.797631: Storing config in KEYRING:persistent:0:0 for krbtgt/XYZ.COM at XYZ.COM: pa_type: 2 [11298] 1473702515.797647: Storing admin at XYZ.COM -> krb5_ccache_conf_data/pa_type/krbtgt\/XYZ.COM\@XYZ.COM at X-CACHECONF: in KEYRING:persistent:0:0 are any pointers as to what could be causing this slowness Thanks Rakesh -------------- next part -------------- An HTML attachment was scrubbed... URL: From rcritten at redhat.com Mon Sep 12 19:48:06 2016 From: rcritten at redhat.com (Rob Crittenden) Date: Mon, 12 Sep 2016 15:48:06 -0400 Subject: [Freeipa-users] ipa: ERROR: Certificate format error: (SEC_ERROR_LEGACY_DATABASE) The certificate/key database is in an old, unsupported format. In-Reply-To: References: <57D015D8.7080507@redhat.com> <57D166B9.3070201@redhat.com> Message-ID: <57D70676.8080107@redhat.com> Natxo Asenjo wrote: > hi, > > I can reproduce this everytime. Restarting httpd fixes it for a while, > but then ik stops working: > > $ ipa cert-show 1 > ipa: ERROR: cannot connect to > 'https://kdc01.unix.domain.tld:443/ca/agent/ca/displayBySerial': > (SEC_ERROR_LEGACY_DATABASE) The certificate/key database is in an old, > unsupported format. It is very strange that it goes from a working to a non-working state. I have only two suggestions: 1. Create /etc/ipa/server.conf with a [global] section and debug=True in it, restart httpd. Your log will be quite a bit more verbose but given it reproduces so quickly hopefully won't be too big a deal. That might show something. 2. Try brute force with strace. Finding the right httpd process to strace can be frustrating but usually there are only 8 and they rotate so eventually you should get the right one. rob > [jose.admin at kdc01 ~]$ sudo /etc/init.d/httpd restart > Stopping httpd: [ OK ] > Starting httpd: [ OK ] > [jose.admin at kdc01 ~]$ ipa cert-show 1 > Certificate: > MIIDnDCCAoSgAwIBAgIBATANBgkqhkiG9w0BAQsFADA7MRkwFwYDVQQKExBVTklY > LklSSVNaT1JHLk5MMR4wHAYDVQQDExVDZXJ0aWZpY2F0ZSBBdXRob3JpdHkwHhcN > MTIxMTA3MjEyNDE1WhcNMjAxMTA3MjEyNDE1WjA7MRkwFwYDVQQKExBVTklYLklS > SVNaT1JHLk5MMR4wHAYDVQQDExVDZXJ0aWZpY2F0ZSBBdXRob3JpdHkwggEiMA0G > CSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQCy2WVy7QkHiuENW/zkMeD4ILoqOruu > YKvb2+rqeuI9iw+zBBt569XSxrgcyeTq0G63RjbXgrAzot4EhYg6MoepDVCn0Bnu > rUfgbCf5R0Eboigjboh5MGnPylHefLRGARNUCwcTGA4uR9ZQL/rEUqWktmZjanYE > vOP8UBeuq5WP5emaX8U03SzMA+cQT9w/zx0eAOYgZW5yx3aA5Q4Fu8qWqMGGAOA6 > yDQWqmIpgxiFHHRa7hQK4AjeHgvaColaU979Lh5jAv/XwrYtok1G+UVEp45INpfx > r5dLe03ognPFPZ0/xwbBqtt/2qn6rk4L4ukH4P9g4Rw0o7U1yJVx/SOJAgMBAAGj > gaowgacwHwYDVR0jBBgwFoAUo5fkii64zz7qM/K8k9Yj3qmENmgwDwYDVR0TAQH/ > BAUwAwEB/zAOBgNVHQ8BAf8EBAMCAcYwHQYDVR0OBBYEFKOX5IouuM8+6jPyvJPW > I96phDZoMEQGCCsGAQUFBwEBBDgwNjA0BggrBgEFBQcwAYYoaHR0cDovL2tkYzAx > LnVuaXguaXJpc3pvcmcubmw6ODAvY2Evb2NzcDANBgkqhkiG9w0BAQsFAAOCAQEA > J28gdozd/ptOM5PTKKwyV+otO/wk3yErslxpNUhRZgSNUwT+t6tfF/j+jJRV5sX+ > jy09c9Do+p3Hy9gRnIVJONDScvMV9nDc75C6JGXU+FdNJJ+Dbpep/RsQjHrZ+unw > IyAWoOpBol8sGzN5tXbeo/M6mGFxaBTH1GKtgv4CKbzQAotvMaGxzKjScHRsGaer > NSCZp/90yRJypC3MOosUFcFl4CoYHB42XDTzjvzZQcaFNcgYXOciujwwYHNzsSqY > cIKFSWuWvN++7g4yxQMlu8QW0Ms/PntmTmO2cDdNI1tujVyBKe599y4O/Es/MBGt > DtVA85ALksJOU27bjtvbBg== > Subject: CN=Certificate Authority,O=UNIX.DOMAIN.TLD > Issuer: CN=Certificate Authority,O=UNIX.DOMAIN.TLD > Not Before: Wed Nov 07 21:24:15 2012 UTC > Not After: Sat Nov 07 21:24:15 2020 UTC > Fingerprint (MD5): 28:18:34:9d:03:99:b8:ff:2b:bd:55:0a:65:bf:d4:f2 > Fingerprint (SHA1): > 6f:e1:a4:4f:47:ec:9c:c4:ad:b9:b9:fc:e8:f4:33:4b:0a:cb:43:3e > Serial number (hex): 0x1 > Serial number: 1 > > And a few minutes later (5, maximum 10), then I get the > SEC_ERROR_LEGACY_DATABASE error. No traceback in /var/log/httpd/error_log. > > This is the first CA domain controller. > > I am leaving this job in a few weeks, so I would like to leave > everything working properly. Would it be better to upgrade the domain > controllers to centos 7 (right now running centos 6.8, fully patched). > > Thanks for your input. > > -- > regards, > natxo > > > > On Thu, Sep 8, 2016 at 6:30 PM, Natxo Asenjo > wrote: > > > > On Thu, Sep 8, 2016 at 3:25 PM, Rob Crittenden > wrote: > > Natxo Asenjo wrote: > > I do see these errors: > [Wed Sep 07 15:56:13 2016] [error] ipa: INFO:: ping(): SUCCESS > [Wed Sep 07 15:56:13 2016] [error] ipa: INFO: : > host_find(u'tftp-1801', > all=False, raw=False, version=u'2.49', no_members=False, > pkey_only=False): CertificateFormatError > [Wed Sep 07 15:56:44 2016] [error] ipa: INFO: : ping(): SUCCESS > [Wed Sep 07 15:56:44 2016] [error] ipa: INFO: : > host_find(u'tftp-1801', > all=False, raw=False, version=u'2.49', no_members=False, > pkey_only=False): CertificateFormatError > [Wed Sep 07 15:57:57 2016] [error] ipa: INFO: : ping(): SUCCESS > [Wed Sep 07 15:57:58 2016] [error] ipa: INFO: : > host_find(u'tftp-1801', > all=False, raw=False, version=u'2.49', no_members=False, > pkey_only=False): CertificateFormatErro > > > On Wed, Sep 7, 2016 at 4:01 PM, Natxo Asenjo > > >> wrote: > > > alas, not woriking again. > > On the one kdc > > $ ipa host-find tftp-1801 > ipa: ERROR: Certificate format error: > (SEC_ERROR_LEGACY_DATABASE) > The certificate/key database is in an old, unsupported > format. > > On the other: > > $ ipa host-find tftp-1801 > -------------- > 1 host matched > -------------- > Host name: tftp-1801.sub.domain.tld > ..... > > After rebooting the kdc with the error, no new > tracebacks in the > error_log > > > No new tracebacks but still not working? > > The CertificateFormatError is the server logging the equivalent > of what you're seeing in the client. > > rob > > > > that's right. > > Is there anything else I can look at? > > > -- > -- > Groeten, > natxo > > > > > -- > -- > Groeten, > natxo From edewata at redhat.com Tue Sep 13 02:35:12 2016 From: edewata at redhat.com (Endi Sukma Dewata) Date: Mon, 12 Sep 2016 21:35:12 -0500 Subject: [Freeipa-users] CA: Cannot add Centos7.2 replica to Centos6.8 ipa server In-Reply-To: References: <45b66871-aa01-3c91-0630-0bbeee488ae7@nelios.com> <323b9de1-adb3-47af-d4a7-d6083f88d75c@nelios.com> <9afdf0f3-97d1-cc6e-5f3d-0e11687a124a@redhat.com> <05b8e986-ceef-8e75-c76d-b38c45e2e3a1@nelios.com> <79903d5c-2b80-d492-6b29-59ac5b39c6f5@redhat.com> Message-ID: <98b16feb-8b70-13bb-4c92-cd9bca3e408d@redhat.com> On 9/9/2016 2:46 PM, Georgios Kafataridis wrote: > I've tried that but still the same result. > > [root at ipa-server /]# ldapsearch -D "cn=directory manager" -W -p 389 -h > localhost -b "uid=admin,ou=people,o=ipaca" > Enter LDAP Password: > # extended LDIF > # > # LDAPv3 > # base with scope subtree > # filter: (objectclass=*) > # requesting: ALL > # > > # search result > search: 2 > result: 32 No such object Hi, The master's logs indicate there's an authentication issue. Could you search the whole directory to find the admin user? $ ldapsearch ... -b "o=ipaca" "(uid=admin)" Try also other suffixes that you have in the DS. If you find it, try to authenticate against DS directly as the admin user. If the authentication fails, try resetting the password. -- Endi S. Dewata From rob.verduijn at gmail.com Tue Sep 13 06:51:48 2016 From: rob.verduijn at gmail.com (Rob Verduijn) Date: Tue, 13 Sep 2016 08:51:48 +0200 Subject: [Freeipa-users] sss / nsswitch Message-ID: Hi all, Yesterday my fedora 24 box received an update for sssd to 1.14.1-2.fc24. Then after the reboot the nfs-idmap service told me it couldn't start because it could not find method sss. So I filed a bug report and tried switching the method nsswitch. But now all files on my kerberos nfs4 shares belong to nobody:nobodyy again. Anybody who has a tip on how to work around this until they fix sssd ? Cheers Rob Verduijn -------------- next part -------------- An HTML attachment was scrubbed... URL: From sbose at redhat.com Tue Sep 13 07:03:54 2016 From: sbose at redhat.com (Sumit Bose) Date: Tue, 13 Sep 2016 09:03:54 +0200 Subject: [Freeipa-users] sss / nsswitch In-Reply-To: References: Message-ID: <20160913070354.GE32073@p.Speedport_W_724V_Typ_A_05011603_00_009> On Tue, Sep 13, 2016 at 08:51:48AM +0200, Rob Verduijn wrote: > Hi all, > > Yesterday my fedora 24 box received an update for sssd to 1.14.1-2.fc24. > > Then after the reboot the nfs-idmap service told me it couldn't start > because it could not find method sss. > > So I filed a bug report and tried switching the method nsswitch. > > But now all files on my kerberos nfs4 shares belong to nobody:nobodyy again. > > Anybody who has a tip on how to work around this until they fix sssd ? The module got its own sub-package. Please install sssd-nfs-idmap. HTH bye, Sumit > > Cheers > Rob Verduijn > -- > Manage your subscription for the Freeipa-users mailing list: > https://www.redhat.com/mailman/listinfo/freeipa-users > Go to http://freeipa.org for more info on the project From deepak_dimri at hotmail.com Tue Sep 13 07:49:26 2016 From: deepak_dimri at hotmail.com (Deepak Dimri) Date: Tue, 13 Sep 2016 03:49:26 -0400 Subject: [Freeipa-users] 2FA using FreeIPA Message-ID: Hi All, I have below lines added to my sshd_config file for testuser. Match User testuser AuthenticationMethods publickey,password:pam publickey,keyboard-interactive:pam I have OTP enable for tapuser in IPA and i am able to login to GUI using the password + OTP. However when i try to ssh i am getting prompted for first factor then second factor and then it ends with "Permission denied (keyboard-interactive)." error. What could be wrong here? Regards,Deepak -------------- next part -------------- An HTML attachment was scrubbed... URL: From jhrozek at redhat.com Tue Sep 13 08:21:39 2016 From: jhrozek at redhat.com (Jakub Hrozek) Date: Tue, 13 Sep 2016 10:21:39 +0200 Subject: [Freeipa-users] About AllowGroups with sshd In-Reply-To: <0afb01d20d0e$d99b9100$8cd2b300$@cyberfuel.com> References: <0afb01d20d0e$d99b9100$8cd2b300$@cyberfuel.com> Message-ID: <20160913082139.gishru3hcpzcpclt@hendrix> On Mon, Sep 12, 2016 at 10:00:57AM -0600, Jose Alvarez R. wrote: > Hello > > > > I have an question > > > > I have an FreeIPA 3.0 server(CentOS 6) with some clients servers(CentOS 6). > I wants enable root a two servers this servers, because they are backup > servers. > > > > I add theses lines in /etc/ssh/sshd_config of a client server. > > > > AllowUsers root at 192.168.20.2 > > AllowUsers root at 192.168.20.90 > > PermitRootLogin yes > > > > This working, but when try login with my user IPA, I can't login. > > > > I add the line "AllowGroups" with my group of users_IPA > > > > AllowGroups > > > > But not working, Can you help me ? > > > > Thanks, Regards > > > > Jose Alvarez. I know I'm not answering your question directly, but isn't it better to use HBAC with IPA and centralize the access control rather than edit config files on the clients? From natxo.asenjo at gmail.com Tue Sep 13 08:22:45 2016 From: natxo.asenjo at gmail.com (Natxo Asenjo) Date: Tue, 13 Sep 2016 10:22:45 +0200 Subject: [Freeipa-users] ipa: ERROR: Certificate format error: (SEC_ERROR_LEGACY_DATABASE) The certificate/key database is in an old, unsupported format. In-Reply-To: <57D70676.8080107@redhat.com> References: <57D015D8.7080507@redhat.com> <57D166B9.3070201@redhat.com> <57D70676.8080107@redhat.com> Message-ID: hi, On Mon, Sep 12, 2016 at 9:48 PM, Rob Crittenden wrote: > Natxo Asenjo wrote: > >> hi, >> >> I can reproduce this everytime. Restarting httpd fixes it for a while, >> but then ik stops working: >> >> $ ipa cert-show 1 >> ipa: ERROR: cannot connect to >> 'https://kdc01.unix.domain.tld:443/ca/agent/ca/displayBySerial': >> (SEC_ERROR_LEGACY_DATABASE) The certificate/key database is in an old, >> unsupported format. >> > > It is very strange that it goes from a working to a non-working state. > > I have only two suggestions: > > 1. Create /etc/ipa/server.conf with a [global] section and debug=True in > it, restart httpd. Your log will be quite a bit more verbose but given it > reproduces so quickly hopefully won't be too big a deal. That might show > something. > > 2. Try brute force with strace. Finding the right httpd process to strace > can be frustrating but usually there are only 8 and they rotate so > eventually you should get the right one. > Could I send you the log files privately? -- Groeten, natxo -------------- next part -------------- An HTML attachment was scrubbed... URL: From sbose at redhat.com Tue Sep 13 08:39:17 2016 From: sbose at redhat.com (Sumit Bose) Date: Tue, 13 Sep 2016 10:39:17 +0200 Subject: [Freeipa-users] sss / nsswitch In-Reply-To: References: <20160913070354.GE32073@p.Speedport_W_724V_Typ_A_05011603_00_009> Message-ID: <20160913083917.GF32073@p.Speedport_W_724V_Typ_A_05011603_00_009> On Tue, Sep 13, 2016 at 10:13:12AM +0200, Rob Verduijn wrote: > Hi, > > Thanks that did it. > > Is there a less painfull way to be notified of these changes ? > > My nfs configuration gets broken much more than I like because of changes > like these. > I know fedora is supposed to be testing grounds unstable software, but I > would really like to hear a heads up more often. The change was mentioned in the upstream release notes of SSSD-1.14.1 https://fedorahosted.org/sssd/wiki/Releases/Notes-1.14.1 but of course I cannot be expected to read all upstream release note before running 'dnf update'. The change was necessary because before the plugin was in the sssd-common package and this caused that some nfs dependencies were pulled in even on systems where nfs is not needed at all. Since neither SSSD nor nfs-idmap strictly require the plugin the new package is not automatically installed during update. bye, Sumit > > Rob Verduijn > > 2016-09-13 9:03 GMT+02:00 Sumit Bose : > > > On Tue, Sep 13, 2016 at 08:51:48AM +0200, Rob Verduijn wrote: > > > Hi all, > > > > > > Yesterday my fedora 24 box received an update for sssd to 1.14.1-2.fc24. > > > > > > Then after the reboot the nfs-idmap service told me it couldn't start > > > because it could not find method sss. > > > > > > So I filed a bug report and tried switching the method nsswitch. > > > > > > But now all files on my kerberos nfs4 shares belong to nobody:nobodyy > > again. > > > > > > Anybody who has a tip on how to work around this until they fix sssd ? > > > > The module got its own sub-package. Please install sssd-nfs-idmap. > > > > HTH > > > > bye, > > Sumit > > > > > > > > Cheers > > > Rob Verduijn > > > > > -- > > > Manage your subscription for the Freeipa-users mailing list: > > > https://www.redhat.com/mailman/listinfo/freeipa-users > > > Go to http://freeipa.org for more info on the project > > > > -- > > Manage your subscription for the Freeipa-users mailing list: > > https://www.redhat.com/mailman/listinfo/freeipa-users > > Go to http://freeipa.org for more info on the project > > From peljasz at yahoo.co.uk Tue Sep 13 09:43:34 2016 From: peljasz at yahoo.co.uk (lejeczek) Date: Tue, 13 Sep 2016 10:43:34 +0100 Subject: [Freeipa-users] ERROR CA configuration failed. - again In-Reply-To: <57D2F611.6060700@redhat.com> References: <57D2F611.6060700@redhat.com> Message-ID: <6bf80bff-5331-5cb2-45eb-c767bade7eb8@yahoo.co.uk> fortunately that was a VM and with libvirt+qemu snaphost feature I reverted filesystem to some older(prior to IPA) state, and yes... that was that only system'slocal problem. On 09/09/16 18:49, Rob Crittenden wrote: > lejeczek wrote: >> hi everybody, >> >> looking at ipareplica-install.log: >> >> raise RuntimeError("%s configuration failed." % >> self.subsystem) >> RuntimeError: CA configuration failed. >> >> 2016-09-09T16:23:17Z DEBUG [error] RuntimeError: CA >> configuration failed. >> 2016-09-09T16:23:17Z DEBUG File >> "/usr/lib/python2.7/site-packages/ipapython/admintool.py", >> line 171, in >> execute >> >> then at /var/log/pki/pki-tomcat/ca/system > > I'd suggest looking at the debug log for more details. > >> 0.localhost-startStop-1 - [09/Sep/2016:16:04:22 BST] [3] >> [3] Cannot >> build CA chain. Error >> java.security.cert.CertificateException: >> Certificate is not a PKCS #11 certificate >> 0.localhost-startStop-1 - [09/Sep/2016:16:04:22 BST] [13] >> [3] authz >> instance DirAclAuthz initialization failed and skipped, >> error=Property >> internaldb.ldapconn.port missing value >> >> I cannot find anything more telling in the logs. Does it >> have anything >> to do with what's in: >> /etc/httpd/alias/ >> ? > > No. > >> I yum removed >> `rpm -qa ipa* 389*` pki-base krb5-pkinit krb5-server >> krb5-workstation >> pki-tomcat certmonger >> rm dirs + reinstalled, yet I cannot find the the root >> cause of this mess. > > I seriously doubt the problem is local to the box. > > rob > From gjn at gjn.priv.at Tue Sep 13 11:57:36 2016 From: gjn at gjn.priv.at (=?ISO-8859-1?Q?G=FCnther_J=2E?= Niederwimmer) Date: Tue, 13 Sep 2016 13:57:36 +0200 Subject: [Freeipa-users] 3rd party Certificate install Message-ID: <1557096.oY3xIzA97I@techz> Hello, FreeIPA 4.3.1 I like to install my new Startcom Cert and have a Problem with the access ? I search and found this ipa-cacert-manage -p '#-!???<<<<<<' -n STARTCOM-ROOT -t C,, install 1_root_bundle.crt but I become this Insufficient access: Invalid credentials The ipa-cacert-manage command failed. Can I test the "DM" Password with a other command or is this a Problem with ipa-cacert-manage? I test it with "kinit admin" and without ? or is this a Problem with the Password when I write this ipa-cacert-manage -p #-!???<<<<<< -n STARTCOM-ROOT -t C,, install 1_root_bundle.crt I have this answer ipa-cacert-manage: error: -p option requires an argument Thanks for a answer, -- mit freundlichen Gr??en / best regards, G?nther J. Niederwimmer From natxo.asenjo at gmail.com Tue Sep 13 12:10:45 2016 From: natxo.asenjo at gmail.com (Natxo Asenjo) Date: Tue, 13 Sep 2016 14:10:45 +0200 Subject: [Freeipa-users] adding replica centos 7 to centos 6 fails [error] ObjectclassViolation: attribute "unhashed#user#password" not allowed Message-ID: hi, when trying to add a replica to the Idm environment of a host running centos 7 (fully patched) to an existing centos 6.8 realm I get this error: [24/38]: setting up initial replication [error] ObjectclassViolation: attribute "unhashed#user#password" not allowed Your system may be partly configured. Run /usr/sbin/ipa-server-install --uninstall to clean up. ipa.ipapython.install.cli.install_tool(Replica): ERROR attribute "unhashed#user#password" not allowed In /var/log/ipareplica-install.log I find this same info with a detailed traceback of the python script, but it boils down to the same, the attribute is not allowed. Any ideas? -- -- Groeten, natxo -------------- next part -------------- An HTML attachment was scrubbed... URL: From natxo.asenjo at gmail.com Tue Sep 13 12:39:52 2016 From: natxo.asenjo at gmail.com (Natxo Asenjo) Date: Tue, 13 Sep 2016 14:39:52 +0200 Subject: [Freeipa-users] adding replica centos 7 to centos 6 fails [error] ObjectclassViolation: attribute "unhashed#user#password" not allowed In-Reply-To: References: Message-ID: On Tue, Sep 13, 2016 at 2:10 PM, Natxo Asenjo wrote: > hi, > > when trying to add a replica to the Idm environment of a host running > centos 7 (fully patched) to an existing centos 6.8 realm I get this error: > ok, some progress. I found this: https://fedorahosted.org/389/ticket/470 So I went ahead and rebooted the master 6.8 kdc I was replicating from and then it failed in the certificate server instance: ipa.ipaserver.install.cainstance.CAInstance: CRITICAL Failed to configure CA instance: Command ''/usr/sbin/pkispawn' '-s' 'CA' '-f' '/tmp/tmpyHV1BW'' returned non-zero exit status 1 ipa.ipaserver.install.cainstance.CAInstance: CRITICAL See the installation logs and the following files/directories for more information: ipa.ipaserver.install.cainstance.CAInstance: CRITICAL /var/log/pki-ca-install.log ipa.ipaserver.install.cainstance.CAInstance: CRITICAL /var/log/pki/pki-tomcat [error] RuntimeError: CA configuration failed. Your system may be partly configured. Run /usr/sbin/ipa-server-install --uninstall to clean up. ipa.ipapython.install.cli.install_tool(Replica): ERROR CA configuration failed. But there is no /var/log/pki-ca-install.log : # ls -ltr /var/log/ total 1708 drwx------. 2 root root 6 Jun 10 2014 ppp drwxr-xr-x. 2 ntp ntp 6 May 31 12:29 ntpstats drwx------. 2 root root 6 Jul 18 17:30 httpd drwxr-x---. 2 sssd sssd 6 Aug 2 18:58 sssd -rw-------. 1 root root 0 Sep 13 13:19 tallylog drwx------. 3 root root 16 Sep 13 13:19 samba -rw-------. 1 root root 0 Sep 13 13:20 spooler drwxr-xr-x. 2 root root 4096 Sep 13 13:23 anaconda drwxr-x---. 2 root root 22 Sep 13 13:23 audit drwxr-xr-x. 2 root root 22 Sep 13 13:23 tuned drwxrwx---. 2 tomcat root 25 Sep 13 13:31 tomcat -rw-------. 1 root root 15126 Sep 13 13:31 yum.log -rw-------. 1 root root 8786 Sep 13 13:31 ipaupgrade.log -rw-r--r--. 1 root root 94862 Sep 13 13:59 dmesg.old -rw-------. 1 root root 18112 Sep 13 14:29 ipaclient-install.log -rw-------. 1 root root 40193 Sep 13 14:29 ipaclient-uninstall.log -rw-------. 1 root root 35796 Sep 13 14:29 ipaserver-uninstall.log -rw-r--r--. 1 root root 94862 Sep 13 14:30 dmesg -rw-r--r--. 1 root root 8591 Sep 13 14:30 boot.log -rw-------. 1 root root 2587 Sep 13 14:30 cron -rw-r--r--. 1 root root 200 Sep 13 14:30 wpa_supplicant.log -rw-------. 1 root root 958 Sep 13 14:30 maillog -rw-------. 1 root utmp 768 Sep 13 14:30 btmp -rw-rw-r--. 1 root utmp 13056 Sep 13 14:30 wtmp -rw-r--r--. 1 root root 291416 Sep 13 14:30 lastlog -rw-------. 1 root root 7318 Sep 13 14:31 ipareplica-conncheck.log drwxr-xr-x. 3 root root 35 Sep 13 14:31 dirsrv drwxr-xr-x. 4 root root 4096 Sep 13 14:32 pki -rw-------. 1 root root 87106 Sep 13 14:32 secure -rw-------. 1 root root 742436 Sep 13 14:32 messages -rw-------. 1 root root 202169 Sep 13 14:33 ipareplica-install.log In the ipa-replica-install.log , though, I find this: pkispawn : ERROR ....... Exception from Java Configuration Servlet: 500 Server Error: Internal Server Error pkispawn : ERROR ....... ParseError: not well-formed (invalid token): line 1, column 0: {"Attributes":{"Attribute":[]},"ClassName":"com.netscape.certsrv.base.PKIException","Code":500,"Message":"Clone does not have all the required certificates"} Any clue? -- -- Groeten, natxo -------------- next part -------------- An HTML attachment was scrubbed... URL: From lslebodn at redhat.com Tue Sep 13 13:07:16 2016 From: lslebodn at redhat.com (Lukas Slebodnik) Date: Tue, 13 Sep 2016 15:07:16 +0200 Subject: [Freeipa-users] sss / nsswitch In-Reply-To: <20160913083917.GF32073@p.Speedport_W_724V_Typ_A_05011603_00_009> References: <20160913070354.GE32073@p.Speedport_W_724V_Typ_A_05011603_00_009> <20160913083917.GF32073@p.Speedport_W_724V_Typ_A_05011603_00_009> Message-ID: <20160913130716.GL6703@10.4.128.1> On (13/09/16 10:39), Sumit Bose wrote: >On Tue, Sep 13, 2016 at 10:13:12AM +0200, Rob Verduijn wrote: >> Hi, >> >> Thanks that did it. >> >> Is there a less painfull way to be notified of these changes ? >> >> My nfs configuration gets broken much more than I like because of changes >> like these. >> I know fedora is supposed to be testing grounds unstable software, but I >> would really like to hear a heads up more often. > >The change was mentioned in the upstream release notes of SSSD-1.14.1 >https://fedorahosted.org/sssd/wiki/Releases/Notes-1.14.1 but of course I >cannot be expected to read all upstream release note before running 'dnf >update'. > >The change was necessary because before the plugin was in the >sssd-common package and this caused that some nfs dependencies were >pulled in even on systems where nfs is not needed at all. Since neither >SSSD nor nfs-idmap strictly require the plugin the new package is not >automatically installed during update. > Sorry for troubles. We can add weak dependency info sssd-common on sssd-nfs-idmap and it might be installed by default. IIRC dnf does not inform about suggested packages; but recommends minght work. Feel free ot file a BZ. The reason why it is in separate package is "container world". You need to have install packge sssd-nfs-idmap on host but sssd can be running in container. LS From natxo.asenjo at gmail.com Tue Sep 13 13:43:41 2016 From: natxo.asenjo at gmail.com (Natxo Asenjo) Date: Tue, 13 Sep 2016 15:43:41 +0200 Subject: [Freeipa-users] adding replica centos 7 to centos 6 fails [error] ObjectclassViolation: attribute "unhashed#user#password" not allowed In-Reply-To: References: Message-ID: ok, again some progress. I found this thread: https://www.mail-archive.com/freeipa-users at redhat.com/msg21107.html And the symptons match: this Idm environment was installed in november 2012, and has been upgraded a few times (time flies). All the certificates are valid: [root at kdc01 ~]$ getcert list | grep expire expires: 2016-10-12 10:49:24 UTC expires: 2016-10-12 10:49:25 UTC expires: 2016-10-12 10:49:24 UTC expires: 2018-09-03 12:24:14 UTC expires: 2018-09-03 12:23:14 UTC expires: 2018-09-03 12:23:14 UTC expires: 2018-09-03 12:23:14 UTC expires: 2018-09-03 12:23:14 UTC root at kdc01 ~]$ pki-server ca-group-member-find "Subsystem Group" User ID: CA-kdc01.unix.iriszorg.nl-9443 Common Name: CA-kdc01.unix.iriszorg.nl-9443 Surname: CA-kdc01.unix.iriszorg.nl-9443 Type: agentType Description: 2;240;CN=Certificate Authority,O=UNIX.IRISZORG.NL;CN=CA Subsystem,O=UNIX.IRISZORG.NL E-mail: User ID: CA-kdc02.unix.iriszorg.nl-9443 Common Name: CA-kdc02.unix.iriszorg.nl-9443 Surname: CA-kdc02.unix.iriszorg.nl-9443 Type: agentType Description: 2;4;CN=Certificate Authority,O=UNIX.IRISZORG.NL;CN=CA Subsystem,O=UNIX.IRISZORG.NL so according to the posts in the mailing list thread, the description field has the serial number of the certificates, in this case, 240 and 4 and that appears to match with this info: [root at kdc01 ~]$ certutil -d /var/lib/pki-ca/alias/ -L -n 'subsystemCert cert-pki-ca' | grep -i serial Serial Number: 240 (0xf0) Serial Number: 4 (0x4) although the one with serial 4 is expired So, how to procede next? Do I need to modify the usercertificate attribute of uid=CA-kdc01.unix.iriszorg.nl-9443,ou=people,o=ipaca? step 1: dump cert info to temp file: certutil -L -d /var/lib/pki-ca/alias/ -n "subsystemCert cert-pki-ca" -a > /tmp/subsystemcert.pem In my case, because of the expired certificate, I get 2 certificates. step 2: strip begin/end headers of temp certificate: echo && cat /tmp/subsystemcert.pem | sed -rn '/^-----BEGIN CERTIFICATE-----$/{:1;n;/^-----END CERTIFICATE-----$/b2;H;b1};:2;${x;s/\s//g;p}' step 3: modify usercertificate attribute of uid=CA-kdc01.unix.iriszorg.nl-9443,ou=people,o=ipaca with string generated in step 2. step 4: reload ipa/reboot server. is my procedure correct? -- regards, Natxo On Tue, Sep 13, 2016 at 2:39 PM, Natxo Asenjo wrote: > > > On Tue, Sep 13, 2016 at 2:10 PM, Natxo Asenjo > wrote: > >> hi, >> >> when trying to add a replica to the Idm environment of a host running >> centos 7 (fully patched) to an existing centos 6.8 realm I get this error: >> > > ok, some progress. I found this: > > https://fedorahosted.org/389/ticket/470 > > So I went ahead and rebooted the master 6.8 kdc I was replicating from and > then it failed in the certificate server instance: > > > ipa.ipaserver.install.cainstance.CAInstance: CRITICAL Failed to configure > CA instance: Command ''/usr/sbin/pkispawn' '-s' 'CA' '-f' '/tmp/tmpyHV1BW'' > returned non-zero exit status 1 > ipa.ipaserver.install.cainstance.CAInstance: CRITICAL See the > installation logs and the following files/directories for more information: > ipa.ipaserver.install.cainstance.CAInstance: CRITICAL > /var/log/pki-ca-install.log > ipa.ipaserver.install.cainstance.CAInstance: CRITICAL > /var/log/pki/pki-tomcat > [error] RuntimeError: CA configuration failed. > Your system may be partly configured. > Run /usr/sbin/ipa-server-install --uninstall to clean up. > > ipa.ipapython.install.cli.install_tool(Replica): ERROR CA > configuration failed. > > > But there is no /var/log/pki-ca-install.log : > > # ls -ltr /var/log/ > total 1708 > drwx------. 2 root root 6 Jun 10 2014 ppp > drwxr-xr-x. 2 ntp ntp 6 May 31 12:29 ntpstats > drwx------. 2 root root 6 Jul 18 17:30 httpd > drwxr-x---. 2 sssd sssd 6 Aug 2 18:58 sssd > -rw-------. 1 root root 0 Sep 13 13:19 tallylog > drwx------. 3 root root 16 Sep 13 13:19 samba > -rw-------. 1 root root 0 Sep 13 13:20 spooler > drwxr-xr-x. 2 root root 4096 Sep 13 13:23 anaconda > drwxr-x---. 2 root root 22 Sep 13 13:23 audit > drwxr-xr-x. 2 root root 22 Sep 13 13:23 tuned > drwxrwx---. 2 tomcat root 25 Sep 13 13:31 tomcat > -rw-------. 1 root root 15126 Sep 13 13:31 yum.log > -rw-------. 1 root root 8786 Sep 13 13:31 ipaupgrade.log > -rw-r--r--. 1 root root 94862 Sep 13 13:59 dmesg.old > -rw-------. 1 root root 18112 Sep 13 14:29 ipaclient-install.log > -rw-------. 1 root root 40193 Sep 13 14:29 ipaclient-uninstall.log > -rw-------. 1 root root 35796 Sep 13 14:29 ipaserver-uninstall.log > -rw-r--r--. 1 root root 94862 Sep 13 14:30 dmesg > -rw-r--r--. 1 root root 8591 Sep 13 14:30 boot.log > -rw-------. 1 root root 2587 Sep 13 14:30 cron > -rw-r--r--. 1 root root 200 Sep 13 14:30 wpa_supplicant.log > -rw-------. 1 root root 958 Sep 13 14:30 maillog > -rw-------. 1 root utmp 768 Sep 13 14:30 btmp > -rw-rw-r--. 1 root utmp 13056 Sep 13 14:30 wtmp > -rw-r--r--. 1 root root 291416 Sep 13 14:30 lastlog > -rw-------. 1 root root 7318 Sep 13 14:31 ipareplica-conncheck.log > drwxr-xr-x. 3 root root 35 Sep 13 14:31 dirsrv > drwxr-xr-x. 4 root root 4096 Sep 13 14:32 pki > -rw-------. 1 root root 87106 Sep 13 14:32 secure > -rw-------. 1 root root 742436 Sep 13 14:32 messages > -rw-------. 1 root root 202169 Sep 13 14:33 ipareplica-install.log > > In the ipa-replica-install.log , though, I find this: > > pkispawn : ERROR ....... Exception from Java Configuration Servlet: > 500 Server Error: Internal Server Error > pkispawn : ERROR ....... ParseError: not well-formed (invalid > token): line 1, column 0: {"Attributes":{"Attribute":[]} > ,"ClassName":"com.netscape.certsrv.base.PKIException","Code":500,"Message":"Clone > does not have all the required certificates"} > > Any clue? > > > > -- > -- > Groeten, > natxo > -- -- Groeten, natxo -------------- next part -------------- An HTML attachment was scrubbed... URL: From flo at redhat.com Tue Sep 13 14:03:06 2016 From: flo at redhat.com (Florence Blanc-Renaud) Date: Tue, 13 Sep 2016 16:03:06 +0200 Subject: [Freeipa-users] 3rd party Certificate install In-Reply-To: <1557096.oY3xIzA97I@techz> References: <1557096.oY3xIzA97I@techz> Message-ID: <4fbd6208-e6b5-e53f-760b-2f8152f2f3e4@redhat.com> Hi, ipa-cacert-manage must be run as root but does not require any Kerberos ticket. You can run the following command to check your directory manager password: /usr/bin/ldapsearch -h localhost -p 389 -D "cn=directory manager" -w '#-!???<<<<<<' -b "" -s base If the password is wrong, you will get an output like this one: ldap_bind: Invalid credentials (49) Otherwise it means that your DM password is OK. HTH, Flo. On 09/13/2016 01:57 PM, G?nther J. Niederwimmer wrote: > Hello, > > FreeIPA 4.3.1 > > I like to install my new Startcom Cert and have a Problem with the access ? > > I search and found this > > ipa-cacert-manage -p '#-!???<<<<<<' -n STARTCOM-ROOT -t C,, install > 1_root_bundle.crt > > but I become this > Insufficient access: Invalid credentials > The ipa-cacert-manage command failed. > > Can I test the "DM" Password with a other command or is this a Problem with > ipa-cacert-manage? > > I test it with "kinit admin" and without ? > > or is this a Problem with the Password when I write this > ipa-cacert-manage -p #-!???<<<<<< -n STARTCOM-ROOT -t C,, install > 1_root_bundle.crt > > I have this answer > > ipa-cacert-manage: error: -p option requires an argument > > Thanks for a answer, > From rob.verduijn at gmail.com Tue Sep 13 14:18:52 2016 From: rob.verduijn at gmail.com (Rob Verduijn) Date: Tue, 13 Sep 2016 16:18:52 +0200 Subject: [Freeipa-users] sss / nsswitch In-Reply-To: <20160913130716.GL6703@10.4.128.1> References: <20160913070354.GE32073@p.Speedport_W_724V_Typ_A_05011603_00_009> <20160913083917.GF32073@p.Speedport_W_724V_Typ_A_05011603_00_009> <20160913130716.GL6703@10.4.128.1> Message-ID: 2016-09-13 15:07 GMT+02:00 Lukas Slebodnik : > On (13/09/16 10:39), Sumit Bose wrote: > >On Tue, Sep 13, 2016 at 10:13:12AM +0200, Rob Verduijn wrote: > >> Hi, > >> > >> Thanks that did it. > >> > >> Is there a less painfull way to be notified of these changes ? > >> > >> My nfs configuration gets broken much more than I like because of > changes > >> like these. > >> I know fedora is supposed to be testing grounds unstable software, but I > >> would really like to hear a heads up more often. > > > >The change was mentioned in the upstream release notes of SSSD-1.14.1 > >https://fedorahosted.org/sssd/wiki/Releases/Notes-1.14.1 but of course I > >cannot be expected to read all upstream release note before running 'dnf > >update'. > > > >The change was necessary because before the plugin was in the > >sssd-common package and this caused that some nfs dependencies were > >pulled in even on systems where nfs is not needed at all. Since neither > >SSSD nor nfs-idmap strictly require the plugin the new package is not > >automatically installed during update. > > > > Sorry for troubles. We can add weak dependency info sssd-common on > sssd-nfs-idmap and it might be installed by default. > IIRC dnf does not inform about suggested packages; but recommends minght > work. Feel free ot file a BZ. > > The reason why it is in separate package is "container world". > You need to have install packge sssd-nfs-idmap on host > but sssd can be running in container. > > LS > I probably should've noticed that the version number went from 1.13.x to 1.14.x which usually is something noteworthy. I'll just add the release notes from sssd to my list of must reads when there is an update. Rob Verduijn -------------- next part -------------- An HTML attachment was scrubbed... URL: From rcritten at redhat.com Tue Sep 13 15:26:25 2016 From: rcritten at redhat.com (Rob Crittenden) Date: Tue, 13 Sep 2016 11:26:25 -0400 Subject: [Freeipa-users] ipa: ERROR: Certificate format error: (SEC_ERROR_LEGACY_DATABASE) The certificate/key database is in an old, unsupported format. In-Reply-To: References: <57D015D8.7080507@redhat.com> <57D166B9.3070201@redhat.com> <57D70676.8080107@redhat.com> Message-ID: <57D81AA1.4050400@redhat.com> Natxo Asenjo wrote: > hi, > > > On Mon, Sep 12, 2016 at 9:48 PM, Rob Crittenden > wrote: > > Natxo Asenjo wrote: > > hi, > > I can reproduce this everytime. Restarting httpd fixes it for a > while, > but then ik stops working: > > $ ipa cert-show 1 > ipa: ERROR: cannot connect to > 'https://kdc01.unix.domain.tld:443/ca/agent/ca/displayBySerial > ': > (SEC_ERROR_LEGACY_DATABASE) The certificate/key database is in > an old, > unsupported format. > > > It is very strange that it goes from a working to a non-working state. > > I have only two suggestions: > > 1. Create /etc/ipa/server.conf with a [global] section and > debug=True in it, restart httpd. Your log will be quite a bit more > verbose but given it reproduces so quickly hopefully won't be too > big a deal. That might show something. > > 2. Try brute force with strace. Finding the right httpd process to > strace can be frustrating but usually there are only 8 and they > rotate so eventually you should get the right one. > > > Could I send you the log files privately? Sure. rob From edewata at redhat.com Tue Sep 13 19:36:13 2016 From: edewata at redhat.com (Endi Sukma Dewata) Date: Tue, 13 Sep 2016 14:36:13 -0500 Subject: [Freeipa-users] CA: Cannot add Centos7.2 replica to Centos6.8 ipa server In-Reply-To: <98b16feb-8b70-13bb-4c92-cd9bca3e408d@redhat.com> References: <45b66871-aa01-3c91-0630-0bbeee488ae7@nelios.com> <323b9de1-adb3-47af-d4a7-d6083f88d75c@nelios.com> <9afdf0f3-97d1-cc6e-5f3d-0e11687a124a@redhat.com> <05b8e986-ceef-8e75-c76d-b38c45e2e3a1@nelios.com> <79903d5c-2b80-d492-6b29-59ac5b39c6f5@redhat.com> <98b16feb-8b70-13bb-4c92-cd9bca3e408d@redhat.com> Message-ID: <0b546b4a-9ef3-7a15-27a7-3a2d59e5281f@redhat.com> On 9/12/2016 9:35 PM, Endi Sukma Dewata wrote: > On 9/9/2016 2:46 PM, Georgios Kafataridis wrote: >> I've tried that but still the same result. >> >> [root at ipa-server /]# ldapsearch -D "cn=directory manager" -W -p 389 -h >> localhost -b "uid=admin,ou=people,o=ipaca" >> Enter LDAP Password: >> # extended LDIF >> # >> # LDAPv3 >> # base with scope subtree >> # filter: (objectclass=*) >> # requesting: ALL >> # >> >> # search result >> search: 2 >> result: 32 No such object > > Hi, > > The master's logs indicate there's an authentication issue. > > Could you search the whole directory to find the admin user? > $ ldapsearch ... -b "o=ipaca" "(uid=admin)" > > Try also other suffixes that you have in the DS. > > If you find it, try to authenticate against DS directly as the admin > user. If the authentication fails, try resetting the password. I believe there is actually another DS instance on CentOS 6.8 running on port 7389, so make sure you check that too. If the admin user is indeed missing, it will need to be recreated, assigned a password and certificate, and added to the appropriate groups. See also: http://pki.fedoraproject.org/wiki/IPA_PKI_Users -- Endi S. Dewata From natxo.asenjo at gmail.com Wed Sep 14 06:20:48 2016 From: natxo.asenjo at gmail.com (Natxo Asenjo) Date: Wed, 14 Sep 2016 08:20:48 +0200 Subject: [Freeipa-users] CA: Cannot add Centos7.2 replica to Centos6.8 ipa server In-Reply-To: <0b546b4a-9ef3-7a15-27a7-3a2d59e5281f@redhat.com> References: <45b66871-aa01-3c91-0630-0bbeee488ae7@nelios.com> <323b9de1-adb3-47af-d4a7-d6083f88d75c@nelios.com> <9afdf0f3-97d1-cc6e-5f3d-0e11687a124a@redhat.com> <05b8e986-ceef-8e75-c76d-b38c45e2e3a1@nelios.com> <79903d5c-2b80-d492-6b29-59ac5b39c6f5@redhat.com> <98b16feb-8b70-13bb-4c92-cd9bca3e408d@redhat.com> <0b546b4a-9ef3-7a15-27a7-3a2d59e5281f@redhat.com> Message-ID: hi, On Tue, Sep 13, 2016 at 9:36 PM, Endi Sukma Dewata wrote: > On 9/12/2016 9:35 PM, Endi Sukma Dewata wrote: > >> On 9/9/2016 2:46 PM, Georgios Kafataridis wrote: >> >>> I've tried that but still the same result. >>> >>> [root at ipa-server /]# ldapsearch -D "cn=directory manager" -W -p 389 -h >>> localhost -b "uid=admin,ou=people,o=ipaca" >>> Enter LDAP Password: >>> # extended LDIF >>> # >>> # LDAPv3 >>> # base with scope subtree >>> # filter: (objectclass=*) >>> # requesting: ALL >>> # >>> >>> # search result >>> search: 2 >>> result: 32 No such object >>> >> >> Hi, >> >> The master's logs indicate there's an authentication issue. >> >> Could you search the whole directory to find the admin user? >> $ ldapsearch ... -b "o=ipaca" "(uid=admin)" >> >> Try also other suffixes that you have in the DS. >> >> If you find it, try to authenticate against DS directly as the admin >> user. If the authentication fails, try resetting the password. >> > > I believe there is actually another DS instance on CentOS 6.8 running on > port 7389, so make sure you check that too. If the admin user is indeed > missing, it will need to be recreated, assigned a password and certificate, > and added to the appropriate groups. > > See also: http://pki.fedoraproject.org/wiki/IPA_PKI_Users > I am having this problem too (see thread with subject " adding replica centos 7 to centos 6 fails [error] ObjectclassViolation: attribute "unhashed#user#password" not allowed" If the usercertificate attribute of this user is expired, could this be the cause of these problems? I can login with the password as user uid=admin,ou=people,o=ipaca, but the certificate expired 2014-10-28 -- regards, Natxo -- -- Groeten, natxo -------------- next part -------------- An HTML attachment was scrubbed... URL: From lkrispen at redhat.com Wed Sep 14 07:01:22 2016 From: lkrispen at redhat.com (Ludwig Krispenz) Date: Wed, 14 Sep 2016 09:01:22 +0200 Subject: [Freeipa-users] Freeipa 4.2.0 hangs intermittently In-Reply-To: References: <57CD659B.6050707@redhat.com> <57CD68BD.9030802@redhat.com> <57CE8CDB.6000809@redhat.com> <57CE9DE5.6020301@redhat.com> <57D0758E.6070406@redhat.com> Message-ID: <57D8F5C2.6090605@redhat.com> Hi, On 09/13/2016 07:37 PM, Rakesh Rajasekharan wrote: > Hi All, > > Have finally made some progress with this.. after changing the > checkpoint interval to 180, my hangs have gone down now.. > > However, I faced a similar hang yesterday... users were not able to > login.. , though this time the ns-slapd did not had any issues and > ldapsearch worked fine possibly due to the changes in checpoint. So, I > think I hit some other issue this time this is a bit confusing, if your server crashes with the attached stacktrace ldapsearch cannot work. About the core, it looks like you are hitting this issue: https://fedorahosted.org/389/ticket/48388 > > I had a core genrated and this is the stacktrace of it.. can you > please go through this and help me identify what could be causing the > issue this time.. I have put in lot of efforts to debug and really > would love to have this working in my prod env.. as it does in my > other envs... > > GNU gdb (GDB) Red Hat Enterprise Linux 7.6.1-80.el7 > Copyright (C) 2013 Free Software Foundation, Inc. > License GPLv3+: GNU GPL version 3 or later > > This is free software: you are free to change and redistribute it. > There is NO WARRANTY, to the extent permitted by law. Type "show copying" > and "show warranty" for details. > This GDB was configured as "x86_64-redhat-linux-gnu". > For bug reporting instructions, please see: > ... > Reading symbols from /usr/sbin/ns-slapd... > warning: the debug information found in > "/usr/lib/debug//usr/sbin/ns-slapd.debug" does not match > "/usr/sbin/ns-slapd" (CRC mismatch). > > > warning: the debug information found in > "/usr/lib/debug/usr/sbin/ns-slapd.debug" does not match > "/usr/sbin/ns-slapd" (CRC mismatch). > > Reading symbols from /usr/sbin/ns-slapd...(no debugging symbols > found)...done. > (no debugging symbols found)...done. > [New LWP 15255] > [New LWP 15286] > [New LWP 15245] > [New LWP 15246] > [New LWP 15247] > [New LWP 15248] > [New LWP 15243] > > warning: the debug information found in > "/usr/lib/debug//usr/lib64/dirsrv/libslapd.so.0.0.0.debug" does not > match "/usr/lib64/dirsrv/libslapd.so.0" (CRC mismatch). > > > warning: the debug information found in > "/usr/lib/debug/usr/lib64/dirsrv/libslapd.so.0.0.0.debug" does not > match "/usr/lib64/dirsrv/libslapd.so.0" (CRC mismatch). > > [Thread debugging using libthread_db enabled] > Using host libthread_db library "/lib64/libthread_db.so.1". > > warning: the debug information found in > "/usr/lib/debug//usr/lib64/dirsrv/plugins/libsyntax-plugin.so.debug" > does not match "/usr/lib64/dirsrv/plugins/libsyntax-plugin.so" (CRC > mismatch). > > > warning: the debug information found in > "/usr/lib/debug/usr/lib64/dirsrv/plugins/libsyntax-plugin.so.debug" > does not match "/usr/lib64/dirsrv/plugins/libsyntax-plugin.so" (CRC > mismatch). > > > warning: the debug information found in > "/usr/lib/debug//usr/lib64/dirsrv/plugins/libbitwise-plugin.so.debug" > does not match "/usr/lib64/dirsrv/plugins/libbitwise-plugin.so" (CRC > mismatch). > > > warning: the debug information found in > "/usr/lib/debug/usr/lib64/dirsrv/plugins/libbitwise-plugin.so.debug" > does not match "/usr/lib64/dirsrv/plugins/libbitwise-plugin.so" (CRC > mismatch). > > ...skipping... > -rw-------. 1 dirsrv dirsrv 0 Sep 8 02:55 audit > -rw-------. 1 dirsrv dirsrv 2551824384 Sep 12 17:32 core.10450 > -rw-------. 1 dirsrv dirsrv 1464463360 Sep 12 19:35 core.14709 > -rw-------. 1 dirsrv dirsrv 4483862528 Sep 13 01:05 core.15243 > -rw-------. 1 dirsrv dirsrv 66288165 Sep 13 02:10 errors > -rw-------. 1 dirsrv dirsrv 104964391 Sep 13 08:30 access.20160913-074214 > -rw-------. 1 dirsrv dirsrv 105021859 Sep 13 09:26 access.20160913-083046 > -rw-------. 1 dirsrv dirsrv 104861746 Sep 13 10:31 access.20160913-092646 > -rw-------. 1 dirsrv dirsrv 105069140 Sep 13 11:36 access.20160913-103137 > -rw-------. 1 dirsrv dirsrv 104913480 Sep 13 12:41 access.20160913-113638 > -rw-------. 1 dirsrv dirsrv 105186788 Sep 13 13:46 access.20160913-124118 > -rw-------. 1 dirsrv dirsrv 105162159 Sep 13 14:51 access.20160913-134619 > -rw-------. 1 dirsrv dirsrv 105256624 Sep 13 15:56 access.20160913-145120 > -rw-------. 1 dirsrv dirsrv 105231158 Sep 13 17:01 access.20160913-155620 > -rw-------. 1 dirsrv dirsrv 1044 Sep 13 17:01 access.rotationinfo > -rw-r--r--. 1 root root 19287 Sep 13 17:28 > stacktrace.1473787719.txt > -rw-------. 1 dirsrv dirsrv 45608914 Sep 13 17:29 access > [root at prod-ipa-master-int slapd-SPRINKLR-COM]# gdb -ex 'set confirm > off' -ex 'set pagination off' -ex 'thread apply all bt full' -ex > 'quit' /usr/sbin/ns-slapd > /var/log/dirsrv/slapd-SPRINKLR-COM/core.15243 stacktrace.`date > +%s`.txt 2>&1^C > [root at prod-ipa-master-int slapd-SPRINKLR-COM]# gdb -ex 'set confirm > off' -ex 'set pagination off' -ex 'thread apply all bt full' -ex > 'quit' /usr/sbin/ns-slapd > /var/log/dirsrv/slapd-SPRINKLR-COM/core.15243 > stacktrace.`date > +%s`.txt 2>&1 > [root at prod-ipa-master-int slapd-SPRINKLR-COM]# ls -ltr > total 6404952 > -rw-------. 1 dirsrv dirsrv 63 Sep 8 02:55 errors.rotationinfo > -rw-------. 1 dirsrv dirsrv 63 Sep 8 02:55 audit.rotationinfo > -rw-------. 1 dirsrv dirsrv 0 Sep 8 02:55 audit > -rw-------. 1 dirsrv dirsrv 2551824384 Sep 12 17:32 core.10450 > -rw-------. 1 dirsrv dirsrv 1464463360 Sep 12 19:35 core.14709 > -rw-------. 1 dirsrv dirsrv 4483862528 Sep 13 01:05 core.15243 > -rw-------. 1 dirsrv dirsrv 66288165 Sep 13 02:10 errors > -rw-------. 1 dirsrv dirsrv 104964391 Sep 13 08:30 access.20160913-074214 > -rw-------. 1 dirsrv dirsrv 105021859 Sep 13 09:26 access.20160913-083046 > -rw-------. 1 dirsrv dirsrv 104861746 Sep 13 10:31 access.20160913-092646 > -rw-------. 1 dirsrv dirsrv 105069140 Sep 13 11:36 access.20160913-103137 > -rw-------. 1 dirsrv dirsrv 104913480 Sep 13 12:41 access.20160913-113638 > -rw-------. 1 dirsrv dirsrv 105186788 Sep 13 13:46 access.20160913-124118 > -rw-------. 1 dirsrv dirsrv 105162159 Sep 13 14:51 access.20160913-134619 > -rw-------. 1 dirsrv dirsrv 105256624 Sep 13 15:56 access.20160913-145120 > -rw-------. 1 dirsrv dirsrv 105231158 Sep 13 17:01 access.20160913-155620 > -rw-------. 1 dirsrv dirsrv 1044 Sep 13 17:01 access.rotationinfo > -rw-r--r--. 1 root root 19287 Sep 13 17:28 > stacktrace.1473787719.txt > -rw-------. 1 dirsrv dirsrv 46949570 Sep 13 17:30 access > -rw-r--r--. 1 root root 20007 Sep 13 17:30 > stacktrace.1473787812.txt > [root at prod-ipa-master-int slapd-SPRINKLR-COM]# less > stacktrace.1473787812.txt > [root at prod-ipa-master-int slapd-SPRINKLR-COM]# cat > stacktrace.1473787812.txt > GNU gdb (GDB) Red Hat Enterprise Linux 7.6.1-80.el7 > Copyright (C) 2013 Free Software Foundation, Inc. > License GPLv3+: GNU GPL version 3 or later > > This is free software: you are free to change and redistribute it. > There is NO WARRANTY, to the extent permitted by law. Type "show copying" > and "show warranty" for details. > This GDB was configured as "x86_64-redhat-linux-gnu". > For bug reporting instructions, please see: > ... > Reading symbols from /usr/sbin/ns-slapd... > warning: the debug information found in > "/usr/lib/debug//usr/sbin/ns-slapd.debug" does not match > "/usr/sbin/ns-slapd" (CRC mismatch). > > > warning: the debug information found in > "/usr/lib/debug/usr/sbin/ns-slapd.debug" does not match > "/usr/sbin/ns-slapd" (CRC mismatch). > > Reading symbols from /usr/sbin/ns-slapd...(no debugging symbols > found)...done. > (no debugging symbols found)...done. > [New LWP 15255] > [New LWP 15286] > [New LWP 15245] > [New LWP 15246] > [New LWP 15247] > [New LWP 15248] > [New LWP 15243] > > warning: the debug information found in > "/usr/lib/debug//usr/lib64/dirsrv/libslapd.so.0.0.0.debug" does not > match "/usr/lib64/dirsrv/libslapd.so.0" (CRC mismatch). > > > warning: the debug information found in > "/usr/lib/debug/usr/lib64/dirsrv/libslapd.so.0.0.0.debug" does not > match "/usr/lib64/dirsrv/libslapd.so.0" (CRC mismatch). > > [Thread debugging using libthread_db enabled] > Using host libthread_db library "/lib64/libthread_db.so.1". > > warning: the debug information found in > "/usr/lib/debug//usr/lib64/dirsrv/plugins/libsyntax-plugin.so.debug" > does not match "/usr/lib64/dirsrv/plugins/libsyntax-plugin.so" (CRC > mismatch). > > > warning: the debug information found in > "/usr/lib/debug/usr/lib64/dirsrv/plugins/libsyntax-plugin.so.debug" > does not match "/usr/lib64/dirsrv/plugins/libsyntax-plugin.so" (CRC > mismatch). > > > warning: the debug information found in > "/usr/lib/debug//usr/lib64/dirsrv/plugins/libbitwise-plugin.so.debug" > does not match "/usr/lib64/dirsrv/plugins/libbitwise-plugin.so" (CRC > mismatch). > > > warning: the debug information found in > "/usr/lib/debug/usr/lib64/dirsrv/plugins/libbitwise-plugin.so.debug" > does not match "/usr/lib64/dirsrv/plugins/libbitwise-plugin.so" (CRC > mismatch). > > > warning: the debug information found in > "/usr/lib/debug//usr/lib64/dirsrv/plugins/libcollation-plugin.so.debug" does > not match "/usr/lib64/dirsrv/plugins/libcollation-plugin.so" (CRC > mismatch). > > > warning: the debug information found in > "/usr/lib/debug/usr/lib64/dirsrv/plugins/libcollation-plugin.so.debug" > does not match "/usr/lib64/dirsrv/plugins/libcollation-plugin.so" (CRC > mismatch). > > > warning: the debug information found in > "/usr/lib/debug//usr/lib64/dirsrv/plugins/libpwdstorage-plugin.so.debug" > does not match "/usr/lib64/dirsrv/plugins/libpwdstorage-plugin.so" > (CRC mismatch). > > > warning: the debug information found in > "/usr/lib/debug/usr/lib64/dirsrv/plugins/libpwdstorage-plugin.so.debug" does > not match "/usr/lib64/dirsrv/plugins/libpwdstorage-plugin.so" (CRC > mismatch). > > > warning: the debug information found in > "/usr/lib/debug//usr/lib64/dirsrv/plugins/libattr-unique-plugin.so.debug" > does not match "/usr/lib64/dirsrv/plugins/libattr-unique-plugin.so" > (CRC mismatch). > > > warning: the debug information found in > "/usr/lib/debug/usr/lib64/dirsrv/plugins/libattr-unique-plugin.so.debug" > does not match "/usr/lib64/dirsrv/plugins/libattr-unique-plugin.so" > (CRC mismatch). > > > warning: the debug information found in > "/usr/lib/debug//usr/lib64/dirsrv/plugins/libacctpolicy-plugin.so.debug" > does not match "/usr/lib64/dirsrv/plugins/libacctpolicy-plugin.so" > (CRC mismatch). > > > warning: the debug information found in > "/usr/lib/debug/usr/lib64/dirsrv/plugins/libacctpolicy-plugin.so.debug" does > not match "/usr/lib64/dirsrv/plugins/libacctpolicy-plugin.so" (CRC > mismatch). > > > warning: the debug information found in > "/usr/lib/debug//usr/lib64/dirsrv/plugins/libacctusability-plugin.so.debug" > does not match "/usr/lib64/dirsrv/plugins/libacctusability-plugin.so" > (CRC mismatch). > > > warning: the debug information found in > "/usr/lib/debug/usr/lib64/dirsrv/plugins/libacctusability-plugin.so.debug" > does not match "/usr/lib64/dirsrv/plugins/libacctusability-plugin.so" > (CRC mismatch). > > > warning: the debug information found in > "/usr/lib/debug//usr/lib64/dirsrv/plugins/libacl-plugin.so.debug" does > not match "/usr/lib64/dirsrv/plugins/libacl-plugin.so" (CRC mismatch). > > > warning: the debug information found in > "/usr/lib/debug/usr/lib64/dirsrv/plugins/libacl-plugin.so.debug" does > not match "/usr/lib64/dirsrv/plugins/libacl-plugin.so" (CRC mismatch). > > > warning: the debug information found in > "/usr/lib/debug//usr/lib64/dirsrv/libns-dshttpd.so.0.0.0.debug" does > not match "/usr/lib64/dirsrv/libns-dshttpd.so.0" (CRC mismatch). > > > warning: the debug information found in > "/usr/lib/debug/usr/lib64/dirsrv/libns-dshttpd.so.0.0.0.debug" does > not match "/usr/lib64/dirsrv/libns-dshttpd.so.0" (CRC mismatch). > > > warning: the debug information found in > "/usr/lib/debug//usr/lib64/dirsrv/plugins/libautomember-plugin.so.debug" > does not match "/usr/lib64/dirsrv/plugins/libautomember-plugin.so" > (CRC mismatch). > > > warning: the debug information found in > "/usr/lib/debug/usr/lib64/dirsrv/plugins/libautomember-plugin.so.debug" does > not match "/usr/lib64/dirsrv/plugins/libautomember-plugin.so" (CRC > mismatch). > > > warning: the debug information found in > "/usr/lib/debug//usr/lib64/dirsrv/plugins/libchainingdb-plugin.so.debug" > does not match "/usr/lib64/dirsrv/plugins/libchainingdb-plugin.so" > (CRC mismatch). > > > warning: the debug information found in > "/usr/lib/debug/usr/lib64/dirsrv/plugins/libchainingdb-plugin.so.debug" does > not match "/usr/lib64/dirsrv/plugins/libchainingdb-plugin.so" (CRC > mismatch). > > > warning: the debug information found in > "/usr/lib/debug//usr/lib64/dirsrv/plugins/libcos-plugin.so.debug" does > not match "/usr/lib64/dirsrv/plugins/libcos-plugin.so" (CRC mismatch). > > > warning: the debug information found in > "/usr/lib/debug/usr/lib64/dirsrv/plugins/libcos-plugin.so.debug" does > not match "/usr/lib64/dirsrv/plugins/libcos-plugin.so" (CRC mismatch). > > > warning: the debug information found in > "/usr/lib/debug//usr/lib64/dirsrv/plugins/libderef-plugin.so.debug" > does not match "/usr/lib64/dirsrv/plugins/libderef-plugin.so" (CRC > mismatch). > > > warning: the debug information found in > "/usr/lib/debug/usr/lib64/dirsrv/plugins/libderef-plugin.so.debug" > does not match "/usr/lib64/dirsrv/plugins/libderef-plugin.so" (CRC > mismatch). > > > warning: the debug information found in > "/usr/lib/debug//usr/lib64/dirsrv/plugins/libdna-plugin.so.debug" does > not match "/usr/lib64/dirsrv/plugins/libdna-plugin.so" (CRC mismatch). > > > warning: the debug information found in > "/usr/lib/debug/usr/lib64/dirsrv/plugins/libdna-plugin.so.debug" does > not match "/usr/lib64/dirsrv/plugins/libdna-plugin.so" (CRC mismatch). > > > warning: the debug information found in > "/usr/lib/debug//usr/lib64/dirsrv/plugins/libhttp-client-plugin.so.debug" > does not match "/usr/lib64/dirsrv/plugins/libhttp-client-plugin.so" > (CRC mismatch). > > > warning: the debug information found in > "/usr/lib/debug/usr/lib64/dirsrv/plugins/libhttp-client-plugin.so.debug" > does not match "/usr/lib64/dirsrv/plugins/libhttp-client-plugin.so" > (CRC mismatch). > > > warning: the debug information found in > "/usr/lib/debug//usr/lib64/dirsrv/plugins/libback-ldbm.so.debug" does > not match "/usr/lib64/dirsrv/plugins/libback-ldbm.so" (CRC mismatch). > > > warning: the debug information found in > "/usr/lib/debug/usr/lib64/dirsrv/plugins/libback-ldbm.so.debug" does > not match "/usr/lib64/dirsrv/plugins/libback-ldbm.so" (CRC mismatch). > > > warning: the debug information found in > "/usr/lib/debug//usr/lib64/dirsrv/plugins/libreplication-plugin.so.debug" > does not match "/usr/lib64/dirsrv/plugins/libreplication-plugin.so" > (CRC mismatch). > > > warning: the debug information found in > "/usr/lib/debug/usr/lib64/dirsrv/plugins/libreplication-plugin.so.debug" > does not match "/usr/lib64/dirsrv/plugins/libreplication-plugin.so" > (CRC mismatch). > > > warning: the debug information found in > "/usr/lib/debug//usr/lib64/dirsrv/plugins/liblinkedattrs-plugin.so.debug" > does not match "/usr/lib64/dirsrv/plugins/liblinkedattrs-plugin.so" > (CRC mismatch). > > > warning: the debug information found in > "/usr/lib/debug/usr/lib64/dirsrv/plugins/liblinkedattrs-plugin.so.debug" > does not match "/usr/lib64/dirsrv/plugins/liblinkedattrs-plugin.so" > (CRC mismatch). > > > warning: the debug information found in > "/usr/lib/debug//usr/lib64/dirsrv/plugins/libmanagedentries-plugin.so.debug" > does not match "/usr/lib64/dirsrv/plugins/libmanagedentries-plugin.so" > (CRC mismatch). > > > warning: the debug information found in > "/usr/lib/debug/usr/lib64/dirsrv/plugins/libmanagedentries-plugin.so.debug" > does not match "/usr/lib64/dirsrv/plugins/libmanagedentries-plugin.so" > (CRC mismatch). > > > warning: the debug information found in > "/usr/lib/debug//usr/lib64/dirsrv/plugins/libmemberof-plugin.so.debug" > does not match "/usr/lib64/dirsrv/plugins/libmemberof-plugin.so" (CRC > mismatch). > > > warning: the debug information found in > "/usr/lib/debug/usr/lib64/dirsrv/plugins/libmemberof-plugin.so.debug" > does not match "/usr/lib64/dirsrv/plugins/libmemberof-plugin.so" (CRC > mismatch). > > > warning: the debug information found in > "/usr/lib/debug//usr/lib64/dirsrv/plugins/libpam-passthru-plugin.so.debug" > does not match "/usr/lib64/dirsrv/plugins/libpam-passthru-plugin.so" > (CRC mismatch). > > > warning: the debug information found in > "/usr/lib/debug/usr/lib64/dirsrv/plugins/libpam-passthru-plugin.so.debug" > does not match "/usr/lib64/dirsrv/plugins/libpam-passthru-plugin.so" > (CRC mismatch). > > > warning: the debug information found in > "/usr/lib/debug//usr/lib64/dirsrv/plugins/libpassthru-plugin.so.debug" > does not match "/usr/lib64/dirsrv/plugins/libpassthru-plugin.so" (CRC > mismatch). > > > warning: the debug information found in > "/usr/lib/debug/usr/lib64/dirsrv/plugins/libpassthru-plugin.so.debug" > does not match "/usr/lib64/dirsrv/plugins/libpassthru-plugin.so" (CRC > mismatch). > > > warning: the debug information found in > "/usr/lib/debug//usr/lib64/dirsrv/plugins/libposix-winsync-plugin.so.debug" > does not match "/usr/lib64/dirsrv/plugins/libposix-winsync-plugin.so" > (CRC mismatch). > > > warning: the debug information found in > "/usr/lib/debug/usr/lib64/dirsrv/plugins/libposix-winsync-plugin.so.debug" > does not match "/usr/lib64/dirsrv/plugins/libposix-winsync-plugin.so" > (CRC mismatch). > > > warning: the debug information found in > "/usr/lib/debug//usr/lib64/dirsrv/plugins/libreferint-plugin.so.debug" > does not match "/usr/lib64/dirsrv/plugins/libreferint-plugin.so" (CRC > mismatch). > > > warning: the debug information found in > "/usr/lib/debug/usr/lib64/dirsrv/plugins/libreferint-plugin.so.debug" > does not match "/usr/lib64/dirsrv/plugins/libreferint-plugin.so" (CRC > mismatch). > > > warning: the debug information found in > "/usr/lib/debug//usr/lib64/dirsrv/plugins/libretrocl-plugin.so.debug" > does not match "/usr/lib64/dirsrv/plugins/libretrocl-plugin.so" (CRC > mismatch). > > > warning: the debug information found in > "/usr/lib/debug/usr/lib64/dirsrv/plugins/libretrocl-plugin.so.debug" > does not match "/usr/lib64/dirsrv/plugins/libretrocl-plugin.so" (CRC > mismatch). > > > warning: the debug information found in > "/usr/lib/debug//usr/lib64/dirsrv/plugins/libroles-plugin.so.debug" > does not match "/usr/lib64/dirsrv/plugins/libroles-plugin.so" (CRC > mismatch). > > > warning: the debug information found in > "/usr/lib/debug/usr/lib64/dirsrv/plugins/libroles-plugin.so.debug" > does not match "/usr/lib64/dirsrv/plugins/libroles-plugin.so" (CRC > mismatch). > > > warning: the debug information found in > "/usr/lib/debug//usr/lib64/dirsrv/plugins/librootdn-access-plugin.so.debug" > does not match "/usr/lib64/dirsrv/plugins/librootdn-access-plugin.so" > (CRC mismatch). > > > warning: the debug information found in > "/usr/lib/debug/usr/lib64/dirsrv/plugins/librootdn-access-plugin.so.debug" > does not match "/usr/lib64/dirsrv/plugins/librootdn-access-plugin.so" > (CRC mismatch). > > > warning: the debug information found in > "/usr/lib/debug//usr/lib64/dirsrv/plugins/libschemareload-plugin.so.debug" > does not match "/usr/lib64/dirsrv/plugins/libschemareload-plugin.so" > (CRC mismatch). > > > warning: the debug information found in > "/usr/lib/debug/usr/lib64/dirsrv/plugins/libschemareload-plugin.so.debug" > does not match "/usr/lib64/dirsrv/plugins/libschemareload-plugin.so" > (CRC mismatch). > > > warning: the debug information found in > "/usr/lib/debug//usr/lib64/dirsrv/plugins/libstatechange-plugin.so.debug" > does not match "/usr/lib64/dirsrv/plugins/libstatechange-plugin.so" > (CRC mismatch). > > > warning: the debug information found in > "/usr/lib/debug/usr/lib64/dirsrv/plugins/libstatechange-plugin.so.debug" > does not match "/usr/lib64/dirsrv/plugins/libstatechange-plugin.so" > (CRC mismatch). > > > warning: the debug information found in > "/usr/lib/debug//usr/lib64/dirsrv/plugins/libusn-plugin.so.debug" does > not match "/usr/lib64/dirsrv/plugins/libusn-plugin.so" (CRC mismatch). > > > warning: the debug information found in > "/usr/lib/debug/usr/lib64/dirsrv/plugins/libusn-plugin.so.debug" does > not match "/usr/lib64/dirsrv/plugins/libusn-plugin.so" (CRC mismatch). > > > warning: the debug information found in > "/usr/lib/debug//usr/lib64/dirsrv/plugins/libviews-plugin.so.debug" > does not match "/usr/lib64/dirsrv/plugins/libviews-plugin.so" (CRC > mismatch). > > > warning: the debug information found in > "/usr/lib/debug/usr/lib64/dirsrv/plugins/libviews-plugin.so.debug" > does not match "/usr/lib64/dirsrv/plugins/libviews-plugin.so" (CRC > mismatch). > > Core was generated by `/usr/sbin/ns-slapd -D > /etc/dirsrv/slapd-SPRINKLR-COM -i /var/run/dirsrv/slapd-S'. > Program terminated with signal 11, Segmentation fault. > #0 0x00007f47ec77b51c in cos_cache_template_index_compare () from > /usr/lib64/dirsrv/plugins/libcos-plugin.so > > Thread 7 (Thread 0x7f47f686a840 (LWP 15243)): > #0 0x00007f47f3f5884d in fsync () from /lib64/libpthread.so.0 > No symbol table info available. > #1 0x00007f47f45ad263 in pt_Fsync () from /lib64/libnspr4.so > No symbol table info available. > #2 0x00007f47f63ae908 in vslapd_log_error () from > /usr/lib64/dirsrv/libslapd.so.0 > No symbol table info available. > #3 0x00007f47f63b3afd in slapd_log_error_proc_internal () from > /usr/lib64/dirsrv/libslapd.so.0 > No symbol table info available. > #4 0x00007f47f63b0f59 in slapd_log_error_proc () from > /usr/lib64/dirsrv/libslapd.so.0 > No symbol table info available. > #5 0x00007f47e9a4a0f8 in dblayer_pre_close () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > No symbol table info available. > #6 0x00007f47e9a4a216 in dblayer_close () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > No symbol table info available. > #7 0x00007f47e9a460f1 in ldbm_back_close () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > No symbol table info available. > #8 0x00007f47f63ce848 in plugin_call_func () from > /usr/lib64/dirsrv/libslapd.so.0 > No symbol table info available. > #9 0x00007f47f63d1feb in plugin_dependency_closeall () from > /usr/lib64/dirsrv/libslapd.so.0 > No symbol table info available. > #10 0x00007f47f63d205e in plugin_closeall () from > /usr/lib64/dirsrv/libslapd.so.0 > No symbol table info available. > #11 0x00007f47f68a8014 in slapd_daemon () > No symbol table info available. > #12 0x00007f47f6898c6c in main () > No symbol table info available. > > Thread 6 (Thread 0x7f47e35d7700 (LWP 15248)): > #0 0x00007f47f3c76413 in select () from /lib64/libc.so.6 > No symbol table info available. > #1 0x00007f47f64020e9 in DS_Sleep () from /usr/lib64/dirsrv/libslapd.so.0 > No symbol table info available. > #2 0x00007f47e9a47707 in perf_threadmain () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > No symbol table info available. > #3 0x00007f47f45b096b in _pt_root () from /lib64/libnspr4.so > No symbol table info available. > #4 0x00007f47f3f51dc5 in start_thread () from /lib64/libpthread.so.0 > No symbol table info available. > #5 0x00007f47f3c7eced in clone () from /lib64/libc.so.6 > No symbol table info available. > > Thread 5 (Thread 0x7f47e3dd8700 (LWP 15247)): > #0 0x00007f47f3c76413 in select () from /lib64/libc.so.6 > No symbol table info available. > #1 0x00007f47f64020e9 in DS_Sleep () from /usr/lib64/dirsrv/libslapd.so.0 > No symbol table info available. > #2 0x00007f47e9a4cb7f in trickle_threadmain () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > No symbol table info available. > #3 0x00007f47f45b096b in _pt_root () from /lib64/libnspr4.so > No symbol table info available. > #4 0x00007f47f3f51dc5 in start_thread () from /lib64/libpthread.so.0 > No symbol table info available. > #5 0x00007f47f3c7eced in clone () from /lib64/libc.so.6 > No symbol table info available. > > Thread 4 (Thread 0x7f47e45d9700 (LWP 15246)): > #0 0x00007f47f3c76413 in select () from /lib64/libc.so.6 > No symbol table info available. > #1 0x00007f47f64020e9 in DS_Sleep () from /usr/lib64/dirsrv/libslapd.so.0 > No symbol table info available. > #2 0x00007f47e9a50a26 in checkpoint_threadmain () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > No symbol table info available. > #3 0x00007f47f45b096b in _pt_root () from /lib64/libnspr4.so > No symbol table info available. > #4 0x00007f47f3f51dc5 in start_thread () from /lib64/libpthread.so.0 > No symbol table info available. > #5 0x00007f47f3c7eced in clone () from /lib64/libc.so.6 > No symbol table info available. > > Thread 3 (Thread 0x7f47e4dda700 (LWP 15245)): > #0 0x00007f47f3c76413 in select () from /lib64/libc.so.6 > No symbol table info available. > #1 0x00007f47f64020e9 in DS_Sleep () from /usr/lib64/dirsrv/libslapd.so.0 > No symbol table info available. > #2 0x00007f47e9a4c907 in deadlock_threadmain () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > No symbol table info available. > #3 0x00007f47f45b096b in _pt_root () from /lib64/libnspr4.so > No symbol table info available. > #4 0x00007f47f3f51dc5 in start_thread () from /lib64/libpthread.so.0 > No symbol table info available. > #5 0x00007f47f3c7eced in clone () from /lib64/libc.so.6 > No symbol table info available. > > Thread 2 (Thread 0x7f47bf7ee700 (LWP 15286)): > #0 0x00007f47f3c76413 in select () from /lib64/libc.so.6 > No symbol table info available. > #1 0x00007f47f64020e9 in DS_Sleep () from /usr/lib64/dirsrv/libslapd.so.0 > No symbol table info available. > #2 0x00007f47f68a3325 in time_thread () > No symbol table info available. > #3 0x00007f47f45b096b in _pt_root () from /lib64/libnspr4.so > No symbol table info available. > #4 0x00007f47f3f51dc5 in start_thread () from /lib64/libpthread.so.0 > No symbol table info available. > #5 0x00007f47f3c7eced in clone () from /lib64/libc.so.6 > No symbol table info available. > > Thread 1 (Thread 0x7f47f67e8700 (LWP 15255)): > #0 0x00007f47ec77b51c in cos_cache_template_index_compare () from > /usr/lib64/dirsrv/plugins/libcos-plugin.so > No symbol table info available. > #1 0x00007f47f3bbef08 in bsearch () from /lib64/libc.so.6 > No symbol table info available. > #2 0x00007f47ec77fddd in cos_cache_change_notify () from > /usr/lib64/dirsrv/plugins/libcos-plugin.so > No symbol table info available. > #3 0x00007f47ec77abed in cos_post_op () from > /usr/lib64/dirsrv/plugins/libcos-plugin.so > No symbol table info available. > #4 0x00007f47f63ce848 in plugin_call_func () from > /usr/lib64/dirsrv/libslapd.so.0 > No symbol table info available. > #5 0x00007f47f63ceb13 in plugin_call_plugins () from > /usr/lib64/dirsrv/libslapd.so.0 > No symbol table info available. > #6 0x00007f47f63832dd in op_shared_delete () from > /usr/lib64/dirsrv/libslapd.so.0 > No symbol table info available. > #7 0x00007f47f6383412 in delete_internal_pb () from > /usr/lib64/dirsrv/libslapd.so.0 > No symbol table info available. > #8 0x00007f47e82f635e in delete_changerecord () from > /usr/lib64/dirsrv/plugins/libretrocl-plugin.so > No symbol table info available. > #9 0x00007f47e82f6631 in changelog_trim_thread_fn () from > /usr/lib64/dirsrv/plugins/libretrocl-plugin.so > No symbol table info available. > #10 0x00007f47f45b096b in _pt_root () from /lib64/libnspr4.so > No symbol table info available. > #11 0x00007f47f3f51dc5 in start_thread () from /lib64/libpthread.so.0 > No symbol table info available. > #12 0x00007f47f3c7eced in clone () from /lib64/libc.so.6 > No symbol table info available. > > Thanks, > > Rakesh > -- Red Hat GmbH, http://www.de.redhat.com/, Registered seat: Grasbrunn, Commercial register: Amtsgericht Muenchen, HRB 153243, Managing Directors: Charles Cachera, Michael Cunningham, Michael O'Neill, Eric Shander From blipton at redhat.com Wed Sep 14 12:45:55 2016 From: blipton at redhat.com (Ben Lipton) Date: Wed, 14 Sep 2016 08:45:55 -0400 Subject: [Freeipa-users] ipa: ERROR: Certificate format error: (SEC_ERROR_LEGACY_DATABASE) The certificate/key database is in an old, unsupported format. In-Reply-To: <57D81AA1.4050400@redhat.com> References: <57D015D8.7080507@redhat.com> <57D166B9.3070201@redhat.com> <57D70676.8080107@redhat.com> <57D81AA1.4050400@redhat.com> Message-ID: This may be resolved already, but just in case it's helpful: On 09/13/2016 11:26 AM, Rob Crittenden wrote: > Natxo Asenjo wrote: >> hi, >> >> >> On Mon, Sep 12, 2016 at 9:48 PM, Rob Crittenden > > wrote: >> >> Natxo Asenjo wrote: >> >> hi, >> >> I can reproduce this everytime. Restarting httpd fixes it for a >> while, >> but then ik stops working: >> >> $ ipa cert-show 1 >> ipa: ERROR: cannot connect to >> 'https://kdc01.unix.domain.tld:443/ca/agent/ca/displayBySerial >> ': >> (SEC_ERROR_LEGACY_DATABASE) The certificate/key database is in >> an old, >> unsupported format. >> >> >> It is very strange that it goes from a working to a non-working >> state. >> >> I have only two suggestions: >> >> 1. Create /etc/ipa/server.conf with a [global] section and >> debug=True in it, restart httpd. Your log will be quite a bit more >> verbose but given it reproduces so quickly hopefully won't be too >> big a deal. That might show something. +1 to this. With debug=True there should be tracebacks for your CertificateFormatErrors. >> >> 2. Try brute force with strace. Finding the right httpd process to >> strace can be frustrating but usually there are only 8 and they >> rotate so eventually you should get the right one. >> >> >> Could I send you the log files privately? > > Sure. > > rob > One other note - this could be a permissions issue. NSS seems to produce this confusing error message when it can't access the database, even if the format of the database is actually fine. $ sudo chown root:root /tmp/certs $ certutil -N -d /tmp/certs certutil: function failed: SEC_ERROR_LEGACY_DATABASE: The certificate/key database is in an old, unsupported format. From jeffrey.armstrong at gasoc.com Wed Sep 14 13:20:17 2016 From: jeffrey.armstrong at gasoc.com (Armstrong, Jeffrey) Date: Wed, 14 Sep 2016 13:20:17 +0000 Subject: [Freeipa-users] Migration Question Message-ID: <3DAC7A5927B8594195EA704FB41255B076BE6534@Supernatural2.gafoc.com> Hi My company is migrating from RedHat Linux to Oracle Linux. I warned them that IdM could be a problem. Does anyone know If IPA works after the migration? Jeff Armstrong -------------- next part -------------- An HTML attachment was scrubbed... URL: From jalvarez at cyberfuel.com Wed Sep 14 14:37:14 2016 From: jalvarez at cyberfuel.com (Jose Alvarez R.) Date: Wed, 14 Sep 2016 08:37:14 -0600 Subject: [Freeipa-users] About AllowGroups with sshd In-Reply-To: <20160913082139.gishru3hcpzcpclt@hendrix> References: <0afb01d20d0e$d99b9100$8cd2b300$@cyberfuel.com> <20160913082139.gishru3hcpzcpclt@hendrix> Message-ID: <09d201d20e95$7cbcce30$76366a90$@cyberfuel.com> Hi Jakub Thanks for your response. It's an option, but my backups servers I will not add to the FreeIPA server. Then, I cannot use the option HBAC, because I want my backup server can connect with root to some client server of my FreeIPA Server. If I'm doing something wrong, please let me know Thanks, Regards Jose Alvarez R. -----Original Message----- From: freeipa-users-bounces at redhat.com [mailto:freeipa-users-bounces at redhat.com] On Behalf Of Jakub Hrozek Sent: martes 13 de septiembre de 2016 02:22 a.m. To: freeipa-users at redhat.com Subject: Re: [Freeipa-users] About AllowGroups with sshd On Mon, Sep 12, 2016 at 10:00:57AM -0600, Jose Alvarez R. wrote: > Hello > > > > I have an question > > > > I have an FreeIPA 3.0 server(CentOS 6) with some clients servers(CentOS 6). > I wants enable root a two servers this servers, because they are > backup servers. > > > > I add theses lines in /etc/ssh/sshd_config of a client server. > > > > AllowUsers root at 192.168.20.2 > > AllowUsers root at 192.168.20.90 > > PermitRootLogin yes > > > > This working, but when try login with my user IPA, I can't login. > > > > I add the line "AllowGroups" with my group of users_IPA > > > > AllowGroups > > > > But not working, Can you help me ? > > > > Thanks, Regards > > > > Jose Alvarez. I know I'm not answering your question directly, but isn't it better to use HBAC with IPA and centralize the access control rather than edit config files on the clients? -- Manage your subscription for the Freeipa-users mailing list: https://www.redhat.com/mailman/listinfo/freeipa-users Go to http://freeipa.org for more info on the project From jgiger at verizon.com Wed Sep 14 15:17:57 2016 From: jgiger at verizon.com (Giger, Justean) Date: Wed, 14 Sep 2016 11:17:57 -0400 Subject: [Freeipa-users] [E] Migration Question Message-ID: We did the same and have had zero issues. In fact, one overzealous colleague moved one out of our 5 IDM servers to Oracle while all the others were still on Red Hat and things still worked. I have not tried to get support for IDM with Oracle though so not sure how that goes. From: > on behalf of "Armstrong, Jeffrey" > Date: Wednesday, September 14, 2016 at 6:20 AM To: "freeipa-users at redhat.com" > Subject: [E] [Freeipa-users] Migration Question Hi My company is migrating from RedHat Linux to Oracle Linux. I warned them that IdM could be a problem. Does anyone know If IPA works after the migration? Jeff Armstrong -------------- next part -------------- An HTML attachment was scrubbed... URL: From bahanw042014 at gmail.com Wed Sep 14 15:59:13 2016 From: bahanw042014 at gmail.com (bahan w) Date: Wed, 14 Sep 2016 17:59:13 +0200 Subject: [Freeipa-users] SSL alert: CERT_VerifyCertificateNow: verify certificate failed for cert Server-Cert of family Message-ID: Hello ! I send you this mail because I cannot restart my test IPA server. When I try to start it with service ipa start, I got the following error message : ### # service ipa start Starting Directory Service Starting dirsrv: ...[14/Sep/2016:17:57:23 +0200] - SSL alert: CERT_VerifyCertificateNow: verify certificate failed for cert Server-Cert of family cn=RSA,cn=encryption,cn=config (Netscape Portable Runtime error -8181 - Peer's Certificate has expired.) [ OK ] PKI-IPA...[14/Sep/2016:17:57:33 +0200] - SSL alert: CERT_VerifyCertificateNow: verify certificate failed for cert Server-Cert of family cn=RSA,cn=encryption,cn=config (Netscape Portable Runtime error -8181 - Peer's Certificate has expired.) [ OK ] Starting KDC Service Starting Kerberos 5 KDC: [ OK ] Starting KPASSWD Service Starting Kerberos 5 Admin Server: [ OK ] Starting MEMCACHE Service Starting ipa_memcached: [ OK ] Starting HTTP Service Starting httpd: [FAILED] Failed to start HTTP Service Shutting down Stopping Kerberos 5 KDC: [ OK ] Stopping Kerberos 5 Admin Server: [ OK ] Stopping ipa_memcached: [ OK ] Stopping httpd: [FAILED] Stopping pki-ca: [ OK ] Shutting down dirsrv: ... [ OK ] PKI-IPA... [ OK ] Aborting ipactl # service ipa status Directory Service: STOPPED Failed to get list of services to probe status: Directory Server is stopped ### Do you know how to renew the SSL certificate used for the IPA Server ? Best regards. Bahan -------------- next part -------------- An HTML attachment was scrubbed... URL: From mbasti at redhat.com Wed Sep 14 16:14:58 2016 From: mbasti at redhat.com (Martin Basti) Date: Wed, 14 Sep 2016 18:14:58 +0200 Subject: [Freeipa-users] SSL alert: CERT_VerifyCertificateNow: verify certificate failed for cert Server-Cert of family In-Reply-To: References: Message-ID: <7d17ebb6-eba6-b560-bcf9-e016546095c1@redhat.com> On 14.09.2016 17:59, bahan w wrote: > Hello ! > > I send you this mail because I cannot restart my test IPA server. > > When I try to start it with service ipa start, I got the following > error message : > ### > # service ipa start > Starting Directory Service > Starting dirsrv: > ...[14/Sep/2016:17:57:23 +0200] - SSL alert: > CERT_VerifyCertificateNow: verify certificate failed for cert > Server-Cert of family cn=RSA,cn=encryption,cn=config (Netscape > Portable Runtime error -8181 - Peer's Certificate has expired.) > [ OK ] > PKI-IPA...[14/Sep/2016:17:57:33 +0200] - SSL alert: > CERT_VerifyCertificateNow: verify certificate failed for cert > Server-Cert of family cn=RSA,cn=encryption,cn=config (Netscape > Portable Runtime error -8181 - Peer's Certificate has expired.) > [ OK ] > Starting KDC Service > Starting Kerberos 5 KDC: [ OK ] > Starting KPASSWD Service > Starting Kerberos 5 Admin Server: [ OK ] > Starting MEMCACHE Service > Starting ipa_memcached: [ OK ] > Starting HTTP Service > Starting httpd: [FAILED] > Failed to start HTTP Service > Shutting down > Stopping Kerberos 5 KDC: [ OK ] > Stopping Kerberos 5 Admin Server: [ OK ] > Stopping ipa_memcached: [ OK ] > Stopping httpd: [FAILED] > Stopping pki-ca: [ OK ] > Shutting down dirsrv: > ... [ OK ] > PKI-IPA... [ OK ] > Aborting ipactl > > # service ipa status > Directory Service: STOPPED > Failed to get list of services to probe status: > Directory Server is stopped > ### > > Do you know how to renew the SSL certificate used for the IPA Server ? > > Best regards. > > Bahan > > > Hello, please run # ipactl start --force # getcert list (to detect which certificate is outdated, I suspect DS cert (or to get more info why it has not been renewed)) If getcert does work (I'm not sure if ti is able to work without httpd), you probable need to move time back to past where cert is valid, start IPA and try again. Please find ID outdated certificate and try resubmit it (CA and DS must be running) # getcert resubmit -i 20160914122036 (use you ID :) ) This should renew cert, check status with getcert list Move time back to future (if needed) Try to restart IPA Martin^2 -------------- next part -------------- An HTML attachment was scrubbed... URL: From bahanw042014 at gmail.com Wed Sep 14 16:18:22 2016 From: bahanw042014 at gmail.com (bahan w) Date: Wed, 14 Sep 2016 18:18:22 +0200 Subject: [Freeipa-users] SSL alert: CERT_VerifyCertificateNow: verify certificate failed for cert Server-Cert of family In-Reply-To: <7d17ebb6-eba6-b560-bcf9-e016546095c1@redhat.com> References: <7d17ebb6-eba6-b560-bcf9-e016546095c1@redhat.com> Message-ID: Oh I forgot to add that my version of ipa is quite old : ### # rpm -qa | grep ipa-server ipa-server-3.0.0-25.el6.x86_64 ### When I try the command you gave me I got the following error : ### # ipactl start --force Usage: ipactl start|stop|restart|status ipactl: error: no such option: --force ### Best regards. Bahan On Wed, Sep 14, 2016 at 6:14 PM, Martin Basti wrote: > > > On 14.09.2016 17:59, bahan w wrote: > > Hello ! > > I send you this mail because I cannot restart my test IPA server. > > When I try to start it with service ipa start, I got the following error > message : > ### > # service ipa start > Starting Directory Service > Starting dirsrv: > ...[14/Sep/2016:17:57:23 +0200] - SSL alert: > CERT_VerifyCertificateNow: verify certificate failed for cert Server-Cert > of family cn=RSA,cn=encryption,cn=config (Netscape Portable Runtime error > -8181 - Peer's Certificate has expired.) > [ OK ] > PKI-IPA...[14/Sep/2016:17:57:33 +0200] - SSL alert: > CERT_VerifyCertificateNow: verify certificate failed for cert Server-Cert > of family cn=RSA,cn=encryption,cn=config (Netscape Portable Runtime error > -8181 - Peer's Certificate has expired.) > [ OK ] > Starting KDC Service > Starting Kerberos 5 KDC: [ OK ] > Starting KPASSWD Service > Starting Kerberos 5 Admin Server: [ OK ] > Starting MEMCACHE Service > Starting ipa_memcached: [ OK ] > Starting HTTP Service > Starting httpd: [FAILED] > Failed to start HTTP Service > Shutting down > Stopping Kerberos 5 KDC: [ OK ] > Stopping Kerberos 5 Admin Server: [ OK ] > Stopping ipa_memcached: [ OK ] > Stopping httpd: [FAILED] > Stopping pki-ca: [ OK ] > Shutting down dirsrv: > ... [ OK ] > PKI-IPA... [ OK ] > Aborting ipactl > > # service ipa status > Directory Service: STOPPED > Failed to get list of services to probe status: > Directory Server is stopped > ### > > Do you know how to renew the SSL certificate used for the IPA Server ? > > Best regards. > > Bahan > > > > > > Hello, > > please run > > # ipactl start --force > # getcert list (to detect which certificate is outdated, I suspect DS cert > (or to get more info why it has not been renewed)) > > If getcert does work (I'm not sure if ti is able to work without httpd), > you probable need to move time back to past where cert is valid, start IPA > and try again. > > Please find ID outdated certificate and try resubmit it (CA and DS must be > running) > > # getcert resubmit -i 20160914122036 (use you ID :) ) > > This should renew cert, check status with getcert list > > Move time back to future (if needed) > > Try to restart IPA > > Martin^2 > -------------- next part -------------- An HTML attachment was scrubbed... URL: From mbasti at redhat.com Wed Sep 14 16:21:10 2016 From: mbasti at redhat.com (Martin Basti) Date: Wed, 14 Sep 2016 18:21:10 +0200 Subject: [Freeipa-users] SSL alert: CERT_VerifyCertificateNow: verify certificate failed for cert Server-Cert of family In-Reply-To: References: <7d17ebb6-eba6-b560-bcf9-e016546095c1@redhat.com> Message-ID: Then you have to start services manually, I don't know if the same steps will work with IPA 3.0.0, I don't remember, but you can try :) On 14.09.2016 18:18, bahan w wrote: > Oh I forgot to add that my version of ipa is quite old : > ### > # rpm -qa | grep ipa-server > ipa-server-3.0.0-25.el6.x86_64 > ### > > When I try the command you gave me I got the following error : > ### > # ipactl start --force > Usage: ipactl start|stop|restart|status > > > ipactl: error: no such option: --force > ### > > Best regards. > > Bahan > > On Wed, Sep 14, 2016 at 6:14 PM, Martin Basti > wrote: > > > > On 14.09.2016 17:59, bahan w wrote: >> Hello ! >> >> I send you this mail because I cannot restart my test IPA server. >> >> When I try to start it with service ipa start, I got the >> following error message : >> ### >> # service ipa start >> Starting Directory Service >> Starting dirsrv: >> ...[14/Sep/2016:17:57:23 +0200] - SSL alert: >> CERT_VerifyCertificateNow: verify certificate failed for cert >> Server-Cert of family cn=RSA,cn=encryption,cn=config (Netscape >> Portable Runtime error -8181 - Peer's Certificate has expired.) >> [ OK ] >> PKI-IPA...[14/Sep/2016:17:57:33 +0200] - SSL alert: >> CERT_VerifyCertificateNow: verify certificate failed for cert >> Server-Cert of family cn=RSA,cn=encryption,cn=config (Netscape >> Portable Runtime error -8181 - Peer's Certificate has expired.) >> [ OK ] >> Starting KDC Service >> Starting Kerberos 5 KDC: [ OK ] >> Starting KPASSWD Service >> Starting Kerberos 5 Admin Server: [ OK ] >> Starting MEMCACHE Service >> Starting ipa_memcached: [ OK ] >> Starting HTTP Service >> Starting httpd: [FAILED] >> Failed to start HTTP Service >> Shutting down >> Stopping Kerberos 5 KDC: [ OK ] >> Stopping Kerberos 5 Admin Server: [ OK ] >> Stopping ipa_memcached: [ OK ] >> Stopping httpd: [FAILED] >> Stopping pki-ca: [ OK ] >> Shutting down dirsrv: >> ... [ OK ] >> PKI-IPA... [ OK ] >> Aborting ipactl >> >> # service ipa status >> Directory Service: STOPPED >> Failed to get list of services to probe status: >> Directory Server is stopped >> ### >> >> Do you know how to renew the SSL certificate used for the IPA >> Server ? >> >> Best regards. >> >> Bahan >> >> >> > > > Hello, > > please run > > # ipactl start --force > # getcert list (to detect which certificate is outdated, I suspect > DS cert (or to get more info why it has not been renewed)) > > If getcert does work (I'm not sure if ti is able to work without > httpd), you probable need to move time back to past where cert is > valid, start IPA and try again. > > Please find ID outdated certificate and try resubmit it (CA and DS > must be running) > > # getcert resubmit -i 20160914122036 (use you ID :) ) > > This should renew cert, check status with getcert list > > Move time back to future (if needed) > > Try to restart IPA > > Martin^2 > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From mbasti at redhat.com Wed Sep 14 16:34:43 2016 From: mbasti at redhat.com (Martin Basti) Date: Wed, 14 Sep 2016 18:34:43 +0200 Subject: [Freeipa-users] SSL alert: CERT_VerifyCertificateNow: verify certificate failed for cert Server-Cert of family In-Reply-To: References: <7d17ebb6-eba6-b560-bcf9-e016546095c1@redhat.com> Message-ID: <250e8b50-5c4b-c8e5-8ec8-f3062cac4527@redhat.com> Please keep freeipa-users in CC, there si no sensitive information in getcert list output (you sanitized it) Folowing certificates are expired, please try to to resubmit them. I'm also worried about this error message: ca-error: Error setting up ccache for local "host" service using default keytab: Cannot contact any KDC for realm ''. is KDC running? > Request ID '20140528063919': > status: MONITORING > ca-error: Error setting up ccache for local "host" service > using default keytab: Cannot contact any KDC for realm ''. > stuck: no > key pair storage: > type=NSSDB,location='/etc/dirsrv/slapd-',nickname='Server-Cert',token='NSS > Certificate DB',pinfile='/etc/dirsrv/slapd-/pwdfile.txt' > certificate: > type=NSSDB,location='/etc/dirsrv/slapd-',nickname='Server-Cert',token='NSS > Certificate DB' > CA: IPA > issuer: CN=Certificate Authority,O= > subject: CN=,O= > expires: 2016-05-28 06:39:18 UTC > eku: id-kp-serverAuth,id-kp-clientAuth > pre-save command: > post-save command: /usr/lib64/ipa/certmonger/restart_dirsrv > > track: yes > auto-renew: yes > Request ID '20140528063953': > status: MONITORING > ca-error: Error setting up ccache for local "host" service > using default keytab: Cannot contact any KDC for realm ''. > stuck: no > key pair storage: > type=NSSDB,location='/etc/dirsrv/slapd-PKI-IPA',nickname='Server-Cert',token='NSS > Certificate DB',pinfile='/etc/dirsrv/slapd-PKI-IPA/pwdfile.txt' > certificate: > type=NSSDB,location='/etc/dirsrv/slapd-PKI-IPA',nickname='Server-Cert',token='NSS > Certificate DB' > CA: IPA > issuer: CN=Certificate Authority,O= > subject: CN=,O= > expires: 2016-05-28 06:39:52 UTC > eku: id-kp-serverAuth,id-kp-clientAuth > pre-save command: > post-save command: /usr/lib64/ipa/certmonger/restart_dirsrv > PKI-IPA > track: yes > auto-renew: yes > Request ID '20140528064145': > status: MONITORING > ca-error: Error setting up ccache for local "host" service > using default keytab: Cannot contact any KDC for realm ''. > stuck: no > key pair storage: > type=NSSDB,location='/etc/httpd/alias',nickname='Server-Cert',token='NSS > Certificate DB',pinfile='/etc/httpd/alias/pwdfile.txt' > certificate: > type=NSSDB,location='/etc/httpd/alias',nickname='Server-Cert',token='NSS > Certificate DB' > CA: IPA > issuer: CN=Certificate Authority,O= > subject: CN=,O= > expires: 2016-05-28 06:41:44 UTC > eku: id-kp-serverAuth,id-kp-clientAuth > pre-save command: > post-save command: /usr/lib64/ipa/certmonger/restart_httpd > track: yes > auto-renew: yes From bahanw042014 at gmail.com Wed Sep 14 16:45:07 2016 From: bahanw042014 at gmail.com (bahan w) Date: Wed, 14 Sep 2016 18:45:07 +0200 Subject: [Freeipa-users] SSL alert: CERT_VerifyCertificateNow: verify certificate failed for cert Server-Cert of family In-Reply-To: References: <7d17ebb6-eba6-b560-bcf9-e016546095c1@redhat.com> Message-ID: I set the date-time when the certificates were valid : ### # date -s '2016-05-27 10:00:00' Fri May 27 10:00:00 CEST 2016 # date Fri May 27 10:00:02 CEST 2016 ### Then I try to renew them : ### # getcert resubmit -i 20140528063919 Resubmitting "20140528063919" to "IPA". # getcert resubmit -i 20140528064145 Resubmitting "20140528064145" to "IPA". # getcert resubmit -i 20140528063953 Resubmitting "20140528063953" to "IPA". ### But when I do the getcert list after, the result is the same. I guess it is because of this ? CA_UNREACHABLE Any idea ? Best regards. Bahan On Wed, Sep 14, 2016 at 6:38 PM, bahan w wrote: > Ok, I managed to restart the IPA service by adding this line in the file > /etc/httpd/conf.d/nss.conf : > ### > NSSEnforceValidCerts off > ### > > But when I do the getcert now I got the following result : > > ### > # getcert list > Number of certificates and requests being tracked: 8. > Request ID '20140528063903': > status: MONITORING > stuck: no > key pair storage: type=NSSDB,location='/var/lib/ > pki-ca/alias',nickname='auditSigningCert cert-pki-ca',token='NSS > Certificate DB',pin='159203530658' > certificate: type=NSSDB,location='/var/lib/ > pki-ca/alias',nickname='auditSigningCert cert-pki-ca',token='NSS > Certificate DB' > CA: dogtag-ipa-renew-agent > issuer: CN=Certificate Authority,O= > subject: CN=CA Audit,O= > expires: 2018-04-09 11:39:16 UTC > pre-save command: /usr/lib64/ipa/certmonger/stop_pkicad > post-save command: /usr/lib64/ipa/certmonger/renew_ca_cert > "auditSigningCert cert-pki-ca" > track: yes > auto-renew: yes > Request ID '20140528063904': > status: MONITORING > stuck: no > key pair storage: type=NSSDB,location='/var/lib/ > pki-ca/alias',nickname='ocspSigningCert cert-pki-ca',token='NSS > Certificate DB',pin='159203530658' > certificate: type=NSSDB,location='/var/lib/ > pki-ca/alias',nickname='ocspSigningCert cert-pki-ca',token='NSS > Certificate DB' > CA: dogtag-ipa-renew-agent > issuer: CN=Certificate Authority,O= > subject: CN=OCSP Subsystem,O= > expires: 2018-04-09 11:38:16 UTC > eku: id-kp-OCSPSigning > pre-save command: /usr/lib64/ipa/certmonger/stop_pkicad > post-save command: /usr/lib64/ipa/certmonger/renew_ca_cert > "ocspSigningCert cert-pki-ca" > track: yes > auto-renew: yes > Request ID '20140528063905': > status: MONITORING > stuck: no > key pair storage: type=NSSDB,location='/var/lib/ > pki-ca/alias',nickname='subsystemCert cert-pki-ca',token='NSS Certificate > DB',pin='159203530658' > certificate: type=NSSDB,location='/var/lib/ > pki-ca/alias',nickname='subsystemCert cert-pki-ca',token='NSS Certificate > DB' > CA: dogtag-ipa-renew-agent > issuer: CN=Certificate Authority,O= > subject: CN=CA Subsystem,O= > expires: 2018-04-09 11:38:16 UTC > eku: id-kp-serverAuth,id-kp-clientAuth > pre-save command: /usr/lib64/ipa/certmonger/stop_pkicad > post-save command: /usr/lib64/ipa/certmonger/renew_ca_cert > "subsystemCert cert-pki-ca" > track: yes > auto-renew: yes > Request ID '20140528063906': > status: MONITORING > stuck: no > key pair storage: type=NSSDB,location='/etc/ > httpd/alias',nickname='ipaCert',token='NSS Certificate > DB',pinfile='/etc/httpd/alias/pwdfile.txt' > certificate: type=NSSDB,location='/etc/httpd/alias',nickname='ipaCert',token='NSS > Certificate DB' > CA: dogtag-ipa-renew-agent > issuer: CN=Certificate Authority,O= > subject: CN=IPA RA,O= > expires: 2018-04-09 11:38:16 UTC > eku: id-kp-serverAuth,id-kp-clientAuth > pre-save command: > post-save command: /usr/lib64/ipa/certmonger/renew_ra_cert > track: yes > auto-renew: yes > Request ID '20140528063907': > status: MONITORING > stuck: no > key pair storage: type=NSSDB,location='/var/lib/ > pki-ca/alias',nickname='Server-Cert cert-pki-ca',token='NSS Certificate > DB',pin='159203530658' > certificate: type=NSSDB,location='/var/lib/ > pki-ca/alias',nickname='Server-Cert cert-pki-ca',token='NSS Certificate > DB' > CA: dogtag-ipa-renew-agent > issuer: CN=Certificate Authority,O= > subject: CN=,O= > expires: 2018-04-09 11:38:16 UTC > eku: id-kp-serverAuth,id-kp-clientAuth > pre-save command: > post-save command: > track: yes > auto-renew: yes > Request ID '20140528063919': > status: CA_UNREACHABLE > ca-error: Server failed request, will retry: -504 (libcurl failed > to execute the HTTP POST transaction. Peer certificate cannot be > authenticated with known CA certificates). > stuck: yes > key pair storage: type=NSSDB,location='/etc/ > dirsrv/slapd-',nickname='Server-Cert',token='NSS Certificate > DB',pinfile='/etc/dirsrv/slapd-/pwdfile.txt' > certificate: type=NSSDB,location='/etc/dirsrv/slapd-', > nickname='Server-Cert',token='NSS Certificate DB' > CA: IPA > issuer: CN=Certificate Authority,O= > subject: CN=,O= > expires: 2016-05-28 06:39:18 UTC > eku: id-kp-serverAuth,id-kp-clientAuth > pre-save command: > post-save command: /usr/lib64/ipa/certmonger/restart_dirsrv > > track: yes > auto-renew: yes > Request ID '20140528063953': > status: CA_UNREACHABLE > ca-error: Server failed request, will retry: -504 (libcurl failed > to execute the HTTP POST transaction. Peer certificate cannot be > authenticated with known CA certificates). > stuck: yes > key pair storage: type=NSSDB,location='/etc/dirsrv/slapd-PKI-IPA', > nickname='Server-Cert',token='NSS Certificate DB',pinfile='/etc/dirsrv/ > slapd-PKI-IPA/pwdfile.txt' > certificate: type=NSSDB,location='/etc/dirsrv/slapd-PKI-IPA', > nickname='Server-Cert',token='NSS Certificate DB' > CA: IPA > issuer: CN=Certificate Authority,O= > subject: CN=,O= > expires: 2016-05-28 06:39:52 UTC > eku: id-kp-serverAuth,id-kp-clientAuth > pre-save command: > post-save command: /usr/lib64/ipa/certmonger/restart_dirsrv > PKI-IPA > track: yes > auto-renew: yes > Request ID '20140528064145': > status: CA_UNREACHABLE > ca-error: Server failed request, will retry: -504 (libcurl failed > to execute the HTTP POST transaction. Peer certificate cannot be > authenticated with known CA certificates). > stuck: yes > key pair storage: type=NSSDB,location='/etc/ > httpd/alias',nickname='Server-Cert',token='NSS Certificate > DB',pinfile='/etc/httpd/alias/pwdfile.txt' > certificate: type=NSSDB,location='/etc/ > httpd/alias',nickname='Server-Cert',token='NSS Certificate DB' > CA: IPA > issuer: CN=Certificate Authority,O= > subject: CN=,O= > expires: 2016-05-28 06:41:44 UTC > eku: id-kp-serverAuth,id-kp-clientAuth > pre-save command: > post-save command: /usr/lib64/ipa/certmonger/restart_httpd > track: yes > auto-renew: yes > ### > > Indeed, the entries outdated are the following : > - for /etc/dirsrv/slapd- : 20140528063919 > - for /etc/dirsrv/slapd-PKI-IPA : 20140528063953 > - for httpd ? : 20140528064145 > > Best regards. > > Bahan > > On Wed, Sep 14, 2016 at 6:28 PM, bahan w wrote: > >> Ok :D >> >> Because to perform the getcert list command, I need to have all the ipa >> services running right ? >> >> Here is the result of the command with the ipa services down. >> ### >> # getcert list >> Number of certificates and requests being tracked: 8. >> Request ID '20140528063903': >> status: MONITORING >> stuck: no >> key pair storage: type=NSSDB,location='/var/lib/ >> pki-ca/alias',nickname='auditSigningCert cert-pki-ca',token='NSS >> Certificate DB',pin='159203530658' >> certificate: type=NSSDB,location='/var/lib/ >> pki-ca/alias',nickname='auditSigningCert cert-pki-ca',token='NSS >> Certificate DB' >> CA: dogtag-ipa-renew-agent >> issuer: CN=Certificate Authority,O= >> subject: CN=CA Audit,O= >> expires: 2018-04-09 11:39:16 UTC >> pre-save command: /usr/lib64/ipa/certmonger/stop_pkicad >> post-save command: /usr/lib64/ipa/certmonger/renew_ca_cert >> "auditSigningCert cert-pki-ca" >> track: yes >> auto-renew: yes >> Request ID '20140528063904': >> status: MONITORING >> stuck: no >> key pair storage: type=NSSDB,location='/var/lib/ >> pki-ca/alias',nickname='ocspSigningCert cert-pki-ca',token='NSS >> Certificate DB',pin='159203530658' >> certificate: type=NSSDB,location='/var/lib/ >> pki-ca/alias',nickname='ocspSigningCert cert-pki-ca',token='NSS >> Certificate DB' >> CA: dogtag-ipa-renew-agent >> issuer: CN=Certificate Authority,O= >> subject: CN=OCSP Subsystem,O= >> expires: 2018-04-09 11:38:16 UTC >> eku: id-kp-OCSPSigning >> pre-save command: /usr/lib64/ipa/certmonger/stop_pkicad >> post-save command: /usr/lib64/ipa/certmonger/renew_ca_cert >> "ocspSigningCert cert-pki-ca" >> track: yes >> auto-renew: yes >> Request ID '20140528063905': >> status: MONITORING >> stuck: no >> key pair storage: type=NSSDB,location='/var/lib/ >> pki-ca/alias',nickname='subsystemCert cert-pki-ca',token='NSS >> Certificate DB',pin='159203530658' >> certificate: type=NSSDB,location='/var/lib/ >> pki-ca/alias',nickname='subsystemCert cert-pki-ca',token='NSS >> Certificate DB' >> CA: dogtag-ipa-renew-agent >> issuer: CN=Certificate Authority,O= >> subject: CN=CA Subsystem,O= >> expires: 2018-04-09 11:38:16 UTC >> eku: id-kp-serverAuth,id-kp-clientAuth >> pre-save command: /usr/lib64/ipa/certmonger/stop_pkicad >> post-save command: /usr/lib64/ipa/certmonger/renew_ca_cert >> "subsystemCert cert-pki-ca" >> track: yes >> auto-renew: yes >> Request ID '20140528063906': >> status: MONITORING >> stuck: no >> key pair storage: type=NSSDB,location='/etc/http >> d/alias',nickname='ipaCert',token='NSS Certificate >> DB',pinfile='/etc/httpd/alias/pwdfile.txt' >> certificate: type=NSSDB,location='/etc/http >> d/alias',nickname='ipaCert',token='NSS Certificate DB' >> CA: dogtag-ipa-renew-agent >> issuer: CN=Certificate Authority,O= >> subject: CN=IPA RA,O= >> expires: 2018-04-09 11:38:16 UTC >> eku: id-kp-serverAuth,id-kp-clientAuth >> pre-save command: >> post-save command: /usr/lib64/ipa/certmonger/renew_ra_cert >> track: yes >> auto-renew: yes >> Request ID '20140528063907': >> status: MONITORING >> stuck: no >> key pair storage: type=NSSDB,location='/var/lib/ >> pki-ca/alias',nickname='Server-Cert cert-pki-ca',token='NSS Certificate >> DB',pin='159203530658' >> certificate: type=NSSDB,location='/var/lib/ >> pki-ca/alias',nickname='Server-Cert cert-pki-ca',token='NSS Certificate >> DB' >> CA: dogtag-ipa-renew-agent >> issuer: CN=Certificate Authority,O= >> subject: CN=,O= >> expires: 2018-04-09 11:38:16 UTC >> eku: id-kp-serverAuth,id-kp-clientAuth >> pre-save command: >> post-save command: >> track: yes >> auto-renew: yes >> Request ID '20140528063919': >> status: MONITORING >> ca-error: Error setting up ccache for local "host" service using >> default keytab: Cannot contact any KDC for realm ''. >> stuck: no >> key pair storage: type=NSSDB,location='/etc/dirs >> rv/slapd-',nickname='Server-Cert',token='NSS Certificate >> DB',pinfile='/etc/dirsrv/slapd-/pwdfile.txt' >> certificate: type=NSSDB,location='/etc/dirs >> rv/slapd-',nickname='Server-Cert',token='NSS Certificate DB' >> CA: IPA >> issuer: CN=Certificate Authority,O= >> subject: CN=,O= >> expires: 2016-05-28 06:39:18 UTC >> eku: id-kp-serverAuth,id-kp-clientAuth >> pre-save command: >> post-save command: /usr/lib64/ipa/certmonger/restart_dirsrv >> >> track: yes >> auto-renew: yes >> Request ID '20140528063953': >> status: MONITORING >> ca-error: Error setting up ccache for local "host" service using >> default keytab: Cannot contact any KDC for realm ''. >> stuck: no >> key pair storage: type=NSSDB,location='/etc/dirs >> rv/slapd-PKI-IPA',nickname='Server-Cert',token='NSS Certificate >> DB',pinfile='/etc/dirsrv/slapd-PKI-IPA/pwdfile.txt' >> certificate: type=NSSDB,location='/etc/dirs >> rv/slapd-PKI-IPA',nickname='Server-Cert',token='NSS Certificate DB' >> CA: IPA >> issuer: CN=Certificate Authority,O= >> subject: CN=,O= >> expires: 2016-05-28 06:39:52 UTC >> eku: id-kp-serverAuth,id-kp-clientAuth >> pre-save command: >> post-save command: /usr/lib64/ipa/certmonger/restart_dirsrv >> PKI-IPA >> track: yes >> auto-renew: yes >> Request ID '20140528064145': >> status: MONITORING >> ca-error: Error setting up ccache for local "host" service using >> default keytab: Cannot contact any KDC for realm ''. >> stuck: no >> key pair storage: type=NSSDB,location='/etc/http >> d/alias',nickname='Server-Cert',token='NSS Certificate >> DB',pinfile='/etc/httpd/alias/pwdfile.txt' >> certificate: type=NSSDB,location='/etc/http >> d/alias',nickname='Server-Cert',token='NSS Certificate DB' >> CA: IPA >> issuer: CN=Certificate Authority,O= >> subject: CN=,O= >> expires: 2016-05-28 06:41:44 UTC >> eku: id-kp-serverAuth,id-kp-clientAuth >> pre-save command: >> post-save command: /usr/lib64/ipa/certmonger/restart_httpd >> track: yes >> auto-renew: yes >> ### >> >> Best regards. >> >> Bahan >> >> On Wed, Sep 14, 2016 at 6:21 PM, Martin Basti wrote: >> >>> >>> Then you have to start services manually, I don't know if the same steps >>> will work with IPA 3.0.0, I don't remember, but you can try :) >>> >>> On 14.09.2016 18:18, bahan w wrote: >>> >>> Oh I forgot to add that my version of ipa is quite old : >>> ### >>> # rpm -qa | grep ipa-server >>> ipa-server-3.0.0-25.el6.x86_64 >>> ### >>> >>> When I try the command you gave me I got the following error : >>> ### >>> # ipactl start --force >>> Usage: ipactl start|stop|restart|status >>> >>> >>> ipactl: error: no such option: --force >>> ### >>> >>> Best regards. >>> >>> Bahan >>> >>> >>> On Wed, Sep 14, 2016 at 6:14 PM, Martin Basti wrote: >>> >>>> >>>> >>>> On 14.09.2016 17:59, bahan w wrote: >>>> >>>> Hello ! >>>> >>>> I send you this mail because I cannot restart my test IPA server. >>>> >>>> When I try to start it with service ipa start, I got the following >>>> error message : >>>> ### >>>> # service ipa start >>>> Starting Directory Service >>>> Starting dirsrv: >>>> ...[14/Sep/2016:17:57:23 +0200] - SSL alert: >>>> CERT_VerifyCertificateNow: verify certificate failed for cert Server-Cert >>>> of family cn=RSA,cn=encryption,cn=config (Netscape Portable Runtime error >>>> -8181 - Peer's Certificate has expired.) >>>> [ OK ] >>>> PKI-IPA...[14/Sep/2016:17:57:33 +0200] - SSL alert: >>>> CERT_VerifyCertificateNow: verify certificate failed for cert Server-Cert >>>> of family cn=RSA,cn=encryption,cn=config (Netscape Portable Runtime error >>>> -8181 - Peer's Certificate has expired.) >>>> [ OK ] >>>> Starting KDC Service >>>> Starting Kerberos 5 KDC: [ OK ] >>>> Starting KPASSWD Service >>>> Starting Kerberos 5 Admin Server: [ OK ] >>>> Starting MEMCACHE Service >>>> Starting ipa_memcached: [ OK ] >>>> Starting HTTP Service >>>> Starting httpd: [FAILED] >>>> Failed to start HTTP Service >>>> Shutting down >>>> Stopping Kerberos 5 KDC: [ OK ] >>>> Stopping Kerberos 5 Admin Server: [ OK ] >>>> Stopping ipa_memcached: [ OK ] >>>> Stopping httpd: [FAILED] >>>> Stopping pki-ca: [ OK ] >>>> Shutting down dirsrv: >>>> ... [ OK ] >>>> PKI-IPA... [ OK ] >>>> Aborting ipactl >>>> >>>> # service ipa status >>>> Directory Service: STOPPED >>>> Failed to get list of services to probe status: >>>> Directory Server is stopped >>>> ### >>>> >>>> Do you know how to renew the SSL certificate used for the IPA Server ? >>>> >>>> Best regards. >>>> >>>> Bahan >>>> >>>> >>>> >>>> >>>> >>>> Hello, >>>> >>>> please run >>>> >>>> # ipactl start --force >>>> # getcert list (to detect which certificate is outdated, I suspect DS >>>> cert (or to get more info why it has not been renewed)) >>>> >>>> If getcert does work (I'm not sure if ti is able to work without >>>> httpd), you probable need to move time back to past where cert is valid, >>>> start IPA and try again. >>>> >>>> Please find ID outdated certificate and try resubmit it (CA and DS must >>>> be running) >>>> >>>> # getcert resubmit -i 20160914122036 (use you ID :) ) >>>> >>>> This should renew cert, check status with getcert list >>>> >>>> Move time back to future (if needed) >>>> >>>> Try to restart IPA >>>> >>>> Martin^2 >>>> >>> >>> >>> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From mbasti at redhat.com Wed Sep 14 16:45:13 2016 From: mbasti at redhat.com (Martin Basti) Date: Wed, 14 Sep 2016 18:45:13 +0200 Subject: [Freeipa-users] SSL alert: CERT_VerifyCertificateNow: verify certificate failed for cert Server-Cert of family In-Reply-To: References: <7d17ebb6-eba6-b560-bcf9-e016546095c1@redhat.com> Message-ID: <1b83e8e5-3702-f388-a864-cc4abb2ecb1d@redhat.com> Please keep freeipa-users in CC, I'm quite lost here ca-error: Server failed request, will retry: -504 (libcurl failed to execute the HTTP POST transaction. Peer certificate cannot be authenticated with known CA certificates). I'm not sure what this does mean, but if this is caused by invalid httpd certificate, solution might be to set time a week before 2016-05-28, restart IPA and try to renew certs again Martin^2 On 14.09.2016 18:38, bahan w wrote: > Ok, I managed to restart the IPA service by adding this line in the > file /etc/httpd/conf.d/nss.conf : > ### > NSSEnforceValidCerts off > ### > > But when I do the getcert now I got the following result : > ### > # getcert list > Number of certificates and requests being tracked: 8. > Request ID '20140528063903': > status: MONITORING > stuck: no > key pair storage: > type=NSSDB,location='/var/lib/pki-ca/alias',nickname='auditSigningCert > cert-pki-ca',token='NSS Certificate DB',pin='159203530658' > certificate: > type=NSSDB,location='/var/lib/pki-ca/alias',nickname='auditSigningCert > cert-pki-ca',token='NSS Certificate DB' > CA: dogtag-ipa-renew-agent > issuer: CN=Certificate Authority,O= > subject: CN=CA Audit,O= > expires: 2018-04-09 11:39:16 UTC > pre-save command: /usr/lib64/ipa/certmonger/stop_pkicad > post-save command: /usr/lib64/ipa/certmonger/renew_ca_cert > "auditSigningCert cert-pki-ca" > track: yes > auto-renew: yes > Request ID '20140528063904': > status: MONITORING > stuck: no > key pair storage: > type=NSSDB,location='/var/lib/pki-ca/alias',nickname='ocspSigningCert > cert-pki-ca',token='NSS Certificate DB',pin='159203530658' > certificate: > type=NSSDB,location='/var/lib/pki-ca/alias',nickname='ocspSigningCert > cert-pki-ca',token='NSS Certificate DB' > CA: dogtag-ipa-renew-agent > issuer: CN=Certificate Authority,O= > subject: CN=OCSP Subsystem,O= > expires: 2018-04-09 11:38:16 UTC > eku: id-kp-OCSPSigning > pre-save command: /usr/lib64/ipa/certmonger/stop_pkicad > post-save command: /usr/lib64/ipa/certmonger/renew_ca_cert > "ocspSigningCert cert-pki-ca" > track: yes > auto-renew: yes > Request ID '20140528063905': > status: MONITORING > stuck: no > key pair storage: > type=NSSDB,location='/var/lib/pki-ca/alias',nickname='subsystemCert > cert-pki-ca',token='NSS Certificate DB',pin='159203530658' > certificate: > type=NSSDB,location='/var/lib/pki-ca/alias',nickname='subsystemCert > cert-pki-ca',token='NSS Certificate DB' > CA: dogtag-ipa-renew-agent > issuer: CN=Certificate Authority,O= > subject: CN=CA Subsystem,O= > expires: 2018-04-09 11:38:16 UTC > eku: id-kp-serverAuth,id-kp-clientAuth > pre-save command: /usr/lib64/ipa/certmonger/stop_pkicad > post-save command: /usr/lib64/ipa/certmonger/renew_ca_cert > "subsystemCert cert-pki-ca" > track: yes > auto-renew: yes > Request ID '20140528063906': > status: MONITORING > stuck: no > key pair storage: > type=NSSDB,location='/etc/httpd/alias',nickname='ipaCert',token='NSS > Certificate DB',pinfile='/etc/httpd/alias/pwdfile.txt' > certificate: > type=NSSDB,location='/etc/httpd/alias',nickname='ipaCert',token='NSS > Certificate DB' > CA: dogtag-ipa-renew-agent > issuer: CN=Certificate Authority,O= > subject: CN=IPA RA,O= > expires: 2018-04-09 11:38:16 UTC > eku: id-kp-serverAuth,id-kp-clientAuth > pre-save command: > post-save command: /usr/lib64/ipa/certmonger/renew_ra_cert > track: yes > auto-renew: yes > Request ID '20140528063907': > status: MONITORING > stuck: no > key pair storage: > type=NSSDB,location='/var/lib/pki-ca/alias',nickname='Server-Cert > cert-pki-ca',token='NSS Certificate DB',pin='159203530658' > certificate: > type=NSSDB,location='/var/lib/pki-ca/alias',nickname='Server-Cert > cert-pki-ca',token='NSS Certificate DB' > CA: dogtag-ipa-renew-agent > issuer: CN=Certificate Authority,O= > subject: CN=,O= > expires: 2018-04-09 11:38:16 UTC > eku: id-kp-serverAuth,id-kp-clientAuth > pre-save command: > post-save command: > track: yes > auto-renew: yes > Request ID '20140528063919': > status: CA_UNREACHABLE > ca-error: Server failed request, will retry: -504 (libcurl > failed to execute the HTTP POST transaction. Peer certificate cannot > be authenticated with known CA certificates). > stuck: yes > key pair storage: > type=NSSDB,location='/etc/dirsrv/slapd-',nickname='Server-Cert',token='NSS > Certificate DB',pinfile='/etc/dirsrv/slapd-/pwdfile.txt' > certificate: > type=NSSDB,location='/etc/dirsrv/slapd-',nickname='Server-Cert',token='NSS > Certificate DB' > CA: IPA > issuer: CN=Certificate Authority,O= > subject: CN=,O= > expires: 2016-05-28 06:39:18 UTC > eku: id-kp-serverAuth,id-kp-clientAuth > pre-save command: > post-save command: /usr/lib64/ipa/certmonger/restart_dirsrv > > track: yes > auto-renew: yes > Request ID '20140528063953': > status: CA_UNREACHABLE > ca-error: Server failed request, will retry: -504 (libcurl > failed to execute the HTTP POST transaction. Peer certificate cannot > be authenticated with known CA certificates). > stuck: yes > key pair storage: > type=NSSDB,location='/etc/dirsrv/slapd-PKI-IPA',nickname='Server-Cert',token='NSS > Certificate DB',pinfile='/etc/dirsrv/slapd-PKI-IPA/pwdfile.txt' > certificate: > type=NSSDB,location='/etc/dirsrv/slapd-PKI-IPA',nickname='Server-Cert',token='NSS > Certificate DB' > CA: IPA > issuer: CN=Certificate Authority,O= > subject: CN=,O= > expires: 2016-05-28 06:39:52 UTC > eku: id-kp-serverAuth,id-kp-clientAuth > pre-save command: > post-save command: /usr/lib64/ipa/certmonger/restart_dirsrv > PKI-IPA > track: yes > auto-renew: yes > Request ID '20140528064145': > status: CA_UNREACHABLE > ca-error: Server failed request, will retry: -504 (libcurl > failed to execute the HTTP POST transaction. Peer certificate cannot > be authenticated with known CA certificates). > stuck: yes > key pair storage: > type=NSSDB,location='/etc/httpd/alias',nickname='Server-Cert',token='NSS > Certificate DB',pinfile='/etc/httpd/alias/pwdfile.txt' > certificate: > type=NSSDB,location='/etc/httpd/alias',nickname='Server-Cert',token='NSS > Certificate DB' > CA: IPA > issuer: CN=Certificate Authority,O= > subject: CN=,O= > expires: 2016-05-28 06:41:44 UTC > eku: id-kp-serverAuth,id-kp-clientAuth > pre-save command: > post-save command: /usr/lib64/ipa/certmonger/restart_httpd > track: yes > auto-renew: yes > ### > > Indeed, the entries outdated are the following : > - for /etc/dirsrv/slapd- : 20140528063919 > - for /etc/dirsrv/slapd-PKI-IPA : 20140528063953 > - for httpd ? : 20140528064145 > > Best regards. > > Bahan > > On Wed, Sep 14, 2016 at 6:28 PM, bahan w > wrote: > > Ok :D > > Because to perform the getcert list command, I need to have all > the ipa services running right ? > > Here is the result of the command with the ipa services down. > ### > # getcert list > Number of certificates and requests being tracked: 8. > Request ID '20140528063903': > status: MONITORING > stuck: no > key pair storage: > type=NSSDB,location='/var/lib/pki-ca/alias',nickname='auditSigningCert > cert-pki-ca',token='NSS Certificate DB',pin='159203530658' > certificate: > type=NSSDB,location='/var/lib/pki-ca/alias',nickname='auditSigningCert > cert-pki-ca',token='NSS Certificate DB' > CA: dogtag-ipa-renew-agent > issuer: CN=Certificate Authority,O= > subject: CN=CA Audit,O= > expires: 2018-04-09 11:39:16 UTC > pre-save command: /usr/lib64/ipa/certmonger/stop_pkicad > post-save command: /usr/lib64/ipa/certmonger/renew_ca_cert > "auditSigningCert cert-pki-ca" > track: yes > auto-renew: yes > Request ID '20140528063904': > status: MONITORING > stuck: no > key pair storage: > type=NSSDB,location='/var/lib/pki-ca/alias',nickname='ocspSigningCert > cert-pki-ca',token='NSS Certificate DB',pin='159203530658' > certificate: > type=NSSDB,location='/var/lib/pki-ca/alias',nickname='ocspSigningCert > cert-pki-ca',token='NSS Certificate DB' > CA: dogtag-ipa-renew-agent > issuer: CN=Certificate Authority,O= > subject: CN=OCSP Subsystem,O= > expires: 2018-04-09 11:38:16 UTC > eku: id-kp-OCSPSigning > pre-save command: /usr/lib64/ipa/certmonger/stop_pkicad > post-save command: /usr/lib64/ipa/certmonger/renew_ca_cert > "ocspSigningCert cert-pki-ca" > track: yes > auto-renew: yes > Request ID '20140528063905': > status: MONITORING > stuck: no > key pair storage: > type=NSSDB,location='/var/lib/pki-ca/alias',nickname='subsystemCert > cert-pki-ca',token='NSS Certificate DB',pin='159203530658' > certificate: > type=NSSDB,location='/var/lib/pki-ca/alias',nickname='subsystemCert > cert-pki-ca',token='NSS Certificate DB' > CA: dogtag-ipa-renew-agent > issuer: CN=Certificate Authority,O= > subject: CN=CA Subsystem,O= > expires: 2018-04-09 11:38:16 UTC > eku: id-kp-serverAuth,id-kp-clientAuth > pre-save command: /usr/lib64/ipa/certmonger/stop_pkicad > post-save command: /usr/lib64/ipa/certmonger/renew_ca_cert > "subsystemCert cert-pki-ca" > track: yes > auto-renew: yes > Request ID '20140528063906': > status: MONITORING > stuck: no > key pair storage: > type=NSSDB,location='/etc/httpd/alias',nickname='ipaCert',token='NSS > Certificate DB',pinfile='/etc/httpd/alias/pwdfile.txt' > certificate: > type=NSSDB,location='/etc/httpd/alias',nickname='ipaCert',token='NSS > Certificate DB' > CA: dogtag-ipa-renew-agent > issuer: CN=Certificate Authority,O= > subject: CN=IPA RA,O= > expires: 2018-04-09 11:38:16 UTC > eku: id-kp-serverAuth,id-kp-clientAuth > pre-save command: > post-save command: /usr/lib64/ipa/certmonger/renew_ra_cert > track: yes > auto-renew: yes > Request ID '20140528063907': > status: MONITORING > stuck: no > key pair storage: > type=NSSDB,location='/var/lib/pki-ca/alias',nickname='Server-Cert > cert-pki-ca',token='NSS Certificate DB',pin='159203530658' > certificate: > type=NSSDB,location='/var/lib/pki-ca/alias',nickname='Server-Cert > cert-pki-ca',token='NSS Certificate DB' > CA: dogtag-ipa-renew-agent > issuer: CN=Certificate Authority,O= > subject: CN=,O= > expires: 2018-04-09 11:38:16 UTC > eku: id-kp-serverAuth,id-kp-clientAuth > pre-save command: > post-save command: > track: yes > auto-renew: yes > Request ID '20140528063919': > status: MONITORING > ca-error: Error setting up ccache for local "host" service > using default keytab: Cannot contact any KDC for realm ''. > stuck: no > key pair storage: > type=NSSDB,location='/etc/dirsrv/slapd-',nickname='Server-Cert',token='NSS > Certificate DB',pinfile='/etc/dirsrv/slapd-/pwdfile.txt' > certificate: > type=NSSDB,location='/etc/dirsrv/slapd-',nickname='Server-Cert',token='NSS > Certificate DB' > CA: IPA > issuer: CN=Certificate Authority,O= > subject: CN=,O= > expires: 2016-05-28 06:39:18 UTC > eku: id-kp-serverAuth,id-kp-clientAuth > pre-save command: > post-save command: > /usr/lib64/ipa/certmonger/restart_dirsrv > track: yes > auto-renew: yes > Request ID '20140528063953': > status: MONITORING > ca-error: Error setting up ccache for local "host" service > using default keytab: Cannot contact any KDC for realm ''. > stuck: no > key pair storage: > type=NSSDB,location='/etc/dirsrv/slapd-PKI-IPA',nickname='Server-Cert',token='NSS > Certificate DB',pinfile='/etc/dirsrv/slapd-PKI-IPA/pwdfile.txt' > certificate: > type=NSSDB,location='/etc/dirsrv/slapd-PKI-IPA',nickname='Server-Cert',token='NSS > Certificate DB' > CA: IPA > issuer: CN=Certificate Authority,O= > subject: CN=,O= > expires: 2016-05-28 06:39:52 UTC > eku: id-kp-serverAuth,id-kp-clientAuth > pre-save command: > post-save command: > /usr/lib64/ipa/certmonger/restart_dirsrv PKI-IPA > track: yes > auto-renew: yes > Request ID '20140528064145': > status: MONITORING > ca-error: Error setting up ccache for local "host" service > using default keytab: Cannot contact any KDC for realm ''. > stuck: no > key pair storage: > type=NSSDB,location='/etc/httpd/alias',nickname='Server-Cert',token='NSS > Certificate DB',pinfile='/etc/httpd/alias/pwdfile.txt' > certificate: > type=NSSDB,location='/etc/httpd/alias',nickname='Server-Cert',token='NSS > Certificate DB' > CA: IPA > issuer: CN=Certificate Authority,O= > subject: CN=,O= > expires: 2016-05-28 06:41:44 UTC > eku: id-kp-serverAuth,id-kp-clientAuth > pre-save command: > post-save command: /usr/lib64/ipa/certmonger/restart_httpd > track: yes > auto-renew: yes > ### > > Best regards. > > Bahan > > On Wed, Sep 14, 2016 at 6:21 PM, Martin Basti > wrote: > > > Then you have to start services manually, I don't know if the > same steps will work with IPA 3.0.0, I don't remember, but you > can try :) > > > On 14.09.2016 18:18, bahan w wrote: >> Oh I forgot to add that my version of ipa is quite old : >> ### >> # rpm -qa | grep ipa-server >> ipa-server-3.0.0-25.el6.x86_64 >> ### >> >> When I try the command you gave me I got the following error : >> ### >> # ipactl start --force >> Usage: ipactl start|stop|restart|status >> >> >> ipactl: error: no such option: --force >> ### >> >> Best regards. >> >> Bahan >> >> On Wed, Sep 14, 2016 at 6:14 PM, Martin Basti >> > wrote: >> >> >> >> On 14.09.2016 17:59, bahan w wrote: >>> Hello ! >>> >>> I send you this mail because I cannot restart my test >>> IPA server. >>> >>> When I try to start it with service ipa start, I got the >>> following error message : >>> ### >>> # service ipa start >>> Starting Directory Service >>> Starting dirsrv: >>> ...[14/Sep/2016:17:57:23 +0200] - SSL alert: >>> CERT_VerifyCertificateNow: verify certificate failed for >>> cert Server-Cert of family >>> cn=RSA,cn=encryption,cn=config (Netscape Portable >>> Runtime error -8181 - Peer's Certificate has expired.) >>> [ OK ] >>> PKI-IPA...[14/Sep/2016:17:57:33 +0200] - SSL alert: >>> CERT_VerifyCertificateNow: verify certificate failed for >>> cert Server-Cert of family >>> cn=RSA,cn=encryption,cn=config (Netscape Portable >>> Runtime error -8181 - Peer's Certificate has expired.) >>> [ OK ] >>> Starting KDC Service >>> Starting Kerberos 5 KDC: [ OK ] >>> Starting KPASSWD Service >>> Starting Kerberos 5 Admin Server: [ OK ] >>> Starting MEMCACHE Service >>> Starting ipa_memcached: [ OK ] >>> Starting HTTP Service >>> Starting httpd: [FAILED] >>> Failed to start HTTP Service >>> Shutting down >>> Stopping Kerberos 5 KDC: [ OK ] >>> Stopping Kerberos 5 Admin Server: [ OK ] >>> Stopping ipa_memcached: [ OK ] >>> Stopping httpd: [FAILED] >>> Stopping pki-ca: [ OK ] >>> Shutting down dirsrv: >>> ... [ OK ] >>> PKI-IPA... [ OK ] >>> Aborting ipactl >>> >>> # service ipa status >>> Directory Service: STOPPED >>> Failed to get list of services to probe status: >>> Directory Server is stopped >>> ### >>> >>> Do you know how to renew the SSL certificate used for >>> the IPA Server ? >>> >>> Best regards. >>> >>> Bahan >>> >>> >>> >> >> >> Hello, >> >> please run >> >> # ipactl start --force >> # getcert list (to detect which certificate is outdated, >> I suspect DS cert (or to get more info why it has not >> been renewed)) >> >> If getcert does work (I'm not sure if ti is able to work >> without httpd), you probable need to move time back to >> past where cert is valid, start IPA and try again. >> >> Please find ID outdated certificate and try resubmit it >> (CA and DS must be running) >> >> # getcert resubmit -i 20160914122036 (use you ID :) ) >> >> This should renew cert, check status with getcert list >> >> Move time back to future (if needed) >> >> Try to restart IPA >> >> Martin^2 >> >> > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From mbasti at redhat.com Wed Sep 14 16:46:39 2016 From: mbasti at redhat.com (Martin Basti) Date: Wed, 14 Sep 2016 18:46:39 +0200 Subject: [Freeipa-users] SSL alert: CERT_VerifyCertificateNow: verify certificate failed for cert Server-Cert of family In-Reply-To: References: <7d17ebb6-eba6-b560-bcf9-e016546095c1@redhat.com> Message-ID: <4a3d91a5-e48c-96cc-b1aa-6082eea2e302@redhat.com> did you restart IPA when you moved time? Is there are more detailed error description in output of getcert list? On 14.09.2016 18:45, bahan w wrote: > I set the date-time when the certificates were valid : > ### > # date -s '2016-05-27 10:00:00' > Fri May 27 10:00:00 CEST 2016 > > # date > Fri May 27 10:00:02 CEST 2016 > ### > > Then I try to renew them : > ### > # getcert resubmit -i 20140528063919 > Resubmitting "20140528063919" to "IPA". > > # getcert resubmit -i 20140528064145 > Resubmitting "20140528064145" to "IPA". > > # getcert resubmit -i 20140528063953 > Resubmitting "20140528063953" to "IPA". > ### > > But when I do the getcert list after, the result is the same. > > I guess it is because of this ? > CA_UNREACHABLE > > Any idea ? > > Best regards. > > Bahan > > On Wed, Sep 14, 2016 at 6:38 PM, bahan w > wrote: > > Ok, I managed to restart the IPA service by adding this line in > the file /etc/httpd/conf.d/nss.conf : > ### > NSSEnforceValidCerts off > ### > > But when I do the getcert now I got the following result : > > ### > # getcert list > Number of certificates and requests being tracked: 8. > Request ID '20140528063903': > status: MONITORING > stuck: no > key pair storage: > type=NSSDB,location='/var/lib/pki-ca/alias',nickname='auditSigningCert > cert-pki-ca',token='NSS Certificate DB',pin='159203530658' > certificate: > type=NSSDB,location='/var/lib/pki-ca/alias',nickname='auditSigningCert > cert-pki-ca',token='NSS Certificate DB' > CA: dogtag-ipa-renew-agent > issuer: CN=Certificate Authority,O= > subject: CN=CA Audit,O= > expires: 2018-04-09 11:39:16 UTC > pre-save command: /usr/lib64/ipa/certmonger/stop_pkicad > post-save command: /usr/lib64/ipa/certmonger/renew_ca_cert > "auditSigningCert cert-pki-ca" > track: yes > auto-renew: yes > Request ID '20140528063904': > status: MONITORING > stuck: no > key pair storage: > type=NSSDB,location='/var/lib/pki-ca/alias',nickname='ocspSigningCert > cert-pki-ca',token='NSS Certificate DB',pin='159203530658' > certificate: > type=NSSDB,location='/var/lib/pki-ca/alias',nickname='ocspSigningCert > cert-pki-ca',token='NSS Certificate DB' > CA: dogtag-ipa-renew-agent > issuer: CN=Certificate Authority,O= > subject: CN=OCSP Subsystem,O= > expires: 2018-04-09 11:38:16 UTC > eku: id-kp-OCSPSigning > pre-save command: /usr/lib64/ipa/certmonger/stop_pkicad > post-save command: /usr/lib64/ipa/certmonger/renew_ca_cert > "ocspSigningCert cert-pki-ca" > track: yes > auto-renew: yes > Request ID '20140528063905': > status: MONITORING > stuck: no > key pair storage: > type=NSSDB,location='/var/lib/pki-ca/alias',nickname='subsystemCert > cert-pki-ca',token='NSS Certificate DB',pin='159203530658' > certificate: > type=NSSDB,location='/var/lib/pki-ca/alias',nickname='subsystemCert > cert-pki-ca',token='NSS Certificate DB' > CA: dogtag-ipa-renew-agent > issuer: CN=Certificate Authority,O= > subject: CN=CA Subsystem,O= > expires: 2018-04-09 11:38:16 UTC > eku: id-kp-serverAuth,id-kp-clientAuth > pre-save command: /usr/lib64/ipa/certmonger/stop_pkicad > post-save command: /usr/lib64/ipa/certmonger/renew_ca_cert > "subsystemCert cert-pki-ca" > track: yes > auto-renew: yes > Request ID '20140528063906': > status: MONITORING > stuck: no > key pair storage: > type=NSSDB,location='/etc/httpd/alias',nickname='ipaCert',token='NSS > Certificate DB',pinfile='/etc/httpd/alias/pwdfile.txt' > certificate: > type=NSSDB,location='/etc/httpd/alias',nickname='ipaCert',token='NSS > Certificate DB' > CA: dogtag-ipa-renew-agent > issuer: CN=Certificate Authority,O= > subject: CN=IPA RA,O= > expires: 2018-04-09 11:38:16 UTC > eku: id-kp-serverAuth,id-kp-clientAuth > pre-save command: > post-save command: /usr/lib64/ipa/certmonger/renew_ra_cert > track: yes > auto-renew: yes > Request ID '20140528063907': > status: MONITORING > stuck: no > key pair storage: > type=NSSDB,location='/var/lib/pki-ca/alias',nickname='Server-Cert > cert-pki-ca',token='NSS Certificate DB',pin='159203530658' > certificate: > type=NSSDB,location='/var/lib/pki-ca/alias',nickname='Server-Cert > cert-pki-ca',token='NSS Certificate DB' > CA: dogtag-ipa-renew-agent > issuer: CN=Certificate Authority,O= > subject: CN=,O= > expires: 2018-04-09 11:38:16 UTC > eku: id-kp-serverAuth,id-kp-clientAuth > pre-save command: > post-save command: > track: yes > auto-renew: yes > Request ID '20140528063919': > status: CA_UNREACHABLE > ca-error: Server failed request, will retry: -504 (libcurl > failed to execute the HTTP POST transaction. Peer certificate > cannot be authenticated with known CA certificates). > stuck: yes > key pair storage: > type=NSSDB,location='/etc/dirsrv/slapd-',nickname='Server-Cert',token='NSS > Certificate DB',pinfile='/etc/dirsrv/slapd-/pwdfile.txt' > certificate: > type=NSSDB,location='/etc/dirsrv/slapd-',nickname='Server-Cert',token='NSS > Certificate DB' > CA: IPA > issuer: CN=Certificate Authority,O= > subject: CN=,O= > expires: 2016-05-28 06:39:18 UTC > eku: id-kp-serverAuth,id-kp-clientAuth > pre-save command: > post-save command: > /usr/lib64/ipa/certmonger/restart_dirsrv > track: yes > auto-renew: yes > Request ID '20140528063953': > status: CA_UNREACHABLE > ca-error: Server failed request, will retry: -504 (libcurl > failed to execute the HTTP POST transaction. Peer certificate > cannot be authenticated with known CA certificates). > stuck: yes > key pair storage: > type=NSSDB,location='/etc/dirsrv/slapd-PKI-IPA',nickname='Server-Cert',token='NSS > Certificate DB',pinfile='/etc/dirsrv/slapd-PKI-IPA/pwdfile.txt' > certificate: > type=NSSDB,location='/etc/dirsrv/slapd-PKI-IPA',nickname='Server-Cert',token='NSS > Certificate DB' > CA: IPA > issuer: CN=Certificate Authority,O= > subject: CN=,O= > expires: 2016-05-28 06:39:52 UTC > eku: id-kp-serverAuth,id-kp-clientAuth > pre-save command: > post-save command: > /usr/lib64/ipa/certmonger/restart_dirsrv PKI-IPA > track: yes > auto-renew: yes > Request ID '20140528064145': > status: CA_UNREACHABLE > ca-error: Server failed request, will retry: -504 (libcurl > failed to execute the HTTP POST transaction. Peer certificate > cannot be authenticated with known CA certificates). > stuck: yes > key pair storage: > type=NSSDB,location='/etc/httpd/alias',nickname='Server-Cert',token='NSS > Certificate DB',pinfile='/etc/httpd/alias/pwdfile.txt' > certificate: > type=NSSDB,location='/etc/httpd/alias',nickname='Server-Cert',token='NSS > Certificate DB' > CA: IPA > issuer: CN=Certificate Authority,O= > subject: CN=,O= > expires: 2016-05-28 06:41:44 UTC > eku: id-kp-serverAuth,id-kp-clientAuth > pre-save command: > post-save command: /usr/lib64/ipa/certmonger/restart_httpd > track: yes > auto-renew: yes > ### > > Indeed, the entries outdated are the following : > - for /etc/dirsrv/slapd- : 20140528063919 > - for /etc/dirsrv/slapd-PKI-IPA : 20140528063953 > - for httpd ? : 20140528064145 > > Best regards. > > Bahan > > On Wed, Sep 14, 2016 at 6:28 PM, bahan w > wrote: > > Ok :D > > Because to perform the getcert list command, I need to have > all the ipa services running right ? > > Here is the result of the command with the ipa services down. > ### > # getcert list > Number of certificates and requests being tracked: 8. > Request ID '20140528063903': > status: MONITORING > stuck: no > key pair storage: > type=NSSDB,location='/var/lib/pki-ca/alias',nickname='auditSigningCert > cert-pki-ca',token='NSS Certificate DB',pin='159203530658' > certificate: > type=NSSDB,location='/var/lib/pki-ca/alias',nickname='auditSigningCert > cert-pki-ca',token='NSS Certificate DB' > CA: dogtag-ipa-renew-agent > issuer: CN=Certificate Authority,O= > subject: CN=CA Audit,O= > expires: 2018-04-09 11:39:16 UTC > pre-save command: /usr/lib64/ipa/certmonger/stop_pkicad > post-save command: > /usr/lib64/ipa/certmonger/renew_ca_cert "auditSigningCert > cert-pki-ca" > track: yes > auto-renew: yes > Request ID '20140528063904': > status: MONITORING > stuck: no > key pair storage: > type=NSSDB,location='/var/lib/pki-ca/alias',nickname='ocspSigningCert > cert-pki-ca',token='NSS Certificate DB',pin='159203530658' > certificate: > type=NSSDB,location='/var/lib/pki-ca/alias',nickname='ocspSigningCert > cert-pki-ca',token='NSS Certificate DB' > CA: dogtag-ipa-renew-agent > issuer: CN=Certificate Authority,O= > subject: CN=OCSP Subsystem,O= > expires: 2018-04-09 11:38:16 UTC > eku: id-kp-OCSPSigning > pre-save command: /usr/lib64/ipa/certmonger/stop_pkicad > post-save command: > /usr/lib64/ipa/certmonger/renew_ca_cert "ocspSigningCert > cert-pki-ca" > track: yes > auto-renew: yes > Request ID '20140528063905': > status: MONITORING > stuck: no > key pair storage: > type=NSSDB,location='/var/lib/pki-ca/alias',nickname='subsystemCert > cert-pki-ca',token='NSS Certificate DB',pin='159203530658' > certificate: > type=NSSDB,location='/var/lib/pki-ca/alias',nickname='subsystemCert > cert-pki-ca',token='NSS Certificate DB' > CA: dogtag-ipa-renew-agent > issuer: CN=Certificate Authority,O= > subject: CN=CA Subsystem,O= > expires: 2018-04-09 11:38:16 UTC > eku: id-kp-serverAuth,id-kp-clientAuth > pre-save command: /usr/lib64/ipa/certmonger/stop_pkicad > post-save command: > /usr/lib64/ipa/certmonger/renew_ca_cert "subsystemCert > cert-pki-ca" > track: yes > auto-renew: yes > Request ID '20140528063906': > status: MONITORING > stuck: no > key pair storage: > type=NSSDB,location='/etc/httpd/alias',nickname='ipaCert',token='NSS > Certificate DB',pinfile='/etc/httpd/alias/pwdfile.txt' > certificate: > type=NSSDB,location='/etc/httpd/alias',nickname='ipaCert',token='NSS > Certificate DB' > CA: dogtag-ipa-renew-agent > issuer: CN=Certificate Authority,O= > subject: CN=IPA RA,O= > expires: 2018-04-09 11:38:16 UTC > eku: id-kp-serverAuth,id-kp-clientAuth > pre-save command: > post-save command: /usr/lib64/ipa/certmonger/renew_ra_cert > track: yes > auto-renew: yes > Request ID '20140528063907': > status: MONITORING > stuck: no > key pair storage: > type=NSSDB,location='/var/lib/pki-ca/alias',nickname='Server-Cert > cert-pki-ca',token='NSS Certificate DB',pin='159203530658' > certificate: > type=NSSDB,location='/var/lib/pki-ca/alias',nickname='Server-Cert > cert-pki-ca',token='NSS Certificate DB' > CA: dogtag-ipa-renew-agent > issuer: CN=Certificate Authority,O= > subject: CN=,O= > expires: 2018-04-09 11:38:16 UTC > eku: id-kp-serverAuth,id-kp-clientAuth > pre-save command: > post-save command: > track: yes > auto-renew: yes > Request ID '20140528063919': > status: MONITORING > ca-error: Error setting up ccache for local "host" > service using default keytab: Cannot contact any KDC for realm > ''. > stuck: no > key pair storage: > type=NSSDB,location='/etc/dirsrv/slapd-',nickname='Server-Cert',token='NSS > Certificate DB',pinfile='/etc/dirsrv/slapd-/pwdfile.txt' > certificate: > type=NSSDB,location='/etc/dirsrv/slapd-',nickname='Server-Cert',token='NSS > Certificate DB' > CA: IPA > issuer: CN=Certificate Authority,O= > subject: CN=,O= > expires: 2016-05-28 06:39:18 UTC > eku: id-kp-serverAuth,id-kp-clientAuth > pre-save command: > post-save command: > /usr/lib64/ipa/certmonger/restart_dirsrv > track: yes > auto-renew: yes > Request ID '20140528063953': > status: MONITORING > ca-error: Error setting up ccache for local "host" > service using default keytab: Cannot contact any KDC for realm > ''. > stuck: no > key pair storage: > type=NSSDB,location='/etc/dirsrv/slapd-PKI-IPA',nickname='Server-Cert',token='NSS > Certificate DB',pinfile='/etc/dirsrv/slapd-PKI-IPA/pwdfile.txt' > certificate: > type=NSSDB,location='/etc/dirsrv/slapd-PKI-IPA',nickname='Server-Cert',token='NSS > Certificate DB' > CA: IPA > issuer: CN=Certificate Authority,O= > subject: CN=,O= > expires: 2016-05-28 06:39:52 UTC > eku: id-kp-serverAuth,id-kp-clientAuth > pre-save command: > post-save command: > /usr/lib64/ipa/certmonger/restart_dirsrv PKI-IPA > track: yes > auto-renew: yes > Request ID '20140528064145': > status: MONITORING > ca-error: Error setting up ccache for local "host" > service using default keytab: Cannot contact any KDC for realm > ''. > stuck: no > key pair storage: > type=NSSDB,location='/etc/httpd/alias',nickname='Server-Cert',token='NSS > Certificate DB',pinfile='/etc/httpd/alias/pwdfile.txt' > certificate: > type=NSSDB,location='/etc/httpd/alias',nickname='Server-Cert',token='NSS > Certificate DB' > CA: IPA > issuer: CN=Certificate Authority,O= > subject: CN=,O= > expires: 2016-05-28 06:41:44 UTC > eku: id-kp-serverAuth,id-kp-clientAuth > pre-save command: > post-save command: /usr/lib64/ipa/certmonger/restart_httpd > track: yes > auto-renew: yes > ### > > Best regards. > > Bahan > > On Wed, Sep 14, 2016 at 6:21 PM, Martin Basti > > wrote: > > > Then you have to start services manually, I don't know if > the same steps will work with IPA 3.0.0, I don't remember, > but you can try :) > > > On 14.09.2016 18:18, bahan w wrote: >> Oh I forgot to add that my version of ipa is quite old : >> ### >> # rpm -qa | grep ipa-server >> ipa-server-3.0.0-25.el6.x86_64 >> ### >> >> When I try the command you gave me I got the following >> error : >> ### >> # ipactl start --force >> Usage: ipactl start|stop|restart|status >> >> >> ipactl: error: no such option: --force >> ### >> >> Best regards. >> >> Bahan >> >> On Wed, Sep 14, 2016 at 6:14 PM, Martin Basti >> > wrote: >> >> >> >> On 14.09.2016 17:59, bahan w wrote: >>> Hello ! >>> >>> I send you this mail because I cannot restart my >>> test IPA server. >>> >>> When I try to start it with service ipa start, I got >>> the following error message : >>> ### >>> # service ipa start >>> Starting Directory Service >>> Starting dirsrv: >>> ...[14/Sep/2016:17:57:23 +0200] - SSL >>> alert: CERT_VerifyCertificateNow: verify certificate >>> failed for cert Server-Cert of family >>> cn=RSA,cn=encryption,cn=config (Netscape Portable >>> Runtime error -8181 - Peer's Certificate has expired.) >>> [ OK ] >>> PKI-IPA...[14/Sep/2016:17:57:33 +0200] - SSL alert: >>> CERT_VerifyCertificateNow: verify certificate failed >>> for cert Server-Cert of family >>> cn=RSA,cn=encryption,cn=config (Netscape Portable >>> Runtime error -8181 - Peer's Certificate has expired.) >>> [ OK ] >>> Starting KDC Service >>> Starting Kerberos 5 KDC: [ OK ] >>> Starting KPASSWD Service >>> Starting Kerberos 5 Admin Server: [ OK ] >>> Starting MEMCACHE Service >>> Starting ipa_memcached: [ OK ] >>> Starting HTTP Service >>> Starting httpd: [FAILED] >>> Failed to start HTTP Service >>> Shutting down >>> Stopping Kerberos 5 KDC: [ OK ] >>> Stopping Kerberos 5 Admin Server: [ OK ] >>> Stopping ipa_memcached: [ OK ] >>> Stopping httpd: [FAILED] >>> Stopping pki-ca: [ OK ] >>> Shutting down dirsrv: >>> ... [ OK ] >>> PKI-IPA... [ OK ] >>> Aborting ipactl >>> >>> # service ipa status >>> Directory Service: STOPPED >>> Failed to get list of services to probe status: >>> Directory Server is stopped >>> ### >>> >>> Do you know how to renew the SSL certificate used >>> for the IPA Server ? >>> >>> Best regards. >>> >>> Bahan >>> >>> >>> >> >> >> Hello, >> >> please run >> >> # ipactl start --force >> # getcert list (to detect which certificate is >> outdated, I suspect DS cert (or to get more info why >> it has not been renewed)) >> >> If getcert does work (I'm not sure if ti is able to >> work without httpd), you probable need to move time >> back to past where cert is valid, start IPA and try >> again. >> >> Please find ID outdated certificate and try resubmit >> it (CA and DS must be running) >> >> # getcert resubmit -i 20160914122036 (use you ID :) ) >> >> This should renew cert, check status with getcert list >> >> Move time back to future (if needed) >> >> Try to restart IPA >> >> Martin^2 >> >> > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jeffrey.armstrong at gasoc.com Wed Sep 14 16:56:08 2016 From: jeffrey.armstrong at gasoc.com (Armstrong, Jeffrey) Date: Wed, 14 Sep 2016 16:56:08 +0000 Subject: [Freeipa-users] [E] Migration Question In-Reply-To: References: Message-ID: <3DAC7A5927B8594195EA704FB41255B076BE65CF@Supernatural2.gafoc.com> Ok. Thank you very much for the information. Jeff From: Giger, Justean [mailto:jgiger at verizon.com] Sent: Wednesday, September 14, 2016 11:18 AM To: Armstrong, Jeffrey ; freeipa-users at redhat.com Subject: Re: [E] [Freeipa-users] Migration Question *External E-Mail* We did the same and have had zero issues. In fact, one overzealous colleague moved one out of our 5 IDM servers to Oracle while all the others were still on Red Hat and things still worked. I have not tried to get support for IDM with Oracle though so not sure how that goes. From: > on behalf of "Armstrong, Jeffrey" > Date: Wednesday, September 14, 2016 at 6:20 AM To: "freeipa-users at redhat.com" > Subject: [E] [Freeipa-users] Migration Question Hi My company is migrating from RedHat Linux to Oracle Linux. I warned them that IdM could be a problem. Does anyone know If IPA works after the migration? Jeff Armstrong -------------- next part -------------- An HTML attachment was scrubbed... URL: From bahanw042014 at gmail.com Wed Sep 14 17:02:22 2016 From: bahanw042014 at gmail.com (bahan w) Date: Wed, 14 Sep 2016 19:02:22 +0200 Subject: [Freeipa-users] SSL alert: CERT_VerifyCertificateNow: verify certificate failed for cert Server-Cert of family In-Reply-To: <4a3d91a5-e48c-96cc-b1aa-6082eea2e302@redhat.com> References: <7d17ebb6-eba6-b560-bcf9-e016546095c1@redhat.com> <4a3d91a5-e48c-96cc-b1aa-6082eea2e302@redhat.com> Message-ID: Sorry Martin, This is not the first time I forgot to add back freeipa users. I have problems with gmail, again sorry. Indeed I figured out that I had to restart the ipa server. So I tried to restart ipa server. But it was not working yet. So I thought it was maybe due to the configuration I performed in the nss.conf. So I rollbacked this conf and restarted ipa-server. Then I retried your commands but it is still the same error. ### Request ID '20140528064145': status: CA_UNREACHABLE ca-error: Server failed request, will retry: 4301 (RPC failed at server. Certificate operation cannot be completed: Unable to communicate with CMS (Not Found)). stuck: yes key pair storage: type=NSSDB,location='/etc/httpd/alias',nickname='Server-Cert',token='NSS Certificate DB',pinfile='/etc/httpd/alias/pwdfile.txt' certificate: type=NSSDB,location='/etc/httpd/alias',nickname='Server-Cert',token='NSS Certificate DB' CA: IPA issuer: CN=Certificate Authority,O= subject: CN=,O= expires: 2016-05-28 06:41:44 UTC eku: id-kp-serverAuth,id-kp-clientAuth pre-save command: post-save command: /usr/lib64/ipa/certmonger/restart_httpd track: yes auto-renew: yes ### Do you know what is the CMS ? ### (RPC failed at server. Certificate operation cannot be completed: Unable to communicate with CMS (Not Found)). ### Best regards. Bahan On Wed, Sep 14, 2016 at 6:46 PM, Martin Basti wrote: > did you restart IPA when you moved time? Is there are more detailed error > description in output of getcert list? > > On 14.09.2016 18:45, bahan w wrote: > > I set the date-time when the certificates were valid : > ### > # date -s '2016-05-27 10:00:00' > Fri May 27 10:00:00 CEST 2016 > > # date > Fri May 27 10:00:02 CEST 2016 > ### > > Then I try to renew them : > ### > # getcert resubmit -i 20140528063919 > Resubmitting "20140528063919" to "IPA". > > # getcert resubmit -i 20140528064145 > Resubmitting "20140528064145" to "IPA". > > # getcert resubmit -i 20140528063953 > Resubmitting "20140528063953" to "IPA". > ### > > But when I do the getcert list after, the result is the same. > > I guess it is because of this ? > CA_UNREACHABLE > > Any idea ? > > Best regards. > > Bahan > > On Wed, Sep 14, 2016 at 6:38 PM, bahan w wrote: > >> Ok, I managed to restart the IPA service by adding this line in the file >> /etc/httpd/conf.d/nss.conf : >> ### >> NSSEnforceValidCerts off >> ### >> >> But when I do the getcert now I got the following result : >> >> ### >> # getcert list >> Number of certificates and requests being tracked: 8. >> Request ID '20140528063903': >> status: MONITORING >> stuck: no >> key pair storage: type=NSSDB,location='/var/lib/ >> pki-ca/alias',nickname='auditSigningCert cert-pki-ca',token='NSS >> Certificate DB',pin='159203530658' >> certificate: type=NSSDB,location='/var/lib/ >> pki-ca/alias',nickname='auditSigningCert cert-pki-ca',token='NSS >> Certificate DB' >> CA: dogtag-ipa-renew-agent >> issuer: CN=Certificate Authority,O= >> subject: CN=CA Audit,O= >> expires: 2018-04-09 11:39:16 UTC >> pre-save command: /usr/lib64/ipa/certmonger/stop_pkicad >> post-save command: /usr/lib64/ipa/certmonger/renew_ca_cert >> "auditSigningCert cert-pki-ca" >> track: yes >> auto-renew: yes >> Request ID '20140528063904': >> status: MONITORING >> stuck: no >> key pair storage: type=NSSDB,location='/var/lib/ >> pki-ca/alias',nickname='ocspSigningCert cert-pki-ca',token='NSS >> Certificate DB',pin='159203530658' >> certificate: type=NSSDB,location='/var/lib/ >> pki-ca/alias',nickname='ocspSigningCert cert-pki-ca',token='NSS >> Certificate DB' >> CA: dogtag-ipa-renew-agent >> issuer: CN=Certificate Authority,O= >> subject: CN=OCSP Subsystem,O= >> expires: 2018-04-09 11:38:16 UTC >> eku: id-kp-OCSPSigning >> pre-save command: /usr/lib64/ipa/certmonger/stop_pkicad >> post-save command: /usr/lib64/ipa/certmonger/renew_ca_cert >> "ocspSigningCert cert-pki-ca" >> track: yes >> auto-renew: yes >> Request ID '20140528063905': >> status: MONITORING >> stuck: no >> key pair storage: type=NSSDB,location='/var/lib/ >> pki-ca/alias',nickname='subsystemCert cert-pki-ca',token='NSS >> Certificate DB',pin='159203530658' >> certificate: type=NSSDB,location='/var/lib/ >> pki-ca/alias',nickname='subsystemCert cert-pki-ca',token='NSS >> Certificate DB' >> CA: dogtag-ipa-renew-agent >> issuer: CN=Certificate Authority,O= >> subject: CN=CA Subsystem,O= >> expires: 2018-04-09 11:38:16 UTC >> eku: id-kp-serverAuth,id-kp-clientAuth >> pre-save command: /usr/lib64/ipa/certmonger/stop_pkicad >> post-save command: /usr/lib64/ipa/certmonger/renew_ca_cert >> "subsystemCert cert-pki-ca" >> track: yes >> auto-renew: yes >> Request ID '20140528063906': >> status: MONITORING >> stuck: no >> key pair storage: type=NSSDB,location='/etc/http >> d/alias',nickname='ipaCert',token='NSS Certificate >> DB',pinfile='/etc/httpd/alias/pwdfile.txt' >> certificate: type=NSSDB,location='/etc/http >> d/alias',nickname='ipaCert',token='NSS Certificate DB' >> CA: dogtag-ipa-renew-agent >> issuer: CN=Certificate Authority,O= >> subject: CN=IPA RA,O= >> expires: 2018-04-09 11:38:16 UTC >> eku: id-kp-serverAuth,id-kp-clientAuth >> pre-save command: >> post-save command: /usr/lib64/ipa/certmonger/renew_ra_cert >> track: yes >> auto-renew: yes >> Request ID '20140528063907': >> status: MONITORING >> stuck: no >> key pair storage: type=NSSDB,location='/var/lib/ >> pki-ca/alias',nickname='Server-Cert cert-pki-ca',token='NSS Certificate >> DB',pin='159203530658' >> certificate: type=NSSDB,location='/var/lib/ >> pki-ca/alias',nickname='Server-Cert cert-pki-ca',token='NSS Certificate >> DB' >> CA: dogtag-ipa-renew-agent >> issuer: CN=Certificate Authority,O= >> subject: CN=,O= >> expires: 2018-04-09 11:38:16 UTC >> eku: id-kp-serverAuth,id-kp-clientAuth >> pre-save command: >> post-save command: >> track: yes >> auto-renew: yes >> Request ID '20140528063919': >> status: CA_UNREACHABLE >> ca-error: Server failed request, will retry: -504 (libcurl failed >> to execute the HTTP POST transaction. Peer certificate cannot be >> authenticated with known CA certificates). >> stuck: yes >> key pair storage: type=NSSDB,location='/etc/dirs >> rv/slapd-',nickname='Server-Cert',token='NSS Certificate >> DB',pinfile='/etc/dirsrv/slapd-/pwdfile.txt' >> certificate: type=NSSDB,location='/etc/dirs >> rv/slapd-',nickname='Server-Cert',token='NSS Certificate DB' >> CA: IPA >> issuer: CN=Certificate Authority,O= >> subject: CN=,O= >> expires: 2016-05-28 06:39:18 UTC >> eku: id-kp-serverAuth,id-kp-clientAuth >> pre-save command: >> post-save command: /usr/lib64/ipa/certmonger/restart_dirsrv >> >> track: yes >> auto-renew: yes >> Request ID '20140528063953': >> status: CA_UNREACHABLE >> ca-error: Server failed request, will retry: -504 (libcurl failed >> to execute the HTTP POST transaction. Peer certificate cannot be >> authenticated with known CA certificates). >> stuck: yes >> key pair storage: type=NSSDB,location='/etc/dirs >> rv/slapd-PKI-IPA',nickname='Server-Cert',token='NSS Certificate >> DB',pinfile='/etc/dirsrv/slapd-PKI-IPA/pwdfile.txt' >> certificate: type=NSSDB,location='/etc/dirs >> rv/slapd-PKI-IPA',nickname='Server-Cert',token='NSS Certificate DB' >> CA: IPA >> issuer: CN=Certificate Authority,O= >> subject: CN=,O= >> expires: 2016-05-28 06:39:52 UTC >> eku: id-kp-serverAuth,id-kp-clientAuth >> pre-save command: >> post-save command: /usr/lib64/ipa/certmonger/restart_dirsrv >> PKI-IPA >> track: yes >> auto-renew: yes >> Request ID '20140528064145': >> status: CA_UNREACHABLE >> ca-error: Server failed request, will retry: -504 (libcurl failed >> to execute the HTTP POST transaction. Peer certificate cannot be >> authenticated with known CA certificates). >> stuck: yes >> key pair storage: type=NSSDB,location='/etc/http >> d/alias',nickname='Server-Cert',token='NSS Certificate >> DB',pinfile='/etc/httpd/alias/pwdfile.txt' >> certificate: type=NSSDB,location='/etc/http >> d/alias',nickname='Server-Cert',token='NSS Certificate DB' >> CA: IPA >> issuer: CN=Certificate Authority,O= >> subject: CN=,O= >> expires: 2016-05-28 06:41:44 UTC >> eku: id-kp-serverAuth,id-kp-clientAuth >> pre-save command: >> post-save command: /usr/lib64/ipa/certmonger/restart_httpd >> track: yes >> auto-renew: yes >> ### >> >> Indeed, the entries outdated are the following : >> - for /etc/dirsrv/slapd- : 20140528063919 >> - for /etc/dirsrv/slapd-PKI-IPA : 20140528063953 >> - for httpd ? : 20140528064145 >> >> Best regards. >> >> Bahan >> >> On Wed, Sep 14, 2016 at 6:28 PM, bahan w wrote: >> >>> Ok :D >>> >>> Because to perform the getcert list command, I need to have all the ipa >>> services running right ? >>> >>> Here is the result of the command with the ipa services down. >>> ### >>> # getcert list >>> Number of certificates and requests being tracked: 8. >>> Request ID '20140528063903': >>> status: MONITORING >>> stuck: no >>> key pair storage: type=NSSDB,location='/var/lib/ >>> pki-ca/alias',nickname='auditSigningCert cert-pki-ca',token='NSS >>> Certificate DB',pin='159203530658' >>> certificate: type=NSSDB,location='/var/lib/ >>> pki-ca/alias',nickname='auditSigningCert cert-pki-ca',token='NSS >>> Certificate DB' >>> CA: dogtag-ipa-renew-agent >>> issuer: CN=Certificate Authority,O= >>> subject: CN=CA Audit,O= >>> expires: 2018-04-09 11:39:16 UTC >>> pre-save command: /usr/lib64/ipa/certmonger/stop_pkicad >>> post-save command: /usr/lib64/ipa/certmonger/renew_ca_cert >>> "auditSigningCert cert-pki-ca" >>> track: yes >>> auto-renew: yes >>> Request ID '20140528063904': >>> status: MONITORING >>> stuck: no >>> key pair storage: type=NSSDB,location='/var/lib/ >>> pki-ca/alias',nickname='ocspSigningCert cert-pki-ca',token='NSS >>> Certificate DB',pin='159203530658' >>> certificate: type=NSSDB,location='/var/lib/ >>> pki-ca/alias',nickname='ocspSigningCert cert-pki-ca',token='NSS >>> Certificate DB' >>> CA: dogtag-ipa-renew-agent >>> issuer: CN=Certificate Authority,O= >>> subject: CN=OCSP Subsystem,O= >>> expires: 2018-04-09 11:38:16 UTC >>> eku: id-kp-OCSPSigning >>> pre-save command: /usr/lib64/ipa/certmonger/stop_pkicad >>> post-save command: /usr/lib64/ipa/certmonger/renew_ca_cert >>> "ocspSigningCert cert-pki-ca" >>> track: yes >>> auto-renew: yes >>> Request ID '20140528063905': >>> status: MONITORING >>> stuck: no >>> key pair storage: type=NSSDB,location='/var/lib/ >>> pki-ca/alias',nickname='subsystemCert cert-pki-ca',token='NSS >>> Certificate DB',pin='159203530658' >>> certificate: type=NSSDB,location='/var/lib/ >>> pki-ca/alias',nickname='subsystemCert cert-pki-ca',token='NSS >>> Certificate DB' >>> CA: dogtag-ipa-renew-agent >>> issuer: CN=Certificate Authority,O= >>> subject: CN=CA Subsystem,O= >>> expires: 2018-04-09 11:38:16 UTC >>> eku: id-kp-serverAuth,id-kp-clientAuth >>> pre-save command: /usr/lib64/ipa/certmonger/stop_pkicad >>> post-save command: /usr/lib64/ipa/certmonger/renew_ca_cert >>> "subsystemCert cert-pki-ca" >>> track: yes >>> auto-renew: yes >>> Request ID '20140528063906': >>> status: MONITORING >>> stuck: no >>> key pair storage: type=NSSDB,location='/etc/http >>> d/alias',nickname='ipaCert',token='NSS Certificate >>> DB',pinfile='/etc/httpd/alias/pwdfile.txt' >>> certificate: type=NSSDB,location='/etc/http >>> d/alias',nickname='ipaCert',token='NSS Certificate DB' >>> CA: dogtag-ipa-renew-agent >>> issuer: CN=Certificate Authority,O= >>> subject: CN=IPA RA,O= >>> expires: 2018-04-09 11:38:16 UTC >>> eku: id-kp-serverAuth,id-kp-clientAuth >>> pre-save command: >>> post-save command: /usr/lib64/ipa/certmonger/renew_ra_cert >>> track: yes >>> auto-renew: yes >>> Request ID '20140528063907': >>> status: MONITORING >>> stuck: no >>> key pair storage: type=NSSDB,location='/var/lib/ >>> pki-ca/alias',nickname='Server-Cert cert-pki-ca',token='NSS Certificate >>> DB',pin='159203530658' >>> certificate: type=NSSDB,location='/var/lib/ >>> pki-ca/alias',nickname='Server-Cert cert-pki-ca',token='NSS Certificate >>> DB' >>> CA: dogtag-ipa-renew-agent >>> issuer: CN=Certificate Authority,O= >>> subject: CN=,O= >>> expires: 2018-04-09 11:38:16 UTC >>> eku: id-kp-serverAuth,id-kp-clientAuth >>> pre-save command: >>> post-save command: >>> track: yes >>> auto-renew: yes >>> Request ID '20140528063919': >>> status: MONITORING >>> ca-error: Error setting up ccache for local "host" service using >>> default keytab: Cannot contact any KDC for realm ''. >>> stuck: no >>> key pair storage: type=NSSDB,location='/etc/dirs >>> rv/slapd-',nickname='Server-Cert',token='NSS Certificate >>> DB',pinfile='/etc/dirsrv/slapd-/pwdfile.txt' >>> certificate: type=NSSDB,location='/etc/dirs >>> rv/slapd-',nickname='Server-Cert',token='NSS Certificate DB' >>> CA: IPA >>> issuer: CN=Certificate Authority,O= >>> subject: CN=,O= >>> expires: 2016-05-28 06:39:18 UTC >>> eku: id-kp-serverAuth,id-kp-clientAuth >>> pre-save command: >>> post-save command: /usr/lib64/ipa/certmonger/restart_dirsrv >>> >>> track: yes >>> auto-renew: yes >>> Request ID '20140528063953': >>> status: MONITORING >>> ca-error: Error setting up ccache for local "host" service using >>> default keytab: Cannot contact any KDC for realm ''. >>> stuck: no >>> key pair storage: type=NSSDB,location='/etc/dirs >>> rv/slapd-PKI-IPA',nickname='Server-Cert',token='NSS Certificate >>> DB',pinfile='/etc/dirsrv/slapd-PKI-IPA/pwdfile.txt' >>> certificate: type=NSSDB,location='/etc/dirs >>> rv/slapd-PKI-IPA',nickname='Server-Cert',token='NSS Certificate DB' >>> CA: IPA >>> issuer: CN=Certificate Authority,O= >>> subject: CN=,O= >>> expires: 2016-05-28 06:39:52 UTC >>> eku: id-kp-serverAuth,id-kp-clientAuth >>> pre-save command: >>> post-save command: /usr/lib64/ipa/certmonger/restart_dirsrv >>> PKI-IPA >>> track: yes >>> auto-renew: yes >>> Request ID '20140528064145': >>> status: MONITORING >>> ca-error: Error setting up ccache for local "host" service using >>> default keytab: Cannot contact any KDC for realm ''. >>> stuck: no >>> key pair storage: type=NSSDB,location='/etc/http >>> d/alias',nickname='Server-Cert',token='NSS Certificate >>> DB',pinfile='/etc/httpd/alias/pwdfile.txt' >>> certificate: type=NSSDB,location='/etc/http >>> d/alias',nickname='Server-Cert',token='NSS Certificate DB' >>> CA: IPA >>> issuer: CN=Certificate Authority,O= >>> subject: CN=,O= >>> expires: 2016-05-28 06:41:44 UTC >>> eku: id-kp-serverAuth,id-kp-clientAuth >>> pre-save command: >>> post-save command: /usr/lib64/ipa/certmonger/restart_httpd >>> track: yes >>> auto-renew: yes >>> ### >>> >>> Best regards. >>> >>> Bahan >>> >>> On Wed, Sep 14, 2016 at 6:21 PM, Martin Basti wrote: >>> >>>> >>>> Then you have to start services manually, I don't know if the same >>>> steps will work with IPA 3.0.0, I don't remember, but you can try :) >>>> >>>> On 14.09.2016 18:18, bahan w wrote: >>>> >>>> Oh I forgot to add that my version of ipa is quite old : >>>> ### >>>> # rpm -qa | grep ipa-server >>>> ipa-server-3.0.0-25.el6.x86_64 >>>> ### >>>> >>>> When I try the command you gave me I got the following error : >>>> ### >>>> # ipactl start --force >>>> Usage: ipactl start|stop|restart|status >>>> >>>> >>>> ipactl: error: no such option: --force >>>> ### >>>> >>>> Best regards. >>>> >>>> Bahan >>>> >>>> >>>> On Wed, Sep 14, 2016 at 6:14 PM, Martin Basti >>>> wrote: >>>> >>>>> >>>>> >>>>> On 14.09.2016 17:59, bahan w wrote: >>>>> >>>>> Hello ! >>>>> >>>>> I send you this mail because I cannot restart my test IPA server. >>>>> >>>>> When I try to start it with service ipa start, I got the following >>>>> error message : >>>>> ### >>>>> # service ipa start >>>>> Starting Directory Service >>>>> Starting dirsrv: >>>>> ...[14/Sep/2016:17:57:23 +0200] - SSL alert: >>>>> CERT_VerifyCertificateNow: verify certificate failed for cert Server-Cert >>>>> of family cn=RSA,cn=encryption,cn=config (Netscape Portable Runtime error >>>>> -8181 - Peer's Certificate has expired.) >>>>> [ OK ] >>>>> PKI-IPA...[14/Sep/2016:17:57:33 +0200] - SSL alert: >>>>> CERT_VerifyCertificateNow: verify certificate failed for cert Server-Cert >>>>> of family cn=RSA,cn=encryption,cn=config (Netscape Portable Runtime error >>>>> -8181 - Peer's Certificate has expired.) >>>>> [ OK ] >>>>> Starting KDC Service >>>>> Starting Kerberos 5 KDC: [ OK ] >>>>> Starting KPASSWD Service >>>>> Starting Kerberos 5 Admin Server: [ OK ] >>>>> Starting MEMCACHE Service >>>>> Starting ipa_memcached: [ OK ] >>>>> Starting HTTP Service >>>>> Starting httpd: [FAILED] >>>>> Failed to start HTTP Service >>>>> Shutting down >>>>> Stopping Kerberos 5 KDC: [ OK ] >>>>> Stopping Kerberos 5 Admin Server: [ OK ] >>>>> Stopping ipa_memcached: [ OK ] >>>>> Stopping httpd: [FAILED] >>>>> Stopping pki-ca: [ OK ] >>>>> Shutting down dirsrv: >>>>> ... [ OK ] >>>>> PKI-IPA... [ OK ] >>>>> Aborting ipactl >>>>> >>>>> # service ipa status >>>>> Directory Service: STOPPED >>>>> Failed to get list of services to probe status: >>>>> Directory Server is stopped >>>>> ### >>>>> >>>>> Do you know how to renew the SSL certificate used for the IPA Server ? >>>>> >>>>> Best regards. >>>>> >>>>> Bahan >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> Hello, >>>>> >>>>> please run >>>>> >>>>> # ipactl start --force >>>>> # getcert list (to detect which certificate is outdated, I suspect DS >>>>> cert (or to get more info why it has not been renewed)) >>>>> >>>>> If getcert does work (I'm not sure if ti is able to work without >>>>> httpd), you probable need to move time back to past where cert is valid, >>>>> start IPA and try again. >>>>> >>>>> Please find ID outdated certificate and try resubmit it (CA and DS >>>>> must be running) >>>>> >>>>> # getcert resubmit -i 20160914122036 (use you ID :) ) >>>>> >>>>> This should renew cert, check status with getcert list >>>>> >>>>> Move time back to future (if needed) >>>>> >>>>> Try to restart IPA >>>>> >>>>> Martin^2 >>>>> >>>> >>>> >>>> >>> >> > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From g.kafataridis at nelios.com Wed Sep 14 17:26:07 2016 From: g.kafataridis at nelios.com (Giorgos Kafataridis) Date: Wed, 14 Sep 2016 20:26:07 +0300 Subject: [Freeipa-users] CA: Cannot add Centos7.2 replica to Centos6.8 ipa server In-Reply-To: <0b546b4a-9ef3-7a15-27a7-3a2d59e5281f@redhat.com> References: <45b66871-aa01-3c91-0630-0bbeee488ae7@nelios.com> <323b9de1-adb3-47af-d4a7-d6083f88d75c@nelios.com> <9afdf0f3-97d1-cc6e-5f3d-0e11687a124a@redhat.com> <05b8e986-ceef-8e75-c76d-b38c45e2e3a1@nelios.com> <79903d5c-2b80-d492-6b29-59ac5b39c6f5@redhat.com> <98b16feb-8b70-13bb-4c92-cd9bca3e408d@redhat.com> <0b546b4a-9ef3-7a15-27a7-3a2d59e5281f@redhat.com> Message-ID: On 09/13/2016 10:36 PM, Endi Sukma Dewata wrote: > On 9/12/2016 9:35 PM, Endi Sukma Dewata wrote: >> On 9/9/2016 2:46 PM, Georgios Kafataridis wrote: >>> I've tried that but still the same result. >>> >>> [root at ipa-server /]# ldapsearch -D "cn=directory manager" -W -p 389 -h >>> localhost -b "uid=admin,ou=people,o=ipaca" >>> Enter LDAP Password: >>> # extended LDIF >>> # >>> # LDAPv3 >>> # base with scope subtree >>> # filter: (objectclass=*) >>> # requesting: ALL >>> # >>> >>> # search result >>> search: 2 >>> result: 32 No such object >> >> Hi, >> >> The master's logs indicate there's an authentication issue. >> >> Could you search the whole directory to find the admin user? >> $ ldapsearch ... -b "o=ipaca" "(uid=admin)" >> >> Try also other suffixes that you have in the DS. >> >> If you find it, try to authenticate against DS directly as the admin >> user. If the authentication fails, try resetting the password. > > I believe there is actually another DS instance on CentOS 6.8 running > on port 7389, so make sure you check that too. If the admin user is > indeed missing, it will need to be recreated, assigned a password and > certificate, and added to the appropriate groups. > > See also: http://pki.fedoraproject.org/wiki/IPA_PKI_Users > Sorry for the delay, crazy office days. Ok, tried that and finally got a hit on the user. Indeed in 6.x you also have 7389 to look for. *Master *#ldapsearch -h localhost -p 7389 -b "o=ipaca" "(uid=admin)" -x -W Enter LDAP Password: # extended LDIF # # LDAPv3 # base with scope subtree # filter: (uid=admin) # requesting: ALL # # admin, people, ipaca dn: uid=admin,ou=people,o=ipaca objectClass: top objectClass: person objectClass: organizationalPerson objectClass: inetOrgPerson objectClass: cmsuser uid: admin sn: admin cn: admin mail: root at localhost usertype: adminType userstate: 1 description: 2;6;CN=Certificate Authority,O=NELIOS;CN=ipa-ca-agent,O=NELIOS userCertificate:: MIIDaTCCAlGgAwIBAgIBBjANBgkqhkiG9w0BAQsFADAxMQ8wDQYDVQQKEwZO.... . . . . # search result search: 2 result: 0 Success # numResponses: 2 # numEntries: 1 *Replica Server* [root at ipa2-server2 ~]# ldapsearch -h ipa-server.nelios -p 7389 -b "o=ipaca" "(uid=admin)" -x -W # extended LDIF # # LDAPv3 # base with scope subtree # filter: (uid=admin) # requesting: ALL # # admin, people, ipaca dn: uid=admin,ou=people,o=ipaca objectClass: top objectClass: person objectClass: organizationalPerson objectClass: inetOrgPerson objectClass: cmsuser uid: admin sn: admin cn: admin mail: root at localhost usertype: adminType userstate: 1 Password is valid in both cases. So the user is there and can be retrieved from replica, assuming that ipa-replica-install also tries 7389 the only thing I can try now is "ipa cert-request --uid admin" to create a new certificate, generate a new cacert.p12 and retry install ? -------------- next part -------------- An HTML attachment was scrubbed... URL: From bahanw042014 at gmail.com Wed Sep 14 17:27:52 2016 From: bahanw042014 at gmail.com (bahan w) Date: Wed, 14 Sep 2016 19:27:52 +0200 Subject: [Freeipa-users] SSL alert: CERT_VerifyCertificateNow: verify certificate failed for cert Server-Cert of family In-Reply-To: References: <7d17ebb6-eba6-b560-bcf9-e016546095c1@redhat.com> <4a3d91a5-e48c-96cc-b1aa-6082eea2e302@redhat.com> Message-ID: I tried also the following commands : ### # ipa cert-show 1 ipa: ERROR: Certificate operation cannot be completed: Unable to communicate with CMS (Not Found) # service ipa status Directory Service: RUNNING KDC Service: RUNNING KPASSWD Service: RUNNING MEMCACHE Service: RUNNING HTTP Service: RUNNING CA Service: RUNNING ### I'm checking the /var/log/pki-ca logs to see if I find something. Best regards. Bahan On Wed, Sep 14, 2016 at 7:02 PM, bahan w wrote: > Sorry Martin, > > This is not the first time I forgot to add back freeipa users. > I have problems with gmail, again sorry. > > Indeed I figured out that I had to restart the ipa server. > So I tried to restart ipa server. > But it was not working yet. > > So I thought it was maybe due to the configuration I performed in the > nss.conf. > So I rollbacked this conf and restarted ipa-server. > Then I retried your commands but it is still the same error. > > ### > Request ID '20140528064145': > status: CA_UNREACHABLE > ca-error: Server failed request, will retry: 4301 (RPC failed at > server. Certificate operation cannot be completed: Unable to communicate > with CMS (Not Found)). > stuck: yes > key pair storage: type=NSSDB,location='/etc/ > httpd/alias',nickname='Server-Cert',token='NSS Certificate > DB',pinfile='/etc/httpd/alias/pwdfile.txt' > certificate: type=NSSDB,location='/etc/ > httpd/alias',nickname='Server-Cert',token='NSS Certificate DB' > CA: IPA > issuer: CN=Certificate Authority,O= > subject: CN=,O= > expires: 2016-05-28 06:41:44 UTC > eku: id-kp-serverAuth,id-kp-clientAuth > pre-save command: > post-save command: /usr/lib64/ipa/certmonger/restart_httpd > track: yes > auto-renew: yes > ### > > Do you know what is the CMS ? > ### > (RPC failed at server. Certificate operation cannot be completed: Unable > to communicate with CMS (Not Found)). > ### > > Best regards. > > Bahan > > > > > > On Wed, Sep 14, 2016 at 6:46 PM, Martin Basti wrote: > >> did you restart IPA when you moved time? Is there are more detailed error >> description in output of getcert list? >> >> On 14.09.2016 18:45, bahan w wrote: >> >> I set the date-time when the certificates were valid : >> ### >> # date -s '2016-05-27 10:00:00' >> Fri May 27 10:00:00 CEST 2016 >> >> # date >> Fri May 27 10:00:02 CEST 2016 >> ### >> >> Then I try to renew them : >> ### >> # getcert resubmit -i 20140528063919 >> Resubmitting "20140528063919" to "IPA". >> >> # getcert resubmit -i 20140528064145 >> Resubmitting "20140528064145" to "IPA". >> >> # getcert resubmit -i 20140528063953 >> Resubmitting "20140528063953" to "IPA". >> ### >> >> But when I do the getcert list after, the result is the same. >> >> I guess it is because of this ? >> CA_UNREACHABLE >> >> Any idea ? >> >> Best regards. >> >> Bahan >> >> On Wed, Sep 14, 2016 at 6:38 PM, bahan w wrote: >> >>> Ok, I managed to restart the IPA service by adding this line in the file >>> /etc/httpd/conf.d/nss.conf : >>> ### >>> NSSEnforceValidCerts off >>> ### >>> >>> But when I do the getcert now I got the following result : >>> >>> ### >>> # getcert list >>> Number of certificates and requests being tracked: 8. >>> Request ID '20140528063903': >>> status: MONITORING >>> stuck: no >>> key pair storage: type=NSSDB,location='/var/lib/ >>> pki-ca/alias',nickname='auditSigningCert cert-pki-ca',token='NSS >>> Certificate DB',pin='159203530658' >>> certificate: type=NSSDB,location='/var/lib/ >>> pki-ca/alias',nickname='auditSigningCert cert-pki-ca',token='NSS >>> Certificate DB' >>> CA: dogtag-ipa-renew-agent >>> issuer: CN=Certificate Authority,O= >>> subject: CN=CA Audit,O= >>> expires: 2018-04-09 11:39:16 UTC >>> pre-save command: /usr/lib64/ipa/certmonger/stop_pkicad >>> post-save command: /usr/lib64/ipa/certmonger/renew_ca_cert >>> "auditSigningCert cert-pki-ca" >>> track: yes >>> auto-renew: yes >>> Request ID '20140528063904': >>> status: MONITORING >>> stuck: no >>> key pair storage: type=NSSDB,location='/var/lib/ >>> pki-ca/alias',nickname='ocspSigningCert cert-pki-ca',token='NSS >>> Certificate DB',pin='159203530658' >>> certificate: type=NSSDB,location='/var/lib/ >>> pki-ca/alias',nickname='ocspSigningCert cert-pki-ca',token='NSS >>> Certificate DB' >>> CA: dogtag-ipa-renew-agent >>> issuer: CN=Certificate Authority,O= >>> subject: CN=OCSP Subsystem,O= >>> expires: 2018-04-09 11:38:16 UTC >>> eku: id-kp-OCSPSigning >>> pre-save command: /usr/lib64/ipa/certmonger/stop_pkicad >>> post-save command: /usr/lib64/ipa/certmonger/renew_ca_cert >>> "ocspSigningCert cert-pki-ca" >>> track: yes >>> auto-renew: yes >>> Request ID '20140528063905': >>> status: MONITORING >>> stuck: no >>> key pair storage: type=NSSDB,location='/var/lib/ >>> pki-ca/alias',nickname='subsystemCert cert-pki-ca',token='NSS >>> Certificate DB',pin='159203530658' >>> certificate: type=NSSDB,location='/var/lib/ >>> pki-ca/alias',nickname='subsystemCert cert-pki-ca',token='NSS >>> Certificate DB' >>> CA: dogtag-ipa-renew-agent >>> issuer: CN=Certificate Authority,O= >>> subject: CN=CA Subsystem,O= >>> expires: 2018-04-09 11:38:16 UTC >>> eku: id-kp-serverAuth,id-kp-clientAuth >>> pre-save command: /usr/lib64/ipa/certmonger/stop_pkicad >>> post-save command: /usr/lib64/ipa/certmonger/renew_ca_cert >>> "subsystemCert cert-pki-ca" >>> track: yes >>> auto-renew: yes >>> Request ID '20140528063906': >>> status: MONITORING >>> stuck: no >>> key pair storage: type=NSSDB,location='/etc/http >>> d/alias',nickname='ipaCert',token='NSS Certificate >>> DB',pinfile='/etc/httpd/alias/pwdfile.txt' >>> certificate: type=NSSDB,location='/etc/http >>> d/alias',nickname='ipaCert',token='NSS Certificate DB' >>> CA: dogtag-ipa-renew-agent >>> issuer: CN=Certificate Authority,O= >>> subject: CN=IPA RA,O= >>> expires: 2018-04-09 11:38:16 UTC >>> eku: id-kp-serverAuth,id-kp-clientAuth >>> pre-save command: >>> post-save command: /usr/lib64/ipa/certmonger/renew_ra_cert >>> track: yes >>> auto-renew: yes >>> Request ID '20140528063907': >>> status: MONITORING >>> stuck: no >>> key pair storage: type=NSSDB,location='/var/lib/ >>> pki-ca/alias',nickname='Server-Cert cert-pki-ca',token='NSS Certificate >>> DB',pin='159203530658' >>> certificate: type=NSSDB,location='/var/lib/ >>> pki-ca/alias',nickname='Server-Cert cert-pki-ca',token='NSS Certificate >>> DB' >>> CA: dogtag-ipa-renew-agent >>> issuer: CN=Certificate Authority,O= >>> subject: CN=,O= >>> expires: 2018-04-09 11:38:16 UTC >>> eku: id-kp-serverAuth,id-kp-clientAuth >>> pre-save command: >>> post-save command: >>> track: yes >>> auto-renew: yes >>> Request ID '20140528063919': >>> status: CA_UNREACHABLE >>> ca-error: Server failed request, will retry: -504 (libcurl >>> failed to execute the HTTP POST transaction. Peer certificate cannot be >>> authenticated with known CA certificates). >>> stuck: yes >>> key pair storage: type=NSSDB,location='/etc/dirs >>> rv/slapd-',nickname='Server-Cert',token='NSS Certificate >>> DB',pinfile='/etc/dirsrv/slapd-/pwdfile.txt' >>> certificate: type=NSSDB,location='/etc/dirs >>> rv/slapd-',nickname='Server-Cert',token='NSS Certificate DB' >>> CA: IPA >>> issuer: CN=Certificate Authority,O= >>> subject: CN=,O= >>> expires: 2016-05-28 06:39:18 UTC >>> eku: id-kp-serverAuth,id-kp-clientAuth >>> pre-save command: >>> post-save command: /usr/lib64/ipa/certmonger/restart_dirsrv >>> >>> track: yes >>> auto-renew: yes >>> Request ID '20140528063953': >>> status: CA_UNREACHABLE >>> ca-error: Server failed request, will retry: -504 (libcurl >>> failed to execute the HTTP POST transaction. Peer certificate cannot be >>> authenticated with known CA certificates). >>> stuck: yes >>> key pair storage: type=NSSDB,location='/etc/dirs >>> rv/slapd-PKI-IPA',nickname='Server-Cert',token='NSS Certificate >>> DB',pinfile='/etc/dirsrv/slapd-PKI-IPA/pwdfile.txt' >>> certificate: type=NSSDB,location='/etc/dirs >>> rv/slapd-PKI-IPA',nickname='Server-Cert',token='NSS Certificate DB' >>> CA: IPA >>> issuer: CN=Certificate Authority,O= >>> subject: CN=,O= >>> expires: 2016-05-28 06:39:52 UTC >>> eku: id-kp-serverAuth,id-kp-clientAuth >>> pre-save command: >>> post-save command: /usr/lib64/ipa/certmonger/restart_dirsrv >>> PKI-IPA >>> track: yes >>> auto-renew: yes >>> Request ID '20140528064145': >>> status: CA_UNREACHABLE >>> ca-error: Server failed request, will retry: -504 (libcurl >>> failed to execute the HTTP POST transaction. Peer certificate cannot be >>> authenticated with known CA certificates). >>> stuck: yes >>> key pair storage: type=NSSDB,location='/etc/http >>> d/alias',nickname='Server-Cert',token='NSS Certificate >>> DB',pinfile='/etc/httpd/alias/pwdfile.txt' >>> certificate: type=NSSDB,location='/etc/http >>> d/alias',nickname='Server-Cert',token='NSS Certificate DB' >>> CA: IPA >>> issuer: CN=Certificate Authority,O= >>> subject: CN=,O= >>> expires: 2016-05-28 06:41:44 UTC >>> eku: id-kp-serverAuth,id-kp-clientAuth >>> pre-save command: >>> post-save command: /usr/lib64/ipa/certmonger/restart_httpd >>> track: yes >>> auto-renew: yes >>> ### >>> >>> Indeed, the entries outdated are the following : >>> - for /etc/dirsrv/slapd- : 20140528063919 >>> - for /etc/dirsrv/slapd-PKI-IPA : 20140528063953 >>> - for httpd ? : 20140528064145 >>> >>> Best regards. >>> >>> Bahan >>> >>> On Wed, Sep 14, 2016 at 6:28 PM, bahan w wrote: >>> >>>> Ok :D >>>> >>>> Because to perform the getcert list command, I need to have all the ipa >>>> services running right ? >>>> >>>> Here is the result of the command with the ipa services down. >>>> ### >>>> # getcert list >>>> Number of certificates and requests being tracked: 8. >>>> Request ID '20140528063903': >>>> status: MONITORING >>>> stuck: no >>>> key pair storage: type=NSSDB,location='/var/lib/ >>>> pki-ca/alias',nickname='auditSigningCert cert-pki-ca',token='NSS >>>> Certificate DB',pin='159203530658' >>>> certificate: type=NSSDB,location='/var/lib/ >>>> pki-ca/alias',nickname='auditSigningCert cert-pki-ca',token='NSS >>>> Certificate DB' >>>> CA: dogtag-ipa-renew-agent >>>> issuer: CN=Certificate Authority,O= >>>> subject: CN=CA Audit,O= >>>> expires: 2018-04-09 11:39:16 UTC >>>> pre-save command: /usr/lib64/ipa/certmonger/stop_pkicad >>>> post-save command: /usr/lib64/ipa/certmonger/renew_ca_cert >>>> "auditSigningCert cert-pki-ca" >>>> track: yes >>>> auto-renew: yes >>>> Request ID '20140528063904': >>>> status: MONITORING >>>> stuck: no >>>> key pair storage: type=NSSDB,location='/var/lib/ >>>> pki-ca/alias',nickname='ocspSigningCert cert-pki-ca',token='NSS >>>> Certificate DB',pin='159203530658' >>>> certificate: type=NSSDB,location='/var/lib/ >>>> pki-ca/alias',nickname='ocspSigningCert cert-pki-ca',token='NSS >>>> Certificate DB' >>>> CA: dogtag-ipa-renew-agent >>>> issuer: CN=Certificate Authority,O= >>>> subject: CN=OCSP Subsystem,O= >>>> expires: 2018-04-09 11:38:16 UTC >>>> eku: id-kp-OCSPSigning >>>> pre-save command: /usr/lib64/ipa/certmonger/stop_pkicad >>>> post-save command: /usr/lib64/ipa/certmonger/renew_ca_cert >>>> "ocspSigningCert cert-pki-ca" >>>> track: yes >>>> auto-renew: yes >>>> Request ID '20140528063905': >>>> status: MONITORING >>>> stuck: no >>>> key pair storage: type=NSSDB,location='/var/lib/ >>>> pki-ca/alias',nickname='subsystemCert cert-pki-ca',token='NSS >>>> Certificate DB',pin='159203530658' >>>> certificate: type=NSSDB,location='/var/lib/ >>>> pki-ca/alias',nickname='subsystemCert cert-pki-ca',token='NSS >>>> Certificate DB' >>>> CA: dogtag-ipa-renew-agent >>>> issuer: CN=Certificate Authority,O= >>>> subject: CN=CA Subsystem,O= >>>> expires: 2018-04-09 11:38:16 UTC >>>> eku: id-kp-serverAuth,id-kp-clientAuth >>>> pre-save command: /usr/lib64/ipa/certmonger/stop_pkicad >>>> post-save command: /usr/lib64/ipa/certmonger/renew_ca_cert >>>> "subsystemCert cert-pki-ca" >>>> track: yes >>>> auto-renew: yes >>>> Request ID '20140528063906': >>>> status: MONITORING >>>> stuck: no >>>> key pair storage: type=NSSDB,location='/etc/http >>>> d/alias',nickname='ipaCert',token='NSS Certificate >>>> DB',pinfile='/etc/httpd/alias/pwdfile.txt' >>>> certificate: type=NSSDB,location='/etc/http >>>> d/alias',nickname='ipaCert',token='NSS Certificate DB' >>>> CA: dogtag-ipa-renew-agent >>>> issuer: CN=Certificate Authority,O= >>>> subject: CN=IPA RA,O= >>>> expires: 2018-04-09 11:38:16 UTC >>>> eku: id-kp-serverAuth,id-kp-clientAuth >>>> pre-save command: >>>> post-save command: /usr/lib64/ipa/certmonger/renew_ra_cert >>>> track: yes >>>> auto-renew: yes >>>> Request ID '20140528063907': >>>> status: MONITORING >>>> stuck: no >>>> key pair storage: type=NSSDB,location='/var/lib/ >>>> pki-ca/alias',nickname='Server-Cert cert-pki-ca',token='NSS >>>> Certificate DB',pin='159203530658' >>>> certificate: type=NSSDB,location='/var/lib/ >>>> pki-ca/alias',nickname='Server-Cert cert-pki-ca',token='NSS >>>> Certificate DB' >>>> CA: dogtag-ipa-renew-agent >>>> issuer: CN=Certificate Authority,O= >>>> subject: CN=,O= >>>> expires: 2018-04-09 11:38:16 UTC >>>> eku: id-kp-serverAuth,id-kp-clientAuth >>>> pre-save command: >>>> post-save command: >>>> track: yes >>>> auto-renew: yes >>>> Request ID '20140528063919': >>>> status: MONITORING >>>> ca-error: Error setting up ccache for local "host" service >>>> using default keytab: Cannot contact any KDC for realm ''. >>>> stuck: no >>>> key pair storage: type=NSSDB,location='/etc/dirs >>>> rv/slapd-',nickname='Server-Cert',token='NSS Certificate >>>> DB',pinfile='/etc/dirsrv/slapd-/pwdfile.txt' >>>> certificate: type=NSSDB,location='/etc/dirs >>>> rv/slapd-',nickname='Server-Cert',token='NSS Certificate DB' >>>> CA: IPA >>>> issuer: CN=Certificate Authority,O= >>>> subject: CN=,O= >>>> expires: 2016-05-28 06:39:18 UTC >>>> eku: id-kp-serverAuth,id-kp-clientAuth >>>> pre-save command: >>>> post-save command: /usr/lib64/ipa/certmonger/restart_dirsrv >>>> >>>> track: yes >>>> auto-renew: yes >>>> Request ID '20140528063953': >>>> status: MONITORING >>>> ca-error: Error setting up ccache for local "host" service >>>> using default keytab: Cannot contact any KDC for realm ''. >>>> stuck: no >>>> key pair storage: type=NSSDB,location='/etc/dirs >>>> rv/slapd-PKI-IPA',nickname='Server-Cert',token='NSS Certificate >>>> DB',pinfile='/etc/dirsrv/slapd-PKI-IPA/pwdfile.txt' >>>> certificate: type=NSSDB,location='/etc/dirs >>>> rv/slapd-PKI-IPA',nickname='Server-Cert',token='NSS Certificate DB' >>>> CA: IPA >>>> issuer: CN=Certificate Authority,O= >>>> subject: CN=,O= >>>> expires: 2016-05-28 06:39:52 UTC >>>> eku: id-kp-serverAuth,id-kp-clientAuth >>>> pre-save command: >>>> post-save command: /usr/lib64/ipa/certmonger/restart_dirsrv >>>> PKI-IPA >>>> track: yes >>>> auto-renew: yes >>>> Request ID '20140528064145': >>>> status: MONITORING >>>> ca-error: Error setting up ccache for local "host" service >>>> using default keytab: Cannot contact any KDC for realm ''. >>>> stuck: no >>>> key pair storage: type=NSSDB,location='/etc/http >>>> d/alias',nickname='Server-Cert',token='NSS Certificate >>>> DB',pinfile='/etc/httpd/alias/pwdfile.txt' >>>> certificate: type=NSSDB,location='/etc/http >>>> d/alias',nickname='Server-Cert',token='NSS Certificate DB' >>>> CA: IPA >>>> issuer: CN=Certificate Authority,O= >>>> subject: CN=,O= >>>> expires: 2016-05-28 06:41:44 UTC >>>> eku: id-kp-serverAuth,id-kp-clientAuth >>>> pre-save command: >>>> post-save command: /usr/lib64/ipa/certmonger/restart_httpd >>>> track: yes >>>> auto-renew: yes >>>> ### >>>> >>>> Best regards. >>>> >>>> Bahan >>>> >>>> On Wed, Sep 14, 2016 at 6:21 PM, Martin Basti >>>> wrote: >>>> >>>>> >>>>> Then you have to start services manually, I don't know if the same >>>>> steps will work with IPA 3.0.0, I don't remember, but you can try :) >>>>> >>>>> On 14.09.2016 18:18, bahan w wrote: >>>>> >>>>> Oh I forgot to add that my version of ipa is quite old : >>>>> ### >>>>> # rpm -qa | grep ipa-server >>>>> ipa-server-3.0.0-25.el6.x86_64 >>>>> ### >>>>> >>>>> When I try the command you gave me I got the following error : >>>>> ### >>>>> # ipactl start --force >>>>> Usage: ipactl start|stop|restart|status >>>>> >>>>> >>>>> ipactl: error: no such option: --force >>>>> ### >>>>> >>>>> Best regards. >>>>> >>>>> Bahan >>>>> >>>>> >>>>> On Wed, Sep 14, 2016 at 6:14 PM, Martin Basti >>>>> wrote: >>>>> >>>>>> >>>>>> >>>>>> On 14.09.2016 17:59, bahan w wrote: >>>>>> >>>>>> Hello ! >>>>>> >>>>>> I send you this mail because I cannot restart my test IPA server. >>>>>> >>>>>> When I try to start it with service ipa start, I got the following >>>>>> error message : >>>>>> ### >>>>>> # service ipa start >>>>>> Starting Directory Service >>>>>> Starting dirsrv: >>>>>> ...[14/Sep/2016:17:57:23 +0200] - SSL alert: >>>>>> CERT_VerifyCertificateNow: verify certificate failed for cert Server-Cert >>>>>> of family cn=RSA,cn=encryption,cn=config (Netscape Portable Runtime error >>>>>> -8181 - Peer's Certificate has expired.) >>>>>> [ OK ] >>>>>> PKI-IPA...[14/Sep/2016:17:57:33 +0200] - SSL alert: >>>>>> CERT_VerifyCertificateNow: verify certificate failed for cert Server-Cert >>>>>> of family cn=RSA,cn=encryption,cn=config (Netscape Portable Runtime error >>>>>> -8181 - Peer's Certificate has expired.) >>>>>> [ OK ] >>>>>> Starting KDC Service >>>>>> Starting Kerberos 5 KDC: [ OK ] >>>>>> Starting KPASSWD Service >>>>>> Starting Kerberos 5 Admin Server: [ OK ] >>>>>> Starting MEMCACHE Service >>>>>> Starting ipa_memcached: [ OK ] >>>>>> Starting HTTP Service >>>>>> Starting httpd: [FAILED] >>>>>> Failed to start HTTP Service >>>>>> Shutting down >>>>>> Stopping Kerberos 5 KDC: [ OK ] >>>>>> Stopping Kerberos 5 Admin Server: [ OK ] >>>>>> Stopping ipa_memcached: [ OK ] >>>>>> Stopping httpd: [FAILED] >>>>>> Stopping pki-ca: [ OK ] >>>>>> Shutting down dirsrv: >>>>>> ... [ OK ] >>>>>> PKI-IPA... [ OK ] >>>>>> Aborting ipactl >>>>>> >>>>>> # service ipa status >>>>>> Directory Service: STOPPED >>>>>> Failed to get list of services to probe status: >>>>>> Directory Server is stopped >>>>>> ### >>>>>> >>>>>> Do you know how to renew the SSL certificate used for the IPA Server ? >>>>>> >>>>>> Best regards. >>>>>> >>>>>> Bahan >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> Hello, >>>>>> >>>>>> please run >>>>>> >>>>>> # ipactl start --force >>>>>> # getcert list (to detect which certificate is outdated, I suspect DS >>>>>> cert (or to get more info why it has not been renewed)) >>>>>> >>>>>> If getcert does work (I'm not sure if ti is able to work without >>>>>> httpd), you probable need to move time back to past where cert is valid, >>>>>> start IPA and try again. >>>>>> >>>>>> Please find ID outdated certificate and try resubmit it (CA and DS >>>>>> must be running) >>>>>> >>>>>> # getcert resubmit -i 20160914122036 (use you ID :) ) >>>>>> >>>>>> This should renew cert, check status with getcert list >>>>>> >>>>>> Move time back to future (if needed) >>>>>> >>>>>> Try to restart IPA >>>>>> >>>>>> Martin^2 >>>>>> >>>>> >>>>> >>>>> >>>> >>> >> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From bahanw042014 at gmail.com Wed Sep 14 17:42:12 2016 From: bahanw042014 at gmail.com (bahan w) Date: Wed, 14 Sep 2016 19:42:12 +0200 Subject: [Freeipa-users] SSL alert: CERT_VerifyCertificateNow: verify certificate failed for cert Server-Cert of family In-Reply-To: References: <7d17ebb6-eba6-b560-bcf9-e016546095c1@redhat.com> <4a3d91a5-e48c-96cc-b1aa-6082eea2e302@redhat.com> Message-ID: Here is what I found : In the catalina.out : ### May 27, 2016 10:51:35 AM org.apache.catalina.core.StandardWrapperValve invoke SEVERE: Servlet.service() for servlet caDisplayBySerial-agent threw exception java.io.IOException: CS server is not ready to serve. at com.netscape.cms.servlet.base.CMSServlet.service(CMSServlet.java:441) at javax.servlet.http.HttpServlet.service(HttpServlet.java:717) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206) at com.netscape.cms.servlet.filter.AgentRequestFilter.doFilter(AgentRequestFilter.java:124) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206) at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233) at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191) at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127) at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102) at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109) at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:298) at org.apache.jk.server.JkCoyoteHandler.invoke(JkCoyoteHandler.java:190) at org.apache.jk.common.HandlerRequest.invoke(HandlerRequest.java:291) at org.apache.jk.common.ChannelSocket.invoke(ChannelSocket.java:769) at org.apache.jk.common.ChannelSocket.processConnection(ChannelSocket.java:698) at org.apache.jk.common.ChannelSocket$SocketConnection.runIt(ChannelSocket.java:891) at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:690) at java.lang.Thread.run(Thread.java:722) ### In the selftests.log in /var/log/pki-ca : ### 24196.main - [27/May/2016:10:50:27 CEST] [20] [1] SelfTestSubsystem: Initializing self test plugins: 24196.main - [27/May/2016:10:50:27 CEST] [20] [1] SelfTestSubsystem: loading all self test plugin logger parameters 24196.main - [27/May/2016:10:50:27 CEST] [20] [1] SelfTestSubsystem: loading all self test plugin instances 24196.main - [27/May/2016:10:50:27 CEST] [20] [1] SelfTestSubsystem: loading all self test plugin instance parameters 24196.main - [27/May/2016:10:50:27 CEST] [20] [1] SelfTestSubsystem: loading self test plugins in on-demand order 24196.main - [27/May/2016:10:50:27 CEST] [20] [1] SelfTestSubsystem: loading self test plugins in startup order 24196.main - [27/May/2016:10:50:27 CEST] [20] [1] SelfTestSubsystem: Self test plugins have been successfully loaded! 24196.main - [27/May/2016:10:50:28 CEST] [20] [1] SelfTestSubsystem: Running self test plugins specified to be executed at startup: 24196.main - [27/May/2016:10:50:28 CEST] [20] [1] CAPresence: CA is present 24196.main - [27/May/2016:10:50:28 CEST] [20] [1] SystemCertsVerification: system certs verification failure 24196.main - [27/May/2016:10:50:28 CEST] [20] [1] SelfTestSubsystem: The CRITICAL self test plugin called selftests.container.instance.SystemC ertsVerification running at startup FAILED! ### But nothing else. Best regards. Bahan On Wed, Sep 14, 2016 at 7:27 PM, bahan w wrote: > I tried also the following commands : > ### > # ipa cert-show 1 > ipa: ERROR: Certificate operation cannot be completed: Unable to > communicate with CMS (Not Found) > > # service ipa status > Directory Service: RUNNING > KDC Service: RUNNING > KPASSWD Service: RUNNING > MEMCACHE Service: RUNNING > HTTP Service: RUNNING > CA Service: RUNNING > ### > > I'm checking the /var/log/pki-ca logs to see if I find something. > > Best regards. > > Bahan > > On Wed, Sep 14, 2016 at 7:02 PM, bahan w wrote: > >> Sorry Martin, >> >> This is not the first time I forgot to add back freeipa users. >> I have problems with gmail, again sorry. >> >> Indeed I figured out that I had to restart the ipa server. >> So I tried to restart ipa server. >> But it was not working yet. >> >> So I thought it was maybe due to the configuration I performed in the >> nss.conf. >> So I rollbacked this conf and restarted ipa-server. >> Then I retried your commands but it is still the same error. >> >> ### >> Request ID '20140528064145': >> status: CA_UNREACHABLE >> ca-error: Server failed request, will retry: 4301 (RPC failed at >> server. Certificate operation cannot be completed: Unable to communicate >> with CMS (Not Found)). >> stuck: yes >> key pair storage: type=NSSDB,location='/etc/http >> d/alias',nickname='Server-Cert',token='NSS Certificate >> DB',pinfile='/etc/httpd/alias/pwdfile.txt' >> certificate: type=NSSDB,location='/etc/http >> d/alias',nickname='Server-Cert',token='NSS Certificate DB' >> CA: IPA >> issuer: CN=Certificate Authority,O= >> subject: CN=,O= >> expires: 2016-05-28 06:41:44 UTC >> eku: id-kp-serverAuth,id-kp-clientAuth >> pre-save command: >> post-save command: /usr/lib64/ipa/certmonger/restart_httpd >> track: yes >> auto-renew: yes >> ### >> >> Do you know what is the CMS ? >> ### >> (RPC failed at server. Certificate operation cannot be completed: Unable >> to communicate with CMS (Not Found)). >> ### >> >> Best regards. >> >> Bahan >> >> >> >> >> >> On Wed, Sep 14, 2016 at 6:46 PM, Martin Basti wrote: >> >>> did you restart IPA when you moved time? Is there are more detailed >>> error description in output of getcert list? >>> >>> On 14.09.2016 18:45, bahan w wrote: >>> >>> I set the date-time when the certificates were valid : >>> ### >>> # date -s '2016-05-27 10:00:00' >>> Fri May 27 10:00:00 CEST 2016 >>> >>> # date >>> Fri May 27 10:00:02 CEST 2016 >>> ### >>> >>> Then I try to renew them : >>> ### >>> # getcert resubmit -i 20140528063919 >>> Resubmitting "20140528063919" to "IPA". >>> >>> # getcert resubmit -i 20140528064145 >>> Resubmitting "20140528064145" to "IPA". >>> >>> # getcert resubmit -i 20140528063953 >>> Resubmitting "20140528063953" to "IPA". >>> ### >>> >>> But when I do the getcert list after, the result is the same. >>> >>> I guess it is because of this ? >>> CA_UNREACHABLE >>> >>> Any idea ? >>> >>> Best regards. >>> >>> Bahan >>> >>> On Wed, Sep 14, 2016 at 6:38 PM, bahan w wrote: >>> >>>> Ok, I managed to restart the IPA service by adding this line in the >>>> file /etc/httpd/conf.d/nss.conf : >>>> ### >>>> NSSEnforceValidCerts off >>>> ### >>>> >>>> But when I do the getcert now I got the following result : >>>> >>>> ### >>>> # getcert list >>>> Number of certificates and requests being tracked: 8. >>>> Request ID '20140528063903': >>>> status: MONITORING >>>> stuck: no >>>> key pair storage: type=NSSDB,location='/var/lib/ >>>> pki-ca/alias',nickname='auditSigningCert cert-pki-ca',token='NSS >>>> Certificate DB',pin='159203530658' >>>> certificate: type=NSSDB,location='/var/lib/ >>>> pki-ca/alias',nickname='auditSigningCert cert-pki-ca',token='NSS >>>> Certificate DB' >>>> CA: dogtag-ipa-renew-agent >>>> issuer: CN=Certificate Authority,O= >>>> subject: CN=CA Audit,O= >>>> expires: 2018-04-09 11:39:16 UTC >>>> pre-save command: /usr/lib64/ipa/certmonger/stop_pkicad >>>> post-save command: /usr/lib64/ipa/certmonger/renew_ca_cert >>>> "auditSigningCert cert-pki-ca" >>>> track: yes >>>> auto-renew: yes >>>> Request ID '20140528063904': >>>> status: MONITORING >>>> stuck: no >>>> key pair storage: type=NSSDB,location='/var/lib/ >>>> pki-ca/alias',nickname='ocspSigningCert cert-pki-ca',token='NSS >>>> Certificate DB',pin='159203530658' >>>> certificate: type=NSSDB,location='/var/lib/ >>>> pki-ca/alias',nickname='ocspSigningCert cert-pki-ca',token='NSS >>>> Certificate DB' >>>> CA: dogtag-ipa-renew-agent >>>> issuer: CN=Certificate Authority,O= >>>> subject: CN=OCSP Subsystem,O= >>>> expires: 2018-04-09 11:38:16 UTC >>>> eku: id-kp-OCSPSigning >>>> pre-save command: /usr/lib64/ipa/certmonger/stop_pkicad >>>> post-save command: /usr/lib64/ipa/certmonger/renew_ca_cert >>>> "ocspSigningCert cert-pki-ca" >>>> track: yes >>>> auto-renew: yes >>>> Request ID '20140528063905': >>>> status: MONITORING >>>> stuck: no >>>> key pair storage: type=NSSDB,location='/var/lib/ >>>> pki-ca/alias',nickname='subsystemCert cert-pki-ca',token='NSS >>>> Certificate DB',pin='159203530658' >>>> certificate: type=NSSDB,location='/var/lib/ >>>> pki-ca/alias',nickname='subsystemCert cert-pki-ca',token='NSS >>>> Certificate DB' >>>> CA: dogtag-ipa-renew-agent >>>> issuer: CN=Certificate Authority,O= >>>> subject: CN=CA Subsystem,O= >>>> expires: 2018-04-09 11:38:16 UTC >>>> eku: id-kp-serverAuth,id-kp-clientAuth >>>> pre-save command: /usr/lib64/ipa/certmonger/stop_pkicad >>>> post-save command: /usr/lib64/ipa/certmonger/renew_ca_cert >>>> "subsystemCert cert-pki-ca" >>>> track: yes >>>> auto-renew: yes >>>> Request ID '20140528063906': >>>> status: MONITORING >>>> stuck: no >>>> key pair storage: type=NSSDB,location='/etc/http >>>> d/alias',nickname='ipaCert',token='NSS Certificate >>>> DB',pinfile='/etc/httpd/alias/pwdfile.txt' >>>> certificate: type=NSSDB,location='/etc/http >>>> d/alias',nickname='ipaCert',token='NSS Certificate DB' >>>> CA: dogtag-ipa-renew-agent >>>> issuer: CN=Certificate Authority,O= >>>> subject: CN=IPA RA,O= >>>> expires: 2018-04-09 11:38:16 UTC >>>> eku: id-kp-serverAuth,id-kp-clientAuth >>>> pre-save command: >>>> post-save command: /usr/lib64/ipa/certmonger/renew_ra_cert >>>> track: yes >>>> auto-renew: yes >>>> Request ID '20140528063907': >>>> status: MONITORING >>>> stuck: no >>>> key pair storage: type=NSSDB,location='/var/lib/ >>>> pki-ca/alias',nickname='Server-Cert cert-pki-ca',token='NSS >>>> Certificate DB',pin='159203530658' >>>> certificate: type=NSSDB,location='/var/lib/ >>>> pki-ca/alias',nickname='Server-Cert cert-pki-ca',token='NSS >>>> Certificate DB' >>>> CA: dogtag-ipa-renew-agent >>>> issuer: CN=Certificate Authority,O= >>>> subject: CN=,O= >>>> expires: 2018-04-09 11:38:16 UTC >>>> eku: id-kp-serverAuth,id-kp-clientAuth >>>> pre-save command: >>>> post-save command: >>>> track: yes >>>> auto-renew: yes >>>> Request ID '20140528063919': >>>> status: CA_UNREACHABLE >>>> ca-error: Server failed request, will retry: -504 (libcurl >>>> failed to execute the HTTP POST transaction. Peer certificate cannot be >>>> authenticated with known CA certificates). >>>> stuck: yes >>>> key pair storage: type=NSSDB,location='/etc/dirs >>>> rv/slapd-',nickname='Server-Cert',token='NSS Certificate >>>> DB',pinfile='/etc/dirsrv/slapd-/pwdfile.txt' >>>> certificate: type=NSSDB,location='/etc/dirs >>>> rv/slapd-',nickname='Server-Cert',token='NSS Certificate DB' >>>> CA: IPA >>>> issuer: CN=Certificate Authority,O= >>>> subject: CN=,O= >>>> expires: 2016-05-28 06:39:18 UTC >>>> eku: id-kp-serverAuth,id-kp-clientAuth >>>> pre-save command: >>>> post-save command: /usr/lib64/ipa/certmonger/restart_dirsrv >>>> >>>> track: yes >>>> auto-renew: yes >>>> Request ID '20140528063953': >>>> status: CA_UNREACHABLE >>>> ca-error: Server failed request, will retry: -504 (libcurl >>>> failed to execute the HTTP POST transaction. Peer certificate cannot be >>>> authenticated with known CA certificates). >>>> stuck: yes >>>> key pair storage: type=NSSDB,location='/etc/dirs >>>> rv/slapd-PKI-IPA',nickname='Server-Cert',token='NSS Certificate >>>> DB',pinfile='/etc/dirsrv/slapd-PKI-IPA/pwdfile.txt' >>>> certificate: type=NSSDB,location='/etc/dirs >>>> rv/slapd-PKI-IPA',nickname='Server-Cert',token='NSS Certificate DB' >>>> CA: IPA >>>> issuer: CN=Certificate Authority,O= >>>> subject: CN=,O= >>>> expires: 2016-05-28 06:39:52 UTC >>>> eku: id-kp-serverAuth,id-kp-clientAuth >>>> pre-save command: >>>> post-save command: /usr/lib64/ipa/certmonger/restart_dirsrv >>>> PKI-IPA >>>> track: yes >>>> auto-renew: yes >>>> Request ID '20140528064145': >>>> status: CA_UNREACHABLE >>>> ca-error: Server failed request, will retry: -504 (libcurl >>>> failed to execute the HTTP POST transaction. Peer certificate cannot be >>>> authenticated with known CA certificates). >>>> stuck: yes >>>> key pair storage: type=NSSDB,location='/etc/http >>>> d/alias',nickname='Server-Cert',token='NSS Certificate >>>> DB',pinfile='/etc/httpd/alias/pwdfile.txt' >>>> certificate: type=NSSDB,location='/etc/http >>>> d/alias',nickname='Server-Cert',token='NSS Certificate DB' >>>> CA: IPA >>>> issuer: CN=Certificate Authority,O= >>>> subject: CN=,O= >>>> expires: 2016-05-28 06:41:44 UTC >>>> eku: id-kp-serverAuth,id-kp-clientAuth >>>> pre-save command: >>>> post-save command: /usr/lib64/ipa/certmonger/restart_httpd >>>> track: yes >>>> auto-renew: yes >>>> ### >>>> >>>> Indeed, the entries outdated are the following : >>>> - for /etc/dirsrv/slapd- : 20140528063919 >>>> - for /etc/dirsrv/slapd-PKI-IPA : 20140528063953 >>>> - for httpd ? : 20140528064145 >>>> >>>> Best regards. >>>> >>>> Bahan >>>> >>>> On Wed, Sep 14, 2016 at 6:28 PM, bahan w >>>> wrote: >>>> >>>>> Ok :D >>>>> >>>>> Because to perform the getcert list command, I need to have all the >>>>> ipa services running right ? >>>>> >>>>> Here is the result of the command with the ipa services down. >>>>> ### >>>>> # getcert list >>>>> Number of certificates and requests being tracked: 8. >>>>> Request ID '20140528063903': >>>>> status: MONITORING >>>>> stuck: no >>>>> key pair storage: type=NSSDB,location='/var/lib/ >>>>> pki-ca/alias',nickname='auditSigningCert cert-pki-ca',token='NSS >>>>> Certificate DB',pin='159203530658' >>>>> certificate: type=NSSDB,location='/var/lib/ >>>>> pki-ca/alias',nickname='auditSigningCert cert-pki-ca',token='NSS >>>>> Certificate DB' >>>>> CA: dogtag-ipa-renew-agent >>>>> issuer: CN=Certificate Authority,O= >>>>> subject: CN=CA Audit,O= >>>>> expires: 2018-04-09 11:39:16 UTC >>>>> pre-save command: /usr/lib64/ipa/certmonger/stop_pkicad >>>>> post-save command: /usr/lib64/ipa/certmonger/renew_ca_cert >>>>> "auditSigningCert cert-pki-ca" >>>>> track: yes >>>>> auto-renew: yes >>>>> Request ID '20140528063904': >>>>> status: MONITORING >>>>> stuck: no >>>>> key pair storage: type=NSSDB,location='/var/lib/ >>>>> pki-ca/alias',nickname='ocspSigningCert cert-pki-ca',token='NSS >>>>> Certificate DB',pin='159203530658' >>>>> certificate: type=NSSDB,location='/var/lib/ >>>>> pki-ca/alias',nickname='ocspSigningCert cert-pki-ca',token='NSS >>>>> Certificate DB' >>>>> CA: dogtag-ipa-renew-agent >>>>> issuer: CN=Certificate Authority,O= >>>>> subject: CN=OCSP Subsystem,O= >>>>> expires: 2018-04-09 11:38:16 UTC >>>>> eku: id-kp-OCSPSigning >>>>> pre-save command: /usr/lib64/ipa/certmonger/stop_pkicad >>>>> post-save command: /usr/lib64/ipa/certmonger/renew_ca_cert >>>>> "ocspSigningCert cert-pki-ca" >>>>> track: yes >>>>> auto-renew: yes >>>>> Request ID '20140528063905': >>>>> status: MONITORING >>>>> stuck: no >>>>> key pair storage: type=NSSDB,location='/var/lib/ >>>>> pki-ca/alias',nickname='subsystemCert cert-pki-ca',token='NSS >>>>> Certificate DB',pin='159203530658' >>>>> certificate: type=NSSDB,location='/var/lib/ >>>>> pki-ca/alias',nickname='subsystemCert cert-pki-ca',token='NSS >>>>> Certificate DB' >>>>> CA: dogtag-ipa-renew-agent >>>>> issuer: CN=Certificate Authority,O= >>>>> subject: CN=CA Subsystem,O= >>>>> expires: 2018-04-09 11:38:16 UTC >>>>> eku: id-kp-serverAuth,id-kp-clientAuth >>>>> pre-save command: /usr/lib64/ipa/certmonger/stop_pkicad >>>>> post-save command: /usr/lib64/ipa/certmonger/renew_ca_cert >>>>> "subsystemCert cert-pki-ca" >>>>> track: yes >>>>> auto-renew: yes >>>>> Request ID '20140528063906': >>>>> status: MONITORING >>>>> stuck: no >>>>> key pair storage: type=NSSDB,location='/etc/http >>>>> d/alias',nickname='ipaCert',token='NSS Certificate >>>>> DB',pinfile='/etc/httpd/alias/pwdfile.txt' >>>>> certificate: type=NSSDB,location='/etc/http >>>>> d/alias',nickname='ipaCert',token='NSS Certificate DB' >>>>> CA: dogtag-ipa-renew-agent >>>>> issuer: CN=Certificate Authority,O= >>>>> subject: CN=IPA RA,O= >>>>> expires: 2018-04-09 11:38:16 UTC >>>>> eku: id-kp-serverAuth,id-kp-clientAuth >>>>> pre-save command: >>>>> post-save command: /usr/lib64/ipa/certmonger/renew_ra_cert >>>>> track: yes >>>>> auto-renew: yes >>>>> Request ID '20140528063907': >>>>> status: MONITORING >>>>> stuck: no >>>>> key pair storage: type=NSSDB,location='/var/lib/ >>>>> pki-ca/alias',nickname='Server-Cert cert-pki-ca',token='NSS >>>>> Certificate DB',pin='159203530658' >>>>> certificate: type=NSSDB,location='/var/lib/ >>>>> pki-ca/alias',nickname='Server-Cert cert-pki-ca',token='NSS >>>>> Certificate DB' >>>>> CA: dogtag-ipa-renew-agent >>>>> issuer: CN=Certificate Authority,O= >>>>> subject: CN=,O= >>>>> expires: 2018-04-09 11:38:16 UTC >>>>> eku: id-kp-serverAuth,id-kp-clientAuth >>>>> pre-save command: >>>>> post-save command: >>>>> track: yes >>>>> auto-renew: yes >>>>> Request ID '20140528063919': >>>>> status: MONITORING >>>>> ca-error: Error setting up ccache for local "host" service >>>>> using default keytab: Cannot contact any KDC for realm ''. >>>>> stuck: no >>>>> key pair storage: type=NSSDB,location='/etc/dirs >>>>> rv/slapd-',nickname='Server-Cert',token='NSS Certificate >>>>> DB',pinfile='/etc/dirsrv/slapd-/pwdfile.txt' >>>>> certificate: type=NSSDB,location='/etc/dirs >>>>> rv/slapd-',nickname='Server-Cert',token='NSS Certificate DB' >>>>> CA: IPA >>>>> issuer: CN=Certificate Authority,O= >>>>> subject: CN=,O= >>>>> expires: 2016-05-28 06:39:18 UTC >>>>> eku: id-kp-serverAuth,id-kp-clientAuth >>>>> pre-save command: >>>>> post-save command: /usr/lib64/ipa/certmonger/restart_dirsrv >>>>> >>>>> track: yes >>>>> auto-renew: yes >>>>> Request ID '20140528063953': >>>>> status: MONITORING >>>>> ca-error: Error setting up ccache for local "host" service >>>>> using default keytab: Cannot contact any KDC for realm ''. >>>>> stuck: no >>>>> key pair storage: type=NSSDB,location='/etc/dirs >>>>> rv/slapd-PKI-IPA',nickname='Server-Cert',token='NSS Certificate >>>>> DB',pinfile='/etc/dirsrv/slapd-PKI-IPA/pwdfile.txt' >>>>> certificate: type=NSSDB,location='/etc/dirs >>>>> rv/slapd-PKI-IPA',nickname='Server-Cert',token='NSS Certificate DB' >>>>> CA: IPA >>>>> issuer: CN=Certificate Authority,O= >>>>> subject: CN=,O= >>>>> expires: 2016-05-28 06:39:52 UTC >>>>> eku: id-kp-serverAuth,id-kp-clientAuth >>>>> pre-save command: >>>>> post-save command: /usr/lib64/ipa/certmonger/restart_dirsrv >>>>> PKI-IPA >>>>> track: yes >>>>> auto-renew: yes >>>>> Request ID '20140528064145': >>>>> status: MONITORING >>>>> ca-error: Error setting up ccache for local "host" service >>>>> using default keytab: Cannot contact any KDC for realm ''. >>>>> stuck: no >>>>> key pair storage: type=NSSDB,location='/etc/http >>>>> d/alias',nickname='Server-Cert',token='NSS Certificate >>>>> DB',pinfile='/etc/httpd/alias/pwdfile.txt' >>>>> certificate: type=NSSDB,location='/etc/http >>>>> d/alias',nickname='Server-Cert',token='NSS Certificate DB' >>>>> CA: IPA >>>>> issuer: CN=Certificate Authority,O= >>>>> subject: CN=,O= >>>>> expires: 2016-05-28 06:41:44 UTC >>>>> eku: id-kp-serverAuth,id-kp-clientAuth >>>>> pre-save command: >>>>> post-save command: /usr/lib64/ipa/certmonger/restart_httpd >>>>> track: yes >>>>> auto-renew: yes >>>>> ### >>>>> >>>>> Best regards. >>>>> >>>>> Bahan >>>>> >>>>> On Wed, Sep 14, 2016 at 6:21 PM, Martin Basti >>>>> wrote: >>>>> >>>>>> >>>>>> Then you have to start services manually, I don't know if the same >>>>>> steps will work with IPA 3.0.0, I don't remember, but you can try :) >>>>>> >>>>>> On 14.09.2016 18:18, bahan w wrote: >>>>>> >>>>>> Oh I forgot to add that my version of ipa is quite old : >>>>>> ### >>>>>> # rpm -qa | grep ipa-server >>>>>> ipa-server-3.0.0-25.el6.x86_64 >>>>>> ### >>>>>> >>>>>> When I try the command you gave me I got the following error : >>>>>> ### >>>>>> # ipactl start --force >>>>>> Usage: ipactl start|stop|restart|status >>>>>> >>>>>> >>>>>> ipactl: error: no such option: --force >>>>>> ### >>>>>> >>>>>> Best regards. >>>>>> >>>>>> Bahan >>>>>> >>>>>> >>>>>> On Wed, Sep 14, 2016 at 6:14 PM, Martin Basti >>>>>> wrote: >>>>>> >>>>>>> >>>>>>> >>>>>>> On 14.09.2016 17:59, bahan w wrote: >>>>>>> >>>>>>> Hello ! >>>>>>> >>>>>>> I send you this mail because I cannot restart my test IPA server. >>>>>>> >>>>>>> When I try to start it with service ipa start, I got the following >>>>>>> error message : >>>>>>> ### >>>>>>> # service ipa start >>>>>>> Starting Directory Service >>>>>>> Starting dirsrv: >>>>>>> ...[14/Sep/2016:17:57:23 +0200] - SSL alert: >>>>>>> CERT_VerifyCertificateNow: verify certificate failed for cert Server-Cert >>>>>>> of family cn=RSA,cn=encryption,cn=config (Netscape Portable Runtime error >>>>>>> -8181 - Peer's Certificate has expired.) >>>>>>> [ OK ] >>>>>>> PKI-IPA...[14/Sep/2016:17:57:33 +0200] - SSL alert: >>>>>>> CERT_VerifyCertificateNow: verify certificate failed for cert Server-Cert >>>>>>> of family cn=RSA,cn=encryption,cn=config (Netscape Portable Runtime error >>>>>>> -8181 - Peer's Certificate has expired.) >>>>>>> [ OK ] >>>>>>> Starting KDC Service >>>>>>> Starting Kerberos 5 KDC: [ OK ] >>>>>>> Starting KPASSWD Service >>>>>>> Starting Kerberos 5 Admin Server: [ OK ] >>>>>>> Starting MEMCACHE Service >>>>>>> Starting ipa_memcached: [ OK ] >>>>>>> Starting HTTP Service >>>>>>> Starting httpd: [FAILED] >>>>>>> Failed to start HTTP Service >>>>>>> Shutting down >>>>>>> Stopping Kerberos 5 KDC: [ OK ] >>>>>>> Stopping Kerberos 5 Admin Server: [ OK ] >>>>>>> Stopping ipa_memcached: [ OK ] >>>>>>> Stopping httpd: [FAILED] >>>>>>> Stopping pki-ca: [ OK ] >>>>>>> Shutting down dirsrv: >>>>>>> ... [ OK ] >>>>>>> PKI-IPA... [ OK ] >>>>>>> Aborting ipactl >>>>>>> >>>>>>> # service ipa status >>>>>>> Directory Service: STOPPED >>>>>>> Failed to get list of services to probe status: >>>>>>> Directory Server is stopped >>>>>>> ### >>>>>>> >>>>>>> Do you know how to renew the SSL certificate used for the IPA Server >>>>>>> ? >>>>>>> >>>>>>> Best regards. >>>>>>> >>>>>>> Bahan >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> Hello, >>>>>>> >>>>>>> please run >>>>>>> >>>>>>> # ipactl start --force >>>>>>> # getcert list (to detect which certificate is outdated, I suspect >>>>>>> DS cert (or to get more info why it has not been renewed)) >>>>>>> >>>>>>> If getcert does work (I'm not sure if ti is able to work without >>>>>>> httpd), you probable need to move time back to past where cert is valid, >>>>>>> start IPA and try again. >>>>>>> >>>>>>> Please find ID outdated certificate and try resubmit it (CA and DS >>>>>>> must be running) >>>>>>> >>>>>>> # getcert resubmit -i 20160914122036 (use you ID :) ) >>>>>>> >>>>>>> This should renew cert, check status with getcert list >>>>>>> >>>>>>> Move time back to future (if needed) >>>>>>> >>>>>>> Try to restart IPA >>>>>>> >>>>>>> Martin^2 >>>>>>> >>>>>> >>>>>> >>>>>> >>>>> >>>> >>> >>> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From lslebodn at redhat.com Wed Sep 14 17:45:38 2016 From: lslebodn at redhat.com (Lukas Slebodnik) Date: Wed, 14 Sep 2016 19:45:38 +0200 Subject: [Freeipa-users] About AllowGroups with sshd In-Reply-To: <09d201d20e95$7cbcce30$76366a90$@cyberfuel.com> References: <0afb01d20d0e$d99b9100$8cd2b300$@cyberfuel.com> <20160913082139.gishru3hcpzcpclt@hendrix> <09d201d20e95$7cbcce30$76366a90$@cyberfuel.com> Message-ID: <20160914174538.GA23179@10.4.128.1> On (14/09/16 08:37), Jose Alvarez R. wrote: >Hi Jakub > >Thanks for your response. It's an option, but my backups servers I will not >add to the FreeIPA server. > >Then, I cannot use the option HBAC, because I want my backup server can >connect with root to some client server of my FreeIPA Server. > root is not handled by sssd/freeIPA. It is a local user; and thus access cannot be denied by HBAC. LS From sfrancisco at muxi.com.br Wed Sep 14 21:19:01 2016 From: sfrancisco at muxi.com.br (Sergio Francisco) Date: Wed, 14 Sep 2016 18:19:01 -0300 Subject: [Freeipa-users] How to make a FreeIPA node replica become Master? Message-ID: Hi, We have a deployment of FreeIPA using 3 nodes (Master with more 2 replicas). Recently, the master node had a problem with the process 'ns-slapd' consuming 100% of CPU. During this problem, DNS service wasn't working, IPA admin UI encountered timeout, SSH keys to access the hosts are not being loaded correctly. We observed in the logs of "dirsrv" that something related to the cachesize wasn't enough to the space needed and then ns-slapd started a process to recover it. We let the server running this operation almost one day and nothing happened. Today, we tried to: 1 - remove the failed server from the deployment, using the command below, but unfortunately, it wasn't possible to do from both the 2 other nodes. ipa-replica-manage del --force mux-idm-p03.muxi.dc --cacert=/etc/ipa/ca.crt unexpected error: cannot connect to 'ldaps://localhost.localdomain:636 2 - tried to upgrade the failed server to a most recent version of IPA using ipa-server-upgrade but it stopped in the step to connect [5/10]: starting directory server 2016-09-14T13:43:28Z ERROR IPA server upgrade failed: Inspect /var/log/ipaupgrade.log and run command ipa-server-upgrade manually. 2016-09-14T13:43:28Z DEBUG The ipa-server-upgrade command failed, exception: error: [Errno 111] Connection refused 2016-09-14T13:43:28Z ERROR [Errno 111] Connection refused 3 - tried to recover the 389-ds database with the command "db_recover -f -v" but nothing happened. 4 - visited similar threads but none of them helped me https://www.redhat.com/archives/freeipa-users/2013-May/msg00015.html https://www.redhat.com/archives/freeipa-users/2015-July/msg00188.html 5 - as we need to urgently recover the service, we tried to rebuild the failed server, removing and reinstalling all the packages needed by ipa-server (yum install ipa-server bind bind-dyndb-ldap ipa-server-dns) and tried to re-join the new server as a replica to receive all the data again, but it doesn't seems to work. The other nodes are working well, resolving DNS requests, allowing users to access the servers using SSH, etc. Any ideas of what I can do to rebuild the server? Versions ipa-server-4.2.0-15.0.1.el7.centos.19.x86_64 ipa-server-dns-4.2.0-15.0.1.el7.centos.19.x86_64 389-ds-base-1.3.4.0-33.el7_2.x86_64 CentOS Linux release 7.2.1511 (Core) -------------- next part -------------- An HTML attachment was scrubbed... URL: From venkataramana.kintali at gmail.com Thu Sep 15 06:56:13 2016 From: venkataramana.kintali at gmail.com (Venkataramana Kintali) Date: Thu, 15 Sep 2016 09:56:13 +0300 Subject: [Freeipa-users] Issues with FreeIPA SSH Key authentication Message-ID: Hi Lukas, Thank you for responding. I compared the configs.(sshd_config and sssd.conf ),they are same. sssd and sshd services are running on all the servers(IPA clients). PubKey Authentication is enabled on all the servers. I am not able to login with sshkeys. But I am able to ssh to these servers from the other IPA clients I am able to connect to with ssh keys(after doing a kinit). Thanks Venkataramana On Fri, Sep 9, 2016 at 1:22 PM, Lukas Slebodnik wrote: > On (07/09/16 17:39), Venkataramana Kintali wrote: > >Hi, > >Of late, I am learning FreeIPA . I have installed IPA server and few > >clients (Version 3.0.0) > >I am facing an issue with ssh key authentication in my setup. > >I generated a putty ssh private key (using putty keygen) ,and uploaded it > >under a user through IPA GUI. > I assume you uploaded public key to the IPA > otherwise you did something wrong and I wonder why it works on some > machines. > > >I am able to login to some IPA clients but not able to login to other IPA > >clients with putty using private key and passphrase. > > > Is sssd_ssh running on all clients? (Is sssd.conf almost the same on all > machines) > Is sshd configuration the same on all machines? > /etc/ssh/ssh_config /etc/ssh/sshd_config > > >Public Key Authentication is enabled on all clients. > >I am able to from one client to other clients successfully (after doing > >kinit) without promting password. > > > >Can someone please throw some light on this as to what the issue could be > >here and what else I can check to understand where the problem is ? > > > >I searched this online but couldn't find any solution in the context of > IPA. > > > > LS > -------------- next part -------------- An HTML attachment was scrubbed... URL: From natxo.asenjo at gmail.com Thu Sep 15 07:04:07 2016 From: natxo.asenjo at gmail.com (Natxo Asenjo) Date: Thu, 15 Sep 2016 09:04:07 +0200 Subject: [Freeipa-users] ipa: ERROR: Certificate format error: (SEC_ERROR_LEGACY_DATABASE) The certificate/key database is in an old, unsupported format. In-Reply-To: References: <57D015D8.7080507@redhat.com> <57D166B9.3070201@redhat.com> <57D70676.8080107@redhat.com> <57D81AA1.4050400@redhat.com> Message-ID: Hi Ben, On Wed, Sep 14, 2016 at 2:45 PM, Ben Lipton wrote: One other note - this could be a permissions issue. NSS seems to produce > this confusing error message when it can't access the database, even if the > format of the database is actually fine. > > $ sudo chown root:root /tmp/certs > $ certutil -N -d /tmp/certs > certutil: function failed: SEC_ERROR_LEGACY_DATABASE: The certificate/key > database is in an old, unsupported format. > Thanks for the tip. What directory should I check? I have checked: [root at kdc01 httpd]$ ls -ltrZ /etc/httpd/alias/ -rw-r-----. root apache unconfined_u:object_r:cert_t:s0 secmod.db.orig -rw-r-----. root apache unconfined_u:object_r:cert_t:s0 key3.db.orig -rw-r-----. root apache unconfined_u:object_r:cert_t:s0 cert8.db.orig -rw-------. root root unconfined_u:object_r:cert_t:s0 install.log -rw-rw----. root apache unconfined_u:object_r:cert_t:s0 pwdfile.txt -rw-rw----. root apache unconfined_u:object_r:cert_t:s0 secmod.db -r--r--r--. root root unconfined_u:object_r:cert_t:s0 cacert.asc.orig -r--r--r--. root root unconfined_u:object_r:cert_t:s0 cacert.asc lrwxrwxrwx. root root system_u:object_r:cert_t:s0 libnssckbi.so -> ../../..//usr/lib/libnssckbi.so -rw-rw----. root apache unconfined_u:object_r:cert_t:s0 key3.db -rw-rw----. root apache unconfined_u:object_r:cert_t:s0 cert8.db [root at kdc01 httpd]$ ls -ltrdZ /etc/httpd/alias/ drwxr-xr-x. root root system_u:object_r:cert_t:s0 /etc/httpd/alias/ Those seem ok. -- Groeten, natxo -------------- next part -------------- An HTML attachment was scrubbed... URL: From lslebodn at redhat.com Thu Sep 15 07:09:49 2016 From: lslebodn at redhat.com (Lukas Slebodnik) Date: Thu, 15 Sep 2016 09:09:49 +0200 Subject: [Freeipa-users] Issues with FreeIPA SSH Key authentication In-Reply-To: References: Message-ID: <20160915070948.GA1485@10.4.128.1> On (15/09/16 09:56), Venkataramana Kintali wrote: >Hi Lukas, >Thank you for responding. >I compared the configs.(sshd_config and sssd.conf ),they are same. Is /etc/ssh/ssh_config the same as well? NOTE: (ssh_config is not the same as sshd_config //extra 'd' in name) >sssd and sshd services are running on all the servers(IPA clients). >PubKey Authentication is enabled on all the servers. >I am not able to login with sshkeys. > >But I am able to ssh to these servers from the other IPA clients I am able >to connect to with ssh keys(after doing a kinit). > If I remeber correctly GSSAPI has higher priority then public keys. So the behaviour is expected. You should decide whether you want to authenticate with ssh keys stored in IPA or with kerberos ticket (GSSAPI) or you can change sshd configuration to allow only authentication with public keys. LS From dkupka at redhat.com Thu Sep 15 07:40:37 2016 From: dkupka at redhat.com (David Kupka) Date: Thu, 15 Sep 2016 09:40:37 +0200 Subject: [Freeipa-users] How to make a FreeIPA node replica become Master? In-Reply-To: References: Message-ID: On 14/09/16 23:19, Sergio Francisco wrote: > Hi, > We have a deployment of FreeIPA using 3 nodes (Master with more 2 replicas). > > Recently, the master node had a problem with the process 'ns-slapd' > consuming 100% of CPU. During this problem, DNS service wasn't working, IPA > admin UI encountered timeout, SSH keys to access the hosts are not being > loaded correctly. > > We observed in the logs of "dirsrv" that something related to the cachesize > wasn't enough to the space needed and then ns-slapd started a process to > recover it. We let the server running this operation almost one day and > nothing happened. > > Today, we tried to: > > 1 - remove the failed server from the deployment, using the command below, > but unfortunately, it wasn't possible to do from both the 2 other nodes. > > ipa-replica-manage del --force mux-idm-p03.muxi.dc --cacert=/etc/ipa/ca.crt > unexpected error: cannot connect to 'ldaps://localhost.localdomain:636 > > 2 - tried to upgrade the failed server to a most recent version of IPA > using ipa-server-upgrade but it stopped in the step to connect > > [5/10]: starting directory server > > 2016-09-14T13:43:28Z ERROR IPA server upgrade failed: Inspect > /var/log/ipaupgrade.log and run command ipa-server-upgrade manually. > 2016-09-14T13:43:28Z DEBUG The ipa-server-upgrade command failed, > exception: error: [Errno 111] Connection refused > 2016-09-14T13:43:28Z ERROR [Errno 111] Connection refused > > 3 - tried to recover the 389-ds database with the command "db_recover -f > -v" but nothing happened. > 4 - visited similar threads but none of them helped me > > https://www.redhat.com/archives/freeipa-users/2013-May/msg00015.html > https://www.redhat.com/archives/freeipa-users/2015-July/msg00188.html > > 5 - as we need to urgently recover the service, we tried to rebuild the > failed server, removing and reinstalling all the packages needed by > ipa-server (yum install ipa-server bind bind-dyndb-ldap ipa-server-dns) and > tried to re-join the new server as a replica to receive all the data again, > but it doesn't seems to work. > > The other nodes are working well, resolving DNS requests, allowing users to > access the servers using SSH, etc. > > Any ideas of what I can do to rebuild the server? > > Versions > ipa-server-4.2.0-15.0.1.el7.centos.19.x86_64 > ipa-server-dns-4.2.0-15.0.1.el7.centos.19.x86_64 > 389-ds-base-1.3.4.0-33.el7_2.x86_64 > CentOS Linux release 7.2.1511 (Core) > > > Hi Sergio, first of all the terms master and replica are misleading. All FreeIPA servers are masters because the backends (389-ds) are configured to maintain multi-master replication. The difference between masters may be in services (CA, DNS, KRA, AD Trust, ...) that was configured on particular master but the data are synchronized among all masters. Looking on the steps you've done it would be best to create new master as a replica of one of the existing masters. Then you will probably need to enable CRL generating on some master because this can be enable only on one master and by default is enabled on first master that is installed with CA. Here you can find more information and how to: https://www.freeipa.org/page/Howto/Promote_CA_to_Renewal_and_CRL_Master HTH, -- David Kupka From natxo.asenjo at gmail.com Thu Sep 15 08:36:12 2016 From: natxo.asenjo at gmail.com (Natxo Asenjo) Date: Thu, 15 Sep 2016 10:36:12 +0200 Subject: [Freeipa-users] adding replica centos 7 to centos 6 fails [error] ObjectclassViolation: attribute "unhashed#user#password" not allowed In-Reply-To: References: Message-ID: hi, the fact the the usercertificate attribute of uid=admin,ou=people,o=ipaca is expired could this be the cause of these problems as well? How can I renew this certificate? -- Groeten, natxo -------------- next part -------------- An HTML attachment was scrubbed... URL: From venkataramana.kintali at gmail.com Thu Sep 15 08:46:03 2016 From: venkataramana.kintali at gmail.com (Venkataramana Kintali) Date: Thu, 15 Sep 2016 11:46:03 +0300 Subject: [Freeipa-users] Issues with FreeIPA SSH Key authentication In-Reply-To: <20160915070948.GA1485@10.4.128.1> References: <20160915070948.GA1485@10.4.128.1> Message-ID: Hi Lukas, ssh_config is also same on all servers. Our need is to do it both ways, to be able to login with ssh public keys(uploaded in IPA) and disable password login, and be able to access allhosts within the same IPA domain silently from any host. Hoping the configs will help, I am including the configurations here. ssh_config file : http://pastebin.com/MWHyH1Qw sshd_config file: http://pastebin.com/gpn5XhXM sssd_config file: http://pastebin.com/5Pby6xKp I just used some placeholders for sssd_config file in pastebin instead of actual values. Thanks Venkataramana On Thu, Sep 15, 2016 at 10:09 AM, Lukas Slebodnik wrote: > On (15/09/16 09:56), Venkataramana Kintali wrote: > >Hi Lukas, > >Thank you for responding. > >I compared the configs.(sshd_config and sssd.conf ),they are same. > Is /etc/ssh/ssh_config the same as well? > NOTE: (ssh_config is not the same as sshd_config //extra 'd' in name) > > >sssd and sshd services are running on all the servers(IPA clients). > >PubKey Authentication is enabled on all the servers. > >I am not able to login with sshkeys. > > > >But I am able to ssh to these servers from the other IPA clients I am able > >to connect to with ssh keys(after doing a kinit). > > > If I remeber correctly GSSAPI has higher priority then public keys. > So the behaviour is expected. > > You should decide whether you want to authenticate > with ssh keys stored in IPA or with kerberos ticket (GSSAPI) > or you can change sshd configuration to allow only authentication > with public keys. > > LS > -------------- next part -------------- An HTML attachment was scrubbed... URL: From natxo.asenjo at gmail.com Thu Sep 15 09:29:22 2016 From: natxo.asenjo at gmail.com (Natxo Asenjo) Date: Thu, 15 Sep 2016 11:29:22 +0200 Subject: [Freeipa-users] certificates not renewing CA_UNREACHEABLE Message-ID: hi, one of our master servers has a problem with its certificates: # getcert list Number of certificates and requests being tracked: 8. Request ID '20121107212513': status: CA_UNREACHABLE ca-error: Server failed request, will retry: 907 (RPC failed at server. cannot connect to ' https://kdc01.unix.iriszorg.nl:443/ca/agent/ca/doRevoke': (SEC_ERROR_BUSY) NSS could not shutdown. Objects are still in use.). stuck: yes key pair storage: type=NSSDB,location='/etc/dirsrv/slapd-UNIX-IRISZORG-NL',nickname='Server-Cert',token='NSS Certificate DB',pinfile='/etc/dirsrv/slapd-UNIX-IRISZORG-NL/pwdfile.txt' certificate: type=NSSDB,location='/etc/dirsrv/slapd-UNIX-IRISZORG-NL',nickname='Server-Cert',token='NSS Certificate DB' CA: IPA issuer: CN=Certificate Authority,O=UNIX.IRISZORG.NL subject: CN=kdc01.unix.iriszorg.nl,O=UNIX.IRISZORG.NL expires: 2016-10-12 10:49:24 UTC eku: id-kp-serverAuth,id-kp-clientAuth pre-save command: post-save command: /usr/lib/ipa/certmonger/restart_dirsrv UNIX-IRISZORG-NL track: yes auto-renew: yes Request ID '20121107212532': status: CA_UNREACHABLE ca-error: Server failed request, will retry: 4301 (RPC failed at server. Certificate operation cannot be completed: Failure decoding Certificate Signing Request). stuck: yes key pair storage: type=NSSDB,location='/etc/dirsrv/slapd-PKI-IPA',nickname='Server-Cert',token='NSS Certificate DB',pinfile='/etc/dirsrv/slapd-PKI-IPA/pwdfile.txt' certificate: type=NSSDB,location='/etc/dirsrv/slapd-PKI-IPA',nickname='Server-Cert',token='NSS Certificate DB' CA: IPA issuer: CN=Certificate Authority,O=UNIX.IRISZORG.NL subject: CN=kdc01.unix.iriszorg.nl,O=UNIX.IRISZORG.NL expires: 2016-10-12 10:49:25 UTC eku: id-kp-serverAuth,id-kp-clientAuth pre-save command: post-save command: track: yes auto-renew: yes Request ID '20121107212548': status: CA_UNREACHABLE ca-error: Server failed request, will retry: 4301 (RPC failed at server. Certificate operation cannot be completed: Failure decoding Certificate Signing Request). stuck: yes key pair storage: type=NSSDB,location='/etc/httpd/alias',nickname='Server-Cert',token='NSS Certificate DB',pinfile='/etc/httpd/alias/pwdfile.txt' certificate: type=NSSDB,location='/etc/httpd/alias',nickname='Server-Cert',token='NSS Certificate DB' CA: IPA issuer: CN=Certificate Authority,O=UNIX.IRISZORG.NL subject: CN=kdc01.unix.iriszorg.nl,O=UNIX.IRISZORG.NL expires: 2016-10-12 10:49:24 UTC eku: id-kp-serverAuth,id-kp-clientAuth pre-save command: post-save command: /usr/lib/ipa/certmonger/restart_httpd track: yes auto-renew: yes Where should I start looking? In /var/log/httpd/error_log there is nothing of consquence. -- -- Groeten, natxo -------------- next part -------------- An HTML attachment was scrubbed... URL: From mbasti at redhat.com Thu Sep 15 10:20:31 2016 From: mbasti at redhat.com (Martin Basti) Date: Thu, 15 Sep 2016 12:20:31 +0200 Subject: [Freeipa-users] SSL alert: CERT_VerifyCertificateNow: verify certificate failed for cert Server-Cert of family In-Reply-To: References: <7d17ebb6-eba6-b560-bcf9-e016546095c1@redhat.com> <4a3d91a5-e48c-96cc-b1aa-6082eea2e302@redhat.com> Message-ID: <7f084151-ac35-5459-8df8-f7e5c8d20bfa@redhat.com> I'm afraid that because you moved time back, the dogtag certificates are before VALIDITY time now. Can you find CA debug log, /var/log/pki/pki-tomcat/ca/debug.log (not sure about the path). There should be exact certificate and reason why cert validation failed On 14.09.2016 19:42, bahan w wrote: > Here is what I found : > > In the catalina.out : > ### > May 27, 2016 10:51:35 AM org.apache.catalina.core.StandardWrapperValve > invoke > SEVERE: Servlet.service() for servlet caDisplayBySerial-agent threw > exception > java.io.IOException: CS server is not ready to serve. > at > com.netscape.cms.servlet.base.CMSServlet.service(CMSServlet.java:441) > at javax.servlet.http.HttpServlet.service(HttpServlet.java:717) > at > org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290) > at > org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206) > at > com.netscape.cms.servlet.filter.AgentRequestFilter.doFilter(AgentRequestFilter.java:124) > at > org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235) > at > org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206) > at > org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233) > at > org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191) > at > org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127) > at > org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102) > at > org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109) > at > org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:298) > at > org.apache.jk.server.JkCoyoteHandler.invoke(JkCoyoteHandler.java:190) > at > org.apache.jk.common.HandlerRequest.invoke(HandlerRequest.java:291) > at > org.apache.jk.common.ChannelSocket.invoke(ChannelSocket.java:769) > at > org.apache.jk.common.ChannelSocket.processConnection(ChannelSocket.java:698) > at > org.apache.jk.common.ChannelSocket$SocketConnection.runIt(ChannelSocket.java:891) > at > org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:690) > at java.lang.Thread.run(Thread.java:722) > ### > > In the selftests.log in /var/log/pki-ca : > ### > 24196.main - [27/May/2016:10:50:27 CEST] [20] [1] SelfTestSubsystem: > Initializing self test plugins: > 24196.main - [27/May/2016:10:50:27 CEST] [20] [1] SelfTestSubsystem: > loading all self test plugin logger parameters > 24196.main - [27/May/2016:10:50:27 CEST] [20] [1] SelfTestSubsystem: > loading all self test plugin instances > 24196.main - [27/May/2016:10:50:27 CEST] [20] [1] SelfTestSubsystem: > loading all self test plugin instance parameters > 24196.main - [27/May/2016:10:50:27 CEST] [20] [1] SelfTestSubsystem: > loading self test plugins in on-demand order > 24196.main - [27/May/2016:10:50:27 CEST] [20] [1] SelfTestSubsystem: > loading self test plugins in startup order > 24196.main - [27/May/2016:10:50:27 CEST] [20] [1] SelfTestSubsystem: > Self test plugins have been successfully loaded! > 24196.main - [27/May/2016:10:50:28 CEST] [20] [1] SelfTestSubsystem: > Running self test plugins specified to be executed at startup: > 24196.main - [27/May/2016:10:50:28 CEST] [20] [1] CAPresence: CA is > present > 24196.main - [27/May/2016:10:50:28 CEST] [20] [1] > SystemCertsVerification: system certs verification failure > 24196.main - [27/May/2016:10:50:28 CEST] [20] [1] SelfTestSubsystem: > The CRITICAL self test plugin called selftests.container.instance.SystemC > ertsVerification running at startup FAILED! > ### > > But nothing else. > > Best regards. > > Bahan > > On Wed, Sep 14, 2016 at 7:27 PM, bahan w > wrote: > > I tried also the following commands : > ### > # ipa cert-show 1 > ipa: ERROR: Certificate operation cannot be completed: Unable to > communicate with CMS (Not Found) > > # service ipa status > Directory Service: RUNNING > KDC Service: RUNNING > KPASSWD Service: RUNNING > MEMCACHE Service: RUNNING > HTTP Service: RUNNING > CA Service: RUNNING > ### > > I'm checking the /var/log/pki-ca logs to see if I find something. > > Best regards. > > Bahan > > On Wed, Sep 14, 2016 at 7:02 PM, bahan w > wrote: > > Sorry Martin, > > This is not the first time I forgot to add back freeipa users. > I have problems with gmail, again sorry. > > Indeed I figured out that I had to restart the ipa server. > So I tried to restart ipa server. > But it was not working yet. > > So I thought it was maybe due to the configuration I performed > in the nss.conf. > So I rollbacked this conf and restarted ipa-server. > Then I retried your commands but it is still the same error. > > ### > Request ID '20140528064145': > status: CA_UNREACHABLE > ca-error: Server failed request, will retry: 4301 (RPC > failed at server. Certificate operation cannot be completed: > Unable to communicate with CMS (Not Found)). > stuck: yes > key pair storage: > type=NSSDB,location='/etc/httpd/alias',nickname='Server-Cert',token='NSS > Certificate DB',pinfile='/etc/httpd/alias/pwdfile.txt' > certificate: > type=NSSDB,location='/etc/httpd/alias',nickname='Server-Cert',token='NSS > Certificate DB' > CA: IPA > issuer: CN=Certificate Authority,O= > subject: CN=,O= > expires: 2016-05-28 06:41:44 UTC > eku: id-kp-serverAuth,id-kp-clientAuth > pre-save command: > post-save command: /usr/lib64/ipa/certmonger/restart_httpd > track: yes > auto-renew: yes > ### > > Do you know what is the CMS ? > ### > (RPC failed at server. Certificate operation cannot be > completed: Unable to communicate with CMS (Not Found)). > ### > > Best regards. > > Bahan > > > > > > On Wed, Sep 14, 2016 at 6:46 PM, Martin Basti > > wrote: > > did you restart IPA when you moved time? Is there are more > detailed error description in output of getcert list? > > > On 14.09.2016 18:45, bahan w wrote: >> I set the date-time when the certificates were valid : >> ### >> # date -s '2016-05-27 10:00:00' >> Fri May 27 10:00:00 CEST 2016 >> >> # date >> Fri May 27 10:00:02 CEST 2016 >> ### >> >> Then I try to renew them : >> ### >> # getcert resubmit -i 20140528063919 >> Resubmitting "20140528063919" to "IPA". >> >> # getcert resubmit -i 20140528064145 >> Resubmitting "20140528064145" to "IPA". >> >> # getcert resubmit -i 20140528063953 >> Resubmitting "20140528063953" to "IPA". >> ### >> >> But when I do the getcert list after, the result is the same. >> >> I guess it is because of this ? >> CA_UNREACHABLE >> >> Any idea ? >> >> Best regards. >> >> Bahan >> >> On Wed, Sep 14, 2016 at 6:38 PM, bahan w >> > >> wrote: >> >> Ok, I managed to restart the IPA service by adding >> this line in the file /etc/httpd/conf.d/nss.conf : >> ### >> NSSEnforceValidCerts off >> ### >> >> But when I do the getcert now I got the following >> result : >> >> ### >> # getcert list >> Number of certificates and requests being tracked: 8. >> Request ID '20140528063903': >> status: MONITORING >> stuck: no >> key pair storage: >> type=NSSDB,location='/var/lib/pki-ca/alias',nickname='auditSigningCert >> cert-pki-ca',token='NSS Certificate >> DB',pin='159203530658' >> certificate: >> type=NSSDB,location='/var/lib/pki-ca/alias',nickname='auditSigningCert >> cert-pki-ca',token='NSS Certificate DB' >> CA: dogtag-ipa-renew-agent >> issuer: CN=Certificate Authority,O= >> subject: CN=CA Audit,O= >> expires: 2018-04-09 11:39:16 UTC >> pre-save command: /usr/lib64/ipa/certmonger/stop_pkicad >> post-save command: >> /usr/lib64/ipa/certmonger/renew_ca_cert >> "auditSigningCert cert-pki-ca" >> track: yes >> auto-renew: yes >> Request ID '20140528063904': >> status: MONITORING >> stuck: no >> key pair storage: >> type=NSSDB,location='/var/lib/pki-ca/alias',nickname='ocspSigningCert >> cert-pki-ca',token='NSS Certificate >> DB',pin='159203530658' >> certificate: >> type=NSSDB,location='/var/lib/pki-ca/alias',nickname='ocspSigningCert >> cert-pki-ca',token='NSS Certificate DB' >> CA: dogtag-ipa-renew-agent >> issuer: CN=Certificate Authority,O= >> subject: CN=OCSP Subsystem,O= >> expires: 2018-04-09 11:38:16 UTC >> eku: id-kp-OCSPSigning >> pre-save command: /usr/lib64/ipa/certmonger/stop_pkicad >> post-save command: >> /usr/lib64/ipa/certmonger/renew_ca_cert >> "ocspSigningCert cert-pki-ca" >> track: yes >> auto-renew: yes >> Request ID '20140528063905': >> status: MONITORING >> stuck: no >> key pair storage: >> type=NSSDB,location='/var/lib/pki-ca/alias',nickname='subsystemCert >> cert-pki-ca',token='NSS Certificate >> DB',pin='159203530658' >> certificate: >> type=NSSDB,location='/var/lib/pki-ca/alias',nickname='subsystemCert >> cert-pki-ca',token='NSS Certificate DB' >> CA: dogtag-ipa-renew-agent >> issuer: CN=Certificate Authority,O= >> subject: CN=CA Subsystem,O= >> expires: 2018-04-09 11:38:16 UTC >> eku: id-kp-serverAuth,id-kp-clientAuth >> pre-save command: /usr/lib64/ipa/certmonger/stop_pkicad >> post-save command: >> /usr/lib64/ipa/certmonger/renew_ca_cert >> "subsystemCert cert-pki-ca" >> track: yes >> auto-renew: yes >> Request ID '20140528063906': >> status: MONITORING >> stuck: no >> key pair storage: >> type=NSSDB,location='/etc/httpd/alias',nickname='ipaCert',token='NSS >> Certificate DB',pinfile='/etc/httpd/alias/pwdfile.txt' >> certificate: >> type=NSSDB,location='/etc/httpd/alias',nickname='ipaCert',token='NSS >> Certificate DB' >> CA: dogtag-ipa-renew-agent >> issuer: CN=Certificate Authority,O= >> subject: CN=IPA RA,O= >> expires: 2018-04-09 11:38:16 UTC >> eku: id-kp-serverAuth,id-kp-clientAuth >> pre-save command: >> post-save command: >> /usr/lib64/ipa/certmonger/renew_ra_cert >> track: yes >> auto-renew: yes >> Request ID '20140528063907': >> status: MONITORING >> stuck: no >> key pair storage: >> type=NSSDB,location='/var/lib/pki-ca/alias',nickname='Server-Cert >> cert-pki-ca',token='NSS Certificate >> DB',pin='159203530658' >> certificate: >> type=NSSDB,location='/var/lib/pki-ca/alias',nickname='Server-Cert >> cert-pki-ca',token='NSS Certificate DB' >> CA: dogtag-ipa-renew-agent >> issuer: CN=Certificate Authority,O= >> subject: CN=,O= >> expires: 2018-04-09 11:38:16 UTC >> eku: id-kp-serverAuth,id-kp-clientAuth >> pre-save command: >> post-save command: >> track: yes >> auto-renew: yes >> Request ID '20140528063919': >> status: CA_UNREACHABLE >> ca-error: Server failed request, will retry: >> -504 (libcurl failed to execute the HTTP POST >> transaction. Peer certificate cannot be authenticated >> with known CA certificates). >> stuck: yes >> key pair storage: >> type=NSSDB,location='/etc/dirsrv/slapd-',nickname='Server-Cert',token='NSS >> Certificate >> DB',pinfile='/etc/dirsrv/slapd-/pwdfile.txt' >> certificate: >> type=NSSDB,location='/etc/dirsrv/slapd-',nickname='Server-Cert',token='NSS >> Certificate DB' >> CA: IPA >> issuer: CN=Certificate Authority,O= >> subject: CN=,O= >> expires: 2016-05-28 06:39:18 UTC >> eku: id-kp-serverAuth,id-kp-clientAuth >> pre-save command: >> post-save command: >> /usr/lib64/ipa/certmonger/restart_dirsrv >> track: yes >> auto-renew: yes >> Request ID '20140528063953': >> status: CA_UNREACHABLE >> ca-error: Server failed request, will retry: >> -504 (libcurl failed to execute the HTTP POST >> transaction. Peer certificate cannot be authenticated >> with known CA certificates). >> stuck: yes >> key pair storage: >> type=NSSDB,location='/etc/dirsrv/slapd-PKI-IPA',nickname='Server-Cert',token='NSS >> Certificate >> DB',pinfile='/etc/dirsrv/slapd-PKI-IPA/pwdfile.txt' >> certificate: >> type=NSSDB,location='/etc/dirsrv/slapd-PKI-IPA',nickname='Server-Cert',token='NSS >> Certificate DB' >> CA: IPA >> issuer: CN=Certificate Authority,O= >> subject: CN=,O= >> expires: 2016-05-28 06:39:52 UTC >> eku: id-kp-serverAuth,id-kp-clientAuth >> pre-save command: >> post-save command: >> /usr/lib64/ipa/certmonger/restart_dirsrv PKI-IPA >> track: yes >> auto-renew: yes >> Request ID '20140528064145': >> status: CA_UNREACHABLE >> ca-error: Server failed request, will retry: >> -504 (libcurl failed to execute the HTTP POST >> transaction. Peer certificate cannot be authenticated >> with known CA certificates). >> stuck: yes >> key pair storage: >> type=NSSDB,location='/etc/httpd/alias',nickname='Server-Cert',token='NSS >> Certificate DB',pinfile='/etc/httpd/alias/pwdfile.txt' >> certificate: >> type=NSSDB,location='/etc/httpd/alias',nickname='Server-Cert',token='NSS >> Certificate DB' >> CA: IPA >> issuer: CN=Certificate Authority,O= >> subject: CN=,O= >> expires: 2016-05-28 06:41:44 UTC >> eku: id-kp-serverAuth,id-kp-clientAuth >> pre-save command: >> post-save command: >> /usr/lib64/ipa/certmonger/restart_httpd >> track: yes >> auto-renew: yes >> ### >> >> Indeed, the entries outdated are the following : >> - for /etc/dirsrv/slapd- : 20140528063919 >> - for /etc/dirsrv/slapd-PKI-IPA : 20140528063953 >> - for httpd ? : 20140528064145 >> >> Best regards. >> >> Bahan >> >> On Wed, Sep 14, 2016 at 6:28 PM, bahan w >> > > wrote: >> >> Ok :D >> >> Because to perform the getcert list command, I >> need to have all the ipa services running right ? >> >> Here is the result of the command with the ipa >> services down. >> ### >> # getcert list >> Number of certificates and requests being tracked: 8. >> Request ID '20140528063903': >> status: MONITORING >> stuck: no >> key pair storage: >> type=NSSDB,location='/var/lib/pki-ca/alias',nickname='auditSigningCert >> cert-pki-ca',token='NSS Certificate >> DB',pin='159203530658' >> certificate: >> type=NSSDB,location='/var/lib/pki-ca/alias',nickname='auditSigningCert >> cert-pki-ca',token='NSS Certificate DB' >> CA: dogtag-ipa-renew-agent >> issuer: CN=Certificate Authority,O= >> subject: CN=CA Audit,O= >> expires: 2018-04-09 11:39:16 UTC >> pre-save command: >> /usr/lib64/ipa/certmonger/stop_pkicad >> post-save command: >> /usr/lib64/ipa/certmonger/renew_ca_cert >> "auditSigningCert cert-pki-ca" >> track: yes >> auto-renew: yes >> Request ID '20140528063904': >> status: MONITORING >> stuck: no >> key pair storage: >> type=NSSDB,location='/var/lib/pki-ca/alias',nickname='ocspSigningCert >> cert-pki-ca',token='NSS Certificate >> DB',pin='159203530658' >> certificate: >> type=NSSDB,location='/var/lib/pki-ca/alias',nickname='ocspSigningCert >> cert-pki-ca',token='NSS Certificate DB' >> CA: dogtag-ipa-renew-agent >> issuer: CN=Certificate Authority,O= >> subject: CN=OCSP Subsystem,O= >> expires: 2018-04-09 11:38:16 UTC >> eku: id-kp-OCSPSigning >> pre-save command: >> /usr/lib64/ipa/certmonger/stop_pkicad >> post-save command: >> /usr/lib64/ipa/certmonger/renew_ca_cert >> "ocspSigningCert cert-pki-ca" >> track: yes >> auto-renew: yes >> Request ID '20140528063905': >> status: MONITORING >> stuck: no >> key pair storage: >> type=NSSDB,location='/var/lib/pki-ca/alias',nickname='subsystemCert >> cert-pki-ca',token='NSS Certificate >> DB',pin='159203530658' >> certificate: >> type=NSSDB,location='/var/lib/pki-ca/alias',nickname='subsystemCert >> cert-pki-ca',token='NSS Certificate DB' >> CA: dogtag-ipa-renew-agent >> issuer: CN=Certificate Authority,O= >> subject: CN=CA Subsystem,O= >> expires: 2018-04-09 11:38:16 UTC >> eku: id-kp-serverAuth,id-kp-clientAuth >> pre-save command: >> /usr/lib64/ipa/certmonger/stop_pkicad >> post-save command: >> /usr/lib64/ipa/certmonger/renew_ca_cert >> "subsystemCert cert-pki-ca" >> track: yes >> auto-renew: yes >> Request ID '20140528063906': >> status: MONITORING >> stuck: no >> key pair storage: >> type=NSSDB,location='/etc/httpd/alias',nickname='ipaCert',token='NSS >> Certificate >> DB',pinfile='/etc/httpd/alias/pwdfile.txt' >> certificate: >> type=NSSDB,location='/etc/httpd/alias',nickname='ipaCert',token='NSS >> Certificate DB' >> CA: dogtag-ipa-renew-agent >> issuer: CN=Certificate Authority,O= >> subject: CN=IPA RA,O= >> expires: 2018-04-09 11:38:16 UTC >> eku: id-kp-serverAuth,id-kp-clientAuth >> pre-save command: >> post-save command: >> /usr/lib64/ipa/certmonger/renew_ra_cert >> track: yes >> auto-renew: yes >> Request ID '20140528063907': >> status: MONITORING >> stuck: no >> key pair storage: >> type=NSSDB,location='/var/lib/pki-ca/alias',nickname='Server-Cert >> cert-pki-ca',token='NSS Certificate >> DB',pin='159203530658' >> certificate: >> type=NSSDB,location='/var/lib/pki-ca/alias',nickname='Server-Cert >> cert-pki-ca',token='NSS Certificate DB' >> CA: dogtag-ipa-renew-agent >> issuer: CN=Certificate Authority,O= >> subject: CN=,O= >> expires: 2018-04-09 11:38:16 UTC >> eku: id-kp-serverAuth,id-kp-clientAuth >> pre-save command: >> post-save command: >> track: yes >> auto-renew: yes >> Request ID '20140528063919': >> status: MONITORING >> ca-error: Error setting up ccache for local >> "host" service using default keytab: Cannot >> contact any KDC for realm ''. >> stuck: no >> key pair storage: >> type=NSSDB,location='/etc/dirsrv/slapd-',nickname='Server-Cert',token='NSS >> Certificate >> DB',pinfile='/etc/dirsrv/slapd-/pwdfile.txt' >> certificate: >> type=NSSDB,location='/etc/dirsrv/slapd-',nickname='Server-Cert',token='NSS >> Certificate DB' >> CA: IPA >> issuer: CN=Certificate Authority,O= >> subject: CN=,O= >> expires: 2016-05-28 06:39:18 UTC >> eku: id-kp-serverAuth,id-kp-clientAuth >> pre-save command: >> post-save command: >> /usr/lib64/ipa/certmonger/restart_dirsrv >> track: yes >> auto-renew: yes >> Request ID '20140528063953': >> status: MONITORING >> ca-error: Error setting up ccache for local >> "host" service using default keytab: Cannot >> contact any KDC for realm ''. >> stuck: no >> key pair storage: >> type=NSSDB,location='/etc/dirsrv/slapd-PKI-IPA',nickname='Server-Cert',token='NSS >> Certificate >> DB',pinfile='/etc/dirsrv/slapd-PKI-IPA/pwdfile.txt' >> certificate: >> type=NSSDB,location='/etc/dirsrv/slapd-PKI-IPA',nickname='Server-Cert',token='NSS >> Certificate DB' >> CA: IPA >> issuer: CN=Certificate Authority,O= >> subject: CN=,O= >> expires: 2016-05-28 06:39:52 UTC >> eku: id-kp-serverAuth,id-kp-clientAuth >> pre-save command: >> post-save command: >> /usr/lib64/ipa/certmonger/restart_dirsrv PKI-IPA >> track: yes >> auto-renew: yes >> Request ID '20140528064145': >> status: MONITORING >> ca-error: Error setting up ccache for local >> "host" service using default keytab: Cannot >> contact any KDC for realm ''. >> stuck: no >> key pair storage: >> type=NSSDB,location='/etc/httpd/alias',nickname='Server-Cert',token='NSS >> Certificate >> DB',pinfile='/etc/httpd/alias/pwdfile.txt' >> certificate: >> type=NSSDB,location='/etc/httpd/alias',nickname='Server-Cert',token='NSS >> Certificate DB' >> CA: IPA >> issuer: CN=Certificate Authority,O= >> subject: CN=,O= >> expires: 2016-05-28 06:41:44 UTC >> eku: id-kp-serverAuth,id-kp-clientAuth >> pre-save command: >> post-save command: >> /usr/lib64/ipa/certmonger/restart_httpd >> track: yes >> auto-renew: yes >> ### >> >> Best regards. >> >> Bahan >> >> On Wed, Sep 14, 2016 at 6:21 PM, Martin Basti >> > wrote: >> >> >> Then you have to start services manually, I >> don't know if the same steps will work with >> IPA 3.0.0, I don't remember, but you can try :) >> >> >> On 14.09.2016 18:18, bahan w wrote: >>> Oh I forgot to add that my version of ipa is >>> quite old : >>> ### >>> # rpm -qa | grep ipa-server >>> ipa-server-3.0.0-25.el6.x86_64 >>> ### >>> >>> When I try the command you gave me I got the >>> following error : >>> ### >>> # ipactl start --force >>> Usage: ipactl start|stop|restart|status >>> >>> >>> ipactl: error: no such option: --force >>> ### >>> >>> Best regards. >>> >>> Bahan >>> >>> On Wed, Sep 14, 2016 at 6:14 PM, Martin >>> Basti >> > wrote: >>> >>> >>> >>> On 14.09.2016 17:59, bahan w wrote: >>>> Hello ! >>>> >>>> I send you this mail because I cannot >>>> restart my test IPA server. >>>> >>>> When I try to start it with service ipa >>>> start, I got the following error message : >>>> ### >>>> # service ipa start >>>> Starting Directory Service >>>> Starting dirsrv: >>>> ...[14/Sep/2016:17:57:23 >>>> +0200] - SSL alert: >>>> CERT_VerifyCertificateNow: verify >>>> certificate failed for cert Server-Cert >>>> of family >>>> cn=RSA,cn=encryption,cn=config >>>> (Netscape Portable Runtime error -8181 >>>> - Peer's Certificate has expired.) >>>> [ OK ] >>>> PKI-IPA...[14/Sep/2016:17:57:33 +0200] >>>> - SSL alert: CERT_VerifyCertificateNow: >>>> verify certificate failed for cert >>>> Server-Cert of family >>>> cn=RSA,cn=encryption,cn=config >>>> (Netscape Portable Runtime error -8181 >>>> - Peer's Certificate has expired.) >>>> [ OK ] >>>> Starting KDC Service >>>> Starting Kerberos 5 KDC: [ OK ] >>>> Starting KPASSWD Service >>>> Starting Kerberos 5 Admin Server: [ OK ] >>>> Starting MEMCACHE Service >>>> Starting ipa_memcached: [ OK ] >>>> Starting HTTP Service >>>> Starting httpd: [FAILED] >>>> Failed to start HTTP Service >>>> Shutting down >>>> Stopping Kerberos 5 KDC: [ OK ] >>>> Stopping Kerberos 5 Admin Server: [ OK ] >>>> Stopping ipa_memcached: [ OK ] >>>> Stopping httpd: [FAILED] >>>> Stopping pki-ca: [ OK ] >>>> Shutting down dirsrv: >>>> ... [ OK ] >>>> PKI-IPA... [ OK ] >>>> Aborting ipactl >>>> >>>> # service ipa status >>>> Directory Service: STOPPED >>>> Failed to get list of services to probe >>>> status: >>>> Directory Server is stopped >>>> ### >>>> >>>> Do you know how to renew the SSL >>>> certificate used for the IPA Server ? >>>> >>>> Best regards. >>>> >>>> Bahan >>>> >>>> >>>> >>> >>> >>> Hello, >>> >>> please run >>> >>> # ipactl start --force >>> # getcert list (to detect which >>> certificate is outdated, I suspect DS >>> cert (or to get more info why it has not >>> been renewed)) >>> >>> If getcert does work (I'm not sure if ti >>> is able to work without httpd), you >>> probable need to move time back to past >>> where cert is valid, start IPA and try >>> again. >>> >>> Please find ID outdated certificate and >>> try resubmit it (CA and DS must be running) >>> >>> # getcert resubmit -i 20160914122036 >>> (use you ID :) ) >>> >>> This should renew cert, check status >>> with getcert list >>> >>> Move time back to future (if needed) >>> >>> Try to restart IPA >>> >>> Martin^2 >>> >>> >> >> >> >> > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From mbasti at redhat.com Thu Sep 15 10:33:21 2016 From: mbasti at redhat.com (Martin Basti) Date: Thu, 15 Sep 2016 12:33:21 +0200 Subject: [Freeipa-users] certificates not renewing CA_UNREACHEABLE In-Reply-To: References: Message-ID: <806bdf32-558c-4972-255a-0901f47c4c4b@redhat.com> On 15.09.2016 11:29, Natxo Asenjo wrote: > hi, > > one of our master servers has a problem with its certificates: > > # getcert list > > Number of certificates and requests being tracked: 8. > Request ID '20121107212513': > status: CA_UNREACHABLE > ca-error: Server failed request, will retry: 907 (RPC failed > at server. cannot connect to > 'https://kdc01.unix.iriszorg.nl:443/ca/agent/ca/doRevoke': > (SEC_ERROR_BUSY) NSS could not shutdown. Objects are still in use.). > stuck: yes > key pair storage: > type=NSSDB,location='/etc/dirsrv/slapd-UNIX-IRISZORG-NL',nickname='Server-Cert',token='NSS > Certificate DB',pinfile='/etc/dirsrv/slapd-UNIX-IRISZORG-NL/pwdfile.txt' > certificate: > type=NSSDB,location='/etc/dirsrv/slapd-UNIX-IRISZORG-NL',nickname='Server-Cert',token='NSS > Certificate DB' > CA: IPA > issuer: CN=Certificate Authority,O=UNIX.IRISZORG.NL > > subject: CN=kdc01.unix.iriszorg.nl > ,O=UNIX.IRISZORG.NL > > expires: 2016-10-12 10:49:24 UTC > eku: id-kp-serverAuth,id-kp-clientAuth > pre-save command: > post-save command: /usr/lib/ipa/certmonger/restart_dirsrv > UNIX-IRISZORG-NL > track: yes > auto-renew: yes > Request ID '20121107212532': > status: CA_UNREACHABLE > ca-error: Server failed request, will retry: 4301 (RPC failed > at server. Certificate operation cannot be completed: Failure > decoding Certificate Signing Request). > stuck: yes > key pair storage: > type=NSSDB,location='/etc/dirsrv/slapd-PKI-IPA',nickname='Server-Cert',token='NSS > Certificate DB',pinfile='/etc/dirsrv/slapd-PKI-IPA/pwdfile.txt' > certificate: > type=NSSDB,location='/etc/dirsrv/slapd-PKI-IPA',nickname='Server-Cert',token='NSS > Certificate DB' > CA: IPA > issuer: CN=Certificate Authority,O=UNIX.IRISZORG.NL > > subject: CN=kdc01.unix.iriszorg.nl > ,O=UNIX.IRISZORG.NL > > expires: 2016-10-12 10:49:25 UTC > eku: id-kp-serverAuth,id-kp-clientAuth > pre-save command: > post-save command: > track: yes > auto-renew: yes > Request ID '20121107212548': > status: CA_UNREACHABLE > ca-error: Server failed request, will retry: 4301 (RPC failed > at server. Certificate operation cannot be completed: Failure > decoding Certificate Signing Request). > stuck: yes > key pair storage: > type=NSSDB,location='/etc/httpd/alias',nickname='Server-Cert',token='NSS > Certificate DB',pinfile='/etc/httpd/alias/pwdfile.txt' > certificate: > type=NSSDB,location='/etc/httpd/alias',nickname='Server-Cert',token='NSS > Certificate DB' > CA: IPA > issuer: CN=Certificate Authority,O=UNIX.IRISZORG.NL > > subject: CN=kdc01.unix.iriszorg.nl > ,O=UNIX.IRISZORG.NL > > expires: 2016-10-12 10:49:24 UTC > eku: id-kp-serverAuth,id-kp-clientAuth > pre-save command: > post-save command: /usr/lib/ipa/certmonger/restart_httpd > track: yes > auto-renew: yes > > > Where should I start looking? > > In /var/log/httpd/error_log there is nothing of consquence. > > -- > -- > Groeten, > natxo > > Hello, usually the most information can be found here /var/log/pki/pki-tomcat/ca/debug Martin -------------- next part -------------- An HTML attachment was scrubbed... URL: From gjn at gjn.priv.at Thu Sep 15 10:42:49 2016 From: gjn at gjn.priv.at (=?ISO-8859-1?Q?G=FCnther_J=2E?= Niederwimmer) Date: Thu, 15 Sep 2016 12:42:49 +0200 Subject: [Freeipa-users] ipa-server-certinstall -w -d mysite.key mysite.crt Message-ID: <3132705.gf7jV6EcmM@techz> Hello, FreeIPA 4.3.1 is it a workaround to install the key and cert with this command I have to insert a password, but the key file have no password? Afterward I have a Error from ipa-server-certinstall ? Thanks for the Help -- mit freundlichen Gr??en / best regards, G?nther J. Niederwimmer From natxo.asenjo at gmail.com Thu Sep 15 10:44:25 2016 From: natxo.asenjo at gmail.com (Natxo Asenjo) Date: Thu, 15 Sep 2016 12:44:25 +0200 Subject: [Freeipa-users] certificates not renewing CA_UNREACHEABLE In-Reply-To: <806bdf32-558c-4972-255a-0901f47c4c4b@redhat.com> References: <806bdf32-558c-4972-255a-0901f47c4c4b@redhat.com> Message-ID: hi, On Thu, Sep 15, 2016 at 12:33 PM, Martin Basti wrote: > > Hello, > > usually the most information can be found here > /var/log/pki/pki-tomcat/ca/debug > mmm, in this centos 6.8 system that does not exist: # ls -l /var/log/pki/pki-tomcat/ca/debug ls: cannot access /var/log/pki/pki-tomcat/ca/debug: No such file or directory I do have a /var/log/pki-ca/debug -- -- Groeten, natxo -------------- next part -------------- An HTML attachment was scrubbed... URL: From mbasti at redhat.com Thu Sep 15 10:49:01 2016 From: mbasti at redhat.com (Martin Basti) Date: Thu, 15 Sep 2016 12:49:01 +0200 Subject: [Freeipa-users] certificates not renewing CA_UNREACHEABLE In-Reply-To: References: <806bdf32-558c-4972-255a-0901f47c4c4b@redhat.com> Message-ID: <6596593a-9470-bebd-f236-f9d17aa4a56c@redhat.com> On 15.09.2016 12:44, Natxo Asenjo wrote: > hi, > > On Thu, Sep 15, 2016 at 12:33 PM, Martin Basti > wrote: > > > Hello, > > usually the most information can be found here > /var/log/pki/pki-tomcat/ca/debug > > > mmm, in this centos 6.8 system that does not exist: > > # ls -l /var/log/pki/pki-tomcat/ca/debug > ls: cannot access /var/log/pki/pki-tomcat/ca/debug: No such file or > directory > > > I do have a /var/log/pki-ca/debug > > Does it contain any information related to your issue? > > > -- > -- > Groeten, > natxo -------------- next part -------------- An HTML attachment was scrubbed... URL: From blipton at redhat.com Thu Sep 15 11:03:22 2016 From: blipton at redhat.com (Ben Lipton) Date: Thu, 15 Sep 2016 07:03:22 -0400 Subject: [Freeipa-users] ipa: ERROR: Certificate format error: (SEC_ERROR_LEGACY_DATABASE) The certificate/key database is in an old, unsupported format. In-Reply-To: References: <57D015D8.7080507@redhat.com> <57D166B9.3070201@redhat.com> <57D70676.8080107@redhat.com> <57D81AA1.4050400@redhat.com> Message-ID: <4eac5df5-ded2-8705-d5ba-6ab9dde5ad4d@redhat.com> On 09/15/2016 03:04 AM, Natxo Asenjo wrote: > Hi Ben, > > On Wed, Sep 14, 2016 at 2:45 PM, Ben Lipton > wrote: > > One other note - this could be a permissions issue. NSS seems to > produce this confusing error message when it can't access the > database, even if the format of the database is actually fine. > > $ sudo chown root:root /tmp/certs > $ certutil -N -d /tmp/certs > certutil: function failed: SEC_ERROR_LEGACY_DATABASE: The > certificate/key database is in an old, unsupported format. > > > Thanks for the tip. What directory should I check? I have checked: > > > [root at kdc01 httpd]$ ls -ltrZ /etc/httpd/alias/ > -rw-r-----. root apache unconfined_u:object_r:cert_t:s0 secmod.db.orig > -rw-r-----. root apache unconfined_u:object_r:cert_t:s0 key3.db.orig > -rw-r-----. root apache unconfined_u:object_r:cert_t:s0 cert8.db.orig > -rw-------. root root unconfined_u:object_r:cert_t:s0 install.log > -rw-rw----. root apache unconfined_u:object_r:cert_t:s0 pwdfile.txt > -rw-rw----. root apache unconfined_u:object_r:cert_t:s0 secmod.db > -r--r--r--. root root unconfined_u:object_r:cert_t:s0 cacert.asc.orig > -r--r--r--. root root unconfined_u:object_r:cert_t:s0 cacert.asc > lrwxrwxrwx. root root system_u:object_r:cert_t:s0 libnssckbi.so -> > ../../..//usr/lib/libnssckbi.so > -rw-rw----. root apache unconfined_u:object_r:cert_t:s0 key3.db > -rw-rw----. root apache unconfined_u:object_r:cert_t:s0 cert8.db > > [root at kdc01 httpd]$ ls -ltrdZ /etc/httpd/alias/ > drwxr-xr-x. root root system_u:object_r:cert_t:s0 /etc/httpd/alias/ > > > Those seem ok. > -- > Groeten, > natxo The other one I know about is: # ls -ltrZ /etc/ipa/nssdb total 80 -rw-------. 1 root root unconfined_u:object_r:cert_t:s0 40 Aug 22 13:13 pwdfile.txt -rw-r--r--. 1 root root unconfined_u:object_r:cert_t:s0 16384 Aug 22 13:13 secmod.db -rw-r--r--. 1 root root unconfined_u:object_r:cert_t:s0 16384 Aug 22 13:13 key3.db -rw-r--r--. 1 root root unconfined_u:object_r:cert_t:s0 65536 Aug 22 13:13 cert8.db # ls -ltrdZ /etc/ipa/nssdb drwxr-xr-x. 2 root root system_u:object_r:cert_t:s0 73 Sep 14 18:08 /etc/ipa/nssdb I still don't have any good ideas for why it would work for 5 minutes and then give an error. If you manage to get a traceback for the CertificateFormatError by enabling debug logging, that could be very helpful. -------------- next part -------------- An HTML attachment was scrubbed... URL: From natxo.asenjo at gmail.com Thu Sep 15 11:03:30 2016 From: natxo.asenjo at gmail.com (Natxo Asenjo) Date: Thu, 15 Sep 2016 13:03:30 +0200 Subject: [Freeipa-users] certificates not renewing CA_UNREACHEABLE In-Reply-To: <6596593a-9470-bebd-f236-f9d17aa4a56c@redhat.com> References: <806bdf32-558c-4972-255a-0901f47c4c4b@redhat.com> <6596593a-9470-bebd-f236-f9d17aa4a56c@redhat.com> Message-ID: On Thu, Sep 15, 2016 at 12:49 PM, Martin Basti wrote: > > > On 15.09.2016 12:44, Natxo Asenjo wrote: > > hi, > > On Thu, Sep 15, 2016 at 12:33 PM, Martin Basti wrote: > >> >> Hello, >> >> usually the most information can be found here >> /var/log/pki/pki-tomcat/ca/debug >> > > mmm, in this centos 6.8 system that does not exist: > > # ls -l /var/log/pki/pki-tomcat/ca/debug > ls: cannot access /var/log/pki/pki-tomcat/ca/debug: No such file or > directory > > > I do have a /var/log/pki-ca/debug > > > > Does it contain any information related to your issue? > I have tried renewing the certificate: ipa-getcert resubmit -i 20121107212513 If I grep that file for that request id I find nothing recent, just in the ipaserver installation log # cd /var/log # grep -ri 20121107212513 *.log ipaserver-install.log:2012-11-07T21:25:13Z DEBUG stdout=New tracking request "20121107212513" added. # grep -ri 20121107212513 pki-ca # -- -- Groeten, natxo -------------- next part -------------- An HTML attachment was scrubbed... URL: From natxo.asenjo at gmail.com Thu Sep 15 11:09:51 2016 From: natxo.asenjo at gmail.com (Natxo Asenjo) Date: Thu, 15 Sep 2016 13:09:51 +0200 Subject: [Freeipa-users] ipa: ERROR: Certificate format error: (SEC_ERROR_LEGACY_DATABASE) The certificate/key database is in an old, unsupported format. In-Reply-To: <4eac5df5-ded2-8705-d5ba-6ab9dde5ad4d@redhat.com> References: <57D015D8.7080507@redhat.com> <57D166B9.3070201@redhat.com> <57D70676.8080107@redhat.com> <57D81AA1.4050400@redhat.com> <4eac5df5-ded2-8705-d5ba-6ab9dde5ad4d@redhat.com> Message-ID: On Thu, Sep 15, 2016 at 1:03 PM, Ben Lipton wrote: > > On 09/15/2016 03:04 AM, Natxo Asenjo wrote: > > Hi Ben, > > On Wed, Sep 14, 2016 at 2:45 PM, Ben Lipton wrote: > > One other note - this could be a permissions issue. NSS seems to produce >> this confusing error message when it can't access the database, even if the >> format of the database is actually fine. >> >> $ sudo chown root:root /tmp/certs >> $ certutil -N -d /tmp/certs >> certutil: function failed: SEC_ERROR_LEGACY_DATABASE: The certificate/key >> database is in an old, unsupported format. >> > > Thanks for the tip. What directory should I check? I have checked: > > > [root at kdc01 httpd]$ ls -ltrZ /etc/httpd/alias/ > -rw-r-----. root apache unconfined_u:object_r:cert_t:s0 secmod.db.orig > -rw-r-----. root apache unconfined_u:object_r:cert_t:s0 key3.db.orig > -rw-r-----. root apache unconfined_u:object_r:cert_t:s0 cert8.db.orig > -rw-------. root root unconfined_u:object_r:cert_t:s0 install.log > -rw-rw----. root apache unconfined_u:object_r:cert_t:s0 pwdfile.txt > -rw-rw----. root apache unconfined_u:object_r:cert_t:s0 secmod.db > -r--r--r--. root root unconfined_u:object_r:cert_t:s0 cacert.asc.orig > -r--r--r--. root root unconfined_u:object_r:cert_t:s0 cacert.asc > lrwxrwxrwx. root root system_u:object_r:cert_t:s0 libnssckbi.so -> > ../../..//usr/lib/libnssckbi.so > -rw-rw----. root apache unconfined_u:object_r:cert_t:s0 key3.db > -rw-rw----. root apache unconfined_u:object_r:cert_t:s0 cert8.db > > [root at kdc01 httpd]$ ls -ltrdZ /etc/httpd/alias/ > drwxr-xr-x. root root system_u:object_r:cert_t:s0 /etc/httpd/alias/ > > > Those seem ok. > -- > Groeten, > natxo > > > The other one I know about is: > # ls -ltrZ /etc/ipa/nssdb > total 80 > -rw-------. 1 root root unconfined_u:object_r:cert_t:s0 40 Aug 22 > 13:13 pwdfile.txt > -rw-r--r--. 1 root root unconfined_u:object_r:cert_t:s0 16384 Aug 22 > 13:13 secmod.db > -rw-r--r--. 1 root root unconfined_u:object_r:cert_t:s0 16384 Aug 22 > 13:13 key3.db > -rw-r--r--. 1 root root unconfined_u:object_r:cert_t:s0 65536 Aug 22 > 13:13 cert8.db > # ls -ltrdZ /etc/ipa/nssdb > drwxr-xr-x. 2 root root system_u:object_r:cert_t:s0 73 Sep 14 18:08 > /etc/ipa/nssdb > > I still don't have any good ideas for why it would work for 5 minutes and > then give an error. If you manage to get a traceback for the > CertificateFormatError by enabling debug logging, that could be very > helpful. > I do not have that directory (centos 6.8): ls -ltrZ /etc/ipa/ -rw-r--r--. root root unconfined_u:object_r:etc_t:s0 default.conf -rw-r--r--. root root unconfined_u:object_r:etc_t:s0 ca.crt drwxr-xr-x. root root system_u:object_r:etc_t:s0 html -rw-r--r--. root root unconfined_u:object_r:etc_t:s0 server.conf.bak -rw-r--r--. root root unconfined_u:object_r:etc_t:s0 server.conf I have enabled debugging: $ cat /etc/ipa/server.conf [global] debug = True Could I send you the logs privately? -- -- Groeten, natxo -------------- next part -------------- An HTML attachment was scrubbed... URL: From natxo.asenjo at gmail.com Thu Sep 15 12:25:51 2016 From: natxo.asenjo at gmail.com (Natxo Asenjo) Date: Thu, 15 Sep 2016 14:25:51 +0200 Subject: [Freeipa-users] ipa: ERROR: Certificate format error: (SEC_ERROR_LEGACY_DATABASE) The certificate/key database is in an old, unsupported format. In-Reply-To: References: <57D015D8.7080507@redhat.com> <57D166B9.3070201@redhat.com> <57D70676.8080107@redhat.com> <57D81AA1.4050400@redhat.com> <4eac5df5-ded2-8705-d5ba-6ab9dde5ad4d@redhat.com> Message-ID: hi, attached error_log On Thu, Sep 15, 2016 at 1:09 PM, Natxo Asenjo wrote: > > > On Thu, Sep 15, 2016 at 1:03 PM, Ben Lipton wrote: > >> >> On 09/15/2016 03:04 AM, Natxo Asenjo wrote: >> >> Hi Ben, >> >> On Wed, Sep 14, 2016 at 2:45 PM, Ben Lipton wrote: >> >> One other note - this could be a permissions issue. NSS seems to produce >>> this confusing error message when it can't access the database, even if the >>> format of the database is actually fine. >>> >>> $ sudo chown root:root /tmp/certs >>> $ certutil -N -d /tmp/certs >>> certutil: function failed: SEC_ERROR_LEGACY_DATABASE: The >>> certificate/key database is in an old, unsupported format. >>> >> >> Thanks for the tip. What directory should I check? I have checked: >> >> >> [root at kdc01 httpd]$ ls -ltrZ /etc/httpd/alias/ >> -rw-r-----. root apache unconfined_u:object_r:cert_t:s0 secmod.db.orig >> -rw-r-----. root apache unconfined_u:object_r:cert_t:s0 key3.db.orig >> -rw-r-----. root apache unconfined_u:object_r:cert_t:s0 cert8.db.orig >> -rw-------. root root unconfined_u:object_r:cert_t:s0 install.log >> -rw-rw----. root apache unconfined_u:object_r:cert_t:s0 pwdfile.txt >> -rw-rw----. root apache unconfined_u:object_r:cert_t:s0 secmod.db >> -r--r--r--. root root unconfined_u:object_r:cert_t:s0 cacert.asc.orig >> -r--r--r--. root root unconfined_u:object_r:cert_t:s0 cacert.asc >> lrwxrwxrwx. root root system_u:object_r:cert_t:s0 libnssckbi.so >> -> ../../..//usr/lib/libnssckbi.so >> -rw-rw----. root apache unconfined_u:object_r:cert_t:s0 key3.db >> -rw-rw----. root apache unconfined_u:object_r:cert_t:s0 cert8.db >> >> [root at kdc01 httpd]$ ls -ltrdZ /etc/httpd/alias/ >> drwxr-xr-x. root root system_u:object_r:cert_t:s0 /etc/httpd/alias/ >> >> >> Those seem ok. >> -- >> Groeten, >> natxo >> >> >> The other one I know about is: >> # ls -ltrZ /etc/ipa/nssdb >> total 80 >> -rw-------. 1 root root unconfined_u:object_r:cert_t:s0 40 Aug 22 >> 13:13 pwdfile.txt >> -rw-r--r--. 1 root root unconfined_u:object_r:cert_t:s0 16384 Aug 22 >> 13:13 secmod.db >> -rw-r--r--. 1 root root unconfined_u:object_r:cert_t:s0 16384 Aug 22 >> 13:13 key3.db >> -rw-r--r--. 1 root root unconfined_u:object_r:cert_t:s0 65536 Aug 22 >> 13:13 cert8.db >> # ls -ltrdZ /etc/ipa/nssdb >> drwxr-xr-x. 2 root root system_u:object_r:cert_t:s0 73 Sep 14 18:08 >> /etc/ipa/nssdb >> >> I still don't have any good ideas for why it would work for 5 minutes and >> then give an error. If you manage to get a traceback for the >> CertificateFormatError by enabling debug logging, that could be very >> helpful. >> > > I do not have that directory (centos 6.8): > > ls -ltrZ /etc/ipa/ > -rw-r--r--. root root unconfined_u:object_r:etc_t:s0 default.conf > -rw-r--r--. root root unconfined_u:object_r:etc_t:s0 ca.crt > drwxr-xr-x. root root system_u:object_r:etc_t:s0 html > -rw-r--r--. root root unconfined_u:object_r:etc_t:s0 server.conf.bak > -rw-r--r--. root root unconfined_u:object_r:etc_t:s0 server.conf > > > I have enabled debugging: > > $ cat /etc/ipa/server.conf > [global] > debug = True > > Could I send you the logs privately? > > > -- > -- > Groeten, > natxo > -- -- Groeten, natxo -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: error_log Type: application/octet-stream Size: 473585 bytes Desc: not available URL: From abrook at bsd.uchicago.edu Thu Sep 15 14:04:18 2016 From: abrook at bsd.uchicago.edu (Brook, Andy [CRI]) Date: Thu, 15 Sep 2016 14:04:18 +0000 Subject: [Freeipa-users] Samba Server setup Message-ID: <73C11E05-045F-4324-9733-44C370584232@bsd.uchicago.edu> All, I?m working on setting up Samba to serve files from a server attached to our IPA domain. I followed the directions in https://www.freeipa.org/page/Howto/Integrating_a_Samba_File_Server_With_IPA. Everything seems to work and I can access the files from another RHEL server attached to the same domain using a Kerberos ticket from a user from the trusted AD domain. However, I can?t access this share from a windows client that is also attached to the trusted AD domain. My smb.conf is as follows: [global] workgroup = IPA realm = IPA.DOMAIN kerberos method = dedicated keytab dedicated keytab file = FILE:/etc/samba/samba.keytab log file = /var/log/samba/log.%m log level = 3 security = ads load printers = no disable spoolss = yes map to guest = Never restrict anonymous = 2 [spacetest] path = /var/www writable = yes browsable = yes I put the keytab in place from the cifs service from the IPA server. I feel like I?m missing something small, but I can?t seem to find it. Logs from samba are here: http://pastebin.com/aMDXfR78 Andy Brook Sr. Systems Administrator | Center for Research Informatics | University of Chicago T: 773-834-0458 | http://cri.uchicago.edu ******************************************************************************** This e-mail is intended only for the use of the individual or entity to which it is addressed and may contain information that is privileged and confidential. If the reader of this e-mail message is not the intended recipient, you are hereby notified that any dissemination, distribution or copying of this communication is prohibited. If you have received this e-mail in error, please notify the sender and destroy all copies of the transmittal. Thank you University of Chicago Medicine and Biological Sciences ******************************************************************************** From alan at instinctualsoftware.com Thu Sep 15 16:07:24 2016 From: alan at instinctualsoftware.com (Alan Latteri) Date: Thu, 15 Sep 2016 09:07:24 -0700 Subject: [Freeipa-users] Samba Server setup In-Reply-To: <73C11E05-045F-4324-9733-44C370584232@bsd.uchicago.edu> References: <73C11E05-045F-4324-9733-44C370584232@bsd.uchicago.edu> Message-ID: I too am running into this problem. Looking forward to some feedback regarding this issue. > On Sep 15, 2016, at 7:04 AM, Brook, Andy [CRI] wrote: > > All, > I?m working on setting up Samba to serve files from a server attached to our IPA domain. I followed the directions in https://www.freeipa.org/page/Howto/Integrating_a_Samba_File_Server_With_IPA. Everything seems to work and I can access the files from another RHEL server attached to the same domain using a Kerberos ticket from a user from the trusted AD domain. However, I can?t access this share from a windows client that is also attached to the trusted AD domain. > > My smb.conf is as follows: > [global] > workgroup = IPA > realm = IPA.DOMAIN > kerberos method = dedicated keytab > dedicated keytab file = FILE:/etc/samba/samba.keytab > log file = /var/log/samba/log.%m > log level = 3 > security = ads > load printers = no > disable spoolss = yes > map to guest = Never > restrict anonymous = 2 > > [spacetest] > path = /var/www > writable = yes > browsable = yes > > I put the keytab in place from the cifs service from the IPA server. > > I feel like I?m missing something small, but I can?t seem to find it. Logs from samba are here: http://pastebin.com/aMDXfR78 > > Andy Brook > Sr. Systems Administrator | Center for Research Informatics | University of Chicago > T: 773-834-0458 | http://cri.uchicago.edu > > ******************************************************************************** > This e-mail is intended only for the use of the individual or entity to which > it is addressed and may contain information that is privileged and confidential. > If the reader of this e-mail message is not the intended recipient, you are > hereby notified that any dissemination, distribution or copying of this > communication is prohibited. If you have received this e-mail in error, please > notify the sender and destroy all copies of the transmittal. > > Thank you > University of Chicago Medicine and Biological Sciences > ******************************************************************************** > > -- > Manage your subscription for the Freeipa-users mailing list: > https://www.redhat.com/mailman/listinfo/freeipa-users > Go to http://freeipa.org for more info on the project From tbordaz at redhat.com Thu Sep 15 16:38:14 2016 From: tbordaz at redhat.com (thierry bordaz) Date: Thu, 15 Sep 2016 18:38:14 +0200 Subject: [Freeipa-users] Freeipa 4.2.0 hangs intermittently In-Reply-To: References: <57C44AC6.1030402@redhat.com> <57CD1FDC.2010109@redhat.com> <57CD5488.60404@redhat.com> <57CD659B.6050707@redhat.com> <57CD68BD.9030802@redhat.com> <57CE8CDB.6000809@redhat.com> <57CE9DE5.6020301@redhat.com> Message-ID: <57DACE76.5070803@redhat.com> Hello Rakesh, Sorry to be back so late. I finally manage to try to reproduce locally the hang of DS... and I fail ;-) I am running 4 VM, with freeipa 4.3.2. One is running a server (where I provisioned some users), with 200Mb cache. The server receives no direct load (SRCH or update). The tree others are running in a loop ipa-client-install and uninstall. So far it has run for several hours without hang. Going through the mail thread I have not found others details. Am I missing something to reproduce ? thanks in advance for your help thierry On 09/07/2016 01:46 PM, Rakesh Rajasekharan wrote: > I changed the nsslapd-db-checkpoint interval first to 80 and increased > till 160.. however the hang has not gone away yet > > > I migrated my data from a openldap, can that cause issues like this.. > If so, i could try creating users from scratch directly on IPA rather > than migrating and test it > > But before that , have taken a fresh pstack just incase if it helps to > nail down the issue > > Thread 44 (Thread 0x7f53ed52f700 (LWP 128165)): > #0 0x00007f54047649b3 in select () from /lib64/libc.so.6 > #1 0x00007f5406eef0e9 in DS_Sleep () from /usr/lib64/dirsrv/libslapd.so.0 > #2 0x00007f53fa53d907 in deadlock_threadmain () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #3 0x00007f540509e7bb in _pt_root () from /lib64/libnspr4.so > #4 0x00007f5404a3fdc5 in start_thread () from /lib64/libpthread.so.0 > #5 0x00007f540476d28d in clone () from /lib64/libc.so.6 > Thread 43 (Thread 0x7f53ecd2e700 (LWP 128166)): > #0 0x00007f54047649b3 in select () from /lib64/libc.so.6 > #1 0x00007f53ff0cbbad in __os_yield () from /lib64/libdb-5.3.so > > #2 0x00007f53ff0c72b3 in __memp_sync_int () from /lib64/libdb-5.3.so > > #3 0x00007f53ff0d7752 in __txn_checkpoint () from /lib64/libdb-5.3.so > > #4 0x00007f53ff0d7b74 in __txn_checkpoint_pp () from > /lib64/libdb-5.3.so > #5 0x00007f53fa541a87 in checkpoint_threadmain () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #6 0x00007f540509e7bb in _pt_root () from /lib64/libnspr4.so > #7 0x00007f5404a3fdc5 in start_thread () from /lib64/libpthread.so.0 > #8 0x00007f540476d28d in clone () from /lib64/libc.so.6 > Thread 42 (Thread 0x7f53ec52d700 (LWP 128167)): > #0 0x00007f54047649b3 in select () from /lib64/libc.so.6 > #1 0x00007f5406eef0e9 in DS_Sleep () from /usr/lib64/dirsrv/libslapd.so.0 > #2 0x00007f53fa53db7f in trickle_threadmain () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #3 0x00007f540509e7bb in _pt_root () from /lib64/libnspr4.so > #4 0x00007f5404a3fdc5 in start_thread () from /lib64/libpthread.so.0 > #5 0x00007f540476d28d in clone () from /lib64/libc.so.6 > Thread 41 (Thread 0x7f53ebd2c700 (LWP 128168)): > #0 0x00007f54047649b3 in select () from /lib64/libc.so.6 > #1 0x00007f5406eef0e9 in DS_Sleep () from /usr/lib64/dirsrv/libslapd.so.0 > #2 0x00007f53fa538707 in perf_threadmain () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #3 0x00007f540509e7bb in _pt_root () from /lib64/libnspr4.so > #4 0x00007f5404a3fdc5 in start_thread () from /lib64/libpthread.so.0 > #5 0x00007f540476d28d in clone () from /lib64/libc.so.6 > Thread 40 (Thread 0x7f53eb322700 (LWP 128204)): > #0 0x00007f5404a436d5 in pthread_cond_wait@@GLIBC_2.3.2 () from > /lib64/libpthread.so.0 > #1 0x00007f5405099050 in PR_WaitCondVar () from /lib64/libnspr4.so > #2 0x00007f5406ede198 in slapi_wait_condvar () from > /usr/lib64/dirsrv/libslapd.so.0 > #3 0x00007f53fd27062e in cos_cache_wait_on_change () from > /usr/lib64/dirsrv/plugins/libcos-plugin.so > #4 0x00007f540509e7bb in _pt_root () from /lib64/libnspr4.so > #5 0x00007f5404a3fdc5 in start_thread () from /lib64/libpthread.so.0 > #6 0x00007f540476d28d in clone () from /lib64/libc.so.6 > Thread 39 (Thread 0x7f53eab21700 (LWP 128206)): > #0 0x00007f5404a436d5 in pthread_cond_wait@@GLIBC_2.3.2 () from > /lib64/libpthread.so.0 > #1 0x00007f5405099050 in PR_WaitCondVar () from /lib64/libnspr4.so > #2 0x00007f5406ede198 in slapi_wait_condvar () from > /usr/lib64/dirsrv/libslapd.so.0 > #3 0x00007f53f8bdbeed in roles_cache_wait_on_change () from > /usr/lib64/dirsrv/plugins/libroles-plugin.so > #4 0x00007f540509e7bb in _pt_root () from /lib64/libnspr4.so > #5 0x00007f5404a3fdc5 in start_thread () from /lib64/libpthread.so.0 > #6 0x00007f540476d28d in clone () from /lib64/libc.so.6 > Thread 38 (Thread 0x7f53ea320700 (LWP 128207)): > #0 0x00007f5404a436d5 in pthread_cond_wait@@GLIBC_2.3.2 () from > /lib64/libpthread.so.0 > #1 0x00007f5405099050 in PR_WaitCondVar () from /lib64/libnspr4.so > #2 0x00007f5406ede198 in slapi_wait_condvar () from > /usr/lib64/dirsrv/libslapd.so.0 > #3 0x00007f53f8bdbeed in roles_cache_wait_on_change () from > /usr/lib64/dirsrv/plugins/libroles-plugin.so > #4 0x00007f540509e7bb in _pt_root () from /lib64/libnspr4.so > #5 0x00007f5404a3fdc5 in start_thread () from /lib64/libpthread.so.0 > #6 0x00007f540476d28d in clone () from /lib64/libc.so.6 > Thread 37 (Thread 0x7f53e9b1f700 (LWP 128208)): > #0 0x00007f5404a436d5 in pthread_cond_wait@@GLIBC_2.3.2 () from > /lib64/libpthread.so.0 > #1 0x00007f5405099050 in PR_WaitCondVar () from /lib64/libnspr4.so > #2 0x00007f5406ede198 in slapi_wait_condvar () from > /usr/lib64/dirsrv/libslapd.so.0 > #3 0x00007f53f8bdbeed in roles_cache_wait_on_change () from > /usr/lib64/dirsrv/plugins/libroles-plugin.so > #4 0x00007f540509e7bb in _pt_root () from /lib64/libnspr4.so > #5 0x00007f5404a3fdc5 in start_thread () from /lib64/libpthread.so.0 > #6 0x00007f540476d28d in clone () from /lib64/libc.so.6 > Thread 36 (Thread 0x7f53e931e700 (LWP 128209)): > #0 0x00007f5404a43a82 in pthread_cond_timedwait@@GLIBC_2.3.2 () from > /lib64/libpthread.so.0 > #1 0x00007f5405098b07 in pt_TimedWait () from /lib64/libnspr4.so > #2 0x00007f5405098fce in PR_WaitCondVar () from /lib64/libnspr4.so > #3 0x00007f5407397d03 in housecleaning () > #4 0x00007f540509e7bb in _pt_root () from /lib64/libnspr4.so > #5 0x00007f5404a3fdc5 in start_thread () from /lib64/libpthread.so.0 > #6 0x00007f540476d28d in clone () from /lib64/libc.so.6 > Thread 35 (Thread 0x7f53e8b1d700 (LWP 128210)): > #0 0x00007f5404a43a82 in pthread_cond_timedwait@@GLIBC_2.3.2 () from > /lib64/libpthread.so.0 > #1 0x00007f5405098b07 in pt_TimedWait () from /lib64/libnspr4.so > #2 0x00007f5405098fce in PR_WaitCondVar () from /lib64/libnspr4.so > #3 0x00007f5406e841e8 in eq_loop () from /usr/lib64/dirsrv/libslapd.so.0 > #4 0x00007f540509e7bb in _pt_root () from /lib64/libnspr4.so > #5 0x00007f5404a3fdc5 in start_thread () from /lib64/libpthread.so.0 > #6 0x00007f540476d28d in clone () from /lib64/libc.so.6 > Thread 34 (Thread 0x7f53db7fe700 (LWP 128212)): > #0 0x00007f5404a436d5 in pthread_cond_wait@@GLIBC_2.3.2 () from > /lib64/libpthread.so.0 > #1 0x00007f53fef9f2f3 in __db_hybrid_mutex_suspend () from > /lib64/libdb-5.3.so > #2 0x00007f53fef9e640 in __db_tas_mutex_lock () from > /lib64/libdb-5.3.so > #3 0x00007f53ff048d3a in __lock_get_internal () from > /lib64/libdb-5.3.so > #4 0x00007f53ff049820 in __lock_get () from /lib64/libdb-5.3.so > > #5 0x00007f53ff075142 in __db_lget () from /lib64/libdb-5.3.so > > #6 0x00007f53fefbc605 in __bam_search () from /lib64/libdb-5.3.so > > #7 0x00007f53fefa7256 in __bamc_search () from /lib64/libdb-5.3.so > > #8 0x00007f53fefa8d0f in __bamc_get () from /lib64/libdb-5.3.so > > #9 0x00007f53ff061ca6 in __dbc_iget () from /lib64/libdb-5.3.so > > #10 0x00007f53ff070b02 in __dbc_get_pp () from /lib64/libdb-5.3.so > > #11 0x00007f53fa54c680 in idl_new_fetch () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #12 0x00007f53fa55ab26 in index_read_ext_allids () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #13 0x00007f53fa545284 in keys2idl () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #14 0x00007f53fa5459e3 in ava_candidates.isra.0 () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #15 0x00007f53fa545fd2 in filter_candidates_ext () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #16 0x00007f53fa547096 in list_candidates () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #17 0x00007f53fa545f40 in filter_candidates_ext () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #18 0x00007f53fa58180c in subtree_candidates () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #19 0x00007f53fa582e7f in ldbm_back_search () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #20 0x00007f5406eb01d6 in op_shared_search () from > /usr/lib64/dirsrv/libslapd.so.0 > #21 0x00007f54073a133e in do_search () > #22 0x00007f540738f567 in connection_threadmain () > #23 0x00007f540509e7bb in _pt_root () from /lib64/libnspr4.so > #24 0x00007f5404a3fdc5 in start_thread () from /lib64/libpthread.so.0 > #25 0x00007f540476d28d in clone () from /lib64/libc.so.6 > Thread 33 (Thread 0x7f53daffd700 (LWP 128213)): > #0 0x00007f5404a436d5 in pthread_cond_wait@@GLIBC_2.3.2 () from > /lib64/libpthread.so.0 > #1 0x00007f53fef9f2f3 in __db_hybrid_mutex_suspend () from > /lib64/libdb-5.3.so > #2 0x00007f53fef9e640 in __db_tas_mutex_lock () from > /lib64/libdb-5.3.so > #3 0x00007f53ff048d3a in __lock_get_internal () from > /lib64/libdb-5.3.so > #4 0x00007f53ff049820 in __lock_get () from /lib64/libdb-5.3.so > > #5 0x00007f53ff075142 in __db_lget () from /lib64/libdb-5.3.so > > #6 0x00007f53fefbc605 in __bam_search () from /lib64/libdb-5.3.so > > #7 0x00007f53fefa7256 in __bamc_search () from /lib64/libdb-5.3.so > > #8 0x00007f53fefa8d0f in __bamc_get () from /lib64/libdb-5.3.so > > #9 0x00007f53ff061ca6 in __dbc_iget () from /lib64/libdb-5.3.so > > #10 0x00007f53ff070b02 in __dbc_get_pp () from /lib64/libdb-5.3.so > > #11 0x00007f53fa54c680 in idl_new_fetch () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #12 0x00007f53fa55ab26 in index_read_ext_allids () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #13 0x00007f53fa545284 in keys2idl () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #14 0x00007f53fa5459e3 in ava_candidates.isra.0 () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #15 0x00007f53fa545fd2 in filter_candidates_ext () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #16 0x00007f53fa547096 in list_candidates () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #17 0x00007f53fa545f40 in filter_candidates_ext () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #18 0x00007f53fa58180c in subtree_candidates () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #19 0x00007f53fa582e7f in ldbm_back_search () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #20 0x00007f5406eb01d6 in op_shared_search () from > /usr/lib64/dirsrv/libslapd.so.0 > #21 0x00007f54073a133e in do_search () > #22 0x00007f540738f567 in connection_threadmain () > #23 0x00007f540509e7bb in _pt_root () from /lib64/libnspr4.so > #24 0x00007f5404a3fdc5 in start_thread () from /lib64/libpthread.so.0 > #25 0x00007f540476d28d in clone () from /lib64/libc.so.6 > Thread 32 (Thread 0x7f53da7fc700 (LWP 128214)): > #0 0x00007f5404a436d5 in pthread_cond_wait@@GLIBC_2.3.2 () from > /lib64/libpthread.so.0 > #1 0x00007f53fef9f2f3 in __db_hybrid_mutex_suspend () from > /lib64/libdb-5.3.so > #2 0x00007f53fef9e640 in __db_tas_mutex_lock () from > /lib64/libdb-5.3.so > #3 0x00007f53ff048d3a in __lock_get_internal () from > /lib64/libdb-5.3.so > #4 0x00007f53ff049820 in __lock_get () from /lib64/libdb-5.3.so > > #5 0x00007f53ff075142 in __db_lget () from /lib64/libdb-5.3.so > > #6 0x00007f53fefbc605 in __bam_search () from /lib64/libdb-5.3.so > > #7 0x00007f53fefa7256 in __bamc_search () from /lib64/libdb-5.3.so > > #8 0x00007f53fefa8d0f in __bamc_get () from /lib64/libdb-5.3.so > > #9 0x00007f53ff061ca6 in __dbc_iget () from /lib64/libdb-5.3.so > > #10 0x00007f53ff070b02 in __dbc_get_pp () from /lib64/libdb-5.3.so > > #11 0x00007f53fa54c680 in idl_new_fetch () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #12 0x00007f53fa55ab26 in index_read_ext_allids () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #13 0x00007f53fa545284 in keys2idl () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #14 0x00007f53fa5459e3 in ava_candidates.isra.0 () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #15 0x00007f53fa545fd2 in filter_candidates_ext () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #16 0x00007f53fa547096 in list_candidates () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #17 0x00007f53fa545f40 in filter_candidates_ext () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #18 0x00007f53fa547096 in list_candidates () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #19 0x00007f53fa545f40 in filter_candidates_ext () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #20 0x00007f53fa547096 in list_candidates () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #21 0x00007f53fa545f40 in filter_candidates_ext () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #22 0x00007f53fa58180c in subtree_candidates () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #23 0x00007f53fa582e7f in ldbm_back_search () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #24 0x00007f5406eb01d6 in op_shared_search () from > /usr/lib64/dirsrv/libslapd.so.0 > #25 0x00007f54073a133e in do_search () > #26 0x00007f540738f3c3 in connection_threadmain () > #27 0x00007f540509e7bb in _pt_root () from /lib64/libnspr4.so > #28 0x00007f5404a3fdc5 in start_thread () from /lib64/libpthread.so.0 > #29 0x00007f540476d28d in clone () from /lib64/libc.so.6 > Thread 31 (Thread 0x7f53d9ffb700 (LWP 128215)): > #0 0x00007f5404a436d5 in pthread_cond_wait@@GLIBC_2.3.2 () from > /lib64/libpthread.so.0 > #1 0x00007f53fef9f2f3 in __db_hybrid_mutex_suspend () from > /lib64/libdb-5.3.so > #2 0x00007f53fef9e640 in __db_tas_mutex_lock () from > /lib64/libdb-5.3.so > #3 0x00007f53ff048d3a in __lock_get_internal () from > /lib64/libdb-5.3.so > #4 0x00007f53ff049820 in __lock_get () from /lib64/libdb-5.3.so > > #5 0x00007f53ff075142 in __db_lget () from /lib64/libdb-5.3.so > > #6 0x00007f53fefbc605 in __bam_search () from /lib64/libdb-5.3.so > > #7 0x00007f53fefa7256 in __bamc_search () from /lib64/libdb-5.3.so > > #8 0x00007f53fefa8d0f in __bamc_get () from /lib64/libdb-5.3.so > > #9 0x00007f53ff061ca6 in __dbc_iget () from /lib64/libdb-5.3.so > > #10 0x00007f53ff070b02 in __dbc_get_pp () from /lib64/libdb-5.3.so > > #11 0x00007f53fa54c680 in idl_new_fetch () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #12 0x00007f53fa55ab26 in index_read_ext_allids () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #13 0x00007f53fa545284 in keys2idl () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #14 0x00007f53fa5459e3 in ava_candidates.isra.0 () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #15 0x00007f53fa545fd2 in filter_candidates_ext () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #16 0x00007f53fa547096 in list_candidates () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #17 0x00007f53fa545f40 in filter_candidates_ext () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #18 0x00007f53fa58180c in subtree_candidates () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #19 0x00007f53fa582e7f in ldbm_back_search () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #20 0x00007f5406eb01d6 in op_shared_search () from > /usr/lib64/dirsrv/libslapd.so.0 > #21 0x00007f54073a133e in do_search () > #22 0x00007f540738f567 in connection_threadmain () > #23 0x00007f540509e7bb in _pt_root () from /lib64/libnspr4.so > #24 0x00007f5404a3fdc5 in start_thread () from /lib64/libpthread.so.0 > #25 0x00007f540476d28d in clone () from /lib64/libc.so.6 > Thread 30 (Thread 0x7f53d97fa700 (LWP 128216)): > #0 0x00007f5404a436d5 in pthread_cond_wait@@GLIBC_2.3.2 () from > /lib64/libpthread.so.0 > #1 0x00007f53fef9f2f3 in __db_hybrid_mutex_suspend () from > /lib64/libdb-5.3.so > #2 0x00007f53fef9e640 in __db_tas_mutex_lock () from > /lib64/libdb-5.3.so > #3 0x00007f53ff048d3a in __lock_get_internal () from > /lib64/libdb-5.3.so > #4 0x00007f53ff049820 in __lock_get () from /lib64/libdb-5.3.so > > #5 0x00007f53ff075142 in __db_lget () from /lib64/libdb-5.3.so > > #6 0x00007f53fefbc605 in __bam_search () from /lib64/libdb-5.3.so > > #7 0x00007f53fefa7256 in __bamc_search () from /lib64/libdb-5.3.so > > #8 0x00007f53fefa8d0f in __bamc_get () from /lib64/libdb-5.3.so > > #9 0x00007f53ff061ca6 in __dbc_iget () from /lib64/libdb-5.3.so > > #10 0x00007f53ff070b02 in __dbc_get_pp () from /lib64/libdb-5.3.so > > #11 0x00007f53fa54c680 in idl_new_fetch () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #12 0x00007f53fa55ab26 in index_read_ext_allids () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #13 0x00007f53fa545284 in keys2idl () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #14 0x00007f53fa5459e3 in ava_candidates.isra.0 () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #15 0x00007f53fa545fd2 in filter_candidates_ext () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #16 0x00007f53fa547096 in list_candidates () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #17 0x00007f53fa545f40 in filter_candidates_ext () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #18 0x00007f53fa58180c in subtree_candidates () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #19 0x00007f53fa582e7f in ldbm_back_search () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #20 0x00007f5406eb01d6 in op_shared_search () from > /usr/lib64/dirsrv/libslapd.so.0 > #21 0x00007f54073a133e in do_search () > #22 0x00007f540738f567 in connection_threadmain () > #23 0x00007f540509e7bb in _pt_root () from /lib64/libnspr4.so > #24 0x00007f5404a3fdc5 in start_thread () from /lib64/libpthread.so.0 > #25 0x00007f540476d28d in clone () from /lib64/libc.so.6 > Thread 29 (Thread 0x7f53d8ff9700 (LWP 128217)): > #0 0x00007f5404a436d5 in pthread_cond_wait@@GLIBC_2.3.2 () from > /lib64/libpthread.so.0 > #1 0x00007f53fef9f2f3 in __db_hybrid_mutex_suspend () from > /lib64/libdb-5.3.so > #2 0x00007f53fef9e640 in __db_tas_mutex_lock () from > /lib64/libdb-5.3.so > #3 0x00007f53ff048d3a in __lock_get_internal () from > /lib64/libdb-5.3.so > #4 0x00007f53ff049820 in __lock_get () from /lib64/libdb-5.3.so > > #5 0x00007f53ff075142 in __db_lget () from /lib64/libdb-5.3.so > > #6 0x00007f53fefbc605 in __bam_search () from /lib64/libdb-5.3.so > > #7 0x00007f53fefa7256 in __bamc_search () from /lib64/libdb-5.3.so > > #8 0x00007f53fefa8d0f in __bamc_get () from /lib64/libdb-5.3.so > > #9 0x00007f53ff061ca6 in __dbc_iget () from /lib64/libdb-5.3.so > > #10 0x00007f53ff070b02 in __dbc_get_pp () from /lib64/libdb-5.3.so > > #11 0x00007f53fa54c680 in idl_new_fetch () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #12 0x00007f53fa55ab26 in index_read_ext_allids () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #13 0x00007f53fa545284 in keys2idl () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #14 0x00007f53fa5459e3 in ava_candidates.isra.0 () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #15 0x00007f53fa545fd2 in filter_candidates_ext () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #16 0x00007f53fa547096 in list_candidates () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #17 0x00007f53fa545f40 in filter_candidates_ext () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #18 0x00007f53fa58180c in subtree_candidates () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #19 0x00007f53fa582e7f in ldbm_back_search () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #20 0x00007f5406eb01d6 in op_shared_search () from > /usr/lib64/dirsrv/libslapd.so.0 > #21 0x00007f54073a133e in do_search () > #22 0x00007f540738f567 in connection_threadmain () > #23 0x00007f540509e7bb in _pt_root () from /lib64/libnspr4.so > #24 0x00007f5404a3fdc5 in start_thread () from /lib64/libpthread.so.0 > #25 0x00007f540476d28d in clone () from /lib64/libc.so.6 > Thread 28 (Thread 0x7f53d87f8700 (LWP 128218)): > #0 0x00007f5404a436d5 in pthread_cond_wait@@GLIBC_2.3.2 () from > /lib64/libpthread.so.0 > #1 0x00007f53fef9f2f3 in __db_hybrid_mutex_suspend () from > /lib64/libdb-5.3.so > #2 0x00007f53fef9e640 in __db_tas_mutex_lock () from > /lib64/libdb-5.3.so > #3 0x00007f53ff048d3a in __lock_get_internal () from > /lib64/libdb-5.3.so > #4 0x00007f53ff049820 in __lock_get () from /lib64/libdb-5.3.so > > #5 0x00007f53ff075142 in __db_lget () from /lib64/libdb-5.3.so > > #6 0x00007f53fefbc605 in __bam_search () from /lib64/libdb-5.3.so > > #7 0x00007f53fefa7256 in __bamc_search () from /lib64/libdb-5.3.so > > #8 0x00007f53fefa8d0f in __bamc_get () from /lib64/libdb-5.3.so > > #9 0x00007f53ff061ca6 in __dbc_iget () from /lib64/libdb-5.3.so > > #10 0x00007f53ff070b02 in __dbc_get_pp () from /lib64/libdb-5.3.so > > #11 0x00007f53fa54c680 in idl_new_fetch () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #12 0x00007f53fa55ab26 in index_read_ext_allids () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #13 0x00007f53fa545284 in keys2idl () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #14 0x00007f53fa5459e3 in ava_candidates.isra.0 () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #15 0x00007f53fa545fd2 in filter_candidates_ext () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #16 0x00007f53fa547096 in list_candidates () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #17 0x00007f53fa545f40 in filter_candidates_ext () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #18 0x00007f53fa58180c in subtree_candidates () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #19 0x00007f53fa582e7f in ldbm_back_search () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #20 0x00007f5406eb01d6 in op_shared_search () from > /usr/lib64/dirsrv/libslapd.so.0 > #21 0x00007f54073a133e in do_search () > #22 0x00007f540738f567 in connection_threadmain () > #23 0x00007f540509e7bb in _pt_root () from /lib64/libnspr4.so > #24 0x00007f5404a3fdc5 in start_thread () from /lib64/libpthread.so.0 > #25 0x00007f540476d28d in clone () from /lib64/libc.so.6 > Thread 27 (Thread 0x7f53d7ff7700 (LWP 128219)): > #0 0x00007f5404a436d5 in pthread_cond_wait@@GLIBC_2.3.2 () from > /lib64/libpthread.so.0 > #1 0x00007f53fef9f2f3 in __db_hybrid_mutex_suspend () from > /lib64/libdb-5.3.so > #2 0x00007f53fef9e640 in __db_tas_mutex_lock () from > /lib64/libdb-5.3.so > #3 0x00007f53ff048d3a in __lock_get_internal () from > /lib64/libdb-5.3.so > #4 0x00007f53ff049820 in __lock_get () from /lib64/libdb-5.3.so > > #5 0x00007f53ff075142 in __db_lget () from /lib64/libdb-5.3.so > > #6 0x00007f53fefbc605 in __bam_search () from /lib64/libdb-5.3.so > > #7 0x00007f53fefa7256 in __bamc_search () from /lib64/libdb-5.3.so > > #8 0x00007f53fefa8d0f in __bamc_get () from /lib64/libdb-5.3.so > > #9 0x00007f53ff061ca6 in __dbc_iget () from /lib64/libdb-5.3.so > > #10 0x00007f53ff070b02 in __dbc_get_pp () from /lib64/libdb-5.3.so > > #11 0x00007f53fa54c680 in idl_new_fetch () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #12 0x00007f53fa55ab26 in index_read_ext_allids () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #13 0x00007f53fa545284 in keys2idl () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #14 0x00007f53fa5459e3 in ava_candidates.isra.0 () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #15 0x00007f53fa545fd2 in filter_candidates_ext () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #16 0x00007f53fa547096 in list_candidates () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #17 0x00007f53fa545f40 in filter_candidates_ext () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #18 0x00007f53fa58180c in subtree_candidates () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #19 0x00007f53fa582e7f in ldbm_back_search () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #20 0x00007f5406eb01d6 in op_shared_search () from > /usr/lib64/dirsrv/libslapd.so.0 > #21 0x00007f54073a133e in do_search () > #22 0x00007f540738f567 in connection_threadmain () > #23 0x00007f540509e7bb in _pt_root () from /lib64/libnspr4.so > #24 0x00007f5404a3fdc5 in start_thread () from /lib64/libpthread.so.0 > #25 0x00007f540476d28d in clone () from /lib64/libc.so.6 > Thread 26 (Thread 0x7f53d77f6700 (LWP 128220)): > #0 0x00007f5404a436d5 in pthread_cond_wait@@GLIBC_2.3.2 () from > /lib64/libpthread.so.0 > #1 0x00007f53fef9f2f3 in __db_hybrid_mutex_suspend () from > /lib64/libdb-5.3.so > #2 0x00007f53fef9e640 in __db_tas_mutex_lock () from > /lib64/libdb-5.3.so > #3 0x00007f53ff048d3a in __lock_get_internal () from > /lib64/libdb-5.3.so > #4 0x00007f53ff049820 in __lock_get () from /lib64/libdb-5.3.so > > #5 0x00007f53ff075142 in __db_lget () from /lib64/libdb-5.3.so > > #6 0x00007f53fefbc605 in __bam_search () from /lib64/libdb-5.3.so > > #7 0x00007f53fefa7256 in __bamc_search () from /lib64/libdb-5.3.so > > #8 0x00007f53fefa8d0f in __bamc_get () from /lib64/libdb-5.3.so > > #9 0x00007f53ff061ca6 in __dbc_iget () from /lib64/libdb-5.3.so > > #10 0x00007f53ff070b02 in __dbc_get_pp () from /lib64/libdb-5.3.so > > #11 0x00007f53fa54c680 in idl_new_fetch () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #12 0x00007f53fa55ab26 in index_read_ext_allids () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #13 0x00007f53fa545284 in keys2idl () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #14 0x00007f53fa5459e3 in ava_candidates.isra.0 () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #15 0x00007f53fa545fd2 in filter_candidates_ext () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #16 0x00007f53fa547096 in list_candidates () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #17 0x00007f53fa545f40 in filter_candidates_ext () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #18 0x00007f53fa58180c in subtree_candidates () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #19 0x00007f53fa582e7f in ldbm_back_search () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #20 0x00007f5406eb01d6 in op_shared_search () from > /usr/lib64/dirsrv/libslapd.so.0 > #21 0x00007f54073a133e in do_search () > #22 0x00007f540738f567 in connection_threadmain () > #23 0x00007f540509e7bb in _pt_root () from /lib64/libnspr4.so > #24 0x00007f5404a3fdc5 in start_thread () from /lib64/libpthread.so.0 > #25 0x00007f540476d28d in clone () from /lib64/libc.so.6 > Thread 25 (Thread 0x7f53d6ff5700 (LWP 128221)): > #0 0x00007f5404a436d5 in pthread_cond_wait@@GLIBC_2.3.2 () from > /lib64/libpthread.so.0 > #1 0x00007f53fef9f2f3 in __db_hybrid_mutex_suspend () from > /lib64/libdb-5.3.so > #2 0x00007f53fef9e640 in __db_tas_mutex_lock () from > /lib64/libdb-5.3.so > #3 0x00007f53ff048d3a in __lock_get_internal () from > /lib64/libdb-5.3.so > #4 0x00007f53ff049820 in __lock_get () from /lib64/libdb-5.3.so > > #5 0x00007f53ff075142 in __db_lget () from /lib64/libdb-5.3.so > > #6 0x00007f53fefbc605 in __bam_search () from /lib64/libdb-5.3.so > > #7 0x00007f53fefa7256 in __bamc_search () from /lib64/libdb-5.3.so > > #8 0x00007f53fefa8d0f in __bamc_get () from /lib64/libdb-5.3.so > > #9 0x00007f53ff061ca6 in __dbc_iget () from /lib64/libdb-5.3.so > > #10 0x00007f53ff070b02 in __dbc_get_pp () from /lib64/libdb-5.3.so > > #11 0x00007f53fa54c680 in idl_new_fetch () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #12 0x00007f53fa55ab26 in index_read_ext_allids () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #13 0x00007f53fa545284 in keys2idl () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #14 0x00007f53fa5459e3 in ava_candidates.isra.0 () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #15 0x00007f53fa545fd2 in filter_candidates_ext () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #16 0x00007f53fa547096 in list_candidates () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #17 0x00007f53fa545f40 in filter_candidates_ext () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #18 0x00007f53fa58180c in subtree_candidates () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #19 0x00007f53fa582e7f in ldbm_back_search () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #20 0x00007f5406eb01d6 in op_shared_search () from > /usr/lib64/dirsrv/libslapd.so.0 > #21 0x00007f54073a133e in do_search () > #22 0x00007f540738f567 in connection_threadmain () > #23 0x00007f540509e7bb in _pt_root () from /lib64/libnspr4.so > #24 0x00007f5404a3fdc5 in start_thread () from /lib64/libpthread.so.0 > #25 0x00007f540476d28d in clone () from /lib64/libc.so.6 > Thread 24 (Thread 0x7f53d67f4700 (LWP 128222)): > #0 0x00007f5404a436d5 in pthread_cond_wait@@GLIBC_2.3.2 () from > /lib64/libpthread.so.0 > #1 0x00007f53fef9f2f3 in __db_hybrid_mutex_suspend () from > /lib64/libdb-5.3.so > #2 0x00007f53fef9e640 in __db_tas_mutex_lock () from > /lib64/libdb-5.3.so > #3 0x00007f53ff048d3a in __lock_get_internal () from > /lib64/libdb-5.3.so > #4 0x00007f53ff049820 in __lock_get () from /lib64/libdb-5.3.so > > #5 0x00007f53ff075142 in __db_lget () from /lib64/libdb-5.3.so > > #6 0x00007f53fefbc605 in __bam_search () from /lib64/libdb-5.3.so > > #7 0x00007f53fefa7256 in __bamc_search () from /lib64/libdb-5.3.so > > #8 0x00007f53fefa8d0f in __bamc_get () from /lib64/libdb-5.3.so > > #9 0x00007f53ff061ca6 in __dbc_iget () from /lib64/libdb-5.3.so > > #10 0x00007f53ff070b02 in __dbc_get_pp () from /lib64/libdb-5.3.so > > #11 0x00007f53fa54c680 in idl_new_fetch () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #12 0x00007f53fa55ab26 in index_read_ext_allids () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #13 0x00007f53fa545284 in keys2idl () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #14 0x00007f53fa5459e3 in ava_candidates.isra.0 () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #15 0x00007f53fa545fd2 in filter_candidates_ext () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #16 0x00007f53fa547096 in list_candidates () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #17 0x00007f53fa545f40 in filter_candidates_ext () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #18 0x00007f53fa58180c in subtree_candidates () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #19 0x00007f53fa582e7f in ldbm_back_search () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #20 0x00007f5406eb01d6 in op_shared_search () from > /usr/lib64/dirsrv/libslapd.so.0 > #21 0x00007f54073a133e in do_search () > #22 0x00007f540738f567 in connection_threadmain () > #23 0x00007f540509e7bb in _pt_root () from /lib64/libnspr4.so > #24 0x00007f5404a3fdc5 in start_thread () from /lib64/libpthread.so.0 > #25 0x00007f540476d28d in clone () from /lib64/libc.so.6 > Thread 23 (Thread 0x7f53d5ff3700 (LWP 128223)): > #0 0x00007f5404a436d5 in pthread_cond_wait@@GLIBC_2.3.2 () from > /lib64/libpthread.so.0 > #1 0x00007f53fef9f2f3 in __db_hybrid_mutex_suspend () from > /lib64/libdb-5.3.so > #2 0x00007f53fef9e640 in __db_tas_mutex_lock () from > /lib64/libdb-5.3.so > #3 0x00007f53ff0b8e1b in __memp_fget () from /lib64/libdb-5.3.so > > #4 0x00007f53fefbc211 in __bam_search () from /lib64/libdb-5.3.so > > #5 0x00007f53fefa7256 in __bamc_search () from /lib64/libdb-5.3.so > > #6 0x00007f53fefab6c4 in __bamc_put () from /lib64/libdb-5.3.so > > #7 0x00007f53ff0626d5 in __dbc_iput () from /lib64/libdb-5.3.so > > #8 0x00007f53ff05da0e in __db_put () from /lib64/libdb-5.3.so > > #9 0x00007f53ff072fa4 in __db_put_pp () from /lib64/libdb-5.3.so > > #10 0x00007f53fa54d4cf in idl_new_insert_key () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #11 0x00007f53fa55c5b5 in addordel_values_sv.isra.3 () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #12 0x00007f53fa55d25a in index_addordel_values_ext_sv () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #13 0x00007f53fa55d694 in index_addordel_values_sv () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #14 0x00007f53fa55d75a in index_addordel_entry () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #15 0x00007f53fa5605fa in ldbm_back_add () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #16 0x00007f5406e62cca in op_shared_add () from > /usr/lib64/dirsrv/libslapd.so.0 > #17 0x00007f5406e64020 in do_add () from /usr/lib64/dirsrv/libslapd.so.0 > #18 0x00007f540738f303 in connection_threadmain () > #19 0x00007f540509e7bb in _pt_root () from /lib64/libnspr4.so > #20 0x00007f5404a3fdc5 in start_thread () from /lib64/libpthread.so.0 > #21 0x00007f540476d28d in clone () from /lib64/libc.so.6 > Thread 22 (Thread 0x7f53d57f2700 (LWP 128224)): > #0 0x00007f5404a436d5 in pthread_cond_wait@@GLIBC_2.3.2 () from > /lib64/libpthread.so.0 > #1 0x00007f53fef9f2f3 in __db_hybrid_mutex_suspend () from > /lib64/libdb-5.3.so > #2 0x00007f53fef9e640 in __db_tas_mutex_lock () from > /lib64/libdb-5.3.so > #3 0x00007f53ff048d3a in __lock_get_internal () from > /lib64/libdb-5.3.so > #4 0x00007f53ff049820 in __lock_get () from /lib64/libdb-5.3.so > > #5 0x00007f53ff075142 in __db_lget () from /lib64/libdb-5.3.so > > #6 0x00007f53fefbc605 in __bam_search () from /lib64/libdb-5.3.so > > #7 0x00007f53fefa7256 in __bamc_search () from /lib64/libdb-5.3.so > > #8 0x00007f53fefa8d0f in __bamc_get () from /lib64/libdb-5.3.so > > #9 0x00007f53ff061ca6 in __dbc_iget () from /lib64/libdb-5.3.so > > #10 0x00007f53ff070b02 in __dbc_get_pp () from /lib64/libdb-5.3.so > > #11 0x00007f53fa54c680 in idl_new_fetch () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #12 0x00007f53fa55ab26 in index_read_ext_allids () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #13 0x00007f53fa545284 in keys2idl () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #14 0x00007f53fa5459e3 in ava_candidates.isra.0 () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #15 0x00007f53fa545fd2 in filter_candidates_ext () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #16 0x00007f53fa547096 in list_candidates () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #17 0x00007f53fa545f40 in filter_candidates_ext () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #18 0x00007f53fa58180c in subtree_candidates () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #19 0x00007f53fa582e7f in ldbm_back_search () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #20 0x00007f5406eb01d6 in op_shared_search () from > /usr/lib64/dirsrv/libslapd.so.0 > #21 0x00007f54073a133e in do_search () > #22 0x00007f540738f567 in connection_threadmain () > #23 0x00007f540509e7bb in _pt_root () from /lib64/libnspr4.so > #24 0x00007f5404a3fdc5 in start_thread () from /lib64/libpthread.so.0 > #25 0x00007f540476d28d in clone () from /lib64/libc.so.6 > Thread 21 (Thread 0x7f53d4ff1700 (LWP 128225)): > #0 0x00007f5404a436d5 in pthread_cond_wait@@GLIBC_2.3.2 () from > /lib64/libpthread.so.0 > #1 0x00007f53fef9f2f3 in __db_hybrid_mutex_suspend () from > /lib64/libdb-5.3.so > #2 0x00007f53fef9e640 in __db_tas_mutex_lock () from > /lib64/libdb-5.3.so > #3 0x00007f53ff048d3a in __lock_get_internal () from > /lib64/libdb-5.3.so > #4 0x00007f53ff049820 in __lock_get () from /lib64/libdb-5.3.so > > #5 0x00007f53ff075142 in __db_lget () from /lib64/libdb-5.3.so > > #6 0x00007f53fefbc605 in __bam_search () from /lib64/libdb-5.3.so > > #7 0x00007f53fefa7256 in __bamc_search () from /lib64/libdb-5.3.so > > #8 0x00007f53fefa8d0f in __bamc_get () from /lib64/libdb-5.3.so > > #9 0x00007f53ff061ca6 in __dbc_iget () from /lib64/libdb-5.3.so > > #10 0x00007f53ff070b02 in __dbc_get_pp () from /lib64/libdb-5.3.so > > #11 0x00007f53fa54c680 in idl_new_fetch () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #12 0x00007f53fa55ab26 in index_read_ext_allids () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #13 0x00007f53fa545284 in keys2idl () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #14 0x00007f53fa5459e3 in ava_candidates.isra.0 () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #15 0x00007f53fa545fd2 in filter_candidates_ext () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #16 0x00007f53fa547096 in list_candidates () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #17 0x00007f53fa545f40 in filter_candidates_ext () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #18 0x00007f53fa58180c in subtree_candidates () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #19 0x00007f53fa582e7f in ldbm_back_search () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #20 0x00007f5406eb01d6 in op_shared_search () from > /usr/lib64/dirsrv/libslapd.so.0 > #21 0x00007f54073a133e in do_search () > #22 0x00007f540738f567 in connection_threadmain () > #23 0x00007f540509e7bb in _pt_root () from /lib64/libnspr4.so > #24 0x00007f5404a3fdc5 in start_thread () from /lib64/libpthread.so.0 > #25 0x00007f540476d28d in clone () from /lib64/libc.so.6 > Thread 20 (Thread 0x7f53d47f0700 (LWP 128226)): > #0 0x00007f5404a436d5 in pthread_cond_wait@@GLIBC_2.3.2 () from > /lib64/libpthread.so.0 > #1 0x00007f53fef9f2f3 in __db_hybrid_mutex_suspend () from > /lib64/libdb-5.3.so > #2 0x00007f53fef9e640 in __db_tas_mutex_lock () from > /lib64/libdb-5.3.so > #3 0x00007f53ff048d3a in __lock_get_internal () from > /lib64/libdb-5.3.so > #4 0x00007f53ff049820 in __lock_get () from /lib64/libdb-5.3.so > > #5 0x00007f53ff075142 in __db_lget () from /lib64/libdb-5.3.so > > #6 0x00007f53fefbc605 in __bam_search () from /lib64/libdb-5.3.so > > #7 0x00007f53fefa7256 in __bamc_search () from /lib64/libdb-5.3.so > > #8 0x00007f53fefa8d0f in __bamc_get () from /lib64/libdb-5.3.so > > #9 0x00007f53ff061ca6 in __dbc_iget () from /lib64/libdb-5.3.so > > #10 0x00007f53ff070b02 in __dbc_get_pp () from /lib64/libdb-5.3.so > > #11 0x00007f53fa54c680 in idl_new_fetch () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #12 0x00007f53fa55ab26 in index_read_ext_allids () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #13 0x00007f53fa545284 in keys2idl () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #14 0x00007f53fa5459e3 in ava_candidates.isra.0 () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #15 0x00007f53fa545fd2 in filter_candidates_ext () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #16 0x00007f53fa547096 in list_candidates () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #17 0x00007f53fa545f40 in filter_candidates_ext () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #18 0x00007f53fa58180c in subtree_candidates () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #19 0x00007f53fa582e7f in ldbm_back_search () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #20 0x00007f5406eb01d6 in op_shared_search () from > /usr/lib64/dirsrv/libslapd.so.0 > #21 0x00007f5406ec094e in search_internal_callback_pb () from > /usr/lib64/dirsrv/libslapd.so.0 > #22 0x00007f5406ec0be8 in search_internal_pb () from > /usr/lib64/dirsrv/libslapd.so.0 > #23 0x00007f53fe118e90 in search_one_berval () from > /usr/lib64/dirsrv/plugins/libattr-unique-plugin.so > #24 0x00007f53fe119242 in search () from > /usr/lib64/dirsrv/plugins/libattr-unique-plugin.so > #25 0x00007f53fe11954c in searchAllSubtrees () from > /usr/lib64/dirsrv/plugins/libattr-unique-plugin.so > #26 0x00007f53fe11a015 in preop_add () from > /usr/lib64/dirsrv/plugins/libattr-unique-plugin.so > #27 0x00007f5406ebb848 in plugin_call_func () from > /usr/lib64/dirsrv/libslapd.so.0 > #28 0x00007f5406ebbb13 in plugin_call_plugins () from > /usr/lib64/dirsrv/libslapd.so.0 > #29 0x00007f5406e62c2b in op_shared_add () from > /usr/lib64/dirsrv/libslapd.so.0 > #30 0x00007f5406e64020 in do_add () from /usr/lib64/dirsrv/libslapd.so.0 > #31 0x00007f540738f303 in connection_threadmain () > #32 0x00007f540509e7bb in _pt_root () from /lib64/libnspr4.so > #33 0x00007f5404a3fdc5 in start_thread () from /lib64/libpthread.so.0 > #34 0x00007f540476d28d in clone () from /lib64/libc.so.6 > Thread 19 (Thread 0x7f53d3fef700 (LWP 128227)): > #0 0x00007f5404a436d5 in pthread_cond_wait@@GLIBC_2.3.2 () from > /lib64/libpthread.so.0 > #1 0x00007f53fef9f2f3 in __db_hybrid_mutex_suspend () from > /lib64/libdb-5.3.so > #2 0x00007f53fef9e640 in __db_tas_mutex_lock () from > /lib64/libdb-5.3.so > #3 0x00007f53ff048d3a in __lock_get_internal () from > /lib64/libdb-5.3.so > #4 0x00007f53ff049820 in __lock_get () from /lib64/libdb-5.3.so > > #5 0x00007f53ff075142 in __db_lget () from /lib64/libdb-5.3.so > > #6 0x00007f53fefbc605 in __bam_search () from /lib64/libdb-5.3.so > > #7 0x00007f53fefa7256 in __bamc_search () from /lib64/libdb-5.3.so > > #8 0x00007f53fefa8d0f in __bamc_get () from /lib64/libdb-5.3.so > > #9 0x00007f53ff061ca6 in __dbc_iget () from /lib64/libdb-5.3.so > > #10 0x00007f53ff070b02 in __dbc_get_pp () from /lib64/libdb-5.3.so > > #11 0x00007f53fa54c680 in idl_new_fetch () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #12 0x00007f53fa55ab26 in index_read_ext_allids () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #13 0x00007f53fa545284 in keys2idl () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #14 0x00007f53fa5459e3 in ava_candidates.isra.0 () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #15 0x00007f53fa545fd2 in filter_candidates_ext () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #16 0x00007f53fa547096 in list_candidates () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #17 0x00007f53fa545f40 in filter_candidates_ext () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #18 0x00007f53fa58180c in subtree_candidates () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #19 0x00007f53fa582e7f in ldbm_back_search () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #20 0x00007f5406eb01d6 in op_shared_search () from > /usr/lib64/dirsrv/libslapd.so.0 > #21 0x00007f54073a133e in do_search () > #22 0x00007f540738f567 in connection_threadmain () > #23 0x00007f540509e7bb in _pt_root () from /lib64/libnspr4.so > #24 0x00007f5404a3fdc5 in start_thread () from /lib64/libpthread.so.0 > #25 0x00007f540476d28d in clone () from /lib64/libc.so.6 > Thread 18 (Thread 0x7f53d37ee700 (LWP 128228)): > #0 0x00007f5404a436d5 in pthread_cond_wait@@GLIBC_2.3.2 () from > /lib64/libpthread.so.0 > #1 0x00007f53fef9f2f3 in __db_hybrid_mutex_suspend () from > /lib64/libdb-5.3.so > #2 0x00007f53fef9e640 in __db_tas_mutex_lock () from > /lib64/libdb-5.3.so > #3 0x00007f53ff048d3a in __lock_get_internal () from > /lib64/libdb-5.3.so > #4 0x00007f53ff049820 in __lock_get () from /lib64/libdb-5.3.so > > #5 0x00007f53ff075142 in __db_lget () from /lib64/libdb-5.3.so > > #6 0x00007f53fefbc605 in __bam_search () from /lib64/libdb-5.3.so > > #7 0x00007f53fefa7256 in __bamc_search () from /lib64/libdb-5.3.so > > #8 0x00007f53fefa8d0f in __bamc_get () from /lib64/libdb-5.3.so > > #9 0x00007f53ff061ca6 in __dbc_iget () from /lib64/libdb-5.3.so > > #10 0x00007f53ff070b02 in __dbc_get_pp () from /lib64/libdb-5.3.so > > #11 0x00007f53fa54c680 in idl_new_fetch () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #12 0x00007f53fa55ab26 in index_read_ext_allids () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #13 0x00007f53fa545284 in keys2idl () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #14 0x00007f53fa5459e3 in ava_candidates.isra.0 () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #15 0x00007f53fa545fd2 in filter_candidates_ext () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #16 0x00007f53fa547096 in list_candidates () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #17 0x00007f53fa545f40 in filter_candidates_ext () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #18 0x00007f53fa58180c in subtree_candidates () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #19 0x00007f53fa582e7f in ldbm_back_search () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #20 0x00007f5406eb01d6 in op_shared_search () from > /usr/lib64/dirsrv/libslapd.so.0 > #21 0x00007f54073a133e in do_search () > #22 0x00007f540738f567 in connection_threadmain () > #23 0x00007f540509e7bb in _pt_root () from /lib64/libnspr4.so > #24 0x00007f5404a3fdc5 in start_thread () from /lib64/libpthread.so.0 > #25 0x00007f540476d28d in clone () from /lib64/libc.so.6 > Thread 17 (Thread 0x7f53d2fed700 (LWP 128229)): > #0 0x00007f5404a436d5 in pthread_cond_wait@@GLIBC_2.3.2 () from > /lib64/libpthread.so.0 > #1 0x00007f53fef9f2f3 in __db_hybrid_mutex_suspend () from > /lib64/libdb-5.3.so > #2 0x00007f53fef9e640 in __db_tas_mutex_lock () from > /lib64/libdb-5.3.so > #3 0x00007f53ff048d3a in __lock_get_internal () from > /lib64/libdb-5.3.so > #4 0x00007f53ff049820 in __lock_get () from /lib64/libdb-5.3.so > > #5 0x00007f53ff075142 in __db_lget () from /lib64/libdb-5.3.so > > #6 0x00007f53fefbc605 in __bam_search () from /lib64/libdb-5.3.so > > #7 0x00007f53fefa7256 in __bamc_search () from /lib64/libdb-5.3.so > > #8 0x00007f53fefa8d0f in __bamc_get () from /lib64/libdb-5.3.so > > #9 0x00007f53ff061ca6 in __dbc_iget () from /lib64/libdb-5.3.so > > #10 0x00007f53ff070b02 in __dbc_get_pp () from /lib64/libdb-5.3.so > > #11 0x00007f53fa54c680 in idl_new_fetch () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #12 0x00007f53fa55ab26 in index_read_ext_allids () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #13 0x00007f53fa545284 in keys2idl () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #14 0x00007f53fa5459e3 in ava_candidates.isra.0 () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #15 0x00007f53fa545fd2 in filter_candidates_ext () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #16 0x00007f53fa547096 in list_candidates () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #17 0x00007f53fa545f40 in filter_candidates_ext () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #18 0x00007f53fa58180c in subtree_candidates () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #19 0x00007f53fa582e7f in ldbm_back_search () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #20 0x00007f5406eb01d6 in op_shared_search () from > /usr/lib64/dirsrv/libslapd.so.0 > #21 0x00007f54073a133e in do_search () > #22 0x00007f540738f567 in connection_threadmain () > #23 0x00007f540509e7bb in _pt_root () from /lib64/libnspr4.so > #24 0x00007f5404a3fdc5 in start_thread () from /lib64/libpthread.so.0 > #25 0x00007f540476d28d in clone () from /lib64/libc.so.6 > Thread 16 (Thread 0x7f53d27ec700 (LWP 128230)): > #0 0x00007f5404a436d5 in pthread_cond_wait@@GLIBC_2.3.2 () from > /lib64/libpthread.so.0 > #1 0x00007f53fef9f2f3 in __db_hybrid_mutex_suspend () from > /lib64/libdb-5.3.so > #2 0x00007f53fef9e640 in __db_tas_mutex_lock () from > /lib64/libdb-5.3.so > #3 0x00007f53ff048d3a in __lock_get_internal () from > /lib64/libdb-5.3.so > #4 0x00007f53ff049820 in __lock_get () from /lib64/libdb-5.3.so > > #5 0x00007f53ff075142 in __db_lget () from /lib64/libdb-5.3.so > > #6 0x00007f53fefbc605 in __bam_search () from /lib64/libdb-5.3.so > > #7 0x00007f53fefa7256 in __bamc_search () from /lib64/libdb-5.3.so > > #8 0x00007f53fefa8d0f in __bamc_get () from /lib64/libdb-5.3.so > > #9 0x00007f53ff061ca6 in __dbc_iget () from /lib64/libdb-5.3.so > > #10 0x00007f53ff070b02 in __dbc_get_pp () from /lib64/libdb-5.3.so > > #11 0x00007f53fa54c680 in idl_new_fetch () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #12 0x00007f53fa55ab26 in index_read_ext_allids () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #13 0x00007f53fa545284 in keys2idl () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #14 0x00007f53fa5459e3 in ava_candidates.isra.0 () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #15 0x00007f53fa545fd2 in filter_candidates_ext () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #16 0x00007f53fa547096 in list_candidates () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #17 0x00007f53fa545f40 in filter_candidates_ext () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #18 0x00007f53fa58180c in subtree_candidates () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #19 0x00007f53fa582e7f in ldbm_back_search () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #20 0x00007f5406eb01d6 in op_shared_search () from > /usr/lib64/dirsrv/libslapd.so.0 > #21 0x00007f54073a133e in do_search () > #22 0x00007f540738f567 in connection_threadmain () > #23 0x00007f540509e7bb in _pt_root () from /lib64/libnspr4.so > #24 0x00007f5404a3fdc5 in start_thread () from /lib64/libpthread.so.0 > #25 0x00007f540476d28d in clone () from /lib64/libc.so.6 > Thread 15 (Thread 0x7f53d1feb700 (LWP 128231)): > #0 0x00007f5404a436d5 in pthread_cond_wait@@GLIBC_2.3.2 () from > /lib64/libpthread.so.0 > #1 0x00007f53fef9f2f3 in __db_hybrid_mutex_suspend () from > /lib64/libdb-5.3.so > #2 0x00007f53fef9e640 in __db_tas_mutex_lock () from > /lib64/libdb-5.3.so > #3 0x00007f53ff048d3a in __lock_get_internal () from > /lib64/libdb-5.3.so > #4 0x00007f53ff049820 in __lock_get () from /lib64/libdb-5.3.so > > #5 0x00007f53ff075142 in __db_lget () from /lib64/libdb-5.3.so > > #6 0x00007f53fefbc605 in __bam_search () from /lib64/libdb-5.3.so > > #7 0x00007f53fefa7256 in __bamc_search () from /lib64/libdb-5.3.so > > #8 0x00007f53fefa8d0f in __bamc_get () from /lib64/libdb-5.3.so > > #9 0x00007f53ff061ca6 in __dbc_iget () from /lib64/libdb-5.3.so > > #10 0x00007f53ff070b02 in __dbc_get_pp () from /lib64/libdb-5.3.so > > #11 0x00007f53fa54c680 in idl_new_fetch () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #12 0x00007f53fa55ab26 in index_read_ext_allids () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #13 0x00007f53fa545284 in keys2idl () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #14 0x00007f53fa5459e3 in ava_candidates.isra.0 () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #15 0x00007f53fa545fd2 in filter_candidates_ext () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #16 0x00007f53fa547096 in list_candidates () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #17 0x00007f53fa545f40 in filter_candidates_ext () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #18 0x00007f53fa58180c in subtree_candidates () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #19 0x00007f53fa582e7f in ldbm_back_search () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #20 0x00007f5406eb01d6 in op_shared_search () from > /usr/lib64/dirsrv/libslapd.so.0 > #21 0x00007f54073a133e in do_search () > #22 0x00007f540738f567 in connection_threadmain () > #23 0x00007f540509e7bb in _pt_root () from /lib64/libnspr4.so > #24 0x00007f5404a3fdc5 in start_thread () from /lib64/libpthread.so.0 > #25 0x00007f540476d28d in clone () from /lib64/libc.so.6 > Thread 14 (Thread 0x7f53d17ea700 (LWP 128232)): > #0 0x00007f5404a436d5 in pthread_cond_wait@@GLIBC_2.3.2 () from > /lib64/libpthread.so.0 > #1 0x00007f53fef9f2f3 in __db_hybrid_mutex_suspend () from > /lib64/libdb-5.3.so > #2 0x00007f53fef9e640 in __db_tas_mutex_lock () from > /lib64/libdb-5.3.so > #3 0x00007f53ff048d3a in __lock_get_internal () from > /lib64/libdb-5.3.so > #4 0x00007f53ff049820 in __lock_get () from /lib64/libdb-5.3.so > > #5 0x00007f53ff075142 in __db_lget () from /lib64/libdb-5.3.so > > #6 0x00007f53fefbc605 in __bam_search () from /lib64/libdb-5.3.so > > #7 0x00007f53fefa7256 in __bamc_search () from /lib64/libdb-5.3.so > > #8 0x00007f53fefa8d0f in __bamc_get () from /lib64/libdb-5.3.so > > #9 0x00007f53ff061ca6 in __dbc_iget () from /lib64/libdb-5.3.so > > #10 0x00007f53ff070b02 in __dbc_get_pp () from /lib64/libdb-5.3.so > > #11 0x00007f53fa54c680 in idl_new_fetch () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #12 0x00007f53fa55ab26 in index_read_ext_allids () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #13 0x00007f53fa545284 in keys2idl () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #14 0x00007f53fa5459e3 in ava_candidates.isra.0 () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #15 0x00007f53fa545fd2 in filter_candidates_ext () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #16 0x00007f53fa547096 in list_candidates () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #17 0x00007f53fa545f40 in filter_candidates_ext () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #18 0x00007f53fa547096 in list_candidates () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #19 0x00007f53fa545f40 in filter_candidates_ext () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #20 0x00007f53fa547096 in list_candidates () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #21 0x00007f53fa545f40 in filter_candidates_ext () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #22 0x00007f53fa58180c in subtree_candidates () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #23 0x00007f53fa582e7f in ldbm_back_search () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #24 0x00007f5406eb01d6 in op_shared_search () from > /usr/lib64/dirsrv/libslapd.so.0 > #25 0x00007f54073a133e in do_search () > #26 0x00007f540738f3c3 in connection_threadmain () > #27 0x00007f540509e7bb in _pt_root () from /lib64/libnspr4.so > #28 0x00007f5404a3fdc5 in start_thread () from /lib64/libpthread.so.0 > #29 0x00007f540476d28d in clone () from /lib64/libc.so.6 > Thread 13 (Thread 0x7f53d0fe9700 (LWP 128233)): > #0 0x00007f5404a436d5 in pthread_cond_wait@@GLIBC_2.3.2 () from > /lib64/libpthread.so.0 > #1 0x00007f53fef9f2f3 in __db_hybrid_mutex_suspend () from > /lib64/libdb-5.3.so > #2 0x00007f53fef9e640 in __db_tas_mutex_lock () from > /lib64/libdb-5.3.so > #3 0x00007f53ff048d3a in __lock_get_internal () from > /lib64/libdb-5.3.so > #4 0x00007f53ff049820 in __lock_get () from /lib64/libdb-5.3.so > > #5 0x00007f53ff075142 in __db_lget () from /lib64/libdb-5.3.so > > #6 0x00007f53fefbc605 in __bam_search () from /lib64/libdb-5.3.so > > #7 0x00007f53fefa7256 in __bamc_search () from /lib64/libdb-5.3.so > > #8 0x00007f53fefa8d0f in __bamc_get () from /lib64/libdb-5.3.so > > #9 0x00007f53ff061ca6 in __dbc_iget () from /lib64/libdb-5.3.so > > #10 0x00007f53ff070b02 in __dbc_get_pp () from /lib64/libdb-5.3.so > > #11 0x00007f53fa54c680 in idl_new_fetch () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #12 0x00007f53fa55ab26 in index_read_ext_allids () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #13 0x00007f53fa545284 in keys2idl () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #14 0x00007f53fa5459e3 in ava_candidates.isra.0 () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #15 0x00007f53fa545fd2 in filter_candidates_ext () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #16 0x00007f53fa547096 in list_candidates () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #17 0x00007f53fa545f40 in filter_candidates_ext () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #18 0x00007f53fa58180c in subtree_candidates () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #19 0x00007f53fa582e7f in ldbm_back_search () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #20 0x00007f5406eb01d6 in op_shared_search () from > /usr/lib64/dirsrv/libslapd.so.0 > #21 0x00007f54073a133e in do_search () > #22 0x00007f540738f567 in connection_threadmain () > #23 0x00007f540509e7bb in _pt_root () from /lib64/libnspr4.so > #24 0x00007f5404a3fdc5 in start_thread () from /lib64/libpthread.so.0 > #25 0x00007f540476d28d in clone () from /lib64/libc.so.6 > Thread 12 (Thread 0x7f53d07e8700 (LWP 128234)): > #0 0x00007f5404a436d5 in pthread_cond_wait@@GLIBC_2.3.2 () from > /lib64/libpthread.so.0 > #1 0x00007f53fef9f2f3 in __db_hybrid_mutex_suspend () from > /lib64/libdb-5.3.so > #2 0x00007f53fef9e640 in __db_tas_mutex_lock () from > /lib64/libdb-5.3.so > #3 0x00007f53ff048d3a in __lock_get_internal () from > /lib64/libdb-5.3.so > #4 0x00007f53ff049820 in __lock_get () from /lib64/libdb-5.3.so > > #5 0x00007f53ff075142 in __db_lget () from /lib64/libdb-5.3.so > > #6 0x00007f53fefbc605 in __bam_search () from /lib64/libdb-5.3.so > > #7 0x00007f53fefa7256 in __bamc_search () from /lib64/libdb-5.3.so > > #8 0x00007f53fefa8d0f in __bamc_get () from /lib64/libdb-5.3.so > > #9 0x00007f53ff061ca6 in __dbc_iget () from /lib64/libdb-5.3.so > > #10 0x00007f53ff070b02 in __dbc_get_pp () from /lib64/libdb-5.3.so > > #11 0x00007f53fa54c680 in idl_new_fetch () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #12 0x00007f53fa55ab26 in index_read_ext_allids () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #13 0x00007f53fa545284 in keys2idl () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #14 0x00007f53fa5459e3 in ava_candidates.isra.0 () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #15 0x00007f53fa545fd2 in filter_candidates_ext () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #16 0x00007f53fa547096 in list_candidates () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #17 0x00007f53fa545f40 in filter_candidates_ext () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #18 0x00007f53fa58180c in subtree_candidates () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #19 0x00007f53fa582e7f in ldbm_back_search () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #20 0x00007f5406eb01d6 in op_shared_search () from > /usr/lib64/dirsrv/libslapd.so.0 > #21 0x00007f54073a133e in do_search () > #22 0x00007f540738f567 in connection_threadmain () > #23 0x00007f540509e7bb in _pt_root () from /lib64/libnspr4.so > #24 0x00007f5404a3fdc5 in start_thread () from /lib64/libpthread.so.0 > #25 0x00007f540476d28d in clone () from /lib64/libc.so.6 > Thread 11 (Thread 0x7f53cffe7700 (LWP 128235)): > #0 0x00007f5404a436d5 in pthread_cond_wait@@GLIBC_2.3.2 () from > /lib64/libpthread.so.0 > #1 0x00007f53fef9f2f3 in __db_hybrid_mutex_suspend () from > /lib64/libdb-5.3.so > #2 0x00007f53fef9e640 in __db_tas_mutex_lock () from > /lib64/libdb-5.3.so > #3 0x00007f53ff048d3a in __lock_get_internal () from > /lib64/libdb-5.3.so > #4 0x00007f53ff049820 in __lock_get () from /lib64/libdb-5.3.so > > #5 0x00007f53ff075142 in __db_lget () from /lib64/libdb-5.3.so > > #6 0x00007f53fefbc605 in __bam_search () from /lib64/libdb-5.3.so > > #7 0x00007f53fefa7256 in __bamc_search () from /lib64/libdb-5.3.so > > #8 0x00007f53fefa8d0f in __bamc_get () from /lib64/libdb-5.3.so > > #9 0x00007f53ff061ca6 in __dbc_iget () from /lib64/libdb-5.3.so > > #10 0x00007f53ff070b02 in __dbc_get_pp () from /lib64/libdb-5.3.so > > #11 0x00007f53fa54c680 in idl_new_fetch () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #12 0x00007f53fa55ab26 in index_read_ext_allids () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #13 0x00007f53fa545284 in keys2idl () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #14 0x00007f53fa5459e3 in ava_candidates.isra.0 () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #15 0x00007f53fa545fd2 in filter_candidates_ext () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #16 0x00007f53fa547096 in list_candidates () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #17 0x00007f53fa545f40 in filter_candidates_ext () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #18 0x00007f53fa58180c in subtree_candidates () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #19 0x00007f53fa582e7f in ldbm_back_search () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #20 0x00007f5406eb01d6 in op_shared_search () from > /usr/lib64/dirsrv/libslapd.so.0 > #21 0x00007f54073a133e in do_search () > #22 0x00007f540738f567 in connection_threadmain () > #23 0x00007f540509e7bb in _pt_root () from /lib64/libnspr4.so > #24 0x00007f5404a3fdc5 in start_thread () from /lib64/libpthread.so.0 > #25 0x00007f540476d28d in clone () from /lib64/libc.so.6 > Thread 10 (Thread 0x7f53cf7e6700 (LWP 128236)): > #0 0x00007f5404a436d5 in pthread_cond_wait@@GLIBC_2.3.2 () from > /lib64/libpthread.so.0 > #1 0x00007f53fef9f2f3 in __db_hybrid_mutex_suspend () from > /lib64/libdb-5.3.so > #2 0x00007f53fef9e640 in __db_tas_mutex_lock () from > /lib64/libdb-5.3.so > #3 0x00007f53ff048d3a in __lock_get_internal () from > /lib64/libdb-5.3.so > #4 0x00007f53ff049820 in __lock_get () from /lib64/libdb-5.3.so > > #5 0x00007f53ff075142 in __db_lget () from /lib64/libdb-5.3.so > > #6 0x00007f53fefbc605 in __bam_search () from /lib64/libdb-5.3.so > > #7 0x00007f53fefa7256 in __bamc_search () from /lib64/libdb-5.3.so > > #8 0x00007f53fefa8d0f in __bamc_get () from /lib64/libdb-5.3.so > > #9 0x00007f53ff061ca6 in __dbc_iget () from /lib64/libdb-5.3.so > > #10 0x00007f53ff070b02 in __dbc_get_pp () from /lib64/libdb-5.3.so > > #11 0x00007f53fa54c680 in idl_new_fetch () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #12 0x00007f53fa55ab26 in index_read_ext_allids () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #13 0x00007f53fa545284 in keys2idl () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #14 0x00007f53fa5459e3 in ava_candidates.isra.0 () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #15 0x00007f53fa545fd2 in filter_candidates_ext () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #16 0x00007f53fa547096 in list_candidates () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #17 0x00007f53fa545f40 in filter_candidates_ext () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #18 0x00007f53fa58180c in subtree_candidates () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #19 0x00007f53fa582e7f in ldbm_back_search () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #20 0x00007f5406eb01d6 in op_shared_search () from > /usr/lib64/dirsrv/libslapd.so.0 > #21 0x00007f54073a133e in do_search () > #22 0x00007f540738f567 in connection_threadmain () > #23 0x00007f540509e7bb in _pt_root () from /lib64/libnspr4.so > #24 0x00007f5404a3fdc5 in start_thread () from /lib64/libpthread.so.0 > #25 0x00007f540476d28d in clone () from /lib64/libc.so.6 > Thread 9 (Thread 0x7f53cefe5700 (LWP 128237)): > #0 0x00007f5404a436d5 in pthread_cond_wait@@GLIBC_2.3.2 () from > /lib64/libpthread.so.0 > #1 0x00007f53fef9f2f3 in __db_hybrid_mutex_suspend () from > /lib64/libdb-5.3.so > #2 0x00007f53fef9e640 in __db_tas_mutex_lock () from > /lib64/libdb-5.3.so > #3 0x00007f53ff048d3a in __lock_get_internal () from > /lib64/libdb-5.3.so > #4 0x00007f53ff049820 in __lock_get () from /lib64/libdb-5.3.so > > #5 0x00007f53ff075142 in __db_lget () from /lib64/libdb-5.3.so > > #6 0x00007f53fefbc605 in __bam_search () from /lib64/libdb-5.3.so > > #7 0x00007f53fefa7256 in __bamc_search () from /lib64/libdb-5.3.so > > #8 0x00007f53fefa8d0f in __bamc_get () from /lib64/libdb-5.3.so > > #9 0x00007f53ff061ca6 in __dbc_iget () from /lib64/libdb-5.3.so > > #10 0x00007f53ff070b02 in __dbc_get_pp () from /lib64/libdb-5.3.so > > #11 0x00007f53fa54c680 in idl_new_fetch () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #12 0x00007f53fa55ab26 in index_read_ext_allids () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #13 0x00007f53fa545284 in keys2idl () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #14 0x00007f53fa5459e3 in ava_candidates.isra.0 () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #15 0x00007f53fa545fd2 in filter_candidates_ext () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #16 0x00007f53fa547096 in list_candidates () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #17 0x00007f53fa545f40 in filter_candidates_ext () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #18 0x00007f53fa58180c in subtree_candidates () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #19 0x00007f53fa582e7f in ldbm_back_search () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #20 0x00007f5406eb01d6 in op_shared_search () from > /usr/lib64/dirsrv/libslapd.so.0 > #21 0x00007f54073a133e in do_search () > #22 0x00007f540738f567 in connection_threadmain () > #23 0x00007f540509e7bb in _pt_root () from /lib64/libnspr4.so > #24 0x00007f5404a3fdc5 in start_thread () from /lib64/libpthread.so.0 > #25 0x00007f540476d28d in clone () from /lib64/libc.so.6 > Thread 8 (Thread 0x7f53ce7e4700 (LWP 128238)): > #0 0x00007f5404a436d5 in pthread_cond_wait@@GLIBC_2.3.2 () from > /lib64/libpthread.so.0 > #1 0x00007f53fef9f2f3 in __db_hybrid_mutex_suspend () from > /lib64/libdb-5.3.so > #2 0x00007f53fef9e640 in __db_tas_mutex_lock () from > /lib64/libdb-5.3.so > #3 0x00007f53ff048d3a in __lock_get_internal () from > /lib64/libdb-5.3.so > #4 0x00007f53ff049820 in __lock_get () from /lib64/libdb-5.3.so > > #5 0x00007f53ff075142 in __db_lget () from /lib64/libdb-5.3.so > > #6 0x00007f53fefbc605 in __bam_search () from /lib64/libdb-5.3.so > > #7 0x00007f53fefa7256 in __bamc_search () from /lib64/libdb-5.3.so > > #8 0x00007f53fefa8d0f in __bamc_get () from /lib64/libdb-5.3.so > > #9 0x00007f53ff061ca6 in __dbc_iget () from /lib64/libdb-5.3.so > > #10 0x00007f53ff070b02 in __dbc_get_pp () from /lib64/libdb-5.3.so > > #11 0x00007f53fa54c680 in idl_new_fetch () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #12 0x00007f53fa55ab26 in index_read_ext_allids () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #13 0x00007f53fa545284 in keys2idl () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #14 0x00007f53fa5459e3 in ava_candidates.isra.0 () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #15 0x00007f53fa545fd2 in filter_candidates_ext () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #16 0x00007f53fa547096 in list_candidates () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #17 0x00007f53fa545f40 in filter_candidates_ext () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #18 0x00007f53fa58180c in subtree_candidates () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #19 0x00007f53fa582e7f in ldbm_back_search () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #20 0x00007f5406eb01d6 in op_shared_search () from > /usr/lib64/dirsrv/libslapd.so.0 > #21 0x00007f54073a133e in do_search () > #22 0x00007f540738f567 in connection_threadmain () > #23 0x00007f540509e7bb in _pt_root () from /lib64/libnspr4.so > #24 0x00007f5404a3fdc5 in start_thread () from /lib64/libpthread.so.0 > #25 0x00007f540476d28d in clone () from /lib64/libc.so.6 > Thread 7 (Thread 0x7f53cdfe3700 (LWP 128239)): > #0 0x00007f5404a436d5 in pthread_cond_wait@@GLIBC_2.3.2 () from > /lib64/libpthread.so.0 > #1 0x00007f53fef9f2f3 in __db_hybrid_mutex_suspend () from > /lib64/libdb-5.3.so > #2 0x00007f53fef9e640 in __db_tas_mutex_lock () from > /lib64/libdb-5.3.so > #3 0x00007f53ff048d3a in __lock_get_internal () from > /lib64/libdb-5.3.so > #4 0x00007f53ff049820 in __lock_get () from /lib64/libdb-5.3.so > > #5 0x00007f53ff075142 in __db_lget () from /lib64/libdb-5.3.so > > #6 0x00007f53fefbc605 in __bam_search () from /lib64/libdb-5.3.so > > #7 0x00007f53fefa7256 in __bamc_search () from /lib64/libdb-5.3.so > > #8 0x00007f53fefa8d0f in __bamc_get () from /lib64/libdb-5.3.so > > #9 0x00007f53ff061ca6 in __dbc_iget () from /lib64/libdb-5.3.so > > #10 0x00007f53ff070b02 in __dbc_get_pp () from /lib64/libdb-5.3.so > > #11 0x00007f53fa54c680 in idl_new_fetch () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #12 0x00007f53fa55ab26 in index_read_ext_allids () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #13 0x00007f53fa545284 in keys2idl () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #14 0x00007f53fa5459e3 in ava_candidates.isra.0 () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #15 0x00007f53fa545fd2 in filter_candidates_ext () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #16 0x00007f53fa547096 in list_candidates () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #17 0x00007f53fa545f40 in filter_candidates_ext () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #18 0x00007f53fa58180c in subtree_candidates () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #19 0x00007f53fa582e7f in ldbm_back_search () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #20 0x00007f5406eb01d6 in op_shared_search () from > /usr/lib64/dirsrv/libslapd.so.0 > #21 0x00007f54073a133e in do_search () > #22 0x00007f540738f567 in connection_threadmain () > #23 0x00007f540509e7bb in _pt_root () from /lib64/libnspr4.so > #24 0x00007f5404a3fdc5 in start_thread () from /lib64/libpthread.so.0 > #25 0x00007f540476d28d in clone () from /lib64/libc.so.6 > Thread 6 (Thread 0x7f53cd7e2700 (LWP 128240)): > #0 0x00007f5404a436d5 in pthread_cond_wait@@GLIBC_2.3.2 () from > /lib64/libpthread.so.0 > #1 0x00007f53fef9f2f3 in __db_hybrid_mutex_suspend () from > /lib64/libdb-5.3.so > #2 0x00007f53fef9eb37 in __db_tas_mutex_readlock () from > /lib64/libdb-5.3.so > #3 0x00007f53ff0b8d35 in __memp_fget () from /lib64/libdb-5.3.so > > #4 0x00007f53ff062333 in __dbc_iget () from /lib64/libdb-5.3.so > > #5 0x00007f53ff070b02 in __dbc_get_pp () from /lib64/libdb-5.3.so > > #6 0x00007f53fa54c822 in idl_new_fetch () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #7 0x00007f53fa55ab26 in index_read_ext_allids () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #8 0x00007f53fa545284 in keys2idl () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #9 0x00007f53fa5459e3 in ava_candidates.isra.0 () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #10 0x00007f53fa545fd2 in filter_candidates_ext () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #11 0x00007f53fa547096 in list_candidates () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #12 0x00007f53fa545f40 in filter_candidates_ext () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #13 0x00007f53fa547096 in list_candidates () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #14 0x00007f53fa545f40 in filter_candidates_ext () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #15 0x00007f53fa547096 in list_candidates () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #16 0x00007f53fa545f40 in filter_candidates_ext () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #17 0x00007f53fa58180c in subtree_candidates () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #18 0x00007f53fa582e7f in ldbm_back_search () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #19 0x00007f5406eb01d6 in op_shared_search () from > /usr/lib64/dirsrv/libslapd.so.0 > #20 0x00007f54073a133e in do_search () > #21 0x00007f540738f3c3 in connection_threadmain () > #22 0x00007f540509e7bb in _pt_root () from /lib64/libnspr4.so > #23 0x00007f5404a3fdc5 in start_thread () from /lib64/libpthread.so.0 > #24 0x00007f540476d28d in clone () from /lib64/libc.so.6 > Thread 5 (Thread 0x7f53ccfe1700 (LWP 128241)): > #0 0x00007f5404a436d5 in pthread_cond_wait@@GLIBC_2.3.2 () from > /lib64/libpthread.so.0 > #1 0x00007f53fef9f2f3 in __db_hybrid_mutex_suspend () from > /lib64/libdb-5.3.so > #2 0x00007f53fef9e640 in __db_tas_mutex_lock () from > /lib64/libdb-5.3.so > #3 0x00007f53ff048d3a in __lock_get_internal () from > /lib64/libdb-5.3.so > #4 0x00007f53ff049820 in __lock_get () from /lib64/libdb-5.3.so > > #5 0x00007f53ff075142 in __db_lget () from /lib64/libdb-5.3.so > > #6 0x00007f53fefbc605 in __bam_search () from /lib64/libdb-5.3.so > > #7 0x00007f53fefa7256 in __bamc_search () from /lib64/libdb-5.3.so > > #8 0x00007f53fefa8d0f in __bamc_get () from /lib64/libdb-5.3.so > > #9 0x00007f53ff061ca6 in __dbc_iget () from /lib64/libdb-5.3.so > > #10 0x00007f53ff070b02 in __dbc_get_pp () from /lib64/libdb-5.3.so > > #11 0x00007f53fa54c680 in idl_new_fetch () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #12 0x00007f53fa55ab26 in index_read_ext_allids () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #13 0x00007f53fa545284 in keys2idl () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #14 0x00007f53fa5459e3 in ava_candidates.isra.0 () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #15 0x00007f53fa545fd2 in filter_candidates_ext () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #16 0x00007f53fa547096 in list_candidates () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #17 0x00007f53fa545f40 in filter_candidates_ext () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #18 0x00007f53fa547096 in list_candidates () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #19 0x00007f53fa545f40 in filter_candidates_ext () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #20 0x00007f53fa547096 in list_candidates () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #21 0x00007f53fa545f40 in filter_candidates_ext () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #22 0x00007f53fa58180c in subtree_candidates () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #23 0x00007f53fa582e7f in ldbm_back_search () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #24 0x00007f5406eb01d6 in op_shared_search () from > /usr/lib64/dirsrv/libslapd.so.0 > #25 0x00007f54073a133e in do_search () > #26 0x00007f540738f3c3 in connection_threadmain () > #27 0x00007f540509e7bb in _pt_root () from /lib64/libnspr4.so > #28 0x00007f5404a3fdc5 in start_thread () from /lib64/libpthread.so.0 > #29 0x00007f540476d28d in clone () from /lib64/libc.so.6 > Thread 4 (Thread 0x7f53cc7e0700 (LWP 128242)): > #0 0x00007f54047649b3 in select () from /lib64/libc.so.6 > #1 0x00007f5406eef0e9 in DS_Sleep () from /usr/lib64/dirsrv/libslapd.so.0 > #2 0x00007f5407390325 in time_thread () > #3 0x00007f540509e7bb in _pt_root () from /lib64/libnspr4.so > #4 0x00007f5404a3fdc5 in start_thread () from /lib64/libpthread.so.0 > #5 0x00007f540476d28d in clone () from /lib64/libc.so.6 > Thread 3 (Thread 0x7f53cb7a0700 (LWP 128259)): > #0 0x00007f5404a436d5 in pthread_cond_wait@@GLIBC_2.3.2 () from > /lib64/libpthread.so.0 > #1 0x00007f5405099050 in PR_WaitCondVar () from /lib64/libnspr4.so > #2 0x00007f540739bf25 in ps_send_results () > #3 0x00007f540509e7bb in _pt_root () from /lib64/libnspr4.so > #4 0x00007f5404a3fdc5 in start_thread () from /lib64/libpthread.so.0 > #5 0x00007f540476d28d in clone () from /lib64/libc.so.6 > Thread 2 (Thread 0x7f53cab9d700 (LWP 9020)): > #0 0x00007f5404a436d5 in pthread_cond_wait@@GLIBC_2.3.2 () from > /lib64/libpthread.so.0 > #1 0x00007f5405099050 in PR_WaitCondVar () from /lib64/libnspr4.so > #2 0x00007f540739bf25 in ps_send_results () > #3 0x00007f540509e7bb in _pt_root () from /lib64/libnspr4.so > #4 0x00007f5404a3fdc5 in start_thread () from /lib64/libpthread.so.0 > #5 0x00007f540476d28d in clone () from /lib64/libc.so.6 > Thread 1 (Thread 0x7f5407357840 (LWP 128124)): > #0 0x00007f5404762c3d in poll () from /lib64/libc.so.6 > #1 0x00007f540509a967 in _pr_poll_with_poll () from /lib64/libnspr4.so > #2 0x00007f5407393ff1 in slapd_daemon () > #3 0x00007f5407385c6c in main () > > > Thanks > > On Tue, Sep 6, 2016 at 5:51 PM, Rakesh Rajasekharan > > > wrote: > > Hi, > > With script provided by Rich, I looked at my db cache usage and it > does not look to be high enough > > # under each db are the list of selected indexes specified with > INDEXLIST > dbname count free free% size > changelog:ent 5 2076333 99.0 4163.8 > changelog:dn 15976 9303557 88.7 74.0 > userroot:ent 4763 175321074 83.6 7221.1 > userroot:dn 4763 9995056 95.3 103.0 > ipaca:ent 425 6337866 60.4 9759.8 > ipaca:dn 425 10451081 99.7 81.6 > > Right now I set it to 200MB. So, I am planning to bring it down to > 100MB > > Also, /|nsslapd-db-checkpoint-interval |/will change it to 80 or > 90 from default of 60 > > I will check this out and the impact it has on the hang issues and > get back > > Thanks > Rakesh > > On Tue, Sep 6, 2016 at 4:13 PM, Ludwig Krispenz > > wrote: > > Hi, > > I think you are running into this bug: > https://bugzilla.redhat.com/show_bug.cgi?id=1349779 > > it was the outcome of an analysis started in this one: > https://bugzilla.redhat.com/show_bug.cgi?id=1347800 > > > the problem is that it looks like the checkpoint thread is > holding a mutex and otehr threads are blocked on it. It looks > like an issue in BerkeleyDB although there is no explanation > from the BDB code why this could happen, we also had a support > case with Oracle to investigate this, with no solution. Oracle > at the end was providing a diagnostic patch, which we did not > test since the customer with the original problem had worked > around this by changing their configuration. > I don't know the status of this patch and if we can make it > available, probably depends on your support status, you need > to contact support for this. > > About the workarounds: > you can try to change timing for the database update and > checkpointing eg by reducing the dbcache size, changing the > checkpoint interval. In the customer case they identified one > substring index, which was frequently updated but not really > required - and removing this index helped. > > Regards, > Ludwig > > On 09/06/2016 11:31 AM, thierry bordaz wrote: >> Hi, >> >> The following thread is hanging most of the others threads >> >> Thread 20 (Thread 0x7f68007f8700 (LWP 4873)): >> #0 0x00007f683d0506d5 in pthread_cond_wait@@GLIBC_2.3.2 >> () from >> /lib64/libpthread.so.0 >> #1 0x00007f68375ac2f3 in __db_hybrid_mutex_suspend () >> from /lib64/libdb-5.3.so >> #2 0x00007f68375ab640 in __db_tas_mutex_lock () from >> /lib64/libdb-5.3.so >> #3 0x00007f68376c5e1b in __memp_fget () from >> /lib64/libdb-5.3.so >> #4 0x00007f68375c9211 in __bam_search () from >> /lib64/libdb-5.3.so >> #5 0x00007f68375b4256 in __bamc_search () from >> /lib64/libdb-5.3.so >> #6 0x00007f68375b86c4 in __bamc_put () from >> /lib64/libdb-5.3.so >> #7 0x00007f683766f6d5 in __dbc_iput () from >> /lib64/libdb-5.3.so >> #8 0x00007f683766aa0e in __db_put () from >> /lib64/libdb-5.3.so >> #9 0x00007f683767ffa4 in __db_put_pp () from >> /lib64/libdb-5.3.so >> #10 0x00007f6832b5a4cf in idl_new_insert_key () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #11 0x00007f6832b695b5 in addordel_values_sv.isra.3 () >> from /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #12 0x00007f6832b6a25a in index_addordel_values_ext_sv () >> from /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #13 0x00007f6832b6a694 in index_addordel_values_sv () >> from /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #14 0x00007f6832b6a75a in index_addordel_entry () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #15 0x00007f6832b6d5fa in ldbm_back_add () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #16 0x00007f683f46fcca in op_shared_add () from >> /usr/lib64/dirsrv/libslapd.so.0 >> #17 0x00007f683f471020 in do_add () from >> /usr/lib64/dirsrv/libslapd.so.0 >> #18 0x00007f683f99c303 in connection_threadmain () >> #19 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so >> #20 0x00007f683d04cdc5 in start_thread () from >> /lib64/libpthread.so.0 >> #21 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 >> >> This thread is likely not hang on 'db_tas_mutex_lock' as the >> db_lock does not show it waiting for something. >> Unfortunately it is not shown in the output of top -H, did >> you get the complete output ? >> Ludwig mentioned a known issue that DS is not scaling well on >> NUMA arch, do you know if the machine is NUMA ? >> A possibility is that this thread get severely slowed down by >> a NUMA remote lock and showing DS has hung. >> This is why it would be interesting to know if it consums CPU. >> >> thanks >> theirry >> >> On 09/06/2016 10:59 AM, Rakesh Rajasekharan wrote: >>> Hi Thierry/Ludwig, >>> >>> I again hit the hang issue. This is the pstack of the ns-slapd >>> >>> So was able to this time get the pstack and db locks >>> >>> db locks details >>> Default locking region information: >>> 451 Last allocated locker ID >>> 0x7fffffff Current maximum unused locker ID >>> 9 Number of lock modes >>> 800 Initial number of locks allocated >>> 0 Initial number of lockers allocated >>> 800 Initial number of lock objects allocated >>> 10000 Maximum number of locks possible >>> 10000 Maximum number of lockers possible >>> 10000 Maximum number of lock objects possible >>> 800 Current number of locks allocated >>> 366 Current number of lockers allocated >>> 800 Current number of lock objects allocated >>> 160 Number of lock object partitions >>> 8191 Size of object hash table >>> 126 Number of current locks >>> 1005 Maximum number of locks at any one time >>> 33 Maximum number of locks in any one bucket >>> 3987 Maximum number of locks stolen by for an empty >>> partition >>> 75 Maximum number of locks stolen for any one partition >>> 309 Number of current lockers >>> 313 Maximum number of lockers at any one time >>> 121 Number of current lock objects >>> 676 Maximum number of lock objects at any one time >>> 3 Maximum number of lock objects in any one bucket >>> 69 Maximum number of objects stolen by for an empty >>> partition >>> 5 Maximum number of objects stolen for any one >>> partition >>> 22M Total number of locks requested (22009642) >>> 21M Total number of locks released (21992878) >>> 0 Total number of locks upgraded >>> 95 Total number of locks downgraded >>> 3348 Lock requests not available due to conflicts, for >>> which we waited >>> 12646 Lock requests not available due to conflicts, for >>> which we did not wait >>> 0 Number of deadlocks >>> 0 Lock timeout value >>> 0 Number of locks that have timed out >>> 0 Transaction timeout value >>> 0 Number of transactions that have timed out >>> 2MB 560KB Region size >>> 71719 The number of partition locks that required >>> waiting (0%) >>> 23105 The maximum number of times any partition lock >>> was waited for (0%) >>> 0 The number of object queue operations that >>> required waiting (0%) >>> 163 The number of locker allocations that required >>> waiting (0%) >>> 3172 The number of region locks that required waiting (0%) >>> 5 Maximum hash bucket length >>> =-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-= >>> Lock REGINFO information: >>> Environment Region type >>> 1 Region ID >>> /var/lib/dirsrv/slapd-SPRINKLR-COM/db/__db.001 Region name >>> 0x7f67ef869000 Region address >>> 0x7f67ef8690a0 Region allocation head >>> 0x7f67ef8712b0 Region primary address >>> 0 Region maximum allocation >>> 0 Region allocated >>> Region allocations: 1581345 allocations, 0 failures, 1580794 >>> frees, 6 longest >>> Allocations by power-of-two sizes: >>> 1KB 1581329 >>> 2KB 3 >>> 4KB 4 >>> 8KB 3 >>> 16KB 3 >>> 32KB 1 >>> 64KB 0 >>> 128KB 0 >>> 256KB 2 >>> 512KB 0 >>> 1024KB 1 >>> REGION_SHARED Region flags >>> =-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-= >>> Lock region parameters: >>> 2 Lock region region mutex [3172/14M 0% !Own] >>> >>> 16381 locker table size >>> 8191 object table size >>> 34128 obj_off >>> 900216 locker_off >>> 0 need_dd >>> =-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-= >>> Lock conflict matrix: >>> =-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-= >>> Locks grouped by lockers: >>> Locker Mode Count Status ----------------- Object >>> --------------- >>> 2 dd=308 locks held 1 write locks 0 pid/thread >>> 4845/140085720139840 flags 10 priority 100 >>> 2 READ 1 HELD userRoot/id2entry.db >>> handle 0 >>> 3 dd=307 locks held 0 write locks 0 pid/thread >>> 4845/140084829472512 flags 0 priority 100 >>> 4 dd=306 locks held 1 write locks 0 pid/thread >>> 4845/140085720139840 flags 10 priority 100 >>> 4 READ 1 HELD ipaca/id2entry.db handle 0 >>> 5 dd=305 locks held 0 write locks 0 pid/thread >>> 4845/140084837865216 flags 0 priority 100 >>> 6 dd=304 locks held 1 write locks 0 pid/thread >>> 4845/140085720139840 flags 10 priority 100 >>> 6 READ 1 HELD ipaca/entryrdn.db handle 0 >>> 7 dd=303 locks held 0 write locks 0 pid/thread >>> 4845/140084804294400 flags 0 priority 100 >>> 8 dd=302 locks held 0 write locks 0 pid/thread >>> 4845/140084720432896 flags 0 priority 100 >>> 9 dd=301 locks held 0 write locks 0 pid/thread >>> 4845/140084837865216 flags 0 priority 100 >>> a dd=300 locks held 1 write locks 0 pid/thread >>> 4845/140085720139840 flags 10 priority 100 >>> a READ 1 HELD >>> ipaca/vlv#allcertspkitomcatindex.db handle 0 >>> e dd=299 locks held 1 write locks 0 pid/thread >>> 4845/140085720139840 flags 10 priority 100 >>> e READ 1 HELD >>> ipaca/vlv#allnonrevokedcertspkitomcatindex.db handle 0 >>> 10 dd=298 locks held 1 write locks 0 pid/thread >>> 4845/140085720139840 flags 10 priority 100 >>> 10 READ 1 HELD >>> ipaca/vlv#allrevokedcertspkitomcatindex.db handle 0 >>> 11 dd=297 locks held 1 write locks 0 pid/thread >>> 4845/140085720139840 flags 10 priority 100 >>> 11 READ 1 HELD >>> ipaca/vlv#allrevokedcertsnotafterpkitomcatindex.db >>> handle 0 >>> 14 dd=296 locks held 1 write locks 0 pid/thread >>> 4845/140085720139840 flags 10 priority 100 >>> 14 READ 1 HELD >>> ipaca/vlv#allrevokedorrevokedexpiredcertspkitomcatindex.db >>> handle 0 >>> 15 dd=295 locks held 1 write locks 0 pid/thread >>> 4845/140085720139840 flags 10 priority 100 >>> 15 READ 1 HELD >>> ipaca/vlv#allvalidcertspkitomcatindex.db handle 0 >>> 16 dd=294 locks held 1 write locks 0 pid/thread >>> 4845/140085720139840 flags 10 priority 100 >>> 16 READ 1 HELD >>> ipaca/vlv#allvalidcertsnotafterpkitomcatindex.db handle 0 >>> 17 dd=293 locks held 1 write locks 0 pid/thread >>> 4845/140085720139840 flags 10 priority 100 >>> 17 READ 1 HELD >>> ipaca/vlv#allvalidorrevokedcertspkitomcatindex.db >>> handle 0 >>> 18 dd=292 locks held 1 write locks 0 pid/thread >>> 4845/140085720139840 flags 10 priority 100 >>> 18 READ 1 HELD >>> ipaca/vlv#caallpkitomcatindex.db handle 0 >>> 1d dd=291 locks held 1 write locks 0 pid/thread >>> 4845/140085720139840 flags 10 priority 100 >>> 1d READ 1 HELD >>> ipaca/vlv#cacompletepkitomcatindex.db handle 0 >>> 1e dd=290 locks held 1 write locks 0 pid/thread >>> 4845/140085720139840 flags 10 priority 100 >>> 1e READ 1 HELD >>> ipaca/vlv#cacompleteenrollmentpkitomcatindex.db handle 0 >>> 20 dd=289 locks held 1 write locks 0 pid/thread >>> 4845/140085720139840 flags 10 priority 100 >>> 20 READ 1 HELD >>> ipaca/vlv#cacompleterevocationpkitomcatindex.db handle 0 >>> 21 dd=288 locks held 1 write locks 0 pid/thread >>> 4845/140085720139840 flags 10 priority 100 >>> 21 READ 1 HELD >>> ipaca/vlv#caenrollmentpkitomcatindex.db handle 0 >>> 2b dd=287 locks held 1 write locks 0 pid/thread >>> 4845/140085720139840 flags 10 priority 100 >>> 2b READ 1 HELD >>> ipaca/vlv#carevocationpkitomcatindex.db handle 0 >>> 2c dd=286 locks held 1 write locks 0 pid/thread >>> 4845/140085720139840 flags 10 priority 100 >>> 2c READ 1 HELD changelog/id2entry.db >>> handle 0 >>> 2d dd=285 locks held 0 write locks 0 pid/thread >>> 4845/140085719607040 flags 0 priority 100 >>> 2e dd=284 locks held 1 write locks 0 pid/thread >>> 4845/140085720139840 flags 10 priority 100 >>> 2e READ 1 HELD changelog/entryusn.db >>> handle 0 >>> 2f dd=283 locks held 0 write locks 0 pid/thread >>> 4845/140085720139840 flags 0 priority 100 >>> 30 dd=282 locks held 1 write locks 0 pid/thread >>> 4845/140085720139840 flags 10 priority 100 >>> 30 READ 1 HELD userRoot/entryusn.db >>> handle 0 >>> 31 dd=281 locks held 0 write locks 0 pid/thread >>> 4845/140085720139840 flags 0 priority 100 >>> 32 dd=280 locks held 1 write locks 0 pid/thread >>> 4845/140085720139840 flags 10 priority 100 >>> 32 READ 1 HELD ipaca/entryusn.db handle 0 >>> 33 dd=279 locks held 0 write locks 0 pid/thread >>> 4845/140085720139840 flags 0 priority 100 >>> 34 dd=278 locks held 1 write locks 0 pid/thread >>> 4845/140085720139840 flags 10 priority 100 >>> 34 READ 1 HELD userRoot/entryrdn.db >>> handle 0 >>> 35 dd=277 locks held 0 write locks 0 pid/thread >>> 4845/140084544186112 flags 0 priority 100 >>> 36 dd=276 locks held 0 write locks 0 pid/thread >>> 4845/140084636505856 flags 0 priority 100 >>> 37 dd=275 locks held 0 write locks 0 pid/thread >>> 4845/140084854650624 flags 0 priority 100 >>> 38 dd=274 locks held 1 write locks 0 pid/thread >>> 4845/140085720139840 flags 10 priority 100 >>> 38 READ 1 HELD userRoot/objectclass.db >>> handle 0 >>> 39 dd=273 locks held 0 write locks 0 pid/thread >>> 4845/140084569364224 flags 0 priority 100 >>> 3a dd=272 locks held 1 write locks 0 pid/thread >>> 4845/140085720139840 flags 10 priority 100 >>> 3a READ 1 HELD userRoot/ancestorid.db >>> handle 0 >>> 3b dd=271 locks held 0 write locks 0 pid/thread >>> 4845/140084854650624 flags 0 priority 100 >>> 3c dd=270 locks held 1 write locks 0 pid/thread >>> 4845/140085720139840 flags 10 priority 100 >>> 3c READ 1 HELD changelog/entryrdn.db >>> handle 0 >>> 3d dd=269 locks held 0 write locks 0 pid/thread >>> 4845/140085719607040 flags 0 priority 100 >>> 3e dd=268 locks held 0 write locks 0 pid/thread >>> 4845/140084602935040 flags 0 priority 100 >>> 3f dd=267 locks held 0 write locks 0 pid/thread >>> 4845/140085719607040 flags 0 priority 100 >>> 40 dd=266 locks held 1 write locks 0 pid/thread >>> 4845/140085720139840 flags 10 priority 100 >>> 40 READ 1 HELD changelog/objectclass.db >>> handle 0 >>> 41 dd=265 locks held 0 write locks 0 pid/thread >>> 4845/140085719607040 flags 0 priority 100 >>> 42 dd=264 locks held 0 write locks 0 pid/thread >>> 4845/140085719607040 flags 0 priority 100 >>> 43 dd=263 locks held 0 write locks 0 pid/thread >>> 4845/140084653291264 flags 0 priority 100 >>> 44 dd=262 locks held 1 write locks 0 pid/thread >>> 4845/140085720139840 flags 10 priority 100 >>> 44 READ 1 HELD ipaca/objectclass.db >>> handle 0 >>> 45 dd=261 locks held 0 write locks 0 pid/thread >>> 4845/140084577756928 flags 0 priority 100 >>> 46 dd=260 locks held 0 write locks 0 pid/thread >>> 4845/140084720432896 flags 0 priority 100 >>> 47 dd=259 locks held 1 write locks 0 pid/thread >>> 4845/140085720139840 flags 10 priority 100 >>> 47 READ 1 HELD changelog/aci.db handle 0 >>> 48 dd=258 locks held 0 write locks 0 pid/thread >>> 4845/140085720139840 flags 0 priority 100 >>> 49 dd=257 locks held 1 write locks 0 pid/thread >>> 4845/140085720139840 flags 10 priority 100 >>> 49 READ 1 HELD userRoot/aci.db handle 0 >>> 4a dd=256 locks held 0 write locks 0 pid/thread >>> 4845/140085720139840 flags 0 priority 100 >>> 4b dd=255 locks held 0 write locks 0 pid/thread >>> 4845/140084535793408 flags 0 priority 100 >>> 4c dd=254 locks held 1 write locks 0 pid/thread >>> 4845/140085720139840 flags 10 priority 100 >>> 4c READ 1 HELD ipaca/aci.db handle 0 >>> 4d dd=253 locks held 0 write locks 0 pid/thread >>> 4845/140085720139840 flags 0 priority 100 >>> 4e dd=252 locks held 1 write locks 0 pid/thread >>> 4845/140085720139840 flags 10 priority 100 >>> 4e READ 1 HELD userRoot/parentid.db >>> handle 0 >>> 4f dd=251 locks held 0 write locks 0 pid/thread >>> 4845/140085720139840 flags 0 priority 100 >>> 50 dd=250 locks held 0 write locks 0 pid/thread >>> 4845/140085719607040 flags 0 priority 100 >>> 51 dd=249 locks held 1 write locks 0 pid/thread >>> 4845/140085720139840 flags 10 priority 100 >>> 51 READ 1 HELD changelog/nsuniqueid.db >>> handle 0 >>> 52 dd=248 locks held 1 write locks 0 pid/thread >>> 4845/140085720139840 flags 10 priority 100 >>> 52 READ 1 HELD changelog/changenumber.db >>> handle 0 >>> 53 dd=247 locks held 0 write locks 0 pid/thread >>> 4845/140084602935040 flags 0 priority 100 >>> 54 dd=246 locks held 0 write locks 0 pid/thread >>> 4845/140085719607040 flags 0 priority 100 >>> 55 dd=245 locks held 1 write locks 0 pid/thread >>> 4845/140085719607040 flags 10 priority 100 >>> 55 READ 1 HELD changelog/targetuniqueid.db >>> handle 0 >>> 56 dd=244 locks held 1 write locks 0 pid/thread >>> 4845/140085719607040 flags 10 priority 100 >>> 56 READ 1 HELD changelog/parentid.db >>> handle 0 >>> 57 dd=243 locks held 1 write locks 0 pid/thread >>> 4845/140085719607040 flags 10 priority 100 >>> 57 READ 1 HELD changelog/ancestorid.db >>> handle 0 >>> 58 dd=242 locks held 1 write locks 0 pid/thread >>> 4845/140085719607040 flags 10 priority 100 >>> 58 READ 1 HELD changelog/numsubordinates.db >>> handle 0 >>> 59 dd=241 locks held 0 write locks 0 pid/thread >>> 4845/140085719607040 flags 0 priority 100 >>> 5a dd=240 locks held 0 write locks 0 pid/thread >>> 4845/140084544186112 flags 0 priority 100 >>> 5b dd=239 locks held 1 write locks 0 pid/thread >>> 4845/140084577756928 flags 10 priority 100 >>> 5b READ 1 HELD userRoot/krbPrincipalName.db >>> handle 0 >>> 5c dd=238 locks held 0 write locks 0 pid/thread >>> 4845/140084636505856 flags 0 priority 100 >>> 5d dd=237 locks held 0 write locks 0 pid/thread >>> 4845/140084695254784 flags 0 priority 100 >>> 5e dd=236 locks held 1 write locks 0 pid/thread >>> 4845/140084560971520 flags 10 priority 100 >>> 5e READ 1 HELD >>> userRoot/ipakrbprincipalalias.db handle 0 >>> 5f dd=235 locks held 0 write locks 0 pid/thread >>> 4845/140084703647488 flags 0 priority 100 >>> 60 dd=234 locks held 0 write locks 0 pid/thread >>> 4845/140084653291264 flags 0 priority 100 >>> 61 dd=233 locks held 0 write locks 0 pid/thread >>> 4845/140084577756928 flags 0 priority 100 >>> 62 dd=232 locks held 0 write locks 0 pid/thread >>> 4845/140084686862080 flags 0 priority 100 >>> 63 dd=231 locks held 1 write locks 0 pid/thread >>> 4845/140084611327744 flags 10 priority 100 >>> 63 READ 1 HELD changelog/seeAlso.db >>> handle 0 >>> 64 dd=230 locks held 0 write locks 0 pid/thread >>> 4845/140084686862080 flags 0 priority 100 >>> 65 dd=229 locks held 1 write locks 0 pid/thread >>> 4845/140084611327744 flags 10 priority 100 >>> 65 READ 1 HELD userRoot/seeAlso.db >>> handle 0 >>> 66 dd=228 locks held 0 write locks 0 pid/thread >>> 4845/140084636505856 flags 0 priority 100 >>> 67 dd=227 locks held 1 write locks 0 pid/thread >>> 4845/140084611327744 flags 10 priority 100 >>> 67 READ 1 HELD ipaca/seeAlso.db handle 0 >>> 68 dd=226 locks held 0 write locks 0 pid/thread >>> 4845/140084686862080 flags 0 priority 100 >>> 69 dd=225 locks held 1 write locks 0 pid/thread >>> 4845/140084678469376 flags 10 priority 100 >>> 69 READ 1 HELD ipaca/parentid.db handle 0 >>> 6a dd=224 locks held 0 write locks 0 pid/thread >>> 4845/140084678469376 flags 0 priority 100 >>> 6b dd=223 locks held 0 write locks 0 pid/thread >>> 4845/140084829472512 flags 0 priority 100 >>> 6c dd=222 locks held 0 write locks 0 pid/thread >>> 4845/140084653291264 flags 0 priority 100 >>> 6d dd=221 locks held 0 write locks 0 pid/thread >>> 4845/140084569364224 flags 0 priority 100 >>> 6e dd=220 locks held 0 write locks 0 pid/thread >>> 4845/140084636505856 flags 0 priority 100 >>> 6e READ 1 WAIT userRoot/objectclass.db page 18 >>> 6f dd=219 locks held 0 write locks 0 pid/thread >>> 4845/140084544186112 flags 0 priority 100 >>> 70 dd=218 locks held 0 write locks 0 pid/thread >>> 4845/140084569364224 flags 0 priority 100 >>> 71 dd=217 locks held 0 write locks 0 pid/thread >>> 4845/140084619720448 flags 0 priority 100 >>> 72 dd=216 locks held 0 write locks 0 pid/thread >>> 4845/140084552578816 flags 0 priority 100 >>> 73 dd=215 locks held 0 write locks 0 pid/thread >>> 4845/140084829472512 flags 0 priority 100 >>> 74 dd=214 locks held 0 write locks 0 pid/thread >>> 4845/140084628113152 flags 0 priority 100 >>> 75 dd=213 locks held 0 write locks 0 pid/thread >>> 4845/140084686862080 flags 0 priority 100 >>> 76 dd=212 locks held 1 write locks 0 pid/thread >>> 4845/140084560971520 flags 10 priority 100 >>> 76 READ 1 HELD userRoot/ipauniqueid.db >>> handle 0 >>> 77 dd=211 locks held 0 write locks 0 pid/thread >>> 4845/140084661683968 flags 0 priority 100 >>> 78 dd=210 locks held 0 write locks 0 pid/thread >>> 4845/140084846257920 flags 0 priority 100 >>> 79 dd=209 locks held 0 write locks 0 pid/thread >>> 4845/140084628113152 flags 0 priority 100 >>> 7a dd=208 locks held 1 write locks 0 pid/thread >>> 4845/140084560971520 flags 10 priority 100 >>> 7a READ 1 HELD userRoot/nsuniqueid.db >>> handle 0 >>> 7b dd=207 locks held 1 write locks 0 pid/thread >>> 4845/140084560971520 flags 10 priority 100 >>> 7b READ 1 HELD userRoot/cn.db handle 0 >>> 7c dd=206 locks held 1 write locks 0 pid/thread >>> 4845/140084560971520 flags 10 priority 100 >>> 7c READ 1 HELD userRoot/fqdn.db handle 0 >>> 7d dd=205 locks held 1 write locks 0 pid/thread >>> 4845/140084560971520 flags 10 priority 100 >>> 7d READ 1 HELD userRoot/managedby.db >>> handle 0 >>> 7e dd=204 locks held 1 write locks 0 pid/thread >>> 4845/140084560971520 flags 10 priority 100 >>> 7e READ 1 HELD userRoot/numsubordinates.db >>> handle 0 >>> 7f dd=203 locks held 1 write locks 0 pid/thread >>> 4845/140084678469376 flags 10 priority 100 >>> 7f READ 1 HELD userRoot/member.db handle 0 >>> 80 dd=202 locks held 0 write locks 0 pid/thread >>> 4845/140084712040192 flags 0 priority 100 >>> 81 dd=201 locks held 1 write locks 0 pid/thread >>> 4845/140084678469376 flags 10 priority 100 >>> 81 READ 1 HELD userRoot/memberUser.db >>> handle 0 >>> 82 dd=200 locks held 0 write locks 0 pid/thread >>> 4845/140084577756928 flags 0 priority 100 >>> 83 dd=199 locks held 1 write locks 0 pid/thread >>> 4845/140084678469376 flags 10 priority 100 >>> 83 READ 1 HELD userRoot/memberHost.db >>> handle 0 >>> 84 dd=198 locks held 0 write locks 0 pid/thread >>> 4845/140084712040192 flags 0 priority 100 >>> 85 dd=197 locks held 0 write locks 0 pid/thread >>> 4845/140084577756928 flags 0 priority 100 >>> 86 dd=196 locks held 0 write locks 0 pid/thread >>> 4845/140084670076672 flags 0 priority 100 >>> 87 dd=195 locks held 0 write locks 0 pid/thread >>> 4845/140084577756928 flags 0 priority 100 >>> 88 dd=194 locks held 0 write locks 0 pid/thread >>> 4845/140084846257920 flags 0 priority 100 >>> 89 dd=193 locks held 0 write locks 0 pid/thread >>> 4845/140084552578816 flags 0 priority 100 >>> 8a dd=192 locks held 0 write locks 0 pid/thread >>> 4845/140084703647488 flags 0 priority 100 >>> 8b dd=191 locks held 0 write locks 0 pid/thread >>> 4845/140084544186112 flags 0 priority 100 >>> 8c dd=190 locks held 0 write locks 0 pid/thread >>> 4845/140084846257920 flags 0 priority 100 >>> 8d dd=189 locks held 0 write locks 0 pid/thread >>> 4845/140084661683968 flags 0 priority 100 >>> 8e dd=188 locks held 1 write locks 0 pid/thread >>> 4845/140085067675392 flags 10 priority 100 >>> 8e READ 1 HELD userRoot/macAddress.db >>> handle 0 >>> 8f dd=187 locks held 0 write locks 0 pid/thread >>> 4845/140085067675392 flags 0 priority 100 >>> 90 dd=186 locks held 0 write locks 0 pid/thread >>> 4845/140084703647488 flags 0 priority 100 >>> 91 dd=185 locks held 0 write locks 0 pid/thread >>> 4845/140084552578816 flags 0 priority 100 >>> 92 dd=184 locks held 0 write locks 0 pid/thread >>> 4845/140084560971520 flags 0 priority 100 >>> 92 READ 1 WAIT userRoot/objectclass.db >>> page 9 >>> 93 dd=183 locks held 0 write locks 0 pid/thread >>> 4845/140084577756928 flags 0 priority 100 >>> 94 dd=182 locks held 0 write locks 0 pid/thread >>> 4845/140084821079808 flags 0 priority 100 >>> 94 READ 1 WAIT userRoot/objectclass.db page 18 >>> 95 dd=181 locks held 0 write locks 0 pid/thread >>> 4845/140084544186112 flags 0 priority 100 >>> 96 dd=180 locks held 0 write locks 0 pid/thread >>> 4845/140084804294400 flags 0 priority 100 >>> 97 dd=179 locks held 1 write locks 0 pid/thread >>> 4845/140084712040192 flags 10 priority 100 >>> 97 READ 1 HELD userRoot/uid.db handle 0 >>> 98 dd=178 locks held 0 write locks 0 pid/thread >>> 4845/140084552578816 flags 0 priority 100 >>> 99 dd=177 locks held 0 write locks 0 pid/thread >>> 4845/140084577756928 flags 0 priority 100 >>> 9a dd=176 locks held 0 write locks 0 pid/thread >>> 4845/140084602935040 flags 0 priority 100 >>> 9b dd=175 locks held 1 write locks 0 pid/thread >>> 4845/140085076068096 flags 10 priority 100 >>> 9b READ 1 HELD userRoot/uniquemember.db >>> handle 0 >>> 9c dd=174 locks held 1 write locks 0 pid/thread >>> 4845/140085076068096 flags 10 priority 100 >>> 9c READ 1 HELD userRoot/owner.db handle 0 >>> 9d dd=173 locks held 1 write locks 0 pid/thread >>> 4845/140085076068096 flags 10 priority 100 >>> 9d READ 1 HELD userRoot/ipaallowedtarget.db >>> handle 0 >>> 9e dd=172 locks held 1 write locks 0 pid/thread >>> 4845/140085076068096 flags 10 priority 100 >>> 9e READ 1 HELD >>> userRoot/ipaMemberCertProfile.db handle 0 >>> 9f dd=171 locks held 1 write locks 0 pid/thread >>> 4845/140085076068096 flags 10 priority 100 >>> 9f READ 1 HELD userRoot/ipaassignedidview.db >>> handle 0 >>> a0 dd=170 locks held 1 write locks 0 pid/thread >>> 4845/140085076068096 flags 10 priority 100 >>> a0 READ 1 HELD userRoot/secretary.db >>> handle 0 >>> a1 dd=169 locks held 1 write locks 0 pid/thread >>> 4845/140085076068096 flags 10 priority 100 >>> a1 READ 1 HELD userRoot/memberdenycmd.db >>> handle 0 >>> a2 dd=168 locks held 1 write locks 0 pid/thread >>> 4845/140085076068096 flags 10 priority 100 >>> a2 READ 1 HELD userRoot/memberallowcmd.db >>> handle 0 >>> a3 dd=167 locks held 1 write locks 0 pid/thread >>> 4845/140085076068096 flags 10 priority 100 >>> a3 READ 1 HELD userRoot/manager.db >>> handle 0 >>> a4 dd=166 locks held 1 write locks 0 pid/thread >>> 4845/140085076068096 flags 10 priority 100 >>> a4 READ 1 HELD userRoot/ipasudorunas.db >>> handle 0 >>> a5 dd=165 locks held 1 write locks 0 pid/thread >>> 4845/140085076068096 flags 10 priority 100 >>> a5 READ 1 HELD userRoot/ipaMemberCa.db >>> handle 0 >>> a6 dd=164 locks held 1 write locks 0 pid/thread >>> 4845/140085076068096 flags 10 priority 100 >>> a6 READ 1 HELD userRoot/ipasudorunasgroup.db >>> handle 0 >>> a7 dd=163 locks held 1 write locks 0 pid/thread >>> 4845/140085076068096 flags 10 priority 100 >>> a7 READ 1 HELD userRoot/sourcehost.db >>> handle 0 >>> a8 dd=162 locks held 1 write locks 0 pid/thread >>> 4845/140085076068096 flags 10 priority 100 >>> a8 READ 1 HELD userRoot/memberservice.db >>> handle 0 >>> a9 dd=161 locks held 1 write locks 0 pid/thread >>> 4845/140085076068096 flags 10 priority 100 >>> a9 READ 1 HELD >>> userRoot/ipatokenradiusconfiglink.db handle 0 >>> aa dd=160 locks held 0 write locks 0 pid/thread >>> 4845/140084703647488 flags 0 priority 100 >>> ab dd=159 locks held 0 write locks 0 pid/thread >>> 4845/140084821079808 flags 0 priority 100 >>> ac dd=158 locks held 0 write locks 0 pid/thread >>> 4845/140084628113152 flags 0 priority 100 >>> ad dd=157 locks held 0 write locks 0 pid/thread >>> 4845/140084829472512 flags 0 priority 100 >>> ae dd=156 locks held 0 write locks 0 pid/thread >>> 4845/140084829472512 flags 0 priority 100 >>> af dd=155 locks held 0 write locks 0 pid/thread >>> 4845/140084854650624 flags 0 priority 100 >>> b0 dd=154 locks held 0 write locks 0 pid/thread >>> 4845/140084854650624 flags 0 priority 100 >>> b1 dd=153 locks held 0 write locks 0 pid/thread >>> 4845/140084837865216 flags 0 priority 100 >>> b2 dd=152 locks held 0 write locks 0 pid/thread >>> 4845/140084804294400 flags 0 priority 100 >>> b3 dd=151 locks held 0 write locks 0 pid/thread >>> 4845/140084812687104 flags 0 priority 100 >>> b4 dd=150 locks held 0 write locks 0 pid/thread >>> 4845/140084804294400 flags 0 priority 100 >>> b5 dd=149 locks held 0 write locks 0 pid/thread >>> 4845/140084837865216 flags 0 priority 100 >>> b6 dd=148 locks held 0 write locks 0 pid/thread >>> 4845/140084569364224 flags 0 priority 100 >>> b7 dd=147 locks held 0 write locks 0 pid/thread >>> 4845/140084829472512 flags 0 priority 100 >>> b8 dd=146 locks held 0 write locks 0 pid/thread >>> 4845/140084586149632 flags 0 priority 100 >>> b9 dd=145 locks held 0 write locks 0 pid/thread >>> 4845/140084653291264 flags 0 priority 100 >>> ba dd=144 locks held 0 write locks 0 pid/thread >>> 4845/140084636505856 flags 0 priority 100 >>> bb dd=143 locks held 0 write locks 0 pid/thread >>> 4845/140084804294400 flags 0 priority 100 >>> bc dd=142 locks held 0 write locks 0 pid/thread >>> 4845/140084804294400 flags 0 priority 100 >>> bd dd=141 locks held 0 write locks 0 pid/thread >>> 4845/140084695254784 flags 0 priority 100 >>> be dd=140 locks held 0 write locks 0 pid/thread >>> 4845/140084670076672 flags 0 priority 100 >>> bf dd=139 locks held 0 write locks 0 pid/thread >>> 4845/140084628113152 flags 0 priority 100 >>> c0 dd=138 locks held 0 write locks 0 pid/thread >>> 4845/140084661683968 flags 0 priority 100 >>> c1 dd=137 locks held 0 write locks 0 pid/thread >>> 4845/140084712040192 flags 0 priority 100 >>> c2 dd=136 locks held 0 write locks 0 pid/thread >>> 4845/140084661683968 flags 0 priority 100 >>> c3 dd=135 locks held 0 write locks 0 pid/thread >>> 4845/140084577756928 flags 0 priority 100 >>> c4 dd=134 locks held 0 write locks 0 pid/thread >>> 4845/140084854650624 flags 0 priority 100 >>> c5 dd=133 locks held 1 write locks 0 pid/thread >>> 4845/140084644898560 flags 10 priority 100 >>> c5 READ 1 HELD userRoot/uidnumber.db >>> handle 0 >>> c6 dd=132 locks held 0 write locks 0 pid/thread >>> 4845/140084594542336 flags 0 priority 100 >>> c8 dd=131 locks held 1 write locks 0 pid/thread >>> 4845/140084678469376 flags 10 priority 100 >>> c8 READ 1 HELD ipaca/certstatus.db >>> handle 0 >>> c9 dd=130 locks held 0 write locks 0 pid/thread >>> 4845/140084720432896 flags 0 priority 100 >>> ca dd=129 locks held 0 write locks 0 pid/thread >>> 4845/140084619720448 flags 0 priority 100 >>> cb dd=128 locks held 0 write locks 0 pid/thread >>> 4845/140084821079808 flags 0 priority 100 >>> cc dd=127 locks held 0 write locks 0 pid/thread >>> 4845/140084594542336 flags 0 priority 100 >>> cd dd=126 locks held 0 write locks 0 pid/thread >>> 4845/140084560971520 flags 0 priority 100 >>> ce dd=125 locks held 1 write locks 0 pid/thread >>> 4845/140084837865216 flags 10 priority 100 >>> ce READ 1 HELD userRoot/gidnumber.db >>> handle 0 >>> cf dd=124 locks held 0 write locks 0 pid/thread >>> 4845/140084720432896 flags 0 priority 100 >>> d0 dd=123 locks held 0 write locks 0 pid/thread >>> 4845/140084569364224 flags 0 priority 100 >>> d1 dd=122 locks held 1 write locks 0 pid/thread >>> 4845/140084678469376 flags 10 priority 100 >>> d1 READ 1 HELD userRoot/nscpEntryDN.db >>> handle 0 >>> d2 dd=121 locks held 0 write locks 0 pid/thread >>> 4845/140084695254784 flags 0 priority 100 >>> d3 dd=120 locks held 0 write locks 0 pid/thread >>> 4845/140084569364224 flags 0 priority 100 >>> d4 dd=119 locks held 0 write locks 0 pid/thread >>> 4845/140084653291264 flags 0 priority 100 >>> d5 dd=118 locks held 0 write locks 0 pid/thread >>> 4845/140084829472512 flags 0 priority 100 >>> d6 dd=117 locks held 0 write locks 0 pid/thread >>> 4845/140084661683968 flags 0 priority 100 >>> d7 dd=116 locks held 0 write locks 0 pid/thread >>> 4845/140084804294400 flags 0 priority 100 >>> d8 dd=115 locks held 1 write locks 0 pid/thread >>> 4845/140084821079808 flags 10 priority 100 >>> d8 READ 1 HELD ipaca/description.db >>> handle 0 >>> d9 dd=114 locks held 0 write locks 0 pid/thread >>> 4845/140084661683968 flags 0 priority 100 >>> da dd=113 locks held 0 write locks 0 pid/thread >>> 4845/140084837865216 flags 0 priority 100 >>> db dd=112 locks held 1 write locks 0 pid/thread >>> 4845/140084678469376 flags 10 priority 100 >>> db READ 1 HELD ipaca/cn.db handle 0 >>> dc dd=111 locks held 0 write locks 0 pid/thread >>> 4845/140084577756928 flags 0 priority 100 >>> dd dd=110 locks held 1 write locks 0 pid/thread >>> 4845/140084678469376 flags 10 priority 100 >>> dd READ 1 HELD ipaca/ancestorid.db >>> handle 0 >>> de dd=109 locks held 0 write locks 0 pid/thread >>> 4845/140084577756928 flags 0 priority 100 >>> df dd=108 locks held 0 write locks 0 pid/thread >>> 4845/140084703647488 flags 0 priority 100 >>> e0 dd=107 locks held 0 write locks 0 pid/thread >>> 4845/140084569364224 flags 0 priority 100 >>> e1 dd=106 locks held 0 write locks 0 pid/thread >>> 4845/140084569364224 flags 0 priority 100 >>> e2 dd=105 locks held 0 write locks 0 pid/thread >>> 4845/140084821079808 flags 0 priority 100 >>> e3 dd=104 locks held 1 write locks 0 pid/thread >>> 4845/140084686862080 flags 10 priority 100 >>> e3 READ 1 HELD ipaca/serialno.db handle 0 >>> e4 dd=103 locks held 0 write locks 0 pid/thread >>> 4845/140084837865216 flags 0 priority 100 >>> e5 dd=102 locks held 1 write locks 0 pid/thread >>> 4845/140084695254784 flags 10 priority 100 >>> e5 READ 1 HELD ipaca/revInfo.db handle 0 >>> e6 dd=101 locks held 1 write locks 0 pid/thread >>> 4845/140084695254784 flags 10 priority 100 >>> e6 READ 1 HELD ipaca/revokedby.db handle 0 >>> e7 dd=100 locks held 1 write locks 0 pid/thread >>> 4845/140084695254784 flags 10 priority 100 >>> e7 READ 1 HELD ipaca/revokedOn.db handle 0 >>> e8 dd=99 locks held 0 write locks 0 pid/thread >>> 4845/140084829472512 flags 0 priority 100 >>> e9 dd=98 locks held 0 write locks 0 pid/thread >>> 4845/140084846257920 flags 0 priority 100 >>> ea dd=97 locks held 1 write locks 0 pid/thread >>> 4845/140084703647488 flags 10 priority 100 >>> ea READ 1 HELD ipaca/nsuniqueid.db >>> handle 0 >>> eb dd=96 locks held 1 write locks 0 pid/thread >>> 4845/140084703647488 flags 10 priority 100 >>> eb READ 1 HELD ipaca/requestid.db handle 0 >>> ec dd=95 locks held 1 write locks 0 pid/thread >>> 4845/140084703647488 flags 10 priority 100 >>> ec READ 1 HELD ipaca/requeststate.db >>> handle 0 >>> ed dd=94 locks held 1 write locks 0 pid/thread >>> 4845/140084703647488 flags 10 priority 100 >>> ed READ 1 HELD ipaca/dateOfCreate.db >>> handle 0 >>> ee dd=93 locks held 1 write locks 0 pid/thread >>> 4845/140084703647488 flags 10 priority 100 >>> ee READ 1 HELD ipaca/requesttype.db >>> handle 0 >>> ef dd=92 locks held 1 write locks 0 pid/thread >>> 4845/140084703647488 flags 10 priority 100 >>> ef READ 1 HELD ipaca/numsubordinates.db >>> handle 0 >>> f0 dd=91 locks held 1 write locks 0 pid/thread >>> 4845/140084611327744 flags 10 priority 100 >>> f0 READ 1 HELD userRoot/userCertificate.db >>> handle 0 >>> f1 dd=90 locks held 0 write locks 0 pid/thread >>> 4845/140084821079808 flags 0 priority 100 >>> f2 dd=89 locks held 0 write locks 0 pid/thread >>> 4845/140084569364224 flags 0 priority 100 >>> f3 dd=88 locks held 0 write locks 0 pid/thread >>> 4845/140084602935040 flags 0 priority 100 >>> f4 dd=87 locks held 0 write locks 0 pid/thread >>> 4845/140084720432896 flags 0 priority 100 >>> f5 dd=86 locks held 0 write locks 0 pid/thread >>> 4845/140084628113152 flags 0 priority 100 >>> f6 dd=85 locks held 0 write locks 0 pid/thread >>> 4845/140084703647488 flags 0 priority 100 >>> f7 dd=84 locks held 0 write locks 0 pid/thread >>> 4845/140084611327744 flags 0 priority 100 >>> f8 dd=83 locks held 0 write locks 0 pid/thread >>> 4845/140084661683968 flags 0 priority 100 >>> f9 dd=82 locks held 0 write locks 0 pid/thread >>> 4845/140084837865216 flags 0 priority 100 >>> fa dd=81 locks held 0 write locks 0 pid/thread >>> 4845/140084544186112 flags 0 priority 100 >>> fb dd=80 locks held 0 write locks 0 pid/thread >>> 4845/140084703647488 flags 0 priority 100 >>> fc dd=79 locks held 0 write locks 0 pid/thread >>> 4845/140084544186112 flags 0 priority 100 >>> fd dd=78 locks held 0 write locks 0 pid/thread >>> 4845/140084535793408 flags 0 priority 100 >>> fe dd=77 locks held 0 write locks 0 pid/thread >>> 4845/140084586149632 flags 0 priority 100 >>> ff dd=76 locks held 0 write locks 0 pid/thread >>> 4845/140084569364224 flags 0 priority 100 >>> 100 dd=75 locks held 0 write locks 0 pid/thread >>> 4845/140084804294400 flags 0 priority 100 >>> 101 dd=74 locks held 0 write locks 0 pid/thread >>> 4845/140084703647488 flags 0 priority 100 >>> 102 dd=73 locks held 0 write locks 0 pid/thread >>> 4845/140084703647488 flags 0 priority 100 >>> 103 dd=72 locks held 0 write locks 0 pid/thread >>> 4845/140084821079808 flags 0 priority 100 >>> 104 dd=71 locks held 0 write locks 0 pid/thread >>> 4845/140084695254784 flags 0 priority 100 >>> 105 dd=70 locks held 0 write locks 0 pid/thread >>> 4845/140084544186112 flags 0 priority 100 >>> 106 dd=69 locks held 0 write locks 0 pid/thread >>> 4845/140084854650624 flags 0 priority 100 >>> 107 dd=68 locks held 0 write locks 0 pid/thread >>> 4845/140084628113152 flags 0 priority 100 >>> 108 dd=67 locks held 0 write locks 0 pid/thread >>> 4845/140084569364224 flags 0 priority 100 >>> 109 dd=66 locks held 0 write locks 0 pid/thread >>> 4845/140084594542336 flags 0 priority 100 >>> 10a dd=65 locks held 0 write locks 0 pid/thread >>> 4845/140084661683968 flags 0 priority 100 >>> 10b dd=64 locks held 0 write locks 0 pid/thread >>> 4845/140084653291264 flags 0 priority 100 >>> 10c dd=63 locks held 0 write locks 0 pid/thread >>> 4845/140084695254784 flags 0 priority 100 >>> 10d dd=62 locks held 0 write locks 0 pid/thread >>> 4845/140084720432896 flags 0 priority 100 >>> 10e dd=61 locks held 0 write locks 0 pid/thread >>> 4845/140084619720448 flags 0 priority 100 >>> 10f dd=60 locks held 0 write locks 0 pid/thread >>> 4845/140084720432896 flags 0 priority 100 >>> 110 dd=59 locks held 0 write locks 0 pid/thread >>> 4845/140084569364224 flags 0 priority 100 >>> 111 dd=58 locks held 0 write locks 0 pid/thread >>> 4845/140084695254784 flags 0 priority 100 >>> 112 dd=57 locks held 0 write locks 0 pid/thread >>> 4845/140084653291264 flags 0 priority 100 >>> 113 dd=56 locks held 0 write locks 0 pid/thread >>> 4845/140084712040192 flags 0 priority 100 >>> 114 dd=55 locks held 0 write locks 0 pid/thread >>> 4845/140084686862080 flags 0 priority 100 >>> 115 dd=54 locks held 0 write locks 0 pid/thread >>> 4845/140084577756928 flags 0 priority 100 >>> 115 READ 1 WAIT userRoot/objectclass.db >>> page 9 >>> 116 dd=53 locks held 0 write locks 0 pid/thread >>> 4845/140084837865216 flags 0 priority 100 >>> 116 READ 1 WAIT userRoot/objectclass.db >>> page 9 >>> 117 dd=52 locks held 0 write locks 0 pid/thread >>> 4845/140084577756928 flags 0 priority 100 >>> 118 dd=51 locks held 0 write locks 0 pid/thread >>> 4845/140084569364224 flags 0 priority 100 >>> 119 dd=50 locks held 0 write locks 0 pid/thread >>> 4845/140084560971520 flags 0 priority 100 >>> 11a dd=49 locks held 1 write locks 0 pid/thread >>> 4845/140084695254784 flags 10 priority 100 >>> 11a READ 1 HELD ipaca/metaInfo.db handle 0 >>> 11b dd=48 locks held 1 write locks 0 pid/thread >>> 4845/140084695254784 flags 10 priority 100 >>> 11b READ 1 HELD ipaca/notbefore.db handle 0 >>> 11c dd=47 locks held 1 write locks 0 pid/thread >>> 4845/140084695254784 flags 10 priority 100 >>> 11c READ 1 HELD ipaca/notafter.db handle 0 >>> 11d dd=46 locks held 1 write locks 0 pid/thread >>> 4845/140084695254784 flags 10 priority 100 >>> 11d READ 1 HELD ipaca/duration.db handle 0 >>> 11e dd=45 locks held 1 write locks 0 pid/thread >>> 4845/140084695254784 flags 10 priority 100 >>> 11e READ 1 HELD ipaca/subjectname.db >>> handle 0 >>> 11f dd=44 locks held 1 write locks 0 pid/thread >>> 4845/140084695254784 flags 10 priority 100 >>> 11f READ 1 HELD ipaca/publicKeyData.db >>> handle 0 >>> 120 dd=43 locks held 1 write locks 0 pid/thread >>> 4845/140084695254784 flags 10 priority 100 >>> 120 READ 1 HELD ipaca/extension.db handle 0 >>> 121 dd=42 locks held 1 write locks 0 pid/thread >>> 4845/140084695254784 flags 10 priority 100 >>> 121 READ 1 HELD ipaca/issuedby.db handle 0 >>> 122 dd=41 locks held 0 write locks 0 pid/thread >>> 4845/140084712040192 flags 0 priority 100 >>> 123 dd=40 locks held 0 write locks 0 pid/thread >>> 4845/140084812687104 flags 0 priority 100 >>> 124 dd=39 locks held 0 write locks 0 pid/thread >>> 4845/140084544186112 flags 0 priority 100 >>> 125 dd=38 locks held 0 write locks 0 pid/thread >>> 4845/140084686862080 flags 0 priority 100 >>> 126 dd=37 locks held 0 write locks 0 pid/thread >>> 4845/140084552578816 flags 0 priority 100 >>> 128 dd=36 locks held 0 write locks 0 pid/thread >>> 4845/140084686862080 flags 0 priority 100 >>> 12a dd=35 locks held 0 write locks 0 pid/thread >>> 4845/140084720432896 flags 0 priority 100 >>> 12b dd=34 locks held 0 write locks 0 pid/thread >>> 4845/140084552578816 flags 0 priority 100 >>> 12c dd=33 locks held 0 write locks 0 pid/thread >>> 4845/140084670076672 flags 0 priority 100 >>> 12d dd=32 locks held 0 write locks 0 pid/thread >>> 4845/140084577756928 flags 0 priority 100 >>> 12e dd=31 locks held 0 write locks 0 pid/thread >>> 4845/140084712040192 flags 0 priority 100 >>> 12f dd=30 locks held 0 write locks 0 pid/thread >>> 4845/140084670076672 flags 0 priority 100 >>> 130 dd=29 locks held 0 write locks 0 pid/thread >>> 4845/140084577756928 flags 0 priority 100 >>> 131 dd=28 locks held 0 write locks 0 pid/thread >>> 4845/140084560971520 flags 0 priority 100 >>> 132 dd=27 locks held 0 write locks 0 pid/thread >>> 4845/140084670076672 flags 0 priority 100 >>> 133 dd=26 locks held 0 write locks 0 pid/thread >>> 4845/140084602935040 flags 0 priority 100 >>> 134 dd=25 locks held 0 write locks 0 pid/thread >>> 4845/140084552578816 flags 0 priority 100 >>> 135 dd=24 locks held 0 write locks 0 pid/thread >>> 4845/140084678469376 flags 0 priority 100 >>> 136 dd=23 locks held 0 write locks 0 pid/thread >>> 4845/140084544186112 flags 0 priority 100 >>> 137 dd=22 locks held 0 write locks 0 pid/thread >>> 4845/140084577756928 flags 0 priority 100 >>> 138 dd=21 locks held 0 write locks 0 pid/thread >>> 4845/140084703647488 flags 0 priority 100 >>> 139 dd=20 locks held 0 write locks 0 pid/thread >>> 4845/140084678469376 flags 0 priority 100 >>> 13b dd=19 locks held 0 write locks 0 pid/thread >>> 4845/140084812687104 flags 0 priority 100 >>> 144 dd=18 locks held 0 write locks 0 pid/thread >>> 4845/140084670076672 flags 0 priority 100 >>> 156 dd=17 locks held 0 write locks 0 pid/thread >>> 4845/140084594542336 flags 0 priority 100 >>> 15a dd=16 locks held 0 write locks 0 pid/thread >>> 4845/140084577756928 flags 0 priority 100 >>> 168 dd=15 locks held 0 write locks 0 pid/thread >>> 4845/140084653291264 flags 0 priority 100 >>> 16a dd=14 locks held 0 write locks 0 pid/thread >>> 4845/140084829472512 flags 0 priority 100 >>> 17d dd=13 locks held 0 write locks 0 pid/thread >>> 4845/140084829472512 flags 0 priority 100 >>> 17e dd=12 locks held 0 write locks 0 pid/thread >>> 4845/140084829472512 flags 0 priority 100 >>> 17f dd=11 locks held 0 write locks 0 pid/thread >>> 4845/140084586149632 flags 0 priority 100 >>> 180 dd=10 locks held 0 write locks 0 pid/thread >>> 4845/140084594542336 flags 0 priority 100 >>> 181 dd= 9 locks held 0 write locks 0 pid/thread >>> 4845/140084577756928 flags 0 priority 100 >>> 193 dd= 8 locks held 0 write locks 0 pid/thread >>> 4845/140084812687104 flags 0 priority 100 >>> 196 dd= 7 locks held 0 write locks 0 pid/thread >>> 4845/140084695254784 flags 0 priority 100 >>> 1a1 dd= 6 locks held 0 write locks 0 pid/thread >>> 4845/140084720432896 flags 0 priority 100 >>> 1a2 dd= 5 locks held 0 write locks 0 pid/thread >>> 4845/140084577756928 flags 0 priority 100 >>> 1a3 dd= 4 locks held 0 write locks 0 pid/thread >>> 4845/140084812687104 flags 0 priority 100 >>> 1a4 dd= 3 locks held 0 write locks 0 pid/thread >>> 4845/140084812687104 flags 0 priority 100 >>> 1a5 dd= 2 locks held 0 write locks 0 pid/thread >>> 4845/140084577756928 flags 0 priority 100 >>> 1a6 dd= 1 locks held 0 write locks 0 pid/thread >>> 4845/140084594542336 flags 0 priority 100 >>> 800b3a33 dd= 0 locks held 26 write locks 22 pid/thread >>> 4845/140084661683968 flags 0 priority 100 >>> 800b3a33 WRITE 1 HELD userRoot/objectclass.db >>> page 9 >>> 800b3a33 WRITE 1 HELD userRoot/objectclass.db >>> page 8 >>> 800b3a33 WRITE 1 HELD userRoot/objectclass.db page 18 >>> 800b3a33 WRITE 1 HELD userRoot/objectclass.db >>> page 7 >>> 800b3a33 WRITE 1 HELD userRoot/objectclass.db page 14 >>> 800b3a33 WRITE 1 HELD userRoot/objectclass.db page 11 >>> 800b3a33 WRITE 2 HELD userRoot/objectclass.db >>> page 3 >>> 800b3a33 WRITE 1 HELD userRoot/cn.db page 150 >>> 800b3a33 WRITE 1 HELD userRoot/cn.db page 89 >>> 800b3a33 WRITE 1 HELD userRoot/cn.db page 53 >>> 800b3a33 WRITE 1 HELD userRoot/cn.db page 131 >>> 800b3a33 WRITE 1 HELD userRoot/cn.db page 117 >>> 800b3a33 WRITE 1 HELD userRoot/cn.db page 11 >>> 800b3a33 WRITE 2 HELD userRoot/cn.db page 141 >>> 800b3a33 WRITE 1 HELD userRoot/cn.db page 41 >>> 800b3a33 WRITE 1 HELD userRoot/cn.db page 40 >>> 800b3a33 WRITE 1 HELD userRoot/cn.db page 143 >>> 800b3a33 WRITE 1 HELD userRoot/cn.db page 30 >>> 800b3a33 WRITE 2 HELD userRoot/id2entry.db page >>> 8184 >>> 800b3a33 WRITE 1 HELD userRoot/id2entry.db page >>> 8183 >>> 800b3a33 WRITE 1 HELD userRoot/id2entry.db page >>> 7779 >>> 800b3a33 WRITE 1 HELD userRoot/id2entry.db >>> page 0 >>> 800b3a33 READ 3 HELD userRoot/entryrdn.db >>> page 3 >>> 800b3a33 READ 1 HELD userRoot/entryrdn.db page 36 >>> 800b3a33 READ 1 HELD userRoot/entryrdn.db >>> page 4 >>> 800b3a33 READ 1 HELD userRoot/nsuniqueid.db page 86 >>> =-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-= >>> Locks grouped by object: >>> Locker Mode Count Status ----------------- Object >>> --------------- >>> a4 READ 1 HELD userRoot/ipasudorunas.db >>> handle 0 >>> >>> 9e READ 1 HELD >>> userRoot/ipaMemberCertProfile.db handle 0 >>> >>> 10 READ 1 HELD >>> ipaca/vlv#allrevokedcertspkitomcatindex.db handle 0 >>> >>> 800b3a33 WRITE 2 HELD userRoot/id2entry.db page >>> 8184 >>> >>> 800b3a33 WRITE 1 HELD userRoot/id2entry.db page >>> 8183 >>> >>> 800b3a33 WRITE 1 HELD userRoot/id2entry.db page >>> 7779 >>> >>> a1 READ 1 HELD userRoot/memberdenycmd.db >>> handle 0 >>> >>> a READ 1 HELD >>> ipaca/vlv#allcertspkitomcatindex.db handle 0 >>> >>> a7 READ 1 HELD userRoot/sourcehost.db >>> handle 0 >>> >>> 57 READ 1 HELD changelog/ancestorid.db >>> handle 0 >>> >>> e5 READ 1 HELD ipaca/revInfo.db handle 0 >>> >>> e6 READ 1 HELD ipaca/revokedby.db handle 0 >>> >>> 11 READ 1 HELD >>> ipaca/vlv#allrevokedcertsnotafterpkitomcatindex.db >>> handle 0 >>> >>> 4e READ 1 HELD userRoot/parentid.db >>> handle 0 >>> >>> 4 READ 1 HELD ipaca/id2entry.db handle 0 >>> >>> 58 READ 1 HELD changelog/numsubordinates.db >>> handle 0 >>> >>> 67 READ 1 HELD ipaca/seeAlso.db handle 0 >>> >>> 2b READ 1 HELD >>> ipaca/vlv#carevocationpkitomcatindex.db handle 0 >>> >>> ec READ 1 HELD ipaca/requeststate.db >>> handle 0 >>> >>> 11b READ 1 HELD ipaca/notbefore.db handle 0 >>> >>> 7e READ 1 HELD userRoot/numsubordinates.db >>> handle 0 >>> >>> a2 READ 1 HELD userRoot/memberallowcmd.db >>> handle 0 >>> >>> 9c READ 1 HELD userRoot/owner.db handle 0 >>> >>> 1e READ 1 HELD >>> ipaca/vlv#cacompleteenrollmentpkitomcatindex.db handle 0 >>> >>> 11c READ 1 HELD ipaca/notafter.db handle 0 >>> >>> 40 READ 1 HELD changelog/objectclass.db >>> handle 0 >>> >>> 2e READ 1 HELD changelog/entryusn.db >>> handle 0 >>> >>> 16 READ 1 HELD >>> ipaca/vlv#allvalidcertsnotafterpkitomcatindex.db handle 0 >>> >>> 15 READ 1 HELD >>> ipaca/vlv#allvalidcertspkitomcatindex.db handle 0 >>> >>> a5 READ 1 HELD userRoot/ipaMemberCa.db >>> handle 0 >>> >>> 18 READ 1 HELD >>> ipaca/vlv#caallpkitomcatindex.db handle 0 >>> >>> a0 READ 1 HELD userRoot/secretary.db >>> handle 0 >>> >>> 81 READ 1 HELD userRoot/memberUser.db >>> handle 0 >>> >>> 9f READ 1 HELD userRoot/ipaassignedidview.db >>> handle 0 >>> >>> a3 READ 1 HELD userRoot/manager.db >>> handle 0 >>> >>> d8 READ 1 HELD ipaca/description.db >>> handle 0 >>> >>> a8 READ 1 HELD userRoot/memberservice.db >>> handle 0 >>> >>> 44 READ 1 HELD ipaca/objectclass.db >>> handle 0 >>> >>> ed READ 1 HELD ipaca/dateOfCreate.db >>> handle 0 >>> >>> 800b3a33 WRITE 1 HELD userRoot/id2entry.db >>> page 0 >>> >>> 2 READ 1 HELD userRoot/id2entry.db >>> handle 0 >>> >>> 5b READ 1 HELD userRoot/krbPrincipalName.db >>> handle 0 >>> >>> ee READ 1 HELD ipaca/requesttype.db >>> handle 0 >>> >>> c8 READ 1 HELD ipaca/certstatus.db >>> handle 0 >>> >>> 56 READ 1 HELD changelog/parentid.db >>> handle 0 >>> >>> 83 READ 1 HELD userRoot/memberHost.db >>> handle 0 >>> >>> 5e READ 1 HELD >>> userRoot/ipakrbprincipalalias.db handle 0 >>> >>> 121 READ 1 HELD ipaca/issuedby.db handle 0 >>> >>> 6 READ 1 HELD ipaca/entryrdn.db handle 0 >>> >>> e7 READ 1 HELD ipaca/revokedOn.db handle 0 >>> >>> 4c READ 1 HELD ipaca/aci.db handle 0 >>> >>> 11d READ 1 HELD ipaca/duration.db handle 0 >>> >>> 14 READ 1 HELD >>> ipaca/vlv#allrevokedorrevokedexpiredcertspkitomcatindex.db >>> handle 0 >>> >>> 21 READ 1 HELD >>> ipaca/vlv#caenrollmentpkitomcatindex.db handle 0 >>> >>> 55 READ 1 HELD changelog/targetuniqueid.db >>> handle 0 >>> >>> 800b3a33 READ 1 HELD userRoot/entryrdn.db page 36 >>> >>> 2c READ 1 HELD changelog/id2entry.db >>> handle 0 >>> >>> 800b3a33 READ 1 HELD userRoot/entryrdn.db >>> page 4 >>> >>> 34 READ 1 HELD userRoot/entryrdn.db >>> handle 0 >>> >>> 800b3a33 READ 3 HELD userRoot/entryrdn.db >>> page 3 >>> >>> eb READ 1 HELD ipaca/requestid.db handle 0 >>> >>> 49 READ 1 HELD userRoot/aci.db handle 0 >>> >>> 76 READ 1 HELD userRoot/ipauniqueid.db >>> handle 0 >>> >>> 63 READ 1 HELD changelog/seeAlso.db >>> handle 0 >>> >>> 7c READ 1 HELD userRoot/fqdn.db handle 0 >>> >>> db READ 1 HELD ipaca/cn.db handle 0 >>> >>> ef READ 1 HELD ipaca/numsubordinates.db >>> handle 0 >>> >>> 32 READ 1 HELD ipaca/entryusn.db handle 0 >>> >>> 1d READ 1 HELD >>> ipaca/vlv#cacompletepkitomcatindex.db handle 0 >>> >>> dd READ 1 HELD ipaca/ancestorid.db >>> handle 0 >>> >>> a9 READ 1 HELD >>> userRoot/ipatokenradiusconfiglink.db handle 0 >>> >>> f0 READ 1 HELD userRoot/userCertificate.db >>> handle 0 >>> >>> 11a READ 1 HELD ipaca/metaInfo.db handle 0 >>> >>> 7d READ 1 HELD userRoot/managedby.db >>> handle 0 >>> >>> ea READ 1 HELD ipaca/nsuniqueid.db >>> handle 0 >>> >>> c5 READ 1 HELD userRoot/uidnumber.db >>> handle 0 >>> >>> d1 READ 1 HELD userRoot/nscpEntryDN.db >>> handle 0 >>> >>> 120 READ 1 HELD ipaca/extension.db handle 0 >>> >>> 800b3a33 WRITE 1 HELD userRoot/cn.db page 150 >>> >>> 69 READ 1 HELD ipaca/parentid.db handle 0 >>> >>> 800b3a33 WRITE 1 HELD userRoot/cn.db page 131 >>> >>> 800b3a33 WRITE 1 HELD userRoot/cn.db page 143 >>> >>> 800b3a33 WRITE 2 HELD userRoot/cn.db page 141 >>> >>> 20 READ 1 HELD >>> ipaca/vlv#cacompleterevocationpkitomcatindex.db handle 0 >>> >>> 8e READ 1 HELD userRoot/macAddress.db >>> handle 0 >>> >>> e READ 1 HELD >>> ipaca/vlv#allnonrevokedcertspkitomcatindex.db handle 0 >>> >>> 17 READ 1 HELD >>> ipaca/vlv#allvalidorrevokedcertspkitomcatindex.db >>> handle 0 >>> >>> 800b3a33 WRITE 1 HELD userRoot/cn.db page 30 >>> >>> 800b3a33 WRITE 1 HELD userRoot/objectclass.db page 14 >>> >>> 800b3a33 WRITE 1 HELD userRoot/objectclass.db >>> page 9 >>> 116 READ 1 WAIT userRoot/objectclass.db >>> page 9 >>> 115 READ 1 WAIT userRoot/objectclass.db >>> page 9 >>> 92 READ 1 WAIT userRoot/objectclass.db >>> page 9 >>> >>> 800b3a33 WRITE 1 HELD userRoot/objectclass.db >>> page 8 >>> >>> 7b READ 1 HELD userRoot/cn.db handle 0 >>> >>> 800b3a33 WRITE 1 HELD userRoot/objectclass.db page 11 >>> >>> 800b3a33 WRITE 1 HELD userRoot/objectclass.db >>> page 7 >>> >>> 800b3a33 WRITE 1 HELD userRoot/cn.db page 11 >>> >>> 38 READ 1 HELD userRoot/objectclass.db >>> handle 0 >>> >>> 800b3a33 WRITE 2 HELD userRoot/objectclass.db >>> page 3 >>> >>> 800b3a33 WRITE 1 HELD userRoot/cn.db page 53 >>> >>> 800b3a33 WRITE 1 HELD userRoot/objectclass.db page 18 >>> 6e READ 1 WAIT userRoot/objectclass.db page 18 >>> 94 READ 1 WAIT userRoot/objectclass.db page 18 >>> >>> 800b3a33 WRITE 1 HELD userRoot/cn.db page 41 >>> >>> 800b3a33 WRITE 1 HELD userRoot/cn.db page 40 >>> >>> 800b3a33 WRITE 1 HELD userRoot/cn.db page 89 >>> >>> 800b3a33 WRITE 1 HELD userRoot/cn.db page 117 >>> >>> 30 READ 1 HELD userRoot/entryusn.db >>> handle 0 >>> >>> 3c READ 1 HELD changelog/entryrdn.db >>> handle 0 >>> >>> 97 READ 1 HELD userRoot/uid.db handle 0 >>> >>> 47 READ 1 HELD changelog/aci.db handle 0 >>> >>> e3 READ 1 HELD ipaca/serialno.db handle 0 >>> >>> 7a READ 1 HELD userRoot/nsuniqueid.db >>> handle 0 >>> >>> 800b3a33 READ 1 HELD userRoot/nsuniqueid.db page 86 >>> >>> 51 READ 1 HELD changelog/nsuniqueid.db >>> handle 0 >>> >>> 11e READ 1 HELD ipaca/subjectname.db >>> handle 0 >>> >>> 7f READ 1 HELD userRoot/member.db handle 0 >>> >>> 11f READ 1 HELD ipaca/publicKeyData.db >>> handle 0 >>> >>> 9b READ 1 HELD userRoot/uniquemember.db >>> handle 0 >>> >>> 52 READ 1 HELD changelog/changenumber.db >>> handle 0 >>> >>> ce READ 1 HELD userRoot/gidnumber.db >>> handle 0 >>> >>> 65 READ 1 HELD userRoot/seeAlso.db >>> handle 0 >>> >>> 9d READ 1 HELD userRoot/ipaallowedtarget.db >>> handle 0 >>> >>> a6 READ 1 HELD userRoot/ipasudorunasgroup.db >>> handle 0 >>> >>> 3a READ 1 HELD userRoot/ancestorid.db >>> handle 0 >>> >>> >>> >>> >>> pstack output >>> >>> root at ipa-master ]# pstack 4845 >>> Thread 44 (Thread 0x7f681dd3a700 (LWP 4847)): >>> #0 0x00007f683cd719b3 in select () from /lib64/libc.so.6 >>> #1 0x00007f683f4fc0e9 in DS_Sleep () from >>> /usr/lib64/dirsrv/libslapd.so.0 >>> #2 0x00007f6832b4a907 in deadlock_threadmain () from >>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>> #3 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so >>> #4 0x00007f683d04cdc5 in start_thread () from >>> /lib64/libpthread.so.0 >>> #5 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 >>> Thread 43 (Thread 0x7f681d539700 (LWP 4848)): >>> #0 0x00007f683cd719b3 in select () from /lib64/libc.so.6 >>> #1 0x00007f68376d8bad in __os_yield () from >>> /lib64/libdb-5.3.so >>> #2 0x00007f68376d42b3 in __memp_sync_int () from >>> /lib64/libdb-5.3.so >>> #3 0x00007f68376e4752 in __txn_checkpoint () from >>> /lib64/libdb-5.3.so >>> #4 0x00007f68376e4b74 in __txn_checkpoint_pp () from >>> /lib64/libdb-5.3.so >>> #5 0x00007f6832b4ea87 in checkpoint_threadmain () from >>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>> #6 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so >>> #7 0x00007f683d04cdc5 in start_thread () from >>> /lib64/libpthread.so.0 >>> #8 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 >>> Thread 42 (Thread 0x7f681cd38700 (LWP 4849)): >>> #0 0x00007f683cd719b3 in select () from /lib64/libc.so.6 >>> #1 0x00007f683f4fc0e9 in DS_Sleep () from >>> /usr/lib64/dirsrv/libslapd.so.0 >>> #2 0x00007f6832b4ab7f in trickle_threadmain () from >>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>> #3 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so >>> #4 0x00007f683d04cdc5 in start_thread () from >>> /lib64/libpthread.so.0 >>> #5 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 >>> Thread 41 (Thread 0x7f681c537700 (LWP 4850)): >>> #0 0x00007f683cd719b3 in select () from /lib64/libc.so.6 >>> #1 0x00007f683f4fc0e9 in DS_Sleep () from >>> /usr/lib64/dirsrv/libslapd.so.0 >>> #2 0x00007f6832b45707 in perf_threadmain () from >>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>> #3 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so >>> #4 0x00007f683d04cdc5 in start_thread () from >>> /lib64/libpthread.so.0 >>> #5 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 >>> Thread 40 (Thread 0x7f681bb2d700 (LWP 4851)): >>> #0 0x00007f683d0506d5 in pthread_cond_wait@@GLIBC_2.3.2 >>> () from >>> /lib64/libpthread.so.0 >>> #1 0x00007f683d6a6050 in PR_WaitCondVar () from >>> /lib64/libnspr4.so >>> #2 0x00007f683f4eb198 in slapi_wait_condvar () from >>> /usr/lib64/dirsrv/libslapd.so.0 >>> #3 0x00007f683587d62e in cos_cache_wait_on_change () from >>> /usr/lib64/dirsrv/plugins/libcos-plugin.so >>> #4 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so >>> #5 0x00007f683d04cdc5 in start_thread () from >>> /lib64/libpthread.so.0 >>> #6 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 >>> Thread 39 (Thread 0x7f681b32c700 (LWP 4852)): >>> #0 0x00007f683d0506d5 in pthread_cond_wait@@GLIBC_2.3.2 >>> () from >>> /lib64/libpthread.so.0 >>> #1 0x00007f683d6a6050 in PR_WaitCondVar () from >>> /lib64/libnspr4.so >>> #2 0x00007f683f4eb198 in slapi_wait_condvar () from >>> /usr/lib64/dirsrv/libslapd.so.0 >>> #3 0x00007f68311e8eed in roles_cache_wait_on_change () from >>> /usr/lib64/dirsrv/plugins/libroles-plugin.so >>> #4 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so >>> #5 0x00007f683d04cdc5 in start_thread () from >>> /lib64/libpthread.so.0 >>> #6 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 >>> Thread 38 (Thread 0x7f681ab2b700 (LWP 4853)): >>> #0 0x00007f683d0506d5 in pthread_cond_wait@@GLIBC_2.3.2 >>> () from >>> /lib64/libpthread.so.0 >>> #1 0x00007f683d6a6050 in PR_WaitCondVar () from >>> /lib64/libnspr4.so >>> #2 0x00007f683f4eb198 in slapi_wait_condvar () from >>> /usr/lib64/dirsrv/libslapd.so.0 >>> #3 0x00007f68311e8eed in roles_cache_wait_on_change () from >>> /usr/lib64/dirsrv/plugins/libroles-plugin.so >>> #4 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so >>> #5 0x00007f683d04cdc5 in start_thread () from >>> /lib64/libpthread.so.0 >>> #6 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 >>> Thread 37 (Thread 0x7f681a32a700 (LWP 4854)): >>> #0 0x00007f683d0506d5 in pthread_cond_wait@@GLIBC_2.3.2 >>> () from >>> /lib64/libpthread.so.0 >>> #1 0x00007f683d6a6050 in PR_WaitCondVar () from >>> /lib64/libnspr4.so >>> #2 0x00007f683f4eb198 in slapi_wait_condvar () from >>> /usr/lib64/dirsrv/libslapd.so.0 >>> #3 0x00007f68311e8eed in roles_cache_wait_on_change () from >>> /usr/lib64/dirsrv/plugins/libroles-plugin.so >>> #4 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so >>> #5 0x00007f683d04cdc5 in start_thread () from >>> /lib64/libpthread.so.0 >>> #6 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 >>> Thread 36 (Thread 0x7f6819b29700 (LWP 4855)): >>> #0 0x00007f683d050a82 in >>> pthread_cond_timedwait@@GLIBC_2.3.2 >>> () from >>> /lib64/libpthread.so.0 >>> #1 0x00007f683d6a5b07 in pt_TimedWait () from >>> /lib64/libnspr4.so >>> #2 0x00007f683d6a5fce in PR_WaitCondVar () from >>> /lib64/libnspr4.so >>> #3 0x00007f683f9a4d03 in housecleaning () >>> #4 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so >>> #5 0x00007f683d04cdc5 in start_thread () from >>> /lib64/libpthread.so.0 >>> #6 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 >>> Thread 35 (Thread 0x7f6819328700 (LWP 4856)): >>> #0 0x00007f683d050a82 in >>> pthread_cond_timedwait@@GLIBC_2.3.2 >>> () from >>> /lib64/libpthread.so.0 >>> #1 0x00007f683d6a5b07 in pt_TimedWait () from >>> /lib64/libnspr4.so >>> #2 0x00007f683d6a5fce in PR_WaitCondVar () from >>> /lib64/libnspr4.so >>> #3 0x00007f683f4911e8 in eq_loop () from >>> /usr/lib64/dirsrv/libslapd.so.0 >>> #4 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so >>> #5 0x00007f683d04cdc5 in start_thread () from >>> /lib64/libpthread.so.0 >>> #6 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 >>> Thread 34 (Thread 0x7f680bfff700 (LWP 4859)): >>> #0 0x00007f683d0506d5 in pthread_cond_wait@@GLIBC_2.3.2 >>> () from >>> /lib64/libpthread.so.0 >>> #1 0x00007f683d6a6050 in PR_WaitCondVar () from >>> /lib64/libnspr4.so >>> #2 0x00007f683f999fee in connection_wait_for_new_work () >>> #3 0x00007f683f99b719 in connection_threadmain () >>> #4 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so >>> #5 0x00007f683d04cdc5 in start_thread () from >>> /lib64/libpthread.so.0 >>> #6 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 >>> Thread 33 (Thread 0x7f680b7fe700 (LWP 4860)): >>> #0 0x00007f683d0506d5 in pthread_cond_wait@@GLIBC_2.3.2 >>> () from >>> /lib64/libpthread.so.0 >>> #1 0x00007f683d6a6050 in PR_WaitCondVar () from >>> /lib64/libnspr4.so >>> #2 0x00007f683f999fee in connection_wait_for_new_work () >>> #3 0x00007f683f99b719 in connection_threadmain () >>> #4 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so >>> #5 0x00007f683d04cdc5 in start_thread () from >>> /lib64/libpthread.so.0 >>> #6 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 >>> Thread 32 (Thread 0x7f680affd700 (LWP 4861)): >>> #0 0x00007f683d0506d5 in pthread_cond_wait@@GLIBC_2.3.2 >>> () from >>> /lib64/libpthread.so.0 >>> #1 0x00007f68375ac2f3 in __db_hybrid_mutex_suspend () from >>> /lib64/libdb-5.3.so >>> #2 0x00007f68375ab640 in __db_tas_mutex_lock () from >>> /lib64/libdb-5.3.so >>> #3 0x00007f6837655d3a in __lock_get_internal () from >>> /lib64/libdb-5.3.so >>> #4 0x00007f6837656820 in __lock_get () from >>> /lib64/libdb-5.3.so >>> #5 0x00007f6837682142 in __db_lget () from >>> /lib64/libdb-5.3.so >>> #6 0x00007f68375c9605 in __bam_search () from >>> /lib64/libdb-5.3.so >>> #7 0x00007f68375b4256 in __bamc_search () from >>> /lib64/libdb-5.3.so >>> #8 0x00007f68375b5d0f in __bamc_get () from >>> /lib64/libdb-5.3.so >>> #9 0x00007f683766eca6 in __dbc_iget () from >>> /lib64/libdb-5.3.so >>> #10 0x00007f683767db02 in __dbc_get_pp () from >>> /lib64/libdb-5.3.so >>> #11 0x00007f6832b59680 in idl_new_fetch () from >>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>> #12 0x00007f6832b67b26 in index_read_ext_allids () from >>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>> #13 0x00007f6832b52284 in keys2idl () from >>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>> #14 0x00007f6832b529e3 in ava_candidates.isra.0 () from >>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>> #15 0x00007f6832b52fd2 in filter_candidates_ext () from >>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>> #16 0x00007f6832b54096 in list_candidates () from >>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>> #17 0x00007f6832b52f40 in filter_candidates_ext () from >>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>> #18 0x00007f6832b54096 in list_candidates () from >>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>> #19 0x00007f6832b52f40 in filter_candidates_ext () from >>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>> #20 0x00007f6832b54096 in list_candidates () from >>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>> #21 0x00007f6832b52f40 in filter_candidates_ext () from >>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>> #22 0x00007f6832b8e80c in subtree_candidates () from >>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>> #23 0x00007f6832b8fe7f in ldbm_back_search () from >>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>> #24 0x00007f683f4bd1d6 in op_shared_search () from >>> /usr/lib64/dirsrv/libslapd.so.0 >>> #25 0x00007f683f9ae33e in do_search () >>> #26 0x00007f683f99c3c3 in connection_threadmain () >>> #27 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so >>> #28 0x00007f683d04cdc5 in start_thread () from >>> /lib64/libpthread.so.0 >>> #29 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 >>> Thread 31 (Thread 0x7f680a7fc700 (LWP 4862)): >>> #0 0x00007f683d0506d5 in pthread_cond_wait@@GLIBC_2.3.2 >>> () from >>> /lib64/libpthread.so.0 >>> #1 0x00007f683d6a62c3 in PR_EnterMonitor () from >>> /lib64/libnspr4.so >>> #2 0x00007f6832b4d3e6 in dblayer_txn_begin () from >>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>> #3 0x00007f6832b89007 in ldbm_back_modify () from >>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>> #4 0x00007f683f4b6541 in op_shared_modify () from >>> /usr/lib64/dirsrv/libslapd.so.0 >>> #5 0x00007f683f4b7004 in modify_internal_pb () from >>> /usr/lib64/dirsrv/libslapd.so.0 >>> #6 0x00007f683483bbbb in ipalockout_postop () from >>> /usr/lib64/dirsrv/plugins/libipa_lockout.so >>> #7 0x00007f683f4c8848 in plugin_call_func () from >>> /usr/lib64/dirsrv/libslapd.so.0 >>> #8 0x00007f683f4c8b13 in plugin_call_plugins () from >>> /usr/lib64/dirsrv/libslapd.so.0 >>> #9 0x00007f683f9954d8 in do_bind () >>> #10 0x00007f683f99c3fd in connection_threadmain () >>> #11 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so >>> #12 0x00007f683d04cdc5 in start_thread () from >>> /lib64/libpthread.so.0 >>> #13 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 >>> Thread 30 (Thread 0x7f6809ffb700 (LWP 4863)): >>> #0 0x00007f683d0506d5 in pthread_cond_wait@@GLIBC_2.3.2 >>> () from >>> /lib64/libpthread.so.0 >>> #1 0x00007f68375ac2f3 in __db_hybrid_mutex_suspend () from >>> /lib64/libdb-5.3.so >>> #2 0x00007f68375ab640 in __db_tas_mutex_lock () from >>> /lib64/libdb-5.3.so >>> #3 0x00007f6837655d3a in __lock_get_internal () from >>> /lib64/libdb-5.3.so >>> #4 0x00007f6837656820 in __lock_get () from >>> /lib64/libdb-5.3.so >>> #5 0x00007f6837682142 in __db_lget () from >>> /lib64/libdb-5.3.so >>> #6 0x00007f68375c9605 in __bam_search () from >>> /lib64/libdb-5.3.so >>> #7 0x00007f68375b4256 in __bamc_search () from >>> /lib64/libdb-5.3.so >>> #8 0x00007f68375b5d0f in __bamc_get () from >>> /lib64/libdb-5.3.so >>> #9 0x00007f683766eca6 in __dbc_iget () from >>> /lib64/libdb-5.3.so >>> #10 0x00007f683767db02 in __dbc_get_pp () from >>> /lib64/libdb-5.3.so >>> #11 0x00007f6832b59680 in idl_new_fetch () from >>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>> #12 0x00007f6832b67b26 in index_read_ext_allids () from >>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>> #13 0x00007f6832b52284 in keys2idl () from >>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>> #14 0x00007f6832b529e3 in ava_candidates.isra.0 () from >>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>> #15 0x00007f6832b52fd2 in filter_candidates_ext () from >>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>> #16 0x00007f6832b54096 in list_candidates () from >>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>> #17 0x00007f6832b52f40 in filter_candidates_ext () from >>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>> #18 0x00007f6832b8e80c in subtree_candidates () from >>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>> #19 0x00007f6832b8fe7f in ldbm_back_search () from >>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>> #20 0x00007f683f4bd1d6 in op_shared_search () from >>> /usr/lib64/dirsrv/libslapd.so.0 >>> #21 0x00007f683f4cd94e in search_internal_callback_pb () >>> from /usr/lib64/dirsrv/libslapd.so.0 >>> #22 0x00007f683f4cdbe8 in search_internal_pb () from >>> /usr/lib64/dirsrv/libslapd.so.0 >>> #23 0x00007f683f9ac49b in ids_sasl_canon_user () >>> #24 0x00007f683d46c93b in _sasl_canon_user () from >>> /lib64/libsasl2.so.3 >>> #25 0x00007f683d46cc4c in _sasl_canon_user_lookup () from >>> /lib64/libsasl2.so.3 >>> #26 0x00007f6837374fb9 in gssapi_server_mech_step () from >>> /usr/lib64/sasl2/libgssapiv2.so >>> #27 0x00007f683d478b9b in sasl_server_step () from >>> /lib64/libsasl2.so.3 >>> #28 0x00007f683f9ad83b in ids_sasl_check_bind () >>> #29 0x00007f683f99561a in do_bind () >>> #30 0x00007f683f99c3fd in connection_threadmain () >>> #31 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so >>> #32 0x00007f683d04cdc5 in start_thread () from >>> /lib64/libpthread.so.0 >>> #33 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 >>> Thread 29 (Thread 0x7f68097fa700 (LWP 4864)): >>> #0 0x00007f683d0506d5 in pthread_cond_wait@@GLIBC_2.3.2 >>> () from >>> /lib64/libpthread.so.0 >>> #1 0x00007f683d6a6050 in PR_WaitCondVar () from >>> /lib64/libnspr4.so >>> #2 0x00007f683f999fee in connection_wait_for_new_work () >>> #3 0x00007f683f99b719 in connection_threadmain () >>> #4 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so >>> #5 0x00007f683d04cdc5 in start_thread () from >>> /lib64/libpthread.so.0 >>> #6 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 >>> Thread 28 (Thread 0x7f6808ff9700 (LWP 4865)): >>> #0 0x00007f683d0506d5 in pthread_cond_wait@@GLIBC_2.3.2 >>> () from >>> /lib64/libpthread.so.0 >>> #1 0x00007f683d6a6050 in PR_WaitCondVar () from >>> /lib64/libnspr4.so >>> #2 0x00007f683f999fee in connection_wait_for_new_work () >>> #3 0x00007f683f99b719 in connection_threadmain () >>> #4 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so >>> #5 0x00007f683d04cdc5 in start_thread () from >>> /lib64/libpthread.so.0 >>> #6 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 >>> Thread 27 (Thread 0x7f6803fff700 (LWP 4866)): >>> #0 0x00007f683d0506d5 in pthread_cond_wait@@GLIBC_2.3.2 >>> () from >>> /lib64/libpthread.so.0 >>> #1 0x00007f683d6a6050 in PR_WaitCondVar () from >>> /lib64/libnspr4.so >>> #2 0x00007f683f999fee in connection_wait_for_new_work () >>> #3 0x00007f683f99b719 in connection_threadmain () >>> #4 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so >>> #5 0x00007f683d04cdc5 in start_thread () from >>> /lib64/libpthread.so.0 >>> #6 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 >>> Thread 26 (Thread 0x7f68037fe700 (LWP 4867)): >>> #0 0x00007f683d0506d5 in pthread_cond_wait@@GLIBC_2.3.2 >>> () from >>> /lib64/libpthread.so.0 >>> #1 0x00007f683d6a6050 in PR_WaitCondVar () from >>> /lib64/libnspr4.so >>> #2 0x00007f683f999fee in connection_wait_for_new_work () >>> #3 0x00007f683f99b719 in connection_threadmain () >>> #4 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so >>> #5 0x00007f683d04cdc5 in start_thread () from >>> /lib64/libpthread.so.0 >>> #6 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 >>> Thread 25 (Thread 0x7f6802ffd700 (LWP 4868)): >>> #0 0x00007f683d0506d5 in pthread_cond_wait@@GLIBC_2.3.2 >>> () from >>> /lib64/libpthread.so.0 >>> #1 0x00007f683d6a6050 in PR_WaitCondVar () from >>> /lib64/libnspr4.so >>> #2 0x00007f683f999fee in connection_wait_for_new_work () >>> #3 0x00007f683f99b719 in connection_threadmain () >>> #4 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so >>> #5 0x00007f683d04cdc5 in start_thread () from >>> /lib64/libpthread.so.0 >>> #6 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 >>> Thread 24 (Thread 0x7f68027fc700 (LWP 4869)): >>> #0 0x00007f683d0506d5 in pthread_cond_wait@@GLIBC_2.3.2 >>> () from >>> /lib64/libpthread.so.0 >>> #1 0x00007f68375ac2f3 in __db_hybrid_mutex_suspend () from >>> /lib64/libdb-5.3.so >>> #2 0x00007f68375abb37 in __db_tas_mutex_readlock () from >>> /lib64/libdb-5.3.so >>> #3 0x00007f68376c5d35 in __memp_fget () from >>> /lib64/libdb-5.3.so >>> #4 0x00007f683766f333 in __dbc_iget () from >>> /lib64/libdb-5.3.so >>> #5 0x00007f683767db02 in __dbc_get_pp () from >>> /lib64/libdb-5.3.so >>> #6 0x00007f6832b59822 in idl_new_fetch () from >>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>> #7 0x00007f6832b67b26 in index_read_ext_allids () from >>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>> #8 0x00007f6832b52284 in keys2idl () from >>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>> #9 0x00007f6832b529e3 in ava_candidates.isra.0 () from >>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>> #10 0x00007f6832b52fd2 in filter_candidates_ext () from >>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>> #11 0x00007f6832b54096 in list_candidates () from >>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>> #12 0x00007f6832b52f40 in filter_candidates_ext () from >>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>> #13 0x00007f6832b54096 in list_candidates () from >>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>> #14 0x00007f6832b52f40 in filter_candidates_ext () from >>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>> #15 0x00007f6832b54096 in list_candidates () from >>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>> #16 0x00007f6832b52f40 in filter_candidates_ext () from >>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>> #17 0x00007f6832b8e80c in subtree_candidates () from >>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>> #18 0x00007f6832b8fe7f in ldbm_back_search () from >>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>> #19 0x00007f683f4bd1d6 in op_shared_search () from >>> /usr/lib64/dirsrv/libslapd.so.0 >>> #20 0x00007f683f9ae33e in do_search () >>> #21 0x00007f683f99c3c3 in connection_threadmain () >>> #22 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so >>> #23 0x00007f683d04cdc5 in start_thread () from >>> /lib64/libpthread.so.0 >>> #24 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 >>> Thread 23 (Thread 0x7f6801ffb700 (LWP 4870)): >>> #0 0x00007f683d0506d5 in pthread_cond_wait@@GLIBC_2.3.2 >>> () from >>> /lib64/libpthread.so.0 >>> #1 0x00007f683d6a6050 in PR_WaitCondVar () from >>> /lib64/libnspr4.so >>> #2 0x00007f683f999fee in connection_wait_for_new_work () >>> #3 0x00007f683f99b719 in connection_threadmain () >>> #4 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so >>> #5 0x00007f683d04cdc5 in start_thread () from >>> /lib64/libpthread.so.0 >>> #6 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 >>> Thread 22 (Thread 0x7f68017fa700 (LWP 4871)): >>> #0 0x00007f683d0506d5 in pthread_cond_wait@@GLIBC_2.3.2 >>> () from >>> /lib64/libpthread.so.0 >>> #1 0x00007f683d6a6050 in PR_WaitCondVar () from >>> /lib64/libnspr4.so >>> #2 0x00007f683f999fee in connection_wait_for_new_work () >>> #3 0x00007f683f99b719 in connection_threadmain () >>> #4 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so >>> #5 0x00007f683d04cdc5 in start_thread () from >>> /lib64/libpthread.so.0 >>> #6 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 >>> Thread 21 (Thread 0x7f6800ff9700 (LWP 4872)): >>> #0 0x00007f683d0506d5 in pthread_cond_wait@@GLIBC_2.3.2 >>> () from >>> /lib64/libpthread.so.0 >>> #1 0x00007f683d6a6050 in PR_WaitCondVar () from >>> /lib64/libnspr4.so >>> #2 0x00007f683f999fee in connection_wait_for_new_work () >>> #3 0x00007f683f99b719 in connection_threadmain () >>> #4 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so >>> #5 0x00007f683d04cdc5 in start_thread () from >>> /lib64/libpthread.so.0 >>> #6 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 >>> Thread 20 (Thread 0x7f68007f8700 (LWP 4873)): >>> #0 0x00007f683d0506d5 in pthread_cond_wait@@GLIBC_2.3.2 >>> () from >>> /lib64/libpthread.so.0 >>> #1 0x00007f68375ac2f3 in __db_hybrid_mutex_suspend () from >>> /lib64/libdb-5.3.so >>> #2 0x00007f68375ab640 in __db_tas_mutex_lock () from >>> /lib64/libdb-5.3.so >>> #3 0x00007f68376c5e1b in __memp_fget () from >>> /lib64/libdb-5.3.so >>> #4 0x00007f68375c9211 in __bam_search () from >>> /lib64/libdb-5.3.so >>> #5 0x00007f68375b4256 in __bamc_search () from >>> /lib64/libdb-5.3.so >>> #6 0x00007f68375b86c4 in __bamc_put () from >>> /lib64/libdb-5.3.so >>> #7 0x00007f683766f6d5 in __dbc_iput () from >>> /lib64/libdb-5.3.so >>> #8 0x00007f683766aa0e in __db_put () from >>> /lib64/libdb-5.3.so >>> #9 0x00007f683767ffa4 in __db_put_pp () from >>> /lib64/libdb-5.3.so >>> #10 0x00007f6832b5a4cf in idl_new_insert_key () from >>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>> #11 0x00007f6832b695b5 in addordel_values_sv.isra.3 () from >>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>> #12 0x00007f6832b6a25a in index_addordel_values_ext_sv () >>> from /usr/lib64/dirsrv/plugins/libback-ldbm.so >>> #13 0x00007f6832b6a694 in index_addordel_values_sv () from >>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>> #14 0x00007f6832b6a75a in index_addordel_entry () from >>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>> #15 0x00007f6832b6d5fa in ldbm_back_add () from >>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>> #16 0x00007f683f46fcca in op_shared_add () from >>> /usr/lib64/dirsrv/libslapd.so.0 >>> #17 0x00007f683f471020 in do_add () from >>> /usr/lib64/dirsrv/libslapd.so.0 >>> #18 0x00007f683f99c303 in connection_threadmain () >>> #19 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so >>> #20 0x00007f683d04cdc5 in start_thread () from >>> /lib64/libpthread.so.0 >>> #21 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 >>> Thread 19 (Thread 0x7f67ffff7700 (LWP 4874)): >>> #0 0x00007f683d0506d5 in pthread_cond_wait@@GLIBC_2.3.2 >>> () from >>> /lib64/libpthread.so.0 >>> #1 0x00007f683d6a62c3 in PR_EnterMonitor () from >>> /lib64/libnspr4.so >>> #2 0x00007f6832b4d3e6 in dblayer_txn_begin () from >>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>> #3 0x00007f6832b89007 in ldbm_back_modify () from >>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>> #4 0x00007f683f4b6541 in op_shared_modify () from >>> /usr/lib64/dirsrv/libslapd.so.0 >>> #5 0x00007f683f4b7004 in modify_internal_pb () from >>> /usr/lib64/dirsrv/libslapd.so.0 >>> #6 0x00007f6832dd3b60 in ipapwd_apply_mods () from >>> /usr/lib64/dirsrv/plugins/libipa_pwd_extop.so >>> #7 0x00007f6832dd78e0 in store_new_keys () from >>> /usr/lib64/dirsrv/plugins/libipa_pwd_extop.so >>> #8 0x00007f6832dd9ad0 in ipapwd_extop () from >>> /usr/lib64/dirsrv/plugins/libipa_pwd_extop.so >>> #9 0x00007f683f4c6ba2 in plugin_call_exop_plugins () from >>> /usr/lib64/dirsrv/libslapd.so.0 >>> #10 0x00007f683f9a2efe in do_extended () >>> #11 0x00007f683f99c41a in connection_threadmain () >>> #12 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so >>> #13 0x00007f683d04cdc5 in start_thread () from >>> /lib64/libpthread.so.0 >>> #14 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 >>> Thread 18 (Thread 0x7f67ff7f6700 (LWP 4875)): >>> #0 0x00007f683d0506d5 in pthread_cond_wait@@GLIBC_2.3.2 >>> () from >>> /lib64/libpthread.so.0 >>> #1 0x00007f683d6a6050 in PR_WaitCondVar () from >>> /lib64/libnspr4.so >>> #2 0x00007f683f999fee in connection_wait_for_new_work () >>> #3 0x00007f683f99b719 in connection_threadmain () >>> #4 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so >>> #5 0x00007f683d04cdc5 in start_thread () from >>> /lib64/libpthread.so.0 >>> #6 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 >>> Thread 17 (Thread 0x7f67feff5700 (LWP 4876)): >>> #0 0x00007f683d0506d5 in pthread_cond_wait@@GLIBC_2.3.2 >>> () from >>> /lib64/libpthread.so.0 >>> #1 0x00007f68375ac2f3 in __db_hybrid_mutex_suspend () from >>> /lib64/libdb-5.3.so >>> #2 0x00007f68375ab640 in __db_tas_mutex_lock () from >>> /lib64/libdb-5.3.so >>> #3 0x00007f6837655d3a in __lock_get_internal () from >>> /lib64/libdb-5.3.so >>> #4 0x00007f6837656820 in __lock_get () from >>> /lib64/libdb-5.3.so >>> #5 0x00007f6837682142 in __db_lget () from >>> /lib64/libdb-5.3.so >>> #6 0x00007f68375c9605 in __bam_search () from >>> /lib64/libdb-5.3.so >>> #7 0x00007f68375b4256 in __bamc_search () from >>> /lib64/libdb-5.3.so >>> #8 0x00007f68375b5d0f in __bamc_get () from >>> /lib64/libdb-5.3.so >>> #9 0x00007f683766eca6 in __dbc_iget () from >>> /lib64/libdb-5.3.so >>> #10 0x00007f683767db02 in __dbc_get_pp () from >>> /lib64/libdb-5.3.so >>> #11 0x00007f6832b59680 in idl_new_fetch () from >>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>> #12 0x00007f6832b67b26 in index_read_ext_allids () from >>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>> #13 0x00007f6832b52284 in keys2idl () from >>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>> #14 0x00007f6832b529e3 in ava_candidates.isra.0 () from >>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>> #15 0x00007f6832b52fd2 in filter_candidates_ext () from >>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>> #16 0x00007f6832b54096 in list_candidates () from >>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>> #17 0x00007f6832b52f40 in filter_candidates_ext () from >>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>> #18 0x00007f6832b8e80c in subtree_candidates () from >>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>> #19 0x00007f6832b8fe7f in ldbm_back_search () from >>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>> #20 0x00007f683f4bd1d6 in op_shared_search () from >>> /usr/lib64/dirsrv/libslapd.so.0 >>> #21 0x00007f683f4cd94e in search_internal_callback_pb () >>> from /usr/lib64/dirsrv/libslapd.so.0 >>> #22 0x00007f683f4cdbe8 in search_internal_pb () from >>> /usr/lib64/dirsrv/libslapd.so.0 >>> #23 0x00007f6832dd75dc in get_entry_by_principal () from >>> /usr/lib64/dirsrv/plugins/libipa_pwd_extop.so >>> #24 0x00007f6832dd8eb1 in ipapwd_extop () from >>> /usr/lib64/dirsrv/plugins/libipa_pwd_extop.so >>> #25 0x00007f683f4c6ba2 in plugin_call_exop_plugins () from >>> /usr/lib64/dirsrv/libslapd.so.0 >>> #26 0x00007f683f9a2efe in do_extended () >>> #27 0x00007f683f99c41a in connection_threadmain () >>> #28 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so >>> #29 0x00007f683d04cdc5 in start_thread () from >>> /lib64/libpthread.so.0 >>> #30 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 >>> Thread 16 (Thread 0x7f67fe7f4700 (LWP 4877)): >>> #0 0x00007f683d0506d5 in pthread_cond_wait@@GLIBC_2.3.2 >>> () from >>> /lib64/libpthread.so.0 >>> #1 0x00007f683d6a6050 in PR_WaitCondVar () from >>> /lib64/libnspr4.so >>> #2 0x00007f683f999fee in connection_wait_for_new_work () >>> #3 0x00007f683f99b719 in connection_threadmain () >>> #4 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so >>> #5 0x00007f683d04cdc5 in start_thread () from >>> /lib64/libpthread.so.0 >>> #6 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 >>> Thread 15 (Thread 0x7f67fdff3700 (LWP 4878)): >>> #0 0x00007f683d0506d5 in pthread_cond_wait@@GLIBC_2.3.2 >>> () from >>> /lib64/libpthread.so.0 >>> #1 0x00007f683d6a6050 in PR_WaitCondVar () from >>> /lib64/libnspr4.so >>> #2 0x00007f683f999fee in connection_wait_for_new_work () >>> #3 0x00007f683f99b719 in connection_threadmain () >>> #4 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so >>> #5 0x00007f683d04cdc5 in start_thread () from >>> /lib64/libpthread.so.0 >>> #6 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 >>> Thread 14 (Thread 0x7f67fd7f2700 (LWP 4879)): >>> #0 0x00007f683d0506d5 in pthread_cond_wait@@GLIBC_2.3.2 >>> () from >>> /lib64/libpthread.so.0 >>> #1 0x00007f683d6a6050 in PR_WaitCondVar () from >>> /lib64/libnspr4.so >>> #2 0x00007f683f999fee in connection_wait_for_new_work () >>> #3 0x00007f683f99b719 in connection_threadmain () >>> #4 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so >>> #5 0x00007f683d04cdc5 in start_thread () from >>> /lib64/libpthread.so.0 >>> #6 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 >>> Thread 13 (Thread 0x7f67fcff1700 (LWP 4880)): >>> #0 0x00007f683d0506d5 in pthread_cond_wait@@GLIBC_2.3.2 >>> () from >>> /lib64/libpthread.so.0 >>> #1 0x00007f683d6a6050 in PR_WaitCondVar () from >>> /lib64/libnspr4.so >>> #2 0x00007f683f999fee in connection_wait_for_new_work () >>> #3 0x00007f683f99b719 in connection_threadmain () >>> #4 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so >>> #5 0x00007f683d04cdc5 in start_thread () from >>> /lib64/libpthread.so.0 >>> #6 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 >>> Thread 12 (Thread 0x7f67fc7f0700 (LWP 4881)): >>> #0 0x00007f683d0506d5 in pthread_cond_wait@@GLIBC_2.3.2 >>> () from >>> /lib64/libpthread.so.0 >>> #1 0x00007f683d6a6050 in PR_WaitCondVar () from >>> /lib64/libnspr4.so >>> #2 0x00007f683f999fee in connection_wait_for_new_work () >>> #3 0x00007f683f99b719 in connection_threadmain () >>> #4 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so >>> #5 0x00007f683d04cdc5 in start_thread () from >>> /lib64/libpthread.so.0 >>> #6 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 >>> Thread 11 (Thread 0x7f67fbfef700 (LWP 4882)): >>> #0 0x00007f683d0506d5 in pthread_cond_wait@@GLIBC_2.3.2 >>> () from >>> /lib64/libpthread.so.0 >>> #1 0x00007f683d6a6050 in PR_WaitCondVar () from >>> /lib64/libnspr4.so >>> #2 0x00007f683f999fee in connection_wait_for_new_work () >>> #3 0x00007f683f99b719 in connection_threadmain () >>> #4 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so >>> #5 0x00007f683d04cdc5 in start_thread () from >>> /lib64/libpthread.so.0 >>> #6 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 >>> Thread 10 (Thread 0x7f67fb7ee700 (LWP 4883)): >>> #0 0x00007f683d0506d5 in pthread_cond_wait@@GLIBC_2.3.2 >>> () from >>> /lib64/libpthread.so.0 >>> #1 0x00007f68375ac2f3 in __db_hybrid_mutex_suspend () from >>> /lib64/libdb-5.3.so >>> #2 0x00007f68375ab640 in __db_tas_mutex_lock () from >>> /lib64/libdb-5.3.so >>> #3 0x00007f6837655d3a in __lock_get_internal () from >>> /lib64/libdb-5.3.so >>> #4 0x00007f6837656820 in __lock_get () from >>> /lib64/libdb-5.3.so >>> #5 0x00007f6837682142 in __db_lget () from >>> /lib64/libdb-5.3.so >>> #6 0x00007f68375c9605 in __bam_search () from >>> /lib64/libdb-5.3.so >>> #7 0x00007f68375b4256 in __bamc_search () from >>> /lib64/libdb-5.3.so >>> #8 0x00007f68375b5d0f in __bamc_get () from >>> /lib64/libdb-5.3.so >>> #9 0x00007f683766eca6 in __dbc_iget () from >>> /lib64/libdb-5.3.so >>> #10 0x00007f683767db02 in __dbc_get_pp () from >>> /lib64/libdb-5.3.so >>> #11 0x00007f6832b59680 in idl_new_fetch () from >>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>> #12 0x00007f6832b67b26 in index_read_ext_allids () from >>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>> #13 0x00007f6832b52284 in keys2idl () from >>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>> #14 0x00007f6832b529e3 in ava_candidates.isra.0 () from >>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>> #15 0x00007f6832b52fd2 in filter_candidates_ext () from >>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>> #16 0x00007f6832b54096 in list_candidates () from >>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>> #17 0x00007f6832b52f40 in filter_candidates_ext () from >>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>> #18 0x00007f6832b54096 in list_candidates () from >>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>> #19 0x00007f6832b52f40 in filter_candidates_ext () from >>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>> #20 0x00007f6832b54096 in list_candidates () from >>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>> #21 0x00007f6832b52f40 in filter_candidates_ext () from >>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>> #22 0x00007f6832b8e80c in subtree_candidates () from >>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>> #23 0x00007f6832b8fe7f in ldbm_back_search () from >>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>> #24 0x00007f683f4bd1d6 in op_shared_search () from >>> /usr/lib64/dirsrv/libslapd.so.0 >>> #25 0x00007f683f9ae33e in do_search () >>> #26 0x00007f683f99c3c3 in connection_threadmain () >>> #27 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so >>> #28 0x00007f683d04cdc5 in start_thread () from >>> /lib64/libpthread.so.0 >>> #29 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 >>> Thread 9 (Thread 0x7f67fafed700 (LWP 4884)): >>> #0 0x00007f683d0506d5 in pthread_cond_wait@@GLIBC_2.3.2 >>> () from >>> /lib64/libpthread.so.0 >>> #1 0x00007f683d6a6050 in PR_WaitCondVar () from >>> /lib64/libnspr4.so >>> #2 0x00007f683f999fee in connection_wait_for_new_work () >>> #3 0x00007f683f99b719 in connection_threadmain () >>> #4 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so >>> #5 0x00007f683d04cdc5 in start_thread () from >>> /lib64/libpthread.so.0 >>> #6 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 >>> Thread 8 (Thread 0x7f67fa7ec700 (LWP 4885)): >>> #0 0x00007f683d0506d5 in pthread_cond_wait@@GLIBC_2.3.2 >>> () from >>> /lib64/libpthread.so.0 >>> #1 0x00007f68375ac2f3 in __db_hybrid_mutex_suspend () from >>> /lib64/libdb-5.3.so >>> #2 0x00007f68375ab640 in __db_tas_mutex_lock () from >>> /lib64/libdb-5.3.so >>> #3 0x00007f6837655d3a in __lock_get_internal () from >>> /lib64/libdb-5.3.so >>> #4 0x00007f6837656820 in __lock_get () from >>> /lib64/libdb-5.3.so >>> #5 0x00007f6837682142 in __db_lget () from >>> /lib64/libdb-5.3.so >>> #6 0x00007f68375c9605 in __bam_search () from >>> /lib64/libdb-5.3.so >>> #7 0x00007f68375b4256 in __bamc_search () from >>> /lib64/libdb-5.3.so >>> #8 0x00007f68375b5d0f in __bamc_get () from >>> /lib64/libdb-5.3.so >>> #9 0x00007f683766eca6 in __dbc_iget () from >>> /lib64/libdb-5.3.so >>> #10 0x00007f683767db02 in __dbc_get_pp () from >>> /lib64/libdb-5.3.so >>> #11 0x00007f6832b59680 in idl_new_fetch () from >>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>> #12 0x00007f6832b67b26 in index_read_ext_allids () from >>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>> #13 0x00007f6832b52284 in keys2idl () from >>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>> #14 0x00007f6832b529e3 in ava_candidates.isra.0 () from >>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>> #15 0x00007f6832b52fd2 in filter_candidates_ext () from >>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>> #16 0x00007f6832b54096 in list_candidates () from >>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>> #17 0x00007f6832b52f40 in filter_candidates_ext () from >>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>> #18 0x00007f6832b54096 in list_candidates () from >>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>> #19 0x00007f6832b52f40 in filter_candidates_ext () from >>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>> #20 0x00007f6832b54096 in list_candidates () from >>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>> #21 0x00007f6832b52f40 in filter_candidates_ext () from >>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>> #22 0x00007f6832b8e80c in subtree_candidates () from >>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>> #23 0x00007f6832b8fe7f in ldbm_back_search () from >>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>> #24 0x00007f683f4bd1d6 in op_shared_search () from >>> /usr/lib64/dirsrv/libslapd.so.0 >>> #25 0x00007f683f9ae33e in do_search () >>> #26 0x00007f683f99c3c3 in connection_threadmain () >>> #27 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so >>> #28 0x00007f683d04cdc5 in start_thread () from >>> /lib64/libpthread.so.0 >>> #29 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 >>> Thread 7 (Thread 0x7f67f9feb700 (LWP 4886)): >>> #0 0x00007f683d0506d5 in pthread_cond_wait@@GLIBC_2.3.2 >>> () from >>> /lib64/libpthread.so.0 >>> #1 0x00007f683d6a6050 in PR_WaitCondVar () from >>> /lib64/libnspr4.so >>> #2 0x00007f683f999fee in connection_wait_for_new_work () >>> #3 0x00007f683f99b719 in connection_threadmain () >>> #4 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so >>> #5 0x00007f683d04cdc5 in start_thread () from >>> /lib64/libpthread.so.0 >>> #6 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 >>> Thread 6 (Thread 0x7f67f97ea700 (LWP 4887)): >>> #0 0x00007f683d0506d5 in pthread_cond_wait@@GLIBC_2.3.2 >>> () from >>> /lib64/libpthread.so.0 >>> #1 0x00007f683d6a6050 in PR_WaitCondVar () from >>> /lib64/libnspr4.so >>> #2 0x00007f683f999fee in connection_wait_for_new_work () >>> #3 0x00007f683f99b719 in connection_threadmain () >>> #4 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so >>> #5 0x00007f683d04cdc5 in start_thread () from >>> /lib64/libpthread.so.0 >>> #6 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 >>> Thread 5 (Thread 0x7f67f8fe9700 (LWP 4888)): >>> #0 0x00007f683d0506d5 in pthread_cond_wait@@GLIBC_2.3.2 >>> () from >>> /lib64/libpthread.so.0 >>> #1 0x00007f683d6a6050 in PR_WaitCondVar () from >>> /lib64/libnspr4.so >>> #2 0x00007f683f999fee in connection_wait_for_new_work () >>> #3 0x00007f683f99b719 in connection_threadmain () >>> #4 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so >>> #5 0x00007f683d04cdc5 in start_thread () from >>> /lib64/libpthread.so.0 >>> #6 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 >>> Thread 4 (Thread 0x7f67f87e8700 (LWP 4889)): >>> #0 0x00007f683cd719b3 in select () from /lib64/libc.so.6 >>> #1 0x00007f683f4fc0e9 in DS_Sleep () from >>> /usr/lib64/dirsrv/libslapd.so.0 >>> #2 0x00007f683f99d325 in time_thread () >>> #3 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so >>> #4 0x00007f683d04cdc5 in start_thread () from >>> /lib64/libpthread.so.0 >>> #5 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 >>> Thread 3 (Thread 0x7f67f7de6700 (LWP 4892)): >>> #0 0x00007f683d0506d5 in pthread_cond_wait@@GLIBC_2.3.2 >>> () from >>> /lib64/libpthread.so.0 >>> #1 0x00007f683d6a6050 in PR_WaitCondVar () from >>> /lib64/libnspr4.so >>> #2 0x00007f683f9a8f25 in ps_send_results () >>> #3 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so >>> #4 0x00007f683d04cdc5 in start_thread () from >>> /lib64/libpthread.so.0 >>> #5 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 >>> Thread 2 (Thread 0x7f67f75e5700 (LWP 5084)): >>> #0 0x00007f683d0506d5 in pthread_cond_wait@@GLIBC_2.3.2 >>> () from >>> /lib64/libpthread.so.0 >>> #1 0x00007f683d6a6050 in PR_WaitCondVar () from >>> /lib64/libnspr4.so >>> #2 0x00007f683f9a8f25 in ps_send_results () >>> #3 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so >>> #4 0x00007f683d04cdc5 in start_thread () from >>> /lib64/libpthread.so.0 >>> #5 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 >>> Thread 1 (Thread 0x7f683f964840 (LWP 4845)): >>> #0 0x00007f683d0506d5 in pthread_cond_wait@@GLIBC_2.3.2 >>> () from >>> /lib64/libpthread.so.0 >>> #1 0x00007f683d6a62c3 in PR_EnterMonitor () from >>> /lib64/libnspr4.so >>> #2 0x00007f683f99d5ed in setup_pr_read_pds () >>> #3 0x00007f683f9a0fd9 in slapd_daemon () >>> #4 0x00007f683f992c6c in main () >>> >>> >>> >>> the top output seems ok, there was just a sudden spike to >>> around 340% cpu then it comes down >>> >>> top - 08:53:12 up 6 days, 23:20, 1 user, load average: >>> 0.00, 0.01, 0.12 >>> Threads: 44 total, 0 running, 44 sleeping, 0 >>> stopped, 0 zombie >>> %Cpu(s): 0.0 us, 0.0 sy, 0.0 ni,100.0 id, 0.0 wa, 0.0 >>> hi, 0.0 si, 0.0 st >>> KiB Mem : 65533176 total, 52165676 free, 6135088 used, >>> 7232412 buff/cache >>> KiB Swap: 3999996 total, 3999996 free, 0 used. >>> 58452040 avail Mem >>> >>> PID USER PR NI VIRT RES SHR S %CPU %MEM >>> TIME+ COMMAND >>> 4845 dirsrv 20 0 5133196 3.150g 281272 S 0.0 5.0 >>> 31:34.51 ns-slapd >>> 4847 dirsrv 20 0 5133196 3.150g 281272 S 0.0 5.0 >>> 0:13.53 ns-slapd >>> 4848 dirsrv 20 0 5133196 3.150g 281272 S 0.0 5.0 >>> 0:13.16 ns-slapd >>> 4849 dirsrv 20 0 5133196 3.150g 281272 S 0.0 5.0 >>> 0:45.27 ns-slapd >>> 4850 dirsrv 20 0 5133196 3.150g 281272 S 0.0 5.0 >>> 0:01.60 ns-slapd >>> 4851 dirsrv 20 0 5133196 3.150g 281272 S 0.0 5.0 >>> 0:00.00 ns-slapd >>> 4852 dirsrv 20 0 5133196 3.150g 281272 S 0.0 5.0 >>> 0:00.00 ns-slapd >>> 4853 dirsrv 20 0 5133196 3.150g 281272 S 0.0 5.0 >>> 0:00.00 ns-slapd >>> 4854 dirsrv 20 0 5133196 3.150g 281272 S 0.0 5.0 >>> 0:00.00 ns-slapd >>> 4855 dirsrv 20 0 5133196 3.150g 281272 S 0.0 5.0 >>> 0:03.02 ns-slapd >>> 4856 dirsrv 20 0 5133196 3.150g 281272 S 0.0 5.0 >>> 0:00.26 ns-slapd >>> 4859 dirsrv 20 0 5133196 3.150g 281272 S 0.0 5.0 >>> 3:18.67 ns-slapd >>> 4860 dirsrv 20 0 5133196 3.150g 281272 S 0.0 5.0 >>> 3:19.82 ns-slapd >>> 4861 dirsrv 20 0 5133196 3.150g 281272 S 0.0 5.0 >>> 3:20.23 ns-slapd >>> 4862 dirsrv 20 0 5133196 3.150g 281272 S 0.0 5.0 >>> 3:22.65 ns-slapd >>> 4863 dirsrv 20 0 5133196 3.150g 281272 S 0.0 5.0 >>> 3:16.64 ns-slapd >>> 4864 dirsrv 20 0 5133196 3.150g 281272 S 0.0 5.0 >>> 3:17.39 ns-slapd >>> 4865 dirsrv 20 0 5133196 3.150g 281272 S 0.0 5.0 >>> 3:17.80 ns-slapd >>> >>> >>> Thanks >>> Rakesh >>> >>> On Tue, Sep 6, 2016 at 12:09 AM, Rakesh Rajasekharan >>> >> > wrote: >>> >>> Hi Thierry/Ludwig, >>> >>> Unfortunately I had by then restarted the service and >>> since then have been trying to hit the hang issue again >>> but no luck this time >>> >>> It has been running fine for the last few hours now >>> in-spite of trying all that I believe could cause the >>> hang like making simultaneous client addition requests. >>> >>> I will continue testing this and next time I get it I >>> will make sure I get the pstack and also get the dump of >>> db lock info. >>> >>> Thanks, >>> Rakesh >>> >>> >>> >>> On Mon, Sep 5, 2016 at 6:14 PM, Ludwig Krispenz >>> > wrote: >>> >>> >>> On 09/05/2016 02:31 PM, thierry bordaz wrote: >>>> Hi Rakesh, >>>> >>>> Thanks you soooo much for you continuous help on >>>> this !! >>>> >>>> If the hang is still going on, would you dump the >>>> DB lock information with: >>>> >>>> db_stat -CA -N -h >>>> /var/lib/dirsrv/slapd-/db >>>> >/tmp/db_lock_info >>>> >>>> It is looking like several threads are waiting for >>>> one/several db locks. With the lock info we will >>>> know which thread acquired it/them. >>>> A guess is that it could be the checkpointing >>>> thread but this one is also waiting (on a memory pool). >>>> >>>> I am copying Ludwig on this thread in case this >>>> pstack rings any bell. >>> the frequent presence of __db_tas_mutex_lock() looks >>> like the NUMA problem of libdb, although the threads >>> already have stopped spinning (maybe this was an >>> effect of gcore). >>> If the hang aoccurs again, could you check cpu >>> consumption >>> top -H -p >>> and get a pstack >>> pstack >>>> >>>> best regards >>>> thierry >>>> >>>> On 09/05/2016 02:05 PM, Rakesh Rajasekharan wrote: >>>>> Hi Thierry, >>>>> >>>>> I was able to now reporduce the issue.. >>>>> >>>>> the ns-slapd process just got hung,.. however, no >>>>> core file was generate.. >>>>> >>>>> I manually created a coredump using gcore and then >>>>> took a strace of it >>>>> this is strace from the coredump generated >>>>> >>>>> GNU gdb (GDB) Red Hat Enterprise Linux 7.6.1-80.el7 >>>>> Copyright (C) 2013 Free Software Foundation, Inc. >>>>> License GPLv3+: GNU GPL version 3 or later >>>>> >>>>> This is free software: you are free to change and >>>>> redistribute it. >>>>> There is NO WARRANTY, to the extent permitted by >>>>> law. Type "show copying" >>>>> and "show warranty" for details. >>>>> This GDB was configured as "x86_64-redhat-linux-gnu". >>>>> For bug reporting instructions, please see: >>>>> >>>> >... >>>>> Reading symbols from /usr/sbin/ns-slapd...Reading >>>>> symbols from /usr/sbin/ns-slapd...(no debugging >>>>> symbols found)...done. >>>>> (no debugging symbols found)...done. >>>>> [New LWP 105159] >>>>> [New LWP 105160] >>>>> [New LWP 105161] >>>>> [New LWP 105162] >>>>> [New LWP 105163] >>>>> [New LWP 105164] >>>>> [New LWP 105165] >>>>> [New LWP 105166] >>>>> [New LWP 105167] >>>>> [New LWP 105168] >>>>> [New LWP 105171] >>>>> [New LWP 105172] >>>>> [New LWP 105173] >>>>> [New LWP 105174] >>>>> [New LWP 105175] >>>>> [New LWP 105176] >>>>> [New LWP 105177] >>>>> [New LWP 105178] >>>>> [New LWP 105179] >>>>> [New LWP 105180] >>>>> [New LWP 105181] >>>>> [New LWP 105182] >>>>> [New LWP 105183] >>>>> [New LWP 105184] >>>>> [New LWP 105185] >>>>> [New LWP 105186] >>>>> [New LWP 105187] >>>>> [New LWP 105188] >>>>> [New LWP 105189] >>>>> [New LWP 105190] >>>>> [New LWP 105191] >>>>> [New LWP 105192] >>>>> [New LWP 105193] >>>>> [New LWP 105194] >>>>> [New LWP 105195] >>>>> [New LWP 105196] >>>>> [New LWP 105197] >>>>> [New LWP 105198] >>>>> [New LWP 105199] >>>>> [New LWP 105200] >>>>> [New LWP 105201] >>>>> [New LWP 105203] >>>>> [New LWP 105215] >>>>> [New LWP 105158] >>>>> [Thread debugging using libthread_db enabled] >>>>> Using host libthread_db library >>>>> "/lib64/libthread_db.so.1". >>>>> Core was generated by `/usr/sbin/ns-slapd'. >>>>> #0 0x00007f1b48ae69b3 in select () from >>>>> /lib64/libc.so.6 >>>>> >>>>> Thread 44 (Thread 0x7f1b4b6d9840 (LWP 105158)): >>>>> #0 0x00007f1b48dc56d5 in >>>>> pthread_cond_wait@@GLIBC_2.3.2 >>>>> () from >>>>> /lib64/libpthread.so.0 >>>>> No symbol table info available. >>>>> #1 0x00007f1b4941b2c3 in PR_EnterMonitor () from >>>>> /lib64/libnspr4.so >>>>> No symbol table info available. >>>>> #2 0x00007f1b4b7125ed in setup_pr_read_pds () >>>>> No symbol table info available. >>>>> #3 0x00007f1b4b715fd9 in slapd_daemon () >>>>> No symbol table info available. >>>>> #4 0x00007f1b4b707c6c in main () >>>>> No symbol table info available. >>>>> >>>>> Thread 43 (Thread 0x7f1a5df0f700 (LWP 105215)): >>>>> #0 0x00007f1b48dc56d5 in >>>>> pthread_cond_wait@@GLIBC_2.3.2 >>>>> () from >>>>> /lib64/libpthread.so.0 >>>>> No symbol table info available. >>>>> #1 0x00007f1b4941b050 in PR_WaitCondVar () from >>>>> /lib64/libnspr4.so >>>>> No symbol table info available. >>>>> #2 0x00007f1b4b71df25 in ps_send_results () >>>>> No symbol table info available. >>>>> #3 0x00007f1b494207bb in _pt_root () from >>>>> /lib64/libnspr4.so >>>>> No symbol table info available. >>>>> #4 0x00007f1b48dc1dc5 in start_thread () from >>>>> /lib64/libpthread.so.0 >>>>> No symbol table info available. >>>>> #5 0x00007f1b48aef28d in clone () from >>>>> /lib64/libc.so.6 >>>>> No symbol table info available. >>>>> >>>>> Thread 42 (Thread 0x7f1b03fe7700 (LWP 105203)): >>>>> #0 0x00007f1b48dc56d5 in >>>>> pthread_cond_wait@@GLIBC_2.3.2 >>>>> () from >>>>> /lib64/libpthread.so.0 >>>>> No symbol table info available. >>>>> #1 0x00007f1b4941b050 in PR_WaitCondVar () from >>>>> /lib64/libnspr4.so >>>>> No symbol table info available. >>>>> #2 0x00007f1b4b71df25 in ps_send_results () >>>>> No symbol table info available. >>>>> #3 0x00007f1b494207bb in _pt_root () from >>>>> /lib64/libnspr4.so >>>>> No symbol table info available. >>>>> #4 0x00007f1b48dc1dc5 in start_thread () from >>>>> /lib64/libpthread.so.0 >>>>> No symbol table info available. >>>>> #5 0x00007f1b48aef28d in clone () from >>>>> /lib64/libc.so.6 >>>>> No symbol table info available. >>>>> >>>>> Thread 41 (Thread 0x7f1b047e8700 (LWP 105201)): >>>>> #0 0x00007f1b48ae69b3 in select () from >>>>> /lib64/libc.so.6 >>>>> No symbol table info available. >>>>> #1 0x00007f1b4b2710e9 in DS_Sleep () from >>>>> /usr/lib64/dirsrv/libslapd.so.0 >>>>> No symbol table info available. >>>>> #2 0x00007f1b4b712325 in time_thread () >>>>> No symbol table info available. >>>>> #3 0x00007f1b494207bb in _pt_root () from >>>>> /lib64/libnspr4.so >>>>> No symbol table info available. >>>>> #4 0x00007f1b48dc1dc5 in start_thread () from >>>>> /lib64/libpthread.so.0 >>>>> No symbol table info available. >>>>> #5 0x00007f1b48aef28d in clone () from >>>>> /lib64/libc.so.6 >>>>> No symbol table info available. >>>>> >>>>> Thread 40 (Thread 0x7f1b04fe9700 (LWP 105200)): >>>>> #0 0x00007f1b48dc56d5 in >>>>> pthread_cond_wait@@GLIBC_2.3.2 >>>>> () from >>>>> /lib64/libpthread.so.0 >>>>> No symbol table info available. >>>>> #1 0x00007f1b433212f3 in __db_hybrid_mutex_suspend >>>>> () from /lib64/libdb-5.3.so >>>>> No symbol table info available. >>>>> #2 0x00007f1b43320640 in __db_tas_mutex_lock () >>>>> from /lib64/libdb-5.3.so >>>>> No symbol table info available. >>>>> #3 0x00007f1b4343ae1b in __memp_fget () from >>>>> /lib64/libdb-5.3.so >>>>> No symbol table info available. >>>>> #4 0x00007f1b4333e211 in __bam_search () from >>>>> /lib64/libdb-5.3.so >>>>> No symbol table info available. >>>>> #5 0x00007f1b43329256 in __bamc_search () from >>>>> /lib64/libdb-5.3.so >>>>> No symbol table info available. >>>>> #6 0x00007f1b4332d6c4 in __bamc_put () from >>>>> /lib64/libdb-5.3.so >>>>> No symbol table info available. >>>>> #7 0x00007f1b433e46d5 in __dbc_iput () from >>>>> /lib64/libdb-5.3.so >>>>> No symbol table info available. >>>>> #8 0x00007f1b433dfa0e in __db_put () from >>>>> /lib64/libdb-5.3.so >>>>> No symbol table info available. >>>>> #9 0x00007f1b433f4fa4 in __db_put_pp () from >>>>> /lib64/libdb-5.3.so >>>>> No symbol table info available. >>>>> #10 0x00007f1b3e8cf4cf in idl_new_insert_key () >>>>> from /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>>> No symbol table info available. >>>>> #11 0x00007f1b3e8de5b5 in >>>>> addordel_values_sv.isra.3 () from >>>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>>> No symbol table info available. >>>>> #12 0x00007f1b3e8df25a in >>>>> index_addordel_values_ext_sv () from >>>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>>> No symbol table info available. >>>>> #13 0x00007f1b3e8df694 in index_addordel_values_sv >>>>> () from /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>>> No symbol table info available. >>>>> #14 0x00007f1b3e8df75a in index_addordel_entry () >>>>> from /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>>> No symbol table info available. >>>>> #15 0x00007f1b3e8e25fa in ldbm_back_add () from >>>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>>> No symbol table info available. >>>>> #16 0x00007f1b4b1e4cca in op_shared_add () from >>>>> /usr/lib64/dirsrv/libslapd.so.0 >>>>> No symbol table info available. >>>>> #17 0x00007f1b4b1e6020 in do_add () from >>>>> /usr/lib64/dirsrv/libslapd.so.0 >>>>> No symbol table info available. >>>>> #18 0x00007f1b4b711303 in connection_threadmain () >>>>> No symbol table info available. >>>>> #19 0x00007f1b494207bb in _pt_root () from >>>>> /lib64/libnspr4.so >>>>> No symbol table info available. >>>>> #20 0x00007f1b48dc1dc5 in start_thread () from >>>>> /lib64/libpthread.so.0 >>>>> No symbol table info available. >>>>> #21 0x00007f1b48aef28d in clone () from >>>>> /lib64/libc.so.6 >>>>> No symbol table info available. >>>>> >>>>> Thread 39 (Thread 0x7f1b057ea700 (LWP 105199)): >>>>> #0 0x00007f1b48dc56d5 in >>>>> pthread_cond_wait@@GLIBC_2.3.2 >>>>> () from >>>>> /lib64/libpthread.so.0 >>>>> No symbol table info available. >>>>> #1 0x00007f1b4941b050 in PR_WaitCondVar () from >>>>> /lib64/libnspr4.so >>>>> No symbol table info available. >>>>> #2 0x00007f1b4b70efee in >>>>> connection_wait_for_new_work () >>>>> No symbol table info available. >>>>> #3 0x00007f1b4b710719 in connection_threadmain () >>>>> No symbol table info available. >>>>> #4 0x00007f1b494207bb in _pt_root () from >>>>> /lib64/libnspr4.so >>>>> No symbol table info available. >>>>> #5 0x00007f1b48dc1dc5 in start_thread () from >>>>> /lib64/libpthread.so.0 >>>>> No symbol table info available. >>>>> #6 0x00007f1b48aef28d in clone () from >>>>> /lib64/libc.so.6 >>>>> No symbol table info available. >>>>> >>>>> Thread 38 (Thread 0x7f1b05feb700 (LWP 105198)): >>>>> #0 0x00007f1b48dc56d5 in >>>>> pthread_cond_wait@@GLIBC_2.3.2 >>>>> () from >>>>> /lib64/libpthread.so.0 >>>>> No symbol table info available. >>>>> #1 0x00007f1b4941b050 in PR_WaitCondVar () from >>>>> /lib64/libnspr4.so >>>>> No symbol table info available. >>>>> #2 0x00007f1b4b70efee in >>>>> connection_wait_for_new_work () >>>>> No symbol table info available. >>>>> #3 0x00007f1b4b710719 in connection_threadmain () >>>>> No symbol table info available. >>>>> #4 0x00007f1b494207bb in _pt_root () from >>>>> /lib64/libnspr4.so >>>>> No symbol table info available. >>>>> #5 0x00007f1b48dc1dc5 in start_thread () from >>>>> /lib64/libpthread.so.0 >>>>> No symbol table info available. >>>>> #6 0x00007f1b48aef28d in clone () from >>>>> /lib64/libc.so.6 >>>>> No symbol table info available. >>>>> >>>>> Thread 37 (Thread 0x7f1b067ec700 (LWP 105197)): >>>>> #0 0x00007f1b48dc56d5 in >>>>> pthread_cond_wait@@GLIBC_2.3.2 >>>>> () from >>>>> /lib64/libpthread.so.0 >>>>> No symbol table info available. >>>>> #1 0x00007f1b4941b050 in PR_WaitCondVar () from >>>>> /lib64/libnspr4.so >>>>> No symbol table info available. >>>>> #2 0x00007f1b4b70efee in >>>>> connection_wait_for_new_work () >>>>> No symbol table info available. >>>>> #3 0x00007f1b4b710719 in connection_threadmain () >>>>> No symbol table info available. >>>>> #4 0x00007f1b494207bb in _pt_root () from >>>>> /lib64/libnspr4.so >>>>> No symbol table info available. >>>>> #5 0x00007f1b48dc1dc5 in start_thread () from >>>>> /lib64/libpthread.so.0 >>>>> No symbol table info available. >>>>> #6 0x00007f1b48aef28d in clone () from >>>>> /lib64/libc.so.6 >>>>> No symbol table info available. >>>>> >>>>> Thread 36 (Thread 0x7f1b06fed700 (LWP 105196)): >>>>> #0 0x00007f1b48dc56d5 in >>>>> pthread_cond_wait@@GLIBC_2.3.2 >>>>> () from >>>>> /lib64/libpthread.so.0 >>>>> No symbol table info available. >>>>> #1 0x00007f1b4941b050 in PR_WaitCondVar () from >>>>> /lib64/libnspr4.so >>>>> No symbol table info available. >>>>> #2 0x00007f1b4b70efee in >>>>> connection_wait_for_new_work () >>>>> No symbol table info available. >>>>> #3 0x00007f1b4b710719 in connection_threadmain () >>>>> No symbol table info available. >>>>> #4 0x00007f1b494207bb in _pt_root () from >>>>> /lib64/libnspr4.so >>>>> No symbol table info available. >>>>> #5 0x00007f1b48dc1dc5 in start_thread () from >>>>> /lib64/libpthread.so.0 >>>>> No symbol table info available. >>>>> #6 0x00007f1b48aef28d in clone () from >>>>> /lib64/libc.so.6 >>>>> No symbol table info available. >>>>> >>>>> Thread 35 (Thread 0x7f1b077ee700 (LWP 105195)): >>>>> #0 0x00007f1b48dc56d5 in >>>>> pthread_cond_wait@@GLIBC_2.3.2 >>>>> () from >>>>> /lib64/libpthread.so.0 >>>>> No symbol table info available. >>>>> #1 0x00007f1b4941b2c3 in PR_EnterMonitor () from >>>>> /lib64/libnspr4.so >>>>> No symbol table info available. >>>>> #2 0x00007f1b3e8c23e6 in dblayer_txn_begin () from >>>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>>> No symbol table info available. >>>>> #3 0x00007f1b3e8fe007 in ldbm_back_modify () from >>>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>>> No symbol table info available. >>>>> #4 0x00007f1b4b22b541 in op_shared_modify () from >>>>> /usr/lib64/dirsrv/libslapd.so.0 >>>>> No symbol table info available. >>>>> #5 0x00007f1b4b22c004 in modify_internal_pb () >>>>> from /usr/lib64/dirsrv/libslapd.so.0 >>>>> No symbol table info available. >>>>> #6 0x00007f1b3eb48b60 in ipapwd_apply_mods () from >>>>> /usr/lib64/dirsrv/plugins/libipa_pwd_extop.so >>>>> No symbol table info available. >>>>> #7 0x00007f1b3eb4c8e0 in store_new_keys () from >>>>> /usr/lib64/dirsrv/plugins/libipa_pwd_extop.so >>>>> No symbol table info available. >>>>> #8 0x00007f1b3eb4ead0 in ipapwd_extop () from >>>>> /usr/lib64/dirsrv/plugins/libipa_pwd_extop.so >>>>> No symbol table info available. >>>>> #9 0x00007f1b4b23bba2 in plugin_call_exop_plugins >>>>> () from /usr/lib64/dirsrv/libslapd.so.0 >>>>> No symbol table info available. >>>>> #10 0x00007f1b4b717efe in do_extended () >>>>> No symbol table info available. >>>>> #11 0x00007f1b4b71141a in connection_threadmain () >>>>> No symbol table info available. >>>>> #12 0x00007f1b494207bb in _pt_root () from >>>>> /lib64/libnspr4.so >>>>> No symbol table info available. >>>>> #13 0x00007f1b48dc1dc5 in start_thread () from >>>>> /lib64/libpthread.so.0 >>>>> No symbol table info available. >>>>> #14 0x00007f1b48aef28d in clone () from >>>>> /lib64/libc.so.6 >>>>> No symbol table info available. >>>>> >>>>> Thread 34 (Thread 0x7f1b07fef700 (LWP 105194)): >>>>> #0 0x00007f1b48dc56d5 in >>>>> pthread_cond_wait@@GLIBC_2.3.2 >>>>> () from >>>>> /lib64/libpthread.so.0 >>>>> No symbol table info available. >>>>> #1 0x00007f1b4941b050 in PR_WaitCondVar () from >>>>> /lib64/libnspr4.so >>>>> No symbol table info available. >>>>> #2 0x00007f1b4b70efee in >>>>> connection_wait_for_new_work () >>>>> No symbol table info available. >>>>> #3 0x00007f1b4b710719 in connection_threadmain () >>>>> No symbol table info available. >>>>> #4 0x00007f1b494207bb in _pt_root () from >>>>> /lib64/libnspr4.so >>>>> No symbol table info available. >>>>> #5 0x00007f1b48dc1dc5 in start_thread () from >>>>> /lib64/libpthread.so.0 >>>>> No symbol table info available. >>>>> #6 0x00007f1b48aef28d in clone () from >>>>> /lib64/libc.so.6 >>>>> No symbol table info available. >>>>> >>>>> Thread 33 (Thread 0x7f1b087f0700 (LWP 105193)): >>>>> #0 0x00007f1b48dc56d5 in >>>>> pthread_cond_wait@@GLIBC_2.3.2 >>>>> () from >>>>> /lib64/libpthread.so.0 >>>>> No symbol table info available. >>>>> #1 0x00007f1b4941b050 in PR_WaitCondVar () from >>>>> /lib64/libnspr4.so >>>>> No symbol table info available. >>>>> #2 0x00007f1b4b70efee in >>>>> connection_wait_for_new_work () >>>>> No symbol table info available. >>>>> #3 0x00007f1b4b710719 in connection_threadmain () >>>>> No symbol table info available. >>>>> #4 0x00007f1b494207bb in _pt_root () from >>>>> /lib64/libnspr4.so >>>>> No symbol table info available. >>>>> #5 0x00007f1b48dc1dc5 in start_thread () from >>>>> /lib64/libpthread.so.0 >>>>> No symbol table info available. >>>>> #6 0x00007f1b48aef28d in clone () from >>>>> /lib64/libc.so.6 >>>>> No symbol table info available. >>>>> >>>>> Thread 32 (Thread 0x7f1b08ff1700 (LWP 105192)): >>>>> #0 0x00007f1b48dc56d5 in >>>>> pthread_cond_wait@@GLIBC_2.3.2 >>>>> () from >>>>> /lib64/libpthread.so.0 >>>>> No symbol table info available. >>>>> #1 0x00007f1b4941b050 in PR_WaitCondVar () from >>>>> /lib64/libnspr4.so >>>>> No symbol table info available. >>>>> #2 0x00007f1b4b70efee in >>>>> connection_wait_for_new_work () >>>>> No symbol table info available. >>>>> #3 0x00007f1b4b710719 in connection_threadmain () >>>>> No symbol table info available. >>>>> #4 0x00007f1b494207bb in _pt_root () from >>>>> /lib64/libnspr4.so >>>>> No symbol table info available. >>>>> #5 0x00007f1b48dc1dc5 in start_thread () from >>>>> /lib64/libpthread.so.0 >>>>> No symbol table info available. >>>>> #6 0x00007f1b48aef28d in clone () from >>>>> /lib64/libc.so.6 >>>>> No symbol table info available. >>>>> >>>>> Thread 31 (Thread 0x7f1b097f2700 (LWP 105191)): >>>>> #0 0x00007f1b48dc56d5 in >>>>> pthread_cond_wait@@GLIBC_2.3.2 >>>>> () from >>>>> /lib64/libpthread.so.0 >>>>> No symbol table info available. >>>>> #1 0x00007f1b4941b050 in PR_WaitCondVar () from >>>>> /lib64/libnspr4.so >>>>> No symbol table info available. >>>>> #2 0x00007f1b4b70efee in >>>>> connection_wait_for_new_work () >>>>> No symbol table info available. >>>>> #3 0x00007f1b4b710719 in connection_threadmain () >>>>> No symbol table info available. >>>>> #4 0x00007f1b494207bb in _pt_root () from >>>>> /lib64/libnspr4.so >>>>> No symbol table info available. >>>>> #5 0x00007f1b48dc1dc5 in start_thread () from >>>>> /lib64/libpthread.so.0 >>>>> No symbol table info available. >>>>> #6 0x00007f1b48aef28d in clone () from >>>>> /lib64/libc.so.6 >>>>> No symbol table info available. >>>>> >>>>> Thread 30 (Thread 0x7f1b09ff3700 (LWP 105190)): >>>>> #0 0x00007f1b48dc56d5 in >>>>> pthread_cond_wait@@GLIBC_2.3.2 >>>>> () from >>>>> /lib64/libpthread.so.0 >>>>> No symbol table info available. >>>>> #1 0x00007f1b433212f3 in __db_hybrid_mutex_suspend >>>>> () from /lib64/libdb-5.3.so >>>>> No symbol table info available. >>>>> #2 0x00007f1b43320640 in __db_tas_mutex_lock () >>>>> from /lib64/libdb-5.3.so >>>>> No symbol table info available. >>>>> #3 0x00007f1b433cad3a in __lock_get_internal () >>>>> from /lib64/libdb-5.3.so >>>>> No symbol table info available. >>>>> #4 0x00007f1b433cb820 in __lock_get () from >>>>> /lib64/libdb-5.3.so >>>>> No symbol table info available. >>>>> #5 0x00007f1b433f7142 in __db_lget () from >>>>> /lib64/libdb-5.3.so >>>>> No symbol table info available. >>>>> #6 0x00007f1b4333e605 in __bam_search () from >>>>> /lib64/libdb-5.3.so >>>>> No symbol table info available. >>>>> #7 0x00007f1b43329256 in __bamc_search () from >>>>> /lib64/libdb-5.3.so >>>>> No symbol table info available. >>>>> #8 0x00007f1b4332ad0f in __bamc_get () from >>>>> /lib64/libdb-5.3.so >>>>> No symbol table info available. >>>>> #9 0x00007f1b433e3ca6 in __dbc_iget () from >>>>> /lib64/libdb-5.3.so >>>>> No symbol table info available. >>>>> #10 0x00007f1b433f2b02 in __dbc_get_pp () from >>>>> /lib64/libdb-5.3.so >>>>> No symbol table info available. >>>>> #11 0x00007f1b3e8ce680 in idl_new_fetch () from >>>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>>> No symbol table info available. >>>>> #12 0x00007f1b3e8dcb26 in index_read_ext_allids () >>>>> from /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>>> No symbol table info available. >>>>> #13 0x00007f1b3e8c7284 in keys2idl () from >>>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>>> No symbol table info available. >>>>> #14 0x00007f1b3e8c79e3 in ava_candidates.isra.0 () >>>>> from /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>>> No symbol table info available. >>>>> #15 0x00007f1b3e8c7fd2 in filter_candidates_ext () >>>>> from /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>>> No symbol table info available. >>>>> #16 0x00007f1b3e8c9096 in list_candidates () from >>>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>>> No symbol table info available. >>>>> #17 0x00007f1b3e8c7f40 in filter_candidates_ext () >>>>> from /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>>> No symbol table info available. >>>>> #18 0x00007f1b3e8c9096 in list_candidates () from >>>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>>> No symbol table info available. >>>>> #19 0x00007f1b3e8c7f40 in filter_candidates_ext () >>>>> from /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>>> No symbol table info available. >>>>> #20 0x00007f1b3e8c9096 in list_candidates () from >>>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>>> No symbol table info available. >>>>> #21 0x00007f1b3e8c7f40 in filter_candidates_ext () >>>>> from /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>>> No symbol table info available. >>>>> #22 0x00007f1b3e90380c in subtree_candidates () >>>>> from /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>>> No symbol table info available. >>>>> #23 0x00007f1b3e904e7f in ldbm_back_search () from >>>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>>> No symbol table info available. >>>>> #24 0x00007f1b4b2321d6 in op_shared_search () from >>>>> /usr/lib64/dirsrv/libslapd.so.0 >>>>> No symbol table info available. >>>>> #25 0x00007f1b4b72333e in do_search () >>>>> No symbol table info available. >>>>> #26 0x00007f1b4b7113c3 in connection_threadmain () >>>>> No symbol table info available. >>>>> #27 0x00007f1b494207bb in _pt_root () from >>>>> /lib64/libnspr4.so >>>>> No symbol table info available. >>>>> #28 0x00007f1b48dc1dc5 in start_thread () from >>>>> /lib64/libpthread.so.0 >>>>> No symbol table info available. >>>>> #29 0x00007f1b48aef28d in clone () from >>>>> /lib64/libc.so.6 >>>>> No symbol table info available. >>>>> >>>>> Thread 29 (Thread 0x7f1b0a7f4700 (LWP 105189)): >>>>> #0 0x00007f1b48dc56d5 in >>>>> pthread_cond_wait@@GLIBC_2.3.2 >>>>> () from >>>>> /lib64/libpthread.so.0 >>>>> No symbol table info available. >>>>> #1 0x00007f1b4941b050 in PR_WaitCondVar () from >>>>> /lib64/libnspr4.so >>>>> No symbol table info available. >>>>> #2 0x00007f1b4b70efee in >>>>> connection_wait_for_new_work () >>>>> No symbol table info available. >>>>> #3 0x00007f1b4b710719 in connection_threadmain () >>>>> No symbol table info available. >>>>> #4 0x00007f1b494207bb in _pt_root () from >>>>> /lib64/libnspr4.so >>>>> No symbol table info available. >>>>> #5 0x00007f1b48dc1dc5 in start_thread () from >>>>> /lib64/libpthread.so.0 >>>>> No symbol table info available. >>>>> #6 0x00007f1b48aef28d in clone () from >>>>> /lib64/libc.so.6 >>>>> No symbol table info available. >>>>> >>>>> Thread 28 (Thread 0x7f1b0aff5700 (LWP 105188)): >>>>> #0 0x00007f1b48dc56d5 in >>>>> pthread_cond_wait@@GLIBC_2.3.2 >>>>> () from >>>>> /lib64/libpthread.so.0 >>>>> No symbol table info available. >>>>> #1 0x00007f1b4941b050 in PR_WaitCondVar () from >>>>> /lib64/libnspr4.so >>>>> No symbol table info available. >>>>> #2 0x00007f1b4b70efee in >>>>> connection_wait_for_new_work () >>>>> No symbol table info available. >>>>> #3 0x00007f1b4b710719 in connection_threadmain () >>>>> No symbol table info available. >>>>> #4 0x00007f1b494207bb in _pt_root () from >>>>> /lib64/libnspr4.so >>>>> No symbol table info available. >>>>> #5 0x00007f1b48dc1dc5 in start_thread () from >>>>> /lib64/libpthread.so.0 >>>>> No symbol table info available. >>>>> #6 0x00007f1b48aef28d in clone () from >>>>> /lib64/libc.so.6 >>>>> No symbol table info available. >>>>> >>>>> Thread 27 (Thread 0x7f1b0b7f6700 (LWP 105187)): >>>>> #0 0x00007f1b48dc56d5 in >>>>> pthread_cond_wait@@GLIBC_2.3.2 >>>>> () from >>>>> /lib64/libpthread.so.0 >>>>> No symbol table info available. >>>>> #1 0x00007f1b4941b050 in PR_WaitCondVar () from >>>>> /lib64/libnspr4.so >>>>> No symbol table info available. >>>>> #2 0x00007f1b4b70efee in >>>>> connection_wait_for_new_work () >>>>> No symbol table info available. >>>>> #3 0x00007f1b4b710719 in connection_threadmain () >>>>> No symbol table info available. >>>>> #4 0x00007f1b494207bb in _pt_root () from >>>>> /lib64/libnspr4.so >>>>> No symbol table info available. >>>>> #5 0x00007f1b48dc1dc5 in start_thread () from >>>>> /lib64/libpthread.so.0 >>>>> No symbol table info available. >>>>> #6 0x00007f1b48aef28d in clone () from >>>>> /lib64/libc.so.6 >>>>> No symbol table info available. >>>>> >>>>> Thread 26 (Thread 0x7f1b0bff7700 (LWP 105186)): >>>>> #0 0x00007f1b48dc56d5 in >>>>> pthread_cond_wait@@GLIBC_2.3.2 >>>>> () from >>>>> /lib64/libpthread.so.0 >>>>> No symbol table info available. >>>>> #1 0x00007f1b433212f3 in __db_hybrid_mutex_suspend >>>>> () from /lib64/libdb-5.3.so >>>>> No symbol table info available. >>>>> #2 0x00007f1b43320640 in __db_tas_mutex_lock () >>>>> from /lib64/libdb-5.3.so >>>>> No symbol table info available. >>>>> #3 0x00007f1b433cad3a in __lock_get_internal () >>>>> from /lib64/libdb-5.3.so >>>>> No symbol table info available. >>>>> #4 0x00007f1b433cb820 in __lock_get () from >>>>> /lib64/libdb-5.3.so >>>>> No symbol table info available. >>>>> #5 0x00007f1b433f7142 in __db_lget () from >>>>> /lib64/libdb-5.3.so >>>>> No symbol table info available. >>>>> #6 0x00007f1b4333e605 in __bam_search () from >>>>> /lib64/libdb-5.3.so >>>>> No symbol table info available. >>>>> #7 0x00007f1b43329256 in __bamc_search () from >>>>> /lib64/libdb-5.3.so >>>>> No symbol table info available. >>>>> #8 0x00007f1b4332ad0f in __bamc_get () from >>>>> /lib64/libdb-5.3.so >>>>> No symbol table info available. >>>>> #9 0x00007f1b433e3ca6 in __dbc_iget () from >>>>> /lib64/libdb-5.3.so >>>>> No symbol table info available. >>>>> #10 0x00007f1b433f2b02 in __dbc_get_pp () from >>>>> /lib64/libdb-5.3.so >>>>> No symbol table info available. >>>>> #11 0x00007f1b3e8ce680 in idl_new_fetch () from >>>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>>> No symbol table info available. >>>>> #12 0x00007f1b3e8dcb26 in index_read_ext_allids () >>>>> from /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>>> No symbol table info available. >>>>> #13 0x00007f1b3e8c7284 in keys2idl () from >>>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>>> No symbol table info available. >>>>> #14 0x00007f1b3e8c79e3 in ava_candidates.isra.0 () >>>>> from /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>>> No symbol table info available. >>>>> #15 0x00007f1b3e8c7fd2 in filter_candidates_ext () >>>>> from /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>>> No symbol table info available. >>>>> #16 0x00007f1b3e8c9096 in list_candidates () from >>>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>>> No symbol table info available. >>>>> #17 0x00007f1b3e8c7f40 in filter_candidates_ext () >>>>> from /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>>> No symbol table info available. >>>>> #18 0x00007f1b3e90380c in subtree_candidates () >>>>> from /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>>> No symbol table info available. >>>>> #19 0x00007f1b3e904e7f in ldbm_back_search () from >>>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>>> No symbol table info available. >>>>> #20 0x00007f1b4b2321d6 in op_shared_search () from >>>>> /usr/lib64/dirsrv/libslapd.so.0 >>>>> No symbol table info available. >>>>> #21 0x00007f1b4b24294e in >>>>> search_internal_callback_pb () from >>>>> /usr/lib64/dirsrv/libslapd.so.0 >>>>> No symbol table info available. >>>>> #22 0x00007f1b4b242be8 in search_internal_pb () >>>>> from /usr/lib64/dirsrv/libslapd.so.0 >>>>> No symbol table info available. >>>>> #23 0x00007f1b4b72149b in ids_sasl_canon_user () >>>>> No symbol table info available. >>>>> #24 0x00007f1b491e193b in _sasl_canon_user () from >>>>> /lib64/libsasl2.so.3 >>>>> No symbol table info available. >>>>> #25 0x00007f1b491e1c4c in _sasl_canon_user_lookup >>>>> () from /lib64/libsasl2.so.3 >>>>> No symbol table info available. >>>>> #26 0x00007f1b430e9fb9 in gssapi_server_mech_step >>>>> () from /usr/lib64/sasl2/libgssapiv2.so >>>>> No symbol table info available. >>>>> #27 0x00007f1b491edb9b in sasl_server_step () from >>>>> /lib64/libsasl2.so.3 >>>>> No symbol table info available. >>>>> #28 0x00007f1b4b72283b in ids_sasl_check_bind () >>>>> No symbol table info available. >>>>> #29 0x00007f1b4b70a61a in do_bind () >>>>> No symbol table info available. >>>>> #30 0x00007f1b4b7113fd in connection_threadmain () >>>>> No symbol table info available. >>>>> #31 0x00007f1b494207bb in _pt_root () from >>>>> /lib64/libnspr4.so >>>>> No symbol table info available. >>>>> #32 0x00007f1b48dc1dc5 in start_thread () from >>>>> /lib64/libpthread.so.0 >>>>> No symbol table info available. >>>>> #33 0x00007f1b48aef28d in clone () from >>>>> /lib64/libc.so.6 >>>>> No symbol table info available. >>>>> >>>>> Thread 25 (Thread 0x7f1b0c7f8700 (LWP 105185)): >>>>> #0 0x00007f1b48dc56d5 in >>>>> pthread_cond_wait@@GLIBC_2.3.2 >>>>> () from >>>>> /lib64/libpthread.so.0 >>>>> No symbol table info available. >>>>> #1 0x00007f1b4941b050 in PR_WaitCondVar () from >>>>> /lib64/libnspr4.so >>>>> No symbol table info available. >>>>> #2 0x00007f1b4b70efee in >>>>> connection_wait_for_new_work () >>>>> No symbol table info available. >>>>> #3 0x00007f1b4b710719 in connection_threadmain () >>>>> No symbol table info available. >>>>> #4 0x00007f1b494207bb in _pt_root () from >>>>> /lib64/libnspr4.so >>>>> No symbol table info available. >>>>> #5 0x00007f1b48dc1dc5 in start_thread () from >>>>> /lib64/libpthread.so.0 >>>>> No symbol table info available. >>>>> #6 0x00007f1b48aef28d in clone () from >>>>> /lib64/libc.so.6 >>>>> No symbol table info available. >>>>> >>>>> Thread 24 (Thread 0x7f1b0cff9700 (LWP 105184)): >>>>> #0 0x00007f1b48dc56d5 in >>>>> pthread_cond_wait@@GLIBC_2.3.2 >>>>> () from >>>>> /lib64/libpthread.so.0 >>>>> No symbol table info available. >>>>> #1 0x00007f1b4941b050 in PR_WaitCondVar () from >>>>> /lib64/libnspr4.so >>>>> No symbol table info available. >>>>> #2 0x00007f1b4b70efee in >>>>> connection_wait_for_new_work () >>>>> No symbol table info available. >>>>> #3 0x00007f1b4b710719 in connection_threadmain () >>>>> No symbol table info available. >>>>> #4 0x00007f1b494207bb in _pt_root () from >>>>> /lib64/libnspr4.so >>>>> No symbol table info available. >>>>> #5 0x00007f1b48dc1dc5 in start_thread () from >>>>> /lib64/libpthread.so.0 >>>>> No symbol table info available. >>>>> #6 0x00007f1b48aef28d in clone () from >>>>> /lib64/libc.so.6 >>>>> No symbol table info available. >>>>> >>>>> Thread 23 (Thread 0x7f1b0d7fa700 (LWP 105183)): >>>>> #0 0x00007f1b48dc56d5 in >>>>> pthread_cond_wait@@GLIBC_2.3.2 >>>>> () from >>>>> /lib64/libpthread.so.0 >>>>> No symbol table info available. >>>>> #1 0x00007f1b4941b050 in PR_WaitCondVar () from >>>>> /lib64/libnspr4.so >>>>> No symbol table info available. >>>>> #2 0x00007f1b4b70efee in >>>>> connection_wait_for_new_work () >>>>> No symbol table info available. >>>>> #3 0x00007f1b4b710719 in connection_threadmain () >>>>> No symbol table info available. >>>>> #4 0x00007f1b494207bb in _pt_root () from >>>>> /lib64/libnspr4.so >>>>> No symbol table info available. >>>>> #5 0x00007f1b48dc1dc5 in start_thread () from >>>>> /lib64/libpthread.so.0 >>>>> No symbol table info available. >>>>> #6 0x00007f1b48aef28d in clone () from >>>>> /lib64/libc.so.6 >>>>> No symbol table info available. >>>>> >>>>> Thread 22 (Thread 0x7f1b0dffb700 (LWP 105182)): >>>>> #0 0x00007f1b48dc56d5 in >>>>> pthread_cond_wait@@GLIBC_2.3.2 >>>>> () from >>>>> /lib64/libpthread.so.0 >>>>> No symbol table info available. >>>>> #1 0x00007f1b4941b050 in PR_WaitCondVar () from >>>>> /lib64/libnspr4.so >>>>> No symbol table info available. >>>>> #2 0x00007f1b4b70efee in >>>>> connection_wait_for_new_work () >>>>> No symbol table info available. >>>>> #3 0x00007f1b4b710719 in connection_threadmain () >>>>> No symbol table info available. >>>>> #4 0x00007f1b494207bb in _pt_root () from >>>>> /lib64/libnspr4.so >>>>> No symbol table info available. >>>>> #5 0x00007f1b48dc1dc5 in start_thread () from >>>>> /lib64/libpthread.so.0 >>>>> No symbol table info available. >>>>> #6 0x00007f1b48aef28d in clone () from >>>>> /lib64/libc.so.6 >>>>> No symbol table info available. >>>>> >>>>> Thread 21 (Thread 0x7f1b0e7fc700 (LWP 105181)): >>>>> #0 0x00007f1b48dc56d5 in >>>>> pthread_cond_wait@@GLIBC_2.3.2 >>>>> () from >>>>> /lib64/libpthread.so.0 >>>>> No symbol table info available. >>>>> #1 0x00007f1b4941b050 in PR_WaitCondVar () from >>>>> /lib64/libnspr4.so >>>>> No symbol table info available. >>>>> #2 0x00007f1b4b70efee in >>>>> connection_wait_for_new_work () >>>>> No symbol table info available. >>>>> #3 0x00007f1b4b710719 in connection_threadmain () >>>>> No symbol table info available. >>>>> #4 0x00007f1b494207bb in _pt_root () from >>>>> /lib64/libnspr4.so >>>>> No symbol table info available. >>>>> #5 0x00007f1b48dc1dc5 in start_thread () from >>>>> /lib64/libpthread.so.0 >>>>> No symbol table info available. >>>>> #6 0x00007f1b48aef28d in clone () from >>>>> /lib64/libc.so.6 >>>>> No symbol table info available. >>>>> >>>>> Thread 20 (Thread 0x7f1b0effd700 (LWP 105180)): >>>>> #0 0x00007f1b48dc56d5 in >>>>> pthread_cond_wait@@GLIBC_2.3.2 >>>>> () from >>>>> /lib64/libpthread.so.0 >>>>> No symbol table info available. >>>>> #1 0x00007f1b433212f3 in __db_hybrid_mutex_suspend >>>>> () from /lib64/libdb-5.3.so >>>>> No symbol table info available. >>>>> #2 0x00007f1b43320b37 in __db_tas_mutex_readlock >>>>> () from /lib64/libdb-5.3.so >>>>> No symbol table info available. >>>>> #3 0x00007f1b4343ad35 in __memp_fget () from >>>>> /lib64/libdb-5.3.so >>>>> No symbol table info available. >>>>> #4 0x00007f1b433e4333 in __dbc_iget () from >>>>> /lib64/libdb-5.3.so >>>>> No symbol table info available. >>>>> #5 0x00007f1b433f2b02 in __dbc_get_pp () from >>>>> /lib64/libdb-5.3.so >>>>> No symbol table info available. >>>>> #6 0x00007f1b3e8ce822 in idl_new_fetch () from >>>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>>> No symbol table info available. >>>>> #7 0x00007f1b3e8dcb26 in index_read_ext_allids () >>>>> from /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>>> No symbol table info available. >>>>> #8 0x00007f1b3e8c7284 in keys2idl () from >>>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>>> No symbol table info available. >>>>> #9 0x00007f1b3e8c79e3 in ava_candidates.isra.0 () >>>>> from /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>>> No symbol table info available. >>>>> #10 0x00007f1b3e8c7fd2 in filter_candidates_ext () >>>>> from /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>>> No symbol table info available. >>>>> #11 0x00007f1b3e8c9096 in list_candidates () from >>>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>>> No symbol table info available. >>>>> #12 0x00007f1b3e8c7f40 in filter_candidates_ext () >>>>> from /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>>> No symbol table info available. >>>>> #13 0x00007f1b3e8c9096 in list_candidates () from >>>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>>> No symbol table info available. >>>>> #14 0x00007f1b3e8c7f40 in filter_candidates_ext () >>>>> from /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>>> No symbol table info available. >>>>> #15 0x00007f1b3e8c9096 in list_candidates () from >>>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>>> No symbol table info available. >>>>> #16 0x00007f1b3e8c7f40 in filter_candidates_ext () >>>>> from /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>>> No symbol table info available. >>>>> #17 0x00007f1b3e90380c in subtree_candidates () >>>>> from /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>>> No symbol table info available. >>>>> #18 0x00007f1b3e904e7f in ldbm_back_search () from >>>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>>> No symbol table info available. >>>>> #19 0x00007f1b4b2321d6 in op_shared_search () from >>>>> /usr/lib64/dirsrv/libslapd.so.0 >>>>> No symbol table info available. >>>>> #20 0x00007f1b4b72333e in do_search () >>>>> No symbol table info available. >>>>> #21 0x00007f1b4b7113c3 in connection_threadmain () >>>>> No symbol table info available. >>>>> #22 0x00007f1b494207bb in _pt_root () from >>>>> /lib64/libnspr4.so >>>>> No symbol table info available. >>>>> #23 0x00007f1b48dc1dc5 in start_thread () from >>>>> /lib64/libpthread.so.0 >>>>> No symbol table info available. >>>>> #24 0x00007f1b48aef28d in clone () from >>>>> /lib64/libc.so.6 >>>>> No symbol table info available. >>>>> >>>>> Thread 19 (Thread 0x7f1b0f7fe700 (LWP 105179)): >>>>> #0 0x00007f1b48dc56d5 in >>>>> pthread_cond_wait@@GLIBC_2.3.2 >>>>> () from >>>>> /lib64/libpthread.so.0 >>>>> No symbol table info available. >>>>> #1 0x00007f1b433212f3 in __db_hybrid_mutex_suspend >>>>> () from /lib64/libdb-5.3.so >>>>> No symbol table info available. >>>>> #2 0x00007f1b43320640 in __db_tas_mutex_lock () >>>>> from /lib64/libdb-5.3.so >>>>> No symbol table info available. >>>>> #3 0x00007f1b433cad3a in __lock_get_internal () >>>>> from /lib64/libdb-5.3.so >>>>> No symbol table info available. >>>>> #4 0x00007f1b433cb820 in __lock_get () from >>>>> /lib64/libdb-5.3.so >>>>> No symbol table info available. >>>>> #5 0x00007f1b433f7142 in __db_lget () from >>>>> /lib64/libdb-5.3.so >>>>> No symbol table info available. >>>>> #6 0x00007f1b4333e605 in __bam_search () from >>>>> /lib64/libdb-5.3.so >>>>> No symbol table info available. >>>>> #7 0x00007f1b43329256 in __bamc_search () from >>>>> /lib64/libdb-5.3.so >>>>> No symbol table info available. >>>>> #8 0x00007f1b4332ad0f in __bamc_get () from >>>>> /lib64/libdb-5.3.so >>>>> No symbol table info available. >>>>> #9 0x00007f1b433e3ca6 in __dbc_iget () from >>>>> /lib64/libdb-5.3.so >>>>> No symbol table info available. >>>>> #10 0x00007f1b433f2b02 in __dbc_get_pp () from >>>>> /lib64/libdb-5.3.so >>>>> No symbol table info available. >>>>> #11 0x00007f1b3e8ce680 in idl_new_fetch () from >>>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>>> No symbol table info available. >>>>> #12 0x00007f1b3e8dcb26 in index_read_ext_allids () >>>>> from /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>>> No symbol table info available. >>>>> #13 0x00007f1b3e8c7284 in keys2idl () from >>>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>>> No symbol table info available. >>>>> #14 0x00007f1b3e8c79e3 in ava_candidates.isra.0 () >>>>> from /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>>> No symbol table info available. >>>>> #15 0x00007f1b3e8c7fd2 in filter_candidates_ext () >>>>> from /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>>> No symbol table info available. >>>>> #16 0x00007f1b3e8c9096 in list_candidates () from >>>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>>> No symbol table info available. >>>>> #17 0x00007f1b3e8c7f40 in filter_candidates_ext () >>>>> from /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>>> No symbol table info available. >>>>> #18 0x00007f1b3e8c9096 in list_candidates () from >>>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>>> No symbol table info available. >>>>> #19 0x00007f1b3e8c7f40 in filter_candidates_ext () >>>>> from /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>>> No symbol table info available. >>>>> #20 0x00007f1b3e8c9096 in list_candidates () from >>>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>>> No symbol table info available. >>>>> #21 0x00007f1b3e8c7f40 in filter_candidates_ext () >>>>> from /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>>> No symbol table info available. >>>>> #22 0x00007f1b3e90380c in subtree_candidates () >>>>> from /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>>> No symbol table info available. >>>>> #23 0x00007f1b3e904e7f in ldbm_back_search () from >>>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>>> No symbol table info available. >>>>> #24 0x00007f1b4b2321d6 in op_shared_search () from >>>>> /usr/lib64/dirsrv/libslapd.so.0 >>>>> No symbol table info available. >>>>> #25 0x00007f1b4b72333e in do_search () >>>>> No symbol table info available. >>>>> #26 0x00007f1b4b7113c3 in connection_threadmain () >>>>> No symbol table info available. >>>>> #27 0x00007f1b494207bb in _pt_root () from >>>>> /lib64/libnspr4.so >>>>> No symbol table info available. >>>>> #28 0x00007f1b48dc1dc5 in start_thread () from >>>>> /lib64/libpthread.so.0 >>>>> No symbol table info available. >>>>> #29 0x00007f1b48aef28d in clone () from >>>>> /lib64/libc.so.6 >>>>> No symbol table info available. >>>>> >>>>> Thread 18 (Thread 0x7f1b0ffff700 (LWP 105178)): >>>>> #0 0x00007f1b48dc56d5 in >>>>> pthread_cond_wait@@GLIBC_2.3.2 >>>>> () from >>>>> /lib64/libpthread.so.0 >>>>> No symbol table info available. >>>>> #1 0x00007f1b4941b050 in PR_WaitCondVar () from >>>>> /lib64/libnspr4.so >>>>> No symbol table info available. >>>>> #2 0x00007f1b4b70efee in >>>>> connection_wait_for_new_work () >>>>> No symbol table info available. >>>>> #3 0x00007f1b4b710719 in connection_threadmain () >>>>> No symbol table info available. >>>>> #4 0x00007f1b494207bb in _pt_root () from >>>>> /lib64/libnspr4.so >>>>> No symbol table info available. >>>>> #5 0x00007f1b48dc1dc5 in start_thread () from >>>>> /lib64/libpthread.so.0 >>>>> No symbol table info available. >>>>> #6 0x00007f1b48aef28d in clone () from >>>>> /lib64/libc.so.6 >>>>> No symbol table info available. >>>>> >>>>> Thread 17 (Thread 0x7f1b14d28700 (LWP 105177)): >>>>> #0 0x00007f1b48dc56d5 in >>>>> pthread_cond_wait@@GLIBC_2.3.2 >>>>> () from >>>>> /lib64/libpthread.so.0 >>>>> No symbol table info available. >>>>> #1 0x00007f1b4941b050 in PR_WaitCondVar () from >>>>> /lib64/libnspr4.so >>>>> No symbol table info available. >>>>> #2 0x00007f1b4b70efee in >>>>> connection_wait_for_new_work () >>>>> No symbol table info available. >>>>> #3 0x00007f1b4b710719 in connection_threadmain () >>>>> No symbol table info available. >>>>> #4 0x00007f1b494207bb in _pt_root () from >>>>> /lib64/libnspr4.so >>>>> No symbol table info available. >>>>> #5 0x00007f1b48dc1dc5 in start_thread () from >>>>> /lib64/libpthread.so.0 >>>>> No symbol table info available. >>>>> #6 0x00007f1b48aef28d in clone () from >>>>> /lib64/libc.so.6 >>>>> No symbol table info available. >>>>> >>>>> Thread 16 (Thread 0x7f1b15529700 (LWP 105176)): >>>>> #0 0x00007f1b48dc56d5 in >>>>> pthread_cond_wait@@GLIBC_2.3.2 >>>>> () from >>>>> /lib64/libpthread.so.0 >>>>> No symbol table info available. >>>>> #1 0x00007f1b4941b050 in PR_WaitCondVar () from >>>>> /lib64/libnspr4.so >>>>> No symbol table info available. >>>>> #2 0x00007f1b4b70efee in >>>>> connection_wait_for_new_work () >>>>> No symbol table info available. >>>>> #3 0x00007f1b4b710719 in connection_threadmain () >>>>> No symbol table info available. >>>>> #4 0x00007f1b494207bb in _pt_root () from >>>>> /lib64/libnspr4.so >>>>> No symbol table info available. >>>>> #5 0x00007f1b48dc1dc5 in start_thread () from >>>>> /lib64/libpthread.so.0 >>>>> No symbol table info available. >>>>> #6 0x00007f1b48aef28d in clone () from >>>>> /lib64/libc.so.6 >>>>> No symbol table info available. >>>>> >>>>> Thread 15 (Thread 0x7f1b15d2a700 (LWP 105175)): >>>>> #0 0x00007f1b48dc56d5 in >>>>> pthread_cond_wait@@GLIBC_2.3.2 >>>>> () from >>>>> /lib64/libpthread.so.0 >>>>> No symbol table info available. >>>>> #1 0x00007f1b4941b050 in PR_WaitCondVar () from >>>>> /lib64/libnspr4.so >>>>> No symbol table info available. >>>>> #2 0x00007f1b4b70efee in >>>>> connection_wait_for_new_work () >>>>> No symbol table info available. >>>>> #3 0x00007f1b4b710719 in connection_threadmain () >>>>> No symbol table info available. >>>>> #4 0x00007f1b494207bb in _pt_root () from >>>>> /lib64/libnspr4.so >>>>> No symbol table info available. >>>>> #5 0x00007f1b48dc1dc5 in start_thread () from >>>>> /lib64/libpthread.so.0 >>>>> No symbol table info available. >>>>> #6 0x00007f1b48aef28d in clone () from >>>>> /lib64/libc.so.6 >>>>> No symbol table info available. >>>>> >>>>> Thread 14 (Thread 0x7f1b1652b700 (LWP 105174)): >>>>> #0 0x00007f1b48dc56d5 in >>>>> pthread_cond_wait@@GLIBC_2.3.2 >>>>> () from >>>>> /lib64/libpthread.so.0 >>>>> No symbol table info available. >>>>> #1 0x00007f1b4941b050 in PR_WaitCondVar () from >>>>> /lib64/libnspr4.so >>>>> No symbol table info available. >>>>> #2 0x00007f1b4b70efee in >>>>> connection_wait_for_new_work () >>>>> No symbol table info available. >>>>> #3 0x00007f1b4b710719 in connection_threadmain () >>>>> No symbol table info available. >>>>> #4 0x00007f1b494207bb in _pt_root () from >>>>> /lib64/libnspr4.so >>>>> No symbol table info available. >>>>> #5 0x00007f1b48dc1dc5 in start_thread () from >>>>> /lib64/libpthread.so.0 >>>>> No symbol table info available. >>>>> #6 0x00007f1b48aef28d in clone () from >>>>> /lib64/libc.so.6 >>>>> No symbol table info available. >>>>> >>>>> Thread 13 (Thread 0x7f1b16d2c700 (LWP 105173)): >>>>> #0 0x00007f1b48dc56d5 in >>>>> pthread_cond_wait@@GLIBC_2.3.2 >>>>> () from >>>>> /lib64/libpthread.so.0 >>>>> No symbol table info available. >>>>> #1 0x00007f1b433212f3 in __db_hybrid_mutex_suspend >>>>> () from /lib64/libdb-5.3.so >>>>> No symbol table info available. >>>>> #2 0x00007f1b43320640 in __db_tas_mutex_lock () >>>>> from /lib64/libdb-5.3.so >>>>> No symbol table info available. >>>>> #3 0x00007f1b433cad3a in __lock_get_internal () >>>>> from /lib64/libdb-5.3.so >>>>> No symbol table info available. >>>>> #4 0x00007f1b433cb820 in __lock_get () from >>>>> /lib64/libdb-5.3.so >>>>> No symbol table info available. >>>>> #5 0x00007f1b433f7142 in __db_lget () from >>>>> /lib64/libdb-5.3.so >>>>> No symbol table info available. >>>>> #6 0x00007f1b4333e605 in __bam_search () from >>>>> /lib64/libdb-5.3.so >>>>> No symbol table info available. >>>>> #7 0x00007f1b43329256 in __bamc_search () from >>>>> /lib64/libdb-5.3.so >>>>> No symbol table info available. >>>>> #8 0x00007f1b4332ad0f in __bamc_get () from >>>>> /lib64/libdb-5.3.so >>>>> No symbol table info available. >>>>> #9 0x00007f1b433e3ca6 in __dbc_iget () from >>>>> /lib64/libdb-5.3.so >>>>> No symbol table info available. >>>>> #10 0x00007f1b433f2b02 in __dbc_get_pp () from >>>>> /lib64/libdb-5.3.so >>>>> No symbol table info available. >>>>> #11 0x00007f1b3e8ce680 in idl_new_fetch () from >>>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>>> No symbol table info available. >>>>> #12 0x00007f1b3e8dcb26 in index_read_ext_allids () >>>>> from /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>>> No symbol table info available. >>>>> #13 0x00007f1b3e8c7284 in keys2idl () from >>>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>>> No symbol table info available. >>>>> #14 0x00007f1b3e8c79e3 in ava_candidates.isra.0 () >>>>> from /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>>> No symbol table info available. >>>>> #15 0x00007f1b3e8c7fd2 in filter_candidates_ext () >>>>> from /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>>> No symbol table info available. >>>>> #16 0x00007f1b3e8c9096 in list_candidates () from >>>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>>> No symbol table info available. >>>>> #17 0x00007f1b3e8c7f40 in filter_candidates_ext () >>>>> from /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>>> No symbol table info available. >>>>> #18 0x00007f1b3e8c9096 in list_candidates () from >>>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>>> No symbol table info available. >>>>> #19 0x00007f1b3e8c7f40 in filter_candidates_ext () >>>>> from /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>>> No symbol table info available. >>>>> #20 0x00007f1b3e8c9096 in list_candidates () from >>>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>>> No symbol table info available. >>>>> #21 0x00007f1b3e8c7f40 in filter_candidates_ext () >>>>> from /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>>> No symbol table info available. >>>>> #22 0x00007f1b3e90380c in subtree_candidates () >>>>> from /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>>> No symbol table info available. >>>>> #23 0x00007f1b3e904e7f in ldbm_back_search () from >>>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>>> No symbol table info available. >>>>> #24 0x00007f1b4b2321d6 in op_shared_search () from >>>>> /usr/lib64/dirsrv/libslapd.so.0 >>>>> No symbol table info available. >>>>> #25 0x00007f1b4b72333e in do_search () >>>>> No symbol table info available. >>>>> #26 0x00007f1b4b7113c3 in connection_threadmain () >>>>> No symbol table info available. >>>>> #27 0x00007f1b494207bb in _pt_root () from >>>>> /lib64/libnspr4.so >>>>> No symbol table info available. >>>>> #28 0x00007f1b48dc1dc5 in start_thread () from >>>>> /lib64/libpthread.so.0 >>>>> No symbol table info available. >>>>> #29 0x00007f1b48aef28d in clone () from >>>>> /lib64/libc.so.6 >>>>> No symbol table info available. >>>>> >>>>> Thread 12 (Thread 0x7f1b1752d700 (LWP 105172)): >>>>> #0 0x00007f1b48dc56d5 in >>>>> pthread_cond_wait@@GLIBC_2.3.2 >>>>> () from >>>>> /lib64/libpthread.so.0 >>>>> No symbol table info available. >>>>> #1 0x00007f1b4941b050 in PR_WaitCondVar () from >>>>> /lib64/libnspr4.so >>>>> No symbol table info available. >>>>> #2 0x00007f1b4b70efee in >>>>> connection_wait_for_new_work () >>>>> No symbol table info available. >>>>> #3 0x00007f1b4b710719 in connection_threadmain () >>>>> No symbol table info available. >>>>> #4 0x00007f1b494207bb in _pt_root () from >>>>> /lib64/libnspr4.so >>>>> No symbol table info available. >>>>> #5 0x00007f1b48dc1dc5 in start_thread () from >>>>> /lib64/libpthread.so.0 >>>>> No symbol table info available. >>>>> #6 0x00007f1b48aef28d in clone () from >>>>> /lib64/libc.so.6 >>>>> No symbol table info available. >>>>> >>>>> Thread 11 (Thread 0x7f1b17d2e700 (LWP 105171)): >>>>> #0 0x00007f1b48dc56d5 in >>>>> pthread_cond_wait@@GLIBC_2.3.2 >>>>> () from >>>>> /lib64/libpthread.so.0 >>>>> No symbol table info available. >>>>> #1 0x00007f1b4941b050 in PR_WaitCondVar () from >>>>> /lib64/libnspr4.so >>>>> No symbol table info available. >>>>> #2 0x00007f1b4b70efee in >>>>> connection_wait_for_new_work () >>>>> No symbol table info available. >>>>> #3 0x00007f1b4b710719 in connection_threadmain () >>>>> No symbol table info available. >>>>> #4 0x00007f1b494207bb in _pt_root () from >>>>> /lib64/libnspr4.so >>>>> No symbol table info available. >>>>> #5 0x00007f1b48dc1dc5 in start_thread () from >>>>> /lib64/libpthread.so.0 >>>>> No symbol table info available. >>>>> #6 0x00007f1b48aef28d in clone () from >>>>> /lib64/libc.so.6 >>>>> No symbol table info available. >>>>> >>>>> Thread 10 (Thread 0x7f1b2509d700 (LWP 105168)): >>>>> #0 0x00007f1b48dc5a82 in >>>>> pthread_cond_timedwait@@GLIBC_2.3.2 >>>>> () >>>>> from /lib64/libpthread.so.0 >>>>> No symbol table info available. >>>>> #1 0x00007f1b4941ab07 in pt_TimedWait () from >>>>> /lib64/libnspr4.so >>>>> No symbol table info available. >>>>> #2 0x00007f1b4941afce in PR_WaitCondVar () from >>>>> /lib64/libnspr4.so >>>>> No symbol table info available. >>>>> #3 0x00007f1b4b2061e8 in eq_loop () from >>>>> /usr/lib64/dirsrv/libslapd.so.0 >>>>> No symbol table info available. >>>>> #4 0x00007f1b494207bb in _pt_root () from >>>>> /lib64/libnspr4.so >>>>> No symbol table info available. >>>>> #5 0x00007f1b48dc1dc5 in start_thread () from >>>>> /lib64/libpthread.so.0 >>>>> No symbol table info available. >>>>> #6 0x00007f1b48aef28d in clone () from >>>>> /lib64/libc.so.6 >>>>> No symbol table info available. >>>>> >>>>> Thread 9 (Thread 0x7f1b2589e700 (LWP 105167)): >>>>> #0 0x00007f1b48dc5a82 in >>>>> pthread_cond_timedwait@@GLIBC_2.3.2 >>>>> () >>>>> from /lib64/libpthread.so.0 >>>>> No symbol table info available. >>>>> #1 0x00007f1b4941ab07 in pt_TimedWait () from >>>>> /lib64/libnspr4.so >>>>> No symbol table info available. >>>>> #2 0x00007f1b4941afce in PR_WaitCondVar () from >>>>> /lib64/libnspr4.so >>>>> No symbol table info available. >>>>> #3 0x00007f1b4b719d03 in housecleaning () >>>>> No symbol table info available. >>>>> #4 0x00007f1b494207bb in _pt_root () from >>>>> /lib64/libnspr4.so >>>>> No symbol table info available. >>>>> #5 0x00007f1b48dc1dc5 in start_thread () from >>>>> /lib64/libpthread.so.0 >>>>> No symbol table info available. >>>>> #6 0x00007f1b48aef28d in clone () from >>>>> /lib64/libc.so.6 >>>>> No symbol table info available. >>>>> >>>>> Thread 8 (Thread 0x7f1b2609f700 (LWP 105166)): >>>>> #0 0x00007f1b48dc56d5 in >>>>> pthread_cond_wait@@GLIBC_2.3.2 >>>>> () from >>>>> /lib64/libpthread.so.0 >>>>> No symbol table info available. >>>>> #1 0x00007f1b4941b050 in PR_WaitCondVar () from >>>>> /lib64/libnspr4.so >>>>> No symbol table info available. >>>>> #2 0x00007f1b4b260198 in slapi_wait_condvar () >>>>> from /usr/lib64/dirsrv/libslapd.so.0 >>>>> No symbol table info available. >>>>> #3 0x00007f1b3cf5deed in >>>>> roles_cache_wait_on_change () from >>>>> /usr/lib64/dirsrv/plugins/libroles-plugin.so >>>>> No symbol table info available. >>>>> #4 0x00007f1b494207bb in _pt_root () from >>>>> /lib64/libnspr4.so >>>>> No symbol table info available. >>>>> #5 0x00007f1b48dc1dc5 in start_thread () from >>>>> /lib64/libpthread.so.0 >>>>> No symbol table info available. >>>>> #6 0x00007f1b48aef28d in clone () from >>>>> /lib64/libc.so.6 >>>>> No symbol table info available. >>>>> >>>>> Thread 7 (Thread 0x7f1b268a0700 (LWP 105165)): >>>>> #0 0x00007f1b48dc56d5 in >>>>> pthread_cond_wait@@GLIBC_2.3.2 >>>>> () from >>>>> /lib64/libpthread.so.0 >>>>> No symbol table info available. >>>>> #1 0x00007f1b4941b050 in PR_WaitCondVar () from >>>>> /lib64/libnspr4.so >>>>> No symbol table info available. >>>>> #2 0x00007f1b4b260198 in slapi_wait_condvar () >>>>> from /usr/lib64/dirsrv/libslapd.so.0 >>>>> No symbol table info available. >>>>> #3 0x00007f1b3cf5deed in >>>>> roles_cache_wait_on_change () from >>>>> /usr/lib64/dirsrv/plugins/libroles-plugin.so >>>>> No symbol table info available. >>>>> #4 0x00007f1b494207bb in _pt_root () from >>>>> /lib64/libnspr4.so >>>>> No symbol table info available. >>>>> #5 0x00007f1b48dc1dc5 in start_thread () from >>>>> /lib64/libpthread.so.0 >>>>> No symbol table info available. >>>>> #6 0x00007f1b48aef28d in clone () from >>>>> /lib64/libc.so.6 >>>>> No symbol table info available. >>>>> >>>>> Thread 6 (Thread 0x7f1b270a1700 (LWP 105164)): >>>>> #0 0x00007f1b48dc56d5 in >>>>> pthread_cond_wait@@GLIBC_2.3.2 >>>>> () from >>>>> /lib64/libpthread.so.0 >>>>> No symbol table info available. >>>>> #1 0x00007f1b4941b050 in PR_WaitCondVar () from >>>>> /lib64/libnspr4.so >>>>> No symbol table info available. >>>>> #2 0x00007f1b4b260198 in slapi_wait_condvar () >>>>> from /usr/lib64/dirsrv/libslapd.so.0 >>>>> No symbol table info available. >>>>> #3 0x00007f1b3cf5deed in >>>>> roles_cache_wait_on_change () from >>>>> /usr/lib64/dirsrv/plugins/libroles-plugin.so >>>>> No symbol table info available. >>>>> #4 0x00007f1b494207bb in _pt_root () from >>>>> /lib64/libnspr4.so >>>>> No symbol table info available. >>>>> #5 0x00007f1b48dc1dc5 in start_thread () from >>>>> /lib64/libpthread.so.0 >>>>> No symbol table info available. >>>>> #6 0x00007f1b48aef28d in clone () from >>>>> /lib64/libc.so.6 >>>>> No symbol table info available. >>>>> >>>>> Thread 5 (Thread 0x7f1b278a2700 (LWP 105163)): >>>>> #0 0x00007f1b48dc56d5 in >>>>> pthread_cond_wait@@GLIBC_2.3.2 >>>>> () from >>>>> /lib64/libpthread.so.0 >>>>> No symbol table info available. >>>>> #1 0x00007f1b4941b050 in PR_WaitCondVar () from >>>>> /lib64/libnspr4.so >>>>> No symbol table info available. >>>>> #2 0x00007f1b4b260198 in slapi_wait_condvar () >>>>> from /usr/lib64/dirsrv/libslapd.so.0 >>>>> No symbol table info available. >>>>> #3 0x00007f1b415f262e in cos_cache_wait_on_change >>>>> () from /usr/lib64/dirsrv/plugins/libcos-plugin.so >>>>> No symbol table info available. >>>>> #4 0x00007f1b494207bb in _pt_root () from >>>>> /lib64/libnspr4.so >>>>> No symbol table info available. >>>>> #5 0x00007f1b48dc1dc5 in start_thread () from >>>>> /lib64/libpthread.so.0 >>>>> No symbol table info available. >>>>> #6 0x00007f1b48aef28d in clone () from >>>>> /lib64/libc.so.6 >>>>> No symbol table info available. >>>>> >>>>> Thread 4 (Thread 0x7f1b282ac700 (LWP 105162)): >>>>> #0 0x00007f1b48ae69b3 in select () from >>>>> /lib64/libc.so.6 >>>>> No symbol table info available. >>>>> #1 0x00007f1b4b2710e9 in DS_Sleep () from >>>>> /usr/lib64/dirsrv/libslapd.so.0 >>>>> No symbol table info available. >>>>> #2 0x00007f1b3e8ba707 in perf_threadmain () from >>>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>>> No symbol table info available. >>>>> #3 0x00007f1b494207bb in _pt_root () from >>>>> /lib64/libnspr4.so >>>>> No symbol table info available. >>>>> #4 0x00007f1b48dc1dc5 in start_thread () from >>>>> /lib64/libpthread.so.0 >>>>> No symbol table info available. >>>>> #5 0x00007f1b48aef28d in clone () from >>>>> /lib64/libc.so.6 >>>>> No symbol table info available. >>>>> >>>>> Thread 3 (Thread 0x7f1b28aad700 (LWP 105161)): >>>>> #0 0x00007f1b48ae69b3 in select () from >>>>> /lib64/libc.so.6 >>>>> No symbol table info available. >>>>> #1 0x00007f1b4b2710e9 in DS_Sleep () from >>>>> /usr/lib64/dirsrv/libslapd.so.0 >>>>> No symbol table info available. >>>>> #2 0x00007f1b3e8bfb7f in trickle_threadmain () >>>>> from /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>>> No symbol table info available. >>>>> #3 0x00007f1b494207bb in _pt_root () from >>>>> /lib64/libnspr4.so >>>>> No symbol table info available. >>>>> #4 0x00007f1b48dc1dc5 in start_thread () from >>>>> /lib64/libpthread.so.0 >>>>> No symbol table info available. >>>>> #5 0x00007f1b48aef28d in clone () from >>>>> /lib64/libc.so.6 >>>>> No symbol table info available. >>>>> >>>>> Thread 2 (Thread 0x7f1b292ae700 (LWP 105160)): >>>>> #0 0x00007f1b48ae69b3 in select () from >>>>> /lib64/libc.so.6 >>>>> No symbol table info available. >>>>> #1 0x00007f1b4344dbad in __os_yield () from >>>>> /lib64/libdb-5.3.so >>>>> No symbol table info available. >>>>> #2 0x00007f1b434492b3 in __memp_sync_int () from >>>>> /lib64/libdb-5.3.so >>>>> No symbol table info available. >>>>> #3 0x00007f1b43459752 in __txn_checkpoint () from >>>>> /lib64/libdb-5.3.so >>>>> No symbol table info available. >>>>> #4 0x00007f1b43459b74 in __txn_checkpoint_pp () >>>>> from /lib64/libdb-5.3.so >>>>> No symbol table info available. >>>>> #5 0x00007f1b3e8c3a87 in checkpoint_threadmain () >>>>> from /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>>> No symbol table info available. >>>>> #6 0x00007f1b494207bb in _pt_root () from >>>>> /lib64/libnspr4.so >>>>> No symbol table info available. >>>>> #7 0x00007f1b48dc1dc5 in start_thread () from >>>>> /lib64/libpthread.so.0 >>>>> No symbol table info available. >>>>> #8 0x00007f1b48aef28d in clone () from >>>>> /lib64/libc.so.6 >>>>> No symbol table info available. >>>>> >>>>> Thread 1 (Thread 0x7f1b29aaf700 (LWP 105159)): >>>>> #0 0x00007f1b48ae69b3 in select () from >>>>> /lib64/libc.so.6 >>>>> No symbol table info available. >>>>> #1 0x00007f1b4b2710e9 in DS_Sleep () from >>>>> /usr/lib64/dirsrv/libslapd.so.0 >>>>> No symbol table info available. >>>>> #2 0x00007f1b3e8bf907 in deadlock_threadmain () >>>>> from /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>>> No symbol table info available. >>>>> #3 0x00007f1b494207bb in _pt_root () from >>>>> /lib64/libnspr4.so >>>>> No symbol table info available. >>>>> #4 0x00007f1b48dc1dc5 in start_thread () from >>>>> /lib64/libpthread.so.0 >>>>> No symbol table info available. >>>>> #5 0x00007f1b48aef28d in clone () from >>>>> /lib64/libc.so.6 >>>>> No symbol table info available. >>>>> >>>>> >>>>> >>>>> Thanks, >>>>> >>>>> Rakesh >>>>> >>>>> >>>>> On Mon, Sep 5, 2016 at 4:48 PM, thierry bordaz >>>>> > >>>>> wrote: >>>>> >>>>> >>>>> >>>>> On 09/05/2016 12:05 PM, Rakesh Rajasekharan wrote: >>>>>> Hi Thierry, >>>>>> >>>>>> >>>>>> I was getting the hang issue while running >>>>>> ipa-client-install simultaneously on few >>>>>> clients.. >>>>>> However, today, I am not able to replicate that. >>>>>> >>>>>> I could not get a gdb . But i will try >>>>>> getting that the next time I face this issue. >>>>>> >>>>>> The CPU does not stay high.. it just >>>>>> momentarily touches a high value and then >>>>>> drops down to around 2-7% >>>>>> >>>>>> One question I have is , is it ok to set it >>>>>> nsslapd-threadnumber to a very high value . >>>>>> I have around 4000 clients and with >>>>>> nsslapd-maxthreadsperconn set to 5..So, can I >>>>>> set nsslapd-threadnumber to around 25000. >>>>> >>>>> Hello, >>>>> >>>>> I know some users running in production with >>>>> several hunderds of threads (>600) and this >>>>> without problem. >>>>> >>>>> I do not recall having suggested to increase >>>>> that number and for what reason. >>>>> Usually 30 workers is a good enough value. It >>>>> can create bootleneck if for some reason each >>>>> operation is very long to satisfy and exhaust >>>>> the number of workers. You can monitor the >>>>> work queue: >>>>> >>>>> ldapsearch -D "cn=directory manager" -w >>>>> xxx -LLL -b "cn=monitor" -s base >>>>> opsinitiated opscompleted >>>>> >>>>> >>>>> If opscompleted-opsinitiated remains close to >>>>> threadnumber, then yes it would be valuable to >>>>> increase it. >>>>> >>>>> The computation #client * #async_op_per_client >>>>> sound an overkill. Even if all clients send at >>>>> the exact same time all their requests, it is >>>>> very likely that some common resource (db >>>>> page, log, allocator...) will serialize them. >>>>> If you monitor a need to increase the work, >>>>> you would for example set it to 50, then >>>>> monitor, then set it to 100, then monitor... >>>>> until you find a good enough value. >>>>> Note the increasing the #thread, increases the >>>>> memory footprint that will reduce the >>>>> efficiency of file system cache and can >>>>> increase the response time. >>>>> >>>>> >>>>> best regards >>>>> thierry >>>>> >>>>>> >>>>>> Thanks >>>>>> >>>>>> On Mon, Sep 5, 2016 at 1:03 PM, thierry >>>>>> bordaz >>>>> > wrote: >>>>>> >>>>>> >>>>>> Hi Rakesh, >>>>>> >>>>>> Were you able to get a pstack or full >>>>>> stack with gdb >>>>>> (http://www.port389.org/docs/389ds/FAQ/faq.html#debugging-crashes) >>>>>> when the server hangs ? >>>>>> >>>>>> If it happens with 500 threads as well as >>>>>> with 30, using 30 threads is a better >>>>>> choice to debug this issue. >>>>>> I will try to reproduce using 150 >>>>>> parallel 'ipa user-find p-testipa' commands >>>>>> >>>>>> Something I am unsure is if the CPU >>>>>> consumption stays high (you mentioned >>>>>> 340% CPU usage) as long as the hang >>>>>> happens or if after a suddent shot up to >>>>>> 340% (that marks the beginning of the >>>>>> hang) it drops and stay hanging ? >>>>>> >>>>>> thanks >>>>>> thierry >>>>>> >>>>>> On 09/04/2016 08:40 PM, Rakesh >>>>>> Rajasekharan wrote: >>>>>>> starce on the slapd process actually had >>>>>>> this in the output.. >>>>>>> FUTEX_WAIT_PRIVATE >>>>>>> >>>>>>> and checking for the number of threads >>>>>>> slapd had.. there were 5015 threads >>>>>>> >>>>>>> ps -efL|grep slapd|wc -l >>>>>>> 5015 >>>>>>> >>>>>>> strace on most of the threads gave this >>>>>>> output >>>>>>> >>>>>>> strace -p 67411 >>>>>>> Process 67411 attached >>>>>>> futex(0x7f3f0226b41c, >>>>>>> FUTEX_WAIT_PRIVATE, 1, NULL) = -1 EAGAIN >>>>>>> (Resource temporarily unavailable) >>>>>>> futex(0x7f3f0226b41c, >>>>>>> FUTEX_WAIT_PRIVATE, 2, NULL^CProcess >>>>>>> 67411 detached >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> On Sun, Sep 4, 2016 at 5:34 PM, Rakesh >>>>>>> Rajasekharan >>>>>>> >>>>>> > >>>>>>> wrote: >>>>>>> >>>>>>> I have again got the issue of IPA >>>>>>> hanging.. The issue came up when i >>>>>>> tried to run ipa-client-isntall on >>>>>>> 142 clients simultaneously >>>>>>> >>>>>>> >>>>>>> None of the IPA commands are >>>>>>> responding, and I see this error >>>>>>> >>>>>>> ipa user-find p-testipa >>>>>>> ipa: ERROR: Insufficient access: >>>>>>> SASL(-1): generic failure: GSSAPI >>>>>>> Error: Unspecified GSS failure. >>>>>>> Minor code may provide more >>>>>>> information (KDC returned error >>>>>>> string: PROCESS_TGS) >>>>>>> >>>>>>> KRB5_TRACE=/dev/stdout kinit admin >>>>>>> [41178] 1472984115.233214: Getting >>>>>>> initial credentials for >>>>>>> admin at XYZ.COM >>>>>>> [41178] 1472984115.235257: Sending >>>>>>> request (167 bytes) to XYZ.COM >>>>>>> >>>>>>> [41178] 1472984115.235419: >>>>>>> Initiating TCP connection to stream >>>>>>> 10.1.3.36:88 >>>>>>> [41178] 1472984115.235685: Sending >>>>>>> TCP request to stream 10.1.3.36:88 >>>>>>> >>>>>>> [41178] 1472984120.238914: Received >>>>>>> answer (174 bytes) from stream >>>>>>> 10.1.3.36:88 >>>>>>> [41178] 1472984120.238925: >>>>>>> Terminating TCP connection to stream >>>>>>> 10.1.3.36:88 >>>>>>> [41178] 1472984120.238993: Response >>>>>>> was from master KDC >>>>>>> [41 >>>>>>> >>>>>>> >>>>>>> Running an ldapsearch to see the >>>>>>> db.. does not give any results and >>>>>>> just hangs there >>>>>>> >>>>>>> ldapsearch -x -D 'cn=Directory >>>>>>> Manager' -W -s one -b >>>>>>> 'cn=kerberos,dc=xyz,dc=com' >>>>>>> Enter LDAP Password: >>>>>>> >>>>>>> even an ldapsearch -x does not respond >>>>>>> At this point, am sure that slapd is >>>>>>> the one causing issues >>>>>>> >>>>>>> Running an strace against the hung >>>>>>> slapd itself seems to get stuck does >>>>>>> not proceed after saying "attaching >>>>>>> to process" >>>>>>> >>>>>>> From some others posts I read >>>>>>> Thierry suggesting to increase the >>>>>>> nsslapd-threadnumber value >>>>>>> >>>>>>> It was set to 30, I think that might >>>>>>> be too low. >>>>>>> >>> ... >>> >>> [Message clipped] >>> >>> >>> >> > > -- > Red Hat GmbH,http://www.de.redhat.com/, Registered seat: Grasbrunn, > Commercial register: Amtsgericht Muenchen, HRB 153243, > Managing Directors: Charles Cachera, Michael Cunningham, Michael O'Neill, Eric Shander > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From abokovoy at redhat.com Thu Sep 15 18:06:06 2016 From: abokovoy at redhat.com (Alexander Bokovoy) Date: Thu, 15 Sep 2016 21:06:06 +0300 Subject: [Freeipa-users] Samba Server setup In-Reply-To: <73C11E05-045F-4324-9733-44C370584232@bsd.uchicago.edu> References: <73C11E05-045F-4324-9733-44C370584232@bsd.uchicago.edu> Message-ID: <20160915180606.v2z6dkdw3gltem34@redhat.com> On Thu, 15 Sep 2016, Brook, Andy [CRI] wrote: >All, > I?m working on setting up Samba to serve files from a server attached > to our IPA domain. I followed the directions in > https://www.freeipa.org/page/Howto/Integrating_a_Samba_File_Server_With_IPA. > Everything seems to work and I can access the files from another RHEL > server attached to the same domain using a Kerberos ticket from a > user from the trusted AD domain. However, I can?t access this share > from a windows client that is also attached to the trusted AD domain. > >My smb.conf is as follows: >[global] > workgroup = IPA > realm = IPA.DOMAIN > kerberos method = dedicated keytab > dedicated keytab file = FILE:/etc/samba/samba.keytab > log file = /var/log/samba/log.%m > log level = 3 > security = ads > load printers = no > disable spoolss = yes > map to guest = Never > restrict anonymous = 2 > >[spacetest] > path = /var/www > writable = yes > browsable = yes > >I put the keytab in place from the cifs service from the IPA server. > >I feel like I?m missing something small, but I can?t seem to find it. >Logs from samba are here: http://pastebin.com/aMDXfR78 These logs show that your Windows client did not use Kerberos but tried to authenticate with password using NTLMSSP. This is not supported yet, as written on the page you used for the setup guidance. You need to find out why Windows client didn't use Kerberos. Is your trust to AD really working? -- / Alexander Bokovoy From peljasz at yahoo.co.uk Thu Sep 15 19:08:28 2016 From: peljasz at yahoo.co.uk (lejeczek) Date: Thu, 15 Sep 2016 20:08:28 +0100 Subject: [Freeipa-users] how to revert ipa-adtrust-install... Message-ID: <1ec82d0c-1350-feb3-4da5-a022eb7ac5e5@yahoo.co.uk> ... in a sense so IPA would keep away from local smb services? many thanks L. From rakesh.rajasekharan at gmail.com Thu Sep 15 19:14:49 2016 From: rakesh.rajasekharan at gmail.com (Rakesh Rajasekharan) Date: Fri, 16 Sep 2016 00:44:49 +0530 Subject: [Freeipa-users] Freeipa 4.2.0 hangs intermittently In-Reply-To: <57DACE76.5070803@redhat.com> References: <57C44AC6.1030402@redhat.com> <57CD1FDC.2010109@redhat.com> <57CD5488.60404@redhat.com> <57CD659B.6050707@redhat.com> <57CD68BD.9030802@redhat.com> <57CE8CDB.6000809@redhat.com> <57CE9DE5.6020301@redhat.com> <57DACE76.5070803@redhat.com> Message-ID: Hi Thierry, I had been doing pretty much the same to reproduce the hang.. After changing the checkpoint interval.. the hangs had become manageable.. However, this time when the hang occurred.. I did not add any clients and also the behavior was bit different. This time my ns-slapd did not hang.. ldapsearch was working fine.. but, the ipa commands were taking a long time to respond and most users were not able to login to any of the clients.. restarting ipa did not help either. I was seeing around 14-15 connections in SYN_RECV state continously and /var.log/messages has entries like "TCP: request_sock_TCP: Possible SYN flooding on port 88. Sending cookies. Check SNMP counter" I also see in the /var/log/krb5kdc.log this error ipa-master-xyz.com krb5kdc[15308](info): TGS_REQ (4 etypes {18 17 16 23}) 10.1.12.155: PROCESS_TGS: authtime 0, host/10.1.12.155 at XYZ.COM for ldap/ ipa-master-xyz.com at XYZ.COM, Clock skew too great there are arounf 7.5k messages like that.. which I guess is too high grep "krbtgt/XYZ.COM at XYZ.COM, Clock skew too great" /var/log/krb5kdc.log | wc -l 7570 Would this give rise to a hang or make IPA nonresponsive.. The kinit admin command itself was taking an unusually long time to respond.. KRB5_TRACE=/dev/stderr kinit admin [11298] 1473702491.60880: Getting initial credentials for admin at XYZ.COM [11298] 1473702491.62981: Sending request (167 bytes) to XYZ.COM [11298] 1473702491.63119: Initiating TCP connection to stream 10.1.3.35:88 [11298] 1473702491.63359: Sending TCP request to stream 10.1.3.35:88 [11298] 1473702493.797835: Received answer (341 bytes) from stream 10.1.3.35:88 [11298] 1473702493.797848: Terminating TCP connection to stream 10.1.3.35:88 [11298] 1473702493.797911: Response was from master KDC [11298] 1473702493.797956: Received error from KDC: -1765328359/Additional pre-authentication required [11298] 1473702493.797993: Processing preauth types: 136, 19, 2, 133 [11298] 1473702493.798005: Selected etype info: etype aes256-cts, salt "V at Cbu147E#1;R0WD", params "" [11298] 1473702493.798009: Received cookie: MIT Password for admin at XYZ.COM: [11298] 1473702498.190064: AS key obtained for encrypted timestamp: aes256-cts/2C9D [11298] 1473702498.190109: Encrypted timestamp (for 1473702498.184527): plain 301AA011180F32303136303931323137343831385AA105020302D0CF, encrypted 25FC8D37EFB6B7837C8D5C6649DFB9972010D40EE29D1222FBA45CAA98428E42C7FCC9B7FE881A04BD3390A6A9EDE9D2D93729FDF3E47B6D [11298] 1473702498.190129: Preauth module encrypted_timestamp (2) (real) returned: 0/Success [11298] 1473702498.190133: Produced preauth for next request: 133, 2 [11298] 1473702498.190148: Sending request (261 bytes) to XYZ.COM [11298] 1473702498.190246: Initiating TCP connection to stream 10.1.3.35:88 [11298] 1473702499.191933: Sending initial UDP request to dgram 10.1.3.35:88 [11298] 1473702502.195157: Sending retry UDP request to dgram 10.1.3.35:88 [11298] 1473702507.200405: Sending retry UDP request to dgram 10.1.3.35:88 [11298] 1473702513.226371: Sending TCP request to stream 10.1.3.35:88 [11298] 1473702515.797243: Received answer (730 bytes) from stream 10.1.3.35:88 [11298] 1473702515.797271: Terminating TCP connection to stream 10.1.3.35:88 [11298] 1473702515.797326: Response was from master KDC [11298] 1473702515.797353: Processing preauth types: 19 [11298] 1473702515.797360: Selected etype info: etype aes256-cts, salt "V at Cbu147E#1;R0WD", params "" [11298] 1473702515.797394: Produced preauth for next request: (empty) [11298] 1473702515.797401: AS key determined by preauth: aes256-cts/2C9D [11298] 1473702515.797445: Decrypted AS reply; session key is: aes256-cts/702E [11298] 1473702515.797460: FAST negotiation: available [11298] 1473702515.797478: Initializing KEYRING:persistent:0:0 with default princ admin at XYZ.COM [11298] 1473702515.797534: Storing admin at XYZ.COM -> krbtgt/XYZ.COM at XYZ.COM in KEYRING:persistent:0:0 [11298] 1473702515.797572: Storing config in KEYRING:persistent:0:0 for krbtgt/XYZ.COM at XYZ.COM: fast_avail: yes [11298] 1473702515.797585: Storing admin at XYZ.COM -> krb5_ccache_conf_data/fast_avail/krbtgt\/XYZ.COM\@XYZ.COM at X-CACHECONF: in KEYRING:persistent:0:0 [11298] 1473702515.797631: Storing config in KEYRING:persistent:0:0 for krbtgt/XYZ.COM at XYZ.COM: pa_type: 2 [11298] 1473702515.797647: Storing admin at XYZ.COM -> krb5_ccache_conf_data/pa_type/krbtgt\/XYZ.COM\@XYZ.COM at X-CACHECONF: in KEYRING:persistent:0:0 Could it be an version specific issue..as I am running 4.2.0 verison.. and yours is 4.3.2 If so, I will try to upgrade my IPA version Thanks Rakesh On Thu, Sep 15, 2016 at 10:08 PM, thierry bordaz wrote: > Hello Rakesh, > > Sorry to be back so late. I finally manage to try to reproduce locally the > hang of DS... and I fail ;-) > > I am running 4 VM, with freeipa 4.3.2. > One is running a server (where I provisioned some users), with 200Mb > cache. The server receives no direct load (SRCH or update). > The tree others are running in a loop ipa-client-install and uninstall. So > far it has run for several hours without hang. > > Going through the mail thread I have not found others details. > Am I missing something to reproduce ? > > thanks in advance for your help > thierry > > On 09/07/2016 01:46 PM, Rakesh Rajasekharan wrote: > > I changed the nsslapd-db-checkpoint interval first to 80 and increased > till 160.. however the hang has not gone away yet > > > I migrated my data from a openldap, can that cause issues like this.. If > so, i could try creating users from scratch directly on IPA rather than > migrating and test it > > But before that , have taken a fresh pstack just incase if it helps to > nail down the issue > > Thread 44 (Thread 0x7f53ed52f700 (LWP 128165)): > #0 0x00007f54047649b3 in select () from /lib64/libc.so.6 > #1 0x00007f5406eef0e9 in DS_Sleep () from /usr/lib64/dirsrv/libslapd.so.0 > #2 0x00007f53fa53d907 in deadlock_threadmain () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #3 0x00007f540509e7bb in _pt_root () from /lib64/libnspr4.so > #4 0x00007f5404a3fdc5 in start_thread () from /lib64/libpthread.so.0 > #5 0x00007f540476d28d in clone () from /lib64/libc.so.6 > Thread 43 (Thread 0x7f53ecd2e700 (LWP 128166)): > #0 0x00007f54047649b3 in select () from /lib64/libc.so.6 > #1 0x00007f53ff0cbbad in __os_yield () from /lib64/libdb-5.3.so > #2 0x00007f53ff0c72b3 in __memp_sync_int () from /lib64/libdb-5.3.so > #3 0x00007f53ff0d7752 in __txn_checkpoint () from /lib64/libdb-5.3.so > #4 0x00007f53ff0d7b74 in __txn_checkpoint_pp () from /lib64/libdb-5.3.so > #5 0x00007f53fa541a87 in checkpoint_threadmain () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #6 0x00007f540509e7bb in _pt_root () from /lib64/libnspr4.so > #7 0x00007f5404a3fdc5 in start_thread () from /lib64/libpthread.so.0 > #8 0x00007f540476d28d in clone () from /lib64/libc.so.6 > Thread 42 (Thread 0x7f53ec52d700 (LWP 128167)): > #0 0x00007f54047649b3 in select () from /lib64/libc.so.6 > #1 0x00007f5406eef0e9 in DS_Sleep () from /usr/lib64/dirsrv/libslapd.so.0 > #2 0x00007f53fa53db7f in trickle_threadmain () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #3 0x00007f540509e7bb in _pt_root () from /lib64/libnspr4.so > #4 0x00007f5404a3fdc5 in start_thread () from /lib64/libpthread.so.0 > #5 0x00007f540476d28d in clone () from /lib64/libc.so.6 > Thread 41 (Thread 0x7f53ebd2c700 (LWP 128168)): > #0 0x00007f54047649b3 in select () from /lib64/libc.so.6 > #1 0x00007f5406eef0e9 in DS_Sleep () from /usr/lib64/dirsrv/libslapd.so.0 > #2 0x00007f53fa538707 in perf_threadmain () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #3 0x00007f540509e7bb in _pt_root () from /lib64/libnspr4.so > #4 0x00007f5404a3fdc5 in start_thread () from /lib64/libpthread.so.0 > #5 0x00007f540476d28d in clone () from /lib64/libc.so.6 > Thread 40 (Thread 0x7f53eb322700 (LWP 128204)): > #0 0x00007f5404a436d5 in pthread_cond_wait@@GLIBC_2.3.2 () from > /lib64/libpthread.so.0 > #1 0x00007f5405099050 in PR_WaitCondVar () from /lib64/libnspr4.so > #2 0x00007f5406ede198 in slapi_wait_condvar () from > /usr/lib64/dirsrv/libslapd.so.0 > #3 0x00007f53fd27062e in cos_cache_wait_on_change () from > /usr/lib64/dirsrv/plugins/libcos-plugin.so > #4 0x00007f540509e7bb in _pt_root () from /lib64/libnspr4.so > #5 0x00007f5404a3fdc5 in start_thread () from /lib64/libpthread.so.0 > #6 0x00007f540476d28d in clone () from /lib64/libc.so.6 > Thread 39 (Thread 0x7f53eab21700 (LWP 128206)): > #0 0x00007f5404a436d5 in pthread_cond_wait@@GLIBC_2.3.2 () from > /lib64/libpthread.so.0 > #1 0x00007f5405099050 in PR_WaitCondVar () from /lib64/libnspr4.so > #2 0x00007f5406ede198 in slapi_wait_condvar () from > /usr/lib64/dirsrv/libslapd.so.0 > #3 0x00007f53f8bdbeed in roles_cache_wait_on_change () from > /usr/lib64/dirsrv/plugins/libroles-plugin.so > #4 0x00007f540509e7bb in _pt_root () from /lib64/libnspr4.so > #5 0x00007f5404a3fdc5 in start_thread () from /lib64/libpthread.so.0 > #6 0x00007f540476d28d in clone () from /lib64/libc.so.6 > Thread 38 (Thread 0x7f53ea320700 (LWP 128207)): > #0 0x00007f5404a436d5 in pthread_cond_wait@@GLIBC_2.3.2 () from > /lib64/libpthread.so.0 > #1 0x00007f5405099050 in PR_WaitCondVar () from /lib64/libnspr4.so > #2 0x00007f5406ede198 in slapi_wait_condvar () from > /usr/lib64/dirsrv/libslapd.so.0 > #3 0x00007f53f8bdbeed in roles_cache_wait_on_change () from > /usr/lib64/dirsrv/plugins/libroles-plugin.so > #4 0x00007f540509e7bb in _pt_root () from /lib64/libnspr4.so > #5 0x00007f5404a3fdc5 in start_thread () from /lib64/libpthread.so.0 > #6 0x00007f540476d28d in clone () from /lib64/libc.so.6 > Thread 37 (Thread 0x7f53e9b1f700 (LWP 128208)): > #0 0x00007f5404a436d5 in pthread_cond_wait@@GLIBC_2.3.2 () from > /lib64/libpthread.so.0 > #1 0x00007f5405099050 in PR_WaitCondVar () from /lib64/libnspr4.so > #2 0x00007f5406ede198 in slapi_wait_condvar () from > /usr/lib64/dirsrv/libslapd.so.0 > #3 0x00007f53f8bdbeed in roles_cache_wait_on_change () from > /usr/lib64/dirsrv/plugins/libroles-plugin.so > #4 0x00007f540509e7bb in _pt_root () from /lib64/libnspr4.so > #5 0x00007f5404a3fdc5 in start_thread () from /lib64/libpthread.so.0 > #6 0x00007f540476d28d in clone () from /lib64/libc.so.6 > Thread 36 (Thread 0x7f53e931e700 (LWP 128209)): > #0 0x00007f5404a43a82 in pthread_cond_timedwait@@GLIBC_2.3.2 () from > /lib64/libpthread.so.0 > #1 0x00007f5405098b07 in pt_TimedWait () from /lib64/libnspr4.so > #2 0x00007f5405098fce in PR_WaitCondVar () from /lib64/libnspr4.so > #3 0x00007f5407397d03 in housecleaning () > #4 0x00007f540509e7bb in _pt_root () from /lib64/libnspr4.so > #5 0x00007f5404a3fdc5 in start_thread () from /lib64/libpthread.so.0 > #6 0x00007f540476d28d in clone () from /lib64/libc.so.6 > Thread 35 (Thread 0x7f53e8b1d700 (LWP 128210)): > #0 0x00007f5404a43a82 in pthread_cond_timedwait@@GLIBC_2.3.2 () from > /lib64/libpthread.so.0 > #1 0x00007f5405098b07 in pt_TimedWait () from /lib64/libnspr4.so > #2 0x00007f5405098fce in PR_WaitCondVar () from /lib64/libnspr4.so > #3 0x00007f5406e841e8 in eq_loop () from /usr/lib64/dirsrv/libslapd.so.0 > #4 0x00007f540509e7bb in _pt_root () from /lib64/libnspr4.so > #5 0x00007f5404a3fdc5 in start_thread () from /lib64/libpthread.so.0 > #6 0x00007f540476d28d in clone () from /lib64/libc.so.6 > Thread 34 (Thread 0x7f53db7fe700 (LWP 128212)): > #0 0x00007f5404a436d5 in pthread_cond_wait@@GLIBC_2.3.2 () from > /lib64/libpthread.so.0 > #1 0x00007f53fef9f2f3 in __db_hybrid_mutex_suspend () from /lib64/ > libdb-5.3.so > #2 0x00007f53fef9e640 in __db_tas_mutex_lock () from /lib64/libdb-5.3.so > #3 0x00007f53ff048d3a in __lock_get_internal () from /lib64/libdb-5.3.so > #4 0x00007f53ff049820 in __lock_get () from /lib64/libdb-5.3.so > #5 0x00007f53ff075142 in __db_lget () from /lib64/libdb-5.3.so > #6 0x00007f53fefbc605 in __bam_search () from /lib64/libdb-5.3.so > #7 0x00007f53fefa7256 in __bamc_search () from /lib64/libdb-5.3.so > #8 0x00007f53fefa8d0f in __bamc_get () from /lib64/libdb-5.3.so > #9 0x00007f53ff061ca6 in __dbc_iget () from /lib64/libdb-5.3.so > #10 0x00007f53ff070b02 in __dbc_get_pp () from /lib64/libdb-5.3.so > #11 0x00007f53fa54c680 in idl_new_fetch () from /usr/lib64/dirsrv/plugins/ > libback-ldbm.so > #12 0x00007f53fa55ab26 in index_read_ext_allids () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #13 0x00007f53fa545284 in keys2idl () from /usr/lib64/dirsrv/plugins/ > libback-ldbm.so > #14 0x00007f53fa5459e3 in ava_candidates.isra.0 () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #15 0x00007f53fa545fd2 in filter_candidates_ext () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #16 0x00007f53fa547096 in list_candidates () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #17 0x00007f53fa545f40 in filter_candidates_ext () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #18 0x00007f53fa58180c in subtree_candidates () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #19 0x00007f53fa582e7f in ldbm_back_search () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #20 0x00007f5406eb01d6 in op_shared_search () from > /usr/lib64/dirsrv/libslapd.so.0 > #21 0x00007f54073a133e in do_search () > #22 0x00007f540738f567 in connection_threadmain () > #23 0x00007f540509e7bb in _pt_root () from /lib64/libnspr4.so > #24 0x00007f5404a3fdc5 in start_thread () from /lib64/libpthread.so.0 > #25 0x00007f540476d28d in clone () from /lib64/libc.so.6 > Thread 33 (Thread 0x7f53daffd700 (LWP 128213)): > #0 0x00007f5404a436d5 in pthread_cond_wait@@GLIBC_2.3.2 () from > /lib64/libpthread.so.0 > #1 0x00007f53fef9f2f3 in __db_hybrid_mutex_suspend () from /lib64/ > libdb-5.3.so > #2 0x00007f53fef9e640 in __db_tas_mutex_lock () from /lib64/libdb-5.3.so > #3 0x00007f53ff048d3a in __lock_get_internal () from /lib64/libdb-5.3.so > #4 0x00007f53ff049820 in __lock_get () from /lib64/libdb-5.3.so > #5 0x00007f53ff075142 in __db_lget () from /lib64/libdb-5.3.so > #6 0x00007f53fefbc605 in __bam_search () from /lib64/libdb-5.3.so > #7 0x00007f53fefa7256 in __bamc_search () from /lib64/libdb-5.3.so > #8 0x00007f53fefa8d0f in __bamc_get () from /lib64/libdb-5.3.so > #9 0x00007f53ff061ca6 in __dbc_iget () from /lib64/libdb-5.3.so > #10 0x00007f53ff070b02 in __dbc_get_pp () from /lib64/libdb-5.3.so > #11 0x00007f53fa54c680 in idl_new_fetch () from /usr/lib64/dirsrv/plugins/ > libback-ldbm.so > #12 0x00007f53fa55ab26 in index_read_ext_allids () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #13 0x00007f53fa545284 in keys2idl () from /usr/lib64/dirsrv/plugins/ > libback-ldbm.so > #14 0x00007f53fa5459e3 in ava_candidates.isra.0 () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #15 0x00007f53fa545fd2 in filter_candidates_ext () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #16 0x00007f53fa547096 in list_candidates () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #17 0x00007f53fa545f40 in filter_candidates_ext () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #18 0x00007f53fa58180c in subtree_candidates () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #19 0x00007f53fa582e7f in ldbm_back_search () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #20 0x00007f5406eb01d6 in op_shared_search () from > /usr/lib64/dirsrv/libslapd.so.0 > #21 0x00007f54073a133e in do_search () > #22 0x00007f540738f567 in connection_threadmain () > #23 0x00007f540509e7bb in _pt_root () from /lib64/libnspr4.so > #24 0x00007f5404a3fdc5 in start_thread () from /lib64/libpthread.so.0 > #25 0x00007f540476d28d in clone () from /lib64/libc.so.6 > Thread 32 (Thread 0x7f53da7fc700 (LWP 128214)): > #0 0x00007f5404a436d5 in pthread_cond_wait@@GLIBC_2.3.2 () from > /lib64/libpthread.so.0 > #1 0x00007f53fef9f2f3 in __db_hybrid_mutex_suspend () from /lib64/ > libdb-5.3.so > #2 0x00007f53fef9e640 in __db_tas_mutex_lock () from /lib64/libdb-5.3.so > #3 0x00007f53ff048d3a in __lock_get_internal () from /lib64/libdb-5.3.so > #4 0x00007f53ff049820 in __lock_get () from /lib64/libdb-5.3.so > #5 0x00007f53ff075142 in __db_lget () from /lib64/libdb-5.3.so > #6 0x00007f53fefbc605 in __bam_search () from /lib64/libdb-5.3.so > #7 0x00007f53fefa7256 in __bamc_search () from /lib64/libdb-5.3.so > #8 0x00007f53fefa8d0f in __bamc_get () from /lib64/libdb-5.3.so > #9 0x00007f53ff061ca6 in __dbc_iget () from /lib64/libdb-5.3.so > #10 0x00007f53ff070b02 in __dbc_get_pp () from /lib64/libdb-5.3.so > #11 0x00007f53fa54c680 in idl_new_fetch () from /usr/lib64/dirsrv/plugins/ > libback-ldbm.so > #12 0x00007f53fa55ab26 in index_read_ext_allids () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #13 0x00007f53fa545284 in keys2idl () from /usr/lib64/dirsrv/plugins/ > libback-ldbm.so > #14 0x00007f53fa5459e3 in ava_candidates.isra.0 () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #15 0x00007f53fa545fd2 in filter_candidates_ext () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #16 0x00007f53fa547096 in list_candidates () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #17 0x00007f53fa545f40 in filter_candidates_ext () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #18 0x00007f53fa547096 in list_candidates () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #19 0x00007f53fa545f40 in filter_candidates_ext () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #20 0x00007f53fa547096 in list_candidates () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #21 0x00007f53fa545f40 in filter_candidates_ext () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #22 0x00007f53fa58180c in subtree_candidates () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #23 0x00007f53fa582e7f in ldbm_back_search () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #24 0x00007f5406eb01d6 in op_shared_search () from > /usr/lib64/dirsrv/libslapd.so.0 > #25 0x00007f54073a133e in do_search () > #26 0x00007f540738f3c3 in connection_threadmain () > #27 0x00007f540509e7bb in _pt_root () from /lib64/libnspr4.so > #28 0x00007f5404a3fdc5 in start_thread () from /lib64/libpthread.so.0 > #29 0x00007f540476d28d in clone () from /lib64/libc.so.6 > Thread 31 (Thread 0x7f53d9ffb700 (LWP 128215)): > #0 0x00007f5404a436d5 in pthread_cond_wait@@GLIBC_2.3.2 () from > /lib64/libpthread.so.0 > #1 0x00007f53fef9f2f3 in __db_hybrid_mutex_suspend () from /lib64/ > libdb-5.3.so > #2 0x00007f53fef9e640 in __db_tas_mutex_lock () from /lib64/libdb-5.3.so > #3 0x00007f53ff048d3a in __lock_get_internal () from /lib64/libdb-5.3.so > #4 0x00007f53ff049820 in __lock_get () from /lib64/libdb-5.3.so > #5 0x00007f53ff075142 in __db_lget () from /lib64/libdb-5.3.so > #6 0x00007f53fefbc605 in __bam_search () from /lib64/libdb-5.3.so > #7 0x00007f53fefa7256 in __bamc_search () from /lib64/libdb-5.3.so > #8 0x00007f53fefa8d0f in __bamc_get () from /lib64/libdb-5.3.so > #9 0x00007f53ff061ca6 in __dbc_iget () from /lib64/libdb-5.3.so > #10 0x00007f53ff070b02 in __dbc_get_pp () from /lib64/libdb-5.3.so > #11 0x00007f53fa54c680 in idl_new_fetch () from /usr/lib64/dirsrv/plugins/ > libback-ldbm.so > #12 0x00007f53fa55ab26 in index_read_ext_allids () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #13 0x00007f53fa545284 in keys2idl () from /usr/lib64/dirsrv/plugins/ > libback-ldbm.so > #14 0x00007f53fa5459e3 in ava_candidates.isra.0 () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #15 0x00007f53fa545fd2 in filter_candidates_ext () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #16 0x00007f53fa547096 in list_candidates () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #17 0x00007f53fa545f40 in filter_candidates_ext () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #18 0x00007f53fa58180c in subtree_candidates () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #19 0x00007f53fa582e7f in ldbm_back_search () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #20 0x00007f5406eb01d6 in op_shared_search () from > /usr/lib64/dirsrv/libslapd.so.0 > #21 0x00007f54073a133e in do_search () > #22 0x00007f540738f567 in connection_threadmain () > #23 0x00007f540509e7bb in _pt_root () from /lib64/libnspr4.so > #24 0x00007f5404a3fdc5 in start_thread () from /lib64/libpthread.so.0 > #25 0x00007f540476d28d in clone () from /lib64/libc.so.6 > Thread 30 (Thread 0x7f53d97fa700 (LWP 128216)): > #0 0x00007f5404a436d5 in pthread_cond_wait@@GLIBC_2.3.2 () from > /lib64/libpthread.so.0 > #1 0x00007f53fef9f2f3 in __db_hybrid_mutex_suspend () from /lib64/ > libdb-5.3.so > #2 0x00007f53fef9e640 in __db_tas_mutex_lock () from /lib64/libdb-5.3.so > #3 0x00007f53ff048d3a in __lock_get_internal () from /lib64/libdb-5.3.so > #4 0x00007f53ff049820 in __lock_get () from /lib64/libdb-5.3.so > #5 0x00007f53ff075142 in __db_lget () from /lib64/libdb-5.3.so > #6 0x00007f53fefbc605 in __bam_search () from /lib64/libdb-5.3.so > #7 0x00007f53fefa7256 in __bamc_search () from /lib64/libdb-5.3.so > #8 0x00007f53fefa8d0f in __bamc_get () from /lib64/libdb-5.3.so > #9 0x00007f53ff061ca6 in __dbc_iget () from /lib64/libdb-5.3.so > #10 0x00007f53ff070b02 in __dbc_get_pp () from /lib64/libdb-5.3.so > #11 0x00007f53fa54c680 in idl_new_fetch () from /usr/lib64/dirsrv/plugins/ > libback-ldbm.so > #12 0x00007f53fa55ab26 in index_read_ext_allids () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #13 0x00007f53fa545284 in keys2idl () from /usr/lib64/dirsrv/plugins/ > libback-ldbm.so > #14 0x00007f53fa5459e3 in ava_candidates.isra.0 () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #15 0x00007f53fa545fd2 in filter_candidates_ext () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #16 0x00007f53fa547096 in list_candidates () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #17 0x00007f53fa545f40 in filter_candidates_ext () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #18 0x00007f53fa58180c in subtree_candidates () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #19 0x00007f53fa582e7f in ldbm_back_search () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #20 0x00007f5406eb01d6 in op_shared_search () from > /usr/lib64/dirsrv/libslapd.so.0 > #21 0x00007f54073a133e in do_search () > #22 0x00007f540738f567 in connection_threadmain () > #23 0x00007f540509e7bb in _pt_root () from /lib64/libnspr4.so > #24 0x00007f5404a3fdc5 in start_thread () from /lib64/libpthread.so.0 > #25 0x00007f540476d28d in clone () from /lib64/libc.so.6 > Thread 29 (Thread 0x7f53d8ff9700 (LWP 128217)): > #0 0x00007f5404a436d5 in pthread_cond_wait@@GLIBC_2.3.2 () from > /lib64/libpthread.so.0 > #1 0x00007f53fef9f2f3 in __db_hybrid_mutex_suspend () from /lib64/ > libdb-5.3.so > #2 0x00007f53fef9e640 in __db_tas_mutex_lock () from /lib64/libdb-5.3.so > #3 0x00007f53ff048d3a in __lock_get_internal () from /lib64/libdb-5.3.so > #4 0x00007f53ff049820 in __lock_get () from /lib64/libdb-5.3.so > #5 0x00007f53ff075142 in __db_lget () from /lib64/libdb-5.3.so > #6 0x00007f53fefbc605 in __bam_search () from /lib64/libdb-5.3.so > #7 0x00007f53fefa7256 in __bamc_search () from /lib64/libdb-5.3.so > #8 0x00007f53fefa8d0f in __bamc_get () from /lib64/libdb-5.3.so > #9 0x00007f53ff061ca6 in __dbc_iget () from /lib64/libdb-5.3.so > #10 0x00007f53ff070b02 in __dbc_get_pp () from /lib64/libdb-5.3.so > #11 0x00007f53fa54c680 in idl_new_fetch () from /usr/lib64/dirsrv/plugins/ > libback-ldbm.so > #12 0x00007f53fa55ab26 in index_read_ext_allids () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #13 0x00007f53fa545284 in keys2idl () from /usr/lib64/dirsrv/plugins/ > libback-ldbm.so > #14 0x00007f53fa5459e3 in ava_candidates.isra.0 () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #15 0x00007f53fa545fd2 in filter_candidates_ext () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #16 0x00007f53fa547096 in list_candidates () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #17 0x00007f53fa545f40 in filter_candidates_ext () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #18 0x00007f53fa58180c in subtree_candidates () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #19 0x00007f53fa582e7f in ldbm_back_search () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #20 0x00007f5406eb01d6 in op_shared_search () from > /usr/lib64/dirsrv/libslapd.so.0 > #21 0x00007f54073a133e in do_search () > #22 0x00007f540738f567 in connection_threadmain () > #23 0x00007f540509e7bb in _pt_root () from /lib64/libnspr4.so > #24 0x00007f5404a3fdc5 in start_thread () from /lib64/libpthread.so.0 > #25 0x00007f540476d28d in clone () from /lib64/libc.so.6 > Thread 28 (Thread 0x7f53d87f8700 (LWP 128218)): > #0 0x00007f5404a436d5 in pthread_cond_wait@@GLIBC_2.3.2 () from > /lib64/libpthread.so.0 > #1 0x00007f53fef9f2f3 in __db_hybrid_mutex_suspend () from /lib64/ > libdb-5.3.so > #2 0x00007f53fef9e640 in __db_tas_mutex_lock () from /lib64/libdb-5.3.so > #3 0x00007f53ff048d3a in __lock_get_internal () from /lib64/libdb-5.3.so > #4 0x00007f53ff049820 in __lock_get () from /lib64/libdb-5.3.so > #5 0x00007f53ff075142 in __db_lget () from /lib64/libdb-5.3.so > #6 0x00007f53fefbc605 in __bam_search () from /lib64/libdb-5.3.so > #7 0x00007f53fefa7256 in __bamc_search () from /lib64/libdb-5.3.so > #8 0x00007f53fefa8d0f in __bamc_get () from /lib64/libdb-5.3.so > #9 0x00007f53ff061ca6 in __dbc_iget () from /lib64/libdb-5.3.so > #10 0x00007f53ff070b02 in __dbc_get_pp () from /lib64/libdb-5.3.so > #11 0x00007f53fa54c680 in idl_new_fetch () from /usr/lib64/dirsrv/plugins/ > libback-ldbm.so > #12 0x00007f53fa55ab26 in index_read_ext_allids () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #13 0x00007f53fa545284 in keys2idl () from /usr/lib64/dirsrv/plugins/ > libback-ldbm.so > #14 0x00007f53fa5459e3 in ava_candidates.isra.0 () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #15 0x00007f53fa545fd2 in filter_candidates_ext () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #16 0x00007f53fa547096 in list_candidates () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #17 0x00007f53fa545f40 in filter_candidates_ext () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #18 0x00007f53fa58180c in subtree_candidates () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #19 0x00007f53fa582e7f in ldbm_back_search () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #20 0x00007f5406eb01d6 in op_shared_search () from > /usr/lib64/dirsrv/libslapd.so.0 > #21 0x00007f54073a133e in do_search () > #22 0x00007f540738f567 in connection_threadmain () > #23 0x00007f540509e7bb in _pt_root () from /lib64/libnspr4.so > #24 0x00007f5404a3fdc5 in start_thread () from /lib64/libpthread.so.0 > #25 0x00007f540476d28d in clone () from /lib64/libc.so.6 > Thread 27 (Thread 0x7f53d7ff7700 (LWP 128219)): > #0 0x00007f5404a436d5 in pthread_cond_wait@@GLIBC_2.3.2 () from > /lib64/libpthread.so.0 > #1 0x00007f53fef9f2f3 in __db_hybrid_mutex_suspend () from /lib64/ > libdb-5.3.so > #2 0x00007f53fef9e640 in __db_tas_mutex_lock () from /lib64/libdb-5.3.so > #3 0x00007f53ff048d3a in __lock_get_internal () from /lib64/libdb-5.3.so > #4 0x00007f53ff049820 in __lock_get () from /lib64/libdb-5.3.so > #5 0x00007f53ff075142 in __db_lget () from /lib64/libdb-5.3.so > #6 0x00007f53fefbc605 in __bam_search () from /lib64/libdb-5.3.so > #7 0x00007f53fefa7256 in __bamc_search () from /lib64/libdb-5.3.so > #8 0x00007f53fefa8d0f in __bamc_get () from /lib64/libdb-5.3.so > #9 0x00007f53ff061ca6 in __dbc_iget () from /lib64/libdb-5.3.so > #10 0x00007f53ff070b02 in __dbc_get_pp () from /lib64/libdb-5.3.so > #11 0x00007f53fa54c680 in idl_new_fetch () from /usr/lib64/dirsrv/plugins/ > libback-ldbm.so > #12 0x00007f53fa55ab26 in index_read_ext_allids () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #13 0x00007f53fa545284 in keys2idl () from /usr/lib64/dirsrv/plugins/ > libback-ldbm.so > #14 0x00007f53fa5459e3 in ava_candidates.isra.0 () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #15 0x00007f53fa545fd2 in filter_candidates_ext () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #16 0x00007f53fa547096 in list_candidates () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #17 0x00007f53fa545f40 in filter_candidates_ext () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #18 0x00007f53fa58180c in subtree_candidates () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #19 0x00007f53fa582e7f in ldbm_back_search () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #20 0x00007f5406eb01d6 in op_shared_search () from > /usr/lib64/dirsrv/libslapd.so.0 > #21 0x00007f54073a133e in do_search () > #22 0x00007f540738f567 in connection_threadmain () > #23 0x00007f540509e7bb in _pt_root () from /lib64/libnspr4.so > #24 0x00007f5404a3fdc5 in start_thread () from /lib64/libpthread.so.0 > #25 0x00007f540476d28d in clone () from /lib64/libc.so.6 > Thread 26 (Thread 0x7f53d77f6700 (LWP 128220)): > #0 0x00007f5404a436d5 in pthread_cond_wait@@GLIBC_2.3.2 () from > /lib64/libpthread.so.0 > #1 0x00007f53fef9f2f3 in __db_hybrid_mutex_suspend () from /lib64/ > libdb-5.3.so > #2 0x00007f53fef9e640 in __db_tas_mutex_lock () from /lib64/libdb-5.3.so > #3 0x00007f53ff048d3a in __lock_get_internal () from /lib64/libdb-5.3.so > #4 0x00007f53ff049820 in __lock_get () from /lib64/libdb-5.3.so > #5 0x00007f53ff075142 in __db_lget () from /lib64/libdb-5.3.so > #6 0x00007f53fefbc605 in __bam_search () from /lib64/libdb-5.3.so > #7 0x00007f53fefa7256 in __bamc_search () from /lib64/libdb-5.3.so > #8 0x00007f53fefa8d0f in __bamc_get () from /lib64/libdb-5.3.so > #9 0x00007f53ff061ca6 in __dbc_iget () from /lib64/libdb-5.3.so > #10 0x00007f53ff070b02 in __dbc_get_pp () from /lib64/libdb-5.3.so > #11 0x00007f53fa54c680 in idl_new_fetch () from /usr/lib64/dirsrv/plugins/ > libback-ldbm.so > #12 0x00007f53fa55ab26 in index_read_ext_allids () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #13 0x00007f53fa545284 in keys2idl () from /usr/lib64/dirsrv/plugins/ > libback-ldbm.so > #14 0x00007f53fa5459e3 in ava_candidates.isra.0 () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #15 0x00007f53fa545fd2 in filter_candidates_ext () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #16 0x00007f53fa547096 in list_candidates () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #17 0x00007f53fa545f40 in filter_candidates_ext () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #18 0x00007f53fa58180c in subtree_candidates () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #19 0x00007f53fa582e7f in ldbm_back_search () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #20 0x00007f5406eb01d6 in op_shared_search () from > /usr/lib64/dirsrv/libslapd.so.0 > #21 0x00007f54073a133e in do_search () > #22 0x00007f540738f567 in connection_threadmain () > #23 0x00007f540509e7bb in _pt_root () from /lib64/libnspr4.so > #24 0x00007f5404a3fdc5 in start_thread () from /lib64/libpthread.so.0 > #25 0x00007f540476d28d in clone () from /lib64/libc.so.6 > Thread 25 (Thread 0x7f53d6ff5700 (LWP 128221)): > #0 0x00007f5404a436d5 in pthread_cond_wait@@GLIBC_2.3.2 () from > /lib64/libpthread.so.0 > #1 0x00007f53fef9f2f3 in __db_hybrid_mutex_suspend () from /lib64/ > libdb-5.3.so > #2 0x00007f53fef9e640 in __db_tas_mutex_lock () from /lib64/libdb-5.3.so > #3 0x00007f53ff048d3a in __lock_get_internal () from /lib64/libdb-5.3.so > #4 0x00007f53ff049820 in __lock_get () from /lib64/libdb-5.3.so > #5 0x00007f53ff075142 in __db_lget () from /lib64/libdb-5.3.so > #6 0x00007f53fefbc605 in __bam_search () from /lib64/libdb-5.3.so > #7 0x00007f53fefa7256 in __bamc_search () from /lib64/libdb-5.3.so > #8 0x00007f53fefa8d0f in __bamc_get () from /lib64/libdb-5.3.so > #9 0x00007f53ff061ca6 in __dbc_iget () from /lib64/libdb-5.3.so > #10 0x00007f53ff070b02 in __dbc_get_pp () from /lib64/libdb-5.3.so > #11 0x00007f53fa54c680 in idl_new_fetch () from /usr/lib64/dirsrv/plugins/ > libback-ldbm.so > #12 0x00007f53fa55ab26 in index_read_ext_allids () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #13 0x00007f53fa545284 in keys2idl () from /usr/lib64/dirsrv/plugins/ > libback-ldbm.so > #14 0x00007f53fa5459e3 in ava_candidates.isra.0 () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #15 0x00007f53fa545fd2 in filter_candidates_ext () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #16 0x00007f53fa547096 in list_candidates () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #17 0x00007f53fa545f40 in filter_candidates_ext () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #18 0x00007f53fa58180c in subtree_candidates () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #19 0x00007f53fa582e7f in ldbm_back_search () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #20 0x00007f5406eb01d6 in op_shared_search () from > /usr/lib64/dirsrv/libslapd.so.0 > #21 0x00007f54073a133e in do_search () > #22 0x00007f540738f567 in connection_threadmain () > #23 0x00007f540509e7bb in _pt_root () from /lib64/libnspr4.so > #24 0x00007f5404a3fdc5 in start_thread () from /lib64/libpthread.so.0 > #25 0x00007f540476d28d in clone () from /lib64/libc.so.6 > Thread 24 (Thread 0x7f53d67f4700 (LWP 128222)): > #0 0x00007f5404a436d5 in pthread_cond_wait@@GLIBC_2.3.2 () from > /lib64/libpthread.so.0 > #1 0x00007f53fef9f2f3 in __db_hybrid_mutex_suspend () from /lib64/ > libdb-5.3.so > #2 0x00007f53fef9e640 in __db_tas_mutex_lock () from /lib64/libdb-5.3.so > #3 0x00007f53ff048d3a in __lock_get_internal () from /lib64/libdb-5.3.so > #4 0x00007f53ff049820 in __lock_get () from /lib64/libdb-5.3.so > #5 0x00007f53ff075142 in __db_lget () from /lib64/libdb-5.3.so > #6 0x00007f53fefbc605 in __bam_search () from /lib64/libdb-5.3.so > #7 0x00007f53fefa7256 in __bamc_search () from /lib64/libdb-5.3.so > #8 0x00007f53fefa8d0f in __bamc_get () from /lib64/libdb-5.3.so > #9 0x00007f53ff061ca6 in __dbc_iget () from /lib64/libdb-5.3.so > #10 0x00007f53ff070b02 in __dbc_get_pp () from /lib64/libdb-5.3.so > #11 0x00007f53fa54c680 in idl_new_fetch () from /usr/lib64/dirsrv/plugins/ > libback-ldbm.so > #12 0x00007f53fa55ab26 in index_read_ext_allids () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #13 0x00007f53fa545284 in keys2idl () from /usr/lib64/dirsrv/plugins/ > libback-ldbm.so > #14 0x00007f53fa5459e3 in ava_candidates.isra.0 () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #15 0x00007f53fa545fd2 in filter_candidates_ext () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #16 0x00007f53fa547096 in list_candidates () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #17 0x00007f53fa545f40 in filter_candidates_ext () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #18 0x00007f53fa58180c in subtree_candidates () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #19 0x00007f53fa582e7f in ldbm_back_search () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #20 0x00007f5406eb01d6 in op_shared_search () from > /usr/lib64/dirsrv/libslapd.so.0 > #21 0x00007f54073a133e in do_search () > #22 0x00007f540738f567 in connection_threadmain () > #23 0x00007f540509e7bb in _pt_root () from /lib64/libnspr4.so > #24 0x00007f5404a3fdc5 in start_thread () from /lib64/libpthread.so.0 > #25 0x00007f540476d28d in clone () from /lib64/libc.so.6 > Thread 23 (Thread 0x7f53d5ff3700 (LWP 128223)): > #0 0x00007f5404a436d5 in pthread_cond_wait@@GLIBC_2.3.2 () from > /lib64/libpthread.so.0 > #1 0x00007f53fef9f2f3 in __db_hybrid_mutex_suspend () from /lib64/ > libdb-5.3.so > #2 0x00007f53fef9e640 in __db_tas_mutex_lock () from /lib64/libdb-5.3.so > #3 0x00007f53ff0b8e1b in __memp_fget () from /lib64/libdb-5.3.so > #4 0x00007f53fefbc211 in __bam_search () from /lib64/libdb-5.3.so > #5 0x00007f53fefa7256 in __bamc_search () from /lib64/libdb-5.3.so > #6 0x00007f53fefab6c4 in __bamc_put () from /lib64/libdb-5.3.so > #7 0x00007f53ff0626d5 in __dbc_iput () from /lib64/libdb-5.3.so > #8 0x00007f53ff05da0e in __db_put () from /lib64/libdb-5.3.so > #9 0x00007f53ff072fa4 in __db_put_pp () from /lib64/libdb-5.3.so > #10 0x00007f53fa54d4cf in idl_new_insert_key () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #11 0x00007f53fa55c5b5 in addordel_values_sv.isra.3 () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #12 0x00007f53fa55d25a in index_addordel_values_ext_sv () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #13 0x00007f53fa55d694 in index_addordel_values_sv () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #14 0x00007f53fa55d75a in index_addordel_entry () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #15 0x00007f53fa5605fa in ldbm_back_add () from /usr/lib64/dirsrv/plugins/ > libback-ldbm.so > #16 0x00007f5406e62cca in op_shared_add () from > /usr/lib64/dirsrv/libslapd.so.0 > #17 0x00007f5406e64020 in do_add () from /usr/lib64/dirsrv/libslapd.so.0 > #18 0x00007f540738f303 in connection_threadmain () > #19 0x00007f540509e7bb in _pt_root () from /lib64/libnspr4.so > #20 0x00007f5404a3fdc5 in start_thread () from /lib64/libpthread.so.0 > #21 0x00007f540476d28d in clone () from /lib64/libc.so.6 > Thread 22 (Thread 0x7f53d57f2700 (LWP 128224)): > #0 0x00007f5404a436d5 in pthread_cond_wait@@GLIBC_2.3.2 () from > /lib64/libpthread.so.0 > #1 0x00007f53fef9f2f3 in __db_hybrid_mutex_suspend () from /lib64/ > libdb-5.3.so > #2 0x00007f53fef9e640 in __db_tas_mutex_lock () from /lib64/libdb-5.3.so > #3 0x00007f53ff048d3a in __lock_get_internal () from /lib64/libdb-5.3.so > #4 0x00007f53ff049820 in __lock_get () from /lib64/libdb-5.3.so > #5 0x00007f53ff075142 in __db_lget () from /lib64/libdb-5.3.so > #6 0x00007f53fefbc605 in __bam_search () from /lib64/libdb-5.3.so > #7 0x00007f53fefa7256 in __bamc_search () from /lib64/libdb-5.3.so > #8 0x00007f53fefa8d0f in __bamc_get () from /lib64/libdb-5.3.so > #9 0x00007f53ff061ca6 in __dbc_iget () from /lib64/libdb-5.3.so > #10 0x00007f53ff070b02 in __dbc_get_pp () from /lib64/libdb-5.3.so > #11 0x00007f53fa54c680 in idl_new_fetch () from /usr/lib64/dirsrv/plugins/ > libback-ldbm.so > #12 0x00007f53fa55ab26 in index_read_ext_allids () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #13 0x00007f53fa545284 in keys2idl () from /usr/lib64/dirsrv/plugins/ > libback-ldbm.so > #14 0x00007f53fa5459e3 in ava_candidates.isra.0 () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #15 0x00007f53fa545fd2 in filter_candidates_ext () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #16 0x00007f53fa547096 in list_candidates () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #17 0x00007f53fa545f40 in filter_candidates_ext () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #18 0x00007f53fa58180c in subtree_candidates () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #19 0x00007f53fa582e7f in ldbm_back_search () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #20 0x00007f5406eb01d6 in op_shared_search () from > /usr/lib64/dirsrv/libslapd.so.0 > #21 0x00007f54073a133e in do_search () > #22 0x00007f540738f567 in connection_threadmain () > #23 0x00007f540509e7bb in _pt_root () from /lib64/libnspr4.so > #24 0x00007f5404a3fdc5 in start_thread () from /lib64/libpthread.so.0 > #25 0x00007f540476d28d in clone () from /lib64/libc.so.6 > Thread 21 (Thread 0x7f53d4ff1700 (LWP 128225)): > #0 0x00007f5404a436d5 in pthread_cond_wait@@GLIBC_2.3.2 () from > /lib64/libpthread.so.0 > #1 0x00007f53fef9f2f3 in __db_hybrid_mutex_suspend () from /lib64/ > libdb-5.3.so > #2 0x00007f53fef9e640 in __db_tas_mutex_lock () from /lib64/libdb-5.3.so > #3 0x00007f53ff048d3a in __lock_get_internal () from /lib64/libdb-5.3.so > #4 0x00007f53ff049820 in __lock_get () from /lib64/libdb-5.3.so > #5 0x00007f53ff075142 in __db_lget () from /lib64/libdb-5.3.so > #6 0x00007f53fefbc605 in __bam_search () from /lib64/libdb-5.3.so > #7 0x00007f53fefa7256 in __bamc_search () from /lib64/libdb-5.3.so > #8 0x00007f53fefa8d0f in __bamc_get () from /lib64/libdb-5.3.so > #9 0x00007f53ff061ca6 in __dbc_iget () from /lib64/libdb-5.3.so > #10 0x00007f53ff070b02 in __dbc_get_pp () from /lib64/libdb-5.3.so > #11 0x00007f53fa54c680 in idl_new_fetch () from /usr/lib64/dirsrv/plugins/ > libback-ldbm.so > #12 0x00007f53fa55ab26 in index_read_ext_allids () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #13 0x00007f53fa545284 in keys2idl () from /usr/lib64/dirsrv/plugins/ > libback-ldbm.so > #14 0x00007f53fa5459e3 in ava_candidates.isra.0 () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #15 0x00007f53fa545fd2 in filter_candidates_ext () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #16 0x00007f53fa547096 in list_candidates () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #17 0x00007f53fa545f40 in filter_candidates_ext () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #18 0x00007f53fa58180c in subtree_candidates () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #19 0x00007f53fa582e7f in ldbm_back_search () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #20 0x00007f5406eb01d6 in op_shared_search () from > /usr/lib64/dirsrv/libslapd.so.0 > #21 0x00007f54073a133e in do_search () > #22 0x00007f540738f567 in connection_threadmain () > #23 0x00007f540509e7bb in _pt_root () from /lib64/libnspr4.so > #24 0x00007f5404a3fdc5 in start_thread () from /lib64/libpthread.so.0 > #25 0x00007f540476d28d in clone () from /lib64/libc.so.6 > Thread 20 (Thread 0x7f53d47f0700 (LWP 128226)): > #0 0x00007f5404a436d5 in pthread_cond_wait@@GLIBC_2.3.2 () from > /lib64/libpthread.so.0 > #1 0x00007f53fef9f2f3 in __db_hybrid_mutex_suspend () from /lib64/ > libdb-5.3.so > #2 0x00007f53fef9e640 in __db_tas_mutex_lock () from /lib64/libdb-5.3.so > #3 0x00007f53ff048d3a in __lock_get_internal () from /lib64/libdb-5.3.so > #4 0x00007f53ff049820 in __lock_get () from /lib64/libdb-5.3.so > #5 0x00007f53ff075142 in __db_lget () from /lib64/libdb-5.3.so > #6 0x00007f53fefbc605 in __bam_search () from /lib64/libdb-5.3.so > #7 0x00007f53fefa7256 in __bamc_search () from /lib64/libdb-5.3.so > #8 0x00007f53fefa8d0f in __bamc_get () from /lib64/libdb-5.3.so > #9 0x00007f53ff061ca6 in __dbc_iget () from /lib64/libdb-5.3.so > #10 0x00007f53ff070b02 in __dbc_get_pp () from /lib64/libdb-5.3.so > #11 0x00007f53fa54c680 in idl_new_fetch () from /usr/lib64/dirsrv/plugins/ > libback-ldbm.so > #12 0x00007f53fa55ab26 in index_read_ext_allids () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #13 0x00007f53fa545284 in keys2idl () from /usr/lib64/dirsrv/plugins/ > libback-ldbm.so > #14 0x00007f53fa5459e3 in ava_candidates.isra.0 () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #15 0x00007f53fa545fd2 in filter_candidates_ext () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #16 0x00007f53fa547096 in list_candidates () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #17 0x00007f53fa545f40 in filter_candidates_ext () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #18 0x00007f53fa58180c in subtree_candidates () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #19 0x00007f53fa582e7f in ldbm_back_search () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #20 0x00007f5406eb01d6 in op_shared_search () from > /usr/lib64/dirsrv/libslapd.so.0 > #21 0x00007f5406ec094e in search_internal_callback_pb () from > /usr/lib64/dirsrv/libslapd.so.0 > #22 0x00007f5406ec0be8 in search_internal_pb () from > /usr/lib64/dirsrv/libslapd.so.0 > #23 0x00007f53fe118e90 in search_one_berval () from > /usr/lib64/dirsrv/plugins/libattr-unique-plugin.so > #24 0x00007f53fe119242 in search () from /usr/lib64/dirsrv/plugins/ > libattr-unique-plugin.so > #25 0x00007f53fe11954c in searchAllSubtrees () from > /usr/lib64/dirsrv/plugins/libattr-unique-plugin.so > #26 0x00007f53fe11a015 in preop_add () from /usr/lib64/dirsrv/plugins/ > libattr-unique-plugin.so > #27 0x00007f5406ebb848 in plugin_call_func () from > /usr/lib64/dirsrv/libslapd.so.0 > #28 0x00007f5406ebbb13 in plugin_call_plugins () from > /usr/lib64/dirsrv/libslapd.so.0 > #29 0x00007f5406e62c2b in op_shared_add () from > /usr/lib64/dirsrv/libslapd.so.0 > #30 0x00007f5406e64020 in do_add () from /usr/lib64/dirsrv/libslapd.so.0 > #31 0x00007f540738f303 in connection_threadmain () > #32 0x00007f540509e7bb in _pt_root () from /lib64/libnspr4.so > #33 0x00007f5404a3fdc5 in start_thread () from /lib64/libpthread.so.0 > #34 0x00007f540476d28d in clone () from /lib64/libc.so.6 > Thread 19 (Thread 0x7f53d3fef700 (LWP 128227)): > #0 0x00007f5404a436d5 in pthread_cond_wait@@GLIBC_2.3.2 () from > /lib64/libpthread.so.0 > #1 0x00007f53fef9f2f3 in __db_hybrid_mutex_suspend () from /lib64/ > libdb-5.3.so > #2 0x00007f53fef9e640 in __db_tas_mutex_lock () from /lib64/libdb-5.3.so > #3 0x00007f53ff048d3a in __lock_get_internal () from /lib64/libdb-5.3.so > #4 0x00007f53ff049820 in __lock_get () from /lib64/libdb-5.3.so > #5 0x00007f53ff075142 in __db_lget () from /lib64/libdb-5.3.so > #6 0x00007f53fefbc605 in __bam_search () from /lib64/libdb-5.3.so > #7 0x00007f53fefa7256 in __bamc_search () from /lib64/libdb-5.3.so > #8 0x00007f53fefa8d0f in __bamc_get () from /lib64/libdb-5.3.so > #9 0x00007f53ff061ca6 in __dbc_iget () from /lib64/libdb-5.3.so > #10 0x00007f53ff070b02 in __dbc_get_pp () from /lib64/libdb-5.3.so > #11 0x00007f53fa54c680 in idl_new_fetch () from /usr/lib64/dirsrv/plugins/ > libback-ldbm.so > #12 0x00007f53fa55ab26 in index_read_ext_allids () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #13 0x00007f53fa545284 in keys2idl () from /usr/lib64/dirsrv/plugins/ > libback-ldbm.so > #14 0x00007f53fa5459e3 in ava_candidates.isra.0 () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #15 0x00007f53fa545fd2 in filter_candidates_ext () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #16 0x00007f53fa547096 in list_candidates () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #17 0x00007f53fa545f40 in filter_candidates_ext () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #18 0x00007f53fa58180c in subtree_candidates () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #19 0x00007f53fa582e7f in ldbm_back_search () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #20 0x00007f5406eb01d6 in op_shared_search () from > /usr/lib64/dirsrv/libslapd.so.0 > #21 0x00007f54073a133e in do_search () > #22 0x00007f540738f567 in connection_threadmain () > #23 0x00007f540509e7bb in _pt_root () from /lib64/libnspr4.so > #24 0x00007f5404a3fdc5 in start_thread () from /lib64/libpthread.so.0 > #25 0x00007f540476d28d in clone () from /lib64/libc.so.6 > Thread 18 (Thread 0x7f53d37ee700 (LWP 128228)): > #0 0x00007f5404a436d5 in pthread_cond_wait@@GLIBC_2.3.2 () from > /lib64/libpthread.so.0 > #1 0x00007f53fef9f2f3 in __db_hybrid_mutex_suspend () from /lib64/ > libdb-5.3.so > #2 0x00007f53fef9e640 in __db_tas_mutex_lock () from /lib64/libdb-5.3.so > #3 0x00007f53ff048d3a in __lock_get_internal () from /lib64/libdb-5.3.so > #4 0x00007f53ff049820 in __lock_get () from /lib64/libdb-5.3.so > #5 0x00007f53ff075142 in __db_lget () from /lib64/libdb-5.3.so > #6 0x00007f53fefbc605 in __bam_search () from /lib64/libdb-5.3.so > #7 0x00007f53fefa7256 in __bamc_search () from /lib64/libdb-5.3.so > #8 0x00007f53fefa8d0f in __bamc_get () from /lib64/libdb-5.3.so > #9 0x00007f53ff061ca6 in __dbc_iget () from /lib64/libdb-5.3.so > #10 0x00007f53ff070b02 in __dbc_get_pp () from /lib64/libdb-5.3.so > #11 0x00007f53fa54c680 in idl_new_fetch () from /usr/lib64/dirsrv/plugins/ > libback-ldbm.so > #12 0x00007f53fa55ab26 in index_read_ext_allids () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #13 0x00007f53fa545284 in keys2idl () from /usr/lib64/dirsrv/plugins/ > libback-ldbm.so > #14 0x00007f53fa5459e3 in ava_candidates.isra.0 () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #15 0x00007f53fa545fd2 in filter_candidates_ext () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #16 0x00007f53fa547096 in list_candidates () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #17 0x00007f53fa545f40 in filter_candidates_ext () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #18 0x00007f53fa58180c in subtree_candidates () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #19 0x00007f53fa582e7f in ldbm_back_search () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #20 0x00007f5406eb01d6 in op_shared_search () from > /usr/lib64/dirsrv/libslapd.so.0 > #21 0x00007f54073a133e in do_search () > #22 0x00007f540738f567 in connection_threadmain () > #23 0x00007f540509e7bb in _pt_root () from /lib64/libnspr4.so > #24 0x00007f5404a3fdc5 in start_thread () from /lib64/libpthread.so.0 > #25 0x00007f540476d28d in clone () from /lib64/libc.so.6 > Thread 17 (Thread 0x7f53d2fed700 (LWP 128229)): > #0 0x00007f5404a436d5 in pthread_cond_wait@@GLIBC_2.3.2 () from > /lib64/libpthread.so.0 > #1 0x00007f53fef9f2f3 in __db_hybrid_mutex_suspend () from /lib64/ > libdb-5.3.so > #2 0x00007f53fef9e640 in __db_tas_mutex_lock () from /lib64/libdb-5.3.so > #3 0x00007f53ff048d3a in __lock_get_internal () from /lib64/libdb-5.3.so > #4 0x00007f53ff049820 in __lock_get () from /lib64/libdb-5.3.so > #5 0x00007f53ff075142 in __db_lget () from /lib64/libdb-5.3.so > #6 0x00007f53fefbc605 in __bam_search () from /lib64/libdb-5.3.so > #7 0x00007f53fefa7256 in __bamc_search () from /lib64/libdb-5.3.so > #8 0x00007f53fefa8d0f in __bamc_get () from /lib64/libdb-5.3.so > #9 0x00007f53ff061ca6 in __dbc_iget () from /lib64/libdb-5.3.so > #10 0x00007f53ff070b02 in __dbc_get_pp () from /lib64/libdb-5.3.so > #11 0x00007f53fa54c680 in idl_new_fetch () from /usr/lib64/dirsrv/plugins/ > libback-ldbm.so > #12 0x00007f53fa55ab26 in index_read_ext_allids () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #13 0x00007f53fa545284 in keys2idl () from /usr/lib64/dirsrv/plugins/ > libback-ldbm.so > #14 0x00007f53fa5459e3 in ava_candidates.isra.0 () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #15 0x00007f53fa545fd2 in filter_candidates_ext () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #16 0x00007f53fa547096 in list_candidates () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #17 0x00007f53fa545f40 in filter_candidates_ext () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #18 0x00007f53fa58180c in subtree_candidates () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #19 0x00007f53fa582e7f in ldbm_back_search () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #20 0x00007f5406eb01d6 in op_shared_search () from > /usr/lib64/dirsrv/libslapd.so.0 > #21 0x00007f54073a133e in do_search () > #22 0x00007f540738f567 in connection_threadmain () > #23 0x00007f540509e7bb in _pt_root () from /lib64/libnspr4.so > #24 0x00007f5404a3fdc5 in start_thread () from /lib64/libpthread.so.0 > #25 0x00007f540476d28d in clone () from /lib64/libc.so.6 > Thread 16 (Thread 0x7f53d27ec700 (LWP 128230)): > #0 0x00007f5404a436d5 in pthread_cond_wait@@GLIBC_2.3.2 () from > /lib64/libpthread.so.0 > #1 0x00007f53fef9f2f3 in __db_hybrid_mutex_suspend () from /lib64/ > libdb-5.3.so > #2 0x00007f53fef9e640 in __db_tas_mutex_lock () from /lib64/libdb-5.3.so > #3 0x00007f53ff048d3a in __lock_get_internal () from /lib64/libdb-5.3.so > #4 0x00007f53ff049820 in __lock_get () from /lib64/libdb-5.3.so > #5 0x00007f53ff075142 in __db_lget () from /lib64/libdb-5.3.so > #6 0x00007f53fefbc605 in __bam_search () from /lib64/libdb-5.3.so > #7 0x00007f53fefa7256 in __bamc_search () from /lib64/libdb-5.3.so > #8 0x00007f53fefa8d0f in __bamc_get () from /lib64/libdb-5.3.so > #9 0x00007f53ff061ca6 in __dbc_iget () from /lib64/libdb-5.3.so > #10 0x00007f53ff070b02 in __dbc_get_pp () from /lib64/libdb-5.3.so > #11 0x00007f53fa54c680 in idl_new_fetch () from /usr/lib64/dirsrv/plugins/ > libback-ldbm.so > #12 0x00007f53fa55ab26 in index_read_ext_allids () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #13 0x00007f53fa545284 in keys2idl () from /usr/lib64/dirsrv/plugins/ > libback-ldbm.so > #14 0x00007f53fa5459e3 in ava_candidates.isra.0 () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #15 0x00007f53fa545fd2 in filter_candidates_ext () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #16 0x00007f53fa547096 in list_candidates () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #17 0x00007f53fa545f40 in filter_candidates_ext () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #18 0x00007f53fa58180c in subtree_candidates () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #19 0x00007f53fa582e7f in ldbm_back_search () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #20 0x00007f5406eb01d6 in op_shared_search () from > /usr/lib64/dirsrv/libslapd.so.0 > #21 0x00007f54073a133e in do_search () > #22 0x00007f540738f567 in connection_threadmain () > #23 0x00007f540509e7bb in _pt_root () from /lib64/libnspr4.so > #24 0x00007f5404a3fdc5 in start_thread () from /lib64/libpthread.so.0 > #25 0x00007f540476d28d in clone () from /lib64/libc.so.6 > Thread 15 (Thread 0x7f53d1feb700 (LWP 128231)): > #0 0x00007f5404a436d5 in pthread_cond_wait@@GLIBC_2.3.2 () from > /lib64/libpthread.so.0 > #1 0x00007f53fef9f2f3 in __db_hybrid_mutex_suspend () from /lib64/ > libdb-5.3.so > #2 0x00007f53fef9e640 in __db_tas_mutex_lock () from /lib64/libdb-5.3.so > #3 0x00007f53ff048d3a in __lock_get_internal () from /lib64/libdb-5.3.so > #4 0x00007f53ff049820 in __lock_get () from /lib64/libdb-5.3.so > #5 0x00007f53ff075142 in __db_lget () from /lib64/libdb-5.3.so > #6 0x00007f53fefbc605 in __bam_search () from /lib64/libdb-5.3.so > #7 0x00007f53fefa7256 in __bamc_search () from /lib64/libdb-5.3.so > #8 0x00007f53fefa8d0f in __bamc_get () from /lib64/libdb-5.3.so > #9 0x00007f53ff061ca6 in __dbc_iget () from /lib64/libdb-5.3.so > #10 0x00007f53ff070b02 in __dbc_get_pp () from /lib64/libdb-5.3.so > #11 0x00007f53fa54c680 in idl_new_fetch () from /usr/lib64/dirsrv/plugins/ > libback-ldbm.so > #12 0x00007f53fa55ab26 in index_read_ext_allids () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #13 0x00007f53fa545284 in keys2idl () from /usr/lib64/dirsrv/plugins/ > libback-ldbm.so > #14 0x00007f53fa5459e3 in ava_candidates.isra.0 () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #15 0x00007f53fa545fd2 in filter_candidates_ext () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #16 0x00007f53fa547096 in list_candidates () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #17 0x00007f53fa545f40 in filter_candidates_ext () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #18 0x00007f53fa58180c in subtree_candidates () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #19 0x00007f53fa582e7f in ldbm_back_search () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #20 0x00007f5406eb01d6 in op_shared_search () from > /usr/lib64/dirsrv/libslapd.so.0 > #21 0x00007f54073a133e in do_search () > #22 0x00007f540738f567 in connection_threadmain () > #23 0x00007f540509e7bb in _pt_root () from /lib64/libnspr4.so > #24 0x00007f5404a3fdc5 in start_thread () from /lib64/libpthread.so.0 > #25 0x00007f540476d28d in clone () from /lib64/libc.so.6 > Thread 14 (Thread 0x7f53d17ea700 (LWP 128232)): > #0 0x00007f5404a436d5 in pthread_cond_wait@@GLIBC_2.3.2 () from > /lib64/libpthread.so.0 > #1 0x00007f53fef9f2f3 in __db_hybrid_mutex_suspend () from /lib64/ > libdb-5.3.so > #2 0x00007f53fef9e640 in __db_tas_mutex_lock () from /lib64/libdb-5.3.so > #3 0x00007f53ff048d3a in __lock_get_internal () from /lib64/libdb-5.3.so > #4 0x00007f53ff049820 in __lock_get () from /lib64/libdb-5.3.so > #5 0x00007f53ff075142 in __db_lget () from /lib64/libdb-5.3.so > #6 0x00007f53fefbc605 in __bam_search () from /lib64/libdb-5.3.so > #7 0x00007f53fefa7256 in __bamc_search () from /lib64/libdb-5.3.so > #8 0x00007f53fefa8d0f in __bamc_get () from /lib64/libdb-5.3.so > #9 0x00007f53ff061ca6 in __dbc_iget () from /lib64/libdb-5.3.so > #10 0x00007f53ff070b02 in __dbc_get_pp () from /lib64/libdb-5.3.so > #11 0x00007f53fa54c680 in idl_new_fetch () from /usr/lib64/dirsrv/plugins/ > libback-ldbm.so > #12 0x00007f53fa55ab26 in index_read_ext_allids () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #13 0x00007f53fa545284 in keys2idl () from /usr/lib64/dirsrv/plugins/ > libback-ldbm.so > #14 0x00007f53fa5459e3 in ava_candidates.isra.0 () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #15 0x00007f53fa545fd2 in filter_candidates_ext () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #16 0x00007f53fa547096 in list_candidates () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #17 0x00007f53fa545f40 in filter_candidates_ext () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #18 0x00007f53fa547096 in list_candidates () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #19 0x00007f53fa545f40 in filter_candidates_ext () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #20 0x00007f53fa547096 in list_candidates () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #21 0x00007f53fa545f40 in filter_candidates_ext () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #22 0x00007f53fa58180c in subtree_candidates () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #23 0x00007f53fa582e7f in ldbm_back_search () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #24 0x00007f5406eb01d6 in op_shared_search () from > /usr/lib64/dirsrv/libslapd.so.0 > #25 0x00007f54073a133e in do_search () > #26 0x00007f540738f3c3 in connection_threadmain () > #27 0x00007f540509e7bb in _pt_root () from /lib64/libnspr4.so > #28 0x00007f5404a3fdc5 in start_thread () from /lib64/libpthread.so.0 > #29 0x00007f540476d28d in clone () from /lib64/libc.so.6 > Thread 13 (Thread 0x7f53d0fe9700 (LWP 128233)): > #0 0x00007f5404a436d5 in pthread_cond_wait@@GLIBC_2.3.2 () from > /lib64/libpthread.so.0 > #1 0x00007f53fef9f2f3 in __db_hybrid_mutex_suspend () from /lib64/ > libdb-5.3.so > #2 0x00007f53fef9e640 in __db_tas_mutex_lock () from /lib64/libdb-5.3.so > #3 0x00007f53ff048d3a in __lock_get_internal () from /lib64/libdb-5.3.so > #4 0x00007f53ff049820 in __lock_get () from /lib64/libdb-5.3.so > #5 0x00007f53ff075142 in __db_lget () from /lib64/libdb-5.3.so > #6 0x00007f53fefbc605 in __bam_search () from /lib64/libdb-5.3.so > #7 0x00007f53fefa7256 in __bamc_search () from /lib64/libdb-5.3.so > #8 0x00007f53fefa8d0f in __bamc_get () from /lib64/libdb-5.3.so > #9 0x00007f53ff061ca6 in __dbc_iget () from /lib64/libdb-5.3.so > #10 0x00007f53ff070b02 in __dbc_get_pp () from /lib64/libdb-5.3.so > #11 0x00007f53fa54c680 in idl_new_fetch () from /usr/lib64/dirsrv/plugins/ > libback-ldbm.so > #12 0x00007f53fa55ab26 in index_read_ext_allids () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #13 0x00007f53fa545284 in keys2idl () from /usr/lib64/dirsrv/plugins/ > libback-ldbm.so > #14 0x00007f53fa5459e3 in ava_candidates.isra.0 () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #15 0x00007f53fa545fd2 in filter_candidates_ext () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #16 0x00007f53fa547096 in list_candidates () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #17 0x00007f53fa545f40 in filter_candidates_ext () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #18 0x00007f53fa58180c in subtree_candidates () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #19 0x00007f53fa582e7f in ldbm_back_search () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #20 0x00007f5406eb01d6 in op_shared_search () from > /usr/lib64/dirsrv/libslapd.so.0 > #21 0x00007f54073a133e in do_search () > #22 0x00007f540738f567 in connection_threadmain () > #23 0x00007f540509e7bb in _pt_root () from /lib64/libnspr4.so > #24 0x00007f5404a3fdc5 in start_thread () from /lib64/libpthread.so.0 > #25 0x00007f540476d28d in clone () from /lib64/libc.so.6 > Thread 12 (Thread 0x7f53d07e8700 (LWP 128234)): > #0 0x00007f5404a436d5 in pthread_cond_wait@@GLIBC_2.3.2 () from > /lib64/libpthread.so.0 > #1 0x00007f53fef9f2f3 in __db_hybrid_mutex_suspend () from /lib64/ > libdb-5.3.so > #2 0x00007f53fef9e640 in __db_tas_mutex_lock () from /lib64/libdb-5.3.so > #3 0x00007f53ff048d3a in __lock_get_internal () from /lib64/libdb-5.3.so > #4 0x00007f53ff049820 in __lock_get () from /lib64/libdb-5.3.so > #5 0x00007f53ff075142 in __db_lget () from /lib64/libdb-5.3.so > #6 0x00007f53fefbc605 in __bam_search () from /lib64/libdb-5.3.so > #7 0x00007f53fefa7256 in __bamc_search () from /lib64/libdb-5.3.so > #8 0x00007f53fefa8d0f in __bamc_get () from /lib64/libdb-5.3.so > #9 0x00007f53ff061ca6 in __dbc_iget () from /lib64/libdb-5.3.so > #10 0x00007f53ff070b02 in __dbc_get_pp () from /lib64/libdb-5.3.so > #11 0x00007f53fa54c680 in idl_new_fetch () from /usr/lib64/dirsrv/plugins/ > libback-ldbm.so > #12 0x00007f53fa55ab26 in index_read_ext_allids () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #13 0x00007f53fa545284 in keys2idl () from /usr/lib64/dirsrv/plugins/ > libback-ldbm.so > #14 0x00007f53fa5459e3 in ava_candidates.isra.0 () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #15 0x00007f53fa545fd2 in filter_candidates_ext () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #16 0x00007f53fa547096 in list_candidates () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #17 0x00007f53fa545f40 in filter_candidates_ext () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #18 0x00007f53fa58180c in subtree_candidates () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #19 0x00007f53fa582e7f in ldbm_back_search () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #20 0x00007f5406eb01d6 in op_shared_search () from > /usr/lib64/dirsrv/libslapd.so.0 > #21 0x00007f54073a133e in do_search () > #22 0x00007f540738f567 in connection_threadmain () > #23 0x00007f540509e7bb in _pt_root () from /lib64/libnspr4.so > #24 0x00007f5404a3fdc5 in start_thread () from /lib64/libpthread.so.0 > #25 0x00007f540476d28d in clone () from /lib64/libc.so.6 > Thread 11 (Thread 0x7f53cffe7700 (LWP 128235)): > #0 0x00007f5404a436d5 in pthread_cond_wait@@GLIBC_2.3.2 () from > /lib64/libpthread.so.0 > #1 0x00007f53fef9f2f3 in __db_hybrid_mutex_suspend () from /lib64/ > libdb-5.3.so > #2 0x00007f53fef9e640 in __db_tas_mutex_lock () from /lib64/libdb-5.3.so > #3 0x00007f53ff048d3a in __lock_get_internal () from /lib64/libdb-5.3.so > #4 0x00007f53ff049820 in __lock_get () from /lib64/libdb-5.3.so > #5 0x00007f53ff075142 in __db_lget () from /lib64/libdb-5.3.so > #6 0x00007f53fefbc605 in __bam_search () from /lib64/libdb-5.3.so > #7 0x00007f53fefa7256 in __bamc_search () from /lib64/libdb-5.3.so > #8 0x00007f53fefa8d0f in __bamc_get () from /lib64/libdb-5.3.so > #9 0x00007f53ff061ca6 in __dbc_iget () from /lib64/libdb-5.3.so > #10 0x00007f53ff070b02 in __dbc_get_pp () from /lib64/libdb-5.3.so > #11 0x00007f53fa54c680 in idl_new_fetch () from /usr/lib64/dirsrv/plugins/ > libback-ldbm.so > #12 0x00007f53fa55ab26 in index_read_ext_allids () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #13 0x00007f53fa545284 in keys2idl () from /usr/lib64/dirsrv/plugins/ > libback-ldbm.so > #14 0x00007f53fa5459e3 in ava_candidates.isra.0 () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #15 0x00007f53fa545fd2 in filter_candidates_ext () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #16 0x00007f53fa547096 in list_candidates () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #17 0x00007f53fa545f40 in filter_candidates_ext () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #18 0x00007f53fa58180c in subtree_candidates () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #19 0x00007f53fa582e7f in ldbm_back_search () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #20 0x00007f5406eb01d6 in op_shared_search () from > /usr/lib64/dirsrv/libslapd.so.0 > #21 0x00007f54073a133e in do_search () > #22 0x00007f540738f567 in connection_threadmain () > #23 0x00007f540509e7bb in _pt_root () from /lib64/libnspr4.so > #24 0x00007f5404a3fdc5 in start_thread () from /lib64/libpthread.so.0 > #25 0x00007f540476d28d in clone () from /lib64/libc.so.6 > Thread 10 (Thread 0x7f53cf7e6700 (LWP 128236)): > #0 0x00007f5404a436d5 in pthread_cond_wait@@GLIBC_2.3.2 () from > /lib64/libpthread.so.0 > #1 0x00007f53fef9f2f3 in __db_hybrid_mutex_suspend () from /lib64/ > libdb-5.3.so > #2 0x00007f53fef9e640 in __db_tas_mutex_lock () from /lib64/libdb-5.3.so > #3 0x00007f53ff048d3a in __lock_get_internal () from /lib64/libdb-5.3.so > #4 0x00007f53ff049820 in __lock_get () from /lib64/libdb-5.3.so > #5 0x00007f53ff075142 in __db_lget () from /lib64/libdb-5.3.so > #6 0x00007f53fefbc605 in __bam_search () from /lib64/libdb-5.3.so > #7 0x00007f53fefa7256 in __bamc_search () from /lib64/libdb-5.3.so > #8 0x00007f53fefa8d0f in __bamc_get () from /lib64/libdb-5.3.so > #9 0x00007f53ff061ca6 in __dbc_iget () from /lib64/libdb-5.3.so > #10 0x00007f53ff070b02 in __dbc_get_pp () from /lib64/libdb-5.3.so > #11 0x00007f53fa54c680 in idl_new_fetch () from /usr/lib64/dirsrv/plugins/ > libback-ldbm.so > #12 0x00007f53fa55ab26 in index_read_ext_allids () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #13 0x00007f53fa545284 in keys2idl () from /usr/lib64/dirsrv/plugins/ > libback-ldbm.so > #14 0x00007f53fa5459e3 in ava_candidates.isra.0 () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #15 0x00007f53fa545fd2 in filter_candidates_ext () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #16 0x00007f53fa547096 in list_candidates () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #17 0x00007f53fa545f40 in filter_candidates_ext () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #18 0x00007f53fa58180c in subtree_candidates () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #19 0x00007f53fa582e7f in ldbm_back_search () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #20 0x00007f5406eb01d6 in op_shared_search () from > /usr/lib64/dirsrv/libslapd.so.0 > #21 0x00007f54073a133e in do_search () > #22 0x00007f540738f567 in connection_threadmain () > #23 0x00007f540509e7bb in _pt_root () from /lib64/libnspr4.so > #24 0x00007f5404a3fdc5 in start_thread () from /lib64/libpthread.so.0 > #25 0x00007f540476d28d in clone () from /lib64/libc.so.6 > Thread 9 (Thread 0x7f53cefe5700 (LWP 128237)): > #0 0x00007f5404a436d5 in pthread_cond_wait@@GLIBC_2.3.2 () from > /lib64/libpthread.so.0 > #1 0x00007f53fef9f2f3 in __db_hybrid_mutex_suspend () from /lib64/ > libdb-5.3.so > #2 0x00007f53fef9e640 in __db_tas_mutex_lock () from /lib64/libdb-5.3.so > #3 0x00007f53ff048d3a in __lock_get_internal () from /lib64/libdb-5.3.so > #4 0x00007f53ff049820 in __lock_get () from /lib64/libdb-5.3.so > #5 0x00007f53ff075142 in __db_lget () from /lib64/libdb-5.3.so > #6 0x00007f53fefbc605 in __bam_search () from /lib64/libdb-5.3.so > #7 0x00007f53fefa7256 in __bamc_search () from /lib64/libdb-5.3.so > #8 0x00007f53fefa8d0f in __bamc_get () from /lib64/libdb-5.3.so > #9 0x00007f53ff061ca6 in __dbc_iget () from /lib64/libdb-5.3.so > #10 0x00007f53ff070b02 in __dbc_get_pp () from /lib64/libdb-5.3.so > #11 0x00007f53fa54c680 in idl_new_fetch () from /usr/lib64/dirsrv/plugins/ > libback-ldbm.so > #12 0x00007f53fa55ab26 in index_read_ext_allids () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #13 0x00007f53fa545284 in keys2idl () from /usr/lib64/dirsrv/plugins/ > libback-ldbm.so > #14 0x00007f53fa5459e3 in ava_candidates.isra.0 () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #15 0x00007f53fa545fd2 in filter_candidates_ext () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #16 0x00007f53fa547096 in list_candidates () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #17 0x00007f53fa545f40 in filter_candidates_ext () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #18 0x00007f53fa58180c in subtree_candidates () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #19 0x00007f53fa582e7f in ldbm_back_search () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #20 0x00007f5406eb01d6 in op_shared_search () from > /usr/lib64/dirsrv/libslapd.so.0 > #21 0x00007f54073a133e in do_search () > #22 0x00007f540738f567 in connection_threadmain () > #23 0x00007f540509e7bb in _pt_root () from /lib64/libnspr4.so > #24 0x00007f5404a3fdc5 in start_thread () from /lib64/libpthread.so.0 > #25 0x00007f540476d28d in clone () from /lib64/libc.so.6 > Thread 8 (Thread 0x7f53ce7e4700 (LWP 128238)): > #0 0x00007f5404a436d5 in pthread_cond_wait@@GLIBC_2.3.2 () from > /lib64/libpthread.so.0 > #1 0x00007f53fef9f2f3 in __db_hybrid_mutex_suspend () from /lib64/ > libdb-5.3.so > #2 0x00007f53fef9e640 in __db_tas_mutex_lock () from /lib64/libdb-5.3.so > #3 0x00007f53ff048d3a in __lock_get_internal () from /lib64/libdb-5.3.so > #4 0x00007f53ff049820 in __lock_get () from /lib64/libdb-5.3.so > #5 0x00007f53ff075142 in __db_lget () from /lib64/libdb-5.3.so > #6 0x00007f53fefbc605 in __bam_search () from /lib64/libdb-5.3.so > #7 0x00007f53fefa7256 in __bamc_search () from /lib64/libdb-5.3.so > #8 0x00007f53fefa8d0f in __bamc_get () from /lib64/libdb-5.3.so > #9 0x00007f53ff061ca6 in __dbc_iget () from /lib64/libdb-5.3.so > #10 0x00007f53ff070b02 in __dbc_get_pp () from /lib64/libdb-5.3.so > #11 0x00007f53fa54c680 in idl_new_fetch () from /usr/lib64/dirsrv/plugins/ > libback-ldbm.so > #12 0x00007f53fa55ab26 in index_read_ext_allids () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #13 0x00007f53fa545284 in keys2idl () from /usr/lib64/dirsrv/plugins/ > libback-ldbm.so > #14 0x00007f53fa5459e3 in ava_candidates.isra.0 () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #15 0x00007f53fa545fd2 in filter_candidates_ext () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #16 0x00007f53fa547096 in list_candidates () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #17 0x00007f53fa545f40 in filter_candidates_ext () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #18 0x00007f53fa58180c in subtree_candidates () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #19 0x00007f53fa582e7f in ldbm_back_search () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #20 0x00007f5406eb01d6 in op_shared_search () from > /usr/lib64/dirsrv/libslapd.so.0 > #21 0x00007f54073a133e in do_search () > #22 0x00007f540738f567 in connection_threadmain () > #23 0x00007f540509e7bb in _pt_root () from /lib64/libnspr4.so > #24 0x00007f5404a3fdc5 in start_thread () from /lib64/libpthread.so.0 > #25 0x00007f540476d28d in clone () from /lib64/libc.so.6 > Thread 7 (Thread 0x7f53cdfe3700 (LWP 128239)): > #0 0x00007f5404a436d5 in pthread_cond_wait@@GLIBC_2.3.2 () from > /lib64/libpthread.so.0 > #1 0x00007f53fef9f2f3 in __db_hybrid_mutex_suspend () from /lib64/ > libdb-5.3.so > #2 0x00007f53fef9e640 in __db_tas_mutex_lock () from /lib64/libdb-5.3.so > #3 0x00007f53ff048d3a in __lock_get_internal () from /lib64/libdb-5.3.so > #4 0x00007f53ff049820 in __lock_get () from /lib64/libdb-5.3.so > #5 0x00007f53ff075142 in __db_lget () from /lib64/libdb-5.3.so > #6 0x00007f53fefbc605 in __bam_search () from /lib64/libdb-5.3.so > #7 0x00007f53fefa7256 in __bamc_search () from /lib64/libdb-5.3.so > #8 0x00007f53fefa8d0f in __bamc_get () from /lib64/libdb-5.3.so > #9 0x00007f53ff061ca6 in __dbc_iget () from /lib64/libdb-5.3.so > #10 0x00007f53ff070b02 in __dbc_get_pp () from /lib64/libdb-5.3.so > #11 0x00007f53fa54c680 in idl_new_fetch () from /usr/lib64/dirsrv/plugins/ > libback-ldbm.so > #12 0x00007f53fa55ab26 in index_read_ext_allids () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #13 0x00007f53fa545284 in keys2idl () from /usr/lib64/dirsrv/plugins/ > libback-ldbm.so > #14 0x00007f53fa5459e3 in ava_candidates.isra.0 () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #15 0x00007f53fa545fd2 in filter_candidates_ext () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #16 0x00007f53fa547096 in list_candidates () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #17 0x00007f53fa545f40 in filter_candidates_ext () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #18 0x00007f53fa58180c in subtree_candidates () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #19 0x00007f53fa582e7f in ldbm_back_search () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #20 0x00007f5406eb01d6 in op_shared_search () from > /usr/lib64/dirsrv/libslapd.so.0 > #21 0x00007f54073a133e in do_search () > #22 0x00007f540738f567 in connection_threadmain () > #23 0x00007f540509e7bb in _pt_root () from /lib64/libnspr4.so > #24 0x00007f5404a3fdc5 in start_thread () from /lib64/libpthread.so.0 > #25 0x00007f540476d28d in clone () from /lib64/libc.so.6 > Thread 6 (Thread 0x7f53cd7e2700 (LWP 128240)): > #0 0x00007f5404a436d5 in pthread_cond_wait@@GLIBC_2.3.2 () from > /lib64/libpthread.so.0 > #1 0x00007f53fef9f2f3 in __db_hybrid_mutex_suspend () from /lib64/ > libdb-5.3.so > #2 0x00007f53fef9eb37 in __db_tas_mutex_readlock () from /lib64/ > libdb-5.3.so > #3 0x00007f53ff0b8d35 in __memp_fget () from /lib64/libdb-5.3.so > #4 0x00007f53ff062333 in __dbc_iget () from /lib64/libdb-5.3.so > #5 0x00007f53ff070b02 in __dbc_get_pp () from /lib64/libdb-5.3.so > #6 0x00007f53fa54c822 in idl_new_fetch () from /usr/lib64/dirsrv/plugins/ > libback-ldbm.so > #7 0x00007f53fa55ab26 in index_read_ext_allids () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #8 0x00007f53fa545284 in keys2idl () from /usr/lib64/dirsrv/plugins/ > libback-ldbm.so > #9 0x00007f53fa5459e3 in ava_candidates.isra.0 () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #10 0x00007f53fa545fd2 in filter_candidates_ext () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #11 0x00007f53fa547096 in list_candidates () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #12 0x00007f53fa545f40 in filter_candidates_ext () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #13 0x00007f53fa547096 in list_candidates () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #14 0x00007f53fa545f40 in filter_candidates_ext () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #15 0x00007f53fa547096 in list_candidates () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #16 0x00007f53fa545f40 in filter_candidates_ext () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #17 0x00007f53fa58180c in subtree_candidates () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #18 0x00007f53fa582e7f in ldbm_back_search () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #19 0x00007f5406eb01d6 in op_shared_search () from > /usr/lib64/dirsrv/libslapd.so.0 > #20 0x00007f54073a133e in do_search () > #21 0x00007f540738f3c3 in connection_threadmain () > #22 0x00007f540509e7bb in _pt_root () from /lib64/libnspr4.so > #23 0x00007f5404a3fdc5 in start_thread () from /lib64/libpthread.so.0 > #24 0x00007f540476d28d in clone () from /lib64/libc.so.6 > Thread 5 (Thread 0x7f53ccfe1700 (LWP 128241)): > #0 0x00007f5404a436d5 in pthread_cond_wait@@GLIBC_2.3.2 () from > /lib64/libpthread.so.0 > #1 0x00007f53fef9f2f3 in __db_hybrid_mutex_suspend () from /lib64/ > libdb-5.3.so > #2 0x00007f53fef9e640 in __db_tas_mutex_lock () from /lib64/libdb-5.3.so > #3 0x00007f53ff048d3a in __lock_get_internal () from /lib64/libdb-5.3.so > #4 0x00007f53ff049820 in __lock_get () from /lib64/libdb-5.3.so > #5 0x00007f53ff075142 in __db_lget () from /lib64/libdb-5.3.so > #6 0x00007f53fefbc605 in __bam_search () from /lib64/libdb-5.3.so > #7 0x00007f53fefa7256 in __bamc_search () from /lib64/libdb-5.3.so > #8 0x00007f53fefa8d0f in __bamc_get () from /lib64/libdb-5.3.so > #9 0x00007f53ff061ca6 in __dbc_iget () from /lib64/libdb-5.3.so > #10 0x00007f53ff070b02 in __dbc_get_pp () from /lib64/libdb-5.3.so > #11 0x00007f53fa54c680 in idl_new_fetch () from /usr/lib64/dirsrv/plugins/ > libback-ldbm.so > #12 0x00007f53fa55ab26 in index_read_ext_allids () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #13 0x00007f53fa545284 in keys2idl () from /usr/lib64/dirsrv/plugins/ > libback-ldbm.so > #14 0x00007f53fa5459e3 in ava_candidates.isra.0 () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #15 0x00007f53fa545fd2 in filter_candidates_ext () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #16 0x00007f53fa547096 in list_candidates () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #17 0x00007f53fa545f40 in filter_candidates_ext () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #18 0x00007f53fa547096 in list_candidates () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #19 0x00007f53fa545f40 in filter_candidates_ext () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #20 0x00007f53fa547096 in list_candidates () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #21 0x00007f53fa545f40 in filter_candidates_ext () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #22 0x00007f53fa58180c in subtree_candidates () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #23 0x00007f53fa582e7f in ldbm_back_search () from > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #24 0x00007f5406eb01d6 in op_shared_search () from > /usr/lib64/dirsrv/libslapd.so.0 > #25 0x00007f54073a133e in do_search () > #26 0x00007f540738f3c3 in connection_threadmain () > #27 0x00007f540509e7bb in _pt_root () from /lib64/libnspr4.so > #28 0x00007f5404a3fdc5 in start_thread () from /lib64/libpthread.so.0 > #29 0x00007f540476d28d in clone () from /lib64/libc.so.6 > Thread 4 (Thread 0x7f53cc7e0700 (LWP 128242)): > #0 0x00007f54047649b3 in select () from /lib64/libc.so.6 > #1 0x00007f5406eef0e9 in DS_Sleep () from /usr/lib64/dirsrv/libslapd.so.0 > #2 0x00007f5407390325 in time_thread () > #3 0x00007f540509e7bb in _pt_root () from /lib64/libnspr4.so > #4 0x00007f5404a3fdc5 in start_thread () from /lib64/libpthread.so.0 > #5 0x00007f540476d28d in clone () from /lib64/libc.so.6 > Thread 3 (Thread 0x7f53cb7a0700 (LWP 128259)): > #0 0x00007f5404a436d5 in pthread_cond_wait@@GLIBC_2.3.2 () from > /lib64/libpthread.so.0 > #1 0x00007f5405099050 in PR_WaitCondVar () from /lib64/libnspr4.so > #2 0x00007f540739bf25 in ps_send_results () > #3 0x00007f540509e7bb in _pt_root () from /lib64/libnspr4.so > #4 0x00007f5404a3fdc5 in start_thread () from /lib64/libpthread.so.0 > #5 0x00007f540476d28d in clone () from /lib64/libc.so.6 > Thread 2 (Thread 0x7f53cab9d700 (LWP 9020)): > #0 0x00007f5404a436d5 in pthread_cond_wait@@GLIBC_2.3.2 () from > /lib64/libpthread.so.0 > #1 0x00007f5405099050 in PR_WaitCondVar () from /lib64/libnspr4.so > #2 0x00007f540739bf25 in ps_send_results () > #3 0x00007f540509e7bb in _pt_root () from /lib64/libnspr4.so > #4 0x00007f5404a3fdc5 in start_thread () from /lib64/libpthread.so.0 > #5 0x00007f540476d28d in clone () from /lib64/libc.so.6 > Thread 1 (Thread 0x7f5407357840 (LWP 128124)): > #0 0x00007f5404762c3d in poll () from /lib64/libc.so.6 > #1 0x00007f540509a967 in _pr_poll_with_poll () from /lib64/libnspr4.so > #2 0x00007f5407393ff1 in slapd_daemon () > #3 0x00007f5407385c6c in main () > > > Thanks > > On Tue, Sep 6, 2016 at 5:51 PM, Rakesh Rajasekharan < > rakesh.rajasekharan at gmail.com> wrote: > >> Hi, >> >> With script provided by Rich, I looked at my db cache usage and it does >> not look to be high enough >> >> # under each db are the list of selected indexes specified with INDEXLIST >> dbname count free free% size >> changelog:ent 5 2076333 99.0 4163.8 >> changelog:dn 15976 9303557 88.7 74.0 >> userroot:ent 4763 175321074 83.6 7221.1 >> userroot:dn 4763 9995056 95.3 103.0 >> ipaca:ent 425 6337866 60.4 9759.8 >> ipaca:dn 425 10451081 99.7 81.6 >> >> Right now I set it to 200MB. So, I am planning to bring it down to 100MB >> >> Also, *nsslapd-db-checkpoint-interval *will change it to 80 or 90 from >> default of 60 >> >> I will check this out and the impact it has on the hang issues and get >> back >> >> Thanks >> Rakesh >> >> On Tue, Sep 6, 2016 at 4:13 PM, Ludwig Krispenz < >> lkrispen at redhat.com> wrote: >> >>> Hi, >>> >>> I think you are running into this bug: >>> >>> https://bugzilla.redhat.com/show_bug.cgi?id=1349779 >>> it was the outcome of an analysis started in this one: >>> https://bugzilla.redhat.com/show_bug.cgi?id=1347800 >>> >>> the problem is that it looks like the checkpoint thread is holding a >>> mutex and otehr threads are blocked on it. It looks like an issue in >>> BerkeleyDB although there is no explanation from the BDB code why this >>> could happen, we also had a support case with Oracle to investigate this, >>> with no solution. Oracle at the end was providing a diagnostic patch, which >>> we did not test since the customer with the original problem had worked >>> around this by changing their configuration. >>> I don't know the status of this patch and if we can make it available, >>> probably depends on your support status, you need to contact support for >>> this. >>> >>> About the workarounds: >>> you can try to change timing for the database update and checkpointing >>> eg by reducing the dbcache size, changing the checkpoint interval. In the >>> customer case they identified one substring index, which was frequently >>> updated but not really required - and removing this index helped. >>> >>> Regards, >>> Ludwig >>> >>> On 09/06/2016 11:31 AM, thierry bordaz wrote: >>> >>> Hi, >>> >>> The following thread is hanging most of the others threads >>> >>> Thread 20 (Thread 0x7f68007f8700 (LWP 4873)): >>> >>> #0 0x00007f683d0506d5 in >>> pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 >>> #1 0x00007f68375ac2f3 in __db_hybrid_mutex_suspend () from /lib64/ >>> libdb-5.3.so >>> #2 0x00007f68375ab640 in __db_tas_mutex_lock () from /lib64/ >>> libdb-5.3.so >>> #3 0x00007f68376c5e1b in __memp_fget () from /lib64/libdb-5.3.so >>> #4 0x00007f68375c9211 in __bam_search () from /lib64/libdb-5.3.so >>> #5 0x00007f68375b4256 in __bamc_search () from /lib64/libdb-5.3.so >>> #6 0x00007f68375b86c4 in __bamc_put () from /lib64/libdb-5.3.so >>> #7 0x00007f683766f6d5 in __dbc_iput () from /lib64/libdb-5.3.so >>> #8 0x00007f683766aa0e in __db_put () from /lib64/libdb-5.3.so >>> #9 0x00007f683767ffa4 in __db_put_pp () from /lib64/libdb-5.3.so >>> #10 0x00007f6832b5a4cf in idl_new_insert_key () from >>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>> #11 0x00007f6832b695b5 in addordel_values_sv.isra.3 () from >>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>> #12 0x00007f6832b6a25a in index_addordel_values_ext_sv () from >>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>> #13 0x00007f6832b6a694 in index_addordel_values_sv () from >>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>> #14 0x00007f6832b6a75a in index_addordel_entry () from >>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>> #15 0x00007f6832b6d5fa in ldbm_back_add () from >>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>> #16 0x00007f683f46fcca in op_shared_add () from >>> /usr/lib64/dirsrv/libslapd.so.0 >>> #17 0x00007f683f471020 in do_add () from /usr/lib64/dirsrv/libslapd.so.0 >>> #18 0x00007f683f99c303 in connection_threadmain () >>> #19 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so >>> #20 0x00007f683d04cdc5 in start_thread () from /lib64/libpthread.so.0 >>> #21 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 >>> >>> This thread is likely not hang on 'db_tas_mutex_lock' as the db_lock >>> does not show it waiting for something. >>> Unfortunately it is not shown in the output of top -H, did you get the >>> complete output ? >>> Ludwig mentioned a known issue that DS is not scaling well on NUMA arch, >>> do you know if the machine is NUMA ? >>> A possibility is that this thread get severely slowed down by a NUMA >>> remote lock and showing DS has hung. >>> This is why it would be interesting to know if it consums CPU. >>> >>> thanks >>> theirry >>> >>> On 09/06/2016 10:59 AM, Rakesh Rajasekharan wrote: >>> >>> Hi Thierry/Ludwig, >>> >>> I again hit the hang issue. This is the pstack of the ns-slapd >>> >>> So was able to this time get the pstack and db locks >>> >>> db locks details >>> Default locking region information: >>> 451 Last allocated locker ID >>> 0x7fffffff Current maximum unused locker ID >>> 9 Number of lock modes >>> 800 Initial number of locks allocated >>> 0 Initial number of lockers allocated >>> 800 Initial number of lock objects allocated >>> 10000 Maximum number of locks possible >>> 10000 Maximum number of lockers possible >>> 10000 Maximum number of lock objects possible >>> 800 Current number of locks allocated >>> 366 Current number of lockers allocated >>> 800 Current number of lock objects allocated >>> 160 Number of lock object partitions >>> 8191 Size of object hash table >>> 126 Number of current locks >>> 1005 Maximum number of locks at any one time >>> 33 Maximum number of locks in any one bucket >>> 3987 Maximum number of locks stolen by for an empty partition >>> 75 Maximum number of locks stolen for any one partition >>> 309 Number of current lockers >>> 313 Maximum number of lockers at any one time >>> 121 Number of current lock objects >>> 676 Maximum number of lock objects at any one time >>> 3 Maximum number of lock objects in any one bucket >>> 69 Maximum number of objects stolen by for an empty partition >>> 5 Maximum number of objects stolen for any one partition >>> 22M Total number of locks requested (22009642) >>> 21M Total number of locks released (21992878) >>> 0 Total number of locks upgraded >>> 95 Total number of locks downgraded >>> 3348 Lock requests not available due to conflicts, for which we >>> waited >>> 12646 Lock requests not available due to conflicts, for which we >>> did not wait >>> 0 Number of deadlocks >>> 0 Lock timeout value >>> 0 Number of locks that have timed out >>> 0 Transaction timeout value >>> 0 Number of transactions that have timed out >>> 2MB 560KB Region size >>> 71719 The number of partition locks that required waiting (0%) >>> 23105 The maximum number of times any partition lock was waited for >>> (0%) >>> 0 The number of object queue operations that required waiting >>> (0%) >>> 163 The number of locker allocations that required waiting (0%) >>> 3172 The number of region locks that required waiting (0%) >>> 5 Maximum hash bucket length >>> =-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-= >>> Lock REGINFO information: >>> Environment Region type >>> 1 Region ID >>> /var/lib/dirsrv/slapd-SPRINKLR-COM/db/__db.001 Region name >>> 0x7f67ef869000 Region address >>> 0x7f67ef8690a0 Region allocation head >>> 0x7f67ef8712b0 Region primary address >>> 0 Region maximum allocation >>> 0 Region allocated >>> Region allocations: 1581345 allocations, 0 failures, 1580794 frees, 6 >>> longest >>> Allocations by power-of-two sizes: >>> 1KB 1581329 >>> 2KB 3 >>> 4KB 4 >>> 8KB 3 >>> 16KB 3 >>> 32KB 1 >>> 64KB 0 >>> 128KB 0 >>> 256KB 2 >>> 512KB 0 >>> 1024KB 1 >>> REGION_SHARED Region flags >>> =-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-= >>> Lock region parameters: >>> 2 Lock region region mutex [3172/14M 0% !Own] >>> 16381 locker table size >>> 8191 object table size >>> 34128 obj_off >>> 900216 locker_off >>> 0 need_dd >>> =-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-= >>> Lock conflict matrix: >>> =-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-= >>> Locks grouped by lockers: >>> Locker Mode Count Status ----------------- Object --------------- >>> 2 dd=308 locks held 1 write locks 0 pid/thread >>> 4845/140085720139840 flags 10 priority 100 >>> 2 READ 1 HELD userRoot/id2entry.db handle >>> 0 >>> 3 dd=307 locks held 0 write locks 0 pid/thread >>> 4845/140084829472512 flags 0 priority 100 >>> 4 dd=306 locks held 1 write locks 0 pid/thread >>> 4845/140085720139840 flags 10 priority 100 >>> 4 READ 1 HELD ipaca/id2entry.db handle >>> 0 >>> 5 dd=305 locks held 0 write locks 0 pid/thread >>> 4845/140084837865216 flags 0 priority 100 >>> 6 dd=304 locks held 1 write locks 0 pid/thread >>> 4845/140085720139840 flags 10 priority 100 >>> 6 READ 1 HELD ipaca/entryrdn.db handle >>> 0 >>> 7 dd=303 locks held 0 write locks 0 pid/thread >>> 4845/140084804294400 flags 0 priority 100 >>> 8 dd=302 locks held 0 write locks 0 pid/thread >>> 4845/140084720432896 flags 0 priority 100 >>> 9 dd=301 locks held 0 write locks 0 pid/thread >>> 4845/140084837865216 flags 0 priority 100 >>> a dd=300 locks held 1 write locks 0 pid/thread >>> 4845/140085720139840 flags 10 priority 100 >>> a READ 1 HELD ipaca/vlv#allcertspkitomcatindex.db >>> handle 0 >>> e dd=299 locks held 1 write locks 0 pid/thread >>> 4845/140085720139840 flags 10 priority 100 >>> e READ 1 HELD ipaca/vlv#allnonrevokedcertspkitomcatindex.db >>> handle 0 >>> 10 dd=298 locks held 1 write locks 0 pid/thread >>> 4845/140085720139840 flags 10 priority 100 >>> 10 READ 1 HELD ipaca/vlv#allrevokedcertspkitomcatindex.db >>> handle 0 >>> 11 dd=297 locks held 1 write locks 0 pid/thread >>> 4845/140085720139840 flags 10 priority 100 >>> 11 READ 1 HELD ipaca/vlv#allrevokedcertsnotafterpkitomcatindex.db >>> handle 0 >>> 14 dd=296 locks held 1 write locks 0 pid/thread >>> 4845/140085720139840 flags 10 priority 100 >>> 14 READ 1 HELD ipaca/vlv#allrevokedorrevokedexpiredcertspkitomcatindex.db >>> handle 0 >>> 15 dd=295 locks held 1 write locks 0 pid/thread >>> 4845/140085720139840 flags 10 priority 100 >>> 15 READ 1 HELD ipaca/vlv#allvalidcertspkitomcatindex.db >>> handle 0 >>> 16 dd=294 locks held 1 write locks 0 pid/thread >>> 4845/140085720139840 flags 10 priority 100 >>> 16 READ 1 HELD ipaca/vlv#allvalidcertsnotafterpkitomcatindex.db >>> handle 0 >>> 17 dd=293 locks held 1 write locks 0 pid/thread >>> 4845/140085720139840 flags 10 priority 100 >>> 17 READ 1 HELD ipaca/vlv#allvalidorrevokedcertspkitomcatindex.db >>> handle 0 >>> 18 dd=292 locks held 1 write locks 0 pid/thread >>> 4845/140085720139840 flags 10 priority 100 >>> 18 READ 1 HELD ipaca/vlv#caallpkitomcatindex.db >>> handle 0 >>> 1d dd=291 locks held 1 write locks 0 pid/thread >>> 4845/140085720139840 flags 10 priority 100 >>> 1d READ 1 HELD ipaca/vlv#cacompletepkitomcatindex.db >>> handle 0 >>> 1e dd=290 locks held 1 write locks 0 pid/thread >>> 4845/140085720139840 flags 10 priority 100 >>> 1e READ 1 HELD ipaca/vlv#cacompleteenrollmentpkitomcatindex.db >>> handle 0 >>> 20 dd=289 locks held 1 write locks 0 pid/thread >>> 4845/140085720139840 flags 10 priority 100 >>> 20 READ 1 HELD ipaca/vlv#cacompleterevocationpkitomcatindex.db >>> handle 0 >>> 21 dd=288 locks held 1 write locks 0 pid/thread >>> 4845/140085720139840 flags 10 priority 100 >>> 21 READ 1 HELD ipaca/vlv#caenrollmentpkitomcatindex.db >>> handle 0 >>> 2b dd=287 locks held 1 write locks 0 pid/thread >>> 4845/140085720139840 flags 10 priority 100 >>> 2b READ 1 HELD ipaca/vlv#carevocationpkitomcatindex.db >>> handle 0 >>> 2c dd=286 locks held 1 write locks 0 pid/thread >>> 4845/140085720139840 flags 10 priority 100 >>> 2c READ 1 HELD changelog/id2entry.db handle >>> 0 >>> 2d dd=285 locks held 0 write locks 0 pid/thread >>> 4845/140085719607040 flags 0 priority 100 >>> 2e dd=284 locks held 1 write locks 0 pid/thread >>> 4845/140085720139840 flags 10 priority 100 >>> 2e READ 1 HELD changelog/entryusn.db handle >>> 0 >>> 2f dd=283 locks held 0 write locks 0 pid/thread >>> 4845/140085720139840 flags 0 priority 100 >>> 30 dd=282 locks held 1 write locks 0 pid/thread >>> 4845/140085720139840 flags 10 priority 100 >>> 30 READ 1 HELD userRoot/entryusn.db handle >>> 0 >>> 31 dd=281 locks held 0 write locks 0 pid/thread >>> 4845/140085720139840 flags 0 priority 100 >>> 32 dd=280 locks held 1 write locks 0 pid/thread >>> 4845/140085720139840 flags 10 priority 100 >>> 32 READ 1 HELD ipaca/entryusn.db handle >>> 0 >>> 33 dd=279 locks held 0 write locks 0 pid/thread >>> 4845/140085720139840 flags 0 priority 100 >>> 34 dd=278 locks held 1 write locks 0 pid/thread >>> 4845/140085720139840 flags 10 priority 100 >>> 34 READ 1 HELD userRoot/entryrdn.db handle >>> 0 >>> 35 dd=277 locks held 0 write locks 0 pid/thread >>> 4845/140084544186112 flags 0 priority 100 >>> 36 dd=276 locks held 0 write locks 0 pid/thread >>> 4845/140084636505856 flags 0 priority 100 >>> 37 dd=275 locks held 0 write locks 0 pid/thread >>> 4845/140084854650624 flags 0 priority 100 >>> 38 dd=274 locks held 1 write locks 0 pid/thread >>> 4845/140085720139840 flags 10 priority 100 >>> 38 READ 1 HELD userRoot/objectclass.db handle >>> 0 >>> 39 dd=273 locks held 0 write locks 0 pid/thread >>> 4845/140084569364224 flags 0 priority 100 >>> 3a dd=272 locks held 1 write locks 0 pid/thread >>> 4845/140085720139840 flags 10 priority 100 >>> 3a READ 1 HELD userRoot/ancestorid.db handle >>> 0 >>> 3b dd=271 locks held 0 write locks 0 pid/thread >>> 4845/140084854650624 flags 0 priority 100 >>> 3c dd=270 locks held 1 write locks 0 pid/thread >>> 4845/140085720139840 flags 10 priority 100 >>> 3c READ 1 HELD changelog/entryrdn.db handle >>> 0 >>> 3d dd=269 locks held 0 write locks 0 pid/thread >>> 4845/140085719607040 flags 0 priority 100 >>> 3e dd=268 locks held 0 write locks 0 pid/thread >>> 4845/140084602935040 flags 0 priority 100 >>> 3f dd=267 locks held 0 write locks 0 pid/thread >>> 4845/140085719607040 flags 0 priority 100 >>> 40 dd=266 locks held 1 write locks 0 pid/thread >>> 4845/140085720139840 flags 10 priority 100 >>> 40 READ 1 HELD changelog/objectclass.db handle >>> 0 >>> 41 dd=265 locks held 0 write locks 0 pid/thread >>> 4845/140085719607040 flags 0 priority 100 >>> 42 dd=264 locks held 0 write locks 0 pid/thread >>> 4845/140085719607040 flags 0 priority 100 >>> 43 dd=263 locks held 0 write locks 0 pid/thread >>> 4845/140084653291264 flags 0 priority 100 >>> 44 dd=262 locks held 1 write locks 0 pid/thread >>> 4845/140085720139840 flags 10 priority 100 >>> 44 READ 1 HELD ipaca/objectclass.db handle >>> 0 >>> 45 dd=261 locks held 0 write locks 0 pid/thread >>> 4845/140084577756928 flags 0 priority 100 >>> 46 dd=260 locks held 0 write locks 0 pid/thread >>> 4845/140084720432896 flags 0 priority 100 >>> 47 dd=259 locks held 1 write locks 0 pid/thread >>> 4845/140085720139840 flags 10 priority 100 >>> 47 READ 1 HELD changelog/aci.db handle >>> 0 >>> 48 dd=258 locks held 0 write locks 0 pid/thread >>> 4845/140085720139840 flags 0 priority 100 >>> 49 dd=257 locks held 1 write locks 0 pid/thread >>> 4845/140085720139840 flags 10 priority 100 >>> 49 READ 1 HELD userRoot/aci.db handle >>> 0 >>> 4a dd=256 locks held 0 write locks 0 pid/thread >>> 4845/140085720139840 flags 0 priority 100 >>> 4b dd=255 locks held 0 write locks 0 pid/thread >>> 4845/140084535793408 flags 0 priority 100 >>> 4c dd=254 locks held 1 write locks 0 pid/thread >>> 4845/140085720139840 flags 10 priority 100 >>> 4c READ 1 HELD ipaca/aci.db handle >>> 0 >>> 4d dd=253 locks held 0 write locks 0 pid/thread >>> 4845/140085720139840 flags 0 priority 100 >>> 4e dd=252 locks held 1 write locks 0 pid/thread >>> 4845/140085720139840 flags 10 priority 100 >>> 4e READ 1 HELD userRoot/parentid.db handle >>> 0 >>> 4f dd=251 locks held 0 write locks 0 pid/thread >>> 4845/140085720139840 flags 0 priority 100 >>> 50 dd=250 locks held 0 write locks 0 pid/thread >>> 4845/140085719607040 flags 0 priority 100 >>> 51 dd=249 locks held 1 write locks 0 pid/thread >>> 4845/140085720139840 flags 10 priority 100 >>> 51 READ 1 HELD changelog/nsuniqueid.db handle >>> 0 >>> 52 dd=248 locks held 1 write locks 0 pid/thread >>> 4845/140085720139840 flags 10 priority 100 >>> 52 READ 1 HELD changelog/changenumber.db handle >>> 0 >>> 53 dd=247 locks held 0 write locks 0 pid/thread >>> 4845/140084602935040 flags 0 priority 100 >>> 54 dd=246 locks held 0 write locks 0 pid/thread >>> 4845/140085719607040 flags 0 priority 100 >>> 55 dd=245 locks held 1 write locks 0 pid/thread >>> 4845/140085719607040 flags 10 priority 100 >>> 55 READ 1 HELD changelog/targetuniqueid.db >>> handle 0 >>> 56 dd=244 locks held 1 write locks 0 pid/thread >>> 4845/140085719607040 flags 10 priority 100 >>> 56 READ 1 HELD changelog/parentid.db handle >>> 0 >>> 57 dd=243 locks held 1 write locks 0 pid/thread >>> 4845/140085719607040 flags 10 priority 100 >>> 57 READ 1 HELD changelog/ancestorid.db handle >>> 0 >>> 58 dd=242 locks held 1 write locks 0 pid/thread >>> 4845/140085719607040 flags 10 priority 100 >>> 58 READ 1 HELD changelog/numsubordinates.db >>> handle 0 >>> 59 dd=241 locks held 0 write locks 0 pid/thread >>> 4845/140085719607040 flags 0 priority 100 >>> 5a dd=240 locks held 0 write locks 0 pid/thread >>> 4845/140084544186112 flags 0 priority 100 >>> 5b dd=239 locks held 1 write locks 0 pid/thread >>> 4845/140084577756928 flags 10 priority 100 >>> 5b READ 1 HELD userRoot/krbPrincipalName.db >>> handle 0 >>> 5c dd=238 locks held 0 write locks 0 pid/thread >>> 4845/140084636505856 flags 0 priority 100 >>> 5d dd=237 locks held 0 write locks 0 pid/thread >>> 4845/140084695254784 flags 0 priority 100 >>> 5e dd=236 locks held 1 write locks 0 pid/thread >>> 4845/140084560971520 flags 10 priority 100 >>> 5e READ 1 HELD userRoot/ipakrbprincipalalias.db >>> handle 0 >>> 5f dd=235 locks held 0 write locks 0 pid/thread >>> 4845/140084703647488 flags 0 priority 100 >>> 60 dd=234 locks held 0 write locks 0 pid/thread >>> 4845/140084653291264 flags 0 priority 100 >>> 61 dd=233 locks held 0 write locks 0 pid/thread >>> 4845/140084577756928 flags 0 priority 100 >>> 62 dd=232 locks held 0 write locks 0 pid/thread >>> 4845/140084686862080 flags 0 priority 100 >>> 63 dd=231 locks held 1 write locks 0 pid/thread >>> 4845/140084611327744 flags 10 priority 100 >>> 63 READ 1 HELD changelog/seeAlso.db handle >>> 0 >>> 64 dd=230 locks held 0 write locks 0 pid/thread >>> 4845/140084686862080 flags 0 priority 100 >>> 65 dd=229 locks held 1 write locks 0 pid/thread >>> 4845/140084611327744 flags 10 priority 100 >>> 65 READ 1 HELD userRoot/seeAlso.db handle >>> 0 >>> 66 dd=228 locks held 0 write locks 0 pid/thread >>> 4845/140084636505856 flags 0 priority 100 >>> 67 dd=227 locks held 1 write locks 0 pid/thread >>> 4845/140084611327744 flags 10 priority 100 >>> 67 READ 1 HELD ipaca/seeAlso.db handle >>> 0 >>> 68 dd=226 locks held 0 write locks 0 pid/thread >>> 4845/140084686862080 flags 0 priority 100 >>> 69 dd=225 locks held 1 write locks 0 pid/thread >>> 4845/140084678469376 flags 10 priority 100 >>> 69 READ 1 HELD ipaca/parentid.db handle >>> 0 >>> 6a dd=224 locks held 0 write locks 0 pid/thread >>> 4845/140084678469376 flags 0 priority 100 >>> 6b dd=223 locks held 0 write locks 0 pid/thread >>> 4845/140084829472512 flags 0 priority 100 >>> 6c dd=222 locks held 0 write locks 0 pid/thread >>> 4845/140084653291264 flags 0 priority 100 >>> 6d dd=221 locks held 0 write locks 0 pid/thread >>> 4845/140084569364224 flags 0 priority 100 >>> 6e dd=220 locks held 0 write locks 0 pid/thread >>> 4845/140084636505856 flags 0 priority 100 >>> 6e READ 1 WAIT userRoot/objectclass.db page >>> 18 >>> 6f dd=219 locks held 0 write locks 0 pid/thread >>> 4845/140084544186112 flags 0 priority 100 >>> 70 dd=218 locks held 0 write locks 0 pid/thread >>> 4845/140084569364224 flags 0 priority 100 >>> 71 dd=217 locks held 0 write locks 0 pid/thread >>> 4845/140084619720448 flags 0 priority 100 >>> 72 dd=216 locks held 0 write locks 0 pid/thread >>> 4845/140084552578816 flags 0 priority 100 >>> 73 dd=215 locks held 0 write locks 0 pid/thread >>> 4845/140084829472512 flags 0 priority 100 >>> 74 dd=214 locks held 0 write locks 0 pid/thread >>> 4845/140084628113152 flags 0 priority 100 >>> 75 dd=213 locks held 0 write locks 0 pid/thread >>> 4845/140084686862080 flags 0 priority 100 >>> 76 dd=212 locks held 1 write locks 0 pid/thread >>> 4845/140084560971520 flags 10 priority 100 >>> 76 READ 1 HELD userRoot/ipauniqueid.db handle >>> 0 >>> 77 dd=211 locks held 0 write locks 0 pid/thread >>> 4845/140084661683968 flags 0 priority 100 >>> 78 dd=210 locks held 0 write locks 0 pid/thread >>> 4845/140084846257920 flags 0 priority 100 >>> 79 dd=209 locks held 0 write locks 0 pid/thread >>> 4845/140084628113152 flags 0 priority 100 >>> 7a dd=208 locks held 1 write locks 0 pid/thread >>> 4845/140084560971520 flags 10 priority 100 >>> 7a READ 1 HELD userRoot/nsuniqueid.db handle >>> 0 >>> 7b dd=207 locks held 1 write locks 0 pid/thread >>> 4845/140084560971520 flags 10 priority 100 >>> 7b READ 1 HELD userRoot/cn.db handle >>> 0 >>> 7c dd=206 locks held 1 write locks 0 pid/thread >>> 4845/140084560971520 flags 10 priority 100 >>> 7c READ 1 HELD userRoot/fqdn.db handle >>> 0 >>> 7d dd=205 locks held 1 write locks 0 pid/thread >>> 4845/140084560971520 flags 10 priority 100 >>> 7d READ 1 HELD userRoot/managedby.db handle >>> 0 >>> 7e dd=204 locks held 1 write locks 0 pid/thread >>> 4845/140084560971520 flags 10 priority 100 >>> 7e READ 1 HELD userRoot/numsubordinates.db >>> handle 0 >>> 7f dd=203 locks held 1 write locks 0 pid/thread >>> 4845/140084678469376 flags 10 priority 100 >>> 7f READ 1 HELD userRoot/member.db handle >>> 0 >>> 80 dd=202 locks held 0 write locks 0 pid/thread >>> 4845/140084712040192 flags 0 priority 100 >>> 81 dd=201 locks held 1 write locks 0 pid/thread >>> 4845/140084678469376 flags 10 priority 100 >>> 81 READ 1 HELD userRoot/memberUser.db handle >>> 0 >>> 82 dd=200 locks held 0 write locks 0 pid/thread >>> 4845/140084577756928 flags 0 priority 100 >>> 83 dd=199 locks held 1 write locks 0 pid/thread >>> 4845/140084678469376 flags 10 priority 100 >>> 83 READ 1 HELD userRoot/memberHost.db handle >>> 0 >>> 84 dd=198 locks held 0 write locks 0 pid/thread >>> 4845/140084712040192 flags 0 priority 100 >>> 85 dd=197 locks held 0 write locks 0 pid/thread >>> 4845/140084577756928 flags 0 priority 100 >>> 86 dd=196 locks held 0 write locks 0 pid/thread >>> 4845/140084670076672 flags 0 priority 100 >>> 87 dd=195 locks held 0 write locks 0 pid/thread >>> 4845/140084577756928 flags 0 priority 100 >>> 88 dd=194 locks held 0 write locks 0 pid/thread >>> 4845/140084846257920 flags 0 priority 100 >>> 89 dd=193 locks held 0 write locks 0 pid/thread >>> 4845/140084552578816 flags 0 priority 100 >>> 8a dd=192 locks held 0 write locks 0 pid/thread >>> 4845/140084703647488 flags 0 priority 100 >>> 8b dd=191 locks held 0 write locks 0 pid/thread >>> 4845/140084544186112 flags 0 priority 100 >>> 8c dd=190 locks held 0 write locks 0 pid/thread >>> 4845/140084846257920 flags 0 priority 100 >>> 8d dd=189 locks held 0 write locks 0 pid/thread >>> 4845/140084661683968 flags 0 priority 100 >>> 8e dd=188 locks held 1 write locks 0 pid/thread >>> 4845/140085067675392 flags 10 priority 100 >>> 8e READ 1 HELD userRoot/macAddress.db handle >>> 0 >>> 8f dd=187 locks held 0 write locks 0 pid/thread >>> 4845/140085067675392 flags 0 priority 100 >>> 90 dd=186 locks held 0 write locks 0 pid/thread >>> 4845/140084703647488 flags 0 priority 100 >>> 91 dd=185 locks held 0 write locks 0 pid/thread >>> 4845/140084552578816 flags 0 priority 100 >>> 92 dd=184 locks held 0 write locks 0 pid/thread >>> 4845/140084560971520 flags 0 priority 100 >>> 92 READ 1 WAIT userRoot/objectclass.db page >>> 9 >>> 93 dd=183 locks held 0 write locks 0 pid/thread >>> 4845/140084577756928 flags 0 priority 100 >>> 94 dd=182 locks held 0 write locks 0 pid/thread >>> 4845/140084821079808 flags 0 priority 100 >>> 94 READ 1 WAIT userRoot/objectclass.db page >>> 18 >>> 95 dd=181 locks held 0 write locks 0 pid/thread >>> 4845/140084544186112 flags 0 priority 100 >>> 96 dd=180 locks held 0 write locks 0 pid/thread >>> 4845/140084804294400 flags 0 priority 100 >>> 97 dd=179 locks held 1 write locks 0 pid/thread >>> 4845/140084712040192 flags 10 priority 100 >>> 97 READ 1 HELD userRoot/uid.db handle >>> 0 >>> 98 dd=178 locks held 0 write locks 0 pid/thread >>> 4845/140084552578816 flags 0 priority 100 >>> 99 dd=177 locks held 0 write locks 0 pid/thread >>> 4845/140084577756928 flags 0 priority 100 >>> 9a dd=176 locks held 0 write locks 0 pid/thread >>> 4845/140084602935040 flags 0 priority 100 >>> 9b dd=175 locks held 1 write locks 0 pid/thread >>> 4845/140085076068096 flags 10 priority 100 >>> 9b READ 1 HELD userRoot/uniquemember.db handle >>> 0 >>> 9c dd=174 locks held 1 write locks 0 pid/thread >>> 4845/140085076068096 flags 10 priority 100 >>> 9c READ 1 HELD userRoot/owner.db handle >>> 0 >>> 9d dd=173 locks held 1 write locks 0 pid/thread >>> 4845/140085076068096 flags 10 priority 100 >>> 9d READ 1 HELD userRoot/ipaallowedtarget.db >>> handle 0 >>> 9e dd=172 locks held 1 write locks 0 pid/thread >>> 4845/140085076068096 flags 10 priority 100 >>> 9e READ 1 HELD userRoot/ipaMemberCertProfile.db >>> handle 0 >>> 9f dd=171 locks held 1 write locks 0 pid/thread >>> 4845/140085076068096 flags 10 priority 100 >>> 9f READ 1 HELD userRoot/ipaassignedidview.db >>> handle 0 >>> a0 dd=170 locks held 1 write locks 0 pid/thread >>> 4845/140085076068096 flags 10 priority 100 >>> a0 READ 1 HELD userRoot/secretary.db handle >>> 0 >>> a1 dd=169 locks held 1 write locks 0 pid/thread >>> 4845/140085076068096 flags 10 priority 100 >>> a1 READ 1 HELD userRoot/memberdenycmd.db handle >>> 0 >>> a2 dd=168 locks held 1 write locks 0 pid/thread >>> 4845/140085076068096 flags 10 priority 100 >>> a2 READ 1 HELD userRoot/memberallowcmd.db >>> handle 0 >>> a3 dd=167 locks held 1 write locks 0 pid/thread >>> 4845/140085076068096 flags 10 priority 100 >>> a3 READ 1 HELD userRoot/manager.db handle >>> 0 >>> a4 dd=166 locks held 1 write locks 0 pid/thread >>> 4845/140085076068096 flags 10 priority 100 >>> a4 READ 1 HELD userRoot/ipasudorunas.db handle >>> 0 >>> a5 dd=165 locks held 1 write locks 0 pid/thread >>> 4845/140085076068096 flags 10 priority 100 >>> a5 READ 1 HELD userRoot/ipaMemberCa.db handle >>> 0 >>> a6 dd=164 locks held 1 write locks 0 pid/thread >>> 4845/140085076068096 flags 10 priority 100 >>> a6 READ 1 HELD userRoot/ipasudorunasgroup.db >>> handle 0 >>> a7 dd=163 locks held 1 write locks 0 pid/thread >>> 4845/140085076068096 flags 10 priority 100 >>> a7 READ 1 HELD userRoot/sourcehost.db handle >>> 0 >>> a8 dd=162 locks held 1 write locks 0 pid/thread >>> 4845/140085076068096 flags 10 priority 100 >>> a8 READ 1 HELD userRoot/memberservice.db handle >>> 0 >>> a9 dd=161 locks held 1 write locks 0 pid/thread >>> 4845/140085076068096 flags 10 priority 100 >>> a9 READ 1 HELD userRoot/ipatokenradiusconfiglink.db >>> handle 0 >>> aa dd=160 locks held 0 write locks 0 pid/thread >>> 4845/140084703647488 flags 0 priority 100 >>> ab dd=159 locks held 0 write locks 0 pid/thread >>> 4845/140084821079808 flags 0 priority 100 >>> ac dd=158 locks held 0 write locks 0 pid/thread >>> 4845/140084628113152 flags 0 priority 100 >>> ad dd=157 locks held 0 write locks 0 pid/thread >>> 4845/140084829472512 flags 0 priority 100 >>> ae dd=156 locks held 0 write locks 0 pid/thread >>> 4845/140084829472512 flags 0 priority 100 >>> af dd=155 locks held 0 write locks 0 pid/thread >>> 4845/140084854650624 flags 0 priority 100 >>> b0 dd=154 locks held 0 write locks 0 pid/thread >>> 4845/140084854650624 flags 0 priority 100 >>> b1 dd=153 locks held 0 write locks 0 pid/thread >>> 4845/140084837865216 flags 0 priority 100 >>> b2 dd=152 locks held 0 write locks 0 pid/thread >>> 4845/140084804294400 flags 0 priority 100 >>> b3 dd=151 locks held 0 write locks 0 pid/thread >>> 4845/140084812687104 flags 0 priority 100 >>> b4 dd=150 locks held 0 write locks 0 pid/thread >>> 4845/140084804294400 flags 0 priority 100 >>> b5 dd=149 locks held 0 write locks 0 pid/thread >>> 4845/140084837865216 flags 0 priority 100 >>> b6 dd=148 locks held 0 write locks 0 pid/thread >>> 4845/140084569364224 flags 0 priority 100 >>> b7 dd=147 locks held 0 write locks 0 pid/thread >>> 4845/140084829472512 flags 0 priority 100 >>> b8 dd=146 locks held 0 write locks 0 pid/thread >>> 4845/140084586149632 flags 0 priority 100 >>> b9 dd=145 locks held 0 write locks 0 pid/thread >>> 4845/140084653291264 flags 0 priority 100 >>> ba dd=144 locks held 0 write locks 0 pid/thread >>> 4845/140084636505856 flags 0 priority 100 >>> bb dd=143 locks held 0 write locks 0 pid/thread >>> 4845/140084804294400 flags 0 priority 100 >>> bc dd=142 locks held 0 write locks 0 pid/thread >>> 4845/140084804294400 flags 0 priority 100 >>> bd dd=141 locks held 0 write locks 0 pid/thread >>> 4845/140084695254784 flags 0 priority 100 >>> be dd=140 locks held 0 write locks 0 pid/thread >>> 4845/140084670076672 flags 0 priority 100 >>> bf dd=139 locks held 0 write locks 0 pid/thread >>> 4845/140084628113152 flags 0 priority 100 >>> c0 dd=138 locks held 0 write locks 0 pid/thread >>> 4845/140084661683968 flags 0 priority 100 >>> c1 dd=137 locks held 0 write locks 0 pid/thread >>> 4845/140084712040192 flags 0 priority 100 >>> c2 dd=136 locks held 0 write locks 0 pid/thread >>> 4845/140084661683968 flags 0 priority 100 >>> c3 dd=135 locks held 0 write locks 0 pid/thread >>> 4845/140084577756928 flags 0 priority 100 >>> c4 dd=134 locks held 0 write locks 0 pid/thread >>> 4845/140084854650624 flags 0 priority 100 >>> c5 dd=133 locks held 1 write locks 0 pid/thread >>> 4845/140084644898560 flags 10 priority 100 >>> c5 READ 1 HELD userRoot/uidnumber.db handle >>> 0 >>> c6 dd=132 locks held 0 write locks 0 pid/thread >>> 4845/140084594542336 flags 0 priority 100 >>> c8 dd=131 locks held 1 write locks 0 pid/thread >>> 4845/140084678469376 flags 10 priority 100 >>> c8 READ 1 HELD ipaca/certstatus.db handle >>> 0 >>> c9 dd=130 locks held 0 write locks 0 pid/thread >>> 4845/140084720432896 flags 0 priority 100 >>> ca dd=129 locks held 0 write locks 0 pid/thread >>> 4845/140084619720448 flags 0 priority 100 >>> cb dd=128 locks held 0 write locks 0 pid/thread >>> 4845/140084821079808 flags 0 priority 100 >>> cc dd=127 locks held 0 write locks 0 pid/thread >>> 4845/140084594542336 flags 0 priority 100 >>> cd dd=126 locks held 0 write locks 0 pid/thread >>> 4845/140084560971520 flags 0 priority 100 >>> ce dd=125 locks held 1 write locks 0 pid/thread >>> 4845/140084837865216 flags 10 priority 100 >>> ce READ 1 HELD userRoot/gidnumber.db handle >>> 0 >>> cf dd=124 locks held 0 write locks 0 pid/thread >>> 4845/140084720432896 flags 0 priority 100 >>> d0 dd=123 locks held 0 write locks 0 pid/thread >>> 4845/140084569364224 flags 0 priority 100 >>> d1 dd=122 locks held 1 write locks 0 pid/thread >>> 4845/140084678469376 flags 10 priority 100 >>> d1 READ 1 HELD userRoot/nscpEntryDN.db handle >>> 0 >>> d2 dd=121 locks held 0 write locks 0 pid/thread >>> 4845/140084695254784 flags 0 priority 100 >>> d3 dd=120 locks held 0 write locks 0 pid/thread >>> 4845/140084569364224 flags 0 priority 100 >>> d4 dd=119 locks held 0 write locks 0 pid/thread >>> 4845/140084653291264 flags 0 priority 100 >>> d5 dd=118 locks held 0 write locks 0 pid/thread >>> 4845/140084829472512 flags 0 priority 100 >>> d6 dd=117 locks held 0 write locks 0 pid/thread >>> 4845/140084661683968 flags 0 priority 100 >>> d7 dd=116 locks held 0 write locks 0 pid/thread >>> 4845/140084804294400 flags 0 priority 100 >>> d8 dd=115 locks held 1 write locks 0 pid/thread >>> 4845/140084821079808 flags 10 priority 100 >>> d8 READ 1 HELD ipaca/description.db handle >>> 0 >>> d9 dd=114 locks held 0 write locks 0 pid/thread >>> 4845/140084661683968 flags 0 priority 100 >>> da dd=113 locks held 0 write locks 0 pid/thread >>> 4845/140084837865216 flags 0 priority 100 >>> db dd=112 locks held 1 write locks 0 pid/thread >>> 4845/140084678469376 flags 10 priority 100 >>> db READ 1 HELD ipaca/cn.db handle >>> 0 >>> dc dd=111 locks held 0 write locks 0 pid/thread >>> 4845/140084577756928 flags 0 priority 100 >>> dd dd=110 locks held 1 write locks 0 pid/thread >>> 4845/140084678469376 flags 10 priority 100 >>> dd READ 1 HELD ipaca/ancestorid.db handle >>> 0 >>> de dd=109 locks held 0 write locks 0 pid/thread >>> 4845/140084577756928 flags 0 priority 100 >>> df dd=108 locks held 0 write locks 0 pid/thread >>> 4845/140084703647488 flags 0 priority 100 >>> e0 dd=107 locks held 0 write locks 0 pid/thread >>> 4845/140084569364224 flags 0 priority 100 >>> e1 dd=106 locks held 0 write locks 0 pid/thread >>> 4845/140084569364224 flags 0 priority 100 >>> e2 dd=105 locks held 0 write locks 0 pid/thread >>> 4845/140084821079808 flags 0 priority 100 >>> e3 dd=104 locks held 1 write locks 0 pid/thread >>> 4845/140084686862080 flags 10 priority 100 >>> e3 READ 1 HELD ipaca/serialno.db handle >>> 0 >>> e4 dd=103 locks held 0 write locks 0 pid/thread >>> 4845/140084837865216 flags 0 priority 100 >>> e5 dd=102 locks held 1 write locks 0 pid/thread >>> 4845/140084695254784 flags 10 priority 100 >>> e5 READ 1 HELD ipaca/revInfo.db handle >>> 0 >>> e6 dd=101 locks held 1 write locks 0 pid/thread >>> 4845/140084695254784 flags 10 priority 100 >>> e6 READ 1 HELD ipaca/revokedby.db handle >>> 0 >>> e7 dd=100 locks held 1 write locks 0 pid/thread >>> 4845/140084695254784 flags 10 priority 100 >>> e7 READ 1 HELD ipaca/revokedOn.db handle >>> 0 >>> e8 dd=99 locks held 0 write locks 0 pid/thread >>> 4845/140084829472512 flags 0 priority 100 >>> e9 dd=98 locks held 0 write locks 0 pid/thread >>> 4845/140084846257920 flags 0 priority 100 >>> ea dd=97 locks held 1 write locks 0 pid/thread >>> 4845/140084703647488 flags 10 priority 100 >>> ea READ 1 HELD ipaca/nsuniqueid.db handle >>> 0 >>> eb dd=96 locks held 1 write locks 0 pid/thread >>> 4845/140084703647488 flags 10 priority 100 >>> eb READ 1 HELD ipaca/requestid.db handle >>> 0 >>> ec dd=95 locks held 1 write locks 0 pid/thread >>> 4845/140084703647488 flags 10 priority 100 >>> ec READ 1 HELD ipaca/requeststate.db handle >>> 0 >>> ed dd=94 locks held 1 write locks 0 pid/thread >>> 4845/140084703647488 flags 10 priority 100 >>> ed READ 1 HELD ipaca/dateOfCreate.db handle >>> 0 >>> ee dd=93 locks held 1 write locks 0 pid/thread >>> 4845/140084703647488 flags 10 priority 100 >>> ee READ 1 HELD ipaca/requesttype.db handle >>> 0 >>> ef dd=92 locks held 1 write locks 0 pid/thread >>> 4845/140084703647488 flags 10 priority 100 >>> ef READ 1 HELD ipaca/numsubordinates.db handle >>> 0 >>> f0 dd=91 locks held 1 write locks 0 pid/thread >>> 4845/140084611327744 flags 10 priority 100 >>> f0 READ 1 HELD userRoot/userCertificate.db >>> handle 0 >>> f1 dd=90 locks held 0 write locks 0 pid/thread >>> 4845/140084821079808 flags 0 priority 100 >>> f2 dd=89 locks held 0 write locks 0 pid/thread >>> 4845/140084569364224 flags 0 priority 100 >>> f3 dd=88 locks held 0 write locks 0 pid/thread >>> 4845/140084602935040 flags 0 priority 100 >>> f4 dd=87 locks held 0 write locks 0 pid/thread >>> 4845/140084720432896 flags 0 priority 100 >>> f5 dd=86 locks held 0 write locks 0 pid/thread >>> 4845/140084628113152 flags 0 priority 100 >>> f6 dd=85 locks held 0 write locks 0 pid/thread >>> 4845/140084703647488 flags 0 priority 100 >>> f7 dd=84 locks held 0 write locks 0 pid/thread >>> 4845/140084611327744 flags 0 priority 100 >>> f8 dd=83 locks held 0 write locks 0 pid/thread >>> 4845/140084661683968 flags 0 priority 100 >>> f9 dd=82 locks held 0 write locks 0 pid/thread >>> 4845/140084837865216 flags 0 priority 100 >>> fa dd=81 locks held 0 write locks 0 pid/thread >>> 4845/140084544186112 flags 0 priority 100 >>> fb dd=80 locks held 0 write locks 0 pid/thread >>> 4845/140084703647488 flags 0 priority 100 >>> fc dd=79 locks held 0 write locks 0 pid/thread >>> 4845/140084544186112 flags 0 priority 100 >>> fd dd=78 locks held 0 write locks 0 pid/thread >>> 4845/140084535793408 flags 0 priority 100 >>> fe dd=77 locks held 0 write locks 0 pid/thread >>> 4845/140084586149632 flags 0 priority 100 >>> ff dd=76 locks held 0 write locks 0 pid/thread >>> 4845/140084569364224 flags 0 priority 100 >>> 100 dd=75 locks held 0 write locks 0 pid/thread >>> 4845/140084804294400 flags 0 priority 100 >>> 101 dd=74 locks held 0 write locks 0 pid/thread >>> 4845/140084703647488 flags 0 priority 100 >>> 102 dd=73 locks held 0 write locks 0 pid/thread >>> 4845/140084703647488 flags 0 priority 100 >>> 103 dd=72 locks held 0 write locks 0 pid/thread >>> 4845/140084821079808 flags 0 priority 100 >>> 104 dd=71 locks held 0 write locks 0 pid/thread >>> 4845/140084695254784 flags 0 priority 100 >>> 105 dd=70 locks held 0 write locks 0 pid/thread >>> 4845/140084544186112 flags 0 priority 100 >>> 106 dd=69 locks held 0 write locks 0 pid/thread >>> 4845/140084854650624 flags 0 priority 100 >>> 107 dd=68 locks held 0 write locks 0 pid/thread >>> 4845/140084628113152 flags 0 priority 100 >>> 108 dd=67 locks held 0 write locks 0 pid/thread >>> 4845/140084569364224 flags 0 priority 100 >>> 109 dd=66 locks held 0 write locks 0 pid/thread >>> 4845/140084594542336 flags 0 priority 100 >>> 10a dd=65 locks held 0 write locks 0 pid/thread >>> 4845/140084661683968 flags 0 priority 100 >>> 10b dd=64 locks held 0 write locks 0 pid/thread >>> 4845/140084653291264 flags 0 priority 100 >>> 10c dd=63 locks held 0 write locks 0 pid/thread >>> 4845/140084695254784 flags 0 priority 100 >>> 10d dd=62 locks held 0 write locks 0 pid/thread >>> 4845/140084720432896 flags 0 priority 100 >>> 10e dd=61 locks held 0 write locks 0 pid/thread >>> 4845/140084619720448 flags 0 priority 100 >>> 10f dd=60 locks held 0 write locks 0 pid/thread >>> 4845/140084720432896 flags 0 priority 100 >>> 110 dd=59 locks held 0 write locks 0 pid/thread >>> 4845/140084569364224 flags 0 priority 100 >>> 111 dd=58 locks held 0 write locks 0 pid/thread >>> 4845/140084695254784 flags 0 priority 100 >>> 112 dd=57 locks held 0 write locks 0 pid/thread >>> 4845/140084653291264 flags 0 priority 100 >>> 113 dd=56 locks held 0 write locks 0 pid/thread >>> 4845/140084712040192 flags 0 priority 100 >>> 114 dd=55 locks held 0 write locks 0 pid/thread >>> 4845/140084686862080 flags 0 priority 100 >>> 115 dd=54 locks held 0 write locks 0 pid/thread >>> 4845/140084577756928 flags 0 priority 100 >>> 115 READ 1 WAIT userRoot/objectclass.db page >>> 9 >>> 116 dd=53 locks held 0 write locks 0 pid/thread >>> 4845/140084837865216 flags 0 priority 100 >>> 116 READ 1 WAIT userRoot/objectclass.db page >>> 9 >>> 117 dd=52 locks held 0 write locks 0 pid/thread >>> 4845/140084577756928 flags 0 priority 100 >>> 118 dd=51 locks held 0 write locks 0 pid/thread >>> 4845/140084569364224 flags 0 priority 100 >>> 119 dd=50 locks held 0 write locks 0 pid/thread >>> 4845/140084560971520 flags 0 priority 100 >>> 11a dd=49 locks held 1 write locks 0 pid/thread >>> 4845/140084695254784 flags 10 priority 100 >>> 11a READ 1 HELD ipaca/metaInfo.db handle >>> 0 >>> 11b dd=48 locks held 1 write locks 0 pid/thread >>> 4845/140084695254784 flags 10 priority 100 >>> 11b READ 1 HELD ipaca/notbefore.db handle >>> 0 >>> 11c dd=47 locks held 1 write locks 0 pid/thread >>> 4845/140084695254784 flags 10 priority 100 >>> 11c READ 1 HELD ipaca/notafter.db handle >>> 0 >>> 11d dd=46 locks held 1 write locks 0 pid/thread >>> 4845/140084695254784 flags 10 priority 100 >>> 11d READ 1 HELD ipaca/duration.db handle >>> 0 >>> 11e dd=45 locks held 1 write locks 0 pid/thread >>> 4845/140084695254784 flags 10 priority 100 >>> 11e READ 1 HELD ipaca/subjectname.db handle >>> 0 >>> 11f dd=44 locks held 1 write locks 0 pid/thread >>> 4845/140084695254784 flags 10 priority 100 >>> 11f READ 1 HELD ipaca/publicKeyData.db handle >>> 0 >>> 120 dd=43 locks held 1 write locks 0 pid/thread >>> 4845/140084695254784 flags 10 priority 100 >>> 120 READ 1 HELD ipaca/extension.db handle >>> 0 >>> 121 dd=42 locks held 1 write locks 0 pid/thread >>> 4845/140084695254784 flags 10 priority 100 >>> 121 READ 1 HELD ipaca/issuedby.db handle >>> 0 >>> 122 dd=41 locks held 0 write locks 0 pid/thread >>> 4845/140084712040192 flags 0 priority 100 >>> 123 dd=40 locks held 0 write locks 0 pid/thread >>> 4845/140084812687104 flags 0 priority 100 >>> 124 dd=39 locks held 0 write locks 0 pid/thread >>> 4845/140084544186112 flags 0 priority 100 >>> 125 dd=38 locks held 0 write locks 0 pid/thread >>> 4845/140084686862080 flags 0 priority 100 >>> 126 dd=37 locks held 0 write locks 0 pid/thread >>> 4845/140084552578816 flags 0 priority 100 >>> 128 dd=36 locks held 0 write locks 0 pid/thread >>> 4845/140084686862080 flags 0 priority 100 >>> 12a dd=35 locks held 0 write locks 0 pid/thread >>> 4845/140084720432896 flags 0 priority 100 >>> 12b dd=34 locks held 0 write locks 0 pid/thread >>> 4845/140084552578816 flags 0 priority 100 >>> 12c dd=33 locks held 0 write locks 0 pid/thread >>> 4845/140084670076672 flags 0 priority 100 >>> 12d dd=32 locks held 0 write locks 0 pid/thread >>> 4845/140084577756928 flags 0 priority 100 >>> 12e dd=31 locks held 0 write locks 0 pid/thread >>> 4845/140084712040192 flags 0 priority 100 >>> 12f dd=30 locks held 0 write locks 0 pid/thread >>> 4845/140084670076672 flags 0 priority 100 >>> 130 dd=29 locks held 0 write locks 0 pid/thread >>> 4845/140084577756928 flags 0 priority 100 >>> 131 dd=28 locks held 0 write locks 0 pid/thread >>> 4845/140084560971520 flags 0 priority 100 >>> 132 dd=27 locks held 0 write locks 0 pid/thread >>> 4845/140084670076672 flags 0 priority 100 >>> 133 dd=26 locks held 0 write locks 0 pid/thread >>> 4845/140084602935040 flags 0 priority 100 >>> 134 dd=25 locks held 0 write locks 0 pid/thread >>> 4845/140084552578816 flags 0 priority 100 >>> 135 dd=24 locks held 0 write locks 0 pid/thread >>> 4845/140084678469376 flags 0 priority 100 >>> 136 dd=23 locks held 0 write locks 0 pid/thread >>> 4845/140084544186112 flags 0 priority 100 >>> 137 dd=22 locks held 0 write locks 0 pid/thread >>> 4845/140084577756928 flags 0 priority 100 >>> 138 dd=21 locks held 0 write locks 0 pid/thread >>> 4845/140084703647488 flags 0 priority 100 >>> 139 dd=20 locks held 0 write locks 0 pid/thread >>> 4845/140084678469376 flags 0 priority 100 >>> 13b dd=19 locks held 0 write locks 0 pid/thread >>> 4845/140084812687104 flags 0 priority 100 >>> 144 dd=18 locks held 0 write locks 0 pid/thread >>> 4845/140084670076672 flags 0 priority 100 >>> 156 dd=17 locks held 0 write locks 0 pid/thread >>> 4845/140084594542336 flags 0 priority 100 >>> 15a dd=16 locks held 0 write locks 0 pid/thread >>> 4845/140084577756928 flags 0 priority 100 >>> 168 dd=15 locks held 0 write locks 0 pid/thread >>> 4845/140084653291264 flags 0 priority 100 >>> 16a dd=14 locks held 0 write locks 0 pid/thread >>> 4845/140084829472512 flags 0 priority 100 >>> 17d dd=13 locks held 0 write locks 0 pid/thread >>> 4845/140084829472512 flags 0 priority 100 >>> 17e dd=12 locks held 0 write locks 0 pid/thread >>> 4845/140084829472512 flags 0 priority 100 >>> 17f dd=11 locks held 0 write locks 0 pid/thread >>> 4845/140084586149632 flags 0 priority 100 >>> 180 dd=10 locks held 0 write locks 0 pid/thread >>> 4845/140084594542336 flags 0 priority 100 >>> 181 dd= 9 locks held 0 write locks 0 pid/thread >>> 4845/140084577756928 flags 0 priority 100 >>> 193 dd= 8 locks held 0 write locks 0 pid/thread >>> 4845/140084812687104 flags 0 priority 100 >>> 196 dd= 7 locks held 0 write locks 0 pid/thread >>> 4845/140084695254784 flags 0 priority 100 >>> 1a1 dd= 6 locks held 0 write locks 0 pid/thread >>> 4845/140084720432896 flags 0 priority 100 >>> 1a2 dd= 5 locks held 0 write locks 0 pid/thread >>> 4845/140084577756928 flags 0 priority 100 >>> 1a3 dd= 4 locks held 0 write locks 0 pid/thread >>> 4845/140084812687104 flags 0 priority 100 >>> 1a4 dd= 3 locks held 0 write locks 0 pid/thread >>> 4845/140084812687104 flags 0 priority 100 >>> 1a5 dd= 2 locks held 0 write locks 0 pid/thread >>> 4845/140084577756928 flags 0 priority 100 >>> 1a6 dd= 1 locks held 0 write locks 0 pid/thread >>> 4845/140084594542336 flags 0 priority 100 >>> 800b3a33 dd= 0 locks held 26 write locks 22 pid/thread >>> 4845/140084661683968 flags 0 priority 100 >>> 800b3a33 WRITE 1 HELD userRoot/objectclass.db page >>> 9 >>> 800b3a33 WRITE 1 HELD userRoot/objectclass.db page >>> 8 >>> 800b3a33 WRITE 1 HELD userRoot/objectclass.db page >>> 18 >>> 800b3a33 WRITE 1 HELD userRoot/objectclass.db page >>> 7 >>> 800b3a33 WRITE 1 HELD userRoot/objectclass.db page >>> 14 >>> 800b3a33 WRITE 1 HELD userRoot/objectclass.db page >>> 11 >>> 800b3a33 WRITE 2 HELD userRoot/objectclass.db page >>> 3 >>> 800b3a33 WRITE 1 HELD userRoot/cn.db page >>> 150 >>> 800b3a33 WRITE 1 HELD userRoot/cn.db page >>> 89 >>> 800b3a33 WRITE 1 HELD userRoot/cn.db page >>> 53 >>> 800b3a33 WRITE 1 HELD userRoot/cn.db page >>> 131 >>> 800b3a33 WRITE 1 HELD userRoot/cn.db page >>> 117 >>> 800b3a33 WRITE 1 HELD userRoot/cn.db page >>> 11 >>> 800b3a33 WRITE 2 HELD userRoot/cn.db page >>> 141 >>> 800b3a33 WRITE 1 HELD userRoot/cn.db page >>> 41 >>> 800b3a33 WRITE 1 HELD userRoot/cn.db page >>> 40 >>> 800b3a33 WRITE 1 HELD userRoot/cn.db page >>> 143 >>> 800b3a33 WRITE 1 HELD userRoot/cn.db page >>> 30 >>> 800b3a33 WRITE 2 HELD userRoot/id2entry.db page >>> 8184 >>> 800b3a33 WRITE 1 HELD userRoot/id2entry.db page >>> 8183 >>> 800b3a33 WRITE 1 HELD userRoot/id2entry.db page >>> 7779 >>> 800b3a33 WRITE 1 HELD userRoot/id2entry.db page >>> 0 >>> 800b3a33 READ 3 HELD userRoot/entryrdn.db page >>> 3 >>> 800b3a33 READ 1 HELD userRoot/entryrdn.db page >>> 36 >>> 800b3a33 READ 1 HELD userRoot/entryrdn.db page >>> 4 >>> 800b3a33 READ 1 HELD userRoot/nsuniqueid.db page >>> 86 >>> =-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-= >>> Locks grouped by object: >>> Locker Mode Count Status ----------------- Object --------------- >>> a4 READ 1 HELD userRoot/ipasudorunas.db handle >>> 0 >>> >>> 9e READ 1 HELD userRoot/ipaMemberCertProfile.db >>> handle 0 >>> >>> 10 READ 1 HELD ipaca/vlv#allrevokedcertspkitomcatindex.db >>> handle 0 >>> >>> 800b3a33 WRITE 2 HELD userRoot/id2entry.db page >>> 8184 >>> >>> 800b3a33 WRITE 1 HELD userRoot/id2entry.db page >>> 8183 >>> >>> 800b3a33 WRITE 1 HELD userRoot/id2entry.db page >>> 7779 >>> >>> a1 READ 1 HELD userRoot/memberdenycmd.db handle >>> 0 >>> >>> a READ 1 HELD ipaca/vlv#allcertspkitomcatindex.db >>> handle 0 >>> >>> a7 READ 1 HELD userRoot/sourcehost.db handle >>> 0 >>> >>> 57 READ 1 HELD changelog/ancestorid.db handle >>> 0 >>> >>> e5 READ 1 HELD ipaca/revInfo.db handle >>> 0 >>> >>> e6 READ 1 HELD ipaca/revokedby.db handle >>> 0 >>> >>> 11 READ 1 HELD ipaca/vlv#allrevokedcertsnotafterpkitomcatindex.db >>> handle 0 >>> >>> 4e READ 1 HELD userRoot/parentid.db handle >>> 0 >>> >>> 4 READ 1 HELD ipaca/id2entry.db handle >>> 0 >>> >>> 58 READ 1 HELD changelog/numsubordinates.db >>> handle 0 >>> >>> 67 READ 1 HELD ipaca/seeAlso.db handle >>> 0 >>> >>> 2b READ 1 HELD ipaca/vlv#carevocationpkitomcatindex.db >>> handle 0 >>> >>> ec READ 1 HELD ipaca/requeststate.db handle >>> 0 >>> >>> 11b READ 1 HELD ipaca/notbefore.db handle >>> 0 >>> >>> 7e READ 1 HELD userRoot/numsubordinates.db >>> handle 0 >>> >>> a2 READ 1 HELD userRoot/memberallowcmd.db >>> handle 0 >>> >>> 9c READ 1 HELD userRoot/owner.db handle >>> 0 >>> >>> 1e READ 1 HELD ipaca/vlv#cacompleteenrollmentpkitomcatindex.db >>> handle 0 >>> >>> 11c READ 1 HELD ipaca/notafter.db handle >>> 0 >>> >>> 40 READ 1 HELD changelog/objectclass.db handle >>> 0 >>> >>> 2e READ 1 HELD changelog/entryusn.db handle >>> 0 >>> >>> 16 READ 1 HELD ipaca/vlv#allvalidcertsnotafterpkitomcatindex.db >>> handle 0 >>> >>> 15 READ 1 HELD ipaca/vlv#allvalidcertspkitomcatindex.db >>> handle 0 >>> >>> a5 READ 1 HELD userRoot/ipaMemberCa.db handle >>> 0 >>> >>> 18 READ 1 HELD ipaca/vlv#caallpkitomcatindex.db >>> handle 0 >>> >>> a0 READ 1 HELD userRoot/secretary.db handle >>> 0 >>> >>> 81 READ 1 HELD userRoot/memberUser.db handle >>> 0 >>> >>> 9f READ 1 HELD userRoot/ipaassignedidview.db >>> handle 0 >>> >>> a3 READ 1 HELD userRoot/manager.db handle >>> 0 >>> >>> d8 READ 1 HELD ipaca/description.db handle >>> 0 >>> >>> a8 READ 1 HELD userRoot/memberservice.db handle >>> 0 >>> >>> 44 READ 1 HELD ipaca/objectclass.db handle >>> 0 >>> >>> ed READ 1 HELD ipaca/dateOfCreate.db handle >>> 0 >>> >>> 800b3a33 WRITE 1 HELD userRoot/id2entry.db page >>> 0 >>> >>> 2 READ 1 HELD userRoot/id2entry.db handle >>> 0 >>> >>> 5b READ 1 HELD userRoot/krbPrincipalName.db >>> handle 0 >>> >>> ee READ 1 HELD ipaca/requesttype.db handle >>> 0 >>> >>> c8 READ 1 HELD ipaca/certstatus.db handle >>> 0 >>> >>> 56 READ 1 HELD changelog/parentid.db handle >>> 0 >>> >>> 83 READ 1 HELD userRoot/memberHost.db handle >>> 0 >>> >>> 5e READ 1 HELD userRoot/ipakrbprincipalalias.db >>> handle 0 >>> >>> 121 READ 1 HELD ipaca/issuedby.db handle >>> 0 >>> >>> 6 READ 1 HELD ipaca/entryrdn.db handle >>> 0 >>> >>> e7 READ 1 HELD ipaca/revokedOn.db handle >>> 0 >>> >>> 4c READ 1 HELD ipaca/aci.db handle >>> 0 >>> >>> 11d READ 1 HELD ipaca/duration.db handle >>> 0 >>> >>> 14 READ 1 HELD ipaca/vlv#allrevokedorrevokedexpiredcertspkitomcatindex.db >>> handle 0 >>> >>> 21 READ 1 HELD ipaca/vlv#caenrollmentpkitomcatindex.db >>> handle 0 >>> >>> 55 READ 1 HELD changelog/targetuniqueid.db >>> handle 0 >>> >>> 800b3a33 READ 1 HELD userRoot/entryrdn.db page >>> 36 >>> >>> 2c READ 1 HELD changelog/id2entry.db handle >>> 0 >>> >>> 800b3a33 READ 1 HELD userRoot/entryrdn.db page >>> 4 >>> >>> 34 READ 1 HELD userRoot/entryrdn.db handle >>> 0 >>> >>> 800b3a33 READ 3 HELD userRoot/entryrdn.db page >>> 3 >>> >>> eb READ 1 HELD ipaca/requestid.db handle >>> 0 >>> >>> 49 READ 1 HELD userRoot/aci.db handle >>> 0 >>> >>> 76 READ 1 HELD userRoot/ipauniqueid.db handle >>> 0 >>> >>> 63 READ 1 HELD changelog/seeAlso.db handle >>> 0 >>> >>> 7c READ 1 HELD userRoot/fqdn.db handle >>> 0 >>> >>> db READ 1 HELD ipaca/cn.db handle >>> 0 >>> >>> ef READ 1 HELD ipaca/numsubordinates.db handle >>> 0 >>> >>> 32 READ 1 HELD ipaca/entryusn.db handle >>> 0 >>> >>> 1d READ 1 HELD ipaca/vlv#cacompletepkitomcatindex.db >>> handle 0 >>> >>> dd READ 1 HELD ipaca/ancestorid.db handle >>> 0 >>> >>> a9 READ 1 HELD userRoot/ipatokenradiusconfiglink.db >>> handle 0 >>> >>> f0 READ 1 HELD userRoot/userCertificate.db >>> handle 0 >>> >>> 11a READ 1 HELD ipaca/metaInfo.db handle >>> 0 >>> >>> 7d READ 1 HELD userRoot/managedby.db handle >>> 0 >>> >>> ea READ 1 HELD ipaca/nsuniqueid.db handle >>> 0 >>> >>> c5 READ 1 HELD userRoot/uidnumber.db handle >>> 0 >>> >>> d1 READ 1 HELD userRoot/nscpEntryDN.db handle >>> 0 >>> >>> 120 READ 1 HELD ipaca/extension.db handle >>> 0 >>> >>> 800b3a33 WRITE 1 HELD userRoot/cn.db page >>> 150 >>> >>> 69 READ 1 HELD ipaca/parentid.db handle >>> 0 >>> >>> 800b3a33 WRITE 1 HELD userRoot/cn.db page >>> 131 >>> >>> 800b3a33 WRITE 1 HELD userRoot/cn.db page >>> 143 >>> >>> 800b3a33 WRITE 2 HELD userRoot/cn.db page >>> 141 >>> >>> 20 READ 1 HELD ipaca/vlv#cacompleterevocationpkitomcatindex.db >>> handle 0 >>> >>> 8e READ 1 HELD userRoot/macAddress.db handle >>> 0 >>> >>> e READ 1 HELD ipaca/vlv#allnonrevokedcertspkitomcatindex.db >>> handle 0 >>> >>> 17 READ 1 HELD ipaca/vlv#allvalidorrevokedcertspkitomcatindex.db >>> handle 0 >>> >>> 800b3a33 WRITE 1 HELD userRoot/cn.db page >>> 30 >>> >>> 800b3a33 WRITE 1 HELD userRoot/objectclass.db page >>> 14 >>> >>> 800b3a33 WRITE 1 HELD userRoot/objectclass.db page >>> 9 >>> 116 READ 1 WAIT userRoot/objectclass.db page >>> 9 >>> 115 READ 1 WAIT userRoot/objectclass.db page >>> 9 >>> 92 READ 1 WAIT userRoot/objectclass.db page >>> 9 >>> >>> 800b3a33 WRITE 1 HELD userRoot/objectclass.db page >>> 8 >>> >>> 7b READ 1 HELD userRoot/cn.db handle >>> 0 >>> >>> 800b3a33 WRITE 1 HELD userRoot/objectclass.db page >>> 11 >>> >>> 800b3a33 WRITE 1 HELD userRoot/objectclass.db page >>> 7 >>> >>> 800b3a33 WRITE 1 HELD userRoot/cn.db page >>> 11 >>> >>> 38 READ 1 HELD userRoot/objectclass.db handle >>> 0 >>> >>> 800b3a33 WRITE 2 HELD userRoot/objectclass.db page >>> 3 >>> >>> 800b3a33 WRITE 1 HELD userRoot/cn.db page >>> 53 >>> >>> 800b3a33 WRITE 1 HELD userRoot/objectclass.db page >>> 18 >>> 6e READ 1 WAIT userRoot/objectclass.db page >>> 18 >>> 94 READ 1 WAIT userRoot/objectclass.db page >>> 18 >>> >>> 800b3a33 WRITE 1 HELD userRoot/cn.db page >>> 41 >>> >>> 800b3a33 WRITE 1 HELD userRoot/cn.db page >>> 40 >>> >>> 800b3a33 WRITE 1 HELD userRoot/cn.db page >>> 89 >>> >>> 800b3a33 WRITE 1 HELD userRoot/cn.db page >>> 117 >>> >>> 30 READ 1 HELD userRoot/entryusn.db handle >>> 0 >>> >>> 3c READ 1 HELD changelog/entryrdn.db handle >>> 0 >>> >>> 97 READ 1 HELD userRoot/uid.db handle >>> 0 >>> >>> 47 READ 1 HELD changelog/aci.db handle >>> 0 >>> >>> e3 READ 1 HELD ipaca/serialno.db handle >>> 0 >>> >>> 7a READ 1 HELD userRoot/nsuniqueid.db handle >>> 0 >>> >>> 800b3a33 READ 1 HELD userRoot/nsuniqueid.db page >>> 86 >>> >>> 51 READ 1 HELD changelog/nsuniqueid.db handle >>> 0 >>> >>> 11e READ 1 HELD ipaca/subjectname.db handle >>> 0 >>> >>> 7f READ 1 HELD userRoot/member.db handle >>> 0 >>> >>> 11f READ 1 HELD ipaca/publicKeyData.db handle >>> 0 >>> >>> 9b READ 1 HELD userRoot/uniquemember.db handle >>> 0 >>> >>> 52 READ 1 HELD changelog/changenumber.db handle >>> 0 >>> >>> ce READ 1 HELD userRoot/gidnumber.db handle >>> 0 >>> >>> 65 READ 1 HELD userRoot/seeAlso.db handle >>> 0 >>> >>> 9d READ 1 HELD userRoot/ipaallowedtarget.db >>> handle 0 >>> >>> a6 READ 1 HELD userRoot/ipasudorunasgroup.db >>> handle 0 >>> >>> 3a READ 1 HELD userRoot/ancestorid.db handle >>> 0 >>> >>> >>> >>> >>> pstack output >>> >>> root at ipa-master ]# pstack 4845 >>> Thread 44 (Thread 0x7f681dd3a700 (LWP 4847)): >>> #0 0x00007f683cd719b3 in select () from /lib64/libc.so.6 >>> #1 0x00007f683f4fc0e9 in DS_Sleep () from /usr/lib64/dirsrv/libslapd.so. >>> 0 >>> #2 0x00007f6832b4a907 in deadlock_threadmain () from >>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>> #3 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so >>> #4 0x00007f683d04cdc5 in start_thread () from /lib64/libpthread.so.0 >>> #5 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 >>> Thread 43 (Thread 0x7f681d539700 (LWP 4848)): >>> #0 0x00007f683cd719b3 in select () from /lib64/libc.so.6 >>> #1 0x00007f68376d8bad in __os_yield () from /lib64/libdb-5.3.so >>> #2 0x00007f68376d42b3 in __memp_sync_int () from /lib64/libdb-5.3.so >>> #3 0x00007f68376e4752 in __txn_checkpoint () from /lib64/libdb-5.3.so >>> #4 0x00007f68376e4b74 in __txn_checkpoint_pp () from /lib64/ >>> libdb-5.3.so >>> #5 0x00007f6832b4ea87 in checkpoint_threadmain () from >>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>> #6 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so >>> #7 0x00007f683d04cdc5 in start_thread () from /lib64/libpthread.so.0 >>> #8 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 >>> Thread 42 (Thread 0x7f681cd38700 (LWP 4849)): >>> #0 0x00007f683cd719b3 in select () from /lib64/libc.so.6 >>> #1 0x00007f683f4fc0e9 in DS_Sleep () from /usr/lib64/dirsrv/libslapd.so. >>> 0 >>> #2 0x00007f6832b4ab7f in trickle_threadmain () from >>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>> #3 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so >>> #4 0x00007f683d04cdc5 in start_thread () from /lib64/libpthread.so.0 >>> #5 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 >>> Thread 41 (Thread 0x7f681c537700 (LWP 4850)): >>> #0 0x00007f683cd719b3 in select () from /lib64/libc.so.6 >>> #1 0x00007f683f4fc0e9 in DS_Sleep () from /usr/lib64/dirsrv/libslapd.so. >>> 0 >>> #2 0x00007f6832b45707 in perf_threadmain () from >>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>> #3 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so >>> #4 0x00007f683d04cdc5 in start_thread () from /lib64/libpthread.so.0 >>> #5 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 >>> Thread 40 (Thread 0x7f681bb2d700 (LWP 4851)): >>> #0 0x00007f683d0506d5 in >>> pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 >>> #1 0x00007f683d6a6050 in PR_WaitCondVar () from /lib64/libnspr4.so >>> #2 0x00007f683f4eb198 in slapi_wait_condvar () from >>> /usr/lib64/dirsrv/libslapd.so.0 >>> #3 0x00007f683587d62e in cos_cache_wait_on_change () from >>> /usr/lib64/dirsrv/plugins/libcos-plugin.so >>> #4 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so >>> #5 0x00007f683d04cdc5 in start_thread () from /lib64/libpthread.so.0 >>> #6 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 >>> Thread 39 (Thread 0x7f681b32c700 (LWP 4852)): >>> #0 0x00007f683d0506d5 in >>> pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 >>> #1 0x00007f683d6a6050 in PR_WaitCondVar () from /lib64/libnspr4.so >>> #2 0x00007f683f4eb198 in slapi_wait_condvar () from >>> /usr/lib64/dirsrv/libslapd.so.0 >>> #3 0x00007f68311e8eed in roles_cache_wait_on_change () from >>> /usr/lib64/dirsrv/plugins/libroles-plugin.so >>> #4 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so >>> #5 0x00007f683d04cdc5 in start_thread () from /lib64/libpthread.so.0 >>> #6 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 >>> Thread 38 (Thread 0x7f681ab2b700 (LWP 4853)): >>> #0 0x00007f683d0506d5 in >>> pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 >>> #1 0x00007f683d6a6050 in PR_WaitCondVar () from /lib64/libnspr4.so >>> #2 0x00007f683f4eb198 in slapi_wait_condvar () from >>> /usr/lib64/dirsrv/libslapd.so.0 >>> #3 0x00007f68311e8eed in roles_cache_wait_on_change () from >>> /usr/lib64/dirsrv/plugins/libroles-plugin.so >>> #4 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so >>> #5 0x00007f683d04cdc5 in start_thread () from /lib64/libpthread.so.0 >>> #6 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 >>> Thread 37 (Thread 0x7f681a32a700 (LWP 4854)): >>> #0 0x00007f683d0506d5 in >>> pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 >>> #1 0x00007f683d6a6050 in PR_WaitCondVar () from /lib64/libnspr4.so >>> #2 0x00007f683f4eb198 in slapi_wait_condvar () from >>> /usr/lib64/dirsrv/libslapd.so.0 >>> #3 0x00007f68311e8eed in roles_cache_wait_on_change () from >>> /usr/lib64/dirsrv/plugins/libroles-plugin.so >>> #4 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so >>> #5 0x00007f683d04cdc5 in start_thread () from /lib64/libpthread.so.0 >>> #6 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 >>> Thread 36 (Thread 0x7f6819b29700 (LWP 4855)): >>> #0 0x00007f683d050a82 in pthread_cond_timedwait@@GLIBC_2.3.2 () from >>> /lib64/libpthread.so.0 >>> #1 0x00007f683d6a5b07 in pt_TimedWait () from /lib64/libnspr4.so >>> #2 0x00007f683d6a5fce in PR_WaitCondVar () from /lib64/libnspr4.so >>> #3 0x00007f683f9a4d03 in housecleaning () >>> #4 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so >>> #5 0x00007f683d04cdc5 in start_thread () from /lib64/libpthread.so.0 >>> #6 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 >>> Thread 35 (Thread 0x7f6819328700 (LWP 4856)): >>> #0 0x00007f683d050a82 in pthread_cond_timedwait@@GLIBC_2.3.2 () from >>> /lib64/libpthread.so.0 >>> #1 0x00007f683d6a5b07 in pt_TimedWait () from /lib64/libnspr4.so >>> #2 0x00007f683d6a5fce in PR_WaitCondVar () from /lib64/libnspr4.so >>> #3 0x00007f683f4911e8 in eq_loop () from /usr/lib64/dirsrv/libslapd.so. >>> 0 >>> #4 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so >>> #5 0x00007f683d04cdc5 in start_thread () from /lib64/libpthread.so.0 >>> #6 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 >>> Thread 34 (Thread 0x7f680bfff700 (LWP 4859)): >>> #0 0x00007f683d0506d5 in >>> pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 >>> #1 0x00007f683d6a6050 in PR_WaitCondVar () from /lib64/libnspr4.so >>> #2 0x00007f683f999fee in connection_wait_for_new_work () >>> #3 0x00007f683f99b719 in connection_threadmain () >>> #4 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so >>> #5 0x00007f683d04cdc5 in start_thread () from /lib64/libpthread.so.0 >>> #6 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 >>> Thread 33 (Thread 0x7f680b7fe700 (LWP 4860)): >>> #0 0x00007f683d0506d5 in >>> pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 >>> #1 0x00007f683d6a6050 in PR_WaitCondVar () from /lib64/libnspr4.so >>> #2 0x00007f683f999fee in connection_wait_for_new_work () >>> #3 0x00007f683f99b719 in connection_threadmain () >>> #4 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so >>> #5 0x00007f683d04cdc5 in start_thread () from /lib64/libpthread.so.0 >>> #6 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 >>> Thread 32 (Thread 0x7f680affd700 (LWP 4861)): >>> #0 0x00007f683d0506d5 in >>> pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 >>> #1 0x00007f68375ac2f3 in __db_hybrid_mutex_suspend () from /lib64/ >>> libdb-5.3.so >>> #2 0x00007f68375ab640 in __db_tas_mutex_lock () from /lib64/ >>> libdb-5.3.so >>> #3 0x00007f6837655d3a in __lock_get_internal () from /lib64/ >>> libdb-5.3.so >>> #4 0x00007f6837656820 in __lock_get () from /lib64/libdb-5.3.so >>> #5 0x00007f6837682142 in __db_lget () from /lib64/libdb-5.3.so >>> #6 0x00007f68375c9605 in __bam_search () from /lib64/libdb-5.3.so >>> #7 0x00007f68375b4256 in __bamc_search () from /lib64/libdb-5.3.so >>> #8 0x00007f68375b5d0f in __bamc_get () from /lib64/libdb-5.3.so >>> #9 0x00007f683766eca6 in __dbc_iget () from /lib64/libdb-5.3.so >>> #10 0x00007f683767db02 in __dbc_get_pp () from /lib64/libdb-5.3.so >>> #11 0x00007f6832b59680 in idl_new_fetch () from >>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>> #12 0x00007f6832b67b26 in index_read_ext_allids () from >>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>> #13 0x00007f6832b52284 in keys2idl () from /usr/lib64/dirsrv/plugins/libb >>> ack-ldbm.so >>> #14 0x00007f6832b529e3 in ava_candidates.isra.0 () from >>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>> #15 0x00007f6832b52fd2 in filter_candidates_ext () from >>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>> #16 0x00007f6832b54096 in list_candidates () from >>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>> #17 0x00007f6832b52f40 in filter_candidates_ext () from >>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>> #18 0x00007f6832b54096 in list_candidates () from >>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>> #19 0x00007f6832b52f40 in filter_candidates_ext () from >>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>> #20 0x00007f6832b54096 in list_candidates () from >>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>> #21 0x00007f6832b52f40 in filter_candidates_ext () from >>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>> #22 0x00007f6832b8e80c in subtree_candidates () from >>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>> #23 0x00007f6832b8fe7f in ldbm_back_search () from >>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>> #24 0x00007f683f4bd1d6 in op_shared_search () from >>> /usr/lib64/dirsrv/libslapd.so.0 >>> #25 0x00007f683f9ae33e in do_search () >>> #26 0x00007f683f99c3c3 in connection_threadmain () >>> #27 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so >>> #28 0x00007f683d04cdc5 in start_thread () from /lib64/libpthread.so.0 >>> #29 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 >>> Thread 31 (Thread 0x7f680a7fc700 (LWP 4862)): >>> #0 0x00007f683d0506d5 in >>> pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 >>> #1 0x00007f683d6a62c3 in PR_EnterMonitor () from /lib64/libnspr4.so >>> #2 0x00007f6832b4d3e6 in dblayer_txn_begin () from >>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>> #3 0x00007f6832b89007 in ldbm_back_modify () from >>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>> #4 0x00007f683f4b6541 in op_shared_modify () from >>> /usr/lib64/dirsrv/libslapd.so.0 >>> #5 0x00007f683f4b7004 in modify_internal_pb () from >>> /usr/lib64/dirsrv/libslapd.so.0 >>> #6 0x00007f683483bbbb in ipalockout_postop () from >>> /usr/lib64/dirsrv/plugins/libipa_lockout.so >>> #7 0x00007f683f4c8848 in plugin_call_func () from >>> /usr/lib64/dirsrv/libslapd.so.0 >>> #8 0x00007f683f4c8b13 in plugin_call_plugins () from >>> /usr/lib64/dirsrv/libslapd.so.0 >>> #9 0x00007f683f9954d8 in do_bind () >>> #10 0x00007f683f99c3fd in connection_threadmain () >>> #11 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so >>> #12 0x00007f683d04cdc5 in start_thread () from /lib64/libpthread.so.0 >>> #13 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 >>> Thread 30 (Thread 0x7f6809ffb700 (LWP 4863)): >>> #0 0x00007f683d0506d5 in >>> pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 >>> #1 0x00007f68375ac2f3 in __db_hybrid_mutex_suspend () from /lib64/ >>> libdb-5.3.so >>> #2 0x00007f68375ab640 in __db_tas_mutex_lock () from /lib64/ >>> libdb-5.3.so >>> #3 0x00007f6837655d3a in __lock_get_internal () from /lib64/ >>> libdb-5.3.so >>> #4 0x00007f6837656820 in __lock_get () from /lib64/libdb-5.3.so >>> #5 0x00007f6837682142 in __db_lget () from /lib64/libdb-5.3.so >>> #6 0x00007f68375c9605 in __bam_search () from /lib64/libdb-5.3.so >>> #7 0x00007f68375b4256 in __bamc_search () from /lib64/libdb-5.3.so >>> #8 0x00007f68375b5d0f in __bamc_get () from /lib64/libdb-5.3.so >>> #9 0x00007f683766eca6 in __dbc_iget () from /lib64/libdb-5.3.so >>> #10 0x00007f683767db02 in __dbc_get_pp () from /lib64/libdb-5.3.so >>> #11 0x00007f6832b59680 in idl_new_fetch () from >>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>> #12 0x00007f6832b67b26 in index_read_ext_allids () from >>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>> #13 0x00007f6832b52284 in keys2idl () from /usr/lib64/dirsrv/plugins/libb >>> ack-ldbm.so >>> #14 0x00007f6832b529e3 in ava_candidates.isra.0 () from >>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>> #15 0x00007f6832b52fd2 in filter_candidates_ext () from >>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>> #16 0x00007f6832b54096 in list_candidates () from >>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>> #17 0x00007f6832b52f40 in filter_candidates_ext () from >>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>> #18 0x00007f6832b8e80c in subtree_candidates () from >>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>> #19 0x00007f6832b8fe7f in ldbm_back_search () from >>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>> #20 0x00007f683f4bd1d6 in op_shared_search () from >>> /usr/lib64/dirsrv/libslapd.so.0 >>> #21 0x00007f683f4cd94e in search_internal_callback_pb () from >>> /usr/lib64/dirsrv/libslapd.so.0 >>> #22 0x00007f683f4cdbe8 in search_internal_pb () from >>> /usr/lib64/dirsrv/libslapd.so.0 >>> #23 0x00007f683f9ac49b in ids_sasl_canon_user () >>> #24 0x00007f683d46c93b in _sasl_canon_user () from /lib64/libsasl2.so.3 >>> #25 0x00007f683d46cc4c in _sasl_canon_user_lookup () from >>> /lib64/libsasl2.so.3 >>> #26 0x00007f6837374fb9 in gssapi_server_mech_step () from >>> /usr/lib64/sasl2/libgssapiv2.so >>> #27 0x00007f683d478b9b in sasl_server_step () from /lib64/libsasl2.so.3 >>> #28 0x00007f683f9ad83b in ids_sasl_check_bind () >>> #29 0x00007f683f99561a in do_bind () >>> #30 0x00007f683f99c3fd in connection_threadmain () >>> #31 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so >>> #32 0x00007f683d04cdc5 in start_thread () from /lib64/libpthread.so.0 >>> #33 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 >>> Thread 29 (Thread 0x7f68097fa700 (LWP 4864)): >>> #0 0x00007f683d0506d5 in >>> pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 >>> #1 0x00007f683d6a6050 in PR_WaitCondVar () from /lib64/libnspr4.so >>> #2 0x00007f683f999fee in connection_wait_for_new_work () >>> #3 0x00007f683f99b719 in connection_threadmain () >>> #4 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so >>> #5 0x00007f683d04cdc5 in start_thread () from /lib64/libpthread.so.0 >>> #6 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 >>> Thread 28 (Thread 0x7f6808ff9700 (LWP 4865)): >>> #0 0x00007f683d0506d5 in >>> pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 >>> #1 0x00007f683d6a6050 in PR_WaitCondVar () from /lib64/libnspr4.so >>> #2 0x00007f683f999fee in connection_wait_for_new_work () >>> #3 0x00007f683f99b719 in connection_threadmain () >>> #4 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so >>> #5 0x00007f683d04cdc5 in start_thread () from /lib64/libpthread.so.0 >>> #6 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 >>> Thread 27 (Thread 0x7f6803fff700 (LWP 4866)): >>> #0 0x00007f683d0506d5 in >>> pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 >>> #1 0x00007f683d6a6050 in PR_WaitCondVar () from /lib64/libnspr4.so >>> #2 0x00007f683f999fee in connection_wait_for_new_work () >>> #3 0x00007f683f99b719 in connection_threadmain () >>> #4 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so >>> #5 0x00007f683d04cdc5 in start_thread () from /lib64/libpthread.so.0 >>> #6 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 >>> Thread 26 (Thread 0x7f68037fe700 (LWP 4867)): >>> #0 0x00007f683d0506d5 in >>> pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 >>> #1 0x00007f683d6a6050 in PR_WaitCondVar () from /lib64/libnspr4.so >>> #2 0x00007f683f999fee in connection_wait_for_new_work () >>> #3 0x00007f683f99b719 in connection_threadmain () >>> #4 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so >>> #5 0x00007f683d04cdc5 in start_thread () from /lib64/libpthread.so.0 >>> #6 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 >>> Thread 25 (Thread 0x7f6802ffd700 (LWP 4868)): >>> #0 0x00007f683d0506d5 in >>> pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 >>> #1 0x00007f683d6a6050 in PR_WaitCondVar () from /lib64/libnspr4.so >>> #2 0x00007f683f999fee in connection_wait_for_new_work () >>> #3 0x00007f683f99b719 in connection_threadmain () >>> #4 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so >>> #5 0x00007f683d04cdc5 in start_thread () from /lib64/libpthread.so.0 >>> #6 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 >>> Thread 24 (Thread 0x7f68027fc700 (LWP 4869)): >>> #0 0x00007f683d0506d5 in >>> pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 >>> #1 0x00007f68375ac2f3 in __db_hybrid_mutex_suspend () from /lib64/ >>> libdb-5.3.so >>> #2 0x00007f68375abb37 in __db_tas_mutex_readlock () from /lib64/ >>> libdb-5.3.so >>> #3 0x00007f68376c5d35 in __memp_fget () from /lib64/libdb-5.3.so >>> #4 0x00007f683766f333 in __dbc_iget () from /lib64/libdb-5.3.so >>> #5 0x00007f683767db02 in __dbc_get_pp () from /lib64/libdb-5.3.so >>> #6 0x00007f6832b59822 in idl_new_fetch () from >>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>> #7 0x00007f6832b67b26 in index_read_ext_allids () from >>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>> #8 0x00007f6832b52284 in keys2idl () from /usr/lib64/dirsrv/plugins/libb >>> ack-ldbm.so >>> #9 0x00007f6832b529e3 in ava_candidates.isra.0 () from >>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>> #10 0x00007f6832b52fd2 in filter_candidates_ext () from >>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>> #11 0x00007f6832b54096 in list_candidates () from >>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>> #12 0x00007f6832b52f40 in filter_candidates_ext () from >>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>> #13 0x00007f6832b54096 in list_candidates () from >>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>> #14 0x00007f6832b52f40 in filter_candidates_ext () from >>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>> #15 0x00007f6832b54096 in list_candidates () from >>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>> #16 0x00007f6832b52f40 in filter_candidates_ext () from >>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>> #17 0x00007f6832b8e80c in subtree_candidates () from >>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>> #18 0x00007f6832b8fe7f in ldbm_back_search () from >>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>> #19 0x00007f683f4bd1d6 in op_shared_search () from >>> /usr/lib64/dirsrv/libslapd.so.0 >>> #20 0x00007f683f9ae33e in do_search () >>> #21 0x00007f683f99c3c3 in connection_threadmain () >>> #22 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so >>> #23 0x00007f683d04cdc5 in start_thread () from /lib64/libpthread.so.0 >>> #24 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 >>> Thread 23 (Thread 0x7f6801ffb700 (LWP 4870)): >>> #0 0x00007f683d0506d5 in >>> pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 >>> #1 0x00007f683d6a6050 in PR_WaitCondVar () from /lib64/libnspr4.so >>> #2 0x00007f683f999fee in connection_wait_for_new_work () >>> #3 0x00007f683f99b719 in connection_threadmain () >>> #4 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so >>> #5 0x00007f683d04cdc5 in start_thread () from /lib64/libpthread.so.0 >>> #6 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 >>> Thread 22 (Thread 0x7f68017fa700 (LWP 4871)): >>> #0 0x00007f683d0506d5 in >>> pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 >>> #1 0x00007f683d6a6050 in PR_WaitCondVar () from /lib64/libnspr4.so >>> #2 0x00007f683f999fee in connection_wait_for_new_work () >>> #3 0x00007f683f99b719 in connection_threadmain () >>> #4 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so >>> #5 0x00007f683d04cdc5 in start_thread () from /lib64/libpthread.so.0 >>> #6 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 >>> Thread 21 (Thread 0x7f6800ff9700 (LWP 4872)): >>> #0 0x00007f683d0506d5 in >>> pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 >>> #1 0x00007f683d6a6050 in PR_WaitCondVar () from /lib64/libnspr4.so >>> #2 0x00007f683f999fee in connection_wait_for_new_work () >>> #3 0x00007f683f99b719 in connection_threadmain () >>> #4 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so >>> #5 0x00007f683d04cdc5 in start_thread () from /lib64/libpthread.so.0 >>> #6 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 >>> Thread 20 (Thread 0x7f68007f8700 (LWP 4873)): >>> #0 0x00007f683d0506d5 in >>> pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 >>> #1 0x00007f68375ac2f3 in __db_hybrid_mutex_suspend () from /lib64/ >>> libdb-5.3.so >>> #2 0x00007f68375ab640 in __db_tas_mutex_lock () from /lib64/ >>> libdb-5.3.so >>> #3 0x00007f68376c5e1b in __memp_fget () from /lib64/libdb-5.3.so >>> #4 0x00007f68375c9211 in __bam_search () from /lib64/libdb-5.3.so >>> #5 0x00007f68375b4256 in __bamc_search () from /lib64/libdb-5.3.so >>> #6 0x00007f68375b86c4 in __bamc_put () from /lib64/libdb-5.3.so >>> #7 0x00007f683766f6d5 in __dbc_iput () from /lib64/libdb-5.3.so >>> #8 0x00007f683766aa0e in __db_put () from /lib64/libdb-5.3.so >>> #9 0x00007f683767ffa4 in __db_put_pp () from /lib64/libdb-5.3.so >>> #10 0x00007f6832b5a4cf in idl_new_insert_key () from >>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>> #11 0x00007f6832b695b5 in addordel_values_sv.isra.3 () from >>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>> #12 0x00007f6832b6a25a in index_addordel_values_ext_sv () from >>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>> #13 0x00007f6832b6a694 in index_addordel_values_sv () from >>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>> #14 0x00007f6832b6a75a in index_addordel_entry () from >>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>> #15 0x00007f6832b6d5fa in ldbm_back_add () from >>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>> #16 0x00007f683f46fcca in op_shared_add () from >>> /usr/lib64/dirsrv/libslapd.so.0 >>> #17 0x00007f683f471020 in do_add () from /usr/lib64/dirsrv/libslapd.so.0 >>> #18 0x00007f683f99c303 in connection_threadmain () >>> #19 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so >>> #20 0x00007f683d04cdc5 in start_thread () from /lib64/libpthread.so.0 >>> #21 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 >>> Thread 19 (Thread 0x7f67ffff7700 (LWP 4874)): >>> #0 0x00007f683d0506d5 in >>> pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 >>> #1 0x00007f683d6a62c3 in PR_EnterMonitor () from /lib64/libnspr4.so >>> #2 0x00007f6832b4d3e6 in dblayer_txn_begin () from >>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>> #3 0x00007f6832b89007 in ldbm_back_modify () from >>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>> #4 0x00007f683f4b6541 in op_shared_modify () from >>> /usr/lib64/dirsrv/libslapd.so.0 >>> #5 0x00007f683f4b7004 in modify_internal_pb () from >>> /usr/lib64/dirsrv/libslapd.so.0 >>> #6 0x00007f6832dd3b60 in ipapwd_apply_mods () from >>> /usr/lib64/dirsrv/plugins/libipa_pwd_extop.so >>> #7 0x00007f6832dd78e0 in store_new_keys () from >>> /usr/lib64/dirsrv/plugins/libipa_pwd_extop.so >>> #8 0x00007f6832dd9ad0 in ipapwd_extop () from >>> /usr/lib64/dirsrv/plugins/libipa_pwd_extop.so >>> #9 0x00007f683f4c6ba2 in plugin_call_exop_plugins () from >>> /usr/lib64/dirsrv/libslapd.so.0 >>> #10 0x00007f683f9a2efe in do_extended () >>> #11 0x00007f683f99c41a in connection_threadmain () >>> #12 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so >>> #13 0x00007f683d04cdc5 in start_thread () from /lib64/libpthread.so.0 >>> #14 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 >>> Thread 18 (Thread 0x7f67ff7f6700 (LWP 4875)): >>> #0 0x00007f683d0506d5 in >>> pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 >>> #1 0x00007f683d6a6050 in PR_WaitCondVar () from /lib64/libnspr4.so >>> #2 0x00007f683f999fee in connection_wait_for_new_work () >>> #3 0x00007f683f99b719 in connection_threadmain () >>> #4 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so >>> #5 0x00007f683d04cdc5 in start_thread () from /lib64/libpthread.so.0 >>> #6 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 >>> Thread 17 (Thread 0x7f67feff5700 (LWP 4876)): >>> #0 0x00007f683d0506d5 in >>> pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 >>> #1 0x00007f68375ac2f3 in __db_hybrid_mutex_suspend () from /lib64/ >>> libdb-5.3.so >>> #2 0x00007f68375ab640 in __db_tas_mutex_lock () from /lib64/ >>> libdb-5.3.so >>> #3 0x00007f6837655d3a in __lock_get_internal () from /lib64/ >>> libdb-5.3.so >>> #4 0x00007f6837656820 in __lock_get () from /lib64/libdb-5.3.so >>> #5 0x00007f6837682142 in __db_lget () from /lib64/libdb-5.3.so >>> #6 0x00007f68375c9605 in __bam_search () from /lib64/libdb-5.3.so >>> #7 0x00007f68375b4256 in __bamc_search () from /lib64/libdb-5.3.so >>> #8 0x00007f68375b5d0f in __bamc_get () from /lib64/libdb-5.3.so >>> #9 0x00007f683766eca6 in __dbc_iget () from /lib64/libdb-5.3.so >>> #10 0x00007f683767db02 in __dbc_get_pp () from /lib64/libdb-5.3.so >>> #11 0x00007f6832b59680 in idl_new_fetch () from >>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>> #12 0x00007f6832b67b26 in index_read_ext_allids () from >>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>> #13 0x00007f6832b52284 in keys2idl () from /usr/lib64/dirsrv/plugins/libb >>> ack-ldbm.so >>> #14 0x00007f6832b529e3 in ava_candidates.isra.0 () from >>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>> #15 0x00007f6832b52fd2 in filter_candidates_ext () from >>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>> #16 0x00007f6832b54096 in list_candidates () from >>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>> #17 0x00007f6832b52f40 in filter_candidates_ext () from >>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>> #18 0x00007f6832b8e80c in subtree_candidates () from >>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>> #19 0x00007f6832b8fe7f in ldbm_back_search () from >>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>> #20 0x00007f683f4bd1d6 in op_shared_search () from >>> /usr/lib64/dirsrv/libslapd.so.0 >>> #21 0x00007f683f4cd94e in search_internal_callback_pb () from >>> /usr/lib64/dirsrv/libslapd.so.0 >>> #22 0x00007f683f4cdbe8 in search_internal_pb () from >>> /usr/lib64/dirsrv/libslapd.so.0 >>> #23 0x00007f6832dd75dc in get_entry_by_principal () from >>> /usr/lib64/dirsrv/plugins/libipa_pwd_extop.so >>> #24 0x00007f6832dd8eb1 in ipapwd_extop () from >>> /usr/lib64/dirsrv/plugins/libipa_pwd_extop.so >>> #25 0x00007f683f4c6ba2 in plugin_call_exop_plugins () from >>> /usr/lib64/dirsrv/libslapd.so.0 >>> #26 0x00007f683f9a2efe in do_extended () >>> #27 0x00007f683f99c41a in connection_threadmain () >>> #28 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so >>> #29 0x00007f683d04cdc5 in start_thread () from /lib64/libpthread.so.0 >>> #30 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 >>> Thread 16 (Thread 0x7f67fe7f4700 (LWP 4877)): >>> #0 0x00007f683d0506d5 in >>> pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 >>> #1 0x00007f683d6a6050 in PR_WaitCondVar () from /lib64/libnspr4.so >>> #2 0x00007f683f999fee in connection_wait_for_new_work () >>> #3 0x00007f683f99b719 in connection_threadmain () >>> #4 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so >>> #5 0x00007f683d04cdc5 in start_thread () from /lib64/libpthread.so.0 >>> #6 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 >>> Thread 15 (Thread 0x7f67fdff3700 (LWP 4878)): >>> #0 0x00007f683d0506d5 in >>> pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 >>> #1 0x00007f683d6a6050 in PR_WaitCondVar () from /lib64/libnspr4.so >>> #2 0x00007f683f999fee in connection_wait_for_new_work () >>> #3 0x00007f683f99b719 in connection_threadmain () >>> #4 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so >>> #5 0x00007f683d04cdc5 in start_thread () from /lib64/libpthread.so.0 >>> #6 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 >>> Thread 14 (Thread 0x7f67fd7f2700 (LWP 4879)): >>> #0 0x00007f683d0506d5 in >>> pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 >>> #1 0x00007f683d6a6050 in PR_WaitCondVar () from /lib64/libnspr4.so >>> #2 0x00007f683f999fee in connection_wait_for_new_work () >>> #3 0x00007f683f99b719 in connection_threadmain () >>> #4 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so >>> #5 0x00007f683d04cdc5 in start_thread () from /lib64/libpthread.so.0 >>> #6 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 >>> Thread 13 (Thread 0x7f67fcff1700 (LWP 4880)): >>> #0 0x00007f683d0506d5 in >>> pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 >>> #1 0x00007f683d6a6050 in PR_WaitCondVar () from /lib64/libnspr4.so >>> #2 0x00007f683f999fee in connection_wait_for_new_work () >>> #3 0x00007f683f99b719 in connection_threadmain () >>> #4 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so >>> #5 0x00007f683d04cdc5 in start_thread () from /lib64/libpthread.so.0 >>> #6 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 >>> Thread 12 (Thread 0x7f67fc7f0700 (LWP 4881)): >>> #0 0x00007f683d0506d5 in >>> pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 >>> #1 0x00007f683d6a6050 in PR_WaitCondVar () from /lib64/libnspr4.so >>> #2 0x00007f683f999fee in connection_wait_for_new_work () >>> #3 0x00007f683f99b719 in connection_threadmain () >>> #4 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so >>> #5 0x00007f683d04cdc5 in start_thread () from /lib64/libpthread.so.0 >>> #6 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 >>> Thread 11 (Thread 0x7f67fbfef700 (LWP 4882)): >>> #0 0x00007f683d0506d5 in >>> pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 >>> #1 0x00007f683d6a6050 in PR_WaitCondVar () from /lib64/libnspr4.so >>> #2 0x00007f683f999fee in connection_wait_for_new_work () >>> #3 0x00007f683f99b719 in connection_threadmain () >>> #4 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so >>> #5 0x00007f683d04cdc5 in start_thread () from /lib64/libpthread.so.0 >>> #6 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 >>> Thread 10 (Thread 0x7f67fb7ee700 (LWP 4883)): >>> #0 0x00007f683d0506d5 in >>> pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 >>> #1 0x00007f68375ac2f3 in __db_hybrid_mutex_suspend () from /lib64/ >>> libdb-5.3.so >>> #2 0x00007f68375ab640 in __db_tas_mutex_lock () from /lib64/ >>> libdb-5.3.so >>> #3 0x00007f6837655d3a in __lock_get_internal () from /lib64/ >>> libdb-5.3.so >>> #4 0x00007f6837656820 in __lock_get () from /lib64/libdb-5.3.so >>> #5 0x00007f6837682142 in __db_lget () from /lib64/libdb-5.3.so >>> #6 0x00007f68375c9605 in __bam_search () from /lib64/libdb-5.3.so >>> #7 0x00007f68375b4256 in __bamc_search () from /lib64/libdb-5.3.so >>> #8 0x00007f68375b5d0f in __bamc_get () from /lib64/libdb-5.3.so >>> #9 0x00007f683766eca6 in __dbc_iget () from /lib64/libdb-5.3.so >>> #10 0x00007f683767db02 in __dbc_get_pp () from /lib64/libdb-5.3.so >>> #11 0x00007f6832b59680 in idl_new_fetch () from >>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>> #12 0x00007f6832b67b26 in index_read_ext_allids () from >>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>> #13 0x00007f6832b52284 in keys2idl () from /usr/lib64/dirsrv/plugins/libb >>> ack-ldbm.so >>> #14 0x00007f6832b529e3 in ava_candidates.isra.0 () from >>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>> #15 0x00007f6832b52fd2 in filter_candidates_ext () from >>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>> #16 0x00007f6832b54096 in list_candidates () from >>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>> #17 0x00007f6832b52f40 in filter_candidates_ext () from >>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>> #18 0x00007f6832b54096 in list_candidates () from >>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>> #19 0x00007f6832b52f40 in filter_candidates_ext () from >>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>> #20 0x00007f6832b54096 in list_candidates () from >>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>> #21 0x00007f6832b52f40 in filter_candidates_ext () from >>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>> #22 0x00007f6832b8e80c in subtree_candidates () from >>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>> #23 0x00007f6832b8fe7f in ldbm_back_search () from >>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>> #24 0x00007f683f4bd1d6 in op_shared_search () from >>> /usr/lib64/dirsrv/libslapd.so.0 >>> #25 0x00007f683f9ae33e in do_search () >>> #26 0x00007f683f99c3c3 in connection_threadmain () >>> #27 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so >>> #28 0x00007f683d04cdc5 in start_thread () from /lib64/libpthread.so.0 >>> #29 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 >>> Thread 9 (Thread 0x7f67fafed700 (LWP 4884)): >>> #0 0x00007f683d0506d5 in >>> pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 >>> #1 0x00007f683d6a6050 in PR_WaitCondVar () from /lib64/libnspr4.so >>> #2 0x00007f683f999fee in connection_wait_for_new_work () >>> #3 0x00007f683f99b719 in connection_threadmain () >>> #4 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so >>> #5 0x00007f683d04cdc5 in start_thread () from /lib64/libpthread.so.0 >>> #6 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 >>> Thread 8 (Thread 0x7f67fa7ec700 (LWP 4885)): >>> #0 0x00007f683d0506d5 in >>> pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 >>> #1 0x00007f68375ac2f3 in __db_hybrid_mutex_suspend () from /lib64/ >>> libdb-5.3.so >>> #2 0x00007f68375ab640 in __db_tas_mutex_lock () from /lib64/ >>> libdb-5.3.so >>> #3 0x00007f6837655d3a in __lock_get_internal () from /lib64/ >>> libdb-5.3.so >>> #4 0x00007f6837656820 in __lock_get () from /lib64/libdb-5.3.so >>> #5 0x00007f6837682142 in __db_lget () from /lib64/libdb-5.3.so >>> #6 0x00007f68375c9605 in __bam_search () from /lib64/libdb-5.3.so >>> #7 0x00007f68375b4256 in __bamc_search () from /lib64/libdb-5.3.so >>> #8 0x00007f68375b5d0f in __bamc_get () from /lib64/libdb-5.3.so >>> #9 0x00007f683766eca6 in __dbc_iget () from /lib64/libdb-5.3.so >>> #10 0x00007f683767db02 in __dbc_get_pp () from /lib64/libdb-5.3.so >>> #11 0x00007f6832b59680 in idl_new_fetch () from >>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>> #12 0x00007f6832b67b26 in index_read_ext_allids () from >>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>> #13 0x00007f6832b52284 in keys2idl () from /usr/lib64/dirsrv/plugins/libb >>> ack-ldbm.so >>> #14 0x00007f6832b529e3 in ava_candidates.isra.0 () from >>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>> #15 0x00007f6832b52fd2 in filter_candidates_ext () from >>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>> #16 0x00007f6832b54096 in list_candidates () from >>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>> #17 0x00007f6832b52f40 in filter_candidates_ext () from >>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>> #18 0x00007f6832b54096 in list_candidates () from >>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>> #19 0x00007f6832b52f40 in filter_candidates_ext () from >>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>> #20 0x00007f6832b54096 in list_candidates () from >>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>> #21 0x00007f6832b52f40 in filter_candidates_ext () from >>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>> #22 0x00007f6832b8e80c in subtree_candidates () from >>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>> #23 0x00007f6832b8fe7f in ldbm_back_search () from >>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>> #24 0x00007f683f4bd1d6 in op_shared_search () from >>> /usr/lib64/dirsrv/libslapd.so.0 >>> #25 0x00007f683f9ae33e in do_search () >>> #26 0x00007f683f99c3c3 in connection_threadmain () >>> #27 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so >>> #28 0x00007f683d04cdc5 in start_thread () from /lib64/libpthread.so.0 >>> #29 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 >>> Thread 7 (Thread 0x7f67f9feb700 (LWP 4886)): >>> #0 0x00007f683d0506d5 in >>> pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 >>> #1 0x00007f683d6a6050 in PR_WaitCondVar () from /lib64/libnspr4.so >>> #2 0x00007f683f999fee in connection_wait_for_new_work () >>> #3 0x00007f683f99b719 in connection_threadmain () >>> #4 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so >>> #5 0x00007f683d04cdc5 in start_thread () from /lib64/libpthread.so.0 >>> #6 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 >>> Thread 6 (Thread 0x7f67f97ea700 (LWP 4887)): >>> #0 0x00007f683d0506d5 in >>> pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 >>> #1 0x00007f683d6a6050 in PR_WaitCondVar () from /lib64/libnspr4.so >>> #2 0x00007f683f999fee in connection_wait_for_new_work () >>> #3 0x00007f683f99b719 in connection_threadmain () >>> #4 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so >>> #5 0x00007f683d04cdc5 in start_thread () from /lib64/libpthread.so.0 >>> #6 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 >>> Thread 5 (Thread 0x7f67f8fe9700 (LWP 4888)): >>> #0 0x00007f683d0506d5 in >>> pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 >>> #1 0x00007f683d6a6050 in PR_WaitCondVar () from /lib64/libnspr4.so >>> #2 0x00007f683f999fee in connection_wait_for_new_work () >>> #3 0x00007f683f99b719 in connection_threadmain () >>> #4 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so >>> #5 0x00007f683d04cdc5 in start_thread () from /lib64/libpthread.so.0 >>> #6 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 >>> Thread 4 (Thread 0x7f67f87e8700 (LWP 4889)): >>> #0 0x00007f683cd719b3 in select () from /lib64/libc.so.6 >>> #1 0x00007f683f4fc0e9 in DS_Sleep () from /usr/lib64/dirsrv/libslapd.so. >>> 0 >>> #2 0x00007f683f99d325 in time_thread () >>> #3 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so >>> #4 0x00007f683d04cdc5 in start_thread () from /lib64/libpthread.so.0 >>> #5 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 >>> Thread 3 (Thread 0x7f67f7de6700 (LWP 4892)): >>> #0 0x00007f683d0506d5 in >>> pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 >>> #1 0x00007f683d6a6050 in PR_WaitCondVar () from /lib64/libnspr4.so >>> #2 0x00007f683f9a8f25 in ps_send_results () >>> #3 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so >>> #4 0x00007f683d04cdc5 in start_thread () from /lib64/libpthread.so.0 >>> #5 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 >>> Thread 2 (Thread 0x7f67f75e5700 (LWP 5084)): >>> #0 0x00007f683d0506d5 in >>> pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 >>> #1 0x00007f683d6a6050 in PR_WaitCondVar () from /lib64/libnspr4.so >>> #2 0x00007f683f9a8f25 in ps_send_results () >>> #3 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so >>> #4 0x00007f683d04cdc5 in start_thread () from /lib64/libpthread.so.0 >>> #5 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 >>> Thread 1 (Thread 0x7f683f964840 (LWP 4845)): >>> #0 0x00007f683d0506d5 in >>> pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 >>> #1 0x00007f683d6a62c3 in PR_EnterMonitor () from /lib64/libnspr4.so >>> #2 0x00007f683f99d5ed in setup_pr_read_pds () >>> #3 0x00007f683f9a0fd9 in slapd_daemon () >>> #4 0x00007f683f992c6c in main () >>> >>> >>> >>> the top output seems ok, there was just a sudden spike to around 340% >>> cpu then it comes down >>> >>> top - 08:53:12 up 6 days, 23:20, 1 user, load average: 0.00, 0.01, 0.12 >>> Threads: 44 total, 0 running, 44 sleeping, 0 stopped, 0 zombie >>> %Cpu(s): 0.0 us, 0.0 sy, 0.0 ni,100.0 id, 0.0 wa, 0.0 hi, 0.0 si, >>> 0.0 st >>> KiB Mem : 65533176 total, 52165676 free, 6135088 used, 7232412 >>> buff/cache >>> KiB Swap: 3999996 total, 3999996 free, 0 used. 58452040 avail >>> Mem >>> >>> PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ >>> COMMAND >>> 4845 dirsrv 20 0 5133196 3.150g 281272 S 0.0 5.0 31:34.51 >>> ns-slapd >>> 4847 dirsrv 20 0 5133196 3.150g 281272 S 0.0 5.0 0:13.53 >>> ns-slapd >>> 4848 dirsrv 20 0 5133196 3.150g 281272 S 0.0 5.0 0:13.16 >>> ns-slapd >>> 4849 dirsrv 20 0 5133196 3.150g 281272 S 0.0 5.0 0:45.27 >>> ns-slapd >>> 4850 dirsrv 20 0 5133196 3.150g 281272 S 0.0 5.0 0:01.60 >>> ns-slapd >>> 4851 dirsrv 20 0 5133196 3.150g 281272 S 0.0 5.0 0:00.00 >>> ns-slapd >>> 4852 dirsrv 20 0 5133196 3.150g 281272 S 0.0 5.0 0:00.00 >>> ns-slapd >>> 4853 dirsrv 20 0 5133196 3.150g 281272 S 0.0 5.0 0:00.00 >>> ns-slapd >>> 4854 dirsrv 20 0 5133196 3.150g 281272 S 0.0 5.0 0:00.00 >>> ns-slapd >>> 4855 dirsrv 20 0 5133196 3.150g 281272 S 0.0 5.0 0:03.02 >>> ns-slapd >>> 4856 dirsrv 20 0 5133196 3.150g 281272 S 0.0 5.0 0:00.26 >>> ns-slapd >>> 4859 dirsrv 20 0 5133196 3.150g 281272 S 0.0 5.0 3:18.67 >>> ns-slapd >>> 4860 dirsrv 20 0 5133196 3.150g 281272 S 0.0 5.0 3:19.82 >>> ns-slapd >>> 4861 dirsrv 20 0 5133196 3.150g 281272 S 0.0 5.0 3:20.23 >>> ns-slapd >>> 4862 dirsrv 20 0 5133196 3.150g 281272 S 0.0 5.0 3:22.65 >>> ns-slapd >>> 4863 dirsrv 20 0 5133196 3.150g 281272 S 0.0 5.0 3:16.64 >>> ns-slapd >>> 4864 dirsrv 20 0 5133196 3.150g 281272 S 0.0 5.0 3:17.39 >>> ns-slapd >>> 4865 dirsrv 20 0 5133196 3.150g 281272 S 0.0 5.0 3:17.80 >>> ns-slapd >>> >>> >>> Thanks >>> Rakesh >>> >>> On Tue, Sep 6, 2016 at 12:09 AM, Rakesh Rajasekharan < >>> rakesh.rajasekharan at gmail.com> wrote: >>> >>>> Hi Thierry/Ludwig, >>>> >>>> Unfortunately I had by then restarted the service and since then have >>>> been trying to hit the hang issue again but no luck this time >>>> >>>> It has been running fine for the last few hours now in-spite of trying >>>> all that I believe could cause the hang like making simultaneous client >>>> addition requests. >>>> >>>> I will continue testing this and next time I get it I will make sure I >>>> get the pstack and also get the dump of db lock info. >>>> >>>> Thanks, >>>> Rakesh >>>> >>>> >>>> >>>> On Mon, Sep 5, 2016 at 6:14 PM, Ludwig Krispenz < >>>> lkrispen at redhat.com> wrote: >>>> >>>>> >>>>> On 09/05/2016 02:31 PM, thierry bordaz wrote: >>>>> >>>>> Hi Rakesh, >>>>> >>>>> Thanks you soooo much for you continuous help on this !! >>>>> >>>>> If the hang is still going on, would you dump the DB lock information >>>>> with: >>>>> >>>>> db_stat -CA -N -h /var/lib/dirsrv/slapd-/db >>>>> >/tmp/db_lock_info >>>>> >>>>> It is looking like several threads are waiting for one/several db >>>>> locks. With the lock info we will know which thread acquired it/them. >>>>> A guess is that it could be the checkpointing thread but this one is >>>>> also waiting (on a memory pool). >>>>> >>>>> I am copying Ludwig on this thread in case this pstack rings any bell. >>>>> >>>>> the frequent presence of __db_tas_mutex_lock() looks like the NUMA >>>>> problem of libdb, although the threads already have stopped spinning (maybe >>>>> this was an effect of gcore). >>>>> If the hang aoccurs again, could you check cpu consumption >>>>> top -H -p >>>>> and get a pstack >>>>> pstack >>>>> >>>>> >>>>> best regards >>>>> thierry >>>>> >>>>> On 09/05/2016 02:05 PM, Rakesh Rajasekharan wrote: >>>>> >>>>> Hi Thierry, >>>>> >>>>> I was able to now reporduce the issue.. >>>>> >>>>> the ns-slapd process just got hung,.. however, no core file was >>>>> generate.. >>>>> >>>>> I manually created a coredump using gcore and then took a strace of it >>>>> this is strace from the coredump generated >>>>> >>>>> GNU gdb (GDB) Red Hat Enterprise Linux 7.6.1-80.el7 >>>>> Copyright (C) 2013 Free Software Foundation, Inc. >>>>> License GPLv3+: GNU GPL version 3 or later < >>>>> http://gnu.org/licenses/gpl.html> >>>>> This is free software: you are free to change and redistribute it. >>>>> There is NO WARRANTY, to the extent permitted by law. Type "show >>>>> copying" >>>>> and "show warranty" for details. >>>>> This GDB was configured as "x86_64-redhat-linux-gnu". >>>>> For bug reporting instructions, please see: >>>>> < http://www.gnu.org/software/g >>>>> db/bugs/>... >>>>> Reading symbols from /usr/sbin/ns-slapd...Reading symbols from >>>>> /usr/sbin/ns-slapd...(no debugging symbols found)...done. >>>>> (no debugging symbols found)...done. >>>>> [New LWP 105159] >>>>> [New LWP 105160] >>>>> [New LWP 105161] >>>>> [New LWP 105162] >>>>> [New LWP 105163] >>>>> [New LWP 105164] >>>>> [New LWP 105165] >>>>> [New LWP 105166] >>>>> [New LWP 105167] >>>>> [New LWP 105168] >>>>> [New LWP 105171] >>>>> [New LWP 105172] >>>>> [New LWP 105173] >>>>> [New LWP 105174] >>>>> [New LWP 105175] >>>>> [New LWP 105176] >>>>> [New LWP 105177] >>>>> [New LWP 105178] >>>>> [New LWP 105179] >>>>> [New LWP 105180] >>>>> [New LWP 105181] >>>>> [New LWP 105182] >>>>> [New LWP 105183] >>>>> [New LWP 105184] >>>>> [New LWP 105185] >>>>> [New LWP 105186] >>>>> [New LWP 105187] >>>>> [New LWP 105188] >>>>> [New LWP 105189] >>>>> [New LWP 105190] >>>>> [New LWP 105191] >>>>> [New LWP 105192] >>>>> [New LWP 105193] >>>>> [New LWP 105194] >>>>> [New LWP 105195] >>>>> [New LWP 105196] >>>>> [New LWP 105197] >>>>> [New LWP 105198] >>>>> [New LWP 105199] >>>>> [New LWP 105200] >>>>> [New LWP 105201] >>>>> [New LWP 105203] >>>>> [New LWP 105215] >>>>> [New LWP 105158] >>>>> [Thread debugging using libthread_db enabled] >>>>> Using host libthread_db library "/lib64/libthread_db.so.1". >>>>> Core was generated by `/usr/sbin/ns-slapd'. >>>>> #0 0x00007f1b48ae69b3 in select () from /lib64/libc.so.6 >>>>> >>>>> Thread 44 (Thread 0x7f1b4b6d9840 (LWP 105158)): >>>>> #0 0x00007f1b48dc56d5 in >>>>> pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 >>>>> No symbol table info available. >>>>> #1 0x00007f1b4941b2c3 in PR_EnterMonitor () from /lib64/libnspr4.so >>>>> No symbol table info available. >>>>> #2 0x00007f1b4b7125ed in setup_pr_read_pds () >>>>> No symbol table info available. >>>>> #3 0x00007f1b4b715fd9 in slapd_daemon () >>>>> No symbol table info available. >>>>> #4 0x00007f1b4b707c6c in main () >>>>> No symbol table info available. >>>>> >>>>> Thread 43 (Thread 0x7f1a5df0f700 (LWP 105215)): >>>>> #0 0x00007f1b48dc56d5 in >>>>> pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 >>>>> No symbol table info available. >>>>> #1 0x00007f1b4941b050 in PR_WaitCondVar () from /lib64/libnspr4.so >>>>> No symbol table info available. >>>>> #2 0x00007f1b4b71df25 in ps_send_results () >>>>> No symbol table info available. >>>>> #3 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so >>>>> No symbol table info available. >>>>> #4 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 >>>>> No symbol table info available. >>>>> #5 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 >>>>> No symbol table info available. >>>>> >>>>> Thread 42 (Thread 0x7f1b03fe7700 (LWP 105203)): >>>>> #0 0x00007f1b48dc56d5 in >>>>> pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 >>>>> No symbol table info available. >>>>> #1 0x00007f1b4941b050 in PR_WaitCondVar () from /lib64/libnspr4.so >>>>> No symbol table info available. >>>>> #2 0x00007f1b4b71df25 in ps_send_results () >>>>> No symbol table info available. >>>>> #3 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so >>>>> No symbol table info available. >>>>> #4 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 >>>>> No symbol table info available. >>>>> #5 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 >>>>> No symbol table info available. >>>>> >>>>> Thread 41 (Thread 0x7f1b047e8700 (LWP 105201)): >>>>> #0 0x00007f1b48ae69b3 in select () from /lib64/libc.so.6 >>>>> No symbol table info available. >>>>> #1 0x00007f1b4b2710e9 in DS_Sleep () from >>>>> /usr/lib64/dirsrv/libslapd.so.0 >>>>> No symbol table info available. >>>>> #2 0x00007f1b4b712325 in time_thread () >>>>> No symbol table info available. >>>>> #3 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so >>>>> No symbol table info available. >>>>> #4 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 >>>>> No symbol table info available. >>>>> #5 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 >>>>> No symbol table info available. >>>>> >>>>> Thread 40 (Thread 0x7f1b04fe9700 (LWP 105200)): >>>>> #0 0x00007f1b48dc56d5 in >>>>> pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 >>>>> No symbol table info available. >>>>> #1 0x00007f1b433212f3 in __db_hybrid_mutex_suspend () from /lib64/ >>>>> libdb-5.3.so >>>>> No symbol table info available. >>>>> #2 0x00007f1b43320640 in __db_tas_mutex_lock () from /lib64/ >>>>> libdb-5.3.so >>>>> No symbol table info available. >>>>> #3 0x00007f1b4343ae1b in __memp_fget () from /lib64/libdb-5.3.so >>>>> No symbol table info available. >>>>> #4 0x00007f1b4333e211 in __bam_search () from /lib64/libdb-5.3.so >>>>> No symbol table info available. >>>>> #5 0x00007f1b43329256 in __bamc_search () from /lib64/libdb-5.3.so >>>>> No symbol table info available. >>>>> #6 0x00007f1b4332d6c4 in __bamc_put () from /lib64/libdb-5.3.so >>>>> No symbol table info available. >>>>> #7 0x00007f1b433e46d5 in __dbc_iput () from /lib64/libdb-5.3.so >>>>> No symbol table info available. >>>>> #8 0x00007f1b433dfa0e in __db_put () from /lib64/libdb-5.3.so >>>>> No symbol table info available. >>>>> #9 0x00007f1b433f4fa4 in __db_put_pp () from /lib64/libdb-5.3.so >>>>> No symbol table info available. >>>>> #10 0x00007f1b3e8cf4cf in idl_new_insert_key () from >>>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>>> No symbol table info available. >>>>> #11 0x00007f1b3e8de5b5 in addordel_values_sv.isra.3 () from >>>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>>> No symbol table info available. >>>>> #12 0x00007f1b3e8df25a in index_addordel_values_ext_sv () from >>>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>>> No symbol table info available. >>>>> #13 0x00007f1b3e8df694 in index_addordel_values_sv () from >>>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>>> No symbol table info available. >>>>> #14 0x00007f1b3e8df75a in index_addordel_entry () from >>>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>>> No symbol table info available. >>>>> #15 0x00007f1b3e8e25fa in ldbm_back_add () from >>>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>>> No symbol table info available. >>>>> #16 0x00007f1b4b1e4cca in op_shared_add () from >>>>> /usr/lib64/dirsrv/libslapd.so.0 >>>>> No symbol table info available. >>>>> #17 0x00007f1b4b1e6020 in do_add () from /usr/lib64/dirsrv/libslapd.so. >>>>> 0 >>>>> No symbol table info available. >>>>> #18 0x00007f1b4b711303 in connection_threadmain () >>>>> No symbol table info available. >>>>> #19 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so >>>>> No symbol table info available. >>>>> #20 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 >>>>> No symbol table info available. >>>>> #21 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 >>>>> No symbol table info available. >>>>> >>>>> Thread 39 (Thread 0x7f1b057ea700 (LWP 105199)): >>>>> #0 0x00007f1b48dc56d5 in >>>>> pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 >>>>> No symbol table info available. >>>>> #1 0x00007f1b4941b050 in PR_WaitCondVar () from /lib64/libnspr4.so >>>>> No symbol table info available. >>>>> #2 0x00007f1b4b70efee in connection_wait_for_new_work () >>>>> No symbol table info available. >>>>> #3 0x00007f1b4b710719 in connection_threadmain () >>>>> No symbol table info available. >>>>> #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so >>>>> No symbol table info available. >>>>> #5 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 >>>>> No symbol table info available. >>>>> #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 >>>>> No symbol table info available. >>>>> >>>>> Thread 38 (Thread 0x7f1b05feb700 (LWP 105198)): >>>>> #0 0x00007f1b48dc56d5 in >>>>> pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 >>>>> No symbol table info available. >>>>> #1 0x00007f1b4941b050 in PR_WaitCondVar () from /lib64/libnspr4.so >>>>> No symbol table info available. >>>>> #2 0x00007f1b4b70efee in connection_wait_for_new_work () >>>>> No symbol table info available. >>>>> #3 0x00007f1b4b710719 in connection_threadmain () >>>>> No symbol table info available. >>>>> #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so >>>>> No symbol table info available. >>>>> #5 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 >>>>> No symbol table info available. >>>>> #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 >>>>> No symbol table info available. >>>>> >>>>> Thread 37 (Thread 0x7f1b067ec700 (LWP 105197)): >>>>> #0 0x00007f1b48dc56d5 in >>>>> pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 >>>>> No symbol table info available. >>>>> #1 0x00007f1b4941b050 in PR_WaitCondVar () from /lib64/libnspr4.so >>>>> No symbol table info available. >>>>> #2 0x00007f1b4b70efee in connection_wait_for_new_work () >>>>> No symbol table info available. >>>>> #3 0x00007f1b4b710719 in connection_threadmain () >>>>> No symbol table info available. >>>>> #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so >>>>> No symbol table info available. >>>>> #5 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 >>>>> No symbol table info available. >>>>> #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 >>>>> No symbol table info available. >>>>> >>>>> Thread 36 (Thread 0x7f1b06fed700 (LWP 105196)): >>>>> #0 0x00007f1b48dc56d5 in >>>>> pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 >>>>> No symbol table info available. >>>>> #1 0x00007f1b4941b050 in PR_WaitCondVar () from /lib64/libnspr4.so >>>>> No symbol table info available. >>>>> #2 0x00007f1b4b70efee in connection_wait_for_new_work () >>>>> No symbol table info available. >>>>> #3 0x00007f1b4b710719 in connection_threadmain () >>>>> No symbol table info available. >>>>> #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so >>>>> No symbol table info available. >>>>> #5 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 >>>>> No symbol table info available. >>>>> #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 >>>>> No symbol table info available. >>>>> >>>>> Thread 35 (Thread 0x7f1b077ee700 (LWP 105195)): >>>>> #0 0x00007f1b48dc56d5 in >>>>> pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 >>>>> No symbol table info available. >>>>> #1 0x00007f1b4941b2c3 in PR_EnterMonitor () from /lib64/libnspr4.so >>>>> No symbol table info available. >>>>> #2 0x00007f1b3e8c23e6 in dblayer_txn_begin () from >>>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>>> No symbol table info available. >>>>> #3 0x00007f1b3e8fe007 in ldbm_back_modify () from >>>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>>> No symbol table info available. >>>>> #4 0x00007f1b4b22b541 in op_shared_modify () from >>>>> /usr/lib64/dirsrv/libslapd.so.0 >>>>> No symbol table info available. >>>>> #5 0x00007f1b4b22c004 in modify_internal_pb () from >>>>> /usr/lib64/dirsrv/libslapd.so.0 >>>>> No symbol table info available. >>>>> #6 0x00007f1b3eb48b60 in ipapwd_apply_mods () from >>>>> /usr/lib64/dirsrv/plugins/libipa_pwd_extop.so >>>>> No symbol table info available. >>>>> #7 0x00007f1b3eb4c8e0 in store_new_keys () from >>>>> /usr/lib64/dirsrv/plugins/libipa_pwd_extop.so >>>>> No symbol table info available. >>>>> #8 0x00007f1b3eb4ead0 in ipapwd_extop () from >>>>> /usr/lib64/dirsrv/plugins/libipa_pwd_extop.so >>>>> No symbol table info available. >>>>> #9 0x00007f1b4b23bba2 in plugin_call_exop_plugins () from >>>>> /usr/lib64/dirsrv/libslapd.so.0 >>>>> No symbol table info available. >>>>> #10 0x00007f1b4b717efe in do_extended () >>>>> No symbol table info available. >>>>> #11 0x00007f1b4b71141a in connection_threadmain () >>>>> No symbol table info available. >>>>> #12 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so >>>>> No symbol table info available. >>>>> #13 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 >>>>> No symbol table info available. >>>>> #14 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 >>>>> No symbol table info available. >>>>> >>>>> Thread 34 (Thread 0x7f1b07fef700 (LWP 105194)): >>>>> #0 0x00007f1b48dc56d5 in >>>>> pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 >>>>> No symbol table info available. >>>>> #1 0x00007f1b4941b050 in PR_WaitCondVar () from /lib64/libnspr4.so >>>>> No symbol table info available. >>>>> #2 0x00007f1b4b70efee in connection_wait_for_new_work () >>>>> No symbol table info available. >>>>> #3 0x00007f1b4b710719 in connection_threadmain () >>>>> No symbol table info available. >>>>> #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so >>>>> No symbol table info available. >>>>> #5 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 >>>>> No symbol table info available. >>>>> #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 >>>>> No symbol table info available. >>>>> >>>>> Thread 33 (Thread 0x7f1b087f0700 (LWP 105193)): >>>>> #0 0x00007f1b48dc56d5 in >>>>> pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 >>>>> No symbol table info available. >>>>> #1 0x00007f1b4941b050 in PR_WaitCondVar () from /lib64/libnspr4.so >>>>> No symbol table info available. >>>>> #2 0x00007f1b4b70efee in connection_wait_for_new_work () >>>>> No symbol table info available. >>>>> #3 0x00007f1b4b710719 in connection_threadmain () >>>>> No symbol table info available. >>>>> #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so >>>>> No symbol table info available. >>>>> #5 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 >>>>> No symbol table info available. >>>>> #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 >>>>> No symbol table info available. >>>>> >>>>> Thread 32 (Thread 0x7f1b08ff1700 (LWP 105192)): >>>>> #0 0x00007f1b48dc56d5 in >>>>> pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 >>>>> No symbol table info available. >>>>> #1 0x00007f1b4941b050 in PR_WaitCondVar () from /lib64/libnspr4.so >>>>> No symbol table info available. >>>>> #2 0x00007f1b4b70efee in connection_wait_for_new_work () >>>>> No symbol table info available. >>>>> #3 0x00007f1b4b710719 in connection_threadmain () >>>>> No symbol table info available. >>>>> #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so >>>>> No symbol table info available. >>>>> #5 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 >>>>> No symbol table info available. >>>>> #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 >>>>> No symbol table info available. >>>>> >>>>> Thread 31 (Thread 0x7f1b097f2700 (LWP 105191)): >>>>> #0 0x00007f1b48dc56d5 in >>>>> pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 >>>>> No symbol table info available. >>>>> #1 0x00007f1b4941b050 in PR_WaitCondVar () from /lib64/libnspr4.so >>>>> No symbol table info available. >>>>> #2 0x00007f1b4b70efee in connection_wait_for_new_work () >>>>> No symbol table info available. >>>>> #3 0x00007f1b4b710719 in connection_threadmain () >>>>> No symbol table info available. >>>>> #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so >>>>> No symbol table info available. >>>>> #5 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 >>>>> No symbol table info available. >>>>> #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 >>>>> No symbol table info available. >>>>> >>>>> Thread 30 (Thread 0x7f1b09ff3700 (LWP 105190)): >>>>> #0 0x00007f1b48dc56d5 in >>>>> pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 >>>>> No symbol table info available. >>>>> #1 0x00007f1b433212f3 in __db_hybrid_mutex_suspend () from /lib64/ >>>>> libdb-5.3.so >>>>> No symbol table info available. >>>>> #2 0x00007f1b43320640 in __db_tas_mutex_lock () from /lib64/ >>>>> libdb-5.3.so >>>>> No symbol table info available. >>>>> #3 0x00007f1b433cad3a in __lock_get_internal () from /lib64/ >>>>> libdb-5.3.so >>>>> No symbol table info available. >>>>> #4 0x00007f1b433cb820 in __lock_get () from /lib64/libdb-5.3.so >>>>> No symbol table info available. >>>>> #5 0x00007f1b433f7142 in __db_lget () from /lib64/libdb-5.3.so >>>>> No symbol table info available. >>>>> #6 0x00007f1b4333e605 in __bam_search () from /lib64/libdb-5.3.so >>>>> No symbol table info available. >>>>> #7 0x00007f1b43329256 in __bamc_search () from /lib64/libdb-5.3.so >>>>> No symbol table info available. >>>>> #8 0x00007f1b4332ad0f in __bamc_get () from /lib64/libdb-5.3.so >>>>> No symbol table info available. >>>>> #9 0x00007f1b433e3ca6 in __dbc_iget () from /lib64/libdb-5.3.so >>>>> No symbol table info available. >>>>> #10 0x00007f1b433f2b02 in __dbc_get_pp () from /lib64/libdb-5.3.so >>>>> No symbol table info available. >>>>> #11 0x00007f1b3e8ce680 in idl_new_fetch () from >>>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>>> No symbol table info available. >>>>> #12 0x00007f1b3e8dcb26 in index_read_ext_allids () from >>>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>>> No symbol table info available. >>>>> #13 0x00007f1b3e8c7284 in keys2idl () from >>>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>>> No symbol table info available. >>>>> #14 0x00007f1b3e8c79e3 in ava_candidates.isra.0 () from >>>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>>> No symbol table info available. >>>>> #15 0x00007f1b3e8c7fd2 in filter_candidates_ext () from >>>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>>> No symbol table info available. >>>>> #16 0x00007f1b3e8c9096 in list_candidates () from >>>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>>> No symbol table info available. >>>>> #17 0x00007f1b3e8c7f40 in filter_candidates_ext () from >>>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>>> No symbol table info available. >>>>> #18 0x00007f1b3e8c9096 in list_candidates () from >>>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>>> No symbol table info available. >>>>> #19 0x00007f1b3e8c7f40 in filter_candidates_ext () from >>>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>>> No symbol table info available. >>>>> #20 0x00007f1b3e8c9096 in list_candidates () from >>>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>>> No symbol table info available. >>>>> #21 0x00007f1b3e8c7f40 in filter_candidates_ext () from >>>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>>> No symbol table info available. >>>>> #22 0x00007f1b3e90380c in subtree_candidates () from >>>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>>> No symbol table info available. >>>>> #23 0x00007f1b3e904e7f in ldbm_back_search () from >>>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>>> No symbol table info available. >>>>> #24 0x00007f1b4b2321d6 in op_shared_search () from >>>>> /usr/lib64/dirsrv/libslapd.so.0 >>>>> No symbol table info available. >>>>> #25 0x00007f1b4b72333e in do_search () >>>>> No symbol table info available. >>>>> #26 0x00007f1b4b7113c3 in connection_threadmain () >>>>> No symbol table info available. >>>>> #27 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so >>>>> No symbol table info available. >>>>> #28 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 >>>>> No symbol table info available. >>>>> #29 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 >>>>> No symbol table info available. >>>>> >>>>> Thread 29 (Thread 0x7f1b0a7f4700 (LWP 105189)): >>>>> #0 0x00007f1b48dc56d5 in >>>>> pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 >>>>> No symbol table info available. >>>>> #1 0x00007f1b4941b050 in PR_WaitCondVar () from /lib64/libnspr4.so >>>>> No symbol table info available. >>>>> #2 0x00007f1b4b70efee in connection_wait_for_new_work () >>>>> No symbol table info available. >>>>> #3 0x00007f1b4b710719 in connection_threadmain () >>>>> No symbol table info available. >>>>> #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so >>>>> No symbol table info available. >>>>> #5 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 >>>>> No symbol table info available. >>>>> #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 >>>>> No symbol table info available. >>>>> >>>>> Thread 28 (Thread 0x7f1b0aff5700 (LWP 105188)): >>>>> #0 0x00007f1b48dc56d5 in >>>>> pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 >>>>> No symbol table info available. >>>>> #1 0x00007f1b4941b050 in PR_WaitCondVar () from /lib64/libnspr4.so >>>>> No symbol table info available. >>>>> #2 0x00007f1b4b70efee in connection_wait_for_new_work () >>>>> No symbol table info available. >>>>> #3 0x00007f1b4b710719 in connection_threadmain () >>>>> No symbol table info available. >>>>> #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so >>>>> No symbol table info available. >>>>> #5 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 >>>>> No symbol table info available. >>>>> #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 >>>>> No symbol table info available. >>>>> >>>>> Thread 27 (Thread 0x7f1b0b7f6700 (LWP 105187)): >>>>> #0 0x00007f1b48dc56d5 in >>>>> pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 >>>>> No symbol table info available. >>>>> #1 0x00007f1b4941b050 in PR_WaitCondVar () from /lib64/libnspr4.so >>>>> No symbol table info available. >>>>> #2 0x00007f1b4b70efee in connection_wait_for_new_work () >>>>> No symbol table info available. >>>>> #3 0x00007f1b4b710719 in connection_threadmain () >>>>> No symbol table info available. >>>>> #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so >>>>> No symbol table info available. >>>>> #5 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 >>>>> No symbol table info available. >>>>> #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 >>>>> No symbol table info available. >>>>> >>>>> Thread 26 (Thread 0x7f1b0bff7700 (LWP 105186)): >>>>> #0 0x00007f1b48dc56d5 in >>>>> pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 >>>>> No symbol table info available. >>>>> #1 0x00007f1b433212f3 in __db_hybrid_mutex_suspend () from /lib64/ >>>>> libdb-5.3.so >>>>> No symbol table info available. >>>>> #2 0x00007f1b43320640 in __db_tas_mutex_lock () from /lib64/ >>>>> libdb-5.3.so >>>>> No symbol table info available. >>>>> #3 0x00007f1b433cad3a in __lock_get_internal () from /lib64/ >>>>> libdb-5.3.so >>>>> No symbol table info available. >>>>> #4 0x00007f1b433cb820 in __lock_get () from /lib64/libdb-5.3.so >>>>> No symbol table info available. >>>>> #5 0x00007f1b433f7142 in __db_lget () from /lib64/libdb-5.3.so >>>>> No symbol table info available. >>>>> #6 0x00007f1b4333e605 in __bam_search () from /lib64/libdb-5.3.so >>>>> No symbol table info available. >>>>> #7 0x00007f1b43329256 in __bamc_search () from /lib64/libdb-5.3.so >>>>> No symbol table info available. >>>>> #8 0x00007f1b4332ad0f in __bamc_get () from /lib64/libdb-5.3.so >>>>> No symbol table info available. >>>>> #9 0x00007f1b433e3ca6 in __dbc_iget () from /lib64/libdb-5.3.so >>>>> No symbol table info available. >>>>> #10 0x00007f1b433f2b02 in __dbc_get_pp () from /lib64/libdb-5.3.so >>>>> No symbol table info available. >>>>> #11 0x00007f1b3e8ce680 in idl_new_fetch () from >>>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>>> No symbol table info available. >>>>> #12 0x00007f1b3e8dcb26 in index_read_ext_allids () from >>>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>>> No symbol table info available. >>>>> #13 0x00007f1b3e8c7284 in keys2idl () from >>>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>>> No symbol table info available. >>>>> #14 0x00007f1b3e8c79e3 in ava_candidates.isra.0 () from >>>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>>> No symbol table info available. >>>>> #15 0x00007f1b3e8c7fd2 in filter_candidates_ext () from >>>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>>> No symbol table info available. >>>>> #16 0x00007f1b3e8c9096 in list_candidates () from >>>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>>> No symbol table info available. >>>>> #17 0x00007f1b3e8c7f40 in filter_candidates_ext () from >>>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>>> No symbol table info available. >>>>> #18 0x00007f1b3e90380c in subtree_candidates () from >>>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>>> No symbol table info available. >>>>> #19 0x00007f1b3e904e7f in ldbm_back_search () from >>>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>>> No symbol table info available. >>>>> #20 0x00007f1b4b2321d6 in op_shared_search () from >>>>> /usr/lib64/dirsrv/libslapd.so.0 >>>>> No symbol table info available. >>>>> #21 0x00007f1b4b24294e in search_internal_callback_pb () from >>>>> /usr/lib64/dirsrv/libslapd.so.0 >>>>> No symbol table info available. >>>>> #22 0x00007f1b4b242be8 in search_internal_pb () from >>>>> /usr/lib64/dirsrv/libslapd.so.0 >>>>> No symbol table info available. >>>>> #23 0x00007f1b4b72149b in ids_sasl_canon_user () >>>>> No symbol table info available. >>>>> #24 0x00007f1b491e193b in _sasl_canon_user () from /lib64/libsasl2.so.3 >>>>> No symbol table info available. >>>>> #25 0x00007f1b491e1c4c in _sasl_canon_user_lookup () from >>>>> /lib64/libsasl2.so.3 >>>>> No symbol table info available. >>>>> #26 0x00007f1b430e9fb9 in gssapi_server_mech_step () from >>>>> /usr/lib64/sasl2/libgssapiv2.so >>>>> No symbol table info available. >>>>> #27 0x00007f1b491edb9b in sasl_server_step () from /lib64/libsasl2.so.3 >>>>> No symbol table info available. >>>>> #28 0x00007f1b4b72283b in ids_sasl_check_bind () >>>>> No symbol table info available. >>>>> #29 0x00007f1b4b70a61a in do_bind () >>>>> No symbol table info available. >>>>> #30 0x00007f1b4b7113fd in connection_threadmain () >>>>> No symbol table info available. >>>>> #31 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so >>>>> No symbol table info available. >>>>> #32 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 >>>>> No symbol table info available. >>>>> #33 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 >>>>> No symbol table info available. >>>>> >>>>> Thread 25 (Thread 0x7f1b0c7f8700 (LWP 105185)): >>>>> #0 0x00007f1b48dc56d5 in >>>>> pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 >>>>> No symbol table info available. >>>>> #1 0x00007f1b4941b050 in PR_WaitCondVar () from /lib64/libnspr4.so >>>>> No symbol table info available. >>>>> #2 0x00007f1b4b70efee in connection_wait_for_new_work () >>>>> No symbol table info available. >>>>> #3 0x00007f1b4b710719 in connection_threadmain () >>>>> No symbol table info available. >>>>> #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so >>>>> No symbol table info available. >>>>> #5 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 >>>>> No symbol table info available. >>>>> #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 >>>>> No symbol table info available. >>>>> >>>>> Thread 24 (Thread 0x7f1b0cff9700 (LWP 105184)): >>>>> #0 0x00007f1b48dc56d5 in >>>>> pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 >>>>> No symbol table info available. >>>>> #1 0x00007f1b4941b050 in PR_WaitCondVar () from /lib64/libnspr4.so >>>>> No symbol table info available. >>>>> #2 0x00007f1b4b70efee in connection_wait_for_new_work () >>>>> No symbol table info available. >>>>> #3 0x00007f1b4b710719 in connection_threadmain () >>>>> No symbol table info available. >>>>> #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so >>>>> No symbol table info available. >>>>> #5 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 >>>>> No symbol table info available. >>>>> #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 >>>>> No symbol table info available. >>>>> >>>>> Thread 23 (Thread 0x7f1b0d7fa700 (LWP 105183)): >>>>> #0 0x00007f1b48dc56d5 in >>>>> pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 >>>>> No symbol table info available. >>>>> #1 0x00007f1b4941b050 in PR_WaitCondVar () from /lib64/libnspr4.so >>>>> No symbol table info available. >>>>> #2 0x00007f1b4b70efee in connection_wait_for_new_work () >>>>> No symbol table info available. >>>>> #3 0x00007f1b4b710719 in connection_threadmain () >>>>> No symbol table info available. >>>>> #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so >>>>> No symbol table info available. >>>>> #5 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 >>>>> No symbol table info available. >>>>> #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 >>>>> No symbol table info available. >>>>> >>>>> Thread 22 (Thread 0x7f1b0dffb700 (LWP 105182)): >>>>> #0 0x00007f1b48dc56d5 in >>>>> pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 >>>>> No symbol table info available. >>>>> #1 0x00007f1b4941b050 in PR_WaitCondVar () from /lib64/libnspr4.so >>>>> No symbol table info available. >>>>> #2 0x00007f1b4b70efee in connection_wait_for_new_work () >>>>> No symbol table info available. >>>>> #3 0x00007f1b4b710719 in connection_threadmain () >>>>> No symbol table info available. >>>>> #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so >>>>> No symbol table info available. >>>>> #5 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 >>>>> No symbol table info available. >>>>> #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 >>>>> No symbol table info available. >>>>> >>>>> Thread 21 (Thread 0x7f1b0e7fc700 (LWP 105181)): >>>>> #0 0x00007f1b48dc56d5 in >>>>> pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 >>>>> No symbol table info available. >>>>> #1 0x00007f1b4941b050 in PR_WaitCondVar () from /lib64/libnspr4.so >>>>> No symbol table info available. >>>>> #2 0x00007f1b4b70efee in connection_wait_for_new_work () >>>>> No symbol table info available. >>>>> #3 0x00007f1b4b710719 in connection_threadmain () >>>>> No symbol table info available. >>>>> #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so >>>>> No symbol table info available. >>>>> #5 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 >>>>> No symbol table info available. >>>>> #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 >>>>> No symbol table info available. >>>>> >>>>> Thread 20 (Thread 0x7f1b0effd700 (LWP 105180)): >>>>> #0 0x00007f1b48dc56d5 in >>>>> pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 >>>>> No symbol table info available. >>>>> #1 0x00007f1b433212f3 in __db_hybrid_mutex_suspend () from /lib64/ >>>>> libdb-5.3.so >>>>> No symbol table info available. >>>>> #2 0x00007f1b43320b37 in __db_tas_mutex_readlock () from /lib64/ >>>>> libdb-5.3.so >>>>> No symbol table info available. >>>>> #3 0x00007f1b4343ad35 in __memp_fget () from /lib64/libdb-5.3.so >>>>> No symbol table info available. >>>>> #4 0x00007f1b433e4333 in __dbc_iget () from /lib64/libdb-5.3.so >>>>> No symbol table info available. >>>>> #5 0x00007f1b433f2b02 in __dbc_get_pp () from /lib64/libdb-5.3.so >>>>> No symbol table info available. >>>>> #6 0x00007f1b3e8ce822 in idl_new_fetch () from >>>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>>> No symbol table info available. >>>>> #7 0x00007f1b3e8dcb26 in index_read_ext_allids () from >>>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>>> No symbol table info available. >>>>> #8 0x00007f1b3e8c7284 in keys2idl () from >>>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>>> No symbol table info available. >>>>> #9 0x00007f1b3e8c79e3 in ava_candidates.isra.0 () from >>>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>>> No symbol table info available. >>>>> #10 0x00007f1b3e8c7fd2 in filter_candidates_ext () from >>>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>>> No symbol table info available. >>>>> #11 0x00007f1b3e8c9096 in list_candidates () from >>>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>>> No symbol table info available. >>>>> #12 0x00007f1b3e8c7f40 in filter_candidates_ext () from >>>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>>> No symbol table info available. >>>>> #13 0x00007f1b3e8c9096 in list_candidates () from >>>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>>> No symbol table info available. >>>>> #14 0x00007f1b3e8c7f40 in filter_candidates_ext () from >>>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>>> No symbol table info available. >>>>> #15 0x00007f1b3e8c9096 in list_candidates () from >>>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>>> No symbol table info available. >>>>> #16 0x00007f1b3e8c7f40 in filter_candidates_ext () from >>>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>>> No symbol table info available. >>>>> #17 0x00007f1b3e90380c in subtree_candidates () from >>>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>>> No symbol table info available. >>>>> #18 0x00007f1b3e904e7f in ldbm_back_search () from >>>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>>> No symbol table info available. >>>>> #19 0x00007f1b4b2321d6 in op_shared_search () from >>>>> /usr/lib64/dirsrv/libslapd.so.0 >>>>> No symbol table info available. >>>>> #20 0x00007f1b4b72333e in do_search () >>>>> No symbol table info available. >>>>> #21 0x00007f1b4b7113c3 in connection_threadmain () >>>>> No symbol table info available. >>>>> #22 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so >>>>> No symbol table info available. >>>>> #23 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 >>>>> No symbol table info available. >>>>> #24 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 >>>>> No symbol table info available. >>>>> >>>>> Thread 19 (Thread 0x7f1b0f7fe700 (LWP 105179)): >>>>> #0 0x00007f1b48dc56d5 in >>>>> pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 >>>>> No symbol table info available. >>>>> #1 0x00007f1b433212f3 in __db_hybrid_mutex_suspend () from /lib64/ >>>>> libdb-5.3.so >>>>> No symbol table info available. >>>>> #2 0x00007f1b43320640 in __db_tas_mutex_lock () from /lib64/ >>>>> libdb-5.3.so >>>>> No symbol table info available. >>>>> #3 0x00007f1b433cad3a in __lock_get_internal () from /lib64/ >>>>> libdb-5.3.so >>>>> No symbol table info available. >>>>> #4 0x00007f1b433cb820 in __lock_get () from /lib64/libdb-5.3.so >>>>> No symbol table info available. >>>>> #5 0x00007f1b433f7142 in __db_lget () from /lib64/libdb-5.3.so >>>>> No symbol table info available. >>>>> #6 0x00007f1b4333e605 in __bam_search () from /lib64/libdb-5.3.so >>>>> No symbol table info available. >>>>> #7 0x00007f1b43329256 in __bamc_search () from /lib64/libdb-5.3.so >>>>> No symbol table info available. >>>>> #8 0x00007f1b4332ad0f in __bamc_get () from /lib64/libdb-5.3.so >>>>> No symbol table info available. >>>>> #9 0x00007f1b433e3ca6 in __dbc_iget () from /lib64/libdb-5.3.so >>>>> No symbol table info available. >>>>> #10 0x00007f1b433f2b02 in __dbc_get_pp () from /lib64/libdb-5.3.so >>>>> No symbol table info available. >>>>> #11 0x00007f1b3e8ce680 in idl_new_fetch () from >>>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>>> No symbol table info available. >>>>> #12 0x00007f1b3e8dcb26 in index_read_ext_allids () from >>>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>>> No symbol table info available. >>>>> #13 0x00007f1b3e8c7284 in keys2idl () from >>>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>>> No symbol table info available. >>>>> #14 0x00007f1b3e8c79e3 in ava_candidates.isra.0 () from >>>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>>> No symbol table info available. >>>>> #15 0x00007f1b3e8c7fd2 in filter_candidates_ext () from >>>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>>> No symbol table info available. >>>>> #16 0x00007f1b3e8c9096 in list_candidates () from >>>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>>> No symbol table info available. >>>>> #17 0x00007f1b3e8c7f40 in filter_candidates_ext () from >>>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>>> No symbol table info available. >>>>> #18 0x00007f1b3e8c9096 in list_candidates () from >>>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>>> No symbol table info available. >>>>> #19 0x00007f1b3e8c7f40 in filter_candidates_ext () from >>>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>>> No symbol table info available. >>>>> #20 0x00007f1b3e8c9096 in list_candidates () from >>>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>>> No symbol table info available. >>>>> #21 0x00007f1b3e8c7f40 in filter_candidates_ext () from >>>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>>> No symbol table info available. >>>>> #22 0x00007f1b3e90380c in subtree_candidates () from >>>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>>> No symbol table info available. >>>>> #23 0x00007f1b3e904e7f in ldbm_back_search () from >>>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>>> No symbol table info available. >>>>> #24 0x00007f1b4b2321d6 in op_shared_search () from >>>>> /usr/lib64/dirsrv/libslapd.so.0 >>>>> No symbol table info available. >>>>> #25 0x00007f1b4b72333e in do_search () >>>>> No symbol table info available. >>>>> #26 0x00007f1b4b7113c3 in connection_threadmain () >>>>> No symbol table info available. >>>>> #27 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so >>>>> No symbol table info available. >>>>> #28 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 >>>>> No symbol table info available. >>>>> #29 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 >>>>> No symbol table info available. >>>>> >>>>> Thread 18 (Thread 0x7f1b0ffff700 (LWP 105178)): >>>>> #0 0x00007f1b48dc56d5 in >>>>> pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 >>>>> No symbol table info available. >>>>> #1 0x00007f1b4941b050 in PR_WaitCondVar () from /lib64/libnspr4.so >>>>> No symbol table info available. >>>>> #2 0x00007f1b4b70efee in connection_wait_for_new_work () >>>>> No symbol table info available. >>>>> #3 0x00007f1b4b710719 in connection_threadmain () >>>>> No symbol table info available. >>>>> #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so >>>>> No symbol table info available. >>>>> #5 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 >>>>> No symbol table info available. >>>>> #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 >>>>> No symbol table info available. >>>>> >>>>> Thread 17 (Thread 0x7f1b14d28700 (LWP 105177)): >>>>> #0 0x00007f1b48dc56d5 in >>>>> pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 >>>>> No symbol table info available. >>>>> #1 0x00007f1b4941b050 in PR_WaitCondVar () from /lib64/libnspr4.so >>>>> No symbol table info available. >>>>> #2 0x00007f1b4b70efee in connection_wait_for_new_work () >>>>> No symbol table info available. >>>>> #3 0x00007f1b4b710719 in connection_threadmain () >>>>> No symbol table info available. >>>>> #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so >>>>> No symbol table info available. >>>>> #5 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 >>>>> No symbol table info available. >>>>> #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 >>>>> No symbol table info available. >>>>> >>>>> Thread 16 (Thread 0x7f1b15529700 (LWP 105176)): >>>>> #0 0x00007f1b48dc56d5 in >>>>> pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 >>>>> No symbol table info available. >>>>> #1 0x00007f1b4941b050 in PR_WaitCondVar () from /lib64/libnspr4.so >>>>> No symbol table info available. >>>>> #2 0x00007f1b4b70efee in connection_wait_for_new_work () >>>>> No symbol table info available. >>>>> #3 0x00007f1b4b710719 in connection_threadmain () >>>>> No symbol table info available. >>>>> #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so >>>>> No symbol table info available. >>>>> #5 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 >>>>> No symbol table info available. >>>>> #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 >>>>> No symbol table info available. >>>>> >>>>> Thread 15 (Thread 0x7f1b15d2a700 (LWP 105175)): >>>>> #0 0x00007f1b48dc56d5 in >>>>> pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 >>>>> No symbol table info available. >>>>> #1 0x00007f1b4941b050 in PR_WaitCondVar () from /lib64/libnspr4.so >>>>> No symbol table info available. >>>>> #2 0x00007f1b4b70efee in connection_wait_for_new_work () >>>>> No symbol table info available. >>>>> #3 0x00007f1b4b710719 in connection_threadmain () >>>>> No symbol table info available. >>>>> #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so >>>>> No symbol table info available. >>>>> #5 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 >>>>> No symbol table info available. >>>>> #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 >>>>> No symbol table info available. >>>>> >>>>> Thread 14 (Thread 0x7f1b1652b700 (LWP 105174)): >>>>> #0 0x00007f1b48dc56d5 in >>>>> pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 >>>>> No symbol table info available. >>>>> #1 0x00007f1b4941b050 in PR_WaitCondVar () from /lib64/libnspr4.so >>>>> No symbol table info available. >>>>> #2 0x00007f1b4b70efee in connection_wait_for_new_work () >>>>> No symbol table info available. >>>>> #3 0x00007f1b4b710719 in connection_threadmain () >>>>> No symbol table info available. >>>>> #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so >>>>> No symbol table info available. >>>>> #5 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 >>>>> No symbol table info available. >>>>> #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 >>>>> No symbol table info available. >>>>> >>>>> Thread 13 (Thread 0x7f1b16d2c700 (LWP 105173)): >>>>> #0 0x00007f1b48dc56d5 in >>>>> pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 >>>>> No symbol table info available. >>>>> #1 0x00007f1b433212f3 in __db_hybrid_mutex_suspend () from /lib64/ >>>>> libdb-5.3.so >>>>> No symbol table info available. >>>>> #2 0x00007f1b43320640 in __db_tas_mutex_lock () from /lib64/ >>>>> libdb-5.3.so >>>>> No symbol table info available. >>>>> #3 0x00007f1b433cad3a in __lock_get_internal () from /lib64/ >>>>> libdb-5.3.so >>>>> No symbol table info available. >>>>> #4 0x00007f1b433cb820 in __lock_get () from /lib64/libdb-5.3.so >>>>> No symbol table info available. >>>>> #5 0x00007f1b433f7142 in __db_lget () from /lib64/libdb-5.3.so >>>>> No symbol table info available. >>>>> #6 0x00007f1b4333e605 in __bam_search () from /lib64/libdb-5.3.so >>>>> No symbol table info available. >>>>> #7 0x00007f1b43329256 in __bamc_search () from /lib64/libdb-5.3.so >>>>> No symbol table info available. >>>>> #8 0x00007f1b4332ad0f in __bamc_get () from /lib64/libdb-5.3.so >>>>> No symbol table info available. >>>>> #9 0x00007f1b433e3ca6 in __dbc_iget () from /lib64/libdb-5.3.so >>>>> No symbol table info available. >>>>> #10 0x00007f1b433f2b02 in __dbc_get_pp () from /lib64/libdb-5.3.so >>>>> No symbol table info available. >>>>> #11 0x00007f1b3e8ce680 in idl_new_fetch () from >>>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>>> No symbol table info available. >>>>> #12 0x00007f1b3e8dcb26 in index_read_ext_allids () from >>>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>>> No symbol table info available. >>>>> #13 0x00007f1b3e8c7284 in keys2idl () from >>>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>>> No symbol table info available. >>>>> #14 0x00007f1b3e8c79e3 in ava_candidates.isra.0 () from >>>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>>> No symbol table info available. >>>>> #15 0x00007f1b3e8c7fd2 in filter_candidates_ext () from >>>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>>> No symbol table info available. >>>>> #16 0x00007f1b3e8c9096 in list_candidates () from >>>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>>> No symbol table info available. >>>>> #17 0x00007f1b3e8c7f40 in filter_candidates_ext () from >>>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>>> No symbol table info available. >>>>> #18 0x00007f1b3e8c9096 in list_candidates () from >>>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>>> No symbol table info available. >>>>> #19 0x00007f1b3e8c7f40 in filter_candidates_ext () from >>>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>>> No symbol table info available. >>>>> #20 0x00007f1b3e8c9096 in list_candidates () from >>>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>>> No symbol table info available. >>>>> #21 0x00007f1b3e8c7f40 in filter_candidates_ext () from >>>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>>> No symbol table info available. >>>>> #22 0x00007f1b3e90380c in subtree_candidates () from >>>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>>> No symbol table info available. >>>>> #23 0x00007f1b3e904e7f in ldbm_back_search () from >>>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>>> No symbol table info available. >>>>> #24 0x00007f1b4b2321d6 in op_shared_search () from >>>>> /usr/lib64/dirsrv/libslapd.so.0 >>>>> No symbol table info available. >>>>> #25 0x00007f1b4b72333e in do_search () >>>>> No symbol table info available. >>>>> #26 0x00007f1b4b7113c3 in connection_threadmain () >>>>> No symbol table info available. >>>>> #27 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so >>>>> No symbol table info available. >>>>> #28 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 >>>>> No symbol table info available. >>>>> #29 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 >>>>> No symbol table info available. >>>>> >>>>> Thread 12 (Thread 0x7f1b1752d700 (LWP 105172)): >>>>> #0 0x00007f1b48dc56d5 in >>>>> pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 >>>>> No symbol table info available. >>>>> #1 0x00007f1b4941b050 in PR_WaitCondVar () from /lib64/libnspr4.so >>>>> No symbol table info available. >>>>> #2 0x00007f1b4b70efee in connection_wait_for_new_work () >>>>> No symbol table info available. >>>>> #3 0x00007f1b4b710719 in connection_threadmain () >>>>> No symbol table info available. >>>>> #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so >>>>> No symbol table info available. >>>>> #5 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 >>>>> No symbol table info available. >>>>> #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 >>>>> No symbol table info available. >>>>> >>>>> Thread 11 (Thread 0x7f1b17d2e700 (LWP 105171)): >>>>> #0 0x00007f1b48dc56d5 in >>>>> pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 >>>>> No symbol table info available. >>>>> #1 0x00007f1b4941b050 in PR_WaitCondVar () from /lib64/libnspr4.so >>>>> No symbol table info available. >>>>> #2 0x00007f1b4b70efee in connection_wait_for_new_work () >>>>> No symbol table info available. >>>>> #3 0x00007f1b4b710719 in connection_threadmain () >>>>> No symbol table info available. >>>>> #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so >>>>> No symbol table info available. >>>>> #5 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 >>>>> No symbol table info available. >>>>> #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 >>>>> No symbol table info available. >>>>> >>>>> Thread 10 (Thread 0x7f1b2509d700 (LWP 105168)): >>>>> #0 0x00007f1b48dc5a82 in pthread_cond_timedwait@@GLIBC_2.3.2 () from >>>>> /lib64/libpthread.so.0 >>>>> No symbol table info available. >>>>> #1 0x00007f1b4941ab07 in pt_TimedWait () from /lib64/libnspr4.so >>>>> No symbol table info available. >>>>> #2 0x00007f1b4941afce in PR_WaitCondVar () from /lib64/libnspr4.so >>>>> No symbol table info available. >>>>> #3 0x00007f1b4b2061e8 in eq_loop () from >>>>> /usr/lib64/dirsrv/libslapd.so.0 >>>>> No symbol table info available. >>>>> #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so >>>>> No symbol table info available. >>>>> #5 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 >>>>> No symbol table info available. >>>>> #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 >>>>> No symbol table info available. >>>>> >>>>> Thread 9 (Thread 0x7f1b2589e700 (LWP 105167)): >>>>> #0 0x00007f1b48dc5a82 in pthread_cond_timedwait@@GLIBC_2.3.2 () from >>>>> /lib64/libpthread.so.0 >>>>> No symbol table info available. >>>>> #1 0x00007f1b4941ab07 in pt_TimedWait () from /lib64/libnspr4.so >>>>> No symbol table info available. >>>>> #2 0x00007f1b4941afce in PR_WaitCondVar () from /lib64/libnspr4.so >>>>> No symbol table info available. >>>>> #3 0x00007f1b4b719d03 in housecleaning () >>>>> No symbol table info available. >>>>> #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so >>>>> No symbol table info available. >>>>> #5 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 >>>>> No symbol table info available. >>>>> #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 >>>>> No symbol table info available. >>>>> >>>>> Thread 8 (Thread 0x7f1b2609f700 (LWP 105166)): >>>>> #0 0x00007f1b48dc56d5 in >>>>> pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 >>>>> No symbol table info available. >>>>> #1 0x00007f1b4941b050 in PR_WaitCondVar () from /lib64/libnspr4.so >>>>> No symbol table info available. >>>>> #2 0x00007f1b4b260198 in slapi_wait_condvar () from >>>>> /usr/lib64/dirsrv/libslapd.so.0 >>>>> No symbol table info available. >>>>> #3 0x00007f1b3cf5deed in roles_cache_wait_on_change () from >>>>> /usr/lib64/dirsrv/plugins/libroles-plugin.so >>>>> No symbol table info available. >>>>> #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so >>>>> No symbol table info available. >>>>> #5 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 >>>>> No symbol table info available. >>>>> #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 >>>>> No symbol table info available. >>>>> >>>>> Thread 7 (Thread 0x7f1b268a0700 (LWP 105165)): >>>>> #0 0x00007f1b48dc56d5 in >>>>> pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 >>>>> No symbol table info available. >>>>> #1 0x00007f1b4941b050 in PR_WaitCondVar () from /lib64/libnspr4.so >>>>> No symbol table info available. >>>>> #2 0x00007f1b4b260198 in slapi_wait_condvar () from >>>>> /usr/lib64/dirsrv/libslapd.so.0 >>>>> No symbol table info available. >>>>> #3 0x00007f1b3cf5deed in roles_cache_wait_on_change () from >>>>> /usr/lib64/dirsrv/plugins/libroles-plugin.so >>>>> No symbol table info available. >>>>> #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so >>>>> No symbol table info available. >>>>> #5 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 >>>>> No symbol table info available. >>>>> #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 >>>>> No symbol table info available. >>>>> >>>>> Thread 6 (Thread 0x7f1b270a1700 (LWP 105164)): >>>>> #0 0x00007f1b48dc56d5 in >>>>> pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 >>>>> No symbol table info available. >>>>> #1 0x00007f1b4941b050 in PR_WaitCondVar () from /lib64/libnspr4.so >>>>> No symbol table info available. >>>>> #2 0x00007f1b4b260198 in slapi_wait_condvar () from >>>>> /usr/lib64/dirsrv/libslapd.so.0 >>>>> No symbol table info available. >>>>> #3 0x00007f1b3cf5deed in roles_cache_wait_on_change () from >>>>> /usr/lib64/dirsrv/plugins/libroles-plugin.so >>>>> No symbol table info available. >>>>> #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so >>>>> No symbol table info available. >>>>> #5 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 >>>>> No symbol table info available. >>>>> #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 >>>>> No symbol table info available. >>>>> >>>>> Thread 5 (Thread 0x7f1b278a2700 (LWP 105163)): >>>>> #0 0x00007f1b48dc56d5 in >>>>> pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 >>>>> No symbol table info available. >>>>> #1 0x00007f1b4941b050 in PR_WaitCondVar () from /lib64/libnspr4.so >>>>> No symbol table info available. >>>>> #2 0x00007f1b4b260198 in slapi_wait_condvar () from >>>>> /usr/lib64/dirsrv/libslapd.so.0 >>>>> No symbol table info available. >>>>> #3 0x00007f1b415f262e in cos_cache_wait_on_change () from >>>>> /usr/lib64/dirsrv/plugins/libcos-plugin.so >>>>> No symbol table info available. >>>>> #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so >>>>> No symbol table info available. >>>>> #5 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 >>>>> No symbol table info available. >>>>> #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 >>>>> No symbol table info available. >>>>> >>>>> Thread 4 (Thread 0x7f1b282ac700 (LWP 105162)): >>>>> #0 0x00007f1b48ae69b3 in select () from /lib64/libc.so.6 >>>>> No symbol table info available. >>>>> #1 0x00007f1b4b2710e9 in DS_Sleep () from >>>>> /usr/lib64/dirsrv/libslapd.so.0 >>>>> No symbol table info available. >>>>> #2 0x00007f1b3e8ba707 in perf_threadmain () from >>>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>>> No symbol table info available. >>>>> #3 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so >>>>> No symbol table info available. >>>>> #4 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 >>>>> No symbol table info available. >>>>> #5 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 >>>>> No symbol table info available. >>>>> >>>>> Thread 3 (Thread 0x7f1b28aad700 (LWP 105161)): >>>>> #0 0x00007f1b48ae69b3 in select () from /lib64/libc.so.6 >>>>> No symbol table info available. >>>>> #1 0x00007f1b4b2710e9 in DS_Sleep () from >>>>> /usr/lib64/dirsrv/libslapd.so.0 >>>>> No symbol table info available. >>>>> #2 0x00007f1b3e8bfb7f in trickle_threadmain () from >>>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>>> No symbol table info available. >>>>> #3 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so >>>>> No symbol table info available. >>>>> #4 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 >>>>> No symbol table info available. >>>>> #5 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 >>>>> No symbol table info available. >>>>> >>>>> Thread 2 (Thread 0x7f1b292ae700 (LWP 105160)): >>>>> #0 0x00007f1b48ae69b3 in select () from /lib64/libc.so.6 >>>>> No symbol table info available. >>>>> #1 0x00007f1b4344dbad in __os_yield () from /lib64/libdb-5.3.so >>>>> No symbol table info available. >>>>> #2 0x00007f1b434492b3 in __memp_sync_int () from /lib64/libdb-5.3.so >>>>> No symbol table info available. >>>>> #3 0x00007f1b43459752 in __txn_checkpoint () from /lib64/libdb-5.3.so >>>>> No symbol table info available. >>>>> #4 0x00007f1b43459b74 in __txn_checkpoint_pp () from /lib64/ >>>>> libdb-5.3.so >>>>> No symbol table info available. >>>>> #5 0x00007f1b3e8c3a87 in checkpoint_threadmain () from >>>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>>> No symbol table info available. >>>>> #6 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so >>>>> No symbol table info available. >>>>> #7 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 >>>>> No symbol table info available. >>>>> #8 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 >>>>> No symbol table info available. >>>>> >>>>> Thread 1 (Thread 0x7f1b29aaf700 (LWP 105159)): >>>>> #0 0x00007f1b48ae69b3 in select () from /lib64/libc.so.6 >>>>> No symbol table info available. >>>>> #1 0x00007f1b4b2710e9 in DS_Sleep () from >>>>> /usr/lib64/dirsrv/libslapd.so.0 >>>>> No symbol table info available. >>>>> #2 0x00007f1b3e8bf907 in deadlock_threadmain () from >>>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>>> No symbol table info available. >>>>> #3 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so >>>>> No symbol table info available. >>>>> #4 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 >>>>> No symbol table info available. >>>>> #5 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 >>>>> No symbol table info available. >>>>> >>>>> >>>>> >>>>> Thanks, >>>>> >>>>> Rakesh >>>>> >>>>> >>>>> On Mon, Sep 5, 2016 at 4:48 PM, thierry bordaz < >>>>> tbordaz at redhat.com> wrote: >>>>> >>>>>> >>>>>> >>>>>> On 09/05/2016 12:05 PM, Rakesh Rajasekharan wrote: >>>>>> >>>>>> Hi Thierry, >>>>>> >>>>>> >>>>>> I was getting the hang issue while running ipa-client-install >>>>>> simultaneously on few clients.. >>>>>> However, today, I am not able to replicate that. >>>>>> >>>>>> I could not get a gdb . But i will try getting that the next time I >>>>>> face this issue. >>>>>> >>>>>> The CPU does not stay high.. it just momentarily touches a high value >>>>>> and then drops down to around 2-7% >>>>>> >>>>>> One question I have is , is it ok to set it nsslapd-threadnumber to a >>>>>> very high value . >>>>>> I have around 4000 clients and with nsslapd-maxthreadsperconn set to >>>>>> 5..So, can I set nsslapd-threadnumber to around 25000. >>>>>> >>>>>> >>>>>> Hello, >>>>>> >>>>>> I know some users running in production with several hunderds of >>>>>> threads (>600) and this without problem. >>>>>> >>>>>> I do not recall having suggested to increase that number and for what >>>>>> reason. >>>>>> Usually 30 workers is a good enough value. It can create bootleneck >>>>>> if for some reason each operation is very long to satisfy and exhaust the >>>>>> number of workers. You can monitor the work queue: >>>>>> >>>>>> ldapsearch -D "cn=directory manager" -w xxx -LLL -b "cn=monitor" -s >>>>>> base opsinitiated opscompleted >>>>>> >>>>>> >>>>>> If opscompleted-opsinitiated remains close to threadnumber, then yes >>>>>> it would be valuable to increase it. >>>>>> >>>>>> The computation #client * #async_op_per_client sound an overkill. >>>>>> Even if all clients send at the exact same time all their requests, it is >>>>>> very likely that some common resource (db page, log, allocator...) will >>>>>> serialize them. If you monitor a need to increase the work, you would for >>>>>> example set it to 50, then monitor, then set it to 100, then monitor... >>>>>> until you find a good enough value. >>>>>> Note the increasing the #thread, increases the memory footprint that >>>>>> will reduce the efficiency of file system cache and can increase the >>>>>> response time. >>>>>> >>>>>> >>>>>> best regards >>>>>> thierry >>>>>> >>>>>> >>>>>> Thanks >>>>>> >>>>>> On Mon, Sep 5, 2016 at 1:03 PM, thierry bordaz < >>>>>> tbordaz at redhat.com> wrote: >>>>>> >>>>>>> >>>>>>> Hi Rakesh, >>>>>>> >>>>>>> Were you able to get a pstack or full stack with gdb ( >>>>>>> http://www.port389.org/docs/3 >>>>>>> 89ds/FAQ/faq.html#debugging-crashes) when the server hangs ? >>>>>>> >>>>>>> If it happens with 500 threads as well as with 30, using 30 threads >>>>>>> is a better choice to debug this issue. >>>>>>> I will try to reproduce using 150 parallel 'ipa user-find p-testipa' >>>>>>> commands >>>>>>> >>>>>>> Something I am unsure is if the CPU consumption stays high (you >>>>>>> mentioned 340% CPU usage) as long as the hang happens or if after a suddent >>>>>>> shot up to 340% (that marks the beginning of the hang) it drops and stay >>>>>>> hanging ? >>>>>>> >>>>>>> thanks >>>>>>> thierry >>>>>>> >>>>>>> On 09/04/2016 08:40 PM, Rakesh Rajasekharan wrote: >>>>>>> >>>>>>> starce on the slapd process actually had this in the output.. >>>>>>> FUTEX_WAIT_PRIVATE >>>>>>> >>>>>>> and checking for the number of threads slapd had.. there were 5015 >>>>>>> threads >>>>>>> >>>>>>> ps -efL|grep slapd|wc -l >>>>>>> 5015 >>>>>>> >>>>>>> strace on most of the threads gave this output >>>>>>> >>>>>>> strace -p 67411 >>>>>>> Process 67411 attached >>>>>>> futex(0x7f3f0226b41c, FUTEX_WAIT_PRIVATE, 1, NULL) = -1 EAGAIN >>>>>>> (Resource temporarily unavailable) >>>>>>> futex(0x7f3f0226b41c, FUTEX_WAIT_PRIVATE, 2, NULL^CProcess 67411 >>>>>>> detached >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> On Sun, Sep 4, 2016 at 5:34 PM, Rakesh Rajasekharan < >>>>>>> rakesh.rajasekharan at gmail.com> wrote: >>>>>>> >>>>>>>> I have again got the issue of IPA hanging.. The issue came up when >>>>>>>> i tried to run ipa-client-isntall on 142 clients simultaneously >>>>>>>> >>>>>>>> >>>>>>>> None of the IPA commands are responding, and I see this error >>>>>>>> >>>>>>>> ipa user-find p-testipa >>>>>>>> ipa: ERROR: Insufficient access: SASL(-1): generic failure: GSSAPI >>>>>>>> Error: Unspecified GSS failure. Minor code may provide more information >>>>>>>> (KDC returned error string: PROCESS_TGS) >>>>>>>> >>>>>>>> KRB5_TRACE=/dev/stdout kinit admin >>>>>>>> [41178] 1472984115.233214: Getting initial credentials for >>>>>>>> admin at XYZ.COM >>>>>>>> [41178] 1472984115.235257: Sending request (167 bytes) to XYZ.COM >>>>>>>> [41178] 1472984115.235419: Initiating TCP connection to stream >>>>>>>> 10.1.3.36:88 >>>>>>>> [41178] 1472984115.235685: Sending TCP request to stream >>>>>>>> 10.1.3.36:88 >>>>>>>> [41178] 1472984120.238914: Received answer (174 bytes) from stream >>>>>>>> 10.1.3.36:88 >>>>>>>> [41178] 1472984120.238925: Terminating TCP connection to stream >>>>>>>> 10.1.3.36:88 >>>>>>>> [41178] 1472984120.238993: Response was from master KDC >>>>>>>> [41 >>>>>>>> >>>>>>>> >>>>>>>> Running an ldapsearch to see the db.. does not give any results and >>>>>>>> just hangs there >>>>>>>> >>>>>>>> ldapsearch -x -D 'cn=Directory Manager' -W -s one -b >>>>>>>> 'cn=kerberos,dc=xyz,dc=com' >>>>>>>> Enter LDAP Password: >>>>>>>> >>>>>>>> even an ldapsearch -x does not respond >>>>>>>> At this point, am sure that slapd is the one causing issues >>>>>>>> >>>>>>>> Running an strace against the hung slapd itself seems to get stuck >>>>>>>> does not proceed after saying "attaching to process" >>>>>>>> >>>>>>>> From some others posts I read Thierry suggesting to increase the >>>>>>>> nsslapd-threadnumber value >>>>>>>> >>>>>>>> It was set to 30, I think that might be too low. >>>>>>>> >>>>>>> ... >>>>> >>>>> [Message clipped] >>>> >>>> >>>> >>> >>> >>> -- >>> Red Hat GmbH, http://www.de.redhat.com/, Registered seat: Grasbrunn, >>> Commercial register: Amtsgericht Muenchen, HRB 153243, >>> Managing Directors: Charles Cachera, Michael Cunningham, Michael O'Neill, Eric Shander >>> >>> >> > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From abokovoy at redhat.com Thu Sep 15 19:17:09 2016 From: abokovoy at redhat.com (Alexander Bokovoy) Date: Thu, 15 Sep 2016 22:17:09 +0300 Subject: [Freeipa-users] how to revert ipa-adtrust-install... In-Reply-To: <1ec82d0c-1350-feb3-4da5-a022eb7ac5e5@yahoo.co.uk> References: <1ec82d0c-1350-feb3-4da5-a022eb7ac5e5@yahoo.co.uk> Message-ID: <20160915191709.6vk7y3gidzxoicen@redhat.com> On Thu, 15 Sep 2016, lejeczek wrote: >... in a sense so IPA would keep away from local smb services? Not supported. -- / Alexander Bokovoy From peljasz at yahoo.co.uk Thu Sep 15 19:54:42 2016 From: peljasz at yahoo.co.uk (lejeczek) Date: Thu, 15 Sep 2016 20:54:42 +0100 Subject: [Freeipa-users] how to revert ipa-adtrust-install... In-Reply-To: <20160915191709.6vk7y3gidzxoicen@redhat.com> References: <1ec82d0c-1350-feb3-4da5-a022eb7ac5e5@yahoo.co.uk> <20160915191709.6vk7y3gidzxoicen@redhat.com> Message-ID: <98df2894-d5c7-082c-8fe8-77ece9474301@yahoo.co.uk> is there any way to tell IPA not to control smb.service? On 15/09/16 20:17, Alexander Bokovoy wrote: > On Thu, 15 Sep 2016, lejeczek wrote: >> ... in a sense so IPA would keep away from local smb >> services? > Not supported. > From abokovoy at redhat.com Thu Sep 15 20:00:55 2016 From: abokovoy at redhat.com (Alexander Bokovoy) Date: Thu, 15 Sep 2016 23:00:55 +0300 Subject: [Freeipa-users] how to revert ipa-adtrust-install... In-Reply-To: <98df2894-d5c7-082c-8fe8-77ece9474301@yahoo.co.uk> References: <1ec82d0c-1350-feb3-4da5-a022eb7ac5e5@yahoo.co.uk> <20160915191709.6vk7y3gidzxoicen@redhat.com> <98df2894-d5c7-082c-8fe8-77ece9474301@yahoo.co.uk> Message-ID: <20160915200055.qpx7gma722tjcfxk@redhat.com> On Thu, 15 Sep 2016, lejeczek wrote: >is there any way to tell IPA not to control smb.service? Do not run ipa-adtrust-install on the IPA master. -- / Alexander Bokovoy From rcritten at redhat.com Thu Sep 15 21:37:11 2016 From: rcritten at redhat.com (Rob Crittenden) Date: Thu, 15 Sep 2016 17:37:11 -0400 Subject: [Freeipa-users] how to revert ipa-adtrust-install... In-Reply-To: <20160915200055.qpx7gma722tjcfxk@redhat.com> References: <1ec82d0c-1350-feb3-4da5-a022eb7ac5e5@yahoo.co.uk> <20160915191709.6vk7y3gidzxoicen@redhat.com> <98df2894-d5c7-082c-8fe8-77ece9474301@yahoo.co.uk> <20160915200055.qpx7gma722tjcfxk@redhat.com> Message-ID: <57DB1487.4000500@redhat.com> Alexander Bokovoy wrote: > On Thu, 15 Sep 2016, lejeczek wrote: >> is there any way to tell IPA not to control smb.service? > Do not run ipa-adtrust-install on the IPA master. > What do you mean control? If you don't want ipactl to manage the smb service, look for an entry in cn=masters,cn=ipa,cn=etc,dc=example,dc=com and delete it if you find it. rob From abrook at bsd.uchicago.edu Thu Sep 15 20:29:28 2016 From: abrook at bsd.uchicago.edu (Brook, Andy [CRI]) Date: Thu, 15 Sep 2016 20:29:28 +0000 Subject: [Freeipa-users] Samba Server setup In-Reply-To: <20160915180606.v2z6dkdw3gltem34@redhat.com> References: <73C11E05-045F-4324-9733-44C370584232@bsd.uchicago.edu> <20160915180606.v2z6dkdw3gltem34@redhat.com> Message-ID: <9A3517B6-CFFB-47ED-BDF2-BE3EE0D06EDE@bsd.uchicago.edu> On 9/15/16, 1:06 PM, "Alexander Bokovoy" wrote: On Thu, 15 Sep 2016, Brook, Andy [CRI] wrote: >All, > I?m working on setting up Samba to serve files from a server attached > to our IPA domain. I followed the directions in > https://www.freeipa.org/page/Howto/Integrating_a_Samba_File_Server_With_IPA. > Everything seems to work and I can access the files from another RHEL > server attached to the same domain using a Kerberos ticket from a > user from the trusted AD domain. However, I can?t access this share > from a windows client that is also attached to the trusted AD domain. > >My smb.conf is as follows: >[global] > workgroup = IPA > realm = IPA.DOMAIN > kerberos method = dedicated keytab > dedicated keytab file = FILE:/etc/samba/samba.keytab > log file = /var/log/samba/log.%m > log level = 3 > security = ads > load printers = no > disable spoolss = yes > map to guest = Never > restrict anonymous = 2 > >[spacetest] > path = /var/www > writable = yes > browsable = yes > >I put the keytab in place from the cifs service from the IPA server. > >I feel like I?m missing something small, but I can?t seem to find it. >Logs from samba are here: http://pastebin.com/aMDXfR78 These logs show that your Windows client did not use Kerberos but tried to authenticate with password using NTLMSSP. This is not supported yet, as written on the page you used for the setup guidance. You need to find out why Windows client didn't use Kerberos. Is your trust to AD really working? We?re authenticating AD users on the hosts that are connected to IPA. We?re able to create external groups and associate them with internal groups for HBAC and sudoers rules. Is there something else I should check to see if the trust is working? It?s entirely possible I missed something somewhere in the setup, but I don?t think I did. Andy Brook Sr. Systems Administrator | Center for Research Informatics | University of Chicago T: 773-834-0458 | http://cri.uchicago.edu ******************************************************************************** This e-mail is intended only for the use of the individual or entity to which it is addressed and may contain information that is privileged and confidential. If the reader of this e-mail message is not the intended recipient, you are hereby notified that any dissemination, distribution or copying of this communication is prohibited. If you have received this e-mail in error, please notify the sender and destroy all copies of the transmittal. Thank you University of Chicago Medicine and Biological Sciences ******************************************************************************** From rakesh.rajasekharan at gmail.com Fri Sep 16 04:50:46 2016 From: rakesh.rajasekharan at gmail.com (Rakesh Rajasekharan) Date: Fri, 16 Sep 2016 10:20:46 +0530 Subject: [Freeipa-users] Freeipa 4.2.0 hangs intermittently In-Reply-To: References: <57C44AC6.1030402@redhat.com> <57CD1FDC.2010109@redhat.com> <57CD5488.60404@redhat.com> <57CD659B.6050707@redhat.com> <57CD68BD.9030802@redhat.com> <57CE8CDB.6000809@redhat.com> <57CE9DE5.6020301@redhat.com> <57DACE76.5070803@redhat.com> Message-ID: My client and IPA server times are in sync.. so not sure why the clock skew message comes On Fri, Sep 16, 2016 at 12:44 AM, Rakesh Rajasekharan < rakesh.rajasekharan at gmail.com> wrote: > Hi Thierry, > > I had been doing pretty much the same to reproduce the hang.. > After changing the checkpoint interval.. the hangs had become manageable.. > > However, this time when the hang occurred.. I did not add any clients and > also the behavior was bit different. This time my ns-slapd did not hang.. > ldapsearch was working fine.. but, the ipa commands were taking a long > time to respond and most users were not able to login to any of the > clients.. > restarting ipa did not help either. > > I was seeing around 14-15 connections in SYN_RECV state continously and > /var.log/messages has entries like "TCP: request_sock_TCP: Possible SYN > flooding on port 88. Sending cookies. Check SNMP counter" > > I also see in the /var/log/krb5kdc.log this error > > ipa-master-xyz.com krb5kdc[15308](info): TGS_REQ (4 etypes {18 17 16 23}) > 10.1.12.155: PROCESS_TGS: authtime 0, host/10.1.12.155 at XYZ.COM for ldap/ > ipa-master-xyz.com at XYZ.COM, Clock skew too great > > there are arounf 7.5k messages like that.. which I guess is too high > grep "krbtgt/XYZ.COM at XYZ.COM, Clock skew too great" /var/log/krb5kdc.log > | wc -l > 7570 > > Would this give rise to a hang or make IPA nonresponsive.. > > The kinit admin command itself was taking an unusually long time to > respond.. > > KRB5_TRACE=/dev/stderr kinit admin > [11298] 1473702491.60880: Getting initial credentials for admin at XYZ.COM > [11298] 1473702491.62981: Sending request (167 bytes) to XYZ.COM > [11298] 1473702491.63119: Initiating TCP connection to stream 10.1.3.35:88 > [11298] 1473702491.63359: Sending TCP request to stream 10.1.3.35:88 > [11298] 1473702493.797835: Received answer (341 bytes) from stream > 10.1.3.35:88 > [11298] 1473702493.797848: Terminating TCP connection to stream > 10.1.3.35:88 > [11298] 1473702493.797911: Response was from master KDC > [11298] 1473702493.797956: Received error from KDC: -1765328359/Additional > pre-authentication required > [11298] 1473702493.797993: Processing preauth types: 136, 19, 2, 133 > [11298] 1473702493.798005: Selected etype info: etype aes256-cts, salt > "V at Cbu147E#1;R0WD", params "" > [11298] 1473702493.798009: Received cookie: MIT > Password for admin at XYZ.COM: > [11298] 1473702498.190064: AS key obtained for encrypted timestamp: > aes256-cts/2C9D > [11298] 1473702498.190109: Encrypted timestamp (for 1473702498.184527): > plain 301AA011180F32303136303931323137343831385AA105020302D0CF, encrypted > 25FC8D37EFB6B7837C8D5C6649DFB9972010D40EE29D1222FBA45CAA9842 > 8E42C7FCC9B7FE881A04BD3390A6A9EDE9D2D93729FDF3E47B6D > [11298] 1473702498.190129: Preauth module encrypted_timestamp (2) (real) > returned: 0/Success > [11298] 1473702498.190133: Produced preauth for next request: 133, 2 > [11298] 1473702498.190148: Sending request (261 bytes) to XYZ.COM > [11298] 1473702498.190246: Initiating TCP connection to stream > 10.1.3.35:88 > [11298] 1473702499.191933: Sending initial UDP request to dgram > 10.1.3.35:88 > [11298] 1473702502.195157: Sending retry UDP request to dgram 10.1.3.35:88 > [11298] 1473702507.200405: Sending retry UDP request to dgram 10.1.3.35:88 > [11298] 1473702513.226371: Sending TCP request to stream 10.1.3.35:88 > [11298] 1473702515.797243: Received answer (730 bytes) from stream > 10.1.3.35:88 > [11298] 1473702515.797271: Terminating TCP connection to stream > 10.1.3.35:88 > [11298] 1473702515.797326: Response was from master KDC > [11298] 1473702515.797353: Processing preauth types: 19 > [11298] 1473702515.797360: Selected etype info: etype aes256-cts, salt > "V at Cbu147E#1;R0WD", params "" > [11298] 1473702515.797394: Produced preauth for next request: (empty) > [11298] 1473702515.797401: AS key determined by preauth: aes256-cts/2C9D > [11298] 1473702515.797445: Decrypted AS reply; session key is: > aes256-cts/702E > [11298] 1473702515.797460: FAST negotiation: available > [11298] 1473702515.797478: Initializing KEYRING:persistent:0:0 with > default princ admin at XYZ.COM > [11298] 1473702515.797534: Storing admin at XYZ.COM -> krbtgt/XYZ.COM at XYZ.COM > in KEYRING:persistent:0:0 > [11298] 1473702515.797572: Storing config in KEYRING:persistent:0:0 for > krbtgt/XYZ.COM at XYZ.COM: fast_avail: yes > [11298] 1473702515.797585: Storing admin at XYZ.COM -> > krb5_ccache_conf_data/fast_avail/krbtgt\/XYZ.COM\@XYZ.COM at X-CACHECONF: in > KEYRING:persistent:0:0 > [11298] 1473702515.797631: Storing config in KEYRING:persistent:0:0 for > krbtgt/XYZ.COM at XYZ.COM: pa_type: 2 > [11298] 1473702515.797647: Storing admin at XYZ.COM -> > krb5_ccache_conf_data/pa_type/krbtgt\/XYZ.COM\@XYZ.COM at X-CACHECONF: in > KEYRING:persistent:0:0 > > > > Could it be an version specific issue..as I am running 4.2.0 verison.. and > yours is 4.3.2 > > If so, I will try to upgrade my IPA version > > Thanks > Rakesh > > > > > > > On Thu, Sep 15, 2016 at 10:08 PM, thierry bordaz > wrote: > >> Hello Rakesh, >> >> Sorry to be back so late. I finally manage to try to reproduce locally >> the hang of DS... and I fail ;-) >> >> I am running 4 VM, with freeipa 4.3.2. >> One is running a server (where I provisioned some users), with 200Mb >> cache. The server receives no direct load (SRCH or update). >> The tree others are running in a loop ipa-client-install and uninstall. >> So far it has run for several hours without hang. >> >> Going through the mail thread I have not found others details. >> Am I missing something to reproduce ? >> >> thanks in advance for your help >> thierry >> >> On 09/07/2016 01:46 PM, Rakesh Rajasekharan wrote: >> >> I changed the nsslapd-db-checkpoint interval first to 80 and increased >> till 160.. however the hang has not gone away yet >> >> >> I migrated my data from a openldap, can that cause issues like this.. If >> so, i could try creating users from scratch directly on IPA rather than >> migrating and test it >> >> But before that , have taken a fresh pstack just incase if it helps to >> nail down the issue >> >> Thread 44 (Thread 0x7f53ed52f700 (LWP 128165)): >> #0 0x00007f54047649b3 in select () from /lib64/libc.so.6 >> #1 0x00007f5406eef0e9 in DS_Sleep () from /usr/lib64/dirsrv/libslapd.so. >> 0 >> #2 0x00007f53fa53d907 in deadlock_threadmain () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #3 0x00007f540509e7bb in _pt_root () from /lib64/libnspr4.so >> #4 0x00007f5404a3fdc5 in start_thread () from /lib64/libpthread.so.0 >> #5 0x00007f540476d28d in clone () from /lib64/libc.so.6 >> Thread 43 (Thread 0x7f53ecd2e700 (LWP 128166)): >> #0 0x00007f54047649b3 in select () from /lib64/libc.so.6 >> #1 0x00007f53ff0cbbad in __os_yield () from /lib64/libdb-5.3.so >> #2 0x00007f53ff0c72b3 in __memp_sync_int () from /lib64/libdb-5.3.so >> #3 0x00007f53ff0d7752 in __txn_checkpoint () from /lib64/libdb-5.3.so >> #4 0x00007f53ff0d7b74 in __txn_checkpoint_pp () from /lib64/libdb-5.3.so >> #5 0x00007f53fa541a87 in checkpoint_threadmain () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #6 0x00007f540509e7bb in _pt_root () from /lib64/libnspr4.so >> #7 0x00007f5404a3fdc5 in start_thread () from /lib64/libpthread.so.0 >> #8 0x00007f540476d28d in clone () from /lib64/libc.so.6 >> Thread 42 (Thread 0x7f53ec52d700 (LWP 128167)): >> #0 0x00007f54047649b3 in select () from /lib64/libc.so.6 >> #1 0x00007f5406eef0e9 in DS_Sleep () from /usr/lib64/dirsrv/libslapd.so. >> 0 >> #2 0x00007f53fa53db7f in trickle_threadmain () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #3 0x00007f540509e7bb in _pt_root () from /lib64/libnspr4.so >> #4 0x00007f5404a3fdc5 in start_thread () from /lib64/libpthread.so.0 >> #5 0x00007f540476d28d in clone () from /lib64/libc.so.6 >> Thread 41 (Thread 0x7f53ebd2c700 (LWP 128168)): >> #0 0x00007f54047649b3 in select () from /lib64/libc.so.6 >> #1 0x00007f5406eef0e9 in DS_Sleep () from /usr/lib64/dirsrv/libslapd.so. >> 0 >> #2 0x00007f53fa538707 in perf_threadmain () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #3 0x00007f540509e7bb in _pt_root () from /lib64/libnspr4.so >> #4 0x00007f5404a3fdc5 in start_thread () from /lib64/libpthread.so.0 >> #5 0x00007f540476d28d in clone () from /lib64/libc.so.6 >> Thread 40 (Thread 0x7f53eb322700 (LWP 128204)): >> #0 0x00007f5404a436d5 in pthread_cond_wait@@GLIBC_2.3.2 () from >> /lib64/libpthread.so.0 >> #1 0x00007f5405099050 in PR_WaitCondVar () from /lib64/libnspr4.so >> #2 0x00007f5406ede198 in slapi_wait_condvar () from >> /usr/lib64/dirsrv/libslapd.so.0 >> #3 0x00007f53fd27062e in cos_cache_wait_on_change () from >> /usr/lib64/dirsrv/plugins/libcos-plugin.so >> #4 0x00007f540509e7bb in _pt_root () from /lib64/libnspr4.so >> #5 0x00007f5404a3fdc5 in start_thread () from /lib64/libpthread.so.0 >> #6 0x00007f540476d28d in clone () from /lib64/libc.so.6 >> Thread 39 (Thread 0x7f53eab21700 (LWP 128206)): >> #0 0x00007f5404a436d5 in pthread_cond_wait@@GLIBC_2.3.2 () from >> /lib64/libpthread.so.0 >> #1 0x00007f5405099050 in PR_WaitCondVar () from /lib64/libnspr4.so >> #2 0x00007f5406ede198 in slapi_wait_condvar () from >> /usr/lib64/dirsrv/libslapd.so.0 >> #3 0x00007f53f8bdbeed in roles_cache_wait_on_change () from >> /usr/lib64/dirsrv/plugins/libroles-plugin.so >> #4 0x00007f540509e7bb in _pt_root () from /lib64/libnspr4.so >> #5 0x00007f5404a3fdc5 in start_thread () from /lib64/libpthread.so.0 >> #6 0x00007f540476d28d in clone () from /lib64/libc.so.6 >> Thread 38 (Thread 0x7f53ea320700 (LWP 128207)): >> #0 0x00007f5404a436d5 in pthread_cond_wait@@GLIBC_2.3.2 () from >> /lib64/libpthread.so.0 >> #1 0x00007f5405099050 in PR_WaitCondVar () from /lib64/libnspr4.so >> #2 0x00007f5406ede198 in slapi_wait_condvar () from >> /usr/lib64/dirsrv/libslapd.so.0 >> #3 0x00007f53f8bdbeed in roles_cache_wait_on_change () from >> /usr/lib64/dirsrv/plugins/libroles-plugin.so >> #4 0x00007f540509e7bb in _pt_root () from /lib64/libnspr4.so >> #5 0x00007f5404a3fdc5 in start_thread () from /lib64/libpthread.so.0 >> #6 0x00007f540476d28d in clone () from /lib64/libc.so.6 >> Thread 37 (Thread 0x7f53e9b1f700 (LWP 128208)): >> #0 0x00007f5404a436d5 in pthread_cond_wait@@GLIBC_2.3.2 () from >> /lib64/libpthread.so.0 >> #1 0x00007f5405099050 in PR_WaitCondVar () from /lib64/libnspr4.so >> #2 0x00007f5406ede198 in slapi_wait_condvar () from >> /usr/lib64/dirsrv/libslapd.so.0 >> #3 0x00007f53f8bdbeed in roles_cache_wait_on_change () from >> /usr/lib64/dirsrv/plugins/libroles-plugin.so >> #4 0x00007f540509e7bb in _pt_root () from /lib64/libnspr4.so >> #5 0x00007f5404a3fdc5 in start_thread () from /lib64/libpthread.so.0 >> #6 0x00007f540476d28d in clone () from /lib64/libc.so.6 >> Thread 36 (Thread 0x7f53e931e700 (LWP 128209)): >> #0 0x00007f5404a43a82 in pthread_cond_timedwait@@GLIBC_2.3.2 () from >> /lib64/libpthread.so.0 >> #1 0x00007f5405098b07 in pt_TimedWait () from /lib64/libnspr4.so >> #2 0x00007f5405098fce in PR_WaitCondVar () from /lib64/libnspr4.so >> #3 0x00007f5407397d03 in housecleaning () >> #4 0x00007f540509e7bb in _pt_root () from /lib64/libnspr4.so >> #5 0x00007f5404a3fdc5 in start_thread () from /lib64/libpthread.so.0 >> #6 0x00007f540476d28d in clone () from /lib64/libc.so.6 >> Thread 35 (Thread 0x7f53e8b1d700 (LWP 128210)): >> #0 0x00007f5404a43a82 in pthread_cond_timedwait@@GLIBC_2.3.2 () from >> /lib64/libpthread.so.0 >> #1 0x00007f5405098b07 in pt_TimedWait () from /lib64/libnspr4.so >> #2 0x00007f5405098fce in PR_WaitCondVar () from /lib64/libnspr4.so >> #3 0x00007f5406e841e8 in eq_loop () from /usr/lib64/dirsrv/libslapd.so.0 >> #4 0x00007f540509e7bb in _pt_root () from /lib64/libnspr4.so >> #5 0x00007f5404a3fdc5 in start_thread () from /lib64/libpthread.so.0 >> #6 0x00007f540476d28d in clone () from /lib64/libc.so.6 >> Thread 34 (Thread 0x7f53db7fe700 (LWP 128212)): >> #0 0x00007f5404a436d5 in pthread_cond_wait@@GLIBC_2.3.2 () from >> /lib64/libpthread.so.0 >> #1 0x00007f53fef9f2f3 in __db_hybrid_mutex_suspend () from /lib64/ >> libdb-5.3.so >> #2 0x00007f53fef9e640 in __db_tas_mutex_lock () from /lib64/libdb-5.3.so >> #3 0x00007f53ff048d3a in __lock_get_internal () from /lib64/libdb-5.3.so >> #4 0x00007f53ff049820 in __lock_get () from /lib64/libdb-5.3.so >> #5 0x00007f53ff075142 in __db_lget () from /lib64/libdb-5.3.so >> #6 0x00007f53fefbc605 in __bam_search () from /lib64/libdb-5.3.so >> #7 0x00007f53fefa7256 in __bamc_search () from /lib64/libdb-5.3.so >> #8 0x00007f53fefa8d0f in __bamc_get () from /lib64/libdb-5.3.so >> #9 0x00007f53ff061ca6 in __dbc_iget () from /lib64/libdb-5.3.so >> #10 0x00007f53ff070b02 in __dbc_get_pp () from /lib64/libdb-5.3.so >> #11 0x00007f53fa54c680 in idl_new_fetch () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #12 0x00007f53fa55ab26 in index_read_ext_allids () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #13 0x00007f53fa545284 in keys2idl () from /usr/lib64/dirsrv/plugins/libb >> ack-ldbm.so >> #14 0x00007f53fa5459e3 in ava_candidates.isra.0 () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #15 0x00007f53fa545fd2 in filter_candidates_ext () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #16 0x00007f53fa547096 in list_candidates () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #17 0x00007f53fa545f40 in filter_candidates_ext () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #18 0x00007f53fa58180c in subtree_candidates () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #19 0x00007f53fa582e7f in ldbm_back_search () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #20 0x00007f5406eb01d6 in op_shared_search () from >> /usr/lib64/dirsrv/libslapd.so.0 >> #21 0x00007f54073a133e in do_search () >> #22 0x00007f540738f567 in connection_threadmain () >> #23 0x00007f540509e7bb in _pt_root () from /lib64/libnspr4.so >> #24 0x00007f5404a3fdc5 in start_thread () from /lib64/libpthread.so.0 >> #25 0x00007f540476d28d in clone () from /lib64/libc.so.6 >> Thread 33 (Thread 0x7f53daffd700 (LWP 128213)): >> #0 0x00007f5404a436d5 in pthread_cond_wait@@GLIBC_2.3.2 () from >> /lib64/libpthread.so.0 >> #1 0x00007f53fef9f2f3 in __db_hybrid_mutex_suspend () from /lib64/ >> libdb-5.3.so >> #2 0x00007f53fef9e640 in __db_tas_mutex_lock () from /lib64/libdb-5.3.so >> #3 0x00007f53ff048d3a in __lock_get_internal () from /lib64/libdb-5.3.so >> #4 0x00007f53ff049820 in __lock_get () from /lib64/libdb-5.3.so >> #5 0x00007f53ff075142 in __db_lget () from /lib64/libdb-5.3.so >> #6 0x00007f53fefbc605 in __bam_search () from /lib64/libdb-5.3.so >> #7 0x00007f53fefa7256 in __bamc_search () from /lib64/libdb-5.3.so >> #8 0x00007f53fefa8d0f in __bamc_get () from /lib64/libdb-5.3.so >> #9 0x00007f53ff061ca6 in __dbc_iget () from /lib64/libdb-5.3.so >> #10 0x00007f53ff070b02 in __dbc_get_pp () from /lib64/libdb-5.3.so >> #11 0x00007f53fa54c680 in idl_new_fetch () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #12 0x00007f53fa55ab26 in index_read_ext_allids () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #13 0x00007f53fa545284 in keys2idl () from /usr/lib64/dirsrv/plugins/libb >> ack-ldbm.so >> #14 0x00007f53fa5459e3 in ava_candidates.isra.0 () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #15 0x00007f53fa545fd2 in filter_candidates_ext () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #16 0x00007f53fa547096 in list_candidates () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #17 0x00007f53fa545f40 in filter_candidates_ext () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #18 0x00007f53fa58180c in subtree_candidates () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #19 0x00007f53fa582e7f in ldbm_back_search () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #20 0x00007f5406eb01d6 in op_shared_search () from >> /usr/lib64/dirsrv/libslapd.so.0 >> #21 0x00007f54073a133e in do_search () >> #22 0x00007f540738f567 in connection_threadmain () >> #23 0x00007f540509e7bb in _pt_root () from /lib64/libnspr4.so >> #24 0x00007f5404a3fdc5 in start_thread () from /lib64/libpthread.so.0 >> #25 0x00007f540476d28d in clone () from /lib64/libc.so.6 >> Thread 32 (Thread 0x7f53da7fc700 (LWP 128214)): >> #0 0x00007f5404a436d5 in pthread_cond_wait@@GLIBC_2.3.2 () from >> /lib64/libpthread.so.0 >> #1 0x00007f53fef9f2f3 in __db_hybrid_mutex_suspend () from /lib64/ >> libdb-5.3.so >> #2 0x00007f53fef9e640 in __db_tas_mutex_lock () from /lib64/libdb-5.3.so >> #3 0x00007f53ff048d3a in __lock_get_internal () from /lib64/libdb-5.3.so >> #4 0x00007f53ff049820 in __lock_get () from /lib64/libdb-5.3.so >> #5 0x00007f53ff075142 in __db_lget () from /lib64/libdb-5.3.so >> #6 0x00007f53fefbc605 in __bam_search () from /lib64/libdb-5.3.so >> #7 0x00007f53fefa7256 in __bamc_search () from /lib64/libdb-5.3.so >> #8 0x00007f53fefa8d0f in __bamc_get () from /lib64/libdb-5.3.so >> #9 0x00007f53ff061ca6 in __dbc_iget () from /lib64/libdb-5.3.so >> #10 0x00007f53ff070b02 in __dbc_get_pp () from /lib64/libdb-5.3.so >> #11 0x00007f53fa54c680 in idl_new_fetch () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #12 0x00007f53fa55ab26 in index_read_ext_allids () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #13 0x00007f53fa545284 in keys2idl () from /usr/lib64/dirsrv/plugins/libb >> ack-ldbm.so >> #14 0x00007f53fa5459e3 in ava_candidates.isra.0 () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #15 0x00007f53fa545fd2 in filter_candidates_ext () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #16 0x00007f53fa547096 in list_candidates () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #17 0x00007f53fa545f40 in filter_candidates_ext () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #18 0x00007f53fa547096 in list_candidates () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #19 0x00007f53fa545f40 in filter_candidates_ext () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #20 0x00007f53fa547096 in list_candidates () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #21 0x00007f53fa545f40 in filter_candidates_ext () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #22 0x00007f53fa58180c in subtree_candidates () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #23 0x00007f53fa582e7f in ldbm_back_search () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #24 0x00007f5406eb01d6 in op_shared_search () from >> /usr/lib64/dirsrv/libslapd.so.0 >> #25 0x00007f54073a133e in do_search () >> #26 0x00007f540738f3c3 in connection_threadmain () >> #27 0x00007f540509e7bb in _pt_root () from /lib64/libnspr4.so >> #28 0x00007f5404a3fdc5 in start_thread () from /lib64/libpthread.so.0 >> #29 0x00007f540476d28d in clone () from /lib64/libc.so.6 >> Thread 31 (Thread 0x7f53d9ffb700 (LWP 128215)): >> #0 0x00007f5404a436d5 in pthread_cond_wait@@GLIBC_2.3.2 () from >> /lib64/libpthread.so.0 >> #1 0x00007f53fef9f2f3 in __db_hybrid_mutex_suspend () from /lib64/ >> libdb-5.3.so >> #2 0x00007f53fef9e640 in __db_tas_mutex_lock () from /lib64/libdb-5.3.so >> #3 0x00007f53ff048d3a in __lock_get_internal () from /lib64/libdb-5.3.so >> #4 0x00007f53ff049820 in __lock_get () from /lib64/libdb-5.3.so >> #5 0x00007f53ff075142 in __db_lget () from /lib64/libdb-5.3.so >> #6 0x00007f53fefbc605 in __bam_search () from /lib64/libdb-5.3.so >> #7 0x00007f53fefa7256 in __bamc_search () from /lib64/libdb-5.3.so >> #8 0x00007f53fefa8d0f in __bamc_get () from /lib64/libdb-5.3.so >> #9 0x00007f53ff061ca6 in __dbc_iget () from /lib64/libdb-5.3.so >> #10 0x00007f53ff070b02 in __dbc_get_pp () from /lib64/libdb-5.3.so >> #11 0x00007f53fa54c680 in idl_new_fetch () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #12 0x00007f53fa55ab26 in index_read_ext_allids () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #13 0x00007f53fa545284 in keys2idl () from /usr/lib64/dirsrv/plugins/libb >> ack-ldbm.so >> #14 0x00007f53fa5459e3 in ava_candidates.isra.0 () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #15 0x00007f53fa545fd2 in filter_candidates_ext () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #16 0x00007f53fa547096 in list_candidates () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #17 0x00007f53fa545f40 in filter_candidates_ext () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #18 0x00007f53fa58180c in subtree_candidates () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #19 0x00007f53fa582e7f in ldbm_back_search () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #20 0x00007f5406eb01d6 in op_shared_search () from >> /usr/lib64/dirsrv/libslapd.so.0 >> #21 0x00007f54073a133e in do_search () >> #22 0x00007f540738f567 in connection_threadmain () >> #23 0x00007f540509e7bb in _pt_root () from /lib64/libnspr4.so >> #24 0x00007f5404a3fdc5 in start_thread () from /lib64/libpthread.so.0 >> #25 0x00007f540476d28d in clone () from /lib64/libc.so.6 >> Thread 30 (Thread 0x7f53d97fa700 (LWP 128216)): >> #0 0x00007f5404a436d5 in pthread_cond_wait@@GLIBC_2.3.2 () from >> /lib64/libpthread.so.0 >> #1 0x00007f53fef9f2f3 in __db_hybrid_mutex_suspend () from /lib64/ >> libdb-5.3.so >> #2 0x00007f53fef9e640 in __db_tas_mutex_lock () from /lib64/libdb-5.3.so >> #3 0x00007f53ff048d3a in __lock_get_internal () from /lib64/libdb-5.3.so >> #4 0x00007f53ff049820 in __lock_get () from /lib64/libdb-5.3.so >> #5 0x00007f53ff075142 in __db_lget () from /lib64/libdb-5.3.so >> #6 0x00007f53fefbc605 in __bam_search () from /lib64/libdb-5.3.so >> #7 0x00007f53fefa7256 in __bamc_search () from /lib64/libdb-5.3.so >> #8 0x00007f53fefa8d0f in __bamc_get () from /lib64/libdb-5.3.so >> #9 0x00007f53ff061ca6 in __dbc_iget () from /lib64/libdb-5.3.so >> #10 0x00007f53ff070b02 in __dbc_get_pp () from /lib64/libdb-5.3.so >> #11 0x00007f53fa54c680 in idl_new_fetch () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #12 0x00007f53fa55ab26 in index_read_ext_allids () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #13 0x00007f53fa545284 in keys2idl () from /usr/lib64/dirsrv/plugins/libb >> ack-ldbm.so >> #14 0x00007f53fa5459e3 in ava_candidates.isra.0 () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #15 0x00007f53fa545fd2 in filter_candidates_ext () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #16 0x00007f53fa547096 in list_candidates () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #17 0x00007f53fa545f40 in filter_candidates_ext () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #18 0x00007f53fa58180c in subtree_candidates () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #19 0x00007f53fa582e7f in ldbm_back_search () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #20 0x00007f5406eb01d6 in op_shared_search () from >> /usr/lib64/dirsrv/libslapd.so.0 >> #21 0x00007f54073a133e in do_search () >> #22 0x00007f540738f567 in connection_threadmain () >> #23 0x00007f540509e7bb in _pt_root () from /lib64/libnspr4.so >> #24 0x00007f5404a3fdc5 in start_thread () from /lib64/libpthread.so.0 >> #25 0x00007f540476d28d in clone () from /lib64/libc.so.6 >> Thread 29 (Thread 0x7f53d8ff9700 (LWP 128217)): >> #0 0x00007f5404a436d5 in pthread_cond_wait@@GLIBC_2.3.2 () from >> /lib64/libpthread.so.0 >> #1 0x00007f53fef9f2f3 in __db_hybrid_mutex_suspend () from /lib64/ >> libdb-5.3.so >> #2 0x00007f53fef9e640 in __db_tas_mutex_lock () from /lib64/libdb-5.3.so >> #3 0x00007f53ff048d3a in __lock_get_internal () from /lib64/libdb-5.3.so >> #4 0x00007f53ff049820 in __lock_get () from /lib64/libdb-5.3.so >> #5 0x00007f53ff075142 in __db_lget () from /lib64/libdb-5.3.so >> #6 0x00007f53fefbc605 in __bam_search () from /lib64/libdb-5.3.so >> #7 0x00007f53fefa7256 in __bamc_search () from /lib64/libdb-5.3.so >> #8 0x00007f53fefa8d0f in __bamc_get () from /lib64/libdb-5.3.so >> #9 0x00007f53ff061ca6 in __dbc_iget () from /lib64/libdb-5.3.so >> #10 0x00007f53ff070b02 in __dbc_get_pp () from /lib64/libdb-5.3.so >> #11 0x00007f53fa54c680 in idl_new_fetch () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #12 0x00007f53fa55ab26 in index_read_ext_allids () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #13 0x00007f53fa545284 in keys2idl () from /usr/lib64/dirsrv/plugins/libb >> ack-ldbm.so >> #14 0x00007f53fa5459e3 in ava_candidates.isra.0 () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #15 0x00007f53fa545fd2 in filter_candidates_ext () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #16 0x00007f53fa547096 in list_candidates () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #17 0x00007f53fa545f40 in filter_candidates_ext () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #18 0x00007f53fa58180c in subtree_candidates () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #19 0x00007f53fa582e7f in ldbm_back_search () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #20 0x00007f5406eb01d6 in op_shared_search () from >> /usr/lib64/dirsrv/libslapd.so.0 >> #21 0x00007f54073a133e in do_search () >> #22 0x00007f540738f567 in connection_threadmain () >> #23 0x00007f540509e7bb in _pt_root () from /lib64/libnspr4.so >> #24 0x00007f5404a3fdc5 in start_thread () from /lib64/libpthread.so.0 >> #25 0x00007f540476d28d in clone () from /lib64/libc.so.6 >> Thread 28 (Thread 0x7f53d87f8700 (LWP 128218)): >> #0 0x00007f5404a436d5 in pthread_cond_wait@@GLIBC_2.3.2 () from >> /lib64/libpthread.so.0 >> #1 0x00007f53fef9f2f3 in __db_hybrid_mutex_suspend () from /lib64/ >> libdb-5.3.so >> #2 0x00007f53fef9e640 in __db_tas_mutex_lock () from /lib64/libdb-5.3.so >> #3 0x00007f53ff048d3a in __lock_get_internal () from /lib64/libdb-5.3.so >> #4 0x00007f53ff049820 in __lock_get () from /lib64/libdb-5.3.so >> #5 0x00007f53ff075142 in __db_lget () from /lib64/libdb-5.3.so >> #6 0x00007f53fefbc605 in __bam_search () from /lib64/libdb-5.3.so >> #7 0x00007f53fefa7256 in __bamc_search () from /lib64/libdb-5.3.so >> #8 0x00007f53fefa8d0f in __bamc_get () from /lib64/libdb-5.3.so >> #9 0x00007f53ff061ca6 in __dbc_iget () from /lib64/libdb-5.3.so >> #10 0x00007f53ff070b02 in __dbc_get_pp () from /lib64/libdb-5.3.so >> #11 0x00007f53fa54c680 in idl_new_fetch () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #12 0x00007f53fa55ab26 in index_read_ext_allids () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #13 0x00007f53fa545284 in keys2idl () from /usr/lib64/dirsrv/plugins/libb >> ack-ldbm.so >> #14 0x00007f53fa5459e3 in ava_candidates.isra.0 () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #15 0x00007f53fa545fd2 in filter_candidates_ext () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #16 0x00007f53fa547096 in list_candidates () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #17 0x00007f53fa545f40 in filter_candidates_ext () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #18 0x00007f53fa58180c in subtree_candidates () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #19 0x00007f53fa582e7f in ldbm_back_search () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #20 0x00007f5406eb01d6 in op_shared_search () from >> /usr/lib64/dirsrv/libslapd.so.0 >> #21 0x00007f54073a133e in do_search () >> #22 0x00007f540738f567 in connection_threadmain () >> #23 0x00007f540509e7bb in _pt_root () from /lib64/libnspr4.so >> #24 0x00007f5404a3fdc5 in start_thread () from /lib64/libpthread.so.0 >> #25 0x00007f540476d28d in clone () from /lib64/libc.so.6 >> Thread 27 (Thread 0x7f53d7ff7700 (LWP 128219)): >> #0 0x00007f5404a436d5 in pthread_cond_wait@@GLIBC_2.3.2 () from >> /lib64/libpthread.so.0 >> #1 0x00007f53fef9f2f3 in __db_hybrid_mutex_suspend () from /lib64/ >> libdb-5.3.so >> #2 0x00007f53fef9e640 in __db_tas_mutex_lock () from /lib64/libdb-5.3.so >> #3 0x00007f53ff048d3a in __lock_get_internal () from /lib64/libdb-5.3.so >> #4 0x00007f53ff049820 in __lock_get () from /lib64/libdb-5.3.so >> #5 0x00007f53ff075142 in __db_lget () from /lib64/libdb-5.3.so >> #6 0x00007f53fefbc605 in __bam_search () from /lib64/libdb-5.3.so >> #7 0x00007f53fefa7256 in __bamc_search () from /lib64/libdb-5.3.so >> #8 0x00007f53fefa8d0f in __bamc_get () from /lib64/libdb-5.3.so >> #9 0x00007f53ff061ca6 in __dbc_iget () from /lib64/libdb-5.3.so >> #10 0x00007f53ff070b02 in __dbc_get_pp () from /lib64/libdb-5.3.so >> #11 0x00007f53fa54c680 in idl_new_fetch () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #12 0x00007f53fa55ab26 in index_read_ext_allids () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #13 0x00007f53fa545284 in keys2idl () from /usr/lib64/dirsrv/plugins/libb >> ack-ldbm.so >> #14 0x00007f53fa5459e3 in ava_candidates.isra.0 () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #15 0x00007f53fa545fd2 in filter_candidates_ext () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #16 0x00007f53fa547096 in list_candidates () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #17 0x00007f53fa545f40 in filter_candidates_ext () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #18 0x00007f53fa58180c in subtree_candidates () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #19 0x00007f53fa582e7f in ldbm_back_search () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #20 0x00007f5406eb01d6 in op_shared_search () from >> /usr/lib64/dirsrv/libslapd.so.0 >> #21 0x00007f54073a133e in do_search () >> #22 0x00007f540738f567 in connection_threadmain () >> #23 0x00007f540509e7bb in _pt_root () from /lib64/libnspr4.so >> #24 0x00007f5404a3fdc5 in start_thread () from /lib64/libpthread.so.0 >> #25 0x00007f540476d28d in clone () from /lib64/libc.so.6 >> Thread 26 (Thread 0x7f53d77f6700 (LWP 128220)): >> #0 0x00007f5404a436d5 in pthread_cond_wait@@GLIBC_2.3.2 () from >> /lib64/libpthread.so.0 >> #1 0x00007f53fef9f2f3 in __db_hybrid_mutex_suspend () from /lib64/ >> libdb-5.3.so >> #2 0x00007f53fef9e640 in __db_tas_mutex_lock () from /lib64/libdb-5.3.so >> #3 0x00007f53ff048d3a in __lock_get_internal () from /lib64/libdb-5.3.so >> #4 0x00007f53ff049820 in __lock_get () from /lib64/libdb-5.3.so >> #5 0x00007f53ff075142 in __db_lget () from /lib64/libdb-5.3.so >> #6 0x00007f53fefbc605 in __bam_search () from /lib64/libdb-5.3.so >> #7 0x00007f53fefa7256 in __bamc_search () from /lib64/libdb-5.3.so >> #8 0x00007f53fefa8d0f in __bamc_get () from /lib64/libdb-5.3.so >> #9 0x00007f53ff061ca6 in __dbc_iget () from /lib64/libdb-5.3.so >> #10 0x00007f53ff070b02 in __dbc_get_pp () from /lib64/libdb-5.3.so >> #11 0x00007f53fa54c680 in idl_new_fetch () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #12 0x00007f53fa55ab26 in index_read_ext_allids () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #13 0x00007f53fa545284 in keys2idl () from /usr/lib64/dirsrv/plugins/libb >> ack-ldbm.so >> #14 0x00007f53fa5459e3 in ava_candidates.isra.0 () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #15 0x00007f53fa545fd2 in filter_candidates_ext () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #16 0x00007f53fa547096 in list_candidates () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #17 0x00007f53fa545f40 in filter_candidates_ext () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #18 0x00007f53fa58180c in subtree_candidates () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #19 0x00007f53fa582e7f in ldbm_back_search () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #20 0x00007f5406eb01d6 in op_shared_search () from >> /usr/lib64/dirsrv/libslapd.so.0 >> #21 0x00007f54073a133e in do_search () >> #22 0x00007f540738f567 in connection_threadmain () >> #23 0x00007f540509e7bb in _pt_root () from /lib64/libnspr4.so >> #24 0x00007f5404a3fdc5 in start_thread () from /lib64/libpthread.so.0 >> #25 0x00007f540476d28d in clone () from /lib64/libc.so.6 >> Thread 25 (Thread 0x7f53d6ff5700 (LWP 128221)): >> #0 0x00007f5404a436d5 in pthread_cond_wait@@GLIBC_2.3.2 () from >> /lib64/libpthread.so.0 >> #1 0x00007f53fef9f2f3 in __db_hybrid_mutex_suspend () from /lib64/ >> libdb-5.3.so >> #2 0x00007f53fef9e640 in __db_tas_mutex_lock () from /lib64/libdb-5.3.so >> #3 0x00007f53ff048d3a in __lock_get_internal () from /lib64/libdb-5.3.so >> #4 0x00007f53ff049820 in __lock_get () from /lib64/libdb-5.3.so >> #5 0x00007f53ff075142 in __db_lget () from /lib64/libdb-5.3.so >> #6 0x00007f53fefbc605 in __bam_search () from /lib64/libdb-5.3.so >> #7 0x00007f53fefa7256 in __bamc_search () from /lib64/libdb-5.3.so >> #8 0x00007f53fefa8d0f in __bamc_get () from /lib64/libdb-5.3.so >> #9 0x00007f53ff061ca6 in __dbc_iget () from /lib64/libdb-5.3.so >> #10 0x00007f53ff070b02 in __dbc_get_pp () from /lib64/libdb-5.3.so >> #11 0x00007f53fa54c680 in idl_new_fetch () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #12 0x00007f53fa55ab26 in index_read_ext_allids () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #13 0x00007f53fa545284 in keys2idl () from /usr/lib64/dirsrv/plugins/libb >> ack-ldbm.so >> #14 0x00007f53fa5459e3 in ava_candidates.isra.0 () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #15 0x00007f53fa545fd2 in filter_candidates_ext () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #16 0x00007f53fa547096 in list_candidates () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #17 0x00007f53fa545f40 in filter_candidates_ext () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #18 0x00007f53fa58180c in subtree_candidates () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #19 0x00007f53fa582e7f in ldbm_back_search () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #20 0x00007f5406eb01d6 in op_shared_search () from >> /usr/lib64/dirsrv/libslapd.so.0 >> #21 0x00007f54073a133e in do_search () >> #22 0x00007f540738f567 in connection_threadmain () >> #23 0x00007f540509e7bb in _pt_root () from /lib64/libnspr4.so >> #24 0x00007f5404a3fdc5 in start_thread () from /lib64/libpthread.so.0 >> #25 0x00007f540476d28d in clone () from /lib64/libc.so.6 >> Thread 24 (Thread 0x7f53d67f4700 (LWP 128222)): >> #0 0x00007f5404a436d5 in pthread_cond_wait@@GLIBC_2.3.2 () from >> /lib64/libpthread.so.0 >> #1 0x00007f53fef9f2f3 in __db_hybrid_mutex_suspend () from /lib64/ >> libdb-5.3.so >> #2 0x00007f53fef9e640 in __db_tas_mutex_lock () from /lib64/libdb-5.3.so >> #3 0x00007f53ff048d3a in __lock_get_internal () from /lib64/libdb-5.3.so >> #4 0x00007f53ff049820 in __lock_get () from /lib64/libdb-5.3.so >> #5 0x00007f53ff075142 in __db_lget () from /lib64/libdb-5.3.so >> #6 0x00007f53fefbc605 in __bam_search () from /lib64/libdb-5.3.so >> #7 0x00007f53fefa7256 in __bamc_search () from /lib64/libdb-5.3.so >> #8 0x00007f53fefa8d0f in __bamc_get () from /lib64/libdb-5.3.so >> #9 0x00007f53ff061ca6 in __dbc_iget () from /lib64/libdb-5.3.so >> #10 0x00007f53ff070b02 in __dbc_get_pp () from /lib64/libdb-5.3.so >> #11 0x00007f53fa54c680 in idl_new_fetch () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #12 0x00007f53fa55ab26 in index_read_ext_allids () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #13 0x00007f53fa545284 in keys2idl () from /usr/lib64/dirsrv/plugins/libb >> ack-ldbm.so >> #14 0x00007f53fa5459e3 in ava_candidates.isra.0 () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #15 0x00007f53fa545fd2 in filter_candidates_ext () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #16 0x00007f53fa547096 in list_candidates () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #17 0x00007f53fa545f40 in filter_candidates_ext () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #18 0x00007f53fa58180c in subtree_candidates () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #19 0x00007f53fa582e7f in ldbm_back_search () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #20 0x00007f5406eb01d6 in op_shared_search () from >> /usr/lib64/dirsrv/libslapd.so.0 >> #21 0x00007f54073a133e in do_search () >> #22 0x00007f540738f567 in connection_threadmain () >> #23 0x00007f540509e7bb in _pt_root () from /lib64/libnspr4.so >> #24 0x00007f5404a3fdc5 in start_thread () from /lib64/libpthread.so.0 >> #25 0x00007f540476d28d in clone () from /lib64/libc.so.6 >> Thread 23 (Thread 0x7f53d5ff3700 (LWP 128223)): >> #0 0x00007f5404a436d5 in pthread_cond_wait@@GLIBC_2.3.2 () from >> /lib64/libpthread.so.0 >> #1 0x00007f53fef9f2f3 in __db_hybrid_mutex_suspend () from /lib64/ >> libdb-5.3.so >> #2 0x00007f53fef9e640 in __db_tas_mutex_lock () from /lib64/libdb-5.3.so >> #3 0x00007f53ff0b8e1b in __memp_fget () from /lib64/libdb-5.3.so >> #4 0x00007f53fefbc211 in __bam_search () from /lib64/libdb-5.3.so >> #5 0x00007f53fefa7256 in __bamc_search () from /lib64/libdb-5.3.so >> #6 0x00007f53fefab6c4 in __bamc_put () from /lib64/libdb-5.3.so >> #7 0x00007f53ff0626d5 in __dbc_iput () from /lib64/libdb-5.3.so >> #8 0x00007f53ff05da0e in __db_put () from /lib64/libdb-5.3.so >> #9 0x00007f53ff072fa4 in __db_put_pp () from /lib64/libdb-5.3.so >> #10 0x00007f53fa54d4cf in idl_new_insert_key () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #11 0x00007f53fa55c5b5 in addordel_values_sv.isra.3 () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #12 0x00007f53fa55d25a in index_addordel_values_ext_sv () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #13 0x00007f53fa55d694 in index_addordel_values_sv () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #14 0x00007f53fa55d75a in index_addordel_entry () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #15 0x00007f53fa5605fa in ldbm_back_add () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #16 0x00007f5406e62cca in op_shared_add () from >> /usr/lib64/dirsrv/libslapd.so.0 >> #17 0x00007f5406e64020 in do_add () from /usr/lib64/dirsrv/libslapd.so.0 >> #18 0x00007f540738f303 in connection_threadmain () >> #19 0x00007f540509e7bb in _pt_root () from /lib64/libnspr4.so >> #20 0x00007f5404a3fdc5 in start_thread () from /lib64/libpthread.so.0 >> #21 0x00007f540476d28d in clone () from /lib64/libc.so.6 >> Thread 22 (Thread 0x7f53d57f2700 (LWP 128224)): >> #0 0x00007f5404a436d5 in pthread_cond_wait@@GLIBC_2.3.2 () from >> /lib64/libpthread.so.0 >> #1 0x00007f53fef9f2f3 in __db_hybrid_mutex_suspend () from /lib64/ >> libdb-5.3.so >> #2 0x00007f53fef9e640 in __db_tas_mutex_lock () from /lib64/libdb-5.3.so >> #3 0x00007f53ff048d3a in __lock_get_internal () from /lib64/libdb-5.3.so >> #4 0x00007f53ff049820 in __lock_get () from /lib64/libdb-5.3.so >> #5 0x00007f53ff075142 in __db_lget () from /lib64/libdb-5.3.so >> #6 0x00007f53fefbc605 in __bam_search () from /lib64/libdb-5.3.so >> #7 0x00007f53fefa7256 in __bamc_search () from /lib64/libdb-5.3.so >> #8 0x00007f53fefa8d0f in __bamc_get () from /lib64/libdb-5.3.so >> #9 0x00007f53ff061ca6 in __dbc_iget () from /lib64/libdb-5.3.so >> #10 0x00007f53ff070b02 in __dbc_get_pp () from /lib64/libdb-5.3.so >> #11 0x00007f53fa54c680 in idl_new_fetch () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #12 0x00007f53fa55ab26 in index_read_ext_allids () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #13 0x00007f53fa545284 in keys2idl () from /usr/lib64/dirsrv/plugins/libb >> ack-ldbm.so >> #14 0x00007f53fa5459e3 in ava_candidates.isra.0 () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #15 0x00007f53fa545fd2 in filter_candidates_ext () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #16 0x00007f53fa547096 in list_candidates () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #17 0x00007f53fa545f40 in filter_candidates_ext () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #18 0x00007f53fa58180c in subtree_candidates () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #19 0x00007f53fa582e7f in ldbm_back_search () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #20 0x00007f5406eb01d6 in op_shared_search () from >> /usr/lib64/dirsrv/libslapd.so.0 >> #21 0x00007f54073a133e in do_search () >> #22 0x00007f540738f567 in connection_threadmain () >> #23 0x00007f540509e7bb in _pt_root () from /lib64/libnspr4.so >> #24 0x00007f5404a3fdc5 in start_thread () from /lib64/libpthread.so.0 >> #25 0x00007f540476d28d in clone () from /lib64/libc.so.6 >> Thread 21 (Thread 0x7f53d4ff1700 (LWP 128225)): >> #0 0x00007f5404a436d5 in pthread_cond_wait@@GLIBC_2.3.2 () from >> /lib64/libpthread.so.0 >> #1 0x00007f53fef9f2f3 in __db_hybrid_mutex_suspend () from /lib64/ >> libdb-5.3.so >> #2 0x00007f53fef9e640 in __db_tas_mutex_lock () from /lib64/libdb-5.3.so >> #3 0x00007f53ff048d3a in __lock_get_internal () from /lib64/libdb-5.3.so >> #4 0x00007f53ff049820 in __lock_get () from /lib64/libdb-5.3.so >> #5 0x00007f53ff075142 in __db_lget () from /lib64/libdb-5.3.so >> #6 0x00007f53fefbc605 in __bam_search () from /lib64/libdb-5.3.so >> #7 0x00007f53fefa7256 in __bamc_search () from /lib64/libdb-5.3.so >> #8 0x00007f53fefa8d0f in __bamc_get () from /lib64/libdb-5.3.so >> #9 0x00007f53ff061ca6 in __dbc_iget () from /lib64/libdb-5.3.so >> #10 0x00007f53ff070b02 in __dbc_get_pp () from /lib64/libdb-5.3.so >> #11 0x00007f53fa54c680 in idl_new_fetch () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #12 0x00007f53fa55ab26 in index_read_ext_allids () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #13 0x00007f53fa545284 in keys2idl () from /usr/lib64/dirsrv/plugins/libb >> ack-ldbm.so >> #14 0x00007f53fa5459e3 in ava_candidates.isra.0 () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #15 0x00007f53fa545fd2 in filter_candidates_ext () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #16 0x00007f53fa547096 in list_candidates () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #17 0x00007f53fa545f40 in filter_candidates_ext () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #18 0x00007f53fa58180c in subtree_candidates () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #19 0x00007f53fa582e7f in ldbm_back_search () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #20 0x00007f5406eb01d6 in op_shared_search () from >> /usr/lib64/dirsrv/libslapd.so.0 >> #21 0x00007f54073a133e in do_search () >> #22 0x00007f540738f567 in connection_threadmain () >> #23 0x00007f540509e7bb in _pt_root () from /lib64/libnspr4.so >> #24 0x00007f5404a3fdc5 in start_thread () from /lib64/libpthread.so.0 >> #25 0x00007f540476d28d in clone () from /lib64/libc.so.6 >> Thread 20 (Thread 0x7f53d47f0700 (LWP 128226)): >> #0 0x00007f5404a436d5 in pthread_cond_wait@@GLIBC_2.3.2 () from >> /lib64/libpthread.so.0 >> #1 0x00007f53fef9f2f3 in __db_hybrid_mutex_suspend () from /lib64/ >> libdb-5.3.so >> #2 0x00007f53fef9e640 in __db_tas_mutex_lock () from /lib64/libdb-5.3.so >> #3 0x00007f53ff048d3a in __lock_get_internal () from /lib64/libdb-5.3.so >> #4 0x00007f53ff049820 in __lock_get () from /lib64/libdb-5.3.so >> #5 0x00007f53ff075142 in __db_lget () from /lib64/libdb-5.3.so >> #6 0x00007f53fefbc605 in __bam_search () from /lib64/libdb-5.3.so >> #7 0x00007f53fefa7256 in __bamc_search () from /lib64/libdb-5.3.so >> #8 0x00007f53fefa8d0f in __bamc_get () from /lib64/libdb-5.3.so >> #9 0x00007f53ff061ca6 in __dbc_iget () from /lib64/libdb-5.3.so >> #10 0x00007f53ff070b02 in __dbc_get_pp () from /lib64/libdb-5.3.so >> #11 0x00007f53fa54c680 in idl_new_fetch () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #12 0x00007f53fa55ab26 in index_read_ext_allids () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #13 0x00007f53fa545284 in keys2idl () from /usr/lib64/dirsrv/plugins/libb >> ack-ldbm.so >> #14 0x00007f53fa5459e3 in ava_candidates.isra.0 () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #15 0x00007f53fa545fd2 in filter_candidates_ext () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #16 0x00007f53fa547096 in list_candidates () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #17 0x00007f53fa545f40 in filter_candidates_ext () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #18 0x00007f53fa58180c in subtree_candidates () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #19 0x00007f53fa582e7f in ldbm_back_search () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #20 0x00007f5406eb01d6 in op_shared_search () from >> /usr/lib64/dirsrv/libslapd.so.0 >> #21 0x00007f5406ec094e in search_internal_callback_pb () from >> /usr/lib64/dirsrv/libslapd.so.0 >> #22 0x00007f5406ec0be8 in search_internal_pb () from >> /usr/lib64/dirsrv/libslapd.so.0 >> #23 0x00007f53fe118e90 in search_one_berval () from >> /usr/lib64/dirsrv/plugins/libattr-unique-plugin.so >> #24 0x00007f53fe119242 in search () from /usr/lib64/dirsrv/plugins/liba >> ttr-unique-plugin.so >> #25 0x00007f53fe11954c in searchAllSubtrees () from >> /usr/lib64/dirsrv/plugins/libattr-unique-plugin.so >> #26 0x00007f53fe11a015 in preop_add () from /usr/lib64/dirsrv/plugins/liba >> ttr-unique-plugin.so >> #27 0x00007f5406ebb848 in plugin_call_func () from >> /usr/lib64/dirsrv/libslapd.so.0 >> #28 0x00007f5406ebbb13 in plugin_call_plugins () from >> /usr/lib64/dirsrv/libslapd.so.0 >> #29 0x00007f5406e62c2b in op_shared_add () from >> /usr/lib64/dirsrv/libslapd.so.0 >> #30 0x00007f5406e64020 in do_add () from /usr/lib64/dirsrv/libslapd.so.0 >> #31 0x00007f540738f303 in connection_threadmain () >> #32 0x00007f540509e7bb in _pt_root () from /lib64/libnspr4.so >> #33 0x00007f5404a3fdc5 in start_thread () from /lib64/libpthread.so.0 >> #34 0x00007f540476d28d in clone () from /lib64/libc.so.6 >> Thread 19 (Thread 0x7f53d3fef700 (LWP 128227)): >> #0 0x00007f5404a436d5 in pthread_cond_wait@@GLIBC_2.3.2 () from >> /lib64/libpthread.so.0 >> #1 0x00007f53fef9f2f3 in __db_hybrid_mutex_suspend () from /lib64/ >> libdb-5.3.so >> #2 0x00007f53fef9e640 in __db_tas_mutex_lock () from /lib64/libdb-5.3.so >> #3 0x00007f53ff048d3a in __lock_get_internal () from /lib64/libdb-5.3.so >> #4 0x00007f53ff049820 in __lock_get () from /lib64/libdb-5.3.so >> #5 0x00007f53ff075142 in __db_lget () from /lib64/libdb-5.3.so >> #6 0x00007f53fefbc605 in __bam_search () from /lib64/libdb-5.3.so >> #7 0x00007f53fefa7256 in __bamc_search () from /lib64/libdb-5.3.so >> #8 0x00007f53fefa8d0f in __bamc_get () from /lib64/libdb-5.3.so >> #9 0x00007f53ff061ca6 in __dbc_iget () from /lib64/libdb-5.3.so >> #10 0x00007f53ff070b02 in __dbc_get_pp () from /lib64/libdb-5.3.so >> #11 0x00007f53fa54c680 in idl_new_fetch () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #12 0x00007f53fa55ab26 in index_read_ext_allids () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #13 0x00007f53fa545284 in keys2idl () from /usr/lib64/dirsrv/plugins/libb >> ack-ldbm.so >> #14 0x00007f53fa5459e3 in ava_candidates.isra.0 () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #15 0x00007f53fa545fd2 in filter_candidates_ext () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #16 0x00007f53fa547096 in list_candidates () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #17 0x00007f53fa545f40 in filter_candidates_ext () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #18 0x00007f53fa58180c in subtree_candidates () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #19 0x00007f53fa582e7f in ldbm_back_search () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #20 0x00007f5406eb01d6 in op_shared_search () from >> /usr/lib64/dirsrv/libslapd.so.0 >> #21 0x00007f54073a133e in do_search () >> #22 0x00007f540738f567 in connection_threadmain () >> #23 0x00007f540509e7bb in _pt_root () from /lib64/libnspr4.so >> #24 0x00007f5404a3fdc5 in start_thread () from /lib64/libpthread.so.0 >> #25 0x00007f540476d28d in clone () from /lib64/libc.so.6 >> Thread 18 (Thread 0x7f53d37ee700 (LWP 128228)): >> #0 0x00007f5404a436d5 in pthread_cond_wait@@GLIBC_2.3.2 () from >> /lib64/libpthread.so.0 >> #1 0x00007f53fef9f2f3 in __db_hybrid_mutex_suspend () from /lib64/ >> libdb-5.3.so >> #2 0x00007f53fef9e640 in __db_tas_mutex_lock () from /lib64/libdb-5.3.so >> #3 0x00007f53ff048d3a in __lock_get_internal () from /lib64/libdb-5.3.so >> #4 0x00007f53ff049820 in __lock_get () from /lib64/libdb-5.3.so >> #5 0x00007f53ff075142 in __db_lget () from /lib64/libdb-5.3.so >> #6 0x00007f53fefbc605 in __bam_search () from /lib64/libdb-5.3.so >> #7 0x00007f53fefa7256 in __bamc_search () from /lib64/libdb-5.3.so >> #8 0x00007f53fefa8d0f in __bamc_get () from /lib64/libdb-5.3.so >> #9 0x00007f53ff061ca6 in __dbc_iget () from /lib64/libdb-5.3.so >> #10 0x00007f53ff070b02 in __dbc_get_pp () from /lib64/libdb-5.3.so >> #11 0x00007f53fa54c680 in idl_new_fetch () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #12 0x00007f53fa55ab26 in index_read_ext_allids () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #13 0x00007f53fa545284 in keys2idl () from /usr/lib64/dirsrv/plugins/libb >> ack-ldbm.so >> #14 0x00007f53fa5459e3 in ava_candidates.isra.0 () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #15 0x00007f53fa545fd2 in filter_candidates_ext () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #16 0x00007f53fa547096 in list_candidates () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #17 0x00007f53fa545f40 in filter_candidates_ext () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #18 0x00007f53fa58180c in subtree_candidates () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #19 0x00007f53fa582e7f in ldbm_back_search () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #20 0x00007f5406eb01d6 in op_shared_search () from >> /usr/lib64/dirsrv/libslapd.so.0 >> #21 0x00007f54073a133e in do_search () >> #22 0x00007f540738f567 in connection_threadmain () >> #23 0x00007f540509e7bb in _pt_root () from /lib64/libnspr4.so >> #24 0x00007f5404a3fdc5 in start_thread () from /lib64/libpthread.so.0 >> #25 0x00007f540476d28d in clone () from /lib64/libc.so.6 >> Thread 17 (Thread 0x7f53d2fed700 (LWP 128229)): >> #0 0x00007f5404a436d5 in pthread_cond_wait@@GLIBC_2.3.2 () from >> /lib64/libpthread.so.0 >> #1 0x00007f53fef9f2f3 in __db_hybrid_mutex_suspend () from /lib64/ >> libdb-5.3.so >> #2 0x00007f53fef9e640 in __db_tas_mutex_lock () from /lib64/libdb-5.3.so >> #3 0x00007f53ff048d3a in __lock_get_internal () from /lib64/libdb-5.3.so >> #4 0x00007f53ff049820 in __lock_get () from /lib64/libdb-5.3.so >> #5 0x00007f53ff075142 in __db_lget () from /lib64/libdb-5.3.so >> #6 0x00007f53fefbc605 in __bam_search () from /lib64/libdb-5.3.so >> #7 0x00007f53fefa7256 in __bamc_search () from /lib64/libdb-5.3.so >> #8 0x00007f53fefa8d0f in __bamc_get () from /lib64/libdb-5.3.so >> #9 0x00007f53ff061ca6 in __dbc_iget () from /lib64/libdb-5.3.so >> #10 0x00007f53ff070b02 in __dbc_get_pp () from /lib64/libdb-5.3.so >> #11 0x00007f53fa54c680 in idl_new_fetch () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #12 0x00007f53fa55ab26 in index_read_ext_allids () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #13 0x00007f53fa545284 in keys2idl () from /usr/lib64/dirsrv/plugins/libb >> ack-ldbm.so >> #14 0x00007f53fa5459e3 in ava_candidates.isra.0 () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #15 0x00007f53fa545fd2 in filter_candidates_ext () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #16 0x00007f53fa547096 in list_candidates () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #17 0x00007f53fa545f40 in filter_candidates_ext () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #18 0x00007f53fa58180c in subtree_candidates () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #19 0x00007f53fa582e7f in ldbm_back_search () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #20 0x00007f5406eb01d6 in op_shared_search () from >> /usr/lib64/dirsrv/libslapd.so.0 >> #21 0x00007f54073a133e in do_search () >> #22 0x00007f540738f567 in connection_threadmain () >> #23 0x00007f540509e7bb in _pt_root () from /lib64/libnspr4.so >> #24 0x00007f5404a3fdc5 in start_thread () from /lib64/libpthread.so.0 >> #25 0x00007f540476d28d in clone () from /lib64/libc.so.6 >> Thread 16 (Thread 0x7f53d27ec700 (LWP 128230)): >> #0 0x00007f5404a436d5 in pthread_cond_wait@@GLIBC_2.3.2 () from >> /lib64/libpthread.so.0 >> #1 0x00007f53fef9f2f3 in __db_hybrid_mutex_suspend () from /lib64/ >> libdb-5.3.so >> #2 0x00007f53fef9e640 in __db_tas_mutex_lock () from /lib64/libdb-5.3.so >> #3 0x00007f53ff048d3a in __lock_get_internal () from /lib64/libdb-5.3.so >> #4 0x00007f53ff049820 in __lock_get () from /lib64/libdb-5.3.so >> #5 0x00007f53ff075142 in __db_lget () from /lib64/libdb-5.3.so >> #6 0x00007f53fefbc605 in __bam_search () from /lib64/libdb-5.3.so >> #7 0x00007f53fefa7256 in __bamc_search () from /lib64/libdb-5.3.so >> #8 0x00007f53fefa8d0f in __bamc_get () from /lib64/libdb-5.3.so >> #9 0x00007f53ff061ca6 in __dbc_iget () from /lib64/libdb-5.3.so >> #10 0x00007f53ff070b02 in __dbc_get_pp () from /lib64/libdb-5.3.so >> #11 0x00007f53fa54c680 in idl_new_fetch () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #12 0x00007f53fa55ab26 in index_read_ext_allids () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #13 0x00007f53fa545284 in keys2idl () from /usr/lib64/dirsrv/plugins/libb >> ack-ldbm.so >> #14 0x00007f53fa5459e3 in ava_candidates.isra.0 () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #15 0x00007f53fa545fd2 in filter_candidates_ext () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #16 0x00007f53fa547096 in list_candidates () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #17 0x00007f53fa545f40 in filter_candidates_ext () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #18 0x00007f53fa58180c in subtree_candidates () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #19 0x00007f53fa582e7f in ldbm_back_search () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #20 0x00007f5406eb01d6 in op_shared_search () from >> /usr/lib64/dirsrv/libslapd.so.0 >> #21 0x00007f54073a133e in do_search () >> #22 0x00007f540738f567 in connection_threadmain () >> #23 0x00007f540509e7bb in _pt_root () from /lib64/libnspr4.so >> #24 0x00007f5404a3fdc5 in start_thread () from /lib64/libpthread.so.0 >> #25 0x00007f540476d28d in clone () from /lib64/libc.so.6 >> Thread 15 (Thread 0x7f53d1feb700 (LWP 128231)): >> #0 0x00007f5404a436d5 in pthread_cond_wait@@GLIBC_2.3.2 () from >> /lib64/libpthread.so.0 >> #1 0x00007f53fef9f2f3 in __db_hybrid_mutex_suspend () from /lib64/ >> libdb-5.3.so >> #2 0x00007f53fef9e640 in __db_tas_mutex_lock () from /lib64/libdb-5.3.so >> #3 0x00007f53ff048d3a in __lock_get_internal () from /lib64/libdb-5.3.so >> #4 0x00007f53ff049820 in __lock_get () from /lib64/libdb-5.3.so >> #5 0x00007f53ff075142 in __db_lget () from /lib64/libdb-5.3.so >> #6 0x00007f53fefbc605 in __bam_search () from /lib64/libdb-5.3.so >> #7 0x00007f53fefa7256 in __bamc_search () from /lib64/libdb-5.3.so >> #8 0x00007f53fefa8d0f in __bamc_get () from /lib64/libdb-5.3.so >> #9 0x00007f53ff061ca6 in __dbc_iget () from /lib64/libdb-5.3.so >> #10 0x00007f53ff070b02 in __dbc_get_pp () from /lib64/libdb-5.3.so >> #11 0x00007f53fa54c680 in idl_new_fetch () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #12 0x00007f53fa55ab26 in index_read_ext_allids () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #13 0x00007f53fa545284 in keys2idl () from /usr/lib64/dirsrv/plugins/libb >> ack-ldbm.so >> #14 0x00007f53fa5459e3 in ava_candidates.isra.0 () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #15 0x00007f53fa545fd2 in filter_candidates_ext () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #16 0x00007f53fa547096 in list_candidates () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #17 0x00007f53fa545f40 in filter_candidates_ext () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #18 0x00007f53fa58180c in subtree_candidates () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #19 0x00007f53fa582e7f in ldbm_back_search () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #20 0x00007f5406eb01d6 in op_shared_search () from >> /usr/lib64/dirsrv/libslapd.so.0 >> #21 0x00007f54073a133e in do_search () >> #22 0x00007f540738f567 in connection_threadmain () >> #23 0x00007f540509e7bb in _pt_root () from /lib64/libnspr4.so >> #24 0x00007f5404a3fdc5 in start_thread () from /lib64/libpthread.so.0 >> #25 0x00007f540476d28d in clone () from /lib64/libc.so.6 >> Thread 14 (Thread 0x7f53d17ea700 (LWP 128232)): >> #0 0x00007f5404a436d5 in pthread_cond_wait@@GLIBC_2.3.2 () from >> /lib64/libpthread.so.0 >> #1 0x00007f53fef9f2f3 in __db_hybrid_mutex_suspend () from /lib64/ >> libdb-5.3.so >> #2 0x00007f53fef9e640 in __db_tas_mutex_lock () from /lib64/libdb-5.3.so >> #3 0x00007f53ff048d3a in __lock_get_internal () from /lib64/libdb-5.3.so >> #4 0x00007f53ff049820 in __lock_get () from /lib64/libdb-5.3.so >> #5 0x00007f53ff075142 in __db_lget () from /lib64/libdb-5.3.so >> #6 0x00007f53fefbc605 in __bam_search () from /lib64/libdb-5.3.so >> #7 0x00007f53fefa7256 in __bamc_search () from /lib64/libdb-5.3.so >> #8 0x00007f53fefa8d0f in __bamc_get () from /lib64/libdb-5.3.so >> #9 0x00007f53ff061ca6 in __dbc_iget () from /lib64/libdb-5.3.so >> #10 0x00007f53ff070b02 in __dbc_get_pp () from /lib64/libdb-5.3.so >> #11 0x00007f53fa54c680 in idl_new_fetch () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #12 0x00007f53fa55ab26 in index_read_ext_allids () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #13 0x00007f53fa545284 in keys2idl () from /usr/lib64/dirsrv/plugins/libb >> ack-ldbm.so >> #14 0x00007f53fa5459e3 in ava_candidates.isra.0 () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #15 0x00007f53fa545fd2 in filter_candidates_ext () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #16 0x00007f53fa547096 in list_candidates () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #17 0x00007f53fa545f40 in filter_candidates_ext () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #18 0x00007f53fa547096 in list_candidates () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #19 0x00007f53fa545f40 in filter_candidates_ext () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #20 0x00007f53fa547096 in list_candidates () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #21 0x00007f53fa545f40 in filter_candidates_ext () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #22 0x00007f53fa58180c in subtree_candidates () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #23 0x00007f53fa582e7f in ldbm_back_search () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #24 0x00007f5406eb01d6 in op_shared_search () from >> /usr/lib64/dirsrv/libslapd.so.0 >> #25 0x00007f54073a133e in do_search () >> #26 0x00007f540738f3c3 in connection_threadmain () >> #27 0x00007f540509e7bb in _pt_root () from /lib64/libnspr4.so >> #28 0x00007f5404a3fdc5 in start_thread () from /lib64/libpthread.so.0 >> #29 0x00007f540476d28d in clone () from /lib64/libc.so.6 >> Thread 13 (Thread 0x7f53d0fe9700 (LWP 128233)): >> #0 0x00007f5404a436d5 in pthread_cond_wait@@GLIBC_2.3.2 () from >> /lib64/libpthread.so.0 >> #1 0x00007f53fef9f2f3 in __db_hybrid_mutex_suspend () from /lib64/ >> libdb-5.3.so >> #2 0x00007f53fef9e640 in __db_tas_mutex_lock () from /lib64/libdb-5.3.so >> #3 0x00007f53ff048d3a in __lock_get_internal () from /lib64/libdb-5.3.so >> #4 0x00007f53ff049820 in __lock_get () from /lib64/libdb-5.3.so >> #5 0x00007f53ff075142 in __db_lget () from /lib64/libdb-5.3.so >> #6 0x00007f53fefbc605 in __bam_search () from /lib64/libdb-5.3.so >> #7 0x00007f53fefa7256 in __bamc_search () from /lib64/libdb-5.3.so >> #8 0x00007f53fefa8d0f in __bamc_get () from /lib64/libdb-5.3.so >> #9 0x00007f53ff061ca6 in __dbc_iget () from /lib64/libdb-5.3.so >> #10 0x00007f53ff070b02 in __dbc_get_pp () from /lib64/libdb-5.3.so >> #11 0x00007f53fa54c680 in idl_new_fetch () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #12 0x00007f53fa55ab26 in index_read_ext_allids () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #13 0x00007f53fa545284 in keys2idl () from /usr/lib64/dirsrv/plugins/libb >> ack-ldbm.so >> #14 0x00007f53fa5459e3 in ava_candidates.isra.0 () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #15 0x00007f53fa545fd2 in filter_candidates_ext () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #16 0x00007f53fa547096 in list_candidates () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #17 0x00007f53fa545f40 in filter_candidates_ext () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #18 0x00007f53fa58180c in subtree_candidates () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #19 0x00007f53fa582e7f in ldbm_back_search () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #20 0x00007f5406eb01d6 in op_shared_search () from >> /usr/lib64/dirsrv/libslapd.so.0 >> #21 0x00007f54073a133e in do_search () >> #22 0x00007f540738f567 in connection_threadmain () >> #23 0x00007f540509e7bb in _pt_root () from /lib64/libnspr4.so >> #24 0x00007f5404a3fdc5 in start_thread () from /lib64/libpthread.so.0 >> #25 0x00007f540476d28d in clone () from /lib64/libc.so.6 >> Thread 12 (Thread 0x7f53d07e8700 (LWP 128234)): >> #0 0x00007f5404a436d5 in pthread_cond_wait@@GLIBC_2.3.2 () from >> /lib64/libpthread.so.0 >> #1 0x00007f53fef9f2f3 in __db_hybrid_mutex_suspend () from /lib64/ >> libdb-5.3.so >> #2 0x00007f53fef9e640 in __db_tas_mutex_lock () from /lib64/libdb-5.3.so >> #3 0x00007f53ff048d3a in __lock_get_internal () from /lib64/libdb-5.3.so >> #4 0x00007f53ff049820 in __lock_get () from /lib64/libdb-5.3.so >> #5 0x00007f53ff075142 in __db_lget () from /lib64/libdb-5.3.so >> #6 0x00007f53fefbc605 in __bam_search () from /lib64/libdb-5.3.so >> #7 0x00007f53fefa7256 in __bamc_search () from /lib64/libdb-5.3.so >> #8 0x00007f53fefa8d0f in __bamc_get () from /lib64/libdb-5.3.so >> #9 0x00007f53ff061ca6 in __dbc_iget () from /lib64/libdb-5.3.so >> #10 0x00007f53ff070b02 in __dbc_get_pp () from /lib64/libdb-5.3.so >> #11 0x00007f53fa54c680 in idl_new_fetch () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #12 0x00007f53fa55ab26 in index_read_ext_allids () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #13 0x00007f53fa545284 in keys2idl () from /usr/lib64/dirsrv/plugins/libb >> ack-ldbm.so >> #14 0x00007f53fa5459e3 in ava_candidates.isra.0 () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #15 0x00007f53fa545fd2 in filter_candidates_ext () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #16 0x00007f53fa547096 in list_candidates () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #17 0x00007f53fa545f40 in filter_candidates_ext () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #18 0x00007f53fa58180c in subtree_candidates () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #19 0x00007f53fa582e7f in ldbm_back_search () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #20 0x00007f5406eb01d6 in op_shared_search () from >> /usr/lib64/dirsrv/libslapd.so.0 >> #21 0x00007f54073a133e in do_search () >> #22 0x00007f540738f567 in connection_threadmain () >> #23 0x00007f540509e7bb in _pt_root () from /lib64/libnspr4.so >> #24 0x00007f5404a3fdc5 in start_thread () from /lib64/libpthread.so.0 >> #25 0x00007f540476d28d in clone () from /lib64/libc.so.6 >> Thread 11 (Thread 0x7f53cffe7700 (LWP 128235)): >> #0 0x00007f5404a436d5 in pthread_cond_wait@@GLIBC_2.3.2 () from >> /lib64/libpthread.so.0 >> #1 0x00007f53fef9f2f3 in __db_hybrid_mutex_suspend () from /lib64/ >> libdb-5.3.so >> #2 0x00007f53fef9e640 in __db_tas_mutex_lock () from /lib64/libdb-5.3.so >> #3 0x00007f53ff048d3a in __lock_get_internal () from /lib64/libdb-5.3.so >> #4 0x00007f53ff049820 in __lock_get () from /lib64/libdb-5.3.so >> #5 0x00007f53ff075142 in __db_lget () from /lib64/libdb-5.3.so >> #6 0x00007f53fefbc605 in __bam_search () from /lib64/libdb-5.3.so >> #7 0x00007f53fefa7256 in __bamc_search () from /lib64/libdb-5.3.so >> #8 0x00007f53fefa8d0f in __bamc_get () from /lib64/libdb-5.3.so >> #9 0x00007f53ff061ca6 in __dbc_iget () from /lib64/libdb-5.3.so >> #10 0x00007f53ff070b02 in __dbc_get_pp () from /lib64/libdb-5.3.so >> #11 0x00007f53fa54c680 in idl_new_fetch () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #12 0x00007f53fa55ab26 in index_read_ext_allids () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #13 0x00007f53fa545284 in keys2idl () from /usr/lib64/dirsrv/plugins/libb >> ack-ldbm.so >> #14 0x00007f53fa5459e3 in ava_candidates.isra.0 () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #15 0x00007f53fa545fd2 in filter_candidates_ext () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #16 0x00007f53fa547096 in list_candidates () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #17 0x00007f53fa545f40 in filter_candidates_ext () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #18 0x00007f53fa58180c in subtree_candidates () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #19 0x00007f53fa582e7f in ldbm_back_search () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #20 0x00007f5406eb01d6 in op_shared_search () from >> /usr/lib64/dirsrv/libslapd.so.0 >> #21 0x00007f54073a133e in do_search () >> #22 0x00007f540738f567 in connection_threadmain () >> #23 0x00007f540509e7bb in _pt_root () from /lib64/libnspr4.so >> #24 0x00007f5404a3fdc5 in start_thread () from /lib64/libpthread.so.0 >> #25 0x00007f540476d28d in clone () from /lib64/libc.so.6 >> Thread 10 (Thread 0x7f53cf7e6700 (LWP 128236)): >> #0 0x00007f5404a436d5 in pthread_cond_wait@@GLIBC_2.3.2 () from >> /lib64/libpthread.so.0 >> #1 0x00007f53fef9f2f3 in __db_hybrid_mutex_suspend () from /lib64/ >> libdb-5.3.so >> #2 0x00007f53fef9e640 in __db_tas_mutex_lock () from /lib64/libdb-5.3.so >> #3 0x00007f53ff048d3a in __lock_get_internal () from /lib64/libdb-5.3.so >> #4 0x00007f53ff049820 in __lock_get () from /lib64/libdb-5.3.so >> #5 0x00007f53ff075142 in __db_lget () from /lib64/libdb-5.3.so >> #6 0x00007f53fefbc605 in __bam_search () from /lib64/libdb-5.3.so >> #7 0x00007f53fefa7256 in __bamc_search () from /lib64/libdb-5.3.so >> #8 0x00007f53fefa8d0f in __bamc_get () from /lib64/libdb-5.3.so >> #9 0x00007f53ff061ca6 in __dbc_iget () from /lib64/libdb-5.3.so >> #10 0x00007f53ff070b02 in __dbc_get_pp () from /lib64/libdb-5.3.so >> #11 0x00007f53fa54c680 in idl_new_fetch () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #12 0x00007f53fa55ab26 in index_read_ext_allids () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #13 0x00007f53fa545284 in keys2idl () from /usr/lib64/dirsrv/plugins/libb >> ack-ldbm.so >> #14 0x00007f53fa5459e3 in ava_candidates.isra.0 () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #15 0x00007f53fa545fd2 in filter_candidates_ext () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #16 0x00007f53fa547096 in list_candidates () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #17 0x00007f53fa545f40 in filter_candidates_ext () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #18 0x00007f53fa58180c in subtree_candidates () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #19 0x00007f53fa582e7f in ldbm_back_search () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #20 0x00007f5406eb01d6 in op_shared_search () from >> /usr/lib64/dirsrv/libslapd.so.0 >> #21 0x00007f54073a133e in do_search () >> #22 0x00007f540738f567 in connection_threadmain () >> #23 0x00007f540509e7bb in _pt_root () from /lib64/libnspr4.so >> #24 0x00007f5404a3fdc5 in start_thread () from /lib64/libpthread.so.0 >> #25 0x00007f540476d28d in clone () from /lib64/libc.so.6 >> Thread 9 (Thread 0x7f53cefe5700 (LWP 128237)): >> #0 0x00007f5404a436d5 in pthread_cond_wait@@GLIBC_2.3.2 () from >> /lib64/libpthread.so.0 >> #1 0x00007f53fef9f2f3 in __db_hybrid_mutex_suspend () from /lib64/ >> libdb-5.3.so >> #2 0x00007f53fef9e640 in __db_tas_mutex_lock () from /lib64/libdb-5.3.so >> #3 0x00007f53ff048d3a in __lock_get_internal () from /lib64/libdb-5.3.so >> #4 0x00007f53ff049820 in __lock_get () from /lib64/libdb-5.3.so >> #5 0x00007f53ff075142 in __db_lget () from /lib64/libdb-5.3.so >> #6 0x00007f53fefbc605 in __bam_search () from /lib64/libdb-5.3.so >> #7 0x00007f53fefa7256 in __bamc_search () from /lib64/libdb-5.3.so >> #8 0x00007f53fefa8d0f in __bamc_get () from /lib64/libdb-5.3.so >> #9 0x00007f53ff061ca6 in __dbc_iget () from /lib64/libdb-5.3.so >> #10 0x00007f53ff070b02 in __dbc_get_pp () from /lib64/libdb-5.3.so >> #11 0x00007f53fa54c680 in idl_new_fetch () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #12 0x00007f53fa55ab26 in index_read_ext_allids () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #13 0x00007f53fa545284 in keys2idl () from /usr/lib64/dirsrv/plugins/libb >> ack-ldbm.so >> #14 0x00007f53fa5459e3 in ava_candidates.isra.0 () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #15 0x00007f53fa545fd2 in filter_candidates_ext () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #16 0x00007f53fa547096 in list_candidates () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #17 0x00007f53fa545f40 in filter_candidates_ext () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #18 0x00007f53fa58180c in subtree_candidates () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #19 0x00007f53fa582e7f in ldbm_back_search () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #20 0x00007f5406eb01d6 in op_shared_search () from >> /usr/lib64/dirsrv/libslapd.so.0 >> #21 0x00007f54073a133e in do_search () >> #22 0x00007f540738f567 in connection_threadmain () >> #23 0x00007f540509e7bb in _pt_root () from /lib64/libnspr4.so >> #24 0x00007f5404a3fdc5 in start_thread () from /lib64/libpthread.so.0 >> #25 0x00007f540476d28d in clone () from /lib64/libc.so.6 >> Thread 8 (Thread 0x7f53ce7e4700 (LWP 128238)): >> #0 0x00007f5404a436d5 in pthread_cond_wait@@GLIBC_2.3.2 () from >> /lib64/libpthread.so.0 >> #1 0x00007f53fef9f2f3 in __db_hybrid_mutex_suspend () from /lib64/ >> libdb-5.3.so >> #2 0x00007f53fef9e640 in __db_tas_mutex_lock () from /lib64/libdb-5.3.so >> #3 0x00007f53ff048d3a in __lock_get_internal () from /lib64/libdb-5.3.so >> #4 0x00007f53ff049820 in __lock_get () from /lib64/libdb-5.3.so >> #5 0x00007f53ff075142 in __db_lget () from /lib64/libdb-5.3.so >> #6 0x00007f53fefbc605 in __bam_search () from /lib64/libdb-5.3.so >> #7 0x00007f53fefa7256 in __bamc_search () from /lib64/libdb-5.3.so >> #8 0x00007f53fefa8d0f in __bamc_get () from /lib64/libdb-5.3.so >> #9 0x00007f53ff061ca6 in __dbc_iget () from /lib64/libdb-5.3.so >> #10 0x00007f53ff070b02 in __dbc_get_pp () from /lib64/libdb-5.3.so >> #11 0x00007f53fa54c680 in idl_new_fetch () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #12 0x00007f53fa55ab26 in index_read_ext_allids () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #13 0x00007f53fa545284 in keys2idl () from /usr/lib64/dirsrv/plugins/libb >> ack-ldbm.so >> #14 0x00007f53fa5459e3 in ava_candidates.isra.0 () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #15 0x00007f53fa545fd2 in filter_candidates_ext () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #16 0x00007f53fa547096 in list_candidates () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #17 0x00007f53fa545f40 in filter_candidates_ext () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #18 0x00007f53fa58180c in subtree_candidates () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #19 0x00007f53fa582e7f in ldbm_back_search () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #20 0x00007f5406eb01d6 in op_shared_search () from >> /usr/lib64/dirsrv/libslapd.so.0 >> #21 0x00007f54073a133e in do_search () >> #22 0x00007f540738f567 in connection_threadmain () >> #23 0x00007f540509e7bb in _pt_root () from /lib64/libnspr4.so >> #24 0x00007f5404a3fdc5 in start_thread () from /lib64/libpthread.so.0 >> #25 0x00007f540476d28d in clone () from /lib64/libc.so.6 >> Thread 7 (Thread 0x7f53cdfe3700 (LWP 128239)): >> #0 0x00007f5404a436d5 in pthread_cond_wait@@GLIBC_2.3.2 () from >> /lib64/libpthread.so.0 >> #1 0x00007f53fef9f2f3 in __db_hybrid_mutex_suspend () from /lib64/ >> libdb-5.3.so >> #2 0x00007f53fef9e640 in __db_tas_mutex_lock () from /lib64/libdb-5.3.so >> #3 0x00007f53ff048d3a in __lock_get_internal () from /lib64/libdb-5.3.so >> #4 0x00007f53ff049820 in __lock_get () from /lib64/libdb-5.3.so >> #5 0x00007f53ff075142 in __db_lget () from /lib64/libdb-5.3.so >> #6 0x00007f53fefbc605 in __bam_search () from /lib64/libdb-5.3.so >> #7 0x00007f53fefa7256 in __bamc_search () from /lib64/libdb-5.3.so >> #8 0x00007f53fefa8d0f in __bamc_get () from /lib64/libdb-5.3.so >> #9 0x00007f53ff061ca6 in __dbc_iget () from /lib64/libdb-5.3.so >> #10 0x00007f53ff070b02 in __dbc_get_pp () from /lib64/libdb-5.3.so >> #11 0x00007f53fa54c680 in idl_new_fetch () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #12 0x00007f53fa55ab26 in index_read_ext_allids () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #13 0x00007f53fa545284 in keys2idl () from /usr/lib64/dirsrv/plugins/libb >> ack-ldbm.so >> #14 0x00007f53fa5459e3 in ava_candidates.isra.0 () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #15 0x00007f53fa545fd2 in filter_candidates_ext () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #16 0x00007f53fa547096 in list_candidates () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #17 0x00007f53fa545f40 in filter_candidates_ext () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #18 0x00007f53fa58180c in subtree_candidates () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #19 0x00007f53fa582e7f in ldbm_back_search () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #20 0x00007f5406eb01d6 in op_shared_search () from >> /usr/lib64/dirsrv/libslapd.so.0 >> #21 0x00007f54073a133e in do_search () >> #22 0x00007f540738f567 in connection_threadmain () >> #23 0x00007f540509e7bb in _pt_root () from /lib64/libnspr4.so >> #24 0x00007f5404a3fdc5 in start_thread () from /lib64/libpthread.so.0 >> #25 0x00007f540476d28d in clone () from /lib64/libc.so.6 >> Thread 6 (Thread 0x7f53cd7e2700 (LWP 128240)): >> #0 0x00007f5404a436d5 in pthread_cond_wait@@GLIBC_2.3.2 () from >> /lib64/libpthread.so.0 >> #1 0x00007f53fef9f2f3 in __db_hybrid_mutex_suspend () from /lib64/ >> libdb-5.3.so >> #2 0x00007f53fef9eb37 in __db_tas_mutex_readlock () from /lib64/ >> libdb-5.3.so >> #3 0x00007f53ff0b8d35 in __memp_fget () from /lib64/libdb-5.3.so >> #4 0x00007f53ff062333 in __dbc_iget () from /lib64/libdb-5.3.so >> #5 0x00007f53ff070b02 in __dbc_get_pp () from /lib64/libdb-5.3.so >> #6 0x00007f53fa54c822 in idl_new_fetch () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #7 0x00007f53fa55ab26 in index_read_ext_allids () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #8 0x00007f53fa545284 in keys2idl () from /usr/lib64/dirsrv/plugins/libb >> ack-ldbm.so >> #9 0x00007f53fa5459e3 in ava_candidates.isra.0 () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #10 0x00007f53fa545fd2 in filter_candidates_ext () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #11 0x00007f53fa547096 in list_candidates () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #12 0x00007f53fa545f40 in filter_candidates_ext () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #13 0x00007f53fa547096 in list_candidates () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #14 0x00007f53fa545f40 in filter_candidates_ext () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #15 0x00007f53fa547096 in list_candidates () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #16 0x00007f53fa545f40 in filter_candidates_ext () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #17 0x00007f53fa58180c in subtree_candidates () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #18 0x00007f53fa582e7f in ldbm_back_search () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #19 0x00007f5406eb01d6 in op_shared_search () from >> /usr/lib64/dirsrv/libslapd.so.0 >> #20 0x00007f54073a133e in do_search () >> #21 0x00007f540738f3c3 in connection_threadmain () >> #22 0x00007f540509e7bb in _pt_root () from /lib64/libnspr4.so >> #23 0x00007f5404a3fdc5 in start_thread () from /lib64/libpthread.so.0 >> #24 0x00007f540476d28d in clone () from /lib64/libc.so.6 >> Thread 5 (Thread 0x7f53ccfe1700 (LWP 128241)): >> #0 0x00007f5404a436d5 in pthread_cond_wait@@GLIBC_2.3.2 () from >> /lib64/libpthread.so.0 >> #1 0x00007f53fef9f2f3 in __db_hybrid_mutex_suspend () from /lib64/ >> libdb-5.3.so >> #2 0x00007f53fef9e640 in __db_tas_mutex_lock () from /lib64/libdb-5.3.so >> #3 0x00007f53ff048d3a in __lock_get_internal () from /lib64/libdb-5.3.so >> #4 0x00007f53ff049820 in __lock_get () from /lib64/libdb-5.3.so >> #5 0x00007f53ff075142 in __db_lget () from /lib64/libdb-5.3.so >> #6 0x00007f53fefbc605 in __bam_search () from /lib64/libdb-5.3.so >> #7 0x00007f53fefa7256 in __bamc_search () from /lib64/libdb-5.3.so >> #8 0x00007f53fefa8d0f in __bamc_get () from /lib64/libdb-5.3.so >> #9 0x00007f53ff061ca6 in __dbc_iget () from /lib64/libdb-5.3.so >> #10 0x00007f53ff070b02 in __dbc_get_pp () from /lib64/libdb-5.3.so >> #11 0x00007f53fa54c680 in idl_new_fetch () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #12 0x00007f53fa55ab26 in index_read_ext_allids () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #13 0x00007f53fa545284 in keys2idl () from /usr/lib64/dirsrv/plugins/libb >> ack-ldbm.so >> #14 0x00007f53fa5459e3 in ava_candidates.isra.0 () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #15 0x00007f53fa545fd2 in filter_candidates_ext () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #16 0x00007f53fa547096 in list_candidates () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #17 0x00007f53fa545f40 in filter_candidates_ext () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #18 0x00007f53fa547096 in list_candidates () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #19 0x00007f53fa545f40 in filter_candidates_ext () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #20 0x00007f53fa547096 in list_candidates () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #21 0x00007f53fa545f40 in filter_candidates_ext () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #22 0x00007f53fa58180c in subtree_candidates () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #23 0x00007f53fa582e7f in ldbm_back_search () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #24 0x00007f5406eb01d6 in op_shared_search () from >> /usr/lib64/dirsrv/libslapd.so.0 >> #25 0x00007f54073a133e in do_search () >> #26 0x00007f540738f3c3 in connection_threadmain () >> #27 0x00007f540509e7bb in _pt_root () from /lib64/libnspr4.so >> #28 0x00007f5404a3fdc5 in start_thread () from /lib64/libpthread.so.0 >> #29 0x00007f540476d28d in clone () from /lib64/libc.so.6 >> Thread 4 (Thread 0x7f53cc7e0700 (LWP 128242)): >> #0 0x00007f54047649b3 in select () from /lib64/libc.so.6 >> #1 0x00007f5406eef0e9 in DS_Sleep () from /usr/lib64/dirsrv/libslapd.so. >> 0 >> #2 0x00007f5407390325 in time_thread () >> #3 0x00007f540509e7bb in _pt_root () from /lib64/libnspr4.so >> #4 0x00007f5404a3fdc5 in start_thread () from /lib64/libpthread.so.0 >> #5 0x00007f540476d28d in clone () from /lib64/libc.so.6 >> Thread 3 (Thread 0x7f53cb7a0700 (LWP 128259)): >> #0 0x00007f5404a436d5 in pthread_cond_wait@@GLIBC_2.3.2 () from >> /lib64/libpthread.so.0 >> #1 0x00007f5405099050 in PR_WaitCondVar () from /lib64/libnspr4.so >> #2 0x00007f540739bf25 in ps_send_results () >> #3 0x00007f540509e7bb in _pt_root () from /lib64/libnspr4.so >> #4 0x00007f5404a3fdc5 in start_thread () from /lib64/libpthread.so.0 >> #5 0x00007f540476d28d in clone () from /lib64/libc.so.6 >> Thread 2 (Thread 0x7f53cab9d700 (LWP 9020)): >> #0 0x00007f5404a436d5 in pthread_cond_wait@@GLIBC_2.3.2 () from >> /lib64/libpthread.so.0 >> #1 0x00007f5405099050 in PR_WaitCondVar () from /lib64/libnspr4.so >> #2 0x00007f540739bf25 in ps_send_results () >> #3 0x00007f540509e7bb in _pt_root () from /lib64/libnspr4.so >> #4 0x00007f5404a3fdc5 in start_thread () from /lib64/libpthread.so.0 >> #5 0x00007f540476d28d in clone () from /lib64/libc.so.6 >> Thread 1 (Thread 0x7f5407357840 (LWP 128124)): >> #0 0x00007f5404762c3d in poll () from /lib64/libc.so.6 >> #1 0x00007f540509a967 in _pr_poll_with_poll () from /lib64/libnspr4.so >> #2 0x00007f5407393ff1 in slapd_daemon () >> #3 0x00007f5407385c6c in main () >> >> >> Thanks >> >> On Tue, Sep 6, 2016 at 5:51 PM, Rakesh Rajasekharan < >> rakesh.rajasekharan at gmail.com> wrote: >> >>> Hi, >>> >>> With script provided by Rich, I looked at my db cache usage and it does >>> not look to be high enough >>> >>> # under each db are the list of selected indexes specified with INDEXLIST >>> dbname count free free% size >>> changelog:ent 5 2076333 99.0 4163.8 >>> changelog:dn 15976 9303557 88.7 74.0 >>> userroot:ent 4763 175321074 83.6 7221.1 >>> userroot:dn 4763 9995056 95.3 103.0 >>> ipaca:ent 425 6337866 60.4 9759.8 >>> ipaca:dn 425 10451081 99.7 81.6 >>> >>> Right now I set it to 200MB. So, I am planning to bring it down to 100MB >>> >>> Also, *nsslapd-db-checkpoint-interval *will change it to 80 or 90 from >>> default of 60 >>> >>> I will check this out and the impact it has on the hang issues and get >>> back >>> >>> Thanks >>> Rakesh >>> >>> On Tue, Sep 6, 2016 at 4:13 PM, Ludwig Krispenz < >>> lkrispen at redhat.com> wrote: >>> >>>> Hi, >>>> >>>> I think you are running into this bug: >>>> >>>> https://bugzilla.redhat.com/show_bug.cgi?id=1349779 >>>> it was the outcome of an analysis started in this one: >>>> https://bugzilla.redhat.com/show_bug.cgi?id=1347800 >>>> >>>> the problem is that it looks like the checkpoint thread is holding a >>>> mutex and otehr threads are blocked on it. It looks like an issue in >>>> BerkeleyDB although there is no explanation from the BDB code why this >>>> could happen, we also had a support case with Oracle to investigate this, >>>> with no solution. Oracle at the end was providing a diagnostic patch, which >>>> we did not test since the customer with the original problem had worked >>>> around this by changing their configuration. >>>> I don't know the status of this patch and if we can make it available, >>>> probably depends on your support status, you need to contact support for >>>> this. >>>> >>>> About the workarounds: >>>> you can try to change timing for the database update and checkpointing >>>> eg by reducing the dbcache size, changing the checkpoint interval. In the >>>> customer case they identified one substring index, which was frequently >>>> updated but not really required - and removing this index helped. >>>> >>>> Regards, >>>> Ludwig >>>> >>>> On 09/06/2016 11:31 AM, thierry bordaz wrote: >>>> >>>> Hi, >>>> >>>> The following thread is hanging most of the others threads >>>> >>>> Thread 20 (Thread 0x7f68007f8700 (LWP 4873)): >>>> >>>> #0 0x00007f683d0506d5 in >>>> pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 >>>> #1 0x00007f68375ac2f3 in __db_hybrid_mutex_suspend () from /lib64/ >>>> libdb-5.3.so >>>> #2 0x00007f68375ab640 in __db_tas_mutex_lock () from /lib64/ >>>> libdb-5.3.so >>>> #3 0x00007f68376c5e1b in __memp_fget () from /lib64/libdb-5.3.so >>>> #4 0x00007f68375c9211 in __bam_search () from /lib64/libdb-5.3.so >>>> #5 0x00007f68375b4256 in __bamc_search () from /lib64/libdb-5.3.so >>>> #6 0x00007f68375b86c4 in __bamc_put () from /lib64/libdb-5.3.so >>>> #7 0x00007f683766f6d5 in __dbc_iput () from /lib64/libdb-5.3.so >>>> #8 0x00007f683766aa0e in __db_put () from /lib64/libdb-5.3.so >>>> #9 0x00007f683767ffa4 in __db_put_pp () from /lib64/libdb-5.3.so >>>> #10 0x00007f6832b5a4cf in idl_new_insert_key () from >>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>> #11 0x00007f6832b695b5 in addordel_values_sv.isra.3 () from >>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>> #12 0x00007f6832b6a25a in index_addordel_values_ext_sv () from >>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>> #13 0x00007f6832b6a694 in index_addordel_values_sv () from >>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>> #14 0x00007f6832b6a75a in index_addordel_entry () from >>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>> #15 0x00007f6832b6d5fa in ldbm_back_add () from >>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>> #16 0x00007f683f46fcca in op_shared_add () from >>>> /usr/lib64/dirsrv/libslapd.so.0 >>>> #17 0x00007f683f471020 in do_add () from /usr/lib64/dirsrv/libslapd.so. >>>> 0 >>>> #18 0x00007f683f99c303 in connection_threadmain () >>>> #19 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so >>>> #20 0x00007f683d04cdc5 in start_thread () from /lib64/libpthread.so.0 >>>> #21 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 >>>> >>>> This thread is likely not hang on 'db_tas_mutex_lock' as the db_lock >>>> does not show it waiting for something. >>>> Unfortunately it is not shown in the output of top -H, did you get the >>>> complete output ? >>>> Ludwig mentioned a known issue that DS is not scaling well on NUMA >>>> arch, do you know if the machine is NUMA ? >>>> A possibility is that this thread get severely slowed down by a NUMA >>>> remote lock and showing DS has hung. >>>> This is why it would be interesting to know if it consums CPU. >>>> >>>> thanks >>>> theirry >>>> >>>> On 09/06/2016 10:59 AM, Rakesh Rajasekharan wrote: >>>> >>>> Hi Thierry/Ludwig, >>>> >>>> I again hit the hang issue. This is the pstack of the ns-slapd >>>> >>>> So was able to this time get the pstack and db locks >>>> >>>> db locks details >>>> Default locking region information: >>>> 451 Last allocated locker ID >>>> 0x7fffffff Current maximum unused locker ID >>>> 9 Number of lock modes >>>> 800 Initial number of locks allocated >>>> 0 Initial number of lockers allocated >>>> 800 Initial number of lock objects allocated >>>> 10000 Maximum number of locks possible >>>> 10000 Maximum number of lockers possible >>>> 10000 Maximum number of lock objects possible >>>> 800 Current number of locks allocated >>>> 366 Current number of lockers allocated >>>> 800 Current number of lock objects allocated >>>> 160 Number of lock object partitions >>>> 8191 Size of object hash table >>>> 126 Number of current locks >>>> 1005 Maximum number of locks at any one time >>>> 33 Maximum number of locks in any one bucket >>>> 3987 Maximum number of locks stolen by for an empty partition >>>> 75 Maximum number of locks stolen for any one partition >>>> 309 Number of current lockers >>>> 313 Maximum number of lockers at any one time >>>> 121 Number of current lock objects >>>> 676 Maximum number of lock objects at any one time >>>> 3 Maximum number of lock objects in any one bucket >>>> 69 Maximum number of objects stolen by for an empty partition >>>> 5 Maximum number of objects stolen for any one partition >>>> 22M Total number of locks requested (22009642) >>>> 21M Total number of locks released (21992878) >>>> 0 Total number of locks upgraded >>>> 95 Total number of locks downgraded >>>> 3348 Lock requests not available due to conflicts, for which we >>>> waited >>>> 12646 Lock requests not available due to conflicts, for which we >>>> did not wait >>>> 0 Number of deadlocks >>>> 0 Lock timeout value >>>> 0 Number of locks that have timed out >>>> 0 Transaction timeout value >>>> 0 Number of transactions that have timed out >>>> 2MB 560KB Region size >>>> 71719 The number of partition locks that required waiting (0%) >>>> 23105 The maximum number of times any partition lock was waited >>>> for (0%) >>>> 0 The number of object queue operations that required waiting >>>> (0%) >>>> 163 The number of locker allocations that required waiting (0%) >>>> 3172 The number of region locks that required waiting (0%) >>>> 5 Maximum hash bucket length >>>> =-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-= >>>> Lock REGINFO information: >>>> Environment Region type >>>> 1 Region ID >>>> /var/lib/dirsrv/slapd-SPRINKLR-COM/db/__db.001 Region name >>>> 0x7f67ef869000 Region address >>>> 0x7f67ef8690a0 Region allocation head >>>> 0x7f67ef8712b0 Region primary address >>>> 0 Region maximum allocation >>>> 0 Region allocated >>>> Region allocations: 1581345 allocations, 0 failures, 1580794 frees, 6 >>>> longest >>>> Allocations by power-of-two sizes: >>>> 1KB 1581329 >>>> 2KB 3 >>>> 4KB 4 >>>> 8KB 3 >>>> 16KB 3 >>>> 32KB 1 >>>> 64KB 0 >>>> 128KB 0 >>>> 256KB 2 >>>> 512KB 0 >>>> 1024KB 1 >>>> REGION_SHARED Region flags >>>> =-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-= >>>> Lock region parameters: >>>> 2 Lock region region mutex [3172/14M 0% !Own] >>>> 16381 locker table size >>>> 8191 object table size >>>> 34128 obj_off >>>> 900216 locker_off >>>> 0 need_dd >>>> =-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-= >>>> Lock conflict matrix: >>>> =-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-= >>>> Locks grouped by lockers: >>>> Locker Mode Count Status ----------------- Object >>>> --------------- >>>> 2 dd=308 locks held 1 write locks 0 pid/thread >>>> 4845/140085720139840 flags 10 priority 100 >>>> 2 READ 1 HELD userRoot/id2entry.db >>>> handle 0 >>>> 3 dd=307 locks held 0 write locks 0 pid/thread >>>> 4845/140084829472512 flags 0 priority 100 >>>> 4 dd=306 locks held 1 write locks 0 pid/thread >>>> 4845/140085720139840 flags 10 priority 100 >>>> 4 READ 1 HELD ipaca/id2entry.db >>>> handle 0 >>>> 5 dd=305 locks held 0 write locks 0 pid/thread >>>> 4845/140084837865216 flags 0 priority 100 >>>> 6 dd=304 locks held 1 write locks 0 pid/thread >>>> 4845/140085720139840 flags 10 priority 100 >>>> 6 READ 1 HELD ipaca/entryrdn.db >>>> handle 0 >>>> 7 dd=303 locks held 0 write locks 0 pid/thread >>>> 4845/140084804294400 flags 0 priority 100 >>>> 8 dd=302 locks held 0 write locks 0 pid/thread >>>> 4845/140084720432896 flags 0 priority 100 >>>> 9 dd=301 locks held 0 write locks 0 pid/thread >>>> 4845/140084837865216 flags 0 priority 100 >>>> a dd=300 locks held 1 write locks 0 pid/thread >>>> 4845/140085720139840 flags 10 priority 100 >>>> a READ 1 HELD ipaca/vlv#allcertspkitomcatindex.db >>>> handle 0 >>>> e dd=299 locks held 1 write locks 0 pid/thread >>>> 4845/140085720139840 flags 10 priority 100 >>>> e READ 1 HELD ipaca/vlv#allnonrevokedcertspkitomcatindex.db >>>> handle 0 >>>> 10 dd=298 locks held 1 write locks 0 pid/thread >>>> 4845/140085720139840 flags 10 priority 100 >>>> 10 READ 1 HELD ipaca/vlv#allrevokedcertspkitomcatindex.db >>>> handle 0 >>>> 11 dd=297 locks held 1 write locks 0 pid/thread >>>> 4845/140085720139840 flags 10 priority 100 >>>> 11 READ 1 HELD ipaca/vlv#allrevokedcertsnotafterpkitomcatindex.db >>>> handle 0 >>>> 14 dd=296 locks held 1 write locks 0 pid/thread >>>> 4845/140085720139840 flags 10 priority 100 >>>> 14 READ 1 HELD ipaca/vlv#allrevokedorrevokedexpiredcertspkitomcatindex.db >>>> handle 0 >>>> 15 dd=295 locks held 1 write locks 0 pid/thread >>>> 4845/140085720139840 flags 10 priority 100 >>>> 15 READ 1 HELD ipaca/vlv#allvalidcertspkitomcatindex.db >>>> handle 0 >>>> 16 dd=294 locks held 1 write locks 0 pid/thread >>>> 4845/140085720139840 flags 10 priority 100 >>>> 16 READ 1 HELD ipaca/vlv#allvalidcertsnotafterpkitomcatindex.db >>>> handle 0 >>>> 17 dd=293 locks held 1 write locks 0 pid/thread >>>> 4845/140085720139840 flags 10 priority 100 >>>> 17 READ 1 HELD ipaca/vlv#allvalidorrevokedcertspkitomcatindex.db >>>> handle 0 >>>> 18 dd=292 locks held 1 write locks 0 pid/thread >>>> 4845/140085720139840 flags 10 priority 100 >>>> 18 READ 1 HELD ipaca/vlv#caallpkitomcatindex.db >>>> handle 0 >>>> 1d dd=291 locks held 1 write locks 0 pid/thread >>>> 4845/140085720139840 flags 10 priority 100 >>>> 1d READ 1 HELD ipaca/vlv#cacompletepkitomcatindex.db >>>> handle 0 >>>> 1e dd=290 locks held 1 write locks 0 pid/thread >>>> 4845/140085720139840 flags 10 priority 100 >>>> 1e READ 1 HELD ipaca/vlv#cacompleteenrollmentpkitomcatindex.db >>>> handle 0 >>>> 20 dd=289 locks held 1 write locks 0 pid/thread >>>> 4845/140085720139840 flags 10 priority 100 >>>> 20 READ 1 HELD ipaca/vlv#cacompleterevocationpkitomcatindex.db >>>> handle 0 >>>> 21 dd=288 locks held 1 write locks 0 pid/thread >>>> 4845/140085720139840 flags 10 priority 100 >>>> 21 READ 1 HELD ipaca/vlv#caenrollmentpkitomcatindex.db >>>> handle 0 >>>> 2b dd=287 locks held 1 write locks 0 pid/thread >>>> 4845/140085720139840 flags 10 priority 100 >>>> 2b READ 1 HELD ipaca/vlv#carevocationpkitomcatindex.db >>>> handle 0 >>>> 2c dd=286 locks held 1 write locks 0 pid/thread >>>> 4845/140085720139840 flags 10 priority 100 >>>> 2c READ 1 HELD changelog/id2entry.db >>>> handle 0 >>>> 2d dd=285 locks held 0 write locks 0 pid/thread >>>> 4845/140085719607040 flags 0 priority 100 >>>> 2e dd=284 locks held 1 write locks 0 pid/thread >>>> 4845/140085720139840 flags 10 priority 100 >>>> 2e READ 1 HELD changelog/entryusn.db >>>> handle 0 >>>> 2f dd=283 locks held 0 write locks 0 pid/thread >>>> 4845/140085720139840 flags 0 priority 100 >>>> 30 dd=282 locks held 1 write locks 0 pid/thread >>>> 4845/140085720139840 flags 10 priority 100 >>>> 30 READ 1 HELD userRoot/entryusn.db >>>> handle 0 >>>> 31 dd=281 locks held 0 write locks 0 pid/thread >>>> 4845/140085720139840 flags 0 priority 100 >>>> 32 dd=280 locks held 1 write locks 0 pid/thread >>>> 4845/140085720139840 flags 10 priority 100 >>>> 32 READ 1 HELD ipaca/entryusn.db >>>> handle 0 >>>> 33 dd=279 locks held 0 write locks 0 pid/thread >>>> 4845/140085720139840 flags 0 priority 100 >>>> 34 dd=278 locks held 1 write locks 0 pid/thread >>>> 4845/140085720139840 flags 10 priority 100 >>>> 34 READ 1 HELD userRoot/entryrdn.db >>>> handle 0 >>>> 35 dd=277 locks held 0 write locks 0 pid/thread >>>> 4845/140084544186112 flags 0 priority 100 >>>> 36 dd=276 locks held 0 write locks 0 pid/thread >>>> 4845/140084636505856 flags 0 priority 100 >>>> 37 dd=275 locks held 0 write locks 0 pid/thread >>>> 4845/140084854650624 flags 0 priority 100 >>>> 38 dd=274 locks held 1 write locks 0 pid/thread >>>> 4845/140085720139840 flags 10 priority 100 >>>> 38 READ 1 HELD userRoot/objectclass.db >>>> handle 0 >>>> 39 dd=273 locks held 0 write locks 0 pid/thread >>>> 4845/140084569364224 flags 0 priority 100 >>>> 3a dd=272 locks held 1 write locks 0 pid/thread >>>> 4845/140085720139840 flags 10 priority 100 >>>> 3a READ 1 HELD userRoot/ancestorid.db >>>> handle 0 >>>> 3b dd=271 locks held 0 write locks 0 pid/thread >>>> 4845/140084854650624 flags 0 priority 100 >>>> 3c dd=270 locks held 1 write locks 0 pid/thread >>>> 4845/140085720139840 flags 10 priority 100 >>>> 3c READ 1 HELD changelog/entryrdn.db >>>> handle 0 >>>> 3d dd=269 locks held 0 write locks 0 pid/thread >>>> 4845/140085719607040 flags 0 priority 100 >>>> 3e dd=268 locks held 0 write locks 0 pid/thread >>>> 4845/140084602935040 flags 0 priority 100 >>>> 3f dd=267 locks held 0 write locks 0 pid/thread >>>> 4845/140085719607040 flags 0 priority 100 >>>> 40 dd=266 locks held 1 write locks 0 pid/thread >>>> 4845/140085720139840 flags 10 priority 100 >>>> 40 READ 1 HELD changelog/objectclass.db >>>> handle 0 >>>> 41 dd=265 locks held 0 write locks 0 pid/thread >>>> 4845/140085719607040 flags 0 priority 100 >>>> 42 dd=264 locks held 0 write locks 0 pid/thread >>>> 4845/140085719607040 flags 0 priority 100 >>>> 43 dd=263 locks held 0 write locks 0 pid/thread >>>> 4845/140084653291264 flags 0 priority 100 >>>> 44 dd=262 locks held 1 write locks 0 pid/thread >>>> 4845/140085720139840 flags 10 priority 100 >>>> 44 READ 1 HELD ipaca/objectclass.db >>>> handle 0 >>>> 45 dd=261 locks held 0 write locks 0 pid/thread >>>> 4845/140084577756928 flags 0 priority 100 >>>> 46 dd=260 locks held 0 write locks 0 pid/thread >>>> 4845/140084720432896 flags 0 priority 100 >>>> 47 dd=259 locks held 1 write locks 0 pid/thread >>>> 4845/140085720139840 flags 10 priority 100 >>>> 47 READ 1 HELD changelog/aci.db >>>> handle 0 >>>> 48 dd=258 locks held 0 write locks 0 pid/thread >>>> 4845/140085720139840 flags 0 priority 100 >>>> 49 dd=257 locks held 1 write locks 0 pid/thread >>>> 4845/140085720139840 flags 10 priority 100 >>>> 49 READ 1 HELD userRoot/aci.db >>>> handle 0 >>>> 4a dd=256 locks held 0 write locks 0 pid/thread >>>> 4845/140085720139840 flags 0 priority 100 >>>> 4b dd=255 locks held 0 write locks 0 pid/thread >>>> 4845/140084535793408 flags 0 priority 100 >>>> 4c dd=254 locks held 1 write locks 0 pid/thread >>>> 4845/140085720139840 flags 10 priority 100 >>>> 4c READ 1 HELD ipaca/aci.db >>>> handle 0 >>>> 4d dd=253 locks held 0 write locks 0 pid/thread >>>> 4845/140085720139840 flags 0 priority 100 >>>> 4e dd=252 locks held 1 write locks 0 pid/thread >>>> 4845/140085720139840 flags 10 priority 100 >>>> 4e READ 1 HELD userRoot/parentid.db >>>> handle 0 >>>> 4f dd=251 locks held 0 write locks 0 pid/thread >>>> 4845/140085720139840 flags 0 priority 100 >>>> 50 dd=250 locks held 0 write locks 0 pid/thread >>>> 4845/140085719607040 flags 0 priority 100 >>>> 51 dd=249 locks held 1 write locks 0 pid/thread >>>> 4845/140085720139840 flags 10 priority 100 >>>> 51 READ 1 HELD changelog/nsuniqueid.db >>>> handle 0 >>>> 52 dd=248 locks held 1 write locks 0 pid/thread >>>> 4845/140085720139840 flags 10 priority 100 >>>> 52 READ 1 HELD changelog/changenumber.db >>>> handle 0 >>>> 53 dd=247 locks held 0 write locks 0 pid/thread >>>> 4845/140084602935040 flags 0 priority 100 >>>> 54 dd=246 locks held 0 write locks 0 pid/thread >>>> 4845/140085719607040 flags 0 priority 100 >>>> 55 dd=245 locks held 1 write locks 0 pid/thread >>>> 4845/140085719607040 flags 10 priority 100 >>>> 55 READ 1 HELD changelog/targetuniqueid.db >>>> handle 0 >>>> 56 dd=244 locks held 1 write locks 0 pid/thread >>>> 4845/140085719607040 flags 10 priority 100 >>>> 56 READ 1 HELD changelog/parentid.db >>>> handle 0 >>>> 57 dd=243 locks held 1 write locks 0 pid/thread >>>> 4845/140085719607040 flags 10 priority 100 >>>> 57 READ 1 HELD changelog/ancestorid.db >>>> handle 0 >>>> 58 dd=242 locks held 1 write locks 0 pid/thread >>>> 4845/140085719607040 flags 10 priority 100 >>>> 58 READ 1 HELD changelog/numsubordinates.db >>>> handle 0 >>>> 59 dd=241 locks held 0 write locks 0 pid/thread >>>> 4845/140085719607040 flags 0 priority 100 >>>> 5a dd=240 locks held 0 write locks 0 pid/thread >>>> 4845/140084544186112 flags 0 priority 100 >>>> 5b dd=239 locks held 1 write locks 0 pid/thread >>>> 4845/140084577756928 flags 10 priority 100 >>>> 5b READ 1 HELD userRoot/krbPrincipalName.db >>>> handle 0 >>>> 5c dd=238 locks held 0 write locks 0 pid/thread >>>> 4845/140084636505856 flags 0 priority 100 >>>> 5d dd=237 locks held 0 write locks 0 pid/thread >>>> 4845/140084695254784 flags 0 priority 100 >>>> 5e dd=236 locks held 1 write locks 0 pid/thread >>>> 4845/140084560971520 flags 10 priority 100 >>>> 5e READ 1 HELD userRoot/ipakrbprincipalalias.db >>>> handle 0 >>>> 5f dd=235 locks held 0 write locks 0 pid/thread >>>> 4845/140084703647488 flags 0 priority 100 >>>> 60 dd=234 locks held 0 write locks 0 pid/thread >>>> 4845/140084653291264 flags 0 priority 100 >>>> 61 dd=233 locks held 0 write locks 0 pid/thread >>>> 4845/140084577756928 flags 0 priority 100 >>>> 62 dd=232 locks held 0 write locks 0 pid/thread >>>> 4845/140084686862080 flags 0 priority 100 >>>> 63 dd=231 locks held 1 write locks 0 pid/thread >>>> 4845/140084611327744 flags 10 priority 100 >>>> 63 READ 1 HELD changelog/seeAlso.db >>>> handle 0 >>>> 64 dd=230 locks held 0 write locks 0 pid/thread >>>> 4845/140084686862080 flags 0 priority 100 >>>> 65 dd=229 locks held 1 write locks 0 pid/thread >>>> 4845/140084611327744 flags 10 priority 100 >>>> 65 READ 1 HELD userRoot/seeAlso.db >>>> handle 0 >>>> 66 dd=228 locks held 0 write locks 0 pid/thread >>>> 4845/140084636505856 flags 0 priority 100 >>>> 67 dd=227 locks held 1 write locks 0 pid/thread >>>> 4845/140084611327744 flags 10 priority 100 >>>> 67 READ 1 HELD ipaca/seeAlso.db >>>> handle 0 >>>> 68 dd=226 locks held 0 write locks 0 pid/thread >>>> 4845/140084686862080 flags 0 priority 100 >>>> 69 dd=225 locks held 1 write locks 0 pid/thread >>>> 4845/140084678469376 flags 10 priority 100 >>>> 69 READ 1 HELD ipaca/parentid.db >>>> handle 0 >>>> 6a dd=224 locks held 0 write locks 0 pid/thread >>>> 4845/140084678469376 flags 0 priority 100 >>>> 6b dd=223 locks held 0 write locks 0 pid/thread >>>> 4845/140084829472512 flags 0 priority 100 >>>> 6c dd=222 locks held 0 write locks 0 pid/thread >>>> 4845/140084653291264 flags 0 priority 100 >>>> 6d dd=221 locks held 0 write locks 0 pid/thread >>>> 4845/140084569364224 flags 0 priority 100 >>>> 6e dd=220 locks held 0 write locks 0 pid/thread >>>> 4845/140084636505856 flags 0 priority 100 >>>> 6e READ 1 WAIT userRoot/objectclass.db page >>>> 18 >>>> 6f dd=219 locks held 0 write locks 0 pid/thread >>>> 4845/140084544186112 flags 0 priority 100 >>>> 70 dd=218 locks held 0 write locks 0 pid/thread >>>> 4845/140084569364224 flags 0 priority 100 >>>> 71 dd=217 locks held 0 write locks 0 pid/thread >>>> 4845/140084619720448 flags 0 priority 100 >>>> 72 dd=216 locks held 0 write locks 0 pid/thread >>>> 4845/140084552578816 flags 0 priority 100 >>>> 73 dd=215 locks held 0 write locks 0 pid/thread >>>> 4845/140084829472512 flags 0 priority 100 >>>> 74 dd=214 locks held 0 write locks 0 pid/thread >>>> 4845/140084628113152 flags 0 priority 100 >>>> 75 dd=213 locks held 0 write locks 0 pid/thread >>>> 4845/140084686862080 flags 0 priority 100 >>>> 76 dd=212 locks held 1 write locks 0 pid/thread >>>> 4845/140084560971520 flags 10 priority 100 >>>> 76 READ 1 HELD userRoot/ipauniqueid.db >>>> handle 0 >>>> 77 dd=211 locks held 0 write locks 0 pid/thread >>>> 4845/140084661683968 flags 0 priority 100 >>>> 78 dd=210 locks held 0 write locks 0 pid/thread >>>> 4845/140084846257920 flags 0 priority 100 >>>> 79 dd=209 locks held 0 write locks 0 pid/thread >>>> 4845/140084628113152 flags 0 priority 100 >>>> 7a dd=208 locks held 1 write locks 0 pid/thread >>>> 4845/140084560971520 flags 10 priority 100 >>>> 7a READ 1 HELD userRoot/nsuniqueid.db >>>> handle 0 >>>> 7b dd=207 locks held 1 write locks 0 pid/thread >>>> 4845/140084560971520 flags 10 priority 100 >>>> 7b READ 1 HELD userRoot/cn.db >>>> handle 0 >>>> 7c dd=206 locks held 1 write locks 0 pid/thread >>>> 4845/140084560971520 flags 10 priority 100 >>>> 7c READ 1 HELD userRoot/fqdn.db >>>> handle 0 >>>> 7d dd=205 locks held 1 write locks 0 pid/thread >>>> 4845/140084560971520 flags 10 priority 100 >>>> 7d READ 1 HELD userRoot/managedby.db >>>> handle 0 >>>> 7e dd=204 locks held 1 write locks 0 pid/thread >>>> 4845/140084560971520 flags 10 priority 100 >>>> 7e READ 1 HELD userRoot/numsubordinates.db >>>> handle 0 >>>> 7f dd=203 locks held 1 write locks 0 pid/thread >>>> 4845/140084678469376 flags 10 priority 100 >>>> 7f READ 1 HELD userRoot/member.db >>>> handle 0 >>>> 80 dd=202 locks held 0 write locks 0 pid/thread >>>> 4845/140084712040192 flags 0 priority 100 >>>> 81 dd=201 locks held 1 write locks 0 pid/thread >>>> 4845/140084678469376 flags 10 priority 100 >>>> 81 READ 1 HELD userRoot/memberUser.db >>>> handle 0 >>>> 82 dd=200 locks held 0 write locks 0 pid/thread >>>> 4845/140084577756928 flags 0 priority 100 >>>> 83 dd=199 locks held 1 write locks 0 pid/thread >>>> 4845/140084678469376 flags 10 priority 100 >>>> 83 READ 1 HELD userRoot/memberHost.db >>>> handle 0 >>>> 84 dd=198 locks held 0 write locks 0 pid/thread >>>> 4845/140084712040192 flags 0 priority 100 >>>> 85 dd=197 locks held 0 write locks 0 pid/thread >>>> 4845/140084577756928 flags 0 priority 100 >>>> 86 dd=196 locks held 0 write locks 0 pid/thread >>>> 4845/140084670076672 flags 0 priority 100 >>>> 87 dd=195 locks held 0 write locks 0 pid/thread >>>> 4845/140084577756928 flags 0 priority 100 >>>> 88 dd=194 locks held 0 write locks 0 pid/thread >>>> 4845/140084846257920 flags 0 priority 100 >>>> 89 dd=193 locks held 0 write locks 0 pid/thread >>>> 4845/140084552578816 flags 0 priority 100 >>>> 8a dd=192 locks held 0 write locks 0 pid/thread >>>> 4845/140084703647488 flags 0 priority 100 >>>> 8b dd=191 locks held 0 write locks 0 pid/thread >>>> 4845/140084544186112 flags 0 priority 100 >>>> 8c dd=190 locks held 0 write locks 0 pid/thread >>>> 4845/140084846257920 flags 0 priority 100 >>>> 8d dd=189 locks held 0 write locks 0 pid/thread >>>> 4845/140084661683968 flags 0 priority 100 >>>> 8e dd=188 locks held 1 write locks 0 pid/thread >>>> 4845/140085067675392 flags 10 priority 100 >>>> 8e READ 1 HELD userRoot/macAddress.db >>>> handle 0 >>>> 8f dd=187 locks held 0 write locks 0 pid/thread >>>> 4845/140085067675392 flags 0 priority 100 >>>> 90 dd=186 locks held 0 write locks 0 pid/thread >>>> 4845/140084703647488 flags 0 priority 100 >>>> 91 dd=185 locks held 0 write locks 0 pid/thread >>>> 4845/140084552578816 flags 0 priority 100 >>>> 92 dd=184 locks held 0 write locks 0 pid/thread >>>> 4845/140084560971520 flags 0 priority 100 >>>> 92 READ 1 WAIT userRoot/objectclass.db >>>> page 9 >>>> 93 dd=183 locks held 0 write locks 0 pid/thread >>>> 4845/140084577756928 flags 0 priority 100 >>>> 94 dd=182 locks held 0 write locks 0 pid/thread >>>> 4845/140084821079808 flags 0 priority 100 >>>> 94 READ 1 WAIT userRoot/objectclass.db page >>>> 18 >>>> 95 dd=181 locks held 0 write locks 0 pid/thread >>>> 4845/140084544186112 flags 0 priority 100 >>>> 96 dd=180 locks held 0 write locks 0 pid/thread >>>> 4845/140084804294400 flags 0 priority 100 >>>> 97 dd=179 locks held 1 write locks 0 pid/thread >>>> 4845/140084712040192 flags 10 priority 100 >>>> 97 READ 1 HELD userRoot/uid.db >>>> handle 0 >>>> 98 dd=178 locks held 0 write locks 0 pid/thread >>>> 4845/140084552578816 flags 0 priority 100 >>>> 99 dd=177 locks held 0 write locks 0 pid/thread >>>> 4845/140084577756928 flags 0 priority 100 >>>> 9a dd=176 locks held 0 write locks 0 pid/thread >>>> 4845/140084602935040 flags 0 priority 100 >>>> 9b dd=175 locks held 1 write locks 0 pid/thread >>>> 4845/140085076068096 flags 10 priority 100 >>>> 9b READ 1 HELD userRoot/uniquemember.db >>>> handle 0 >>>> 9c dd=174 locks held 1 write locks 0 pid/thread >>>> 4845/140085076068096 flags 10 priority 100 >>>> 9c READ 1 HELD userRoot/owner.db >>>> handle 0 >>>> 9d dd=173 locks held 1 write locks 0 pid/thread >>>> 4845/140085076068096 flags 10 priority 100 >>>> 9d READ 1 HELD userRoot/ipaallowedtarget.db >>>> handle 0 >>>> 9e dd=172 locks held 1 write locks 0 pid/thread >>>> 4845/140085076068096 flags 10 priority 100 >>>> 9e READ 1 HELD userRoot/ipaMemberCertProfile.db >>>> handle 0 >>>> 9f dd=171 locks held 1 write locks 0 pid/thread >>>> 4845/140085076068096 flags 10 priority 100 >>>> 9f READ 1 HELD userRoot/ipaassignedidview.db >>>> handle 0 >>>> a0 dd=170 locks held 1 write locks 0 pid/thread >>>> 4845/140085076068096 flags 10 priority 100 >>>> a0 READ 1 HELD userRoot/secretary.db >>>> handle 0 >>>> a1 dd=169 locks held 1 write locks 0 pid/thread >>>> 4845/140085076068096 flags 10 priority 100 >>>> a1 READ 1 HELD userRoot/memberdenycmd.db >>>> handle 0 >>>> a2 dd=168 locks held 1 write locks 0 pid/thread >>>> 4845/140085076068096 flags 10 priority 100 >>>> a2 READ 1 HELD userRoot/memberallowcmd.db >>>> handle 0 >>>> a3 dd=167 locks held 1 write locks 0 pid/thread >>>> 4845/140085076068096 flags 10 priority 100 >>>> a3 READ 1 HELD userRoot/manager.db >>>> handle 0 >>>> a4 dd=166 locks held 1 write locks 0 pid/thread >>>> 4845/140085076068096 flags 10 priority 100 >>>> a4 READ 1 HELD userRoot/ipasudorunas.db >>>> handle 0 >>>> a5 dd=165 locks held 1 write locks 0 pid/thread >>>> 4845/140085076068096 flags 10 priority 100 >>>> a5 READ 1 HELD userRoot/ipaMemberCa.db >>>> handle 0 >>>> a6 dd=164 locks held 1 write locks 0 pid/thread >>>> 4845/140085076068096 flags 10 priority 100 >>>> a6 READ 1 HELD userRoot/ipasudorunasgroup.db >>>> handle 0 >>>> a7 dd=163 locks held 1 write locks 0 pid/thread >>>> 4845/140085076068096 flags 10 priority 100 >>>> a7 READ 1 HELD userRoot/sourcehost.db >>>> handle 0 >>>> a8 dd=162 locks held 1 write locks 0 pid/thread >>>> 4845/140085076068096 flags 10 priority 100 >>>> a8 READ 1 HELD userRoot/memberservice.db >>>> handle 0 >>>> a9 dd=161 locks held 1 write locks 0 pid/thread >>>> 4845/140085076068096 flags 10 priority 100 >>>> a9 READ 1 HELD userRoot/ipatokenradiusconfiglink.db >>>> handle 0 >>>> aa dd=160 locks held 0 write locks 0 pid/thread >>>> 4845/140084703647488 flags 0 priority 100 >>>> ab dd=159 locks held 0 write locks 0 pid/thread >>>> 4845/140084821079808 flags 0 priority 100 >>>> ac dd=158 locks held 0 write locks 0 pid/thread >>>> 4845/140084628113152 flags 0 priority 100 >>>> ad dd=157 locks held 0 write locks 0 pid/thread >>>> 4845/140084829472512 flags 0 priority 100 >>>> ae dd=156 locks held 0 write locks 0 pid/thread >>>> 4845/140084829472512 flags 0 priority 100 >>>> af dd=155 locks held 0 write locks 0 pid/thread >>>> 4845/140084854650624 flags 0 priority 100 >>>> b0 dd=154 locks held 0 write locks 0 pid/thread >>>> 4845/140084854650624 flags 0 priority 100 >>>> b1 dd=153 locks held 0 write locks 0 pid/thread >>>> 4845/140084837865216 flags 0 priority 100 >>>> b2 dd=152 locks held 0 write locks 0 pid/thread >>>> 4845/140084804294400 flags 0 priority 100 >>>> b3 dd=151 locks held 0 write locks 0 pid/thread >>>> 4845/140084812687104 flags 0 priority 100 >>>> b4 dd=150 locks held 0 write locks 0 pid/thread >>>> 4845/140084804294400 flags 0 priority 100 >>>> b5 dd=149 locks held 0 write locks 0 pid/thread >>>> 4845/140084837865216 flags 0 priority 100 >>>> b6 dd=148 locks held 0 write locks 0 pid/thread >>>> 4845/140084569364224 flags 0 priority 100 >>>> b7 dd=147 locks held 0 write locks 0 pid/thread >>>> 4845/140084829472512 flags 0 priority 100 >>>> b8 dd=146 locks held 0 write locks 0 pid/thread >>>> 4845/140084586149632 flags 0 priority 100 >>>> b9 dd=145 locks held 0 write locks 0 pid/thread >>>> 4845/140084653291264 flags 0 priority 100 >>>> ba dd=144 locks held 0 write locks 0 pid/thread >>>> 4845/140084636505856 flags 0 priority 100 >>>> bb dd=143 locks held 0 write locks 0 pid/thread >>>> 4845/140084804294400 flags 0 priority 100 >>>> bc dd=142 locks held 0 write locks 0 pid/thread >>>> 4845/140084804294400 flags 0 priority 100 >>>> bd dd=141 locks held 0 write locks 0 pid/thread >>>> 4845/140084695254784 flags 0 priority 100 >>>> be dd=140 locks held 0 write locks 0 pid/thread >>>> 4845/140084670076672 flags 0 priority 100 >>>> bf dd=139 locks held 0 write locks 0 pid/thread >>>> 4845/140084628113152 flags 0 priority 100 >>>> c0 dd=138 locks held 0 write locks 0 pid/thread >>>> 4845/140084661683968 flags 0 priority 100 >>>> c1 dd=137 locks held 0 write locks 0 pid/thread >>>> 4845/140084712040192 flags 0 priority 100 >>>> c2 dd=136 locks held 0 write locks 0 pid/thread >>>> 4845/140084661683968 flags 0 priority 100 >>>> c3 dd=135 locks held 0 write locks 0 pid/thread >>>> 4845/140084577756928 flags 0 priority 100 >>>> c4 dd=134 locks held 0 write locks 0 pid/thread >>>> 4845/140084854650624 flags 0 priority 100 >>>> c5 dd=133 locks held 1 write locks 0 pid/thread >>>> 4845/140084644898560 flags 10 priority 100 >>>> c5 READ 1 HELD userRoot/uidnumber.db >>>> handle 0 >>>> c6 dd=132 locks held 0 write locks 0 pid/thread >>>> 4845/140084594542336 flags 0 priority 100 >>>> c8 dd=131 locks held 1 write locks 0 pid/thread >>>> 4845/140084678469376 flags 10 priority 100 >>>> c8 READ 1 HELD ipaca/certstatus.db >>>> handle 0 >>>> c9 dd=130 locks held 0 write locks 0 pid/thread >>>> 4845/140084720432896 flags 0 priority 100 >>>> ca dd=129 locks held 0 write locks 0 pid/thread >>>> 4845/140084619720448 flags 0 priority 100 >>>> cb dd=128 locks held 0 write locks 0 pid/thread >>>> 4845/140084821079808 flags 0 priority 100 >>>> cc dd=127 locks held 0 write locks 0 pid/thread >>>> 4845/140084594542336 flags 0 priority 100 >>>> cd dd=126 locks held 0 write locks 0 pid/thread >>>> 4845/140084560971520 flags 0 priority 100 >>>> ce dd=125 locks held 1 write locks 0 pid/thread >>>> 4845/140084837865216 flags 10 priority 100 >>>> ce READ 1 HELD userRoot/gidnumber.db >>>> handle 0 >>>> cf dd=124 locks held 0 write locks 0 pid/thread >>>> 4845/140084720432896 flags 0 priority 100 >>>> d0 dd=123 locks held 0 write locks 0 pid/thread >>>> 4845/140084569364224 flags 0 priority 100 >>>> d1 dd=122 locks held 1 write locks 0 pid/thread >>>> 4845/140084678469376 flags 10 priority 100 >>>> d1 READ 1 HELD userRoot/nscpEntryDN.db >>>> handle 0 >>>> d2 dd=121 locks held 0 write locks 0 pid/thread >>>> 4845/140084695254784 flags 0 priority 100 >>>> d3 dd=120 locks held 0 write locks 0 pid/thread >>>> 4845/140084569364224 flags 0 priority 100 >>>> d4 dd=119 locks held 0 write locks 0 pid/thread >>>> 4845/140084653291264 flags 0 priority 100 >>>> d5 dd=118 locks held 0 write locks 0 pid/thread >>>> 4845/140084829472512 flags 0 priority 100 >>>> d6 dd=117 locks held 0 write locks 0 pid/thread >>>> 4845/140084661683968 flags 0 priority 100 >>>> d7 dd=116 locks held 0 write locks 0 pid/thread >>>> 4845/140084804294400 flags 0 priority 100 >>>> d8 dd=115 locks held 1 write locks 0 pid/thread >>>> 4845/140084821079808 flags 10 priority 100 >>>> d8 READ 1 HELD ipaca/description.db >>>> handle 0 >>>> d9 dd=114 locks held 0 write locks 0 pid/thread >>>> 4845/140084661683968 flags 0 priority 100 >>>> da dd=113 locks held 0 write locks 0 pid/thread >>>> 4845/140084837865216 flags 0 priority 100 >>>> db dd=112 locks held 1 write locks 0 pid/thread >>>> 4845/140084678469376 flags 10 priority 100 >>>> db READ 1 HELD ipaca/cn.db >>>> handle 0 >>>> dc dd=111 locks held 0 write locks 0 pid/thread >>>> 4845/140084577756928 flags 0 priority 100 >>>> dd dd=110 locks held 1 write locks 0 pid/thread >>>> 4845/140084678469376 flags 10 priority 100 >>>> dd READ 1 HELD ipaca/ancestorid.db >>>> handle 0 >>>> de dd=109 locks held 0 write locks 0 pid/thread >>>> 4845/140084577756928 flags 0 priority 100 >>>> df dd=108 locks held 0 write locks 0 pid/thread >>>> 4845/140084703647488 flags 0 priority 100 >>>> e0 dd=107 locks held 0 write locks 0 pid/thread >>>> 4845/140084569364224 flags 0 priority 100 >>>> e1 dd=106 locks held 0 write locks 0 pid/thread >>>> 4845/140084569364224 flags 0 priority 100 >>>> e2 dd=105 locks held 0 write locks 0 pid/thread >>>> 4845/140084821079808 flags 0 priority 100 >>>> e3 dd=104 locks held 1 write locks 0 pid/thread >>>> 4845/140084686862080 flags 10 priority 100 >>>> e3 READ 1 HELD ipaca/serialno.db >>>> handle 0 >>>> e4 dd=103 locks held 0 write locks 0 pid/thread >>>> 4845/140084837865216 flags 0 priority 100 >>>> e5 dd=102 locks held 1 write locks 0 pid/thread >>>> 4845/140084695254784 flags 10 priority 100 >>>> e5 READ 1 HELD ipaca/revInfo.db >>>> handle 0 >>>> e6 dd=101 locks held 1 write locks 0 pid/thread >>>> 4845/140084695254784 flags 10 priority 100 >>>> e6 READ 1 HELD ipaca/revokedby.db >>>> handle 0 >>>> e7 dd=100 locks held 1 write locks 0 pid/thread >>>> 4845/140084695254784 flags 10 priority 100 >>>> e7 READ 1 HELD ipaca/revokedOn.db >>>> handle 0 >>>> e8 dd=99 locks held 0 write locks 0 pid/thread >>>> 4845/140084829472512 flags 0 priority 100 >>>> e9 dd=98 locks held 0 write locks 0 pid/thread >>>> 4845/140084846257920 flags 0 priority 100 >>>> ea dd=97 locks held 1 write locks 0 pid/thread >>>> 4845/140084703647488 flags 10 priority 100 >>>> ea READ 1 HELD ipaca/nsuniqueid.db >>>> handle 0 >>>> eb dd=96 locks held 1 write locks 0 pid/thread >>>> 4845/140084703647488 flags 10 priority 100 >>>> eb READ 1 HELD ipaca/requestid.db >>>> handle 0 >>>> ec dd=95 locks held 1 write locks 0 pid/thread >>>> 4845/140084703647488 flags 10 priority 100 >>>> ec READ 1 HELD ipaca/requeststate.db >>>> handle 0 >>>> ed dd=94 locks held 1 write locks 0 pid/thread >>>> 4845/140084703647488 flags 10 priority 100 >>>> ed READ 1 HELD ipaca/dateOfCreate.db >>>> handle 0 >>>> ee dd=93 locks held 1 write locks 0 pid/thread >>>> 4845/140084703647488 flags 10 priority 100 >>>> ee READ 1 HELD ipaca/requesttype.db >>>> handle 0 >>>> ef dd=92 locks held 1 write locks 0 pid/thread >>>> 4845/140084703647488 flags 10 priority 100 >>>> ef READ 1 HELD ipaca/numsubordinates.db >>>> handle 0 >>>> f0 dd=91 locks held 1 write locks 0 pid/thread >>>> 4845/140084611327744 flags 10 priority 100 >>>> f0 READ 1 HELD userRoot/userCertificate.db >>>> handle 0 >>>> f1 dd=90 locks held 0 write locks 0 pid/thread >>>> 4845/140084821079808 flags 0 priority 100 >>>> f2 dd=89 locks held 0 write locks 0 pid/thread >>>> 4845/140084569364224 flags 0 priority 100 >>>> f3 dd=88 locks held 0 write locks 0 pid/thread >>>> 4845/140084602935040 flags 0 priority 100 >>>> f4 dd=87 locks held 0 write locks 0 pid/thread >>>> 4845/140084720432896 flags 0 priority 100 >>>> f5 dd=86 locks held 0 write locks 0 pid/thread >>>> 4845/140084628113152 flags 0 priority 100 >>>> f6 dd=85 locks held 0 write locks 0 pid/thread >>>> 4845/140084703647488 flags 0 priority 100 >>>> f7 dd=84 locks held 0 write locks 0 pid/thread >>>> 4845/140084611327744 flags 0 priority 100 >>>> f8 dd=83 locks held 0 write locks 0 pid/thread >>>> 4845/140084661683968 flags 0 priority 100 >>>> f9 dd=82 locks held 0 write locks 0 pid/thread >>>> 4845/140084837865216 flags 0 priority 100 >>>> fa dd=81 locks held 0 write locks 0 pid/thread >>>> 4845/140084544186112 flags 0 priority 100 >>>> fb dd=80 locks held 0 write locks 0 pid/thread >>>> 4845/140084703647488 flags 0 priority 100 >>>> fc dd=79 locks held 0 write locks 0 pid/thread >>>> 4845/140084544186112 flags 0 priority 100 >>>> fd dd=78 locks held 0 write locks 0 pid/thread >>>> 4845/140084535793408 flags 0 priority 100 >>>> fe dd=77 locks held 0 write locks 0 pid/thread >>>> 4845/140084586149632 flags 0 priority 100 >>>> ff dd=76 locks held 0 write locks 0 pid/thread >>>> 4845/140084569364224 flags 0 priority 100 >>>> 100 dd=75 locks held 0 write locks 0 pid/thread >>>> 4845/140084804294400 flags 0 priority 100 >>>> 101 dd=74 locks held 0 write locks 0 pid/thread >>>> 4845/140084703647488 flags 0 priority 100 >>>> 102 dd=73 locks held 0 write locks 0 pid/thread >>>> 4845/140084703647488 flags 0 priority 100 >>>> 103 dd=72 locks held 0 write locks 0 pid/thread >>>> 4845/140084821079808 flags 0 priority 100 >>>> 104 dd=71 locks held 0 write locks 0 pid/thread >>>> 4845/140084695254784 flags 0 priority 100 >>>> 105 dd=70 locks held 0 write locks 0 pid/thread >>>> 4845/140084544186112 flags 0 priority 100 >>>> 106 dd=69 locks held 0 write locks 0 pid/thread >>>> 4845/140084854650624 flags 0 priority 100 >>>> 107 dd=68 locks held 0 write locks 0 pid/thread >>>> 4845/140084628113152 flags 0 priority 100 >>>> 108 dd=67 locks held 0 write locks 0 pid/thread >>>> 4845/140084569364224 flags 0 priority 100 >>>> 109 dd=66 locks held 0 write locks 0 pid/thread >>>> 4845/140084594542336 flags 0 priority 100 >>>> 10a dd=65 locks held 0 write locks 0 pid/thread >>>> 4845/140084661683968 flags 0 priority 100 >>>> 10b dd=64 locks held 0 write locks 0 pid/thread >>>> 4845/140084653291264 flags 0 priority 100 >>>> 10c dd=63 locks held 0 write locks 0 pid/thread >>>> 4845/140084695254784 flags 0 priority 100 >>>> 10d dd=62 locks held 0 write locks 0 pid/thread >>>> 4845/140084720432896 flags 0 priority 100 >>>> 10e dd=61 locks held 0 write locks 0 pid/thread >>>> 4845/140084619720448 flags 0 priority 100 >>>> 10f dd=60 locks held 0 write locks 0 pid/thread >>>> 4845/140084720432896 flags 0 priority 100 >>>> 110 dd=59 locks held 0 write locks 0 pid/thread >>>> 4845/140084569364224 flags 0 priority 100 >>>> 111 dd=58 locks held 0 write locks 0 pid/thread >>>> 4845/140084695254784 flags 0 priority 100 >>>> 112 dd=57 locks held 0 write locks 0 pid/thread >>>> 4845/140084653291264 flags 0 priority 100 >>>> 113 dd=56 locks held 0 write locks 0 pid/thread >>>> 4845/140084712040192 flags 0 priority 100 >>>> 114 dd=55 locks held 0 write locks 0 pid/thread >>>> 4845/140084686862080 flags 0 priority 100 >>>> 115 dd=54 locks held 0 write locks 0 pid/thread >>>> 4845/140084577756928 flags 0 priority 100 >>>> 115 READ 1 WAIT userRoot/objectclass.db >>>> page 9 >>>> 116 dd=53 locks held 0 write locks 0 pid/thread >>>> 4845/140084837865216 flags 0 priority 100 >>>> 116 READ 1 WAIT userRoot/objectclass.db >>>> page 9 >>>> 117 dd=52 locks held 0 write locks 0 pid/thread >>>> 4845/140084577756928 flags 0 priority 100 >>>> 118 dd=51 locks held 0 write locks 0 pid/thread >>>> 4845/140084569364224 flags 0 priority 100 >>>> 119 dd=50 locks held 0 write locks 0 pid/thread >>>> 4845/140084560971520 flags 0 priority 100 >>>> 11a dd=49 locks held 1 write locks 0 pid/thread >>>> 4845/140084695254784 flags 10 priority 100 >>>> 11a READ 1 HELD ipaca/metaInfo.db >>>> handle 0 >>>> 11b dd=48 locks held 1 write locks 0 pid/thread >>>> 4845/140084695254784 flags 10 priority 100 >>>> 11b READ 1 HELD ipaca/notbefore.db >>>> handle 0 >>>> 11c dd=47 locks held 1 write locks 0 pid/thread >>>> 4845/140084695254784 flags 10 priority 100 >>>> 11c READ 1 HELD ipaca/notafter.db >>>> handle 0 >>>> 11d dd=46 locks held 1 write locks 0 pid/thread >>>> 4845/140084695254784 flags 10 priority 100 >>>> 11d READ 1 HELD ipaca/duration.db >>>> handle 0 >>>> 11e dd=45 locks held 1 write locks 0 pid/thread >>>> 4845/140084695254784 flags 10 priority 100 >>>> 11e READ 1 HELD ipaca/subjectname.db >>>> handle 0 >>>> 11f dd=44 locks held 1 write locks 0 pid/thread >>>> 4845/140084695254784 flags 10 priority 100 >>>> 11f READ 1 HELD ipaca/publicKeyData.db >>>> handle 0 >>>> 120 dd=43 locks held 1 write locks 0 pid/thread >>>> 4845/140084695254784 flags 10 priority 100 >>>> 120 READ 1 HELD ipaca/extension.db >>>> handle 0 >>>> 121 dd=42 locks held 1 write locks 0 pid/thread >>>> 4845/140084695254784 flags 10 priority 100 >>>> 121 READ 1 HELD ipaca/issuedby.db >>>> handle 0 >>>> 122 dd=41 locks held 0 write locks 0 pid/thread >>>> 4845/140084712040192 flags 0 priority 100 >>>> 123 dd=40 locks held 0 write locks 0 pid/thread >>>> 4845/140084812687104 flags 0 priority 100 >>>> 124 dd=39 locks held 0 write locks 0 pid/thread >>>> 4845/140084544186112 flags 0 priority 100 >>>> 125 dd=38 locks held 0 write locks 0 pid/thread >>>> 4845/140084686862080 flags 0 priority 100 >>>> 126 dd=37 locks held 0 write locks 0 pid/thread >>>> 4845/140084552578816 flags 0 priority 100 >>>> 128 dd=36 locks held 0 write locks 0 pid/thread >>>> 4845/140084686862080 flags 0 priority 100 >>>> 12a dd=35 locks held 0 write locks 0 pid/thread >>>> 4845/140084720432896 flags 0 priority 100 >>>> 12b dd=34 locks held 0 write locks 0 pid/thread >>>> 4845/140084552578816 flags 0 priority 100 >>>> 12c dd=33 locks held 0 write locks 0 pid/thread >>>> 4845/140084670076672 flags 0 priority 100 >>>> 12d dd=32 locks held 0 write locks 0 pid/thread >>>> 4845/140084577756928 flags 0 priority 100 >>>> 12e dd=31 locks held 0 write locks 0 pid/thread >>>> 4845/140084712040192 flags 0 priority 100 >>>> 12f dd=30 locks held 0 write locks 0 pid/thread >>>> 4845/140084670076672 flags 0 priority 100 >>>> 130 dd=29 locks held 0 write locks 0 pid/thread >>>> 4845/140084577756928 flags 0 priority 100 >>>> 131 dd=28 locks held 0 write locks 0 pid/thread >>>> 4845/140084560971520 flags 0 priority 100 >>>> 132 dd=27 locks held 0 write locks 0 pid/thread >>>> 4845/140084670076672 flags 0 priority 100 >>>> 133 dd=26 locks held 0 write locks 0 pid/thread >>>> 4845/140084602935040 flags 0 priority 100 >>>> 134 dd=25 locks held 0 write locks 0 pid/thread >>>> 4845/140084552578816 flags 0 priority 100 >>>> 135 dd=24 locks held 0 write locks 0 pid/thread >>>> 4845/140084678469376 flags 0 priority 100 >>>> 136 dd=23 locks held 0 write locks 0 pid/thread >>>> 4845/140084544186112 flags 0 priority 100 >>>> 137 dd=22 locks held 0 write locks 0 pid/thread >>>> 4845/140084577756928 flags 0 priority 100 >>>> 138 dd=21 locks held 0 write locks 0 pid/thread >>>> 4845/140084703647488 flags 0 priority 100 >>>> 139 dd=20 locks held 0 write locks 0 pid/thread >>>> 4845/140084678469376 flags 0 priority 100 >>>> 13b dd=19 locks held 0 write locks 0 pid/thread >>>> 4845/140084812687104 flags 0 priority 100 >>>> 144 dd=18 locks held 0 write locks 0 pid/thread >>>> 4845/140084670076672 flags 0 priority 100 >>>> 156 dd=17 locks held 0 write locks 0 pid/thread >>>> 4845/140084594542336 flags 0 priority 100 >>>> 15a dd=16 locks held 0 write locks 0 pid/thread >>>> 4845/140084577756928 flags 0 priority 100 >>>> 168 dd=15 locks held 0 write locks 0 pid/thread >>>> 4845/140084653291264 flags 0 priority 100 >>>> 16a dd=14 locks held 0 write locks 0 pid/thread >>>> 4845/140084829472512 flags 0 priority 100 >>>> 17d dd=13 locks held 0 write locks 0 pid/thread >>>> 4845/140084829472512 flags 0 priority 100 >>>> 17e dd=12 locks held 0 write locks 0 pid/thread >>>> 4845/140084829472512 flags 0 priority 100 >>>> 17f dd=11 locks held 0 write locks 0 pid/thread >>>> 4845/140084586149632 flags 0 priority 100 >>>> 180 dd=10 locks held 0 write locks 0 pid/thread >>>> 4845/140084594542336 flags 0 priority 100 >>>> 181 dd= 9 locks held 0 write locks 0 pid/thread >>>> 4845/140084577756928 flags 0 priority 100 >>>> 193 dd= 8 locks held 0 write locks 0 pid/thread >>>> 4845/140084812687104 flags 0 priority 100 >>>> 196 dd= 7 locks held 0 write locks 0 pid/thread >>>> 4845/140084695254784 flags 0 priority 100 >>>> 1a1 dd= 6 locks held 0 write locks 0 pid/thread >>>> 4845/140084720432896 flags 0 priority 100 >>>> 1a2 dd= 5 locks held 0 write locks 0 pid/thread >>>> 4845/140084577756928 flags 0 priority 100 >>>> 1a3 dd= 4 locks held 0 write locks 0 pid/thread >>>> 4845/140084812687104 flags 0 priority 100 >>>> 1a4 dd= 3 locks held 0 write locks 0 pid/thread >>>> 4845/140084812687104 flags 0 priority 100 >>>> 1a5 dd= 2 locks held 0 write locks 0 pid/thread >>>> 4845/140084577756928 flags 0 priority 100 >>>> 1a6 dd= 1 locks held 0 write locks 0 pid/thread >>>> 4845/140084594542336 flags 0 priority 100 >>>> 800b3a33 dd= 0 locks held 26 write locks 22 pid/thread >>>> 4845/140084661683968 flags 0 priority 100 >>>> 800b3a33 WRITE 1 HELD userRoot/objectclass.db >>>> page 9 >>>> 800b3a33 WRITE 1 HELD userRoot/objectclass.db >>>> page 8 >>>> 800b3a33 WRITE 1 HELD userRoot/objectclass.db page >>>> 18 >>>> 800b3a33 WRITE 1 HELD userRoot/objectclass.db >>>> page 7 >>>> 800b3a33 WRITE 1 HELD userRoot/objectclass.db page >>>> 14 >>>> 800b3a33 WRITE 1 HELD userRoot/objectclass.db page >>>> 11 >>>> 800b3a33 WRITE 2 HELD userRoot/objectclass.db >>>> page 3 >>>> 800b3a33 WRITE 1 HELD userRoot/cn.db page >>>> 150 >>>> 800b3a33 WRITE 1 HELD userRoot/cn.db page >>>> 89 >>>> 800b3a33 WRITE 1 HELD userRoot/cn.db page >>>> 53 >>>> 800b3a33 WRITE 1 HELD userRoot/cn.db page >>>> 131 >>>> 800b3a33 WRITE 1 HELD userRoot/cn.db page >>>> 117 >>>> 800b3a33 WRITE 1 HELD userRoot/cn.db page >>>> 11 >>>> 800b3a33 WRITE 2 HELD userRoot/cn.db page >>>> 141 >>>> 800b3a33 WRITE 1 HELD userRoot/cn.db page >>>> 41 >>>> 800b3a33 WRITE 1 HELD userRoot/cn.db page >>>> 40 >>>> 800b3a33 WRITE 1 HELD userRoot/cn.db page >>>> 143 >>>> 800b3a33 WRITE 1 HELD userRoot/cn.db page >>>> 30 >>>> 800b3a33 WRITE 2 HELD userRoot/id2entry.db page >>>> 8184 >>>> 800b3a33 WRITE 1 HELD userRoot/id2entry.db page >>>> 8183 >>>> 800b3a33 WRITE 1 HELD userRoot/id2entry.db page >>>> 7779 >>>> 800b3a33 WRITE 1 HELD userRoot/id2entry.db >>>> page 0 >>>> 800b3a33 READ 3 HELD userRoot/entryrdn.db >>>> page 3 >>>> 800b3a33 READ 1 HELD userRoot/entryrdn.db page >>>> 36 >>>> 800b3a33 READ 1 HELD userRoot/entryrdn.db >>>> page 4 >>>> 800b3a33 READ 1 HELD userRoot/nsuniqueid.db page >>>> 86 >>>> =-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-= >>>> Locks grouped by object: >>>> Locker Mode Count Status ----------------- Object >>>> --------------- >>>> a4 READ 1 HELD userRoot/ipasudorunas.db >>>> handle 0 >>>> >>>> 9e READ 1 HELD userRoot/ipaMemberCertProfile.db >>>> handle 0 >>>> >>>> 10 READ 1 HELD ipaca/vlv#allrevokedcertspkitomcatindex.db >>>> handle 0 >>>> >>>> 800b3a33 WRITE 2 HELD userRoot/id2entry.db page >>>> 8184 >>>> >>>> 800b3a33 WRITE 1 HELD userRoot/id2entry.db page >>>> 8183 >>>> >>>> 800b3a33 WRITE 1 HELD userRoot/id2entry.db page >>>> 7779 >>>> >>>> a1 READ 1 HELD userRoot/memberdenycmd.db >>>> handle 0 >>>> >>>> a READ 1 HELD ipaca/vlv#allcertspkitomcatindex.db >>>> handle 0 >>>> >>>> a7 READ 1 HELD userRoot/sourcehost.db >>>> handle 0 >>>> >>>> 57 READ 1 HELD changelog/ancestorid.db >>>> handle 0 >>>> >>>> e5 READ 1 HELD ipaca/revInfo.db >>>> handle 0 >>>> >>>> e6 READ 1 HELD ipaca/revokedby.db >>>> handle 0 >>>> >>>> 11 READ 1 HELD ipaca/vlv#allrevokedcertsnotafterpkitomcatindex.db >>>> handle 0 >>>> >>>> 4e READ 1 HELD userRoot/parentid.db >>>> handle 0 >>>> >>>> 4 READ 1 HELD ipaca/id2entry.db >>>> handle 0 >>>> >>>> 58 READ 1 HELD changelog/numsubordinates.db >>>> handle 0 >>>> >>>> 67 READ 1 HELD ipaca/seeAlso.db >>>> handle 0 >>>> >>>> 2b READ 1 HELD ipaca/vlv#carevocationpkitomcatindex.db >>>> handle 0 >>>> >>>> ec READ 1 HELD ipaca/requeststate.db >>>> handle 0 >>>> >>>> 11b READ 1 HELD ipaca/notbefore.db >>>> handle 0 >>>> >>>> 7e READ 1 HELD userRoot/numsubordinates.db >>>> handle 0 >>>> >>>> a2 READ 1 HELD userRoot/memberallowcmd.db >>>> handle 0 >>>> >>>> 9c READ 1 HELD userRoot/owner.db >>>> handle 0 >>>> >>>> 1e READ 1 HELD ipaca/vlv#cacompleteenrollmentpkitomcatindex.db >>>> handle 0 >>>> >>>> 11c READ 1 HELD ipaca/notafter.db >>>> handle 0 >>>> >>>> 40 READ 1 HELD changelog/objectclass.db >>>> handle 0 >>>> >>>> 2e READ 1 HELD changelog/entryusn.db >>>> handle 0 >>>> >>>> 16 READ 1 HELD ipaca/vlv#allvalidcertsnotafterpkitomcatindex.db >>>> handle 0 >>>> >>>> 15 READ 1 HELD ipaca/vlv#allvalidcertspkitomcatindex.db >>>> handle 0 >>>> >>>> a5 READ 1 HELD userRoot/ipaMemberCa.db >>>> handle 0 >>>> >>>> 18 READ 1 HELD ipaca/vlv#caallpkitomcatindex.db >>>> handle 0 >>>> >>>> a0 READ 1 HELD userRoot/secretary.db >>>> handle 0 >>>> >>>> 81 READ 1 HELD userRoot/memberUser.db >>>> handle 0 >>>> >>>> 9f READ 1 HELD userRoot/ipaassignedidview.db >>>> handle 0 >>>> >>>> a3 READ 1 HELD userRoot/manager.db >>>> handle 0 >>>> >>>> d8 READ 1 HELD ipaca/description.db >>>> handle 0 >>>> >>>> a8 READ 1 HELD userRoot/memberservice.db >>>> handle 0 >>>> >>>> 44 READ 1 HELD ipaca/objectclass.db >>>> handle 0 >>>> >>>> ed READ 1 HELD ipaca/dateOfCreate.db >>>> handle 0 >>>> >>>> 800b3a33 WRITE 1 HELD userRoot/id2entry.db >>>> page 0 >>>> >>>> 2 READ 1 HELD userRoot/id2entry.db >>>> handle 0 >>>> >>>> 5b READ 1 HELD userRoot/krbPrincipalName.db >>>> handle 0 >>>> >>>> ee READ 1 HELD ipaca/requesttype.db >>>> handle 0 >>>> >>>> c8 READ 1 HELD ipaca/certstatus.db >>>> handle 0 >>>> >>>> 56 READ 1 HELD changelog/parentid.db >>>> handle 0 >>>> >>>> 83 READ 1 HELD userRoot/memberHost.db >>>> handle 0 >>>> >>>> 5e READ 1 HELD userRoot/ipakrbprincipalalias.db >>>> handle 0 >>>> >>>> 121 READ 1 HELD ipaca/issuedby.db >>>> handle 0 >>>> >>>> 6 READ 1 HELD ipaca/entryrdn.db >>>> handle 0 >>>> >>>> e7 READ 1 HELD ipaca/revokedOn.db >>>> handle 0 >>>> >>>> 4c READ 1 HELD ipaca/aci.db >>>> handle 0 >>>> >>>> 11d READ 1 HELD ipaca/duration.db >>>> handle 0 >>>> >>>> 14 READ 1 HELD ipaca/vlv#allrevokedorrevokedexpiredcertspkitomcatindex.db >>>> handle 0 >>>> >>>> 21 READ 1 HELD ipaca/vlv#caenrollmentpkitomcatindex.db >>>> handle 0 >>>> >>>> 55 READ 1 HELD changelog/targetuniqueid.db >>>> handle 0 >>>> >>>> 800b3a33 READ 1 HELD userRoot/entryrdn.db page >>>> 36 >>>> >>>> 2c READ 1 HELD changelog/id2entry.db >>>> handle 0 >>>> >>>> 800b3a33 READ 1 HELD userRoot/entryrdn.db >>>> page 4 >>>> >>>> 34 READ 1 HELD userRoot/entryrdn.db >>>> handle 0 >>>> >>>> 800b3a33 READ 3 HELD userRoot/entryrdn.db >>>> page 3 >>>> >>>> eb READ 1 HELD ipaca/requestid.db >>>> handle 0 >>>> >>>> 49 READ 1 HELD userRoot/aci.db >>>> handle 0 >>>> >>>> 76 READ 1 HELD userRoot/ipauniqueid.db >>>> handle 0 >>>> >>>> 63 READ 1 HELD changelog/seeAlso.db >>>> handle 0 >>>> >>>> 7c READ 1 HELD userRoot/fqdn.db >>>> handle 0 >>>> >>>> db READ 1 HELD ipaca/cn.db >>>> handle 0 >>>> >>>> ef READ 1 HELD ipaca/numsubordinates.db >>>> handle 0 >>>> >>>> 32 READ 1 HELD ipaca/entryusn.db >>>> handle 0 >>>> >>>> 1d READ 1 HELD ipaca/vlv#cacompletepkitomcatindex.db >>>> handle 0 >>>> >>>> dd READ 1 HELD ipaca/ancestorid.db >>>> handle 0 >>>> >>>> a9 READ 1 HELD userRoot/ipatokenradiusconfiglink.db >>>> handle 0 >>>> >>>> f0 READ 1 HELD userRoot/userCertificate.db >>>> handle 0 >>>> >>>> 11a READ 1 HELD ipaca/metaInfo.db >>>> handle 0 >>>> >>>> 7d READ 1 HELD userRoot/managedby.db >>>> handle 0 >>>> >>>> ea READ 1 HELD ipaca/nsuniqueid.db >>>> handle 0 >>>> >>>> c5 READ 1 HELD userRoot/uidnumber.db >>>> handle 0 >>>> >>>> d1 READ 1 HELD userRoot/nscpEntryDN.db >>>> handle 0 >>>> >>>> 120 READ 1 HELD ipaca/extension.db >>>> handle 0 >>>> >>>> 800b3a33 WRITE 1 HELD userRoot/cn.db page >>>> 150 >>>> >>>> 69 READ 1 HELD ipaca/parentid.db >>>> handle 0 >>>> >>>> 800b3a33 WRITE 1 HELD userRoot/cn.db page >>>> 131 >>>> >>>> 800b3a33 WRITE 1 HELD userRoot/cn.db page >>>> 143 >>>> >>>> 800b3a33 WRITE 2 HELD userRoot/cn.db page >>>> 141 >>>> >>>> 20 READ 1 HELD ipaca/vlv#cacompleterevocationpkitomcatindex.db >>>> handle 0 >>>> >>>> 8e READ 1 HELD userRoot/macAddress.db >>>> handle 0 >>>> >>>> e READ 1 HELD ipaca/vlv#allnonrevokedcertspkitomcatindex.db >>>> handle 0 >>>> >>>> 17 READ 1 HELD ipaca/vlv#allvalidorrevokedcertspkitomcatindex.db >>>> handle 0 >>>> >>>> 800b3a33 WRITE 1 HELD userRoot/cn.db page >>>> 30 >>>> >>>> 800b3a33 WRITE 1 HELD userRoot/objectclass.db page >>>> 14 >>>> >>>> 800b3a33 WRITE 1 HELD userRoot/objectclass.db >>>> page 9 >>>> 116 READ 1 WAIT userRoot/objectclass.db >>>> page 9 >>>> 115 READ 1 WAIT userRoot/objectclass.db >>>> page 9 >>>> 92 READ 1 WAIT userRoot/objectclass.db >>>> page 9 >>>> >>>> 800b3a33 WRITE 1 HELD userRoot/objectclass.db >>>> page 8 >>>> >>>> 7b READ 1 HELD userRoot/cn.db >>>> handle 0 >>>> >>>> 800b3a33 WRITE 1 HELD userRoot/objectclass.db page >>>> 11 >>>> >>>> 800b3a33 WRITE 1 HELD userRoot/objectclass.db >>>> page 7 >>>> >>>> 800b3a33 WRITE 1 HELD userRoot/cn.db page >>>> 11 >>>> >>>> 38 READ 1 HELD userRoot/objectclass.db >>>> handle 0 >>>> >>>> 800b3a33 WRITE 2 HELD userRoot/objectclass.db >>>> page 3 >>>> >>>> 800b3a33 WRITE 1 HELD userRoot/cn.db page >>>> 53 >>>> >>>> 800b3a33 WRITE 1 HELD userRoot/objectclass.db page >>>> 18 >>>> 6e READ 1 WAIT userRoot/objectclass.db page >>>> 18 >>>> 94 READ 1 WAIT userRoot/objectclass.db page >>>> 18 >>>> >>>> 800b3a33 WRITE 1 HELD userRoot/cn.db page >>>> 41 >>>> >>>> 800b3a33 WRITE 1 HELD userRoot/cn.db page >>>> 40 >>>> >>>> 800b3a33 WRITE 1 HELD userRoot/cn.db page >>>> 89 >>>> >>>> 800b3a33 WRITE 1 HELD userRoot/cn.db page >>>> 117 >>>> >>>> 30 READ 1 HELD userRoot/entryusn.db >>>> handle 0 >>>> >>>> 3c READ 1 HELD changelog/entryrdn.db >>>> handle 0 >>>> >>>> 97 READ 1 HELD userRoot/uid.db >>>> handle 0 >>>> >>>> 47 READ 1 HELD changelog/aci.db >>>> handle 0 >>>> >>>> e3 READ 1 HELD ipaca/serialno.db >>>> handle 0 >>>> >>>> 7a READ 1 HELD userRoot/nsuniqueid.db >>>> handle 0 >>>> >>>> 800b3a33 READ 1 HELD userRoot/nsuniqueid.db page >>>> 86 >>>> >>>> 51 READ 1 HELD changelog/nsuniqueid.db >>>> handle 0 >>>> >>>> 11e READ 1 HELD ipaca/subjectname.db >>>> handle 0 >>>> >>>> 7f READ 1 HELD userRoot/member.db >>>> handle 0 >>>> >>>> 11f READ 1 HELD ipaca/publicKeyData.db >>>> handle 0 >>>> >>>> 9b READ 1 HELD userRoot/uniquemember.db >>>> handle 0 >>>> >>>> 52 READ 1 HELD changelog/changenumber.db >>>> handle 0 >>>> >>>> ce READ 1 HELD userRoot/gidnumber.db >>>> handle 0 >>>> >>>> 65 READ 1 HELD userRoot/seeAlso.db >>>> handle 0 >>>> >>>> 9d READ 1 HELD userRoot/ipaallowedtarget.db >>>> handle 0 >>>> >>>> a6 READ 1 HELD userRoot/ipasudorunasgroup.db >>>> handle 0 >>>> >>>> 3a READ 1 HELD userRoot/ancestorid.db >>>> handle 0 >>>> >>>> >>>> >>>> >>>> pstack output >>>> >>>> root at ipa-master ]# pstack 4845 >>>> Thread 44 (Thread 0x7f681dd3a700 (LWP 4847)): >>>> #0 0x00007f683cd719b3 in select () from /lib64/libc.so.6 >>>> #1 0x00007f683f4fc0e9 in DS_Sleep () from >>>> /usr/lib64/dirsrv/libslapd.so.0 >>>> #2 0x00007f6832b4a907 in deadlock_threadmain () from >>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>> #3 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so >>>> #4 0x00007f683d04cdc5 in start_thread () from /lib64/libpthread.so.0 >>>> #5 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 >>>> Thread 43 (Thread 0x7f681d539700 (LWP 4848)): >>>> #0 0x00007f683cd719b3 in select () from /lib64/libc.so.6 >>>> #1 0x00007f68376d8bad in __os_yield () from /lib64/libdb-5.3.so >>>> #2 0x00007f68376d42b3 in __memp_sync_int () from /lib64/libdb-5.3.so >>>> #3 0x00007f68376e4752 in __txn_checkpoint () from /lib64/libdb-5.3.so >>>> #4 0x00007f68376e4b74 in __txn_checkpoint_pp () from /lib64/ >>>> libdb-5.3.so >>>> #5 0x00007f6832b4ea87 in checkpoint_threadmain () from >>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>> #6 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so >>>> #7 0x00007f683d04cdc5 in start_thread () from /lib64/libpthread.so.0 >>>> #8 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 >>>> Thread 42 (Thread 0x7f681cd38700 (LWP 4849)): >>>> #0 0x00007f683cd719b3 in select () from /lib64/libc.so.6 >>>> #1 0x00007f683f4fc0e9 in DS_Sleep () from >>>> /usr/lib64/dirsrv/libslapd.so.0 >>>> #2 0x00007f6832b4ab7f in trickle_threadmain () from >>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>> #3 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so >>>> #4 0x00007f683d04cdc5 in start_thread () from /lib64/libpthread.so.0 >>>> #5 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 >>>> Thread 41 (Thread 0x7f681c537700 (LWP 4850)): >>>> #0 0x00007f683cd719b3 in select () from /lib64/libc.so.6 >>>> #1 0x00007f683f4fc0e9 in DS_Sleep () from >>>> /usr/lib64/dirsrv/libslapd.so.0 >>>> #2 0x00007f6832b45707 in perf_threadmain () from >>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>> #3 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so >>>> #4 0x00007f683d04cdc5 in start_thread () from /lib64/libpthread.so.0 >>>> #5 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 >>>> Thread 40 (Thread 0x7f681bb2d700 (LWP 4851)): >>>> #0 0x00007f683d0506d5 in >>>> pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 >>>> #1 0x00007f683d6a6050 in PR_WaitCondVar () from /lib64/libnspr4.so >>>> #2 0x00007f683f4eb198 in slapi_wait_condvar () from >>>> /usr/lib64/dirsrv/libslapd.so.0 >>>> #3 0x00007f683587d62e in cos_cache_wait_on_change () from >>>> /usr/lib64/dirsrv/plugins/libcos-plugin.so >>>> #4 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so >>>> #5 0x00007f683d04cdc5 in start_thread () from /lib64/libpthread.so.0 >>>> #6 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 >>>> Thread 39 (Thread 0x7f681b32c700 (LWP 4852)): >>>> #0 0x00007f683d0506d5 in >>>> pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 >>>> #1 0x00007f683d6a6050 in PR_WaitCondVar () from /lib64/libnspr4.so >>>> #2 0x00007f683f4eb198 in slapi_wait_condvar () from >>>> /usr/lib64/dirsrv/libslapd.so.0 >>>> #3 0x00007f68311e8eed in roles_cache_wait_on_change () from >>>> /usr/lib64/dirsrv/plugins/libroles-plugin.so >>>> #4 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so >>>> #5 0x00007f683d04cdc5 in start_thread () from /lib64/libpthread.so.0 >>>> #6 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 >>>> Thread 38 (Thread 0x7f681ab2b700 (LWP 4853)): >>>> #0 0x00007f683d0506d5 in >>>> pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 >>>> #1 0x00007f683d6a6050 in PR_WaitCondVar () from /lib64/libnspr4.so >>>> #2 0x00007f683f4eb198 in slapi_wait_condvar () from >>>> /usr/lib64/dirsrv/libslapd.so.0 >>>> #3 0x00007f68311e8eed in roles_cache_wait_on_change () from >>>> /usr/lib64/dirsrv/plugins/libroles-plugin.so >>>> #4 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so >>>> #5 0x00007f683d04cdc5 in start_thread () from /lib64/libpthread.so.0 >>>> #6 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 >>>> Thread 37 (Thread 0x7f681a32a700 (LWP 4854)): >>>> #0 0x00007f683d0506d5 in >>>> pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 >>>> #1 0x00007f683d6a6050 in PR_WaitCondVar () from /lib64/libnspr4.so >>>> #2 0x00007f683f4eb198 in slapi_wait_condvar () from >>>> /usr/lib64/dirsrv/libslapd.so.0 >>>> #3 0x00007f68311e8eed in roles_cache_wait_on_change () from >>>> /usr/lib64/dirsrv/plugins/libroles-plugin.so >>>> #4 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so >>>> #5 0x00007f683d04cdc5 in start_thread () from /lib64/libpthread.so.0 >>>> #6 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 >>>> Thread 36 (Thread 0x7f6819b29700 (LWP 4855)): >>>> #0 0x00007f683d050a82 in pthread_cond_timedwait@@GLIBC_2.3.2 () from >>>> /lib64/libpthread.so.0 >>>> #1 0x00007f683d6a5b07 in pt_TimedWait () from /lib64/libnspr4.so >>>> #2 0x00007f683d6a5fce in PR_WaitCondVar () from /lib64/libnspr4.so >>>> #3 0x00007f683f9a4d03 in housecleaning () >>>> #4 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so >>>> #5 0x00007f683d04cdc5 in start_thread () from /lib64/libpthread.so.0 >>>> #6 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 >>>> Thread 35 (Thread 0x7f6819328700 (LWP 4856)): >>>> #0 0x00007f683d050a82 in pthread_cond_timedwait@@GLIBC_2.3.2 () from >>>> /lib64/libpthread.so.0 >>>> #1 0x00007f683d6a5b07 in pt_TimedWait () from /lib64/libnspr4.so >>>> #2 0x00007f683d6a5fce in PR_WaitCondVar () from /lib64/libnspr4.so >>>> #3 0x00007f683f4911e8 in eq_loop () from /usr/lib64/dirsrv/libslapd.so. >>>> 0 >>>> #4 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so >>>> #5 0x00007f683d04cdc5 in start_thread () from /lib64/libpthread.so.0 >>>> #6 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 >>>> Thread 34 (Thread 0x7f680bfff700 (LWP 4859)): >>>> #0 0x00007f683d0506d5 in >>>> pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 >>>> #1 0x00007f683d6a6050 in PR_WaitCondVar () from /lib64/libnspr4.so >>>> #2 0x00007f683f999fee in connection_wait_for_new_work () >>>> #3 0x00007f683f99b719 in connection_threadmain () >>>> #4 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so >>>> #5 0x00007f683d04cdc5 in start_thread () from /lib64/libpthread.so.0 >>>> #6 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 >>>> Thread 33 (Thread 0x7f680b7fe700 (LWP 4860)): >>>> #0 0x00007f683d0506d5 in >>>> pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 >>>> #1 0x00007f683d6a6050 in PR_WaitCondVar () from /lib64/libnspr4.so >>>> #2 0x00007f683f999fee in connection_wait_for_new_work () >>>> #3 0x00007f683f99b719 in connection_threadmain () >>>> #4 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so >>>> #5 0x00007f683d04cdc5 in start_thread () from /lib64/libpthread.so.0 >>>> #6 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 >>>> Thread 32 (Thread 0x7f680affd700 (LWP 4861)): >>>> #0 0x00007f683d0506d5 in >>>> pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 >>>> #1 0x00007f68375ac2f3 in __db_hybrid_mutex_suspend () from /lib64/ >>>> libdb-5.3.so >>>> #2 0x00007f68375ab640 in __db_tas_mutex_lock () from /lib64/ >>>> libdb-5.3.so >>>> #3 0x00007f6837655d3a in __lock_get_internal () from /lib64/ >>>> libdb-5.3.so >>>> #4 0x00007f6837656820 in __lock_get () from /lib64/libdb-5.3.so >>>> #5 0x00007f6837682142 in __db_lget () from /lib64/libdb-5.3.so >>>> #6 0x00007f68375c9605 in __bam_search () from /lib64/libdb-5.3.so >>>> #7 0x00007f68375b4256 in __bamc_search () from /lib64/libdb-5.3.so >>>> #8 0x00007f68375b5d0f in __bamc_get () from /lib64/libdb-5.3.so >>>> #9 0x00007f683766eca6 in __dbc_iget () from /lib64/libdb-5.3.so >>>> #10 0x00007f683767db02 in __dbc_get_pp () from /lib64/libdb-5.3.so >>>> #11 0x00007f6832b59680 in idl_new_fetch () from >>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>> #12 0x00007f6832b67b26 in index_read_ext_allids () from >>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>> #13 0x00007f6832b52284 in keys2idl () from >>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>> #14 0x00007f6832b529e3 in ava_candidates.isra.0 () from >>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>> #15 0x00007f6832b52fd2 in filter_candidates_ext () from >>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>> #16 0x00007f6832b54096 in list_candidates () from >>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>> #17 0x00007f6832b52f40 in filter_candidates_ext () from >>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>> #18 0x00007f6832b54096 in list_candidates () from >>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>> #19 0x00007f6832b52f40 in filter_candidates_ext () from >>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>> #20 0x00007f6832b54096 in list_candidates () from >>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>> #21 0x00007f6832b52f40 in filter_candidates_ext () from >>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>> #22 0x00007f6832b8e80c in subtree_candidates () from >>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>> #23 0x00007f6832b8fe7f in ldbm_back_search () from >>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>> #24 0x00007f683f4bd1d6 in op_shared_search () from >>>> /usr/lib64/dirsrv/libslapd.so.0 >>>> #25 0x00007f683f9ae33e in do_search () >>>> #26 0x00007f683f99c3c3 in connection_threadmain () >>>> #27 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so >>>> #28 0x00007f683d04cdc5 in start_thread () from /lib64/libpthread.so.0 >>>> #29 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 >>>> Thread 31 (Thread 0x7f680a7fc700 (LWP 4862)): >>>> #0 0x00007f683d0506d5 in >>>> pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 >>>> #1 0x00007f683d6a62c3 in PR_EnterMonitor () from /lib64/libnspr4.so >>>> #2 0x00007f6832b4d3e6 in dblayer_txn_begin () from >>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>> #3 0x00007f6832b89007 in ldbm_back_modify () from >>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>> #4 0x00007f683f4b6541 in op_shared_modify () from >>>> /usr/lib64/dirsrv/libslapd.so.0 >>>> #5 0x00007f683f4b7004 in modify_internal_pb () from >>>> /usr/lib64/dirsrv/libslapd.so.0 >>>> #6 0x00007f683483bbbb in ipalockout_postop () from >>>> /usr/lib64/dirsrv/plugins/libipa_lockout.so >>>> #7 0x00007f683f4c8848 in plugin_call_func () from >>>> /usr/lib64/dirsrv/libslapd.so.0 >>>> #8 0x00007f683f4c8b13 in plugin_call_plugins () from >>>> /usr/lib64/dirsrv/libslapd.so.0 >>>> #9 0x00007f683f9954d8 in do_bind () >>>> #10 0x00007f683f99c3fd in connection_threadmain () >>>> #11 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so >>>> #12 0x00007f683d04cdc5 in start_thread () from /lib64/libpthread.so.0 >>>> #13 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 >>>> Thread 30 (Thread 0x7f6809ffb700 (LWP 4863)): >>>> #0 0x00007f683d0506d5 in >>>> pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 >>>> #1 0x00007f68375ac2f3 in __db_hybrid_mutex_suspend () from /lib64/ >>>> libdb-5.3.so >>>> #2 0x00007f68375ab640 in __db_tas_mutex_lock () from /lib64/ >>>> libdb-5.3.so >>>> #3 0x00007f6837655d3a in __lock_get_internal () from /lib64/ >>>> libdb-5.3.so >>>> #4 0x00007f6837656820 in __lock_get () from /lib64/libdb-5.3.so >>>> #5 0x00007f6837682142 in __db_lget () from /lib64/libdb-5.3.so >>>> #6 0x00007f68375c9605 in __bam_search () from /lib64/libdb-5.3.so >>>> #7 0x00007f68375b4256 in __bamc_search () from /lib64/libdb-5.3.so >>>> #8 0x00007f68375b5d0f in __bamc_get () from /lib64/libdb-5.3.so >>>> #9 0x00007f683766eca6 in __dbc_iget () from /lib64/libdb-5.3.so >>>> #10 0x00007f683767db02 in __dbc_get_pp () from /lib64/libdb-5.3.so >>>> #11 0x00007f6832b59680 in idl_new_fetch () from >>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>> #12 0x00007f6832b67b26 in index_read_ext_allids () from >>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>> #13 0x00007f6832b52284 in keys2idl () from >>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>> #14 0x00007f6832b529e3 in ava_candidates.isra.0 () from >>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>> #15 0x00007f6832b52fd2 in filter_candidates_ext () from >>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>> #16 0x00007f6832b54096 in list_candidates () from >>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>> #17 0x00007f6832b52f40 in filter_candidates_ext () from >>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>> #18 0x00007f6832b8e80c in subtree_candidates () from >>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>> #19 0x00007f6832b8fe7f in ldbm_back_search () from >>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>> #20 0x00007f683f4bd1d6 in op_shared_search () from >>>> /usr/lib64/dirsrv/libslapd.so.0 >>>> #21 0x00007f683f4cd94e in search_internal_callback_pb () from >>>> /usr/lib64/dirsrv/libslapd.so.0 >>>> #22 0x00007f683f4cdbe8 in search_internal_pb () from >>>> /usr/lib64/dirsrv/libslapd.so.0 >>>> #23 0x00007f683f9ac49b in ids_sasl_canon_user () >>>> #24 0x00007f683d46c93b in _sasl_canon_user () from /lib64/libsasl2.so.3 >>>> #25 0x00007f683d46cc4c in _sasl_canon_user_lookup () from >>>> /lib64/libsasl2.so.3 >>>> #26 0x00007f6837374fb9 in gssapi_server_mech_step () from >>>> /usr/lib64/sasl2/libgssapiv2.so >>>> #27 0x00007f683d478b9b in sasl_server_step () from /lib64/libsasl2.so.3 >>>> #28 0x00007f683f9ad83b in ids_sasl_check_bind () >>>> #29 0x00007f683f99561a in do_bind () >>>> #30 0x00007f683f99c3fd in connection_threadmain () >>>> #31 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so >>>> #32 0x00007f683d04cdc5 in start_thread () from /lib64/libpthread.so.0 >>>> #33 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 >>>> Thread 29 (Thread 0x7f68097fa700 (LWP 4864)): >>>> #0 0x00007f683d0506d5 in >>>> pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 >>>> #1 0x00007f683d6a6050 in PR_WaitCondVar () from /lib64/libnspr4.so >>>> #2 0x00007f683f999fee in connection_wait_for_new_work () >>>> #3 0x00007f683f99b719 in connection_threadmain () >>>> #4 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so >>>> #5 0x00007f683d04cdc5 in start_thread () from /lib64/libpthread.so.0 >>>> #6 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 >>>> Thread 28 (Thread 0x7f6808ff9700 (LWP 4865)): >>>> #0 0x00007f683d0506d5 in >>>> pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 >>>> #1 0x00007f683d6a6050 in PR_WaitCondVar () from /lib64/libnspr4.so >>>> #2 0x00007f683f999fee in connection_wait_for_new_work () >>>> #3 0x00007f683f99b719 in connection_threadmain () >>>> #4 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so >>>> #5 0x00007f683d04cdc5 in start_thread () from /lib64/libpthread.so.0 >>>> #6 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 >>>> Thread 27 (Thread 0x7f6803fff700 (LWP 4866)): >>>> #0 0x00007f683d0506d5 in >>>> pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 >>>> #1 0x00007f683d6a6050 in PR_WaitCondVar () from /lib64/libnspr4.so >>>> #2 0x00007f683f999fee in connection_wait_for_new_work () >>>> #3 0x00007f683f99b719 in connection_threadmain () >>>> #4 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so >>>> #5 0x00007f683d04cdc5 in start_thread () from /lib64/libpthread.so.0 >>>> #6 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 >>>> Thread 26 (Thread 0x7f68037fe700 (LWP 4867)): >>>> #0 0x00007f683d0506d5 in >>>> pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 >>>> #1 0x00007f683d6a6050 in PR_WaitCondVar () from /lib64/libnspr4.so >>>> #2 0x00007f683f999fee in connection_wait_for_new_work () >>>> #3 0x00007f683f99b719 in connection_threadmain () >>>> #4 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so >>>> #5 0x00007f683d04cdc5 in start_thread () from /lib64/libpthread.so.0 >>>> #6 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 >>>> Thread 25 (Thread 0x7f6802ffd700 (LWP 4868)): >>>> #0 0x00007f683d0506d5 in >>>> pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 >>>> #1 0x00007f683d6a6050 in PR_WaitCondVar () from /lib64/libnspr4.so >>>> #2 0x00007f683f999fee in connection_wait_for_new_work () >>>> #3 0x00007f683f99b719 in connection_threadmain () >>>> #4 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so >>>> #5 0x00007f683d04cdc5 in start_thread () from /lib64/libpthread.so.0 >>>> #6 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 >>>> Thread 24 (Thread 0x7f68027fc700 (LWP 4869)): >>>> #0 0x00007f683d0506d5 in >>>> pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 >>>> #1 0x00007f68375ac2f3 in __db_hybrid_mutex_suspend () from /lib64/ >>>> libdb-5.3.so >>>> #2 0x00007f68375abb37 in __db_tas_mutex_readlock () from /lib64/ >>>> libdb-5.3.so >>>> #3 0x00007f68376c5d35 in __memp_fget () from /lib64/libdb-5.3.so >>>> #4 0x00007f683766f333 in __dbc_iget () from /lib64/libdb-5.3.so >>>> #5 0x00007f683767db02 in __dbc_get_pp () from /lib64/libdb-5.3.so >>>> #6 0x00007f6832b59822 in idl_new_fetch () from >>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>> #7 0x00007f6832b67b26 in index_read_ext_allids () from >>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>> #8 0x00007f6832b52284 in keys2idl () from >>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>> #9 0x00007f6832b529e3 in ava_candidates.isra.0 () from >>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>> #10 0x00007f6832b52fd2 in filter_candidates_ext () from >>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>> #11 0x00007f6832b54096 in list_candidates () from >>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>> #12 0x00007f6832b52f40 in filter_candidates_ext () from >>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>> #13 0x00007f6832b54096 in list_candidates () from >>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>> #14 0x00007f6832b52f40 in filter_candidates_ext () from >>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>> #15 0x00007f6832b54096 in list_candidates () from >>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>> #16 0x00007f6832b52f40 in filter_candidates_ext () from >>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>> #17 0x00007f6832b8e80c in subtree_candidates () from >>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>> #18 0x00007f6832b8fe7f in ldbm_back_search () from >>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>> #19 0x00007f683f4bd1d6 in op_shared_search () from >>>> /usr/lib64/dirsrv/libslapd.so.0 >>>> #20 0x00007f683f9ae33e in do_search () >>>> #21 0x00007f683f99c3c3 in connection_threadmain () >>>> #22 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so >>>> #23 0x00007f683d04cdc5 in start_thread () from /lib64/libpthread.so.0 >>>> #24 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 >>>> Thread 23 (Thread 0x7f6801ffb700 (LWP 4870)): >>>> #0 0x00007f683d0506d5 in >>>> pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 >>>> #1 0x00007f683d6a6050 in PR_WaitCondVar () from /lib64/libnspr4.so >>>> #2 0x00007f683f999fee in connection_wait_for_new_work () >>>> #3 0x00007f683f99b719 in connection_threadmain () >>>> #4 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so >>>> #5 0x00007f683d04cdc5 in start_thread () from /lib64/libpthread.so.0 >>>> #6 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 >>>> Thread 22 (Thread 0x7f68017fa700 (LWP 4871)): >>>> #0 0x00007f683d0506d5 in >>>> pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 >>>> #1 0x00007f683d6a6050 in PR_WaitCondVar () from /lib64/libnspr4.so >>>> #2 0x00007f683f999fee in connection_wait_for_new_work () >>>> #3 0x00007f683f99b719 in connection_threadmain () >>>> #4 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so >>>> #5 0x00007f683d04cdc5 in start_thread () from /lib64/libpthread.so.0 >>>> #6 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 >>>> Thread 21 (Thread 0x7f6800ff9700 (LWP 4872)): >>>> #0 0x00007f683d0506d5 in >>>> pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 >>>> #1 0x00007f683d6a6050 in PR_WaitCondVar () from /lib64/libnspr4.so >>>> #2 0x00007f683f999fee in connection_wait_for_new_work () >>>> #3 0x00007f683f99b719 in connection_threadmain () >>>> #4 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so >>>> #5 0x00007f683d04cdc5 in start_thread () from /lib64/libpthread.so.0 >>>> #6 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 >>>> Thread 20 (Thread 0x7f68007f8700 (LWP 4873)): >>>> #0 0x00007f683d0506d5 in >>>> pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 >>>> #1 0x00007f68375ac2f3 in __db_hybrid_mutex_suspend () from /lib64/ >>>> libdb-5.3.so >>>> #2 0x00007f68375ab640 in __db_tas_mutex_lock () from /lib64/ >>>> libdb-5.3.so >>>> #3 0x00007f68376c5e1b in __memp_fget () from /lib64/libdb-5.3.so >>>> #4 0x00007f68375c9211 in __bam_search () from /lib64/libdb-5.3.so >>>> #5 0x00007f68375b4256 in __bamc_search () from /lib64/libdb-5.3.so >>>> #6 0x00007f68375b86c4 in __bamc_put () from /lib64/libdb-5.3.so >>>> #7 0x00007f683766f6d5 in __dbc_iput () from /lib64/libdb-5.3.so >>>> #8 0x00007f683766aa0e in __db_put () from /lib64/libdb-5.3.so >>>> #9 0x00007f683767ffa4 in __db_put_pp () from /lib64/libdb-5.3.so >>>> #10 0x00007f6832b5a4cf in idl_new_insert_key () from >>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>> #11 0x00007f6832b695b5 in addordel_values_sv.isra.3 () from >>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>> #12 0x00007f6832b6a25a in index_addordel_values_ext_sv () from >>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>> #13 0x00007f6832b6a694 in index_addordel_values_sv () from >>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>> #14 0x00007f6832b6a75a in index_addordel_entry () from >>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>> #15 0x00007f6832b6d5fa in ldbm_back_add () from >>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>> #16 0x00007f683f46fcca in op_shared_add () from >>>> /usr/lib64/dirsrv/libslapd.so.0 >>>> #17 0x00007f683f471020 in do_add () from /usr/lib64/dirsrv/libslapd.so. >>>> 0 >>>> #18 0x00007f683f99c303 in connection_threadmain () >>>> #19 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so >>>> #20 0x00007f683d04cdc5 in start_thread () from /lib64/libpthread.so.0 >>>> #21 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 >>>> Thread 19 (Thread 0x7f67ffff7700 (LWP 4874)): >>>> #0 0x00007f683d0506d5 in >>>> pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 >>>> #1 0x00007f683d6a62c3 in PR_EnterMonitor () from /lib64/libnspr4.so >>>> #2 0x00007f6832b4d3e6 in dblayer_txn_begin () from >>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>> #3 0x00007f6832b89007 in ldbm_back_modify () from >>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>> #4 0x00007f683f4b6541 in op_shared_modify () from >>>> /usr/lib64/dirsrv/libslapd.so.0 >>>> #5 0x00007f683f4b7004 in modify_internal_pb () from >>>> /usr/lib64/dirsrv/libslapd.so.0 >>>> #6 0x00007f6832dd3b60 in ipapwd_apply_mods () from >>>> /usr/lib64/dirsrv/plugins/libipa_pwd_extop.so >>>> #7 0x00007f6832dd78e0 in store_new_keys () from >>>> /usr/lib64/dirsrv/plugins/libipa_pwd_extop.so >>>> #8 0x00007f6832dd9ad0 in ipapwd_extop () from >>>> /usr/lib64/dirsrv/plugins/libipa_pwd_extop.so >>>> #9 0x00007f683f4c6ba2 in plugin_call_exop_plugins () from >>>> /usr/lib64/dirsrv/libslapd.so.0 >>>> #10 0x00007f683f9a2efe in do_extended () >>>> #11 0x00007f683f99c41a in connection_threadmain () >>>> #12 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so >>>> #13 0x00007f683d04cdc5 in start_thread () from /lib64/libpthread.so.0 >>>> #14 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 >>>> Thread 18 (Thread 0x7f67ff7f6700 (LWP 4875)): >>>> #0 0x00007f683d0506d5 in >>>> pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 >>>> #1 0x00007f683d6a6050 in PR_WaitCondVar () from /lib64/libnspr4.so >>>> #2 0x00007f683f999fee in connection_wait_for_new_work () >>>> #3 0x00007f683f99b719 in connection_threadmain () >>>> #4 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so >>>> #5 0x00007f683d04cdc5 in start_thread () from /lib64/libpthread.so.0 >>>> #6 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 >>>> Thread 17 (Thread 0x7f67feff5700 (LWP 4876)): >>>> #0 0x00007f683d0506d5 in >>>> pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 >>>> #1 0x00007f68375ac2f3 in __db_hybrid_mutex_suspend () from /lib64/ >>>> libdb-5.3.so >>>> #2 0x00007f68375ab640 in __db_tas_mutex_lock () from /lib64/ >>>> libdb-5.3.so >>>> #3 0x00007f6837655d3a in __lock_get_internal () from /lib64/ >>>> libdb-5.3.so >>>> #4 0x00007f6837656820 in __lock_get () from /lib64/libdb-5.3.so >>>> #5 0x00007f6837682142 in __db_lget () from /lib64/libdb-5.3.so >>>> #6 0x00007f68375c9605 in __bam_search () from /lib64/libdb-5.3.so >>>> #7 0x00007f68375b4256 in __bamc_search () from /lib64/libdb-5.3.so >>>> #8 0x00007f68375b5d0f in __bamc_get () from /lib64/libdb-5.3.so >>>> #9 0x00007f683766eca6 in __dbc_iget () from /lib64/libdb-5.3.so >>>> #10 0x00007f683767db02 in __dbc_get_pp () from /lib64/libdb-5.3.so >>>> #11 0x00007f6832b59680 in idl_new_fetch () from >>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>> #12 0x00007f6832b67b26 in index_read_ext_allids () from >>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>> #13 0x00007f6832b52284 in keys2idl () from >>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>> #14 0x00007f6832b529e3 in ava_candidates.isra.0 () from >>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>> #15 0x00007f6832b52fd2 in filter_candidates_ext () from >>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>> #16 0x00007f6832b54096 in list_candidates () from >>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>> #17 0x00007f6832b52f40 in filter_candidates_ext () from >>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>> #18 0x00007f6832b8e80c in subtree_candidates () from >>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>> #19 0x00007f6832b8fe7f in ldbm_back_search () from >>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>> #20 0x00007f683f4bd1d6 in op_shared_search () from >>>> /usr/lib64/dirsrv/libslapd.so.0 >>>> #21 0x00007f683f4cd94e in search_internal_callback_pb () from >>>> /usr/lib64/dirsrv/libslapd.so.0 >>>> #22 0x00007f683f4cdbe8 in search_internal_pb () from >>>> /usr/lib64/dirsrv/libslapd.so.0 >>>> #23 0x00007f6832dd75dc in get_entry_by_principal () from >>>> /usr/lib64/dirsrv/plugins/libipa_pwd_extop.so >>>> #24 0x00007f6832dd8eb1 in ipapwd_extop () from >>>> /usr/lib64/dirsrv/plugins/libipa_pwd_extop.so >>>> #25 0x00007f683f4c6ba2 in plugin_call_exop_plugins () from >>>> /usr/lib64/dirsrv/libslapd.so.0 >>>> #26 0x00007f683f9a2efe in do_extended () >>>> #27 0x00007f683f99c41a in connection_threadmain () >>>> #28 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so >>>> #29 0x00007f683d04cdc5 in start_thread () from /lib64/libpthread.so.0 >>>> #30 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 >>>> Thread 16 (Thread 0x7f67fe7f4700 (LWP 4877)): >>>> #0 0x00007f683d0506d5 in >>>> pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 >>>> #1 0x00007f683d6a6050 in PR_WaitCondVar () from /lib64/libnspr4.so >>>> #2 0x00007f683f999fee in connection_wait_for_new_work () >>>> #3 0x00007f683f99b719 in connection_threadmain () >>>> #4 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so >>>> #5 0x00007f683d04cdc5 in start_thread () from /lib64/libpthread.so.0 >>>> #6 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 >>>> Thread 15 (Thread 0x7f67fdff3700 (LWP 4878)): >>>> #0 0x00007f683d0506d5 in >>>> pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 >>>> #1 0x00007f683d6a6050 in PR_WaitCondVar () from /lib64/libnspr4.so >>>> #2 0x00007f683f999fee in connection_wait_for_new_work () >>>> #3 0x00007f683f99b719 in connection_threadmain () >>>> #4 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so >>>> #5 0x00007f683d04cdc5 in start_thread () from /lib64/libpthread.so.0 >>>> #6 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 >>>> Thread 14 (Thread 0x7f67fd7f2700 (LWP 4879)): >>>> #0 0x00007f683d0506d5 in >>>> pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 >>>> #1 0x00007f683d6a6050 in PR_WaitCondVar () from /lib64/libnspr4.so >>>> #2 0x00007f683f999fee in connection_wait_for_new_work () >>>> #3 0x00007f683f99b719 in connection_threadmain () >>>> #4 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so >>>> #5 0x00007f683d04cdc5 in start_thread () from /lib64/libpthread.so.0 >>>> #6 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 >>>> Thread 13 (Thread 0x7f67fcff1700 (LWP 4880)): >>>> #0 0x00007f683d0506d5 in >>>> pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 >>>> #1 0x00007f683d6a6050 in PR_WaitCondVar () from /lib64/libnspr4.so >>>> #2 0x00007f683f999fee in connection_wait_for_new_work () >>>> #3 0x00007f683f99b719 in connection_threadmain () >>>> #4 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so >>>> #5 0x00007f683d04cdc5 in start_thread () from /lib64/libpthread.so.0 >>>> #6 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 >>>> Thread 12 (Thread 0x7f67fc7f0700 (LWP 4881)): >>>> #0 0x00007f683d0506d5 in >>>> pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 >>>> #1 0x00007f683d6a6050 in PR_WaitCondVar () from /lib64/libnspr4.so >>>> #2 0x00007f683f999fee in connection_wait_for_new_work () >>>> #3 0x00007f683f99b719 in connection_threadmain () >>>> #4 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so >>>> #5 0x00007f683d04cdc5 in start_thread () from /lib64/libpthread.so.0 >>>> #6 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 >>>> Thread 11 (Thread 0x7f67fbfef700 (LWP 4882)): >>>> #0 0x00007f683d0506d5 in >>>> pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 >>>> #1 0x00007f683d6a6050 in PR_WaitCondVar () from /lib64/libnspr4.so >>>> #2 0x00007f683f999fee in connection_wait_for_new_work () >>>> #3 0x00007f683f99b719 in connection_threadmain () >>>> #4 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so >>>> #5 0x00007f683d04cdc5 in start_thread () from /lib64/libpthread.so.0 >>>> #6 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 >>>> Thread 10 (Thread 0x7f67fb7ee700 (LWP 4883)): >>>> #0 0x00007f683d0506d5 in >>>> pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 >>>> #1 0x00007f68375ac2f3 in __db_hybrid_mutex_suspend () from /lib64/ >>>> libdb-5.3.so >>>> #2 0x00007f68375ab640 in __db_tas_mutex_lock () from /lib64/ >>>> libdb-5.3.so >>>> #3 0x00007f6837655d3a in __lock_get_internal () from /lib64/ >>>> libdb-5.3.so >>>> #4 0x00007f6837656820 in __lock_get () from /lib64/libdb-5.3.so >>>> #5 0x00007f6837682142 in __db_lget () from /lib64/libdb-5.3.so >>>> #6 0x00007f68375c9605 in __bam_search () from /lib64/libdb-5.3.so >>>> #7 0x00007f68375b4256 in __bamc_search () from /lib64/libdb-5.3.so >>>> #8 0x00007f68375b5d0f in __bamc_get () from /lib64/libdb-5.3.so >>>> #9 0x00007f683766eca6 in __dbc_iget () from /lib64/libdb-5.3.so >>>> #10 0x00007f683767db02 in __dbc_get_pp () from /lib64/libdb-5.3.so >>>> #11 0x00007f6832b59680 in idl_new_fetch () from >>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>> #12 0x00007f6832b67b26 in index_read_ext_allids () from >>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>> #13 0x00007f6832b52284 in keys2idl () from >>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>> #14 0x00007f6832b529e3 in ava_candidates.isra.0 () from >>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>> #15 0x00007f6832b52fd2 in filter_candidates_ext () from >>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>> #16 0x00007f6832b54096 in list_candidates () from >>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>> #17 0x00007f6832b52f40 in filter_candidates_ext () from >>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>> #18 0x00007f6832b54096 in list_candidates () from >>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>> #19 0x00007f6832b52f40 in filter_candidates_ext () from >>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>> #20 0x00007f6832b54096 in list_candidates () from >>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>> #21 0x00007f6832b52f40 in filter_candidates_ext () from >>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>> #22 0x00007f6832b8e80c in subtree_candidates () from >>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>> #23 0x00007f6832b8fe7f in ldbm_back_search () from >>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>> #24 0x00007f683f4bd1d6 in op_shared_search () from >>>> /usr/lib64/dirsrv/libslapd.so.0 >>>> #25 0x00007f683f9ae33e in do_search () >>>> #26 0x00007f683f99c3c3 in connection_threadmain () >>>> #27 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so >>>> #28 0x00007f683d04cdc5 in start_thread () from /lib64/libpthread.so.0 >>>> #29 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 >>>> Thread 9 (Thread 0x7f67fafed700 (LWP 4884)): >>>> #0 0x00007f683d0506d5 in >>>> pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 >>>> #1 0x00007f683d6a6050 in PR_WaitCondVar () from /lib64/libnspr4.so >>>> #2 0x00007f683f999fee in connection_wait_for_new_work () >>>> #3 0x00007f683f99b719 in connection_threadmain () >>>> #4 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so >>>> #5 0x00007f683d04cdc5 in start_thread () from /lib64/libpthread.so.0 >>>> #6 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 >>>> Thread 8 (Thread 0x7f67fa7ec700 (LWP 4885)): >>>> #0 0x00007f683d0506d5 in >>>> pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 >>>> #1 0x00007f68375ac2f3 in __db_hybrid_mutex_suspend () from /lib64/ >>>> libdb-5.3.so >>>> #2 0x00007f68375ab640 in __db_tas_mutex_lock () from /lib64/ >>>> libdb-5.3.so >>>> #3 0x00007f6837655d3a in __lock_get_internal () from /lib64/ >>>> libdb-5.3.so >>>> #4 0x00007f6837656820 in __lock_get () from /lib64/libdb-5.3.so >>>> #5 0x00007f6837682142 in __db_lget () from /lib64/libdb-5.3.so >>>> #6 0x00007f68375c9605 in __bam_search () from /lib64/libdb-5.3.so >>>> #7 0x00007f68375b4256 in __bamc_search () from /lib64/libdb-5.3.so >>>> #8 0x00007f68375b5d0f in __bamc_get () from /lib64/libdb-5.3.so >>>> #9 0x00007f683766eca6 in __dbc_iget () from /lib64/libdb-5.3.so >>>> #10 0x00007f683767db02 in __dbc_get_pp () from /lib64/libdb-5.3.so >>>> #11 0x00007f6832b59680 in idl_new_fetch () from >>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>> #12 0x00007f6832b67b26 in index_read_ext_allids () from >>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>> #13 0x00007f6832b52284 in keys2idl () from >>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>> #14 0x00007f6832b529e3 in ava_candidates.isra.0 () from >>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>> #15 0x00007f6832b52fd2 in filter_candidates_ext () from >>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>> #16 0x00007f6832b54096 in list_candidates () from >>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>> #17 0x00007f6832b52f40 in filter_candidates_ext () from >>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>> #18 0x00007f6832b54096 in list_candidates () from >>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>> #19 0x00007f6832b52f40 in filter_candidates_ext () from >>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>> #20 0x00007f6832b54096 in list_candidates () from >>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>> #21 0x00007f6832b52f40 in filter_candidates_ext () from >>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>> #22 0x00007f6832b8e80c in subtree_candidates () from >>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>> #23 0x00007f6832b8fe7f in ldbm_back_search () from >>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>> #24 0x00007f683f4bd1d6 in op_shared_search () from >>>> /usr/lib64/dirsrv/libslapd.so.0 >>>> #25 0x00007f683f9ae33e in do_search () >>>> #26 0x00007f683f99c3c3 in connection_threadmain () >>>> #27 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so >>>> #28 0x00007f683d04cdc5 in start_thread () from /lib64/libpthread.so.0 >>>> #29 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 >>>> Thread 7 (Thread 0x7f67f9feb700 (LWP 4886)): >>>> #0 0x00007f683d0506d5 in >>>> pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 >>>> #1 0x00007f683d6a6050 in PR_WaitCondVar () from /lib64/libnspr4.so >>>> #2 0x00007f683f999fee in connection_wait_for_new_work () >>>> #3 0x00007f683f99b719 in connection_threadmain () >>>> #4 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so >>>> #5 0x00007f683d04cdc5 in start_thread () from /lib64/libpthread.so.0 >>>> #6 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 >>>> Thread 6 (Thread 0x7f67f97ea700 (LWP 4887)): >>>> #0 0x00007f683d0506d5 in >>>> pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 >>>> #1 0x00007f683d6a6050 in PR_WaitCondVar () from /lib64/libnspr4.so >>>> #2 0x00007f683f999fee in connection_wait_for_new_work () >>>> #3 0x00007f683f99b719 in connection_threadmain () >>>> #4 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so >>>> #5 0x00007f683d04cdc5 in start_thread () from /lib64/libpthread.so.0 >>>> #6 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 >>>> Thread 5 (Thread 0x7f67f8fe9700 (LWP 4888)): >>>> #0 0x00007f683d0506d5 in >>>> pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 >>>> #1 0x00007f683d6a6050 in PR_WaitCondVar () from /lib64/libnspr4.so >>>> #2 0x00007f683f999fee in connection_wait_for_new_work () >>>> #3 0x00007f683f99b719 in connection_threadmain () >>>> #4 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so >>>> #5 0x00007f683d04cdc5 in start_thread () from /lib64/libpthread.so.0 >>>> #6 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 >>>> Thread 4 (Thread 0x7f67f87e8700 (LWP 4889)): >>>> #0 0x00007f683cd719b3 in select () from /lib64/libc.so.6 >>>> #1 0x00007f683f4fc0e9 in DS_Sleep () from >>>> /usr/lib64/dirsrv/libslapd.so.0 >>>> #2 0x00007f683f99d325 in time_thread () >>>> #3 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so >>>> #4 0x00007f683d04cdc5 in start_thread () from /lib64/libpthread.so.0 >>>> #5 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 >>>> Thread 3 (Thread 0x7f67f7de6700 (LWP 4892)): >>>> #0 0x00007f683d0506d5 in >>>> pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 >>>> #1 0x00007f683d6a6050 in PR_WaitCondVar () from /lib64/libnspr4.so >>>> #2 0x00007f683f9a8f25 in ps_send_results () >>>> #3 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so >>>> #4 0x00007f683d04cdc5 in start_thread () from /lib64/libpthread.so.0 >>>> #5 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 >>>> Thread 2 (Thread 0x7f67f75e5700 (LWP 5084)): >>>> #0 0x00007f683d0506d5 in >>>> pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 >>>> #1 0x00007f683d6a6050 in PR_WaitCondVar () from /lib64/libnspr4.so >>>> #2 0x00007f683f9a8f25 in ps_send_results () >>>> #3 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so >>>> #4 0x00007f683d04cdc5 in start_thread () from /lib64/libpthread.so.0 >>>> #5 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 >>>> Thread 1 (Thread 0x7f683f964840 (LWP 4845)): >>>> #0 0x00007f683d0506d5 in >>>> pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 >>>> #1 0x00007f683d6a62c3 in PR_EnterMonitor () from /lib64/libnspr4.so >>>> #2 0x00007f683f99d5ed in setup_pr_read_pds () >>>> #3 0x00007f683f9a0fd9 in slapd_daemon () >>>> #4 0x00007f683f992c6c in main () >>>> >>>> >>>> >>>> the top output seems ok, there was just a sudden spike to around 340% >>>> cpu then it comes down >>>> >>>> top - 08:53:12 up 6 days, 23:20, 1 user, load average: 0.00, 0.01, >>>> 0.12 >>>> Threads: 44 total, 0 running, 44 sleeping, 0 stopped, 0 zombie >>>> %Cpu(s): 0.0 us, 0.0 sy, 0.0 ni,100.0 id, 0.0 wa, 0.0 hi, 0.0 >>>> si, 0.0 st >>>> KiB Mem : 65533176 total, 52165676 free, 6135088 used, 7232412 >>>> buff/cache >>>> KiB Swap: 3999996 total, 3999996 free, 0 used. 58452040 avail >>>> Mem >>>> >>>> PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ >>>> COMMAND >>>> 4845 dirsrv 20 0 5133196 3.150g 281272 S 0.0 5.0 31:34.51 >>>> ns-slapd >>>> 4847 dirsrv 20 0 5133196 3.150g 281272 S 0.0 5.0 0:13.53 >>>> ns-slapd >>>> 4848 dirsrv 20 0 5133196 3.150g 281272 S 0.0 5.0 0:13.16 >>>> ns-slapd >>>> 4849 dirsrv 20 0 5133196 3.150g 281272 S 0.0 5.0 0:45.27 >>>> ns-slapd >>>> 4850 dirsrv 20 0 5133196 3.150g 281272 S 0.0 5.0 0:01.60 >>>> ns-slapd >>>> 4851 dirsrv 20 0 5133196 3.150g 281272 S 0.0 5.0 0:00.00 >>>> ns-slapd >>>> 4852 dirsrv 20 0 5133196 3.150g 281272 S 0.0 5.0 0:00.00 >>>> ns-slapd >>>> 4853 dirsrv 20 0 5133196 3.150g 281272 S 0.0 5.0 0:00.00 >>>> ns-slapd >>>> 4854 dirsrv 20 0 5133196 3.150g 281272 S 0.0 5.0 0:00.00 >>>> ns-slapd >>>> 4855 dirsrv 20 0 5133196 3.150g 281272 S 0.0 5.0 0:03.02 >>>> ns-slapd >>>> 4856 dirsrv 20 0 5133196 3.150g 281272 S 0.0 5.0 0:00.26 >>>> ns-slapd >>>> 4859 dirsrv 20 0 5133196 3.150g 281272 S 0.0 5.0 3:18.67 >>>> ns-slapd >>>> 4860 dirsrv 20 0 5133196 3.150g 281272 S 0.0 5.0 3:19.82 >>>> ns-slapd >>>> 4861 dirsrv 20 0 5133196 3.150g 281272 S 0.0 5.0 3:20.23 >>>> ns-slapd >>>> 4862 dirsrv 20 0 5133196 3.150g 281272 S 0.0 5.0 3:22.65 >>>> ns-slapd >>>> 4863 dirsrv 20 0 5133196 3.150g 281272 S 0.0 5.0 3:16.64 >>>> ns-slapd >>>> 4864 dirsrv 20 0 5133196 3.150g 281272 S 0.0 5.0 3:17.39 >>>> ns-slapd >>>> 4865 dirsrv 20 0 5133196 3.150g 281272 S 0.0 5.0 3:17.80 >>>> ns-slapd >>>> >>>> >>>> Thanks >>>> Rakesh >>>> >>>> On Tue, Sep 6, 2016 at 12:09 AM, Rakesh Rajasekharan < >>>> rakesh.rajasekharan at gmail.com> wrote: >>>> >>>>> Hi Thierry/Ludwig, >>>>> >>>>> Unfortunately I had by then restarted the service and since then have >>>>> been trying to hit the hang issue again but no luck this time >>>>> >>>>> It has been running fine for the last few hours now in-spite of trying >>>>> all that I believe could cause the hang like making simultaneous client >>>>> addition requests. >>>>> >>>>> I will continue testing this and next time I get it I will make sure I >>>>> get the pstack and also get the dump of db lock info. >>>>> >>>>> Thanks, >>>>> Rakesh >>>>> >>>>> >>>>> >>>>> On Mon, Sep 5, 2016 at 6:14 PM, Ludwig Krispenz < >>>>> lkrispen at redhat.com> wrote: >>>>> >>>>>> >>>>>> On 09/05/2016 02:31 PM, thierry bordaz wrote: >>>>>> >>>>>> Hi Rakesh, >>>>>> >>>>>> Thanks you soooo much for you continuous help on this !! >>>>>> >>>>>> If the hang is still going on, would you dump the DB lock information >>>>>> with: >>>>>> >>>>>> db_stat -CA -N -h /var/lib/dirsrv/slapd-/db >>>>>> >/tmp/db_lock_info >>>>>> >>>>>> It is looking like several threads are waiting for one/several db >>>>>> locks. With the lock info we will know which thread acquired it/them. >>>>>> A guess is that it could be the checkpointing thread but this one is >>>>>> also waiting (on a memory pool). >>>>>> >>>>>> I am copying Ludwig on this thread in case this pstack rings any bell. >>>>>> >>>>>> the frequent presence of __db_tas_mutex_lock() looks like the NUMA >>>>>> problem of libdb, although the threads already have stopped spinning (maybe >>>>>> this was an effect of gcore). >>>>>> If the hang aoccurs again, could you check cpu consumption >>>>>> top -H -p >>>>>> and get a pstack >>>>>> pstack >>>>>> >>>>>> >>>>>> best regards >>>>>> thierry >>>>>> >>>>>> On 09/05/2016 02:05 PM, Rakesh Rajasekharan wrote: >>>>>> >>>>>> Hi Thierry, >>>>>> >>>>>> I was able to now reporduce the issue.. >>>>>> >>>>>> the ns-slapd process just got hung,.. however, no core file was >>>>>> generate.. >>>>>> >>>>>> I manually created a coredump using gcore and then took a strace of it >>>>>> this is strace from the coredump generated >>>>>> >>>>>> GNU gdb (GDB) Red Hat Enterprise Linux 7.6.1-80.el7 >>>>>> Copyright (C) 2013 Free Software Foundation, Inc. >>>>>> License GPLv3+: GNU GPL version 3 or later < >>>>>> http://gnu.org/licenses/gpl.html> >>>>>> This is free software: you are free to change and redistribute it. >>>>>> There is NO WARRANTY, to the extent permitted by law. Type "show >>>>>> copying" >>>>>> and "show warranty" for details. >>>>>> This GDB was configured as "x86_64-redhat-linux-gnu". >>>>>> For bug reporting instructions, please see: >>>>>> < >>>>>> http://www.gnu.org/software/gdb/bugs/>... >>>>>> Reading symbols from /usr/sbin/ns-slapd...Reading symbols from >>>>>> /usr/sbin/ns-slapd...(no debugging symbols found)...done. >>>>>> (no debugging symbols found)...done. >>>>>> [New LWP 105159] >>>>>> [New LWP 105160] >>>>>> [New LWP 105161] >>>>>> [New LWP 105162] >>>>>> [New LWP 105163] >>>>>> [New LWP 105164] >>>>>> [New LWP 105165] >>>>>> [New LWP 105166] >>>>>> [New LWP 105167] >>>>>> [New LWP 105168] >>>>>> [New LWP 105171] >>>>>> [New LWP 105172] >>>>>> [New LWP 105173] >>>>>> [New LWP 105174] >>>>>> [New LWP 105175] >>>>>> [New LWP 105176] >>>>>> [New LWP 105177] >>>>>> [New LWP 105178] >>>>>> [New LWP 105179] >>>>>> [New LWP 105180] >>>>>> [New LWP 105181] >>>>>> [New LWP 105182] >>>>>> [New LWP 105183] >>>>>> [New LWP 105184] >>>>>> [New LWP 105185] >>>>>> [New LWP 105186] >>>>>> [New LWP 105187] >>>>>> [New LWP 105188] >>>>>> [New LWP 105189] >>>>>> [New LWP 105190] >>>>>> [New LWP 105191] >>>>>> [New LWP 105192] >>>>>> [New LWP 105193] >>>>>> [New LWP 105194] >>>>>> [New LWP 105195] >>>>>> [New LWP 105196] >>>>>> [New LWP 105197] >>>>>> [New LWP 105198] >>>>>> [New LWP 105199] >>>>>> [New LWP 105200] >>>>>> [New LWP 105201] >>>>>> [New LWP 105203] >>>>>> [New LWP 105215] >>>>>> [New LWP 105158] >>>>>> [Thread debugging using libthread_db enabled] >>>>>> Using host libthread_db library "/lib64/libthread_db.so.1". >>>>>> Core was generated by `/usr/sbin/ns-slapd'. >>>>>> #0 0x00007f1b48ae69b3 in select () from /lib64/libc.so.6 >>>>>> >>>>>> Thread 44 (Thread 0x7f1b4b6d9840 (LWP 105158)): >>>>>> #0 0x00007f1b48dc56d5 in >>>>>> pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 >>>>>> No symbol table info available. >>>>>> #1 0x00007f1b4941b2c3 in PR_EnterMonitor () from /lib64/libnspr4.so >>>>>> No symbol table info available. >>>>>> #2 0x00007f1b4b7125ed in setup_pr_read_pds () >>>>>> No symbol table info available. >>>>>> #3 0x00007f1b4b715fd9 in slapd_daemon () >>>>>> No symbol table info available. >>>>>> #4 0x00007f1b4b707c6c in main () >>>>>> No symbol table info available. >>>>>> >>>>>> Thread 43 (Thread 0x7f1a5df0f700 (LWP 105215)): >>>>>> #0 0x00007f1b48dc56d5 in >>>>>> pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 >>>>>> No symbol table info available. >>>>>> #1 0x00007f1b4941b050 in PR_WaitCondVar () from /lib64/libnspr4.so >>>>>> No symbol table info available. >>>>>> #2 0x00007f1b4b71df25 in ps_send_results () >>>>>> No symbol table info available. >>>>>> #3 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so >>>>>> No symbol table info available. >>>>>> #4 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 >>>>>> No symbol table info available. >>>>>> #5 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 >>>>>> No symbol table info available. >>>>>> >>>>>> Thread 42 (Thread 0x7f1b03fe7700 (LWP 105203)): >>>>>> #0 0x00007f1b48dc56d5 in >>>>>> pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 >>>>>> No symbol table info available. >>>>>> #1 0x00007f1b4941b050 in PR_WaitCondVar () from /lib64/libnspr4.so >>>>>> No symbol table info available. >>>>>> #2 0x00007f1b4b71df25 in ps_send_results () >>>>>> No symbol table info available. >>>>>> #3 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so >>>>>> No symbol table info available. >>>>>> #4 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 >>>>>> No symbol table info available. >>>>>> #5 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 >>>>>> No symbol table info available. >>>>>> >>>>>> Thread 41 (Thread 0x7f1b047e8700 (LWP 105201)): >>>>>> #0 0x00007f1b48ae69b3 in select () from /lib64/libc.so.6 >>>>>> No symbol table info available. >>>>>> #1 0x00007f1b4b2710e9 in DS_Sleep () from >>>>>> /usr/lib64/dirsrv/libslapd.so.0 >>>>>> No symbol table info available. >>>>>> #2 0x00007f1b4b712325 in time_thread () >>>>>> No symbol table info available. >>>>>> #3 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so >>>>>> No symbol table info available. >>>>>> #4 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 >>>>>> No symbol table info available. >>>>>> #5 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 >>>>>> No symbol table info available. >>>>>> >>>>>> Thread 40 (Thread 0x7f1b04fe9700 (LWP 105200)): >>>>>> #0 0x00007f1b48dc56d5 in >>>>>> pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 >>>>>> No symbol table info available. >>>>>> #1 0x00007f1b433212f3 in __db_hybrid_mutex_suspend () from /lib64/ >>>>>> libdb-5.3.so >>>>>> No symbol table info available. >>>>>> #2 0x00007f1b43320640 in __db_tas_mutex_lock () from /lib64/ >>>>>> libdb-5.3.so >>>>>> No symbol table info available. >>>>>> #3 0x00007f1b4343ae1b in __memp_fget () from /lib64/libdb-5.3.so >>>>>> No symbol table info available. >>>>>> #4 0x00007f1b4333e211 in __bam_search () from /lib64/libdb-5.3.so >>>>>> No symbol table info available. >>>>>> #5 0x00007f1b43329256 in __bamc_search () from /lib64/libdb-5.3.so >>>>>> No symbol table info available. >>>>>> #6 0x00007f1b4332d6c4 in __bamc_put () from /lib64/libdb-5.3.so >>>>>> No symbol table info available. >>>>>> #7 0x00007f1b433e46d5 in __dbc_iput () from /lib64/libdb-5.3.so >>>>>> No symbol table info available. >>>>>> #8 0x00007f1b433dfa0e in __db_put () from /lib64/libdb-5.3.so >>>>>> No symbol table info available. >>>>>> #9 0x00007f1b433f4fa4 in __db_put_pp () from /lib64/libdb-5.3.so >>>>>> No symbol table info available. >>>>>> #10 0x00007f1b3e8cf4cf in idl_new_insert_key () from >>>>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>>>> No symbol table info available. >>>>>> #11 0x00007f1b3e8de5b5 in addordel_values_sv.isra.3 () from >>>>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>>>> No symbol table info available. >>>>>> #12 0x00007f1b3e8df25a in index_addordel_values_ext_sv () from >>>>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>>>> No symbol table info available. >>>>>> #13 0x00007f1b3e8df694 in index_addordel_values_sv () from >>>>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>>>> No symbol table info available. >>>>>> #14 0x00007f1b3e8df75a in index_addordel_entry () from >>>>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>>>> No symbol table info available. >>>>>> #15 0x00007f1b3e8e25fa in ldbm_back_add () from >>>>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>>>> No symbol table info available. >>>>>> #16 0x00007f1b4b1e4cca in op_shared_add () from >>>>>> /usr/lib64/dirsrv/libslapd.so.0 >>>>>> No symbol table info available. >>>>>> #17 0x00007f1b4b1e6020 in do_add () from >>>>>> /usr/lib64/dirsrv/libslapd.so.0 >>>>>> No symbol table info available. >>>>>> #18 0x00007f1b4b711303 in connection_threadmain () >>>>>> No symbol table info available. >>>>>> #19 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so >>>>>> No symbol table info available. >>>>>> #20 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 >>>>>> No symbol table info available. >>>>>> #21 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 >>>>>> No symbol table info available. >>>>>> >>>>>> Thread 39 (Thread 0x7f1b057ea700 (LWP 105199)): >>>>>> #0 0x00007f1b48dc56d5 in >>>>>> pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 >>>>>> No symbol table info available. >>>>>> #1 0x00007f1b4941b050 in PR_WaitCondVar () from /lib64/libnspr4.so >>>>>> No symbol table info available. >>>>>> #2 0x00007f1b4b70efee in connection_wait_for_new_work () >>>>>> No symbol table info available. >>>>>> #3 0x00007f1b4b710719 in connection_threadmain () >>>>>> No symbol table info available. >>>>>> #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so >>>>>> No symbol table info available. >>>>>> #5 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 >>>>>> No symbol table info available. >>>>>> #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 >>>>>> No symbol table info available. >>>>>> >>>>>> Thread 38 (Thread 0x7f1b05feb700 (LWP 105198)): >>>>>> #0 0x00007f1b48dc56d5 in >>>>>> pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 >>>>>> No symbol table info available. >>>>>> #1 0x00007f1b4941b050 in PR_WaitCondVar () from /lib64/libnspr4.so >>>>>> No symbol table info available. >>>>>> #2 0x00007f1b4b70efee in connection_wait_for_new_work () >>>>>> No symbol table info available. >>>>>> #3 0x00007f1b4b710719 in connection_threadmain () >>>>>> No symbol table info available. >>>>>> #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so >>>>>> No symbol table info available. >>>>>> #5 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 >>>>>> No symbol table info available. >>>>>> #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 >>>>>> No symbol table info available. >>>>>> >>>>>> Thread 37 (Thread 0x7f1b067ec700 (LWP 105197)): >>>>>> #0 0x00007f1b48dc56d5 in >>>>>> pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 >>>>>> No symbol table info available. >>>>>> #1 0x00007f1b4941b050 in PR_WaitCondVar () from /lib64/libnspr4.so >>>>>> No symbol table info available. >>>>>> #2 0x00007f1b4b70efee in connection_wait_for_new_work () >>>>>> No symbol table info available. >>>>>> #3 0x00007f1b4b710719 in connection_threadmain () >>>>>> No symbol table info available. >>>>>> #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so >>>>>> No symbol table info available. >>>>>> #5 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 >>>>>> No symbol table info available. >>>>>> #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 >>>>>> No symbol table info available. >>>>>> >>>>>> Thread 36 (Thread 0x7f1b06fed700 (LWP 105196)): >>>>>> #0 0x00007f1b48dc56d5 in >>>>>> pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 >>>>>> No symbol table info available. >>>>>> #1 0x00007f1b4941b050 in PR_WaitCondVar () from /lib64/libnspr4.so >>>>>> No symbol table info available. >>>>>> #2 0x00007f1b4b70efee in connection_wait_for_new_work () >>>>>> No symbol table info available. >>>>>> #3 0x00007f1b4b710719 in connection_threadmain () >>>>>> No symbol table info available. >>>>>> #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so >>>>>> No symbol table info available. >>>>>> #5 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 >>>>>> No symbol table info available. >>>>>> #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 >>>>>> No symbol table info available. >>>>>> >>>>>> Thread 35 (Thread 0x7f1b077ee700 (LWP 105195)): >>>>>> #0 0x00007f1b48dc56d5 in >>>>>> pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 >>>>>> No symbol table info available. >>>>>> #1 0x00007f1b4941b2c3 in PR_EnterMonitor () from /lib64/libnspr4.so >>>>>> No symbol table info available. >>>>>> #2 0x00007f1b3e8c23e6 in dblayer_txn_begin () from >>>>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>>>> No symbol table info available. >>>>>> #3 0x00007f1b3e8fe007 in ldbm_back_modify () from >>>>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>>>> No symbol table info available. >>>>>> #4 0x00007f1b4b22b541 in op_shared_modify () from >>>>>> /usr/lib64/dirsrv/libslapd.so.0 >>>>>> No symbol table info available. >>>>>> #5 0x00007f1b4b22c004 in modify_internal_pb () from >>>>>> /usr/lib64/dirsrv/libslapd.so.0 >>>>>> No symbol table info available. >>>>>> #6 0x00007f1b3eb48b60 in ipapwd_apply_mods () from >>>>>> /usr/lib64/dirsrv/plugins/libipa_pwd_extop.so >>>>>> No symbol table info available. >>>>>> #7 0x00007f1b3eb4c8e0 in store_new_keys () from >>>>>> /usr/lib64/dirsrv/plugins/libipa_pwd_extop.so >>>>>> No symbol table info available. >>>>>> #8 0x00007f1b3eb4ead0 in ipapwd_extop () from >>>>>> /usr/lib64/dirsrv/plugins/libipa_pwd_extop.so >>>>>> No symbol table info available. >>>>>> #9 0x00007f1b4b23bba2 in plugin_call_exop_plugins () from >>>>>> /usr/lib64/dirsrv/libslapd.so.0 >>>>>> No symbol table info available. >>>>>> #10 0x00007f1b4b717efe in do_extended () >>>>>> No symbol table info available. >>>>>> #11 0x00007f1b4b71141a in connection_threadmain () >>>>>> No symbol table info available. >>>>>> #12 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so >>>>>> No symbol table info available. >>>>>> #13 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 >>>>>> No symbol table info available. >>>>>> #14 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 >>>>>> No symbol table info available. >>>>>> >>>>>> Thread 34 (Thread 0x7f1b07fef700 (LWP 105194)): >>>>>> #0 0x00007f1b48dc56d5 in >>>>>> pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 >>>>>> No symbol table info available. >>>>>> #1 0x00007f1b4941b050 in PR_WaitCondVar () from /lib64/libnspr4.so >>>>>> No symbol table info available. >>>>>> #2 0x00007f1b4b70efee in connection_wait_for_new_work () >>>>>> No symbol table info available. >>>>>> #3 0x00007f1b4b710719 in connection_threadmain () >>>>>> No symbol table info available. >>>>>> #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so >>>>>> No symbol table info available. >>>>>> #5 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 >>>>>> No symbol table info available. >>>>>> #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 >>>>>> No symbol table info available. >>>>>> >>>>>> Thread 33 (Thread 0x7f1b087f0700 (LWP 105193)): >>>>>> #0 0x00007f1b48dc56d5 in >>>>>> pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 >>>>>> No symbol table info available. >>>>>> #1 0x00007f1b4941b050 in PR_WaitCondVar () from /lib64/libnspr4.so >>>>>> No symbol table info available. >>>>>> #2 0x00007f1b4b70efee in connection_wait_for_new_work () >>>>>> No symbol table info available. >>>>>> #3 0x00007f1b4b710719 in connection_threadmain () >>>>>> No symbol table info available. >>>>>> #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so >>>>>> No symbol table info available. >>>>>> #5 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 >>>>>> No symbol table info available. >>>>>> #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 >>>>>> No symbol table info available. >>>>>> >>>>>> Thread 32 (Thread 0x7f1b08ff1700 (LWP 105192)): >>>>>> #0 0x00007f1b48dc56d5 in >>>>>> pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 >>>>>> No symbol table info available. >>>>>> #1 0x00007f1b4941b050 in PR_WaitCondVar () from /lib64/libnspr4.so >>>>>> No symbol table info available. >>>>>> #2 0x00007f1b4b70efee in connection_wait_for_new_work () >>>>>> No symbol table info available. >>>>>> #3 0x00007f1b4b710719 in connection_threadmain () >>>>>> No symbol table info available. >>>>>> #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so >>>>>> No symbol table info available. >>>>>> #5 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 >>>>>> No symbol table info available. >>>>>> #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 >>>>>> No symbol table info available. >>>>>> >>>>>> Thread 31 (Thread 0x7f1b097f2700 (LWP 105191)): >>>>>> #0 0x00007f1b48dc56d5 in >>>>>> pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 >>>>>> No symbol table info available. >>>>>> #1 0x00007f1b4941b050 in PR_WaitCondVar () from /lib64/libnspr4.so >>>>>> No symbol table info available. >>>>>> #2 0x00007f1b4b70efee in connection_wait_for_new_work () >>>>>> No symbol table info available. >>>>>> #3 0x00007f1b4b710719 in connection_threadmain () >>>>>> No symbol table info available. >>>>>> #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so >>>>>> No symbol table info available. >>>>>> #5 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 >>>>>> No symbol table info available. >>>>>> #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 >>>>>> No symbol table info available. >>>>>> >>>>>> Thread 30 (Thread 0x7f1b09ff3700 (LWP 105190)): >>>>>> #0 0x00007f1b48dc56d5 in >>>>>> pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 >>>>>> No symbol table info available. >>>>>> #1 0x00007f1b433212f3 in __db_hybrid_mutex_suspend () from /lib64/ >>>>>> libdb-5.3.so >>>>>> No symbol table info available. >>>>>> #2 0x00007f1b43320640 in __db_tas_mutex_lock () from /lib64/ >>>>>> libdb-5.3.so >>>>>> No symbol table info available. >>>>>> #3 0x00007f1b433cad3a in __lock_get_internal () from /lib64/ >>>>>> libdb-5.3.so >>>>>> No symbol table info available. >>>>>> #4 0x00007f1b433cb820 in __lock_get () from /lib64/libdb-5.3.so >>>>>> No symbol table info available. >>>>>> #5 0x00007f1b433f7142 in __db_lget () from /lib64/libdb-5.3.so >>>>>> No symbol table info available. >>>>>> #6 0x00007f1b4333e605 in __bam_search () from /lib64/libdb-5.3.so >>>>>> No symbol table info available. >>>>>> #7 0x00007f1b43329256 in __bamc_search () from /lib64/libdb-5.3.so >>>>>> No symbol table info available. >>>>>> #8 0x00007f1b4332ad0f in __bamc_get () from /lib64/libdb-5.3.so >>>>>> No symbol table info available. >>>>>> #9 0x00007f1b433e3ca6 in __dbc_iget () from /lib64/libdb-5.3.so >>>>>> No symbol table info available. >>>>>> #10 0x00007f1b433f2b02 in __dbc_get_pp () from /lib64/libdb-5.3.so >>>>>> No symbol table info available. >>>>>> #11 0x00007f1b3e8ce680 in idl_new_fetch () from >>>>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>>>> No symbol table info available. >>>>>> #12 0x00007f1b3e8dcb26 in index_read_ext_allids () from >>>>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>>>> No symbol table info available. >>>>>> #13 0x00007f1b3e8c7284 in keys2idl () from >>>>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>>>> No symbol table info available. >>>>>> #14 0x00007f1b3e8c79e3 in ava_candidates.isra.0 () from >>>>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>>>> No symbol table info available. >>>>>> #15 0x00007f1b3e8c7fd2 in filter_candidates_ext () from >>>>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>>>> No symbol table info available. >>>>>> #16 0x00007f1b3e8c9096 in list_candidates () from >>>>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>>>> No symbol table info available. >>>>>> #17 0x00007f1b3e8c7f40 in filter_candidates_ext () from >>>>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>>>> No symbol table info available. >>>>>> #18 0x00007f1b3e8c9096 in list_candidates () from >>>>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>>>> No symbol table info available. >>>>>> #19 0x00007f1b3e8c7f40 in filter_candidates_ext () from >>>>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>>>> No symbol table info available. >>>>>> #20 0x00007f1b3e8c9096 in list_candidates () from >>>>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>>>> No symbol table info available. >>>>>> #21 0x00007f1b3e8c7f40 in filter_candidates_ext () from >>>>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>>>> No symbol table info available. >>>>>> #22 0x00007f1b3e90380c in subtree_candidates () from >>>>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>>>> No symbol table info available. >>>>>> #23 0x00007f1b3e904e7f in ldbm_back_search () from >>>>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>>>> No symbol table info available. >>>>>> #24 0x00007f1b4b2321d6 in op_shared_search () from >>>>>> /usr/lib64/dirsrv/libslapd.so.0 >>>>>> No symbol table info available. >>>>>> #25 0x00007f1b4b72333e in do_search () >>>>>> No symbol table info available. >>>>>> #26 0x00007f1b4b7113c3 in connection_threadmain () >>>>>> No symbol table info available. >>>>>> #27 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so >>>>>> No symbol table info available. >>>>>> #28 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 >>>>>> No symbol table info available. >>>>>> #29 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 >>>>>> No symbol table info available. >>>>>> >>>>>> Thread 29 (Thread 0x7f1b0a7f4700 (LWP 105189)): >>>>>> #0 0x00007f1b48dc56d5 in >>>>>> pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 >>>>>> No symbol table info available. >>>>>> #1 0x00007f1b4941b050 in PR_WaitCondVar () from /lib64/libnspr4.so >>>>>> No symbol table info available. >>>>>> #2 0x00007f1b4b70efee in connection_wait_for_new_work () >>>>>> No symbol table info available. >>>>>> #3 0x00007f1b4b710719 in connection_threadmain () >>>>>> No symbol table info available. >>>>>> #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so >>>>>> No symbol table info available. >>>>>> #5 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 >>>>>> No symbol table info available. >>>>>> #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 >>>>>> No symbol table info available. >>>>>> >>>>>> Thread 28 (Thread 0x7f1b0aff5700 (LWP 105188)): >>>>>> #0 0x00007f1b48dc56d5 in >>>>>> pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 >>>>>> No symbol table info available. >>>>>> #1 0x00007f1b4941b050 in PR_WaitCondVar () from /lib64/libnspr4.so >>>>>> No symbol table info available. >>>>>> #2 0x00007f1b4b70efee in connection_wait_for_new_work () >>>>>> No symbol table info available. >>>>>> #3 0x00007f1b4b710719 in connection_threadmain () >>>>>> No symbol table info available. >>>>>> #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so >>>>>> No symbol table info available. >>>>>> #5 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 >>>>>> No symbol table info available. >>>>>> #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 >>>>>> No symbol table info available. >>>>>> >>>>>> Thread 27 (Thread 0x7f1b0b7f6700 (LWP 105187)): >>>>>> #0 0x00007f1b48dc56d5 in >>>>>> pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 >>>>>> No symbol table info available. >>>>>> #1 0x00007f1b4941b050 in PR_WaitCondVar () from /lib64/libnspr4.so >>>>>> No symbol table info available. >>>>>> #2 0x00007f1b4b70efee in connection_wait_for_new_work () >>>>>> No symbol table info available. >>>>>> #3 0x00007f1b4b710719 in connection_threadmain () >>>>>> No symbol table info available. >>>>>> #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so >>>>>> No symbol table info available. >>>>>> #5 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 >>>>>> No symbol table info available. >>>>>> #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 >>>>>> No symbol table info available. >>>>>> >>>>>> Thread 26 (Thread 0x7f1b0bff7700 (LWP 105186)): >>>>>> #0 0x00007f1b48dc56d5 in >>>>>> pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 >>>>>> No symbol table info available. >>>>>> #1 0x00007f1b433212f3 in __db_hybrid_mutex_suspend () from /lib64/ >>>>>> libdb-5.3.so >>>>>> No symbol table info available. >>>>>> #2 0x00007f1b43320640 in __db_tas_mutex_lock () from /lib64/ >>>>>> libdb-5.3.so >>>>>> No symbol table info available. >>>>>> #3 0x00007f1b433cad3a in __lock_get_internal () from /lib64/ >>>>>> libdb-5.3.so >>>>>> No symbol table info available. >>>>>> #4 0x00007f1b433cb820 in __lock_get () from /lib64/libdb-5.3.so >>>>>> No symbol table info available. >>>>>> #5 0x00007f1b433f7142 in __db_lget () from /lib64/libdb-5.3.so >>>>>> No symbol table info available. >>>>>> #6 0x00007f1b4333e605 in __bam_search () from /lib64/libdb-5.3.so >>>>>> No symbol table info available. >>>>>> #7 0x00007f1b43329256 in __bamc_search () from /lib64/libdb-5.3.so >>>>>> No symbol table info available. >>>>>> #8 0x00007f1b4332ad0f in __bamc_get () from /lib64/libdb-5.3.so >>>>>> No symbol table info available. >>>>>> #9 0x00007f1b433e3ca6 in __dbc_iget () from /lib64/libdb-5.3.so >>>>>> No symbol table info available. >>>>>> #10 0x00007f1b433f2b02 in __dbc_get_pp () from /lib64/libdb-5.3.so >>>>>> No symbol table info available. >>>>>> #11 0x00007f1b3e8ce680 in idl_new_fetch () from >>>>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>>>> No symbol table info available. >>>>>> #12 0x00007f1b3e8dcb26 in index_read_ext_allids () from >>>>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>>>> No symbol table info available. >>>>>> #13 0x00007f1b3e8c7284 in keys2idl () from >>>>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>>>> No symbol table info available. >>>>>> #14 0x00007f1b3e8c79e3 in ava_candidates.isra.0 () from >>>>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>>>> No symbol table info available. >>>>>> #15 0x00007f1b3e8c7fd2 in filter_candidates_ext () from >>>>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>>>> No symbol table info available. >>>>>> #16 0x00007f1b3e8c9096 in list_candidates () from >>>>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>>>> No symbol table info available. >>>>>> #17 0x00007f1b3e8c7f40 in filter_candidates_ext () from >>>>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>>>> No symbol table info available. >>>>>> #18 0x00007f1b3e90380c in subtree_candidates () from >>>>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>>>> No symbol table info available. >>>>>> #19 0x00007f1b3e904e7f in ldbm_back_search () from >>>>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>>>> No symbol table info available. >>>>>> #20 0x00007f1b4b2321d6 in op_shared_search () from >>>>>> /usr/lib64/dirsrv/libslapd.so.0 >>>>>> No symbol table info available. >>>>>> #21 0x00007f1b4b24294e in search_internal_callback_pb () from >>>>>> /usr/lib64/dirsrv/libslapd.so.0 >>>>>> No symbol table info available. >>>>>> #22 0x00007f1b4b242be8 in search_internal_pb () from >>>>>> /usr/lib64/dirsrv/libslapd.so.0 >>>>>> No symbol table info available. >>>>>> #23 0x00007f1b4b72149b in ids_sasl_canon_user () >>>>>> No symbol table info available. >>>>>> #24 0x00007f1b491e193b in _sasl_canon_user () from >>>>>> /lib64/libsasl2.so.3 >>>>>> No symbol table info available. >>>>>> #25 0x00007f1b491e1c4c in _sasl_canon_user_lookup () from >>>>>> /lib64/libsasl2.so.3 >>>>>> No symbol table info available. >>>>>> #26 0x00007f1b430e9fb9 in gssapi_server_mech_step () from >>>>>> /usr/lib64/sasl2/libgssapiv2.so >>>>>> No symbol table info available. >>>>>> #27 0x00007f1b491edb9b in sasl_server_step () from >>>>>> /lib64/libsasl2.so.3 >>>>>> No symbol table info available. >>>>>> #28 0x00007f1b4b72283b in ids_sasl_check_bind () >>>>>> No symbol table info available. >>>>>> #29 0x00007f1b4b70a61a in do_bind () >>>>>> No symbol table info available. >>>>>> #30 0x00007f1b4b7113fd in connection_threadmain () >>>>>> No symbol table info available. >>>>>> #31 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so >>>>>> No symbol table info available. >>>>>> #32 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 >>>>>> No symbol table info available. >>>>>> #33 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 >>>>>> No symbol table info available. >>>>>> >>>>>> Thread 25 (Thread 0x7f1b0c7f8700 (LWP 105185)): >>>>>> #0 0x00007f1b48dc56d5 in >>>>>> pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 >>>>>> No symbol table info available. >>>>>> #1 0x00007f1b4941b050 in PR_WaitCondVar () from /lib64/libnspr4.so >>>>>> No symbol table info available. >>>>>> #2 0x00007f1b4b70efee in connection_wait_for_new_work () >>>>>> No symbol table info available. >>>>>> #3 0x00007f1b4b710719 in connection_threadmain () >>>>>> No symbol table info available. >>>>>> #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so >>>>>> No symbol table info available. >>>>>> #5 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 >>>>>> No symbol table info available. >>>>>> #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 >>>>>> No symbol table info available. >>>>>> >>>>>> Thread 24 (Thread 0x7f1b0cff9700 (LWP 105184)): >>>>>> #0 0x00007f1b48dc56d5 in >>>>>> pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 >>>>>> No symbol table info available. >>>>>> #1 0x00007f1b4941b050 in PR_WaitCondVar () from /lib64/libnspr4.so >>>>>> No symbol table info available. >>>>>> #2 0x00007f1b4b70efee in connection_wait_for_new_work () >>>>>> No symbol table info available. >>>>>> #3 0x00007f1b4b710719 in connection_threadmain () >>>>>> No symbol table info available. >>>>>> #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so >>>>>> No symbol table info available. >>>>>> #5 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 >>>>>> No symbol table info available. >>>>>> #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 >>>>>> No symbol table info available. >>>>>> >>>>>> Thread 23 (Thread 0x7f1b0d7fa700 (LWP 105183)): >>>>>> #0 0x00007f1b48dc56d5 in >>>>>> pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 >>>>>> No symbol table info available. >>>>>> #1 0x00007f1b4941b050 in PR_WaitCondVar () from /lib64/libnspr4.so >>>>>> No symbol table info available. >>>>>> #2 0x00007f1b4b70efee in connection_wait_for_new_work () >>>>>> No symbol table info available. >>>>>> #3 0x00007f1b4b710719 in connection_threadmain () >>>>>> No symbol table info available. >>>>>> #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so >>>>>> No symbol table info available. >>>>>> #5 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 >>>>>> No symbol table info available. >>>>>> #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 >>>>>> No symbol table info available. >>>>>> >>>>>> Thread 22 (Thread 0x7f1b0dffb700 (LWP 105182)): >>>>>> #0 0x00007f1b48dc56d5 in >>>>>> pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 >>>>>> No symbol table info available. >>>>>> #1 0x00007f1b4941b050 in PR_WaitCondVar () from /lib64/libnspr4.so >>>>>> No symbol table info available. >>>>>> #2 0x00007f1b4b70efee in connection_wait_for_new_work () >>>>>> No symbol table info available. >>>>>> #3 0x00007f1b4b710719 in connection_threadmain () >>>>>> No symbol table info available. >>>>>> #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so >>>>>> No symbol table info available. >>>>>> #5 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 >>>>>> No symbol table info available. >>>>>> #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 >>>>>> No symbol table info available. >>>>>> >>>>>> Thread 21 (Thread 0x7f1b0e7fc700 (LWP 105181)): >>>>>> #0 0x00007f1b48dc56d5 in >>>>>> pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 >>>>>> No symbol table info available. >>>>>> #1 0x00007f1b4941b050 in PR_WaitCondVar () from /lib64/libnspr4.so >>>>>> No symbol table info available. >>>>>> #2 0x00007f1b4b70efee in connection_wait_for_new_work () >>>>>> No symbol table info available. >>>>>> #3 0x00007f1b4b710719 in connection_threadmain () >>>>>> No symbol table info available. >>>>>> #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so >>>>>> No symbol table info available. >>>>>> #5 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 >>>>>> No symbol table info available. >>>>>> #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 >>>>>> No symbol table info available. >>>>>> >>>>>> Thread 20 (Thread 0x7f1b0effd700 (LWP 105180)): >>>>>> #0 0x00007f1b48dc56d5 in >>>>>> pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 >>>>>> No symbol table info available. >>>>>> #1 0x00007f1b433212f3 in __db_hybrid_mutex_suspend () from /lib64/ >>>>>> libdb-5.3.so >>>>>> No symbol table info available. >>>>>> #2 0x00007f1b43320b37 in __db_tas_mutex_readlock () from /lib64/ >>>>>> libdb-5.3.so >>>>>> No symbol table info available. >>>>>> #3 0x00007f1b4343ad35 in __memp_fget () from /lib64/libdb-5.3.so >>>>>> No symbol table info available. >>>>>> #4 0x00007f1b433e4333 in __dbc_iget () from /lib64/libdb-5.3.so >>>>>> No symbol table info available. >>>>>> #5 0x00007f1b433f2b02 in __dbc_get_pp () from /lib64/libdb-5.3.so >>>>>> No symbol table info available. >>>>>> #6 0x00007f1b3e8ce822 in idl_new_fetch () from >>>>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>>>> No symbol table info available. >>>>>> #7 0x00007f1b3e8dcb26 in index_read_ext_allids () from >>>>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>>>> No symbol table info available. >>>>>> #8 0x00007f1b3e8c7284 in keys2idl () from >>>>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>>>> No symbol table info available. >>>>>> #9 0x00007f1b3e8c79e3 in ava_candidates.isra.0 () from >>>>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>>>> No symbol table info available. >>>>>> #10 0x00007f1b3e8c7fd2 in filter_candidates_ext () from >>>>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>>>> No symbol table info available. >>>>>> #11 0x00007f1b3e8c9096 in list_candidates () from >>>>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>>>> No symbol table info available. >>>>>> #12 0x00007f1b3e8c7f40 in filter_candidates_ext () from >>>>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>>>> No symbol table info available. >>>>>> #13 0x00007f1b3e8c9096 in list_candidates () from >>>>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>>>> No symbol table info available. >>>>>> #14 0x00007f1b3e8c7f40 in filter_candidates_ext () from >>>>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>>>> No symbol table info available. >>>>>> #15 0x00007f1b3e8c9096 in list_candidates () from >>>>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>>>> No symbol table info available. >>>>>> #16 0x00007f1b3e8c7f40 in filter_candidates_ext () from >>>>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>>>> No symbol table info available. >>>>>> #17 0x00007f1b3e90380c in subtree_candidates () from >>>>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>>>> No symbol table info available. >>>>>> #18 0x00007f1b3e904e7f in ldbm_back_search () from >>>>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>>>> No symbol table info available. >>>>>> #19 0x00007f1b4b2321d6 in op_shared_search () from >>>>>> /usr/lib64/dirsrv/libslapd.so.0 >>>>>> No symbol table info available. >>>>>> #20 0x00007f1b4b72333e in do_search () >>>>>> No symbol table info available. >>>>>> #21 0x00007f1b4b7113c3 in connection_threadmain () >>>>>> No symbol table info available. >>>>>> #22 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so >>>>>> No symbol table info available. >>>>>> #23 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 >>>>>> No symbol table info available. >>>>>> #24 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 >>>>>> No symbol table info available. >>>>>> >>>>>> Thread 19 (Thread 0x7f1b0f7fe700 (LWP 105179)): >>>>>> #0 0x00007f1b48dc56d5 in >>>>>> pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 >>>>>> No symbol table info available. >>>>>> #1 0x00007f1b433212f3 in __db_hybrid_mutex_suspend () from /lib64/ >>>>>> libdb-5.3.so >>>>>> No symbol table info available. >>>>>> #2 0x00007f1b43320640 in __db_tas_mutex_lock () from /lib64/ >>>>>> libdb-5.3.so >>>>>> No symbol table info available. >>>>>> #3 0x00007f1b433cad3a in __lock_get_internal () from /lib64/ >>>>>> libdb-5.3.so >>>>>> No symbol table info available. >>>>>> #4 0x00007f1b433cb820 in __lock_get () from /lib64/libdb-5.3.so >>>>>> No symbol table info available. >>>>>> #5 0x00007f1b433f7142 in __db_lget () from /lib64/libdb-5.3.so >>>>>> No symbol table info available. >>>>>> #6 0x00007f1b4333e605 in __bam_search () from /lib64/libdb-5.3.so >>>>>> No symbol table info available. >>>>>> #7 0x00007f1b43329256 in __bamc_search () from /lib64/libdb-5.3.so >>>>>> No symbol table info available. >>>>>> #8 0x00007f1b4332ad0f in __bamc_get () from /lib64/libdb-5.3.so >>>>>> No symbol table info available. >>>>>> #9 0x00007f1b433e3ca6 in __dbc_iget () from /lib64/libdb-5.3.so >>>>>> No symbol table info available. >>>>>> #10 0x00007f1b433f2b02 in __dbc_get_pp () from /lib64/libdb-5.3.so >>>>>> No symbol table info available. >>>>>> #11 0x00007f1b3e8ce680 in idl_new_fetch () from >>>>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>>>> No symbol table info available. >>>>>> #12 0x00007f1b3e8dcb26 in index_read_ext_allids () from >>>>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>>>> No symbol table info available. >>>>>> #13 0x00007f1b3e8c7284 in keys2idl () from >>>>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>>>> No symbol table info available. >>>>>> #14 0x00007f1b3e8c79e3 in ava_candidates.isra.0 () from >>>>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>>>> No symbol table info available. >>>>>> #15 0x00007f1b3e8c7fd2 in filter_candidates_ext () from >>>>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>>>> No symbol table info available. >>>>>> #16 0x00007f1b3e8c9096 in list_candidates () from >>>>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>>>> No symbol table info available. >>>>>> #17 0x00007f1b3e8c7f40 in filter_candidates_ext () from >>>>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>>>> No symbol table info available. >>>>>> #18 0x00007f1b3e8c9096 in list_candidates () from >>>>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>>>> No symbol table info available. >>>>>> #19 0x00007f1b3e8c7f40 in filter_candidates_ext () from >>>>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>>>> No symbol table info available. >>>>>> #20 0x00007f1b3e8c9096 in list_candidates () from >>>>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>>>> No symbol table info available. >>>>>> #21 0x00007f1b3e8c7f40 in filter_candidates_ext () from >>>>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>>>> No symbol table info available. >>>>>> #22 0x00007f1b3e90380c in subtree_candidates () from >>>>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>>>> No symbol table info available. >>>>>> #23 0x00007f1b3e904e7f in ldbm_back_search () from >>>>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>>>> No symbol table info available. >>>>>> #24 0x00007f1b4b2321d6 in op_shared_search () from >>>>>> /usr/lib64/dirsrv/libslapd.so.0 >>>>>> No symbol table info available. >>>>>> #25 0x00007f1b4b72333e in do_search () >>>>>> No symbol table info available. >>>>>> #26 0x00007f1b4b7113c3 in connection_threadmain () >>>>>> No symbol table info available. >>>>>> #27 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so >>>>>> No symbol table info available. >>>>>> #28 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 >>>>>> No symbol table info available. >>>>>> #29 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 >>>>>> No symbol table info available. >>>>>> >>>>>> Thread 18 (Thread 0x7f1b0ffff700 (LWP 105178)): >>>>>> #0 0x00007f1b48dc56d5 in >>>>>> pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 >>>>>> No symbol table info available. >>>>>> #1 0x00007f1b4941b050 in PR_WaitCondVar () from /lib64/libnspr4.so >>>>>> No symbol table info available. >>>>>> #2 0x00007f1b4b70efee in connection_wait_for_new_work () >>>>>> No symbol table info available. >>>>>> #3 0x00007f1b4b710719 in connection_threadmain () >>>>>> No symbol table info available. >>>>>> #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so >>>>>> No symbol table info available. >>>>>> #5 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 >>>>>> No symbol table info available. >>>>>> #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 >>>>>> No symbol table info available. >>>>>> >>>>>> Thread 17 (Thread 0x7f1b14d28700 (LWP 105177)): >>>>>> #0 0x00007f1b48dc56d5 in >>>>>> pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 >>>>>> No symbol table info available. >>>>>> #1 0x00007f1b4941b050 in PR_WaitCondVar () from /lib64/libnspr4.so >>>>>> No symbol table info available. >>>>>> #2 0x00007f1b4b70efee in connection_wait_for_new_work () >>>>>> No symbol table info available. >>>>>> #3 0x00007f1b4b710719 in connection_threadmain () >>>>>> No symbol table info available. >>>>>> #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so >>>>>> No symbol table info available. >>>>>> #5 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 >>>>>> No symbol table info available. >>>>>> #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 >>>>>> No symbol table info available. >>>>>> >>>>>> Thread 16 (Thread 0x7f1b15529700 (LWP 105176)): >>>>>> #0 0x00007f1b48dc56d5 in >>>>>> pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 >>>>>> No symbol table info available. >>>>>> #1 0x00007f1b4941b050 in PR_WaitCondVar () from /lib64/libnspr4.so >>>>>> No symbol table info available. >>>>>> #2 0x00007f1b4b70efee in connection_wait_for_new_work () >>>>>> No symbol table info available. >>>>>> #3 0x00007f1b4b710719 in connection_threadmain () >>>>>> No symbol table info available. >>>>>> #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so >>>>>> No symbol table info available. >>>>>> #5 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 >>>>>> No symbol table info available. >>>>>> #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 >>>>>> No symbol table info available. >>>>>> >>>>>> Thread 15 (Thread 0x7f1b15d2a700 (LWP 105175)): >>>>>> #0 0x00007f1b48dc56d5 in >>>>>> pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 >>>>>> No symbol table info available. >>>>>> #1 0x00007f1b4941b050 in PR_WaitCondVar () from /lib64/libnspr4.so >>>>>> No symbol table info available. >>>>>> #2 0x00007f1b4b70efee in connection_wait_for_new_work () >>>>>> No symbol table info available. >>>>>> #3 0x00007f1b4b710719 in connection_threadmain () >>>>>> No symbol table info available. >>>>>> #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so >>>>>> No symbol table info available. >>>>>> #5 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 >>>>>> No symbol table info available. >>>>>> #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 >>>>>> No symbol table info available. >>>>>> >>>>>> Thread 14 (Thread 0x7f1b1652b700 (LWP 105174)): >>>>>> #0 0x00007f1b48dc56d5 in >>>>>> pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 >>>>>> No symbol table info available. >>>>>> #1 0x00007f1b4941b050 in PR_WaitCondVar () from /lib64/libnspr4.so >>>>>> No symbol table info available. >>>>>> #2 0x00007f1b4b70efee in connection_wait_for_new_work () >>>>>> No symbol table info available. >>>>>> #3 0x00007f1b4b710719 in connection_threadmain () >>>>>> No symbol table info available. >>>>>> #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so >>>>>> No symbol table info available. >>>>>> #5 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 >>>>>> No symbol table info available. >>>>>> #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 >>>>>> No symbol table info available. >>>>>> >>>>>> Thread 13 (Thread 0x7f1b16d2c700 (LWP 105173)): >>>>>> #0 0x00007f1b48dc56d5 in >>>>>> pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 >>>>>> No symbol table info available. >>>>>> #1 0x00007f1b433212f3 in __db_hybrid_mutex_suspend () from /lib64/ >>>>>> libdb-5.3.so >>>>>> No symbol table info available. >>>>>> #2 0x00007f1b43320640 in __db_tas_mutex_lock () from /lib64/ >>>>>> libdb-5.3.so >>>>>> No symbol table info available. >>>>>> #3 0x00007f1b433cad3a in __lock_get_internal () from /lib64/ >>>>>> libdb-5.3.so >>>>>> No symbol table info available. >>>>>> #4 0x00007f1b433cb820 in __lock_get () from /lib64/libdb-5.3.so >>>>>> No symbol table info available. >>>>>> #5 0x00007f1b433f7142 in __db_lget () from /lib64/libdb-5.3.so >>>>>> No symbol table info available. >>>>>> #6 0x00007f1b4333e605 in __bam_search () from /lib64/libdb-5.3.so >>>>>> No symbol table info available. >>>>>> #7 0x00007f1b43329256 in __bamc_search () from /lib64/libdb-5.3.so >>>>>> No symbol table info available. >>>>>> #8 0x00007f1b4332ad0f in __bamc_get () from /lib64/libdb-5.3.so >>>>>> No symbol table info available. >>>>>> #9 0x00007f1b433e3ca6 in __dbc_iget () from /lib64/libdb-5.3.so >>>>>> No symbol table info available. >>>>>> #10 0x00007f1b433f2b02 in __dbc_get_pp () from /lib64/libdb-5.3.so >>>>>> No symbol table info available. >>>>>> #11 0x00007f1b3e8ce680 in idl_new_fetch () from >>>>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>>>> No symbol table info available. >>>>>> #12 0x00007f1b3e8dcb26 in index_read_ext_allids () from >>>>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>>>> No symbol table info available. >>>>>> #13 0x00007f1b3e8c7284 in keys2idl () from >>>>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>>>> No symbol table info available. >>>>>> #14 0x00007f1b3e8c79e3 in ava_candidates.isra.0 () from >>>>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>>>> No symbol table info available. >>>>>> #15 0x00007f1b3e8c7fd2 in filter_candidates_ext () from >>>>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>>>> No symbol table info available. >>>>>> #16 0x00007f1b3e8c9096 in list_candidates () from >>>>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>>>> No symbol table info available. >>>>>> #17 0x00007f1b3e8c7f40 in filter_candidates_ext () from >>>>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>>>> No symbol table info available. >>>>>> #18 0x00007f1b3e8c9096 in list_candidates () from >>>>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>>>> No symbol table info available. >>>>>> #19 0x00007f1b3e8c7f40 in filter_candidates_ext () from >>>>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>>>> No symbol table info available. >>>>>> #20 0x00007f1b3e8c9096 in list_candidates () from >>>>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>>>> No symbol table info available. >>>>>> #21 0x00007f1b3e8c7f40 in filter_candidates_ext () from >>>>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>>>> No symbol table info available. >>>>>> #22 0x00007f1b3e90380c in subtree_candidates () from >>>>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>>>> No symbol table info available. >>>>>> #23 0x00007f1b3e904e7f in ldbm_back_search () from >>>>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>>>> No symbol table info available. >>>>>> #24 0x00007f1b4b2321d6 in op_shared_search () from >>>>>> /usr/lib64/dirsrv/libslapd.so.0 >>>>>> No symbol table info available. >>>>>> #25 0x00007f1b4b72333e in do_search () >>>>>> No symbol table info available. >>>>>> #26 0x00007f1b4b7113c3 in connection_threadmain () >>>>>> No symbol table info available. >>>>>> #27 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so >>>>>> No symbol table info available. >>>>>> #28 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 >>>>>> No symbol table info available. >>>>>> #29 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 >>>>>> No symbol table info available. >>>>>> >>>>>> Thread 12 (Thread 0x7f1b1752d700 (LWP 105172)): >>>>>> #0 0x00007f1b48dc56d5 in >>>>>> pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 >>>>>> No symbol table info available. >>>>>> #1 0x00007f1b4941b050 in PR_WaitCondVar () from /lib64/libnspr4.so >>>>>> No symbol table info available. >>>>>> #2 0x00007f1b4b70efee in connection_wait_for_new_work () >>>>>> No symbol table info available. >>>>>> #3 0x00007f1b4b710719 in connection_threadmain () >>>>>> No symbol table info available. >>>>>> #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so >>>>>> No symbol table info available. >>>>>> #5 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 >>>>>> No symbol table info available. >>>>>> #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 >>>>>> No symbol table info available. >>>>>> >>>>>> Thread 11 (Thread 0x7f1b17d2e700 (LWP 105171)): >>>>>> #0 0x00007f1b48dc56d5 in >>>>>> pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 >>>>>> No symbol table info available. >>>>>> #1 0x00007f1b4941b050 in PR_WaitCondVar () from /lib64/libnspr4.so >>>>>> No symbol table info available. >>>>>> #2 0x00007f1b4b70efee in connection_wait_for_new_work () >>>>>> No symbol table info available. >>>>>> #3 0x00007f1b4b710719 in connection_threadmain () >>>>>> No symbol table info available. >>>>>> #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so >>>>>> No symbol table info available. >>>>>> #5 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 >>>>>> No symbol table info available. >>>>>> #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 >>>>>> No symbol table info available. >>>>>> >>>>>> Thread 10 (Thread 0x7f1b2509d700 (LWP 105168)): >>>>>> #0 0x00007f1b48dc5a82 in pthread_cond_timedwait@@GLIBC_2.3.2 () >>>>>> from /lib64/libpthread.so.0 >>>>>> No symbol table info available. >>>>>> #1 0x00007f1b4941ab07 in pt_TimedWait () from /lib64/libnspr4.so >>>>>> No symbol table info available. >>>>>> #2 0x00007f1b4941afce in PR_WaitCondVar () from /lib64/libnspr4.so >>>>>> No symbol table info available. >>>>>> #3 0x00007f1b4b2061e8 in eq_loop () from >>>>>> /usr/lib64/dirsrv/libslapd.so.0 >>>>>> No symbol table info available. >>>>>> #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so >>>>>> No symbol table info available. >>>>>> #5 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 >>>>>> No symbol table info available. >>>>>> #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 >>>>>> No symbol table info available. >>>>>> >>>>>> Thread 9 (Thread 0x7f1b2589e700 (LWP 105167)): >>>>>> #0 0x00007f1b48dc5a82 in pthread_cond_timedwait@@GLIBC_2.3.2 () >>>>>> from /lib64/libpthread.so.0 >>>>>> No symbol table info available. >>>>>> #1 0x00007f1b4941ab07 in pt_TimedWait () from /lib64/libnspr4.so >>>>>> No symbol table info available. >>>>>> #2 0x00007f1b4941afce in PR_WaitCondVar () from /lib64/libnspr4.so >>>>>> No symbol table info available. >>>>>> #3 0x00007f1b4b719d03 in housecleaning () >>>>>> No symbol table info available. >>>>>> #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so >>>>>> No symbol table info available. >>>>>> #5 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 >>>>>> No symbol table info available. >>>>>> #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 >>>>>> No symbol table info available. >>>>>> >>>>>> Thread 8 (Thread 0x7f1b2609f700 (LWP 105166)): >>>>>> #0 0x00007f1b48dc56d5 in >>>>>> pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 >>>>>> No symbol table info available. >>>>>> #1 0x00007f1b4941b050 in PR_WaitCondVar () from /lib64/libnspr4.so >>>>>> No symbol table info available. >>>>>> #2 0x00007f1b4b260198 in slapi_wait_condvar () from >>>>>> /usr/lib64/dirsrv/libslapd.so.0 >>>>>> No symbol table info available. >>>>>> #3 0x00007f1b3cf5deed in roles_cache_wait_on_change () from >>>>>> /usr/lib64/dirsrv/plugins/libroles-plugin.so >>>>>> No symbol table info available. >>>>>> #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so >>>>>> No symbol table info available. >>>>>> #5 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 >>>>>> No symbol table info available. >>>>>> #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 >>>>>> No symbol table info available. >>>>>> >>>>>> Thread 7 (Thread 0x7f1b268a0700 (LWP 105165)): >>>>>> #0 0x00007f1b48dc56d5 in >>>>>> pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 >>>>>> No symbol table info available. >>>>>> #1 0x00007f1b4941b050 in PR_WaitCondVar () from /lib64/libnspr4.so >>>>>> No symbol table info available. >>>>>> #2 0x00007f1b4b260198 in slapi_wait_condvar () from >>>>>> /usr/lib64/dirsrv/libslapd.so.0 >>>>>> No symbol table info available. >>>>>> #3 0x00007f1b3cf5deed in roles_cache_wait_on_change () from >>>>>> /usr/lib64/dirsrv/plugins/libroles-plugin.so >>>>>> No symbol table info available. >>>>>> #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so >>>>>> No symbol table info available. >>>>>> #5 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 >>>>>> No symbol table info available. >>>>>> #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 >>>>>> No symbol table info available. >>>>>> >>>>>> Thread 6 (Thread 0x7f1b270a1700 (LWP 105164)): >>>>>> #0 0x00007f1b48dc56d5 in >>>>>> pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 >>>>>> No symbol table info available. >>>>>> #1 0x00007f1b4941b050 in PR_WaitCondVar () from /lib64/libnspr4.so >>>>>> No symbol table info available. >>>>>> #2 0x00007f1b4b260198 in slapi_wait_condvar () from >>>>>> /usr/lib64/dirsrv/libslapd.so.0 >>>>>> No symbol table info available. >>>>>> #3 0x00007f1b3cf5deed in roles_cache_wait_on_change () from >>>>>> /usr/lib64/dirsrv/plugins/libroles-plugin.so >>>>>> No symbol table info available. >>>>>> #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so >>>>>> No symbol table info available. >>>>>> #5 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 >>>>>> No symbol table info available. >>>>>> #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 >>>>>> No symbol table info available. >>>>>> >>>>>> Thread 5 (Thread 0x7f1b278a2700 (LWP 105163)): >>>>>> #0 0x00007f1b48dc56d5 in >>>>>> pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 >>>>>> No symbol table info available. >>>>>> #1 0x00007f1b4941b050 in PR_WaitCondVar () from /lib64/libnspr4.so >>>>>> No symbol table info available. >>>>>> #2 0x00007f1b4b260198 in slapi_wait_condvar () from >>>>>> /usr/lib64/dirsrv/libslapd.so.0 >>>>>> No symbol table info available. >>>>>> #3 0x00007f1b415f262e in cos_cache_wait_on_change () from >>>>>> /usr/lib64/dirsrv/plugins/libcos-plugin.so >>>>>> No symbol table info available. >>>>>> #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so >>>>>> No symbol table info available. >>>>>> #5 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 >>>>>> No symbol table info available. >>>>>> #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 >>>>>> No symbol table info available. >>>>>> >>>>>> Thread 4 (Thread 0x7f1b282ac700 (LWP 105162)): >>>>>> #0 0x00007f1b48ae69b3 in select () from /lib64/libc.so.6 >>>>>> No symbol table info available. >>>>>> #1 0x00007f1b4b2710e9 in DS_Sleep () from >>>>>> /usr/lib64/dirsrv/libslapd.so.0 >>>>>> No symbol table info available. >>>>>> #2 0x00007f1b3e8ba707 in perf_threadmain () from >>>>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>>>> No symbol table info available. >>>>>> #3 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so >>>>>> No symbol table info available. >>>>>> #4 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 >>>>>> No symbol table info available. >>>>>> #5 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 >>>>>> No symbol table info available. >>>>>> >>>>>> Thread 3 (Thread 0x7f1b28aad700 (LWP 105161)): >>>>>> #0 0x00007f1b48ae69b3 in select () from /lib64/libc.so.6 >>>>>> No symbol table info available. >>>>>> #1 0x00007f1b4b2710e9 in DS_Sleep () from >>>>>> /usr/lib64/dirsrv/libslapd.so.0 >>>>>> No symbol table info available. >>>>>> #2 0x00007f1b3e8bfb7f in trickle_threadmain () from >>>>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>>>> No symbol table info available. >>>>>> #3 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so >>>>>> No symbol table info available. >>>>>> #4 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 >>>>>> No symbol table info available. >>>>>> #5 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 >>>>>> No symbol table info available. >>>>>> >>>>>> Thread 2 (Thread 0x7f1b292ae700 (LWP 105160)): >>>>>> #0 0x00007f1b48ae69b3 in select () from /lib64/libc.so.6 >>>>>> No symbol table info available. >>>>>> #1 0x00007f1b4344dbad in __os_yield () from /lib64/libdb-5.3.so >>>>>> No symbol table info available. >>>>>> #2 0x00007f1b434492b3 in __memp_sync_int () from /lib64/libdb-5.3.so >>>>>> No symbol table info available. >>>>>> #3 0x00007f1b43459752 in __txn_checkpoint () from /lib64/ >>>>>> libdb-5.3.so >>>>>> No symbol table info available. >>>>>> #4 0x00007f1b43459b74 in __txn_checkpoint_pp () from /lib64/ >>>>>> libdb-5.3.so >>>>>> No symbol table info available. >>>>>> #5 0x00007f1b3e8c3a87 in checkpoint_threadmain () from >>>>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>>>> No symbol table info available. >>>>>> #6 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so >>>>>> No symbol table info available. >>>>>> #7 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 >>>>>> No symbol table info available. >>>>>> #8 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 >>>>>> No symbol table info available. >>>>>> >>>>>> Thread 1 (Thread 0x7f1b29aaf700 (LWP 105159)): >>>>>> #0 0x00007f1b48ae69b3 in select () from /lib64/libc.so.6 >>>>>> No symbol table info available. >>>>>> #1 0x00007f1b4b2710e9 in DS_Sleep () from >>>>>> /usr/lib64/dirsrv/libslapd.so.0 >>>>>> No symbol table info available. >>>>>> #2 0x00007f1b3e8bf907 in deadlock_threadmain () from >>>>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>>>> No symbol table info available. >>>>>> #3 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so >>>>>> No symbol table info available. >>>>>> #4 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 >>>>>> No symbol table info available. >>>>>> #5 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 >>>>>> No symbol table info available. >>>>>> >>>>>> >>>>>> >>>>>> Thanks, >>>>>> >>>>>> Rakesh >>>>>> >>>>>> >>>>>> On Mon, Sep 5, 2016 at 4:48 PM, thierry bordaz < >>>>>> tbordaz at redhat.com> wrote: >>>>>> >>>>>>> >>>>>>> >>>>>>> On 09/05/2016 12:05 PM, Rakesh Rajasekharan wrote: >>>>>>> >>>>>>> Hi Thierry, >>>>>>> >>>>>>> >>>>>>> I was getting the hang issue while running ipa-client-install >>>>>>> simultaneously on few clients.. >>>>>>> However, today, I am not able to replicate that. >>>>>>> >>>>>>> I could not get a gdb . But i will try getting that the next time I >>>>>>> face this issue. >>>>>>> >>>>>>> The CPU does not stay high.. it just momentarily touches a high >>>>>>> value and then drops down to around 2-7% >>>>>>> >>>>>>> One question I have is , is it ok to set it nsslapd-threadnumber to >>>>>>> a very high value . >>>>>>> I have around 4000 clients and with nsslapd-maxthreadsperconn set to >>>>>>> 5..So, can I set nsslapd-threadnumber to around 25000. >>>>>>> >>>>>>> >>>>>>> Hello, >>>>>>> >>>>>>> I know some users running in production with several hunderds of >>>>>>> threads (>600) and this without problem. >>>>>>> >>>>>>> I do not recall having suggested to increase that number and for >>>>>>> what reason. >>>>>>> Usually 30 workers is a good enough value. It can create bootleneck >>>>>>> if for some reason each operation is very long to satisfy and exhaust the >>>>>>> number of workers. You can monitor the work queue: >>>>>>> >>>>>>> ldapsearch -D "cn=directory manager" -w xxx -LLL -b "cn=monitor" -s >>>>>>> base opsinitiated opscompleted >>>>>>> >>>>>>> >>>>>>> If opscompleted-opsinitiated remains close to threadnumber, then yes >>>>>>> it would be valuable to increase it. >>>>>>> >>>>>>> The computation #client * #async_op_per_client sound an overkill. >>>>>>> Even if all clients send at the exact same time all their requests, it is >>>>>>> very likely that some common resource (db page, log, allocator...) will >>>>>>> serialize them. If you monitor a need to increase the work, you would for >>>>>>> example set it to 50, then monitor, then set it to 100, then monitor... >>>>>>> until you find a good enough value. >>>>>>> Note the increasing the #thread, increases the memory footprint that >>>>>>> will reduce the efficiency of file system cache and can increase the >>>>>>> response time. >>>>>>> >>>>>>> >>>>>>> best regards >>>>>>> thierry >>>>>>> >>>>>>> >>>>>>> Thanks >>>>>>> >>>>>>> On Mon, Sep 5, 2016 at 1:03 PM, thierry bordaz < >>>>>>> tbordaz at redhat.com> wrote: >>>>>>> >>>>>>>> >>>>>>>> Hi Rakesh, >>>>>>>> >>>>>>>> Were you able to get a pstack or full stack with gdb ( >>>>>>>> http://www.port389.org/docs/3 >>>>>>>> 89ds/FAQ/faq.html#debugging-crashes) when the server hangs ? >>>>>>>> >>>>>>>> If it happens with 500 threads as well as with 30, using 30 threads >>>>>>>> is a better choice to debug this issue. >>>>>>>> I will try to reproduce using 150 parallel 'ipa user-find >>>>>>>> p-testipa' commands >>>>>>>> >>>>>>>> Something I am unsure is if the CPU consumption stays high (you >>>>>>>> mentioned 340% CPU usage) as long as the hang happens or if after a suddent >>>>>>>> shot up to 340% (that marks the beginning of the hang) it drops and stay >>>>>>>> hanging ? >>>>>>>> >>>>>>>> thanks >>>>>>>> thierry >>>>>>>> >>>>>>>> On 09/04/2016 08:40 PM, Rakesh Rajasekharan wrote: >>>>>>>> >>>>>>>> starce on the slapd process actually had this in the output.. >>>>>>>> FUTEX_WAIT_PRIVATE >>>>>>>> >>>>>>>> and checking for the number of threads slapd had.. there were 5015 >>>>>>>> threads >>>>>>>> >>>>>>>> ps -efL|grep slapd|wc -l >>>>>>>> 5015 >>>>>>>> >>>>>>>> strace on most of the threads gave this output >>>>>>>> >>>>>>>> strace -p 67411 >>>>>>>> Process 67411 attached >>>>>>>> futex(0x7f3f0226b41c, FUTEX_WAIT_PRIVATE, 1, NULL) = -1 EAGAIN >>>>>>>> (Resource temporarily unavailable) >>>>>>>> futex(0x7f3f0226b41c, FUTEX_WAIT_PRIVATE, 2, NULL^CProcess 67411 >>>>>>>> detached >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> On Sun, Sep 4, 2016 at 5:34 PM, Rakesh Rajasekharan < >>>>>>>> rakesh.rajasekharan at gmail.com> >>>>>>>> wrote: >>>>>>>> >>>>>>>>> I have again got the issue of IPA hanging.. The issue came up when >>>>>>>>> i tried to run ipa-client-isntall on 142 clients simultaneously >>>>>>>>> >>>>>>>>> >>>>>>>>> None of the IPA commands are responding, and I see this error >>>>>>>>> >>>>>>>>> ipa user-find p-testipa >>>>>>>>> ipa: ERROR: Insufficient access: SASL(-1): generic failure: GSSAPI >>>>>>>>> Error: Unspecified GSS failure. Minor code may provide more information >>>>>>>>> (KDC returned error string: PROCESS_TGS) >>>>>>>>> >>>>>>>>> KRB5_TRACE=/dev/stdout kinit admin >>>>>>>>> [41178] 1472984115.233214: Getting initial credentials for >>>>>>>>> admin at XYZ.COM >>>>>>>>> [41178] 1472984115.235257: Sending request (167 bytes) to XYZ.COM >>>>>>>>> [41178] 1472984115.235419: Initiating TCP connection to stream >>>>>>>>> 10.1.3.36:88 >>>>>>>>> [41178] 1472984115.235685: Sending TCP request to stream >>>>>>>>> 10.1.3.36:88 >>>>>>>>> [41178] 1472984120.238914: Received answer (174 bytes) from stream >>>>>>>>> 10.1.3.36:88 >>>>>>>>> [41178] 1472984120.238925: Terminating TCP connection to stream >>>>>>>>> 10.1.3.36:88 >>>>>>>>> [41178] 1472984120.238993: Response was from master KDC >>>>>>>>> [41 >>>>>>>>> >>>>>>>>> >>>>>>>>> Running an ldapsearch to see the db.. does not give any results >>>>>>>>> and just hangs there >>>>>>>>> >>>>>>>>> ldapsearch -x -D 'cn=Directory Manager' -W -s one -b >>>>>>>>> 'cn=kerberos,dc=xyz,dc=com' >>>>>>>>> Enter LDAP Password: >>>>>>>>> >>>>>>>>> even an ldapsearch -x does not respond >>>>>>>>> At this point, am sure that slapd is the one causing issues >>>>>>>>> >>>>>>>>> Running an strace against the hung slapd itself seems to get stuck >>>>>>>>> does not proceed after saying "attaching to process" >>>>>>>>> >>>>>>>>> From some others posts I read Thierry suggesting to increase the >>>>>>>>> nsslapd-threadnumber value >>>>>>>>> >>>>>>>>> It was set to 30, I think that might be too low. >>>>>>>>> >>>>>>>> ... >>>>>> >>>>>> [Message clipped] >>>>> >>>>> >>>>> >>>> >>>> >>>> -- >>>> Red Hat GmbH, http://www.de.redhat.com/, Registered seat: Grasbrunn, >>>> Commercial register: Amtsgericht Muenchen, HRB 153243, >>>> Managing Directors: Charles Cachera, Michael Cunningham, Michael O'Neill, Eric Shander >>>> >>>> >>> >> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From abokovoy at redhat.com Fri Sep 16 04:57:04 2016 From: abokovoy at redhat.com (Alexander Bokovoy) Date: Fri, 16 Sep 2016 07:57:04 +0300 Subject: [Freeipa-users] how to revert ipa-adtrust-install... In-Reply-To: <57DB1487.4000500@redhat.com> References: <1ec82d0c-1350-feb3-4da5-a022eb7ac5e5@yahoo.co.uk> <20160915191709.6vk7y3gidzxoicen@redhat.com> <98df2894-d5c7-082c-8fe8-77ece9474301@yahoo.co.uk> <20160915200055.qpx7gma722tjcfxk@redhat.com> <57DB1487.4000500@redhat.com> Message-ID: <20160916045704.5emkdqda3uvqdzlc@redhat.com> On Thu, 15 Sep 2016, Rob Crittenden wrote: >Alexander Bokovoy wrote: >>On Thu, 15 Sep 2016, lejeczek wrote: >>>is there any way to tell IPA not to control smb.service? >>Do not run ipa-adtrust-install on the IPA master. >> > >What do you mean control? If you don't want ipactl to manage the smb >service, look for an entry in >cn=masters,cn=ipa,cn=etc,dc=example,dc=com and delete it if you find >it. At some point we decided to not do 'ipa-adtrust-install --uninstall' because restoring previous Samba configuration is not easy. Aside from smb.conf, there are multiple databases in /var/lib/samba which will have no meaning if smb.conf is restored to pre-'ipa-adtrust-install' set up. Removing (or recovering from backup) those databases wouldn't necessary create a working Samba configuration as it was in pre-'ipa-adtrust-install' stage. Which is especially doubtful to restore to with the case when all users are actually in IPA LDAP and there is no Samba passdb module that could readily consume IPA LDAP schema, other than ipasam module that IPA provides. At this point the question would be why to revert to non-IPA configuration if it could only work with IPA LDAP when using ipasam module which is configured by ipa-adtrust-install. So we decided to not perform 'ipa-adtrust-install --uninstall' as it makes no sense. If somebode is willing to uninstall 'ipa-adtrust-install', then need to realize what they are doing as it would need to remove certain configuration in IPA LDAP because there are actual 389-ds plugins that depend on the configuration and work jointly with ipasam module in Samba to provide common setup. If 'ipasam' is missing, those modules also become useless. -- / Alexander Bokovoy From abokovoy at redhat.com Fri Sep 16 05:04:35 2016 From: abokovoy at redhat.com (Alexander Bokovoy) Date: Fri, 16 Sep 2016 08:04:35 +0300 Subject: [Freeipa-users] Samba Server setup In-Reply-To: <9A3517B6-CFFB-47ED-BDF2-BE3EE0D06EDE@bsd.uchicago.edu> References: <73C11E05-045F-4324-9733-44C370584232@bsd.uchicago.edu> <20160915180606.v2z6dkdw3gltem34@redhat.com> <9A3517B6-CFFB-47ED-BDF2-BE3EE0D06EDE@bsd.uchicago.edu> Message-ID: <20160916050435.l4k6rshziftp6pzd@redhat.com> On Thu, 15 Sep 2016, Brook, Andy [CRI] wrote: >On 9/15/16, 1:06 PM, "Alexander Bokovoy" wrote: > > On Thu, 15 Sep 2016, Brook, Andy [CRI] wrote: > >All, > > I?m working on setting up Samba to serve files from a server attached > > to our IPA domain. I followed the directions in > > https://www.freeipa.org/page/Howto/Integrating_a_Samba_File_Server_With_IPA. > > Everything seems to work and I can access the files from another RHEL > > server attached to the same domain using a Kerberos ticket from a > > user from the trusted AD domain. However, I can?t access this share > > from a windows client that is also attached to the trusted AD domain. > > > >My smb.conf is as follows: > >[global] > > workgroup = IPA > > realm = IPA.DOMAIN > > kerberos method = dedicated keytab > > dedicated keytab file = FILE:/etc/samba/samba.keytab > > log file = /var/log/samba/log.%m > > log level = 3 > > security = ads > > load printers = no > > disable spoolss = yes > > map to guest = Never > > restrict anonymous = 2 > > > >[spacetest] > > path = /var/www > > writable = yes > > browsable = yes > > > >I put the keytab in place from the cifs service from the IPA server. > > > >I feel like I?m missing something small, but I can?t seem to find it. > >Logs from samba are here: http://pastebin.com/aMDXfR78 > These logs show that your Windows client did not use Kerberos but tried > to authenticate with password using NTLMSSP. This is not supported yet, > as written on the page you used for the setup guidance. > > You need to find out why Windows client didn't use Kerberos. > Is your trust to AD really working? > >We?re authenticating AD users on the hosts that are connected to IPA. >We?re able to create external groups and associate them with internal >groups for HBAC and sudoers rules. Is there something else I should >check to see if the trust is working? It?s entirely possible I missed >something somewhere in the setup, but I don?t think I did. Start by listing your configuration. Show: - ipa service-show cifs/samba.server.host (using FQDN hostname) - ipa trust-show ad.domain - kinit user at AD.DOMAIN ; KRB5_TRACE=/dev/stderr smbclient -k //samba.server.host/share - Try to access \\samba.server.host\share from Windows host and then show 'klist' in the Windows shell, or alternatively, - if you are using Windows Server 2012 or later, show output of 'klist get cifs/samba.server.host at IPA.DOMAIN' - show any lines related to use user at AD.DOMAIN and cifs/samba.server.host from /var/log/krb5kdc.log on IPA master You can replace actual hostnames/realm names/IP addresses by something more generic in the output when sending to the list, but please do it consistently. -- / Alexander Bokovoy From tjaalton at ubuntu.com Fri Sep 16 06:18:25 2016 From: tjaalton at ubuntu.com (Timo Aaltonen) Date: Fri, 16 Sep 2016 09:18:25 +0300 Subject: [Freeipa-users] Ubuntu 16.04 released with FreeIPA 4.3.1 In-Reply-To: <57C749C8.908@ubuntu.com> References: <57192380.4090400@ubuntu.com> <57C749C8.908@ubuntu.com> Message-ID: <060f6d55-5c78-622e-91bc-892e3750ddaa@ubuntu.com> On 01.09.2016 00:19, Timo Aaltonen wrote: > On 31.08.2016 11:18, Petr Spacek wrote: >> On 31.8.2016 00:23, Timo Aaltonen wrote: >>> On 29.08.2016 10:34, Timo Aaltonen wrote: >>>> On 21.04.2016 22:01, Timo Aaltonen wrote: >>>>> >>>>> ps. Debian unstable will have 4.3.1 once the package has gone through >>>>> the NEW queue because the packaging got split in certain ways >>>> >>>> No it did not, because the ftpmaster rejected the upload since it ships >>>> with minified javascript which is not considered modifiable source code. >>>> And the old version has now been removed from Debian because it was >>>> unmaintainable. >>>> >>>> So I hope #5639 will be resolved at some point. Note that Debian doesn't >>>> require the javascript to be minified during package build, just that >>>> the source would ship the unminified copy as well. >>> >>> Turns out it wasn't too much of an effort to pull in unminified bits of >>> everything that is shipped minified (just ~630kB..), so I guess Freeipa >>> will be uploaded back fairly soon... >> >> Timo, >> >> can you share script/procedure you used? It would save us some time spent on >> re-inventing what you have done :-) >> >> We need to see how complex change it would be so we could pull it into master >> eventually. > > I put it in https://fedorahosted.org/freeipa/ticket/5639 > > for dojo & build I looked at the profile.js files. But now I see that I > didn't look at webui.profile.js... could be something is missing still. well, at least the Debian ftpmaster was happy enough to accept 4.3.2-1 to experimental with the new list, so the exile didn't last for too long. -- t From natxo.asenjo at gmail.com Fri Sep 16 07:38:18 2016 From: natxo.asenjo at gmail.com (Natxo Asenjo) Date: Fri, 16 Sep 2016 09:38:18 +0200 Subject: [Freeipa-users] certificates not renewing CA_UNREACHEABLE In-Reply-To: References: <806bdf32-558c-4972-255a-0901f47c4c4b@redhat.com> <6596593a-9470-bebd-f236-f9d17aa4a56c@redhat.com> Message-ID: hi, On Thu, Sep 15, 2016 at 1:03 PM, Natxo Asenjo > > On Thu, Sep 15, 2016 at 12:49 PM, Martin Basti wrote: > >> >> >> On 15.09.2016 12:44, Natxo Asenjo wrote: >> >> hi, >> >> On Thu, Sep 15, 2016 at 12:33 PM, Martin Basti wrote: >> >>> >>> Hello, >>> >>> usually the most information can be found here >>> /var/log/pki/pki-tomcat/ca/debug >>> >> >> mmm, in this centos 6.8 system that does not exist: >> >> # ls -l /var/log/pki/pki-tomcat/ca/debug >> ls: cannot access /var/log/pki/pki-tomcat/ca/debug: No such file or >> directory >> >> >> I do have a /var/log/pki-ca/debug >> >> >> >> Does it contain any information related to your issue? >> > > I have tried renewing the certificate: > > ipa-getcert resubmit -i 20121107212513 > > > If I grep that file for that request id I find nothing recent, just in the > ipaserver installation log > > # cd /var/log > # grep -ri 20121107212513 *.log > ipaserver-install.log:2012-11-07T21:25:13Z DEBUG stdout=New tracking > request "20121107212513" added. > > # grep -ri 20121107212513 pki-ca > # > > Any clues? -- Groeten, natxo -------------- next part -------------- An HTML attachment was scrubbed... URL: From natxo.asenjo at gmail.com Fri Sep 16 07:39:58 2016 From: natxo.asenjo at gmail.com (Natxo Asenjo) Date: Fri, 16 Sep 2016 09:39:58 +0200 Subject: [Freeipa-users] ipa: ERROR: Certificate format error: (SEC_ERROR_LEGACY_DATABASE) The certificate/key database is in an old, unsupported format. In-Reply-To: References: <57D015D8.7080507@redhat.com> <57D166B9.3070201@redhat.com> <57D70676.8080107@redhat.com> <57D81AA1.4050400@redhat.com> <4eac5df5-ded2-8705-d5ba-6ab9dde5ad4d@redhat.com> Message-ID: hi, On Thu, Sep 15, 2016 at 2:25 PM, Natxo Asenjo wrote: > hi, > > attached error_log > Any clues? Thanks! -- -- Groeten, natxo -------------- next part -------------- An HTML attachment was scrubbed... URL: From tbordaz at redhat.com Fri Sep 16 08:29:43 2016 From: tbordaz at redhat.com (thierry bordaz) Date: Fri, 16 Sep 2016 10:29:43 +0200 Subject: [Freeipa-users] Freeipa 4.2.0 hangs intermittently In-Reply-To: References: <57CD1FDC.2010109@redhat.com> <57CD5488.60404@redhat.com> <57CD659B.6050707@redhat.com> <57CD68BD.9030802@redhat.com> <57CE8CDB.6000809@redhat.com> <57CE9DE5.6020301@redhat.com> <57DACE76.5070803@redhat.com> Message-ID: <57DBAD77.8090308@redhat.com> Hi Rakesh, Thank you so much for those precious feedback. You are right I did test 4.3 while you were testing 4.2. Upgrading is a good option but regarding the suspected bug it should not have impact but the dynamic of the install is likely quite different. I do not know from where is coming this flow of Clock skew messages. Is ntpd still running ? At the time of the hang, the provided pstacks contained some SRCHs and sometime ADD. Do you remember the kind of search/update you did apply when the hang occurred ? From server or client ? Also after configuring client, did you promote them to replica or just keep them as client ? thanks thierry On 09/15/2016 09:14 PM, Rakesh Rajasekharan wrote: > Hi Thierry, > > I had been doing pretty much the same to reproduce the hang.. > After changing the checkpoint interval.. the hangs had become > manageable.. > > However, this time when the hang occurred.. I did not add any clients > and also the behavior was bit different. This time my ns-slapd did not > hang.. ldapsearch was working fine.. but, the ipa commands were > taking a long time to respond and most users were not able to login to > any of the clients.. > restarting ipa did not help either. > > I was seeing around 14-15 connections in SYN_RECV state continously > and /var.log/messages has entries like "TCP: request_sock_TCP: > Possible SYN flooding on port 88. Sending cookies. Check SNMP counter" > > I also see in the /var/log/krb5kdc.log this error > > ipa-master-xyz.com krb5kdc[15308](info): > TGS_REQ (4 etypes {18 17 16 23}) 10.1.12.155 : > PROCESS_TGS: authtime 0, host/10.1.12.155 at XYZ.COM > for ldap/ipa-master-xyz.com at XYZ.COM > , Clock skew too great > > there are arounf 7.5k messages like that.. which I guess is too high > grep "krbtgt/XYZ.COM at XYZ.COM , Clock skew too > great" /var/log/krb5kdc.log | wc -l > 7570 > > Would this give rise to a hang or make IPA nonresponsive.. > > The kinit admin command itself was taking an unusually long time to > respond.. > > KRB5_TRACE=/dev/stderr kinit admin > [11298] 1473702491.60880: Getting initial credentials for > admin at XYZ.COM > [11298] 1473702491.62981: Sending request (167 bytes) to XYZ.COM > > [11298] 1473702491.63119: Initiating TCP connection to stream > 10.1.3.35:88 > [11298] 1473702491.63359: Sending TCP request to stream 10.1.3.35:88 > > [11298] 1473702493.797835: Received answer (341 bytes) from stream > 10.1.3.35:88 > [11298] 1473702493.797848: Terminating TCP connection to stream > 10.1.3.35:88 > [11298] 1473702493.797911: Response was from master KDC > [11298] 1473702493.797956: Received error from KDC: > -1765328359/Additional pre-authentication required > [11298] 1473702493.797993: Processing preauth types: 136, 19, 2, 133 > [11298] 1473702493.798005: Selected etype info: etype aes256-cts, salt > "V at Cbu147E#1;R0WD", params "" > [11298] 1473702493.798009: Received cookie: MIT > Password for admin at XYZ.COM : > [11298] 1473702498.190064: AS key obtained for encrypted timestamp: > aes256-cts/2C9D > [11298] 1473702498.190109: Encrypted timestamp (for > 1473702498.184527): plain > 301AA011180F32303136303931323137343831385AA105020302D0CF, encrypted > 25FC8D37EFB6B7837C8D5C6649DFB9972010D40EE29D1222FBA45CAA98428E42C7FCC9B7FE881A04BD3390A6A9EDE9D2D93729FDF3E47B6D > [11298] 1473702498.190129: Preauth module encrypted_timestamp (2) > (real) returned: 0/Success > [11298] 1473702498.190133: Produced preauth for next request: 133, 2 > [11298] 1473702498.190148: Sending request (261 bytes) to XYZ.COM > > [11298] 1473702498.190246: Initiating TCP connection to stream > 10.1.3.35:88 > [11298] 1473702499.191933: Sending initial UDP request to dgram > 10.1.3.35:88 > [11298] 1473702502.195157: Sending retry UDP request to dgram > 10.1.3.35:88 > [11298] 1473702507.200405: Sending retry UDP request to dgram > 10.1.3.35:88 > [11298] 1473702513.226371: Sending TCP request to stream 10.1.3.35:88 > > [11298] 1473702515.797243: Received answer (730 bytes) from stream > 10.1.3.35:88 > [11298] 1473702515.797271: Terminating TCP connection to stream > 10.1.3.35:88 > [11298] 1473702515.797326: Response was from master KDC > [11298] 1473702515.797353: Processing preauth types: 19 > [11298] 1473702515.797360: Selected etype info: etype aes256-cts, salt > "V at Cbu147E#1;R0WD", params "" > [11298] 1473702515.797394: Produced preauth for next request: (empty) > [11298] 1473702515.797401: AS key determined by preauth: aes256-cts/2C9D > [11298] 1473702515.797445: Decrypted AS reply; session key is: > aes256-cts/702E > [11298] 1473702515.797460: FAST negotiation: available > [11298] 1473702515.797478: Initializing KEYRING:persistent:0:0 with > default princ admin at XYZ.COM > [11298] 1473702515.797534: Storing admin at XYZ.COM > -> krbtgt/XYZ.COM at XYZ.COM > in KEYRING:persistent:0:0 > [11298] 1473702515.797572: Storing config in KEYRING:persistent:0:0 > for krbtgt/XYZ.COM at XYZ.COM : fast_avail: yes > [11298] 1473702515.797585: Storing admin at XYZ.COM > -> > krb5_ccache_conf_data/fast_avail/krbtgt\/XYZ.COM > \@XYZ.COM at X-CACHECONF: in KEYRING:persistent:0:0 > [11298] 1473702515.797631: Storing config in KEYRING:persistent:0:0 > for krbtgt/XYZ.COM at XYZ.COM : pa_type: 2 > [11298] 1473702515.797647: Storing admin at XYZ.COM > -> > krb5_ccache_conf_data/pa_type/krbtgt\/XYZ.COM > \@XYZ.COM at X-CACHECONF: in KEYRING:persistent:0:0 > > > > Could it be an version specific issue..as I am running 4.2.0 verison.. > and yours is 4.3.2 > > If so, I will try to upgrade my IPA version > > Thanks > Rakesh > > > > > > > On Thu, Sep 15, 2016 at 10:08 PM, thierry bordaz > wrote: > > Hello Rakesh, > > Sorry to be back so late. I finally manage to try to reproduce > locally the hang of DS... and I fail ;-) > > I am running 4 VM, with freeipa 4.3.2. > One is running a server (where I provisioned some users), with > 200Mb cache. The server receives no direct load (SRCH or update). > The tree others are running in a loop ipa-client-install and > uninstall. So far it has run for several hours without hang. > > Going through the mail thread I have not found others details. > Am I missing something to reproduce ? > > thanks in advance for your help > thierry > > On 09/07/2016 01:46 PM, Rakesh Rajasekharan wrote: >> I changed the nsslapd-db-checkpoint interval first to 80 and >> increased till 160.. however the hang has not gone away yet >> >> >> I migrated my data from a openldap, can that cause issues like >> this.. If so, i could try creating users from scratch directly on >> IPA rather than migrating and test it >> >> But before that , have taken a fresh pstack just incase if it >> helps to nail down the issue >> >> Thread 44 (Thread 0x7f53ed52f700 (LWP 128165)): >> #0 0x00007f54047649b3 in select () from /lib64/libc.so.6 >> #1 0x00007f5406eef0e9 in DS_Sleep () from >> /usr/lib64/dirsrv/libslapd.so.0 >> #2 0x00007f53fa53d907 in deadlock_threadmain () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #3 0x00007f540509e7bb in _pt_root () from /lib64/libnspr4.so >> #4 0x00007f5404a3fdc5 in start_thread () from /lib64/libpthread.so.0 >> #5 0x00007f540476d28d in clone () from /lib64/libc.so.6 >> Thread 43 (Thread 0x7f53ecd2e700 (LWP 128166)): >> #0 0x00007f54047649b3 in select () from /lib64/libc.so.6 >> #1 0x00007f53ff0cbbad in __os_yield () from /lib64/libdb-5.3.so >> >> #2 0x00007f53ff0c72b3 in __memp_sync_int () from >> /lib64/libdb-5.3.so >> #3 0x00007f53ff0d7752 in __txn_checkpoint () from >> /lib64/libdb-5.3.so >> #4 0x00007f53ff0d7b74 in __txn_checkpoint_pp () from >> /lib64/libdb-5.3.so >> #5 0x00007f53fa541a87 in checkpoint_threadmain () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #6 0x00007f540509e7bb in _pt_root () from /lib64/libnspr4.so >> #7 0x00007f5404a3fdc5 in start_thread () from /lib64/libpthread.so.0 >> #8 0x00007f540476d28d in clone () from /lib64/libc.so.6 >> Thread 42 (Thread 0x7f53ec52d700 (LWP 128167)): >> #0 0x00007f54047649b3 in select () from /lib64/libc.so.6 >> #1 0x00007f5406eef0e9 in DS_Sleep () from >> /usr/lib64/dirsrv/libslapd.so.0 >> #2 0x00007f53fa53db7f in trickle_threadmain () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #3 0x00007f540509e7bb in _pt_root () from /lib64/libnspr4.so >> #4 0x00007f5404a3fdc5 in start_thread () from /lib64/libpthread.so.0 >> #5 0x00007f540476d28d in clone () from /lib64/libc.so.6 >> Thread 41 (Thread 0x7f53ebd2c700 (LWP 128168)): >> #0 0x00007f54047649b3 in select () from /lib64/libc.so.6 >> #1 0x00007f5406eef0e9 in DS_Sleep () from >> /usr/lib64/dirsrv/libslapd.so.0 >> #2 0x00007f53fa538707 in perf_threadmain () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #3 0x00007f540509e7bb in _pt_root () from /lib64/libnspr4.so >> #4 0x00007f5404a3fdc5 in start_thread () from /lib64/libpthread.so.0 >> #5 0x00007f540476d28d in clone () from /lib64/libc.so.6 >> Thread 40 (Thread 0x7f53eb322700 (LWP 128204)): >> #0 0x00007f5404a436d5 in pthread_cond_wait@@GLIBC_2.3.2 >> () from >> /lib64/libpthread.so.0 >> #1 0x00007f5405099050 in PR_WaitCondVar () from /lib64/libnspr4.so >> #2 0x00007f5406ede198 in slapi_wait_condvar () from >> /usr/lib64/dirsrv/libslapd.so.0 >> #3 0x00007f53fd27062e in cos_cache_wait_on_change () from >> /usr/lib64/dirsrv/plugins/libcos-plugin.so >> #4 0x00007f540509e7bb in _pt_root () from /lib64/libnspr4.so >> #5 0x00007f5404a3fdc5 in start_thread () from /lib64/libpthread.so.0 >> #6 0x00007f540476d28d in clone () from /lib64/libc.so.6 >> Thread 39 (Thread 0x7f53eab21700 (LWP 128206)): >> #0 0x00007f5404a436d5 in pthread_cond_wait@@GLIBC_2.3.2 >> () from >> /lib64/libpthread.so.0 >> #1 0x00007f5405099050 in PR_WaitCondVar () from /lib64/libnspr4.so >> #2 0x00007f5406ede198 in slapi_wait_condvar () from >> /usr/lib64/dirsrv/libslapd.so.0 >> #3 0x00007f53f8bdbeed in roles_cache_wait_on_change () from >> /usr/lib64/dirsrv/plugins/libroles-plugin.so >> #4 0x00007f540509e7bb in _pt_root () from /lib64/libnspr4.so >> #5 0x00007f5404a3fdc5 in start_thread () from /lib64/libpthread.so.0 >> #6 0x00007f540476d28d in clone () from /lib64/libc.so.6 >> Thread 38 (Thread 0x7f53ea320700 (LWP 128207)): >> #0 0x00007f5404a436d5 in pthread_cond_wait@@GLIBC_2.3.2 >> () from >> /lib64/libpthread.so.0 >> #1 0x00007f5405099050 in PR_WaitCondVar () from /lib64/libnspr4.so >> #2 0x00007f5406ede198 in slapi_wait_condvar () from >> /usr/lib64/dirsrv/libslapd.so.0 >> #3 0x00007f53f8bdbeed in roles_cache_wait_on_change () from >> /usr/lib64/dirsrv/plugins/libroles-plugin.so >> #4 0x00007f540509e7bb in _pt_root () from /lib64/libnspr4.so >> #5 0x00007f5404a3fdc5 in start_thread () from /lib64/libpthread.so.0 >> #6 0x00007f540476d28d in clone () from /lib64/libc.so.6 >> Thread 37 (Thread 0x7f53e9b1f700 (LWP 128208)): >> #0 0x00007f5404a436d5 in pthread_cond_wait@@GLIBC_2.3.2 >> () from >> /lib64/libpthread.so.0 >> #1 0x00007f5405099050 in PR_WaitCondVar () from /lib64/libnspr4.so >> #2 0x00007f5406ede198 in slapi_wait_condvar () from >> /usr/lib64/dirsrv/libslapd.so.0 >> #3 0x00007f53f8bdbeed in roles_cache_wait_on_change () from >> /usr/lib64/dirsrv/plugins/libroles-plugin.so >> #4 0x00007f540509e7bb in _pt_root () from /lib64/libnspr4.so >> #5 0x00007f5404a3fdc5 in start_thread () from /lib64/libpthread.so.0 >> #6 0x00007f540476d28d in clone () from /lib64/libc.so.6 >> Thread 36 (Thread 0x7f53e931e700 (LWP 128209)): >> #0 0x00007f5404a43a82 in pthread_cond_timedwait@@GLIBC_2.3.2 >> () from >> /lib64/libpthread.so.0 >> #1 0x00007f5405098b07 in pt_TimedWait () from /lib64/libnspr4.so >> #2 0x00007f5405098fce in PR_WaitCondVar () from /lib64/libnspr4.so >> #3 0x00007f5407397d03 in housecleaning () >> #4 0x00007f540509e7bb in _pt_root () from /lib64/libnspr4.so >> #5 0x00007f5404a3fdc5 in start_thread () from /lib64/libpthread.so.0 >> #6 0x00007f540476d28d in clone () from /lib64/libc.so.6 >> Thread 35 (Thread 0x7f53e8b1d700 (LWP 128210)): >> #0 0x00007f5404a43a82 in pthread_cond_timedwait@@GLIBC_2.3.2 >> () from >> /lib64/libpthread.so.0 >> #1 0x00007f5405098b07 in pt_TimedWait () from /lib64/libnspr4.so >> #2 0x00007f5405098fce in PR_WaitCondVar () from /lib64/libnspr4.so >> #3 0x00007f5406e841e8 in eq_loop () from >> /usr/lib64/dirsrv/libslapd.so.0 >> #4 0x00007f540509e7bb in _pt_root () from /lib64/libnspr4.so >> #5 0x00007f5404a3fdc5 in start_thread () from /lib64/libpthread.so.0 >> #6 0x00007f540476d28d in clone () from /lib64/libc.so.6 >> Thread 34 (Thread 0x7f53db7fe700 (LWP 128212)): >> #0 0x00007f5404a436d5 in pthread_cond_wait@@GLIBC_2.3.2 >> () from >> /lib64/libpthread.so.0 >> #1 0x00007f53fef9f2f3 in __db_hybrid_mutex_suspend () from >> /lib64/libdb-5.3.so >> #2 0x00007f53fef9e640 in __db_tas_mutex_lock () from >> /lib64/libdb-5.3.so >> #3 0x00007f53ff048d3a in __lock_get_internal () from >> /lib64/libdb-5.3.so >> #4 0x00007f53ff049820 in __lock_get () from /lib64/libdb-5.3.so >> >> #5 0x00007f53ff075142 in __db_lget () from /lib64/libdb-5.3.so >> >> #6 0x00007f53fefbc605 in __bam_search () from >> /lib64/libdb-5.3.so >> #7 0x00007f53fefa7256 in __bamc_search () from >> /lib64/libdb-5.3.so >> #8 0x00007f53fefa8d0f in __bamc_get () from /lib64/libdb-5.3.so >> >> #9 0x00007f53ff061ca6 in __dbc_iget () from /lib64/libdb-5.3.so >> >> #10 0x00007f53ff070b02 in __dbc_get_pp () from >> /lib64/libdb-5.3.so >> #11 0x00007f53fa54c680 in idl_new_fetch () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #12 0x00007f53fa55ab26 in index_read_ext_allids () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #13 0x00007f53fa545284 in keys2idl () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #14 0x00007f53fa5459e3 in ava_candidates.isra.0 () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #15 0x00007f53fa545fd2 in filter_candidates_ext () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #16 0x00007f53fa547096 in list_candidates () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #17 0x00007f53fa545f40 in filter_candidates_ext () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #18 0x00007f53fa58180c in subtree_candidates () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #19 0x00007f53fa582e7f in ldbm_back_search () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #20 0x00007f5406eb01d6 in op_shared_search () from >> /usr/lib64/dirsrv/libslapd.so.0 >> #21 0x00007f54073a133e in do_search () >> #22 0x00007f540738f567 in connection_threadmain () >> #23 0x00007f540509e7bb in _pt_root () from /lib64/libnspr4.so >> #24 0x00007f5404a3fdc5 in start_thread () from /lib64/libpthread.so.0 >> #25 0x00007f540476d28d in clone () from /lib64/libc.so.6 >> Thread 33 (Thread 0x7f53daffd700 (LWP 128213)): >> #0 0x00007f5404a436d5 in pthread_cond_wait@@GLIBC_2.3.2 >> () from >> /lib64/libpthread.so.0 >> #1 0x00007f53fef9f2f3 in __db_hybrid_mutex_suspend () from >> /lib64/libdb-5.3.so >> #2 0x00007f53fef9e640 in __db_tas_mutex_lock () from >> /lib64/libdb-5.3.so >> #3 0x00007f53ff048d3a in __lock_get_internal () from >> /lib64/libdb-5.3.so >> #4 0x00007f53ff049820 in __lock_get () from /lib64/libdb-5.3.so >> >> #5 0x00007f53ff075142 in __db_lget () from /lib64/libdb-5.3.so >> >> #6 0x00007f53fefbc605 in __bam_search () from >> /lib64/libdb-5.3.so >> #7 0x00007f53fefa7256 in __bamc_search () from >> /lib64/libdb-5.3.so >> #8 0x00007f53fefa8d0f in __bamc_get () from /lib64/libdb-5.3.so >> >> #9 0x00007f53ff061ca6 in __dbc_iget () from /lib64/libdb-5.3.so >> >> #10 0x00007f53ff070b02 in __dbc_get_pp () from >> /lib64/libdb-5.3.so >> #11 0x00007f53fa54c680 in idl_new_fetch () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #12 0x00007f53fa55ab26 in index_read_ext_allids () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #13 0x00007f53fa545284 in keys2idl () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #14 0x00007f53fa5459e3 in ava_candidates.isra.0 () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #15 0x00007f53fa545fd2 in filter_candidates_ext () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #16 0x00007f53fa547096 in list_candidates () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #17 0x00007f53fa545f40 in filter_candidates_ext () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #18 0x00007f53fa58180c in subtree_candidates () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #19 0x00007f53fa582e7f in ldbm_back_search () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #20 0x00007f5406eb01d6 in op_shared_search () from >> /usr/lib64/dirsrv/libslapd.so.0 >> #21 0x00007f54073a133e in do_search () >> #22 0x00007f540738f567 in connection_threadmain () >> #23 0x00007f540509e7bb in _pt_root () from /lib64/libnspr4.so >> #24 0x00007f5404a3fdc5 in start_thread () from /lib64/libpthread.so.0 >> #25 0x00007f540476d28d in clone () from /lib64/libc.so.6 >> Thread 32 (Thread 0x7f53da7fc700 (LWP 128214)): >> #0 0x00007f5404a436d5 in pthread_cond_wait@@GLIBC_2.3.2 >> () from >> /lib64/libpthread.so.0 >> #1 0x00007f53fef9f2f3 in __db_hybrid_mutex_suspend () from >> /lib64/libdb-5.3.so >> #2 0x00007f53fef9e640 in __db_tas_mutex_lock () from >> /lib64/libdb-5.3.so >> #3 0x00007f53ff048d3a in __lock_get_internal () from >> /lib64/libdb-5.3.so >> #4 0x00007f53ff049820 in __lock_get () from /lib64/libdb-5.3.so >> >> #5 0x00007f53ff075142 in __db_lget () from /lib64/libdb-5.3.so >> >> #6 0x00007f53fefbc605 in __bam_search () from >> /lib64/libdb-5.3.so >> #7 0x00007f53fefa7256 in __bamc_search () from >> /lib64/libdb-5.3.so >> #8 0x00007f53fefa8d0f in __bamc_get () from /lib64/libdb-5.3.so >> >> #9 0x00007f53ff061ca6 in __dbc_iget () from /lib64/libdb-5.3.so >> >> #10 0x00007f53ff070b02 in __dbc_get_pp () from >> /lib64/libdb-5.3.so >> #11 0x00007f53fa54c680 in idl_new_fetch () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #12 0x00007f53fa55ab26 in index_read_ext_allids () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #13 0x00007f53fa545284 in keys2idl () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #14 0x00007f53fa5459e3 in ava_candidates.isra.0 () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #15 0x00007f53fa545fd2 in filter_candidates_ext () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #16 0x00007f53fa547096 in list_candidates () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #17 0x00007f53fa545f40 in filter_candidates_ext () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #18 0x00007f53fa547096 in list_candidates () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #19 0x00007f53fa545f40 in filter_candidates_ext () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #20 0x00007f53fa547096 in list_candidates () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #21 0x00007f53fa545f40 in filter_candidates_ext () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #22 0x00007f53fa58180c in subtree_candidates () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #23 0x00007f53fa582e7f in ldbm_back_search () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #24 0x00007f5406eb01d6 in op_shared_search () from >> /usr/lib64/dirsrv/libslapd.so.0 >> #25 0x00007f54073a133e in do_search () >> #26 0x00007f540738f3c3 in connection_threadmain () >> #27 0x00007f540509e7bb in _pt_root () from /lib64/libnspr4.so >> #28 0x00007f5404a3fdc5 in start_thread () from /lib64/libpthread.so.0 >> #29 0x00007f540476d28d in clone () from /lib64/libc.so.6 >> Thread 31 (Thread 0x7f53d9ffb700 (LWP 128215)): >> #0 0x00007f5404a436d5 in pthread_cond_wait@@GLIBC_2.3.2 >> () from >> /lib64/libpthread.so.0 >> #1 0x00007f53fef9f2f3 in __db_hybrid_mutex_suspend () from >> /lib64/libdb-5.3.so >> #2 0x00007f53fef9e640 in __db_tas_mutex_lock () from >> /lib64/libdb-5.3.so >> #3 0x00007f53ff048d3a in __lock_get_internal () from >> /lib64/libdb-5.3.so >> #4 0x00007f53ff049820 in __lock_get () from /lib64/libdb-5.3.so >> >> #5 0x00007f53ff075142 in __db_lget () from /lib64/libdb-5.3.so >> >> #6 0x00007f53fefbc605 in __bam_search () from >> /lib64/libdb-5.3.so >> #7 0x00007f53fefa7256 in __bamc_search () from >> /lib64/libdb-5.3.so >> #8 0x00007f53fefa8d0f in __bamc_get () from /lib64/libdb-5.3.so >> >> #9 0x00007f53ff061ca6 in __dbc_iget () from /lib64/libdb-5.3.so >> >> #10 0x00007f53ff070b02 in __dbc_get_pp () from >> /lib64/libdb-5.3.so >> #11 0x00007f53fa54c680 in idl_new_fetch () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #12 0x00007f53fa55ab26 in index_read_ext_allids () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #13 0x00007f53fa545284 in keys2idl () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #14 0x00007f53fa5459e3 in ava_candidates.isra.0 () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #15 0x00007f53fa545fd2 in filter_candidates_ext () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #16 0x00007f53fa547096 in list_candidates () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #17 0x00007f53fa545f40 in filter_candidates_ext () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #18 0x00007f53fa58180c in subtree_candidates () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #19 0x00007f53fa582e7f in ldbm_back_search () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #20 0x00007f5406eb01d6 in op_shared_search () from >> /usr/lib64/dirsrv/libslapd.so.0 >> #21 0x00007f54073a133e in do_search () >> #22 0x00007f540738f567 in connection_threadmain () >> #23 0x00007f540509e7bb in _pt_root () from /lib64/libnspr4.so >> #24 0x00007f5404a3fdc5 in start_thread () from /lib64/libpthread.so.0 >> #25 0x00007f540476d28d in clone () from /lib64/libc.so.6 >> Thread 30 (Thread 0x7f53d97fa700 (LWP 128216)): >> #0 0x00007f5404a436d5 in pthread_cond_wait@@GLIBC_2.3.2 >> () from >> /lib64/libpthread.so.0 >> #1 0x00007f53fef9f2f3 in __db_hybrid_mutex_suspend () from >> /lib64/libdb-5.3.so >> #2 0x00007f53fef9e640 in __db_tas_mutex_lock () from >> /lib64/libdb-5.3.so >> #3 0x00007f53ff048d3a in __lock_get_internal () from >> /lib64/libdb-5.3.so >> #4 0x00007f53ff049820 in __lock_get () from /lib64/libdb-5.3.so >> >> #5 0x00007f53ff075142 in __db_lget () from /lib64/libdb-5.3.so >> >> #6 0x00007f53fefbc605 in __bam_search () from >> /lib64/libdb-5.3.so >> #7 0x00007f53fefa7256 in __bamc_search () from >> /lib64/libdb-5.3.so >> #8 0x00007f53fefa8d0f in __bamc_get () from /lib64/libdb-5.3.so >> >> #9 0x00007f53ff061ca6 in __dbc_iget () from /lib64/libdb-5.3.so >> >> #10 0x00007f53ff070b02 in __dbc_get_pp () from >> /lib64/libdb-5.3.so >> #11 0x00007f53fa54c680 in idl_new_fetch () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #12 0x00007f53fa55ab26 in index_read_ext_allids () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #13 0x00007f53fa545284 in keys2idl () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #14 0x00007f53fa5459e3 in ava_candidates.isra.0 () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #15 0x00007f53fa545fd2 in filter_candidates_ext () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #16 0x00007f53fa547096 in list_candidates () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #17 0x00007f53fa545f40 in filter_candidates_ext () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #18 0x00007f53fa58180c in subtree_candidates () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #19 0x00007f53fa582e7f in ldbm_back_search () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #20 0x00007f5406eb01d6 in op_shared_search () from >> /usr/lib64/dirsrv/libslapd.so.0 >> #21 0x00007f54073a133e in do_search () >> #22 0x00007f540738f567 in connection_threadmain () >> #23 0x00007f540509e7bb in _pt_root () from /lib64/libnspr4.so >> #24 0x00007f5404a3fdc5 in start_thread () from /lib64/libpthread.so.0 >> #25 0x00007f540476d28d in clone () from /lib64/libc.so.6 >> Thread 29 (Thread 0x7f53d8ff9700 (LWP 128217)): >> #0 0x00007f5404a436d5 in pthread_cond_wait@@GLIBC_2.3.2 >> () from >> /lib64/libpthread.so.0 >> #1 0x00007f53fef9f2f3 in __db_hybrid_mutex_suspend () from >> /lib64/libdb-5.3.so >> #2 0x00007f53fef9e640 in __db_tas_mutex_lock () from >> /lib64/libdb-5.3.so >> #3 0x00007f53ff048d3a in __lock_get_internal () from >> /lib64/libdb-5.3.so >> #4 0x00007f53ff049820 in __lock_get () from /lib64/libdb-5.3.so >> >> #5 0x00007f53ff075142 in __db_lget () from /lib64/libdb-5.3.so >> >> #6 0x00007f53fefbc605 in __bam_search () from >> /lib64/libdb-5.3.so >> #7 0x00007f53fefa7256 in __bamc_search () from >> /lib64/libdb-5.3.so >> #8 0x00007f53fefa8d0f in __bamc_get () from /lib64/libdb-5.3.so >> >> #9 0x00007f53ff061ca6 in __dbc_iget () from /lib64/libdb-5.3.so >> >> #10 0x00007f53ff070b02 in __dbc_get_pp () from >> /lib64/libdb-5.3.so >> #11 0x00007f53fa54c680 in idl_new_fetch () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #12 0x00007f53fa55ab26 in index_read_ext_allids () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #13 0x00007f53fa545284 in keys2idl () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #14 0x00007f53fa5459e3 in ava_candidates.isra.0 () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #15 0x00007f53fa545fd2 in filter_candidates_ext () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #16 0x00007f53fa547096 in list_candidates () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #17 0x00007f53fa545f40 in filter_candidates_ext () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #18 0x00007f53fa58180c in subtree_candidates () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #19 0x00007f53fa582e7f in ldbm_back_search () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #20 0x00007f5406eb01d6 in op_shared_search () from >> /usr/lib64/dirsrv/libslapd.so.0 >> #21 0x00007f54073a133e in do_search () >> #22 0x00007f540738f567 in connection_threadmain () >> #23 0x00007f540509e7bb in _pt_root () from /lib64/libnspr4.so >> #24 0x00007f5404a3fdc5 in start_thread () from /lib64/libpthread.so.0 >> #25 0x00007f540476d28d in clone () from /lib64/libc.so.6 >> Thread 28 (Thread 0x7f53d87f8700 (LWP 128218)): >> #0 0x00007f5404a436d5 in pthread_cond_wait@@GLIBC_2.3.2 >> () from >> /lib64/libpthread.so.0 >> #1 0x00007f53fef9f2f3 in __db_hybrid_mutex_suspend () from >> /lib64/libdb-5.3.so >> #2 0x00007f53fef9e640 in __db_tas_mutex_lock () from >> /lib64/libdb-5.3.so >> #3 0x00007f53ff048d3a in __lock_get_internal () from >> /lib64/libdb-5.3.so >> #4 0x00007f53ff049820 in __lock_get () from /lib64/libdb-5.3.so >> >> #5 0x00007f53ff075142 in __db_lget () from /lib64/libdb-5.3.so >> >> #6 0x00007f53fefbc605 in __bam_search () from >> /lib64/libdb-5.3.so >> #7 0x00007f53fefa7256 in __bamc_search () from >> /lib64/libdb-5.3.so >> #8 0x00007f53fefa8d0f in __bamc_get () from /lib64/libdb-5.3.so >> >> #9 0x00007f53ff061ca6 in __dbc_iget () from /lib64/libdb-5.3.so >> >> #10 0x00007f53ff070b02 in __dbc_get_pp () from >> /lib64/libdb-5.3.so >> #11 0x00007f53fa54c680 in idl_new_fetch () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #12 0x00007f53fa55ab26 in index_read_ext_allids () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #13 0x00007f53fa545284 in keys2idl () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #14 0x00007f53fa5459e3 in ava_candidates.isra.0 () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #15 0x00007f53fa545fd2 in filter_candidates_ext () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #16 0x00007f53fa547096 in list_candidates () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #17 0x00007f53fa545f40 in filter_candidates_ext () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #18 0x00007f53fa58180c in subtree_candidates () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #19 0x00007f53fa582e7f in ldbm_back_search () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #20 0x00007f5406eb01d6 in op_shared_search () from >> /usr/lib64/dirsrv/libslapd.so.0 >> #21 0x00007f54073a133e in do_search () >> #22 0x00007f540738f567 in connection_threadmain () >> #23 0x00007f540509e7bb in _pt_root () from /lib64/libnspr4.so >> #24 0x00007f5404a3fdc5 in start_thread () from /lib64/libpthread.so.0 >> #25 0x00007f540476d28d in clone () from /lib64/libc.so.6 >> Thread 27 (Thread 0x7f53d7ff7700 (LWP 128219)): >> #0 0x00007f5404a436d5 in pthread_cond_wait@@GLIBC_2.3.2 >> () from >> /lib64/libpthread.so.0 >> #1 0x00007f53fef9f2f3 in __db_hybrid_mutex_suspend () from >> /lib64/libdb-5.3.so >> #2 0x00007f53fef9e640 in __db_tas_mutex_lock () from >> /lib64/libdb-5.3.so >> #3 0x00007f53ff048d3a in __lock_get_internal () from >> /lib64/libdb-5.3.so >> #4 0x00007f53ff049820 in __lock_get () from /lib64/libdb-5.3.so >> >> #5 0x00007f53ff075142 in __db_lget () from /lib64/libdb-5.3.so >> >> #6 0x00007f53fefbc605 in __bam_search () from >> /lib64/libdb-5.3.so >> #7 0x00007f53fefa7256 in __bamc_search () from >> /lib64/libdb-5.3.so >> #8 0x00007f53fefa8d0f in __bamc_get () from /lib64/libdb-5.3.so >> >> #9 0x00007f53ff061ca6 in __dbc_iget () from /lib64/libdb-5.3.so >> >> #10 0x00007f53ff070b02 in __dbc_get_pp () from >> /lib64/libdb-5.3.so >> #11 0x00007f53fa54c680 in idl_new_fetch () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #12 0x00007f53fa55ab26 in index_read_ext_allids () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #13 0x00007f53fa545284 in keys2idl () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #14 0x00007f53fa5459e3 in ava_candidates.isra.0 () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #15 0x00007f53fa545fd2 in filter_candidates_ext () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #16 0x00007f53fa547096 in list_candidates () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #17 0x00007f53fa545f40 in filter_candidates_ext () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #18 0x00007f53fa58180c in subtree_candidates () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #19 0x00007f53fa582e7f in ldbm_back_search () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #20 0x00007f5406eb01d6 in op_shared_search () from >> /usr/lib64/dirsrv/libslapd.so.0 >> #21 0x00007f54073a133e in do_search () >> #22 0x00007f540738f567 in connection_threadmain () >> #23 0x00007f540509e7bb in _pt_root () from /lib64/libnspr4.so >> #24 0x00007f5404a3fdc5 in start_thread () from /lib64/libpthread.so.0 >> #25 0x00007f540476d28d in clone () from /lib64/libc.so.6 >> Thread 26 (Thread 0x7f53d77f6700 (LWP 128220)): >> #0 0x00007f5404a436d5 in pthread_cond_wait@@GLIBC_2.3.2 >> () from >> /lib64/libpthread.so.0 >> #1 0x00007f53fef9f2f3 in __db_hybrid_mutex_suspend () from >> /lib64/libdb-5.3.so >> #2 0x00007f53fef9e640 in __db_tas_mutex_lock () from >> /lib64/libdb-5.3.so >> #3 0x00007f53ff048d3a in __lock_get_internal () from >> /lib64/libdb-5.3.so >> #4 0x00007f53ff049820 in __lock_get () from /lib64/libdb-5.3.so >> >> #5 0x00007f53ff075142 in __db_lget () from /lib64/libdb-5.3.so >> >> #6 0x00007f53fefbc605 in __bam_search () from >> /lib64/libdb-5.3.so >> #7 0x00007f53fefa7256 in __bamc_search () from >> /lib64/libdb-5.3.so >> #8 0x00007f53fefa8d0f in __bamc_get () from /lib64/libdb-5.3.so >> >> #9 0x00007f53ff061ca6 in __dbc_iget () from /lib64/libdb-5.3.so >> >> #10 0x00007f53ff070b02 in __dbc_get_pp () from >> /lib64/libdb-5.3.so >> #11 0x00007f53fa54c680 in idl_new_fetch () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #12 0x00007f53fa55ab26 in index_read_ext_allids () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #13 0x00007f53fa545284 in keys2idl () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #14 0x00007f53fa5459e3 in ava_candidates.isra.0 () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #15 0x00007f53fa545fd2 in filter_candidates_ext () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #16 0x00007f53fa547096 in list_candidates () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #17 0x00007f53fa545f40 in filter_candidates_ext () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #18 0x00007f53fa58180c in subtree_candidates () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #19 0x00007f53fa582e7f in ldbm_back_search () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #20 0x00007f5406eb01d6 in op_shared_search () from >> /usr/lib64/dirsrv/libslapd.so.0 >> #21 0x00007f54073a133e in do_search () >> #22 0x00007f540738f567 in connection_threadmain () >> #23 0x00007f540509e7bb in _pt_root () from /lib64/libnspr4.so >> #24 0x00007f5404a3fdc5 in start_thread () from /lib64/libpthread.so.0 >> #25 0x00007f540476d28d in clone () from /lib64/libc.so.6 >> Thread 25 (Thread 0x7f53d6ff5700 (LWP 128221)): >> #0 0x00007f5404a436d5 in pthread_cond_wait@@GLIBC_2.3.2 >> () from >> /lib64/libpthread.so.0 >> #1 0x00007f53fef9f2f3 in __db_hybrid_mutex_suspend () from >> /lib64/libdb-5.3.so >> #2 0x00007f53fef9e640 in __db_tas_mutex_lock () from >> /lib64/libdb-5.3.so >> #3 0x00007f53ff048d3a in __lock_get_internal () from >> /lib64/libdb-5.3.so >> #4 0x00007f53ff049820 in __lock_get () from /lib64/libdb-5.3.so >> >> #5 0x00007f53ff075142 in __db_lget () from /lib64/libdb-5.3.so >> >> #6 0x00007f53fefbc605 in __bam_search () from >> /lib64/libdb-5.3.so >> #7 0x00007f53fefa7256 in __bamc_search () from >> /lib64/libdb-5.3.so >> #8 0x00007f53fefa8d0f in __bamc_get () from /lib64/libdb-5.3.so >> >> #9 0x00007f53ff061ca6 in __dbc_iget () from /lib64/libdb-5.3.so >> >> #10 0x00007f53ff070b02 in __dbc_get_pp () from >> /lib64/libdb-5.3.so >> #11 0x00007f53fa54c680 in idl_new_fetch () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #12 0x00007f53fa55ab26 in index_read_ext_allids () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #13 0x00007f53fa545284 in keys2idl () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #14 0x00007f53fa5459e3 in ava_candidates.isra.0 () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #15 0x00007f53fa545fd2 in filter_candidates_ext () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #16 0x00007f53fa547096 in list_candidates () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #17 0x00007f53fa545f40 in filter_candidates_ext () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #18 0x00007f53fa58180c in subtree_candidates () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #19 0x00007f53fa582e7f in ldbm_back_search () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #20 0x00007f5406eb01d6 in op_shared_search () from >> /usr/lib64/dirsrv/libslapd.so.0 >> #21 0x00007f54073a133e in do_search () >> #22 0x00007f540738f567 in connection_threadmain () >> #23 0x00007f540509e7bb in _pt_root () from /lib64/libnspr4.so >> #24 0x00007f5404a3fdc5 in start_thread () from /lib64/libpthread.so.0 >> #25 0x00007f540476d28d in clone () from /lib64/libc.so.6 >> Thread 24 (Thread 0x7f53d67f4700 (LWP 128222)): >> #0 0x00007f5404a436d5 in pthread_cond_wait@@GLIBC_2.3.2 >> () from >> /lib64/libpthread.so.0 >> #1 0x00007f53fef9f2f3 in __db_hybrid_mutex_suspend () from >> /lib64/libdb-5.3.so >> #2 0x00007f53fef9e640 in __db_tas_mutex_lock () from >> /lib64/libdb-5.3.so >> #3 0x00007f53ff048d3a in __lock_get_internal () from >> /lib64/libdb-5.3.so >> #4 0x00007f53ff049820 in __lock_get () from /lib64/libdb-5.3.so >> >> #5 0x00007f53ff075142 in __db_lget () from /lib64/libdb-5.3.so >> >> #6 0x00007f53fefbc605 in __bam_search () from >> /lib64/libdb-5.3.so >> #7 0x00007f53fefa7256 in __bamc_search () from >> /lib64/libdb-5.3.so >> #8 0x00007f53fefa8d0f in __bamc_get () from /lib64/libdb-5.3.so >> >> #9 0x00007f53ff061ca6 in __dbc_iget () from /lib64/libdb-5.3.so >> >> #10 0x00007f53ff070b02 in __dbc_get_pp () from >> /lib64/libdb-5.3.so >> #11 0x00007f53fa54c680 in idl_new_fetch () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #12 0x00007f53fa55ab26 in index_read_ext_allids () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #13 0x00007f53fa545284 in keys2idl () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #14 0x00007f53fa5459e3 in ava_candidates.isra.0 () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #15 0x00007f53fa545fd2 in filter_candidates_ext () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #16 0x00007f53fa547096 in list_candidates () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #17 0x00007f53fa545f40 in filter_candidates_ext () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #18 0x00007f53fa58180c in subtree_candidates () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #19 0x00007f53fa582e7f in ldbm_back_search () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #20 0x00007f5406eb01d6 in op_shared_search () from >> /usr/lib64/dirsrv/libslapd.so.0 >> #21 0x00007f54073a133e in do_search () >> #22 0x00007f540738f567 in connection_threadmain () >> #23 0x00007f540509e7bb in _pt_root () from /lib64/libnspr4.so >> #24 0x00007f5404a3fdc5 in start_thread () from /lib64/libpthread.so.0 >> #25 0x00007f540476d28d in clone () from /lib64/libc.so.6 >> Thread 23 (Thread 0x7f53d5ff3700 (LWP 128223)): >> #0 0x00007f5404a436d5 in pthread_cond_wait@@GLIBC_2.3.2 >> () from >> /lib64/libpthread.so.0 >> #1 0x00007f53fef9f2f3 in __db_hybrid_mutex_suspend () from >> /lib64/libdb-5.3.so >> #2 0x00007f53fef9e640 in __db_tas_mutex_lock () from >> /lib64/libdb-5.3.so >> #3 0x00007f53ff0b8e1b in __memp_fget () from /lib64/libdb-5.3.so >> >> #4 0x00007f53fefbc211 in __bam_search () from >> /lib64/libdb-5.3.so >> #5 0x00007f53fefa7256 in __bamc_search () from >> /lib64/libdb-5.3.so >> #6 0x00007f53fefab6c4 in __bamc_put () from /lib64/libdb-5.3.so >> >> #7 0x00007f53ff0626d5 in __dbc_iput () from /lib64/libdb-5.3.so >> >> #8 0x00007f53ff05da0e in __db_put () from /lib64/libdb-5.3.so >> >> #9 0x00007f53ff072fa4 in __db_put_pp () from /lib64/libdb-5.3.so >> >> #10 0x00007f53fa54d4cf in idl_new_insert_key () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #11 0x00007f53fa55c5b5 in addordel_values_sv.isra.3 () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #12 0x00007f53fa55d25a in index_addordel_values_ext_sv () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #13 0x00007f53fa55d694 in index_addordel_values_sv () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #14 0x00007f53fa55d75a in index_addordel_entry () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #15 0x00007f53fa5605fa in ldbm_back_add () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #16 0x00007f5406e62cca in op_shared_add () from >> /usr/lib64/dirsrv/libslapd.so.0 >> #17 0x00007f5406e64020 in do_add () from >> /usr/lib64/dirsrv/libslapd.so.0 >> #18 0x00007f540738f303 in connection_threadmain () >> #19 0x00007f540509e7bb in _pt_root () from /lib64/libnspr4.so >> #20 0x00007f5404a3fdc5 in start_thread () from /lib64/libpthread.so.0 >> #21 0x00007f540476d28d in clone () from /lib64/libc.so.6 >> Thread 22 (Thread 0x7f53d57f2700 (LWP 128224)): >> #0 0x00007f5404a436d5 in pthread_cond_wait@@GLIBC_2.3.2 >> () from >> /lib64/libpthread.so.0 >> #1 0x00007f53fef9f2f3 in __db_hybrid_mutex_suspend () from >> /lib64/libdb-5.3.so >> #2 0x00007f53fef9e640 in __db_tas_mutex_lock () from >> /lib64/libdb-5.3.so >> #3 0x00007f53ff048d3a in __lock_get_internal () from >> /lib64/libdb-5.3.so >> #4 0x00007f53ff049820 in __lock_get () from /lib64/libdb-5.3.so >> >> #5 0x00007f53ff075142 in __db_lget () from /lib64/libdb-5.3.so >> >> #6 0x00007f53fefbc605 in __bam_search () from >> /lib64/libdb-5.3.so >> #7 0x00007f53fefa7256 in __bamc_search () from >> /lib64/libdb-5.3.so >> #8 0x00007f53fefa8d0f in __bamc_get () from /lib64/libdb-5.3.so >> >> #9 0x00007f53ff061ca6 in __dbc_iget () from /lib64/libdb-5.3.so >> >> #10 0x00007f53ff070b02 in __dbc_get_pp () from >> /lib64/libdb-5.3.so >> #11 0x00007f53fa54c680 in idl_new_fetch () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #12 0x00007f53fa55ab26 in index_read_ext_allids () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #13 0x00007f53fa545284 in keys2idl () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #14 0x00007f53fa5459e3 in ava_candidates.isra.0 () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #15 0x00007f53fa545fd2 in filter_candidates_ext () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #16 0x00007f53fa547096 in list_candidates () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #17 0x00007f53fa545f40 in filter_candidates_ext () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #18 0x00007f53fa58180c in subtree_candidates () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #19 0x00007f53fa582e7f in ldbm_back_search () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #20 0x00007f5406eb01d6 in op_shared_search () from >> /usr/lib64/dirsrv/libslapd.so.0 >> #21 0x00007f54073a133e in do_search () >> #22 0x00007f540738f567 in connection_threadmain () >> #23 0x00007f540509e7bb in _pt_root () from /lib64/libnspr4.so >> #24 0x00007f5404a3fdc5 in start_thread () from /lib64/libpthread.so.0 >> #25 0x00007f540476d28d in clone () from /lib64/libc.so.6 >> Thread 21 (Thread 0x7f53d4ff1700 (LWP 128225)): >> #0 0x00007f5404a436d5 in pthread_cond_wait@@GLIBC_2.3.2 >> () from >> /lib64/libpthread.so.0 >> #1 0x00007f53fef9f2f3 in __db_hybrid_mutex_suspend () from >> /lib64/libdb-5.3.so >> #2 0x00007f53fef9e640 in __db_tas_mutex_lock () from >> /lib64/libdb-5.3.so >> #3 0x00007f53ff048d3a in __lock_get_internal () from >> /lib64/libdb-5.3.so >> #4 0x00007f53ff049820 in __lock_get () from /lib64/libdb-5.3.so >> >> #5 0x00007f53ff075142 in __db_lget () from /lib64/libdb-5.3.so >> >> #6 0x00007f53fefbc605 in __bam_search () from >> /lib64/libdb-5.3.so >> #7 0x00007f53fefa7256 in __bamc_search () from >> /lib64/libdb-5.3.so >> #8 0x00007f53fefa8d0f in __bamc_get () from /lib64/libdb-5.3.so >> >> #9 0x00007f53ff061ca6 in __dbc_iget () from /lib64/libdb-5.3.so >> >> #10 0x00007f53ff070b02 in __dbc_get_pp () from >> /lib64/libdb-5.3.so >> #11 0x00007f53fa54c680 in idl_new_fetch () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #12 0x00007f53fa55ab26 in index_read_ext_allids () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #13 0x00007f53fa545284 in keys2idl () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #14 0x00007f53fa5459e3 in ava_candidates.isra.0 () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #15 0x00007f53fa545fd2 in filter_candidates_ext () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #16 0x00007f53fa547096 in list_candidates () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #17 0x00007f53fa545f40 in filter_candidates_ext () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #18 0x00007f53fa58180c in subtree_candidates () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #19 0x00007f53fa582e7f in ldbm_back_search () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #20 0x00007f5406eb01d6 in op_shared_search () from >> /usr/lib64/dirsrv/libslapd.so.0 >> #21 0x00007f54073a133e in do_search () >> #22 0x00007f540738f567 in connection_threadmain () >> #23 0x00007f540509e7bb in _pt_root () from /lib64/libnspr4.so >> #24 0x00007f5404a3fdc5 in start_thread () from /lib64/libpthread.so.0 >> #25 0x00007f540476d28d in clone () from /lib64/libc.so.6 >> Thread 20 (Thread 0x7f53d47f0700 (LWP 128226)): >> #0 0x00007f5404a436d5 in pthread_cond_wait@@GLIBC_2.3.2 >> () from >> /lib64/libpthread.so.0 >> #1 0x00007f53fef9f2f3 in __db_hybrid_mutex_suspend () from >> /lib64/libdb-5.3.so >> #2 0x00007f53fef9e640 in __db_tas_mutex_lock () from >> /lib64/libdb-5.3.so >> #3 0x00007f53ff048d3a in __lock_get_internal () from >> /lib64/libdb-5.3.so >> #4 0x00007f53ff049820 in __lock_get () from /lib64/libdb-5.3.so >> >> #5 0x00007f53ff075142 in __db_lget () from /lib64/libdb-5.3.so >> >> #6 0x00007f53fefbc605 in __bam_search () from >> /lib64/libdb-5.3.so >> #7 0x00007f53fefa7256 in __bamc_search () from >> /lib64/libdb-5.3.so >> #8 0x00007f53fefa8d0f in __bamc_get () from /lib64/libdb-5.3.so >> >> #9 0x00007f53ff061ca6 in __dbc_iget () from /lib64/libdb-5.3.so >> >> #10 0x00007f53ff070b02 in __dbc_get_pp () from >> /lib64/libdb-5.3.so >> #11 0x00007f53fa54c680 in idl_new_fetch () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #12 0x00007f53fa55ab26 in index_read_ext_allids () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #13 0x00007f53fa545284 in keys2idl () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #14 0x00007f53fa5459e3 in ava_candidates.isra.0 () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #15 0x00007f53fa545fd2 in filter_candidates_ext () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #16 0x00007f53fa547096 in list_candidates () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #17 0x00007f53fa545f40 in filter_candidates_ext () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #18 0x00007f53fa58180c in subtree_candidates () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #19 0x00007f53fa582e7f in ldbm_back_search () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #20 0x00007f5406eb01d6 in op_shared_search () from >> /usr/lib64/dirsrv/libslapd.so.0 >> #21 0x00007f5406ec094e in search_internal_callback_pb () from >> /usr/lib64/dirsrv/libslapd.so.0 >> #22 0x00007f5406ec0be8 in search_internal_pb () from >> /usr/lib64/dirsrv/libslapd.so.0 >> #23 0x00007f53fe118e90 in search_one_berval () from >> /usr/lib64/dirsrv/plugins/libattr-unique-plugin.so >> #24 0x00007f53fe119242 in search () from >> /usr/lib64/dirsrv/plugins/libattr-unique-plugin.so >> #25 0x00007f53fe11954c in searchAllSubtrees () from >> /usr/lib64/dirsrv/plugins/libattr-unique-plugin.so >> #26 0x00007f53fe11a015 in preop_add () from >> /usr/lib64/dirsrv/plugins/libattr-unique-plugin.so >> #27 0x00007f5406ebb848 in plugin_call_func () from >> /usr/lib64/dirsrv/libslapd.so.0 >> #28 0x00007f5406ebbb13 in plugin_call_plugins () from >> /usr/lib64/dirsrv/libslapd.so.0 >> #29 0x00007f5406e62c2b in op_shared_add () from >> /usr/lib64/dirsrv/libslapd.so.0 >> #30 0x00007f5406e64020 in do_add () from >> /usr/lib64/dirsrv/libslapd.so.0 >> #31 0x00007f540738f303 in connection_threadmain () >> #32 0x00007f540509e7bb in _pt_root () from /lib64/libnspr4.so >> #33 0x00007f5404a3fdc5 in start_thread () from /lib64/libpthread.so.0 >> #34 0x00007f540476d28d in clone () from /lib64/libc.so.6 >> Thread 19 (Thread 0x7f53d3fef700 (LWP 128227)): >> #0 0x00007f5404a436d5 in pthread_cond_wait@@GLIBC_2.3.2 >> () from >> /lib64/libpthread.so.0 >> #1 0x00007f53fef9f2f3 in __db_hybrid_mutex_suspend () from >> /lib64/libdb-5.3.so >> #2 0x00007f53fef9e640 in __db_tas_mutex_lock () from >> /lib64/libdb-5.3.so >> #3 0x00007f53ff048d3a in __lock_get_internal () from >> /lib64/libdb-5.3.so >> #4 0x00007f53ff049820 in __lock_get () from /lib64/libdb-5.3.so >> >> #5 0x00007f53ff075142 in __db_lget () from /lib64/libdb-5.3.so >> >> #6 0x00007f53fefbc605 in __bam_search () from >> /lib64/libdb-5.3.so >> #7 0x00007f53fefa7256 in __bamc_search () from >> /lib64/libdb-5.3.so >> #8 0x00007f53fefa8d0f in __bamc_get () from /lib64/libdb-5.3.so >> >> #9 0x00007f53ff061ca6 in __dbc_iget () from /lib64/libdb-5.3.so >> >> #10 0x00007f53ff070b02 in __dbc_get_pp () from >> /lib64/libdb-5.3.so >> #11 0x00007f53fa54c680 in idl_new_fetch () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #12 0x00007f53fa55ab26 in index_read_ext_allids () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #13 0x00007f53fa545284 in keys2idl () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #14 0x00007f53fa5459e3 in ava_candidates.isra.0 () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #15 0x00007f53fa545fd2 in filter_candidates_ext () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #16 0x00007f53fa547096 in list_candidates () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #17 0x00007f53fa545f40 in filter_candidates_ext () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #18 0x00007f53fa58180c in subtree_candidates () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #19 0x00007f53fa582e7f in ldbm_back_search () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #20 0x00007f5406eb01d6 in op_shared_search () from >> /usr/lib64/dirsrv/libslapd.so.0 >> #21 0x00007f54073a133e in do_search () >> #22 0x00007f540738f567 in connection_threadmain () >> #23 0x00007f540509e7bb in _pt_root () from /lib64/libnspr4.so >> #24 0x00007f5404a3fdc5 in start_thread () from /lib64/libpthread.so.0 >> #25 0x00007f540476d28d in clone () from /lib64/libc.so.6 >> Thread 18 (Thread 0x7f53d37ee700 (LWP 128228)): >> #0 0x00007f5404a436d5 in pthread_cond_wait@@GLIBC_2.3.2 >> () from >> /lib64/libpthread.so.0 >> #1 0x00007f53fef9f2f3 in __db_hybrid_mutex_suspend () from >> /lib64/libdb-5.3.so >> #2 0x00007f53fef9e640 in __db_tas_mutex_lock () from >> /lib64/libdb-5.3.so >> #3 0x00007f53ff048d3a in __lock_get_internal () from >> /lib64/libdb-5.3.so >> #4 0x00007f53ff049820 in __lock_get () from /lib64/libdb-5.3.so >> >> #5 0x00007f53ff075142 in __db_lget () from /lib64/libdb-5.3.so >> >> #6 0x00007f53fefbc605 in __bam_search () from >> /lib64/libdb-5.3.so >> #7 0x00007f53fefa7256 in __bamc_search () from >> /lib64/libdb-5.3.so >> #8 0x00007f53fefa8d0f in __bamc_get () from /lib64/libdb-5.3.so >> >> #9 0x00007f53ff061ca6 in __dbc_iget () from /lib64/libdb-5.3.so >> >> #10 0x00007f53ff070b02 in __dbc_get_pp () from >> /lib64/libdb-5.3.so >> #11 0x00007f53fa54c680 in idl_new_fetch () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #12 0x00007f53fa55ab26 in index_read_ext_allids () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #13 0x00007f53fa545284 in keys2idl () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #14 0x00007f53fa5459e3 in ava_candidates.isra.0 () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #15 0x00007f53fa545fd2 in filter_candidates_ext () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #16 0x00007f53fa547096 in list_candidates () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #17 0x00007f53fa545f40 in filter_candidates_ext () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #18 0x00007f53fa58180c in subtree_candidates () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #19 0x00007f53fa582e7f in ldbm_back_search () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #20 0x00007f5406eb01d6 in op_shared_search () from >> /usr/lib64/dirsrv/libslapd.so.0 >> #21 0x00007f54073a133e in do_search () >> #22 0x00007f540738f567 in connection_threadmain () >> #23 0x00007f540509e7bb in _pt_root () from /lib64/libnspr4.so >> #24 0x00007f5404a3fdc5 in start_thread () from /lib64/libpthread.so.0 >> #25 0x00007f540476d28d in clone () from /lib64/libc.so.6 >> Thread 17 (Thread 0x7f53d2fed700 (LWP 128229)): >> #0 0x00007f5404a436d5 in pthread_cond_wait@@GLIBC_2.3.2 >> () from >> /lib64/libpthread.so.0 >> #1 0x00007f53fef9f2f3 in __db_hybrid_mutex_suspend () from >> /lib64/libdb-5.3.so >> #2 0x00007f53fef9e640 in __db_tas_mutex_lock () from >> /lib64/libdb-5.3.so >> #3 0x00007f53ff048d3a in __lock_get_internal () from >> /lib64/libdb-5.3.so >> #4 0x00007f53ff049820 in __lock_get () from /lib64/libdb-5.3.so >> >> #5 0x00007f53ff075142 in __db_lget () from /lib64/libdb-5.3.so >> >> #6 0x00007f53fefbc605 in __bam_search () from >> /lib64/libdb-5.3.so >> #7 0x00007f53fefa7256 in __bamc_search () from >> /lib64/libdb-5.3.so >> #8 0x00007f53fefa8d0f in __bamc_get () from /lib64/libdb-5.3.so >> >> #9 0x00007f53ff061ca6 in __dbc_iget () from /lib64/libdb-5.3.so >> >> #10 0x00007f53ff070b02 in __dbc_get_pp () from >> /lib64/libdb-5.3.so >> #11 0x00007f53fa54c680 in idl_new_fetch () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #12 0x00007f53fa55ab26 in index_read_ext_allids () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #13 0x00007f53fa545284 in keys2idl () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #14 0x00007f53fa5459e3 in ava_candidates.isra.0 () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #15 0x00007f53fa545fd2 in filter_candidates_ext () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #16 0x00007f53fa547096 in list_candidates () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #17 0x00007f53fa545f40 in filter_candidates_ext () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #18 0x00007f53fa58180c in subtree_candidates () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #19 0x00007f53fa582e7f in ldbm_back_search () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #20 0x00007f5406eb01d6 in op_shared_search () from >> /usr/lib64/dirsrv/libslapd.so.0 >> #21 0x00007f54073a133e in do_search () >> #22 0x00007f540738f567 in connection_threadmain () >> #23 0x00007f540509e7bb in _pt_root () from /lib64/libnspr4.so >> #24 0x00007f5404a3fdc5 in start_thread () from /lib64/libpthread.so.0 >> #25 0x00007f540476d28d in clone () from /lib64/libc.so.6 >> Thread 16 (Thread 0x7f53d27ec700 (LWP 128230)): >> #0 0x00007f5404a436d5 in pthread_cond_wait@@GLIBC_2.3.2 >> () from >> /lib64/libpthread.so.0 >> #1 0x00007f53fef9f2f3 in __db_hybrid_mutex_suspend () from >> /lib64/libdb-5.3.so >> #2 0x00007f53fef9e640 in __db_tas_mutex_lock () from >> /lib64/libdb-5.3.so >> #3 0x00007f53ff048d3a in __lock_get_internal () from >> /lib64/libdb-5.3.so >> #4 0x00007f53ff049820 in __lock_get () from /lib64/libdb-5.3.so >> >> #5 0x00007f53ff075142 in __db_lget () from /lib64/libdb-5.3.so >> >> #6 0x00007f53fefbc605 in __bam_search () from >> /lib64/libdb-5.3.so >> #7 0x00007f53fefa7256 in __bamc_search () from >> /lib64/libdb-5.3.so >> #8 0x00007f53fefa8d0f in __bamc_get () from /lib64/libdb-5.3.so >> >> #9 0x00007f53ff061ca6 in __dbc_iget () from /lib64/libdb-5.3.so >> >> #10 0x00007f53ff070b02 in __dbc_get_pp () from >> /lib64/libdb-5.3.so >> #11 0x00007f53fa54c680 in idl_new_fetch () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #12 0x00007f53fa55ab26 in index_read_ext_allids () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #13 0x00007f53fa545284 in keys2idl () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #14 0x00007f53fa5459e3 in ava_candidates.isra.0 () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #15 0x00007f53fa545fd2 in filter_candidates_ext () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #16 0x00007f53fa547096 in list_candidates () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #17 0x00007f53fa545f40 in filter_candidates_ext () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #18 0x00007f53fa58180c in subtree_candidates () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #19 0x00007f53fa582e7f in ldbm_back_search () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #20 0x00007f5406eb01d6 in op_shared_search () from >> /usr/lib64/dirsrv/libslapd.so.0 >> #21 0x00007f54073a133e in do_search () >> #22 0x00007f540738f567 in connection_threadmain () >> #23 0x00007f540509e7bb in _pt_root () from /lib64/libnspr4.so >> #24 0x00007f5404a3fdc5 in start_thread () from /lib64/libpthread.so.0 >> #25 0x00007f540476d28d in clone () from /lib64/libc.so.6 >> Thread 15 (Thread 0x7f53d1feb700 (LWP 128231)): >> #0 0x00007f5404a436d5 in pthread_cond_wait@@GLIBC_2.3.2 >> () from >> /lib64/libpthread.so.0 >> #1 0x00007f53fef9f2f3 in __db_hybrid_mutex_suspend () from >> /lib64/libdb-5.3.so >> #2 0x00007f53fef9e640 in __db_tas_mutex_lock () from >> /lib64/libdb-5.3.so >> #3 0x00007f53ff048d3a in __lock_get_internal () from >> /lib64/libdb-5.3.so >> #4 0x00007f53ff049820 in __lock_get () from /lib64/libdb-5.3.so >> >> #5 0x00007f53ff075142 in __db_lget () from /lib64/libdb-5.3.so >> >> #6 0x00007f53fefbc605 in __bam_search () from >> /lib64/libdb-5.3.so >> #7 0x00007f53fefa7256 in __bamc_search () from >> /lib64/libdb-5.3.so >> #8 0x00007f53fefa8d0f in __bamc_get () from /lib64/libdb-5.3.so >> >> #9 0x00007f53ff061ca6 in __dbc_iget () from /lib64/libdb-5.3.so >> >> #10 0x00007f53ff070b02 in __dbc_get_pp () from >> /lib64/libdb-5.3.so >> #11 0x00007f53fa54c680 in idl_new_fetch () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #12 0x00007f53fa55ab26 in index_read_ext_allids () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #13 0x00007f53fa545284 in keys2idl () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #14 0x00007f53fa5459e3 in ava_candidates.isra.0 () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #15 0x00007f53fa545fd2 in filter_candidates_ext () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #16 0x00007f53fa547096 in list_candidates () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #17 0x00007f53fa545f40 in filter_candidates_ext () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #18 0x00007f53fa58180c in subtree_candidates () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #19 0x00007f53fa582e7f in ldbm_back_search () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #20 0x00007f5406eb01d6 in op_shared_search () from >> /usr/lib64/dirsrv/libslapd.so.0 >> #21 0x00007f54073a133e in do_search () >> #22 0x00007f540738f567 in connection_threadmain () >> #23 0x00007f540509e7bb in _pt_root () from /lib64/libnspr4.so >> #24 0x00007f5404a3fdc5 in start_thread () from /lib64/libpthread.so.0 >> #25 0x00007f540476d28d in clone () from /lib64/libc.so.6 >> Thread 14 (Thread 0x7f53d17ea700 (LWP 128232)): >> #0 0x00007f5404a436d5 in pthread_cond_wait@@GLIBC_2.3.2 >> () from >> /lib64/libpthread.so.0 >> #1 0x00007f53fef9f2f3 in __db_hybrid_mutex_suspend () from >> /lib64/libdb-5.3.so >> #2 0x00007f53fef9e640 in __db_tas_mutex_lock () from >> /lib64/libdb-5.3.so >> #3 0x00007f53ff048d3a in __lock_get_internal () from >> /lib64/libdb-5.3.so >> #4 0x00007f53ff049820 in __lock_get () from /lib64/libdb-5.3.so >> >> #5 0x00007f53ff075142 in __db_lget () from /lib64/libdb-5.3.so >> >> #6 0x00007f53fefbc605 in __bam_search () from >> /lib64/libdb-5.3.so >> #7 0x00007f53fefa7256 in __bamc_search () from >> /lib64/libdb-5.3.so >> #8 0x00007f53fefa8d0f in __bamc_get () from /lib64/libdb-5.3.so >> >> #9 0x00007f53ff061ca6 in __dbc_iget () from /lib64/libdb-5.3.so >> >> #10 0x00007f53ff070b02 in __dbc_get_pp () from >> /lib64/libdb-5.3.so >> #11 0x00007f53fa54c680 in idl_new_fetch () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #12 0x00007f53fa55ab26 in index_read_ext_allids () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #13 0x00007f53fa545284 in keys2idl () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #14 0x00007f53fa5459e3 in ava_candidates.isra.0 () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #15 0x00007f53fa545fd2 in filter_candidates_ext () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #16 0x00007f53fa547096 in list_candidates () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #17 0x00007f53fa545f40 in filter_candidates_ext () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #18 0x00007f53fa547096 in list_candidates () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #19 0x00007f53fa545f40 in filter_candidates_ext () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #20 0x00007f53fa547096 in list_candidates () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #21 0x00007f53fa545f40 in filter_candidates_ext () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #22 0x00007f53fa58180c in subtree_candidates () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #23 0x00007f53fa582e7f in ldbm_back_search () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #24 0x00007f5406eb01d6 in op_shared_search () from >> /usr/lib64/dirsrv/libslapd.so.0 >> #25 0x00007f54073a133e in do_search () >> #26 0x00007f540738f3c3 in connection_threadmain () >> #27 0x00007f540509e7bb in _pt_root () from /lib64/libnspr4.so >> #28 0x00007f5404a3fdc5 in start_thread () from /lib64/libpthread.so.0 >> #29 0x00007f540476d28d in clone () from /lib64/libc.so.6 >> Thread 13 (Thread 0x7f53d0fe9700 (LWP 128233)): >> #0 0x00007f5404a436d5 in pthread_cond_wait@@GLIBC_2.3.2 >> () from >> /lib64/libpthread.so.0 >> #1 0x00007f53fef9f2f3 in __db_hybrid_mutex_suspend () from >> /lib64/libdb-5.3.so >> #2 0x00007f53fef9e640 in __db_tas_mutex_lock () from >> /lib64/libdb-5.3.so >> #3 0x00007f53ff048d3a in __lock_get_internal () from >> /lib64/libdb-5.3.so >> #4 0x00007f53ff049820 in __lock_get () from /lib64/libdb-5.3.so >> >> #5 0x00007f53ff075142 in __db_lget () from /lib64/libdb-5.3.so >> >> #6 0x00007f53fefbc605 in __bam_search () from >> /lib64/libdb-5.3.so >> #7 0x00007f53fefa7256 in __bamc_search () from >> /lib64/libdb-5.3.so >> #8 0x00007f53fefa8d0f in __bamc_get () from /lib64/libdb-5.3.so >> >> #9 0x00007f53ff061ca6 in __dbc_iget () from /lib64/libdb-5.3.so >> >> #10 0x00007f53ff070b02 in __dbc_get_pp () from >> /lib64/libdb-5.3.so >> #11 0x00007f53fa54c680 in idl_new_fetch () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #12 0x00007f53fa55ab26 in index_read_ext_allids () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #13 0x00007f53fa545284 in keys2idl () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #14 0x00007f53fa5459e3 in ava_candidates.isra.0 () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #15 0x00007f53fa545fd2 in filter_candidates_ext () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #16 0x00007f53fa547096 in list_candidates () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #17 0x00007f53fa545f40 in filter_candidates_ext () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #18 0x00007f53fa58180c in subtree_candidates () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #19 0x00007f53fa582e7f in ldbm_back_search () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #20 0x00007f5406eb01d6 in op_shared_search () from >> /usr/lib64/dirsrv/libslapd.so.0 >> #21 0x00007f54073a133e in do_search () >> #22 0x00007f540738f567 in connection_threadmain () >> #23 0x00007f540509e7bb in _pt_root () from /lib64/libnspr4.so >> #24 0x00007f5404a3fdc5 in start_thread () from /lib64/libpthread.so.0 >> #25 0x00007f540476d28d in clone () from /lib64/libc.so.6 >> Thread 12 (Thread 0x7f53d07e8700 (LWP 128234)): >> #0 0x00007f5404a436d5 in pthread_cond_wait@@GLIBC_2.3.2 >> () from >> /lib64/libpthread.so.0 >> #1 0x00007f53fef9f2f3 in __db_hybrid_mutex_suspend () from >> /lib64/libdb-5.3.so >> #2 0x00007f53fef9e640 in __db_tas_mutex_lock () from >> /lib64/libdb-5.3.so >> #3 0x00007f53ff048d3a in __lock_get_internal () from >> /lib64/libdb-5.3.so >> #4 0x00007f53ff049820 in __lock_get () from /lib64/libdb-5.3.so >> >> #5 0x00007f53ff075142 in __db_lget () from /lib64/libdb-5.3.so >> >> #6 0x00007f53fefbc605 in __bam_search () from >> /lib64/libdb-5.3.so >> #7 0x00007f53fefa7256 in __bamc_search () from >> /lib64/libdb-5.3.so >> #8 0x00007f53fefa8d0f in __bamc_get () from /lib64/libdb-5.3.so >> >> #9 0x00007f53ff061ca6 in __dbc_iget () from /lib64/libdb-5.3.so >> >> #10 0x00007f53ff070b02 in __dbc_get_pp () from >> /lib64/libdb-5.3.so >> #11 0x00007f53fa54c680 in idl_new_fetch () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #12 0x00007f53fa55ab26 in index_read_ext_allids () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #13 0x00007f53fa545284 in keys2idl () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #14 0x00007f53fa5459e3 in ava_candidates.isra.0 () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #15 0x00007f53fa545fd2 in filter_candidates_ext () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #16 0x00007f53fa547096 in list_candidates () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #17 0x00007f53fa545f40 in filter_candidates_ext () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #18 0x00007f53fa58180c in subtree_candidates () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #19 0x00007f53fa582e7f in ldbm_back_search () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #20 0x00007f5406eb01d6 in op_shared_search () from >> /usr/lib64/dirsrv/libslapd.so.0 >> #21 0x00007f54073a133e in do_search () >> #22 0x00007f540738f567 in connection_threadmain () >> #23 0x00007f540509e7bb in _pt_root () from /lib64/libnspr4.so >> #24 0x00007f5404a3fdc5 in start_thread () from /lib64/libpthread.so.0 >> #25 0x00007f540476d28d in clone () from /lib64/libc.so.6 >> Thread 11 (Thread 0x7f53cffe7700 (LWP 128235)): >> #0 0x00007f5404a436d5 in pthread_cond_wait@@GLIBC_2.3.2 >> () from >> /lib64/libpthread.so.0 >> #1 0x00007f53fef9f2f3 in __db_hybrid_mutex_suspend () from >> /lib64/libdb-5.3.so >> #2 0x00007f53fef9e640 in __db_tas_mutex_lock () from >> /lib64/libdb-5.3.so >> #3 0x00007f53ff048d3a in __lock_get_internal () from >> /lib64/libdb-5.3.so >> #4 0x00007f53ff049820 in __lock_get () from /lib64/libdb-5.3.so >> >> #5 0x00007f53ff075142 in __db_lget () from /lib64/libdb-5.3.so >> >> #6 0x00007f53fefbc605 in __bam_search () from >> /lib64/libdb-5.3.so >> #7 0x00007f53fefa7256 in __bamc_search () from >> /lib64/libdb-5.3.so >> #8 0x00007f53fefa8d0f in __bamc_get () from /lib64/libdb-5.3.so >> >> #9 0x00007f53ff061ca6 in __dbc_iget () from /lib64/libdb-5.3.so >> >> #10 0x00007f53ff070b02 in __dbc_get_pp () from >> /lib64/libdb-5.3.so >> #11 0x00007f53fa54c680 in idl_new_fetch () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #12 0x00007f53fa55ab26 in index_read_ext_allids () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #13 0x00007f53fa545284 in keys2idl () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #14 0x00007f53fa5459e3 in ava_candidates.isra.0 () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #15 0x00007f53fa545fd2 in filter_candidates_ext () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #16 0x00007f53fa547096 in list_candidates () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #17 0x00007f53fa545f40 in filter_candidates_ext () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #18 0x00007f53fa58180c in subtree_candidates () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #19 0x00007f53fa582e7f in ldbm_back_search () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #20 0x00007f5406eb01d6 in op_shared_search () from >> /usr/lib64/dirsrv/libslapd.so.0 >> #21 0x00007f54073a133e in do_search () >> #22 0x00007f540738f567 in connection_threadmain () >> #23 0x00007f540509e7bb in _pt_root () from /lib64/libnspr4.so >> #24 0x00007f5404a3fdc5 in start_thread () from /lib64/libpthread.so.0 >> #25 0x00007f540476d28d in clone () from /lib64/libc.so.6 >> Thread 10 (Thread 0x7f53cf7e6700 (LWP 128236)): >> #0 0x00007f5404a436d5 in pthread_cond_wait@@GLIBC_2.3.2 >> () from >> /lib64/libpthread.so.0 >> #1 0x00007f53fef9f2f3 in __db_hybrid_mutex_suspend () from >> /lib64/libdb-5.3.so >> #2 0x00007f53fef9e640 in __db_tas_mutex_lock () from >> /lib64/libdb-5.3.so >> #3 0x00007f53ff048d3a in __lock_get_internal () from >> /lib64/libdb-5.3.so >> #4 0x00007f53ff049820 in __lock_get () from /lib64/libdb-5.3.so >> >> #5 0x00007f53ff075142 in __db_lget () from /lib64/libdb-5.3.so >> >> #6 0x00007f53fefbc605 in __bam_search () from >> /lib64/libdb-5.3.so >> #7 0x00007f53fefa7256 in __bamc_search () from >> /lib64/libdb-5.3.so >> #8 0x00007f53fefa8d0f in __bamc_get () from /lib64/libdb-5.3.so >> >> #9 0x00007f53ff061ca6 in __dbc_iget () from /lib64/libdb-5.3.so >> >> #10 0x00007f53ff070b02 in __dbc_get_pp () from >> /lib64/libdb-5.3.so >> #11 0x00007f53fa54c680 in idl_new_fetch () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #12 0x00007f53fa55ab26 in index_read_ext_allids () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #13 0x00007f53fa545284 in keys2idl () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #14 0x00007f53fa5459e3 in ava_candidates.isra.0 () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #15 0x00007f53fa545fd2 in filter_candidates_ext () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #16 0x00007f53fa547096 in list_candidates () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #17 0x00007f53fa545f40 in filter_candidates_ext () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #18 0x00007f53fa58180c in subtree_candidates () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #19 0x00007f53fa582e7f in ldbm_back_search () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #20 0x00007f5406eb01d6 in op_shared_search () from >> /usr/lib64/dirsrv/libslapd.so.0 >> #21 0x00007f54073a133e in do_search () >> #22 0x00007f540738f567 in connection_threadmain () >> #23 0x00007f540509e7bb in _pt_root () from /lib64/libnspr4.so >> #24 0x00007f5404a3fdc5 in start_thread () from /lib64/libpthread.so.0 >> #25 0x00007f540476d28d in clone () from /lib64/libc.so.6 >> Thread 9 (Thread 0x7f53cefe5700 (LWP 128237)): >> #0 0x00007f5404a436d5 in pthread_cond_wait@@GLIBC_2.3.2 >> () from >> /lib64/libpthread.so.0 >> #1 0x00007f53fef9f2f3 in __db_hybrid_mutex_suspend () from >> /lib64/libdb-5.3.so >> #2 0x00007f53fef9e640 in __db_tas_mutex_lock () from >> /lib64/libdb-5.3.so >> #3 0x00007f53ff048d3a in __lock_get_internal () from >> /lib64/libdb-5.3.so >> #4 0x00007f53ff049820 in __lock_get () from /lib64/libdb-5.3.so >> >> #5 0x00007f53ff075142 in __db_lget () from /lib64/libdb-5.3.so >> >> #6 0x00007f53fefbc605 in __bam_search () from >> /lib64/libdb-5.3.so >> #7 0x00007f53fefa7256 in __bamc_search () from >> /lib64/libdb-5.3.so >> #8 0x00007f53fefa8d0f in __bamc_get () from /lib64/libdb-5.3.so >> >> #9 0x00007f53ff061ca6 in __dbc_iget () from /lib64/libdb-5.3.so >> >> #10 0x00007f53ff070b02 in __dbc_get_pp () from >> /lib64/libdb-5.3.so >> #11 0x00007f53fa54c680 in idl_new_fetch () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #12 0x00007f53fa55ab26 in index_read_ext_allids () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #13 0x00007f53fa545284 in keys2idl () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #14 0x00007f53fa5459e3 in ava_candidates.isra.0 () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #15 0x00007f53fa545fd2 in filter_candidates_ext () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #16 0x00007f53fa547096 in list_candidates () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #17 0x00007f53fa545f40 in filter_candidates_ext () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #18 0x00007f53fa58180c in subtree_candidates () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #19 0x00007f53fa582e7f in ldbm_back_search () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #20 0x00007f5406eb01d6 in op_shared_search () from >> /usr/lib64/dirsrv/libslapd.so.0 >> #21 0x00007f54073a133e in do_search () >> #22 0x00007f540738f567 in connection_threadmain () >> #23 0x00007f540509e7bb in _pt_root () from /lib64/libnspr4.so >> #24 0x00007f5404a3fdc5 in start_thread () from /lib64/libpthread.so.0 >> #25 0x00007f540476d28d in clone () from /lib64/libc.so.6 >> Thread 8 (Thread 0x7f53ce7e4700 (LWP 128238)): >> #0 0x00007f5404a436d5 in pthread_cond_wait@@GLIBC_2.3.2 >> () from >> /lib64/libpthread.so.0 >> #1 0x00007f53fef9f2f3 in __db_hybrid_mutex_suspend () from >> /lib64/libdb-5.3.so >> #2 0x00007f53fef9e640 in __db_tas_mutex_lock () from >> /lib64/libdb-5.3.so >> #3 0x00007f53ff048d3a in __lock_get_internal () from >> /lib64/libdb-5.3.so >> #4 0x00007f53ff049820 in __lock_get () from /lib64/libdb-5.3.so >> >> #5 0x00007f53ff075142 in __db_lget () from /lib64/libdb-5.3.so >> >> #6 0x00007f53fefbc605 in __bam_search () from >> /lib64/libdb-5.3.so >> #7 0x00007f53fefa7256 in __bamc_search () from >> /lib64/libdb-5.3.so >> #8 0x00007f53fefa8d0f in __bamc_get () from /lib64/libdb-5.3.so >> >> #9 0x00007f53ff061ca6 in __dbc_iget () from /lib64/libdb-5.3.so >> >> #10 0x00007f53ff070b02 in __dbc_get_pp () from >> /lib64/libdb-5.3.so >> #11 0x00007f53fa54c680 in idl_new_fetch () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #12 0x00007f53fa55ab26 in index_read_ext_allids () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #13 0x00007f53fa545284 in keys2idl () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #14 0x00007f53fa5459e3 in ava_candidates.isra.0 () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #15 0x00007f53fa545fd2 in filter_candidates_ext () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #16 0x00007f53fa547096 in list_candidates () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #17 0x00007f53fa545f40 in filter_candidates_ext () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #18 0x00007f53fa58180c in subtree_candidates () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #19 0x00007f53fa582e7f in ldbm_back_search () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #20 0x00007f5406eb01d6 in op_shared_search () from >> /usr/lib64/dirsrv/libslapd.so.0 >> #21 0x00007f54073a133e in do_search () >> #22 0x00007f540738f567 in connection_threadmain () >> #23 0x00007f540509e7bb in _pt_root () from /lib64/libnspr4.so >> #24 0x00007f5404a3fdc5 in start_thread () from /lib64/libpthread.so.0 >> #25 0x00007f540476d28d in clone () from /lib64/libc.so.6 >> Thread 7 (Thread 0x7f53cdfe3700 (LWP 128239)): >> #0 0x00007f5404a436d5 in pthread_cond_wait@@GLIBC_2.3.2 >> () from >> /lib64/libpthread.so.0 >> #1 0x00007f53fef9f2f3 in __db_hybrid_mutex_suspend () from >> /lib64/libdb-5.3.so >> #2 0x00007f53fef9e640 in __db_tas_mutex_lock () from >> /lib64/libdb-5.3.so >> #3 0x00007f53ff048d3a in __lock_get_internal () from >> /lib64/libdb-5.3.so >> #4 0x00007f53ff049820 in __lock_get () from /lib64/libdb-5.3.so >> >> #5 0x00007f53ff075142 in __db_lget () from /lib64/libdb-5.3.so >> >> #6 0x00007f53fefbc605 in __bam_search () from >> /lib64/libdb-5.3.so >> #7 0x00007f53fefa7256 in __bamc_search () from >> /lib64/libdb-5.3.so >> #8 0x00007f53fefa8d0f in __bamc_get () from /lib64/libdb-5.3.so >> >> #9 0x00007f53ff061ca6 in __dbc_iget () from /lib64/libdb-5.3.so >> >> #10 0x00007f53ff070b02 in __dbc_get_pp () from >> /lib64/libdb-5.3.so >> #11 0x00007f53fa54c680 in idl_new_fetch () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #12 0x00007f53fa55ab26 in index_read_ext_allids () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #13 0x00007f53fa545284 in keys2idl () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #14 0x00007f53fa5459e3 in ava_candidates.isra.0 () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #15 0x00007f53fa545fd2 in filter_candidates_ext () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #16 0x00007f53fa547096 in list_candidates () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #17 0x00007f53fa545f40 in filter_candidates_ext () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #18 0x00007f53fa58180c in subtree_candidates () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #19 0x00007f53fa582e7f in ldbm_back_search () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #20 0x00007f5406eb01d6 in op_shared_search () from >> /usr/lib64/dirsrv/libslapd.so.0 >> #21 0x00007f54073a133e in do_search () >> #22 0x00007f540738f567 in connection_threadmain () >> #23 0x00007f540509e7bb in _pt_root () from /lib64/libnspr4.so >> #24 0x00007f5404a3fdc5 in start_thread () from /lib64/libpthread.so.0 >> #25 0x00007f540476d28d in clone () from /lib64/libc.so.6 >> Thread 6 (Thread 0x7f53cd7e2700 (LWP 128240)): >> #0 0x00007f5404a436d5 in pthread_cond_wait@@GLIBC_2.3.2 >> () from >> /lib64/libpthread.so.0 >> #1 0x00007f53fef9f2f3 in __db_hybrid_mutex_suspend () from >> /lib64/libdb-5.3.so >> #2 0x00007f53fef9eb37 in __db_tas_mutex_readlock () from >> /lib64/libdb-5.3.so >> #3 0x00007f53ff0b8d35 in __memp_fget () from /lib64/libdb-5.3.so >> >> #4 0x00007f53ff062333 in __dbc_iget () from /lib64/libdb-5.3.so >> >> #5 0x00007f53ff070b02 in __dbc_get_pp () from >> /lib64/libdb-5.3.so >> #6 0x00007f53fa54c822 in idl_new_fetch () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #7 0x00007f53fa55ab26 in index_read_ext_allids () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #8 0x00007f53fa545284 in keys2idl () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #9 0x00007f53fa5459e3 in ava_candidates.isra.0 () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #10 0x00007f53fa545fd2 in filter_candidates_ext () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #11 0x00007f53fa547096 in list_candidates () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #12 0x00007f53fa545f40 in filter_candidates_ext () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #13 0x00007f53fa547096 in list_candidates () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #14 0x00007f53fa545f40 in filter_candidates_ext () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #15 0x00007f53fa547096 in list_candidates () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #16 0x00007f53fa545f40 in filter_candidates_ext () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #17 0x00007f53fa58180c in subtree_candidates () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #18 0x00007f53fa582e7f in ldbm_back_search () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #19 0x00007f5406eb01d6 in op_shared_search () from >> /usr/lib64/dirsrv/libslapd.so.0 >> #20 0x00007f54073a133e in do_search () >> #21 0x00007f540738f3c3 in connection_threadmain () >> #22 0x00007f540509e7bb in _pt_root () from /lib64/libnspr4.so >> #23 0x00007f5404a3fdc5 in start_thread () from /lib64/libpthread.so.0 >> #24 0x00007f540476d28d in clone () from /lib64/libc.so.6 >> Thread 5 (Thread 0x7f53ccfe1700 (LWP 128241)): >> #0 0x00007f5404a436d5 in pthread_cond_wait@@GLIBC_2.3.2 >> () from >> /lib64/libpthread.so.0 >> #1 0x00007f53fef9f2f3 in __db_hybrid_mutex_suspend () from >> /lib64/libdb-5.3.so >> #2 0x00007f53fef9e640 in __db_tas_mutex_lock () from >> /lib64/libdb-5.3.so >> #3 0x00007f53ff048d3a in __lock_get_internal () from >> /lib64/libdb-5.3.so >> #4 0x00007f53ff049820 in __lock_get () from /lib64/libdb-5.3.so >> >> #5 0x00007f53ff075142 in __db_lget () from /lib64/libdb-5.3.so >> >> #6 0x00007f53fefbc605 in __bam_search () from >> /lib64/libdb-5.3.so >> #7 0x00007f53fefa7256 in __bamc_search () from >> /lib64/libdb-5.3.so >> #8 0x00007f53fefa8d0f in __bamc_get () from /lib64/libdb-5.3.so >> >> #9 0x00007f53ff061ca6 in __dbc_iget () from /lib64/libdb-5.3.so >> >> #10 0x00007f53ff070b02 in __dbc_get_pp () from >> /lib64/libdb-5.3.so >> #11 0x00007f53fa54c680 in idl_new_fetch () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #12 0x00007f53fa55ab26 in index_read_ext_allids () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #13 0x00007f53fa545284 in keys2idl () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #14 0x00007f53fa5459e3 in ava_candidates.isra.0 () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #15 0x00007f53fa545fd2 in filter_candidates_ext () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #16 0x00007f53fa547096 in list_candidates () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #17 0x00007f53fa545f40 in filter_candidates_ext () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #18 0x00007f53fa547096 in list_candidates () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #19 0x00007f53fa545f40 in filter_candidates_ext () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #20 0x00007f53fa547096 in list_candidates () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #21 0x00007f53fa545f40 in filter_candidates_ext () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #22 0x00007f53fa58180c in subtree_candidates () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #23 0x00007f53fa582e7f in ldbm_back_search () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #24 0x00007f5406eb01d6 in op_shared_search () from >> /usr/lib64/dirsrv/libslapd.so.0 >> #25 0x00007f54073a133e in do_search () >> #26 0x00007f540738f3c3 in connection_threadmain () >> #27 0x00007f540509e7bb in _pt_root () from /lib64/libnspr4.so >> #28 0x00007f5404a3fdc5 in start_thread () from /lib64/libpthread.so.0 >> #29 0x00007f540476d28d in clone () from /lib64/libc.so.6 >> Thread 4 (Thread 0x7f53cc7e0700 (LWP 128242)): >> #0 0x00007f54047649b3 in select () from /lib64/libc.so.6 >> #1 0x00007f5406eef0e9 in DS_Sleep () from >> /usr/lib64/dirsrv/libslapd.so.0 >> #2 0x00007f5407390325 in time_thread () >> #3 0x00007f540509e7bb in _pt_root () from /lib64/libnspr4.so >> #4 0x00007f5404a3fdc5 in start_thread () from /lib64/libpthread.so.0 >> #5 0x00007f540476d28d in clone () from /lib64/libc.so.6 >> Thread 3 (Thread 0x7f53cb7a0700 (LWP 128259)): >> #0 0x00007f5404a436d5 in pthread_cond_wait@@GLIBC_2.3.2 >> () from >> /lib64/libpthread.so.0 >> #1 0x00007f5405099050 in PR_WaitCondVar () from /lib64/libnspr4.so >> #2 0x00007f540739bf25 in ps_send_results () >> #3 0x00007f540509e7bb in _pt_root () from /lib64/libnspr4.so >> #4 0x00007f5404a3fdc5 in start_thread () from /lib64/libpthread.so.0 >> #5 0x00007f540476d28d in clone () from /lib64/libc.so.6 >> Thread 2 (Thread 0x7f53cab9d700 (LWP 9020)): >> #0 0x00007f5404a436d5 in pthread_cond_wait@@GLIBC_2.3.2 >> () from >> /lib64/libpthread.so.0 >> #1 0x00007f5405099050 in PR_WaitCondVar () from /lib64/libnspr4.so >> #2 0x00007f540739bf25 in ps_send_results () >> #3 0x00007f540509e7bb in _pt_root () from /lib64/libnspr4.so >> #4 0x00007f5404a3fdc5 in start_thread () from /lib64/libpthread.so.0 >> #5 0x00007f540476d28d in clone () from /lib64/libc.so.6 >> Thread 1 (Thread 0x7f5407357840 (LWP 128124)): >> #0 0x00007f5404762c3d in poll () from /lib64/libc.so.6 >> #1 0x00007f540509a967 in _pr_poll_with_poll () from >> /lib64/libnspr4.so >> #2 0x00007f5407393ff1 in slapd_daemon () >> #3 0x00007f5407385c6c in main () >> >> >> Thanks >> >> On Tue, Sep 6, 2016 at 5:51 PM, Rakesh Rajasekharan >> > > wrote: >> >> Hi, >> >> With script provided by Rich, I looked at my db cache usage >> and it does not look to be high enough >> >> # under each db are the list of selected indexes specified >> with INDEXLIST >> dbname count free free% size >> changelog:ent 5 2076333 99.0 4163.8 >> changelog:dn 15976 9303557 88.7 74.0 >> userroot:ent 4763 175321074 83.6 7221.1 >> userroot:dn 4763 9995056 95.3 103.0 >> ipaca:ent 425 6337866 60.4 9759.8 >> ipaca:dn 425 10451081 99.7 81.6 >> >> Right now I set it to 200MB. So, I am planning to bring it >> down to 100MB >> >> Also, /|nsslapd-db-checkpoint-interval |/will change it to 80 >> or 90 from default of 60 >> >> I will check this out and the impact it has on the hang >> issues and get back >> >> Thanks >> Rakesh >> >> On Tue, Sep 6, 2016 at 4:13 PM, Ludwig Krispenz >> > wrote: >> >> Hi, >> >> I think you are running into this bug: >> https://bugzilla.redhat.com/show_bug.cgi?id=1349779 >> it was the outcome of an analysis started in this one: >> https://bugzilla.redhat.com/show_bug.cgi?id=1347800 >> >> >> the problem is that it looks like the checkpoint thread >> is holding a mutex and otehr threads are blocked on it. >> It looks like an issue in BerkeleyDB although there is no >> explanation from the BDB code why this could happen, we >> also had a support case with Oracle to investigate this, >> with no solution. Oracle at the end was providing a >> diagnostic patch, which we did not test since the >> customer with the original problem had worked around this >> by changing their configuration. >> I don't know the status of this patch and if we can make >> it available, probably depends on your support status, >> you need to contact support for this. >> >> About the workarounds: >> you can try to change timing for the database update and >> checkpointing eg by reducing the dbcache size, changing >> the checkpoint interval. In the customer case they >> identified one substring index, which was frequently >> updated but not really required - and removing this index >> helped. >> >> Regards, >> Ludwig >> >> On 09/06/2016 11:31 AM, thierry bordaz wrote: >>> Hi, >>> >>> The following thread is hanging most of the others threads >>> >>> Thread 20 (Thread 0x7f68007f8700 (LWP 4873)): >>> #0 0x00007f683d0506d5 in >>> pthread_cond_wait@@GLIBC_2.3.2 >>> () from >>> /lib64/libpthread.so.0 >>> #1 0x00007f68375ac2f3 in __db_hybrid_mutex_suspend >>> () from /lib64/libdb-5.3.so >>> #2 0x00007f68375ab640 in __db_tas_mutex_lock () >>> from /lib64/libdb-5.3.so >>> #3 0x00007f68376c5e1b in __memp_fget () from >>> /lib64/libdb-5.3.so >>> #4 0x00007f68375c9211 in __bam_search () from >>> /lib64/libdb-5.3.so >>> #5 0x00007f68375b4256 in __bamc_search () from >>> /lib64/libdb-5.3.so >>> #6 0x00007f68375b86c4 in __bamc_put () from >>> /lib64/libdb-5.3.so >>> #7 0x00007f683766f6d5 in __dbc_iput () from >>> /lib64/libdb-5.3.so >>> #8 0x00007f683766aa0e in __db_put () from >>> /lib64/libdb-5.3.so >>> #9 0x00007f683767ffa4 in __db_put_pp () from >>> /lib64/libdb-5.3.so >>> #10 0x00007f6832b5a4cf in idl_new_insert_key () from >>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>> #11 0x00007f6832b695b5 in addordel_values_sv.isra.3 >>> () from /usr/lib64/dirsrv/plugins/libback-ldbm.so >>> #12 0x00007f6832b6a25a in >>> index_addordel_values_ext_sv () from >>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>> #13 0x00007f6832b6a694 in index_addordel_values_sv >>> () from /usr/lib64/dirsrv/plugins/libback-ldbm.so >>> #14 0x00007f6832b6a75a in index_addordel_entry () >>> from /usr/lib64/dirsrv/plugins/libback-ldbm.so >>> #15 0x00007f6832b6d5fa in ldbm_back_add () from >>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>> #16 0x00007f683f46fcca in op_shared_add () from >>> /usr/lib64/dirsrv/libslapd.so.0 >>> #17 0x00007f683f471020 in do_add () from >>> /usr/lib64/dirsrv/libslapd.so.0 >>> #18 0x00007f683f99c303 in connection_threadmain () >>> #19 0x00007f683d6ab7bb in _pt_root () from >>> /lib64/libnspr4.so >>> #20 0x00007f683d04cdc5 in start_thread () from >>> /lib64/libpthread.so.0 >>> #21 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 >>> >>> This thread is likely not hang on 'db_tas_mutex_lock' as >>> the db_lock does not show it waiting for something. >>> Unfortunately it is not shown in the output of top -H, >>> did you get the complete output ? >>> Ludwig mentioned a known issue that DS is not scaling >>> well on NUMA arch, do you know if the machine is NUMA ? >>> A possibility is that this thread get severely slowed >>> down by a NUMA remote lock and showing DS has hung. >>> This is why it would be interesting to know if it >>> consums CPU. >>> >>> thanks >>> theirry >>> >>> On 09/06/2016 10:59 AM, Rakesh Rajasekharan wrote: >>>> Hi Thierry/Ludwig, >>>> >>>> I again hit the hang issue. This is the pstack of the >>>> ns-slapd >>>> >>>> So was able to this time get the pstack and db locks >>>> >>>> db locks details >>>> Default locking region information: >>>> 451 Last allocated locker ID >>>> 0x7fffffff Current maximum unused locker ID >>>> 9 Number of lock modes >>>> 800 Initial number of locks allocated >>>> 0 Initial number of lockers allocated >>>> 800 Initial number of lock objects allocated >>>> 10000 Maximum number of locks possible >>>> 10000 Maximum number of lockers possible >>>> 10000 Maximum number of lock objects possible >>>> 800 Current number of locks allocated >>>> 366 Current number of lockers allocated >>>> 800 Current number of lock objects allocated >>>> 160 Number of lock object partitions >>>> 8191 Size of object hash table >>>> 126 Number of current locks >>>> 1005 Maximum number of locks at any one time >>>> 33 Maximum number of locks in any one bucket >>>> 3987 Maximum number of locks stolen by for an >>>> empty partition >>>> 75 Maximum number of locks stolen for any one >>>> partition >>>> 309 Number of current lockers >>>> 313 Maximum number of lockers at any one time >>>> 121 Number of current lock objects >>>> 676 Maximum number of lock objects at any one time >>>> 3 Maximum number of lock objects in any one bucket >>>> 69 Maximum number of objects stolen by for an >>>> empty partition >>>> 5 Maximum number of objects stolen for any one >>>> partition >>>> 22M Total number of locks requested (22009642) >>>> 21M Total number of locks released (21992878) >>>> 0 Total number of locks upgraded >>>> 95 Total number of locks downgraded >>>> 3348 Lock requests not available due to >>>> conflicts, for which we waited >>>> 12646 Lock requests not available due to >>>> conflicts, for which we did not wait >>>> 0 Number of deadlocks >>>> 0 Lock timeout value >>>> 0 Number of locks that have timed out >>>> 0 Transaction timeout value >>>> 0 Number of transactions that have timed out >>>> 2MB 560KB Region size >>>> 71719 The number of partition locks that required >>>> waiting (0%) >>>> 23105 The maximum number of times any partition >>>> lock was waited for (0%) >>>> 0 The number of object queue operations that >>>> required waiting (0%) >>>> 163 The number of locker allocations that >>>> required waiting (0%) >>>> 3172 The number of region locks that required >>>> waiting (0%) >>>> 5 Maximum hash bucket length >>>> =-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-= >>>> Lock REGINFO information: >>>> Environment Region type >>>> 1 Region ID >>>> /var/lib/dirsrv/slapd-SPRINKLR-COM/db/__db.001 >>>> Region name >>>> 0x7f67ef869000 Region address >>>> 0x7f67ef8690a0 Region allocation head >>>> 0x7f67ef8712b0 Region primary address >>>> 0 Region maximum allocation >>>> 0 Region allocated >>>> Region allocations: 1581345 allocations, 0 failures, >>>> 1580794 frees, 6 longest >>>> Allocations by power-of-two sizes: >>>> 1KB 1581329 >>>> 2KB 3 >>>> 4KB 4 >>>> 8KB 3 >>>> 16KB 3 >>>> 32KB 1 >>>> 64KB 0 >>>> 128KB 0 >>>> 256KB 2 >>>> 512KB 0 >>>> 1024KB 1 >>>> REGION_SHARED Region flags >>>> =-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-= >>>> Lock region parameters: >>>> 2 Lock region region mutex [3172/14M 0% !Own] >>>> >>>> 16381 locker table size >>>> 8191 object table size >>>> 34128 obj_off >>>> 900216 locker_off >>>> 0 need_dd >>>> =-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-= >>>> Lock conflict matrix: >>>> =-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-= >>>> Locks grouped by lockers: >>>> Locker Mode Count Status ----------------- >>>> Object --------------- >>>> 2 dd=308 locks held 1 write locks 0 >>>> pid/thread 4845/140085720139840 flags 10 priority 100 >>>> 2 READ 1 HELD userRoot/id2entry.db >>>> handle 0 >>>> 3 dd=307 locks held 0 write locks 0 >>>> pid/thread 4845/140084829472512 flags 0 priority 100 >>>> 4 dd=306 locks held 1 write locks 0 >>>> pid/thread 4845/140085720139840 flags 10 priority 100 >>>> 4 READ 1 HELD ipaca/id2entry.db >>>> handle 0 >>>> 5 dd=305 locks held 0 write locks 0 >>>> pid/thread 4845/140084837865216 flags 0 priority 100 >>>> 6 dd=304 locks held 1 write locks 0 >>>> pid/thread 4845/140085720139840 flags 10 priority 100 >>>> 6 READ 1 HELD ipaca/entryrdn.db >>>> handle 0 >>>> 7 dd=303 locks held 0 write locks 0 >>>> pid/thread 4845/140084804294400 flags 0 priority 100 >>>> 8 dd=302 locks held 0 write locks 0 >>>> pid/thread 4845/140084720432896 flags 0 priority 100 >>>> 9 dd=301 locks held 0 write locks 0 >>>> pid/thread 4845/140084837865216 flags 0 priority 100 >>>> a dd=300 locks held 1 write locks 0 >>>> pid/thread 4845/140085720139840 flags 10 priority 100 >>>> a READ 1 HELD >>>> ipaca/vlv#allcertspkitomcatindex.db handle 0 >>>> e dd=299 locks held 1 write locks 0 >>>> pid/thread 4845/140085720139840 flags 10 priority 100 >>>> e READ 1 HELD >>>> ipaca/vlv#allnonrevokedcertspkitomcatindex.db >>>> handle 0 >>>> 10 dd=298 locks held 1 write locks 0 >>>> pid/thread 4845/140085720139840 flags 10 priority 100 >>>> 10 READ 1 HELD >>>> ipaca/vlv#allrevokedcertspkitomcatindex.db handle 0 >>>> 11 dd=297 locks held 1 write locks 0 >>>> pid/thread 4845/140085720139840 flags 10 priority 100 >>>> 11 READ 1 HELD >>>> ipaca/vlv#allrevokedcertsnotafterpkitomcatindex.db >>>> handle 0 >>>> 14 dd=296 locks held 1 write locks 0 >>>> pid/thread 4845/140085720139840 flags 10 priority 100 >>>> 14 READ 1 HELD >>>> ipaca/vlv#allrevokedorrevokedexpiredcertspkitomcatindex.db >>>> handle 0 >>>> 15 dd=295 locks held 1 write locks 0 >>>> pid/thread 4845/140085720139840 flags 10 priority 100 >>>> 15 READ 1 HELD >>>> ipaca/vlv#allvalidcertspkitomcatindex.db handle 0 >>>> 16 dd=294 locks held 1 write locks 0 >>>> pid/thread 4845/140085720139840 flags 10 priority 100 >>>> 16 READ 1 HELD >>>> ipaca/vlv#allvalidcertsnotafterpkitomcatindex.db >>>> handle 0 >>>> 17 dd=293 locks held 1 write locks 0 >>>> pid/thread 4845/140085720139840 flags 10 priority 100 >>>> 17 READ 1 HELD >>>> ipaca/vlv#allvalidorrevokedcertspkitomcatindex.db >>>> handle 0 >>>> 18 dd=292 locks held 1 write locks 0 >>>> pid/thread 4845/140085720139840 flags 10 priority 100 >>>> 18 READ 1 HELD >>>> ipaca/vlv#caallpkitomcatindex.db handle 0 >>>> 1d dd=291 locks held 1 write locks 0 >>>> pid/thread 4845/140085720139840 flags 10 priority 100 >>>> 1d READ 1 HELD >>>> ipaca/vlv#cacompletepkitomcatindex.db handle 0 >>>> 1e dd=290 locks held 1 write locks 0 >>>> pid/thread 4845/140085720139840 flags 10 priority 100 >>>> 1e READ 1 HELD >>>> ipaca/vlv#cacompleteenrollmentpkitomcatindex.db >>>> handle 0 >>>> 20 dd=289 locks held 1 write locks 0 >>>> pid/thread 4845/140085720139840 flags 10 priority 100 >>>> 20 READ 1 HELD >>>> ipaca/vlv#cacompleterevocationpkitomcatindex.db >>>> handle 0 >>>> 21 dd=288 locks held 1 write locks 0 >>>> pid/thread 4845/140085720139840 flags 10 priority 100 >>>> 21 READ 1 HELD >>>> ipaca/vlv#caenrollmentpkitomcatindex.db handle 0 >>>> 2b dd=287 locks held 1 write locks 0 >>>> pid/thread 4845/140085720139840 flags 10 priority 100 >>>> 2b READ 1 HELD >>>> ipaca/vlv#carevocationpkitomcatindex.db handle 0 >>>> 2c dd=286 locks held 1 write locks 0 >>>> pid/thread 4845/140085720139840 flags 10 priority 100 >>>> 2c READ 1 HELD changelog/id2entry.db >>>> handle 0 >>>> 2d dd=285 locks held 0 write locks 0 >>>> pid/thread 4845/140085719607040 flags 0 priority 100 >>>> 2e dd=284 locks held 1 write locks 0 >>>> pid/thread 4845/140085720139840 flags 10 priority 100 >>>> 2e READ 1 HELD changelog/entryusn.db >>>> handle 0 >>>> 2f dd=283 locks held 0 write locks 0 >>>> pid/thread 4845/140085720139840 flags 0 priority 100 >>>> 30 dd=282 locks held 1 write locks 0 >>>> pid/thread 4845/140085720139840 flags 10 priority 100 >>>> 30 READ 1 HELD userRoot/entryusn.db >>>> handle 0 >>>> 31 dd=281 locks held 0 write locks 0 >>>> pid/thread 4845/140085720139840 flags 0 priority 100 >>>> 32 dd=280 locks held 1 write locks 0 >>>> pid/thread 4845/140085720139840 flags 10 priority 100 >>>> 32 READ 1 HELD ipaca/entryusn.db >>>> handle 0 >>>> 33 dd=279 locks held 0 write locks 0 >>>> pid/thread 4845/140085720139840 flags 0 priority 100 >>>> 34 dd=278 locks held 1 write locks 0 >>>> pid/thread 4845/140085720139840 flags 10 priority 100 >>>> 34 READ 1 HELD userRoot/entryrdn.db >>>> handle 0 >>>> 35 dd=277 locks held 0 write locks 0 >>>> pid/thread 4845/140084544186112 flags 0 priority 100 >>>> 36 dd=276 locks held 0 write locks 0 >>>> pid/thread 4845/140084636505856 flags 0 priority 100 >>>> 37 dd=275 locks held 0 write locks 0 >>>> pid/thread 4845/140084854650624 flags 0 priority 100 >>>> 38 dd=274 locks held 1 write locks 0 >>>> pid/thread 4845/140085720139840 flags 10 priority 100 >>>> 38 READ 1 HELD userRoot/objectclass.db >>>> handle 0 >>>> 39 dd=273 locks held 0 write locks 0 >>>> pid/thread 4845/140084569364224 flags 0 priority 100 >>>> 3a dd=272 locks held 1 write locks 0 >>>> pid/thread 4845/140085720139840 flags 10 priority 100 >>>> 3a READ 1 HELD userRoot/ancestorid.db >>>> handle 0 >>>> 3b dd=271 locks held 0 write locks 0 >>>> pid/thread 4845/140084854650624 flags 0 priority 100 >>>> 3c dd=270 locks held 1 write locks 0 >>>> pid/thread 4845/140085720139840 flags 10 priority 100 >>>> 3c READ 1 HELD changelog/entryrdn.db >>>> handle 0 >>>> 3d dd=269 locks held 0 write locks 0 >>>> pid/thread 4845/140085719607040 flags 0 priority 100 >>>> 3e dd=268 locks held 0 write locks 0 >>>> pid/thread 4845/140084602935040 flags 0 priority 100 >>>> 3f dd=267 locks held 0 write locks 0 >>>> pid/thread 4845/140085719607040 flags 0 priority 100 >>>> 40 dd=266 locks held 1 write locks 0 >>>> pid/thread 4845/140085720139840 flags 10 priority 100 >>>> 40 READ 1 HELD changelog/objectclass.db >>>> handle 0 >>>> 41 dd=265 locks held 0 write locks 0 >>>> pid/thread 4845/140085719607040 flags 0 priority 100 >>>> 42 dd=264 locks held 0 write locks 0 >>>> pid/thread 4845/140085719607040 flags 0 priority 100 >>>> 43 dd=263 locks held 0 write locks 0 >>>> pid/thread 4845/140084653291264 flags 0 priority 100 >>>> 44 dd=262 locks held 1 write locks 0 >>>> pid/thread 4845/140085720139840 flags 10 priority 100 >>>> 44 READ 1 HELD ipaca/objectclass.db >>>> handle 0 >>>> 45 dd=261 locks held 0 write locks 0 >>>> pid/thread 4845/140084577756928 flags 0 priority 100 >>>> 46 dd=260 locks held 0 write locks 0 >>>> pid/thread 4845/140084720432896 flags 0 priority 100 >>>> 47 dd=259 locks held 1 write locks 0 >>>> pid/thread 4845/140085720139840 flags 10 priority 100 >>>> 47 READ 1 HELD changelog/aci.db >>>> handle 0 >>>> 48 dd=258 locks held 0 write locks 0 >>>> pid/thread 4845/140085720139840 flags 0 priority 100 >>>> 49 dd=257 locks held 1 write locks 0 >>>> pid/thread 4845/140085720139840 flags 10 priority 100 >>>> 49 READ 1 HELD userRoot/aci.db >>>> handle 0 >>>> 4a dd=256 locks held 0 write locks 0 >>>> pid/thread 4845/140085720139840 flags 0 priority 100 >>>> 4b dd=255 locks held 0 write locks 0 >>>> pid/thread 4845/140084535793408 flags 0 priority 100 >>>> 4c dd=254 locks held 1 write locks 0 >>>> pid/thread 4845/140085720139840 flags 10 priority 100 >>>> 4c READ 1 HELD ipaca/aci.db handle 0 >>>> 4d dd=253 locks held 0 write locks 0 >>>> pid/thread 4845/140085720139840 flags 0 priority 100 >>>> 4e dd=252 locks held 1 write locks 0 >>>> pid/thread 4845/140085720139840 flags 10 priority 100 >>>> 4e READ 1 HELD userRoot/parentid.db >>>> handle 0 >>>> 4f dd=251 locks held 0 write locks 0 >>>> pid/thread 4845/140085720139840 flags 0 priority 100 >>>> 50 dd=250 locks held 0 write locks 0 >>>> pid/thread 4845/140085719607040 flags 0 priority 100 >>>> 51 dd=249 locks held 1 write locks 0 >>>> pid/thread 4845/140085720139840 flags 10 priority 100 >>>> 51 READ 1 HELD changelog/nsuniqueid.db >>>> handle 0 >>>> 52 dd=248 locks held 1 write locks 0 >>>> pid/thread 4845/140085720139840 flags 10 priority 100 >>>> 52 READ 1 HELD changelog/changenumber.db >>>> handle 0 >>>> 53 dd=247 locks held 0 write locks 0 >>>> pid/thread 4845/140084602935040 flags 0 priority 100 >>>> 54 dd=246 locks held 0 write locks 0 >>>> pid/thread 4845/140085719607040 flags 0 priority 100 >>>> 55 dd=245 locks held 1 write locks 0 >>>> pid/thread 4845/140085719607040 flags 10 priority 100 >>>> 55 READ 1 HELD >>>> changelog/targetuniqueid.db handle 0 >>>> 56 dd=244 locks held 1 write locks 0 >>>> pid/thread 4845/140085719607040 flags 10 priority 100 >>>> 56 READ 1 HELD changelog/parentid.db >>>> handle 0 >>>> 57 dd=243 locks held 1 write locks 0 >>>> pid/thread 4845/140085719607040 flags 10 priority 100 >>>> 57 READ 1 HELD changelog/ancestorid.db >>>> handle 0 >>>> 58 dd=242 locks held 1 write locks 0 >>>> pid/thread 4845/140085719607040 flags 10 priority 100 >>>> 58 READ 1 HELD >>>> changelog/numsubordinates.db handle 0 >>>> 59 dd=241 locks held 0 write locks 0 >>>> pid/thread 4845/140085719607040 flags 0 priority 100 >>>> 5a dd=240 locks held 0 write locks 0 >>>> pid/thread 4845/140084544186112 flags 0 priority 100 >>>> 5b dd=239 locks held 1 write locks 0 >>>> pid/thread 4845/140084577756928 flags 10 priority 100 >>>> 5b READ 1 HELD >>>> userRoot/krbPrincipalName.db handle 0 >>>> 5c dd=238 locks held 0 write locks 0 >>>> pid/thread 4845/140084636505856 flags 0 priority 100 >>>> 5d dd=237 locks held 0 write locks 0 >>>> pid/thread 4845/140084695254784 flags 0 priority 100 >>>> 5e dd=236 locks held 1 write locks 0 >>>> pid/thread 4845/140084560971520 flags 10 priority 100 >>>> 5e READ 1 HELD >>>> userRoot/ipakrbprincipalalias.db handle 0 >>>> 5f dd=235 locks held 0 write locks 0 >>>> pid/thread 4845/140084703647488 flags 0 priority 100 >>>> 60 dd=234 locks held 0 write locks 0 >>>> pid/thread 4845/140084653291264 flags 0 priority 100 >>>> 61 dd=233 locks held 0 write locks 0 >>>> pid/thread 4845/140084577756928 flags 0 priority 100 >>>> 62 dd=232 locks held 0 write locks 0 >>>> pid/thread 4845/140084686862080 flags 0 priority 100 >>>> 63 dd=231 locks held 1 write locks 0 >>>> pid/thread 4845/140084611327744 flags 10 priority 100 >>>> 63 READ 1 HELD changelog/seeAlso.db >>>> handle 0 >>>> 64 dd=230 locks held 0 write locks 0 >>>> pid/thread 4845/140084686862080 flags 0 priority 100 >>>> 65 dd=229 locks held 1 write locks 0 >>>> pid/thread 4845/140084611327744 flags 10 priority 100 >>>> 65 READ 1 HELD userRoot/seeAlso.db >>>> handle 0 >>>> 66 dd=228 locks held 0 write locks 0 >>>> pid/thread 4845/140084636505856 flags 0 priority 100 >>>> 67 dd=227 locks held 1 write locks 0 >>>> pid/thread 4845/140084611327744 flags 10 priority 100 >>>> 67 READ 1 HELD ipaca/seeAlso.db >>>> handle 0 >>>> 68 dd=226 locks held 0 write locks 0 >>>> pid/thread 4845/140084686862080 flags 0 priority 100 >>>> 69 dd=225 locks held 1 write locks 0 >>>> pid/thread 4845/140084678469376 flags 10 priority 100 >>>> 69 READ 1 HELD ipaca/parentid.db >>>> handle 0 >>>> 6a dd=224 locks held 0 write locks 0 >>>> pid/thread 4845/140084678469376 flags 0 priority 100 >>>> 6b dd=223 locks held 0 write locks 0 >>>> pid/thread 4845/140084829472512 flags 0 priority 100 >>>> 6c dd=222 locks held 0 write locks 0 >>>> pid/thread 4845/140084653291264 flags 0 priority 100 >>>> 6d dd=221 locks held 0 write locks 0 >>>> pid/thread 4845/140084569364224 flags 0 priority 100 >>>> 6e dd=220 locks held 0 write locks 0 >>>> pid/thread 4845/140084636505856 flags 0 priority 100 >>>> 6e READ 1 WAIT userRoot/objectclass.db >>>> page 18 >>>> 6f dd=219 locks held 0 write locks 0 >>>> pid/thread 4845/140084544186112 flags 0 priority 100 >>>> 70 dd=218 locks held 0 write locks 0 >>>> pid/thread 4845/140084569364224 flags 0 priority 100 >>>> 71 dd=217 locks held 0 write locks 0 >>>> pid/thread 4845/140084619720448 flags 0 priority 100 >>>> 72 dd=216 locks held 0 write locks 0 >>>> pid/thread 4845/140084552578816 flags 0 priority 100 >>>> 73 dd=215 locks held 0 write locks 0 >>>> pid/thread 4845/140084829472512 flags 0 priority 100 >>>> 74 dd=214 locks held 0 write locks 0 >>>> pid/thread 4845/140084628113152 flags 0 priority 100 >>>> 75 dd=213 locks held 0 write locks 0 >>>> pid/thread 4845/140084686862080 flags 0 priority 100 >>>> 76 dd=212 locks held 1 write locks 0 >>>> pid/thread 4845/140084560971520 flags 10 priority 100 >>>> 76 READ 1 HELD userRoot/ipauniqueid.db >>>> handle 0 >>>> 77 dd=211 locks held 0 write locks 0 >>>> pid/thread 4845/140084661683968 flags 0 priority 100 >>>> 78 dd=210 locks held 0 write locks 0 >>>> pid/thread 4845/140084846257920 flags 0 priority 100 >>>> 79 dd=209 locks held 0 write locks 0 >>>> pid/thread 4845/140084628113152 flags 0 priority 100 >>>> 7a dd=208 locks held 1 write locks 0 >>>> pid/thread 4845/140084560971520 flags 10 priority 100 >>>> 7a READ 1 HELD userRoot/nsuniqueid.db >>>> handle 0 >>>> 7b dd=207 locks held 1 write locks 0 >>>> pid/thread 4845/140084560971520 flags 10 priority 100 >>>> 7b READ 1 HELD userRoot/cn.db >>>> handle 0 >>>> 7c dd=206 locks held 1 write locks 0 >>>> pid/thread 4845/140084560971520 flags 10 priority 100 >>>> 7c READ 1 HELD userRoot/fqdn.db >>>> handle 0 >>>> 7d dd=205 locks held 1 write locks 0 >>>> pid/thread 4845/140084560971520 flags 10 priority 100 >>>> 7d READ 1 HELD userRoot/managedby.db >>>> handle 0 >>>> 7e dd=204 locks held 1 write locks 0 >>>> pid/thread 4845/140084560971520 flags 10 priority 100 >>>> 7e READ 1 HELD >>>> userRoot/numsubordinates.db handle 0 >>>> 7f dd=203 locks held 1 write locks 0 >>>> pid/thread 4845/140084678469376 flags 10 priority 100 >>>> 7f READ 1 HELD userRoot/member.db >>>> handle 0 >>>> 80 dd=202 locks held 0 write locks 0 >>>> pid/thread 4845/140084712040192 flags 0 priority 100 >>>> 81 dd=201 locks held 1 write locks 0 >>>> pid/thread 4845/140084678469376 flags 10 priority 100 >>>> 81 READ 1 HELD userRoot/memberUser.db >>>> handle 0 >>>> 82 dd=200 locks held 0 write locks 0 >>>> pid/thread 4845/140084577756928 flags 0 priority 100 >>>> 83 dd=199 locks held 1 write locks 0 >>>> pid/thread 4845/140084678469376 flags 10 priority 100 >>>> 83 READ 1 HELD userRoot/memberHost.db >>>> handle 0 >>>> 84 dd=198 locks held 0 write locks 0 >>>> pid/thread 4845/140084712040192 flags 0 priority 100 >>>> 85 dd=197 locks held 0 write locks 0 >>>> pid/thread 4845/140084577756928 flags 0 priority 100 >>>> 86 dd=196 locks held 0 write locks 0 >>>> pid/thread 4845/140084670076672 flags 0 priority 100 >>>> 87 dd=195 locks held 0 write locks 0 >>>> pid/thread 4845/140084577756928 flags 0 priority 100 >>>> 88 dd=194 locks held 0 write locks 0 >>>> pid/thread 4845/140084846257920 flags 0 priority 100 >>>> 89 dd=193 locks held 0 write locks 0 >>>> pid/thread 4845/140084552578816 flags 0 priority 100 >>>> 8a dd=192 locks held 0 write locks 0 >>>> pid/thread 4845/140084703647488 flags 0 priority 100 >>>> 8b dd=191 locks held 0 write locks 0 >>>> pid/thread 4845/140084544186112 flags 0 priority 100 >>>> 8c dd=190 locks held 0 write locks 0 >>>> pid/thread 4845/140084846257920 flags 0 priority 100 >>>> 8d dd=189 locks held 0 write locks 0 >>>> pid/thread 4845/140084661683968 flags 0 priority 100 >>>> 8e dd=188 locks held 1 write locks 0 >>>> pid/thread 4845/140085067675392 flags 10 priority 100 >>>> 8e READ 1 HELD userRoot/macAddress.db >>>> handle 0 >>>> 8f dd=187 locks held 0 write locks 0 >>>> pid/thread 4845/140085067675392 flags 0 priority 100 >>>> 90 dd=186 locks held 0 write locks 0 >>>> pid/thread 4845/140084703647488 flags 0 priority 100 >>>> 91 dd=185 locks held 0 write locks 0 >>>> pid/thread 4845/140084552578816 flags 0 priority 100 >>>> 92 dd=184 locks held 0 write locks 0 >>>> pid/thread 4845/140084560971520 flags 0 priority 100 >>>> 92 READ 1 WAIT userRoot/objectclass.db >>>> page 9 >>>> 93 dd=183 locks held 0 write locks 0 >>>> pid/thread 4845/140084577756928 flags 0 priority 100 >>>> 94 dd=182 locks held 0 write locks 0 >>>> pid/thread 4845/140084821079808 flags 0 priority 100 >>>> 94 READ 1 WAIT userRoot/objectclass.db >>>> page 18 >>>> 95 dd=181 locks held 0 write locks 0 >>>> pid/thread 4845/140084544186112 flags 0 priority 100 >>>> 96 dd=180 locks held 0 write locks 0 >>>> pid/thread 4845/140084804294400 flags 0 priority 100 >>>> 97 dd=179 locks held 1 write locks 0 >>>> pid/thread 4845/140084712040192 flags 10 priority 100 >>>> 97 READ 1 HELD userRoot/uid.db >>>> handle 0 >>>> 98 dd=178 locks held 0 write locks 0 >>>> pid/thread 4845/140084552578816 flags 0 priority 100 >>>> 99 dd=177 locks held 0 write locks 0 >>>> pid/thread 4845/140084577756928 flags 0 priority 100 >>>> 9a dd=176 locks held 0 write locks 0 >>>> pid/thread 4845/140084602935040 flags 0 priority 100 >>>> 9b dd=175 locks held 1 write locks 0 >>>> pid/thread 4845/140085076068096 flags 10 priority 100 >>>> 9b READ 1 HELD userRoot/uniquemember.db >>>> handle 0 >>>> 9c dd=174 locks held 1 write locks 0 >>>> pid/thread 4845/140085076068096 flags 10 priority 100 >>>> 9c READ 1 HELD userRoot/owner.db >>>> handle 0 >>>> 9d dd=173 locks held 1 write locks 0 >>>> pid/thread 4845/140085076068096 flags 10 priority 100 >>>> 9d READ 1 HELD >>>> userRoot/ipaallowedtarget.db handle 0 >>>> 9e dd=172 locks held 1 write locks 0 >>>> pid/thread 4845/140085076068096 flags 10 priority 100 >>>> 9e READ 1 HELD >>>> userRoot/ipaMemberCertProfile.db handle 0 >>>> 9f dd=171 locks held 1 write locks 0 >>>> pid/thread 4845/140085076068096 flags 10 priority 100 >>>> 9f READ 1 HELD >>>> userRoot/ipaassignedidview.db handle 0 >>>> a0 dd=170 locks held 1 write locks 0 >>>> pid/thread 4845/140085076068096 flags 10 priority 100 >>>> a0 READ 1 HELD userRoot/secretary.db >>>> handle 0 >>>> a1 dd=169 locks held 1 write locks 0 >>>> pid/thread 4845/140085076068096 flags 10 priority 100 >>>> a1 READ 1 HELD userRoot/memberdenycmd.db >>>> handle 0 >>>> a2 dd=168 locks held 1 write locks 0 >>>> pid/thread 4845/140085076068096 flags 10 priority 100 >>>> a2 READ 1 HELD >>>> userRoot/memberallowcmd.db handle 0 >>>> a3 dd=167 locks held 1 write locks 0 >>>> pid/thread 4845/140085076068096 flags 10 priority 100 >>>> a3 READ 1 HELD userRoot/manager.db >>>> handle 0 >>>> a4 dd=166 locks held 1 write locks 0 >>>> pid/thread 4845/140085076068096 flags 10 priority 100 >>>> a4 READ 1 HELD userRoot/ipasudorunas.db >>>> handle 0 >>>> a5 dd=165 locks held 1 write locks 0 >>>> pid/thread 4845/140085076068096 flags 10 priority 100 >>>> a5 READ 1 HELD userRoot/ipaMemberCa.db >>>> handle 0 >>>> a6 dd=164 locks held 1 write locks 0 >>>> pid/thread 4845/140085076068096 flags 10 priority 100 >>>> a6 READ 1 HELD >>>> userRoot/ipasudorunasgroup.db handle 0 >>>> a7 dd=163 locks held 1 write locks 0 >>>> pid/thread 4845/140085076068096 flags 10 priority 100 >>>> a7 READ 1 HELD userRoot/sourcehost.db >>>> handle 0 >>>> a8 dd=162 locks held 1 write locks 0 >>>> pid/thread 4845/140085076068096 flags 10 priority 100 >>>> a8 READ 1 HELD userRoot/memberservice.db >>>> handle 0 >>>> a9 dd=161 locks held 1 write locks 0 >>>> pid/thread 4845/140085076068096 flags 10 priority 100 >>>> a9 READ 1 HELD >>>> userRoot/ipatokenradiusconfiglink.db handle 0 >>>> aa dd=160 locks held 0 write locks 0 >>>> pid/thread 4845/140084703647488 flags 0 priority 100 >>>> ab dd=159 locks held 0 write locks 0 >>>> pid/thread 4845/140084821079808 flags 0 priority 100 >>>> ac dd=158 locks held 0 write locks 0 >>>> pid/thread 4845/140084628113152 flags 0 priority 100 >>>> ad dd=157 locks held 0 write locks 0 >>>> pid/thread 4845/140084829472512 flags 0 priority 100 >>>> ae dd=156 locks held 0 write locks 0 >>>> pid/thread 4845/140084829472512 flags 0 priority 100 >>>> af dd=155 locks held 0 write locks 0 >>>> pid/thread 4845/140084854650624 flags 0 priority 100 >>>> b0 dd=154 locks held 0 write locks 0 >>>> pid/thread 4845/140084854650624 flags 0 priority 100 >>>> b1 dd=153 locks held 0 write locks 0 >>>> pid/thread 4845/140084837865216 flags 0 priority 100 >>>> b2 dd=152 locks held 0 write locks 0 >>>> pid/thread 4845/140084804294400 flags 0 priority 100 >>>> b3 dd=151 locks held 0 write locks 0 >>>> pid/thread 4845/140084812687104 flags 0 priority 100 >>>> b4 dd=150 locks held 0 write locks 0 >>>> pid/thread 4845/140084804294400 flags 0 priority 100 >>>> b5 dd=149 locks held 0 write locks 0 >>>> pid/thread 4845/140084837865216 flags 0 priority 100 >>>> b6 dd=148 locks held 0 write locks 0 >>>> pid/thread 4845/140084569364224 flags 0 priority 100 >>>> b7 dd=147 locks held 0 write locks 0 >>>> pid/thread 4845/140084829472512 flags 0 priority 100 >>>> b8 dd=146 locks held 0 write locks 0 >>>> pid/thread 4845/140084586149632 flags 0 priority 100 >>>> b9 dd=145 locks held 0 write locks 0 >>>> pid/thread 4845/140084653291264 flags 0 priority 100 >>>> ba dd=144 locks held 0 write locks 0 >>>> pid/thread 4845/140084636505856 flags 0 priority 100 >>>> bb dd=143 locks held 0 write locks 0 >>>> pid/thread 4845/140084804294400 flags 0 priority 100 >>>> bc dd=142 locks held 0 write locks 0 >>>> pid/thread 4845/140084804294400 flags 0 priority 100 >>>> bd dd=141 locks held 0 write locks 0 >>>> pid/thread 4845/140084695254784 flags 0 priority 100 >>>> be dd=140 locks held 0 write locks 0 >>>> pid/thread 4845/140084670076672 flags 0 priority 100 >>>> bf dd=139 locks held 0 write locks 0 >>>> pid/thread 4845/140084628113152 flags 0 priority 100 >>>> c0 dd=138 locks held 0 write locks 0 >>>> pid/thread 4845/140084661683968 flags 0 priority 100 >>>> c1 dd=137 locks held 0 write locks 0 >>>> pid/thread 4845/140084712040192 flags 0 priority 100 >>>> c2 dd=136 locks held 0 write locks 0 >>>> pid/thread 4845/140084661683968 flags 0 priority 100 >>>> c3 dd=135 locks held 0 write locks 0 >>>> pid/thread 4845/140084577756928 flags 0 priority 100 >>>> c4 dd=134 locks held 0 write locks 0 >>>> pid/thread 4845/140084854650624 flags 0 priority 100 >>>> c5 dd=133 locks held 1 write locks 0 >>>> pid/thread 4845/140084644898560 flags 10 priority 100 >>>> c5 READ 1 HELD userRoot/uidnumber.db >>>> handle 0 >>>> c6 dd=132 locks held 0 write locks 0 >>>> pid/thread 4845/140084594542336 flags 0 priority 100 >>>> c8 dd=131 locks held 1 write locks 0 >>>> pid/thread 4845/140084678469376 flags 10 priority 100 >>>> c8 READ 1 HELD ipaca/certstatus.db >>>> handle 0 >>>> c9 dd=130 locks held 0 write locks 0 >>>> pid/thread 4845/140084720432896 flags 0 priority 100 >>>> ca dd=129 locks held 0 write locks 0 >>>> pid/thread 4845/140084619720448 flags 0 priority 100 >>>> cb dd=128 locks held 0 write locks 0 >>>> pid/thread 4845/140084821079808 flags 0 priority 100 >>>> cc dd=127 locks held 0 write locks 0 >>>> pid/thread 4845/140084594542336 flags 0 priority 100 >>>> cd dd=126 locks held 0 write locks 0 >>>> pid/thread 4845/140084560971520 flags 0 priority 100 >>>> ce dd=125 locks held 1 write locks 0 >>>> pid/thread 4845/140084837865216 flags 10 priority 100 >>>> ce READ 1 HELD userRoot/gidnumber.db >>>> handle 0 >>>> cf dd=124 locks held 0 write locks 0 >>>> pid/thread 4845/140084720432896 flags 0 priority 100 >>>> d0 dd=123 locks held 0 write locks 0 >>>> pid/thread 4845/140084569364224 flags 0 priority 100 >>>> d1 dd=122 locks held 1 write locks 0 >>>> pid/thread 4845/140084678469376 flags 10 priority 100 >>>> d1 READ 1 HELD userRoot/nscpEntryDN.db >>>> handle 0 >>>> d2 dd=121 locks held 0 write locks 0 >>>> pid/thread 4845/140084695254784 flags 0 priority 100 >>>> d3 dd=120 locks held 0 write locks 0 >>>> pid/thread 4845/140084569364224 flags 0 priority 100 >>>> d4 dd=119 locks held 0 write locks 0 >>>> pid/thread 4845/140084653291264 flags 0 priority 100 >>>> d5 dd=118 locks held 0 write locks 0 >>>> pid/thread 4845/140084829472512 flags 0 priority 100 >>>> d6 dd=117 locks held 0 write locks 0 >>>> pid/thread 4845/140084661683968 flags 0 priority 100 >>>> d7 dd=116 locks held 0 write locks 0 >>>> pid/thread 4845/140084804294400 flags 0 priority 100 >>>> d8 dd=115 locks held 1 write locks 0 >>>> pid/thread 4845/140084821079808 flags 10 priority 100 >>>> d8 READ 1 HELD ipaca/description.db >>>> handle 0 >>>> d9 dd=114 locks held 0 write locks 0 >>>> pid/thread 4845/140084661683968 flags 0 priority 100 >>>> da dd=113 locks held 0 write locks 0 >>>> pid/thread 4845/140084837865216 flags 0 priority 100 >>>> db dd=112 locks held 1 write locks 0 >>>> pid/thread 4845/140084678469376 flags 10 priority 100 >>>> db READ 1 HELD ipaca/cn.db handle 0 >>>> dc dd=111 locks held 0 write locks 0 >>>> pid/thread 4845/140084577756928 flags 0 priority 100 >>>> dd dd=110 locks held 1 write locks 0 >>>> pid/thread 4845/140084678469376 flags 10 priority 100 >>>> dd READ 1 HELD ipaca/ancestorid.db >>>> handle 0 >>>> de dd=109 locks held 0 write locks 0 >>>> pid/thread 4845/140084577756928 flags 0 priority 100 >>>> df dd=108 locks held 0 write locks 0 >>>> pid/thread 4845/140084703647488 flags 0 priority 100 >>>> e0 dd=107 locks held 0 write locks 0 >>>> pid/thread 4845/140084569364224 flags 0 priority 100 >>>> e1 dd=106 locks held 0 write locks 0 >>>> pid/thread 4845/140084569364224 flags 0 priority 100 >>>> e2 dd=105 locks held 0 write locks 0 >>>> pid/thread 4845/140084821079808 flags 0 priority 100 >>>> e3 dd=104 locks held 1 write locks 0 >>>> pid/thread 4845/140084686862080 flags 10 priority 100 >>>> e3 READ 1 HELD ipaca/serialno.db >>>> handle 0 >>>> e4 dd=103 locks held 0 write locks 0 >>>> pid/thread 4845/140084837865216 flags 0 priority 100 >>>> e5 dd=102 locks held 1 write locks 0 >>>> pid/thread 4845/140084695254784 flags 10 priority 100 >>>> e5 READ 1 HELD ipaca/revInfo.db >>>> handle 0 >>>> e6 dd=101 locks held 1 write locks 0 >>>> pid/thread 4845/140084695254784 flags 10 priority 100 >>>> e6 READ 1 HELD ipaca/revokedby.db >>>> handle 0 >>>> e7 dd=100 locks held 1 write locks 0 >>>> pid/thread 4845/140084695254784 flags 10 priority 100 >>>> e7 READ 1 HELD ipaca/revokedOn.db >>>> handle 0 >>>> e8 dd=99 locks held 0 write locks 0 pid/thread >>>> 4845/140084829472512 flags 0 priority 100 >>>> e9 dd=98 locks held 0 write locks 0 pid/thread >>>> 4845/140084846257920 flags 0 priority 100 >>>> ea dd=97 locks held 1 write locks 0 pid/thread >>>> 4845/140084703647488 flags 10 priority 100 >>>> ea READ 1 HELD ipaca/nsuniqueid.db >>>> handle 0 >>>> eb dd=96 locks held 1 write locks 0 pid/thread >>>> 4845/140084703647488 flags 10 priority 100 >>>> eb READ 1 HELD ipaca/requestid.db >>>> handle 0 >>>> ec dd=95 locks held 1 write locks 0 pid/thread >>>> 4845/140084703647488 flags 10 priority 100 >>>> ec READ 1 HELD ipaca/requeststate.db >>>> handle 0 >>>> ed dd=94 locks held 1 write locks 0 pid/thread >>>> 4845/140084703647488 flags 10 priority 100 >>>> ed READ 1 HELD ipaca/dateOfCreate.db >>>> handle 0 >>>> ee dd=93 locks held 1 write locks 0 pid/thread >>>> 4845/140084703647488 flags 10 priority 100 >>>> ee READ 1 HELD ipaca/requesttype.db >>>> handle 0 >>>> ef dd=92 locks held 1 write locks 0 pid/thread >>>> 4845/140084703647488 flags 10 priority 100 >>>> ef READ 1 HELD ipaca/numsubordinates.db >>>> handle 0 >>>> f0 dd=91 locks held 1 write locks 0 pid/thread >>>> 4845/140084611327744 flags 10 priority 100 >>>> f0 READ 1 HELD >>>> userRoot/userCertificate.db handle 0 >>>> f1 dd=90 locks held 0 write locks 0 pid/thread >>>> 4845/140084821079808 flags 0 priority 100 >>>> f2 dd=89 locks held 0 write locks 0 pid/thread >>>> 4845/140084569364224 flags 0 priority 100 >>>> f3 dd=88 locks held 0 write locks 0 pid/thread >>>> 4845/140084602935040 flags 0 priority 100 >>>> f4 dd=87 locks held 0 write locks 0 pid/thread >>>> 4845/140084720432896 flags 0 priority 100 >>>> f5 dd=86 locks held 0 write locks 0 pid/thread >>>> 4845/140084628113152 flags 0 priority 100 >>>> f6 dd=85 locks held 0 write locks 0 pid/thread >>>> 4845/140084703647488 flags 0 priority 100 >>>> f7 dd=84 locks held 0 write locks 0 pid/thread >>>> 4845/140084611327744 flags 0 priority 100 >>>> f8 dd=83 locks held 0 write locks 0 pid/thread >>>> 4845/140084661683968 flags 0 priority 100 >>>> f9 dd=82 locks held 0 write locks 0 pid/thread >>>> 4845/140084837865216 flags 0 priority 100 >>>> fa dd=81 locks held 0 write locks 0 pid/thread >>>> 4845/140084544186112 flags 0 priority 100 >>>> fb dd=80 locks held 0 write locks 0 pid/thread >>>> 4845/140084703647488 flags 0 priority 100 >>>> fc dd=79 locks held 0 write locks 0 pid/thread >>>> 4845/140084544186112 flags 0 priority 100 >>>> fd dd=78 locks held 0 write locks 0 pid/thread >>>> 4845/140084535793408 flags 0 priority 100 >>>> fe dd=77 locks held 0 write locks 0 pid/thread >>>> 4845/140084586149632 flags 0 priority 100 >>>> ff dd=76 locks held 0 write locks 0 pid/thread >>>> 4845/140084569364224 flags 0 priority 100 >>>> 100 dd=75 locks held 0 write locks 0 pid/thread >>>> 4845/140084804294400 flags 0 priority 100 >>>> 101 dd=74 locks held 0 write locks 0 pid/thread >>>> 4845/140084703647488 flags 0 priority 100 >>>> 102 dd=73 locks held 0 write locks 0 pid/thread >>>> 4845/140084703647488 flags 0 priority 100 >>>> 103 dd=72 locks held 0 write locks 0 pid/thread >>>> 4845/140084821079808 flags 0 priority 100 >>>> 104 dd=71 locks held 0 write locks 0 pid/thread >>>> 4845/140084695254784 flags 0 priority 100 >>>> 105 dd=70 locks held 0 write locks 0 pid/thread >>>> 4845/140084544186112 flags 0 priority 100 >>>> 106 dd=69 locks held 0 write locks 0 pid/thread >>>> 4845/140084854650624 flags 0 priority 100 >>>> 107 dd=68 locks held 0 write locks 0 pid/thread >>>> 4845/140084628113152 flags 0 priority 100 >>>> 108 dd=67 locks held 0 write locks 0 pid/thread >>>> 4845/140084569364224 flags 0 priority 100 >>>> 109 dd=66 locks held 0 write locks 0 pid/thread >>>> 4845/140084594542336 flags 0 priority 100 >>>> 10a dd=65 locks held 0 write locks 0 pid/thread >>>> 4845/140084661683968 flags 0 priority 100 >>>> 10b dd=64 locks held 0 write locks 0 pid/thread >>>> 4845/140084653291264 flags 0 priority 100 >>>> 10c dd=63 locks held 0 write locks 0 pid/thread >>>> 4845/140084695254784 flags 0 priority 100 >>>> 10d dd=62 locks held 0 write locks 0 pid/thread >>>> 4845/140084720432896 flags 0 priority 100 >>>> 10e dd=61 locks held 0 write locks 0 pid/thread >>>> 4845/140084619720448 flags 0 priority 100 >>>> 10f dd=60 locks held 0 write locks 0 pid/thread >>>> 4845/140084720432896 flags 0 priority 100 >>>> 110 dd=59 locks held 0 write locks 0 pid/thread >>>> 4845/140084569364224 flags 0 priority 100 >>>> 111 dd=58 locks held 0 write locks 0 pid/thread >>>> 4845/140084695254784 flags 0 priority 100 >>>> 112 dd=57 locks held 0 write locks 0 pid/thread >>>> 4845/140084653291264 flags 0 priority 100 >>>> 113 dd=56 locks held 0 write locks 0 pid/thread >>>> 4845/140084712040192 flags 0 priority 100 >>>> 114 dd=55 locks held 0 write locks 0 pid/thread >>>> 4845/140084686862080 flags 0 priority 100 >>>> 115 dd=54 locks held 0 write locks 0 pid/thread >>>> 4845/140084577756928 flags 0 priority 100 >>>> 115 READ 1 WAIT userRoot/objectclass.db >>>> page 9 >>>> 116 dd=53 locks held 0 write locks 0 pid/thread >>>> 4845/140084837865216 flags 0 priority 100 >>>> 116 READ 1 WAIT userRoot/objectclass.db >>>> page 9 >>>> 117 dd=52 locks held 0 write locks 0 pid/thread >>>> 4845/140084577756928 flags 0 priority 100 >>>> 118 dd=51 locks held 0 write locks 0 pid/thread >>>> 4845/140084569364224 flags 0 priority 100 >>>> 119 dd=50 locks held 0 write locks 0 pid/thread >>>> 4845/140084560971520 flags 0 priority 100 >>>> 11a dd=49 locks held 1 write locks 0 pid/thread >>>> 4845/140084695254784 flags 10 priority 100 >>>> 11a READ 1 HELD ipaca/metaInfo.db >>>> handle 0 >>>> 11b dd=48 locks held 1 write locks 0 pid/thread >>>> 4845/140084695254784 flags 10 priority 100 >>>> 11b READ 1 HELD ipaca/notbefore.db >>>> handle 0 >>>> 11c dd=47 locks held 1 write locks 0 pid/thread >>>> 4845/140084695254784 flags 10 priority 100 >>>> 11c READ 1 HELD ipaca/notafter.db >>>> handle 0 >>>> 11d dd=46 locks held 1 write locks 0 pid/thread >>>> 4845/140084695254784 flags 10 priority 100 >>>> 11d READ 1 HELD ipaca/duration.db >>>> handle 0 >>>> 11e dd=45 locks held 1 write locks 0 pid/thread >>>> 4845/140084695254784 flags 10 priority 100 >>>> 11e READ 1 HELD ipaca/subjectname.db >>>> handle 0 >>>> 11f dd=44 locks held 1 write locks 0 pid/thread >>>> 4845/140084695254784 flags 10 priority 100 >>>> 11f READ 1 HELD ipaca/publicKeyData.db >>>> handle 0 >>>> 120 dd=43 locks held 1 write locks 0 pid/thread >>>> 4845/140084695254784 flags 10 priority 100 >>>> 120 READ 1 HELD ipaca/extension.db >>>> handle 0 >>>> 121 dd=42 locks held 1 write locks 0 pid/thread >>>> 4845/140084695254784 flags 10 priority 100 >>>> 121 READ 1 HELD ipaca/issuedby.db >>>> handle 0 >>>> 122 dd=41 locks held 0 write locks 0 pid/thread >>>> 4845/140084712040192 flags 0 priority 100 >>>> 123 dd=40 locks held 0 write locks 0 pid/thread >>>> 4845/140084812687104 flags 0 priority 100 >>>> 124 dd=39 locks held 0 write locks 0 pid/thread >>>> 4845/140084544186112 flags 0 priority 100 >>>> 125 dd=38 locks held 0 write locks 0 pid/thread >>>> 4845/140084686862080 flags 0 priority 100 >>>> 126 dd=37 locks held 0 write locks 0 pid/thread >>>> 4845/140084552578816 flags 0 priority 100 >>>> 128 dd=36 locks held 0 write locks 0 pid/thread >>>> 4845/140084686862080 flags 0 priority 100 >>>> 12a dd=35 locks held 0 write locks 0 pid/thread >>>> 4845/140084720432896 flags 0 priority 100 >>>> 12b dd=34 locks held 0 write locks 0 pid/thread >>>> 4845/140084552578816 flags 0 priority 100 >>>> 12c dd=33 locks held 0 write locks 0 pid/thread >>>> 4845/140084670076672 flags 0 priority 100 >>>> 12d dd=32 locks held 0 write locks 0 pid/thread >>>> 4845/140084577756928 flags 0 priority 100 >>>> 12e dd=31 locks held 0 write locks 0 pid/thread >>>> 4845/140084712040192 flags 0 priority 100 >>>> 12f dd=30 locks held 0 write locks 0 pid/thread >>>> 4845/140084670076672 flags 0 priority 100 >>>> 130 dd=29 locks held 0 write locks 0 pid/thread >>>> 4845/140084577756928 flags 0 priority 100 >>>> 131 dd=28 locks held 0 write locks 0 pid/thread >>>> 4845/140084560971520 flags 0 priority 100 >>>> 132 dd=27 locks held 0 write locks 0 pid/thread >>>> 4845/140084670076672 flags 0 priority 100 >>>> 133 dd=26 locks held 0 write locks 0 pid/thread >>>> 4845/140084602935040 flags 0 priority 100 >>>> 134 dd=25 locks held 0 write locks 0 pid/thread >>>> 4845/140084552578816 flags 0 priority 100 >>>> 135 dd=24 locks held 0 write locks 0 pid/thread >>>> 4845/140084678469376 flags 0 priority 100 >>>> 136 dd=23 locks held 0 write locks 0 pid/thread >>>> 4845/140084544186112 flags 0 priority 100 >>>> 137 dd=22 locks held 0 write locks 0 pid/thread >>>> 4845/140084577756928 flags 0 priority 100 >>>> 138 dd=21 locks held 0 write locks 0 pid/thread >>>> 4845/140084703647488 flags 0 priority 100 >>>> 139 dd=20 locks held 0 write locks 0 pid/thread >>>> 4845/140084678469376 flags 0 priority 100 >>>> 13b dd=19 locks held 0 write locks 0 pid/thread >>>> 4845/140084812687104 flags 0 priority 100 >>>> 144 dd=18 locks held 0 write locks 0 pid/thread >>>> 4845/140084670076672 flags 0 priority 100 >>>> 156 dd=17 locks held 0 write locks 0 pid/thread >>>> 4845/140084594542336 flags 0 priority 100 >>>> 15a dd=16 locks held 0 write locks 0 pid/thread >>>> 4845/140084577756928 flags 0 priority 100 >>>> 168 dd=15 locks held 0 write locks 0 pid/thread >>>> 4845/140084653291264 flags 0 priority 100 >>>> 16a dd=14 locks held 0 write locks 0 pid/thread >>>> 4845/140084829472512 flags 0 priority 100 >>>> 17d dd=13 locks held 0 write locks 0 pid/thread >>>> 4845/140084829472512 flags 0 priority 100 >>>> 17e dd=12 locks held 0 write locks 0 pid/thread >>>> 4845/140084829472512 flags 0 priority 100 >>>> 17f dd=11 locks held 0 write locks 0 pid/thread >>>> 4845/140084586149632 flags 0 priority 100 >>>> 180 dd=10 locks held 0 write locks 0 pid/thread >>>> 4845/140084594542336 flags 0 priority 100 >>>> 181 dd= 9 locks held 0 write locks 0 pid/thread >>>> 4845/140084577756928 flags 0 priority 100 >>>> 193 dd= 8 locks held 0 write locks 0 pid/thread >>>> 4845/140084812687104 flags 0 priority 100 >>>> 196 dd= 7 locks held 0 write locks 0 pid/thread >>>> 4845/140084695254784 flags 0 priority 100 >>>> 1a1 dd= 6 locks held 0 write locks 0 pid/thread >>>> 4845/140084720432896 flags 0 priority 100 >>>> 1a2 dd= 5 locks held 0 write locks 0 pid/thread >>>> 4845/140084577756928 flags 0 priority 100 >>>> 1a3 dd= 4 locks held 0 write locks 0 pid/thread >>>> 4845/140084812687104 flags 0 priority 100 >>>> 1a4 dd= 3 locks held 0 write locks 0 pid/thread >>>> 4845/140084812687104 flags 0 priority 100 >>>> 1a5 dd= 2 locks held 0 write locks 0 pid/thread >>>> 4845/140084577756928 flags 0 priority 100 >>>> 1a6 dd= 1 locks held 0 write locks 0 pid/thread >>>> 4845/140084594542336 flags 0 priority 100 >>>> 800b3a33 dd= 0 locks held 26 write locks 22 >>>> pid/thread 4845/140084661683968 flags 0 priority 100 >>>> 800b3a33 WRITE 1 HELD userRoot/objectclass.db >>>> page 9 >>>> 800b3a33 WRITE 1 HELD userRoot/objectclass.db >>>> page 8 >>>> 800b3a33 WRITE 1 HELD userRoot/objectclass.db >>>> page 18 >>>> 800b3a33 WRITE 1 HELD userRoot/objectclass.db >>>> page 7 >>>> 800b3a33 WRITE 1 HELD userRoot/objectclass.db >>>> page 14 >>>> 800b3a33 WRITE 1 HELD userRoot/objectclass.db >>>> page 11 >>>> 800b3a33 WRITE 2 HELD userRoot/objectclass.db >>>> page 3 >>>> 800b3a33 WRITE 1 HELD userRoot/cn.db >>>> page 150 >>>> 800b3a33 WRITE 1 HELD userRoot/cn.db >>>> page 89 >>>> 800b3a33 WRITE 1 HELD userRoot/cn.db >>>> page 53 >>>> 800b3a33 WRITE 1 HELD userRoot/cn.db >>>> page 131 >>>> 800b3a33 WRITE 1 HELD userRoot/cn.db >>>> page 117 >>>> 800b3a33 WRITE 1 HELD userRoot/cn.db >>>> page 11 >>>> 800b3a33 WRITE 2 HELD userRoot/cn.db >>>> page 141 >>>> 800b3a33 WRITE 1 HELD userRoot/cn.db >>>> page 41 >>>> 800b3a33 WRITE 1 HELD userRoot/cn.db >>>> page 40 >>>> 800b3a33 WRITE 1 HELD userRoot/cn.db >>>> page 143 >>>> 800b3a33 WRITE 1 HELD userRoot/cn.db >>>> page 30 >>>> 800b3a33 WRITE 2 HELD userRoot/id2entry.db >>>> page 8184 >>>> 800b3a33 WRITE 1 HELD userRoot/id2entry.db >>>> page 8183 >>>> 800b3a33 WRITE 1 HELD userRoot/id2entry.db >>>> page 7779 >>>> 800b3a33 WRITE 1 HELD userRoot/id2entry.db >>>> page 0 >>>> 800b3a33 READ 3 HELD userRoot/entryrdn.db >>>> page 3 >>>> 800b3a33 READ 1 HELD userRoot/entryrdn.db >>>> page 36 >>>> 800b3a33 READ 1 HELD userRoot/entryrdn.db >>>> page 4 >>>> 800b3a33 READ 1 HELD userRoot/nsuniqueid.db >>>> page 86 >>>> =-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-= >>>> Locks grouped by object: >>>> Locker Mode Count Status ----------------- >>>> Object --------------- >>>> a4 READ 1 HELD userRoot/ipasudorunas.db >>>> handle 0 >>>> >>>> 9e READ 1 HELD >>>> userRoot/ipaMemberCertProfile.db handle 0 >>>> >>>> 10 READ 1 HELD >>>> ipaca/vlv#allrevokedcertspkitomcatindex.db handle 0 >>>> >>>> 800b3a33 WRITE 2 HELD userRoot/id2entry.db >>>> page 8184 >>>> >>>> 800b3a33 WRITE 1 HELD userRoot/id2entry.db >>>> page 8183 >>>> >>>> 800b3a33 WRITE 1 HELD userRoot/id2entry.db >>>> page 7779 >>>> >>>> a1 READ 1 HELD userRoot/memberdenycmd.db >>>> handle 0 >>>> >>>> a READ 1 HELD >>>> ipaca/vlv#allcertspkitomcatindex.db handle 0 >>>> >>>> a7 READ 1 HELD userRoot/sourcehost.db >>>> handle 0 >>>> >>>> 57 READ 1 HELD changelog/ancestorid.db >>>> handle 0 >>>> >>>> e5 READ 1 HELD ipaca/revInfo.db >>>> handle 0 >>>> >>>> e6 READ 1 HELD ipaca/revokedby.db >>>> handle 0 >>>> >>>> 11 READ 1 HELD >>>> ipaca/vlv#allrevokedcertsnotafterpkitomcatindex.db >>>> handle 0 >>>> >>>> 4e READ 1 HELD userRoot/parentid.db >>>> handle 0 >>>> >>>> 4 READ 1 HELD ipaca/id2entry.db >>>> handle 0 >>>> >>>> 58 READ 1 HELD >>>> changelog/numsubordinates.db handle 0 >>>> >>>> 67 READ 1 HELD ipaca/seeAlso.db >>>> handle 0 >>>> >>>> 2b READ 1 HELD >>>> ipaca/vlv#carevocationpkitomcatindex.db handle 0 >>>> >>>> ec READ 1 HELD ipaca/requeststate.db >>>> handle 0 >>>> >>>> 11b READ 1 HELD ipaca/notbefore.db >>>> handle 0 >>>> >>>> 7e READ 1 HELD >>>> userRoot/numsubordinates.db handle 0 >>>> >>>> a2 READ 1 HELD >>>> userRoot/memberallowcmd.db handle 0 >>>> >>>> 9c READ 1 HELD userRoot/owner.db >>>> handle 0 >>>> >>>> 1e READ 1 HELD >>>> ipaca/vlv#cacompleteenrollmentpkitomcatindex.db >>>> handle 0 >>>> >>>> 11c READ 1 HELD ipaca/notafter.db >>>> handle 0 >>>> >>>> 40 READ 1 HELD changelog/objectclass.db >>>> handle 0 >>>> >>>> 2e READ 1 HELD changelog/entryusn.db >>>> handle 0 >>>> >>>> 16 READ 1 HELD >>>> ipaca/vlv#allvalidcertsnotafterpkitomcatindex.db >>>> handle 0 >>>> >>>> 15 READ 1 HELD >>>> ipaca/vlv#allvalidcertspkitomcatindex.db handle 0 >>>> >>>> a5 READ 1 HELD userRoot/ipaMemberCa.db >>>> handle 0 >>>> >>>> 18 READ 1 HELD >>>> ipaca/vlv#caallpkitomcatindex.db handle 0 >>>> >>>> a0 READ 1 HELD userRoot/secretary.db >>>> handle 0 >>>> >>>> 81 READ 1 HELD userRoot/memberUser.db >>>> handle 0 >>>> >>>> 9f READ 1 HELD >>>> userRoot/ipaassignedidview.db handle 0 >>>> >>>> a3 READ 1 HELD userRoot/manager.db >>>> handle 0 >>>> >>>> d8 READ 1 HELD ipaca/description.db >>>> handle 0 >>>> >>>> a8 READ 1 HELD userRoot/memberservice.db >>>> handle 0 >>>> >>>> 44 READ 1 HELD ipaca/objectclass.db >>>> handle 0 >>>> >>>> ed READ 1 HELD ipaca/dateOfCreate.db >>>> handle 0 >>>> >>>> 800b3a33 WRITE 1 HELD userRoot/id2entry.db >>>> page 0 >>>> >>>> 2 READ 1 HELD userRoot/id2entry.db >>>> handle 0 >>>> >>>> 5b READ 1 HELD >>>> userRoot/krbPrincipalName.db handle 0 >>>> >>>> ee READ 1 HELD ipaca/requesttype.db >>>> handle 0 >>>> >>>> c8 READ 1 HELD ipaca/certstatus.db >>>> handle 0 >>>> >>>> 56 READ 1 HELD changelog/parentid.db >>>> handle 0 >>>> >>>> 83 READ 1 HELD userRoot/memberHost.db >>>> handle 0 >>>> >>>> 5e READ 1 HELD >>>> userRoot/ipakrbprincipalalias.db handle 0 >>>> >>>> 121 READ 1 HELD ipaca/issuedby.db >>>> handle 0 >>>> >>>> 6 READ 1 HELD ipaca/entryrdn.db >>>> handle 0 >>>> >>>> e7 READ 1 HELD ipaca/revokedOn.db >>>> handle 0 >>>> >>>> 4c READ 1 HELD ipaca/aci.db handle 0 >>>> >>>> 11d READ 1 HELD ipaca/duration.db >>>> handle 0 >>>> >>>> 14 READ 1 HELD >>>> ipaca/vlv#allrevokedorrevokedexpiredcertspkitomcatindex.db >>>> handle 0 >>>> >>>> 21 READ 1 HELD >>>> ipaca/vlv#caenrollmentpkitomcatindex.db handle 0 >>>> >>>> 55 READ 1 HELD >>>> changelog/targetuniqueid.db handle 0 >>>> >>>> 800b3a33 READ 1 HELD userRoot/entryrdn.db >>>> page 36 >>>> >>>> 2c READ 1 HELD changelog/id2entry.db >>>> handle 0 >>>> >>>> 800b3a33 READ 1 HELD userRoot/entryrdn.db >>>> page 4 >>>> >>>> 34 READ 1 HELD userRoot/entryrdn.db >>>> handle 0 >>>> >>>> 800b3a33 READ 3 HELD userRoot/entryrdn.db >>>> page 3 >>>> >>>> eb READ 1 HELD ipaca/requestid.db >>>> handle 0 >>>> >>>> 49 READ 1 HELD userRoot/aci.db >>>> handle 0 >>>> >>>> 76 READ 1 HELD userRoot/ipauniqueid.db >>>> handle 0 >>>> >>>> 63 READ 1 HELD changelog/seeAlso.db >>>> handle 0 >>>> >>>> 7c READ 1 HELD userRoot/fqdn.db >>>> handle 0 >>>> >>>> db READ 1 HELD ipaca/cn.db handle 0 >>>> >>>> ef READ 1 HELD ipaca/numsubordinates.db >>>> handle 0 >>>> >>>> 32 READ 1 HELD ipaca/entryusn.db >>>> handle 0 >>>> >>>> 1d READ 1 HELD >>>> ipaca/vlv#cacompletepkitomcatindex.db handle 0 >>>> >>>> dd READ 1 HELD ipaca/ancestorid.db >>>> handle 0 >>>> >>>> a9 READ 1 HELD >>>> userRoot/ipatokenradiusconfiglink.db handle 0 >>>> >>>> f0 READ 1 HELD >>>> userRoot/userCertificate.db handle 0 >>>> >>>> 11a READ 1 HELD ipaca/metaInfo.db >>>> handle 0 >>>> >>>> 7d READ 1 HELD userRoot/managedby.db >>>> handle 0 >>>> >>>> ea READ 1 HELD ipaca/nsuniqueid.db >>>> handle 0 >>>> >>>> c5 READ 1 HELD userRoot/uidnumber.db >>>> handle 0 >>>> >>>> d1 READ 1 HELD userRoot/nscpEntryDN.db >>>> handle 0 >>>> >>>> 120 READ 1 HELD ipaca/extension.db >>>> handle 0 >>>> >>>> 800b3a33 WRITE 1 HELD userRoot/cn.db >>>> page 150 >>>> >>>> 69 READ 1 HELD ipaca/parentid.db >>>> handle 0 >>>> >>>> 800b3a33 WRITE 1 HELD userRoot/cn.db >>>> page 131 >>>> >>>> 800b3a33 WRITE 1 HELD userRoot/cn.db >>>> page 143 >>>> >>>> 800b3a33 WRITE 2 HELD userRoot/cn.db >>>> page 141 >>>> >>>> 20 READ 1 HELD >>>> ipaca/vlv#cacompleterevocationpkitomcatindex.db >>>> handle 0 >>>> >>>> 8e READ 1 HELD userRoot/macAddress.db >>>> handle 0 >>>> >>>> e READ 1 HELD >>>> ipaca/vlv#allnonrevokedcertspkitomcatindex.db >>>> handle 0 >>>> >>>> 17 READ 1 HELD >>>> ipaca/vlv#allvalidorrevokedcertspkitomcatindex.db >>>> handle 0 >>>> >>>> 800b3a33 WRITE 1 HELD userRoot/cn.db >>>> page 30 >>>> >>>> 800b3a33 WRITE 1 HELD userRoot/objectclass.db >>>> page 14 >>>> >>>> 800b3a33 WRITE 1 HELD userRoot/objectclass.db >>>> page 9 >>>> 116 READ 1 WAIT userRoot/objectclass.db >>>> page 9 >>>> 115 READ 1 WAIT userRoot/objectclass.db >>>> page 9 >>>> 92 READ 1 WAIT userRoot/objectclass.db >>>> page 9 >>>> >>>> 800b3a33 WRITE 1 HELD userRoot/objectclass.db >>>> page 8 >>>> >>>> 7b READ 1 HELD userRoot/cn.db >>>> handle 0 >>>> >>>> 800b3a33 WRITE 1 HELD userRoot/objectclass.db >>>> page 11 >>>> >>>> 800b3a33 WRITE 1 HELD userRoot/objectclass.db >>>> page 7 >>>> >>>> 800b3a33 WRITE 1 HELD userRoot/cn.db >>>> page 11 >>>> >>>> 38 READ 1 HELD userRoot/objectclass.db >>>> handle 0 >>>> >>>> 800b3a33 WRITE 2 HELD userRoot/objectclass.db >>>> page 3 >>>> >>>> 800b3a33 WRITE 1 HELD userRoot/cn.db >>>> page 53 >>>> >>>> 800b3a33 WRITE 1 HELD userRoot/objectclass.db >>>> page 18 >>>> 6e READ 1 WAIT userRoot/objectclass.db >>>> page 18 >>>> 94 READ 1 WAIT userRoot/objectclass.db >>>> page 18 >>>> >>>> 800b3a33 WRITE 1 HELD userRoot/cn.db >>>> page 41 >>>> >>>> 800b3a33 WRITE 1 HELD userRoot/cn.db >>>> page 40 >>>> >>>> 800b3a33 WRITE 1 HELD userRoot/cn.db >>>> page 89 >>>> >>>> 800b3a33 WRITE 1 HELD userRoot/cn.db >>>> page 117 >>>> >>>> 30 READ 1 HELD userRoot/entryusn.db >>>> handle 0 >>>> >>>> 3c READ 1 HELD changelog/entryrdn.db >>>> handle 0 >>>> >>>> 97 READ 1 HELD userRoot/uid.db >>>> handle 0 >>>> >>>> 47 READ 1 HELD changelog/aci.db >>>> handle 0 >>>> >>>> e3 READ 1 HELD ipaca/serialno.db >>>> handle 0 >>>> >>>> 7a READ 1 HELD userRoot/nsuniqueid.db >>>> handle 0 >>>> >>>> 800b3a33 READ 1 HELD userRoot/nsuniqueid.db >>>> page 86 >>>> >>>> 51 READ 1 HELD changelog/nsuniqueid.db >>>> handle 0 >>>> >>>> 11e READ 1 HELD ipaca/subjectname.db >>>> handle 0 >>>> >>>> 7f READ 1 HELD userRoot/member.db >>>> handle 0 >>>> >>>> 11f READ 1 HELD ipaca/publicKeyData.db >>>> handle 0 >>>> >>>> 9b READ 1 HELD userRoot/uniquemember.db >>>> handle 0 >>>> >>>> 52 READ 1 HELD changelog/changenumber.db >>>> handle 0 >>>> >>>> ce READ 1 HELD userRoot/gidnumber.db >>>> handle 0 >>>> >>>> 65 READ 1 HELD userRoot/seeAlso.db >>>> handle 0 >>>> >>>> 9d READ 1 HELD >>>> userRoot/ipaallowedtarget.db handle 0 >>>> >>>> a6 READ 1 HELD >>>> userRoot/ipasudorunasgroup.db handle 0 >>>> >>>> 3a READ 1 HELD userRoot/ancestorid.db >>>> handle 0 >>>> >>>> >>>> >>>> >>>> pstack output >>>> >>>> root at ipa-master ]# pstack 4845 >>>> Thread 44 (Thread 0x7f681dd3a700 (LWP 4847)): >>>> #0 0x00007f683cd719b3 in select () from /lib64/libc.so.6 >>>> #1 0x00007f683f4fc0e9 in DS_Sleep () from >>>> /usr/lib64/dirsrv/libslapd.so.0 >>>> #2 0x00007f6832b4a907 in deadlock_threadmain () from >>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>> #3 0x00007f683d6ab7bb in _pt_root () from >>>> /lib64/libnspr4.so >>>> #4 0x00007f683d04cdc5 in start_thread () from >>>> /lib64/libpthread.so.0 >>>> #5 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 >>>> Thread 43 (Thread 0x7f681d539700 (LWP 4848)): >>>> #0 0x00007f683cd719b3 in select () from /lib64/libc.so.6 >>>> #1 0x00007f68376d8bad in __os_yield () from >>>> /lib64/libdb-5.3.so >>>> #2 0x00007f68376d42b3 in __memp_sync_int () from >>>> /lib64/libdb-5.3.so >>>> #3 0x00007f68376e4752 in __txn_checkpoint () from >>>> /lib64/libdb-5.3.so >>>> #4 0x00007f68376e4b74 in __txn_checkpoint_pp () from >>>> /lib64/libdb-5.3.so >>>> #5 0x00007f6832b4ea87 in checkpoint_threadmain () from >>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>> #6 0x00007f683d6ab7bb in _pt_root () from >>>> /lib64/libnspr4.so >>>> #7 0x00007f683d04cdc5 in start_thread () from >>>> /lib64/libpthread.so.0 >>>> #8 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 >>>> Thread 42 (Thread 0x7f681cd38700 (LWP 4849)): >>>> #0 0x00007f683cd719b3 in select () from /lib64/libc.so.6 >>>> #1 0x00007f683f4fc0e9 in DS_Sleep () from >>>> /usr/lib64/dirsrv/libslapd.so.0 >>>> #2 0x00007f6832b4ab7f in trickle_threadmain () from >>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>> #3 0x00007f683d6ab7bb in _pt_root () from >>>> /lib64/libnspr4.so >>>> #4 0x00007f683d04cdc5 in start_thread () from >>>> /lib64/libpthread.so.0 >>>> #5 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 >>>> Thread 41 (Thread 0x7f681c537700 (LWP 4850)): >>>> #0 0x00007f683cd719b3 in select () from /lib64/libc.so.6 >>>> #1 0x00007f683f4fc0e9 in DS_Sleep () from >>>> /usr/lib64/dirsrv/libslapd.so.0 >>>> #2 0x00007f6832b45707 in perf_threadmain () from >>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>> #3 0x00007f683d6ab7bb in _pt_root () from >>>> /lib64/libnspr4.so >>>> #4 0x00007f683d04cdc5 in start_thread () from >>>> /lib64/libpthread.so.0 >>>> #5 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 >>>> Thread 40 (Thread 0x7f681bb2d700 (LWP 4851)): >>>> #0 0x00007f683d0506d5 in >>>> pthread_cond_wait@@GLIBC_2.3.2 >>>> () from >>>> /lib64/libpthread.so.0 >>>> #1 0x00007f683d6a6050 in PR_WaitCondVar () from >>>> /lib64/libnspr4.so >>>> #2 0x00007f683f4eb198 in slapi_wait_condvar () from >>>> /usr/lib64/dirsrv/libslapd.so.0 >>>> #3 0x00007f683587d62e in cos_cache_wait_on_change () >>>> from /usr/lib64/dirsrv/plugins/libcos-plugin.so >>>> #4 0x00007f683d6ab7bb in _pt_root () from >>>> /lib64/libnspr4.so >>>> #5 0x00007f683d04cdc5 in start_thread () from >>>> /lib64/libpthread.so.0 >>>> #6 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 >>>> Thread 39 (Thread 0x7f681b32c700 (LWP 4852)): >>>> #0 0x00007f683d0506d5 in >>>> pthread_cond_wait@@GLIBC_2.3.2 >>>> () from >>>> /lib64/libpthread.so.0 >>>> #1 0x00007f683d6a6050 in PR_WaitCondVar () from >>>> /lib64/libnspr4.so >>>> #2 0x00007f683f4eb198 in slapi_wait_condvar () from >>>> /usr/lib64/dirsrv/libslapd.so.0 >>>> #3 0x00007f68311e8eed in roles_cache_wait_on_change () >>>> from /usr/lib64/dirsrv/plugins/libroles-plugin.so >>>> #4 0x00007f683d6ab7bb in _pt_root () from >>>> /lib64/libnspr4.so >>>> #5 0x00007f683d04cdc5 in start_thread () from >>>> /lib64/libpthread.so.0 >>>> #6 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 >>>> Thread 38 (Thread 0x7f681ab2b700 (LWP 4853)): >>>> #0 0x00007f683d0506d5 in >>>> pthread_cond_wait@@GLIBC_2.3.2 >>>> () from >>>> /lib64/libpthread.so.0 >>>> #1 0x00007f683d6a6050 in PR_WaitCondVar () from >>>> /lib64/libnspr4.so >>>> #2 0x00007f683f4eb198 in slapi_wait_condvar () from >>>> /usr/lib64/dirsrv/libslapd.so.0 >>>> #3 0x00007f68311e8eed in roles_cache_wait_on_change () >>>> from /usr/lib64/dirsrv/plugins/libroles-plugin.so >>>> #4 0x00007f683d6ab7bb in _pt_root () from >>>> /lib64/libnspr4.so >>>> #5 0x00007f683d04cdc5 in start_thread () from >>>> /lib64/libpthread.so.0 >>>> #6 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 >>>> Thread 37 (Thread 0x7f681a32a700 (LWP 4854)): >>>> #0 0x00007f683d0506d5 in >>>> pthread_cond_wait@@GLIBC_2.3.2 >>>> () from >>>> /lib64/libpthread.so.0 >>>> #1 0x00007f683d6a6050 in PR_WaitCondVar () from >>>> /lib64/libnspr4.so >>>> #2 0x00007f683f4eb198 in slapi_wait_condvar () from >>>> /usr/lib64/dirsrv/libslapd.so.0 >>>> #3 0x00007f68311e8eed in roles_cache_wait_on_change () >>>> from /usr/lib64/dirsrv/plugins/libroles-plugin.so >>>> #4 0x00007f683d6ab7bb in _pt_root () from >>>> /lib64/libnspr4.so >>>> #5 0x00007f683d04cdc5 in start_thread () from >>>> /lib64/libpthread.so.0 >>>> #6 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 >>>> Thread 36 (Thread 0x7f6819b29700 (LWP 4855)): >>>> #0 0x00007f683d050a82 in >>>> pthread_cond_timedwait@@GLIBC_2.3.2 >>>> () from >>>> /lib64/libpthread.so.0 >>>> #1 0x00007f683d6a5b07 in pt_TimedWait () from >>>> /lib64/libnspr4.so >>>> #2 0x00007f683d6a5fce in PR_WaitCondVar () from >>>> /lib64/libnspr4.so >>>> #3 0x00007f683f9a4d03 in housecleaning () >>>> #4 0x00007f683d6ab7bb in _pt_root () from >>>> /lib64/libnspr4.so >>>> #5 0x00007f683d04cdc5 in start_thread () from >>>> /lib64/libpthread.so.0 >>>> #6 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 >>>> Thread 35 (Thread 0x7f6819328700 (LWP 4856)): >>>> #0 0x00007f683d050a82 in >>>> pthread_cond_timedwait@@GLIBC_2.3.2 >>>> () from >>>> /lib64/libpthread.so.0 >>>> #1 0x00007f683d6a5b07 in pt_TimedWait () from >>>> /lib64/libnspr4.so >>>> #2 0x00007f683d6a5fce in PR_WaitCondVar () from >>>> /lib64/libnspr4.so >>>> #3 0x00007f683f4911e8 in eq_loop () from >>>> /usr/lib64/dirsrv/libslapd.so.0 >>>> #4 0x00007f683d6ab7bb in _pt_root () from >>>> /lib64/libnspr4.so >>>> #5 0x00007f683d04cdc5 in start_thread () from >>>> /lib64/libpthread.so.0 >>>> #6 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 >>>> Thread 34 (Thread 0x7f680bfff700 (LWP 4859)): >>>> #0 0x00007f683d0506d5 in >>>> pthread_cond_wait@@GLIBC_2.3.2 >>>> () from >>>> /lib64/libpthread.so.0 >>>> #1 0x00007f683d6a6050 in PR_WaitCondVar () from >>>> /lib64/libnspr4.so >>>> #2 0x00007f683f999fee in connection_wait_for_new_work () >>>> #3 0x00007f683f99b719 in connection_threadmain () >>>> #4 0x00007f683d6ab7bb in _pt_root () from >>>> /lib64/libnspr4.so >>>> #5 0x00007f683d04cdc5 in start_thread () from >>>> /lib64/libpthread.so.0 >>>> #6 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 >>>> Thread 33 (Thread 0x7f680b7fe700 (LWP 4860)): >>>> #0 0x00007f683d0506d5 in >>>> pthread_cond_wait@@GLIBC_2.3.2 >>>> () from >>>> /lib64/libpthread.so.0 >>>> #1 0x00007f683d6a6050 in PR_WaitCondVar () from >>>> /lib64/libnspr4.so >>>> #2 0x00007f683f999fee in connection_wait_for_new_work () >>>> #3 0x00007f683f99b719 in connection_threadmain () >>>> #4 0x00007f683d6ab7bb in _pt_root () from >>>> /lib64/libnspr4.so >>>> #5 0x00007f683d04cdc5 in start_thread () from >>>> /lib64/libpthread.so.0 >>>> #6 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 >>>> Thread 32 (Thread 0x7f680affd700 (LWP 4861)): >>>> #0 0x00007f683d0506d5 in >>>> pthread_cond_wait@@GLIBC_2.3.2 >>>> () from >>>> /lib64/libpthread.so.0 >>>> #1 0x00007f68375ac2f3 in __db_hybrid_mutex_suspend () >>>> from /lib64/libdb-5.3.so >>>> #2 0x00007f68375ab640 in __db_tas_mutex_lock () from >>>> /lib64/libdb-5.3.so >>>> #3 0x00007f6837655d3a in __lock_get_internal () from >>>> /lib64/libdb-5.3.so >>>> #4 0x00007f6837656820 in __lock_get () from >>>> /lib64/libdb-5.3.so >>>> #5 0x00007f6837682142 in __db_lget () from >>>> /lib64/libdb-5.3.so >>>> #6 0x00007f68375c9605 in __bam_search () from >>>> /lib64/libdb-5.3.so >>>> #7 0x00007f68375b4256 in __bamc_search () from >>>> /lib64/libdb-5.3.so >>>> #8 0x00007f68375b5d0f in __bamc_get () from >>>> /lib64/libdb-5.3.so >>>> #9 0x00007f683766eca6 in __dbc_iget () from >>>> /lib64/libdb-5.3.so >>>> #10 0x00007f683767db02 in __dbc_get_pp () from >>>> /lib64/libdb-5.3.so >>>> #11 0x00007f6832b59680 in idl_new_fetch () from >>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>> #12 0x00007f6832b67b26 in index_read_ext_allids () from >>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>> #13 0x00007f6832b52284 in keys2idl () from >>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>> #14 0x00007f6832b529e3 in ava_candidates.isra.0 () from >>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>> #15 0x00007f6832b52fd2 in filter_candidates_ext () from >>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>> #16 0x00007f6832b54096 in list_candidates () from >>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>> #17 0x00007f6832b52f40 in filter_candidates_ext () from >>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>> #18 0x00007f6832b54096 in list_candidates () from >>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>> #19 0x00007f6832b52f40 in filter_candidates_ext () from >>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>> #20 0x00007f6832b54096 in list_candidates () from >>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>> #21 0x00007f6832b52f40 in filter_candidates_ext () from >>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>> #22 0x00007f6832b8e80c in subtree_candidates () from >>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>> #23 0x00007f6832b8fe7f in ldbm_back_search () from >>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>> #24 0x00007f683f4bd1d6 in op_shared_search () from >>>> /usr/lib64/dirsrv/libslapd.so.0 >>>> #25 0x00007f683f9ae33e in do_search () >>>> #26 0x00007f683f99c3c3 in connection_threadmain () >>>> #27 0x00007f683d6ab7bb in _pt_root () from >>>> /lib64/libnspr4.so >>>> #28 0x00007f683d04cdc5 in start_thread () from >>>> /lib64/libpthread.so.0 >>>> #29 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 >>>> Thread 31 (Thread 0x7f680a7fc700 (LWP 4862)): >>>> #0 0x00007f683d0506d5 in >>>> pthread_cond_wait@@GLIBC_2.3.2 >>>> () from >>>> /lib64/libpthread.so.0 >>>> #1 0x00007f683d6a62c3 in PR_EnterMonitor () from >>>> /lib64/libnspr4.so >>>> #2 0x00007f6832b4d3e6 in dblayer_txn_begin () from >>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>> #3 0x00007f6832b89007 in ldbm_back_modify () from >>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>> #4 0x00007f683f4b6541 in op_shared_modify () from >>>> /usr/lib64/dirsrv/libslapd.so.0 >>>> #5 0x00007f683f4b7004 in modify_internal_pb () from >>>> /usr/lib64/dirsrv/libslapd.so.0 >>>> #6 0x00007f683483bbbb in ipalockout_postop () from >>>> /usr/lib64/dirsrv/plugins/libipa_lockout.so >>>> #7 0x00007f683f4c8848 in plugin_call_func () from >>>> /usr/lib64/dirsrv/libslapd.so.0 >>>> #8 0x00007f683f4c8b13 in plugin_call_plugins () from >>>> /usr/lib64/dirsrv/libslapd.so.0 >>>> #9 0x00007f683f9954d8 in do_bind () >>>> #10 0x00007f683f99c3fd in connection_threadmain () >>>> #11 0x00007f683d6ab7bb in _pt_root () from >>>> /lib64/libnspr4.so >>>> #12 0x00007f683d04cdc5 in start_thread () from >>>> /lib64/libpthread.so.0 >>>> #13 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 >>>> Thread 30 (Thread 0x7f6809ffb700 (LWP 4863)): >>>> #0 0x00007f683d0506d5 in >>>> pthread_cond_wait@@GLIBC_2.3.2 >>>> () from >>>> /lib64/libpthread.so.0 >>>> #1 0x00007f68375ac2f3 in __db_hybrid_mutex_suspend () >>>> from /lib64/libdb-5.3.so >>>> #2 0x00007f68375ab640 in __db_tas_mutex_lock () from >>>> /lib64/libdb-5.3.so >>>> #3 0x00007f6837655d3a in __lock_get_internal () from >>>> /lib64/libdb-5.3.so >>>> #4 0x00007f6837656820 in __lock_get () from >>>> /lib64/libdb-5.3.so >>>> #5 0x00007f6837682142 in __db_lget () from >>>> /lib64/libdb-5.3.so >>>> #6 0x00007f68375c9605 in __bam_search () from >>>> /lib64/libdb-5.3.so >>>> #7 0x00007f68375b4256 in __bamc_search () from >>>> /lib64/libdb-5.3.so >>>> #8 0x00007f68375b5d0f in __bamc_get () from >>>> /lib64/libdb-5.3.so >>>> #9 0x00007f683766eca6 in __dbc_iget () from >>>> /lib64/libdb-5.3.so >>>> #10 0x00007f683767db02 in __dbc_get_pp () from >>>> /lib64/libdb-5.3.so >>>> #11 0x00007f6832b59680 in idl_new_fetch () from >>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>> #12 0x00007f6832b67b26 in index_read_ext_allids () from >>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>> #13 0x00007f6832b52284 in keys2idl () from >>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>> #14 0x00007f6832b529e3 in ava_candidates.isra.0 () from >>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>> #15 0x00007f6832b52fd2 in filter_candidates_ext () from >>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>> #16 0x00007f6832b54096 in list_candidates () from >>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>> #17 0x00007f6832b52f40 in filter_candidates_ext () from >>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>> #18 0x00007f6832b8e80c in subtree_candidates () from >>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>> #19 0x00007f6832b8fe7f in ldbm_back_search () from >>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>> #20 0x00007f683f4bd1d6 in op_shared_search () from >>>> /usr/lib64/dirsrv/libslapd.so.0 >>>> #21 0x00007f683f4cd94e in search_internal_callback_pb >>>> () from /usr/lib64/dirsrv/libslapd.so.0 >>>> #22 0x00007f683f4cdbe8 in search_internal_pb () from >>>> /usr/lib64/dirsrv/libslapd.so.0 >>>> #23 0x00007f683f9ac49b in ids_sasl_canon_user () >>>> #24 0x00007f683d46c93b in _sasl_canon_user () from >>>> /lib64/libsasl2.so.3 >>>> #25 0x00007f683d46cc4c in _sasl_canon_user_lookup () >>>> from /lib64/libsasl2.so.3 >>>> #26 0x00007f6837374fb9 in gssapi_server_mech_step () >>>> from /usr/lib64/sasl2/libgssapiv2.so >>>> #27 0x00007f683d478b9b in sasl_server_step () from >>>> /lib64/libsasl2.so.3 >>>> #28 0x00007f683f9ad83b in ids_sasl_check_bind () >>>> #29 0x00007f683f99561a in do_bind () >>>> #30 0x00007f683f99c3fd in connection_threadmain () >>>> #31 0x00007f683d6ab7bb in _pt_root () from >>>> /lib64/libnspr4.so >>>> #32 0x00007f683d04cdc5 in start_thread () from >>>> /lib64/libpthread.so.0 >>>> #33 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 >>>> Thread 29 (Thread 0x7f68097fa700 (LWP 4864)): >>>> #0 0x00007f683d0506d5 in >>>> pthread_cond_wait@@GLIBC_2.3.2 >>>> () from >>>> /lib64/libpthread.so.0 >>>> #1 0x00007f683d6a6050 in PR_WaitCondVar () from >>>> /lib64/libnspr4.so >>>> #2 0x00007f683f999fee in connection_wait_for_new_work () >>>> #3 0x00007f683f99b719 in connection_threadmain () >>>> #4 0x00007f683d6ab7bb in _pt_root () from >>>> /lib64/libnspr4.so >>>> #5 0x00007f683d04cdc5 in start_thread () from >>>> /lib64/libpthread.so.0 >>>> #6 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 >>>> Thread 28 (Thread 0x7f6808ff9700 (LWP 4865)): >>>> #0 0x00007f683d0506d5 in >>>> pthread_cond_wait@@GLIBC_2.3.2 >>>> () from >>>> /lib64/libpthread.so.0 >>>> #1 0x00007f683d6a6050 in PR_WaitCondVar () from >>>> /lib64/libnspr4.so >>>> #2 0x00007f683f999fee in connection_wait_for_new_work () >>>> #3 0x00007f683f99b719 in connection_threadmain () >>>> #4 0x00007f683d6ab7bb in _pt_root () from >>>> /lib64/libnspr4.so >>>> #5 0x00007f683d04cdc5 in start_thread () from >>>> /lib64/libpthread.so.0 >>>> #6 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 >>>> Thread 27 (Thread 0x7f6803fff700 (LWP 4866)): >>>> #0 0x00007f683d0506d5 in >>>> pthread_cond_wait@@GLIBC_2.3.2 >>>> () from >>>> /lib64/libpthread.so.0 >>>> #1 0x00007f683d6a6050 in PR_WaitCondVar () from >>>> /lib64/libnspr4.so >>>> #2 0x00007f683f999fee in connection_wait_for_new_work () >>>> #3 0x00007f683f99b719 in connection_threadmain () >>>> #4 0x00007f683d6ab7bb in _pt_root () from >>>> /lib64/libnspr4.so >>>> #5 0x00007f683d04cdc5 in start_thread () from >>>> /lib64/libpthread.so.0 >>>> #6 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 >>>> Thread 26 (Thread 0x7f68037fe700 (LWP 4867)): >>>> #0 0x00007f683d0506d5 in >>>> pthread_cond_wait@@GLIBC_2.3.2 >>>> () from >>>> /lib64/libpthread.so.0 >>>> #1 0x00007f683d6a6050 in PR_WaitCondVar () from >>>> /lib64/libnspr4.so >>>> #2 0x00007f683f999fee in connection_wait_for_new_work () >>>> #3 0x00007f683f99b719 in connection_threadmain () >>>> #4 0x00007f683d6ab7bb in _pt_root () from >>>> /lib64/libnspr4.so >>>> #5 0x00007f683d04cdc5 in start_thread () from >>>> /lib64/libpthread.so.0 >>>> #6 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 >>>> Thread 25 (Thread 0x7f6802ffd700 (LWP 4868)): >>>> #0 0x00007f683d0506d5 in >>>> pthread_cond_wait@@GLIBC_2.3.2 >>>> () from >>>> /lib64/libpthread.so.0 >>>> #1 0x00007f683d6a6050 in PR_WaitCondVar () from >>>> /lib64/libnspr4.so >>>> #2 0x00007f683f999fee in connection_wait_for_new_work () >>>> #3 0x00007f683f99b719 in connection_threadmain () >>>> #4 0x00007f683d6ab7bb in _pt_root () from >>>> /lib64/libnspr4.so >>>> #5 0x00007f683d04cdc5 in start_thread () from >>>> /lib64/libpthread.so.0 >>>> #6 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 >>>> Thread 24 (Thread 0x7f68027fc700 (LWP 4869)): >>>> #0 0x00007f683d0506d5 in >>>> pthread_cond_wait@@GLIBC_2.3.2 >>>> () from >>>> /lib64/libpthread.so.0 >>>> #1 0x00007f68375ac2f3 in __db_hybrid_mutex_suspend () >>>> from /lib64/libdb-5.3.so >>>> #2 0x00007f68375abb37 in __db_tas_mutex_readlock () >>>> from /lib64/libdb-5.3.so >>>> #3 0x00007f68376c5d35 in __memp_fget () from >>>> /lib64/libdb-5.3.so >>>> #4 0x00007f683766f333 in __dbc_iget () from >>>> /lib64/libdb-5.3.so >>>> #5 0x00007f683767db02 in __dbc_get_pp () from >>>> /lib64/libdb-5.3.so >>>> #6 0x00007f6832b59822 in idl_new_fetch () from >>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>> #7 0x00007f6832b67b26 in index_read_ext_allids () from >>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>> #8 0x00007f6832b52284 in keys2idl () from >>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>> #9 0x00007f6832b529e3 in ava_candidates.isra.0 () from >>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>> #10 0x00007f6832b52fd2 in filter_candidates_ext () from >>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>> #11 0x00007f6832b54096 in list_candidates () from >>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>> #12 0x00007f6832b52f40 in filter_candidates_ext () from >>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>> #13 0x00007f6832b54096 in list_candidates () from >>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>> #14 0x00007f6832b52f40 in filter_candidates_ext () from >>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>> #15 0x00007f6832b54096 in list_candidates () from >>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>> #16 0x00007f6832b52f40 in filter_candidates_ext () from >>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>> #17 0x00007f6832b8e80c in subtree_candidates () from >>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>> #18 0x00007f6832b8fe7f in ldbm_back_search () from >>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>> #19 0x00007f683f4bd1d6 in op_shared_search () from >>>> /usr/lib64/dirsrv/libslapd.so.0 >>>> #20 0x00007f683f9ae33e in do_search () >>>> #21 0x00007f683f99c3c3 in connection_threadmain () >>>> #22 0x00007f683d6ab7bb in _pt_root () from >>>> /lib64/libnspr4.so >>>> #23 0x00007f683d04cdc5 in start_thread () from >>>> /lib64/libpthread.so.0 >>>> #24 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 >>>> Thread 23 (Thread 0x7f6801ffb700 (LWP 4870)): >>>> #0 0x00007f683d0506d5 in >>>> pthread_cond_wait@@GLIBC_2.3.2 >>>> () from >>>> /lib64/libpthread.so.0 >>>> #1 0x00007f683d6a6050 in PR_WaitCondVar () from >>>> /lib64/libnspr4.so >>>> #2 0x00007f683f999fee in connection_wait_for_new_work () >>>> #3 0x00007f683f99b719 in connection_threadmain () >>>> #4 0x00007f683d6ab7bb in _pt_root () from >>>> /lib64/libnspr4.so >>>> #5 0x00007f683d04cdc5 in start_thread () from >>>> /lib64/libpthread.so.0 >>>> #6 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 >>>> Thread 22 (Thread 0x7f68017fa700 (LWP 4871)): >>>> #0 0x00007f683d0506d5 in >>>> pthread_cond_wait@@GLIBC_2.3.2 >>>> () from >>>> /lib64/libpthread.so.0 >>>> #1 0x00007f683d6a6050 in PR_WaitCondVar () from >>>> /lib64/libnspr4.so >>>> #2 0x00007f683f999fee in connection_wait_for_new_work () >>>> #3 0x00007f683f99b719 in connection_threadmain () >>>> #4 0x00007f683d6ab7bb in _pt_root () from >>>> /lib64/libnspr4.so >>>> #5 0x00007f683d04cdc5 in start_thread () from >>>> /lib64/libpthread.so.0 >>>> #6 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 >>>> Thread 21 (Thread 0x7f6800ff9700 (LWP 4872)): >>>> #0 0x00007f683d0506d5 in >>>> pthread_cond_wait@@GLIBC_2.3.2 >>>> () from >>>> /lib64/libpthread.so.0 >>>> #1 0x00007f683d6a6050 in PR_WaitCondVar () from >>>> /lib64/libnspr4.so >>>> #2 0x00007f683f999fee in connection_wait_for_new_work () >>>> #3 0x00007f683f99b719 in connection_threadmain () >>>> #4 0x00007f683d6ab7bb in _pt_root () from >>>> /lib64/libnspr4.so >>>> #5 0x00007f683d04cdc5 in start_thread () from >>>> /lib64/libpthread.so.0 >>>> #6 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 >>>> Thread 20 (Thread 0x7f68007f8700 (LWP 4873)): >>>> #0 0x00007f683d0506d5 in >>>> pthread_cond_wait@@GLIBC_2.3.2 >>>> () from >>>> /lib64/libpthread.so.0 >>>> #1 0x00007f68375ac2f3 in __db_hybrid_mutex_suspend () >>>> from /lib64/libdb-5.3.so >>>> #2 0x00007f68375ab640 in __db_tas_mutex_lock () from >>>> /lib64/libdb-5.3.so >>>> #3 0x00007f68376c5e1b in __memp_fget () from >>>> /lib64/libdb-5.3.so >>>> #4 0x00007f68375c9211 in __bam_search () from >>>> /lib64/libdb-5.3.so >>>> #5 0x00007f68375b4256 in __bamc_search () from >>>> /lib64/libdb-5.3.so >>>> #6 0x00007f68375b86c4 in __bamc_put () from >>>> /lib64/libdb-5.3.so >>>> #7 0x00007f683766f6d5 in __dbc_iput () from >>>> /lib64/libdb-5.3.so >>>> #8 0x00007f683766aa0e in __db_put () from >>>> /lib64/libdb-5.3.so >>>> #9 0x00007f683767ffa4 in __db_put_pp () from >>>> /lib64/libdb-5.3.so >>>> #10 0x00007f6832b5a4cf in idl_new_insert_key () from >>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>> #11 0x00007f6832b695b5 in addordel_values_sv.isra.3 () >>>> from /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>> #12 0x00007f6832b6a25a in index_addordel_values_ext_sv >>>> () from /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>> #13 0x00007f6832b6a694 in index_addordel_values_sv () >>>> from /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>> #14 0x00007f6832b6a75a in index_addordel_entry () from >>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>> #15 0x00007f6832b6d5fa in ldbm_back_add () from >>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>> #16 0x00007f683f46fcca in op_shared_add () from >>>> /usr/lib64/dirsrv/libslapd.so.0 >>>> #17 0x00007f683f471020 in do_add () from >>>> /usr/lib64/dirsrv/libslapd.so.0 >>>> #18 0x00007f683f99c303 in connection_threadmain () >>>> #19 0x00007f683d6ab7bb in _pt_root () from >>>> /lib64/libnspr4.so >>>> #20 0x00007f683d04cdc5 in start_thread () from >>>> /lib64/libpthread.so.0 >>>> #21 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 >>>> Thread 19 (Thread 0x7f67ffff7700 (LWP 4874)): >>>> #0 0x00007f683d0506d5 in >>>> pthread_cond_wait@@GLIBC_2.3.2 >>>> () from >>>> /lib64/libpthread.so.0 >>>> #1 0x00007f683d6a62c3 in PR_EnterMonitor () from >>>> /lib64/libnspr4.so >>>> #2 0x00007f6832b4d3e6 in dblayer_txn_begin () from >>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>> #3 0x00007f6832b89007 in ldbm_back_modify () from >>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>> #4 0x00007f683f4b6541 in op_shared_modify () from >>>> /usr/lib64/dirsrv/libslapd.so.0 >>>> #5 0x00007f683f4b7004 in modify_internal_pb () from >>>> /usr/lib64/dirsrv/libslapd.so.0 >>>> #6 0x00007f6832dd3b60 in ipapwd_apply_mods () from >>>> /usr/lib64/dirsrv/plugins/libipa_pwd_extop.so >>>> #7 0x00007f6832dd78e0 in store_new_keys () from >>>> /usr/lib64/dirsrv/plugins/libipa_pwd_extop.so >>>> #8 0x00007f6832dd9ad0 in ipapwd_extop () from >>>> /usr/lib64/dirsrv/plugins/libipa_pwd_extop.so >>>> #9 0x00007f683f4c6ba2 in plugin_call_exop_plugins () >>>> from /usr/lib64/dirsrv/libslapd.so.0 >>>> #10 0x00007f683f9a2efe in do_extended () >>>> #11 0x00007f683f99c41a in connection_threadmain () >>>> #12 0x00007f683d6ab7bb in _pt_root () from >>>> /lib64/libnspr4.so >>>> #13 0x00007f683d04cdc5 in start_thread () from >>>> /lib64/libpthread.so.0 >>>> #14 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 >>>> Thread 18 (Thread 0x7f67ff7f6700 (LWP 4875)): >>>> #0 0x00007f683d0506d5 in >>>> pthread_cond_wait@@GLIBC_2.3.2 >>>> () from >>>> /lib64/libpthread.so.0 >>>> #1 0x00007f683d6a6050 in PR_WaitCondVar () from >>>> /lib64/libnspr4.so >>>> #2 0x00007f683f999fee in connection_wait_for_new_work () >>>> #3 0x00007f683f99b719 in connection_threadmain () >>>> #4 0x00007f683d6ab7bb in _pt_root () from >>>> /lib64/libnspr4.so >>>> #5 0x00007f683d04cdc5 in start_thread () from >>>> /lib64/libpthread.so.0 >>>> #6 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 >>>> Thread 17 (Thread 0x7f67feff5700 (LWP 4876)): >>>> #0 0x00007f683d0506d5 in >>>> pthread_cond_wait@@GLIBC_2.3.2 >>>> () from >>>> /lib64/libpthread.so.0 >>>> #1 0x00007f68375ac2f3 in __db_hybrid_mutex_suspend () >>>> from /lib64/libdb-5.3.so >>>> #2 0x00007f68375ab640 in __db_tas_mutex_lock () from >>>> /lib64/libdb-5.3.so >>>> #3 0x00007f6837655d3a in __lock_get_internal () from >>>> /lib64/libdb-5.3.so >>>> #4 0x00007f6837656820 in __lock_get () from >>>> /lib64/libdb-5.3.so >>>> #5 0x00007f6837682142 in __db_lget () from >>>> /lib64/libdb-5.3.so >>>> #6 0x00007f68375c9605 in __bam_search () from >>>> /lib64/libdb-5.3.so >>>> #7 0x00007f68375b4256 in __bamc_search () from >>>> /lib64/libdb-5.3.so >>>> #8 0x00007f68375b5d0f in __bamc_get () from >>>> /lib64/libdb-5.3.so >>>> #9 0x00007f683766eca6 in __dbc_iget () from >>>> /lib64/libdb-5.3.so >>>> #10 0x00007f683767db02 in __dbc_get_pp () from >>>> /lib64/libdb-5.3.so >>>> #11 0x00007f6832b59680 in idl_new_fetch () from >>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>> #12 0x00007f6832b67b26 in index_read_ext_allids () from >>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>> #13 0x00007f6832b52284 in keys2idl () from >>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>> #14 0x00007f6832b529e3 in ava_candidates.isra.0 () from >>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>> #15 0x00007f6832b52fd2 in filter_candidates_ext () from >>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>> #16 0x00007f6832b54096 in list_candidates () from >>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>> #17 0x00007f6832b52f40 in filter_candidates_ext () from >>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>> #18 0x00007f6832b8e80c in subtree_candidates () from >>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>> #19 0x00007f6832b8fe7f in ldbm_back_search () from >>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>> #20 0x00007f683f4bd1d6 in op_shared_search () from >>>> /usr/lib64/dirsrv/libslapd.so.0 >>>> #21 0x00007f683f4cd94e in search_internal_callback_pb >>>> () from /usr/lib64/dirsrv/libslapd.so.0 >>>> #22 0x00007f683f4cdbe8 in search_internal_pb () from >>>> /usr/lib64/dirsrv/libslapd.so.0 >>>> #23 0x00007f6832dd75dc in get_entry_by_principal () >>>> from /usr/lib64/dirsrv/plugins/libipa_pwd_extop.so >>>> #24 0x00007f6832dd8eb1 in ipapwd_extop () from >>>> /usr/lib64/dirsrv/plugins/libipa_pwd_extop.so >>>> #25 0x00007f683f4c6ba2 in plugin_call_exop_plugins () >>>> from /usr/lib64/dirsrv/libslapd.so.0 >>>> #26 0x00007f683f9a2efe in do_extended () >>>> #27 0x00007f683f99c41a in connection_threadmain () >>>> #28 0x00007f683d6ab7bb in _pt_root () from >>>> /lib64/libnspr4.so >>>> #29 0x00007f683d04cdc5 in start_thread () from >>>> /lib64/libpthread.so.0 >>>> #30 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 >>>> Thread 16 (Thread 0x7f67fe7f4700 (LWP 4877)): >>>> #0 0x00007f683d0506d5 in >>>> pthread_cond_wait@@GLIBC_2.3.2 >>>> () from >>>> /lib64/libpthread.so.0 >>>> #1 0x00007f683d6a6050 in PR_WaitCondVar () from >>>> /lib64/libnspr4.so >>>> #2 0x00007f683f999fee in connection_wait_for_new_work () >>>> #3 0x00007f683f99b719 in connection_threadmain () >>>> #4 0x00007f683d6ab7bb in _pt_root () from >>>> /lib64/libnspr4.so >>>> #5 0x00007f683d04cdc5 in start_thread () from >>>> /lib64/libpthread.so.0 >>>> #6 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 >>>> Thread 15 (Thread 0x7f67fdff3700 (LWP 4878)): >>>> #0 0x00007f683d0506d5 in >>>> pthread_cond_wait@@GLIBC_2.3.2 >>>> () from >>>> /lib64/libpthread.so.0 >>>> #1 0x00007f683d6a6050 in PR_WaitCondVar () from >>>> /lib64/libnspr4.so >>>> #2 0x00007f683f999fee in connection_wait_for_new_work () >>>> #3 0x00007f683f99b719 in connection_threadmain () >>>> #4 0x00007f683d6ab7bb in _pt_root () from >>>> /lib64/libnspr4.so >>>> #5 0x00007f683d04cdc5 in start_thread () from >>>> /lib64/libpthread.so.0 >>>> #6 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 >>>> Thread 14 (Thread 0x7f67fd7f2700 (LWP 4879)): >>>> #0 0x00007f683d0506d5 in >>>> pthread_cond_wait@@GLIBC_2.3.2 >>>> () from >>>> /lib64/libpthread.so.0 >>>> #1 0x00007f683d6a6050 in PR_WaitCondVar () from >>>> /lib64/libnspr4.so >>>> #2 0x00007f683f999fee in connection_wait_for_new_work () >>>> #3 0x00007f683f99b719 in connection_threadmain () >>>> #4 0x00007f683d6ab7bb in _pt_root () from >>>> /lib64/libnspr4.so >>>> #5 0x00007f683d04cdc5 in start_thread () from >>>> /lib64/libpthread.so.0 >>>> #6 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 >>>> Thread 13 (Thread 0x7f67fcff1700 (LWP 4880)): >>>> #0 0x00007f683d0506d5 in >>>> pthread_cond_wait@@GLIBC_2.3.2 >>>> () from >>>> /lib64/libpthread.so.0 >>>> #1 0x00007f683d6a6050 in PR_WaitCondVar () from >>>> /lib64/libnspr4.so >>>> #2 0x00007f683f999fee in connection_wait_for_new_work () >>>> #3 0x00007f683f99b719 in connection_threadmain () >>>> #4 0x00007f683d6ab7bb in _pt_root () from >>>> /lib64/libnspr4.so >>>> #5 0x00007f683d04cdc5 in start_thread () from >>>> /lib64/libpthread.so.0 >>>> #6 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 >>>> Thread 12 (Thread 0x7f67fc7f0700 (LWP 4881)): >>>> #0 0x00007f683d0506d5 in >>>> pthread_cond_wait@@GLIBC_2.3.2 >>>> () from >>>> /lib64/libpthread.so.0 >>>> #1 0x00007f683d6a6050 in PR_WaitCondVar () from >>>> /lib64/libnspr4.so >>>> #2 0x00007f683f999fee in connection_wait_for_new_work () >>>> #3 0x00007f683f99b719 in connection_threadmain () >>>> #4 0x00007f683d6ab7bb in _pt_root () from >>>> /lib64/libnspr4.so >>>> #5 0x00007f683d04cdc5 in start_thread () from >>>> /lib64/libpthread.so.0 >>>> #6 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 >>>> Thread 11 (Thread 0x7f67fbfef700 (LWP 4882)): >>>> #0 0x00007f683d0506d5 in >>>> pthread_cond_wait@@GLIBC_2.3.2 >>>> () from >>>> /lib64/libpthread.so.0 >>>> #1 0x00007f683d6a6050 in PR_WaitCondVar () from >>>> /lib64/libnspr4.so >>>> #2 0x00007f683f999fee in connection_wait_for_new_work () >>>> #3 0x00007f683f99b719 in connection_threadmain () >>>> #4 0x00007f683d6ab7bb in _pt_root () from >>>> /lib64/libnspr4.so >>>> #5 0x00007f683d04cdc5 in start_thread () from >>>> /lib64/libpthread.so.0 >>>> #6 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 >>>> Thread 10 (Thread 0x7f67fb7ee700 (LWP 4883)): >>>> #0 0x00007f683d0506d5 in >>>> pthread_cond_wait@@GLIBC_2.3.2 >>>> () from >>>> /lib64/libpthread.so.0 >>>> #1 0x00007f68375ac2f3 in __db_hybrid_mutex_suspend () >>>> from /lib64/libdb-5.3.so >>>> #2 0x00007f68375ab640 in __db_tas_mutex_lock () from >>>> /lib64/libdb-5.3.so >>>> #3 0x00007f6837655d3a in __lock_get_internal () from >>>> /lib64/libdb-5.3.so >>>> #4 0x00007f6837656820 in __lock_get () from >>>> /lib64/libdb-5.3.so >>>> #5 0x00007f6837682142 in __db_lget () from >>>> /lib64/libdb-5.3.so >>>> #6 0x00007f68375c9605 in __bam_search () from >>>> /lib64/libdb-5.3.so >>>> #7 0x00007f68375b4256 in __bamc_search () from >>>> /lib64/libdb-5.3.so >>>> #8 0x00007f68375b5d0f in __bamc_get () from >>>> /lib64/libdb-5.3.so >>>> #9 0x00007f683766eca6 in __dbc_iget () from >>>> /lib64/libdb-5.3.so >>>> #10 0x00007f683767db02 in __dbc_get_pp () from >>>> /lib64/libdb-5.3.so >>>> #11 0x00007f6832b59680 in idl_new_fetch () from >>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>> #12 0x00007f6832b67b26 in index_read_ext_allids () from >>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>> #13 0x00007f6832b52284 in keys2idl () from >>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>> #14 0x00007f6832b529e3 in ava_candidates.isra.0 () from >>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>> #15 0x00007f6832b52fd2 in filter_candidates_ext () from >>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>> #16 0x00007f6832b54096 in list_candidates () from >>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>> #17 0x00007f6832b52f40 in filter_candidates_ext () from >>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>> #18 0x00007f6832b54096 in list_candidates () from >>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>> #19 0x00007f6832b52f40 in filter_candidates_ext () from >>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>> #20 0x00007f6832b54096 in list_candidates () from >>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>> #21 0x00007f6832b52f40 in filter_candidates_ext () from >>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>> #22 0x00007f6832b8e80c in subtree_candidates () from >>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>> #23 0x00007f6832b8fe7f in ldbm_back_search () from >>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>> #24 0x00007f683f4bd1d6 in op_shared_search () from >>>> /usr/lib64/dirsrv/libslapd.so.0 >>>> #25 0x00007f683f9ae33e in do_search () >>>> #26 0x00007f683f99c3c3 in connection_threadmain () >>>> #27 0x00007f683d6ab7bb in _pt_root () from >>>> /lib64/libnspr4.so >>>> #28 0x00007f683d04cdc5 in start_thread () from >>>> /lib64/libpthread.so.0 >>>> #29 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 >>>> Thread 9 (Thread 0x7f67fafed700 (LWP 4884)): >>>> #0 0x00007f683d0506d5 in >>>> pthread_cond_wait@@GLIBC_2.3.2 >>>> () from >>>> /lib64/libpthread.so.0 >>>> #1 0x00007f683d6a6050 in PR_WaitCondVar () from >>>> /lib64/libnspr4.so >>>> #2 0x00007f683f999fee in connection_wait_for_new_work () >>>> #3 0x00007f683f99b719 in connection_threadmain () >>>> #4 0x00007f683d6ab7bb in _pt_root () from >>>> /lib64/libnspr4.so >>>> #5 0x00007f683d04cdc5 in start_thread () from >>>> /lib64/libpthread.so.0 >>>> #6 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 >>>> Thread 8 (Thread 0x7f67fa7ec700 (LWP 4885)): >>>> #0 0x00007f683d0506d5 in >>>> pthread_cond_wait@@GLIBC_2.3.2 >>>> () from >>>> /lib64/libpthread.so.0 >>>> #1 0x00007f68375ac2f3 in __db_hybrid_mutex_suspend () >>>> from /lib64/libdb-5.3.so >>>> #2 0x00007f68375ab640 in __db_tas_mutex_lock () from >>>> /lib64/libdb-5.3.so >>>> #3 0x00007f6837655d3a in __lock_get_internal () from >>>> /lib64/libdb-5.3.so >>>> #4 0x00007f6837656820 in __lock_get () from >>>> /lib64/libdb-5.3.so >>>> #5 0x00007f6837682142 in __db_lget () from >>>> /lib64/libdb-5.3.so >>>> #6 0x00007f68375c9605 in __bam_search () from >>>> /lib64/libdb-5.3.so >>>> #7 0x00007f68375b4256 in __bamc_search () from >>>> /lib64/libdb-5.3.so >>>> #8 0x00007f68375b5d0f in __bamc_get () from >>>> /lib64/libdb-5.3.so >>>> #9 0x00007f683766eca6 in __dbc_iget () from >>>> /lib64/libdb-5.3.so >>>> #10 0x00007f683767db02 in __dbc_get_pp () from >>>> /lib64/libdb-5.3.so >>>> #11 0x00007f6832b59680 in idl_new_fetch () from >>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>> #12 0x00007f6832b67b26 in index_read_ext_allids () from >>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>> #13 0x00007f6832b52284 in keys2idl () from >>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>> #14 0x00007f6832b529e3 in ava_candidates.isra.0 () from >>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>> #15 0x00007f6832b52fd2 in filter_candidates_ext () from >>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>> #16 0x00007f6832b54096 in list_candidates () from >>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>> #17 0x00007f6832b52f40 in filter_candidates_ext () from >>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>> #18 0x00007f6832b54096 in list_candidates () from >>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>> #19 0x00007f6832b52f40 in filter_candidates_ext () from >>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>> #20 0x00007f6832b54096 in list_candidates () from >>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>> #21 0x00007f6832b52f40 in filter_candidates_ext () from >>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>> #22 0x00007f6832b8e80c in subtree_candidates () from >>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>> #23 0x00007f6832b8fe7f in ldbm_back_search () from >>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>> #24 0x00007f683f4bd1d6 in op_shared_search () from >>>> /usr/lib64/dirsrv/libslapd.so.0 >>>> #25 0x00007f683f9ae33e in do_search () >>>> #26 0x00007f683f99c3c3 in connection_threadmain () >>>> #27 0x00007f683d6ab7bb in _pt_root () from >>>> /lib64/libnspr4.so >>>> #28 0x00007f683d04cdc5 in start_thread () from >>>> /lib64/libpthread.so.0 >>>> #29 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 >>>> Thread 7 (Thread 0x7f67f9feb700 (LWP 4886)): >>>> #0 0x00007f683d0506d5 in >>>> pthread_cond_wait@@GLIBC_2.3.2 >>>> () from >>>> /lib64/libpthread.so.0 >>>> #1 0x00007f683d6a6050 in PR_WaitCondVar () from >>>> /lib64/libnspr4.so >>>> #2 0x00007f683f999fee in connection_wait_for_new_work () >>>> #3 0x00007f683f99b719 in connection_threadmain () >>>> #4 0x00007f683d6ab7bb in _pt_root () from >>>> /lib64/libnspr4.so >>>> #5 0x00007f683d04cdc5 in start_thread () from >>>> /lib64/libpthread.so.0 >>>> #6 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 >>>> Thread 6 (Thread 0x7f67f97ea700 (LWP 4887)): >>>> #0 0x00007f683d0506d5 in >>>> pthread_cond_wait@@GLIBC_2.3.2 >>>> () from >>>> /lib64/libpthread.so.0 >>>> #1 0x00007f683d6a6050 in PR_WaitCondVar () from >>>> /lib64/libnspr4.so >>>> #2 0x00007f683f999fee in connection_wait_for_new_work () >>>> #3 0x00007f683f99b719 in connection_threadmain () >>>> #4 0x00007f683d6ab7bb in _pt_root () from >>>> /lib64/libnspr4.so >>>> #5 0x00007f683d04cdc5 in start_thread () from >>>> /lib64/libpthread.so.0 >>>> #6 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 >>>> Thread 5 (Thread 0x7f67f8fe9700 (LWP 4888)): >>>> #0 0x00007f683d0506d5 in >>>> pthread_cond_wait@@GLIBC_2.3.2 >>>> () from >>>> /lib64/libpthread.so.0 >>>> #1 0x00007f683d6a6050 in PR_WaitCondVar () from >>>> /lib64/libnspr4.so >>>> #2 0x00007f683f999fee in connection_wait_for_new_work () >>>> #3 0x00007f683f99b719 in connection_threadmain () >>>> #4 0x00007f683d6ab7bb in _pt_root () from >>>> /lib64/libnspr4.so >>>> #5 0x00007f683d04cdc5 in start_thread () from >>>> /lib64/libpthread.so.0 >>>> #6 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 >>>> Thread 4 (Thread 0x7f67f87e8700 (LWP 4889)): >>>> #0 0x00007f683cd719b3 in select () from /lib64/libc.so.6 >>>> #1 0x00007f683f4fc0e9 in DS_Sleep () from >>>> /usr/lib64/dirsrv/libslapd.so.0 >>>> #2 0x00007f683f99d325 in time_thread () >>>> #3 0x00007f683d6ab7bb in _pt_root () from >>>> /lib64/libnspr4.so >>>> #4 0x00007f683d04cdc5 in start_thread () from >>>> /lib64/libpthread.so.0 >>>> #5 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 >>>> Thread 3 (Thread 0x7f67f7de6700 (LWP 4892)): >>>> #0 0x00007f683d0506d5 in >>>> pthread_cond_wait@@GLIBC_2.3.2 >>>> () from >>>> /lib64/libpthread.so.0 >>>> #1 0x00007f683d6a6050 in PR_WaitCondVar () from >>>> /lib64/libnspr4.so >>>> #2 0x00007f683f9a8f25 in ps_send_results () >>>> #3 0x00007f683d6ab7bb in _pt_root () from >>>> /lib64/libnspr4.so >>>> #4 0x00007f683d04cdc5 in start_thread () from >>>> /lib64/libpthread.so.0 >>>> #5 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 >>>> Thread 2 (Thread 0x7f67f75e5700 (LWP 5084)): >>>> #0 0x00007f683d0506d5 in >>>> pthread_cond_wait@@GLIBC_2.3.2 >>>> () from >>>> /lib64/libpthread.so.0 >>>> #1 0x00007f683d6a6050 in PR_WaitCondVar () from >>>> /lib64/libnspr4.so >>>> #2 0x00007f683f9a8f25 in ps_send_results () >>>> #3 0x00007f683d6ab7bb in _pt_root () from >>>> /lib64/libnspr4.so >>>> #4 0x00007f683d04cdc5 in start_thread () from >>>> /lib64/libpthread.so.0 >>>> #5 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 >>>> Thread 1 (Thread 0x7f683f964840 (LWP 4845)): >>>> #0 0x00007f683d0506d5 in >>>> pthread_cond_wait@@GLIBC_2.3.2 >>>> () from >>>> /lib64/libpthread.so.0 >>>> #1 0x00007f683d6a62c3 in PR_EnterMonitor () from >>>> /lib64/libnspr4.so >>>> #2 0x00007f683f99d5ed in setup_pr_read_pds () >>>> #3 0x00007f683f9a0fd9 in slapd_daemon () >>>> #4 0x00007f683f992c6c in main () >>>> >>>> >>>> >>>> the top output seems ok, there was just a sudden spike >>>> to around 340% cpu then it comes down >>>> >>>> top - 08:53:12 up 6 days, 23:20, 1 user, load >>>> average: 0.00, 0.01, 0.12 >>>> Threads: 44 total, 0 running, 44 sleeping, 0 >>>> stopped, 0 zombie >>>> %Cpu(s): 0.0 us, 0.0 sy, 0.0 ni,100.0 id, 0.0 wa, >>>> 0.0 hi, 0.0 si, 0.0 st >>>> KiB Mem : 65533176 total, 52165676 free, 6135088 used, >>>> 7232412 buff/cache >>>> KiB Swap: 3999996 total, 3999996 free, 0 used. >>>> 58452040 avail Mem >>>> >>>> PID USER PR NI VIRT RES SHR S %CPU >>>> %MEM TIME+ COMMAND >>>> 4845 dirsrv 20 0 5133196 3.150g 281272 S 0.0 >>>> 5.0 31:34.51 ns-slapd >>>> 4847 dirsrv 20 0 5133196 3.150g 281272 S 0.0 >>>> 5.0 0:13.53 ns-slapd >>>> 4848 dirsrv 20 0 5133196 3.150g 281272 S 0.0 >>>> 5.0 0:13.16 ns-slapd >>>> 4849 dirsrv 20 0 5133196 3.150g 281272 S 0.0 >>>> 5.0 0:45.27 ns-slapd >>>> 4850 dirsrv 20 0 5133196 3.150g 281272 S 0.0 >>>> 5.0 0:01.60 ns-slapd >>>> 4851 dirsrv 20 0 5133196 3.150g 281272 S 0.0 >>>> 5.0 0:00.00 ns-slapd >>>> 4852 dirsrv 20 0 5133196 3.150g 281272 S 0.0 >>>> 5.0 0:00.00 ns-slapd >>>> 4853 dirsrv 20 0 5133196 3.150g 281272 S 0.0 >>>> 5.0 0:00.00 ns-slapd >>>> 4854 dirsrv 20 0 5133196 3.150g 281272 S 0.0 >>>> 5.0 0:00.00 ns-slapd >>>> 4855 dirsrv 20 0 5133196 3.150g 281272 S 0.0 >>>> 5.0 0:03.02 ns-slapd >>>> 4856 dirsrv 20 0 5133196 3.150g 281272 S 0.0 >>>> 5.0 0:00.26 ns-slapd >>>> 4859 dirsrv 20 0 5133196 3.150g 281272 S 0.0 >>>> 5.0 3:18.67 ns-slapd >>>> 4860 dirsrv 20 0 5133196 3.150g 281272 S 0.0 >>>> 5.0 3:19.82 ns-slapd >>>> 4861 dirsrv 20 0 5133196 3.150g 281272 S 0.0 >>>> 5.0 3:20.23 ns-slapd >>>> 4862 dirsrv 20 0 5133196 3.150g 281272 S 0.0 >>>> 5.0 3:22.65 ns-slapd >>>> 4863 dirsrv 20 0 5133196 3.150g 281272 S 0.0 >>>> 5.0 3:16.64 ns-slapd >>>> 4864 dirsrv 20 0 5133196 3.150g 281272 S 0.0 >>>> 5.0 3:17.39 ns-slapd >>>> 4865 dirsrv 20 0 5133196 3.150g 281272 S 0.0 >>>> 5.0 3:17.80 ns-slapd >>>> >>>> >>>> Thanks >>>> Rakesh >>>> >>>> On Tue, Sep 6, 2016 at 12:09 AM, Rakesh Rajasekharan >>>> >>> > wrote: >>>> >>>> Hi Thierry/Ludwig, >>>> >>>> Unfortunately I had by then restarted the service >>>> and since then have been trying to hit the hang >>>> issue again but no luck this time >>>> >>>> It has been running fine for the last few hours now >>>> in-spite of trying all that I believe could cause >>>> the hang like making simultaneous client addition >>>> requests. >>>> >>>> I will continue testing this and next time I get it >>>> I will make sure I get the pstack and also get the >>>> dump of db lock info. >>>> >>>> Thanks, >>>> Rakesh >>>> >>>> >>>> >>>> On Mon, Sep 5, 2016 at 6:14 PM, Ludwig Krispenz >>>> > >>>> wrote: >>>> >>>> >>>> On 09/05/2016 02:31 PM, thierry bordaz wrote: >>>>> Hi Rakesh, >>>>> >>>>> Thanks you soooo much for you continuous help >>>>> on this !! >>>>> >>>>> If the hang is still going on, would you dump >>>>> the DB lock information with: >>>>> >>>>> db_stat -CA -N -h >>>>> /var/lib/dirsrv/slapd-/db >>>>> >/tmp/db_lock_info >>>>> >>>>> It is looking like several threads are waiting >>>>> for one/several db locks. With the lock info >>>>> we will know which thread acquired it/them. >>>>> A guess is that it could be the checkpointing >>>>> thread but this one is also waiting (on a >>>>> memory pool). >>>>> >>>>> I am copying Ludwig on this thread in case >>>>> this pstack rings any bell. >>>> the frequent presence of __db_tas_mutex_lock() >>>> looks like the NUMA problem of libdb, although >>>> the threads already have stopped spinning >>>> (maybe this was an effect of gcore). >>>> If the hang aoccurs again, could you check cpu >>>> consumption >>>> top -H -p >>>> and get a pstack >>>> pstack >>>>> >>>>> best regards >>>>> thierry >>>>> >>>>> On 09/05/2016 02:05 PM, Rakesh Rajasekharan wrote: >>>>>> Hi Thierry, >>>>>> >>>>>> I was able to now reporduce the issue.. >>>>>> >>>>>> the ns-slapd process just got hung,.. >>>>>> however, no core file was generate.. >>>>>> >>>>>> I manually created a coredump using gcore and >>>>>> then took a strace of it >>>>>> this is strace from the coredump generated >>>>>> >>>>>> GNU gdb (GDB) Red Hat Enterprise Linux >>>>>> 7.6.1-80.el7 >>>>>> Copyright (C) 2013 Free Software Foundation, Inc. >>>>>> License GPLv3+: GNU GPL version 3 or later >>>>>> >>>>>> This is free software: you are free to change >>>>>> and redistribute it. >>>>>> There is NO WARRANTY, to the extent permitted >>>>>> by law. Type "show copying" >>>>>> and "show warranty" for details. >>>>>> This GDB was configured as >>>>>> "x86_64-redhat-linux-gnu". >>>>>> For bug reporting instructions, please see: >>>>>> ... >>>>>> Reading symbols from >>>>>> /usr/sbin/ns-slapd...Reading symbols from >>>>>> /usr/sbin/ns-slapd...(no debugging symbols >>>>>> found)...done. >>>>>> (no debugging symbols found)...done. >>>>>> [New LWP 105159] >>>>>> [New LWP 105160] >>>>>> [New LWP 105161] >>>>>> [New LWP 105162] >>>>>> [New LWP 105163] >>>>>> [New LWP 105164] >>>>>> [New LWP 105165] >>>>>> [New LWP 105166] >>>>>> [New LWP 105167] >>>>>> [New LWP 105168] >>>>>> [New LWP 105171] >>>>>> [New LWP 105172] >>>>>> [New LWP 105173] >>>>>> [New LWP 105174] >>>>>> [New LWP 105175] >>>>>> [New LWP 105176] >>>>>> [New LWP 105177] >>>>>> [New LWP 105178] >>>>>> [New LWP 105179] >>>>>> [New LWP 105180] >>>>>> [New LWP 105181] >>>>>> [New LWP 105182] >>>>>> [New LWP 105183] >>>>>> [New LWP 105184] >>>>>> [New LWP 105185] >>>>>> [New LWP 105186] >>>>>> [New LWP 105187] >>>>>> [New LWP 105188] >>>>>> [New LWP 105189] >>>>>> [New LWP 105190] >>>>>> [New LWP 105191] >>>>>> [New LWP 105192] >>>>>> [New LWP 105193] >>>>>> [New LWP 105194] >>>>>> [New LWP 105195] >>>>>> [New LWP 105196] >>>>>> [New LWP 105197] >>>>>> [New LWP 105198] >>>>>> [New LWP 105199] >>>>>> [New LWP 105200] >>>>>> [New LWP 105201] >>>>>> [New LWP 105203] >>>>>> [New LWP 105215] >>>>>> [New LWP 105158] >>>>>> [Thread debugging using libthread_db enabled] >>>>>> Using host libthread_db library >>>>>> "/lib64/libthread_db.so.1". >>>>>> Core was generated by `/usr/sbin/ns-slapd'. >>>>>> #0 0x00007f1b48ae69b3 in select () from >>>>>> /lib64/libc.so.6 >>>>>> >>>>>> Thread 44 (Thread 0x7f1b4b6d9840 (LWP 105158)): >>>>>> #0 0x00007f1b48dc56d5 in >>>>>> pthread_cond_wait@@GLIBC_2.3.2 >>>>>> () >>>>>> from /lib64/libpthread.so.0 >>>>>> No symbol table info available. >>>>>> #1 0x00007f1b4941b2c3 in PR_EnterMonitor () >>>>>> from /lib64/libnspr4.so >>>>>> No symbol table info available. >>>>>> #2 0x00007f1b4b7125ed in setup_pr_read_pds () >>>>>> No symbol table info available. >>>>>> #3 0x00007f1b4b715fd9 in slapd_daemon () >>>>>> No symbol table info available. >>>>>> #4 0x00007f1b4b707c6c in main () >>>>>> No symbol table info available. >>>>>> >>>>>> Thread 43 (Thread 0x7f1a5df0f700 (LWP 105215)): >>>>>> #0 0x00007f1b48dc56d5 in >>>>>> pthread_cond_wait@@GLIBC_2.3.2 >>>>>> () >>>>>> from /lib64/libpthread.so.0 >>>>>> No symbol table info available. >>>>>> #1 0x00007f1b4941b050 in PR_WaitCondVar () >>>>>> from /lib64/libnspr4.so >>>>>> No symbol table info available. >>>>>> #2 0x00007f1b4b71df25 in ps_send_results () >>>>>> No symbol table info available. >>>>>> #3 0x00007f1b494207bb in _pt_root () from >>>>>> /lib64/libnspr4.so >>>>>> No symbol table info available. >>>>>> #4 0x00007f1b48dc1dc5 in start_thread () from >>>>>> /lib64/libpthread.so.0 >>>>>> No symbol table info available. >>>>>> #5 0x00007f1b48aef28d in clone () from >>>>>> /lib64/libc.so.6 >>>>>> No symbol table info available. >>>>>> >>>>>> Thread 42 (Thread 0x7f1b03fe7700 (LWP 105203)): >>>>>> #0 0x00007f1b48dc56d5 in >>>>>> pthread_cond_wait@@GLIBC_2.3.2 >>>>>> () >>>>>> from /lib64/libpthread.so.0 >>>>>> No symbol table info available. >>>>>> #1 0x00007f1b4941b050 in PR_WaitCondVar () >>>>>> from /lib64/libnspr4.so >>>>>> No symbol table info available. >>>>>> #2 0x00007f1b4b71df25 in ps_send_results () >>>>>> No symbol table info available. >>>>>> #3 0x00007f1b494207bb in _pt_root () from >>>>>> /lib64/libnspr4.so >>>>>> No symbol table info available. >>>>>> #4 0x00007f1b48dc1dc5 in start_thread () from >>>>>> /lib64/libpthread.so.0 >>>>>> No symbol table info available. >>>>>> #5 0x00007f1b48aef28d in clone () from >>>>>> /lib64/libc.so.6 >>>>>> No symbol table info available. >>>>>> >>>>>> Thread 41 (Thread 0x7f1b047e8700 (LWP 105201)): >>>>>> #0 0x00007f1b48ae69b3 in select () from >>>>>> /lib64/libc.so.6 >>>>>> No symbol table info available. >>>>>> #1 0x00007f1b4b2710e9 in DS_Sleep () from >>>>>> /usr/lib64/dirsrv/libslapd.so.0 >>>>>> No symbol table info available. >>>>>> #2 0x00007f1b4b712325 in time_thread () >>>>>> No symbol table info available. >>>>>> #3 0x00007f1b494207bb in _pt_root () from >>>>>> /lib64/libnspr4.so >>>>>> No symbol table info available. >>>>>> #4 0x00007f1b48dc1dc5 in start_thread () from >>>>>> /lib64/libpthread.so.0 >>>>>> No symbol table info available. >>>>>> #5 0x00007f1b48aef28d in clone () from >>>>>> /lib64/libc.so.6 >>>>>> No symbol table info available. >>>>>> >>>>>> Thread 40 (Thread 0x7f1b04fe9700 (LWP 105200)): >>>>>> #0 0x00007f1b48dc56d5 in >>>>>> pthread_cond_wait@@GLIBC_2.3.2 >>>>>> () >>>>>> from /lib64/libpthread.so.0 >>>>>> No symbol table info available. >>>>>> #1 0x00007f1b433212f3 in >>>>>> __db_hybrid_mutex_suspend () from >>>>>> /lib64/libdb-5.3.so >>>>>> No symbol table info available. >>>>>> #2 0x00007f1b43320640 in __db_tas_mutex_lock >>>>>> () from /lib64/libdb-5.3.so >>>>>> No symbol table info available. >>>>>> #3 0x00007f1b4343ae1b in __memp_fget () from >>>>>> /lib64/libdb-5.3.so >>>>>> No symbol table info available. >>>>>> #4 0x00007f1b4333e211 in __bam_search () from >>>>>> /lib64/libdb-5.3.so >>>>>> No symbol table info available. >>>>>> #5 0x00007f1b43329256 in __bamc_search () >>>>>> from /lib64/libdb-5.3.so >>>>>> No symbol table info available. >>>>>> #6 0x00007f1b4332d6c4 in __bamc_put () from >>>>>> /lib64/libdb-5.3.so >>>>>> No symbol table info available. >>>>>> #7 0x00007f1b433e46d5 in __dbc_iput () from >>>>>> /lib64/libdb-5.3.so >>>>>> No symbol table info available. >>>>>> #8 0x00007f1b433dfa0e in __db_put () from >>>>>> /lib64/libdb-5.3.so >>>>>> No symbol table info available. >>>>>> #9 0x00007f1b433f4fa4 in __db_put_pp () from >>>>>> /lib64/libdb-5.3.so >>>>>> No symbol table info available. >>>>>> #10 0x00007f1b3e8cf4cf in idl_new_insert_key >>>>>> () from /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>>>> No symbol table info available. >>>>>> #11 0x00007f1b3e8de5b5 in >>>>>> addordel_values_sv.isra.3 () from >>>>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>>>> No symbol table info available. >>>>>> #12 0x00007f1b3e8df25a in >>>>>> index_addordel_values_ext_sv () from >>>>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>>>> No symbol table info available. >>>>>> #13 0x00007f1b3e8df694 in >>>>>> index_addordel_values_sv () from >>>>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>>>> No symbol table info available. >>>>>> #14 0x00007f1b3e8df75a in >>>>>> index_addordel_entry () from >>>>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>>>> No symbol table info available. >>>>>> #15 0x00007f1b3e8e25fa in ldbm_back_add () >>>>>> from /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>>>> No symbol table info available. >>>>>> #16 0x00007f1b4b1e4cca in op_shared_add () >>>>>> from /usr/lib64/dirsrv/libslapd.so.0 >>>>>> No symbol table info available. >>>>>> #17 0x00007f1b4b1e6020 in do_add () from >>>>>> /usr/lib64/dirsrv/libslapd.so.0 >>>>>> No symbol table info available. >>>>>> #18 0x00007f1b4b711303 in >>>>>> connection_threadmain () >>>>>> No symbol table info available. >>>>>> #19 0x00007f1b494207bb in _pt_root () from >>>>>> /lib64/libnspr4.so >>>>>> No symbol table info available. >>>>>> #20 0x00007f1b48dc1dc5 in start_thread () >>>>>> from /lib64/libpthread.so.0 >>>>>> No symbol table info available. >>>>>> #21 0x00007f1b48aef28d in clone () from >>>>>> /lib64/libc.so.6 >>>>>> No symbol table info available. >>>>>> >>>>>> Thread 39 (Thread 0x7f1b057ea700 (LWP 105199)): >>>>>> #0 0x00007f1b48dc56d5 in >>>>>> pthread_cond_wait@@GLIBC_2.3.2 >>>>>> () >>>>>> from /lib64/libpthread.so.0 >>>>>> No symbol table info available. >>>>>> #1 0x00007f1b4941b050 in PR_WaitCondVar () >>>>>> from /lib64/libnspr4.so >>>>>> No symbol table info available. >>>>>> #2 0x00007f1b4b70efee in >>>>>> connection_wait_for_new_work () >>>>>> No symbol table info available. >>>>>> #3 0x00007f1b4b710719 in connection_threadmain () >>>>>> No symbol table info available. >>>>>> #4 0x00007f1b494207bb in _pt_root () from >>>>>> /lib64/libnspr4.so >>>>>> No symbol table info available. >>>>>> #5 0x00007f1b48dc1dc5 in start_thread () from >>>>>> /lib64/libpthread.so.0 >>>>>> No symbol table info available. >>>>>> #6 0x00007f1b48aef28d in clone () from >>>>>> /lib64/libc.so.6 >>>>>> No symbol table info available. >>>>>> >>>>>> Thread 38 (Thread 0x7f1b05feb700 (LWP 105198)): >>>>>> #0 0x00007f1b48dc56d5 in >>>>>> pthread_cond_wait@@GLIBC_2.3.2 >>>>>> () >>>>>> from /lib64/libpthread.so.0 >>>>>> No symbol table info available. >>>>>> #1 0x00007f1b4941b050 in PR_WaitCondVar () >>>>>> from /lib64/libnspr4.so >>>>>> No symbol table info available. >>>>>> #2 0x00007f1b4b70efee in >>>>>> connection_wait_for_new_work () >>>>>> No symbol table info available. >>>>>> #3 0x00007f1b4b710719 in connection_threadmain () >>>>>> No symbol table info available. >>>>>> #4 0x00007f1b494207bb in _pt_root () from >>>>>> /lib64/libnspr4.so >>>>>> No symbol table info available. >>>>>> #5 0x00007f1b48dc1dc5 in start_thread () from >>>>>> /lib64/libpthread.so.0 >>>>>> No symbol table info available. >>>>>> #6 0x00007f1b48aef28d in clone () from >>>>>> /lib64/libc.so.6 >>>>>> No symbol table info available. >>>>>> >>>>>> Thread 37 (Thread 0x7f1b067ec700 (LWP 105197)): >>>>>> #0 0x00007f1b48dc56d5 in >>>>>> pthread_cond_wait@@GLIBC_2.3.2 >>>>>> () >>>>>> from /lib64/libpthread.so.0 >>>>>> No symbol table info available. >>>>>> #1 0x00007f1b4941b050 in PR_WaitCondVar () >>>>>> from /lib64/libnspr4.so >>>>>> No symbol table info available. >>>>>> #2 0x00007f1b4b70efee in >>>>>> connection_wait_for_new_work () >>>>>> No symbol table info available. >>>>>> #3 0x00007f1b4b710719 in connection_threadmain () >>>>>> No symbol table info available. >>>>>> #4 0x00007f1b494207bb in _pt_root () from >>>>>> /lib64/libnspr4.so >>>>>> No symbol table info available. >>>>>> #5 0x00007f1b48dc1dc5 in start_thread () from >>>>>> /lib64/libpthread.so.0 >>>>>> No symbol table info available. >>>>>> #6 0x00007f1b48aef28d in clone () from >>>>>> /lib64/libc.so.6 >>>>>> No symbol table info available. >>>>>> >>>>>> Thread 36 (Thread 0x7f1b06fed700 (LWP 105196)): >>>>>> #0 0x00007f1b48dc56d5 in >>>>>> pthread_cond_wait@@GLIBC_2.3.2 >>>>>> () >>>>>> from /lib64/libpthread.so.0 >>>>>> No symbol table info available. >>>>>> #1 0x00007f1b4941b050 in PR_WaitCondVar () >>>>>> from /lib64/libnspr4.so >>>>>> No symbol table info available. >>>>>> #2 0x00007f1b4b70efee in >>>>>> connection_wait_for_new_work () >>>>>> No symbol table info available. >>>>>> #3 0x00007f1b4b710719 in connection_threadmain () >>>>>> No symbol table info available. >>>>>> #4 0x00007f1b494207bb in _pt_root () from >>>>>> /lib64/libnspr4.so >>>>>> No symbol table info available. >>>>>> #5 0x00007f1b48dc1dc5 in start_thread () from >>>>>> /lib64/libpthread.so.0 >>>>>> No symbol table info available. >>>>>> #6 0x00007f1b48aef28d in clone () from >>>>>> /lib64/libc.so.6 >>>>>> No symbol table info available. >>>>>> >>>>>> Thread 35 (Thread 0x7f1b077ee700 (LWP 105195)): >>>>>> #0 0x00007f1b48dc56d5 in >>>>>> pthread_cond_wait@@GLIBC_2.3.2 >>>>>> () >>>>>> from /lib64/libpthread.so.0 >>>>>> No symbol table info available. >>>>>> #1 0x00007f1b4941b2c3 in PR_EnterMonitor () >>>>>> from /lib64/libnspr4.so >>>>>> No symbol table info available. >>>>>> #2 0x00007f1b3e8c23e6 in dblayer_txn_begin () >>>>>> from /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>>>> No symbol table info available. >>>>>> #3 0x00007f1b3e8fe007 in ldbm_back_modify () >>>>>> from /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>>>> No symbol table info available. >>>>>> #4 0x00007f1b4b22b541 in op_shared_modify () >>>>>> from /usr/lib64/dirsrv/libslapd.so.0 >>>>>> No symbol table info available. >>>>>> #5 0x00007f1b4b22c004 in modify_internal_pb >>>>>> () from /usr/lib64/dirsrv/libslapd.so.0 >>>>>> No symbol table info available. >>>>>> #6 0x00007f1b3eb48b60 in ipapwd_apply_mods () >>>>>> from >>>>>> /usr/lib64/dirsrv/plugins/libipa_pwd_extop.so >>>>>> No symbol table info available. >>>>>> #7 0x00007f1b3eb4c8e0 in store_new_keys () >>>>>> from >>>>>> /usr/lib64/dirsrv/plugins/libipa_pwd_extop.so >>>>>> No symbol table info available. >>>>>> #8 0x00007f1b3eb4ead0 in ipapwd_extop () from >>>>>> /usr/lib64/dirsrv/plugins/libipa_pwd_extop.so >>>>>> No symbol table info available. >>>>>> #9 0x00007f1b4b23bba2 in >>>>>> plugin_call_exop_plugins () from >>>>>> /usr/lib64/dirsrv/libslapd.so.0 >>>>>> No symbol table info available. >>>>>> #10 0x00007f1b4b717efe in do_extended () >>>>>> No symbol table info available. >>>>>> #11 0x00007f1b4b71141a in >>>>>> connection_threadmain () >>>>>> No symbol table info available. >>>>>> #12 0x00007f1b494207bb in _pt_root () from >>>>>> /lib64/libnspr4.so >>>>>> No symbol table info available. >>>>>> #13 0x00007f1b48dc1dc5 in start_thread () >>>>>> from /lib64/libpthread.so.0 >>>>>> No symbol table info available. >>>>>> #14 0x00007f1b48aef28d in clone () from >>>>>> /lib64/libc.so.6 >>>>>> No symbol table info available. >>>>>> >>>>>> Thread 34 (Thread 0x7f1b07fef700 (LWP 105194)): >>>>>> #0 0x00007f1b48dc56d5 in >>>>>> pthread_cond_wait@@GLIBC_2.3.2 >>>>>> () >>>>>> from /lib64/libpthread.so.0 >>>>>> No symbol table info available. >>>>>> #1 0x00007f1b4941b050 in PR_WaitCondVar () >>>>>> from /lib64/libnspr4.so >>>>>> No symbol table info available. >>>>>> #2 0x00007f1b4b70efee in >>>>>> connection_wait_for_new_work () >>>>>> No symbol table info available. >>>>>> #3 0x00007f1b4b710719 in connection_threadmain () >>>>>> No symbol table info available. >>>>>> #4 0x00007f1b494207bb in _pt_root () from >>>>>> /lib64/libnspr4.so >>>>>> No symbol table info available. >>>>>> #5 0x00007f1b48dc1dc5 in start_thread () from >>>>>> /lib64/libpthread.so.0 >>>>>> No symbol table info available. >>>>>> #6 0x00007f1b48aef28d in clone () from >>>>>> /lib64/libc.so.6 >>>>>> No symbol table info available. >>>>>> >>>>>> Thread 33 (Thread 0x7f1b087f0700 (LWP 105193)): >>>>>> #0 0x00007f1b48dc56d5 in >>>>>> pthread_cond_wait@@GLIBC_2.3.2 >>>>>> () >>>>>> from /lib64/libpthread.so.0 >>>>>> No symbol table info available. >>>>>> #1 0x00007f1b4941b050 in PR_WaitCondVar () >>>>>> from /lib64/libnspr4.so >>>>>> No symbol table info available. >>>>>> #2 0x00007f1b4b70efee in >>>>>> connection_wait_for_new_work () >>>>>> No symbol table info available. >>>>>> #3 0x00007f1b4b710719 in connection_threadmain () >>>>>> No symbol table info available. >>>>>> #4 0x00007f1b494207bb in _pt_root () from >>>>>> /lib64/libnspr4.so >>>>>> No symbol table info available. >>>>>> #5 0x00007f1b48dc1dc5 in start_thread () from >>>>>> /lib64/libpthread.so.0 >>>>>> No symbol table info available. >>>>>> #6 0x00007f1b48aef28d in clone () from >>>>>> /lib64/libc.so.6 >>>>>> No symbol table info available. >>>>>> >>>>>> Thread 32 (Thread 0x7f1b08ff1700 (LWP 105192)): >>>>>> #0 0x00007f1b48dc56d5 in >>>>>> pthread_cond_wait@@GLIBC_2.3.2 >>>>>> () >>>>>> from /lib64/libpthread.so.0 >>>>>> No symbol table info available. >>>>>> #1 0x00007f1b4941b050 in PR_WaitCondVar () >>>>>> from /lib64/libnspr4.so >>>>>> No symbol table info available. >>>>>> #2 0x00007f1b4b70efee in >>>>>> connection_wait_for_new_work () >>>>>> No symbol table info available. >>>>>> #3 0x00007f1b4b710719 in connection_threadmain () >>>>>> No symbol table info available. >>>>>> #4 0x00007f1b494207bb in _pt_root () from >>>>>> /lib64/libnspr4.so >>>>>> No symbol table info available. >>>>>> #5 0x00007f1b48dc1dc5 in start_thread () from >>>>>> /lib64/libpthread.so.0 >>>>>> No symbol table info available. >>>>>> #6 0x00007f1b48aef28d in clone () from >>>>>> /lib64/libc.so.6 >>>>>> No symbol table info available. >>>>>> >>>>>> Thread 31 (Thread 0x7f1b097f2700 (LWP 105191)): >>>>>> #0 0x00007f1b48dc56d5 in >>>>>> pthread_cond_wait@@GLIBC_2.3.2 >>>>>> () >>>>>> from /lib64/libpthread.so.0 >>>>>> No symbol table info available. >>>>>> #1 0x00007f1b4941b050 in PR_WaitCondVar () >>>>>> from /lib64/libnspr4.so >>>>>> No symbol table info available. >>>>>> #2 0x00007f1b4b70efee in >>>>>> connection_wait_for_new_work () >>>>>> No symbol table info available. >>>>>> #3 0x00007f1b4b710719 in connection_threadmain () >>>>>> No symbol table info available. >>>>>> #4 0x00007f1b494207bb in _pt_root () from >>>>>> /lib64/libnspr4.so >>>>>> No symbol table info available. >>>>>> #5 0x00007f1b48dc1dc5 in start_thread () from >>>>>> /lib64/libpthread.so.0 >>>>>> No symbol table info available. >>>>>> #6 0x00007f1b48aef28d in clone () from >>>>>> /lib64/libc.so.6 >>>>>> No symbol table info available. >>>>>> >>>>>> Thread 30 (Thread 0x7f1b09ff3700 (LWP 105190)): >>>>>> #0 0x00007f1b48dc56d5 in >>>>>> pthread_cond_wait@@GLIBC_2.3.2 >>>>>> () >>>>>> from /lib64/libpthread.so.0 >>>>>> No symbol table info available. >>>>>> #1 0x00007f1b433212f3 in >>>>>> __db_hybrid_mutex_suspend () from >>>>>> /lib64/libdb-5.3.so >>>>>> No symbol table info available. >>>>>> #2 0x00007f1b43320640 in __db_tas_mutex_lock >>>>>> () from /lib64/libdb-5.3.so >>>>>> No symbol table info available. >>>>>> #3 0x00007f1b433cad3a in __lock_get_internal >>>>>> () from /lib64/libdb-5.3.so >>>>>> No symbol table info available. >>>>>> #4 0x00007f1b433cb820 in __lock_get () from >>>>>> /lib64/libdb-5.3.so >>>>>> No symbol table info available. >>>>>> #5 0x00007f1b433f7142 in __db_lget () from >>>>>> /lib64/libdb-5.3.so >>>>>> No symbol table info available. >>>>>> #6 0x00007f1b4333e605 in __bam_search () from >>>>>> /lib64/libdb-5.3.so >>>>>> No symbol table info available. >>>>>> #7 0x00007f1b43329256 in __bamc_search () >>>>>> from /lib64/libdb-5.3.so >>>>>> No symbol table info available. >>>>>> #8 0x00007f1b4332ad0f in __bamc_get () from >>>>>> /lib64/libdb-5.3.so >>>>>> No symbol table info available. >>>>>> #9 0x00007f1b433e3ca6 in __dbc_iget () from >>>>>> /lib64/libdb-5.3.so >>>>>> No symbol table info available. >>>>>> #10 0x00007f1b433f2b02 in __dbc_get_pp () >>>>>> from /lib64/libdb-5.3.so >>>>>> No symbol table info available. >>>>>> #11 0x00007f1b3e8ce680 in idl_new_fetch () >>>>>> from /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>>>> No symbol table info available. >>>>>> #12 0x00007f1b3e8dcb26 in >>>>>> index_read_ext_allids () from >>>>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>>>> No symbol table info available. >>>>>> #13 0x00007f1b3e8c7284 in keys2idl () from >>>>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>>>> No symbol table info available. >>>>>> #14 0x00007f1b3e8c79e3 in >>>>>> ava_candidates.isra.0 () from >>>>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>>>> No symbol table info available. >>>>>> #15 0x00007f1b3e8c7fd2 in >>>>>> filter_candidates_ext () from >>>>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>>>> No symbol table info available. >>>>>> #16 0x00007f1b3e8c9096 in list_candidates () >>>>>> from /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>>>> No symbol table info available. >>>>>> #17 0x00007f1b3e8c7f40 in >>>>>> filter_candidates_ext () from >>>>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>>>> No symbol table info available. >>>>>> #18 0x00007f1b3e8c9096 in list_candidates () >>>>>> from /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>>>> No symbol table info available. >>>>>> #19 0x00007f1b3e8c7f40 in >>>>>> filter_candidates_ext () from >>>>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>>>> No symbol table info available. >>>>>> #20 0x00007f1b3e8c9096 in list_candidates () >>>>>> from /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>>>> No symbol table info available. >>>>>> #21 0x00007f1b3e8c7f40 in >>>>>> filter_candidates_ext () from >>>>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>>>> No symbol table info available. >>>>>> #22 0x00007f1b3e90380c in subtree_candidates >>>>>> () from /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>>>> No symbol table info available. >>>>>> #23 0x00007f1b3e904e7f in ldbm_back_search () >>>>>> from /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>>>> No symbol table info available. >>>>>> #24 0x00007f1b4b2321d6 in op_shared_search () >>>>>> from /usr/lib64/dirsrv/libslapd.so.0 >>>>>> No symbol table info available. >>>>>> #25 0x00007f1b4b72333e in do_search () >>>>>> No symbol table info available. >>>>>> #26 0x00007f1b4b7113c3 in >>>>>> connection_threadmain () >>>>>> No symbol table info available. >>>>>> #27 0x00007f1b494207bb in _pt_root () from >>>>>> /lib64/libnspr4.so >>>>>> No symbol table info available. >>>>>> #28 0x00007f1b48dc1dc5 in start_thread () >>>>>> from /lib64/libpthread.so.0 >>>>>> No symbol table info available. >>>>>> #29 0x00007f1b48aef28d in clone () from >>>>>> /lib64/libc.so.6 >>>>>> No symbol table info available. >>>>>> >>>>>> Thread 29 (Thread 0x7f1b0a7f4700 (LWP 105189)): >>>>>> #0 0x00007f1b48dc56d5 in >>>>>> pthread_cond_wait@@GLIBC_2.3.2 >>>>>> () >>>>>> from /lib64/libpthread.so.0 >>>>>> No symbol table info available. >>>>>> #1 0x00007f1b4941b050 in PR_WaitCondVar () >>>>>> from /lib64/libnspr4.so >>>>>> No symbol table info available. >>>>>> #2 0x00007f1b4b70efee in >>>>>> connection_wait_for_new_work () >>>>>> No symbol table info available. >>>>>> #3 0x00007f1b4b710719 in connection_threadmain () >>>>>> No symbol table info available. >>>>>> #4 0x00007f1b494207bb in _pt_root () from >>>>>> /lib64/libnspr4.so >>>>>> No symbol table info available. >>>>>> #5 0x00007f1b48dc1dc5 in start_thread () from >>>>>> /lib64/libpthread.so.0 >>>>>> No symbol table info available. >>>>>> #6 0x00007f1b48aef28d in clone () from >>>>>> /lib64/libc.so.6 >>>>>> No symbol table info available. >>>>>> >>>>>> Thread 28 (Thread 0x7f1b0aff5700 (LWP 105188)): >>>>>> #0 0x00007f1b48dc56d5 in >>>>>> pthread_cond_wait@@GLIBC_2.3.2 >>>>>> () >>>>>> from /lib64/libpthread.so.0 >>>>>> No symbol table info available. >>>>>> #1 0x00007f1b4941b050 in PR_WaitCondVar () >>>>>> from /lib64/libnspr4.so >>>>>> No symbol table info available. >>>>>> #2 0x00007f1b4b70efee in >>>>>> connection_wait_for_new_work () >>>>>> No symbol table info available. >>>>>> #3 0x00007f1b4b710719 in connection_threadmain () >>>>>> No symbol table info available. >>>>>> #4 0x00007f1b494207bb in _pt_root () from >>>>>> /lib64/libnspr4.so >>>>>> No symbol table info available. >>>>>> #5 0x00007f1b48dc1dc5 in start_thread () from >>>>>> /lib64/libpthread.so.0 >>>>>> No symbol table info available. >>>>>> #6 0x00007f1b48aef28d in clone () from >>>>>> /lib64/libc.so.6 >>>>>> No symbol table info available. >>>>>> >>>>>> Thread 27 (Thread 0x7f1b0b7f6700 (LWP 105187)): >>>>>> #0 0x00007f1b48dc56d5 in >>>>>> pthread_cond_wait@@GLIBC_2.3.2 >>>>>> () >>>>>> from /lib64/libpthread.so.0 >>>>>> No symbol table info available. >>>>>> #1 0x00007f1b4941b050 in PR_WaitCondVar () >>>>>> from /lib64/libnspr4.so >>>>>> No symbol table info available. >>>>>> #2 0x00007f1b4b70efee in >>>>>> connection_wait_for_new_work () >>>>>> No symbol table info available. >>>>>> #3 0x00007f1b4b710719 in connection_threadmain () >>>>>> No symbol table info available. >>>>>> #4 0x00007f1b494207bb in _pt_root () from >>>>>> /lib64/libnspr4.so >>>>>> No symbol table info available. >>>>>> #5 0x00007f1b48dc1dc5 in start_thread () from >>>>>> /lib64/libpthread.so.0 >>>>>> No symbol table info available. >>>>>> #6 0x00007f1b48aef28d in clone () from >>>>>> /lib64/libc.so.6 >>>>>> No symbol table info available. >>>>>> >>>>>> Thread 26 (Thread 0x7f1b0bff7700 (LWP 105186)): >>>>>> #0 0x00007f1b48dc56d5 in >>>>>> pthread_cond_wait@@GLIBC_2.3.2 >>>>>> () >>>>>> from /lib64/libpthread.so.0 >>>>>> No symbol table info available. >>>>>> #1 0x00007f1b433212f3 in >>>>>> __db_hybrid_mutex_suspend () from >>>>>> /lib64/libdb-5.3.so >>>>>> No symbol table info available. >>>>>> #2 0x00007f1b43320640 in __db_tas_mutex_lock >>>>>> () from /lib64/libdb-5.3.so >>>>>> No symbol table info available. >>>>>> #3 0x00007f1b433cad3a in __lock_get_internal >>>>>> () from /lib64/libdb-5.3.so >>>>>> No symbol table info available. >>>>>> #4 0x00007f1b433cb820 in __lock_get () from >>>>>> /lib64/libdb-5.3.so >>>>>> No symbol table info available. >>>>>> #5 0x00007f1b433f7142 in __db_lget () from >>>>>> /lib64/libdb-5.3.so >>>>>> No symbol table info available. >>>>>> #6 0x00007f1b4333e605 in __bam_search () from >>>>>> /lib64/libdb-5.3.so >>>>>> No symbol table info available. >>>>>> #7 0x00007f1b43329256 in __bamc_search () >>>>>> from /lib64/libdb-5.3.so >>>>>> No symbol table info available. >>>>>> #8 0x00007f1b4332ad0f in __bamc_get () from >>>>>> /lib64/libdb-5.3.so >>>>>> No symbol table info available. >>>>>> #9 0x00007f1b433e3ca6 in __dbc_iget () from >>>>>> /lib64/libdb-5.3.so >>>>>> No symbol table info available. >>>>>> #10 0x00007f1b433f2b02 in __dbc_get_pp () >>>>>> from /lib64/libdb-5.3.so >>>>>> No symbol table info available. >>>>>> #11 0x00007f1b3e8ce680 in idl_new_fetch () >>>>>> from /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>>>> No symbol table info available. >>>>>> #12 0x00007f1b3e8dcb26 in >>>>>> index_read_ext_allids () from >>>>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>>>> No symbol table info available. >>>>>> #13 0x00007f1b3e8c7284 in keys2idl () from >>>>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>>>> No symbol table info available. >>>>>> #14 0x00007f1b3e8c79e3 in >>>>>> ava_candidates.isra.0 () from >>>>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>>>> No symbol table info available. >>>>>> #15 0x00007f1b3e8c7fd2 in >>>>>> filter_candidates_ext () from >>>>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>>>> No symbol table info available. >>>>>> #16 0x00007f1b3e8c9096 in list_candidates () >>>>>> from /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>>>> No symbol table info available. >>>>>> #17 0x00007f1b3e8c7f40 in >>>>>> filter_candidates_ext () from >>>>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>>>> No symbol table info available. >>>>>> #18 0x00007f1b3e90380c in subtree_candidates >>>>>> () from /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>>>> No symbol table info available. >>>>>> #19 0x00007f1b3e904e7f in ldbm_back_search () >>>>>> from /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>>>> No symbol table info available. >>>>>> #20 0x00007f1b4b2321d6 in op_shared_search () >>>>>> from /usr/lib64/dirsrv/libslapd.so.0 >>>>>> No symbol table info available. >>>>>> #21 0x00007f1b4b24294e in >>>>>> search_internal_callback_pb () from >>>>>> /usr/lib64/dirsrv/libslapd.so.0 >>>>>> No symbol table info available. >>>>>> #22 0x00007f1b4b242be8 in search_internal_pb >>>>>> () from /usr/lib64/dirsrv/libslapd.so.0 >>>>>> No symbol table info available. >>>>>> #23 0x00007f1b4b72149b in ids_sasl_canon_user () >>>>>> No symbol table info available. >>>>>> #24 0x00007f1b491e193b in _sasl_canon_user () >>>>>> from /lib64/libsasl2.so.3 >>>>>> No symbol table info available. >>>>>> #25 0x00007f1b491e1c4c in >>>>>> _sasl_canon_user_lookup () from >>>>>> /lib64/libsasl2.so.3 >>>>>> No symbol table info available. >>>>>> #26 0x00007f1b430e9fb9 in >>>>>> gssapi_server_mech_step () from >>>>>> /usr/lib64/sasl2/libgssapiv2.so >>>>>> No symbol table info available. >>>>>> #27 0x00007f1b491edb9b in sasl_server_step () >>>>>> from /lib64/libsasl2.so.3 >>>>>> No symbol table info available. >>>>>> #28 0x00007f1b4b72283b in ids_sasl_check_bind () >>>>>> No symbol table info available. >>>>>> #29 0x00007f1b4b70a61a in do_bind () >>>>>> No symbol table info available. >>>>>> #30 0x00007f1b4b7113fd in >>>>>> connection_threadmain () >>>>>> No symbol table info available. >>>>>> #31 0x00007f1b494207bb in _pt_root () from >>>>>> /lib64/libnspr4.so >>>>>> No symbol table info available. >>>>>> #32 0x00007f1b48dc1dc5 in start_thread () >>>>>> from /lib64/libpthread.so.0 >>>>>> No symbol table info available. >>>>>> #33 0x00007f1b48aef28d in clone () from >>>>>> /lib64/libc.so.6 >>>>>> No symbol table info available. >>>>>> >>>>>> Thread 25 (Thread 0x7f1b0c7f8700 (LWP 105185)): >>>>>> #0 0x00007f1b48dc56d5 in >>>>>> pthread_cond_wait@@GLIBC_2.3.2 >>>>>> () >>>>>> from /lib64/libpthread.so.0 >>>>>> No symbol table info available. >>>>>> #1 0x00007f1b4941b050 in PR_WaitCondVar () >>>>>> from /lib64/libnspr4.so >>>>>> No symbol table info available. >>>>>> #2 0x00007f1b4b70efee in >>>>>> connection_wait_for_new_work () >>>>>> No symbol table info available. >>>>>> #3 0x00007f1b4b710719 in connection_threadmain () >>>>>> No symbol table info available. >>>>>> #4 0x00007f1b494207bb in _pt_root () from >>>>>> /lib64/libnspr4.so >>>>>> No symbol table info available. >>>>>> #5 0x00007f1b48dc1dc5 in start_thread () from >>>>>> /lib64/libpthread.so.0 >>>>>> No symbol table info available. >>>>>> #6 0x00007f1b48aef28d in clone () from >>>>>> /lib64/libc.so.6 >>>>>> No symbol table info available. >>>>>> >>>>>> Thread 24 (Thread 0x7f1b0cff9700 (LWP 105184)): >>>>>> #0 0x00007f1b48dc56d5 in >>>>>> pthread_cond_wait@@GLIBC_2.3.2 >>>>>> () >>>>>> from /lib64/libpthread.so.0 >>>>>> No symbol table info available. >>>>>> #1 0x00007f1b4941b050 in PR_WaitCondVar () >>>>>> from /lib64/libnspr4.so >>>>>> No symbol table info available. >>>>>> #2 0x00007f1b4b70efee in >>>>>> connection_wait_for_new_work () >>>>>> No symbol table info available. >>>>>> #3 0x00007f1b4b710719 in connection_threadmain () >>>>>> No symbol table info available. >>>>>> #4 0x00007f1b494207bb in _pt_root () from >>>>>> /lib64/libnspr4.so >>>>>> No symbol table info available. >>>>>> #5 0x00007f1b48dc1dc5 in start_thread () from >>>>>> /lib64/libpthread.so.0 >>>>>> No symbol table info available. >>>>>> #6 0x00007f1b48aef28d in clone () from >>>>>> /lib64/libc.so.6 >>>>>> No symbol table info available. >>>>>> >>>>>> Thread 23 (Thread 0x7f1b0d7fa700 (LWP 105183)): >>>>>> #0 0x00007f1b48dc56d5 in >>>>>> pthread_cond_wait@@GLIBC_2.3.2 >>>>>> () >>>>>> from /lib64/libpthread.so.0 >>>>>> No symbol table info available. >>>>>> #1 0x00007f1b4941b050 in PR_WaitCondVar () >>>>>> from /lib64/libnspr4.so >>>>>> No symbol table info available. >>>>>> #2 0x00007f1b4b70efee in >>>>>> connection_wait_for_new_work () >>>>>> No symbol table info available. >>>>>> #3 0x00007f1b4b710719 in connection_threadmain () >>>>>> No symbol table info available. >>>>>> #4 0x00007f1b494207bb in _pt_root () from >>>>>> /lib64/libnspr4.so >>>>>> No symbol table info available. >>>>>> #5 0x00007f1b48dc1dc5 in start_thread () from >>>>>> /lib64/libpthread.so.0 >>>>>> No symbol table info available. >>>>>> #6 0x00007f1b48aef28d in clone () from >>>>>> /lib64/libc.so.6 >>>>>> No symbol table info available. >>>>>> >>>>>> Thread 22 (Thread 0x7f1b0dffb700 (LWP 105182)): >>>>>> #0 0x00007f1b48dc56d5 in >>>>>> pthread_cond_wait@@GLIBC_2.3.2 >>>>>> () >>>>>> from /lib64/libpthread.so.0 >>>>>> No symbol table info available. >>>>>> #1 0x00007f1b4941b050 in PR_WaitCondVar () >>>>>> from /lib64/libnspr4.so >>>>>> No symbol table info available. >>>>>> #2 0x00007f1b4b70efee in >>>>>> connection_wait_for_new_work () >>>>>> No symbol table info available. >>>>>> #3 0x00007f1b4b710719 in connection_threadmain () >>>>>> No symbol table info available. >>>>>> #4 0x00007f1b494207bb in _pt_root () from >>>>>> /lib64/libnspr4.so >>>>>> No symbol table info available. >>>>>> #5 0x00007f1b48dc1dc5 in start_thread () from >>>>>> /lib64/libpthread.so.0 >>>>>> No symbol table info available. >>>>>> #6 0x00007f1b48aef28d in clone () from >>>>>> /lib64/libc.so.6 >>>>>> No symbol table info available. >>>>>> >>>>>> Thread 21 (Thread 0x7f1b0e7fc700 (LWP 105181)): >>>>>> #0 0x00007f1b48dc56d5 in >>>>>> pthread_cond_wait@@GLIBC_2.3.2 >>>>>> () >>>>>> from /lib64/libpthread.so.0 >>>>>> No symbol table info available. >>>>>> #1 0x00007f1b4941b050 in PR_WaitCondVar () >>>>>> from /lib64/libnspr4.so >>>>>> No symbol table info available. >>>>>> #2 0x00007f1b4b70efee in >>>>>> connection_wait_for_new_work () >>>>>> No symbol table info available. >>>>>> #3 0x00007f1b4b710719 in connection_threadmain () >>>>>> No symbol table info available. >>>>>> #4 0x00007f1b494207bb in _pt_root () from >>>>>> /lib64/libnspr4.so >>>>>> No symbol table info available. >>>>>> #5 0x00007f1b48dc1dc5 in start_thread () from >>>>>> /lib64/libpthread.so.0 >>>>>> No symbol table info available. >>>>>> #6 0x00007f1b48aef28d in clone () from >>>>>> /lib64/libc.so.6 >>>>>> No symbol table info available. >>>>>> >>>>>> Thread 20 (Thread 0x7f1b0effd700 (LWP 105180)): >>>>>> #0 0x00007f1b48dc56d5 in >>>>>> pthread_cond_wait@@GLIBC_2.3.2 >>>>>> () >>>>>> from /lib64/libpthread.so.0 >>>>>> No symbol table info available. >>>>>> #1 0x00007f1b433212f3 in >>>>>> __db_hybrid_mutex_suspend () from >>>>>> /lib64/libdb-5.3.so >>>>>> No symbol table info available. >>>>>> #2 0x00007f1b43320b37 in >>>>>> __db_tas_mutex_readlock () from >>>>>> /lib64/libdb-5.3.so >>>>>> No symbol table info available. >>>>>> #3 0x00007f1b4343ad35 in __memp_fget () from >>>>>> /lib64/libdb-5.3.so >>>>>> No symbol table info available. >>>>>> #4 0x00007f1b433e4333 in __dbc_iget () from >>>>>> /lib64/libdb-5.3.so >>>>>> No symbol table info available. >>>>>> #5 0x00007f1b433f2b02 in __dbc_get_pp () from >>>>>> /lib64/libdb-5.3.so >>>>>> No symbol table info available. >>>>>> #6 0x00007f1b3e8ce822 in idl_new_fetch () >>>>>> from /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>>>> No symbol table info available. >>>>>> #7 0x00007f1b3e8dcb26 in >>>>>> index_read_ext_allids () from >>>>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>>>> No symbol table info available. >>>>>> #8 0x00007f1b3e8c7284 in keys2idl () from >>>>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>>>> No symbol table info available. >>>>>> #9 0x00007f1b3e8c79e3 in >>>>>> ava_candidates.isra.0 () from >>>>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>>>> No symbol table info available. >>>>>> #10 0x00007f1b3e8c7fd2 in >>>>>> filter_candidates_ext () from >>>>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>>>> No symbol table info available. >>>>>> #11 0x00007f1b3e8c9096 in list_candidates () >>>>>> from /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>>>> No symbol table info available. >>>>>> #12 0x00007f1b3e8c7f40 in >>>>>> filter_candidates_ext () from >>>>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>>>> No symbol table info available. >>>>>> #13 0x00007f1b3e8c9096 in list_candidates () >>>>>> from /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>>>> No symbol table info available. >>>>>> #14 0x00007f1b3e8c7f40 in >>>>>> filter_candidates_ext () from >>>>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>>>> No symbol table info available. >>>>>> #15 0x00007f1b3e8c9096 in list_candidates () >>>>>> from /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>>>> No symbol table info available. >>>>>> #16 0x00007f1b3e8c7f40 in >>>>>> filter_candidates_ext () from >>>>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>>>> No symbol table info available. >>>>>> #17 0x00007f1b3e90380c in subtree_candidates >>>>>> () from /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>>>> No symbol table info available. >>>>>> #18 0x00007f1b3e904e7f in ldbm_back_search () >>>>>> from /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>>>> No symbol table info available. >>>>>> #19 0x00007f1b4b2321d6 in op_shared_search () >>>>>> from /usr/lib64/dirsrv/libslapd.so.0 >>>>>> No symbol table info available. >>>>>> #20 0x00007f1b4b72333e in do_search () >>>>>> No symbol table info available. >>>>>> #21 0x00007f1b4b7113c3 in >>>>>> connection_threadmain () >>>>>> No symbol table info available. >>>>>> #22 0x00007f1b494207bb in _pt_root () from >>>>>> /lib64/libnspr4.so >>>>>> No symbol table info available. >>>>>> #23 0x00007f1b48dc1dc5 in start_thread () >>>>>> from /lib64/libpthread.so.0 >>>>>> No symbol table info available. >>>>>> #24 0x00007f1b48aef28d in clone () from >>>>>> /lib64/libc.so.6 >>>>>> No symbol table info available. >>>>>> >>>>>> Thread 19 (Thread 0x7f1b0f7fe700 (LWP 105179)): >>>>>> #0 0x00007f1b48dc56d5 in >>>>>> pthread_cond_wait@@GLIBC_2.3.2 >>>>>> () >>>>>> from /lib64/libpthread.so.0 >>>>>> No symbol table info available. >>>>>> #1 0x00007f1b433212f3 in >>>>>> __db_hybrid_mutex_suspend () from >>>>>> /lib64/libdb-5.3.so >>>>>> No symbol table info available. >>>>>> #2 0x00007f1b43320640 in __db_tas_mutex_lock >>>>>> () from /lib64/libdb-5.3.so >>>>>> No symbol table info available. >>>>>> #3 0x00007f1b433cad3a in __lock_get_internal >>>>>> () from /lib64/libdb-5.3.so >>>>>> No symbol table info available. >>>>>> #4 0x00007f1b433cb820 in __lock_get () from >>>>>> /lib64/libdb-5.3.so >>>>>> No symbol table info available. >>>>>> #5 0x00007f1b433f7142 in __db_lget () from >>>>>> /lib64/libdb-5.3.so >>>>>> No symbol table info available. >>>>>> #6 0x00007f1b4333e605 in __bam_search () from >>>>>> /lib64/libdb-5.3.so >>>>>> No symbol table info available. >>>>>> #7 0x00007f1b43329256 in __bamc_search () >>>>>> from /lib64/libdb-5.3.so >>>>>> No symbol table info available. >>>>>> #8 0x00007f1b4332ad0f in __bamc_get () from >>>>>> /lib64/libdb-5.3.so >>>>>> No symbol table info available. >>>>>> #9 0x00007f1b433e3ca6 in __dbc_iget () from >>>>>> /lib64/libdb-5.3.so >>>>>> No symbol table info available. >>>>>> #10 0x00007f1b433f2b02 in __dbc_get_pp () >>>>>> from /lib64/libdb-5.3.so >>>>>> No symbol table info available. >>>>>> #11 0x00007f1b3e8ce680 in idl_new_fetch () >>>>>> from /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>>>> No symbol table info available. >>>>>> #12 0x00007f1b3e8dcb26 in >>>>>> index_read_ext_allids () from >>>>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>>>> No symbol table info available. >>>>>> #13 0x00007f1b3e8c7284 in keys2idl () from >>>>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>>>> No symbol table info available. >>>>>> #14 0x00007f1b3e8c79e3 in >>>>>> ava_candidates.isra.0 () from >>>>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>>>> No symbol table info available. >>>>>> #15 0x00007f1b3e8c7fd2 in >>>>>> filter_candidates_ext () from >>>>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>>>> No symbol table info available. >>>>>> #16 0x00007f1b3e8c9096 in list_candidates () >>>>>> from /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>>>> No symbol table info available. >>>>>> #17 0x00007f1b3e8c7f40 in >>>>>> filter_candidates_ext () from >>>>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>>>> No symbol table info available. >>>>>> #18 0x00007f1b3e8c9096 in list_candidates () >>>>>> from /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>>>> No symbol table info available. >>>>>> #19 0x00007f1b3e8c7f40 in >>>>>> filter_candidates_ext () from >>>>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>>>> No symbol table info available. >>>>>> #20 0x00007f1b3e8c9096 in list_candidates () >>>>>> from /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>>>> No symbol table info available. >>>>>> #21 0x00007f1b3e8c7f40 in >>>>>> filter_candidates_ext () from >>>>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>>>> No symbol table info available. >>>>>> #22 0x00007f1b3e90380c in subtree_candidates >>>>>> () from /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>>>> No symbol table info available. >>>>>> #23 0x00007f1b3e904e7f in ldbm_back_search () >>>>>> from /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>>>> No symbol table info available. >>>>>> #24 0x00007f1b4b2321d6 in op_shared_search () >>>>>> from /usr/lib64/dirsrv/libslapd.so.0 >>>>>> No symbol table info available. >>>>>> #25 0x00007f1b4b72333e in do_search () >>>>>> No symbol table info available. >>>>>> #26 0x00007f1b4b7113c3 in >>>>>> connection_threadmain () >>>>>> No symbol table info available. >>>>>> #27 0x00007f1b494207bb in _pt_root () from >>>>>> /lib64/libnspr4.so >>>>>> No symbol table info available. >>>>>> #28 0x00007f1b48dc1dc5 in start_thread () >>>>>> from /lib64/libpthread.so.0 >>>>>> No symbol table info available. >>>>>> #29 0x00007f1b48aef28d in clone () from >>>>>> /lib64/libc.so.6 >>>>>> No symbol table info available. >>>>>> >>>>>> Thread 18 (Thread 0x7f1b0ffff700 (LWP 105178)): >>>>>> #0 0x00007f1b48dc56d5 in >>>>>> pthread_cond_wait@@GLIBC_2.3.2 >>>>>> () >>>>>> from /lib64/libpthread.so.0 >>>>>> No symbol table info available. >>>>>> #1 0x00007f1b4941b050 in PR_WaitCondVar () >>>>>> from /lib64/libnspr4.so >>>>>> No symbol table info available. >>>>>> #2 0x00007f1b4b70efee in >>>>>> connection_wait_for_new_work () >>>>>> No symbol table info available. >>>>>> #3 0x00007f1b4b710719 in connection_threadmain () >>>>>> No symbol table info available. >>>>>> #4 0x00007f1b494207bb in _pt_root () from >>>>>> /lib64/libnspr4.so >>>>>> No symbol table info available. >>>>>> #5 0x00007f1b48dc1dc5 in start_thread () from >>>>>> /lib64/libpthread.so.0 >>>>>> No symbol table info available. >>>>>> #6 0x00007f1b48aef28d in clone () from >>>>>> /lib64/libc.so.6 >>>>>> No symbol table info available. >>>>>> >>>>>> Thread 17 (Thread 0x7f1b14d28700 (LWP 105177)): >>>>>> #0 0x00007f1b48dc56d5 in >>>>>> pthread_cond_wait@@GLIBC_2.3.2 >>>>>> () >>>>>> from /lib64/libpthread.so.0 >>>>>> No symbol table info available. >>>>>> #1 0x00007f1b4941b050 in PR_WaitCondVar () >>>>>> from /lib64/libnspr4.so >>>>>> No symbol table info available. >>>>>> #2 0x00007f1b4b70efee in >>>>>> connection_wait_for_new_work () >>>>>> No symbol table info available. >>>>>> #3 0x00007f1b4b710719 in connection_threadmain () >>>>>> No symbol table info available. >>>>>> #4 0x00007f1b494207bb in _pt_root () from >>>>>> /lib64/libnspr4.so >>>>>> No symbol table info available. >>>>>> #5 0x00007f1b48dc1dc5 in start_thread () from >>>>>> /lib64/libpthread.so.0 >>>>>> No symbol table info available. >>>>>> #6 0x00007f1b48aef28d in clone () from >>>>>> /lib64/libc.so.6 >>>>>> No symbol table info available. >>>>>> >>>>>> Thread 16 (Thread 0x7f1b15529700 (LWP 105176)): >>>>>> #0 0x00007f1b48dc56d5 in >>>>>> pthread_cond_wait@@GLIBC_2.3.2 >>>>>> () >>>>>> from /lib64/libpthread.so.0 >>>>>> No symbol table info available. >>>>>> #1 0x00007f1b4941b050 in PR_WaitCondVar () >>>>>> from /lib64/libnspr4.so >>>>>> No symbol table info available. >>>>>> #2 0x00007f1b4b70efee in >>>>>> connection_wait_for_new_work () >>>>>> No symbol table info available. >>>>>> #3 0x00007f1b4b710719 in connection_threadmain () >>>>>> No symbol table info available. >>>>>> #4 0x00007f1b494207bb in _pt_root () from >>>>>> /lib64/libnspr4.so >>>>>> No symbol table info available. >>>>>> #5 0x00007f1b48dc1dc5 in start_thread () from >>>>>> /lib64/libpthread.so.0 >>>>>> No symbol table info available. >>>>>> #6 0x00007f1b48aef28d in clone () from >>>>>> /lib64/libc.so.6 >>>>>> No symbol table info available. >>>>>> >>>>>> Thread 15 (Thread 0x7f1b15d2a700 (LWP 105175)): >>>>>> #0 0x00007f1b48dc56d5 in >>>>>> pthread_cond_wait@@GLIBC_2.3.2 >>>>>> () >>>>>> from /lib64/libpthread.so.0 >>>>>> No symbol table info available. >>>>>> #1 0x00007f1b4941b050 in PR_WaitCondVar () >>>>>> from /lib64/libnspr4.so >>>>>> No symbol table info available. >>>>>> #2 0x00007f1b4b70efee in >>>>>> connection_wait_for_new_work () >>>>>> No symbol table info available. >>>>>> #3 0x00007f1b4b710719 in connection_threadmain () >>>>>> No symbol table info available. >>>>>> #4 0x00007f1b494207bb in _pt_root () from >>>>>> /lib64/libnspr4.so >>>>>> No symbol table info available. >>>>>> #5 0x00007f1b48dc1dc5 in start_thread () from >>>>>> /lib64/libpthread.so.0 >>>>>> No symbol table info available. >>>>>> #6 0x00007f1b48aef28d in clone () from >>>>>> /lib64/libc.so.6 >>>>>> No symbol table info available. >>>>>> >>>>>> Thread 14 (Thread 0x7f1b1652b700 (LWP 105174)): >>>>>> #0 0x00007f1b48dc56d5 in >>>>>> pthread_cond_wait@@GLIBC_2.3.2 >>>>>> () >>>>>> from /lib64/libpthread.so.0 >>>>>> No symbol table info available. >>>>>> #1 0x00007f1b4941b050 in PR_WaitCondVar () >>>>>> from /lib64/libnspr4.so >>>>>> No symbol table info available. >>>>>> #2 0x00007f1b4b70efee in >>>>>> connection_wait_for_new_work () >>>>>> No symbol table info available. >>>>>> #3 0x00007f1b4b710719 in connection_threadmain () >>>>>> No symbol table info available. >>>>>> #4 0x00007f1b494207bb in _pt_root () from >>>>>> /lib64/libnspr4.so >>>>>> No symbol table info available. >>>>>> #5 0x00007f1b48dc1dc5 in start_thread () from >>>>>> /lib64/libpthread.so.0 >>>>>> No symbol table info available. >>>>>> #6 0x00007f1b48aef28d in clone () from >>>>>> /lib64/libc.so.6 >>>>>> No symbol table info available. >>>>>> >>>>>> Thread 13 (Thread 0x7f1b16d2c700 (LWP 105173)): >>>>>> #0 0x00007f1b48dc56d5 in >>>>>> pthread_cond_wait@@GLIBC_2.3.2 >>>>>> () >>>>>> from /lib64/libpthread.so.0 >>>>>> No symbol table info available. >>>>>> #1 0x00007f1b433212f3 in >>>>>> __db_hybrid_mutex_suspend () from >>>>>> /lib64/libdb-5.3.so >>>>>> No symbol table info available. >>>>>> #2 0x00007f1b43320640 in __db_tas_mutex_lock >>>>>> () from /lib64/libdb-5.3.so >>>>>> No symbol table info available. >>>>>> #3 0x00007f1b433cad3a in __lock_get_internal >>>>>> () from /lib64/libdb-5.3.so >>>>>> No symbol table info available. >>>>>> #4 0x00007f1b433cb820 in __lock_get () from >>>>>> /lib64/libdb-5.3.so >>>>>> No symbol table info available. >>>>>> #5 0x00007f1b433f7142 in __db_lget () from >>>>>> /lib64/libdb-5.3.so >>>>>> No symbol table info available. >>>>>> #6 0x00007f1b4333e605 in __bam_search () from >>>>>> /lib64/libdb-5.3.so >>>>>> No symbol table info available. >>>>>> #7 0x00007f1b43329256 in __bamc_search () >>>>>> from /lib64/libdb-5.3.so >>>>>> No symbol table info available. >>>>>> #8 0x00007f1b4332ad0f in __bamc_get () from >>>>>> /lib64/libdb-5.3.so >>>>>> No symbol table info available. >>>>>> #9 0x00007f1b433e3ca6 in __dbc_iget () from >>>>>> /lib64/libdb-5.3.so >>>>>> No symbol table info available. >>>>>> #10 0x00007f1b433f2b02 in __dbc_get_pp () >>>>>> from /lib64/libdb-5.3.so >>>>>> No symbol table info available. >>>>>> #11 0x00007f1b3e8ce680 in idl_new_fetch () >>>>>> from /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>>>> No symbol table info available. >>>>>> #12 0x00007f1b3e8dcb26 in >>>>>> index_read_ext_allids () from >>>>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>>>> No symbol table info available. >>>>>> #13 0x00007f1b3e8c7284 in keys2idl () from >>>>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>>>> No symbol table info available. >>>>>> #14 0x00007f1b3e8c79e3 in >>>>>> ava_candidates.isra.0 () from >>>>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>>>> No symbol table info available. >>>>>> #15 0x00007f1b3e8c7fd2 in >>>>>> filter_candidates_ext () from >>>>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>>>> No symbol table info available. >>>>>> #16 0x00007f1b3e8c9096 in list_candidates () >>>>>> from /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>>>> No symbol table info available. >>>>>> #17 0x00007f1b3e8c7f40 in >>>>>> filter_candidates_ext () from >>>>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>>>> No symbol table info available. >>>>>> #18 0x00007f1b3e8c9096 in list_candidates () >>>>>> from /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>>>> No symbol table info available. >>>>>> #19 0x00007f1b3e8c7f40 in >>>>>> filter_candidates_ext () from >>>>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>>>> No symbol table info available. >>>>>> #20 0x00007f1b3e8c9096 in list_candidates () >>>>>> from /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>>>> No symbol table info available. >>>>>> #21 0x00007f1b3e8c7f40 in >>>>>> filter_candidates_ext () from >>>>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>>>> No symbol table info available. >>>>>> #22 0x00007f1b3e90380c in subtree_candidates >>>>>> () from /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>>>> No symbol table info available. >>>>>> #23 0x00007f1b3e904e7f in ldbm_back_search () >>>>>> from /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>>>> No symbol table info available. >>>>>> #24 0x00007f1b4b2321d6 in op_shared_search () >>>>>> from /usr/lib64/dirsrv/libslapd.so.0 >>>>>> No symbol table info available. >>>>>> #25 0x00007f1b4b72333e in do_search () >>>>>> No symbol table info available. >>>>>> #26 0x00007f1b4b7113c3 in >>>>>> connection_threadmain () >>>>>> No symbol table info available. >>>>>> #27 0x00007f1b494207bb in _pt_root () from >>>>>> /lib64/libnspr4.so >>>>>> No symbol table info available. >>>>>> #28 0x00007f1b48dc1dc5 in start_thread () >>>>>> from /lib64/libpthread.so.0 >>>>>> No symbol table info available. >>>>>> #29 0x00007f1b48aef28d in clone () from >>>>>> /lib64/libc.so.6 >>>>>> No symbol table info available. >>>>>> >>>>>> Thread 12 (Thread 0x7f1b1752d700 (LWP 105172)): >>>>>> #0 0x00007f1b48dc56d5 in >>>>>> pthread_cond_wait@@GLIBC_2.3.2 >>>>>> () >>>>>> from /lib64/libpthread.so.0 >>>>>> No symbol table info available. >>>>>> #1 0x00007f1b4941b050 in PR_WaitCondVar () >>>>>> from /lib64/libnspr4.so >>>>>> No symbol table info available. >>>>>> #2 0x00007f1b4b70efee in >>>>>> connection_wait_for_new_work () >>>>>> No symbol table info available. >>>>>> #3 0x00007f1b4b710719 in connection_threadmain () >>>>>> No symbol table info available. >>>>>> #4 0x00007f1b494207bb in _pt_root () from >>>>>> /lib64/libnspr4.so >>>>>> No symbol table info available. >>>>>> #5 0x00007f1b48dc1dc5 in start_thread () from >>>>>> /lib64/libpthread.so.0 >>>>>> No symbol table info available. >>>>>> #6 0x00007f1b48aef28d in clone () from >>>>>> /lib64/libc.so.6 >>>>>> No symbol table info available. >>>>>> >>>>>> Thread 11 (Thread 0x7f1b17d2e700 (LWP 105171)): >>>>>> #0 0x00007f1b48dc56d5 in >>>>>> pthread_cond_wait@@GLIBC_2.3.2 >>>>>> () >>>>>> from /lib64/libpthread.so.0 >>>>>> No symbol table info available. >>>>>> #1 0x00007f1b4941b050 in PR_WaitCondVar () >>>>>> from /lib64/libnspr4.so >>>>>> No symbol table info available. >>>>>> #2 0x00007f1b4b70efee in >>>>>> connection_wait_for_new_work () >>>>>> No symbol table info available. >>>>>> #3 0x00007f1b4b710719 in connection_threadmain () >>>>>> No symbol table info available. >>>>>> #4 0x00007f1b494207bb in _pt_root () from >>>>>> /lib64/libnspr4.so >>>>>> No symbol table info available. >>>>>> #5 0x00007f1b48dc1dc5 in start_thread () from >>>>>> /lib64/libpthread.so.0 >>>>>> No symbol table info available. >>>>>> #6 0x00007f1b48aef28d in clone () from >>>>>> /lib64/libc.so.6 >>>>>> No symbol table info available. >>>>>> >>>>>> Thread 10 (Thread 0x7f1b2509d700 (LWP 105168)): >>>>>> #0 0x00007f1b48dc5a82 in >>>>>> pthread_cond_timedwait@@GLIBC_2.3.2 >>>>>> >>>>>> () from /lib64/libpthread.so.0 >>>>>> No symbol table info available. >>>>>> #1 0x00007f1b4941ab07 in pt_TimedWait () from >>>>>> /lib64/libnspr4.so >>>>>> No symbol table info available. >>>>>> #2 0x00007f1b4941afce in PR_WaitCondVar () >>>>>> from /lib64/libnspr4.so >>>>>> No symbol table info available. >>>>>> #3 0x00007f1b4b2061e8 in eq_loop () from >>>>>> /usr/lib64/dirsrv/libslapd.so.0 >>>>>> No symbol table info available. >>>>>> #4 0x00007f1b494207bb in _pt_root () from >>>>>> /lib64/libnspr4.so >>>>>> No symbol table info available. >>>>>> #5 0x00007f1b48dc1dc5 in start_thread () from >>>>>> /lib64/libpthread.so.0 >>>>>> No symbol table info available. >>>>>> #6 0x00007f1b48aef28d in clone () from >>>>>> /lib64/libc.so.6 >>>>>> No symbol table info available. >>>>>> >>>>>> Thread 9 (Thread 0x7f1b2589e700 (LWP 105167)): >>>>>> #0 0x00007f1b48dc5a82 in >>>>>> pthread_cond_timedwait@@GLIBC_2.3.2 >>>>>> >>>>>> () from /lib64/libpthread.so.0 >>>>>> No symbol table info available. >>>>>> #1 0x00007f1b4941ab07 in pt_TimedWait () from >>>>>> /lib64/libnspr4.so >>>>>> No symbol table info available. >>>>>> #2 0x00007f1b4941afce in PR_WaitCondVar () >>>>>> from /lib64/libnspr4.so >>>>>> No symbol table info available. >>>>>> #3 0x00007f1b4b719d03 in housecleaning () >>>>>> No symbol table info available. >>>>>> #4 0x00007f1b494207bb in _pt_root () from >>>>>> /lib64/libnspr4.so >>>>>> No symbol table info available. >>>>>> #5 0x00007f1b48dc1dc5 in start_thread () from >>>>>> /lib64/libpthread.so.0 >>>>>> No symbol table info available. >>>>>> #6 0x00007f1b48aef28d in clone () from >>>>>> /lib64/libc.so.6 >>>>>> No symbol table info available. >>>>>> >>>>>> Thread 8 (Thread 0x7f1b2609f700 (LWP 105166)): >>>>>> #0 0x00007f1b48dc56d5 in >>>>>> pthread_cond_wait@@GLIBC_2.3.2 >>>>>> () >>>>>> from /lib64/libpthread.so.0 >>>>>> No symbol table info available. >>>>>> #1 0x00007f1b4941b050 in PR_WaitCondVar () >>>>>> from /lib64/libnspr4.so >>>>>> No symbol table info available. >>>>>> #2 0x00007f1b4b260198 in slapi_wait_condvar >>>>>> () from /usr/lib64/dirsrv/libslapd.so.0 >>>>>> No symbol table info available. >>>>>> #3 0x00007f1b3cf5deed in >>>>>> roles_cache_wait_on_change () from >>>>>> /usr/lib64/dirsrv/plugins/libroles-plugin.so >>>>>> No symbol table info available. >>>>>> #4 0x00007f1b494207bb in _pt_root () from >>>>>> /lib64/libnspr4.so >>>>>> No symbol table info available. >>>>>> #5 0x00007f1b48dc1dc5 in start_thread () from >>>>>> /lib64/libpthread.so.0 >>>>>> No symbol table info available. >>>>>> #6 0x00007f1b48aef28d in clone () from >>>>>> /lib64/libc.so.6 >>>>>> No symbol table info available. >>>>>> >>>>>> Thread 7 (Thread 0x7f1b268a0700 (LWP 105165)): >>>>>> #0 0x00007f1b48dc56d5 in >>>>>> pthread_cond_wait@@GLIBC_2.3.2 >>>>>> () >>>>>> from /lib64/libpthread.so.0 >>>>>> No symbol table info available. >>>>>> #1 0x00007f1b4941b050 in PR_WaitCondVar () >>>>>> from /lib64/libnspr4.so >>>>>> No symbol table info available. >>>>>> #2 0x00007f1b4b260198 in slapi_wait_condvar >>>>>> () from /usr/lib64/dirsrv/libslapd.so.0 >>>>>> No symbol table info available. >>>>>> #3 0x00007f1b3cf5deed in >>>>>> roles_cache_wait_on_change () from >>>>>> /usr/lib64/dirsrv/plugins/libroles-plugin.so >>>>>> No symbol table info available. >>>>>> #4 0x00007f1b494207bb in _pt_root () from >>>>>> /lib64/libnspr4.so >>>>>> No symbol table info available. >>>>>> #5 0x00007f1b48dc1dc5 in start_thread () from >>>>>> /lib64/libpthread.so.0 >>>>>> No symbol table info available. >>>>>> #6 0x00007f1b48aef28d in clone () from >>>>>> /lib64/libc.so.6 >>>>>> No symbol table info available. >>>>>> >>>>>> Thread 6 (Thread 0x7f1b270a1700 (LWP 105164)): >>>>>> #0 0x00007f1b48dc56d5 in >>>>>> pthread_cond_wait@@GLIBC_2.3.2 >>>>>> () >>>>>> from /lib64/libpthread.so.0 >>>>>> No symbol table info available. >>>>>> #1 0x00007f1b4941b050 in PR_WaitCondVar () >>>>>> from /lib64/libnspr4.so >>>>>> No symbol table info available. >>>>>> #2 0x00007f1b4b260198 in slapi_wait_condvar >>>>>> () from /usr/lib64/dirsrv/libslapd.so.0 >>>>>> No symbol table info available. >>>>>> #3 0x00007f1b3cf5deed in >>>>>> roles_cache_wait_on_change () from >>>>>> /usr/lib64/dirsrv/plugins/libroles-plugin.so >>>>>> No symbol table info available. >>>>>> #4 0x00007f1b494207bb in _pt_root () from >>>>>> /lib64/libnspr4.so >>>>>> No symbol table info available. >>>>>> #5 0x00007f1b48dc1dc5 in start_thread () from >>>>>> /lib64/libpthread.so.0 >>>>>> No symbol table info available. >>>>>> #6 0x00007f1b48aef28d in clone () from >>>>>> /lib64/libc.so.6 >>>>>> No symbol table info available. >>>>>> >>>>>> Thread 5 (Thread 0x7f1b278a2700 (LWP 105163)): >>>>>> #0 0x00007f1b48dc56d5 in >>>>>> pthread_cond_wait@@GLIBC_2.3.2 >>>>>> () >>>>>> from /lib64/libpthread.so.0 >>>>>> No symbol table info available. >>>>>> #1 0x00007f1b4941b050 in PR_WaitCondVar () >>>>>> from /lib64/libnspr4.so >>>>>> No symbol table info available. >>>>>> #2 0x00007f1b4b260198 in slapi_wait_condvar >>>>>> () from /usr/lib64/dirsrv/libslapd.so.0 >>>>>> No symbol table info available. >>>>>> #3 0x00007f1b415f262e in >>>>>> cos_cache_wait_on_change () from >>>>>> /usr/lib64/dirsrv/plugins/libcos-plugin.so >>>>>> No symbol table info available. >>>>>> #4 0x00007f1b494207bb in _pt_root () from >>>>>> /lib64/libnspr4.so >>>>>> No symbol table info available. >>>>>> #5 0x00007f1b48dc1dc5 in start_thread () from >>>>>> /lib64/libpthread.so.0 >>>>>> No symbol table info available. >>>>>> #6 0x00007f1b48aef28d in clone () from >>>>>> /lib64/libc.so.6 >>>>>> No symbol table info available. >>>>>> >>>>>> Thread 4 (Thread 0x7f1b282ac700 (LWP 105162)): >>>>>> #0 0x00007f1b48ae69b3 in select () from >>>>>> /lib64/libc.so.6 >>>>>> No symbol table info available. >>>>>> #1 0x00007f1b4b2710e9 in DS_Sleep () from >>>>>> /usr/lib64/dirsrv/libslapd.so.0 >>>>>> No symbol table info available. >>>>>> #2 0x00007f1b3e8ba707 in perf_threadmain () >>>>>> from /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>>>> No symbol table info available. >>>>>> #3 0x00007f1b494207bb in _pt_root () from >>>>>> /lib64/libnspr4.so >>>>>> No symbol table info available. >>>>>> #4 0x00007f1b48dc1dc5 in start_thread () from >>>>>> /lib64/libpthread.so.0 >>>>>> No symbol table info available. >>>>>> #5 0x00007f1b48aef28d in clone () from >>>>>> /lib64/libc.so.6 >>>>>> No symbol table info available. >>>>>> >>>>>> Thread 3 (Thread 0x7f1b28aad700 (LWP 105161)): >>>>>> #0 0x00007f1b48ae69b3 in select () from >>>>>> /lib64/libc.so.6 >>>>>> No symbol table info available. >>>>>> #1 0x00007f1b4b2710e9 in DS_Sleep () from >>>>>> /usr/lib64/dirsrv/libslapd.so.0 >>>>>> No symbol table info available. >>>>>> #2 0x00007f1b3e8bfb7f in trickle_threadmain >>>>>> () from /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>>>> No symbol table info available. >>>>>> #3 0x00007f1b494207bb in _pt_root () from >>>>>> /lib64/libnspr4.so >>>>>> No symbol table info available. >>>>>> #4 0x00007f1b48dc1dc5 in start_thread () from >>>>>> /lib64/libpthread.so.0 >>>>>> No symbol table info available. >>>>>> #5 0x00007f1b48aef28d in clone () from >>>>>> /lib64/libc.so.6 >>>>>> No symbol table info available. >>>>>> >>>>>> Thread 2 (Thread 0x7f1b292ae700 (LWP 105160)): >>>>>> #0 0x00007f1b48ae69b3 in select () from >>>>>> /lib64/libc.so.6 >>>>>> No symbol table info available. >>>>>> #1 0x00007f1b4344dbad in __os_yield () from >>>>>> /lib64/libdb-5.3.so >>>>>> No symbol table info available. >>>>>> #2 0x00007f1b434492b3 in __memp_sync_int () >>>>>> from /lib64/libdb-5.3.so >>>>>> No symbol table info available. >>>>>> #3 0x00007f1b43459752 in __txn_checkpoint () >>>>>> from /lib64/libdb-5.3.so >>>>>> No symbol table info available. >>>>>> #4 0x00007f1b43459b74 in __txn_checkpoint_pp >>>>>> () from /lib64/libdb-5.3.so >>>>>> No symbol table info available. >>>>>> #5 0x00007f1b3e8c3a87 in >>>>>> checkpoint_threadmain () from >>>>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>>>> No symbol table info available. >>>>>> #6 0x00007f1b494207bb in _pt_root () from >>>>>> /lib64/libnspr4.so >>>>>> No symbol table info available. >>>>>> #7 0x00007f1b48dc1dc5 in start_thread () from >>>>>> /lib64/libpthread.so.0 >>>>>> No symbol table info available. >>>>>> #8 0x00007f1b48aef28d in clone () from >>>>>> /lib64/libc.so.6 >>>>>> No symbol table info available. >>>>>> >>>>>> Thread 1 (Thread 0x7f1b29aaf700 (LWP 105159)): >>>>>> #0 0x00007f1b48ae69b3 in select () from >>>>>> /lib64/libc.so.6 >>>>>> No symbol table info available. >>>>>> #1 0x00007f1b4b2710e9 in DS_Sleep () from >>>>>> /usr/lib64/dirsrv/libslapd.so.0 >>>>>> No symbol table info available. >>>>>> #2 0x00007f1b3e8bf907 in deadlock_threadmain >>>>>> () from /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>>>> No symbol table info available. >>>>>> #3 0x00007f1b494207bb in _pt_root () from >>>>>> /lib64/libnspr4.so >>>>>> No symbol table info available. >>>>>> #4 0x00007f1b48dc1dc5 in start_thread () from >>>>>> /lib64/libpthread.so.0 >>>>>> No symbol table info available. >>>>>> #5 0x00007f1b48aef28d in clone () from >>>>>> /lib64/libc.so.6 >>>>>> No symbol table info available. >>>>>> >>>>>> >>>>>> >>>>>> Thanks, >>>>>> >>>>>> Rakesh >>>>>> >>>>>> >>>>>> On Mon, Sep 5, 2016 at 4:48 PM, thierry >>>>>> bordaz >>>>> > wrote: >>>>>> >>>>>> >>>>>> >>>>>> On 09/05/2016 12:05 PM, Rakesh >>>>>> Rajasekharan wrote: >>>>>>> Hi Thierry, >>>>>>> >>>>>>> >>>>>>> I was getting the hang issue while >>>>>>> running ipa-client-install >>>>>>> simultaneously on few clients.. >>>>>>> However, today, I am not able to >>>>>>> replicate that. >>>>>>> >>>>>>> I could not get a gdb . But i will try >>>>>>> getting that the next time I face this >>>>>>> issue. >>>>>>> >>>>>>> The CPU does not stay high.. it just >>>>>>> momentarily touches a high value and >>>>>>> then drops down to around 2-7% >>>>>>> >>>>>>> One question I have is , is it ok to set >>>>>>> it nsslapd-threadnumber to a very high >>>>>>> value . >>>>>>> I have around 4000 clients and with >>>>>>> nsslapd-maxthreadsperconn set to 5..So, >>>>>>> can I set nsslapd-threadnumber to around >>>>>>> 25000. >>>>>> >>>>>> Hello, >>>>>> >>>>>> I know some users running in production >>>>>> with several hunderds of threads (>600) >>>>>> and this without problem. >>>>>> >>>>>> I do not recall having suggested to >>>>>> increase that number and for what reason. >>>>>> Usually 30 workers is a good enough >>>>>> value. It can create bootleneck if for >>>>>> some reason each operation is very long >>>>>> to satisfy and exhaust the number of >>>>>> workers. You can monitor the work queue: >>>>>> >>>>>> ldapsearch -D "cn=directory manager" >>>>>> -w xxx -LLL -b "cn=monitor" -s base >>>>>> opsinitiated opscompleted >>>>>> >>>>>> >>>>>> If opscompleted-opsinitiated remains >>>>>> close to threadnumber, then yes it would >>>>>> be valuable to increase it. >>>>>> >>>>>> The computation #client * >>>>>> #async_op_per_client sound an overkill. >>>>>> Even if all clients send at the exact >>>>>> same time all their requests, it is very >>>>>> likely that some common resource (db >>>>>> page, log, allocator...) will serialize >>>>>> them. If you monitor a need to increase >>>>>> the work, you would for example set it to >>>>>> 50, then monitor, then set it to 100, >>>>>> then monitor... until you find a good >>>>>> enough value. >>>>>> Note the increasing the #thread, >>>>>> increases the memory footprint that will >>>>>> reduce the efficiency of file system >>>>>> cache and can increase the response time. >>>>>> >>>>>> >>>>>> best regards >>>>>> thierry >>>>>> >>>>>>> >>>>>>> Thanks >>>>>>> >>>>>>> On Mon, Sep 5, 2016 at 1:03 PM, thierry >>>>>>> bordaz >>>>>> > wrote: >>>>>>> >>>>>>> >>>>>>> Hi Rakesh, >>>>>>> >>>>>>> Were you able to get a pstack or >>>>>>> full stack with gdb >>>>>>> (http://www.port389.org/docs/389ds/FAQ/faq.html#debugging-crashes) >>>>>>> when the server hangs ? >>>>>>> >>>>>>> If it happens with 500 threads as >>>>>>> well as with 30, using 30 threads is >>>>>>> a better choice to debug this issue. >>>>>>> I will try to reproduce using 150 >>>>>>> parallel 'ipa user-find p-testipa' >>>>>>> commands >>>>>>> >>>>>>> Something I am unsure is if the CPU >>>>>>> consumption stays high (you >>>>>>> mentioned 340% CPU usage) as long as >>>>>>> the hang happens or if after a >>>>>>> suddent shot up to 340% (that marks >>>>>>> the beginning of the hang) it drops >>>>>>> and stay hanging ? >>>>>>> >>>>>>> thanks >>>>>>> thierry >>>>>>> >>>>>>> On 09/04/2016 08:40 PM, Rakesh >>>>>>> Rajasekharan wrote: >>>>>>>> starce on the slapd process >>>>>>>> actually had this in the output.. >>>>>>>> FUTEX_WAIT_PRIVATE >>>>>>>> >>>>>>>> and checking for the number of >>>>>>>> threads slapd had.. there were 5015 >>>>>>>> threads >>>>>>>> >>>>>>>> ps -efL|grep slapd|wc -l >>>>>>>> 5015 >>>>>>>> >>>>>>>> strace on most of the threads gave >>>>>>>> this output >>>>>>>> >>>>>>>> strace -p 67411 >>>>>>>> Process 67411 attached >>>>>>>> futex(0x7f3f0226b41c, >>>>>>>> FUTEX_WAIT_PRIVATE, 1, NULL) = -1 >>>>>>>> EAGAIN (Resource temporarily >>>>>>>> unavailable) >>>>>>>> futex(0x7f3f0226b41c, >>>>>>>> FUTEX_WAIT_PRIVATE, 2, >>>>>>>> NULL^CProcess 67411 detached >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> On Sun, Sep 4, 2016 at 5:34 PM, >>>>>>>> Rakesh Rajasekharan >>>>>>>> >>>>>>> > >>>>>>>> wrote: >>>>>>>> >>>>>>>> I have again got the issue of >>>>>>>> IPA hanging.. The issue came up >>>>>>>> when i tried to run >>>>>>>> ipa-client-isntall on 142 >>>>>>>> clients simultaneously >>>>>>>> >>>>>>>> >>>>>>>> None of the IPA commands are >>>>>>>> responding, and I see this error >>>>>>>> >>>>>>>> ipa user-find p-testipa >>>>>>>> ipa: ERROR: Insufficient >>>>>>>> access: SASL(-1): generic >>>>>>>> failure: GSSAPI Error: >>>>>>>> Unspecified GSS failure. Minor >>>>>>>> code may provide more >>>>>>>> information (KDC returned error >>>>>>>> string: PROCESS_TGS) >>>>>>>> >>>>>>>> KRB5_TRACE=/dev/stdout kinit admin >>>>>>>> [41178] 1472984115.233214: >>>>>>>> Getting initial credentials for >>>>>>>> admin at XYZ.COM >>>>>>>> >>>>>>>> [41178] 1472984115.235257: >>>>>>>> Sending request (167 bytes) to >>>>>>>> XYZ.COM >>>>>>>> [41178] 1472984115.235419: >>>>>>>> Initiating TCP connection to >>>>>>>> stream 10.1.3.36:88 >>>>>>>> >>>>>>>> [41178] 1472984115.235685: >>>>>>>> Sending TCP request to stream >>>>>>>> 10.1.3.36:88 >>>>>>>> [41178] 1472984120.238914: >>>>>>>> Received answer (174 bytes) >>>>>>>> from stream 10.1.3.36:88 >>>>>>>> >>>>>>>> [41178] 1472984120.238925: >>>>>>>> Terminating TCP connection to >>>>>>>> stream 10.1.3.36:88 >>>>>>>> >>>>>>>> [41178] 1472984120.238993: >>>>>>>> Response was from master KDC >>>>>>>> [41 >>>>>>>> >>>>>>>> >>>>>>>> Running an ldapsearch to see >>>>>>>> the db.. does not give any >>>>>>>> results and just hangs there >>>>>>>> >>>>>>>> ldapsearch -x -D 'cn=Directory >>>>>>>> Manager' -W -s one -b >>>>>>>> 'cn=kerberos,dc=xyz,dc=com' >>>>>>>> Enter LDAP Password: >>>>>>>> >>>>>>>> even an ldapsearch -x does not >>>>>>>> respond >>>>>>>> At this point, am sure that >>>>>>>> slapd is the one causing issues >>>>>>>> >>>>>>>> Running an strace against the >>>>>>>> hung slapd itself seems to get >>>>>>>> stuck does not proceed after >>>>>>>> saying "attaching to process" >>>>>>>> >>>>>>>> From some others posts I read >>>>>>>> Thierry suggesting to increase >>>>>>>> the nsslapd-threadnumber value >>>>>>>> >>>>>>>> It was set to 30, I think that >>>>>>>> might be too low. >>>>>>>> >>>> ... >>>> >>>> [Message clipped] >>>> >>>> >>>> >>> >> >> -- >> Red Hat GmbH,http://www.de.redhat.com/, Registered seat: Grasbrunn, >> Commercial register: Amtsgericht Muenchen, HRB 153243, >> Managing Directors: Charles Cachera, Michael Cunningham, Michael O'Neill, Eric Shander >> >> >> > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From mbasti at redhat.com Fri Sep 16 08:34:34 2016 From: mbasti at redhat.com (Martin Basti) Date: Fri, 16 Sep 2016 10:34:34 +0200 Subject: [Freeipa-users] certificates not renewing CA_UNREACHEABLE In-Reply-To: References: <806bdf32-558c-4972-255a-0901f47c4c4b@redhat.com> <6596593a-9470-bebd-f236-f9d17aa4a56c@redhat.com> Message-ID: On 16.09.2016 09:38, Natxo Asenjo wrote: > hi, > > > On Thu, Sep 15, 2016 at 1:03 PM, Natxo Asenjo > > On Thu, Sep 15, 2016 at 12:49 PM, Martin Basti > wrote: > > > > On 15.09.2016 12:44, Natxo Asenjo wrote: >> hi, >> >> On Thu, Sep 15, 2016 at 12:33 PM, Martin Basti >> > wrote: >> >> >> Hello, >> >> usually the most information can be found here >> /var/log/pki/pki-tomcat/ca/debug >> >> >> mmm, in this centos 6.8 system that does not exist: >> >> # ls -l /var/log/pki/pki-tomcat/ca/debug >> ls: cannot access /var/log/pki/pki-tomcat/ca/debug: No such >> file or directory >> >> >> I do have a /var/log/pki-ca/debug >> >> > Does it contain any information related to your issue? > > > I have tried renewing the certificate: > > ipa-getcert resubmit -i 20121107212513 > > > If I grep that file for that request id I find nothing recent, > just in the ipaserver installation log > > # cd /var/log > # grep -ri 20121107212513 *.log > ipaserver-install.log:2012-11-07T21:25:13Z DEBUG stdout=New > tracking request "20121107212513" added. > > # grep -ri 20121107212513 pki-ca > # > > > Any clues? > > > -- > Groeten, > natxo Sorry, I'm quite lost here, maybe somebody from dogtag can help what might be reason of those CA errors -------------- next part -------------- An HTML attachment was scrubbed... URL: From th at casalogic.dk Fri Sep 16 08:42:32 2016 From: th at casalogic.dk (Troels Hansen) Date: Fri, 16 Sep 2016 10:42:32 +0200 (CEST) Subject: [Freeipa-users] ipa trust-add using password Message-ID: <210728865.288453.1474015352468.JavaMail.zimbra@casalogic.dk> Hi, we are having some issues creating a IPA-AD trust, using password, and not shared secret, because of the error where name routing not getting created on AD if using shared secret. We have a AD domain tree with a top level domain and a domain below that where the users are located. We try to join the top level domain as a trust exists between those tow domains. Everything worked in our test setup, where we joined using a shared secret. We try to join our AD using this command: ipa trust-add --type=ad --admin @ --password However, we receive one of these two error messages: ipa: ERROR: CIFS server communication error: code "- 1073741712 ", message "Invalid workstation" (both may be "None") ipa: ERROR: AD domain controller complains about communication sequence. It may mean unsynchronized time on both sides, for example I think the first message was caused by some login restrictions on the user used to join, as it seems we don't receive that error massage anymore, and we receive the second error every time we try to join. We have tried pointing it to a specific server with the "--server" option, but that didn't change anything. -- Med venlig hilsen Troels Hansen Systemkonsulent Casalogic A/S T (+45) 70 20 10 63 M (+45) 22 43 71 57 Red Hat, SUSE, VMware, Citrix, Novell, Yellowfin BI, EnterpriseDB, Sophos og meget mere. -------------- next part -------------- An HTML attachment was scrubbed... URL: From lslebodn at redhat.com Fri Sep 16 08:43:26 2016 From: lslebodn at redhat.com (Lukas Slebodnik) Date: Fri, 16 Sep 2016 10:43:26 +0200 Subject: [Freeipa-users] 2FA using FreeIPA In-Reply-To: References: Message-ID: <20160916084326.GA10405@10.4.128.1> On (13/09/16 03:49), Deepak Dimri wrote: >Hi All, >I have below lines added to my sshd_config file for testuser. > > > >Match User testuser > AuthenticationMethods publickey,password:pam publickey,keyboard-interactive:pam >I have OTP enable for tapuser in IPA and i am able to login to GUI using the password + OTP. However when i try to ssh i am getting prompted for first factor then second factor and then it ends with "Permission denied (keyboard-interactive)." error. What could be wrong here? >Regards,Deepak > Please provide versions of freeIPA server packages, version of sssd. And it would be good to seed the exact output of ssh authentication. LS From abokovoy at redhat.com Fri Sep 16 08:50:12 2016 From: abokovoy at redhat.com (Alexander Bokovoy) Date: Fri, 16 Sep 2016 11:50:12 +0300 Subject: [Freeipa-users] ipa trust-add using password In-Reply-To: <210728865.288453.1474015352468.JavaMail.zimbra@casalogic.dk> References: <210728865.288453.1474015352468.JavaMail.zimbra@casalogic.dk> Message-ID: <20160916085012.joed3d6renupfush@redhat.com> On Fri, 16 Sep 2016, Troels Hansen wrote: >Hi, we are having some issues creating a IPA-AD trust, using password, and not shared secret, because of the error where name routing not getting created on AD if using shared secret. > >We have a AD domain tree with a top level domain and a domain below that where the users are located. We try to join the top level domain as a trust exists between those tow domains. > >Everything worked in our test setup, where we joined using a shared secret. > >We try to join our AD using this command: >ipa trust-add --type=ad --admin @ --password > >However, we receive one of these two error messages: > >ipa: ERROR: CIFS server communication error: code "- 1073741712 ", >message "Invalid workstation" (both may be "None") > >ipa: ERROR: AD domain controller complains about communication >sequence. It may mean unsynchronized time on both sides, for example > >I think the first message was caused by some login restrictions on the user used to join, as it seems we don't receive that error massage anymore, and we receive the second error every time we try to join. > >We have tried pointing it to a specific server with the "--server" option, but that didn't change anything. > If you add 'log level = 50' to /usr/share/ipa/smb.conf.empty, then /var/log/httpd/error_log will contain detailed debug information from IPA attempts to talk to AD DCs. -- / Alexander Bokovoy From lslebodn at redhat.com Fri Sep 16 08:51:38 2016 From: lslebodn at redhat.com (Lukas Slebodnik) Date: Fri, 16 Sep 2016 10:51:38 +0200 Subject: [Freeipa-users] Issues with FreeIPA SSH Key authentication In-Reply-To: References: <20160915070948.GA1485@10.4.128.1> Message-ID: <20160916085137.GB10405@10.4.128.1> On (15/09/16 11:46), Venkataramana Kintali wrote: >Hi Lukas, >ssh_config is also same on all servers. >Our need is to do it both ways, to be able to login with ssh public >keys(uploaded in IPA) and disable password login, and be able to access >allhosts within the same IPA domain silently from any host. >Hoping the configs will help, I am including the configurations here. > >ssh_config file : http://pastebin.com/MWHyH1Qw >sshd_config file: http://pastebin.com/gpn5XhXM >sssd_config file: http://pastebin.com/5Pby6xKp > Looks good to me >I just used some placeholders for sssd_config file in pastebin instead of >actual values. > In initial mail you wrote: >I am able to login to some IPA clients but not able to login to other IPA >clients with putty using private key and passphrase. Therefore your previous test case is wrong. If you want to test authentication with public keys then you cannot obtain krb5 ticket with kinit. I would also recommend to call kdestory before authentication with ssh to be sure that gssapi authentication will not be used. I would recomment to set "debug_level = 7" in domain and ssh section on the server where you woudl like to authenticate. then restart sssd and try to authenticate with ssh + verbose mode e.g. ssh -v user at remote.host Then I would recommend to compare logs from working server and from broken server. LS From rakesh.rajasekharan at gmail.com Fri Sep 16 10:12:47 2016 From: rakesh.rajasekharan at gmail.com (Rakesh Rajasekharan) Date: Fri, 16 Sep 2016 15:42:47 +0530 Subject: [Freeipa-users] Freeipa 4.2.0 hangs intermittently In-Reply-To: <57DBAD77.8090308@redhat.com> References: <57CD1FDC.2010109@redhat.com> <57CD5488.60404@redhat.com> <57CD659B.6050707@redhat.com> <57CD68BD.9030802@redhat.com> <57CE8CDB.6000809@redhat.com> <57CE9DE5.6020301@redhat.com> <57DACE76.5070803@redhat.com> <57DBAD77.8090308@redhat.com> Message-ID: Hi Thierry, I am not quite sure on the SRCH or ADD operations as the environment was live and few other guys also are using the command line to search or add users. I did not promote it to replica and was running just as a client Thanks, Rakesh On Fri, Sep 16, 2016 at 1:59 PM, thierry bordaz wrote: > Hi Rakesh, > > Thank you so much for those precious feedback. > You are right I did test 4.3 while you were testing 4.2. Upgrading is a > good option but regarding the suspected bug it should not have impact but > the dynamic of the install is likely quite different. > > I do not know from where is coming this flow of Clock skew messages. Is > ntpd still running ? > > > At the time of the hang, the provided pstacks contained some SRCHs and > sometime ADD. Do you remember the kind of search/update you did apply when > the hang occurred ? From server or client ? > Also after configuring client, did you promote them to replica or just > keep them as client ? > > thanks > thierry > > On 09/15/2016 09:14 PM, Rakesh Rajasekharan wrote: > > Hi Thierry, > > I had been doing pretty much the same to reproduce the hang.. > After changing the checkpoint interval.. the hangs had become manageable.. > > However, this time when the hang occurred.. I did not add any clients and > also the behavior was bit different. This time my ns-slapd did not hang.. > ldapsearch was working fine.. but, the ipa commands were taking a long > time to respond and most users were not able to login to any of the > clients.. > restarting ipa did not help either. > > I was seeing around 14-15 connections in SYN_RECV state continously and > /var.log/messages has entries like "TCP: request_sock_TCP: Possible SYN > flooding on port 88. Sending cookies. Check SNMP counter" > > I also see in the /var/log/krb5kdc.log this error > > ipa-master-xyz.com krb5kdc[15308](info): TGS_REQ (4 etypes {18 17 16 23}) > 10.1.12.155: PROCESS_TGS: authtime 0, host/10.1.12.155 at XYZ.COM for ldap/ > ipa-master-xyz.com at XYZ.COM, Clock skew too great > > there are arounf 7.5k messages like that.. which I guess is too high > grep "krbtgt/XYZ.COM at XYZ.COM, Clock skew too great" /var/log/krb5kdc.log > | wc -l > 7570 > > Would this give rise to a hang or make IPA nonresponsive.. > > The kinit admin command itself was taking an unusually long time to > respond.. > > KRB5_TRACE=/dev/stderr kinit admin > [11298] 1473702491.60880: Getting initial credentials for > admin at XYZ.COM > [11298] 1473702491.62981: Sending request (167 bytes) to XYZ.COM > [11298] 1473702491.63119: Initiating TCP connection to stream 10.1.3.35:88 > [11298] 1473702491.63359: Sending TCP request to stream 10.1.3.35:88 > [11298] 1473702493.797835: Received answer (341 bytes) from stream > 10.1.3.35:88 > [11298] 1473702493.797848: Terminating TCP connection to stream > 10.1.3.35:88 > [11298] 1473702493.797911: Response was from master KDC > [11298] 1473702493.797956: Received error from KDC: -1765328359/Additional > pre-authentication required > [11298] 1473702493.797993: Processing preauth types: 136, 19, 2, 133 > [11298] 1473702493.798005: Selected etype info: etype aes256-cts, salt > "V at Cbu147E#1;R0WD", params "" > [11298] 1473702493.798009: Received cookie: MIT > Password for admin at XYZ.COM: > [11298] 1473702498.190064: AS key obtained for encrypted timestamp: > aes256-cts/2C9D > [11298] 1473702498.190109: Encrypted timestamp (for 1473702498.184527): > plain 301AA011180F32303136303931323137343831385AA105020302D0CF, encrypted > 25FC8D37EFB6B7837C8D5C6649DFB9972010D40EE29D1222FBA45CAA9842 > 8E42C7FCC9B7FE881A04BD3390A6A9EDE9D2D93729FDF3E47B6D > [11298] 1473702498.190129: Preauth module encrypted_timestamp (2) (real) > returned: 0/Success > [11298] 1473702498.190133: Produced preauth for next request: 133, 2 > [11298] 1473702498.190148: Sending request (261 bytes) to XYZ.COM > [11298] 1473702498.190246: Initiating TCP connection to stream > 10.1.3.35:88 > [11298] 1473702499.191933: Sending initial UDP request to dgram > 10.1.3.35:88 > [11298] 1473702502.195157: Sending retry UDP request to dgram 10.1.3.35:88 > [11298] 1473702507.200405: Sending retry UDP request to dgram 10.1.3.35:88 > [11298] 1473702513.226371: Sending TCP request to stream 10.1.3.35:88 > [11298] 1473702515.797243: Received answer (730 bytes) from stream > 10.1.3.35:88 > [11298] 1473702515.797271: Terminating TCP connection to stream > 10.1.3.35:88 > [11298] 1473702515.797326: Response was from master KDC > [11298] 1473702515.797353: Processing preauth types: 19 > [11298] 1473702515.797360: Selected etype info: etype aes256-cts, salt > "V at Cbu147E#1;R0WD", params "" > [11298] 1473702515.797394: Produced preauth for next request: (empty) > [11298] 1473702515.797401: AS key determined by preauth: aes256-cts/2C9D > [11298] 1473702515.797445: Decrypted AS reply; session key is: > aes256-cts/702E > [11298] 1473702515.797460: FAST negotiation: available > [11298] 1473702515.797478: Initializing KEYRING:persistent:0:0 with > default princ admin at XYZ.COM > [11298] 1473702515.797534: Storing admin at XYZ.COM -> krbtgt/ > XYZ.COM at XYZ.COM in KEYRING:persistent:0:0 > [11298] 1473702515.797572: Storing config in KEYRING:persistent:0:0 for > krbtgt/XYZ.COM at XYZ.COM: fast_avail: yes > [11298] 1473702515.797585: Storing admin at XYZ.COM -> > krb5_ccache_conf_data/fast_avail/krbtgt\/XYZ.COM\@XYZ.COM at X-CACHECONF: in > KEYRING:persistent:0:0 > [11298] 1473702515.797631: Storing config in KEYRING:persistent:0:0 for > krbtgt/XYZ.COM at XYZ.COM: pa_type: 2 > [11298] 1473702515.797647: Storing admin at XYZ.COM -> > krb5_ccache_conf_data/pa_type/krbtgt\/XYZ.COM\@XYZ.COM at X-CACHECONF: in > KEYRING:persistent:0:0 > > > > Could it be an version specific issue..as I am running 4.2.0 verison.. and > yours is 4.3.2 > > If so, I will try to upgrade my IPA version > > Thanks > Rakesh > > > > > > > On Thu, Sep 15, 2016 at 10:08 PM, thierry bordaz > wrote: > >> Hello Rakesh, >> >> Sorry to be back so late. I finally manage to try to reproduce locally >> the hang of DS... and I fail ;-) >> >> I am running 4 VM, with freeipa 4.3.2. >> One is running a server (where I provisioned some users), with 200Mb >> cache. The server receives no direct load (SRCH or update). >> The tree others are running in a loop ipa-client-install and uninstall. >> So far it has run for several hours without hang. >> >> Going through the mail thread I have not found others details. >> Am I missing something to reproduce ? >> >> thanks in advance for your help >> thierry >> >> On 09/07/2016 01:46 PM, Rakesh Rajasekharan wrote: >> >> I changed the nsslapd-db-checkpoint interval first to 80 and increased >> till 160.. however the hang has not gone away yet >> >> >> I migrated my data from a openldap, can that cause issues like this.. If >> so, i could try creating users from scratch directly on IPA rather than >> migrating and test it >> >> But before that , have taken a fresh pstack just incase if it helps to >> nail down the issue >> >> Thread 44 (Thread 0x7f53ed52f700 (LWP 128165)): >> #0 0x00007f54047649b3 in select () from /lib64/libc.so.6 >> #1 0x00007f5406eef0e9 in DS_Sleep () from /usr/lib64/dirsrv/libslapd.so. >> 0 >> #2 0x00007f53fa53d907 in deadlock_threadmain () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #3 0x00007f540509e7bb in _pt_root () from /lib64/libnspr4.so >> #4 0x00007f5404a3fdc5 in start_thread () from /lib64/libpthread.so.0 >> #5 0x00007f540476d28d in clone () from /lib64/libc.so.6 >> Thread 43 (Thread 0x7f53ecd2e700 (LWP 128166)): >> #0 0x00007f54047649b3 in select () from /lib64/libc.so.6 >> #1 0x00007f53ff0cbbad in __os_yield () from /lib64/libdb-5.3.so >> #2 0x00007f53ff0c72b3 in __memp_sync_int () from /lib64/libdb-5.3.so >> #3 0x00007f53ff0d7752 in __txn_checkpoint () from /lib64/libdb-5.3.so >> #4 0x00007f53ff0d7b74 in __txn_checkpoint_pp () from /lib64/libdb-5.3.so >> #5 0x00007f53fa541a87 in checkpoint_threadmain () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #6 0x00007f540509e7bb in _pt_root () from /lib64/libnspr4.so >> #7 0x00007f5404a3fdc5 in start_thread () from /lib64/libpthread.so.0 >> #8 0x00007f540476d28d in clone () from /lib64/libc.so.6 >> Thread 42 (Thread 0x7f53ec52d700 (LWP 128167)): >> #0 0x00007f54047649b3 in select () from /lib64/libc.so.6 >> #1 0x00007f5406eef0e9 in DS_Sleep () from /usr/lib64/dirsrv/libslapd.so. >> 0 >> #2 0x00007f53fa53db7f in trickle_threadmain () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #3 0x00007f540509e7bb in _pt_root () from /lib64/libnspr4.so >> #4 0x00007f5404a3fdc5 in start_thread () from /lib64/libpthread.so.0 >> #5 0x00007f540476d28d in clone () from /lib64/libc.so.6 >> Thread 41 (Thread 0x7f53ebd2c700 (LWP 128168)): >> #0 0x00007f54047649b3 in select () from /lib64/libc.so.6 >> #1 0x00007f5406eef0e9 in DS_Sleep () from /usr/lib64/dirsrv/libslapd.so. >> 0 >> #2 0x00007f53fa538707 in perf_threadmain () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #3 0x00007f540509e7bb in _pt_root () from /lib64/libnspr4.so >> #4 0x00007f5404a3fdc5 in start_thread () from /lib64/libpthread.so.0 >> #5 0x00007f540476d28d in clone () from /lib64/libc.so.6 >> Thread 40 (Thread 0x7f53eb322700 (LWP 128204)): >> #0 0x00007f5404a436d5 in pthread_cond_wait@@GLIBC_2.3.2 () from >> /lib64/libpthread.so.0 >> #1 0x00007f5405099050 in PR_WaitCondVar () from /lib64/libnspr4.so >> #2 0x00007f5406ede198 in slapi_wait_condvar () from >> /usr/lib64/dirsrv/libslapd.so.0 >> #3 0x00007f53fd27062e in cos_cache_wait_on_change () from >> /usr/lib64/dirsrv/plugins/libcos-plugin.so >> #4 0x00007f540509e7bb in _pt_root () from /lib64/libnspr4.so >> #5 0x00007f5404a3fdc5 in start_thread () from /lib64/libpthread.so.0 >> #6 0x00007f540476d28d in clone () from /lib64/libc.so.6 >> Thread 39 (Thread 0x7f53eab21700 (LWP 128206)): >> #0 0x00007f5404a436d5 in pthread_cond_wait@@GLIBC_2.3.2 () from >> /lib64/libpthread.so.0 >> #1 0x00007f5405099050 in PR_WaitCondVar () from /lib64/libnspr4.so >> #2 0x00007f5406ede198 in slapi_wait_condvar () from >> /usr/lib64/dirsrv/libslapd.so.0 >> #3 0x00007f53f8bdbeed in roles_cache_wait_on_change () from >> /usr/lib64/dirsrv/plugins/libroles-plugin.so >> #4 0x00007f540509e7bb in _pt_root () from /lib64/libnspr4.so >> #5 0x00007f5404a3fdc5 in start_thread () from /lib64/libpthread.so.0 >> #6 0x00007f540476d28d in clone () from /lib64/libc.so.6 >> Thread 38 (Thread 0x7f53ea320700 (LWP 128207)): >> #0 0x00007f5404a436d5 in pthread_cond_wait@@GLIBC_2.3.2 () from >> /lib64/libpthread.so.0 >> #1 0x00007f5405099050 in PR_WaitCondVar () from /lib64/libnspr4.so >> #2 0x00007f5406ede198 in slapi_wait_condvar () from >> /usr/lib64/dirsrv/libslapd.so.0 >> #3 0x00007f53f8bdbeed in roles_cache_wait_on_change () from >> /usr/lib64/dirsrv/plugins/libroles-plugin.so >> #4 0x00007f540509e7bb in _pt_root () from /lib64/libnspr4.so >> #5 0x00007f5404a3fdc5 in start_thread () from /lib64/libpthread.so.0 >> #6 0x00007f540476d28d in clone () from /lib64/libc.so.6 >> Thread 37 (Thread 0x7f53e9b1f700 (LWP 128208)): >> #0 0x00007f5404a436d5 in pthread_cond_wait@@GLIBC_2.3.2 () from >> /lib64/libpthread.so.0 >> #1 0x00007f5405099050 in PR_WaitCondVar () from /lib64/libnspr4.so >> #2 0x00007f5406ede198 in slapi_wait_condvar () from >> /usr/lib64/dirsrv/libslapd.so.0 >> #3 0x00007f53f8bdbeed in roles_cache_wait_on_change () from >> /usr/lib64/dirsrv/plugins/libroles-plugin.so >> #4 0x00007f540509e7bb in _pt_root () from /lib64/libnspr4.so >> #5 0x00007f5404a3fdc5 in start_thread () from /lib64/libpthread.so.0 >> #6 0x00007f540476d28d in clone () from /lib64/libc.so.6 >> Thread 36 (Thread 0x7f53e931e700 (LWP 128209)): >> #0 0x00007f5404a43a82 in pthread_cond_timedwait@@GLIBC_2.3.2 () from >> /lib64/libpthread.so.0 >> #1 0x00007f5405098b07 in pt_TimedWait () from /lib64/libnspr4.so >> #2 0x00007f5405098fce in PR_WaitCondVar () from /lib64/libnspr4.so >> #3 0x00007f5407397d03 in housecleaning () >> #4 0x00007f540509e7bb in _pt_root () from /lib64/libnspr4.so >> #5 0x00007f5404a3fdc5 in start_thread () from /lib64/libpthread.so.0 >> #6 0x00007f540476d28d in clone () from /lib64/libc.so.6 >> Thread 35 (Thread 0x7f53e8b1d700 (LWP 128210)): >> #0 0x00007f5404a43a82 in pthread_cond_timedwait@@GLIBC_2.3.2 () from >> /lib64/libpthread.so.0 >> #1 0x00007f5405098b07 in pt_TimedWait () from /lib64/libnspr4.so >> #2 0x00007f5405098fce in PR_WaitCondVar () from /lib64/libnspr4.so >> #3 0x00007f5406e841e8 in eq_loop () from /usr/lib64/dirsrv/libslapd.so.0 >> #4 0x00007f540509e7bb in _pt_root () from /lib64/libnspr4.so >> #5 0x00007f5404a3fdc5 in start_thread () from /lib64/libpthread.so.0 >> #6 0x00007f540476d28d in clone () from /lib64/libc.so.6 >> Thread 34 (Thread 0x7f53db7fe700 (LWP 128212)): >> #0 0x00007f5404a436d5 in pthread_cond_wait@@GLIBC_2.3.2 () from >> /lib64/libpthread.so.0 >> #1 0x00007f53fef9f2f3 in __db_hybrid_mutex_suspend () from /lib64/ >> libdb-5.3.so >> #2 0x00007f53fef9e640 in __db_tas_mutex_lock () from /lib64/libdb-5.3.so >> #3 0x00007f53ff048d3a in __lock_get_internal () from /lib64/libdb-5.3.so >> #4 0x00007f53ff049820 in __lock_get () from /lib64/libdb-5.3.so >> #5 0x00007f53ff075142 in __db_lget () from /lib64/libdb-5.3.so >> #6 0x00007f53fefbc605 in __bam_search () from /lib64/libdb-5.3.so >> #7 0x00007f53fefa7256 in __bamc_search () from /lib64/libdb-5.3.so >> #8 0x00007f53fefa8d0f in __bamc_get () from /lib64/libdb-5.3.so >> #9 0x00007f53ff061ca6 in __dbc_iget () from /lib64/libdb-5.3.so >> #10 0x00007f53ff070b02 in __dbc_get_pp () from /lib64/libdb-5.3.so >> #11 0x00007f53fa54c680 in idl_new_fetch () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #12 0x00007f53fa55ab26 in index_read_ext_allids () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #13 0x00007f53fa545284 in keys2idl () from /usr/lib64/dirsrv/plugins/libb >> ack-ldbm.so >> #14 0x00007f53fa5459e3 in ava_candidates.isra.0 () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #15 0x00007f53fa545fd2 in filter_candidates_ext () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #16 0x00007f53fa547096 in list_candidates () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #17 0x00007f53fa545f40 in filter_candidates_ext () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #18 0x00007f53fa58180c in subtree_candidates () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #19 0x00007f53fa582e7f in ldbm_back_search () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #20 0x00007f5406eb01d6 in op_shared_search () from >> /usr/lib64/dirsrv/libslapd.so.0 >> #21 0x00007f54073a133e in do_search () >> #22 0x00007f540738f567 in connection_threadmain () >> #23 0x00007f540509e7bb in _pt_root () from /lib64/libnspr4.so >> #24 0x00007f5404a3fdc5 in start_thread () from /lib64/libpthread.so.0 >> #25 0x00007f540476d28d in clone () from /lib64/libc.so.6 >> Thread 33 (Thread 0x7f53daffd700 (LWP 128213)): >> #0 0x00007f5404a436d5 in pthread_cond_wait@@GLIBC_2.3.2 () from >> /lib64/libpthread.so.0 >> #1 0x00007f53fef9f2f3 in __db_hybrid_mutex_suspend () from /lib64/ >> libdb-5.3.so >> #2 0x00007f53fef9e640 in __db_tas_mutex_lock () from /lib64/libdb-5.3.so >> #3 0x00007f53ff048d3a in __lock_get_internal () from /lib64/libdb-5.3.so >> #4 0x00007f53ff049820 in __lock_get () from /lib64/libdb-5.3.so >> #5 0x00007f53ff075142 in __db_lget () from /lib64/libdb-5.3.so >> #6 0x00007f53fefbc605 in __bam_search () from /lib64/libdb-5.3.so >> #7 0x00007f53fefa7256 in __bamc_search () from /lib64/libdb-5.3.so >> #8 0x00007f53fefa8d0f in __bamc_get () from /lib64/libdb-5.3.so >> #9 0x00007f53ff061ca6 in __dbc_iget () from /lib64/libdb-5.3.so >> #10 0x00007f53ff070b02 in __dbc_get_pp () from /lib64/libdb-5.3.so >> #11 0x00007f53fa54c680 in idl_new_fetch () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #12 0x00007f53fa55ab26 in index_read_ext_allids () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #13 0x00007f53fa545284 in keys2idl () from /usr/lib64/dirsrv/plugins/libb >> ack-ldbm.so >> #14 0x00007f53fa5459e3 in ava_candidates.isra.0 () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #15 0x00007f53fa545fd2 in filter_candidates_ext () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #16 0x00007f53fa547096 in list_candidates () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #17 0x00007f53fa545f40 in filter_candidates_ext () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #18 0x00007f53fa58180c in subtree_candidates () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #19 0x00007f53fa582e7f in ldbm_back_search () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #20 0x00007f5406eb01d6 in op_shared_search () from >> /usr/lib64/dirsrv/libslapd.so.0 >> #21 0x00007f54073a133e in do_search () >> #22 0x00007f540738f567 in connection_threadmain () >> #23 0x00007f540509e7bb in _pt_root () from /lib64/libnspr4.so >> #24 0x00007f5404a3fdc5 in start_thread () from /lib64/libpthread.so.0 >> #25 0x00007f540476d28d in clone () from /lib64/libc.so.6 >> Thread 32 (Thread 0x7f53da7fc700 (LWP 128214)): >> #0 0x00007f5404a436d5 in pthread_cond_wait@@GLIBC_2.3.2 () from >> /lib64/libpthread.so.0 >> #1 0x00007f53fef9f2f3 in __db_hybrid_mutex_suspend () from /lib64/ >> libdb-5.3.so >> #2 0x00007f53fef9e640 in __db_tas_mutex_lock () from /lib64/libdb-5.3.so >> #3 0x00007f53ff048d3a in __lock_get_internal () from /lib64/libdb-5.3.so >> #4 0x00007f53ff049820 in __lock_get () from /lib64/libdb-5.3.so >> #5 0x00007f53ff075142 in __db_lget () from /lib64/libdb-5.3.so >> #6 0x00007f53fefbc605 in __bam_search () from /lib64/libdb-5.3.so >> #7 0x00007f53fefa7256 in __bamc_search () from /lib64/libdb-5.3.so >> #8 0x00007f53fefa8d0f in __bamc_get () from /lib64/libdb-5.3.so >> #9 0x00007f53ff061ca6 in __dbc_iget () from /lib64/libdb-5.3.so >> #10 0x00007f53ff070b02 in __dbc_get_pp () from /lib64/libdb-5.3.so >> #11 0x00007f53fa54c680 in idl_new_fetch () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #12 0x00007f53fa55ab26 in index_read_ext_allids () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #13 0x00007f53fa545284 in keys2idl () from /usr/lib64/dirsrv/plugins/libb >> ack-ldbm.so >> #14 0x00007f53fa5459e3 in ava_candidates.isra.0 () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #15 0x00007f53fa545fd2 in filter_candidates_ext () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #16 0x00007f53fa547096 in list_candidates () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #17 0x00007f53fa545f40 in filter_candidates_ext () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #18 0x00007f53fa547096 in list_candidates () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #19 0x00007f53fa545f40 in filter_candidates_ext () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #20 0x00007f53fa547096 in list_candidates () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #21 0x00007f53fa545f40 in filter_candidates_ext () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #22 0x00007f53fa58180c in subtree_candidates () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #23 0x00007f53fa582e7f in ldbm_back_search () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #24 0x00007f5406eb01d6 in op_shared_search () from >> /usr/lib64/dirsrv/libslapd.so.0 >> #25 0x00007f54073a133e in do_search () >> #26 0x00007f540738f3c3 in connection_threadmain () >> #27 0x00007f540509e7bb in _pt_root () from /lib64/libnspr4.so >> #28 0x00007f5404a3fdc5 in start_thread () from /lib64/libpthread.so.0 >> #29 0x00007f540476d28d in clone () from /lib64/libc.so.6 >> Thread 31 (Thread 0x7f53d9ffb700 (LWP 128215)): >> #0 0x00007f5404a436d5 in pthread_cond_wait@@GLIBC_2.3.2 () from >> /lib64/libpthread.so.0 >> #1 0x00007f53fef9f2f3 in __db_hybrid_mutex_suspend () from /lib64/ >> libdb-5.3.so >> #2 0x00007f53fef9e640 in __db_tas_mutex_lock () from /lib64/libdb-5.3.so >> #3 0x00007f53ff048d3a in __lock_get_internal () from /lib64/libdb-5.3.so >> #4 0x00007f53ff049820 in __lock_get () from /lib64/libdb-5.3.so >> #5 0x00007f53ff075142 in __db_lget () from /lib64/libdb-5.3.so >> #6 0x00007f53fefbc605 in __bam_search () from /lib64/libdb-5.3.so >> #7 0x00007f53fefa7256 in __bamc_search () from /lib64/libdb-5.3.so >> #8 0x00007f53fefa8d0f in __bamc_get () from /lib64/libdb-5.3.so >> #9 0x00007f53ff061ca6 in __dbc_iget () from /lib64/libdb-5.3.so >> #10 0x00007f53ff070b02 in __dbc_get_pp () from /lib64/libdb-5.3.so >> #11 0x00007f53fa54c680 in idl_new_fetch () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #12 0x00007f53fa55ab26 in index_read_ext_allids () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #13 0x00007f53fa545284 in keys2idl () from /usr/lib64/dirsrv/plugins/libb >> ack-ldbm.so >> #14 0x00007f53fa5459e3 in ava_candidates.isra.0 () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #15 0x00007f53fa545fd2 in filter_candidates_ext () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #16 0x00007f53fa547096 in list_candidates () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #17 0x00007f53fa545f40 in filter_candidates_ext () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #18 0x00007f53fa58180c in subtree_candidates () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #19 0x00007f53fa582e7f in ldbm_back_search () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #20 0x00007f5406eb01d6 in op_shared_search () from >> /usr/lib64/dirsrv/libslapd.so.0 >> #21 0x00007f54073a133e in do_search () >> #22 0x00007f540738f567 in connection_threadmain () >> #23 0x00007f540509e7bb in _pt_root () from /lib64/libnspr4.so >> #24 0x00007f5404a3fdc5 in start_thread () from /lib64/libpthread.so.0 >> #25 0x00007f540476d28d in clone () from /lib64/libc.so.6 >> Thread 30 (Thread 0x7f53d97fa700 (LWP 128216)): >> #0 0x00007f5404a436d5 in pthread_cond_wait@@GLIBC_2.3.2 () from >> /lib64/libpthread.so.0 >> #1 0x00007f53fef9f2f3 in __db_hybrid_mutex_suspend () from /lib64/ >> libdb-5.3.so >> #2 0x00007f53fef9e640 in __db_tas_mutex_lock () from /lib64/libdb-5.3.so >> #3 0x00007f53ff048d3a in __lock_get_internal () from /lib64/libdb-5.3.so >> #4 0x00007f53ff049820 in __lock_get () from /lib64/libdb-5.3.so >> #5 0x00007f53ff075142 in __db_lget () from /lib64/libdb-5.3.so >> #6 0x00007f53fefbc605 in __bam_search () from /lib64/libdb-5.3.so >> #7 0x00007f53fefa7256 in __bamc_search () from /lib64/libdb-5.3.so >> #8 0x00007f53fefa8d0f in __bamc_get () from /lib64/libdb-5.3.so >> #9 0x00007f53ff061ca6 in __dbc_iget () from /lib64/libdb-5.3.so >> #10 0x00007f53ff070b02 in __dbc_get_pp () from /lib64/libdb-5.3.so >> #11 0x00007f53fa54c680 in idl_new_fetch () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #12 0x00007f53fa55ab26 in index_read_ext_allids () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #13 0x00007f53fa545284 in keys2idl () from /usr/lib64/dirsrv/plugins/libb >> ack-ldbm.so >> #14 0x00007f53fa5459e3 in ava_candidates.isra.0 () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #15 0x00007f53fa545fd2 in filter_candidates_ext () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #16 0x00007f53fa547096 in list_candidates () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #17 0x00007f53fa545f40 in filter_candidates_ext () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #18 0x00007f53fa58180c in subtree_candidates () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #19 0x00007f53fa582e7f in ldbm_back_search () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #20 0x00007f5406eb01d6 in op_shared_search () from >> /usr/lib64/dirsrv/libslapd.so.0 >> #21 0x00007f54073a133e in do_search () >> #22 0x00007f540738f567 in connection_threadmain () >> #23 0x00007f540509e7bb in _pt_root () from /lib64/libnspr4.so >> #24 0x00007f5404a3fdc5 in start_thread () from /lib64/libpthread.so.0 >> #25 0x00007f540476d28d in clone () from /lib64/libc.so.6 >> Thread 29 (Thread 0x7f53d8ff9700 (LWP 128217)): >> #0 0x00007f5404a436d5 in pthread_cond_wait@@GLIBC_2.3.2 () from >> /lib64/libpthread.so.0 >> #1 0x00007f53fef9f2f3 in __db_hybrid_mutex_suspend () from /lib64/ >> libdb-5.3.so >> #2 0x00007f53fef9e640 in __db_tas_mutex_lock () from /lib64/libdb-5.3.so >> #3 0x00007f53ff048d3a in __lock_get_internal () from /lib64/libdb-5.3.so >> #4 0x00007f53ff049820 in __lock_get () from /lib64/libdb-5.3.so >> #5 0x00007f53ff075142 in __db_lget () from /lib64/libdb-5.3.so >> #6 0x00007f53fefbc605 in __bam_search () from /lib64/libdb-5.3.so >> #7 0x00007f53fefa7256 in __bamc_search () from /lib64/libdb-5.3.so >> #8 0x00007f53fefa8d0f in __bamc_get () from /lib64/libdb-5.3.so >> #9 0x00007f53ff061ca6 in __dbc_iget () from /lib64/libdb-5.3.so >> #10 0x00007f53ff070b02 in __dbc_get_pp () from /lib64/libdb-5.3.so >> #11 0x00007f53fa54c680 in idl_new_fetch () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #12 0x00007f53fa55ab26 in index_read_ext_allids () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #13 0x00007f53fa545284 in keys2idl () from /usr/lib64/dirsrv/plugins/libb >> ack-ldbm.so >> #14 0x00007f53fa5459e3 in ava_candidates.isra.0 () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #15 0x00007f53fa545fd2 in filter_candidates_ext () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #16 0x00007f53fa547096 in list_candidates () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #17 0x00007f53fa545f40 in filter_candidates_ext () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #18 0x00007f53fa58180c in subtree_candidates () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #19 0x00007f53fa582e7f in ldbm_back_search () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #20 0x00007f5406eb01d6 in op_shared_search () from >> /usr/lib64/dirsrv/libslapd.so.0 >> #21 0x00007f54073a133e in do_search () >> #22 0x00007f540738f567 in connection_threadmain () >> #23 0x00007f540509e7bb in _pt_root () from /lib64/libnspr4.so >> #24 0x00007f5404a3fdc5 in start_thread () from /lib64/libpthread.so.0 >> #25 0x00007f540476d28d in clone () from /lib64/libc.so.6 >> Thread 28 (Thread 0x7f53d87f8700 (LWP 128218)): >> #0 0x00007f5404a436d5 in pthread_cond_wait@@GLIBC_2.3.2 () from >> /lib64/libpthread.so.0 >> #1 0x00007f53fef9f2f3 in __db_hybrid_mutex_suspend () from /lib64/ >> libdb-5.3.so >> #2 0x00007f53fef9e640 in __db_tas_mutex_lock () from /lib64/libdb-5.3.so >> #3 0x00007f53ff048d3a in __lock_get_internal () from /lib64/libdb-5.3.so >> #4 0x00007f53ff049820 in __lock_get () from /lib64/libdb-5.3.so >> #5 0x00007f53ff075142 in __db_lget () from /lib64/libdb-5.3.so >> #6 0x00007f53fefbc605 in __bam_search () from /lib64/libdb-5.3.so >> #7 0x00007f53fefa7256 in __bamc_search () from /lib64/libdb-5.3.so >> #8 0x00007f53fefa8d0f in __bamc_get () from /lib64/libdb-5.3.so >> #9 0x00007f53ff061ca6 in __dbc_iget () from /lib64/libdb-5.3.so >> #10 0x00007f53ff070b02 in __dbc_get_pp () from /lib64/libdb-5.3.so >> #11 0x00007f53fa54c680 in idl_new_fetch () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #12 0x00007f53fa55ab26 in index_read_ext_allids () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #13 0x00007f53fa545284 in keys2idl () from /usr/lib64/dirsrv/plugins/libb >> ack-ldbm.so >> #14 0x00007f53fa5459e3 in ava_candidates.isra.0 () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #15 0x00007f53fa545fd2 in filter_candidates_ext () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #16 0x00007f53fa547096 in list_candidates () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #17 0x00007f53fa545f40 in filter_candidates_ext () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #18 0x00007f53fa58180c in subtree_candidates () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #19 0x00007f53fa582e7f in ldbm_back_search () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #20 0x00007f5406eb01d6 in op_shared_search () from >> /usr/lib64/dirsrv/libslapd.so.0 >> #21 0x00007f54073a133e in do_search () >> #22 0x00007f540738f567 in connection_threadmain () >> #23 0x00007f540509e7bb in _pt_root () from /lib64/libnspr4.so >> #24 0x00007f5404a3fdc5 in start_thread () from /lib64/libpthread.so.0 >> #25 0x00007f540476d28d in clone () from /lib64/libc.so.6 >> Thread 27 (Thread 0x7f53d7ff7700 (LWP 128219)): >> #0 0x00007f5404a436d5 in pthread_cond_wait@@GLIBC_2.3.2 () from >> /lib64/libpthread.so.0 >> #1 0x00007f53fef9f2f3 in __db_hybrid_mutex_suspend () from /lib64/ >> libdb-5.3.so >> #2 0x00007f53fef9e640 in __db_tas_mutex_lock () from /lib64/libdb-5.3.so >> #3 0x00007f53ff048d3a in __lock_get_internal () from /lib64/libdb-5.3.so >> #4 0x00007f53ff049820 in __lock_get () from /lib64/libdb-5.3.so >> #5 0x00007f53ff075142 in __db_lget () from /lib64/libdb-5.3.so >> #6 0x00007f53fefbc605 in __bam_search () from /lib64/libdb-5.3.so >> #7 0x00007f53fefa7256 in __bamc_search () from /lib64/libdb-5.3.so >> #8 0x00007f53fefa8d0f in __bamc_get () from /lib64/libdb-5.3.so >> #9 0x00007f53ff061ca6 in __dbc_iget () from /lib64/libdb-5.3.so >> #10 0x00007f53ff070b02 in __dbc_get_pp () from /lib64/libdb-5.3.so >> #11 0x00007f53fa54c680 in idl_new_fetch () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #12 0x00007f53fa55ab26 in index_read_ext_allids () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #13 0x00007f53fa545284 in keys2idl () from /usr/lib64/dirsrv/plugins/libb >> ack-ldbm.so >> #14 0x00007f53fa5459e3 in ava_candidates.isra.0 () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #15 0x00007f53fa545fd2 in filter_candidates_ext () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #16 0x00007f53fa547096 in list_candidates () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #17 0x00007f53fa545f40 in filter_candidates_ext () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #18 0x00007f53fa58180c in subtree_candidates () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #19 0x00007f53fa582e7f in ldbm_back_search () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #20 0x00007f5406eb01d6 in op_shared_search () from >> /usr/lib64/dirsrv/libslapd.so.0 >> #21 0x00007f54073a133e in do_search () >> #22 0x00007f540738f567 in connection_threadmain () >> #23 0x00007f540509e7bb in _pt_root () from /lib64/libnspr4.so >> #24 0x00007f5404a3fdc5 in start_thread () from /lib64/libpthread.so.0 >> #25 0x00007f540476d28d in clone () from /lib64/libc.so.6 >> Thread 26 (Thread 0x7f53d77f6700 (LWP 128220)): >> #0 0x00007f5404a436d5 in pthread_cond_wait@@GLIBC_2.3.2 () from >> /lib64/libpthread.so.0 >> #1 0x00007f53fef9f2f3 in __db_hybrid_mutex_suspend () from /lib64/ >> libdb-5.3.so >> #2 0x00007f53fef9e640 in __db_tas_mutex_lock () from /lib64/libdb-5.3.so >> #3 0x00007f53ff048d3a in __lock_get_internal () from /lib64/libdb-5.3.so >> #4 0x00007f53ff049820 in __lock_get () from /lib64/libdb-5.3.so >> #5 0x00007f53ff075142 in __db_lget () from /lib64/libdb-5.3.so >> #6 0x00007f53fefbc605 in __bam_search () from /lib64/libdb-5.3.so >> #7 0x00007f53fefa7256 in __bamc_search () from /lib64/libdb-5.3.so >> #8 0x00007f53fefa8d0f in __bamc_get () from /lib64/libdb-5.3.so >> #9 0x00007f53ff061ca6 in __dbc_iget () from /lib64/libdb-5.3.so >> #10 0x00007f53ff070b02 in __dbc_get_pp () from /lib64/libdb-5.3.so >> #11 0x00007f53fa54c680 in idl_new_fetch () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #12 0x00007f53fa55ab26 in index_read_ext_allids () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #13 0x00007f53fa545284 in keys2idl () from /usr/lib64/dirsrv/plugins/libb >> ack-ldbm.so >> #14 0x00007f53fa5459e3 in ava_candidates.isra.0 () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #15 0x00007f53fa545fd2 in filter_candidates_ext () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #16 0x00007f53fa547096 in list_candidates () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #17 0x00007f53fa545f40 in filter_candidates_ext () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #18 0x00007f53fa58180c in subtree_candidates () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #19 0x00007f53fa582e7f in ldbm_back_search () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #20 0x00007f5406eb01d6 in op_shared_search () from >> /usr/lib64/dirsrv/libslapd.so.0 >> #21 0x00007f54073a133e in do_search () >> #22 0x00007f540738f567 in connection_threadmain () >> #23 0x00007f540509e7bb in _pt_root () from /lib64/libnspr4.so >> #24 0x00007f5404a3fdc5 in start_thread () from /lib64/libpthread.so.0 >> #25 0x00007f540476d28d in clone () from /lib64/libc.so.6 >> Thread 25 (Thread 0x7f53d6ff5700 (LWP 128221)): >> #0 0x00007f5404a436d5 in pthread_cond_wait@@GLIBC_2.3.2 () from >> /lib64/libpthread.so.0 >> #1 0x00007f53fef9f2f3 in __db_hybrid_mutex_suspend () from /lib64/ >> libdb-5.3.so >> #2 0x00007f53fef9e640 in __db_tas_mutex_lock () from /lib64/libdb-5.3.so >> #3 0x00007f53ff048d3a in __lock_get_internal () from /lib64/libdb-5.3.so >> #4 0x00007f53ff049820 in __lock_get () from /lib64/libdb-5.3.so >> #5 0x00007f53ff075142 in __db_lget () from /lib64/libdb-5.3.so >> #6 0x00007f53fefbc605 in __bam_search () from /lib64/libdb-5.3.so >> #7 0x00007f53fefa7256 in __bamc_search () from /lib64/libdb-5.3.so >> #8 0x00007f53fefa8d0f in __bamc_get () from /lib64/libdb-5.3.so >> #9 0x00007f53ff061ca6 in __dbc_iget () from /lib64/libdb-5.3.so >> #10 0x00007f53ff070b02 in __dbc_get_pp () from /lib64/libdb-5.3.so >> #11 0x00007f53fa54c680 in idl_new_fetch () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #12 0x00007f53fa55ab26 in index_read_ext_allids () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #13 0x00007f53fa545284 in keys2idl () from /usr/lib64/dirsrv/plugins/libb >> ack-ldbm.so >> #14 0x00007f53fa5459e3 in ava_candidates.isra.0 () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #15 0x00007f53fa545fd2 in filter_candidates_ext () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #16 0x00007f53fa547096 in list_candidates () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #17 0x00007f53fa545f40 in filter_candidates_ext () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #18 0x00007f53fa58180c in subtree_candidates () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #19 0x00007f53fa582e7f in ldbm_back_search () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #20 0x00007f5406eb01d6 in op_shared_search () from >> /usr/lib64/dirsrv/libslapd.so.0 >> #21 0x00007f54073a133e in do_search () >> #22 0x00007f540738f567 in connection_threadmain () >> #23 0x00007f540509e7bb in _pt_root () from /lib64/libnspr4.so >> #24 0x00007f5404a3fdc5 in start_thread () from /lib64/libpthread.so.0 >> #25 0x00007f540476d28d in clone () from /lib64/libc.so.6 >> Thread 24 (Thread 0x7f53d67f4700 (LWP 128222)): >> #0 0x00007f5404a436d5 in pthread_cond_wait@@GLIBC_2.3.2 () from >> /lib64/libpthread.so.0 >> #1 0x00007f53fef9f2f3 in __db_hybrid_mutex_suspend () from /lib64/ >> libdb-5.3.so >> #2 0x00007f53fef9e640 in __db_tas_mutex_lock () from /lib64/libdb-5.3.so >> #3 0x00007f53ff048d3a in __lock_get_internal () from /lib64/libdb-5.3.so >> #4 0x00007f53ff049820 in __lock_get () from /lib64/libdb-5.3.so >> #5 0x00007f53ff075142 in __db_lget () from /lib64/libdb-5.3.so >> #6 0x00007f53fefbc605 in __bam_search () from /lib64/libdb-5.3.so >> #7 0x00007f53fefa7256 in __bamc_search () from /lib64/libdb-5.3.so >> #8 0x00007f53fefa8d0f in __bamc_get () from /lib64/libdb-5.3.so >> #9 0x00007f53ff061ca6 in __dbc_iget () from /lib64/libdb-5.3.so >> #10 0x00007f53ff070b02 in __dbc_get_pp () from /lib64/libdb-5.3.so >> #11 0x00007f53fa54c680 in idl_new_fetch () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #12 0x00007f53fa55ab26 in index_read_ext_allids () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #13 0x00007f53fa545284 in keys2idl () from /usr/lib64/dirsrv/plugins/libb >> ack-ldbm.so >> #14 0x00007f53fa5459e3 in ava_candidates.isra.0 () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #15 0x00007f53fa545fd2 in filter_candidates_ext () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #16 0x00007f53fa547096 in list_candidates () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #17 0x00007f53fa545f40 in filter_candidates_ext () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #18 0x00007f53fa58180c in subtree_candidates () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #19 0x00007f53fa582e7f in ldbm_back_search () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #20 0x00007f5406eb01d6 in op_shared_search () from >> /usr/lib64/dirsrv/libslapd.so.0 >> #21 0x00007f54073a133e in do_search () >> #22 0x00007f540738f567 in connection_threadmain () >> #23 0x00007f540509e7bb in _pt_root () from /lib64/libnspr4.so >> #24 0x00007f5404a3fdc5 in start_thread () from /lib64/libpthread.so.0 >> #25 0x00007f540476d28d in clone () from /lib64/libc.so.6 >> Thread 23 (Thread 0x7f53d5ff3700 (LWP 128223)): >> #0 0x00007f5404a436d5 in pthread_cond_wait@@GLIBC_2.3.2 () from >> /lib64/libpthread.so.0 >> #1 0x00007f53fef9f2f3 in __db_hybrid_mutex_suspend () from /lib64/ >> libdb-5.3.so >> #2 0x00007f53fef9e640 in __db_tas_mutex_lock () from /lib64/libdb-5.3.so >> #3 0x00007f53ff0b8e1b in __memp_fget () from /lib64/libdb-5.3.so >> #4 0x00007f53fefbc211 in __bam_search () from /lib64/libdb-5.3.so >> #5 0x00007f53fefa7256 in __bamc_search () from /lib64/libdb-5.3.so >> #6 0x00007f53fefab6c4 in __bamc_put () from /lib64/libdb-5.3.so >> #7 0x00007f53ff0626d5 in __dbc_iput () from /lib64/libdb-5.3.so >> #8 0x00007f53ff05da0e in __db_put () from /lib64/libdb-5.3.so >> #9 0x00007f53ff072fa4 in __db_put_pp () from /lib64/libdb-5.3.so >> #10 0x00007f53fa54d4cf in idl_new_insert_key () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #11 0x00007f53fa55c5b5 in addordel_values_sv.isra.3 () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #12 0x00007f53fa55d25a in index_addordel_values_ext_sv () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #13 0x00007f53fa55d694 in index_addordel_values_sv () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #14 0x00007f53fa55d75a in index_addordel_entry () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #15 0x00007f53fa5605fa in ldbm_back_add () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #16 0x00007f5406e62cca in op_shared_add () from >> /usr/lib64/dirsrv/libslapd.so.0 >> #17 0x00007f5406e64020 in do_add () from /usr/lib64/dirsrv/libslapd.so.0 >> #18 0x00007f540738f303 in connection_threadmain () >> #19 0x00007f540509e7bb in _pt_root () from /lib64/libnspr4.so >> #20 0x00007f5404a3fdc5 in start_thread () from /lib64/libpthread.so.0 >> #21 0x00007f540476d28d in clone () from /lib64/libc.so.6 >> Thread 22 (Thread 0x7f53d57f2700 (LWP 128224)): >> #0 0x00007f5404a436d5 in pthread_cond_wait@@GLIBC_2.3.2 () from >> /lib64/libpthread.so.0 >> #1 0x00007f53fef9f2f3 in __db_hybrid_mutex_suspend () from /lib64/ >> libdb-5.3.so >> #2 0x00007f53fef9e640 in __db_tas_mutex_lock () from /lib64/libdb-5.3.so >> #3 0x00007f53ff048d3a in __lock_get_internal () from /lib64/libdb-5.3.so >> #4 0x00007f53ff049820 in __lock_get () from /lib64/libdb-5.3.so >> #5 0x00007f53ff075142 in __db_lget () from /lib64/libdb-5.3.so >> #6 0x00007f53fefbc605 in __bam_search () from /lib64/libdb-5.3.so >> #7 0x00007f53fefa7256 in __bamc_search () from /lib64/libdb-5.3.so >> #8 0x00007f53fefa8d0f in __bamc_get () from /lib64/libdb-5.3.so >> #9 0x00007f53ff061ca6 in __dbc_iget () from /lib64/libdb-5.3.so >> #10 0x00007f53ff070b02 in __dbc_get_pp () from /lib64/libdb-5.3.so >> #11 0x00007f53fa54c680 in idl_new_fetch () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #12 0x00007f53fa55ab26 in index_read_ext_allids () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #13 0x00007f53fa545284 in keys2idl () from /usr/lib64/dirsrv/plugins/libb >> ack-ldbm.so >> #14 0x00007f53fa5459e3 in ava_candidates.isra.0 () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #15 0x00007f53fa545fd2 in filter_candidates_ext () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #16 0x00007f53fa547096 in list_candidates () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #17 0x00007f53fa545f40 in filter_candidates_ext () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #18 0x00007f53fa58180c in subtree_candidates () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #19 0x00007f53fa582e7f in ldbm_back_search () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #20 0x00007f5406eb01d6 in op_shared_search () from >> /usr/lib64/dirsrv/libslapd.so.0 >> #21 0x00007f54073a133e in do_search () >> #22 0x00007f540738f567 in connection_threadmain () >> #23 0x00007f540509e7bb in _pt_root () from /lib64/libnspr4.so >> #24 0x00007f5404a3fdc5 in start_thread () from /lib64/libpthread.so.0 >> #25 0x00007f540476d28d in clone () from /lib64/libc.so.6 >> Thread 21 (Thread 0x7f53d4ff1700 (LWP 128225)): >> #0 0x00007f5404a436d5 in pthread_cond_wait@@GLIBC_2.3.2 () from >> /lib64/libpthread.so.0 >> #1 0x00007f53fef9f2f3 in __db_hybrid_mutex_suspend () from /lib64/ >> libdb-5.3.so >> #2 0x00007f53fef9e640 in __db_tas_mutex_lock () from /lib64/libdb-5.3.so >> #3 0x00007f53ff048d3a in __lock_get_internal () from /lib64/libdb-5.3.so >> #4 0x00007f53ff049820 in __lock_get () from /lib64/libdb-5.3.so >> #5 0x00007f53ff075142 in __db_lget () from /lib64/libdb-5.3.so >> #6 0x00007f53fefbc605 in __bam_search () from /lib64/libdb-5.3.so >> #7 0x00007f53fefa7256 in __bamc_search () from /lib64/libdb-5.3.so >> #8 0x00007f53fefa8d0f in __bamc_get () from /lib64/libdb-5.3.so >> #9 0x00007f53ff061ca6 in __dbc_iget () from /lib64/libdb-5.3.so >> #10 0x00007f53ff070b02 in __dbc_get_pp () from /lib64/libdb-5.3.so >> #11 0x00007f53fa54c680 in idl_new_fetch () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #12 0x00007f53fa55ab26 in index_read_ext_allids () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #13 0x00007f53fa545284 in keys2idl () from /usr/lib64/dirsrv/plugins/libb >> ack-ldbm.so >> #14 0x00007f53fa5459e3 in ava_candidates.isra.0 () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #15 0x00007f53fa545fd2 in filter_candidates_ext () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #16 0x00007f53fa547096 in list_candidates () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #17 0x00007f53fa545f40 in filter_candidates_ext () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #18 0x00007f53fa58180c in subtree_candidates () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #19 0x00007f53fa582e7f in ldbm_back_search () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #20 0x00007f5406eb01d6 in op_shared_search () from >> /usr/lib64/dirsrv/libslapd.so.0 >> #21 0x00007f54073a133e in do_search () >> #22 0x00007f540738f567 in connection_threadmain () >> #23 0x00007f540509e7bb in _pt_root () from /lib64/libnspr4.so >> #24 0x00007f5404a3fdc5 in start_thread () from /lib64/libpthread.so.0 >> #25 0x00007f540476d28d in clone () from /lib64/libc.so.6 >> Thread 20 (Thread 0x7f53d47f0700 (LWP 128226)): >> #0 0x00007f5404a436d5 in pthread_cond_wait@@GLIBC_2.3.2 () from >> /lib64/libpthread.so.0 >> #1 0x00007f53fef9f2f3 in __db_hybrid_mutex_suspend () from /lib64/ >> libdb-5.3.so >> #2 0x00007f53fef9e640 in __db_tas_mutex_lock () from /lib64/libdb-5.3.so >> #3 0x00007f53ff048d3a in __lock_get_internal () from /lib64/libdb-5.3.so >> #4 0x00007f53ff049820 in __lock_get () from /lib64/libdb-5.3.so >> #5 0x00007f53ff075142 in __db_lget () from /lib64/libdb-5.3.so >> #6 0x00007f53fefbc605 in __bam_search () from /lib64/libdb-5.3.so >> #7 0x00007f53fefa7256 in __bamc_search () from /lib64/libdb-5.3.so >> #8 0x00007f53fefa8d0f in __bamc_get () from /lib64/libdb-5.3.so >> #9 0x00007f53ff061ca6 in __dbc_iget () from /lib64/libdb-5.3.so >> #10 0x00007f53ff070b02 in __dbc_get_pp () from /lib64/libdb-5.3.so >> #11 0x00007f53fa54c680 in idl_new_fetch () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #12 0x00007f53fa55ab26 in index_read_ext_allids () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #13 0x00007f53fa545284 in keys2idl () from /usr/lib64/dirsrv/plugins/libb >> ack-ldbm.so >> #14 0x00007f53fa5459e3 in ava_candidates.isra.0 () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #15 0x00007f53fa545fd2 in filter_candidates_ext () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #16 0x00007f53fa547096 in list_candidates () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #17 0x00007f53fa545f40 in filter_candidates_ext () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #18 0x00007f53fa58180c in subtree_candidates () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #19 0x00007f53fa582e7f in ldbm_back_search () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #20 0x00007f5406eb01d6 in op_shared_search () from >> /usr/lib64/dirsrv/libslapd.so.0 >> #21 0x00007f5406ec094e in search_internal_callback_pb () from >> /usr/lib64/dirsrv/libslapd.so.0 >> #22 0x00007f5406ec0be8 in search_internal_pb () from >> /usr/lib64/dirsrv/libslapd.so.0 >> #23 0x00007f53fe118e90 in search_one_berval () from >> /usr/lib64/dirsrv/plugins/libattr-unique-plugin.so >> #24 0x00007f53fe119242 in search () from /usr/lib64/dirsrv/plugins/liba >> ttr-unique-plugin.so >> #25 0x00007f53fe11954c in searchAllSubtrees () from >> /usr/lib64/dirsrv/plugins/libattr-unique-plugin.so >> #26 0x00007f53fe11a015 in preop_add () from /usr/lib64/dirsrv/plugins/liba >> ttr-unique-plugin.so >> #27 0x00007f5406ebb848 in plugin_call_func () from >> /usr/lib64/dirsrv/libslapd.so.0 >> #28 0x00007f5406ebbb13 in plugin_call_plugins () from >> /usr/lib64/dirsrv/libslapd.so.0 >> #29 0x00007f5406e62c2b in op_shared_add () from >> /usr/lib64/dirsrv/libslapd.so.0 >> #30 0x00007f5406e64020 in do_add () from /usr/lib64/dirsrv/libslapd.so.0 >> #31 0x00007f540738f303 in connection_threadmain () >> #32 0x00007f540509e7bb in _pt_root () from /lib64/libnspr4.so >> #33 0x00007f5404a3fdc5 in start_thread () from /lib64/libpthread.so.0 >> #34 0x00007f540476d28d in clone () from /lib64/libc.so.6 >> Thread 19 (Thread 0x7f53d3fef700 (LWP 128227)): >> #0 0x00007f5404a436d5 in pthread_cond_wait@@GLIBC_2.3.2 () from >> /lib64/libpthread.so.0 >> #1 0x00007f53fef9f2f3 in __db_hybrid_mutex_suspend () from /lib64/ >> libdb-5.3.so >> #2 0x00007f53fef9e640 in __db_tas_mutex_lock () from /lib64/libdb-5.3.so >> #3 0x00007f53ff048d3a in __lock_get_internal () from /lib64/libdb-5.3.so >> #4 0x00007f53ff049820 in __lock_get () from /lib64/libdb-5.3.so >> #5 0x00007f53ff075142 in __db_lget () from /lib64/libdb-5.3.so >> #6 0x00007f53fefbc605 in __bam_search () from /lib64/libdb-5.3.so >> #7 0x00007f53fefa7256 in __bamc_search () from /lib64/libdb-5.3.so >> #8 0x00007f53fefa8d0f in __bamc_get () from /lib64/libdb-5.3.so >> #9 0x00007f53ff061ca6 in __dbc_iget () from /lib64/libdb-5.3.so >> #10 0x00007f53ff070b02 in __dbc_get_pp () from /lib64/libdb-5.3.so >> #11 0x00007f53fa54c680 in idl_new_fetch () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #12 0x00007f53fa55ab26 in index_read_ext_allids () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #13 0x00007f53fa545284 in keys2idl () from /usr/lib64/dirsrv/plugins/libb >> ack-ldbm.so >> #14 0x00007f53fa5459e3 in ava_candidates.isra.0 () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #15 0x00007f53fa545fd2 in filter_candidates_ext () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #16 0x00007f53fa547096 in list_candidates () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #17 0x00007f53fa545f40 in filter_candidates_ext () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #18 0x00007f53fa58180c in subtree_candidates () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #19 0x00007f53fa582e7f in ldbm_back_search () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #20 0x00007f5406eb01d6 in op_shared_search () from >> /usr/lib64/dirsrv/libslapd.so.0 >> #21 0x00007f54073a133e in do_search () >> #22 0x00007f540738f567 in connection_threadmain () >> #23 0x00007f540509e7bb in _pt_root () from /lib64/libnspr4.so >> #24 0x00007f5404a3fdc5 in start_thread () from /lib64/libpthread.so.0 >> #25 0x00007f540476d28d in clone () from /lib64/libc.so.6 >> Thread 18 (Thread 0x7f53d37ee700 (LWP 128228)): >> #0 0x00007f5404a436d5 in pthread_cond_wait@@GLIBC_2.3.2 () from >> /lib64/libpthread.so.0 >> #1 0x00007f53fef9f2f3 in __db_hybrid_mutex_suspend () from /lib64/ >> libdb-5.3.so >> #2 0x00007f53fef9e640 in __db_tas_mutex_lock () from /lib64/libdb-5.3.so >> #3 0x00007f53ff048d3a in __lock_get_internal () from /lib64/libdb-5.3.so >> #4 0x00007f53ff049820 in __lock_get () from /lib64/libdb-5.3.so >> #5 0x00007f53ff075142 in __db_lget () from /lib64/libdb-5.3.so >> #6 0x00007f53fefbc605 in __bam_search () from /lib64/libdb-5.3.so >> #7 0x00007f53fefa7256 in __bamc_search () from /lib64/libdb-5.3.so >> #8 0x00007f53fefa8d0f in __bamc_get () from /lib64/libdb-5.3.so >> #9 0x00007f53ff061ca6 in __dbc_iget () from /lib64/libdb-5.3.so >> #10 0x00007f53ff070b02 in __dbc_get_pp () from /lib64/libdb-5.3.so >> #11 0x00007f53fa54c680 in idl_new_fetch () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #12 0x00007f53fa55ab26 in index_read_ext_allids () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #13 0x00007f53fa545284 in keys2idl () from /usr/lib64/dirsrv/plugins/libb >> ack-ldbm.so >> #14 0x00007f53fa5459e3 in ava_candidates.isra.0 () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #15 0x00007f53fa545fd2 in filter_candidates_ext () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #16 0x00007f53fa547096 in list_candidates () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #17 0x00007f53fa545f40 in filter_candidates_ext () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #18 0x00007f53fa58180c in subtree_candidates () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #19 0x00007f53fa582e7f in ldbm_back_search () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #20 0x00007f5406eb01d6 in op_shared_search () from >> /usr/lib64/dirsrv/libslapd.so.0 >> #21 0x00007f54073a133e in do_search () >> #22 0x00007f540738f567 in connection_threadmain () >> #23 0x00007f540509e7bb in _pt_root () from /lib64/libnspr4.so >> #24 0x00007f5404a3fdc5 in start_thread () from /lib64/libpthread.so.0 >> #25 0x00007f540476d28d in clone () from /lib64/libc.so.6 >> Thread 17 (Thread 0x7f53d2fed700 (LWP 128229)): >> #0 0x00007f5404a436d5 in pthread_cond_wait@@GLIBC_2.3.2 () from >> /lib64/libpthread.so.0 >> #1 0x00007f53fef9f2f3 in __db_hybrid_mutex_suspend () from /lib64/ >> libdb-5.3.so >> #2 0x00007f53fef9e640 in __db_tas_mutex_lock () from /lib64/libdb-5.3.so >> #3 0x00007f53ff048d3a in __lock_get_internal () from /lib64/libdb-5.3.so >> #4 0x00007f53ff049820 in __lock_get () from /lib64/libdb-5.3.so >> #5 0x00007f53ff075142 in __db_lget () from /lib64/libdb-5.3.so >> #6 0x00007f53fefbc605 in __bam_search () from /lib64/libdb-5.3.so >> #7 0x00007f53fefa7256 in __bamc_search () from /lib64/libdb-5.3.so >> #8 0x00007f53fefa8d0f in __bamc_get () from /lib64/libdb-5.3.so >> #9 0x00007f53ff061ca6 in __dbc_iget () from /lib64/libdb-5.3.so >> #10 0x00007f53ff070b02 in __dbc_get_pp () from /lib64/libdb-5.3.so >> #11 0x00007f53fa54c680 in idl_new_fetch () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #12 0x00007f53fa55ab26 in index_read_ext_allids () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #13 0x00007f53fa545284 in keys2idl () from /usr/lib64/dirsrv/plugins/libb >> ack-ldbm.so >> #14 0x00007f53fa5459e3 in ava_candidates.isra.0 () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #15 0x00007f53fa545fd2 in filter_candidates_ext () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #16 0x00007f53fa547096 in list_candidates () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #17 0x00007f53fa545f40 in filter_candidates_ext () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #18 0x00007f53fa58180c in subtree_candidates () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #19 0x00007f53fa582e7f in ldbm_back_search () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #20 0x00007f5406eb01d6 in op_shared_search () from >> /usr/lib64/dirsrv/libslapd.so.0 >> #21 0x00007f54073a133e in do_search () >> #22 0x00007f540738f567 in connection_threadmain () >> #23 0x00007f540509e7bb in _pt_root () from /lib64/libnspr4.so >> #24 0x00007f5404a3fdc5 in start_thread () from /lib64/libpthread.so.0 >> #25 0x00007f540476d28d in clone () from /lib64/libc.so.6 >> Thread 16 (Thread 0x7f53d27ec700 (LWP 128230)): >> #0 0x00007f5404a436d5 in pthread_cond_wait@@GLIBC_2.3.2 () from >> /lib64/libpthread.so.0 >> #1 0x00007f53fef9f2f3 in __db_hybrid_mutex_suspend () from /lib64/ >> libdb-5.3.so >> #2 0x00007f53fef9e640 in __db_tas_mutex_lock () from /lib64/libdb-5.3.so >> #3 0x00007f53ff048d3a in __lock_get_internal () from /lib64/libdb-5.3.so >> #4 0x00007f53ff049820 in __lock_get () from /lib64/libdb-5.3.so >> #5 0x00007f53ff075142 in __db_lget () from /lib64/libdb-5.3.so >> #6 0x00007f53fefbc605 in __bam_search () from /lib64/libdb-5.3.so >> #7 0x00007f53fefa7256 in __bamc_search () from /lib64/libdb-5.3.so >> #8 0x00007f53fefa8d0f in __bamc_get () from /lib64/libdb-5.3.so >> #9 0x00007f53ff061ca6 in __dbc_iget () from /lib64/libdb-5.3.so >> #10 0x00007f53ff070b02 in __dbc_get_pp () from /lib64/libdb-5.3.so >> #11 0x00007f53fa54c680 in idl_new_fetch () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #12 0x00007f53fa55ab26 in index_read_ext_allids () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #13 0x00007f53fa545284 in keys2idl () from /usr/lib64/dirsrv/plugins/libb >> ack-ldbm.so >> #14 0x00007f53fa5459e3 in ava_candidates.isra.0 () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #15 0x00007f53fa545fd2 in filter_candidates_ext () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #16 0x00007f53fa547096 in list_candidates () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #17 0x00007f53fa545f40 in filter_candidates_ext () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #18 0x00007f53fa58180c in subtree_candidates () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #19 0x00007f53fa582e7f in ldbm_back_search () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #20 0x00007f5406eb01d6 in op_shared_search () from >> /usr/lib64/dirsrv/libslapd.so.0 >> #21 0x00007f54073a133e in do_search () >> #22 0x00007f540738f567 in connection_threadmain () >> #23 0x00007f540509e7bb in _pt_root () from /lib64/libnspr4.so >> #24 0x00007f5404a3fdc5 in start_thread () from /lib64/libpthread.so.0 >> #25 0x00007f540476d28d in clone () from /lib64/libc.so.6 >> Thread 15 (Thread 0x7f53d1feb700 (LWP 128231)): >> #0 0x00007f5404a436d5 in pthread_cond_wait@@GLIBC_2.3.2 () from >> /lib64/libpthread.so.0 >> #1 0x00007f53fef9f2f3 in __db_hybrid_mutex_suspend () from /lib64/ >> libdb-5.3.so >> #2 0x00007f53fef9e640 in __db_tas_mutex_lock () from /lib64/libdb-5.3.so >> #3 0x00007f53ff048d3a in __lock_get_internal () from /lib64/libdb-5.3.so >> #4 0x00007f53ff049820 in __lock_get () from /lib64/libdb-5.3.so >> #5 0x00007f53ff075142 in __db_lget () from /lib64/libdb-5.3.so >> #6 0x00007f53fefbc605 in __bam_search () from /lib64/libdb-5.3.so >> #7 0x00007f53fefa7256 in __bamc_search () from /lib64/libdb-5.3.so >> #8 0x00007f53fefa8d0f in __bamc_get () from /lib64/libdb-5.3.so >> #9 0x00007f53ff061ca6 in __dbc_iget () from /lib64/libdb-5.3.so >> #10 0x00007f53ff070b02 in __dbc_get_pp () from /lib64/libdb-5.3.so >> #11 0x00007f53fa54c680 in idl_new_fetch () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #12 0x00007f53fa55ab26 in index_read_ext_allids () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #13 0x00007f53fa545284 in keys2idl () from /usr/lib64/dirsrv/plugins/libb >> ack-ldbm.so >> #14 0x00007f53fa5459e3 in ava_candidates.isra.0 () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #15 0x00007f53fa545fd2 in filter_candidates_ext () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #16 0x00007f53fa547096 in list_candidates () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #17 0x00007f53fa545f40 in filter_candidates_ext () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #18 0x00007f53fa58180c in subtree_candidates () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #19 0x00007f53fa582e7f in ldbm_back_search () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #20 0x00007f5406eb01d6 in op_shared_search () from >> /usr/lib64/dirsrv/libslapd.so.0 >> #21 0x00007f54073a133e in do_search () >> #22 0x00007f540738f567 in connection_threadmain () >> #23 0x00007f540509e7bb in _pt_root () from /lib64/libnspr4.so >> #24 0x00007f5404a3fdc5 in start_thread () from /lib64/libpthread.so.0 >> #25 0x00007f540476d28d in clone () from /lib64/libc.so.6 >> Thread 14 (Thread 0x7f53d17ea700 (LWP 128232)): >> #0 0x00007f5404a436d5 in pthread_cond_wait@@GLIBC_2.3.2 () from >> /lib64/libpthread.so.0 >> #1 0x00007f53fef9f2f3 in __db_hybrid_mutex_suspend () from /lib64/ >> libdb-5.3.so >> #2 0x00007f53fef9e640 in __db_tas_mutex_lock () from /lib64/libdb-5.3.so >> #3 0x00007f53ff048d3a in __lock_get_internal () from /lib64/libdb-5.3.so >> #4 0x00007f53ff049820 in __lock_get () from /lib64/libdb-5.3.so >> #5 0x00007f53ff075142 in __db_lget () from /lib64/libdb-5.3.so >> #6 0x00007f53fefbc605 in __bam_search () from /lib64/libdb-5.3.so >> #7 0x00007f53fefa7256 in __bamc_search () from /lib64/libdb-5.3.so >> #8 0x00007f53fefa8d0f in __bamc_get () from /lib64/libdb-5.3.so >> #9 0x00007f53ff061ca6 in __dbc_iget () from /lib64/libdb-5.3.so >> #10 0x00007f53ff070b02 in __dbc_get_pp () from /lib64/libdb-5.3.so >> #11 0x00007f53fa54c680 in idl_new_fetch () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #12 0x00007f53fa55ab26 in index_read_ext_allids () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #13 0x00007f53fa545284 in keys2idl () from /usr/lib64/dirsrv/plugins/libb >> ack-ldbm.so >> #14 0x00007f53fa5459e3 in ava_candidates.isra.0 () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #15 0x00007f53fa545fd2 in filter_candidates_ext () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #16 0x00007f53fa547096 in list_candidates () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #17 0x00007f53fa545f40 in filter_candidates_ext () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #18 0x00007f53fa547096 in list_candidates () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #19 0x00007f53fa545f40 in filter_candidates_ext () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #20 0x00007f53fa547096 in list_candidates () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #21 0x00007f53fa545f40 in filter_candidates_ext () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #22 0x00007f53fa58180c in subtree_candidates () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #23 0x00007f53fa582e7f in ldbm_back_search () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #24 0x00007f5406eb01d6 in op_shared_search () from >> /usr/lib64/dirsrv/libslapd.so.0 >> #25 0x00007f54073a133e in do_search () >> #26 0x00007f540738f3c3 in connection_threadmain () >> #27 0x00007f540509e7bb in _pt_root () from /lib64/libnspr4.so >> #28 0x00007f5404a3fdc5 in start_thread () from /lib64/libpthread.so.0 >> #29 0x00007f540476d28d in clone () from /lib64/libc.so.6 >> Thread 13 (Thread 0x7f53d0fe9700 (LWP 128233)): >> #0 0x00007f5404a436d5 in pthread_cond_wait@@GLIBC_2.3.2 () from >> /lib64/libpthread.so.0 >> #1 0x00007f53fef9f2f3 in __db_hybrid_mutex_suspend () from /lib64/ >> libdb-5.3.so >> #2 0x00007f53fef9e640 in __db_tas_mutex_lock () from /lib64/libdb-5.3.so >> #3 0x00007f53ff048d3a in __lock_get_internal () from /lib64/libdb-5.3.so >> #4 0x00007f53ff049820 in __lock_get () from /lib64/libdb-5.3.so >> #5 0x00007f53ff075142 in __db_lget () from /lib64/libdb-5.3.so >> #6 0x00007f53fefbc605 in __bam_search () from /lib64/libdb-5.3.so >> #7 0x00007f53fefa7256 in __bamc_search () from /lib64/libdb-5.3.so >> #8 0x00007f53fefa8d0f in __bamc_get () from /lib64/libdb-5.3.so >> #9 0x00007f53ff061ca6 in __dbc_iget () from /lib64/libdb-5.3.so >> #10 0x00007f53ff070b02 in __dbc_get_pp () from /lib64/libdb-5.3.so >> #11 0x00007f53fa54c680 in idl_new_fetch () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #12 0x00007f53fa55ab26 in index_read_ext_allids () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #13 0x00007f53fa545284 in keys2idl () from /usr/lib64/dirsrv/plugins/libb >> ack-ldbm.so >> #14 0x00007f53fa5459e3 in ava_candidates.isra.0 () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #15 0x00007f53fa545fd2 in filter_candidates_ext () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #16 0x00007f53fa547096 in list_candidates () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #17 0x00007f53fa545f40 in filter_candidates_ext () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #18 0x00007f53fa58180c in subtree_candidates () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #19 0x00007f53fa582e7f in ldbm_back_search () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #20 0x00007f5406eb01d6 in op_shared_search () from >> /usr/lib64/dirsrv/libslapd.so.0 >> #21 0x00007f54073a133e in do_search () >> #22 0x00007f540738f567 in connection_threadmain () >> #23 0x00007f540509e7bb in _pt_root () from /lib64/libnspr4.so >> #24 0x00007f5404a3fdc5 in start_thread () from /lib64/libpthread.so.0 >> #25 0x00007f540476d28d in clone () from /lib64/libc.so.6 >> Thread 12 (Thread 0x7f53d07e8700 (LWP 128234)): >> #0 0x00007f5404a436d5 in pthread_cond_wait@@GLIBC_2.3.2 () from >> /lib64/libpthread.so.0 >> #1 0x00007f53fef9f2f3 in __db_hybrid_mutex_suspend () from /lib64/ >> libdb-5.3.so >> #2 0x00007f53fef9e640 in __db_tas_mutex_lock () from /lib64/libdb-5.3.so >> #3 0x00007f53ff048d3a in __lock_get_internal () from /lib64/libdb-5.3.so >> #4 0x00007f53ff049820 in __lock_get () from /lib64/libdb-5.3.so >> #5 0x00007f53ff075142 in __db_lget () from /lib64/libdb-5.3.so >> #6 0x00007f53fefbc605 in __bam_search () from /lib64/libdb-5.3.so >> #7 0x00007f53fefa7256 in __bamc_search () from /lib64/libdb-5.3.so >> #8 0x00007f53fefa8d0f in __bamc_get () from /lib64/libdb-5.3.so >> #9 0x00007f53ff061ca6 in __dbc_iget () from /lib64/libdb-5.3.so >> #10 0x00007f53ff070b02 in __dbc_get_pp () from /lib64/libdb-5.3.so >> #11 0x00007f53fa54c680 in idl_new_fetch () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #12 0x00007f53fa55ab26 in index_read_ext_allids () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #13 0x00007f53fa545284 in keys2idl () from /usr/lib64/dirsrv/plugins/libb >> ack-ldbm.so >> #14 0x00007f53fa5459e3 in ava_candidates.isra.0 () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #15 0x00007f53fa545fd2 in filter_candidates_ext () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #16 0x00007f53fa547096 in list_candidates () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #17 0x00007f53fa545f40 in filter_candidates_ext () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #18 0x00007f53fa58180c in subtree_candidates () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #19 0x00007f53fa582e7f in ldbm_back_search () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #20 0x00007f5406eb01d6 in op_shared_search () from >> /usr/lib64/dirsrv/libslapd.so.0 >> #21 0x00007f54073a133e in do_search () >> #22 0x00007f540738f567 in connection_threadmain () >> #23 0x00007f540509e7bb in _pt_root () from /lib64/libnspr4.so >> #24 0x00007f5404a3fdc5 in start_thread () from /lib64/libpthread.so.0 >> #25 0x00007f540476d28d in clone () from /lib64/libc.so.6 >> Thread 11 (Thread 0x7f53cffe7700 (LWP 128235)): >> #0 0x00007f5404a436d5 in pthread_cond_wait@@GLIBC_2.3.2 () from >> /lib64/libpthread.so.0 >> #1 0x00007f53fef9f2f3 in __db_hybrid_mutex_suspend () from /lib64/ >> libdb-5.3.so >> #2 0x00007f53fef9e640 in __db_tas_mutex_lock () from /lib64/libdb-5.3.so >> #3 0x00007f53ff048d3a in __lock_get_internal () from /lib64/libdb-5.3.so >> #4 0x00007f53ff049820 in __lock_get () from /lib64/libdb-5.3.so >> #5 0x00007f53ff075142 in __db_lget () from /lib64/libdb-5.3.so >> #6 0x00007f53fefbc605 in __bam_search () from /lib64/libdb-5.3.so >> #7 0x00007f53fefa7256 in __bamc_search () from /lib64/libdb-5.3.so >> #8 0x00007f53fefa8d0f in __bamc_get () from /lib64/libdb-5.3.so >> #9 0x00007f53ff061ca6 in __dbc_iget () from /lib64/libdb-5.3.so >> #10 0x00007f53ff070b02 in __dbc_get_pp () from /lib64/libdb-5.3.so >> #11 0x00007f53fa54c680 in idl_new_fetch () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #12 0x00007f53fa55ab26 in index_read_ext_allids () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #13 0x00007f53fa545284 in keys2idl () from /usr/lib64/dirsrv/plugins/libb >> ack-ldbm.so >> #14 0x00007f53fa5459e3 in ava_candidates.isra.0 () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #15 0x00007f53fa545fd2 in filter_candidates_ext () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #16 0x00007f53fa547096 in list_candidates () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #17 0x00007f53fa545f40 in filter_candidates_ext () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #18 0x00007f53fa58180c in subtree_candidates () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #19 0x00007f53fa582e7f in ldbm_back_search () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #20 0x00007f5406eb01d6 in op_shared_search () from >> /usr/lib64/dirsrv/libslapd.so.0 >> #21 0x00007f54073a133e in do_search () >> #22 0x00007f540738f567 in connection_threadmain () >> #23 0x00007f540509e7bb in _pt_root () from /lib64/libnspr4.so >> #24 0x00007f5404a3fdc5 in start_thread () from /lib64/libpthread.so.0 >> #25 0x00007f540476d28d in clone () from /lib64/libc.so.6 >> Thread 10 (Thread 0x7f53cf7e6700 (LWP 128236)): >> #0 0x00007f5404a436d5 in pthread_cond_wait@@GLIBC_2.3.2 () from >> /lib64/libpthread.so.0 >> #1 0x00007f53fef9f2f3 in __db_hybrid_mutex_suspend () from /lib64/ >> libdb-5.3.so >> #2 0x00007f53fef9e640 in __db_tas_mutex_lock () from /lib64/libdb-5.3.so >> #3 0x00007f53ff048d3a in __lock_get_internal () from /lib64/libdb-5.3.so >> #4 0x00007f53ff049820 in __lock_get () from /lib64/libdb-5.3.so >> #5 0x00007f53ff075142 in __db_lget () from /lib64/libdb-5.3.so >> #6 0x00007f53fefbc605 in __bam_search () from /lib64/libdb-5.3.so >> #7 0x00007f53fefa7256 in __bamc_search () from /lib64/libdb-5.3.so >> #8 0x00007f53fefa8d0f in __bamc_get () from /lib64/libdb-5.3.so >> #9 0x00007f53ff061ca6 in __dbc_iget () from /lib64/libdb-5.3.so >> #10 0x00007f53ff070b02 in __dbc_get_pp () from /lib64/libdb-5.3.so >> #11 0x00007f53fa54c680 in idl_new_fetch () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #12 0x00007f53fa55ab26 in index_read_ext_allids () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #13 0x00007f53fa545284 in keys2idl () from /usr/lib64/dirsrv/plugins/libb >> ack-ldbm.so >> #14 0x00007f53fa5459e3 in ava_candidates.isra.0 () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #15 0x00007f53fa545fd2 in filter_candidates_ext () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #16 0x00007f53fa547096 in list_candidates () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #17 0x00007f53fa545f40 in filter_candidates_ext () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #18 0x00007f53fa58180c in subtree_candidates () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #19 0x00007f53fa582e7f in ldbm_back_search () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #20 0x00007f5406eb01d6 in op_shared_search () from >> /usr/lib64/dirsrv/libslapd.so.0 >> #21 0x00007f54073a133e in do_search () >> #22 0x00007f540738f567 in connection_threadmain () >> #23 0x00007f540509e7bb in _pt_root () from /lib64/libnspr4.so >> #24 0x00007f5404a3fdc5 in start_thread () from /lib64/libpthread.so.0 >> #25 0x00007f540476d28d in clone () from /lib64/libc.so.6 >> Thread 9 (Thread 0x7f53cefe5700 (LWP 128237)): >> #0 0x00007f5404a436d5 in pthread_cond_wait@@GLIBC_2.3.2 () from >> /lib64/libpthread.so.0 >> #1 0x00007f53fef9f2f3 in __db_hybrid_mutex_suspend () from /lib64/ >> libdb-5.3.so >> #2 0x00007f53fef9e640 in __db_tas_mutex_lock () from /lib64/libdb-5.3.so >> #3 0x00007f53ff048d3a in __lock_get_internal () from /lib64/libdb-5.3.so >> #4 0x00007f53ff049820 in __lock_get () from /lib64/libdb-5.3.so >> #5 0x00007f53ff075142 in __db_lget () from /lib64/libdb-5.3.so >> #6 0x00007f53fefbc605 in __bam_search () from /lib64/libdb-5.3.so >> #7 0x00007f53fefa7256 in __bamc_search () from /lib64/libdb-5.3.so >> #8 0x00007f53fefa8d0f in __bamc_get () from /lib64/libdb-5.3.so >> #9 0x00007f53ff061ca6 in __dbc_iget () from /lib64/libdb-5.3.so >> #10 0x00007f53ff070b02 in __dbc_get_pp () from /lib64/libdb-5.3.so >> #11 0x00007f53fa54c680 in idl_new_fetch () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #12 0x00007f53fa55ab26 in index_read_ext_allids () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #13 0x00007f53fa545284 in keys2idl () from /usr/lib64/dirsrv/plugins/libb >> ack-ldbm.so >> #14 0x00007f53fa5459e3 in ava_candidates.isra.0 () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #15 0x00007f53fa545fd2 in filter_candidates_ext () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #16 0x00007f53fa547096 in list_candidates () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #17 0x00007f53fa545f40 in filter_candidates_ext () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #18 0x00007f53fa58180c in subtree_candidates () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #19 0x00007f53fa582e7f in ldbm_back_search () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #20 0x00007f5406eb01d6 in op_shared_search () from >> /usr/lib64/dirsrv/libslapd.so.0 >> #21 0x00007f54073a133e in do_search () >> #22 0x00007f540738f567 in connection_threadmain () >> #23 0x00007f540509e7bb in _pt_root () from /lib64/libnspr4.so >> #24 0x00007f5404a3fdc5 in start_thread () from /lib64/libpthread.so.0 >> #25 0x00007f540476d28d in clone () from /lib64/libc.so.6 >> Thread 8 (Thread 0x7f53ce7e4700 (LWP 128238)): >> #0 0x00007f5404a436d5 in pthread_cond_wait@@GLIBC_2.3.2 () from >> /lib64/libpthread.so.0 >> #1 0x00007f53fef9f2f3 in __db_hybrid_mutex_suspend () from /lib64/ >> libdb-5.3.so >> #2 0x00007f53fef9e640 in __db_tas_mutex_lock () from /lib64/libdb-5.3.so >> #3 0x00007f53ff048d3a in __lock_get_internal () from /lib64/libdb-5.3.so >> #4 0x00007f53ff049820 in __lock_get () from /lib64/libdb-5.3.so >> #5 0x00007f53ff075142 in __db_lget () from /lib64/libdb-5.3.so >> #6 0x00007f53fefbc605 in __bam_search () from /lib64/libdb-5.3.so >> #7 0x00007f53fefa7256 in __bamc_search () from /lib64/libdb-5.3.so >> #8 0x00007f53fefa8d0f in __bamc_get () from /lib64/libdb-5.3.so >> #9 0x00007f53ff061ca6 in __dbc_iget () from /lib64/libdb-5.3.so >> #10 0x00007f53ff070b02 in __dbc_get_pp () from /lib64/libdb-5.3.so >> #11 0x00007f53fa54c680 in idl_new_fetch () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #12 0x00007f53fa55ab26 in index_read_ext_allids () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #13 0x00007f53fa545284 in keys2idl () from /usr/lib64/dirsrv/plugins/libb >> ack-ldbm.so >> #14 0x00007f53fa5459e3 in ava_candidates.isra.0 () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #15 0x00007f53fa545fd2 in filter_candidates_ext () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #16 0x00007f53fa547096 in list_candidates () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #17 0x00007f53fa545f40 in filter_candidates_ext () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #18 0x00007f53fa58180c in subtree_candidates () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #19 0x00007f53fa582e7f in ldbm_back_search () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #20 0x00007f5406eb01d6 in op_shared_search () from >> /usr/lib64/dirsrv/libslapd.so.0 >> #21 0x00007f54073a133e in do_search () >> #22 0x00007f540738f567 in connection_threadmain () >> #23 0x00007f540509e7bb in _pt_root () from /lib64/libnspr4.so >> #24 0x00007f5404a3fdc5 in start_thread () from /lib64/libpthread.so.0 >> #25 0x00007f540476d28d in clone () from /lib64/libc.so.6 >> Thread 7 (Thread 0x7f53cdfe3700 (LWP 128239)): >> #0 0x00007f5404a436d5 in pthread_cond_wait@@GLIBC_2.3.2 () from >> /lib64/libpthread.so.0 >> #1 0x00007f53fef9f2f3 in __db_hybrid_mutex_suspend () from /lib64/ >> libdb-5.3.so >> #2 0x00007f53fef9e640 in __db_tas_mutex_lock () from /lib64/libdb-5.3.so >> #3 0x00007f53ff048d3a in __lock_get_internal () from /lib64/libdb-5.3.so >> #4 0x00007f53ff049820 in __lock_get () from /lib64/libdb-5.3.so >> #5 0x00007f53ff075142 in __db_lget () from /lib64/libdb-5.3.so >> #6 0x00007f53fefbc605 in __bam_search () from /lib64/libdb-5.3.so >> #7 0x00007f53fefa7256 in __bamc_search () from /lib64/libdb-5.3.so >> #8 0x00007f53fefa8d0f in __bamc_get () from /lib64/libdb-5.3.so >> #9 0x00007f53ff061ca6 in __dbc_iget () from /lib64/libdb-5.3.so >> #10 0x00007f53ff070b02 in __dbc_get_pp () from /lib64/libdb-5.3.so >> #11 0x00007f53fa54c680 in idl_new_fetch () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #12 0x00007f53fa55ab26 in index_read_ext_allids () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #13 0x00007f53fa545284 in keys2idl () from /usr/lib64/dirsrv/plugins/libb >> ack-ldbm.so >> #14 0x00007f53fa5459e3 in ava_candidates.isra.0 () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #15 0x00007f53fa545fd2 in filter_candidates_ext () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #16 0x00007f53fa547096 in list_candidates () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #17 0x00007f53fa545f40 in filter_candidates_ext () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #18 0x00007f53fa58180c in subtree_candidates () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #19 0x00007f53fa582e7f in ldbm_back_search () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #20 0x00007f5406eb01d6 in op_shared_search () from >> /usr/lib64/dirsrv/libslapd.so.0 >> #21 0x00007f54073a133e in do_search () >> #22 0x00007f540738f567 in connection_threadmain () >> #23 0x00007f540509e7bb in _pt_root () from /lib64/libnspr4.so >> #24 0x00007f5404a3fdc5 in start_thread () from /lib64/libpthread.so.0 >> #25 0x00007f540476d28d in clone () from /lib64/libc.so.6 >> Thread 6 (Thread 0x7f53cd7e2700 (LWP 128240)): >> #0 0x00007f5404a436d5 in pthread_cond_wait@@GLIBC_2.3.2 () from >> /lib64/libpthread.so.0 >> #1 0x00007f53fef9f2f3 in __db_hybrid_mutex_suspend () from /lib64/ >> libdb-5.3.so >> #2 0x00007f53fef9eb37 in __db_tas_mutex_readlock () from /lib64/ >> libdb-5.3.so >> #3 0x00007f53ff0b8d35 in __memp_fget () from /lib64/libdb-5.3.so >> #4 0x00007f53ff062333 in __dbc_iget () from /lib64/libdb-5.3.so >> #5 0x00007f53ff070b02 in __dbc_get_pp () from /lib64/libdb-5.3.so >> #6 0x00007f53fa54c822 in idl_new_fetch () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #7 0x00007f53fa55ab26 in index_read_ext_allids () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #8 0x00007f53fa545284 in keys2idl () from /usr/lib64/dirsrv/plugins/libb >> ack-ldbm.so >> #9 0x00007f53fa5459e3 in ava_candidates.isra.0 () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #10 0x00007f53fa545fd2 in filter_candidates_ext () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #11 0x00007f53fa547096 in list_candidates () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #12 0x00007f53fa545f40 in filter_candidates_ext () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #13 0x00007f53fa547096 in list_candidates () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #14 0x00007f53fa545f40 in filter_candidates_ext () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #15 0x00007f53fa547096 in list_candidates () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #16 0x00007f53fa545f40 in filter_candidates_ext () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #17 0x00007f53fa58180c in subtree_candidates () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #18 0x00007f53fa582e7f in ldbm_back_search () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #19 0x00007f5406eb01d6 in op_shared_search () from >> /usr/lib64/dirsrv/libslapd.so.0 >> #20 0x00007f54073a133e in do_search () >> #21 0x00007f540738f3c3 in connection_threadmain () >> #22 0x00007f540509e7bb in _pt_root () from /lib64/libnspr4.so >> #23 0x00007f5404a3fdc5 in start_thread () from /lib64/libpthread.so.0 >> #24 0x00007f540476d28d in clone () from /lib64/libc.so.6 >> Thread 5 (Thread 0x7f53ccfe1700 (LWP 128241)): >> #0 0x00007f5404a436d5 in pthread_cond_wait@@GLIBC_2.3.2 () from >> /lib64/libpthread.so.0 >> #1 0x00007f53fef9f2f3 in __db_hybrid_mutex_suspend () from /lib64/ >> libdb-5.3.so >> #2 0x00007f53fef9e640 in __db_tas_mutex_lock () from /lib64/libdb-5.3.so >> #3 0x00007f53ff048d3a in __lock_get_internal () from /lib64/libdb-5.3.so >> #4 0x00007f53ff049820 in __lock_get () from /lib64/libdb-5.3.so >> #5 0x00007f53ff075142 in __db_lget () from /lib64/libdb-5.3.so >> #6 0x00007f53fefbc605 in __bam_search () from /lib64/libdb-5.3.so >> #7 0x00007f53fefa7256 in __bamc_search () from /lib64/libdb-5.3.so >> #8 0x00007f53fefa8d0f in __bamc_get () from /lib64/libdb-5.3.so >> #9 0x00007f53ff061ca6 in __dbc_iget () from /lib64/libdb-5.3.so >> #10 0x00007f53ff070b02 in __dbc_get_pp () from /lib64/libdb-5.3.so >> #11 0x00007f53fa54c680 in idl_new_fetch () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #12 0x00007f53fa55ab26 in index_read_ext_allids () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #13 0x00007f53fa545284 in keys2idl () from /usr/lib64/dirsrv/plugins/libb >> ack-ldbm.so >> #14 0x00007f53fa5459e3 in ava_candidates.isra.0 () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #15 0x00007f53fa545fd2 in filter_candidates_ext () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #16 0x00007f53fa547096 in list_candidates () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #17 0x00007f53fa545f40 in filter_candidates_ext () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #18 0x00007f53fa547096 in list_candidates () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #19 0x00007f53fa545f40 in filter_candidates_ext () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #20 0x00007f53fa547096 in list_candidates () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #21 0x00007f53fa545f40 in filter_candidates_ext () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #22 0x00007f53fa58180c in subtree_candidates () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #23 0x00007f53fa582e7f in ldbm_back_search () from >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #24 0x00007f5406eb01d6 in op_shared_search () from >> /usr/lib64/dirsrv/libslapd.so.0 >> #25 0x00007f54073a133e in do_search () >> #26 0x00007f540738f3c3 in connection_threadmain () >> #27 0x00007f540509e7bb in _pt_root () from /lib64/libnspr4.so >> #28 0x00007f5404a3fdc5 in start_thread () from /lib64/libpthread.so.0 >> #29 0x00007f540476d28d in clone () from /lib64/libc.so.6 >> Thread 4 (Thread 0x7f53cc7e0700 (LWP 128242)): >> #0 0x00007f54047649b3 in select () from /lib64/libc.so.6 >> #1 0x00007f5406eef0e9 in DS_Sleep () from /usr/lib64/dirsrv/libslapd.so. >> 0 >> #2 0x00007f5407390325 in time_thread () >> #3 0x00007f540509e7bb in _pt_root () from /lib64/libnspr4.so >> #4 0x00007f5404a3fdc5 in start_thread () from /lib64/libpthread.so.0 >> #5 0x00007f540476d28d in clone () from /lib64/libc.so.6 >> Thread 3 (Thread 0x7f53cb7a0700 (LWP 128259)): >> #0 0x00007f5404a436d5 in pthread_cond_wait@@GLIBC_2.3.2 () from >> /lib64/libpthread.so.0 >> #1 0x00007f5405099050 in PR_WaitCondVar () from /lib64/libnspr4.so >> #2 0x00007f540739bf25 in ps_send_results () >> #3 0x00007f540509e7bb in _pt_root () from /lib64/libnspr4.so >> #4 0x00007f5404a3fdc5 in start_thread () from /lib64/libpthread.so.0 >> #5 0x00007f540476d28d in clone () from /lib64/libc.so.6 >> Thread 2 (Thread 0x7f53cab9d700 (LWP 9020)): >> #0 0x00007f5404a436d5 in pthread_cond_wait@@GLIBC_2.3.2 () from >> /lib64/libpthread.so.0 >> #1 0x00007f5405099050 in PR_WaitCondVar () from /lib64/libnspr4.so >> #2 0x00007f540739bf25 in ps_send_results () >> #3 0x00007f540509e7bb in _pt_root () from /lib64/libnspr4.so >> #4 0x00007f5404a3fdc5 in start_thread () from /lib64/libpthread.so.0 >> #5 0x00007f540476d28d in clone () from /lib64/libc.so.6 >> Thread 1 (Thread 0x7f5407357840 (LWP 128124)): >> #0 0x00007f5404762c3d in poll () from /lib64/libc.so.6 >> #1 0x00007f540509a967 in _pr_poll_with_poll () from /lib64/libnspr4.so >> #2 0x00007f5407393ff1 in slapd_daemon () >> #3 0x00007f5407385c6c in main () >> >> >> Thanks >> >> On Tue, Sep 6, 2016 at 5:51 PM, Rakesh Rajasekharan < >> rakesh.rajasekharan at gmail.com> wrote: >> >>> Hi, >>> >>> With script provided by Rich, I looked at my db cache usage and it does >>> not look to be high enough >>> >>> # under each db are the list of selected indexes specified with INDEXLIST >>> dbname count free free% size >>> changelog:ent 5 2076333 99.0 4163.8 >>> changelog:dn 15976 9303557 88.7 74.0 >>> userroot:ent 4763 175321074 83.6 7221.1 >>> userroot:dn 4763 9995056 95.3 103.0 >>> ipaca:ent 425 6337866 60.4 9759.8 >>> ipaca:dn 425 10451081 99.7 81.6 >>> >>> Right now I set it to 200MB. So, I am planning to bring it down to 100MB >>> >>> Also, *nsslapd-db-checkpoint-interval *will change it to 80 or 90 from >>> default of 60 >>> >>> I will check this out and the impact it has on the hang issues and get >>> back >>> >>> Thanks >>> Rakesh >>> >>> On Tue, Sep 6, 2016 at 4:13 PM, Ludwig Krispenz < >>> lkrispen at redhat.com> wrote: >>> >>>> Hi, >>>> >>>> I think you are running into this bug: >>>> https://bugzilla.redhat.com/show_bug.cgi?id=1349779 >>>> it was the outcome of an analysis started in this one: >>>> https://bugzilla.redhat.com/show_bug.cgi?id=1347800 >>>> >>>> the problem is that it looks like the checkpoint thread is holding a >>>> mutex and otehr threads are blocked on it. It looks like an issue in >>>> BerkeleyDB although there is no explanation from the BDB code why this >>>> could happen, we also had a support case with Oracle to investigate this, >>>> with no solution. Oracle at the end was providing a diagnostic patch, which >>>> we did not test since the customer with the original problem had worked >>>> around this by changing their configuration. >>>> I don't know the status of this patch and if we can make it available, >>>> probably depends on your support status, you need to contact support for >>>> this. >>>> >>>> About the workarounds: >>>> you can try to change timing for the database update and checkpointing >>>> eg by reducing the dbcache size, changing the checkpoint interval. In the >>>> customer case they identified one substring index, which was frequently >>>> updated but not really required - and removing this index helped. >>>> >>>> Regards, >>>> Ludwig >>>> >>>> On 09/06/2016 11:31 AM, thierry bordaz wrote: >>>> >>>> Hi, >>>> >>>> The following thread is hanging most of the others threads >>>> >>>> Thread 20 (Thread 0x7f68007f8700 (LWP 4873)): >>>> >>>> #0 0x00007f683d0506d5 in >>>> pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 >>>> #1 0x00007f68375ac2f3 in __db_hybrid_mutex_suspend () from /lib64/ >>>> libdb-5.3.so >>>> #2 0x00007f68375ab640 in __db_tas_mutex_lock () from /lib64/ >>>> libdb-5.3.so >>>> #3 0x00007f68376c5e1b in __memp_fget () from /lib64/libdb-5.3.so >>>> #4 0x00007f68375c9211 in __bam_search () from /lib64/libdb-5.3.so >>>> #5 0x00007f68375b4256 in __bamc_search () from /lib64/libdb-5.3.so >>>> #6 0x00007f68375b86c4 in __bamc_put () from /lib64/libdb-5.3.so >>>> #7 0x00007f683766f6d5 in __dbc_iput () from /lib64/libdb-5.3.so >>>> #8 0x00007f683766aa0e in __db_put () from /lib64/libdb-5.3.so >>>> #9 0x00007f683767ffa4 in __db_put_pp () from /lib64/libdb-5.3.so >>>> #10 0x00007f6832b5a4cf in idl_new_insert_key () from >>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>> #11 0x00007f6832b695b5 in addordel_values_sv.isra.3 () from >>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>> #12 0x00007f6832b6a25a in index_addordel_values_ext_sv () from >>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>> #13 0x00007f6832b6a694 in index_addordel_values_sv () from >>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>> #14 0x00007f6832b6a75a in index_addordel_entry () from >>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>> #15 0x00007f6832b6d5fa in ldbm_back_add () from >>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>> #16 0x00007f683f46fcca in op_shared_add () from >>>> /usr/lib64/dirsrv/libslapd.so.0 >>>> #17 0x00007f683f471020 in do_add () from /usr/lib64/dirsrv/libslapd.so. >>>> 0 >>>> #18 0x00007f683f99c303 in connection_threadmain () >>>> #19 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so >>>> #20 0x00007f683d04cdc5 in start_thread () from /lib64/libpthread.so.0 >>>> #21 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 >>>> >>>> This thread is likely not hang on 'db_tas_mutex_lock' as the db_lock >>>> does not show it waiting for something. >>>> Unfortunately it is not shown in the output of top -H, did you get the >>>> complete output ? >>>> Ludwig mentioned a known issue that DS is not scaling well on NUMA >>>> arch, do you know if the machine is NUMA ? >>>> A possibility is that this thread get severely slowed down by a NUMA >>>> remote lock and showing DS has hung. >>>> This is why it would be interesting to know if it consums CPU. >>>> >>>> thanks >>>> theirry >>>> >>>> On 09/06/2016 10:59 AM, Rakesh Rajasekharan wrote: >>>> >>>> Hi Thierry/Ludwig, >>>> >>>> I again hit the hang issue. This is the pstack of the ns-slapd >>>> >>>> So was able to this time get the pstack and db locks >>>> >>>> db locks details >>>> Default locking region information: >>>> 451 Last allocated locker ID >>>> 0x7fffffff Current maximum unused locker ID >>>> 9 Number of lock modes >>>> 800 Initial number of locks allocated >>>> 0 Initial number of lockers allocated >>>> 800 Initial number of lock objects allocated >>>> 10000 Maximum number of locks possible >>>> 10000 Maximum number of lockers possible >>>> 10000 Maximum number of lock objects possible >>>> 800 Current number of locks allocated >>>> 366 Current number of lockers allocated >>>> 800 Current number of lock objects allocated >>>> 160 Number of lock object partitions >>>> 8191 Size of object hash table >>>> 126 Number of current locks >>>> 1005 Maximum number of locks at any one time >>>> 33 Maximum number of locks in any one bucket >>>> 3987 Maximum number of locks stolen by for an empty partition >>>> 75 Maximum number of locks stolen for any one partition >>>> 309 Number of current lockers >>>> 313 Maximum number of lockers at any one time >>>> 121 Number of current lock objects >>>> 676 Maximum number of lock objects at any one time >>>> 3 Maximum number of lock objects in any one bucket >>>> 69 Maximum number of objects stolen by for an empty partition >>>> 5 Maximum number of objects stolen for any one partition >>>> 22M Total number of locks requested (22009642) >>>> 21M Total number of locks released (21992878) >>>> 0 Total number of locks upgraded >>>> 95 Total number of locks downgraded >>>> 3348 Lock requests not available due to conflicts, for which we >>>> waited >>>> 12646 Lock requests not available due to conflicts, for which we >>>> did not wait >>>> 0 Number of deadlocks >>>> 0 Lock timeout value >>>> 0 Number of locks that have timed out >>>> 0 Transaction timeout value >>>> 0 Number of transactions that have timed out >>>> 2MB 560KB Region size >>>> 71719 The number of partition locks that required waiting (0%) >>>> 23105 The maximum number of times any partition lock was waited >>>> for (0%) >>>> 0 The number of object queue operations that required waiting >>>> (0%) >>>> 163 The number of locker allocations that required waiting (0%) >>>> 3172 The number of region locks that required waiting (0%) >>>> 5 Maximum hash bucket length >>>> =-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-= >>>> Lock REGINFO information: >>>> Environment Region type >>>> 1 Region ID >>>> /var/lib/dirsrv/slapd-SPRINKLR-COM/db/__db.001 Region name >>>> 0x7f67ef869000 Region address >>>> 0x7f67ef8690a0 Region allocation head >>>> 0x7f67ef8712b0 Region primary address >>>> 0 Region maximum allocation >>>> 0 Region allocated >>>> Region allocations: 1581345 allocations, 0 failures, 1580794 frees, 6 >>>> longest >>>> Allocations by power-of-two sizes: >>>> 1KB 1581329 >>>> 2KB 3 >>>> 4KB 4 >>>> 8KB 3 >>>> 16KB 3 >>>> 32KB 1 >>>> 64KB 0 >>>> 128KB 0 >>>> 256KB 2 >>>> 512KB 0 >>>> 1024KB 1 >>>> REGION_SHARED Region flags >>>> =-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-= >>>> Lock region parameters: >>>> 2 Lock region region mutex [3172/14M 0% !Own] >>>> 16381 locker table size >>>> 8191 object table size >>>> 34128 obj_off >>>> 900216 locker_off >>>> 0 need_dd >>>> =-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-= >>>> Lock conflict matrix: >>>> =-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-= >>>> Locks grouped by lockers: >>>> Locker Mode Count Status ----------------- Object >>>> --------------- >>>> 2 dd=308 locks held 1 write locks 0 pid/thread >>>> 4845/140085720139840 flags 10 priority 100 >>>> 2 READ 1 HELD userRoot/id2entry.db >>>> handle 0 >>>> 3 dd=307 locks held 0 write locks 0 pid/thread >>>> 4845/140084829472512 flags 0 priority 100 >>>> 4 dd=306 locks held 1 write locks 0 pid/thread >>>> 4845/140085720139840 flags 10 priority 100 >>>> 4 READ 1 HELD ipaca/id2entry.db >>>> handle 0 >>>> 5 dd=305 locks held 0 write locks 0 pid/thread >>>> 4845/140084837865216 flags 0 priority 100 >>>> 6 dd=304 locks held 1 write locks 0 pid/thread >>>> 4845/140085720139840 flags 10 priority 100 >>>> 6 READ 1 HELD ipaca/entryrdn.db >>>> handle 0 >>>> 7 dd=303 locks held 0 write locks 0 pid/thread >>>> 4845/140084804294400 flags 0 priority 100 >>>> 8 dd=302 locks held 0 write locks 0 pid/thread >>>> 4845/140084720432896 flags 0 priority 100 >>>> 9 dd=301 locks held 0 write locks 0 pid/thread >>>> 4845/140084837865216 flags 0 priority 100 >>>> a dd=300 locks held 1 write locks 0 pid/thread >>>> 4845/140085720139840 flags 10 priority 100 >>>> a READ 1 HELD ipaca/vlv#allcertspkitomcatindex.db >>>> handle 0 >>>> e dd=299 locks held 1 write locks 0 pid/thread >>>> 4845/140085720139840 flags 10 priority 100 >>>> e READ 1 HELD ipaca/vlv#allnonrevokedcertspkitomcatindex.db >>>> handle 0 >>>> 10 dd=298 locks held 1 write locks 0 pid/thread >>>> 4845/140085720139840 flags 10 priority 100 >>>> 10 READ 1 HELD ipaca/vlv#allrevokedcertspkitomcatindex.db >>>> handle 0 >>>> 11 dd=297 locks held 1 write locks 0 pid/thread >>>> 4845/140085720139840 flags 10 priority 100 >>>> 11 READ 1 HELD ipaca/vlv#allrevokedcertsnotafterpkitomcatindex.db >>>> handle 0 >>>> 14 dd=296 locks held 1 write locks 0 pid/thread >>>> 4845/140085720139840 flags 10 priority 100 >>>> 14 READ 1 HELD ipaca/vlv#allrevokedorrevokedexpiredcertspkitomcatindex.db >>>> handle 0 >>>> 15 dd=295 locks held 1 write locks 0 pid/thread >>>> 4845/140085720139840 flags 10 priority 100 >>>> 15 READ 1 HELD ipaca/vlv#allvalidcertspkitomcatindex.db >>>> handle 0 >>>> 16 dd=294 locks held 1 write locks 0 pid/thread >>>> 4845/140085720139840 flags 10 priority 100 >>>> 16 READ 1 HELD ipaca/vlv#allvalidcertsnotafterpkitomcatindex.db >>>> handle 0 >>>> 17 dd=293 locks held 1 write locks 0 pid/thread >>>> 4845/140085720139840 flags 10 priority 100 >>>> 17 READ 1 HELD ipaca/vlv#allvalidorrevokedcertspkitomcatindex.db >>>> handle 0 >>>> 18 dd=292 locks held 1 write locks 0 pid/thread >>>> 4845/140085720139840 flags 10 priority 100 >>>> 18 READ 1 HELD ipaca/vlv#caallpkitomcatindex.db >>>> handle 0 >>>> 1d dd=291 locks held 1 write locks 0 pid/thread >>>> 4845/140085720139840 flags 10 priority 100 >>>> 1d READ 1 HELD ipaca/vlv#cacompletepkitomcatindex.db >>>> handle 0 >>>> 1e dd=290 locks held 1 write locks 0 pid/thread >>>> 4845/140085720139840 flags 10 priority 100 >>>> 1e READ 1 HELD ipaca/vlv#cacompleteenrollmentpkitomcatindex.db >>>> handle 0 >>>> 20 dd=289 locks held 1 write locks 0 pid/thread >>>> 4845/140085720139840 flags 10 priority 100 >>>> 20 READ 1 HELD ipaca/vlv#cacompleterevocationpkitomcatindex.db >>>> handle 0 >>>> 21 dd=288 locks held 1 write locks 0 pid/thread >>>> 4845/140085720139840 flags 10 priority 100 >>>> 21 READ 1 HELD ipaca/vlv#caenrollmentpkitomcatindex.db >>>> handle 0 >>>> 2b dd=287 locks held 1 write locks 0 pid/thread >>>> 4845/140085720139840 flags 10 priority 100 >>>> 2b READ 1 HELD ipaca/vlv#carevocationpkitomcatindex.db >>>> handle 0 >>>> 2c dd=286 locks held 1 write locks 0 pid/thread >>>> 4845/140085720139840 flags 10 priority 100 >>>> 2c READ 1 HELD changelog/id2entry.db >>>> handle 0 >>>> 2d dd=285 locks held 0 write locks 0 pid/thread >>>> 4845/140085719607040 flags 0 priority 100 >>>> 2e dd=284 locks held 1 write locks 0 pid/thread >>>> 4845/140085720139840 flags 10 priority 100 >>>> 2e READ 1 HELD changelog/entryusn.db >>>> handle 0 >>>> 2f dd=283 locks held 0 write locks 0 pid/thread >>>> 4845/140085720139840 flags 0 priority 100 >>>> 30 dd=282 locks held 1 write locks 0 pid/thread >>>> 4845/140085720139840 flags 10 priority 100 >>>> 30 READ 1 HELD userRoot/entryusn.db >>>> handle 0 >>>> 31 dd=281 locks held 0 write locks 0 pid/thread >>>> 4845/140085720139840 flags 0 priority 100 >>>> 32 dd=280 locks held 1 write locks 0 pid/thread >>>> 4845/140085720139840 flags 10 priority 100 >>>> 32 READ 1 HELD ipaca/entryusn.db >>>> handle 0 >>>> 33 dd=279 locks held 0 write locks 0 pid/thread >>>> 4845/140085720139840 flags 0 priority 100 >>>> 34 dd=278 locks held 1 write locks 0 pid/thread >>>> 4845/140085720139840 flags 10 priority 100 >>>> 34 READ 1 HELD userRoot/entryrdn.db >>>> handle 0 >>>> 35 dd=277 locks held 0 write locks 0 pid/thread >>>> 4845/140084544186112 flags 0 priority 100 >>>> 36 dd=276 locks held 0 write locks 0 pid/thread >>>> 4845/140084636505856 flags 0 priority 100 >>>> 37 dd=275 locks held 0 write locks 0 pid/thread >>>> 4845/140084854650624 flags 0 priority 100 >>>> 38 dd=274 locks held 1 write locks 0 pid/thread >>>> 4845/140085720139840 flags 10 priority 100 >>>> 38 READ 1 HELD userRoot/objectclass.db >>>> handle 0 >>>> 39 dd=273 locks held 0 write locks 0 pid/thread >>>> 4845/140084569364224 flags 0 priority 100 >>>> 3a dd=272 locks held 1 write locks 0 pid/thread >>>> 4845/140085720139840 flags 10 priority 100 >>>> 3a READ 1 HELD userRoot/ancestorid.db >>>> handle 0 >>>> 3b dd=271 locks held 0 write locks 0 pid/thread >>>> 4845/140084854650624 flags 0 priority 100 >>>> 3c dd=270 locks held 1 write locks 0 pid/thread >>>> 4845/140085720139840 flags 10 priority 100 >>>> 3c READ 1 HELD changelog/entryrdn.db >>>> handle 0 >>>> 3d dd=269 locks held 0 write locks 0 pid/thread >>>> 4845/140085719607040 flags 0 priority 100 >>>> 3e dd=268 locks held 0 write locks 0 pid/thread >>>> 4845/140084602935040 flags 0 priority 100 >>>> 3f dd=267 locks held 0 write locks 0 pid/thread >>>> 4845/140085719607040 flags 0 priority 100 >>>> 40 dd=266 locks held 1 write locks 0 pid/thread >>>> 4845/140085720139840 flags 10 priority 100 >>>> 40 READ 1 HELD changelog/objectclass.db >>>> handle 0 >>>> 41 dd=265 locks held 0 write locks 0 pid/thread >>>> 4845/140085719607040 flags 0 priority 100 >>>> 42 dd=264 locks held 0 write locks 0 pid/thread >>>> 4845/140085719607040 flags 0 priority 100 >>>> 43 dd=263 locks held 0 write locks 0 pid/thread >>>> 4845/140084653291264 flags 0 priority 100 >>>> 44 dd=262 locks held 1 write locks 0 pid/thread >>>> 4845/140085720139840 flags 10 priority 100 >>>> 44 READ 1 HELD ipaca/objectclass.db >>>> handle 0 >>>> 45 dd=261 locks held 0 write locks 0 pid/thread >>>> 4845/140084577756928 flags 0 priority 100 >>>> 46 dd=260 locks held 0 write locks 0 pid/thread >>>> 4845/140084720432896 flags 0 priority 100 >>>> 47 dd=259 locks held 1 write locks 0 pid/thread >>>> 4845/140085720139840 flags 10 priority 100 >>>> 47 READ 1 HELD changelog/aci.db >>>> handle 0 >>>> 48 dd=258 locks held 0 write locks 0 pid/thread >>>> 4845/140085720139840 flags 0 priority 100 >>>> 49 dd=257 locks held 1 write locks 0 pid/thread >>>> 4845/140085720139840 flags 10 priority 100 >>>> 49 READ 1 HELD userRoot/aci.db >>>> handle 0 >>>> 4a dd=256 locks held 0 write locks 0 pid/thread >>>> 4845/140085720139840 flags 0 priority 100 >>>> 4b dd=255 locks held 0 write locks 0 pid/thread >>>> 4845/140084535793408 flags 0 priority 100 >>>> 4c dd=254 locks held 1 write locks 0 pid/thread >>>> 4845/140085720139840 flags 10 priority 100 >>>> 4c READ 1 HELD ipaca/aci.db >>>> handle 0 >>>> 4d dd=253 locks held 0 write locks 0 pid/thread >>>> 4845/140085720139840 flags 0 priority 100 >>>> 4e dd=252 locks held 1 write locks 0 pid/thread >>>> 4845/140085720139840 flags 10 priority 100 >>>> 4e READ 1 HELD userRoot/parentid.db >>>> handle 0 >>>> 4f dd=251 locks held 0 write locks 0 pid/thread >>>> 4845/140085720139840 flags 0 priority 100 >>>> 50 dd=250 locks held 0 write locks 0 pid/thread >>>> 4845/140085719607040 flags 0 priority 100 >>>> 51 dd=249 locks held 1 write locks 0 pid/thread >>>> 4845/140085720139840 flags 10 priority 100 >>>> 51 READ 1 HELD changelog/nsuniqueid.db >>>> handle 0 >>>> 52 dd=248 locks held 1 write locks 0 pid/thread >>>> 4845/140085720139840 flags 10 priority 100 >>>> 52 READ 1 HELD changelog/changenumber.db >>>> handle 0 >>>> 53 dd=247 locks held 0 write locks 0 pid/thread >>>> 4845/140084602935040 flags 0 priority 100 >>>> 54 dd=246 locks held 0 write locks 0 pid/thread >>>> 4845/140085719607040 flags 0 priority 100 >>>> 55 dd=245 locks held 1 write locks 0 pid/thread >>>> 4845/140085719607040 flags 10 priority 100 >>>> 55 READ 1 HELD changelog/targetuniqueid.db >>>> handle 0 >>>> 56 dd=244 locks held 1 write locks 0 pid/thread >>>> 4845/140085719607040 flags 10 priority 100 >>>> 56 READ 1 HELD changelog/parentid.db >>>> handle 0 >>>> 57 dd=243 locks held 1 write locks 0 pid/thread >>>> 4845/140085719607040 flags 10 priority 100 >>>> 57 READ 1 HELD changelog/ancestorid.db >>>> handle 0 >>>> 58 dd=242 locks held 1 write locks 0 pid/thread >>>> 4845/140085719607040 flags 10 priority 100 >>>> 58 READ 1 HELD changelog/numsubordinates.db >>>> handle 0 >>>> 59 dd=241 locks held 0 write locks 0 pid/thread >>>> 4845/140085719607040 flags 0 priority 100 >>>> 5a dd=240 locks held 0 write locks 0 pid/thread >>>> 4845/140084544186112 flags 0 priority 100 >>>> 5b dd=239 locks held 1 write locks 0 pid/thread >>>> 4845/140084577756928 flags 10 priority 100 >>>> 5b READ 1 HELD userRoot/krbPrincipalName.db >>>> handle 0 >>>> 5c dd=238 locks held 0 write locks 0 pid/thread >>>> 4845/140084636505856 flags 0 priority 100 >>>> 5d dd=237 locks held 0 write locks 0 pid/thread >>>> 4845/140084695254784 flags 0 priority 100 >>>> 5e dd=236 locks held 1 write locks 0 pid/thread >>>> 4845/140084560971520 flags 10 priority 100 >>>> 5e READ 1 HELD userRoot/ipakrbprincipalalias.db >>>> handle 0 >>>> 5f dd=235 locks held 0 write locks 0 pid/thread >>>> 4845/140084703647488 flags 0 priority 100 >>>> 60 dd=234 locks held 0 write locks 0 pid/thread >>>> 4845/140084653291264 flags 0 priority 100 >>>> 61 dd=233 locks held 0 write locks 0 pid/thread >>>> 4845/140084577756928 flags 0 priority 100 >>>> 62 dd=232 locks held 0 write locks 0 pid/thread >>>> 4845/140084686862080 flags 0 priority 100 >>>> 63 dd=231 locks held 1 write locks 0 pid/thread >>>> 4845/140084611327744 flags 10 priority 100 >>>> 63 READ 1 HELD changelog/seeAlso.db >>>> handle 0 >>>> 64 dd=230 locks held 0 write locks 0 pid/thread >>>> 4845/140084686862080 flags 0 priority 100 >>>> 65 dd=229 locks held 1 write locks 0 pid/thread >>>> 4845/140084611327744 flags 10 priority 100 >>>> 65 READ 1 HELD userRoot/seeAlso.db >>>> handle 0 >>>> 66 dd=228 locks held 0 write locks 0 pid/thread >>>> 4845/140084636505856 flags 0 priority 100 >>>> 67 dd=227 locks held 1 write locks 0 pid/thread >>>> 4845/140084611327744 flags 10 priority 100 >>>> 67 READ 1 HELD ipaca/seeAlso.db >>>> handle 0 >>>> 68 dd=226 locks held 0 write locks 0 pid/thread >>>> 4845/140084686862080 flags 0 priority 100 >>>> 69 dd=225 locks held 1 write locks 0 pid/thread >>>> 4845/140084678469376 flags 10 priority 100 >>>> 69 READ 1 HELD ipaca/parentid.db >>>> handle 0 >>>> 6a dd=224 locks held 0 write locks 0 pid/thread >>>> 4845/140084678469376 flags 0 priority 100 >>>> 6b dd=223 locks held 0 write locks 0 pid/thread >>>> 4845/140084829472512 flags 0 priority 100 >>>> 6c dd=222 locks held 0 write locks 0 pid/thread >>>> 4845/140084653291264 flags 0 priority 100 >>>> 6d dd=221 locks held 0 write locks 0 pid/thread >>>> 4845/140084569364224 flags 0 priority 100 >>>> 6e dd=220 locks held 0 write locks 0 pid/thread >>>> 4845/140084636505856 flags 0 priority 100 >>>> 6e READ 1 WAIT userRoot/objectclass.db page >>>> 18 >>>> 6f dd=219 locks held 0 write locks 0 pid/thread >>>> 4845/140084544186112 flags 0 priority 100 >>>> 70 dd=218 locks held 0 write locks 0 pid/thread >>>> 4845/140084569364224 flags 0 priority 100 >>>> 71 dd=217 locks held 0 write locks 0 pid/thread >>>> 4845/140084619720448 flags 0 priority 100 >>>> 72 dd=216 locks held 0 write locks 0 pid/thread >>>> 4845/140084552578816 flags 0 priority 100 >>>> 73 dd=215 locks held 0 write locks 0 pid/thread >>>> 4845/140084829472512 flags 0 priority 100 >>>> 74 dd=214 locks held 0 write locks 0 pid/thread >>>> 4845/140084628113152 flags 0 priority 100 >>>> 75 dd=213 locks held 0 write locks 0 pid/thread >>>> 4845/140084686862080 flags 0 priority 100 >>>> 76 dd=212 locks held 1 write locks 0 pid/thread >>>> 4845/140084560971520 flags 10 priority 100 >>>> 76 READ 1 HELD userRoot/ipauniqueid.db >>>> handle 0 >>>> 77 dd=211 locks held 0 write locks 0 pid/thread >>>> 4845/140084661683968 flags 0 priority 100 >>>> 78 dd=210 locks held 0 write locks 0 pid/thread >>>> 4845/140084846257920 flags 0 priority 100 >>>> 79 dd=209 locks held 0 write locks 0 pid/thread >>>> 4845/140084628113152 flags 0 priority 100 >>>> 7a dd=208 locks held 1 write locks 0 pid/thread >>>> 4845/140084560971520 flags 10 priority 100 >>>> 7a READ 1 HELD userRoot/nsuniqueid.db >>>> handle 0 >>>> 7b dd=207 locks held 1 write locks 0 pid/thread >>>> 4845/140084560971520 flags 10 priority 100 >>>> 7b READ 1 HELD userRoot/cn.db >>>> handle 0 >>>> 7c dd=206 locks held 1 write locks 0 pid/thread >>>> 4845/140084560971520 flags 10 priority 100 >>>> 7c READ 1 HELD userRoot/fqdn.db >>>> handle 0 >>>> 7d dd=205 locks held 1 write locks 0 pid/thread >>>> 4845/140084560971520 flags 10 priority 100 >>>> 7d READ 1 HELD userRoot/managedby.db >>>> handle 0 >>>> 7e dd=204 locks held 1 write locks 0 pid/thread >>>> 4845/140084560971520 flags 10 priority 100 >>>> 7e READ 1 HELD userRoot/numsubordinates.db >>>> handle 0 >>>> 7f dd=203 locks held 1 write locks 0 pid/thread >>>> 4845/140084678469376 flags 10 priority 100 >>>> 7f READ 1 HELD userRoot/member.db >>>> handle 0 >>>> 80 dd=202 locks held 0 write locks 0 pid/thread >>>> 4845/140084712040192 flags 0 priority 100 >>>> 81 dd=201 locks held 1 write locks 0 pid/thread >>>> 4845/140084678469376 flags 10 priority 100 >>>> 81 READ 1 HELD userRoot/memberUser.db >>>> handle 0 >>>> 82 dd=200 locks held 0 write locks 0 pid/thread >>>> 4845/140084577756928 flags 0 priority 100 >>>> 83 dd=199 locks held 1 write locks 0 pid/thread >>>> 4845/140084678469376 flags 10 priority 100 >>>> 83 READ 1 HELD userRoot/memberHost.db >>>> handle 0 >>>> 84 dd=198 locks held 0 write locks 0 pid/thread >>>> 4845/140084712040192 flags 0 priority 100 >>>> 85 dd=197 locks held 0 write locks 0 pid/thread >>>> 4845/140084577756928 flags 0 priority 100 >>>> 86 dd=196 locks held 0 write locks 0 pid/thread >>>> 4845/140084670076672 flags 0 priority 100 >>>> 87 dd=195 locks held 0 write locks 0 pid/thread >>>> 4845/140084577756928 flags 0 priority 100 >>>> 88 dd=194 locks held 0 write locks 0 pid/thread >>>> 4845/140084846257920 flags 0 priority 100 >>>> 89 dd=193 locks held 0 write locks 0 pid/thread >>>> 4845/140084552578816 flags 0 priority 100 >>>> 8a dd=192 locks held 0 write locks 0 pid/thread >>>> 4845/140084703647488 flags 0 priority 100 >>>> 8b dd=191 locks held 0 write locks 0 pid/thread >>>> 4845/140084544186112 flags 0 priority 100 >>>> 8c dd=190 locks held 0 write locks 0 pid/thread >>>> 4845/140084846257920 flags 0 priority 100 >>>> 8d dd=189 locks held 0 write locks 0 pid/thread >>>> 4845/140084661683968 flags 0 priority 100 >>>> 8e dd=188 locks held 1 write locks 0 pid/thread >>>> 4845/140085067675392 flags 10 priority 100 >>>> 8e READ 1 HELD userRoot/macAddress.db >>>> handle 0 >>>> 8f dd=187 locks held 0 write locks 0 pid/thread >>>> 4845/140085067675392 flags 0 priority 100 >>>> 90 dd=186 locks held 0 write locks 0 pid/thread >>>> 4845/140084703647488 flags 0 priority 100 >>>> 91 dd=185 locks held 0 write locks 0 pid/thread >>>> 4845/140084552578816 flags 0 priority 100 >>>> 92 dd=184 locks held 0 write locks 0 pid/thread >>>> 4845/140084560971520 flags 0 priority 100 >>>> 92 READ 1 WAIT userRoot/objectclass.db >>>> page 9 >>>> 93 dd=183 locks held 0 write locks 0 pid/thread >>>> 4845/140084577756928 flags 0 priority 100 >>>> 94 dd=182 locks held 0 write locks 0 pid/thread >>>> 4845/140084821079808 flags 0 priority 100 >>>> 94 READ 1 WAIT userRoot/objectclass.db page >>>> 18 >>>> 95 dd=181 locks held 0 write locks 0 pid/thread >>>> 4845/140084544186112 flags 0 priority 100 >>>> 96 dd=180 locks held 0 write locks 0 pid/thread >>>> 4845/140084804294400 flags 0 priority 100 >>>> 97 dd=179 locks held 1 write locks 0 pid/thread >>>> 4845/140084712040192 flags 10 priority 100 >>>> 97 READ 1 HELD userRoot/uid.db >>>> handle 0 >>>> 98 dd=178 locks held 0 write locks 0 pid/thread >>>> 4845/140084552578816 flags 0 priority 100 >>>> 99 dd=177 locks held 0 write locks 0 pid/thread >>>> 4845/140084577756928 flags 0 priority 100 >>>> 9a dd=176 locks held 0 write locks 0 pid/thread >>>> 4845/140084602935040 flags 0 priority 100 >>>> 9b dd=175 locks held 1 write locks 0 pid/thread >>>> 4845/140085076068096 flags 10 priority 100 >>>> 9b READ 1 HELD userRoot/uniquemember.db >>>> handle 0 >>>> 9c dd=174 locks held 1 write locks 0 pid/thread >>>> 4845/140085076068096 flags 10 priority 100 >>>> 9c READ 1 HELD userRoot/owner.db >>>> handle 0 >>>> 9d dd=173 locks held 1 write locks 0 pid/thread >>>> 4845/140085076068096 flags 10 priority 100 >>>> 9d READ 1 HELD userRoot/ipaallowedtarget.db >>>> handle 0 >>>> 9e dd=172 locks held 1 write locks 0 pid/thread >>>> 4845/140085076068096 flags 10 priority 100 >>>> 9e READ 1 HELD userRoot/ipaMemberCertProfile.db >>>> handle 0 >>>> 9f dd=171 locks held 1 write locks 0 pid/thread >>>> 4845/140085076068096 flags 10 priority 100 >>>> 9f READ 1 HELD userRoot/ipaassignedidview.db >>>> handle 0 >>>> a0 dd=170 locks held 1 write locks 0 pid/thread >>>> 4845/140085076068096 flags 10 priority 100 >>>> a0 READ 1 HELD userRoot/secretary.db >>>> handle 0 >>>> a1 dd=169 locks held 1 write locks 0 pid/thread >>>> 4845/140085076068096 flags 10 priority 100 >>>> a1 READ 1 HELD userRoot/memberdenycmd.db >>>> handle 0 >>>> a2 dd=168 locks held 1 write locks 0 pid/thread >>>> 4845/140085076068096 flags 10 priority 100 >>>> a2 READ 1 HELD userRoot/memberallowcmd.db >>>> handle 0 >>>> a3 dd=167 locks held 1 write locks 0 pid/thread >>>> 4845/140085076068096 flags 10 priority 100 >>>> a3 READ 1 HELD userRoot/manager.db >>>> handle 0 >>>> a4 dd=166 locks held 1 write locks 0 pid/thread >>>> 4845/140085076068096 flags 10 priority 100 >>>> a4 READ 1 HELD userRoot/ipasudorunas.db >>>> handle 0 >>>> a5 dd=165 locks held 1 write locks 0 pid/thread >>>> 4845/140085076068096 flags 10 priority 100 >>>> a5 READ 1 HELD userRoot/ipaMemberCa.db >>>> handle 0 >>>> a6 dd=164 locks held 1 write locks 0 pid/thread >>>> 4845/140085076068096 flags 10 priority 100 >>>> a6 READ 1 HELD userRoot/ipasudorunasgroup.db >>>> handle 0 >>>> a7 dd=163 locks held 1 write locks 0 pid/thread >>>> 4845/140085076068096 flags 10 priority 100 >>>> a7 READ 1 HELD userRoot/sourcehost.db >>>> handle 0 >>>> a8 dd=162 locks held 1 write locks 0 pid/thread >>>> 4845/140085076068096 flags 10 priority 100 >>>> a8 READ 1 HELD userRoot/memberservice.db >>>> handle 0 >>>> a9 dd=161 locks held 1 write locks 0 pid/thread >>>> 4845/140085076068096 flags 10 priority 100 >>>> a9 READ 1 HELD userRoot/ipatokenradiusconfiglink.db >>>> handle 0 >>>> aa dd=160 locks held 0 write locks 0 pid/thread >>>> 4845/140084703647488 flags 0 priority 100 >>>> ab dd=159 locks held 0 write locks 0 pid/thread >>>> 4845/140084821079808 flags 0 priority 100 >>>> ac dd=158 locks held 0 write locks 0 pid/thread >>>> 4845/140084628113152 flags 0 priority 100 >>>> ad dd=157 locks held 0 write locks 0 pid/thread >>>> 4845/140084829472512 flags 0 priority 100 >>>> ae dd=156 locks held 0 write locks 0 pid/thread >>>> 4845/140084829472512 flags 0 priority 100 >>>> af dd=155 locks held 0 write locks 0 pid/thread >>>> 4845/140084854650624 flags 0 priority 100 >>>> b0 dd=154 locks held 0 write locks 0 pid/thread >>>> 4845/140084854650624 flags 0 priority 100 >>>> b1 dd=153 locks held 0 write locks 0 pid/thread >>>> 4845/140084837865216 flags 0 priority 100 >>>> b2 dd=152 locks held 0 write locks 0 pid/thread >>>> 4845/140084804294400 flags 0 priority 100 >>>> b3 dd=151 locks held 0 write locks 0 pid/thread >>>> 4845/140084812687104 flags 0 priority 100 >>>> b4 dd=150 locks held 0 write locks 0 pid/thread >>>> 4845/140084804294400 flags 0 priority 100 >>>> b5 dd=149 locks held 0 write locks 0 pid/thread >>>> 4845/140084837865216 flags 0 priority 100 >>>> b6 dd=148 locks held 0 write locks 0 pid/thread >>>> 4845/140084569364224 flags 0 priority 100 >>>> b7 dd=147 locks held 0 write locks 0 pid/thread >>>> 4845/140084829472512 flags 0 priority 100 >>>> b8 dd=146 locks held 0 write locks 0 pid/thread >>>> 4845/140084586149632 flags 0 priority 100 >>>> b9 dd=145 locks held 0 write locks 0 pid/thread >>>> 4845/140084653291264 flags 0 priority 100 >>>> ba dd=144 locks held 0 write locks 0 pid/thread >>>> 4845/140084636505856 flags 0 priority 100 >>>> bb dd=143 locks held 0 write locks 0 pid/thread >>>> 4845/140084804294400 flags 0 priority 100 >>>> bc dd=142 locks held 0 write locks 0 pid/thread >>>> 4845/140084804294400 flags 0 priority 100 >>>> bd dd=141 locks held 0 write locks 0 pid/thread >>>> 4845/140084695254784 flags 0 priority 100 >>>> be dd=140 locks held 0 write locks 0 pid/thread >>>> 4845/140084670076672 flags 0 priority 100 >>>> bf dd=139 locks held 0 write locks 0 pid/thread >>>> 4845/140084628113152 flags 0 priority 100 >>>> c0 dd=138 locks held 0 write locks 0 pid/thread >>>> 4845/140084661683968 flags 0 priority 100 >>>> c1 dd=137 locks held 0 write locks 0 pid/thread >>>> 4845/140084712040192 flags 0 priority 100 >>>> c2 dd=136 locks held 0 write locks 0 pid/thread >>>> 4845/140084661683968 flags 0 priority 100 >>>> c3 dd=135 locks held 0 write locks 0 pid/thread >>>> 4845/140084577756928 flags 0 priority 100 >>>> c4 dd=134 locks held 0 write locks 0 pid/thread >>>> 4845/140084854650624 flags 0 priority 100 >>>> c5 dd=133 locks held 1 write locks 0 pid/thread >>>> 4845/140084644898560 flags 10 priority 100 >>>> c5 READ 1 HELD userRoot/uidnumber.db >>>> handle 0 >>>> c6 dd=132 locks held 0 write locks 0 pid/thread >>>> 4845/140084594542336 flags 0 priority 100 >>>> c8 dd=131 locks held 1 write locks 0 pid/thread >>>> 4845/140084678469376 flags 10 priority 100 >>>> c8 READ 1 HELD ipaca/certstatus.db >>>> handle 0 >>>> c9 dd=130 locks held 0 write locks 0 pid/thread >>>> 4845/140084720432896 flags 0 priority 100 >>>> ca dd=129 locks held 0 write locks 0 pid/thread >>>> 4845/140084619720448 flags 0 priority 100 >>>> cb dd=128 locks held 0 write locks 0 pid/thread >>>> 4845/140084821079808 flags 0 priority 100 >>>> cc dd=127 locks held 0 write locks 0 pid/thread >>>> 4845/140084594542336 flags 0 priority 100 >>>> cd dd=126 locks held 0 write locks 0 pid/thread >>>> 4845/140084560971520 flags 0 priority 100 >>>> ce dd=125 locks held 1 write locks 0 pid/thread >>>> 4845/140084837865216 flags 10 priority 100 >>>> ce READ 1 HELD userRoot/gidnumber.db >>>> handle 0 >>>> cf dd=124 locks held 0 write locks 0 pid/thread >>>> 4845/140084720432896 flags 0 priority 100 >>>> d0 dd=123 locks held 0 write locks 0 pid/thread >>>> 4845/140084569364224 flags 0 priority 100 >>>> d1 dd=122 locks held 1 write locks 0 pid/thread >>>> 4845/140084678469376 flags 10 priority 100 >>>> d1 READ 1 HELD userRoot/nscpEntryDN.db >>>> handle 0 >>>> d2 dd=121 locks held 0 write locks 0 pid/thread >>>> 4845/140084695254784 flags 0 priority 100 >>>> d3 dd=120 locks held 0 write locks 0 pid/thread >>>> 4845/140084569364224 flags 0 priority 100 >>>> d4 dd=119 locks held 0 write locks 0 pid/thread >>>> 4845/140084653291264 flags 0 priority 100 >>>> d5 dd=118 locks held 0 write locks 0 pid/thread >>>> 4845/140084829472512 flags 0 priority 100 >>>> d6 dd=117 locks held 0 write locks 0 pid/thread >>>> 4845/140084661683968 flags 0 priority 100 >>>> d7 dd=116 locks held 0 write locks 0 pid/thread >>>> 4845/140084804294400 flags 0 priority 100 >>>> d8 dd=115 locks held 1 write locks 0 pid/thread >>>> 4845/140084821079808 flags 10 priority 100 >>>> d8 READ 1 HELD ipaca/description.db >>>> handle 0 >>>> d9 dd=114 locks held 0 write locks 0 pid/thread >>>> 4845/140084661683968 flags 0 priority 100 >>>> da dd=113 locks held 0 write locks 0 pid/thread >>>> 4845/140084837865216 flags 0 priority 100 >>>> db dd=112 locks held 1 write locks 0 pid/thread >>>> 4845/140084678469376 flags 10 priority 100 >>>> db READ 1 HELD ipaca/cn.db >>>> handle 0 >>>> dc dd=111 locks held 0 write locks 0 pid/thread >>>> 4845/140084577756928 flags 0 priority 100 >>>> dd dd=110 locks held 1 write locks 0 pid/thread >>>> 4845/140084678469376 flags 10 priority 100 >>>> dd READ 1 HELD ipaca/ancestorid.db >>>> handle 0 >>>> de dd=109 locks held 0 write locks 0 pid/thread >>>> 4845/140084577756928 flags 0 priority 100 >>>> df dd=108 locks held 0 write locks 0 pid/thread >>>> 4845/140084703647488 flags 0 priority 100 >>>> e0 dd=107 locks held 0 write locks 0 pid/thread >>>> 4845/140084569364224 flags 0 priority 100 >>>> e1 dd=106 locks held 0 write locks 0 pid/thread >>>> 4845/140084569364224 flags 0 priority 100 >>>> e2 dd=105 locks held 0 write locks 0 pid/thread >>>> 4845/140084821079808 flags 0 priority 100 >>>> e3 dd=104 locks held 1 write locks 0 pid/thread >>>> 4845/140084686862080 flags 10 priority 100 >>>> e3 READ 1 HELD ipaca/serialno.db >>>> handle 0 >>>> e4 dd=103 locks held 0 write locks 0 pid/thread >>>> 4845/140084837865216 flags 0 priority 100 >>>> e5 dd=102 locks held 1 write locks 0 pid/thread >>>> 4845/140084695254784 flags 10 priority 100 >>>> e5 READ 1 HELD ipaca/revInfo.db >>>> handle 0 >>>> e6 dd=101 locks held 1 write locks 0 pid/thread >>>> 4845/140084695254784 flags 10 priority 100 >>>> e6 READ 1 HELD ipaca/revokedby.db >>>> handle 0 >>>> e7 dd=100 locks held 1 write locks 0 pid/thread >>>> 4845/140084695254784 flags 10 priority 100 >>>> e7 READ 1 HELD ipaca/revokedOn.db >>>> handle 0 >>>> e8 dd=99 locks held 0 write locks 0 pid/thread >>>> 4845/140084829472512 flags 0 priority 100 >>>> e9 dd=98 locks held 0 write locks 0 pid/thread >>>> 4845/140084846257920 flags 0 priority 100 >>>> ea dd=97 locks held 1 write locks 0 pid/thread >>>> 4845/140084703647488 flags 10 priority 100 >>>> ea READ 1 HELD ipaca/nsuniqueid.db >>>> handle 0 >>>> eb dd=96 locks held 1 write locks 0 pid/thread >>>> 4845/140084703647488 flags 10 priority 100 >>>> eb READ 1 HELD ipaca/requestid.db >>>> handle 0 >>>> ec dd=95 locks held 1 write locks 0 pid/thread >>>> 4845/140084703647488 flags 10 priority 100 >>>> ec READ 1 HELD ipaca/requeststate.db >>>> handle 0 >>>> ed dd=94 locks held 1 write locks 0 pid/thread >>>> 4845/140084703647488 flags 10 priority 100 >>>> ed READ 1 HELD ipaca/dateOfCreate.db >>>> handle 0 >>>> ee dd=93 locks held 1 write locks 0 pid/thread >>>> 4845/140084703647488 flags 10 priority 100 >>>> ee READ 1 HELD ipaca/requesttype.db >>>> handle 0 >>>> ef dd=92 locks held 1 write locks 0 pid/thread >>>> 4845/140084703647488 flags 10 priority 100 >>>> ef READ 1 HELD ipaca/numsubordinates.db >>>> handle 0 >>>> f0 dd=91 locks held 1 write locks 0 pid/thread >>>> 4845/140084611327744 flags 10 priority 100 >>>> f0 READ 1 HELD userRoot/userCertificate.db >>>> handle 0 >>>> f1 dd=90 locks held 0 write locks 0 pid/thread >>>> 4845/140084821079808 flags 0 priority 100 >>>> f2 dd=89 locks held 0 write locks 0 pid/thread >>>> 4845/140084569364224 flags 0 priority 100 >>>> f3 dd=88 locks held 0 write locks 0 pid/thread >>>> 4845/140084602935040 flags 0 priority 100 >>>> f4 dd=87 locks held 0 write locks 0 pid/thread >>>> 4845/140084720432896 flags 0 priority 100 >>>> f5 dd=86 locks held 0 write locks 0 pid/thread >>>> 4845/140084628113152 flags 0 priority 100 >>>> f6 dd=85 locks held 0 write locks 0 pid/thread >>>> 4845/140084703647488 flags 0 priority 100 >>>> f7 dd=84 locks held 0 write locks 0 pid/thread >>>> 4845/140084611327744 flags 0 priority 100 >>>> f8 dd=83 locks held 0 write locks 0 pid/thread >>>> 4845/140084661683968 flags 0 priority 100 >>>> f9 dd=82 locks held 0 write locks 0 pid/thread >>>> 4845/140084837865216 flags 0 priority 100 >>>> fa dd=81 locks held 0 write locks 0 pid/thread >>>> 4845/140084544186112 flags 0 priority 100 >>>> fb dd=80 locks held 0 write locks 0 pid/thread >>>> 4845/140084703647488 flags 0 priority 100 >>>> fc dd=79 locks held 0 write locks 0 pid/thread >>>> 4845/140084544186112 flags 0 priority 100 >>>> fd dd=78 locks held 0 write locks 0 pid/thread >>>> 4845/140084535793408 flags 0 priority 100 >>>> fe dd=77 locks held 0 write locks 0 pid/thread >>>> 4845/140084586149632 flags 0 priority 100 >>>> ff dd=76 locks held 0 write locks 0 pid/thread >>>> 4845/140084569364224 flags 0 priority 100 >>>> 100 dd=75 locks held 0 write locks 0 pid/thread >>>> 4845/140084804294400 flags 0 priority 100 >>>> 101 dd=74 locks held 0 write locks 0 pid/thread >>>> 4845/140084703647488 flags 0 priority 100 >>>> 102 dd=73 locks held 0 write locks 0 pid/thread >>>> 4845/140084703647488 flags 0 priority 100 >>>> 103 dd=72 locks held 0 write locks 0 pid/thread >>>> 4845/140084821079808 flags 0 priority 100 >>>> 104 dd=71 locks held 0 write locks 0 pid/thread >>>> 4845/140084695254784 flags 0 priority 100 >>>> 105 dd=70 locks held 0 write locks 0 pid/thread >>>> 4845/140084544186112 flags 0 priority 100 >>>> 106 dd=69 locks held 0 write locks 0 pid/thread >>>> 4845/140084854650624 flags 0 priority 100 >>>> 107 dd=68 locks held 0 write locks 0 pid/thread >>>> 4845/140084628113152 flags 0 priority 100 >>>> 108 dd=67 locks held 0 write locks 0 pid/thread >>>> 4845/140084569364224 flags 0 priority 100 >>>> 109 dd=66 locks held 0 write locks 0 pid/thread >>>> 4845/140084594542336 flags 0 priority 100 >>>> 10a dd=65 locks held 0 write locks 0 pid/thread >>>> 4845/140084661683968 flags 0 priority 100 >>>> 10b dd=64 locks held 0 write locks 0 pid/thread >>>> 4845/140084653291264 flags 0 priority 100 >>>> 10c dd=63 locks held 0 write locks 0 pid/thread >>>> 4845/140084695254784 flags 0 priority 100 >>>> 10d dd=62 locks held 0 write locks 0 pid/thread >>>> 4845/140084720432896 flags 0 priority 100 >>>> 10e dd=61 locks held 0 write locks 0 pid/thread >>>> 4845/140084619720448 flags 0 priority 100 >>>> 10f dd=60 locks held 0 write locks 0 pid/thread >>>> 4845/140084720432896 flags 0 priority 100 >>>> 110 dd=59 locks held 0 write locks 0 pid/thread >>>> 4845/140084569364224 flags 0 priority 100 >>>> 111 dd=58 locks held 0 write locks 0 pid/thread >>>> 4845/140084695254784 flags 0 priority 100 >>>> 112 dd=57 locks held 0 write locks 0 pid/thread >>>> 4845/140084653291264 flags 0 priority 100 >>>> 113 dd=56 locks held 0 write locks 0 pid/thread >>>> 4845/140084712040192 flags 0 priority 100 >>>> 114 dd=55 locks held 0 write locks 0 pid/thread >>>> 4845/140084686862080 flags 0 priority 100 >>>> 115 dd=54 locks held 0 write locks 0 pid/thread >>>> 4845/140084577756928 flags 0 priority 100 >>>> 115 READ 1 WAIT userRoot/objectclass.db >>>> page 9 >>>> 116 dd=53 locks held 0 write locks 0 pid/thread >>>> 4845/140084837865216 flags 0 priority 100 >>>> 116 READ 1 WAIT userRoot/objectclass.db >>>> page 9 >>>> 117 dd=52 locks held 0 write locks 0 pid/thread >>>> 4845/140084577756928 flags 0 priority 100 >>>> 118 dd=51 locks held 0 write locks 0 pid/thread >>>> 4845/140084569364224 flags 0 priority 100 >>>> 119 dd=50 locks held 0 write locks 0 pid/thread >>>> 4845/140084560971520 flags 0 priority 100 >>>> 11a dd=49 locks held 1 write locks 0 pid/thread >>>> 4845/140084695254784 flags 10 priority 100 >>>> 11a READ 1 HELD ipaca/metaInfo.db >>>> handle 0 >>>> 11b dd=48 locks held 1 write locks 0 pid/thread >>>> 4845/140084695254784 flags 10 priority 100 >>>> 11b READ 1 HELD ipaca/notbefore.db >>>> handle 0 >>>> 11c dd=47 locks held 1 write locks 0 pid/thread >>>> 4845/140084695254784 flags 10 priority 100 >>>> 11c READ 1 HELD ipaca/notafter.db >>>> handle 0 >>>> 11d dd=46 locks held 1 write locks 0 pid/thread >>>> 4845/140084695254784 flags 10 priority 100 >>>> 11d READ 1 HELD ipaca/duration.db >>>> handle 0 >>>> 11e dd=45 locks held 1 write locks 0 pid/thread >>>> 4845/140084695254784 flags 10 priority 100 >>>> 11e READ 1 HELD ipaca/subjectname.db >>>> handle 0 >>>> 11f dd=44 locks held 1 write locks 0 pid/thread >>>> 4845/140084695254784 flags 10 priority 100 >>>> 11f READ 1 HELD ipaca/publicKeyData.db >>>> handle 0 >>>> 120 dd=43 locks held 1 write locks 0 pid/thread >>>> 4845/140084695254784 flags 10 priority 100 >>>> 120 READ 1 HELD ipaca/extension.db >>>> handle 0 >>>> 121 dd=42 locks held 1 write locks 0 pid/thread >>>> 4845/140084695254784 flags 10 priority 100 >>>> 121 READ 1 HELD ipaca/issuedby.db >>>> handle 0 >>>> 122 dd=41 locks held 0 write locks 0 pid/thread >>>> 4845/140084712040192 flags 0 priority 100 >>>> 123 dd=40 locks held 0 write locks 0 pid/thread >>>> 4845/140084812687104 flags 0 priority 100 >>>> 124 dd=39 locks held 0 write locks 0 pid/thread >>>> 4845/140084544186112 flags 0 priority 100 >>>> 125 dd=38 locks held 0 write locks 0 pid/thread >>>> 4845/140084686862080 flags 0 priority 100 >>>> 126 dd=37 locks held 0 write locks 0 pid/thread >>>> 4845/140084552578816 flags 0 priority 100 >>>> 128 dd=36 locks held 0 write locks 0 pid/thread >>>> 4845/140084686862080 flags 0 priority 100 >>>> 12a dd=35 locks held 0 write locks 0 pid/thread >>>> 4845/140084720432896 flags 0 priority 100 >>>> 12b dd=34 locks held 0 write locks 0 pid/thread >>>> 4845/140084552578816 flags 0 priority 100 >>>> 12c dd=33 locks held 0 write locks 0 pid/thread >>>> 4845/140084670076672 flags 0 priority 100 >>>> 12d dd=32 locks held 0 write locks 0 pid/thread >>>> 4845/140084577756928 flags 0 priority 100 >>>> 12e dd=31 locks held 0 write locks 0 pid/thread >>>> 4845/140084712040192 flags 0 priority 100 >>>> 12f dd=30 locks held 0 write locks 0 pid/thread >>>> 4845/140084670076672 flags 0 priority 100 >>>> 130 dd=29 locks held 0 write locks 0 pid/thread >>>> 4845/140084577756928 flags 0 priority 100 >>>> 131 dd=28 locks held 0 write locks 0 pid/thread >>>> 4845/140084560971520 flags 0 priority 100 >>>> 132 dd=27 locks held 0 write locks 0 pid/thread >>>> 4845/140084670076672 flags 0 priority 100 >>>> 133 dd=26 locks held 0 write locks 0 pid/thread >>>> 4845/140084602935040 flags 0 priority 100 >>>> 134 dd=25 locks held 0 write locks 0 pid/thread >>>> 4845/140084552578816 flags 0 priority 100 >>>> 135 dd=24 locks held 0 write locks 0 pid/thread >>>> 4845/140084678469376 flags 0 priority 100 >>>> 136 dd=23 locks held 0 write locks 0 pid/thread >>>> 4845/140084544186112 flags 0 priority 100 >>>> 137 dd=22 locks held 0 write locks 0 pid/thread >>>> 4845/140084577756928 flags 0 priority 100 >>>> 138 dd=21 locks held 0 write locks 0 pid/thread >>>> 4845/140084703647488 flags 0 priority 100 >>>> 139 dd=20 locks held 0 write locks 0 pid/thread >>>> 4845/140084678469376 flags 0 priority 100 >>>> 13b dd=19 locks held 0 write locks 0 pid/thread >>>> 4845/140084812687104 flags 0 priority 100 >>>> 144 dd=18 locks held 0 write locks 0 pid/thread >>>> 4845/140084670076672 flags 0 priority 100 >>>> 156 dd=17 locks held 0 write locks 0 pid/thread >>>> 4845/140084594542336 flags 0 priority 100 >>>> 15a dd=16 locks held 0 write locks 0 pid/thread >>>> 4845/140084577756928 flags 0 priority 100 >>>> 168 dd=15 locks held 0 write locks 0 pid/thread >>>> 4845/140084653291264 flags 0 priority 100 >>>> 16a dd=14 locks held 0 write locks 0 pid/thread >>>> 4845/140084829472512 flags 0 priority 100 >>>> 17d dd=13 locks held 0 write locks 0 pid/thread >>>> 4845/140084829472512 flags 0 priority 100 >>>> 17e dd=12 locks held 0 write locks 0 pid/thread >>>> 4845/140084829472512 flags 0 priority 100 >>>> 17f dd=11 locks held 0 write locks 0 pid/thread >>>> 4845/140084586149632 flags 0 priority 100 >>>> 180 dd=10 locks held 0 write locks 0 pid/thread >>>> 4845/140084594542336 flags 0 priority 100 >>>> 181 dd= 9 locks held 0 write locks 0 pid/thread >>>> 4845/140084577756928 flags 0 priority 100 >>>> 193 dd= 8 locks held 0 write locks 0 pid/thread >>>> 4845/140084812687104 flags 0 priority 100 >>>> 196 dd= 7 locks held 0 write locks 0 pid/thread >>>> 4845/140084695254784 flags 0 priority 100 >>>> 1a1 dd= 6 locks held 0 write locks 0 pid/thread >>>> 4845/140084720432896 flags 0 priority 100 >>>> 1a2 dd= 5 locks held 0 write locks 0 pid/thread >>>> 4845/140084577756928 flags 0 priority 100 >>>> 1a3 dd= 4 locks held 0 write locks 0 pid/thread >>>> 4845/140084812687104 flags 0 priority 100 >>>> 1a4 dd= 3 locks held 0 write locks 0 pid/thread >>>> 4845/140084812687104 flags 0 priority 100 >>>> 1a5 dd= 2 locks held 0 write locks 0 pid/thread >>>> 4845/140084577756928 flags 0 priority 100 >>>> 1a6 dd= 1 locks held 0 write locks 0 pid/thread >>>> 4845/140084594542336 flags 0 priority 100 >>>> 800b3a33 dd= 0 locks held 26 write locks 22 pid/thread >>>> 4845/140084661683968 flags 0 priority 100 >>>> 800b3a33 WRITE 1 HELD userRoot/objectclass.db >>>> page 9 >>>> 800b3a33 WRITE 1 HELD userRoot/objectclass.db >>>> page 8 >>>> 800b3a33 WRITE 1 HELD userRoot/objectclass.db page >>>> 18 >>>> 800b3a33 WRITE 1 HELD userRoot/objectclass.db >>>> page 7 >>>> 800b3a33 WRITE 1 HELD userRoot/objectclass.db page >>>> 14 >>>> 800b3a33 WRITE 1 HELD userRoot/objectclass.db page >>>> 11 >>>> 800b3a33 WRITE 2 HELD userRoot/objectclass.db >>>> page 3 >>>> 800b3a33 WRITE 1 HELD userRoot/cn.db page >>>> 150 >>>> 800b3a33 WRITE 1 HELD userRoot/cn.db page >>>> 89 >>>> 800b3a33 WRITE 1 HELD userRoot/cn.db page >>>> 53 >>>> 800b3a33 WRITE 1 HELD userRoot/cn.db page >>>> 131 >>>> 800b3a33 WRITE 1 HELD userRoot/cn.db page >>>> 117 >>>> 800b3a33 WRITE 1 HELD userRoot/cn.db page >>>> 11 >>>> 800b3a33 WRITE 2 HELD userRoot/cn.db page >>>> 141 >>>> 800b3a33 WRITE 1 HELD userRoot/cn.db page >>>> 41 >>>> 800b3a33 WRITE 1 HELD userRoot/cn.db page >>>> 40 >>>> 800b3a33 WRITE 1 HELD userRoot/cn.db page >>>> 143 >>>> 800b3a33 WRITE 1 HELD userRoot/cn.db page >>>> 30 >>>> 800b3a33 WRITE 2 HELD userRoot/id2entry.db page >>>> 8184 >>>> 800b3a33 WRITE 1 HELD userRoot/id2entry.db page >>>> 8183 >>>> 800b3a33 WRITE 1 HELD userRoot/id2entry.db page >>>> 7779 >>>> 800b3a33 WRITE 1 HELD userRoot/id2entry.db >>>> page 0 >>>> 800b3a33 READ 3 HELD userRoot/entryrdn.db >>>> page 3 >>>> 800b3a33 READ 1 HELD userRoot/entryrdn.db page >>>> 36 >>>> 800b3a33 READ 1 HELD userRoot/entryrdn.db >>>> page 4 >>>> 800b3a33 READ 1 HELD userRoot/nsuniqueid.db page >>>> 86 >>>> =-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-= >>>> Locks grouped by object: >>>> Locker Mode Count Status ----------------- Object >>>> --------------- >>>> a4 READ 1 HELD userRoot/ipasudorunas.db >>>> handle 0 >>>> >>>> 9e READ 1 HELD userRoot/ipaMemberCertProfile.db >>>> handle 0 >>>> >>>> 10 READ 1 HELD ipaca/vlv#allrevokedcertspkitomcatindex.db >>>> handle 0 >>>> >>>> 800b3a33 WRITE 2 HELD userRoot/id2entry.db page >>>> 8184 >>>> >>>> 800b3a33 WRITE 1 HELD userRoot/id2entry.db page >>>> 8183 >>>> >>>> 800b3a33 WRITE 1 HELD userRoot/id2entry.db page >>>> 7779 >>>> >>>> a1 READ 1 HELD userRoot/memberdenycmd.db >>>> handle 0 >>>> >>>> a READ 1 HELD ipaca/vlv#allcertspkitomcatindex.db >>>> handle 0 >>>> >>>> a7 READ 1 HELD userRoot/sourcehost.db >>>> handle 0 >>>> >>>> 57 READ 1 HELD changelog/ancestorid.db >>>> handle 0 >>>> >>>> e5 READ 1 HELD ipaca/revInfo.db >>>> handle 0 >>>> >>>> e6 READ 1 HELD ipaca/revokedby.db >>>> handle 0 >>>> >>>> 11 READ 1 HELD ipaca/vlv#allrevokedcertsnotafterpkitomcatindex.db >>>> handle 0 >>>> >>>> 4e READ 1 HELD userRoot/parentid.db >>>> handle 0 >>>> >>>> 4 READ 1 HELD ipaca/id2entry.db >>>> handle 0 >>>> >>>> 58 READ 1 HELD changelog/numsubordinates.db >>>> handle 0 >>>> >>>> 67 READ 1 HELD ipaca/seeAlso.db >>>> handle 0 >>>> >>>> 2b READ 1 HELD ipaca/vlv#carevocationpkitomcatindex.db >>>> handle 0 >>>> >>>> ec READ 1 HELD ipaca/requeststate.db >>>> handle 0 >>>> >>>> 11b READ 1 HELD ipaca/notbefore.db >>>> handle 0 >>>> >>>> 7e READ 1 HELD userRoot/numsubordinates.db >>>> handle 0 >>>> >>>> a2 READ 1 HELD userRoot/memberallowcmd.db >>>> handle 0 >>>> >>>> 9c READ 1 HELD userRoot/owner.db >>>> handle 0 >>>> >>>> 1e READ 1 HELD ipaca/vlv#cacompleteenrollmentpkitomcatindex.db >>>> handle 0 >>>> >>>> 11c READ 1 HELD ipaca/notafter.db >>>> handle 0 >>>> >>>> 40 READ 1 HELD changelog/objectclass.db >>>> handle 0 >>>> >>>> 2e READ 1 HELD changelog/entryusn.db >>>> handle 0 >>>> >>>> 16 READ 1 HELD ipaca/vlv#allvalidcertsnotafterpkitomcatindex.db >>>> handle 0 >>>> >>>> 15 READ 1 HELD ipaca/vlv#allvalidcertspkitomcatindex.db >>>> handle 0 >>>> >>>> a5 READ 1 HELD userRoot/ipaMemberCa.db >>>> handle 0 >>>> >>>> 18 READ 1 HELD ipaca/vlv#caallpkitomcatindex.db >>>> handle 0 >>>> >>>> a0 READ 1 HELD userRoot/secretary.db >>>> handle 0 >>>> >>>> 81 READ 1 HELD userRoot/memberUser.db >>>> handle 0 >>>> >>>> 9f READ 1 HELD userRoot/ipaassignedidview.db >>>> handle 0 >>>> >>>> a3 READ 1 HELD userRoot/manager.db >>>> handle 0 >>>> >>>> d8 READ 1 HELD ipaca/description.db >>>> handle 0 >>>> >>>> a8 READ 1 HELD userRoot/memberservice.db >>>> handle 0 >>>> >>>> 44 READ 1 HELD ipaca/objectclass.db >>>> handle 0 >>>> >>>> ed READ 1 HELD ipaca/dateOfCreate.db >>>> handle 0 >>>> >>>> 800b3a33 WRITE 1 HELD userRoot/id2entry.db >>>> page 0 >>>> >>>> 2 READ 1 HELD userRoot/id2entry.db >>>> handle 0 >>>> >>>> 5b READ 1 HELD userRoot/krbPrincipalName.db >>>> handle 0 >>>> >>>> ee READ 1 HELD ipaca/requesttype.db >>>> handle 0 >>>> >>>> c8 READ 1 HELD ipaca/certstatus.db >>>> handle 0 >>>> >>>> 56 READ 1 HELD changelog/parentid.db >>>> handle 0 >>>> >>>> 83 READ 1 HELD userRoot/memberHost.db >>>> handle 0 >>>> >>>> 5e READ 1 HELD userRoot/ipakrbprincipalalias.db >>>> handle 0 >>>> >>>> 121 READ 1 HELD ipaca/issuedby.db >>>> handle 0 >>>> >>>> 6 READ 1 HELD ipaca/entryrdn.db >>>> handle 0 >>>> >>>> e7 READ 1 HELD ipaca/revokedOn.db >>>> handle 0 >>>> >>>> 4c READ 1 HELD ipaca/aci.db >>>> handle 0 >>>> >>>> 11d READ 1 HELD ipaca/duration.db >>>> handle 0 >>>> >>>> 14 READ 1 HELD ipaca/vlv#allrevokedorrevokedexpiredcertspkitomcatindex.db >>>> handle 0 >>>> >>>> 21 READ 1 HELD ipaca/vlv#caenrollmentpkitomcatindex.db >>>> handle 0 >>>> >>>> 55 READ 1 HELD changelog/targetuniqueid.db >>>> handle 0 >>>> >>>> 800b3a33 READ 1 HELD userRoot/entryrdn.db page >>>> 36 >>>> >>>> 2c READ 1 HELD changelog/id2entry.db >>>> handle 0 >>>> >>>> 800b3a33 READ 1 HELD userRoot/entryrdn.db >>>> page 4 >>>> >>>> 34 READ 1 HELD userRoot/entryrdn.db >>>> handle 0 >>>> >>>> 800b3a33 READ 3 HELD userRoot/entryrdn.db >>>> page 3 >>>> >>>> eb READ 1 HELD ipaca/requestid.db >>>> handle 0 >>>> >>>> 49 READ 1 HELD userRoot/aci.db >>>> handle 0 >>>> >>>> 76 READ 1 HELD userRoot/ipauniqueid.db >>>> handle 0 >>>> >>>> 63 READ 1 HELD changelog/seeAlso.db >>>> handle 0 >>>> >>>> 7c READ 1 HELD userRoot/fqdn.db >>>> handle 0 >>>> >>>> db READ 1 HELD ipaca/cn.db >>>> handle 0 >>>> >>>> ef READ 1 HELD ipaca/numsubordinates.db >>>> handle 0 >>>> >>>> 32 READ 1 HELD ipaca/entryusn.db >>>> handle 0 >>>> >>>> 1d READ 1 HELD ipaca/vlv#cacompletepkitomcatindex.db >>>> handle 0 >>>> >>>> dd READ 1 HELD ipaca/ancestorid.db >>>> handle 0 >>>> >>>> a9 READ 1 HELD userRoot/ipatokenradiusconfiglink.db >>>> handle 0 >>>> >>>> f0 READ 1 HELD userRoot/userCertificate.db >>>> handle 0 >>>> >>>> 11a READ 1 HELD ipaca/metaInfo.db >>>> handle 0 >>>> >>>> 7d READ 1 HELD userRoot/managedby.db >>>> handle 0 >>>> >>>> ea READ 1 HELD ipaca/nsuniqueid.db >>>> handle 0 >>>> >>>> c5 READ 1 HELD userRoot/uidnumber.db >>>> handle 0 >>>> >>>> d1 READ 1 HELD userRoot/nscpEntryDN.db >>>> handle 0 >>>> >>>> 120 READ 1 HELD ipaca/extension.db >>>> handle 0 >>>> >>>> 800b3a33 WRITE 1 HELD userRoot/cn.db page >>>> 150 >>>> >>>> 69 READ 1 HELD ipaca/parentid.db >>>> handle 0 >>>> >>>> 800b3a33 WRITE 1 HELD userRoot/cn.db page >>>> 131 >>>> >>>> 800b3a33 WRITE 1 HELD userRoot/cn.db page >>>> 143 >>>> >>>> 800b3a33 WRITE 2 HELD userRoot/cn.db page >>>> 141 >>>> >>>> 20 READ 1 HELD ipaca/vlv#cacompleterevocationpkitomcatindex.db >>>> handle 0 >>>> >>>> 8e READ 1 HELD userRoot/macAddress.db >>>> handle 0 >>>> >>>> e READ 1 HELD ipaca/vlv#allnonrevokedcertspkitomcatindex.db >>>> handle 0 >>>> >>>> 17 READ 1 HELD ipaca/vlv#allvalidorrevokedcertspkitomcatindex.db >>>> handle 0 >>>> >>>> 800b3a33 WRITE 1 HELD userRoot/cn.db page >>>> 30 >>>> >>>> 800b3a33 WRITE 1 HELD userRoot/objectclass.db page >>>> 14 >>>> >>>> 800b3a33 WRITE 1 HELD userRoot/objectclass.db >>>> page 9 >>>> 116 READ 1 WAIT userRoot/objectclass.db >>>> page 9 >>>> 115 READ 1 WAIT userRoot/objectclass.db >>>> page 9 >>>> 92 READ 1 WAIT userRoot/objectclass.db >>>> page 9 >>>> >>>> 800b3a33 WRITE 1 HELD userRoot/objectclass.db >>>> page 8 >>>> >>>> 7b READ 1 HELD userRoot/cn.db >>>> handle 0 >>>> >>>> 800b3a33 WRITE 1 HELD userRoot/objectclass.db page >>>> 11 >>>> >>>> 800b3a33 WRITE 1 HELD userRoot/objectclass.db >>>> page 7 >>>> >>>> 800b3a33 WRITE 1 HELD userRoot/cn.db page >>>> 11 >>>> >>>> 38 READ 1 HELD userRoot/objectclass.db >>>> handle 0 >>>> >>>> 800b3a33 WRITE 2 HELD userRoot/objectclass.db >>>> page 3 >>>> >>>> 800b3a33 WRITE 1 HELD userRoot/cn.db page >>>> 53 >>>> >>>> 800b3a33 WRITE 1 HELD userRoot/objectclass.db page >>>> 18 >>>> 6e READ 1 WAIT userRoot/objectclass.db page >>>> 18 >>>> 94 READ 1 WAIT userRoot/objectclass.db page >>>> 18 >>>> >>>> 800b3a33 WRITE 1 HELD userRoot/cn.db page >>>> 41 >>>> >>>> 800b3a33 WRITE 1 HELD userRoot/cn.db page >>>> 40 >>>> >>>> 800b3a33 WRITE 1 HELD userRoot/cn.db page >>>> 89 >>>> >>>> 800b3a33 WRITE 1 HELD userRoot/cn.db page >>>> 117 >>>> >>>> 30 READ 1 HELD userRoot/entryusn.db >>>> handle 0 >>>> >>>> 3c READ 1 HELD changelog/entryrdn.db >>>> handle 0 >>>> >>>> 97 READ 1 HELD userRoot/uid.db >>>> handle 0 >>>> >>>> 47 READ 1 HELD changelog/aci.db >>>> handle 0 >>>> >>>> e3 READ 1 HELD ipaca/serialno.db >>>> handle 0 >>>> >>>> 7a READ 1 HELD userRoot/nsuniqueid.db >>>> handle 0 >>>> >>>> 800b3a33 READ 1 HELD userRoot/nsuniqueid.db page >>>> 86 >>>> >>>> 51 READ 1 HELD changelog/nsuniqueid.db >>>> handle 0 >>>> >>>> 11e READ 1 HELD ipaca/subjectname.db >>>> handle 0 >>>> >>>> 7f READ 1 HELD userRoot/member.db >>>> handle 0 >>>> >>>> 11f READ 1 HELD ipaca/publicKeyData.db >>>> handle 0 >>>> >>>> 9b READ 1 HELD userRoot/uniquemember.db >>>> handle 0 >>>> >>>> 52 READ 1 HELD changelog/changenumber.db >>>> handle 0 >>>> >>>> ce READ 1 HELD userRoot/gidnumber.db >>>> handle 0 >>>> >>>> 65 READ 1 HELD userRoot/seeAlso.db >>>> handle 0 >>>> >>>> 9d READ 1 HELD userRoot/ipaallowedtarget.db >>>> handle 0 >>>> >>>> a6 READ 1 HELD userRoot/ipasudorunasgroup.db >>>> handle 0 >>>> >>>> 3a READ 1 HELD userRoot/ancestorid.db >>>> handle 0 >>>> >>>> >>>> >>>> >>>> pstack output >>>> >>>> root at ipa-master ]# pstack 4845 >>>> Thread 44 (Thread 0x7f681dd3a700 (LWP 4847)): >>>> #0 0x00007f683cd719b3 in select () from /lib64/libc.so.6 >>>> #1 0x00007f683f4fc0e9 in DS_Sleep () from >>>> /usr/lib64/dirsrv/libslapd.so.0 >>>> #2 0x00007f6832b4a907 in deadlock_threadmain () from >>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>> #3 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so >>>> #4 0x00007f683d04cdc5 in start_thread () from /lib64/libpthread.so.0 >>>> #5 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 >>>> Thread 43 (Thread 0x7f681d539700 (LWP 4848)): >>>> #0 0x00007f683cd719b3 in select () from /lib64/libc.so.6 >>>> #1 0x00007f68376d8bad in __os_yield () from /lib64/libdb-5.3.so >>>> #2 0x00007f68376d42b3 in __memp_sync_int () from /lib64/libdb-5.3.so >>>> #3 0x00007f68376e4752 in __txn_checkpoint () from /lib64/libdb-5.3.so >>>> #4 0x00007f68376e4b74 in __txn_checkpoint_pp () from /lib64/ >>>> libdb-5.3.so >>>> #5 0x00007f6832b4ea87 in checkpoint_threadmain () from >>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>> #6 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so >>>> #7 0x00007f683d04cdc5 in start_thread () from /lib64/libpthread.so.0 >>>> #8 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 >>>> Thread 42 (Thread 0x7f681cd38700 (LWP 4849)): >>>> #0 0x00007f683cd719b3 in select () from /lib64/libc.so.6 >>>> #1 0x00007f683f4fc0e9 in DS_Sleep () from >>>> /usr/lib64/dirsrv/libslapd.so.0 >>>> #2 0x00007f6832b4ab7f in trickle_threadmain () from >>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>> #3 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so >>>> #4 0x00007f683d04cdc5 in start_thread () from /lib64/libpthread.so.0 >>>> #5 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 >>>> Thread 41 (Thread 0x7f681c537700 (LWP 4850)): >>>> #0 0x00007f683cd719b3 in select () from /lib64/libc.so.6 >>>> #1 0x00007f683f4fc0e9 in DS_Sleep () from >>>> /usr/lib64/dirsrv/libslapd.so.0 >>>> #2 0x00007f6832b45707 in perf_threadmain () from >>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>> #3 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so >>>> #4 0x00007f683d04cdc5 in start_thread () from /lib64/libpthread.so.0 >>>> #5 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 >>>> Thread 40 (Thread 0x7f681bb2d700 (LWP 4851)): >>>> #0 0x00007f683d0506d5 in >>>> pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 >>>> #1 0x00007f683d6a6050 in PR_WaitCondVar () from /lib64/libnspr4.so >>>> #2 0x00007f683f4eb198 in slapi_wait_condvar () from >>>> /usr/lib64/dirsrv/libslapd.so.0 >>>> #3 0x00007f683587d62e in cos_cache_wait_on_change () from >>>> /usr/lib64/dirsrv/plugins/libcos-plugin.so >>>> #4 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so >>>> #5 0x00007f683d04cdc5 in start_thread () from /lib64/libpthread.so.0 >>>> #6 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 >>>> Thread 39 (Thread 0x7f681b32c700 (LWP 4852)): >>>> #0 0x00007f683d0506d5 in >>>> pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 >>>> #1 0x00007f683d6a6050 in PR_WaitCondVar () from /lib64/libnspr4.so >>>> #2 0x00007f683f4eb198 in slapi_wait_condvar () from >>>> /usr/lib64/dirsrv/libslapd.so.0 >>>> #3 0x00007f68311e8eed in roles_cache_wait_on_change () from >>>> /usr/lib64/dirsrv/plugins/libroles-plugin.so >>>> #4 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so >>>> #5 0x00007f683d04cdc5 in start_thread () from /lib64/libpthread.so.0 >>>> #6 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 >>>> Thread 38 (Thread 0x7f681ab2b700 (LWP 4853)): >>>> #0 0x00007f683d0506d5 in >>>> pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 >>>> #1 0x00007f683d6a6050 in PR_WaitCondVar () from /lib64/libnspr4.so >>>> #2 0x00007f683f4eb198 in slapi_wait_condvar () from >>>> /usr/lib64/dirsrv/libslapd.so.0 >>>> #3 0x00007f68311e8eed in roles_cache_wait_on_change () from >>>> /usr/lib64/dirsrv/plugins/libroles-plugin.so >>>> #4 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so >>>> #5 0x00007f683d04cdc5 in start_thread () from /lib64/libpthread.so.0 >>>> #6 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 >>>> Thread 37 (Thread 0x7f681a32a700 (LWP 4854)): >>>> #0 0x00007f683d0506d5 in >>>> pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 >>>> #1 0x00007f683d6a6050 in PR_WaitCondVar () from /lib64/libnspr4.so >>>> #2 0x00007f683f4eb198 in slapi_wait_condvar () from >>>> /usr/lib64/dirsrv/libslapd.so.0 >>>> #3 0x00007f68311e8eed in roles_cache_wait_on_change () from >>>> /usr/lib64/dirsrv/plugins/libroles-plugin.so >>>> #4 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so >>>> #5 0x00007f683d04cdc5 in start_thread () from /lib64/libpthread.so.0 >>>> #6 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 >>>> Thread 36 (Thread 0x7f6819b29700 (LWP 4855)): >>>> #0 0x00007f683d050a82 in pthread_cond_timedwait@@GLIBC_2.3.2 () from >>>> /lib64/libpthread.so.0 >>>> #1 0x00007f683d6a5b07 in pt_TimedWait () from /lib64/libnspr4.so >>>> #2 0x00007f683d6a5fce in PR_WaitCondVar () from /lib64/libnspr4.so >>>> #3 0x00007f683f9a4d03 in housecleaning () >>>> #4 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so >>>> #5 0x00007f683d04cdc5 in start_thread () from /lib64/libpthread.so.0 >>>> #6 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 >>>> Thread 35 (Thread 0x7f6819328700 (LWP 4856)): >>>> #0 0x00007f683d050a82 in pthread_cond_timedwait@@GLIBC_2.3.2 () from >>>> /lib64/libpthread.so.0 >>>> #1 0x00007f683d6a5b07 in pt_TimedWait () from /lib64/libnspr4.so >>>> #2 0x00007f683d6a5fce in PR_WaitCondVar () from /lib64/libnspr4.so >>>> #3 0x00007f683f4911e8 in eq_loop () from /usr/lib64/dirsrv/libslapd.so. >>>> 0 >>>> #4 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so >>>> #5 0x00007f683d04cdc5 in start_thread () from /lib64/libpthread.so.0 >>>> #6 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 >>>> Thread 34 (Thread 0x7f680bfff700 (LWP 4859)): >>>> #0 0x00007f683d0506d5 in >>>> pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 >>>> #1 0x00007f683d6a6050 in PR_WaitCondVar () from /lib64/libnspr4.so >>>> #2 0x00007f683f999fee in connection_wait_for_new_work () >>>> #3 0x00007f683f99b719 in connection_threadmain () >>>> #4 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so >>>> #5 0x00007f683d04cdc5 in start_thread () from /lib64/libpthread.so.0 >>>> #6 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 >>>> Thread 33 (Thread 0x7f680b7fe700 (LWP 4860)): >>>> #0 0x00007f683d0506d5 in >>>> pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 >>>> #1 0x00007f683d6a6050 in PR_WaitCondVar () from /lib64/libnspr4.so >>>> #2 0x00007f683f999fee in connection_wait_for_new_work () >>>> #3 0x00007f683f99b719 in connection_threadmain () >>>> #4 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so >>>> #5 0x00007f683d04cdc5 in start_thread () from /lib64/libpthread.so.0 >>>> #6 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 >>>> Thread 32 (Thread 0x7f680affd700 (LWP 4861)): >>>> #0 0x00007f683d0506d5 in >>>> pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 >>>> #1 0x00007f68375ac2f3 in __db_hybrid_mutex_suspend () from /lib64/ >>>> libdb-5.3.so >>>> #2 0x00007f68375ab640 in __db_tas_mutex_lock () from /lib64/ >>>> libdb-5.3.so >>>> #3 0x00007f6837655d3a in __lock_get_internal () from /lib64/ >>>> libdb-5.3.so >>>> #4 0x00007f6837656820 in __lock_get () from /lib64/libdb-5.3.so >>>> #5 0x00007f6837682142 in __db_lget () from /lib64/libdb-5.3.so >>>> #6 0x00007f68375c9605 in __bam_search () from /lib64/libdb-5.3.so >>>> #7 0x00007f68375b4256 in __bamc_search () from /lib64/libdb-5.3.so >>>> #8 0x00007f68375b5d0f in __bamc_get () from /lib64/libdb-5.3.so >>>> #9 0x00007f683766eca6 in __dbc_iget () from /lib64/libdb-5.3.so >>>> #10 0x00007f683767db02 in __dbc_get_pp () from /lib64/libdb-5.3.so >>>> #11 0x00007f6832b59680 in idl_new_fetch () from >>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>> #12 0x00007f6832b67b26 in index_read_ext_allids () from >>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>> #13 0x00007f6832b52284 in keys2idl () from >>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>> #14 0x00007f6832b529e3 in ava_candidates.isra.0 () from >>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>> #15 0x00007f6832b52fd2 in filter_candidates_ext () from >>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>> #16 0x00007f6832b54096 in list_candidates () from >>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>> #17 0x00007f6832b52f40 in filter_candidates_ext () from >>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>> #18 0x00007f6832b54096 in list_candidates () from >>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>> #19 0x00007f6832b52f40 in filter_candidates_ext () from >>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>> #20 0x00007f6832b54096 in list_candidates () from >>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>> #21 0x00007f6832b52f40 in filter_candidates_ext () from >>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>> #22 0x00007f6832b8e80c in subtree_candidates () from >>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>> #23 0x00007f6832b8fe7f in ldbm_back_search () from >>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>> #24 0x00007f683f4bd1d6 in op_shared_search () from >>>> /usr/lib64/dirsrv/libslapd.so.0 >>>> #25 0x00007f683f9ae33e in do_search () >>>> #26 0x00007f683f99c3c3 in connection_threadmain () >>>> #27 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so >>>> #28 0x00007f683d04cdc5 in start_thread () from /lib64/libpthread.so.0 >>>> #29 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 >>>> Thread 31 (Thread 0x7f680a7fc700 (LWP 4862)): >>>> #0 0x00007f683d0506d5 in >>>> pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 >>>> #1 0x00007f683d6a62c3 in PR_EnterMonitor () from /lib64/libnspr4.so >>>> #2 0x00007f6832b4d3e6 in dblayer_txn_begin () from >>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>> #3 0x00007f6832b89007 in ldbm_back_modify () from >>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>> #4 0x00007f683f4b6541 in op_shared_modify () from >>>> /usr/lib64/dirsrv/libslapd.so.0 >>>> #5 0x00007f683f4b7004 in modify_internal_pb () from >>>> /usr/lib64/dirsrv/libslapd.so.0 >>>> #6 0x00007f683483bbbb in ipalockout_postop () from >>>> /usr/lib64/dirsrv/plugins/libipa_lockout.so >>>> #7 0x00007f683f4c8848 in plugin_call_func () from >>>> /usr/lib64/dirsrv/libslapd.so.0 >>>> #8 0x00007f683f4c8b13 in plugin_call_plugins () from >>>> /usr/lib64/dirsrv/libslapd.so.0 >>>> #9 0x00007f683f9954d8 in do_bind () >>>> #10 0x00007f683f99c3fd in connection_threadmain () >>>> #11 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so >>>> #12 0x00007f683d04cdc5 in start_thread () from /lib64/libpthread.so.0 >>>> #13 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 >>>> Thread 30 (Thread 0x7f6809ffb700 (LWP 4863)): >>>> #0 0x00007f683d0506d5 in >>>> pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 >>>> #1 0x00007f68375ac2f3 in __db_hybrid_mutex_suspend () from /lib64/ >>>> libdb-5.3.so >>>> #2 0x00007f68375ab640 in __db_tas_mutex_lock () from /lib64/ >>>> libdb-5.3.so >>>> #3 0x00007f6837655d3a in __lock_get_internal () from /lib64/ >>>> libdb-5.3.so >>>> #4 0x00007f6837656820 in __lock_get () from /lib64/libdb-5.3.so >>>> #5 0x00007f6837682142 in __db_lget () from /lib64/libdb-5.3.so >>>> #6 0x00007f68375c9605 in __bam_search () from /lib64/libdb-5.3.so >>>> #7 0x00007f68375b4256 in __bamc_search () from /lib64/libdb-5.3.so >>>> #8 0x00007f68375b5d0f in __bamc_get () from /lib64/libdb-5.3.so >>>> #9 0x00007f683766eca6 in __dbc_iget () from /lib64/libdb-5.3.so >>>> #10 0x00007f683767db02 in __dbc_get_pp () from /lib64/libdb-5.3.so >>>> #11 0x00007f6832b59680 in idl_new_fetch () from >>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>> #12 0x00007f6832b67b26 in index_read_ext_allids () from >>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>> #13 0x00007f6832b52284 in keys2idl () from >>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>> #14 0x00007f6832b529e3 in ava_candidates.isra.0 () from >>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>> #15 0x00007f6832b52fd2 in filter_candidates_ext () from >>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>> #16 0x00007f6832b54096 in list_candidates () from >>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>> #17 0x00007f6832b52f40 in filter_candidates_ext () from >>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>> #18 0x00007f6832b8e80c in subtree_candidates () from >>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>> #19 0x00007f6832b8fe7f in ldbm_back_search () from >>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>> #20 0x00007f683f4bd1d6 in op_shared_search () from >>>> /usr/lib64/dirsrv/libslapd.so.0 >>>> #21 0x00007f683f4cd94e in search_internal_callback_pb () from >>>> /usr/lib64/dirsrv/libslapd.so.0 >>>> #22 0x00007f683f4cdbe8 in search_internal_pb () from >>>> /usr/lib64/dirsrv/libslapd.so.0 >>>> #23 0x00007f683f9ac49b in ids_sasl_canon_user () >>>> #24 0x00007f683d46c93b in _sasl_canon_user () from /lib64/libsasl2.so.3 >>>> #25 0x00007f683d46cc4c in _sasl_canon_user_lookup () from >>>> /lib64/libsasl2.so.3 >>>> #26 0x00007f6837374fb9 in gssapi_server_mech_step () from >>>> /usr/lib64/sasl2/libgssapiv2.so >>>> #27 0x00007f683d478b9b in sasl_server_step () from /lib64/libsasl2.so.3 >>>> #28 0x00007f683f9ad83b in ids_sasl_check_bind () >>>> #29 0x00007f683f99561a in do_bind () >>>> #30 0x00007f683f99c3fd in connection_threadmain () >>>> #31 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so >>>> #32 0x00007f683d04cdc5 in start_thread () from /lib64/libpthread.so.0 >>>> #33 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 >>>> Thread 29 (Thread 0x7f68097fa700 (LWP 4864)): >>>> #0 0x00007f683d0506d5 in >>>> pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 >>>> #1 0x00007f683d6a6050 in PR_WaitCondVar () from /lib64/libnspr4.so >>>> #2 0x00007f683f999fee in connection_wait_for_new_work () >>>> #3 0x00007f683f99b719 in connection_threadmain () >>>> #4 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so >>>> #5 0x00007f683d04cdc5 in start_thread () from /lib64/libpthread.so.0 >>>> #6 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 >>>> Thread 28 (Thread 0x7f6808ff9700 (LWP 4865)): >>>> #0 0x00007f683d0506d5 in >>>> pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 >>>> #1 0x00007f683d6a6050 in PR_WaitCondVar () from /lib64/libnspr4.so >>>> #2 0x00007f683f999fee in connection_wait_for_new_work () >>>> #3 0x00007f683f99b719 in connection_threadmain () >>>> #4 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so >>>> #5 0x00007f683d04cdc5 in start_thread () from /lib64/libpthread.so.0 >>>> #6 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 >>>> Thread 27 (Thread 0x7f6803fff700 (LWP 4866)): >>>> #0 0x00007f683d0506d5 in >>>> pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 >>>> #1 0x00007f683d6a6050 in PR_WaitCondVar () from /lib64/libnspr4.so >>>> #2 0x00007f683f999fee in connection_wait_for_new_work () >>>> #3 0x00007f683f99b719 in connection_threadmain () >>>> #4 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so >>>> #5 0x00007f683d04cdc5 in start_thread () from /lib64/libpthread.so.0 >>>> #6 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 >>>> Thread 26 (Thread 0x7f68037fe700 (LWP 4867)): >>>> #0 0x00007f683d0506d5 in >>>> pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 >>>> #1 0x00007f683d6a6050 in PR_WaitCondVar () from /lib64/libnspr4.so >>>> #2 0x00007f683f999fee in connection_wait_for_new_work () >>>> #3 0x00007f683f99b719 in connection_threadmain () >>>> #4 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so >>>> #5 0x00007f683d04cdc5 in start_thread () from /lib64/libpthread.so.0 >>>> #6 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 >>>> Thread 25 (Thread 0x7f6802ffd700 (LWP 4868)): >>>> #0 0x00007f683d0506d5 in >>>> pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 >>>> #1 0x00007f683d6a6050 in PR_WaitCondVar () from /lib64/libnspr4.so >>>> #2 0x00007f683f999fee in connection_wait_for_new_work () >>>> #3 0x00007f683f99b719 in connection_threadmain () >>>> #4 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so >>>> #5 0x00007f683d04cdc5 in start_thread () from /lib64/libpthread.so.0 >>>> #6 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 >>>> Thread 24 (Thread 0x7f68027fc700 (LWP 4869)): >>>> #0 0x00007f683d0506d5 in >>>> pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 >>>> #1 0x00007f68375ac2f3 in __db_hybrid_mutex_suspend () from /lib64/ >>>> libdb-5.3.so >>>> #2 0x00007f68375abb37 in __db_tas_mutex_readlock () from /lib64/ >>>> libdb-5.3.so >>>> #3 0x00007f68376c5d35 in __memp_fget () from /lib64/libdb-5.3.so >>>> #4 0x00007f683766f333 in __dbc_iget () from /lib64/libdb-5.3.so >>>> #5 0x00007f683767db02 in __dbc_get_pp () from /lib64/libdb-5.3.so >>>> #6 0x00007f6832b59822 in idl_new_fetch () from >>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>> #7 0x00007f6832b67b26 in index_read_ext_allids () from >>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>> #8 0x00007f6832b52284 in keys2idl () from >>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>> #9 0x00007f6832b529e3 in ava_candidates.isra.0 () from >>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>> #10 0x00007f6832b52fd2 in filter_candidates_ext () from >>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>> #11 0x00007f6832b54096 in list_candidates () from >>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>> #12 0x00007f6832b52f40 in filter_candidates_ext () from >>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>> #13 0x00007f6832b54096 in list_candidates () from >>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>> #14 0x00007f6832b52f40 in filter_candidates_ext () from >>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>> #15 0x00007f6832b54096 in list_candidates () from >>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>> #16 0x00007f6832b52f40 in filter_candidates_ext () from >>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>> #17 0x00007f6832b8e80c in subtree_candidates () from >>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>> #18 0x00007f6832b8fe7f in ldbm_back_search () from >>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>> #19 0x00007f683f4bd1d6 in op_shared_search () from >>>> /usr/lib64/dirsrv/libslapd.so.0 >>>> #20 0x00007f683f9ae33e in do_search () >>>> #21 0x00007f683f99c3c3 in connection_threadmain () >>>> #22 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so >>>> #23 0x00007f683d04cdc5 in start_thread () from /lib64/libpthread.so.0 >>>> #24 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 >>>> Thread 23 (Thread 0x7f6801ffb700 (LWP 4870)): >>>> #0 0x00007f683d0506d5 in >>>> pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 >>>> #1 0x00007f683d6a6050 in PR_WaitCondVar () from /lib64/libnspr4.so >>>> #2 0x00007f683f999fee in connection_wait_for_new_work () >>>> #3 0x00007f683f99b719 in connection_threadmain () >>>> #4 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so >>>> #5 0x00007f683d04cdc5 in start_thread () from /lib64/libpthread.so.0 >>>> #6 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 >>>> Thread 22 (Thread 0x7f68017fa700 (LWP 4871)): >>>> #0 0x00007f683d0506d5 in >>>> pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 >>>> #1 0x00007f683d6a6050 in PR_WaitCondVar () from /lib64/libnspr4.so >>>> #2 0x00007f683f999fee in connection_wait_for_new_work () >>>> #3 0x00007f683f99b719 in connection_threadmain () >>>> #4 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so >>>> #5 0x00007f683d04cdc5 in start_thread () from /lib64/libpthread.so.0 >>>> #6 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 >>>> Thread 21 (Thread 0x7f6800ff9700 (LWP 4872)): >>>> #0 0x00007f683d0506d5 in >>>> pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 >>>> #1 0x00007f683d6a6050 in PR_WaitCondVar () from /lib64/libnspr4.so >>>> #2 0x00007f683f999fee in connection_wait_for_new_work () >>>> #3 0x00007f683f99b719 in connection_threadmain () >>>> #4 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so >>>> #5 0x00007f683d04cdc5 in start_thread () from /lib64/libpthread.so.0 >>>> #6 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 >>>> Thread 20 (Thread 0x7f68007f8700 (LWP 4873)): >>>> #0 0x00007f683d0506d5 in >>>> pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 >>>> #1 0x00007f68375ac2f3 in __db_hybrid_mutex_suspend () from /lib64/ >>>> libdb-5.3.so >>>> #2 0x00007f68375ab640 in __db_tas_mutex_lock () from /lib64/ >>>> libdb-5.3.so >>>> #3 0x00007f68376c5e1b in __memp_fget () from /lib64/libdb-5.3.so >>>> #4 0x00007f68375c9211 in __bam_search () from /lib64/libdb-5.3.so >>>> #5 0x00007f68375b4256 in __bamc_search () from /lib64/libdb-5.3.so >>>> #6 0x00007f68375b86c4 in __bamc_put () from /lib64/libdb-5.3.so >>>> #7 0x00007f683766f6d5 in __dbc_iput () from /lib64/libdb-5.3.so >>>> #8 0x00007f683766aa0e in __db_put () from /lib64/libdb-5.3.so >>>> #9 0x00007f683767ffa4 in __db_put_pp () from /lib64/libdb-5.3.so >>>> #10 0x00007f6832b5a4cf in idl_new_insert_key () from >>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>> #11 0x00007f6832b695b5 in addordel_values_sv.isra.3 () from >>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>> #12 0x00007f6832b6a25a in index_addordel_values_ext_sv () from >>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>> #13 0x00007f6832b6a694 in index_addordel_values_sv () from >>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>> #14 0x00007f6832b6a75a in index_addordel_entry () from >>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>> #15 0x00007f6832b6d5fa in ldbm_back_add () from >>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>> #16 0x00007f683f46fcca in op_shared_add () from >>>> /usr/lib64/dirsrv/libslapd.so.0 >>>> #17 0x00007f683f471020 in do_add () from /usr/lib64/dirsrv/libslapd.so. >>>> 0 >>>> #18 0x00007f683f99c303 in connection_threadmain () >>>> #19 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so >>>> #20 0x00007f683d04cdc5 in start_thread () from /lib64/libpthread.so.0 >>>> #21 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 >>>> Thread 19 (Thread 0x7f67ffff7700 (LWP 4874)): >>>> #0 0x00007f683d0506d5 in >>>> pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 >>>> #1 0x00007f683d6a62c3 in PR_EnterMonitor () from /lib64/libnspr4.so >>>> #2 0x00007f6832b4d3e6 in dblayer_txn_begin () from >>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>> #3 0x00007f6832b89007 in ldbm_back_modify () from >>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>> #4 0x00007f683f4b6541 in op_shared_modify () from >>>> /usr/lib64/dirsrv/libslapd.so.0 >>>> #5 0x00007f683f4b7004 in modify_internal_pb () from >>>> /usr/lib64/dirsrv/libslapd.so.0 >>>> #6 0x00007f6832dd3b60 in ipapwd_apply_mods () from >>>> /usr/lib64/dirsrv/plugins/libipa_pwd_extop.so >>>> #7 0x00007f6832dd78e0 in store_new_keys () from >>>> /usr/lib64/dirsrv/plugins/libipa_pwd_extop.so >>>> #8 0x00007f6832dd9ad0 in ipapwd_extop () from >>>> /usr/lib64/dirsrv/plugins/libipa_pwd_extop.so >>>> #9 0x00007f683f4c6ba2 in plugin_call_exop_plugins () from >>>> /usr/lib64/dirsrv/libslapd.so.0 >>>> #10 0x00007f683f9a2efe in do_extended () >>>> #11 0x00007f683f99c41a in connection_threadmain () >>>> #12 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so >>>> #13 0x00007f683d04cdc5 in start_thread () from /lib64/libpthread.so.0 >>>> #14 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 >>>> Thread 18 (Thread 0x7f67ff7f6700 (LWP 4875)): >>>> #0 0x00007f683d0506d5 in >>>> pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 >>>> #1 0x00007f683d6a6050 in PR_WaitCondVar () from /lib64/libnspr4.so >>>> #2 0x00007f683f999fee in connection_wait_for_new_work () >>>> #3 0x00007f683f99b719 in connection_threadmain () >>>> #4 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so >>>> #5 0x00007f683d04cdc5 in start_thread () from /lib64/libpthread.so.0 >>>> #6 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 >>>> Thread 17 (Thread 0x7f67feff5700 (LWP 4876)): >>>> #0 0x00007f683d0506d5 in >>>> pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 >>>> #1 0x00007f68375ac2f3 in __db_hybrid_mutex_suspend () from /lib64/ >>>> libdb-5.3.so >>>> #2 0x00007f68375ab640 in __db_tas_mutex_lock () from /lib64/ >>>> libdb-5.3.so >>>> #3 0x00007f6837655d3a in __lock_get_internal () from /lib64/ >>>> libdb-5.3.so >>>> #4 0x00007f6837656820 in __lock_get () from /lib64/libdb-5.3.so >>>> #5 0x00007f6837682142 in __db_lget () from /lib64/libdb-5.3.so >>>> #6 0x00007f68375c9605 in __bam_search () from /lib64/libdb-5.3.so >>>> #7 0x00007f68375b4256 in __bamc_search () from /lib64/libdb-5.3.so >>>> #8 0x00007f68375b5d0f in __bamc_get () from /lib64/libdb-5.3.so >>>> #9 0x00007f683766eca6 in __dbc_iget () from /lib64/libdb-5.3.so >>>> #10 0x00007f683767db02 in __dbc_get_pp () from /lib64/libdb-5.3.so >>>> #11 0x00007f6832b59680 in idl_new_fetch () from >>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>> #12 0x00007f6832b67b26 in index_read_ext_allids () from >>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>> #13 0x00007f6832b52284 in keys2idl () from >>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>> #14 0x00007f6832b529e3 in ava_candidates.isra.0 () from >>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>> #15 0x00007f6832b52fd2 in filter_candidates_ext () from >>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>> #16 0x00007f6832b54096 in list_candidates () from >>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>> #17 0x00007f6832b52f40 in filter_candidates_ext () from >>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>> #18 0x00007f6832b8e80c in subtree_candidates () from >>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>> #19 0x00007f6832b8fe7f in ldbm_back_search () from >>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>> #20 0x00007f683f4bd1d6 in op_shared_search () from >>>> /usr/lib64/dirsrv/libslapd.so.0 >>>> #21 0x00007f683f4cd94e in search_internal_callback_pb () from >>>> /usr/lib64/dirsrv/libslapd.so.0 >>>> #22 0x00007f683f4cdbe8 in search_internal_pb () from >>>> /usr/lib64/dirsrv/libslapd.so.0 >>>> #23 0x00007f6832dd75dc in get_entry_by_principal () from >>>> /usr/lib64/dirsrv/plugins/libipa_pwd_extop.so >>>> #24 0x00007f6832dd8eb1 in ipapwd_extop () from >>>> /usr/lib64/dirsrv/plugins/libipa_pwd_extop.so >>>> #25 0x00007f683f4c6ba2 in plugin_call_exop_plugins () from >>>> /usr/lib64/dirsrv/libslapd.so.0 >>>> #26 0x00007f683f9a2efe in do_extended () >>>> #27 0x00007f683f99c41a in connection_threadmain () >>>> #28 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so >>>> #29 0x00007f683d04cdc5 in start_thread () from /lib64/libpthread.so.0 >>>> #30 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 >>>> Thread 16 (Thread 0x7f67fe7f4700 (LWP 4877)): >>>> #0 0x00007f683d0506d5 in >>>> pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 >>>> #1 0x00007f683d6a6050 in PR_WaitCondVar () from /lib64/libnspr4.so >>>> #2 0x00007f683f999fee in connection_wait_for_new_work () >>>> #3 0x00007f683f99b719 in connection_threadmain () >>>> #4 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so >>>> #5 0x00007f683d04cdc5 in start_thread () from /lib64/libpthread.so.0 >>>> #6 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 >>>> Thread 15 (Thread 0x7f67fdff3700 (LWP 4878)): >>>> #0 0x00007f683d0506d5 in >>>> pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 >>>> #1 0x00007f683d6a6050 in PR_WaitCondVar () from /lib64/libnspr4.so >>>> #2 0x00007f683f999fee in connection_wait_for_new_work () >>>> #3 0x00007f683f99b719 in connection_threadmain () >>>> #4 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so >>>> #5 0x00007f683d04cdc5 in start_thread () from /lib64/libpthread.so.0 >>>> #6 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 >>>> Thread 14 (Thread 0x7f67fd7f2700 (LWP 4879)): >>>> #0 0x00007f683d0506d5 in >>>> pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 >>>> #1 0x00007f683d6a6050 in PR_WaitCondVar () from /lib64/libnspr4.so >>>> #2 0x00007f683f999fee in connection_wait_for_new_work () >>>> #3 0x00007f683f99b719 in connection_threadmain () >>>> #4 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so >>>> #5 0x00007f683d04cdc5 in start_thread () from /lib64/libpthread.so.0 >>>> #6 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 >>>> Thread 13 (Thread 0x7f67fcff1700 (LWP 4880)): >>>> #0 0x00007f683d0506d5 in >>>> pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 >>>> #1 0x00007f683d6a6050 in PR_WaitCondVar () from /lib64/libnspr4.so >>>> #2 0x00007f683f999fee in connection_wait_for_new_work () >>>> #3 0x00007f683f99b719 in connection_threadmain () >>>> #4 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so >>>> #5 0x00007f683d04cdc5 in start_thread () from /lib64/libpthread.so.0 >>>> #6 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 >>>> Thread 12 (Thread 0x7f67fc7f0700 (LWP 4881)): >>>> #0 0x00007f683d0506d5 in >>>> pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 >>>> #1 0x00007f683d6a6050 in PR_WaitCondVar () from /lib64/libnspr4.so >>>> #2 0x00007f683f999fee in connection_wait_for_new_work () >>>> #3 0x00007f683f99b719 in connection_threadmain () >>>> #4 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so >>>> #5 0x00007f683d04cdc5 in start_thread () from /lib64/libpthread.so.0 >>>> #6 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 >>>> Thread 11 (Thread 0x7f67fbfef700 (LWP 4882)): >>>> #0 0x00007f683d0506d5 in >>>> pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 >>>> #1 0x00007f683d6a6050 in PR_WaitCondVar () from /lib64/libnspr4.so >>>> #2 0x00007f683f999fee in connection_wait_for_new_work () >>>> #3 0x00007f683f99b719 in connection_threadmain () >>>> #4 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so >>>> #5 0x00007f683d04cdc5 in start_thread () from /lib64/libpthread.so.0 >>>> #6 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 >>>> Thread 10 (Thread 0x7f67fb7ee700 (LWP 4883)): >>>> #0 0x00007f683d0506d5 in >>>> pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 >>>> #1 0x00007f68375ac2f3 in __db_hybrid_mutex_suspend () from /lib64/ >>>> libdb-5.3.so >>>> #2 0x00007f68375ab640 in __db_tas_mutex_lock () from /lib64/ >>>> libdb-5.3.so >>>> #3 0x00007f6837655d3a in __lock_get_internal () from /lib64/ >>>> libdb-5.3.so >>>> #4 0x00007f6837656820 in __lock_get () from /lib64/libdb-5.3.so >>>> #5 0x00007f6837682142 in __db_lget () from /lib64/libdb-5.3.so >>>> #6 0x00007f68375c9605 in __bam_search () from /lib64/libdb-5.3.so >>>> #7 0x00007f68375b4256 in __bamc_search () from /lib64/libdb-5.3.so >>>> #8 0x00007f68375b5d0f in __bamc_get () from /lib64/libdb-5.3.so >>>> #9 0x00007f683766eca6 in __dbc_iget () from /lib64/libdb-5.3.so >>>> #10 0x00007f683767db02 in __dbc_get_pp () from /lib64/libdb-5.3.so >>>> #11 0x00007f6832b59680 in idl_new_fetch () from >>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>> #12 0x00007f6832b67b26 in index_read_ext_allids () from >>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>> #13 0x00007f6832b52284 in keys2idl () from >>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>> #14 0x00007f6832b529e3 in ava_candidates.isra.0 () from >>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>> #15 0x00007f6832b52fd2 in filter_candidates_ext () from >>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>> #16 0x00007f6832b54096 in list_candidates () from >>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>> #17 0x00007f6832b52f40 in filter_candidates_ext () from >>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>> #18 0x00007f6832b54096 in list_candidates () from >>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>> #19 0x00007f6832b52f40 in filter_candidates_ext () from >>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>> #20 0x00007f6832b54096 in list_candidates () from >>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>> #21 0x00007f6832b52f40 in filter_candidates_ext () from >>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>> #22 0x00007f6832b8e80c in subtree_candidates () from >>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>> #23 0x00007f6832b8fe7f in ldbm_back_search () from >>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>> #24 0x00007f683f4bd1d6 in op_shared_search () from >>>> /usr/lib64/dirsrv/libslapd.so.0 >>>> #25 0x00007f683f9ae33e in do_search () >>>> #26 0x00007f683f99c3c3 in connection_threadmain () >>>> #27 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so >>>> #28 0x00007f683d04cdc5 in start_thread () from /lib64/libpthread.so.0 >>>> #29 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 >>>> Thread 9 (Thread 0x7f67fafed700 (LWP 4884)): >>>> #0 0x00007f683d0506d5 in >>>> pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 >>>> #1 0x00007f683d6a6050 in PR_WaitCondVar () from /lib64/libnspr4.so >>>> #2 0x00007f683f999fee in connection_wait_for_new_work () >>>> #3 0x00007f683f99b719 in connection_threadmain () >>>> #4 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so >>>> #5 0x00007f683d04cdc5 in start_thread () from /lib64/libpthread.so.0 >>>> #6 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 >>>> Thread 8 (Thread 0x7f67fa7ec700 (LWP 4885)): >>>> #0 0x00007f683d0506d5 in >>>> pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 >>>> #1 0x00007f68375ac2f3 in __db_hybrid_mutex_suspend () from /lib64/ >>>> libdb-5.3.so >>>> #2 0x00007f68375ab640 in __db_tas_mutex_lock () from /lib64/ >>>> libdb-5.3.so >>>> #3 0x00007f6837655d3a in __lock_get_internal () from /lib64/ >>>> libdb-5.3.so >>>> #4 0x00007f6837656820 in __lock_get () from /lib64/libdb-5.3.so >>>> #5 0x00007f6837682142 in __db_lget () from /lib64/libdb-5.3.so >>>> #6 0x00007f68375c9605 in __bam_search () from /lib64/libdb-5.3.so >>>> #7 0x00007f68375b4256 in __bamc_search () from /lib64/libdb-5.3.so >>>> #8 0x00007f68375b5d0f in __bamc_get () from /lib64/libdb-5.3.so >>>> #9 0x00007f683766eca6 in __dbc_iget () from /lib64/libdb-5.3.so >>>> #10 0x00007f683767db02 in __dbc_get_pp () from /lib64/libdb-5.3.so >>>> #11 0x00007f6832b59680 in idl_new_fetch () from >>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>> #12 0x00007f6832b67b26 in index_read_ext_allids () from >>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>> #13 0x00007f6832b52284 in keys2idl () from >>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>> #14 0x00007f6832b529e3 in ava_candidates.isra.0 () from >>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>> #15 0x00007f6832b52fd2 in filter_candidates_ext () from >>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>> #16 0x00007f6832b54096 in list_candidates () from >>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>> #17 0x00007f6832b52f40 in filter_candidates_ext () from >>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>> #18 0x00007f6832b54096 in list_candidates () from >>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>> #19 0x00007f6832b52f40 in filter_candidates_ext () from >>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>> #20 0x00007f6832b54096 in list_candidates () from >>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>> #21 0x00007f6832b52f40 in filter_candidates_ext () from >>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>> #22 0x00007f6832b8e80c in subtree_candidates () from >>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>> #23 0x00007f6832b8fe7f in ldbm_back_search () from >>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>> #24 0x00007f683f4bd1d6 in op_shared_search () from >>>> /usr/lib64/dirsrv/libslapd.so.0 >>>> #25 0x00007f683f9ae33e in do_search () >>>> #26 0x00007f683f99c3c3 in connection_threadmain () >>>> #27 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so >>>> #28 0x00007f683d04cdc5 in start_thread () from /lib64/libpthread.so.0 >>>> #29 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 >>>> Thread 7 (Thread 0x7f67f9feb700 (LWP 4886)): >>>> #0 0x00007f683d0506d5 in >>>> pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 >>>> #1 0x00007f683d6a6050 in PR_WaitCondVar () from /lib64/libnspr4.so >>>> #2 0x00007f683f999fee in connection_wait_for_new_work () >>>> #3 0x00007f683f99b719 in connection_threadmain () >>>> #4 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so >>>> #5 0x00007f683d04cdc5 in start_thread () from /lib64/libpthread.so.0 >>>> #6 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 >>>> Thread 6 (Thread 0x7f67f97ea700 (LWP 4887)): >>>> #0 0x00007f683d0506d5 in >>>> pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 >>>> #1 0x00007f683d6a6050 in PR_WaitCondVar () from /lib64/libnspr4.so >>>> #2 0x00007f683f999fee in connection_wait_for_new_work () >>>> #3 0x00007f683f99b719 in connection_threadmain () >>>> #4 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so >>>> #5 0x00007f683d04cdc5 in start_thread () from /lib64/libpthread.so.0 >>>> #6 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 >>>> Thread 5 (Thread 0x7f67f8fe9700 (LWP 4888)): >>>> #0 0x00007f683d0506d5 in >>>> pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 >>>> #1 0x00007f683d6a6050 in PR_WaitCondVar () from /lib64/libnspr4.so >>>> #2 0x00007f683f999fee in connection_wait_for_new_work () >>>> #3 0x00007f683f99b719 in connection_threadmain () >>>> #4 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so >>>> #5 0x00007f683d04cdc5 in start_thread () from /lib64/libpthread.so.0 >>>> #6 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 >>>> Thread 4 (Thread 0x7f67f87e8700 (LWP 4889)): >>>> #0 0x00007f683cd719b3 in select () from /lib64/libc.so.6 >>>> #1 0x00007f683f4fc0e9 in DS_Sleep () from >>>> /usr/lib64/dirsrv/libslapd.so.0 >>>> #2 0x00007f683f99d325 in time_thread () >>>> #3 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so >>>> #4 0x00007f683d04cdc5 in start_thread () from /lib64/libpthread.so.0 >>>> #5 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 >>>> Thread 3 (Thread 0x7f67f7de6700 (LWP 4892)): >>>> #0 0x00007f683d0506d5 in >>>> pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 >>>> #1 0x00007f683d6a6050 in PR_WaitCondVar () from /lib64/libnspr4.so >>>> #2 0x00007f683f9a8f25 in ps_send_results () >>>> #3 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so >>>> #4 0x00007f683d04cdc5 in start_thread () from /lib64/libpthread.so.0 >>>> #5 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 >>>> Thread 2 (Thread 0x7f67f75e5700 (LWP 5084)): >>>> #0 0x00007f683d0506d5 in >>>> pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 >>>> #1 0x00007f683d6a6050 in PR_WaitCondVar () from /lib64/libnspr4.so >>>> #2 0x00007f683f9a8f25 in ps_send_results () >>>> #3 0x00007f683d6ab7bb in _pt_root () from /lib64/libnspr4.so >>>> #4 0x00007f683d04cdc5 in start_thread () from /lib64/libpthread.so.0 >>>> #5 0x00007f683cd7a28d in clone () from /lib64/libc.so.6 >>>> Thread 1 (Thread 0x7f683f964840 (LWP 4845)): >>>> #0 0x00007f683d0506d5 in >>>> pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 >>>> #1 0x00007f683d6a62c3 in PR_EnterMonitor () from /lib64/libnspr4.so >>>> #2 0x00007f683f99d5ed in setup_pr_read_pds () >>>> #3 0x00007f683f9a0fd9 in slapd_daemon () >>>> #4 0x00007f683f992c6c in main () >>>> >>>> >>>> >>>> the top output seems ok, there was just a sudden spike to around 340% >>>> cpu then it comes down >>>> >>>> top - 08:53:12 up 6 days, 23:20, 1 user, load average: 0.00, 0.01, >>>> 0.12 >>>> Threads: 44 total, 0 running, 44 sleeping, 0 stopped, 0 zombie >>>> %Cpu(s): 0.0 us, 0.0 sy, 0.0 ni,100.0 id, 0.0 wa, 0.0 hi, 0.0 >>>> si, 0.0 st >>>> KiB Mem : 65533176 total, 52165676 free, 6135088 used, 7232412 >>>> buff/cache >>>> KiB Swap: 3999996 total, 3999996 free, 0 used. 58452040 avail >>>> Mem >>>> >>>> PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ >>>> COMMAND >>>> 4845 dirsrv 20 0 5133196 3.150g 281272 S 0.0 5.0 31:34.51 >>>> ns-slapd >>>> 4847 dirsrv 20 0 5133196 3.150g 281272 S 0.0 5.0 0:13.53 >>>> ns-slapd >>>> 4848 dirsrv 20 0 5133196 3.150g 281272 S 0.0 5.0 0:13.16 >>>> ns-slapd >>>> 4849 dirsrv 20 0 5133196 3.150g 281272 S 0.0 5.0 0:45.27 >>>> ns-slapd >>>> 4850 dirsrv 20 0 5133196 3.150g 281272 S 0.0 5.0 0:01.60 >>>> ns-slapd >>>> 4851 dirsrv 20 0 5133196 3.150g 281272 S 0.0 5.0 0:00.00 >>>> ns-slapd >>>> 4852 dirsrv 20 0 5133196 3.150g 281272 S 0.0 5.0 0:00.00 >>>> ns-slapd >>>> 4853 dirsrv 20 0 5133196 3.150g 281272 S 0.0 5.0 0:00.00 >>>> ns-slapd >>>> 4854 dirsrv 20 0 5133196 3.150g 281272 S 0.0 5.0 0:00.00 >>>> ns-slapd >>>> 4855 dirsrv 20 0 5133196 3.150g 281272 S 0.0 5.0 0:03.02 >>>> ns-slapd >>>> 4856 dirsrv 20 0 5133196 3.150g 281272 S 0.0 5.0 0:00.26 >>>> ns-slapd >>>> 4859 dirsrv 20 0 5133196 3.150g 281272 S 0.0 5.0 3:18.67 >>>> ns-slapd >>>> 4860 dirsrv 20 0 5133196 3.150g 281272 S 0.0 5.0 3:19.82 >>>> ns-slapd >>>> 4861 dirsrv 20 0 5133196 3.150g 281272 S 0.0 5.0 3:20.23 >>>> ns-slapd >>>> 4862 dirsrv 20 0 5133196 3.150g 281272 S 0.0 5.0 3:22.65 >>>> ns-slapd >>>> 4863 dirsrv 20 0 5133196 3.150g 281272 S 0.0 5.0 3:16.64 >>>> ns-slapd >>>> 4864 dirsrv 20 0 5133196 3.150g 281272 S 0.0 5.0 3:17.39 >>>> ns-slapd >>>> 4865 dirsrv 20 0 5133196 3.150g 281272 S 0.0 5.0 3:17.80 >>>> ns-slapd >>>> >>>> >>>> Thanks >>>> Rakesh >>>> >>>> On Tue, Sep 6, 2016 at 12:09 AM, Rakesh Rajasekharan < >>>> rakesh.rajasekharan at gmail.com> wrote: >>>> >>>>> Hi Thierry/Ludwig, >>>>> >>>>> Unfortunately I had by then restarted the service and since then have >>>>> been trying to hit the hang issue again but no luck this time >>>>> >>>>> It has been running fine for the last few hours now in-spite of trying >>>>> all that I believe could cause the hang like making simultaneous client >>>>> addition requests. >>>>> >>>>> I will continue testing this and next time I get it I will make sure I >>>>> get the pstack and also get the dump of db lock info. >>>>> >>>>> Thanks, >>>>> Rakesh >>>>> >>>>> >>>>> >>>>> On Mon, Sep 5, 2016 at 6:14 PM, Ludwig Krispenz < >>>>> lkrispen at redhat.com> wrote: >>>>> >>>>>> >>>>>> On 09/05/2016 02:31 PM, thierry bordaz wrote: >>>>>> >>>>>> Hi Rakesh, >>>>>> >>>>>> Thanks you soooo much for you continuous help on this !! >>>>>> >>>>>> If the hang is still going on, would you dump the DB lock information >>>>>> with: >>>>>> >>>>>> db_stat -CA -N -h /var/lib/dirsrv/slapd-/db >>>>>> >/tmp/db_lock_info >>>>>> >>>>>> It is looking like several threads are waiting for one/several db >>>>>> locks. With the lock info we will know which thread acquired it/them. >>>>>> A guess is that it could be the checkpointing thread but this one is >>>>>> also waiting (on a memory pool). >>>>>> >>>>>> I am copying Ludwig on this thread in case this pstack rings any bell. >>>>>> >>>>>> the frequent presence of __db_tas_mutex_lock() looks like the NUMA >>>>>> problem of libdb, although the threads already have stopped spinning (maybe >>>>>> this was an effect of gcore). >>>>>> If the hang aoccurs again, could you check cpu consumption >>>>>> top -H -p >>>>>> and get a pstack >>>>>> pstack >>>>>> >>>>>> >>>>>> best regards >>>>>> thierry >>>>>> >>>>>> On 09/05/2016 02:05 PM, Rakesh Rajasekharan wrote: >>>>>> >>>>>> Hi Thierry, >>>>>> >>>>>> I was able to now reporduce the issue.. >>>>>> >>>>>> the ns-slapd process just got hung,.. however, no core file was >>>>>> generate.. >>>>>> >>>>>> I manually created a coredump using gcore and then took a strace of it >>>>>> this is strace from the coredump generated >>>>>> >>>>>> GNU gdb (GDB) Red Hat Enterprise Linux 7.6.1-80.el7 >>>>>> Copyright (C) 2013 Free Software Foundation, Inc. >>>>>> License GPLv3+: GNU GPL version 3 or later < >>>>>> http://gnu.org/licenses/gpl.html> >>>>>> This is free software: you are free to change and redistribute it. >>>>>> There is NO WARRANTY, to the extent permitted by law. Type "show >>>>>> copying" >>>>>> and "show warranty" for details. >>>>>> This GDB was configured as "x86_64-redhat-linux-gnu". >>>>>> For bug reporting instructions, please see: >>>>>> < http://www.gnu.org/software/g >>>>>> db/bugs/>... >>>>>> Reading symbols from /usr/sbin/ns-slapd...Reading symbols from >>>>>> /usr/sbin/ns-slapd...(no debugging symbols found)...done. >>>>>> (no debugging symbols found)...done. >>>>>> [New LWP 105159] >>>>>> [New LWP 105160] >>>>>> [New LWP 105161] >>>>>> [New LWP 105162] >>>>>> [New LWP 105163] >>>>>> [New LWP 105164] >>>>>> [New LWP 105165] >>>>>> [New LWP 105166] >>>>>> [New LWP 105167] >>>>>> [New LWP 105168] >>>>>> [New LWP 105171] >>>>>> [New LWP 105172] >>>>>> [New LWP 105173] >>>>>> [New LWP 105174] >>>>>> [New LWP 105175] >>>>>> [New LWP 105176] >>>>>> [New LWP 105177] >>>>>> [New LWP 105178] >>>>>> [New LWP 105179] >>>>>> [New LWP 105180] >>>>>> [New LWP 105181] >>>>>> [New LWP 105182] >>>>>> [New LWP 105183] >>>>>> [New LWP 105184] >>>>>> [New LWP 105185] >>>>>> [New LWP 105186] >>>>>> [New LWP 105187] >>>>>> [New LWP 105188] >>>>>> [New LWP 105189] >>>>>> [New LWP 105190] >>>>>> [New LWP 105191] >>>>>> [New LWP 105192] >>>>>> [New LWP 105193] >>>>>> [New LWP 105194] >>>>>> [New LWP 105195] >>>>>> [New LWP 105196] >>>>>> [New LWP 105197] >>>>>> [New LWP 105198] >>>>>> [New LWP 105199] >>>>>> [New LWP 105200] >>>>>> [New LWP 105201] >>>>>> [New LWP 105203] >>>>>> [New LWP 105215] >>>>>> [New LWP 105158] >>>>>> [Thread debugging using libthread_db enabled] >>>>>> Using host libthread_db library "/lib64/libthread_db.so.1". >>>>>> Core was generated by `/usr/sbin/ns-slapd'. >>>>>> #0 0x00007f1b48ae69b3 in select () from /lib64/libc.so.6 >>>>>> >>>>>> Thread 44 (Thread 0x7f1b4b6d9840 (LWP 105158)): >>>>>> #0 0x00007f1b48dc56d5 in >>>>>> pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 >>>>>> No symbol table info available. >>>>>> #1 0x00007f1b4941b2c3 in PR_EnterMonitor () from /lib64/libnspr4.so >>>>>> No symbol table info available. >>>>>> #2 0x00007f1b4b7125ed in setup_pr_read_pds () >>>>>> No symbol table info available. >>>>>> #3 0x00007f1b4b715fd9 in slapd_daemon () >>>>>> No symbol table info available. >>>>>> #4 0x00007f1b4b707c6c in main () >>>>>> No symbol table info available. >>>>>> >>>>>> Thread 43 (Thread 0x7f1a5df0f700 (LWP 105215)): >>>>>> #0 0x00007f1b48dc56d5 in >>>>>> pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 >>>>>> No symbol table info available. >>>>>> #1 0x00007f1b4941b050 in PR_WaitCondVar () from /lib64/libnspr4.so >>>>>> No symbol table info available. >>>>>> #2 0x00007f1b4b71df25 in ps_send_results () >>>>>> No symbol table info available. >>>>>> #3 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so >>>>>> No symbol table info available. >>>>>> #4 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 >>>>>> No symbol table info available. >>>>>> #5 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 >>>>>> No symbol table info available. >>>>>> >>>>>> Thread 42 (Thread 0x7f1b03fe7700 (LWP 105203)): >>>>>> #0 0x00007f1b48dc56d5 in >>>>>> pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 >>>>>> No symbol table info available. >>>>>> #1 0x00007f1b4941b050 in PR_WaitCondVar () from /lib64/libnspr4.so >>>>>> No symbol table info available. >>>>>> #2 0x00007f1b4b71df25 in ps_send_results () >>>>>> No symbol table info available. >>>>>> #3 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so >>>>>> No symbol table info available. >>>>>> #4 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 >>>>>> No symbol table info available. >>>>>> #5 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 >>>>>> No symbol table info available. >>>>>> >>>>>> Thread 41 (Thread 0x7f1b047e8700 (LWP 105201)): >>>>>> #0 0x00007f1b48ae69b3 in select () from /lib64/libc.so.6 >>>>>> No symbol table info available. >>>>>> #1 0x00007f1b4b2710e9 in DS_Sleep () from >>>>>> /usr/lib64/dirsrv/libslapd.so.0 >>>>>> No symbol table info available. >>>>>> #2 0x00007f1b4b712325 in time_thread () >>>>>> No symbol table info available. >>>>>> #3 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so >>>>>> No symbol table info available. >>>>>> #4 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 >>>>>> No symbol table info available. >>>>>> #5 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 >>>>>> No symbol table info available. >>>>>> >>>>>> Thread 40 (Thread 0x7f1b04fe9700 (LWP 105200)): >>>>>> #0 0x00007f1b48dc56d5 in >>>>>> pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 >>>>>> No symbol table info available. >>>>>> #1 0x00007f1b433212f3 in __db_hybrid_mutex_suspend () from /lib64/ >>>>>> libdb-5.3.so >>>>>> No symbol table info available. >>>>>> #2 0x00007f1b43320640 in __db_tas_mutex_lock () from /lib64/ >>>>>> libdb-5.3.so >>>>>> No symbol table info available. >>>>>> #3 0x00007f1b4343ae1b in __memp_fget () from /lib64/libdb-5.3.so >>>>>> No symbol table info available. >>>>>> #4 0x00007f1b4333e211 in __bam_search () from /lib64/libdb-5.3.so >>>>>> No symbol table info available. >>>>>> #5 0x00007f1b43329256 in __bamc_search () from /lib64/libdb-5.3.so >>>>>> No symbol table info available. >>>>>> #6 0x00007f1b4332d6c4 in __bamc_put () from /lib64/libdb-5.3.so >>>>>> No symbol table info available. >>>>>> #7 0x00007f1b433e46d5 in __dbc_iput () from /lib64/libdb-5.3.so >>>>>> No symbol table info available. >>>>>> #8 0x00007f1b433dfa0e in __db_put () from /lib64/libdb-5.3.so >>>>>> No symbol table info available. >>>>>> #9 0x00007f1b433f4fa4 in __db_put_pp () from /lib64/libdb-5.3.so >>>>>> No symbol table info available. >>>>>> #10 0x00007f1b3e8cf4cf in idl_new_insert_key () from >>>>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>>>> No symbol table info available. >>>>>> #11 0x00007f1b3e8de5b5 in addordel_values_sv.isra.3 () from >>>>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>>>> No symbol table info available. >>>>>> #12 0x00007f1b3e8df25a in index_addordel_values_ext_sv () from >>>>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>>>> No symbol table info available. >>>>>> #13 0x00007f1b3e8df694 in index_addordel_values_sv () from >>>>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>>>> No symbol table info available. >>>>>> #14 0x00007f1b3e8df75a in index_addordel_entry () from >>>>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>>>> No symbol table info available. >>>>>> #15 0x00007f1b3e8e25fa in ldbm_back_add () from >>>>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>>>> No symbol table info available. >>>>>> #16 0x00007f1b4b1e4cca in op_shared_add () from >>>>>> /usr/lib64/dirsrv/libslapd.so.0 >>>>>> No symbol table info available. >>>>>> #17 0x00007f1b4b1e6020 in do_add () from >>>>>> /usr/lib64/dirsrv/libslapd.so.0 >>>>>> No symbol table info available. >>>>>> #18 0x00007f1b4b711303 in connection_threadmain () >>>>>> No symbol table info available. >>>>>> #19 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so >>>>>> No symbol table info available. >>>>>> #20 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 >>>>>> No symbol table info available. >>>>>> #21 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 >>>>>> No symbol table info available. >>>>>> >>>>>> Thread 39 (Thread 0x7f1b057ea700 (LWP 105199)): >>>>>> #0 0x00007f1b48dc56d5 in >>>>>> pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 >>>>>> No symbol table info available. >>>>>> #1 0x00007f1b4941b050 in PR_WaitCondVar () from /lib64/libnspr4.so >>>>>> No symbol table info available. >>>>>> #2 0x00007f1b4b70efee in connection_wait_for_new_work () >>>>>> No symbol table info available. >>>>>> #3 0x00007f1b4b710719 in connection_threadmain () >>>>>> No symbol table info available. >>>>>> #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so >>>>>> No symbol table info available. >>>>>> #5 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 >>>>>> No symbol table info available. >>>>>> #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 >>>>>> No symbol table info available. >>>>>> >>>>>> Thread 38 (Thread 0x7f1b05feb700 (LWP 105198)): >>>>>> #0 0x00007f1b48dc56d5 in >>>>>> pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 >>>>>> No symbol table info available. >>>>>> #1 0x00007f1b4941b050 in PR_WaitCondVar () from /lib64/libnspr4.so >>>>>> No symbol table info available. >>>>>> #2 0x00007f1b4b70efee in connection_wait_for_new_work () >>>>>> No symbol table info available. >>>>>> #3 0x00007f1b4b710719 in connection_threadmain () >>>>>> No symbol table info available. >>>>>> #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so >>>>>> No symbol table info available. >>>>>> #5 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 >>>>>> No symbol table info available. >>>>>> #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 >>>>>> No symbol table info available. >>>>>> >>>>>> Thread 37 (Thread 0x7f1b067ec700 (LWP 105197)): >>>>>> #0 0x00007f1b48dc56d5 in >>>>>> pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 >>>>>> No symbol table info available. >>>>>> #1 0x00007f1b4941b050 in PR_WaitCondVar () from /lib64/libnspr4.so >>>>>> No symbol table info available. >>>>>> #2 0x00007f1b4b70efee in connection_wait_for_new_work () >>>>>> No symbol table info available. >>>>>> #3 0x00007f1b4b710719 in connection_threadmain () >>>>>> No symbol table info available. >>>>>> #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so >>>>>> No symbol table info available. >>>>>> #5 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 >>>>>> No symbol table info available. >>>>>> #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 >>>>>> No symbol table info available. >>>>>> >>>>>> Thread 36 (Thread 0x7f1b06fed700 (LWP 105196)): >>>>>> #0 0x00007f1b48dc56d5 in >>>>>> pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 >>>>>> No symbol table info available. >>>>>> #1 0x00007f1b4941b050 in PR_WaitCondVar () from /lib64/libnspr4.so >>>>>> No symbol table info available. >>>>>> #2 0x00007f1b4b70efee in connection_wait_for_new_work () >>>>>> No symbol table info available. >>>>>> #3 0x00007f1b4b710719 in connection_threadmain () >>>>>> No symbol table info available. >>>>>> #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so >>>>>> No symbol table info available. >>>>>> #5 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 >>>>>> No symbol table info available. >>>>>> #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 >>>>>> No symbol table info available. >>>>>> >>>>>> Thread 35 (Thread 0x7f1b077ee700 (LWP 105195)): >>>>>> #0 0x00007f1b48dc56d5 in >>>>>> pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 >>>>>> No symbol table info available. >>>>>> #1 0x00007f1b4941b2c3 in PR_EnterMonitor () from /lib64/libnspr4.so >>>>>> No symbol table info available. >>>>>> #2 0x00007f1b3e8c23e6 in dblayer_txn_begin () from >>>>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>>>> No symbol table info available. >>>>>> #3 0x00007f1b3e8fe007 in ldbm_back_modify () from >>>>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>>>> No symbol table info available. >>>>>> #4 0x00007f1b4b22b541 in op_shared_modify () from >>>>>> /usr/lib64/dirsrv/libslapd.so.0 >>>>>> No symbol table info available. >>>>>> #5 0x00007f1b4b22c004 in modify_internal_pb () from >>>>>> /usr/lib64/dirsrv/libslapd.so.0 >>>>>> No symbol table info available. >>>>>> #6 0x00007f1b3eb48b60 in ipapwd_apply_mods () from >>>>>> /usr/lib64/dirsrv/plugins/libipa_pwd_extop.so >>>>>> No symbol table info available. >>>>>> #7 0x00007f1b3eb4c8e0 in store_new_keys () from >>>>>> /usr/lib64/dirsrv/plugins/libipa_pwd_extop.so >>>>>> No symbol table info available. >>>>>> #8 0x00007f1b3eb4ead0 in ipapwd_extop () from >>>>>> /usr/lib64/dirsrv/plugins/libipa_pwd_extop.so >>>>>> No symbol table info available. >>>>>> #9 0x00007f1b4b23bba2 in plugin_call_exop_plugins () from >>>>>> /usr/lib64/dirsrv/libslapd.so.0 >>>>>> No symbol table info available. >>>>>> #10 0x00007f1b4b717efe in do_extended () >>>>>> No symbol table info available. >>>>>> #11 0x00007f1b4b71141a in connection_threadmain () >>>>>> No symbol table info available. >>>>>> #12 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so >>>>>> No symbol table info available. >>>>>> #13 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 >>>>>> No symbol table info available. >>>>>> #14 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 >>>>>> No symbol table info available. >>>>>> >>>>>> Thread 34 (Thread 0x7f1b07fef700 (LWP 105194)): >>>>>> #0 0x00007f1b48dc56d5 in >>>>>> pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 >>>>>> No symbol table info available. >>>>>> #1 0x00007f1b4941b050 in PR_WaitCondVar () from /lib64/libnspr4.so >>>>>> No symbol table info available. >>>>>> #2 0x00007f1b4b70efee in connection_wait_for_new_work () >>>>>> No symbol table info available. >>>>>> #3 0x00007f1b4b710719 in connection_threadmain () >>>>>> No symbol table info available. >>>>>> #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so >>>>>> No symbol table info available. >>>>>> #5 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 >>>>>> No symbol table info available. >>>>>> #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 >>>>>> No symbol table info available. >>>>>> >>>>>> Thread 33 (Thread 0x7f1b087f0700 (LWP 105193)): >>>>>> #0 0x00007f1b48dc56d5 in >>>>>> pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 >>>>>> No symbol table info available. >>>>>> #1 0x00007f1b4941b050 in PR_WaitCondVar () from /lib64/libnspr4.so >>>>>> No symbol table info available. >>>>>> #2 0x00007f1b4b70efee in connection_wait_for_new_work () >>>>>> No symbol table info available. >>>>>> #3 0x00007f1b4b710719 in connection_threadmain () >>>>>> No symbol table info available. >>>>>> #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so >>>>>> No symbol table info available. >>>>>> #5 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 >>>>>> No symbol table info available. >>>>>> #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 >>>>>> No symbol table info available. >>>>>> >>>>>> Thread 32 (Thread 0x7f1b08ff1700 (LWP 105192)): >>>>>> #0 0x00007f1b48dc56d5 in >>>>>> pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 >>>>>> No symbol table info available. >>>>>> #1 0x00007f1b4941b050 in PR_WaitCondVar () from /lib64/libnspr4.so >>>>>> No symbol table info available. >>>>>> #2 0x00007f1b4b70efee in connection_wait_for_new_work () >>>>>> No symbol table info available. >>>>>> #3 0x00007f1b4b710719 in connection_threadmain () >>>>>> No symbol table info available. >>>>>> #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so >>>>>> No symbol table info available. >>>>>> #5 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 >>>>>> No symbol table info available. >>>>>> #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 >>>>>> No symbol table info available. >>>>>> >>>>>> Thread 31 (Thread 0x7f1b097f2700 (LWP 105191)): >>>>>> #0 0x00007f1b48dc56d5 in >>>>>> pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 >>>>>> No symbol table info available. >>>>>> #1 0x00007f1b4941b050 in PR_WaitCondVar () from /lib64/libnspr4.so >>>>>> No symbol table info available. >>>>>> #2 0x00007f1b4b70efee in connection_wait_for_new_work () >>>>>> No symbol table info available. >>>>>> #3 0x00007f1b4b710719 in connection_threadmain () >>>>>> No symbol table info available. >>>>>> #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so >>>>>> No symbol table info available. >>>>>> #5 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 >>>>>> No symbol table info available. >>>>>> #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 >>>>>> No symbol table info available. >>>>>> >>>>>> Thread 30 (Thread 0x7f1b09ff3700 (LWP 105190)): >>>>>> #0 0x00007f1b48dc56d5 in >>>>>> pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 >>>>>> No symbol table info available. >>>>>> #1 0x00007f1b433212f3 in __db_hybrid_mutex_suspend () from /lib64/ >>>>>> libdb-5.3.so >>>>>> No symbol table info available. >>>>>> #2 0x00007f1b43320640 in __db_tas_mutex_lock () from /lib64/ >>>>>> libdb-5.3.so >>>>>> No symbol table info available. >>>>>> #3 0x00007f1b433cad3a in __lock_get_internal () from /lib64/ >>>>>> libdb-5.3.so >>>>>> No symbol table info available. >>>>>> #4 0x00007f1b433cb820 in __lock_get () from /lib64/libdb-5.3.so >>>>>> No symbol table info available. >>>>>> #5 0x00007f1b433f7142 in __db_lget () from /lib64/libdb-5.3.so >>>>>> No symbol table info available. >>>>>> #6 0x00007f1b4333e605 in __bam_search () from /lib64/libdb-5.3.so >>>>>> No symbol table info available. >>>>>> #7 0x00007f1b43329256 in __bamc_search () from /lib64/libdb-5.3.so >>>>>> No symbol table info available. >>>>>> #8 0x00007f1b4332ad0f in __bamc_get () from /lib64/libdb-5.3.so >>>>>> No symbol table info available. >>>>>> #9 0x00007f1b433e3ca6 in __dbc_iget () from /lib64/libdb-5.3.so >>>>>> No symbol table info available. >>>>>> #10 0x00007f1b433f2b02 in __dbc_get_pp () from /lib64/libdb-5.3.so >>>>>> No symbol table info available. >>>>>> #11 0x00007f1b3e8ce680 in idl_new_fetch () from >>>>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>>>> No symbol table info available. >>>>>> #12 0x00007f1b3e8dcb26 in index_read_ext_allids () from >>>>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>>>> No symbol table info available. >>>>>> #13 0x00007f1b3e8c7284 in keys2idl () from >>>>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>>>> No symbol table info available. >>>>>> #14 0x00007f1b3e8c79e3 in ava_candidates.isra.0 () from >>>>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>>>> No symbol table info available. >>>>>> #15 0x00007f1b3e8c7fd2 in filter_candidates_ext () from >>>>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>>>> No symbol table info available. >>>>>> #16 0x00007f1b3e8c9096 in list_candidates () from >>>>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>>>> No symbol table info available. >>>>>> #17 0x00007f1b3e8c7f40 in filter_candidates_ext () from >>>>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>>>> No symbol table info available. >>>>>> #18 0x00007f1b3e8c9096 in list_candidates () from >>>>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>>>> No symbol table info available. >>>>>> #19 0x00007f1b3e8c7f40 in filter_candidates_ext () from >>>>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>>>> No symbol table info available. >>>>>> #20 0x00007f1b3e8c9096 in list_candidates () from >>>>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>>>> No symbol table info available. >>>>>> #21 0x00007f1b3e8c7f40 in filter_candidates_ext () from >>>>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>>>> No symbol table info available. >>>>>> #22 0x00007f1b3e90380c in subtree_candidates () from >>>>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>>>> No symbol table info available. >>>>>> #23 0x00007f1b3e904e7f in ldbm_back_search () from >>>>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>>>> No symbol table info available. >>>>>> #24 0x00007f1b4b2321d6 in op_shared_search () from >>>>>> /usr/lib64/dirsrv/libslapd.so.0 >>>>>> No symbol table info available. >>>>>> #25 0x00007f1b4b72333e in do_search () >>>>>> No symbol table info available. >>>>>> #26 0x00007f1b4b7113c3 in connection_threadmain () >>>>>> No symbol table info available. >>>>>> #27 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so >>>>>> No symbol table info available. >>>>>> #28 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 >>>>>> No symbol table info available. >>>>>> #29 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 >>>>>> No symbol table info available. >>>>>> >>>>>> Thread 29 (Thread 0x7f1b0a7f4700 (LWP 105189)): >>>>>> #0 0x00007f1b48dc56d5 in >>>>>> pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 >>>>>> No symbol table info available. >>>>>> #1 0x00007f1b4941b050 in PR_WaitCondVar () from /lib64/libnspr4.so >>>>>> No symbol table info available. >>>>>> #2 0x00007f1b4b70efee in connection_wait_for_new_work () >>>>>> No symbol table info available. >>>>>> #3 0x00007f1b4b710719 in connection_threadmain () >>>>>> No symbol table info available. >>>>>> #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so >>>>>> No symbol table info available. >>>>>> #5 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 >>>>>> No symbol table info available. >>>>>> #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 >>>>>> No symbol table info available. >>>>>> >>>>>> Thread 28 (Thread 0x7f1b0aff5700 (LWP 105188)): >>>>>> #0 0x00007f1b48dc56d5 in >>>>>> pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 >>>>>> No symbol table info available. >>>>>> #1 0x00007f1b4941b050 in PR_WaitCondVar () from /lib64/libnspr4.so >>>>>> No symbol table info available. >>>>>> #2 0x00007f1b4b70efee in connection_wait_for_new_work () >>>>>> No symbol table info available. >>>>>> #3 0x00007f1b4b710719 in connection_threadmain () >>>>>> No symbol table info available. >>>>>> #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so >>>>>> No symbol table info available. >>>>>> #5 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 >>>>>> No symbol table info available. >>>>>> #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 >>>>>> No symbol table info available. >>>>>> >>>>>> Thread 27 (Thread 0x7f1b0b7f6700 (LWP 105187)): >>>>>> #0 0x00007f1b48dc56d5 in >>>>>> pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 >>>>>> No symbol table info available. >>>>>> #1 0x00007f1b4941b050 in PR_WaitCondVar () from /lib64/libnspr4.so >>>>>> No symbol table info available. >>>>>> #2 0x00007f1b4b70efee in connection_wait_for_new_work () >>>>>> No symbol table info available. >>>>>> #3 0x00007f1b4b710719 in connection_threadmain () >>>>>> No symbol table info available. >>>>>> #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so >>>>>> No symbol table info available. >>>>>> #5 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 >>>>>> No symbol table info available. >>>>>> #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 >>>>>> No symbol table info available. >>>>>> >>>>>> Thread 26 (Thread 0x7f1b0bff7700 (LWP 105186)): >>>>>> #0 0x00007f1b48dc56d5 in >>>>>> pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 >>>>>> No symbol table info available. >>>>>> #1 0x00007f1b433212f3 in __db_hybrid_mutex_suspend () from /lib64/ >>>>>> libdb-5.3.so >>>>>> No symbol table info available. >>>>>> #2 0x00007f1b43320640 in __db_tas_mutex_lock () from /lib64/ >>>>>> libdb-5.3.so >>>>>> No symbol table info available. >>>>>> #3 0x00007f1b433cad3a in __lock_get_internal () from /lib64/ >>>>>> libdb-5.3.so >>>>>> No symbol table info available. >>>>>> #4 0x00007f1b433cb820 in __lock_get () from /lib64/libdb-5.3.so >>>>>> No symbol table info available. >>>>>> #5 0x00007f1b433f7142 in __db_lget () from /lib64/libdb-5.3.so >>>>>> No symbol table info available. >>>>>> #6 0x00007f1b4333e605 in __bam_search () from /lib64/libdb-5.3.so >>>>>> No symbol table info available. >>>>>> #7 0x00007f1b43329256 in __bamc_search () from /lib64/libdb-5.3.so >>>>>> No symbol table info available. >>>>>> #8 0x00007f1b4332ad0f in __bamc_get () from /lib64/libdb-5.3.so >>>>>> No symbol table info available. >>>>>> #9 0x00007f1b433e3ca6 in __dbc_iget () from /lib64/libdb-5.3.so >>>>>> No symbol table info available. >>>>>> #10 0x00007f1b433f2b02 in __dbc_get_pp () from /lib64/libdb-5.3.so >>>>>> No symbol table info available. >>>>>> #11 0x00007f1b3e8ce680 in idl_new_fetch () from >>>>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>>>> No symbol table info available. >>>>>> #12 0x00007f1b3e8dcb26 in index_read_ext_allids () from >>>>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>>>> No symbol table info available. >>>>>> #13 0x00007f1b3e8c7284 in keys2idl () from >>>>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>>>> No symbol table info available. >>>>>> #14 0x00007f1b3e8c79e3 in ava_candidates.isra.0 () from >>>>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>>>> No symbol table info available. >>>>>> #15 0x00007f1b3e8c7fd2 in filter_candidates_ext () from >>>>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>>>> No symbol table info available. >>>>>> #16 0x00007f1b3e8c9096 in list_candidates () from >>>>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>>>> No symbol table info available. >>>>>> #17 0x00007f1b3e8c7f40 in filter_candidates_ext () from >>>>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>>>> No symbol table info available. >>>>>> #18 0x00007f1b3e90380c in subtree_candidates () from >>>>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>>>> No symbol table info available. >>>>>> #19 0x00007f1b3e904e7f in ldbm_back_search () from >>>>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>>>> No symbol table info available. >>>>>> #20 0x00007f1b4b2321d6 in op_shared_search () from >>>>>> /usr/lib64/dirsrv/libslapd.so.0 >>>>>> No symbol table info available. >>>>>> #21 0x00007f1b4b24294e in search_internal_callback_pb () from >>>>>> /usr/lib64/dirsrv/libslapd.so.0 >>>>>> No symbol table info available. >>>>>> #22 0x00007f1b4b242be8 in search_internal_pb () from >>>>>> /usr/lib64/dirsrv/libslapd.so.0 >>>>>> No symbol table info available. >>>>>> #23 0x00007f1b4b72149b in ids_sasl_canon_user () >>>>>> No symbol table info available. >>>>>> #24 0x00007f1b491e193b in _sasl_canon_user () from >>>>>> /lib64/libsasl2.so.3 >>>>>> No symbol table info available. >>>>>> #25 0x00007f1b491e1c4c in _sasl_canon_user_lookup () from >>>>>> /lib64/libsasl2.so.3 >>>>>> No symbol table info available. >>>>>> #26 0x00007f1b430e9fb9 in gssapi_server_mech_step () from >>>>>> /usr/lib64/sasl2/libgssapiv2.so >>>>>> No symbol table info available. >>>>>> #27 0x00007f1b491edb9b in sasl_server_step () from >>>>>> /lib64/libsasl2.so.3 >>>>>> No symbol table info available. >>>>>> #28 0x00007f1b4b72283b in ids_sasl_check_bind () >>>>>> No symbol table info available. >>>>>> #29 0x00007f1b4b70a61a in do_bind () >>>>>> No symbol table info available. >>>>>> #30 0x00007f1b4b7113fd in connection_threadmain () >>>>>> No symbol table info available. >>>>>> #31 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so >>>>>> No symbol table info available. >>>>>> #32 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 >>>>>> No symbol table info available. >>>>>> #33 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 >>>>>> No symbol table info available. >>>>>> >>>>>> Thread 25 (Thread 0x7f1b0c7f8700 (LWP 105185)): >>>>>> #0 0x00007f1b48dc56d5 in >>>>>> pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 >>>>>> No symbol table info available. >>>>>> #1 0x00007f1b4941b050 in PR_WaitCondVar () from /lib64/libnspr4.so >>>>>> No symbol table info available. >>>>>> #2 0x00007f1b4b70efee in connection_wait_for_new_work () >>>>>> No symbol table info available. >>>>>> #3 0x00007f1b4b710719 in connection_threadmain () >>>>>> No symbol table info available. >>>>>> #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so >>>>>> No symbol table info available. >>>>>> #5 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 >>>>>> No symbol table info available. >>>>>> #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 >>>>>> No symbol table info available. >>>>>> >>>>>> Thread 24 (Thread 0x7f1b0cff9700 (LWP 105184)): >>>>>> #0 0x00007f1b48dc56d5 in >>>>>> pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 >>>>>> No symbol table info available. >>>>>> #1 0x00007f1b4941b050 in PR_WaitCondVar () from /lib64/libnspr4.so >>>>>> No symbol table info available. >>>>>> #2 0x00007f1b4b70efee in connection_wait_for_new_work () >>>>>> No symbol table info available. >>>>>> #3 0x00007f1b4b710719 in connection_threadmain () >>>>>> No symbol table info available. >>>>>> #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so >>>>>> No symbol table info available. >>>>>> #5 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 >>>>>> No symbol table info available. >>>>>> #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 >>>>>> No symbol table info available. >>>>>> >>>>>> Thread 23 (Thread 0x7f1b0d7fa700 (LWP 105183)): >>>>>> #0 0x00007f1b48dc56d5 in >>>>>> pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 >>>>>> No symbol table info available. >>>>>> #1 0x00007f1b4941b050 in PR_WaitCondVar () from /lib64/libnspr4.so >>>>>> No symbol table info available. >>>>>> #2 0x00007f1b4b70efee in connection_wait_for_new_work () >>>>>> No symbol table info available. >>>>>> #3 0x00007f1b4b710719 in connection_threadmain () >>>>>> No symbol table info available. >>>>>> #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so >>>>>> No symbol table info available. >>>>>> #5 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 >>>>>> No symbol table info available. >>>>>> #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 >>>>>> No symbol table info available. >>>>>> >>>>>> Thread 22 (Thread 0x7f1b0dffb700 (LWP 105182)): >>>>>> #0 0x00007f1b48dc56d5 in >>>>>> pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 >>>>>> No symbol table info available. >>>>>> #1 0x00007f1b4941b050 in PR_WaitCondVar () from /lib64/libnspr4.so >>>>>> No symbol table info available. >>>>>> #2 0x00007f1b4b70efee in connection_wait_for_new_work () >>>>>> No symbol table info available. >>>>>> #3 0x00007f1b4b710719 in connection_threadmain () >>>>>> No symbol table info available. >>>>>> #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so >>>>>> No symbol table info available. >>>>>> #5 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 >>>>>> No symbol table info available. >>>>>> #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 >>>>>> No symbol table info available. >>>>>> >>>>>> Thread 21 (Thread 0x7f1b0e7fc700 (LWP 105181)): >>>>>> #0 0x00007f1b48dc56d5 in >>>>>> pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 >>>>>> No symbol table info available. >>>>>> #1 0x00007f1b4941b050 in PR_WaitCondVar () from /lib64/libnspr4.so >>>>>> No symbol table info available. >>>>>> #2 0x00007f1b4b70efee in connection_wait_for_new_work () >>>>>> No symbol table info available. >>>>>> #3 0x00007f1b4b710719 in connection_threadmain () >>>>>> No symbol table info available. >>>>>> #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so >>>>>> No symbol table info available. >>>>>> #5 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 >>>>>> No symbol table info available. >>>>>> #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 >>>>>> No symbol table info available. >>>>>> >>>>>> Thread 20 (Thread 0x7f1b0effd700 (LWP 105180)): >>>>>> #0 0x00007f1b48dc56d5 in >>>>>> pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 >>>>>> No symbol table info available. >>>>>> #1 0x00007f1b433212f3 in __db_hybrid_mutex_suspend () from /lib64/ >>>>>> libdb-5.3.so >>>>>> No symbol table info available. >>>>>> #2 0x00007f1b43320b37 in __db_tas_mutex_readlock () from /lib64/ >>>>>> libdb-5.3.so >>>>>> No symbol table info available. >>>>>> #3 0x00007f1b4343ad35 in __memp_fget () from /lib64/libdb-5.3.so >>>>>> No symbol table info available. >>>>>> #4 0x00007f1b433e4333 in __dbc_iget () from /lib64/libdb-5.3.so >>>>>> No symbol table info available. >>>>>> #5 0x00007f1b433f2b02 in __dbc_get_pp () from /lib64/libdb-5.3.so >>>>>> No symbol table info available. >>>>>> #6 0x00007f1b3e8ce822 in idl_new_fetch () from >>>>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>>>> No symbol table info available. >>>>>> #7 0x00007f1b3e8dcb26 in index_read_ext_allids () from >>>>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>>>> No symbol table info available. >>>>>> #8 0x00007f1b3e8c7284 in keys2idl () from >>>>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>>>> No symbol table info available. >>>>>> #9 0x00007f1b3e8c79e3 in ava_candidates.isra.0 () from >>>>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>>>> No symbol table info available. >>>>>> #10 0x00007f1b3e8c7fd2 in filter_candidates_ext () from >>>>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>>>> No symbol table info available. >>>>>> #11 0x00007f1b3e8c9096 in list_candidates () from >>>>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>>>> No symbol table info available. >>>>>> #12 0x00007f1b3e8c7f40 in filter_candidates_ext () from >>>>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>>>> No symbol table info available. >>>>>> #13 0x00007f1b3e8c9096 in list_candidates () from >>>>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>>>> No symbol table info available. >>>>>> #14 0x00007f1b3e8c7f40 in filter_candidates_ext () from >>>>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>>>> No symbol table info available. >>>>>> #15 0x00007f1b3e8c9096 in list_candidates () from >>>>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>>>> No symbol table info available. >>>>>> #16 0x00007f1b3e8c7f40 in filter_candidates_ext () from >>>>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>>>> No symbol table info available. >>>>>> #17 0x00007f1b3e90380c in subtree_candidates () from >>>>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>>>> No symbol table info available. >>>>>> #18 0x00007f1b3e904e7f in ldbm_back_search () from >>>>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>>>> No symbol table info available. >>>>>> #19 0x00007f1b4b2321d6 in op_shared_search () from >>>>>> /usr/lib64/dirsrv/libslapd.so.0 >>>>>> No symbol table info available. >>>>>> #20 0x00007f1b4b72333e in do_search () >>>>>> No symbol table info available. >>>>>> #21 0x00007f1b4b7113c3 in connection_threadmain () >>>>>> No symbol table info available. >>>>>> #22 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so >>>>>> No symbol table info available. >>>>>> #23 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 >>>>>> No symbol table info available. >>>>>> #24 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 >>>>>> No symbol table info available. >>>>>> >>>>>> Thread 19 (Thread 0x7f1b0f7fe700 (LWP 105179)): >>>>>> #0 0x00007f1b48dc56d5 in >>>>>> pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 >>>>>> No symbol table info available. >>>>>> #1 0x00007f1b433212f3 in __db_hybrid_mutex_suspend () from /lib64/ >>>>>> libdb-5.3.so >>>>>> No symbol table info available. >>>>>> #2 0x00007f1b43320640 in __db_tas_mutex_lock () from /lib64/ >>>>>> libdb-5.3.so >>>>>> No symbol table info available. >>>>>> #3 0x00007f1b433cad3a in __lock_get_internal () from /lib64/ >>>>>> libdb-5.3.so >>>>>> No symbol table info available. >>>>>> #4 0x00007f1b433cb820 in __lock_get () from /lib64/libdb-5.3.so >>>>>> No symbol table info available. >>>>>> #5 0x00007f1b433f7142 in __db_lget () from /lib64/libdb-5.3.so >>>>>> No symbol table info available. >>>>>> #6 0x00007f1b4333e605 in __bam_search () from /lib64/libdb-5.3.so >>>>>> No symbol table info available. >>>>>> #7 0x00007f1b43329256 in __bamc_search () from /lib64/libdb-5.3.so >>>>>> No symbol table info available. >>>>>> #8 0x00007f1b4332ad0f in __bamc_get () from /lib64/libdb-5.3.so >>>>>> No symbol table info available. >>>>>> #9 0x00007f1b433e3ca6 in __dbc_iget () from /lib64/libdb-5.3.so >>>>>> No symbol table info available. >>>>>> #10 0x00007f1b433f2b02 in __dbc_get_pp () from /lib64/libdb-5.3.so >>>>>> No symbol table info available. >>>>>> #11 0x00007f1b3e8ce680 in idl_new_fetch () from >>>>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>>>> No symbol table info available. >>>>>> #12 0x00007f1b3e8dcb26 in index_read_ext_allids () from >>>>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>>>> No symbol table info available. >>>>>> #13 0x00007f1b3e8c7284 in keys2idl () from >>>>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>>>> No symbol table info available. >>>>>> #14 0x00007f1b3e8c79e3 in ava_candidates.isra.0 () from >>>>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>>>> No symbol table info available. >>>>>> #15 0x00007f1b3e8c7fd2 in filter_candidates_ext () from >>>>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>>>> No symbol table info available. >>>>>> #16 0x00007f1b3e8c9096 in list_candidates () from >>>>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>>>> No symbol table info available. >>>>>> #17 0x00007f1b3e8c7f40 in filter_candidates_ext () from >>>>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>>>> No symbol table info available. >>>>>> #18 0x00007f1b3e8c9096 in list_candidates () from >>>>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>>>> No symbol table info available. >>>>>> #19 0x00007f1b3e8c7f40 in filter_candidates_ext () from >>>>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>>>> No symbol table info available. >>>>>> #20 0x00007f1b3e8c9096 in list_candidates () from >>>>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>>>> No symbol table info available. >>>>>> #21 0x00007f1b3e8c7f40 in filter_candidates_ext () from >>>>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>>>> No symbol table info available. >>>>>> #22 0x00007f1b3e90380c in subtree_candidates () from >>>>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>>>> No symbol table info available. >>>>>> #23 0x00007f1b3e904e7f in ldbm_back_search () from >>>>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>>>> No symbol table info available. >>>>>> #24 0x00007f1b4b2321d6 in op_shared_search () from >>>>>> /usr/lib64/dirsrv/libslapd.so.0 >>>>>> No symbol table info available. >>>>>> #25 0x00007f1b4b72333e in do_search () >>>>>> No symbol table info available. >>>>>> #26 0x00007f1b4b7113c3 in connection_threadmain () >>>>>> No symbol table info available. >>>>>> #27 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so >>>>>> No symbol table info available. >>>>>> #28 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 >>>>>> No symbol table info available. >>>>>> #29 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 >>>>>> No symbol table info available. >>>>>> >>>>>> Thread 18 (Thread 0x7f1b0ffff700 (LWP 105178)): >>>>>> #0 0x00007f1b48dc56d5 in >>>>>> pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 >>>>>> No symbol table info available. >>>>>> #1 0x00007f1b4941b050 in PR_WaitCondVar () from /lib64/libnspr4.so >>>>>> No symbol table info available. >>>>>> #2 0x00007f1b4b70efee in connection_wait_for_new_work () >>>>>> No symbol table info available. >>>>>> #3 0x00007f1b4b710719 in connection_threadmain () >>>>>> No symbol table info available. >>>>>> #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so >>>>>> No symbol table info available. >>>>>> #5 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 >>>>>> No symbol table info available. >>>>>> #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 >>>>>> No symbol table info available. >>>>>> >>>>>> Thread 17 (Thread 0x7f1b14d28700 (LWP 105177)): >>>>>> #0 0x00007f1b48dc56d5 in >>>>>> pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 >>>>>> No symbol table info available. >>>>>> #1 0x00007f1b4941b050 in PR_WaitCondVar () from /lib64/libnspr4.so >>>>>> No symbol table info available. >>>>>> #2 0x00007f1b4b70efee in connection_wait_for_new_work () >>>>>> No symbol table info available. >>>>>> #3 0x00007f1b4b710719 in connection_threadmain () >>>>>> No symbol table info available. >>>>>> #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so >>>>>> No symbol table info available. >>>>>> #5 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 >>>>>> No symbol table info available. >>>>>> #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 >>>>>> No symbol table info available. >>>>>> >>>>>> Thread 16 (Thread 0x7f1b15529700 (LWP 105176)): >>>>>> #0 0x00007f1b48dc56d5 in >>>>>> pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 >>>>>> No symbol table info available. >>>>>> #1 0x00007f1b4941b050 in PR_WaitCondVar () from /lib64/libnspr4.so >>>>>> No symbol table info available. >>>>>> #2 0x00007f1b4b70efee in connection_wait_for_new_work () >>>>>> No symbol table info available. >>>>>> #3 0x00007f1b4b710719 in connection_threadmain () >>>>>> No symbol table info available. >>>>>> #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so >>>>>> No symbol table info available. >>>>>> #5 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 >>>>>> No symbol table info available. >>>>>> #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 >>>>>> No symbol table info available. >>>>>> >>>>>> Thread 15 (Thread 0x7f1b15d2a700 (LWP 105175)): >>>>>> #0 0x00007f1b48dc56d5 in >>>>>> pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 >>>>>> No symbol table info available. >>>>>> #1 0x00007f1b4941b050 in PR_WaitCondVar () from /lib64/libnspr4.so >>>>>> No symbol table info available. >>>>>> #2 0x00007f1b4b70efee in connection_wait_for_new_work () >>>>>> No symbol table info available. >>>>>> #3 0x00007f1b4b710719 in connection_threadmain () >>>>>> No symbol table info available. >>>>>> #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so >>>>>> No symbol table info available. >>>>>> #5 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 >>>>>> No symbol table info available. >>>>>> #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 >>>>>> No symbol table info available. >>>>>> >>>>>> Thread 14 (Thread 0x7f1b1652b700 (LWP 105174)): >>>>>> #0 0x00007f1b48dc56d5 in >>>>>> pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 >>>>>> No symbol table info available. >>>>>> #1 0x00007f1b4941b050 in PR_WaitCondVar () from /lib64/libnspr4.so >>>>>> No symbol table info available. >>>>>> #2 0x00007f1b4b70efee in connection_wait_for_new_work () >>>>>> No symbol table info available. >>>>>> #3 0x00007f1b4b710719 in connection_threadmain () >>>>>> No symbol table info available. >>>>>> #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so >>>>>> No symbol table info available. >>>>>> #5 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 >>>>>> No symbol table info available. >>>>>> #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 >>>>>> No symbol table info available. >>>>>> >>>>>> Thread 13 (Thread 0x7f1b16d2c700 (LWP 105173)): >>>>>> #0 0x00007f1b48dc56d5 in >>>>>> pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 >>>>>> No symbol table info available. >>>>>> #1 0x00007f1b433212f3 in __db_hybrid_mutex_suspend () from /lib64/ >>>>>> libdb-5.3.so >>>>>> No symbol table info available. >>>>>> #2 0x00007f1b43320640 in __db_tas_mutex_lock () from /lib64/ >>>>>> libdb-5.3.so >>>>>> No symbol table info available. >>>>>> #3 0x00007f1b433cad3a in __lock_get_internal () from /lib64/ >>>>>> libdb-5.3.so >>>>>> No symbol table info available. >>>>>> #4 0x00007f1b433cb820 in __lock_get () from /lib64/libdb-5.3.so >>>>>> No symbol table info available. >>>>>> #5 0x00007f1b433f7142 in __db_lget () from /lib64/libdb-5.3.so >>>>>> No symbol table info available. >>>>>> #6 0x00007f1b4333e605 in __bam_search () from /lib64/libdb-5.3.so >>>>>> No symbol table info available. >>>>>> #7 0x00007f1b43329256 in __bamc_search () from /lib64/libdb-5.3.so >>>>>> No symbol table info available. >>>>>> #8 0x00007f1b4332ad0f in __bamc_get () from /lib64/libdb-5.3.so >>>>>> No symbol table info available. >>>>>> #9 0x00007f1b433e3ca6 in __dbc_iget () from /lib64/libdb-5.3.so >>>>>> No symbol table info available. >>>>>> #10 0x00007f1b433f2b02 in __dbc_get_pp () from /lib64/libdb-5.3.so >>>>>> No symbol table info available. >>>>>> #11 0x00007f1b3e8ce680 in idl_new_fetch () from >>>>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>>>> No symbol table info available. >>>>>> #12 0x00007f1b3e8dcb26 in index_read_ext_allids () from >>>>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>>>> No symbol table info available. >>>>>> #13 0x00007f1b3e8c7284 in keys2idl () from >>>>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>>>> No symbol table info available. >>>>>> #14 0x00007f1b3e8c79e3 in ava_candidates.isra.0 () from >>>>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>>>> No symbol table info available. >>>>>> #15 0x00007f1b3e8c7fd2 in filter_candidates_ext () from >>>>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>>>> No symbol table info available. >>>>>> #16 0x00007f1b3e8c9096 in list_candidates () from >>>>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>>>> No symbol table info available. >>>>>> #17 0x00007f1b3e8c7f40 in filter_candidates_ext () from >>>>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>>>> No symbol table info available. >>>>>> #18 0x00007f1b3e8c9096 in list_candidates () from >>>>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>>>> No symbol table info available. >>>>>> #19 0x00007f1b3e8c7f40 in filter_candidates_ext () from >>>>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>>>> No symbol table info available. >>>>>> #20 0x00007f1b3e8c9096 in list_candidates () from >>>>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>>>> No symbol table info available. >>>>>> #21 0x00007f1b3e8c7f40 in filter_candidates_ext () from >>>>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>>>> No symbol table info available. >>>>>> #22 0x00007f1b3e90380c in subtree_candidates () from >>>>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>>>> No symbol table info available. >>>>>> #23 0x00007f1b3e904e7f in ldbm_back_search () from >>>>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>>>> No symbol table info available. >>>>>> #24 0x00007f1b4b2321d6 in op_shared_search () from >>>>>> /usr/lib64/dirsrv/libslapd.so.0 >>>>>> No symbol table info available. >>>>>> #25 0x00007f1b4b72333e in do_search () >>>>>> No symbol table info available. >>>>>> #26 0x00007f1b4b7113c3 in connection_threadmain () >>>>>> No symbol table info available. >>>>>> #27 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so >>>>>> No symbol table info available. >>>>>> #28 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 >>>>>> No symbol table info available. >>>>>> #29 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 >>>>>> No symbol table info available. >>>>>> >>>>>> Thread 12 (Thread 0x7f1b1752d700 (LWP 105172)): >>>>>> #0 0x00007f1b48dc56d5 in >>>>>> pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 >>>>>> No symbol table info available. >>>>>> #1 0x00007f1b4941b050 in PR_WaitCondVar () from /lib64/libnspr4.so >>>>>> No symbol table info available. >>>>>> #2 0x00007f1b4b70efee in connection_wait_for_new_work () >>>>>> No symbol table info available. >>>>>> #3 0x00007f1b4b710719 in connection_threadmain () >>>>>> No symbol table info available. >>>>>> #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so >>>>>> No symbol table info available. >>>>>> #5 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 >>>>>> No symbol table info available. >>>>>> #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 >>>>>> No symbol table info available. >>>>>> >>>>>> Thread 11 (Thread 0x7f1b17d2e700 (LWP 105171)): >>>>>> #0 0x00007f1b48dc56d5 in >>>>>> pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 >>>>>> No symbol table info available. >>>>>> #1 0x00007f1b4941b050 in PR_WaitCondVar () from /lib64/libnspr4.so >>>>>> No symbol table info available. >>>>>> #2 0x00007f1b4b70efee in connection_wait_for_new_work () >>>>>> No symbol table info available. >>>>>> #3 0x00007f1b4b710719 in connection_threadmain () >>>>>> No symbol table info available. >>>>>> #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so >>>>>> No symbol table info available. >>>>>> #5 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 >>>>>> No symbol table info available. >>>>>> #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 >>>>>> No symbol table info available. >>>>>> >>>>>> Thread 10 (Thread 0x7f1b2509d700 (LWP 105168)): >>>>>> #0 0x00007f1b48dc5a82 in pthread_cond_timedwait@@GLIBC_2.3.2 () >>>>>> from /lib64/libpthread.so.0 >>>>>> No symbol table info available. >>>>>> #1 0x00007f1b4941ab07 in pt_TimedWait () from /lib64/libnspr4.so >>>>>> No symbol table info available. >>>>>> #2 0x00007f1b4941afce in PR_WaitCondVar () from /lib64/libnspr4.so >>>>>> No symbol table info available. >>>>>> #3 0x00007f1b4b2061e8 in eq_loop () from >>>>>> /usr/lib64/dirsrv/libslapd.so.0 >>>>>> No symbol table info available. >>>>>> #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so >>>>>> No symbol table info available. >>>>>> #5 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 >>>>>> No symbol table info available. >>>>>> #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 >>>>>> No symbol table info available. >>>>>> >>>>>> Thread 9 (Thread 0x7f1b2589e700 (LWP 105167)): >>>>>> #0 0x00007f1b48dc5a82 in pthread_cond_timedwait@@GLIBC_2.3.2 () >>>>>> from /lib64/libpthread.so.0 >>>>>> No symbol table info available. >>>>>> #1 0x00007f1b4941ab07 in pt_TimedWait () from /lib64/libnspr4.so >>>>>> No symbol table info available. >>>>>> #2 0x00007f1b4941afce in PR_WaitCondVar () from /lib64/libnspr4.so >>>>>> No symbol table info available. >>>>>> #3 0x00007f1b4b719d03 in housecleaning () >>>>>> No symbol table info available. >>>>>> #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so >>>>>> No symbol table info available. >>>>>> #5 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 >>>>>> No symbol table info available. >>>>>> #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 >>>>>> No symbol table info available. >>>>>> >>>>>> Thread 8 (Thread 0x7f1b2609f700 (LWP 105166)): >>>>>> #0 0x00007f1b48dc56d5 in >>>>>> pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 >>>>>> No symbol table info available. >>>>>> #1 0x00007f1b4941b050 in PR_WaitCondVar () from /lib64/libnspr4.so >>>>>> No symbol table info available. >>>>>> #2 0x00007f1b4b260198 in slapi_wait_condvar () from >>>>>> /usr/lib64/dirsrv/libslapd.so.0 >>>>>> No symbol table info available. >>>>>> #3 0x00007f1b3cf5deed in roles_cache_wait_on_change () from >>>>>> /usr/lib64/dirsrv/plugins/libroles-plugin.so >>>>>> No symbol table info available. >>>>>> #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so >>>>>> No symbol table info available. >>>>>> #5 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 >>>>>> No symbol table info available. >>>>>> #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 >>>>>> No symbol table info available. >>>>>> >>>>>> Thread 7 (Thread 0x7f1b268a0700 (LWP 105165)): >>>>>> #0 0x00007f1b48dc56d5 in >>>>>> pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 >>>>>> No symbol table info available. >>>>>> #1 0x00007f1b4941b050 in PR_WaitCondVar () from /lib64/libnspr4.so >>>>>> No symbol table info available. >>>>>> #2 0x00007f1b4b260198 in slapi_wait_condvar () from >>>>>> /usr/lib64/dirsrv/libslapd.so.0 >>>>>> No symbol table info available. >>>>>> #3 0x00007f1b3cf5deed in roles_cache_wait_on_change () from >>>>>> /usr/lib64/dirsrv/plugins/libroles-plugin.so >>>>>> No symbol table info available. >>>>>> #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so >>>>>> No symbol table info available. >>>>>> #5 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 >>>>>> No symbol table info available. >>>>>> #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 >>>>>> No symbol table info available. >>>>>> >>>>>> Thread 6 (Thread 0x7f1b270a1700 (LWP 105164)): >>>>>> #0 0x00007f1b48dc56d5 in >>>>>> pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 >>>>>> No symbol table info available. >>>>>> #1 0x00007f1b4941b050 in PR_WaitCondVar () from /lib64/libnspr4.so >>>>>> No symbol table info available. >>>>>> #2 0x00007f1b4b260198 in slapi_wait_condvar () from >>>>>> /usr/lib64/dirsrv/libslapd.so.0 >>>>>> No symbol table info available. >>>>>> #3 0x00007f1b3cf5deed in roles_cache_wait_on_change () from >>>>>> /usr/lib64/dirsrv/plugins/libroles-plugin.so >>>>>> No symbol table info available. >>>>>> #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so >>>>>> No symbol table info available. >>>>>> #5 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 >>>>>> No symbol table info available. >>>>>> #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 >>>>>> No symbol table info available. >>>>>> >>>>>> Thread 5 (Thread 0x7f1b278a2700 (LWP 105163)): >>>>>> #0 0x00007f1b48dc56d5 in >>>>>> pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 >>>>>> No symbol table info available. >>>>>> #1 0x00007f1b4941b050 in PR_WaitCondVar () from /lib64/libnspr4.so >>>>>> No symbol table info available. >>>>>> #2 0x00007f1b4b260198 in slapi_wait_condvar () from >>>>>> /usr/lib64/dirsrv/libslapd.so.0 >>>>>> No symbol table info available. >>>>>> #3 0x00007f1b415f262e in cos_cache_wait_on_change () from >>>>>> /usr/lib64/dirsrv/plugins/libcos-plugin.so >>>>>> No symbol table info available. >>>>>> #4 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so >>>>>> No symbol table info available. >>>>>> #5 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 >>>>>> No symbol table info available. >>>>>> #6 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 >>>>>> No symbol table info available. >>>>>> >>>>>> Thread 4 (Thread 0x7f1b282ac700 (LWP 105162)): >>>>>> #0 0x00007f1b48ae69b3 in select () from /lib64/libc.so.6 >>>>>> No symbol table info available. >>>>>> #1 0x00007f1b4b2710e9 in DS_Sleep () from >>>>>> /usr/lib64/dirsrv/libslapd.so.0 >>>>>> No symbol table info available. >>>>>> #2 0x00007f1b3e8ba707 in perf_threadmain () from >>>>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>>>> No symbol table info available. >>>>>> #3 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so >>>>>> No symbol table info available. >>>>>> #4 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 >>>>>> No symbol table info available. >>>>>> #5 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 >>>>>> No symbol table info available. >>>>>> >>>>>> Thread 3 (Thread 0x7f1b28aad700 (LWP 105161)): >>>>>> #0 0x00007f1b48ae69b3 in select () from /lib64/libc.so.6 >>>>>> No symbol table info available. >>>>>> #1 0x00007f1b4b2710e9 in DS_Sleep () from >>>>>> /usr/lib64/dirsrv/libslapd.so.0 >>>>>> No symbol table info available. >>>>>> #2 0x00007f1b3e8bfb7f in trickle_threadmain () from >>>>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>>>> No symbol table info available. >>>>>> #3 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so >>>>>> No symbol table info available. >>>>>> #4 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 >>>>>> No symbol table info available. >>>>>> #5 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 >>>>>> No symbol table info available. >>>>>> >>>>>> Thread 2 (Thread 0x7f1b292ae700 (LWP 105160)): >>>>>> #0 0x00007f1b48ae69b3 in select () from /lib64/libc.so.6 >>>>>> No symbol table info available. >>>>>> #1 0x00007f1b4344dbad in __os_yield () from /lib64/libdb-5.3.so >>>>>> No symbol table info available. >>>>>> #2 0x00007f1b434492b3 in __memp_sync_int () from /lib64/libdb-5.3.so >>>>>> No symbol table info available. >>>>>> #3 0x00007f1b43459752 in __txn_checkpoint () from /lib64/ >>>>>> libdb-5.3.so >>>>>> No symbol table info available. >>>>>> #4 0x00007f1b43459b74 in __txn_checkpoint_pp () from /lib64/ >>>>>> libdb-5.3.so >>>>>> No symbol table info available. >>>>>> #5 0x00007f1b3e8c3a87 in checkpoint_threadmain () from >>>>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>>>> No symbol table info available. >>>>>> #6 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so >>>>>> No symbol table info available. >>>>>> #7 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 >>>>>> No symbol table info available. >>>>>> #8 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 >>>>>> No symbol table info available. >>>>>> >>>>>> Thread 1 (Thread 0x7f1b29aaf700 (LWP 105159)): >>>>>> #0 0x00007f1b48ae69b3 in select () from /lib64/libc.so.6 >>>>>> No symbol table info available. >>>>>> #1 0x00007f1b4b2710e9 in DS_Sleep () from >>>>>> /usr/lib64/dirsrv/libslapd.so.0 >>>>>> No symbol table info available. >>>>>> #2 0x00007f1b3e8bf907 in deadlock_threadmain () from >>>>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>>>>> No symbol table info available. >>>>>> #3 0x00007f1b494207bb in _pt_root () from /lib64/libnspr4.so >>>>>> No symbol table info available. >>>>>> #4 0x00007f1b48dc1dc5 in start_thread () from /lib64/libpthread.so.0 >>>>>> No symbol table info available. >>>>>> #5 0x00007f1b48aef28d in clone () from /lib64/libc.so.6 >>>>>> No symbol table info available. >>>>>> >>>>>> >>>>>> >>>>>> Thanks, >>>>>> >>>>>> Rakesh >>>>>> >>>>>> >>>>>> On Mon, Sep 5, 2016 at 4:48 PM, thierry bordaz < >>>>>> tbordaz at redhat.com> wrote: >>>>>> >>>>>>> >>>>>>> >>>>>>> On 09/05/2016 12:05 PM, Rakesh Rajasekharan wrote: >>>>>>> >>>>>>> Hi Thierry, >>>>>>> >>>>>>> >>>>>>> I was getting the hang issue while running ipa-client-install >>>>>>> simultaneously on few clients.. >>>>>>> However, today, I am not able to replicate that. >>>>>>> >>>>>>> I could not get a gdb . But i will try getting that the next time I >>>>>>> face this issue. >>>>>>> >>>>>>> The CPU does not stay high.. it just momentarily touches a high >>>>>>> value and then drops down to around 2-7% >>>>>>> >>>>>>> One question I have is , is it ok to set it nsslapd-threadnumber to >>>>>>> a very high value . >>>>>>> I have around 4000 clients and with nsslapd-maxthreadsperconn set to >>>>>>> 5..So, can I set nsslapd-threadnumber to around 25000. >>>>>>> >>>>>>> >>>>>>> Hello, >>>>>>> >>>>>>> I know some users running in production with several hunderds of >>>>>>> threads (>600) and this without problem. >>>>>>> >>>>>>> I do not recall having suggested to increase that number and for >>>>>>> what reason. >>>>>>> Usually 30 workers is a good enough value. It can create bootleneck >>>>>>> if for some reason each operation is very long to satisfy and exhaust the >>>>>>> number of workers. You can monitor the work queue: >>>>>>> >>>>>>> ldapsearch -D "cn=directory manager" -w xxx -LLL -b "cn=monitor" -s >>>>>>> base opsinitiated opscompleted >>>>>>> >>>>>>> >>>>>>> If opscompleted-opsinitiated remains close to threadnumber, then yes >>>>>>> it would be valuable to increase it. >>>>>>> >>>>>>> The computation #client * #async_op_per_client sound an overkill. >>>>>>> Even if all clients send at the exact same time all their requests, it is >>>>>>> very likely that some common resource (db page, log, allocator...) will >>>>>>> serialize them. If you monitor a need to increase the work, you would for >>>>>>> example set it to 50, then monitor, then set it to 100, then monitor... >>>>>>> until you find a good enough value. >>>>>>> Note the increasing the #thread, increases the memory footprint that >>>>>>> will reduce the efficiency of file system cache and can increase the >>>>>>> response time. >>>>>>> >>>>>>> >>>>>>> best regards >>>>>>> thierry >>>>>>> >>>>>>> >>>>>>> Thanks >>>>>>> >>>>>>> On Mon, Sep 5, 2016 at 1:03 PM, thierry bordaz < >>>>>>> tbordaz at redhat.com> wrote: >>>>>>> >>>>>>>> >>>>>>>> Hi Rakesh, >>>>>>>> >>>>>>>> Were you able to get a pstack or full stack with gdb ( >>>>>>>> http://www.port389.org/docs/3 >>>>>>>> 89ds/FAQ/faq.html#debugging-crashes) when the server hangs ? >>>>>>>> >>>>>>>> If it happens with 500 threads as well as with 30, using 30 threads >>>>>>>> is a better choice to debug this issue. >>>>>>>> I will try to reproduce using 150 parallel 'ipa user-find >>>>>>>> p-testipa' commands >>>>>>>> >>>>>>>> Something I am unsure is if the CPU consumption stays high (you >>>>>>>> mentioned 340% CPU usage) as long as the hang happens or if after a suddent >>>>>>>> shot up to 340% (that marks the beginning of the hang) it drops and stay >>>>>>>> hanging ? >>>>>>>> >>>>>>>> thanks >>>>>>>> thierry >>>>>>>> >>>>>>>> On 09/04/2016 08:40 PM, Rakesh Rajasekharan wrote: >>>>>>>> >>>>>>>> starce on the slapd process actually had this in the output.. >>>>>>>> FUTEX_WAIT_PRIVATE >>>>>>>> >>>>>>>> and checking for the number of threads slapd had.. there were 5015 >>>>>>>> threads >>>>>>>> >>>>>>>> ps -efL|grep slapd|wc -l >>>>>>>> 5015 >>>>>>>> >>>>>>>> strace on most of the threads gave this output >>>>>>>> >>>>>>>> strace -p 67411 >>>>>>>> Process 67411 attached >>>>>>>> futex(0x7f3f0226b41c, FUTEX_WAIT_PRIVATE, 1, NULL) = -1 EAGAIN >>>>>>>> (Resource temporarily unavailable) >>>>>>>> futex(0x7f3f0226b41c, FUTEX_WAIT_PRIVATE, 2, NULL^CProcess 67411 >>>>>>>> detached >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> On Sun, Sep 4, 2016 at 5:34 PM, Rakesh Rajasekharan < >>>>>>>> rakesh.rajasekharan at gmail.com> >>>>>>>> wrote: >>>>>>>> >>>>>>>>> I have again got the issue of IPA hanging.. The issue came up when >>>>>>>>> i tried to run ipa-client-isntall on 142 clients simultaneously >>>>>>>>> >>>>>>>>> >>>>>>>>> None of the IPA commands are responding, and I see this error >>>>>>>>> >>>>>>>>> ipa user-find p-testipa >>>>>>>>> ipa: ERROR: Insufficient access: SASL(-1): generic failure: GSSAPI >>>>>>>>> Error: Unspecified GSS failure. Minor code may provide more information >>>>>>>>> (KDC returned error string: PROCESS_TGS) >>>>>>>>> >>>>>>>>> KRB5_TRACE=/dev/stdout kinit admin >>>>>>>>> [41178] 1472984115.233214: Getting initial credentials for >>>>>>>>> admin at XYZ.COM >>>>>>>>> [41178] 1472984115.235257: Sending request (167 bytes) to XYZ.COM >>>>>>>>> [41178] 1472984115.235419: Initiating TCP connection to stream >>>>>>>>> 10.1.3.36:88 >>>>>>>>> [41178] 1472984115.235685: Sending TCP request to stream >>>>>>>>> 10.1.3.36:88 >>>>>>>>> [41178] 1472984120.238914: Received answer (174 bytes) from stream >>>>>>>>> 10.1.3.36:88 >>>>>>>>> [41178] 1472984120.238925: Terminating TCP connection to stream >>>>>>>>> 10.1.3.36:88 >>>>>>>>> [41178] 1472984120.238993: Response was from master KDC >>>>>>>>> [41 >>>>>>>>> >>>>>>>>> >>>>>>>>> Running an ldapsearch to see the db.. does not give any results >>>>>>>>> and just hangs there >>>>>>>>> >>>>>>>>> ldapsearch -x -D 'cn=Directory Manager' -W -s one -b >>>>>>>>> 'cn=kerberos,dc=xyz,dc=com' >>>>>>>>> Enter LDAP Password: >>>>>>>>> >>>>>>>>> even an ldapsearch -x does not respond >>>>>>>>> At this point, am sure that slapd is the one causing issues >>>>>>>>> >>>>>>>>> Running an strace against the hung slapd itself seems to get stuck >>>>>>>>> does not proceed after saying "attaching to process" >>>>>>>>> >>>>>>>>> From some others posts I read Thierry suggesting to increase the >>>>>>>>> nsslapd-threadnumber value >>>>>>>>> >>>>>>>>> It was set to 30, I think that might be too low. >>>>>>>>> >>>>>>>> ... >>>>>> >>>>>> [Message clipped] >>>>> >>>>> >>>>> >>>> >>>> >>>> -- >>>> Red Hat GmbH, http://www.de.redhat.com/, Registered seat: Grasbrunn, >>>> Commercial register: Amtsgericht Muenchen, HRB 153243, >>>> Managing Directors: Charles Cachera, Michael Cunningham, Michael O'Neill, Eric Shander >>>> >>>> >>> >> >> > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From peljasz at yahoo.co.uk Fri Sep 16 11:51:33 2016 From: peljasz at yahoo.co.uk (lejeczek) Date: Fri, 16 Sep 2016 12:51:33 +0100 Subject: [Freeipa-users] how to revert ipa-adtrust-install... In-Reply-To: <20160916045704.5emkdqda3uvqdzlc@redhat.com> References: <1ec82d0c-1350-feb3-4da5-a022eb7ac5e5@yahoo.co.uk> <20160915191709.6vk7y3gidzxoicen@redhat.com> <98df2894-d5c7-082c-8fe8-77ece9474301@yahoo.co.uk> <20160915200055.qpx7gma722tjcfxk@redhat.com> <57DB1487.4000500@redhat.com> <20160916045704.5emkdqda3uvqdzlc@redhat.com> Message-ID: I appreciate the fact that it might be a complex task, however a supported(CLI) way to if not revert it all back to pre-install state but at least to take samba out of IPA's hands would be nice to have. Would it be ok to leave IPA+ds389 part as is and only change, reconfigure Samba - I believe so - if yes then a CLI option to achieve this would be very desired. Rob's workaround only... "built-in". many thanks On 16/09/16 05:57, Alexander Bokovoy wrote: > So we decided to not perform 'ipa-adtrust-install > --uninstall' as it > makes no sense. If somebode is willing to uninstall > 'ipa-adtrust-install', then need to realize what they are > doing as it > would need to remove certain configuration in IPA LDAP > because there are > actual 389-ds plugins that depend on the configuration and > work jointly > with ipasam module in Samba to provide common setup. If > 'ipasam' is > missing, those modules also become useless. From natxo.asenjo at gmail.com Fri Sep 16 12:26:24 2016 From: natxo.asenjo at gmail.com (Natxo Asenjo) Date: Fri, 16 Sep 2016 14:26:24 +0200 Subject: [Freeipa-users] certificates not renewing CA_UNREACHEABLE In-Reply-To: References: <806bdf32-558c-4972-255a-0901f47c4c4b@redhat.com> <6596593a-9470-bebd-f236-f9d17aa4a56c@redhat.com> Message-ID: hi, On Fri, Sep 16, 2016 at 10:34 AM, Martin Basti wrote: > > > On 16.09.2016 09:38, Natxo Asenjo wrote: > > hi, > > > On Thu, Sep 15, 2016 at 1:03 PM, Natxo Asenjo >> >> On Thu, Sep 15, 2016 at 12:49 PM, Martin Basti wrote: >> >>> >>> >>> On 15.09.2016 12:44, Natxo Asenjo wrote: >>> >>> hi, >>> >>> On Thu, Sep 15, 2016 at 12:33 PM, Martin Basti >>> wrote: >>> >>>> >>>> Hello, >>>> >>>> usually the most information can be found here >>>> /var/log/pki/pki-tomcat/ca/debug >>>> >>> >>> mmm, in this centos 6.8 system that does not exist: >>> >>> # ls -l /var/log/pki/pki-tomcat/ca/debug >>> ls: cannot access /var/log/pki/pki-tomcat/ca/debug: No such file or >>> directory >>> >>> >>> I do have a /var/log/pki-ca/debug >>> >>> >>> >>> Does it contain any information related to your issue? >>> >> >> I have tried renewing the certificate: >> >> ipa-getcert resubmit -i 20121107212513 >> >> >> If I grep that file for that request id I find nothing recent, just in >> the ipaserver installation log >> >> # cd /var/log >> # grep -ri 20121107212513 *.log >> ipaserver-install.log:2012-11-07T21:25:13Z DEBUG stdout=New tracking >> request "20121107212513" added. >> >> # grep -ri 20121107212513 pki-ca >> # >> >> > Any clues? > > > -- > Groeten, > natxo > > > > Sorry, I'm quite lost here, maybe somebody from dogtag can help what might > be reason of those CA errors > do I need to ask in the dogtag list? -- -- Groeten, natxo -------------- next part -------------- An HTML attachment was scrubbed... URL: From abokovoy at redhat.com Fri Sep 16 12:31:41 2016 From: abokovoy at redhat.com (Alexander Bokovoy) Date: Fri, 16 Sep 2016 15:31:41 +0300 Subject: [Freeipa-users] how to revert ipa-adtrust-install... In-Reply-To: References: <1ec82d0c-1350-feb3-4da5-a022eb7ac5e5@yahoo.co.uk> <20160915191709.6vk7y3gidzxoicen@redhat.com> <98df2894-d5c7-082c-8fe8-77ece9474301@yahoo.co.uk> <20160915200055.qpx7gma722tjcfxk@redhat.com> <57DB1487.4000500@redhat.com> <20160916045704.5emkdqda3uvqdzlc@redhat.com> Message-ID: <20160916123141.gvmpct77ogri5vqj@redhat.com> On Fri, 16 Sep 2016, lejeczek wrote: >I appreciate the fact that it might be a complex task, however a >supported(CLI) way to if not revert it all back to pre-install state >but at least to take samba out of IPA's hands would be nice to have. >Would it be ok to leave IPA+ds389 part as is and only change, >reconfigure Samba - I believe so - if yes then a CLI option to achieve >this would be very desired. >Rob's workaround only... "built-in". We looked at it from multiple angles and decided against it. You are welcome to create a script for that and publish it. -- / Alexander Bokovoy From gjn at gjn.priv.at Fri Sep 16 13:06:16 2016 From: gjn at gjn.priv.at (=?ISO-8859-1?Q?G=FCnther_J=2E?= Niederwimmer) Date: Fri, 16 Sep 2016 15:06:16 +0200 Subject: [Freeipa-users] 3rd party Cert install now IPA total broken Message-ID: <1825019.y8rA6b7sXH@techz> Hello, Freeipa 4.3.1 I have now install a 3rd Party Certificat from Startcom now my IPA is total broken? I make this ipa-cacert-manage -p 'xxxxxxxxxxxxxxxx' -n STARTCOM-ROOT -t C,, install root.crt ipa-certupdate ipa-server-certinstall -w -d ipa_3rd_ca.p12 I create this p12 with key.pem, cert.pem root.crt I insert also in the cert.pem the intermediate.crt the kerberos don't start anymore ? The Error Is Unspecified GSS failure.Minor (2529639068): Cannot contact any KDC for realm '4GJN.COM' after insert in nss.conf "NSSEnforceValidCerts off" ipactl restart is starting (?) but ipactl status tell me Directory Service: RUNNING krb5kdc Service: RUNNING kadmin Service: RUNNING named Service: RUNNING ipa_memcached Service: RUNNING httpd Service: RUNNING ipa-custodia Service: RUNNING pki-tomcatd Service: RUNNING ipa-otpd Service: RUNNING ipa-ods-exporter Service: STOPPED ods-enforcerd Service: RUNNING ipa-dnskeysyncd Service: RUNNING ipa: INFO: The ipactl command was successful with certutil -d /etc/httpd/alias -L I have now this Certificate Nickname Trust Attributes SSL,S/MIME,JAR/XPI Signing-Cert u,u,u 4GJN_CA_FILE u,u,u ipaCert u,u,u 4GJN.COM IPA CA CT,C,C STARTCOM-ROOT C,, I can Insert in nss.conf by the #NSSNickname "Signing-Cert" original or NSSNickname 4GJN_CA_FILE but all is now broken ? I also add this, found in Bugzilla certutil -d /var/lib/pki/pki-tomcat/alias -L Certificate Nickname Trust Attributes SSL,S/MIME,JAR/XPI ocspSigningCert cert-pki-ca u,u,u subsystemCert cert-pki-ca u,u,u caSigningCert cert-pki-ca CTu,Cu,Cu Server-Cert cert-pki-ca u,u,u auditSigningCert cert-pki-ca u,u,Pu STARTCOM-ROOT CT,, this is created in the certutil -d /etc/dirsrv/slapd-4GJN.COM -L Certificate Nickname Trust Attributes SSL,S/MIME,JAR/XPI 4GJN_CA_FILE u,u,u 4GJN.COM IPA CA CT,C,C STARTCOM-ROOT C,, Can any help a little, please ;-) The bad Problem, I tested this with my master server with DNS / DNSSEC I can't new install (DNSSEC Keys) -- mit freundlichen Gr??en / best regards, G?nther J. Niederwimmer From blipton at redhat.com Fri Sep 16 14:15:15 2016 From: blipton at redhat.com (Ben Lipton) Date: Fri, 16 Sep 2016 10:15:15 -0400 Subject: [Freeipa-users] ipa: ERROR: Certificate format error: (SEC_ERROR_LEGACY_DATABASE) The certificate/key database is in an old, unsupported format. In-Reply-To: References: <57D015D8.7080507@redhat.com> <57D166B9.3070201@redhat.com> <57D70676.8080107@redhat.com> <57D81AA1.4050400@redhat.com> <4eac5df5-ded2-8705-d5ba-6ab9dde5ad4d@redhat.com> Message-ID: <01d22dde-941b-588c-3842-03c2159b9246@redhat.com> On 09/16/2016 03:39 AM, Natxo Asenjo wrote: > hi, > > > On Thu, Sep 15, 2016 at 2:25 PM, Natxo Asenjo > wrote: > > hi, > > attached error_log > > > > Any clues? > > Thanks! > > -- > -- > Groeten, > natxo Sorry, I'm not having any luck tracking down the answer. Maybe someone else has an idea. Looking at the code, /etc/httpd/alias does seem to be the relevant database. One last thought: does anything change if selinux is in permissive mode? Is there anything interesting in /var/log/audit/audit.log? Again, it doesn't make a whole lot of sense because it works sometimes, but maybe something is changing a few minutes after boot time? -------------- next part -------------- An HTML attachment was scrubbed... URL: From rcritten at redhat.com Fri Sep 16 14:22:01 2016 From: rcritten at redhat.com (Rob Crittenden) Date: Fri, 16 Sep 2016 10:22:01 -0400 Subject: [Freeipa-users] certificates not renewing CA_UNREACHEABLE In-Reply-To: References: <806bdf32-558c-4972-255a-0901f47c4c4b@redhat.com> <6596593a-9470-bebd-f236-f9d17aa4a56c@redhat.com> Message-ID: <57DC0009.4030602@redhat.com> Natxo Asenjo wrote: > hi, > > > On Fri, Sep 16, 2016 at 10:34 AM, Martin Basti > wrote: > > > > On 16.09.2016 09:38, Natxo Asenjo wrote: >> hi, >> >> >> On Thu, Sep 15, 2016 at 1:03 PM, Natxo Asenjo >> >> >> On Thu, Sep 15, 2016 at 12:49 PM, Martin Basti >> > wrote: >> >> >> >> On 15.09.2016 12:44, Natxo Asenjo wrote: >>> hi, >>> >>> On Thu, Sep 15, 2016 at 12:33 PM, Martin Basti >>> > wrote: >>> >>> >>> Hello, >>> >>> usually the most information can be found here >>> /var/log/pki/pki-tomcat/ca/debug >>> >>> >>> mmm, in this centos 6.8 system that does not exist: >>> >>> # ls -l /var/log/pki/pki-tomcat/ca/debug >>> ls: cannot access /var/log/pki/pki-tomcat/ca/debug: No >>> such file or directory >>> >>> >>> I do have a /var/log/pki-ca/debug >>> >>> >> Does it contain any information related to your issue? >> >> >> I have tried renewing the certificate: >> >> ipa-getcert resubmit -i 20121107212513 >> >> >> If I grep that file for that request id I find nothing recent, >> just in the ipaserver installation log >> >> # cd /var/log >> # grep -ri 20121107212513 *.log >> ipaserver-install.log:2012-11-07T21:25:13Z DEBUG stdout=New >> tracking request "20121107212513" added. >> >> # grep -ri 20121107212513 pki-ca >> # >> >> >> Any clues? >> >> >> -- >> Groeten, >> natxo > > > Sorry, I'm quite lost here, maybe somebody from dogtag can help what > might be reason of those CA errors > > > > do I need to ask in the dogtag list? You won't find any errors on this in the dogtag logs because it isn't getting that far. The 3 certs you list are the ones that are renewed via the IPA API (as opposed to the subsystem certs renewed directly by dogtag). I think the failures are all related. I had someone else report the CSR decoding failure and he just restarted IPA and that fixes things for him though it was a rather unsatisfying fix. What I'd do is this. Assuming each step works, move onto the next. 1. ipa cert-show 1 The serial # picked more or less at random, we're testing connectivity and that the CA is up and operational. 2. I assume that getcert list | grep expire shows all certs currently valid? The IPA service certs expire in a month, how about the CA subsystem certs? 3. Is this the same server having problems talking to the CA due to the other NSS errors? If so what I'd do is restart httpd then immediately use ipa-getcert to resubmit the requests to try to get into that few minute window. If this is the same box you already have debugging enabled so seeing what that shows might be helpful. rob From pvoborni at redhat.com Fri Sep 16 14:41:08 2016 From: pvoborni at redhat.com (Petr Vobornik) Date: Fri, 16 Sep 2016 16:41:08 +0200 Subject: [Freeipa-users] ipa: ERROR: Certificate format error: (SEC_ERROR_LEGACY_DATABASE) The certificate/key database is in an old, unsupported format. In-Reply-To: References: <57D015D8.7080507@redhat.com> <57D166B9.3070201@redhat.com> <57D70676.8080107@redhat.com> <57D81AA1.4050400@redhat.com> <4eac5df5-ded2-8705-d5ba-6ab9dde5ad4d@redhat.com> Message-ID: <2205f14d-a876-fb89-1fe5-e9076eb778ec@redhat.com> On 09/16/2016 09:39 AM, Natxo Asenjo wrote: > hi, > > > Any clues? > output of $ cat error_log | grep INFO -A 1 | cut -c -120 shows that first cert-show was successful. It was followed by cert-request. cert-request internally called - host-show - cert_show(1) success - cert_show(162) success - ipaserver.plugins.dogtag.ra.get_certificate() https_request 'https://xx.xxx.xxx.xx:443/ca/agent/ca/displayBySerial' - cert_revoke(162, recvocation_reason=4) - cert_show(162) success - cert_show(1) - success - ipaserver.plugins.dogtag.ra.revoke_certificate() - https_request 'https://xx.xxx.xxx.xx:443/ca/agent/ca/doRevoke' ends with: NetworkError [Thu Sep 15 13:08:23 2016] [error] ipa: DEBUG: response: NetworkError: cannot connect to 'https://xx.xxx.xxx.xx:443/ca/agent/ca/doRevoke': (SEC_ERROR_BUSY) NSS could not shutdown. Objects are still in use. After it every other communication with CA ends with the issue in subject: cert_show(u'15'): NetworkError [Thu Sep 15 13:08:26 2016] [error] ipa: DEBUG: response: NetworkError: cannot connect to 'https://xx.xxx.xxx.xxl:443/ca/agent/ca/displayBySerial': (SEC_ERROR_LEGACY_DATABASE) The certificate/key database is in an old, unsupported format. So the main issue is "NSS could not shutdown." Investigation of that is beyond me. Maybe a workaround can be do first revoke existing cert for the host and then request a new one - which might trigger a different sequence of calls and hopefully not reproduce the issue. But the issue will be still present. -- Petr Vobornik From pvoborni at redhat.com Fri Sep 16 14:43:36 2016 From: pvoborni at redhat.com (Petr Vobornik) Date: Fri, 16 Sep 2016 16:43:36 +0200 Subject: [Freeipa-users] certificates not renewing CA_UNREACHEABLE In-Reply-To: <57DC0009.4030602@redhat.com> References: <806bdf32-558c-4972-255a-0901f47c4c4b@redhat.com> <6596593a-9470-bebd-f236-f9d17aa4a56c@redhat.com> <57DC0009.4030602@redhat.com> Message-ID: On 09/16/2016 04:22 PM, Rob Crittenden wrote: > Natxo Asenjo wrote: >> hi, >> >> >> On Fri, Sep 16, 2016 at 10:34 AM, Martin Basti > > wrote: >> >> >> >> On 16.09.2016 09:38, Natxo Asenjo wrote: >>> hi, >>> >>> >>> On Thu, Sep 15, 2016 at 1:03 PM, Natxo Asenjo >>> >>> >>> On Thu, Sep 15, 2016 at 12:49 PM, Martin Basti >>> > wrote: >>> >>> >>> >>> On 15.09.2016 12:44, Natxo Asenjo wrote: >>>> hi, >>>> >>>> On Thu, Sep 15, 2016 at 12:33 PM, Martin Basti >>>> > wrote: >>>> >>>> >>>> Hello, >>>> >>>> usually the most information can be found here >>>> /var/log/pki/pki-tomcat/ca/debug >>>> >>>> >>>> mmm, in this centos 6.8 system that does not exist: >>>> >>>> # ls -l /var/log/pki/pki-tomcat/ca/debug >>>> ls: cannot access /var/log/pki/pki-tomcat/ca/debug: No >>>> such file or directory >>>> >>>> >>>> I do have a /var/log/pki-ca/debug >>>> >>>> >>> Does it contain any information related to your issue? >>> >>> >>> I have tried renewing the certificate: >>> >>> ipa-getcert resubmit -i 20121107212513 >>> >>> >>> If I grep that file for that request id I find nothing recent, >>> just in the ipaserver installation log >>> >>> # cd /var/log >>> # grep -ri 20121107212513 *.log >>> ipaserver-install.log:2012-11-07T21:25:13Z DEBUG stdout=New >>> tracking request "20121107212513" added. >>> >>> # grep -ri 20121107212513 pki-ca >>> # >>> >>> >>> Any clues? >>> >>> >>> -- >>> Groeten, >>> natxo >> >> >> Sorry, I'm quite lost here, maybe somebody from dogtag can help what >> might be reason of those CA errors >> >> >> >> do I need to ask in the dogtag list? > > You won't find any errors on this in the dogtag logs because it isn't > getting that far. > > The 3 certs you list are the ones that are renewed via the IPA API (as > opposed to the subsystem certs renewed directly by dogtag). I think the > failures are all related. I had someone else report the CSR decoding > failure and he just restarted IPA and that fixes things for him though > it was a rather unsatisfying fix. > > What I'd do is this. Assuming each step works, move onto the next. > > 1. ipa cert-show 1 > > The serial # picked more or less at random, we're testing connectivity > and that the CA is up and operational. > > 2. I assume that getcert list | grep expire shows all certs currently > valid? The IPA service certs expire in a month, how about the CA > subsystem certs? > > 3. Is this the same server having problems talking to the CA due to the > other NSS errors? If so what I'd do is restart httpd then immediately > use ipa-getcert to resubmit the requests to try to get into that few > minute window. The error log from thread [Freeipa-users] ipa: ERROR: Certificate format error: (SEC_ERROR_LEGACY_DATABASE) The certificate/key database is in an old, unsupported format." looks like it. > > If this is the same box you already have debugging enabled so seeing > what that shows might be helpful. > > rob > -- Petr Vobornik From abrook at bsd.uchicago.edu Fri Sep 16 14:47:20 2016 From: abrook at bsd.uchicago.edu (Brook, Andy [CRI]) Date: Fri, 16 Sep 2016 14:47:20 +0000 Subject: [Freeipa-users] Samba Server setup In-Reply-To: <20160916050435.l4k6rshziftp6pzd@redhat.com> References: <73C11E05-045F-4324-9733-44C370584232@bsd.uchicago.edu> <20160915180606.v2z6dkdw3gltem34@redhat.com> <9A3517B6-CFFB-47ED-BDF2-BE3EE0D06EDE@bsd.uchicago.edu> <20160916050435.l4k6rshziftp6pzd@redhat.com> Message-ID: <38F25AFE-C5E8-4CE9-BD18-26949884625A@bsd.uchicago.edu> On 9/16/16, 12:04 AM, "Alexander Bokovoy" wrote: On Thu, 15 Sep 2016, Brook, Andy [CRI] wrote: >On 9/15/16, 1:06 PM, "Alexander Bokovoy" wrote: > > On Thu, 15 Sep 2016, Brook, Andy [CRI] wrote: > >All, > > I?m working on setting up Samba to serve files from a server attached > > to our IPA domain. I followed the directions in > > https://www.freeipa.org/page/Howto/Integrating_a_Samba_File_Server_With_IPA. > > Everything seems to work and I can access the files from another RHEL > > server attached to the same domain using a Kerberos ticket from a > > user from the trusted AD domain. However, I can?t access this share > > from a windows client that is also attached to the trusted AD domain. > > > >My smb.conf is as follows: > >[global] > > workgroup = IPA > > realm = IPA.DOMAIN > > kerberos method = dedicated keytab > > dedicated keytab file = FILE:/etc/samba/samba.keytab > > log file = /var/log/samba/log.%m > > log level = 3 > > security = ads > > load printers = no > > disable spoolss = yes > > map to guest = Never > > restrict anonymous = 2 > > > >[spacetest] > > path = /var/www > > writable = yes > > browsable = yes > > > >I put the keytab in place from the cifs service from the IPA server. > > > >I feel like I?m missing something small, but I can?t seem to find it. > >Logs from samba are here: http://pastebin.com/aMDXfR78 > These logs show that your Windows client did not use Kerberos but tried > to authenticate with password using NTLMSSP. This is not supported yet, > as written on the page you used for the setup guidance. > > You need to find out why Windows client didn't use Kerberos. > Is your trust to AD really working? > >We?re authenticating AD users on the hosts that are connected to IPA. >We?re able to create external groups and associate them with internal >groups for HBAC and sudoers rules. Is there something else I should >check to see if the trust is working? It?s entirely possible I missed >something somewhere in the setup, but I don?t think I did. Start by listing your configuration. Show: - ipa service-show cifs/samba.server.host (using FQDN hostname) - ipa trust-show ad.domain - kinit user at AD.DOMAIN ; KRB5_TRACE=/dev/stderr smbclient -k //samba.server.host/share - Try to access \\samba.server.host\share from Windows host and then show 'klist' in the Windows shell, or alternatively, - if you are using Windows Server 2012 or later, show output of 'klist get cifs/samba.server.host at IPA.DOMAIN' - show any lines related to use user at AD.DOMAIN and cifs/samba.server.host from /var/log/krb5kdc.log on IPA master You can replace actual hostnames/realm names/IP addresses by something more generic in the output when sending to the list, but please do it consistently. I?m sorry. I thought I had been consistent when making changes, but from your response, it looks like I wasn?t. I?m sorry about that. I got yelled at by our security team last time we sent logs to a public list that had any type of identifiable information in them, so it?s sort of a new process for me. I think I have it down now. The results of the commands are here: http://pastebin.com/PRwr7wv6 Andy Brook Sr. Systems Administrator | Center for Research Informatics | University of Chicago T: 773-834-0458 | http://cri.uchicago.edu ******************************************************************************** This e-mail is intended only for the use of the individual or entity to which it is addressed and may contain information that is privileged and confidential. If the reader of this e-mail message is not the intended recipient, you are hereby notified that any dissemination, distribution or copying of this communication is prohibited. If you have received this e-mail in error, please notify the sender and destroy all copies of the transmittal. Thank you University of Chicago Medicine and Biological Sciences ******************************************************************************** From pvoborni at redhat.com Fri Sep 16 15:05:34 2016 From: pvoborni at redhat.com (Petr Vobornik) Date: Fri, 16 Sep 2016 17:05:34 +0200 Subject: [Freeipa-users] how to revert ipa-adtrust-install... In-Reply-To: References: <1ec82d0c-1350-feb3-4da5-a022eb7ac5e5@yahoo.co.uk> <20160915191709.6vk7y3gidzxoicen@redhat.com> <98df2894-d5c7-082c-8fe8-77ece9474301@yahoo.co.uk> <20160915200055.qpx7gma722tjcfxk@redhat.com> <57DB1487.4000500@redhat.com> <20160916045704.5emkdqda3uvqdzlc@redhat.com> Message-ID: <4ae11a44-cf3a-39ce-0732-ba2ca2186a74@redhat.com> On 09/16/2016 01:51 PM, lejeczek wrote: > I appreciate the fact that it might be a complex task, however a > supported(CLI) way to if not revert it all back to pre-install state but > at least to take samba out of IPA's hands would be nice to have. > Would it be ok to leave IPA+ds389 part as is and only change, > reconfigure Samba - I believe so - if yes then a CLI option to achieve > this would be very desired. > Rob's workaround only... "built-in". Out of curiosity: is there a partial broader use case behind this feature request? > > many thanks > > On 16/09/16 05:57, Alexander Bokovoy wrote: >> So we decided to not perform 'ipa-adtrust-install --uninstall' as it >> makes no sense. If somebode is willing to uninstall >> 'ipa-adtrust-install', then need to realize what they are doing as it >> would need to remove certain configuration in IPA LDAP because there are >> actual 389-ds plugins that depend on the configuration and work jointly >> with ipasam module in Samba to provide common setup. If 'ipasam' is >> missing, those modules also become useless. > -- Petr Vobornik From pvoborni at redhat.com Fri Sep 16 15:28:36 2016 From: pvoborni at redhat.com (Petr Vobornik) Date: Fri, 16 Sep 2016 17:28:36 +0200 Subject: [Freeipa-users] ipa-server-certinstall -w -d mysite.key mysite.crt In-Reply-To: <3132705.gf7jV6EcmM@techz> References: <3132705.gf7jV6EcmM@techz> Message-ID: On 09/15/2016 12:42 PM, G?nther J. Niederwimmer wrote: > Hello, > > FreeIPA 4.3.1 > > is it a workaround to install the key and cert > > with this command I have to insert a password, but the key file have no > password? > > Afterward I have a Error from ipa-server-certinstall ? > > Thanks for the Help > Looks to me as bug: https://fedorahosted.org/freeipa/ticket/6032 It was fixed in FreeIPA 4.4.1 (will be in Fedora 25) -- Petr Vobornik From pvoborni at redhat.com Fri Sep 16 15:39:33 2016 From: pvoborni at redhat.com (Petr Vobornik) Date: Fri, 16 Sep 2016 17:39:33 +0200 Subject: [Freeipa-users] CA: Cannot add Centos7.2 replica to Centos6.8 ipa server In-Reply-To: References: <45b66871-aa01-3c91-0630-0bbeee488ae7@nelios.com> <323b9de1-adb3-47af-d4a7-d6083f88d75c@nelios.com> <9afdf0f3-97d1-cc6e-5f3d-0e11687a124a@redhat.com> <05b8e986-ceef-8e75-c76d-b38c45e2e3a1@nelios.com> <79903d5c-2b80-d492-6b29-59ac5b39c6f5@redhat.com> <98b16feb-8b70-13bb-4c92-cd9bca3e408d@redhat.com> <0b546b4a-9ef3-7a15-27a7-3a2d59e5281f@redhat.com> Message-ID: <93f6dbfd-dff8-aa92-b540-cb7281fd29de@redhat.com> On 09/14/2016 07:26 PM, Giorgos Kafataridis wrote: > > > On 09/13/2016 10:36 PM, Endi Sukma Dewata wrote: >> On 9/12/2016 9:35 PM, Endi Sukma Dewata wrote: >>> On 9/9/2016 2:46 PM, Georgios Kafataridis wrote: >>>> I've tried that but still the same result. >>>> >>>> [root at ipa-server /]# ldapsearch -D "cn=directory manager" -W -p 389 -h >>>> localhost -b "uid=admin,ou=people,o=ipaca" >>>> Enter LDAP Password: >>>> # extended LDIF >>>> # >>>> # LDAPv3 >>>> # base with scope subtree >>>> # filter: (objectclass=*) >>>> # requesting: ALL >>>> # >>>> >>>> # search result >>>> search: 2 >>>> result: 32 No such object >>> >>> Hi, >>> >>> The master's logs indicate there's an authentication issue. >>> >>> Could you search the whole directory to find the admin user? >>> $ ldapsearch ... -b "o=ipaca" "(uid=admin)" >>> >>> Try also other suffixes that you have in the DS. >>> >>> If you find it, try to authenticate against DS directly as the admin >>> user. If the authentication fails, try resetting the password. >> >> I believe there is actually another DS instance on CentOS 6.8 running on port >> 7389, so make sure you check that too. If the admin user is indeed missing, it >> will need to be recreated, assigned a password and certificate, and added to >> the appropriate groups. >> >> See also: http://pki.fedoraproject.org/wiki/IPA_PKI_Users >> > > Sorry for the delay, crazy office days. > > Ok, tried that and finally got a hit on the user. Indeed in 6.x you also have > 7389 to look for. > > *Master > > *#ldapsearch -h localhost -p 7389 -b "o=ipaca" "(uid=admin)" -x -W > Enter LDAP Password: > > # extended LDIF > # > # LDAPv3 > # base with scope subtree > # filter: (uid=admin) > # requesting: ALL > # > > # admin, people, ipaca > dn: uid=admin,ou=people,o=ipaca > objectClass: top > objectClass: person > objectClass: organizationalPerson > objectClass: inetOrgPerson > objectClass: cmsuser > uid: admin > sn: admin > cn: admin > mail: root at localhost > usertype: adminType > userstate: 1 > description: 2;6;CN=Certificate Authority,O=NELIOS;CN=ipa-ca-agent,O=NELIOS > userCertificate:: MIIDaTCCAlGgAwIBAgIBBjANBgkqhkiG9w0BAQsFADAxMQ8wDQYDVQQKEwZO.... > . > . > . > . > # search result > search: 2 > result: 0 Success > > # numResponses: 2 > # numEntries: 1 > > > *Replica Server* > > [root at ipa2-server2 ~]# ldapsearch -h ipa-server.nelios -p 7389 -b "o=ipaca" > "(uid=admin)" -x -W > # extended LDIF > # > # LDAPv3 > # base with scope subtree > # filter: (uid=admin) > # requesting: ALL > # > > # admin, people, ipaca > dn: uid=admin,ou=people,o=ipaca > objectClass: top > objectClass: person > objectClass: organizationalPerson > objectClass: inetOrgPerson > objectClass: cmsuser > uid: admin > sn: admin > cn: admin > mail: root at localhost > usertype: adminType > userstate: 1 > > Password is valid in both cases. > > So the user is there and can be retrieved from replica, assuming that > ipa-replica-install also tries 7389 the only thing I can try now is "ipa > cert-request --uid admin" to create a new certificate, generate a new cacert.p12 > and retry install ? > > In the other subthred there was a log from CentOS6 machine: /var/log/pki-ca/system 5337.TP-Processor3 - [09/Sep/2016:22:59:42 EEST] [6] [6] Failed to authenticate as admin UID=admin. Error: netscape.ldap.LDAPException: error result (49) 5337.TP-Processor3 - [09/Sep/2016:22:59:42 EEST] [3] [3] Servlet caGetCookie: Error getting servlet output stream when rendering template. Error Invalid Credential.. Which to me looks like a wrong password. Which indicates my original theory that IPA admin user shared with CA admin user the same password but it got out of sync. During replica installation it uses IPA admin user for authenticating as PKI admin user. If that is correct then changing PKI admin user's ( uid=admin,ou=people,o=ipaca ) password to IPA admin user's password might fix the issue. -- Petr Vobornik From abokovoy at redhat.com Fri Sep 16 17:02:16 2016 From: abokovoy at redhat.com (Alexander Bokovoy) Date: Fri, 16 Sep 2016 20:02:16 +0300 Subject: [Freeipa-users] Samba Server setup In-Reply-To: <38F25AFE-C5E8-4CE9-BD18-26949884625A@bsd.uchicago.edu> References: <73C11E05-045F-4324-9733-44C370584232@bsd.uchicago.edu> <20160915180606.v2z6dkdw3gltem34@redhat.com> <9A3517B6-CFFB-47ED-BDF2-BE3EE0D06EDE@bsd.uchicago.edu> <20160916050435.l4k6rshziftp6pzd@redhat.com> <38F25AFE-C5E8-4CE9-BD18-26949884625A@bsd.uchicago.edu> Message-ID: <20160916170216.jxpico2fscjcswq4@redhat.com> On Fri, 16 Sep 2016, Brook, Andy [CRI] wrote: > You can replace actual hostnames/realm names/IP addresses by something more generic > in the output when sending to the list, but please do it consistently. > >I?m sorry. I thought I had been consistent when making changes, but >from your response, it looks like I wasn?t. I?m sorry about that. I got >yelled at by our security team last time we sent logs to a public list >that had any type of identifiable information in them, so it?s sort of >a new process for me. I think I have it down now. > >The results of the commands are here: http://pastebin.com/PRwr7wv6 So IPA side works fine -- on IPA client you can kinit as AD user and then obtain cross-realm TGT to IPA realm and use that cross-realm TGT to request a service ticket to cifs/... service. That's good. You need to identify what happens on AD side. A possible issue is that name suffix routing to IPA domain is disabled. Can you provide output of netdom.exe run on Windows side: netdom trust addom.domain /namesuffixes: ipa.domain You should get something like example 28 on the page https://msdn.microsoft.com/en-us/library/cc776879(v=ws.10).aspx -- / Alexander Bokovoy From larry.rosen at JDRSolutions.com Fri Sep 16 16:53:12 2016 From: larry.rosen at JDRSolutions.com (Larry Rosen) Date: Fri, 16 Sep 2016 16:53:12 +0000 Subject: [Freeipa-users] login auth fails then success Message-ID: <79B7CEE400C91A4C9FD8BF082D8226073E376FBA@JDRPDC.JDRSolutions.local> We have a web app that logs in using a service (automated login user, non-expiring, non-failure count) account that leaves these log entries all day long. This does not appear to cause any problems, it just make my logs grow unnecessarily and creates a lot of "noise" in the log. Any ideas why it initially fails and then works? Larry -------------- next part -------------- An HTML attachment was scrubbed... URL: From rcritten at redhat.com Fri Sep 16 17:38:59 2016 From: rcritten at redhat.com (Rob Crittenden) Date: Fri, 16 Sep 2016 13:38:59 -0400 Subject: [Freeipa-users] login auth fails then success In-Reply-To: <79B7CEE400C91A4C9FD8BF082D8226073E376FBA@JDRPDC.JDRSolutions.local> References: <79B7CEE400C91A4C9FD8BF082D8226073E376FBA@JDRPDC.JDRSolutions.local> Message-ID: <57DC2E33.6040407@redhat.com> Larry Rosen wrote: > We have a web app that logs in using a service (automated login user, > non-expiring, non-failure count) account that leaves these log entries > all day long. This does not appear to cause any problems, it just make > my logs grow unnecessarily and creates a lot of ?noise? in the log. > > Any ideas why it initially fails and then works?** Logs where? Can we see them? rob From larry.rosen at JDRSolutions.com Fri Sep 16 18:23:03 2016 From: larry.rosen at JDRSolutions.com (Larry Rosen) Date: Fri, 16 Sep 2016 18:23:03 +0000 Subject: [Freeipa-users] login auth fails then success In-Reply-To: <57DC2E33.6040407@redhat.com> References: <79B7CEE400C91A4C9FD8BF082D8226073E376FBA@JDRPDC.JDRSolutions.local> <57DC2E33.6040407@redhat.com> Message-ID: <79B7CEE400C91A4C9FD8BF082D8226073E377113@JDRPDC.JDRSolutions.local> Sorry I thought I had pasted these previously: What other logs do I need to add (maybe from the IPA server)? Client system's /var/log/secure: Sep 13 19:12:33 il10-app-xfs udcs: pam_unix(login:auth): authentication failure; logname= uid=0 euid=0 tty= ruser= rhost= user=il10web Sep 13 19:12:33 il10-app-xfs udcs: pam_sss(login:auth): authentication success; logname= uid=0 euid=0 tty= ruser= rhost= user=il10web Sep 13 19:18:11 il10-app-xfs udcs: pam_unix(login:auth): authentication failure; logname= uid=0 euid=0 tty= ruser= rhost= user=il10web Sep 13 19:18:11 il10-app-xfs udcs: pam_sss(login:auth): authentication success; logname= uid=0 euid=0 tty= ruser= rhost= user=il10web Sep 13 19:22:52 il10-app-xfs udcs: pam_unix(login:auth): authentication failure; logname= uid=0 euid=0 tty= ruser= rhost= user=il10web Sep 13 19:22:53 il10-app-xfs udcs: pam_sss(login:auth): authentication success; logname= uid=0 euid=0 tty= ruser= rhost= user=il10web Sep 13 19:23:49 il10-app-xfs udcs: pam_unix(login:auth): authentication failure; logname= uid=0 euid=0 tty= ruser= rhost= user=il10web Sep 13 19:23:49 il10-app-xfs udcs: pam_sss(login:auth): authentication success; logname= uid=0 euid=0 tty= ruser= rhost= user=il10web Sep 13 19:28:24 il10-app-xfs udcs: pam_unix(login:auth): authentication failure; logname= uid=0 euid=0 tty= ruser= rhost= user=il10web Sep 13 19:28:24 il10-app-xfs udcs: pam_sss(login:auth): authentication success; logname= uid=0 euid=0 tty= ruser= rhost= user=il10web Sep 13 19:29:27 il10-app-xfs udcs: pam_unix(login:auth): authentication failure; logname= uid=0 euid=0 tty= ruser= rhost= user=il10web Sep 13 19:29:27 il10-app-xfs udcs: pam_sss(login:auth): authentication success; logname= uid=0 euid=0 tty= ruser= rhost= user=il10web -----Original Message----- From: Rob Crittenden [mailto:rcritten at redhat.com] Sent: Friday, September 16, 2016 1:39 PM To: Larry Rosen ; freeipa-users at redhat.com Subject: Re: [Freeipa-users] login auth fails then success Larry Rosen wrote: > We have a web app that logs in using a service (automated login user, > non-expiring, non-failure count) account that leaves these log entries > all day long. This does not appear to cause any problems, it just make > my logs grow unnecessarily and creates a lot of "noise" in the log. > > Any ideas why it initially fails and then works?** Logs where? Can we see them? rob From APtashnik at cccis.com Fri Sep 16 19:16:05 2016 From: APtashnik at cccis.com (Andrey Ptashnik) Date: Fri, 16 Sep 2016 19:16:05 +0000 Subject: [Freeipa-users] IPA server as a domain controller for more than one domain Message-ID: Hi IPA team, Can I use the same FreeIPA server to be a domain controller for more than one domain? Regards, Andrey Ptashnik From abokovoy at redhat.com Fri Sep 16 19:56:33 2016 From: abokovoy at redhat.com (Alexander Bokovoy) Date: Fri, 16 Sep 2016 22:56:33 +0300 Subject: [Freeipa-users] IPA server as a domain controller for more than one domain In-Reply-To: References: Message-ID: <20160916195633.kgl4qpbcat52sybr@redhat.com> On Fri, 16 Sep 2016, Andrey Ptashnik wrote: >Hi IPA team, > >Can I use the same FreeIPA server to be a domain controller for more than one domain? Define 'domain', please. The term is so overloaded that it is hard to understand what exactly you need to control. -- / Alexander Bokovoy From detlev.habicht at ims.uni-hannover.de Sat Sep 17 06:20:35 2016 From: detlev.habicht at ims.uni-hannover.de (Detlev Habicht) Date: Sat, 17 Sep 2016 08:20:35 +0200 Subject: [Freeipa-users] Problems with mount and user logins Message-ID: Hi all, i am setting up IPA the first time for real life and have now some big problems. First for testing i setup an IPA-Server, a NFS server and up to 3 clients. The server are running Scientic Linux and the clients Federo 24 (setup via Cobbler server). The setup is based on the Red Hat Linux 7 IPA Docs. This was running well over several weeks. I can reinstall my clients via cobbler and everything was good.But mostly with one user (me). Now i was adding 20 hosts and the first time big problems are coming. First, one problem has nothing to do with IPA: I found bug reports about new autofs problems with Fedora 24. autofs is starting at the wrong time and sssd too. Second, my problem has to do with mounting or accessing directories: When i login in a host, sometimes mounting directories is not allowed and sometimes it is possible. And there is no system, sometimes it works, sometimes it works not. Well, important: i am login at several hosts at the same time! Mostly with one user! Me! I install the clients with ipa-client-install and ipa-client-automount (first time by hand, for reinstall with kickstart). I do not something with the user .-files like .cshrc or change any other dot-file. I read now at some MIT-Docs, it is a good idea to add kdestroy in dot-files for logout. I can?t expect big help with my poor report here, but i have now two questions: Have i to setup the users dot-files in some way? Is there any good documentation for a users home directory setup? Is login in several clients at the same time with the same shared home directory a problem? Thank you for any help! Detlev -- Detlev | Institut fuer Mikroelektronische Systeme Habicht | D-30167 Hannover +49 511 76219662 habicht at ims.uni-hannover.de --------+-------- Handy +49 172 5415752 --------------------------- -------------- next part -------------- An HTML attachment was scrubbed... URL: From lslebodn at redhat.com Sat Sep 17 08:25:46 2016 From: lslebodn at redhat.com (Lukas Slebodnik) Date: Sat, 17 Sep 2016 10:25:46 +0200 Subject: [Freeipa-users] Problems with mount and user logins In-Reply-To: References: Message-ID: <20160917082545.GA22593@10.4.128.1> On (17/09/16 08:20), Detlev Habicht wrote: >Hi all, > >i am setting up IPA the first time for real life and have now >some big problems. > >First for testing i setup an IPA-Server, a NFS server and up >to 3 clients. The server are running Scientic Linux and the clients >Federo 24 (setup via Cobbler server). The setup is based on the Red Hat Linux 7 IPA Docs. > >This was running well over several weeks. I can reinstall my clients >via cobbler and everything was good.But mostly with one user (me). > >Now i was adding 20 hosts and the first time big problems >are coming. > >First, one problem has nothing to do with IPA: I found bug reports >about new autofs problems with Fedora 24. autofs is starting >at the wrong time and sssd too. > There are some patches pending review which should help in this case https://lists.fedorahosted.org/archives/list/sssd-devel at lists.fedorahosted.org/message/UKWFVD7NBYTLW6VW5W6TBSTCWOMP5C2V/ I plan to backport them to fedora immediately after accepting in upstream. LS From peljasz at yahoo.co.uk Sat Sep 17 09:23:04 2016 From: peljasz at yahoo.co.uk (lejeczek) Date: Sat, 17 Sep 2016 10:23:04 +0100 Subject: [Freeipa-users] how to revert ipa-adtrust-install... In-Reply-To: <4ae11a44-cf3a-39ce-0732-ba2ca2186a74@redhat.com> References: <1ec82d0c-1350-feb3-4da5-a022eb7ac5e5@yahoo.co.uk> <20160915191709.6vk7y3gidzxoicen@redhat.com> <98df2894-d5c7-082c-8fe8-77ece9474301@yahoo.co.uk> <20160915200055.qpx7gma722tjcfxk@redhat.com> <57DB1487.4000500@redhat.com> <20160916045704.5emkdqda3uvqdzlc@redhat.com> <4ae11a44-cf3a-39ce-0732-ba2ca2186a74@redhat.com> Message-ID: gee... I'd like to think that I'm not the only one who at some point, for whatever reason there might decide to have Samba freed from IPA. For me it's because I want my Samba to be a slave to AD and inter-domain trusts are impossible to use for administrative, bureaucratic reasons, but I absolutely need IPA (for it's great (piece of work)) (both software on the same box). On 16/09/16 16:05, Petr Vobornik wrote: > Out of curiosity: is there a partial broader use case behind this > feature request? From peljasz at yahoo.co.uk Sat Sep 17 10:43:46 2016 From: peljasz at yahoo.co.uk (lejeczek) Date: Sat, 17 Sep 2016 11:43:46 +0100 Subject: [Freeipa-users] how to revert ipa-adtrust-install... In-Reply-To: <57DB1487.4000500@redhat.com> References: <1ec82d0c-1350-feb3-4da5-a022eb7ac5e5@yahoo.co.uk> <20160915191709.6vk7y3gidzxoicen@redhat.com> <98df2894-d5c7-082c-8fe8-77ece9474301@yahoo.co.uk> <20160915200055.qpx7gma722tjcfxk@redhat.com> <57DB1487.4000500@redhat.com> Message-ID: On 15/09/16 22:37, Rob Crittenden wrote: > What do you mean control? If you don't want ipactl to > manage the smb service, look for an entry in > cn=masters,cn=ipa,cn=etc,dc=example,dc=com and delete it > if you find it. > > rob all I find there is: objectClass: nsContainer objectClass: top cn: masters From peljasz at yahoo.co.uk Sat Sep 17 11:02:34 2016 From: peljasz at yahoo.co.uk (lejeczek) Date: Sat, 17 Sep 2016 12:02:34 +0100 Subject: [Freeipa-users] SELinux is preventing /usr/sbin/krb5kdc from write access on the sock_file /var/lib/sss/pipes/pac. Message-ID: <13db3059-5a1d-6585-85fa-15f9a7ed1a1e@yahoo.co.uk> before I drop above onto SELinux team - do you guys think SE should be doing that? Does it impair IPA in some ways? many thanks L. From lslebodn at redhat.com Sat Sep 17 11:59:44 2016 From: lslebodn at redhat.com (Lukas Slebodnik) Date: Sat, 17 Sep 2016 13:59:44 +0200 Subject: [Freeipa-users] SELinux is preventing /usr/sbin/krb5kdc from write access on the sock_file /var/lib/sss/pipes/pac. In-Reply-To: <13db3059-5a1d-6585-85fa-15f9a7ed1a1e@yahoo.co.uk> References: <13db3059-5a1d-6585-85fa-15f9a7ed1a1e@yahoo.co.uk> Message-ID: <20160917115943.GA24223@10.4.128.1> On (17/09/16 12:02), lejeczek wrote: >before I drop above onto SELinux team - do you guys think SE should be doing >that? Does it impair IPA in some ways? > It would be god to see more details. Do you know which action trigger AVCs? Could you also provide detail about AVC? ausearch -m avc -i ts recent LS From peljasz at yahoo.co.uk Sat Sep 17 12:18:25 2016 From: peljasz at yahoo.co.uk (lejeczek) Date: Sat, 17 Sep 2016 13:18:25 +0100 Subject: [Freeipa-users] SELinux is preventing /usr/sbin/krb5kdc from write access on the sock_file /var/lib/sss/pipes/pac. In-Reply-To: <20160917115943.GA24223@10.4.128.1> References: <13db3059-5a1d-6585-85fa-15f9a7ed1a1e@yahoo.co.uk> <20160917115943.GA24223@10.4.128.1> Message-ID: I think one case it when I sudo not much there really, building up an semodule out of the alerts would end up in: allow krb5kdc_t sssd_var_lib_t:sock_file write; On 17/09/16 12:59, Lukas Slebodnik wrote: > On (17/09/16 12:02), lejeczek wrote: >> before I drop above onto SELinux team - do you guys think SE should be doing >> that? Does it impair IPA in some ways? >> > It would be god to see more details. Do you know which action trigger > AVCs? > > Could you also provide detail about AVC? > ausearch -m avc -i ts recent > > LS From jhrozek at redhat.com Sun Sep 18 20:14:59 2016 From: jhrozek at redhat.com (Jakub Hrozek) Date: Sun, 18 Sep 2016 22:14:59 +0200 Subject: [Freeipa-users] login auth fails then success In-Reply-To: <79B7CEE400C91A4C9FD8BF082D8226073E377113@JDRPDC.JDRSolutions.local> References: <79B7CEE400C91A4C9FD8BF082D8226073E376FBA@JDRPDC.JDRSolutions.local> <57DC2E33.6040407@redhat.com> <79B7CEE400C91A4C9FD8BF082D8226073E377113@JDRPDC.JDRSolutions.local> Message-ID: <20160918201459.uhijnc4gyfykgzic@hendrix> On Fri, Sep 16, 2016 at 06:23:03PM +0000, Larry Rosen wrote: > Sorry I thought I had pasted these previously: > > What other logs do I need to add (maybe from the IPA server)? > > Client system's /var/log/secure: > > Sep 13 19:12:33 il10-app-xfs udcs: pam_unix(login:auth): authentication failure; logname= uid=0 euid=0 tty= ruser= rhost= user=il10web > Sep 13 19:12:33 il10-app-xfs udcs: pam_sss(login:auth): authentication success; logname= uid=0 euid=0 tty= ruser= rhost= user=il10web > Sep 13 19:18:11 il10-app-xfs udcs: pam_unix(login:auth): authentication failure; logname= uid=0 euid=0 tty= ruser= rhost= user=il10web > Sep 13 19:18:11 il10-app-xfs udcs: pam_sss(login:auth): authentication success; logname= uid=0 euid=0 tty= ruser= rhost= user=il10web > Sep 13 19:22:52 il10-app-xfs udcs: pam_unix(login:auth): authentication failure; logname= uid=0 euid=0 tty= ruser= rhost= user=il10web > Sep 13 19:22:53 il10-app-xfs udcs: pam_sss(login:auth): authentication success; logname= uid=0 euid=0 tty= ruser= rhost= user=il10web > Sep 13 19:23:49 il10-app-xfs udcs: pam_unix(login:auth): authentication failure; logname= uid=0 euid=0 tty= ruser= rhost= user=il10web > Sep 13 19:23:49 il10-app-xfs udcs: pam_sss(login:auth): authentication success; logname= uid=0 euid=0 tty= ruser= rhost= user=il10web > Sep 13 19:28:24 il10-app-xfs udcs: pam_unix(login:auth): authentication failure; logname= uid=0 euid=0 tty= ruser= rhost= user=il10web > Sep 13 19:28:24 il10-app-xfs udcs: pam_sss(login:auth): authentication success; logname= uid=0 euid=0 tty= ruser= rhost= user=il10web > Sep 13 19:29:27 il10-app-xfs udcs: pam_unix(login:auth): authentication failure; logname= uid=0 euid=0 tty= ruser= rhost= user=il10web > Sep 13 19:29:27 il10-app-xfs udcs: pam_sss(login:auth): authentication success; logname= uid=0 euid=0 tty= ruser= rhost= user=il10web I think these are expected. Authentication using pam_unix fails because pam_unix doesn't know this particular users and then pam_sss succeeds. I wonder if the best way to deal with the log messages is just to configure logrotate a bit more aggressively? > > -----Original Message----- > From: Rob Crittenden [mailto:rcritten at redhat.com] > Sent: Friday, September 16, 2016 1:39 PM > To: Larry Rosen ; freeipa-users at redhat.com > Subject: Re: [Freeipa-users] login auth fails then success > > Larry Rosen wrote: > > We have a web app that logs in using a service (automated login user, > > non-expiring, non-failure count) account that leaves these log entries > > all day long. This does not appear to cause any problems, it just make > > my logs grow unnecessarily and creates a lot of "noise" in the log. > > > > Any ideas why it initially fails and then works?** > > Logs where? Can we see them? > > rob > > > -- > Manage your subscription for the Freeipa-users mailing list: > https://www.redhat.com/mailman/listinfo/freeipa-users > Go to http://freeipa.org for more info on the project From datakid at gmail.com Mon Sep 19 01:12:27 2016 From: datakid at gmail.com (Lachlan Musicman) Date: Mon, 19 Sep 2016 11:12:27 +1000 Subject: [Freeipa-users] In webgui, ID Views slow, to crashingly slow Message-ID: Hi Sometimes when I visit the ID Views page in the webgui, it is crushingly slow, and often it times out. Centos 7, ipa --version VERSION: 4.2.0, API_VERSION: 2.156 Is there a reason, can I do something to fix this? cheers L. ------ The most dangerous phrase in the language is, "We've always done it this way." - Grace Hopper -------------- next part -------------- An HTML attachment was scrubbed... URL: From datakid at gmail.com Mon Sep 19 06:43:05 2016 From: datakid at gmail.com (Lachlan Musicman) Date: Mon, 19 Sep 2016 16:43:05 +1000 Subject: [Freeipa-users] HBAC doesn't work issues Message-ID: I must have made an error again: - ipa hbactest gives seemingly correct answer on both server and client - user can't actually use sudo on client? Centos 7, freeipa 4.2.o/2.156; sssd 1.14.1 from COPR >From the server: [root at vmdv-linuxidm1 ~]# ipa hbactest --user=lsimpson at petermac.org.au --host=vmts-linuxclient1.unixdev.petermac.org.au --service=sudo -------------------- Access granted: True -------------------- Matched rules: Cluster Admin Users (sudo) Not matched rules: Cluster Users [root at vmdv-linuxidm1 ~]# >From the host in question: [root at vmts-linuxclient1 ~]# ipa hbactest --user lsimpson at petermac.org.au --host `hostname` --service sudo -------------------- Access granted: True -------------------- Matched rules: Cluster Admin Users (sudo) Not matched rules: Cluster Users [root at vmts-linuxclient1 ~]# [lsimpson at petermac.org.au@vmts-linuxclient1 ~]$ sudo reboot [sudo] password for lsimpson at petermac.org.au: lsimpson at petermac.org.au is not allowed to run sudo on vmts-linuxclient1. This incident will be reported. On the client, in the sssd_sudo.log I can see (debug_level=6) a number of lines, most notably three that start "Searching sysdb with" and then follow with all my ipa and AD groups - both groups that would give me HBAC sudo are listed in those log entries. What should I try next? cheers L. ------ The most dangerous phrase in the language is, "We've always done it this way." - Grace Hopper -------------- next part -------------- An HTML attachment was scrubbed... URL: From natxo.asenjo at gmail.com Mon Sep 19 07:05:57 2016 From: natxo.asenjo at gmail.com (Natxo Asenjo) Date: Mon, 19 Sep 2016 09:05:57 +0200 Subject: [Freeipa-users] certificates not renewing CA_UNREACHEABLE In-Reply-To: <57DC0009.4030602@redhat.com> References: <806bdf32-558c-4972-255a-0901f47c4c4b@redhat.com> <6596593a-9470-bebd-f236-f9d17aa4a56c@redhat.com> <57DC0009.4030602@redhat.com> Message-ID: hi, On Fri, Sep 16, 2016 at 4:22 PM, Rob Crittenden wrote: > The 3 certs you list are the ones that are renewed via the IPA API (as > opposed to the subsystem certs renewed directly by dogtag). I think the > failures are all related. I had someone else report the CSR decoding > failure and he just restarted IPA and that fixes things for him though it > was a rather unsatisfying fix. > > What I'd do is this. Assuming each step works, move onto the next. > > 1. ipa cert-show 1 > > The serial # picked more or less at random, we're testing connectivity and > that the CA is up and operational. > > 2. I assume that getcert list | grep expire shows all certs currently > valid? The IPA service certs expire in a month, how about the CA subsystem > certs? > > 3. Is this the same server having problems talking to the CA due to the > other NSS errors? If so what I'd do is restart httpd then immediately use > ipa-getcert to resubmit the requests to try to get into that few minute > window. > > If this is the same box you already have debugging enabled so seeing what > that shows might be helpful. > > rob > yes, all certs are valid (see attachment getcert.txt). So I restarted httpd, I could execute ipa cert-show 1 and get an answer, inmediately after I run $ sudo ipa-getcert resubmit -i 20121107212513 Resubmitting "20121107212513" to "IPA". and now the status is the one you see in the attached getcert.txt file. The server failed request, will retry. I do not know if it's important, but I saw that the usercertificate attribute of the pki user admin was expired.1 I attach the error_log of httpd as well. -- -- Groeten, natxo -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- Number of certificates and requests being tracked: 8. Request ID '20121107212513': status: CA_UNREACHABLE ca-error: Server failed request, will retry: 907 (RPC failed at server. cannot connect to 'https://kdc01.unix.iriszorg.nl:443/ca/agent/ca/doRevoke': (SEC_ERROR_BUSY) NSS could not shutdown. Objects are still in use.). stuck: yes key pair storage: type=NSSDB,location='/etc/dirsrv/slapd-UNIX-IRISZORG-NL',nickname='Server-Cert',token='NSS Certificate DB',pinfile='/etc/dirsrv/slapd-UNIX-IRISZORG-NL/pwdfile.txt' certificate: type=NSSDB,location='/etc/dirsrv/slapd-UNIX-IRISZORG-NL',nickname='Server-Cert',token='NSS Certificate DB' CA: IPA issuer: CN=Certificate Authority,O=UNIX.IRISZORG.NL subject: CN=kdc01.unix.iriszorg.nl,O=UNIX.IRISZORG.NL expires: 2016-10-12 10:49:24 UTC eku: id-kp-serverAuth,id-kp-clientAuth pre-save command: post-save command: /usr/lib/ipa/certmonger/restart_dirsrv UNIX-IRISZORG-NL track: yes auto-renew: yes Request ID '20121107212532': status: CA_UNREACHABLE ca-error: Server failed request, will retry: 907 (RPC failed at server. cannot connect to 'https://kdc01.unix.iriszorg.nl:443/ca/agent/ca/doRevoke': (SEC_ERROR_BUSY) NSS could not shutdown. Objects are still in use.). stuck: yes key pair storage: type=NSSDB,location='/etc/dirsrv/slapd-PKI-IPA',nickname='Server-Cert',token='NSS Certificate DB',pinfile='/etc/dirsrv/slapd-PKI-IPA/pwdfile.txt' certificate: type=NSSDB,location='/etc/dirsrv/slapd-PKI-IPA',nickname='Server-Cert',token='NSS Certificate DB' CA: IPA issuer: CN=Certificate Authority,O=UNIX.IRISZORG.NL subject: CN=kdc01.unix.iriszorg.nl,O=UNIX.IRISZORG.NL expires: 2016-10-12 10:49:25 UTC eku: id-kp-serverAuth,id-kp-clientAuth pre-save command: post-save command: track: yes auto-renew: yes Request ID '20121107212548': status: CA_UNREACHABLE ca-error: Server failed request, will retry: 907 (RPC failed at server. cannot connect to 'https://kdc01.unix.iriszorg.nl:443/ca/agent/ca/doRevoke': (SEC_ERROR_BUSY) NSS could not shutdown. Objects are still in use.). stuck: yes key pair storage: type=NSSDB,location='/etc/httpd/alias',nickname='Server-Cert',token='NSS Certificate DB',pinfile='/etc/httpd/alias/pwdfile.txt' certificate: type=NSSDB,location='/etc/httpd/alias',nickname='Server-Cert',token='NSS Certificate DB' CA: IPA issuer: CN=Certificate Authority,O=UNIX.IRISZORG.NL subject: CN=kdc01.unix.iriszorg.nl,O=UNIX.IRISZORG.NL expires: 2016-10-12 10:49:24 UTC eku: id-kp-serverAuth,id-kp-clientAuth pre-save command: post-save command: /usr/lib/ipa/certmonger/restart_httpd track: yes auto-renew: yes Request ID '20130312123435': status: MONITORING stuck: no key pair storage: type=NSSDB,location='/var/lib/pki-ca/alias',nickname='auditSigningCert cert-pki-ca',token='NSS Certificate DB',pin='289224738763' certificate: type=NSSDB,location='/var/lib/pki-ca/alias',nickname='auditSigningCert cert-pki-ca',token='NSS Certificate DB' CA: dogtag-ipa-renew-agent issuer: CN=Certificate Authority,O=UNIX.IRISZORG.NL subject: CN=CA Audit,O=UNIX.IRISZORG.NL expires: 2018-09-03 12:24:14 UTC pre-save command: /usr/lib/ipa/certmonger/stop_pkicad post-save command: /usr/lib/ipa/certmonger/renew_ca_cert "auditSigningCert cert-pki-ca" track: yes auto-renew: yes Request ID '20130312123436': status: MONITORING stuck: no key pair storage: type=NSSDB,location='/var/lib/pki-ca/alias',nickname='ocspSigningCert cert-pki-ca',token='NSS Certificate DB',pin='289224738763' certificate: type=NSSDB,location='/var/lib/pki-ca/alias',nickname='ocspSigningCert cert-pki-ca',token='NSS Certificate DB' CA: dogtag-ipa-renew-agent issuer: CN=Certificate Authority,O=UNIX.IRISZORG.NL subject: CN=OCSP Subsystem,O=UNIX.IRISZORG.NL expires: 2018-09-03 12:23:14 UTC eku: id-kp-OCSPSigning pre-save command: /usr/lib/ipa/certmonger/stop_pkicad post-save command: /usr/lib/ipa/certmonger/renew_ca_cert "ocspSigningCert cert-pki-ca" track: yes auto-renew: yes Request ID '20130312123437': status: MONITORING stuck: no key pair storage: type=NSSDB,location='/var/lib/pki-ca/alias',nickname='subsystemCert cert-pki-ca',token='NSS Certificate DB',pin='289224738763' certificate: type=NSSDB,location='/var/lib/pki-ca/alias',nickname='subsystemCert cert-pki-ca',token='NSS Certificate DB' CA: dogtag-ipa-renew-agent issuer: CN=Certificate Authority,O=UNIX.IRISZORG.NL subject: CN=CA Subsystem,O=UNIX.IRISZORG.NL expires: 2018-09-03 12:23:14 UTC eku: id-kp-serverAuth,id-kp-clientAuth pre-save command: /usr/lib/ipa/certmonger/stop_pkicad post-save command: /usr/lib/ipa/certmonger/renew_ca_cert "subsystemCert cert-pki-ca" track: yes auto-renew: yes Request ID '20130312123438': status: MONITORING stuck: no key pair storage: type=NSSDB,location='/etc/httpd/alias',nickname='ipaCert',token='NSS Certificate DB',pinfile='/etc/httpd/alias/pwdfile.txt' certificate: type=NSSDB,location='/etc/httpd/alias',nickname='ipaCert',token='NSS Certificate DB' CA: dogtag-ipa-renew-agent issuer: CN=Certificate Authority,O=UNIX.IRISZORG.NL subject: CN=IPA RA,O=UNIX.IRISZORG.NL expires: 2018-09-03 12:23:14 UTC eku: id-kp-serverAuth,id-kp-clientAuth pre-save command: post-save command: /usr/lib/ipa/certmonger/renew_ra_cert track: yes auto-renew: yes Request ID '20130312123439': status: MONITORING stuck: no key pair storage: type=NSSDB,location='/var/lib/pki-ca/alias',nickname='Server-Cert cert-pki-ca',token='NSS Certificate DB',pin='289224738763' certificate: type=NSSDB,location='/var/lib/pki-ca/alias',nickname='Server-Cert cert-pki-ca',token='NSS Certificate DB' CA: dogtag-ipa-renew-agent issuer: CN=Certificate Authority,O=UNIX.IRISZORG.NL subject: CN=kdc01.unix.iriszorg.nl,O=UNIX.IRISZORG.NL expires: 2018-09-03 12:23:14 UTC eku: id-kp-serverAuth,id-kp-clientAuth pre-save command: post-save command: track: yes auto-renew: yes -------------- next part -------------- A non-text attachment was scrubbed... Name: error_log Type: application/octet-stream Size: 2400723 bytes Desc: not available URL: From th at casalogic.dk Mon Sep 19 07:07:41 2016 From: th at casalogic.dk (Troels Hansen) Date: Mon, 19 Sep 2016 09:07:41 +0200 (CEST) Subject: [Freeipa-users] ipa trust-add using password In-Reply-To: <20160916085012.joed3d6renupfush@redhat.com> References: <210728865.288453.1474015352468.JavaMail.zimbra@casalogic.dk> <20160916085012.joed3d6renupfush@redhat.com> Message-ID: <476548999.299147.1474268861554.JavaMail.zimbra@casalogic.dk> > If you add 'log level = 50' to /usr/share/ipa/smb.conf.empty, then > /var/log/httpd/error_log will contain detailed debug information from > IPA attempts to talk to AD DCs. > > -- > / Alexander Bokovoy Hi Alexander I added the log level, and had the domain admin try to create the trust, and today it just worked, soo... not any further on finding out what went wrong last week, but the trust got created so not going to spent more time on this. Anyway, thanks for the help. I have made a mental note on debugging IPA-AD trust creation. From mbabinsk at redhat.com Mon Sep 19 07:49:32 2016 From: mbabinsk at redhat.com (Martin Babinsky) Date: Mon, 19 Sep 2016 09:49:32 +0200 Subject: [Freeipa-users] how to revert ipa-adtrust-install... In-Reply-To: References: <1ec82d0c-1350-feb3-4da5-a022eb7ac5e5@yahoo.co.uk> <20160915191709.6vk7y3gidzxoicen@redhat.com> <98df2894-d5c7-082c-8fe8-77ece9474301@yahoo.co.uk> <20160915200055.qpx7gma722tjcfxk@redhat.com> <57DB1487.4000500@redhat.com> Message-ID: <7d742171-ce87-aff0-5aef-903f109abc94@redhat.com> On 09/17/2016 12:43 PM, lejeczek wrote: > > > On 15/09/16 22:37, Rob Crittenden wrote: >> What do you mean control? If you don't want ipactl to manage the smb >> service, look for an entry in >> cn=masters,cn=ipa,cn=etc,dc=example,dc=com and delete it if you find it. >> >> rob > all I find there is: > > objectClass: nsContainer > objectClass: top > cn: masters > You must perform subtree search and search for the entry named 'cn=ADTRUST', like so: """ ldapsearch -Y GSSAPI -b 'cn=masters,cn=ipa,cn=etc,dc=ipa,dc=test' '(cn=ADTRUST)' SASL/GSSAPI authentication started SASL username: admin at IPA.TEST SASL SSF: 56 SASL data security layer installed. # extended LDIF # # LDAPv3 # base with scope subtree # filter: (cn=ADTRUST) # requesting: ALL # # ADTRUST, master1.ipa.test, masters, ipa, etc, ipa.test dn: cn=ADTRUST,cn=master1.ipa.test,cn=masters,cn=ipa,cn=etc,dc=ipa,dc=test objectClass: ipaConfigObject objectClass: nsContainer objectClass: top ipaConfigString: startOrder 60 ipaConfigString: enabledService cn: ADTRUST # search result search: 4 result: 0 Success # numResponses: 2 # numEntries: 1 """ Then remove the "ipaConfigString: enabledService" attribute from the entry to tell "ipactl" that it should not control this service anymore: [root at master1 ~]# ldapmodify -Y GSSAPI SASL/GSSAPI authentication started SASL username: admin at IPA.TEST SASL SSF: 56 SASL data security layer installed. dn: cn=ADTRUST,cn=master1.ipa.test,cn=masters,cn=ipa,cn=etc,dc=ipa,dc=test changetype: modify delete: ipaConfigString ipaConfigString: enabledService modifying entry "cn=ADTRUST,cn=master1.ipa.test,cn=masters,cn=ipa,cn=etc,dc=ipa,dc=test" If you then do "ipactl restart" and "ipactl status", it should not display smb.service anymore and you are free to use them as you wish. -- Martin^3 Babinsky From mbabinsk at redhat.com Mon Sep 19 07:54:38 2016 From: mbabinsk at redhat.com (Martin Babinsky) Date: Mon, 19 Sep 2016 09:54:38 +0200 Subject: [Freeipa-users] how to revert ipa-adtrust-install... In-Reply-To: <7d742171-ce87-aff0-5aef-903f109abc94@redhat.com> References: <1ec82d0c-1350-feb3-4da5-a022eb7ac5e5@yahoo.co.uk> <20160915191709.6vk7y3gidzxoicen@redhat.com> <98df2894-d5c7-082c-8fe8-77ece9474301@yahoo.co.uk> <20160915200055.qpx7gma722tjcfxk@redhat.com> <57DB1487.4000500@redhat.com> <7d742171-ce87-aff0-5aef-903f109abc94@redhat.com> Message-ID: <9a426178-f16b-8282-0f84-3d026567ed30@redhat.com> On 09/19/2016 09:49 AM, Martin Babinsky wrote: > On 09/17/2016 12:43 PM, lejeczek wrote: >> >> >> On 15/09/16 22:37, Rob Crittenden wrote: >>> What do you mean control? If you don't want ipactl to manage the smb >>> service, look for an entry in >>> cn=masters,cn=ipa,cn=etc,dc=example,dc=com and delete it if you find it. >>> >>> rob >> all I find there is: >> >> objectClass: nsContainer >> objectClass: top >> cn: masters >> > > You must perform subtree search and search for the entry named > 'cn=ADTRUST', like so: > > """ > ldapsearch -Y GSSAPI -b 'cn=masters,cn=ipa,cn=etc,dc=ipa,dc=test' > '(cn=ADTRUST)' > SASL/GSSAPI authentication started > SASL username: admin at IPA.TEST > SASL SSF: 56 > SASL data security layer installed. > # extended LDIF > # > # LDAPv3 > # base with scope subtree > # filter: (cn=ADTRUST) > # requesting: ALL > # > > # ADTRUST, master1.ipa.test, masters, ipa, etc, ipa.test > dn: cn=ADTRUST,cn=master1.ipa.test,cn=masters,cn=ipa,cn=etc,dc=ipa,dc=test > objectClass: ipaConfigObject > objectClass: nsContainer > objectClass: top > ipaConfigString: startOrder 60 > ipaConfigString: enabledService > cn: ADTRUST > > # search result > search: 4 > result: 0 Success > > # numResponses: 2 > # numEntries: 1 > """ > > Then remove the "ipaConfigString: enabledService" attribute from the > entry to tell "ipactl" that it should not control this service anymore: > > [root at master1 ~]# ldapmodify -Y GSSAPI > SASL/GSSAPI authentication started > SASL username: admin at IPA.TEST > SASL SSF: 56 > SASL data security layer installed. > dn: cn=ADTRUST,cn=master1.ipa.test,cn=masters,cn=ipa,cn=etc,dc=ipa,dc=test > changetype: modify > delete: ipaConfigString > ipaConfigString: enabledService > > modifying entry > "cn=ADTRUST,cn=master1.ipa.test,cn=masters,cn=ipa,cn=etc,dc=ipa,dc=test" > > If you then do "ipactl restart" and "ipactl status", it should not > display smb.service anymore and you are free to use them as you wish. > Note that you might do the same for the 'cn=EXTID' entry which controls the winbind service. -- Martin^3 Babinsky From mbabinsk at redhat.com Mon Sep 19 08:09:22 2016 From: mbabinsk at redhat.com (Martin Babinsky) Date: Mon, 19 Sep 2016 10:09:22 +0200 Subject: [Freeipa-users] In webgui, ID Views slow, to crashingly slow In-Reply-To: References: Message-ID: <90ec609b-f939-a04c-cab9-d34ce955d81d@redhat.com> On 09/19/2016 03:12 AM, Lachlan Musicman wrote: > Hi > > Sometimes when I visit the ID Views page in the webgui, it is crushingly > slow, and often it times out. > > Centos 7, ipa --version > VERSION: 4.2.0, API_VERSION: 2.156 > > Is there a reason, can I do something to fix this? > > cheers > L. > ------ > The most dangerous phrase in the language is, "We've always done it this > way." > > - Grace Hopper > > What kind of ID Views do you use? Do you use them to override AD users? Is there any useful info in '/var/log/httpd/error_log'? -- Martin^3 Babinsky From lslebodn at redhat.com Mon Sep 19 08:21:17 2016 From: lslebodn at redhat.com (Lukas Slebodnik) Date: Mon, 19 Sep 2016 10:21:17 +0200 Subject: [Freeipa-users] HBAC doesn't work issues In-Reply-To: References: Message-ID: <20160919082116.GF4644@10.4.128.1> On (19/09/16 16:43), Lachlan Musicman wrote: >I must have made an error again: > >- ipa hbactest gives seemingly correct answer on both server and client >- user can't actually use sudo on client? > >Centos 7, freeipa 4.2.o/2.156; sssd 1.14.1 from COPR > >>From the server: > >[root at vmdv-linuxidm1 ~]# ipa hbactest --user=lsimpson at petermac.org.au >--host=vmts-linuxclient1.unixdev.petermac.org.au --service=sudo >-------------------- >Access granted: True >-------------------- > Matched rules: Cluster Admin Users (sudo) > Not matched rules: Cluster Users >[root at vmdv-linuxidm1 ~]# > > >>From the host in question: > >[root at vmts-linuxclient1 ~]# ipa hbactest --user lsimpson at petermac.org.au >--host `hostname` --service sudo >-------------------- >Access granted: True >-------------------- > Matched rules: Cluster Admin Users (sudo) > Not matched rules: Cluster Users >[root at vmts-linuxclient1 ~]# > > >[lsimpson at petermac.org.au@vmts-linuxclient1 ~]$ sudo reboot >[sudo] password for lsimpson at petermac.org.au: >lsimpson at petermac.org.au is not allowed to run sudo on vmts-linuxclient1. >This incident will be reported. > Did you configure sudo rules for such user? What is an output of "sudo -l" LS From peljasz at yahoo.co.uk Mon Sep 19 09:01:42 2016 From: peljasz at yahoo.co.uk (lejeczek) Date: Mon, 19 Sep 2016 10:01:42 +0100 Subject: [Freeipa-users] how to revert ipa-adtrust-install... In-Reply-To: <7d742171-ce87-aff0-5aef-903f109abc94@redhat.com> References: <1ec82d0c-1350-feb3-4da5-a022eb7ac5e5@yahoo.co.uk> <20160915191709.6vk7y3gidzxoicen@redhat.com> <98df2894-d5c7-082c-8fe8-77ece9474301@yahoo.co.uk> <20160915200055.qpx7gma722tjcfxk@redhat.com> <57DB1487.4000500@redhat.com> <7d742171-ce87-aff0-5aef-903f109abc94@redhat.com> Message-ID: <43163c2c-95e7-32ba-f927-ddb6140a7b1f@yahoo.co.uk> thanks a lot! On 19/09/16 08:49, Martin Babinsky wrote: > ipaConfigString: enabledService From flo at redhat.com Mon Sep 19 10:02:19 2016 From: flo at redhat.com (Florence Blanc-Renaud) Date: Mon, 19 Sep 2016 12:02:19 +0200 Subject: [Freeipa-users] 3rd party Cert install now IPA total broken In-Reply-To: <1825019.y8rA6b7sXH@techz> References: <1825019.y8rA6b7sXH@techz> Message-ID: On 09/16/2016 03:06 PM, G?nther J. Niederwimmer wrote: > Hello, > Freeipa 4.3.1 > > I have now install a 3rd Party Certificat from Startcom now my IPA is total > broken? > I make this > > ipa-cacert-manage -p 'xxxxxxxxxxxxxxxx' -n STARTCOM-ROOT -t C,, install > root.crt > > ipa-certupdate > > ipa-server-certinstall -w -d ipa_3rd_ca.p12 > > I create this p12 with key.pem, cert.pem root.crt > > I insert also in the cert.pem the intermediate.crt > Hi, there were some issues with ipa-server-certinstall (see tickets #4785, #4786 and #6263). In order to check your configuration, you must make sure that the NSS DBs for Apache and the LDAP server (/etc/httpd/alias, /var/lib/pki/pki-tomcat/alias, /etc/dirsrv/slapd-DOMxx) contain: - the server certificate with flags u,u,u (= the one contained in ipa_3rd_ca.p12) - the certificate of the CA which signed the server certificate, with flags C,, (= the one contained in root.rt) Then you can also check if the nickname for the server cert is properly set in /etc/httpd/conf.d/nss.conf (in the directive NSSNickname), and in the LDAP entry cn=RSA,cn=encryption,cn=config (in the attribute nsSSLPersonalitySSL). If this doesn't fix the issue, the logs of pki-tomcat/ca/debug may provide more information. Also note that it is important to run ipa-certupdate on all the clients and replicas in order to install the new certificates in the NSS DBs *before* you run ipa-server-certinstall. Hope this helps, Flo. > the kerberos don't start anymore ? > The Error Is > Unspecified GSS failure.Minor (2529639068): Cannot contact any KDC for realm > '4GJN.COM' > > after insert in nss.conf > "NSSEnforceValidCerts off" > > ipactl restart is starting (?) but > > ipactl status tell me > Directory Service: RUNNING > krb5kdc Service: RUNNING > kadmin Service: RUNNING > named Service: RUNNING > ipa_memcached Service: RUNNING > httpd Service: RUNNING > ipa-custodia Service: RUNNING > pki-tomcatd Service: RUNNING > ipa-otpd Service: RUNNING > ipa-ods-exporter Service: STOPPED > ods-enforcerd Service: RUNNING > ipa-dnskeysyncd Service: RUNNING > ipa: INFO: The ipactl command was successful > > with certutil -d /etc/httpd/alias -L I have now this > Certificate Nickname Trust Attributes > SSL,S/MIME,JAR/XPI > > Signing-Cert u,u,u > 4GJN_CA_FILE u,u,u > ipaCert u,u,u > 4GJN.COM IPA CA CT,C,C > STARTCOM-ROOT C,, > > I can Insert in nss.conf by the > #NSSNickname "Signing-Cert" original > or > NSSNickname 4GJN_CA_FILE but all is now broken ? > > I also add this, found in Bugzilla > certutil -d /var/lib/pki/pki-tomcat/alias -L > > Certificate Nickname Trust Attributes > SSL,S/MIME,JAR/XPI > > ocspSigningCert cert-pki-ca u,u,u > subsystemCert cert-pki-ca u,u,u > caSigningCert cert-pki-ca CTu,Cu,Cu > Server-Cert cert-pki-ca u,u,u > auditSigningCert cert-pki-ca u,u,Pu > STARTCOM-ROOT CT,, > > this is created in the > > certutil -d /etc/dirsrv/slapd-4GJN.COM -L > > Certificate Nickname Trust Attributes > SSL,S/MIME,JAR/XPI > > 4GJN_CA_FILE u,u,u > 4GJN.COM IPA CA CT,C,C > STARTCOM-ROOT C,, > > Can any help a little, please ;-) > > The bad Problem, I tested this with my master server with DNS / DNSSEC I can't > new install (DNSSEC Keys) > From deepak_dimri at hotmail.com Mon Sep 19 11:31:11 2016 From: deepak_dimri at hotmail.com (Deepak Dimri) Date: Mon, 19 Sep 2016 07:31:11 -0400 Subject: [Freeipa-users] Want to extend schema for ipahost Message-ID: Hi All, I want to add couple of custom attribute to IPA Host. I have already added custom attributes and objectclass "AWSInstanceDetails" to my schema succesfully but when i am trying to modify existing host to include the new objectclass i am getting below error ldap_modify: Object class violation (65) additional info: missing attribute "sn" required by object class "AWSInstanceDetails" my ldif file to add the newly created objectclass. dn: fqdn=testhost,dc=ddiam,dd=online changetype: modify add: objectclass objectclass: AWSInstanceDetails How can i extend my ipahost objectclass to include additional attributes? i followed this document to extend ipa userobjectclass https://www.freeipa.org/images/5/5b/FreeIPA33-extending-freeipa.pdf but now i need help with ipahost As always any help would be much appreciated! Thanks,Deepak -------------- next part -------------- An HTML attachment was scrubbed... URL: From flo at redhat.com Mon Sep 19 11:41:00 2016 From: flo at redhat.com (Florence Blanc-Renaud) Date: Mon, 19 Sep 2016 13:41:00 +0200 Subject: [Freeipa-users] Want to extend schema for ipahost In-Reply-To: References: Message-ID: <34bbe704-d45d-6fad-4bf1-74f08b449409@redhat.com> On 09/19/2016 01:31 PM, Deepak Dimri wrote: > Hi All, > > I want to add couple of custom attribute to IPA Host. I have already > added custom attributes and objectclass "AWSInstanceDetails" to my > schema succesfully but when i am trying to modify existing host to > include the new objectclass i am getting below error > > ldap_modify: Object class violation (65) > > additional info: missing attribute "sn" required by object class > "AWSInstanceDetails" > > > my ldif file to add the newly created objectclass. > > > dn: fqdn=testhost,dc=ddiam,dd=online > > changetype: modify > > add: objectclass > > objectclass: AWSInstanceDetails > > > How can i extend my ipahost objectclass to include additional > attributes? i followed this document to extend ipa > userobjectclass https://www.freeipa.org/images/5/5b/FreeIPA33-extending-freeipa.pdf but > now i need help with ipahost > > > As always any help would be much appreciated! > > > Thanks, > > Deepak > > > Hi Deepak, What is your schema definition for AWSInstanceDetails? If it requires the "sn" attribute as a mandatory attribute (i.e in the MUST section), then you need to define a value for sn in your ldif file. Otherwise the schema would not be respected by your object. For instance: dn: fqdn=testhost,dc=ddiam,dd=online changetype: modify add: objectclass objectclass: AWSInstanceDetails - add: sn sn: myValue If, on the contrary, you do not want the attribute to be mandatory, you can define the AWSInstanceDetails objectclass with an optional "sn" attribute, by putting sn in the MAY section. Hope this helps, Flo. From th at casalogic.dk Mon Sep 19 11:47:22 2016 From: th at casalogic.dk (Troels Hansen) Date: Mon, 19 Sep 2016 13:47:22 +0200 (CEST) Subject: [Freeipa-users] User gecos in IPA-AD trust Message-ID: <2061013941.305488.1474285642236.JavaMail.zimbra@casalogic.dk> Hi, i'm having some problems setting user's gecos in AD trust environment. No matter what I change ldap_user_gecos to its not changes on AD users. I guess its because I can only set it on the IPA domain, in SSSD config, and it can't be added to the subdomain_inherit ? -- Med venlig hilsen Troels Hansen Systemkonsulent Casalogic A/S T (+45) 70 20 10 63 M (+45) 22 43 71 57 Red Hat, SUSE, VMware, Citrix, Novell, Yellowfin BI, EnterpriseDB, Sophos og meget mere. -------------- next part -------------- An HTML attachment was scrubbed... URL: From jhrozek at redhat.com Mon Sep 19 11:56:39 2016 From: jhrozek at redhat.com (Jakub Hrozek) Date: Mon, 19 Sep 2016 13:56:39 +0200 Subject: [Freeipa-users] User gecos in IPA-AD trust In-Reply-To: <2061013941.305488.1474285642236.JavaMail.zimbra@casalogic.dk> References: <2061013941.305488.1474285642236.JavaMail.zimbra@casalogic.dk> Message-ID: <20160919115639.s7w3ooaytz6f2fcl@hendrix> On Mon, Sep 19, 2016 at 01:47:22PM +0200, Troels Hansen wrote: > Hi, i'm having some problems setting user's gecos in AD trust environment. > > No matter what I change ldap_user_gecos to its not changes on AD users. > > I guess its because I can only set it on the IPA domain, in SSSD config, and it can't be added to the subdomain_inherit ? Currently it's not possible. From deepak_dimri at hotmail.com Mon Sep 19 12:00:39 2016 From: deepak_dimri at hotmail.com (Deepak Dimri) Date: Mon, 19 Sep 2016 08:00:39 -0400 Subject: [Freeipa-users] Want to extend schema for ipahost In-Reply-To: <34bbe704-d45d-6fad-4bf1-74f08b449409@redhat.com> References: , <34bbe704-d45d-6fad-4bf1-74f08b449409@redhat.com> Message-ID: Thank You Flo This helped!!! Best regards,Deepak > Subject: Re: [Freeipa-users] Want to extend schema for ipahost > To: deepak_dimri at hotmail.com; freeipa-users at redhat.com > From: flo at redhat.com > Date: Mon, 19 Sep 2016 13:41:00 +0200 > > On 09/19/2016 01:31 PM, Deepak Dimri wrote: > > Hi All, > > > > I want to add couple of custom attribute to IPA Host. I have already > > added custom attributes and objectclass "AWSInstanceDetails" to my > > schema succesfully but when i am trying to modify existing host to > > include the new objectclass i am getting below error > > > > ldap_modify: Object class violation (65) > > > > additional info: missing attribute "sn" required by object class > > "AWSInstanceDetails" > > > > > > my ldif file to add the newly created objectclass. > > > > > > dn: fqdn=testhost,dc=ddiam,dd=online > > > > changetype: modify > > > > add: objectclass > > > > objectclass: AWSInstanceDetails > > > > > > How can i extend my ipahost objectclass to include additional > > attributes? i followed this document to extend ipa > > userobjectclass https://www.freeipa.org/images/5/5b/FreeIPA33-extending-freeipa.pdf but > > now i need help with ipahost > > > > > > As always any help would be much appreciated! > > > > > > Thanks, > > > > Deepak > > > > > > > > Hi Deepak, > > What is your schema definition for AWSInstanceDetails? If it requires > the "sn" attribute as a mandatory attribute (i.e in the MUST section), > then you need to define a value for sn in your ldif file. Otherwise the > schema would not be respected by your object. > > For instance: > dn: fqdn=testhost,dc=ddiam,dd=online > changetype: modify > add: objectclass > objectclass: AWSInstanceDetails > - > add: sn > sn: myValue > > If, on the contrary, you do not want the attribute to be mandatory, you > can define the AWSInstanceDetails objectclass with an optional "sn" > attribute, by putting sn in the MAY section. > > Hope this helps, > Flo. > -------------- next part -------------- An HTML attachment was scrubbed... URL: From mbasti at redhat.com Mon Sep 19 12:01:04 2016 From: mbasti at redhat.com (Martin Basti) Date: Mon, 19 Sep 2016 14:01:04 +0200 Subject: [Freeipa-users] Want to extend schema for ipahost In-Reply-To: <34bbe704-d45d-6fad-4bf1-74f08b449409@redhat.com> References: <34bbe704-d45d-6fad-4bf1-74f08b449409@redhat.com> Message-ID: On 19.09.2016 13:41, Florence Blanc-Renaud wrote: > On 09/19/2016 01:31 PM, Deepak Dimri wrote: >> Hi All, >> >> I want to add couple of custom attribute to IPA Host. I have already >> added custom attributes and objectclass "AWSInstanceDetails" to my >> schema succesfully but when i am trying to modify existing host to >> include the new objectclass i am getting below error >> >> ldap_modify: Object class violation (65) >> >> additional info: missing attribute "sn" required by object class >> "AWSInstanceDetails" >> >> >> my ldif file to add the newly created objectclass. >> >> >> dn: fqdn=testhost,dc=ddiam,dd=online >> >> changetype: modify >> >> add: objectclass >> >> objectclass: AWSInstanceDetails >> >> >> How can i extend my ipahost objectclass to include additional >> attributes? i followed this document to extend ipa >> userobjectclass >> https://www.freeipa.org/images/5/5b/FreeIPA33-extending-freeipa.pdf but >> now i need help with ipahost >> >> >> As always any help would be much appreciated! >> >> >> Thanks, >> >> Deepak >> >> >> > > Hi Deepak, > > What is your schema definition for AWSInstanceDetails? If it requires > the "sn" attribute as a mandatory attribute (i.e in the MUST section), > then you need to define a value for sn in your ldif file. Otherwise > the schema would not be respected by your object. > > For instance: > dn: fqdn=testhost,dc=ddiam,dd=online > changetype: modify > add: objectclass > objectclass: AWSInstanceDetails > - > add: sn > sn: myValue > > If, on the contrary, you do not want the attribute to be mandatory, > you can define the AWSInstanceDetails objectclass with an optional > "sn" attribute, by putting sn in the MAY section. > > Hope this helps, > Flo. > Yes please use only MAY attributes otherwise you will not be able to create new entries using IPA CLI/webUI Martin^2 From pspacek at redhat.com Mon Sep 19 12:16:55 2016 From: pspacek at redhat.com (Petr Spacek) Date: Mon, 19 Sep 2016 14:16:55 +0200 Subject: [Freeipa-users] bind crashes on rndc reload In-Reply-To: <2892489.MTAJ5pYV2v@ws1.m.messinet.com> References: <77002552-83b0-7e5d-0950-9d8f78b504de@winteltosh.de> <2892489.MTAJ5pYV2v@ws1.m.messinet.com> Message-ID: On 12.9.2016 11:55, Anthony Joseph Messina wrote: > On Monday, September 12, 2016 10:31:10 AM CDT Jochen Demmer wrote: >> Hi, >> >> I have a major issue with my setup: >> Fedora 24 >> freeipa-common-4.3.2-2.fc24.noarch >> freeipa-admintools-4.3.2-2.fc24.noarch >> freeipa-server-dns-4.3.2-2.fc24.noarch >> freeipa-client-common-4.3.2-2.fc24.noarch >> freeipa-server-4.3.2-2.fc24.x86_64 >> freeipa-server-common-4.3.2-2.fc24.noarch >> freeipa-client-4.3.2-2.fc24.x86_64 >> bind-dyndb-ldap-9.0-3.fc24.x86_64 >> bind-libs-lite-9.10.4-1.P2.fc24.x86_64 >> bind-pkcs11-libs-9.10.4-1.P2.fc24.x86_64 >> bind99-libs-9.9.9-1.P2.fc24.x86_64 >> bind-utils-9.10.4-1.P2.fc24.x86_64 >> rpcbind-0.2.3-11.rc1.fc24.x86_64 >> bind-license-9.10.4-1.P2.fc24.noarch >> bind-pkcs11-9.10.4-1.P2.fc24.x86_64 >> bind-9.10.4-1.P2.fc24.x86_64 >> bind-libs-9.10.4-1.P2.fc24.x86_64 >> bind99-license-9.9.9-1.P2.fc24.noarch >> bind-pkcs11-utils-9.10.4-1.P2.fc24.x86_64 >> >> It seems that there is a regular but not daily "rndc reload" sent to the >> nameserver that leads to a crash of it. I sent a SIGHUP to the named >> process, but that didn't lead to a crash. Only "rndc reload" does. It >> does not crash EVERY time, but most of the times. I need to do an >> "ipactl restart" in order to get the nameserver up and running again. >> >> I found this thread, but this doesn't give me any clues: >> https://www.redhat.com/archives/freeipa-users/2012-May/msg00340.html >> >> This is what the log says: >> http://paste.debian.net/818354/ >> Please understand that I obfuscated my IP addresses and domain names >> >> This is the strace: >> http://paste.debian.net/818365/ >> >> This is my named.conf: >> http://paste.debian.net/818368/ >> >> Hope someone can help. >> Jochen > > I wish I could, as I have the same issue, usually early Sunday morning after > some cron/timer job that reloads: > > https://bugzilla.redhat.com/show_bug.cgi?id=1362162 Could you please try bind-dyndb-ldap-10.1-1.fc24 from updates-testing? Alternatively the package can be downloaded from http://koji.fedoraproject.org/koji/buildinfo?buildID=792505 Please let me know if it fixes your problem or not. -- Petr^2 Spacek From g.kafataridis at nelios.com Mon Sep 19 12:51:39 2016 From: g.kafataridis at nelios.com (Giorgos Kafataridis) Date: Mon, 19 Sep 2016 15:51:39 +0300 Subject: [Freeipa-users] CA: Cannot add Centos7.2 replica to Centos6.8 ipa server In-Reply-To: <93f6dbfd-dff8-aa92-b540-cb7281fd29de@redhat.com> References: <45b66871-aa01-3c91-0630-0bbeee488ae7@nelios.com> <323b9de1-adb3-47af-d4a7-d6083f88d75c@nelios.com> <9afdf0f3-97d1-cc6e-5f3d-0e11687a124a@redhat.com> <05b8e986-ceef-8e75-c76d-b38c45e2e3a1@nelios.com> <79903d5c-2b80-d492-6b29-59ac5b39c6f5@redhat.com> <98b16feb-8b70-13bb-4c92-cd9bca3e408d@redhat.com> <0b546b4a-9ef3-7a15-27a7-3a2d59e5281f@redhat.com> <93f6dbfd-dff8-aa92-b540-cb7281fd29de@redhat.com> Message-ID: On 09/16/2016 06:39 PM, Petr Vobornik wrote: > On 09/14/2016 07:26 PM, Giorgos Kafataridis wrote: >> >> On 09/13/2016 10:36 PM, Endi Sukma Dewata wrote: >>> On 9/12/2016 9:35 PM, Endi Sukma Dewata wrote: >>>> On 9/9/2016 2:46 PM, Georgios Kafataridis wrote: >>>>> I've tried that but still the same result. >>>>> >>>>> [root at ipa-server /]# ldapsearch -D "cn=directory manager" -W -p 389 -h >>>>> localhost -b "uid=admin,ou=people,o=ipaca" >>>>> Enter LDAP Password: >>>>> # extended LDIF >>>>> # >>>>> # LDAPv3 >>>>> # base with scope subtree >>>>> # filter: (objectclass=*) >>>>> # requesting: ALL >>>>> # >>>>> >>>>> # search result >>>>> search: 2 >>>>> result: 32 No such object >>>> Hi, >>>> >>>> The master's logs indicate there's an authentication issue. >>>> >>>> Could you search the whole directory to find the admin user? >>>> $ ldapsearch ... -b "o=ipaca" "(uid=admin)" >>>> >>>> Try also other suffixes that you have in the DS. >>>> >>>> If you find it, try to authenticate against DS directly as the admin >>>> user. If the authentication fails, try resetting the password. >>> I believe there is actually another DS instance on CentOS 6.8 running on port >>> 7389, so make sure you check that too. If the admin user is indeed missing, it >>> will need to be recreated, assigned a password and certificate, and added to >>> the appropriate groups. >>> >>> See also: http://pki.fedoraproject.org/wiki/IPA_PKI_Users >>> >> Sorry for the delay, crazy office days. >> >> Ok, tried that and finally got a hit on the user. Indeed in 6.x you also have >> 7389 to look for. >> >> *Master >> >> *#ldapsearch -h localhost -p 7389 -b "o=ipaca" "(uid=admin)" -x -W >> Enter LDAP Password: >> >> # extended LDIF >> # >> # LDAPv3 >> # base with scope subtree >> # filter: (uid=admin) >> # requesting: ALL >> # >> >> # admin, people, ipaca >> dn: uid=admin,ou=people,o=ipaca >> objectClass: top >> objectClass: person >> objectClass: organizationalPerson >> objectClass: inetOrgPerson >> objectClass: cmsuser >> uid: admin >> sn: admin >> cn: admin >> mail: root at localhost >> usertype: adminType >> userstate: 1 >> description: 2;6;CN=Certificate Authority,O=NELIOS;CN=ipa-ca-agent,O=NELIOS >> userCertificate:: MIIDaTCCAlGgAwIBAgIBBjANBgkqhkiG9w0BAQsFADAxMQ8wDQYDVQQKEwZO.... >> . >> . >> . >> . >> # search result >> search: 2 >> result: 0 Success >> >> # numResponses: 2 >> # numEntries: 1 >> >> >> *Replica Server* >> >> [root at ipa2-server2 ~]# ldapsearch -h ipa-server.nelios -p 7389 -b "o=ipaca" >> "(uid=admin)" -x -W >> # extended LDIF >> # >> # LDAPv3 >> # base with scope subtree >> # filter: (uid=admin) >> # requesting: ALL >> # >> >> # admin, people, ipaca >> dn: uid=admin,ou=people,o=ipaca >> objectClass: top >> objectClass: person >> objectClass: organizationalPerson >> objectClass: inetOrgPerson >> objectClass: cmsuser >> uid: admin >> sn: admin >> cn: admin >> mail: root at localhost >> usertype: adminType >> userstate: 1 >> >> Password is valid in both cases. >> >> So the user is there and can be retrieved from replica, assuming that >> ipa-replica-install also tries 7389 the only thing I can try now is "ipa >> cert-request --uid admin" to create a new certificate, generate a new cacert.p12 >> and retry install ? >> >> > In the other subthred there was a log from CentOS6 machine: > > /var/log/pki-ca/system > > 5337.TP-Processor3 - [09/Sep/2016:22:59:42 EEST] [6] [6] Failed to > authenticate as admin UID=admin. Error: netscape.ldap.LDAPException: > error result (49) > 5337.TP-Processor3 - [09/Sep/2016:22:59:42 EEST] [3] [3] Servlet > caGetCookie: Error getting servlet output stream when rendering > template. Error Invalid Credential.. > > Which to me looks like a wrong password. Which indicates my original > theory that IPA admin user shared with CA admin user the same password > but it got out of sync. During replica installation it uses IPA admin > user for authenticating as PKI admin user. > > If that is correct then changing PKI admin user's ( > uid=admin,ou=people,o=ipaca ) password to IPA admin user's password > might fix the issue. > Thanks! Ok, I will look into that more. In any case, I'll keep you posted. From rcritten at redhat.com Mon Sep 19 15:27:39 2016 From: rcritten at redhat.com (Rob Crittenden) Date: Mon, 19 Sep 2016 11:27:39 -0400 Subject: [Freeipa-users] certificates not renewing CA_UNREACHEABLE In-Reply-To: References: <806bdf32-558c-4972-255a-0901f47c4c4b@redhat.com> <6596593a-9470-bebd-f236-f9d17aa4a56c@redhat.com> <57DC0009.4030602@redhat.com> Message-ID: <57E003EB.3050409@redhat.com> Natxo Asenjo wrote: > hi, > > > On Fri, Sep 16, 2016 at 4:22 PM, Rob Crittenden > wrote: > > The 3 certs you list are the ones that are renewed via the IPA API > (as opposed to the subsystem certs renewed directly by dogtag). I > think the failures are all related. I had someone else report the > CSR decoding failure and he just restarted IPA and that fixes things > for him though it was a rather unsatisfying fix. > > What I'd do is this. Assuming each step works, move onto the next. > > 1. ipa cert-show 1 > > The serial # picked more or less at random, we're testing > connectivity and that the CA is up and operational. > > 2. I assume that getcert list | grep expire shows all certs > currently valid? The IPA service certs expire in a month, how about > the CA subsystem certs? > > 3. Is this the same server having problems talking to the CA due to > the other NSS errors? If so what I'd do is restart httpd then > immediately use ipa-getcert to resubmit the requests to try to get > into that few minute window. > > If this is the same box you already have debugging enabled so seeing > what that shows might be helpful. > > rob > > > > yes, all certs are valid (see attachment getcert.txt). > > So I restarted httpd, I could execute ipa cert-show 1 and get an answer, > inmediately after I run > > $ sudo ipa-getcert resubmit -i 20121107212513 > Resubmitting "20121107212513" to "IPA". > > and now the status is the one you see in the attached getcert.txt file. > The server failed request, will retry. > > I do not know if it's important, but I saw that the usercertificate > attribute of the pki user admin was expired.1 > > I attach the error_log of httpd as well. Ok, how about we work around the problem. Since it is failing on the revocation what you might try is removing the userCertificate value from the ldap/kdc01.unix.iriszorg.nl service entry. I think this will work: $ ipa service-show ldap/kdc01.unix.iriszorg.nl |grep Serial $ ipa service-mod --certificate= ldap/kdc01.unix.iriszorg.nl If this doesn't work you can use ldapmodify to delete the usercertificate value. This will remove the certificate value so there is nothing to revoke and a new cert will be saved (hopefully). Now try to resubmit the request via certmonger. It if works then you can run ipa cert-revooke It isn't a great answer long-term because it is really just working around the problem but it should get the certs renewed. rob From rharwood at redhat.com Mon Sep 19 16:45:31 2016 From: rharwood at redhat.com (Robbie Harwood) Date: Mon, 19 Sep 2016 12:45:31 -0400 Subject: [Freeipa-users] Fwd: Re: Increase ListenBacklog for httpd In-Reply-To: <9819313c-6018-728f-4c9e-77371b493eb1@redhat.com> References: <9819313c-6018-728f-4c9e-77371b493eb1@redhat.com> Message-ID: Rakesh Rajasekharan writes: > On Mon, Sep 12, 2016 at 10:13 AM, Rakesh Rajasekharan > > > wrote: > > sorry I guess I did not put the question correctly.... > > I wanted to know .. like we have the ListenBacklog for apache to > basically define the number of connections it can handle.. do we > have some thing similar for our krb5kdc service.. as the SYN floodin > at 88 looks like krb5kdc service is not able to handle sudden spurt > in connections or the number of connections are more than it could > handle.. > > So, would be great if I could know how many connection it can > support at any given time ..most of the times I see this error while > i add clients to IPA master.. so if thers a known limit , I could > first check netstat to see how many connections I have at any point > and if its below the limit only then setup ipa-client-install We intentionally do not have such a parameter in krb5. We call listen(5) internally, but please note this is probably not the parameter you want to be able to tune. The listen() backlog is the number of connections that are waiting to be accept()ed by the process. They sit in the kernel, not receiving SYNACK. This number does not count connections that the process - here krb5kdc - has accept()ed and is currently processing. If you're truly seeing connections faster than they can be accept()ed, you have a load problem that tuning this parameter likely won't fix. You should probably configure replicas: krb5 will fall back if the connection is refused from one kdc to the next configured one. This will result in faster operation for your users than waiting on an enormous listen() backlog will as well. A tunable for the listen value may be added in the future, but is not available at the present time. -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 818 bytes Desc: not available URL: From Lachlan.Simpson at petermac.org Mon Sep 19 22:17:28 2016 From: Lachlan.Simpson at petermac.org (Simpson Lachlan) Date: Mon, 19 Sep 2016 22:17:28 +0000 Subject: [Freeipa-users] In webgui, ID Views slow, to crashingly slow In-Reply-To: <90ec609b-f939-a04c-cab9-d34ce955d81d@redhat.com> References: <90ec609b-f939-a04c-cab9-d34ce955d81d@redhat.com> Message-ID: <0137003026EBE54FBEC540C5600C03C439A098@PAPR-EXMBX1.petermac.org.au> > -----Original Message----- > > On 09/19/2016 03:12 AM, Lachlan Musicman wrote: > > Hi > > > > Sometimes when I visit the ID Views page in the webgui, it is > > crushingly slow, and often it times out. > > > > Centos 7, ipa --version > > VERSION: 4.2.0, API_VERSION: 2.156 > > > > Is there a reason, can I do something to fix this? > > > > What kind of ID Views do you use? Do you use them to override AD users? > Is there any useful info in '/var/log/httpd/error_log'? There is the single ID View Name, Default Trust View, and in that we have a number of users over riding the AD usernames and home dirs. The httpd error log is relatively large, tbh, but there's nothing in there that looks like an obvious reason. In fact, for an error log, there is a hell of a lot of "SUCCESS" messages? The most obvious culprit in the error log is jsonserver_session... Next time I see it (I only see it intermittently), I'll grab the logs and have a look. Cheers L. This email (including any attachments or links) may contain confidential and/or legally privileged information and is intended only to be read or used by the addressee. If you are not the intended addressee, any use, distribution, disclosure or copying of this email is strictly prohibited. Confidentiality and legal privilege attached to this email (including any attachments) are not waived or lost by reason of its mistaken delivery to you. If you have received this email in error, please delete it and notify us immediately by telephone or email. Peter MacCallum Cancer Centre provides no guarantee that this transmission is free of virus or that it has not been intercepted or altered and will not be liable for any delay in its receipt. From amessina at messinet.com Mon Sep 19 22:33:47 2016 From: amessina at messinet.com (Anthony Joseph Messina) Date: Mon, 19 Sep 2016 17:33:47 -0500 Subject: [Freeipa-users] bind crashes on rndc reload In-Reply-To: References: <77002552-83b0-7e5d-0950-9d8f78b504de@winteltosh.de> <2892489.MTAJ5pYV2v@ws1.m.messinet.com> Message-ID: <1537316.S96Dxyse1G@linux-ws1.messinet.com> On Monday, September 19, 2016 2:16:55 PM CDT Petr Spacek wrote: > On 12.9.2016 11:55, Anthony Joseph Messina wrote: > > On Monday, September 12, 2016 10:31:10 AM CDT Jochen Demmer wrote: > >> Hi, > >> > >> I have a major issue with my setup: > >> Fedora 24 > >> freeipa-common-4.3.2-2.fc24.noarch > >> freeipa-admintools-4.3.2-2.fc24.noarch > >> freeipa-server-dns-4.3.2-2.fc24.noarch > >> freeipa-client-common-4.3.2-2.fc24.noarch > >> freeipa-server-4.3.2-2.fc24.x86_64 > >> freeipa-server-common-4.3.2-2.fc24.noarch > >> freeipa-client-4.3.2-2.fc24.x86_64 > >> bind-dyndb-ldap-9.0-3.fc24.x86_64 > >> bind-libs-lite-9.10.4-1.P2.fc24.x86_64 > >> bind-pkcs11-libs-9.10.4-1.P2.fc24.x86_64 > >> bind99-libs-9.9.9-1.P2.fc24.x86_64 > >> bind-utils-9.10.4-1.P2.fc24.x86_64 > >> rpcbind-0.2.3-11.rc1.fc24.x86_64 > >> bind-license-9.10.4-1.P2.fc24.noarch > >> bind-pkcs11-9.10.4-1.P2.fc24.x86_64 > >> bind-9.10.4-1.P2.fc24.x86_64 > >> bind-libs-9.10.4-1.P2.fc24.x86_64 > >> bind99-license-9.9.9-1.P2.fc24.noarch > >> bind-pkcs11-utils-9.10.4-1.P2.fc24.x86_64 > >> > >> It seems that there is a regular but not daily "rndc reload" sent to the > >> nameserver that leads to a crash of it. I sent a SIGHUP to the named > >> process, but that didn't lead to a crash. Only "rndc reload" does. It > >> does not crash EVERY time, but most of the times. I need to do an > >> "ipactl restart" in order to get the nameserver up and running again. > >> > >> I found this thread, but this doesn't give me any clues: > >> https://www.redhat.com/archives/freeipa-users/2012-May/msg00340.html > >> > >> This is what the log says: > >> http://paste.debian.net/818354/ > >> Please understand that I obfuscated my IP addresses and domain names > >> > >> This is the strace: > >> http://paste.debian.net/818365/ > >> > >> This is my named.conf: > >> http://paste.debian.net/818368/ > >> > >> Hope someone can help. > >> Jochen > > > > I wish I could, as I have the same issue, usually early Sunday morning > > after some cron/timer job that reloads: > > > > https://bugzilla.redhat.com/show_bug.cgi?id=1362162 > > Could you please try bind-dyndb-ldap-10.1-1.fc24 from updates-testing? > > Alternatively the package can be downloaded from > http://koji.fedoraproject.org/koji/buildinfo?buildID=792505 > > Please let me know if it fixes your problem or not. bind-dyndb-ldap-10.1-1.fc24 from updates-testing seems to work, or at least is does not crash on rndc reload. I'll give it some time and see what happens, since it didn't crash every time before either. Thank you Petr. -A -- Anthony - https://messinet.com/ - https://messinet.com/~amessina/gallery F9B6 560E 68EA 037D 8C3D D1C9 FF31 3BDB D9D8 99B6 -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 819 bytes Desc: This is a digitally signed message part. URL: From datakid at gmail.com Mon Sep 19 23:57:38 2016 From: datakid at gmail.com (Lachlan Musicman) Date: Tue, 20 Sep 2016 09:57:38 +1000 Subject: [Freeipa-users] HBAC doesn't work issues In-Reply-To: <20160919082116.GF4644@10.4.128.1> References: <20160919082116.GF4644@10.4.128.1> Message-ID: We have one "allow all" sudo rule (anyone, any host, any command). Matching Defaults entries for root on this host: requiretty, !visiblepw, always_set_home, env_reset, env_keep="COLORS DISPLAY HOSTNAME HISTSIZE INPUTRC KDEDIR LS_COLORS", env_keep+="MAIL PS1 PS2 QTDIR USERNAME LANG LC_ADDRESS LC_CTYPE", env_keep+="LC_COLLATE LC_IDENTIFICATION LC_MEASUREMENT LC_MESSAGES", env_keep+="LC_MONETARY LC_NAME LC_NUMERIC LC_PAPER LC_TELEPHONE", env_keep+="LC_TIME LC_ALL LANGUAGE LINGUAS _XKB_CHARSET XAUTHORITY", secure_path=/sbin\:/bin\:/usr/sbin\:/usr/bin User root may run the following commands on this host: (ALL) ALL My sssd.conf has: [domain/unixdev.etc] ... sudo_provider = ldap ldap_uri = ldap://vmdv-linuxidm1.unixdev.petermac.org.au ldap_sudo_search_base = or=sudoers,dc=unixdev,dc=petermac,dc=org,dc=au ldap_sasl_mech = GSSAPI ldap_sasl_authid = host/vmdv-linuxidm1.unixdev.petermac.org.au ldap_sasl_realm = UNIXDEV.PETERMAC.ORG.AU krb5_server = vmdv-linuxidm1.unixdev.petermac.org.au [sssd] services = nss, sudo, pam, ssh config_file_version = 2 domains = unixdev.petermac.org.au debug_level = 6 [sudo] debug_level = 6 but only on the server - does that need to filter down to each client? The client side sssd.confs seem to be auto created when ipa-client-install is run, and are stripped down... cheers L. ------ The most dangerous phrase in the language is, "We've always done it this way." - Grace Hopper On 19 September 2016 at 18:21, Lukas Slebodnik wrote: > On (19/09/16 16:43), Lachlan Musicman wrote: > >I must have made an error again: > > > >- ipa hbactest gives seemingly correct answer on both server and client > >- user can't actually use sudo on client? > > > >Centos 7, freeipa 4.2.o/2.156; sssd 1.14.1 from COPR > > > >>From the server: > > > >[root at vmdv-linuxidm1 ~]# ipa hbactest --user=lsimpson at petermac.org.au > >--host=vmts-linuxclient1.unixdev.petermac.org.au --service=sudo > >-------------------- > >Access granted: True > >-------------------- > > Matched rules: Cluster Admin Users (sudo) > > Not matched rules: Cluster Users > >[root at vmdv-linuxidm1 ~]# > > > > > >>From the host in question: > > > >[root at vmts-linuxclient1 ~]# ipa hbactest --user lsimpson at petermac.org.au > >--host `hostname` --service sudo > >-------------------- > >Access granted: True > >-------------------- > > Matched rules: Cluster Admin Users (sudo) > > Not matched rules: Cluster Users > >[root at vmts-linuxclient1 ~]# > > > > > >[lsimpson at petermac.org.au@vmts-linuxclient1 ~]$ sudo reboot > >[sudo] password for lsimpson at petermac.org.au: > >lsimpson at petermac.org.au is not allowed to run sudo on vmts-linuxclient1. > >This incident will be reported. > > > Did you configure sudo rules for such user? > What is an output of "sudo -l" > > LS > -------------- next part -------------- An HTML attachment was scrubbed... URL: From datakid at gmail.com Tue Sep 20 00:23:24 2016 From: datakid at gmail.com (Lachlan Musicman) Date: Tue, 20 Sep 2016 10:23:24 +1000 Subject: [Freeipa-users] HBAC doesn't work issues In-Reply-To: References: <20160919082116.GF4644@10.4.128.1> Message-ID: (redface) It seems to be working. Thanks ------ The most dangerous phrase in the language is, "We've always done it this way." - Grace Hopper On 20 September 2016 at 09:57, Lachlan Musicman wrote: > We have one "allow all" sudo rule (anyone, any host, any command). > > Matching Defaults entries for root on this host: > requiretty, !visiblepw, always_set_home, env_reset, env_keep="COLORS > DISPLAY HOSTNAME HISTSIZE INPUTRC KDEDIR LS_COLORS", env_keep+="MAIL PS1 > PS2 QTDIR USERNAME LANG LC_ADDRESS LC_CTYPE", env_keep+="LC_COLLATE > LC_IDENTIFICATION LC_MEASUREMENT LC_MESSAGES", env_keep+="LC_MONETARY > LC_NAME LC_NUMERIC LC_PAPER LC_TELEPHONE", env_keep+="LC_TIME LC_ALL > LANGUAGE LINGUAS _XKB_CHARSET XAUTHORITY", > secure_path=/sbin\:/bin\:/usr/sbin\:/usr/bin > > User root may run the following commands on this host: > (ALL) ALL > > > My sssd.conf has: > > [domain/unixdev.etc] > ... > sudo_provider = ldap > ldap_uri = ldap://vmdv-linuxidm1.unixdev.petermac.org.au > ldap_sudo_search_base = or=sudoers,dc=unixdev,dc=petermac,dc=org,dc=au > ldap_sasl_mech = GSSAPI > ldap_sasl_authid = host/vmdv-linuxidm1.unixdev.petermac.org.au > ldap_sasl_realm = UNIXDEV.PETERMAC.ORG.AU > krb5_server = vmdv-linuxidm1.unixdev.petermac.org.au > > [sssd] > services = nss, sudo, pam, ssh > config_file_version = 2 > domains = unixdev.petermac.org.au > debug_level = 6 > > [sudo] > debug_level = 6 > > but only on the server - does that need to filter down to each client? The > client side sssd.confs seem to be auto created when ipa-client-install is > run, and are stripped down... > > cheers > L. > > ------ > The most dangerous phrase in the language is, "We've always done it this > way." > > - Grace Hopper > > On 19 September 2016 at 18:21, Lukas Slebodnik > wrote: > >> On (19/09/16 16:43), Lachlan Musicman wrote: >> >I must have made an error again: >> > >> >- ipa hbactest gives seemingly correct answer on both server and client >> >- user can't actually use sudo on client? >> > >> >Centos 7, freeipa 4.2.o/2.156; sssd 1.14.1 from COPR >> > >> >>From the server: >> > >> >[root at vmdv-linuxidm1 ~]# ipa hbactest --user=lsimpson at petermac.org.au >> >--host=vmts-linuxclient1.unixdev.petermac.org.au --service=sudo >> >-------------------- >> >Access granted: True >> >-------------------- >> > Matched rules: Cluster Admin Users (sudo) >> > Not matched rules: Cluster Users >> >[root at vmdv-linuxidm1 ~]# >> > >> > >> >>From the host in question: >> > >> >[root at vmts-linuxclient1 ~]# ipa hbactest --user lsimpson at petermac.org.au >> >--host `hostname` --service sudo >> >-------------------- >> >Access granted: True >> >-------------------- >> > Matched rules: Cluster Admin Users (sudo) >> > Not matched rules: Cluster Users >> >[root at vmts-linuxclient1 ~]# >> > >> > >> >[lsimpson at petermac.org.au@vmts-linuxclient1 ~]$ sudo reboot >> >[sudo] password for lsimpson at petermac.org.au: >> >lsimpson at petermac.org.au is not allowed to run sudo on >> vmts-linuxclient1. >> >This incident will be reported. >> > >> Did you configure sudo rules for such user? >> What is an output of "sudo -l" >> >> LS >> > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From datakid at gmail.com Tue Sep 20 05:06:30 2016 From: datakid at gmail.com (Lachlan Musicman) Date: Tue, 20 Sep 2016 15:06:30 +1000 Subject: [Freeipa-users] sssd.conf - the server and host-client relationship Message-ID: Hola, What is the relationship between the IPA server, host-clients and the sssd.conf? >From what I can tell, sssd.conf is edited/changed by the ipa-client-install process on the host-client. What level of similarity does there need to be between the two sssd.confs? My server's sssd.conf has a significant number of extra parameters set that are not getting put onto the clients. Debug levels are the most obvious, and understandable, omissions - but some others are frustrating. The (non debug_level) parameters missing are: ---------------------- [domain/unixdev.etc] ignore_group_members = True ldap_purge_cache_timeout = 0 subdomain_inherit = ignore_group_members, ldap_purge_cache_timeout selinux_provider = none ipa_server_mode = True sudo_provider = ldap ldap_uri = ldap://vmdv-linuxidm1.unixdev.petermac.org.au ldap_sudo_search_base = or=sudoers,dc=unixdev,dc=petermac,dc=org,dc=au ldap_sasl_mech = GSSAPI ldap_sasl_authid = host/vmdv-linuxidm1.unixdev.petermac.org.au ldap_sasl_realm = UNIXDEV.PETERMAC.ORG.AU krb5_server = vmdv-linuxidm1.unixdev.petermac.org.au [sssd] config_file_version = 2 domains = unixdev.etc [nss] memcache_timeout = 600 ---------------------- The other diff is that the host has: ipa_server = vmdv-linuxidm1.unixdev.petermac.org.au client has: ipa_server = _srv_, vmdv-linuxidm1.unixdev.petermac.org.au Which I presume is expected/desired. And the reason I ask is because we have selinux disabled, and without the "selinux_provider = none" line, we would get kicked out as soon as freeipa had logged us in with message: Connection to test_client.unixdev.petermac.org.au closed by remote host. and on that host-client there was a brand new selinux_child.log that I'd never seen before. cheers L. ------ The most dangerous phrase in the language is, "We've always done it this way." - Grace Hopper -------------- next part -------------- An HTML attachment was scrubbed... URL: From mbabinsk at redhat.com Tue Sep 20 05:44:51 2016 From: mbabinsk at redhat.com (Martin Babinsky) Date: Tue, 20 Sep 2016 07:44:51 +0200 Subject: [Freeipa-users] In webgui, ID Views slow, to crashingly slow In-Reply-To: <0137003026EBE54FBEC540C5600C03C439A098@PAPR-EXMBX1.petermac.org.au> References: <90ec609b-f939-a04c-cab9-d34ce955d81d@redhat.com> <0137003026EBE54FBEC540C5600C03C439A098@PAPR-EXMBX1.petermac.org.au> Message-ID: <132f65a8-65ab-048a-64fc-2df5921c5a94@redhat.com> On 09/20/2016 12:17 AM, Simpson Lachlan wrote: >> -----Original Message----- >> >> On 09/19/2016 03:12 AM, Lachlan Musicman wrote: >>> Hi >>> >>> Sometimes when I visit the ID Views page in the webgui, it is >>> crushingly slow, and often it times out. >>> >>> Centos 7, ipa --version >>> VERSION: 4.2.0, API_VERSION: 2.156 >>> >>> Is there a reason, can I do something to fix this? >>> >> >> What kind of ID Views do you use? Do you use them to override AD users? >> Is there any useful info in '/var/log/httpd/error_log'? > > There is the single ID View Name, Default Trust View, and in that we have a number of users over riding the AD usernames and home dirs. > > The httpd error log is relatively large, tbh, but there's nothing in there that looks like an obvious reason. In fact, for an error log, there is a hell of a lot of "SUCCESS" messages? The most obvious culprit in the error log is jsonserver_session... > > Next time I see it (I only see it intermittently), I'll grab the logs and have a look. > > Cheers > L. > > > > This email (including any attachments or links) may contain > confidential and/or legally privileged information and is > intended only to be read or used by the addressee. If you > are not the intended addressee, any use, distribution, > disclosure or copying of this email is strictly > prohibited. > Confidentiality and legal privilege attached to this email > (including any attachments) are not waived or lost by > reason of its mistaken delivery to you. > If you have received this email in error, please delete it > and notify us immediately by telephone or email. Peter > MacCallum Cancer Centre provides no guarantee that this > transmission is free of virus or that it has not been > intercepted or altered and will not be liable for any delay > in its receipt. > One thing that crossed my mind is to check the connectivity to the AD domain controllers. To resolve AD user overrides, FreeIPA uses SSSD to contact AD DCs to do the username -> SID translation. If there is some problem contacting them, then there may be hangs/timeouts when resolving override anchors. Thus you may also want to to check SSSD logs (see https://fedorahosted.org/sssd/wiki/Troubleshooting) to see whether this is not the case. -- Martin^3 Babinsky From abokovoy at redhat.com Tue Sep 20 06:33:21 2016 From: abokovoy at redhat.com (Alexander Bokovoy) Date: Tue, 20 Sep 2016 09:33:21 +0300 Subject: [Freeipa-users] In webgui, ID Views slow, to crashingly slow In-Reply-To: <132f65a8-65ab-048a-64fc-2df5921c5a94@redhat.com> References: <90ec609b-f939-a04c-cab9-d34ce955d81d@redhat.com> <0137003026EBE54FBEC540C5600C03C439A098@PAPR-EXMBX1.petermac.org.au> <132f65a8-65ab-048a-64fc-2df5921c5a94@redhat.com> Message-ID: <20160920063321.4guq4fze4p2jgu7w@redhat.com> On Tue, 20 Sep 2016, Martin Babinsky wrote: >On 09/20/2016 12:17 AM, Simpson Lachlan wrote: >>>-----Original Message----- >>> >>>On 09/19/2016 03:12 AM, Lachlan Musicman wrote: >>>>Hi >>>> >>>>Sometimes when I visit the ID Views page in the webgui, it is >>>>crushingly slow, and often it times out. >>>> >>>>Centos 7, ipa --version >>>>VERSION: 4.2.0, API_VERSION: 2.156 >>>> >>>>Is there a reason, can I do something to fix this? >>>> >>> >>>What kind of ID Views do you use? Do you use them to override AD users? >>>Is there any useful info in '/var/log/httpd/error_log'? >> >>There is the single ID View Name, Default Trust View, and in that we have a number of users over riding the AD usernames and home dirs. >> >>The httpd error log is relatively large, tbh, but there's nothing in there that looks like an obvious reason. In fact, for an error log, there is a hell of a lot of "SUCCESS" messages? The most obvious culprit in the error log is jsonserver_session... >> >>Next time I see it (I only see it intermittently), I'll grab the logs and have a look. >> >>Cheers >>L. >> >> >> >>This email (including any attachments or links) may contain >>confidential and/or legally privileged information and is >>intended only to be read or used by the addressee. If you >>are not the intended addressee, any use, distribution, >>disclosure or copying of this email is strictly >>prohibited. >>Confidentiality and legal privilege attached to this email >>(including any attachments) are not waived or lost by >>reason of its mistaken delivery to you. >>If you have received this email in error, please delete it >>and notify us immediately by telephone or email. Peter >>MacCallum Cancer Centre provides no guarantee that this >>transmission is free of virus or that it has not been >>intercepted or altered and will not be liable for any delay >>in its receipt. >> > >One thing that crossed my mind is to check the connectivity to the AD >domain controllers. To resolve AD user overrides, FreeIPA uses SSSD to >contact AD DCs to do the username -> SID translation. If there is some >problem contacting them, then there may be hangs/timeouts when >resolving override anchors. Internally IPA framework attempts to resolve ID override anchors. We can actually optimize this code as ipaOriginalUID attribute contains normalized name already, written their when the override is created and never changed afterwards. This should speed up the resolution of large overrides. Martin, can you file a ticket for that? The code in question is baseidoverride.convert_anchor_to_human_readable_form() which could benefit from passing in entry_attrs and checking ipaoriginaluid there. If 'ipaoriginaluid' is missing, do analysis of ipaanchoruuid like it is done now. -- / Alexander Bokovoy From pspacek at redhat.com Tue Sep 20 06:53:58 2016 From: pspacek at redhat.com (Petr Spacek) Date: Tue, 20 Sep 2016 08:53:58 +0200 Subject: [Freeipa-users] bind crashes on rndc reload In-Reply-To: <1537316.S96Dxyse1G@linux-ws1.messinet.com> References: <77002552-83b0-7e5d-0950-9d8f78b504de@winteltosh.de> <2892489.MTAJ5pYV2v@ws1.m.messinet.com> <1537316.S96Dxyse1G@linux-ws1.messinet.com> Message-ID: On 20.9.2016 00:33, Anthony Joseph Messina wrote: > On Monday, September 19, 2016 2:16:55 PM CDT Petr Spacek wrote: >> On 12.9.2016 11:55, Anthony Joseph Messina wrote: >>> On Monday, September 12, 2016 10:31:10 AM CDT Jochen Demmer wrote: >>>> Hi, >>>> >>>> I have a major issue with my setup: >>>> Fedora 24 >>>> freeipa-common-4.3.2-2.fc24.noarch >>>> freeipa-admintools-4.3.2-2.fc24.noarch >>>> freeipa-server-dns-4.3.2-2.fc24.noarch >>>> freeipa-client-common-4.3.2-2.fc24.noarch >>>> freeipa-server-4.3.2-2.fc24.x86_64 >>>> freeipa-server-common-4.3.2-2.fc24.noarch >>>> freeipa-client-4.3.2-2.fc24.x86_64 >>>> bind-dyndb-ldap-9.0-3.fc24.x86_64 >>>> bind-libs-lite-9.10.4-1.P2.fc24.x86_64 >>>> bind-pkcs11-libs-9.10.4-1.P2.fc24.x86_64 >>>> bind99-libs-9.9.9-1.P2.fc24.x86_64 >>>> bind-utils-9.10.4-1.P2.fc24.x86_64 >>>> rpcbind-0.2.3-11.rc1.fc24.x86_64 >>>> bind-license-9.10.4-1.P2.fc24.noarch >>>> bind-pkcs11-9.10.4-1.P2.fc24.x86_64 >>>> bind-9.10.4-1.P2.fc24.x86_64 >>>> bind-libs-9.10.4-1.P2.fc24.x86_64 >>>> bind99-license-9.9.9-1.P2.fc24.noarch >>>> bind-pkcs11-utils-9.10.4-1.P2.fc24.x86_64 >>>> >>>> It seems that there is a regular but not daily "rndc reload" sent to the >>>> nameserver that leads to a crash of it. I sent a SIGHUP to the named >>>> process, but that didn't lead to a crash. Only "rndc reload" does. It >>>> does not crash EVERY time, but most of the times. I need to do an >>>> "ipactl restart" in order to get the nameserver up and running again. >>>> >>>> I found this thread, but this doesn't give me any clues: >>>> https://www.redhat.com/archives/freeipa-users/2012-May/msg00340.html >>>> >>>> This is what the log says: >>>> http://paste.debian.net/818354/ >>>> Please understand that I obfuscated my IP addresses and domain names >>>> >>>> This is the strace: >>>> http://paste.debian.net/818365/ >>>> >>>> This is my named.conf: >>>> http://paste.debian.net/818368/ >>>> >>>> Hope someone can help. >>>> Jochen >>> >>> I wish I could, as I have the same issue, usually early Sunday morning >>> after some cron/timer job that reloads: >>> >>> https://bugzilla.redhat.com/show_bug.cgi?id=1362162 >> >> Could you please try bind-dyndb-ldap-10.1-1.fc24 from updates-testing? >> >> Alternatively the package can be downloaded from >> http://koji.fedoraproject.org/koji/buildinfo?buildID=792505 >> >> Please let me know if it fixes your problem or not. > > bind-dyndb-ldap-10.1-1.fc24 from updates-testing seems to work, or at least is > does not crash on rndc reload. I'll give it some time and see what happens, > since it didn't crash every time before either. Thank you Petr. -A Thanks, I will be waiting for your observations! -- Petr^2 Spacek From venkataramana.kintali at gmail.com Tue Sep 20 07:04:59 2016 From: venkataramana.kintali at gmail.com (Venkataramana Kintali) Date: Tue, 20 Sep 2016 10:04:59 +0300 Subject: [Freeipa-users] Issues with FreeIPA SSH Key authentication In-Reply-To: <20160916085137.GB10405@10.4.128.1> References: <20160915070948.GA1485@10.4.128.1> <20160916085137.GB10405@10.4.128.1> Message-ID: Thank you Lukas. The issue , not being able to login to some servers in our setup with ssh keys, was due to incorrect permissions on /usr directory,per the following entry in /var/log/secure. *sshd[12856]: error: bad ownership or modes for AuthorizedKeysCommand path component "/usr"* After setting up the permissions for /usr to 755, I was able to login to these servers with ssh private keys. Thank you again,Lukas, for your help. Regards Venkataramana On Fri, Sep 16, 2016 at 11:51 AM, Lukas Slebodnik wrote: > On (15/09/16 11:46), Venkataramana Kintali wrote: > >Hi Lukas, > >ssh_config is also same on all servers. > >Our need is to do it both ways, to be able to login with ssh public > >keys(uploaded in IPA) and disable password login, and be able to access > >allhosts within the same IPA domain silently from any host. > >Hoping the configs will help, I am including the configurations here. > > > >ssh_config file : http://pastebin.com/MWHyH1Qw > >sshd_config file: http://pastebin.com/gpn5XhXM > >sssd_config file: http://pastebin.com/5Pby6xKp > > > Looks good to me > > >I just used some placeholders for sssd_config file in pastebin instead of > >actual values. > > > > In initial mail you wrote: > >I am able to login to some IPA clients but not able to login to other IPA > >clients with putty using private key and passphrase. > Therefore your previous test case is wrong. > If you want to test authentication with public keys > then you cannot obtain krb5 ticket with kinit. > > I would also recommend to call kdestory before > authentication with ssh to be sure that gssapi > authentication will not be used. > > I would recomment to set "debug_level = 7" in domain and ssh section > on the server where you woudl like to authenticate. > then restart sssd and try to authenticate with ssh + verbose mode > e.g. ssh -v user at remote.host > > Then I would recommend to compare logs from working server > and from broken server. > > LS > -------------- next part -------------- An HTML attachment was scrubbed... URL: From lslebodn at redhat.com Tue Sep 20 07:11:44 2016 From: lslebodn at redhat.com (Lukas Slebodnik) Date: Tue, 20 Sep 2016 09:11:44 +0200 Subject: [Freeipa-users] sssd.conf - the server and host-client relationship In-Reply-To: References: Message-ID: <20160920071144.GA2435@10.4.128.1> On (20/09/16 15:06), Lachlan Musicman wrote: >Hola, > >What is the relationship between the IPA server, host-clients and the >sssd.conf? > >>From what I can tell, sssd.conf is edited/changed by the ipa-client-install >process on the host-client. > >What level of similarity does there need to be between the two sssd.confs? > >My server's sssd.conf has a significant number of extra parameters set that >are not getting put onto the clients. > >Debug levels are the most obvious, and understandable, omissions - but some >others are frustrating. > >The (non debug_level) parameters missing are: >---------------------- >[domain/unixdev.etc] >ignore_group_members = True It was probably set as a result of performance tuning. >ldap_purge_cache_timeout = 0 That's default since 1.13.0 >subdomain_inherit = ignore_group_members, ldap_purge_cache_timeout that's specific option for sssd on IPA server >selinux_provider = none It was probably set as a workaround of bug which have been already fixed. >ipa_server_mode = True that's specific option for sssd on IPA server >sudo_provider = ldap >ldap_uri = ldap://vmdv-linuxidm1.unixdev.petermac.org.au >ldap_sudo_search_base = or=sudoers,dc=unixdev,dc=petermac,dc=org,dc=au >ldap_sasl_mech = GSSAPI >ldap_sasl_authid = host/vmdv-linuxidm1.unixdev.petermac.org.au >ldap_sasl_realm = UNIXDEV.PETERMAC.ORG.AU >krb5_server = vmdv-linuxidm1.unixdev.petermac.org.au Previous 7 options are not required since sssd-1.10 > >[sssd] >config_file_version = 2 >domains = unixdev.etc > >[nss] >memcache_timeout = 600 This option is se by ipa-*-install on ipa server mode. >---------------------- > >The other diff is that the > >host has: ipa_server = vmdv-linuxidm1.unixdev.petermac.org.au >client has: ipa_server = _srv_, vmdv-linuxidm1.unixdev.petermac.org.au > >Which I presume is expected/desired. > >And the reason I ask is because we have selinux disabled, and without the Do you eman disabled or permissive? BTW freeIPA works well with SELinux in enforcing mode >"selinux_provider = none" line, we would get kicked out as soon as freeipa >had logged us in with message: > disabled SELinux should not affected authentication; but I didn't test that. >Connection to test_client.unixdev.petermac.org.au closed by remote host. > >and on that host-client there was a brand new selinux_child.log that I'd >never seen before. > LS From rakesh.rajasekharan at gmail.com Tue Sep 20 07:31:54 2016 From: rakesh.rajasekharan at gmail.com (Rakesh Rajasekharan) Date: Tue, 20 Sep 2016 13:01:54 +0530 Subject: [Freeipa-users] Fwd: Re: Increase ListenBacklog for httpd In-Reply-To: References: <9819313c-6018-728f-4c9e-77371b493eb1@redhat.com> Message-ID: Thanks Robbie for the inputs.. the load should not have been high as I have around 4000 clients with 160 users which should be manageable However, I saw a lot of clock skew too great errors in my krb5kdc.log... however I haven't been able to verify if those were genuine... Can too many clock skew errors take down the kerberos service.. On Mon, Sep 19, 2016 at 10:15 PM, Robbie Harwood wrote: > Rakesh Rajasekharan writes: > > > On Mon, Sep 12, 2016 at 10:13 AM, Rakesh Rajasekharan > > > > > wrote: > > > > sorry I guess I did not put the question correctly.... > > > > I wanted to know .. like we have the ListenBacklog for apache to > > basically define the number of connections it can handle.. do we > > have some thing similar for our krb5kdc service.. as the SYN floodin > > at 88 looks like krb5kdc service is not able to handle sudden spurt > > in connections or the number of connections are more than it could > > handle.. > > > > So, would be great if I could know how many connection it can > > support at any given time ..most of the times I see this error while > > i add clients to IPA master.. so if thers a known limit , I could > > first check netstat to see how many connections I have at any point > > and if its below the limit only then setup ipa-client-install > > We intentionally do not have such a parameter in krb5. We call > listen(5) internally, but please note this is probably not the parameter > you want to be able to tune. > > The listen() backlog is the number of connections that are waiting to be > accept()ed by the process. They sit in the kernel, not receiving > SYNACK. This number does not count connections that the process - here > krb5kdc - has accept()ed and is currently processing. > > If you're truly seeing connections faster than they can be accept()ed, > you have a load problem that tuning this parameter likely won't fix. > You should probably configure replicas: krb5 will fall back if the > connection is refused from one kdc to the next configured one. This > will result in faster operation for your users than waiting on an > enormous listen() backlog will as well. > > A tunable for the listen value may be added in the future, but is not > available at the present time. > -------------- next part -------------- An HTML attachment was scrubbed... URL: From sbose at redhat.com Tue Sep 20 07:49:53 2016 From: sbose at redhat.com (Sumit Bose) Date: Tue, 20 Sep 2016 09:49:53 +0200 Subject: [Freeipa-users] In webgui, ID Views slow, to crashingly slow In-Reply-To: <20160920063321.4guq4fze4p2jgu7w@redhat.com> References: <90ec609b-f939-a04c-cab9-d34ce955d81d@redhat.com> <0137003026EBE54FBEC540C5600C03C439A098@PAPR-EXMBX1.petermac.org.au> <132f65a8-65ab-048a-64fc-2df5921c5a94@redhat.com> <20160920063321.4guq4fze4p2jgu7w@redhat.com> Message-ID: <20160920074953.GR2761@p.Speedport_W_724V_Typ_A_05011603_00_009> On Tue, Sep 20, 2016 at 09:33:21AM +0300, Alexander Bokovoy wrote: > On Tue, 20 Sep 2016, Martin Babinsky wrote: > > On 09/20/2016 12:17 AM, Simpson Lachlan wrote: > > > > -----Original Message----- > > > > > > > > On 09/19/2016 03:12 AM, Lachlan Musicman wrote: > > > > > Hi > > > > > > > > > > Sometimes when I visit the ID Views page in the webgui, it is > > > > > crushingly slow, and often it times out. > > > > > > > > > > Centos 7, ipa --version > > > > > VERSION: 4.2.0, API_VERSION: 2.156 > > > > > > > > > > Is there a reason, can I do something to fix this? > > > > > > > > > > > > > What kind of ID Views do you use? Do you use them to override AD users? > > > > Is there any useful info in '/var/log/httpd/error_log'? > > > > > > There is the single ID View Name, Default Trust View, and in that we have a number of users over riding the AD usernames and home dirs. > > > > > > The httpd error log is relatively large, tbh, but there's nothing in there that looks like an obvious reason. In fact, for an error log, there is a hell of a lot of "SUCCESS" messages? The most obvious culprit in the error log is jsonserver_session... > > > > > > Next time I see it (I only see it intermittently), I'll grab the logs and have a look. > > > > > > Cheers > > > L. > > > > > > > > > > > > This email (including any attachments or links) may contain > > > confidential and/or legally privileged information and is > > > intended only to be read or used by the addressee. If you > > > are not the intended addressee, any use, distribution, > > > disclosure or copying of this email is strictly > > > prohibited. > > > Confidentiality and legal privilege attached to this email > > > (including any attachments) are not waived or lost by > > > reason of its mistaken delivery to you. > > > If you have received this email in error, please delete it > > > and notify us immediately by telephone or email. Peter > > > MacCallum Cancer Centre provides no guarantee that this > > > transmission is free of virus or that it has not been > > > intercepted or altered and will not be liable for any delay > > > in its receipt. > > > > > > > One thing that crossed my mind is to check the connectivity to the AD > > domain controllers. To resolve AD user overrides, FreeIPA uses SSSD to > > contact AD DCs to do the username -> SID translation. If there is some > > problem contacting them, then there may be hangs/timeouts when resolving > > override anchors. > Internally IPA framework attempts to resolve ID override anchors. We can > actually optimize this code as ipaOriginalUID attribute contains > normalized name already, written their when the override is created and > never changed afterwards. This should speed up the resolution of large > overrides. > > Martin, can you file a ticket for that? The code in question is > baseidoverride.convert_anchor_to_human_readable_form() which could > benefit from passing in entry_attrs and checking ipaoriginaluid there. > If 'ipaoriginaluid' is missing, do analysis of ipaanchoruuid like it is > done now. As an alternative I wonder if the WebUI can be made asynchronous in the sense that it displays the raw data (the SID in this case) first and run the lookups in the background and replace the SID when the name is available. I've seen some Windows tools behaving this when looking up large numbers of SIDs. bye, Sumit > > -- > / Alexander Bokovoy > > -- > Manage your subscription for the Freeipa-users mailing list: > https://www.redhat.com/mailman/listinfo/freeipa-users > Go to http://freeipa.org for more info on the project From mbabinsk at redhat.com Tue Sep 20 08:02:20 2016 From: mbabinsk at redhat.com (Martin Babinsky) Date: Tue, 20 Sep 2016 10:02:20 +0200 Subject: [Freeipa-users] In webgui, ID Views slow, to crashingly slow In-Reply-To: <20160920063321.4guq4fze4p2jgu7w@redhat.com> References: <90ec609b-f939-a04c-cab9-d34ce955d81d@redhat.com> <0137003026EBE54FBEC540C5600C03C439A098@PAPR-EXMBX1.petermac.org.au> <132f65a8-65ab-048a-64fc-2df5921c5a94@redhat.com> <20160920063321.4guq4fze4p2jgu7w@redhat.com> Message-ID: On 09/20/2016 08:33 AM, Alexander Bokovoy wrote: > On Tue, 20 Sep 2016, Martin Babinsky wrote: >> On 09/20/2016 12:17 AM, Simpson Lachlan wrote: >>>> -----Original Message----- >>>> >>>> On 09/19/2016 03:12 AM, Lachlan Musicman wrote: >>>>> Hi >>>>> >>>>> Sometimes when I visit the ID Views page in the webgui, it is >>>>> crushingly slow, and often it times out. >>>>> >>>>> Centos 7, ipa --version >>>>> VERSION: 4.2.0, API_VERSION: 2.156 >>>>> >>>>> Is there a reason, can I do something to fix this? >>>>> >>>> >>>> What kind of ID Views do you use? Do you use them to override AD >>>> users? >>>> Is there any useful info in '/var/log/httpd/error_log'? >>> >>> There is the single ID View Name, Default Trust View, and in that we >>> have a number of users over riding the AD usernames and home dirs. >>> >>> The httpd error log is relatively large, tbh, but there's nothing in >>> there that looks like an obvious reason. In fact, for an error log, >>> there is a hell of a lot of "SUCCESS" messages? The most obvious >>> culprit in the error log is jsonserver_session... >>> >>> Next time I see it (I only see it intermittently), I'll grab the logs >>> and have a look. >>> >>> Cheers >>> L. >>> >>> >>> >>> This email (including any attachments or links) may contain >>> confidential and/or legally privileged information and is >>> intended only to be read or used by the addressee. If you >>> are not the intended addressee, any use, distribution, >>> disclosure or copying of this email is strictly >>> prohibited. >>> Confidentiality and legal privilege attached to this email >>> (including any attachments) are not waived or lost by >>> reason of its mistaken delivery to you. >>> If you have received this email in error, please delete it >>> and notify us immediately by telephone or email. Peter >>> MacCallum Cancer Centre provides no guarantee that this >>> transmission is free of virus or that it has not been >>> intercepted or altered and will not be liable for any delay >>> in its receipt. >>> >> >> One thing that crossed my mind is to check the connectivity to the AD >> domain controllers. To resolve AD user overrides, FreeIPA uses SSSD to >> contact AD DCs to do the username -> SID translation. If there is some >> problem contacting them, then there may be hangs/timeouts when >> resolving override anchors. > Internally IPA framework attempts to resolve ID override anchors. We can > actually optimize this code as ipaOriginalUID attribute contains > normalized name already, written their when the override is created and > never changed afterwards. This should speed up the resolution of large > overrides. > > Martin, can you file a ticket for that? The code in question is > baseidoverride.convert_anchor_to_human_readable_form() which could > benefit from passing in entry_attrs and checking ipaoriginaluid there. > If 'ipaoriginaluid' is missing, do analysis of ipaanchoruuid like it is > done now. > Done: https://fedorahosted.org/freeipa/ticket/6339 -- Martin^3 Babinsky From datakid at gmail.com Tue Sep 20 08:02:55 2016 From: datakid at gmail.com (Lachlan Musicman) Date: Tue, 20 Sep 2016 18:02:55 +1000 Subject: [Freeipa-users] In webgui, ID Views slow, to crashingly slow In-Reply-To: <20160920074953.GR2761@p.Speedport_W_724V_Typ_A_05011603_00_009> References: <90ec609b-f939-a04c-cab9-d34ce955d81d@redhat.com> <0137003026EBE54FBEC540C5600C03C439A098@PAPR-EXMBX1.petermac.org.au> <132f65a8-65ab-048a-64fc-2df5921c5a94@redhat.com> <20160920063321.4guq4fze4p2jgu7w@redhat.com> <20160920074953.GR2761@p.Speedport_W_724V_Typ_A_05011603_00_009> Message-ID: I've actually seen that on occasion - when it's loading sometimes that happens already? ------ The most dangerous phrase in the language is, "We've always done it this way." - Grace Hopper On 20 September 2016 at 17:49, Sumit Bose wrote: > On Tue, Sep 20, 2016 at 09:33:21AM +0300, Alexander Bokovoy wrote: > > On Tue, 20 Sep 2016, Martin Babinsky wrote: > > > On 09/20/2016 12:17 AM, Simpson Lachlan wrote: > > > > > -----Original Message----- > > > > > > > > > > On 09/19/2016 03:12 AM, Lachlan Musicman wrote: > > > > > > Hi > > > > > > > > > > > > Sometimes when I visit the ID Views page in the webgui, it is > > > > > > crushingly slow, and often it times out. > > > > > > > > > > > > Centos 7, ipa --version > > > > > > VERSION: 4.2.0, API_VERSION: 2.156 > > > > > > > > > > > > Is there a reason, can I do something to fix this? > > > > > > > > > > > > > > > > What kind of ID Views do you use? Do you use them to override AD > users? > > > > > Is there any useful info in '/var/log/httpd/error_log'? > > > > > > > > There is the single ID View Name, Default Trust View, and in that we > have a number of users over riding the AD usernames and home dirs. > > > > > > > > The httpd error log is relatively large, tbh, but there's nothing in > there that looks like an obvious reason. In fact, for an error log, there > is a hell of a lot of "SUCCESS" messages? The most obvious culprit in the > error log is jsonserver_session... > > > > > > > > Next time I see it (I only see it intermittently), I'll grab the > logs and have a look. > > > > > > > > Cheers > > > > L. > > > > > > > > > > > > > > > > This email (including any attachments or links) may contain > > > > confidential and/or legally privileged information and is > > > > intended only to be read or used by the addressee. If you > > > > are not the intended addressee, any use, distribution, > > > > disclosure or copying of this email is strictly > > > > prohibited. > > > > Confidentiality and legal privilege attached to this email > > > > (including any attachments) are not waived or lost by > > > > reason of its mistaken delivery to you. > > > > If you have received this email in error, please delete it > > > > and notify us immediately by telephone or email. Peter > > > > MacCallum Cancer Centre provides no guarantee that this > > > > transmission is free of virus or that it has not been > > > > intercepted or altered and will not be liable for any delay > > > > in its receipt. > > > > > > > > > > One thing that crossed my mind is to check the connectivity to the AD > > > domain controllers. To resolve AD user overrides, FreeIPA uses SSSD to > > > contact AD DCs to do the username -> SID translation. If there is some > > > problem contacting them, then there may be hangs/timeouts when > resolving > > > override anchors. > > Internally IPA framework attempts to resolve ID override anchors. We can > > actually optimize this code as ipaOriginalUID attribute contains > > normalized name already, written their when the override is created and > > never changed afterwards. This should speed up the resolution of large > > overrides. > > > > Martin, can you file a ticket for that? The code in question is > > baseidoverride.convert_anchor_to_human_readable_form() which could > > benefit from passing in entry_attrs and checking ipaoriginaluid there. > > If 'ipaoriginaluid' is missing, do analysis of ipaanchoruuid like it is > > done now. > > As an alternative I wonder if the WebUI can be made asynchronous in the > sense that it displays the raw data (the SID in this case) first and run > the lookups in the background and replace the SID when the name is > available. I've seen some Windows tools behaving this when looking up > large numbers of SIDs. > > bye, > Sumit > > > > > -- > > / Alexander Bokovoy > > > > -- > > Manage your subscription for the Freeipa-users mailing list: > > https://www.redhat.com/mailman/listinfo/freeipa-users > > Go to http://freeipa.org for more info on the project > > -- > Manage your subscription for the Freeipa-users mailing list: > https://www.redhat.com/mailman/listinfo/freeipa-users > Go to http://freeipa.org for more info on the project > -------------- next part -------------- An HTML attachment was scrubbed... URL: From abokovoy at redhat.com Tue Sep 20 08:10:46 2016 From: abokovoy at redhat.com (Alexander Bokovoy) Date: Tue, 20 Sep 2016 11:10:46 +0300 Subject: [Freeipa-users] In webgui, ID Views slow, to crashingly slow In-Reply-To: <20160920074953.GR2761@p.Speedport_W_724V_Typ_A_05011603_00_009> References: <90ec609b-f939-a04c-cab9-d34ce955d81d@redhat.com> <0137003026EBE54FBEC540C5600C03C439A098@PAPR-EXMBX1.petermac.org.au> <132f65a8-65ab-048a-64fc-2df5921c5a94@redhat.com> <20160920063321.4guq4fze4p2jgu7w@redhat.com> <20160920074953.GR2761@p.Speedport_W_724V_Typ_A_05011603_00_009> Message-ID: <20160920081046.rx6kbjzgam5xk4ic@redhat.com> On Tue, 20 Sep 2016, Sumit Bose wrote: >On Tue, Sep 20, 2016 at 09:33:21AM +0300, Alexander Bokovoy wrote: >> On Tue, 20 Sep 2016, Martin Babinsky wrote: >> > On 09/20/2016 12:17 AM, Simpson Lachlan wrote: >> > > > -----Original Message----- >> > > > >> > > > On 09/19/2016 03:12 AM, Lachlan Musicman wrote: >> > > > > Hi >> > > > > >> > > > > Sometimes when I visit the ID Views page in the webgui, it is >> > > > > crushingly slow, and often it times out. >> > > > > >> > > > > Centos 7, ipa --version >> > > > > VERSION: 4.2.0, API_VERSION: 2.156 >> > > > > >> > > > > Is there a reason, can I do something to fix this? >> > > > > >> > > > >> > > > What kind of ID Views do you use? Do you use them to override AD users? >> > > > Is there any useful info in '/var/log/httpd/error_log'? >> > > >> > > There is the single ID View Name, Default Trust View, and in that we have a number of users over riding the AD usernames and home dirs. >> > > >> > > The httpd error log is relatively large, tbh, but there's nothing in there that looks like an obvious reason. In fact, for an error log, there is a hell of a lot of "SUCCESS" messages? The most obvious culprit in the error log is jsonserver_session... >> > > >> > > Next time I see it (I only see it intermittently), I'll grab the logs and have a look. >> > > >> > > Cheers >> > > L. >> > > >> > > >> > > >> > > This email (including any attachments or links) may contain >> > > confidential and/or legally privileged information and is >> > > intended only to be read or used by the addressee. If you >> > > are not the intended addressee, any use, distribution, >> > > disclosure or copying of this email is strictly >> > > prohibited. >> > > Confidentiality and legal privilege attached to this email >> > > (including any attachments) are not waived or lost by >> > > reason of its mistaken delivery to you. >> > > If you have received this email in error, please delete it >> > > and notify us immediately by telephone or email. Peter >> > > MacCallum Cancer Centre provides no guarantee that this >> > > transmission is free of virus or that it has not been >> > > intercepted or altered and will not be liable for any delay >> > > in its receipt. >> > > >> > >> > One thing that crossed my mind is to check the connectivity to the AD >> > domain controllers. To resolve AD user overrides, FreeIPA uses SSSD to >> > contact AD DCs to do the username -> SID translation. If there is some >> > problem contacting them, then there may be hangs/timeouts when resolving >> > override anchors. >> Internally IPA framework attempts to resolve ID override anchors. We can >> actually optimize this code as ipaOriginalUID attribute contains >> normalized name already, written their when the override is created and >> never changed afterwards. This should speed up the resolution of large >> overrides. >> >> Martin, can you file a ticket for that? The code in question is >> baseidoverride.convert_anchor_to_human_readable_form() which could >> benefit from passing in entry_attrs and checking ipaoriginaluid there. >> If 'ipaoriginaluid' is missing, do analysis of ipaanchoruuid like it is >> done now. > >As an alternative I wonder if the WebUI can be made asynchronous in the >sense that it displays the raw data (the SID in this case) first and run >the lookups in the background and replace the SID when the name is >available. I've seen some Windows tools behaving this when looking up >large numbers of SIDs. We have that for external group members already. However, in the ID overrides there is no SID as it is. The RDN of the ID override is 'ipaAnchorUUID' attribute which is an encoding of a target reference. We don't need to resolve it because we already have it resolved in 'ipaOriginalUid' attribute -- this was done to help Schema Compatibility and SASL mapping plugins which cannot resolve ipaAnchorUUID value. We just need to make its use consistent across the IPA framework. -- / Alexander Bokovoy From abokovoy at redhat.com Tue Sep 20 08:11:28 2016 From: abokovoy at redhat.com (Alexander Bokovoy) Date: Tue, 20 Sep 2016 11:11:28 +0300 Subject: [Freeipa-users] In webgui, ID Views slow, to crashingly slow In-Reply-To: References: <90ec609b-f939-a04c-cab9-d34ce955d81d@redhat.com> <0137003026EBE54FBEC540C5600C03C439A098@PAPR-EXMBX1.petermac.org.au> <132f65a8-65ab-048a-64fc-2df5921c5a94@redhat.com> <20160920063321.4guq4fze4p2jgu7w@redhat.com> <20160920074953.GR2761@p.Speedport_W_724V_Typ_A_05011603_00_009> Message-ID: <20160920081128.dz667pm2z6qc2a5y@redhat.com> On Tue, 20 Sep 2016, Lachlan Musicman wrote: >I've actually seen that on occasion - when it's loading sometimes that >happens already? For external group members -- yes, not for ID overrides. See my other answer. -- / Alexander Bokovoy From datakid at gmail.com Tue Sep 20 08:53:16 2016 From: datakid at gmail.com (Lachlan Musicman) Date: Tue, 20 Sep 2016 18:53:16 +1000 Subject: [Freeipa-users] In webgui, ID Views slow, to crashingly slow In-Reply-To: <20160920081046.rx6kbjzgam5xk4ic@redhat.com> References: <90ec609b-f939-a04c-cab9-d34ce955d81d@redhat.com> <0137003026EBE54FBEC540C5600C03C439A098@PAPR-EXMBX1.petermac.org.au> <132f65a8-65ab-048a-64fc-2df5921c5a94@redhat.com> <20160920063321.4guq4fze4p2jgu7w@redhat.com> <20160920074953.GR2761@p.Speedport_W_724V_Typ_A_05011603_00_009> <20160920081046.rx6kbjzgam5xk4ic@redhat.com> Message-ID: I concede - FreeIPA is big and hard and I am new. Evidence would suggest that you know exactly what's going on under the hood. :) Thanks everyone. ------ The most dangerous phrase in the language is, "We've always done it this way." - Grace Hopper On 20 September 2016 at 18:10, Alexander Bokovoy wrote: > On Tue, 20 Sep 2016, Sumit Bose wrote: > >> On Tue, Sep 20, 2016 at 09:33:21AM +0300, Alexander Bokovoy wrote: >> >>> On Tue, 20 Sep 2016, Martin Babinsky wrote: >>> > On 09/20/2016 12:17 AM, Simpson Lachlan wrote: >>> > > > -----Original Message----- >>> > > > >>> > > > On 09/19/2016 03:12 AM, Lachlan Musicman wrote: >>> > > > > Hi >>> > > > > >>> > > > > Sometimes when I visit the ID Views page in the webgui, it is >>> > > > > crushingly slow, and often it times out. >>> > > > > >>> > > > > Centos 7, ipa --version >>> > > > > VERSION: 4.2.0, API_VERSION: 2.156 >>> > > > > >>> > > > > Is there a reason, can I do something to fix this? >>> > > > > >>> > > > >>> > > > What kind of ID Views do you use? Do you use them to override AD >>> users? >>> > > > Is there any useful info in '/var/log/httpd/error_log'? >>> > > >>> > > There is the single ID View Name, Default Trust View, and in that we >>> have a number of users over riding the AD usernames and home dirs. >>> > > >>> > > The httpd error log is relatively large, tbh, but there's nothing in >>> there that looks like an obvious reason. In fact, for an error log, there >>> is a hell of a lot of "SUCCESS" messages? The most obvious culprit in the >>> error log is jsonserver_session... >>> > > >>> > > Next time I see it (I only see it intermittently), I'll grab the >>> logs and have a look. >>> > > >>> > > Cheers >>> > > L. >>> > > >>> > > >>> > > >>> > > This email (including any attachments or links) may contain >>> > > confidential and/or legally privileged information and is >>> > > intended only to be read or used by the addressee. If you >>> > > are not the intended addressee, any use, distribution, >>> > > disclosure or copying of this email is strictly >>> > > prohibited. >>> > > Confidentiality and legal privilege attached to this email >>> > > (including any attachments) are not waived or lost by >>> > > reason of its mistaken delivery to you. >>> > > If you have received this email in error, please delete it >>> > > and notify us immediately by telephone or email. Peter >>> > > MacCallum Cancer Centre provides no guarantee that this >>> > > transmission is free of virus or that it has not been >>> > > intercepted or altered and will not be liable for any delay >>> > > in its receipt. >>> > > >>> > >>> > One thing that crossed my mind is to check the connectivity to the AD >>> > domain controllers. To resolve AD user overrides, FreeIPA uses SSSD to >>> > contact AD DCs to do the username -> SID translation. If there is some >>> > problem contacting them, then there may be hangs/timeouts when >>> resolving >>> > override anchors. >>> Internally IPA framework attempts to resolve ID override anchors. We can >>> actually optimize this code as ipaOriginalUID attribute contains >>> normalized name already, written their when the override is created and >>> never changed afterwards. This should speed up the resolution of large >>> overrides. >>> >>> Martin, can you file a ticket for that? The code in question is >>> baseidoverride.convert_anchor_to_human_readable_form() which could >>> benefit from passing in entry_attrs and checking ipaoriginaluid there. >>> If 'ipaoriginaluid' is missing, do analysis of ipaanchoruuid like it is >>> done now. >>> >> >> As an alternative I wonder if the WebUI can be made asynchronous in the >> sense that it displays the raw data (the SID in this case) first and run >> the lookups in the background and replace the SID when the name is >> available. I've seen some Windows tools behaving this when looking up >> large numbers of SIDs. >> > We have that for external group members already. > > However, in the ID overrides there is no SID as it is. The RDN of the ID > override is 'ipaAnchorUUID' attribute which is an encoding of a target > reference. We don't need to resolve it because we already have it > resolved in 'ipaOriginalUid' attribute -- this was done to help Schema > Compatibility and SASL mapping plugins which cannot resolve > ipaAnchorUUID value. We just need to make its use consistent across the > IPA framework. > > > -- > / Alexander Bokovoy > > -- > Manage your subscription for the Freeipa-users mailing list: > https://www.redhat.com/mailman/listinfo/freeipa-users > Go to http://freeipa.org for more info on the project > -------------- next part -------------- An HTML attachment was scrubbed... URL: From natxo.asenjo at gmail.com Tue Sep 20 09:49:58 2016 From: natxo.asenjo at gmail.com (Natxo Asenjo) Date: Tue, 20 Sep 2016 11:49:58 +0200 Subject: [Freeipa-users] certificates not renewing CA_UNREACHEABLE In-Reply-To: <57E003EB.3050409@redhat.com> References: <806bdf32-558c-4972-255a-0901f47c4c4b@redhat.com> <6596593a-9470-bebd-f236-f9d17aa4a56c@redhat.com> <57DC0009.4030602@redhat.com> <57E003EB.3050409@redhat.com> Message-ID: On Mon, Sep 19, 2016 at 5:27 PM, Rob Crittenden wrote: > Natxo Asenjo wrote: > >> hi, >> >> >> On Fri, Sep 16, 2016 at 4:22 PM, Rob Crittenden > > Ok, how about we work around the problem. > Gladly ;-) > Since it is failing on the revocation what you might try is removing the > userCertificate value from the ldap/kdc01.unix.iriszorg.nl service entry. > > I think this will work: > > $ ipa service-show ldap/kdc01.unix.iriszorg.nl |grep Serial > > > $ ipa service-mod --certificate= ldap/kdc01.unix.iriszorg.nl > > If this doesn't work you can use ldapmodify to delete the usercertificate > value. > > This will remove the certificate value so there is nothing to revoke and a > new cert will be saved (hopefully). > > Now try to resubmit the request via certmonger. > > It if works then you can run ipa cert-revooke > > It isn't a great answer long-term because it is really just working around > the problem but it should get the certs renewed. > > ok, so I restarted the httpd service then I could use ipa service-show: $ ipa service-show ldap/kdc01.unix.iriszorg.nl |grep Serial Serial Number: 175 Serial Number (hex): 0xAF bash-4.1$ ipa service-mod --certificate= ldap/kdc01.unix.iriszorg.nl --------------------------------------------------------------- Modified service "ldap/kdc01.unix.iriszorg.nl at UNIX.IRISZORG.NL" --------------------------------------------------------------- Principal: ldap/kdc01.unix.iriszorg.nl at UNIX.IRISZORG.NL Managed by: kdc01.unix.iriszorg.nl bash-4.1$ sudo ipa-getcert resubmit -i 20121107212513 Resubmitting "20121107212513" to "IPA". bash-4.1$ sudo getcert list Number of certificates and requests being tracked: 8. Request ID '20121107212513': status: CA_UNREACHABLE ca-error: Server failed request, will retry: 4301 (RPC failed at server. Certificate operation cannot be completed: Failure decoding Certificate Signing Request). stuck: yes key pair storage: type=NSSDB,location='/etc/dirsrv/slapd-UNIX-IRISZORG-NL',nickname='Server-Cert',token='NSS Certificate DB',pinfile='/etc/dirsrv/slapd-UNIX-IRISZORG-NL/pwdfile.txt' certificate: type=NSSDB,location='/etc/dirsrv/slapd-UNIX-IRISZORG-NL',nickname='Server-Cert',token='NSS Certificate DB' CA: IPA issuer: CN=Certificate Authority,O=UNIX.IRISZORG.NL subject: CN=kdc01.unix.iriszorg.nl,O=UNIX.IRISZORG.NL expires: 2016-10-12 10:49:24 UTC eku: id-kp-serverAuth,id-kp-clientAuth pre-save command: post-save command: /usr/lib/ipa/certmonger/restart_dirsrv UNIX-IRISZORG-NL track: yes auto-renew: yes the certificate is gone: $ ipa service-show ldap/kdc01.unix.iriszorg.nl ipa: ERROR: Could not create log_dir u'/home/jose.admin/.ipa/log' Principal: ldap/kdc01.unix.iriszorg.nl at UNIX.IRISZORG.NL Keytab: True Managed by: kdc01.unix.iriszorg.nl But then I thought, what the hell, let's try again, restarted httpd, resubmitted it, and now it did work ;-) $ ipa service-show ldap/kdc01.unix.iriszorg.nl Principal: ldap/kdc01.unix.iriszorg.nl at UNIX.IRISZORG.NL Certificate: MIIDrDCCApSgAwIBAgICAPUwDQYJKoZIhvcNAQELBQAwOzEZMBcGA1UEChMQVU5JWC5JUklTWk9SRy5OTDEeMBwGA1UEAxMVQ2VydGlmaWNhdGUgQXV0aG9yaXR5MB4XDTE2MDkyMDA4MDY1OFoXDTE4MDkyMTA4MDY1OFowPDEZMBcGA1UEChMQVU5JWC5JUklTWk9SRy5OTDEfMB0GA1UEAxMWa2RjMDEudW5peC5pcmlzem9yZy5ubDCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAO2QVqrFRb/Q5dhkAi7BK29BJhqTvbaH3bNDLvhe1snyChdlr/AIwrJj/53Ti2eJ7u1BtV7u3gSwQ3/xJ0HwUZmOEQHCNDrjcGy+iw7lqkC5NaZ8AGt8bSTGWwnJvEGWrb3uEJzVZf+xB5eZa8vFXr+Jlcfoq8DbVZhX274pmpVfQOnRckD+AmncuEItHpcJCCHneF0QzA5DQqlTPUFerFm3F/iI/k6g9XbHQaNejcUYdhXpy9q0mEuBIIsEzTeNWTTEsUYX5TPVEsN3x2feA0icxR6bUTeg2BqSu7ZOuM55iBp3l0d9UAQ7W7yh76FI/Bqz8vIMdS6VsurPS4asLa8CAwEAAaOBuDCBtTAfBgNVHSMEGDAWgBSjl+SKLrjPPuoz8ryT1iPeqYQ2aDBEBggrBgEFBQcBAQQ4MDYwNAYIKwYBBQUHMAGGKGh0dHA6Ly9rZGMwMS51bml4LmlyaXN6b3JnLm5sOjgwL2NhL29jc3AwDgYDVR0PAQH/BAQDAgTwMB0GA1UdJQQWMBQGCCsGAQUFBwMBBggrBgEFBQcDAjAdBgNVHQ4EFgQUBIRsG98GBkIyB/BgQKloUlLEJeEwDQYJKoZIhvcNAQELBQADggEBAHN+ggklVf2uzaePwEI9rMObe0WZeOyCLZxEtigDaJIHkq3GzkugxcG8ivD/LnuF0D8m07npfpIMC3QRUJQjFjz6E3rKtqau0QY0BO+Dwg1TzItQqXxgHtCqcQ7bmahj2AMPRNUXeZck0p/eueG4wj2kbLwTLU6cOfwnT4IOfszAS9GCql6oQIXlOfG6i6DAodBpgWziDfIrRJsJi4ZE+FvJL/ImJDdW+En50UyGp0n31oMSDIxWf1bdWUctSEYhcy9JftzkitNm1FD+a1HzeYyuHthzlHHcSIXN/kXRSGktpe8VHE5XLtKnH92vmkMnyxZvE///2+ExHXIAOkwq3ck= Keytab: True Managed by: kdc01.unix.iriszorg.nl Subject: CN=kdc01.unix.iriszorg.nl,O=UNIX.IRISZORG.NL Serial Number: 245 Serial Number (hex): 0xF5 Issuer: CN=Certificate Authority,O=UNIX.IRISZORG.NL Not Before: Tue Sep 20 08:06:58 2016 UTC Not After: Fri Sep 21 08:06:58 2018 UTC Fingerprint (MD5): f8:d3:cb:6f:4c:ca:e4:f3:47:65:51:d3:2c:69:84:df Fingerprint (SHA1): e3:0a:66:19:d7:36:fe:c4:ff:58:bf:90:35:3e:0b:31:cb:a0:58:37 So I could revoke the old one: $ ipa cert-revoke 175 Revoked: True and now getcert list shows the certificate is ok: Number of certificates and requests being tracked: 8. Request ID '20121107212513': status: MONITORING stuck: no key pair storage: type=NSSDB,location='/etc/dirsrv/slapd-UNIX-IRISZORG-NL',nickname='Server-Cert',token='NSS Certificate DB',pinfile='/etc/dirsrv/slapd-UNIX-IRISZORG-NL/pwdfile.txt' certificate: type=NSSDB,location='/etc/dirsrv/slapd-UNIX-IRISZORG-NL',nickname='Server-Cert',token='NSS Certificate DB' CA: IPA issuer: CN=Certificate Authority,O=UNIX.IRISZORG.NL subject: CN=kdc01.unix.iriszorg.nl,O=UNIX.IRISZORG.NL expires: 2018-09-21 08:06:58 UTC eku: id-kp-serverAuth,id-kp-clientAuth pre-save command: post-save command: /usr/lib/ipa/certmonger/restart_dirsrv UNIX-IRISZORG-NL track: yes auto-renew: yes So one down, two to go, it seems. -- Groeten, natxo -------------- next part -------------- An HTML attachment was scrubbed... URL: From g.kafataridis at nelios.com Tue Sep 20 09:51:16 2016 From: g.kafataridis at nelios.com (Giorgos Kafataridis) Date: Tue, 20 Sep 2016 12:51:16 +0300 Subject: [Freeipa-users] CA: Cannot add Centos7.2 replica to Centos6.8 ipa server [SOLVED] In-Reply-To: References: <45b66871-aa01-3c91-0630-0bbeee488ae7@nelios.com> <323b9de1-adb3-47af-d4a7-d6083f88d75c@nelios.com> <9afdf0f3-97d1-cc6e-5f3d-0e11687a124a@redhat.com> <05b8e986-ceef-8e75-c76d-b38c45e2e3a1@nelios.com> <79903d5c-2b80-d492-6b29-59ac5b39c6f5@redhat.com> <98b16feb-8b70-13bb-4c92-cd9bca3e408d@redhat.com> <0b546b4a-9ef3-7a15-27a7-3a2d59e5281f@redhat.com> <93f6dbfd-dff8-aa92-b540-cb7281fd29de@redhat.com> Message-ID: <5e9c1f1a-9728-936d-3786-688ff9687085@nelios.com> On 09/19/2016 03:51 PM, Giorgos Kafataridis wrote: > > > On 09/16/2016 06:39 PM, Petr Vobornik wrote: >> On 09/14/2016 07:26 PM, Giorgos Kafataridis wrote: >>> >>> On 09/13/2016 10:36 PM, Endi Sukma Dewata wrote: >>>> On 9/12/2016 9:35 PM, Endi Sukma Dewata wrote: >>>>> On 9/9/2016 2:46 PM, Georgios Kafataridis wrote: >>>>>> I've tried that but still the same result. >>>>>> >>>>>> [root at ipa-server /]# ldapsearch -D "cn=directory manager" -W -p >>>>>> 389 -h >>>>>> localhost -b "uid=admin,ou=people,o=ipaca" >>>>>> Enter LDAP Password: >>>>>> # extended LDIF >>>>>> # >>>>>> # LDAPv3 >>>>>> # base with scope subtree >>>>>> # filter: (objectclass=*) >>>>>> # requesting: ALL >>>>>> # >>>>>> >>>>>> # search result >>>>>> search: 2 >>>>>> result: 32 No such object >>>>> Hi, >>>>> >>>>> The master's logs indicate there's an authentication issue. >>>>> >>>>> Could you search the whole directory to find the admin user? >>>>> $ ldapsearch ... -b "o=ipaca" "(uid=admin)" >>>>> >>>>> Try also other suffixes that you have in the DS. >>>>> >>>>> If you find it, try to authenticate against DS directly as the admin >>>>> user. If the authentication fails, try resetting the password. >>>> I believe there is actually another DS instance on CentOS 6.8 >>>> running on port >>>> 7389, so make sure you check that too. If the admin user is indeed >>>> missing, it >>>> will need to be recreated, assigned a password and certificate, and >>>> added to >>>> the appropriate groups. >>>> >>>> See also: http://pki.fedoraproject.org/wiki/IPA_PKI_Users >>>> >>> Sorry for the delay, crazy office days. >>> >>> Ok, tried that and finally got a hit on the user. Indeed in 6.x you >>> also have >>> 7389 to look for. >>> >>> *Master >>> >>> *#ldapsearch -h localhost -p 7389 -b "o=ipaca" "(uid=admin)" -x -W >>> Enter LDAP Password: >>> >>> # extended LDIF >>> # >>> # LDAPv3 >>> # base with scope subtree >>> # filter: (uid=admin) >>> # requesting: ALL >>> # >>> >>> # admin, people, ipaca >>> dn: uid=admin,ou=people,o=ipaca >>> objectClass: top >>> objectClass: person >>> objectClass: organizationalPerson >>> objectClass: inetOrgPerson >>> objectClass: cmsuser >>> uid: admin >>> sn: admin >>> cn: admin >>> mail: root at localhost >>> usertype: adminType >>> userstate: 1 >>> description: 2;6;CN=Certificate >>> Authority,O=NELIOS;CN=ipa-ca-agent,O=NELIOS >>> userCertificate:: >>> MIIDaTCCAlGgAwIBAgIBBjANBgkqhkiG9w0BAQsFADAxMQ8wDQYDVQQKEwZO.... >>> . >>> . >>> . >>> . >>> # search result >>> search: 2 >>> result: 0 Success >>> >>> # numResponses: 2 >>> # numEntries: 1 >>> >>> >>> *Replica Server* >>> >>> [root at ipa2-server2 ~]# ldapsearch -h ipa-server.nelios -p 7389 -b >>> "o=ipaca" >>> "(uid=admin)" -x -W >>> # extended LDIF >>> # >>> # LDAPv3 >>> # base with scope subtree >>> # filter: (uid=admin) >>> # requesting: ALL >>> # >>> >>> # admin, people, ipaca >>> dn: uid=admin,ou=people,o=ipaca >>> objectClass: top >>> objectClass: person >>> objectClass: organizationalPerson >>> objectClass: inetOrgPerson >>> objectClass: cmsuser >>> uid: admin >>> sn: admin >>> cn: admin >>> mail: root at localhost >>> usertype: adminType >>> userstate: 1 >>> >>> Password is valid in both cases. >>> >>> So the user is there and can be retrieved from replica, assuming that >>> ipa-replica-install also tries 7389 the only thing I can try now is >>> "ipa >>> cert-request --uid admin" to create a new certificate, generate a >>> new cacert.p12 >>> and retry install ? >>> >>> >> In the other subthred there was a log from CentOS6 machine: >> >> /var/log/pki-ca/system >> >> 5337.TP-Processor3 - [09/Sep/2016:22:59:42 EEST] [6] [6] Failed to >> authenticate as admin UID=admin. Error: netscape.ldap.LDAPException: >> error result (49) >> 5337.TP-Processor3 - [09/Sep/2016:22:59:42 EEST] [3] [3] Servlet >> caGetCookie: Error getting servlet output stream when rendering >> template. Error Invalid Credential.. >> >> Which to me looks like a wrong password. Which indicates my original >> theory that IPA admin user shared with CA admin user the same password >> but it got out of sync. During replica installation it uses IPA admin >> user for authenticating as PKI admin user. >> >> If that is correct then changing PKI admin user's ( >> uid=admin,ou=people,o=ipaca ) password to IPA admin user's password >> might fix the issue. >> > Thanks! Ok, I will look into that more. In any case, I'll keep you > posted. > I had to follow https://www.freeipa.org/page/Howto/Change_Directory_Manager_Password#3._Update_PKI_admin_password and do it in my master and it worked at last***"ipa.ipapython.install.cli.install_tool(Replica): INFO The ipa-replica-install command was successful"!* -------------- next part -------------- An HTML attachment was scrubbed... URL: From deepak_dimri at hotmail.com Tue Sep 20 10:45:56 2016 From: deepak_dimri at hotmail.com (Deepak Dimri) Date: Tue, 20 Sep 2016 06:45:56 -0400 Subject: [Freeipa-users] IPA Server is not coming backup Message-ID: Hi All, My IPA Server was working all fine until i tried restarting it using "ipactl restart" and now i am ended with these errors :( [root at ip-172-31-25-165 plugins]# ipactl restartStarting Directory ServiceRestarting krb5kdc ServiceRestarting kadmin ServiceStarting named ServiceJob for named-pkcs11.service failed because the control process exited with error code. See "systemctl status named-pkcs11.service" and "journalctl -xe" for details.Failed to start named ServiceShutting down Aborting ipactl This is what i get with "systemctl status named-pkcs11.service" [root at ip-172-31-25-165 plugins]# systemctl status named-pkcs11.service? named-pkcs11.service - Berkeley Internet Name Domain (DNS) with native PKCS#11 Loaded: loaded (/usr/lib/systemd/system/named-pkcs11.service; disabled; vendor preset: disabled) Active: failed (Result: exit-code) since Tue 2016-09-20 06:28:03 EDT; 1min 2s ago Process: 3281 ExecStart=/usr/sbin/named-pkcs11 -u named $OPTIONS (code=exited, status=1/FAILURE) Process: 3278 ExecStartPre=/bin/bash -c if [ ! "$DISABLE_ZONE_CHECKING" == "yes" ]; then /usr/sbin/named-checkconf -z /etc/named.conf; else echo "Checking of zone files is disabled"; fi (code=exited, status=0/SUCCESS) Sep 20 06:28:03 ip-172-31-25-165.us-west-2.compute.internal named-pkcs11[3284]: GSSAPI Error: Unspecified GSS failure. Minor code may provide more information (Server krbtgt/US-WEST-2.C...database)Sep 20 06:28:03 ip-172-31-25-165.us-west-2.compute.internal named-pkcs11[3284]: LDAP error: Local error: SASL(-1): generic failure: GSSAPI Error: Unspecified GSS failure. Minor code may...er failedSep 20 06:28:03 ip-172-31-25-165.us-west-2.compute.internal named-pkcs11[3284]: couldn't establish connection in LDAP connection pool: failureSep 20 06:28:03 ip-172-31-25-165.us-west-2.compute.internal named-pkcs11[3284]: dynamic database 'ipa' configuration failed: failureSep 20 06:28:03 ip-172-31-25-165.us-west-2.compute.internal named-pkcs11[3284]: loading configuration: failureSep 20 06:28:03 ip-172-31-25-165.us-west-2.compute.internal named-pkcs11[3284]: exiting (due to fatal error)Sep 20 06:28:03 ip-172-31-25-165.us-west-2.compute.internal systemd[1]: named-pkcs11.service: control process exited, code=exited status=1Sep 20 06:28:03 ip-172-31-25-165.us-west-2.compute.internal systemd[1]: Failed to start Berkeley Internet Name Domain (DNS) with native PKCS#11.Sep 20 06:28:03 ip-172-31-25-165.us-west-2.compute.internal systemd[1]: Unit named-pkcs11.service entered failed state.Sep 20 06:28:03 ip-172-31-25-165.us-west-2.compute.internal systemd[1]: named-pkcs11.service failed. Hint: Some lines were ellipsized, use -l to show in full. output from "journalctl -xe" is as below: [root at ip-172-31-25-165 ec2-user]# journalctl -xeSep 20 06:37:00 ip-172-31-25-165.us-west-2.compute.internal named-pkcs11[3511]: option 'serial_autoincrement' is not supported, ignoringSep 20 06:37:00 ip-172-31-25-165.us-west-2.compute.internal named-pkcs11[3511]: GSSAPI client step 1Sep 20 06:37:00 ip-172-31-25-165.us-west-2.compute.internal named-pkcs11[3511]: GSSAPI client step 1Sep 20 06:37:00 ip-172-31-25-165.us-west-2.compute.internal named-pkcs11[3511]: GSSAPI Error: Unspecified GSS failure. Minor code may provide more information Sep 20 06:37:00 ip-172-31-25-165.us-west-2.compute.internal named-pkcs11[3511]: LDAP error: Local error: SASL(-1): generic failure: GSSAPI Error: Unspecified GSSep 20 06:37:00 ip-172-31-25-165.us-west-2.compute.internal named-pkcs11[3511]: couldn't establish connection in LDAP connection pool: failureSep 20 06:37:00 ip-172-31-25-165.us-west-2.compute.internal named-pkcs11[3511]: dynamic database 'ipa' configuration failed: failureSep 20 06:37:00 ip-172-31-25-165.us-west-2.compute.internal named-pkcs11[3511]: loading configuration: failureSep 20 06:37:00 ip-172-31-25-165.us-west-2.compute.internal named-pkcs11[3511]: exiting (due to fatal error)Sep 20 06:37:00 ip-172-31-25-165.us-west-2.compute.internal systemd[1]: named-pkcs11.service: control process exited, code=exited status=1Sep 20 06:37:00 ip-172-31-25-165.us-west-2.compute.internal systemd[1]: Failed to start Berkeley Internet Name Domain (DNS) with native PKCS#11.-- Subject: Unit named-pkcs11.service has failed-- Defined-By: systemd-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel-- -- Unit named-pkcs11.service has failed.-- -- The result is failed.Sep 20 06:37:00 ip-172-31-25-165.us-west-2.compute.internal systemd[1]: Unit named-pkcs11.service entered failed state.Sep 20 06:37:00 ip-172-31-25-165.us-west-2.compute.internal systemd[1]: named-pkcs11.service failed.Sep 20 06:37:00 ip-172-31-25-165.us-west-2.compute.internal polkitd[529]: Unregistered Authentication Agent for unix-process:3498:364279453 (system bus name :1.Sep 20 06:37:00 ip-172-31-25-165.us-west-2.compute.internal polkitd[529]: Registered Authentication Agent for unix-process:3518:364279465 (system bus name :1.96Sep 20 06:37:00 ip-172-31-25-165.us-west-2.compute.internal systemd[1]: Stopping 389 Directory Server US-WEST-2-COMPUTE-AMAZONAWS-COM....-- Subject: Unit dirsrv at US-WEST-2-COMPUTE-AMAZONAWS-COM.service has begun shutting down-- Defined-By: systemd-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel-- -- Unit dirsrv at US-WEST-2-COMPUTE-AMAZONAWS-COM.service has begun shutting down.Sep 20 06:37:05 ip-172-31-25-165.us-west-2.compute.internal systemd[1]: Stopped 389 Directory Server US-WEST-2-COMPUTE-AMAZONAWS-COM..-- Subject: Unit dirsrv at US-WEST-2-COMPUTE-AMAZONAWS-COM.service has finished shutting down-- Defined-By: systemd-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel-- -- Unit dirsrv at US-WEST-2-COMPUTE-AMAZONAWS-COM.service has finished shutting down.Sep 20 06:37:05 ip-172-31-25-165.us-west-2.compute.internal polkitd[529]: Unregistered Authentication Agent for unix-process:3518:364279465 (system bus name :1.Sep 20 06:37:15 ip-172-31-25-165.us-west-2.compute.internal unix_chkpwd[3531]: password check failed for user (root)Sep 20 06:37:15 ip-172-31-25-165.us-west-2.compute.internal sshd[3530]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=Sep 20 06:37:17 ip-172-31-25-165.us-west-2.compute.internal sshd[3528]: error: PAM: Authentication failure for root from 221.229.172.103Sep 20 06:37:18 ip-172-31-25-165.us-west-2.compute.internal unix_chkpwd[3533]: password check failed for user (root)Sep 20 06:37:18 ip-172-31-25-165.us-west-2.compute.internal sshd[3532]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=Sep 20 06:37:20 ip-172-31-25-165.us-west-2.compute.internal sshd[3528]: error: PAM: Authentication failure for root from 221.229.172.103Sep 20 06:37:20 ip-172-31-25-165.us-west-2.compute.internal unix_chkpwd[3535]: password check failed for user (root)Sep 20 06:37:20 ip-172-31-25-165.us-west-2.compute.internal sshd[3534]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=Sep 20 06:37:23 ip-172-31-25-165.us-west-2.compute.internal sshd[3528]: error: PAM: Authentication failure for root from 221.229.172.103 Sep 20 06:37:23 ip-172-31-25-165.us-west-2.compute.internal sshd[3528]: Received disconnect from 221.229.172.103: 11: [preauth] Would really be thankful if you can get me backup with my IPA Server.. Many Thanks,Deepak -------------- next part -------------- An HTML attachment was scrubbed... URL: From pspacek at redhat.com Tue Sep 20 11:21:47 2016 From: pspacek at redhat.com (Petr Spacek) Date: Tue, 20 Sep 2016 13:21:47 +0200 Subject: [Freeipa-users] IPA Server is not coming backup In-Reply-To: References: Message-ID: <85a15594-3cc7-7442-5912-f1d679ad6cc4@redhat.com> Hi, The important line is around > named-pkcs11[3511]: GSSAPI Error: Unspecified GSS failure. Minor code may provide more information Unfortunately the log is truncated so it does not show the actual error. Please see https://fedorahosted.org/bind-dyndb-ldap/wiki/BIND9/NamedCannotStart I hope it helps. Petr^2 Spacek On 20.9.2016 12:45, Deepak Dimri wrote: > Hi All, > My IPA Server was working all fine until i tried restarting it using "ipactl restart" and now i am ended with these errors :( > > > > > > > > > [root at ip-172-31-25-165 plugins]# ipactl restartStarting Directory ServiceRestarting krb5kdc ServiceRestarting kadmin ServiceStarting named ServiceJob for named-pkcs11.service failed because the control process exited with error code. See "systemctl status named-pkcs11.service" and "journalctl -xe" for details.Failed to start named ServiceShutting down > > > > > > > > > > > > > > > > Aborting ipactl > This is what i get with "systemctl status named-pkcs11.service" > [root at ip-172-31-25-165 plugins]# systemctl status named-pkcs11.service? named-pkcs11.service - Berkeley Internet Name Domain (DNS) with native PKCS#11 Loaded: loaded (/usr/lib/systemd/system/named-pkcs11.service; disabled; vendor preset: disabled) Active: failed (Result: exit-code) since Tue 2016-09-20 06:28:03 EDT; 1min 2s ago Process: 3281 ExecStart=/usr/sbin/named-pkcs11 -u named $OPTIONS (code=exited, status=1/FAILURE) Process: 3278 ExecStartPre=/bin/bash -c if [ ! "$DISABLE_ZONE_CHECKING" == "yes" ]; then /usr/sbin/named-checkconf -z /etc/named.conf; else echo "Checking of zone files is disabled"; fi (code=exited, status=0/SUCCESS) > Sep 20 06:28:03 ip-172-31-25-165.us-west-2.compute.internal named-pkcs11[3284]: GSSAPI Error: Unspecified GSS failure. Minor code may provide more information (Server krbtgt/US-WEST-2.C...database)Sep 20 06:28:03 ip-172-31-25-165.us-west-2.compute.internal named-pkcs11[3284]: LDAP error: Local error: SASL(-1): generic failure: GSSAPI Error: Unspecified GSS failure. Minor code may...er failedSep 20 06:28:03 ip-172-31-25-165.us-west-2.compute.internal named-pkcs11[3284]: couldn't establish connection in LDAP connection pool: failureSep 20 06:28:03 ip-172-31-25-165.us-west-2.compute.internal named-pkcs11[3284]: dynamic database 'ipa' configuration failed: failureSep 20 06:28:03 ip-172-31-25-165.us-west-2.compute.internal named-pkcs11[3284]: loading configuration: failureSep 20 06:28:03 ip-172-31-25-165.us-west-2.compute.internal named-pkcs11[3284]: exiting (due to fatal error)Sep 20 06:28:03 ip-172-31-25-165.us-west-2.compute.internal systemd[1]: named-pkcs11.service: control process exited, code=exited status=1Sep 20 06:28:03 ip-172-31-25-165.us-west-2.compute.internal systemd[1]: Failed to start Berkeley Internet Name Domain (DNS) with native PKCS#11.Sep 20 06:28:03 ip-172-31-25-165.us-west-2.compute.internal systemd[1]: Unit named-pkcs11.service entered failed state.Sep 20 06:28:03 ip-172-31-25-165.us-west-2.compute.internal systemd[1]: named-pkcs11.service failed. > > > > > > > > > > > > > > > > > > > > > > > > > Hint: Some lines were ellipsized, use -l to show in full. > output from "journalctl -xe" is as below: > [root at ip-172-31-25-165 ec2-user]# journalctl -xeSep 20 06:37:00 ip-172-31-25-165.us-west-2.compute.internal named-pkcs11[3511]: option 'serial_autoincrement' is not supported, ignoringSep 20 06:37:00 ip-172-31-25-165.us-west-2.compute.internal named-pkcs11[3511]: GSSAPI client step 1Sep 20 06:37:00 ip-172-31-25-165.us-west-2.compute.internal named-pkcs11[3511]: GSSAPI client step 1Sep 20 06:37:00 ip-172-31-25-165.us-west-2.compute.internal named-pkcs11[3511]: GSSAPI Error: Unspecified GSS failure. Minor code may provide more information Sep 20 06:37:00 ip-172-31-25-165.us-west-2.compute.internal named-pkcs11[3511]: LDAP error: Local error: SASL(-1): generic failure: GSSAPI Error: Unspecified GSSep 20 06:37:00 ip-172-31-25-165.us-west-2.compute.internal named-pkcs11[3511]: couldn't establish connection in LDAP connection pool: failureSep 20 06:37:00 ip-172-31-25-165.us-west-2.compute.internal named-pkcs11[3511]: dynamic database 'ipa' configuration failed: failureSep 20 06:37:00 ip-172-31-25-165.us-west-2.compute.internal named-pkcs11[3511]: loading configuration: failureSep 20 06:37:00 ip-172-31-25-165.us-west-2.compute.internal named-pkcs11[3511]: exiting (due to fatal error)Sep 20 06:37:00 ip-172-31-25-165.us-west-2.compute.internal systemd[1]: named-pkcs11.service: control process exited, code=exited status=1Sep 20 06:37:00 ip-172-31-25-165.us-west-2.compute.internal systemd[1]: Failed to start Berkeley Internet Name Domain (DNS) with native PKCS#11.-- Subject: Unit named-pkcs11.service has failed-- Defined-By: systemd-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel-- -- Unit named-pkcs11.service has failed.-- -- The result is failed.Sep 20 06:37:00 ip-172-31-25-165.us-west-2.compute.internal systemd[1]: Unit named-pkcs11.service entered failed state.Sep 20 06:37:00 ip-172-31-25-165.us-west-2.compute.internal systemd[1]: named-pkcs11.service failed.Sep 20 06:37:00 ip-172-31-25-165.us-west-2.compute.internal polkitd[529]: Unregistered Authentication Agent for unix-process:3498:36427945 3 (system bus name :1.Sep 20 06:37:00 ip-172-31-25-165.us-west-2.compute.internal polkitd[529]: Registered Authentication Agent for unix-process:3518:364279465 (system bus name :1.96Sep 20 06:37:00 ip-172-31-25-165.us-west-2.compute.internal systemd[1]: Stopping 389 Directory Server US-WEST-2-COMPUTE-AMAZONAWS-COM....-- Subject: Unit dirsrv at US-WEST-2-COMPUTE-AMAZONAWS-COM.service has begun shutting down-- Defined-By: systemd-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel-- -- Unit dirsrv at US-WEST-2-COMPUTE-AMAZONAWS-COM.service has begun shutting down.Sep 20 06:37:05 ip-172-31-25-165.us-west-2.compute.internal systemd[1]: Stopped 389 Directory Server US-WEST-2-COMPUTE-AMAZONAWS-COM..-- Subject: Unit dirsrv at US-WEST-2-COMPUTE-AMAZONAWS-COM.service has finished shutting down-- Defined-By: systemd-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel-- -- Unit dirsrv at US-WEST-2-COMPUTE-AMAZONAWS-COM.service has finished shutting down.Sep 20 06:37:05 ip-172-31-25-165.us-west-2.compute.internal polkitd[529]: Unregistered Authentication Agent for unix-process:3518:364279465 (system bus name :1.Sep 20 06:37:15 ip-172-31-25-165.us-west-2.compute.internal unix_chkpwd[3531]: password check failed for user (root)Sep 20 06:37:15 ip-172-31-25-165.us-west-2.compute.internal sshd[3530]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=Sep 20 06:37:17 ip-172-31-25-165.us-west-2.compute.internal sshd[3528]: error: PAM: Authentication failure for root from 221.229.172.103Sep 20 06:37:18 ip-172-31-25-165.us-west-2.compute.internal unix_chkpwd[3533]: password check failed for user (root)Sep 20 06:37:18 ip-172-31-25-165.us-west-2.compute.internal sshd[3532]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=Sep 20 06:37:20 ip-172-31-25-165.us-west-2.compute.internal sshd[3528]: error: PAM: Authentication failure for root from 221.229.172.103Sep 20 06:37:20 ip-172-31-25-165.us-west-2.compute.internal unix_chkpwd[3535]: pa ssword check failed for user (root)Sep 20 06:37:20 ip-172-31-25-165.us-west-2.compute.internal sshd[3534]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=Sep 20 06:37:23 ip-172-31-25-165.us-west-2.compute.internal sshd[3528]: error: PAM: Authentication failure for root from 221.229.172.103 > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Sep 20 06:37:23 ip-172-31-25-165.us-west-2.compute.internal sshd[3528]: Received disconnect from 221.229.172.103: 11: [preauth] > > Would really be thankful if you can get me backup with my IPA Server.. > Many Thanks,Deepak > > > > > > > > > > > -- Petr^2 Spacek From deepak_dimri at hotmail.com Tue Sep 20 11:28:27 2016 From: deepak_dimri at hotmail.com (Deepak Dimri) Date: Tue, 20 Sep 2016 07:28:27 -0400 Subject: [Freeipa-users] IPA Server is not coming backup In-Reply-To: References: Message-ID: Hi All, My IPA Server was working all fine until i tried restarting it using "ipactl restart" and now i am ended with these errors :( [root at ip-172-31-25-165 plugins]# ipactl restartStarting Directory ServiceRestarting krb5kdc ServiceRestarting kadmin ServiceStarting named ServiceJob for named-pkcs11.service failed because the control process exited with error code. See "systemctl status named-pkcs11.service" and "journalctl -xe" for details.Failed to start named ServiceShutting down Aborting ipactl This is what i get with "systemctl status named-pkcs11.service" [root at ip-172-31-25-165 plugins]# systemctl status named-pkcs11.service? named-pkcs11.service - Berkeley Internet Name Domain (DNS) with native PKCS#11 Loaded: loaded (/usr/lib/systemd/system/named-pkcs11.service; disabled; vendor preset: disabled) Active: failed (Result: exit-code) since Tue 2016-09-20 06:28:03 EDT; 1min 2s ago Process: 3281 ExecStart=/usr/sbin/named-pkcs11 -u named $OPTIONS (code=exited, status=1/FAILURE) Process: 3278 ExecStartPre=/bin/bash -c if [ ! "$DISABLE_ZONE_CHECKING" == "yes" ]; then /usr/sbin/named-checkconf -z /etc/named.conf; else echo "Checking of zone files is disabled"; fi (code=exited, status=0/SUCCESS) Sep 20 06:28:03 ip-172-31-25-165.us-west-2.compute.internal named-pkcs11[3284]: GSSAPI Error: Unspecified GSS failure. Minor code may provide more information (Server krbtgt/US-WEST-2.C...database)Sep 20 06:28:03 ip-172-31-25-165.us-west-2.compute.internal named-pkcs11[3284]: LDAP error: Local error: SASL(-1): generic failure: GSSAPI Error: Unspecified GSS failure. Minor code may...er failedSep 20 06:28:03 ip-172-31-25-165.us-west-2.compute.internal named-pkcs11[3284]: couldn't establish connection in LDAP connection pool: failureSep 20 06:28:03 ip-172-31-25-165.us-west-2.compute.internal named-pkcs11[3284]: dynamic database 'ipa' configuration failed: failureSep 20 06:28:03 ip-172-31-25-165.us-west-2.compute.internal named-pkcs11[3284]: loading configuration: failureSep 20 06:28:03 ip-172-31-25-165.us-west-2.compute.internal named-pkcs11[3284]: exiting (due to fatal error)Sep 20 06:28:03 ip-172-31-25-165.us-west-2.compute.internal systemd[1]: named-pkcs11.service: control process exited, code=exited status=1Sep 20 06:28:03 ip-172-31-25-165.us-west-2.compute.internal systemd[1]: Failed to start Berkeley Internet Name Domain (DNS) with native PKCS#11.Sep 20 06:28:03 ip-172-31-25-165.us-west-2.compute.internal systemd[1]: Unit named-pkcs11.service entered failed state.Sep 20 06:28:03 ip-172-31-25-165.us-west-2.compute.internal systemd[1]: named-pkcs11.service failed. Hint: Some lines were ellipsized, use -l to show in full. output from "journalctl -xe" is as below: [root at ip-172-31-25-165 ec2-user]# journalctl -xeSep 20 06:37:00 ip-172-31-25-165.us-west-2.compute.internal named-pkcs11[3511]: option 'serial_autoincrement' is not supported, ignoringSep 20 06:37:00 ip-172-31-25-165.us-west-2.compute.internal named-pkcs11[3511]: GSSAPI client step 1Sep 20 06:37:00 ip-172-31-25-165.us-west-2.compute.internal named-pkcs11[3511]: GSSAPI client step 1Sep 20 06:37:00 ip-172-31-25-165.us-west-2.compute.internal named-pkcs11[3511]: GSSAPI Error: Unspecified GSS failure. Minor code may provide more information Sep 20 06:37:00 ip-172-31-25-165.us-west-2.compute.internal named-pkcs11[3511]: LDAP error: Local error: SASL(-1): generic failure: GSSAPI Error: Unspecified GSSep 20 06:37:00 ip-172-31-25-165.us-west-2.compute.internal named-pkcs11[3511]: couldn't establish connection in LDAP connection pool: failureSep 20 06:37:00 ip-172-31-25-165.us-west-2.compute.internal named-pkcs11[3511]: dynamic database 'ipa' configuration failed: failureSep 20 06:37:00 ip-172-31-25-165.us-west-2.compute.internal named-pkcs11[3511]: loading configuration: failureSep 20 06:37:00 ip-172-31-25-165.us-west-2.compute.internal named-pkcs11[3511]: exiting (due to fatal error)Sep 20 06:37:00 ip-172-31-25-165.us-west-2.compute.internal systemd[1]: named-pkcs11.service: control process exited, code=exited status=1Sep 20 06:37:00 ip-172-31-25-165.us-west-2.compute.internal systemd[1]: Failed to start Berkeley Internet Name Domain (DNS) with native PKCS#11.-- Subject: Unit named-pkcs11.service has failed-- Defined-By: systemd-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel-- -- Unit named-pkcs11.service has failed.-- -- The result is failed.Sep 20 06:37:00 ip-172-31-25-165.us-west-2.compute.internal systemd[1]: Unit named-pkcs11.service entered failed state.Sep 20 06:37:00 ip-172-31-25-165.us-west-2.compute.internal systemd[1]: named-pkcs11.service failed.Sep 20 06:37:00 ip-172-31-25-165.us-west-2.compute.internal polkitd[529]: Unregistered Authentication Agent for unix-process:3498:364279453 (system bus name :1.Sep 20 06:37:00 ip-172-31-25-165.us-west-2.compute.internal polkitd[529]: Registered Authentication Agent for unix-process:3518:364279465 (system bus name :1.96Sep 20 06:37:00 ip-172-31-25-165.us-west-2.compute.internal systemd[1]: Stopping 389 Directory Server US-WEST-2-COMPUTE-AMAZONAWS-COM....-- Subject: Unit dirsrv at US-WEST-2-COMPUTE-AMAZONAWS-COM.service has begun shutting down-- Defined-By: systemd-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel-- -- Unit dirsrv at US-WEST-2-COMPUTE-AMAZONAWS-COM.service has begun shutting down.Sep 20 06:37:05 ip-172-31-25-165.us-west-2.compute.internal systemd[1]: Stopped 389 Directory Server US-WEST-2-COMPUTE-AMAZONAWS-COM..-- Subject: Unit dirsrv at US-WEST-2-COMPUTE-AMAZONAWS-COM.service has finished shutting down-- Defined-By: systemd-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel-- -- Unit dirsrv at US-WEST-2-COMPUTE-AMAZONAWS-COM.service has finished shutting down.Sep 20 06:37:05 ip-172-31-25-165.us-west-2.compute.internal polkitd[529]: Unregistered Authentication Agent for unix-process:3518:364279465 (system bus name :1.Sep 20 06:37:15 ip-172-31-25-165.us-west-2.compute.internal unix_chkpwd[3531]: password check failed for user (root)Sep 20 06:37:15 ip-172-31-25-165.us-west-2.compute.internal sshd[3530]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=Sep 20 06:37:17 ip-172-31-25-165.us-west-2.compute.internal sshd[3528]: error: PAM: Authentication failure for root from 221.229.172.103Sep 20 06:37:18 ip-172-31-25-165.us-west-2.compute.internal unix_chkpwd[3533]: password check failed for user (root)Sep 20 06:37:18 ip-172-31-25-165.us-west-2.compute.internal sshd[3532]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=Sep 20 06:37:20 ip-172-31-25-165.us-west-2.compute.internal sshd[3528]: error: PAM: Authentication failure for root from 221.229.172.103Sep 20 06:37:20 ip-172-31-25-165.us-west-2.compute.internal unix_chkpwd[3535]: password check failed for user (root)Sep 20 06:37:20 ip-172-31-25-165.us-west-2.compute.internal sshd[3534]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=Sep 20 06:37:23 ip-172-31-25-165.us-west-2.compute.internal sshd[3528]: error: PAM: Authentication failure for root from 221.229.172.103 Sep 20 06:37:23 ip-172-31-25-165.us-west-2.compute.internal sshd[3528]: Received disconnect from 221.229.172.103: 11: [preauth] Would really be thankful if you can get me backup with my IPA Server.. Many Thanks,Deepak -------------- next part -------------- An HTML attachment was scrubbed... URL: From gjn at gjn.priv.at Tue Sep 20 12:15:19 2016 From: gjn at gjn.priv.at (=?ISO-8859-1?Q?G=FCnther_J=2E?= Niederwimmer) Date: Tue, 20 Sep 2016 14:15:19 +0200 Subject: [Freeipa-users] 3rd party Cert install now IPA total broken In-Reply-To: References: <1825019.y8rA6b7sXH@techz> Message-ID: <1693814.UKm3IGFT8S@techz> Hello. Thanks for the first help, Am Montag, 19. September 2016, 12:02:19 schrieb Florence Blanc-Renaud: > On 09/16/2016 03:06 PM, G?nther J. Niederwimmer wrote: > > Hello, > > Freeipa 4.3.1 > > > > I have now install a 3rd Party Certificat from Startcom now my IPA is > > total > > broken? > > ipa-cacert-manage -p 'xxxxxxxxxxxxxxxx' -n STARTCOM-ROOT -t C,, install > > root.crt I mean this is the wrong cert I installed :-(. Is it possible to overwrite or delete and make it new. this file is the ROOT-CA from STARTCOM ("30 Years") > > ipa-certupdate > > > > ipa-server-certinstall -w -d ipa_3rd_ca.p12 This was wrong, I delete all this installed certs with Certutil -d . -D -n xxxxxxx > > I create this p12 with key.pem, cert.pem root.crt now i create a new p12 with I hope the correct certs I become from Startcom a httpd zip file with 1_root_bundle.crt ("15 Years")and my wild-card Certificate this I included in my new created p12 with my key.pem. This p12 I Installed on the first master with pk12util -v -i ipa_4gjn_ca.p12 -d /etc/httpd/alias -k /etc/httpd/alias/pwdfile.txt -W xxxxxxxx pk12util -v -i ipa_4gjn_ca.p12 -d /etc/dirsrv/slapd-4GJN-COM -k /etc/dirsrv/slapd-4GJN-COM/pwdfile.txt -W xxxxxxx and pk12util -v -i ipa_4gjn_ca.p12 -d /var/lib/pki/pki-tomcat/alias -k /etc/pki/pki-tomcat/pwdfile.txt -W xxxxxxxxx I change the nss.conf and I hope the correct file in /etc/dirsrv/slapd- XXXX/dsl.ldif Then I change in all NSS DB the StartCOM Cert (1_root_bundle.crt) with name STARTCOM-ROOT to certutil -d . -M -t C,, -n STARCOM-ROOT afterward I make a reboot and a test ipactl status Directory Service: RUNNING krb5kdc Service: RUNNING kadmin Service: RUNNING named Service: RUNNING ipa_memcached Service: RUNNING httpd Service: RUNNING ipa-custodia Service: RUNNING pki-tomcatd Service: RUNNING ipa-otpd Service: RUNNING ipa-ods-exporter Service: STOPPED ods-enforcerd Service: RUNNING ipa-dnskeysyncd Service: RUNNING ipa: INFO: The ipactl command was successful Why is ipa-ods-exporter Service always STOPPED ?? The next I Test a login on the Web UI from IPA, this is now also working ;-) the QUESTION is now what is with the second master and the IPA- clients Now (?) I have also found the ipa-backup ;-) OK, for the next Problem now I know it :-). Have I to repeat this all on the second Master ? and what is the correct way to inform the clients ? Thanks again for a answer, > Hi, > > there were some issues with ipa-server-certinstall (see tickets #4785, > #4786 and #6263). > In order to check your configuration, you must make sure that the NSS > DBs for Apache and the LDAP server (/etc/httpd/alias, > /var/lib/pki/pki-tomcat/alias, /etc/dirsrv/slapd-DOMxx) contain: > - the server certificate with flags u,u,u (= the one contained in > ipa_3rd_ca.p12) > - the certificate of the CA which signed the server certificate, with > flags C,, (= the one contained in root.rt) > > Then you can also check if the nickname for the server cert is properly > set in /etc/httpd/conf.d/nss.conf (in the directive NSSNickname), and in > the LDAP entry cn=RSA,cn=encryption,cn=config (in the attribute > nsSSLPersonalitySSL). > > If this doesn't fix the issue, the logs of pki-tomcat/ca/debug may > provide more information. > > Also note that it is important to run ipa-certupdate on all the clients > and replicas in order to install the new certificates in the NSS DBs > *before* you run ipa-server-certinstall. > > Hope this helps, > Flo. > > > the kerberos don't start anymore ? > > The Error Is > > > > Unspecified GSS failure.Minor (2529639068): Cannot contact any KDC for > > realm> > > '4GJN.COM' > > > > after insert in nss.conf > > "NSSEnforceValidCerts off" > > > > ipactl restart is starting (?) but > > > > ipactl status tell me > > Directory Service: RUNNING > > krb5kdc Service: RUNNING > > kadmin Service: RUNNING > > named Service: RUNNING > > ipa_memcached Service: RUNNING > > httpd Service: RUNNING > > ipa-custodia Service: RUNNING > > pki-tomcatd Service: RUNNING > > ipa-otpd Service: RUNNING > > ipa-ods-exporter Service: STOPPED > > ods-enforcerd Service: RUNNING > > ipa-dnskeysyncd Service: RUNNING > > ipa: INFO: The ipactl command was successful > > > > with certutil -d /etc/httpd/alias -L I have now this > > Certificate Nickname Trust > > Attributes> > > SSL,S/MIME,JA > > R/XPI > > > > Signing-Cert u,u,u > > 4GJN_CA_FILE u,u,u > > ipaCert u,u,u > > 4GJN.COM IPA CA CT,C,C > > STARTCOM-ROOT C,, > > > > I can Insert in nss.conf by the > > #NSSNickname "Signing-Cert" original > > or > > NSSNickname 4GJN_CA_FILE but all is now broken ? > > > > I also add this, found in Bugzilla > > > > certutil -d /var/lib/pki/pki-tomcat/alias -L > > > > Certificate Nickname Trust > > Attributes> > > SSL,S/MIME,JA > > R/XPI > > > > ocspSigningCert cert-pki-ca u,u,u > > subsystemCert cert-pki-ca u,u,u > > caSigningCert cert-pki-ca CTu,Cu,Cu > > Server-Cert cert-pki-ca u,u,u > > auditSigningCert cert-pki-ca u,u,Pu > > STARTCOM-ROOT CT,, > > > > this is created in the > > > > certutil -d /etc/dirsrv/slapd-4GJN.COM -L > > > > Certificate Nickname Trust > > Attributes> > > SSL,S/MIME,JA > > R/XPI > > > > 4GJN_CA_FILE u,u,u > > 4GJN.COM IPA CA CT,C,C > > STARTCOM-ROOT C,, > > > > Can any help a little, please ;-) > > > > The bad Problem, I tested this with my master server with DNS / DNSSEC I > > can't new install (DNSSEC Keys) -- mit freundlichen Gr??en / best regards, G?nther J. Niederwimmer From natxo.asenjo at gmail.com Tue Sep 20 12:36:17 2016 From: natxo.asenjo at gmail.com (Natxo Asenjo) Date: Tue, 20 Sep 2016 14:36:17 +0200 Subject: [Freeipa-users] certificates not renewing CA_UNREACHEABLE In-Reply-To: References: <806bdf32-558c-4972-255a-0901f47c4c4b@redhat.com> <6596593a-9470-bebd-f236-f9d17aa4a56c@redhat.com> <57DC0009.4030602@redhat.com> <57E003EB.3050409@redhat.com> Message-ID: ok, so all certs are renewed (dogldap and http). On Tue, Sep 20, 2016 at 11:49 AM, Natxo Asenjo wrote: > > > On Mon, Sep 19, 2016 at 5:27 PM, Rob Crittenden > wrote: > >> Natxo Asenjo wrote: >> >>> hi, >>> >>> >>> On Fri, Sep 16, 2016 at 4:22 PM, Rob Crittenden >> >> Ok, how about we work around the problem. >> > > Gladly ;-) > > >> Since it is failing on the revocation what you might try is removing the >> userCertificate value from the ldap/kdc01.unix.iriszorg.nl service entry. >> >> I think this will work: >> >> $ ipa service-show ldap/kdc01.unix.iriszorg.nl |grep Serial >> >> >> $ ipa service-mod --certificate= ldap/kdc01.unix.iriszorg.nl >> >> If this doesn't work you can use ldapmodify to delete the usercertificate >> value. >> >> This will remove the certificate value so there is nothing to revoke and >> a new cert will be saved (hopefully). >> >> Now try to resubmit the request via certmonger. >> >> It if works then you can run ipa cert-revooke >> >> It isn't a great answer long-term because it is really just working >> around the problem but it should get the certs renewed. >> >> > ok, so I restarted the httpd service then I could use ipa service-show: > > $ ipa service-show ldap/kdc01.unix.iriszorg.nl |grep Serial > Serial Number: 175 > Serial Number (hex): 0xAF > bash-4.1$ ipa service-mod --certificate= ldap/kdc01.unix.iriszorg.nl > --------------------------------------------------------------- > Modified service "ldap/kdc01.unix.iriszorg.nl at UNIX.IRISZORG.NL" > --------------------------------------------------------------- > Principal: ldap/kdc01.unix.iriszorg.nl at UNIX.IRISZORG.NL > Managed by: kdc01.unix.iriszorg.nl > > > bash-4.1$ sudo ipa-getcert resubmit -i 20121107212513 > Resubmitting "20121107212513" to "IPA". > bash-4.1$ sudo getcert list > Number of certificates and requests being tracked: 8. > Request ID '20121107212513': > status: CA_UNREACHABLE > ca-error: Server failed request, will retry: 4301 (RPC failed at > server. Certificate operation cannot be completed: Failure decoding > Certificate Signing Request). > stuck: yes > key pair storage: type=NSSDB,location='/etc/ > dirsrv/slapd-UNIX-IRISZORG-NL',nickname='Server-Cert',token='NSS > Certificate DB',pinfile='/etc/dirsrv/slapd-UNIX-IRISZORG-NL/pwdfile.txt' > certificate: type=NSSDB,location='/etc/ > dirsrv/slapd-UNIX-IRISZORG-NL',nickname='Server-Cert',token='NSS > Certificate DB' > CA: IPA > issuer: CN=Certificate Authority,O=UNIX.IRISZORG.NL > subject: CN=kdc01.unix.iriszorg.nl,O=UNIX.IRISZORG.NL > expires: 2016-10-12 10:49:24 UTC > eku: id-kp-serverAuth,id-kp-clientAuth > pre-save command: > post-save command: /usr/lib/ipa/certmonger/restart_dirsrv > UNIX-IRISZORG-NL > track: yes > auto-renew: yes > > > > the certificate is gone: > $ ipa service-show ldap/kdc01.unix.iriszorg.nl > ipa: ERROR: Could not create log_dir u'/home/jose.admin/.ipa/log' > Principal: ldap/kdc01.unix.iriszorg.nl at UNIX.IRISZORG.NL > Keytab: True > Managed by: kdc01.unix.iriszorg.nl > > > But then I thought, what the hell, let's try again, restarted httpd, > resubmitted it, and now it did work ;-) > > $ ipa service-show ldap/kdc01.unix.iriszorg.nl > Principal: ldap/kdc01.unix.iriszorg.nl at UNIX.IRISZORG.NL > Certificate: MIIDrDCCApSgAwIBAgICAPUwDQYJKo > ZIhvcNAQELBQAwOzEZMBcGA1UEChMQVU5JWC5JUklTWk9SRy5OTDEeMBwGA1 > UEAxMVQ2VydGlmaWNhdGUgQXV0aG9yaXR5MB4XDTE2MDkyMDA4MDY1OFoXDT > E4MDkyMTA4MDY1OFowPDEZMBcGA1UEChMQVU5JWC5JUklTWk9SRy5OTDEfMB > 0GA1UEAxMWa2RjMDEudW5peC5pcmlzem9yZy5ubDCCASIwDQYJKoZIhvcNAQ > EBBQADggEPADCCAQoCggEBAO2QVqrFRb/Q5dhkAi7BK29BJhqTvbaH3bNDLvhe1 > snyChdlr/AIwrJj/53Ti2eJ7u1BtV7u3gSwQ3/xJ0HwUZmOEQHCNDrjcGy+ > iw7lqkC5NaZ8AGt8bSTGWwnJvEGWrb3uEJzVZf+xB5eZa8vFXr+ > Jlcfoq8DbVZhX274pmpVfQOnRckD+AmncuEItHpcJCCHneF0QzA5DQqlTPUFerFm3F/iI/ > k6g9XbHQaNejcUYdhXpy9q0mEuBIIsEzTeNWTTEsUYX5TPVEsN3x2feA0icx > R6bUTeg2BqSu7ZOuM55iBp3l0d9UAQ7W7yh76FI/Bqz8vIMdS6VsurPS4asLa8CAwEAAaO > BuDCBtTAfBgNVHSMEGDAWgBSjl+SKLrjPPuoz8ryT1iPeqYQ2aDBEBggr > BgEFBQcBAQQ4MDYwNAYIKwYBBQUHMAGGKGh0dHA6Ly9rZGMwMS51bml4Lmly > aXN6b3JnLm5sOjgwL2NhL29jc3AwDgYDVR0PAQH/BAQDAgTwMB0GA1UdJQQWMBQGCCsGAQ > UFBwMBBggrBgEFBQcDAjAdBgNVHQ4EFgQUBIRsG98GBkIyB/ > BgQKloUlLEJeEwDQYJKoZIhvcNAQELBQADggEBAHN+ggklVf2uzaePwEI9rMObe0WZeOyCLZ > xEtigDaJIHkq3GzkugxcG8ivD/LnuF0D8m07npfpIMC3QRUJQjFjz6E3rKtqau0QY0BO+ > Dwg1TzItQqXxgHtCqcQ7bmahj2AMPRNUXeZck0p/eueG4wj2kbLwTLU6cOfwnT4IOfszAS > 9GCql6oQIXlOfG6i6DAodBpgWziDfIrRJsJi4ZE+FvJL/ImJDdW+ > En50UyGp0n31oMSDIxWf1bdWUctSEYhcy9JftzkitNm1FD+a1HzeYyuHthzlHHcSIXN/ > kXRSGktpe8VHE5XLtKnH92vmkMnyxZvE///2+ExHXIAOkwq3ck= > Keytab: True > Managed by: kdc01.unix.iriszorg.nl > Subject: CN=kdc01.unix.iriszorg.nl,O=UNIX.IRISZORG.NL > Serial Number: 245 > Serial Number (hex): 0xF5 > Issuer: CN=Certificate Authority,O=UNIX.IRISZORG.NL > Not Before: Tue Sep 20 08:06:58 2016 UTC > Not After: Fri Sep 21 08:06:58 2018 UTC > Fingerprint (MD5): f8:d3:cb:6f:4c:ca:e4:f3:47:65:51:d3:2c:69:84:df > Fingerprint (SHA1): e3:0a:66:19:d7:36:fe:c4:ff:58: > bf:90:35:3e:0b:31:cb:a0:58:37 > > So I could revoke the old one: > > $ ipa cert-revoke 175 > Revoked: True > > > and now getcert list shows the certificate is ok: > > Number of certificates and requests being tracked: 8. > Request ID '20121107212513': > status: MONITORING > stuck: no > key pair storage: type=NSSDB,location='/etc/ > dirsrv/slapd-UNIX-IRISZORG-NL',nickname='Server-Cert',token='NSS > Certificate DB',pinfile='/etc/dirsrv/slapd-UNIX-IRISZORG-NL/pwdfile.txt' > certificate: type=NSSDB,location='/etc/ > dirsrv/slapd-UNIX-IRISZORG-NL',nickname='Server-Cert',token='NSS > Certificate DB' > CA: IPA > issuer: CN=Certificate Authority,O=UNIX.IRISZORG.NL > subject: CN=kdc01.unix.iriszorg.nl,O=UNIX.IRISZORG.NL > expires: 2018-09-21 08:06:58 UTC > eku: id-kp-serverAuth,id-kp-clientAuth > pre-save command: > post-save command: /usr/lib/ipa/certmonger/restart_dirsrv > UNIX-IRISZORG-NL > track: yes > auto-renew: yes > > > So one down, two to go, it seems. > > > > > -- > Groeten, > natxo > -- -- Groeten, natxo -------------- next part -------------- An HTML attachment was scrubbed... URL: From larry.rosen at JDRSolutions.com Tue Sep 20 14:03:38 2016 From: larry.rosen at JDRSolutions.com (Larry Rosen) Date: Tue, 20 Sep 2016 14:03:38 +0000 Subject: [Freeipa-users] login auth fails then success Message-ID: <79B7CEE400C91A4C9FD8BF082D8226073E379115@JDRPDC.JDRSolutions.local> Thanks, that explains a lot (I didn't catch the difference in auth services). Would this be mitigated by putting sss in front of files in nsswitch.conf)? /etc/nsswitchconf: passwd: files sss shadow: files sss group: files sss Date: Sun, 18 Sep 2016 22:14:59 +0200 From: Jakub Hrozek To: freeipa-users at redhat.com Subject: Re: [Freeipa-users] login auth fails then success Message-ID: <20160918201459.uhijnc4gyfykgzic at hendrix> Content-Type: text/plain; charset=us-ascii On Fri, Sep 16, 2016 at 06:23:03PM +0000, Larry Rosen wrote: > Sorry I thought I had pasted these previously: > > What other logs do I need to add (maybe from the IPA server)? > > Client system's /var/log/secure: > > Sep 13 19:12:33 il10-app-xfs udcs: pam_unix(login:auth): > authentication failure; logname= uid=0 euid=0 tty= ruser= rhost= > user=il10web Sep 13 19:12:33 il10-app-xfs udcs: pam_sss(login:auth): > authentication success; logname= uid=0 euid=0 tty= ruser= rhost= > user=il10web Sep 13 19:18:11 il10-app-xfs udcs: pam_unix(login:auth): > authentication failure; logname= uid=0 euid=0 tty= ruser= rhost= > user=il10web Sep 13 19:18:11 il10-app-xfs udcs: pam_sss(login:auth): > authentication success; logname= uid=0 euid=0 tty= ruser= rhost= > user=il10web Sep 13 19:22:52 il10-app-xfs udcs: pam_unix(login:auth): > authentication failure; logname= uid=0 euid=0 tty= ruser= rhost= > user=il10web Sep 13 19:22:53 il10-app-xfs udcs: pam_sss(login:auth): > authentication success; logname= uid=0 euid=0 tty= ruser= rhost= > user=il10web Sep 13 19:23:49 il10-app-xfs udcs: pam_unix(login:auth): > authentication failure; logname= uid=0 euid=0 tty= ruser= rhost= > user=il10web Sep 13 19:23:49 il10-app-xfs udcs: pam_sss(login:auth): > authentication success; logname= uid=0 euid=0 tty= ruser= rhost= > user=il10web Sep 13 19:28:24 il10-app-xfs udcs: pam_unix(login:auth): > authentication failure; logname= uid=0 euid=0 tty= ruser= rhost= > user=il10web Sep 13 19:28:24 il10-app-xfs udcs: pam_sss(login:auth): > authentication success; logname= uid=0 euid=0 tty= ruser= rhost= > user=il10web Sep 13 19:29:27 il10-app-xfs udcs: pam_unix(login:auth): > authentication failure; logname= uid=0 euid=0 tty= ruser= rhost= > user=il10web Sep 13 19:29:27 il10-app-xfs udcs: pam_sss(login:auth): > authentication success; logname= uid=0 euid=0 tty= ruser= rhost= > user=il10web I think these are expected. Authentication using pam_unix fails because pam_unix doesn't know this particular users and then pam_sss succeeds. I wonder if the best way to deal with the log messages is just to configure logrotate a bit more aggressively? > > -----Original Message----- > From: Rob Crittenden [mailto:rcritten at redhat.com] > Sent: Friday, September 16, 2016 1:39 PM > To: Larry Rosen ; > freeipa-users at redhat.com > Subject: Re: [Freeipa-users] login auth fails then success > > Larry Rosen wrote: > > We have a web app that logs in using a service (automated login > > user, non-expiring, non-failure count) account that leaves these log > > entries all day long. This does not appear to cause any problems, > > it just make my logs grow unnecessarily and creates a lot of "noise" in the log. > > > > Any ideas why it initially fails and then works?** > > Logs where? Can we see them? > > rob > > > -- > Manage your subscription for the Freeipa-users mailing list: > https://www.redhat.com/mailman/listinfo/freeipa-users > Go to http://freeipa.org for more info on the project From jhrozek at redhat.com Tue Sep 20 14:41:20 2016 From: jhrozek at redhat.com (Jakub Hrozek) Date: Tue, 20 Sep 2016 16:41:20 +0200 Subject: [Freeipa-users] login auth fails then success In-Reply-To: <79B7CEE400C91A4C9FD8BF082D8226073E379115@JDRPDC.JDRSolutions.local> References: <79B7CEE400C91A4C9FD8BF082D8226073E379115@JDRPDC.JDRSolutions.local> Message-ID: <20160920144120.xyloiqjg2j6qikne@hendrix> On Tue, Sep 20, 2016 at 02:03:38PM +0000, Larry Rosen wrote: > Thanks, that explains a lot (I didn't catch the difference in auth services). > Would this be mitigated by putting sss in front of files in nsswitch.conf)? > > /etc/nsswitchconf: > passwd: files sss > shadow: files sss > group: files sss No, NSS is a separate interface. You can experiment with adding pam_localuser.so before pam_unix, though. btw this is how recent Fedora releases configure their PAM stack: auth required pam_env.so auth sufficient pam_fprintd.so auth [default=1 success=ok] pam_localuser.so auth [success=done ignore=ignore default=die] pam_unix.so nullok try_first_pass auth requisite pam_succeed_if.so uid >= 1000 quiet_success auth sufficient pam_sss.so forward_pass auth required pam_deny.so But watch out, PAM stacks are inherently distro-specific and I don't remember what exactly you're running. From ianh at brownpapertickets.com Tue Sep 20 18:51:08 2016 From: ianh at brownpapertickets.com (Ian Harding) Date: Tue, 20 Sep 2016 11:51:08 -0700 Subject: [Freeipa-users] Me Again Message-ID: <722fd8d1-5ede-12dc-45d6-952be7e53dcd@brownpapertickets.com> I used to have a lot of replicas, but like a house of cards, it all came crashing down. I was down to two, that seemed to be replicating, but last few days I've noticed that they haven't always been. freeipa-sea.bpt.rocks is where we do all our admin. seattlenfs.bpt.rocks is also up and running and can be used for authentication. When I noticed that logins were failing after password changes I did ipa-replica-manage re-initialize --from=freeipa-sea.bpt.rocks on seattlenfs.bpt.rocks and replication appeared to be working again. Well it happened again, and this time I peeked at the dirsrv errors log and saw some scary things having to do with the CA. [19/Sep/2016:02:55:50 -0700] slapd_ldap_sasl_interactive_bind - Error: could not perform interactive bind for id [] mech [GSSAPI]: LDAP error -1 (Can't contact LDAP server) ((null)) errno 0 (Success) [19/Sep/2016:02:55:50 -0700] slapi_ldap_bind - Error: could not perform interactive bind for id [] authentication mechanism [GSSAPI]: error -1 (Can't contact LDAP server) [19/Sep/2016:02:55:50 -0700] NSMMReplicationPlugin - agmt="cn=meTofreeipa-sea.bpt.rocks" (freeipa-sea:389): Replication bind with GSSAPI auth failed: LDAP error -1 (Can't contact LDAP server) () [19/Sep/2016:02:56:04 -0700] NSMMReplicationPlugin - agmt="cn=meTofreeipa-sea.bpt.rocks" (freeipa-sea:389): Replication bind with GSSAPI auth resumed [20/Sep/2016:10:18:25 -0700] NSMMReplicationPlugin - multimaster_be_state_change: replica dc=bpt,dc=rocks is going offline; disabling replication [20/Sep/2016:10:18:26 -0700] - WARNING: Import is running with nsslapd-db-private-import-mem on; No other process is allowed to access the database [20/Sep/2016:10:18:29 -0700] - import userRoot: Workers finished; cleaning up... [20/Sep/2016:10:18:29 -0700] - import userRoot: Workers cleaned up. [20/Sep/2016:10:18:29 -0700] - import userRoot: Indexing complete. Post-processing... [20/Sep/2016:10:18:29 -0700] - import userRoot: Generating numsubordinates (this may take several minutes to complete)... [20/Sep/2016:10:18:29 -0700] - import userRoot: Generating numSubordinates complete. [20/Sep/2016:10:18:29 -0700] - import userRoot: Gathering ancestorid non-leaf IDs... [20/Sep/2016:10:18:29 -0700] - import userRoot: Finished gathering ancestorid non-leaf IDs. [20/Sep/2016:10:18:29 -0700] - import userRoot: Creating ancestorid index (new idl)... [20/Sep/2016:10:18:29 -0700] - import userRoot: Created ancestorid index (new idl). [20/Sep/2016:10:18:29 -0700] - import userRoot: Flushing caches... [20/Sep/2016:10:18:29 -0700] - import userRoot: Closing files... [20/Sep/2016:10:18:29 -0700] - import userRoot: Import complete. Processed 1324 entries in 3 seconds. (441.33 entries/sec) [20/Sep/2016:10:18:29 -0700] NSMMReplicationPlugin - multimaster_be_state_change: replica dc=bpt,dc=rocks is coming online; enabling replication [20/Sep/2016:10:18:29 -0700] NSMMReplicationPlugin - replica_reload_ruv: Warning: new data for replica dc=bpt,dc=rocks does not match the data in the changelog. Recreating the changelog file. This could affect replication with replica's consumers in which case the consumers should be reinitialized. [20/Sep/2016:10:18:29 -0700] NSACLPlugin - The ACL target cn=groups,cn=compat,dc=bpt,dc=rocks does not exist [20/Sep/2016:10:18:29 -0700] NSACLPlugin - The ACL target cn=computers,cn=compat,dc=bpt,dc=rocks does not exist [20/Sep/2016:10:18:29 -0700] NSACLPlugin - The ACL target cn=ng,cn=compat,dc=bpt,dc=rocks does not exist [20/Sep/2016:10:18:29 -0700] NSACLPlugin - The ACL target ou=sudoers,dc=bpt,dc=rocks does not exist [20/Sep/2016:10:18:29 -0700] NSACLPlugin - The ACL target cn=users,cn=compat,dc=bpt,dc=rocks does not exist [20/Sep/2016:10:18:29 -0700] NSACLPlugin - The ACL target cn=vaults,cn=kra,dc=bpt,dc=rocks does not exist [20/Sep/2016:10:18:29 -0700] NSACLPlugin - The ACL target cn=vaults,cn=kra,dc=bpt,dc=rocks does not exist [20/Sep/2016:10:18:29 -0700] NSACLPlugin - The ACL target cn=vaults,cn=kra,dc=bpt,dc=rocks does not exist [20/Sep/2016:10:18:29 -0700] NSACLPlugin - The ACL target cn=vaults,cn=kra,dc=bpt,dc=rocks does not exist [20/Sep/2016:10:18:29 -0700] NSACLPlugin - The ACL target cn=vaults,cn=kra,dc=bpt,dc=rocks does not exist [20/Sep/2016:10:18:29 -0700] NSACLPlugin - The ACL target cn=vaults,cn=kra,dc=bpt,dc=rocks does not exist [20/Sep/2016:10:18:29 -0700] NSACLPlugin - The ACL target cn=vaults,cn=kra,dc=bpt,dc=rocks does not exist [20/Sep/2016:10:18:29 -0700] NSACLPlugin - The ACL target cn=vaults,cn=kra,dc=bpt,dc=rocks does not exist [20/Sep/2016:10:18:29 -0700] NSACLPlugin - The ACL target cn=vaults,cn=kra,dc=bpt,dc=rocks does not exist [20/Sep/2016:10:18:29 -0700] NSACLPlugin - The ACL target cn=vaults,cn=kra,dc=bpt,dc=rocks does not exist [20/Sep/2016:10:18:29 -0700] NSACLPlugin - The ACL target cn=vaults,cn=kra,dc=bpt,dc=rocks does not exist [20/Sep/2016:10:18:29 -0700] NSACLPlugin - The ACL target cn=ad,cn=etc,dc=bpt,dc=rocks does not exist [20/Sep/2016:10:18:29 -0700] NSACLPlugin - The ACL target cn=casigningcert cert-pki-ca,cn=ca_renewal,cn=ipa,cn=etc,dc=bpt,dc=rocks does not exist [20/Sep/2016:10:18:29 -0700] NSACLPlugin - The ACL target cn=casigningcert cert-pki-ca,cn=ca_renewal,cn=ipa,cn=etc,dc=bpt,dc=rocks does not exist Any clues what that's about? The high numbered RUV are for CA RUV I assume. Those other machines are still installed but IPA services turned off so people can't authenticate to them because replication was not working. If there was some way I could go down to one machine (freeipa-sea) and get it all cleaned up, no ghost RUV, everything quiet in the logs, and start over creating replicas, I would love to do that. Seems like someone smarter than me could stop the server, back up the ldif files and edit them to make all the cruft go away. Is that possible? I've started a conversation with RedHat about getting on board with the official bits and support but I want to know if it's possible/cost effective to do what I describe, along with, I assume, migrating to the official versions of Spacewalk and FreeIPA. Thanks! Ian -- Ian Harding IT Director Brown Paper Tickets 1-800-838-3006 ext 7186 http://www.brownpapertickets.com From deepak_dimri at hotmail.com Wed Sep 21 06:41:23 2016 From: deepak_dimri at hotmail.com (Deepak Dimri) Date: Wed, 21 Sep 2016 02:41:23 -0400 Subject: [Freeipa-users] FreeIPA client installation on ubuntu 14.04 Message-ID: Hi All, I am trying to install freeipa client on my ubuntu client via ansible script. I have "apt-get update" and "apt-get install freeipa-client -y" these basic commands added in my playbook but the problem is when i run "apt-get install freeipa-client" with or without -y option it opens up some graphical interface confirming the IPA realm and other details. I did not find any option with in "apt-get install freeipa-client"to make it deployment unattended. Can anyone please tell me the how i can automate ipa-client installation on ubuntu? The same process works fine with RHEL using yum but i am unable to do so for ubuntu with apt-get Thanks,Deepak -------------- next part -------------- An HTML attachment was scrubbed... URL: From flo at redhat.com Wed Sep 21 07:04:50 2016 From: flo at redhat.com (Florence Blanc-Renaud) Date: Wed, 21 Sep 2016 09:04:50 +0200 Subject: [Freeipa-users] 3rd party Cert install now IPA total broken In-Reply-To: <1693814.UKm3IGFT8S@techz> References: <1825019.y8rA6b7sXH@techz> <1693814.UKm3IGFT8S@techz> Message-ID: On 09/20/2016 02:15 PM, G?nther J. Niederwimmer wrote: > Hello. > > Thanks for the first help, > > Am Montag, 19. September 2016, 12:02:19 schrieb Florence Blanc-Renaud: >> On 09/16/2016 03:06 PM, G?nther J. Niederwimmer wrote: >>> Hello, >>> Freeipa 4.3.1 >>> >>> I have now install a 3rd Party Certificat from Startcom now my IPA is >>> total >>> broken? > >>> ipa-cacert-manage -p 'xxxxxxxxxxxxxxxx' -n STARTCOM-ROOT -t C,, install >>> root.crt > > I mean this is the wrong cert I installed :-(. > > Is it possible to overwrite or delete and make it new. this file is the ROOT-CA > from STARTCOM ("30 Years") > Hi, ipa-cacert-manage install *adds* the CA certificate to the list of CA certs (it does not replace the CA cert), meaning that it can be run multiple times with different certificates. After this step, you can find all your CA certificates in the ldap server, below cn=certificates,cn=ipa,cn=etc,$BASEDN So in your case, you can re-run this command, this time with the right CA cert. Then do not forget to run ipa-certupdate on all the ipa replicas/clients in order to install the new CA cert on the relevant NSS databases. It is important to run ipa-certupdate before IPA services are restarted with the new certs (otherwise ipa-certupdate cannot contact the LDAP server to download the new certificates). If you forgot to run ipa-certupdate on the clients, I guess you can fix this by installing the new CA cert in /etc/ipa/nssdb with C,, flags. HTH, Flo >>> ipa-certupdate >>> >>> ipa-server-certinstall -w -d ipa_3rd_ca.p12 > > This was wrong, I delete all this installed certs with > Certutil -d . -D -n xxxxxxx > >>> I create this p12 with key.pem, cert.pem root.crt > > now i create a new p12 with I hope the correct certs > > I become from Startcom a httpd zip file with 1_root_bundle.crt ("15 Years")and > my wild-card Certificate this I included in my new created p12 with my key.pem. > > This p12 I Installed on the first master with > > pk12util -v -i ipa_4gjn_ca.p12 -d /etc/httpd/alias -k > /etc/httpd/alias/pwdfile.txt -W xxxxxxxx > > pk12util -v -i ipa_4gjn_ca.p12 -d /etc/dirsrv/slapd-4GJN-COM -k > /etc/dirsrv/slapd-4GJN-COM/pwdfile.txt -W xxxxxxx > and > pk12util -v -i ipa_4gjn_ca.p12 -d /var/lib/pki/pki-tomcat/alias -k > /etc/pki/pki-tomcat/pwdfile.txt -W xxxxxxxxx > > I change the nss.conf and I hope the correct file in /etc/dirsrv/slapd- > XXXX/dsl.ldif > > Then I change in all NSS DB the StartCOM Cert (1_root_bundle.crt) with name > STARTCOM-ROOT to > certutil -d . -M -t C,, -n STARCOM-ROOT > > > afterward I make a reboot and a test > ipactl status > Directory Service: RUNNING > krb5kdc Service: RUNNING > kadmin Service: RUNNING > named Service: RUNNING > ipa_memcached Service: RUNNING > httpd Service: RUNNING > ipa-custodia Service: RUNNING > pki-tomcatd Service: RUNNING > ipa-otpd Service: RUNNING > ipa-ods-exporter Service: STOPPED > ods-enforcerd Service: RUNNING > ipa-dnskeysyncd Service: RUNNING > ipa: INFO: The ipactl command was successful > > Why is ipa-ods-exporter Service always STOPPED ?? > > The next I Test a login on the Web UI from IPA, this is now also working ;-) > > > the QUESTION is now what is with the second master and the IPA- clients > Now (?) I have also found the ipa-backup ;-) OK, for the next Problem now I > know it :-). > > Have I to repeat this all on the second Master ? > > and what is the correct way to inform the clients ? > > Thanks again for a answer, > >> Hi, >> >> there were some issues with ipa-server-certinstall (see tickets #4785, >> #4786 and #6263). >> In order to check your configuration, you must make sure that the NSS >> DBs for Apache and the LDAP server (/etc/httpd/alias, >> /var/lib/pki/pki-tomcat/alias, /etc/dirsrv/slapd-DOMxx) contain: >> - the server certificate with flags u,u,u (= the one contained in >> ipa_3rd_ca.p12) >> - the certificate of the CA which signed the server certificate, with >> flags C,, (= the one contained in root.rt) >> >> Then you can also check if the nickname for the server cert is properly >> set in /etc/httpd/conf.d/nss.conf (in the directive NSSNickname), and in >> the LDAP entry cn=RSA,cn=encryption,cn=config (in the attribute >> nsSSLPersonalitySSL). >> >> If this doesn't fix the issue, the logs of pki-tomcat/ca/debug may >> provide more information. >> >> Also note that it is important to run ipa-certupdate on all the clients >> and replicas in order to install the new certificates in the NSS DBs >> *before* you run ipa-server-certinstall. >> >> Hope this helps, >> Flo. >> >>> the kerberos don't start anymore ? >>> The Error Is >>> >>> Unspecified GSS failure.Minor (2529639068): Cannot contact any KDC for >>> realm> >>> '4GJN.COM' >>> >>> after insert in nss.conf >>> "NSSEnforceValidCerts off" >>> >>> ipactl restart is starting (?) but >>> >>> ipactl status tell me >>> Directory Service: RUNNING >>> krb5kdc Service: RUNNING >>> kadmin Service: RUNNING >>> named Service: RUNNING >>> ipa_memcached Service: RUNNING >>> httpd Service: RUNNING >>> ipa-custodia Service: RUNNING >>> pki-tomcatd Service: RUNNING >>> ipa-otpd Service: RUNNING >>> ipa-ods-exporter Service: STOPPED >>> ods-enforcerd Service: RUNNING >>> ipa-dnskeysyncd Service: RUNNING >>> ipa: INFO: The ipactl command was successful >>> >>> with certutil -d /etc/httpd/alias -L I have now this >>> Certificate Nickname Trust >>> Attributes> >>> SSL,S/MIME,JA >>> R/XPI >>> >>> Signing-Cert u,u,u >>> 4GJN_CA_FILE u,u,u >>> ipaCert u,u,u >>> 4GJN.COM IPA CA CT,C,C >>> STARTCOM-ROOT C,, >>> >>> I can Insert in nss.conf by the >>> #NSSNickname "Signing-Cert" original >>> or >>> NSSNickname 4GJN_CA_FILE but all is now broken ? >>> >>> I also add this, found in Bugzilla >>> >>> certutil -d /var/lib/pki/pki-tomcat/alias -L >>> >>> Certificate Nickname Trust >>> Attributes> >>> SSL,S/MIME,JA >>> R/XPI >>> >>> ocspSigningCert cert-pki-ca u,u,u >>> subsystemCert cert-pki-ca u,u,u >>> caSigningCert cert-pki-ca CTu,Cu,Cu >>> Server-Cert cert-pki-ca u,u,u >>> auditSigningCert cert-pki-ca u,u,Pu >>> STARTCOM-ROOT CT,, >>> >>> this is created in the >>> >>> certutil -d /etc/dirsrv/slapd-4GJN.COM -L >>> >>> Certificate Nickname Trust >>> Attributes> >>> SSL,S/MIME,JA >>> R/XPI >>> >>> 4GJN_CA_FILE u,u,u >>> 4GJN.COM IPA CA CT,C,C >>> STARTCOM-ROOT C,, >>> >>> Can any help a little, please ;-) >>> >>> The bad Problem, I tested this with my master server with DNS / DNSSEC I >>> can't new install (DNSSEC Keys) > From tjaalton at ubuntu.com Wed Sep 21 07:26:41 2016 From: tjaalton at ubuntu.com (Timo Aaltonen) Date: Wed, 21 Sep 2016 10:26:41 +0300 Subject: [Freeipa-users] FreeIPA client installation on ubuntu 14.04 In-Reply-To: References: Message-ID: <57E23631.2040207@ubuntu.com> On 21.09.2016 09:41, Deepak Dimri wrote: > Hi All, > > I am trying to install freeipa client on my ubuntu client via ansible > script. I have "apt-get update" and "apt-get install freeipa-client -y" > these basic commands added in my playbook but the problem is when i run > "apt-get install freeipa-client" with or without -y option it opens up > some graphical interface confirming the IPA realm and other details. I > did not find any option with in "apt-get install freeipa-client"to make > it deployment unattended. Can anyone please tell me the how i can > automate ipa-client installation on ubuntu? > > The same process works fine with RHEL using yum but i am unable to do so > for ubuntu with apt-get the dialog is from krb5-common, and you can skip it with DEBIAN_FRONTEND=noninteractive apt-get install ... From jan.karasek at elostech.cz Wed Sep 21 07:47:12 2016 From: jan.karasek at elostech.cz (Jan =?utf-8?Q?Kar=C3=A1sek?=) Date: Wed, 21 Sep 2016 09:47:12 +0200 (CEST) Subject: [Freeipa-users] SSH public user's key stored in AD POSIX attribute In-Reply-To: References: Message-ID: <635468628.1622501.1474444032930.JavaMail.zimbra@elostech.cz> Hi, I have a question about the IPA-AD trust scenario where POSIX attributes are store in AD. I would like to know if it's possible to store public SSH user key in Active Directory in some user's object attribute - the same way as uidNumber or loginShell. I can't find any suitable attribute for ssh in AD schema but the uidNumber,gidNumber and others are already presented (win2012). So is there any chance to extend AD schema and let the IPA server get public ssh user's key from AD the same way as other POSIX attributes ? Is it IPA ready for that and how that attribute should be named in AD ? Thanks, Jan -------------- next part -------------- An HTML attachment was scrubbed... URL: From sbose at redhat.com Wed Sep 21 08:34:21 2016 From: sbose at redhat.com (Sumit Bose) Date: Wed, 21 Sep 2016 10:34:21 +0200 Subject: [Freeipa-users] SSH public user's key stored in AD POSIX attribute In-Reply-To: <635468628.1622501.1474444032930.JavaMail.zimbra@elostech.cz> References: <635468628.1622501.1474444032930.JavaMail.zimbra@elostech.cz> Message-ID: <20160921083421.GD6019@p.Speedport_W_724V_Typ_A_05011603_00_009> On Wed, Sep 21, 2016 at 09:47:12AM +0200, Jan Kar?sek wrote: > Hi, > > I have a question about the IPA-AD trust scenario where POSIX attributes are store in AD. Although I describe some possible solution below I wonder if using IPA overrides which allow to add public ssh keys for AD user would work for you as well? > > I would like to know if it's possible to store public SSH user key in Active Directory in some user's object attribute - the same way as uidNumber or loginShell. I can't find any suitable attribute for ssh in AD schema but the uidNumber,gidNumber and others are already presented (win2012). In general it is possible either extend the schema or use an existing attribute, see e.g. https://social.technet.microsoft.com/Forums/en-US/8aa28e34-2007-49fe-a689-e28e19b2757b/is-there-a-way-to-link-ssh-key-in-ad?forum=winserverDS for details. But given the recent activities in areas of Powershell and OpenSSH for Windows I wonder if there might be some "official" attributes coming sooner or later. Currently I'm not aware of any plans here but maybe other readers on the list have more insight here? > > So is there any chance to extend AD schema and let the IPA server get public ssh user's key from AD the same way as other POSIX attributes ? Is it IPA ready for that and how that attribute should be named in AD ? You have to configure SSSD on the IPA server to read the attribute and forward it to the clients, for this you need (at least) to add [domain/EXAMPLE] ldap_user_extra_attrs = adAttributeName:sshPublicKey (see sssd-ldap man page for details) bye, Sumit > > Thanks, > > Jan > -- > Manage your subscription for the Freeipa-users mailing list: > https://www.redhat.com/mailman/listinfo/freeipa-users > Go to http://freeipa.org for more info on the project From deepak_dimri at hotmail.com Wed Sep 21 08:34:42 2016 From: deepak_dimri at hotmail.com (Deepak Dimri) Date: Wed, 21 Sep 2016 04:34:42 -0400 Subject: [Freeipa-users] FreeIPA client installation on ubuntu 14.04 In-Reply-To: <57E23631.2040207@ubuntu.com> References: , <57E23631.2040207@ubuntu.com> Message-ID: Thanks Timo, The "DEBIAN_FRONTEND=noninteractive apt-get install freeipa-client -y" command works on the terminal but within ansible playbook i am getting [Errno 2] No such file or directory", "rc": 2} when adding command: DEBIAN_FRONTEND=noninteractive apt-get install freeipa-client -y any idea how can i get this resolved for ansible? i tried "export DEBIAN_FRONTEND=noninteractive" and then "apt-get install freeipa-client -y" but that did not help either still getting [Errno 2] No such file or directory", "rc": 2} Thanks again,Deepak > Subject: Re: [Freeipa-users] FreeIPA client installation on ubuntu 14.04 > To: deepak_dimri at hotmail.com; freeipa-users at redhat.com > From: tjaalton at ubuntu.com > Date: Wed, 21 Sep 2016 10:26:41 +0300 > > On 21.09.2016 09:41, Deepak Dimri wrote: > > Hi All, > > > > I am trying to install freeipa client on my ubuntu client via ansible > > script. I have "apt-get update" and "apt-get install freeipa-client -y" > > these basic commands added in my playbook but the problem is when i run > > "apt-get install freeipa-client" with or without -y option it opens up > > some graphical interface confirming the IPA realm and other details. I > > did not find any option with in "apt-get install freeipa-client"to make > > it deployment unattended. Can anyone please tell me the how i can > > automate ipa-client installation on ubuntu? > > > > The same process works fine with RHEL using yum but i am unable to do so > > for ubuntu with apt-get > > the dialog is from krb5-common, and you can skip it with > > DEBIAN_FRONTEND=noninteractive apt-get install ... > -------------- next part -------------- An HTML attachment was scrubbed... URL: From natxo.asenjo at gmail.com Wed Sep 21 08:50:47 2016 From: natxo.asenjo at gmail.com (Natxo Asenjo) Date: Wed, 21 Sep 2016 10:50:47 +0200 Subject: [Freeipa-users] replica added, but clients still try renewing certificates with old master Message-ID: hi, I followed the instructions here: https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/7/html/Linux_Domain_Identity_Authentication_and_Policy_Guide/upgrading.html and now after some issues I have a replica with both pki and dns data running centos 7. So now I have 3 replicas: centos 6.8: kdc01.unix.iriszorg.nl kdc02.unix.iriszorg.nl centos 7.2 kdc03.unix.iriszorg.nl The replica was created with an agreement to kdc01.unix.iriszorg.nl which was the master for crl updates. I followed the steps to disabled crlcache and crlupdates on the kdc01 and to enable them on the kdc03. So in the kdc01 I edited /etc/httpd/conf.d/ipa-pki-proxy.conf and uncommented # Only enable this on servers that are not generating a CRL RewriteRule ^/ipa/crl/MasterCRL.bin https://kdc03.unix.iriszorg.nl/ca/ee/ca/getCRL?op=getCRL&crlIssuingPoint=MasterCRL [L,R=301,NC] and on the kdc03 i commented this out: # Only enable this on servers that are not generating a CRL #RewriteRule ^/ipa/crl/MasterCRL.bin https://kdc03.unix.iriszorg.nl/ca/ee/ca/getCRL?op=getCRL&crlIssuingPoint=MasterCRL [L,R=301,NC] When I try to resubmit certificates from certmonger they still hit the kdc01 web server, so the requests hang on an status: CA_UNREACHABLE ca-error: Server failed request, will retry: 4301 (RPC failed at server. Certificate operation cannot be completed: Failure decoding Certificate Signing Request). Which was the problem on a recent thread on the list (trying to get rid of this replica now to fix this problem as well). So something is not redirecting properly and I would appreciate your assistance. TIA. -- Groeten, natxo -------------- next part -------------- An HTML attachment was scrubbed... URL: From tjaalton at ubuntu.com Wed Sep 21 11:40:17 2016 From: tjaalton at ubuntu.com (Timo Aaltonen) Date: Wed, 21 Sep 2016 14:40:17 +0300 Subject: [Freeipa-users] FreeIPA client installation on ubuntu 14.04 In-Reply-To: References: <57E23631.2040207@ubuntu.com> Message-ID: <57E271A1.1080902@ubuntu.com> On 21.09.2016 11:34, Deepak Dimri wrote: > Thanks Timo, > > The "DEBIAN_FRONTEND=noninteractive apt-get install freeipa-client -y" > command works on the terminal but within ansible playbook i am getting > > [Errno 2] No such file or directory", "rc": 2} when adding > command: DEBIAN_FRONTEND=noninteractive apt-get install freeipa-client -y > > > any idea how can i get this resolved for ansible? i tried > "export DEBIAN_FRONTEND=noninteractive" and then "apt-get install > freeipa-client -y" but that did not help either still getting [Errno 2] > No such file or directory", "rc": 2} no idea about that, but you could also preseed the debconf priority beforehand and then run apt-get, something like: echo 'debconf debconf/priority select critical' > /tmp/preseed debconf-set-selections /tmp/preseed apt-get ... From Johan.Petersson at sscspace.com Wed Sep 21 12:27:13 2016 From: Johan.Petersson at sscspace.com (Johan Petersson) Date: Wed, 21 Sep 2016 12:27:13 +0000 Subject: [Freeipa-users] Central logging docker image Message-ID: <558C15177F5E714F83334217C9A197DF023B1DDC3A@SSC-MBX2.ssc.internal> Hi, When i was evaluating the configuration of the central logging proof of concept docker image described here: https://www.freeipa.org/page/Centralized_Logging i noticed that the rsyslog mmnormalization rules did not work properly and failed to parse keywords. Elasticsearch indexes does not get properly filled. This lead to the visualizations and dashboards are broken in Kibana. The reason is that custom repos was being used to install newer version of rsyslog, rsyslog-mmnormalize and rsyslog-elasticsearch (8.8.0-2), liblognorm (1.1.1-1 and liblogging (1.0.4-4). These repos do not work anymore since the content has been removed. The repos are: https://copr-be.cloud.fedoraproject.org/results/jhrozek/rsyslog https://copr-be.cloud.fedoraproject.org/results/jhrozek/liblognorm https://copr-be.cloud.fedoraproject.org/results/jhrozek/liblogging Liblognorm version in RHEL 7 are normally 0.3.7. It would be of interest if these newer versions could be made available for RHEL 7 in the near future since they would make RHEL 7 as a log server platform much more interesting. This e-mail is private and confidential between the sender and the addressee. In the event of misdirection, the recipient is prohibited from using, copying or disseminating it or any information in it. Please notify the above if any misdirection. -------------- next part -------------- An HTML attachment was scrubbed... URL: From koreyc at gmail.com Wed Sep 21 00:13:18 2016 From: koreyc at gmail.com (Korey Chapman) Date: Tue, 20 Sep 2016 17:13:18 -0700 Subject: [Freeipa-users] CA Fails to build Replica (w/External CA) Message-ID: Hello list, I'm currently attempting to add a second CA server to our IPA cluster (all servers Centos 7.2 with IPA 4.2.0). However, it is failing no matter how I try to setup the CA (ipa-replica-install with --setup-ca or ipa-replica-install followed by ipa-ca-install). The only useful thing in the logs is an error about a missing key for "trust_flags" in the pki setup. Our infrastructure uses FreeIPA with an external CA. Any ideas/help would be greatly appreciated. Here are the logs snips from my most recent attempt: Command output snip from "ipa-replica-install /root/replica-info-auth-002.XXX.gpg --setup-ca" Configuring certificate server (pki-tomcatd). Estimated time: 3 minutes 30 seconds [1/24]: creating certificate server user [2/24]: configuring certificate server instance ipa.ipaserver.install.cainstance.CAInstance: CRITICAL Failed to configure CA instance: Command ''/usr/sbin/pkispawn' '-s' 'CA' '-f' '/tmp/tmpYofMPt'' returned non-zero exit status 1 ipa.ipaserver.install.cainstance.CAInstance: CRITICAL See the installation logs and the following files/directories for more information: ipa.ipaserver.install.cainstance.CAInstance: CRITICAL /var/log/pki-ca-install.log ipa.ipaserver.install.cainstance.CAInstance: CRITICAL /var/log/pki/pki-tomcat [error] RuntimeError: CA configuration failed. Your system may be partly configured. Run /usr/sbin/ipa-server-install --uninstall to clean up. ipa.ipapython.install.cli.install_tool(Replica): ERROR CA configuration failed Log snip from ipareplica-install.log: 2016-09-20T23:42:27Z DEBUG Starting external process 2016-09-20T23:42:27Z DEBUG args='/usr/sbin/pkispawn' '-s' 'CA' '-f' '/tmp/tmpYofMPt' 2016-09-20T23:42:31Z DEBUG Process finished, return code=1 2016-09-20T23:42:31Z DEBUG stdout=Log file: /var/log/pki/pki-ca-spawn.20160920234227.log Loading deployment configuration from /tmp/tmpYofMPt. Installing CA into /var/lib/pki/pki-tomcat. Storing deployment configuration into /etc/sysconfig/pki/tomcat/pki-tomcat/ca/deployment.cfg. Installation failed. 2016-09-20T23:42:31Z DEBUG stderr=/usr/lib/python2.7/site-packages/urllib3/connectionpool.py:769: InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.org/en/latest/security.html InsecureRequestWarning) Traceback (most recent call last): File "/bin/pki", line 254, in cli.execute(sys.argv) File "/bin/pki", line 240, in execute module.execute(module_args) File "/usr/lib/python2.7/site-packages/pki/cli/__init__.py", line 195, in execute module.execute(module_args) File "/usr/lib/python2.7/site-packages/pki/cli/pkcs12.py", line 222, in execute trust_flags = cert_info['trust_flags'] KeyError: 'trust_flags' -- Korey -------------- next part -------------- An HTML attachment was scrubbed... URL: From deepak_dimri at hotmail.com Wed Sep 21 12:49:13 2016 From: deepak_dimri at hotmail.com (Deepak Dimri) Date: Wed, 21 Sep 2016 08:49:13 -0400 Subject: [Freeipa-users] 2FA using FreeIPA In-Reply-To: <20160916084326.GA10405@10.4.128.1> References: , <20160916084326.GA10405@10.4.128.1> Message-ID: hi LS, I am using IPA Server - VERSION: 4.2.0, API_VERSION: 2.156sssd version on my IPA server: 1.13.0sssd version on my IPA client (ubuntu): 1.11.8 I have new "testhip2user" created in IPA Server with 2FA enabled. My /etc/ssh/sshd_config has this entry AuthorizedKeysFile %h/.ssh/authorized_keys #ChallengeResponseAuthentication no PasswordAuthentication noMatch User testhip2user AuthenticationMethods publickey,password:pam publickey,keyboard-interactive:pam When i am trying to ssh with private key of testhip2user into IPA client then this what i see in ssh auth.log as keep getting prompted for password and then it end with permission denied error Sep 21 12:42:04 ip-172-31-30-146 sshd[7530]: error: Disabled method "password" in AuthenticationMethods list "publickey,password:pam" Sep 21 12:42:04 ip-172-31-30-146 sshd[7530]: Authentication methods list "publickey,password:pam" contains disabled method, skipping Sep 21 12:42:04 ip-172-31-30-146 sshd[7530]: error: Disabled method "password" in AuthenticationMethods list "publickey,password:pam" [preauth] Sep 21 12:42:04 ip-172-31-30-146 sshd[7530]: Authentication methods list "publickey,password:pam" contains disabled method, skipping [preauth]Sep 21 12:42:50 ip-172-31-30-146 sshd[7533]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=50-201-125-254-static.hfc.comcastbusiness.net user=testhip2userSep 21 12:42:50 ip-172-31-30-146 sshd[7533]: pam_sss(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=50-201-125-254-static.hfc.comcastbusiness.net user=testhip2userSep 21 12:42:50 ip-172-31-30-146 sshd[7533]: pam_sss(sshd:auth): received for user testhip2user: 6 (Permission denied)Sep 21 12:42:53 ip-172-31-30-146 sshd[7530]: error: PAM: Authentication failure for testhip2user from 50-201-125-254-static.hfc.comcastbusiness.net Thanks for your time and helping me with this Best Regards,Deepak > Date: Fri, 16 Sep 2016 10:43:26 +0200 > From: lslebodn at redhat.com > To: deepak_dimri at hotmail.com > CC: freeipa-users at redhat.com > Subject: Re: [Freeipa-users] 2FA using FreeIPA > > On (13/09/16 03:49), Deepak Dimri wrote: > >Hi All, > >I have below lines added to my sshd_config file for testuser. > > > > > > > >Match User testuser > > AuthenticationMethods publickey,password:pam publickey,keyboard-interactive:pam > >I have OTP enable for tapuser in IPA and i am able to login to GUI using the password + OTP. However when i try to ssh i am getting prompted for first factor then second factor and then it ends with "Permission denied (keyboard-interactive)." error. What could be wrong here? > >Regards,Deepak > > > Please provide versions of freeIPA server packages, version of sssd. > And it would be good to seed the exact output of ssh authentication. > > LS -------------- next part -------------- An HTML attachment was scrubbed... URL: From deepak_dimri at hotmail.com Wed Sep 21 13:00:12 2016 From: deepak_dimri at hotmail.com (Deepak Dimri) Date: Wed, 21 Sep 2016 09:00:12 -0400 Subject: [Freeipa-users] FreeIPA client installation on ubuntu 14.04 In-Reply-To: <57E271A1.1080902@ubuntu.com> References: , <57E23631.2040207@ubuntu.com> , <57E271A1.1080902@ubuntu.com> Message-ID: I got it fixed my adding these in my playbook - command: sudo env DEBIAN_FRONTEND=noninteractive - shell: "DEBIAN_FRONTEND=noninteractive apt-get install freeipa-client -y" Thanks,Deepak > Subject: Re: [Freeipa-users] FreeIPA client installation on ubuntu 14.04 > To: deepak_dimri at hotmail.com; freeipa-users at redhat.com > From: tjaalton at ubuntu.com > Date: Wed, 21 Sep 2016 14:40:17 +0300 > > On 21.09.2016 11:34, Deepak Dimri wrote: > > Thanks Timo, > > > > The "DEBIAN_FRONTEND=noninteractive apt-get install freeipa-client -y" > > command works on the terminal but within ansible playbook i am getting > > > > [Errno 2] No such file or directory", "rc": 2} when adding > > command: DEBIAN_FRONTEND=noninteractive apt-get install freeipa-client -y > > > > > > any idea how can i get this resolved for ansible? i tried > > "export DEBIAN_FRONTEND=noninteractive" and then "apt-get install > > freeipa-client -y" but that did not help either still getting [Errno 2] > > No such file or directory", "rc": 2} > > no idea about that, but you could also preseed the debconf priority > beforehand and then run apt-get, something like: > > echo 'debconf debconf/priority select critical' > /tmp/preseed > debconf-set-selections /tmp/preseed > apt-get ... > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From akor72 at gmail.com Wed Sep 21 12:43:29 2016 From: akor72 at gmail.com (Alexander K) Date: Wed, 21 Sep 2016 17:43:29 +0500 Subject: [Freeipa-users] AD users can't login to IPA client Message-ID: Hello, I'm having troubles with AD users authentication on IPA client. I have 3 VMs in my test inveronment: win-dc.windc.local 10.1.97.122 - AD DC server 2012R2 fedora-dc.demo.loc 10.1.97.120 - fedora 24 + FreeIPA wks.demo.loc 10.1.97.121 - IPA client I have done IPA AD trust setup https://www.freeipa.org/page/Active_Directory_trust_setup AD user can access IPA server: login as: Administrator at windc.local Administrator at windc.local@10.1.97.120's password: Last login: Wed Sep 21 13:59:36 2016 from 192.168.70.26 Could not chdir to home directory /home/windc.local/administrator: No such file or directory -sh-4.3$ IPA user can login IPA client: login as: admin admin at 10.1.97.121's password: Last login: Wed Sep 21 16:12:31 2016 from 192.168.70.26 [admin at wks ~]$ But AD user can't access IPA client: login as: Administrator at windc.local Administrator at windc.local@10.1.97.121's password: Access denied On another hand, ID works correct for AD users: [root at wks ~]# id Administrator at windc.local uid=429000500(administrator at windc.local) gid=429000500(administrator at windc.local) groups=429000500(administrator at windc.local),429000520(group policy creator owners at windc.local),429000519(enterprise admins at windc.local),429000513(domain users at windc.local),429000518(schema admins at windc.local),429000512(domain admins at windc.local) I have attached logs (Last login time is 17:29-17:30) Any help would be appreciated! -- Best regards, Alexander K -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: sssd_nss.log Type: application/octet-stream Size: 177799 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: sssd_demo.loc.log Type: application/octet-stream Size: 971133 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: sssd_pam.log Type: application/octet-stream Size: 293364 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: krb5_child.log Type: application/octet-stream Size: 94982 bytes Desc: not available URL: From jhrozek at redhat.com Wed Sep 21 13:44:15 2016 From: jhrozek at redhat.com (Jakub Hrozek) Date: Wed, 21 Sep 2016 15:44:15 +0200 Subject: [Freeipa-users] AD users can't login to IPA client In-Reply-To: References: Message-ID: <20160921134415.sm6hjy5qigvj4tek@hendrix> On Wed, Sep 21, 2016 at 05:43:29PM +0500, Alexander K wrote: > Hello, > > I'm having troubles with AD users authentication on IPA client. > I have 3 VMs in my test inveronment: > win-dc.windc.local 10.1.97.122 - AD DC server 2012R2 > fedora-dc.demo.loc 10.1.97.120 - fedora 24 + FreeIPA > wks.demo.loc 10.1.97.121 - IPA client > > I have done IPA AD trust setup > https://www.freeipa.org/page/Active_Directory_trust_setup > > AD user can access IPA server: > login as: Administrator at windc.local > Administrator at windc.local@10.1.97.120's password: > Last login: Wed Sep 21 13:59:36 2016 from 192.168.70.26 > Could not chdir to home directory /home/windc.local/administrator: No such > file or directory > -sh-4.3$ > > IPA user can login IPA client: > login as: admin > admin at 10.1.97.121's password: > Last login: Wed Sep 21 16:12:31 2016 from 192.168.70.26 > [admin at wks ~]$ > > > But AD user can't access IPA client: > login as: Administrator at windc.local > Administrator at windc.local@10.1.97.121's password: > Access denied > > On another hand, ID works correct for AD users: > [root at wks ~]# id Administrator at windc.local > uid=429000500(administrator at windc.local) > gid=429000500(administrator at windc.local) > groups=429000500(administrator at windc.local),429000520(group policy creator > owners at windc.local),429000519(enterprise admins at windc.local),429000513(domain > users at windc.local),429000518(schema admins at windc.local),429000512(domain > admins at windc.local) > > I have attached logs > (Last login time is 17:29-17:30) The domain logs say the authentication takes too long, it might be due to processing the PAC. Try increasing the authentication timeout (krb5_auth_timeout). > > > Any help would be appreciated! > > > -- > Best regards, > Alexander K > -- > Manage your subscription for the Freeipa-users mailing list: > https://www.redhat.com/mailman/listinfo/freeipa-users > Go to http://freeipa.org for more info on the project From tkrizek at redhat.com Wed Sep 21 13:47:15 2016 From: tkrizek at redhat.com (Tomas Krizek) Date: Wed, 21 Sep 2016 15:47:15 +0200 Subject: [Freeipa-users] CA Fails to build Replica (w/External CA) In-Reply-To: References: Message-ID: <64066e0c-a299-cfa8-f1e5-1585692c15ae@redhat.com> On 09/21/2016 02:13 AM, Korey Chapman wrote: > Hello list, > > I'm currently attempting to add a second CA server to our IPA cluster > (all servers Centos 7.2 with IPA 4.2.0). However, it is failing no > matter how I try to setup the CA (ipa-replica-install with --setup-ca > or ipa-replica-install followed by ipa-ca-install). The only useful > thing in the logs is an error about a missing key for "trust_flags" in > the pki setup. Our infrastructure uses FreeIPA with an external CA. > > Any ideas/help would be greatly appreciated. Here are the logs snips > from my most recent attempt: > > Command output snip from "ipa-replica-install > /root/replica-info-auth-002.XXX.gpg --setup-ca" > Configuring certificate server (pki-tomcatd). Estimated time: 3 > minutes 30 seconds > [1/24]: creating certificate server user > [2/24]: configuring certificate server instance > ipa.ipaserver.install.cainstance.CAInstance: CRITICAL Failed to > configure CA instance: Command ''/usr/sbin/pkispawn' '-s' 'CA' '-f' > '/tmp/tmpYofMPt'' returned non-zero exit status 1 > ipa.ipaserver.install.cainstance.CAInstance: CRITICAL See the > installation logs and the following files/directories for more > information: > ipa.ipaserver.install.cainstance.CAInstance: CRITICAL > /var/log/pki-ca-install.log > ipa.ipaserver.install.cainstance.CAInstance: CRITICAL > /var/log/pki/pki-tomcat > [error] RuntimeError: CA configuration failed. > Your system may be partly configured. > Run /usr/sbin/ipa-server-install --uninstall to clean up. > > ipa.ipapython.install.cli.install_tool(Replica): ERROR CA > configuration failed > > > Log snip from ipareplica-install.log: > > 2016-09-20T23:42:27Z DEBUG Starting external process > 2016-09-20T23:42:27Z DEBUG args='/usr/sbin/pkispawn' '-s' 'CA' '-f' > '/tmp/tmpYofMPt' > 2016-09-20T23:42:31Z DEBUG Process finished, return code=1 > 2016-09-20T23:42:31Z DEBUG stdout=Log file: > /var/log/pki/pki-ca-spawn.20160920234227.log > Loading deployment configuration from /tmp/tmpYofMPt. > Installing CA into /var/lib/pki/pki-tomcat. > Storing deployment configuration into > /etc/sysconfig/pki/tomcat/pki-tomcat/ca/deployment.cfg. > > Installation failed. > > > 2016-09-20T23:42:31Z DEBUG > stderr=/usr/lib/python2.7/site-packages/urllib3/connectionpool.py:769: > InsecureRequestWarning: Unverified HTTPS request is being made. Adding > certificate verification is strongly advised. See: > https://urllib3.readthedocs.org/en/latest/security.html > InsecureRequestWarning) > Traceback (most recent call last): > File "/bin/pki", line 254, in > cli.execute(sys.argv) > File "/bin/pki", line 240, in execute > module.execute(module_args) > File "/usr/lib/python2.7/site-packages/pki/cli/__init__.py", line > 195, in execute > module.execute(module_args) > File "/usr/lib/python2.7/site-packages/pki/cli/pkcs12.py", line 222, > in execute > trust_flags = cert_info['trust_flags'] > KeyError: 'trust_flags' > > > -- > Korey > > Hi Korey, could you check if there is any more info in /var/log/pki/pki-ca-spawn log? It might also be helpful verify if correct trust flags are set in nssdb: certutil -d /etc/pki/pki-tomcat/alias/ -L Finally, can you check that LDAPS is running on port 636 on the replica where you're trying to install the CA (i.e. by nmap localhost)? -- Tomas Krizek -------------- next part -------------- An HTML attachment was scrubbed... URL: From pvoborni at redhat.com Wed Sep 21 14:38:25 2016 From: pvoborni at redhat.com (Petr Vobornik) Date: Wed, 21 Sep 2016 16:38:25 +0200 Subject: [Freeipa-users] replica added, but clients still try renewing certificates with old master In-Reply-To: References: Message-ID: <89facc8a-74d1-5c00-dbd9-6cafa20f0e80@redhat.com> On 09/21/2016 10:50 AM, Natxo Asenjo wrote: > hi, > > I followed the instructions here: > https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/7/html/Linux_Domain_Identity_Authentication_and_Policy_Guide/upgrading.html > > and now after some issues I have a replica with both pki and dns data running > centos 7. > > So now I have 3 replicas: > > centos 6.8: > kdc01.unix.iriszorg.nl > kdc02.unix.iriszorg.nl > > centos 7.2 > kdc03.unix.iriszorg.nl > > The replica was created with an agreement to kdc01.unix.iriszorg.nl > which was the master for crl updates. I followed > the steps to disabled crlcache and crlupdates on the kdc01 and to enable them on > the kdc03. > > So in the kdc01 I edited /etc/httpd/conf.d/ipa-pki-proxy.conf and uncommented > > # Only enable this on servers that are not generating a CRL > RewriteRule ^/ipa/crl/MasterCRL.bin > https://kdc03.unix.iriszorg.nl/ca/ee/ca/getCRL?op=getCRL&crlIssuingPoint=MasterCRL > [L,R=301,NC] > > and on the kdc03 i commented this out: > > # Only enable this on servers that are not generating a CRL > #RewriteRule ^/ipa/crl/MasterCRL.bin > https://kdc03.unix.iriszorg.nl/ca/ee/ca/getCRL?op=getCRL&crlIssuingPoint=MasterCRL > [L,R=301,NC] > > > When I try to resubmit certificates from certmonger they still hit the kdc01 web > server, so the requests hang on an status: CA_UNREACHABLE > ca-error: Server failed request, will retry: 4301 (RPC failed at server. > Certificate operation cannot be completed: Failure decoding Certificate Signing > Request). Where does it happen? On arbitrary client which was installed in a past against the removed kdc01? If so could you look into /etc/ipa/default.conf and change host option from kdc01 to the 7.2 IPA sever? If this is correct then IMO it is quite a serious bug which needs to be fixed (i.e. DNS discovery needs to be used). > > > Which was the problem on a recent thread on the list (trying to get rid of this > replica now to fix this problem as well). > > So something is not redirecting properly and I would appreciate your assistance. > > TIA. > -- > Groeten, > natxo > -- Petr Vobornik From abrook at bsd.uchicago.edu Wed Sep 21 13:52:54 2016 From: abrook at bsd.uchicago.edu (Brook, Andy [CRI]) Date: Wed, 21 Sep 2016 13:52:54 +0000 Subject: [Freeipa-users] Samba Server setup In-Reply-To: <20160916170216.jxpico2fscjcswq4@redhat.com> References: <73C11E05-045F-4324-9733-44C370584232@bsd.uchicago.edu> <20160915180606.v2z6dkdw3gltem34@redhat.com> <9A3517B6-CFFB-47ED-BDF2-BE3EE0D06EDE@bsd.uchicago.edu> <20160916050435.l4k6rshziftp6pzd@redhat.com> <38F25AFE-C5E8-4CE9-BD18-26949884625A@bsd.uchicago.edu> <20160916170216.jxpico2fscjcswq4@redhat.com> Message-ID: <08B39CF6-36E9-41C4-88A8-F1E4077B75AF@bsd.uchicago.edu> On 9/16/16, 12:02 PM, "Alexander Bokovoy" wrote: On Fri, 16 Sep 2016, Brook, Andy [CRI] wrote: > You can replace actual hostnames/realm names/IP addresses by something more generic > in the output when sending to the list, but please do it consistently. > >I?m sorry. I thought I had been consistent when making changes, but >from your response, it looks like I wasn?t. I?m sorry about that. I got >yelled at by our security team last time we sent logs to a public list >that had any type of identifiable information in them, so it?s sort of >a new process for me. I think I have it down now. > >The results of the commands are here: http://pastebin.com/PRwr7wv6 So IPA side works fine -- on IPA client you can kinit as AD user and then obtain cross-realm TGT to IPA realm and use that cross-realm TGT to request a service ticket to cifs/... service. That's good. You need to identify what happens on AD side. A possible issue is that name suffix routing to IPA domain is disabled. Can you provide output of netdom.exe run on Windows side: netdom trust addom.domain /namesuffixes: ipa.domain You should get something like example 28 on the page https://msdn.microsoft.com/en-us/library/cc776879(v=ws.10).aspx Thank you for this. I went to run the command and kept getting an ?Incorrect parameter? error. After that I talked to one of our Active Directory admins and he mentioned that we are working on resolving a disjoint namespace error on addom. I don?t understand enough about it, but do know that it can cause issues with Kerberos authentication across domains. That should get fixed soon. Once that gets fixed, I?ll test again. I have one more related question. The instruction page states that NTLMSSP authentication isn?t working as of yet, as well as you mentioned it earlier in this thread. Is there a bug or feature request that is tracking that? Andy Brook Sr. Systems Administrator | Center for Research Informatics | University of Chicago T: 773-834-0458 | http://cri.uchicago.edu ******************************************************************************** This e-mail is intended only for the use of the individual or entity to which it is addressed and may contain information that is privileged and confidential. If the reader of this e-mail message is not the intended recipient, you are hereby notified that any dissemination, distribution or copying of this communication is prohibited. If you have received this e-mail in error, please notify the sender and destroy all copies of the transmittal. Thank you University of Chicago Medicine and Biological Sciences ******************************************************************************** From natxo.asenjo at gmail.com Wed Sep 21 15:06:09 2016 From: natxo.asenjo at gmail.com (Natxo Asenjo) Date: Wed, 21 Sep 2016 17:06:09 +0200 Subject: [Freeipa-users] replica added, but clients still try renewing certificates with old master In-Reply-To: <89facc8a-74d1-5c00-dbd9-6cafa20f0e80@redhat.com> References: <89facc8a-74d1-5c00-dbd9-6cafa20f0e80@redhat.com> Message-ID: hi Petr, On Wed, Sep 21, 2016 at 4:38 PM, Petr Vobornik wrote: > On 09/21/2016 10:50 AM, Natxo Asenjo wrote: > > > When I try to resubmit certificates from certmonger they still hit the > kdc01 web > > server, so the requests hang on an status: CA_UNREACHABLE > > ca-error: Server failed request, will retry: 4301 (RPC failed at > server. > > Certificate operation cannot be completed: Failure decoding Certificate > Signing > > Request). > > Where does it happen? On arbitrary client which was installed in a past > against the removed kdc01? > yes. > > If so could you look into /etc/ipa/default.conf and change host option > from kdc01 to the 7.2 IPA sever? > > ok, done. In fact, change both the domain as the xmlrpc_uri directives in the global section was necessary. Now It worked :-) So, what should be the correct value for dns discovery for both directives using dns discovery? thanks! -- Groeten, natxo -------------- next part -------------- An HTML attachment was scrubbed... URL: From koreyc at gmail.com Wed Sep 21 15:38:37 2016 From: koreyc at gmail.com (Korey Chapman) Date: Wed, 21 Sep 2016 08:38:37 -0700 Subject: [Freeipa-users] CA Fails to build Replica (w/External CA) In-Reply-To: <64066e0c-a299-cfa8-f1e5-1585692c15ae@redhat.com> References: <64066e0c-a299-cfa8-f1e5-1585692c15ae@redhat.com> Message-ID: On Wed, Sep 21, 2016 at 6:47 AM, Tomas Krizek wrote: > On 09/21/2016 02:13 AM, Korey Chapman wrote: > > Hello list, > > I'm currently attempting to add a second CA server to our IPA cluster (all > servers Centos 7.2 with IPA 4.2.0). However, it is failing no matter how I > try to setup the CA (ipa-replica-install with --setup-ca or > ipa-replica-install followed by ipa-ca-install). The only useful thing in > the logs is an error about a missing key for "trust_flags" in the pki setup. > Our infrastructure uses FreeIPA with an external CA. > > Any ideas/help would be greatly appreciated. Here are the logs snips from my > most recent attempt: > > Command output snip from "ipa-replica-install > /root/replica-info-auth-002.XXX.gpg --setup-ca" > Configuring certificate server (pki-tomcatd). Estimated time: 3 minutes 30 > seconds > [1/24]: creating certificate server user > [2/24]: configuring certificate server instance > ipa.ipaserver.install.cainstance.CAInstance: CRITICAL Failed to configure CA > instance: Command ''/usr/sbin/pkispawn' '-s' 'CA' '-f' '/tmp/tmpYofMPt'' > returned non-zero exit status 1 > ipa.ipaserver.install.cainstance.CAInstance: CRITICAL See the installation > logs and the following files/directories for more information: > ipa.ipaserver.install.cainstance.CAInstance: CRITICAL > /var/log/pki-ca-install.log > ipa.ipaserver.install.cainstance.CAInstance: CRITICAL > /var/log/pki/pki-tomcat > [error] RuntimeError: CA configuration failed. > Your system may be partly configured. > Run /usr/sbin/ipa-server-install --uninstall to clean up. > > ipa.ipapython.install.cli.install_tool(Replica): ERROR CA configuration > failed > > > Log snip from ipareplica-install.log: > > 2016-09-20T23:42:27Z DEBUG Starting external process > 2016-09-20T23:42:27Z DEBUG args='/usr/sbin/pkispawn' '-s' 'CA' '-f' > '/tmp/tmpYofMPt' > 2016-09-20T23:42:31Z DEBUG Process finished, return code=1 > 2016-09-20T23:42:31Z DEBUG stdout=Log file: > /var/log/pki/pki-ca-spawn.20160920234227.log > Loading deployment configuration from /tmp/tmpYofMPt. > Installing CA into /var/lib/pki/pki-tomcat. > Storing deployment configuration into > /etc/sysconfig/pki/tomcat/pki-tomcat/ca/deployment.cfg. > > Installation failed. > > > 2016-09-20T23:42:31Z DEBUG > stderr=/usr/lib/python2.7/site-packages/urllib3/connectionpool.py:769: > InsecureRequestWarning: Unverified HTTPS request is being made. Adding > certificate verification is strongly advised. See: > https://urllib3.readthedocs.org/en/latest/security.html > InsecureRequestWarning) > Traceback (most recent call last): > File "/bin/pki", line 254, in > cli.execute(sys.argv) > File "/bin/pki", line 240, in execute > module.execute(module_args) > File "/usr/lib/python2.7/site-packages/pki/cli/__init__.py", line 195, in > execute > module.execute(module_args) > File "/usr/lib/python2.7/site-packages/pki/cli/pkcs12.py", line 222, in > execute > trust_flags = cert_info['trust_flags'] > KeyError: 'trust_flags' > > > -- > Korey > > > Hi Korey, > > could you check if there is any more info in /var/log/pki/pki-ca-spawn log? Nothing really useful I see in the spawn log: 2016-09-20 23:42:31 pkispawn : DEBUG ....... Error Type: CalledProcessError 2016-09-20 23:42:31 pkispawn : DEBUG ....... Error Message: Command '['pki', '-d', '/etc/pki/pki-tomcat/alias', '-C', '/etc/pki/pki-tomcat/pfile', 'pkcs12-import', '--pkcs12-file', '/tmp/ca.p12', '--pkcs12-password-file', '/tmp/tmps5OOav/password.txt', '--no-user-certs']' returned non-zero exit status 1 2016-09-20 23:42:31 pkispawn : DEBUG ....... File "/usr/sbin/pkispawn", line 597, in main rv = scriptlet.spawn(deployer) File "/usr/lib/python2.7/site-packages/pki/server/deployment/scriptlets/security_databases.py", line 104, in spawn no_user_certs=True) File "/usr/lib/python2.7/site-packages/pki/nssdb.py", line 538, in import_pkcs12 subprocess.check_call(cmd) File "/usr/lib64/python2.7/subprocess.py", line 542, in check_call raise CalledProcessError(retcode, cmd) > > It might also be helpful verify if correct trust flags are set in nssdb: > certutil -d /etc/pki/pki-tomcat/alias/ -L > Run on the source ipa server (current CA server): $ certutil -d /etc/pki/pki-tomcat/alias/ -L Certificate Nickname Trust Attributes SSL,S/MIME,JAR/XPI XXX Certificate Authority CT,c, Server-Cert cert-pki-ca u,u,u auditSigningCert cert-pki-ca u,u,Pu caSigningCert cert-pki-ca CTu,Cu,Cu ocspSigningCert cert-pki-ca u,u,u subsystemCert cert-pki-ca u,u,u Run on the destination ipa server: $ certutil -d /etc/pki/pki-tomcat/alias/ -L Certificate Nickname Trust Attributes SSL,S/MIME,JAR/XPI > Finally, can you check that LDAPS is running on port 636 on the replica > where you're trying to install the CA (i.e. by nmap localhost)? Run on the new replica: $ nmap localhost Starting Nmap 6.40 ( http://nmap.org ) at 2016-09-21 15:29 UTC Nmap scan report for localhost (127.0.0.1) Host is up (0.0000040s latency). Other addresses for localhost (not scanned): 127.0.0.1 Not shown: 995 closed ports PORT STATE SERVICE 22/tcp open ssh 25/tcp open smtp 53/tcp open domain 389/tcp open ldap 636/tcp open ldapssl > > -- > Tomas Krizek -- Korey From abokovoy at redhat.com Wed Sep 21 15:49:00 2016 From: abokovoy at redhat.com (Alexander Bokovoy) Date: Wed, 21 Sep 2016 18:49:00 +0300 Subject: [Freeipa-users] Samba Server setup In-Reply-To: <08B39CF6-36E9-41C4-88A8-F1E4077B75AF@bsd.uchicago.edu> References: <73C11E05-045F-4324-9733-44C370584232@bsd.uchicago.edu> <20160915180606.v2z6dkdw3gltem34@redhat.com> <9A3517B6-CFFB-47ED-BDF2-BE3EE0D06EDE@bsd.uchicago.edu> <20160916050435.l4k6rshziftp6pzd@redhat.com> <38F25AFE-C5E8-4CE9-BD18-26949884625A@bsd.uchicago.edu> <20160916170216.jxpico2fscjcswq4@redhat.com> <08B39CF6-36E9-41C4-88A8-F1E4077B75AF@bsd.uchicago.edu> Message-ID: <20160921154900.ti6d224dhtyjfdk5@redhat.com> On Wed, 21 Sep 2016, Brook, Andy [CRI] wrote: >On 9/16/16, 12:02 PM, "Alexander Bokovoy" wrote: > > On Fri, 16 Sep 2016, Brook, Andy [CRI] wrote: > > You can replace actual hostnames/realm names/IP addresses by something more generic > > in the output when sending to the list, but please do it consistently. > > > >I?m sorry. I thought I had been consistent when making changes, but > >from your response, it looks like I wasn?t. I?m sorry about that. I got > >yelled at by our security team last time we sent logs to a public list > >that had any type of identifiable information in them, so it?s sort of > >a new process for me. I think I have it down now. > > > >The results of the commands are here: http://pastebin.com/PRwr7wv6 > So IPA side works fine -- on IPA client you can kinit as AD user and > then obtain cross-realm TGT to IPA realm and use that cross-realm TGT to > request a service ticket to cifs/... service. That's good. > > You need to identify what happens on AD side. A possible issue is that > name suffix routing to IPA domain is disabled. > > Can you provide output of netdom.exe run on Windows side: > > netdom trust addom.domain /namesuffixes: ipa.domain > > You should get something like example 28 on the page > https://msdn.microsoft.com/en-us/library/cc776879(v=ws.10).aspx > >Thank you for this. I went to run the command and kept getting an >?Incorrect parameter? error. After that I talked to one of our Active >Directory admins and he mentioned that we are working on resolving a >disjoint namespace error on addom. I don?t understand enough about it, >but do know that it can cause issues with Kerberos authentication >across domains. That should get fixed soon. Once that gets fixed, I?ll >test again. > >I have one more related question. The instruction page states that >NTLMSSP authentication isn?t working as of yet, as well as you >mentioned it earlier in this thread. Is there a bug or feature request >that is tracking that? https://fedorahosted.org/sssd/ticket/2012 is a tracker. We have gss-ntlmssp implemented but it depends on winbindd and there are things which are not done yet in making sssd/winbindd co-working. We had few talks about possible ways to integrate around that topic at SambaXP 2016 conference: https://sambaxp.org/archive_data/SambaXP2016-SLIDES/wed/sambaxp2016-wed-Simo_Sorce-SambaAndLinuxDistributionsLetsIntegrateBetter.pdf https://sambaxp.org/archive_data/SambaXP2016-SLIDES/wed/sambaxp2016-wed-Sumit_Bose-WinbindAndSSSDCanTheyBeFriends/ -- / Alexander Bokovoy From julliot at ljll.math.upmc.fr Wed Sep 21 18:20:43 2016 From: julliot at ljll.math.upmc.fr (Sebastien Julliot) Date: Wed, 21 Sep 2016 20:20:43 +0200 Subject: [Freeipa-users] FreeIPA client installation on ubuntu, 14.04 In-Reply-To: References: Message-ID: Hello Deepak, If you know in advance what infos you want to enter input, you can try putting them in a file "inputs" and execute apt-get install freeipa-client -y < inputs > I am trying to install freeipa client on my ubuntu client via ansible > script. I have "apt-get update" and "apt-get install freeipa-client -y" > these basic commands added in my playbook but the problem is when i run > "apt-get install freeipa-client" with or without -y option it opens up > some graphical interface confirming the IPA realm and other details. I > did not find any option with in "apt-get install freeipa-client"to make > it deployment unattended. Can anyone please tell me the how i can > automate ipa-client installation on ubuntu? From rcritten at redhat.com Wed Sep 21 18:43:08 2016 From: rcritten at redhat.com (Rob Crittenden) Date: Wed, 21 Sep 2016 14:43:08 -0400 Subject: [Freeipa-users] Replication issues (was Me Again) In-Reply-To: <722fd8d1-5ede-12dc-45d6-952be7e53dcd@brownpapertickets.com> References: <722fd8d1-5ede-12dc-45d6-952be7e53dcd@brownpapertickets.com> Message-ID: <57E2D4BC.5060102@redhat.com> Ian Harding wrote: > I used to have a lot of replicas, but like a house of cards, it all came > crashing down. > > I was down to two, that seemed to be replicating, but last few days I've > noticed that they haven't always been. > > freeipa-sea.bpt.rocks is where we do all our admin. > seattlenfs.bpt.rocks is also up and running and can be used for > authentication. > > When I noticed that logins were failing after password changes I did > > ipa-replica-manage re-initialize --from=freeipa-sea.bpt.rocks Note that this is the hammer approach. Diagnosing the underlying issues would probably be best. What is the output of: $ rpm -q 389-ds-base freeipa-server (or ipa-server depending on distro). That will give us the info needed to suggest what else to look for. rob > > on seattlenfs.bpt.rocks and replication appeared to be working again. > > Well it happened again, and this time I peeked at the dirsrv errors log > and saw some scary things having to do with the CA. > > [19/Sep/2016:02:55:50 -0700] slapd_ldap_sasl_interactive_bind - Error: > could not perform interactive bind for id [] mech [GSSAPI]: LDAP error > -1 (Can't contact LDAP server) ((null)) errno 0 (Success) > [19/Sep/2016:02:55:50 -0700] slapi_ldap_bind - Error: could not perform > interactive bind for id [] authentication mechanism [GSSAPI]: error -1 > (Can't contact LDAP server) > [19/Sep/2016:02:55:50 -0700] NSMMReplicationPlugin - > agmt="cn=meTofreeipa-sea.bpt.rocks" (freeipa-sea:389): Replication bind > with GSSAPI auth failed: LDAP error -1 (Can't contact LDAP server) () > [19/Sep/2016:02:56:04 -0700] NSMMReplicationPlugin - > agmt="cn=meTofreeipa-sea.bpt.rocks" (freeipa-sea:389): Replication bind > with GSSAPI auth resumed > [20/Sep/2016:10:18:25 -0700] NSMMReplicationPlugin - > multimaster_be_state_change: replica dc=bpt,dc=rocks is going offline; > disabling replication > [20/Sep/2016:10:18:26 -0700] - WARNING: Import is running with > nsslapd-db-private-import-mem on; No other process is allowed to access > the database > [20/Sep/2016:10:18:29 -0700] - import userRoot: Workers finished; > cleaning up... > [20/Sep/2016:10:18:29 -0700] - import userRoot: Workers cleaned up. > [20/Sep/2016:10:18:29 -0700] - import userRoot: Indexing complete. > Post-processing... > [20/Sep/2016:10:18:29 -0700] - import userRoot: Generating > numsubordinates (this may take several minutes to complete)... > [20/Sep/2016:10:18:29 -0700] - import userRoot: Generating > numSubordinates complete. > [20/Sep/2016:10:18:29 -0700] - import userRoot: Gathering ancestorid > non-leaf IDs... > [20/Sep/2016:10:18:29 -0700] - import userRoot: Finished gathering > ancestorid non-leaf IDs. > [20/Sep/2016:10:18:29 -0700] - import userRoot: Creating ancestorid > index (new idl)... > [20/Sep/2016:10:18:29 -0700] - import userRoot: Created ancestorid index > (new idl). > [20/Sep/2016:10:18:29 -0700] - import userRoot: Flushing caches... > [20/Sep/2016:10:18:29 -0700] - import userRoot: Closing files... > [20/Sep/2016:10:18:29 -0700] - import userRoot: Import complete. > Processed 1324 entries in 3 seconds. (441.33 entries/sec) > [20/Sep/2016:10:18:29 -0700] NSMMReplicationPlugin - > multimaster_be_state_change: replica dc=bpt,dc=rocks is coming online; > enabling replication > [20/Sep/2016:10:18:29 -0700] NSMMReplicationPlugin - replica_reload_ruv: > Warning: new data for replica dc=bpt,dc=rocks does not match the data in > the changelog. > Recreating the changelog file. This could affect replication with > replica's consumers in which case the consumers should be reinitialized. > [20/Sep/2016:10:18:29 -0700] NSACLPlugin - The ACL target > cn=groups,cn=compat,dc=bpt,dc=rocks does not exist > [20/Sep/2016:10:18:29 -0700] NSACLPlugin - The ACL target > cn=computers,cn=compat,dc=bpt,dc=rocks does not exist > [20/Sep/2016:10:18:29 -0700] NSACLPlugin - The ACL target > cn=ng,cn=compat,dc=bpt,dc=rocks does not exist > [20/Sep/2016:10:18:29 -0700] NSACLPlugin - The ACL target > ou=sudoers,dc=bpt,dc=rocks does not exist > [20/Sep/2016:10:18:29 -0700] NSACLPlugin - The ACL target > cn=users,cn=compat,dc=bpt,dc=rocks does not exist > [20/Sep/2016:10:18:29 -0700] NSACLPlugin - The ACL target > cn=vaults,cn=kra,dc=bpt,dc=rocks does not exist > [20/Sep/2016:10:18:29 -0700] NSACLPlugin - The ACL target > cn=vaults,cn=kra,dc=bpt,dc=rocks does not exist > [20/Sep/2016:10:18:29 -0700] NSACLPlugin - The ACL target > cn=vaults,cn=kra,dc=bpt,dc=rocks does not exist > [20/Sep/2016:10:18:29 -0700] NSACLPlugin - The ACL target > cn=vaults,cn=kra,dc=bpt,dc=rocks does not exist > [20/Sep/2016:10:18:29 -0700] NSACLPlugin - The ACL target > cn=vaults,cn=kra,dc=bpt,dc=rocks does not exist > [20/Sep/2016:10:18:29 -0700] NSACLPlugin - The ACL target > cn=vaults,cn=kra,dc=bpt,dc=rocks does not exist > [20/Sep/2016:10:18:29 -0700] NSACLPlugin - The ACL target > cn=vaults,cn=kra,dc=bpt,dc=rocks does not exist > [20/Sep/2016:10:18:29 -0700] NSACLPlugin - The ACL target > cn=vaults,cn=kra,dc=bpt,dc=rocks does not exist > [20/Sep/2016:10:18:29 -0700] NSACLPlugin - The ACL target > cn=vaults,cn=kra,dc=bpt,dc=rocks does not exist > [20/Sep/2016:10:18:29 -0700] NSACLPlugin - The ACL target > cn=vaults,cn=kra,dc=bpt,dc=rocks does not exist > [20/Sep/2016:10:18:29 -0700] NSACLPlugin - The ACL target > cn=vaults,cn=kra,dc=bpt,dc=rocks does not exist > [20/Sep/2016:10:18:29 -0700] NSACLPlugin - The ACL target > cn=ad,cn=etc,dc=bpt,dc=rocks does not exist > [20/Sep/2016:10:18:29 -0700] NSACLPlugin - The ACL target > cn=casigningcert cert-pki-ca,cn=ca_renewal,cn=ipa,cn=etc,dc=bpt,dc=rocks > does not exist > [20/Sep/2016:10:18:29 -0700] NSACLPlugin - The ACL target > cn=casigningcert cert-pki-ca,cn=ca_renewal,cn=ipa,cn=etc,dc=bpt,dc=rocks > does not exist > > Any clues what that's about? The high numbered RUV are for CA RUV I > assume. Those other machines are still installed but IPA services > turned off so people can't authenticate to them because replication was > not working. > > If there was some way I could go down to one machine (freeipa-sea) and > get it all cleaned up, no ghost RUV, everything quiet in the logs, and > start over creating replicas, I would love to do that. Seems like > someone smarter than me could stop the server, back up the ldif files > and edit them to make all the cruft go away. > > Is that possible? I've started a conversation with RedHat about getting > on board with the official bits and support but I want to know if it's > possible/cost effective to do what I describe, along with, I assume, > migrating to the official versions of Spacewalk and FreeIPA. > > Thanks! > > Ian > From ianh at brownpapertickets.com Wed Sep 21 18:53:55 2016 From: ianh at brownpapertickets.com (Ian Harding) Date: Wed, 21 Sep 2016 11:53:55 -0700 Subject: [Freeipa-users] Replication issues (was Me Again) In-Reply-To: <57E2D4BC.5060102@redhat.com> References: <722fd8d1-5ede-12dc-45d6-952be7e53dcd@brownpapertickets.com> <57E2D4BC.5060102@redhat.com> Message-ID: <22975394-5382-8c6b-1fdb-abf25c90787a@brownpapertickets.com> On 09/21/2016 11:43 AM, Rob Crittenden wrote: > Ian Harding wrote: >> I used to have a lot of replicas, but like a house of cards, it all came >> crashing down. >> >> I was down to two, that seemed to be replicating, but last few days I've >> noticed that they haven't always been. >> >> freeipa-sea.bpt.rocks is where we do all our admin. >> seattlenfs.bpt.rocks is also up and running and can be used for >> authentication. >> >> When I noticed that logins were failing after password changes I did >> >> ipa-replica-manage re-initialize --from=freeipa-sea.bpt.rocks > > Note that this is the hammer approach. Diagnosing the underlying issues > would probably be best. > > What is the output of: > > $ rpm -q 389-ds-base freeipa-server > > (or ipa-server depending on distro). > > That will give us the info needed to suggest what else to look for. > > rob > Hammer sounds pretty good. # rpm -q 389-ds-base ipa-server 389-ds-base-1.3.4.0-33.el7_2.x86_64 ipa-server-4.2.0-15.0.1.el7.centos.19.x86_64 >> >> on seattlenfs.bpt.rocks and replication appeared to be working again. >> >> Well it happened again, and this time I peeked at the dirsrv errors log >> and saw some scary things having to do with the CA. >> >> [19/Sep/2016:02:55:50 -0700] slapd_ldap_sasl_interactive_bind - Error: >> could not perform interactive bind for id [] mech [GSSAPI]: LDAP error >> -1 (Can't contact LDAP server) ((null)) errno 0 (Success) >> [19/Sep/2016:02:55:50 -0700] slapi_ldap_bind - Error: could not perform >> interactive bind for id [] authentication mechanism [GSSAPI]: error -1 >> (Can't contact LDAP server) >> [19/Sep/2016:02:55:50 -0700] NSMMReplicationPlugin - >> agmt="cn=meTofreeipa-sea.bpt.rocks" (freeipa-sea:389): Replication bind >> with GSSAPI auth failed: LDAP error -1 (Can't contact LDAP server) () >> [19/Sep/2016:02:56:04 -0700] NSMMReplicationPlugin - >> agmt="cn=meTofreeipa-sea.bpt.rocks" (freeipa-sea:389): Replication bind >> with GSSAPI auth resumed >> [20/Sep/2016:10:18:25 -0700] NSMMReplicationPlugin - >> multimaster_be_state_change: replica dc=bpt,dc=rocks is going offline; >> disabling replication >> [20/Sep/2016:10:18:26 -0700] - WARNING: Import is running with >> nsslapd-db-private-import-mem on; No other process is allowed to access >> the database >> [20/Sep/2016:10:18:29 -0700] - import userRoot: Workers finished; >> cleaning up... >> [20/Sep/2016:10:18:29 -0700] - import userRoot: Workers cleaned up. >> [20/Sep/2016:10:18:29 -0700] - import userRoot: Indexing complete. >> Post-processing... >> [20/Sep/2016:10:18:29 -0700] - import userRoot: Generating >> numsubordinates (this may take several minutes to complete)... >> [20/Sep/2016:10:18:29 -0700] - import userRoot: Generating >> numSubordinates complete. >> [20/Sep/2016:10:18:29 -0700] - import userRoot: Gathering ancestorid >> non-leaf IDs... >> [20/Sep/2016:10:18:29 -0700] - import userRoot: Finished gathering >> ancestorid non-leaf IDs. >> [20/Sep/2016:10:18:29 -0700] - import userRoot: Creating ancestorid >> index (new idl)... >> [20/Sep/2016:10:18:29 -0700] - import userRoot: Created ancestorid index >> (new idl). >> [20/Sep/2016:10:18:29 -0700] - import userRoot: Flushing caches... >> [20/Sep/2016:10:18:29 -0700] - import userRoot: Closing files... >> [20/Sep/2016:10:18:29 -0700] - import userRoot: Import complete. >> Processed 1324 entries in 3 seconds. (441.33 entries/sec) >> [20/Sep/2016:10:18:29 -0700] NSMMReplicationPlugin - >> multimaster_be_state_change: replica dc=bpt,dc=rocks is coming online; >> enabling replication >> [20/Sep/2016:10:18:29 -0700] NSMMReplicationPlugin - replica_reload_ruv: >> Warning: new data for replica dc=bpt,dc=rocks does not match the data in >> the changelog. >> Recreating the changelog file. This could affect replication with >> replica's consumers in which case the consumers should be reinitialized. >> [20/Sep/2016:10:18:29 -0700] NSACLPlugin - The ACL target >> cn=groups,cn=compat,dc=bpt,dc=rocks does not exist >> [20/Sep/2016:10:18:29 -0700] NSACLPlugin - The ACL target >> cn=computers,cn=compat,dc=bpt,dc=rocks does not exist >> [20/Sep/2016:10:18:29 -0700] NSACLPlugin - The ACL target >> cn=ng,cn=compat,dc=bpt,dc=rocks does not exist >> [20/Sep/2016:10:18:29 -0700] NSACLPlugin - The ACL target >> ou=sudoers,dc=bpt,dc=rocks does not exist >> [20/Sep/2016:10:18:29 -0700] NSACLPlugin - The ACL target >> cn=users,cn=compat,dc=bpt,dc=rocks does not exist >> [20/Sep/2016:10:18:29 -0700] NSACLPlugin - The ACL target >> cn=vaults,cn=kra,dc=bpt,dc=rocks does not exist >> [20/Sep/2016:10:18:29 -0700] NSACLPlugin - The ACL target >> cn=vaults,cn=kra,dc=bpt,dc=rocks does not exist >> [20/Sep/2016:10:18:29 -0700] NSACLPlugin - The ACL target >> cn=vaults,cn=kra,dc=bpt,dc=rocks does not exist >> [20/Sep/2016:10:18:29 -0700] NSACLPlugin - The ACL target >> cn=vaults,cn=kra,dc=bpt,dc=rocks does not exist >> [20/Sep/2016:10:18:29 -0700] NSACLPlugin - The ACL target >> cn=vaults,cn=kra,dc=bpt,dc=rocks does not exist >> [20/Sep/2016:10:18:29 -0700] NSACLPlugin - The ACL target >> cn=vaults,cn=kra,dc=bpt,dc=rocks does not exist >> [20/Sep/2016:10:18:29 -0700] NSACLPlugin - The ACL target >> cn=vaults,cn=kra,dc=bpt,dc=rocks does not exist >> [20/Sep/2016:10:18:29 -0700] NSACLPlugin - The ACL target >> cn=vaults,cn=kra,dc=bpt,dc=rocks does not exist >> [20/Sep/2016:10:18:29 -0700] NSACLPlugin - The ACL target >> cn=vaults,cn=kra,dc=bpt,dc=rocks does not exist >> [20/Sep/2016:10:18:29 -0700] NSACLPlugin - The ACL target >> cn=vaults,cn=kra,dc=bpt,dc=rocks does not exist >> [20/Sep/2016:10:18:29 -0700] NSACLPlugin - The ACL target >> cn=vaults,cn=kra,dc=bpt,dc=rocks does not exist >> [20/Sep/2016:10:18:29 -0700] NSACLPlugin - The ACL target >> cn=ad,cn=etc,dc=bpt,dc=rocks does not exist >> [20/Sep/2016:10:18:29 -0700] NSACLPlugin - The ACL target >> cn=casigningcert cert-pki-ca,cn=ca_renewal,cn=ipa,cn=etc,dc=bpt,dc=rocks >> does not exist >> [20/Sep/2016:10:18:29 -0700] NSACLPlugin - The ACL target >> cn=casigningcert cert-pki-ca,cn=ca_renewal,cn=ipa,cn=etc,dc=bpt,dc=rocks >> does not exist >> >> Any clues what that's about? The high numbered RUV are for CA RUV I >> assume. Those other machines are still installed but IPA services >> turned off so people can't authenticate to them because replication was >> not working. >> >> If there was some way I could go down to one machine (freeipa-sea) and >> get it all cleaned up, no ghost RUV, everything quiet in the logs, and >> start over creating replicas, I would love to do that. Seems like >> someone smarter than me could stop the server, back up the ldif files >> and edit them to make all the cruft go away. >> >> Is that possible? I've started a conversation with RedHat about getting >> on board with the official bits and support but I want to know if it's >> possible/cost effective to do what I describe, along with, I assume, >> migrating to the official versions of Spacewalk and FreeIPA. >> >> Thanks! >> >> Ian >> > -- Ian Harding IT Director Brown Paper Tickets 1-800-838-3006 ext 7186 http://www.brownpapertickets.com From pgb205 at yahoo.com Wed Sep 21 21:25:59 2016 From: pgb205 at yahoo.com (pgb205) Date: Wed, 21 Sep 2016 21:25:59 +0000 (UTC) Subject: [Freeipa-users] down master still in ldap, prevents re-enrolement References: <1562005586.2453821.1474493159566.ref@mail.yahoo.com> Message-ID: <1562005586.2453821.1474493159566@mail.yahoo.com> topology prior to deletion master1<->master2 master2 deleted with ipa-server --uninstall command During re-installation I get error that the replication agreement still exists on master1.I do see this using ipa-replica-manage list. Tried deleting replication agreement withipa-replica-manage disconnect but receive 'no such replication agreement exist' Force deletion and cleanup do not workreceive?unexpected error: Server is unwilling to perform: database is read-only? removing directly from ldap gives me:?ldapdelete -r -x -D "cn=Directory Manager" -W 'cn=fqdn,cn=masters,cn=ipa,cn=etc,dc=domain,dc=com' Enter LDAP Password:ldap_delete: Server is unwilling to perform (53)ldap_delete: Server is unwilling to perform (53)? ? ? ? additional info: database is read-only But I am not sure if I'm not using correct path or if it's something else. Might be related to ?Bug 826677 ? IPA cannot remove disconnected replica data to reconnect | | | Bug 826677 ? IPA cannot remove disconnected replica data to reconnect | | | -------------- next part -------------- An HTML attachment was scrubbed... URL: From datakid at gmail.com Wed Sep 21 22:53:12 2016 From: datakid at gmail.com (Lachlan Musicman) Date: Thu, 22 Sep 2016 08:53:12 +1000 Subject: [Freeipa-users] sssd.conf - the server and host-client relationship In-Reply-To: <20160920071144.GA2435@10.4.128.1> References: <20160920071144.GA2435@10.4.128.1> Message-ID: My translations of your comments are in line, if you could correct, I'd appreciate that. On 20 September 2016 at 17:11, Lukas Slebodnik wrote: > >---------------------- > >[domain/unixdev.etc] > >ignore_group_members = True > It was probably set as a result of performance tuning. > > >ldap_purge_cache_timeout = 0 > That's default since 1.13.0 > > >subdomain_inherit = ignore_group_members, ldap_purge_cache_timeout > that's specific option for sssd on IPA server > I presume your comment suggests ignore_group_members is no longer needed, and since the lpct=0 is now default, then subdomain_inherit is also superfluous? > >selinux_provider = none > It was probably set as a workaround of bug which have been already > fixed. > We set this because of an error in libsemanage, but I think that was an upstream (selinux) issue? https://www.redhat.com/archives/freeipa-users/2016-July/msg00244.html Not sure if I should disable just yet - was this fixed? > > >ipa_server_mode = True > that's specific option for sssd on IPA server > > I take it that this means it's still used. > >sudo_provider = ldap > >ldap_uri = ldap://vmdv-linuxidm1.unixdev.petermac.org.au > >ldap_sudo_search_base = or=sudoers,dc=unixdev,dc=petermac,dc=org,dc=au > >ldap_sasl_mech = GSSAPI > >ldap_sasl_authid = host/vmdv-linuxidm1.unixdev.petermac.org.au > >ldap_sasl_realm = UNIXDEV.PETERMAC.ORG.AU > >krb5_server = vmdv-linuxidm1.unixdev.petermac.org.au > Previous 7 options are not required since sssd-1.10 > Yep, I added those because of disconnect between the different info sources made it hard to tell what was canonical, so I followed the red hat guide: https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/6/html/Deployment_Guide/sssd-ldap-sudo.html mostly because I didn't quite understand the sssd-sudo man page (because sometimes I find man pages obtuse), but also there was an inconsistency with the local man page and the die.net mirror https://linux.die.net/man/5/sssd-sudo and this howto https://blog-rcritten.rhcloud.com/?p=52 > > > >[sssd] > >config_file_version = 2 > >domains = unixdev.etc > > > >[nss] > >memcache_timeout = 600 > This option is se by ipa-*-install on ipa server mode. > These I will leave. Cheers L. -------------- next part -------------- An HTML attachment was scrubbed... URL: From natxo.asenjo at gmail.com Thu Sep 22 05:56:02 2016 From: natxo.asenjo at gmail.com (Natxo Asenjo) Date: Thu, 22 Sep 2016 07:56:02 +0200 Subject: [Freeipa-users] replica added, but clients still try renewing certificates with old master In-Reply-To: References: <89facc8a-74d1-5c00-dbd9-6cafa20f0e80@redhat.com> Message-ID: On Wed, Sep 21, 2016 at 5:06 PM, Natxo Asenjo wrote: > ok, done. > > In fact, change both the domain as the xmlrpc_uri directives in the global > section was necessary. Now It worked :-) > I meant the server, not the domain options obviously. -- Groeten, natxo -------------- next part -------------- An HTML attachment was scrubbed... URL: From deepak_dimri at hotmail.com Thu Sep 22 08:17:21 2016 From: deepak_dimri at hotmail.com (Deepak Dimri) Date: Thu, 22 Sep 2016 08:17:21 +0000 Subject: [Freeipa-users] key + 2FA (password+OTP) is not working Message-ID: Hi All, I am trying hard to get my 2FA working with FreeIPA but every effort of mine going waste! I have referred earlier forum emails but could not find any good reply on the issue i am facing. This is what i am trying I have a test user created in my IPA server enabled with Two factor authentication (password + OTP) and has ssh public key added in its profile. I want this test user to ssh into my ipa client (ubuntu 14.04) using key + password + OTP. I woudl ceryainly prefer just the key+ OTP only ( no password) but that seems far sighted as i cannot even make it work with what it supposed to work password + OTP. My /etc/ssh/sshd_conf file has almost everything default except i added these two lines at the end of it Match Group testusergroup AuthenticationMethods publickey,password:pam publickey,keyboard-interactive:pam i also tried with below but no luck Match Group testusergroup AuthenticationMethods publickey,keyboard-interactive my /etc/pam.d/sshd has these two changes, rest i kept default: # Standard Un*x authentication. #@include common-auth auth required pam_sss.so Now when i try to ssh into ipa client i either keep getting promptS for the password or it gets into a loop asking me to change the password ;complaining falsely that it has expired. I have tried multiple combinations of configurations by referring earlier email threads but none i found helpful. I cant make simple 2FA login to work with freeIPA. Normal password and key works just fine. its the 2FA which does not work for me. Would really be thankful if some one can help me with this issue.. is there any good freeIPA 2FA configuration document that i can refer? What should the steps for it work seamlessly? Many Thanks, Deepak -------------- next part -------------- An HTML attachment was scrubbed... URL: From sbose at redhat.com Thu Sep 22 08:45:34 2016 From: sbose at redhat.com (Sumit Bose) Date: Thu, 22 Sep 2016 10:45:34 +0200 Subject: [Freeipa-users] key + 2FA (password+OTP) is not working In-Reply-To: References: Message-ID: <20160922084534.GI6019@p.Speedport_W_724V_Typ_A_05011603_00_009> On Thu, Sep 22, 2016 at 08:17:21AM +0000, Deepak Dimri wrote: > Hi All, > > > I am trying hard to get my 2FA working with FreeIPA but every effort of mine going waste! I have referred earlier forum emails but could not find any good reply on the issue i am facing. > > > This is what i am trying > > > I have a test user created in my IPA server enabled with Two factor authentication (password + OTP) and has ssh public key added in its profile. I want this test user to ssh into my ipa client (ubuntu 14.04) using key + password + OTP. I woudl ceryainly prefer just the key+ OTP only ( no password) but that seems far sighted as i cannot even make it work with what it supposed to work password + OTP. > > > My /etc/ssh/sshd_conf file has almost everything default except i added these two lines at the end of it > > Match Group testusergroup > > AuthenticationMethods publickey,password:pam publickey,keyboard-interactive:pam > > i also tried with below but no luck > > Match Group testusergroup > > AuthenticationMethods publickey,keyboard-interactive > > > my /etc/pam.d/sshd has these two changes, rest i kept default: > > > # Standard Un*x authentication. > > #@include common-auth > > > auth required pam_sss.so > > > Now when i try to ssh into ipa client i either keep getting promptS for the password or it gets into a loop asking me to change the password ;complaining falsely that it has expired. I have tried multiple combinations of configurations by referring earlier email threads but none i found helpful. I cant make simple 2FA login to work with freeIPA. Normal password and key works just fine. its the 2FA which does not work for me. > > > Would really be thankful if some one can help me with this issue.. is there any good freeIPA 2FA configuration document that i can refer? Please add debug_level=10 to the [pam] and [domain/...] section of sssd.conf, restart SSSD, re-run the authentication and send the generated debug logs together with your sssd.conf and the full /etc/pam.d/sshd. Please see https://fedorahosted.org/sssd/wiki/Troubleshooting for details. > > What should the steps for it work seamlessly? In general it should work out of the box with SSSD's ipa provider. bye, Sumit > > > Many Thanks, > > Deepak > > -- > Manage your subscription for the Freeipa-users mailing list: > https://www.redhat.com/mailman/listinfo/freeipa-users > Go to http://freeipa.org for more info on the project From flo at redhat.com Thu Sep 22 08:52:57 2016 From: flo at redhat.com (Florence Blanc-Renaud) Date: Thu, 22 Sep 2016 10:52:57 +0200 Subject: [Freeipa-users] CA Fails to build Replica (w/External CA) In-Reply-To: References: <64066e0c-a299-cfa8-f1e5-1585692c15ae@redhat.com> Message-ID: Hi Korey, I believe that you are hitting Dogtag issue #2255 [1]. The file /tmp/ca.p12 probably doesn't contain the trust flags for some certificates. You can check by running pki pkcs12-cert-find --pkcs12-file /tmp/ca.p12 --pkcs12-password password and see if the output displays "Trust Flags: xxx" for all the certs. Flo. [1] https://fedorahosted.org/pki/ticket/2255 On 09/21/2016 05:38 PM, Korey Chapman wrote: > On Wed, Sep 21, 2016 at 6:47 AM, Tomas Krizek wrote: >> On 09/21/2016 02:13 AM, Korey Chapman wrote: >> >> Hello list, >> >> I'm currently attempting to add a second CA server to our IPA cluster (all >> servers Centos 7.2 with IPA 4.2.0). However, it is failing no matter how I >> try to setup the CA (ipa-replica-install with --setup-ca or >> ipa-replica-install followed by ipa-ca-install). The only useful thing in >> the logs is an error about a missing key for "trust_flags" in the pki setup. >> Our infrastructure uses FreeIPA with an external CA. >> >> Any ideas/help would be greatly appreciated. Here are the logs snips from my >> most recent attempt: >> >> Command output snip from "ipa-replica-install >> /root/replica-info-auth-002.XXX.gpg --setup-ca" >> Configuring certificate server (pki-tomcatd). Estimated time: 3 minutes 30 >> seconds >> [1/24]: creating certificate server user >> [2/24]: configuring certificate server instance >> ipa.ipaserver.install.cainstance.CAInstance: CRITICAL Failed to configure CA >> instance: Command ''/usr/sbin/pkispawn' '-s' 'CA' '-f' '/tmp/tmpYofMPt'' >> returned non-zero exit status 1 >> ipa.ipaserver.install.cainstance.CAInstance: CRITICAL See the installation >> logs and the following files/directories for more information: >> ipa.ipaserver.install.cainstance.CAInstance: CRITICAL >> /var/log/pki-ca-install.log >> ipa.ipaserver.install.cainstance.CAInstance: CRITICAL >> /var/log/pki/pki-tomcat >> [error] RuntimeError: CA configuration failed. >> Your system may be partly configured. >> Run /usr/sbin/ipa-server-install --uninstall to clean up. >> >> ipa.ipapython.install.cli.install_tool(Replica): ERROR CA configuration >> failed >> >> >> Log snip from ipareplica-install.log: >> >> 2016-09-20T23:42:27Z DEBUG Starting external process >> 2016-09-20T23:42:27Z DEBUG args='/usr/sbin/pkispawn' '-s' 'CA' '-f' >> '/tmp/tmpYofMPt' >> 2016-09-20T23:42:31Z DEBUG Process finished, return code=1 >> 2016-09-20T23:42:31Z DEBUG stdout=Log file: >> /var/log/pki/pki-ca-spawn.20160920234227.log >> Loading deployment configuration from /tmp/tmpYofMPt. >> Installing CA into /var/lib/pki/pki-tomcat. >> Storing deployment configuration into >> /etc/sysconfig/pki/tomcat/pki-tomcat/ca/deployment.cfg. >> >> Installation failed. >> >> >> 2016-09-20T23:42:31Z DEBUG >> stderr=/usr/lib/python2.7/site-packages/urllib3/connectionpool.py:769: >> InsecureRequestWarning: Unverified HTTPS request is being made. Adding >> certificate verification is strongly advised. See: >> https://urllib3.readthedocs.org/en/latest/security.html >> InsecureRequestWarning) >> Traceback (most recent call last): >> File "/bin/pki", line 254, in >> cli.execute(sys.argv) >> File "/bin/pki", line 240, in execute >> module.execute(module_args) >> File "/usr/lib/python2.7/site-packages/pki/cli/__init__.py", line 195, in >> execute >> module.execute(module_args) >> File "/usr/lib/python2.7/site-packages/pki/cli/pkcs12.py", line 222, in >> execute >> trust_flags = cert_info['trust_flags'] >> KeyError: 'trust_flags' >> >> >> -- >> Korey >> >> >> Hi Korey, >> >> could you check if there is any more info in /var/log/pki/pki-ca-spawn log? > > Nothing really useful I see in the spawn log: > 2016-09-20 23:42:31 pkispawn : DEBUG ....... Error Type: > CalledProcessError > 2016-09-20 23:42:31 pkispawn : DEBUG ....... Error Message: > Command '['pki', '-d', '/etc/pki/pki-tomcat/alias', '-C', > '/etc/pki/pki-tomcat/pfile', 'pkcs12-import', '--pkcs12-file', > '/tmp/ca.p12', '--pkcs12-password-file', > '/tmp/tmps5OOav/password.txt', '--no-user-certs']' returned non-zero > exit status 1 > 2016-09-20 23:42:31 pkispawn : DEBUG ....... File > "/usr/sbin/pkispawn", line 597, in main > rv = scriptlet.spawn(deployer) > File "/usr/lib/python2.7/site-packages/pki/server/deployment/scriptlets/security_databases.py", > line 104, in spawn > no_user_certs=True) > File "/usr/lib/python2.7/site-packages/pki/nssdb.py", line 538, in > import_pkcs12 > subprocess.check_call(cmd) > File "/usr/lib64/python2.7/subprocess.py", line 542, in check_call > raise CalledProcessError(retcode, cmd) > >> >> It might also be helpful verify if correct trust flags are set in nssdb: >> certutil -d /etc/pki/pki-tomcat/alias/ -L >> > > Run on the source ipa server (current CA server): > $ certutil -d /etc/pki/pki-tomcat/alias/ -L > > Certificate Nickname Trust Attributes > SSL,S/MIME,JAR/XPI > > XXX Certificate Authority CT,c, > Server-Cert cert-pki-ca u,u,u > auditSigningCert cert-pki-ca u,u,Pu > caSigningCert cert-pki-ca CTu,Cu,Cu > ocspSigningCert cert-pki-ca u,u,u > subsystemCert cert-pki-ca u,u,u > > > Run on the destination ipa server: > $ certutil -d /etc/pki/pki-tomcat/alias/ -L > > Certificate Nickname Trust Attributes > SSL,S/MIME,JAR/XPI > >> Finally, can you check that LDAPS is running on port 636 on the replica >> where you're trying to install the CA (i.e. by nmap localhost)? > > Run on the new replica: > $ nmap localhost > > Starting Nmap 6.40 ( http://nmap.org ) at 2016-09-21 15:29 UTC > Nmap scan report for localhost (127.0.0.1) > Host is up (0.0000040s latency). > Other addresses for localhost (not scanned): 127.0.0.1 > Not shown: 995 closed ports > PORT STATE SERVICE > 22/tcp open ssh > 25/tcp open smtp > 53/tcp open domain > 389/tcp open ldap > 636/tcp open ldapssl > >> >> -- >> Tomas Krizek > > > From harenberg at physik.uni-wuppertal.de Thu Sep 22 09:39:04 2016 From: harenberg at physik.uni-wuppertal.de (Torsten Harenberg) Date: Thu, 22 Sep 2016 11:39:04 +0200 Subject: [Freeipa-users] OT: slow NFS4 (kerberos) since moving to IPA Message-ID: <0496bcaf-0339-9b56-23b2-a0eea228a0c3@physik.uni-wuppertal.de> Dear all, please excuse, this is slightly off-topic. We run an NFS4 server (running Ubuntu 14.04) serving about 40 clients on (running Ubuntu and Mint as well as a few CentOS 6 nodes). We moved all our infrastructure to IPA to have Kerberos security. File systems are mounted with home -fstype=nfs4,rw,sec=krb5i using the automounter. That all works fine. Now the clients show timeouts in the system logs physikd98kuech kernel: [ 595.004026] nfs: server whep-nfs.pleiades.uni-wuppertal.de not responding, still trying and sometimes the NFS server has a whep-nfs kernel: [ 1129.628090] RPC: AUTH_GSS upcall failed. Please check user daemon is running. or whep-nfs rpc.gssd[462]: destroying client /run/rpc_pipefs/nfsd4_cb/clnt151 in their logs. We already increased the debug levels (for example of idmapd) but haven't had anything obvious so far. Surprisingly, we have three main "public login machines" running CentOS 6, they seem to run fine. So probably it's not a NFS server issue. We increased the number of NFS server processes running on the NFS server from the default 8 (which might be too little for a setup like this) to 128. Also I upgraded the sssd on the server to 1.12.5 (compared to the 1.11 which comes with Ubuntu). All with little success, users are still complaining about the slow connection. Our munin monitoring shows no satuaration on the NFS server. The load is also nearly 0. The real file system is mounted through a high-perfomance FibreChannel system. Any thoughts / ideas are appreciated. Thanks Torsten -- <><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><> <> <> <> Dr. Torsten Harenberg harenberg at physik.uni-wuppertal.de <> <> Bergische Universitaet <> <> FB C - Physik Tel.: +49 (0)202 439-3521 <> <> Gaussstr. 20 Fax : +49 (0)202 439-2811 <> <> 42097 Wuppertal <> <> <> <><><><><><><>< Of course it runs NetBSD http://www.netbsd.org ><> From lslebodn at redhat.com Thu Sep 22 12:15:46 2016 From: lslebodn at redhat.com (Lukas Slebodnik) Date: Thu, 22 Sep 2016 14:15:46 +0200 Subject: [Freeipa-users] sssd.conf - the server and host-client relationship In-Reply-To: References: <20160920071144.GA2435@10.4.128.1> Message-ID: <20160922121545.GG5249@10.4.128.1> On (22/09/16 08:53), Lachlan Musicman wrote: >My translations of your comments are in line, if you could correct, I'd >appreciate that. > >On 20 September 2016 at 17:11, Lukas Slebodnik wrote: > >> >---------------------- >> >[domain/unixdev.etc] >> >ignore_group_members = True >> It was probably set as a result of performance tuning. >> >> >ldap_purge_cache_timeout = 0 >> That's default since 1.13.0 >> >> >subdomain_inherit = ignore_group_members, ldap_purge_cache_timeout >> that's specific option for sssd on IPA server >> > > >I presume your comment suggests ignore_group_members is no longer needed, >and since the lpct=0 is now default, then subdomain_inherit is also >superfluous? > I have no idea why the option ignore_group_members was set. My assumption is that you wanted to reduce loading data from IPA/AD because they were many members in groups and it was slow. > > >> >selinux_provider = none >> It was probably set as a workaround of bug which have been already >> fixed. >> > >We set this because of an error in libsemanage, but I think that was an >upstream (selinux) issue? >https://www.redhat.com/archives/freeipa-users/2016-July/msg00244.html > >Not sure if I should disable just yet - was this fixed? It should be fixed if not file a bug. >> >> >ipa_server_mode = True >> that's specific option for sssd on IPA server >> >> >I take it that this means it's still used. > yes, but it is used only on in sssd which is on IPA server. > >> >sudo_provider = ldap >> >ldap_uri = ldap://vmdv-linuxidm1.unixdev.petermac.org.au >> >ldap_sudo_search_base = or=sudoers,dc=unixdev,dc=petermac,dc=org,dc=au >> >ldap_sasl_mech = GSSAPI >> >ldap_sasl_authid = host/vmdv-linuxidm1.unixdev.petermac.org.au >> >ldap_sasl_realm = UNIXDEV.PETERMAC.ORG.AU >> >krb5_server = vmdv-linuxidm1.unixdev.petermac.org.au >> Previous 7 options are not required since sssd-1.10 >> > >Yep, I added those because of disconnect between the different info sources >made it hard to tell what was canonical, so I followed the red hat guide: > >https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/6/html/Deployment_Guide/sssd-ldap-sudo.html > >mostly because I didn't quite understand the sssd-sudo man page (because >sometimes I find man pages obtuse), but also there was an inconsistency >with the local man page and the die.net mirror >https://linux.die.net/man/5/sssd-sudo and this howto >https://blog-rcritten.rhcloud.com/?p=52 > The best is to check version of man page sssd-sudo on the machine But as I wrote "sudo_provider = ldap" is not required for ipa client since sssd-1.10 and most of current distributions has newer version of sssd. LS From pvoborni at redhat.com Thu Sep 22 12:45:40 2016 From: pvoborni at redhat.com (Petr Vobornik) Date: Thu, 22 Sep 2016 14:45:40 +0200 Subject: [Freeipa-users] down master still in ldap, prevents re-enrolement In-Reply-To: <1562005586.2453821.1474493159566@mail.yahoo.com> References: <1562005586.2453821.1474493159566.ref@mail.yahoo.com> <1562005586.2453821.1474493159566@mail.yahoo.com> Message-ID: On 09/21/2016 11:25 PM, pgb205 wrote: > topology prior to deletion > > master1<->master2 > > master2 deleted with ipa-server --uninstall command > > During re-installation I get error that the replication agreement still exists > on master1. > I do see this using ipa-replica-manage list. > > Tried deleting replication agreement with > ipa-replica-manage disconnect but receive 'no such replication agreement exist' > > Force deletion and cleanup do not work > receive unexpected error: Server is unwilling to perform: database is read-only > > > removing directly from ldap gives me: > ldapdelete -r -x -D "cn=Directory Manager" -W > 'cn=fqdn,cn=masters,cn=ipa,cn=etc,dc=domain,dc=com' > Enter LDAP Password: > ldap_delete: Server is unwilling to perform (53) > ldap_delete: Server is unwilling to perform (53) > additional info: database is read-only > > But I am not sure if I'm not using correct path or if it's something else. > > Might be related to Bug 826677 ? IPA cannot remove disconnected replica data to > reconnect > > > > > Bug 826677 ? IPA cannot remove disconnected replica data to reconnect > > > > > run on master1: ipa-csreplica-manage del master2 --force --clean ipa-replica-manage del master2 --force --clean In that order. First step only if master2 was installed with CA. Those command should clean left-over data from master2. In standard situation, recommended uninstallation procedure for IPAs prior FreeIPA 4.4 is: master1# ipa-csreplica-manage del master2 master1# ipa-replica-manage del master2 master2# ipa-server-install --uninstall -- Petr Vobornik From rcritten at redhat.com Thu Sep 22 13:51:08 2016 From: rcritten at redhat.com (Rob Crittenden) Date: Thu, 22 Sep 2016 09:51:08 -0400 Subject: [Freeipa-users] down master still in ldap, prevents re-enrolement In-Reply-To: References: <1562005586.2453821.1474493159566.ref@mail.yahoo.com> <1562005586.2453821.1474493159566@mail.yahoo.com> Message-ID: <57E3E1CC.2050802@redhat.com> Petr Vobornik wrote: > On 09/21/2016 11:25 PM, pgb205 wrote: >> topology prior to deletion >> >> master1<->master2 >> >> master2 deleted with ipa-server --uninstall command >> >> During re-installation I get error that the replication agreement still exists >> on master1. >> I do see this using ipa-replica-manage list. >> >> Tried deleting replication agreement with >> ipa-replica-manage disconnect but receive 'no such replication agreement exist' >> >> Force deletion and cleanup do not work >> receive unexpected error: Server is unwilling to perform: database is read-only >> >> >> removing directly from ldap gives me: >> ldapdelete -r -x -D "cn=Directory Manager" -W >> 'cn=fqdn,cn=masters,cn=ipa,cn=etc,dc=domain,dc=com' >> Enter LDAP Password: >> ldap_delete: Server is unwilling to perform (53) >> ldap_delete: Server is unwilling to perform (53) >> additional info: database is read-only >> >> But I am not sure if I'm not using correct path or if it's something else. >> >> Might be related to Bug 826677 ? IPA cannot remove disconnected replica data to >> reconnect >> >> >> >> >> Bug 826677 ? IPA cannot remove disconnected replica data to reconnect >> >> >> >> >> > > run on master1: > ipa-csreplica-manage del master2 --force --clean > ipa-replica-manage del master2 --force --clean > > In that order. First step only if master2 was installed with CA. > > Those command should clean left-over data from master2. > > In standard situation, recommended uninstallation procedure for IPAs > prior FreeIPA 4.4 is: > master1# ipa-csreplica-manage del master2 > master1# ipa-replica-manage del master2 > master2# ipa-server-install --uninstall > Ultimately the problem is that the database is set to read only. $ ldapsearch -x -D 'cn=directory manager' -W -s base -b 'cn=userRoot, cn=ldbm database, cn=plugins, cn=config' nsslapd-readonly rob From koreyc at gmail.com Thu Sep 22 14:44:02 2016 From: koreyc at gmail.com (Korey Chapman) Date: Thu, 22 Sep 2016 07:44:02 -0700 Subject: [Freeipa-users] CA Fails to build Replica (w/External CA) In-Reply-To: References: <64066e0c-a299-cfa8-f1e5-1585692c15ae@redhat.com> Message-ID: On Thu, Sep 22, 2016 at 1:52 AM, Florence Blanc-Renaud wrote: > Hi Korey, > > I believe that you are hitting Dogtag issue #2255 [1]. The file /tmp/ca.p12 > probably doesn't contain the trust flags for some certificates. > You can check by running > pki pkcs12-cert-find --pkcs12-file /tmp/ca.p12 --pkcs12-password password > and see if the output displays "Trust Flags: xxx" for all the certs. > > Flo. > > [1] https://fedorahosted.org/pki/ticket/2255 > > > On 09/21/2016 05:38 PM, Korey Chapman wrote: >> >> On Wed, Sep 21, 2016 at 6:47 AM, Tomas Krizek wrote: >>> >>> On 09/21/2016 02:13 AM, Korey Chapman wrote: >>> >>> Hello list, >>> >>> I'm currently attempting to add a second CA server to our IPA cluster >>> (all >>> servers Centos 7.2 with IPA 4.2.0). However, it is failing no matter how >>> I >>> try to setup the CA (ipa-replica-install with --setup-ca or >>> ipa-replica-install followed by ipa-ca-install). The only useful thing in >>> the logs is an error about a missing key for "trust_flags" in the pki >>> setup. >>> Our infrastructure uses FreeIPA with an external CA. >>> >>> Any ideas/help would be greatly appreciated. Here are the logs snips from >>> my >>> most recent attempt: >>> >>> Command output snip from "ipa-replica-install >>> /root/replica-info-auth-002.XXX.gpg --setup-ca" >>> Configuring certificate server (pki-tomcatd). Estimated time: 3 minutes >>> 30 >>> seconds >>> [1/24]: creating certificate server user >>> [2/24]: configuring certificate server instance >>> ipa.ipaserver.install.cainstance.CAInstance: CRITICAL Failed to configure >>> CA >>> instance: Command ''/usr/sbin/pkispawn' '-s' 'CA' '-f' '/tmp/tmpYofMPt'' >>> returned non-zero exit status 1 >>> ipa.ipaserver.install.cainstance.CAInstance: CRITICAL See the >>> installation >>> logs and the following files/directories for more information: >>> ipa.ipaserver.install.cainstance.CAInstance: CRITICAL >>> /var/log/pki-ca-install.log >>> ipa.ipaserver.install.cainstance.CAInstance: CRITICAL >>> /var/log/pki/pki-tomcat >>> [error] RuntimeError: CA configuration failed. >>> Your system may be partly configured. >>> Run /usr/sbin/ipa-server-install --uninstall to clean up. >>> >>> ipa.ipapython.install.cli.install_tool(Replica): ERROR CA >>> configuration >>> failed >>> >>> >>> Log snip from ipareplica-install.log: >>> >>> 2016-09-20T23:42:27Z DEBUG Starting external process >>> 2016-09-20T23:42:27Z DEBUG args='/usr/sbin/pkispawn' '-s' 'CA' '-f' >>> '/tmp/tmpYofMPt' >>> 2016-09-20T23:42:31Z DEBUG Process finished, return code=1 >>> 2016-09-20T23:42:31Z DEBUG stdout=Log file: >>> /var/log/pki/pki-ca-spawn.20160920234227.log >>> Loading deployment configuration from /tmp/tmpYofMPt. >>> Installing CA into /var/lib/pki/pki-tomcat. >>> Storing deployment configuration into >>> /etc/sysconfig/pki/tomcat/pki-tomcat/ca/deployment.cfg. >>> >>> Installation failed. >>> >>> >>> 2016-09-20T23:42:31Z DEBUG >>> stderr=/usr/lib/python2.7/site-packages/urllib3/connectionpool.py:769: >>> InsecureRequestWarning: Unverified HTTPS request is being made. Adding >>> certificate verification is strongly advised. See: >>> https://urllib3.readthedocs.org/en/latest/security.html >>> InsecureRequestWarning) >>> Traceback (most recent call last): >>> File "/bin/pki", line 254, in >>> cli.execute(sys.argv) >>> File "/bin/pki", line 240, in execute >>> module.execute(module_args) >>> File "/usr/lib/python2.7/site-packages/pki/cli/__init__.py", line 195, >>> in >>> execute >>> module.execute(module_args) >>> File "/usr/lib/python2.7/site-packages/pki/cli/pkcs12.py", line 222, in >>> execute >>> trust_flags = cert_info['trust_flags'] >>> KeyError: 'trust_flags' >>> >>> >>> -- >>> Korey >>> >>> >>> Hi Korey, >>> >>> could you check if there is any more info in /var/log/pki/pki-ca-spawn >>> log? >> >> >> Nothing really useful I see in the spawn log: >> 2016-09-20 23:42:31 pkispawn : DEBUG ....... Error Type: >> CalledProcessError >> 2016-09-20 23:42:31 pkispawn : DEBUG ....... Error Message: >> Command '['pki', '-d', '/etc/pki/pki-tomcat/alias', '-C', >> '/etc/pki/pki-tomcat/pfile', 'pkcs12-import', '--pkcs12-file', >> '/tmp/ca.p12', '--pkcs12-password-file', >> '/tmp/tmps5OOav/password.txt', '--no-user-certs']' returned non-zero >> exit status 1 >> 2016-09-20 23:42:31 pkispawn : DEBUG ....... File >> "/usr/sbin/pkispawn", line 597, in main >> rv = scriptlet.spawn(deployer) >> File >> "/usr/lib/python2.7/site-packages/pki/server/deployment/scriptlets/security_databases.py", >> line 104, in spawn >> no_user_certs=True) >> File "/usr/lib/python2.7/site-packages/pki/nssdb.py", line 538, in >> import_pkcs12 >> subprocess.check_call(cmd) >> File "/usr/lib64/python2.7/subprocess.py", line 542, in check_call >> raise CalledProcessError(retcode, cmd) >> >>> >>> It might also be helpful verify if correct trust flags are set in nssdb: >>> certutil -d /etc/pki/pki-tomcat/alias/ -L >>> >> >> Run on the source ipa server (current CA server): >> $ certutil -d /etc/pki/pki-tomcat/alias/ -L >> >> Certificate Nickname Trust >> Attributes >> >> SSL,S/MIME,JAR/XPI >> >> XXX Certificate Authority CT,c, >> Server-Cert cert-pki-ca u,u,u >> auditSigningCert cert-pki-ca u,u,Pu >> caSigningCert cert-pki-ca CTu,Cu,Cu >> ocspSigningCert cert-pki-ca u,u,u >> subsystemCert cert-pki-ca u,u,u >> >> >> Run on the destination ipa server: >> $ certutil -d /etc/pki/pki-tomcat/alias/ -L >> >> Certificate Nickname Trust >> Attributes >> >> SSL,S/MIME,JAR/XPI >> >>> Finally, can you check that LDAPS is running on port 636 on the replica >>> where you're trying to install the CA (i.e. by nmap localhost)? >> >> >> Run on the new replica: >> $ nmap localhost >> >> Starting Nmap 6.40 ( http://nmap.org ) at 2016-09-21 15:29 UTC >> Nmap scan report for localhost (127.0.0.1) >> Host is up (0.0000040s latency). >> Other addresses for localhost (not scanned): 127.0.0.1 >> Not shown: 995 closed ports >> PORT STATE SERVICE >> 22/tcp open ssh >> 25/tcp open smtp >> 53/tcp open domain >> 389/tcp open ldap >> 636/tcp open ldapssl >> >>> >>> -- >>> Tomas Krizek >> >> >> >> > Looks like that was it. I updated the pki-* components on the master server and it imported fine after a fresh ipa-replica-prepare. I didn't think to look at the master server for the source of the problem. Thanks Florance and Thomas! -- Korey From bahanw042014 at gmail.com Thu Sep 22 14:57:22 2016 From: bahanw042014 at gmail.com (bahan w) Date: Thu, 22 Sep 2016 16:57:22 +0200 Subject: [Freeipa-users] Problem with a filer and FreeIPA Message-ID: Hello ! I contact you because I have a problem with a filer mounted on a server on which I installed freeipa client. I'm using FreeIPA 3.0.0-47 for both client and servers. The filer is mounted on /myfiler I have a user defined in freeipa : User1 I have a group defined in freeipa : Group1 I have another user defined in freeipa : User2 User2 belongs to group Group1. Test 1 : I create a folder Folder1 outside of the filer, in /usr for example. /usr/folder1 I set the posix permissions 750 and owner = user1 and group=group1. I connect with user2 and tries to read the content of the folder /usr/folder1. It works fine. Test 2 : I create a folder Folder2 inside the filer, in /myfiler for example. /myfiler/folder2 I set the posix permissions 750 and owner = user1 and group=group1. I connect with user2 and tries to read the content of the folder /usr/folder1. It does not work with the following error : permission denied. Is there something to do from filer side to plugin with FreeIPA server ? Best regards. Bahan -------------- next part -------------- An HTML attachment was scrubbed... URL: From rharwood at redhat.com Thu Sep 22 18:21:39 2016 From: rharwood at redhat.com (Robbie Harwood) Date: Thu, 22 Sep 2016 14:21:39 -0400 Subject: [Freeipa-users] Fwd: Re: Increase ListenBacklog for httpd In-Reply-To: References: <9819313c-6018-728f-4c9e-77371b493eb1@redhat.com> Message-ID: Rakesh Rajasekharan writes: > Thanks Robbie for the inputs.. the load should not have been high as I > have around 4000 clients with 160 users which should be manageable > > However, I saw a lot of clock skew too great errors in my > krb5kdc.log... however I haven't been able to verify if those were > genuine... > > Can too many clock skew errors take down the kerberos service.. Too much load of any kind will take down the service, but clock skew problems shouldn't be particularly resource intensive. Depending on your client behavior, they may cause retries, which will mean about twice as much traffic as normal, if memory serves. You'll want to fix the clock skew problem regardless. -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 818 bytes Desc: not available URL: From linuxguru.co at gmail.com Fri Sep 23 05:29:53 2016 From: linuxguru.co at gmail.com (Devin Acosta) Date: Thu, 22 Sep 2016 22:29:53 -0700 Subject: [Freeipa-users] FreeIPA upgrade from ipa-server-4.2.0-15.0.1.el7.centos.18 to ipa-server-4.2.0-15.0.1.el7.centos.19 (went sideways) Message-ID: Tonight, I noticed there was like 30 packages to be applied on my IPA server. I did the normal 'yum update' process and it completed. I then rebooted the box for the new kernel to take affect and then that is when IPA stopped working completely. When I try to start the dirsrv at RSINC-LOCAL.service, it throws up with: [23/Sep/2016:05:19:38 +0000] - SSL alert: Configured NSS Ciphers [23/Sep/2016:05:19:38 +0000] - SSL alert: TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384: enabled [23/Sep/2016:05:19:38 +0000] - SSL alert: TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA: enabled [23/Sep/2016:05:19:38 +0000] - SSL alert: TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA384: enabled [23/Sep/2016:05:19:38 +0000] - SSL alert: TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256: enabled [23/Sep/2016:05:19:38 +0000] - SSL alert: TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA: enabled [23/Sep/2016:05:19:38 +0000] - SSL alert: TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256: enabled [23/Sep/2016:05:19:38 +0000] - SSL alert: TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384: enabled [23/Sep/2016:05:19:38 +0000] - SSL alert: TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA: enabled [23/Sep/2016:05:19:38 +0000] - SSL alert: TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384: enabled [23/Sep/2016:05:19:38 +0000] - SSL alert: TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256: enabled [23/Sep/2016:05:19:38 +0000] - SSL alert: TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA: enabled [23/Sep/2016:05:19:38 +0000] - SSL alert: TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256: enabled [23/Sep/2016:05:19:38 +0000] - SSL alert: TLS_DHE_RSA_WITH_AES_256_GCM_SHA384: enabled [23/Sep/2016:05:19:38 +0000] - SSL alert: TLS_DHE_DSS_WITH_AES_256_GCM_SHA384: enabled [23/Sep/2016:05:19:38 +0000] - SSL alert: TLS_DHE_RSA_WITH_AES_256_CBC_SHA: enabled [23/Sep/2016:05:19:38 +0000] - SSL alert: TLS_DHE_DSS_WITH_AES_256_CBC_SHA: enabled [23/Sep/2016:05:19:38 +0000] - SSL alert: TLS_DHE_RSA_WITH_AES_256_CBC_SHA256: enabled [23/Sep/2016:05:19:38 +0000] - SSL alert: TLS_DHE_DSS_WITH_AES_256_CBC_SHA256: enabled [23/Sep/2016:05:19:38 +0000] - SSL alert: TLS_DHE_RSA_WITH_CAMELLIA_256_CBC_SHA: enabled [23/Sep/2016:05:19:38 +0000] - SSL alert: TLS_DHE_DSS_WITH_CAMELLIA_256_CBC_SHA: enabled [23/Sep/2016:05:19:38 +0000] - SSL alert: TLS_DHE_RSA_WITH_AES_128_GCM_SHA256: enabled [23/Sep/2016:05:19:38 +0000] - SSL alert: TLS_DHE_DSS_WITH_AES_128_GCM_SHA256: enabled [23/Sep/2016:05:19:38 +0000] - SSL alert: TLS_DHE_RSA_WITH_AES_128_CBC_SHA: enabled [23/Sep/2016:05:19:38 +0000] - SSL alert: TLS_DHE_DSS_WITH_AES_128_CBC_SHA: enabled [23/Sep/2016:05:19:38 +0000] - SSL alert: TLS_DHE_RSA_WITH_AES_128_CBC_SHA256: enabled [23/Sep/2016:05:19:38 +0000] - SSL alert: TLS_DHE_DSS_WITH_AES_128_CBC_SHA256: enabled [23/Sep/2016:05:19:38 +0000] - SSL alert: TLS_DHE_RSA_WITH_CAMELLIA_128_CBC_SHA: enabled [23/Sep/2016:05:19:38 +0000] - SSL alert: TLS_DHE_DSS_WITH_CAMELLIA_128_CBC_SHA: enabled [23/Sep/2016:05:19:38 +0000] - SSL alert: TLS_ECDH_ECDSA_WITH_AES_128_CBC_SHA: enabled [23/Sep/2016:05:19:38 +0000] - SSL alert: TLS_ECDH_RSA_WITH_AES_128_CBC_SHA: enabled [23/Sep/2016:05:19:38 +0000] - SSL alert: TLS_ECDH_ECDSA_WITH_AES_256_CBC_SHA: enabled [23/Sep/2016:05:19:38 +0000] - SSL alert: TLS_ECDH_RSA_WITH_AES_256_CBC_SHA: enabled [23/Sep/2016:05:19:38 +0000] - SSL alert: TLS_RSA_WITH_AES_256_GCM_SHA384: enabled [23/Sep/2016:05:19:38 +0000] - SSL alert: TLS_RSA_WITH_AES_256_CBC_SHA: enabled [23/Sep/2016:05:19:38 +0000] - SSL alert: TLS_RSA_WITH_AES_256_CBC_SHA256: enabled [23/Sep/2016:05:19:38 +0000] - SSL alert: TLS_RSA_WITH_CAMELLIA_256_CBC_SHA: enabled [23/Sep/2016:05:19:38 +0000] - SSL alert: TLS_RSA_WITH_AES_128_GCM_SHA256: enabled [23/Sep/2016:05:19:38 +0000] - SSL alert: TLS_RSA_WITH_AES_128_CBC_SHA: enabled [23/Sep/2016:05:19:38 +0000] - SSL alert: TLS_RSA_WITH_AES_128_CBC_SHA256: enabled [23/Sep/2016:05:19:38 +0000] - SSL alert: TLS_RSA_WITH_CAMELLIA_128_CBC_SHA: enabled [23/Sep/2016:05:19:38 +0000] - SSL alert: TLS_RSA_WITH_SEED_CBC_SHA: enabled [23/Sep/2016:05:19:38 +0000] SSL Initialization - Configured SSL version range: min: TLS1.0, max: TLS1.2 [23/Sep/2016:05:19:38 +0000] - Shutting down due to possible conflicts with other slapd processes *I am not sure what to do about the error "Shutting down due to possible conflicts with other slapd processes"??* The dirserv won't start, and therefore IPA won't start either. Is there some way to do some cleanup or to have it repair the issue? Any help is greatly appreciated!!! Devin. -------------- next part -------------- An HTML attachment was scrubbed... URL: From deepak_dimri at HOTMAIL.COM Fri Sep 23 07:12:44 2016 From: deepak_dimri at HOTMAIL.COM (Deepak Dimri) Date: Fri, 23 Sep 2016 07:12:44 +0000 Subject: [Freeipa-users] key + 2FA (password+OTP) is not working In-Reply-To: References: Message-ID: Hi All, I am trying hard to get my 2FA working with FreeIPA but every effort of mine going waste! I have referred earlier forum emails but could not find any good reply on the issue i am facing. This is what i am trying I have a test user created in my IPA server enabled with Two factor authentication (password + OTP) and has ssh public key added in its profile. I want this test user to ssh into my ipa client (ubuntu 14.04) using key + password + OTP. I woudl ceryainly prefer just the key+ OTP only ( no password) but that seems far sighted as i cannot even make it work with what it supposed to work password + OTP. My /etc/ssh/sshd_conf file has almost everything default except i added these two lines at the end of it Match Group testusergroup AuthenticationMethods publickey,password:pam publickey,keyboard-interactive:pam i also tried with below but no luck Match Group testusergroup AuthenticationMethods publickey,keyboard-interactive my /etc/pam.d/sshd has these two changes, rest i kept default: # Standard Un*x authentication. #@include common-auth auth required pam_sss.so Now when i try to ssh into ipa client i either keep getting promptS for the password or it gets into a loop asking me to change the password ;complaining falsely that it has expired. I have tried multiple combinations of configurations by referring earlier email threads but none i found helpful. I cant make simple 2FA login to work with freeIPA. Normal password and key works just fine. its the 2FA which does not work for me. Would really be thankful if some one can help me with this issue.. is there any good freeIPA 2FA configuration document that i can refer? What should the steps for it work seamlessly? Many Thanks, Deepak -------------- next part -------------- An HTML attachment was scrubbed... URL: From lkrispen at redhat.com Fri Sep 23 07:18:40 2016 From: lkrispen at redhat.com (Ludwig Krispenz) Date: Fri, 23 Sep 2016 09:18:40 +0200 Subject: [Freeipa-users] FreeIPA upgrade from ipa-server-4.2.0-15.0.1.el7.centos.18 to ipa-server-4.2.0-15.0.1.el7.centos.19 (went sideways) In-Reply-To: References: Message-ID: <57E4D750.9080204@redhat.com> can you check if you have /var/lock/dirsrv/slapd-RSINC-LOCAL if the server user has permissions to write into this directory and its subdirs or if any pid file still exists in /var/lock/dirsrv/slapd-RSINC-LOCAL/server On 09/23/2016 07:29 AM, Devin Acosta wrote: > > Tonight, > > I noticed there was like 30 packages to be applied on my IPA server. I > did the normal 'yum update' process and it completed. I then rebooted > the box for the new kernel to take affect and then that is when IPA > stopped working completely. > > When I try to start the dirsrv at RSINC-LOCAL.service, it throws up with: > > [23/Sep/2016:05:19:38 +0000] - SSL alert: Configured NSS Ciphers > [23/Sep/2016:05:19:38 +0000] - SSL alert: > TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384: enabled > [23/Sep/2016:05:19:38 +0000] - SSL alert: > TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA: enabled > [23/Sep/2016:05:19:38 +0000] - SSL alert: > TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA384: enabled > [23/Sep/2016:05:19:38 +0000] - SSL alert: > TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256: enabled > [23/Sep/2016:05:19:38 +0000] - SSL alert: > TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA: enabled > [23/Sep/2016:05:19:38 +0000] - SSL alert: > TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256: enabled > [23/Sep/2016:05:19:38 +0000] - SSL alert: > TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384: enabled > [23/Sep/2016:05:19:38 +0000] - SSL alert: > TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA: enabled > [23/Sep/2016:05:19:38 +0000] - SSL alert: > TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384: enabled > [23/Sep/2016:05:19:38 +0000] - SSL alert: > TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256: enabled > [23/Sep/2016:05:19:38 +0000] - SSL alert: > TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA: enabled > [23/Sep/2016:05:19:38 +0000] - SSL alert: > TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256: enabled > [23/Sep/2016:05:19:38 +0000] - SSL alert: > TLS_DHE_RSA_WITH_AES_256_GCM_SHA384: enabled > [23/Sep/2016:05:19:38 +0000] - SSL alert: > TLS_DHE_DSS_WITH_AES_256_GCM_SHA384: enabled > [23/Sep/2016:05:19:38 +0000] - SSL alert: > TLS_DHE_RSA_WITH_AES_256_CBC_SHA: enabled > [23/Sep/2016:05:19:38 +0000] - SSL alert: > TLS_DHE_DSS_WITH_AES_256_CBC_SHA: enabled > [23/Sep/2016:05:19:38 +0000] - SSL alert: > TLS_DHE_RSA_WITH_AES_256_CBC_SHA256: enabled > [23/Sep/2016:05:19:38 +0000] - SSL alert: > TLS_DHE_DSS_WITH_AES_256_CBC_SHA256: enabled > [23/Sep/2016:05:19:38 +0000] - SSL alert: > TLS_DHE_RSA_WITH_CAMELLIA_256_CBC_SHA: enabled > [23/Sep/2016:05:19:38 +0000] - SSL alert: > TLS_DHE_DSS_WITH_CAMELLIA_256_CBC_SHA: enabled > [23/Sep/2016:05:19:38 +0000] - SSL alert: > TLS_DHE_RSA_WITH_AES_128_GCM_SHA256: enabled > [23/Sep/2016:05:19:38 +0000] - SSL alert: > TLS_DHE_DSS_WITH_AES_128_GCM_SHA256: enabled > [23/Sep/2016:05:19:38 +0000] - SSL alert: > TLS_DHE_RSA_WITH_AES_128_CBC_SHA: enabled > [23/Sep/2016:05:19:38 +0000] - SSL alert: > TLS_DHE_DSS_WITH_AES_128_CBC_SHA: enabled > [23/Sep/2016:05:19:38 +0000] - SSL alert: > TLS_DHE_RSA_WITH_AES_128_CBC_SHA256: enabled > [23/Sep/2016:05:19:38 +0000] - SSL alert: > TLS_DHE_DSS_WITH_AES_128_CBC_SHA256: enabled > [23/Sep/2016:05:19:38 +0000] - SSL alert: > TLS_DHE_RSA_WITH_CAMELLIA_128_CBC_SHA: enabled > [23/Sep/2016:05:19:38 +0000] - SSL alert: > TLS_DHE_DSS_WITH_CAMELLIA_128_CBC_SHA: enabled > [23/Sep/2016:05:19:38 +0000] - SSL alert: > TLS_ECDH_ECDSA_WITH_AES_128_CBC_SHA: enabled > [23/Sep/2016:05:19:38 +0000] - SSL alert: > TLS_ECDH_RSA_WITH_AES_128_CBC_SHA: enabled > [23/Sep/2016:05:19:38 +0000] - SSL alert: > TLS_ECDH_ECDSA_WITH_AES_256_CBC_SHA: enabled > [23/Sep/2016:05:19:38 +0000] - SSL alert: > TLS_ECDH_RSA_WITH_AES_256_CBC_SHA: enabled > [23/Sep/2016:05:19:38 +0000] - SSL alert: > TLS_RSA_WITH_AES_256_GCM_SHA384: enabled > [23/Sep/2016:05:19:38 +0000] - SSL alert: > TLS_RSA_WITH_AES_256_CBC_SHA: enabled > [23/Sep/2016:05:19:38 +0000] - SSL alert: > TLS_RSA_WITH_AES_256_CBC_SHA256: enabled > [23/Sep/2016:05:19:38 +0000] - SSL alert: > TLS_RSA_WITH_CAMELLIA_256_CBC_SHA: enabled > [23/Sep/2016:05:19:38 +0000] - SSL alert: > TLS_RSA_WITH_AES_128_GCM_SHA256: enabled > [23/Sep/2016:05:19:38 +0000] - SSL alert: > TLS_RSA_WITH_AES_128_CBC_SHA: enabled > [23/Sep/2016:05:19:38 +0000] - SSL alert: > TLS_RSA_WITH_AES_128_CBC_SHA256: enabled > [23/Sep/2016:05:19:38 +0000] - SSL alert: > TLS_RSA_WITH_CAMELLIA_128_CBC_SHA: enabled > [23/Sep/2016:05:19:38 +0000] - SSL alert: TLS_RSA_WITH_SEED_CBC_SHA: > enabled > [23/Sep/2016:05:19:38 +0000] SSL Initialization - Configured SSL > version range: min: TLS1.0, max: TLS1.2 > [23/Sep/2016:05:19:38 +0000] - Shutting down due to possible conflicts > with other slapd processes > > *I am not sure what to do about the error "Shutting down due to > possible conflicts with other slapd processes"??* > The dirserv won't start, and therefore IPA won't start either. Is > there some way to do some cleanup or to have it repair the issue? > > Any help is greatly appreciated!!! > > Devin. > > > > > -- Red Hat GmbH, http://www.de.redhat.com/, Registered seat: Grasbrunn, Commercial register: Amtsgericht Muenchen, HRB 153243, Managing Directors: Charles Cachera, Michael Cunningham, Michael O'Neill, Eric Shander -------------- next part -------------- An HTML attachment was scrubbed... URL: From abokovoy at redhat.com Fri Sep 23 07:25:25 2016 From: abokovoy at redhat.com (Alexander Bokovoy) Date: Fri, 23 Sep 2016 10:25:25 +0300 Subject: [Freeipa-users] key + 2FA (password+OTP) is not working In-Reply-To: References: Message-ID: <20160923072525.mf4i74mymjjjrxvm@redhat.com> On Fri, 23 Sep 2016, Deepak Dimri wrote: > >Hi All, > > >I am trying hard to get my 2FA working with FreeIPA but every effort of >mine going waste! I have referred earlier forum emails but could not >find any good reply on the issue i am facing. > > >This is what i am trying > > >I have a test user created in my IPA server enabled with Two factor >authentication (password + OTP) and has ssh public key added in its >profile. I want this test user to ssh into my ipa client (ubuntu >14.04) using key + password + OTP. I woudl ceryainly prefer just the >key+ OTP only ( no password) but that seems far sighted as i cannot >even make it work with what it supposed to work password + OTP. Can you make it working on Fedora 24 or CentOS 7.2? I.e. on the platforms where we know it works for sure (for me, at least). This would allow us to reduce problem space to the client side. >My /etc/ssh/sshd_conf file has almost everything default except i >added these two lines at the end of it > >Match Group testusergroup > > AuthenticationMethods publickey,password:pam publickey,keyboard-interactive:pam > >i also tried with below but no luck > >Match Group testusergroup > > AuthenticationMethods publickey,keyboard-interactive > > >my /etc/pam.d/sshd has these two changes, rest i kept default: > > ># Standard Un*x authentication. > >#@include common-auth > > >auth required pam_sss.so > > >Now when i try to ssh into ipa client i either keep getting promptS for >the password or it gets into a loop asking me to change the password >;complaining falsely that it has expired. I have tried multiple >combinations of configurations by referring earlier email threads but >none i found helpful. I cant make simple 2FA login to work with >freeIPA. Normal password and key works just fine. its the 2FA which >does not work for me. > > >Would really be thankful if some one can help me with this issue.. is >there any good freeIPA 2FA configuration document that i can refer? > >What should the steps for it work seamlessly? > > >Many Thanks, > >Deepak > >-- >Manage your subscription for the Freeipa-users mailing list: >https://www.redhat.com/mailman/listinfo/freeipa-users >Go to http://freeipa.org for more info on the project -- / Alexander Bokovoy From pvoborni at redhat.com Fri Sep 23 07:29:12 2016 From: pvoborni at redhat.com (Petr Vobornik) Date: Fri, 23 Sep 2016 09:29:12 +0200 Subject: [Freeipa-users] replica added, but clients still try renewing certificates with old master In-Reply-To: References: <89facc8a-74d1-5c00-dbd9-6cafa20f0e80@redhat.com> Message-ID: <4d78dba1-4ad2-77e8-ba4f-de9f86ed21eb@redhat.com> On 09/21/2016 05:06 PM, Natxo Asenjo wrote: > hi Petr, > > On Wed, Sep 21, 2016 at 4:38 PM, Petr Vobornik > wrote: > > On 09/21/2016 10:50 AM, Natxo Asenjo wrote: > > > When I try to resubmit certificates from certmonger they still hit the kdc01 web > > server, so the requests hang on an status: CA_UNREACHABLE > > ca-error: Server failed request, will retry: 4301 (RPC failed at server. > > Certificate operation cannot be completed: Failure decoding Certificate Signing > > Request). > > Where does it happen? On arbitrary client which was installed in a past > against the removed kdc01? > > > yes. > > > If so could you look into /etc/ipa/default.conf and change host option > from kdc01 to the 7.2 IPA sever? > > > ok, done. > > In fact, change both the domain as the xmlrpc_uri directives in the global > section was necessary. Now It worked :-) > > So, what should be the correct value for dns discovery for both directives using > dns discovery? I don't think there is a support for DNS discovery in Certmonger. CCing Rob. > > thanks! > -- > Groeten, > natxo > -- Petr Vobornik From deepak_dimri at hotmail.com Fri Sep 23 07:47:46 2016 From: deepak_dimri at hotmail.com (Deepak Dimri) Date: Fri, 23 Sep 2016 07:47:46 +0000 Subject: [Freeipa-users] key + 2FA (password+OTP) is not working In-Reply-To: <20160923072525.mf4i74mymjjjrxvm@redhat.com> References: , <20160923072525.mf4i74mymjjjrxvm@redhat.com> Message-ID: Hi Alexander, I am using AWS to do a pilot on freeIPA & unfortunately AWS does not provide fedora or centos as part of its freetier setup so i have to live with ubuntu, redhat , suse etc. I have same problem with ubuntu and redhat though! Just one basic question.. what are the steps i should be following to make it work assuming i am trying on centos or fedora regards, Deepak ________________________________ From: Alexander Bokovoy Sent: Friday, September 23, 2016 3:25 AM To: Deepak Dimri Cc: freeipa-users at redhat.com Subject: Re: [Freeipa-users] key + 2FA (password+OTP) is not working On Fri, 23 Sep 2016, Deepak Dimri wrote: > >Hi All, > > >I am trying hard to get my 2FA working with FreeIPA but every effort of >mine going waste! I have referred earlier forum emails but could not >find any good reply on the issue i am facing. > > >This is what i am trying > > >I have a test user created in my IPA server enabled with Two factor >authentication (password + OTP) and has ssh public key added in its >profile. I want this test user to ssh into my ipa client (ubuntu >14.04) using key + password + OTP. I woudl ceryainly prefer just the >key+ OTP only ( no password) but that seems far sighted as i cannot >even make it work with what it supposed to work password + OTP. Can you make it working on Fedora 24 or CentOS 7.2? I.e. on the platforms where we know it works for sure (for me, at least). This would allow us to reduce problem space to the client side. >My /etc/ssh/sshd_conf file has almost everything default except i >added these two lines at the end of it > >Match Group testusergroup > > AuthenticationMethods publickey,password:pam publickey,keyboard-interactive:pam > >i also tried with below but no luck > >Match Group testusergroup > > AuthenticationMethods publickey,keyboard-interactive > > >my /etc/pam.d/sshd has these two changes, rest i kept default: > > ># Standard Un*x authentication. > >#@include common-auth > > >auth required pam_sss.so > > >Now when i try to ssh into ipa client i either keep getting promptS for >the password or it gets into a loop asking me to change the password >;complaining falsely that it has expired. I have tried multiple >combinations of configurations by referring earlier email threads but >none i found helpful. I cant make simple 2FA login to work with >freeIPA. Normal password and key works just fine. its the 2FA which >does not work for me. > > >Would really be thankful if some one can help me with this issue.. is >there any good freeIPA 2FA configuration document that i can refer? > >What should the steps for it work seamlessly? > > >Many Thanks, > >Deepak > >-- >Manage your subscription for the Freeipa-users mailing list: >https://www.redhat.com/mailman/listinfo/freeipa-users Freeipa-users Info Page - Red Hat www.redhat.com Freeipa-users -- List dedicated to discussions about use, configuration and deployment of the IPA server. About Freeipa-users >Go to http://freeipa.org for more info on the project -- / Alexander Bokovoy -------------- next part -------------- An HTML attachment was scrubbed... URL: From lslebodn at redhat.com Fri Sep 23 08:27:11 2016 From: lslebodn at redhat.com (Lukas Slebodnik) Date: Fri, 23 Sep 2016 10:27:11 +0200 Subject: [Freeipa-users] sss / nsswitch In-Reply-To: References: <20160913070354.GE32073@p.Speedport_W_724V_Typ_A_05011603_00_009> <20160913083917.GF32073@p.Speedport_W_724V_Typ_A_05011603_00_009> <20160913130716.GL6703@10.4.128.1> Message-ID: <20160923082710.GA10982@10.4.128.1> On (13/09/16 16:18), Rob Verduijn wrote: >2016-09-13 15:07 GMT+02:00 Lukas Slebodnik : > >> On (13/09/16 10:39), Sumit Bose wrote: >> >On Tue, Sep 13, 2016 at 10:13:12AM +0200, Rob Verduijn wrote: >> >> Hi, >> >> >> >> Thanks that did it. >> >> >> >> Is there a less painfull way to be notified of these changes ? >> >> >> >> My nfs configuration gets broken much more than I like because of >> changes >> >> like these. >> >> I know fedora is supposed to be testing grounds unstable software, but I >> >> would really like to hear a heads up more often. >> > >> >The change was mentioned in the upstream release notes of SSSD-1.14.1 >> >https://fedorahosted.org/sssd/wiki/Releases/Notes-1.14.1 but of course I >> >cannot be expected to read all upstream release note before running 'dnf >> >update'. >> > >> >The change was necessary because before the plugin was in the >> >sssd-common package and this caused that some nfs dependencies were >> >pulled in even on systems where nfs is not needed at all. Since neither >> >SSSD nor nfs-idmap strictly require the plugin the new package is not >> >automatically installed during update. >> > >> >> Sorry for troubles. We can add weak dependency info sssd-common on >> sssd-nfs-idmap and it might be installed by default. >> IIRC dnf does not inform about suggested packages; but recommends minght >> work. Feel free ot file a BZ. >> >> The reason why it is in separate package is "container world". >> You need to have install packge sssd-nfs-idmap on host >> but sssd can be running in container. >> >> LS >> > > >I probably should've noticed that the version number went from 1.13.x to >1.14.x which usually is something noteworthy. >I'll just add the release notes from sssd to my list of must reads when >there is an update. > The package sssd-nfs-idmap should be installed with sssd-1.14.1-3 It needn't be due to weak dependencies. But recommended packages are installed by default with dnf. rpm -q --recommends sssd-common-1.14.1-3 libsss_autofs(x86-64) = 1.14.1-3.fc24 libsss_sudo = 1.14.1-3.fc24 sssd-nfs-idmap = 1.14.1-3.fc24 LS From rob.verduijn at gmail.com Fri Sep 23 08:31:32 2016 From: rob.verduijn at gmail.com (Rob Verduijn) Date: Fri, 23 Sep 2016 10:31:32 +0200 Subject: [Freeipa-users] sss / nsswitch In-Reply-To: <20160923082710.GA10982@10.4.128.1> References: <20160913070354.GE32073@p.Speedport_W_724V_Typ_A_05011603_00_009> <20160913083917.GF32073@p.Speedport_W_724V_Typ_A_05011603_00_009> <20160913130716.GL6703@10.4.128.1> <20160923082710.GA10982@10.4.128.1> Message-ID: 2016-09-23 10:27 GMT+02:00 Lukas Slebodnik : > On (13/09/16 16:18), Rob Verduijn wrote: > >2016-09-13 15:07 GMT+02:00 Lukas Slebodnik : > > > >> On (13/09/16 10:39), Sumit Bose wrote: > >> >On Tue, Sep 13, 2016 at 10:13:12AM +0200, Rob Verduijn wrote: > >> >> Hi, > >> >> > >> >> Thanks that did it. > >> >> > >> >> Is there a less painfull way to be notified of these changes ? > >> >> > >> >> My nfs configuration gets broken much more than I like because of > >> changes > >> >> like these. > >> >> I know fedora is supposed to be testing grounds unstable software, > but I > >> >> would really like to hear a heads up more often. > >> > > >> >The change was mentioned in the upstream release notes of SSSD-1.14.1 > >> >https://fedorahosted.org/sssd/wiki/Releases/Notes-1.14.1 but of > course I > >> >cannot be expected to read all upstream release note before running > 'dnf > >> >update'. > >> > > >> >The change was necessary because before the plugin was in the > >> >sssd-common package and this caused that some nfs dependencies were > >> >pulled in even on systems where nfs is not needed at all. Since neither > >> >SSSD nor nfs-idmap strictly require the plugin the new package is not > >> >automatically installed during update. > >> > > >> > >> Sorry for troubles. We can add weak dependency info sssd-common on > >> sssd-nfs-idmap and it might be installed by default. > >> IIRC dnf does not inform about suggested packages; but recommends minght > >> work. Feel free ot file a BZ. > >> > >> The reason why it is in separate package is "container world". > >> You need to have install packge sssd-nfs-idmap on host > >> but sssd can be running in container. > >> > >> LS > >> > > > > > >I probably should've noticed that the version number went from 1.13.x to > >1.14.x which usually is something noteworthy. > >I'll just add the release notes from sssd to my list of must reads when > >there is an update. > > > The package sssd-nfs-idmap should be installed with sssd-1.14.1-3 > It needn't be due to weak dependencies. But recommended packages > are installed by default with dnf. > > rpm -q --recommends sssd-common-1.14.1-3 > libsss_autofs(x86-64) = 1.14.1-3.fc24 > libsss_sudo = 1.14.1-3.fc24 > sssd-nfs-idmap = 1.14.1-3.fc24 > > LS > Does this also apply when you run dnf update ? Rob Verduijn -------------- next part -------------- An HTML attachment was scrubbed... URL: From deepak_dimri at hotmail.com Fri Sep 23 08:44:44 2016 From: deepak_dimri at hotmail.com (Deepak Dimri) Date: Fri, 23 Sep 2016 08:44:44 +0000 Subject: [Freeipa-users] key + 2FA (password+OTP) is not working In-Reply-To: <20160923072525.mf4i74mymjjjrxvm@redhat.com> References: , <20160923072525.mf4i74mymjjjrxvm@redhat.com> Message-ID: Hi Alexander, I somehow manage to try it on fedora and it did work fine for me.. Now is there any way i can restrict the login to OTP only? and not password + OTP? Best Regards, Deepak ________________________________ From: Alexander Bokovoy Sent: Friday, September 23, 2016 3:25 AM To: Deepak Dimri Cc: freeipa-users at redhat.com Subject: Re: [Freeipa-users] key + 2FA (password+OTP) is not working On Fri, 23 Sep 2016, Deepak Dimri wrote: > >Hi All, > > >I am trying hard to get my 2FA working with FreeIPA but every effort of >mine going waste! I have referred earlier forum emails but could not >find any good reply on the issue i am facing. > > >This is what i am trying > > >I have a test user created in my IPA server enabled with Two factor >authentication (password + OTP) and has ssh public key added in its >profile. I want this test user to ssh into my ipa client (ubuntu >14.04) using key + password + OTP. I woudl ceryainly prefer just the >key+ OTP only ( no password) but that seems far sighted as i cannot >even make it work with what it supposed to work password + OTP. Can you make it working on Fedora 24 or CentOS 7.2? I.e. on the platforms where we know it works for sure (for me, at least). This would allow us to reduce problem space to the client side. >My /etc/ssh/sshd_conf file has almost everything default except i >added these two lines at the end of it > >Match Group testusergroup > > AuthenticationMethods publickey,password:pam publickey,keyboard-interactive:pam > >i also tried with below but no luck > >Match Group testusergroup > > AuthenticationMethods publickey,keyboard-interactive > > >my /etc/pam.d/sshd has these two changes, rest i kept default: > > ># Standard Un*x authentication. > >#@include common-auth > > >auth required pam_sss.so > > >Now when i try to ssh into ipa client i either keep getting promptS for >the password or it gets into a loop asking me to change the password >;complaining falsely that it has expired. I have tried multiple >combinations of configurations by referring earlier email threads but >none i found helpful. I cant make simple 2FA login to work with >freeIPA. Normal password and key works just fine. its the 2FA which >does not work for me. > > >Would really be thankful if some one can help me with this issue.. is >there any good freeIPA 2FA configuration document that i can refer? > >What should the steps for it work seamlessly? > > >Many Thanks, > >Deepak > >-- >Manage your subscription for the Freeipa-users mailing list: >https://www.redhat.com/mailman/listinfo/freeipa-users Freeipa-users Info Page - Red Hat www.redhat.com Freeipa-users -- List dedicated to discussions about use, configuration and deployment of the IPA server. About Freeipa-users >Go to http://freeipa.org for more info on the project -- / Alexander Bokovoy -------------- next part -------------- An HTML attachment was scrubbed... URL: From abokovoy at redhat.com Fri Sep 23 09:06:24 2016 From: abokovoy at redhat.com (Alexander Bokovoy) Date: Fri, 23 Sep 2016 12:06:24 +0300 Subject: [Freeipa-users] key + 2FA (password+OTP) is not working In-Reply-To: References: <20160923072525.mf4i74mymjjjrxvm@redhat.com> Message-ID: <20160923090624.wrl5twqdf3evvowa@redhat.com> On Fri, 23 Sep 2016, Deepak Dimri wrote: >Hi Alexander, I am using AWS to do a pilot on freeIPA & unfortunately >AWS does not provide fedora or centos as part of its freetier setup so >i have to live with ubuntu, redhat , suse etc. I have same problem >with ubuntu and redhat though! CentOS 7 is available and eligible for free tier: https://aws.amazon.com/marketplace/pp/B00O7WM7QW >Just one basic question.. what are the steps i should be following to >make it work assuming i am trying on centos or fedora Literally what you describe in your setup, except that 'passwod:pam' seems to be broken in OpenSSH -- given that you are using PAM already for password checks, removing :pam should just work. It works for me with Match Group twofa AllowGroups twofa AuthenticationMethods publickey,password publickey,keyboard-interactive as the last statement in the sshd_config. Sep 23 11:55:50 f24-master.ipa.ad.test sshd[2965]: debug3: monitor_child_preauth: method publickey: partial ... Sep 23 11:56:07 f24-master.ipa.ad.test sshd[2965]: debug3: PAM: sshpam_passwd_conv called with 2 messages Sep 23 11:56:08 f24-master.ipa.ad.test ipa-otpd[2892]: foobar at IPA.AD.TEST: request received Sep 23 11:56:08 f24-master.ipa.ad.test ipa-otpd[2892]: foobar at IPA.AD.TEST: user query start Sep 23 11:56:08 f24-master.ipa.ad.test ipa-otpd[2892]: foobar at IPA.AD.TEST: user query end: uid=foobar,cn=users,cn=accounts,dc=ipa,dc=ad,dc=test Sep 23 11:56:08 f24-master.ipa.ad.test ipa-otpd[2892]: foobar at IPA.AD.TEST: bind start: uid=foobar,cn=users,cn=accounts,dc=ipa,dc=ad,dc=test Sep 23 11:56:08 f24-master.ipa.ad.test ipa-otpd[2892]: foobar at IPA.AD.TEST: bind end: success Sep 23 11:56:08 f24-master.ipa.ad.test ipa-otpd[2892]: foobar at IPA.AD.TEST: response sent: Access-Accept Sep 23 11:56:10 f24-master.ipa.ad.test audit[2965]: USER_AUTH pid=2965 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:sshd_t:s0-s0:c0.c1023 msg='op=PAM:authentication grantors=pam_succeed_if,pam_sss acct="foobar" exe="/usr/sbin/sshd" hostname=192.168.5.136 addr=192.168.5.136 terminal=ssh res=success' Sep 23 11:56:10 f24-master.ipa.ad.test sshd[2965]: pam_sss(sshd:auth): authentication success; logname= uid=0 euid=0 tty=ssh ruser= rhost=192.168.5.136 user=foobar Sep 23 11:56:10 f24-master.ipa.ad.test sshd[2965]: debug1: PAM: password authentication accepted for foobar Sep 23 11:56:10 f24-master.ipa.ad.test sshd[2965]: debug3: mm_answer_authpassword: sending result 1 Sep 23 11:56:10 f24-master.ipa.ad.test sshd[2965]: debug3: mm_request_send entering: type 13 Sep 23 11:56:10 f24-master.ipa.ad.test sshd[2965]: debug3: auth2_update_methods_lists: updating methods list after "password" Sep 23 11:56:10 f24-master.ipa.ad.test sshd[2965]: debug2: authentication methods list 0 complete Sep 23 11:56:10 f24-master.ipa.ad.test sshd[2965]: debug3: mm_request_receive_expect entering: type 102 Sep 23 11:56:10 f24-master.ipa.ad.test sshd[2965]: debug3: mm_request_receive entering Sep 23 11:56:10 f24-master.ipa.ad.test sshd[2965]: debug1: do_pam_account: called Sep 23 11:56:12 f24-master.ipa.ad.test audit[2965]: USER_ACCT pid=2965 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:sshd_t:s0-s0:c0.c1023 msg='op=PAM:accounting grantors=pam_unix,pam_sss,pam_permit acct="foobar" exe="/usr/sbin/sshd" hostname=192.168.5.136 addr=192.168.5.136 terminal=ssh res=success' Sep 23 11:56:12 f24-master.ipa.ad.test sshd[2965]: debug3: PAM: do_pam_account pam_acct_mgmt = 0 (Success) Sep 23 11:56:12 f24-master.ipa.ad.test sshd[2965]: debug3: mm_request_send entering: type 103 Sep 23 11:56:12 f24-master.ipa.ad.test sshd[2965]: Accepted password for foobar from 192.168.5.136 port 33466 ssh2 Sep 23 11:56:12 f24-master.ipa.ad.test sshd[2965]: debug1: monitor_child_preauth: foobar has been authenticated by privileged process The first line above says that publickey method was successful but not enough to allow login (partial) because password is also required. The client got a request to enter password+OTP value. As you can see the user is only allowed to login with an OTP token. $ ssh foobar at 192.168.5.117 foobar at 192.168.5.117's password: Last login: Fri Sep 23 11:49:17 2016 -sh-4.3$ id uid=903200044(foobar) gid=903200044(foobar) groups=903200044(foobar),903200046(twofa) context=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023 -sh-4.3$ klist Ticket cache: KEYRING:persistent:903200044:krb_ccache_Dk553LV Default principal: foobar at IPA.AD.TEST Valid starting Expires Service principal 09/23/2016 11:56:08 09/24/2016 11:56:08 krbtgt/IPA.AD.TEST at IPA.AD.TEST -sh-4.3$ ipa user-show foobar User login: foobar First name: Test Last name: Foo Home directory: /home/foobar Login shell: /bin/sh Principal name: foobar at IPA.AD.TEST Principal alias: foobar at IPA.AD.TEST Email address: foobar at ipa.ad.test UID: 903200044 GID: 903200044 User authentication types: otp Account disabled: False Password: True Member of groups: twofa, ipausers Kerberos keys available: True -- / Alexander Bokovoy From abokovoy at redhat.com Fri Sep 23 09:07:34 2016 From: abokovoy at redhat.com (Alexander Bokovoy) Date: Fri, 23 Sep 2016 12:07:34 +0300 Subject: [Freeipa-users] key + 2FA (password+OTP) is not working In-Reply-To: References: <20160923072525.mf4i74mymjjjrxvm@redhat.com> Message-ID: <20160923090734.llhpefsjpvfaazke@redhat.com> On Fri, 23 Sep 2016, Deepak Dimri wrote: >Hi Alexander, > > >I somehow manage to try it on fedora and it did work fine for me.. > > >Now is there any way i can restrict the login to OTP only? and not password + OTP? No, this is not supported. OTP value only is not secure enough (6 digits by default, really low entropy). > > >Best Regards, > >Deepak > > >________________________________ >From: Alexander Bokovoy >Sent: Friday, September 23, 2016 3:25 AM >To: Deepak Dimri >Cc: freeipa-users at redhat.com >Subject: Re: [Freeipa-users] key + 2FA (password+OTP) is not working > >On Fri, 23 Sep 2016, Deepak Dimri wrote: >> >>Hi All, >> >> >>I am trying hard to get my 2FA working with FreeIPA but every effort of >>mine going waste! I have referred earlier forum emails but could not >>find any good reply on the issue i am facing. >> >> >>This is what i am trying >> >> >>I have a test user created in my IPA server enabled with Two factor >>authentication (password + OTP) and has ssh public key added in its >>profile. I want this test user to ssh into my ipa client (ubuntu >>14.04) using key + password + OTP. I woudl ceryainly prefer just the >>key+ OTP only ( no password) but that seems far sighted as i cannot >>even make it work with what it supposed to work password + OTP. >Can you make it working on Fedora 24 or CentOS 7.2? I.e. on the >platforms where we know it works for sure (for me, at least). > >This would allow us to reduce problem space to the client side. > >>My /etc/ssh/sshd_conf file has almost everything default except i >>added these two lines at the end of it >> >>Match Group testusergroup >> >> AuthenticationMethods publickey,password:pam publickey,keyboard-interactive:pam >> >>i also tried with below but no luck >> >>Match Group testusergroup >> >> AuthenticationMethods publickey,keyboard-interactive >> >> >>my /etc/pam.d/sshd has these two changes, rest i kept default: >> >> >># Standard Un*x authentication. >> >>#@include common-auth >> >> >>auth required pam_sss.so >> >> >>Now when i try to ssh into ipa client i either keep getting promptS for >>the password or it gets into a loop asking me to change the password >>;complaining falsely that it has expired. I have tried multiple >>combinations of configurations by referring earlier email threads but >>none i found helpful. I cant make simple 2FA login to work with >>freeIPA. Normal password and key works just fine. its the 2FA which >>does not work for me. >> >> >>Would really be thankful if some one can help me with this issue.. is >>there any good freeIPA 2FA configuration document that i can refer? >> >>What should the steps for it work seamlessly? >> >> >>Many Thanks, >> >>Deepak >> > >>-- >>Manage your subscription for the Freeipa-users mailing list: >>https://www.redhat.com/mailman/listinfo/freeipa-users >Freeipa-users Info Page - Red Hat >www.redhat.com >Freeipa-users -- List dedicated to discussions about use, configuration and deployment of the IPA server. About Freeipa-users > > > >>Go to http://freeipa.org for more info on the project > > >-- >/ Alexander Bokovoy -- / Alexander Bokovoy From natxo.asenjo at gmail.com Fri Sep 23 09:40:11 2016 From: natxo.asenjo at gmail.com (Natxo Asenjo) Date: Fri, 23 Sep 2016 11:40:11 +0200 Subject: [Freeipa-users] replica added, but clients still try renewing certificates with old master In-Reply-To: <4d78dba1-4ad2-77e8-ba4f-de9f86ed21eb@redhat.com> References: <89facc8a-74d1-5c00-dbd9-6cafa20f0e80@redhat.com> <4d78dba1-4ad2-77e8-ba4f-de9f86ed21eb@redhat.com> Message-ID: On Fri, Sep 23, 2016 at 9:29 AM, Petr Vobornik wrote: > On 09/21/2016 05:06 PM, Natxo Asenjo wrote: > > > So, what should be the correct value for dns discovery for both > directives using > > dns discovery? > > I don't think there is a support for DNS discovery in Certmonger. CCing > Rob. > Well, as soon as I remove the old replica running centos 6.8, I will create a dns A record with the old replica host name pointing to the new replica. So I think that will solve this particular problem. It would be much more convinient to have dns discovery in certmonger though. Thanks! -- Groeten, natxo -------------- next part -------------- An HTML attachment was scrubbed... URL: From malo at avast.com Fri Sep 23 11:09:13 2016 From: malo at avast.com (malo) Date: Fri, 23 Sep 2016 13:09:13 +0200 Subject: [Freeipa-users] Port and protocol for winsync Message-ID: Hello, I am currently trying to setup the winsyncagreement between my AD and my FreeIPA servers. The network topology allows me to only connect the FreeIPA server to the 636 port of AD, using TLS. It seems that FreeIPA wants to connect to the port 389 using StartTLS when I run the ipa-replica-manage command to create the winsync agreement. I know that I can modify the parameters of the winsync agreement once it is established, by modifying the cn=replica,cn=XXXXcom,cn=mapping tree,cn=config elements. But is there a way to specify the port as well as the protocol to use on the first configuration of the winsync agreement ? Thank you for your help, Best regards, Nathan M. From mbabinsk at redhat.com Fri Sep 23 12:26:38 2016 From: mbabinsk at redhat.com (Martin Babinsky) Date: Fri, 23 Sep 2016 14:26:38 +0200 Subject: [Freeipa-users] Port and protocol for winsync In-Reply-To: References: Message-ID: <7605080d-6f0b-db10-b999-69240ba51cc8@redhat.com> On 09/23/2016 01:09 PM, malo wrote: > Hello, > > > I am currently trying to setup the winsyncagreement between my AD and > my FreeIPA servers. The network topology allows me to only connect the > FreeIPA server to the 636 port of AD, using TLS. > > It seems that FreeIPA wants to connect to the port 389 using StartTLS > when I run the ipa-replica-manage command to create the winsync agreement. > > I know that I can modify the parameters of the winsync agreement once it > is established, by modifying the cn=replica,cn=XXXXcom,cn=mapping > tree,cn=config elements. > > > But is there a way to specify the port as well as the protocol to use on > the first configuration of the winsync agreement ? > > > Thank you for your help, > > Best regards, > > > Nathan M. > I am afraid that this is hardcoded in ipa-replica-manage and there is no way to force the command to use LDAPS connection. Is there any particular reason why incoming connections on AD DC's port 389 are blocked in your network? -- Martin^3 Babinsky From malo at avast.com Fri Sep 23 12:35:28 2016 From: malo at avast.com (malo) Date: Fri, 23 Sep 2016 14:35:28 +0200 Subject: [Freeipa-users] Port and protocol for winsync In-Reply-To: <7605080d-6f0b-db10-b999-69240ba51cc8@redhat.com> References: <7605080d-6f0b-db10-b999-69240ba51cc8@redhat.com> Message-ID: <2b74fa31-fc31-6349-b2b8-154bdb7b2938@avast.com> Thank you for your response Martin ! This restriction is due to the architecture of the in place network. This is sadly not something that I can change. Regards, Nathan On 09/23/2016 02:26 PM, Martin Babinsky wrote: > On 09/23/2016 01:09 PM, malo wrote: >> Hello, >> >> >> I am currently trying to setup the winsyncagreement between my AD and >> my FreeIPA servers. The network topology allows me to only connect the >> FreeIPA server to the 636 port of AD, using TLS. >> >> It seems that FreeIPA wants to connect to the port 389 using StartTLS >> when I run the ipa-replica-manage command to create the winsync >> agreement. >> >> I know that I can modify the parameters of the winsync agreement once it >> is established, by modifying the cn=replica,cn=XXXXcom,cn=mapping >> tree,cn=config elements. >> >> >> But is there a way to specify the port as well as the protocol to use on >> the first configuration of the winsync agreement ? >> >> >> Thank you for your help, >> >> Best regards, >> >> >> Nathan M. >> > I am afraid that this is hardcoded in ipa-replica-manage and there is > no way to force the command to use LDAPS connection. > > Is there any particular reason why incoming connections on AD DC's > port 389 are blocked in your network? > From linuxguru.co at gmail.com Fri Sep 23 14:42:36 2016 From: linuxguru.co at gmail.com (Devin Acosta) Date: Fri, 23 Sep 2016 07:42:36 -0700 Subject: [Freeipa-users] FreeIPA upgrade from ipa-server-4.2.0-15.0.1.el7.centos.18 to ipa-server-4.2.0-15.0.1.el7.centos.19 (went sideways) In-Reply-To: <57E4D750.9080204@redhat.com> References: <57E4D750.9080204@redhat.com> Message-ID: Ludwig, Thanks for that, for some reason I had to re-create the /var/lock/ dirsrv/slapd-RSINC-LOCAL/server directory tree, it did not exist. Once I re-created it now the server starts. Should it have disappeared like that? On Fri, Sep 23, 2016 at 12:18 AM, Ludwig Krispenz wrote: > can you check if you have > /var/lock/dirsrv/slapd-RSINC-LOCAL > > if the server user has permissions to write into this directory and its > subdirs or if any pid file still exists in /var/lock/dirsrv/slapd-RSINC- > LOCAL/server > > > On 09/23/2016 07:29 AM, Devin Acosta wrote: > > > Tonight, > > I noticed there was like 30 packages to be applied on my IPA server. I did > the normal 'yum update' process and it completed. I then rebooted the box > for the new kernel to take affect and then that is when IPA stopped working > completely. > > When I try to start the dirsrv at RSINC-LOCAL.service, it throws up with: > > [23/Sep/2016:05:19:38 +0000] - SSL alert: Configured NSS Ciphers > [23/Sep/2016:05:19:38 +0000] - SSL alert: TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384: > enabled > [23/Sep/2016:05:19:38 +0000] - SSL alert: TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA: > enabled > [23/Sep/2016:05:19:38 +0000] - SSL alert: TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA384: > enabled > [23/Sep/2016:05:19:38 +0000] - SSL alert: TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256: > enabled > [23/Sep/2016:05:19:38 +0000] - SSL alert: TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA: > enabled > [23/Sep/2016:05:19:38 +0000] - SSL alert: TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256: > enabled > [23/Sep/2016:05:19:38 +0000] - SSL alert: TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384: > enabled > [23/Sep/2016:05:19:38 +0000] - SSL alert: TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA: > enabled > [23/Sep/2016:05:19:38 +0000] - SSL alert: TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384: > enabled > [23/Sep/2016:05:19:38 +0000] - SSL alert: TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256: > enabled > [23/Sep/2016:05:19:38 +0000] - SSL alert: TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA: > enabled > [23/Sep/2016:05:19:38 +0000] - SSL alert: TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256: > enabled > [23/Sep/2016:05:19:38 +0000] - SSL alert: TLS_DHE_RSA_WITH_AES_256_GCM_SHA384: > enabled > [23/Sep/2016:05:19:38 +0000] - SSL alert: TLS_DHE_DSS_WITH_AES_256_GCM_SHA384: > enabled > [23/Sep/2016:05:19:38 +0000] - SSL alert: TLS_DHE_RSA_WITH_AES_256_CBC_SHA: > enabled > [23/Sep/2016:05:19:38 +0000] - SSL alert: TLS_DHE_DSS_WITH_AES_256_CBC_SHA: > enabled > [23/Sep/2016:05:19:38 +0000] - SSL alert: TLS_DHE_RSA_WITH_AES_256_CBC_SHA256: > enabled > [23/Sep/2016:05:19:38 +0000] - SSL alert: TLS_DHE_DSS_WITH_AES_256_CBC_SHA256: > enabled > [23/Sep/2016:05:19:38 +0000] - SSL alert: TLS_DHE_RSA_WITH_CAMELLIA_256_CBC_SHA: > enabled > [23/Sep/2016:05:19:38 +0000] - SSL alert: TLS_DHE_DSS_WITH_CAMELLIA_256_CBC_SHA: > enabled > [23/Sep/2016:05:19:38 +0000] - SSL alert: TLS_DHE_RSA_WITH_AES_128_GCM_SHA256: > enabled > [23/Sep/2016:05:19:38 +0000] - SSL alert: TLS_DHE_DSS_WITH_AES_128_GCM_SHA256: > enabled > [23/Sep/2016:05:19:38 +0000] - SSL alert: TLS_DHE_RSA_WITH_AES_128_CBC_SHA: > enabled > [23/Sep/2016:05:19:38 +0000] - SSL alert: TLS_DHE_DSS_WITH_AES_128_CBC_SHA: > enabled > [23/Sep/2016:05:19:38 +0000] - SSL alert: TLS_DHE_RSA_WITH_AES_128_CBC_SHA256: > enabled > [23/Sep/2016:05:19:38 +0000] - SSL alert: TLS_DHE_DSS_WITH_AES_128_CBC_SHA256: > enabled > [23/Sep/2016:05:19:38 +0000] - SSL alert: TLS_DHE_RSA_WITH_CAMELLIA_128_CBC_SHA: > enabled > [23/Sep/2016:05:19:38 +0000] - SSL alert: TLS_DHE_DSS_WITH_CAMELLIA_128_CBC_SHA: > enabled > [23/Sep/2016:05:19:38 +0000] - SSL alert: TLS_ECDH_ECDSA_WITH_AES_128_CBC_SHA: > enabled > [23/Sep/2016:05:19:38 +0000] - SSL alert: TLS_ECDH_RSA_WITH_AES_128_CBC_SHA: > enabled > [23/Sep/2016:05:19:38 +0000] - SSL alert: TLS_ECDH_ECDSA_WITH_AES_256_CBC_SHA: > enabled > [23/Sep/2016:05:19:38 +0000] - SSL alert: TLS_ECDH_RSA_WITH_AES_256_CBC_SHA: > enabled > [23/Sep/2016:05:19:38 +0000] - SSL alert: TLS_RSA_WITH_AES_256_GCM_SHA384: > enabled > [23/Sep/2016:05:19:38 +0000] - SSL alert: TLS_RSA_WITH_AES_256_CBC_SHA: > enabled > [23/Sep/2016:05:19:38 +0000] - SSL alert: TLS_RSA_WITH_AES_256_CBC_SHA256: > enabled > [23/Sep/2016:05:19:38 +0000] - SSL alert: TLS_RSA_WITH_CAMELLIA_256_CBC_SHA: > enabled > [23/Sep/2016:05:19:38 +0000] - SSL alert: TLS_RSA_WITH_AES_128_GCM_SHA256: > enabled > [23/Sep/2016:05:19:38 +0000] - SSL alert: TLS_RSA_WITH_AES_128_CBC_SHA: > enabled > [23/Sep/2016:05:19:38 +0000] - SSL alert: TLS_RSA_WITH_AES_128_CBC_SHA256: > enabled > [23/Sep/2016:05:19:38 +0000] - SSL alert: TLS_RSA_WITH_CAMELLIA_128_CBC_SHA: > enabled > [23/Sep/2016:05:19:38 +0000] - SSL alert: TLS_RSA_WITH_SEED_CBC_SHA: > enabled > [23/Sep/2016:05:19:38 +0000] SSL Initialization - Configured SSL version > range: min: TLS1.0, max: TLS1.2 > [23/Sep/2016:05:19:38 +0000] - Shutting down due to possible conflicts > with other slapd processes > > *I am not sure what to do about the error "Shutting down due to possible > conflicts with other slapd processes"??* > The dirserv won't start, and therefore IPA won't start either. Is there > some way to do some cleanup or to have it repair the issue? > > Any help is greatly appreciated!!! > > Devin. > > > > > > > -- > Red Hat GmbH, http://www.de.redhat.com/, Registered seat: Grasbrunn, > Commercial register: Amtsgericht Muenchen, HRB 153243, > Managing Directors: Charles Cachera, Michael Cunningham, Michael O'Neill, Eric Shander > > > -- > Manage your subscription for the Freeipa-users mailing list: > https://www.redhat.com/mailman/listinfo/freeipa-users > Go to http://freeipa.org for more info on the project > -------------- next part -------------- An HTML attachment was scrubbed... URL: From lkrispen at redhat.com Fri Sep 23 14:55:57 2016 From: lkrispen at redhat.com (Ludwig Krispenz) Date: Fri, 23 Sep 2016 16:55:57 +0200 Subject: [Freeipa-users] FreeIPA upgrade from ipa-server-4.2.0-15.0.1.el7.centos.18 to ipa-server-4.2.0-15.0.1.el7.centos.19 (went sideways) In-Reply-To: References: <57E4D750.9080204@redhat.com> Message-ID: <57E5427D.6000202@redhat.com> On 09/23/2016 04:42 PM, Devin Acosta wrote: > > Ludwig, > > Thanks for that, for some reason I had to re-create the > /var/lock/dirsrv/slapd-RSINC-LOCAL/server directory tree, it did not > exist. Once I re-created it now the server starts. Should it have > disappeared like that? no. I don't know what was going during update and reboot. There have been cases when a dse.ldif was lost after crashing/rebooting a VM, but the missing lock directory is new to me. > > > > On Fri, Sep 23, 2016 at 12:18 AM, Ludwig Krispenz > wrote: > > can you check if you have > /var/lock/dirsrv/slapd-RSINC-LOCAL > > if the server user has permissions to write into this directory > and its subdirs or if any pid file still exists in > /var/lock/dirsrv/slapd-RSINC-LOCAL/server > > > On 09/23/2016 07:29 AM, Devin Acosta wrote: >> >> Tonight, >> >> I noticed there was like 30 packages to be applied on my IPA >> server. I did the normal 'yum update' process and it completed. I >> then rebooted the box for the new kernel to take affect and then >> that is when IPA stopped working completely. >> >> When I try to start the dirsrv at RSINC-LOCAL.service >> , it throws up with: >> >> [23/Sep/2016:05:19:38 +0000] - SSL alert: Configured NSS Ciphers >> [23/Sep/2016:05:19:38 +0000] - SSL alert: >> TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384: enabled >> [23/Sep/2016:05:19:38 +0000] - SSL alert: >> TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA: enabled >> [23/Sep/2016:05:19:38 +0000] - SSL alert: >> TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA384: enabled >> [23/Sep/2016:05:19:38 +0000] - SSL alert: >> TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256: enabled >> [23/Sep/2016:05:19:38 +0000] - SSL alert: >> TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA: enabled >> [23/Sep/2016:05:19:38 +0000] - SSL alert: >> TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256: enabled >> [23/Sep/2016:05:19:38 +0000] - SSL alert: >> TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384: enabled >> [23/Sep/2016:05:19:38 +0000] - SSL alert: >> TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA: enabled >> [23/Sep/2016:05:19:38 +0000] - SSL alert: >> TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384: enabled >> [23/Sep/2016:05:19:38 +0000] - SSL alert: >> TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256: enabled >> [23/Sep/2016:05:19:38 +0000] - SSL alert: >> TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA: enabled >> [23/Sep/2016:05:19:38 +0000] - SSL alert: >> TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256: enabled >> [23/Sep/2016:05:19:38 +0000] - SSL alert: >> TLS_DHE_RSA_WITH_AES_256_GCM_SHA384: enabled >> [23/Sep/2016:05:19:38 +0000] - SSL alert: >> TLS_DHE_DSS_WITH_AES_256_GCM_SHA384: enabled >> [23/Sep/2016:05:19:38 +0000] - SSL alert: >> TLS_DHE_RSA_WITH_AES_256_CBC_SHA: enabled >> [23/Sep/2016:05:19:38 +0000] - SSL alert: >> TLS_DHE_DSS_WITH_AES_256_CBC_SHA: enabled >> [23/Sep/2016:05:19:38 +0000] - SSL alert: >> TLS_DHE_RSA_WITH_AES_256_CBC_SHA256: enabled >> [23/Sep/2016:05:19:38 +0000] - SSL alert: >> TLS_DHE_DSS_WITH_AES_256_CBC_SHA256: enabled >> [23/Sep/2016:05:19:38 +0000] - SSL alert: >> TLS_DHE_RSA_WITH_CAMELLIA_256_CBC_SHA: enabled >> [23/Sep/2016:05:19:38 +0000] - SSL alert: >> TLS_DHE_DSS_WITH_CAMELLIA_256_CBC_SHA: enabled >> [23/Sep/2016:05:19:38 +0000] - SSL alert: >> TLS_DHE_RSA_WITH_AES_128_GCM_SHA256: enabled >> [23/Sep/2016:05:19:38 +0000] - SSL alert: >> TLS_DHE_DSS_WITH_AES_128_GCM_SHA256: enabled >> [23/Sep/2016:05:19:38 +0000] - SSL alert: >> TLS_DHE_RSA_WITH_AES_128_CBC_SHA: enabled >> [23/Sep/2016:05:19:38 +0000] - SSL alert: >> TLS_DHE_DSS_WITH_AES_128_CBC_SHA: enabled >> [23/Sep/2016:05:19:38 +0000] - SSL alert: >> TLS_DHE_RSA_WITH_AES_128_CBC_SHA256: enabled >> [23/Sep/2016:05:19:38 +0000] - SSL alert: >> TLS_DHE_DSS_WITH_AES_128_CBC_SHA256: enabled >> [23/Sep/2016:05:19:38 +0000] - SSL alert: >> TLS_DHE_RSA_WITH_CAMELLIA_128_CBC_SHA: enabled >> [23/Sep/2016:05:19:38 +0000] - SSL alert: >> TLS_DHE_DSS_WITH_CAMELLIA_128_CBC_SHA: enabled >> [23/Sep/2016:05:19:38 +0000] - SSL alert: >> TLS_ECDH_ECDSA_WITH_AES_128_CBC_SHA: enabled >> [23/Sep/2016:05:19:38 +0000] - SSL alert: >> TLS_ECDH_RSA_WITH_AES_128_CBC_SHA: enabled >> [23/Sep/2016:05:19:38 +0000] - SSL alert: >> TLS_ECDH_ECDSA_WITH_AES_256_CBC_SHA: enabled >> [23/Sep/2016:05:19:38 +0000] - SSL alert: >> TLS_ECDH_RSA_WITH_AES_256_CBC_SHA: enabled >> [23/Sep/2016:05:19:38 +0000] - SSL alert: >> TLS_RSA_WITH_AES_256_GCM_SHA384: enabled >> [23/Sep/2016:05:19:38 +0000] - SSL alert: >> TLS_RSA_WITH_AES_256_CBC_SHA: enabled >> [23/Sep/2016:05:19:38 +0000] - SSL alert: >> TLS_RSA_WITH_AES_256_CBC_SHA256: enabled >> [23/Sep/2016:05:19:38 +0000] - SSL alert: >> TLS_RSA_WITH_CAMELLIA_256_CBC_SHA: enabled >> [23/Sep/2016:05:19:38 +0000] - SSL alert: >> TLS_RSA_WITH_AES_128_GCM_SHA256: enabled >> [23/Sep/2016:05:19:38 +0000] - SSL alert: >> TLS_RSA_WITH_AES_128_CBC_SHA: enabled >> [23/Sep/2016:05:19:38 +0000] - SSL alert: >> TLS_RSA_WITH_AES_128_CBC_SHA256: enabled >> [23/Sep/2016:05:19:38 +0000] - SSL alert: >> TLS_RSA_WITH_CAMELLIA_128_CBC_SHA: enabled >> [23/Sep/2016:05:19:38 +0000] - SSL alert: >> TLS_RSA_WITH_SEED_CBC_SHA: enabled >> [23/Sep/2016:05:19:38 +0000] SSL Initialization - Configured SSL >> version range: min: TLS1.0, max: TLS1.2 >> [23/Sep/2016:05:19:38 +0000] - Shutting down due to possible >> conflicts with other slapd processes >> >> *I am not sure what to do about the error "Shutting down due to >> possible conflicts with other slapd processes"??* >> The dirserv won't start, and therefore IPA won't start either. Is >> there some way to do some cleanup or to have it repair the issue? >> >> Any help is greatly appreciated!!! >> >> Devin. >> >> >> >> >> > > -- > Red Hat GmbH,http://www.de.redhat.com/, Registered seat: Grasbrunn, > Commercial register: Amtsgericht Muenchen, HRB 153243, > Managing Directors: Charles Cachera, Michael Cunningham, Michael O'Neill, Eric Shander > > > -- > Manage your subscription for the Freeipa-users mailing list: > https://www.redhat.com/mailman/listinfo/freeipa-users > > Go to http://freeipa.org for more info on the project > > -- Red Hat GmbH, http://www.de.redhat.com/, Registered seat: Grasbrunn, Commercial register: Amtsgericht Muenchen, HRB 153243, Managing Directors: Charles Cachera, Michael Cunningham, Michael O'Neill, Eric Shander -------------- next part -------------- An HTML attachment was scrubbed... URL: From rcritten at redhat.com Fri Sep 23 15:16:26 2016 From: rcritten at redhat.com (Rob Crittenden) Date: Fri, 23 Sep 2016 11:16:26 -0400 Subject: [Freeipa-users] replica added, but clients still try renewing certificates with old master In-Reply-To: <4d78dba1-4ad2-77e8-ba4f-de9f86ed21eb@redhat.com> References: <89facc8a-74d1-5c00-dbd9-6cafa20f0e80@redhat.com> <4d78dba1-4ad2-77e8-ba4f-de9f86ed21eb@redhat.com> Message-ID: <57E5474A.2050905@redhat.com> Petr Vobornik wrote: > On 09/21/2016 05:06 PM, Natxo Asenjo wrote: >> hi Petr, >> >> On Wed, Sep 21, 2016 at 4:38 PM, Petr Vobornik > > wrote: >> >> On 09/21/2016 10:50 AM, Natxo Asenjo wrote: >> >> > When I try to resubmit certificates from certmonger they still hit the kdc01 web >> > server, so the requests hang on an status: CA_UNREACHABLE >> > ca-error: Server failed request, will retry: 4301 (RPC failed at server. >> > Certificate operation cannot be completed: Failure decoding Certificate Signing >> > Request). >> >> Where does it happen? On arbitrary client which was installed in a past >> against the removed kdc01? >> >> >> yes. >> >> >> If so could you look into /etc/ipa/default.conf and change host option >> from kdc01 to the 7.2 IPA sever? >> >> >> ok, done. >> >> In fact, change both the domain as the xmlrpc_uri directives in the global >> section was necessary. Now It worked :-) >> >> So, what should be the correct value for dns discovery for both directives using >> dns discovery? > > I don't think there is a support for DNS discovery in Certmonger. CCing Rob. That is correct, it uses the value from the ipa config file. rob From mike.driscoll at oracle.com Fri Sep 23 15:51:56 2016 From: mike.driscoll at oracle.com (Mike Driscoll) Date: Fri, 23 Sep 2016 08:51:56 -0700 Subject: [Freeipa-users] Server replication stopped working Message-ID: <7854DA02-1936-448E-B4F7-791289F1F1A0@oracle.com> Hello. I have four IPA servers replicating in full mesh. All four servers are running ipa-server-4.2.0-15.0.1.el7_2.19.x86_64. This was working for some time but now I see that no replication is occurring automatically at present. When I update a user attribute on an IPA server, I see errors like these: [22/Sep/2016:16:53:49 -0700] attrlist_replace - attr_replace (nsslapd-referral, ldap://ldap03.xx.com:389/o%3Dipaca) failed. [22/Sep/2016:16:58:56 -0700] NSMMReplicationPlugin - agmt="cn=masterAgreement1-ldap03.xx.com-pki-tomcat" (ldap03:389): Incremental update failed and requires administrator action I can reinitialize without errors. ipa-csreplica-manage re-initialize --from=ldap01.xx.com ipa-replica-manage re-initialize --from=ldap01.xx.com Afterwards I see my attribute (and other) changes are replicated on each server I re-initialize from. But subsequently, replication doesn?t seem to be happening. I reinitialized according to the steps in Table 8.7, ?Replication Errors?, but subsequent replication isn?t occurring. Any suggestions? Is it safe to identify one of my four servers as containing up-to-date data, then sever and reinstate replication relationships with the other three? Mike -------------- next part -------------- An HTML attachment was scrubbed... URL: From gjn at gjn.priv.at Sat Sep 24 12:37:34 2016 From: gjn at gjn.priv.at (=?ISO-8859-1?Q?G=FCnther_J=2E?= Niederwimmer) Date: Sat, 24 Sep 2016 14:37:34 +0200 Subject: [Freeipa-users] Question Test 3rd Party Certificate Message-ID: <1480663.xLjgThpfif@techz> Hello, what is the best way to test a new installed 3rd Party certificate ? I hope i have now install (with big problems) the new certificate on clients and servers. But now is the big question is this all working correct together (?), or have i make a mistake ? I like to install this on a productive server with two master and 8 clients Freeipa 4.2 Centos 7 with all Updates with MailServer, private Cloud, webserver, DNS server ..... the next question is, what is in three years when the certificates expire ? Is there a tested way to renew the certificate ? I have search a long time in the internet but I can't found answers ? Thanks for a answer, -- mit freundlichen Gr??en / best regards, G?nther J. Niederwimmer From amessina at messinet.com Sun Sep 25 16:39:10 2016 From: amessina at messinet.com (Anthony Joseph Messina) Date: Sun, 25 Sep 2016 11:39:10 -0500 Subject: [Freeipa-users] bind crashes on rndc reload In-Reply-To: References: <77002552-83b0-7e5d-0950-9d8f78b504de@winteltosh.de> <1537316.S96Dxyse1G@linux-ws1.messinet.com> Message-ID: <4157170.3LmGYRYUmi@linux-ws1.messinet.com> On Tuesday, September 20, 2016 8:53:58 AM CDT Petr Spacek wrote: > On 20.9.2016 00:33, Anthony Joseph Messina wrote: > > On Monday, September 19, 2016 2:16:55 PM CDT Petr Spacek wrote: > >> On 12.9.2016 11:55, Anthony Joseph Messina wrote: > >>> On Monday, September 12, 2016 10:31:10 AM CDT Jochen Demmer wrote: > >>>> Hi, > >>>> > >>>> I have a major issue with my setup: > >>>> Fedora 24 > >>>> freeipa-common-4.3.2-2.fc24.noarch > >>>> freeipa-admintools-4.3.2-2.fc24.noarch > >>>> freeipa-server-dns-4.3.2-2.fc24.noarch > >>>> freeipa-client-common-4.3.2-2.fc24.noarch > >>>> freeipa-server-4.3.2-2.fc24.x86_64 > >>>> freeipa-server-common-4.3.2-2.fc24.noarch > >>>> freeipa-client-4.3.2-2.fc24.x86_64 > >>>> bind-dyndb-ldap-9.0-3.fc24.x86_64 > >>>> bind-libs-lite-9.10.4-1.P2.fc24.x86_64 > >>>> bind-pkcs11-libs-9.10.4-1.P2.fc24.x86_64 > >>>> bind99-libs-9.9.9-1.P2.fc24.x86_64 > >>>> bind-utils-9.10.4-1.P2.fc24.x86_64 > >>>> rpcbind-0.2.3-11.rc1.fc24.x86_64 > >>>> bind-license-9.10.4-1.P2.fc24.noarch > >>>> bind-pkcs11-9.10.4-1.P2.fc24.x86_64 > >>>> bind-9.10.4-1.P2.fc24.x86_64 > >>>> bind-libs-9.10.4-1.P2.fc24.x86_64 > >>>> bind99-license-9.9.9-1.P2.fc24.noarch > >>>> bind-pkcs11-utils-9.10.4-1.P2.fc24.x86_64 > >>>> > >>>> It seems that there is a regular but not daily "rndc reload" sent to > >>>> the > >>>> nameserver that leads to a crash of it. I sent a SIGHUP to the named > >>>> process, but that didn't lead to a crash. Only "rndc reload" does. It > >>>> does not crash EVERY time, but most of the times. I need to do an > >>>> "ipactl restart" in order to get the nameserver up and running again. > >>>> > >>>> I found this thread, but this doesn't give me any clues: > >>>> https://www.redhat.com/archives/freeipa-users/2012-May/msg00340.html > >>>> > >>>> This is what the log says: > >>>> http://paste.debian.net/818354/ > >>>> Please understand that I obfuscated my IP addresses and domain names > >>>> > >>>> This is the strace: > >>>> http://paste.debian.net/818365/ > >>>> > >>>> This is my named.conf: > >>>> http://paste.debian.net/818368/ > >>>> > >>>> Hope someone can help. > >>>> Jochen > >>> > >>> I wish I could, as I have the same issue, usually early Sunday morning > >>> after some cron/timer job that reloads: > >>> > >>> https://bugzilla.redhat.com/show_bug.cgi?id=1362162 > >> > >> Could you please try bind-dyndb-ldap-10.1-1.fc24 from updates-testing? > >> > >> Alternatively the package can be downloaded from > >> http://koji.fedoraproject.org/koji/buildinfo?buildID=792505 > >> > >> Please let me know if it fixes your problem or not. > > > > bind-dyndb-ldap-10.1-1.fc24 from updates-testing seems to work, or at > > least is does not crash on rndc reload. I'll give it some time and see > > what happens, since it didn't crash every time before either. Thank you > > Petr. -A > Thanks, I will be waiting for your observations! Petr, https://bodhi.fedoraproject.org/updates/FEDORA-2016-ea30aafae1 seems to have resolved this issue. I've updated Bodhi/Bugzilla. Thanks. -A -- Anthony - https://messinet.com/ - https://messinet.com/~amessina/gallery F9B6 560E 68EA 037D 8C3D D1C9 FF31 3BDB D9D8 99B6 -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 819 bytes Desc: This is a digitally signed message part. URL: From piolet.y at gmail.com Sun Sep 25 19:35:11 2016 From: piolet.y at gmail.com (Youenn PIOLET) Date: Sun, 25 Sep 2016 21:35:11 +0200 Subject: [Freeipa-users] Server replication stopped working In-Reply-To: <7854DA02-1936-448E-B4F7-791289F1F1A0@oracle.com> References: <7854DA02-1936-448E-B4F7-791289F1F1A0@oracle.com> Message-ID: Hi there, Same issue for me in a my 15 ipa-servers multi-master grid just after the update. The replication is completely broken except on 3/15 nodes. This is the second time I have to fully reinitialize the whole cluster for similar reason. I don't know what to do to clean this mess... For more information: this cluster has been initialized on a fedora 4.1.4 more than one year ago then complemetely migrated to Centos 7, IPA 4.2. Example on fr-master03 error logs: [25/Sep/2016:19:27:31 +0000] NSMMReplicationPlugin - changelog program - agmt="cn=meTofr-master01.domain" (fr-master01:389): CSN 57e3ffcc0003001a0000 not found, we aren't as up to date, or we purged [25/Sep/2016:19:27:31 +0000] NSMMReplicationPlugin - agmt="cn=meTofr-master01.domain" (fr-master01:389): Data required to update replica has been purged. The replica must be reinitialized. [25/Sep/2016:19:27:31 +0000] NSMMReplicationPlugin - agmt="cn=meTofr-master01.domain" (fr-master01:389): Incremental update failed and requires administrator action ipa: INFO: The ipactl command was successful [25/Sep/2016:19:27:35 +0000] agmt="cn=meTofr-master02.domain" (fr-master02:389) - Can't locate CSN 57e3ffcc0003001a0000 in the changelog (DB rc=-30988). If replication stops, the consumer may need to be reinitialized. [25/Sep/2016:19:27:35 +0000] NSMMReplicationPlugin - changelog program - agmt="cn=meTofr-master02.domain" (fr-master02:389): CSN 57e3ffcc0003001a0000 not found, we aren't as up to date, or we purged [25/Sep/2016:19:27:35 +0000] NSMMReplicationPlugin - agmt="cn=meTofr-master02.domain" (fr-master02:389): Data required to update replica has been purged. The replica must be reinitialized. [25/Sep/2016:19:27:35 +0000] NSMMReplicationPlugin - agmt="cn=meTofr-master02.domain" (fr-master02:389): Incremental update failed and requires administrator action Regards, -- Youenn Piolet piolet.y at gmail.com 2016-09-23 17:51 GMT+02:00 Mike Driscoll : > Hello. I have four IPA servers replicating in full mesh. All four > servers are running ipa-server-4.2.0-15.0.1.el7_2.19.x86_64. > > This was working for some time but now I see that no replication is > occurring automatically at present. > > When I update a user attribute on an IPA server, I see errors like these: > [22/Sep/2016:16:53:49 -0700] attrlist_replace - attr_replace > (nsslapd-referral, ldap://ldap03.xx.com:389/o%3Dipaca) failed. > [22/Sep/2016:16:58:56 -0700] NSMMReplicationPlugin - agmt="cn= > masterAgreement1-ldap03.xx.com > -pki-tomcat" (ldap03:389): Incremental update failed and requires > administrator action > > I can reinitialize without errors. > ipa-csreplica-manage re-initialize --from=ldap01.xx.com > > ipa-replica-manage re-initialize --from=ldap01.xx.com > Afterwards I see my attribute (and other) changes are replicated on each > server I re-initialize from. But subsequently, replication doesn?t seem to > be happening. > > I reinitialized according to the steps in Table 8.7, ?Replication Errors?, > but subsequent replication isn?t occurring. Any suggestions? Is it safe > to identify one of my four servers as containing up-to-date data, then > sever and reinstate replication relationships with the other three? > > Mike > > > > > > > -- > Manage your subscription for the Freeipa-users mailing list: > https://www.redhat.com/mailman/listinfo/freeipa-users > Go to http://freeipa.org for more info on the project > -------------- next part -------------- An HTML attachment was scrubbed... URL: From matt at indigo.nu Mon Sep 26 00:37:29 2016 From: matt at indigo.nu (Matthew Sellers) Date: Sun, 25 Sep 2016 19:37:29 -0500 Subject: [Freeipa-users] Distributing user keytabs for non-interactive auth question Message-ID: Hi Guys, What is the best way to distribute a 'user' keytab to distribute keytabs to allow 'system users' to run scripts with non-interactive auth? Is it possible to use the ipa-getkeytab feature ( with "-r" option ) to request a keytab for a user principal? I see support for HOST and SERVICE keytabs, but nothing specific to user keytabs? Concept Example: ipa-getkeytab -s ipa_server -p cron_runner at REALM.COM -k ipa_cron.keytab -r KRB5_KTNAME=ipa_cron.keytab service.py Actual Results ( tried with tgt for cron_runner or admin ): [sysadmin at 01 ~]$ ipa-getkeytab -s coipa100 -p cron_runner at REALM.COM -kipa_cron.keytab -r Failed to parse result: Insufficient access rights My only other option is grab the keytab and copy it around after initial creation ( understanding that each keytab requests bumps the KVNO ). My goal is to make password-less authentication for automated processes as easy as possible to setup....ipa-getkeytab seems like its almost there? Love the work you guys are putting out, its a really cool system. Thanks, Matt From mrorourke at earthlink.net Mon Sep 26 02:03:00 2016 From: mrorourke at earthlink.net (Michael ORourke) Date: Sun, 25 Sep 2016 22:03:00 -0400 (EDT) Subject: [Freeipa-users] Distributing user keytabs for non-interactive auth question Message-ID: <23846404.1474855380692.JavaMail.wam@elwamui-rubis.atl.sa.earthlink.net> Matt, Try the following... # Get admin TGT kinit admin at REALM.COM # Get keytab for user account ipa-getkeytab -s coipa100 -p cron_runner at REALM.COM -k ipa_cron_runner.keytab # Clear tickets kdestroy # Request TGT using the keytab kinit -k -t ./cron_runner.keytab cron_runner at REALM.COM # List tickets klist I recommend including the username somewhere in the name of the keytab file itself which makes it easier to remember. Of course be careful with the permissions on the keytab file, because anyone that has read access to the keytab can get a TGT as that user. -Mike -----Original Message----- >From: Matthew Sellers >Sent: Sep 25, 2016 8:37 PM >To: freeipa-users at redhat.com >Subject: [Freeipa-users] Distributing user keytabs for non-interactive auth question > >Hi Guys, > >What is the best way to distribute a 'user' keytab to distribute >keytabs to allow 'system users' to run scripts with non-interactive >auth? Is it possible to use the ipa-getkeytab feature ( with "-r" >option ) to request a keytab for a user principal? I see support for >HOST and SERVICE keytabs, but nothing specific to user keytabs? > >Concept Example: > >ipa-getkeytab -s ipa_server -p cron_runner at REALM.COM -k ipa_cron.keytab -r >KRB5_KTNAME=ipa_cron.keytab service.py > >Actual Results ( tried with tgt for cron_runner or admin ): > >[sysadmin at 01 ~]$ ipa-getkeytab -s coipa100 -p cron_runner at REALM.COM >-kipa_cron.keytab -r >Failed to parse result: Insufficient access rights > >My only other option is grab the keytab and copy it around after >initial creation ( understanding that each keytab requests bumps the >KVNO ). My goal is to make password-less authentication for automated >processes as easy as possible to setup....ipa-getkeytab seems like its >almost there? > >Love the work you guys are putting out, its a really cool system. > >Thanks, >Matt > >-- >Manage your subscription for the Freeipa-users mailing list: >https://www.redhat.com/mailman/listinfo/freeipa-users >Go to http://freeipa.org for more info on the project From william.muriithi at gmail.com Mon Sep 26 02:08:08 2016 From: william.muriithi at gmail.com (William Muriithi) Date: Sun, 25 Sep 2016 22:08:08 -0400 Subject: [Freeipa-users] NFS permissions after migrating to FreeIPA Message-ID: Hello, First, this may just be a coincidence and may have nothing to do with FreeIPA. However, I am running out of ideas and just wonder if anyone has seen it. The only change was to move them from openLDAP to FreeIPA. The automount were in place before this weekend and were working fine. However, I seem to have permission when using wildcard permissions. If I add the line below to /etc/exports, it works fine. This fixes it: /export platinum.eng.example.com(rw,sync,no_root_squash) However, its clearly covered by this wildcard permissions. All three lines the same access options - rw, sync and ro root squash [root at platinum ~]# showmount -e silicon Export list for silicon: /export/eng *.eng.example.com /export *.eng.example.com However, I do get an error that I don't have proper rights. Sep 25 21:54:15 platinum automount[13480]: mount_mount: mount(nfs): calling mkdir_path /home/rtdamgr Sep 25 21:54:15 platinum automount[13480]: mount_mount: mount(nfs): calling mount -t nfs -s -o intr 192.168.20.14:/export/eng/home/rtdamgr /home/rtdamgr Sep 25 21:54:15 platinum automount[13480]: >> mount.nfs: access denied by server while mounting 192.168.20.14:/export/eng/home/rtdamgr Sep 25 21:54:15 platinum automount[13480]: mount(nfs): nfs: mount failure 192.168.20.14:/export/eng/home/rtdamgr on /home/rtdamgr Would anyone know why NFS wouldn't respect wildcard hostnames? Regards, William From matt at indigo.nu Mon Sep 26 02:22:27 2016 From: matt at indigo.nu (Matthew Sellers) Date: Sun, 25 Sep 2016 21:22:27 -0500 Subject: [Freeipa-users] Distributing user keytabs for non-interactive auth question In-Reply-To: <23846404.1474855380692.JavaMail.wam@elwamui-rubis.atl.sa.earthlink.net> References: <23846404.1474855380692.JavaMail.wam@elwamui-rubis.atl.sa.earthlink.net> Message-ID: Hey Mike, Thanks for the reply. I did use this originally when deploying my 'kerberized' service on my first host. What I am trying to do is use ipa-getkeytab for keytab distribution on say...100 hosts, without having to copy around keytabs from host to host. Since using ipa-getkeytab without the '-r' option just creates a new keytab with bumped KVNO ..and.. when I do use '-r' I recieve a message for 'Insufficient access rights' I am still fuzzy.... Can ipa-getkeytab be used for mass distribution of user keytabs with the -r option? Thanks Again! Matt On Sun, Sep 25, 2016 at 9:03 PM, Michael ORourke wrote: > Matt, > > Try the following... > > # Get admin TGT > kinit admin at REALM.COM > > # Get keytab for user account > ipa-getkeytab -s coipa100 -p cron_runner at REALM.COM -k ipa_cron_runner.keytab > > # Clear tickets > kdestroy > > # Request TGT using the keytab > kinit -k -t ./cron_runner.keytab cron_runner at REALM.COM > > # List tickets > klist > > I recommend including the username somewhere in the name of the keytab file itself which makes it easier to remember. Of course be careful with the permissions on the keytab file, because anyone that has read access to the keytab can get a TGT as that user. > > -Mike > > -----Original Message----- >>From: Matthew Sellers >>Sent: Sep 25, 2016 8:37 PM >>To: freeipa-users at redhat.com >>Subject: [Freeipa-users] Distributing user keytabs for non-interactive auth question >> >>Hi Guys, >> >>What is the best way to distribute a 'user' keytab to distribute >>keytabs to allow 'system users' to run scripts with non-interactive >>auth? Is it possible to use the ipa-getkeytab feature ( with "-r" >>option ) to request a keytab for a user principal? I see support for >>HOST and SERVICE keytabs, but nothing specific to user keytabs? >> >>Concept Example: >> >>ipa-getkeytab -s ipa_server -p cron_runner at REALM.COM -k ipa_cron.keytab -r >>KRB5_KTNAME=ipa_cron.keytab service.py >> >>Actual Results ( tried with tgt for cron_runner or admin ): >> >>[sysadmin at 01 ~]$ ipa-getkeytab -s coipa100 -p cron_runner at REALM.COM >>-kipa_cron.keytab -r >>Failed to parse result: Insufficient access rights >> >>My only other option is grab the keytab and copy it around after >>initial creation ( understanding that each keytab requests bumps the >>KVNO ). My goal is to make password-less authentication for automated >>processes as easy as possible to setup....ipa-getkeytab seems like its >>almost there? >> >>Love the work you guys are putting out, its a really cool system. >> >>Thanks, >>Matt >> >>-- >>Manage your subscription for the Freeipa-users mailing list: >>https://www.redhat.com/mailman/listinfo/freeipa-users >>Go to http://freeipa.org for more info on the project > > -- > Manage your subscription for the Freeipa-users mailing list: > https://www.redhat.com/mailman/listinfo/freeipa-users > Go to http://freeipa.org for more info on the project From mbabinsk at redhat.com Mon Sep 26 07:22:46 2016 From: mbabinsk at redhat.com (Martin Babinsky) Date: Mon, 26 Sep 2016 09:22:46 +0200 Subject: [Freeipa-users] Distributing user keytabs for non-interactive auth question In-Reply-To: References: <23846404.1474855380692.JavaMail.wam@elwamui-rubis.atl.sa.earthlink.net> Message-ID: <8934c824-37b3-aac8-568e-f0d45288a1a8@redhat.com> On 09/26/2016 04:22 AM, Matthew Sellers wrote: > Hey Mike, > > Thanks for the reply. I did use this originally when deploying my > 'kerberized' service on my first host. What I am trying to do is use > ipa-getkeytab for keytab distribution on say...100 hosts, without > having to copy around keytabs from host to host. > > Since using ipa-getkeytab without the '-r' option just creates a new > keytab with bumped KVNO ..and.. when I do use '-r' I recieve a message > for 'Insufficient access rights' I am still fuzzy.... > > Can ipa-getkeytab be used for mass distribution of user keytabs with > the -r option? > > Thanks Again! > Matt > > > > On Sun, Sep 25, 2016 at 9:03 PM, Michael ORourke > wrote: >> Matt, >> >> Try the following... >> >> # Get admin TGT >> kinit admin at REALM.COM >> >> # Get keytab for user account >> ipa-getkeytab -s coipa100 -p cron_runner at REALM.COM -k ipa_cron_runner.keytab >> >> # Clear tickets >> kdestroy >> >> # Request TGT using the keytab >> kinit -k -t ./cron_runner.keytab cron_runner at REALM.COM >> >> # List tickets >> klist >> >> I recommend including the username somewhere in the name of the keytab file itself which makes it easier to remember. Of course be careful with the permissions on the keytab file, because anyone that has read access to the keytab can get a TGT as that user. >> >> -Mike >> >> -----Original Message----- >>> From: Matthew Sellers >>> Sent: Sep 25, 2016 8:37 PM >>> To: freeipa-users at redhat.com >>> Subject: [Freeipa-users] Distributing user keytabs for non-interactive auth question >>> >>> Hi Guys, >>> >>> What is the best way to distribute a 'user' keytab to distribute >>> keytabs to allow 'system users' to run scripts with non-interactive >>> auth? Is it possible to use the ipa-getkeytab feature ( with "-r" >>> option ) to request a keytab for a user principal? I see support for >>> HOST and SERVICE keytabs, but nothing specific to user keytabs? >>> >>> Concept Example: >>> >>> ipa-getkeytab -s ipa_server -p cron_runner at REALM.COM -k ipa_cron.keytab -r >>> KRB5_KTNAME=ipa_cron.keytab service.py >>> >>> Actual Results ( tried with tgt for cron_runner or admin ): >>> >>> [sysadmin at 01 ~]$ ipa-getkeytab -s coipa100 -p cron_runner at REALM.COM >>> -kipa_cron.keytab -r >>> Failed to parse result: Insufficient access rights >>> >>> My only other option is grab the keytab and copy it around after >>> initial creation ( understanding that each keytab requests bumps the >>> KVNO ). My goal is to make password-less authentication for automated >>> processes as easy as possible to setup....ipa-getkeytab seems like its >>> almost there? >>> >>> Love the work you guys are putting out, its a really cool system. >>> >>> Thanks, >>> Matt >>> >>> -- >>> Manage your subscription for the Freeipa-users mailing list: >>> https://www.redhat.com/mailman/listinfo/freeipa-users >>> Go to http://freeipa.org for more info on the project >> >> -- >> Manage your subscription for the Freeipa-users mailing list: >> https://www.redhat.com/mailman/listinfo/freeipa-users >> Go to http://freeipa.org for more info on the project > The problem is that in order to retrieve an existing Kerberos keys the getkeytab extended operation need to be able to read them. The support for these permissions is currently implemented for hosts and services only (see http://www.freeipa.org/page/V4/Keytab_Retrieval_Management for more details). Maybe you can workaround this by retrieving keytabs as a directory manager but then you have to enter directory manager password everywhere. Also there is a considerable security risk involved in storing user keytabs e.g. in their home directories, as anyone who gains privileged access to the enrolled machine can then impersonate any user that has a keytab stored on it. -- Martin^3 Babinsky From th at casalogic.dk Mon Sep 26 07:25:46 2016 From: th at casalogic.dk (Troels Hansen) Date: Mon, 26 Sep 2016 09:25:46 +0200 (CEST) Subject: [Freeipa-users] SSH using putty to IPA client Message-ID: <1098516169.379802.1474874746409.JavaMail.zimbra@casalogic.dk> After we installed a new set of IPA servers for prod, and joined AD using username and password to have AD create a correct suffix routing everythin seems to work, and the suffix routing is created correctly on AD. However, trying to SSH from Windows using Putty and kerberos fails: Putty log shows: Event Log: GSSAPI authentication initialisation failed Event Log: No authority could be contacted for authentication.The domain name of the authenticating party could be wrong, the domain could be unreachable, or there might have been a trust relationship failure. DNS is on AD (manually added, and IPA have no DNS installed. Kerberos DNS is correct: # dig _kerberos._tcp.lx.dr.dk SRV .... ;; ANSWER SECTION: _kerberos._tcp.lx.dr.dk. 3600 IN SRV 0 100 88 ipa01.lx.dr.dk. _kerberos._tcp.lx.dr.dk. 3600 IN SRV 0 100 88 ipa02.lx.dr.dk. ;; ADDITIONAL SECTION: ipa01.lx.dr.dk. 3600 IN A x.y.z.135 ipa02.lx.dr.dk. 3600 IN A x.y.z.134 # dig _kerberos._tcp.dc._msdcs.lx.dr.dk SRV ... ;; ANSWER SECTION: _kerberos._tcp.dc._msdcs.lx.dr.dk. 3600 IN SRV 0 100 88 ipa02.lx.dr.dk. _kerberos._tcp.dc._msdcs.lx.dr.dk. 3600 IN SRV 0 100 88 ipa01.lx.dr.dk. ;; ADDITIONAL SECTION: ipa02.lx.dr.dk. 3600 IN A x.y.z.134 ipa01.lx.dr.dk. 3600 IN A x.y.z.135 Klist on Windows shows I have a TGT for the LX domain (but only a TGT), sorry for the danish. #0> Klient: drextrha @ NET.DR.DK Server: krbtgt/LX.DR.DK @ PLACE.DR.DK KerbTicket-krypteringstype: AES-256-CTS-HMAC-SHA1-96 Billetflag 0x40a50000 -> forwardable renewable pre_authent ok_as_delegate name_canonicalize Starttidspunkt: 9/21/2016 14:58:36 (lokal) Sluttidspunkt: 9/21/2016 23:16:09 (lokal) Fornyelsestidspunkt: 9/28/2016 13:16:09 (lokal) Sessionsn?gletype: AES-256-CTS-HMAC-SHA1-96 I can't see whats wrong and can't seem to find out whats wrong? Suggestions welcome :-) -------------- next part -------------- An HTML attachment was scrubbed... URL: From lkrispen at redhat.com Mon Sep 26 07:42:04 2016 From: lkrispen at redhat.com (Ludwig Krispenz) Date: Mon, 26 Sep 2016 09:42:04 +0200 Subject: [Freeipa-users] Server replication stopped working In-Reply-To: References: <7854DA02-1936-448E-B4F7-791289F1F1A0@oracle.com> Message-ID: <57E8D14C.4070203@redhat.com> On 09/25/2016 09:35 PM, Youenn PIOLET wrote: > Hi there, > > Same issue for me in a my 15 ipa-servers multi-master grid just after > the update. > The replication is completely broken except on 3/15 nodes. > > This is the second time I have to fully reinitialize the whole cluster > for similar reason. I don't know what to do to clean this mess... > For more information: this cluster has been initialized on a fedora > 4.1.4 more than one year ago then complemetely migrated to Centos 7, > IPA 4.2. what is the exact version of 389-ds-base you are running ? did these errors come out of the blue or are they related to some activities ? The csn which is not found has a timestamp of "Thu, 22 Sep 2016 15:59:08 GMT" did anything happen around this time ? > > Example on fr-master03 error logs: > > [25/Sep/2016:19:27:31 +0000] NSMMReplicationPlugin - changelog program > - agmt="cn=meTofr-master01.domain" (fr-master01:389): CSN > 57e3ffcc0003001a0000 not found, we aren't as up to date, or we purged > [25/Sep/2016:19:27:31 +0000] NSMMReplicationPlugin - > agmt="cn=meTofr-master01.domain" (fr-master01:389): Data required to > update replica has been purged. The replica must be reinitialized. > [25/Sep/2016:19:27:31 +0000] NSMMReplicationPlugin - > agmt="cn=meTofr-master01.domain" (fr-master01:389): Incremental update > failed and requires administrator action > ipa: INFO: The ipactl command was successful > [25/Sep/2016:19:27:35 +0000] agmt="cn=meTofr-master02.domain" > (fr-master02:389) - Can't locate CSN 57e3ffcc0003001a0000 in the > changelog (DB rc=-30988). If replication stops, the consumer may need > to be reinitialized. > [25/Sep/2016:19:27:35 +0000] NSMMReplicationPlugin - changelog program > - agmt="cn=meTofr-master02.domain" (fr-master02:389): CSN > 57e3ffcc0003001a0000 not found, we aren't as up to date, or we purged > [25/Sep/2016:19:27:35 +0000] NSMMReplicationPlugin - > agmt="cn=meTofr-master02.domain" (fr-master02:389): Data required to > update replica has been purged. The replica must be reinitialized. > [25/Sep/2016:19:27:35 +0000] NSMMReplicationPlugin - > agmt="cn=meTofr-master02.domain" (fr-master02:389): Incremental update > failed and requires administrator action > > Regards, > > -- > Youenn Piolet > piolet.y at gmail.com > / > / > > 2016-09-23 17:51 GMT+02:00 Mike Driscoll >: > > Hello. I have four IPA servers replicating in full mesh. All > four servers are running ipa-server-4.2.0-15.0.1.el7_2.19.x86_64. > > This was working for some time but now I see that no replication > is occurring automatically at present. > > When I update a user attribute on an IPA server, I see errors like > these: > [22/Sep/2016:16:53:49 -0700] attrlist_replace - attr_replace > (nsslapd-referral, ldap://ldap03.xx.com:389/o%3Dipaca) failed. > [22/Sep/2016:16:58:56 -0700] NSMMReplicationPlugin - > agmt="cn=masterAgreement1-ldap03.xx.com > -pki-tomcat" (ldap03:389): > Incremental update failed and requires administrator action > > I can reinitialize without errors. > ipa-csreplica-manage re-initialize --from=ldap01.xx.com > > ipa-replica-manage re-initialize --from=ldap01.xx.com > > Afterwards I see my attribute (and other) changes are replicated > on each server I re-initialize from. But subsequently, > replication doesn't seem to be happening. > > I reinitialized according to the steps in Table 8.7, "Replication > Errors", but subsequent replication isn't occurring. Any > suggestions? Is it safe to identify one of my four servers as > containing up-to-date data, then sever and reinstate replication > relationships with the other three? > > Mike > > > > > > > -- > Manage your subscription for the Freeipa-users mailing list: > https://www.redhat.com/mailman/listinfo/freeipa-users > > Go to http://freeipa.org for more info on the project > > > > -- Red Hat GmbH, http://www.de.redhat.com/, Registered seat: Grasbrunn, Commercial register: Amtsgericht Muenchen, HRB 153243, Managing Directors: Charles Cachera, Michael Cunningham, Michael O'Neill, Eric Shander -------------- next part -------------- An HTML attachment was scrubbed... URL: From deepak_dimri at hotmail.com Mon Sep 26 07:57:10 2016 From: deepak_dimri at hotmail.com (Deepak Dimri) Date: Mon, 26 Sep 2016 07:57:10 +0000 Subject: [Freeipa-users] SSH key based login for the users Message-ID: Hi All, Can i have my IPA server pre-configured with RSA and public key authentication enabled (passwordauthentication no) for its users and at the same time have users to automatically register with their ssh key pair during first time login process so that they can login with the keys? i am wondering what would the advisable workflow to issue private keys to users so that they can login with it to freeIPA server? This is a great forum hence thought of asking this question while i am still exploring on this. Thanks & Regards, Deepak -------------- next part -------------- An HTML attachment was scrubbed... URL: From sbose at redhat.com Mon Sep 26 08:18:53 2016 From: sbose at redhat.com (Sumit Bose) Date: Mon, 26 Sep 2016 10:18:53 +0200 Subject: [Freeipa-users] SSH using putty to IPA client In-Reply-To: <1098516169.379802.1474874746409.JavaMail.zimbra@casalogic.dk> References: <1098516169.379802.1474874746409.JavaMail.zimbra@casalogic.dk> Message-ID: <20160926081853.GQ6019@p.Speedport_W_724V_Typ_A_05011603_00_009> On Mon, Sep 26, 2016 at 09:25:46AM +0200, Troels Hansen wrote: > After we installed a new set of IPA servers for prod, and joined AD using username and password to have AD create a correct suffix routing everythin seems to work, and the suffix routing is created correctly on AD. > > However, trying to SSH from Windows using Putty and kerberos fails: > > Putty log shows: > Event Log: GSSAPI authentication initialisation failed > Event Log: No authority could be contacted for authentication.The domain name of the authenticating party could be wrong, the domain could be unreachable, or there might have been a trust relationship failure. > > DNS is on AD (manually added, and IPA have no DNS installed. > > Kerberos DNS is correct: > > # dig _kerberos._tcp.lx.dr.dk SRV > .... > ;; ANSWER SECTION: > _kerberos._tcp.lx.dr.dk. 3600 IN SRV 0 100 88 ipa01.lx.dr.dk. > _kerberos._tcp.lx.dr.dk. 3600 IN SRV 0 100 88 ipa02.lx.dr.dk. > > ;; ADDITIONAL SECTION: > ipa01.lx.dr.dk. 3600 IN A x.y.z.135 > ipa02.lx.dr.dk. 3600 IN A x.y.z.134 > > > # dig _kerberos._tcp.dc._msdcs.lx.dr.dk SRV > ... > ;; ANSWER SECTION: > _kerberos._tcp.dc._msdcs.lx.dr.dk. 3600 IN SRV 0 100 88 ipa02.lx.dr.dk. > _kerberos._tcp.dc._msdcs.lx.dr.dk. 3600 IN SRV 0 100 88 ipa01.lx.dr.dk. > > ;; ADDITIONAL SECTION: > ipa02.lx.dr.dk. 3600 IN A x.y.z.134 > ipa01.lx.dr.dk. 3600 IN A x.y.z.135 > > > Klist on Windows shows I have a TGT for the LX domain (but only a TGT), sorry for the danish. > > #0> Klient: drextrha @ NET.DR.DK > Server: krbtgt/LX.DR.DK @ PLACE.DR.DK > KerbTicket-krypteringstype: AES-256-CTS-HMAC-SHA1-96 > Billetflag 0x40a50000 -> forwardable renewable pre_authent ok_as_delegate name_canonicalize > Starttidspunkt: 9/21/2016 14:58:36 (lokal) > Sluttidspunkt: 9/21/2016 23:16:09 (lokal) > Fornyelsestidspunkt: 9/28/2016 13:16:09 (lokal) > Sessionsn?gletype: AES-256-CTS-HMAC-SHA1-96 > > > I can't see whats wrong and can't seem to find out whats wrong? > Suggestions welcome :-) Have you checked the firewalls? AD clients must be able to talk to the KDC port (88 udp and tcp) on the IPA servers to get service tickets for IPA hosts. HTH bye, Sumit > -- > Manage your subscription for the Freeipa-users mailing list: > https://www.redhat.com/mailman/listinfo/freeipa-users > Go to http://freeipa.org for more info on the project From flo at redhat.com Mon Sep 26 08:45:12 2016 From: flo at redhat.com (Florence Blanc-Renaud) Date: Mon, 26 Sep 2016 10:45:12 +0200 Subject: [Freeipa-users] Question Test 3rd Party Certificate In-Reply-To: <1480663.xLjgThpfif@techz> References: <1480663.xLjgThpfif@techz> Message-ID: On 09/24/2016 02:37 PM, G?nther J. Niederwimmer wrote: > Hello, > > what is the best way to test a new installed 3rd Party certificate ? > I hope i have now install (with big problems) the new certificate on clients > and servers. > > But now is the big question is this all working correct together (?), or have > i make a mistake ? > > I like to install this on a productive server with two master and 8 clients > Freeipa 4.2 Centos 7 with all Updates > > with MailServer, private Cloud, webserver, DNS server ..... > > the next question is, what is in three years when the certificates expire ? > Is there a tested way to renew the certificate ? > > I have search a long time in the internet but I can't found answers ? Hi, you can find the supported procedure here: Using 3rd part certificates for HTTP/LDAP [1]. We are currently working on improving the chapter "Managing Certificates and Certificate Authorities" of the "Linux Domain Identity, Authentication, and Policy Guide" [2]. If you feel that some information is missing, please file documentation bugs so that we can take your comments into account for the next revision. Depending on your deployment constraints, you may also consider installing FreeIPA's certificate authority using ipa-ca-install. This would allow to have HTTP/LDAP certificates issued *and renewed automatically* by FreeIPA CA. Hope this helps, Flo. [1] http://www.freeipa.org/page/Using_3rd_part_certificates_for_HTTP/LDAP [2] https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/7/html/Linux_Domain_Identity_Authentication_and_Policy_Guide/config-certificates.html > > Thanks for a answer, > From th at casalogic.dk Mon Sep 26 11:11:49 2016 From: th at casalogic.dk (Troels Hansen) Date: Mon, 26 Sep 2016 13:11:49 +0200 (CEST) Subject: [Freeipa-users] SSH using putty to IPA client In-Reply-To: <20160926081853.GQ6019@p.Speedport_W_724V_Typ_A_05011603_00_009> References: <1098516169.379802.1474874746409.JavaMail.zimbra@casalogic.dk> <20160926081853.GQ6019@p.Speedport_W_724V_Typ_A_05011603_00_009> Message-ID: <1313111984.386014.1474888309619.JavaMail.zimbra@casalogic.dk> ----- On Sep 26, 2016, at 10:18 AM, Sumit Bose sbose at redhat.com wrote: > > Have you checked the firewalls? AD clients must be able to talk to the > KDC port (88 udp and tcp) on the IPA servers to get service tickets for > IPA hosts. > KDC ports seems to work.... Besides, I don't have a TGT for the IPA (LX) domain, untill I try to SSH to it. I guess I shouldn't be able to if KDC traffic was blocked? From abokovoy at redhat.com Mon Sep 26 11:24:20 2016 From: abokovoy at redhat.com (Alexander Bokovoy) Date: Mon, 26 Sep 2016 14:24:20 +0300 Subject: [Freeipa-users] SSH using putty to IPA client In-Reply-To: <1313111984.386014.1474888309619.JavaMail.zimbra@casalogic.dk> References: <1098516169.379802.1474874746409.JavaMail.zimbra@casalogic.dk> <20160926081853.GQ6019@p.Speedport_W_724V_Typ_A_05011603_00_009> <1313111984.386014.1474888309619.JavaMail.zimbra@casalogic.dk> Message-ID: <20160926112420.k522kqnbawqc3n7o@redhat.com> On ma, 26 syys 2016, Troels Hansen wrote: > > >----- On Sep 26, 2016, at 10:18 AM, Sumit Bose sbose at redhat.com wrote: > >> >> Have you checked the firewalls? AD clients must be able to talk to the >> KDC port (88 udp and tcp) on the IPA servers to get service tickets for >> IPA hosts. >> > > >KDC ports seems to work.... Besides, I don't have a TGT for the IPA >(LX) domain, untill I try to SSH to it. I guess I shouldn't be able to >if KDC traffic was blocked? cross-realm TGT is issued by your AD DC. -- / Alexander Bokovoy From sbose at redhat.com Mon Sep 26 11:30:18 2016 From: sbose at redhat.com (Sumit Bose) Date: Mon, 26 Sep 2016 13:30:18 +0200 Subject: [Freeipa-users] SSH using putty to IPA client In-Reply-To: <1313111984.386014.1474888309619.JavaMail.zimbra@casalogic.dk> References: <1098516169.379802.1474874746409.JavaMail.zimbra@casalogic.dk> <20160926081853.GQ6019@p.Speedport_W_724V_Typ_A_05011603_00_009> <1313111984.386014.1474888309619.JavaMail.zimbra@casalogic.dk> Message-ID: <20160926113018.GU6019@p.Speedport_W_724V_Typ_A_05011603_00_009> On Mon, Sep 26, 2016 at 01:11:49PM +0200, Troels Hansen wrote: > > > ----- On Sep 26, 2016, at 10:18 AM, Sumit Bose sbose at redhat.com wrote: > > > > > Have you checked the firewalls? AD clients must be able to talk to the > > KDC port (88 udp and tcp) on the IPA servers to get service tickets for > > IPA hosts. > > > > > KDC ports seems to work.... Besides, I don't have a TGT for the IPA (LX) domain, untill I try to SSH to it. I guess I shouldn't be able to if KDC traffic was blocked? The cross-realm TGT 'krbtgt/LX.DR.DK @ PLACE.DR.DK' is issued by the AD DC. So this is not indication that the IPA KDC can be reached by the AD client. Do you see and log messages in the krb5kdc.log on the IPA server? If it is not the firewall I would suggest to record the IP traffic of the AD client and check what it tries to do after the AD DC send the cross-realm TGT. About the DNS SRV records, did you add matching records for _udp as well? I'm not sure if the AD client will fallback to _tcp if they are missing or just stop? HTH bye, Sumit From natxo.asenjo at gmail.com Mon Sep 26 11:36:05 2016 From: natxo.asenjo at gmail.com (Natxo Asenjo) Date: Mon, 26 Sep 2016 13:36:05 +0200 Subject: [Freeipa-users] replicas removed, but incorrectly Message-ID: hi, I recently upgraded a centos 6.8 realm to centos 7.2 and it almost went correctly. Now I see some errors in /var/log/dirsrv/slapd-INSTANCENAME/errors 26/Sep/2016:13:20:15 +0200] attrlist_replace - attr_replace (nsslapd-referral, ldap://kdc03.unix.iriszorg.nl:389/o%3Dipaca) failed and according to http://www.freeipa.org/page/Troubleshooting#Replication_issues this points to a ruv problem. So let's enumerate. We had kdc01 replicating to kdc02 (both 6.8). Then I created a replica from kdc01 to kdc03 (running 7.2). And from kdc03 to kdc04 (both 7.2). kdc01 and kdc02 are decommissioned, but kdc02 still shows in both kdc03 and kdc04: $ ipa-replica-manage list kdc02.unix.iriszorg.nl: master kdc03.unix.iriszorg.nl: master kdc04.unix.iriszorg.nl: master and in $ ipa-csreplica-manage list Directory Manager password: kdc02.unix.iriszorg.nl: master kdc03.unix.iriszorg.nl: master kdc04.unix.iriszorg.nl: master >From kdc03: $ ldapsearch -Z -h kdc04.unix.iriszorg.nl -D "cn=Directory Manager" -W -b "o=ipaca" "(&(objectclass=nstombstone)(nsUniqueId=ffffffff-ffffffff-ffffffff-ffffffff))" | grep "nsds50ruv\|nsDS5ReplicaId" Enter LDAP Password: nsDS5ReplicaId: 1095 nsds50ruv: {replicageneration} 50c1015c000000600000 nsds50ruv: {replica 1095 ldap://kdc04.unix.iriszorg.nl:389} 57e4d75a0000044700 nsds50ruv: {replica 66 ldap://kdc03.unix.iriszorg.nl:389} 57e23f66000000420000 nsds50ruv: {replica 96 ldap://kdc01.unix.iriszorg.nl:7389} 50c1016c00000060000 nsds50ruv: {replica 71 ldap://kdc03.unix.iriszorg.nl:389} 57e140c7000000470000 nsds50ruv: {replica 97 ldap://kdc02.unix.iriszorg.nl:7389} 50c1016800000061000 and from kdc04: # ldapsearch -Z -h kdc04.unix.iriszorg.nl -D "cn=Directory Manager" -W -b "o=ipaca" "(&(objectclass=nstombstone)(nsUniqueId=ffffffff-ffffffff-ffffffff-ffffffff))" | grep "nsds50ruv\|nsDS5ReplicaId" Enter LDAP Password: nsDS5ReplicaId: 1095 nsds50ruv: {replicageneration} 50c1015c000000600000 nsds50ruv: {replica 1095 ldap://kdc04.unix.iriszorg.nl:389} 57e4d75a0000044700 nsds50ruv: {replica 66 ldap://kdc03.unix.iriszorg.nl:389} 57e23f66000000420000 nsds50ruv: {replica 96 ldap://kdc01.unix.iriszorg.nl:7389} 50c1016c00000060000 nsds50ruv: {replica 71 ldap://kdc03.unix.iriszorg.nl:389} 57e140c7000000470000 nsds50ruv: {replica 97 ldap://kdc02.unix.iriszorg.nl:7389} 50c1016800000061000 So now I have to run a clen ruv task like this (as seen in https://www.redhat.com/archives/freeipa-users/2016-May/msg00043.html): # ldapmodify -ZZ -D "cn=directory manager" -W -a dn: cn=clean 13, cn=cleanallruv, cn=tasks, cn=config objectclass: extensibleObject replica-base-dn: o=ipaca replica-id: 13 cn: clean 13 And in my example, the replica id would be 66, 96, 71 and 97, correct? Thanks for confirming this, never done it before. -- Groeten, natxo -------------- next part -------------- An HTML attachment was scrubbed... URL: From lkrispen at redhat.com Mon Sep 26 11:50:24 2016 From: lkrispen at redhat.com (Ludwig Krispenz) Date: Mon, 26 Sep 2016 13:50:24 +0200 Subject: [Freeipa-users] replicas removed, but incorrectly In-Reply-To: References: Message-ID: <57E90B80.1020106@redhat.com> On 09/26/2016 01:36 PM, Natxo Asenjo wrote: > hi, > > I recently upgraded a centos 6.8 realm to centos 7.2 and it almost > went correctly. > > Now I see some errors in /var/log/dirsrv/slapd-INSTANCENAME/errors > > 26/Sep/2016:13:20:15 +0200] attrlist_replace - attr_replace > (nsslapd-referral, ldap://kdc03.unix.iriszorg.nl:389/o%3Dipaca > ) failed > > and according to > http://www.freeipa.org/page/Troubleshooting#Replication_issues this > points to a ruv problem. > > So let's enumerate. > > We had kdc01 replicating to kdc02 (both 6.8). > > Then I created a replica from kdc01 to kdc03 (running 7.2). > > And from kdc03 to kdc04 (both 7.2). > > kdc01 and kdc02 are decommissioned, but kdc02 still shows in both > kdc03 and kdc04: > > $ ipa-replica-manage list > kdc02.unix.iriszorg.nl : master > kdc03.unix.iriszorg.nl : master > kdc04.unix.iriszorg.nl : master > > and in > > $ ipa-csreplica-manage list > Directory Manager password: > kdc02.unix.iriszorg.nl : master > kdc03.unix.iriszorg.nl : master > kdc04.unix.iriszorg.nl : master > > > >From kdc03: > $ ldapsearch -Z -h kdc04.unix.iriszorg.nl > -D "cn=Directory Manager" -W -b > "o=ipaca" > "(&(objectclass=nstombstone)(nsUniqueId=ffffffff-ffffffff-ffffffff-ffffffff))" > | grep "nsds50ruv\|nsDS5ReplicaId" > Enter LDAP Password: > nsDS5ReplicaId: 1095 > nsds50ruv: {replicageneration} 50c1015c000000600000 > nsds50ruv: {replica 1095 ldap://kdc04.unix.iriszorg.nl:389 > } 57e4d75a0000044700 > nsds50ruv: {replica 66 ldap://kdc03.unix.iriszorg.nl:389 > } 57e23f66000000420000 > nsds50ruv: {replica 96 ldap://kdc01.unix.iriszorg.nl:7389 > } 50c1016c00000060000 > nsds50ruv: {replica 71 ldap://kdc03.unix.iriszorg.nl:389 > } 57e140c7000000470000 > nsds50ruv: {replica 97 ldap://kdc02.unix.iriszorg.nl:7389 > } 50c1016800000061000 > > and from kdc04: > > # ldapsearch -Z -h kdc04.unix.iriszorg.nl > -D "cn=Directory Manager" -W -b > "o=ipaca" > "(&(objectclass=nstombstone)(nsUniqueId=ffffffff-ffffffff-ffffffff-ffffffff))" > | grep "nsds50ruv\|nsDS5ReplicaId" > Enter LDAP Password: > nsDS5ReplicaId: 1095 > nsds50ruv: {replicageneration} 50c1015c000000600000 > nsds50ruv: {replica 1095 ldap://kdc04.unix.iriszorg.nl:389 > } 57e4d75a0000044700 > nsds50ruv: {replica 66 ldap://kdc03.unix.iriszorg.nl:389 > } 57e23f66000000420000 > nsds50ruv: {replica 96 ldap://kdc01.unix.iriszorg.nl:7389 > } 50c1016c00000060000 > nsds50ruv: {replica 71 ldap://kdc03.unix.iriszorg.nl:389 > } 57e140c7000000470000 > nsds50ruv: {replica 97 ldap://kdc02.unix.iriszorg.nl:7389 > } 50c1016800000061000 > > > So now I have to run a clen ruv task like this (as seen in > https://www.redhat.com/archives/freeipa-users/2016-May/msg00043.html): > > # ldapmodify -ZZ -D "cn=directory manager" -W -a > dn: cn=clean 13, cn=cleanallruv, cn=tasks, cn=config > objectclass: extensibleObject > replica-base-dn: o=ipaca > replica-id: 13 > cn: clean 13 > > And in my example, the replica id would be 66, 96, 71 and 97, correct? no, I don't think so. you searched 2 times the same host "-h kdc04.unix.iriszorg.nl ". you need to search on kdc03 to find the current replicaid of kdc03 and you have to keep it. > Thanks for confirming this, never done it before. > -- > Groeten, > natxo > > -- Red Hat GmbH, http://www.de.redhat.com/, Registered seat: Grasbrunn, Commercial register: Amtsgericht Muenchen, HRB 153243, Managing Directors: Charles Cachera, Michael Cunningham, Michael O'Neill, Eric Shander -------------- next part -------------- An HTML attachment was scrubbed... URL: From natxo.asenjo at gmail.com Mon Sep 26 11:54:29 2016 From: natxo.asenjo at gmail.com (Natxo Asenjo) Date: Mon, 26 Sep 2016 13:54:29 +0200 Subject: [Freeipa-users] replicas removed, but incorrectly In-Reply-To: <57E90B80.1020106@redhat.com> References: <57E90B80.1020106@redhat.com> Message-ID: On Mon, Sep 26, 2016 at 1:50 PM, Ludwig Krispenz wrote: > > On 09/26/2016 01:36 PM, Natxo Asenjo wrote: > > hi, > > I recently upgraded a centos 6.8 realm to centos 7.2 and it almost went > correctly. > > Now I see some errors in /var/log/dirsrv/slapd-INSTANCENAME/errors > > 26/Sep/2016:13:20:15 +0200] attrlist_replace - attr_replace > (nsslapd-referral, ldap://kdc03.unix.iriszorg.nl:389/o%3Dipaca) failed > > and according to http://www.freeipa.org/page/Troubleshooting#Replication_ > issues this points to a ruv problem. > > So let's enumerate. > > We had kdc01 replicating to kdc02 (both 6.8). > > Then I created a replica from kdc01 to kdc03 (running 7.2). > > And from kdc03 to kdc04 (both 7.2). > > kdc01 and kdc02 are decommissioned, but kdc02 still shows in both kdc03 > and kdc04: > > $ ipa-replica-manage list > kdc02.unix.iriszorg.nl: master > kdc03.unix.iriszorg.nl: master > kdc04.unix.iriszorg.nl: master > > and in > > $ ipa-csreplica-manage list > Directory Manager password: > kdc02.unix.iriszorg.nl: master > kdc03.unix.iriszorg.nl: master > kdc04.unix.iriszorg.nl: master > > > >From kdc03: > $ ldapsearch -Z -h kdc04.unix.iriszorg.nl -D "cn=Directory Manager" -W -b > "o=ipaca" "(&(objectclass=nstombstone)(nsUniqueId=ffffffff-ffffffff-ffffffff-ffffffff))" > | grep "nsds50ruv\|nsDS5ReplicaId" > Enter LDAP Password: > nsDS5ReplicaId: 1095 > nsds50ruv: {replicageneration} 50c1015c000000600000 > nsds50ruv: {replica 1095 ldap://kdc04.unix.iriszorg.nl:389} > 57e4d75a0000044700 > nsds50ruv: {replica 66 ldap://kdc03.unix.iriszorg.nl:389} > 57e23f66000000420000 > nsds50ruv: {replica 96 ldap://kdc01.unix.iriszorg.nl:7389} > 50c1016c00000060000 > nsds50ruv: {replica 71 ldap://kdc03.unix.iriszorg.nl:389} > 57e140c7000000470000 > nsds50ruv: {replica 97 ldap://kdc02.unix.iriszorg.nl:7389} > 50c1016800000061000 > > and from kdc04: > > # ldapsearch -Z -h kdc04.unix.iriszorg.nl -D "cn=Directory Manager" -W -b > "o=ipaca" "(&(objectclass=nstombstone)(nsUniqueId=ffffffff-ffffffff-ffffffff-ffffffff))" > | grep "nsds50ruv\|nsDS5ReplicaId" > Enter LDAP Password: > nsDS5ReplicaId: 1095 > nsds50ruv: {replicageneration} 50c1015c000000600000 > nsds50ruv: {replica 1095 ldap://kdc04.unix.iriszorg.nl:389} > 57e4d75a0000044700 > nsds50ruv: {replica 66 ldap://kdc03.unix.iriszorg.nl:389} > 57e23f66000000420000 > nsds50ruv: {replica 96 ldap://kdc01.unix.iriszorg.nl:7389} > 50c1016c00000060000 > nsds50ruv: {replica 71 ldap://kdc03.unix.iriszorg.nl:389} > 57e140c7000000470000 > nsds50ruv: {replica 97 ldap://kdc02.unix.iriszorg.nl:7389} > 50c1016800000061000 > > > So now I have to run a clen ruv task like this (as seen in > https://www.redhat.com/archives/freeipa-users/2016-May/msg00043.html): > > # ldapmodify -ZZ -D "cn=directory manager" -W -a > dn: cn=clean 13, cn=cleanallruv, cn=tasks, cn=config > objectclass: extensibleObject > replica-base-dn: o=ipaca > replica-id: 13 > cn: clean 13 > > > And in my example, the replica id would be 66, 96, 71 and 97, correct? > > no, I don't think so. you searched 2 times the same host "-h > kdc04.unix.iriszorg.nl". > you need to search on kdc03 to find the current replicaid of kdc03 and you > have to keep it. > yes, you are right :( $ ldapsearch -Z -h kdc03.unix.iriszorg.nl -D "cn=Directory Manager" -W -b "o=ipaca" "(&(objectclass=nstombstone)(nsUniqueId=ffffffff-ffffffff-ffffffff-ffffffff))" | grep "nsds50ruv\|nsDS5ReplicaId" Enter LDAP Password: nsDS5ReplicaId: 66 nsds50ruv: {replicageneration} 50c1015c000000600000 nsds50ruv: {replica 66 ldap://kdc03.unix.iriszorg.nl:389} 57e23f66000000420000 nsds50ruv: {replica 1095 ldap://kdc04.unix.iriszorg.nl:389} 57e4d75a0000044700 nsds50ruv: {replica 96 ldap://kdc01.unix.iriszorg.nl:7389} 50c1016c00000060000 nsds50ruv: {replica 71 ldap://kdc03.unix.iriszorg.nl:389} 57e140c7000000470000 nsds50ruv: {replica 97 ldap://kdc02.unix.iriszorg.nl:7389} 50c1016800000061000 so I need to keep 66 and 1095, and run the task on 96, 71 and 97, it would seem. Thanks for spotting my error. -- regards, natxo -------------- next part -------------- An HTML attachment was scrubbed... URL: From th at casalogic.dk Mon Sep 26 12:53:18 2016 From: th at casalogic.dk (Troels Hansen) Date: Mon, 26 Sep 2016 14:53:18 +0200 (CEST) Subject: [Freeipa-users] SSH using putty to IPA client In-Reply-To: <20160926113018.GU6019@p.Speedport_W_724V_Typ_A_05011603_00_009> References: <1098516169.379802.1474874746409.JavaMail.zimbra@casalogic.dk> <20160926081853.GQ6019@p.Speedport_W_724V_Typ_A_05011603_00_009> <1313111984.386014.1474888309619.JavaMail.zimbra@casalogic.dk> <20160926113018.GU6019@p.Speedport_W_724V_Typ_A_05011603_00_009> Message-ID: <1606822054.387886.1474894398116.JavaMail.zimbra@casalogic.dk> ----- On Sep 26, 2016, at 1:30 PM, Sumit Bose sbose at redhat.com wrote: > > Do you see and log messages in the krb5kdc.log on the IPA server? If it > is not the firewall I would suggest to record the IP traffic of the AD > client and check what it tries to do after the AD DC send the > cross-realm TGT. > > About the DNS SRV records, did you add matching records for _udp as > well? I'm not sure if the AD client will fallback to _tcp if they are > missing or just stop? > Yes, _udp recores is created (at least now, as they actually wasn't). krb5kdc.log shows nothing, so I guess its still have no access to IPA? From natxo.asenjo at gmail.com Mon Sep 26 12:56:01 2016 From: natxo.asenjo at gmail.com (Natxo Asenjo) Date: Mon, 26 Sep 2016 14:56:01 +0200 Subject: [Freeipa-users] replicas removed, but incorrectly In-Reply-To: References: <57E90B80.1020106@redhat.com> Message-ID: On Mon, Sep 26, 2016 at 1:54 PM, Natxo Asenjo wrote: > > > > On Mon, Sep 26, 2016 at 1:50 PM, Ludwig Krispenz > wrote: > >> >> On 09/26/2016 01:36 PM, Natxo Asenjo wrote: >> >> And in my example, the replica id would be 66, 96, 71 and 97, correct? >> >> no, I don't think so. you searched 2 times the same host "-h >> kdc04.unix.iriszorg.nl". >> you need to search on kdc03 to find the current replicaid of kdc03 and >> you have to keep it. >> > > > yes, you are right :( > > $ ldapsearch -Z -h kdc03.unix.iriszorg.nl -D "cn=Directory Manager" -W > -b "o=ipaca" "(&(objectclass=nstombstone)(nsUniqueId=ffffffff-ffffffff-ffffffff-ffffffff))" > | grep "nsds50ruv\|nsDS5ReplicaId" > Enter LDAP Password: > nsDS5ReplicaId: 66 > nsds50ruv: {replicageneration} 50c1015c000000600000 > nsds50ruv: {replica 66 ldap://kdc03.unix.iriszorg.nl:389} > 57e23f66000000420000 > nsds50ruv: {replica 1095 ldap://kdc04.unix.iriszorg.nl:389} > 57e4d75a0000044700 > nsds50ruv: {replica 96 ldap://kdc01.unix.iriszorg.nl:7389} > 50c1016c00000060000 > nsds50ruv: {replica 71 ldap://kdc03.unix.iriszorg.nl:389} > 57e140c7000000470000 > nsds50ruv: {replica 97 ldap://kdc02.unix.iriszorg.nl:7389} > 50c1016800000061000 > > > so I need to keep 66 and 1095, and run the task on 96, 71 and 97, it would > seem. > > Thanks for spotting my error. > ok, so I have now run the commands against both ldap hosts (the kdc03 and the kdc04), and now I have this: # ldapsearch -Z -h kdc04.unix.iriszorg.nl -D "cn=Directory Manager" -W -b "o=ipaca" "(&(objectclass=nstombstone)(nsUniqueId=ffffffff-ffffffff-ffffffff-ffffffff))" | grep "nsds50ruv\|nsDS5ReplicaId" Enter LDAP Password: nsDS5ReplicaId: 1095 nsds50ruv: {replicageneration} 50c1015c000000600000 nsds50ruv: {replica 1095 ldap://kdc04.unix.iriszorg.nl:389} 57e4d75a0000044700 nsds50ruv: {replica 66 ldap://kdc03.unix.iriszorg.nl:389} 57e23f66000000420000 # ldapsearch -Z -h kdc03.unix.iriszorg.nl -D "cn=Directory Manager" -W -b "o=ipaca" "(&(objectclass=nstombstone)(nsUniqueId=ffffffff-ffffffff-ffffffff-ffffffff))" | grep "nsds50ruv\|nsDS5ReplicaId" Enter LDAP Password: nsDS5ReplicaId: 66 nsds50ruv: {replicageneration} 50c1015c000000600000 nsds50ruv: {replica 66 ldap://kdc03.unix.iriszorg.nl:389} 57e23f66000000420000 nsds50ruv: {replica 1095 ldap://kdc04.unix.iriszorg.nl:389} 57e4d75a0000044700 nsds50ruv: {replica 96 ldap://kdc01.unix.iriszorg.nl:7389} 50c1016c00000060000 nsds50ruv: {replica 71 ldap://kdc03.unix.iriszorg.nl:389} 57e140c7000000470000 nsds50ruv: {replica 97 ldap://kdc02.unix.iriszorg.nl:7389} 50c1016800000061000 so the command has not been successful in the kdc03. in the dirsrv errors log I see: [26/Sep/2016:14:50:54 +0200] NSMMReplicationPlugin - CleanAllRUV Task (rid 71): Not all replicas online, retrying in 640 seconds... [26/Sep/2016:14:51:00 +0200] slapi_ldap_bind - Error: could not send startTLS request: error -1 (Can't contact LDAP server) errno 107 (Transport endpoint is not connected) but those replicas are gone (decommissioned). So how can I remove them? -- regards, Natxo -- -- Groeten, natxo -------------- next part -------------- An HTML attachment was scrubbed... URL: From lkrispen at redhat.com Mon Sep 26 13:06:59 2016 From: lkrispen at redhat.com (Ludwig Krispenz) Date: Mon, 26 Sep 2016 15:06:59 +0200 Subject: [Freeipa-users] replicas removed, but incorrectly In-Reply-To: References: <57E90B80.1020106@redhat.com> Message-ID: <57E91D73.3000606@redhat.com> On 09/26/2016 02:56 PM, Natxo Asenjo wrote: > > > On Mon, Sep 26, 2016 at 1:54 PM, Natxo Asenjo > wrote: > > > > > On Mon, Sep 26, 2016 at 1:50 PM, Ludwig Krispenz > > wrote: > > > On 09/26/2016 01:36 PM, Natxo Asenjo wrote: >> And in my example, the replica id would be 66, 96, 71 and 97, correct? > no, I don't think so. you searched 2 times the same host "-h > kdc04.unix.iriszorg.nl ". > you need to search on kdc03 to find the current replicaid of > kdc03 and you have to keep it. > > > > yes, you are right :( > > $ ldapsearch -Z -h kdc03.unix.iriszorg.nl > -D "cn=Directory Manager" -W -b > "o=ipaca" > "(&(objectclass=nstombstone)(nsUniqueId=ffffffff-ffffffff-ffffffff-ffffffff))" > | grep "nsds50ruv\|nsDS5ReplicaId" > Enter LDAP Password: > nsDS5ReplicaId: 66 > nsds50ruv: {replicageneration} 50c1015c000000600000 > nsds50ruv: {replica 66 ldap://kdc03.unix.iriszorg.nl:389 > } 57e23f66000000420000 > nsds50ruv: {replica 1095 ldap://kdc04.unix.iriszorg.nl:389 > } 57e4d75a0000044700 > nsds50ruv: {replica 96 ldap://kdc01.unix.iriszorg.nl:7389 > } 50c1016c00000060000 > nsds50ruv: {replica 71 ldap://kdc03.unix.iriszorg.nl:389 > } 57e140c7000000470000 > nsds50ruv: {replica 97 ldap://kdc02.unix.iriszorg.nl:7389 > } 50c1016800000061000 > > > so I need to keep 66 and 1095, and run the task on 96, 71 and 97, > it would seem. > > Thanks for spotting my error. > > > > ok, so I have now run the commands against both ldap hosts (the kdc03 > and the kdc04), and now I have this: you need to run it only against one host, it will propagate itself to the other replicas, if it can - see below. > > # ldapsearch -Z -h kdc04.unix.iriszorg.nl > -D "cn=Directory Manager" -W -b > "o=ipaca" > "(&(objectclass=nstombstone)(nsUniqueId=ffffffff-ffffffff-ffffffff-ffffffff))" > | grep "nsds50ruv\|nsDS5ReplicaId" > Enter LDAP Password: > nsDS5ReplicaId: 1095 > nsds50ruv: {replicageneration} 50c1015c000000600000 > nsds50ruv: {replica 1095 ldap://kdc04.unix.iriszorg.nl:389 > } 57e4d75a0000044700 > nsds50ruv: {replica 66 ldap://kdc03.unix.iriszorg.nl:389 > } 57e23f66000000420000 > > # ldapsearch -Z -h kdc03.unix.iriszorg.nl > -D "cn=Directory Manager" -W -b > "o=ipaca" > "(&(objectclass=nstombstone)(nsUniqueId=ffffffff-ffffffff-ffffffff-ffffffff))" > | grep "nsds50ruv\|nsDS5ReplicaId" > Enter LDAP Password: > nsDS5ReplicaId: 66 > nsds50ruv: {replicageneration} 50c1015c000000600000 > nsds50ruv: {replica 66 ldap://kdc03.unix.iriszorg.nl:389 > } 57e23f66000000420000 > nsds50ruv: {replica 1095 ldap://kdc04.unix.iriszorg.nl:389 > } 57e4d75a0000044700 > nsds50ruv: {replica 96 ldap://kdc01.unix.iriszorg.nl:7389 > } 50c1016c00000060000 > nsds50ruv: {replica 71 ldap://kdc03.unix.iriszorg.nl:389 > } 57e140c7000000470000 > nsds50ruv: {replica 97 ldap://kdc02.unix.iriszorg.nl:7389 > } 50c1016800000061000 > > so the command has not been successful in the kdc03. in the dirsrv > errors log I see: > > [26/Sep/2016:14:50:54 +0200] NSMMReplicationPlugin - CleanAllRUV Task > (rid 71): Not all replicas online, retrying in 640 seconds... this looks like there is still a replication agreement to one of the no longer existing servers. can you search for "... -b "cn=config" "objectclass=nsds5replicationagreement" and remove the ones no longer needed. > [26/Sep/2016:14:51:00 +0200] slapi_ldap_bind - Error: could not send > startTLS request: error -1 (Can't contact LDAP server) errno 107 > (Transport endpoint is not connected) > > but those replicas are gone (decommissioned). So how can I remove them? > > > -- > regards, > Natxo > > > > > > -- > -- > Groeten, > natxo > > -- Red Hat GmbH, http://www.de.redhat.com/, Registered seat: Grasbrunn, Commercial register: Amtsgericht Muenchen, HRB 153243, Managing Directors: Charles Cachera, Michael Cunningham, Michael O'Neill, Eric Shander -------------- next part -------------- An HTML attachment was scrubbed... URL: From natxo.asenjo at gmail.com Mon Sep 26 13:32:11 2016 From: natxo.asenjo at gmail.com (Natxo Asenjo) Date: Mon, 26 Sep 2016 15:32:11 +0200 Subject: [Freeipa-users] replicas removed, but incorrectly In-Reply-To: <57E91D73.3000606@redhat.com> References: <57E90B80.1020106@redhat.com> <57E91D73.3000606@redhat.com> Message-ID: hi, On Mon, Sep 26, 2016 at 3:06 PM, Ludwig Krispenz wrote: > > On 09/26/2016 02:56 PM, Natxo Asenjo wrote: > > > so the command has not been successful in the kdc03. in the dirsrv errors > log I see: > > [26/Sep/2016:14:50:54 +0200] NSMMReplicationPlugin - CleanAllRUV Task (rid > 71): Not all replicas online, retrying in 640 seconds... > > this looks like there is still a replication agreement to one of the no > longer existing servers. > > can you search for "... -b "cn=config" "objectclass=nsds5replicationagreement" > > > and remove the ones no longer needed. > allow me to post the output of both commands as separate files I am not really sure which one I need to remove. -- Groeten, natxo -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: kdc03.log Type: text/x-csrc Size: 6416 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: kdc04.log Type: text/x-csrc Size: 4516 bytes Desc: not available URL: From natxo.asenjo at gmail.com Mon Sep 26 13:35:45 2016 From: natxo.asenjo at gmail.com (Natxo Asenjo) Date: Mon, 26 Sep 2016 15:35:45 +0200 Subject: [Freeipa-users] replicas removed, but incorrectly In-Reply-To: References: <57E90B80.1020106@redhat.com> <57E91D73.3000606@redhat.com> Message-ID: hi, or do I need to remove: dn: cn=cloneAgreement1-kdc03.unix.iriszorg.nl-pki-tomcat,cn=replica,cn=o\3Dipa ca,cn=mapping tree,cn=config because it has this: nsds5replicaLastUpdateStatus: -1 Unable to acquire replicaLDAP error: Can't co ntact LDAP server nsds5replicaUpdateInProgress: FALSE and this: dn: cn=masterAgreement1-kdc04.unix.iriszorg.nl-pki-tomcat,cn=replica,cn=o\3Dip aca,cn=mapping tree,cn=config nsds5replicaLastUpdateStatus: -1 Incremental update has failed and requires ad ministrator actionLDAP error: Can't contact LDAP server On Mon, Sep 26, 2016 at 3:32 PM, Natxo Asenjo wrote: > hi, > > > > On Mon, Sep 26, 2016 at 3:06 PM, Ludwig Krispenz > wrote: > >> >> On 09/26/2016 02:56 PM, Natxo Asenjo wrote: >> >> >> so the command has not been successful in the kdc03. in the dirsrv errors >> log I see: >> >> [26/Sep/2016:14:50:54 +0200] NSMMReplicationPlugin - CleanAllRUV Task >> (rid 71): Not all replicas online, retrying in 640 seconds... >> >> this looks like there is still a replication agreement to one of the no >> longer existing servers. >> >> can you search for "... -b "cn=config" "objectclass=nsds5replicationagreement" >> >> >> and remove the ones no longer needed. >> > > allow me to post the output of both commands as separate files > > I am not really sure which one I need to remove. > > -- > Groeten, > natxo > -- -- Groeten, natxo -------------- next part -------------- An HTML attachment was scrubbed... URL: From lkrispen at redhat.com Mon Sep 26 13:52:32 2016 From: lkrispen at redhat.com (Ludwig Krispenz) Date: Mon, 26 Sep 2016 15:52:32 +0200 Subject: [Freeipa-users] replicas removed, but incorrectly In-Reply-To: References: <57E90B80.1020106@redhat.com> <57E91D73.3000606@redhat.com> Message-ID: <57E92820.9020705@redhat.com> you should only remove agreements to no longer existing servers, eg where: nsDS5ReplicaHost: kdc01.unix.iriszorg.nl the other one should remain, not sure why it cannot contact the server On 09/26/2016 03:35 PM, Natxo Asenjo wrote: > hi, > > or do I need to remove: > > dn: > cn=cloneAgreement1-kdc03.unix.iriszorg.nl-pki-tomcat,cn=replica,cn=o\3Dipa > ca,cn=mapping tree,cn=config > > because it has this: > nsds5replicaLastUpdateStatus: -1 Unable to acquire replicaLDAP error: > Can't co > ntact LDAP server > nsds5replicaUpdateInProgress: FALSE > > and this: > > dn: > cn=masterAgreement1-kdc04.unix.iriszorg.nl-pki-tomcat,cn=replica,cn=o\3Dip > aca,cn=mapping tree,cn=config > > nsds5replicaLastUpdateStatus: -1 Incremental update has failed and > requires ad > ministrator actionLDAP error: Can't contact LDAP server > > > > On Mon, Sep 26, 2016 at 3:32 PM, Natxo Asenjo > wrote: > > hi, > > > > On Mon, Sep 26, 2016 at 3:06 PM, Ludwig Krispenz > > wrote: > > > On 09/26/2016 02:56 PM, Natxo Asenjo wrote: >> >> so the command has not been successful in the kdc03. in the >> dirsrv errors log I see: >> >> [26/Sep/2016:14:50:54 +0200] NSMMReplicationPlugin - >> CleanAllRUV Task (rid 71): Not all replicas online, retrying >> in 640 seconds... > this looks like there is still a replication agreement to one > of the no longer existing servers. > > can you search for "... -b "cn=config" > "objectclass=nsds5replicationagreement" > > and remove the ones no longer needed. > > > allow me to post the output of both commands as separate files > > I am not really sure which one I need to remove. > > -- > Groeten, > natxo > > > > > -- > -- > Groeten, > natxo > > -- Red Hat GmbH, http://www.de.redhat.com/, Registered seat: Grasbrunn, Commercial register: Amtsgericht Muenchen, HRB 153243, Managing Directors: Charles Cachera, Michael Cunningham, Michael O'Neill, Eric Shander -------------- next part -------------- An HTML attachment was scrubbed... URL: From matt at indigo.nu Mon Sep 26 14:38:50 2016 From: matt at indigo.nu (Matthew Sellers) Date: Mon, 26 Sep 2016 09:38:50 -0500 Subject: [Freeipa-users] Distributing user keytabs for non-interactive auth question In-Reply-To: <8934c824-37b3-aac8-568e-f0d45288a1a8@redhat.com> References: <23846404.1474855380692.JavaMail.wam@elwamui-rubis.atl.sa.earthlink.net> <8934c824-37b3-aac8-568e-f0d45288a1a8@redhat.com> Message-ID: Hi Martin, Thank you for clarification. In my example I am configuring 'unprivileged' service users. Specifically, I wrote a script to pull data from IPA from its wonderful REST interface that will run on a group of hosts. Since this has to run non-interactively I would like to use a keytab. I initially went down the road of grabbing an initial keytab with ipa-getkeytab and then distributing the 'service user keytab' with Puppet. I see the security implications here the same as un-encrypted ssh keys for service users. My second option was to jump on the host, grab a TGT for the service user with kinit, and download the lastest KVNO of my service user keytab using ipa-getkeytab with '-r' option. This seemed pretty cool and solved the issue of asking the KDC 'give me the lastest keytab for service user abc_service'. What is the best way to do this? If anybody can share similar deployment stories that would be great. Thank You! Matt On Mon, Sep 26, 2016 at 2:22 AM, Martin Babinsky wrote: > On 09/26/2016 04:22 AM, Matthew Sellers wrote: >> >> Hey Mike, >> >> Thanks for the reply. I did use this originally when deploying my >> 'kerberized' service on my first host. What I am trying to do is use >> ipa-getkeytab for keytab distribution on say...100 hosts, without >> having to copy around keytabs from host to host. >> >> Since using ipa-getkeytab without the '-r' option just creates a new >> keytab with bumped KVNO ..and.. when I do use '-r' I recieve a message >> for 'Insufficient access rights' I am still fuzzy.... >> >> Can ipa-getkeytab be used for mass distribution of user keytabs with >> the -r option? >> >> Thanks Again! >> Matt >> >> >> >> On Sun, Sep 25, 2016 at 9:03 PM, Michael ORourke >> wrote: >>> >>> Matt, >>> >>> Try the following... >>> >>> # Get admin TGT >>> kinit admin at REALM.COM >>> >>> # Get keytab for user account >>> ipa-getkeytab -s coipa100 -p cron_runner at REALM.COM -k >>> ipa_cron_runner.keytab >>> >>> # Clear tickets >>> kdestroy >>> >>> # Request TGT using the keytab >>> kinit -k -t ./cron_runner.keytab cron_runner at REALM.COM >>> >>> # List tickets >>> klist >>> >>> I recommend including the username somewhere in the name of the keytab >>> file itself which makes it easier to remember. Of course be careful with >>> the permissions on the keytab file, because anyone that has read access to >>> the keytab can get a TGT as that user. >>> >>> -Mike >>> >>> -----Original Message----- >>>> >>>> From: Matthew Sellers >>>> Sent: Sep 25, 2016 8:37 PM >>>> To: freeipa-users at redhat.com >>>> Subject: [Freeipa-users] Distributing user keytabs for non-interactive >>>> auth question >>>> >>>> Hi Guys, >>>> >>>> What is the best way to distribute a 'user' keytab to distribute >>>> keytabs to allow 'system users' to run scripts with non-interactive >>>> auth? Is it possible to use the ipa-getkeytab feature ( with "-r" >>>> option ) to request a keytab for a user principal? I see support for >>>> HOST and SERVICE keytabs, but nothing specific to user keytabs? >>>> >>>> Concept Example: >>>> >>>> ipa-getkeytab -s ipa_server -p cron_runner at REALM.COM -k ipa_cron.keytab >>>> -r >>>> KRB5_KTNAME=ipa_cron.keytab service.py >>>> >>>> Actual Results ( tried with tgt for cron_runner or admin ): >>>> >>>> [sysadmin at 01 ~]$ ipa-getkeytab -s coipa100 -p cron_runner at REALM.COM >>>> -kipa_cron.keytab -r >>>> Failed to parse result: Insufficient access rights >>>> >>>> My only other option is grab the keytab and copy it around after >>>> initial creation ( understanding that each keytab requests bumps the >>>> KVNO ). My goal is to make password-less authentication for automated >>>> processes as easy as possible to setup....ipa-getkeytab seems like its >>>> almost there? >>>> >>>> Love the work you guys are putting out, its a really cool system. >>>> >>>> Thanks, >>>> Matt >>>> >>>> -- >>>> Manage your subscription for the Freeipa-users mailing list: >>>> https://www.redhat.com/mailman/listinfo/freeipa-users >>>> Go to http://freeipa.org for more info on the project >>> >>> >>> -- >>> Manage your subscription for the Freeipa-users mailing list: >>> https://www.redhat.com/mailman/listinfo/freeipa-users >>> Go to http://freeipa.org for more info on the project >> >> > > The problem is that in order to retrieve an existing Kerberos keys the > getkeytab extended operation need to be able to read them. The support for > these permissions is currently implemented for hosts and services only (see > http://www.freeipa.org/page/V4/Keytab_Retrieval_Management for more > details). > > Maybe you can workaround this by retrieving keytabs as a directory manager > but then you have to enter directory manager password everywhere. > > Also there is a considerable security risk involved in storing user keytabs > e.g. in their home directories, as anyone who gains privileged access to the > enrolled machine can then impersonate any user that has a keytab stored on > it. > > -- > Martin^3 Babinsky > > > -- > Manage your subscription for the Freeipa-users mailing list: > https://www.redhat.com/mailman/listinfo/freeipa-users > Go to http://freeipa.org for more info on the project From abokovoy at redhat.com Mon Sep 26 14:59:53 2016 From: abokovoy at redhat.com (Alexander Bokovoy) Date: Mon, 26 Sep 2016 17:59:53 +0300 Subject: [Freeipa-users] Distributing user keytabs for non-interactive auth question In-Reply-To: References: <23846404.1474855380692.JavaMail.wam@elwamui-rubis.atl.sa.earthlink.net> <8934c824-37b3-aac8-568e-f0d45288a1a8@redhat.com> Message-ID: <20160926145953.5ezxd2dkgraqrnvh@redhat.com> On ma, 26 syys 2016, Matthew Sellers wrote: >Hi Martin, > >Thank you for clarification. In my example I am configuring >'unprivileged' service users. Specifically, I wrote a script to pull >data from IPA from its wonderful REST interface that will run on a >group of hosts. Since this has to run non-interactively I would like >to use a keytab. > >I initially went down the road of grabbing an initial keytab with >ipa-getkeytab and then distributing the 'service user keytab' with >Puppet. I see the security implications here the same as >un-encrypted ssh keys for service users. > >My second option was to jump on the host, grab a TGT for the service >user with kinit, and download the lastest KVNO of my service user >keytab using ipa-getkeytab with '-r' option. > >This seemed pretty cool and solved the issue of asking the KDC 'give >me the lastest keytab for service user abc_service'. What is the >best way to do this? > > >If anybody can share similar deployment stories that would be great. If you are not tied to POSIX users, you can create actual Kerberos services and use them to talk to IPA framework. To the framework there is no difference whether it is a user or a kerberos service that is authenticating. You can create roles that reference Kerberos services instead of users when assigning permissions to access certain objects and their attributes. For services there is already a good way to delegate retrieval rights for keytabs. > >Thank You! >Matt > > > >On Mon, Sep 26, 2016 at 2:22 AM, Martin Babinsky wrote: >> On 09/26/2016 04:22 AM, Matthew Sellers wrote: >>> >>> Hey Mike, >>> >>> Thanks for the reply. I did use this originally when deploying my >>> 'kerberized' service on my first host. What I am trying to do is use >>> ipa-getkeytab for keytab distribution on say...100 hosts, without >>> having to copy around keytabs from host to host. >>> >>> Since using ipa-getkeytab without the '-r' option just creates a new >>> keytab with bumped KVNO ..and.. when I do use '-r' I recieve a message >>> for 'Insufficient access rights' I am still fuzzy.... >>> >>> Can ipa-getkeytab be used for mass distribution of user keytabs with >>> the -r option? >>> >>> Thanks Again! >>> Matt >>> >>> >>> >>> On Sun, Sep 25, 2016 at 9:03 PM, Michael ORourke >>> wrote: >>>> >>>> Matt, >>>> >>>> Try the following... >>>> >>>> # Get admin TGT >>>> kinit admin at REALM.COM >>>> >>>> # Get keytab for user account >>>> ipa-getkeytab -s coipa100 -p cron_runner at REALM.COM -k >>>> ipa_cron_runner.keytab >>>> >>>> # Clear tickets >>>> kdestroy >>>> >>>> # Request TGT using the keytab >>>> kinit -k -t ./cron_runner.keytab cron_runner at REALM.COM >>>> >>>> # List tickets >>>> klist >>>> >>>> I recommend including the username somewhere in the name of the keytab >>>> file itself which makes it easier to remember. Of course be careful with >>>> the permissions on the keytab file, because anyone that has read access to >>>> the keytab can get a TGT as that user. >>>> >>>> -Mike >>>> >>>> -----Original Message----- >>>>> >>>>> From: Matthew Sellers >>>>> Sent: Sep 25, 2016 8:37 PM >>>>> To: freeipa-users at redhat.com >>>>> Subject: [Freeipa-users] Distributing user keytabs for non-interactive >>>>> auth question >>>>> >>>>> Hi Guys, >>>>> >>>>> What is the best way to distribute a 'user' keytab to distribute >>>>> keytabs to allow 'system users' to run scripts with non-interactive >>>>> auth? Is it possible to use the ipa-getkeytab feature ( with "-r" >>>>> option ) to request a keytab for a user principal? I see support for >>>>> HOST and SERVICE keytabs, but nothing specific to user keytabs? >>>>> >>>>> Concept Example: >>>>> >>>>> ipa-getkeytab -s ipa_server -p cron_runner at REALM.COM -k ipa_cron.keytab >>>>> -r >>>>> KRB5_KTNAME=ipa_cron.keytab service.py >>>>> >>>>> Actual Results ( tried with tgt for cron_runner or admin ): >>>>> >>>>> [sysadmin at 01 ~]$ ipa-getkeytab -s coipa100 -p cron_runner at REALM.COM >>>>> -kipa_cron.keytab -r >>>>> Failed to parse result: Insufficient access rights >>>>> >>>>> My only other option is grab the keytab and copy it around after >>>>> initial creation ( understanding that each keytab requests bumps the >>>>> KVNO ). My goal is to make password-less authentication for automated >>>>> processes as easy as possible to setup....ipa-getkeytab seems like its >>>>> almost there? >>>>> >>>>> Love the work you guys are putting out, its a really cool system. >>>>> >>>>> Thanks, >>>>> Matt >>>>> >>>>> -- >>>>> Manage your subscription for the Freeipa-users mailing list: >>>>> https://www.redhat.com/mailman/listinfo/freeipa-users >>>>> Go to http://freeipa.org for more info on the project >>>> >>>> >>>> -- >>>> Manage your subscription for the Freeipa-users mailing list: >>>> https://www.redhat.com/mailman/listinfo/freeipa-users >>>> Go to http://freeipa.org for more info on the project >>> >>> >> >> The problem is that in order to retrieve an existing Kerberos keys the >> getkeytab extended operation need to be able to read them. The support for >> these permissions is currently implemented for hosts and services only (see >> http://www.freeipa.org/page/V4/Keytab_Retrieval_Management for more >> details). >> >> Maybe you can workaround this by retrieving keytabs as a directory manager >> but then you have to enter directory manager password everywhere. >> >> Also there is a considerable security risk involved in storing user keytabs >> e.g. in their home directories, as anyone who gains privileged access to the >> enrolled machine can then impersonate any user that has a keytab stored on >> it. >> >> -- >> Martin^3 Babinsky >> >> >> -- >> Manage your subscription for the Freeipa-users mailing list: >> https://www.redhat.com/mailman/listinfo/freeipa-users >> Go to http://freeipa.org for more info on the project > >-- >Manage your subscription for the Freeipa-users mailing list: >https://www.redhat.com/mailman/listinfo/freeipa-users >Go to http://freeipa.org for more info on the project -- / Alexander Bokovoy From tgeier at accertify.com Mon Sep 26 19:17:01 2016 From: tgeier at accertify.com (Timothy Geier) Date: Mon, 26 Sep 2016 19:17:01 +0000 Subject: [Freeipa-users] Replication broken Message-ID: This issue started when trying to remove a user; ipa user-del showed ?operation failed? and the user was not removed. The same ipa user-del command was performed on a replica and completed successfully, but it was then immediately apparent that this change did not replicate anywhere else. All of the replicas then were re-initalized using "ipa-replica-manage re-initialize? and now the LDAP trees/users are consistent though no further changes have been made. The slapd error logs are showing repeated instances of DSRetroclPlugin - replog: an error occured while adding change number 112697, dn = changenumber=112697,cn=changelog: Already exists. retrocl-plugin - retrocl_postob: operation failure [68] Package versions are ipa-server-4.2.0-15.0.1.el7.centos.6.1.x86_64 and 389-ds-base-1.3.4.0-29.el7_2.x86_64 ipa-replica-manage list-ruv ipa: WARNING: session memcached servers not running unable to decode: {replica 11} 56044ef50000000b0000 56044ef50000000b0000 unable to decode: {replica 7} 561f17ba000800070000 561f17ba000800070000 unable to decode: {replica 5} 561f17bc000300050000 561f17bc000300050000 unable to decode: {replica 9} 561f17ba000a00090000 561f17ba000a00090000 unable to decode: {replica 4} 561f17ba000300040000 561f17ba000300040000 (These are likely leftovers from the previous incarnation of these servers on a RHEL6-like setup) ipa07:389: 16 ipa02:389: 13 ipa03:389: 14 ipa01:389: 12 ipa04:389: 15 ipa05:389: 17 Thanks much, "This message and any attachments may contain confidential information. If you have received this message in error, any use or distribution is prohibited. Please notify us by reply e-mail if you have mistakenly received this message, and immediately and permanently delete it and any attachments. Thank you." From natxo.asenjo at gmail.com Mon Sep 26 20:03:08 2016 From: natxo.asenjo at gmail.com (Natxo Asenjo) Date: Mon, 26 Sep 2016 22:03:08 +0200 Subject: [Freeipa-users] another certmonger question Message-ID: hi, after our upgrade from centos 6.8 to 7.2, when I renew a certificate using ipa-getcert resubmit -i xxxxxx the certificate is properly renewed, but the info on ipa host-show still shows the old certificate info. Is this normal? $ sudo getcert list | grep expires expires: 2018-09-27 19:46:03 UTC so that certificate has successfully been renewed, but this is the host's info: $ ipa host-show hostname | grep -i after Not After: Wed Jun 07 14:30:47 2017 UTC and I see there as well more than one certificate for that host: $ ipa cert-find --subject=hostname ---------------------- 5 certificates matched ---------------------- Serial number (hex): 0xFF90008 Serial number: 267976712 Status: VALID Subject: CN=hostname.unix.iriszorg.nl,O=UNIX.IRISZORG.NL Serial number (hex): 0xFF90009 Serial number: 267976713 Status: VALID Subject: CN=hostname.unix.iriszorg.nl,O=UNIX.IRISZORG.NL Serial number (hex): 0xFF9000A Serial number: 267976714 Status: VALID Subject: CN=hostname.unix.iriszorg.nl,O=UNIX.IRISZORG.NL Serial number (hex): 0xFFF001D Serial number: 268369949 Status: REVOKED_EXPIRED Subject: CN=hostname.unix.iriszorg.nl,O=UNIX.IRISZORG.NL Serial number (hex): 0xFFF0093 Serial number: 268370067 Status: REVOKED Subject: CN=hostname.unix.iriszorg.nl,O=UNIX.IRISZORG.NL ---------------------------- Number of entries returned 5 ---------------------------- And three of them are still valid. As a comparison, another hosts which was installed about the same time also has 5 certificates, but 4 are revoked and the expires info of getcert list and of the valid certificate are the same. So how do I correct this? -- -- Groeten, natxo -------------- next part -------------- An HTML attachment was scrubbed... URL: From tgeier at accertify.com Mon Sep 26 21:07:25 2016 From: tgeier at accertify.com (Timothy Geier) Date: Mon, 26 Sep 2016 21:07:25 +0000 Subject: [Freeipa-users] Replication broken In-Reply-To: References: Message-ID: > On Sep 26, 2016, at 2:17 PM, Timothy Geier wrote: > > This issue started when trying to remove a user; ipa user-del showed ?operation failed? and the user was not removed. The same ipa user-del command was performed on a replica and completed successfully, but it was then immediately apparent that this change did not replicate anywhere else. All of the replicas then were re-initalized using "ipa-replica-manage re-initialize? and now the LDAP trees/users are consistent though no further changes have been made. > > The slapd error logs are showing repeated instances of > > DSRetroclPlugin - replog: an error occured while adding change number 112697, dn = changenumber=112697,cn=changelog: Already exists. > retrocl-plugin - retrocl_postob: operation failure [68] > > Package versions are > ipa-server-4.2.0-15.0.1.el7.centos.6.1.x86_64 > and > 389-ds-base-1.3.4.0-29.el7_2.x86_64 > > ipa-replica-manage list-ruv > ipa: WARNING: session memcached servers not running > unable to decode: {replica 11} 56044ef50000000b0000 56044ef50000000b0000 > unable to decode: {replica 7} 561f17ba000800070000 561f17ba000800070000 > unable to decode: {replica 5} 561f17bc000300050000 561f17bc000300050000 > unable to decode: {replica 9} 561f17ba000a00090000 561f17ba000a00090000 > unable to decode: {replica 4} 561f17ba000300040000 561f17ba000300040000 > (These are likely leftovers from the previous incarnation of these servers on a RHEL6-like setup) > ipa07:389: 16 > ipa02:389: 13 > ipa03:389: 14 > ipa01:389: 12 > ipa04:389: 15 > ipa05:389: 17 > > Thanks much, After not taking any action, this error has stopped but has been replaced with [26/Sep/2016:15:54:54 -0500] NSMMReplicationPlugin - agmt="cn=meToipa03" (ipa03:389): Missing data encountered [26/Sep/2016:15:54:54 -0500] NSMMReplicationPlugin - agmt="cn=meToipa03" (ipa03:389): Incremental update failed and requires administrator action for all of the replicas and things are slightly out of sync everywhere. Is the best course of action here to declare one a new master and do a ipa-replica-manage re-initialize to all of the others from that one? "This message and any attachments may contain confidential information. If you have received this message in error, any use or distribution is prohibited. Please notify us by reply e-mail if you have mistakenly received this message, and immediately and permanently delete it and any attachments. Thank you." From tgeier at accertify.com Mon Sep 26 22:37:03 2016 From: tgeier at accertify.com (Timothy Geier) Date: Mon, 26 Sep 2016 22:37:03 +0000 Subject: [Freeipa-users] Replication broken In-Reply-To: References: Message-ID: <55D4C985-EEB3-46AA-A92F-61EFA1EA7B3F@accertify.com> On Sep 26, 2016, at 4:07 PM, Timothy Geier > wrote: On Sep 26, 2016, at 2:17 PM, Timothy Geier > wrote: This issue started when trying to remove a user; ipa user-del showed ?operation failed? and the user was not removed. The same ipa user-del command was performed on a replica and completed successfully, but it was then immediately apparent that this change did not replicate anywhere else. All of the replicas then were re-initalized using "ipa-replica-manage re-initialize? and now the LDAP trees/users are consistent though no further changes have been made. The slapd error logs are showing repeated instances of DSRetroclPlugin - replog: an error occured while adding change number 112697, dn = changenumber=112697,cn=changelog: Already exists. retrocl-plugin - retrocl_postob: operation failure [68] Package versions are ipa-server-4.2.0-15.0.1.el7.centos.6.1.x86_64 and 389-ds-base-1.3.4.0-29.el7_2.x86_64 ipa-replica-manage list-ruv ipa: WARNING: session memcached servers not running unable to decode: {replica 11} 56044ef50000000b0000 56044ef50000000b0000 unable to decode: {replica 7} 561f17ba000800070000 561f17ba000800070000 unable to decode: {replica 5} 561f17bc000300050000 561f17bc000300050000 unable to decode: {replica 9} 561f17ba000a00090000 561f17ba000a00090000 unable to decode: {replica 4} 561f17ba000300040000 561f17ba000300040000 (These are likely leftovers from the previous incarnation of these servers on a RHEL6-like setup) ipa07:389: 16 ipa02:389: 13 ipa03:389: 14 ipa01:389: 12 ipa04:389: 15 ipa05:389: 17 Thanks much, After not taking any action, this error has stopped but has been replaced with [26/Sep/2016:15:54:54 -0500] NSMMReplicationPlugin - agmt="cn=meToipa03" (ipa03:389): Missing data encountered [26/Sep/2016:15:54:54 -0500] NSMMReplicationPlugin - agmt="cn=meToipa03" (ipa03:389): Incremental update failed and requires administrator action for all of the replicas and things are slightly out of sync everywhere. Is the best course of action here to declare one a new master and do a ipa-replica-manage re-initialize to all of the others from that one? After doing some testing, that?s exactly what we did and replication is now working again. It is odd that the DSRetroclPlugin errors stopped on their own (after approximately 3 hours); the only action taken there was looking at the cn=changelog base using ldapvi to see what number it was on but that has to be a sheer coincidence; absolutely no changes were made. We?re also still unsure what caused this; our best theory at the moment is a race condition where everything that could have gone wrong at that exact moment did..is there any validity to this? Thanks, "This message and any attachments may contain confidential information. If you have received this message in error, any use or distribution is prohibited. Please notify us by reply e-mail if you have mistakenly received this message, and immediately and permanently delete it and any attachments. Thank you." -------------- next part -------------- An HTML attachment was scrubbed... URL: From matt at indigo.nu Mon Sep 26 23:01:13 2016 From: matt at indigo.nu (Matthew Sellers) Date: Mon, 26 Sep 2016 18:01:13 -0500 Subject: [Freeipa-users] Distributing user keytabs for non-interactive auth question In-Reply-To: <20160926145953.5ezxd2dkgraqrnvh@redhat.com> References: <23846404.1474855380692.JavaMail.wam@elwamui-rubis.atl.sa.earthlink.net> <8934c824-37b3-aac8-568e-f0d45288a1a8@redhat.com> <20160926145953.5ezxd2dkgraqrnvh@redhat.com> Message-ID: Hi Alexander, Thanks for the hint, I was able to create a keytab and authenticate as a service, but using this method IPA returns zero results ( u'result' : [] ) for calls like "dnszone_find()". I tried adding some of the predefined ROLES to the service principal, but no luck there. Any hints are greatly appreciated. Thank You! Matt On Mon, Sep 26, 2016 at 9:59 AM, Alexander Bokovoy wrote: > On ma, 26 syys 2016, Matthew Sellers wrote: >> >> Hi Martin, >> >> Thank you for clarification. In my example I am configuring >> 'unprivileged' service users. Specifically, I wrote a script to pull >> data from IPA from its wonderful REST interface that will run on a >> group of hosts. Since this has to run non-interactively I would like >> to use a keytab. >> >> I initially went down the road of grabbing an initial keytab with >> ipa-getkeytab and then distributing the 'service user keytab' with >> Puppet. I see the security implications here the same as >> un-encrypted ssh keys for service users. >> >> My second option was to jump on the host, grab a TGT for the service >> user with kinit, and download the lastest KVNO of my service user >> keytab using ipa-getkeytab with '-r' option. >> >> This seemed pretty cool and solved the issue of asking the KDC 'give >> me the lastest keytab for service user abc_service'. What is the >> best way to do this? >> >> >> If anybody can share similar deployment stories that would be great. > > If you are not tied to POSIX users, you can create actual Kerberos > services and use them to talk to IPA framework. To the framework there > is no difference whether it is a user or a kerberos service that is > authenticating. You can create roles that reference Kerberos services > instead of users when assigning permissions to access certain objects > and their attributes. > > For services there is already a good way to delegate retrieval rights > for keytabs. > > > >> >> Thank You! >> Matt >> >> >> >> On Mon, Sep 26, 2016 at 2:22 AM, Martin Babinsky >> wrote: >>> >>> On 09/26/2016 04:22 AM, Matthew Sellers wrote: >>>> >>>> >>>> Hey Mike, >>>> >>>> Thanks for the reply. I did use this originally when deploying my >>>> 'kerberized' service on my first host. What I am trying to do is use >>>> ipa-getkeytab for keytab distribution on say...100 hosts, without >>>> having to copy around keytabs from host to host. >>>> >>>> Since using ipa-getkeytab without the '-r' option just creates a new >>>> keytab with bumped KVNO ..and.. when I do use '-r' I recieve a message >>>> for 'Insufficient access rights' I am still fuzzy.... >>>> >>>> Can ipa-getkeytab be used for mass distribution of user keytabs with >>>> the -r option? >>>> >>>> Thanks Again! >>>> Matt >>>> >>>> >>>> >>>> On Sun, Sep 25, 2016 at 9:03 PM, Michael ORourke >>>> wrote: >>>>> >>>>> >>>>> Matt, >>>>> >>>>> Try the following... >>>>> >>>>> # Get admin TGT >>>>> kinit admin at REALM.COM >>>>> >>>>> # Get keytab for user account >>>>> ipa-getkeytab -s coipa100 -p cron_runner at REALM.COM -k >>>>> ipa_cron_runner.keytab >>>>> >>>>> # Clear tickets >>>>> kdestroy >>>>> >>>>> # Request TGT using the keytab >>>>> kinit -k -t ./cron_runner.keytab cron_runner at REALM.COM >>>>> >>>>> # List tickets >>>>> klist >>>>> >>>>> I recommend including the username somewhere in the name of the keytab >>>>> file itself which makes it easier to remember. Of course be careful >>>>> with >>>>> the permissions on the keytab file, because anyone that has read access >>>>> to >>>>> the keytab can get a TGT as that user. >>>>> >>>>> -Mike >>>>> >>>>> -----Original Message----- >>>>>> >>>>>> >>>>>> From: Matthew Sellers >>>>>> Sent: Sep 25, 2016 8:37 PM >>>>>> To: freeipa-users at redhat.com >>>>>> Subject: [Freeipa-users] Distributing user keytabs for non-interactive >>>>>> auth question >>>>>> >>>>>> Hi Guys, >>>>>> >>>>>> What is the best way to distribute a 'user' keytab to distribute >>>>>> keytabs to allow 'system users' to run scripts with non-interactive >>>>>> auth? Is it possible to use the ipa-getkeytab feature ( with "-r" >>>>>> option ) to request a keytab for a user principal? I see support for >>>>>> HOST and SERVICE keytabs, but nothing specific to user keytabs? >>>>>> >>>>>> Concept Example: >>>>>> >>>>>> ipa-getkeytab -s ipa_server -p cron_runner at REALM.COM -k >>>>>> ipa_cron.keytab >>>>>> -r >>>>>> KRB5_KTNAME=ipa_cron.keytab service.py >>>>>> >>>>>> Actual Results ( tried with tgt for cron_runner or admin ): >>>>>> >>>>>> [sysadmin at 01 ~]$ ipa-getkeytab -s coipa100 -p cron_runner at REALM.COM >>>>>> -kipa_cron.keytab -r >>>>>> Failed to parse result: Insufficient access rights >>>>>> >>>>>> My only other option is grab the keytab and copy it around after >>>>>> initial creation ( understanding that each keytab requests bumps the >>>>>> KVNO ). My goal is to make password-less authentication for automated >>>>>> processes as easy as possible to setup....ipa-getkeytab seems like its >>>>>> almost there? >>>>>> >>>>>> Love the work you guys are putting out, its a really cool system. >>>>>> >>>>>> Thanks, >>>>>> Matt >>>>>> >>>>>> -- >>>>>> Manage your subscription for the Freeipa-users mailing list: >>>>>> https://www.redhat.com/mailman/listinfo/freeipa-users >>>>>> Go to http://freeipa.org for more info on the project >>>>> >>>>> >>>>> >>>>> -- >>>>> Manage your subscription for the Freeipa-users mailing list: >>>>> https://www.redhat.com/mailman/listinfo/freeipa-users >>>>> Go to http://freeipa.org for more info on the project >>>> >>>> >>>> >>> >>> The problem is that in order to retrieve an existing Kerberos keys the >>> getkeytab extended operation need to be able to read them. The support >>> for >>> these permissions is currently implemented for hosts and services only >>> (see >>> http://www.freeipa.org/page/V4/Keytab_Retrieval_Management for more >>> details). >>> >>> Maybe you can workaround this by retrieving keytabs as a directory >>> manager >>> but then you have to enter directory manager password everywhere. >>> >>> Also there is a considerable security risk involved in storing user >>> keytabs >>> e.g. in their home directories, as anyone who gains privileged access to >>> the >>> enrolled machine can then impersonate any user that has a keytab stored >>> on >>> it. >>> >>> -- >>> Martin^3 Babinsky >>> >>> >>> -- >>> Manage your subscription for the Freeipa-users mailing list: >>> https://www.redhat.com/mailman/listinfo/freeipa-users >>> Go to http://freeipa.org for more info on the project >> >> >> -- >> Manage your subscription for the Freeipa-users mailing list: >> https://www.redhat.com/mailman/listinfo/freeipa-users >> Go to http://freeipa.org for more info on the project > > > -- > / Alexander Bokovoy From tbordaz at redhat.com Tue Sep 27 10:47:59 2016 From: tbordaz at redhat.com (thierry bordaz) Date: Tue, 27 Sep 2016 12:47:59 +0200 Subject: [Freeipa-users] Replication broken In-Reply-To: <55D4C985-EEB3-46AA-A92F-61EFA1EA7B3F@accertify.com> References: <55D4C985-EEB3-46AA-A92F-61EFA1EA7B3F@accertify.com> Message-ID: <57EA4E5F.6050009@redhat.com> Hi Timothy, The changenumber counter is protected by a lock and we should not see duplicate value.. except if there is a bug :-( Retrieving the time when changenumber=112697,cn=changelog was created and the time when you saw the error, can you see any error in operations (access log) or in the error log ? Or did you disabled/enable retorCL between those two times ? regards thiery On 09/27/2016 12:37 AM, Timothy Geier wrote: > >> On Sep 26, 2016, at 4:07 PM, Timothy Geier > > wrote: >> >>> >>> On Sep 26, 2016, at 2:17 PM, Timothy Geier >> > wrote: >>> >>> This issue started when trying to remove a user; ipa user-del showed >>> ?operation failed? and the user was not removed. The same ipa >>> user-del command was performed on a replica and completed >>> successfully, but it was then immediately apparent that this change >>> did not replicate anywhere else. All of the replicas then were >>> re-initalized using "ipa-replica-manage re-initialize? and now the >>> LDAP trees/users are consistent though no further changes have been >>> made. >>> >>> The slapd error logs are showing repeated instances of >>> >>> DSRetroclPlugin - replog: an error occured while adding change >>> number 112697, dn = changenumber=112697,cn=changelog: Already exists. >>> retrocl-plugin - retrocl_postob: operation failure [68] >>> >>> Package versions are >>> ipa-server-4.2.0-15.0.1.el7.centos.6.1.x86_64 >>> and >>> 389-ds-base-1.3.4.0-29.el7_2.x86_64 >>> >>> ipa-replica-manage list-ruv >>> ipa: WARNING: session memcached servers not running >>> unable to decode: {replica 11} 56044ef50000000b0000 56044ef50000000b0000 >>> unable to decode: {replica 7} 561f17ba000800070000 561f17ba000800070000 >>> unable to decode: {replica 5} 561f17bc000300050000 561f17bc000300050000 >>> unable to decode: {replica 9} 561f17ba000a00090000 561f17ba000a00090000 >>> unable to decode: {replica 4} 561f17ba000300040000 561f17ba000300040000 >>> (These are likely leftovers from the previous incarnation of these >>> servers on a RHEL6-like setup) >>> ipa07:389: 16 >>> ipa02:389: 13 >>> ipa03:389: 14 >>> ipa01:389: 12 >>> ipa04:389: 15 >>> ipa05:389: 17 >>> >>> Thanks much, >> >> After not taking any action, this error has stopped but has been >> replaced with >> >> [26/Sep/2016:15:54:54 -0500] NSMMReplicationPlugin - >> agmt="cn=meToipa03" (ipa03:389): Missing data encountered >> [26/Sep/2016:15:54:54 -0500] NSMMReplicationPlugin - >> agmt="cn=meToipa03" (ipa03:389): Incremental update failed and >> requires administrator action >> >> for all of the replicas and things are slightly out of sync everywhere. >> >> Is the best course of action here to declare one a new master and do >> a ipa-replica-manage re-initialize to all of the others from that one? >> >> > > After doing some testing, that?s exactly what we did and replication > is now working again. It is odd that the DSRetroclPlugin errors > stopped on their own (after approximately 3 hours); the only action > taken there was looking at the cn=changelog base using ldapvi to see > what number it was on but that has to be a sheer coincidence; > absolutely no changes were made. > > We?re also still unsure what caused this; our best theory at the > moment is a race condition where everything that could have gone wrong > at that exact moment did..is there any validity to this? > > Thanks, > "This message and any attachments may contain confidential information. If you > have received this message in error, any use or distribution is prohibited. > Please notify us by reply e-mail if you have mistakenly received this message, > and immediately and permanently delete it and any attachments. Thank you." > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From rcritten at redhat.com Tue Sep 27 11:42:15 2016 From: rcritten at redhat.com (Rob Crittenden) Date: Tue, 27 Sep 2016 13:42:15 +0200 Subject: [Freeipa-users] another certmonger question In-Reply-To: References: Message-ID: <57EA5B17.8000307@redhat.com> Natxo Asenjo wrote: > hi, > > after our upgrade from centos 6.8 to 7.2, when I renew a certificate > using ipa-getcert resubmit -i xxxxxx the certificate is properly > renewed, but the info on ipa host-show still shows the old certificate > info. Is this normal? > > $ sudo getcert list | grep expires > expires: 2018-09-27 19:46:03 UTC > > so that certificate has successfully been renewed, but this is the > host's info: > > $ ipa host-show hostname | grep -i after > Not After: Wed Jun 07 14:30:47 2017 UTC > > and I see there as well more than one certificate for that host: > > $ ipa cert-find --subject=hostname > ---------------------- > 5 certificates matched > ---------------------- > Serial number (hex): 0xFF90008 > Serial number: 267976712 > Status: VALID > Subject: CN=hostname.unix.iriszorg.nl > ,O=UNIX.IRISZORG.NL > > > Serial number (hex): 0xFF90009 > Serial number: 267976713 > Status: VALID > Subject: CN=hostname.unix.iriszorg.nl > ,O=UNIX.IRISZORG.NL > > > Serial number (hex): 0xFF9000A > Serial number: 267976714 > Status: VALID > Subject: CN=hostname.unix.iriszorg.nl > ,O=UNIX.IRISZORG.NL > > > Serial number (hex): 0xFFF001D > Serial number: 268369949 > Status: REVOKED_EXPIRED > Subject: CN=hostname.unix.iriszorg.nl > ,O=UNIX.IRISZORG.NL > > > Serial number (hex): 0xFFF0093 > Serial number: 268370067 > Status: REVOKED > Subject: CN=hostname.unix.iriszorg.nl > ,O=UNIX.IRISZORG.NL > > ---------------------------- > Number of entries returned 5 > ---------------------------- > > And three of them are still valid. As a comparison, another hosts which > was installed about the same time also has 5 certificates, but 4 are > revoked and the expires info of getcert list and of the valid > certificate are the same. > > So how do I correct this? It's hard to say, it may in fact not be a problem. It is really a matter of what service the certificate(s) are related to. I'd look at the serial numbers and then correlate those to the issued certificates. I'd also do a service-find on the hostname to see if any services have certificates issued and with what serial numbers. rob From bret.wortman at damascusgrp.com Tue Sep 27 14:20:01 2016 From: bret.wortman at damascusgrp.com (Bret Wortman) Date: Tue, 27 Sep 2016 10:20:01 -0400 Subject: [Freeipa-users] How to get a new cert Message-ID: <7b627575-bc54-438b-3d2f-dec72615c95b@damascusgrp.com> Is there a guide anywhere for how to obtain an SSL certificate for a new server & service from the IPA CA master? Most of the guides I'm seeing online use web pages at the major CAs to do this and I'd like to keep it in the family. Thanks! -- *Bret Wortman* http://wrapbuddies.co/ -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- URL signature expired From prashant at apigee.com Tue Sep 27 15:16:06 2016 From: prashant at apigee.com (Prashant Bapat) Date: Tue, 27 Sep 2016 20:46:06 +0530 Subject: [Freeipa-users] RBAC - User Administrator - OTP tokens Message-ID: RBAC Role "User Administrator" should have access to all users OTP tokens. Specifically to remove if some one has lost their token. We get this a lot. I found no permissions that give this access. Can someone explain if this can be added easily either from the WebUI or CLI. Thanks. --Prashant -------------- next part -------------- An HTML attachment was scrubbed... URL: From flo at redhat.com Tue Sep 27 15:28:04 2016 From: flo at redhat.com (Florence Blanc-Renaud) Date: Tue, 27 Sep 2016 17:28:04 +0200 Subject: [Freeipa-users] How to get a new cert In-Reply-To: <7b627575-bc54-438b-3d2f-dec72615c95b@damascusgrp.com> References: <7b627575-bc54-438b-3d2f-dec72615c95b@damascusgrp.com> Message-ID: Hi Bret, would the following be helpful? In "Linux Domain Identity, Authentication, and Policy Guide", Chapter 17.1.1 Requesting New Certificates for a User, Host, or Service [1] Flo. [1] https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/7/html/Linux_Domain_Identity_Authentication_and_Policy_Guide/certificates.html#certificate-request On 09/27/2016 04:20 PM, Bret Wortman wrote: > Is there a guide anywhere for how to obtain an SSL certificate for a new > server & service from the IPA CA master? Most of the guides I'm seeing > online use web pages at the major CAs to do this and I'd like to keep it > in the family. > > Thanks! > > > -- > *Bret Wortman* > > http://wrapbuddies.co/ > > From tgeier at accertify.com Tue Sep 27 15:28:32 2016 From: tgeier at accertify.com (Timothy Geier) Date: Tue, 27 Sep 2016 15:28:32 +0000 Subject: [Freeipa-users] Replication broken In-Reply-To: <57EA4E5F.6050009@redhat.com> References: <55D4C985-EEB3-46AA-A92F-61EFA1EA7B3F@accertify.com> <57EA4E5F.6050009@redhat.com> Message-ID: <1474990112.3794.8.camel@accertify.com> On Tue, 2016-09-27 at 12:47 +0200, thierry bordaz wrote: > Hi Timothy, > > The changenumber counter is protected by a lock and we should not see > duplicate value.. except if there is a bug :-( > > Retrieving the time when changenumber=112697,cn=changelog was created > and the time when you saw the error, can you see any error in > operations (access log) or in the error log ? > > Or did you disabled/enable retorCL between those two times ? > > regards > thiery Unfortunately, the issue appears to be a certain username that starts with a '1'..in both cases, trying to delete this user caused (and is causing) the exact same issue. Are there any known bugs relating to this? > > > > On 09/27/2016 12:37 AM, Timothy Geier wrote: > > > > > > On Sep 26, 2016, at 4:07 PM, Timothy Geier > > > wrote: > > > > > > > > > > > On Sep 26, 2016, at 2:17 PM, Timothy Geier > > > > wrote: > > > > > > > > This issue started when trying to remove a user; ipa user-del > > > > showed ?operation failed? and the user was not removed. The > > > > same ipa user-del command was performed on a replica and > > > > completed successfully, but it was then immediately apparent > > > > that this change did not replicate anywhere else. All of the > > > > replicas then were re-initalized using "ipa-replica-manage > > > > re-initialize? and now the LDAP trees/users are consistent > > > > though no further changes have been made. > > > > > > > > The slapd error logs are showing repeated instances of > > > > > > > > DSRetroclPlugin - replog: an error occured while adding change > > > > number 112697, dn = changenumber=112697,cn=changelog: Already > > > > exists. > > > > retrocl-plugin - retrocl_postob: operation failure [68] > > > > > > > > Package versions are > > > > ipa-server-4.2.0-15.0.1.el7.centos.6.1.x86_64 > > > > and > > > > 389-ds-base-1.3.4.0-29.el7_2.x86_64 > > > > > > > > ipa-replica-manage list-ruv > > > > ipa: WARNING: session memcached servers not running > > > > unable to decode: {replica 11} 56044ef50000000b0000 > > > > 56044ef50000000b0000 > > > > unable to decode: {replica 7} 561f17ba000800070000 > > > > 561f17ba000800070000 > > > > unable to decode: {replica 5} 561f17bc000300050000 > > > > 561f17bc000300050000 > > > > unable to decode: {replica 9} 561f17ba000a00090000 > > > > 561f17ba000a00090000 > > > > unable to decode: {replica 4} 561f17ba000300040000 > > > > 561f17ba000300040000 > > > > (These are likely leftovers from the previous incarnation of > > > > these servers on a RHEL6-like setup) > > > > ipa07:389: 16 > > > > ipa02:389: 13 > > > > ipa03:389: 14 > > > > ipa01:389: 12 > > > > ipa04:389: 15 > > > > ipa05:389: 17 > > > > > > > > Thanks much, > > > > > > After not taking any action, this error has stopped but has been > > > replaced with > > > > > > [26/Sep/2016:15:54:54 -0500] NSMMReplicationPlugin - > > > agmt="cn=meToipa03" (ipa03:389): Missing data encountered > > > [26/Sep/2016:15:54:54 -0500] NSMMReplicationPlugin - > > > agmt="cn=meToipa03" (ipa03:389): Incremental update failed and > > > requires administrator action > > > > > > for all of the replicas and things are slightly out of sync > > > everywhere. > > > > > > Is the best course of action here to declare one a new master and > > > do a ipa-replica-manage re-initialize to all of the others from > > > that one? > > > > > > > > > > > > > > > After doing some testing, that?s exactly what we did and replication > > is now working again. It is odd that the DSRetroclPlugin errors > > stopped on their own (after approximately 3 hours); the only action > > taken there was looking at the cn=changelog base using ldapvi to see > > what number it was on but that has to be a sheer coincidence; > > absolutely no changes were made. > > > > > > We?re also still unsure what caused this; our best theory at the > > moment is a race condition where everything that could have gone > > wrong at that exact moment did..is there any validity to this? > > > > > > Thanks, > > "This message and any attachments may contain confidential information. If you > > have received this message in error, any use or distribution is prohibited. > > Please notify us by reply e-mail if you have mistakenly received this message, > > and immediately and permanently delete it and any attachments. Thank you." > > > > > -- Timothy R. Geier Sr. Linux Systems Administrator Accertify, Inc. an American Express Company 2 Pierce Place Suite 900 Itasca, IL 60143 Office: + 1 (630) 735-4785 tgeier at accertify.com "This message and any attachments may contain confidential information. If you have received this message in error, any use or distribution is prohibited. Please notify us by reply e-mail if you have mistakenly received this message, and immediately and permanently delete it and any attachments. Thank you." From piolet.y at gmail.com Tue Sep 27 16:04:37 2016 From: piolet.y at gmail.com (Youenn PIOLET) Date: Tue, 27 Sep 2016 18:04:37 +0200 Subject: [Freeipa-users] Server replication stopped working In-Reply-To: <57E8D14C.4070203@redhat.com> References: <7854DA02-1936-448E-B4F7-791289F1F1A0@oracle.com> <57E8D14C.4070203@redhat.com> Message-ID: Hi Ludwig, Version: 389-ds-base-1.3.4.0-33.el7_2.x86_64 The timestamp probably matches the last time I've done a ipa-replica-manage re-initialize. I have to do it every day (many times a day actually!), as replication is broken, This CSN changes all the time. My main goal is to rebuilt everything from a clean base. I've got no master without errors. What is the easiest way to rebuilt everything? ipa-[cs]replica-manage re-initialize isn't very effective. Thanks by advance, Regards -- Youenn Piolet piolet.y at gmail.com 2016-09-26 9:42 GMT+02:00 Ludwig Krispenz : > > On 09/25/2016 09:35 PM, Youenn PIOLET wrote: > > Hi there, > > Same issue for me in a my 15 ipa-servers multi-master grid just after the > update. > The replication is completely broken except on 3/15 nodes. > > This is the second time I have to fully reinitialize the whole cluster for > similar reason. I don't know what to do to clean this mess... > For more information: this cluster has been initialized on a fedora 4.1.4 > more than one year ago then complemetely migrated to Centos 7, IPA 4.2. > > what is the exact version of 389-ds-base you are running ? > > did these errors come out of the blue or are they related to some > activities ? The csn which is not found has a timestamp of "Thu, 22 Sep > 2016 15:59:08 GMT" did anything happen around this time ? > > > Example on fr-master03 error logs: > > [25/Sep/2016:19:27:31 +0000] NSMMReplicationPlugin - changelog program - > agmt="cn=meTofr-master01.domain" (fr-master01:389): CSN > 57e3ffcc0003001a0000 not found, we aren't as up to date, or we purged > [25/Sep/2016:19:27:31 +0000] NSMMReplicationPlugin - > agmt="cn=meTofr-master01.domain" (fr-master01:389): Data required to > update replica has been purged. The replica must be reinitialized. > [25/Sep/2016:19:27:31 +0000] NSMMReplicationPlugin - > agmt="cn=meTofr-master01.domain" (fr-master01:389): Incremental update > failed and requires administrator action > ipa: INFO: The ipactl command was successful > [25/Sep/2016:19:27:35 +0000] agmt="cn=meTofr-master02.domain" > (fr-master02:389) - Can't locate CSN 57e3ffcc0003001a0000 in the changelog > (DB rc=-30988). If replication stops, the consumer may need to be > reinitialized. > [25/Sep/2016:19:27:35 +0000] NSMMReplicationPlugin - changelog program - > agmt="cn=meTofr-master02.domain" (fr-master02:389): CSN > 57e3ffcc0003001a0000 not found, we aren't as up to date, or we purged > [25/Sep/2016:19:27:35 +0000] NSMMReplicationPlugin - > agmt="cn=meTofr-master02.domain" (fr-master02:389): Data required to > update replica has been purged. The replica must be reinitialized. > [25/Sep/2016:19:27:35 +0000] NSMMReplicationPlugin - > agmt="cn=meTofr-master02.domain" (fr-master02:389): Incremental update > failed and requires administrator action > > Regards, > > -- > Youenn Piolet > piolet.y at gmail.com > > > 2016-09-23 17:51 GMT+02:00 Mike Driscoll : > >> Hello. I have four IPA servers replicating in full mesh. All four >> servers are running ipa-server-4.2.0-15.0.1.el7_2.19.x86_64. >> >> This was working for some time but now I see that no replication is >> occurring automatically at present. >> >> When I update a user attribute on an IPA server, I see errors like these: >> [22/Sep/2016:16:53:49 -0700] attrlist_replace - attr_replace >> (nsslapd-referral, ldap://ldap03.xx.com:389/o%3Dipaca) failed. >> [22/Sep/2016:16:58:56 -0700] NSMMReplicationPlugin - agmt="cn= >> masterAgreement1-ldap03.xx.com >> -pki-tomcat" (ldap03:389): Incremental update failed and requires >> administrator action >> >> I can reinitialize without errors. >> ipa-csreplica-manage re-initialize --from=ldap01.xx.com >> >> ipa-replica-manage re-initialize --from=ldap01.xx.com >> Afterwards I see my attribute (and other) changes are replicated on each >> server I re-initialize from. But subsequently, replication doesn?t seem to >> be happening. >> >> I reinitialized according to the steps in Table 8.7, ?Replication >> Errors?, but subsequent replication isn?t occurring. Any suggestions? Is >> it safe to identify one of my four servers as containing up-to-date data, >> then sever and reinstate replication relationships with the other three? >> >> Mike >> >> >> >> >> >> >> -- >> Manage your subscription for the Freeipa-users mailing list: >> https://www.redhat.com/mailman/listinfo/freeipa-users >> Go to http://freeipa.org for more info on the project >> > > > > > -- > Red Hat GmbH, http://www.de.redhat.com/, Registered seat: Grasbrunn, > Commercial register: Amtsgericht Muenchen, HRB 153243, > Managing Directors: Charles Cachera, Michael Cunningham, Michael O'Neill, Eric Shander > > > -- > Manage your subscription for the Freeipa-users mailing list: > https://www.redhat.com/mailman/listinfo/freeipa-users > Go to http://freeipa.org for more info on the project > -------------- next part -------------- An HTML attachment was scrubbed... URL: From lkrispen at redhat.com Tue Sep 27 16:34:44 2016 From: lkrispen at redhat.com (Ludwig Krispenz) Date: Tue, 27 Sep 2016 18:34:44 +0200 Subject: [Freeipa-users] Server replication stopped working In-Reply-To: References: <7854DA02-1936-448E-B4F7-791289F1F1A0@oracle.com> <57E8D14C.4070203@redhat.com> Message-ID: <57EA9FA4.4010403@redhat.com> On 09/27/2016 06:04 PM, Youenn PIOLET wrote: > Hi Ludwig, > > Version: > 389-ds-base-1.3.4.0-33.el7_2.x86_64 we have identified an issue with this version, it includes a fix for 389-ds ticket #48766, which was incomplete and resolved shortly after the release of this version (it is missing the latest patch for #49766 and for #48954). You can try to go back to 1.3.4.0-32 or if you have support get a hotfix from our support. Sorry for this, Ludwig > > The timestamp probably matches the last time I've done a > ipa-replica-manage re-initialize. > I have to do it every day (many times a day actually!), as replication > is broken, This CSN changes all the time. > > My main goal is to rebuilt everything from a clean base. > I've got no master without errors. > > What is the easiest way to rebuilt everything? > ipa-[cs]replica-manage re-initialize isn't very effective. > > Thanks by advance, > Regards > > -- > Youenn Piolet > piolet.y at gmail.com > / > / > > 2016-09-26 9:42 GMT+02:00 Ludwig Krispenz >: > > > On 09/25/2016 09:35 PM, Youenn PIOLET wrote: >> Hi there, >> >> Same issue for me in a my 15 ipa-servers multi-master grid just >> after the update. >> The replication is completely broken except on 3/15 nodes. >> >> This is the second time I have to fully reinitialize the whole >> cluster for similar reason. I don't know what to do to clean this >> mess... >> For more information: this cluster has been initialized on a >> fedora 4.1.4 more than one year ago then complemetely migrated to >> Centos 7, IPA 4.2. > what is the exact version of 389-ds-base you are running ? > > did these errors come out of the blue or are they related to some > activities ? The csn which is not found has a timestamp of "Thu, > 22 Sep 2016 15:59:08 GMT" did anything happen around this time ? > >> >> Example on fr-master03 error logs: >> >> [25/Sep/2016:19:27:31 +0000] NSMMReplicationPlugin - changelog >> program - agmt="cn=meTofr-master01.domain" (fr-master01:389): CSN >> 57e3ffcc0003001a0000 not found, we aren't as up to date, or we purged >> [25/Sep/2016:19:27:31 +0000] NSMMReplicationPlugin - >> agmt="cn=meTofr-master01.domain" (fr-master01:389): Data required >> to update replica has been purged. The replica must be reinitialized. >> [25/Sep/2016:19:27:31 +0000] NSMMReplicationPlugin - >> agmt="cn=meTofr-master01.domain" (fr-master01:389): Incremental >> update failed and requires administrator action >> ipa: INFO: The ipactl command was successful >> [25/Sep/2016:19:27:35 +0000] agmt="cn=meTofr-master02.domain" >> (fr-master02:389) - Can't locate CSN 57e3ffcc0003001a0000 in the >> changelog (DB rc=-30988). If replication stops, the consumer may >> need to be reinitialized. >> [25/Sep/2016:19:27:35 +0000] NSMMReplicationPlugin - changelog >> program - agmt="cn=meTofr-master02.domain" (fr-master02:389): CSN >> 57e3ffcc0003001a0000 not found, we aren't as up to date, or we purged >> [25/Sep/2016:19:27:35 +0000] NSMMReplicationPlugin - >> agmt="cn=meTofr-master02.domain" (fr-master02:389): Data required >> to update replica has been purged. The replica must be reinitialized. >> [25/Sep/2016:19:27:35 +0000] NSMMReplicationPlugin - >> agmt="cn=meTofr-master02.domain" (fr-master02:389): Incremental >> update failed and requires administrator action >> >> Regards, >> >> -- >> Youenn Piolet >> piolet.y at gmail.com >> / >> / >> >> 2016-09-23 17:51 GMT+02:00 Mike Driscoll >> >: >> >> Hello. I have four IPA servers replicating in full mesh. >> All four servers are running >> ipa-server-4.2.0-15.0.1.el7_2.19.x86_64. >> >> This was working for some time but now I see that no >> replication is occurring automatically at present. >> >> When I update a user attribute on an IPA server, I see errors >> like these: >> [22/Sep/2016:16:53:49 -0700] attrlist_replace - attr_replace >> (nsslapd-referral, ldap://ldap03.xx.com:389/o%3Dipaca) failed. >> [22/Sep/2016:16:58:56 -0700] NSMMReplicationPlugin - >> agmt="cn=masterAgreement1-ldap03.xx.com >> -pki-tomcat" (ldap03:389): >> Incremental update failed and requires administrator action >> >> I can reinitialize without errors. >> ipa-csreplica-manage re-initialize --from=ldap01.xx.com >> >> ipa-replica-manage re-initialize --from=ldap01.xx.com >> >> Afterwards I see my attribute (and other) changes are >> replicated on each server I re-initialize from. But >> subsequently, replication doesn?t seem to be happening. >> >> I reinitialized according to the steps in Table 8.7, >> ?Replication Errors?, but subsequent replication isn?t >> occurring. Any suggestions? Is it safe to identify one of my >> four servers as containing up-to-date data, then sever and >> reinstate replication relationships with the other three? >> >> Mike >> >> >> >> >> >> >> -- >> Manage your subscription for the Freeipa-users mailing list: >> https://www.redhat.com/mailman/listinfo/freeipa-users >> >> Go to http://freeipa.org for more info on the project >> >> >> >> > > -- > Red Hat GmbH,http://www.de.redhat.com/, Registered seat: Grasbrunn, > Commercial register: Amtsgericht Muenchen, HRB 153243, > Managing Directors: Charles Cachera, Michael Cunningham, Michael O'Neill, Eric Shander > > > -- > Manage your subscription for the Freeipa-users mailing list: > https://www.redhat.com/mailman/listinfo/freeipa-users > > Go to http://freeipa.org for more info on the project > > -- Red Hat GmbH, http://www.de.redhat.com/, Registered seat: Grasbrunn, Commercial register: Amtsgericht Muenchen, HRB 153243, Managing Directors: Charles Cachera, Michael Cunningham, Michael O'Neill, Eric Shander -------------- next part -------------- An HTML attachment was scrubbed... URL: From tbordaz at redhat.com Tue Sep 27 16:38:58 2016 From: tbordaz at redhat.com (thierry bordaz) Date: Tue, 27 Sep 2016 18:38:58 +0200 Subject: [Freeipa-users] Replication broken In-Reply-To: <1474990112.3794.8.camel@accertify.com> References: <55D4C985-EEB3-46AA-A92F-61EFA1EA7B3F@accertify.com> <57EA4E5F.6050009@redhat.com> <1474990112.3794.8.camel@accertify.com> Message-ID: <57EAA0A2.7030602@redhat.com> Hi Timothy, When you say username do you mean 'User login' (uid) ? I can create such entry (with heading '1') on 4.2 and after. I see no reason why the uid value ('1' is valid in uid syntax) would trigger that failure when adding an entry 'cn=changelog'. I think something wrong happened to the retroCL counter and hoping to see what in the error log. Is it possible you send me the error log between the time changenumber=112697 was created and when DSRetroclPlugin reports a failure ? thanks thierry On 09/27/2016 05:28 PM, Timothy Geier wrote: > On Tue, 2016-09-27 at 12:47 +0200, thierry bordaz wrote: >> Hi Timothy, >> >> The changenumber counter is protected by a lock and we should not see >> duplicate value.. except if there is a bug :-( >> >> Retrieving the time when changenumber=112697,cn=changelog was created >> and the time when you saw the error, can you see any error in >> operations (access log) or in the error log ? >> >> Or did you disabled/enable retorCL between those two times ? >> >> regards >> thiery > Unfortunately, the issue appears to be a certain username that starts > with a '1'..in both cases, trying to delete this user caused (and is > causing) the exact same issue. Are there any known bugs relating to > this? > >> >> >> On 09/27/2016 12:37 AM, Timothy Geier wrote: >> >>>> On Sep 26, 2016, at 4:07 PM, Timothy Geier >>>> wrote: >>>> >>>>> On Sep 26, 2016, at 2:17 PM, Timothy Geier >>>>> wrote: >>>>> >>>>> This issue started when trying to remove a user; ipa user-del >>>>> showed ?operation failed? and the user was not removed. The >>>>> same ipa user-del command was performed on a replica and >>>>> completed successfully, but it was then immediately apparent >>>>> that this change did not replicate anywhere else. All of the >>>>> replicas then were re-initalized using "ipa-replica-manage >>>>> re-initialize? and now the LDAP trees/users are consistent >>>>> though no further changes have been made. >>>>> >>>>> The slapd error logs are showing repeated instances of >>>>> >>>>> DSRetroclPlugin - replog: an error occured while adding change >>>>> number 112697, dn = changenumber=112697,cn=changelog: Already >>>>> exists. >>>>> retrocl-plugin - retrocl_postob: operation failure [68] >>>>> >>>>> Package versions are >>>>> ipa-server-4.2.0-15.0.1.el7.centos.6.1.x86_64 >>>>> and >>>>> 389-ds-base-1.3.4.0-29.el7_2.x86_64 >>>>> >>>>> ipa-replica-manage list-ruv >>>>> ipa: WARNING: session memcached servers not running >>>>> unable to decode: {replica 11} 56044ef50000000b0000 >>>>> 56044ef50000000b0000 >>>>> unable to decode: {replica 7} 561f17ba000800070000 >>>>> 561f17ba000800070000 >>>>> unable to decode: {replica 5} 561f17bc000300050000 >>>>> 561f17bc000300050000 >>>>> unable to decode: {replica 9} 561f17ba000a00090000 >>>>> 561f17ba000a00090000 >>>>> unable to decode: {replica 4} 561f17ba000300040000 >>>>> 561f17ba000300040000 >>>>> (These are likely leftovers from the previous incarnation of >>>>> these servers on a RHEL6-like setup) >>>>> ipa07:389: 16 >>>>> ipa02:389: 13 >>>>> ipa03:389: 14 >>>>> ipa01:389: 12 >>>>> ipa04:389: 15 >>>>> ipa05:389: 17 >>>>> >>>>> Thanks much, >>>> After not taking any action, this error has stopped but has been >>>> replaced with >>>> >>>> [26/Sep/2016:15:54:54 -0500] NSMMReplicationPlugin - >>>> agmt="cn=meToipa03" (ipa03:389): Missing data encountered >>>> [26/Sep/2016:15:54:54 -0500] NSMMReplicationPlugin - >>>> agmt="cn=meToipa03" (ipa03:389): Incremental update failed and >>>> requires administrator action >>>> >>>> for all of the replicas and things are slightly out of sync >>>> everywhere. >>>> >>>> Is the best course of action here to declare one a new master and >>>> do a ipa-replica-manage re-initialize to all of the others from >>>> that one? >>>> >>>> >>>> >>> >>> After doing some testing, that?s exactly what we did and replication >>> is now working again. It is odd that the DSRetroclPlugin errors >>> stopped on their own (after approximately 3 hours); the only action >>> taken there was looking at the cn=changelog base using ldapvi to see >>> what number it was on but that has to be a sheer coincidence; >>> absolutely no changes were made. >>> >>> >>> We?re also still unsure what caused this; our best theory at the >>> moment is a race condition where everything that could have gone >>> wrong at that exact moment did..is there any validity to this? >>> >>> >>> Thanks, >>> "This message and any attachments may contain confidential information. If you >>> have received this message in error, any use or distribution is prohibited. >>> Please notify us by reply e-mail if you have mistakenly received this message, >>> and immediately and permanently delete it and any attachments. Thank you." >>> >>> From matt at indigo.nu Tue Sep 27 17:03:41 2016 From: matt at indigo.nu (Matthew Sellers) Date: Tue, 27 Sep 2016 12:03:41 -0500 Subject: [Freeipa-users] Distributing user keytabs for non-interactive auth question In-Reply-To: References: <23846404.1474855380692.JavaMail.wam@elwamui-rubis.atl.sa.earthlink.net> <8934c824-37b3-aac8-568e-f0d45288a1a8@redhat.com> <20160926145953.5ezxd2dkgraqrnvh@redhat.com> Message-ID: Hi Guys, I found docs on roles and was able to permission my IPA DNS scripts as follows: ipa role-add "DNS Admin" ipa role-add-privilege "DNS Admin" --privileges="DNS Administrator" ipa role-add-member "DNS Admin" --services="buildslave/host.domain.com at REALM.COM Now I am able to authenticate with this service principal and access LDAP attributes from IPA as you suggested. I was a little caught off-guard as it seems even the built-in 'admin type' roles do not have access to read DNS ? Adding this role puts a smile on my face. ipa-getkeytab -s myipahost -p buildslave/host.domain.com at REALM.COM -k /etc/named/ipa_automation.keytab -r export KRB5_CLIENT_KTNAME=ipa_automation.keytab export KRB5_KTNAME=ipa_automation.keytab ./buildslave.py Thanks, Matt On Mon, Sep 26, 2016 at 6:01 PM, Matthew Sellers wrote: > Hi Alexander, > > Thanks for the hint, I was able to create a keytab and authenticate as > a service, but using this method IPA returns zero results ( u'result' > : [] ) for calls like "dnszone_find()". I tried adding some of the > predefined ROLES to the service principal, but no luck there. Any > hints are greatly appreciated. > > Thank You! > Matt > > > > On Mon, Sep 26, 2016 at 9:59 AM, Alexander Bokovoy wrote: >> On ma, 26 syys 2016, Matthew Sellers wrote: >>> >>> Hi Martin, >>> >>> Thank you for clarification. In my example I am configuring >>> 'unprivileged' service users. Specifically, I wrote a script to pull >>> data from IPA from its wonderful REST interface that will run on a >>> group of hosts. Since this has to run non-interactively I would like >>> to use a keytab. >>> >>> I initially went down the road of grabbing an initial keytab with >>> ipa-getkeytab and then distributing the 'service user keytab' with >>> Puppet. I see the security implications here the same as >>> un-encrypted ssh keys for service users. >>> >>> My second option was to jump on the host, grab a TGT for the service >>> user with kinit, and download the lastest KVNO of my service user >>> keytab using ipa-getkeytab with '-r' option. >>> >>> This seemed pretty cool and solved the issue of asking the KDC 'give >>> me the lastest keytab for service user abc_service'. What is the >>> best way to do this? >>> >>> >>> If anybody can share similar deployment stories that would be great. >> >> If you are not tied to POSIX users, you can create actual Kerberos >> services and use them to talk to IPA framework. To the framework there >> is no difference whether it is a user or a kerberos service that is >> authenticating. You can create roles that reference Kerberos services >> instead of users when assigning permissions to access certain objects >> and their attributes. >> >> For services there is already a good way to delegate retrieval rights >> for keytabs. >> >> >> >>> >>> Thank You! >>> Matt >>> >>> >>> >>> On Mon, Sep 26, 2016 at 2:22 AM, Martin Babinsky >>> wrote: >>>> >>>> On 09/26/2016 04:22 AM, Matthew Sellers wrote: >>>>> >>>>> >>>>> Hey Mike, >>>>> >>>>> Thanks for the reply. I did use this originally when deploying my >>>>> 'kerberized' service on my first host. What I am trying to do is use >>>>> ipa-getkeytab for keytab distribution on say...100 hosts, without >>>>> having to copy around keytabs from host to host. >>>>> >>>>> Since using ipa-getkeytab without the '-r' option just creates a new >>>>> keytab with bumped KVNO ..and.. when I do use '-r' I recieve a message >>>>> for 'Insufficient access rights' I am still fuzzy.... >>>>> >>>>> Can ipa-getkeytab be used for mass distribution of user keytabs with >>>>> the -r option? >>>>> >>>>> Thanks Again! >>>>> Matt >>>>> >>>>> >>>>> >>>>> On Sun, Sep 25, 2016 at 9:03 PM, Michael ORourke >>>>> wrote: >>>>>> >>>>>> >>>>>> Matt, >>>>>> >>>>>> Try the following... >>>>>> >>>>>> # Get admin TGT >>>>>> kinit admin at REALM.COM >>>>>> >>>>>> # Get keytab for user account >>>>>> ipa-getkeytab -s coipa100 -p cron_runner at REALM.COM -k >>>>>> ipa_cron_runner.keytab >>>>>> >>>>>> # Clear tickets >>>>>> kdestroy >>>>>> >>>>>> # Request TGT using the keytab >>>>>> kinit -k -t ./cron_runner.keytab cron_runner at REALM.COM >>>>>> >>>>>> # List tickets >>>>>> klist >>>>>> >>>>>> I recommend including the username somewhere in the name of the keytab >>>>>> file itself which makes it easier to remember. Of course be careful >>>>>> with >>>>>> the permissions on the keytab file, because anyone that has read access >>>>>> to >>>>>> the keytab can get a TGT as that user. >>>>>> >>>>>> -Mike >>>>>> >>>>>> -----Original Message----- >>>>>>> >>>>>>> >>>>>>> From: Matthew Sellers >>>>>>> Sent: Sep 25, 2016 8:37 PM >>>>>>> To: freeipa-users at redhat.com >>>>>>> Subject: [Freeipa-users] Distributing user keytabs for non-interactive >>>>>>> auth question >>>>>>> >>>>>>> Hi Guys, >>>>>>> >>>>>>> What is the best way to distribute a 'user' keytab to distribute >>>>>>> keytabs to allow 'system users' to run scripts with non-interactive >>>>>>> auth? Is it possible to use the ipa-getkeytab feature ( with "-r" >>>>>>> option ) to request a keytab for a user principal? I see support for >>>>>>> HOST and SERVICE keytabs, but nothing specific to user keytabs? >>>>>>> >>>>>>> Concept Example: >>>>>>> >>>>>>> ipa-getkeytab -s ipa_server -p cron_runner at REALM.COM -k >>>>>>> ipa_cron.keytab >>>>>>> -r >>>>>>> KRB5_KTNAME=ipa_cron.keytab service.py >>>>>>> >>>>>>> Actual Results ( tried with tgt for cron_runner or admin ): >>>>>>> >>>>>>> [sysadmin at 01 ~]$ ipa-getkeytab -s coipa100 -p cron_runner at REALM.COM >>>>>>> -kipa_cron.keytab -r >>>>>>> Failed to parse result: Insufficient access rights >>>>>>> >>>>>>> My only other option is grab the keytab and copy it around after >>>>>>> initial creation ( understanding that each keytab requests bumps the >>>>>>> KVNO ). My goal is to make password-less authentication for automated >>>>>>> processes as easy as possible to setup....ipa-getkeytab seems like its >>>>>>> almost there? >>>>>>> >>>>>>> Love the work you guys are putting out, its a really cool system. >>>>>>> >>>>>>> Thanks, >>>>>>> Matt >>>>>>> >>>>>>> -- >>>>>>> Manage your subscription for the Freeipa-users mailing list: >>>>>>> https://www.redhat.com/mailman/listinfo/freeipa-users >>>>>>> Go to http://freeipa.org for more info on the project >>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> Manage your subscription for the Freeipa-users mailing list: >>>>>> https://www.redhat.com/mailman/listinfo/freeipa-users >>>>>> Go to http://freeipa.org for more info on the project >>>>> >>>>> >>>>> >>>> >>>> The problem is that in order to retrieve an existing Kerberos keys the >>>> getkeytab extended operation need to be able to read them. The support >>>> for >>>> these permissions is currently implemented for hosts and services only >>>> (see >>>> http://www.freeipa.org/page/V4/Keytab_Retrieval_Management for more >>>> details). >>>> >>>> Maybe you can workaround this by retrieving keytabs as a directory >>>> manager >>>> but then you have to enter directory manager password everywhere. >>>> >>>> Also there is a considerable security risk involved in storing user >>>> keytabs >>>> e.g. in their home directories, as anyone who gains privileged access to >>>> the >>>> enrolled machine can then impersonate any user that has a keytab stored >>>> on >>>> it. >>>> >>>> -- >>>> Martin^3 Babinsky >>>> >>>> >>>> -- >>>> Manage your subscription for the Freeipa-users mailing list: >>>> https://www.redhat.com/mailman/listinfo/freeipa-users >>>> Go to http://freeipa.org for more info on the project >>> >>> >>> -- >>> Manage your subscription for the Freeipa-users mailing list: >>> https://www.redhat.com/mailman/listinfo/freeipa-users >>> Go to http://freeipa.org for more info on the project >> >> >> -- >> / Alexander Bokovoy From bret.wortman at damascusgrp.com Tue Sep 27 18:00:19 2016 From: bret.wortman at damascusgrp.com (Bret Wortman) Date: Tue, 27 Sep 2016 14:00:19 -0400 Subject: [Freeipa-users] How to get a new cert In-Reply-To: References: <7b627575-bc54-438b-3d2f-dec72615c95b@damascusgrp.com> Message-ID: <3b75947f-b104-659f-c785-8e8a34b6ed54@damascusgrp.com> That looks like it worked, but I have a follow-on question: I need to provide my RabbitMQ instance with a cacert file, a cert, and a key file. These seem to be .pem files. Is there an easy way to gather these 3 files from a typical IPA client node? Merci! Bret On 09/27/2016 11:28 AM, Florence Blanc-Renaud wrote: > Hi Bret, > > would the following be helpful? In "Linux Domain Identity, > Authentication, and Policy Guide", Chapter 17.1.1 Requesting New > Certificates for a User, Host, or Service [1] > > Flo. > > [1] > https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/7/html/Linux_Domain_Identity_Authentication_and_Policy_Guide/certificates.html#certificate-request > > On 09/27/2016 04:20 PM, Bret Wortman wrote: >> Is there a guide anywhere for how to obtain an SSL certificate for a new >> server & service from the IPA CA master? Most of the guides I'm seeing >> online use web pages at the major CAs to do this and I'd like to keep it >> in the family. >> >> Thanks! >> >> >> -- >> *Bret Wortman* >> >> http://wrapbuddies.co/ >> >> > From as23151767 at gmail.com Tue Sep 27 20:13:01 2016 From: as23151767 at gmail.com (Mike K) Date: Tue, 27 Sep 2016 16:13:01 -0400 Subject: [Freeipa-users] Possibly revoked my CA? Message-ID: We have several IPA servers, recently they got out of sync and in the course of fixing things, I think we inadvertently revoked the CA. When I try to get to ipa01 (the first one we built) in Firefox I get this error: An error occurred during a connection to ipa01-reston.xco.qq. Peer's Certificate has been revoked. Error code: SEC_ERROR_REVOKED_CERTIFICATE I can login to 02 & 03 just fine. But when I try to administer anything certificate related under the GUI I get this error: IPA Error 4301: CertificateOperationError Certificate operation cannot be completed: Unable to communicate with CMS (Internal Server Error) === 2016-09-23T18:53:54Z 7241 MainThread ipa INFO Deleting schedule 2358-2359 0 from agreement cn=meToipa01,cn=replica,cn=dc\=xxx\,dc\=xx,cn=mapping tree,cn=config 2016-09-23T18:53:55Z 7241 MainThread ipa INFO Replication Update in progress: FALSE: status: -1 Incremental update has failed and requires administrator actionLDAP error: Can't contact LDAP server: start: 0: end: 0 2016-09-27T18:23:10Z 30695 MainThread ipa INFO Getting ldap service principals for conversion: (krbprincipalname=ldap/ipa01-xxx at XXX.XX) and (krbprincipalname=ldap/ipa04.xxx.xx at XXX.XX) I'm thinking the cert is only revoked on 01, it should be replicated to 02-09. Is there any way to make sure that it doesn't fully replicate revokation and bring it back to 01? If that's even the problem! Thanks much, Mike -------------- next part -------------- An HTML attachment was scrubbed... URL: From jrichard at placeiq.com Tue Sep 27 22:39:36 2016 From: jrichard at placeiq.com (Jim Richard) Date: Tue, 27 Sep 2016 18:39:36 -0400 Subject: [Freeipa-users] Replica created with expired certs Message-ID: <5EE38BB7-ED9B-4001-8F64-DB7CD2319C02@placeiq.com> I have a master with apparently correct, non expired certs but when I create a new replica master I end up with expired certs. How is this possible, why and of course, how do I fix? first set is the original master and the second is the certs I get on the new replica [root at sso-110:(NYM) nssdb]$ getcert list Number of certificates and requests being tracked: 8. Request ID '20140923213643': status: MONITORING stuck: no key pair storage: type=NSSDB,location='/etc/dirsrv/slapd-PKI-IPA',nickname='Server-Cert',token='NSS Certificate DB',pinfile='/etc/dirsrv/slapd-PKI-IPA/pwdfile .txt' certificate: type=NSSDB,location='/etc/dirsrv/slapd-PKI-IPA',nickname='Server-Cert',token='NSS Certificate DB' CA: IPA issuer: CN=Certificate Authority,O=PLACEIQ.NET subject: CN=sso-110.nym1.placeiq.net,O=PLACEIQ.NET expires: 2018-08-28 10:36:04 UTC key usage: digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment eku: id-kp-serverAuth,id-kp-clientAuth pre-save command: post-save command: /usr/lib64/ipa/certmonger/restart_dirsrv PKI-IPA track: yes auto-renew: yes Request ID '20140923213732': status: MONITORING stuck: no key pair storage: type=NSSDB,location='/var/lib/pki-ca/alias',nickname='Server-Cert cert-pki-ca',token='NSS Certificate DB',pin set certificate: type=NSSDB,location='/var/lib/pki-ca/alias',nickname='Server-Cert cert-pki-ca',token='NSS Certificate DB' CA: dogtag-ipa-renew-agent issuer: CN=Certificate Authority,O=PLACEIQ.NET subject: CN=sso-110.nym1.placeiq.net,O=PLACEIQ.NET expires: 2018-08-06 10:36:02 UTC key usage: digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment eku: id-kp-serverAuth,id-kp-clientAuth pre-save command: post-save command: track: yes auto-renew: yes Request ID '20140923213814': status: MONITORING stuck: no key pair storage: type=NSSDB,location='/etc/dirsrv/slapd-PLACEIQ-NET',nickname='Server-Cert',token='NSS Certificate DB',pinfile='/etc/dirsrv/slapd-PLACEIQ-NET /pwdfile.txt' certificate: type=NSSDB,location='/etc/dirsrv/slapd-PLACEIQ-NET',nickname='Server-Cert',token='NSS Certificate DB' CA: IPA issuer: CN=Certificate Authority,O=PLACEIQ.NET subject: CN=sso-110.nym1.placeiq.net,O=PLACEIQ.NET expires: 2018-08-28 10:36:04 UTC key usage: digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment pre-save command: post-save command: /usr/lib64/ipa/certmonger/restart_dirsrv PLACEIQ-NET track: yes auto-renew: yes Request ID '20140923213856': status: MONITORING stuck: no key pair storage: type=NSSDB,location='/etc/httpd/alias',nickname='Server-Cert',token='NSS Certificate DB',pinfile='/etc/httpd/alias/pwdfile.txt' certificate: type=NSSDB,location='/etc/httpd/alias',nickname='Server-Cert',token='NSS Certificate DB' CA: IPA issuer: CN=Certificate Authority,O=PLACEIQ.NET subject: CN=sso-110.nym1.placeiq.net,O=PLACEIQ.NET expires: 2018-08-28 10:36:04 UTC key usage: digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment eku: id-kp-serverAuth,id-kp-clientAuth pre-save command: post-save command: /usr/lib64/ipa/certmonger/restart_httpd track: yes auto-renew: yes Request ID '20160119021025': status: MONITORING stuck: no key pair storage: type=NSSDB,location='/var/lib/pki-ca/alias',nickname='auditSigningCert cert-pki-ca',token='NSS Certificate DB',pin set certificate: type=NSSDB,location='/var/lib/pki-ca/alias',nickname='auditSigningCert cert-pki-ca',token='NSS Certificate DB' CA: dogtag-ipa-renew-agent issuer: CN=Certificate Authority,O=PLACEIQ.NET subject: CN=CA Audit,O=PLACEIQ.NET expires: 2017-10-26 04:38:19 UTC key usage: digitalSignature,nonRepudiation pre-save command: /usr/lib64/ipa/certmonger/stop_pkicad post-save command: /usr/lib64/ipa/certmonger/renew_ca_cert "auditSigningCert cert-pki-ca" track: yes auto-renew: yes Request ID '20160119021038': status: MONITORING stuck: no key pair storage: type=NSSDB,location='/var/lib/pki-ca/alias',nickname='ocspSigningCert cert-pki-ca',token='NSS Certificate DB',pin set certificate: type=NSSDB,location='/var/lib/pki-ca/alias',nickname='ocspSigningCert cert-pki-ca',token='NSS Certificate DB' CA: dogtag-ipa-renew-agent issuer: CN=Certificate Authority,O=PLACEIQ.NET subject: CN=OCSP Subsystem,O=PLACEIQ.NET expires: 2017-10-26 04:37:19 UTC eku: id-kp-OCSPSigning pre-save command: /usr/lib64/ipa/certmonger/stop_pkicad post-save command: /usr/lib64/ipa/certmonger/renew_ca_cert "ocspSigningCert cert-pki-ca" track: yes auto-renew: yes Request ID '20160119021055': status: MONITORING stuck: no key pair storage: type=NSSDB,location='/var/lib/pki-ca/alias',nickname='subsystemCert cert-pki-ca',token='NSS Certificate DB',pin set certificate: type=NSSDB,location='/var/lib/pki-ca/alias',nickname='subsystemCert cert-pki-ca',token='NSS Certificate DB' CA: dogtag-ipa-renew-agent issuer: CN=Certificate Authority,O=PLACEIQ.NET subject: CN=CA Subsystem,O=PLACEIQ.NET expires: 2017-10-26 04:37:19 UTC key usage: digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment eku: id-kp-serverAuth,id-kp-clientAuth pre-save command: /usr/lib64/ipa/certmonger/stop_pkicad post-save command: /usr/lib64/ipa/certmonger/renew_ca_cert "subsystemCert cert-pki-ca" track: yes auto-renew: yes Request ID '20160119021104': status: MONITORING stuck: no key pair storage: type=NSSDB,location='/etc/httpd/alias',nickname='ipaCert',token='NSS Certificate DB',pinfile='/etc/httpd/alias/pwdfile.txt' certificate: type=NSSDB,location='/etc/httpd/alias',nickname='ipaCert',token='NSS Certificate DB' CA: dogtag-ipa-renew-agent issuer: CN=Certificate Authority,O=PLACEIQ.NET subject: CN=IPA RA,O=PLACEIQ.NET expires: 2017-10-26 04:37:19 UTC key usage: digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment eku: id-kp-serverAuth,id-kp-clientAuth pre-save command: post-save command: /usr/lib64/ipa/certmonger/renew_ra_cert track: yes auto-renew: yes The new replica: [root at sso-108:(NYM) ~]$ getcert list Number of certificates and requests being tracked: 8. Request ID '20160927191253': status: MONITORING stuck: no key pair storage: type=NSSDB,location='/etc/dirsrv/slapd-PKI-IPA',nickname='Server-Cert',token='NSS Certificate DB',pinfile='/etc/dirsrv/slapd-PKI-IPA/pwdfile .txt' certificate: type=NSSDB,location='/etc/dirsrv/slapd-PKI-IPA',nickname='Server-Cert',token='NSS Certificate DB' CA: IPA issuer: CN=Certificate Authority,O=PLACEIQ.NET subject: CN=sso-108.nym1.placeiq.net,O=PLACEIQ.NET expires: 2018-09-28 19:10:33 UTC key usage: digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment eku: id-kp-serverAuth,id-kp-clientAuth pre-save command: post-save command: /usr/lib64/ipa/certmonger/restart_dirsrv PKI-IPA track: yes auto-renew: yes Request ID '20160927191452': status: CA_WORKING stuck: no key pair storage: type=NSSDB,location='/var/lib/pki-ca/alias',nickname='auditSigningCert cert-pki-ca',token='NSS Certificate DB',pin set certificate: type=NSSDB,location='/var/lib/pki-ca/alias',nickname='auditSigningCert cert-pki-ca',token='NSS Certificate DB' CA: dogtag-ipa-retrieve-agent-submit issuer: CN=Certificate Authority,O=PLACEIQ.NET subject: CN=CA Audit,O=PLACEIQ.NET expires: 2015-12-03 21:57:56 UTC key usage: digitalSignature,nonRepudiation pre-save command: /usr/lib64/ipa/certmonger/stop_pkicad post-save command: /usr/lib64/ipa/certmonger/restart_pkicad "auditSigningCert cert-pki-ca" track: yes auto-renew: yes Request ID '20160927191453': status: CA_WORKING stuck: no key pair storage: type=NSSDB,location='/var/lib/pki-ca/alias',nickname='ocspSigningCert cert-pki-ca',token='NSS Certificate DB',pin set certificate: type=NSSDB,location='/var/lib/pki-ca/alias',nickname='ocspSigningCert cert-pki-ca',token='NSS Certificate DB' CA: dogtag-ipa-retrieve-agent-submit issuer: CN=Certificate Authority,O=PLACEIQ.NET subject: CN=OCSP Subsystem,O=PLACEIQ.NET expires: 2015-12-03 21:57:56 UTC key usage: digitalSignature,nonRepudiation,keyCertSign,cRLSign eku: id-kp-OCSPSigning pre-save command: /usr/lib64/ipa/certmonger/stop_pkicad post-save command: /usr/lib64/ipa/certmonger/restart_pkicad "ocspSigningCert cert-pki-ca" track: yes auto-renew: yes Request ID '20160927191454': status: CA_WORKING stuck: no key pair storage: type=NSSDB,location='/var/lib/pki-ca/alias',nickname='subsystemCert cert-pki-ca',token='NSS Certificate DB',pin set certificate: type=NSSDB,location='/var/lib/pki-ca/alias',nickname='subsystemCert cert-pki-ca',token='NSS Certificate DB' CA: dogtag-ipa-retrieve-agent-submit issuer: CN=Certificate Authority,O=PLACEIQ.NET subject: CN=CA Subsystem,O=PLACEIQ.NET expires: 2015-12-03 21:57:56 UTC key usage: digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment eku: id-kp-serverAuth,id-kp-clientAuth pre-save command: /usr/lib64/ipa/certmonger/stop_pkicad post-save command: /usr/lib64/ipa/certmonger/restart_pkicad "subsystemCert cert-pki-ca" track: yes auto-renew: yes Request ID '20160927191455': status: MONITORING stuck: no key pair storage: type=NSSDB,location='/var/lib/pki-ca/alias',nickname='Server-Cert cert-pki-ca',token='NSS Certificate DB',pin set certificate: type=NSSDB,location='/var/lib/pki-ca/alias',nickname='Server-Cert cert-pki-ca',token='NSS Certificate DB' CA: dogtag-ipa-renew-agent issuer: CN=Certificate Authority,O=PLACEIQ.NET subject: CN=sso-108.nym1.placeiq.net,O=PLACEIQ.NET expires: 2018-09-17 19:14:36 UTC key usage: digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment eku: id-kp-serverAuth pre-save command: post-save command: track: yes auto-renew: yes Request ID '20160927191540': status: MONITORING stuck: no key pair storage: type=NSSDB,location='/etc/dirsrv/slapd-PLACEIQ-NET',nickname='Server-Cert',token='NSS Certificate DB',pinfile='/etc/dirsrv/slapd-PLACEIQ-NET /pwdfile.txt' certificate: type=NSSDB,location='/etc/dirsrv/slapd-PLACEIQ-NET',nickname='Server-Cert',token='NSS Certificate DB' CA: IPA issuer: CN=Certificate Authority,O=PLACEIQ.NET subject: CN=sso-108.nym1.placeiq.net,O=PLACEIQ.NET expires: 2018-09-28 19:10:32 UTC key usage: digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment eku: id-kp-serverAuth,id-kp-clientAuth pre-save command: post-save command: /usr/lib64/ipa/certmonger/restart_dirsrv PLACEIQ-NET track: yes auto-renew: yes Request ID '20160927192114': status: MONITORING stuck: no key pair storage: type=NSSDB,location='/etc/httpd/alias',nickname='Server-Cert',token='NSS Certificate DB',pinfile='/etc/httpd/alias/pwdfile.txt' certificate: type=NSSDB,location='/etc/httpd/alias',nickname='Server-Cert',token='NSS Certificate DB' CA: IPA issuer: CN=Certificate Authority,O=PLACEIQ.NET subject: CN=sso-108.nym1.placeiq.net,O=PLACEIQ.NET expires: 2018-09-28 19:10:34 UTC key usage: digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment eku: id-kp-serverAuth,id-kp-clientAuth pre-save command: post-save command: /usr/lib64/ipa/certmonger/restart_httpd track: yes auto-renew: yes Request ID '20160927192146': status: MONITORING stuck: no key pair storage: type=NSSDB,location='/etc/httpd/alias',nickname='ipaCert',token='NSS Certificate DB',pinfile='/etc/httpd/alias/pwdfile.txt' certificate: type=NSSDB,location='/etc/httpd/alias',nickname='ipaCert',token='NSS Certificate DB' CA: dogtag-ipa-retrieve-agent-submit issuer: CN=Certificate Authority,O=PLACEIQ.NET subject: CN=IPA RA,O=PLACEIQ.NET expires: 2017-10-26 04:37:19 UTC key usage: digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment eku: id-kp-serverAuth,id-kp-clientAuth pre-save command: post-save command: /usr/lib64/ipa/certmonger/restart_httpd track: yes auto-renew: yes Jim Richard SYSTEM ADMINISTRATOR III (646) 338-8905 -------------- next part -------------- An HTML attachment was scrubbed... URL: From jrichard at placeiq.com Tue Sep 27 22:42:38 2016 From: jrichard at placeiq.com (Jim Richard) Date: Tue, 27 Sep 2016 18:42:38 -0400 Subject: [Freeipa-users] Certificate format error reported by GUI Message-ID: <14E35142-15BB-4108-BFC9-A7BB84CB60CF@placeiq.com> When I try to look at hosts under the hosts tab. ipactl restart or just restarting httpd seems to clear it up for a short period. Three replicas in the environment, it only happens when I look at hosts using the GUI at one of the three replicas. Certificate format error: (SEC_ERROR_LEGACY_DATABASE) The certificate/key database is in an old, unsupported format. Jim Richard SYSTEM ADMINISTRATOR III (646) 338-8905 -------------- next part -------------- An HTML attachment was scrubbed... URL: From th at casalogic.dk Wed Sep 28 07:19:37 2016 From: th at casalogic.dk (Troels Hansen) Date: Wed, 28 Sep 2016 09:19:37 +0200 (CEST) Subject: [Freeipa-users] SSH using putty to IPA client In-Reply-To: <20160926113018.GU6019@p.Speedport_W_724V_Typ_A_05011603_00_009> References: <1098516169.379802.1474874746409.JavaMail.zimbra@casalogic.dk> <20160926081853.GQ6019@p.Speedport_W_724V_Typ_A_05011603_00_009> <1313111984.386014.1474888309619.JavaMail.zimbra@casalogic.dk> <20160926113018.GU6019@p.Speedport_W_724V_Typ_A_05011603_00_009> Message-ID: <820303837.407114.1475047177588.JavaMail.zimbra@casalogic.dk> ----- On Sep 26, 2016, at 1:30 PM, Sumit Bose sbose at redhat.com wrote: > About the DNS SRV records, did you add matching records for _udp as > well? I'm not sure if the AD client will fallback to _tcp if they are > missing or just stop? > Ok, finally got some time to debug this. tcpdump'ing in the IPA server and logging in, and analyzing the traffic in wireshark I can see that some KRB5KDC_ERR_PREAUTH_REQUIRED traffic to both of the KDC's as expected, followed by some AS-REQ and AS-REP, finally followed by KRB5KRB_ERR-RESPONSE_TOO_BIG, source MAC is a Cisco router despite the server being HP, so somewhere in the network a Cisco router is breaking our Kerberos. I'll start hunting a solution somewhere else but IPA...... From sbose at redhat.com Wed Sep 28 08:06:20 2016 From: sbose at redhat.com (Sumit Bose) Date: Wed, 28 Sep 2016 10:06:20 +0200 Subject: [Freeipa-users] SSH using putty to IPA client In-Reply-To: <820303837.407114.1475047177588.JavaMail.zimbra@casalogic.dk> References: <1098516169.379802.1474874746409.JavaMail.zimbra@casalogic.dk> <20160926081853.GQ6019@p.Speedport_W_724V_Typ_A_05011603_00_009> <1313111984.386014.1474888309619.JavaMail.zimbra@casalogic.dk> <20160926113018.GU6019@p.Speedport_W_724V_Typ_A_05011603_00_009> <820303837.407114.1475047177588.JavaMail.zimbra@casalogic.dk> Message-ID: <20160928080620.GY6019@p.Speedport_W_724V_Typ_A_05011603_00_009> On Wed, Sep 28, 2016 at 09:19:37AM +0200, Troels Hansen wrote: > > > ----- On Sep 26, 2016, at 1:30 PM, Sumit Bose sbose at redhat.com wrote: > > > About the DNS SRV records, did you add matching records for _udp as > > well? I'm not sure if the AD client will fallback to _tcp if they are > > missing or just stop? > > > > > Ok, finally got some time to debug this. > > tcpdump'ing in the IPA server and logging in, and analyzing the traffic in wireshark I can see that some KRB5KDC_ERR_PREAUTH_REQUIRED traffic to both of the KDC's as expected, followed by some AS-REQ and AS-REP, finally followed by KRB5KRB_ERR-RESPONSE_TOO_BIG, source MAC is a Cisco router despite the server being HP, so somewhere in the network a Cisco router is breaking our Kerberos. KRB5KRB_ERR-RESPONSE_TOO_BIG is an expected return code here. The Kerberos communication is typically started via UDP. But the PAC data in the ticket is typically larger than a single UPD packet. The KDC tells the client wit KRB5KRB_ERR-RESPONSE_TOO_BIG to switch to tcp so that the response can be reliably send in multiple tcp packets. If KRB5KRB_ERR-RESPONSE_TOO_BIG is the last you see on the wire I would suspect that port 88 tcp is blocked somewhere. HTH bye, Sumit > > I'll start hunting a solution somewhere else but IPA...... From th at casalogic.dk Wed Sep 28 08:33:43 2016 From: th at casalogic.dk (Troels Hansen) Date: Wed, 28 Sep 2016 10:33:43 +0200 (CEST) Subject: [Freeipa-users] SSH using putty to IPA client In-Reply-To: <20160928080620.GY6019@p.Speedport_W_724V_Typ_A_05011603_00_009> References: <1098516169.379802.1474874746409.JavaMail.zimbra@casalogic.dk> <20160926081853.GQ6019@p.Speedport_W_724V_Typ_A_05011603_00_009> <1313111984.386014.1474888309619.JavaMail.zimbra@casalogic.dk> <20160926113018.GU6019@p.Speedport_W_724V_Typ_A_05011603_00_009> <820303837.407114.1475047177588.JavaMail.zimbra@casalogic.dk> <20160928080620.GY6019@p.Speedport_W_724V_Typ_A_05011603_00_009> Message-ID: <585899542.408376.1475051623551.JavaMail.zimbra@casalogic.dk> ----- On Sep 28, 2016, at 10:06 AM, Sumit Bose sbose at redhat.com wrote: > KRB5KRB_ERR-RESPONSE_TOO_BIG is an expected return code here. The > Kerberos communication is typically started via UDP. But the PAC data in > the ticket is typically larger than a single UPD packet. The KDC tells > the client wit KRB5KRB_ERR-RESPONSE_TOO_BIG to switch to tcp so that the > response can be reliably send in multiple tcp packets. If > KRB5KRB_ERR-RESPONSE_TOO_BIG is the last you see on the wire I would > suspect that port 88 tcp is blocked somewhere. > Yes, you are absolutely correct. We actually switch to TCP after the initial try on UDP. I can see that we send a TGS-REQ over TCP to the AD for the current domain (NET), and AD answers back with a TGS-REP where I can see "KerberosString" tor the root domain (PLACE), and we then ads the DC for PLACE, with a TGS-REQ and get a TGS-REP with KerberosString for the IPA domain. So, actually kerberos traffic seems to be OK.... From sbose at redhat.com Wed Sep 28 08:43:31 2016 From: sbose at redhat.com (Sumit Bose) Date: Wed, 28 Sep 2016 10:43:31 +0200 Subject: [Freeipa-users] SSH using putty to IPA client In-Reply-To: <585899542.408376.1475051623551.JavaMail.zimbra@casalogic.dk> References: <1098516169.379802.1474874746409.JavaMail.zimbra@casalogic.dk> <20160926081853.GQ6019@p.Speedport_W_724V_Typ_A_05011603_00_009> <1313111984.386014.1474888309619.JavaMail.zimbra@casalogic.dk> <20160926113018.GU6019@p.Speedport_W_724V_Typ_A_05011603_00_009> <820303837.407114.1475047177588.JavaMail.zimbra@casalogic.dk> <20160928080620.GY6019@p.Speedport_W_724V_Typ_A_05011603_00_009> <585899542.408376.1475051623551.JavaMail.zimbra@casalogic.dk> Message-ID: <20160928084331.GA24101@p.Speedport_W_724V_Typ_A_05011603_00_009> On Wed, Sep 28, 2016 at 10:33:43AM +0200, Troels Hansen wrote: > > > ----- On Sep 28, 2016, at 10:06 AM, Sumit Bose sbose at redhat.com wrote: > > KRB5KRB_ERR-RESPONSE_TOO_BIG is an expected return code here. The > > Kerberos communication is typically started via UDP. But the PAC data in > > the ticket is typically larger than a single UPD packet. The KDC tells > > the client wit KRB5KRB_ERR-RESPONSE_TOO_BIG to switch to tcp so that the > > response can be reliably send in multiple tcp packets. If > > KRB5KRB_ERR-RESPONSE_TOO_BIG is the last you see on the wire I would > > suspect that port 88 tcp is blocked somewhere. > > > > > Yes, you are absolutely correct. We actually switch to TCP after the initial try on UDP. > > I can see that we send a TGS-REQ over TCP to the AD for the current domain (NET), and AD answers back with a TGS-REP where I can see "KerberosString" tor the root domain (PLACE), and we then ads the DC for PLACE, with a TGS-REQ and get a TGS-REP with KerberosString for the IPA domain. > > So, actually kerberos traffic seems to be OK.... Yes, this makes sense as well. If you are not in the forest root you first need a cross-realm TGT for your domain and the forest root. Then you need a cross-realm TGT for the forest root and the IPA domain. As a next step you should see a request to the IPA KDC to get the actual service ticket for the host in the IPA domain. bye, Sumit From th at casalogic.dk Wed Sep 28 09:30:56 2016 From: th at casalogic.dk (Troels Hansen) Date: Wed, 28 Sep 2016 11:30:56 +0200 (CEST) Subject: [Freeipa-users] SSH using putty to IPA client In-Reply-To: <20160928084331.GA24101@p.Speedport_W_724V_Typ_A_05011603_00_009> References: <1098516169.379802.1474874746409.JavaMail.zimbra@casalogic.dk> <20160926081853.GQ6019@p.Speedport_W_724V_Typ_A_05011603_00_009> <1313111984.386014.1474888309619.JavaMail.zimbra@casalogic.dk> <20160926113018.GU6019@p.Speedport_W_724V_Typ_A_05011603_00_009> <820303837.407114.1475047177588.JavaMail.zimbra@casalogic.dk> <20160928080620.GY6019@p.Speedport_W_724V_Typ_A_05011603_00_009> <585899542.408376.1475051623551.JavaMail.zimbra@casalogic.dk> <20160928084331.GA24101@p.Speedport_W_724V_Typ_A_05011603_00_009> Message-ID: <734257841.409436.1475055056872.JavaMail.zimbra@casalogic.dk> > Yes, this makes sense as well. If you are not in the forest root you > first need a cross-realm TGT for your domain and the forest root. Then > you need a cross-realm TGT for the forest root and the IPA domain. > > As a next step you should see a request to the IPA KDC to get the actual > service ticket for the host in the IPA domain. Yes, this is the traffic that's never seen in the capture. It seems Windows(Putty) never asks for at host ticket for the IPA host. I receive the krbtgt for the IPA domain, but never sees any traffic from the Windows client to IPA, and thus, never receives the host ticket on the Windows client. I'm not at all sure how Kerberos works in Putty, but it seems it uses its own Kerberos libraryes and that these fail. I Linux not joined to IPA, just installed with kerberos and use dns config in krb5.conf can kinit in the NET domain, and ssh to IPA using kerberos just fine, so it seems the problem just relates to putty. From sbose at redhat.com Wed Sep 28 09:48:02 2016 From: sbose at redhat.com (Sumit Bose) Date: Wed, 28 Sep 2016 11:48:02 +0200 Subject: [Freeipa-users] SSH using putty to IPA client In-Reply-To: <734257841.409436.1475055056872.JavaMail.zimbra@casalogic.dk> References: <1098516169.379802.1474874746409.JavaMail.zimbra@casalogic.dk> <20160926081853.GQ6019@p.Speedport_W_724V_Typ_A_05011603_00_009> <1313111984.386014.1474888309619.JavaMail.zimbra@casalogic.dk> <20160926113018.GU6019@p.Speedport_W_724V_Typ_A_05011603_00_009> <820303837.407114.1475047177588.JavaMail.zimbra@casalogic.dk> <20160928080620.GY6019@p.Speedport_W_724V_Typ_A_05011603_00_009> <585899542.408376.1475051623551.JavaMail.zimbra@casalogic.dk> <20160928084331.GA24101@p.Speedport_W_724V_Typ_A_05011603_00_009> <734257841.409436.1475055056872.JavaMail.zimbra@casalogic.dk> Message-ID: <20160928094802.GB24352@p.Speedport_W_724V_Typ_A_05011603_00_009> On Wed, Sep 28, 2016 at 11:30:56AM +0200, Troels Hansen wrote: > > > Yes, this makes sense as well. If you are not in the forest root you > > first need a cross-realm TGT for your domain and the forest root. Then > > you need a cross-realm TGT for the forest root and the IPA domain. > > > > As a next step you should see a request to the IPA KDC to get the actual > > service ticket for the host in the IPA domain. > > Yes, this is the traffic that's never seen in the capture. > It seems Windows(Putty) never asks for at host ticket for the IPA host. I receive the krbtgt for the IPA domain, but never sees any traffic from the Windows client to IPA, and thus, never receives the host ticket on the Windows client. Please check the other traffic on the client after receiving the cross-realm ticket for the IPA domain. Since the client get the name to the IPA realm from the AD DC in the last response I would expect that it will try some DNS SRV lookups to find a KDC in the IPA realm. HTH bye, Sumit > > I'm not at all sure how Kerberos works in Putty, but it seems it uses its own Kerberos libraryes and that these fail. > > I Linux not joined to IPA, just installed with kerberos and use dns config in krb5.conf can kinit in the NET domain, and ssh to IPA using kerberos just fine, so it seems the problem just relates to putty. From rcritten at redhat.com Wed Sep 28 11:44:35 2016 From: rcritten at redhat.com (Rob Crittenden) Date: Wed, 28 Sep 2016 13:44:35 +0200 Subject: [Freeipa-users] Replica created with expired certs In-Reply-To: <5EE38BB7-ED9B-4001-8F64-DB7CD2319C02@placeiq.com> References: <5EE38BB7-ED9B-4001-8F64-DB7CD2319C02@placeiq.com> Message-ID: <57EBAD23.4090604@redhat.com> Jim Richard wrote: > I have a master with apparently correct, non expired certs but when I > create a new replica master I end up with expired certs. > How is this possible, why and of course, how do I fix? I assume you are running IPA v3.0.0? The problem is that the root CA stash isn't updated when a replica file is prepared in that version (fixed in 3.3 IIRC). You can do this manually with something like: # PKCS12Export -d /var/lib/pki-ca/alias -p /root/dbpass -w /root/dmpass -o /root/cacert.p12 where /root/dmpass is a file that contains the Directory Manager password. Then rerun ipa-replica-prepare and things should work. You can look at the certs in /root/cacert.p12 util pk12util to see the change. rob > > first set is the original master and the second is the certs I get on > the new replica > > [root at sso-110:(NYM) nssdb]$ getcert list > Number of certificates and requests being tracked: 8. > Request ID '20140923213643': > status: MONITORING > stuck: no > key pair storage: > type=NSSDB,location='/etc/dirsrv/slapd-PKI-IPA',nickname='Server-Cert',token='NSS > Certificate DB',pinfile='/etc/dirsrv/slapd-PKI-IPA/pwdfile > .txt' > certificate: > type=NSSDB,location='/etc/dirsrv/slapd-PKI-IPA',nickname='Server-Cert',token='NSS > Certificate DB' > CA: IPA > issuer: CN=Certificate Authority,O=PLACEIQ.NET > subject: CN=sso-110.nym1.placeiq.net > ,O=PLACEIQ.NET > expires: 2018-08-28 10:36:04 UTC > key usage: > digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment > eku: id-kp-serverAuth,id-kp-clientAuth > pre-save command: > post-save command: /usr/lib64/ipa/certmonger/restart_dirsrv PKI-IPA > track: yes > auto-renew: yes > Request ID '20140923213732': > status: MONITORING > stuck: no > key pair storage: > type=NSSDB,location='/var/lib/pki-ca/alias',nickname='Server-Cert > cert-pki-ca',token='NSS Certificate DB',pin set > certificate: > type=NSSDB,location='/var/lib/pki-ca/alias',nickname='Server-Cert > cert-pki-ca',token='NSS Certificate DB' > CA: dogtag-ipa-renew-agent > issuer: CN=Certificate Authority,O=PLACEIQ.NET > subject: CN=sso-110.nym1.placeiq.net > ,O=PLACEIQ.NET > expires: 2018-08-06 10:36:02 UTC > key usage: > digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment > eku: id-kp-serverAuth,id-kp-clientAuth > pre-save command: > post-save command: > track: yes > auto-renew: yes > Request ID '20140923213814': > status: MONITORING > stuck: no > key pair storage: > type=NSSDB,location='/etc/dirsrv/slapd-PLACEIQ-NET',nickname='Server-Cert',token='NSS > Certificate DB',pinfile='/etc/dirsrv/slapd-PLACEIQ-NET > /pwdfile.txt' > certificate: > type=NSSDB,location='/etc/dirsrv/slapd-PLACEIQ-NET',nickname='Server-Cert',token='NSS > Certificate DB' > CA: IPA > issuer: CN=Certificate Authority,O=PLACEIQ.NET > subject: CN=sso-110.nym1.placeiq.net > ,O=PLACEIQ.NET > expires: 2018-08-28 10:36:04 UTC > key usage: > digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment > pre-save command: > post-save command: /usr/lib64/ipa/certmonger/restart_dirsrv > PLACEIQ-NET > track: yes > auto-renew: yes > Request ID '20140923213856': > status: MONITORING > stuck: no > key pair storage: > type=NSSDB,location='/etc/httpd/alias',nickname='Server-Cert',token='NSS > Certificate DB',pinfile='/etc/httpd/alias/pwdfile.txt' > certificate: > type=NSSDB,location='/etc/httpd/alias',nickname='Server-Cert',token='NSS > Certificate DB' > CA: IPA > issuer: CN=Certificate Authority,O=PLACEIQ.NET > subject: CN=sso-110.nym1.placeiq.net > ,O=PLACEIQ.NET > expires: 2018-08-28 10:36:04 UTC > key usage: > digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment > eku: id-kp-serverAuth,id-kp-clientAuth > pre-save command: > post-save command: /usr/lib64/ipa/certmonger/restart_httpd > track: yes > auto-renew: yes > Request ID '20160119021025': > status: MONITORING > stuck: no > key pair storage: > type=NSSDB,location='/var/lib/pki-ca/alias',nickname='auditSigningCert > cert-pki-ca',token='NSS Certificate DB',pin set > certificate: > type=NSSDB,location='/var/lib/pki-ca/alias',nickname='auditSigningCert > cert-pki-ca',token='NSS Certificate DB' > CA: dogtag-ipa-renew-agent > issuer: CN=Certificate Authority,O=PLACEIQ.NET > subject: CN=CA Audit,O=PLACEIQ.NET > expires: 2017-10-26 04:38:19 UTC > key usage: digitalSignature,nonRepudiation > pre-save command: /usr/lib64/ipa/certmonger/stop_pkicad > post-save command: /usr/lib64/ipa/certmonger/renew_ca_cert > "auditSigningCert cert-pki-ca" > track: yes > auto-renew: yes > Request ID '20160119021038': > status: MONITORING > stuck: no > key pair storage: > type=NSSDB,location='/var/lib/pki-ca/alias',nickname='ocspSigningCert > cert-pki-ca',token='NSS Certificate DB',pin set > certificate: > type=NSSDB,location='/var/lib/pki-ca/alias',nickname='ocspSigningCert > cert-pki-ca',token='NSS Certificate DB' > CA: dogtag-ipa-renew-agent > issuer: CN=Certificate Authority,O=PLACEIQ.NET > subject: CN=OCSP Subsystem,O=PLACEIQ.NET > expires: 2017-10-26 04:37:19 UTC > eku: id-kp-OCSPSigning > pre-save command: /usr/lib64/ipa/certmonger/stop_pkicad > post-save command: /usr/lib64/ipa/certmonger/renew_ca_cert > "ocspSigningCert cert-pki-ca" > track: yes > auto-renew: yes > Request ID '20160119021055': > status: MONITORING > stuck: no > key pair storage: > type=NSSDB,location='/var/lib/pki-ca/alias',nickname='subsystemCert > cert-pki-ca',token='NSS Certificate DB',pin set > certificate: > type=NSSDB,location='/var/lib/pki-ca/alias',nickname='subsystemCert > cert-pki-ca',token='NSS Certificate DB' > CA: dogtag-ipa-renew-agent > issuer: CN=Certificate Authority,O=PLACEIQ.NET > subject: CN=CA Subsystem,O=PLACEIQ.NET > expires: 2017-10-26 04:37:19 UTC > key usage: > digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment > eku: id-kp-serverAuth,id-kp-clientAuth > pre-save command: /usr/lib64/ipa/certmonger/stop_pkicad > post-save command: /usr/lib64/ipa/certmonger/renew_ca_cert > "subsystemCert cert-pki-ca" > track: yes > auto-renew: yes > Request ID '20160119021104': > status: MONITORING > stuck: no > key pair storage: > type=NSSDB,location='/etc/httpd/alias',nickname='ipaCert',token='NSS > Certificate DB',pinfile='/etc/httpd/alias/pwdfile.txt' > certificate: > type=NSSDB,location='/etc/httpd/alias',nickname='ipaCert',token='NSS > Certificate DB' > CA: dogtag-ipa-renew-agent > issuer: CN=Certificate Authority,O=PLACEIQ.NET > subject: CN=IPA RA,O=PLACEIQ.NET > expires: 2017-10-26 04:37:19 UTC > key usage: > digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment > eku: id-kp-serverAuth,id-kp-clientAuth > pre-save command: > post-save command: /usr/lib64/ipa/certmonger/renew_ra_cert > track: yes > auto-renew: yes > > > The new replica: > > [root at sso-108:(NYM) ~]$ getcert list > Number of certificates and requests being tracked: 8. > Request ID '20160927191253': > status: MONITORING > stuck: no > key pair storage: > type=NSSDB,location='/etc/dirsrv/slapd-PKI-IPA',nickname='Server-Cert',token='NSS > Certificate DB',pinfile='/etc/dirsrv/slapd-PKI-IPA/pwdfile > .txt' > certificate: > type=NSSDB,location='/etc/dirsrv/slapd-PKI-IPA',nickname='Server-Cert',token='NSS > Certificate DB' > CA: IPA > issuer: CN=Certificate Authority,O=PLACEIQ.NET > subject: CN=sso-108.nym1.placeiq.net > ,O=PLACEIQ.NET > expires: 2018-09-28 19:10:33 UTC > key usage: > digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment > eku: id-kp-serverAuth,id-kp-clientAuth > pre-save command: > post-save command: /usr/lib64/ipa/certmonger/restart_dirsrv PKI-IPA > track: yes > auto-renew: yes > Request ID '20160927191452': > status: CA_WORKING > stuck: no > key pair storage: > type=NSSDB,location='/var/lib/pki-ca/alias',nickname='auditSigningCert > cert-pki-ca',token='NSS Certificate DB',pin set > certificate: > type=NSSDB,location='/var/lib/pki-ca/alias',nickname='auditSigningCert > cert-pki-ca',token='NSS Certificate DB' > CA: dogtag-ipa-retrieve-agent-submit > issuer: CN=Certificate Authority,O=PLACEIQ.NET > subject: CN=CA Audit,O=PLACEIQ.NET > expires: 2015-12-03 21:57:56 UTC > key usage: digitalSignature,nonRepudiation > pre-save command: /usr/lib64/ipa/certmonger/stop_pkicad > post-save command: /usr/lib64/ipa/certmonger/restart_pkicad > "auditSigningCert cert-pki-ca" > track: yes > auto-renew: yes > Request ID '20160927191453': > status: CA_WORKING > stuck: no > key pair storage: > type=NSSDB,location='/var/lib/pki-ca/alias',nickname='ocspSigningCert > cert-pki-ca',token='NSS Certificate DB',pin set > certificate: > type=NSSDB,location='/var/lib/pki-ca/alias',nickname='ocspSigningCert > cert-pki-ca',token='NSS Certificate DB' > CA: dogtag-ipa-retrieve-agent-submit > issuer: CN=Certificate Authority,O=PLACEIQ.NET > subject: CN=OCSP Subsystem,O=PLACEIQ.NET > expires: 2015-12-03 21:57:56 UTC > key usage: digitalSignature,nonRepudiation,keyCertSign,cRLSign > eku: id-kp-OCSPSigning > pre-save command: /usr/lib64/ipa/certmonger/stop_pkicad > post-save command: /usr/lib64/ipa/certmonger/restart_pkicad > "ocspSigningCert cert-pki-ca" > track: yes > auto-renew: yes > Request ID '20160927191454': > status: CA_WORKING > stuck: no > key pair storage: > type=NSSDB,location='/var/lib/pki-ca/alias',nickname='subsystemCert > cert-pki-ca',token='NSS Certificate DB',pin set > certificate: > type=NSSDB,location='/var/lib/pki-ca/alias',nickname='subsystemCert > cert-pki-ca',token='NSS Certificate DB' > CA: dogtag-ipa-retrieve-agent-submit > issuer: CN=Certificate Authority,O=PLACEIQ.NET > subject: CN=CA Subsystem,O=PLACEIQ.NET > expires: 2015-12-03 21:57:56 UTC > key usage: > digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment > eku: id-kp-serverAuth,id-kp-clientAuth > pre-save command: /usr/lib64/ipa/certmonger/stop_pkicad > post-save command: /usr/lib64/ipa/certmonger/restart_pkicad > "subsystemCert cert-pki-ca" > track: yes > auto-renew: yes > Request ID '20160927191455': > status: MONITORING > stuck: no > key pair storage: > type=NSSDB,location='/var/lib/pki-ca/alias',nickname='Server-Cert > cert-pki-ca',token='NSS Certificate DB',pin set > certificate: > type=NSSDB,location='/var/lib/pki-ca/alias',nickname='Server-Cert > cert-pki-ca',token='NSS Certificate DB' > CA: dogtag-ipa-renew-agent > issuer: CN=Certificate Authority,O=PLACEIQ.NET > subject: CN=sso-108.nym1.placeiq.net > ,O=PLACEIQ.NET > expires: 2018-09-17 19:14:36 UTC > key usage: > digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment > eku: id-kp-serverAuth > pre-save command: > post-save command: > track: yes > auto-renew: yes > Request ID '20160927191540': > status: MONITORING > stuck: no > key pair storage: > type=NSSDB,location='/etc/dirsrv/slapd-PLACEIQ-NET',nickname='Server-Cert',token='NSS > Certificate DB',pinfile='/etc/dirsrv/slapd-PLACEIQ-NET > /pwdfile.txt' > certificate: > type=NSSDB,location='/etc/dirsrv/slapd-PLACEIQ-NET',nickname='Server-Cert',token='NSS > Certificate DB' > CA: IPA > issuer: CN=Certificate Authority,O=PLACEIQ.NET > subject: CN=sso-108.nym1.placeiq.net > ,O=PLACEIQ.NET > expires: 2018-09-28 19:10:32 UTC > key usage: > digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment > eku: id-kp-serverAuth,id-kp-clientAuth > pre-save command: > post-save command: /usr/lib64/ipa/certmonger/restart_dirsrv > PLACEIQ-NET > track: yes > auto-renew: yes > Request ID '20160927192114': > status: MONITORING > stuck: no > key pair storage: > type=NSSDB,location='/etc/httpd/alias',nickname='Server-Cert',token='NSS > Certificate DB',pinfile='/etc/httpd/alias/pwdfile.txt' > certificate: > type=NSSDB,location='/etc/httpd/alias',nickname='Server-Cert',token='NSS > Certificate DB' > CA: IPA > issuer: CN=Certificate Authority,O=PLACEIQ.NET > subject: CN=sso-108.nym1.placeiq.net > ,O=PLACEIQ.NET > expires: 2018-09-28 19:10:34 UTC > key usage: > digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment > eku: id-kp-serverAuth,id-kp-clientAuth > pre-save command: > post-save command: /usr/lib64/ipa/certmonger/restart_httpd > track: yes > auto-renew: yes > Request ID '20160927192146': > status: MONITORING > stuck: no > key pair storage: > type=NSSDB,location='/etc/httpd/alias',nickname='ipaCert',token='NSS > Certificate DB',pinfile='/etc/httpd/alias/pwdfile.txt' > certificate: > type=NSSDB,location='/etc/httpd/alias',nickname='ipaCert',token='NSS > Certificate DB' > CA: dogtag-ipa-retrieve-agent-submit > issuer: CN=Certificate Authority,O=PLACEIQ.NET > subject: CN=IPA RA,O=PLACEIQ.NET > expires: 2017-10-26 04:37:19 UTC > key usage: > digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment > eku: id-kp-serverAuth,id-kp-clientAuth > pre-save command: > post-save command: /usr/lib64/ipa/certmonger/restart_httpd > track: yes > auto-renew: yes > > > > > > > Jim Richard > > > > SYSTEM ADMINISTRATOR III > /(646) 338-8905 / > > > PlaceIQ:Location > Data Accuracy > > > > > > From bret.wortman at damascusgrp.com Wed Sep 28 13:30:37 2016 From: bret.wortman at damascusgrp.com (Bret Wortman) Date: Wed, 28 Sep 2016 09:30:37 -0400 Subject: [Freeipa-users] How to get a new cert In-Reply-To: <3b75947f-b104-659f-c785-8e8a34b6ed54@damascusgrp.com> References: <7b627575-bc54-438b-3d2f-dec72615c95b@damascusgrp.com> <3b75947f-b104-659f-c785-8e8a34b6ed54@damascusgrp.com> Message-ID: Yeah, I'm still not getting this, and I'm probably missing something painfully obvious. I follow the steps here: 1. Log into the server for which I need the cert. 2. # certutil -R -d /etc/pki/nssdb -a -g 2048 -s "CN=testesk1.internal.net,O=INTERNAL.NET" > ssl.csr I then copy the contents of the csr file and paste it into the FreeIPA UI after selecting Actions->New Certificiate from the Host Settings page. 3. I then click Actions->Get Certificate on that same page to extract the contents and paste it into a new .pem file on the requesting host. But how do I get at the key that was used in the creation of this cert? I can get the cacert, and I've got the newly-issued cert, but what about the key? Thanks! Bret On 09/27/2016 02:00 PM, Bret Wortman wrote: > That looks like it worked, but I have a follow-on question: > > I need to provide my RabbitMQ instance with a cacert file, a cert, and > a key file. These seem to be .pem files. Is there an easy way to > gather these 3 files from a typical IPA client node? > > Merci! > > > Bret > > > On 09/27/2016 11:28 AM, Florence Blanc-Renaud wrote: >> Hi Bret, >> >> would the following be helpful? In "Linux Domain Identity, >> Authentication, and Policy Guide", Chapter 17.1.1 Requesting New >> Certificates for a User, Host, or Service [1] >> >> Flo. >> >> [1] >> https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/7/html/Linux_Domain_Identity_Authentication_and_Policy_Guide/certificates.html#certificate-request >> >> On 09/27/2016 04:20 PM, Bret Wortman wrote: >>> Is there a guide anywhere for how to obtain an SSL certificate for a >>> new >>> server & service from the IPA CA master? Most of the guides I'm seeing >>> online use web pages at the major CAs to do this and I'd like to >>> keep it >>> in the family. >>> >>> Thanks! >>> >>> >>> -- >>> *Bret Wortman* >>> >>> http://wrapbuddies.co/ >>> >>> >> > From flo at redhat.com Wed Sep 28 13:47:00 2016 From: flo at redhat.com (Florence Blanc-Renaud) Date: Wed, 28 Sep 2016 15:47:00 +0200 Subject: [Freeipa-users] How to get a new cert In-Reply-To: <3b75947f-b104-659f-c785-8e8a34b6ed54@damascusgrp.com> References: <7b627575-bc54-438b-3d2f-dec72615c95b@damascusgrp.com> <3b75947f-b104-659f-c785-8e8a34b6ed54@damascusgrp.com> Message-ID: <0c8fc35b-03e8-1402-6cf2-89eea08e0891@redhat.com> On 09/27/2016 08:00 PM, Bret Wortman wrote: > That looks like it worked, but I have a follow-on question: > > I need to provide my RabbitMQ instance with a cacert file, a cert, and a > key file. These seem to be .pem files. Is there an easy way to gather > these 3 files from a typical IPA client node? > Hi, you can retrieve the new cert using the GUI. Navigate to Identity tab, then Users or Hosts or Services and pick your user, host or service. You will find in the "Actions" button a command to "Get Certificate". This will open a new window with the content of the cert, that you can copy/paste into mycert.pem. Once you have obtained mycert.pem, you can add it to the NSS database that you used previously in order to generate the CSR: $ certutil -A -d path_to_database -i mycert.pem -t u,u,u -n mycert Add IPA CA to the nss database: $ certutil -A -d path_to_database -n "IPA CA" -t CT,, -a < /etc/ipa/ca.crt Then pk12util and openssl will allow you to extract the key and certs through a temp keys.p12 file: $ pk12util -o keys.p12 -n mycert -d path_to_database $ openssl pkcs12 -in keys.p12 -out mykey.pem -nodes The output is mykey.pem which contains the key, the new certificate and IPA CA certificate. HTH, Flo. > Merci! > > > Bret > > > On 09/27/2016 11:28 AM, Florence Blanc-Renaud wrote: >> Hi Bret, >> >> would the following be helpful? In "Linux Domain Identity, >> Authentication, and Policy Guide", Chapter 17.1.1 Requesting New >> Certificates for a User, Host, or Service [1] >> >> Flo. >> >> [1] >> https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/7/html/Linux_Domain_Identity_Authentication_and_Policy_Guide/certificates.html#certificate-request >> >> >> On 09/27/2016 04:20 PM, Bret Wortman wrote: >>> Is there a guide anywhere for how to obtain an SSL certificate for a new >>> server & service from the IPA CA master? Most of the guides I'm seeing >>> online use web pages at the major CAs to do this and I'd like to keep it >>> in the family. >>> >>> Thanks! >>> >>> >>> -- >>> *Bret Wortman* >>> >>> http://wrapbuddies.co/ >>> >>> >> > From bret.wortman at damascusgrp.com Wed Sep 28 13:50:56 2016 From: bret.wortman at damascusgrp.com (Bret Wortman) Date: Wed, 28 Sep 2016 09:50:56 -0400 Subject: [Freeipa-users] How to get a new cert In-Reply-To: <0c8fc35b-03e8-1402-6cf2-89eea08e0891@redhat.com> References: <7b627575-bc54-438b-3d2f-dec72615c95b@damascusgrp.com> <3b75947f-b104-659f-c785-8e8a34b6ed54@damascusgrp.com> <0c8fc35b-03e8-1402-6cf2-89eea08e0891@redhat.com> Message-ID: Perfect. That did the trick. Many thanks, Flo. Bret On 09/28/2016 09:47 AM, Florence Blanc-Renaud wrote: > On 09/27/2016 08:00 PM, Bret Wortman wrote: >> That looks like it worked, but I have a follow-on question: >> >> I need to provide my RabbitMQ instance with a cacert file, a cert, and a >> key file. These seem to be .pem files. Is there an easy way to gather >> these 3 files from a typical IPA client node? >> > Hi, > > you can retrieve the new cert using the GUI. Navigate to Identity tab, > then Users or Hosts or Services and pick your user, host or service. > You will find in the "Actions" button a command to "Get Certificate". > This will open a new window with the content of the cert, that you can > copy/paste into mycert.pem. > > Once you have obtained mycert.pem, you can add it to the NSS database > that you used previously in order to generate the CSR: > $ certutil -A -d path_to_database -i mycert.pem -t u,u,u -n mycert > > Add IPA CA to the nss database: > $ certutil -A -d path_to_database -n "IPA CA" -t CT,, -a < > /etc/ipa/ca.crt > > Then pk12util and openssl will allow you to extract the key and certs > through a temp keys.p12 file: > $ pk12util -o keys.p12 -n mycert -d path_to_database > $ openssl pkcs12 -in keys.p12 -out mykey.pem -nodes > > The output is mykey.pem which contains the key, the new certificate > and IPA CA certificate. > > HTH, > Flo. > >> Merci! >> >> >> Bret >> >> >> On 09/27/2016 11:28 AM, Florence Blanc-Renaud wrote: >>> Hi Bret, >>> >>> would the following be helpful? In "Linux Domain Identity, >>> Authentication, and Policy Guide", Chapter 17.1.1 Requesting New >>> Certificates for a User, Host, or Service [1] >>> >>> Flo. >>> >>> [1] >>> https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/7/html/Linux_Domain_Identity_Authentication_and_Policy_Guide/certificates.html#certificate-request >>> >>> >>> >>> On 09/27/2016 04:20 PM, Bret Wortman wrote: >>>> Is there a guide anywhere for how to obtain an SSL certificate for >>>> a new >>>> server & service from the IPA CA master? Most of the guides I'm seeing >>>> online use web pages at the major CAs to do this and I'd like to >>>> keep it >>>> in the family. >>>> >>>> Thanks! >>>> >>>> >>>> -- >>>> *Bret Wortman* >>>> >>>> http://wrapbuddies.co/ >>>> >>>> >>> >> > From freeipa at jacobdevans.com Wed Sep 28 19:15:03 2016 From: freeipa at jacobdevans.com (Jake) Date: Wed, 28 Sep 2016 15:15:03 -0400 (EDT) Subject: [Freeipa-users] unable to add or remove host (exists but doesn't exist) 4.2 Message-ID: <1585133803.2460.1475090103899@vegas.jacobdevans.com> One of my tech's had an issue adding a machine, now it seems to be stuck. it can neither be added or removed -bash-4.2$ ipa host-add server100 .example.com --force ipa: ERROR: host with name " server100 .example.com " already exists -bash-4.2$ ipa host-del server100 .example.com ipa: ERROR: server100.example.com: host not found IP web client gives this error: Operations Error Some operations failed. [ https://c05-rd-ipa01.ipa.clarkinc.io/ipa/ui/# | Hide details ] * server100.example.com: host not found Attempts to delete it via webui Operations Error Some entries were not deleted [ https://c05-rd-ipa01.ipa.clarkinc.io/ipa/ui/# | Hide details ] * server100.example.com : host not found -------------- next part -------------- An HTML attachment was scrubbed... URL: From razvan.vilt at me.com Wed Sep 28 19:44:36 2016 From: razvan.vilt at me.com (=?utf-8?Q?=22R=C4=83zvan_Corneliu_C=2ER=2E_VILT=22?=) Date: Wed, 28 Sep 2016 22:44:36 +0300 Subject: [Freeipa-users] unable to add or remove host (exists but doesn't exist) 4.2 In-Reply-To: <1585133803.2460.1475090103899@vegas.jacobdevans.com> References: <1585133803.2460.1475090103899@vegas.jacobdevans.com> Message-ID: <33BBB335-8103-43FF-B352-28BD418A2023@me.com> Hi Jake, From the output below (assuming a non-altered copy+paste), the host-add request is interpreted by ipa as "server100.example.com " while the delete is interpreted as "server100.example.com ". If that's the case, try using quotes or searching the LDAP manually for the entry to see the actual registered hostname. If what I'm seeing over here is correct, you might be able to fix it by issuing: ipa host-del "server100.example.com " (use quotes when issuing the command). Then again, it might be a wrong copy+paste. Cheers, R?zvan > On 28 Sep 2016, at 22:15, Jake wrote: > > One of my tech's had an issue adding a machine, now it seems to be stuck. > > it can neither be added or removed > > -bash-4.2$ ipa host-add server100.example.com --force > ipa: ERROR: host with name " server100.example.com" already exists > > > -bash-4.2$ ipa host-del server100.example.com > ipa: ERROR: server100.example.com: host not found > > > IP web client gives this error: > > > Operations Error > Some operations failed. > Hide details > server100.example.com: host not found > > Attempts to delete it via webui > Operations Error > Some entries were not deleted > Hide details > server100.example.com: host not found > > > -- > Manage your subscription for the Freeipa-users mailing list: > https://www.redhat.com/mailman/listinfo/freeipa-users > Go to http://freeipa.org for more info on the project -------------- next part -------------- An HTML attachment was scrubbed... URL: From floyd.lorch at gmail.com Wed Sep 28 21:36:34 2016 From: floyd.lorch at gmail.com (Floyd Lorch) Date: Wed, 28 Sep 2016 17:36:34 -0400 Subject: [Freeipa-users] New Installation won't start after reboot Message-ID: Installed FreeIPA 4.2 on a fresh CentOS 7.2. After initial setup and configuration a one way trust was added to windows AD. Server was shut down and moved to a different rack. When it was started back up IPA no longer runs. ipa-admintools.x86_64 4.2.0-15.0.1.el7.centos.19 ipa-client.x86_64 4.2.0-15.0.1.el7.centos.19 ipa-python.x86_64 4.2.0-15.0.1.el7.centos.19 ipa-server.x86_64 4.2.0-15.0.1.el7.centos.19 ipa-server-dns.x86_64 4.2.0-15.0.1.el7.centos.19 ipa-server-trust-ad.x86_64 4.2.0-15.0.1.el7.centos.19 libipa_hbac.x86_64 1.13.0-40.el7_2.12 python-iniparse.noarch 0.4-9.el7 python-libipa_hbac.x86_64 1.13.0-40.el7_2.12 sssd-ipa.x86_64 1.13.0-40.el7_2.12 In the service list IPA, polkit. postfix, and smb service are currently failed. kadmin is also failed sometimes, however I'm able to start it occasionally without explanation. running IPA restart results in the following error: [root at SERVER ~]# ipactl restart Starting Directory Service Stopping pki-tomcatd Service Restarting krb5kdc Service Restarting kadmin Service Restarting named Service Restarting ipa_memcached Service Restarting httpd Service Restarting pki-tomcatd Service Starting smb Service Job for smb.service failed because the control process exited with error code. See "systemctl status smb.service" and "journalctl -xe" for details. Failed to start smb Service Shutting down Aborting ipactl Checking the SMB service I get the following [root at SERVER ~]# systemctl status smb ? smb.service - Samba SMB Daemon Loaded: loaded (/usr/lib/systemd/system/smb.service; disabled; vendor preset: disabled) Active: failed (Result: exit-code) since Wed 2016-09-28 17:15:50 EDT; 43s ago Process: 7186 ExecStart=/usr/sbin/smbd $SMBDOPTIONS (code=exited, status=1/FAILURE) Main PID: 7186 (code=exited, status=1/FAILURE) Status: "Starting process..." Sep 28 17:15:49 SERVER.SUB.DOMAIN.COM smbd[7186]: [2016/09/28 17:15:49.718669, 0] ipa_sam.c:4208(bind_callback_cleanup) Sep 28 17:15:49 SERVER.SUB.DOMAIN.COM smbd[7186]: kerberos error: code=-1765328203, message=Keytab contains no suitable keys for cifs/ SERVER.SUB.DOMAIN.COM at SUB.DOMAIN.COM Sep 28 17:15:50 SERVER.SUB.DOMAIN.COM smbd[7186]: [2016/09/28 17:15:50.718869, 0] ipa_sam.c:4520(pdb_init_ipasam) Sep 28 17:15:50 SERVER.SUB.DOMAIN.COM smbd[7186]: Failed to get base DN. Sep 28 17:15:50 SERVER.SUB.DOMAIN.COM smbd[7186]: [2016/09/28 17:15:50.718900, 0] ../source3/passdb/pdb_interface.c:179(make_pdb_method_name) Sep 28 17:15:50 SERVER.SUB.DOMAIN.COM smbd[7186]: pdb backend ipasam:ldapi://%2fvar%2frun%2fslapd-SUB.DOMAIN.COM.socket did not correctly init (error was NT_STATUS_UNSUCCESSFUL) Sep 28 17:15:50 SERVER.SUB.DOMAIN.COM systemd[1]: smb.service: main process exited, code=exited, status=1/FAILURE Sep 28 17:15:50 SERVER.SUB.DOMAIN.COM systemd[1]: Failed to start Samba SMB Daemon. Sep 28 17:15:50 SERVER.SUB.DOMAIN.COM systemd[1]: Unit smb.service entered failed state. Sep 28 17:15:50 SERVER.SUB.DOMAIN.COM systemd[1]: smb.service failed. I had issues when first trying to add one way trust with SMB but I was able to restart the service and move forward. That doesn't appear to be happening this time. I've also tried using kinit to obtain a ticket but that doesn't work. I'm not sure if that should work at this juncture or not. After restarting ipa the kadmin service also fails to start some of the time. if kadmin is running ======================= [root at SERVER ~]# kinit -V admin Using default cache: persistent:0:0 Using principal: admin at SUB.DOMAIN.COM kinit: Generic error (see e-text) while getting initial credentials if kadmin isn't running ======================= [root at SERVER ~]# kinit -V admin Using default cache: persistent:0:0 Using principal: admin at SUB.DOMAIN.COM kinit: Cannot contact any KDC for realm 'SUB.DOMAIN.COM' while getting initial credentials Attempts to get kadmin to run ======================= [root at SERVER ~]# systemctl start kadmin Job for kadmin.service failed because the control process exited with error code. See "systemctl status kadmin.service" and "journalctl -xe" for details. Journal for kadmin attempt ====================== -- Subject: Unit kadmin.service has begun start-up -- Defined-By: systemd -- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel -- -- Unit kadmin.service has begun starting up. Sep 28 17:22:38 SERVER.SUB.DOMAIN.COM _kadmind[7978]: kadmind: kadmind: Server error while initializing, aborting Sep 28 17:22:38 SERVER.SUB.DOMAIN.COM systemd[1]: kadmin.service: control process exited, code=exited status=1 Sep 28 17:22:38 SERVER.SUB.DOMAIN.COM systemd[1]: Failed to start Kerberos 5 Password-changing and Administration. -- Subject: Unit kadmin.service has failed -- Defined-By: systemd -- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel -- -- Unit kadmin.service has failed. -- -- The result is failed. Sep 28 17:22:38 SERVER.SUB.DOMAIN.COM systemd[1]: Unit kadmin.service entered failed state. Sep 28 17:22:38 SERVER.SUB.DOMAIN.COM systemd[1]: kadmin.service failed. Sep 28 17:22:38 SERVER.SUB.DOMAIN.COM polkitd[6092]: Unregistered Authentication Agent for unix-process:7973:9203783 (system bus name :1.639, object path /org/freedesktop/PolicyKit1/AuthenticationAgent, locale en_US.UTF-8) (dis -------------- next part -------------- An HTML attachment was scrubbed... URL: From rcritten at redhat.com Wed Sep 28 21:57:32 2016 From: rcritten at redhat.com (Rob Crittenden) Date: Wed, 28 Sep 2016 23:57:32 +0200 Subject: [Freeipa-users] unable to add or remove host (exists but doesn't exist) 4.2 In-Reply-To: <33BBB335-8103-43FF-B352-28BD418A2023@me.com> References: <1585133803.2460.1475090103899@vegas.jacobdevans.com> <33BBB335-8103-43FF-B352-28BD418A2023@me.com> Message-ID: <57EC3CCC.6080901@redhat.com> "R?zvan Corneliu C.R. VILT" wrote: > Hi Jake, > > From the output below (assuming a non-altered copy+paste), the host-add > request is interpreted by ipa as "server100.example.com > " while the delete is interpreted as > "server100.example.com ". If that's the > case, try using quotes or searching the LDAP manually for the entry to > see the actual registered hostname. If what I'm seeing over here is > correct, you might be able to fix it by issuing: > ipa host-del "server100.example.com " > (use quotes when issuing the command). > > Then again, it might be a wrong copy+paste. Yeah, that quoted space sure looks strange. I think this suggestion is good, though I'm not sure how an embedded space could happen. An ldapsearch might give additional feedback, something like: $ kinit admin $ ldapsearch -Y GSSAPI -b cn=computers,cn=accounts,dc=example,dc=com fqdn="*server100*" rob > > Cheers, > R?zvan >> On 28 Sep 2016, at 22:15, Jake > > wrote: >> >> One of my tech's had an issue adding a machine, now it seems to be stuck. >> >> it can neither be added or removed >> >> -bash-4.2$ ipa host-add server100.example.com --force >> ipa: ERROR: host with name " server100.example.com >> " already exists >> >> >> -bash-4.2$ ipa host-del server100.example.com >> ipa: ERROR: server100.example.com : host >> not found >> >> >> IP web client gives this error: >> >> >> Operations Error >> >> Some operations failed. >> Hide details >> >> * >> server100.example.com : host not found >> >> >> Attempts to delete it via webui >> >> >> Operations Error >> >> Some entries were not deleted >> Hide details >> >> * >> server100.example.com : host not found >> >> >> >> -- >> Manage your subscription for the Freeipa-users mailing list: >> https://www.redhat.com/mailman/listinfo/freeipa-users >> Go to http://freeipa.org for more info on the project > > > From prasun.gera at gmail.com Thu Sep 29 04:47:34 2016 From: prasun.gera at gmail.com (Prasun Gera) Date: Thu, 29 Sep 2016 00:47:34 -0400 Subject: [Freeipa-users] SELinux errors with sssd-krb5-common-1.13.0-40.el7_2.12.x86_64 Message-ID: I started seeing some selinux errors on one of my RHEL 7 clients recently (possibly after a recent yum update ?), which prevents users from logging in with passwords. I've put SELinux in permissive mode for now. Logs follow SELinux is preventing /usr/libexec/sssd/krb5_child from read access on the key Unknown. ***** Plugin catchall (100. confidence) suggests ************************** If you believe that krb5_child should be allowed read access on the Unknown key by default. Then you should report this as a bug. You can generate a local policy module to allow this access. Do allow this access for now by executing: # grep krb5_child /var/log/audit/audit.log | audit2allow -M mypol # semodule -i mypol.pp Additional Information: Source Context system_u:system_r:sssd_t:s0 Target Context system_u:system_r:unconfined_service_t:s0 Target Objects Unknown [ key ] Source krb5_child Source Path /usr/libexec/sssd/krb5_child Port Host Source RPM Packages sssd-krb5-common-1.13.0-40.el7_2.12.x86_64 Target RPM Packages Policy RPM selinux-policy-3.13.1-60.el7_2.9.noarch Selinux Enabled True Policy Type targeted Enforcing Mode Permissive Host Name example.com Platform Linux example.com 4.4.19-1.el7.x86_64 #1 SMP Mon Aug 29 18:38:32 EDT 2016 x86_64 x86_64 Alert Count 38 First Seen 2016-09-28 18:37:43 EDT Last Seen 2016-09-28 22:08:41 EDT Local ID aa5271fa-f708-46b0-a382-fb1f90ce8973 Raw Audit Messages type=AVC msg=audit(1475114921.376:90787): avc: denied { read } for pid=8272 comm="krb5_child" scontext=system_u:system_r:sssd_t:s0 tcontext=system_u:system_r:unconfined_service_t:s0 tclass=key permissive=0 type=SYSCALL msg=audit(1475114921.376:90787): arch=x86_64 syscall=keyctl success=yes exit=EINTR a0=b a1=333b5463 a2=0 a3=0 items=0 ppid=891 pid=8272 auid=4294967295 uid=1388200053 gid=1388200053 euid=1388200053 suid=1388200053 fsuid=1388200053 egid=1388200053 sgid=1388200053 fsgid=1388200053 tty=(none) ses=4294967295 comm=krb5_child exe=/usr/libexec/sssd/krb5_child subj=system_u:system_r:sssd_t:s0 key=(null) Hash: krb5_child,sssd_t,unconfined_service_t,key,read -------------------------------------------------------------------------------- SELinux is preventing /usr/libexec/sssd/krb5_child from view access on the key Unknown. ***** Plugin catchall (100. confidence) suggests ************************** If you believe that krb5_child should be allowed view access on the Unknown key by default. Then you should report this as a bug. You can generate a local policy module to allow this access. Do allow this access for now by executing: # grep krb5_child /var/log/audit/audit.log | audit2allow -M mypol # semodule -i mypol.pp Additional Information: Source Context system_u:system_r:sssd_t:s0 Target Context system_u:system_r:unconfined_service_t:s0 Target Objects Unknown [ key ] Source krb5_child Source Path /usr/libexec/sssd/krb5_child Port Host Source RPM Packages sssd-krb5-common-1.13.0-40.el7_2.12.x86_64 Target RPM Packages Policy RPM selinux-policy-3.13.1-60.el7_2.9.noarch Selinux Enabled True Policy Type targeted Enforcing Mode Permissive Host Name example.com Platform Linux example.com 4.4.19-1.el7.x86_64 #1 SMP Mon Aug 29 18:38:32 EDT 2016 x86_64 x86_64 Alert Count 10 First Seen 2016-09-28 18:40:00 EDT Last Seen 2016-09-28 22:08:41 EDT Local ID 22ec0970-9447-444a-9631-69749e4e7226 Raw Audit Messages type=AVC msg=audit(1475114921.376:90789): avc: denied { view } for pid=8272 comm="krb5_child" scontext=system_u:system_r:sssd_t:s0 tcontext=system_u:system_r:unconfined_service_t:s0 tclass=key permissive=0 type=SYSCALL msg=audit(1475114921.376:90789): arch=x86_64 syscall=keyctl success=no exit=EACCES a0=6 a1=2e1c07f1 a2=0 a3=0 items=0 ppid=891 pid=8272 auid=4294967295 uid=1388200053 gid=1388200053 euid=1388200053 suid=1388200053 fsuid=1388200053 egid=1388200053 sgid=1388200053 fsgid=1388200053 tty=(none) ses=4294967295 comm=krb5_child exe=/usr/libexec/sssd/krb5_child subj=system_u:system_r:sssd_t:s0 key=(null) Hash: krb5_child,sssd_t,unconfined_service_t,key,view -------------------------------------------------------------------------------- SELinux is preventing /usr/libexec/sssd/krb5_child from write access on the key Unknown. ***** Plugin catchall (100. confidence) suggests ************************** If you believe that krb5_child should be allowed write access on the Unknown key by default. Then you should report this as a bug. You can generate a local policy module to allow this access. Do allow this access for now by executing: # grep krb5_child /var/log/audit/audit.log | audit2allow -M mypol # semodule -i mypol.pp Additional Information: Source Context system_u:system_r:sssd_t:s0 Target Context system_u:system_r:unconfined_service_t:s0 Target Objects Unknown [ key ] Source krb5_child Source Path /usr/libexec/sssd/krb5_child Port Host Source RPM Packages sssd-krb5-common-1.13.0-40.el7_2.12.x86_64 Target RPM Packages Policy RPM selinux-policy-3.13.1-60.el7_2.9.noarch Selinux Enabled True Policy Type targeted Enforcing Mode Permissive Host Name example.com Platform Linux example.com 4.4.19-1.el7.x86_64 #1 SMP Mon Aug 29 18:38:32 EDT 2016 x86_64 x86_64 Alert Count 10 First Seen 2016-09-28 18:40:00 EDT Last Seen 2016-09-28 22:08:41 EDT Local ID 8982bbec-38db-485b-9266-57fdaa8a3621 Raw Audit Messages type=AVC msg=audit(1475114921.376:90790): avc: denied { write } for pid=8272 comm="krb5_child" scontext=system_u:system_r:sssd_t:s0 tcontext=system_u:system_r:unconfined_service_t:s0 tclass=key permissive=0 type=SYSCALL msg=audit(1475114921.376:90790): arch=x86_64 syscall=add_key success=no exit=EACCES a0=7f6987905ffc a1=7ffeed78b1f0 a2=0 a3=0 items=0 ppid=891 pid=8272 auid=4294967295 uid=1388200053 gid=1388200053 euid=1388200053 suid=1388200053 fsuid=1388200053 egid=1388200053 sgid=1388200053 fsgid=1388200053 tty=(none) ses=4294967295 comm=krb5_child exe=/usr/libexec/sssd/krb5_child subj=system_u:system_r:sssd_t:s0 key=(null) Hash: krb5_child,sssd_t,unconfined_service_t,key,write -------------- next part -------------- An HTML attachment was scrubbed... URL: From jrichard at placeiq.com Thu Sep 29 05:37:03 2016 From: jrichard at placeiq.com (Jim Richard) Date: Thu, 29 Sep 2016 01:37:03 -0400 Subject: [Freeipa-users] Replica created with expired certs In-Reply-To: <57EBAD23.4090604@redhat.com> References: <5EE38BB7-ED9B-4001-8F64-DB7CD2319C02@placeiq.com> <57EBAD23.4090604@redhat.com> Message-ID: <17A0F251-F5DF-447C-AB64-C053E83369D0@placeiq.com> Thanks Rob, that worked. Still on the subject of certs, any idea how to solve this error: Certificate format error: (SEC_ERROR_LEGACY_DATABASE) The certificate/key database is in an old, unsupported format. I see that in the gui when querying hosts as well as from cli when I ipa-show or ipa-find Jim Richard SYSTEM ADMINISTRATOR III (646) 338-8905 > On Sep 28, 2016, at 7:44 AM, Rob Crittenden wrote: > > Jim Richard wrote: >> I have a master with apparently correct, non expired certs but when I >> create a new replica master I end up with expired certs. >> How is this possible, why and of course, how do I fix? > > I assume you are running IPA v3.0.0? > > The problem is that the root CA stash isn't updated when a replica file is prepared in that version (fixed in 3.3 IIRC). You can do this manually with something like: > > # PKCS12Export -d /var/lib/pki-ca/alias -p /root/dbpass -w /root/dmpass -o /root/cacert.p12 > > where /root/dmpass is a file that contains the Directory Manager password. > > Then rerun ipa-replica-prepare and things should work. > > You can look at the certs in /root/cacert.p12 util pk12util to see the change. > > rob > >> >> first set is the original master and the second is the certs I get on >> the new replica >> >> [root at sso-110:(NYM) nssdb]$ getcert list >> Number of certificates and requests being tracked: 8. >> Request ID '20140923213643': >> status: MONITORING >> stuck: no >> key pair storage: >> type=NSSDB,location='/etc/dirsrv/slapd-PKI-IPA',nickname='Server-Cert',token='NSS >> Certificate DB',pinfile='/etc/dirsrv/slapd-PKI-IPA/pwdfile >> .txt' >> certificate: >> type=NSSDB,location='/etc/dirsrv/slapd-PKI-IPA',nickname='Server-Cert',token='NSS >> Certificate DB' >> CA: IPA >> issuer: CN=Certificate Authority,O=PLACEIQ.NET >> subject: CN=sso-110.nym1.placeiq.net >> ,O=PLACEIQ.NET >> expires: 2018-08-28 10:36:04 UTC >> key usage: >> digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment >> eku: id-kp-serverAuth,id-kp-clientAuth >> pre-save command: >> post-save command: /usr/lib64/ipa/certmonger/restart_dirsrv PKI-IPA >> track: yes >> auto-renew: yes >> Request ID '20140923213732': >> status: MONITORING >> stuck: no >> key pair storage: >> type=NSSDB,location='/var/lib/pki-ca/alias',nickname='Server-Cert >> cert-pki-ca',token='NSS Certificate DB',pin set >> certificate: >> type=NSSDB,location='/var/lib/pki-ca/alias',nickname='Server-Cert >> cert-pki-ca',token='NSS Certificate DB' >> CA: dogtag-ipa-renew-agent >> issuer: CN=Certificate Authority,O=PLACEIQ.NET >> subject: CN=sso-110.nym1.placeiq.net >> ,O=PLACEIQ.NET >> expires: 2018-08-06 10:36:02 UTC >> key usage: >> digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment >> eku: id-kp-serverAuth,id-kp-clientAuth >> pre-save command: >> post-save command: >> track: yes >> auto-renew: yes >> Request ID '20140923213814': >> status: MONITORING >> stuck: no >> key pair storage: >> type=NSSDB,location='/etc/dirsrv/slapd-PLACEIQ-NET',nickname='Server-Cert',token='NSS >> Certificate DB',pinfile='/etc/dirsrv/slapd-PLACEIQ-NET >> /pwdfile.txt' >> certificate: >> type=NSSDB,location='/etc/dirsrv/slapd-PLACEIQ-NET',nickname='Server-Cert',token='NSS >> Certificate DB' >> CA: IPA >> issuer: CN=Certificate Authority,O=PLACEIQ.NET >> subject: CN=sso-110.nym1.placeiq.net >> ,O=PLACEIQ.NET >> expires: 2018-08-28 10:36:04 UTC >> key usage: >> digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment >> pre-save command: >> post-save command: /usr/lib64/ipa/certmonger/restart_dirsrv >> PLACEIQ-NET >> track: yes >> auto-renew: yes >> Request ID '20140923213856': >> status: MONITORING >> stuck: no >> key pair storage: >> type=NSSDB,location='/etc/httpd/alias',nickname='Server-Cert',token='NSS >> Certificate DB',pinfile='/etc/httpd/alias/pwdfile.txt' >> certificate: >> type=NSSDB,location='/etc/httpd/alias',nickname='Server-Cert',token='NSS >> Certificate DB' >> CA: IPA >> issuer: CN=Certificate Authority,O=PLACEIQ.NET >> subject: CN=sso-110.nym1.placeiq.net >> ,O=PLACEIQ.NET >> expires: 2018-08-28 10:36:04 UTC >> key usage: >> digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment >> eku: id-kp-serverAuth,id-kp-clientAuth >> pre-save command: >> post-save command: /usr/lib64/ipa/certmonger/restart_httpd >> track: yes >> auto-renew: yes >> Request ID '20160119021025': >> status: MONITORING >> stuck: no >> key pair storage: >> type=NSSDB,location='/var/lib/pki-ca/alias',nickname='auditSigningCert >> cert-pki-ca',token='NSS Certificate DB',pin set >> certificate: >> type=NSSDB,location='/var/lib/pki-ca/alias',nickname='auditSigningCert >> cert-pki-ca',token='NSS Certificate DB' >> CA: dogtag-ipa-renew-agent >> issuer: CN=Certificate Authority,O=PLACEIQ.NET >> subject: CN=CA Audit,O=PLACEIQ.NET >> expires: 2017-10-26 04:38:19 UTC >> key usage: digitalSignature,nonRepudiation >> pre-save command: /usr/lib64/ipa/certmonger/stop_pkicad >> post-save command: /usr/lib64/ipa/certmonger/renew_ca_cert >> "auditSigningCert cert-pki-ca" >> track: yes >> auto-renew: yes >> Request ID '20160119021038': >> status: MONITORING >> stuck: no >> key pair storage: >> type=NSSDB,location='/var/lib/pki-ca/alias',nickname='ocspSigningCert >> cert-pki-ca',token='NSS Certificate DB',pin set >> certificate: >> type=NSSDB,location='/var/lib/pki-ca/alias',nickname='ocspSigningCert >> cert-pki-ca',token='NSS Certificate DB' >> CA: dogtag-ipa-renew-agent >> issuer: CN=Certificate Authority,O=PLACEIQ.NET >> subject: CN=OCSP Subsystem,O=PLACEIQ.NET >> expires: 2017-10-26 04:37:19 UTC >> eku: id-kp-OCSPSigning >> pre-save command: /usr/lib64/ipa/certmonger/stop_pkicad >> post-save command: /usr/lib64/ipa/certmonger/renew_ca_cert >> "ocspSigningCert cert-pki-ca" >> track: yes >> auto-renew: yes >> Request ID '20160119021055': >> status: MONITORING >> stuck: no >> key pair storage: >> type=NSSDB,location='/var/lib/pki-ca/alias',nickname='subsystemCert >> cert-pki-ca',token='NSS Certificate DB',pin set >> certificate: >> type=NSSDB,location='/var/lib/pki-ca/alias',nickname='subsystemCert >> cert-pki-ca',token='NSS Certificate DB' >> CA: dogtag-ipa-renew-agent >> issuer: CN=Certificate Authority,O=PLACEIQ.NET >> subject: CN=CA Subsystem,O=PLACEIQ.NET >> expires: 2017-10-26 04:37:19 UTC >> key usage: >> digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment >> eku: id-kp-serverAuth,id-kp-clientAuth >> pre-save command: /usr/lib64/ipa/certmonger/stop_pkicad >> post-save command: /usr/lib64/ipa/certmonger/renew_ca_cert >> "subsystemCert cert-pki-ca" >> track: yes >> auto-renew: yes >> Request ID '20160119021104': >> status: MONITORING >> stuck: no >> key pair storage: >> type=NSSDB,location='/etc/httpd/alias',nickname='ipaCert',token='NSS >> Certificate DB',pinfile='/etc/httpd/alias/pwdfile.txt' >> certificate: >> type=NSSDB,location='/etc/httpd/alias',nickname='ipaCert',token='NSS >> Certificate DB' >> CA: dogtag-ipa-renew-agent >> issuer: CN=Certificate Authority,O=PLACEIQ.NET >> subject: CN=IPA RA,O=PLACEIQ.NET >> expires: 2017-10-26 04:37:19 UTC >> key usage: >> digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment >> eku: id-kp-serverAuth,id-kp-clientAuth >> pre-save command: >> post-save command: /usr/lib64/ipa/certmonger/renew_ra_cert >> track: yes >> auto-renew: yes >> >> >> The new replica: >> >> [root at sso-108:(NYM) ~]$ getcert list >> Number of certificates and requests being tracked: 8. >> Request ID '20160927191253': >> status: MONITORING >> stuck: no >> key pair storage: >> type=NSSDB,location='/etc/dirsrv/slapd-PKI-IPA',nickname='Server-Cert',token='NSS >> Certificate DB',pinfile='/etc/dirsrv/slapd-PKI-IPA/pwdfile >> .txt' >> certificate: >> type=NSSDB,location='/etc/dirsrv/slapd-PKI-IPA',nickname='Server-Cert',token='NSS >> Certificate DB' >> CA: IPA >> issuer: CN=Certificate Authority,O=PLACEIQ.NET >> subject: CN=sso-108.nym1.placeiq.net >> ,O=PLACEIQ.NET >> expires: 2018-09-28 19:10:33 UTC >> key usage: >> digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment >> eku: id-kp-serverAuth,id-kp-clientAuth >> pre-save command: >> post-save command: /usr/lib64/ipa/certmonger/restart_dirsrv PKI-IPA >> track: yes >> auto-renew: yes >> Request ID '20160927191452': >> status: CA_WORKING >> stuck: no >> key pair storage: >> type=NSSDB,location='/var/lib/pki-ca/alias',nickname='auditSigningCert >> cert-pki-ca',token='NSS Certificate DB',pin set >> certificate: >> type=NSSDB,location='/var/lib/pki-ca/alias',nickname='auditSigningCert >> cert-pki-ca',token='NSS Certificate DB' >> CA: dogtag-ipa-retrieve-agent-submit >> issuer: CN=Certificate Authority,O=PLACEIQ.NET >> subject: CN=CA Audit,O=PLACEIQ.NET >> expires: 2015-12-03 21:57:56 UTC >> key usage: digitalSignature,nonRepudiation >> pre-save command: /usr/lib64/ipa/certmonger/stop_pkicad >> post-save command: /usr/lib64/ipa/certmonger/restart_pkicad >> "auditSigningCert cert-pki-ca" >> track: yes >> auto-renew: yes >> Request ID '20160927191453': >> status: CA_WORKING >> stuck: no >> key pair storage: >> type=NSSDB,location='/var/lib/pki-ca/alias',nickname='ocspSigningCert >> cert-pki-ca',token='NSS Certificate DB',pin set >> certificate: >> type=NSSDB,location='/var/lib/pki-ca/alias',nickname='ocspSigningCert >> cert-pki-ca',token='NSS Certificate DB' >> CA: dogtag-ipa-retrieve-agent-submit >> issuer: CN=Certificate Authority,O=PLACEIQ.NET >> subject: CN=OCSP Subsystem,O=PLACEIQ.NET >> expires: 2015-12-03 21:57:56 UTC >> key usage: digitalSignature,nonRepudiation,keyCertSign,cRLSign >> eku: id-kp-OCSPSigning >> pre-save command: /usr/lib64/ipa/certmonger/stop_pkicad >> post-save command: /usr/lib64/ipa/certmonger/restart_pkicad >> "ocspSigningCert cert-pki-ca" >> track: yes >> auto-renew: yes >> Request ID '20160927191454': >> status: CA_WORKING >> stuck: no >> key pair storage: >> type=NSSDB,location='/var/lib/pki-ca/alias',nickname='subsystemCert >> cert-pki-ca',token='NSS Certificate DB',pin set >> certificate: >> type=NSSDB,location='/var/lib/pki-ca/alias',nickname='subsystemCert >> cert-pki-ca',token='NSS Certificate DB' >> CA: dogtag-ipa-retrieve-agent-submit >> issuer: CN=Certificate Authority,O=PLACEIQ.NET >> subject: CN=CA Subsystem,O=PLACEIQ.NET >> expires: 2015-12-03 21:57:56 UTC >> key usage: >> digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment >> eku: id-kp-serverAuth,id-kp-clientAuth >> pre-save command: /usr/lib64/ipa/certmonger/stop_pkicad >> post-save command: /usr/lib64/ipa/certmonger/restart_pkicad >> "subsystemCert cert-pki-ca" >> track: yes >> auto-renew: yes >> Request ID '20160927191455': >> status: MONITORING >> stuck: no >> key pair storage: >> type=NSSDB,location='/var/lib/pki-ca/alias',nickname='Server-Cert >> cert-pki-ca',token='NSS Certificate DB',pin set >> certificate: >> type=NSSDB,location='/var/lib/pki-ca/alias',nickname='Server-Cert >> cert-pki-ca',token='NSS Certificate DB' >> CA: dogtag-ipa-renew-agent >> issuer: CN=Certificate Authority,O=PLACEIQ.NET >> subject: CN=sso-108.nym1.placeiq.net >> ,O=PLACEIQ.NET >> expires: 2018-09-17 19:14:36 UTC >> key usage: >> digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment >> eku: id-kp-serverAuth >> pre-save command: >> post-save command: >> track: yes >> auto-renew: yes >> Request ID '20160927191540': >> status: MONITORING >> stuck: no >> key pair storage: >> type=NSSDB,location='/etc/dirsrv/slapd-PLACEIQ-NET',nickname='Server-Cert',token='NSS >> Certificate DB',pinfile='/etc/dirsrv/slapd-PLACEIQ-NET >> /pwdfile.txt' >> certificate: >> type=NSSDB,location='/etc/dirsrv/slapd-PLACEIQ-NET',nickname='Server-Cert',token='NSS >> Certificate DB' >> CA: IPA >> issuer: CN=Certificate Authority,O=PLACEIQ.NET >> subject: CN=sso-108.nym1.placeiq.net >> ,O=PLACEIQ.NET >> expires: 2018-09-28 19:10:32 UTC >> key usage: >> digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment >> eku: id-kp-serverAuth,id-kp-clientAuth >> pre-save command: >> post-save command: /usr/lib64/ipa/certmonger/restart_dirsrv >> PLACEIQ-NET >> track: yes >> auto-renew: yes >> Request ID '20160927192114': >> status: MONITORING >> stuck: no >> key pair storage: >> type=NSSDB,location='/etc/httpd/alias',nickname='Server-Cert',token='NSS >> Certificate DB',pinfile='/etc/httpd/alias/pwdfile.txt' >> certificate: >> type=NSSDB,location='/etc/httpd/alias',nickname='Server-Cert',token='NSS >> Certificate DB' >> CA: IPA >> issuer: CN=Certificate Authority,O=PLACEIQ.NET >> subject: CN=sso-108.nym1.placeiq.net >> ,O=PLACEIQ.NET >> expires: 2018-09-28 19:10:34 UTC >> key usage: >> digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment >> eku: id-kp-serverAuth,id-kp-clientAuth >> pre-save command: >> post-save command: /usr/lib64/ipa/certmonger/restart_httpd >> track: yes >> auto-renew: yes >> Request ID '20160927192146': >> status: MONITORING >> stuck: no >> key pair storage: >> type=NSSDB,location='/etc/httpd/alias',nickname='ipaCert',token='NSS >> Certificate DB',pinfile='/etc/httpd/alias/pwdfile.txt' >> certificate: >> type=NSSDB,location='/etc/httpd/alias',nickname='ipaCert',token='NSS >> Certificate DB' >> CA: dogtag-ipa-retrieve-agent-submit >> issuer: CN=Certificate Authority,O=PLACEIQ.NET >> subject: CN=IPA RA,O=PLACEIQ.NET >> expires: 2017-10-26 04:37:19 UTC >> key usage: >> digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment >> eku: id-kp-serverAuth,id-kp-clientAuth >> pre-save command: >> post-save command: /usr/lib64/ipa/certmonger/restart_httpd >> track: yes >> auto-renew: yes >> >> >> >> >> >> >> Jim Richard >> >> >> >> SYSTEM ADMINISTRATOR III >> /(646) 338-8905 / >> >> >> dvertising > -initiative-nai-as-100th-member/>PlaceIQ:Location >> Data Accuracy >> >> >> >> >> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From natxo.asenjo at gmail.com Thu Sep 29 07:25:27 2016 From: natxo.asenjo at gmail.com (Natxo Asenjo) Date: Thu, 29 Sep 2016 09:25:27 +0200 Subject: [Freeipa-users] Replica created with expired certs In-Reply-To: <17A0F251-F5DF-447C-AB64-C053E83369D0@placeiq.com> References: <5EE38BB7-ED9B-4001-8F64-DB7CD2319C02@placeiq.com> <57EBAD23.4090604@redhat.com> <17A0F251-F5DF-447C-AB64-C053E83369D0@placeiq.com> Message-ID: hi Jim, On Thu, Sep 29, 2016 at 7:37 AM, Jim Richard wrote: > Thanks Rob, that worked. > > Still on the subject of certs, any idea how to solve this error: > > Certificate format error: (SEC_ERROR_LEGACY_DATABASE) The certificate/key > database is in an old, unsupported format. > > I see that in the gui when querying hosts as well as from cli when I > ipa-show or ipa-find > I have had this too, and we did not find a solution (search my recent posts on the archives). As a workaround I have created replicas and decommissioned the older replicas. -- Groeten, natxo -------------- next part -------------- An HTML attachment was scrubbed... URL: From sbose at redhat.com Thu Sep 29 07:42:41 2016 From: sbose at redhat.com (Sumit Bose) Date: Thu, 29 Sep 2016 09:42:41 +0200 Subject: [Freeipa-users] SELinux errors with sssd-krb5-common-1.13.0-40.el7_2.12.x86_64 In-Reply-To: References: Message-ID: <20160929074241.GF24352@p.Speedport_W_724V_Typ_A_05011603_00_009> On Thu, Sep 29, 2016 at 12:47:34AM -0400, Prasun Gera wrote: > I started seeing some selinux errors on one of my RHEL 7 clients recently > (possibly after a recent yum update ?), which prevents users from logging > in with passwords. I've put SELinux in permissive mode for now. Logs follow This sounds like https://bugzilla.redhat.com/show_bug.cgi?id=1301686 . Would you mind adding your findings and the SSSD logs as described in https://bugzilla.redhat.com/show_bug.cgi?id=1301686#c2 to the bugzilla ticket. Thank you. bye, Sumit > > > SELinux is preventing /usr/libexec/sssd/krb5_child from read access on the > key Unknown. > > ***** Plugin catchall (100. confidence) suggests > ************************** > > If you believe that krb5_child should be allowed read access on the Unknown > key by default. > Then you should report this as a bug. > You can generate a local policy module to allow this access. > Do > allow this access for now by executing: > # grep krb5_child /var/log/audit/audit.log | audit2allow -M mypol > # semodule -i mypol.pp > > > Additional Information: > Source Context system_u:system_r:sssd_t:s0 > Target Context system_u:system_r:unconfined_service_t:s0 > Target Objects Unknown [ key ] > Source krb5_child > Source Path /usr/libexec/sssd/krb5_child > Port > Host > Source RPM Packages sssd-krb5-common-1.13.0-40.el7_2.12.x86_64 > Target RPM Packages > Policy RPM selinux-policy-3.13.1-60.el7_2.9.noarch > Selinux Enabled True > Policy Type targeted > Enforcing Mode Permissive > Host Name example.com > Platform Linux example.com 4.4.19-1.el7.x86_64 > #1 SMP Mon Aug 29 18:38:32 EDT 2016 x86_64 > x86_64 > Alert Count 38 > First Seen 2016-09-28 18:37:43 EDT > Last Seen 2016-09-28 22:08:41 EDT > Local ID aa5271fa-f708-46b0-a382-fb1f90ce8973 > Raw Audit Messages > type=AVC msg=audit(1475114921.376:90787): avc: denied { read } for > pid=8272 comm="krb5_child" scontext=system_u:system_r:sssd_t:s0 > tcontext=system_u:system_r:unconfined_service_t:s0 tclass=key permissive=0 > > > type=SYSCALL msg=audit(1475114921.376:90787): arch=x86_64 syscall=keyctl > success=yes exit=EINTR a0=b a1=333b5463 a2=0 a3=0 items=0 ppid=891 pid=8272 > auid=4294967295 uid=1388200053 gid=1388200053 euid=1388200053 > suid=1388200053 fsuid=1388200053 egid=1388200053 sgid=1388200053 > fsgid=1388200053 tty=(none) ses=4294967295 comm=krb5_child > exe=/usr/libexec/sssd/krb5_child subj=system_u:system_r:sssd_t:s0 key=(null) > > Hash: krb5_child,sssd_t,unconfined_service_t,key,read > > -------------------------------------------------------------------------------- > > SELinux is preventing /usr/libexec/sssd/krb5_child from view access on the > key Unknown. > > ***** Plugin catchall (100. confidence) suggests > ************************** > > If you believe that krb5_child should be allowed view access on the Unknown > key by default. > Then you should report this as a bug. > You can generate a local policy module to allow this access. > Do > allow this access for now by executing: > # grep krb5_child /var/log/audit/audit.log | audit2allow -M mypol > # semodule -i mypol.pp > > > Additional Information: > Source Context system_u:system_r:sssd_t:s0 > Target Context system_u:system_r:unconfined_service_t:s0 > Target Objects Unknown [ key ] > Source krb5_child > Source Path /usr/libexec/sssd/krb5_child > Port > Host > Source RPM Packages sssd-krb5-common-1.13.0-40.el7_2.12.x86_64 > Target RPM Packages > Policy RPM selinux-policy-3.13.1-60.el7_2.9.noarch > Selinux Enabled True > Policy Type targeted > Enforcing Mode Permissive > Host Name example.com > Platform Linux example.com 4.4.19-1.el7.x86_64 > #1 SMP Mon Aug 29 18:38:32 EDT 2016 x86_64 > x86_64 > Alert Count 10 > First Seen 2016-09-28 18:40:00 EDT > Last Seen 2016-09-28 22:08:41 EDT > Local ID 22ec0970-9447-444a-9631-69749e4e7226 > Raw Audit Messages > type=AVC msg=audit(1475114921.376:90789): avc: denied { view } for > pid=8272 comm="krb5_child" scontext=system_u:system_r:sssd_t:s0 > tcontext=system_u:system_r:unconfined_service_t:s0 tclass=key permissive=0 > > > type=SYSCALL msg=audit(1475114921.376:90789): arch=x86_64 syscall=keyctl > success=no exit=EACCES a0=6 a1=2e1c07f1 a2=0 a3=0 items=0 ppid=891 pid=8272 > auid=4294967295 uid=1388200053 gid=1388200053 euid=1388200053 > suid=1388200053 fsuid=1388200053 egid=1388200053 sgid=1388200053 > fsgid=1388200053 tty=(none) ses=4294967295 comm=krb5_child > exe=/usr/libexec/sssd/krb5_child subj=system_u:system_r:sssd_t:s0 key=(null) > > Hash: krb5_child,sssd_t,unconfined_service_t,key,view > > -------------------------------------------------------------------------------- > > SELinux is preventing /usr/libexec/sssd/krb5_child from write access on the > key Unknown. > > ***** Plugin catchall (100. confidence) suggests > ************************** > > If you believe that krb5_child should be allowed write access on the > Unknown key by default. > Then you should report this as a bug. > You can generate a local policy module to allow this access. > Do > allow this access for now by executing: > # grep krb5_child /var/log/audit/audit.log | audit2allow -M mypol > # semodule -i mypol.pp > > > Additional Information: > Source Context system_u:system_r:sssd_t:s0 > Target Context system_u:system_r:unconfined_service_t:s0 > Target Objects Unknown [ key ] > Source krb5_child > Source Path /usr/libexec/sssd/krb5_child > Port > Host > Source RPM Packages sssd-krb5-common-1.13.0-40.el7_2.12.x86_64 > Target RPM Packages > Policy RPM selinux-policy-3.13.1-60.el7_2.9.noarch > Selinux Enabled True > Policy Type targeted > Enforcing Mode Permissive > Host Name example.com > Platform Linux example.com 4.4.19-1.el7.x86_64 > #1 SMP Mon Aug 29 18:38:32 EDT 2016 x86_64 > x86_64 > Alert Count 10 > First Seen 2016-09-28 18:40:00 EDT > Last Seen 2016-09-28 22:08:41 EDT > Local ID 8982bbec-38db-485b-9266-57fdaa8a3621 > > Raw Audit Messages > type=AVC msg=audit(1475114921.376:90790): avc: denied { write } for > pid=8272 comm="krb5_child" scontext=system_u:system_r:sssd_t:s0 > tcontext=system_u:system_r:unconfined_service_t:s0 tclass=key permissive=0 > > type=SYSCALL msg=audit(1475114921.376:90790): arch=x86_64 syscall=add_key > success=no exit=EACCES a0=7f6987905ffc a1=7ffeed78b1f0 a2=0 a3=0 items=0 > ppid=891 pid=8272 auid=4294967295 uid=1388200053 gid=1388200053 > euid=1388200053 suid=1388200053 fsuid=1388200053 egid=1388200053 > sgid=1388200053 fsgid=1388200053 tty=(none) ses=4294967295 comm=krb5_child > exe=/usr/libexec/sssd/krb5_child subj=system_u:system_r:sssd_t:s0 key=(null) > > Hash: krb5_child,sssd_t,unconfined_service_t,key,write > -- > Manage your subscription for the Freeipa-users mailing list: > https://www.redhat.com/mailman/listinfo/freeipa-users > Go to http://freeipa.org for more info on the project From deepak_dimri at hotmail.com Thu Sep 29 08:22:03 2016 From: deepak_dimri at hotmail.com (Deepak Dimri) Date: Thu, 29 Sep 2016 08:22:03 +0000 Subject: [Freeipa-users] Sudo Rule not working Message-ID: Hi All, I have added sudo rule having allowed command for sudo su for a test user. When i login with this test user to my IPA client (ubuntu). I am getting a message that "the user is not in the sudoers file. This incident will be reported." and it works fine if i add the user to sudoers file then the user can switch to sudo and is able to run all the commands even the commands i have included in "deny" list in my IPA server. Do we need to have user/group added sudoers list for IPA sudo rule to work? if so then how can i make it work with IPA sudo rules? Thanks, Deepak -------------- next part -------------- An HTML attachment was scrubbed... URL: From jhrozek at redhat.com Thu Sep 29 08:30:34 2016 From: jhrozek at redhat.com (Jakub Hrozek) Date: Thu, 29 Sep 2016 10:30:34 +0200 Subject: [Freeipa-users] Sudo Rule not working In-Reply-To: References: Message-ID: <20160929083034.xztyvdiwzjkz5rmu@hendrix> On Thu, Sep 29, 2016 at 08:22:03AM +0000, Deepak Dimri wrote: > Hi All, > > I have added sudo rule having allowed command for sudo su for a test user. When i login with this test user to my IPA client (ubuntu). I am getting a message that "the user is not in the sudoers file. This incident will be reported." and it works fine if i add the user to sudoers file then the user can switch to sudo and is able to run all the commands even the commands i have included in "deny" list in my IPA server. > > > Do we need to have user/group added sudoers list for IPA sudo rule to work? if so then how can i make it work with IPA sudo rules? Please check out: https://fedorahosted.org/sssd/wiki/HOWTO_Troubleshoot_SUDO From detlev.habicht at ims.uni-hannover.de Thu Sep 29 08:58:34 2016 From: detlev.habicht at ims.uni-hannover.de (Detlev Habicht) Date: Thu, 29 Sep 2016 10:58:34 +0200 Subject: [Freeipa-users] Unspecified GSS failure: No credentials cache found Message-ID: <29BAC69D-7B96-4B5B-8026-EA44702A86F7@ims.uni-hannover.de> Hi all, based on the Red Hat docs i setup a Kerberized NFS Server with IPA and i course a lot of clients. The IPA services are running on an own host. The servers are running Scientic Linux and the clients Fedora. Samba and NFS is running well - i think. I see no problems. But i see a lot of this messages on the server and also on the clients: Sep 29 10:50:34 sorix gssproxy: gssproxy[1013]: (OID: { 1 2 840 113554 1 2 2 }) Unspecified GSS failure. Minor code may provide more information, No credentials cache found Sep 29 10:50:34 sorix gssproxy: gssproxy[1013]: (OID: { 1 2 840 113554 1 2 2 }) Unspecified GSS failure. Minor code may provide more information, No credentials cache found Sep 29 10:50:34 sorix gssproxy: gssproxy[1013]: (OID: { 1 2 840 113554 1 2 2 }) Unspecified GSS failure. Minor code may provide more information, No credentials cache found Sep 29 10:50:34 sorix gssproxy: gssproxy[1013]: (OID: { 1 2 840 113554 1 2 2 }) Unspecified GSS failure. Minor code may provide more information, No credentials cache found What is wrong? Thank you for any help! Detlev -- Detlev | Institut fuer Mikroelektronische Systeme Habicht | D-30167 Hannover +49 511 76219662 habicht at ims.uni-hannover.de --------+-------- Handy +49 172 5415752 --------------------------- -------------- next part -------------- An HTML attachment was scrubbed... URL: From beeth2006 at gmail.com Thu Sep 29 09:03:52 2016 From: beeth2006 at gmail.com (beeth beeth) Date: Thu, 29 Sep 2016 05:03:52 -0400 Subject: [Freeipa-users] Install IPA Servers with third-party certificate(external CA) Message-ID: I am trying to set up IPA servers with Verisign certificate, so that the Admin Web console can use public signed certificate to meet company's security requirement. But when I try to follow Red Hat's instructions at https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/7/html/Linux_Domain_Identity_Authentication_and_Policy_Guide/install-server.html#install-server-external-ca, 2.3.5. Installing a Server with an External CA as the Root CA, at the first step it says to generate CSR by adding the --external-ca option to the ipa-server-install utility, which does generate a CRS at /root/ipa.csr. However, the ipa-server-install command in fact doesn't ask for Distinguished Name (DN) or the organization info(like country, state, etc.), which are required in the CSR. Without a valid CSR file, I can't request for new Verisign certs. Did I miss something? Originally I once tried to change the default certificate for Apache(the Web Admin console) ONLY to the Verisign one, by adding the certificates to the /etc/httpd/alias database with the command: # ipa-server-certinstall -w --http_pin=test verisign.pk12 And updated the nss.conf for httpd, so that the new Nickname is used to point to the Verisign certs. That worked well for the website. However, the IPA client installation failed after that for the "ipa-client-install": ERROR Joining realm failed: libcurl failed to execute the HTTP POST transaction, explaining: Peer's certificate issuer has been marked as not trusted by the user. Even I tried to also update the certificate for the Directory service(ipa-server-certinstall -d ... ), the client installation still failed. I believe the new Verisign cert messed up the communication of the IPA components. Then I am thinking to install the IPA server from scratch with the Verisign cert, but then I hit the CSR problem described above. Please advise. Thanks! -------------- next part -------------- An HTML attachment was scrubbed... URL: From flo at redhat.com Thu Sep 29 09:25:34 2016 From: flo at redhat.com (Florence Blanc-Renaud) Date: Thu, 29 Sep 2016 11:25:34 +0200 Subject: [Freeipa-users] Install IPA Servers with third-party certificate(external CA) In-Reply-To: References: Message-ID: <08f285b8-0e59-4f3a-deae-479660a193c3@redhat.com> Hi, The instructions that you followed are used when you want to install FreeIPA with an embedded Certificate Authority (ie FreeIPA is able to issue certificates), and FreeIPA CA is signed by a 3rd party CA. Maybe your goal is just to use a 3rd party certificate for IPA's LDAP server and Web UI. In this case, you do not need to install FreeIPA with an embedded CA. You can follow the instructions for Installing without a CA [1], where you will need to provide a 3rd-part certificate. Hope this clarifies, Flo. [1] https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/7/html/Linux_Domain_Identity_Authentication_and_Policy_Guide/install-server.html#install-server-without-ca On 09/29/2016 11:03 AM, beeth beeth wrote: > I am trying to set up IPA servers with Verisign certificate, so that the > Admin Web console can use public signed certificate to meet company's > security requirement. But when I try to follow Red Hat's instructions at > https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/7/html/Linux_Domain_Identity_Authentication_and_Policy_Guide/install-server.html#install-server-external-ca, > > 2.3.5. Installing a Server with an External CA as the Root CA, > at the first step it says to generate CSR by adding the --external-ca > option to the ipa-server-install utility, which does generate a CRS at > /root/ipa.csr. However, the ipa-server-install command in fact doesn't > ask for Distinguished Name (DN) or the organization info(like country, > state, etc.), which are required in the CSR. Without a valid CSR file, I > can't request for new Verisign certs. Did I miss something? > > Originally I once tried to change the default certificate for Apache(the > Web Admin console) ONLY to the Verisign one, by adding the certificates > to the /etc/httpd/alias database with the command: > # ipa-server-certinstall -w --http_pin=test verisign.pk12 > And updated the nss.conf for httpd, so that the new Nickname is used to > point to the Verisign certs. That worked well for the website. However, > the IPA client installation failed after that for the "ipa-client-install": > > ERROR Joining realm failed: libcurl failed to execute the HTTP POST > transaction, explaining: Peer's certificate issuer has been marked as > not trusted by the user. > > Even I tried to also update the certificate for the Directory > service(ipa-server-certinstall -d ... ), the client installation still > failed. I believe the new Verisign cert messed up the communication of > the IPA components. Then I am thinking to install the IPA server from > scratch with the Verisign cert, but then I hit the CSR problem described > above. > > Please advise. Thanks! > > From beeth2006 at gmail.com Thu Sep 29 09:29:48 2016 From: beeth2006 at gmail.com (beeth beeth) Date: Thu, 29 Sep 2016 05:29:48 -0400 Subject: [Freeipa-users] Install IPA Servers with third-party certificate(external CA) In-Reply-To: References: Message-ID: Also, I once followed the instruction about "Using 3rd part certificates for HTTP/LDAP" at https://www.freeipa.org/page/Using_3rd_part_certificates_for_HTTP/LDAP, for my environment: IPA 4.2 on RHEL7 # ipa-cacert-manage -p DM_PASSWORD -n NICKNAME -t C,, install ca.crt # ipa-certupdate # ipa-server-certinstall -w -d mysite.key mysite.crt # systemctl restart httpd.service # systemctl restart dirsrv at MY-REALM.service It failed at the step to restart httpd.service. Thanks! On Thu, Sep 29, 2016 at 5:03 AM, beeth beeth wrote: > I am trying to set up IPA servers with Verisign certificate, so that the > Admin Web console can use public signed certificate to meet company's > security requirement. But when I try to follow Red Hat's instructions at > https://access.redhat.com/documentation/en-US/Red_Hat_ > Enterprise_Linux/7/html/Linux_Domain_Identity_Authentication_and_Policy_ > Guide/install-server.html#install-server-external-ca, > 2.3.5. Installing a Server with an External CA as the Root CA, > at the first step it says to generate CSR by adding the --external-ca > option to the ipa-server-install utility, which does generate a CRS at > /root/ipa.csr. However, the ipa-server-install command in fact doesn't ask > for Distinguished Name (DN) or the organization info(like country, state, > etc.), which are required in the CSR. Without a valid CSR file, I can't > request for new Verisign certs. Did I miss something? > > Originally I once tried to change the default certificate for Apache(the > Web Admin console) ONLY to the Verisign one, by adding the certificates to > the /etc/httpd/alias database with the command: > # ipa-server-certinstall -w --http_pin=test verisign.pk12 > And updated the nss.conf for httpd, so that the new Nickname is used to > point to the Verisign certs. That worked well for the website. However, the > IPA client installation failed after that for the "ipa-client-install": > > ERROR Joining realm failed: libcurl failed to execute the HTTP POST > transaction, explaining: Peer's certificate issuer has been marked as not > trusted by the user. > > Even I tried to also update the certificate for the Directory > service(ipa-server-certinstall -d ... ), the client installation still > failed. I believe the new Verisign cert messed up the communication of the > IPA components. Then I am thinking to install the IPA server from scratch > with the Verisign cert, but then I hit the CSR problem described above. > > Please advise. Thanks! > -------------- next part -------------- An HTML attachment was scrubbed... URL: From beeth2006 at gmail.com Thu Sep 29 09:46:37 2016 From: beeth2006 at gmail.com (beeth beeth) Date: Thu, 29 Sep 2016 05:46:37 -0400 Subject: [Freeipa-users] Install IPA Servers with third-party certificate(external CA) In-Reply-To: <08f285b8-0e59-4f3a-deae-479660a193c3@redhat.com> References: <08f285b8-0e59-4f3a-deae-479660a193c3@redhat.com> Message-ID: Thanks for the quick response Florence! My goal is the use a 3rd party certificate(such as Verisign cert) for Web UI(company security requirement), in fact we are not required to use 3rd party certificate for the LDAP server, but as I mentioned earlier, I couldn't make the new Verisign cert to work with the Web UI, without messing up the IPA function(after I updated the nss.conf to use the new cert in the /etc/httpd/alias db, the ipa_client_install failed). So I tried to follow the Redhat instruction, to see if I can get the Verisign cert installed at the most beginning, without using FreeIPA's own/default certificate), but I got the CSR question. I did install IPA without a CA, by following the instruction at https://www.freeipa.org/page/Using_3rd_part_certificates_for_HTTP/LDAP, but failed to restart HTTPD. When and how can I provide the 3rd-party certificate? Could you please point me a document about the detail? Thanks again! On Thu, Sep 29, 2016 at 5:25 AM, Florence Blanc-Renaud wrote: > Hi, > > The instructions that you followed are used when you want to install > FreeIPA with an embedded Certificate Authority (ie FreeIPA is able to issue > certificates), and FreeIPA CA is signed by a 3rd party CA. > > Maybe your goal is just to use a 3rd party certificate for IPA's LDAP > server and Web UI. In this case, you do not need to install FreeIPA with an > embedded CA. You can follow the instructions for Installing without a CA > [1], where you will need to provide a 3rd-part certificate. > > Hope this clarifies, > Flo. > > [1] https://access.redhat.com/documentation/en-US/Red_Hat_Enterp > rise_Linux/7/html/Linux_Domain_Identity_Authentication_and_ > Policy_Guide/install-server.html#install-server-without-ca > > > > On 09/29/2016 11:03 AM, beeth beeth wrote: > >> I am trying to set up IPA servers with Verisign certificate, so that the >> Admin Web console can use public signed certificate to meet company's >> security requirement. But when I try to follow Red Hat's instructions at >> https://access.redhat.com/documentation/en-US/Red_Hat_Enterp >> rise_Linux/7/html/Linux_Domain_Identity_Authentication_and_ >> Policy_Guide/install-server.html#install-server-external-ca, >> >> 2.3.5. Installing a Server with an External CA as the Root CA, >> at the first step it says to generate CSR by adding the --external-ca >> option to the ipa-server-install utility, which does generate a CRS at >> /root/ipa.csr. However, the ipa-server-install command in fact doesn't >> ask for Distinguished Name (DN) or the organization info(like country, >> state, etc.), which are required in the CSR. Without a valid CSR file, I >> can't request for new Verisign certs. Did I miss something? >> >> Originally I once tried to change the default certificate for Apache(the >> Web Admin console) ONLY to the Verisign one, by adding the certificates >> to the /etc/httpd/alias database with the command: >> # ipa-server-certinstall -w --http_pin=test verisign.pk12 >> And updated the nss.conf for httpd, so that the new Nickname is used to >> point to the Verisign certs. That worked well for the website. However, >> the IPA client installation failed after that for the >> "ipa-client-install": >> >> ERROR Joining realm failed: libcurl failed to execute the HTTP POST >> transaction, explaining: Peer's certificate issuer has been marked as >> not trusted by the user. >> >> Even I tried to also update the certificate for the Directory >> service(ipa-server-certinstall -d ... ), the client installation still >> failed. I believe the new Verisign cert messed up the communication of >> the IPA components. Then I am thinking to install the IPA server from >> scratch with the Verisign cert, but then I hit the CSR problem described >> above. >> >> Please advise. Thanks! >> >> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From beeth2006 at gmail.com Thu Sep 29 09:56:00 2016 From: beeth2006 at gmail.com (beeth beeth) Date: Thu, 29 Sep 2016 05:56:00 -0400 Subject: [Freeipa-users] Install IPA Servers with third-party certificate(external CA) In-Reply-To: <08f285b8-0e59-4f3a-deae-479660a193c3@redhat.com> References: <08f285b8-0e59-4f3a-deae-479660a193c3@redhat.com> Message-ID: Ok, I will try out the "2.3.6. Installing Without a CA", and keep you posted. BTW, I noticed that the key needs to be encrypted, is that true? Thanks! On Thu, Sep 29, 2016 at 5:25 AM, Florence Blanc-Renaud wrote: > Hi, > > The instructions that you followed are used when you want to install > FreeIPA with an embedded Certificate Authority (ie FreeIPA is able to issue > certificates), and FreeIPA CA is signed by a 3rd party CA. > > Maybe your goal is just to use a 3rd party certificate for IPA's LDAP > server and Web UI. In this case, you do not need to install FreeIPA with an > embedded CA. You can follow the instructions for Installing without a CA > [1], where you will need to provide a 3rd-part certificate. > > Hope this clarifies, > Flo. > > [1] https://access.redhat.com/documentation/en-US/Red_Hat_Enterp > rise_Linux/7/html/Linux_Domain_Identity_Authentication_and_ > Policy_Guide/install-server.html#install-server-without-ca > > > > On 09/29/2016 11:03 AM, beeth beeth wrote: > >> I am trying to set up IPA servers with Verisign certificate, so that the >> Admin Web console can use public signed certificate to meet company's >> security requirement. But when I try to follow Red Hat's instructions at >> https://access.redhat.com/documentation/en-US/Red_Hat_Enterp >> rise_Linux/7/html/Linux_Domain_Identity_Authentication_and_ >> Policy_Guide/install-server.html#install-server-external-ca, >> >> 2.3.5. Installing a Server with an External CA as the Root CA, >> at the first step it says to generate CSR by adding the --external-ca >> option to the ipa-server-install utility, which does generate a CRS at >> /root/ipa.csr. However, the ipa-server-install command in fact doesn't >> ask for Distinguished Name (DN) or the organization info(like country, >> state, etc.), which are required in the CSR. Without a valid CSR file, I >> can't request for new Verisign certs. Did I miss something? >> >> Originally I once tried to change the default certificate for Apache(the >> Web Admin console) ONLY to the Verisign one, by adding the certificates >> to the /etc/httpd/alias database with the command: >> # ipa-server-certinstall -w --http_pin=test verisign.pk12 >> And updated the nss.conf for httpd, so that the new Nickname is used to >> point to the Verisign certs. That worked well for the website. However, >> the IPA client installation failed after that for the >> "ipa-client-install": >> >> ERROR Joining realm failed: libcurl failed to execute the HTTP POST >> transaction, explaining: Peer's certificate issuer has been marked as >> not trusted by the user. >> >> Even I tried to also update the certificate for the Directory >> service(ipa-server-certinstall -d ... ), the client installation still >> failed. I believe the new Verisign cert messed up the communication of >> the IPA components. Then I am thinking to install the IPA server from >> scratch with the Verisign cert, but then I hit the CSR problem described >> above. >> >> Please advise. Thanks! >> >> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From flo at redhat.com Thu Sep 29 10:03:52 2016 From: flo at redhat.com (Florence Blanc-Renaud) Date: Thu, 29 Sep 2016 12:03:52 +0200 Subject: [Freeipa-users] Install IPA Servers with third-party certificate(external CA) In-Reply-To: References: <08f285b8-0e59-4f3a-deae-479660a193c3@redhat.com> Message-ID: On 09/29/2016 11:43 AM, beeth beeth wrote: > Thanks for the quick response Florence! > > My goal is the use a 3rd party certificate(such as Verisign cert) for > Web UI(company security requirement), in fact we are not required to use > 3rd party certificate for the LDAP server, but as I mentioned earlier, I > couldn't make the new Verisign cert to work with the Web UI, without > messing up the IPA function(after I updated the nss.conf to use the new > cert in the /etc/httpd/alias db, the ipa_client_install failed). So I > tried to follow the Redhat instruction, to see if I can get the Verisign > cert installed at the most beginning, without using FreeIPA's > own/default certificate), but I got the CSR question. > > I did install IPA without a CA, by following the instruction at > https://www.freeipa.org/page/Using_3rd_part_certificates_for_HTTP/LDAP, > but failed to restart HTTPD. When and how can I provide the 3rd-party > certificate? Could you please point me a document about the detail? Hi, you need first to clarify if you want FreeIPA to act as a CA or not. The setup will depend on this choice. - option a) FreeIPA with an embedded CA: you can install FreeIPA with a self-signed CA, then follow the instructions at https://www.freeipa.org/page/Using_3rd_part_certificates_for_HTTP/LDAP in order to replace the WebUI certificate. Please note that there were some bugs in ipa-server-certinstall, preventing httpd from starting (Ticket #4786 [1]). The workaround is to manually update nss.conf (as you did) and manually import the CA certificate into /etc/pki/pki-tomcat/alias, for instance with $ certutil -A -d /etc/pki/pki-tomcat/alias -i cacert.pem -n nickname -t C,, - option b) Free IPA without CA the installation instructions are in Installing without a CA [2]. You will provide the certificate that will be used by both the LDAP server and the WebUI in the command options. HTH, Flo. [1] https://fedorahosted.org/freeipa/ticket/4786 [2] https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/7/html/Linux_Domain_Identity_Authentication_and_Policy_Guide/install-server.html#install-server-without-ca > Thanks again! > > > On Thu, Sep 29, 2016 at 5:25 AM, Florence Blanc-Renaud > wrote: > > Hi, > > The instructions that you followed are used when you want to install > FreeIPA with an embedded Certificate Authority (ie FreeIPA is able > to issue certificates), and FreeIPA CA is signed by a 3rd party CA. > > Maybe your goal is just to use a 3rd party certificate for IPA's > LDAP server and Web UI. In this case, you do not need to install > FreeIPA with an embedded CA. You can follow the instructions for > Installing without a CA [1], where you will need to provide a > 3rd-part certificate. > > Hope this clarifies, > Flo. > > [1] > https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/7/html/Linux_Domain_Identity_Authentication_and_Policy_Guide/install-server.html#install-server-without-ca > > > > > On 09/29/2016 11:03 AM, beeth beeth wrote: > > I am trying to set up IPA servers with Verisign certificate, so > that the > Admin Web console can use public signed certificate to meet > company's > security requirement. But when I try to follow Red Hat's > instructions at > https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/7/html/Linux_Domain_Identity_Authentication_and_Policy_Guide/install-server.html#install-server-external-ca > , > > 2.3.5. Installing a Server with an External CA as the Root CA, > at the first step it says to generate CSR by adding the > --external-ca > option to the ipa-server-install utility, which does generate a > CRS at > /root/ipa.csr. However, the ipa-server-install command in fact > doesn't > ask for Distinguished Name (DN) or the organization info(like > country, > state, etc.), which are required in the CSR. Without a valid CSR > file, I > can't request for new Verisign certs. Did I miss something? > > Originally I once tried to change the default certificate for > Apache(the > Web Admin console) ONLY to the Verisign one, by adding the > certificates > to the /etc/httpd/alias database with the command: > # ipa-server-certinstall -w --http_pin=test verisign.pk12 > And updated the nss.conf for httpd, so that the new Nickname is > used to > point to the Verisign certs. That worked well for the website. > However, > the IPA client installation failed after that for the > "ipa-client-install": > > ERROR Joining realm failed: libcurl failed to execute the HTTP POST > transaction, explaining: Peer's certificate issuer has been > marked as > not trusted by the user. > > Even I tried to also update the certificate for the Directory > service(ipa-server-certinstall -d ... ), the client installation > still > failed. I believe the new Verisign cert messed up the > communication of > the IPA components. Then I am thinking to install the IPA server > from > scratch with the Verisign cert, but then I hit the CSR problem > described > above. > > Please advise. Thanks! > > > > From deepak_dimri at hotmail.com Thu Sep 29 10:11:00 2016 From: deepak_dimri at hotmail.com (Deepak Dimri) Date: Thu, 29 Sep 2016 10:11:00 +0000 Subject: [Freeipa-users] Install IPA Servers with third-party certificate(external CA) In-Reply-To: References: <08f285b8-0e59-4f3a-deae-479660a193c3@redhat.com> , Message-ID: Thanks, Florence It works now.. my /etc/sssd/sssd.conf was missing with sudo service.. adding below line fixed the issue services = nss, sudo, pam, ssh" Many Thanks Again! Best Regards, Deepak ________________________________ From: freeipa-users-bounces at redhat.com on behalf of Florence Blanc-Renaud Sent: Thursday, September 29, 2016 6:03 AM To: beeth beeth Cc: Freeipa-users Subject: Re: [Freeipa-users] Install IPA Servers with third-party certificate(external CA) On 09/29/2016 11:43 AM, beeth beeth wrote: > Thanks for the quick response Florence! > > My goal is the use a 3rd party certificate(such as Verisign cert) for > Web UI(company security requirement), in fact we are not required to use > 3rd party certificate for the LDAP server, but as I mentioned earlier, I > couldn't make the new Verisign cert to work with the Web UI, without > messing up the IPA function(after I updated the nss.conf to use the new > cert in the /etc/httpd/alias db, the ipa_client_install failed). So I > tried to follow the Redhat instruction, to see if I can get the Verisign > cert installed at the most beginning, without using FreeIPA's > own/default certificate), but I got the CSR question. > > I did install IPA without a CA, by following the instruction at > https://www.freeipa.org/page/Using_3rd_part_certificates_for_HTTP/LDAP, Using 3rd part certificates for HTTP/LDAP - FreeIPA www.freeipa.org The following command will allow you to use a 3rd party certificate after initially deploying the FreeIPA system. You will need the following files: > but failed to restart HTTPD. When and how can I provide the 3rd-party > certificate? Could you please point me a document about the detail? Hi, you need first to clarify if you want FreeIPA to act as a CA or not. The setup will depend on this choice. - option a) FreeIPA with an embedded CA: you can install FreeIPA with a self-signed CA, then follow the instructions at https://www.freeipa.org/page/Using_3rd_part_certificates_for_HTTP/LDAP Using 3rd part certificates for HTTP/LDAP - FreeIPA www.freeipa.org The following command will allow you to use a 3rd party certificate after initially deploying the FreeIPA system. You will need the following files: in order to replace the WebUI certificate. Please note that there were some bugs in ipa-server-certinstall, preventing httpd from starting (Ticket #4786 [1]). The workaround is to manually update nss.conf (as you did) and manually import the CA certificate into /etc/pki/pki-tomcat/alias, for instance with $ certutil -A -d /etc/pki/pki-tomcat/alias -i cacert.pem -n nickname -t C,, - option b) Free IPA without CA the installation instructions are in Installing without a CA [2]. You will provide the certificate that will be used by both the LDAP server and the WebUI in the command options. HTH, Flo. [1] https://fedorahosted.org/freeipa/ticket/4786 [2] https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/7/html/Linux_Domain_Identity_Authentication_and_Policy_Guide/install-server.html#install-server-without-ca > Thanks again! > > > On Thu, Sep 29, 2016 at 5:25 AM, Florence Blanc-Renaud > wrote: > > Hi, > > The instructions that you followed are used when you want to install > FreeIPA with an embedded Certificate Authority (ie FreeIPA is able > to issue certificates), and FreeIPA CA is signed by a 3rd party CA. > > Maybe your goal is just to use a 3rd party certificate for IPA's > LDAP server and Web UI. In this case, you do not need to install > FreeIPA with an embedded CA. You can follow the instructions for > Installing without a CA [1], where you will need to provide a > 3rd-part certificate. > > Hope this clarifies, > Flo. > > [1] > https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/7/html/Linux_Domain_Identity_Authentication_and_Policy_Guide/install-server.html#install-server-without-ca > > > > > On 09/29/2016 11:03 AM, beeth beeth wrote: > > I am trying to set up IPA servers with Verisign certificate, so > that the > Admin Web console can use public signed certificate to meet > company's > security requirement. But when I try to follow Red Hat's > instructions at > https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/7/html/Linux_Domain_Identity_Authentication_and_Policy_Guide/install-server.html#install-server-external-ca > , > > 2.3.5. Installing a Server with an External CA as the Root CA, > at the first step it says to generate CSR by adding the > --external-ca > option to the ipa-server-install utility, which does generate a > CRS at > /root/ipa.csr. However, the ipa-server-install command in fact > doesn't > ask for Distinguished Name (DN) or the organization info(like > country, > state, etc.), which are required in the CSR. Without a valid CSR > file, I > can't request for new Verisign certs. Did I miss something? > > Originally I once tried to change the default certificate for > Apache(the > Web Admin console) ONLY to the Verisign one, by adding the > certificates > to the /etc/httpd/alias database with the command: > # ipa-server-certinstall -w --http_pin=test verisign.pk12 > And updated the nss.conf for httpd, so that the new Nickname is > used to > point to the Verisign certs. That worked well for the website. > However, > the IPA client installation failed after that for the > "ipa-client-install": > > ERROR Joining realm failed: libcurl failed to execute the HTTP POST > transaction, explaining: Peer's certificate issuer has been > marked as > not trusted by the user. > > Even I tried to also update the certificate for the Directory > service(ipa-server-certinstall -d ... ), the client installation > still > failed. I believe the new Verisign cert messed up the > communication of > the IPA components. Then I am thinking to install the IPA server > from > scratch with the Verisign cert, but then I hit the CSR problem > described > above. > > Please advise. Thanks! > > > > -- Manage your subscription for the Freeipa-users mailing list: https://www.redhat.com/mailman/listinfo/freeipa-users Go to http://freeipa.org for more info on the project -------------- next part -------------- An HTML attachment was scrubbed... URL: From natxo.asenjo at gmail.com Thu Sep 29 10:30:06 2016 From: natxo.asenjo at gmail.com (Natxo Asenjo) Date: Thu, 29 Sep 2016 12:30:06 +0200 Subject: [Freeipa-users] another certmonger question In-Reply-To: <57EA5B17.8000307@redhat.com> References: <57EA5B17.8000307@redhat.com> Message-ID: On Tue, Sep 27, 2016 at 1:42 PM, Rob Crittenden wrote: > > It's hard to say, it may in fact not be a problem. > > It is really a matter of what service the certificate(s) are related to. > I'd look at the serial numbers and then correlate those to the issued > certificates. > > I'd also do a service-find on the hostname to see if any services have > certificates issued and with what serial numbers. > I agree, it could be that. But just for testing I have created a vm, joined it to the domain and resubmitted the certificate. Now there are two valid host certificates with the same subject: $ ipa cert-find --subject=throwaway.unix.iriszorg.nl ---------------------- 2 certificates matched ---------------------- Serial number (hex): 0x3FFE0002 Serial number: 1073610754 Status: VALID Subject: CN=throwaway.unix.iriszorg.nl,O=UNIX.IRISZORG.NL Serial number (hex): 0x3FFE0003 Serial number: 1073610755 Status: VALID Subject: CN=throwaway.unix.iriszorg.nl,O=UNIX.IRISZORG.NL ---------------------------- Number of entries returned 2 ---------------------------- So it certmonger in this centos 6.8 32bit host is renewing but not having the old certificate revoked. -- Groeten, natxo -------------- next part -------------- An HTML attachment was scrubbed... URL: From rcritten at redhat.com Thu Sep 29 11:16:30 2016 From: rcritten at redhat.com (Rob Crittenden) Date: Thu, 29 Sep 2016 13:16:30 +0200 Subject: [Freeipa-users] another certmonger question In-Reply-To: References: <57EA5B17.8000307@redhat.com> Message-ID: <57ECF80E.6010406@redhat.com> Natxo Asenjo wrote: > > > On Tue, Sep 27, 2016 at 1:42 PM, Rob Crittenden > wrote: > > > It's hard to say, it may in fact not be a problem. > > It is really a matter of what service the certificate(s) are related > to. I'd look at the serial numbers and then correlate those to the > issued certificates. > > I'd also do a service-find on the hostname to see if any services > have certificates issued and with what serial numbers. > > > I agree, it could be that. But just for testing I have created a vm, > joined it to the domain and resubmitted the certificate. > > Now there are two valid host certificates with the same subject: > > > $ ipa cert-find --subject=throwaway.unix.iriszorg.nl > > ---------------------- > 2 certificates matched > ---------------------- > Serial number (hex): 0x3FFE0002 > Serial number: 1073610754 > Status: VALID > Subject: CN=throwaway.unix.iriszorg.nl > ,O=UNIX.IRISZORG.NL > > > Serial number (hex): 0x3FFE0003 > Serial number: 1073610755 > Status: VALID > Subject: CN=throwaway.unix.iriszorg.nl > ,O=UNIX.IRISZORG.NL > > ---------------------------- > Number of entries returned 2 > ---------------------------- > > > So it certmonger in this centos 6.8 32bit host is renewing but not > having the old certificate revoked. I'd check the Apache log to find the cert_request call to see if you can see if there are any issues raised. It should be doing a cert_revoke at the same time. Can you should how this certificate is being tracked? rob From beeth2006 at gmail.com Thu Sep 29 11:26:10 2016 From: beeth2006 at gmail.com (beeth beeth) Date: Thu, 29 Sep 2016 07:26:10 -0400 Subject: [Freeipa-users] Install IPA Servers with third-party certificate(external CA) In-Reply-To: References: <08f285b8-0e59-4f3a-deae-479660a193c3@redhat.com> Message-ID: Hi Florence, I previously tried option a) and failed(need to find out why later), but I was able to successfully reinstall the server and the client with option b), thanks a lot! So when it says "Installing Without a CA", it means without a "embeded CA"(the IPA's own CA), is that right? Another main problem comes up for option b): now I am going to install the replica server(ipa2), if I do the same as I did before: [root at ipa1 ~]# ipa-replica-prepare ipa2.example.com copy the gpg file from ipa1 to ipa2 [root at ipa2 ~]# ipa-replica-install /var/lib/ipa/replica-info-ipa2.example.com.gpg Then I believe the Apache on ipa2(the replica server) will use the Verisign certificate with the same hostname(DN): ipa1.example.com, NOT ipa2.example.com, hence the users who visit https://ipa2.example.com will experience security warning from the browser, as expected... What could be a solution for this? Thanks again! On Thu, Sep 29, 2016 at 6:03 AM, Florence Blanc-Renaud wrote: > On 09/29/2016 11:43 AM, beeth beeth wrote: > >> Thanks for the quick response Florence! >> >> My goal is the use a 3rd party certificate(such as Verisign cert) for >> Web UI(company security requirement), in fact we are not required to use >> 3rd party certificate for the LDAP server, but as I mentioned earlier, I >> couldn't make the new Verisign cert to work with the Web UI, without >> messing up the IPA function(after I updated the nss.conf to use the new >> cert in the /etc/httpd/alias db, the ipa_client_install failed). So I >> tried to follow the Redhat instruction, to see if I can get the Verisign >> cert installed at the most beginning, without using FreeIPA's >> own/default certificate), but I got the CSR question. >> >> I did install IPA without a CA, by following the instruction at >> https://www.freeipa.org/page/Using_3rd_part_certificates_for_HTTP/LDAP, >> but failed to restart HTTPD. When and how can I provide the 3rd-party >> certificate? Could you please point me a document about the detail? >> > Hi, > > you need first to clarify if you want FreeIPA to act as a CA or not. The > setup will depend on this choice. > > - option a) FreeIPA with an embedded CA: > you can install FreeIPA with a self-signed CA, then follow the > instructions at https://www.freeipa.org/page/U > sing_3rd_part_certificates_for_HTTP/LDAP in order to replace the WebUI > certificate. Please note that there were some bugs in > ipa-server-certinstall, preventing httpd from starting (Ticket #4786 [1]). > The workaround is to manually update nss.conf (as you did) and manually > import the CA certificate into /etc/pki/pki-tomcat/alias, for instance with > $ certutil -A -d /etc/pki/pki-tomcat/alias -i cacert.pem -n nickname -t C,, > > > - option b) Free IPA without CA > the installation instructions are in Installing without a CA [2]. You will > provide the certificate that will be used by both the LDAP server and the > WebUI in the command options. > > HTH, > Flo. > > [1] https://fedorahosted.org/freeipa/ticket/4786 > [2] https://access.redhat.com/documentation/en-US/Red_Hat_Enterp > rise_Linux/7/html/Linux_Domain_Identity_Authentication_and_ > Policy_Guide/install-server.html#install-server-without-ca > -------------- next part -------------- An HTML attachment was scrubbed... URL: From rcritten at redhat.com Thu Sep 29 12:11:11 2016 From: rcritten at redhat.com (Rob Crittenden) Date: Thu, 29 Sep 2016 14:11:11 +0200 Subject: [Freeipa-users] Replica created with expired certs In-Reply-To: References: <5EE38BB7-ED9B-4001-8F64-DB7CD2319C02@placeiq.com> <57EBAD23.4090604@redhat.com> <17A0F251-F5DF-447C-AB64-C053E83369D0@placeiq.com> Message-ID: <57ED04DF.2040707@redhat.com> Natxo Asenjo wrote: > hi Jim, > > On Thu, Sep 29, 2016 at 7:37 AM, Jim Richard > wrote: > > Thanks Rob, that worked. > > Still on the subject of certs, any idea how to solve this error: > > Certificate format error: (SEC_ERROR_LEGACY_DATABASE) The > certificate/key database is in an old, unsupported format. > > I see that in the gui when querying hosts as well as from cli when I > ipa-show or ipa-find > > > I have had this too, and we did not find a solution (search my recent > posts on the archives). As a workaround I have created replicas and > decommissioned the older replicas. On the one hand I'm glad this fixed it for you. On the other it is a rather unsatisfying answer. Unfortunately NSS doesn't always provide the most context with its error messages. This error is usually seen when one tries to open a non-existent database, which in this case is a very strange thing, especially since it goes from working to non-working in the same apache process over a few minutes. I'm not sure how I'd troubleshoot this if it were easily reproducible. I suspect we'd need to figure out which database cannot be found (most likely /etc/httpd/alias) and go from there. An strace is a brute-force way to see the file open but finding the right process to attach to is a bit of an art. rob From rcritten at redhat.com Thu Sep 29 12:12:52 2016 From: rcritten at redhat.com (Rob Crittenden) Date: Thu, 29 Sep 2016 14:12:52 +0200 Subject: [Freeipa-users] Install IPA Servers with third-party certificate(external CA) In-Reply-To: References: <08f285b8-0e59-4f3a-deae-479660a193c3@redhat.com> Message-ID: <57ED0544.9090503@redhat.com> beeth beeth wrote: > Hi Florence, > > I previously tried option a) and failed(need to find out why later), but > I was able to successfully reinstall the server and the client with > option b), thanks a lot! So when it says "Installing Without a CA", it > means without a "embeded CA"(the IPA's own CA), is that right? > > Another main problem comes up for option b): now I am going to install > the replica server(ipa2), if I do the same as I did before: > > [root at ipa1 ~]# ipa-replica-prepare ipa2.example.com > > > copy the gpg file from ipa1 to ipa2 > > [root at ipa2 ~]# ipa-replica-install > /var/lib/ipa/replica-info-ipa2.example.com.gpg > > Then I believe the Apache on ipa2(the replica server) will use the > Verisign certificate with the same hostname(DN): ipa1.example.com > , NOT ipa2.example.com > , hence the users who visit > https://ipa2.example.com will experience security warning from the > browser, as expected... > What could be a solution for this? > > Thanks again! > > > On Thu, Sep 29, 2016 at 6:03 AM, Florence Blanc-Renaud > wrote: > > On 09/29/2016 11:43 AM, beeth beeth wrote: > > Thanks for the quick response Florence! > > My goal is the use a 3rd party certificate(such as Verisign > cert) for > Web UI(company security requirement), in fact we are not > required to use > 3rd party certificate for the LDAP server, but as I mentioned > earlier, I > couldn't make the new Verisign cert to work with the Web UI, without > messing up the IPA function(after I updated the nss.conf to use > the new > cert in the /etc/httpd/alias db, the ipa_client_install failed). > So I > tried to follow the Redhat instruction, to see if I can get the > Verisign > cert installed at the most beginning, without using FreeIPA's > own/default certificate), but I got the CSR question. > > I did install IPA without a CA, by following the instruction at > https://www.freeipa.org/page/Using_3rd_part_certificates_for_HTTP/LDAP > , > but failed to restart HTTPD. When and how can I provide the > 3rd-party > certificate? Could you please point me a document about the detail? > > Hi, > > you need first to clarify if you want FreeIPA to act as a CA or not. > The setup will depend on this choice. > > - option a) FreeIPA with an embedded CA: > you can install FreeIPA with a self-signed CA, then follow the > instructions at > https://www.freeipa.org/page/Using_3rd_part_certificates_for_HTTP/LDAP > > in order to replace the WebUI certificate. Please note that there > were some bugs in ipa-server-certinstall, preventing httpd from > starting (Ticket #4786 [1]). The workaround is to manually update > nss.conf (as you did) and manually import the CA certificate into > /etc/pki/pki-tomcat/alias, for instance with > $ certutil -A -d /etc/pki/pki-tomcat/alias -i cacert.pem -n nickname > -t C,, > > > - option b) Free IPA without CA > the installation instructions are in Installing without a CA [2]. > You will provide the certificate that will be used by both the LDAP > server and the WebUI in the command options. You'd need either a separate certificate or one with multiple subject alternative names, one for each master. I also imagine you'd need to provide this certificate at replica preparation time if you've installed without a CA. rob > > HTH, > Flo. > > [1] https://fedorahosted.org/freeipa/ticket/4786 > > [2] > https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/7/html/Linux_Domain_Identity_Authentication_and_Policy_Guide/install-server.html#install-server-without-ca > > > > From richard.harmonson at gmail.com Thu Sep 29 13:10:52 2016 From: richard.harmonson at gmail.com (Richard Harmonson) Date: Thu, 29 Sep 2016 06:10:52 -0700 Subject: [Freeipa-users] oVirt 3.6 and Fedora 24: How to change display resolution from 1024 x 768? Message-ID: I am unable to change the display for a Fedora 24 Workstation using Gnome 3.20 from its default 1024 x 768. I, also, tried a number of spins but the behavior persist. Installing on a physical desktop does not reproduce the symptom. Installing CentOS 7 does not reproduce the symptom. Under the virtual machine "Applications" it shows: kernel-4.7.4-200.fc24 ovirt-guest-agent-common-1.0.12-3.fc24 xorg-x11-drv-qxl-0.1.4-7.fc24 -------------- next part -------------- An HTML attachment was scrubbed... URL: From richard.harmonson at gmail.com Thu Sep 29 13:23:05 2016 From: richard.harmonson at gmail.com (Richard Harmonson) Date: Thu, 29 Sep 2016 06:23:05 -0700 Subject: [Freeipa-users] oVirt 3.6 and Fedora 24: How to change display resolution from 1024 x 768? In-Reply-To: References: Message-ID: Sorry folks! Sent to the wrong list. Please disregard. On Thu, Sep 29, 2016 at 6:10 AM, Richard Harmonson < richard.harmonson at gmail.com> wrote: > I am unable to change the display for a Fedora 24 Workstation using Gnome > 3.20 from its default 1024 x 768. I, also, tried a number of spins but the > behavior persist. Installing on a physical desktop does not reproduce the > symptom. Installing CentOS 7 does not reproduce the symptom. > > Under the virtual machine "Applications" it shows: > > kernel-4.7.4-200.fc24 > ovirt-guest-agent-common-1.0.12-3.fc24 > xorg-x11-drv-qxl-0.1.4-7.fc24 > -------------- next part -------------- An HTML attachment was scrubbed... URL: From flo at redhat.com Thu Sep 29 13:36:07 2016 From: flo at redhat.com (Florence Blanc-Renaud) Date: Thu, 29 Sep 2016 15:36:07 +0200 Subject: [Freeipa-users] Install IPA Servers with third-party certificate(external CA) In-Reply-To: <57ED0544.9090503@redhat.com> References: <08f285b8-0e59-4f3a-deae-479660a193c3@redhat.com> <57ED0544.9090503@redhat.com> Message-ID: On 09/29/2016 02:12 PM, Rob Crittenden wrote: > beeth beeth wrote: >> Hi Florence, >> >> I previously tried option a) and failed(need to find out why later), but >> I was able to successfully reinstall the server and the client with >> option b), thanks a lot! So when it says "Installing Without a CA", it >> means without a "embeded CA"(the IPA's own CA), is that right? >> >> Another main problem comes up for option b): now I am going to install >> the replica server(ipa2), if I do the same as I did before: >> >> [root at ipa1 ~]# ipa-replica-prepare ipa2.example.com >> >> >> copy the gpg file from ipa1 to ipa2 >> >> [root at ipa2 ~]# ipa-replica-install >> /var/lib/ipa/replica-info-ipa2.example.com.gpg >> >> Then I believe the Apache on ipa2(the replica server) will use the >> Verisign certificate with the same hostname(DN): ipa1.example.com >> , NOT ipa2.example.com >> , hence the users who visit >> https://ipa2.example.com will experience security warning from the >> browser, as expected... >> What could be a solution for this? >> >> Thanks again! >> >> >> On Thu, Sep 29, 2016 at 6:03 AM, Florence Blanc-Renaud > > wrote: >> >> On 09/29/2016 11:43 AM, beeth beeth wrote: >> >> Thanks for the quick response Florence! >> >> My goal is the use a 3rd party certificate(such as Verisign >> cert) for >> Web UI(company security requirement), in fact we are not >> required to use >> 3rd party certificate for the LDAP server, but as I mentioned >> earlier, I >> couldn't make the new Verisign cert to work with the Web UI, >> without >> messing up the IPA function(after I updated the nss.conf to use >> the new >> cert in the /etc/httpd/alias db, the ipa_client_install failed). >> So I >> tried to follow the Redhat instruction, to see if I can get the >> Verisign >> cert installed at the most beginning, without using FreeIPA's >> own/default certificate), but I got the CSR question. >> >> I did install IPA without a CA, by following the instruction at >> >> https://www.freeipa.org/page/Using_3rd_part_certificates_for_HTTP/LDAP >> >> , >> but failed to restart HTTPD. When and how can I provide the >> 3rd-party >> certificate? Could you please point me a document about the >> detail? >> >> Hi, >> >> you need first to clarify if you want FreeIPA to act as a CA or not. >> The setup will depend on this choice. >> >> - option a) FreeIPA with an embedded CA: >> you can install FreeIPA with a self-signed CA, then follow the >> instructions at >> >> https://www.freeipa.org/page/Using_3rd_part_certificates_for_HTTP/LDAP >> >> >> in order to replace the WebUI certificate. Please note that there >> were some bugs in ipa-server-certinstall, preventing httpd from >> starting (Ticket #4786 [1]). The workaround is to manually update >> nss.conf (as you did) and manually import the CA certificate into >> /etc/pki/pki-tomcat/alias, for instance with >> $ certutil -A -d /etc/pki/pki-tomcat/alias -i cacert.pem -n nickname >> -t C,, >> >> >> - option b) Free IPA without CA >> the installation instructions are in Installing without a CA [2]. >> You will provide the certificate that will be used by both the LDAP >> server and the WebUI in the command options. > > You'd need either a separate certificate or one with multiple subject > alternative names, one for each master. I also imagine you'd need to > provide this certificate at replica preparation time if you've installed > without a CA. > Yes, that's right. You can use the command ipa-replica-prepare with the options --dirsrv-cert-file / --dirsrv-pin and --http-cert-file / --http-pin to provide the replica's certificate and key. They will be embedded in the replica file and used during the replica installation. Flo. > rob > >> >> HTH, >> Flo. >> >> [1] https://fedorahosted.org/freeipa/ticket/4786 >> >> [2] >> >> https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/7/html/Linux_Domain_Identity_Authentication_and_Policy_Guide/install-server.html#install-server-without-ca >> >> >> >> >> >> >> > From jgoddard at emerlyn.com Thu Sep 29 13:39:29 2016 From: jgoddard at emerlyn.com (Jeff Goddard) Date: Thu, 29 Sep 2016 09:39:29 -0400 Subject: [Freeipa-users] Sudo Rule not working In-Reply-To: References: Message-ID: I had a similar issue. To see the details and solution search the list for: Re: [Freeipa-users] sudo rules question on ubuntu 16.0.1 Jeff On Thu, Sep 29, 2016 at 4:22 AM, Deepak Dimri wrote: > Hi All, > > I have added sudo rule having allowed command for sudo su for a test > user. When i login with this test user to my IPA client (ubuntu). I am > getting a message that "the user is not in the sudoers file. This > incident will be reported." and it works fine if i add the user to sudoers > file then the user can switch to sudo and is able to run all the commands > even the commands i have included in "deny" list in my IPA server. > > > Do we need to have user/group added sudoers list for IPA sudo rule to > work? if so then how can i make it work with IPA sudo rules? > > > Thanks, > > Deepak > > > > -- > Manage your subscription for the Freeipa-users mailing list: > https://www.redhat.com/mailman/listinfo/freeipa-users > Go to http://freeipa.org for more info on the project > -------------- next part -------------- An HTML attachment was scrubbed... URL: From pvomacka at redhat.com Thu Sep 29 15:58:37 2016 From: pvomacka at redhat.com (Pavel Vomacka) Date: Thu, 29 Sep 2016 17:58:37 +0200 Subject: [Freeipa-users] Certificate format error reported by GUI In-Reply-To: <14E35142-15BB-4108-BFC9-A7BB84CB60CF@placeiq.com> References: <14E35142-15BB-4108-BFC9-A7BB84CB60CF@placeiq.com> Message-ID: <1e2b525b-3aff-636f-ff84-e781299ce86f@redhat.com> Hello, which version of FreeIPA do you use? On 09/28/2016 12:42 AM, Jim Richard wrote: > When I try to look at hosts under the hosts tab. ipactl restart or > just restarting httpd seems to clear it up for a short period. > > Three replicas in the environment, it only happens when I look at > hosts using the GUI at one of the three replicas. > > > Certificate format error: (SEC_ERROR_LEGACY_DATABASE) The > certificate/key database is in an old, unsupported format. > > > Jim Richard > > > SYSTEM ADMINISTRATOR III > /(646) 338-8905 / > > > PlaceIQ:Location Data Accuracy > > > > > > -- Pavel^3 Vomacka -------------- next part -------------- An HTML attachment was scrubbed... URL: From prasun.gera at gmail.com Thu Sep 29 16:07:13 2016 From: prasun.gera at gmail.com (Prasun Gera) Date: Thu, 29 Sep 2016 12:07:13 -0400 Subject: [Freeipa-users] SELinux errors with sssd-krb5-common-1.13.0-40.el7_2.12.x86_64 In-Reply-To: <20160929074241.GF24352@p.Speedport_W_724V_Typ_A_05011603_00_009> References: <20160929074241.GF24352@p.Speedport_W_724V_Typ_A_05011603_00_009> Message-ID: I need to set SELinux to enforcing to get the relevant SSSD logs, right ? On Thu, Sep 29, 2016 at 3:42 AM, Sumit Bose wrote: > On Thu, Sep 29, 2016 at 12:47:34AM -0400, Prasun Gera wrote: > > I started seeing some selinux errors on one of my RHEL 7 clients recently > > (possibly after a recent yum update ?), which prevents users from logging > > in with passwords. I've put SELinux in permissive mode for now. Logs > follow > > This sounds like https://bugzilla.redhat.com/show_bug.cgi?id=1301686 . > Would you mind adding your findings and the SSSD logs as described in > https://bugzilla.redhat.com/show_bug.cgi?id=1301686#c2 to the bugzilla > ticket. > > Thank you. > > bye, > Sumit > > > > > > > SELinux is preventing /usr/libexec/sssd/krb5_child from read access on > the > > key Unknown. > > > > ***** Plugin catchall (100. confidence) suggests > > ************************** > > > > If you believe that krb5_child should be allowed read access on the > Unknown > > key by default. > > Then you should report this as a bug. > > You can generate a local policy module to allow this access. > > Do > > allow this access for now by executing: > > # grep krb5_child /var/log/audit/audit.log | audit2allow -M mypol > > # semodule -i mypol.pp > > > > > > Additional Information: > > Source Context system_u:system_r:sssd_t:s0 > > Target Context system_u:system_r:unconfined_service_t:s0 > > Target Objects Unknown [ key ] > > Source krb5_child > > Source Path /usr/libexec/sssd/krb5_child > > Port > > Host > > Source RPM Packages sssd-krb5-common-1.13.0-40.el7_2.12.x86_64 > > Target RPM Packages > > Policy RPM selinux-policy-3.13.1-60.el7_2.9.noarch > > Selinux Enabled True > > Policy Type targeted > > Enforcing Mode Permissive > > Host Name example.com > > Platform Linux example.com 4.4.19-1.el7.x86_64 > > #1 SMP Mon Aug 29 18:38:32 EDT 2016 x86_64 > > x86_64 > > Alert Count 38 > > First Seen 2016-09-28 18:37:43 EDT > > Last Seen 2016-09-28 22:08:41 EDT > > Local ID aa5271fa-f708-46b0-a382-fb1f90ce8973 > > Raw Audit Messages > > type=AVC msg=audit(1475114921.376:90787): avc: denied { read } for > > pid=8272 comm="krb5_child" scontext=system_u:system_r:sssd_t:s0 > > tcontext=system_u:system_r:unconfined_service_t:s0 tclass=key > permissive=0 > > > > > > type=SYSCALL msg=audit(1475114921.376:90787): arch=x86_64 syscall=keyctl > > success=yes exit=EINTR a0=b a1=333b5463 a2=0 a3=0 items=0 ppid=891 > pid=8272 > > auid=4294967295 uid=1388200053 gid=1388200053 euid=1388200053 > > suid=1388200053 fsuid=1388200053 egid=1388200053 sgid=1388200053 > > fsgid=1388200053 tty=(none) ses=4294967295 comm=krb5_child > > exe=/usr/libexec/sssd/krb5_child subj=system_u:system_r:sssd_t:s0 > key=(null) > > > > Hash: krb5_child,sssd_t,unconfined_service_t,key,read > > > > ------------------------------------------------------------ > -------------------- > > > > SELinux is preventing /usr/libexec/sssd/krb5_child from view access on > the > > key Unknown. > > > > ***** Plugin catchall (100. confidence) suggests > > ************************** > > > > If you believe that krb5_child should be allowed view access on the > Unknown > > key by default. > > Then you should report this as a bug. > > You can generate a local policy module to allow this access. > > Do > > allow this access for now by executing: > > # grep krb5_child /var/log/audit/audit.log | audit2allow -M mypol > > # semodule -i mypol.pp > > > > > > Additional Information: > > Source Context system_u:system_r:sssd_t:s0 > > Target Context system_u:system_r:unconfined_service_t:s0 > > Target Objects Unknown [ key ] > > Source krb5_child > > Source Path /usr/libexec/sssd/krb5_child > > Port > > Host > > Source RPM Packages sssd-krb5-common-1.13.0-40.el7_2.12.x86_64 > > Target RPM Packages > > Policy RPM selinux-policy-3.13.1-60.el7_2.9.noarch > > Selinux Enabled True > > Policy Type targeted > > Enforcing Mode Permissive > > Host Name example.com > > Platform Linux example.com 4.4.19-1.el7.x86_64 > > #1 SMP Mon Aug 29 18:38:32 EDT 2016 x86_64 > > x86_64 > > Alert Count 10 > > First Seen 2016-09-28 18:40:00 EDT > > Last Seen 2016-09-28 22:08:41 EDT > > Local ID 22ec0970-9447-444a-9631-69749e4e7226 > > Raw Audit Messages > > type=AVC msg=audit(1475114921.376:90789): avc: denied { view } for > > pid=8272 comm="krb5_child" scontext=system_u:system_r:sssd_t:s0 > > tcontext=system_u:system_r:unconfined_service_t:s0 tclass=key > permissive=0 > > > > > > type=SYSCALL msg=audit(1475114921.376:90789): arch=x86_64 syscall=keyctl > > success=no exit=EACCES a0=6 a1=2e1c07f1 a2=0 a3=0 items=0 ppid=891 > pid=8272 > > auid=4294967295 uid=1388200053 gid=1388200053 euid=1388200053 > > suid=1388200053 fsuid=1388200053 egid=1388200053 sgid=1388200053 > > fsgid=1388200053 tty=(none) ses=4294967295 comm=krb5_child > > exe=/usr/libexec/sssd/krb5_child subj=system_u:system_r:sssd_t:s0 > key=(null) > > > > Hash: krb5_child,sssd_t,unconfined_service_t,key,view > > > > ------------------------------------------------------------ > -------------------- > > > > SELinux is preventing /usr/libexec/sssd/krb5_child from write access on > the > > key Unknown. > > > > ***** Plugin catchall (100. confidence) suggests > > ************************** > > > > If you believe that krb5_child should be allowed write access on the > > Unknown key by default. > > Then you should report this as a bug. > > You can generate a local policy module to allow this access. > > Do > > allow this access for now by executing: > > # grep krb5_child /var/log/audit/audit.log | audit2allow -M mypol > > # semodule -i mypol.pp > > > > > > Additional Information: > > Source Context system_u:system_r:sssd_t:s0 > > Target Context system_u:system_r:unconfined_service_t:s0 > > Target Objects Unknown [ key ] > > Source krb5_child > > Source Path /usr/libexec/sssd/krb5_child > > Port > > Host > > Source RPM Packages sssd-krb5-common-1.13.0-40.el7_2.12.x86_64 > > Target RPM Packages > > Policy RPM selinux-policy-3.13.1-60.el7_2.9.noarch > > Selinux Enabled True > > Policy Type targeted > > Enforcing Mode Permissive > > Host Name example.com > > Platform Linux example.com 4.4.19-1.el7.x86_64 > > #1 SMP Mon Aug 29 18:38:32 EDT 2016 x86_64 > > x86_64 > > Alert Count 10 > > First Seen 2016-09-28 18:40:00 EDT > > Last Seen 2016-09-28 22:08:41 EDT > > Local ID 8982bbec-38db-485b-9266-57fdaa8a3621 > > > > Raw Audit Messages > > type=AVC msg=audit(1475114921.376:90790): avc: denied { write } for > > pid=8272 comm="krb5_child" scontext=system_u:system_r:sssd_t:s0 > > tcontext=system_u:system_r:unconfined_service_t:s0 tclass=key > permissive=0 > > > > type=SYSCALL msg=audit(1475114921.376:90790): arch=x86_64 > syscall=add_key > > success=no exit=EACCES a0=7f6987905ffc a1=7ffeed78b1f0 a2=0 a3=0 items=0 > > ppid=891 pid=8272 auid=4294967295 uid=1388200053 gid=1388200053 > > euid=1388200053 suid=1388200053 fsuid=1388200053 egid=1388200053 > > sgid=1388200053 fsgid=1388200053 tty=(none) ses=4294967295 > comm=krb5_child > > exe=/usr/libexec/sssd/krb5_child subj=system_u:system_r:sssd_t:s0 > key=(null) > > > > Hash: krb5_child,sssd_t,unconfined_service_t,key,write > > > -- > > Manage your subscription for the Freeipa-users mailing list: > > https://www.redhat.com/mailman/listinfo/freeipa-users > > Go to http://freeipa.org for more info on the project > > -- > Manage your subscription for the Freeipa-users mailing list: > https://www.redhat.com/mailman/listinfo/freeipa-users > Go to http://freeipa.org for more info on the project > -------------- next part -------------- An HTML attachment was scrubbed... URL: From julliot at ljll.math.upmc.fr Thu Sep 29 16:10:10 2016 From: julliot at ljll.math.upmc.fr (=?UTF-8?Q?S=c3=a9bastien_Julliot?=) Date: Thu, 29 Sep 2016 18:10:10 +0200 Subject: [Freeipa-users] Freeipa-users Digest, Vol 98, Issue 84 In-Reply-To: References: Message-ID: <576f1b89-e227-90a2-ccc0-4156ff643f88@ljll.math.upmc.fr> Hello everyone, I am trying to integrate a samba server over my freeipa install. For the moment, basics first, the samba server is on the same machine as freeipa (which fqdn is "freeipa2.ljll.math.upmc.fr"). Yet I am unable to make it work correctly following the official howto . /I must precise that I am running on an Ubuntu 16.04 server./ Here are the steps : 0) ipa-adtrust-install -> everything correct 1) installing required packages -> needed to change the names to "apt-get install freeipa-client libwbclient-sssd samba samba-client" but worked fine 2) |ipa-client-install --mkhomedir| -> as we are on the same server, ipa-client is already installed 3) |ipa service-add cifs/freeipa2.ljll.math.upmc.fr| || -> seems to be working fine, yet not doing it gives the exact same results later .. 4) |ipa-getkeytab -s freeipa2.ljll.math.upmc.fr -p cifs/freeipa2.ljll.math.upmc.fr -k /etc/samba/samba.keytab| | -> OK| |5) Editing /etc/samba/smb.conf| |6) enabling samba /home sharing -> no selinux here so nothing to do 7) restart samba -> OK | ||||After getting a kerberos ticket, `|smbclient -k -L freeipa2.ljll.math.upmc.fr` gives : | ||krb5_init_context failed (invalid argument) smb_krb5_context_init_basic failed (invalid argument) Failed to initialize kerberos context! (invalid argument) session setup failed: NT_STATUS_NO_MEMORY Editing /etc/samba/smb.conf to comment the `security = ads` line makes it more verbose: krb5_init_context failed (Argument invalide) smb_krb5_context_init_basic failed (Argument invalide) Domain=[LJLL] OS=[Windows 6.1] Server=[Samba 4.3.11-Ubuntu] Sharename Type Comment --------- ---- ------- krb5_init_context failed (Argument invalide) smb_krb5_context_init_basic failed (Argument invalide) print$ Disk Printer Drivers shared Disk IPC$ IPC IPC Service (freeipa2 server (Samba, Ubuntu)) krb5_init_context failed (Argument invalide) smb_krb5_context_init_basic failed (Argument invalide) Domain=[LJLL] OS=[Windows 6.1] Server=[Samba 4.3.11-Ubuntu] Server Comment --------- ------- FREEIPA2 freeipa2 server (Samba, Ubuntu) Workgroup Master --------- ------- LJLL Does anyone have ideas how to solve this ? Many thanks in advance, Sebastien. -------------- next part -------------- An HTML attachment was scrubbed... URL: From julliot at ljll.math.upmc.fr Thu Sep 29 16:15:44 2016 From: julliot at ljll.math.upmc.fr (=?UTF-8?Q?S=c3=a9bastien_Julliot?=) Date: Thu, 29 Sep 2016 18:15:44 +0200 Subject: [Freeipa-users] SAMBA + freeIPA over Ubuntu server In-Reply-To: <576f1b89-e227-90a2-ccc0-4156ff643f88@ljll.math.upmc.fr> References: <576f1b89-e227-90a2-ccc0-4156ff643f88@ljll.math.upmc.fr> Message-ID: <9d9ee2d4-4d99-ea00-3915-ca424c5f106f@ljll.math.upmc.fr> Hello everyone, I am trying to integrate a samba server over my freeipa install. For the moment, basics first, the samba server is on the same machine as freeipa (which fqdn is "freeipa2.ljll.math.upmc.fr"). Yet I am unable to make it work correctly following the official howto . /I must precise that I am running on an Ubuntu 16.04 server./ Here are the steps : 0) ipa-adtrust-install -> everything correct 1) installing required packages -> needed to change the names to "apt-get install freeipa-client libwbclient-sssd samba samba-client" but worked fine 2) |ipa-client-install --mkhomedir| -> as we are on the same server, ipa-client is already installed 3) |ipa service-add cifs/freeipa2.ljll.math.upmc.fr| || -> seems to be working fine, yet not doing it gives the exact same results later .. 4) |ipa-getkeytab -s freeipa2.ljll.math.upmc.fr -p cifs/freeipa2.ljll.math.upmc.fr -k /etc/samba/samba.keytab| | -> OK| |5) Editing /etc/samba/smb.conf| |6) enabling samba /home sharing -> no selinux here so nothing to do 7) restart samba -> OK | ||||After getting a kerberos ticket, `|smbclient -k -L freeipa2.ljll.math.upmc.fr` gives : | ||krb5_init_context failed (invalid argument) smb_krb5_context_init_basic failed (invalid argument) Failed to initialize kerberos context! (invalid argument) session setup failed: NT_STATUS_NO_MEMORY Editing /etc/samba/smb.conf to comment the `security = ads` line makes it more verbose: krb5_init_context failed (Argument invalide) smb_krb5_context_init_basic failed (Argument invalide) Domain=[LJLL] OS=[Windows 6.1] Server=[Samba 4.3.11-Ubuntu] Sharename Type Comment --------- ---- ------- krb5_init_context failed (Argument invalide) smb_krb5_context_init_basic failed (Argument invalide) print$ Disk Printer Drivers shared Disk IPC$ IPC IPC Service (freeipa2 server (Samba, Ubuntu)) krb5_init_context failed (Argument invalide) smb_krb5_context_init_basic failed (Argument invalide) Domain=[LJLL] OS=[Windows 6.1] Server=[Samba 4.3.11-Ubuntu] Server Comment --------- ------- FREEIPA2 freeipa2 server (Samba, Ubuntu) Workgroup Master --------- ------- LJLL Does anyone have ideas how to solve this ? Many thanks in advance, Sebastien. -------------- next part -------------- An HTML attachment was scrubbed... URL: From abokovoy at redhat.com Thu Sep 29 16:17:35 2016 From: abokovoy at redhat.com (Alexander Bokovoy) Date: Thu, 29 Sep 2016 19:17:35 +0300 Subject: [Freeipa-users] Freeipa-users Digest, Vol 98, Issue 84 In-Reply-To: <576f1b89-e227-90a2-ccc0-4156ff643f88@ljll.math.upmc.fr> References: <576f1b89-e227-90a2-ccc0-4156ff643f88@ljll.math.upmc.fr> Message-ID: <20160929161735.wdsj576mgdbn7eq6@redhat.com> On to, 29 syys 2016, S?bastien Julliot wrote: >Hello everyone, > > >I am trying to integrate a samba server over my freeipa install. For the >moment, basics first, > >the samba server is on the same machine as freeipa (which fqdn is >"freeipa2.ljll.math.upmc.fr"). > >Yet I am unable to make it work correctly following the official howto >. > >/I must precise that I am running on an Ubuntu 16.04 server./ > >Here are the steps : > >0) ipa-adtrust-install > > -> everything correct No, it is not. Read this: https://bugs.launchpad.net/ubuntu/+source/samba/+bug/1552249 All your errors are due to mix of Heimdal and MIT Kerberos libraries in the same process namespace. There is no solution for Ubuntu atm. > >1) installing required packages > > -> needed to change the names to "apt-get install freeipa-client libwbclient-sssd samba samba-client" but worked fine > >2) |ipa-client-install --mkhomedir| > > -> as we are on the same server, ipa-client is already installed > >3) |ipa service-add cifs/freeipa2.ljll.math.upmc.fr| > >|| > > -> seems to be working fine, yet not doing it gives the exact same results later .. > >4) |ipa-getkeytab -s freeipa2.ljll.math.upmc.fr -p >cifs/freeipa2.ljll.math.upmc.fr -k /etc/samba/samba.keytab| > >| -> OK| > >|5) Editing /etc/samba/smb.conf| > >|6) enabling samba /home sharing -> no selinux here so nothing to do 7) >restart samba -> OK | > >||||After getting a kerberos ticket, `|smbclient -k -L >freeipa2.ljll.math.upmc.fr` gives : > >| > >||krb5_init_context failed (invalid argument) > >smb_krb5_context_init_basic failed (invalid argument) > >Failed to initialize kerberos context! (invalid argument) > >session setup failed: NT_STATUS_NO_MEMORY > > > >Editing /etc/samba/smb.conf to comment the `security = ads` line makes >it more verbose: > >krb5_init_context failed (Argument invalide) > >smb_krb5_context_init_basic failed (Argument invalide) > >Domain=[LJLL] OS=[Windows 6.1] Server=[Samba 4.3.11-Ubuntu] > > Sharename Type Comment > > --------- ---- ------- > >krb5_init_context failed (Argument invalide) > >smb_krb5_context_init_basic failed (Argument invalide) > > print$ Disk Printer Drivers > > shared Disk > > IPC$ IPC IPC Service (freeipa2 server (Samba, Ubuntu)) > >krb5_init_context failed (Argument invalide) > >smb_krb5_context_init_basic failed (Argument invalide) > >Domain=[LJLL] OS=[Windows 6.1] Server=[Samba 4.3.11-Ubuntu] > > Server Comment > > --------- ------- > > FREEIPA2 freeipa2 server (Samba, Ubuntu) > > Workgroup Master > > --------- ------- > > LJLL > > >Does anyone have ideas how to solve this ? > >Many thanks in advance, >Sebastien. >-- >Manage your subscription for the Freeipa-users mailing list: >https://www.redhat.com/mailman/listinfo/freeipa-users >Go to http://freeipa.org for more info on the project -- / Alexander Bokovoy From natxo.asenjo at gmail.com Thu Sep 29 17:28:26 2016 From: natxo.asenjo at gmail.com (Natxo Asenjo) Date: Thu, 29 Sep 2016 19:28:26 +0200 Subject: [Freeipa-users] Replica created with expired certs In-Reply-To: <57ED04DF.2040707@redhat.com> References: <5EE38BB7-ED9B-4001-8F64-DB7CD2319C02@placeiq.com> <57EBAD23.4090604@redhat.com> <17A0F251-F5DF-447C-AB64-C053E83369D0@placeiq.com> <57ED04DF.2040707@redhat.com> Message-ID: hi, On Thu, Sep 29, 2016 at 2:11 PM, Rob Crittenden wrote: > Natxo Asenjo wrote: > >> hi Jim, >> >> On Thu, Sep 29, 2016 at 7:37 AM, Jim Richard > > wrote: >> >> Thanks Rob, that worked. >> >> Still on the subject of certs, any idea how to solve this error: >> >> Certificate format error: (SEC_ERROR_LEGACY_DATABASE) The >> certificate/key database is in an old, unsupported format. >> >> I see that in the gui when querying hosts as well as from cli when I >> ipa-show or ipa-find >> >> >> I have had this too, and we did not find a solution (search my recent >> posts on the archives). As a workaround I have created replicas and >> decommissioned the older replicas. >> > > On the one hand I'm glad this fixed it for you. On the other it is a > rather unsatisfying answer. Unfortunately NSS doesn't always provide the > most context with its error messages. This error is usually seen when one > tries to open a non-existent database, which in this case is a very strange > thing, especially since it goes from working to non-working in the same > apache process over a few minutes. > I totally agree. I did not have enough time to investigate it further because I'm changing jobs, so I really wanted to leave a working situation behind me. -- Groeten, natxo -------------- next part -------------- An HTML attachment was scrubbed... URL: From russellshackleford35 at gmail.com Thu Sep 29 20:35:58 2016 From: russellshackleford35 at gmail.com (Rusty Shackleford) Date: Thu, 29 Sep 2016 16:35:58 -0400 Subject: [Freeipa-users] external groups and /etc/group Message-ID: If I create an external group in freeIPA and add a user to that group, does that mean if that group exists on a host in /etc/group that the user will be a member of that group on that host? I've been trying to achieve that result but am failing and I don't know if I'm failing because I misunderstand what an external group is for, if I've missed a config option somewhere in freeipa or sssd, or if I'm simply using software that is too old. I'm using ipa-server-3.0.0-50.el6.centos.2 and sssd-1.13.3-22.el6_8.4 from the centos 6 updates repo. -------------- next part -------------- An HTML attachment was scrubbed... URL: From natxo.asenjo at gmail.com Thu Sep 29 20:46:56 2016 From: natxo.asenjo at gmail.com (Natxo Asenjo) Date: Thu, 29 Sep 2016 22:46:56 +0200 Subject: [Freeipa-users] another certmonger question In-Reply-To: <57ECF80E.6010406@redhat.com> References: <57EA5B17.8000307@redhat.com> <57ECF80E.6010406@redhat.com> Message-ID: On Thu, Sep 29, 2016 at 1:16 PM, Rob Crittenden wrote: > Natxo Asenjo wrote: > >> >> >> On Tue, Sep 27, 2016 at 1:42 PM, Rob Crittenden > > wrote: >> >> >> It's hard to say, it may in fact not be a problem. >> >> It is really a matter of what service the certificate(s) are related >> to. I'd look at the serial numbers and then correlate those to the >> issued certificates. >> >> I'd also do a service-find on the hostname to see if any services >> have certificates issued and with what serial numbers. >> >> >> I agree, it could be that. But just for testing I have created a vm, >> joined it to the domain and resubmitted the certificate. >> >> Now there are two valid host certificates with the same subject: >> >> >> $ ipa cert-find --subject=throwaway.unix.iriszorg.nl >> >> ---------------------- >> 2 certificates matched >> ---------------------- >> Serial number (hex): 0x3FFE0002 >> Serial number: 1073610754 >> Status: VALID >> Subject: CN=throwaway.unix.iriszorg.nl >> ,O=UNIX.IRISZORG.NL >> >> >> Serial number (hex): 0x3FFE0003 >> Serial number: 1073610755 >> Status: VALID >> Subject: CN=throwaway.unix.iriszorg.nl >> ,O=UNIX.IRISZORG.NL >> >> ---------------------------- >> Number of entries returned 2 >> ---------------------------- >> >> >> So it certmonger in this centos 6.8 32bit host is renewing but not >> having the old certificate revoked. >> > > I'd check the Apache log to find the cert_request call to see if you can > see if there are any issues raised. It should be doing a cert_revoke at the > same time. > > Can you should how this certificate is being tracked? > sure: $ sudo getcert list Number of certificates and requests being tracked: 1. Request ID '20160929100945': status: MONITORING stuck: no key pair storage: type=NSSDB,location='/etc/pki/nssdb',nickname='IPA Machine Certificate - throwaway.unix.iriszorg.nl',token='NSS Certificate DB' certificate: type=NSSDB,location='/etc/pki/nssdb',nickname='IPA Machine Certificate - throwaway.unix.iriszorg.nl',token='NSS Certificate DB' CA: IPA issuer: CN=Certificate Authority,O=UNIX.IRISZORG.NL subject: CN=throwaway.unix.iriszorg.nl,O=UNIX.IRISZORG.NL expires: 2018-09-30 10:13:17 UTC principal name: host/throwaway.unix.iriszorg.nl at UNIX.IRISZORG.NL key usage: digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment eku: id-kp-serverAuth,id-kp-clientAuth pre-save command: post-save command: track: yes auto-renew: yes now, let's resubmit: $ sudo ipa-getcert resubmit -i 20160929100945 Resubmitting "20160929100945" to "IPA". [jose.admin at throwaway ~]$ sudo getcert list Number of certificates and requests being tracked: 1. Request ID '20160929100945': status: MONITORING stuck: no key pair storage: type=NSSDB,location='/etc/pki/nssdb',nickname='IPA Machine Certificate - throwaway.unix.iriszorg.nl',token='NSS Certificate DB' certificate: type=NSSDB,location='/etc/pki/nssdb',nickname='IPA Machine Certificate - throwaway.unix.iriszorg.nl',token='NSS Certificate DB' CA: IPA issuer: CN=Certificate Authority,O=UNIX.IRISZORG.NL subject: CN=throwaway.unix.iriszorg.nl,O=UNIX.IRISZORG.NL expires: 2018-09-30 20:41:28 UTC principal name: host/throwaway.unix.iriszorg.nl at UNIX.IRISZORG.NL key usage: digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment eku: id-kp-serverAuth,id-kp-clientAuth pre-save command: post-save command: track: yes auto-renew: yes so it has been successfully renewed. In the access_log of the kdc I see this: 172.20.4.228 - - [29/Sep/2016:22:41:27 +0200] "POST https://kdc03.unix.iriszorg.nl:443/ca/eeca/ca/profileSubmitSSLClient HTTP/1.1" 200 1913 172.20.6.81 - host/throwaway.unix.iriszorg.nl at UNIX.IRISZORG.NL [29/Sep/2016:22:41:27 +0200] "POST /ipa/xml HTTP/1.1" 200 2929 and in the error_log: [Thu Sep 29 22:41:28.626669 2016] [:error] [pid 4617] ipa: INFO: [xmlserver] host/throwaway.unix.iriszorg.nl at UNIX.IRISZORG.NL: cert_request(u'MIID6DCCAtACAQAwQDEZMBcGA1UEChMQVU5JWC5JUklTWk9SRy5OTDEjMCEGA1UEAxMadGhyb3dhd2F5LnVuaXguaXJpc3pvcmcubmwwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQC4jBk7V2D5pX12kYrr++lwsWq1UWHy6PM9O+B/GvxaI0JoARBrhR6MKI1Ev+DV2r5ukNNWHj5+/kKbtW9XI2XMZ9pIBSwG3SG4m9s3gQV3dGQjlRCcU+MgXiDxRtRy2Vdzd1fZ9xdB1txH3ZnZfceTosNw4Jp3bm/VtPChWJeN6K671FLRCzJkI1KrC+LHfGbvyTtOipB5O9t8RkN4Qh01r/rphPvt9Gh+/mTlHnmGP9+sseqHHsgv2fPvRQowpJDEytTX5w/8pLrUCATqJUYfxK5RDuwD1304p3WXDFLoU6p2xaR63h34muj1a5NV1CvQFqJapHB5B/w6uUbLzjg3AgMBAAGgggFhMHcGCSqGSIb3DQEJFDFqHmgASQBQAEEAIABNAGEAYwBoAGkAbgBlACAAQwBlAHIAdABpAGYAaQBjAGEAdABlACAALQAgAHQAaAByAG8AdwBhAHcAYQB5AC4AdQBuAGkAeAAuAGkAcgBpAHMAegBvAHIAZwAuAG4AbDCB5QYJKoZIhvcNAQkOMYHXMIHUMIGhBgNVHREBAQAEgZYwgZOgQAYKKwYBBAGCNxQCA6AyDDBob3N0L3Rocm93YXdheS51bml4LmlyaXN6b3JnLm5sQFVOSVguSVJJU1pPUkcuTkygTwYGKwYBBQICoEUwQ6ASGxBVTklYLklSSVNaT1JHLk5MoS0wK6ADAgEBoSQwIhsEaG9zdBsadGhyb3dhd2F5LnVuaXguaXJpc3pvcmcubmwwDAYDVR0TAQH/BAIwADAgBgNVHQ4BAQAEFgQUgXWL3vdW/I31tQxv5YjyMZy4x8kwDQYJKoZIhvcNAQELBQADggEBAD674/oGYlQTQDSvwf0muYoxBsj1dc6gnArw0JJpGVCNMv/J3FdgOLcOhxzZcOfZiQr4NdYoV+/6mISOhknMa4ErJhqSAWbUA+w3+lL3CHfdDtNueUjZRbPZezcC0rhAlnXBT7iakjuhE56WkZz7AihEU8RAvnZfSRi1mhehf3wFRYKWuzK9AW1DTY/uGMmHXiFtvINpfAJ3yL66xPwTj4087nz9w4YUqNyCX+hYL+7idCJeoMjDyCqYQpjFkdfZhRuNd+rrKWTgYvKN3w/5+ItefDCYy8py91V2kXS7BrsYjd+2YHtQ2AbjgIW2xpTr/+PetToZyL50oWCpduT5t+M=', principal=u'host/throwaway.unix.iriszorg.nl at UNIX.IRISZORG.NL', add=True, version=u'2.51'): SUCCESS and now I have 3 valid certificates: $ ipa cert-find --subject=throwaway.unix.iriszorg.nl ---------------------- 3 certificates matched ---------------------- Serial number (hex): 0xFF9000D Serial number: 267976717 Status: VALID Subject: CN=throwaway.unix.iriszorg.nl,O=UNIX.IRISZORG.NL Serial number (hex): 0x3FFE0002 Serial number: 1073610754 Status: VALID Subject: CN=throwaway.unix.iriszorg.nl,O=UNIX.IRISZORG.NL Serial number (hex): 0x3FFE0003 Serial number: 1073610755 Status: VALID Subject: CN=throwaway.unix.iriszorg.nl,O=UNIX.IRISZORG.NL ---------------------------- Number of entries returned 3 ---------------------------- -- Groeten, natxo -------------- next part -------------- An HTML attachment was scrubbed... URL: From jhrozek at redhat.com Thu Sep 29 20:47:19 2016 From: jhrozek at redhat.com (Jakub Hrozek) Date: Thu, 29 Sep 2016 22:47:19 +0200 Subject: [Freeipa-users] external groups and /etc/group In-Reply-To: References: Message-ID: <20160929204719.bz5icuo2hmtapkaq@hendrix> On Thu, Sep 29, 2016 at 04:35:58PM -0400, Rusty Shackleford wrote: > If I create an external group in freeIPA and add a user to that group, does > that mean if that group exists on a host in /etc/group that the user will > be a member of that group on that host? I've been trying to achieve that > result but am failing and I don't know if I'm failing because I > misunderstand what an external group is for, if I've missed a config option > somewhere in freeipa or sssd, or if I'm simply using software that is too > old. I'm using ipa-server-3.0.0-50.el6.centos.2 and sssd-1.13.3-22.el6_8.4 > from the centos 6 updates repo. I think you are looking for: https://sourceware.org/glibc/wiki/Proposals/GroupMerging From mc at carcano.ch Thu Sep 29 21:13:22 2016 From: mc at carcano.ch (Marco Antonio Carcano) Date: Thu, 29 Sep 2016 23:13:22 +0200 Subject: [Freeipa-users] certificate list problems using web ui after upgrading to FreeIPA 4.2.0-15 Message-ID: <01cafbc4-68a6-bcb7-bcec-4716b2ce068a@carcano.ch> Hi all, I?ve just upgraded from FreeIPA 4.1 to FreeIPA 4.2.0-15 on a CentOS 7 (7.2.1511) and I?m no more able to list certificates using the web ui when I go on ?Authentication?, ?Certificates? and chose ?Certificates? I got the following error Certificate operation cannot be completed: Unable to communicate with CMS (Internal Server Error) and tomcat logs contain the following exception: Sep 29, 2016 4:54:35 PM org.apache.catalina.core.StandardWrapperValve invoke SEVERE: Allocate exception for servlet Resteasy java.lang.ClassNotFoundException: com.netscape.ca.CertificateAuthorityApplication at org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1720) at org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1571) at org.jboss.resteasy.spi.ResteasyDeployment.createApplication(ResteasyDeployment.java:28 at org.jboss.resteasy.plugins.server.servlet.ServletContainerDispatcher.init(ServletContainerDispatcher.java:95) at org.jboss.resteasy.plugins.server.servlet.HttpServletDispatcher.init(HttpServletDispatcher.java:36) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.apache.catalina.security.SecurityUtil$1.run(SecurityUtil.java:277) at org.apache.catalina.security.SecurityUtil$1.run(SecurityUtil.java:274) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAsPrivileged(Subject.java:536) at org.apache.catalina.security.SecurityUtil.execute(SecurityUtil.java:309) at org.apache.catalina.security.SecurityUtil.doAsPrivilege(SecurityUtil.java:169) at org.apache.catalina.security.SecurityUtil.doAsPrivilege(SecurityUtil.java:123) at org.apache.catalina.core.StandardWrapper.initServlet(StandardWrapper.java:1272) at org.apache.catalina.core.StandardWrapper.loadServlet(StandardWrapper.java:1197) at org.apache.catalina.core.StandardWrapper.allocate(StandardWrapper.java:864) at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:134) at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:122) at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:501) at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:171) at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102) at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:116) at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:40 at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1040) at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:607) at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:314) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61) at java.lang.Thread.run(Thread.java:745) So it complains it cannot find class com.netscape.ca.CertificateAuthorityApplication - that?s right The funny thing is that command line works like a charm pa caacl-find ---------------- 1 CA ACL matched ---------------- ACL name: hosts_services_caIPAserviceCert Enabled: TRUE Host category: all Service category: all Profiles: caIPAserviceCert ---------------------------- Number of entries returned 1 ?????????????? ipa cert-show Serial number: 1 Certificate: MIIDjzCCAnegAwIBAgIBATANBgkqhkiG9w0BAQsFADA2MRQwEgYDVQQKEwtJVEM0 VS5MT0NBTDEeMBwGA1UEAxMVQ2VydGlmaWNhdGUgQXV0aG9yaXR5 ? iI2rFqRTA+AF3xpqYBtOP+WwcBaue+OZ/GEsPOiyvcV1ZX6FWcKsmBf/T t7A9 Subject: CN=Certificate Authority,O=ME.LOCAL Issuer: CN=Certificate Authority,O=ME.LOCAL Not Before: Tue Dec 02 08:05:42 2014 UTC Not After: Sat Dec 02 08:05:42 2034 UTC Fingerprint (MD5): 59:4c:bb:dc:6a:e2:ff:17:6c:34:3e:f4:7e:fa:69:2e Fingerprint (SHA1): 74:c1:b3:a1:a1:25:5c:02:e8:ef:c5:30:14:fd:f0:58:79:6d:60:33 Serial number (hex): 0x1 Serial number: 1 By the way, the weird thing is that before migrating I added a replica node (so a fresh installation of FreeIPA 4.2.0-15) and the replica works perfectly, without this problem It seems to be a problem somehow related to the upgrade process How can I manage? Any suggestion? By the way, does anybody know which JAR contains com.netscape.ca.CertificateAuthorityApplication? I suppose it was /usr/share/java/pki/pki-ca.jar, but it contains only CertificateAuthority class: jar tf /usr/share/java/pki/pki-ca.jar |grep "CertificateAuthority" com/netscape/ca/CertificateAuthority.class Thanks Marco From russellshackleford35 at gmail.com Fri Sep 30 00:01:59 2016 From: russellshackleford35 at gmail.com (Rusty Shackleford) Date: Thu, 29 Sep 2016 20:01:59 -0400 Subject: [Freeipa-users] external groups and /etc/group In-Reply-To: <20160929204719.bz5icuo2hmtapkaq@hendrix> References: <20160929204719.bz5icuo2hmtapkaq@hendrix> Message-ID: On Thu, Sep 29, 2016 at 4:47 PM, Jakub Hrozek wrote: > > I think you are looking for: > https://sourceware.org/glibc/wiki/Proposals/GroupMerging > Well that's a bummer. Thanks for getting back to me. -------------- next part -------------- An HTML attachment was scrubbed... URL: From jrichard at placeiq.com Fri Sep 30 00:40:20 2016 From: jrichard at placeiq.com (Jim Richard) Date: Thu, 29 Sep 2016 20:40:20 -0400 Subject: [Freeipa-users] Certificate format error reported by GUI In-Reply-To: <1e2b525b-3aff-636f-ff84-e781299ce86f@redhat.com> References: <14E35142-15BB-4108-BFC9-A7BB84CB60CF@placeiq.com> <1e2b525b-3aff-636f-ff84-e781299ce86f@redhat.com> Message-ID: Hi Paul, 3.0.0 on Centos 6.8 Jim Richard SYSTEM ADMINISTRATOR III (646) 338-8905 > On Sep 29, 2016, at 11:58 AM, Pavel Vomacka wrote: > > Hello, > > which version of FreeIPA do you use? > On 09/28/2016 12:42 AM, Jim Richard wrote: >> When I try to look at hosts under the hosts tab. ipactl restart or just restarting httpd seems to clear it up for a short period. >> >> Three replicas in the environment, it only happens when I look at hosts using the GUI at one of the three replicas. >> >> >> Certificate format error: (SEC_ERROR_LEGACY_DATABASE) The certificate/key database is in an old, unsupported format. >> >> >> Jim Richard >> SYSTEM ADMINISTRATOR III >> (646) 338-8905 >> >> >> >> >> > > -- > Pavel^3 Vomacka -------------- next part -------------- An HTML attachment was scrubbed... URL: From jrichard at placeiq.com Fri Sep 30 00:53:57 2016 From: jrichard at placeiq.com (Jim Richard) Date: Thu, 29 Sep 2016 20:53:57 -0400 Subject: [Freeipa-users] Replica created with expired certs In-Reply-To: <57ED04DF.2040707@redhat.com> References: <5EE38BB7-ED9B-4001-8F64-DB7CD2319C02@placeiq.com> <57EBAD23.4090604@redhat.com> <17A0F251-F5DF-447C-AB64-C053E83369D0@placeiq.com> <57ED04DF.2040707@redhat.com> Message-ID: <0B1B6EC8-C4DF-4BFB-9157-51A0814DB73C@placeiq.com> another interesting thing, my httpd/error_logs are constantly getting spammed with: (I removed the stuff between the single quotes) Notice those names don?t match, should they? Me thinks not since those ?principal=? items are ALMOST all hosts that no longer exist in the FreeIPA system. I rare few do exist. So, that?s weird :) [Thu Sep 29 20:44:59 2016] [error] ipa: INFO: host/aerospike-cl1-203.nym1.placeiq.net at PLACEIQ.NET: cert_request(u????????..', principal=u'host/sbtt-nyc1-028.thum01.nym1.placeiq.net at PLACEIQ.NET', add=True): CertificateOperationError [Thu Sep 29 20:45:06 2016] [error] ipa: INFO: host/aerospike-cl2-210.nym1.placeiq.net at PLACEIQ.NET: cert_request(u????????..', principal=u'host/017.prod07.nym1.placeiq.net at PLACEIQ.NET', add=True): CertificateOperationError [Thu Sep 29 20:45:09 2016] [error] ipa: INFO: host/adsgateway-14.nym1.placeiq.net at PLACEIQ.NET: cert_request(u?????????...', principal=u'host/025.prod07.nym1.placeiq.net at PLACEIQ.NET', add=True): CertificateOperationError [Thu Sep 29 20:45:29 2016] [error] ipa: INFO: host/ttsandbox-022.nym1.placeiq.net at PLACEIQ.NET: cert_request(u??????????.', principal=u'host/sbtt-nyc1-022.thum01.nym1.placeiq.net at PLACEIQ.NET', add=True): CertificateOperationError Jim Richard SYSTEM ADMINISTRATOR III (646) 338-8905 > On Sep 29, 2016, at 8:11 AM, Rob Crittenden wrote: > > Natxo Asenjo wrote: >> hi Jim, >> >> On Thu, Sep 29, 2016 at 7:37 AM, Jim Richard > > wrote: >> >> Thanks Rob, that worked. >> >> Still on the subject of certs, any idea how to solve this error: >> >> Certificate format error: (SEC_ERROR_LEGACY_DATABASE) The >> certificate/key database is in an old, unsupported format. >> >> I see that in the gui when querying hosts as well as from cli when I >> ipa-show or ipa-find >> >> >> I have had this too, and we did not find a solution (search my recent >> posts on the archives). As a workaround I have created replicas and >> decommissioned the older replicas. > > On the one hand I'm glad this fixed it for you. On the other it is a rather unsatisfying answer. Unfortunately NSS doesn't always provide the most context with its error messages. This error is usually seen when one tries to open a non-existent database, which in this case is a very strange thing, especially since it goes from working to non-working in the same apache process over a few minutes. > > I'm not sure how I'd troubleshoot this if it were easily reproducible. I suspect we'd need to figure out which database cannot be found (most likely /etc/httpd/alias) and go from there. An strace is a brute-force way to see the file open but finding the right process to attach to is a bit of an art. > > rob > > -- > Manage your subscription for the Freeipa-users mailing list: > https://www.redhat.com/mailman/listinfo/freeipa-users > Go to http://freeipa.org for more info on the project -------------- next part -------------- An HTML attachment was scrubbed... URL: From jrichard at placeiq.com Fri Sep 30 01:06:38 2016 From: jrichard at placeiq.com (Jim Richard) Date: Thu, 29 Sep 2016 21:06:38 -0400 Subject: [Freeipa-users] Replica created with expired certs In-Reply-To: <0B1B6EC8-C4DF-4BFB-9157-51A0814DB73C@placeiq.com> References: <5EE38BB7-ED9B-4001-8F64-DB7CD2319C02@placeiq.com> <57EBAD23.4090604@redhat.com> <17A0F251-F5DF-447C-AB64-C053E83369D0@placeiq.com> <57ED04DF.2040707@redhat.com> <0B1B6EC8-C4DF-4BFB-9157-51A0814DB73C@placeiq.com> Message-ID: Can I and how? delete all certs for all hosts I mean, we only use FreeIPA for user login/sssd That said, do we even need those certs? Jim Richard SYSTEM ADMINISTRATOR III (646) 338-8905 > On Sep 29, 2016, at 8:53 PM, Jim Richard wrote: > > another interesting thing, my httpd/error_logs are constantly getting spammed with: (I removed the stuff between the single quotes) > > Notice those names don?t match, should they? > > Me thinks not since those ?principal=? items are ALMOST all hosts that no longer exist in the FreeIPA system. I rare few do exist. > > So, that?s weird :) > > [Thu Sep 29 20:44:59 2016] [error] ipa: INFO: host/aerospike-cl1-203.nym1.placeiq.net at PLACEIQ.NET : cert_request(u????????..', principal=u'host/sbtt-nyc1-028.thum01.nym1.placeiq.net at PLACEIQ.NET ', add=True): CertificateOperationError > > [Thu Sep 29 20:45:06 2016] [error] ipa: INFO: host/aerospike-cl2-210.nym1.placeiq.net at PLACEIQ.NET : cert_request(u????????..', principal=u'host/017.prod07.nym1.placeiq.net at PLACEIQ.NET ', add=True): CertificateOperationError > > [Thu Sep 29 20:45:09 2016] [error] ipa: INFO: host/adsgateway-14.nym1.placeiq.net at PLACEIQ.NET : cert_request(u?????????...', principal=u'host/025.prod07.nym1.placeiq.net at PLACEIQ.NET ', add=True): CertificateOperationError > > [Thu Sep 29 20:45:29 2016] [error] ipa: INFO: host/ttsandbox-022.nym1.placeiq.net at PLACEIQ.NET : cert_request(u??????????.', principal=u'host/sbtt-nyc1-022.thum01.nym1.placeiq.net at PLACEIQ.NET ', add=True): CertificateOperationError > > > > > > > Jim Richard > SYSTEM ADMINISTRATOR III > (646) 338-8905 > > > > >> On Sep 29, 2016, at 8:11 AM, Rob Crittenden > wrote: >> >> Natxo Asenjo wrote: >>> hi Jim, >>> >>> On Thu, Sep 29, 2016 at 7:37 AM, Jim Richard >>> >> wrote: >>> >>> Thanks Rob, that worked. >>> >>> Still on the subject of certs, any idea how to solve this error: >>> >>> Certificate format error: (SEC_ERROR_LEGACY_DATABASE) The >>> certificate/key database is in an old, unsupported format. >>> >>> I see that in the gui when querying hosts as well as from cli when I >>> ipa-show or ipa-find >>> >>> >>> I have had this too, and we did not find a solution (search my recent >>> posts on the archives). As a workaround I have created replicas and >>> decommissioned the older replicas. >> >> On the one hand I'm glad this fixed it for you. On the other it is a rather unsatisfying answer. Unfortunately NSS doesn't always provide the most context with its error messages. This error is usually seen when one tries to open a non-existent database, which in this case is a very strange thing, especially since it goes from working to non-working in the same apache process over a few minutes. >> >> I'm not sure how I'd troubleshoot this if it were easily reproducible. I suspect we'd need to figure out which database cannot be found (most likely /etc/httpd/alias) and go from there. An strace is a brute-force way to see the file open but finding the right process to attach to is a bit of an art. >> >> rob >> >> -- >> Manage your subscription for the Freeipa-users mailing list: >> https://www.redhat.com/mailman/listinfo/freeipa-users >> Go to http://freeipa.org for more info on the project > -------------- next part -------------- An HTML attachment was scrubbed... URL: From orion at cora.nwra.com Fri Sep 30 01:51:14 2016 From: orion at cora.nwra.com (Orion Poplawski) Date: Thu, 29 Sep 2016 19:51:14 -0600 Subject: [Freeipa-users] HBAC rules stop working Message-ID: <656f42fb-6614-e8a9-8861-84ac93838875@cora.nwra.com> server: ipa-server-4.2.0-15.sl7_2.19.x86_64 sssd-1.13.0-40.el7_2.12.x86_64 client: sssd-1.14.1-3.el7.centos.x86_64 AD trust - users are in AD. HBAC rule in place for client to allow a user to login/ssh/su/etc. This seems to have happened a couple times now, and again today after rebooting the IPA server. sssd was denying the user to ssh into the client by pam rules. Logged on to the IPA server and disabled and then re-enabled the HBAC rule for the client and then was able to log back in again. Has anyone else seen this before? client sssd_pam just went from: (Thu Sep 29 19:30:40 2016) [sssd[pam]] [pam_reply] (0x0200): pam_reply called with result [6]: Permission denied. to (Thu Sep 29 19:37:04 2016) [sssd[pam]] [pam_reply] (0x0200): pam_reply called with result [0]: Success. so I assume I'll need to collect debug logs from sssd on the server next time. -- Orion Poplawski Technical Manager 303-415-9701 x222 NWRA/CoRA Division FAX: 303-415-9702 3380 Mitchell Lane orion at cora.nwra.com Boulder, CO 80301 http://www.cora.nwra.com From beeth2006 at gmail.com Fri Sep 30 02:03:08 2016 From: beeth2006 at gmail.com (beeth beeth) Date: Thu, 29 Sep 2016 22:03:08 -0400 Subject: [Freeipa-users] Install IPA Servers with third-party certificate(external CA) In-Reply-To: References: <08f285b8-0e59-4f3a-deae-479660a193c3@redhat.com> <57ED0544.9090503@redhat.com> Message-ID: Thanks Florence and Rob! The replica worked after adding the certs during the replica preparation. Now I got several IPA clients installed with user authentication(ssh login with the users in IPA) working after some work. However, one of them failed during login with the following messages in syslog: Sep 29 21:41:13 ipaclient3 [sssd[krb5_child[2527]]]: Credentials cache permissions incorrect Sep 29 21:41:13 ipaclient3 [sssd[krb5_child[2527]]]: Decrypt integrity check failed Sep 29 21:41:13 ipaclient3 [sssd[krb5_child[2527]]]: Decrypt integrity check failed I tried this on ipaclient3: # kinit admin # ipa-getkeytab -s ipa1.example.com -p host/ipaclient3.example.com -k /etc/krb5.keytab No help. I also tried this on ipaclient3(which I don't think is relevant to the krb5 error): # wget -O /etc/ipa/ca.crt https://ipa1.example.com/ipa/config/ca.crt # certutil -A -d /etc/pki/nssdb -n "IPA CA" -t CT,C,C -a -i /etc/ipa/ca.crt Any idea about such krb5 issue? Thanks again! On Thu, Sep 29, 2016 at 9:36 AM, Florence Blanc-Renaud wrote: > On 09/29/2016 02:12 PM, Rob Crittenden wrote: > >> beeth beeth wrote: >> >>> Hi Florence, >>> >>> I previously tried option a) and failed(need to find out why later), but >>> I was able to successfully reinstall the server and the client with >>> option b), thanks a lot! So when it says "Installing Without a CA", it >>> means without a "embeded CA"(the IPA's own CA), is that right? >>> >>> Another main problem comes up for option b): now I am going to install >>> the replica server(ipa2), if I do the same as I did before: >>> >>> [root at ipa1 ~]# ipa-replica-prepare ipa2.example.com >>> >>> >>> copy the gpg file from ipa1 to ipa2 >>> >>> [root at ipa2 ~]# ipa-replica-install >>> /var/lib/ipa/replica-info-ipa2.example.com.gpg >>> >>> Then I believe the Apache on ipa2(the replica server) will use the >>> Verisign certificate with the same hostname(DN): ipa1.example.com >>> , NOT ipa2.example.com >>> , hence the users who visit >>> https://ipa2.example.com will experience security warning from the >>> browser, as expected... >>> What could be a solution for this? >>> >>> Thanks again! >>> >>> >>> On Thu, Sep 29, 2016 at 6:03 AM, Florence Blanc-Renaud >> > wrote: >>> >>> On 09/29/2016 11:43 AM, beeth beeth wrote: >>> >>> Thanks for the quick response Florence! >>> >>> My goal is the use a 3rd party certificate(such as Verisign >>> cert) for >>> Web UI(company security requirement), in fact we are not >>> required to use >>> 3rd party certificate for the LDAP server, but as I mentioned >>> earlier, I >>> couldn't make the new Verisign cert to work with the Web UI, >>> without >>> messing up the IPA function(after I updated the nss.conf to use >>> the new >>> cert in the /etc/httpd/alias db, the ipa_client_install failed). >>> So I >>> tried to follow the Redhat instruction, to see if I can get the >>> Verisign >>> cert installed at the most beginning, without using FreeIPA's >>> own/default certificate), but I got the CSR question. >>> >>> I did install IPA without a CA, by following the instruction at >>> >>> https://www.freeipa.org/page/Using_3rd_part_certificates_for_HTTP/LDAP >>> >>> >> >, >>> but failed to restart HTTPD. When and how can I provide the >>> 3rd-party >>> certificate? Could you please point me a document about the >>> detail? >>> >>> Hi, >>> >>> you need first to clarify if you want FreeIPA to act as a CA or not. >>> The setup will depend on this choice. >>> >>> - option a) FreeIPA with an embedded CA: >>> you can install FreeIPA with a self-signed CA, then follow the >>> instructions at >>> >>> https://www.freeipa.org/page/Using_3rd_part_certificates_for_HTTP/LDAP >>> >>> >>> in order to replace the WebUI certificate. Please note that there >>> were some bugs in ipa-server-certinstall, preventing httpd from >>> starting (Ticket #4786 [1]). The workaround is to manually update >>> nss.conf (as you did) and manually import the CA certificate into >>> /etc/pki/pki-tomcat/alias, for instance with >>> $ certutil -A -d /etc/pki/pki-tomcat/alias -i cacert.pem -n nickname >>> -t C,, >>> >>> >>> - option b) Free IPA without CA >>> the installation instructions are in Installing without a CA [2]. >>> You will provide the certificate that will be used by both the LDAP >>> server and the WebUI in the command options. >>> >> >> You'd need either a separate certificate or one with multiple subject >> alternative names, one for each master. I also imagine you'd need to >> provide this certificate at replica preparation time if you've installed >> without a CA. >> >> Yes, that's right. You can use the command ipa-replica-prepare with the > options --dirsrv-cert-file / --dirsrv-pin and --http-cert-file / --http-pin > to provide the replica's certificate and key. They will be embedded in the > replica file and used during the replica installation. > > Flo. > > > rob >> >> >>> HTH, >>> Flo. >>> >>> [1] https://fedorahosted.org/freeipa/ticket/4786 >>> >>> [2] >>> >>> https://access.redhat.com/documentation/en-US/Red_Hat_Enterp >>> rise_Linux/7/html/Linux_Domain_Identity_Authentication_and_ >>> Policy_Guide/install-server.html#install-server-without-ca >>> >>> >>> >> prise_Linux/7/html/Linux_Domain_Identity_Authentication_and_ >>> Policy_Guide/install-server.html#install-server-without-ca> >>> >>> >>> >>> >>> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From rakesh.rajasekharan at gmail.com Fri Sep 30 05:19:57 2016 From: rakesh.rajasekharan at gmail.com (Rakesh Rajasekharan) Date: Fri, 30 Sep 2016 10:49:57 +0530 Subject: [Freeipa-users] kinit: admin account getting locked out frequently Message-ID: Hi All , In my FreeIPA setup, I frequently seeing this error "kinit: Clients credentials have been revoked while getting initial credentials" while i try "kinit admin" I have tried decreasing the "--failinterval" and increasing the "--maxfail" values However, I still continue to see this error and it does not get unlocked. I have to manually unlock using "modprinc -unlock admin at XYZ.com" In the history on the IPA admin server.. I do not see any instances of "kinit admin" being run. Is there anything else that I should check to trace the cause of this. Thanks. Rakesh -------------- next part -------------- An HTML attachment was scrubbed... URL: From jhrozek at redhat.com Fri Sep 30 06:53:40 2016 From: jhrozek at redhat.com (Jakub Hrozek) Date: Fri, 30 Sep 2016 08:53:40 +0200 Subject: [Freeipa-users] external groups and /etc/group In-Reply-To: References: <20160929204719.bz5icuo2hmtapkaq@hendrix> Message-ID: <20160930065340.oquybv6ij72c6lnz@hendrix> On Thu, Sep 29, 2016 at 08:01:59PM -0400, Rusty Shackleford wrote: > On Thu, Sep 29, 2016 at 4:47 PM, Jakub Hrozek wrote: > > > > > I think you are looking for: > > https://sourceware.org/glibc/wiki/Proposals/GroupMerging > > > > Well that's a bummer. Thanks for getting back to me. The functionality is already implemented and should work in newer Fedoras. Unfortunately it is not backported to RHEL/centos.. From jhrozek at redhat.com Fri Sep 30 06:56:48 2016 From: jhrozek at redhat.com (Jakub Hrozek) Date: Fri, 30 Sep 2016 08:56:48 +0200 Subject: [Freeipa-users] HBAC rules stop working In-Reply-To: <656f42fb-6614-e8a9-8861-84ac93838875@cora.nwra.com> References: <656f42fb-6614-e8a9-8861-84ac93838875@cora.nwra.com> Message-ID: <20160930065648.so77ieldhqtmrhwt@hendrix> On Thu, Sep 29, 2016 at 07:51:14PM -0600, Orion Poplawski wrote: > server: > ipa-server-4.2.0-15.sl7_2.19.x86_64 > sssd-1.13.0-40.el7_2.12.x86_64 > > client: > sssd-1.14.1-3.el7.centos.x86_64 > > AD trust - users are in AD. HBAC rule in place for client to allow a user > to login/ssh/su/etc. > > This seems to have happened a couple times now, and again today after > rebooting the IPA server. sssd was denying the user to ssh into the client > by pam rules. Logged on to the IPA server and disabled and then re-enabled > the HBAC rule for the client and then was able to log back in again. Has > anyone else seen this before? > > client sssd_pam just went from: > > (Thu Sep 29 19:30:40 2016) [sssd[pam]] [pam_reply] (0x0200): pam_reply > called with result [6]: Permission denied. > > to > > (Thu Sep 29 19:37:04 2016) [sssd[pam]] [pam_reply] (0x0200): pam_reply > called with result [0]: Success. > > so I assume I'll need to collect debug logs from sssd on the server next > time. Yes..please try to collect logs from a machine that exhibits the bug. I suspect this is not related to HBAC per se, but rather to external group memberships, so it would also be nice to check if the groups are resolved on the faulty machine. And if they wouldn't be, please also check if they are resolved on the server itself (and collect logs there..) From jhrozek at redhat.com Fri Sep 30 06:59:38 2016 From: jhrozek at redhat.com (Jakub Hrozek) Date: Fri, 30 Sep 2016 08:59:38 +0200 Subject: [Freeipa-users] Install IPA Servers with third-party certificate(external CA) In-Reply-To: References: <08f285b8-0e59-4f3a-deae-479660a193c3@redhat.com> <57ED0544.9090503@redhat.com> Message-ID: <20160930065938.twbz4ukq4khjfolm@hendrix> On Thu, Sep 29, 2016 at 10:03:08PM -0400, beeth beeth wrote: > Thanks Florence and Rob! The replica worked after adding the certs during > the replica preparation. > > Now I got several IPA clients installed with user authentication(ssh login > with the users in IPA) working after some work. However, one of them failed > during login with the following messages in syslog: > > Sep 29 21:41:13 ipaclient3 [sssd[krb5_child[2527]]]: Credentials cache > permissions incorrect This is RHEL-7, right? Then I'm not sure why would ccache permissions be incorrect, maybe except for an SELinux issue.. (you are using the KEYRING ccache, right?) > Sep 29 21:41:13 ipaclient3 [sssd[krb5_child[2527]]]: Decrypt integrity > check failed > Sep 29 21:41:13 ipaclient3 [sssd[krb5_child[2527]]]: Decrypt integrity > check failed These two mean a wrong password was supplied. You can enable sssd debugging and take a look into krb5_child.log. If you crank up the debug_level all the way up to 10, then you'll also see KRB5_TRACE-level messages.. From yamakasi.014 at gmail.com Fri Sep 30 07:17:35 2016 From: yamakasi.014 at gmail.com (Matt .) Date: Fri, 30 Sep 2016 09:17:35 +0200 Subject: [Freeipa-users] FreeIPA as CA for your own internal webservices Message-ID: Hi Guys, I'm wondering how it's possible to use FreeIPA as your own CA for apache vhosts and such. I need to many certificates for subdomains (wildcards) that its undoable and I would like to use my FreeIAP installs for this. I installed the root certificate on windows from my IPA install and that works, FreeIPA itself is now trusted. But how to do this for other webservices no matter what software I use ? I hope someone can give me direction here. Thanks! Matt From sbose at redhat.com Fri Sep 30 07:30:30 2016 From: sbose at redhat.com (Sumit Bose) Date: Fri, 30 Sep 2016 09:30:30 +0200 Subject: [Freeipa-users] SELinux errors with sssd-krb5-common-1.13.0-40.el7_2.12.x86_64 In-Reply-To: References: <20160929074241.GF24352@p.Speedport_W_724V_Typ_A_05011603_00_009> Message-ID: <20160930073030.GK24352@p.Speedport_W_724V_Typ_A_05011603_00_009> On Thu, Sep 29, 2016 at 12:07:13PM -0400, Prasun Gera wrote: > I need to set SELinux to enforcing to get the relevant SSSD logs, right ? yes, I think this would help to identify the operation which triggers the AVC because it should fail. bye, Sumit > > On Thu, Sep 29, 2016 at 3:42 AM, Sumit Bose wrote: > > > On Thu, Sep 29, 2016 at 12:47:34AM -0400, Prasun Gera wrote: > > > I started seeing some selinux errors on one of my RHEL 7 clients recently > > > (possibly after a recent yum update ?), which prevents users from logging > > > in with passwords. I've put SELinux in permissive mode for now. Logs > > follow > > > > This sounds like https://bugzilla.redhat.com/show_bug.cgi?id=1301686 . > > Would you mind adding your findings and the SSSD logs as described in > > https://bugzilla.redhat.com/show_bug.cgi?id=1301686#c2 to the bugzilla > > ticket. > > > > Thank you. > > > > bye, > > Sumit > > > > > > > > > > > SELinux is preventing /usr/libexec/sssd/krb5_child from read access on > > the > > > key Unknown. > > > > > > ***** Plugin catchall (100. confidence) suggests > > > ************************** > > > > > > If you believe that krb5_child should be allowed read access on the > > Unknown > > > key by default. > > > Then you should report this as a bug. > > > You can generate a local policy module to allow this access. > > > Do > > > allow this access for now by executing: > > > # grep krb5_child /var/log/audit/audit.log | audit2allow -M mypol > > > # semodule -i mypol.pp > > > > > > > > > Additional Information: > > > Source Context system_u:system_r:sssd_t:s0 > > > Target Context system_u:system_r:unconfined_service_t:s0 > > > Target Objects Unknown [ key ] > > > Source krb5_child > > > Source Path /usr/libexec/sssd/krb5_child > > > Port > > > Host > > > Source RPM Packages sssd-krb5-common-1.13.0-40.el7_2.12.x86_64 > > > Target RPM Packages > > > Policy RPM selinux-policy-3.13.1-60.el7_2.9.noarch > > > Selinux Enabled True > > > Policy Type targeted > > > Enforcing Mode Permissive > > > Host Name example.com > > > Platform Linux example.com 4.4.19-1.el7.x86_64 > > > #1 SMP Mon Aug 29 18:38:32 EDT 2016 x86_64 > > > x86_64 > > > Alert Count 38 > > > First Seen 2016-09-28 18:37:43 EDT > > > Last Seen 2016-09-28 22:08:41 EDT > > > Local ID aa5271fa-f708-46b0-a382-fb1f90ce8973 > > > Raw Audit Messages > > > type=AVC msg=audit(1475114921.376:90787): avc: denied { read } for > > > pid=8272 comm="krb5_child" scontext=system_u:system_r:sssd_t:s0 > > > tcontext=system_u:system_r:unconfined_service_t:s0 tclass=key > > permissive=0 > > > > > > > > > type=SYSCALL msg=audit(1475114921.376:90787): arch=x86_64 syscall=keyctl > > > success=yes exit=EINTR a0=b a1=333b5463 a2=0 a3=0 items=0 ppid=891 > > pid=8272 > > > auid=4294967295 uid=1388200053 gid=1388200053 euid=1388200053 > > > suid=1388200053 fsuid=1388200053 egid=1388200053 sgid=1388200053 > > > fsgid=1388200053 tty=(none) ses=4294967295 comm=krb5_child > > > exe=/usr/libexec/sssd/krb5_child subj=system_u:system_r:sssd_t:s0 > > key=(null) > > > > > > Hash: krb5_child,sssd_t,unconfined_service_t,key,read > > > > > > ------------------------------------------------------------ > > -------------------- > > > > > > SELinux is preventing /usr/libexec/sssd/krb5_child from view access on > > the > > > key Unknown. > > > > > > ***** Plugin catchall (100. confidence) suggests > > > ************************** > > > > > > If you believe that krb5_child should be allowed view access on the > > Unknown > > > key by default. > > > Then you should report this as a bug. > > > You can generate a local policy module to allow this access. > > > Do > > > allow this access for now by executing: > > > # grep krb5_child /var/log/audit/audit.log | audit2allow -M mypol > > > # semodule -i mypol.pp > > > > > > > > > Additional Information: > > > Source Context system_u:system_r:sssd_t:s0 > > > Target Context system_u:system_r:unconfined_service_t:s0 > > > Target Objects Unknown [ key ] > > > Source krb5_child > > > Source Path /usr/libexec/sssd/krb5_child > > > Port > > > Host > > > Source RPM Packages sssd-krb5-common-1.13.0-40.el7_2.12.x86_64 > > > Target RPM Packages > > > Policy RPM selinux-policy-3.13.1-60.el7_2.9.noarch > > > Selinux Enabled True > > > Policy Type targeted > > > Enforcing Mode Permissive > > > Host Name example.com > > > Platform Linux example.com 4.4.19-1.el7.x86_64 > > > #1 SMP Mon Aug 29 18:38:32 EDT 2016 x86_64 > > > x86_64 > > > Alert Count 10 > > > First Seen 2016-09-28 18:40:00 EDT > > > Last Seen 2016-09-28 22:08:41 EDT > > > Local ID 22ec0970-9447-444a-9631-69749e4e7226 > > > Raw Audit Messages > > > type=AVC msg=audit(1475114921.376:90789): avc: denied { view } for > > > pid=8272 comm="krb5_child" scontext=system_u:system_r:sssd_t:s0 > > > tcontext=system_u:system_r:unconfined_service_t:s0 tclass=key > > permissive=0 > > > > > > > > > type=SYSCALL msg=audit(1475114921.376:90789): arch=x86_64 syscall=keyctl > > > success=no exit=EACCES a0=6 a1=2e1c07f1 a2=0 a3=0 items=0 ppid=891 > > pid=8272 > > > auid=4294967295 uid=1388200053 gid=1388200053 euid=1388200053 > > > suid=1388200053 fsuid=1388200053 egid=1388200053 sgid=1388200053 > > > fsgid=1388200053 tty=(none) ses=4294967295 comm=krb5_child > > > exe=/usr/libexec/sssd/krb5_child subj=system_u:system_r:sssd_t:s0 > > key=(null) > > > > > > Hash: krb5_child,sssd_t,unconfined_service_t,key,view > > > > > > ------------------------------------------------------------ > > -------------------- > > > > > > SELinux is preventing /usr/libexec/sssd/krb5_child from write access on > > the > > > key Unknown. > > > > > > ***** Plugin catchall (100. confidence) suggests > > > ************************** > > > > > > If you believe that krb5_child should be allowed write access on the > > > Unknown key by default. > > > Then you should report this as a bug. > > > You can generate a local policy module to allow this access. > > > Do > > > allow this access for now by executing: > > > # grep krb5_child /var/log/audit/audit.log | audit2allow -M mypol > > > # semodule -i mypol.pp > > > > > > > > > Additional Information: > > > Source Context system_u:system_r:sssd_t:s0 > > > Target Context system_u:system_r:unconfined_service_t:s0 > > > Target Objects Unknown [ key ] > > > Source krb5_child > > > Source Path /usr/libexec/sssd/krb5_child > > > Port > > > Host > > > Source RPM Packages sssd-krb5-common-1.13.0-40.el7_2.12.x86_64 > > > Target RPM Packages > > > Policy RPM selinux-policy-3.13.1-60.el7_2.9.noarch > > > Selinux Enabled True > > > Policy Type targeted > > > Enforcing Mode Permissive > > > Host Name example.com > > > Platform Linux example.com 4.4.19-1.el7.x86_64 > > > #1 SMP Mon Aug 29 18:38:32 EDT 2016 x86_64 > > > x86_64 > > > Alert Count 10 > > > First Seen 2016-09-28 18:40:00 EDT > > > Last Seen 2016-09-28 22:08:41 EDT > > > Local ID 8982bbec-38db-485b-9266-57fdaa8a3621 > > > > > > Raw Audit Messages > > > type=AVC msg=audit(1475114921.376:90790): avc: denied { write } for > > > pid=8272 comm="krb5_child" scontext=system_u:system_r:sssd_t:s0 > > > tcontext=system_u:system_r:unconfined_service_t:s0 tclass=key > > permissive=0 > > > > > > type=SYSCALL msg=audit(1475114921.376:90790): arch=x86_64 > > syscall=add_key > > > success=no exit=EACCES a0=7f6987905ffc a1=7ffeed78b1f0 a2=0 a3=0 items=0 > > > ppid=891 pid=8272 auid=4294967295 uid=1388200053 gid=1388200053 > > > euid=1388200053 suid=1388200053 fsuid=1388200053 egid=1388200053 > > > sgid=1388200053 fsgid=1388200053 tty=(none) ses=4294967295 > > comm=krb5_child > > > exe=/usr/libexec/sssd/krb5_child subj=system_u:system_r:sssd_t:s0 > > key=(null) > > > > > > Hash: krb5_child,sssd_t,unconfined_service_t,key,write > > > > > -- > > > Manage your subscription for the Freeipa-users mailing list: > > > https://www.redhat.com/mailman/listinfo/freeipa-users > > > Go to http://freeipa.org for more info on the project > > > > -- > > Manage your subscription for the Freeipa-users mailing list: > > https://www.redhat.com/mailman/listinfo/freeipa-users > > Go to http://freeipa.org for more info on the project > > From pvomacka at redhat.com Fri Sep 30 07:42:33 2016 From: pvomacka at redhat.com (Pavel Vomacka) Date: Fri, 30 Sep 2016 09:42:33 +0200 Subject: [Freeipa-users] Certificate format error reported by GUI In-Reply-To: References: <14E35142-15BB-4108-BFC9-A7BB84CB60CF@placeiq.com> <1e2b525b-3aff-636f-ff84-e781299ce86f@redhat.com> Message-ID: <45626024-0cbb-88be-6adc-63dd47bb3d4b@redhat.com> Ah, ok, does /var/log/httpd/error_log contain any error after looking at hosts using GUI? And could you please send output of ipactl status after the error ocurres? On 09/30/2016 02:40 AM, Jim Richard wrote: > Hi Paul, 3.0.0 on Centos 6.8 > > > Jim Richard > > > SYSTEM ADMINISTRATOR III > /(646) 338-8905 / > > > PlaceIQ:Location Data Accuracy > > > > >> On Sep 29, 2016, at 11:58 AM, Pavel Vomacka > > wrote: >> >> Hello, >> >> which version of FreeIPA do you use? >> >> On 09/28/2016 12:42 AM, Jim Richard wrote: >>> When I try to look at hosts under the hosts tab. ipactl restart or >>> just restarting httpd seems to clear it up for a short period. >>> >>> Three replicas in the environment, it only happens when I look at >>> hosts using the GUI at one of the three replicas. >>> >>> >>> Certificate format error: (SEC_ERROR_LEGACY_DATABASE) The >>> certificate/key database is in an old, unsupported format. >>> >>> >>> Jim Richard >>> >>> >>> SYSTEM ADMINISTRATOR III >>> /(646) 338-8905 / >>> >>> >>> PlaceIQ:Location Data Accuracy >>> >>> >>> >>> >>> >>> >> >> -- >> Pavel^3 Vomacka > -- Pavel^3 Vomacka -------------- next part -------------- An HTML attachment was scrubbed... URL: From rcritten at redhat.com Fri Sep 30 08:45:23 2016 From: rcritten at redhat.com (Rob Crittenden) Date: Fri, 30 Sep 2016 10:45:23 +0200 Subject: [Freeipa-users] another certmonger question In-Reply-To: References: <57EA5B17.8000307@redhat.com> <57ECF80E.6010406@redhat.com> Message-ID: <57EE2623.3010303@redhat.com> Natxo Asenjo wrote: > > > On Thu, Sep 29, 2016 at 1:16 PM, Rob Crittenden > wrote: > > Natxo Asenjo wrote: > > > > On Tue, Sep 27, 2016 at 1:42 PM, Rob Crittenden > > >> wrote: > > > It's hard to say, it may in fact not be a problem. > > It is really a matter of what service the certificate(s) > are related > to. I'd look at the serial numbers and then correlate those > to the > issued certificates. > > I'd also do a service-find on the hostname to see if any > services > have certificates issued and with what serial numbers. > > > I agree, it could be that. But just for testing I have created a vm, > joined it to the domain and resubmitted the certificate. > > Now there are two valid host certificates with the same subject: > > > $ ipa cert-find --subject=throwaway.unix.iriszorg.nl > > > > ---------------------- > 2 certificates matched > ---------------------- > Serial number (hex): 0x3FFE0002 > Serial number: 1073610754 > Status: VALID > Subject: CN=throwaway.unix.iriszorg.nl > > >,O=UNIX.IRISZORG.NL > > > > Serial number (hex): 0x3FFE0003 > Serial number: 1073610755 > Status: VALID > Subject: CN=throwaway.unix.iriszorg.nl > > >,O=UNIX.IRISZORG.NL > > > ---------------------------- > Number of entries returned 2 > ---------------------------- > > > So it certmonger in this centos 6.8 32bit host is renewing but not > having the old certificate revoked. > > > I'd check the Apache log to find the cert_request call to see if you > can see if there are any issues raised. It should be doing a > cert_revoke at the same time. > > Can you should how this certificate is being tracked? > > > sure: > > $ sudo getcert list > Number of certificates and requests being tracked: 1. > Request ID '20160929100945': > status: MONITORING > stuck: no > key pair storage: > type=NSSDB,location='/etc/pki/nssdb',nickname='IPA Machine Certificate - > throwaway.unix.iriszorg.nl > ',token='NSS Certificate DB' > certificate: type=NSSDB,location='/etc/pki/nssdb',nickname='IPA > Machine Certificate - throwaway.unix.iriszorg.nl > ',token='NSS Certificate DB' > CA: IPA > issuer: CN=Certificate Authority,O=UNIX.IRISZORG.NL > > subject: CN=throwaway.unix.iriszorg.nl > ,O=UNIX.IRISZORG.NL > > expires: 2018-09-30 10:13:17 UTC > principal name: host/throwaway.unix.iriszorg.nl at UNIX.IRISZORG.NL > > key usage: > digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment > eku: id-kp-serverAuth,id-kp-clientAuth > pre-save command: > post-save command: > track: yes > auto-renew: yes > > now, let's resubmit: > > $ sudo ipa-getcert resubmit -i 20160929100945 > Resubmitting "20160929100945" to "IPA". > [jose.admin at throwaway ~]$ sudo getcert list > Number of certificates and requests being tracked: 1. > Request ID '20160929100945': > status: MONITORING > stuck: no > key pair storage: > type=NSSDB,location='/etc/pki/nssdb',nickname='IPA Machine Certificate - > throwaway.unix.iriszorg.nl > ',token='NSS Certificate DB' > certificate: type=NSSDB,location='/etc/pki/nssdb',nickname='IPA > Machine Certificate - throwaway.unix.iriszorg.nl > ',token='NSS Certificate DB' > CA: IPA > issuer: CN=Certificate Authority,O=UNIX.IRISZORG.NL > > subject: CN=throwaway.unix.iriszorg.nl > ,O=UNIX.IRISZORG.NL > > expires: 2018-09-30 20:41:28 UTC > principal name: host/throwaway.unix.iriszorg.nl at UNIX.IRISZORG.NL > > key usage: > digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment > eku: id-kp-serverAuth,id-kp-clientAuth > pre-save command: > post-save command: > track: yes > auto-renew: yes > > so it has been successfully renewed. > > In the access_log of the kdc I see this: > > 172.20.4.228 - - [29/Sep/2016:22:41:27 +0200] "POST > https://kdc03.unix.iriszorg.nl:443/ca/eeca/ca/profileSubmitSSLClient > HTTP/1.1" 200 1913 > 172.20.6.81 - host/throwaway.unix.iriszorg.nl at UNIX.IRISZORG.NL > > [29/Sep/2016:22:41:27 +0200] "POST /ipa/xml HTTP/1.1" 200 2929 > > and in the error_log: > [Thu Sep 29 22:41:28.626669 2016] [:error] [pid 4617] ipa: INFO: > [xmlserver] host/throwaway.unix.iriszorg.nl at UNIX.IRISZORG.NL > : > cert_request(u'MIID6DCCAtACAQAwQDEZMBcGA1UEChMQVU5JWC5JUklTWk9SRy5OTDEjMCEGA1UEAxMadGhyb3dhd2F5LnVuaXguaXJpc3pvcmcubmwwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQC4jBk7V2D5pX12kYrr++lwsWq1UWHy6PM9O+B/GvxaI0JoARBrhR6MKI1Ev+DV2r5ukNNWHj5+/kKbtW9XI2XMZ9pIBSwG3SG4m9s3gQV3dGQjlRCcU+MgXiDxRtRy2Vdzd1fZ9xdB1txH3ZnZfceTosNw4Jp3bm/VtPChWJeN6K671FLRCzJkI1KrC+LHfGbvyTtOipB5O9t8RkN4Qh01r/rphPvt9Gh+/mTlHnmGP9+sseqHHsgv2fPvRQowpJDEytTX5w/8pLrUCATqJUYfxK5RDuwD1304p3WXDFLoU6p2xaR63h34muj1a5NV1CvQFqJapHB5B/w6uUbLzjg3AgMBAAGgggFhMHcGCSqGSIb3DQEJFDFqHmgASQBQAEEAIABNAGEAYwBoAGkAbgBlACAAQwBlAHIAdABpAGYAaQBjAGEAdABlACAALQAgAHQAaAByAG8AdwBhAHcAYQB5AC4AdQBuAGkAeAAuAGkAcgBpAHMAegBvAHIAZwAuAG4AbDCB5QYJKoZIhvcNAQkOMYHXMIHUMIGhBgNVHREBAQAEgZYwgZOgQAYKKwYBBAGCNxQCA6AyDDBob3N0L3Rocm93YXdheS51bml4LmlyaXN6b3JnLm5sQFVOSVguSVJJU1pPUkcuTkygTwYGKwYBBQICoEUwQ6ASGxBVTklYLklSSVNaT1JHLk5MoS0wK6ADAgEBoSQwIhsEaG9zdBsadGhyb3dhd2F5LnVuaXguaXJpc3pvcmcubmwwDAYDVR0TAQH/BAIwADAgBgNVHQ4BAQAEFgQUgXWL3vdW/I31tQxv5YjyMZy4x8kw! DQYJKoZIhv cNAQELBQADggEBAD674/oGYlQTQDSvwf0muYoxBsj1dc6gnArw0JJpGVCNMv/J3FdgOLcOhxzZcOfZiQr4NdYoV+/6mISOhknMa4ErJhqSAWbUA+w3+lL3CHfdDtNueUjZRbPZezcC0rhAlnXBT7iakjuhE56WkZz7AihEU8RAvnZfSRi1mhehf3wFRYKWuzK9AW1DTY/uGMmHXiFtvINpfAJ3yL66xPwTj4087nz9w4YUqNyCX+hYL+7idCJeoMjDyCqYQpjFkdfZhRuNd+rrKWTgYvKN3w/5+ItefDCYy8py91V2kXS7BrsYjd+2YHtQ2AbjgIW2xpTr/+PetToZyL50oWCpduT5t+M=', > principal=u'host/throwaway.unix.iriszorg.nl at UNIX.IRISZORG.NL > ', add=True, > version=u'2.51'): SUCCESS > > and now I have 3 valid certificates: > > $ ipa cert-find --subject=throwaway.unix.iriszorg.nl > > ---------------------- > 3 certificates matched > ---------------------- > Serial number (hex): 0xFF9000D > Serial number: 267976717 > Status: VALID > Subject: CN=throwaway.unix.iriszorg.nl > ,O=UNIX.IRISZORG.NL > > > Serial number (hex): 0x3FFE0002 > Serial number: 1073610754 > Status: VALID > Subject: CN=throwaway.unix.iriszorg.nl > ,O=UNIX.IRISZORG.NL > > > Serial number (hex): 0x3FFE0003 > Serial number: 1073610755 > Status: VALID > Subject: CN=throwaway.unix.iriszorg.nl > ,O=UNIX.IRISZORG.NL > > ---------------------------- > Number of entries returned 3 > ---------------------------- Ok, let me start by saying that this is not a bug in either certmonger or dogtag. IPA is supposed to do the revocation in the cert_request command. The steps IPA _should_ be taking are: 1. Figure out if we are doing a certificate for a host or a service. 2. See if the requester is allowed to manage this entry 3. Look at the entry to see if it has a usercertificate attribute. If so revoke that serial number, then clear the usercertificate value in the host or service entry (via service_mod or host_mod) 4. Request a new certificate 5. Update IPA with the new value Does a certificate appear in ipa host-show throwaway.unix.iriszorg.nl, and which certificate serial number? rob From rcritten at redhat.com Fri Sep 30 08:49:15 2016 From: rcritten at redhat.com (Rob Crittenden) Date: Fri, 30 Sep 2016 10:49:15 +0200 Subject: [Freeipa-users] Replica created with expired certs In-Reply-To: <0B1B6EC8-C4DF-4BFB-9157-51A0814DB73C@placeiq.com> References: <5EE38BB7-ED9B-4001-8F64-DB7CD2319C02@placeiq.com> <57EBAD23.4090604@redhat.com> <17A0F251-F5DF-447C-AB64-C053E83369D0@placeiq.com> <57ED04DF.2040707@redhat.com> <0B1B6EC8-C4DF-4BFB-9157-51A0814DB73C@placeiq.com> Message-ID: <57EE270B.9090009@redhat.com> Jim Richard wrote: > another interesting thing, my httpd/error_logs are constantly getting > spammed with: (I removed the stuff between the single quotes) > > Notice those names don?t match, should they? > > Me thinks not since those ?principal=? items are ALMOST all hosts that > no longer exist in the FreeIPA system. I rare few do exist. > > So, that?s weird :) I suspect that certmonger is still tracking certificate(s) on those hosts. You should be able to clear things up on those hosts with something like: # ipa-getcert list # ipa-getcert stop-tracking -i It's hard to say if the hostname mismatch is expected or not, it depends on how the requests were done initially. The first value in the log represents the principal that did the BIND, so the host to look on is aerospike-cl1-203.nym1.placeiq.net. The second hostname is the principal that the certificate is being requested _for_. This is basically a delegated request. rob > > [Thu Sep 29 20:44:59 2016] [error] ipa: INFO: > host/aerospike-cl1-203.nym1.placeiq.net at PLACEIQ.NET > : > cert_request(u????????..', > principal=u'host/sbtt-nyc1-028.thum01.nym1.placeiq.net at PLACEIQ.NET > ', > add=True): CertificateOperationError > > [Thu Sep 29 20:45:06 2016] [error] ipa: INFO: > host/aerospike-cl2-210.nym1.placeiq.net at PLACEIQ.NET > : > cert_request(u????????..', > principal=u'host/017.prod07.nym1.placeiq.net at PLACEIQ.NET > ', > add=True): CertificateOperationError > > [Thu Sep 29 20:45:09 2016] [error] ipa: INFO: > host/adsgateway-14.nym1.placeiq.net at PLACEIQ.NET > : > cert_request(u?????????...', > principal=u'host/025.prod07.nym1.placeiq.net at PLACEIQ.NET > ', > add=True): CertificateOperationError > > [Thu Sep 29 20:45:29 2016] [error] ipa: INFO: > host/ttsandbox-022.nym1.placeiq.net at PLACEIQ.NET > : > cert_request(u??????????.', > principal=u'host/sbtt-nyc1-022.thum01.nym1.placeiq.net at PLACEIQ.NET > ', > add=True): CertificateOperationError > > > > > > > > Jim Richard > > > > SYSTEM ADMINISTRATOR III > /(646) 338-8905 / > > > PlaceIQ:Location > Data Accuracy > > > > >> On Sep 29, 2016, at 8:11 AM, Rob Crittenden > > wrote: >> >> Natxo Asenjo wrote: >>> hi Jim, >>> >>> On Thu, Sep 29, 2016 at 7:37 AM, Jim Richard >> >>> > wrote: >>> >>> Thanks Rob, that worked. >>> >>> Still on the subject of certs, any idea how to solve this error: >>> >>> Certificate format error: (SEC_ERROR_LEGACY_DATABASE) The >>> certificate/key database is in an old, unsupported format. >>> >>> I see that in the gui when querying hosts as well as from cli when I >>> ipa-show or ipa-find >>> >>> >>> I have had this too, and we did not find a solution (search my recent >>> posts on the archives). As a workaround I have created replicas and >>> decommissioned the older replicas. >> >> On the one hand I'm glad this fixed it for you. On the other it is a >> rather unsatisfying answer. Unfortunately NSS doesn't always provide >> the most context with its error messages. This error is usually seen >> when one tries to open a non-existent database, which in this case is >> a very strange thing, especially since it goes from working to >> non-working in the same apache process over a few minutes. >> >> I'm not sure how I'd troubleshoot this if it were easily reproducible. >> I suspect we'd need to figure out which database cannot be found (most >> likely /etc/httpd/alias) and go from there. An strace is a brute-force >> way to see the file open but finding the right process to attach to is >> a bit of an art. >> >> rob >> >> -- >> Manage your subscription for the Freeipa-users mailing list: >> https://www.redhat.com/mailman/listinfo/freeipa-users >> Go to http://freeipa.org for more info on the project > From rcritten at redhat.com Fri Sep 30 08:53:10 2016 From: rcritten at redhat.com (Rob Crittenden) Date: Fri, 30 Sep 2016 10:53:10 +0200 Subject: [Freeipa-users] Replica created with expired certs In-Reply-To: References: <5EE38BB7-ED9B-4001-8F64-DB7CD2319C02@placeiq.com> <57EBAD23.4090604@redhat.com> <17A0F251-F5DF-447C-AB64-C053E83369D0@placeiq.com> <57ED04DF.2040707@redhat.com> <0B1B6EC8-C4DF-4BFB-9157-51A0814DB73C@placeiq.com> Message-ID: <57EE27F6.3080904@redhat.com> Jim Richard wrote: > Can I and how? > > delete all certs for all hosts > > I mean, we only use FreeIPA for user login/sssd > > That said, do we even need those certs? There is no simple answer, really. Yes, you can deleted all certs for all hosts (not recommended as some of those are for IPA services). I doubt it would do anything positive and if the certificate is tracked by certmonger on the client it would eventually renew. Do you need the certs? Only you would know that, but chances are the vast majority aren't being used. In 3.0 when a client is registered a host certificate is obtained for it. This certificate was never used and in 4.something it isn't requested at all unless an option is passed to ipa-client-install. rob > > > > > Jim Richard > > > > SYSTEM ADMINISTRATOR III > /(646) 338-8905 / > > > PlaceIQ:Location > Data Accuracy > > > > >> On Sep 29, 2016, at 8:53 PM, Jim Richard > > wrote: >> >> another interesting thing, my httpd/error_logs are constantly getting >> spammed with: (I removed the stuff between the single quotes) >> >> Notice those names don?t match, should they? >> >> Me thinks not since those ?principal=? items are ALMOST all hosts that >> no longer exist in the FreeIPA system. I rare few do exist. >> >> So, that?s weird :) >> >> [Thu Sep 29 20:44:59 2016] [error] ipa: INFO: >> host/aerospike-cl1-203.nym1.placeiq.net at PLACEIQ.NET >> : >> cert_request(u????????..', >> principal=u'host/sbtt-nyc1-028.thum01.nym1.placeiq.net at PLACEIQ.NET >> ', >> add=True): CertificateOperationError >> >> [Thu Sep 29 20:45:06 2016] [error] ipa: INFO: >> host/aerospike-cl2-210.nym1.placeiq.net at PLACEIQ.NET >> : >> cert_request(u????????..', >> principal=u'host/017.prod07.nym1.placeiq.net at PLACEIQ.NET >> ', >> add=True): CertificateOperationError >> >> [Thu Sep 29 20:45:09 2016] [error] ipa: INFO: >> host/adsgateway-14.nym1.placeiq.net at PLACEIQ.NET >> : >> cert_request(u?????????...', >> principal=u'host/025.prod07.nym1.placeiq.net at PLACEIQ.NET >> ', >> add=True): CertificateOperationError >> >> [Thu Sep 29 20:45:29 2016] [error] ipa: INFO: >> host/ttsandbox-022.nym1.placeiq.net at PLACEIQ.NET >> : >> cert_request(u??????????.', >> principal=u'host/sbtt-nyc1-022.thum01.nym1.placeiq.net at PLACEIQ.NET >> ', >> add=True): CertificateOperationError >> >> >> >> >> >> >> >> Jim Richard >> >> >> >> SYSTEM ADMINISTRATOR III >> /(646) 338-8905 / >> >> >> PlaceIQ:Location >> Data Accuracy >> >> >> >> >>> On Sep 29, 2016, at 8:11 AM, Rob Crittenden >> > wrote: >>> >>> Natxo Asenjo wrote: >>>> hi Jim, >>>> >>>> On Thu, Sep 29, 2016 at 7:37 AM, Jim Richard >>> >>>> > wrote: >>>> >>>> Thanks Rob, that worked. >>>> >>>> Still on the subject of certs, any idea how to solve this error: >>>> >>>> Certificate format error: (SEC_ERROR_LEGACY_DATABASE) The >>>> certificate/key database is in an old, unsupported format. >>>> >>>> I see that in the gui when querying hosts as well as from cli when I >>>> ipa-show or ipa-find >>>> >>>> >>>> I have had this too, and we did not find a solution (search my recent >>>> posts on the archives). As a workaround I have created replicas and >>>> decommissioned the older replicas. >>> >>> On the one hand I'm glad this fixed it for you. On the other it is a >>> rather unsatisfying answer. Unfortunately NSS doesn't always provide >>> the most context with its error messages. This error is usually seen >>> when one tries to open a non-existent database, which in this case is >>> a very strange thing, especially since it goes from working to >>> non-working in the same apache process over a few minutes. >>> >>> I'm not sure how I'd troubleshoot this if it were easily >>> reproducible. I suspect we'd need to figure out which database cannot >>> be found (most likely /etc/httpd/alias) and go from there. An strace >>> is a brute-force way to see the file open but finding the right >>> process to attach to is a bit of an art. >>> >>> rob >>> >>> -- >>> Manage your subscription for the Freeipa-users mailing list: >>> https://www.redhat.com/mailman/listinfo/freeipa-users >>> Go to http://freeipa.org for more info on the project >> > From mbasti at redhat.com Fri Sep 30 10:00:54 2016 From: mbasti at redhat.com (Martin Basti) Date: Fri, 30 Sep 2016 12:00:54 +0200 Subject: [Freeipa-users] RBAC - User Administrator - OTP tokens In-Reply-To: References: Message-ID: On 27.09.2016 17:16, Prashant Bapat wrote: > RBAC Role "User Administrator" should have access to all users OTP > tokens. Specifically to remove if some one has lost their token. We > get this a lot. > > I found no permissions that give this access. > > Can someone explain if this can be added easily either from the WebUI > or CLI. > > Thanks. > --Prashant > > > > Hello, OTP related access control is bounded with token owner and token manager, we don't have any system permission created for that. Feel free to open ticket (just for deleting OTP): https://fedorahosted.org/freeipa/newticket We will see if it is feasible. You can create your own permission in RBAC tab in permissions section and assign this to User Administrator privilege but be careful with extending permissions related to OTP, it may open an attack vector. http://www.freeipa.org/page/V4/OTP#Permissions Martin^2 -------------- next part -------------- An HTML attachment was scrubbed... URL: From natxo.asenjo at gmail.com Fri Sep 30 11:54:16 2016 From: natxo.asenjo at gmail.com (Natxo Asenjo) Date: Fri, 30 Sep 2016 13:54:16 +0200 Subject: [Freeipa-users] another certmonger question In-Reply-To: <57EE2623.3010303@redhat.com> References: <57EA5B17.8000307@redhat.com> <57ECF80E.6010406@redhat.com> <57EE2623.3010303@redhat.com> Message-ID: On Fri, Sep 30, 2016 at 10:45 AM, Rob Crittenden wrote: > Natxo Asenjo wrote: > >> >> >> On Thu, Sep 29, 2016 at 1:16 PM, Rob Crittenden > > wrote: >> >> Natxo Asenjo wrote: >> >> >> >> On Tue, Sep 27, 2016 at 1:42 PM, Rob Crittenden >> >> >> wrote: >> >> >> It's hard to say, it may in fact not be a problem. >> >> It is really a matter of what service the certificate(s) >> are related >> to. I'd look at the serial numbers and then correlate those >> to the >> issued certificates. >> >> I'd also do a service-find on the hostname to see if any >> services >> have certificates issued and with what serial numbers. >> >> >> I agree, it could be that. But just for testing I have created a >> vm, >> joined it to the domain and resubmitted the certificate. >> >> Now there are two valid host certificates with the same subject: >> >> >> $ ipa cert-find --subject=throwaway.unix.iriszorg.nl >> >> > > >> ---------------------- >> 2 certificates matched >> ---------------------- >> Serial number (hex): 0x3FFE0002 >> Serial number: 1073610754 >> Status: VALID >> Subject: CN=throwaway.unix.iriszorg.nl >> >> > >,O=UNIX.IRISZORG.NL >> >> >> >> Serial number (hex): 0x3FFE0003 >> Serial number: 1073610755 >> Status: VALID >> Subject: CN=throwaway.unix.iriszorg.nl >> >> > >,O=UNIX.IRISZORG.NL >> >> >> ---------------------------- >> Number of entries returned 2 >> ---------------------------- >> >> >> So it certmonger in this centos 6.8 32bit host is renewing but not >> having the old certificate revoked. >> >> >> I'd check the Apache log to find the cert_request call to see if you >> can see if there are any issues raised. It should be doing a >> cert_revoke at the same time. >> >> Can you should how this certificate is being tracked? >> >> >> sure: >> >> $ sudo getcert list >> Number of certificates and requests being tracked: 1. >> Request ID '20160929100945': >> status: MONITORING >> stuck: no >> key pair storage: >> type=NSSDB,location='/etc/pki/nssdb',nickname='IPA Machine Certificate - >> throwaway.unix.iriszorg.nl >> ',token='NSS Certificate DB' >> certificate: type=NSSDB,location='/etc/pki/nssdb',nickname='IPA >> Machine Certificate - throwaway.unix.iriszorg.nl >> ',token='NSS Certificate DB' >> CA: IPA >> issuer: CN=Certificate Authority,O=UNIX.IRISZORG.NL >> >> subject: CN=throwaway.unix.iriszorg.nl >> ,O=UNIX.IRISZORG.NL >> >> expires: 2018-09-30 10:13:17 UTC >> principal name: host/throwaway.unix.iriszorg.nl at UNIX.IRISZORG.NL >> >> key usage: >> digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment >> eku: id-kp-serverAuth,id-kp-clientAuth >> pre-save command: >> post-save command: >> track: yes >> auto-renew: yes >> >> now, let's resubmit: >> >> $ sudo ipa-getcert resubmit -i 20160929100945 >> Resubmitting "20160929100945" to "IPA". >> [jose.admin at throwaway ~]$ sudo getcert list >> Number of certificates and requests being tracked: 1. >> Request ID '20160929100945': >> status: MONITORING >> stuck: no >> key pair storage: >> type=NSSDB,location='/etc/pki/nssdb',nickname='IPA Machine Certificate - >> throwaway.unix.iriszorg.nl >> ',token='NSS Certificate DB' >> certificate: type=NSSDB,location='/etc/pki/nssdb',nickname='IPA >> Machine Certificate - throwaway.unix.iriszorg.nl >> ',token='NSS Certificate DB' >> CA: IPA >> issuer: CN=Certificate Authority,O=UNIX.IRISZORG.NL >> >> subject: CN=throwaway.unix.iriszorg.nl >> ,O=UNIX.IRISZORG.NL >> >> expires: 2018-09-30 20:41:28 UTC >> principal name: host/throwaway.unix.iriszorg.nl at UNIX.IRISZORG.NL >> >> key usage: >> digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment >> eku: id-kp-serverAuth,id-kp-clientAuth >> pre-save command: >> post-save command: >> track: yes >> auto-renew: yes >> >> so it has been successfully renewed. >> >> In the access_log of the kdc I see this: >> >> 172.20.4.228 - - [29/Sep/2016:22:41:27 +0200] "POST >> https://kdc03.unix.iriszorg.nl:443/ca/eeca/ca/profileSubmitSSLClient >> HTTP/1.1" 200 1913 >> 172.20.6.81 - host/throwaway.unix.iriszorg.nl at UNIX.IRISZORG.NL >> >> [29/Sep/2016:22:41:27 +0200] "POST /ipa/xml HTTP/1.1" 200 2929 >> >> and in the error_log: >> [Thu Sep 29 22:41:28.626669 2016] [:error] [pid 4617] ipa: INFO: >> [xmlserver] host/throwaway.unix.iriszorg.nl at UNIX.IRISZORG.NL >> : >> cert_request(u'MIID6DCCAtACAQAwQDEZMBcGA1UEChMQVU5JWC5JUklTW >> k9SRy5OTDEjMCEGA1UEAxMadGhyb3dhd2F5LnVuaXguaXJpc3pvcmcubmwwg >> gEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQC4jBk7V2D5pX12kYrr+ >> +lwsWq1UWHy6PM9O+B/GvxaI0JoARBrhR6MKI1Ev+DV2r5ukNNWHj5+/kKbt >> W9XI2XMZ9pIBSwG3SG4m9s3gQV3dGQjlRCcU+MgXiDxRtRy2Vdzd1fZ9xdB1 >> txH3ZnZfceTosNw4Jp3bm/VtPChWJeN6K671FLRCzJkI1KrC+LHfGbvyTtOi >> pB5O9t8RkN4Qh01r/rphPvt9Gh+/mTlHnmGP9+sseqHHsgv2fPvRQowpJD >> EytTX5w/8pLrUCATqJUYfxK5RDuwD1304p3WXDFLoU6p2xaR63h34muj1a5N >> V1CvQFqJapHB5B/w6uUbLzjg3AgMBAAGgggFhMHcGCSqGSIb3DQEJFDFqHmg >> ASQBQAEEAIABNAGEAYwBoAGkAbgBlACAAQwBlAHIAdABpAGYAaQBjAGEAdAB >> lACAALQAgAHQAaAByAG8AdwBhAHcAYQB5AC4AdQBuAGkAeAAuAGkAcgBpAHM >> AegBvAHIAZwAuAG4AbDCB5QYJKoZIhvcNAQkOMYHXMIHUMIGhBgNVHREBAQA >> EgZYwgZOgQAYKKwYBBAGCNxQCA6AyDDBob3N0L3Rocm93YXdheS51bml4Lml >> yaXN6b3JnLm5sQFVOSVguSVJJU1pPUkcuTkygTwYGKwYBBQICoEUwQ6ASGxB >> VTklYLklSSVNaT1JHLk5MoS0wK6ADAgEBoSQwIhsEaG9zdBsadGhyb3dhd2F >> 5LnVuaXguaXJpc3pvcmcubmwwDAYDVR0TAQH/BAIwADAgBgNVHQ4BAQAEFgQ >> UgXWL3vdW/I31tQxv5YjyMZy4x8kw! >> > DQYJKoZIhv > cNAQELBQADggEBAD674/oGYlQTQDSvwf0muYoxBsj1dc6gnArw0JJpGVCNMv > /J3FdgOLcOhxzZcOfZiQr4NdYoV+/6mISOhknMa4ErJhqSAWbUA+w3+lL3CH > fdDtNueUjZRbPZezcC0rhAlnXBT7iakjuhE56WkZz7AihEU8RAvnZfSRi1mh > ehf3wFRYKWuzK9AW1DTY/uGMmHXiFtvINpfAJ3yL66xPwTj4087nz9w4YUqN > yCX+hYL+7idCJeoMjDyCqYQpjFkdfZhRuNd+rrKWTgYvKN3w/5+ItefDCYy8 > py91V2kXS7BrsYjd+2YHtQ2AbjgIW2xpTr/+PetToZyL50oWCpduT5t+M=', > >> principal=u'host/throwaway.unix.iriszorg.nl at UNIX.IRISZORG.NL >> ', add=True, >> version=u'2.51'): SUCCESS >> >> and now I have 3 valid certificates: >> >> $ ipa cert-find --subject=throwaway.unix.iriszorg.nl >> >> ---------------------- >> 3 certificates matched >> ---------------------- >> Serial number (hex): 0xFF9000D >> Serial number: 267976717 >> Status: VALID >> Subject: CN=throwaway.unix.iriszorg.nl >> ,O=UNIX.IRISZORG.NL >> >> >> Serial number (hex): 0x3FFE0002 >> Serial number: 1073610754 >> Status: VALID >> Subject: CN=throwaway.unix.iriszorg.nl >> ,O=UNIX.IRISZORG.NL >> >> >> Serial number (hex): 0x3FFE0003 >> Serial number: 1073610755 >> Status: VALID >> Subject: CN=throwaway.unix.iriszorg.nl >> ,O=UNIX.IRISZORG.NL >> >> ---------------------------- >> Number of entries returned 3 >> ---------------------------- >> > > Ok, let me start by saying that this is not a bug in either certmonger or > dogtag. IPA is supposed to do the revocation in the cert_request command. > > The steps IPA _should_ be taking are: > > 1. Figure out if we are doing a certificate for a host or a service. > 2. See if the requester is allowed to manage this entry > 3. Look at the entry to see if it has a usercertificate attribute. If so > revoke that serial number, then clear the usercertificate value in the host > or service entry (via service_mod or host_mod) > 4. Request a new certificate > 5. Update IPA with the new value > > Does a certificate appear in ipa host-show throwaway.unix.iriszorg.nl, > and which certificate serial number? > $ ipa host-show throwaway Host name: throwaway.unix.iriszorg.nl Certificate: MIIE0DCCA7igAwIBAgIED/kADTANBgkqhkiG9w0BAQsFADA7MRkwFwYDVQQKExBVTklYLklSSVNaT1JHLk5MMR4wHAYDVQQDExVDZXJ0aWZpY2F0ZSBBdXRob3JpdHkwHhcNMTYwOTI5MjA0MTI4WhcNMTgwOTMwMjA0MTI4WjBAMRkwFwYDVQQKDBBVTklYLklSSVNaT1JHLk5MMSMwIQYDVQQDDBp0aHJvd2F3YXkudW5peC5pcmlzem9yZy5ubDCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBALiMGTtXYPmlfXaRiuv76XCxarVRYfLo8z074H8a/FojQmgBEGuFHowojUS/4NXavm6Q01YePn7+Qpu1b1cjZcxn2kgFLAbdIbib2zeBBXd0ZCOVEJxT4yBeIPFG1HLZV3N3V9n3F0HW3Efdmdl9x5Oiw3Dgmndub9W08KFYl43orrvUUtELMmQjUqsL4sd8Zu/JO06KkHk723xGQ3hCHTWv+umE++30aH7+ZOUeeYY/36yx6oceyC/Z8+9FCjCkkMTK1NfnD/ykutQIBOolRh/ErlEO7APXfTindZcMUuhTqnbFpHreHfia6PVrk1XUK9AWolqkcHkH/Dq5RsvOODcCAwEAAaOCAdUwggHRMB8GA1UdIwQYMBaAFKOX5IouuM8+6jPyvJPWI96phDZoMEIGCCsGAQUFBwEBBDYwNDAyBggrBgEFBQcwAYYmaHR0cDovL2lwYS1jYS51bml4LmlyaXN6b3JnLm5sL2NhL29jc3AwDgYDVR0PAQH/BAQDAgTwMB0GA1UdJQQWMBQGCCsGAQUFBwMBBggrBgEFBQcDAjB7BgNVHR8EdDByMHCgOKA2hjRodHRwOi8vaXBhLWNhLnVuaXguaXJpc3pvcmcubmwvaXBhL2NybC9NYXN0ZXJDUkwuYmluojSkMjAwMQ4wDAYDVQQKDAVpcGFjYTEeMBwGA1UEAwwVQ2VydGlmaWNhdGUgQXV0aG9yaXR5MB0GA1UdDgQWBBSBdYve91b8jfW1DG/liPIxnLjHyTCBngYDVR0RBIGWMIGToEAGCisGAQQBgjcUAgOgMgwwaG9zdC90aHJvd2F3YXkudW5peC5pcmlzem9yZy5ubEBVTklYLklSSVNaT1JHLk5MoE8GBisGAQUCAqBFMEOgEhsQVU5JWC5JUklTWk9SRy5OTKEtMCugAwIBAaEkMCIbBGhvc3QbGnRocm93YXdheS51bml4LmlyaXN6b3JnLm5sMA0GCSqGSIb3DQEBCwUAA4IBAQB6KplOoHG5d2+3c5J/TSE/qxWkfObqPdpzYSMg5ma+PKL7ofuEgtozfoZfH/GXKIKtUpPZCJL2NQQKauagdIwto7PX184FcohCjJxHD30RRbc6q2rm3PE7Q1vDzSxW1ZCFELmpvK0XN4YX1tz6iaPgo2B7wuyhbZpT4Vd2itT1rOQ6cAnAB7BFhVNj5XDPDoMaHabtWRJVLlIOMeGyrZDUMxmoPys5g1c1SZM1ld+8+zdgqIVEsdTo9qThqPraTUi8GTjP4QvuQkLbnlFO6KQe5RqfbVXA0gKVWDtepY7h+cBQxMa/eHROFtnhW/w1+FdKHDPvOdj8aTF1tSIU2RYP, MIIE0DCCA7igAwIBAgIEP/4AAjANBgkqhkiG9w0BAQsFADA7MRkwFwYDVQQKExBVTklYLklSSVNaT1JHLk5MMR4wHAYDVQQDExVDZXJ0aWZpY2F0ZSBBdXRob3JpdHkwHhcNMTYwOTI5MTAwOTUwWhcNMTgwOTMwMTAwOTUwWjBAMRkwFwYDVQQKDBBVTklYLklSSVNaT1JHLk5MMSMwIQYDVQQDDBp0aHJvd2F3YXkudW5peC5pcmlzem9yZy5ubDCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBALiMGTtXYPmlfXaRiuv76XCxarVRYfLo8z074H8a/FojQmgBEGuFHowojUS/4NXavm6Q01YePn7+Qpu1b1cjZcxn2kgFLAbdIbib2zeBBXd0ZCOVEJxT4yBeIPFG1HLZV3N3V9n3F0HW3Efdmdl9x5Oiw3Dgmndub9W08KFYl43orrvUUtELMmQjUqsL4sd8Zu/JO06KkHk723xGQ3hCHTWv+umE++30aH7+ZOUeeYY/36yx6oceyC/Z8+9FCjCkkMTK1NfnD/ykutQIBOolRh/ErlEO7APXfTindZcMUuhTqnbFpHreHfia6PVrk1XUK9AWolqkcHkH/Dq5RsvOODcCAwEAAaOCAdUwggHRMB8GA1UdIwQYMBaAFKOX5IouuM8+6jPyvJPWI96phDZoMEIGCCsGAQUFBwEBBDYwNDAyBggrBgEFBQcwAYYmaHR0cDovL2lwYS1jYS51bml4LmlyaXN6b3JnLm5sL2NhL29jc3AwDgYDVR0PAQH/BAQDAgTwMB0GA1UdJQQWMBQGCCsGAQUFBwMBBggrBgEFBQcDAjB7BgNVHR8EdDByMHCgOKA2hjRodHRwOi8vaXBhLWNhLnVuaXguaXJpc3pvcmcubmwvaXBhL2NybC9NYXN0ZXJDUkwuYmluojSkMjAwMQ4wDAYDVQQKDAVpcGFjYTEeMBwGA1UEAwwVQ2VydGlmaWNhdGUgQXV0aG9yaXR5MB0GA1UdDgQWBBSBdYve91b8jfW1DG/liPIxnLjHyTCBngYDVR0RBIGWMIGToEAGCisGAQQBgjcUAgOgMgwwaG9zdC90aHJvd2F3YXkudW5peC5pcmlzem9yZy5ubEBVTklYLklSSVNaT1JHLk5MoE8GBisGAQUCAqBFMEOgEhsQVU5JWC5JUklTWk9SRy5OTKEtMCugAwIBAaEkMCIbBGhvc3QbGnRocm93YXdheS51bml4LmlyaXN6b3JnLm5sMA0GCSqGSIb3DQEBCwUAA4IBAQCvTRaJrl3J7Ky4VkFVfkwIGoaxocXrllYSjXZzhzHV0zJtlVeQGmHwulyrEbEzaRuMqbXe7c8WseOgU/K+UwByGiZoyxUmHgBmu2mv8Cln48UbESEAm0py4hRMmE7UzIhsHzTAKjUfyQXujB21S+FYwd97QymGRgn7kJ2TtH99zslQO0kMC//LmctUxIfTOOcrBgOojIEpcbzTeWNcyuN5+MHr6H2DNUYQZpvnDBv7XVphrk7ACrh4ETeYW5E1fFl84CdSxWehhWILF6t2WdA4RSjvtg3zvMPL+uVU8w1aru33dMuCKqvMG3iaRrDjVZ4k9/36lpf4/r1PwKYxusvg, MIIE0DCCA7igAwIBAgIEP/4AAzANBgkqhkiG9w0BAQsFADA7MRkwFwYDVQQKExBVTklYLklSSVNaT1JHLk5MMR4wHAYDVQQDExVDZXJ0aWZpY2F0ZSBBdXRob3JpdHkwHhcNMTYwOTI5MTAxMzE3WhcNMTgwOTMwMTAxMzE3WjBAMRkwFwYDVQQKDBBVTklYLklSSVNaT1JHLk5MMSMwIQYDVQQDDBp0aHJvd2F3YXkudW5peC5pcmlzem9yZy5ubDCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBALiMGTtXYPmlfXaRiuv76XCxarVRYfLo8z074H8a/FojQmgBEGuFHowojUS/4NXavm6Q01YePn7+Qpu1b1cjZcxn2kgFLAbdIbib2zeBBXd0ZCOVEJxT4yBeIPFG1HLZV3N3V9n3F0HW3Efdmdl9x5Oiw3Dgmndub9W08KFYl43orrvUUtELMmQjUqsL4sd8Zu/JO06KkHk723xGQ3hCHTWv+umE++30aH7+ZOUeeYY/36yx6oceyC/Z8+9FCjCkkMTK1NfnD/ykutQIBOolRh/ErlEO7APXfTindZcMUuhTqnbFpHreHfia6PVrk1XUK9AWolqkcHkH/Dq5RsvOODcCAwEAAaOCAdUwggHRMB8GA1UdIwQYMBaAFKOX5IouuM8+6jPyvJPWI96phDZoMEIGCCsGAQUFBwEBBDYwNDAyBggrBgEFBQcwAYYmaHR0cDovL2lwYS1jYS51bml4LmlyaXN6b3JnLm5sL2NhL29jc3AwDgYDVR0PAQH/BAQDAgTwMB0GA1UdJQQWMBQGCCsGAQUFBwMBBggrBgEFBQcDAjB7BgNVHR8EdDByMHCgOKA2hjRodHRwOi8vaXBhLWNhLnVuaXguaXJpc3pvcmcubmwvaXBhL2NybC9NYXN0ZXJDUkwuYmluojSkMjAwMQ4wDAYDVQQKDAVpcGFjYTEeMBwGA1UEAwwVQ2VydGlmaWNhdGUgQXV0aG9yaXR5MB0GA1UdDgQWBBSBdYve91b8jfW1DG/liPIxnLjHyTCBngYDVR0RBIGWMIGToEAGCisGAQQBgjcUAgOgMgwwaG9zdC90aHJvd2F3YXkudW5peC5pcmlzem9yZy5ubEBVTklYLklSSVNaT1JHLk5MoE8GBisGAQUCAqBFMEOgEhsQVU5JWC5JUklTWk9SRy5OTKEtMCugAwIBAaEkMCIbBGhvc3QbGnRocm93YXdheS51bml4LmlyaXN6b3JnLm5sMA0GCSqGSIb3DQEBCwUAA4IBAQCh6lySZa1AyUyP8AuaLUDj6X0Lt/tGS+ZIw/O248FVMJDwvLvkFUxOjTAK1mip0AHxkib+QtKqFgN9lbidnxeKFYNN2komTfLgFV+G+8kBIInxWbU1OsuYw4J6xCu5IE+F7jfdHX1yw6HSgDixYgKHe9mw+8HTbUR1a/ntZ90pmai8I7daem9bMrPHGSSChjcbjif6YNZ8ibmilqq0vw8CEwQopXFToO/mHfbXNDw6gJY5rKu19fWPi3VRQdQxKKtwY/gXg39q4FWBymDaMwjErC7G4AnGeeTYp4iFYZkfcjYvdxGXGF0CpLgunvcMMQ0rTYx5w1MrLbbnqjq1qBZO Principal name: host/throwaway.unix.iriszorg.nl at UNIX.IRISZORG.NL Password: False Keytab: True Managed by: throwaway.unix.iriszorg.nl Subject: CN=throwaway.unix.iriszorg.nl,O=UNIX.IRISZORG.NL Serial Number: 267976717 Serial Number (hex): 0xFF9000D Issuer: CN=Certificate Authority,O=UNIX.IRISZORG.NL Not Before: Thu Sep 29 20:41:28 2016 UTC Not After: Sun Sep 30 20:41:28 2018 UTC Fingerprint (MD5): 52:a1:06:a1:39:27:bc:ed:dd:45:f5:36:32:11:99:c1 Fingerprint (SHA1): 81:d4:01:5a:26:83:9c:c4:fb:76:fb:c3:29:cd:32:c1:8a:4c:eb:45 SSH public key fingerprint: 61:66:4D:D7:E6:83:B3:31:BB:50:C3:28:11:79:FD:42 (ssh-rsa), 71:80:40:26:50:64:CD:FE:9A:FB:8D:DA:55:56:18:95 (ssh-dss) so it shows the three certificates but the serial is 267976717 -- Groeten, natxo -------------- next part -------------- An HTML attachment was scrubbed... URL: From matt.wells at mosaic451.com Fri Sep 30 20:41:20 2016 From: matt.wells at mosaic451.com (Matt Wells) Date: Fri, 30 Sep 2016 13:41:20 -0700 Subject: [Freeipa-users] cleanallruv - no replica's :( Message-ID: Hey all I hoped anyone may be able to assist. I had 2 dead replica's and use the cleanallruv.pl as they refused to leave otherwise. ` /usr/sbin/cleanallruv.pl -v -D "cn=directory manager" -w - -b 'dc=mosaic451,dc=com' -r 17 ` 17 being the bad guy. Well it ran `woohoo` but deleted all of my replica's. The state it's in now is I can make changes on Box1 ( the one I ran it on ) and they replicate to Box2 but never come back. If I delete it on Box2 it never get's to Box1 however Box2 say's he has that happy replication agreement. So it's almost a split brain scenario. I hoped someone may be able to assist. Can I just re-cut the replication agreement from Box2 and run it on Box1; he's a full grown IPA so if I did that wouldn't I need to --uninstall him? What do you guys think? -------------- next part -------------- An HTML attachment was scrubbed... URL: