[Freeipa-users] Replication has stopped and server errors

Mon Jan 9 19:30:57 UTC 2017

I am happy to report this appears to be resolved. I found this post: https://www.redhat.com/archives/freeipa-users/2014-February/msg00007.html which pointed me to the csn skew issue which was causing all my replication failures. I performed the steps in the post and things look much better so far.
Thank you.

      From: sipazzo <sipazzo at yahoo.com>
 To: Martin Basti <mbasti at redhat.com>; Freeipa-users <freeipa-users at redhat.com> 
 Sent: Friday, January 6, 2017 1:03 PM
 Subject: Re: [Freeipa-users] Replication has stopped and server errors

I have changed the number of db locks to 40000. After restart, each server reports a lot of these type errors:
DSRetroclPlugin - delete_changerecord: could not delete change record 6038434 

As well as immediately coming up with these errors (even after re-initializing)

06/Jan/2017:12:10:12 -0800] NSMMReplicationPlugin - changelog program - agmt="cn=meToipa1-dev.example.local" (ipa1-corp:389): CSN 586d8aab000400110000 not found, we aren't as up to date, or we purged
[06/Jan/2017:12:10:12 -0800] NSMMReplicationPlugin - agmt="cn=meToipa1-corp.example.local" (ipa1-dev:389): Data required to update replica has been purged. The replica must be reinitialized.
[06/Jan/2017:12:10:12 -0800] NSMMReplicationPlugin - agmt="cn=meToipa1-prod.example.local" (ipa1-xo:389): Incremental update failed and requires administrator action
[06/Jan/2017:12:10:12 -0800] NSMMReplicationPlugin - agmt="cn=meToipa1-dev.example.local" (ipa1-corp:389): Incremental update failed and requires administrator action
06/Jan/2017:12:15:47 -0800] agmt="cn=meToipa1-dr.example.local" (ipa1-io:389) - Can't locate CSN 586ffaf5000300100000 in the changelog (DB rc=-30988). If replication stops, the consumer may need to be reinitialized.
[06/Jan/2017:12:15:49 -0800] agmt="cn=meToipa1-dr.example.local" (ipa1-io:389) - Can't locate CSN 586ffaf7000000100000 in the changelog (DB rc=-30988). If replication stops, the consumer may need to be reinitialized.

Replication topology is:3 geographic locations each with 2 ipa servers (dr, prod, dev)
ipa1-dev replicates with all servers (ipa2-dev,ipa1-dr, ipa2-dr, ipa1-prod, ipa2-prod)ipa1-dr also replicates with ipa2-dripa1-prod also replicates with ipa2-prod

As a test I deleted one host on each of the servers. I have waited 30 minutes and the results are:ipa1-dev - deletion replicated to all serversipa2-dr - deletion replicated to all servers
ipa1-dr, ipa1-prod, ipa2-dev, ipa2-prod - deletions not replicated

      From: Martin Basti <mbasti at redhat.com>
 To: sipazzo <sipazzo at yahoo.com>; Freeipa-users <freeipa-users at redhat.com> 
 Sent: Friday, January 6, 2017 8:58 AM
 Subject: Re: [Freeipa-users] Replication has stopped and server errors

 On 06.01.2017 00:29, sipazzo wrote:

  I have 6 ipa servers in 3 locations running 4.2.0-15.0.1on RHEL 7. Ipa1-dev is the CA Renewal and CRL Master server and where most of our updates  (host enrollment, password changes) end up taking place.   Servers had been running fine. Over the holidays we started having some replication issues and looking at /var/log/dirsrv/slapd-REALM-COM/errors showed the following: 
  All servers currently have these errors for each replica the respective IPA servers are connected to: NSMMReplicationPlugin - agmt="cn=meToipa2-dr.example.local" (ipa2-dr:389): Incremental update failed and requires administrator action [04/Jan/2017:15:39:48 -0800] agmt="cn=meToipa1-dr.example.local" (ipa1-dr:389) - Can't locate CSN 583c8e74000600110000 in the changelog (DB  rc=-30988). If replication stops, the consumer may need to be reinitialized NSMMReplicationPlugin - agmt="cn=meToipa1-prod.example.local" (ipa1-prod:389): Data required to update replica has been purged. The replica must be reinitialized. [04/Jan/2017:13:33:26 -0800] NSMMReplicationPlugin - agmt="cn=meToipa2-dev.example.local" (ipa2-dev:389): Incremental update failed and requires administrator action  [04/Jan/2017:13:33:26 -0800] NSMMReplicationPlugin - agmt="cn=meToipa1-prod.example.local" (ipa1-prod:389): Incremental update failed and requires administrator action [04/Jan/2017:13:33:27 -0800] agmt="cn=meToipa2-prod.example.local" (ipa2-prod:389) - Can't locate CSN 586d69f0000400120000 in the changelog (DB rc=-30988). If replication stops, the consumer may need to be reinitialized.   And all servers have these types of errors which are worrisome but they go back quite a way
  NSACLPlugin - The ACL target cn=dns,dc=example,dc=local does not exist NSACLPlugin - The ACL target cn=dns,dc=example,dc=local does not exist NSACLPlugin - The ACL target cn=groups,cn=compat,dc=example,dc=local does not exist NSACLPlugin - The ACL target cn=computers,cn=compat,dc=example,dc=local does not exist NSACLPlugin - The ACL target cn=casigningcert cert-pki-ca,cn=ca_renewal,cn=ipa,cn=etc,dc=example,dc=local does not exist NSACLPlugin - The ACL target cn=casigningcert cert-pki-ca,cn=ca_renewal,cn=ipa,cn=etc,dc=example,dc=local does not exist NSACLPlugin - The ACL target ou=sudoers,dc=networkfleet,dc=local does not exist  
 ^^^ just INFO messages, you can ignore them

    All servers except one have a lot of these DSRetroclPlugin - delete_changerecord: could not delete change record   Ipa1-dev only has this
  04/Jan/2017:18:36:52 -0800] NSMMReplicationPlugin -agmt="cn=masterAgreement1-ipa1-prod.example.local-pki-tomcat" (ipa1-prod:389): Replication bind with SIMPLE auth resumed [04/Jan/2017:18:36:52 -0800] NSMMReplicationPlugin - agmt="cn=masterAgreement1-ipa2-dr.example.local-pki-tomcat" (ipa2-dr:389): Replication bind with SIMPLE auth resumed [04/Jan/2017:18:36:52 -0800] NSMMReplicationPlugin - agmt="cn=masterAgreement1-ipa1-dr.example.local-pki-tomcat" (ipa1-dr:389): Replication bind with SIMPLE auth resumed [04/Jan/2017:18:36:53 -0800] NSMMReplicationPlugin -agmt="cn=masterAgreement1-ipa2-prod.example.local-pki-tomcat" (ipa2-prod:389): Replication bind with SIMPLE auth resumed   3 servers (ipa1-dr ipa2-dr ipa2-prod) have these errors:  [01/Jan/2017:14:43:06 -0800] - libdb: BDB2055 Lock table is out of available lock entries [01/Jan/2017:14:43:06 -0800] - compactdb: failed to compact changelog; db error - 12 Cannot allocate memory  

 you probably need https://access.redhat.com/solutions/1241063 to increase number of locks (or in this threadhttps://lists.fedoraproject.org/pipermail/389-users/2011-June/013299.html)

 I would first increase the number of locks, and then look if something improved.
 We also don't know how your topology looks like, which servers are connected together.

 Martin

    4 servers (ipa1-dev, ipa2-dev, ipa1-dr and ipa2-dr) have these errors
  [04/Jan/2017:15:37:21 -0800] slapd_ldap_sasl_interactive_bind - Error: could not perform interactive bind for id [] mech [GSSAPI]: LDAP error -1 (Can't contact LDAP server) ((null)) errno 107 (Transport endpoint is not connected) [04/Jan/2017:15:37:24 -0800] slapd_ldap_sasl_interactive_bind - Error: could not perform interactive bind for id [] mech [GSSAPI]: LDAP error -1 (Can't contact LDAP server) ((null)) errno 107 (Transport endpoint is not connected)  
  I have tried various combinations or restarting, re-initializing, disconnecting and reconnecting replicas but am down to only two servers replicating with each other currently (ipa1-dev and ipa2-dev). We did have a power outage at the dev location but it does not seem to correspond to when the errors started? Not sure how to recover from this. Any help is appreciated      

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/freeipa-users/attachments/20170109/9b733623/attachment.htm>