[Freeipa-users] Replication has stopped and server errors

[sipazzo]

I have 6 ipaservers in 3 locations running 4.2.0-15.0.1on RHEL 7. Ipa1-dev is the CARenewal and CRL Master server and where most of our updates (host enrollment,password changes) end up taking place. Servers hadbeen running fine. Over the holidays we started having some replication issuesand looking at /var/log/dirsrv/slapd-REALM-COM/errors showed the following:
All servers currently have these errors for each replica the respective IPA servers areconnected to:NSMMReplicationPlugin- agmt="cn=meToipa2-dr.example.local" (ipa2-dr:389): Incrementalupdate failed and requires administrator action[04/Jan/2017:15:39:48-0800] agmt="cn=meToipa1-dr.example.local" (ipa1-dr:389) - Can'tlocate CSN 583c8e74000600110000 in the changelog (DB rc=-30988). If replicationstops, the consumer may need to be reinitializedNSMMReplicationPlugin- agmt="cn=meToipa1-prod.example.local" (ipa1-prod:389): Datarequired to update replica has been purged. The replica must be reinitialized.[04/Jan/2017:13:33:26-0800] NSMMReplicationPlugin - agmt="cn=meToipa2-dev.example.local"(ipa2-dev:389): Incremental update failed and requires administrator action [04/Jan/2017:13:33:26 -0800]NSMMReplicationPlugin - agmt="cn=meToipa1-prod.example.local"(ipa1-prod:389): Incremental update failed and requires administrator action[04/Jan/2017:13:33:27-0800] agmt="cn=meToipa2-prod.example.local" (ipa2-prod:389) - Can'tlocate CSN 586d69f0000400120000 in the changelog (DB rc=-30988). If replicationstops, the consumer may need to be reinitialized. And allservers have these types of errors which are worrisome but they go back quite a way
NSACLPlugin - The ACL target cn=dns,dc=example,dc=localdoes not existNSACLPlugin - The ACL target cn=dns,dc=example,dc=localdoes not existNSACLPlugin - The ACL targetcn=groups,cn=compat,dc=example,dc=local does not existNSACLPlugin - The ACL targetcn=computers,cn=compat,dc=example,dc=local does not existNSACLPlugin - The ACL target cn=casigningcertcert-pki-ca,cn=ca_renewal,cn=ipa,cn=etc,dc=example,dc=local does not existNSACLPlugin - The ACL target cn=casigningcertcert-pki-ca,cn=ca_renewal,cn=ipa,cn=etc,dc=example,dc=local does not existNSACLPlugin - The ACL targetou=sudoers,dc=networkfleet,dc=local does not exist All servers except one have a lot of theseDSRetroclPlugin - delete_changerecord: could not delete change record Ipa1-dev only has this
04/Jan/2017:18:36:52-0800] NSMMReplicationPlugin -agmt="cn=masterAgreement1-ipa1-prod.example.local-pki-tomcat"(ipa1-prod:389): Replication bind with SIMPLE auth resumed[04/Jan/2017:18:36:52-0800] NSMMReplicationPlugin -agmt="cn=masterAgreement1-ipa2-dr.example.local-pki-tomcat"(ipa2-dr:389): Replication bind with SIMPLE auth resumed[04/Jan/2017:18:36:52-0800] NSMMReplicationPlugin -agmt="cn=masterAgreement1-ipa1-dr.example.local-pki-tomcat"(ipa1-dr:389): Replication bind with SIMPLE auth resumed[04/Jan/2017:18:36:53-0800] NSMMReplicationPlugin -agmt="cn=masterAgreement1-ipa2-prod.example.local-pki-tomcat"(ipa2-prod:389): Replication bind with SIMPLE auth resumed 3 servers(ipa1-dr ipa2-dr ipa2-prod) have these errors: [01/Jan/2017:14:43:06 -0800] - libdb: BDB2055 Lock table is out ofavailable lock entries[01/Jan/2017:14:43:06 -0800] - compactdb: failed to compact changelog;db error - 12 Cannot allocate memory 4 servers (ipa1-dev, ipa2-dev, ipa1-dr and ipa2-dr) have these errors
[04/Jan/2017:15:37:21 -0800] slapd_ldap_sasl_interactive_bind - Error:could not perform interactive bind for id [] mech [GSSAPI]: LDAP error -1(Can't contact LDAP server) ((null)) errno 107 (Transport endpoint isnot connected)[04/Jan/2017:15:37:24 -0800] slapd_ldap_sasl_interactive_bind - Error:could not perform interactive bind for id [] mech [GSSAPI]: LDAP error -1(Can't contact LDAP server) ((null)) errno 107 (Transport endpoint isnot connected) 
I have tried various combinations or restarting, re-initializing, disconnecting and reconnecting replicas but am down toonly two servers replicating with each other currently (ipa1-dev and ipa2-dev). We did have a power outageat the dev location but it does not seem to correspond to when the errors started? Not sure howto recover from this. Any help is appreciated  
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/freeipa-users/attachments/20170105/86d51cac/attachment.htm>