<html><head></head><body><div style="color:#000; background-color:#fff; font-family:bookman old style, new york, times, serif;font-size:13px"><div id="yui_3_16_0_ym19_1_1483645394100_4471">I have<span style="font-size:10.0pt;font-family:Calibri;mso-ascii-theme-font:major-latin;mso-hansi-theme-font:major-latin" id="yui_3_16_0_ym19_1_1483645394100_4472"> 6 ipa
servers in 3 locations running 4.2.0-15.0.1on RHEL 7. Ipa1-dev is the CA
Renewal and CRL Master server and where most of our updates (host enrollment,
password changes) end up taking place.</span></div>
<div id="yui_3_16_0_ym19_1_1483645394100_4473"><span style="font-size:10.0pt;font-family:Calibri;mso-ascii-theme-font:major-latin;mso-hansi-theme-font:major-latin" id="yui_3_16_0_ym19_1_1483645394100_4474"> </span></div>
<div id="yui_3_16_0_ym19_1_1483645394100_4475"><span style="font-size:10.0pt;font-family:Calibri;mso-ascii-theme-font:major-latin;mso-hansi-theme-font:major-latin" id="yui_3_16_0_ym19_1_1483645394100_4476">Servers had
been running fine. Over the holidays we started having some replication issues
and looking at /var/log/dirsrv/slapd-REALM-COM/errors showed the following:</span></div><div id="yui_3_16_0_ym19_1_1483645394100_6911"><br></div><div id="yui_3_16_0_ym19_1_1483645394100_4670"><span style="font-size:10.0pt;font-family:Calibri;mso-ascii-theme-font:major-latin;mso-hansi-theme-font:major-latin;mso-bidi-font-family:Calibri" id="yui_3_16_0_ym19_1_1483645394100_4671">All servers currently have these errors for each replica the respective IPA servers are
connected to:</span></div>
<div id="yui_3_16_0_ym19_1_1483645394100_4672"><span style="font-size:10.0pt;font-family:Calibri;mso-ascii-theme-font:major-latin;mso-hansi-theme-font:major-latin" id="yui_3_16_0_ym19_1_1483645394100_4673">NSMMReplicationPlugin
- agmt="cn=meToipa2-dr.example.local" (ipa2-dr:389): Incremental
update failed and requires administrator action</span></div>
<div id="yui_3_16_0_ym19_1_1483645394100_4674"><span style="font-size:10.0pt;font-family:Calibri;mso-ascii-theme-font:major-latin;mso-hansi-theme-font:major-latin" id="yui_3_16_0_ym19_1_1483645394100_4675">[04/Jan/2017:15:39:48
-0800] agmt="cn=meToipa1-dr.example.local" (ipa1-dr:389) - Can't
locate CSN 583c8e74000600110000 in the changelog (DB rc=-30988). If replication
stops, the consumer may need to be reinitialized</span></div>
<div id="yui_3_16_0_ym19_1_1483645394100_4676"><span style="font-size:10.0pt;font-family:Calibri;mso-ascii-theme-font:major-latin;mso-hansi-theme-font:major-latin" id="yui_3_16_0_ym19_1_1483645394100_4677">NSMMReplicationPlugin
- agmt="cn=meToipa1-prod.example.local" (ipa1-prod:389): Data
required to update replica has been purged. The replica must be reinitialized.</span></div>
<div id="yui_3_16_0_ym19_1_1483645394100_4678"><span style="font-size:10.0pt;font-family:Calibri;mso-ascii-theme-font:major-latin;mso-hansi-theme-font:major-latin" id="yui_3_16_0_ym19_1_1483645394100_4679">[04/Jan/2017:13:33:26
-0800] NSMMReplicationPlugin - agmt="cn=meToipa2-dev.example.local"
(ipa2-dev:389): Incremental update failed and requires administrator action</span></div>
<div id="yui_3_16_0_ym19_1_1483645394100_4680"><span style="font-size:10.0pt;font-family:Calibri;mso-ascii-theme-font:major-latin;mso-hansi-theme-font:major-latin" id="yui_3_16_0_ym19_1_1483645394100_4681"><span style="mso-spacerun:yes" id="yui_3_16_0_ym19_1_1483645394100_4682"> </span>[04/Jan/2017:13:33:26 -0800]
NSMMReplicationPlugin - agmt="cn=meToipa1-prod.example.local"
(ipa1-prod:389): Incremental update failed and requires administrator action</span></div>
<div id="yui_3_16_0_ym19_1_1483645394100_4683"><span style="font-size:10.0pt;font-family:Calibri;mso-ascii-theme-font:major-latin;mso-hansi-theme-font:major-latin" id="yui_3_16_0_ym19_1_1483645394100_4684">[04/Jan/2017:13:33:27
-0800] agmt="cn=meToipa2-prod.example.local" (ipa2-prod:389) - Can't
locate CSN 586d69f0000400120000 in the changelog (DB rc=-30988). If replication
stops, the consumer may need to be reinitialized.</span></div>
<div id="yui_3_16_0_ym19_1_1483645394100_4685"><span style="font-size:10.0pt;font-family:Calibri;mso-ascii-theme-font:major-latin;mso-hansi-theme-font:major-latin" id="yui_3_16_0_ym19_1_1483645394100_4686"> </span></div>
<div id="yui_3_16_0_ym19_1_1483645394100_4687"><span style="font-size:10.0pt;font-family:Calibri;mso-ascii-theme-font:major-latin;mso-hansi-theme-font:major-latin" id="yui_3_16_0_ym19_1_1483645394100_4688">And all
servers have these types of errors which are worrisome but they go back quite a way<br></span></div>
<div id="yui_3_16_0_ym19_1_1483645394100_4689"><b id="yui_3_16_0_ym19_1_1483645394100_4690"><span style="font-size:10.0pt;font-family:Calibri;mso-ascii-theme-font:major-latin;mso-hansi-theme-font:major-latin" id="yui_3_16_0_ym19_1_1483645394100_4691">NSACL</span></b><span style="font-size:10.0pt;font-family:Calibri;mso-ascii-theme-font:major-latin;mso-hansi-theme-font:major-latin" id="yui_3_16_0_ym19_1_1483645394100_4692">Plugin - The ACL target cn=dns,dc=example,dc=local
does not exist</span></div>
<div id="yui_3_16_0_ym19_1_1483645394100_4693"><b id="yui_3_16_0_ym19_1_1483645394100_4694"><span style="font-size:10.0pt;font-family:Calibri;mso-ascii-theme-font:major-latin;mso-hansi-theme-font:major-latin" id="yui_3_16_0_ym19_1_1483645394100_4695">NSACL</span></b><span style="font-size:10.0pt;font-family:Calibri;mso-ascii-theme-font:major-latin;mso-hansi-theme-font:major-latin" id="yui_3_16_0_ym19_1_1483645394100_4696">Plugin - The ACL target cn=dns,dc=example,dc=local
does not exist</span></div>
<div id="yui_3_16_0_ym19_1_1483645394100_4697"><b id="yui_3_16_0_ym19_1_1483645394100_4698"><span style="font-size:10.0pt;font-family:Calibri;mso-ascii-theme-font:major-latin;mso-hansi-theme-font:major-latin" id="yui_3_16_0_ym19_1_1483645394100_4699">NSACL</span></b><span style="font-size:10.0pt;font-family:Calibri;mso-ascii-theme-font:major-latin;mso-hansi-theme-font:major-latin" id="yui_3_16_0_ym19_1_1483645394100_4700">Plugin - The ACL target
cn=groups,cn=compat,dc=example,dc=local does not exist</span></div>
<div id="yui_3_16_0_ym19_1_1483645394100_4701"><b id="yui_3_16_0_ym19_1_1483645394100_4702"><span style="font-size:10.0pt;font-family:Calibri;mso-ascii-theme-font:major-latin;mso-hansi-theme-font:major-latin" id="yui_3_16_0_ym19_1_1483645394100_4703">NSACL</span></b><span style="font-size:10.0pt;font-family:Calibri;mso-ascii-theme-font:major-latin;mso-hansi-theme-font:major-latin" id="yui_3_16_0_ym19_1_1483645394100_4704">Plugin - The ACL target
cn=computers,cn=compat,dc=example,dc=local does not exist</span></div>
<div id="yui_3_16_0_ym19_1_1483645394100_4705"><b id="yui_3_16_0_ym19_1_1483645394100_4706"><span style="font-size:10.0pt;font-family:Calibri;mso-ascii-theme-font:major-latin;mso-hansi-theme-font:major-latin" id="yui_3_16_0_ym19_1_1483645394100_4707">NSACL</span></b><span style="font-size:10.0pt;font-family:Calibri;mso-ascii-theme-font:major-latin;mso-hansi-theme-font:major-latin" id="yui_3_16_0_ym19_1_1483645394100_4708">Plugin - The ACL target cn=casigningcert
cert-pki-ca,cn=ca_renewal,cn=ipa,cn=etc,dc=example,dc=local does not exist</span></div>
<div id="yui_3_16_0_ym19_1_1483645394100_4709"><b id="yui_3_16_0_ym19_1_1483645394100_4710"><span style="font-size:10.0pt;font-family:Calibri;mso-ascii-theme-font:major-latin;mso-hansi-theme-font:major-latin" id="yui_3_16_0_ym19_1_1483645394100_4711">NSACL</span></b><span style="font-size:10.0pt;font-family:Calibri;mso-ascii-theme-font:major-latin;mso-hansi-theme-font:major-latin" id="yui_3_16_0_ym19_1_1483645394100_4712">Plugin - The ACL target cn=casigningcert
cert-pki-ca,cn=ca_renewal,cn=ipa,cn=etc,dc=example,dc=local does not exist</span></div>
<div id="yui_3_16_0_ym19_1_1483645394100_4713"><b id="yui_3_16_0_ym19_1_1483645394100_4714"><span style="font-size:10.0pt;font-family:Calibri;mso-ascii-theme-font:major-latin;mso-hansi-theme-font:major-latin" id="yui_3_16_0_ym19_1_1483645394100_4715">NSACL</span></b><span style="font-size:10.0pt;font-family:Calibri;mso-ascii-theme-font:major-latin;mso-hansi-theme-font:major-latin" id="yui_3_16_0_ym19_1_1483645394100_4716">Plugin - The ACL target
ou=sudoers,dc=networkfleet,dc=local does not exist</span></div>
<div id="yui_3_16_0_ym19_1_1483645394100_4717"><span style="font-size:10.0pt;font-family:Calibri;mso-ascii-theme-font:major-latin;mso-hansi-theme-font:major-latin;mso-bidi-font-family:Calibri" id="yui_3_16_0_ym19_1_1483645394100_4718"> </span></div>
<div id="yui_3_16_0_ym19_1_1483645394100_4719"><span style="font-size:10.0pt;font-family:Calibri;mso-ascii-theme-font:major-latin;mso-hansi-theme-font:major-latin;mso-bidi-font-family:Calibri" id="yui_3_16_0_ym19_1_1483645394100_4720">All servers except one have a lot of these</span></div>
<div dir="ltr" id="yui_3_16_0_ym19_1_1483645394100_4721"><span style="font-size:10.0pt;font-family:Calibri;mso-ascii-theme-font:major-latin;mso-hansi-theme-font:major-latin;mso-bidi-font-family:Calibri" id="yui_3_16_0_ym19_1_1483645394100_4722">DSRetroclPlugin - delete_changerecord: could not delete change record</span></div>
<div id="yui_3_16_0_ym19_1_1483645394100_4477"><span style="font-size:10.0pt;font-family:Calibri;mso-ascii-theme-font:major-latin;mso-hansi-theme-font:major-latin" id="yui_3_16_0_ym19_1_1483645394100_4478"> </span></div>
<div id="yui_3_16_0_ym19_1_1483645394100_4479"><span style="font-size:10.0pt;font-family:Calibri;mso-ascii-theme-font:major-latin;mso-hansi-theme-font:major-latin" id="yui_3_16_0_ym19_1_1483645394100_4480">Ipa1-dev only has this<br></span></div>
<div id="yui_3_16_0_ym19_1_1483645394100_4481"><span style="font-size:10.0pt;font-family:Calibri;mso-ascii-theme-font:major-latin;mso-hansi-theme-font:major-latin" id="yui_3_16_0_ym19_1_1483645394100_4482">04/Jan/2017:18:36:52
-0800] NSMMReplicationPlugin -
agmt="cn=masterAgreement1-ipa1-prod.example.local-pki-tomcat"
(ipa1-prod:389): Replication bind with <b id="yui_3_16_0_ym19_1_1483645394100_4483">SIMPLE</b> auth resumed</span></div>
<div id="yui_3_16_0_ym19_1_1483645394100_4484"><span style="font-size:10.0pt;font-family:Calibri;mso-ascii-theme-font:major-latin;mso-hansi-theme-font:major-latin" id="yui_3_16_0_ym19_1_1483645394100_4485">[04/Jan/2017:18:36:52
-0800] NSMMReplicationPlugin -
agmt="cn=masterAgreement1-ipa2-dr.example.local-pki-tomcat"
(ipa2-dr:389): Replication bind with <b id="yui_3_16_0_ym19_1_1483645394100_4486">SIMPLE</b> auth resumed</span></div>
<div id="yui_3_16_0_ym19_1_1483645394100_4487"><span style="font-size:10.0pt;font-family:Calibri;mso-ascii-theme-font:major-latin;mso-hansi-theme-font:major-latin" id="yui_3_16_0_ym19_1_1483645394100_4488">[04/Jan/2017:18:36:52
-0800] NSMMReplicationPlugin -
agmt="cn=masterAgreement1-ipa1-dr.example.local-pki-tomcat"
(ipa1-dr:389): Replication bind with <b id="yui_3_16_0_ym19_1_1483645394100_4489">SIMPLE</b> auth resumed</span></div>
<div id="yui_3_16_0_ym19_1_1483645394100_4490"><span style="font-size:10.0pt;font-family:Calibri;mso-ascii-theme-font:major-latin;mso-hansi-theme-font:major-latin" id="yui_3_16_0_ym19_1_1483645394100_4491">[04/Jan/2017:18:36:53
-0800] NSMMReplicationPlugin -
agmt="cn=masterAgreement1-ipa2-prod.example.local-pki-tomcat"
(ipa2-prod:389): Replication bind with <b id="yui_3_16_0_ym19_1_1483645394100_4492">SIMPLE</b> auth resumed</span></div>
<div id="yui_3_16_0_ym19_1_1483645394100_4493"><span style="font-size:10.0pt;font-family:Calibri;mso-ascii-theme-font:major-latin;mso-hansi-theme-font:major-latin" id="yui_3_16_0_ym19_1_1483645394100_4494"> </span></div>
<div id="yui_3_16_0_ym19_1_1483645394100_4495" dir="ltr"><span style="font-size:10.0pt;font-family:Calibri;mso-ascii-theme-font:major-latin;mso-hansi-theme-font:major-latin" id="yui_3_16_0_ym19_1_1483645394100_4496">3 servers
(ipa1-dr ipa2-dr ipa2-prod) have these errors: </span></div>
<div style="mso-pagination:none;mso-layout-grid-align:none;text-autospace:none" id="yui_3_16_0_ym19_1_1483645394100_4497"><span style="font-size:10.0pt;font-family:Calibri;mso-ascii-theme-font:major-latin;mso-hansi-theme-font:major-latin;mso-bidi-font-family:Calibri" id="yui_3_16_0_ym19_1_1483645394100_4498">[01/Jan/2017:14:43:06 -0800] - libdb: BDB2055 Lock table is out of
available lock entries</span></div>
<div id="yui_3_16_0_ym19_1_1483645394100_4499"><span style="font-size:10.0pt;font-family:Calibri;mso-ascii-theme-font:major-latin;mso-hansi-theme-font:major-latin;mso-bidi-font-family:Calibri" id="yui_3_16_0_ym19_1_1483645394100_4500">[01/Jan/2017:14:43:06 -0800] - compactdb: failed to compact changelog;
db error - 12 Cannot allocate memory</span></div>
<div id="yui_3_16_0_ym19_1_1483645394100_4501"><span style="font-size:10.0pt;font-family:Calibri;mso-ascii-theme-font:major-latin;mso-hansi-theme-font:major-latin;mso-bidi-font-family:Calibri" id="yui_3_16_0_ym19_1_1483645394100_4502"> </span></div>
<div id="yui_3_16_0_ym19_1_1483645394100_4503" dir="ltr"><span style="font-size:10.0pt;font-family:Calibri;mso-ascii-theme-font:major-latin;mso-hansi-theme-font:major-latin;mso-bidi-font-family:Calibri" id="yui_3_16_0_ym19_1_1483645394100_4504">4 servers (ipa1-dev, ipa2-dev, ipa1-dr and ipa2-dr) have these errors<br></span></div>
<div id="yui_3_16_0_ym19_1_1483645394100_4505"><span style="font-size:10.0pt;font-family:Calibri;mso-ascii-theme-font:major-latin;mso-hansi-theme-font:major-latin;mso-bidi-font-family:Calibri" id="yui_3_16_0_ym19_1_1483645394100_4506">[04/Jan/2017:15:37:21 -0800] slapd_ldap_sasl_interactive_bind - Error:
could not perform interactive bind for id [] mech [GSSAPI]: LDAP error -1
(Can't contact LDAP server) ((null)) errno 107 (<b id="yui_3_16_0_ym19_1_1483645394100_4507">Transport</b> endpoint is
not connected)</span></div>
<div id="yui_3_16_0_ym19_1_1483645394100_4508"><span style="font-size:10.0pt;font-family:Calibri;mso-ascii-theme-font:major-latin;mso-hansi-theme-font:major-latin;mso-bidi-font-family:Calibri" id="yui_3_16_0_ym19_1_1483645394100_4509">[04/Jan/2017:15:37:24 -0800] slapd_ldap_sasl_interactive_bind - Error:
could not perform interactive bind for id [] mech [GSSAPI]: LDAP error -1
(Can't contact LDAP server) ((null)) errno 107 (<b id="yui_3_16_0_ym19_1_1483645394100_4510">Transport</b> endpoint is
not connected)</span></div>
<div id="yui_3_16_0_ym19_1_1483645394100_4511"><span style="font-size:10.0pt;font-family:Calibri;mso-ascii-theme-font:major-latin;mso-hansi-theme-font:major-latin;mso-bidi-font-family:Calibri" id="yui_3_16_0_ym19_1_1483645394100_4512"> </span><br></div><span style="font-size:10.0pt;font-family:Calibri;mso-ascii-theme-font:major-latin;mso-hansi-theme-font:major-latin;mso-bidi-font-family:Calibri" id="yui_3_16_0_ym19_1_1483645394100_4571">I have tried various combinations or restarting, re-initializing, disconnecting and reconnecting replicas but am down to
only two servers replicating with each other currently (ipa1-dev and ipa2-dev). We did have a power outage
at the dev location but it does not seem to correspond to when the errors started? Not sure how
to recover from this. Any help is appreciated</span>
<div id="yui_3_16_0_ym19_1_1483645394100_4572"><span style="font-size:10.0pt;font-family:Calibri;mso-ascii-theme-font:major-latin;mso-hansi-theme-font:major-latin" id="yui_3_16_0_ym19_1_1483645394100_4573"><span style="mso-spacerun:yes" id="yui_3_16_0_ym19_1_1483645394100_4574"> </span></span></div>
<div id="yui_3_16_0_ym19_1_1483645394100_4575"><span style="font-size:10.0pt;font-family:Calibri;mso-ascii-theme-font:major-latin;mso-hansi-theme-font:major-latin" id="yui_3_16_0_ym19_1_1483645394100_4576"> </span></div>
</div></body></html>