<html><head></head><body><div style="color:#000; background-color:#fff; font-family:bookman old style, new york, times, serif;font-size:13px"><div id="yui_3_16_0_ym19_1_1483732279514_6483">I have changed the number of db locks to 40000. After restart, each server reports a lot of these type errors:<br></div><div id="yui_3_16_0_ym19_1_1483732279514_6484">DSRetroclPlugin - delete_changerecord: could not delete change record 6038434 <br></div><div id="yui_3_16_0_ym19_1_1483732279514_6486" dir="ltr"><br></div><div dir="ltr" id="yui_3_16_0_ym19_1_1483732279514_7569">As well as immediately coming up with these errors (even after re-initializing)<br></div><div dir="ltr" id="yui_3_16_0_ym19_1_1483732279514_6882"><br></div><div id="yui_3_16_0_ym19_1_1483732279514_6339" dir="ltr"><span id="yui_3_16_0_ym19_1_1483732279514_6861">06/Jan/2017:12:10:12 -0800] NSMMReplicationPlugin - changelog program - agmt="cn=meToipa1-dev.example.local" (ipa1-corp:389): CSN 586d8aab000400110000 not found, we aren't as up to date, or we purged<br id="yui_3_16_0_ym19_1_1483732279514_6794">[06/Jan/2017:12:10:12 -0800] NSMMReplicationPlugin - agmt="cn=meToipa1-corp.</span><span id="yui_3_16_0_ym19_1_1483732279514_6865"><span id="yui_3_16_0_ym19_1_1483732279514_6862">example</span>.local" (ipa1-dev:389): Data required to update replica has been purged. The replica must be reinitialized.<br id="yui_3_16_0_ym19_1_1483732279514_6795">[06/Jan/2017:12:10:12 -0800] NSMMReplicationPlugin - agmt="cn=meToipa1-prod.</span><span id="yui_3_16_0_ym19_1_1483732279514_6869"><span id="yui_3_16_0_ym19_1_1483732279514_6866">example</span>.local" (ipa1-xo:389): Incremental update failed and requires administrator action<br id="yui_3_16_0_ym19_1_1483732279514_6796">[06/Jan/2017:12:10:12 -0800] NSMMReplicationPlugin - agmt="cn=meToipa1-dev.</span><span id="yui_3_16_0_ym19_1_1483732279514_6797"><span id="yui_3_16_0_ym19_1_1483732279514_6870">example</span>.local" (ipa1-corp:389): Incremental update failed and requires administrator action<br></span></div><div dir="ltr" id="yui_3_16_0_ym19_1_1483732279514_6944"><span id="yui_3_16_0_ym19_1_1483732279514_6797">06/Jan/2017:12:15:47 -0800] agmt="cn=meToipa1-dr.example.local" (ipa1-io:389) - Can't locate CSN 586ffaf5000300100000 in the changelog (DB rc=-30988). If replication stops, the consumer may need to be reinitialized.<br id="yui_3_16_0_ym19_1_1483732279514_6947">[06/Jan/2017:12:15:49 -0800] agmt="cn=meToipa1-dr.example.local" (ipa1-io:389) - Can't locate CSN 586ffaf7000000100000 in the changelog (DB rc=-30988). If replication stops, the consumer may need to be reinitialized.<br></span></div><div dir="ltr" id="yui_3_16_0_ym19_1_1483732279514_7007"><span id="yui_3_16_0_ym19_1_1483732279514_6797"><br></span></div><div dir="ltr" id="yui_3_16_0_ym19_1_1483732279514_7286"><span id="yui_3_16_0_ym19_1_1483732279514_6797">Replication topology is:</span></div><div dir="ltr" id="yui_3_16_0_ym19_1_1483732279514_7293"><span id="yui_3_16_0_ym19_1_1483732279514_6797">3 geographic locations each with 2 ipa servers (dr, prod, dev)<br></span></div><div dir="ltr" id="yui_3_16_0_ym19_1_1483732279514_7267"><span id="yui_3_16_0_ym19_1_1483732279514_6797">ipa1-dev replicates with all servers (ipa2-dev,ipa1-dr, ipa2-dr, ipa1-prod, ipa2-prod)</span></div><div dir="ltr" id="yui_3_16_0_ym19_1_1483732279514_7314"><span id="yui_3_16_0_ym19_1_1483732279514_6797">ipa1-dr also replicates with ipa2-dr</span></div><div dir="ltr" id="yui_3_16_0_ym19_1_1483732279514_7387"><span id="yui_3_16_0_ym19_1_1483732279514_6797">ipa1-prod also replicates with ipa2-prod<br></span></div><div dir="ltr" id="yui_3_16_0_ym19_1_1483732279514_7572"><span id="yui_3_16_0_ym19_1_1483732279514_6797"><br></span></div><div dir="ltr" id="yui_3_16_0_ym19_1_1483732279514_7667"><span id="yui_3_16_0_ym19_1_1483732279514_6797">As a test I deleted one host on each of the servers. I have waited 30 minutes and the results are:</span></div><div dir="ltr" id="yui_3_16_0_ym19_1_1483732279514_7668"><span id="yui_3_16_0_ym19_1_1483732279514_6797">ipa1-dev - deletion replicated to all servers</span></div><div dir="ltr" id="yui_3_16_0_ym19_1_1483732279514_7671"><span id="yui_3_16_0_ym19_1_1483732279514_6797">ipa2-dr - deletion replicated to all servers<br></span></div><div dir="ltr" id="yui_3_16_0_ym19_1_1483732279514_7279"><span id="yui_3_16_0_ym19_1_1483732279514_6797">ipa1-dr, ipa1-prod, ipa2-dev, ipa2-prod - deletions not replicated<br></span></div><div dir="ltr" id="yui_3_16_0_ym19_1_1483732279514_7000"><span id="yui_3_16_0_ym19_1_1483732279514_6797"><br></span></div><div class="qtdSeparateBR"><br><br></div><div class="yahoo_quoted" id="yui_3_16_0_ym19_1_1483732279514_6345" style="display: block;"> <div style="font-family: bookman old style, new york, times, serif; font-size: 13px;" id="yui_3_16_0_ym19_1_1483732279514_6344"> <div style="font-family: HelveticaNeue, Helvetica Neue, Helvetica, Arial, Lucida Grande, sans-serif; font-size: 16px;" id="yui_3_16_0_ym19_1_1483732279514_6343"> <div dir="ltr" id="yui_3_16_0_ym19_1_1483732279514_6342"> <font id="yui_3_16_0_ym19_1_1483732279514_6346" face="Arial" size="2"> <hr id="yui_3_16_0_ym19_1_1483732279514_6964" size="1"> <b><span style="font-weight:bold;">From:</span></b> Martin Basti <mbasti@redhat.com><br> <b><span style="font-weight: bold;">To:</span></b> sipazzo <sipazzo@yahoo.com>; Freeipa-users <freeipa-users@redhat.com> <br> <b><span style="font-weight: bold;">Sent:</span></b> Friday, January 6, 2017 8:58 AM<br> <b><span style="font-weight: bold;">Subject:</span></b> Re: [Freeipa-users] Replication has stopped and server errors<br> </font> </div> <div class="y_msg_container" id="yui_3_16_0_ym19_1_1483732279514_6354"><br><div id="yiv2997724678"><div id="yui_3_16_0_ym19_1_1483732279514_6353">
<div id="yui_3_16_0_ym19_1_1483732279514_6451"><br clear="none">
</div>
<br clear="none">
<div class="yiv2997724678moz-cite-prefix" id="yui_3_16_0_ym19_1_1483732279514_6452">On 06.01.2017 00:29, sipazzo wrote:<br clear="none">
</div>
<blockquote type="cite" id="yui_3_16_0_ym19_1_1483732279514_6362">
<div style="color:#000;background-color:#fff;font-family:bookman old style, new york, times, serif;font-size:13px;" id="yui_3_16_0_ym19_1_1483732279514_6361">
<div id="yiv2997724678yui_3_16_0_ym19_1_1483645394100_4471">I have<span id="yiv2997724678yui_3_16_0_ym19_1_1483645394100_4472" style="font-size:10.0pt;font-family:Calibri;"> 6 ipa
servers in 3 locations running 4.2.0-15.0.1on RHEL 7.
Ipa1-dev is the CA
Renewal and CRL Master server and where most of our updates
(host enrollment,
password changes) end up taking place.</span></div>
<div id="yiv2997724678yui_3_16_0_ym19_1_1483645394100_4473"><span id="yiv2997724678yui_3_16_0_ym19_1_1483645394100_4474" style="font-size:10.0pt;font-family:Calibri;"> </span></div>
<div id="yiv2997724678yui_3_16_0_ym19_1_1483645394100_4475"><span id="yiv2997724678yui_3_16_0_ym19_1_1483645394100_4476" style="font-size:10.0pt;font-family:Calibri;">Servers had
been running fine. Over the holidays we started having some
replication issues
and looking at /var/log/dirsrv/slapd-REALM-COM/errors showed
the following:</span></div>
<div id="yiv2997724678yui_3_16_0_ym19_1_1483645394100_6911"><br clear="none">
</div>
<div id="yiv2997724678yui_3_16_0_ym19_1_1483645394100_4670"><span id="yiv2997724678yui_3_16_0_ym19_1_1483645394100_4671" style="font-size:10.0pt;font-family:Calibri;">All servers
currently have these errors for each replica the respective
IPA servers are
connected to:</span></div>
<div id="yiv2997724678yui_3_16_0_ym19_1_1483645394100_4672"><span id="yiv2997724678yui_3_16_0_ym19_1_1483645394100_4673" style="font-size:10.0pt;font-family:Calibri;">NSMMReplicationPlugin
- agmt="cn=meToipa2-dr.example.local" (ipa2-dr:389):
Incremental
update failed and requires administrator action</span></div>
<div id="yiv2997724678yui_3_16_0_ym19_1_1483645394100_4674"><span id="yiv2997724678yui_3_16_0_ym19_1_1483645394100_4675" style="font-size:10.0pt;font-family:Calibri;">[04/Jan/2017:15:39:48
-0800] agmt="cn=meToipa1-dr.example.local" (ipa1-dr:389) -
Can't
locate CSN 583c8e74000600110000 in the changelog (DB
rc=-30988). If replication
stops, the consumer may need to be reinitialized</span></div>
<div id="yiv2997724678yui_3_16_0_ym19_1_1483645394100_4676"><span id="yiv2997724678yui_3_16_0_ym19_1_1483645394100_4677" style="font-size:10.0pt;font-family:Calibri;">NSMMReplicationPlugin
- agmt="cn=meToipa1-prod.example.local" (ipa1-prod:389):
Data
required to update replica has been purged. The replica must
be reinitialized.</span></div>
<div id="yiv2997724678yui_3_16_0_ym19_1_1483645394100_4678"><span id="yiv2997724678yui_3_16_0_ym19_1_1483645394100_4679" style="font-size:10.0pt;font-family:Calibri;">[04/Jan/2017:13:33:26
-0800] NSMMReplicationPlugin -
agmt="cn=meToipa2-dev.example.local"
(ipa2-dev:389): Incremental update failed and requires
administrator action</span></div>
<div id="yiv2997724678yui_3_16_0_ym19_1_1483645394100_4680"><span id="yiv2997724678yui_3_16_0_ym19_1_1483645394100_4681" style="font-size:10.0pt;font-family:Calibri;"><span id="yiv2997724678yui_3_16_0_ym19_1_1483645394100_4682" style=""> </span>[04/Jan/2017:13:33:26
-0800]
NSMMReplicationPlugin -
agmt="cn=meToipa1-prod.example.local"
(ipa1-prod:389): Incremental update failed and requires
administrator action</span></div>
<div id="yiv2997724678yui_3_16_0_ym19_1_1483645394100_4683"><span id="yiv2997724678yui_3_16_0_ym19_1_1483645394100_4684" style="font-size:10.0pt;font-family:Calibri;">[04/Jan/2017:13:33:27
-0800] agmt="cn=meToipa2-prod.example.local" (ipa2-prod:389)
- Can't
locate CSN 586d69f0000400120000 in the changelog (DB
rc=-30988). If replication
stops, the consumer may need to be reinitialized.</span></div>
<div id="yiv2997724678yui_3_16_0_ym19_1_1483645394100_4685"><span id="yiv2997724678yui_3_16_0_ym19_1_1483645394100_4686" style="font-size:10.0pt;font-family:Calibri;"> </span></div>
<div id="yiv2997724678yui_3_16_0_ym19_1_1483645394100_4687"><span id="yiv2997724678yui_3_16_0_ym19_1_1483645394100_4688" style="font-size:10.0pt;font-family:Calibri;">And all
servers have these types of errors which are worrisome but
they go back quite a way<br clear="none">
</span></div>
<div id="yiv2997724678yui_3_16_0_ym19_1_1483645394100_4689"><b id="yiv2997724678yui_3_16_0_ym19_1_1483645394100_4690"><span id="yiv2997724678yui_3_16_0_ym19_1_1483645394100_4691" style="font-size:10.0pt;font-family:Calibri;">NSACL</span></b><span id="yiv2997724678yui_3_16_0_ym19_1_1483645394100_4692" style="font-size:10.0pt;font-family:Calibri;">Plugin - The ACL
target cn=dns,dc=example,dc=local
does not exist</span></div>
<div id="yiv2997724678yui_3_16_0_ym19_1_1483645394100_4693"><b id="yiv2997724678yui_3_16_0_ym19_1_1483645394100_4694"><span id="yiv2997724678yui_3_16_0_ym19_1_1483645394100_4695" style="font-size:10.0pt;font-family:Calibri;">NSACL</span></b><span id="yiv2997724678yui_3_16_0_ym19_1_1483645394100_4696" style="font-size:10.0pt;font-family:Calibri;">Plugin - The ACL
target cn=dns,dc=example,dc=local
does not exist</span></div>
<div id="yiv2997724678yui_3_16_0_ym19_1_1483645394100_4697"><b id="yiv2997724678yui_3_16_0_ym19_1_1483645394100_4698"><span id="yiv2997724678yui_3_16_0_ym19_1_1483645394100_4699" style="font-size:10.0pt;font-family:Calibri;">NSACL</span></b><span id="yiv2997724678yui_3_16_0_ym19_1_1483645394100_4700" style="font-size:10.0pt;font-family:Calibri;">Plugin - The ACL
target
cn=groups,cn=compat,dc=example,dc=local does not exist</span></div>
<div id="yiv2997724678yui_3_16_0_ym19_1_1483645394100_4701"><b id="yiv2997724678yui_3_16_0_ym19_1_1483645394100_4702"><span id="yiv2997724678yui_3_16_0_ym19_1_1483645394100_4703" style="font-size:10.0pt;font-family:Calibri;">NSACL</span></b><span id="yiv2997724678yui_3_16_0_ym19_1_1483645394100_4704" style="font-size:10.0pt;font-family:Calibri;">Plugin - The ACL
target
cn=computers,cn=compat,dc=example,dc=local does not exist</span></div>
<div id="yiv2997724678yui_3_16_0_ym19_1_1483645394100_4705"><b id="yiv2997724678yui_3_16_0_ym19_1_1483645394100_4706"><span id="yiv2997724678yui_3_16_0_ym19_1_1483645394100_4707" style="font-size:10.0pt;font-family:Calibri;">NSACL</span></b><span id="yiv2997724678yui_3_16_0_ym19_1_1483645394100_4708" style="font-size:10.0pt;font-family:Calibri;">Plugin - The ACL
target cn=casigningcert
cert-pki-ca,cn=ca_renewal,cn=ipa,cn=etc,dc=example,dc=local
does not exist</span></div>
<div id="yiv2997724678yui_3_16_0_ym19_1_1483645394100_4709"><b id="yiv2997724678yui_3_16_0_ym19_1_1483645394100_4710"><span id="yiv2997724678yui_3_16_0_ym19_1_1483645394100_4711" style="font-size:10.0pt;font-family:Calibri;">NSACL</span></b><span id="yiv2997724678yui_3_16_0_ym19_1_1483645394100_4712" style="font-size:10.0pt;font-family:Calibri;">Plugin - The ACL
target cn=casigningcert
cert-pki-ca,cn=ca_renewal,cn=ipa,cn=etc,dc=example,dc=local
does not exist</span></div>
<div id="yiv2997724678yui_3_16_0_ym19_1_1483645394100_4713"><b id="yiv2997724678yui_3_16_0_ym19_1_1483645394100_4714"><span id="yiv2997724678yui_3_16_0_ym19_1_1483645394100_4715" style="font-size:10.0pt;font-family:Calibri;">NSACL</span></b><span id="yiv2997724678yui_3_16_0_ym19_1_1483645394100_4716" style="font-size:10.0pt;font-family:Calibri;">Plugin - The ACL
target
ou=sudoers,dc=networkfleet,dc=local does not exist</span></div>
</div>
</blockquote>
^^^ just INFO messages, you can ignore them<br clear="none">
<br clear="none">
<br clear="none">
<blockquote type="cite" id="yui_3_16_0_ym19_1_1483732279514_6368">
<div style="color:#000;background-color:#fff;font-family:bookman old style, new york, times, serif;font-size:13px;" id="yui_3_16_0_ym19_1_1483732279514_6367">
<div id="yiv2997724678yui_3_16_0_ym19_1_1483645394100_4717"><span id="yiv2997724678yui_3_16_0_ym19_1_1483645394100_4718" style="font-size:10.0pt;font-family:Calibri;"> </span></div>
<div id="yiv2997724678yui_3_16_0_ym19_1_1483645394100_4719"><span id="yiv2997724678yui_3_16_0_ym19_1_1483645394100_4720" style="font-size:10.0pt;font-family:Calibri;">All servers except
one have a lot of these</span></div>
<div dir="ltr" id="yiv2997724678yui_3_16_0_ym19_1_1483645394100_4721"><span id="yiv2997724678yui_3_16_0_ym19_1_1483645394100_4722" style="font-size:10.0pt;font-family:Calibri;">DSRetroclPlugin -
delete_changerecord: could not delete change record</span></div>
<div id="yiv2997724678yui_3_16_0_ym19_1_1483645394100_4477"><span id="yiv2997724678yui_3_16_0_ym19_1_1483645394100_4478" style="font-size:10.0pt;font-family:Calibri;"> </span></div>
<div id="yiv2997724678yui_3_16_0_ym19_1_1483645394100_4479"><span id="yiv2997724678yui_3_16_0_ym19_1_1483645394100_4480" style="font-size:10.0pt;font-family:Calibri;">Ipa1-dev only has
this<br clear="none">
</span></div>
<div id="yiv2997724678yui_3_16_0_ym19_1_1483645394100_4481"><span id="yiv2997724678yui_3_16_0_ym19_1_1483645394100_4482" style="font-size:10.0pt;font-family:Calibri;">04/Jan/2017:18:36:52
-0800] NSMMReplicationPlugin -
agmt="cn=masterAgreement1-ipa1-prod.example.local-pki-tomcat"
(ipa1-prod:389): Replication bind with <b id="yiv2997724678yui_3_16_0_ym19_1_1483645394100_4483">SIMPLE</b> auth
resumed</span></div>
<div id="yiv2997724678yui_3_16_0_ym19_1_1483645394100_4484"><span id="yiv2997724678yui_3_16_0_ym19_1_1483645394100_4485" style="font-size:10.0pt;font-family:Calibri;">[04/Jan/2017:18:36:52
-0800] NSMMReplicationPlugin -
agmt="cn=masterAgreement1-ipa2-dr.example.local-pki-tomcat"
(ipa2-dr:389): Replication bind with <b id="yiv2997724678yui_3_16_0_ym19_1_1483645394100_4486">SIMPLE</b> auth
resumed</span></div>
<div id="yiv2997724678yui_3_16_0_ym19_1_1483645394100_4487"><span id="yiv2997724678yui_3_16_0_ym19_1_1483645394100_4488" style="font-size:10.0pt;font-family:Calibri;">[04/Jan/2017:18:36:52
-0800] NSMMReplicationPlugin -
agmt="cn=masterAgreement1-ipa1-dr.example.local-pki-tomcat"
(ipa1-dr:389): Replication bind with <b id="yiv2997724678yui_3_16_0_ym19_1_1483645394100_4489">SIMPLE</b> auth
resumed</span></div>
<div id="yiv2997724678yui_3_16_0_ym19_1_1483645394100_4490"><span id="yiv2997724678yui_3_16_0_ym19_1_1483645394100_4491" style="font-size:10.0pt;font-family:Calibri;">[04/Jan/2017:18:36:53
-0800] NSMMReplicationPlugin -
agmt="cn=masterAgreement1-ipa2-prod.example.local-pki-tomcat"
(ipa2-prod:389): Replication bind with <b id="yiv2997724678yui_3_16_0_ym19_1_1483645394100_4492">SIMPLE</b> auth
resumed</span></div>
<div id="yiv2997724678yui_3_16_0_ym19_1_1483645394100_4493"><span id="yiv2997724678yui_3_16_0_ym19_1_1483645394100_4494" style="font-size:10.0pt;font-family:Calibri;"> </span></div>
<div dir="ltr" id="yiv2997724678yui_3_16_0_ym19_1_1483645394100_4495"><span id="yiv2997724678yui_3_16_0_ym19_1_1483645394100_4496" style="font-size:10.0pt;font-family:Calibri;">3 servers
(ipa1-dr ipa2-dr ipa2-prod) have these errors: </span></div>
<div id="yiv2997724678yui_3_16_0_ym19_1_1483645394100_4497" style=""><span id="yiv2997724678yui_3_16_0_ym19_1_1483645394100_4498" style="font-size:10.0pt;font-family:Calibri;">[01/Jan/2017:14:43:06
-0800] - libdb: BDB2055 Lock table is out of
available lock entries</span></div>
<div id="yiv2997724678yui_3_16_0_ym19_1_1483645394100_4499"><span id="yiv2997724678yui_3_16_0_ym19_1_1483645394100_4500" style="font-size:10.0pt;font-family:Calibri;">[01/Jan/2017:14:43:06
-0800] - compactdb: failed to compact changelog;
db error - 12 Cannot allocate memory</span></div>
</div>
</blockquote>
<br clear="none">
you probably need <a rel="nofollow" shape="rect" class="yiv2997724678moz-txt-link-freetext" target="_blank" href="https://access.redhat.com/solutions/1241063">https://access.redhat.com/solutions/1241063</a> to
increase number of locks (or in this thread
<a rel="nofollow" shape="rect" class="yiv2997724678moz-txt-link-freetext" target="_blank" href="https://lists.fedoraproject.org/pipermail/389-users/2011-June/013299.html">https://lists.fedoraproject.org/pipermail/389-users/2011-June/013299.html</a>)<br clear="none">
<br clear="none">
I would first increase the number of locks, and then look if
something improved.<br clear="none">
We also don't know how your topology looks like, which servers are
connected together.<br clear="none">
<br clear="none">
Martin<div class="yiv2997724678yqt8663937149" id="yiv2997724678yqtfd12957"><br clear="none">
<br clear="none">
<blockquote type="cite">
<div style="color:#000;background-color:#fff;font-family:bookman old style, new york, times, serif;font-size:13px;">
<div id="yiv2997724678yui_3_16_0_ym19_1_1483645394100_4501"><span id="yiv2997724678yui_3_16_0_ym19_1_1483645394100_4502" style="font-size:10.0pt;font-family:Calibri;"> </span></div>
<div dir="ltr" id="yiv2997724678yui_3_16_0_ym19_1_1483645394100_4503"><span id="yiv2997724678yui_3_16_0_ym19_1_1483645394100_4504" style="font-size:10.0pt;font-family:Calibri;">4 servers
(ipa1-dev, ipa2-dev, ipa1-dr and ipa2-dr) have these errors<br clear="none">
</span></div>
<div id="yiv2997724678yui_3_16_0_ym19_1_1483645394100_4505"><span id="yiv2997724678yui_3_16_0_ym19_1_1483645394100_4506" style="font-size:10.0pt;font-family:Calibri;">[04/Jan/2017:15:37:21
-0800] slapd_ldap_sasl_interactive_bind - Error:
could not perform interactive bind for id [] mech [GSSAPI]:
LDAP error -1
(Can't contact LDAP server) ((null)) errno 107 (<b id="yiv2997724678yui_3_16_0_ym19_1_1483645394100_4507">Transport</b>
endpoint is
not connected)</span></div>
<div id="yiv2997724678yui_3_16_0_ym19_1_1483645394100_4508"><span id="yiv2997724678yui_3_16_0_ym19_1_1483645394100_4509" style="font-size:10.0pt;font-family:Calibri;">[04/Jan/2017:15:37:24
-0800] slapd_ldap_sasl_interactive_bind - Error:
could not perform interactive bind for id [] mech [GSSAPI]:
LDAP error -1
(Can't contact LDAP server) ((null)) errno 107 (<b id="yiv2997724678yui_3_16_0_ym19_1_1483645394100_4510">Transport</b>
endpoint is
not connected)</span></div>
<div id="yiv2997724678yui_3_16_0_ym19_1_1483645394100_4511"><span id="yiv2997724678yui_3_16_0_ym19_1_1483645394100_4512" style="font-size:10.0pt;font-family:Calibri;"> </span><br clear="none">
</div>
<span id="yiv2997724678yui_3_16_0_ym19_1_1483645394100_4571" style="font-size:10.0pt;font-family:Calibri;">I have tried various
combinations or restarting, re-initializing, disconnecting and
reconnecting replicas but am down to
only two servers replicating with each other currently
(ipa1-dev and ipa2-dev). We did have a power outage
at the dev location but it does not seem to correspond to when
the errors started? Not sure how
to recover from this. Any help is appreciated</span>
<div id="yiv2997724678yui_3_16_0_ym19_1_1483645394100_4572"><span id="yiv2997724678yui_3_16_0_ym19_1_1483645394100_4573" style="font-size:10.0pt;font-family:Calibri;"><span id="yiv2997724678yui_3_16_0_ym19_1_1483645394100_4574" style=""> </span></span></div>
<div id="yiv2997724678yui_3_16_0_ym19_1_1483645394100_4575"><span id="yiv2997724678yui_3_16_0_ym19_1_1483645394100_4576" style="font-size:10.0pt;font-family:Calibri;"> </span></div>
</div>
<br clear="none">
<fieldset class="yiv2997724678mimeAttachmentHeader"></fieldset>
<br clear="none">
</blockquote>
<br clear="none">
</div></div></div><br><br></div> </div> </div> </div></div></body></html>