<html>
<head>
<meta content="text/html; charset=windows-1252"
http-equiv="Content-Type">
</head>
<body text="#000000" bgcolor="#FFFFFF">
Hi Timothy,<br>
<br>
The changenumber counter is protected by a lock and we should not
see duplicate value.. except if there is a bug :-(<br>
<br>
Retrieving the time when changenumber=112697,cn=changelog was
created and the time when you saw the error, can you see any error
in operations (access log) or in the error log ?<br>
<br>
Or did you disabled/enable retorCL between those two times ?<br>
<br>
regards<br>
thiery<br>
<br>
<br>
<br>
<div class="moz-cite-prefix">On 09/27/2016 12:37 AM, Timothy Geier
wrote:<br>
</div>
<blockquote
cite="mid:55D4C985-EEB3-46AA-A92F-61EFA1EA7B3F@accertify.com"
type="cite">
<meta http-equiv="Content-Type" content="text/html;
charset=windows-1252">
<br class="">
<div>
<blockquote type="cite" class="">
<div class="">On Sep 26, 2016, at 4:07 PM, Timothy Geier <<a
moz-do-not-send="true" href="mailto:tgeier@accertify.com"
class=""><a class="moz-txt-link-abbreviated" href="mailto:tgeier@accertify.com">tgeier@accertify.com</a></a>> wrote:</div>
<br class="Apple-interchange-newline">
<div class="">
<blockquote type="cite" style="font-family: Helvetica;
font-size: 12px; font-style: normal; font-variant: normal;
font-weight: normal; letter-spacing: normal; line-height:
normal; orphans: auto; text-align: start; text-indent:
0px; text-transform: none; white-space: normal; widows:
auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;"
class="">
<br class="Apple-interchange-newline">
On Sep 26, 2016, at 2:17 PM, Timothy Geier <<a
moz-do-not-send="true"
href="mailto:tgeier@accertify.com" class=""><a class="moz-txt-link-abbreviated" href="mailto:tgeier@accertify.com">tgeier@accertify.com</a></a>>
wrote:<br class="">
<br class="">
This issue started when trying to remove a user; ipa
user-del showed “operation failed” and the user was not
removed. The same ipa user-del command was performed on a
replica and completed successfully, but it was then
immediately apparent that this change did not replicate
anywhere else. All of the replicas then were
re-initalized using "ipa-replica-manage re-initialize” and
now the LDAP trees/users are consistent though no further
changes have been made.<br class="">
<br class="">
The slapd error logs are showing repeated instances of<br
class="">
<br class="">
DSRetroclPlugin - replog: an error occured while adding
change number 112697, dn =
changenumber=112697,cn=changelog: Already exists.<br
class="">
retrocl-plugin - retrocl_postob: operation failure [68]<br
class="">
<br class="">
Package versions are<br class="">
ipa-server-4.2.0-15.0.1.el7.centos.6.1.x86_64<br class="">
and<span class="Apple-converted-space"> </span><br
class="">
389-ds-base-1.3.4.0-29.el7_2.x86_64<br class="">
<br class="">
ipa-replica-manage list-ruv<br class="">
ipa: WARNING: session memcached servers not running<br
class="">
unable to decode: {replica 11} 56044ef50000000b0000
56044ef50000000b0000<br class="">
unable to decode: {replica 7} 561f17ba000800070000
561f17ba000800070000<br class="">
unable to decode: {replica 5} 561f17bc000300050000
561f17bc000300050000<br class="">
unable to decode: {replica 9} 561f17ba000a00090000
561f17ba000a00090000<br class="">
unable to decode: {replica 4} 561f17ba000300040000
561f17ba000300040000<span class="Apple-converted-space"> </span><br
class="">
(These are likely leftovers from the previous incarnation
of these servers on a RHEL6-like setup)<br class="">
ipa07:389: 16<br class="">
ipa02:389: 13<br class="">
ipa03:389: 14<br class="">
ipa01:389: 12<br class="">
ipa04:389: 15<br class="">
ipa05:389: 17<br class="">
<br class="">
Thanks much,<br class="">
</blockquote>
<br style="font-family: Helvetica; font-size: 12px;
font-style: normal; font-variant: normal; font-weight:
normal; letter-spacing: normal; line-height: normal;
orphans: auto; text-align: start; text-indent: 0px;
text-transform: none; white-space: normal; widows: auto;
word-spacing: 0px; -webkit-text-stroke-width: 0px;"
class="">
<span style="font-family: Helvetica; font-size: 12px;
font-style: normal; font-variant: normal; font-weight:
normal; letter-spacing: normal; line-height: normal;
orphans: auto; text-align: start; text-indent: 0px;
text-transform: none; white-space: normal; widows: auto;
word-spacing: 0px; -webkit-text-stroke-width: 0px; float:
none; display: inline !important;" class="">After not
taking any action, this error has stopped but has been
replaced with</span><br style="font-family: Helvetica;
font-size: 12px; font-style: normal; font-variant: normal;
font-weight: normal; letter-spacing: normal; line-height:
normal; orphans: auto; text-align: start; text-indent:
0px; text-transform: none; white-space: normal; widows:
auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;"
class="">
<br style="font-family: Helvetica; font-size: 12px;
font-style: normal; font-variant: normal; font-weight:
normal; letter-spacing: normal; line-height: normal;
orphans: auto; text-align: start; text-indent: 0px;
text-transform: none; white-space: normal; widows: auto;
word-spacing: 0px; -webkit-text-stroke-width: 0px;"
class="">
<span style="font-family: Helvetica; font-size: 12px;
font-style: normal; font-variant: normal; font-weight:
normal; letter-spacing: normal; line-height: normal;
orphans: auto; text-align: start; text-indent: 0px;
text-transform: none; white-space: normal; widows: auto;
word-spacing: 0px; -webkit-text-stroke-width: 0px; float:
none; display: inline !important;" class="">[26/Sep/2016:15:54:54
-0500] NSMMReplicationPlugin - agmt="cn=meToipa03"
(ipa03:389): Missing data encountered</span><br
style="font-family: Helvetica; font-size: 12px;
font-style: normal; font-variant: normal; font-weight:
normal; letter-spacing: normal; line-height: normal;
orphans: auto; text-align: start; text-indent: 0px;
text-transform: none; white-space: normal; widows: auto;
word-spacing: 0px; -webkit-text-stroke-width: 0px;"
class="">
<span style="font-family: Helvetica; font-size: 12px;
font-style: normal; font-variant: normal; font-weight:
normal; letter-spacing: normal; line-height: normal;
orphans: auto; text-align: start; text-indent: 0px;
text-transform: none; white-space: normal; widows: auto;
word-spacing: 0px; -webkit-text-stroke-width: 0px; float:
none; display: inline !important;" class="">[26/Sep/2016:15:54:54
-0500] NSMMReplicationPlugin - agmt="cn=meToipa03"
(ipa03:389): Incremental update failed and requires
administrator action</span><br style="font-family:
Helvetica; font-size: 12px; font-style: normal;
font-variant: normal; font-weight: normal; letter-spacing:
normal; line-height: normal; orphans: auto; text-align:
start; text-indent: 0px; text-transform: none;
white-space: normal; widows: auto; word-spacing: 0px;
-webkit-text-stroke-width: 0px;" class="">
<br style="font-family: Helvetica; font-size: 12px;
font-style: normal; font-variant: normal; font-weight:
normal; letter-spacing: normal; line-height: normal;
orphans: auto; text-align: start; text-indent: 0px;
text-transform: none; white-space: normal; widows: auto;
word-spacing: 0px; -webkit-text-stroke-width: 0px;"
class="">
<span style="font-family: Helvetica; font-size: 12px;
font-style: normal; font-variant: normal; font-weight:
normal; letter-spacing: normal; line-height: normal;
orphans: auto; text-align: start; text-indent: 0px;
text-transform: none; white-space: normal; widows: auto;
word-spacing: 0px; -webkit-text-stroke-width: 0px; float:
none; display: inline !important;" class="">for all of the
replicas and things are slightly out of sync everywhere. </span><br
style="font-family: Helvetica; font-size: 12px;
font-style: normal; font-variant: normal; font-weight:
normal; letter-spacing: normal; line-height: normal;
orphans: auto; text-align: start; text-indent: 0px;
text-transform: none; white-space: normal; widows: auto;
word-spacing: 0px; -webkit-text-stroke-width: 0px;"
class="">
<br style="font-family: Helvetica; font-size: 12px;
font-style: normal; font-variant: normal; font-weight:
normal; letter-spacing: normal; line-height: normal;
orphans: auto; text-align: start; text-indent: 0px;
text-transform: none; white-space: normal; widows: auto;
word-spacing: 0px; -webkit-text-stroke-width: 0px;"
class="">
<span style="font-family: Helvetica; font-size: 12px;
font-style: normal; font-variant: normal; font-weight:
normal; letter-spacing: normal; line-height: normal;
orphans: auto; text-align: start; text-indent: 0px;
text-transform: none; white-space: normal; widows: auto;
word-spacing: 0px; -webkit-text-stroke-width: 0px; float:
none; display: inline !important;" class="">Is the best
course of action here to declare one a new master and do a
ipa-replica-manage re-initialize to all of the others from
that one?</span><br style="font-family: Helvetica;
font-size: 12px; font-style: normal; font-variant: normal;
font-weight: normal; letter-spacing: normal; line-height:
normal; orphans: auto; text-align: start; text-indent:
0px; text-transform: none; white-space: normal; widows:
auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;"
class="">
<br style="font-family: Helvetica; font-size: 12px;
font-style: normal; font-variant: normal; font-weight:
normal; letter-spacing: normal; line-height: normal;
orphans: auto; text-align: start; text-indent: 0px;
text-transform: none; white-space: normal; widows: auto;
word-spacing: 0px; -webkit-text-stroke-width: 0px;"
class="">
<br class="">
</div>
</blockquote>
<br class="">
</div>
<div>After doing some testing, that’s exactly what we did and
replication is now working again. It is odd that the
DSRetroclPlugin errors stopped on their own (after approximately
3 hours); the only action taken there was looking at the
cn=changelog base using ldapvi to see what number it was on but
that has to be a sheer coincidence; absolutely no changes were
made. </div>
<div><br class="">
</div>
<div>We’re also still unsure what caused this; our best theory at
the moment is a race condition where everything that could have
gone wrong at that exact moment did..is there any validity to
this?</div>
<div><br class="">
</div>
<div>Thanks,</div>
<pre>
"This message and any attachments may contain confidential information. If you
have received this message in error, any use or distribution is prohibited.
Please notify us by reply e-mail if you have mistakenly received this message,
and immediately and permanently delete it and any attachments. Thank you."</pre>
<br>
<fieldset class="mimeAttachmentHeader"></fieldset>
<br>
</blockquote>
<br>
</body>
</html>