<html>
<head>
<meta content="text/html; charset=ISO-8859-1"
http-equiv="Content-Type">
</head>
<body bgcolor="#FFFFFF" text="#000000">
On 05/20/2012 02:28 AM, Gelen James wrote:
<blockquote
cite="mid:1337502493.99772.YahooMailNeo@web160705.mail.bf1.yahoo.com"
type="cite">
<div style="color: rgb(0, 0, 0); background-color: rgb(255, 255,
255); font-family: times new roman,new york,times,serif;
font-size: 12pt;">
<div><span>rebuild the old IPA master A is half success too.
The error also happens at CA replication side. </span></div>
<div><span><br>
</span></div>
<div><span>After replica preparation at replica B, nuke and
reinstall old A, and create A from the replica info file
prepared on B, The user LDAP replication works fine. while
the CA replication broken terribly. the error messages on A
inside file /var/log/dirsrv/slapd-PKI-IPA/errors are pasted
below:</span></div>
<div><span><br>
</span></div>
<div><span>
<div>[20/May/2012:01:17:36 -0700] - 389-Directory/1.2.9.16
B2012.023.214 starting up</div>
<div>[20/May/2012:01:17:36 -0700] NSMMReplicationPlugin -
replica_check_for_data_reload: Warning: data for replica
o=ipaca does not match the data in the changelog (replica
data (4fb8a7f3000404430000) > changelog
(4fb84ba7000000560000)). Recreating the changelog file.
This could affect replication with replica's consumers in
which case the consumers should be reinitialized.</div>
</span></div>
</div>
</blockquote>
<br>
This error message is normal - you should only see this once, just
after a replica has been initialized.<br>
<br>
<blockquote
cite="mid:1337502493.99772.YahooMailNeo@web160705.mail.bf1.yahoo.com"
type="cite">
<div style="color: rgb(0, 0, 0); background-color: rgb(255, 255,
255); font-family: times new roman,new york,times,serif;
font-size: 12pt;">
<div><span>
<div>[20/May/2012:01:17:37 -0700] - slapd started.
Listening on All Interfaces port 7389 for LDAP requests</div>
<div>[20/May/2012:01:17:37 -0700] - Listening on All
Interfaces port 7390 for LDAPS requests</div>
<div>[root@<A> ~]# </div>
<div><br>
</div>
<div>check the RUV records shows a number too big: 1091,
while all others are smaller than 100.</div>
</span></div>
</div>
</blockquote>
<br>
It's not "too big" as far as the protocol is concerned, but it is
strange that it is so much larger than the other values.<br>
<br>
<br>
<blockquote
cite="mid:1337502493.99772.YahooMailNeo@web160705.mail.bf1.yahoo.com"
type="cite">
<div style="color:#000; background-color:#fff; font-family:times
new roman, new york, times, serif;font-size:12pt">
<div><span>
<div>There are no RUV records to delete/clear.</div>
<div><br>
</div>
<div>
<div>dn:
nsuniqueid=ffffffff-ffffffff-ffffffff-ffffffff,o=ipaca</div>
<div>objectClass: top</div>
<div>objectClass: nsTombstone</div>
<div>objectClass: extensibleobject</div>
<div>nsds50ruv: {replicageneration} 4fb8187f000000600000</div>
<div>nsds50ruv: {replica 97 <a class="moz-txt-link-freetext" href="ldap://B.example.com:7389">ldap://B.example.com:7389</a>}
4fb81886000000</div>
<div> 610000 4fb8a7ca000100610000</div>
<div>nsds50ruv: {replica 1091 <a class="moz-txt-link-freetext" href="ldap://A.example.com:7389">ldap://A.example.com:7389</a>}
4fb8a7c60001044</div>
<div> 30000 4fb8a8a9000104430000</div>
<div>nsds50ruv: {replica 91 <a class="moz-txt-link-freetext" href="ldap://C.example.com:7389">ldap://C.example.com:7389</a>}
4fb81f54000000</div>
<div> 5b0000 4fb84db60000005b0000</div>
<div>nsds50ruv: {replica 86 <a class="moz-txt-link-freetext" href="ldap://D.example.com:7389">ldap://D.example.com:7389</a>}
4fb821a6000000</div>
<div> 560000 4fb84ba7000000560000</div>
<div>o: ipaca </div>
<div>nsruvReplicaLastModified: {replica 97
<a class="moz-txt-link-freetext" href="ldap://B.example.com:7389">ldap://B.example.com:7389</a>}</div>
<div> 4fb8a7c7</div>
<div>nsruvReplicaLastModified: {replica 1091
<a class="moz-txt-link-freetext" href="ldap://A.example.com:7389">ldap://A.example.com:7389</a>} </div>
<div> 4fb8a8a6</div>
<div>nsruvReplicaLastModified: {replica 91
<a class="moz-txt-link-freetext" href="ldap://C.example.com:7389">ldap://C.example.com:7389</a>}</div>
<div> 00000000</div>
<div>nsruvReplicaLastModified: {replica 86
<a class="moz-txt-link-freetext" href="ldap://D.example.com:7389">ldap://D.example.com:7389</a>}</div>
<div> 00000000</div>
<div><br>
</div>
<div>Please advise. Thanks.</div>
<div><br>
</div>
<div>--Gelen</div>
</div>
<div><br>
</div>
<div><br>
</div>
</span></div>
<div><span><br>
</span></div>
<div><span><br>
</span></div>
<div><span><br>
</span></div>
<div><span> </span></div>
<div><br>
</div>
<div style="font-size: 12pt; font-family: 'times new roman',
'new york', times, serif; ">
<div style="font-size: 12pt; font-family: 'times new roman',
'new york', times, serif; ">
<div dir="ltr"> <font face="Arial" size="2">
<hr size="1"> <b><span style="font-weight:bold;">From:</span></b>
Gelen James <a class="moz-txt-link-rfc2396E" href="mailto:hahaha_30k@yahoo.com"><hahaha_30k@yahoo.com></a><br>
<b><span style="font-weight: bold;">To:</span></b> Rob
Crittenden <a class="moz-txt-link-rfc2396E" href="mailto:rcritten@redhat.com"><rcritten@redhat.com></a>; Dmitri Pal
<a class="moz-txt-link-rfc2396E" href="mailto:dpal@redhat.com"><dpal@redhat.com></a> <br>
<b><span style="font-weight: bold;">Cc:</span></b>
<a class="moz-txt-link-rfc2396E" href="mailto:Freeipa-users@redhat.com">"Freeipa-users@redhat.com"</a>
<a class="moz-txt-link-rfc2396E" href="mailto:Freeipa-users@redhat.com"><Freeipa-users@redhat.com></a> <br>
<b><span style="font-weight: bold;">Sent:</span></b>
Sunday, May 20, 2012 12:08 AM<br>
<b><span style="font-weight: bold;">Subject:</span></b>
Re: [Freeipa-users] Please help: How to restore IPA
Master/Replicas from daily IPA Replica setup???<br>
</font> </div>
<br>
<div id="yiv600681867">
<div>
<div style="color: rgb(0, 0, 0); background-color:
rgb(255, 255, 255); font-size: 12pt; font-family:
'times new roman', 'new york', times, serif; ">
<div><span>Hi Mmitri, Rob and all.</span></div>
<div><span><br>
</span></div>
<div><span> Thanks for your instructions. I've
performed your steps on case#1: replacing failed
IPA master. The results, and my confusion and
questions, are all detailed below. In general,
please setup your own real test environment, and
write down the detailed steps one by one clearly.</span></div>
<div><span><br>
</span></div>
<div><span> </span>It took me more than one week and
still no clues. Frankly, your steps in the formal
email are kind of over-simplified for normal IPA
users, and not covering how the CA LDAP backend will
be handled.</div>
<div><span><br>
</span></div>
<div><span>The problem is the CA backend. All the
replicas still trying to sync to old failed IPA
master, even after reboot. </span></div>
<div><span><br>
</span></div>
<div><span>Could be that the 'ipa-replica-manage' only
manages the user data replication? and
'ipa-csreplica-manage' only handles CA-end
replication? </span>In other words, when build, or
tear down, IPA replication between two servers, do
we need to deal with both replication types with
'ipa-replica-mange' AND 'ipa-csreplica-manage'? If
so, then why who should run first?</div>
<div><span><br>
</span></div>
<div><span>The error messages in
/var/log/dirsrv/slapd-PKI-IPA/errors are attached,
same from B,C,D replicas. </span></div>
<div><span><br>
</span></div>
<div>[19/May/2012:19:40:48 -0700] -
389-Directory/1.2.9.16 B2012.023.214 starting up</div>
<div>[19/May/2012:19:40:48 -0700] - slapd started.
Listening on All Interfaces port 7389 for LDAP
requests</div>
<div>[19/May/2012:19:40:48 -0700] - Listening on All
Interfaces port 7390 for LDAPS requests</div>
<div>[19/May/2012:19:40:50 -0700] slapi_ldap_bind -
Error: could not send startTLS request: error -1
(Can't contact LDAP server)</div>
<div>[19/May/2012:19:40:50 -0700]
NSMMReplicationPlugin -
agmt="cn=cloneAgreement1-B.example.com-pki-ca"
(<A>:7389): Replication bind with SIMPLE auth
failed: LDAP error -1 (Can't contact LDAP server)
((null))</div>
<div>[19/May/2012:19:40:57 -0700] slapi_ldap_bind -
Error: could not send startTLS request: error -1
(Can't contact LDAP server)</div>
<div>[19/May/2012:19:41:03 -0700] slapi_ldap_bind -
Error: could not send startTLS request: error -1
(Can't contact LDAP server)</div>
<div>[19/May/2012:19:41:15 -0700] slapi_ldap_bind -
Error: could not send startTLS request: error -1
(Can't contact LDAP server)</div>
<div>[19/May/2012:19:41:39 -0700] slapi_ldap_bind -
Error: could not send startTLS request: error -1
(Can't contact LDAP server)</div>
<div>[19/May/2012:19:42:27 -0700] slapi_ldap_bind -
Error: could not send startTLS request: error -1
(Can't contact LDAP server)</div>
<div>[19/May/2012:19:44:03 -0700] slapi_ldap_bind -
Error: could not send startTLS request: error -1
(Can't contact LDAP server)</div>
<div>[19/May/2012:19:47:15 -0700] slapi_ldap_bind -
Error: could not send startTLS request: error -1
(Can't contact LDAP server)</div>
<div><span>[root@<B> ~]# </span></div>
<div><br>
</div>
<div>After seeing the above messages, I tried to run
similar commands for CA replication, it shows that
replication agreement (which replication agreement?
User data, or CA data ?? ) exists already.</div>
<div><br>
</div>
<div>on B,</div>
<div> </div>
<div>ipa-csreplica-manage connect C</div>
<div>ipa-csreplica-manage connect D</div>
<div>ipa-csreplica-manage del A --force</div>
<div>ipactl restart </div>
<div><br>
</div>
<div>on C, </div>
<div>
<div>ipa-csreplica-manage del A --force</div>
<div>ipactl restart </div>
<div><br>
</div>
<div>on D,</div>
<div>
<div>ipa-csreplica-manage del A --force</div>
<div>ipactl restart </div>
</div>
</div>
<div><br>
</div>
<div><br>
</div>
<div>
<div>[root@B ~]# ipa-csreplica-manage
--password=xxxxxxx connect <a
moz-do-not-send="true" target="_blank"
href="http://C.example.com">C.example.com</a></div>
<div>This replication agreement already exists.</div>
<div>[root@B ~]# </div>
</div>
<div><br>
</div>
<div>
<div>[root@B ~]# ipa-csreplica-manage
--password=xxxxxxx connect <a
moz-do-not-send="true" target="_blank"
href="http://D.example.com">D.example.com</a></div>
<div>This replication agreement already exists.</div>
<div>[root@B ~]# </div>
<div><br>
</div>
<div>
<div>[root@B ~]# ipa-csreplica-manage
--password=xxxxxxx del C.example.com --force</div>
<div>
<div>Unable to connect to replica <a
moz-do-not-send="true" target="_blank"
href="http://A.example.com">A.example.com</a>,
forcing removal</div>
<div>Failed to get data from 'A.example.com':
Can't contact LDAP server</div>
<div>Forcing removal on '<a
moz-do-not-send="true" target="_blank"
href="http://B.example.com">B.example.com</a>'</div>
</div>
<div>[root@B ~]# </div>
<div><br>
</div>
<div>....</div>
<div><br>
</div>
<div>After restarting IPA services on B, C, D, and
now the error messages finally got away from CA
errors log file. </div>
<div><br>
</div>
<div>But we still can not find the CA replication
setups. Please see the difference of output from
'ipa-replica-manage' and 'ipa-csreplica-manage':</div>
<div><br>
</div>
<div>[root@B ~] ipa-replica-manage list</div>
<div>B.example.com</div>
<div>C.example.com</div>
<div>D.example.com</div>
<div><br>
</div>
<div>
<div>[root@B ~] ipa-csreplica-manage list</div>
<div>B.example.com</div>
<div>C.example.com</div>
<div>D.example.com</div>
<div><br>
</div>
<div>
<div>[root@B ~] ipa-replica-manage list
B.example.com</div>
<div>C.example.com</div>
<div>D.example.com</div>
<div><br>
</div>
<div>
<div>[root@B ~] ipa-csreplica-manage list
B.example.com</div>
<div>## Nothing at all!</div>
<div><br>
</div>
<div>Please have a check and give correct
command and sequences for us IPA users. It
is such a pain to spend so much time and
still can not get restoration work as
expected. Even worse is, Have no idea how
the 'ipa-replica-manage' and
'ipa-csreplica-manage' work together
behind the scene.</div>
<div><br>
</div>
<div>Thanks a lot.</div>
<div><br>
</div>
<div>--Gelen</div>
</div>
</div>
<div><br>
</div>
</div>
<div><br>
</div>
<div><br>
</div>
<div><br>
</div>
</div>
</div>
<div style="font-size: 12pt; font-family: times,
serif; ">
<div style="font-size: 12pt; font-family: times,
serif; ">
<div dir="ltr"> <font face="Arial" size="2">
<hr size="1"> <b><span
style="font-weight:bold;">From:</span></b>
Rob Crittenden <a class="moz-txt-link-rfc2396E" href="mailto:rcritten@redhat.com"><rcritten@redhat.com></a><br>
<b><span style="font-weight:bold;">To:</span></b>
Robinson Tiemuqinke
<a class="moz-txt-link-rfc2396E" href="mailto:hahaha_30k@yahoo.com"><hahaha_30k@yahoo.com></a> <br>
<b><span style="font-weight:bold;">Cc:</span></b>
<a class="moz-txt-link-rfc2396E" href="mailto:Freeipa-users@redhat.com">"Freeipa-users@redhat.com"</a>
<a class="moz-txt-link-rfc2396E" href="mailto:Freeipa-users@redhat.com"><Freeipa-users@redhat.com></a>; Rich
Megginson <a class="moz-txt-link-rfc2396E" href="mailto:rmeggins@redhat.com"><rmeggins@redhat.com></a>; Dmitri
Pal <a class="moz-txt-link-rfc2396E" href="mailto:dpal@redhat.com"><dpal@redhat.com></a> <br>
<b><span style="font-weight:bold;">Sent:</span></b>
Tuesday, May 15, 2012 9:57 AM<br>
<b><span style="font-weight:bold;">Subject:</span></b>
Re: [Freeipa-users] Please help: How to
restore IPA Master/Replicas from daily IPA
Replica setup???<br>
</font> </div>
<br>
Robinson Tiemuqinke wrote:<br>
> Hi Dmitri, Rich and all,<br>
><br>
> I am a newbie to Redhat IPA, It looks like
pretty cool compared with<br>
> other solutions I've tried before. Thanks a
lot for this great product! :)<br>
><br>
> But there are still some things I needs your
help. My main question is:<br>
> How to restore the IPA setup with a daily
machine-level IPA Replica backup?<br>
><br>
> Please let me explain my IPA setup background
and backup/restore goals<br>
> trying to reach:<br>
><br>
> I'm running IPA 2.1.3 on Redhat Enterprise
6.2. The IPA master is setup<br>
> with Dogtag CA system. It is installed first.
Then two IPA replicas are<br>
> installed -- with '--setup-ca' options -- for
load balancing and<br>
> failover purposes.<br>
><br>
> To describe my problems/objectives, I'll name
the IPA Master as machine<br>
> A, IPA replicas as B and C. and now I've one
more extra IPA replica 'D'<br>
> (virtual machine) setup ONLY for backup
purposes.<br>
> The setup looks like the following, A is the
configuration Hub. B,C,D<br>
> are siblings.<br>
><br>
> A<br>
> / | \<br>
> B C D<br>
><br>
> The following are the steps I backup IPA
setups and LDAP backends daily<br>
> -- it is a whole machine-level backup
(through virtual machine D).<br>
><br>
> 1, First, IPA replica D is backed up daily.
The backup happens like this:<br>
><br>
> 1.1 on IP replica D, run 'service IPA stop'.
Then run 'shutdown -h <D>'.<br>
> On the Hypervisor which holds virtual machine
D, do a daily backup of<br>
> the whole virtual disk that D is on.<br>
> 1.2 turn on the IP replica D again.<br>
> 1.3 after virtual machine D is up, on D
optionally run a<br>
> 'ipa-replica-manage --force-sync --from
<A>' to sync the IPA databases<br>
> forcibly.<br>
><br>
> Now comes to restore part, which is pretty
confusing to me. I've tried<br>
> several times, and every times it comes this
or that kinds of issues and<br>
> so I am wondering that correct
steps/ineraction of IPA Master/replicas<br>
> are the king :(<br>
><br>
> 2, case #1, A is broken, like disc failure,
and then re-imaged after<br>
> several days.<br>
><br>
> 2.1 How to rebuild the IPA Master/Hub A after
A is re-imaged, with the<br>
> daily backup from IPA replica D?<br>
<br>
The first thing you'll need to do is to connect
your other replias <br>
together, either by picking a new hub or adding
links to each one. Then <br>
you'll need to delete the replication agreement to
A. You should be left <br>
with a set of servers that continues to replicate.<br>
<br>
So, for arguments sake, we promote B to be the new
hub:<br>
<br>
On B:<br>
<br>
# ipa-replica-manage connect C<br>
# ipa-replica-manage connect D<br>
# ipa-replica-manage del --force A<br>
# ipactl restart<br>
<br>
On C:<br>
<br>
# ipa-replica-manage del --force A<br>
# ipactl restart<br>
<br>
On D:<br>
<br>
# ipa-replica-manage del --force A<br>
# ipactl restart<br>
<br>
It is unclear what you mean by re-imaged. Are you
restoring from backup <br>
or installing it fresh? I'll assume it is a new
install. You'll need to <br>
prepare a replica file for A and install it as a
replica. Then if you <br>
want to keep A as the primary you'll need to
change the replication <br>
agreements back to it is the hub (using
ipa-replica-manage connect and <br>
disconnect).<br>
<br>
When you install the new A server it should get
all the changes needed, <br>
you should be done.<br>
<br>
You'll want to check the documentation on
promoting a master to verify <br>
that only one server is the CRL generator (at this
point there may be none).<br>
<br>
> 2.2 do I have to check some files on A into
subversion immediately after<br>
> A was initially installed?<br>
<br>
The only thing you really need to save is the
cacert.p12 file. This is <br>
your root CA.<br>
<br>
> 2.3 Please describe the steps. I'll follow
exactly and report the results.<br>
><br>
> 3, case #2, A is working, but either B, or C
is broken.<br>
><br>
> 3.1 It looks that I don't need the daily
backup of D to kick in, is that<br>
> right?<br>
<br>
No, D is unrelated.<br>
<br>
> 3.2 What are the correct steps on A; and B
after it is re-imaged?<br>
<br>
On A:<br>
# ipa-replica-manage del B<br>
# ipactl restart<br>
# ipa-replica-prepare B<br>
<br>
On B<br>
# ipa-replica-install B<br>
<br>
You'll probably need/want to clean RUV, <br>
<a class="moz-txt-link-freetext" href="http://directory.fedoraproject.org/wiki/Howto:CLEANRUV">http://directory.fedoraproject.org/wiki/Howto:CLEANRUV</a><br>
<br>
> 3.3 Please describe the steps. I'll follow
exactly and report the results.<br>
><br>
> 4, case #3, If some un-expected IPA changes
happens on A -- like all<br>
> users are deleted by human mistakes --, and
even worse, all the changes<br>
> are propagated to B and C in minutes.<br>
><br>
> 4.1 How can I recover the IPA setup from
daily backup from D?<br>
<br>
We have not yet documented how to recover from
tombstones or an offline <br>
replica.<br>
<br>
> 4.2 which IPA master/replicas I should
recover first? IPA master A, or<br>
> IPA replicas B/C? and then how to recover
others left one by one?<br>
<br>
If the entries are re-added on any of the replicas
it will be propogated <br>
out.<br>
<br>
> 4.3 Do I have to disconnect replication
agreement of B,C,D from A first?<br>
<br>
Depends on how 4.1 gets answered which we are
still investigating.<br>
<br>
> 4.4 Please describe the steps. I'll follow
exactly and report the results.<br>
><br>
> I've heard something about tombstone records
too, Not sure whether the<br>
> problem still exists in 2.1.3, or 2.2.0(on
6.3Beta)? If so, How can I<br>
> avoid it with correct recovery
steps/interactions.<br>
<br>
It is RUV that is the problem. This 389-ds wiki
page describes how to <br>
clean up: <a moz-do-not-send="true"
rel="nofollow" target="_blank"
href="http://directory.fedoraproject.org/wiki/Howto:CLEANRUV">http://directory.fedoraproject.org/wiki/Howto:CLEANRUV</a><br>
<br>
The 389-ds team is working to make this less
manual.<br>
<br>
rob<br>
<br>
<br>
</div>
</div>
</div>
</div>
</div>
<br>
_______________________________________________<br>
Freeipa-users mailing list<br>
<a moz-do-not-send="true"
ymailto="mailto:Freeipa-users@redhat.com"
href="mailto:Freeipa-users@redhat.com">Freeipa-users@redhat.com</a><br>
<a moz-do-not-send="true"
href="https://www.redhat.com/mailman/listinfo/freeipa-users"
target="_blank">https://www.redhat.com/mailman/listinfo/freeipa-users</a><br>
<br>
</div>
</div>
</div>
<br>
<fieldset class="mimeAttachmentHeader"></fieldset>
<br>
<pre wrap="">_______________________________________________
Freeipa-users mailing list
<a class="moz-txt-link-abbreviated" href="mailto:Freeipa-users@redhat.com">Freeipa-users@redhat.com</a>
<a class="moz-txt-link-freetext" href="https://www.redhat.com/mailman/listinfo/freeipa-users">https://www.redhat.com/mailman/listinfo/freeipa-users</a></pre>
</blockquote>
<br>
</body>
</html>