[Freeipa-users] Please help: How to restore IPA Master/Replicas from daily IPA Replica setup???

Mon May 21 14:10:16 UTC 2012

Gelen James wrote:
> Hi Mmitri, Rob and all.
>
> Thanks for your instructions. I've performed your steps on case#1:
> replacing failed IPA master. The results, and my confusion and
> questions, are all detailed below. In general, please setup your own
> real test environment, and write down the detailed steps one by one clearly.
>
> It took me more than one week and still no clues. Frankly, your steps in
> the formal email are kind of over-simplified for normal IPA users, and
> not covering how the CA LDAP backend will be handled.
>
> The problem is the CA backend. All the replicas still trying to sync to
> old failed IPA master, even after reboot.
>
> Could be that the 'ipa-replica-manage' only manages the user data
> replication? and 'ipa-csreplica-manage' only handles CA-end replication?
> In other words, when build, or tear down, IPA replication between two
> servers, do we need to deal with both replication types with
> 'ipa-replica-mange' AND 'ipa-csreplica-manage'? If so, then why who
> should run first?

Yes, the replication agreements are managed separately which is why 
there are separate tools. This allows you to have a different 
replication topology for the CA than IPA user data.

The order the commands are executed doesn't matter.

>
> The error messages in /var/log/dirsrv/slapd-PKI-IPA/errors are attached,
> same from B,C,D replicas.
>
> [19/May/2012:19:40:48 -0700] - 389-Directory/1.2.9.16 B2012.023.214
> starting up
> [19/May/2012:19:40:48 -0700] - slapd started. Listening on All
> Interfaces port 7389 for LDAP requests
> [19/May/2012:19:40:48 -0700] - Listening on All Interfaces port 7390 for
> LDAPS requests
> [19/May/2012:19:40:50 -0700] slapi_ldap_bind - Error: could not send
> startTLS request: error -1 (Can't contact LDAP server)
> [19/May/2012:19:40:50 -0700] NSMMReplicationPlugin -
> agmt="cn=cloneAgreement1-B.example.com-pki-ca" (<A>:7389): Replication
> bind with SIMPLE auth failed: LDAP error -1 (Can't contact LDAP server)
> ((null))
> [19/May/2012:19:40:57 -0700] slapi_ldap_bind - Error: could not send
> startTLS request: error -1 (Can't contact LDAP server)
> [19/May/2012:19:41:03 -0700] slapi_ldap_bind - Error: could not send
> startTLS request: error -1 (Can't contact LDAP server)
> [19/May/2012:19:41:15 -0700] slapi_ldap_bind - Error: could not send
> startTLS request: error -1 (Can't contact LDAP server)
> [19/May/2012:19:41:39 -0700] slapi_ldap_bind - Error: could not send
> startTLS request: error -1 (Can't contact LDAP server)
> [19/May/2012:19:42:27 -0700] slapi_ldap_bind - Error: could not send
> startTLS request: error -1 (Can't contact LDAP server)
> [19/May/2012:19:44:03 -0700] slapi_ldap_bind - Error: could not send
> startTLS request: error -1 (Can't contact LDAP server)
> [19/May/2012:19:47:15 -0700] slapi_ldap_bind - Error: could not send
> startTLS request: error -1 (Can't contact LDAP server)
> [root@<B> ~]#
>
> After seeing the above messages, I tried to run similar commands for CA
> replication, it shows that replication agreement (which replication
> agreement? User data, or CA data ?? ) exists already.
>
> on B,
> ipa-csreplica-manage connect C
> ipa-csreplica-manage connect D
> ipa-csreplica-manage del A --force
> ipactl restart
>
> on C,
> ipa-csreplica-manage del A --force
> ipactl restart
>
> on D,
> ipa-csreplica-manage del A --force
> ipactl restart
>
>
> [root at B ~]# ipa-csreplica-manage --password=xxxxxxx connect C.example.com
> This replication agreement already exists.
> [root at B ~]#
>
> [root at B ~]# ipa-csreplica-manage --password=xxxxxxx connect D.example.com
> This replication agreement already exists.
> [root at B ~]#
>
> [root at B ~]# ipa-csreplica-manage --password=xxxxxxx del C.example.com
> --force
> Unable to connect to replica A.example.com, forcing removal
> Failed to get data from 'A.example.com': Can't contact LDAP server
> Forcing removal on 'B.example.com'
> [root at B ~]#
>
> ....
>
> After restarting IPA services on B, C, D, and now the error messages
> finally got away from CA errors log file.
>
> But we still can not find the CA replication setups. Please see the
> difference of output from 'ipa-replica-manage' and 'ipa-csreplica-manage':
>
> [root at B ~] ipa-replica-manage list
> B.example.com
> C.example.com
> D.example.com
>
> [root at B ~] ipa-csreplica-manage list
> B.example.com
> C.example.com
> D.example.com
>
> [root at B ~] ipa-replica-manage list B.example.com
> C.example.com
> D.example.com
>
> [root at B ~] ipa-csreplica-manage list B.example.com
> ## Nothing at all!
>
> Please have a check and give correct command and sequences for us IPA
> users. It is such a pain to spend so much time and still can not get
> restoration work as expected. Even worse is, Have no idea how the
> 'ipa-replica-manage' and 'ipa-csreplica-manage' work together behind the
> scene.
>
> Thanks a lot.
>
> --Gelen
>
>
>
>
> ------------------------------------------------------------------------
> *From:* Rob Crittenden <rcritten at redhat.com>
> *To:* Robinson Tiemuqinke <hahaha_30k at yahoo.com>
> *Cc:* "Freeipa-users at redhat.com" <Freeipa-users at redhat.com>; Rich
> Megginson <rmeggins at redhat.com>; Dmitri Pal <dpal at redhat.com>
> *Sent:* Tuesday, May 15, 2012 9:57 AM
> *Subject:* Re: [Freeipa-users] Please help: How to restore IPA
> Master/Replicas from daily IPA Replica setup???
>
> Robinson Tiemuqinke wrote:
>  > Hi Dmitri, Rich and all,
>  >
>  > I am a newbie to Redhat IPA, It looks like pretty cool compared with
>  > other solutions I've tried before. Thanks a lot for this great
> product! :)
>  >
>  > But there are still some things I needs your help. My main question is:
>  > How to restore the IPA setup with a daily machine-level IPA Replica
> backup?
>  >
>  > Please let me explain my IPA setup background and backup/restore goals
>  > trying to reach:
>  >
>  > I'm running IPA 2.1.3 on Redhat Enterprise 6.2. The IPA master is setup
>  > with Dogtag CA system. It is installed first. Then two IPA replicas are
>  > installed -- with '--setup-ca' options -- for load balancing and
>  > failover purposes.
>  >
>  > To describe my problems/objectives, I'll name the IPA Master as machine
>  > A, IPA replicas as B and C. and now I've one more extra IPA replica 'D'
>  > (virtual machine) setup ONLY for backup purposes.
>  > The setup looks like the following, A is the configuration Hub. B,C,D
>  > are siblings.
>  >
>  > A
>  > / | \
>  > B C D
>  >
>  > The following are the steps I backup IPA setups and LDAP backends daily
>  > -- it is a whole machine-level backup (through virtual machine D).
>  >
>  > 1, First, IPA replica D is backed up daily. The backup happens like this:
>  >
>  > 1.1 on IP replica D, run 'service IPA stop'. Then run 'shutdown -h <D>'.
>  > On the Hypervisor which holds virtual machine D, do a daily backup of
>  > the whole virtual disk that D is on.
>  > 1.2 turn on the IP replica D again.
>  > 1.3 after virtual machine D is up, on D optionally run a
>  > 'ipa-replica-manage --force-sync --from <A>' to sync the IPA databases
>  > forcibly.
>  >
>  > Now comes to restore part, which is pretty confusing to me. I've tried
>  > several times, and every times it comes this or that kinds of issues and
>  > so I am wondering that correct steps/ineraction of IPA Master/replicas
>  > are the king :(
>  >
>  > 2, case #1, A is broken, like disc failure, and then re-imaged after
>  > several days.
>  >
>  > 2.1 How to rebuild the IPA Master/Hub A after A is re-imaged, with the
>  > daily backup from IPA replica D?
>
> The first thing you'll need to do is to connect your other replias
> together, either by picking a new hub or adding links to each one. Then
> you'll need to delete the replication agreement to A. You should be left
> with a set of servers that continues to replicate.
>
> So, for arguments sake, we promote B to be the new hub:
>
> On B:
>
> # ipa-replica-manage connect C
> # ipa-replica-manage connect D
> # ipa-replica-manage del --force A
> # ipactl restart
>
> On C:
>
> # ipa-replica-manage del --force A
> # ipactl restart
>
> On D:
>
> # ipa-replica-manage del --force A
> # ipactl restart
>
> It is unclear what you mean by re-imaged. Are you restoring from backup
> or installing it fresh? I'll assume it is a new install. You'll need to
> prepare a replica file for A and install it as a replica. Then if you
> want to keep A as the primary you'll need to change the replication
> agreements back to it is the hub (using ipa-replica-manage connect and
> disconnect).
>
> When you install the new A server it should get all the changes needed,
> you should be done.
>
> You'll want to check the documentation on promoting a master to verify
> that only one server is the CRL generator (at this point there may be none).
>
>  > 2.2 do I have to check some files on A into subversion immediately after
>  > A was initially installed?
>
> The only thing you really need to save is the cacert.p12 file. This is
> your root CA.
>
>  > 2.3 Please describe the steps. I'll follow exactly and report the
> results.
>  >
>  > 3, case #2, A is working, but either B, or C is broken.
>  >
>  > 3.1 It looks that I don't need the daily backup of D to kick in, is that
>  > right?
>
> No, D is unrelated.
>
>  > 3.2 What are the correct steps on A; and B after it is re-imaged?
>
> On A:
> # ipa-replica-manage del B
> # ipactl restart
> # ipa-replica-prepare B
>
> On B
> # ipa-replica-install B
>
> You'll probably need/want to clean RUV,
> http://directory.fedoraproject.org/wiki/Howto:CLEANRUV
>
>  > 3.3 Please describe the steps. I'll follow exactly and report the
> results.
>  >
>  > 4, case #3, If some un-expected IPA changes happens on A -- like all
>  > users are deleted by human mistakes --, and even worse, all the changes
>  > are propagated to B and C in minutes.
>  >
>  > 4.1 How can I recover the IPA setup from daily backup from D?
>
> We have not yet documented how to recover from tombstones or an offline
> replica.
>
>  > 4.2 which IPA master/replicas I should recover first? IPA master A, or
>  > IPA replicas B/C? and then how to recover others left one by one?
>
> If the entries are re-added on any of the replicas it will be propogated
> out.
>
>  > 4.3 Do I have to disconnect replication agreement of B,C,D from A first?
>
> Depends on how 4.1 gets answered which we are still investigating.
>
>  > 4.4 Please describe the steps. I'll follow exactly and report the
> results.
>  >
>  > I've heard something about tombstone records too, Not sure whether the
>  > problem still exists in 2.1.3, or 2.2.0(on 6.3Beta)? If so, How can I
>  > avoid it with correct recovery steps/interactions.
>
> It is RUV that is the problem. This 389-ds wiki page describes how to
> clean up: http://directory.fedoraproject.org/wiki/Howto:CLEANRUV
>
> The 389-ds team is working to make this less manual.
>
> rob
>
>