[Freeipa-users] Please help: How to restore IPA Master/Replicas from daily IPA Replica setup???

[Rob Crittenden]

Robinson Tiemuqinke wrote:
> Hi Dmitri, Rich and all,
>
> I am a newbie to Redhat IPA, It looks like pretty cool compared with
> other solutions I've tried before. Thanks a lot for this great product! :)
>
> But there are still some things I needs your help. My main question is:
> How to restore the IPA setup with a daily machine-level IPA Replica backup?
>
> Please let me explain my IPA setup background and backup/restore goals
> trying to reach:
>
> I'm running IPA 2.1.3 on Redhat Enterprise 6.2. The IPA master is setup
> with Dogtag CA system. It is installed first. Then two IPA replicas are
> installed -- with '--setup-ca' options -- for load balancing and
> failover purposes.
>
> To describe my problems/objectives, I'll name the IPA Master as machine
> A, IPA replicas as B and C. and now I've one more extra IPA replica 'D'
> (virtual machine) setup ONLY for backup purposes.
> The setup looks like the following, A is the configuration Hub. B,C,D
> are siblings.
>
> A
> / | \
> B C D
>
> The following are the steps I backup IPA setups and LDAP backends daily
> -- it is a whole machine-level backup (through virtual machine D).
>
> 1, First, IPA replica D is backed up daily. The backup happens like this:
>
> 1.1 on IP replica D, run 'service IPA stop'. Then run 'shutdown -h <D>'.
> On the Hypervisor which holds virtual machine D, do a daily backup of
> the whole virtual disk that D is on.
> 1.2 turn on the IP replica D again.
> 1.3 after virtual machine D is up, on D optionally run a
> 'ipa-replica-manage --force-sync --from <A>' to sync the IPA databases
> forcibly.
>
> Now comes to restore part, which is pretty confusing to me. I've tried
> several times, and every times it comes this or that kinds of issues and
> so I am wondering that correct steps/ineraction of IPA Master/replicas
> are the king :(
>
> 2, case #1, A is broken, like disc failure, and then re-imaged after
> several days.
>
> 2.1 How to rebuild the IPA Master/Hub A after A is re-imaged, with the
> daily backup from IPA replica D?

The first thing you'll need to do is to connect your other replias 
together, either by picking a new hub or adding links to each one. Then 
you'll need to delete the replication agreement to A. You should be left 
with a set of servers that continues to replicate.

So, for arguments sake, we promote B to be the new hub:

On B:

# ipa-replica-manage connect C
# ipa-replica-manage connect D
# ipa-replica-manage del --force A
# ipactl restart

On C:

# ipa-replica-manage del --force A
# ipactl restart

On D:

# ipa-replica-manage del --force A
# ipactl restart

It is unclear what you mean by re-imaged. Are you restoring from backup 
or installing it fresh? I'll assume it is a new install. You'll need to 
prepare a replica file for A and install it as a replica. Then if you 
want to keep A as the primary you'll need to change the replication 
agreements back to it is the hub (using ipa-replica-manage connect and 
disconnect).

When you install the new A server it should get all the changes needed, 
you should be done.

You'll want to check the documentation on promoting a master to verify 
that only one server is the CRL generator (at this point there may be none).

> 2.2 do I have to check some files on A into subversion immediately after
> A was initially installed?

The only thing you really need to save is the cacert.p12 file. This is 
your root CA.

> 2.3 Please describe the steps. I'll follow exactly and report the results.
>
> 3, case #2, A is working, but either B, or C is broken.
>
> 3.1 It looks that I don't need the daily backup of D to kick in, is that
> right?

No, D is unrelated.

> 3.2 What are the correct steps on A; and B after it is re-imaged?

On A:
# ipa-replica-manage del B
# ipactl restart
# ipa-replica-prepare B

On B
# ipa-replica-install B

You'll probably need/want to clean RUV, 
http://directory.fedoraproject.org/wiki/Howto:CLEANRUV

> 3.3 Please describe the steps. I'll follow exactly and report the results.
>
> 4, case #3, If some un-expected IPA changes happens on A -- like all
> users are deleted by human mistakes --, and even worse, all the changes
> are propagated to B and C in minutes.
>
> 4.1 How can I recover the IPA setup from daily backup from D?

We have not yet documented how to recover from tombstones or an offline 
replica.

> 4.2 which IPA master/replicas I should recover first? IPA master A, or
> IPA replicas B/C? and then how to recover others left one by one?

If the entries are re-added on any of the replicas it will be propogated 
out.

> 4.3 Do I have to disconnect replication agreement of B,C,D from A first?

Depends on how 4.1 gets answered which we are still investigating.

> 4.4 Please describe the steps. I'll follow exactly and report the results.
>
> I've heard something about tombstone records too, Not sure whether the
> problem still exists in 2.1.3, or 2.2.0(on 6.3Beta)? If so, How can I
> avoid it with correct recovery steps/interactions.

It is RUV that is the problem. This 389-ds wiki page describes how to 
clean up: http://directory.fedoraproject.org/wiki/Howto:CLEANRUV

The 389-ds team is working to make this less manual.

rob