[Freeipa-users] Loss of initial master in multi master setup

[Neal Harrington | i-Neda Ltd]

> > Hi IPA Gurus,
> >
> >
> > I had a 3 site multi master IPA replication setup (1 office and 2
> > datacentres) with 2 IPA servers at each site. Each server was
> > replicating successfully to 3 other servers (the other local site
> > server and one server at each of the two remote sites). Everything is
> > running on the default packages from CentOS 7.2 and each server is a
> > full replica (ipa-replica-install
> > /var/lib/ipa/replica-info-id-myserver.fqdn.com.gpg  --setup-ca
> > --setup-dns --mkhomedir --forwarder 8.8.8.8)
> >
> >
> > Everything was ticking over nicely until we had notice that the office
> > site was moving on short notice.
> >
> >
> > I successfully created IPA servers at the new site, setup replication
> > again between the new office and the two datacentres that were to
> > remain online, tested and everything worked as expected -
> > unfortunately in the rush I did not have time to properly retire the
> > IPA servers in the old office.
> >
> >
> > The problem this has caused is that I only ever created users in one
> > of the IPA servers in the original office - so only those servers have
> > a DNA range and I am now unable to create new users on the active
> servers.
> > The original office servers are still in the IPA replication and
> > powered on but offline so potential split brain?
> >
> >
> > I now have two things I would like to know before proceeding:
> >
> >   * Is the best fix here to force remove the original IPA servers and
> >     manually add a new dna range significantly different from the
> >     original to avoid overlaps?
> >   * Is there anything else I should check? I can't see any issues
> >     however did not notice the DNA range until I tried to create a user.
> >
> > Any pointers greatly appreciated.
> >
> >
> > Thanks,
> >
> > Neal.
>
> Hi Neal,
>
> If you already disconnected/decomissioned the old masters then I thnk the
> best you can do is option a, i.e. re-set DNA ranges on replicas to new values
> while avioding overlap with old ranges.
>
> We have an upstream document[1] describing the procedure. Hope it helps.
>
> Also make sure that you migrated CA renewal and CRL master responsibilities
> to the new replicas, otherwise you may get problems with expiring
> certificates which are really hard to solve. See the following guide for details.
> [2]
>
> [1] http://www.freeipa.org/page/V3/Recover_DNA_Ranges
> [2]
> http://www.freeipa.org/page/Howto/Promote_CA_to_Renewal_and_CRL_
> Master
>
> --
> Martin^3 Babinsky

Hi Martin & Rob,

Thank you very much for the pointers. I have added a new range to a IPA server I used the top half of the previous range, I only had 30 ish ID's used so far)
# ipa-replica-manage dnarange-set office03.fqdn.com 310300000-310399999
and this has allowed me to add a user on that server. However when I try to add a user on a different server it still fails with "allocation of new value for range". I was expecting this to request a new range and halve the currently assigned range. Robs link included this command:
# ldapsearch -x -D 'cn=Directory Manager' -W -b cn=posix-ids,cn=dna,cn=ipa,cn=etc,dc=int,dc=i-neda,dc=com
...Which seems to list all of the other servers, including office03.fqdn.com which it shows as having 99999 dnaRemainingValues (all the rest have 0) so the server that cannot add users can see office03 has 99999 unused.

However of more immediate concern now I can create user accounts is the CA replication which I seem to have completely messed up. Most CA replication went back to the (now offline) office and even what I have does not seem to work as expected. Eg on Office03:
# ldapsearch -H ldap://$HOSTNAME -D 'cn=Directory Manager' -W -b 'cn=masters,cn=ipa,cn=etc,dc=example,dc=com' '(&(cn=CA)(ipaConfigString=caRenewalMaster))' dn
<SNIP>
search result
search: 2
result: 32 No such object

Following the instructions to set the master seems to work at first (no errors) but the ldap search for renewal master still returns "result: 32 No Such Object"
# ipa-csreplica-manage set-renewal-master
ipa: WARNING: session memcached servers not running
Directory Manager password:
office03.fqdn.com is now the renewal master

re running the set-renwal-master command reports that this server is already the renewal master.

I think I need to reinitialize the CA replication and connect everything up in a redundant loop as I have with the main replication - however the LDAP query not returning the replication master does not seem right. I have not added any IPA servers since these network changes happened a week ago, is it reasonably safe to assume no certificates will have been created so all servers are effectively in sync?

Your help with this is greatly appreciated. On the plus side the systems we use this for are all dev, not live, so it is a good learning experience for me if nothing else!

Best Regards,
Neal.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/freeipa-users/attachments/20161201/d8cbd015/attachment.htm>