[Freeipa-users] How to make a FreeIPA node replica become Master?

[Sergio Francisco]

Hi,
We have a deployment of FreeIPA using 3 nodes (Master with more 2 replicas).

Recently, the master node had a problem with the process 'ns-slapd'
consuming 100% of CPU. During this problem, DNS service wasn't working, IPA
admin UI encountered timeout, SSH keys to access the hosts are not being
loaded correctly.

We observed in the logs of "dirsrv" that something related to the cachesize
wasn't enough to the space needed and then ns-slapd started a process to
recover it. We let the server running this operation almost one day and
nothing happened.

Today, we tried to:

1 - remove the failed server from the deployment, using the command below,
but unfortunately, it wasn't possible to do from both the 2 other nodes.

ipa-replica-manage del --force mux-idm-p03.muxi.dc --cacert=/etc/ipa/ca.crt
unexpected error: cannot connect to 'ldaps://localhost.localdomain:636

2 - tried to upgrade the failed server to a most recent version of IPA
using ipa-server-upgrade but it stopped in the step to connect

  [5/10]: starting directory server

2016-09-14T13:43:28Z ERROR IPA server upgrade failed: Inspect
/var/log/ipaupgrade.log and run command ipa-server-upgrade manually.
2016-09-14T13:43:28Z DEBUG The ipa-server-upgrade command failed,
exception: error: [Errno 111] Connection refused
2016-09-14T13:43:28Z ERROR [Errno 111] Connection refused

3 - tried to recover the 389-ds database with the command "db_recover -f
-v" but nothing happened.
4 - visited similar threads but none of them helped me

https://www.redhat.com/archives/freeipa-users/2013-May/msg00015.html
https://www.redhat.com/archives/freeipa-users/2015-July/msg00188.html

5 - as we need to urgently recover the service, we tried to rebuild the
failed server, removing and reinstalling all the packages needed by
ipa-server (yum install ipa-server bind bind-dyndb-ldap ipa-server-dns) and
tried to re-join the new server as a replica to receive all the data again,
but it doesn't seems to work.

The other nodes are working well, resolving DNS requests, allowing users to
access the servers using SSH, etc.

Any ideas of what I can do to rebuild the server?

Versions
ipa-server-4.2.0-15.0.1.el7.centos.19.x86_64
ipa-server-dns-4.2.0-15.0.1.el7.centos.19.x86_64
389-ds-base-1.3.4.0-33.el7_2.x86_64
CentOS Linux release 7.2.1511 (Core)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/freeipa-users/attachments/20160914/7fa74dbb/attachment.htm>