[Freeipa-users] ipa-replica-manage failing to delete a node

[Linder, Rolf]

Hello

First, we really would like to thank the developers / community for the great work doing with FreeIPA!

At our company, we're using a CentOS7 based FreeIPA installation (uspidm01 primary and uspidm02 replica) and it worked like a charm the last couple of months. Last week we suffered a severe outage (DNS related) and are still suffering from this on. We have a similar issue as reported by

https://bugzilla.redhat.com/show_bug.cgi?id=826677  (upstream https://pagure.io/freeipa/issue/2797)
https://www.redhat.com/archives/freeipa-users/2013-May/msg00034.html
https://www.redhat.com/archives/freeipa-users/2012-June/msg00382.html

mainly our synchronization stopped with uspidm02 (replica) logging:

"[27/Mar/2017:11:57:39.756880208 +0200] NSMMReplicationPlugin - agmt="cn=meTouspidm01.[domainname].[tld]" (uspidm01:389): Data required to update replica has been purged from the changelog. The replica must be reinitialized."

We tried to re-initialize using "ipa-replica-manage re-initialize --from uspidm01.[domain].[tld]" but this failed. After this we headed for a "clean" first remove then add again solution (knowing that we will temporarily loss the replica and loss any unsynchronized changes). We followed upstream documentation from RedHat (see below) on this.

Unfortunately, the "ipa-replica-manage list" command still lists both servers (uspidm01 and uspidm02). The error given by a forced removal using "ipa-replica-manage del --no-lookup --force --cleanup uspidm02.[domain].[tld]" is

Cleaning a master is irreversible.
This should not normally be require, so use cautiously.
Continue to clean master? [no]: yes
unexpected error: This entry already exists

we then tried to further debug the python code used (ipa-replica-manage) and could identify using PDB that the function "replica_cleanup" from "/usr/lib/python2.7/site-packages/ipaserver/install/replication.py" complains about duplicate entries:


/usr/lib/python2.7/site-packages/ipaserver/install/replication.py(1203)replica_cleanup()
-> self.conn.delete_entry(entry)
(Pdb) n
DuplicateEntry: Duplicat...exists',)
> /usr/lib/python2.7/site-packages/ipaserver/install/replication.py(1203)replica_cleanup()
-> self.conn.delete_entry(entry)
(Pdb) n
> /usr/lib/python2.7/site-packages/ipaserver/install/replication.py(1204)replica_cleanup()
-> except errors.NotFound:
(Pdb) n
> /usr/lib/python2.7/site-packages/ipaserver/install/replication.py(1206)replica_cleanup()
-> except Exception, e:
...

Using LDAPSearch we can confirm there are still entries listed for the ghost/offline server uspidm02 (which seems the reason why ipa-replica-manage still lists it). But we cannot identify where a duplicate entry is exactly. As long as there are entries for this host, it can not be added again (a ipa-server cannot be removed using "ipa host-del" and adding a new also fails).

Our situation for now is we're having a "read-only" IDM solution since any modification (password change, adding new servers, ...) fails. Adding a new replica (new name) is also failing. We suspect if we could clean up the ghost replica entry we should be able to restore IDM / replica again.

Any help would be greatly appreciated!!

Best regards,
Rolf

Documentation used:
Uninstallation: https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/7/html/Linux_Domain_Identity_Authentication_and_Policy_Guide/replica-uninstall.html
New installation: https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/7/html/Linux_Domain_Identity_Authentication_and_Policy_Guide/creating-the-replica.html

Versions in use: initially both servers were updated to ipa-server-4.4.0-14.el7.centos.6.x86_64, uspidm01 was rollbacked to ipa-server-4.2.0-15.0.1.el7.centos.19.x86_64 (eliminating any upgrade issues)

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/freeipa-users/attachments/20170328/32b804da/attachment.htm>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/x-pkcs7-signature
Size: 5507 bytes
Desc: not available
URL: <http://listman.redhat.com/archives/freeipa-users/attachments/20170328/32b804da/attachment.bin>