[Freeipa-users] Replicas in a state of confusion

[Ian Levesque]

Hello,

On our production IPA servers, we have been running in a multi-master state successfully for several weeks. Yesterday, while attempting to modify some permissions and roles using the web UI, we had an odd problem where the web UI became unresponsive. In an attempt to resolve the issue, I issued an `ipactl restart` and when that didn't fix the web UI, I rebooted the VM. When IPA services came back up, the replica would try to sync and the primary would crash. I noticed that if IPA on the replica was off, the primary server was fine.  So, after fighting with this for a few hours I decided to remove the replica and start the replication process again.

Replica reinstall didn't go so well:

	[root at sbgrid-directory ~]# ipa-replica-manage disconnect sbgrid-directory-replica.in.hwlab
	[root at sbgrid-directory ~]# ipa-replica-manage del sbgrid-directory-replica.in.hwlab
	(this failed, unfortunately I didn't record the error)

	[root at sbgrid-directory ~]# ipa-replica-manage del -f sbgrid-directory-replica.in.hwlab

	[root at sbgrid-directory-replica ~]# ipa-server-install --uninstall 
	[root at sbgrid-directory-replica ~]# ipa-replica-install /root/replica-info-sbgrid-directory-replica.in.hwlab.gpg
	(...all ok...)
	Starting replication, please wait until this has completed.
	[sbgrid-directory.in.hwlab] reports: Update failed! Status: [-2  - System error]
	creation of replica failed: Failed to start replication
	
	Your system may be partly configured.
	Run /usr/sbin/ipa-server-install --uninstall to clean up.

When I try to start the primary (sbgrid-directory) server, I see these errors:

/var/log/messages:

	ns-slapd: GSSAPI Error: Unspecified GSS failure.  Minor code may provide more information (Cannot contact any KDC for requested realm)

/var/log/dirsrv/slapd-SBGRID-ORG/errors:

	NSMMReplicationPlugin - repl_set_mtn_referrals: could not set referrals for replica dc=sbgrid,dc=org: 20

	set_krb5_creds - Could not get initial credentials for principal [ldap/sbgrid-directory.in.hwlab at SBGRID.ORG] in keytab [WRFILE:/etc/dirsrv/ds.keytab]: -1765328228 (Cannot contact any KDC for requested realm)

	slapd_ldap_sasl_interactive_bind - Error: could not perform interactive bind for id [] mech [GSSAPI]: error -2 (Local error) (SASL(-1): generic failure: GSSAPI Error: Unspecified GSS failure.  Minor code may provide more information (Credentials cache file '/tmp/krb5cc_496' not found))

	slapi_ldap_bind - Error: could not perform interactive bind for id [] mech [GSSAPI]: error -2 (Local error)


Yikes, what a mess -- thanks for any help.
Ian