[Freeipa-users] re-initialize replica

[Andrew E. Bruno]

On Mon, Oct 05, 2015 at 12:40:42PM +0200, Martin Kosek wrote:
> On 10/02/2015 06:00 PM, Andrew E. Bruno wrote:
> > On Fri, Oct 02, 2015 at 09:56:47AM -0400, Andrew E. Bruno wrote:
> >> What's the best way to re-initialize a replica? 
> >>
> >> Suppose one of your replicas goes south.. is there a command to tell
> >> that replicate to re-initialize from the first master (instead of
> >> removing/re-adding the replica from the topology)?
> > 
> > Found the command I was looking for:
> >    ipa-replica-manage re-initialize --from xxx
> > 
> > However, one of our replicates is down and can't seem to re-initialize
> > it. Starting ipa fails (via systemctl restart ipa):
> > 
> > ipactl status
> > Directory Service: RUNNING
> > krb5kdc Service: STOPPED
> > kadmin Service: STOPPED
> > named Service: STOPPED
> > ipa_memcached Service: STOPPED
> > httpd Service: STOPPED
> > pki-tomcatd Service: STOPPED
> > ipa-otpd Service: STOPPED
> > ipa: INFO: The ipactl command was successful
> > 
> > 
> > Errors from the dirsrv show:
> > 
> > : GSSAPI Error: Unspecified GSS failure.  Minor code may provide more information (No Kerberos credentials available)) errno 0 (Success)
> > [02/Oct/2015:11:45:05 -0400] slapi_ldap_bind - Error: could not perform interactive bind for id [] authentication mechanism [GSSAPI]: error -2 (Local error)
> > [02/Oct/2015:11:50:05 -0400] set_krb5_creds - Could not get initial credentials for principal [ldap/server at realm] in keytab [FILE:/etc/dirsrv/ds.keytab]: -1765328228 (Cannot contact any KDC for requested realm)
> > [02/Oct/2015:11:50:05 -0400] slapd_ldap_sasl_interactive_bind - Error: could not perform interactive bind for id [] mech [GSSAPI]: LDAP error -2 (Local error) (SASL(-1): generic failure: GSSAPI Error: Unspecified GSS failure.  Minor code may provide more information (No Kerberos credentials available)) errno 0 (Success)
> > [02/Oct/2015:11:50:05 -0400] slapi_ldap_bind - Error: could not perform interactive bind for id [] authentication mechanism [GSSAPI]: error -2 (Local error)
> > 
> > 
> > Attempting to re-initialize fails:
> > 
> > ipa-replica-manage re-initialize --from master
> > Connection timed out.
> > 
> > 
> > I verified time is in sync and DNS forward/reverse resolution is working.
> > 
> > Any pointers on what else to try?
> > 
> > Thanks!
> > 
> > --Andrew
> 
> Given that your Kerberos server instance is down, I would start investigating
> Kerberos logs to see why.


So looks like the dirsrv service comes up but with GSS errors about kerb
credentials. However, the rest of the services including the krb5kdc
fail to come up. Errors from the kdc logs suggest DNS:

 LOOKING_UP_CLIENT: DNS/replica at REALM Server error

FreeIPA is configured to serve DNS and this replica resolves it's own
DNS in /etc/resolv.conf (127.0.0.1)

I tried pointing /etc/resolv.conf to another (good) replica and even
tried adjusting /etc/krb5.conf to point at another kdc to try and get a
ticket however it still tries to connect to the local kdc (which fails
to start).  

I'm inclined to re-install this replica and start fresh. I'm curious if
we can re-kickstart this host from a fresh os/freeipa install and run
the  ipa-replica-manage re-initialize --from master command. The replica
will have the same name.. is this possible? Would we need to backup the
/var/lib/ipa/replica-info-XXX.gpg file?


--Andrew