[Freeipa-users] ipa-server-upgrade fails and CA cannot start

[Andrew C. Dingman]

On Tue, 2016-05-10 at 10:16 +0200, Petr Vobornik wrote:
> On 05/08/2016 09:49 PM, Andrew C. Dingman wrote:
> > 
> > "getcert list" successfully shows 8 certificate requests being
> > tracked.
> > Four are in "MONITORING" status, four in "NEED_CA". The NEED_CA
> > requests all indicate expiration back in February, and look like
> > crucial certificates: CN=CA Subsystem, CN=IPA RA, CN=CA Audit
> > and CN=OCSP Subsystem.
> > 
> > On the working replica, all eight are in "MONITORING" status and
> > have
> > expiration dates in 2017 or later. I have not attempted the package
> > update on that system. Should I consider promoting this one to CA
> > master, force-deleting the old one, and reinstalling it as a new
> > system?
> > 
> > Please let me know what other information would be helpful for
> > diagnostics. The current state of all packages on the broken master
> > is
> > up to earlier today from the official Red Hat content distribution
> > network.
> > 
> Hello Andrew,
> 
> Could you paste output of `ipactl start` ?

[andrew at ipa2 ~]$ sudo ipactl start
Starting Directory Service
Starting krb5kdc Service
Starting kadmin Service
Starting named Service
Starting ipa_memcached Service
Starting httpd Service
Starting pki-tomcatd Service
Failed to start pki-tomcatd Service
Shutting down
Aborting ipactl
[andrew at ipa2 ~]$

There's a pause of several minutes between "Starting pki-tomcatd
Service" and "Failed". Full output from "sudo ipactl -d start" is at  h
ttps://paste.fedoraproject.org/364876/14629214/ but it mostly consists
of:

    ipa: DEBUG: stderr=
    ipa: DEBUG: wait_for_open_ports: localhost [8080, 8443] timeout 300
    ipa: DEBUG: Waiting until the CA is running
    ipa: DEBUG: Starting external process
    ipa: DEBUG: args='/usr/bin/wget' '-S' '-O' '-' '--timeout=30' '--no-
    check-certificate' 'https://ipa2.acdingman.com:8443/ca/admin/ca/getS
    tatus'
    ipa: DEBUG: Process finished, return code=8
    ipa: DEBUG: stdout=
    ipa: DEBUG: stderr=--2016-05-10 18:53:33--  https://ipa2.acdingman.c
    om:8443/ca/admin/ca/getStatus
    Resolving ipa2.acdingman.com (ipa2.acdingman.com)...
    2001:19f0:300:2a63::64, 104.156.251.79
    Connecting to ipa2.acdingman.com
    (ipa2.acdingman.com)|2001:19f0:300:2a63::64|:8443... connected.
    HTTP request sent, awaiting response... 
      HTTP/1.1 500 Internal Server Error
      Server: Apache-Coyote/1.1
      Content-Type: text/html;charset=utf-8
      Content-Language: en
      Content-Length: 2134
      Date: Tue, 10 May 2016 22:53:55 GMT
      Connection: close
    2016-05-10 18:53:55 ERROR 500: Internal Server Error.

repeated once a second for nearly five minutes.

> Also when upgrader fails it tends to leave directory server not
> accessible by changing 389 and 636 port.
> 
> It could be verified by:
> 
> $ ldapsearch -ZZ -h `hostname` -D "cn=Directory Manager" -W -s base
> -b
> "cn=config" | grep "nsslapd-security\|nsslapd-port"
> Enter LDAP Password:
> nsslapd-requiresrestart: cn=config:nsslapd-port
> nsslapd-port: 389
> nsslapd-security: on
> 
> If there are values other than '389' and 'on' (usually '0' and 'off')
> then it might the reason why IPA doesn't start. Changing them back to
> 'on' and 389 might help.

Nope, my output looks just like your sample.

> But it won't say why the upgrader failed. Maybe it was a one-time
> glitch
> or it was related to the expired certs.
> 
> The error message you got is in code which creates connection to
> certmonger.
> 
> But if there are expired certificates. The usual recovery is to move
> back time a day or two before the first certificate expires and let
> certmonger to renew the certs. Optionally the renewal can be forced
> by
> `getcert resubmit -i $certid` command.

Do I risk hurting the functional replica if I do that? I presume with
time months off from each other they wouldn't even talk until I got the
time correct on the broken system, but that's based on the assumption
that they mostly use GSSAPI authentication. If anything is certificate-
based the time tolerances could be much larger.

-Andrew