[Freeipa-users] IPA 4.2: pki-tomcatd in terrible shape

Rob Crittenden rcritten at redhat.com
Fri Feb 5 08:00:44 UTC 2016 

Timothy Geier wrote:
> Greetings all,
>
> For the record,this is a CentOS 7.2 box with all current patches. (ipa-server-4.2.0-15.el7.centos.3.x86_64, etc.)
>
> The situation is that pki-tomcatd on the lone CA server in our IPA cluster refuses to start cleanly.  The issues started earlier this week after the certs
> subsystemCert, ocspSigningCert, and auditSigningCert all simultaneously expired without warning; apparently, certmonger failed to renew them automatically.  We
> attempted timeshifting and following instructions for what appeared to be similar issues, but nothing at all has worked.
>
> Today, we attempted removing the certificates in question (of course, the files in /etc/pki/pki-tomcat/alias were backed up beforehand) and using certutil to issue new  certificates.   This process worked but pki-tomcatd is still refusing to start.  We can get IPA to run on this server by manually starting pki-tomcatd, running ipactl start, and then ctrl-c’ing it when it gets to "Starting pki-tomcatd" but this is not a tenable long-term solution.
>
> Relevant log entries/information:
>
> /var/log/pki/pki-tomcat/ca/debug:
> Could not connect to LDAP server host ipa01.XXXXXXXXX.net port 636 Error netscape.ldap.LDAPException: IO Error creating JSS SSL Socket (-1)
> Internal Database Error encountered: Could not connect to LDAP server host ipa01.XXXXXXXXX.net port 636 Error netscape.ldap.LDAPException: IO Error creating JSS SSL Socket (-1)
> Internal Database Error encountered: Could not connect to LDAP server host ipa01.XXXXXXXXX.net port 636 Error netscape.ldap.LDAPException: Authentication failed (49)
>
> /var/log/pki/pki-tomcat/localhost.2016-02-04.log:
> org.apache.catalina.core.StandardContext loadOnStartup
> SEVERE: Servlet /ca threw load() exception
> java.lang.NullPointerException
>
> # getcert list:
>
> Number of certificates and requests being tracked: 8.
> Request ID '20151015022737':
> 	status: MONITORING
> 	ca-error: Error setting up ccache for "host" service on client using default keytab: Generic error (see e-text).
> 	stuck: no
> 	key pair storage: type=NSSDB,location='/etc/dirsrv/slapd-XXXXXXXXX-NET',nickname='Server-Cert',token='NSS Certificate DB',pinfile='/etc/dirsrv/slapd-XXXXXXXXX-NET/pwdfile.txt'
> 	expires: 2017-10-15 02:09:06 UTC
> 	track: yes
> 	auto-renew: yes
> Request ID '20151015022949':
> 	status: MONITORING
> 	ca-error: Error setting up ccache for "host" service on client using default keytab: Generic error (see e-text).
> 	stuck: no
> 	key pair storage: type=NSSDB,location='/etc/httpd/alias',nickname='Server-Cert',token='NSS Certificate DB',pinfile='/etc/httpd/alias/pwdfile.txt'
> 	certificate: type=NSSDB,location='/etc/httpd/alias',nickname='Server-Cert',token='NSS Certificate DB'
> 	expires: 2017-10-15 02:09:10 UTC
> 	track: yes
> 	auto-renew: yes
> Request ID '20160127202548':
> 	status: MONITORING
> 	stuck: no
> 	key pair storage: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='caSigningCert cert-pki-ca',token='NSS Certificate DB',pin set
> 	certificate: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='caSigningCert cert-pki-ca',token='NSS Certificate DB'
> 	expires: 2034-02-11 19:46:43 UTC
> 	track: yes
> 	auto-renew: yes
> Request ID '20160127202549':
> 	status: MONITORING
> 	stuck: no
> 	key pair storage: type=NSSDB,location='/etc/httpd/alias',nickname='ipaCert',token='NSS Certificate DB',pinfile='/etc/httpd/alias/pwdfile.txt'
> 	certificate: type=NSSDB,location='/etc/httpd/alias',nickname='ipaCert',token='NSS Certificate DB'
> 	expires: 2017-12-25 04:27:49 UTC
> 	key usage: digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment
> 	track: yes
> 	auto-renew: yes
> Request ID '20160127202550':
> 	status: MONITORING
> 	ca-error: Server at "http://ipa01.XXXXXXXXX.net:8080/ca/ee/ca/profileSubmit" replied: Profile caServerCert Not Found
> 	stuck: no
> 	key pair storage: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='Server-Cert cert-pki-ca',token='NSS Certificate DB',pin set
> 	certificate: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='Server-Cert cert-pki-ca',token='NSS Certificate DB'
> 	expires: 2017-10-04 02:28:53 UTC
> 	track: yes
> 	auto-renew: yes
> Request ID '20160204165453':
> 	status: MONITORING
> 	stuck: no
> 	key pair storage: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='auditSigningCert cert-pki-ca',token='NSS Certificate DB',pin set
> 	certificate: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='auditSigningCert cert-pki-ca',token='NSS Certificate DB'
> 	expires: 2016-05-04 16:40:23 UTC
> 	track: yes
> 	auto-renew: yes
> Request ID '20160204170246':
> 	status: MONITORING
> 	stuck: no
> 	key pair storage: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='ocspSigningCert cert-pki-ca',token='NSS Certificate DB',pin set
> 	certificate: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='ocspSigningCert cert-pki-ca',token='NSS Certificate DB'
> 	expires: 2016-05-04 16:59:18 UTC
> 	track: yes
> 	auto-renew: yes
> Request ID '20160204170752':
> 	status: MONITORING
> 	stuck: no
> 	key pair storage: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='subsystemCert cert-pki-ca',token='NSS Certificate DB',pin set
> 	certificate: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='subsystemCert cert-pki-ca',token='NSS Certificate DB'
> 	expires: 2016-05-04 17:05:29 UTC
> 	track: yes
> 	auto-renew: yes
>
> # certutil -L -d /var/lib/pki/pki-tomcat/alias/
>
> Certificate Nickname                                         Trust Attributes
>                                                               SSL,S/MIME,JAR/XPI
> auditSigningCert cert-pki-ca                                 u,u,Pu
> ocspSigningCert cert-pki-ca                                  u,u,u
> caSigningCert cert-pki-ca                                    CTu,Cu,Cu
> subsystemCert cert-pki-ca                                    u,u,u
> Server-Cert cert-pki-ca                                      u,u,u
>
> # certutil -L -d /etc/dirsrv/slapd-XXXXXXXXX-NET/
>
> Certificate Nickname                                         Trust Attributes
>                                                               SSL,S/MIME,JAR/XPI
> Server-Cert                                                                 u,u,u
> XXXXXXXXX.NET IPA CA                                         CT,C,C
>
>
>
> The only thing that making new certs seemed to resolve was removing these errors from /var/log/pki/pki-tomcat/ca/system :
>
> Cannot authenticate agent with certificate Serial <redacted> Subject DN CN=IPA RA,O=XXXXXXXXX.NET. Error: User not found
>
> Thus, the root cause(s) appears to be something else entirely that we are totally unfamilar with..we can provide any other required information to help with troubleshooting.

You can't manually re-issue the CA certificates using certutil.

Your best bet is to restore the original database and try going back in 
time again and we can start troubleshooting from that point. And it is 
likely to fail again if you've changed the way the certificates are 
tracked by certmonger.

There are some notable things missing from your certmonger output 
including the CA and the pre and post command scripts. We would 
definitely need to see those in order to troubleshoot.

Is this your originally installed CA or did that die at some point and 
this is a replica of it?

rob

More information about the Freeipa-users mailing list