[Freeipa-users] pki-tomcat failure
Petr Vobornik
pvoborni at redhat.com
Wed Jan 11 13:55:06 UTC 2017
On 01/10/2017 09:31 PM, Bob Hinton wrote:
> Hi,
>
> The pki-tomcatd services on our IPA servers seem to have stopped working.
>
> This seems to be related to the expiry of several certificates -
>
> [root at ipa001 ~]# getcert list | more
> Number of certificates and requests being tracked: 8.
> Request ID '20161230150048':
> status: MONITORING
> stuck: no
> key pair storage:
> type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='auditSigningCert
> cert-pki-ca',token='NSS Certificate DB',pin set
> certificate:
> type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='auditSigningCert
> cert-pki-ca',token='NSS Certificate DB'
> CA: dogtag-ipa-ca-renew-agent
> issuer: CN=Certificate Authority,O=LOCAL.COM
> subject: CN=CA Audit,O=LOCAL.COM
> expires: 2017-01-09 08:21:45 UTC
> key usage: digitalSignature,nonRepudiation
> pre-save command: /usr/libexec/ipa/certmonger/stop_pkicad
> post-save command: /usr/libexec/ipa/certmonger/renew_ca_cert
> "auditSigningCert cert-pki-ca"
> track: yes
> auto-renew: yes
> Request ID '20161230150049':
> status: MONITORING
> stuck: no
> key pair storage:
> type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='ocspSigningCert
> cert-pki-ca',token='NSS Certificate DB',pin set
> certificate:
> type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='ocspSigningCert
> cert-pki-ca',token='NSS Certificate DB'
> CA: dogtag-ipa-ca-renew-agent
> issuer: CN=Certificate Authority,O=LOCAL.COM
> subject: CN=OCSP Subsystem,O=LOCAL.COM
> expires: 2017-01-09 08:21:45 UTC
> key usage: digitalSignature,nonRepudiation,keyCertSign,cRLSign
> eku: id-kp-OCSPSigning
> pre-save command: /usr/libexec/ipa/certmonger/stop_pkicad
> post-save command: /usr/libexec/ipa/certmonger/renew_ca_cert
> "ocspSigningCert cert-pki-ca"
> track: yes
> auto-renew: yes
>
> These were originally in CA_WORKING state, but I moved the clock back
> and restarted certmonger to try to renew them.
Certs above have:
expires: 2017-01-09 08:21:45 UTC
But log has 10/Jan so the log is from the time when certs are expired.
Move time back when all certs reported by `getcert list` are valid.
Restart IPA. Resubmit all certs which are about to expire. Move time back.
>
>
> /var/log/pki/pki-tomcat/ca/debug contains
>
> [10/Jan/2017:18:35:37][localhost-startStop-1]: makeConnection:
> errorIfDown true
> [10/Jan/2017:18:35:37][localhost-startStop-1]:
> SSLClientCertificateSelectionCB: Setting desired cert nickname to:
> subsystemCert cert-pki-ca
> [10/Jan/2017:18:35:37][localhost-startStop-1]: LdapJssSSLSocket: set
> client auth cert nickname subsystemCert cert-pki-ca
> [10/Jan/2017:18:35:37][localhost-startStop-1]:
> SSLClientCertificatSelectionCB: Entering!
> [10/Jan/2017:18:35:37][localhost-startStop-1]: Candidate cert:
> caSigningCert cert-pki-ca
> [10/Jan/2017:18:35:37][localhost-startStop-1]: Candidate cert:
> Server-Cert cert-pki-ca
> [10/Jan/2017:18:35:37][localhost-startStop-1]:
> SSLClientCertificateSelectionCB: returning: null
> [10/Jan/2017:18:35:37][localhost-startStop-1]: SSL handshake happened
> Could not connect to LDAP server host ipa001.mgmt.local.com port 636
> Error netscape.ldap.LDAPException: Authentication failed (48)
> at
> com.netscape.cmscore.ldapconn.LdapBoundConnFactory.makeConnection(LdapBoundConnFactory.java:205)
> at
> com.netscape.cmscore.ldapconn.LdapBoundConnFactory.init(LdapBoundConnFactory.java:166)
> at
> com.netscape.cmscore.ldapconn.LdapBoundConnFactory.init(LdapBoundConnFactory.java:130)
> at com.netscape.cmscore.dbs.DBSubsystem.init(DBSubsystem.java:654)
> at
> com.netscape.cmscore.apps.CMSEngine.initSubsystem(CMSEngine.java:1169)
> at
> com.netscape.cmscore.apps.CMSEngine.initSubsystems(CMSEngine.java:1075)
> at com.netscape.cmscore.apps.CMSEngine.init(CMSEngine.java:571)
> at com.netscape.certsrv.apps.CMS.init(CMS.java:187)
> at com.netscape.certsrv.apps.CMS.start(CMS.java:1616)
> at
> com.netscape.cms.servlet.base.CMSStartServlet.init(CMSStartServlet.java:114)
> at javax.servlet.GenericServlet.init(GenericServlet.java:158)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.apache.catalina.security.SecurityUtil$1.run(SecurityUtil.java:288)
> at
> org.apache.catalina.security.SecurityUtil$1.run(SecurityUtil.java:285)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAsPrivileged(Subject.java:549)
> at
> org.apache.catalina.security.SecurityUtil.execute(SecurityUtil.java:320)
> at
> org.apache.catalina.security.SecurityUtil.doAsPrivilege(SecurityUtil.java:175)
> at
> org.apache.catalina.security.SecurityUtil.doAsPrivilege(SecurityUtil.java:124)
> at
> org.apache.catalina.core.StandardWrapper.initServlet(StandardWrapper.java:1270)
> at
> org.apache.catalina.core.StandardWrapper.loadServlet(StandardWrapper.java:1195)
> at
> org.apache.catalina.core.StandardWrapper.load(StandardWrapper.java:1085)
> at
> org.apache.catalina.core.StandardContext.loadOnStartup(StandardContext.java:5318)
> at
> org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5610)
> at
> org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:147)
> at
> org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:899)
> at
> org.apache.catalina.core.ContainerBase.access$000(ContainerBase.java:133)
> at
> org.apache.catalina.core.ContainerBase$PrivilegedAddChild.run(ContainerBase.java:156)
> at
> org.apache.catalina.core.ContainerBase$PrivilegedAddChild.run(ContainerBase.java:145)
> at java.security.AccessController.doPrivileged(Native Method)
> at
> org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:873)
> at
> org.apache.catalina.core.StandardHost.addChild(StandardHost.java:652)
> at
> org.apache.catalina.startup.HostConfig.deployDescriptor(HostConfig.java:679)
> at
> org.apache.catalina.startup.HostConfig$DeployDescriptor.run(HostConfig.java:1966)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> Internal Database Error encountered: Could not connect to LDAP server
> host ipa001.mgmt.local.com port 636 Error netscape.ldap.LDAPException:
> Authentication failed (48)
> at com.netscape.cmscore.dbs.DBSubsystem.init(DBSubsystem.java:676)
>
> The only connection attempt I can find relating to err=48 in the slapd
> access log is -
>
>
> [10/Jan/2017:18:21:08.884446519 +0000] conn=59668 fd=83 slot=83 SSL
> connection from 10.220.6.250 to 10.220.6.250
> [10/Jan/2017:18:21:08.898844561 +0000] conn=59668 TLS1.2 256-bit AES
> [10/Jan/2017:18:21:08.917314723 +0000] conn=59668 op=0 BIND dn=""
> method=sasl version=3 mech=EXTERNAL
> [10/Jan/2017:18:21:08.919725280 +0000] conn=59668 op=0 RESULT err=48
> tag=97 nentries=0 etime=0
> [10/Jan/2017:18:21:09.590236408 +0000] conn=59637 op=88 EXT
> oid="2.16.840.1.113730.3.5.12" name="replication-multimaster-extop"
>
> We recent upgraded ipa from 4.2 to 4.4 and I wonder if that broke something.
>
> ipa --version
> VERSION: 4.4.0, API_VERSION: 2.213
>
> The /etc/ca.crt cert was originally created on an ipa 3.3 server that no
> longer exists, I don't know if that's relevant.
>
> Anyway, I'm stumped on how to fix this so could anyone please help.
>
> Many thanks
>
> Bob
>
--
Petr Vobornik
More information about the Freeipa-users
mailing list