[Freeipa-users] pki-tomcatd fails to start
Florence Blanc-Renaud
flo at redhat.com
Fri Jan 6 16:23:53 UTC 2017
On 01/06/2017 04:47 PM, Jeff Goddard wrote:
> Sorry for the typo. here is the correct output:
> ldapsearch -h id-management-1.internal.emerlyn.com
> <http://id-management-1.internal.emerlyn.com>
> SASL/EXTERNAL authentication started
> ldap_sasl_interactive_bind_s: Unknown authentication method (-6)
> additional info: SASL(-4): no mechanism available:
>
>
>
>
> When I look at the certificates I get errors regarding a host service in
> the keytab. Here is the output:
>
> [root at id-management-1 ca]# getcert list
> Number of certificates and requests being tracked: 8.
> Request ID '20150116161829':
> status: MONITORING
> ca-error: Error setting up ccache for "host" service on client
> using default keytab: Keytab contains no suitable keys for
> host/id-management-1.internal.emerlyn.com at INTERNAL.EMERLYN.COM
> <mailto:id-management-1.internal.emerlyn.com at INTERNAL.EMERLYN.COM>.
> stuck: no
> key pair storage:
> type=NSSDB,location='/etc/dirsrv/slapd-INTERNAL-EMERLYN-COM',nickname='Server-Cert',token='NSS
> Certificate DB',pinfile='/etc/dirsrv/slapd-INTERNAL-EMERLYN-COM/pwdfile.txt'
> certificate:
> type=NSSDB,location='/etc/dirsrv/slapd-INTERNAL-EMERLYN-COM',nickname='Server-Cert',token='NSS
> Certificate DB'
> CA: IPA
> issuer: CN=Certificate Authority,O=INTERNAL.EMERLYN.COM
> <http://INTERNAL.EMERLYN.COM>
> subject: CN=id-management-1.internal.emerlyn.com
> <http://id-management-1.internal.emerlyn.com>,O=INTERNAL.EMERLYN.COM
> <http://INTERNAL.EMERLYN.COM>
> expires: 2017-01-16 16:18:29 UTC
> key usage:
> digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment
> eku: id-kp-serverAuth,id-kp-clientAuth
> pre-save command:
> post-save command: /usr/lib64/ipa/certmonger/restart_dirsrv
> INTERNAL-EMERLYN-COM
> track: yes
> auto-renew: yes
> Request ID '20150116162120':
> status: MONITORING
> ca-error: Error setting up ccache for "host" service on client
> using default keytab: Keytab contains no suitable keys for
> host/id-management-1.internal.emerlyn.com at INTERNAL.EMERLYN.COM
> <mailto:id-management-1.internal.emerlyn.com at INTERNAL.EMERLYN.COM>.
> stuck: no
> key pair storage:
> type=NSSDB,location='/etc/httpd/alias',nickname='Server-Cert',token='NSS
> Certificate DB',pinfile='/etc/httpd/alias/pwdfile.txt'
> certificate:
> type=NSSDB,location='/etc/httpd/alias',nickname='Server-Cert',token='NSS
> Certificate DB'
> CA: IPA
> issuer: CN=Certificate Authority,O=INTERNAL.EMERLYN.COM
> <http://INTERNAL.EMERLYN.COM>
> subject: CN=id-management-1.internal.emerlyn.com
> <http://id-management-1.internal.emerlyn.com>,O=INTERNAL.EMERLYN.COM
> <http://INTERNAL.EMERLYN.COM>
> expires: 2017-01-16 16:21:20 UTC
> key usage:
> digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment
> eku: id-kp-serverAuth,id-kp-clientAuth
> pre-save command:
> post-save command: /usr/lib64/ipa/certmonger/restart_httpd
> track: yes
> auto-renew: yes
> Request ID '20151217174142':
> status: CA_UNREACHABLE
> ca-error: Internal error
> stuck: no
> key pair storage:
> type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='auditSigningCert
> cert-pki-ca',token='NSS Certificate DB',pin set
> certificate:
> type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='auditSigningCert
> cert-pki-ca',token='NSS Certificate DB'
> CA: dogtag-ipa-ca-renew-agent
> issuer: CN=Certificate Authority,O=INTERNAL.EMERLYN.COM
> <http://INTERNAL.EMERLYN.COM>
> subject: CN=CA Audit,O=INTERNAL.EMERLYN.COM
> <http://INTERNAL.EMERLYN.COM>
> expires: 2017-01-05 16:18:01 UTC
> key usage: digitalSignature,nonRepudiation
> pre-save command: /usr/lib64/ipa/certmonger/stop_pkicad
> post-save command: /usr/lib64/ipa/certmonger/renew_ca_cert
> "auditSigningCert cert-pki-ca"
> track: yes
> auto-renew: yes
> Request ID '20151217174143':
> status: CA_UNREACHABLE
> ca-error: Internal error
> stuck: no
> key pair storage:
> type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='ocspSigningCert cert-pki-ca',token='NSS
> Certificate DB',pin set
> certificate:
> type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='ocspSigningCert cert-pki-ca',token='NSS
> Certificate DB'
> CA: dogtag-ipa-ca-renew-agent
> issuer: CN=Certificate Authority,O=INTERNAL.EMERLYN.COM
> <http://INTERNAL.EMERLYN.COM>
> subject: CN=OCSP Subsystem,O=INTERNAL.EMERLYN.COM
> <http://INTERNAL.EMERLYN.COM>
> expires: 2017-01-05 16:17:58 UTC
> key usage: digitalSignature,nonRepudiation,keyCertSign,cRLSign
> eku: id-kp-OCSPSigning
> pre-save command: /usr/lib64/ipa/certmonger/stop_pkicad
> post-save command: /usr/lib64/ipa/certmonger/renew_ca_cert
> "ocspSigningCert cert-pki-ca"
> track: yes
> auto-renew: yes
> Request ID '20151217174144':
> status: CA_UNREACHABLE
> ca-error: Internal error
> stuck: no
> key pair storage:
> type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='subsystemCert
> cert-pki-ca',token='NSS Certificate DB',pin set
> certificate:
> type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='subsystemCert
> cert-pki-ca',token='NSS Certificate DB'
> CA: dogtag-ipa-ca-renew-agent
> issuer: CN=Certificate Authority,O=INTERNAL.EMERLYN.COM
> <http://INTERNAL.EMERLYN.COM>
> subject: CN=CA Subsystem,O=INTERNAL.EMERLYN.COM
> <http://INTERNAL.EMERLYN.COM>
> expires: 2017-01-05 16:17:59 UTC
> key usage:
> digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment
> eku: id-kp-serverAuth,id-kp-clientAuth
> pre-save command: /usr/lib64/ipa/certmonger/stop_pkicad
> post-save command: /usr/lib64/ipa/certmonger/renew_ca_cert
> "subsystemCert cert-pki-ca"
> track: yes
> auto-renew: yes
> Request ID '20151217174145':
> status: CA_UNREACHABLE
> ca-error: Internal error
> stuck: no
> key pair storage:
> type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='caSigningCert
> cert-pki-ca',token='NSS Certificate DB',pin set
> certificate:
> type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='caSigningCert
> cert-pki-ca',token='NSS Certificate DB'
> CA: dogtag-ipa-ca-renew-agent
> issuer: CN=Certificate Authority,O=INTERNAL.EMERLYN.COM
> <http://INTERNAL.EMERLYN.COM>
> subject: CN=Certificate Authority,O=INTERNAL.EMERLYN.COM
> <http://INTERNAL.EMERLYN.COM>
> expires: 2035-01-16 16:17:57 UTC
> key usage: digitalSignature,nonRepudiation,keyCertSign,cRLSign
> pre-save command: /usr/lib64/ipa/certmonger/stop_pkicad
> post-save command: /usr/lib64/ipa/certmonger/renew_ca_cert
> "caSigningCert cert-pki-ca"
> track: yes
> auto-renew: yes
> Request ID '20151217174146':
> status: CA_UNREACHABLE
> ca-error: Internal error
> stuck: no
> key pair storage:
> type=NSSDB,location='/etc/httpd/alias',nickname='ipaCert',token='NSS
> Certificate DB',pinfile='/etc/httpd/alias/pwdfile.txt'
> certificate:
> type=NSSDB,location='/etc/httpd/alias',nickname='ipaCert',token='NSS
> Certificate DB'
> CA: dogtag-ipa-ca-renew-agent
> issuer: CN=Certificate Authority,O=INTERNAL.EMERLYN.COM
> <http://INTERNAL.EMERLYN.COM>
> subject: CN=IPA RA,O=INTERNAL.EMERLYN.COM
> <http://INTERNAL.EMERLYN.COM>
> expires: 2017-01-05 16:18:23 UTC
> key usage:
> digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment
> eku: id-kp-serverAuth,id-kp-clientAuth
> pre-save command: /usr/lib64/ipa/certmonger/renew_ra_cert_pre
> post-save command: /usr/lib64/ipa/certmonger/renew_ra_cert
> track: yes
> auto-renew: yes
> Request ID '20151217174147':
> status: CA_UNREACHABLE
> ca-error: Error 60 connecting to
> https://id-management-1.internal.emerlyn.com:8443/ca/agent/ca/profileReview:
> Peer certificate cannot be authenticated with given CA certificates.
> stuck: no
> key pair storage:
> type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='Server-Cert
> cert-pki-ca',token='NSS Certificate DB',pin set
> certificate:
> type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='Server-Cert
> cert-pki-ca',token='NSS Certificate DB'
> CA: dogtag-ipa-renew-agent
> issuer: CN=Certificate Authority,O=INTERNAL.EMERLYN.COM
> <http://INTERNAL.EMERLYN.COM>
> subject: CN=id-management-1.internal.emerlyn.com
> <http://id-management-1.internal.emerlyn.com>,O=INTERNAL.EMERLYN.COM
> <http://INTERNAL.EMERLYN.COM>
> expires: 2017-01-05 16:17:59 UTC
> key usage:
> digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment
> eku: id-kp-serverAuth
> pre-save command: /usr/lib64/ipa/certmonger/stop_pkicad
> post-save command: /usr/lib64/ipa/certmonger/renew_ca_cert
> "Server-Cert cert-pki-ca"
> track: yes
> auto-renew: yes
>
> Looking at the content of /etc/krb5.keytab results in no host entry found:
>
> ktutil
> ktutil: read_kt /etc/krb5.keytab
> ktutil: list
> slot KVNO Principal
> ---- ----
> ---------------------------------------------------------------------
> 1 1 cifs/shares-01.internal.emerlyn.com at INTERNAL.EMERLYN.COM
> <mailto:shares-01.internal.emerlyn.com at INTERNAL.EMERLYN.COM>
> 2 1 cifs/shares-01.internal.emerlyn.com at INTERNAL.EMERLYN.COM
> <mailto:shares-01.internal.emerlyn.com at INTERNAL.EMERLYN.COM>
> 3 1 cifs/shares-01.internal.emerlyn.com at INTERNAL.EMERLYN.COM
> <mailto:shares-01.internal.emerlyn.com at INTERNAL.EMERLYN.COM>
> 4 1 cifs/shares-01.internal.emerlyn.com at INTERNAL.EMERLYN.COM
> <mailto:shares-01.internal.emerlyn.com at INTERNAL.EMERLYN.COM>
> 5 1 cifs/files-01.internal.emerlyn.com at INTERNAL.EMERLYN.COM
> <mailto:files-01.internal.emerlyn.com at INTERNAL.EMERLYN.COM>
> 6 1 cifs/files-01.internal.emerlyn.com at INTERNAL.EMERLYN.COM
> <mailto:files-01.internal.emerlyn.com at INTERNAL.EMERLYN.COM>
> 7 1 cifs/files-01.internal.emerlyn.com at INTERNAL.EMERLYN.COM
> <mailto:files-01.internal.emerlyn.com at INTERNAL.EMERLYN.COM>
> 8 1 cifs/files-01.internal.emerlyn.com at INTERNAL.EMERLYN.COM
> <mailto:files-01.internal.emerlyn.com at INTERNAL.EMERLYN.COM>
> 9 2 host/files-01.internal.emerlyn.com at INTERNAL.EMERLYN.COM
> <mailto:files-01.internal.emerlyn.com at INTERNAL.EMERLYN.COM>
> 10 2 host/files-01.internal.emerlyn.com at INTERNAL.EMERLYN.COM
> <mailto:files-01.internal.emerlyn.com at INTERNAL.EMERLYN.COM>
> 11 2 host/files-01.internal.emerlyn.com at INTERNAL.EMERLYN.COM
> <mailto:files-01.internal.emerlyn.com at INTERNAL.EMERLYN.COM>
> 12 2 host/files-01.internal.emerlyn.com at INTERNAL.EMERLYN.COM
> <mailto:files-01.internal.emerlyn.com at INTERNAL.EMERLYN.COM>
>
>
> Trying to add a host entry:
> kadmin -q "ktadd -k /etc/krb5.keytab
> host/id-management-1.internal.emerlyn.com
> <http://id-management-1.internal.emerlyn.com>"
> Authenticating as principal admin/admin at INTERNAL.EMERLYN.COM
> <mailto:admin at INTERNAL.EMERLYN.COM> with password.
> kadmin: Client 'admin/admin at INTERNAL.EMERLYN.COM
> <mailto:admin at INTERNAL.EMERLYN.COM>' not found in Kerberos database
> while initializing kadmin interface
>
> Yet if I issue kinit admin I get a password prompt and appear to get a
> ticket. What am I missing?
>
>
>
>
>
> On Fri, Jan 6, 2017 at 10:19 AM, Rob Crittenden <rcritten at redhat.com
> <mailto:rcritten at redhat.com>> wrote:
>
> Jeff Goddard wrote:
> > My environment is freeipa 4.4; centos 7.3. This system was upgraded as
> > of yesterday afternoon. I'm unable to start pki-tomcat. The debug log
> > show this entry:
> >
> > Internal Database Error encountered: Could not connect to LDAP server
> > host id-management-1.internal.emerlyn.com
> <http://id-management-1.internal.emerlyn.com>
> > <http://id-management-1.internal.emerlyn.com
> <http://id-management-1.internal.emerlyn.com>> port 636 Error
> > netscape.ldap.LDAPException: Authentication failed (48)
> > at
> com.netscape.cmscore.dbs.DBSubsystem.init(DBSubsystem.java:676)
> > at
> > com.netscape.cmscore.apps.CMSEngine.initSubsystem(CMSEngine.java:1169)
> > at
> >
> com.netscape.cmscore.apps.CMSEngine.initSubsystems(CMSEngine.java:1075)
> > at
> com.netscape.cmscore.apps.CMSEngine.init(CMSEngine.java:571)
> > at com.netscape.certsrv.apps.CMS.init(CMS.java:187)
> > at com.netscape.certsrv.apps.CMS.start(CMS.java:1616)
> > at
> >
> com.netscape.cms.servlet.base.CMSStartServlet.init(CMSStartServlet.java:114)
> > at javax.servlet.GenericServlet.init(GenericServlet.java:158)
> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> > at
> >
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> > at
> >
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> > at java.lang.reflect.Method.invoke(Method.java:498)
> > at
> > org.apache.catalina.security.SecurityUtil$1.run(SecurityUtil.java:288)
> > at
> > org.apache.catalina.security.SecurityUtil$1.run(SecurityUtil.java:285)
> > at java.security.AccessController.doPrivileged(Native Method)
> > at
> javax.security.auth.Subject.doAsPrivileged(Subject.java:549)
> > at
> >
> org.apache.catalina.security.SecurityUtil.execute(SecurityUtil.java:320)
> > at
> >
> org.apache.catalina.security.SecurityUtil.doAsPrivilege(SecurityUtil.java:175)
> > at
> >
> org.apache.catalina.security.SecurityUtil.doAsPrivilege(SecurityUtil.java:124)
> > at
> >
> org.apache.catalina.core.StandardWrapper.initServlet(StandardWrapper.java:1270)
> > at
> >
> org.apache.catalina.core.StandardWrapper.loadServlet(StandardWrapper.java:1195)
> > at
> >
> org.apache.catalina.core.StandardWrapper.load(StandardWrapper.java:1085)
> > at
> >
> org.apache.catalina.core.StandardContext.loadOnStartup(StandardContext.java:5318)
> > at
> >
> org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5610)
> > at
> > org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:147)
> > at
> >
> org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:899)
> > at
> >
> org.apache.catalina.core.ContainerBase.access$000(ContainerBase.java:133)
> > at
> >
> org.apache.catalina.core.ContainerBase$PrivilegedAddChild.run(ContainerBase.java:156)
> > at
> >
> org.apache.catalina.core.ContainerBase$PrivilegedAddChild.run(ContainerBase.java:145)
> > at java.security.AccessController.doPrivileged(Native Method)
> > at
> >
> org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:873)
> > at
> > org.apache.catalina.core.StandardHost.addChild(StandardHost.java:652)
> > at
> >
> org.apache.catalina.startup.HostConfig.deployDescriptor(HostConfig.java:679)
> > at
> >
> org.apache.catalina.startup.HostConfig$DeployDescriptor.run(HostConfig.java:1966)
> > at
> >
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> > at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> > at
> >
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> > at
> >
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> > at java.lang.Thread.run(Thread.java:745)
> >
> >
> > I'm able to get a kerberos ticket using kinit but ldap search
> gives this
> > error:
> >
> > ldapsearch -h id-manaement-1.internal.emerlyn.com
> <http://id-manaement-1.internal.emerlyn.com>
> > <http://id-manaement-1.internal.emerlyn.com
> <http://id-manaement-1.internal.emerlyn.com>> -x -b
> > "cn=CAcert,cn=ipa,cn=etc,dc=internal,dc=emerlyn,dc=com"
> > ldap_sasl_bind(SIMPLE): Can't contact LDAP server (-1)
> >
> > adding the -d1 debugging tag results in:
> >
> > ldap_create
> > ldap_url_parse_ext(ldap://id-manaement-1.internal.emerlyn.com
> <http://id-manaement-1.internal.emerlyn.com>
> > <http://id-manaement-1.internal.emerlyn.com
> <http://id-manaement-1.internal.emerlyn.com>>)
> > ldap_sasl_bind
> > ldap_send_initial_request
> > ldap_new_connection 1 1 0
> > ldap_int_open_connection
> > ldap_connect_to_host: TCP id-manaement-1.internal.emerlyn.com:389
> <http://id-manaement-1.internal.emerlyn.com:389>
> > <http://id-manaement-1.internal.emerlyn.com:389
> <http://id-manaement-1.internal.emerlyn.com:389>>
> > ldap_connect_to_host: getaddrinfo failed: Name or service not known
> > ldap_err2string
> > ldap_sasl_bind(SIMPLE): Can't contact LDAP server (-1)
> >
> > I'm able to resolve the hostname via nslookup and /etc/hosts has the
> > correct mapping entry.
> >
> > I'm kind of lost at this point and could use some help.
> >
> > Thanks in advance.
>
> You have a typo in the hostname you're trying to connect to, missing the
> 'g' in management.
>
> I have a vague memory from other reports of this issue that the problem
> may be that the value of the certificate(s) in CS.cfg is different from
> the dogtag NSS database. I'd see if those line up.
>
> rob
>
>
>
>
> --
> Jeff
>
>
>
Hi Jeff,
according to the output of getcert list, many certificates expired just
yesterday (auditSigningCert cert-pki-ca, ocspSigningCert cert-pki-ca,
subsystemCert cert-pki-ca, Server-Cert cert-pki-ca in the PKI NSS DB and
ipaCert in /etc/httpd/alias).
You can refer to this page:
https://access.redhat.com/solutions/643753
to fix the issue.
It is likely that dogtag cannot authenticate to LDAP because its
certificate is expired, and hence refuses to start. IMHO the upgrade is
just an unlucky coincidence (happening the same day as cert expiration)
but not the root cause.
HTH,
Flo.
More information about the Freeipa-users
mailing list