<html><head></head><body><div style="color:#000; background-color:#fff; font-family:bookman old style, new york, times, serif;font-size:13px"><div dir="ltr" id="yui_3_16_0_ym19_1_1469807164755_3130">I have seen many threads on this so sorry to bring it up again but I have a freeipa domain, with 4 ipa servers running on redhat 6 version 3.0.0-50. The certificates are expired/expiring and will not renew and it is causing many issues for us. I have tried the many suggestions I have see in the archives such as changing the time to prior to expiration and attempting renew by resubmitting the requests but they never renew. An example of getcert list from the first server that expired:</div><div id="yui_3_16_0_ym19_1_1469807164755_3739"> </div><div dir="ltr" id="yui_3_16_0_ym19_1_1469807164755_3165">Number of certificates and requests being tracked: 8. Request ID '20140618161026': status: CA_UNREACHABLE ca-error: Server at https://ipa1.example.com/ipa/xml failed request, will retry: -504 (libcurl failed to execute the HTTP POST transaction. Peer certificate cannot be authenticated with known CA certificates). stuck: no key pair storage: type=NSSDB,location='/etc/dirsrv/slapd-PKI-IPA',nickname='Server-Cert',token='NSS Certificate DB',pinfile='/etc/dirsrv/slapd-PKI-IPA/pwdfile.txt' certificate: type=NSSDB,location='/etc/dirsrv/slapd-PKI-IPA',nickname='Server-Cert',token='NSS Certificate DB' CA: IPA issuer: CN=Certificate Authority,O=EXAMPLE.COM subject: CN=idm1-io.example.com,O=EXAMPLE.COM expires: 2016-06-18 00:09:05 UTC key usage: digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment eku: id-kp-serverAuth,id-kp-clientAuth pre-save command: post-save command: /usr/lib64/ipa/certmonger/restart_dirsrv PKI-IPA track: yes auto-renew: yes Request ID '20140618161126': status: MONITORING ca-error: Internal error: no response to "http://ipa1-io.example.com:9180/ca/ee/ca/profileSubmit?profileId=auditSigningCert+cert-pki-ca&serial_num=5&renewal=true&xml=true". stuck: no key pair storage: type=NSSDB,location='/var/lib/pki-ca/alias',nickname='auditSigningCert cert-pki-ca',token='NSS Certificate DB',pin set certificate: type=NSSDB,location='/var/lib/pki-ca/alias',nickname='auditSigningCert cert-pki-ca',token='NSS Certificate DB' CA: dogtag-ipa-renew-agent issuer: CN=Certificate Authority,O=EXAMPLE.COM subject: CN=CA Audit,O=EXAMPLE.COM expires: 2016-06-06 23:36:29 UTC key usage: digitalSignature,nonRepudiation pre-save command: /usr/lib64/ipa/certmonger/stop_pkicad post-save command: /usr/lib64/ipa/certmonger/restart_pkicad "auditSigningCert cert-pki-ca" track: yes auto-renew: yes Request ID '20140618161127': status: MONITORING ca-error: Internal error: no response to "http://ipa1.example.com:9180/ca/ee/ca/profileSubmit?profileId=ocspSigningCert+cert-pki-ca&serial_num=2&renewal=true&xml=true". stuck: no key pair storage: type=NSSDB,location='/var/lib/pki-ca/alias',nickname='ocspSigningCert cert-pki-ca',token='NSS Certificate DB',pin set certificate: type=NSSDB,location='/var/lib/pki-ca/alias',nickname='ocspSigningCert cert-pki-ca',token='NSS Certificate DB' CA: dogtag-ipa-renew-agent issuer: CN=Certificate Authority,O=EXAMPLE.COM subject: CN=OCSP Subsystem,O=EXAMPLE.COM expires: 2016-06-06 23:36:28 UTC key usage: digitalSignature,nonRepudiation,keyCertSign,cRLSign eku: id-kp-OCSPSigning pre-save command: /usr/lib64/ipa/certmonger/stop_pkicad post-save command: /usr/lib64/ipa/certmonger/restart_pkicad "ocspSigningCert cert-pki-ca" track: yes auto-renew: yes Request ID '20140618161128': status: MONITORING ca-error: Internal error: no response to "http://ipa1.example.com:9180/ca/ee/ca/profileSubmit?profileId=subsystemCert+cert-pki-ca&serial_num=4&renewal=true&xml=true". stuck: no key pair storage: type=NSSDB,location='/var/lib/pki-ca/alias',nickname='subsystemCert cert-pki-ca',token='NSS Certificate DB',pin set certificate: type=NSSDB,location='/var/lib/pki-ca/alias',nickname='subsystemCert cert-pki-ca',token='NSS Certificate DB' CA: dogtag-ipa-renew-agent issuer: CN=Certificate Authority,O=EXAMPLE.COM subject: CN=CA Subsystem,O=EXAMPLE.COM expires: 2016-06-06 23:36:28 UTC key usage: digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment eku: id-kp-serverAuth,id-kp-clientAuth pre-save command: /usr/lib64/ipa/certmonger/stop_pkicad post-save command: /usr/lib64/ipa/certmonger/restart_pkicad "subsystemCert cert-pki-ca" track: yes auto-renew: yes Request ID '20140618161129': status: MONITORING ca-error: Internal error: no response to "http://ipa1.example.com:9180/ca/ee/ca/profileSubmit?profileId=caServerCert&serial_num=268304385&renewal=true&xml=true". stuck: no key pair storage: type=NSSDB,location='/var/lib/pki-ca/alias',nickname='Server-Cert cert-pki-ca',token='NSS Certificate DB',pin set certificate: type=NSSDB,location='/var/lib/pki-ca/alias',nickname='Server-Cert cert-pki-ca',token='NSS Certificate DB' CA: dogtag-ipa-renew-agent issuer: CN=Certificate Authority,O=EXAMPLE.COM subject: CN=ipa1.example.com,O=EXAMPLE.COM expires: 2016-06-07 16:11:22 UTC key usage: digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment eku: id-kp-serverAuth pre-save command: post-save command: track: yes auto-renew: yes Request ID '20140618161217': status: NEED_CSR_GEN_TOKEN stuck: yes key pair storage: type=NSSDB,location='/etc/dirsrv/slapd-example-COM',nickname='Server-Cert',token='NSS Certificate DB',pinfile='/etc/dirsrv/slapd-example-COM/pwdfile.txt' certificate: type=NSSDB,location='/etc/dirsrv/slapd-example-COM',nickname='Server-Cert',token='NSS Certificate DB' CA: IPA issuer: CN=Certificate Authority,O=EXAMPLE.COM subject: CN=ipa1.example.com,O=EXAMPLE.COM expires: 2016-06-18 00:09:05 UTC key usage: digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment eku: id-kp-serverAuth,id-kp-clientAuth pre-save command: post-save command: /usr/lib64/ipa/certmonger/restart_dirsrv example-COM track: yes auto-renew: yes Request ID '20140618161317': status: CA_UNREACHABLE ca-error: Server at https://ipa1.example.com/ipa/xml failed request, will retry: -504 (libcurl failed to execute the HTTP POST transaction. Peer certificate cannot be authenticated with known CA certificates). stuck: no key pair storage: type=NSSDB,location='/etc/httpd/alias',nickname='Server-Cert',token='NSS Certificate DB',pinfile='/etc/httpd/alias/pwdfile.txt' certificate: type=NSSDB,location='/etc/httpd/alias',nickname='Server-Cert',token='NSS Certificate DB' CA: IPA issuer: CN=Certificate Authority,O=EXAMPLE.COM subject: CN=idm1-io.example.com,O=EXAMPLE.COM expires: 2016-06-18 00:09:06 UTC key usage: digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment eku: id-kp-serverAuth,id-kp-clientAuth pre-save command: post-save command: /usr/lib64/ipa/certmonger/restart_httpd track: yes auto-renew: yes Request ID '20140618161338': status: MONITORING ca-error: Internal error: no response to "http://ipa1.example.com:9180/ca/ee/ca/profileSubmit?profileId=ipaCert&serial_num=7&renewal=true&xml=true". stuck: no key pair storage: type=NSSDB,location='/etc/httpd/alias',nickname='ipaCert',token='NSS Certificate DB',pinfile='/etc/httpd/alias/pwdfile.txt' certificate: type=NSSDB,location='/etc/httpd/alias',nickname='ipaCert',token='NSS Certificate DB' CA: dogtag-ipa-renew-agent issuer: CN=Certificate Authority,O=EXAMPLE.COM subject: CN=IPA RA,O=EXAMPLE.COM expires: 2016-06-06 23:37:09 UTC key usage: digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment eku: id-kp-serverAuth,id-kp-clientAuth pre-save command: post-save command: /usr/lib64/ipa/certmonger/restart_httpd track: yes auto-renew: yes</div><div id="yui_3_16_0_ym19_1_1469807164755_3601" dir="ltr"> </div><div id="yui_3_16_0_ym19_1_1469807164755_3609" dir="ltr">localhost log in /var/log/pki-ca have errors like:</div><div id="yui_3_16_0_ym19_1_1469807164755_3640" dir="ltr">tail localhost.2016-07-29.log Jul 29, 2016 8:55:51 AM org.apache.catalina.core.StandardWrapperValve invoke SEVERE: Servlet.service() for servlet caProfileSubmit threw exception java.io.IOException: CS server is not ready to serve. at com.netscape.cms.servlet.base.CMSServlet.service(CMSServlet.java:441) at javax.servlet.http.HttpServlet.service(HttpServlet.java:723) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206) at com.netscape.cms.servlet.filter.EERequestFilter.doFilter(EERequestFilter.java:176) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235) at org.</div><div id="yui_3_16_0_ym19_1_1469807164755_3660" dir="ltr"> </div><div id="yui_3_16_0_ym19_1_1469807164755_3692" dir="ltr">Debug log in /var/log/pki-cacd </div><div id="yui_3_16_0_ym19_1_1469807164755_3691" dir="ltr"> tail debug [29/Jul/2016:08:49:08][Timer-0]: CMSEngine: getPasswordStore(): password store initialized before. [29/Jul/2016:08:49:08][Timer-0]: CMSEngine: getPasswordStore(): password store initialized. [29/Jul/2016:08:49:08][Timer-0]: SecurityDomainSessionTable getLDAPConn: netscape.ldap.LDAPException: error result (49) [29/Jul/2016:08:49:08][Timer-0]: SecurityDomainSessionTable: unable to query sessionIds: java.io.IOException: Failed to connect to the internal database. [29/Jul/2016:08:49:08][Timer-0]: SecurityDomainSessionTable: getSessionIds: Error in disconnecting from database: java.lang.NullPointerException [29/Jul/2016:08:54:08][Timer-0]: CMSEngine: getPasswordStore(): password store initialized before. [29/Jul/2016:08:54:08][Timer-0]: CMSEngine: getPasswordStore(): password store initialized. [29/Jul/2016:08:54:08][Timer-0]: SecurityDomainSessionTable getLDAPConn: netscape.ldap.LDAPException: error result (49) [29/Jul/2016:08:54:08][Timer-0]: SecurityDomainSessionTable: unable to query sessionIds: java.io.IOException: Failed to connect to the internal database. [29/Jul/2016:08:54:08][Timer-0]: SecurityDomainSessionTable: getSessionIds: Error in disconnecting from database: java.lang.NullPointerException</div><div id="yui_3_16_0_ym19_1_1469807164755_3749" dir="ltr"> </div><div id="yui_3_16_0_ym19_1_1469807164755_3750" dir="ltr"> </div><div id="yui_3_16_0_ym19_1_1469807164755_3751" dir="ltr">Performing most IPA commands results in errors such as ipa: ERROR: cert validation failed for "CN=ipa1.example.com,O=EXAMPLE.COM" ((SEC_ERROR_EXPIRED_CERTIFICATE) Peer's Certificate has expired.) </div><div id="yui_3_16_0_ym19_1_1469807164755_2942"> </div><div id="yui_3_16_0_ym19_1_1469807164755_3869">Not sure if it is related but we lost our first IPA server some time ago and had to promote another to the CA master. Also, due to someone leaving the company at the beginning of the year we had to change the directory manager password. I followed all the directions to do so but it does not seem like it was a completely smooth transaction. </div></div></body></html>