[Freeipa-users] certificates expired - won't renew

Fri Jul 29 16:06:05 UTC 2016

I have seen many threads on this so sorry to bring it up again but I have a freeipa domain, with 4 ipa servers running on redhat 6 version 3.0.0-50. The certificates are expired/expiring and will not renew and it is causing many issues for us. I have tried the many suggestions I have see in the archives such as changing the time to prior to expiration and attempting renew by resubmitting the requests but they never renew. An example of getcert list from the first server that expired:
Number of certificates and requests being tracked: 8.
Request ID '20140618161026':
    status: CA_UNREACHABLE
    ca-error: Server at https://ipa1.example.com/ipa/xml failed request, will retry: -504 (libcurl failed to execute the HTTP POST transaction.  Peer certificate cannot be authenticated with known CA certificates).
    stuck: no
    key pair storage: type=NSSDB,location='/etc/dirsrv/slapd-PKI-IPA',nickname='Server-Cert',token='NSS Certificate DB',pinfile='/etc/dirsrv/slapd-PKI-IPA/pwdfile.txt'
    certificate: type=NSSDB,location='/etc/dirsrv/slapd-PKI-IPA',nickname='Server-Cert',token='NSS Certificate DB'
    CA: IPA
    issuer: CN=Certificate Authority,O=EXAMPLE.COM
    subject: CN=idm1-io.example.com,O=EXAMPLE.COM
    expires: 2016-06-18 00:09:05 UTC
    key usage: digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment
    eku: id-kp-serverAuth,id-kp-clientAuth
    pre-save command: 
    post-save command: /usr/lib64/ipa/certmonger/restart_dirsrv PKI-IPA
    track: yes
    auto-renew: yes
Request ID '20140618161126':
    status: MONITORING
    ca-error: Internal error: no response to "http://ipa1-io.example.com:9180/ca/ee/ca/profileSubmit?profileId=auditSigningCert+cert-pki-ca&serial_num=5&renewal=true&xml=true".
    stuck: no
    key pair storage: type=NSSDB,location='/var/lib/pki-ca/alias',nickname='auditSigningCert cert-pki-ca',token='NSS Certificate DB',pin set
    certificate: type=NSSDB,location='/var/lib/pki-ca/alias',nickname='auditSigningCert cert-pki-ca',token='NSS Certificate DB'
    CA: dogtag-ipa-renew-agent
    issuer: CN=Certificate Authority,O=EXAMPLE.COM
    subject: CN=CA Audit,O=EXAMPLE.COM
    expires: 2016-06-06 23:36:29 UTC
    key usage: digitalSignature,nonRepudiation
    pre-save command: /usr/lib64/ipa/certmonger/stop_pkicad
    post-save command: /usr/lib64/ipa/certmonger/restart_pkicad "auditSigningCert cert-pki-ca"
    track: yes
    auto-renew: yes
Request ID '20140618161127':
    status: MONITORING
    ca-error: Internal error: no response to "http://ipa1.example.com:9180/ca/ee/ca/profileSubmit?profileId=ocspSigningCert+cert-pki-ca&serial_num=2&renewal=true&xml=true".
    stuck: no
    key pair storage: type=NSSDB,location='/var/lib/pki-ca/alias',nickname='ocspSigningCert cert-pki-ca',token='NSS Certificate DB',pin set
    certificate: type=NSSDB,location='/var/lib/pki-ca/alias',nickname='ocspSigningCert cert-pki-ca',token='NSS Certificate DB'
    CA: dogtag-ipa-renew-agent
    issuer: CN=Certificate Authority,O=EXAMPLE.COM
    subject: CN=OCSP Subsystem,O=EXAMPLE.COM
    expires: 2016-06-06 23:36:28 UTC
    key usage: digitalSignature,nonRepudiation,keyCertSign,cRLSign
    eku: id-kp-OCSPSigning
    pre-save command: /usr/lib64/ipa/certmonger/stop_pkicad
    post-save command: /usr/lib64/ipa/certmonger/restart_pkicad "ocspSigningCert cert-pki-ca"
    track: yes
    auto-renew: yes
Request ID '20140618161128':
    status: MONITORING
    ca-error: Internal error: no response to "http://ipa1.example.com:9180/ca/ee/ca/profileSubmit?profileId=subsystemCert+cert-pki-ca&serial_num=4&renewal=true&xml=true".
    stuck: no
    key pair storage: type=NSSDB,location='/var/lib/pki-ca/alias',nickname='subsystemCert cert-pki-ca',token='NSS Certificate DB',pin set
    certificate: type=NSSDB,location='/var/lib/pki-ca/alias',nickname='subsystemCert cert-pki-ca',token='NSS Certificate DB'
    CA: dogtag-ipa-renew-agent
    issuer: CN=Certificate Authority,O=EXAMPLE.COM
    subject: CN=CA Subsystem,O=EXAMPLE.COM
    expires: 2016-06-06 23:36:28 UTC
    key usage: digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment
    eku: id-kp-serverAuth,id-kp-clientAuth
    pre-save command: /usr/lib64/ipa/certmonger/stop_pkicad
    post-save command: /usr/lib64/ipa/certmonger/restart_pkicad "subsystemCert cert-pki-ca"
    track: yes
    auto-renew: yes
Request ID '20140618161129':
    status: MONITORING
    ca-error: Internal error: no response to "http://ipa1.example.com:9180/ca/ee/ca/profileSubmit?profileId=caServerCert&serial_num=268304385&renewal=true&xml=true".
    stuck: no
    key pair storage: type=NSSDB,location='/var/lib/pki-ca/alias',nickname='Server-Cert cert-pki-ca',token='NSS Certificate DB',pin set
    certificate: type=NSSDB,location='/var/lib/pki-ca/alias',nickname='Server-Cert cert-pki-ca',token='NSS Certificate DB'
    CA: dogtag-ipa-renew-agent
    issuer: CN=Certificate Authority,O=EXAMPLE.COM
    subject: CN=ipa1.example.com,O=EXAMPLE.COM
    expires: 2016-06-07 16:11:22 UTC
    key usage: digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment
    eku: id-kp-serverAuth
    pre-save command: 
    post-save command: 
    track: yes
    auto-renew: yes
Request ID '20140618161217':
    status: NEED_CSR_GEN_TOKEN
    stuck: yes
    key pair storage: type=NSSDB,location='/etc/dirsrv/slapd-example-COM',nickname='Server-Cert',token='NSS Certificate DB',pinfile='/etc/dirsrv/slapd-example-COM/pwdfile.txt'
    certificate: type=NSSDB,location='/etc/dirsrv/slapd-example-COM',nickname='Server-Cert',token='NSS Certificate DB'
    CA: IPA
    issuer: CN=Certificate Authority,O=EXAMPLE.COM
    subject: CN=ipa1.example.com,O=EXAMPLE.COM
    expires: 2016-06-18 00:09:05 UTC
    key usage: digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment
    eku: id-kp-serverAuth,id-kp-clientAuth
    pre-save command: 
    post-save command: /usr/lib64/ipa/certmonger/restart_dirsrv example-COM
    track: yes
    auto-renew: yes
Request ID '20140618161317':
    status: CA_UNREACHABLE
    ca-error: Server at https://ipa1.example.com/ipa/xml failed request, will retry: -504 (libcurl failed to execute the HTTP POST transaction.  Peer certificate cannot be authenticated with known CA certificates).
    stuck: no
    key pair storage: type=NSSDB,location='/etc/httpd/alias',nickname='Server-Cert',token='NSS Certificate DB',pinfile='/etc/httpd/alias/pwdfile.txt'
    certificate: type=NSSDB,location='/etc/httpd/alias',nickname='Server-Cert',token='NSS Certificate DB'
    CA: IPA
    issuer: CN=Certificate Authority,O=EXAMPLE.COM
    subject: CN=idm1-io.example.com,O=EXAMPLE.COM
    expires: 2016-06-18 00:09:06 UTC
    key usage: digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment
    eku: id-kp-serverAuth,id-kp-clientAuth
    pre-save command: 
    post-save command: /usr/lib64/ipa/certmonger/restart_httpd
    track: yes
    auto-renew: yes
Request ID '20140618161338':
    status: MONITORING
    ca-error: Internal error: no response to "http://ipa1.example.com:9180/ca/ee/ca/profileSubmit?profileId=ipaCert&serial_num=7&renewal=true&xml=true".
    stuck: no
    key pair storage: type=NSSDB,location='/etc/httpd/alias',nickname='ipaCert',token='NSS Certificate DB',pinfile='/etc/httpd/alias/pwdfile.txt'
    certificate: type=NSSDB,location='/etc/httpd/alias',nickname='ipaCert',token='NSS Certificate DB'
    CA: dogtag-ipa-renew-agent
    issuer: CN=Certificate Authority,O=EXAMPLE.COM
    subject: CN=IPA RA,O=EXAMPLE.COM
    expires: 2016-06-06 23:37:09 UTC
    key usage: digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment
    eku: id-kp-serverAuth,id-kp-clientAuth
    pre-save command: 
    post-save command: /usr/lib64/ipa/certmonger/restart_httpd
    track: yes
    auto-renew: yes
localhost log in /var/log/pki-ca have errors like:tail localhost.2016-07-29.log
Jul 29, 2016 8:55:51 AM org.apache.catalina.core.StandardWrapperValve invoke
SEVERE: Servlet.service() for servlet caProfileSubmit threw exception
java.io.IOException: CS server is not ready to serve.
    at com.netscape.cms.servlet.base.CMSServlet.service(CMSServlet.java:441)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:723)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
    at com.netscape.cms.servlet.filter.EERequestFilter.doFilter(EERequestFilter.java:176)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
    at org.
Debug log in /var/log/pki-cacd
 tail debug
[29/Jul/2016:08:49:08][Timer-0]: CMSEngine: getPasswordStore(): password store initialized before.
[29/Jul/2016:08:49:08][Timer-0]: CMSEngine: getPasswordStore(): password store initialized.
[29/Jul/2016:08:49:08][Timer-0]: SecurityDomainSessionTable getLDAPConn: netscape.ldap.LDAPException: error result (49)
[29/Jul/2016:08:49:08][Timer-0]: SecurityDomainSessionTable: unable to query sessionIds: java.io.IOException: Failed to connect to the internal database.
[29/Jul/2016:08:49:08][Timer-0]: SecurityDomainSessionTable: getSessionIds: Error in disconnecting from database: java.lang.NullPointerException
[29/Jul/2016:08:54:08][Timer-0]: CMSEngine: getPasswordStore(): password store initialized before.
[29/Jul/2016:08:54:08][Timer-0]: CMSEngine: getPasswordStore(): password store initialized.
[29/Jul/2016:08:54:08][Timer-0]: SecurityDomainSessionTable getLDAPConn: netscape.ldap.LDAPException: error result (49)
[29/Jul/2016:08:54:08][Timer-0]: SecurityDomainSessionTable: unable to query sessionIds: java.io.IOException: Failed to connect to the internal database.
[29/Jul/2016:08:54:08][Timer-0]: SecurityDomainSessionTable: getSessionIds: Error in disconnecting from database: java.lang.NullPointerException

Performing most IPA commands results in errors such as ipa: ERROR: cert validation failed for "CN=ipa1.example.com,O=EXAMPLE.COM" ((SEC_ERROR_EXPIRED_CERTIFICATE) Peer's Certificate has expired.)

Not sure if it is related but we lost our first IPA server some time ago and had to promote another to the CA master. Also, due to someone leaving the company at the beginning of the year we had to change the directory manager password. I followed all the directions to do so but it does not seem like it was a completely smooth transaction. 
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/freeipa-users/attachments/20160729/2982dd8e/attachment.htm>