<html> <head> <meta content="text/html; charset=windows-1252" http-equiv="Content-Type"> </head> <body text="#000000" bgcolor="#FFFFFF"> <div class="moz-cite-prefix">On 04/26/2016 01:45 PM, Rob Crittenden wrote: </div> <blockquote cite="mid:571FA944.8040003@redhat.com" type="cite">Bret Wortman wrote: <blockquote type="cite">I think I've found a deeper problem, in that I can't update these because IPA simply won't start at all now. I mistyped one of these -- the 2016-03-11 is actually 2018-03-11, and 2016-04-01 is actually 2036-04-01. As for the unknowns, the first says status: CA_REJECTED and the error says "hostname in subject of request 'zw198.private.net' does not match principal hostname 'private.net'", with stuck: yes. The second is similar, but for a different host. </blockquote> Is it really a different host and why? I think we'd need to see the full output to know what's going on. </blockquote> Full output: Number of certificates and requests being tracked: 10. Request ID '20140428181940': status: MONITORING stuck: no key pair storage: type=NSSDB,location='/etc/dirsrv/slapd-PRIVATE-NET',nickname='Server-Cert',token='NSS Certificate DB',pinfile='/etc/dirsrv/slapd-PRIVATE-NET/pwdfile.txt' certificate: type=NSSDB,location='/etc/dirsrv/slapd-PRIVATE-NET',nickname='Server-Cert',token='NSS Certificate DB' CA: IPA issuer: CN=Certificate Authority,O=PRIVATE.NET subject: CN=zsipa.private.net,O=PRIVATE.NET expires: 2018-04-02 13:04:51 UTC principal name: <a class="moz-txt-link-abbreviated" href="mailto:ldap/zsipa.private.net@PRIVATE.NET">ldap/zsipa.private.net@PRIVATE.NET</a> key usage: digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment eku: id-kp-serverAuth,id-kp-clientAuth pre-save command: post-save command: track: yes auto-renew: yes Request ID '20140428182016': status: MONITORING stuck: no key pair storage: type=NSSDB,location='/etc/httpd/alias',nickname='Server-Cert',token='NSS Certificate DB',pinfile='/etc/httpd/alias/pwdfile.txt' certificate: type=NSSDB,location='/etc/httpd/alias',nickname='Server-Cert',token='NSS Certificate DB' CA: IPA issuer: CN=Certificate Authority,O=PRIVATE.NET subject: CN=zsipa.private.net,O=PRIVATE.NET expires: 2018-04-02 13:04:31 UTC principal name: <a class="moz-txt-link-abbreviated" href="mailto:HTTP/zsipa.private.net@PRIVATE.NET">HTTP/zsipa.private.net@PRIVATE.NET</a> key usage: digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment eku: id-kp-serverAuth,id-kp-clientAuth pre-save command: post-save command: track: yes auto-renew: yes Request ID '20150211141945': status: CA_REJECTED ca-error: Server at <a class="moz-txt-link-freetext" href="https://zsipa.private.net/ipa/xml">https://zsipa.private.net/ipa/xml</a> denied our request, giving up: 2100 (RPC failed at server. Insufficient access: hostname in subject of request 'zw198.private.net' does not match principal hostname 'private.net'). stuck: yes key pair storage: type=NSSDB,location='/etc/pki/nssdb',nickname='Server-Cert',token='NSS Certificate DB' certificate: type=NSSDB,location='/etc/pki/nssdb',nickname='Server-Cert' CA: IPA issuer: subject: expires: unknown pre-save command: post-save command: track: yes auto-renew: yes Request ID '20150816194107': status: CA_UNREACHABLE ca-error: Internal error stuck: no key pair storage: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='auditSigningCert cert-pki-ca',token='NSS Certificate DB',pin='424151811070' certificate: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='auditSigningCert cert-pki-ca',token='NSS Certificate DB' CA: dogtag-ipa-ca-renew-agent issuer: CN=Certificate Authority,O=PRIVATE.NET subject: CN=CA Audit,O=PRIVATE.NET expires: 2016-04-17 18:19:19 UTC key usage: digitalSignature,nonRepudiation pre-save command: post-save command: track: yes auto-renew: yes Request ID '20150816194108': status: CA_UNREACHABLE ca-error: Internal error stuck: no key pair storage: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='ocspSigningCert cert-pki-ca',token='NSS Certificate DB',pin='424151811070' certificate: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='ocspSigningCert cert-pki-ca',token='NSS Certificate DB' CA: dogtag-ipa-ca-renew-agent issuer: CN=Certificate Authority,O=PRIVATE.NET subject: CN=OCSP Subsystem,O=PRIVATE.NET expires: 2016-04-17 18:19:18 UTC key usage: digitalSignature,nonRepudiation,keyCertSign,cRLSign eku: id-kp-OCSPSigning pre-save command: post-save command: track: yes auto-renew: yes Request ID '20150816194109': status: CA_UNREACHABLE ca-error: Internal error stuck: no key pair storage: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='subsystemCert cert-pki-ca',token='NSS Certificate DB',pin='424151811070' certificate: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='subsystemCert cert-pki-ca',token='NSS Certificate DB' CA: dogtag-ipa-ca-renew-agent issuer: CN=Certificate Authority,O=PRIVATE.NET subject: CN=CA Subsystem,O=PRIVATE.NET expires: 2016-04-17 18:19:19 UTC key usage: digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment eku: id-kp-serverAuth,id-kp-clientAuth pre-save command: post-save command: track: yes auto-renew: yes Request ID '20150816194110': status: MONITORING stuck: no key pair storage: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='caSigningCert cert-pki-ca',token='NSS Certificate DB',pin='424151811070' certificate: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='caSigningCert cert-pki-ca',token='NSS Certificate DB' CA: dogtag-ipa-ca-renew-agent issuer: CN=Certificate Authority,O=PRIVATE.NET subject: CN=Certificate Authority,O=PRIVATE.NET expires: 2036-04-01 20:16:39 UTC key usage: digitalSignature,nonRepudiation,keyCertSign,cRLSign pre-save command: post-save command: track: yes auto-renew: yes Request ID '20150816194111': status: CA_UNREACHABLE ca-error: Internal error stuck: no key pair storage: type=NSSDB,location='/etc/httpd/alias',nickname='ipaCert',token='NSS Certificate DB',pinfile='/etc/httpd/alias/pwdfile.txt' certificate: type=NSSDB,location='/etc/httpd/alias',nickname='ipaCert',token='NSS Certificate DB' CA: dogtag-ipa-ca-renew-agent issuer: CN=Certificate Authority,O=PRIVATE.NET subject: CN=IPA RA,O=PRIVATE.NET expires: 2016-04-17 18:19:35 UTC key usage: digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment eku: id-kp-serverAuth,id-kp-clientAuth pre-save command: post-save command: track: yes auto-renew: yes Request ID '20150816194112': status: MONITORING stuck: no key pair storage: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='Server-Cert cert-pki-ca',token='NSS Certificate DB',pin='424151811070' certificate: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='Server-Cert cert-pki-ca',token='NSS Certificate DB' CA: dogtag-ipa-renew-agent issuer: CN=Certificate Authority,O=PRIVATE.NET subject: CN=zsipa.private.net,O=PRIVATE.NET expires: 2018-03-11 13:04:29 UTC key usage: digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment eku: id-kp-serverAuth,id-kp-clientAuth pre-save command: post-save command: track: yes auto-renew: yes Request ID '20151214165433': status: CA_REJECTED ca-error: Server at <a class="moz-txt-link-freetext" href="https://zsipa.private.net/ipa/xml">https://zsipa.private.net/ipa/xml</a> denied our request, giving up: 2100 (RPC failed at server. Insufficient access: hostname in subject of request 'zsipa.private.net' does not match principal hostname '<a class="moz-txt-link-abbreviated" href="http://www.private.net">www.private.net</a>'). stuck: yes key pair storage: type=FILE,location='/etc/pki/tls/private/www.private.net.key' certificate: type=FILE,location='/etc/pki/tls/certs/www.private.net.crt' CA: IPA issuer: subject: expires: unknown pre-save command: post-save command: track: yes auto-renew: yes <blockquote cite="mid:571FA944.8040003@redhat.com" type="cite">A given host can only get certificates for itself or those delegated to it. Hostnames are used for this enforcement so if they don't line up you'll see this type of rejection. <blockquote type="cite"> No idea what's wrong with the rest, or why nothing will start. Near as I can tell, Kerberos is failing to start, which is causing everything else to go toes up. Early in the startup, in /var/log/messages, there's: ns-slapd: GSSAPI Error: Unspecified GSS failure. Minor code may provide more information (No Kerberos credentials available) </blockquote> Without more context it's hard to say. 389 is rather chatty about things and of course when it starts it has no ticket so it logs a bunch of stuff, eventually (hopefully) gets one, and then shuts up. <blockquote type="cite"> After that, I get a jar file read pboelm on log4j.jar, then a series of property setting attempts that don't find matching properties. Then some cipher errors, then it looks like named starts up okay, and everything pauses for about 5 minutes before it all comes crashing back down. </blockquote> I wouldn't get too hung up on particular services just yet. Without valid certs things will fail and those problems will cascade. I think we just need more details at this point. rob <blockquote type="cite"> Bret On 04/26/2016 12:40 PM, Petr Vobornik wrote: <blockquote type="cite">On 04/26/2016 06:00 PM, Bret Wortman wrote: <blockquote type="cite"># getcert list | grep expires expires: 2018-04-02 13:04:51 UTC expires: 2018-04-02 13:04:31 UTC expires: unknown expires: 2016-04-17 18:19:19 UTC expires: 2016-04-17 18:19:18 UTC expires: 2016-04-17 18:19:19 UTC expires: 2016-04-01 20:16:39 UTC expires: 2016-04-17 18:19:35 UTC expires: 2016-03-11 13:04:29 UTC expires: unknown # So some got updated and most didn't. Is there a recommended way to update these all? The system is still backdated to 3 April (ntpd disabled) at this point. </blockquote> It's usually good to start renewing(when it doesn't happen automatically from some reason) with the cert which is about to expired first, i.e. the one with "2016-03-11 13:04:29" The process is: - move date before the cert is about to expired - leave it up to certmonger or manually force resubmit by `getcert resubmit -i $REQUEST_ID`, where request ID is in `getcert list` output. I'm little worried about the fact that CA cert was renewed at date which is after expiration of the other certs. Also the `expires: unknown` doesn't look good. Check `getcert list` output for errors related to the cert. <blockquote type="cite"> Bret On 04/26/2016 11:46 AM, Petr Vobornik wrote: <blockquote type="cite">On 04/26/2016 03:26 PM, Bret Wortman wrote: <blockquote type="cite">On our non-CA IPA server, this is happening, in case it's related and illustrative: # ipa host-del zw113.private.net ipa: ERROR: Certificate format error: (SEC_ERROR_LEGACY_DATABASE) The certificate/key database is in an old, unsupported format. # </blockquote> I would start with checking on all IPA servers if and what certificates are expired: # getcert list or short version to check if there are any: # getcert list | grep expires When CA cert is renewed, it is not automatically transfered to clients. There one must run: # ipa-certupdate <blockquote type="cite">On 04/26/2016 09:24 AM, Bret Wortman wrote: <blockquote type="cite">I rolled the date on the IPA server in question back to April 1 and ran "ipa-cacert-manage renew", which said it completed successfully. I rolled the date back to current and tried restarting ipa using ipactl stop && ipactl start, but no joy. No more ca renewal errors, but right after the pause I see this in /var/log/messages: systemd: kadmin.service: main process exited, code=exited, status=2/INVALIDARGUMENT systemd: Unit kadmin.service entered failed state. systemd: kadmin.service failed. I rebooted the server just in case, and it's still getting stuck at the same place. ipa-otpd doesn't get around to starting. Bret After the several-minutes-long pause after ipactl start outputs "Starting pki-tomcatd Service", I get the On 04/26/2016 08:14 AM, Bret Wortman wrote: <blockquote type="cite">I have an IPA server on a private network which has apparently run into certificate issues this morning. It's been running without issue for quite a while, and is on 4.1.4-1 on fedora 21. This morning, the gui started giving: IPA Error 907: NetworkError with description "cannot connect to '<a class="moz-txt-link-freetext" href="https://zsipa.private.net:443/ca/agent/ca/displayBySerial">https://zsipa.private.net:443/ca/agent/ca/displayBySerial</a>': (SSL_ERROR_EXPIRED_CERRT_ALERT) SSL peer rejected your certificate as expired." I dug into the logs and after trying to restart ipa using ipactl, there was a length pause, then: dogtag-ipa-ca-renew-agent-submit: Updated certificate not available certmonger: Certificate named "ipaCert" in token "NSS Certificate DB" in database "/etc/httpd/alias" is no longer valid. dogtag-ipa-ca-renew-agent-submit: Updated certificate not available certmonger: Certificate named "ocspSigningCert cert-pki-ca" in token "NSS Certificate DB" in database "/etc/pki/pki-tomcat/alias" is no longer valid. dogtag-ipa-ca-renew-agent-submit: Updated certificate not available. named-pkcs11[3437]: client 192.168.208.205#57832: update '208.168.192.in-addr.arpa/IN' denied and then things start shutting down. I can't start ipa at all using ipactl. So at present, our DNS is down. Authentication should work for a while, but I'd like to get this working again as quickly as possible. Any ideas? I deal with certificates so infrequently (like only when something like this happens) that I'm not sure where to start. Thanks! -- *Bret Wortman* /Coming soon to Kickstarter.../ <a class="moz-txt-link-rfc2396E" href="http://wrapbuddies.co/"><http://wrapbuddies.co/></a> <a class="moz-txt-link-freetext" href="http://wrapbuddies.co/">http://wrapbuddies.co/</a> </blockquote> </blockquote> </blockquote> </blockquote> </blockquote> </blockquote> </blockquote> </blockquote> </body> </html>