<html> <head> <meta http-equiv="Content-Type" content="text/html; charset=utf-8"> </head> <body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class=""> <br class=""> <div> <blockquote type="cite" class=""> <div class="">On Feb 8, 2016, at 4:28 AM, Rob Crittenden <<a href="mailto:rcritten@redhat.com" class="">rcritten@redhat.com</a>> wrote:</div> <br class="Apple-interchange-newline"> <div class=""><span style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; float: none; display: inline !important;" class="">Timothy Geier wrote:</span><br style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""> <blockquote type="cite" style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""> Greetings all,<br class=""> <br class=""> For the record,this is a CentOS 7.2 box with all current patches. (ipa-server-4.2.0-15.el7.centos.3.x86_64, etc.)<br class=""> <br class=""> The situation is that pki-tomcatd on the lone CA server in our IPA cluster refuses to start cleanly. The issues started earlier this week after the certs<br class=""> subsystemCert, ocspSigningCert, and auditSigningCert all simultaneously expired without warning; apparently, certmonger failed to renew them automatically. We<br class=""> attempted timeshifting and following instructions for what appeared to be similar issues, but nothing at all has worked.<br class=""> <br class=""> Today, we attempted removing the certificates in question (of course, the files in /etc/pki/pki-tomcat/alias were backed up beforehand) and using certutil to issue new certificates. This process worked but pki-tomcatd is still refusing to start. We can get IPA to run on this server by manually starting pki-tomcatd, running ipactl start, and then ctrl-c’ing it when it gets to "Starting pki-tomcatd" but this is not a tenable long-term solution.<br class=""> <br class=""> Relevant log entries/information:<br class=""> <br class=""> /var/log/pki/pki-tomcat/ca/debug:<br class=""> Could not connect to LDAP server host <a href="http://ipa01.XXXXXXXXX.net" class=""> ipa01.XXXXXXXXX.net</a> port 636 Error netscape.ldap.LDAPException: IO Error creating JSS SSL Socket (-1)<br class=""> Internal Database Error encountered: Could not connect to LDAP server host <a href="http://ipa01.XXXXXXXXX.net" class=""> ipa01.XXXXXXXXX.net</a> port 636 Error netscape.ldap.LDAPException: IO Error creating JSS SSL Socket (-1)<br class=""> Internal Database Error encountered: Could not connect to LDAP server host <a href="http://ipa01.XXXXXXXXX.net" class=""> ipa01.XXXXXXXXX.net</a> port 636 Error netscape.ldap.LDAPException: Authentication failed (49)<br class=""> <br class=""> /var/log/pki/pki-tomcat/localhost.2016-02-04.log:<br class=""> org.apache.catalina.core.StandardContext loadOnStartup<br class=""> SEVERE: Servlet /ca threw load() exception<br class=""> java.lang.NullPointerException<br class=""> <br class=""> # getcert list:<br class=""> <br class=""> Number of certificates and requests being tracked: 8.<br class=""> Request ID '20151015022737':<br class=""> <span class="Apple-tab-span" style="white-space: pre;"></span>status: MONITORING<br class=""> <span class="Apple-tab-span" style="white-space: pre;"></span>ca-error: Error setting up ccache for "host" service on client using default keytab: Generic error (see e-text).<br class=""> <span class="Apple-tab-span" style="white-space: pre;"></span>stuck: no<br class=""> <span class="Apple-tab-span" style="white-space: pre;"></span>key pair storage: type=NSSDB,location='/etc/dirsrv/slapd-XXXXXXXXX-NET',nickname='Server-Cert',token='NSS Certificate DB',pinfile='/etc/dirsrv/slapd-XXXXXXXXX-NET/pwdfile.txt'<br class=""> <span class="Apple-tab-span" style="white-space: pre;"></span>expires: 2017-10-15 02:09:06 UTC<br class=""> <span class="Apple-tab-span" style="white-space: pre;"></span>track: yes<br class=""> <span class="Apple-tab-span" style="white-space: pre;"></span>auto-renew: yes<br class=""> Request ID '20151015022949':<br class=""> <span class="Apple-tab-span" style="white-space: pre;"></span>status: MONITORING<br class=""> <span class="Apple-tab-span" style="white-space: pre;"></span>ca-error: Error setting up ccache for "host" service on client using default keytab: Generic error (see e-text).<br class=""> <span class="Apple-tab-span" style="white-space: pre;"></span>stuck: no<br class=""> <span class="Apple-tab-span" style="white-space: pre;"></span>key pair storage: type=NSSDB,location='/etc/httpd/alias',nickname='Server-Cert',token='NSS Certificate DB',pinfile='/etc/httpd/alias/pwdfile.txt'<br class=""> <span class="Apple-tab-span" style="white-space: pre;"></span>certificate: type=NSSDB,location='/etc/httpd/alias',nickname='Server-Cert',token='NSS Certificate DB'<br class=""> <span class="Apple-tab-span" style="white-space: pre;"></span>expires: 2017-10-15 02:09:10 UTC<br class=""> <span class="Apple-tab-span" style="white-space: pre;"></span>track: yes<br class=""> <span class="Apple-tab-span" style="white-space: pre;"></span>auto-renew: yes<br class=""> Request ID '20160127202548':<br class=""> <span class="Apple-tab-span" style="white-space: pre;"></span>status: MONITORING<br class=""> <span class="Apple-tab-span" style="white-space: pre;"></span>stuck: no<br class=""> <span class="Apple-tab-span" style="white-space: pre;"></span>key pair storage: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='caSigningCert cert-pki-ca',token='NSS Certificate DB',pin set<br class=""> <span class="Apple-tab-span" style="white-space: pre;"></span>certificate: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='caSigningCert cert-pki-ca',token='NSS Certificate DB'<br class=""> <span class="Apple-tab-span" style="white-space: pre;"></span>expires: 2034-02-11 19:46:43 UTC<br class=""> <span class="Apple-tab-span" style="white-space: pre;"></span>track: yes<br class=""> <span class="Apple-tab-span" style="white-space: pre;"></span>auto-renew: yes<br class=""> Request ID '20160127202549':<br class=""> <span class="Apple-tab-span" style="white-space: pre;"></span>status: MONITORING<br class=""> <span class="Apple-tab-span" style="white-space: pre;"></span>stuck: no<br class=""> <span class="Apple-tab-span" style="white-space: pre;"></span>key pair storage: type=NSSDB,location='/etc/httpd/alias',nickname='ipaCert',token='NSS Certificate DB',pinfile='/etc/httpd/alias/pwdfile.txt'<br class=""> <span class="Apple-tab-span" style="white-space: pre;"></span>certificate: type=NSSDB,location='/etc/httpd/alias',nickname='ipaCert',token='NSS Certificate DB'<br class=""> <span class="Apple-tab-span" style="white-space: pre;"></span>expires: 2017-12-25 04:27:49 UTC<br class=""> <span class="Apple-tab-span" style="white-space: pre;"></span>key usage: digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment<br class=""> <span class="Apple-tab-span" style="white-space: pre;"></span>track: yes<br class=""> <span class="Apple-tab-span" style="white-space: pre;"></span>auto-renew: yes<br class=""> Request ID '20160127202550':<br class=""> <span class="Apple-tab-span" style="white-space: pre;"></span>status: MONITORING<br class=""> <span class="Apple-tab-span" style="white-space: pre;"></span>ca-error: Server at "<a href="http://ipa01.XXXXXXXXX.net:8080/ca/ee/ca/profileSubmit" class="">http://ipa01.XXXXXXXXX.net:8080/ca/ee/ca/profileSubmit</a>" replied: Profile caServerCert Not Found<br class=""> <span class="Apple-tab-span" style="white-space: pre;"></span>stuck: no<br class=""> <span class="Apple-tab-span" style="white-space: pre;"></span>key pair storage: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='Server-Cert cert-pki-ca',token='NSS Certificate DB',pin set<br class=""> <span class="Apple-tab-span" style="white-space: pre;"></span>certificate: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='Server-Cert cert-pki-ca',token='NSS Certificate DB'<br class=""> <span class="Apple-tab-span" style="white-space: pre;"></span>expires: 2017-10-04 02:28:53 UTC<br class=""> <span class="Apple-tab-span" style="white-space: pre;"></span>track: yes<br class=""> <span class="Apple-tab-span" style="white-space: pre;"></span>auto-renew: yes<br class=""> Request ID '20160204165453':<br class=""> <span class="Apple-tab-span" style="white-space: pre;"></span>status: MONITORING<br class=""> <span class="Apple-tab-span" style="white-space: pre;"></span>stuck: no<br class=""> <span class="Apple-tab-span" style="white-space: pre;"></span>key pair storage: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='auditSigningCert cert-pki-ca',token='NSS Certificate DB',pin set<br class=""> <span class="Apple-tab-span" style="white-space: pre;"></span>certificate: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='auditSigningCert cert-pki-ca',token='NSS Certificate DB'<br class=""> <span class="Apple-tab-span" style="white-space: pre;"></span>expires: 2016-05-04 16:40:23 UTC<br class=""> <span class="Apple-tab-span" style="white-space: pre;"></span>track: yes<br class=""> <span class="Apple-tab-span" style="white-space: pre;"></span>auto-renew: yes<br class=""> Request ID '20160204170246':<br class=""> <span class="Apple-tab-span" style="white-space: pre;"></span>status: MONITORING<br class=""> <span class="Apple-tab-span" style="white-space: pre;"></span>stuck: no<br class=""> <span class="Apple-tab-span" style="white-space: pre;"></span>key pair storage: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='ocspSigningCert cert-pki-ca',token='NSS Certificate DB',pin set<br class=""> <span class="Apple-tab-span" style="white-space: pre;"></span>certificate: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='ocspSigningCert cert-pki-ca',token='NSS Certificate DB'<br class=""> <span class="Apple-tab-span" style="white-space: pre;"></span>expires: 2016-05-04 16:59:18 UTC<br class=""> <span class="Apple-tab-span" style="white-space: pre;"></span>track: yes<br class=""> <span class="Apple-tab-span" style="white-space: pre;"></span>auto-renew: yes<br class=""> Request ID '20160204170752':<br class=""> <span class="Apple-tab-span" style="white-space: pre;"></span>status: MONITORING<br class=""> <span class="Apple-tab-span" style="white-space: pre;"></span>stuck: no<br class=""> <span class="Apple-tab-span" style="white-space: pre;"></span>key pair storage: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='subsystemCert cert-pki-ca',token='NSS Certificate DB',pin set<br class=""> <span class="Apple-tab-span" style="white-space: pre;"></span>certificate: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='subsystemCert cert-pki-ca',token='NSS Certificate DB'<br class=""> <span class="Apple-tab-span" style="white-space: pre;"></span>expires: 2016-05-04 17:05:29 UTC<br class=""> <span class="Apple-tab-span" style="white-space: pre;"></span>track: yes<br class=""> <span class="Apple-tab-span" style="white-space: pre;"></span>auto-renew: yes<br class=""> <br class=""> # certutil -L -d /var/lib/pki/pki-tomcat/alias/<br class=""> <br class=""> Certificate Nickname Trust Attributes<br class=""> SSL,S/MIME,JAR/XPI<br class=""> auditSigningCert cert-pki-ca u,u,Pu<br class=""> ocspSigningCert cert-pki-ca u,u,u<br class=""> caSigningCert cert-pki-ca CTu,Cu,Cu<br class=""> subsystemCert cert-pki-ca u,u,u<br class=""> Server-Cert cert-pki-ca u,u,u<br class=""> <br class=""> # certutil -L -d /etc/dirsrv/slapd-XXXXXXXXX-NET/<br class=""> <br class=""> Certificate Nickname Trust Attributes<br class=""> SSL,S/MIME,JAR/XPI<br class=""> Server-Cert u,u,u<br class=""> <a href="http://XXXXXXXXX.NET" class="">XXXXXXXXX.NET</a> IPA CA CT,C,C<br class=""> <br class=""> <br class=""> <br class=""> The only thing that making new certs seemed to resolve was removing these errors from /var/log/pki/pki-tomcat/ca/system :<br class=""> <br class=""> Cannot authenticate agent with certificate Serial <redacted> Subject DN CN=IPA RA,O=<a href="http://XXXXXXXXX.NET" class="">XXXXXXXXX.NET</a>. Error: User not found<br class=""> <br class=""> Thus, the root cause(s) appears to be something else entirely that we are totally unfamilar with..we can provide any other required information to help with troubleshooting.<br class=""> </blockquote> <br style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""> <span style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; float: none; display: inline !important;" class="">It appears that the CA is not fully starting, perhaps due to these renewal issues, perhaps something else. You'll need to dig into the logs. I'd start with /var/lib/pki/pki-ca/pki-tomcat/logs/debug and selftests.log.</span><br style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""> <br style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""> </div> </blockquote> <div><br class=""> </div> <div>The debug log has a lot of instances of:</div> <div><br class=""> </div> <div>Could not connect to LDAP server host xxx.xxxx port 636 Error netscape.ldap.LDAPException: IO Error creating JSS SSL Socket (-1)</div> <div>Internal Database Error encountered: Could not connect to LDAP server host xxx.xxxx port 636 Error netscape.ldap.LDAPException: IO Error creating JSS SSL Socket (-1)</div> <div><br class=""> </div> <div>but nothing else of note other than those errors. </div> <div><br class=""> </div> <div>We’ve also noticed lots of 500 errors in /var/log/pki/pki-tomcat/localhost_access.log</div> <div>[08/Feb/2016:10:34:29 -0600] "GET /ca/ee/ca/profileSubmit?profileId=caServerCert&serial_num=5&renewal=true&xml=true HTTP/1.1" 500 2134</div> <div>[08/Feb/2016:10:34:32 -0600] "GET /ca/ee/ca/profileSubmit?profileId=caServerCert&serial_num=2&renewal=true&xml=true HTTP/1.1" 500 2134</div> <div>[08/Feb/2016:10:34:50 -0600] "GET /ca/ee/ca/profileSubmit?profileId=caServerCert&serial_num=4&renewal=true&xml=true HTTP/1.1" 500 2134</div> <div><br class=""> </div> <div>which looks like certmonger is continuously trying to renew the 3 certs.</div> <div><br class=""> </div> <div>These dates and times from selftests.log are not accurate and are from an earlier attempt to renew the certs while time shifted:</div> <div><br class=""> </div> <div> <div>0.localhost-startStop-1 - [15/Jan/2016:17:39:34 CST] [20] [1] SelfTestSubsystem: Initializing self test plugins:</div> <div>0.localhost-startStop-1 - [15/Jan/2016:17:39:34 CST] [20] [1] SelfTestSubsystem: loading all self test plugin logger parameters</div> <div>0.localhost-startStop-1 - [15/Jan/2016:17:39:34 CST] [20] [1] SelfTestSubsystem: loading all self test plugin instances</div> <div>0.localhost-startStop-1 - [15/Jan/2016:17:39:34 CST] [20] [1] SelfTestSubsystem: loading all self test plugin instance parameters</div> <div>0.localhost-startStop-1 - [15/Jan/2016:17:39:34 CST] [20] [1] SelfTestSubsystem: loading self test plugins in on-demand order</div> <div>0.localhost-startStop-1 - [15/Jan/2016:17:39:34 CST] [20] [1] SelfTestSubsystem: loading self test plugins in startup order</div> <div>0.localhost-startStop-1 - [15/Jan/2016:17:39:34 CST] [20] [1] SelfTestSubsystem: Self test plugins have been successfully loaded!</div> <div>0.localhost-startStop-1 - [15/Jan/2016:17:39:38 CST] [20] [1] SelfTestSubsystem: Running self test plugins specified to be executed at startup:</div> <div>0.localhost-startStop-1 - [15/Jan/2016:17:39:38 CST] [20] [1] CAPresence: CA is present</div> <div>0.localhost-startStop-1 - [15/Jan/2016:17:39:38 CST] [20] [1] SystemCertsVerification: system certs verification success</div> <div>0.localhost-startStop-1 - [15/Jan/2016:17:39:38 CST] [20] [1] SelfTestSubsystem: All CRITICAL self test plugins ran SUCCESSFULLY at startup!</div> </div> <div><br class=""> </div> <div>There’s nothing in that log with any February dates, so when we try to start pki-tomcatd in real time, it's likely not even getting this far.</div> <div><br class=""> </div> <blockquote type="cite" class=""> <div class=""><span style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; float: none; display: inline !important;" class="">You mentioned privately that you renamed the IPA host. This is probably what broke half of the renewals. The hosts and keytabs and many entries in IPA have the hostname baked in so you can't simply rename the host.</span><br style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""> <br style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""> </div> </blockquote> <div><br class=""> </div> <div>Technically, the host wasn’t renamed; a new CentOS 7 host was added to the existing IPA cluster that had 4 hosts (1 master CA) at CentOS 6 (using the documentation at <a href="https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/7/html/Linux_Domain_Identity_Authentication_and_Policy_Guide/upgrading.html" class="">https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/7/html/Linux_Domain_Identity_Authentication_and_Policy_Guide/upgrading.html</a>), it was promoted to the master CA, all of the C6 hosts were decommissioned/removed from replication, and then a new set of C7 hosts were created and added as replicas.</div> <div><br class=""> </div> <div>Is this the correct procedure to follow when time shifted? </div> <div><br class=""> </div> <div>- Stop IPA</div> <div>- Change the system clock (and the hardware clock) to a point before the expiration</div> <div>- Start IPA</div> <div>- Run getcert resubmit on the appropriate request IDs</div> <div>- Stop IPA</div> <div>- Return to real time</div> <div>- Start IPA</div> <div><br class=""> </div> <div>We haven’t tried it this week yet but all attempts at it last week failed without any indication as to why the certs weren’t renewing; are there any other logs to check/other steps in the procedure?</div> <div><br class=""> </div> <div>Thanks much,</div> <div><br class=""> </div> <blockquote type="cite" class=""> <div class=""><span style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; float: none; display: inline !important;" class="">rob</span></div> </blockquote> </div> <br class=""> <br class=""> </body> </html> <pre> "This message and any attachments may contain confidential information. If you have received this message in error, any use or distribution is prohibited. Please notify us by reply e-mail if you have mistakenly received this message, and immediately and permanently delete it and any attachments. Thank you."</pre>