<div dir="ltr">Further attempts to fix the IPA server start has revealed that the ca admin getStatus is returning a server error (500).<div><br></div><div>This has come up during restarts and ipa-server-upgrade.<br><div><br></div><div><div><font color="#999999" face="monospace, monospace">ipa: DEBUG: Waiting for CA to start...</font></div><div><font color="#999999" face="monospace, monospace">ipa: DEBUG: request POST <a href="http://wwgwho01.webwim.com:8080/ca/admin/ca/getStatus" target="_blank">http://wwgwho01.webwim.com:<wbr>8080/ca/admin/ca/getStatus</a></font></div><div><font color="#999999" face="monospace, monospace">ipa: DEBUG: request body ''</font></div><div><font color="#999999" face="monospace, monospace">ipa: DEBUG: response status 500</font></div><div><font color="#999999" face="monospace, monospace">ipa: DEBUG: response headers {'content-length': '2133', 'content-language': 'en', 'server': 'Apache-Coyote/1.1', 'connection': 'close', 'date': 'Sat, 31 Dec 2016 18:44:55 GMT', 'content-type': 'text/html;charset=utf-8'}</font></div><div><font color="#999999" face="monospace, monospace">ipa: DEBUG: response body '<html><head><title>Apache Tomcat/7.0.69 - Error report</title><style></style> </head><body><h1>HTTP Status 500 - Subsystem unavailable</h1><HR size="1" noshade="noshade"><p><b>type</<wbr>b> Exception report</p><p><b>message</b> <u>Subsystem unavailable</u></p><p><b><wbr>description</b> <u>The server encountered an internal error that prevented it from fulfilling this request.</u></p><p><b><wbr>exception</b> <pre><a href="http://javax.ws.rs">javax.ws.rs</a>.<wbr>ServiceUnavailableException: Subsystem unavailable\n\tcom.netscape.<wbr>cms.tomcat.ProxyRealm.<wbr>findSecurityConstraints(<wbr>ProxyRealm.java:145)\n\torg.<wbr>apache.catalina.authenticator.<wbr>AuthenticatorBase.invoke(<wbr>AuthenticatorBase.java:499)\n\<wbr>torg.apache.catalina.valves.<wbr>ErrorReportValve.invoke(<wbr>ErrorReportValve.java:103)\n\<wbr>torg.apache.catalina.<wbr>connector.CoyoteAdapter.<wbr>service(CoyoteAdapter.java:<wbr>436)\n\torg.apache.coyote.<wbr>http11.<wbr>AbstractHttp11Processor.<wbr>process(<wbr>AbstractHttp11Processor.java:<wbr>1078)\n\torg.apache.coyote.<wbr>AbstractProtocol$<wbr>AbstractConnectionHandler.<wbr>process(AbstractProtocol.java:<wbr>625)\n\torg.apache.tomcat.<wbr>util.net.JIoEndpoint$<wbr>SocketProcessor.run(<wbr>JIoEndpoint.java:316)\n\tjava.<wbr>util.concurrent.<wbr>ThreadPoolExecutor.runWorker(<wbr>ThreadPoolExecutor.java:1142)\<wbr>n\tjava.util.concurrent.<wbr>ThreadPoolExecutor$Worker.run(<wbr>ThreadPoolExecutor.java:617)\<wbr>n\torg.apache.tomcat.util.<wbr>threads.TaskThread$<wbr>WrappingRunnable.run(<wbr>TaskThread.java:61)\n\tjava.<wbr>lang.Thread.run(Thread.java:<wbr>745)\n</pre></p><p><b>note</b> <u>The full stack trace of the root cause is available in the Apache Tomcat/7.0.69 logs.</u></p><HR size="1" noshade="noshade"><h3>Apache Tomcat/7.0.69</h3></body></<wbr>html>'</font></div><div><font color="#999999" face="monospace, monospace">ipa: DEBUG: The CA status is: check interrupted due to error: Retrieving CA status failed with status 500</font></div></div><div><div><font color="#999999" face="monospace, monospace">ipa: DEBUG: Waiting for CA to start...</font></div><div><font color="#999999" face="monospace, monospace">ipa: DEBUG: request POST <a href="http://wwgwho01.webwim.com:8080/ca/admin/ca/getStatus">http://wwgwho01.webwim.com:8080/ca/admin/ca/getStatus</a></font></div><div><font color="#999999" face="monospace, monospace">ipa: DEBUG: request body ''</font></div><div><font color="#999999" face="monospace, monospace">ipa: DEBUG: response status 500</font></div><div><font color="#999999" face="monospace, monospace">ipa: DEBUG: response headers {'content-length': '2133', 'content-language': 'en', 'server': 'Apache-Coyote/1.1', 'connection': 'close', 'date': 'Sat, 31 Dec 2016 18:44:56 GMT', 'content-type': 'text/html;charset=utf-8'}</font></div><div><font color="#999999" face="monospace, monospace">ipa: DEBUG: response body '<html><head><title>Apache Tomcat/7.0.69 - Error report</title><style></style> </head><body><h1>HTTP Status 500 - Subsystem unavailable</h1><HR size="1" noshade="noshade"><p><b>type</b> Exception report</p><p><b>message</b> <u>Subsystem unavailable</u></p><p><b>description</b> <u>The server encountered an internal error that prevented it from fulfilling this request.</u></p><p><b>exception</b> <pre>javax.ws.rs.ServiceUnavailableException: Subsystem unavailable\n\tcom.netscape.cms.tomcat.ProxyRealm.findSecurityConstraints(ProxyRealm.java:145)\n\torg.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:499)\n\torg.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:103)\n\torg.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:436)\n\torg.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1078)\n\torg.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:625)\n\torg.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:316)\n\tjava.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)\n\tjava.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)\n\torg.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)\n\tjava.lang.Thread.run(Thread.java:745)\n</pre></p><p><b>note</b> <u>The full stack trace of the root cause is available in the Apache Tomcat/7.0.69 logs.</u></p><HR size="1" noshade="noshade"><h3>Apache Tomcat/7.0.69</h3></body></html>'</font></div><div><font color="#999999" face="monospace, monospace">ipa: DEBUG: The CA status is: check interrupted due to error: Retrieving CA status failed with status 500</font></div><div><font color="#999999" face="monospace, monospace">ipa: DEBUG: Waiting for CA to start...</font></div><div><font color="#999999" face="monospace, monospace">ipa.ipaserver.install.ipa_server_upgrade.ServerUpgrade: ERROR: IPA server upgrade failed: Inspect /var/log/ipaupgrade.log and run command ipa-server-upgrade manually.</font></div><div><font color="#999999" face="monospace, monospace">ipa.ipaserver.install.ipa_server_upgrade.ServerUpgrade: DEBUG: File "/usr/lib/python2.7/site-packages/ipapython/admintool.py", line 171, in execute</font></div><div><font color="#999999" face="monospace, monospace"> return_value = self.run()</font></div><div><font color="#999999" face="monospace, monospace"> File "/usr/lib/python2.7/site-packages/ipaserver/install/ipa_server_upgrade.py", line 48, in run</font></div><div><font color="#999999" face="monospace, monospace"> raise admintool.ScriptError(str(e))</font></div><div><font color="#999999" face="monospace, monospace"><br></font></div><div><font color="#999999" face="monospace, monospace">ipa.ipaserver.install.ipa_server_upgrade.ServerUpgrade: DEBUG: The ipa-server-upgrade command failed, exception: ScriptError: CA did not start in 300.0s</font></div><div><font color="#999999" face="monospace, monospace">ipa.ipaserver.install.ipa_server_upgrade.ServerUpgrade: ERROR: CA did not start in 300.0s</font></div><div><font color="#999999" face="monospace, monospace">ipa.ipaserver.install.ipa_server_upgrade.ServerUpgrade: ERROR: The ipa-server-upgrade command failed. See /var/log/ipaupgrade.log for more information</font></div></div><div><br></div><div><br></div><div>with following in the syslog</div><div><div><font face="monospace, monospace" color="#999999">Dec 31, 2016 12:48:51 PM org.apache.catalina.core.<wbr>ContainerBase backgroundProcess</font></div><div><font face="monospace, monospace" color="#999999">WARNING: Exception processing realm com.netscape.cms.tomcat.<wbr>ProxyRealm@38406d47 background process</font></div><div><font face="monospace, monospace" color="#999999"><a href="http://javax.ws.rs">javax.ws.rs</a>.<wbr>ServiceUnavailableException: Subsystem unavailable</font></div><div><font face="monospace, monospace" color="#999999">at com.netscape.cms.tomcat.<wbr>ProxyRealm.backgroundProcess(<wbr>ProxyRealm.java:137)</font></div><div><font face="monospace, monospace" color="#999999">at org.apache.catalina.core.<wbr>ContainerBase.<wbr>backgroundProcess(<wbr>ContainerBase.java:1357)</font></div><div><font face="monospace, monospace" color="#999999">at org.apache.catalina.core.<wbr>ContainerBase$<wbr>ContainerBackgroundProcessor.<wbr>processChildren(ContainerBase.<wbr>java:1543)</font></div><div><font face="monospace, monospace" color="#999999">at org.apache.catalina.core.<wbr>ContainerBase$<wbr>ContainerBackgroundProcessor.<wbr>processChildren(ContainerBase.<wbr>java:1553)</font></div><div><font face="monospace, monospace" color="#999999">at org.apache.catalina.core.<wbr>ContainerBase$<wbr>ContainerBackgroundProcessor.<wbr>processChildren(ContainerBase.<wbr>java:1553)</font></div><div><font face="monospace, monospace" color="#999999">at org.apache.catalina.core.<wbr>ContainerBase$<wbr>ContainerBackgroundProcessor.<wbr>run(ContainerBase.java:1521)</font></div><div><font face="monospace, monospace" color="#999999">at java.lang.Thread.run(Thread.<wbr>java:745)</font></div></div><div><br></div><div class="gmail_extra"><br><div class="gmail_quote">2016-12-28 18:45 GMT-06:00 Daniel Schimpfoessl <span dir="ltr"><<a href="mailto:daniel@schimpfoessl.com" target="_blank">daniel@schimpfoessl.com</a>></span>:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">Rob/Florence,<div><br></div><div>do you have any pointers on how to troubleshoot, reinstall/configure, update or fix the PKI server to function properly?</div><div>Also if you know of any documentation or video that could be helpful. </div><div>I researched the typical suspects youtube and <a href="http://freeipa.org" target="_blank">freeipa.org</a> without luck.</div><span class="gmail-m_2055790589579790999HOEnZb"><font color="#888888"><div><br></div><div>Daniel</div></font></span></div><div class="gmail-m_2055790589579790999HOEnZb"><div class="gmail-m_2055790589579790999h5"><div class="gmail_extra"><br><div class="gmail_quote">2016-12-22 18:08 GMT-06:00 Daniel Schimpfoessl <span dir="ltr"><<a href="mailto:daniel@schimpfoessl.com" target="_blank">daniel@schimpfoessl.com</a>></span>:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">I do not believe I changed the DM password. I know I had to update the admin passwords regularly.<div><div><br></div></div><div>Only during the startup using ipactl start --force I am able to connect to the service using the password for DM and it returns:</div><div><br></div><div><div># extended LDIF</div><div>#</div><div># LDAPv3</div><div># base <> with scope baseObject</div><div># filter: (objectclass=*)</div><div># requesting: ALL</div><div>#</div><div><br></div><div>#</div><div>dn:</div><div>objectClass: top</div><div>namingContexts: cn=changelog</div><div>namingContexts: dc=myorg,dc=com</div><div>namingContexts: o=ipaca</div><div>defaultnamingcontext: dc=myorg,dc=com</div><div>supportedExtension: 2.16.840.1.113730.3.5.7</div><div>supportedExtension: 2.16.840.1.113730.3.5.8</div><div>supportedExtension: 2.16.840.1.113730.3.5.10</div><div>supportedExtension: 2.16.840.1.113730.3.8.10.3</div><div>supportedExtension: 2.16.840.1.113730.3.8.10.4</div><div>supportedExtension: 2.16.840.1.113730.3.8.10.4.1</div><div>supportedExtension: 1.3.6.1.4.1.4203.1.11.1</div><div>supportedExtension: 2.16.840.1.113730.3.8.10.1</div><div>supportedExtension: 2.16.840.1.113730.3.8.10.5</div><div>supportedExtension: 2.16.840.1.113730.3.5.3</div><div>supportedExtension: 2.16.840.1.113730.3.5.12</div><div>supportedExtension: 2.16.840.1.113730.3.5.5</div><div>supportedExtension: 2.16.840.1.113730.3.5.6</div><div>supportedExtension: 2.16.840.1.113730.3.5.9</div><div>supportedExtension: 2.16.840.1.113730.3.5.4</div><div>supportedExtension: 2.16.840.1.113730.3.6.5</div><div>supportedExtension: 2.16.840.1.113730.3.6.6</div><div>supportedExtension: 2.16.840.1.113730.3.6.7</div><div>supportedExtension: 2.16.840.1.113730.3.6.8</div><div>supportedExtension: 1.3.6.1.4.1.1466.20037</div><div>supportedControl: 2.16.840.1.113730.3.4.2</div><div>supportedControl: 2.16.840.1.113730.3.4.3</div><div>supportedControl: 2.16.840.1.113730.3.4.4</div><div>supportedControl: 2.16.840.1.113730.3.4.5</div><div>supportedControl: 1.2.840.113556.1.4.473</div><div>supportedControl: 2.16.840.1.113730.3.4.9</div><div>supportedControl: 2.16.840.1.113730.3.4.16</div><div>supportedControl: 2.16.840.1.113730.3.4.15</div><div>supportedControl: 2.16.840.1.113730.3.4.17</div><div>supportedControl: 2.16.840.1.113730.3.4.19</div><div>supportedControl: 1.3.6.1.1.13.1</div><div>supportedControl: 1.3.6.1.1.13.2</div><div>supportedControl: 1.3.6.1.4.1.42.2.27.8.5.1</div><div>supportedControl: 1.3.6.1.4.1.42.2.27.9.5.2</div><div>supportedControl: 1.2.840.113556.1.4.319</div><div>supportedControl: 1.3.6.1.4.1.42.2.27.9.5.8</div><div>supportedControl: 1.3.6.1.4.1.4203.666.5.16</div><div>supportedControl: 2.16.840.1.113730.3.8.10.6</div><div>supportedControl: 2.16.840.1.113730.3.4.14</div><div>supportedControl: 2.16.840.1.113730.3.4.20</div><div>supportedControl: 1.3.6.1.4.1.1466.29539.12</div><div>supportedControl: 2.16.840.1.113730.3.4.12</div><div>supportedControl: 2.16.840.1.113730.3.4.18</div><div>supportedControl: 2.16.840.1.113730.3.4.13</div><div>supportedControl: 1.3.6.1.4.1.4203.1.9.1.1</div><div>supportedSASLMechanisms: EXTERNAL</div><div>supportedSASLMechanisms: GSS-SPNEGO</div><div>supportedSASLMechanisms: GSSAPI</div><div>supportedSASLMechanisms: DIGEST-MD5</div><div>supportedSASLMechanisms: CRAM-MD5</div><div>supportedSASLMechanisms: ANONYMOUS</div><div>supportedLDAPVersion: 2</div><div>supportedLDAPVersion: 3</div><div>vendorName: 389 Project</div><div>vendorVersion: 389-Directory/<a href="http://1.3.4.0" target="_blank">1.3.4.0</a> B2016.215.1556</div><div>dataversion: 020161222235947020161222235947<wbr>020161222235947</div><div>netscapemdsuffix: cn=ldap://dc=wwgwho01,dc=myorg<wbr>,dc=com:389</div><div>lastusn: 8690425</div><div>changeLog: cn=changelog</div><div>firstchangenumber: 2752153</div><div>lastchangenumber: 2752346</div><div><br></div><div># search result</div><div>search: 2</div><div>result: 0 Success</div><div><br></div><div># numResponses: 2</div><div># numEntries: 1</div></div><div><br></div></div><div class="gmail_extra"><br><div class="gmail_quote">2016-12-21 9:27 GMT-06:00 Rob Crittenden <span dir="ltr"><<a href="mailto:rcritten@redhat.com" target="_blank">rcritten@redhat.com</a>></span>:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">Daniel Schimpfoessl wrote:<br> > Thanks for getting back to me.<br> ><br> > getcert list | grep expires shows dates years in the future for all<br> > certificates<br> > Inline-Bild 1<br> ><br> > ipactl start --force<br> ><br> > Eventually the system started with:<br> > Forced start, ignoring pki-tomcatd Service, continuing normal<br> > operations.<br> ><br> > systemctl status ipa shows: failed<br> <br> I don't think this is a certificate problem at all. I think the timing<br> with your renewal is just coincidence.<br> <br> Did you change your Directory Manager password at some point?<br> <br> ><br> > ldapsearch -H ldaps://localhost:636 -D "cn=directory manager" -w<br> > password -b "" -s base<br> > ldapsearch -H ldaps://localhost:636 -D "cn=directory manager" -w<br> > *********** -b "" -s base<br> > Inline-Bild 2<br> <br> You need the -x flag to indicate simple bind.<br> <br> rob<br> <br> > The logs have thousands of lines like it, what am I looking for<br> > specifically?<br> ><br> > Daniel<br> ><br> ><br> > 2016-12-20 4:18 GMT-06:00 Florence Blanc-Renaud <<a href="mailto:flo@redhat.com" target="_blank">flo@redhat.com</a><br> > <mailto:<a href="mailto:flo@redhat.com" target="_blank">flo@redhat.com</a>>>:<br> <span>><br> > On 12/19/2016 07:15 PM, Daniel Schimpfoessl wrote:<br> ><br> > Good day and happy holidays,<br> ><br> > I have been running a freeIPA instance for a few years and been very<br> > happy. Recently the certificate expired and I updated it using the<br> > documented methods. At first all seemed fine. Added a Nagios<br> > monitor for<br> > the certificate expiration and restarted the server (single<br> > server). I<br> > have weekly snapshots, daily backups (using Amanda on the entire<br> > disk).<br> ><br> > One day the services relying on IPA failed to authenticate.<br> > Looking at<br> > the server the ipa service had stopped. Restarting the service<br> > fails.<br> > Restoring a few weeks old snapshot does not start either.<br> > Resetting the<br> > date to a few month back does not work either as httpd fails to<br> > start .<br> ><br> > I am at a loss.<br> ><br> </span><span>> Here a few details:<br> > # ipa --version<br> > VERSION: 4.4.0, API_VERSION: 2.213<br> ><br> ><br> > # /usr/sbin/ipactl start<br> > ...<br> > out -> Failed to start pki-tomcatd Service<br> > /var/log/pki/pki-tomcat/ca/de<wbr>bug -> Could not connect to LDAP server<br> </span>> host <a href="http://ipa.myorg.com" rel="noreferrer" target="_blank">ipa.myorg.com</a> <<a href="http://ipa.myorg.com" rel="noreferrer" target="_blank">http://ipa.myorg.com</a>> <<a href="http://ipa.myorg.com" rel="noreferrer" target="_blank">http://ipa.myorg.com</a>><br> <span>> port 636 Error<br> > netscape.ldap.LDAPException: Authentication failed (48)<br> > 2016-12-19T03:02:16Z DEBUG The CA status is: check interrupted<br> > due to<br> > error: Retrieving CA status failed with status 500<br> ><br> > Any help would be appreciated as all connected services are now<br> > down.<br> ><br> > Thanks,<br> ><br> > Daniel<br> ><br> ><br> ><br> ><br> </span>> Hi Daniel,<br> ><br> > more information would be required to understand what is going on.<br> > First of all, which certificate did you renew? Can you check with<br> > $ getcert list<br> > if other certificates also expired?<br> ><br> > PKI fails to start and the error seems linked to the SSL connection<br> > with the LDAP server. You may want to check if the LDAP server is<br> > listening on the LDAPs port:<br> > - start the stack with<br> > $ ipactl start --force<br> > - check the LDAPs port with<br> > $ ldapsearch -H ldaps://localhost:636 -D "cn=directory manager" -w<br> > password -b "" -s base<br> ><br> > The communication between PKI and the LDAP server is authenticated<br> > with the certificate 'subsystemCert cert-pki-ca' located in<br> > /etc/pki/pki-tomcat/alias, so you may also want to check if it is<br> > still valid.<br> > The directory server access logs (in<br> > /var/log/dirsrv/slapd-DOMAIN-<wbr>COM/access) would also show the<br> > connection with logs similar to:<br> ><br> > [...] conn=47 fd=84 slot=84 SSL connection from 10.34.58.150 to<br> > 10.34.58.150<br> > [...] conn=47 TLS1.2 128-bit AES; client CN=CA<br> > Subsystem,O=<a href="http://DOMAIN.COM" rel="noreferrer" target="_blank">DOMAIN.COM</a> <<a href="http://DOMAIN.COM" rel="noreferrer" target="_blank">http://DOMAIN.COM</a>>; issuer CN=Certificate<br> > Authority,O=<a href="http://DOMAIN.COM" rel="noreferrer" target="_blank">DOMAIN.COM</a> <<a href="http://DOMAIN.COM" rel="noreferrer" target="_blank">http://DOMAIN.COM</a>><br> > [...] conn=47 TLS1.2 client bound as uid=pkidbuser,ou=people,o=ipac<wbr>a<br> > [...] conn=47 op=0 BIND dn="" method=sasl version=3 mech=EXTERNAL<br> > [...] conn=47 op=0 RESULT err=0 tag=97 nentries=0 etime=0<br> > dn="uid=pkidbuser,ou=people,o<wbr>=ipaca"<br> ><br> ><br> ><br> > HTH,<br> > Flo<br> ><br> ><br> ><br> ><br> <br> </blockquote></div><br></div> </blockquote></div><br></div> </div></div></blockquote></div><br></div></div></div>