[Freeipa-users] Asking for help with crashed freeIPA istance
Florence Blanc-Renaud
flo at redhat.com
Mon Jan 2 09:45:04 UTC 2017
On 12/31/2016 07:51 PM, Daniel Schimpfoessl wrote:
> Further attempts to fix the IPA server start has revealed that the ca
> admin getStatus is returning a server error (500).
>
> This has come up during restarts and ipa-server-upgrade.
>
> ipa: DEBUG: Waiting for CA to start...
> ipa: DEBUG: request POST
> http://wwgwho01.webwim.com:8080/ca/admin/ca/getStatus
> <http://wwgwho01.webwim.com:8080/ca/admin/ca/getStatus>
> ipa: DEBUG: request body ''
> ipa: DEBUG: response status 500
> ipa: DEBUG: response headers {'content-length': '2133',
> 'content-language': 'en', 'server': 'Apache-Coyote/1.1', 'connection':
> 'close', 'date': 'Sat, 31 Dec 2016 18:44:55 GMT', 'content-type':
> 'text/html;charset=utf-8'}
> ipa: DEBUG: response body '<html><head><title>Apache Tomcat/7.0.69 -
> Error report</title><style><!--H1
> {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:22px;}
> H2
> {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:16px;}
> H3
> {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:14px;}
> BODY
> {font-family:Tahoma,Arial,sans-serif;color:black;background-color:white;} B
> {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;}
> P
> {font-family:Tahoma,Arial,sans-serif;background:white;color:black;font-size:12px;}A
> {color : black;}A.name {color : black;}HR {color : #525D76;}--></style>
> </head><body><h1>HTTP Status 500 - Subsystem unavailable</h1><HR
> size="1" noshade="noshade"><p><b>type</b> Exception
> report</p><p><b>message</b> <u>Subsystem
> unavailable</u></p><p><b>description</b> <u>The server encountered an
> internal error that prevented it from fulfilling this
> request.</u></p><p><b>exception</b> <pre>javax.ws.rs
> <http://javax.ws.rs>.ServiceUnavailableException: Subsystem
> unavailable\n\tcom.netscape.cms.tomcat.ProxyRealm.findSecurityConstraints(ProxyRealm.java:145)\n\torg.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:499)\n\torg.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:103)\n\torg.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:436)\n\torg.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1078)\n\torg.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:625)\n\torg.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:316)\n\tjava.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)\n\tjava.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)\n\torg.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)\n\tjava.lang.Thread.run(Thread.java:745)\n</pre></p><p><b>note</b>
> <u>The full stack trace of the root cause is available in the Apache
> Tomcat/7.0.69 logs.</u></p><HR size="1" noshade="noshade"><h3>Apache
> Tomcat/7.0.69</h3></body></html>'
> ipa: DEBUG: The CA status is: check interrupted due to error: Retrieving
> CA status failed with status 500
> ipa: DEBUG: Waiting for CA to start...
> ipa: DEBUG: request POST
> http://wwgwho01.webwim.com:8080/ca/admin/ca/getStatus
> ipa: DEBUG: request body ''
> ipa: DEBUG: response status 500
> ipa: DEBUG: response headers {'content-length': '2133',
> 'content-language': 'en', 'server': 'Apache-Coyote/1.1', 'connection':
> 'close', 'date': 'Sat, 31 Dec 2016 18:44:56 GMT', 'content-type':
> 'text/html;charset=utf-8'}
> ipa: DEBUG: response body '<html><head><title>Apache Tomcat/7.0.69 -
> Error report</title><style><!--H1
> {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:22px;}
> H2
> {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:16px;}
> H3
> {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:14px;}
> BODY
> {font-family:Tahoma,Arial,sans-serif;color:black;background-color:white;} B
> {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;}
> P
> {font-family:Tahoma,Arial,sans-serif;background:white;color:black;font-size:12px;}A
> {color : black;}A.name {color : black;}HR {color : #525D76;}--></style>
> </head><body><h1>HTTP Status 500 - Subsystem unavailable</h1><HR
> size="1" noshade="noshade"><p><b>type</b> Exception
> report</p><p><b>message</b> <u>Subsystem
> unavailable</u></p><p><b>description</b> <u>The server encountered an
> internal error that prevented it from fulfilling this
> request.</u></p><p><b>exception</b>
> <pre>javax.ws.rs.ServiceUnavailableException: Subsystem
> unavailable\n\tcom.netscape.cms.tomcat.ProxyRealm.findSecurityConstraints(ProxyRealm.java:145)\n\torg.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:499)\n\torg.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:103)\n\torg.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:436)\n\torg.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1078)\n\torg.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:625)\n\torg.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:316)\n\tjava.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)\n\tjava.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)\n\torg.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)\n\tjava.lang.Thread.run(Thread.java:745)\n</pre></p><p><b>note</b>
> <u>The full stack trace of the root cause is available in the Apache
> Tomcat/7.0.69 logs.</u></p><HR size="1" noshade="noshade"><h3>Apache
> Tomcat/7.0.69</h3></body></html>'
> ipa: DEBUG: The CA status is: check interrupted due to error: Retrieving
> CA status failed with status 500
> ipa: DEBUG: Waiting for CA to start...
> ipa.ipaserver.install.ipa_server_upgrade.ServerUpgrade: ERROR: IPA
> server upgrade failed: Inspect /var/log/ipaupgrade.log and run command
> ipa-server-upgrade manually.
> ipa.ipaserver.install.ipa_server_upgrade.ServerUpgrade: DEBUG: File
> "/usr/lib/python2.7/site-packages/ipapython/admintool.py", line 171, in
> execute
> return_value = self.run()
> File
> "/usr/lib/python2.7/site-packages/ipaserver/install/ipa_server_upgrade.py",
> line 48, in run
> raise admintool.ScriptError(str(e))
>
> ipa.ipaserver.install.ipa_server_upgrade.ServerUpgrade: DEBUG: The
> ipa-server-upgrade command failed, exception: ScriptError: CA did not
> start in 300.0s
> ipa.ipaserver.install.ipa_server_upgrade.ServerUpgrade: ERROR: CA did
> not start in 300.0s
> ipa.ipaserver.install.ipa_server_upgrade.ServerUpgrade: ERROR: The
> ipa-server-upgrade command failed. See /var/log/ipaupgrade.log for more
> information
>
>
> with following in the syslog
> Dec 31, 2016 12:48:51 PM org.apache.catalina.core.ContainerBase
> backgroundProcess
> WARNING: Exception processing realm
> com.netscape.cms.tomcat.ProxyRealm at 38406d47 background process
> javax.ws.rs <http://javax.ws.rs>.ServiceUnavailableException: Subsystem
> unavailable
> at com.netscape.cms.tomcat.ProxyRealm.backgroundProcess(ProxyRealm.java:137)
> at
> org.apache.catalina.core.ContainerBase.backgroundProcess(ContainerBase.java:1357)
> at
> org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1543)
> at
> org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1553)
> at
> org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1553)
> at
> org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.run(ContainerBase.java:1521)
> at java.lang.Thread.run(Thread.java:745)
>
>
> 2016-12-28 18:45 GMT-06:00 Daniel Schimpfoessl <daniel at schimpfoessl.com
> <mailto:daniel at schimpfoessl.com>>:
>
> Rob/Florence,
>
> do you have any pointers on how to troubleshoot,
> reinstall/configure, update or fix the PKI server to function properly?
> Also if you know of any documentation or video that could be helpful.
> I researched the typical suspects youtube and freeipa.org
> <http://freeipa.org> without luck.
>
> Daniel
>
> 2016-12-22 18:08 GMT-06:00 Daniel Schimpfoessl
> <daniel at schimpfoessl.com <mailto:daniel at schimpfoessl.com>>:
>
> I do not believe I changed the DM password. I know I had to
> update the admin passwords regularly.
>
> Only during the startup using ipactl start --force I am able to
> connect to the service using the password for DM and it returns:
>
> # extended LDIF
> #
> # LDAPv3
> # base <> with scope baseObject
> # filter: (objectclass=*)
> # requesting: ALL
> #
>
> #
> dn:
> objectClass: top
> namingContexts: cn=changelog
> namingContexts: dc=myorg,dc=com
> namingContexts: o=ipaca
> defaultnamingcontext: dc=myorg,dc=com
> supportedExtension: 2.16.840.1.113730.3.5.7
> supportedExtension: 2.16.840.1.113730.3.5.8
> supportedExtension: 2.16.840.1.113730.3.5.10
> supportedExtension: 2.16.840.1.113730.3.8.10.3
> supportedExtension: 2.16.840.1.113730.3.8.10.4
> supportedExtension: 2.16.840.1.113730.3.8.10.4.1
> supportedExtension: 1.3.6.1.4.1.4203.1.11.1
> supportedExtension: 2.16.840.1.113730.3.8.10.1
> supportedExtension: 2.16.840.1.113730.3.8.10.5
> supportedExtension: 2.16.840.1.113730.3.5.3
> supportedExtension: 2.16.840.1.113730.3.5.12
> supportedExtension: 2.16.840.1.113730.3.5.5
> supportedExtension: 2.16.840.1.113730.3.5.6
> supportedExtension: 2.16.840.1.113730.3.5.9
> supportedExtension: 2.16.840.1.113730.3.5.4
> supportedExtension: 2.16.840.1.113730.3.6.5
> supportedExtension: 2.16.840.1.113730.3.6.6
> supportedExtension: 2.16.840.1.113730.3.6.7
> supportedExtension: 2.16.840.1.113730.3.6.8
> supportedExtension: 1.3.6.1.4.1.1466.20037
> supportedControl: 2.16.840.1.113730.3.4.2
> supportedControl: 2.16.840.1.113730.3.4.3
> supportedControl: 2.16.840.1.113730.3.4.4
> supportedControl: 2.16.840.1.113730.3.4.5
> supportedControl: 1.2.840.113556.1.4.473
> supportedControl: 2.16.840.1.113730.3.4.9
> supportedControl: 2.16.840.1.113730.3.4.16
> supportedControl: 2.16.840.1.113730.3.4.15
> supportedControl: 2.16.840.1.113730.3.4.17
> supportedControl: 2.16.840.1.113730.3.4.19
> supportedControl: 1.3.6.1.1.13.1
> supportedControl: 1.3.6.1.1.13.2
> supportedControl: 1.3.6.1.4.1.42.2.27.8.5.1
> supportedControl: 1.3.6.1.4.1.42.2.27.9.5.2
> supportedControl: 1.2.840.113556.1.4.319
> supportedControl: 1.3.6.1.4.1.42.2.27.9.5.8
> supportedControl: 1.3.6.1.4.1.4203.666.5.16
> supportedControl: 2.16.840.1.113730.3.8.10.6
> supportedControl: 2.16.840.1.113730.3.4.14
> supportedControl: 2.16.840.1.113730.3.4.20
> supportedControl: 1.3.6.1.4.1.1466.29539.12
> supportedControl: 2.16.840.1.113730.3.4.12
> supportedControl: 2.16.840.1.113730.3.4.18
> supportedControl: 2.16.840.1.113730.3.4.13
> supportedControl: 1.3.6.1.4.1.4203.1.9.1.1
> supportedSASLMechanisms: EXTERNAL
> supportedSASLMechanisms: GSS-SPNEGO
> supportedSASLMechanisms: GSSAPI
> supportedSASLMechanisms: DIGEST-MD5
> supportedSASLMechanisms: CRAM-MD5
> supportedSASLMechanisms: ANONYMOUS
> supportedLDAPVersion: 2
> supportedLDAPVersion: 3
> vendorName: 389 Project
> vendorVersion: 389-Directory/1.3.4.0 <http://1.3.4.0> B2016.215.1556
> dataversion: 020161222235947020161222235947020161222235947
> netscapemdsuffix: cn=ldap://dc=wwgwho01,dc=myorg,dc=com:389
> lastusn: 8690425
> changeLog: cn=changelog
> firstchangenumber: 2752153
> lastchangenumber: 2752346
>
> # search result
> search: 2
> result: 0 Success
>
> # numResponses: 2
> # numEntries: 1
>
>
> 2016-12-21 9:27 GMT-06:00 Rob Crittenden <rcritten at redhat.com
> <mailto:rcritten at redhat.com>>:
>
> Daniel Schimpfoessl wrote:
> > Thanks for getting back to me.
> >
> > getcert list | grep expires shows dates years in the
> future for all
> > certificates
> > Inline-Bild 1
> >
> > ipactl start --force
> >
> > Eventually the system started with:
> > Forced start, ignoring pki-tomcatd Service,
> continuing normal
> > operations.
> >
> > systemctl status ipa shows: failed
>
> I don't think this is a certificate problem at all. I think
> the timing
> with your renewal is just coincidence.
>
> Did you change your Directory Manager password at some point?
>
> >
> > ldapsearch -H ldaps://localhost:636 -D "cn=directory
> manager" -w
> > password -b "" -s base
> > ldapsearch -H ldaps://localhost:636 -D "cn=directory
> manager" -w
> > *********** -b "" -s base
> > Inline-Bild 2
>
> You need the -x flag to indicate simple bind.
>
> rob
>
> > The logs have thousands of lines like it, what am I
> looking for
> > specifically?
> >
> > Daniel
> >
> >
> > 2016-12-20 4:18 GMT-06:00 Florence Blanc-Renaud
> <flo at redhat.com <mailto:flo at redhat.com>
> > <mailto:flo at redhat.com <mailto:flo at redhat.com>>>:
> >
> > On 12/19/2016 07:15 PM, Daniel Schimpfoessl wrote:
> >
> > Good day and happy holidays,
> >
> > I have been running a freeIPA instance for a few years and been very
> > happy. Recently the certificate expired and I updated it using the
> > documented methods. At first all seemed fine. Added a Nagios
> > monitor for
> > the certificate expiration and restarted the server (single
> > server). I
> > have weekly snapshots, daily backups (using Amanda on the entire
> > disk).
> >
> > One day the services relying on IPA failed to authenticate.
> > Looking at
> > the server the ipa service had stopped. Restarting the service
> > fails.
> > Restoring a few weeks old snapshot does not start either.
> > Resetting the
> > date to a few month back does not work either as httpd fails to
> > start .
> >
> > I am at a loss.
> >
> > Here a few details:
> > # ipa --version
> > VERSION: 4.4.0, API_VERSION: 2.213
> >
> >
> > # /usr/sbin/ipactl start
> > ...
> > out -> Failed to start pki-tomcatd Service
> > /var/log/pki/pki-tomcat/ca/debug -> Could not connect to LDAP server
> > host ipa.myorg.com <http://ipa.myorg.com>
> <http://ipa.myorg.com> <http://ipa.myorg.com>
> > port 636 Error
> > netscape.ldap.LDAPException: Authentication failed (48)
> > 2016-12-19T03:02:16Z DEBUG The CA status is: check interrupted
> > due to
> > error: Retrieving CA status failed with status 500
> >
> > Any help would be appreciated as all connected services are now
> > down.
> >
> > Thanks,
> >
> > Daniel
> >
> >
> >
> >
> > Hi Daniel,
> >
> > more information would be required to understand what
> is going on.
> > First of all, which certificate did you renew? Can you
> check with
> > $ getcert list
> > if other certificates also expired?
> >
> > PKI fails to start and the error seems linked to the
> SSL connection
> > with the LDAP server. You may want to check if the
> LDAP server is
> > listening on the LDAPs port:
> > - start the stack with
> > $ ipactl start --force
> > - check the LDAPs port with
> > $ ldapsearch -H ldaps://localhost:636 -D "cn=directory
> manager" -w
> > password -b "" -s base
> >
> > The communication between PKI and the LDAP server is
> authenticated
> > with the certificate 'subsystemCert cert-pki-ca'
> located in
> > /etc/pki/pki-tomcat/alias, so you may also want to
> check if it is
> > still valid.
> > The directory server access logs (in
> > /var/log/dirsrv/slapd-DOMAIN-COM/access) would also
> show the
> > connection with logs similar to:
> >
> > [...] conn=47 fd=84 slot=84 SSL connection from
> 10.34.58.150 to
> > 10.34.58.150
> > [...] conn=47 TLS1.2 128-bit AES; client CN=CA
> > Subsystem,O=DOMAIN.COM <http://DOMAIN.COM>
> <http://DOMAIN.COM>; issuer CN=Certificate
> > Authority,O=DOMAIN.COM <http://DOMAIN.COM>
> <http://DOMAIN.COM>
> > [...] conn=47 TLS1.2 client bound as
> uid=pkidbuser,ou=people,o=ipaca
> > [...] conn=47 op=0 BIND dn="" method=sasl version=3
> mech=EXTERNAL
> > [...] conn=47 op=0 RESULT err=0 tag=97 nentries=0 etime=0
> > dn="uid=pkidbuser,ou=people,o=ipaca"
> >
> >
> >
> > HTH,
> > Flo
> >
> >
> >
> >
>
>
>
>
Hi Daniel,
the server error 500 means that PKI is not started. You can have a look
at /var/log/pki/pki-tomcat/ca/debug, especially the logs generated when
you try to start the service with
$ systemctl start pki-tomcatd at pki-tomcat.service
HTH,
Flo
More information about the Freeipa-users
mailing list