[Freeipa-users] Asking for help with crashed freeIPA istance
Daniel Schimpfoessl
daniel at schimpfoessl.com
Sat Dec 31 18:51:25 UTC 2016
Further attempts to fix the IPA server start has revealed that the ca admin
getStatus is returning a server error (500).
This has come up during restarts and ipa-server-upgrade.
ipa: DEBUG: Waiting for CA to start...
ipa: DEBUG: request POST http://wwgwho01.webwim.com:
8080/ca/admin/ca/getStatus
ipa: DEBUG: request body ''
ipa: DEBUG: response status 500
ipa: DEBUG: response headers {'content-length': '2133', 'content-language':
'en', 'server': 'Apache-Coyote/1.1', 'connection': 'close', 'date': 'Sat,
31 Dec 2016 18:44:55 GMT', 'content-type': 'text/html;charset=utf-8'}
ipa: DEBUG: response body '<html><head><title>Apache Tomcat/7.0.69 - Error
report</title><style><!--H1 {font-family:Tahoma,Arial,
sans-serif;color:white;background-color:#525D76;font-size:22px;} H2
{font-family:Tahoma,Arial,sans-serif;color:white;
background-color:#525D76;font-size:16px;} H3 {font-family:Tahoma,Arial,
sans-serif;color:white;background-color:#525D76;font-size:14px;} BODY
{font-family:Tahoma,Arial,sans-serif;color:black;background-color:white;} B
{font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;}
P {font-family:Tahoma,Arial,sans-serif;background:white;color:black;font-size:12px;}A
{color : black;}A.name {color : black;}HR {color : #525D76;}--></style>
</head><body><h1>HTTP Status 500 - Subsystem unavailable</h1><HR size="1"
noshade="noshade"><p><b>type</b> Exception report</p><p><b>message</b>
<u>Subsystem unavailable</u></p><p><b>description</b> <u>The server
encountered an internal error that prevented it from fulfilling this
request.</u></p><p><b>exception</b>
<pre>javax.ws.rs.ServiceUnavailableException:
Subsystem unavailable\n\tcom.netscape.cms.tomcat.ProxyRealm.
findSecurityConstraints(ProxyRealm.java:145)\n\torg.
apache.catalina.authenticator.AuthenticatorBase.invoke(
AuthenticatorBase.java:499)\n\torg.apache.catalina.valves.
ErrorReportValve.invoke(ErrorReportValve.java:103)\n\torg.apache.catalina.
connector.CoyoteAdapter.service(CoyoteAdapter.java:
436)\n\torg.apache.coyote.http11.AbstractHttp11Processor.process(
AbstractHttp11Processor.java:1078)\n\torg.apache.coyote.AbstractProtocol$
AbstractConnectionHandler.process(AbstractProtocol.java:
625)\n\torg.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(
JIoEndpoint.java:316)\n\tjava.util.concurrent.ThreadPoolExecutor.runWorker(
ThreadPoolExecutor.java:1142)\n\tjava.util.concurrent.
ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)\
n\torg.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(
TaskThread.java:61)\n\tjava.lang.Thread.run(Thread.java:745)\n</pre></p><p><b>note</b>
<u>The full stack trace of the root cause is available in the Apache
Tomcat/7.0.69 logs.</u></p><HR size="1" noshade="noshade"><h3>Apache
Tomcat/7.0.69</h3></body></html>'
ipa: DEBUG: The CA status is: check interrupted due to error: Retrieving CA
status failed with status 500
ipa: DEBUG: Waiting for CA to start...
ipa: DEBUG: request POST
http://wwgwho01.webwim.com:8080/ca/admin/ca/getStatus
ipa: DEBUG: request body ''
ipa: DEBUG: response status 500
ipa: DEBUG: response headers {'content-length': '2133', 'content-language':
'en', 'server': 'Apache-Coyote/1.1', 'connection': 'close', 'date': 'Sat,
31 Dec 2016 18:44:56 GMT', 'content-type': 'text/html;charset=utf-8'}
ipa: DEBUG: response body '<html><head><title>Apache Tomcat/7.0.69 - Error
report</title><style><!--H1
{font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:22px;}
H2
{font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:16px;}
H3
{font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:14px;}
BODY
{font-family:Tahoma,Arial,sans-serif;color:black;background-color:white;} B
{font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;}
P
{font-family:Tahoma,Arial,sans-serif;background:white;color:black;font-size:12px;}A
{color : black;}A.name {color : black;}HR {color : #525D76;}--></style>
</head><body><h1>HTTP Status 500 - Subsystem unavailable</h1><HR size="1"
noshade="noshade"><p><b>type</b> Exception report</p><p><b>message</b>
<u>Subsystem unavailable</u></p><p><b>description</b> <u>The server
encountered an internal error that prevented it from fulfilling this
request.</u></p><p><b>exception</b>
<pre>javax.ws.rs.ServiceUnavailableException: Subsystem
unavailable\n\tcom.netscape.cms.tomcat.ProxyRealm.findSecurityConstraints(ProxyRealm.java:145)\n\torg.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:499)\n\torg.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:103)\n\torg.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:436)\n\torg.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1078)\n\torg.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:625)\n\torg.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:316)\n\tjava.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)\n\tjava.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)\n\torg.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)\n\tjava.lang.Thread.run(Thread.java:745)\n</pre></p><p><b>note</b>
<u>The full stack trace of the root cause is available in the Apache
Tomcat/7.0.69 logs.</u></p><HR size="1" noshade="noshade"><h3>Apache
Tomcat/7.0.69</h3></body></html>'
ipa: DEBUG: The CA status is: check interrupted due to error: Retrieving CA
status failed with status 500
ipa: DEBUG: Waiting for CA to start...
ipa.ipaserver.install.ipa_server_upgrade.ServerUpgrade: ERROR: IPA server
upgrade failed: Inspect /var/log/ipaupgrade.log and run command
ipa-server-upgrade manually.
ipa.ipaserver.install.ipa_server_upgrade.ServerUpgrade: DEBUG: File
"/usr/lib/python2.7/site-packages/ipapython/admintool.py", line 171, in
execute
return_value = self.run()
File
"/usr/lib/python2.7/site-packages/ipaserver/install/ipa_server_upgrade.py",
line 48, in run
raise admintool.ScriptError(str(e))
ipa.ipaserver.install.ipa_server_upgrade.ServerUpgrade: DEBUG: The
ipa-server-upgrade command failed, exception: ScriptError: CA did not start
in 300.0s
ipa.ipaserver.install.ipa_server_upgrade.ServerUpgrade: ERROR: CA did not
start in 300.0s
ipa.ipaserver.install.ipa_server_upgrade.ServerUpgrade: ERROR: The
ipa-server-upgrade command failed. See /var/log/ipaupgrade.log for more
information
with following in the syslog
Dec 31, 2016 12:48:51 PM org.apache.catalina.core.ContainerBase
backgroundProcess
WARNING: Exception processing realm com.netscape.cms.tomcat.
ProxyRealm at 38406d47 background process
javax.ws.rs.ServiceUnavailableException: Subsystem unavailable
at com.netscape.cms.tomcat.ProxyRealm.backgroundProcess(ProxyRealm.java:137)
at org.apache.catalina.core.ContainerBase.backgroundProcess(
ContainerBase.java:1357)
at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.
processChildren(ContainerBase.java:1543)
at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.
processChildren(ContainerBase.java:1553)
at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.
processChildren(ContainerBase.java:1553)
at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.
run(ContainerBase.java:1521)
at java.lang.Thread.run(Thread.java:745)
2016-12-28 18:45 GMT-06:00 Daniel Schimpfoessl <daniel at schimpfoessl.com>:
> Rob/Florence,
>
> do you have any pointers on how to troubleshoot, reinstall/configure,
> update or fix the PKI server to function properly?
> Also if you know of any documentation or video that could be helpful.
> I researched the typical suspects youtube and freeipa.org without luck.
>
> Daniel
>
> 2016-12-22 18:08 GMT-06:00 Daniel Schimpfoessl <daniel at schimpfoessl.com>:
>
>> I do not believe I changed the DM password. I know I had to update the
>> admin passwords regularly.
>>
>> Only during the startup using ipactl start --force I am able to connect
>> to the service using the password for DM and it returns:
>>
>> # extended LDIF
>> #
>> # LDAPv3
>> # base <> with scope baseObject
>> # filter: (objectclass=*)
>> # requesting: ALL
>> #
>>
>> #
>> dn:
>> objectClass: top
>> namingContexts: cn=changelog
>> namingContexts: dc=myorg,dc=com
>> namingContexts: o=ipaca
>> defaultnamingcontext: dc=myorg,dc=com
>> supportedExtension: 2.16.840.1.113730.3.5.7
>> supportedExtension: 2.16.840.1.113730.3.5.8
>> supportedExtension: 2.16.840.1.113730.3.5.10
>> supportedExtension: 2.16.840.1.113730.3.8.10.3
>> supportedExtension: 2.16.840.1.113730.3.8.10.4
>> supportedExtension: 2.16.840.1.113730.3.8.10.4.1
>> supportedExtension: 1.3.6.1.4.1.4203.1.11.1
>> supportedExtension: 2.16.840.1.113730.3.8.10.1
>> supportedExtension: 2.16.840.1.113730.3.8.10.5
>> supportedExtension: 2.16.840.1.113730.3.5.3
>> supportedExtension: 2.16.840.1.113730.3.5.12
>> supportedExtension: 2.16.840.1.113730.3.5.5
>> supportedExtension: 2.16.840.1.113730.3.5.6
>> supportedExtension: 2.16.840.1.113730.3.5.9
>> supportedExtension: 2.16.840.1.113730.3.5.4
>> supportedExtension: 2.16.840.1.113730.3.6.5
>> supportedExtension: 2.16.840.1.113730.3.6.6
>> supportedExtension: 2.16.840.1.113730.3.6.7
>> supportedExtension: 2.16.840.1.113730.3.6.8
>> supportedExtension: 1.3.6.1.4.1.1466.20037
>> supportedControl: 2.16.840.1.113730.3.4.2
>> supportedControl: 2.16.840.1.113730.3.4.3
>> supportedControl: 2.16.840.1.113730.3.4.4
>> supportedControl: 2.16.840.1.113730.3.4.5
>> supportedControl: 1.2.840.113556.1.4.473
>> supportedControl: 2.16.840.1.113730.3.4.9
>> supportedControl: 2.16.840.1.113730.3.4.16
>> supportedControl: 2.16.840.1.113730.3.4.15
>> supportedControl: 2.16.840.1.113730.3.4.17
>> supportedControl: 2.16.840.1.113730.3.4.19
>> supportedControl: 1.3.6.1.1.13.1
>> supportedControl: 1.3.6.1.1.13.2
>> supportedControl: 1.3.6.1.4.1.42.2.27.8.5.1
>> supportedControl: 1.3.6.1.4.1.42.2.27.9.5.2
>> supportedControl: 1.2.840.113556.1.4.319
>> supportedControl: 1.3.6.1.4.1.42.2.27.9.5.8
>> supportedControl: 1.3.6.1.4.1.4203.666.5.16
>> supportedControl: 2.16.840.1.113730.3.8.10.6
>> supportedControl: 2.16.840.1.113730.3.4.14
>> supportedControl: 2.16.840.1.113730.3.4.20
>> supportedControl: 1.3.6.1.4.1.1466.29539.12
>> supportedControl: 2.16.840.1.113730.3.4.12
>> supportedControl: 2.16.840.1.113730.3.4.18
>> supportedControl: 2.16.840.1.113730.3.4.13
>> supportedControl: 1.3.6.1.4.1.4203.1.9.1.1
>> supportedSASLMechanisms: EXTERNAL
>> supportedSASLMechanisms: GSS-SPNEGO
>> supportedSASLMechanisms: GSSAPI
>> supportedSASLMechanisms: DIGEST-MD5
>> supportedSASLMechanisms: CRAM-MD5
>> supportedSASLMechanisms: ANONYMOUS
>> supportedLDAPVersion: 2
>> supportedLDAPVersion: 3
>> vendorName: 389 Project
>> vendorVersion: 389-Directory/1.3.4.0 B2016.215.1556
>> dataversion: 020161222235947020161222235947020161222235947
>> netscapemdsuffix: cn=ldap://dc=wwgwho01,dc=myorg,dc=com:389
>> lastusn: 8690425
>> changeLog: cn=changelog
>> firstchangenumber: 2752153
>> lastchangenumber: 2752346
>>
>> # search result
>> search: 2
>> result: 0 Success
>>
>> # numResponses: 2
>> # numEntries: 1
>>
>>
>> 2016-12-21 9:27 GMT-06:00 Rob Crittenden <rcritten at redhat.com>:
>>
>>> Daniel Schimpfoessl wrote:
>>> > Thanks for getting back to me.
>>> >
>>> > getcert list | grep expires shows dates years in the future for all
>>> > certificates
>>> > Inline-Bild 1
>>> >
>>> > ipactl start --force
>>> >
>>> > Eventually the system started with:
>>> > Forced start, ignoring pki-tomcatd Service, continuing normal
>>> > operations.
>>> >
>>> > systemctl status ipa shows: failed
>>>
>>> I don't think this is a certificate problem at all. I think the timing
>>> with your renewal is just coincidence.
>>>
>>> Did you change your Directory Manager password at some point?
>>>
>>> >
>>> > ldapsearch -H ldaps://localhost:636 -D "cn=directory manager" -w
>>> > password -b "" -s base
>>> > ldapsearch -H ldaps://localhost:636 -D "cn=directory manager" -w
>>> > *********** -b "" -s base
>>> > Inline-Bild 2
>>>
>>> You need the -x flag to indicate simple bind.
>>>
>>> rob
>>>
>>> > The logs have thousands of lines like it, what am I looking for
>>> > specifically?
>>> >
>>> > Daniel
>>> >
>>> >
>>> > 2016-12-20 4:18 GMT-06:00 Florence Blanc-Renaud <flo at redhat.com
>>> > <mailto:flo at redhat.com>>:
>>> >
>>> > On 12/19/2016 07:15 PM, Daniel Schimpfoessl wrote:
>>> >
>>> > Good day and happy holidays,
>>> >
>>> > I have been running a freeIPA instance for a few years and
>>> been very
>>> > happy. Recently the certificate expired and I updated it using
>>> the
>>> > documented methods. At first all seemed fine. Added a Nagios
>>> > monitor for
>>> > the certificate expiration and restarted the server (single
>>> > server). I
>>> > have weekly snapshots, daily backups (using Amanda on the
>>> entire
>>> > disk).
>>> >
>>> > One day the services relying on IPA failed to authenticate.
>>> > Looking at
>>> > the server the ipa service had stopped. Restarting the service
>>> > fails.
>>> > Restoring a few weeks old snapshot does not start either.
>>> > Resetting the
>>> > date to a few month back does not work either as httpd fails to
>>> > start .
>>> >
>>> > I am at a loss.
>>> >
>>> > Here a few details:
>>> > # ipa --version
>>> > VERSION: 4.4.0, API_VERSION: 2.213
>>> >
>>> >
>>> > # /usr/sbin/ipactl start
>>> > ...
>>> > out -> Failed to start pki-tomcatd Service
>>> > /var/log/pki/pki-tomcat/ca/debug -> Could not connect to LDAP
>>> server
>>> > host ipa.myorg.com <http://ipa.myorg.com> <
>>> http://ipa.myorg.com>
>>> > port 636 Error
>>> > netscape.ldap.LDAPException: Authentication failed (48)
>>> > 2016-12-19T03:02:16Z DEBUG The CA status is: check interrupted
>>> > due to
>>> > error: Retrieving CA status failed with status 500
>>> >
>>> > Any help would be appreciated as all connected services are now
>>> > down.
>>> >
>>> > Thanks,
>>> >
>>> > Daniel
>>> >
>>> >
>>> >
>>> >
>>> > Hi Daniel,
>>> >
>>> > more information would be required to understand what is going on.
>>> > First of all, which certificate did you renew? Can you check with
>>> > $ getcert list
>>> > if other certificates also expired?
>>> >
>>> > PKI fails to start and the error seems linked to the SSL connection
>>> > with the LDAP server. You may want to check if the LDAP server is
>>> > listening on the LDAPs port:
>>> > - start the stack with
>>> > $ ipactl start --force
>>> > - check the LDAPs port with
>>> > $ ldapsearch -H ldaps://localhost:636 -D "cn=directory manager" -w
>>> > password -b "" -s base
>>> >
>>> > The communication between PKI and the LDAP server is authenticated
>>> > with the certificate 'subsystemCert cert-pki-ca' located in
>>> > /etc/pki/pki-tomcat/alias, so you may also want to check if it is
>>> > still valid.
>>> > The directory server access logs (in
>>> > /var/log/dirsrv/slapd-DOMAIN-COM/access) would also show the
>>> > connection with logs similar to:
>>> >
>>> > [...] conn=47 fd=84 slot=84 SSL connection from 10.34.58.150 to
>>> > 10.34.58.150
>>> > [...] conn=47 TLS1.2 128-bit AES; client CN=CA
>>> > Subsystem,O=DOMAIN.COM <http://DOMAIN.COM>; issuer CN=Certificate
>>> > Authority,O=DOMAIN.COM <http://DOMAIN.COM>
>>> > [...] conn=47 TLS1.2 client bound as uid=pkidbuser,ou=people,o=ipac
>>> a
>>> > [...] conn=47 op=0 BIND dn="" method=sasl version=3 mech=EXTERNAL
>>> > [...] conn=47 op=0 RESULT err=0 tag=97 nentries=0 etime=0
>>> > dn="uid=pkidbuser,ou=people,o=ipaca"
>>> >
>>> >
>>> >
>>> > HTH,
>>> > Flo
>>> >
>>> >
>>> >
>>> >
>>>
>>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/freeipa-users/attachments/20161231/0f77ed61/attachment.htm>
More information about the Freeipa-users
mailing list