[Spacewalk-list] API requests overloading Tomcat???
Matthew Madey
mattmadey at gmail.com
Wed Aug 26 16:52:36 UTC 2015
I use the API heavily for automation purposes. While load starts to
increase on the application server, we reach a breaking point where
Spacewalk becomes unresponsive, and throws 500 internal server errors.
Listed at the bottom are some of the errors I see when this happens. I'm
looking for advice on how to better diagnose and\or tune my settings to
optimize Tomcat performance. I feel like the system is beefy enough to
handle this load, but Tomcat appears to be my bottleneck.. How should I go
about resolving this? I have played around with the AJP connector settings
quite a bit, but can't seem to find suitable parameters.
Spacewalk 2.2
OS: RHEL6.6
RAM: 64GB (Please note, under heavy load, we are not utilizing
even 50% of RAM.. it seems to be all CPU, but I'm not sure how to get
Tomcat to utilize more RAM other than boosting the Xmx settings which I
have already done)
CPU: 16 (vCPU)
Tomcat:
apache-tomcat-apis-0.1-1.el6.noarch
tomcat6-el-2.1-api-6.0.24-83.el6_6.x86_64
tomcat6-lib-6.0.24-83.el6_6.x86_64
tomcat6-servlet-2.5-api-6.0.24-83.el6_6.x86_64
tomcat6-6.0.24-83.el6_6.x86_64
tomcat5-jsp-2.0-api-5.5.27-7.jpp5.noarch
tomcat6-jsp-2.1-api-6.0.24-83.el6_6.x86_64
tomcat5-servlet-2.4-api-5.5.27-7.jpp5.noarch
Java:
java-1.6.0-openjdk-1.6.0.36-1.13.8.1.el6_7.x86_64
Oracle client:
oracle-instantclient11.2-basic-11.2.0.3.0-1.x86_64
JAVA_OPTS="-ea -Xms512m -Xmx4096m -Djava.awt.headless=true
-Dorg.xml.sax.driver=org.apache.xerces.parsers.SAXParser
-Dorg.apache.tomcat.util.http.Parameters.MAX_COUNT=1024 -XX:MaxNewSize=256
-XX:-UseConcMarkSweepGC -Dnet.sf.ehcache.skipUpdateCheck=true
-Djavax.sql.DataSource.Factory=org.apache.commons.dbcp.BasicDataSourceFactory
-Djava.library.path=\${system_property:java.library.path}:/usr/lib:/usr/lib64/oracle/11.2/client/lib"
Connector settings:
<Connector port="8080" protocol="HTTP/1.1" connectionTimeout="20000"
redirectPort="8443" URIEncoding="UTF-8" address="127.0.0.1"
maxThreads="1024" maxKeepAliveRequests="1000"/>
<!-- Define an AJP 1.3 Connector on port 8009 -->
<Connector port="8009" protocol="AJP/1.3" redirectPort="8443"
URIEncoding="UTF-8" address="127.0.0.1" maxThreads="1024"/>
<Connector port="8009" protocol="AJP/1.3" redirectPort="8443"
URIEncoding="UTF-8" address="::1" maxThreads="1024"/>
>From /var/log/tomcat6/catalina.out
Aug 25, 2015 1:33:23 AM org.apache.jk.core.MsgContext action
WARNING: Unable to send headers
java.net.SocketException: Broken pipe
at java.net.SocketOutputStream.socketWrite0(Native Method)
at
java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
at java.net.SocketOutputStream.write(SocketOutputStream.java:153)
at org.apache.jk.common.ChannelSocket.send(ChannelSocket.java:532)
at
org.apache.jk.common.JkInputStream.appendHead(JkInputStream.java:326)
at org.apache.jk.core.MsgContext.action(MsgContext.java:266)
at org.apache.coyote.Response.action(Response.java:183)
at org.apache.coyote.Response.sendHeaders(Response.java:379)
at
org.apache.catalina.connector.OutputBuffer.doFlush(OutputBuffer.java:305)
at
org.apache.catalina.connector.OutputBuffer.flush(OutputBuffer.java:288)
at
org.apache.catalina.connector.CoyoteWriter.flush(CoyoteWriter.java:95)
at
org.apache.jasper.runtime.JspWriterImpl.flush(JspWriterImpl.java:175)
at
org.apache.jasper.runtime.JspRuntimeLibrary.include(JspRuntimeLibrary.java:956)
at
org.apache.jasper.runtime.PageContextImpl.include(PageContextImpl.java:622)
at
com.opensymphony.module.sitemesh.taglib.page.ApplyDecoratorTag.doEndTag(ApplyDecoratorTag.java:258)
at
org.apache.jsp.WEB_002dINF.pages.common.errors._500_jsp._jspx_meth_page_005fapplyDecorator_005f0(Unknown
Source)
at
org.apache.jsp.WEB_002dINF.pages.common.errors._500_jsp._jspService(Unknown
Source)
at
org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:70)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:717)
at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290)
at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
at
org.apache.catalina.core.ApplicationDispatcher.invoke(ApplicationDispatcher.java:646)
at
org.apache.catalina.core.ApplicationDispatcher.processRequest(ApplicationDispatcher.java:438)
at
org.apache.catalina.core.ApplicationDispatcher.doForward(ApplicationDispatcher.java:374)
at
org.apache.catalina.core.ApplicationDispatcher.forward(ApplicationDispatcher.java:302)
at
org.apache.catalina.core.StandardHostValve.custom(StandardHostValve.java:415)
at
org.apache.catalina.core.StandardHostValve.status(StandardHostValve.java:342)
at
org.apache.catalina.core.StandardHostValve.throwable(StandardHostValve.java:286)
at
org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:141)
at
org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)
at
org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
at
org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:298)
at
org.apache.jk.server.JkCoyoteHandler.invoke(JkCoyoteHandler.java:190)
at
org.apache.jk.common.HandlerRequest.invoke(HandlerRequest.java:291)
at org.apache.jk.common.ChannelSocket.invoke(ChannelSocket.java:769)
at
org.apache.jk.common.ChannelSocket.processConnection(ChannelSocket.java:698)
at
org.apache.jk.common.ChannelSocket$SocketConnection.runIt(ChannelSocket.java:891)
at
org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:690)
at java.lang.Thread.run(Thread.java:701)
SEVERE: Exception initializing page context
java.lang.IllegalStateException: Cannot create a session after the response
has been committed
at
org.apache.catalina.connector.Request.doGetSession(Request.java:2402)
at
org.apache.catalina.connector.Request.getSession(Request.java:2119)
at
org.apache.catalina.connector.RequestFacade.getSession(RequestFacade.java:833)
at
javax.servlet.http.HttpServletRequestWrapper.getSession(HttpServletRequestWrapper.java:216)
at
org.apache.catalina.core.ApplicationHttpRequest.getSession(ApplicationHttpRequest.java:544)
at
javax.servlet.http.HttpServletRequestWrapper.getSession(HttpServletRequestWrapper.java:216)
at
org.apache.catalina.core.ApplicationHttpRequest.getSession(ApplicationHttpRequest.java:544)
at
javax.servlet.http.HttpServletRequestWrapper.getSession(HttpServletRequestWrapper.java:216)
at
org.apache.catalina.core.ApplicationHttpRequest.getSession(ApplicationHttpRequest.java:544)
at
org.apache.catalina.core.ApplicationHttpRequest.getSession(ApplicationHttpRequest.java:493)
at
org.apache.jasper.runtime.PageContextImpl._initialize(PageContextImpl.java:146)
at
org.apache.jasper.runtime.PageContextImpl.initialize(PageContextImpl.java:124)
at
org.apache.jasper.runtime.JspFactoryImpl.internalGetPageContext(JspFactoryImpl.java:107)
at
org.apache.jasper.runtime.JspFactoryImpl.getPageContext(JspFactoryImpl.java:63)
at
org.apache.jsp.WEB_002dINF.includes.footer_jsp._jspService(Unknown Source)
at
org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:70)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:717)
at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290)
at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
at
org.apache.catalina.core.ApplicationDispatcher.invoke(ApplicationDispatcher.java:646)
at
org.apache.catalina.core.ApplicationDispatcher.doInclude(ApplicationDispatcher.java:551)
at
org.apache.catalina.core.ApplicationDispatcher.include(ApplicationDispatcher.java:488)
at
org.apache.jasper.runtime.JspRuntimeLibrary.include(JspRuntimeLibrary.java:968)
at
org.apache.jsp.WEB_002dINF.decorators.layout_005fc_jsp._jspService(Unknown
Source)
at
org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:70)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:717)
at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290)
at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
at
org.apache.catalina.core.ApplicationDispatcher.invoke(ApplicationDispatcher.java:646)
at
org.apache.catalina.core.ApplicationDispatcher.doInclude(ApplicationDispatcher.java:551)
at
org.apache.catalina.core.ApplicationDispatcher.include(ApplicationDispatcher.java:488)
at
org.apache.jasper.runtime.JspRuntimeLibrary.include(JspRuntimeLibrary.java:968)
at
org.apache.jasper.runtime.PageContextImpl.include(PageContextImpl.java:622)
at
com.opensymphony.module.sitemesh.taglib.page.ApplyDecoratorTag.doEndTag(ApplyDecoratorTag.java:258)
at
org.apache.jsp.WEB_002dINF.pages.common.errors._500_jsp._jspx_meth_page_005fapplyDecorator_005f0(Unknown
Source)
at
org.apache.jsp.WEB_002dINF.pages.common.errors._500_jsp._jspService(Unknown
Source)
at
org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:70)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:717)
at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290)
at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
at
org.apache.catalina.core.ApplicationDispatcher.invoke(ApplicationDispatcher.java:646)
at
org.apache.catalina.core.ApplicationDispatcher.processRequest(ApplicationDispatcher.java:438)
at
org.apache.catalina.core.ApplicationDispatcher.doForward(ApplicationDispatcher.java:374)
at
org.apache.catalina.core.ApplicationDispatcher.forward(ApplicationDispatcher.java:302)
at
org.apache.catalina.core.StandardHostValve.custom(StandardHostValve.java:415)
at
org.apache.catalina.core.StandardHostValve.status(StandardHostValve.java:342)
at
org.apache.catalina.core.StandardHostValve.throwable(StandardHostValve.java:286)
at
org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:141)
at
org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)
at
org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
at
org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:298)
at
org.apache.jk.server.JkCoyoteHandler.invoke(JkCoyoteHandler.java:190)
at
org.apache.jk.common.HandlerRequest.invoke(HandlerRequest.java:291)
at org.apache.jk.common.ChannelSocket.invoke(ChannelSocket.java:769)
at
org.apache.jk.common.ChannelSocket.processConnection(ChannelSocket.java:698)
at
org.apache.jk.common.ChannelSocket$SocketConnection.runIt(ChannelSocket.java:891)
at
org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:690)
at java.lang.Thread.run(Thread.java:701)
I also see lots of AJP timeouts in the apache error log
[Tue Aug 25 04:21:18 2015] [error] (70007)The timeout specified has
expired: proxy: read response failed from (null) (localhost)
[Tue Aug 25 04:21:18 2015] [error] (70007)The timeout specified has
expired: ajp_ilink_receive() can't receive header
[Tue Aug 25 04:21:18 2015] [error] ajp_read_header: ajp_ilink_receive failed
[Tue Aug 25 04:21:18 2015] [error] (70007)The timeout specified has
expired: proxy: read response failed from (null) (localhost)
[Tue Aug 25 04:21:18 2015] [error] (70007)The timeout specified has
expired: ajp_ilink_receive() can't receive header
[Tue Aug 25 04:21:18 2015] [error] ajp_read_header: ajp_ilink_receive failed
[Tue Aug 25 04:21:18 2015] [error] (70007)The timeout specified has
expired: proxy: read response failed from (null) (localhost)
[Tue Aug 25 04:21:18 2015] [error] (70007)The timeout specified has
expired: ajp_ilink_receive() can't receive header
[Tue Aug 25 04:21:18 2015] [error] ajp_read_header: ajp_ilink_receive failed
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/spacewalk-list/attachments/20150826/d113f80f/attachment.htm>
More information about the Spacewalk-list
mailing list