[Spacewalk-list] Spacewalk CPU lack / Scaling

[Matthew Madey]

We had a similar issue with Tomcat.. our Spacewalk server has 8000+ clients
and is now running smoothly after some performance tuning. See the code
snippets below, hopefully they'll help you out.


Add maxThreads to /etc/tomcat6/server.xml

    <Connector port="8080" protocol="HTTP/1.1"
connectionTimeout="20000" redirectPort="8443" URIEncoding="UTF-8"
address="127.0.0.1" maxThreads="1024" maxKeepAliveRequests="1000"/>
    <!-- A "Connector" using the shared thread pool-->
    <!--
    <Connector executor="tomcatThreadPool"
               port="8080" protocol="HTTP/1.1"
               connectionTimeout="20000"
               redirectPort="8443" />


    <!-- Define an AJP 1.3 Connector on port 8009 -->
    <Connector port="8009" protocol="AJP/1.3" redirectPort="8443"
URIEncoding="UTF-8" address="127.0.0.1" maxThreads="1024"/>

    <Connector port="8009" protocol="AJP/1.3" redirectPort="8443"
URIEncoding="UTF-8" address="::1" maxThreads="1024"/>






Tune Apache to service more requests
/etc/httpd/conf.d/zz-spacewalk-server.conf


#######################################################
# Authorship and versioning info
# $Author$
# $Date$
# $URL$
# $Rev$
# deployment_location: /etc/httpd/conf.d/
#######################################################
# ** DO NOT EDIT **
# Master configuration file for the rhn_server setup
#

##
## Spacewalk settings
##

<VirtualHost *>

<IfModule mod_jk.c>
    # Inherit the mod_jk settings defined in zz-spacewalk-www.conf
    JkMountCopy On
</IfModule>

<Directory "/var/www/html/*">
        AllowOverride all
</Directory>

RewriteEngine on
RewriteOptions inherit
</VirtualHost>

# Override default httpd prefork settings
<IfModule prefork.c>
StartServers       8
MinSpareServers    400
MaxSpareServers   400
ServerLimit      1024
MaxClients       1024
MaxRequestsPerChild  200
</IfModule>

Include /etc/rhn/satellite-httpd/conf/rhn/rhn_monitoring.conf







Also added some network tuning to /etc/sysctl.conf

net.ipv4.icmp_echo_ignore_broadcasts = 1
net.ipv4.conf.all.secure_redirects = 0
net.ipv4.tcp_max_syn_backlog = 8192
net.ipv4.conf.default.secure_redirects = 0
net.ipv4.tcp_syncookies = 1
net.ipv4.conf.all.accept_source_route = 0
net.ipv4.conf.all.rp_filter = 1
net.ipv4.conf.all.send_redirects = 0
net.ipv4.conf.default.accept_redirects = 0
net.ipv4.conf.all.accept_redirects = 0
net.ipv4.conf.default.send_redirects = 0

net.core.somaxconn = 1536

net.core.dev_weight = 512

##3x normal for a queue and budget suited to networks greater than 100mbps
net.core.netdev_budget = 10000
net.core.netdev_max_backlog = 30000



On Fri, Oct 10, 2014 at 4:35 AM, Michael Mraka <michael.mraka at redhat.com>
wrote:

> Nahelou Sébastien wrote:
> % Hi Spacewalk lovers
>
> Hello Sebastian,
>
> % Since we upgrade from 2.0 to 2.2, we encounter a capacity problem due to
> CPU usage overall.
> % When users load a system page in the GUI, the tomcat process grow to 90%
> usage. Api and GUI become unavailable or very slow.
>
> Are you able to identify which page causes such high load?
>
> % Our architecture is as following:
> % 5 rhn proxies
> % LVS Master/slave mode
> % 1 front master: SERVICES="jabberd $TOMCAT httpd osa-dispatcher
> Monitoring MonitoringScout rhn-search cobblerd taskomatic"
> % 1 front slave: SERVICES="jabberd $TOMCAT httpd osa-dispatcher Monitoring
> MonitoringScout "
> % 1 cluster PGSQL
> %
> % Our masters have 2* Intel(R) Xeon(R) CPU E5-2609 / 92 RAM / raid 1 using
> CentOS 6.5.
> %
> % We have about 6000 systems registered,  about 40 organizations and more
> than 600 channels (including cloned)
> %
> % My questions are:
> % - Is it possible to scale spacewalk horizontally ? By changing lvs
> configuration to Active/Active ? By using HAProxy ? other ?
>
> We've never tested Active/Active cluster. I dare to say it won't work.
> Our recomendation is to use one Spacewalk Proxy for every 5000 systems.
>
> % - Is it possible to tune tomcat better than adding memory ? (we use
> actually -Xms8196m -Xmx16384m)
>
> I don't know. Frankly I've never heard a report about necessity to do so.
>
> % - What is the maximum number of systems that we can register in 1
> organisation ? (Actually our max seems to be 1700)
>
> We know about installations with tens to hundereds of thousands systems.
>
> % - Is there a maximum number of systems that we can register in a
> Spacewalk cluster ?
>
> No, it's limmited only by numbers in your spacewalk certificate.
> Default certificate is limited to 20000. If you need more you can
> generate your own.
>
> % Thank in advance
> %
> % Sébastien NAHELOU
> % System engineer
> % Worldline - Seclin - France
>
>
> Regards,
>
> --
> Michael Mráka
> Satellite Engineering, Red Hat
>
> _______________________________________________
> Spacewalk-list mailing list
> Spacewalk-list at redhat.com
> https://www.redhat.com/mailman/listinfo/spacewalk-list
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/spacewalk-list/attachments/20141010/ed8f2179/attachment.htm>