[Spacewalk-list] Spacewalk CPU lack / Scaling

Mon Oct 13 07:43:20 UTC 2014

Thank you so much for your answers.

Hangs happens when we load the "all systems" page when organisation. I will try your configuration tuning.

Last question, what do you preconize for PGSQL tuning ? Particularly, How many maximum connections do you configure ?

Thank again
Sebastien

De : spacewalk-list-bounces at redhat.com [mailto:spacewalk-list-bounces at redhat.com] De la part de Matthew Madey
Envoyé : vendredi 10 octobre 2014 18:25
À : spacewalk-list at redhat.com
Objet : Re: [Spacewalk-list] Spacewalk CPU lack / Scaling

We had a similar issue with Tomcat.. our Spacewalk server has 8000+ clients and is now running smoothly after some performance tuning. See the code snippets below, hopefully they'll help you out.

Add maxThreads to /etc/tomcat6/server.xml

    <Connector port="8080" protocol="HTTP/1.1" connectionTimeout="20000" redirectPort="8443" URIEncoding="UTF-8" address="127.0.0.1" maxThreads="1024" maxKeepAliveRequests="1000"/>

    <!-- A "Connector" using the shared thread pool-->

    <!--

    <Connector executor="tomcatThreadPool"

               port="8080" protocol="HTTP/1.1"

               connectionTimeout="20000"

               redirectPort="8443" />

    <!-- Define an AJP 1.3 Connector on port 8009 -->

    <Connector port="8009" protocol="AJP/1.3" redirectPort="8443" URIEncoding="UTF-8" address="127.0.0.1" maxThreads="1024"/>

    <Connector port="8009" protocol="AJP/1.3" redirectPort="8443" URIEncoding="UTF-8" address="::1" maxThreads="1024"/>

Tune Apache to service more requests /etc/httpd/conf.d/zz-spacewalk-server.conf

#######################################################

# Authorship and versioning info

# $Author$

# $Date$

# $URL$

# $Rev$

# deployment_location: /etc/httpd/conf.d/

#######################################################

# ** DO NOT EDIT **

# Master configuration file for the rhn_server setup

#

##

## Spacewalk settings

##

<VirtualHost *>

<IfModule mod_jk.c>

    # Inherit the mod_jk settings defined in zz-spacewalk-www.conf

    JkMountCopy On

</IfModule>

<Directory "/var/www/html/*">

        AllowOverride all

</Directory>

RewriteEngine on

RewriteOptions inherit

</VirtualHost>

# Override default httpd prefork settings

<IfModule prefork.c>

StartServers       8

MinSpareServers    400

MaxSpareServers   400

ServerLimit      1024

MaxClients       1024

MaxRequestsPerChild  200

</IfModule>

Include /etc/rhn/satellite-httpd/conf/rhn/rhn_monitoring.conf

Also added some network tuning to /etc/sysctl.conf

net.ipv4.icmp_echo_ignore_broadcasts = 1

net.ipv4.conf.all.secure_redirects = 0

net.ipv4.tcp_max_syn_backlog = 8192

net.ipv4.conf.default.secure_redirects = 0

net.ipv4.tcp_syncookies = 1

net.ipv4.conf.all.accept_source_route = 0

net.ipv4.conf.all.rp_filter = 1

net.ipv4.conf.all.send_redirects = 0

net.ipv4.conf.default.accept_redirects = 0

net.ipv4.conf.all.accept_redirects = 0

net.ipv4.conf.default.send_redirects = 0

net.core.somaxconn = 1536

net.core.dev_weight = 512

##3x normal for a queue and budget suited to networks greater than 100mbps

net.core.netdev_budget = 10000

net.core.netdev_max_backlog = 30000

On Fri, Oct 10, 2014 at 4:35 AM, Michael Mraka <michael.mraka at redhat.com<mailto:michael.mraka at redhat.com>> wrote:
Nahelou Sébastien wrote:
% Hi Spacewalk lovers

Hello Sebastian,

% Since we upgrade from 2.0 to 2.2, we encounter a capacity problem due to CPU usage overall.
% When users load a system page in the GUI, the tomcat process grow to 90% usage. Api and GUI become unavailable or very slow.

Are you able to identify which page causes such high load?

% Our architecture is as following:
% 5 rhn proxies
% LVS Master/slave mode
% 1 front master: SERVICES="jabberd $TOMCAT httpd osa-dispatcher Monitoring MonitoringScout rhn-search cobblerd taskomatic"
% 1 front slave: SERVICES="jabberd $TOMCAT httpd osa-dispatcher Monitoring MonitoringScout "
% 1 cluster PGSQL
%
% Our masters have 2* Intel(R) Xeon(R) CPU E5-2609 / 92 RAM / raid 1 using CentOS 6.5.
%
% We have about 6000 systems registered,  about 40 organizations and more than 600 channels (including cloned)
%
% My questions are:
% - Is it possible to scale spacewalk horizontally ? By changing lvs configuration to Active/Active ? By using HAProxy ? other ?

We've never tested Active/Active cluster. I dare to say it won't work.
Our recomendation is to use one Spacewalk Proxy for every 5000 systems.

% - Is it possible to tune tomcat better than adding memory ? (we use actually -Xms8196m -Xmx16384m)

I don't know. Frankly I've never heard a report about necessity to do so.

% - What is the maximum number of systems that we can register in 1 organisation ? (Actually our max seems to be 1700)

We know about installations with tens to hundereds of thousands systems.

% - Is there a maximum number of systems that we can register in a Spacewalk cluster ?

No, it's limmited only by numbers in your spacewalk certificate.
Default certificate is limited to 20000. If you need more you can
generate your own.

% Thank in advance
%
% Sébastien NAHELOU
% System engineer
% Worldline - Seclin - France

Regards,

--
Michael Mráka
Satellite Engineering, Red Hat

_______________________________________________
Spacewalk-list mailing list
Spacewalk-list at redhat.com<mailto:Spacewalk-list at redhat.com>
https://www.redhat.com/mailman/listinfo/spacewalk-list

________________________________

Ce message et les pièces jointes sont confidentiels et réservés à l'usage exclusif de ses destinataires. Il peut également être protégé par le secret professionnel. Si vous recevez ce message par erreur, merci d'en avertir immédiatement l'expéditeur et de le détruire. L'intégrité du message ne pouvant être assurée sur Internet, la responsabilité de Worldline ne pourra être recherchée quant au contenu de ce message. Bien que les meilleurs efforts soient faits pour maintenir cette transmission exempte de tout virus, l'expéditeur ne donne aucune garantie à cet égard et sa responsabilité ne saurait être recherchée pour tout dommage résultant d'un virus transmis.

This e-mail and the documents attached are confidential and intended solely for the addressee; it may also be privileged. If you receive this e-mail in error, please notify the sender immediately and destroy it. As its integrity cannot be secured on the Internet, the Worldline liability cannot be triggered for the message content. Although the sender endeavours to maintain a computer virus-free network, the sender does not warrant that this transmission is virus-free and will not be liable for any damages resulting from any virus transmitted.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/spacewalk-list/attachments/20141013/174b4b1e/attachment.htm>