[Spacewalk-list] Spacewalk performance tuning for deployments with 1000+ hosts

Tue Oct 28 16:41:39 UTC 2014

Here are some configurations you might find helpful for tuning
Apache\Tomcat\Java\Networking..
Like others have mentioned.. when you get over 1000+ clients, it's a good
idea to start scaling horizontally with Spacewalk Proxies. We use 4 proxies
in our production environment and are servicing 8000+ clients. We can
actually patch 1600 clients at a time and the GUI is still pretty
responsive. Can't guarantee this will resolve your issue, but this worked
for us.

Add maxThreads to /etc/tomcat6/server.xml

    <Connector port="8080" protocol="HTTP/1.1"
connectionTimeout="20000" redirectPort="8443" URIEncoding="UTF-8"
address="127.0.0.1" maxThreads="1024" maxKeepAliveRequests="1000"/>

    <!-- A "Connector" using the shared thread pool-->

    <!--

    <Connector executor="tomcatThreadPool"

               port="8080" protocol="HTTP/1.1"

               connectionTimeout="20000"

               redirectPort="8443" />

    <!-- Define an AJP 1.3 Connector on port 8009 -->

    <Connector port="8009" protocol="AJP/1.3" redirectPort="8443"
URIEncoding="UTF-8" address="127.0.0.1" maxThreads="1024"/>

    <Connector port="8009" protocol="AJP/1.3" redirectPort="8443"
URIEncoding="UTF-8" address="::1" maxThreads="1024"/>

Tune Apache to service more requests /etc/httpd/conf.d/zz-spacewalk-server.conf

#######################################################

# Authorship and versioning info

# $Author$

# $Date$

# $URL$

# $Rev$

# deployment_location: /etc/httpd/conf.d/

#######################################################

# ** DO NOT EDIT **

# Master configuration file for the rhn_server setup

#

##

## Spacewalk settings

##

<VirtualHost *>

<IfModule mod_jk.c>

    # Inherit the mod_jk settings defined in zz-spacewalk-www.conf

    JkMountCopy On

</IfModule>

<Directory "/var/www/html/*">

        AllowOverride all

</Directory>

RewriteEngine on

RewriteOptions inherit

</VirtualHost>

# Override default httpd prefork settings

<IfModule prefork.c>

StartServers       8

MinSpareServers    400

MaxSpareServers   400

ServerLimit      1024

MaxClients       1024

MaxRequestsPerChild  200

</IfModule>

Include /etc/rhn/satellite-httpd/conf/rhn/rhn_monitoring.conf

 Also added some network tuning to /etc/sysctl.conf

net.ipv4.icmp_echo_ignore_broadcasts = 1

net.ipv4.conf.all.secure_redirects = 0

net.ipv4.tcp_max_syn_backlog = 8192

net.ipv4.conf.default.secure_redirects = 0

net.ipv4.tcp_syncookies = 1

net.ipv4.conf.all.accept_source_route = 0

net.ipv4.conf.all.rp_filter = 1

net.ipv4.conf.all.send_redirects = 0

net.ipv4.conf.default.accept_redirects = 0

net.ipv4.conf.all.accept_redirects = 0

net.ipv4.conf.default.send_redirects = 0

net.core.somaxconn = 1536

net.core.dev_weight = 512

##3x normal for a queue and budget suited to networks greater than 100mbps

net.core.netdev_budget = 10000

net.core.netdev_max_backlog = 30000

Depending on the amount of memory on your Spacewalk server, you may
want to increase your JAVA_OPTS Xms and Xmx settings to something a
little higher. Typically only needed if you are seeing Java Heap out
of memory errors in your Spacewalk logs.

On Tue, Oct 28, 2014 at 11:21 AM, Waldirio Manhães Pinheiro <
waldirio at gmail.com> wrote:

> Dear Krzysztof
>
> Have you checked your numa configuration ?!
>
> Maybe you can customize you environment to use the same bus to application
> / memory.
>
> ______________
> Atenciosamente
> Waldirio
> msn: waldirio at gmail.com
> Skype: waldirio
> Site: www.waldirio.com.br
> Blog: blog.waldirio.com.br
> LinkedIn: http://br.linkedin.com/pub/waldirio-pinheiro/22/b21/646
> PGP: www.waldirio.com.br/public.html
>
> On Tue, Oct 28, 2014 at 2:10 PM, Krzysztof Pawłowski <msciciel at msciciel.eu
> > wrote:
>
>> We have dedicated machine for db pgsql (16GB RAM , 8 cores)  and seperate
>> for spacewalk (16GB RAM, 8 Cores).
>>
>> I think that problem is with enormous number of queries to database.
>> During such request db is not utilized 100% and spacewalk is also not 100%
>> utilized.
>>
>>
>>
>> 2014-10-28 16:25 GMT+01:00 Götz Reinicke - IT Koordinator <
>> goetz.reinicke at filmakademie.de>:
>>
>>> Hi,
>>> Am 28.10.14 um 12:57 schrieb Krzysztof Pawłowski:
>>> > Hi,
>>> > Is there any guide about tuning spacewalk performance ? With every new
>>> > host spacewalk is getting slowly. Using SSM with more than 200-300
>>> hosts
>>> > is impossible due timeouts. It's also not possible to deploy config
>>> > files to all hosts.
>>> > Standard java tuning was done, java gc is not the problem now.
>>> >
>>> > Any suggestions ?
>>>
>>> what server hardware do you use? What is the systemload while performing
>>> that tasks? CPU, RAM, Disksystem, IO, Network .... ?
>>>
>>>
>>>         /Götz
>>>
>>> --
>>> Götz Reinicke
>>> IT-Koordinator
>>>
>>> Tel. +49 7141 969 82 420
>>> E-Mail goetz.reinicke at filmakademie.de
>>>
>>> Filmakademie Baden-Württemberg GmbH
>>> Akademiehof 10
>>> 71638 Ludwigsburg
>>> www.filmakademie.de
>>>
>>> Eintragung Amtsgericht Stuttgart HRB 205016
>>>
>>> Vorsitzender des Aufsichtsrats: Jürgen Walter MdL
>>> Staatssekretär im Ministerium für Wissenschaft,
>>> Forschung und Kunst Baden-Württemberg
>>>
>>> Geschäftsführer: Prof. Thomas Schadt
>>>
>>>
>>> _______________________________________________
>>> Spacewalk-list mailing list
>>> Spacewalk-list at redhat.com
>>> https://www.redhat.com/mailman/listinfo/spacewalk-list
>>>
>>
>>
>> _______________________________________________
>> Spacewalk-list mailing list
>> Spacewalk-list at redhat.com
>> https://www.redhat.com/mailman/listinfo/spacewalk-list
>>
>
>
> _______________________________________________
> Spacewalk-list mailing list
> Spacewalk-list at redhat.com
> https://www.redhat.com/mailman/listinfo/spacewalk-list
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/spacewalk-list/attachments/20141028/62ad8f29/attachment.htm>