[Spacewalk-list] Spacewalk performance tuning for deployments with 1000+ hosts
Boyd, Robert
Robert.Boyd at peoplefluent.com
Mon Nov 24 22:20:48 UTC 2014
Matthew,
I tried these tuning adjustments on my RHEL 6.6 / SW 2.0 Server and it led to some problems. How did you arrive at the numbers you suggested? What size RAM do you have on your master server?
Robert Boyd
Sr. Systems Engineer
PeopleFluent
From: spacewalk-list-bounces at redhat.com [mailto:spacewalk-list-bounces at redhat.com] On Behalf Of Matthew Madey
Sent: Tuesday, October 28, 2014 12:42 PM
To: spacewalk-list at redhat.com
Subject: Re: [Spacewalk-list] Spacewalk performance tuning for deployments with 1000+ hosts
Here are some configurations you might find helpful for tuning Apache\Tomcat\Java\Networking.. Like others have mentioned.. when you get over 1000+ clients, it's a good idea to start scaling horizontally with Spacewalk Proxies. We use 4 proxies in our production environment and are servicing 8000+ clients. We can actually patch 1600 clients at a time and the GUI is still pretty responsive. Can't guarantee this will resolve your issue, but this worked for us.
Add maxThreads to /etc/tomcat6/server.xml
<Connector port="8080" protocol="HTTP/1.1" connectionTimeout="20000" redirectPort="8443" URIEncoding="UTF-8" address="127.0.0.1" maxThreads="1024" maxKeepAliveRequests="1000"/>
<!-- A "Connector" using the shared thread pool-->
<!--
<Connector executor="tomcatThreadPool"
port="8080" protocol="HTTP/1.1"
connectionTimeout="20000"
redirectPort="8443" />
<!-- Define an AJP 1.3 Connector on port 8009 -->
<Connector port="8009" protocol="AJP/1.3" redirectPort="8443" URIEncoding="UTF-8" address="127.0.0.1" maxThreads="1024"/>
<Connector port="8009" protocol="AJP/1.3" redirectPort="8443" URIEncoding="UTF-8" address="::1" maxThreads="1024"/>
Tune Apache to service more requests /etc/httpd/conf.d/zz-spacewalk-server.conf
#######################################################
# Authorship and versioning info
# $Author$
# $Date$
# $URL$
# $Rev$
# deployment_location: /etc/httpd/conf.d/
#######################################################
# ** DO NOT EDIT **
# Master configuration file for the rhn_server setup
#
##
## Spacewalk settings
##
<VirtualHost *>
<IfModule mod_jk.c>
# Inherit the mod_jk settings defined in zz-spacewalk-www.conf
JkMountCopy On
</IfModule>
<Directory "/var/www/html/*">
AllowOverride all
</Directory>
RewriteEngine on
RewriteOptions inherit
</VirtualHost>
# Override default httpd prefork settings
<IfModule prefork.c>
StartServers 8
MinSpareServers 400
MaxSpareServers 400
ServerLimit 1024
MaxClients 1024
MaxRequestsPerChild 200
</IfModule>
Include /etc/rhn/satellite-httpd/conf/rhn/rhn_monitoring.conf
Also added some network tuning to /etc/sysctl.conf
net.ipv4.icmp_echo_ignore_broadcasts = 1
net.ipv4.conf.all.secure_redirects = 0
net.ipv4.tcp_max_syn_backlog = 8192
net.ipv4.conf.default.secure_redirects = 0
net.ipv4.tcp_syncookies = 1
net.ipv4.conf.all.accept_source_route = 0
net.ipv4.conf.all.rp_filter = 1
net.ipv4.conf.all.send_redirects = 0
net.ipv4.conf.default.accept_redirects = 0
net.ipv4.conf.all.accept_redirects = 0
net.ipv4.conf.default.send_redirects = 0
net.core.somaxconn = 1536
net.core.dev_weight = 512
##3x normal for a queue and budget suited to networks greater than 100mbps
net.core.netdev_budget = 10000
net.core.netdev_max_backlog = 30000
Depending on the amount of memory on your Spacewalk server, you may want to increase your JAVA_OPTS Xms and Xmx settings to something a little higher. Typically only needed if you are seeing Java Heap out of memory errors in your Spacewalk logs.
On Tue, Oct 28, 2014 at 11:21 AM, Waldirio Manhães Pinheiro <waldirio at gmail.com<mailto:waldirio at gmail.com>> wrote:
Dear Krzysztof
Have you checked your numa configuration ?!
Maybe you can customize you environment to use the same bus to application / memory.
______________
Atenciosamente
Waldirio
msn: waldirio at gmail.com<mailto:waldirio at gmail.com>
Skype: waldirio
Site: www.waldirio.com.br<http://www.waldirio.com.br>
Blog: blog.waldirio.com.br<http://blog.waldirio.com.br>
LinkedIn: http://br.linkedin.com/pub/waldirio-pinheiro/22/b21/646
PGP: www.waldirio.com.br/public.html<http://www.waldirio.com.br/public.html>
On Tue, Oct 28, 2014 at 2:10 PM, Krzysztof Pawłowski <msciciel at msciciel.eu<mailto:msciciel at msciciel.eu>> wrote:
We have dedicated machine for db pgsql (16GB RAM , 8 cores) and seperate for spacewalk (16GB RAM, 8 Cores).
I think that problem is with enormous number of queries to database. During such request db is not utilized 100% and spacewalk is also not 100% utilized.
2014-10-28 16:25 GMT+01:00 Götz Reinicke - IT Koordinator <goetz.reinicke at filmakademie.de<mailto:goetz.reinicke at filmakademie.de>>:
Hi,
Am 28.10.14 um 12:57 schrieb Krzysztof Pawłowski:
> Hi,
> Is there any guide about tuning spacewalk performance ? With every new
> host spacewalk is getting slowly. Using SSM with more than 200-300 hosts
> is impossible due timeouts. It's also not possible to deploy config
> files to all hosts.
> Standard java tuning was done, java gc is not the problem now.
>
> Any suggestions ?
what server hardware do you use? What is the systemload while performing
that tasks? CPU, RAM, Disksystem, IO, Network .... ?
/Götz
--
Götz Reinicke
IT-Koordinator
Tel. +49 7141 969 82 420<tel:%2B49%207141%20969%2082%20420>
E-Mail goetz.reinicke at filmakademie.de<mailto:goetz.reinicke at filmakademie.de>
Filmakademie Baden-Württemberg GmbH
Akademiehof 10
71638 Ludwigsburg
www.filmakademie.de<http://www.filmakademie.de>
Eintragung Amtsgericht Stuttgart HRB 205016
Vorsitzender des Aufsichtsrats: Jürgen Walter MdL
Staatssekretär im Ministerium für Wissenschaft,
Forschung und Kunst Baden-Württemberg
Geschäftsführer: Prof. Thomas Schadt
_______________________________________________
Spacewalk-list mailing list
Spacewalk-list at redhat.com<mailto:Spacewalk-list at redhat.com>
https://www.redhat.com/mailman/listinfo/spacewalk-list
_______________________________________________
Spacewalk-list mailing list
Spacewalk-list at redhat.com<mailto:Spacewalk-list at redhat.com>
https://www.redhat.com/mailman/listinfo/spacewalk-list
_______________________________________________
Spacewalk-list mailing list
Spacewalk-list at redhat.com<mailto:Spacewalk-list at redhat.com>
https://www.redhat.com/mailman/listinfo/spacewalk-list
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/spacewalk-list/attachments/20141124/41396c86/attachment.htm>
More information about the Spacewalk-list
mailing list