<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40"><head><meta http-equiv=Content-Type content="text/html; charset=utf-8"><meta name=Generator content="Microsoft Word 12 (filtered medium)"><style></style></head><body lang=EN-US link=blue vlink=purple><div class=WordSection1>Matthew,<o:p></o:p><o:p> </o:p>I tried these tuning adjustments on my RHEL 6.6 / SW 2.0 Server and it led to some problems. How did you arrive at the numbers you suggested? What size RAM do you have on your master server?<o:p></o:p><o:p> </o:p>Robert Boyd<o:p></o:p>Sr. Systems Engineer <o:p></o:p>PeopleFluent<o:p></o:p><o:p> </o:p><div style='border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0in 0in 0in'>From: spacewalk-list-bounces@redhat.com [mailto:spacewalk-list-bounces@redhat.com] On Behalf Of Matthew Madey Sent: Tuesday, October 28, 2014 12:42 PM To: spacewalk-list@redhat.com Subject: Re: [Spacewalk-list] Spacewalk performance tuning for deployments with 1000+ hosts<o:p></o:p></div><o:p> </o:p><div><div><o:p> </o:p></div>Here are some configurations you might find helpful for tuning Apache\Tomcat\Java\Networking.. Like others have mentioned.. when you get over 1000+ clients, it's a good idea to start scaling horizontally with Spacewalk Proxies. We use 4 proxies in our production environment and are servicing 8000+ clients. We can actually patch 1600 clients at a time and the GUI is still pretty responsive. Can't guarantee this will resolve your issue, but this worked for us. <o:p></o:p><div><o:p> </o:p></div><div>Add maxThreads to /etc/tomcat6/server.xml<o:p></o:p><div> <o:p></o:p></div><div><pre style='white-space:pre-wrap;word-wrap:break-word'> <Connector port="8080" protocol="HTTP/1.1" connectionTimeout="20000" redirectPort="8443" URIEncoding="UTF-8" address="127.0.0.1" maxThreads="1024" maxKeepAliveRequests="1000"/><o:p></o:p></pre><pre style='white-space:pre-wrap'> <o:p></o:p></pre><pre style='white-space:pre-wrap'> <o:p></o:p></pre><pre style='white-space:pre-wrap'> <Connector port="8009" protocol="AJP/1.3" redirectPort="8443" URIEncoding="UTF-8" address="127.0.0.1" maxThreads="1024"/><o:p></o:p></pre><pre style='white-space:pre-wrap'> <o:p></o:p></pre><pre style='white-space:pre-wrap'> <Connector port="8009" protocol="AJP/1.3" redirectPort="8443" URIEncoding="UTF-8" address="::1" maxThreads="1024"/><o:p></o:p></pre><div><pre style='white-space:pre-wrap'> <o:p></o:p></pre></div><div><pre style='white-space:pre-wrap'> <o:p></o:p></pre></div><div><pre style='white-space:pre-wrap'>Tune Apache to service more requests /etc/httpd/conf.d/zz-spacewalk-server.conf<o:p></o:p></pre></div><div><pre style='white-space:pre-wrap'> <o:p></o:p></pre></div><div><pre style='white-space:pre-wrap'> <o:p></o:p></pre></div><div><pre style='white-space:pre-wrap;word-wrap:break-word'>#######################################################<o:p></o:p></pre><pre style='white-space:pre-wrap'># Authorship and versioning info<o:p></o:p></pre><pre style='white-space:pre-wrap'># $Author$<o:p></o:p></pre><pre style='white-space:pre-wrap'># $Date$<o:p></o:p></pre><pre style='white-space:pre-wrap'># $URL$<o:p></o:p></pre><pre style='white-space:pre-wrap'># $Rev$<o:p></o:p></pre><pre style='white-space:pre-wrap'># deployment_location: /etc/httpd/conf.d/<o:p></o:p></pre><pre style='white-space:pre-wrap'>#######################################################<o:p></o:p></pre><pre style='white-space:pre-wrap'># ** DO NOT EDIT **<o:p></o:p></pre><pre style='white-space:pre-wrap'># Master configuration file for the rhn_server setup<o:p></o:p></pre><pre style='white-space:pre-wrap'>#<o:p></o:p></pre><pre style='white-space:pre-wrap'> <o:p></o:p></pre><pre style='white-space:pre-wrap'>##<o:p></o:p></pre><pre style='white-space:pre-wrap'>## Spacewalk settings<o:p></o:p></pre><pre style='white-space:pre-wrap'>##<o:p></o:p></pre><pre style='white-space:pre-wrap'> <o:p></o:p></pre><pre style='white-space:pre-wrap'><VirtualHost *><o:p></o:p></pre><pre style='white-space:pre-wrap'> <o:p></o:p></pre><pre style='white-space:pre-wrap'><IfModule mod_jk.c><o:p></o:p></pre><pre style='white-space:pre-wrap'> # Inherit the mod_jk settings defined in zz-spacewalk-www.conf<o:p></o:p></pre><pre style='white-space:pre-wrap'> JkMountCopy On<o:p></o:p></pre><pre style='white-space:pre-wrap'></IfModule><o:p></o:p></pre><pre style='white-space:pre-wrap'> <o:p></o:p></pre><pre style='white-space:pre-wrap'><Directory "/var/www/html/*"><o:p></o:p></pre><pre style='white-space:pre-wrap'> AllowOverride all<o:p></o:p></pre><pre style='white-space:pre-wrap'></Directory><o:p></o:p></pre><pre style='white-space:pre-wrap'> <o:p></o:p></pre><pre style='white-space:pre-wrap'>RewriteEngine on<o:p></o:p></pre><pre style='white-space:pre-wrap'>RewriteOptions inherit<o:p></o:p></pre><pre style='white-space:pre-wrap'></VirtualHost><o:p></o:p></pre><pre style='white-space:pre-wrap'> <o:p></o:p></pre><pre style='white-space:pre-wrap'># Override default httpd prefork settings<o:p></o:p></pre><pre style='white-space:pre-wrap'><IfModule prefork.c><o:p></o:p></pre><pre style='white-space:pre-wrap'>StartServers 8<o:p></o:p></pre><pre style='white-space:pre-wrap'>MinSpareServers 400<o:p></o:p></pre><pre style='white-space:pre-wrap'>MaxSpareServers 400<o:p></o:p></pre><pre style='white-space:pre-wrap'>ServerLimit 1024<o:p></o:p></pre><pre style='white-space:pre-wrap'>MaxClients 1024<o:p></o:p></pre><pre style='white-space:pre-wrap'>MaxRequestsPerChild 200<o:p></o:p></pre><pre style='white-space:pre-wrap'></IfModule><o:p></o:p></pre><pre style='white-space:pre-wrap'> <o:p></o:p></pre><pre style='white-space:pre-wrap'>Include /etc/rhn/satellite-httpd/conf/rhn/rhn_monitoring.conf<o:p></o:p></pre><div><pre style='white-space:pre-wrap'> <o:p></o:p></pre></div><div><pre style='white-space:pre-wrap'> <o:p></o:p></pre></div><div><pre style='white-space:pre-wrap'> <o:p></o:p></pre></div></div><div><pre style='white-space:pre-wrap'> Also added some network tuning to /etc/sysctl.conf<o:p></o:p></pre></div><div><pre style='white-space:pre-wrap'> <o:p></o:p></pre></div><div><pre style='white-space:pre-wrap'>net.ipv4.icmp_echo_ignore_broadcasts = 1<o:p></o:p></pre><pre style='white-space:pre-wrap'>net.ipv4.conf.all.secure_redirects = 0<o:p></o:p></pre><pre style='white-space:pre-wrap'>net.ipv4.tcp_max_syn_backlog = 8192<o:p></o:p></pre><pre style='white-space:pre-wrap'>net.ipv4.conf.default.secure_redirects = 0<o:p></o:p></pre><pre style='white-space:pre-wrap'>net.ipv4.tcp_syncookies = 1<o:p></o:p></pre><pre style='white-space:pre-wrap'>net.ipv4.conf.all.accept_source_route = 0<o:p></o:p></pre><pre style='white-space:pre-wrap'>net.ipv4.conf.all.rp_filter = 1<o:p></o:p></pre><pre style='white-space:pre-wrap'>net.ipv4.conf.all.send_redirects = 0<o:p></o:p></pre><pre style='white-space:pre-wrap'>net.ipv4.conf.default.accept_redirects = 0<o:p></o:p></pre><pre style='white-space:pre-wrap'>net.ipv4.conf.all.accept_redirects = 0<o:p></o:p></pre><pre style='white-space:pre-wrap'>net.ipv4.conf.default.send_redirects = 0<o:p></o:p></pre><pre style='white-space:pre-wrap;word-wrap:break-word'>net.core.somaxconn = 1536<o:p></o:p></pre><pre style='white-space:pre-wrap;word-wrap:break-word'>net.core.dev_weight = 512<o:p></o:p></pre><pre style='white-space:pre-wrap;word-wrap:break-word'>##3x normal for a queue and budget suited to networks greater than 100mbps<o:p></o:p></pre><pre style='white-space:pre-wrap'>net.core.netdev_budget = 10000<o:p></o:p></pre><pre style='white-space:pre-wrap'>net.core.netdev_max_backlog = 30000<o:p></o:p></pre><pre style='white-space:pre-wrap'><o:p> </o:p></pre><pre style='white-space:pre-wrap'><o:p> </o:p></pre><pre style='white-space:pre-wrap'>Depending on the amount of memory on your Spacewalk server, you may want to increase your JAVA_OPTS Xms and Xmx settings to something a little higher. Typically only needed if you are seeing Java Heap out of memory errors in your Spacewalk logs.<o:p></o:p></pre></div><div><o:p> </o:p></div></div></div></div><div><o:p> </o:p><div>On Tue, Oct 28, 2014 at 11:21 AM, Waldirio Manhães Pinheiro <<a href="mailto:waldirio@gmail.com" target="_blank">waldirio@gmail.com</a>> wrote:<o:p></o:p><div>Dear Krzysztof<o:p></o:p><div><o:p> </o:p></div><div>Have you checked your numa configuration ?!<o:p></o:p></div><div><o:p> </o:p></div><div>Maybe you can customize you environment to use the same bus to application / memory.<o:p></o:p></div></div><div> <o:p></o:p><div><div>______________ Atenciosamente Waldirio msn: <a href="mailto:waldirio@gmail.com" target="_blank">waldirio@gmail.com</a> Skype: waldirio Site: <a href="http://www.waldirio.com.br" target="_blank">www.waldirio.com.br</a> Blog: <a href="http://blog.waldirio.com.br" target="_blank">blog.waldirio.com.br</a><o:p></o:p></div><div>LinkedIn: <a href="http://br.linkedin.com/pub/waldirio-pinheiro/22/b21/646" target="_blank">http://br.linkedin.com/pub/waldirio-pinheiro/22/b21/646</a> PGP: <a href="http://www.waldirio.com.br/public.html" target="_blank">www.waldirio.com.br/public.html</a><o:p></o:p></div></div><div><div><o:p> </o:p><div>On Tue, Oct 28, 2014 at 2:10 PM, Krzysztof Pawłowski <<a href="mailto:msciciel@msciciel.eu" target="_blank">msciciel@msciciel.eu</a>> wrote:<o:p></o:p><div><div><div>We have dedicated machine for db pgsql (16GB RAM , 8 cores) and seperate for spacewalk (16GB RAM, 8 Cores).<o:p></o:p></div><o:p> </o:p></div>I think that problem is with enormous number of queries to database. During such request db is not utilized 100% and spacewalk is also not 100% utilized.<o:p></o:p><div><div><o:p> </o:p></div></div></div><div><o:p> </o:p><div><div><div>2014-10-28 16:25 GMT+01:00 Götz Reinicke - IT Koordinator <<a href="mailto:goetz.reinicke@filmakademie.de" target="_blank">goetz.reinicke@filmakademie.de</a>>:<o:p></o:p></div></div><blockquote style='border:none;border-left:solid #CCCCCC 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-right:0in'><div><div>Hi, Am 28.10.14 um 12:57 schrieb Krzysztof Pawłowski:<o:p></o:p><div><div>> Hi, > Is there any guide about tuning spacewalk performance ? With every new > host spacewalk is getting slowly. Using SSM with more than 200-300 hosts > is impossible due timeouts. It's also not possible to deploy config > files to all hosts. > Standard java tuning was done, java gc is not the problem now. > > Any suggestions ?<o:p></o:p></div></div>what server hardware do you use? What is the systemload while performing that tasks? CPU, RAM, Disksystem, IO, Network .... ? /Götz -- Götz Reinicke IT-Koordinator Tel. <a href="tel:%2B49%207141%20969%2082%20420" target="_blank">+49 7141 969 82 420</a> E-Mail <a href="mailto:goetz.reinicke@filmakademie.de" target="_blank">goetz.reinicke@filmakademie.de</a> Filmakademie Baden-Württemberg GmbH Akademiehof 10 71638 Ludwigsburg <a href="http://www.filmakademie.de" target="_blank">www.filmakademie.de</a> Eintragung Amtsgericht Stuttgart HRB 205016 Vorsitzender des Aufsichtsrats: Jürgen Walter MdL Staatssekretär im Ministerium für Wissenschaft, Forschung und Kunst Baden-Württemberg Geschäftsführer: Prof. Thomas Schadt <o:p></o:p></div></div>_______________________________________________ Spacewalk-list mailing list <a href="mailto:Spacewalk-list@redhat.com" target="_blank">Spacewalk-list@redhat.com</a> <a href="https://www.redhat.com/mailman/listinfo/spacewalk-list" target="_blank">https://www.redhat.com/mailman/listinfo/spacewalk-list</a><o:p></o:p></blockquote></div><o:p> </o:p></div> _______________________________________________ Spacewalk-list mailing list <a href="mailto:Spacewalk-list@redhat.com" target="_blank">Spacewalk-list@redhat.com</a> <a href="https://www.redhat.com/mailman/listinfo/spacewalk-list" target="_blank">https://www.redhat.com/mailman/listinfo/spacewalk-list</a><o:p></o:p></div><o:p> </o:p></div></div></div> _______________________________________________ Spacewalk-list mailing list <a href="mailto:Spacewalk-list@redhat.com">Spacewalk-list@redhat.com</a> <a href="https://www.redhat.com/mailman/listinfo/spacewalk-list" target="_blank">https://www.redhat.com/mailman/listinfo/spacewalk-list</a><o:p></o:p></div><o:p> </o:p></div></div></body></html>