<div dir="ltr">Hello,<div>I made the same conclusion one year ago about the "All System" performance:</div><div><a href="https://www.redhat.com/archives/spacewalk-list/2014-March/msg00072.html">https://www.redhat.com/archives/spacewalk-list/2014-March/msg00072.html</a> </div><div> </div><div>I'm happy to see that someone else noticed that this page is slower on his set up too!</div><div> </div><div>Pierre</div></div><div class="gmail_extra"> <div class="gmail_quote">2014-10-14 8:56 GMT+02:00 Nahelou Sébastien <<a href="mailto:sebastien.nahelou@worldline.com" target="_blank">sebastien.nahelou@worldline.com</a>>: <blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"> <div lang="FR" link="blue" vlink="purple"> <div> Ok, like us. Thank you for your help. Sebastien De : <a href="mailto:spacewalk-list-bounces@redhat.com" target="_blank">spacewalk-list-bounces@redhat.com</a> [mailto:<a href="mailto:spacewalk-list-bounces@redhat.com" target="_blank">spacewalk-list-bounces@redhat.com</a>] De la part de Matthew Madey Envoyé : lundi 13 octobre 2014 20:30<div><div class="h5"> À : <a href="mailto:spacewalk-list@redhat.com" target="_blank">spacewalk-list@redhat.com</a> Objet : Re: [Spacewalk-list] Spacewalk CPU lack / Scaling</div></div><div><div class="h5"> <div> We use Oracle, so I can't comment on PGSQL directly.. however I believe we have our concurrent connection limit set to around 2000. </div> <div> <div> On Mon, Oct 13, 2014 at 2:43 AM, Nahelou Sébastien <<a href="mailto:sebastien.nahelou@worldline.com" target="_blank">sebastien.nahelou@worldline.com</a>> wrote: <div> <div> Thank you so much for your answers. Hangs happens when we load the "all systems" page when organisation. I will try your configuration tuning. Last question, what do you preconize for PGSQL tuning ? Particularly, How many maximum connections do you configure ? Thank again Sebastien De : <a href="mailto:spacewalk-list-bounces@redhat.com" target="_blank">spacewalk-list-bounces@redhat.com</a> [mailto:<a href="mailto:spacewalk-list-bounces@redhat.com" target="_blank">spacewalk-list-bounces@redhat.com</a>] De la part de Matthew Madey Envoyé : vendredi 10 octobre 2014 18:25 À : <a href="mailto:spacewalk-list@redhat.com" target="_blank">spacewalk-list@redhat.com</a> Objet : Re: [Spacewalk-list] Spacewalk CPU lack / Scaling <div> <div> <div> We had a similar issue with Tomcat.. our Spacewalk server has 8000+ clients and is now running smoothly after some performance tuning. See the code snippets below, hopefully they'll help you out. <div> </div> <div> </div> <div> Add maxThreads to /etc/tomcat6/server.xml <div> </div> <div> <pre style="word-wrap:break-word;white-space:pre-wrap"> <Connector port="8080" protocol="HTTP/1.1" connectionTimeout="20000" redirectPort="8443" URIEncoding="UTF-8" address="127.0.0.1" maxThreads="1024" maxKeepAliveRequests="1000"/></pre> <pre> </pre> <pre> </pre> <pre> <Connector port="8009" protocol="AJP/1.3" redirectPort="8443" URIEncoding="UTF-8" address="127.0.0.1" maxThreads="1024"/></pre> <pre> </pre> <pre> <Connector port="8009" protocol="AJP/1.3" redirectPort="8443" URIEncoding="UTF-8" address="::1" maxThreads="1024"/></pre> <div> <pre> </pre> </div> <div> <pre> </pre> </div> <div> <pre> </pre> </div> <div> <pre> </pre> </div> <div> <pre> </pre> </div> <div> <pre>Tune Apache to service more requests /etc/httpd/conf.d/zz-spacewalk-server.conf</pre> </div> <div> <pre> </pre> </div> <div> <pre> </pre> </div> <div> <pre style="word-wrap:break-word;white-space:pre-wrap">#######################################################</pre> <pre># Authorship and versioning info</pre> <pre># $Author$</pre> <pre># $Date$</pre> <pre># $URL$</pre> <pre># $Rev$</pre> <pre># deployment_location: /etc/httpd/conf.d/</pre> <pre>#######################################################</pre> <pre># ** DO NOT EDIT **</pre> <pre># Master configuration file for the rhn_server setup</pre> <pre>#</pre> <pre> </pre> <pre>##</pre> <pre>## Spacewalk settings</pre> <pre>##</pre> <pre> </pre> <pre><VirtualHost *></pre> <pre> </pre> <pre><IfModule mod_jk.c></pre> <pre> # Inherit the mod_jk settings defined in zz-spacewalk-www.conf</pre> <pre> JkMountCopy On</pre> <pre></IfModule></pre> <pre> </pre> <pre><Directory "/var/www/html/*"></pre> <pre> AllowOverride all</pre> <pre></Directory></pre> <pre> </pre> <pre>RewriteEngine on</pre> <pre>RewriteOptions inherit</pre> <pre></VirtualHost></pre> <pre> </pre> <pre># Override default httpd prefork settings</pre> <pre><IfModule prefork.c></pre> <pre>StartServers 8</pre> <pre>MinSpareServers 400</pre> <pre>MaxSpareServers 400</pre> <pre>ServerLimit 1024</pre> <pre>MaxClients 1024</pre> <pre>MaxRequestsPerChild 200</pre> <pre></IfModule></pre> <pre> </pre> <pre>Include /etc/rhn/satellite-httpd/conf/rhn/rhn_monitoring.conf</pre> <div> <pre> </pre> </div> <div> <pre> </pre> </div> <div> <pre> </pre> </div> </div> <div> <pre> </pre> </div> <div> <pre> </pre> </div> <div> <pre> </pre> </div> <div> <pre>Also added some network tuning to /etc/sysctl.conf</pre> </div> <div> <pre> </pre> </div> <div> <pre>net.ipv4.icmp_echo_ignore_broadcasts = 1</pre> <pre>net.ipv4.conf.all.secure_redirects = 0</pre> <pre>net.ipv4.tcp_max_syn_backlog = 8192</pre> <pre>net.ipv4.conf.default.secure_redirects = 0</pre> <pre>net.ipv4.tcp_syncookies = 1</pre> <pre>net.ipv4.conf.all.accept_source_route = 0</pre> <pre>net.ipv4.conf.all.rp_filter = 1</pre> <pre>net.ipv4.conf.all.send_redirects = 0</pre> <pre>net.ipv4.conf.default.accept_redirects = 0</pre> <pre>net.ipv4.conf.all.accept_redirects = 0</pre> <pre>net.ipv4.conf.default.send_redirects = 0</pre> <pre style="word-wrap:break-word;white-space:pre-wrap">net.core.somaxconn = 1536</pre> <pre style="word-wrap:break-word;white-space:pre-wrap">net.core.dev_weight = 512</pre> <pre style="word-wrap:break-word;white-space:pre-wrap">##3x normal for a queue and budget suited to networks greater than 100mbps</pre> <pre>net.core.netdev_budget = 10000</pre> <pre>net.core.netdev_max_backlog = 30000</pre> </div> <div> <pre> </pre> </div> </div> </div> </div> <div> <div> On Fri, Oct 10, 2014 at 4:35 AM, Michael Mraka <<a href="mailto:michael.mraka@redhat.com" target="_blank">michael.mraka@redhat.com</a>> wrote: Nahelou Sébastien wrote: % Hi Spacewalk lovers Hello Sebastian, % Since we upgrade from 2.0 to 2.2, we encounter a capacity problem due to CPU usage overall. % When users load a system page in the GUI, the tomcat process grow to 90% usage. Api and GUI become unavailable or very slow. Are you able to identify which page causes such high load? % Our architecture is as following: % 5 rhn proxies % LVS Master/slave mode % 1 front master: SERVICES="jabberd $TOMCAT httpd osa-dispatcher Monitoring MonitoringScout rhn-search cobblerd taskomatic" % 1 front slave: SERVICES="jabberd $TOMCAT httpd osa-dispatcher Monitoring MonitoringScout " % 1 cluster PGSQL % % Our masters have 2* Intel(R) Xeon(R) CPU E5-2609 / 92 RAM / raid 1 using CentOS 6.5. % % We have about 6000 systems registered, about 40 organizations and more than 600 channels (including cloned) % % My questions are: % - Is it possible to scale spacewalk horizontally ? By changing lvs configuration to Active/Active ? By using HAProxy ? other ? We've never tested Active/Active cluster. I dare to say it won't work. Our recomendation is to use one Spacewalk Proxy for every 5000 systems. % - Is it possible to tune tomcat better than adding memory ? (we use actually -Xms8196m -Xmx16384m) I don't know. Frankly I've never heard a report about necessity to do so. % - What is the maximum number of systems that we can register in 1 organisation ? (Actually our max seems to be 1700) We know about installations with tens to hundereds of thousands systems. % - Is there a maximum number of systems that we can register in a Spacewalk cluster ? No, it's limmited only by numbers in your spacewalk certificate. Default certificate is limited to 20000. If you need more you can generate your own. % Thank in advance % % Sébastien NAHELOU % System engineer % Worldline - Seclin - France Regards, -- Michael Mráka Satellite Engineering, Red Hat _______________________________________________ Spacewalk-list mailing list <a href="mailto:Spacewalk-list@redhat.com" target="_blank">Spacewalk-list@redhat.com</a> <a href="https://www.redhat.com/mailman/listinfo/spacewalk-list" target="_blank">https://www.redhat.com/mailman/listinfo/spacewalk-list</a> </div> </div> </div> </div> </div> <div class="MsoNormal" align="center" style="text-align:center"> <hr size="2" width="100%" align="center"> </div> Ce message et les pièces jointes sont confidentiels et réservés à l'usage exclusif de ses destinataires. Il peut également être protégé par le secret professionnel. Si vous recevez ce message par erreur, merci d'en avertir immédiatement l'expéditeur et de le détruire. L'intégrité du message ne pouvant être assurée sur Internet, la responsabilité de Worldline ne pourra être recherchée quant au contenu de ce message. Bien que les meilleurs efforts soient faits pour maintenir cette transmission exempte de tout virus, l'expéditeur ne donne aucune garantie à cet égard et sa responsabilité ne saurait être recherchée pour tout dommage résultant d'un virus transmis. This e-mail and the documents attached are confidential and intended solely for the addressee; it may also be privileged. If you receive this e-mail in error, please notify the sender immediately and destroy it. As its integrity cannot be secured on the Internet, the Worldline liability cannot be triggered for the message content. Although the sender endeavours to maintain a computer virus-free network, the sender does not warrant that this transmission is virus-free and will not be liable for any damages resulting from any virus transmitted. </div> _______________________________________________ Spacewalk-list mailing list <a href="mailto:Spacewalk-list@redhat.com" target="_blank">Spacewalk-list@redhat.com</a> <a href="https://www.redhat.com/mailman/listinfo/spacewalk-list" target="_blank">https://www.redhat.com/mailman/listinfo/spacewalk-list</a> </div> </div> </div></div></div><div><div class="h5"> <hr> Ce message et les pièces jointes sont confidentiels et réservés à l'usage exclusif de ses destinataires. Il peut également être protégé par le secret professionnel. Si vous recevez ce message par erreur, merci d'en avertir immédiatement l'expéditeur et de le détruire. L'intégrité du message ne pouvant être assurée sur Internet, la responsabilité de Worldline ne pourra être recherchée quant au contenu de ce message. Bien que les meilleurs efforts soient faits pour maintenir cette transmission exempte de tout virus, l'expéditeur ne donne aucune garantie à cet égard et sa responsabilité ne saurait être recherchée pour tout dommage résultant d'un virus transmis. This e-mail and the documents attached are confidential and intended solely for the addressee; it may also be privileged. If you receive this e-mail in error, please notify the sender immediately and destroy it. As its integrity cannot be secured on the Internet, the Worldline liability cannot be triggered for the message content. Although the sender endeavours to maintain a computer virus-free network, the sender does not warrant that this transmission is virus-free and will not be liable for any damages resulting from any virus transmitted. </div></div></div> _______________________________________________ Spacewalk-list mailing list <a href="mailto:Spacewalk-list@redhat.com">Spacewalk-list@redhat.com</a> <a href="https://www.redhat.com/mailman/listinfo/spacewalk-list" target="_blank">https://www.redhat.com/mailman/listinfo/spacewalk-list</a> </blockquote></div> </div>