<div dir="ltr">I observed that during high db load rhn-search on spacewalk server is very busy. Is there configuration options to increase number of db connections used by this daemon to increase parallelism ? My db backend is pgsql and with only single connection all sql queries are bound to single cpu core.</div><div class="gmail_extra"> <div class="gmail_quote">2014-10-30 17:15 GMT+01:00 Nahelou Sébastien <<a href="mailto:sebastien.nahelou@worldline.com" target="_blank">sebastien.nahelou@worldline.com</a>>: <blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"> <div lang="FR" vlink="purple" link="blue"> <div> Hi In my case, I use <Connector port="8080" protocol="HTTP/1.1" connectionTimeout="50000" redirectPort="8443" URIEncoding="UTF-8" address="127.0.0.1" maxThreads="1024" maxKeepAliveRequests="1000" /> But increase timeout is just a workaround and won’t solve this problem… Sebastien De : <a href="mailto:spacewalk-list-bounces@redhat.com" target="_blank">spacewalk-list-bounces@redhat.com</a> [mailto:<a href="mailto:spacewalk-list-bounces@redhat.com" target="_blank">spacewalk-list-bounces@redhat.com</a>] De la part de Krzysztof Pawlowski Envoyé : jeudi 30 octobre 2014 17:00 À : <a href="mailto:spacewalk-list@redhat.com" target="_blank">spacewalk-list@redhat.com</a> Objet : Re: [Spacewalk-list] Spacewalk performance tuning for deployments with 1000+ hosts<div><div class="h5"> <div> <div> I've increase timeout settings: /etc/httpd/conf/httpd.conf:Timeout 600 /etc/httpd/conf.d/zz-spacewalk-www.conf:ProxyTimeout 600 /etc/httpd/conf.d/ssl.conf:SSLSessionCacheTimeout 300 /etc/tomcat6/server.xml: <Connector port="8080" protocol="HTTP/1.1" connectionTimeout="200000" redirectPort="8443" URIEncoding="UTF-8" address="127.0.0.1" maxThreads="1024"/> /etc/tomcat6/web.xml: <session-timeout>30</session-timeout> </div> But above is not enough. </div> <div> <div> 2014-10-29 22:35 GMT+01:00 Matthew Madey <<a href="mailto:mattmadey@gmail.com" target="_blank">mattmadey@gmail.com</a>>: <div> Do you have any errors in the tomcat or httpd error logs that could show the root cause? I'm thinking it's probably a timeout setting that needs to be increased. </div> <div> <div> <div> <div> On Wed, Oct 29, 2014 at 10:56 AM, Amedeo Salvati <<a href="mailto:amedeo@oscert.net" target="_blank">amedeo@oscert.net</a>> wrote: </div> </div> <blockquote style="border-width:medium medium medium 1pt;border-style:none none none solid;border-color:currentColor currentColor currentColor rgb(204,204,204);padding:0cm 0cm 0cm 6pt;margin-right:0cm;margin-left:4.8pt"> <div> <div> <div> <div> this will scale out (not scale UP) your env, but you will change where your client will be connected -> to spacewalk or to spacewalk proxy <div> Da: <a href="mailto:spacewalk-list-bounces@redhat.com" target="_blank">spacewalk-list-bounces@redhat.com</a> </div> <div> A: <a href="mailto:spacewalk-list@redhat.com" target="_blank">spacewalk-list@redhat.com</a> </div> <div> Cc: </div> <div> Data: Wed, 29 Oct 2014 15:49:09 +0100 </div> <div> <div> <div> Oggetto: Re: [Spacewalk-list] Spacewalk performance tuning for deployments with 1000+ hosts </div> <div> <div> <div> > Does this solve problem with web ui ? > Sent from my Windows Phone </div> </div> <div> <div align="center" class="MsoNormal" style="text-align:center"> <hr width="100%" size="2" align="center"> </div> > From: <a href="mailto:amedeo@oscert.net" target="_blank">Amedeo Salvati</a> > Sent: ‎2014-‎10-‎29 12:03 > To: <a href="mailto:spacewalk-list@redhat.com" target="_blank">spacewalk-list@redhat.com</a> > Cc: <a href="mailto:spacewalk-list@redhat.com" target="_blank">spacewalk-list@redhat.com</a> > Subject: Re: [Spacewalk-list]Spacewalk performance tuning for deployments with 1000+ hosts </div> <div> <div> > repeat I think you must use rhn|spacewalk proxies > best regards > a <div> > Da: <a href="mailto:spacewalk-list-bounces@redhat.com" target="_blank">spacewalk-list-bounces@redhat.com</a> </div> <div> > A: <a href="mailto:spacewalk-list@redhat.com" target="_blank">spacewalk-list@redhat.com</a> </div> <div> > Cc: </div> <div> > Data: Wed, 29 Oct 2014 07:22:30 +0100 </div> <div> > Oggetto: Re: [Spacewalk-list] Spacewalk performance tuning for deployments with 1000+ hosts </div> <div> <div> <div> > > Unfortunately it doesn't help :( </div> > > When i want do sth with all 1500 systems i get : <div> <h1>> > Service Temporarily Unavailable</h1> > > The server is temporarily unable to service your request due to maintenance downtime or capacity problems. Please try again later. </div> </div> <div> <div> > > 2014-10-28 17:41 GMT+01:00 Matthew Madey <<a href="mailto:mattmadey@gmail.com" target="_blank">mattmadey@gmail.com</a>>: <div> <div> </div> > > Here are some configurations you might find helpful for tuning Apache\Tomcat\Java\Networking.. Like others have mentioned.. when you get over 1000+ clients, it's a good idea to start scaling horizontally with Spacewalk Proxies. We use 4 proxies in our production environment and are servicing 8000+ clients. We can actually patch 1600 clients at a time and the GUI is still pretty responsive. Can't guarantee this will resolve your issue, but this worked for us. <div> </div> <div> > > Add maxThreads to /etc/tomcat6/server.xml <div> > > </div> <div> <pre style="white-space:pre-wrap">> > <Connector port="8080" protocol="HTTP/1.1" connectionTimeout="20000" redirectPort="8443" URIEncoding="UTF-8" address="127.0.0.1" maxThreads="1024" maxKeepAliveRequests="1000"/></pre> <pre style="white-space:pre-wrap">> > </pre> <pre style="white-space:pre-wrap">> > </pre> <pre style="white-space:pre-wrap">> > <Connector port="8009" protocol="AJP/1.3" redirectPort="8443" URIEncoding="UTF-8" address="127.0.0.1" maxThreads="1024"/></pre> <pre style="white-space:pre-wrap">> > </pre> <pre style="white-space:pre-wrap">> > <Connector port="8009" protocol="AJP/1.3" redirectPort="8443" URIEncoding="UTF-8" address="::1" maxThreads="1024"/></pre> <div> <pre style="white-space:pre-wrap">> > </pre> </div> <div> <pre style="white-space:pre-wrap">> > </pre> </div> <div> <pre style="white-space:pre-wrap">> > Tune Apache to service more requests /etc/httpd/conf.d/zz-spacewalk-server.conf</pre> </div> <div> <pre style="white-space:pre-wrap">> > </pre> </div> <div> <pre style="white-space:pre-wrap">> > </pre> </div> <div> <pre style="white-space:pre-wrap">> > #######################################################</pre> <pre style="white-space:pre-wrap">> > # Authorship and versioning info</pre> <pre style="white-space:pre-wrap">> > # $Author$</pre> <pre style="white-space:pre-wrap">> > # $Date$</pre> <pre style="white-space:pre-wrap">> > # $URL$</pre> <pre style="white-space:pre-wrap">> > # $Rev$</pre> <pre style="white-space:pre-wrap">> > # deployment_location: /etc/httpd/conf.d/</pre> <pre style="white-space:pre-wrap">> > #######################################################</pre> <pre style="white-space:pre-wrap">> > # ** DO NOT EDIT **</pre> <pre style="white-space:pre-wrap">> > # Master configuration file for the rhn_server setup</pre> <pre style="white-space:pre-wrap">> > #</pre> <pre style="white-space:pre-wrap">> > </pre> <pre style="white-space:pre-wrap">> > ##</pre> <pre style="white-space:pre-wrap">> > ## Spacewalk settings</pre> <pre style="white-space:pre-wrap">> > ##</pre> <pre style="white-space:pre-wrap">> > </pre> <pre style="white-space:pre-wrap">> > <VirtualHost *></pre> <pre style="white-space:pre-wrap">> > </pre> <pre style="white-space:pre-wrap">> > <IfModule mod_jk.c></pre> <pre style="white-space:pre-wrap">> > # Inherit the mod_jk settings defined in zz-spacewalk-www.conf</pre> <pre style="white-space:pre-wrap">> > JkMountCopy On</pre> <pre style="white-space:pre-wrap">> > </IfModule></pre> <pre style="white-space:pre-wrap">> > </pre> <pre style="white-space:pre-wrap">> > <Directory "/var/www/html/*"></pre> <pre style="white-space:pre-wrap">> > AllowOverride all</pre> <pre style="white-space:pre-wrap">> > </Directory></pre> <pre style="white-space:pre-wrap">> > </pre> <pre style="white-space:pre-wrap">> > RewriteEngine on</pre> <pre style="white-space:pre-wrap">> > RewriteOptions inherit</pre> <pre style="white-space:pre-wrap">> > </VirtualHost></pre> <pre style="white-space:pre-wrap">> > </pre> <pre style="white-space:pre-wrap">> > # Override default httpd prefork settings</pre> <pre style="white-space:pre-wrap">> > <IfModule prefork.c></pre> <pre style="white-space:pre-wrap">> > StartServers 8</pre> <pre style="white-space:pre-wrap">> > MinSpareServers 400</pre> <pre style="white-space:pre-wrap">> > MaxSpareServers 400</pre> <pre style="white-space:pre-wrap">> > ServerLimit 1024</pre> <pre style="white-space:pre-wrap">> > MaxClients 1024</pre> <pre style="white-space:pre-wrap">> > MaxRequestsPerChild 200</pre> <pre style="white-space:pre-wrap">> > </IfModule></pre> <pre style="white-space:pre-wrap">> > </pre> <pre style="white-space:pre-wrap">> > Include /etc/rhn/satellite-httpd/conf/rhn/rhn_monitoring.conf</pre> <div> <pre style="white-space:pre-wrap">> > </pre> </div> <div> <pre style="white-space:pre-wrap">> > </pre> </div> <div> <pre style="white-space:pre-wrap">> > </pre> </div> </div> <div> <pre style="white-space:pre-wrap">> > Also added some network tuning to /etc/sysctl.conf</pre> </div> <div> <pre style="white-space:pre-wrap">> > </pre> </div> <div> <pre style="white-space:pre-wrap">> > net.ipv4.icmp_echo_ignore_broadcasts = 1</pre> <pre style="white-space:pre-wrap">> > net.ipv4.conf.all.secure_redirects = 0</pre> <pre style="white-space:pre-wrap">> > net.ipv4.tcp_max_syn_backlog = 8192</pre> <pre style="white-space:pre-wrap">> > net.ipv4.conf.default.secure_redirects = 0</pre> <pre style="white-space:pre-wrap">> > net.ipv4.tcp_syncookies = 1</pre> <pre style="white-space:pre-wrap">> > net.ipv4.conf.all.accept_source_route = 0</pre> <pre style="white-space:pre-wrap">> > net.ipv4.conf.all.rp_filter = 1</pre> <pre style="white-space:pre-wrap">> > net.ipv4.conf.all.send_redirects = 0</pre> <pre style="white-space:pre-wrap">> > net.ipv4.conf.default.accept_redirects = 0</pre> <pre style="white-space:pre-wrap">> > net.ipv4.conf.all.accept_redirects = 0</pre> <pre style="white-space:pre-wrap">> > net.ipv4.conf.default.send_redirects = 0</pre> <pre style="white-space:pre-wrap">> > net.core.somaxconn = 1536</pre> <pre style="white-space:pre-wrap">> > net.core.dev_weight = 512</pre> <pre style="white-space:pre-wrap">> > ##3x normal for a queue and budget suited to networks greater than 100mbps</pre> <pre style="white-space:pre-wrap">> > net.core.netdev_budget = 10000</pre> <pre style="white-space:pre-wrap">> > net.core.netdev_max_backlog = 30000</pre> <pre style="white-space:pre-wrap"> </pre> <pre style="white-space:pre-wrap"> </pre> <pre style="white-space:pre-wrap">> > Depending on the amount of memory on your Spacewalk server, you may want to increase your JAVA_OPTS Xms and Xmx settings to something a little higher. Typically only needed if you are seeing Java Heap out of memory errors in your Spacewalk logs.</pre> </div> <div> </div> </div> </div> </div> <div> <div> <div> <div> > > On Tue, Oct 28, 2014 at 11:21 AM, Waldirio Manhães Pinheiro <<a href="mailto:waldirio@gmail.com" target="_blank">waldirio@gmail.com</a>> wrote: <div> > > Dear Krzysztof <div> </div> <div> > > Have you checked your numa configuration ?! </div> <div> </div> <div> > > Maybe you can customize you environment to use the same bus to application / memory. </div> </div> <div> <div> <div> > > ______________ > > Atenciosamente > > Waldirio > > msn: <a href="mailto:waldirio@gmail.com" target="_blank">waldirio@gmail.com</a> > > Skype: waldirio > > Site: <a href="http://www.waldirio.com.br" target="_blank">www.waldirio.com.br</a> > > Blog: <a href="http://blog.waldirio.com.br" target="_blank">blog.waldirio.com.br</a> </div> <div> > > LinkedIn: <a href="http://br.linkedin.com/pub/waldirio-pinheiro/22/b21/646" target="_blank"> http://br.linkedin.com/pub/waldirio-pinheiro/22/b21/646</a> > > PGP: <a href="http://www.waldirio.com.br/public.html" target="_blank">www.waldirio.com.br/public.html</a> </div> </div> <div> <div> <div> > > On Tue, Oct 28, 2014 at 2:10 PM, Krzysztof Pawłowski <<a href="mailto:msciciel@msciciel.eu" target="_blank">msciciel@msciciel.eu</a>> wrote: <div> <div> <div> > > We have dedicated machine for db pgsql (16GB RAM , 8 cores) and seperate for spacewalk (16GB RAM, 8 Cores). </div> </div> > > I think that problem is with enormous number of queries to database. During such request db is not utilized 100% and spacewalk is also not 100% utilized. <div> <div> </div> </div> </div> <div> <div> <div> <div> > > 2014-10-28 16:25 GMT+01:00 Götz Reinicke - IT Koordinator <<a href="mailto:goetz.reinicke@filmakademie.de" target="_blank">goetz.reinicke@filmakademie.de</a>>: </div> </div> <blockquote style="border-width:medium medium medium 1pt;border-style:none none none solid;border-color:currentColor currentColor currentColor rgb(204,204,204);padding:0cm 0cm 0cm 6pt;margin-right:0cm;margin-left:4.8pt"> <div> <div> > > Hi, > > Am 28.10.14 um 12:57 schrieb Krzysztof Pawłowski: <div> <div> > > > Hi, > > > Is there any guide about tuning spacewalk performance ? With every new > > > host spacewalk is getting slowly. Using SSM with more than 200-300 hosts > > > is impossible due timeouts. It's also not possible to deploy config > > > files to all hosts. > > > Standard java tuning was done, java gc is not the problem now. > > > > > > Any suggestions ? </div> </div> > > what server hardware do you use? What is the systemload while performing > > that tasks? CPU, RAM, Disksystem, IO, Network .... ? > > /Götz > > -- > > Götz Reinicke > > IT-Koordinator > > Tel. <a href="tel:%2B49%207141%20969%2082%20420" target="_blank">+49 7141 969 82 420</a> > > E-Mail <a href="mailto:goetz.reinicke@filmakademie.de" target="_blank">goetz.reinicke@filmakademie.de</a> > > Filmakademie Baden-Württemberg GmbH > > Akademiehof 10 > > 71638 Ludwigsburg <a href="http://www.filmakademie.de" target="_blank">> > www.filmakademie.de</a> > > Eintragung Amtsgericht Stuttgart HRB 205016 > > Vorsitzender des Aufsichtsrats: Jürgen Walter MdL </div> </div> </blockquote> </div> </div> </div> </div> </div> </div> </div> </div> </div> </div> </div> </div> </div> </div> </div> <div> > [The entire original message is not included.] </div> </div> </div> </div> </div> </div> </div> </div> _______________________________________________ Spacewalk-list mailing list <a href="mailto:Spacewalk-list@redhat.com" target="_blank">Spacewalk-list@redhat.com</a> <a href="https://www.redhat.com/mailman/listinfo/spacewalk-list" target="_blank">https://www.redhat.com/mailman/listinfo/spacewalk-list</a> </blockquote> </div> </div> _______________________________________________ Spacewalk-list mailing list <a href="mailto:Spacewalk-list@redhat.com" target="_blank">Spacewalk-list@redhat.com</a> <a href="https://www.redhat.com/mailman/listinfo/spacewalk-list" target="_blank">https://www.redhat.com/mailman/listinfo/spacewalk-list</a> </div> </div> </div></div></div> <hr> Ce message et les pièces jointes sont confidentiels et réservés à l'usage exclusif de ses destinataires. Il peut également être protégé par le secret professionnel. Si vous recevez ce message par erreur, merci d'en avertir immédiatement l'expéditeur et de le détruire. L'intégrité du message ne pouvant être assurée sur Internet, la responsabilité de Worldline ne pourra être recherchée quant au contenu de ce message. Bien que les meilleurs efforts soient faits pour maintenir cette transmission exempte de tout virus, l'expéditeur ne donne aucune garantie à cet égard et sa responsabilité ne saurait être recherchée pour tout dommage résultant d'un virus transmis. This e-mail and the documents attached are confidential and intended solely for the addressee; it may also be privileged. If you receive this e-mail in error, please notify the sender immediately and destroy it. As its integrity cannot be secured on the Internet, the Worldline liability cannot be triggered for the message content. Although the sender endeavours to maintain a computer virus-free network, the sender does not warrant that this transmission is virus-free and will not be liable for any damages resulting from any virus transmitted. </div> _______________________________________________ Spacewalk-list mailing list <a href="mailto:Spacewalk-list@redhat.com">Spacewalk-list@redhat.com</a> <a href="https://www.redhat.com/mailman/listinfo/spacewalk-list" target="_blank">https://www.redhat.com/mailman/listinfo/spacewalk-list</a> </blockquote></div> </div>