[Spacewalk-list] Spacewalk performance tuning for deployments with 1000+ hosts
Krzysztof Pawłowski
msciciel at msciciel.eu
Tue Nov 18 21:11:48 UTC 2014
I observed that during high db load rhn-search on spacewalk server is very
busy. Is there configuration options to increase number of db connections
used by this daemon to increase parallelism ? My db backend is pgsql and
with only single connection all sql queries are bound to single cpu core.
2014-10-30 17:15 GMT+01:00 Nahelou Sébastien <
sebastien.nahelou at worldline.com>:
> Hi
>
>
>
> In my case, I use
>
> <Connector port="8080" protocol="HTTP/1.1" connectionTimeout="50000"
> redirectPort="8443" URIEncoding="UTF-8" address="127.0.0.1"
> maxThreads="1024" maxKeepAliveRequests="1000" />
>
> But increase timeout is just a workaround and won’t solve this problem…
>
>
>
> Sebastien
>
>
>
> *De :* spacewalk-list-bounces at redhat.com [mailto:
> spacewalk-list-bounces at redhat.com] *De la part de* Krzysztof Pawlowski
> *Envoyé :* jeudi 30 octobre 2014 17:00
> *À :* spacewalk-list at redhat.com
> *Objet :* Re: [Spacewalk-list] Spacewalk performance tuning for
> deployments with 1000+ hosts
>
>
>
> I've increase timeout settings:
>
> /etc/httpd/conf/httpd.conf:Timeout 600
> /etc/httpd/conf.d/zz-spacewalk-www.conf:ProxyTimeout 600
> /etc/httpd/conf.d/ssl.conf:SSLSessionCacheTimeout 300
>
> /etc/tomcat6/server.xml: <Connector port="8080" protocol="HTTP/1.1"
> connectionTimeout="200000" redirectPort="8443" URIEncoding="UTF-8"
> address="127.0.0.1" maxThreads="1024"/>
> /etc/tomcat6/web.xml: <session-timeout>30</session-timeout>
>
> But above is not enough.
>
>
>
> 2014-10-29 22:35 GMT+01:00 Matthew Madey <mattmadey at gmail.com>:
>
> Do you have any errors in the tomcat or httpd error logs that could show
> the root cause? I'm thinking it's probably a timeout setting that needs to
> be increased.
>
>
>
> On Wed, Oct 29, 2014 at 10:56 AM, Amedeo Salvati <amedeo at oscert.net>
> wrote:
>
> this will scale out (not scale UP) your env, but you will change where
> your client will be connected -> to spacewalk or to spacewalk proxy
>
> Da: spacewalk-list-bounces at redhat.com
>
> A: spacewalk-list at redhat.com
>
> Cc:
>
> Data: Wed, 29 Oct 2014 15:49:09 +0100
>
> Oggetto: Re: [Spacewalk-list] Spacewalk performance tuning for
> deployments with 1000+ hosts
>
>
>
> > Does this solve problem with web ui ?
>
> > Sent from my Windows Phone
> ------------------------------
>
> *> From: *Amedeo Salvati <amedeo at oscert.net>
> *> Sent: *2014-10-29 12:03
> *> To: *spacewalk-list at redhat.com
> *> Cc: *spacewalk-list at redhat.com
> *> Subject: *Re: [Spacewalk-list]Spacewalk performance tuning for
> deployments with 1000+ hosts
>
> > repeat I think you must use rhn|spacewalk proxies
>
> > best regards
> > a
>
> > Da: spacewalk-list-bounces at redhat.com
>
> > A: spacewalk-list at redhat.com
>
> > Cc:
>
> > Data: Wed, 29 Oct 2014 07:22:30 +0100
>
> > Oggetto: Re: [Spacewalk-list] Spacewalk performance tuning for
> deployments with 1000+ hosts
>
>
>
> > > Unfortunately it doesn't help :(
>
> > > When i want do sth with all 1500 systems i get :
> > > Service Temporarily Unavailable
>
> > > The server is temporarily unable to service your request due to
> maintenance downtime or capacity problems. Please try again later.
>
>
>
>
>
> > > 2014-10-28 17:41 GMT+01:00 Matthew Madey <mattmadey at gmail.com>:
>
>
>
> > > Here are some configurations you might find helpful
> for tuning Apache\Tomcat\Java\Networking.. Like others have mentioned..
> when you get over 1000+ clients, it's a good idea to start scaling
> horizontally with Spacewalk Proxies. We use 4 proxies in our production
> environment and are servicing 8000+ clients. We can actually patch 1600
> clients at a time and the GUI is still pretty responsive. Can't guarantee
> this will resolve your issue, but this worked for us.
>
>
>
> > > Add maxThreads to /etc/tomcat6/server.xml
>
> > >
>
> > > <Connector port="8080" protocol="HTTP/1.1" connectionTimeout="20000" redirectPort="8443" URIEncoding="UTF-8" address="127.0.0.1" maxThreads="1024" maxKeepAliveRequests="1000"/>
>
> > > <!-- A "Connector" using the shared thread pool-->
>
> > > <!--
>
> > > <Connector executor="tomcatThreadPool"
>
> > > port="8080" protocol="HTTP/1.1"
>
> > > connectionTimeout="20000"
>
> > > redirectPort="8443" />
>
> > >
>
> > > <!-- Define an AJP 1.3 Connector on port 8009 -->
>
> > > <Connector port="8009" protocol="AJP/1.3" redirectPort="8443" URIEncoding="UTF-8" address="127.0.0.1" maxThreads="1024"/>
>
> > >
>
> > > <Connector port="8009" protocol="AJP/1.3" redirectPort="8443" URIEncoding="UTF-8" address="::1" maxThreads="1024"/>
>
> > >
>
> > >
>
> > > Tune Apache to service more requests /etc/httpd/conf.d/zz-spacewalk-server.conf
>
> > >
>
> > >
>
> > > #######################################################
>
> > > # Authorship and versioning info
>
> > > # $Author$
>
> > > # $Date$
>
> > > # $URL$
>
> > > # $Rev$
>
> > > # deployment_location: /etc/httpd/conf.d/
>
> > > #######################################################
>
> > > # ** DO NOT EDIT **
>
> > > # Master configuration file for the rhn_server setup
>
> > > #
>
> > >
>
> > > ##
>
> > > ## Spacewalk settings
>
> > > ##
>
> > >
>
> > > <VirtualHost *>
>
> > >
>
> > > <IfModule mod_jk.c>
>
> > > # Inherit the mod_jk settings defined in zz-spacewalk-www.conf
>
> > > JkMountCopy On
>
> > > </IfModule>
>
> > >
>
> > > <Directory "/var/www/html/*">
>
> > > AllowOverride all
>
> > > </Directory>
>
> > >
>
> > > RewriteEngine on
>
> > > RewriteOptions inherit
>
> > > </VirtualHost>
>
> > >
>
> > > # Override default httpd prefork settings
>
> > > <IfModule prefork.c>
>
> > > StartServers 8
>
> > > MinSpareServers 400
>
> > > MaxSpareServers 400
>
> > > ServerLimit 1024
>
> > > MaxClients 1024
>
> > > MaxRequestsPerChild 200
>
> > > </IfModule>
>
> > >
>
> > > Include /etc/rhn/satellite-httpd/conf/rhn/rhn_monitoring.conf
>
> > >
>
> > >
>
> > >
>
> > > Also added some network tuning to /etc/sysctl.conf
>
> > >
>
> > > net.ipv4.icmp_echo_ignore_broadcasts = 1
>
> > > net.ipv4.conf.all.secure_redirects = 0
>
> > > net.ipv4.tcp_max_syn_backlog = 8192
>
> > > net.ipv4.conf.default.secure_redirects = 0
>
> > > net.ipv4.tcp_syncookies = 1
>
> > > net.ipv4.conf.all.accept_source_route = 0
>
> > > net.ipv4.conf.all.rp_filter = 1
>
> > > net.ipv4.conf.all.send_redirects = 0
>
> > > net.ipv4.conf.default.accept_redirects = 0
>
> > > net.ipv4.conf.all.accept_redirects = 0
>
> > > net.ipv4.conf.default.send_redirects = 0
>
> > > net.core.somaxconn = 1536
>
> > > net.core.dev_weight = 512
>
> > > ##3x normal for a queue and budget suited to networks greater than 100mbps
>
> > > net.core.netdev_budget = 10000
>
> > > net.core.netdev_max_backlog = 30000
>
>
>
>
>
> > > Depending on the amount of memory on your Spacewalk server, you may want to increase your JAVA_OPTS Xms and Xmx settings to something a little higher. Typically only needed if you are seeing Java Heap out of memory errors in your Spacewalk logs.
>
>
>
>
>
> > > On Tue, Oct 28, 2014 at 11:21 AM, Waldirio Manhães Pinheiro <
> waldirio at gmail.com> wrote:
>
> > > Dear Krzysztof
>
>
>
> > > Have you checked your numa configuration ?!
>
>
>
> > > Maybe you can customize you environment to use the same bus to
> application / memory.
>
>
> > > ______________
> > > Atenciosamente
> > > Waldirio
> > > msn: waldirio at gmail.com
> > > Skype: waldirio
> > > Site: www.waldirio.com.br
> > > Blog: blog.waldirio.com.br
>
> > > LinkedIn: http://br.linkedin.com/pub/waldirio-pinheiro/22/b21/646
> > > PGP: www.waldirio.com.br/public.html
>
>
>
> > > On Tue, Oct 28, 2014 at 2:10 PM, Krzysztof Pawłowski <
> msciciel at msciciel.eu> wrote:
>
> > > We have dedicated machine for db pgsql (16GB RAM , 8 cores) and
> seperate for spacewalk (16GB RAM, 8 Cores).
>
>
>
> > > I think that problem is with enormous number of queries to database.
> During such request db is not utilized 100% and spacewalk is also not 100%
> utilized.
>
>
>
>
>
> > > 2014-10-28 16:25 GMT+01:00 Götz Reinicke - IT Koordinator <
> goetz.reinicke at filmakademie.de>:
>
> > > Hi,
> > > Am 28.10.14 um 12:57 schrieb Krzysztof Pawłowski:
>
> > > > Hi,
> > > > Is there any guide about tuning spacewalk performance ? With every
> new
> > > > host spacewalk is getting slowly. Using SSM with more than 200-300
> hosts
> > > > is impossible due timeouts. It's also not possible to deploy config
> > > > files to all hosts.
> > > > Standard java tuning was done, java gc is not the problem now.
> > > >
> > > > Any suggestions ?
>
> > > what server hardware do you use? What is the systemload while
> performing
> > > that tasks? CPU, RAM, Disksystem, IO, Network .... ?
>
>
> > > /Götz
>
> > > --
> > > Götz Reinicke
> > > IT-Koordinator
>
> > > Tel. +49 7141 969 82 420
> > > E-Mail goetz.reinicke at filmakademie.de
>
> > > Filmakademie Baden-Württemberg GmbH
> > > Akademiehof 10
> > > 71638 Ludwigsburg
> > > www.filmakademie.de <http://www.filmakademie.de>
>
> > > Eintragung Amtsgericht Stuttgart HRB 205016
>
> > > Vorsitzender des Aufsichtsrats: Jürgen Walter MdL
>
>
>
> > [The entire original message is not included.]
>
>
>
> _______________________________________________
> Spacewalk-list mailing list
> Spacewalk-list at redhat.com
> https://www.redhat.com/mailman/listinfo/spacewalk-list
>
>
>
>
> _______________________________________________
> Spacewalk-list mailing list
> Spacewalk-list at redhat.com
> https://www.redhat.com/mailman/listinfo/spacewalk-list
>
>
>
> ------------------------------
>
> Ce message et les pièces jointes sont confidentiels et réservés à l'usage
> exclusif de ses destinataires. Il peut également être protégé par le secret
> professionnel. Si vous recevez ce message par erreur, merci d'en avertir
> immédiatement l'expéditeur et de le détruire. L'intégrité du message ne
> pouvant être assurée sur Internet, la responsabilité de Worldline ne pourra
> être recherchée quant au contenu de ce message. Bien que les meilleurs
> efforts soient faits pour maintenir cette transmission exempte de tout
> virus, l'expéditeur ne donne aucune garantie à cet égard et sa
> responsabilité ne saurait être recherchée pour tout dommage résultant d'un
> virus transmis.
>
> This e-mail and the documents attached are confidential and intended
> solely for the addressee; it may also be privileged. If you receive this
> e-mail in error, please notify the sender immediately and destroy it. As
> its integrity cannot be secured on the Internet, the Worldline liability
> cannot be triggered for the message content. Although the sender endeavours
> to maintain a computer virus-free network, the sender does not warrant that
> this transmission is virus-free and will not be liable for any damages
> resulting from any virus transmitted.
>
> _______________________________________________
> Spacewalk-list mailing list
> Spacewalk-list at redhat.com
> https://www.redhat.com/mailman/listinfo/spacewalk-list
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/spacewalk-list/attachments/20141118/9105e79e/attachment.htm>
More information about the Spacewalk-list
mailing list