[Spacewalk-list] Spacewalk performance tuning for deployments with 1000+ hosts
Krzysztof Pawłowski
msciciel at msciciel.eu
Thu Oct 30 15:59:59 UTC 2014
I've increase timeout settings:
/etc/httpd/conf/httpd.conf:Timeout 600
/etc/httpd/conf.d/zz-spacewalk-www.conf:ProxyTimeout 600
/etc/httpd/conf.d/ssl.conf:SSLSessionCacheTimeout 300
/etc/tomcat6/server.xml: <Connector port="8080" protocol="HTTP/1.1"
connectionTimeout="200000" redirectPort="8443" URIEncoding="UTF-8"
address="127.0.0.1" maxThreads="1024"/>
/etc/tomcat6/web.xml: <session-timeout>30</session-timeout>
But above is not enough.
2014-10-29 22:35 GMT+01:00 Matthew Madey <mattmadey at gmail.com>:
> Do you have any errors in the tomcat or httpd error logs that could show
> the root cause? I'm thinking it's probably a timeout setting that needs to
> be increased.
>
> On Wed, Oct 29, 2014 at 10:56 AM, Amedeo Salvati <amedeo at oscert.net>
> wrote:
>
>> this will scale out (not scale UP) your env, but you will change where
>> your client will be connected -> to spacewalk or to spacewalk proxy
>>
>> Da: spacewalk-list-bounces at redhat.com
>> A: spacewalk-list at redhat.com
>> Cc:
>> Data: Wed, 29 Oct 2014 15:49:09 +0100
>> Oggetto: Re: [Spacewalk-list] Spacewalk performance tuning for
>> deployments with 1000+ hosts
>>
>> > Does this solve problem with web ui ?
>>
>> > Sent from my Windows Phone
>> ------------------------------
>> > From: Amedeo Salvati <amedeo at oscert.net>
>> > Sent: 2014-10-29 12:03
>> > To: spacewalk-list at redhat.com
>> > Cc: spacewalk-list at redhat.com
>> > Subject: Re: [Spacewalk-list]Spacewalk performance tuning for
>> deployments with 1000+ hosts
>>
>> > repeat I think you must use rhn|spacewalk proxies
>>
>> > best regards
>> > a
>>
>>
>> > Da: spacewalk-list-bounces at redhat.com
>> > A: spacewalk-list at redhat.com
>> > Cc:
>> > Data: Wed, 29 Oct 2014 07:22:30 +0100
>> > Oggetto: Re: [Spacewalk-list] Spacewalk performance tuning for
>> deployments with 1000+ hosts
>>
>> > > Unfortunately it doesn't help :(
>>
>> > > When i want do sth with all 1500 systems i get :
>> > > Service Temporarily Unavailable
>>
>> > > The server is temporarily unable to service your request due to
>> maintenance downtime or capacity problems. Please try again later.
>>
>>
>> > > 2014-10-28 17:41 GMT+01:00 Matthew Madey <mattmadey at gmail.com>:
>>
>>>
>>> > > Here are some configurations you might find helpful for tuning Apache\Tomcat\Java\Networking..
>>> Like others have mentioned.. when you get over 1000+ clients, it's a good
>>> idea to start scaling horizontally with Spacewalk Proxies. We use 4 proxies
>>> in our production environment and are servicing 8000+ clients. We can
>>> actually patch 1600 clients at a time and the GUI is still pretty
>>> responsive. Can't guarantee this will resolve your issue, but this worked
>>> for us.
>>>
>>> > > Add maxThreads to /etc/tomcat6/server.xml
>>>
>>> > >
>>>
>>> > > <Connector port="8080" protocol="HTTP/1.1" connectionTimeout="20000" redirectPort="8443" URIEncoding="UTF-8" address="127.0.0.1" maxThreads="1024" maxKeepAliveRequests="1000"/>
>>>
>>> > > <!-- A "Connector" using the shared thread pool-->
>>>
>>> > > <!--
>>>
>>> > > <Connector executor="tomcatThreadPool"
>>>
>>> > > port="8080" protocol="HTTP/1.1"
>>>
>>> > > connectionTimeout="20000"
>>>
>>> > > redirectPort="8443" />
>>>
>>> > >
>>>
>>> > > <!-- Define an AJP 1.3 Connector on port 8009 -->
>>>
>>> > > <Connector port="8009" protocol="AJP/1.3" redirectPort="8443" URIEncoding="UTF-8" address="127.0.0.1" maxThreads="1024"/>
>>>
>>> > >
>>>
>>> > > <Connector port="8009" protocol="AJP/1.3" redirectPort="8443" URIEncoding="UTF-8" address="::1" maxThreads="1024"/>
>>>
>>> > >
>>>
>>> > >
>>>
>>> > > Tune Apache to service more requests /etc/httpd/conf.d/zz-spacewalk-server.conf
>>>
>>> > >
>>>
>>> > >
>>>
>>> > > #######################################################
>>>
>>> > > # Authorship and versioning info
>>>
>>> > > # $Author$
>>>
>>> > > # $Date$
>>>
>>> > > # $URL$
>>>
>>> > > # $Rev$
>>>
>>> > > # deployment_location: /etc/httpd/conf.d/
>>>
>>> > > #######################################################
>>>
>>> > > # ** DO NOT EDIT **
>>>
>>> > > # Master configuration file for the rhn_server setup
>>>
>>> > > #
>>>
>>> > >
>>>
>>> > > ##
>>>
>>> > > ## Spacewalk settings
>>>
>>> > > ##
>>>
>>> > >
>>>
>>> > > <VirtualHost *>
>>>
>>> > >
>>>
>>> > > <IfModule mod_jk.c>
>>>
>>> > > # Inherit the mod_jk settings defined in zz-spacewalk-www.conf
>>>
>>> > > JkMountCopy On
>>>
>>> > > </IfModule>
>>>
>>> > >
>>>
>>> > > <Directory "/var/www/html/*">
>>>
>>> > > AllowOverride all
>>>
>>> > > </Directory>
>>>
>>> > >
>>>
>>> > > RewriteEngine on
>>>
>>> > > RewriteOptions inherit
>>>
>>> > > </VirtualHost>
>>>
>>> > >
>>>
>>> > > # Override default httpd prefork settings
>>>
>>> > > <IfModule prefork.c>
>>>
>>> > > StartServers 8
>>>
>>> > > MinSpareServers 400
>>>
>>> > > MaxSpareServers 400
>>>
>>> > > ServerLimit 1024
>>>
>>> > > MaxClients 1024
>>>
>>> > > MaxRequestsPerChild 200
>>>
>>> > > </IfModule>
>>>
>>> > >
>>>
>>> > > Include /etc/rhn/satellite-httpd/conf/rhn/rhn_monitoring.conf
>>>
>>> > >
>>>
>>> > >
>>>
>>> > >
>>>
>>> > > Also added some network tuning to /etc/sysctl.conf
>>>
>>> > >
>>>
>>> > > net.ipv4.icmp_echo_ignore_broadcasts = 1
>>>
>>> > > net.ipv4.conf.all.secure_redirects = 0
>>>
>>> > > net.ipv4.tcp_max_syn_backlog = 8192
>>>
>>> > > net.ipv4.conf.default.secure_redirects = 0
>>>
>>> > > net.ipv4.tcp_syncookies = 1
>>>
>>> > > net.ipv4.conf.all.accept_source_route = 0
>>>
>>> > > net.ipv4.conf.all.rp_filter = 1
>>>
>>> > > net.ipv4.conf.all.send_redirects = 0
>>>
>>> > > net.ipv4.conf.default.accept_redirects = 0
>>>
>>> > > net.ipv4.conf.all.accept_redirects = 0
>>>
>>> > > net.ipv4.conf.default.send_redirects = 0
>>>
>>> > > net.core.somaxconn = 1536
>>>
>>> > > net.core.dev_weight = 512
>>>
>>> > > ##3x normal for a queue and budget suited to networks greater than 100mbps
>>>
>>> > > net.core.netdev_budget = 10000
>>>
>>> > > net.core.netdev_max_backlog = 30000
>>>
>>>
>>>
>>> > > Depending on the amount of memory on your Spacewalk server, you may want to increase your JAVA_OPTS Xms and Xmx settings to something a little higher. Typically only needed if you are seeing Java Heap out of memory errors in your Spacewalk logs.
>>>
>>>
>>>
>>> > > On Tue, Oct 28, 2014 at 11:21 AM, Waldirio Manhães Pinheiro <
>>> waldirio at gmail.com> wrote:
>>>
>>>> > > Dear Krzysztof
>>>>
>>>> > > Have you checked your numa configuration ?!
>>>>
>>>> > > Maybe you can customize you environment to use the same bus to
>>>> application / memory.
>>>>
>>>> > > ______________
>>>> > > Atenciosamente
>>>> > > Waldirio
>>>> > > msn: waldirio at gmail.com
>>>> > > Skype: waldirio
>>>> > > Site: www.waldirio.com.br
>>>> > > Blog: blog.waldirio.com.br
>>>> > > LinkedIn: http://br.linkedin.com/pub/waldirio-pinheiro/22/b21/646
>>>> > > PGP: www.waldirio.com.br/public.html
>>>>
>>>> > > On Tue, Oct 28, 2014 at 2:10 PM, Krzysztof Pawłowski <
>>>> msciciel at msciciel.eu> wrote:
>>>>
>>>>> > > We have dedicated machine for db pgsql (16GB RAM , 8 cores) and
>>>>> seperate for spacewalk (16GB RAM, 8 Cores).
>>>>>
>>>>> > > I think that problem is with enormous number of queries to
>>>>> database. During such request db is not utilized 100% and spacewalk is also
>>>>> not 100% utilized.
>>>>>
>>>>>
>>>>>
>>>>> > > 2014-10-28 16:25 GMT+01:00 Götz Reinicke - IT Koordinator <
>>>>> goetz.reinicke at filmakademie.de>:
>>>>>
>>>>>> > > Hi,
>>>>>> > > Am 28.10.14 um 12:57 schrieb Krzysztof Pawłowski:
>>>>>> > > > Hi,
>>>>>> > > > Is there any guide about tuning spacewalk performance ? With
>>>>>> every new
>>>>>> > > > host spacewalk is getting slowly. Using SSM with more than
>>>>>> 200-300 hosts
>>>>>> > > > is impossible due timeouts. It's also not possible to deploy
>>>>>> config
>>>>>> > > > files to all hosts.
>>>>>> > > > Standard java tuning was done, java gc is not the problem now.
>>>>>> > > >
>>>>>> > > > Any suggestions ?
>>>>>>
>>>>>> > > what server hardware do you use? What is the systemload while
>>>>>> performing
>>>>>> > > that tasks? CPU, RAM, Disksystem, IO, Network .... ?
>>>>>>
>>>>>>
>>>>>> > > /Götz
>>>>>>
>>>>>> > > --
>>>>>> > > Götz Reinicke
>>>>>> > > IT-Koordinator
>>>>>>
>>>>>> > > Tel. +49 7141 969 82 420
>>>>>> > > E-Mail goetz.reinicke at filmakademie.de
>>>>>>
>>>>>> > > Filmakademie Baden-Württemberg GmbH
>>>>>> > > Akademiehof 10
>>>>>> > > 71638 Ludwigsburg
>>>>>> > > www.filmakademie.de <http://www.filmakademie.de>
>>>>>>
>>>>>> > > Eintragung Amtsgericht Stuttgart HRB 205016
>>>>>>
>>>>>> > > Vorsitzender des Aufsichtsrats: Jürgen Walter MdL
>>>>>>
>>>>>
>> > [The entire original message is not included.]
>>
>> _______________________________________________
>> Spacewalk-list mailing list
>> Spacewalk-list at redhat.com
>> https://www.redhat.com/mailman/listinfo/spacewalk-list
>>
>
>
> _______________________________________________
> Spacewalk-list mailing list
> Spacewalk-list at redhat.com
> https://www.redhat.com/mailman/listinfo/spacewalk-list
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/spacewalk-list/attachments/20141030/eec1b6f1/attachment.htm>
More information about the Spacewalk-list
mailing list