[Spacewalk-list] Spacewalk CPU lack / Scaling
Pierre Casenove
pcasenove at gmail.com
Tue Oct 14 07:33:04 UTC 2014
Hello,
I made the same conclusion one year ago about the "All System" performance:
https://www.redhat.com/archives/spacewalk-list/2014-March/msg00072.html
I'm happy to see that someone else noticed that this page is slower on his
set up too!
Pierre
2014-10-14 8:56 GMT+02:00 Nahelou Sébastien <sebastien.nahelou at worldline.com
>:
> Ok, like us.
>
> Thank you for your help.
>
>
>
> Sebastien
>
>
>
> *De :* spacewalk-list-bounces at redhat.com [mailto:
> spacewalk-list-bounces at redhat.com] *De la part de* Matthew Madey
> *Envoyé :* lundi 13 octobre 2014 20:30
>
> *À :* spacewalk-list at redhat.com
> *Objet :* Re: [Spacewalk-list] Spacewalk CPU lack / Scaling
>
>
>
> We use Oracle, so I can't comment on PGSQL directly.. however I believe we
> have our concurrent connection limit set to around 2000.
>
>
>
> On Mon, Oct 13, 2014 at 2:43 AM, Nahelou Sébastien <
> sebastien.nahelou at worldline.com> wrote:
>
> Thank you so much for your answers.
>
>
>
> Hangs happens when we load the "all systems" page when organisation. I
> will try your configuration tuning.
>
>
>
> Last question, what do you preconize for PGSQL tuning ? Particularly, How
> many maximum connections do you configure ?
>
>
>
> Thank again
>
> Sebastien
>
>
>
> *De :* spacewalk-list-bounces at redhat.com [mailto:
> spacewalk-list-bounces at redhat.com] *De la part de* Matthew Madey
> *Envoyé :* vendredi 10 octobre 2014 18:25
> *À :* spacewalk-list at redhat.com
> *Objet :* Re: [Spacewalk-list] Spacewalk CPU lack / Scaling
>
>
>
> We had a similar issue with Tomcat.. our Spacewalk server has 8000+
> clients and is now running smoothly after some performance tuning. See the
> code snippets below, hopefully they'll help you out.
>
>
>
>
>
> Add maxThreads to /etc/tomcat6/server.xml
>
>
>
> <Connector port="8080" protocol="HTTP/1.1" connectionTimeout="20000" redirectPort="8443" URIEncoding="UTF-8" address="127.0.0.1" maxThreads="1024" maxKeepAliveRequests="1000"/>
>
> <!-- A "Connector" using the shared thread pool-->
>
> <!--
>
> <Connector executor="tomcatThreadPool"
>
> port="8080" protocol="HTTP/1.1"
>
> connectionTimeout="20000"
>
> redirectPort="8443" />
>
>
>
> <!-- Define an AJP 1.3 Connector on port 8009 -->
>
> <Connector port="8009" protocol="AJP/1.3" redirectPort="8443" URIEncoding="UTF-8" address="127.0.0.1" maxThreads="1024"/>
>
>
>
> <Connector port="8009" protocol="AJP/1.3" redirectPort="8443" URIEncoding="UTF-8" address="::1" maxThreads="1024"/>
>
>
>
>
>
>
>
>
>
>
>
> Tune Apache to service more requests /etc/httpd/conf.d/zz-spacewalk-server.conf
>
>
>
>
>
> #######################################################
>
> # Authorship and versioning info
>
> # $Author$
>
> # $Date$
>
> # $URL$
>
> # $Rev$
>
> # deployment_location: /etc/httpd/conf.d/
>
> #######################################################
>
> # ** DO NOT EDIT **
>
> # Master configuration file for the rhn_server setup
>
> #
>
>
>
> ##
>
> ## Spacewalk settings
>
> ##
>
>
>
> <VirtualHost *>
>
>
>
> <IfModule mod_jk.c>
>
> # Inherit the mod_jk settings defined in zz-spacewalk-www.conf
>
> JkMountCopy On
>
> </IfModule>
>
>
>
> <Directory "/var/www/html/*">
>
> AllowOverride all
>
> </Directory>
>
>
>
> RewriteEngine on
>
> RewriteOptions inherit
>
> </VirtualHost>
>
>
>
> # Override default httpd prefork settings
>
> <IfModule prefork.c>
>
> StartServers 8
>
> MinSpareServers 400
>
> MaxSpareServers 400
>
> ServerLimit 1024
>
> MaxClients 1024
>
> MaxRequestsPerChild 200
>
> </IfModule>
>
>
>
> Include /etc/rhn/satellite-httpd/conf/rhn/rhn_monitoring.conf
>
>
>
>
>
>
>
>
>
>
>
>
>
> Also added some network tuning to /etc/sysctl.conf
>
>
>
> net.ipv4.icmp_echo_ignore_broadcasts = 1
>
> net.ipv4.conf.all.secure_redirects = 0
>
> net.ipv4.tcp_max_syn_backlog = 8192
>
> net.ipv4.conf.default.secure_redirects = 0
>
> net.ipv4.tcp_syncookies = 1
>
> net.ipv4.conf.all.accept_source_route = 0
>
> net.ipv4.conf.all.rp_filter = 1
>
> net.ipv4.conf.all.send_redirects = 0
>
> net.ipv4.conf.default.accept_redirects = 0
>
> net.ipv4.conf.all.accept_redirects = 0
>
> net.ipv4.conf.default.send_redirects = 0
>
> net.core.somaxconn = 1536
>
> net.core.dev_weight = 512
>
> ##3x normal for a queue and budget suited to networks greater than 100mbps
>
> net.core.netdev_budget = 10000
>
> net.core.netdev_max_backlog = 30000
>
>
>
>
>
> On Fri, Oct 10, 2014 at 4:35 AM, Michael Mraka <michael.mraka at redhat.com>
> wrote:
>
> Nahelou Sébastien wrote:
> % Hi Spacewalk lovers
>
> Hello Sebastian,
>
> % Since we upgrade from 2.0 to 2.2, we encounter a capacity problem due to
> CPU usage overall.
> % When users load a system page in the GUI, the tomcat process grow to 90%
> usage. Api and GUI become unavailable or very slow.
>
> Are you able to identify which page causes such high load?
>
> % Our architecture is as following:
> % 5 rhn proxies
> % LVS Master/slave mode
> % 1 front master: SERVICES="jabberd $TOMCAT httpd osa-dispatcher
> Monitoring MonitoringScout rhn-search cobblerd taskomatic"
> % 1 front slave: SERVICES="jabberd $TOMCAT httpd osa-dispatcher Monitoring
> MonitoringScout "
> % 1 cluster PGSQL
> %
> % Our masters have 2* Intel(R) Xeon(R) CPU E5-2609 / 92 RAM / raid 1 using
> CentOS 6.5.
> %
> % We have about 6000 systems registered, about 40 organizations and more
> than 600 channels (including cloned)
> %
> % My questions are:
> % - Is it possible to scale spacewalk horizontally ? By changing lvs
> configuration to Active/Active ? By using HAProxy ? other ?
>
> We've never tested Active/Active cluster. I dare to say it won't work.
> Our recomendation is to use one Spacewalk Proxy for every 5000 systems.
>
> % - Is it possible to tune tomcat better than adding memory ? (we use
> actually -Xms8196m -Xmx16384m)
>
> I don't know. Frankly I've never heard a report about necessity to do so.
>
> % - What is the maximum number of systems that we can register in 1
> organisation ? (Actually our max seems to be 1700)
>
> We know about installations with tens to hundereds of thousands systems.
>
> % - Is there a maximum number of systems that we can register in a
> Spacewalk cluster ?
>
> No, it's limmited only by numbers in your spacewalk certificate.
> Default certificate is limited to 20000. If you need more you can
> generate your own.
>
> % Thank in advance
> %
> % Sébastien NAHELOU
> % System engineer
> % Worldline - Seclin - France
>
>
> Regards,
>
> --
> Michael Mráka
> Satellite Engineering, Red Hat
>
> _______________________________________________
> Spacewalk-list mailing list
> Spacewalk-list at redhat.com
> https://www.redhat.com/mailman/listinfo/spacewalk-list
>
>
>
>
> ------------------------------
>
>
> Ce message et les pièces jointes sont confidentiels et réservés à l'usage
> exclusif de ses destinataires. Il peut également être protégé par le secret
> professionnel. Si vous recevez ce message par erreur, merci d'en avertir
> immédiatement l'expéditeur et de le détruire. L'intégrité du message ne
> pouvant être assurée sur Internet, la responsabilité de Worldline ne pourra
> être recherchée quant au contenu de ce message. Bien que les meilleurs
> efforts soient faits pour maintenir cette transmission exempte de tout
> virus, l'expéditeur ne donne aucune garantie à cet égard et sa
> responsabilité ne saurait être recherchée pour tout dommage résultant d'un
> virus transmis.
>
> This e-mail and the documents attached are confidential and intended
> solely for the addressee; it may also be privileged. If you receive this
> e-mail in error, please notify the sender immediately and destroy it. As
> its integrity cannot be secured on the Internet, the Worldline liability
> cannot be triggered for the message content. Although the sender endeavours
> to maintain a computer virus-free network, the sender does not warrant that
> this transmission is virus-free and will not be liable for any damages
> resulting from any virus transmitted.
>
>
> _______________________________________________
> Spacewalk-list mailing list
> Spacewalk-list at redhat.com
> https://www.redhat.com/mailman/listinfo/spacewalk-list
>
>
>
> ------------------------------
>
> Ce message et les pièces jointes sont confidentiels et réservés à l'usage
> exclusif de ses destinataires. Il peut également être protégé par le secret
> professionnel. Si vous recevez ce message par erreur, merci d'en avertir
> immédiatement l'expéditeur et de le détruire. L'intégrité du message ne
> pouvant être assurée sur Internet, la responsabilité de Worldline ne pourra
> être recherchée quant au contenu de ce message. Bien que les meilleurs
> efforts soient faits pour maintenir cette transmission exempte de tout
> virus, l'expéditeur ne donne aucune garantie à cet égard et sa
> responsabilité ne saurait être recherchée pour tout dommage résultant d'un
> virus transmis.
>
> This e-mail and the documents attached are confidential and intended
> solely for the addressee; it may also be privileged. If you receive this
> e-mail in error, please notify the sender immediately and destroy it. As
> its integrity cannot be secured on the Internet, the Worldline liability
> cannot be triggered for the message content. Although the sender endeavours
> to maintain a computer virus-free network, the sender does not warrant that
> this transmission is virus-free and will not be liable for any damages
> resulting from any virus transmitted.
>
> _______________________________________________
> Spacewalk-list mailing list
> Spacewalk-list at redhat.com
> https://www.redhat.com/mailman/listinfo/spacewalk-list
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/spacewalk-list/attachments/20141014/c341dd93/attachment.htm>
More information about the Spacewalk-list
mailing list