[Spacewalk-list] Performance Tuning

[Matthew Madey]

Looking to get some feedback on performance settings any of you may have
done on your Spacewalk server and\or external Oracle Database servers.

I'm looking to patch 1600 systems simultaneously to meet our business
requirements.. at our current settings with an 8 CPU \ 32 GB RAM Spacewalk
server, I can patch 400 systems simultaneously without much issue. Much of
our bottleneck appears to be disk await time on the Oracle database side,
which we are addressing, but I'm wondering if there are other
kernel\tomcat\apache settings I should be looking at to improve handling of
all the XMLRPC traffic coming in to the Spacewalk server.
400 systems appears to be our hard limit at the moment, adding just an
additional 50 or so leads to some pretty catastrophic effects. I haven't
seen any GC or JVM errors in catalina.out thus far, but our memory usage
goes through the roof and begin swapping badly. My guess is that the wait
time on the database is so bad the Spacewalk server starts backlogging
requests until it runs out of memory.

Has anyone else attempted to patch this volume of systems at one time? If
so, what changes did you make to the server configuration to accommodate
that?



Below are some of the network tuning parameters we've entered.

net.ipv4.tcp_max_syn_backlog = 8192
net.ipv4.tcp_syncookies = 1
net.ipv4.conf.all.rp_filter = 1
net.ipv4.conf.all.accept_source_route = 0
net.ipv4.conf.all.accept_redirects = 0
net.ipv4.conf.all.secure_redirects = 0
net.ipv4.conf.default.rp_filter = 1
net.ipv4.conf.default.accept_source_route = 0
net.ipv4.conf.default.accept_redirects = 0
net.ipv4.conf.default.secure_redirects = 0
net.ipv4.icmp_echo_ignore_broadcasts = 1
net.ipv4.ip_forward = 0
net.ipv4.conf.all.send_redirects = 0
net.ipv4.conf.default.send_redirects = 0
net.ipv4.neigh.default.unres_qlen = 3

#Added OSE Tuning
###8x normal for faster network queue draining
net.core.dev_weight = 512
#
###3x normal for a queue and budget suited to networks greater than 100mbps
net.core.netdev_budget = 10000
net.core.netdev_max_backlog = 30000
net.core.rmem_max = 4194304
net.core.wmem_max = 1048586

####min, default (BDP), max (2xBDP) for window scaling (BDP calculated for
10gbps link speed and near zero latency in a LAN)
net.ipv4.tcp_rmem = 4096 87380  4194304
net.ipv4.tcp_wmem = 4096 16384 4194304

####Affects UDP streams similarly to the above for TCP.
net.core.rmem_default = 312500
net.core.wmem_default = 312500

###Enable Packetization Layer Path MTU Discovery (PLPMTUD)
net.ipv4.tcp_mtu_probing = 1

net.core.somaxconn = 1536
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/spacewalk-list/attachments/20140418/70a46699/attachment.htm>