[Spacewalk-list] ? Failures Cloning Channels with spacewalk-manage-channel-lifecycle

[Boyd, Robert]

Here's some additional info about one of the failure modes I'm seeing.  I kept cranking up the memory limits until I now have the limits at default =4192 max=16388.   Some channels are handled ok by this tool now.   

Others fail pretty quickly while clearing the first target channel.

One thing I see that concerns me are processes I see running Java that don't have the increased memory allocations.   Like this one:

root      3595  3349  6 14:32 ?        00:00:13 
/usr/bin/java -Dibm.dst.compatibility=true -Xms1024m -Xmx4096m 
-Djava.library.path=/usr/lib:/usr/lib64:/usr/lib/oracle/11.2/client64/lib:/usr/lib/oracle/11.2/client/lib 
-classpath /usr/share/java/tanukiwrapper.jar:/usr/share/rhn/classes:/usr/share/java/struts.jar:/usr/share/java/jfreechart.jar:/usr/share/java/jpam.jar:/usr/share/java/javamail.jar:/usr/share/java/axis/axis-ant.jar:/usr/share/java/quartz.jar:/usr/share/java/commons-codec.jar:/usr/share/java/commons-beanutils.jar:/usr/share/java/ojdbc14.jar:/usr/share/java/jta.jar:/usr/share/java/concurrent.jar:/usr/share/rhn/lib/spacewalk-asm.jar:/usr/share/java/axis/jaxrpc.jar:/usr/share/java/commons-collections.jar:/usr/share/java/taglibs-standard.jar:/usr/share/java/axis/axis.jar:/usr/share/java/xalan-j2.jar:/usr/share/java/commons-validator.jar:/usr/share/java/asm/asm-attrs.jar:/usr/share/java/jaf.jar:/usr/share/java/jdom.jar:/usr/share/java/oro.jar:/usr/share/rhn/lib/rhn.jar:/usr/share/java/redstone-xmlrpc.jar:/usr/share/java/oscache.jar:/usr/share/java/log4j.jar:/usr/share/java/wsdl4j.jar:/usr/share/java/jcommon.jar:/usr/share/java/commons-el.jar:/usr/share/java/taglibs-core.jar:/usr/share/java/commons-lang.jar:/usr/share/java/commons-digester.jar:/usr/share/java/jasper5-runtime.jar:/usr/share/java/jspapi.jar:/usr/share/java/c3p0.jar:/usr/share/java/sitemesh.jar:/usr/share/java/jasper5-compiler.jar:/usr/share/java/axis/saaj.jar:/usr/share/java/commons-logging.jar:/usr/share/java/commons-discovery.jar:/usr/share/java/xml-commons-apis.jar:/usr/share/java/axis/jaxrpc.jar:/usr/share/java/commons-cli.jar:/usr/share/java/bcel.jar:/usr/share/java/antlr.jar:/usr/share/java/xerces-j2.jar:/usr/share/java/hibernate3.jar:/usr/share/java/cglib.jar:/usr/share/java/dom4j.jar:/usr/share/rhn/lib/java-branding.jar:/usr/share/java/slf4j/api.jar:/usr/share/java/slf4j/jcl.jar:/usr/share/java/simple-core.jar:/usr/share/java/commons-dbcp.jar:/usr/share/java/commons-pool.jar:/usr/share/java/quartz-oracle.jar:/usr/share/java/postgresql-jdbc.jar:/usr/share/java/mchange-commons.jar:/usr/share/java/commons-io.jar 
-Dwrapper.key=Ny4YUY7QmPOi432B 
-Dwrapper.port=32001 
-Dwrapper.jvm.port.min=31000 
-Dwrapper.jvm.port.max=31999 
-Dwrapper.pid=3349 -Dwrapper.version=3.2.3 
-Dwrapper.native_library=wrapper 
-Dwrapper.service=TRUE 
-Dwrapper.cpu.timeout=10 
-Dwrapper.jvmid=3
com.redhat.rhn.taskomatic.core.TaskomaticDaemon

What is the configuration file that controls the memory allocation for this process?

I already modified /usr/share/rhn/config-defaults/rhn_taskomatic_daemon.conf:

# Initial Java Heap Size (in MB)
wrapper.java.initmemory=4196

# Maximum Java Heap Size (in MB)
wrapper.java.maxmemory=16388

If the process I show is supposed to be controlled by this file, the memory allocation specified is being ignored, or applied in some way that doesn't match with the process shown.


: spacewalk-list-bounces at redhat.com [mailto:spacewalk-list-bounces at redhat.com] On Behalf Of Boyd, Robert
Sent: Friday, January 29, 2016 12:27 PM
To: spacewalk-list at redhat.com
Subject: Re: [Spacewalk-list] ? Failures Cloning Channels with spacewalk-manage-channel-lifecycle

Thanks Avi,

I tried what you suggested and got some minor improvement -- the tool runs longer before things blow up.   I tried increasing the memory limits even more -- but that doesn't seem to make any difference.   It's still eventually blowing up the same way, and the logs you suggested looking at don't show anything interesting around the time of the failures.

Any other suggestions?   I could keep increasing the memory limits even farther, but I'm not sure that's a good idea.

Thanks,
Robert

-----Original Message-----
From: spacewalk-list-bounces at redhat.com [mailto:spacewalk-list-bounces at redhat.com] On Behalf Of Avi Miller
Sent: Thursday, January 28, 2016 6:12 PM
To: spacewalk-list at redhat.com
Subject: Re: [Spacewalk-list] ? Failures Cloning Channels with spacewalk-manage-channel-lifecycle

Hey,

> On 29 Jan 2016, at 9:56 AM, Boyd, Robert <Robert.Boyd at peoplefluent.com> wrote:
> 
> I have tried poking at log files, but I haven’t found where there seems to be an indication of a problem.   Can someone point me to a prioritized list of logs I should be looking at to find the nature of what’s dying on the back end?  I suspect there is a java resource failure or some other error and would like to track it down and fix it rather than resort to manual/scripted channel by channel cloning( which might fail in the same way).

You may want to increase the memory available to Tomcat and Taskomatic: 

http://docs.oracle.com/cd/E37670_01/E58680/html/sw22-issues-memory.html

This feels like a Tomcat issue to me, the logs are in /var/log/tomcat6 on the Spacewalk server. The Taskomatic logs are in /var/log/rhn/rhn_taskomatic_daemon.log

Cheers,
Avi

--
Oracle <http://www.oracle.com>
Avi Miller | Product Management Director | +61 (3) 8616 3496 Oracle Linux and Virtualization
417 St Kilda Road, Melbourne, Victoria 3004 Australia