[Spacewalk-list] Taskomatic won't start in 0.8 and 1.0
Justin Sherrill
jsherril at redhat.com
Tue May 18 15:30:33 UTC 2010
On 5/17/10 1:45 PM, Mike Hanby wrote:
> Figured I'd update the thread with my latest tests:
>
> I stopped all SW processes and renamed /var/cache/rhn, created a new /var/cache/rhn giving apache ownership and 750 permissions.
>
> Started everything back up and taskomatic still fails to start up properly.
>
> Is there any way to get debugging/logging from the JVM itself, or does the wrapper provide the only logging?
>
> I also tried to increasing the ping timeout, but that also did not result in any different behavior.
>
> Mike
>
> -----Original Message-----
> From: spacewalk-list-bounces at redhat.com [mailto:spacewalk-list-bounces at redhat.com] On Behalf Of Mike Hanby
> Sent: Friday, May 14, 2010 3:43 PM
> To: spacewalk-list at redhat.com
> Subject: Re: [Spacewalk-list] Taskomatic won't start in 0.8 and 1.0
>
> Here's an example of my EPEL 5 x86_64 channel details page:
> Last Modified: 2010-05-14 05:16:39 EDT
> Last Repo Build: 2010-04-21 18:05:08 EDT
>
> Showing that spacewalk-repo-sync inserted new packages this morning, yet the repo build is still very old.
>
> -----Original Message-----
> From: spacewalk-list-bounces at redhat.com [mailto:spacewalk-list-bounces at redhat.com] On Behalf Of Mike Hanby
> Sent: Friday, May 14, 2010 3:24 PM
> To: spacewalk-list at redhat.com
> Subject: [Spacewalk-list] Taskomatic won't start in 0.8 and 1.0
>
> Howdy,
>
> The last time any of my /var/cache/rhn/repodata was updated was Apr 23. This was also the day I upgraded to 0.8 from 0.7 AND ported my database from OracleXE to Oracle 11gR2.
>
> Everything else appears to be working. My nightly /usr/bin/spacewalk-repo-sync successfully pulls updated packages from the various public mirrors and inserts them into the database. I can browse the Spacewalk web interface and perform searches to see the latest packages and my SW clients can all download packages via yum from the spacewalk server. The catch there is, only packages that were part of the system Apr 23 and prior since the repodata is old.
>
> I enabled DEBUG in the rhn_taskomatic_daemon.conf file for console and started taskomatic to gather the output:
>
> # /etc/init.d/taskomatic console
> Running RHN Taskomatic...
> wrapper | --> Wrapper Started as Console
> wrapper | Using tick timer.
> wrapperp | server listening on port 32001.
> wrapper | Command[0] : /usr/bin/java
> wrapper | Command[1] : -Dibm.dst.compatibility=true
> wrapper | Command[2] : -Xms256m
> wrapper | Command[3] : -Xmx512m
> wrapper | Command[4] : -Djava.library.path=/usr/lib:/usr/lib64
> wrapper | Command[5] : -classpath
> wrapper | Command[6] : /usr/share/java/tanukiwrapper.jar:/usr/share/rhn/classes:/usr/share/java/struts.jar:/usr/share/java/jfreechart.jar:/usr/share/java/jpam.jar:/usr/share/java/javamail.jar:/usr/share/java/axis/axis-ant.jar:/usr/share/java/quartz.jar:/usr/share/java/commons-codec.jar:/usr/share/java/commons-beanutils.jar:/usr/share/java/ojdbc14.jar:/usr/share/java/jta.jar:/usr/share/java/concurrent.jar:/usr/share/java/asm/asm.jar:/usr/share/java/axis/jaxrpc.jar:/usr/share/java/commons-collections.jar:/usr/share/java/taglibs-standard.jar:/usr/share/java/axis/axis.jar:/usr/share/java/xalan-j2.jar:/usr/share/java/commons-validator.jar:/usr/share/java/asm/asm-attrs.jar:/usr/share/java/jaf.jar:/usr/share/java/jdom.jar:/usr/share/java/oro.jar:/usr/share/rhn/lib/rhn.jar:/usr/share/java/redstone-xmlrpc.jar:/usr/share/java/oscache.jar:/usr/share/java/log4j.jar:/usr/share/java/wsdl4j.jar:/usr/share/java/jcommon.jar:/usr/share/java/commons-el.jar:/usr/share/java/taglibs-core.jar:/u
s!
> r/share/java/commons-lang.jar:/usr/share/java/commons-digester.jar:/usr/share/java/jasper5-runtime.jar:/usr/share/java/jspapi.jar:/usr/share/java/commons-fileupload.jar:/usr/share/java/c3p0.jar:/usr/share/java/sitemesh.jar:/usr/share/java/jasper5-compiler.jar:/usr/share/java/axis/saaj.jar:/usr/share/java/commons-logging.jar:/usr/share/java/commons-discovery.jar:/usr/share/java/xml-commons-apis.jar:/usr/share/java/axis/jaxrpc.jar:/usr/share/java/commons-cli.jar:/usr/share/java/bcel.jar:/usr/share/java/antlr.jar:/usr/share/java/xerces-j2.jar:/usr/share/java/hibernate3.jar:/usr/share/java/cglib.jar:/usr/share/java/dom4j.jar:/usr/share/rhn/lib/java-branding.jar
> wrapper | Command[7] : -Dwrapper.key=gR3LlUzzdxJT0EFJ
> wrapper | Command[8] : -Dwrapper.port=32001
> wrapper | Command[9] : -Dwrapper.jvm.port.min=31000
> wrapper | Command[10] : -Dwrapper.jvm.port.max=31999
> wrapper | Command[11] : -Dwrapper.debug=TRUE
> wrapper | Command[12] : -Dwrapper.pid=6329
> wrapper | Command[13] : -Dwrapper.version=3.2.1
> wrapper | Command[14] : -Dwrapper.native_library=wrapper
> wrapper | Command[15] : -Dwrapper.cpu.timeout=10
> wrapper | Command[16] : -Dwrapper.jvmid=1
> wrapper | Command[17] : com.redhat.rhn.taskomatic.core.TaskomaticDaemon
> wrapper | Launching a JVM...
> jvm 1 | WrapperManager class initialized by thread: main Using classloader: sun.misc.Launcher$AppClassLoader at 64601bb1
> jvm 1 | Wrapper (Version 3.2.1) http://wrapper.tanukisoftware.org
> jvm 1 |
> jvm 1 | Wrapper Manager: JVM #1
> jvm 1 | Running a 64-bit JVM.
> jvm 1 | Wrapper Manager: Registering shutdown hook
> jvm 1 | Wrapper Manager: Using wrapper
> jvm 1 | Load native library. One or more attempts may fail if platform specific libraries do not exist.
> jvm 1 | Loading native library failed: libwrapper-linux-x86-64.so Cause: java.lang.UnsatisfiedLinkError: no wrapper-linux-x86-64 in java.library.path
> jvm 1 | Loaded native library: libwrapper.so
> jvm 1 | Calling native initialization method.
> jvm 1 | Inside native WrapperManager initialization method
> jvm 1 | Java Version : 1.6.0-b09 OpenJDK 64-Bit Server VM
> jvm 1 | Java VM Vendor : Sun Microsystems Inc.
> jvm 1 |
> jvm 1 | WrapperManager.start(com.redhat.rhn.taskomatic.core.TaskomaticDaemon at 3487a5cc, args[]) called by thread: main
> jvm 1 | Control event monitor thread started.
> jvm 1 | Startup runner thread started.
> jvm 1 | Communications runner thread started.
> jvm 1 | Open socket to wrapper...Wrapper-Connection
> jvm 1 | Failed attempt to bind using local port 31000
> jvm 1 | Opened Socket from 31001 to 32001
> jvm 1 | Send a packet KEY : gR3LlUzzdxJT0EFJ
> jvm 1 | handleSocket(Socket[addr=/127.0.0.1,port=32001,localport=31001])
> wrapperp | accepted a socket from 127.0.0.1 on port 31001
> wrapperp | read a packet KEY : gR3LlUzzdxJT0EFJ
> wrapper | Got key from JVM: gR3LlUzzdxJT0EFJ
> wrapperp | send a packet LOW_LOG_LEVEL : 1
> wrapperp | send a packet PING_TIMEOUT : 30
> wrapperp | send a packet PROPERTIES : (Property Values)
> wrapper | Start Application.
> wrapperp | send a packet START : start
> jvm 1 | Received a packet LOW_LOG_LEVEL : 1
> jvm 1 | Wrapper Manager: LowLogLevel from Wrapper is 1
> jvm 1 | Received a packet PING_TIMEOUT : 30
> jvm 1 | PingTimeout from Wrapper is 30000
> jvm 1 | Received a packet PROPERTIES : (Property Values)
> jvm 1 | Received a packet START : start
> jvm 1 | calling WrapperListener.start()
> jvm 1 | Waiting for WrapperListener.start runner thread to complete.
> jvm 1 | WrapperListener.start runner thread started.
> wrapper | Startup failed: Timed out waiting for signal from JVM.
> wrapper | JVM did not exit on request, terminated
> wrapper | Signal trapped. Details:
> wrapper | signal number=17 (SIGCHLD), source="unknown"
> wrapper | Received SIGCHLD, calling wait().
> wrapper | wait() returned, child process should be gone.
> wrapperp | server listening on port 32001.
> wrapper | JVM was only running for 29 seconds leading to a failed restart count of 1.
> wrapper | Waiting 5 seconds before launching another JVM.
> wrapper | Command[0] : /usr/bin/java
> wrapper | Command[1] : -Dibm.dst.compatibility=true
> wrapper | Command[2] : -Xms256m
> wrapper | Command[3] : -Xmx512m
> wrapper | Command[4] : -Djava.library.path=/usr/lib:/usr/lib64
> wrapper | Command[5] : -classpath
> ... repeats thru jvm 5 ...
> jvm 5 | Received a packet PROPERTIES : (Property Values)
> jvm 5 | Received a packet START : start
> jvm 5 | calling WrapperListener.start()
> jvm 5 | Waiting for WrapperListener.start runner thread to complete.
> jvm 5 | WrapperListener.start runner thread started.
> wrapper | Startup failed: Timed out waiting for signal from JVM.
> wrapper | JVM did not exit on request, terminated
> wrapper | Signal trapped. Details:
> wrapper | signal number=17 (SIGCHLD), source="unknown"
> wrapper | Received SIGCHLD, calling wait().
> wrapper | wait() returned, child process should be gone.
> wrapperp | server listening on port 32001.
> wrapper | JVM was only running for 29 seconds leading to a failed restart count of 5.
> wrapper | There were 5 failed launches in a row, each lasting less than 300 seconds. Giving up.
> wrapper | There may be a configuration problem: please check the logs.
> wrapper | <-- Wrapper Stopped
>
>
> I tried deleting the contents of the rhnTaskQueue, but the taskomatic service still fails to start
> $ sqlpus '/ as sysdba'
> SQL> delete from SPACEWALK.rhnTaskQueue;
>
> 20 rows deleted.
>
> SQL> select * from SPACEWALK.rhnTaskQueue;
>
> no rows selected
>
>
> Anyone have any suggestions on what to try next? While I debug this issue, is there a way from the command line to force the repodata creation for each channel?
>
> Thanks, Mike
Hey Mike,
Someone was on #spaceawlk the other day with this exact problem that we
couldn't figure out. Was it you (just curious)?
I'd try this:
edit /usr/share/rhn/classes/log4j.properties
and add:
log4j.logger.com.redhat.rhn.taskomatic=DEBUG
Then restart taskomatic and monitor /var/log/rhn/rhn_taskomatic.log
Also, can you upload your /etc/rhn/default/rhn_taskomatic* files?
Thanks
-Justin Sherrill
>
> =================================
> Mike Hanby
> mhanby at uab.edu
> Information Systems Specialist II
> IT HPCS / Research Computing
>
>
>
> _______________________________________________
> Spacewalk-list mailing list
> Spacewalk-list at redhat.com
> https://www.redhat.com/mailman/listinfo/spacewalk-list
>
> _______________________________________________
> Spacewalk-list mailing list
> Spacewalk-list at redhat.com
> https://www.redhat.com/mailman/listinfo/spacewalk-list
>
> _______________________________________________
> Spacewalk-list mailing list
> Spacewalk-list at redhat.com
> https://www.redhat.com/mailman/listinfo/spacewalk-list
--
Justin Sherrill, RHCA 1801 Varsity Drive.
Software Engineer Raleigh, NC 27603
Red Hat, Inc.
More information about the Spacewalk-list
mailing list