[Spacewalk-list] Taskomatic won't start in 0.8 and 1.0

Mike Hanby mhanby at uab.edu
Tue May 18 17:48:02 UTC 2010


Howdy Justin and Brian,

Yup, that was me, aka flakrat :-)

Brian, yep, I read through your email thread before I got to this one and I am also unable to connect to http://{127.0.0.1,FQN}/rpc/api but can do so over https. I'm getting the same errors in /var/log/httpd/error_log whenever I attempt to start taskomatic.

My zz-spacewalk-www.conf file looks like this at the bottom (where there are two rewrites concerning rpc):

# Turn rewrite engine on so we can use it for
# kickstart requests.
RewriteEngine on
RewriteOptions inherit
SSLProxyEngine on

<IfModule proxy_ajp_module>
RewriteRule ^/rhn(.*) ajp://localhost:8009/rhn$1 [P]
RewriteRule ^(/.*\.(do|jsp)(\?.*)?)$ ajp://localhost:8009/$1 [P]
</IfModule>

RedirectMatch ^/renew/.* http://rhn.redhat.com
RedirectMatch ^/index\.html$ /rhn/Login.do


# Disable TRACE and TRACK
RewriteCond %{REQUEST_METHOD} ^(TRACE|TRACK)
RewriteRule .* - [F]

ServerTokens Prod
ServerSignature Off

# This rule handles incoming kickstart file requests from
# machines actually performing a kickstart. This rule
# processes the incoming URL and converts it into something
# slightly more Struts friendly.
RewriteRule ^/ks/cfg([-a-zA-Z0-9\._/\%\ ]*)$ /rhn/kickstart/DownloadFile.do?ksurl=$1 [P]
RewriteRule ^/download/(.*)$ /rhn/common/DownloadFile.do?url=$1 [P]
RewriteRule ^/rpc/api /rhn/rpc/api [P]
RewriteRule ^/ks/dist(.*)$ /rhn/common/DownloadFile.do?url=/ks/dist$1 [P]

# For rhn-custom-info
RewriteRule ^/WEBRPC /rhn/rpc/api [P]

-----Original Message-----
From: spacewalk-list-bounces at redhat.com [mailto:spacewalk-list-bounces at redhat.com] On Behalf Of Brian T. O'Neill
Sent: Tuesday, May 18, 2010 10:45 AM
To: spacewalk-list at redhat.com
Subject: Re: [Spacewalk-list] Taskomatic won't start in 0.8 and 1.0

This could be the same issue I'm having. Take a look at
/var/log/httpd/error_log and see if you have:
[Mon May 17 16:32:02 2010] [error] [client 127.0.0.1] File does not
exist: /var/www/html/rpc

They would be showing up at 4 second intervals during each of the 5
attempts  that it tries to start taskomatic.

If it is, sounds like you and I have the same problem of /rpc/api not
working over http.

Brian



-----Original Message-----
From: spacewalk-list-bounces at redhat.com
[mailto:spacewalk-list-bounces at redhat.com] On Behalf Of Justin Sherrill
Sent: Tuesday, May 18, 2010 11:31 AM
To: spacewalk-list at redhat.com
Subject: Re: [Spacewalk-list] Taskomatic won't start in 0.8 and 1.0

On 5/17/10 1:45 PM, Mike Hanby wrote:
> Figured I'd update the thread with my latest tests:
> 
> I stopped all SW processes and renamed /var/cache/rhn, created a new
/var/cache/rhn giving apache ownership and 750 permissions.
> 
> Started everything back up and taskomatic still fails to start up
properly.
> 
> Is there any way to get debugging/logging from the JVM itself, or does
the wrapper provide the only logging?
> 
> I also tried to increasing the ping timeout, but that also did not
result in any different behavior.
> 
> Mike
> 
> -----Original Message-----
> From: spacewalk-list-bounces at redhat.com
[mailto:spacewalk-list-bounces at redhat.com] On Behalf Of Mike Hanby
> Sent: Friday, May 14, 2010 3:43 PM
> To: spacewalk-list at redhat.com
> Subject: Re: [Spacewalk-list] Taskomatic won't start in 0.8 and 1.0
> 
> Here's an example of my EPEL 5 x86_64 channel details page:
> Last Modified:  	2010-05-14 05:16:39 EDT
> Last Repo Build: 	2010-04-21 18:05:08 EDT
> 
> Showing that spacewalk-repo-sync inserted new packages this morning,
yet the repo build is still very old.
> 
> -----Original Message-----
> From: spacewalk-list-bounces at redhat.com
[mailto:spacewalk-list-bounces at redhat.com] On Behalf Of Mike Hanby
> Sent: Friday, May 14, 2010 3:24 PM
> To: spacewalk-list at redhat.com
> Subject: [Spacewalk-list] Taskomatic won't start in 0.8 and 1.0
> 
> Howdy,
> 
> The last time any of my /var/cache/rhn/repodata was updated was Apr
23. This was also the day I upgraded to 0.8 from 0.7 AND ported my
database from OracleXE to Oracle 11gR2.
> 
> Everything else appears to be working. My nightly
/usr/bin/spacewalk-repo-sync successfully pulls updated packages from
the various public mirrors and inserts them into the database. I can
browse the Spacewalk web interface and perform searches to see the
latest packages and my SW clients can all download packages via yum from
the spacewalk server. The catch there is, only packages that were part
of the system Apr 23 and prior since the repodata is old.
> 
> I enabled DEBUG in the rhn_taskomatic_daemon.conf file for console and
started taskomatic to gather the output:
> 
> # /etc/init.d/taskomatic console
> Running RHN Taskomatic...
> wrapper  | --> Wrapper Started as Console
> wrapper  | Using tick timer.
> wrapperp | server listening on port 32001.
> wrapper  | Command[0] : /usr/bin/java
> wrapper  | Command[1] : -Dibm.dst.compatibility=true
> wrapper  | Command[2] : -Xms256m
> wrapper  | Command[3] : -Xmx512m
> wrapper  | Command[4] : -Djava.library.path=/usr/lib:/usr/lib64
> wrapper  | Command[5] : -classpath
> wrapper  | Command[6] :
/usr/share/java/tanukiwrapper.jar:/usr/share/rhn/classes:/usr/share/java
/struts.jar:/usr/share/java/jfreechart.jar:/usr/share/java/jpam.jar:/usr
/share/java/javamail.jar:/usr/share/java/axis/axis-ant.jar:/usr/share/ja
va/quartz.jar:/usr/share/java/commons-codec.jar:/usr/share/java/commons-
beanutils.jar:/usr/share/java/ojdbc14.jar:/usr/share/java/jta.jar:/usr/s
hare/java/concurrent.jar:/usr/share/java/asm/asm.jar:/usr/share/java/axi
s/jaxrpc.jar:/usr/share/java/commons-collections.jar:/usr/share/java/tag
libs-standard.jar:/usr/share/java/axis/axis.jar:/usr/share/java/xalan-j2
.jar:/usr/share/java/commons-validator.jar:/usr/share/java/asm/asm-attrs
.jar:/usr/share/java/jaf.jar:/usr/share/java/jdom.jar:/usr/share/java/or
o.jar:/usr/share/rhn/lib/rhn.jar:/usr/share/java/redstone-xmlrpc.jar:/us
r/share/java/oscache.jar:/usr/share/java/log4j.jar:/usr/share/java/wsdl4
j.jar:/usr/share/java/jcommon.jar:/usr/share/java/commons-el.jar:/usr/sh
are/java/taglibs-core.jar:/u
s!
>
r/share/java/commons-lang.jar:/usr/share/java/commons-digester.jar:/usr/
share/java/jasper5-runtime.jar:/usr/share/java/jspapi.jar:/usr/share/jav
a/commons-fileupload.jar:/usr/share/java/c3p0.jar:/usr/share/java/siteme
sh.jar:/usr/share/java/jasper5-compiler.jar:/usr/share/java/axis/saaj.ja
r:/usr/share/java/commons-logging.jar:/usr/share/java/commons-discovery.
jar:/usr/share/java/xml-commons-apis.jar:/usr/share/java/axis/jaxrpc.jar
:/usr/share/java/commons-cli.jar:/usr/share/java/bcel.jar:/usr/share/jav
a/antlr.jar:/usr/share/java/xerces-j2.jar:/usr/share/java/hibernate3.jar
:/usr/share/java/cglib.jar:/usr/share/java/dom4j.jar:/usr/share/rhn/lib/
java-branding.jar
> wrapper  | Command[7] : -Dwrapper.key=gR3LlUzzdxJT0EFJ
> wrapper  | Command[8] : -Dwrapper.port=32001
> wrapper  | Command[9] : -Dwrapper.jvm.port.min=31000
> wrapper  | Command[10] : -Dwrapper.jvm.port.max=31999
> wrapper  | Command[11] : -Dwrapper.debug=TRUE
> wrapper  | Command[12] : -Dwrapper.pid=6329
> wrapper  | Command[13] : -Dwrapper.version=3.2.1
> wrapper  | Command[14] : -Dwrapper.native_library=wrapper
> wrapper  | Command[15] : -Dwrapper.cpu.timeout=10
> wrapper  | Command[16] : -Dwrapper.jvmid=1
> wrapper  | Command[17] :
com.redhat.rhn.taskomatic.core.TaskomaticDaemon
> wrapper  | Launching a JVM...
> jvm 1    | WrapperManager class initialized by thread: main  Using
classloader: sun.misc.Launcher$AppClassLoader at 64601bb1
> jvm 1    | Wrapper (Version 3.2.1) http://wrapper.tanukisoftware.org
> jvm 1    | 
> jvm 1    | Wrapper Manager: JVM #1
> jvm 1    | Running a 64-bit JVM.
> jvm 1    | Wrapper Manager: Registering shutdown hook
> jvm 1    | Wrapper Manager: Using wrapper
> jvm 1    | Load native library.  One or more attempts may fail if
platform specific libraries do not exist.
> jvm 1    | Loading native library failed: libwrapper-linux-x86-64.so
Cause: java.lang.UnsatisfiedLinkError: no wrapper-linux-x86-64 in
java.library.path
> jvm 1    | Loaded native library: libwrapper.so
> jvm 1    | Calling native initialization method.
> jvm 1    | Inside native WrapperManager initialization method
> jvm 1    | Java Version   : 1.6.0-b09 OpenJDK 64-Bit Server VM
> jvm 1    | Java VM Vendor : Sun Microsystems Inc.
> jvm 1    | 
> jvm 1    |
WrapperManager.start(com.redhat.rhn.taskomatic.core.TaskomaticDaemon at 348
7a5cc, args[]) called by thread: main
> jvm 1    | Control event monitor thread started.
> jvm 1    | Startup runner thread started.
> jvm 1    | Communications runner thread started.
> jvm 1    | Open socket to wrapper...Wrapper-Connection
> jvm 1    | Failed attempt to bind using local port 31000
> jvm 1    | Opened Socket from 31001 to 32001
> jvm 1    | Send a packet KEY : gR3LlUzzdxJT0EFJ
> jvm 1    |
handleSocket(Socket[addr=/127.0.0.1,port=32001,localport=31001])
> wrapperp | accepted a socket from 127.0.0.1 on port 31001
> wrapperp | read a packet KEY : gR3LlUzzdxJT0EFJ
> wrapper  | Got key from JVM: gR3LlUzzdxJT0EFJ
> wrapperp | send a packet LOW_LOG_LEVEL : 1
> wrapperp | send a packet PING_TIMEOUT : 30
> wrapperp | send a packet PROPERTIES : (Property Values)
> wrapper  | Start Application.
> wrapperp | send a packet START : start
> jvm 1    | Received a packet LOW_LOG_LEVEL : 1
> jvm 1    | Wrapper Manager: LowLogLevel from Wrapper is 1
> jvm 1    | Received a packet PING_TIMEOUT : 30
> jvm 1    | PingTimeout from Wrapper is 30000
> jvm 1    | Received a packet PROPERTIES : (Property Values)
> jvm 1    | Received a packet START : start
> jvm 1    | calling WrapperListener.start()
> jvm 1    | Waiting for WrapperListener.start runner thread to
complete.
> jvm 1    | WrapperListener.start runner thread started.
> wrapper  | Startup failed: Timed out waiting for signal from JVM.
> wrapper  | JVM did not exit on request, terminated
> wrapper  | Signal trapped.  Details:
> wrapper  |   signal number=17 (SIGCHLD), source="unknown"
> wrapper  | Received SIGCHLD, calling wait().
> wrapper  | wait() returned, child process should be gone.
> wrapperp | server listening on port 32001.
> wrapper  | JVM was only running for 29 seconds leading to a failed
restart count of 1.
> wrapper  | Waiting 5 seconds before launching another JVM.
> wrapper  | Command[0] : /usr/bin/java
> wrapper  | Command[1] : -Dibm.dst.compatibility=true
> wrapper  | Command[2] : -Xms256m
> wrapper  | Command[3] : -Xmx512m
> wrapper  | Command[4] : -Djava.library.path=/usr/lib:/usr/lib64
> wrapper  | Command[5] : -classpath
> ... repeats thru jvm 5 ...
> jvm 5    | Received a packet PROPERTIES : (Property Values)
> jvm 5    | Received a packet START : start
> jvm 5    | calling WrapperListener.start()
> jvm 5    | Waiting for WrapperListener.start runner thread to
complete.
> jvm 5    | WrapperListener.start runner thread started.
> wrapper  | Startup failed: Timed out waiting for signal from JVM.
> wrapper  | JVM did not exit on request, terminated
> wrapper  | Signal trapped.  Details:
> wrapper  |   signal number=17 (SIGCHLD), source="unknown"
> wrapper  | Received SIGCHLD, calling wait().
> wrapper  | wait() returned, child process should be gone.
> wrapperp | server listening on port 32001.
> wrapper  | JVM was only running for 29 seconds leading to a failed
restart count of 5.
> wrapper  | There were 5 failed launches in a row, each lasting less
than 300 seconds.  Giving up.
> wrapper  |   There may be a configuration problem: please check the
logs.
> wrapper  | <-- Wrapper Stopped
> 
> 
> I tried deleting the contents of the rhnTaskQueue, but the taskomatic
service still fails to start
> $ sqlpus '/ as sysdba'
> SQL> delete from SPACEWALK.rhnTaskQueue;
> 
> 20 rows deleted.
> 
> SQL> select * from SPACEWALK.rhnTaskQueue;
> 
> no rows selected
> 
> 
> Anyone have any suggestions on what to try next? While I debug this
issue, is there a way from the command line to force the repodata
creation for each channel?
> 
> Thanks, Mike


Hey Mike,

Someone was on #spaceawlk the other day with this exact problem that we
couldn't figure out.  Was it you (just curious)?


I'd try this:
edit /usr/share/rhn/classes/log4j.properties

and add:

log4j.logger.com.redhat.rhn.taskomatic=DEBUG

Then restart taskomatic and monitor /var/log/rhn/rhn_taskomatic.log


Also, can you upload your  /etc/rhn/default/rhn_taskomatic*  files?

Thanks

-Justin Sherrill

> 
> =================================
> Mike Hanby
> mhanby at uab.edu
> Information Systems Specialist II
> IT HPCS / Research Computing
> 
> 
> 
> _______________________________________________
> Spacewalk-list mailing list
> Spacewalk-list at redhat.com
> https://www.redhat.com/mailman/listinfo/spacewalk-list
> 
> _______________________________________________
> Spacewalk-list mailing list
> Spacewalk-list at redhat.com
> https://www.redhat.com/mailman/listinfo/spacewalk-list
> 
> _______________________________________________
> Spacewalk-list mailing list
> Spacewalk-list at redhat.com
> https://www.redhat.com/mailman/listinfo/spacewalk-list


-- 
Justin Sherrill, RHCA          1801 Varsity Drive.
Software Engineer                Raleigh, NC 27603
Red Hat, Inc.

_______________________________________________
Spacewalk-list mailing list
Spacewalk-list at redhat.com
https://www.redhat.com/mailman/listinfo/spacewalk-list


_______________________________________________
Spacewalk-list mailing list
Spacewalk-list at redhat.com
https://www.redhat.com/mailman/listinfo/spacewalk-list




More information about the Spacewalk-list mailing list