[Spacewalk-list] Spacewalk Startup Problems after yum update

[Boyd, Robert]

I'm running spacewalk 1.7 on RHEL6.3 and Friday I applied yum updates.   This morning I discovered that there is something severely wrong with the server.   I attempted to sign in to the spacewalk management website and got a 404 error:

HTTP Status 404 -
________________________________
type Status report
message
description The requested resource () is not available.
________________________________
Apache Tomcat/6.0.24

I see that there is a series of error messages about JVM failing to start in /var/log/messages:

Sep 24 11:09:00 rhqengbld03 wrapper[2230]: Startup failed: Timed out waiting for signal from JVM.
Sep 24 11:09:01 rhqengbld03 wrapper[2230]: JVM did not exit on request, terminated
Sep 24 11:09:01 rhqengbld03 wrapper[2230]: JVM exited in response to signal SIGKILL (9).
Sep 24 11:09:05 rhqengbld03 wrapper[2230]: Launching a JVM...
Sep 24 11:09:34 rhqengbld03 wrapper[2230]: Startup failed: Timed out waiting for signal from JVM.
Sep 24 11:09:34 rhqengbld03 wrapper[2230]: JVM did not exit on request, terminated
Sep 24 11:09:35 rhqengbld03 wrapper[2230]: JVM exited in response to signal SIGKILL (9).
Sep 24 11:09:39 rhqengbld03 wrapper[2230]: Launching a JVM...
Sep 24 11:10:08 rhqengbld03 wrapper[2230]: Startup failed: Timed out waiting for signal from JVM.
Sep 24 11:10:08 rhqengbld03 wrapper[2230]: JVM did not exit on request, terminated
Sep 24 11:10:09 rhqengbld03 wrapper[2230]: JVM exited in response to signal SIGKILL (9).
Sep 24 11:10:13 rhqengbld03 wrapper[2230]: Launching a JVM...
Sep 24 11:10:42 rhqengbld03 wrapper[2230]: Startup failed: Timed out waiting for signal from JVM.
Sep 24 11:10:42 rhqengbld03 wrapper[2230]: JVM did not exit on request, terminated
Sep 24 11:10:43 rhqengbld03 wrapper[2230]: JVM exited in response to signal SIGKILL (9).
Sep 24 11:10:47 rhqengbld03 wrapper[2230]: Launching a JVM...
Sep 24 11:11:16 rhqengbld03 wrapper[2230]: Startup failed: Timed out waiting for signal from JVM.
Sep 24 11:11:16 rhqengbld03 wrapper[2230]: JVM did not exit on request, terminated
Sep 24 11:11:17 rhqengbld03 wrapper[2230]: JVM exited in response to signal SIGKILL (9).
Sep 24 11:11:17 rhqengbld03 wrapper[2230]: There were 5 failed launches in a row, each lasting less than 300 seconds.  Giving up.
Sep 24 11:11:17 rhqengbld03 wrapper[2230]:   There may be a configuration problem: please check the logs.
Sep 24 11:11:17 rhqengbld03 wrapper[2230]: <-- Wrapper Stopped

Looking at /var/log/tomcat6/catalina logs I found several messages.   One that looks like it might be central to the problems is this:

INFO: Deploying configuration descriptor rhn.xml
Sep 24, 2012 11:08:12 AM org.apache.catalina.loader.WebappLoader start
SEVERE: LifecycleException
java.io.IOException: Failed to access resource /WEB-INF/lib/postgresql-jdbc.jar
        at org.apache.catalina.loader.WebappLoader.setRepositories(WebappLoader.java:1022)
        at org.apache.catalina.loader.WebappLoader.start(WebappLoader.java:653)
        at org.apache.catalina.core.StandardContext.start(StandardContext.java:4341)
        at org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:791)
        at org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:771)
        at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:526)
        at org.apache.catalina.startup.HostConfig.deployDescriptor(HostConfig.java:637)
        at org.apache.catalina.startup.HostConfig.deployDescriptors(HostConfig.java:563)
        at org.apache.catalina.startup.HostConfig.deployApps(HostConfig.java:498)


I looked for the listed postgresql-jdbc.jar file and see that there is a failed link:

lrwxrwxrwx. 1 tomcat tomcat 35 Aug  3 19:02 /var/lib/tomcat6/webapps/rhn/WEB-INF/lib/postgresql-jdbc.jar -> /usr/share/java/postgresql-jdbc.jar

In /usr/share/java there are these 2 files:

-rwxr-xr-x. 1 root root 502118 Nov 18  2010 postgresql-8.4-702.jdbc3.jar
-rwxr-xr-x. 1 root root 539510 Nov 18  2010 postgresql-8.4-702.jdbc4.jar

There is no plain vanilla versionless postgresql-jdbc.jar

So I linked the missing name to the postgresql-8.4-702.jdbc4.jar to see what will happen.

I restarted the server and now the spacewalk web service appears to be fully operational again.


My question is:  why would this link have been trashed?  Is there anything I can do preemptively to prevent getting caught like this again?


Robert Boyd
Senior Systems Engineer
Phone: 919-645-2972
Mobile: 919-306-4681
Peoplefluent
434 Fayetteville Street
Raleigh, NC  27601

robert.boyd at peoplefluent.com<mailto:michael.dileonardo at peoplefluent.com>

[cid:image001.png at 01CD9A4E.D98CEAB0]



This email message is for the sole use of the intended recipient(s) and may contain confidential information. Any unauthorized review, use, disclosure or distribution is prohibited. If you are not the intended recipient, please contact the sender by reply email and destroy all copies of the original message.

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/spacewalk-list/attachments/20120924/f3301756/attachment.htm>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image001.png
Type: image/png
Size: 4488 bytes
Desc: image001.png
URL: <http://listman.redhat.com/archives/spacewalk-list/attachments/20120924/f3301756/attachment.png>