<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40"><head><meta http-equiv=Content-Type content="text/html; charset=us-ascii"><meta name=Generator content="Microsoft Word 12 (filtered medium)"><!--[if !mso]><style>v\:* {behavior:url(#default#VML);}
o\:* {behavior:url(#default#VML);}
w\:* {behavior:url(#default#VML);}
.shape {behavior:url(#default#VML);}
</style><![endif]--><style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:Tahoma;
panose-1:2 11 6 4 3 5 4 4 2 4;}
@font-face
{font-family:"Arial Narrow";
panose-1:2 11 6 6 2 2 2 3 2 4;}
@font-face
{font-family:Verdana;
panose-1:2 11 6 4 3 5 4 4 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri","sans-serif";}
h1
{mso-style-priority:9;
mso-style-link:"Heading 1 Char";
mso-margin-top-alt:auto;
margin-right:0in;
mso-margin-bottom-alt:auto;
margin-left:0in;
font-size:24.0pt;
font-family:"Times New Roman","serif";
font-weight:bold;}
h3
{mso-style-priority:9;
mso-style-link:"Heading 3 Char";
mso-margin-top-alt:auto;
margin-right:0in;
mso-margin-bottom-alt:auto;
margin-left:0in;
font-size:13.5pt;
font-family:"Times New Roman","serif";
font-weight:bold;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
p
{mso-style-priority:99;
mso-margin-top-alt:auto;
margin-right:0in;
mso-margin-bottom-alt:auto;
margin-left:0in;
font-size:12.0pt;
font-family:"Times New Roman","serif";}
p.MsoAcetate, li.MsoAcetate, div.MsoAcetate
{mso-style-priority:99;
mso-style-link:"Balloon Text Char";
margin:0in;
margin-bottom:.0001pt;
font-size:8.0pt;
font-family:"Tahoma","sans-serif";}
span.EmailStyle17
{mso-style-type:personal-compose;
font-family:"Calibri","sans-serif";
color:windowtext;}
span.BalloonTextChar
{mso-style-name:"Balloon Text Char";
mso-style-priority:99;
mso-style-link:"Balloon Text";
font-family:"Tahoma","sans-serif";}
span.Heading1Char
{mso-style-name:"Heading 1 Char";
mso-style-priority:9;
mso-style-link:"Heading 1";
font-family:"Times New Roman","serif";
font-weight:bold;}
span.Heading3Char
{mso-style-name:"Heading 3 Char";
mso-style-priority:9;
mso-style-link:"Heading 3";
font-family:"Times New Roman","serif";
font-weight:bold;}
span.apple-converted-space
{mso-style-name:apple-converted-space;}
.MsoChpDefault
{mso-style-type:export-only;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="2050" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]--></head><body lang=EN-US link=blue vlink=purple><div class=WordSection1><p class=MsoNormal>I’m running spacewalk 1.7 on RHEL6.3 and Friday I applied yum updates. This morning I discovered that there is something severely wrong with the server. I attempted to sign in to the spacewalk management website and got a 404 error:<o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;background:#525D76'><b><span style='font-size:16.5pt;font-family:"Tahoma","sans-serif";color:white'>HTTP Status 404 -<o:p></o:p></span></b></p><div class=MsoNormal align=center style='text-align:center'><span style='font-size:12.0pt;font-family:"Times New Roman","serif"'><hr size=1 width="100%" noshade style='color:#525D76' align=center></span></div><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;background:white'><b><span style='font-size:9.0pt;font-family:"Tahoma","sans-serif";color:white;background:#525D76'>type</span></b><span style='font-size:9.0pt;font-family:"Tahoma","sans-serif";color:black'> </span><span style='font-size:9.0pt;font-family:"Tahoma","sans-serif";color:black'>Status report<o:p></o:p></span></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;background:white'><b><span style='font-size:9.0pt;font-family:"Tahoma","sans-serif";color:white;background:#525D76'>message</span></b><span style='font-size:9.0pt;font-family:"Tahoma","sans-serif";color:black'><o:p></o:p></span></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;background:white'><b><span style='font-size:9.0pt;font-family:"Tahoma","sans-serif";color:white;background:#525D76'>description</span></b><span style='font-size:9.0pt;font-family:"Tahoma","sans-serif";color:black'> </span><u><span style='font-size:9.0pt;font-family:"Tahoma","sans-serif";color:black'>The requested resource () is not available.</span></u><span style='font-size:9.0pt;font-family:"Tahoma","sans-serif";color:black'><o:p></o:p></span></p><div class=MsoNormal align=center style='text-align:center'><span style='font-size:12.0pt;font-family:"Times New Roman","serif"'><hr size=1 width="100%" noshade style='color:#525D76' align=center></span></div><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;background:#525D76'><b><span style='font-size:10.5pt;font-family:"Tahoma","sans-serif";color:white'>Apache Tomcat/6.0.24<o:p></o:p></span></b></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>I see that there is a series of error messages about JVM failing to start in /var/log/messages: <o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>Sep 24 11:09:00 rhqengbld03 wrapper[2230]: Startup failed: Timed out waiting for signal from JVM.<o:p></o:p></p><p class=MsoNormal>Sep 24 11:09:01 rhqengbld03 wrapper[2230]: JVM did not exit on request, terminated<o:p></o:p></p><p class=MsoNormal>Sep 24 11:09:01 rhqengbld03 wrapper[2230]: JVM exited in response to signal SIGKILL (9).<o:p></o:p></p><p class=MsoNormal>Sep 24 11:09:05 rhqengbld03 wrapper[2230]: Launching a JVM...<o:p></o:p></p><p class=MsoNormal>Sep 24 11:09:34 rhqengbld03 wrapper[2230]: Startup failed: Timed out waiting for signal from JVM.<o:p></o:p></p><p class=MsoNormal>Sep 24 11:09:34 rhqengbld03 wrapper[2230]: JVM did not exit on request, terminated<o:p></o:p></p><p class=MsoNormal>Sep 24 11:09:35 rhqengbld03 wrapper[2230]: JVM exited in response to signal SIGKILL (9).<o:p></o:p></p><p class=MsoNormal>Sep 24 11:09:39 rhqengbld03 wrapper[2230]: Launching a JVM...<o:p></o:p></p><p class=MsoNormal>Sep 24 11:10:08 rhqengbld03 wrapper[2230]: Startup failed: Timed out waiting for signal from JVM.<o:p></o:p></p><p class=MsoNormal>Sep 24 11:10:08 rhqengbld03 wrapper[2230]: JVM did not exit on request, terminated<o:p></o:p></p><p class=MsoNormal>Sep 24 11:10:09 rhqengbld03 wrapper[2230]: JVM exited in response to signal SIGKILL (9).<o:p></o:p></p><p class=MsoNormal>Sep 24 11:10:13 rhqengbld03 wrapper[2230]: Launching a JVM...<o:p></o:p></p><p class=MsoNormal>Sep 24 11:10:42 rhqengbld03 wrapper[2230]: Startup failed: Timed out waiting for signal from JVM.<o:p></o:p></p><p class=MsoNormal>Sep 24 11:10:42 rhqengbld03 wrapper[2230]: JVM did not exit on request, terminated<o:p></o:p></p><p class=MsoNormal>Sep 24 11:10:43 rhqengbld03 wrapper[2230]: JVM exited in response to signal SIGKILL (9).<o:p></o:p></p><p class=MsoNormal>Sep 24 11:10:47 rhqengbld03 wrapper[2230]: Launching a JVM...<o:p></o:p></p><p class=MsoNormal>Sep 24 11:11:16 rhqengbld03 wrapper[2230]: Startup failed: Timed out waiting for signal from JVM.<o:p></o:p></p><p class=MsoNormal>Sep 24 11:11:16 rhqengbld03 wrapper[2230]: JVM did not exit on request, terminated<o:p></o:p></p><p class=MsoNormal>Sep 24 11:11:17 rhqengbld03 wrapper[2230]: JVM exited in response to signal SIGKILL (9).<o:p></o:p></p><p class=MsoNormal>Sep 24 11:11:17 rhqengbld03 wrapper[2230]: There were 5 failed launches in a row, each lasting less than 300 seconds. Giving up.<o:p></o:p></p><p class=MsoNormal>Sep 24 11:11:17 rhqengbld03 wrapper[2230]: There may be a configuration problem: please check the logs.<o:p></o:p></p><p class=MsoNormal>Sep 24 11:11:17 rhqengbld03 wrapper[2230]: <-- Wrapper Stopped<o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>Looking at /var/log/tomcat6/catalina logs I found several messages. One that looks like it might be central to the problems is this:<o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>INFO: Deploying configuration descriptor rhn.xml<o:p></o:p></p><p class=MsoNormal>Sep 24, 2012 11:08:12 AM org.apache.catalina.loader.WebappLoader start<o:p></o:p></p><p class=MsoNormal>SEVERE: LifecycleException <o:p></o:p></p><p class=MsoNormal>java.io.IOException: <span style='background:yellow;mso-highlight:yellow'>Failed to access resource /WEB-INF/lib/postgresql-jdbc.jar</span><o:p></o:p></p><p class=MsoNormal> at org.apache.catalina.loader.WebappLoader.setRepositories(WebappLoader.java:1022)<o:p></o:p></p><p class=MsoNormal> at org.apache.catalina.loader.WebappLoader.start(WebappLoader.java:653)<o:p></o:p></p><p class=MsoNormal> at org.apache.catalina.core.StandardContext.start(StandardContext.java:4341)<o:p></o:p></p><p class=MsoNormal> at org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:791)<o:p></o:p></p><p class=MsoNormal> at org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:771)<o:p></o:p></p><p class=MsoNormal> at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:526)<o:p></o:p></p><p class=MsoNormal> at org.apache.catalina.startup.HostConfig.deployDescriptor(HostConfig.java:637)<o:p></o:p></p><p class=MsoNormal> at org.apache.catalina.startup.HostConfig.deployDescriptors(HostConfig.java:563)<o:p></o:p></p><p class=MsoNormal> at org.apache.catalina.startup.HostConfig.deployApps(HostConfig.java:498)<o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>I looked for the listed postgresql-jdbc.jar file and see that there is a failed link:<o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>lrwxrwxrwx. 1 tomcat tomcat 35 Aug 3 19:02 /var/lib/tomcat6/webapps/rhn/WEB-INF/lib/postgresql-jdbc.jar -> /usr/share/java/postgresql-jdbc.jar<o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>In /usr/share/java there are these 2 files:<o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>-rwxr-xr-x. 1 root root 502118 Nov 18 2010 postgresql-8.4-702.jdbc3.jar<o:p></o:p></p><p class=MsoNormal>-rwxr-xr-x. 1 root root 539510 Nov 18 2010 postgresql-8.4-702.jdbc4.jar<o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>There is no plain vanilla versionless postgresql-jdbc.jar <o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>So I linked the missing name to the postgresql-8.4-702.jdbc4.jar to see what will happen.<o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>I restarted the server and now the spacewalk web service appears to be fully operational again.<o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>My question is: why would this link have been trashed? Is there anything I can do preemptively to prevent getting caught like this again?<o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal style='line-height:11.25pt'><span style='font-size:10.0pt;font-family:"Arial Narrow","sans-serif";color:black'><br><b>Robert Boyd</b><br>Senior Systems Engineer<br>Phone: 919-645-2972</span><span style='font-size:10.0pt;font-family:"Arial Narrow","sans-serif";color:black'><o:p></o:p></span></p><p class=MsoNormal style='line-height:11.25pt'><span style='font-size:10.0pt;font-family:"Arial Narrow","sans-serif";color:black'>Mobile: 919-306-4681<o:p></o:p></span></p><p class=MsoNormal style='line-height:11.25pt'><span style='font-size:10.0pt;font-family:"Arial Narrow","sans-serif";color:black'>Peoplefluent<o:p></o:p></span></p><p class=MsoNormal style='line-height:11.25pt'><span style='font-size:10.0pt;font-family:"Arial Narrow","sans-serif";color:black'>434 Fayetteville Street <o:p></o:p></span></p><p class=MsoNormal style='line-height:11.25pt'><span style='font-size:10.0pt;font-family:"Arial Narrow","sans-serif";color:black'>Raleigh, NC 27601<o:p></o:p></span></p><p class=MsoNormal style='line-height:11.25pt'><span style='font-size:10.0pt;font-family:"Arial Narrow","sans-serif";color:black'><br><a href="mailto:michael.dileonardo@peoplefluent.com"><span style='color:blue'>robert.boyd@peoplefluent.com</span></a><o:p></o:p></span></p><p class=MsoNormal><span style='font-size:8.0pt;font-family:"Arial Narrow","sans-serif";color:#1F497D'><o:p> </o:p></span></p><table class=MsoNormalTable border=0 cellspacing=8 cellpadding=0><tr><td style='padding:0in 0in 0in 0in'><p class=MsoNormal style='line-height:11.25pt'><img border=0 width=179 height=65 id="Picture_x0020_1" src="cid:image001.png@01CD9A4E.D98CEAB0" alt="Description: Description: Description: cid:image001.png@01CC344B.23823870"><span style='font-size:8.0pt;font-family:"Arial Narrow","sans-serif"'><o:p></o:p></span></p></td><td style='padding:0in 0in 0in 0in'><p class=MsoNormal style='line-height:11.25pt'><span style='font-size:8.0pt;font-family:"Verdana","sans-serif"'><o:p> </o:p></span></p></td></tr></table><p class=MsoNormal><span style='font-size:7.5pt;font-family:"Arial Narrow","sans-serif";color:#333333'>This email message is for the sole use of the intended recipient(s) and may contain confidential information. Any unauthorized review, use, disclosure or distribution is prohibited. If you are not the intended recipient, please contact the sender by reply email and destroy all copies of the original message.</span><o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p></div></body></html>