[Spacewalk-list] 404 from spacewalk 1.7 because no space left to create mutex

[john miller]

I was getting a 404 from the spacewalk server. running 1.7 (i know its old,
upgrading soon)

spacewalk-service restart was succeeding but it did not fix it.

tail --lines 10 /var/log/messages
Apr 30 11:13:41 localhost wrapper[983]: Startup failed: Timed out waiting
for signal from JVM.
Apr 30 11:13:42 localhost wrapper[983]: JVM did not exit on request,
terminated
Apr 30 11:13:42 localhost wrapper[983]: JVM exited in response to signal
SIGKILL (9).
Apr 30 11:13:46 localhost wrapper[983]: Launching a JVM...
Apr 30 11:14:15 localhost wrapper[983]: Startup failed: Timed out waiting
for signal from JVM.
Apr 30 11:14:15 localhost wrapper[983]: JVM did not exit on request,
terminated
Apr 30 11:14:16 localhost wrapper[983]: JVM exited in response to signal
SIGKILL (9).
Apr 30 11:14:16 localhost wrapper[983]: There were 5 failed launches in a
row, each lasting less than 300 seconds.  Giving up.
Apr 30 11:14:16 localhost wrapper[983]:   There may be a configuration
problem: please check the logs.
Apr 30 11:14:16 localhost wrapper[983]: <-- Wrapper Stopped

/var/log/tomcat6/catalina.out
Apr 30, 2013 11:10:57 AM org.apache.coyote.http11.Http11Protocol pause
INFO: Pausing Coyote HTTP/1.1 on http-127.0.0.1-8080
Apr 30, 2013 11:10:58 AM org.apache.catalina.core.StandardService stop
INFO: Stopping service Catalina
Apr 30, 2013 11:10:58 AM org.apache.catalina.loader.WebappClassLoader
clearReferencesJdbc
SEVERE: The web application [/rhn] registered the JDBC driver
[org.postgresql.Driver] but failed to unregister it when the web
application was stopped. To prevent a memory leak, the JDBC Driver has been
forcibly unregistered.
Apr 30, 2013 11:10:58 AM org.apache.catalina.loader.WebappClassLoader
clearReferencesThreads
SEVERE: The web application [/rhn] appears to have started a thread named
[RHN Message Dispatcher] but has failed to stop it. This is very likely to
create a memory leak.
Apr 30, 2013 11:11:00 AM org.apache.coyote.http11.Http11Protocol destroy
tail --lines 10 /var/log/httpd/error_log
[Tue Apr 30 11:11:27 2013] [notice] SELinux policy enabled; httpd running
as context system_u:system_r:httpd_t:s0
[Tue Apr 30 11:11:27 2013] [notice] suEXEC mechanism enabled (wrapper:
/usr/sbin/suexec)
[Tue Apr 30 11:11:27 2013] [notice] SSL FIPS mode disabled
[Tue Apr 30 11:11:27 2013] [notice] Digest: generating secret for digest
authentication ...
[Tue Apr 30 11:11:27 2013] [notice] Digest: done
[Tue Apr 30 11:11:27 2013] [notice] mod_python: Creating 4 session mutexes
based on 256 max processes and 0 max threads.
[Tue Apr 30 11:11:27 2013] [notice] mod_python: using mutex_directory /tmp
[Tue Apr 30 11:11:27 2013] [error] (28)No space left on device: mod_python:
Failed to create global mutex 1 of 4 (/tmp/mpmtx8361).
Configuration Failed
from https://bugzilla.redhat.com/show_bug.cgi?id=120515 it seems that this
was a known issue in mod_python-3.1.3-1 but I am running
mod_python-3.3.1-16.fc15.x86_64. Maybe there was a regression or maybe
something else is leaking the semaphores now. To fix:

1. delete leaked semaphores
ipcs -s | perl -ane '/^0x00000000/ && `ipcrm -s $F[1]`'
2. allow more semaphores

echo “kernel.sem = 512 32000 100 512″ >> /etc/sysctl.conf; sysctl -p

3. restart spacewalk

spacewalk-service restart
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/spacewalk-list/attachments/20130430/d95e6f59/attachment.htm>