[Spacewalk-list] Trouble with spacewalk after long time running
Thomas Schweikle
tschweikle at gmail.com
Wed May 17 08:55:42 UTC 2017
Hi!
Since yesterday, after months of running more or less seamless, I
only get "HTTP 404" errors trying to connect to spacewalk's web page.
Within catalina.out:
# cat catalina.out
Mai 17, 2017 9:37:15 AM org.apache.catalina.core.AprLifecycleListener init
INFORMATION: The APR based Apache Tomcat Native library which allows
optimal performance in production environments was not found on the
java.library.path:
/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib
Mai 17, 2017 9:37:16 AM org.apache.coyote.http11.Http11Protocol init
INFORMATION: Initializing Coyote HTTP/1.1 on http-127.0.0.1-8080
Mai 17, 2017 9:37:16 AM org.apache.catalina.startup.Catalina load
INFORMATION: Initialization processed in 781 ms
Mai 17, 2017 9:37:16 AM org.apache.catalina.core.StandardService start
INFORMATION: Starting service Catalina
Mai 17, 2017 9:37:16 AM org.apache.catalina.core.StandardEngine start
INFORMATION: Starting Servlet Engine: Apache Tomcat/6.0.24
Mai 17, 2017 9:37:16 AM org.apache.catalina.startup.HostConfig deployDescriptor
INFORMATION: Deploying configuration descriptor rhn.xml
Mai 17, 2017 9:37:21 AM org.apache.catalina.core.StandardContext start
SCHWERWIEGEND: Error listenerStart
Mai 17, 2017 9:37:21 AM org.apache.catalina.core.StandardContext start
SCHWERWIEGEND: Context [/rhn] startup failed due to previous errors
Mai 17, 2017 9:37:21 AM org.apache.catalina.loader.WebappClassLoader
clearReferencesThreads
SCHWERWIEGEND: A web application appears to have started a thread
named [RHN Message Dispatcher] but has failed to stop it. This is very
likely to create a memory leak.
Mai 17, 2017 9:37:22 AM org.apache.coyote.http11.Http11Protocol start
INFORMATION: Starting Coyote HTTP/1.1 on http-127.0.0.1-8080
Mai 17, 2017 9:37:22 AM org.apache.jk.common.ChannelSocket init
INFORMATION: JK: ajp13 listening on /127.0.0.1:8009
Mai 17, 2017 9:37:22 AM org.apache.jk.server.JkMain start
INFORMATION: Jk running ID=0 time=0/21 config=null
Mai 17, 2017 9:37:22 AM org.apache.jk.common.ChannelSocket init
INFORMATION: JK: ajp13 listening on /0:0:0:0:0:0:0:1:8009
Mai 17, 2017 9:37:22 AM org.apache.jk.server.JkMain start
INFORMATION: Jk running ID=0 time=0/4 config=null
Mai 17, 2017 9:37:22 AM org.apache.catalina.startup.Catalina start
INFORMATION: Server startup in 6939 ms
There are some failures "Error listenerStart". But I do not know where
to look next.
In /var/log/rhn/rhn_taskomatic_daemon.log I could find a variety of
error messages:
INFO | jvm 1 | 2017/05/17 09:37:33 | 2017-05-17 09:37:33,741
[DefaultQuartzScheduler_Worker-2] ERROR
com.redhat.rhn.manager.kickstart.cobbler.CobblerLoginCommand -
XmlRpcFault while logging in. most likely user doesn't have
permissions.
INFO | jvm 1 | 2017/05/17 09:37:33 | redstone.xmlrpc.XmlRpcFault:
<class 'cobbler.cexceptions.CX'>:'login failed (taskomatic_user)'
INFO | jvm 1 | 2017/05/17 09:37:33 | at
redstone.xmlrpc.XmlRpcClient.handleResponse(XmlRpcClient.java:443)
INFO | jvm 1 | 2017/05/17 09:37:33 | at
redstone.xmlrpc.XmlRpcClient.endCall(XmlRpcClient.java:376)
INFO | jvm 1 | 2017/05/17 09:37:33 | at
redstone.xmlrpc.XmlRpcClient.invoke(XmlRpcClient.java:165)
INFO | jvm 1 | 2017/05/17 09:37:33 | at
com.redhat.rhn.manager.kickstart.cobbler.CobblerXMLRPCHelper.invokeMethod(CobblerXMLRPCHelper.java:69)
INFO | jvm 1 | 2017/05/17 09:37:33 | at
com.redhat.rhn.manager.kickstart.cobbler.CobblerLoginCommand.login(CobblerLoginCommand.java:52)
INFO | jvm 1 | 2017/05/17 09:37:33 | at
com.redhat.rhn.frontend.integration.IntegrationService.authorize(IntegrationService.java:114)
INFO | jvm 1 | 2017/05/17 09:37:33 | at
com.redhat.rhn.frontend.integration.IntegrationService.getAuthToken(IntegrationService.java:67)
INFO | jvm 1 | 2017/05/17 09:37:33 | at
com.redhat.rhn.manager.kickstart.cobbler.CobblerCommand.<init>(CobblerCommand.java:68)
INFO | jvm 1 | 2017/05/17 09:37:33 | at
com.redhat.rhn.manager.kickstart.cobbler.CobblerDistroSyncCommand.<init>(CobblerDistroSyncCommand.java:52)
INFO | jvm 1 | 2017/05/17 09:37:33 | at
com.redhat.rhn.taskomatic.task.CobblerSyncTask.execute(CobblerSyncTask.java:78)
INFO | jvm 1 | 2017/05/17 09:37:33 | at
com.redhat.rhn.taskomatic.task.RhnJavaJob.execute(RhnJavaJob.java:88)
INFO | jvm 1 | 2017/05/17 09:37:33 | at
com.redhat.rhn.taskomatic.TaskoJob.execute(TaskoJob.java:186)
INFO | jvm 1 | 2017/05/17 09:37:33 | at
org.quartz.core.JobRunShell.run(JobRunShell.java:216)
INFO | jvm 1 | 2017/05/17 09:37:33 | at
org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:549)
Trying to log into the database using the credentials given in
/etc/rhn/rhn.conf works:
# psql -U rhnuser -p 5432 -h localhost rhnschema
Password for user rhnuser:
psql (9.6.3)
Type "help" for help.
rhnschema=>
Are there any other locations I have to configure credentials?
Anything else what could make spacewalk fail without reason?
selinux is set to permissive.
Versions are:
# rpm -qa | grep 'space\|rhn' | sort
perl-XML-NamespaceSupport-1.10-3.el6.noarch
rhncfg-5.10.99-1.el6.noarch
rhncfg-actions-5.10.99-1.el6.noarch
rhncfg-client-5.10.99-1.el6.noarch
rhn-check-2.6.8-1.el6.noarch
rhn-client-tools-2.6.8-1.el6.noarch
rhnlib-2.6.3-1.el6.noarch
rhn-org-httpd-ssl-key-pair-spacewalk-dmz-1.0-1.noarch
rhnpush-5.5.101-1.el6.noarch
rhnsd-5.0.25-1.el6.x86_64
rhn-setup-2.6.8-1.el6.noarch
spacecmd-2.6.16-1.el6.noarch
spacewalk-admin-2.6.1-1.el6.noarch
spacewalk-backend-2.6.77-1.el6.noarch
spacewalk-backend-app-2.6.77-1.el6.noarch
spacewalk-backend-applet-2.6.77-1.el6.noarch
spacewalk-backend-config-files-2.6.77-1.el6.noarch
spacewalk-backend-config-files-common-2.6.77-1.el6.noarch
spacewalk-backend-config-files-tool-2.6.77-1.el6.noarch
spacewalk-backend-iss-2.6.77-1.el6.noarch
spacewalk-backend-iss-export-2.6.77-1.el6.noarch
spacewalk-backend-libs-2.6.77-1.el6.noarch
spacewalk-backend-package-push-server-2.6.77-1.el6.noarch
spacewalk-backend-server-2.6.77-1.el6.noarch
spacewalk-backend-sql-2.6.77-1.el6.noarch
spacewalk-backend-sql-postgresql-2.6.77-1.el6.noarch
spacewalk-backend-tools-2.6.77-1.el6.noarch
spacewalk-backend-usix-2.6.77-1.el6.noarch
spacewalk-backend-xml-export-libs-2.6.77-1.el6.noarch
spacewalk-backend-xmlrpc-2.6.77-1.el6.noarch
spacewalk-base-2.6.6-1.el6.noarch
spacewalk-base-minimal-2.6.6-1.el6.noarch
spacewalk-base-minimal-config-2.6.6-1.el6.noarch
spacewalk-branding-2.5.3-1.el6.noarch
spacewalk-certs-tools-2.5.3-1.el6.noarch
spacewalk-common-2.6.1-1.el6.noarch
spacewalk-config-2.6.5-1.el6.noarch
spacewalk-doc-indexes-2.6.2-1.el6.noarch
spacewalk-html-2.6.6-1.el6.noarch
spacewalk-java-2.6.49-1.el6.noarch
spacewalk-java-config-2.6.49-1.el6.noarch
spacewalk-java-lib-2.6.49-1.el6.noarch
spacewalk-java-postgresql-2.6.49-1.el6.noarch
spacewalk-jpp-workaround-2.3.5-1.el6.noarch
spacewalk-repo-2.6-0.el6.noarch
spacewalk-schema-2.6.17-1.el6.noarch
spacewalk-search-2.6.1-1.el6.noarch
spacewalk-selinux-2.3.2-1.el6.noarch
spacewalk-setup-2.6.2-1.el6.noarch
spacewalk-setup-jabberd-2.3.2-1.el6.noarch
spacewalk-setup-postgresql-2.6.2-1.el6.noarch
spacewalk-slf4j-1.6.1-6.el6.noarch
spacewalk-taskomatic-2.6.49-1.el6.noarch
yum-rhn-plugin-2.6.3-1.el6.noarch
# rpm -qa | grep python | sort
audit-libs-python-2.4.5-6.el6.x86_64
dbus-python-0.83.0-6.1.el6.x86_64
gnome-python2-2.28.0-3.el6.x86_64
gnome-python2-bonobo-2.28.0-3.el6.x86_64
gnome-python2-canvas-2.28.0-3.el6.x86_64
gnome-python2-gnome-2.28.0-3.el6.x86_64
gnome-python2-gnomevfs-2.28.0-3.el6.x86_64
libproxy-python-0.3.0-10.el6.x86_64
libreport-python-2.0.9-33.0.1.el6.x86_64
libselinux-python-2.0.94-7.el6.x86_64
libsemanage-python-2.0.43-5.1.el6.x86_64
libxml2-python-2.7.6-21.0.1.el6_8.1.x86_64
newt-python-0.52.11-4.el6.x86_64
policycoreutils-python-2.0.83-30.1.0.1.el6_8.x86_64
postgresql96-plpython-9.6.3-1PGDG.rhel6.x86_64
python-2.6.6-66.0.1.el6_8.x86_64
python-argparse-1.2.1-3.rhel6.noarch
python-backports-1.0-5.el6.x86_64
python-backports-ssl_match_hostname-3.4.0.2-5.el6.noarch
python-chardet-2.2.1-1.el6.noarch
python-cheetah-2.4.1-1.el6.x86_64
python-debian-0.1.27-3.el6.noarch
python-deltarpm-3.5-0.5.20090913git.el6.x86_64
python-devel-2.6.6-66.0.1.el6_8.x86_64
python-dmidecode-3.10.15-1.el6.x86_64
python-ethtool-0.6-6.el6.x86_64
python-gudev-147.1-4.el6_0.1.x86_64
python-gzipstream-2.3.3-1.el6.noarch
python-hwdata-1.7.3-1.el6.noarch
python-iniparse-0.3.1-2.1.el6.noarch
python-iwlib-0.1-1.2.el6.x86_64
python-libs-2.6.6-66.0.1.el6_8.x86_64
python-markdown-2.0.1-4.el6.noarch
python-netaddr-0.7.5-4.el6.noarch
python-psycopg2-2.7.1-1.rhel6.x86_64
python-pycurl-7.19.0-9.el6.x86_64
python-pygments-1.1.1-2.el6.noarch
python-requests-2.6.0-4.el6.noarch
python-setuptools-0.6.10-3.el6.noarch
python-simplejson-3.8.2-1.rhel6.x86_64
python-six-1.9.0-2.el6.noarch
python-sss-1.13.3-56.el6.x86_64
python-sssdconfig-1.13.3-56.el6.noarch
python-urlgrabber-3.9.1-11.0.1.el6.noarch
python-urllib3-1.10.2-3.el6.noarch
rpm-python-4.8.0-55.el6.x86_64
setools-libs-python-3.3.7-4.el6.x86_64
# java -version
openjdk version "1.8.0_131"
OpenJDK Runtime Environment (build 1.8.0_131-b11)
OpenJDK 64-Bit Server VM (build 25.131-b11, mixed mode)
# rpm -qa | grep tom | sort
apache-tomcat-apis-0.1-1.el6.noarch
jquery-ui-1.10.4.custom-3.el6.noarch
tomcat5-jsp-2.0-api-5.5.27-7.jpp5.noarch
tomcat5-servlet-2.4-api-5.5.27-7.jpp5.noarch
tomcat6-6.0.24-105.el6_8.noarch
tomcat6-el-2.1-api-6.0.24-105.el6_8.noarch
tomcat6-jsp-2.1-api-6.0.24-105.el6_8.noarch
tomcat6-lib-6.0.24-105.el6_8.noarch
tomcat6-servlet-2.5-api-6.0.24-105.el6_8.noarch
# rpm -qa | grep postgres | sort
postgresql96-9.6.3-1PGDG.rhel6.x86_64
postgresql96-contrib-9.6.3-1PGDG.rhel6.x86_64
postgresql96-libs-9.6.3-1PGDG.rhel6.x86_64
postgresql96-plperl-9.6.3-1PGDG.rhel6.x86_64
postgresql96-plpython-9.6.3-1PGDG.rhel6.x86_64
postgresql96-pltcl-9.6.3-1PGDG.rhel6.x86_64
postgresql96-server-9.6.3-1PGDG.rhel6.x86_64
postgresql-jdbc-42.1.1-1.rhel6.noarch
spacewalk-backend-sql-postgresql-2.6.77-1.el6.noarch
spacewalk-java-postgresql-2.6.49-1.el6.noarch
spacewalk-setup-postgresql-2.6.2-1.el6.noarch
for postgresql 9.6 I had to patch spacewalk within
/usr/share/rhn/server/handlers/xmlrpc/queue.py addin "as"
(http://git.net/ml/spacewalk-devel/2016-11/msg00004.html and
https://github.com/spacewalkproject/spacewalk/blob/master/backend/server/handlers/xmlrpc/queue.py
-- worked like charm!)
Hopefully I did not forget some package spacewalk depends on ...
--
Thomas
More information about the Spacewalk-list
mailing list