[Spacewalk-list] Trouble with spacewalk after long time running
Thomas Schweikle
tschweikle at gmail.com
Wed May 17 15:03:37 UTC 2017
Further things I found within httpd logs:
access_log:
[17/May/2017:16:27:21 +0200] "POST /cobbler_api HTTP/1.1" 200 144 "-"
"Java/1.8.0_131"
[17/May/2017:16:27:21 +0200] "POST /rpc/api HTTP/1.0" 404 952 "-"
"xmlrpclib.py/1.0.1 (by www.pythonware.com)"
[17/May/2017:16:27:21 +0200] "POST /cobbler_api HTTP/1.1" 200 326 "-"
"Java/1.8.0_131"
[17/May/2017:16:28:00 +0200] "POST /cobbler_api HTTP/1.1" 200 144 "-"
"Java/1.8.0_131"
[17/May/2017:16:28:00 +0200] "POST /rpc/api HTTP/1.0" 404 952 "-"
"xmlrpclib.py/1.0.1 (by www.pythonware.com)"
Since 200 is OK, 404 is not. Apache seems to have problems with /rpc/api!
Within error_log there is nothing remarkable. Python 2.6.6 is OK or
does spacewalk rely on Python 3.x?
[notice] SELinux policy enabled; httpd running as context
unconfined_u:system_r:httpd_t:s0
[notice] suEXEC mechanism enabled (wrapper: /usr/sbin/suexec)
[notice] Digest: generating secret for digest authentication ...
[notice] Digest: done
[notice] Apache/2.2.15 (Unix) DAV/2 mod_ssl/2.2.15 OpenSSL/1.0.1e-fips
mod_wsgi/3.2 Python/2.6.6 mod_apreq2-20090110/2.8.0 mod_perl/2.0.4
Perl/v5.10.1 configured -- resuming normal operations
Client request seem to be handled OK:
[17/May/2017:16:36:04 +0200] "POST /XMLRPC HTTP/1.1" 200 112276
[17/May/2017:16:36:06 +0200] "POST /XMLRPC HTTP/1.1" 200 915
[17/May/2017:16:36:07 +0200] "POST /XMLRPC HTTP/1.1" 200 121
[17/May/2017:16:36:07 +0200] "POST /XMLRPC HTTP/1.1" 200 126
[17/May/2017:16:36:08 +0200] "POST /XMLRPC HTTP/1.1" 200 1082
[17/May/2017:16:36:08 +0200] "GET
/XMLRPC/GET-REQ/ol6-imisterm/repodata/repomd.xml HTTP/1.1" 200 1302
While web-requests are not:
[17/May/2017:16:36:34 +0200] "GET /rhn/schedule/CompletedActions.do
HTTP/1.1" 404 952
[17/May/2017:16:36:50 +0200] "GET /rhn/Login.do HTTP/1.1" 404 952
[17/May/2017:16:37:34 +0200] "GET /rhn/schedule/CompletedActions.do
HTTP/1.1" 404 952
[17/May/2017:16:37:50 +0200] "GET /rhn/Login.do HTTP/1.1" 404 952
ssl_error_log is empty -- no certificate errors so far ;-)
ssl_request_log seems good too:
TLSv1.2 ECDHE-RSA-AES256-GCM-SHA384 "GET
/rhn/schedule/CompletedActions.do HTTP/1.1" 952
TLSv1.2 ECDHE-RSA-AES256-GCM-SHA384 "GET /rhn/Login.do HTTP/1.1" 952
TLSv1.2 ECDHE-RSA-AES256-GCM-SHA384 "GET
/rhn/schedule/CompletedActions.do HTTP/1.1" 952
TLSv1.2 ECDHE-RSA-AES256-GCM-SHA384 "GET /rhn/Login.do HTTP/1.1" 952
TLSv1.2 ECDHE-RSA-AES256-GCM-SHA384 "GET
/rhn/schedule/CompletedActions.do HTTP/1.1" 952
TLSv1.2 ECDHE-RSA-AES256-GCM-SHA384 "GET /rhn/Login.do HTTP/1.1" 952
TLSv1.2 ECDHE-RSA-AES256-GCM-SHA384 "GET
/rhn/schedule/CompletedActions.do HTTP/1.1" 952
TLSv1.2 ECDHE-RSA-AES256-GCM-SHA384 "GET /rhn/Login.do HTTP/1.1" 952
A bit clueless now! Looks like tomcat6 running, but connection to
apache broken. Maybe same with python, perl and what else spacewalk
may need to work. But why? Looks like all modules necessary are
loaded:
Loaded Modules:
mod_wsgi.c, mod_ssl.c, mod_perl.c, mod_apreq2.c, mod_version.c,
mod_cgi.c, mod_disk_cache.c, mod_suexec.c, mod_cache.c,
mod_proxy_connect.c, mod_proxy_ajp.c, mod_proxy_http.c,
mod_proxy_ftp.c, mod_proxy_balancer.c, mod_proxy.c, mod_rewrite.c,
mod_substitute.c, mod_alias.c, mod_userdir.c, mod_speling.c,
mod_actions.c, mod_dir.c, mod_negotiation.c, mod_vhost_alias.c,
mod_dav_fs.c, mod_info.c, mod_autoindex.c, mod_status.c, mod_dav.c,
mod_mime.c, mod_setenvif.c, mod_usertrack.c, mod_headers.c,
mod_deflate.c, mod_expires.c, mod_mime_magic.c, mod_ext_filter.c,
mod_env.c, mod_logio.c, mod_log_config.c, mod_include.c,
mod_authnz_ldap.c, util_ldap.c, mod_authz_default.c, mod_authz_dbm.c,
mod_authz_groupfile.c, mod_authz_owner.c, mod_authz_user.c,
mod_authz_host.c, mod_authn_default.c, mod_authn_dbm.c,
mod_authn_anon.c, mod_authn_alias.c, mod_authn_file.c,
mod_auth_digest.c, mod_auth_basic.c, mod_so.c, http_core.c, prefork.c,
core.c
Any idea, help anyone?
2017-05-17 10:55 GMT+02:00 Thomas Schweikle <tschweikle at gmail.com>:
> Hi!
>
> Since yesterday, after months of running more or less seamless, I
> only get "HTTP 404" errors trying to connect to spacewalk's web page.
> Within catalina.out:
> # cat catalina.out
> Mai 17, 2017 9:37:15 AM org.apache.catalina.core.AprLifecycleListener init
> INFORMATION: The APR based Apache Tomcat Native library which allows
> optimal performance in production environments was not found on the
> java.library.path:
> /usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib
> Mai 17, 2017 9:37:16 AM org.apache.coyote.http11.Http11Protocol init
> INFORMATION: Initializing Coyote HTTP/1.1 on http-127.0.0.1-8080
> Mai 17, 2017 9:37:16 AM org.apache.catalina.startup.Catalina load
> INFORMATION: Initialization processed in 781 ms
> Mai 17, 2017 9:37:16 AM org.apache.catalina.core.StandardService start
> INFORMATION: Starting service Catalina
> Mai 17, 2017 9:37:16 AM org.apache.catalina.core.StandardEngine start
> INFORMATION: Starting Servlet Engine: Apache Tomcat/6.0.24
> Mai 17, 2017 9:37:16 AM org.apache.catalina.startup.HostConfig deployDescriptor
> INFORMATION: Deploying configuration descriptor rhn.xml
> Mai 17, 2017 9:37:21 AM org.apache.catalina.core.StandardContext start
> SCHWERWIEGEND: Error listenerStart
> Mai 17, 2017 9:37:21 AM org.apache.catalina.core.StandardContext start
> SCHWERWIEGEND: Context [/rhn] startup failed due to previous errors
> Mai 17, 2017 9:37:21 AM org.apache.catalina.loader.WebappClassLoader
> clearReferencesThreads
> SCHWERWIEGEND: A web application appears to have started a thread
> named [RHN Message Dispatcher] but has failed to stop it. This is very
> likely to create a memory leak.
> Mai 17, 2017 9:37:22 AM org.apache.coyote.http11.Http11Protocol start
> INFORMATION: Starting Coyote HTTP/1.1 on http-127.0.0.1-8080
> Mai 17, 2017 9:37:22 AM org.apache.jk.common.ChannelSocket init
> INFORMATION: JK: ajp13 listening on /127.0.0.1:8009
> Mai 17, 2017 9:37:22 AM org.apache.jk.server.JkMain start
> INFORMATION: Jk running ID=0 time=0/21 config=null
> Mai 17, 2017 9:37:22 AM org.apache.jk.common.ChannelSocket init
> INFORMATION: JK: ajp13 listening on /0:0:0:0:0:0:0:1:8009
> Mai 17, 2017 9:37:22 AM org.apache.jk.server.JkMain start
> INFORMATION: Jk running ID=0 time=0/4 config=null
> Mai 17, 2017 9:37:22 AM org.apache.catalina.startup.Catalina start
> INFORMATION: Server startup in 6939 ms
>
> There are some failures "Error listenerStart". But I do not know where
> to look next.
>
> In /var/log/rhn/rhn_taskomatic_daemon.log I could find a variety of
> error messages:
> INFO | jvm 1 | 2017/05/17 09:37:33 | 2017-05-17 09:37:33,741
> [DefaultQuartzScheduler_Worker-2] ERROR
> com.redhat.rhn.manager.kickstart.cobbler.CobblerLoginCommand -
> XmlRpcFault while logging in. most likely user doesn't have
> permissions.
> INFO | jvm 1 | 2017/05/17 09:37:33 | redstone.xmlrpc.XmlRpcFault:
> <class 'cobbler.cexceptions.CX'>:'login failed (taskomatic_user)'
> INFO | jvm 1 | 2017/05/17 09:37:33 | at
> redstone.xmlrpc.XmlRpcClient.handleResponse(XmlRpcClient.java:443)
> INFO | jvm 1 | 2017/05/17 09:37:33 | at
> redstone.xmlrpc.XmlRpcClient.endCall(XmlRpcClient.java:376)
> INFO | jvm 1 | 2017/05/17 09:37:33 | at
> redstone.xmlrpc.XmlRpcClient.invoke(XmlRpcClient.java:165)
> INFO | jvm 1 | 2017/05/17 09:37:33 | at
> com.redhat.rhn.manager.kickstart.cobbler.CobblerXMLRPCHelper.invokeMethod(CobblerXMLRPCHelper.java:69)
> INFO | jvm 1 | 2017/05/17 09:37:33 | at
> com.redhat.rhn.manager.kickstart.cobbler.CobblerLoginCommand.login(CobblerLoginCommand.java:52)
> INFO | jvm 1 | 2017/05/17 09:37:33 | at
> com.redhat.rhn.frontend.integration.IntegrationService.authorize(IntegrationService.java:114)
> INFO | jvm 1 | 2017/05/17 09:37:33 | at
> com.redhat.rhn.frontend.integration.IntegrationService.getAuthToken(IntegrationService.java:67)
> INFO | jvm 1 | 2017/05/17 09:37:33 | at
> com.redhat.rhn.manager.kickstart.cobbler.CobblerCommand.<init>(CobblerCommand.java:68)
> INFO | jvm 1 | 2017/05/17 09:37:33 | at
> com.redhat.rhn.manager.kickstart.cobbler.CobblerDistroSyncCommand.<init>(CobblerDistroSyncCommand.java:52)
> INFO | jvm 1 | 2017/05/17 09:37:33 | at
> com.redhat.rhn.taskomatic.task.CobblerSyncTask.execute(CobblerSyncTask.java:78)
> INFO | jvm 1 | 2017/05/17 09:37:33 | at
> com.redhat.rhn.taskomatic.task.RhnJavaJob.execute(RhnJavaJob.java:88)
> INFO | jvm 1 | 2017/05/17 09:37:33 | at
> com.redhat.rhn.taskomatic.TaskoJob.execute(TaskoJob.java:186)
> INFO | jvm 1 | 2017/05/17 09:37:33 | at
> org.quartz.core.JobRunShell.run(JobRunShell.java:216)
> INFO | jvm 1 | 2017/05/17 09:37:33 | at
> org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:549)
>
> Trying to log into the database using the credentials given in
> /etc/rhn/rhn.conf works:
> # psql -U rhnuser -p 5432 -h localhost rhnschema
> Password for user rhnuser:
> psql (9.6.3)
> Type "help" for help.
>
> rhnschema=>
>
> Are there any other locations I have to configure credentials?
> Anything else what could make spacewalk fail without reason?
>
> selinux is set to permissive.
>
> Versions are:
> # rpm -qa | grep 'space\|rhn' | sort
> perl-XML-NamespaceSupport-1.10-3.el6.noarch
> rhncfg-5.10.99-1.el6.noarch
> rhncfg-actions-5.10.99-1.el6.noarch
> rhncfg-client-5.10.99-1.el6.noarch
> rhn-check-2.6.8-1.el6.noarch
> rhn-client-tools-2.6.8-1.el6.noarch
> rhnlib-2.6.3-1.el6.noarch
> rhn-org-httpd-ssl-key-pair-spacewalk-dmz-1.0-1.noarch
> rhnpush-5.5.101-1.el6.noarch
> rhnsd-5.0.25-1.el6.x86_64
> rhn-setup-2.6.8-1.el6.noarch
> spacecmd-2.6.16-1.el6.noarch
> spacewalk-admin-2.6.1-1.el6.noarch
> spacewalk-backend-2.6.77-1.el6.noarch
> spacewalk-backend-app-2.6.77-1.el6.noarch
> spacewalk-backend-applet-2.6.77-1.el6.noarch
> spacewalk-backend-config-files-2.6.77-1.el6.noarch
> spacewalk-backend-config-files-common-2.6.77-1.el6.noarch
> spacewalk-backend-config-files-tool-2.6.77-1.el6.noarch
> spacewalk-backend-iss-2.6.77-1.el6.noarch
> spacewalk-backend-iss-export-2.6.77-1.el6.noarch
> spacewalk-backend-libs-2.6.77-1.el6.noarch
> spacewalk-backend-package-push-server-2.6.77-1.el6.noarch
> spacewalk-backend-server-2.6.77-1.el6.noarch
> spacewalk-backend-sql-2.6.77-1.el6.noarch
> spacewalk-backend-sql-postgresql-2.6.77-1.el6.noarch
> spacewalk-backend-tools-2.6.77-1.el6.noarch
> spacewalk-backend-usix-2.6.77-1.el6.noarch
> spacewalk-backend-xml-export-libs-2.6.77-1.el6.noarch
> spacewalk-backend-xmlrpc-2.6.77-1.el6.noarch
> spacewalk-base-2.6.6-1.el6.noarch
> spacewalk-base-minimal-2.6.6-1.el6.noarch
> spacewalk-base-minimal-config-2.6.6-1.el6.noarch
> spacewalk-branding-2.5.3-1.el6.noarch
> spacewalk-certs-tools-2.5.3-1.el6.noarch
> spacewalk-common-2.6.1-1.el6.noarch
> spacewalk-config-2.6.5-1.el6.noarch
> spacewalk-doc-indexes-2.6.2-1.el6.noarch
> spacewalk-html-2.6.6-1.el6.noarch
> spacewalk-java-2.6.49-1.el6.noarch
> spacewalk-java-config-2.6.49-1.el6.noarch
> spacewalk-java-lib-2.6.49-1.el6.noarch
> spacewalk-java-postgresql-2.6.49-1.el6.noarch
> spacewalk-jpp-workaround-2.3.5-1.el6.noarch
> spacewalk-repo-2.6-0.el6.noarch
> spacewalk-schema-2.6.17-1.el6.noarch
> spacewalk-search-2.6.1-1.el6.noarch
> spacewalk-selinux-2.3.2-1.el6.noarch
> spacewalk-setup-2.6.2-1.el6.noarch
> spacewalk-setup-jabberd-2.3.2-1.el6.noarch
> spacewalk-setup-postgresql-2.6.2-1.el6.noarch
> spacewalk-slf4j-1.6.1-6.el6.noarch
> spacewalk-taskomatic-2.6.49-1.el6.noarch
> yum-rhn-plugin-2.6.3-1.el6.noarch
>
> # rpm -qa | grep python | sort
> audit-libs-python-2.4.5-6.el6.x86_64
> dbus-python-0.83.0-6.1.el6.x86_64
> gnome-python2-2.28.0-3.el6.x86_64
> gnome-python2-bonobo-2.28.0-3.el6.x86_64
> gnome-python2-canvas-2.28.0-3.el6.x86_64
> gnome-python2-gnome-2.28.0-3.el6.x86_64
> gnome-python2-gnomevfs-2.28.0-3.el6.x86_64
> libproxy-python-0.3.0-10.el6.x86_64
> libreport-python-2.0.9-33.0.1.el6.x86_64
> libselinux-python-2.0.94-7.el6.x86_64
> libsemanage-python-2.0.43-5.1.el6.x86_64
> libxml2-python-2.7.6-21.0.1.el6_8.1.x86_64
> newt-python-0.52.11-4.el6.x86_64
> policycoreutils-python-2.0.83-30.1.0.1.el6_8.x86_64
> postgresql96-plpython-9.6.3-1PGDG.rhel6.x86_64
> python-2.6.6-66.0.1.el6_8.x86_64
> python-argparse-1.2.1-3.rhel6.noarch
> python-backports-1.0-5.el6.x86_64
> python-backports-ssl_match_hostname-3.4.0.2-5.el6.noarch
> python-chardet-2.2.1-1.el6.noarch
> python-cheetah-2.4.1-1.el6.x86_64
> python-debian-0.1.27-3.el6.noarch
> python-deltarpm-3.5-0.5.20090913git.el6.x86_64
> python-devel-2.6.6-66.0.1.el6_8.x86_64
> python-dmidecode-3.10.15-1.el6.x86_64
> python-ethtool-0.6-6.el6.x86_64
> python-gudev-147.1-4.el6_0.1.x86_64
> python-gzipstream-2.3.3-1.el6.noarch
> python-hwdata-1.7.3-1.el6.noarch
> python-iniparse-0.3.1-2.1.el6.noarch
> python-iwlib-0.1-1.2.el6.x86_64
> python-libs-2.6.6-66.0.1.el6_8.x86_64
> python-markdown-2.0.1-4.el6.noarch
> python-netaddr-0.7.5-4.el6.noarch
> python-psycopg2-2.7.1-1.rhel6.x86_64
> python-pycurl-7.19.0-9.el6.x86_64
> python-pygments-1.1.1-2.el6.noarch
> python-requests-2.6.0-4.el6.noarch
> python-setuptools-0.6.10-3.el6.noarch
> python-simplejson-3.8.2-1.rhel6.x86_64
> python-six-1.9.0-2.el6.noarch
> python-sss-1.13.3-56.el6.x86_64
> python-sssdconfig-1.13.3-56.el6.noarch
> python-urlgrabber-3.9.1-11.0.1.el6.noarch
> python-urllib3-1.10.2-3.el6.noarch
> rpm-python-4.8.0-55.el6.x86_64
> setools-libs-python-3.3.7-4.el6.x86_64
>
> # java -version
> openjdk version "1.8.0_131"
> OpenJDK Runtime Environment (build 1.8.0_131-b11)
> OpenJDK 64-Bit Server VM (build 25.131-b11, mixed mode)
>
> # rpm -qa | grep tom | sort
> apache-tomcat-apis-0.1-1.el6.noarch
> jquery-ui-1.10.4.custom-3.el6.noarch
> tomcat5-jsp-2.0-api-5.5.27-7.jpp5.noarch
> tomcat5-servlet-2.4-api-5.5.27-7.jpp5.noarch
> tomcat6-6.0.24-105.el6_8.noarch
> tomcat6-el-2.1-api-6.0.24-105.el6_8.noarch
> tomcat6-jsp-2.1-api-6.0.24-105.el6_8.noarch
> tomcat6-lib-6.0.24-105.el6_8.noarch
> tomcat6-servlet-2.5-api-6.0.24-105.el6_8.noarch
>
> # rpm -qa | grep postgres | sort
> postgresql96-9.6.3-1PGDG.rhel6.x86_64
> postgresql96-contrib-9.6.3-1PGDG.rhel6.x86_64
> postgresql96-libs-9.6.3-1PGDG.rhel6.x86_64
> postgresql96-plperl-9.6.3-1PGDG.rhel6.x86_64
> postgresql96-plpython-9.6.3-1PGDG.rhel6.x86_64
> postgresql96-pltcl-9.6.3-1PGDG.rhel6.x86_64
> postgresql96-server-9.6.3-1PGDG.rhel6.x86_64
> postgresql-jdbc-42.1.1-1.rhel6.noarch
> spacewalk-backend-sql-postgresql-2.6.77-1.el6.noarch
> spacewalk-java-postgresql-2.6.49-1.el6.noarch
> spacewalk-setup-postgresql-2.6.2-1.el6.noarch
>
> for postgresql 9.6 I had to patch spacewalk within
> /usr/share/rhn/server/handlers/xmlrpc/queue.py addin "as"
> (http://git.net/ml/spacewalk-devel/2016-11/msg00004.html and
> https://github.com/spacewalkproject/spacewalk/blob/master/backend/server/handlers/xmlrpc/queue.py
> -- worked like charm!)
>
> Hopefully I did not forget some package spacewalk depends on ...
>
> --
> Thomas
--
Thomas
More information about the Spacewalk-list
mailing list