[Spacewalk-list] Trouble with spacewalk after long time running

[Michael Mraka]

Thomas Schweikle:
> Further things I found within httpd logs:
> access_log:
> [17/May/2017:16:27:21 +0200] "POST /cobbler_api HTTP/1.1" 200 144 "-"
> "Java/1.8.0_131"
> [17/May/2017:16:27:21 +0200] "POST /rpc/api HTTP/1.0" 404 952 "-"
> "xmlrpclib.py/1.0.1 (by www.pythonware.com)"
...
> Since 200 is OK, 404 is not. Apache seems to have problems with /rpc/api!
> Within error_log there is nothing remarkable. Python 2.6.6 is OK or
> does spacewalk rely on Python 3.x?

No python 3, spacewalk still uses python 2.

> [notice] SELinux policy enabled; httpd running as context
> unconfined_u:system_r:httpd_t:s0
> [notice] suEXEC mechanism enabled (wrapper: /usr/sbin/suexec)
> [notice] Digest: generating secret for digest authentication ...
> [notice] Digest: done
> [notice] Apache/2.2.15 (Unix) DAV/2 mod_ssl/2.2.15 OpenSSL/1.0.1e-fips
> mod_wsgi/3.2 Python/2.6.6 mod_apreq2-20090110/2.8.0 mod_perl/2.0.4
> Perl/v5.10.1 configured -- resuming normal operations
> 
> Client request seem to be handled OK:
> [17/May/2017:16:36:04 +0200] "POST /XMLRPC HTTP/1.1" 200 112276
...

/XMLRPC is handeled by spacewalk-backend so the python part is ok.

> While web-requests are not:
> [17/May/2017:16:36:34 +0200] "GET /rhn/schedule/CompletedActions.do
> HTTP/1.1" 404 952
...

This is handled by java/tomcat.

> ssl_error_log is empty -- no certificate errors so far ;-)
> 
> ssl_request_log seems good too:
> TLSv1.2 ECDHE-RSA-AES256-GCM-SHA384 "GET
> /rhn/schedule/CompletedActions.do HTTP/1.1" 952
> TLSv1.2 ECDHE-RSA-AES256-GCM-SHA384 "GET /rhn/Login.do HTTP/1.1" 952
...
> A bit clueless now! Looks like tomcat6 running, but connection to
> apache broken. Maybe same with python, perl and what else spacewalk
> may need to work. But why? Looks like all modules necessary are
> loaded:
...
> Any idea, help anyone?
> > # cat catalina.out
...
> > INFORMATION: Server startup in 6939 ms
> >
> > There are some failures "Error listenerStart". But I do not know where
> > to look next.

Enable tomcat verbose logging and restart it, hopefully it will give you
better hint what's going on.
(see https://www.jvmhost.com/articles/increase-tomcat-logging-verbosity-quick-problem-resolution/ hot to increase logging verbosity) 

> > In /var/log/rhn/rhn_taskomatic_daemon.log I could find a variety of
> > error messages:
> > INFO   | jvm 1    | 2017/05/17 09:37:33 | 2017-05-17 09:37:33,741
> > [DefaultQuartzScheduler_Worker-2] ERROR
> > com.redhat.rhn.manager.kickstart.cobbler.CobblerLoginCommand -
> > XmlRpcFault while logging in.  most likely user doesn't have
> > permissions.

It tries to contact running tomcat so I believe this will disappear once tomcat is fixed.


--
Michael Mráka
System Management Engineering, Red Hat