[Spacewalk-list] Trouble with spacewalk after long time running

Tue May 23 17:51:01 UTC 2017

2017-05-23 13:57 GMT+02:00 Michael Mraka <michael.mraka at redhat.com>:
> Thomas Schweikle:
>> Further things I found within httpd logs:
>> access_log:
>> [17/May/2017:16:27:21 +0200] "POST /cobbler_api HTTP/1.1" 200 144 "-"
>> "Java/1.8.0_131"
>> [17/May/2017:16:27:21 +0200] "POST /rpc/api HTTP/1.0" 404 952 "-"
>> "xmlrpclib.py/1.0.1 (by www.pythonware.com)"
> ...
>> Since 200 is OK, 404 is not. Apache seems to have problems with /rpc/api!
>> Within error_log there is nothing remarkable. Python 2.6.6 is OK or
>> does spacewalk rely on Python 3.x?
>
> No python 3, spacewalk still uses python 2.
>
>> [notice] SELinux policy enabled; httpd running as context
>> unconfined_u:system_r:httpd_t:s0
>> [notice] suEXEC mechanism enabled (wrapper: /usr/sbin/suexec)
>> [notice] Digest: generating secret for digest authentication ...
>> [notice] Digest: done
>> [notice] Apache/2.2.15 (Unix) DAV/2 mod_ssl/2.2.15 OpenSSL/1.0.1e-fips
>> mod_wsgi/3.2 Python/2.6.6 mod_apreq2-20090110/2.8.0 mod_perl/2.0.4
>> Perl/v5.10.1 configured -- resuming normal operations
>>
>> Client request seem to be handled OK:
>> [17/May/2017:16:36:04 +0200] "POST /XMLRPC HTTP/1.1" 200 112276
> ...
>
> /XMLRPC is handeled by spacewalk-backend so the python part is ok.
>
>> While web-requests are not:
>> [17/May/2017:16:36:34 +0200] "GET /rhn/schedule/CompletedActions.do
>> HTTP/1.1" 404 952
> ...
>
> This is handled by java/tomcat.
>
>> ssl_error_log is empty -- no certificate errors so far ;-)
>>
>> ssl_request_log seems good too:
>> TLSv1.2 ECDHE-RSA-AES256-GCM-SHA384 "GET
>> /rhn/schedule/CompletedActions.do HTTP/1.1" 952
>> TLSv1.2 ECDHE-RSA-AES256-GCM-SHA384 "GET /rhn/Login.do HTTP/1.1" 952
> ...
>> A bit clueless now! Looks like tomcat6 running, but connection to
>> apache broken. Maybe same with python, perl and what else spacewalk
>> may need to work. But why? Looks like all modules necessary are
>> loaded:
> ...
>> Any idea, help anyone?
>> > # cat catalina.out
> ...
>> > INFORMATION: Server startup in 6939 ms
>> >
>> > There are some failures "Error listenerStart". But I do not know where
>> > to look next.
>
> Enable tomcat verbose logging and restart it, hopefully it will give you
> better hint what's going on.
> (see https://www.jvmhost.com/articles/increase-tomcat-logging-verbosity-quick-problem-resolution/ hot to increase logging verbosity)

This was a helpful hint! Now I have:
SCHWERWIEGEND: Exception sending context initialized event to listener
instance of class com.redhat.rhn.webapp.RhnServletListener
java.lang.UnsupportedClassVersionError: org/postgresql/Driver :
Unsupported major.minor version 52.0 (unable to load class
org.postgresql.Driver)
        at org.apache.catalina.loader.WebappClassLoader.findClassInternal(WebappClassLoader.java:2334)
        at org.apache.catalina.loader.WebappClassLoader.findClass(WebappClassLoader.java:976)
        at org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1450)
        at org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1329)
        at java.lang.Class.forName0(Native Method)
        at java.lang.Class.forName(Class.java:195)
        at org.hibernate.connection.C3P0ConnectionProvider.configure(C3P0ConnectionProvider.java:92)
        at org.hibernate.connection.ConnectionProviderFactory.newConnectionProvider(ConnectionProviderFactory.java:124)
        at org.hibernate.connection.ConnectionProviderFactory.newConnectionProvider(ConnectionProviderFactory.java:56)
        at org.hibernate.cfg.SettingsFactory.createConnectionProvider(SettingsFactory.java:414)
        at org.hibernate.cfg.SettingsFactory.buildSettings(SettingsFactory.java:62)
        at org.hibernate.cfg.Configuration.buildSettings(Configuration.java:2009)
        at org.hibernate.cfg.Configuration.buildSessionFactory(Configuration.java:1292)
        at com.redhat.rhn.common.hibernate.ConnectionManager.createSessionFactory(ConnectionManager.java:205)
        at com.redhat.rhn.common.hibernate.ConnectionManager.initialize(ConnectionManager.java:134)
        at com.redhat.rhn.common.hibernate.HibernateFactory.createSessionFactory(HibernateFactory.java:97)
        at com.redhat.rhn.webapp.RhnServletListener.startHibernate(RhnServletListener.java:94)
        at com.redhat.rhn.webapp.RhnServletListener.contextInitialized(RhnServletListener.java:116)
        at org.apache.catalina.core.StandardContext.listenerStart(StandardContext.java:4244)
        at org.apache.catalina.core.StandardContext.start(StandardContext.java:4743)
        at org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:791)
        at org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:771)
        at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:526)
        at org.apache.catalina.startup.HostConfig.deployDescriptor(HostConfig.java:637)
        at org.apache.catalina.startup.HostConfig.deployDescriptors(HostConfig.java:563)
        at org.apache.catalina.startup.HostConfig.deployApps(HostConfig.java:498)
        at org.apache.catalina.startup.HostConfig.start(HostConfig.java:1277)
        at org.apache.catalina.startup.HostConfig.lifecycleEvent(HostConfig.java:321)
        at org.apache.catalina.util.LifecycleSupport.fireLifecycleEvent(LifecycleSupport.java:142)
        at org.apache.catalina.core.ContainerBase.start(ContainerBase.java:1053)
        at org.apache.catalina.core.StandardHost.start(StandardHost.java:722)

jdbc installed is: postgresql-jdbc-42.1.1-1.rhel6.noarch
As far as I found this version is the right one for PostgreSQL 9.6.3?

Having installed:
- Spacewalk 2.6
- PostgreSQL 9.6
- JDK 1.8.0_131

Statements from https://jdbc.postgresql.org/download.html
- "It supports PostgreSQL 8.2 or newer and requires Java 6 or newer.
It contains support for SSL and the javax.sql package."
- "If you are using Java 8 or newer then you should use the JDBC 4.2 version."

JDBC is installed by rpm into:
# rpm -ql postgresql-jdbc-42.1.1-1.rhel6.noarch
/usr/share/doc/postgresql-jdbc-42.1.1
/usr/share/doc/postgresql-jdbc-42.1.1/LICENSE
/usr/share/doc/postgresql-jdbc-42.1.1/README.md
/usr/share/java/postgresql-jdbc.jar
/usr/share/java/postgresql-jdbc2.jar
/usr/share/java/postgresql-jdbc2ee.jar
/usr/share/java/postgresql-jdbc3.jar
/usr/share/maven2/poms/JPP-postgresql-jdbc.pom

Within WEB-INF/lib/ there is a link to this jar. I'd assume this jar
being used then?

A little bit confused about the "Unsupported major.minor version 52.0".
It can't load a driver, but which one? and how to fix?

>> > In /var/log/rhn/rhn_taskomatic_daemon.log I could find a variety of
>> > error messages:
>> > INFO   | jvm 1    | 2017/05/17 09:37:33 | 2017-05-17 09:37:33,741
>> > [DefaultQuartzScheduler_Worker-2] ERROR
>> > com.redhat.rhn.manager.kickstart.cobbler.CobblerLoginCommand -
>> > XmlRpcFault while logging in.  most likely user doesn't have
>> > permissions.
>
> It tries to contact running tomcat so I believe this will disappear once tomcat is fixed.

-- 
Thomas