[Spacewalk-list] Trouble with spacewalk after long time running

Wed May 24 06:58:27 UTC 2017

2017-05-23 19:51 GMT+02:00 Thomas Schweikle <tschweikle at gmail.com>:
> 2017-05-23 13:57 GMT+02:00 Michael Mraka <michael.mraka at redhat.com>:
>> Thomas Schweikle:
>>> Further things I found within httpd logs:
>>> access_log:
>>> [17/May/2017:16:27:21 +0200] "POST /cobbler_api HTTP/1.1" 200 144 "-"
>>> "Java/1.8.0_131"
>>> [17/May/2017:16:27:21 +0200] "POST /rpc/api HTTP/1.0" 404 952 "-"
>>> "xmlrpclib.py/1.0.1 (by www.pythonware.com)"
>> ...
>>> Since 200 is OK, 404 is not. Apache seems to have problems with /rpc/api!
>>> Within error_log there is nothing remarkable. Python 2.6.6 is OK or
>>> does spacewalk rely on Python 3.x?
>>
>> No python 3, spacewalk still uses python 2.
>>
>>> [notice] SELinux policy enabled; httpd running as context
>>> unconfined_u:system_r:httpd_t:s0
>>> [notice] suEXEC mechanism enabled (wrapper: /usr/sbin/suexec)
>>> [notice] Digest: generating secret for digest authentication ...
>>> [notice] Digest: done
>>> [notice] Apache/2.2.15 (Unix) DAV/2 mod_ssl/2.2.15 OpenSSL/1.0.1e-fips
>>> mod_wsgi/3.2 Python/2.6.6 mod_apreq2-20090110/2.8.0 mod_perl/2.0.4
>>> Perl/v5.10.1 configured -- resuming normal operations
>>>
>>> Client request seem to be handled OK:
>>> [17/May/2017:16:36:04 +0200] "POST /XMLRPC HTTP/1.1" 200 112276
>> ...
>>
>> /XMLRPC is handeled by spacewalk-backend so the python part is ok.
>>
>>> While web-requests are not:
>>> [17/May/2017:16:36:34 +0200] "GET /rhn/schedule/CompletedActions.do
>>> HTTP/1.1" 404 952
>> ...
>>
>> This is handled by java/tomcat.
>>
>>> ssl_error_log is empty -- no certificate errors so far ;-)
>>>
>>> ssl_request_log seems good too:
>>> TLSv1.2 ECDHE-RSA-AES256-GCM-SHA384 "GET
>>> /rhn/schedule/CompletedActions.do HTTP/1.1" 952
>>> TLSv1.2 ECDHE-RSA-AES256-GCM-SHA384 "GET /rhn/Login.do HTTP/1.1" 952
>> ...
>>> A bit clueless now! Looks like tomcat6 running, but connection to
>>> apache broken. Maybe same with python, perl and what else spacewalk
>>> may need to work. But why? Looks like all modules necessary are
>>> loaded:
>> ...
>>> Any idea, help anyone?
>>> > # cat catalina.out
>> ...
>>> > INFORMATION: Server startup in 6939 ms
>>> >
>>> > There are some failures "Error listenerStart". But I do not know where
>>> > to look next.
>>
>> Enable tomcat verbose logging and restart it, hopefully it will give you
>> better hint what's going on.
>> (see https://www.jvmhost.com/articles/increase-tomcat-logging-verbosity-quick-problem-resolution/ hot to increase logging verbosity)
>
> This was a helpful hint! Now I have:
> SCHWERWIEGEND: Exception sending context initialized event to listener
> instance of class com.redhat.rhn.webapp.RhnServletListener
> java.lang.UnsupportedClassVersionError: org/postgresql/Driver :
> Unsupported major.minor version 52.0 (unable to load class
> org.postgresql.Driver)
>         at org.apache.catalina.loader.WebappClassLoader.findClassInternal(WebappClassLoader.java:2334)
>         at org.apache.catalina.loader.WebappClassLoader.findClass(WebappClassLoader.java:976)
>         at org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1450)
>         at org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1329)
>         at java.lang.Class.forName0(Native Method)
>         at java.lang.Class.forName(Class.java:195)
>         at org.hibernate.connection.C3P0ConnectionProvider.configure(C3P0ConnectionProvider.java:92)
>         at org.hibernate.connection.ConnectionProviderFactory.newConnectionProvider(ConnectionProviderFactory.java:124)
>         at org.hibernate.connection.ConnectionProviderFactory.newConnectionProvider(ConnectionProviderFactory.java:56)
>         at org.hibernate.cfg.SettingsFactory.createConnectionProvider(SettingsFactory.java:414)
>         at org.hibernate.cfg.SettingsFactory.buildSettings(SettingsFactory.java:62)
>         at org.hibernate.cfg.Configuration.buildSettings(Configuration.java:2009)
>         at org.hibernate.cfg.Configuration.buildSessionFactory(Configuration.java:1292)
>         at com.redhat.rhn.common.hibernate.ConnectionManager.createSessionFactory(ConnectionManager.java:205)
>         at com.redhat.rhn.common.hibernate.ConnectionManager.initialize(ConnectionManager.java:134)
>         at com.redhat.rhn.common.hibernate.HibernateFactory.createSessionFactory(HibernateFactory.java:97)
>         at com.redhat.rhn.webapp.RhnServletListener.startHibernate(RhnServletListener.java:94)
>         at com.redhat.rhn.webapp.RhnServletListener.contextInitialized(RhnServletListener.java:116)
>         at org.apache.catalina.core.StandardContext.listenerStart(StandardContext.java:4244)
>         at org.apache.catalina.core.StandardContext.start(StandardContext.java:4743)
>         at org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:791)
>         at org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:771)
>         at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:526)
>         at org.apache.catalina.startup.HostConfig.deployDescriptor(HostConfig.java:637)
>         at org.apache.catalina.startup.HostConfig.deployDescriptors(HostConfig.java:563)
>         at org.apache.catalina.startup.HostConfig.deployApps(HostConfig.java:498)
>         at org.apache.catalina.startup.HostConfig.start(HostConfig.java:1277)
>         at org.apache.catalina.startup.HostConfig.lifecycleEvent(HostConfig.java:321)
>         at org.apache.catalina.util.LifecycleSupport.fireLifecycleEvent(LifecycleSupport.java:142)
>         at org.apache.catalina.core.ContainerBase.start(ContainerBase.java:1053)
>         at org.apache.catalina.core.StandardHost.start(StandardHost.java:722)
>
> jdbc installed is: postgresql-jdbc-42.1.1-1.rhel6.noarch
> As far as I found this version is the right one for PostgreSQL 9.6.3?
>
> Having installed:
> - Spacewalk 2.6
> - PostgreSQL 9.6
> - JDK 1.8.0_131
>
> Statements from https://jdbc.postgresql.org/download.html
> - "It supports PostgreSQL 8.2 or newer and requires Java 6 or newer.
> It contains support for SSL and the javax.sql package."
> - "If you are using Java 8 or newer then you should use the JDBC 4.2 version."
>
> JDBC is installed by rpm into:
> # rpm -ql postgresql-jdbc-42.1.1-1.rhel6.noarch
> /usr/share/doc/postgresql-jdbc-42.1.1
> /usr/share/doc/postgresql-jdbc-42.1.1/LICENSE
> /usr/share/doc/postgresql-jdbc-42.1.1/README.md
> /usr/share/java/postgresql-jdbc.jar
> /usr/share/java/postgresql-jdbc2.jar
> /usr/share/java/postgresql-jdbc2ee.jar
> /usr/share/java/postgresql-jdbc3.jar
> /usr/share/maven2/poms/JPP-postgresql-jdbc.pom
>
> Within WEB-INF/lib/ there is a link to this jar. I'd assume this jar
> being used then?
>
> A little bit confused about the "Unsupported major.minor version 52.0".
> It can't load a driver, but which one? and how to fix?

Found the problem: some links in
/var/lib/tomcat/webapp/rhn/WEB-INF/lib where not correct: they pointed
to wrong jar-files. After reinstalling the packages they belonged to
this was corrected. Now it is working again!

>>> > In /var/log/rhn/rhn_taskomatic_daemon.log I could find a variety of
>>> > error messages:
>>> > INFO   | jvm 1    | 2017/05/17 09:37:33 | 2017-05-17 09:37:33,741
>>> > [DefaultQuartzScheduler_Worker-2] ERROR
>>> > com.redhat.rhn.manager.kickstart.cobbler.CobblerLoginCommand -
>>> > XmlRpcFault while logging in.  most likely user doesn't have
>>> > permissions.
>>
>> It tries to contact running tomcat so I believe this will disappear once tomcat is fixed.

Thanks for all the help!

-- 
Thomas