[Spacewalk-list] Really slow unresponsive GUI

Lionel Caignec caignec at cines.fr
Thu Mar 14 11:58:23 UTC 2019 

 Hi,

finally i managed to get the source of the bug, it's about then link between tomcat and cobbler. The bug appears only when i try to view kickstart profiles.

In cobbler i get this error message : 
Wed Mar 13 12:06:01 2019 - INFO | Exception Info:
  File "/usr/lib/python2.7/site-packages/cobbler/remote.py", line 1770, in _dispatch
    return method_handle(*params)
   File "/usr/lib/python2.7/site-packages/cobbler/remote.py", line 1598, in token_check
    self.__validate_token(token)
   File "/usr/lib/python2.7/site-packages/cobbler/remote.py", line 1510, in __validate_token
    raise CX("invalid token: %s" % token)


In tomcat log i get this error :
mars 13, 2019 12:11:38 PM org.apache.catalina.core.StandardWrapperValve invoke
GRAVE: Servlet.service() for servlet [action] in context with path [/rhn] threw exception [java.lang.RuntimeException: XmlRpcException calling cobbler.] with root cause
java.io.IOException: Server returned HTTP response code: 502 for URL: http://FQDN:80/cobbler_api
    at sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1894)
    at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1492)
    at redstone.xmlrpc.XmlRpcClient.handleResponse(XmlRpcClient.java:420)
    at redstone.xmlrpc.XmlRpcClient.endCall(XmlRpcClient.java:376)
    at redstone.xmlrpc.XmlRpcClient.invoke(XmlRpcClient.java:165)
    at com.redhat.rhn.manager.kickstart.cobbler.CobblerXMLRPCHelper.invokeMethod(CobblerXMLRPCHelper.java:69)
    at com.redhat.rhn.manager.kickstart.cobbler.CobblerLoginCommand.login(CobblerLoginCommand.java:52)
    at com.redhat.rhn.frontend.integration.IntegrationService.authorize(IntegrationService.java:114)
    at com.redhat.rhn.frontend.integration.IntegrationService.getAuthToken(IntegrationService.java:74)
    at com.redhat.rhn.manager.kickstart.cobbler.CobblerXMLRPCHelper.getConnection(CobblerXMLRPCHelper.java:93)
    at com.redhat.rhn.manager.kickstart.cobbler.CobblerXMLRPCHelper.getConnection(CobblerXMLRPCHelper.java:83)
    at com.redhat.rhn.manager.kickstart.KickstartLister.listCobblerProfiles(KickstartLister.java:436)
    at com.redhat.rhn.frontend.action.kickstart.KickstartsSetupAction.execute(KickstartsSetupAction.java:64)
    at org.apache.struts.action.RequestProcessor.processActionPerform(RequestProcessor.java:425)
    at org.apache.struts.action.RequestProcessor.process(RequestProcessor.java:229)
    at com.redhat.rhn.frontend.struts.RhnRequestProcessor.process(RhnRequestProcessor.java:105)
    at org.apache.struts.action.ActionServlet.process(ActionServlet.java:1926)
    at org.apache.struts.action.ActionServlet.doGet(ActionServlet.java:451)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:624)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:731)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:303)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
    at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
    at com.redhat.rhn.frontend.servlets.AuthFilter.doFilter(AuthFilter.java:107)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
    at com.opensymphony.sitemesh.webapp.SiteMeshFilter.obtainContent(SiteMeshFilter.java:129)
    at com.opensymphony.sitemesh.webapp.SiteMeshFilter.doFilter(SiteMeshFilter.java:77)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
    at com.redhat.rhn.frontend.servlets.LocalizedEnvironmentFilter.doFilter(LocalizedEnvironmentFilter.java:67)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
    at com.redhat.rhn.frontend.servlets.EnvironmentFilter.doFilter(EnvironmentFilter.java:101)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
    at com.redhat.rhn.frontend.servlets.SessionFilter.doFilter(SessionFilter.java:58)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
    at com.redhat.rhn.frontend.servlets.SetCharacterEncodingFilter.doFilter(SetCharacterEncodingFilter.java:97)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
    at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:218)
    at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:110)
    at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:506)
    at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:169)
    at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:103)
    at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:962)
    at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:116)
    at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:445)
    at org.apache.coyote.ajp.AjpProcessor.process(AjpProcessor.java:190)
    at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:637)
    at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:316)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
    at java.lang.Thread.run(Thread.java:748)


Someone have an idea please?

Regards Lionel

----- Mail original -----
De: "Lionel Caignec" <caignec at cines.fr>
À: "spacewalk-list" <spacewalk-list at redhat.com>
Envoyé: Lundi 11 Mars 2019 10:14:05
Objet: Re: [Spacewalk-list] Really slow unresponsive GUI

Yes i even rebooted the server.

Here is the exact error message in sync log : 
ERROR: (23, 'ERROR: duplicate key value violates unique constraint "rhn_cnp_cid_nid_uq"', 'Could not update database entry.')


I double checked this morning all is good until i click on "kickstart --> distributions" or "kickstart --> profiles" then all server is unresponsive.

So my idea is to create a trying delete "kickstart profile" but i cannot do it with gui/api so can i delete in database directly? In which table ?


Regards,
Lionel.

----- Mail original -----
De: "P Cookson" <P.Cookson at bham.ac.uk>
À: "spacewalk-list" <spacewalk-list at redhat.com>
Envoyé: Lundi 11 Mars 2019 09:22:57
Objet: Re: [Spacewalk-list] Really slow unresponsive GUI

Just checking, did you re-start the Taskomatic service after changing the setting too?

Regards
Phil

-----Original Message-----
From: spacewalk-list-bounces at redhat.com <spacewalk-list-bounces at redhat.com> On Behalf Of caignec at cines.fr
Sent: 08 March 2019 15:56
To: spacewalk-list <spacewalk-list at redhat.com>
Subject: Re: [Spacewalk-list] Really slow unresponsive GUI

Hi Phil,

thank for your help. No error about java heap space, but tried to configure taskomatic memory like you said without success.

I setup the kickstart part long time ago, just using satellite redhat doc.

----- Mail original -----
De: "P Cookson" <P.Cookson at bham.ac.uk>
À: "spacewalk-list" <spacewalk-list at redhat.com>
Envoyé: Vendredi 8 Mars 2019 16:43:23
Objet: Re: [Spacewalk-list] Really slow unresponsive GUI

Hi Lionel

Have you looked in /var/log/rhn/rhn_taskomatic_daemon.log too? After seeing "java.lang.OutOfMemoryError: Java heap space" messages, in there, I changed config for Taskomatic but not sure if this is relevant for you?

Current maxmemory (Xmx) can be seen by checking status of Taskomatic:

# systemctl status taskomatic.service

I increased from 256 to 4096:

# grep taskomatic.java.maxmemory /etc/rhn/rhn.conf
taskomatic.java.maxmemory=4096
#

Then re-started Spacewalk:

# spacewalk-service restart

Also, I've noted you've said something about using Kickstart. After patching for some time I need to start looking at provisioning, too, so if you have any notes/links about setup that would be useful.


Regards
Phil

-----Original Message-----
From: spacewalk-list-bounces at redhat.com <spacewalk-list-bounces at redhat.com> On Behalf Of caignec at cines.fr
Sent: 08 March 2019 14:59
To: spacewalk-list <spacewalk-list at redhat.com>
Subject: [Spacewalk-list] Really slow unresponsive GUI

Hi,

i'm writing here because i've a strange behaviour with no clue on how to resolve it. 
There is no load on CPU, ram is mostly free, no IO wait, no processus stuck a 100%. In log file (tomcat/postresql,...) nothing about error or warning.
No recent modification on the system.

When i want to access spacewalk web interface (or API) this one sometimes answer after a very long time, and sometimes get error 500. Same behaviour with the spacecmd cli command.

I really don't understand because i've another spacewalk on another server and this one work like a charm.

My setup : VM with 2vpcu 8Gb RAM near 200 clients spacewalk 2.8, used only for patching and sometimes kickstart new server.
 JAVA_OPTS -Xms2048m -Xmx2048m, tried with 4096 but no difference.

I don't know which log to put here due to the fact i found no error trace.

For now the only solution I imagine is to backup and restore to a new server freshly installed my spacewalk.

Thank you for help.

--
Lionel Caignec 

Centre Informatique National de l' Enseignement Supérieur 
950 rue de Saint Priest 
34097 MONTPELLIER Cedex 5 
Tel : (33) 04 67 14 14 14
Fax : (33)04 67 52 37 63 
https://www.cines.fr

_______________________________________________
Spacewalk-list mailing list
Spacewalk-list at redhat.com
https://www.redhat.com/mailman/listinfo/spacewalk-list

_______________________________________________
Spacewalk-list mailing list
Spacewalk-list at redhat.com
https://www.redhat.com/mailman/listinfo/spacewalk-list
_______________________________________________
Spacewalk-list mailing list
Spacewalk-list at redhat.com
https://www.redhat.com/mailman/listinfo/spacewalk-list
-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 2049 bytes
Desc: S/MIME Cryptographic Signature
URL: <http://listman.redhat.com/archives/spacewalk-list/attachments/20190314/12364ee3/attachment.p7s>

More information about the Spacewalk-list mailing list