[Spacewalk-list] Really slow unresponsive GUI

Fri Mar 15 07:55:37 UTC 2019

> Gesendet: Freitag, 15. März 2019 um 08:35 Uhr
> Von: "Robert Paschedag" <robert.paschedag at web.de>
> An: "Lionel Caignec" <caignec at cines.fr>
> Cc: "spacewalk-listredhat.com" <spacewalk-list at redhat.com>
> Betreff: Re: [Spacewalk-list] Really slow unresponsive GUI
>
> 
> 
> > Gesendet: Donnerstag, 14. März 2019 um 21:53 Uhr
> > Von: "Lionel Caignec" <caignec at cines.fr>
> > An: "Robert Paschedag" <robert.paschedag at web.de>
> > Betreff: Re: [Spacewalk-list] Really slow unresponsive GUI
> >
> > no problem with cobbler is running like all other service.
> > when i try to telnet to cobbler port i get process responding.
> > 
> > I'm really lost. Is it possible my kickstart profile are corrupted which give a bug with cobbler?
> > 
> > ----- Mail original -----
> > De: "Robert Paschedag" <robert.paschedag at web.de>
> > À: "spacewalk-list" <spacewalk-list at redhat.com>, "Lionel Caignec" <caignec at cines.fr>, "spacewalk-list" <spacewalk-list at redhat.com>
> > Envoyé: Jeudi 14 Mars 2019 17:24:16
> > Objet: Re: [Spacewalk-list] Really slow unresponsive GUI
> > 
> > Am 14. März 2019 12:58:23 MEZ schrieb Lionel Caignec <caignec at cines.fr>:
> > > Hi,
> > >
> > >finally i managed to get the source of the bug, it's about then link
> > >between tomcat and cobbler. The bug appears only when i try to view
> > >kickstart profiles.
> > >
> > >In cobbler i get this error message : 
> > >Wed Mar 13 12:06:01 2019 - INFO | Exception Info:
> > >File "/usr/lib/python2.7/site-packages/cobbler/remote.py", line 1770,
> > >in _dispatch
> > >    return method_handle(*params)
> > >File "/usr/lib/python2.7/site-packages/cobbler/remote.py", line 1598,
> > >in token_check
> > >    self.__validate_token(token)
> > >File "/usr/lib/python2.7/site-packages/cobbler/remote.py", line 1510,
> > >in __validate_token
> > >    raise CX("invalid token: %s" % token)
> > >
> > >
> > >In tomcat log i get this error :
> > >mars 13, 2019 12:11:38 PM org.apache.catalina.core.StandardWrapperValve
> > >invoke
> > >GRAVE: Servlet.service() for servlet [action] in context with path
> > >[/rhn] threw exception [java.lang.RuntimeException: XmlRpcException
> > >calling cobbler.] with root cause
> 
> 
> > >java.io.IOException: Server returned HTTP response code: 502 for URL:
> > >http://FQDN:80/cobbler_api
> 
> Hmm....did you replace the original server hostname with "FQDN" here in the log output?
> 
> Robert

Please see this test script. Does this succeed?

https://github.com/spacewalkproject/spacewalk/blob/9be948a4a01d0dd41baed0c6d0032a2078d336d3/java/scripts/test_cobbler_sw_login.py

Robert

> 
> > >at
> > >sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1894)
> > >at
> > >sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1492)
> > >  at redstone.xmlrpc.XmlRpcClient.handleResponse(XmlRpcClient.java:420)
> > >    at redstone.xmlrpc.XmlRpcClient.endCall(XmlRpcClient.java:376)
> > >    at redstone.xmlrpc.XmlRpcClient.invoke(XmlRpcClient.java:165)
> > >at
> > >com.redhat.rhn.manager.kickstart.cobbler.CobblerXMLRPCHelper.invokeMethod(CobblerXMLRPCHelper.java:69)
> > >at
> > >com.redhat.rhn.manager.kickstart.cobbler.CobblerLoginCommand.login(CobblerLoginCommand.java:52)
> > >at
> > >com.redhat.rhn.frontend.integration.IntegrationService.authorize(IntegrationService.java:114)
> > >at
> > >com.redhat.rhn.frontend.integration.IntegrationService.getAuthToken(IntegrationService.java:74)
> > >at
> > >com.redhat.rhn.manager.kickstart.cobbler.CobblerXMLRPCHelper.getConnection(CobblerXMLRPCHelper.java:93)
> > >at
> > >com.redhat.rhn.manager.kickstart.cobbler.CobblerXMLRPCHelper.getConnection(CobblerXMLRPCHelper.java:83)
> > >at
> > >com.redhat.rhn.manager.kickstart.KickstartLister.listCobblerProfiles(KickstartLister.java:436)
> > >at
> > >com.redhat.rhn.frontend.action.kickstart.KickstartsSetupAction.execute(KickstartsSetupAction.java:64)
> > >at
> > >org.apache.struts.action.RequestProcessor.processActionPerform(RequestProcessor.java:425)
> > >at
> > >org.apache.struts.action.RequestProcessor.process(RequestProcessor.java:229)
> > >at
> > >com.redhat.rhn.frontend.struts.RhnRequestProcessor.process(RhnRequestProcessor.java:105)
> > >at
> > >org.apache.struts.action.ActionServlet.process(ActionServlet.java:1926)
> > >at org.apache.struts.action.ActionServlet.doGet(ActionServlet.java:451)
> > >    at javax.servlet.http.HttpServlet.service(HttpServlet.java:624)
> > >    at javax.servlet.http.HttpServlet.service(HttpServlet.java:731)
> > >at
> > >org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:303)
> > >at
> > >org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
> > >at
> > >org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52)
> > >at
> > >org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
> > >at
> > >org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
> > >at
> > >com.redhat.rhn.frontend.servlets.AuthFilter.doFilter(AuthFilter.java:107)
> > >at
> > >org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
> > >at
> > >org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
> > >at
> > >com.opensymphony.sitemesh.webapp.SiteMeshFilter.obtainContent(SiteMeshFilter.java:129)
> > >at
> > >com.opensymphony.sitemesh.webapp.SiteMeshFilter.doFilter(SiteMeshFilter.java:77)
> > >at
> > >org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
> > >at
> > >org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
> > >at
> > >com.redhat.rhn.frontend.servlets.LocalizedEnvironmentFilter.doFilter(LocalizedEnvironmentFilter.java:67)
> > >at
> > >org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
> > >at
> > >org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
> > >at
> > >com.redhat.rhn.frontend.servlets.EnvironmentFilter.doFilter(EnvironmentFilter.java:101)
> > >at
> > >org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
> > >at
> > >org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
> > >at
> > >com.redhat.rhn.frontend.servlets.SessionFilter.doFilter(SessionFilter.java:58)
> > >at
> > >org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
> > >at
> > >org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
> > >at
> > >com.redhat.rhn.frontend.servlets.SetCharacterEncodingFilter.doFilter(SetCharacterEncodingFilter.java:97)
> > >at
> > >org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
> > >at
> > >org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
> > >at
> > >org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:218)
> > >at
> > >org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:110)
> > >at
> > >org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:506)
> > >at
> > >org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:169)
> > >at
> > >org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:103)
> > >at
> > >org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:962)
> > >at
> > >org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:116)
> > >at
> > >org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:445)
> > >   at org.apache.coyote.ajp.AjpProcessor.process(AjpProcessor.java:190)
> > >at
> > >org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:637)
> > >at
> > >org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:316)
> > >at
> > >java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> > >at
> > >java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> > >at
> > >org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
> > >    at java.lang.Thread.run(Thread.java:748)
> > >
> > >
> > >Someone have an idea please?
> > >
> > >Regards Lionel
> > >
> > >----- Mail original -----
> > >De: "Lionel Caignec" <caignec at cines.fr>
> > >À: "spacewalk-list" <spacewalk-list at redhat.com>
> > >Envoyé: Lundi 11 Mars 2019 10:14:05
> > >Objet: Re: [Spacewalk-list] Really slow unresponsive GUI
> > >
> > >Yes i even rebooted the server.
> > >
> > >Here is the exact error message in sync log : 
> > >ERROR: (23, 'ERROR: duplicate key value violates unique constraint
> > >"rhn_cnp_cid_nid_uq"', 'Could not update database entry.')
> > >
> > >
> > >I double checked this morning all is good until i click on "kickstart
> > >--> distributions" or "kickstart --> profiles" then all server is
> > >unresponsive.
> > >
> > >So my idea is to create a trying delete "kickstart profile" but i
> > >cannot do it with gui/api so can i delete in database directly? In
> > >which table ?
> > >
> > >
> > >Regards,
> > >Lionel.
> > >
> > >----- Mail original -----
> > >De: "P Cookson" <P.Cookson at bham.ac.uk>
> > >À: "spacewalk-list" <spacewalk-list at redhat.com>
> > >Envoyé: Lundi 11 Mars 2019 09:22:57
> > >Objet: Re: [Spacewalk-list] Really slow unresponsive GUI
> > >
> > >Just checking, did you re-start the Taskomatic service after changing
> > >the setting too?
> > >
> > >Regards
> > >Phil
> > >
> > >-----Original Message-----
> > >From: spacewalk-list-bounces at redhat.com
> > ><spacewalk-list-bounces at redhat.com> On Behalf Of caignec at cines.fr
> > >Sent: 08 March 2019 15:56
> > >To: spacewalk-list <spacewalk-list at redhat.com>
> > >Subject: Re: [Spacewalk-list] Really slow unresponsive GUI
> > >
> > >Hi Phil,
> > >
> > >thank for your help. No error about java heap space, but tried to
> > >configure taskomatic memory like you said without success.
> > >
> > >I setup the kickstart part long time ago, just using satellite redhat
> > >doc.
> > >
> > >----- Mail original -----
> > >De: "P Cookson" <P.Cookson at bham.ac.uk>
> > >À: "spacewalk-list" <spacewalk-list at redhat.com>
> > >Envoyé: Vendredi 8 Mars 2019 16:43:23
> > >Objet: Re: [Spacewalk-list] Really slow unresponsive GUI
> > >
> > >Hi Lionel
> > >
> > >Have you looked in /var/log/rhn/rhn_taskomatic_daemon.log too? After
> > >seeing "java.lang.OutOfMemoryError: Java heap space" messages, in
> > >there, I changed config for Taskomatic but not sure if this is relevant
> > >for you?
> > >
> > >Current maxmemory (Xmx) can be seen by checking status of Taskomatic:
> > >
> > ># systemctl status taskomatic.service
> > >
> > >I increased from 256 to 4096:
> > >
> > ># grep taskomatic.java.maxmemory /etc/rhn/rhn.conf
> > >taskomatic.java.maxmemory=4096
> > >#
> > >
> > >Then re-started Spacewalk:
> > >
> > ># spacewalk-service restart
> > >
> > >Also, I've noted you've said something about using Kickstart. After
> > >patching for some time I need to start looking at provisioning, too, so
> > >if you have any notes/links about setup that would be useful.
> > >
> > >
> > >Regards
> > >Phil
> > >
> > >-----Original Message-----
> > >From: spacewalk-list-bounces at redhat.com
> > ><spacewalk-list-bounces at redhat.com> On Behalf Of caignec at cines.fr
> > >Sent: 08 March 2019 14:59
> > >To: spacewalk-list <spacewalk-list at redhat.com>
> > >Subject: [Spacewalk-list] Really slow unresponsive GUI
> > >
> > >Hi,
> > >
> > >i'm writing here because i've a strange behaviour with no clue on how
> > >to resolve it. 
> > >There is no load on CPU, ram is mostly free, no IO wait, no processus
> > >stuck a 100%. In log file (tomcat/postresql,...) nothing about error or
> > >warning.
> > >No recent modification on the system.
> > >
> > >When i want to access spacewalk web interface (or API) this one
> > >sometimes answer after a very long time, and sometimes get error 500.
> > >Same behaviour with the spacecmd cli command.
> > >
> > >I really don't understand because i've another spacewalk on another
> > >server and this one work like a charm.
> > >
> > >My setup : VM with 2vpcu 8Gb RAM near 200 clients spacewalk 2.8, used
> > >only for patching and sometimes kickstart new server.
> > > JAVA_OPTS -Xms2048m -Xmx2048m, tried with 4096 but no difference.
> > >
> > >I don't know which log to put here due to the fact i found no error
> > >trace.
> > >
> > >For now the only solution I imagine is to backup and restore to a new
> > >server freshly installed my spacewalk.
> > >
> > >Thank you for help.
> > >
> > >--
> > >Lionel Caignec 
> > >
> > >Centre Informatique National de l' Enseignement Supérieur 
> > >950 rue de Saint Priest 
> > >34097 MONTPELLIER Cedex 5 
> > >Tel : (33) 04 67 14 14 14
> > >Fax : (33)04 67 52 37 63 
> > >https://www.cines.fr
> > >
> > >_______________________________________________
> > >Spacewalk-list mailing list
> > >Spacewalk-list at redhat.com
> > >https://www.redhat.com/mailman/listinfo/spacewalk-list
> > >
> > >_______________________________________________
> > >Spacewalk-list mailing list
> > >Spacewalk-list at redhat.com
> > >https://www.redhat.com/mailman/listinfo/spacewalk-list
> > >_______________________________________________
> > >Spacewalk-list mailing list
> > >Spacewalk-list at redhat.com
> > >https://www.redhat.com/mailman/listinfo/spacewalk-list
> > 
> > cobbler not running??
> > 
> > 
> > -- 
> > sent from my mobile device
> 
> _______________________________________________
> Spacewalk-list mailing list
> Spacewalk-list at redhat.com
> https://www.redhat.com/mailman/listinfo/spacewalk-list