[Spacewalk-list] Spacewalk proxy: a lot of TRACEBACK messages: error in return cPickle.loads(pickled): EOFError

Sat Jan 9 21:09:00 UTC 2016

Hello,

Few days ago I've started to be spammed with TRACEBACK mails from our
Spacewalk proxy system; the erorrs are generated each two-three minutes.
The error stacktrace in Spacewalk proxy Apache error log is:

===
[Sat Jan 09 21:56:42 2016] [error] Exception reported from
spacewalk.fr.cloud.com
[Sat Jan 09 21:56:42 2016] [error] Time: Sat Jan  9 21:56:42 2016
[Sat Jan 09 21:56:42 2016] [error] Exception type <type
'exceptions.EOFError'>
[Sat Jan 09 21:56:42 2016] [error] Exception while handling function handler
[Sat Jan 09 21:56:42 2016] [error] Request object information:
[Sat Jan 09 21:56:42 2016] [error] URI: /XMLRPC
[Sat Jan 09 21:56:42 2016] [error] Remote Host: jboss.fr.cloud.com
[Sat Jan 09 21:56:42 2016] [error] Server Name: spacewalk.fr.cloud.com:80
[Sat Jan 09 21:56:42 2016] [error] Headers passed in:
[Sat Jan 09 21:56:42 2016] [error] \tAccept-Encoding: identity
[Sat Jan 09 21:56:42 2016] [error] \tCONTENT_LENGTH: 122
[Sat Jan 09 21:56:42 2016] [error] \tCONTENT_TYPE: text/xml
[Sat Jan 09 21:56:42 2016] [error] \tDOCUMENT_ROOT: /var/www/html
[Sat Jan 09 21:56:42 2016] [error] \tGATEWAY_INTERFACE: CGI/1.1
[Sat Jan 09 21:56:42 2016] [error] \tHTTP_ACCEPT_ENCODING: identity
[Sat Jan 09 21:56:42 2016] [error] \tHTTP_HOST: spacewalk.fr.cloud.com
[Sat Jan 09 21:56:42 2016] [error] \tHTTP_USER_AGENT:
rhn.rpclib.py/2.5.72-1.el6
[Sat Jan 09 21:56:42 2016] [error] \tHTTP_X_CLIENT_VERSION: 1
[Sat Jan 09 21:56:42 2016] [error] \tHTTP_X_INFO: RPC Processor (C) Red
Hat, Inc (version 2.5.72-1.el6)
[Sat Jan 09 21:56:42 2016] [error] \tHTTP_X_RHN_TRANSPORT_CAPABILITY:
follow-redirects=3
[Sat Jan 09 21:56:42 2016] [error] \tHTTP_X_TRANSPORT_INFO: Extended
Capabilities Transport (C) Red Hat, Inc (version 2.5.72-1.el6)
[Sat Jan 09 21:56:42 2016] [error] \tHost: spacewalk.fr.cloud.com
[Sat Jan 09 21:56:42 2016] [error] \tPATH_INFO:
[Sat Jan 09 21:56:42 2016] [error] \tQUERY_STRING:
[Sat Jan 09 21:56:42 2016] [error] \tREMOTE_ADDR: 10.0.0.100
[Sat Jan 09 21:56:42 2016] [error] \tREMOTE_PORT: 45909
[Sat Jan 09 21:56:42 2016] [error] \tREQUEST_METHOD: POST
[Sat Jan 09 21:56:42 2016] [error] \tREQUEST_URI: /XMLRPC
[Sat Jan 09 21:56:42 2016] [error] \tSCRIPT_FILENAME:
/usr/share/rhn/wsgi/xmlrpc.py
[Sat Jan 09 21:56:42 2016] [error] \tSCRIPT_NAME: /XMLRPC
[Sat Jan 09 21:56:42 2016] [error] \tSCRIPT_URI:
http://spacewalk.fr.cloud.com/XMLRPC
[Sat Jan 09 21:56:42 2016] [error] \tSCRIPT_URL: /XMLRPC
[Sat Jan 09 21:56:42 2016] [error] \tSERVER_ADDR: 10.0.0.10
[Sat Jan 09 21:56:42 2016] [error] \tSERVER_ADMIN: root at localhost
[Sat Jan 09 21:56:42 2016] [error] \tSERVER_NAME: spacewalk.fr.cloud.com
[Sat Jan 09 21:56:42 2016] [error] \tSERVER_PORT: 80
[Sat Jan 09 21:56:42 2016] [error] \tSERVER_PROTOCOL: HTTP/1.1
[Sat Jan 09 21:56:42 2016] [error] \tSERVER_SIGNATURE:
<address>Apache/2.2.15 (CentOS) Server at spacewalk.fr.cloud.com Port
80</address>
[Sat Jan 09 21:56:42 2016] [error]
[Sat Jan 09 21:56:42 2016] [error] \tSERVER_SOFTWARE: Apache/2.2.15 (CentOS)
[Sat Jan 09 21:56:42 2016] [error] \tUser-Agent: rhn.rpclib.py/2.5.72-1.el6
[Sat Jan 09 21:56:42 2016] [error] \tX-Client-Version: 1
[Sat Jan 09 21:56:42 2016] [error] \tX-Info: RPC Processor (C) Red Hat, Inc
(version 2.5.72-1.el6)
[Sat Jan 09 21:56:42 2016] [error] \tX-RHN-Transport-Capability:
follow-redirects=3
[Sat Jan 09 21:56:42 2016] [error] \tX-Transport-Info: Extended
Capabilities Transport (C) Red Hat, Inc (version 2.5.72-1.el6)
[Sat Jan 09 21:56:42 2016] [error] \tmod_ssl.is_https: <built-in method
ssl_is_https of mod_wsgi.Adapter object at 0x7f9b94ef8030>
[Sat Jan 09 21:56:42 2016] [error] \tmod_ssl.var_lookup: <built-in method
ssl_var_lookup of mod_wsgi.Adapter object at 0x7f9b94ef8030>
[Sat Jan 09 21:56:42 2016] [error] \tmod_wsgi.application_group:
spacewalk.fr.cloud.com|/xmlrpc
[Sat Jan 09 21:56:42 2016] [error] \tmod_wsgi.callable_object: application
[Sat Jan 09 21:56:42 2016] [error] \tmod_wsgi.handler_script:
[Sat Jan 09 21:56:42 2016] [error] \tmod_wsgi.input_chunked: 0
[Sat Jan 09 21:56:42 2016] [error] \tmod_wsgi.listener_host:
[Sat Jan 09 21:56:42 2016] [error] \tmod_wsgi.listener_port: 80
[Sat Jan 09 21:56:42 2016] [error] \tmod_wsgi.process_group:
[Sat Jan 09 21:56:42 2016] [error] \tmod_wsgi.request_handler: wsgi-script
[Sat Jan 09 21:56:42 2016] [error] \tmod_wsgi.script_reloading: 1
[Sat Jan 09 21:56:42 2016] [error] \tmod_wsgi.version: (3, 2)
[Sat Jan 09 21:56:42 2016] [error] \twsgi.errors: <mod_wsgi.Log object at
0x7f9b94ea6f30>
[Sat Jan 09 21:56:42 2016] [error] \twsgi.file_wrapper: <built-in method
file_wrapper of mod_wsgi.Adapter object at 0x7f9b94ef8030>
[Sat Jan 09 21:56:42 2016] [error] \twsgi.input: <mod_wsgi.Input object at
0x7f9b94ea6ef0>
[Sat Jan 09 21:56:42 2016] [error] \twsgi.multiprocess: True
[Sat Jan 09 21:56:42 2016] [error] \twsgi.multithread: False
[Sat Jan 09 21:56:42 2016] [error] \twsgi.run_once: False
[Sat Jan 09 21:56:42 2016] [error] \twsgi.url_scheme: http
[Sat Jan 09 21:56:42 2016] [error] \twsgi.version: (1, 1)
[Sat Jan 09 21:56:42 2016] [error] Extra information about this error:
[Sat Jan 09 21:56:42 2016] [error] Unhandled exception type
[Sat Jan 09 21:56:42 2016] [error]
[Sat Jan 09 21:56:42 2016] [error] Exception Handler Information
[Sat Jan 09 21:56:42 2016] [error] Traceback (most recent call last):
[Sat Jan 09 21:56:42 2016] [error]   File
"/usr/share/rhn/proxy/apacheServer.py", line 62, in __call__
[Sat Jan 09 21:56:42 2016] [error]     ret = f(req)
[Sat Jan 09 21:56:42 2016] [error]   File
"/usr/share/rhn/proxy/apacheHandler.py", line 355, in handler
[Sat Jan 09 21:56:42 2016] [error]     ret = handlerObj.handler()
[Sat Jan 09 21:56:42 2016] [error]   File
"/usr/share/rhn/proxy/broker/rhnBroker.py", line 218, in handler
[Sat Jan 09 21:56:42 2016] [error]     authToken =
self.proxyAuth.check_cached_token()
[Sat Jan 09 21:56:42 2016] [error]   File
"/usr/share/rhn/proxy/rhnProxyAuth.py", line 125, in check_cached_token
[Sat Jan 09 21:56:42 2016] [error]     oldToken = self.get_cached_token()
[Sat Jan 09 21:56:42 2016] [error]   File
"/usr/share/rhn/proxy/rhnProxyAuth.py", line 142, in get_cached_token
[Sat Jan 09 21:56:42 2016] [error]     return shelf[key]
[Sat Jan 09 21:56:42 2016] [error]   File
"/usr/share/rhn/proxy/rhnProxyAuth.py", line 398, in __getitem__
[Sat Jan 09 21:56:42 2016] [error]     val = rhnCache.get(rkey,
missing_is_null=0)
[Sat Jan 09 21:56:42 2016] [error]   File
"/usr/lib/python2.6/site-packages/spacewalk/common/rhnCache.py", line 67,
in get
[Sat Jan 09 21:56:42 2016] [error]     return cache.get(name, modified)
[Sat Jan 09 21:56:42 2016] [error]   File
"/usr/lib/python2.6/site-packages/spacewalk/common/rhnCache.py", line 356,
in get
[Sat Jan 09 21:56:42 2016] [error]     return cPickle.loads(pickled)
[Sat Jan 09 21:56:42 2016] [error] EOFError
[Sat Jan 09 21:56:42 2016] [error]
[Sat Jan 09 21:56:58 2016] [error] Exception reported from
spacewalk.fr.cloud.com
===

The versions of installed Spacewalk proxy packages are:

===
# rpm -qa | grep spacewalk
spacewalk-proxy-package-manager-2.2.10-1.el6.noarch
spacewalk-proxy-redirect-2.2.10-1.el6.noarch
spacewalk-proxy-selinux-2.0.1-1.el6.noarch
spacewalk-repo-2.2-1.el6.noarch
spacewalk-certs-tools-2.2.1-1.el6.noarch
spacewalk-proxy-broker-2.2.10-1.el6.noarch
spacewalk-base-minimal-2.2.33-1.el6.noarch
spacewalk-setup-jabberd-2.0.1-1.el6.noarch
spacewalk-proxy-monitoring-2.0.1-1.el6.noarch
spacewalk-proxy-common-2.2.10-1.el6.noarch
spacewalk-proxy-installer-2.2.1-1.el6.noarch
spacewalk-monitoring-selinux-2.2.1-1.el6.noarch
spacewalk-client-repo-2.2-1.el6.noarch
spacewalk-ssl-cert-check-2.3-1.el6.noarch
spacewalk-proxy-management-2.2.10-1.el6.noarch
spacewalk-backend-libs-2.2.43-1.el6.noarch
spacewalk-proxy-docs-2.0.1-1.el6.noarch
spacewalk-proxy-html-2.0.1-1.el6.noarch
spacewalk-backend-2.2.43-1.el6.noarch
spacewalk-base-minimal-config-2.2.33-1.el6.noarch
===

I have no ideas on what could be wrong. The server has enough disk space
available; other proxy servers in the organization are working correctly.

Thanks in advance for your help,

-- 
Best regards,
* Igor BAREEV *
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/spacewalk-list/attachments/20160109/51992c56/attachment.htm>