[Spacewalk-list] new version of rhnlib problems with spacewalk proxy

Thu Dec 29 16:24:19 UTC 2016

Hello,

just an FYI, not sure if it's because I still run spacewalk 2.0 and 
spacewalk-proxy with 2.0 but recently I installed 2 new spacewalk 
proxies and I noticed the first worked and the second did not ...
Platform centos 6.8 x64 with latest updates but still running spacewalk 2.0
After I compared both I figured out this was the difference ( left 
working, right not working):
rhnlib-2.5.65-1.el6.noarch      |rhnlib-2.5.72-1.el6.noarch

when I downgraded this on the newer host to this version all started to 
work again:

logs of the errors I saw with this newer rhnlib taken from spacewalkproxy:

==> /var/log/httpd/error_log <==
[Thu Dec 29 15:12:21 2016] [error] Exception reported from 
SPACEWALKPROXYHOSTNAME
[Thu Dec 29 15:12:21 2016] [error] Time: Thu Dec 29 15:12:21 2016
[Thu Dec 29 15:12:21 2016] [error] Exception type <type 
'exceptions.KeyError'>
[Thu Dec 29 15:12:21 2016] [error] Exception while handling function handler
[Thu Dec 29 15:12:21 2016] [error] Request object information:
[Thu Dec 29 15:12:21 2016] [error] URI: /XMLRPC
[Thu Dec 29 15:12:21 2016] [error] Remote Host: CLIENTHOSTNAME
[Thu Dec 29 15:12:21 2016] [error] Server Name: SPACEWALKPROXYHOSTNAME:443
[Thu Dec 29 15:12:21 2016] [error] Headers passed in:
[Thu Dec 29 15:12:21 2016] [error] \tAccept-Encoding: identity
[Thu Dec 29 15:12:21 2016] [error] \tCONTENT_LENGTH: 2150
[Thu Dec 29 15:12:21 2016] [error] \tCONTENT_TYPE: text/xml
[Thu Dec 29 15:12:21 2016] [error] \tDOCUMENT_ROOT: /var/www/html
[Thu Dec 29 15:12:21 2016] [error] \tGATEWAY_INTERFACE: CGI/1.1
[Thu Dec 29 15:12:21 2016] [error] \tHTTPS: 1
[Thu Dec 29 15:12:21 2016] [error] \tHTTP_ACCEPT_ENCODING: identity
[Thu Dec 29 15:12:21 2016] [error] \tHTTP_HOST: SPACEWALKPROXYHOSTNAME
[Thu Dec 29 15:12:21 2016] [error] \tHTTP_USER_AGENT: 
rhn.rpclib.py/2.5.72-1.el6
[Thu Dec 29 15:12:21 2016] [error] \tHTTP_X_CLIENT_VERSION: 1
[Thu Dec 29 15:12:21 2016] [error] \tHTTP_X_INFO: RPC Processor (C) Red 
Hat, Inc (version 2.5.72-1.el6)
[Thu Dec 29 15:12:21 2016] [error] \tHTTP_X_RHN_CLIENT_CAPABILITY: 
packages.verify(1)=1,packages.extended_profile(2)=1,configfiles.base64_enc(1)=1,reboot.reboot(1)=1,configfiles.deploy(1)=1,caneatCheese(1)=1,packages.rollBack(1)=1,configfiles.mtime_upload(1)=1,configfiles.diff(1)=1,packages.update(2)=2,script.run(1)=1,configfiles.upload(1)=1,packages.runTransaction(1)=1,packages.verify(1)=1,packages.extended_profile(2)=1,configfiles.base64_enc(1)=1,reboot.reboot(1)=1,configfiles.deploy(1)=1,caneatCheese(1)=1,packages.rollBack(1)=1,configfiles.mtime_upload(1)=1,configfiles.diff(1)=1,packages.update(2)=2,script.run(1)=1,configfiles.upload(1)=1,packages.runTransaction(1)=1
[Thu Dec 29 15:12:21 2016] [error] \tHTTP_X_RHN_TRANSPORT_CAPABILITY: 
follow-redirects=3
[Thu Dec 29 15:12:21 2016] [error] \tHTTP_X_TRANSPORT_INFO: Extended 
Capabilities Transport (C) Red Hat, Inc (version 2.5.72-1.el6)
[Thu Dec 29 15:12:21 2016] [error] \tHTTP_X_UP2DATE_VERSION: 2.2.7-1.el6
[Thu Dec 29 15:12:21 2016] [error] \tHost: SPACEWALKPROXYHOSTNAME
[Thu Dec 29 15:12:21 2016] [error] \tPATH_INFO:
[Thu Dec 29 15:12:21 2016] [error] \tQUERY_STRING:
[Thu Dec 29 15:12:21 2016] [error] \tREMOTE_ADDR: IPOFCLIENT
[Thu Dec 29 15:12:21 2016] [error] \tREMOTE_PORT: 36878
[Thu Dec 29 15:12:21 2016] [error] \tREQUEST_METHOD: POST
[Thu Dec 29 15:12:21 2016] [error] \tREQUEST_URI: /XMLRPC
[Thu Dec 29 15:12:21 2016] [error] \tSCRIPT_FILENAME: 
/usr/share/rhn/wsgi/xmlrpc.py
[Thu Dec 29 15:12:21 2016] [error] \tSCRIPT_NAME: /XMLRPC
[Thu Dec 29 15:12:21 2016] [error] \tSCRIPT_URI: 
https://SPACEWALKPROXYHOSTNAME/XMLRPC
[Thu Dec 29 15:12:21 2016] [error] \tSCRIPT_URL: /XMLRPC
[Thu Dec 29 15:12:21 2016] [error] \tSERVER_ADDR: IPOFSPACEWALKPROXY
[Thu Dec 29 15:12:21 2016] [error] \tSERVER_ADMIN: root at localhost
[Thu Dec 29 15:12:21 2016] [error] \tSERVER_NAME: SPACEWALKPROXYHOSTNAME
[Thu Dec 29 15:12:21 2016] [error] \tSERVER_PORT: 443
[Thu Dec 29 15:12:21 2016] [error] \tSERVER_PROTOCOL: HTTP/1.1
[Thu Dec 29 15:12:21 2016] [error] \tSERVER_SIGNATURE: 
<address>Apache/2.2.15 (CentOS) Server at SPACEWALKPROXYHOSTNAME Port 
443</address>
[Thu Dec 29 15:12:21 2016] [error]
[Thu Dec 29 15:12:21 2016] [error] \tSERVER_SOFTWARE: Apache/2.2.15 (CentOS)
[Thu Dec 29 15:12:21 2016] [error] \tUser-Agent: rhn.rpclib.py/2.5.72-1.el6
[Thu Dec 29 15:12:21 2016] [error] \tX-Client-Version: 1
[Thu Dec 29 15:12:21 2016] [error] \tX-Info: RPC Processor (C) Red Hat, 
Inc (version 2.5.72-1.el6)
[Thu Dec 29 15:12:21 2016] [error] \tX-RHN-Client-Capability: 
packages.verify(1)=1,packages.extended_profile(2)=1,configfiles.base64_enc(1)=1,reboot.reboot(1)=1,configfiles.deploy(1)=1,caneatCheese(1)=1,packages.rollBack(1)=1,configfiles.mtime_upload(1)=1,configfiles.diff(1)=1,packages.update(2)=2,script.run(1)=1,configfiles.upload(1)=1,packages.runTransaction(1)=1,packages.verify(1)=1,packages.extended_profile(2)=1,configfiles.base64_enc(1)=1,reboot.reboot(1)=1,configfiles.deploy(1)=1,caneatCheese(1)=1,packages.rollBack(1)=1,configfiles.mtime_upload(1)=1,configfiles.diff(1)=1,packages.update(2)=2,script.run(1)=1,configfiles.upload(1)=1,packages.runTransaction(1)=1
[Thu Dec 29 15:12:21 2016] [error] \tX-RHN-Transport-Capability: 
follow-redirects=3
[Thu Dec 29 15:12:21 2016] [error] \tX-Transport-Info: Extended 
Capabilities Transport (C) Red Hat, Inc (version 2.5.72-1.el6)
[Thu Dec 29 15:12:21 2016] [error] \tX-Up2Date-Version: 2.2.7-1.el6
[Thu Dec 29 15:12:21 2016] [error] \tmod_ssl.is_https: <built-in method 
ssl_is_https of mod_wsgi.Adapter object at 0x7f9e46b25030>
[Thu Dec 29 15:12:21 2016] [error] \tmod_ssl.var_lookup: <built-in 
method ssl_var_lookup of mod_wsgi.Adapter object at 0x7f9e46b25030>
[Thu Dec 29 15:12:21 2016] [error] \tmod_wsgi.application_group: 
SPACEWALKPROXYHOSTNAME|/xmlrpc
[Thu Dec 29 15:12:21 2016] [error] \tmod_wsgi.callable_object: application
[Thu Dec 29 15:12:21 2016] [error] \tmod_wsgi.handler_script:
[Thu Dec 29 15:12:21 2016] [error] \tmod_wsgi.input_chunked: 0
[Thu Dec 29 15:12:21 2016] [error] \tmod_wsgi.listener_host:
[Thu Dec 29 15:12:21 2016] [error] \tmod_wsgi.listener_port: 443
[Thu Dec 29 15:12:21 2016] [error] \tmod_wsgi.process_group:
[Thu Dec 29 15:12:21 2016] [error] \tmod_wsgi.request_handler: wsgi-script
[Thu Dec 29 15:12:21 2016] [error] \tmod_wsgi.script_reloading: 1
[Thu Dec 29 15:12:21 2016] [error] \tmod_wsgi.version: (3, 2)
[Thu Dec 29 15:12:21 2016] [error] \twsgi.errors: <mod_wsgi.Log object 
at 0x7f9e46ad3f30>
[Thu Dec 29 15:12:21 2016] [error] \twsgi.file_wrapper: <built-in method 
file_wrapper of mod_wsgi.Adapter object at 0x7f9e46b25030>
[Thu Dec 29 15:12:21 2016] [error] \twsgi.input: <mod_wsgi.Input object 
at 0x7f9e46ad3ef0>
[Thu Dec 29 15:12:21 2016] [error] \twsgi.multiprocess: True
[Thu Dec 29 15:12:21 2016] [error] \twsgi.multithread: False
[Thu Dec 29 15:12:21 2016] [error] \twsgi.run_once: False
[Thu Dec 29 15:12:21 2016] [error] \twsgi.url_scheme: https
[Thu Dec 29 15:12:21 2016] [error] \twsgi.version: (1, 1)
[Thu Dec 29 15:12:21 2016] [error] Extra information about this error:
[Thu Dec 29 15:12:21 2016] [error] Unhandled exception type
[Thu Dec 29 15:12:21 2016] [error]
[Thu Dec 29 15:12:21 2016] [error] Exception Handler Information
[Thu Dec 29 15:12:21 2016] [error] Traceback (most recent call last):
[Thu Dec 29 15:12:21 2016] [error]   File 
"/usr/share/rhn/proxy/apacheServer.py", line 62, in __call__
[Thu Dec 29 15:12:21 2016] [error]     ret = f(req)
[Thu Dec 29 15:12:21 2016] [error]   File 
"/usr/share/rhn/proxy/apacheHandler.py", line 355, in handler
[Thu Dec 29 15:12:21 2016] [error]     ret = handlerObj.handler()
[Thu Dec 29 15:12:21 2016] [error]   File 
"/usr/share/rhn/proxy/broker/rhnBroker.py", line 157, in handler
[Thu Dec 29 15:12:21 2016] [error]     ip_path = _oto['X-RHN-IP-Path']
[Thu Dec 29 15:12:21 2016] [error]   File 
"/usr/lib/python2.6/site-packages/rhn/UserDictCase.py", line 44, in 
__getitem__
[Thu Dec 29 15:12:21 2016] [error]     return self.data[key]
[Thu Dec 29 15:12:21 2016] [error] KeyError: 'x-rhn-ip-path'
[Thu Dec 29 15:12:21 2016] [error]

==> /var/log/httpd/ssl_error_log <==
[Thu Dec 29 15:12:21 2016] [error] [client IPOFCLIENT] mod_wsgi 
(pid=1560): Exception occurred processing WSGI script 
'/usr/share/rhn/wsgi/xmlrpc.py'.
[Thu Dec 29 15:12:21 2016] [error] [client IPOFCLIENT] Traceback (most 
recent call last):
[Thu Dec 29 15:12:21 2016] [error] [client IPOFCLIENT]   File 
"/usr/share/rhn/wsgi/xmlrpc.py", line 22, in application
[Thu Dec 29 15:12:21 2016] [error] [client IPOFCLIENT] 
"proxy.apacheServer")
[Thu Dec 29 15:12:21 2016] [error] [client IPOFCLIENT]   File 
"/usr/share/rhn/wsgi/wsgiHandler.py", line 45, in handle
[Thu Dec 29 15:12:21 2016] [error] [client IPOFCLIENT]     ret = 
appServ(req)
[Thu Dec 29 15:12:21 2016] [error] [client IPOFCLIENT]   File 
"/usr/share/rhn/proxy/apacheServer.py", line 68, in __call__
[Thu Dec 29 15:12:21 2016] [error] [client IPOFCLIENT] 
severity="unhandled")
[Thu Dec 29 15:12:21 2016] [error] [client IPOFCLIENT]   File 
"/usr/lib/python2.6/site-packages/spacewalk/common/rhnTB.py", line 180, 
in Traceback
[Thu Dec 29 15:12:21 2016] [error] [client IPOFCLIENT] 
rhnMail.send(headers, outstring)
[Thu Dec 29 15:12:21 2016] [error] [client IPOFCLIENT]   File 
"/usr/lib/python2.6/site-packages/spacewalk/common/rhnMail.py", line 55, 
in send
[Thu Dec 29 15:12:21 2016] [error] [client IPOFCLIENT]     server = 
smtplib.SMTP('localhost')
[Thu Dec 29 15:12:21 2016] [error] [client IPOFCLIENT]   File 
"/usr/lib64/python2.6/smtplib.py", line 244, in __init__
[Thu Dec 29 15:12:21 2016] [error] [client IPOFCLIENT]     (code, msg) = 
self.connect(host, port)
[Thu Dec 29 15:12:21 2016] [error] [client IPOFCLIENT]   File 
"/usr/lib64/python2.6/smtplib.py", line 300, in connect
[Thu Dec 29 15:12:21 2016] [error] [client IPOFCLIENT]     self.sock = 
self._get_socket(host, port, self.timeout)
[Thu Dec 29 15:12:21 2016] [error] [client IPOFCLIENT]   File 
"/usr/lib64/python2.6/smtplib.py", line 278, in _get_socket
[Thu Dec 29 15:12:21 2016] [error] [client IPOFCLIENT]     return 
socket.create_connection((port, host), timeout)
[Thu Dec 29 15:12:21 2016] [error] [client IPOFCLIENT]   File 
"/usr/lib64/python2.6/socket.py", line 567, in create_connection
[Thu Dec 29 15:12:21 2016] [error] [client IPOFCLIENT]     raise error, msg
[Thu Dec 29 15:12:21 2016] [error] [client IPOFCLIENT] error: [Errno 
113] No route to host

==> /var/log/httpd/ssl_access_log <==
IPOFCLIENT - - [29/Dec/2016:15:12:21 +0000] "POST /XMLRPC HTTP/1.1" 500 635

==> /var/log/httpd/ssl_request_log <==
[29/Dec/2016:15:12:21 +0000] IPOFCLIENT TLSv1.2 
ECDHE-RSA-AES256-GCM-SHA384 "POST /XMLRPC HTTP/1.1" 635