<html> <head> <meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1"> <style type="text/css" style="display:none;"></style> </head> <body dir="ltr"> <div id="divtagdefaultwrapper" dir="ltr" style="font-size: 12pt; color: rgb(0, 0, 0); font-family: Calibri, Arial, Helvetica, sans-serif, EmojiFont, "Apple Color Emoji", "Segoe UI Emoji", NotoColorEmoji, "Segoe UI Symbol", "Android Emoji", EmojiSymbols;"> <div id="divtagdefaultwrapper" dir="ltr" style="font-size: 12pt; color: rgb(0, 0, 0); font-family: Calibri, Arial, Helvetica, sans-serif, EmojiFont, "Apple Color Emoji", "Segoe UI Emoji", NotoColorEmoji, "Segoe UI Symbol", "Android Emoji", EmojiSymbols, EmojiFont, "Apple Color Emoji", "Segoe UI Emoji", NotoColorEmoji, "Segoe UI Symbol", "Android Emoji", EmojiSymbols;"> <p>Last week 218 systems became "Inactive Systems". All of these systems are in the DMZ and connect via a proxy server. I discovered that /var on the proxy had filled, and I'm guessing corrupted the Berkeley Database. I shutdown Jabberd, cleaned everything out, restarted jabberd, and tried to re-authenticate the inactive systems. However, I'm not able to get them to re-authenticate.</p> <p><br> </p> <p>There are multiple errors in multiple error logs.</p> <p><br> </p> <p>This is a snippet from the /var/log/httpd/error log:</p> <p><br> </p> <p></p> <div><i>[Mon Jun 05 10:41:19 2017] [error] Exception reported from <FQDN proxy server></i></div> <div><i>[Mon Jun 05 10:41:19 2017] [error] Time: Mon Jun 5 10:41:19 2017</i></div> <div><i>[Mon Jun 05 10:41:19 2017] [error] Exception type <type 'exceptions.EOFError'></i></div> <div><i>[Mon Jun 05 10:41:19 2017] [error] Exception while handling function handler</i></div> <div><i>[Mon Jun 05 10:41:19 2017] [error] Request object information:</i></div> <div><i>[Mon Jun 05 10:41:19 2017] [error] URI: /XMLRPC</i></div> <div><i>[Mon Jun 05 10:41:19 2017] [error] Remote Host: <web-server></i></div> <div><i>[Mon Jun 05 10:41:19 2017] [error] Server Name: <FQDN proxy server>:443</i></div> <div><i>[Mon Jun 05 10:41:19 2017] [error] Headers passed in:</i></div> <div><i>[Mon Jun 05 10:41:19 2017] [error] \tAccept-Encoding: identity</i></div> <div><i>[Mon Jun 05 10:41:19 2017] [error] \tCONTENT_LENGTH: 122</i></div> <div><i>[Mon Jun 05 10:41:19 2017] [error] \tCONTENT_TYPE: text/xml</i></div> <div><i>[Mon Jun 05 10:41:19 2017] [error] \tDOCUMENT_ROOT: /var/www/html</i></div> <div><i>[Mon Jun 05 10:41:19 2017] [error] \tGATEWAY_INTERFACE: CGI/1.1</i></div> <div><i>[Mon Jun 05 10:41:19 2017] [error] \tHTTPS: 1</i></div> <div><i>[Mon Jun 05 10:41:19 2017] [error] \tHTTP_ACCEPT_ENCODING: identity</i></div> <div><i>[Mon Jun 05 10:41:19 2017] [error] \tHTTP_HOST: <FQDN proxy server></i></div> <div><i>[Mon Jun 05 10:41:19 2017] [error] \tHTTP_USER_AGENT: rhn.rpclib.py/2.5.75-1.el6</i></div> <div><i>[Mon Jun 05 10:41:19 2017] [error] \tHTTP_X_CLIENT_VERSION: 1</i></div> <div><i>[Mon Jun 05 10:41:19 2017] [error] \tHTTP_X_INFO: RPC Processor (C) Red Hat, Inc (version 2.5.75-1.el6)</i></div> <div><i>[Mon Jun 05 10:41:19 2017] [error] \tHTTP_X_RHN_TRANSPORT_CAPABILITY: follow-redirects=3</i></div> <div><i>[Mon Jun 05 10:41:19 2017] [error] \tHTTP_X_TRANSPORT_INFO: Extended Capabilities Transport (C) Red Hat, Inc (version 2.5.75-1.el6)</i></div> <div><i>[Mon Jun 05 10:41:19 2017] [error] \tHost: <<i style="font-family: Calibri, Arial, Helvetica, sans-serif, EmojiFont, "Apple Color Emoji", "Segoe UI Emoji", NotoColorEmoji, "Segoe UI Symbol", "Android Emoji", EmojiSymbols, EmojiFont, "Apple Color Emoji", "Segoe UI Emoji", NotoColorEmoji, "Segoe UI Symbol", "Android Emoji", EmojiSymbols; font-size: 16px;">FQDN </i>proxy server></i></div> <div><i>[Mon Jun 05 10:41:19 2017] [error] \tPATH_INFO: </i></div> <div><i>[Mon Jun 05 10:41:19 2017] [error] \tQUERY_STRING: </i></div> <div><i>[Mon Jun 05 10:41:19 2017] [error] \tREMOTE_ADDR: 172.19.216.14</i></div> <div><i>[Mon Jun 05 10:41:19 2017] [error] \tREMOTE_PORT: 43218</i></div> <div><i>[Mon Jun 05 10:41:19 2017] [error] \tREQUEST_METHOD: POST</i></div> <div><i>[Mon Jun 05 10:41:19 2017] [error] \tREQUEST_URI: /XMLRPC</i></div> <div><i>[Mon Jun 05 10:41:19 2017] [error] \tSCRIPT_FILENAME: /usr/share/rhn/wsgi/xmlrpc.py</i></div> <div><i>[Mon Jun 05 10:41:19 2017] [error] \tSCRIPT_NAME: /XMLRPC</i></div> <div><i>[Mon Jun 05 10:41:19 2017] [error] \tSCRIPT_URI: https://<<i style="font-family: Calibri, Arial, Helvetica, sans-serif, EmojiFont, "Apple Color Emoji", "Segoe UI Emoji", NotoColorEmoji, "Segoe UI Symbol", "Android Emoji", EmojiSymbols, EmojiFont, "Apple Color Emoji", "Segoe UI Emoji", NotoColorEmoji, "Segoe UI Symbol", "Android Emoji", EmojiSymbols; font-size: 16px;">FQDN </i>proxy server>/XMLRPC</i></div> <div><i>[Mon Jun 05 10:41:19 2017] [error] \tSCRIPT_URL: /XMLRPC</i></div> <div><i>[Mon Jun 05 10:41:19 2017] [error] \tSERVER_ADDR: 172.17.210.59</i></div> <div><i>[Mon Jun 05 10:41:19 2017] [error] \tSERVER_ADMIN: root@localhost</i></div> <div><i>[Mon Jun 05 10:41:19 2017] [error] \tSERVER_NAME: <<i style="font-family: Calibri, Arial, Helvetica, sans-serif, EmojiFont, "Apple Color Emoji", "Segoe UI Emoji", NotoColorEmoji, "Segoe UI Symbol", "Android Emoji", EmojiSymbols, EmojiFont, "Apple Color Emoji", "Segoe UI Emoji", NotoColorEmoji, "Segoe UI Symbol", "Android Emoji", EmojiSymbols; font-size: 16px;">FQDN </i>proxy server></i></div> <div><i>[Mon Jun 05 10:41:19 2017] [error] \tSERVER_PORT: 443</i></div> <div><i>[Mon Jun 05 10:41:19 2017] [error] \tSERVER_PROTOCOL: HTTP/1.1</i></div> <div><i>[Mon Jun 05 10:41:19 2017] [error] \tSERVER_SIGNATURE: <address>Apache/2.2.15 (Red Hat) Server at <<i style="font-family: Calibri, Arial, Helvetica, sans-serif, EmojiFont, "Apple Color Emoji", "Segoe UI Emoji", NotoColorEmoji, "Segoe UI Symbol", "Android Emoji", EmojiSymbols, EmojiFont, "Apple Color Emoji", "Segoe UI Emoji", NotoColorEmoji, "Segoe UI Symbol", "Android Emoji", EmojiSymbols; font-size: 16px;">FQDN </i>proxy server> Port 443</address></i></div> <div><i>[Mon Jun 05 10:41:19 2017] [error] </i></div> <div><i>[Mon Jun 05 10:41:19 2017] [error] \tSERVER_SOFTWARE: Apache/2.2.15 (Red Hat)</i></div> <div><i>[Mon Jun 05 10:41:19 2017] [error] \tUser-Agent: rhn.rpclib.py/2.5.75-1.el6</i></div> <div><i>[Mon Jun 05 10:41:19 2017] [error] \tX-Client-Version: 1</i></div> <div><i>[Mon Jun 05 10:41:19 2017] [error] \tX-Info: RPC Processor (C) Red Hat, Inc (version 2.5.75-1.el6)</i></div> <div><i>[Mon Jun 05 10:41:19 2017] [error] \tX-RHN-Transport-Capability: follow-redirects=3</i></div> <div><i>[Mon Jun 05 10:41:19 2017] [error] \tX-Transport-Info: Extended Capabilities Transport (C) Red Hat, Inc (version 2.5.75-1.el6)</i></div> <div><i>[Mon Jun 05 10:41:19 2017] [error] \tmod_ssl.is_https: <built-in method ssl_is_https of mod_wsgi.Adapter object at 0x7fd673532a08></i></div> <div><i>[Mon Jun 05 10:41:19 2017] [error] \tmod_ssl.var_lookup: <built-in method ssl_var_lookup of mod_wsgi.Adapter object at 0x7fd673532a08></i></div> <div><i>[Mon Jun 05 10:41:19 2017] [error] \tmod_wsgi.application_group: corp-spwlkprxy-prod01.dtndmz.com|/xmlrpc</i></div> <div><i>[Mon Jun 05 10:41:19 2017] [error] \tmod_wsgi.callable_object: application</i></div> <div><i>[Mon Jun 05 10:41:19 2017] [error] \tmod_wsgi.handler_script: </i></div> <div><i>[Mon Jun 05 10:41:19 2017] [error] \tmod_wsgi.input_chunked: 0</i></div> <div><i>[Mon Jun 05 10:41:19 2017] [error] \tmod_wsgi.listener_host: </i></div> <div><i>[Mon Jun 05 10:41:19 2017] [error] \tmod_wsgi.listener_port: 443</i></div> <div><i>[Mon Jun 05 10:41:19 2017] [error] \tmod_wsgi.process_group: </i></div> <div><i>[Mon Jun 05 10:41:19 2017] [error] \tmod_wsgi.request_handler: wsgi-script</i></div> <div><i>[Mon Jun 05 10:41:19 2017] [error] \tmod_wsgi.script_reloading: 1</i></div> <div><i>[Mon Jun 05 10:41:19 2017] [error] \tmod_wsgi.version: (3, 2)</i></div> <div><i>[Mon Jun 05 10:41:19 2017] [error] \twsgi.errors: <mod_wsgi.Log object at 0x7fd673415ab0></i></div> <div><i>[Mon Jun 05 10:41:19 2017] [error] \twsgi.file_wrapper: <built-in method file_wrapper of mod_wsgi.Adapter object at 0x7fd673532a08></i></div> <div><i>[Mon Jun 05 10:41:19 2017] [error] \twsgi.input: <cStringIO.StringO object at 0x7fd6733ffab0></i></div> <div><i>[Mon Jun 05 10:41:19 2017] [error] \twsgi.multiprocess: True</i></div> <div><i>[Mon Jun 05 10:41:19 2017] [error] \twsgi.multithread: False</i></div> <div><i>[Mon Jun 05 10:41:19 2017] [error] \twsgi.run_once: False</i></div> <div><i>[Mon Jun 05 10:41:19 2017] [error] \twsgi.url_scheme: https</i></div> <div><i>[Mon Jun 05 10:41:19 2017] [error] \twsgi.version: (1, 1)</i></div> <div><i>[Mon Jun 05 10:41:19 2017] [error] Extra information about this error:</i></div> <div><i>[Mon Jun 05 10:41:19 2017] [error] Unhandled exception type</i></div> <div><i>[Mon Jun 05 10:41:19 2017] [error] </i></div> <div><i>[Mon Jun 05 10:41:19 2017] [error] Exception Handler Information</i></div> <div><i>[Mon Jun 05 10:41:19 2017] [error] Traceback (most recent call last):</i></div> <div><i>[Mon Jun 05 10:41:19 2017] [error] File "/usr/share/rhn/proxy/apacheServer.py", line 63, in __call__</i></div> <div><i>[Mon Jun 05 10:41:19 2017] [error] ret = f(req)</i></div> <div><i>[Mon Jun 05 10:41:19 2017] [error] File "/usr/share/rhn/proxy/apacheHandler.py", line 357, in handler</i></div> <div><i>[Mon Jun 05 10:41:19 2017] [error] ret = handlerObj.handler()</i></div> <div><i>[Mon Jun 05 10:41:19 2017] [error] File "/usr/share/rhn/proxy/broker/rhnBroker.py", line 220, in handler</i></div> <div><i>[Mon Jun 05 10:41:19 2017] [error] authToken = self.proxyAuth.check_cached_token()</i></div> <div><i>[Mon Jun 05 10:41:19 2017] [error] File "/usr/share/rhn/proxy/rhnProxyAuth.py", line 130, in check_cached_token</i></div> <div><i>[Mon Jun 05 10:41:19 2017] [error] oldToken = self.get_cached_token()</i></div> <div><i>[Mon Jun 05 10:41:19 2017] [error] File "/usr/share/rhn/proxy/rhnProxyAuth.py", line 147, in get_cached_token</i></div> <div><i>[Mon Jun 05 10:41:19 2017] [error] return shelf[key]</i></div> <div><i>[Mon Jun 05 10:41:19 2017] [error] File "/usr/share/rhn/proxy/rhnProxyAuth.py", line 450, in __getitem__</i></div> <div><i>[Mon Jun 05 10:41:19 2017] [error] val = rhnCache.get(rkey, missing_is_null=0)</i></div> <div><i>[Mon Jun 05 10:41:19 2017] [error] File "/usr/lib/python2.6/site-packages/spacewalk/common/rhnCache.py", line 70, in get</i></div> <div><i>[Mon Jun 05 10:41:19 2017] [error] return cache.get(name, modified)</i></div> <div><i>[Mon Jun 05 10:41:19 2017] [error] File "/usr/lib/python2.6/site-packages/spacewalk/common/rhnCache.py", line 366, in get</i></div> <div><i>[Mon Jun 05 10:41:19 2017] [error] return cPickle.loads(pickled)</i></div> <div><i>[Mon Jun 05 10:41:19 2017] [error] EOFError</i></div> <div><br> </div> This is from one of the clients that can't connect:</div> <div id="divtagdefaultwrapper" dir="ltr" style="font-size: 12pt; color: rgb(0, 0, 0); font-family: Calibri, Arial, Helvetica, sans-serif, EmojiFont, "Apple Color Emoji", "Segoe UI Emoji", NotoColorEmoji, "Segoe UI Symbol", "Android Emoji", EmojiSymbols, EmojiFont, "Apple Color Emoji", "Segoe UI Emoji", NotoColorEmoji, "Segoe UI Symbol", "Android Emoji", EmojiSymbols;"> <br> <p></p> <p><i></i></p> <div><i></i></div> <div><i><span style="color:rgb(0,0,0)">2017-06-05 12:46:48 rhn_log.log_error: Error caught:</span></i></div> <div><i><span style="color:rgb(0,0,0)">2017-06-05 12:46:48 rhn_log.log_error: Traceback (most recent call last):</span></i></div> <div><i><span style="color:rgb(0,0,0)"> File "/usr/share/rhn/osad/jabber_lib.py", line 127, in main</span></i></div> <div><i><span style="color:rgb(0,0,0)"> self.fix_connection(c)</span></i></div> <div><i><span style="color:rgb(0,0,0)"> File "/usr/share/rhn/osad/osad.py", line 202, in fix_connection</span></i></div> <div><i><span style="color:rgb(0,0,0)"> server.registration.welcome_message()</span></i></div> <div><i><span style="color:rgb(0,0,0)"> File "/usr/lib64/python2.6/site-packages/rhn/rpclib.py", line 662, in __call__</span></i></div> <div><i><span style="color:rgb(0,0,0)"> return self._send(self._name, args)</span></i></div> <div><i><span style="color:rgb(0,0,0)"> File "/usr/lib64/python2.6/site-packages/rhn/rpclib.py", line 394, in _request</span></i></div> <div><i><span style="color:rgb(0,0,0)"> self._handler, request, verbose=self._verbose)</span></i></div> <div><i><span style="color:rgb(0,0,0)"> File "/usr/lib64/python2.6/site-packages/rhn/transports.py", line 177, in request</span></i></div> <div><i><span style="color:rgb(0,0,0)"> headers, fd = req.send_http(host, handler)</span></i></div> <div><i><span style="color:rgb(0,0,0)"> File "/usr/lib64/python2.6/site-packages/rhn/transports.py", line 740, in send_http</span></i></div> <div><i><span style="color:rgb(0,0,0)"> response.status, response.reason, response.msg)</span></i></div> <div><i><span style="color:rgb(0,0,0)">ProtocolError: <ProtocolError for <FQDN Proxy Server> /XMLRPC: 500 Internal Server Error></span></i></div> <i></i> <p><br> </p> <p>My inital thoughts turn to the certifiacation, but It that shouldn't be the case, because this has been working fine for over a year. As a matter of fact, I can see that the client estabilishes a connection on with the proxy server on port 5222. </p> <p><br> </p> <p>Anyone have any thoughts on what could be causing this?</p> <p><br> </p> <p>Thank you.</p> <p><br> </p> <p>Daryl </p> <p><br> </p> <p><br> </p> </div> </div> </body> </html>