[Spacewalk-list] SW 1.9 abrt crash data upload fails

Tue Apr 9 10:56:05 UTC 2013

> Hey
> 
> When I try to sync a client's abrt data (spacewalk-abrt --sync), or when
> a client tries to upload new data, it always errors out. The entry gets
> created in spacewalk and I can see it in the webinterface, but the crash
> dump/data itself isn't getting sent.
> 
> _Client side error (/var/log/up2date):_
> [Tue Apr  9 10:06:03 2013] up2date A protocol error occurred: Internal
> Server Error , attempt #1,
> [Tue Apr  9 10:06:03 2013] up2date
> Traceback (most recent call last):
>    File "/usr/bin/spacewalk-abrt", line 86, in <module>
>      cli.run()
>    File "/usr/share/rhn/up2date_client/rhncli.py", line 96, in run
>      sys.exit(self.main() or 0)
>    File "/usr/bin/spacewalk-abrt", line 79, in main
>      return (spacewalk_abrt.abrt.sync())
>    File "/usr/share/rhn/spacewalk_abrt/abrt.py", line 147, in sync
>      report(problem_dir)
>    File "/usr/share/rhn/spacewalk_abrt/abrt.py", line 107, in report
>      server.abrt.upload_crash_file(systemid, basename, crash_file_data)
>    File "/usr/share/rhn/up2date_client/rhnserver.py", line 63, in __call__
>      return rpcServer.doCall(method, *args, **kwargs)
>    File "/usr/share/rhn/up2date_client/rpcServer.py", line 203, in doCall
>      ret = method(*args, **kwargs)
>    File "/usr/lib64/python2.6/xmlrpclib.py", line 1199, in __call__
>      return self.__send(self.__name, args)
>    File "/usr/share/rhn/up2date_client/rpcServer.py", line 38, in _request1
>      ret = self._request(methodname, params)
>    File "/usr/lib/python2.6/site-packages/rhn/rpclib.py", line 381, in
> _request
>      self._handler, request, verbose=self._verbose)
>    File "/usr/lib/python2.6/site-packages/rhn/transports.py", line 167,
> in request
>      headers, fd = req.send_http(host, handler)
>    File "/usr/lib/python2.6/site-packages/rhn/transports.py", line 707,
> in send_http
>      response.status, response.reason, response.msg)
> <class 'up2date_client.up2dateErrors.CommunicationError'>: Error
> communicating with server. The message was:
> Internal Server Error
> 
> _Server side error (/var/log/httpd/ssl_error_log__):_
> [Tue Apr 09 10:18:08 2013] [error] [client ip-address-here] mod_wsgi
> (pid=10434): Exception occurred processing WSGI script
> '/usr/share/rhn/wsgi/xmlrpc.py'.
> [Tue Apr 09 10:18:08 2013] [error] [client ip-address-here] Traceback
> (most recent call last):
> [Tue Apr 09 10:18:08 2013] [error] [client ip-address-here]   File
> "/usr/share/rhn/wsgi/xmlrpc.py", line 22, in application
> [Tue Apr 09 10:18:08 2013] [error] [client ip-address-here] return
> wsgiHandler.handle(environ, start_response, "xmlrpc", "server.xmlrpc")
> [Tue Apr 09 10:18:08 2013] [error] [client ip-address-here]   File
> "/usr/share/rhn/wsgi/wsgiHandler.py", line 45, in handle
> [Tue Apr 09 10:18:08 2013] [error] [client ip-address-here]     ret =
> appServ(req)
> [Tue Apr 09 10:18:08 2013] [error] [client ip-address-here]   File
> "/usr/lib/python2.6/site-packages/spacewalk/server/apacheHandler.py",
> line 196, in handler
> [Tue Apr 09 10:18:08 2013] [error] [client ip-address-here]     ret =
> self._req_processor.process()
> [Tue Apr 09 10:18:08 2013] [error] [client ip-address-here]   File
> "/usr/lib/python2.6/site-packages/spacewalk/server/apacheRequest.py",
> line 476, in process
> [Tue Apr 09 10:18:08 2013] [error] [client ip-address-here] return
> self.call_function(method, params)
> [Tue Apr 09 10:18:08 2013] [error] [client ip-address-here]   File
> "/usr/lib/python2.6/site-packages/spacewalk/server/apacheRequest.py",
> line 174, in call_function
> [Tue Apr 09 10:18:08 2013] [error] [client ip-address-here]
> Traceback(method, self.req, severity="unhandled")
> [Tue Apr 09 10:18:08 2013] [error] [client ip-address-here]   File
> "/usr/lib/python2.6/site-packages/spacewalk/common/rhnTB.py", line 180,
> in Traceback
> [Tue Apr 09 10:18:08 2013] [error] [client ip-address-here]
> rhnMail.send(headers, outstring)
> [Tue Apr 09 10:18:08 2013] [error] [client ip-address-here]   File
> "/usr/lib/python2.6/site-packages/spacewalk/common/rhnMail.py", line 54,
> in send
> [Tue Apr 09 10:18:08 2013] [error] [client ip-address-here] server =
> smtplib.SMTP('localhost')
> [Tue Apr 09 10:18:08 2013] [error] [client ip-address-here]   File
> "/usr/lib64/python2.6/smtplib.py", line 239, in __init__
> [Tue Apr 09 10:18:08 2013] [error] [client ip-address-here] (code, msg)
> = self.connect(host, port)
> [Tue Apr 09 10:18:08 2013] [error] [client ip-address-here]   File
> "/usr/lib64/python2.6/smtplib.py", line 295, in connect
> [Tue Apr 09 10:18:08 2013] [error] [client ip-address-here] self.sock =
> self._get_socket(host, port, self.timeout)
> [Tue Apr 09 10:18:08 2013] [error] [client ip-address-here]   File
> "/usr/lib64/python2.6/smtplib.py", line 273, in _get_socket
> [Tue Apr 09 10:18:08 2013] [error] [client ip-address-here] return
> socket.create_connection((port, host), timeout)
> [Tue Apr 09 10:18:08 2013] [error] [client ip-address-here]   File
> "/usr/lib64/python2.6/socket.py", line 567, in create_connection
> [Tue Apr 09 10:18:08 2013] [error] [client ip-address-here] raise error,
> msg [Tue Apr 09 10:18:08 2013] [error] [client ip-address-here] error:
> [Errno 101] Network is unreachable
> 
> Now given the error messages I'm inclined to think it's a
> network/firewall issue, though I'm not sure about the way the data gets
> sent. Spacewalk server is 1.9 on CentOS 6.4 fully updated, in DMZ and is
> reachable on ports 80/443 from the outside. Clients are RHEL 6.3 with
> spacewalk client version 1.9 packages. All other spacewalk communication
> we use (repo's, config management, scheduling scripts/updates, ...)
> works like it's supposed to.
> 
> When I check the connections being made from the client I only see
> connections on 443 so I'm not sure what else I need to open up or change
> to get it to work. Anyone have any pointers?

Does

     /var/log/rhn/rhn_server_xmlrpc.log

contain anything related to this error?

-MZ