[Spacewalk-list] Spacewalk Proxies failing to serve large packages

Mon May 11 19:48:11 UTC 2015

I was able to get some more verbose output, which is showing a socket
timeout. (I have tried with and without using SSL to download the package).

[Mon May 11 15:35:04 2015] [error] Exception reported from
spw-proxy.ourcompany.com
[Mon May 11 15:35:04 2015] [error] Time: Mon May 11 15:35:04 2015
[Mon May 11 15:35:04 2015] [error] Exception type <class 'socket.timeout'>
[Mon May 11 15:35:04 2015] [error] Exception while handling function
SharedHandler._serverCommo
[Mon May 11 15:35:04 2015] [error] Request object information:
[Mon May 11 15:35:04 2015] [error] URI:
/XMLRPC/GET-REQ/rhel-x86_64-server-6-datacenter-hpsa/getPackage/infx_12.10FC3-1.1-1.x86_64.rpm
[Mon May 11 15:35:04 2015] [error] Remote Host: katello-poc.ourcompany.com
[Mon May 11 15:35:04 2015] [error] Server Name: spw-proxy.ourcompany.com:443
[Mon May 11 15:35:04 2015] [error] Headers passed in:
[Mon May 11 15:35:04 2015] [error] \tAccept: */*
[Mon May 11 15:35:04 2015] [error] \tDOCUMENT_ROOT: /var/www/html
[Mon May 11 15:35:04 2015] [error] \tGATEWAY_INTERFACE: CGI/1.1
[Mon May 11 15:35:04 2015] [error] \tHTTPS: 1
[Mon May 11 15:35:04 2015] [error] \tHTTP_ACCEPT: */*
[Mon May 11 15:35:04 2015] [error] \tHTTP_HOST: spw-proxy.ourcompany.com
[Mon May 11 15:35:04 2015] [error] \tHTTP_USER_AGENT:
rhn.rpclib.py/2.5.65-1.el6
[Mon May 11 15:35:04 2015] [error]
\tHTTP_X_LIBCURL_EMPTY_HEADER_WORKAROUND: *
[Mon May 11 15:35:04 2015] [error] \tHTTP_X_RHN_AUTH:
vEqLA0CcEy/Ll5TPJQ1En9WOgWUuiBjHfDsoe30ksmU=
[Mon May 11 15:35:04 2015] [error] \tHTTP_X_RHN_AUTH_EXPIRE_OFFSET: 3600.0
[Mon May 11 15:35:04 2015] [error] \tHTTP_X_RHN_AUTH_SERVER_TIME:
1431372774.47
[Mon May 11 15:35:04 2015] [error] \tHTTP_X_RHN_AUTH_USER_ID:
[Mon May 11 15:35:04 2015] [error] \tHTTP_X_RHN_SERVER_ID: 1000015573
[Mon May 11 15:35:04 2015] [error] \tHTTP_X_RHN_TRANSPORT_CAPABILITY:
follow-redirects=3
[Mon May 11 15:35:04 2015] [error] \tHost: spw-proxy.ourcompany.com
[Mon May 11 15:35:04 2015] [error] \tPATH_INFO:
/GET-REQ/rhel-x86_64-server-6-datacenter-hpsa/getPackage/infx_12.10FC3-1.1-1.x86_64.rpm
[Mon May 11 15:35:04 2015] [error] \tPATH_TRANSLATED:
/var/www/html/GET-REQ/rhel-x86_64-server-6-datacenter-hpsa/getPackage/infx_12.10FC3-1.1-1.x86_64.rpm
[Mon May 11 15:35:04 2015] [error] \tQUERY_STRING:
[Mon May 11 15:35:04 2015] [error] \tREMOTE_ADDR: 172.18.121.224
[Mon May 11 15:35:04 2015] [error] \tREMOTE_PORT: 55117
[Mon May 11 15:35:04 2015] [error] \tREQUEST_METHOD: GET
[Mon May 11 15:35:04 2015] [error] \tREQUEST_URI:
/XMLRPC/GET-REQ/rhel-x86_64-server-6-datacenter-hpsa/getPackage/infx_12.10FC3-1.1-1.x86_64.rpm
[Mon May 11 15:35:04 2015] [error] \tSCRIPT_FILENAME:
/usr/share/rhn/wsgi/xmlrpc.py
[Mon May 11 15:35:04 2015] [error] \tSCRIPT_NAME: /XMLRPC
[Mon May 11 15:35:04 2015] [error] \tSCRIPT_URI:
https://spw-proxy.ourcompany.com/XMLRPC/GET-REQ/rhel-x86_64-server-6-datacenter-hpsa/getPackage/infx_12.10FC3-1.1-1.x86_64.rpm
[Mon May 11 15:35:04 2015] [error] \tSCRIPT_URL:
/XMLRPC/GET-REQ/rhel-x86_64-server-6-datacenter-hpsa/getPackage/infx_12.10FC3-1.1-1.x86_64.rpm
[Mon May 11 15:35:04 2015] [error] \tSERVER_ADDR: 172.18.138.140
[Mon May 11 15:35:04 2015] [error] \tSERVER_ADMIN: root at localhost
[Mon May 11 15:35:04 2015] [error] \tSERVER_NAME: spw-proxy.ourcompany.com
[Mon May 11 15:35:04 2015] [error] \tSERVER_PORT: 443
[Mon May 11 15:35:04 2015] [error] \tSERVER_PROTOCOL: HTTP/1.1
[Mon May 11 15:35:04 2015] [error] \tSERVER_SIGNATURE:
<address>Apache/2.2.15 (Red Hat) Server at spw-proxy.ourcompany.com Port
443</address>
[Mon May 11 15:35:04 2015] [error]
[Mon May 11 15:35:04 2015] [error] \tSERVER_SOFTWARE: Apache/2.2.15 (Red
Hat)
[Mon May 11 15:35:04 2015] [error] \tSSL_TLS_SNI: spw-proxy.ourcompany.com
[Mon May 11 15:35:04 2015] [error] \tUser-Agent: rhn.rpclib.py/2.5.65-1.el6
[Mon May 11 15:35:04 2015] [error] \tX-Libcurl-Empty-Header-Workaround: *
[Mon May 11 15:35:04 2015] [error] \tX-RHN-Auth:
vEqLA0CcEy/Ll5TPJQ1En9WOgWUuiBjHfDsoe30ksmU=
[Mon May 11 15:35:04 2015] [error] \tX-RHN-Auth-Expire-Offset: 3600.0
[Mon May 11 15:35:04 2015] [error] \tX-RHN-Auth-Server-Time: 1431372774.47
[Mon May 11 15:35:04 2015] [error] \tX-RHN-Auth-User-Id:
[Mon May 11 15:35:04 2015] [error] \tX-RHN-Server-Id: 1000015573
[Mon May 11 15:35:04 2015] [error] \tX-RHN-Transport-Capability:
follow-redirects=3
[Mon May 11 15:35:04 2015] [error] \tmod_ssl.is_https: <built-in method
ssl_is_https of mod_wsgi.Adapter object at 0x7f9fece290a8>
[Mon May 11 15:35:04 2015] [error] \tmod_ssl.var_lookup: <built-in method
ssl_var_lookup of mod_wsgi.Adapter object at 0x7f9fece290a8>
[Mon May 11 15:35:04 2015] [error] \tmod_wsgi.application_group:
spw-proxy.ourcompany.com|/xmlrpc
[Mon May 11 15:35:04 2015] [error] \tmod_wsgi.callable_object: application
[Mon May 11 15:35:04 2015] [error] \tmod_wsgi.handler_script:
[Mon May 11 15:35:04 2015] [error] \tmod_wsgi.input_chunked: 0
[Mon May 11 15:35:04 2015] [error] \tmod_wsgi.listener_host:
[Mon May 11 15:35:04 2015] [error] \tmod_wsgi.listener_port: 443
[Mon May 11 15:35:04 2015] [error] \tmod_wsgi.process_group:
[Mon May 11 15:35:04 2015] [error] \tmod_wsgi.request_handler: wsgi-script
[Mon May 11 15:35:04 2015] [error] \tmod_wsgi.script_reloading: 1
[Mon May 11 15:35:04 2015] [error] \tmod_wsgi.version: (3, 2)
[Mon May 11 15:35:04 2015] [error] \twsgi.errors: <mod_wsgi.Log object at
0x7f9fecea0eb0>
[Mon May 11 15:35:04 2015] [error] \twsgi.file_wrapper: <built-in method
file_wrapper of mod_wsgi.Adapter object at 0x7f9fece290a8>
[Mon May 11 15:35:04 2015] [error] \twsgi.input: <mod_wsgi.Input object at
0x7f9fed20eef0>
[Mon May 11 15:35:04 2015] [error] \twsgi.multiprocess: True
[Mon May 11 15:35:04 2015] [error] \twsgi.multithread: False
[Mon May 11 15:35:04 2015] [error] \twsgi.run_once: False
[Mon May 11 15:35:04 2015] [error] \twsgi.url_scheme: https
[Mon May 11 15:35:04 2015] [error] \twsgi.version: (1, 1)
[Mon May 11 15:35:04 2015] [error]
[Mon May 11 15:35:04 2015] [error] Exception Handler Information
[Mon May 11 15:35:04 2015] [error] Traceback (most recent call last):
[Mon May 11 15:35:04 2015] [error]   File
"/usr/share/rhn/proxy/rhnShared.py", line 201, in _serverCommo
[Mon May 11 15:35:04 2015] [error]     status, headers, bodyFd =
self._proxy2server()
[Mon May 11 15:35:04 2015] [error]   File
"/usr/share/rhn/proxy/rhnShared.py", line 375, in _proxy2server
[Mon May 11 15:35:04 2015] [error]     response =
http_connection.getresponse()
[Mon May 11 15:35:04 2015] [error]   File
"/usr/lib/python2.6/site-packages/rhn/connections.py", line 93, in
getresponse
[Mon May 11 15:35:04 2015] [error]     response.begin()
[Mon May 11 15:35:04 2015] [error]   File
"/usr/lib64/python2.6/httplib.py", line 391, in begin
[Mon May 11 15:35:04 2015] [error]     version, status, reason =
self._read_status()
[Mon May 11 15:35:04 2015] [error]   File
"/usr/lib64/python2.6/httplib.py", line 349, in _read_status
[Mon May 11 15:35:04 2015] [error]     line = self.fp.readline()
[Mon May 11 15:35:04 2015] [error]   File "/usr/lib64/python2.6/socket.py",
line 433, in readline
[Mon May 11 15:35:04 2015] [error]     data = recv(1)
[Mon May 11 15:35:04 2015] [error] timeout: timed out
[Mon May 11 15:35:04 2015] [error]
[Mon May 11 15:35:22 2015] [error] [client 172.18.138.140] mod_wsgi
(pid=25120): Exception occurred processing WSGI script
'/usr/share/rhn/wsgi/xmlrpc.py'.
[Mon May 11 15:35:22 2015] [error] [client 172.18.138.140] IOError: failed
to write data

On Mon, May 11, 2015 at 1:39 PM, Matthew Madey <mattmadey at gmail.com> wrote:

> Thanks for the responses. No, there are no other proxies between the
> Spacewalk Proxy and client, and I don't have any timeout settings specified
> in rhn.conf, yum.conf, or up2date.
> If I register the client directly to the Spacewalk server, I'm able to
> download the package without an issue.. so the problem definitely lies in
> the caching mechanism of the Spacewalk proxy. What concerns me is the below
> error I see on the Spacewalk Proxy when attempting to retrieve the package.
>
> [Fri May 08 18:42:28 2015] [error] [client 172.20.165.201] mod_wsgi
> (pid=9403): Exception occurred processing WSGI script
> '/usr/share/rhn/wsgi/xmlrpc.py'.
> [Fri May 08 18:42:28 2015] [error] [client 172.20.165.201] IOError: failed
> to write data
>
>
>
> On Mon, May 11, 2015 at 11:21 AM, Michael Calmer <mc at suse.de> wrote:
>
>> Hi,
>>
>> Am Freitag, 8. Mai 2015, 17:52:29 schrieb Matthew Madey:
>> > I'm seeing an issue with packages 300MB and larger being served by
>> > Spacewalk proxies. When I run yum install on a client, it gets to 97% of
>> > the download or so and fails with the below message:
>> >
>> > Error Downloading Packages:
>> >  somepackage.x86_64: failed to retrieve
>> getPackage/somepackage.x86_64.rpm
>> > from rhel-x86_64-server-6-datacenter
>> > error was [Errno 14] PYCURL ERROR 18 - "transfer closed with 1499666
>> bytes
>> > remaining to read"
>> >
>> > Looking at the Spacewalk proxy, I see the below error:
>> >
>> > [Fri May 08 18:42:28 2015] [error] [client 172.20.165.201] mod_wsgi
>> > (pid=9403): Exception occurred processing WSGI script
>> > '/usr/share/rhn/wsgi/xmlrpc.py'.
>> > [Fri May 08 18:42:28 2015] [error] [client 172.20.165.201] IOError:
>> failed
>> > to write data
>> >
>> > Here's what I've tried so far with no luck:
>> >
>> > 1. Increase max_object_size parameter in squid.conf to 500MB
>> > 2. Turn on KeepAlive parameter in httpd.conf
>> > 3. Increase KeepAlive timeout in httpd.conf to 120 seconds
>> > 4. Increase Timeout in httpd.conf to 120 seconds
>>
>> What about the timeout parameter in rhn.conf (server and proxy) and in
>> up2date
>> conf (for yum - or was it in a special yum config file?)
>>
>> --
>> Regards
>>
>>         Michael Calmer
>>
>> --------------------------------------------------------------------------
>> Michael Calmer
>> SUSE LINUX GmbH, Maxfeldstr. 5, D-90409 Nuernberg
>> T: +49 (0) 911 74053 0
>> F: +49 (0) 911 74053575  - e-mail: Michael.Calmer at suse.com
>> --------------------------------------------------------------------------
>> SUSE Linux GmbH, GF: Felix Imendörffer, Jane Smithard, Dilip Upmanyu,
>>                      Graham Norton, HRB 21284 (AG Nürnberg)
>>
>>
>> _______________________________________________
>> Spacewalk-list mailing list
>> Spacewalk-list at redhat.com
>> https://www.redhat.com/mailman/listinfo/spacewalk-list
>>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/spacewalk-list/attachments/20150511/e82416bc/attachment.htm>