[Spacewalk-list] 1 proxy cant download channel's /repodata/primary.xml.gz

Fri Feb 20 17:04:31 UTC 2015

I would look at your squid logs for more information. I've had some weird
situations with packages that didn't get cached correctly and caused
problems. As a last resort I cleared the whole squid cache and bounced the
proxy services.. but hopefully you can find the root cause in the squid
logs.

rm -Rf /var/spool/squid/*
rhn-proxy restart

On Fri, Feb 20, 2015 at 10:37 AM, Matthew Rinella <MRinella at apptio.com>
wrote:

>
>
> I have a spacewalk installation with a mater and a few proxies.   All of a
> sudden today, one of my more remote proxies cannot serve
> repodata/primary.xml.gz for the channel.  It really looks like nothing
> changed on that proxy and its been in service for a couple months now.
>
>
>
>
>
> When clients hit the proxy, it serves a 404 for that file.
>
>
>
> The error log for a request:
>
> [Fri Feb 20 08:23:40 2015] [error] Exception reported from <hostname
> scrubbed>
>
> [Fri Feb 20 08:23:40 2015] [error] Time: Fri Feb 20 08:23:40 2015
>
> [Fri Feb 20 08:23:40 2015] [error] Exception type <class 'socket.timeout'>
>
> [Fri Feb 20 08:23:40 2015] [error] Exception while handling function
> SharedHandler._serverCommo
>
> [Fri Feb 20 08:23:40 2015] [error] Request object information:
>
> [Fri Feb 20 08:23:40 2015] [error] URI: /XMLRPC/GET-REQ/<channel name
> scrubbed> /repodata/primary.xml.gz
>
> [Fri Feb 20 08:23:40 2015] [error] Remote Host: <hostname scrubbed>
>
> [Fri Feb 20 08:23:40 2015] [error] Server Name: <hostname scrubbed>:80
>
> [Fri Feb 20 08:23:40 2015] [error] Headers passed in:
>
> [Fri Feb 20 08:23:40 2015] [error] \tAccept: */*
>
> [Fri Feb 20 08:23:40 2015] [error] \tDOCUMENT_ROOT: /var/www/html
>
> [Fri Feb 20 08:23:40 2015] [error] \tGATEWAY_INTERFACE: CGI/1.1
>
> [Fri Feb 20 08:23:40 2015] [error] \tHTTP_ACCEPT: */*
>
> [Fri Feb 20 08:23:40 2015] [error] \tHTTP_HOST: <hostname scrubbed>
>
> [Fri Feb 20 08:23:40 2015] [error] \tHTTP_PRAGMA: no-cache
>
> [Fri Feb 20 08:23:40 2015] [error] \tHTTP_USER_AGENT:
> rhn.rpclib.py/2.5.72-1.el6
>
> [Fri Feb 20 08:23:40 2015] [error]
> \tHTTP_X_LIBCURL_EMPTY_HEADER_WORKAROUND: *
>
> [Fri Feb 20 08:23:40 2015] [error] \tHTTP_X_RHN_AUTH:
> apQHxvpocp9VUQKDAKU9whrNlbnhbimIwxV+f+qSaLA=
>
> [Fri Feb 20 08:23:40 2015] [error] \tHTTP_X_RHN_AUTH_EXPIRE_OFFSET: 3600.0
>
> [Fri Feb 20 08:23:40 2015] [error] \tHTTP_X_RHN_AUTH_SERVER_TIME:
> 1424449299.62
>
> [Fri Feb 20 08:23:40 2015] [error] \tHTTP_X_RHN_AUTH_USER_ID:
>
> [Fri Feb 20 08:23:40 2015] [error] \tHTTP_X_RHN_SERVER_ID: 1000010032
>
> [Fri Feb 20 08:23:40 2015] [error] \tHTTP_X_RHN_TRANSPORT_CAPABILITY:
> follow-redirects=3
>
> [Fri Feb 20 08:23:40 2015] [error] \tHost: <hostname scrubbed>
>
> [Fri Feb 20 08:23:40 2015] [error] \tPATH_INFO: /GET-REQ/<channel name
> scrubbed>/repodata/primary.xml.gz
>
> [Fri Feb 20 08:23:40 2015] [error] \tPATH_TRANSLATED:
> /var/www/html/GET-REQ/<channel name scrubbed>/repodata/primary.xml.gz
>
> [Fri Feb 20 08:23:40 2015] [error] \tPragma: no-cache
>
> [Fri Feb 20 08:23:40 2015] [error] \tQUERY_STRING:
>
> [Fri Feb 20 08:23:40 2015] [error] \tREMOTE_ADDR: <IP scrubbed>
>
> [Fri Feb 20 08:23:40 2015] [error] \tREMOTE_PORT: 60139
>
> [Fri Feb 20 08:23:40 2015] [error] \tREQUEST_METHOD: GET
>
> [Fri Feb 20 08:23:40 2015] [error] \tREQUEST_URI: /XMLRPC/GET-REQ/<channel
> name scrubbed>/repodata/primary.xml.gz
>
> [Fri Feb 20 08:23:40 2015] [error] \tSCRIPT_FILENAME:
> /usr/share/rhn/wsgi/xmlrpc.py
>
> [Fri Feb 20 08:23:40 2015] [error] \tSCRIPT_NAME: /XMLRPC
>
> [Fri Feb 20 08:23:40 2015] [error] \tSCRIPT_URI: http://<hostname
> scrubbed>/XMLRPC/GET-REQ/<channel name scrubbed>/repodata/primary.xml.gz
>
> [Fri Feb 20 08:23:40 2015] [error] \tSCRIPT_URL: /XMLRPC/GET-REQ/<channel
> name scrubbed>/repodata/primary.xml.gz
>
> [Fri Feb 20 08:23:40 2015] [error] \tSERVER_ADDR: 172.23.1.93
>
> [Fri Feb 20 08:23:40 2015] [error] \tSERVER_ADMIN: root at localhost
>
> [Fri Feb 20 08:23:40 2015] [error] \tSERVER_NAME: <hostname scrubbed>
>
> [Fri Feb 20 08:23:40 2015] [error] \tSERVER_PORT: 80
>
> [Fri Feb 20 08:23:40 2015] [error] \tSERVER_PROTOCOL: HTTP/1.1
>
> [Fri Feb 20 08:23:40 2015] [error] \tSERVER_SIGNATURE:
> <address>Apache/2.2.15 (CentOS) Server at <hostname scrubbed> Port
> 80</address>
>
> [Fri Feb 20 08:23:40 2015] [error] \tSERVER_SOFTWARE: Apache/2.2.15
> (CentOS)
>
> [Fri Feb 20 08:23:40 2015] [error] \tUser-Agent:
> rhn.rpclib.py/2.5.72-1.el6
>
> [Fri Feb 20 08:23:40 2015] [error] \tX-Libcurl-Empty-Header-Workaround: *
>
> [Fri Feb 20 08:23:40 2015] [error] \tX-RHN-Auth:
> apQHxvpocp9VUQKDAKU9whrNlbnhbimIwxV+f+qSaLA=
>
> [Fri Feb 20 08:23:40 2015] [error] \tX-RHN-Auth-Expire-Offset: 3600.0
>
> [Fri Feb 20 08:23:40 2015] [error] \tX-RHN-Auth-Server-Time: 1424449299.62
>
> [Fri Feb 20 08:23:40 2015] [error] \tX-RHN-Auth-User-Id:
>
> [Fri Feb 20 08:23:40 2015] [error] \tX-RHN-Server-Id: 1000010032
>
> [Fri Feb 20 08:23:40 2015] [error] \tX-RHN-Transport-Capability:
> follow-redirects=3
>
> [Fri Feb 20 08:23:40 2015] [error] \tmod_ssl.is_https: <built-in method
> ssl_is_https of mod_wsgi.Adapter object at 0x7f29cd864030>
>
> [Fri Feb 20 08:23:40 2015] [error] \tmod_ssl.var_lookup: <built-in method
> ssl_var_lookup of mod_wsgi.Adapter object at 0x7f29cd864030>
>
> [Fri Feb 20 08:23:40 2015] [error] \tmod_wsgi.application_group: <hostname
> scrubbed>|/xmlrpc
>
> [Fri Feb 20 08:23:40 2015] [error] \tmod_wsgi.callable_object: application
>
> [Fri Feb 20 08:23:40 2015] [error] \tmod_wsgi.handler_script:
>
> [Fri Feb 20 08:23:40 2015] [error] \tmod_wsgi.input_chunked: 0
>
> [Fri Feb 20 08:23:40 2015] [error] \tmod_wsgi.listener_host:
>
> [Fri Feb 20 08:23:40 2015] [error] \tmod_wsgi.listener_port: 80
>
> [Fri Feb 20 08:23:40 2015] [error] \tmod_wsgi.process_group:
>
> [Fri Feb 20 08:23:40 2015] [error] \tmod_wsgi.request_handler: wsgi-script
>
> [Fri Feb 20 08:23:40 2015] [error] \tmod_wsgi.script_reloading: 1
>
> [Fri Feb 20 08:23:40 2015] [error] \tmod_wsgi.version: (3, 2)
>
> [Fri Feb 20 08:23:40 2015] [error] \twsgi.errors: <mod_wsgi.Log object at
> 0x7f29cd812f30>
>
> [Fri Feb 20 08:23:40 2015] [error] \twsgi.file_wrapper: <built-in method
> file_wrapper of mod_wsgi.Adapter object at 0x7f29cd864030>
>
> [Fri Feb 20 08:23:40 2015] [error] \twsgi.input: <mod_wsgi.Input object at
> 0x7f29cd812f70>
>
> [Fri Feb 20 08:23:40 2015] [error] \twsgi.multiprocess: True
>
> [Fri Feb 20 08:23:40 2015] [error] \twsgi.multithread: False
>
> [Fri Feb 20 08:23:40 2015] [error] \twsgi.run_once: False
>
> [Fri Feb 20 08:23:40 2015] [error] \twsgi.url_scheme: http
>
> [Fri Feb 20 08:23:40 2015] [error] \twsgi.version: (1, 1)
>
> [Fri Feb 20 08:23:40 2015] [error]
>
> [Fri Feb 20 08:23:40 2015] [error] Exception Handler Information
>
> [Fri Feb 20 08:23:40 2015] [error] Traceback (most recent call last):
>
> [Fri Feb 20 08:23:40 2015] [error]   File
> "/usr/share/rhn/proxy/rhnShared.py", line 201, in _serverCommo
>
> [Fri Feb 20 08:23:40 2015] [error]     status, headers, bodyFd =
> self._proxy2server()
>
> [Fri Feb 20 08:23:40 2015] [error]   File
> "/usr/share/rhn/proxy/rhnShared.py", line 375, in _proxy2server
>
> [Fri Feb 20 08:23:40 2015] [error]     response =
> http_connection.getresponse()
>
> [Fri Feb 20 08:23:40 2015] [error]   File
> "/usr/lib/python2.6/site-packages/rhn/connections.py", line 93, in
> getresponse
>
> [Fri Feb 20 08:23:40 2015] [error]     response.begin()
>
> [Fri Feb 20 08:23:40 2015] [error]   File
> "/usr/lib64/python2.6/httplib.py", line 391, in begin
>
> [Fri Feb 20 08:23:40 2015] [error]     version, status, reason =
> self._read_status()
>
> [Fri Feb 20 08:23:40 2015] [error]   File
> "/usr/lib64/python2.6/httplib.py", line 349, in _read_status
>
> [Fri Feb 20 08:23:40 2015] [error]     line = self.fp.readline()
>
> [Fri Feb 20 08:23:40 2015] [error]   File
> "/usr/lib64/python2.6/socket.py", line 433, in readline
>
> [Fri Feb 20 08:23:40 2015] [error]     data = recv(1)
>
> [Fri Feb 20 08:23:40 2015] [error] timeout: timed out
>
>
>
> The proxy is not having any actual network issues connecting to the
> master, and I can see in the UI that hosts going through that proxy are
> checking in as expected (one in the last minute even, so the networking is
> working fine, but it is a little slow since its remote).
>
>
>
> All of my other proxies are not having this problem
>
> _______________________________________________
> Spacewalk-list mailing list
> Spacewalk-list at redhat.com
> https://www.redhat.com/mailman/listinfo/spacewalk-list
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/spacewalk-list/attachments/20150220/b892c62b/attachment.htm>