<div dir="ltr"><div dir="ltr"><div>This turned out to be a timeout issue with squid. I increased timeouts and it now works:</div><div>connect_timeout 3 minute<br>read_timeout 3 minute<br>write_timeout 3 minute<br></div><div><br></div><div>What is a reasonable squid config in this respect?</div><div><br></div><div>Thanks!!<br></div></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Mon, May 6, 2019 at 5:38 PM Guy Matz <<a href="mailto:guymatz@gmail.com">guymatz@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div>Hi!</div><div>I am upgrading from CentOS 7.4 to 7.6 and have added the new 7.6 channel on my master . . . I have modified my kickstart to point to the new release, however the proxy is not sending the squashfs.img file upon request. I can't even curl it from the proxy. I get the following error in my apache log on the proxy:<br></div><div><br></div><div>[Wed May 01 09:53:37.244532 2019] [:error] [pid 22642] Exception reported from <a href="http://nyc-pkgms-lv01.schrodinger.com" target="_blank">nyc-pkgms-lv01.schrodinger.com</a><br>[Wed May 01 09:53:37.244569 2019] [:error] [pid 22642] Time: Wed May 1 09:53:37 2019<br>[Wed May 01 09:53:37.244573 2019] [:error] [pid 22642] Exception type <class 'socket.timeout'><br>[Wed May 01 09:53:37.244577 2019] [:error] [pid 22642] Exception while handling function SharedHandler._serverCommo<br>[Wed May 01 09:53:37.244581 2019] [:error] [pid 22642] Request object information:<br>[Wed May 01 09:53:37.244584 2019] [:error] [pid 22642] URI: /ks/dist/org/1/centos7-base-x86_64/LiveOS/squashfs.img<br>[Wed May 01 09:53:37.244587 2019] [:error] [pid 22642] Remote Host: 172.18.9.25<br>[Wed May 01 09:53:37.244590 2019] [:error] [pid 22642] Server Name: <a href="http://nyc-pkgms-lv01.schrodinger.com:80" target="_blank">nyc-pkgms-lv01.schrodinger.com:80</a><br>[Wed May 01 09:53:37.244594 2019] [:error] [pid 22642] Headers passed in:<br>[Wed May 01 09:53:37.244597 2019] [:error] [pid 22642] \tAccept: */*<br>[Wed May 01 09:53:37.244600 2019] [:error] [pid 22642] \tCONTEXT_DOCUMENT_ROOT: /var/www/html<br>[Wed May 01 09:53:37.244603 2019] [:error] [pid 22642] \tCONTEXT_PREFIX:<br>[Wed May 01 09:53:37.244606 2019] [:error] [pid 22642] \tDOCUMENT_ROOT: /var/www/html<br>[Wed May 01 09:53:37.244609 2019] [:error] [pid 22642] \tGATEWAY_INTERFACE: CGI/1.1<br>[Wed May 01 09:53:37.244612 2019] [:error] [pid 22642] \tHTTP_ACCEPT: */*<br>[Wed May 01 09:53:37.244615 2019] [:error] [pid 22642] \tHTTP_HOST: <a href="http://nyc-pkgms-lv01.schrodinger.com" target="_blank">nyc-pkgms-lv01.schrodinger.com</a><br>[Wed May 01 09:53:37.244618 2019] [:error] [pid 22642] \tHTTP_USER_AGENT: curl/7.29.0<br>[Wed May 01 09:53:37.244621 2019] [:error] [pid 22642] \tHTTP_X_ANACONDA_ARCHITECTURE: x86_64 [Wed May 01 09:53:37.244624 2019] [:error] [pid 22642] \tHTTP_X_ANACONDA_SYSTEM_RELEASE: CentOS<br>[Wed May 01 09:53:37.244627 2019] [:error] [pid 22642] \tHost: <a href="http://nyc-pkgms-lv01.schrodinger.com" target="_blank">nyc-pkgms-lv01.schrodinger.com</a><br>[Wed May 01 09:53:37.244630 2019] [:error] [pid 22642] \tPATH_INFO: /dist/org/1/centos7-base-x86_64/LiveOS/squashfs.img</div><div>[Wed May 01 09:53:37.244633 2019] [:error] [pid 22642] \tPATH_TRANSLATED: /var/www/html/dist/org/1/centos7-base-x86_64/LiveOS/squashfs.img<br>[Wed May 01 09:53:37.244637 2019] [:error] [pid 22642] \tQUERY_STRING:<br>[Wed May 01 09:53:37.244640 2019] [:error] [pid 22642] \tREMOTE_ADDR: 172.18.9.25<br>[Wed May 01 09:53:37.244643 2019] [:error] [pid 22642] \tREMOTE_PORT: 38484<br>[Wed May 01 09:53:37.244646 2019] [:error] [pid 22642] \tREQUEST_METHOD: GET<br>[Wed May 01 09:53:37.244649 2019] [:error] [pid 22642] \tREQUEST_SCHEME: http<br>[Wed May 01 09:53:37.244652 2019] [:error] [pid 22642] \tREQUEST_URI: /ks/dist/org/1/centos7-base-x86_64/LiveOS/squashfs.img<br>[Wed May 01 09:53:37.244655 2019] [:error] [pid 22642] \tSCRIPT_FILENAME: /usr/share/rhn/wsgi/xmlrpc.py<br>[Wed May 01 09:53:37.244659 2019] [:error] [pid 22642] \tSCRIPT_NAME: /ks<br>[Wed May 01 09:53:37.244662 2019] [:error] [pid 22642] \tSCRIPT_URI: <a href="http://nyc-pkgms-lv01.schrodinger.com/ks/dist/org/1/centos7-base-x86_64/LiveOS/squashfs.img" target="_blank">http://nyc-pkgms-lv01.schrodinger.com/ks/dist/org/1/centos7-base-x86_64/LiveOS/squashfs.img</a><br>[Wed May 01 09:53:37.244665 2019] [:error] [pid 22642] \tSCRIPT_URL: /ks/dist/org/1/centos7-base-x86_64/LiveOS/squashfs.img<br>[Wed May 01 09:53:37.244668 2019] [:error] [pid 22642] \tSERVER_ADDR: 172.18.4.65<br>[Wed May 01 09:53:37.244671 2019] [:error] [pid 22642] \tSERVER_ADMIN: root@localhost<br>[Wed May 01 09:53:37.244674 2019] [:error] [pid 22642] \tSERVER_NAME: <a href="http://nyc-pkgms-lv01.schrodinger.com" target="_blank">nyc-pkgms-lv01.schrodinger.com</a><br>[Wed May 01 09:53:37.244677 2019] [:error] [pid 22642] \tSERVER_PORT: 80<br>[Wed May 01 09:53:37.244680 2019] [:error] [pid 22642] \tSERVER_PROTOCOL: HTTP/1.1<br>[Wed May 01 09:53:37.244683 2019] [:error] [pid 22642] \tSERVER_SIGNATURE:<br>[Wed May 01 09:53:37.244686 2019] [:error] [pid 22642] \tSERVER_SOFTWARE: Apache/2.4.6 (CentOS) OpenSSL/1.0.2k-fips mod_wsgi/3.4 Python/2.7.5<br>[Wed May 01 09:53:37.244704 2019] [:error] [pid 22642] \tUNIQUE_ID: XMmkaWXwRKo2zJygVH2NzAAAAAI<br>[Wed May 01 09:53:37.244708 2019] [:error] [pid 22642] \tUser-Agent: curl/7.29.0<br>[Wed May 01 09:53:37.244712 2019] [:error] [pid 22642] \tX-Anaconda-Architecture: x86_64<br>[Wed May 01 09:53:37.244715 2019] [:error] [pid 22642] \tX-Anaconda-System-Release: CentOS<br>[Wed May 01 09:53:37.244718 2019] [:error] [pid 22642] \tmod_wsgi.application_group: <a href="http://nyc-pkgms-lv01.schrodinger.com" target="_blank">nyc-pkgms-lv01.schrodinger.com</a>|/ks<br>[Wed May 01 09:53:37.244721 2019] [:error] [pid 22642] \tmod_wsgi.callable_object: application<br>[Wed May 01 09:53:37.244724 2019] [:error] [pid 22642] \tmod_wsgi.enable_sendfile: 0<br>[Wed May 01 09:53:37.244727 2019] [:error] [pid 22642] \tmod_wsgi.handler_script:<br>[Wed May 01 09:53:37.244730 2019] [:error] [pid 22642] \tmod_wsgi.input_chunked: 0<br>[Wed May 01 09:53:37.244733 2019] [:error] [pid 22642] \tmod_wsgi.listener_host:<br>[Wed May 01 09:53:37.244737 2019] [:error] [pid 22642] \tmod_wsgi.listener_port: 80<br>[Wed May 01 09:53:37.244740 2019] [:error] [pid 22642] \tmod_wsgi.process_group:<br>[Wed May 01 09:53:37.244743 2019] [:error] [pid 22642] \tmod_wsgi.queue_start: 1556718697157324<br>[Wed May 01 09:53:37.244746 2019] [:error] [pid 22642] \tmod_wsgi.request_handler: wsgi-script<br>[Wed May 01 09:53:37.244749 2019] [:error] [pid 22642] \tmod_wsgi.script_reloading: 1<br>[Wed May 01 09:53:37.244752 2019] [:error] [pid 22642] \tmod_wsgi.version: (3, 4)<br>[Wed May 01 09:53:37.244755 2019] [:error] [pid 22642] \twsgi.errors: <mod_wsgi.Log object at 0x55e380bc98f0><br>[Wed May 01 09:53:37.244759 2019] [:error] [pid 22642] \twsgi.file_wrapper: <built-in method file_wrapper of mod_wsgi.Adapter object at 0x55e380cbe6c0><br>[Wed May 01 09:53:37.244762 2019] [:error] [pid 22642] \twsgi.input: <cStringIO.StringO object at 0x55e380f71e30><br>[Wed May 01 09:53:37.244765 2019] [:error] [pid 22642] \twsgi.multiprocess: True<br>[Wed May 01 09:53:37.244768 2019] [:error] [pid 22642] \twsgi.multithread: False<br>[Wed May 01 09:53:37.244771 2019] [:error] [pid 22642] \twsgi.run_once: False<br>[Wed May 01 09:53:37.244775 2019] [:error] [pid 22642] \twsgi.url_scheme: http<br>[Wed May 01 09:53:37.244778 2019] [:error] [pid 22642] \twsgi.version: (1, 0)<br>[Wed May 01 09:53:37.244781 2019] [:error] [pid 22642]<br>[Wed May 01 09:53:37.244784 2019] [:error] [pid 22642] Exception Handler Information<br>[Wed May 01 09:53:37.244787 2019] [:error] [pid 22642] Traceback (most recent call last):<br>[Wed May 01 09:53:37.244790 2019] [:error] [pid 22642] File "/usr/share/rhn/proxy/rhnShared.py", line 210, in _serverCommo<br>[Wed May 01 09:53:37.244793 2019] [:error] [pid 22642] status, headers, bodyFd = self._proxy2server()<br>[Wed May 01 09:53:37.244796 2019] [:error] [pid 22642] File "/usr/share/rhn/proxy/rhnShared.py", line 385, in _proxy2server<br>[Wed May 01 09:53:37.244800 2019] [:error] [pid 22642] response = http_connection.getresponse()<br>[Wed May 01 09:53:37.244803 2019] [:error] [pid 22642] File "/usr/lib/python2.7/site-packages/rhn/connections.py", line 102, in getresponse<br>[Wed May 01 09:53:37.244806 2019] [:error] [pid 22642] response.begin()<br>[Wed May 01 09:53:37.244809 2019] [:error] [pid 22642] File "/usr/lib64/python2.7/httplib.py", line 444, in begin<br>[Wed May 01 09:53:37.244812 2019] [:error] [pid 22642] version, status, reason = self._read_status()<br>[Wed May 01 09:53:37.244816 2019] [:error] [pid 22642] File "/usr/lib64/python2.7/httplib.py", line 400, in _read_status<br>[Wed May 01 09:53:37.244819 2019] [:error] [pid 22642] line = self.fp.readline(_MAXLINE + 1)<br>[Wed May 01 09:53:37.244822 2019] [:error] [pid 22642] File "/usr/lib64/python2.7/socket.py", line 476, in readline<br>[Wed May 01 09:53:37.244825 2019] [:error] [pid 22642] data = self._sock.recv(self._rbufsize)<br>[Wed May 01 09:53:37.244828 2019] [:error] [pid 22642] timeout: timed out<br>[Wed May 01 09:53:37.244834 2019] [:error] [pid 22642]<br>[Wed May 01 09:53:48.826978 2019] [:error] [pid 22863] [client <a href="http://172.18.4.65:55712" target="_blank">172.18.4.65:55712</a>] mod_wsgi (pid=22863): Exception occurred processing WSGI script '/usr/share/rhn/wsgi/xmlrpc.py'.<br>[Wed May 01 09:53:48.827058 2019] [:error] [pid 22863] [client <a href="http://172.18.4.65:55712" target="_blank">172.18.4.65:55712</a>] IOError: failed to write data<br><br></div><div>I see "IOError: failed to write data" and "timeout: timed out" but I can't tell where this is going wrong. Does anyone have a recommendation for how to debug xmlrpc.py?</div><div><br></div><div>Thanks,</div><div>Guy<br></div></div></div></div> </blockquote></div>