[Pulp-list] 503 errors attempting to download packages

[Jason B. Nance]

Hello,

I'm running a Katello 3.3 server which comes with Pulp 2.9.3.  Recently I configured the server to use an http proxy to pull content from upstream repositories.  Shortly after, I discovered some issues with the proxy which was blocking the server from downloading packages.  That issue was resolved, but now I'm getting the nectar errors below.  It seemed as if I was running into this issue (though now I'm not so sure):

https://pulp.plan.io/issues/2344

...which is fixed in Pulp 2.10.1.  I'm currently unable to upgrade my Katello server due to an unrelated bug, so in the meantime I'm allowing the server to connect directly to retrieve packages, removed the proxy configuration from Katello, restarted everything, and performed a resync of all of my repositories.

However, I'm still getting the same nectar errors.

Mar 23 22:05:54 sl1mmgplsat0001.ipa.gen.zone pulp_streamer[90439]: requests.packages.urllib3.connectionpool:INFO: Starting new HTTPS connection (41): dl.fedoraproject.org
Mar 23 22:05:54 sl1mmgplsat0001.ipa.gen.zone pulp_streamer[90439]: requests.packages.urllib3.connectionpool:INFO: Starting new HTTPS connection (42): dl.fedoraproject.org
Mar 23 22:05:54 sl1mmgplsat0001.ipa.gen.zone pulp_streamer[90439]: nectar.downloaders.threaded:ERROR: Skipping requests to dl.fedoraproject.org due to repeated connection failures: [Errno 2] No such file or directory
Mar 23 22:05:54 sl1mmgplsat0001.ipa.gen.zone pulp_streamer[90439]: [-] 127.0.0.1 - - [24/Mar/2017:03:05:54 +0000] "GET /var/lib/pulp/content/units/rpm/75/0c1e41c96eb177be37db9c96b62bfca16127acee984f1d0ddfba97bb7d4634/qpid-proton-c-0.17.0-1.el7.x86_64.rpm HTTP/1.1" 503 - "-" "urlgrabber/3.10 yum/3.4.3"
Mar 23 22:05:54 sl1mmgplsat0001.ipa.gen.zone pulp_streamer[90439]: nectar.downloaders.threaded:ERROR: Skipping requests to dl.fedoraproject.org due to repeated connection failures: [Errno 2] No such file or directory
Mar 23 22:05:54 sl1mmgplsat0001.ipa.gen.zone pulp_streamer[90439]: [-] 127.0.0.1 - - [24/Mar/2017:03:05:54 +0000] "GET /var/lib/pulp/content/units/rpm/01/47d525166c41e61a359eadd535ae2a3675ca309e2ad3de69b8aafdcef44ed3/python-qpid-proton-0.17.0-1.el7.x86_64.rpm HTTP/1.1" 503 - "-" "urlgrabber/3.10 yum/3.4.3"

On my system, this does not exist:

/var/lib/pulp/content/units/rpm/75/0c1e41c96eb177be37db9c96b62bfca16127acee984f1d0ddfba97bb7d4634/qpid-proton-c-0.17.0-1.el7.x86_64.rpm

...but this does (real file):

/var/lib/pulp/content/units/rpm/e2/23414bf8eb85648d5146fe5d21d19bbaf5026fdb122463ae231b418df5fe10/qpid-proton-c-0.14.0-2.el7.x86_64.rpm

And this does not exist:

/var/lib/pulp/content/units/rpm/01/47d525166c41e61a359eadd535ae2a3675ca309e2ad3de69b8aafdcef44ed3/python-qpid-proton-0.17.0-1.el7.x86_64.rpm

...but this does (real file):

/var/lib/pulp/content/units/rpm/d6/2607d38c3808c4fb6c83575c6bbcae774c52680c15aa05e5b244471336d32b/python-qpid-proton-0.14.0-2.el7.x86_64.rpm

I'm using the on_demand sync type if that matters.

Any ideas on resolving this?

Thanks,

j