[Pulp-list] Apache 502 proxy errors when performing Pulp 3 to Pulp 3 syncs

[Eric Helms]

Howdy,

Some quick background, over in the Katello project we deploy the
pulpcore-content service via a unix socket with Apache serving as a reverse
proxy. Today, we deploy pulpcore-content with two gunicorn workers.
Tomorrow we are considering changing this to 2 * CPU + 1 per gunicorn
documentation.

The issue we are running into is intermittent 502s from Apache caused by
being unable to make a connection to the underlying pulpcore-content app.
This manifests itself primarily during a Pulp 3 to Pulp 3 sync. That is,
when we sync our content proxy from the main servers Pulp 3. The sync can
result in a large number of parallel connections back to the
pulpcore-content application running on the main server.

In an issue for aiohttp which is used by the project, and whose worker is
used for gunicorn [1] they talk about the issues with Apache and aiohttp.
In that issue there are two suggestions that I could extract:

 1) set disablereuse=on the Apache reverse proxy declarations for the
content app
 2) change the default Apache worker type to be more like Nginx

There are performance tradeoffs with #1, however, I do not fully grasp if
they are relative to our primary use case when it comes to the content app.

So, I am coming to the experts here to try to get some insight into what
changes we should pursue to ensure the optimal default performance for our
deployment. And to, as best as we can, limit these kind of intermittent
failures to extreme cases. Because today, we see this intermittent failure
with Pulp 3 running the same test suite we did with Pulp 2.

Related, is there retry support built into syncing?

Thanks!

[1] https://github.com/aio-libs/aiohttp/issues/2687

-- 
Eric Helms
Principal Software Engineer
Satellite
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/pulp-list/attachments/20210205/69659520/attachment.htm>