[Pulp-list] Tuning Pulpcore Worker Count in Katello

[Eric Helms]

Are Pulp 3 workers running in a threaded manner ? There is a related
concern, that is more of a concern for Katello installations, around the
number of database connections needed to prevent starvation and ensure
PostgreSQL is tuned correctly to handle this. For Foreman/Katello tihs
means we need to count across Foreman, Foreman's task handler, Candlepin
and Pulp connections.

If you can also speak to the requirements for API and Content app that
would be helpful as well rounding out the information.

On Mon, Jun 15, 2020 at 2:14 PM Brian Bouterse <bmbouter at redhat.com> wrote:

>
>
> On Mon, Jun 15, 2020 at 1:09 PM William Clark <wclark at redhat.com> wrote:
>
>> Hello Pulp Community!
>>
>> I'm working on a feature to allow the foreman-installer to set the number
>> of Pulpcore workers deployed on Katello or Content Proxy, and I require
>> some assistance from the Pulp community in setting sane defaults and limits.
>>
>> With Pulp 2 in Katello, the default behavior was that the worker count
>> would match the number of logical CPUs up to a soft limit of 8. We advised
>> that users could tune the worker count higher but it was expected to cause
>> performance degradation in most cases due to I/O blocking. The largest
>> scale installation to my knowledge uses a Pulp 2 worker count of 16.
>>
> I believe having one pulpcore-worker per CPU is still a good practice. We
> haven't gotten a lot of feedback on right-sizing an installation so I won't
> claim it's the absolute best practice, but it's what I recommend currently.
> We have an issue to document sizing recommendations here that also has some
> similar/more info:  https://pulp.plan.io/issues/6856
>
> The I/O blocking concern is roughly about the same as Pulp2; during sync
> operations the workload could be I/O bound.
>
> A lot more CPU processing has been moved into postgresql which auto forks
> postgresql processes per client connection, which in this case is a 1-1
> pairing with each pulpcore-worker. So when it's under heavy load I expect
> postresql to scale out a process and the workload could become constrained
> on the postgresql CPU itself. In that case, lowering the worker to half of
> the available processors would likely improve throughput. Moving postgresql
> to another, dedicated box is also an option.
>
> If you'd be willing to share your findings with the Pulp community that
> would be really great.
>
>
>> With Pulp 2 being replaced and rebuilt with Pulpcore, I'm looking to
>> understand what are the tuning best practices for the new technology, so
>> that we could apply them to Katello.
>>
> Please let us know what other questions we can help answer.
>
>
>> I am looking forward to hearing from you,
>>
> Thank you for reaching out.
>
>
>> --
>>
>> William Clark, RHCA
>>
>> He/Him/His
>>
>> Software Engineer
>>
>> Red Hat <https://www.redhat.com>
>>
>> IM: wclark
>> <https://www.redhat.com>
>> _______________________________________________
>> Pulp-list mailing list
>> Pulp-list at redhat.com
>> https://www.redhat.com/mailman/listinfo/pulp-list
>
> _______________________________________________
> Pulp-list mailing list
> Pulp-list at redhat.com
> https://www.redhat.com/mailman/listinfo/pulp-list



-- 
Eric Helms
Principal Software Engineer
Satellite and Cloud Services
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/pulp-list/attachments/20200615/c8142290/attachment.htm>