[Pulp-list] Tuning Pulpcore Worker Count in Katello

[Brian Bouterse]

On Mon, Jun 15, 2020 at 1:09 PM William Clark <wclark at redhat.com> wrote:

> Hello Pulp Community!
>
> I'm working on a feature to allow the foreman-installer to set the number
> of Pulpcore workers deployed on Katello or Content Proxy, and I require
> some assistance from the Pulp community in setting sane defaults and limits.
>
> With Pulp 2 in Katello, the default behavior was that the worker count
> would match the number of logical CPUs up to a soft limit of 8. We advised
> that users could tune the worker count higher but it was expected to cause
> performance degradation in most cases due to I/O blocking. The largest
> scale installation to my knowledge uses a Pulp 2 worker count of 16.
>
I believe having one pulpcore-worker per CPU is still a good practice. We
haven't gotten a lot of feedback on right-sizing an installation so I won't
claim it's the absolute best practice, but it's what I recommend currently.
We have an issue to document sizing recommendations here that also has some
similar/more info:  https://pulp.plan.io/issues/6856

The I/O blocking concern is roughly about the same as Pulp2; during sync
operations the workload could be I/O bound.

A lot more CPU processing has been moved into postgresql which auto forks
postgresql processes per client connection, which in this case is a 1-1
pairing with each pulpcore-worker. So when it's under heavy load I expect
postresql to scale out a process and the workload could become constrained
on the postgresql CPU itself. In that case, lowering the worker to half of
the available processors would likely improve throughput. Moving postgresql
to another, dedicated box is also an option.

If you'd be willing to share your findings with the Pulp community that
would be really great.


> With Pulp 2 being replaced and rebuilt with Pulpcore, I'm looking to
> understand what are the tuning best practices for the new technology, so
> that we could apply them to Katello.
>
Please let us know what other questions we can help answer.


> I am looking forward to hearing from you,
>
Thank you for reaching out.


> --
>
> William Clark, RHCA
>
> He/Him/His
>
> Software Engineer
>
> Red Hat <https://www.redhat.com>
>
> IM: wclark
> <https://www.redhat.com>
> _______________________________________________
> Pulp-list mailing list
> Pulp-list at redhat.com
> https://www.redhat.com/mailman/listinfo/pulp-list
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/pulp-list/attachments/20200615/b8539ea6/attachment.htm>