<div dir="ltr">Are Pulp 3 workers running in a threaded manner ? There is a related concern, that is more of a concern for Katello installations, around the number of database connections needed to prevent starvation and ensure PostgreSQL is tuned correctly to handle this. For Foreman/Katello tihs means we need to count across Foreman, Foreman's task handler, Candlepin and Pulp connections.<div><br></div><div>If you can also speak to the requirements for API and Content app that would be helpful as well rounding out the information.</div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Mon, Jun 15, 2020 at 2:14 PM Brian Bouterse <<a href="mailto:bmbouter@redhat.com">bmbouter@redhat.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div><br></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Mon, Jun 15, 2020 at 1:09 PM William Clark <<a href="mailto:wclark@redhat.com" target="_blank">wclark@redhat.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div><div><div><div>Hello Pulp Community!<br><br></div>I'm working on a
feature to allow the foreman-installer to set the number of Pulpcore
workers deployed on Katello or Content Proxy, and I require some
assistance from the Pulp community in setting sane defaults and limits.<br><br></div>With
Pulp 2 in Katello, the default behavior was that the worker count would
match the number of logical CPUs up to a soft limit of 8. We advised
that users could tune the worker count higher but it was expected to
cause performance degradation in most cases due to I/O blocking. The
largest scale installation to my knowledge uses a Pulp 2 worker count of
16.<br></div></div></div></blockquote><div>I believe having one pulpcore-worker per CPU is still a good practice. We haven't gotten a lot of feedback on right-sizing an installation so I won't claim it's the absolute best practice, but it's what I recommend currently. We have an issue to document sizing recommendations here that also has some similar/more info: <a href="https://pulp.plan.io/issues/6856" target="_blank">https://pulp.plan.io/issues/6856</a></div><div><br></div><div>The I/O blocking concern is roughly about the same as Pulp2; during sync operations the workload could be I/O bound.<br></div><div><br></div><div>A lot more CPU processing has been moved into postgresql which auto forks postgresql processes per client connection, which in this case is a 1-1 pairing with each pulpcore-worker. So when it's under heavy load I expect postresql to scale out a process and the workload could become constrained on the postgresql CPU itself. In that case, lowering the worker to half of the available processors would likely improve throughput. Moving postgresql to another, dedicated box is also an option.<br></div><div><br></div><div>If you'd be willing to share your findings with the Pulp community that would be really great.<br></div><div><br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div><div><br></div>With Pulp 2 being replaced and rebuilt with Pulpcore,
I'm looking to understand what are the tuning best practices for the new
technology, so that we could apply them to Katello.<br></div></div></blockquote><div>Please let us know what other questions we can help answer.<br></div><div><br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div><br></div>I am looking forward to hearing from you,<br clear="all"></div></blockquote><div>Thank you for reaching out.</div><div> <br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><br>-- <br><div dir="ltr"><div dir="ltr"><div>
<p style="font-weight:bold;margin:0px;padding:0px;font-size:14px;font-family:RedHatText,sans-serif">
<span>William</span> <span>Clark</span><span style="color:rgb(170,170,170);margin:0px">, RHCA</span>
</p>
<p style="font-weight:normal;font-size:12px;margin:0px 0px 4px;text-transform:capitalize;font-family:RedHatText,sans-serif">He/Him/His</p>
<p style="font-weight:normal;font-size:12px;margin:0px;font-family:RedHatText,sans-serif">
<span>Software Engineer</span>
</p>
<p style="font-weight:normal;margin:0px 0px 4px;font-size:12px;font-family:RedHatText,sans-serif">
<a style="color:rgb(0,136,206);font-size:12px;margin:0px;text-decoration:none;font-family:RedHatText,sans-serif" href="https://www.redhat.com" target="_blank">Red Hat <span></span></a>
</p>
<div style="margin-bottom:4px">
</div>
<p style="font-weight:normal;margin:0px;font-size:12px;font-family:RedHatText,sans-serif">
<span>IM: <span>wclark</span></span>
</p>
<div style="margin-top:12px">
<table border="0">
<tbody><tr>
<td width="100px"><a href="https://www.redhat.com" target="_blank"> <img src="https://static.redhat.com/libs/redhat/brand-assets/2/corp/logo--200.png" width="90" height="auto"></a> </td>
</tr>
</tbody></table>
</div>
</div></div></div></div>
_______________________________________________<br>
Pulp-list mailing list<br>
<a href="mailto:Pulp-list@redhat.com" target="_blank">Pulp-list@redhat.com</a><br>
<a href="https://www.redhat.com/mailman/listinfo/pulp-list" rel="noreferrer" target="_blank">https://www.redhat.com/mailman/listinfo/pulp-list</a></blockquote></div></div>
_______________________________________________<br>
Pulp-list mailing list<br>
<a href="mailto:Pulp-list@redhat.com" target="_blank">Pulp-list@redhat.com</a><br>
<a href="https://www.redhat.com/mailman/listinfo/pulp-list" rel="noreferrer" target="_blank">https://www.redhat.com/mailman/listinfo/pulp-list</a></blockquote></div><br clear="all"><div><br></div>-- <br><div dir="ltr" class="gmail_signature"><div dir="ltr"><div><div dir="ltr">Eric Helms<div>Principal Software Engineer</div><div>Satellite and Cloud Services</div></div></div></div></div>