<div dir="ltr"><div dir="ltr"><br></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Mon, Jun 15, 2020 at 2:27 PM Eric Helms <<a href="mailto:ehelms@redhat.com">ehelms@redhat.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">Are Pulp 3 workers running in a threaded manner ? There is a related concern, that is more of a concern for Katello installations, around the number of database connections needed to prevent starvation and ensure PostgreSQL is tuned correctly to handle this. For Foreman/Katello tihs means we need to count across Foreman, Foreman's task handler, Candlepin and Pulp connections.</div></blockquote><div>Generally there are no threads but there are caveats. First, I don't know of any Pulp code that uses threads, but it's possible code in the future would. I know of no plans currently though. Second, aiofiles (a dependency of Pulp) does use threads to move files around. These threads should not make db connections though. Third, plugins can ship their own tasks that can be run so this isn't entirely in our control.<br></div><div><br></div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div><br></div><div>If you can also speak to the requirements for API and Content app that would be helpful as well rounding out the information.</div></div></blockquote><div>Sure. The Content App and the API are expected to not load system resources significantly, so for those it's mainly about how much capacity you want. For example, if you want to serve lots of clients binary data yourself with the content app (when not using S3 or Azure like pulp supports), then you'll want "more" content app workers. For the API, the capacity there is about the # API operations per second. More workers the more concurrent API operations per second you can do. Generally I expect you'll want a small number of API workers and a variable (potentially large) number of content app workers. You'll also need to make sure your reverse proxy configs can match the connection throughput desired also.<br></div><div><br></div><div>Workers in these cases are typically gunicorn workers in a single gunicorn process, but to really scale a system we'll need to have both more gunicorn workers per proces to vertically scale on a single node, and more gunicorn processes themselves (each with N gunicorn workers) to horizontally scale. Both vertical and horizontal scaling can be deployed manually and fully today. The installer currently can vertically scale the number of content and API gunicorn workers per gunicorn process, we cannot currently perform clustered installations for horizontal scaling well. We are working on that clustered install capability now actually.<br></div><div> </div><div>More questions are welcome; this topic is important and can be a bit complicated with our lack of docs. :/<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Mon, Jun 15, 2020 at 2:14 PM Brian Bouterse <<a href="mailto:bmbouter@redhat.com" target="_blank">bmbouter@redhat.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div><br></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Mon, Jun 15, 2020 at 1:09 PM William Clark <<a href="mailto:wclark@redhat.com" target="_blank">wclark@redhat.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div><div><div><div>Hello Pulp Community!<br><br></div>I'm working on a
feature to allow the foreman-installer to set the number of Pulpcore
workers deployed on Katello or Content Proxy, and I require some
assistance from the Pulp community in setting sane defaults and limits.<br><br></div>With
Pulp 2 in Katello, the default behavior was that the worker count would
match the number of logical CPUs up to a soft limit of 8. We advised
that users could tune the worker count higher but it was expected to
cause performance degradation in most cases due to I/O blocking. The
largest scale installation to my knowledge uses a Pulp 2 worker count of
16.<br></div></div></div></blockquote><div>I believe having one pulpcore-worker per CPU is still a good practice. We haven't gotten a lot of feedback on right-sizing an installation so I won't claim it's the absolute best practice, but it's what I recommend currently. We have an issue to document sizing recommendations here that also has some similar/more info: <a href="https://pulp.plan.io/issues/6856" target="_blank">https://pulp.plan.io/issues/6856</a></div><div><br></div><div>The I/O blocking concern is roughly about the same as Pulp2; during sync operations the workload could be I/O bound.<br></div><div><br></div><div>A lot more CPU processing has been moved into postgresql which auto forks postgresql processes per client connection, which in this case is a 1-1 pairing with each pulpcore-worker. So when it's under heavy load I expect postresql to scale out a process and the workload could become constrained on the postgresql CPU itself. In that case, lowering the worker to half of the available processors would likely improve throughput. Moving postgresql to another, dedicated box is also an option.<br></div><div><br></div><div>If you'd be willing to share your findings with the Pulp community that would be really great.<br></div><div><br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div><div><br></div>With Pulp 2 being replaced and rebuilt with Pulpcore,
I'm looking to understand what are the tuning best practices for the new
technology, so that we could apply them to Katello.<br></div></div></blockquote><div>Please let us know what other questions we can help answer.<br></div><div><br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div><br></div>I am looking forward to hearing from you,<br clear="all"></div></blockquote><div>Thank you for reaching out.</div><div> <br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><br>-- <br><div dir="ltr"><div dir="ltr"><div>
<p style="font-weight:bold;margin:0px;padding:0px;font-size:14px;font-family:RedHatText,sans-serif">
<span>William</span> <span>Clark</span><span style="color:rgb(170,170,170);margin:0px">, RHCA</span>
</p>
<p style="font-weight:normal;font-size:12px;margin:0px 0px 4px;text-transform:capitalize;font-family:RedHatText,sans-serif">He/Him/His</p>
<p style="font-weight:normal;font-size:12px;margin:0px;font-family:RedHatText,sans-serif">
<span>Software Engineer</span>
</p>
<p style="font-weight:normal;margin:0px 0px 4px;font-size:12px;font-family:RedHatText,sans-serif">
<a style="color:rgb(0,136,206);font-size:12px;margin:0px;text-decoration:none;font-family:RedHatText,sans-serif" href="https://www.redhat.com" target="_blank">Red Hat <span></span></a>
</p>
<div style="margin-bottom:4px">
</div>
<p style="font-weight:normal;margin:0px;font-size:12px;font-family:RedHatText,sans-serif">
<span>IM: <span>wclark</span></span>
</p>
<div style="margin-top:12px">
<table border="0">
<tbody><tr>
<td width="100px"><a href="https://www.redhat.com" target="_blank"> <img src="https://static.redhat.com/libs/redhat/brand-assets/2/corp/logo--200.png" width="90" height="auto"></a> </td>
</tr>
</tbody></table>
</div>
</div></div></div></div>
_______________________________________________<br>
Pulp-list mailing list<br>
<a href="mailto:Pulp-list@redhat.com" target="_blank">Pulp-list@redhat.com</a><br>
<a href="https://www.redhat.com/mailman/listinfo/pulp-list" rel="noreferrer" target="_blank">https://www.redhat.com/mailman/listinfo/pulp-list</a></blockquote></div></div>
_______________________________________________<br>
Pulp-list mailing list<br>
<a href="mailto:Pulp-list@redhat.com" target="_blank">Pulp-list@redhat.com</a><br>
<a href="https://www.redhat.com/mailman/listinfo/pulp-list" rel="noreferrer" target="_blank">https://www.redhat.com/mailman/listinfo/pulp-list</a></blockquote></div><br clear="all"><div><br></div>-- <br><div dir="ltr"><div dir="ltr"><div><div dir="ltr">Eric Helms<div>Principal Software Engineer</div><div>Satellite and Cloud Services</div></div></div></div></div>
</blockquote></div></div>