<html><head><title></title></head><body><div class="rte-style-maintainer rte-pre-wrap" data-color="global-default" bbg-color="default" data-bb-font-size="medium"bbg-font-size="medium" style="white-space: pre-wrap; font-size: small; font-family: "Courier New", Courier, "BB.FixedWidth";"style="white-space: pre-wrap; font-size: small; font-family: "Courier New", Courier, "BB.FixedWidth"; color: rgb(0, 0, 0);">We will look into upgrade 3.7.3 to 3.14. <div>For now, I have updated number of worker a few times. We are having 50 workers running. I no longer see the timed out messages but the TIME_WAIT is still around 5k.</div><div><br></div><div># netstat -an | grep -i TIME_WAIT |grep 24816 | wc -l</div><div>5473</div><div><br></div><div>Also notice the database connection is over 60.</div><div>=> select count(*) from pg_stat_activity where usename = 'pulp';</div><div> count </div><div>-------</div><div> 63</div><div>(1 row)</div><div><br></div><div>Should I keep adding workers until the queue comes down? We still have plenty of cpu and memory on the host.</div><div><br><br><div class="rte-style-maintainer" data-color="global-default"bbg-color="default" data-bb-font-size="medium" bbg-font-size="medium" style="font-size: small; font-family: "Courier New", Courier, "BB.FixedWidth";"style="font-size: small; font-family: "Courier New", Courier, "BB.FixedWidth"; color: rgb(0, 0, 0);"><div><div class="bbg-rte-fold-content" data-header="From: bmbouter@redhat.com At: 06/22/21 12:01:30 UTC-4:00"data-digest="From: bmbouter@redhat.com At: 06/22/21 12:01:30 UTC-4:00" style=""><div class="bbg-rte-fold-summary">From: bmbouter@redhat.com At: 06/22/21 12:01:30 UTC-4:00</div>To: <a spellcheck="false" bbg-destination="mailto:danny.sauer@konghq.com" href="mailto:danny.sauer@konghq.com"> danny.sauer@konghq.com</a><br>Cc: <a spellcheck="false"bbg-destination="mailto:bli111@bloomberg.net" href="mailto:bli111@bloomberg.net"> Bin Li (BLOOMBERG/ 120 PARK ) </a>, <a spellcheck="false"bbg-destination="mailto:pulp-list@redhat.com" href="mailto:pulp-list@redhat.com"> pulp-list@redhat.com</a><br>Subject: Re: [Pulp-list] Content server Performance</div><br></div><div class="rte-internet-block-wrapper" style="background: white; color: black; font-family: Arial, "BB.Proportional"; font-size: small; white-space: normal;"><div class="rte-internet-block"><blockquote><div dir="ltr"><div dir="ltr"><br></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Tue, Jun 22, 2021 at 11:56 AM Danny Sauer <<a spellcheck="false"bbg-destination="mailto:rte:bind" class="" href="mailto:danny.sauer@konghq.com" data-destination="mailto:rte:bind">danny.sauer@konghq.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">You can certainly run multiple instances of the content server. It just needs a connection to the database and access to the storage.</div></blockquote><div>Agreed, you could deploy additional content servers and have your nginx/apache load balance them.</div><div> <br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div><br></div><div>Have you tuned the number of worker processes in Gunicorn? It defaults to 1, but should almost certainly be increased for any sort of volume. <a spellcheck="false"bbg-destination="rte:bind" class="" href="https://docs.gunicorn.org/en/stable/settings.html#worker-processes"data-destination="rte:bind">https://docs.gunicorn.org/en/stable/settings.html#worker-processes</a></div></div></blockquote><div>Pulp changed the default gunicorn worker processes to 8 maybe a release or two ago. See the `pulp_content_workers` variable in the installer here <a spellcheck="false"bbg-destination="rte:bind" class="" href="https://pulp-installer.readthedocs.io/en/latest/roles/pulp_content/#role-variables"data-destination="rte:bind">https://pulp-installer.readthedocs.io/en/latest/roles/pulp_content/#role-variables</a></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div><br></div><div>There are several moving pieces, but that's really all I had to touch here.</div><div><br></div><div>--Danny</div></div></blockquote><div>With pulpcore==3.14 there is a significant performance improvement being reviewed now <a spellcheck="false"bbg-destination="rte:bind" class="" href="https://pulp.plan.io/issues/8805" data-destination="rte:bind">https://pulp.plan.io/issues/8805</a> . In addition to resolving it with methods like ^, when 3.14 comes out (scheduled for June 29th) it would be great if you could report on if the improvements helped you.<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Tue, Jun 22, 2021 at 10:34 AM Bin Li (BLOOMBERG/ 120 PARK) <<a spellcheck="false"bbg-destination="mailto:rte:bind" class="" href="mailto:bli111@bloomberg.net" data-destination="mailto:rte:bind">bli111@bloomberg.net</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div style="font-size: small; font-family: "Courier New", Courier, "BB.FixedWidth"; white-space: pre-wrap;">We recently add more clients to use the pulp content server. The processes run out the file descriptor first. We then increased both nginx and pulp-content by creating a override.conf <div>/etc/systemd/system/pulpcore-content.service.d # cat override.conf </div><div>[Service]</div><div>LimitNOFILE=65536</div><div><br><div>and updated nginx.conf</div><div># Gunicorn docs suggest this value.</div><div>worker_processes 1;</div><div>events {</div><div> worker_connections 10000; # increase if you have lots of clients</div><div> accept_mutex off; # set to 'on' if nginx worker_processes > 1</div><div>}</div><div><br></div><div>worker_rlimit_nofile 20000;</div><div><br></div><div><br></div><div>Now we are keep getting this error.</div></div><div>2021/06/22 11:26:36 [error] 78373#0: *112823 upstream timed out (110: Connection timed out) while connecting to upstream, client:</div><div><br></div><div>It looks like pulp-content server cannot keep up with requests. Is there anything we could do to increase the performance of the content server?</div></div>_______________________________________________<br>Pulp-list mailing list<br><a spellcheck="false"bbg-destination="mailto:rte:bind" class="" href="mailto:Pulp-list@redhat.com" data-destination="mailto:rte:bind">Pulp-list@redhat.com</a><br><a spellcheck="false"bbg-destination="rte:bind" class="" href="https://listman.redhat.com/mailman/listinfo/pulp-list"data-destination="rte:bind">https://listman.redhat.com/mailman/listinfo/pulp-list</a></blockquote></div> _______________________________________________<br>Pulp-list mailing list<br><a spellcheck="false"bbg-destination="mailto:rte:bind" class="" href="mailto:Pulp-list@redhat.com" data-destination="mailto:rte:bind">Pulp-list@redhat.com</a><br><a spellcheck="false"bbg-destination="rte:bind" class="" href="https://listman.redhat.com/mailman/listinfo/pulp-list"data-destination="rte:bind">https://listman.redhat.com/mailman/listinfo/pulp-list</a></blockquote></div></div></blockquote><br></div></div></div></div></div></body></html>