<div dir="ltr">Hi everyone,<div><br></div><div>I would like to share my findings when syncing with a large requirements.yml [1] on pulp_ansible 0.5.1<br>The requirements file contains: </div><div> - 723 collections</div><div> -  4922 collections versions</div><div><br></div><div>First, it failed to read MANIFEST.json [2] for <span style="color:rgb(0,0,0);white-space:pre-wrap">rshad.collection_demo collection [3]</span></div><div><span style="color:rgb(0,0,0);white-space:pre-wrap">After removing [3] from the requirements file, I started to get 520 [4] randomly.</span></div><div>From [4]: "This happens mainly when the server is unable to send a timely response due to a resource-intensive process".</div><div>I decided to comment the requests for docs-blob endpoint, I could sync more collections, but this wasn't enough. So in order to investigate, I added a retry for 520 status.</div><div>After doing that, I could find 2 other issues:<br>- tag name with more than 32 characters [5]</div><div>- Sync breaks when it is not possible to determine the highest collection version [6]</div><div><br></div><div><b>Total requests:</b></div><div>723 - collections endpoint e.g. <a href="https://galaxy.ansible.com/api/v2/collections/pulp/pulp_installer/">https://galaxy.ansible.com/api/v2/collections/pulp/pulp_installer/</a></div><div>723 - collections versions endpoint e.g  <a href="https://galaxy.ansible.com/api/v2/collections/pulp/pulp_installer/versions/">https://galaxy.ansible.com/api/v2/collections/pulp/pulp_installer/versions/</a><br></div><div>4922 - collection version endpoint e.g. <a href="https://galaxy.ansible.com/api/v2/collections/pulp/pulp_installer/versions/3.8.1/">https://galaxy.ansible.com/api/v2/collections/pulp/pulp_installer/versions/3.8.1/</a></div><div>4922 - docs-blob endpoint e.g. <a href="https://galaxy.ansible.com/api/v2/collections/pulp/pulp_installer/versions/3.8.1/docs-blob/">https://galaxy.ansible.com/api/v2/collections/pulp/pulp_installer/versions/3.8.1/docs-blob/</a></div><div>4922 - tar file endpoint e.g. <a href="https://galaxy.ansible.com/download/pulp-pulp_installer-3.8.1.tar.gz">https://galaxy.ansible.com/download/pulp-pulp_installer-3.8.1.tar.gz</a></div><div>__________________________________</div><div>2(723) + 3(4922) = 16212 requests</div><div><br></div><div>So we are doing 2(#collections) + 3(#collection versions)</div><div>For reducing docs-blob requests we filed [7] and [8]</div><div><br></div><div>[1] <a href="https://gist.githubusercontent.com/fao89/bace0360f7f1756cae026573b4732dd5/raw/8ee2401bc311053e02d6ff0168bc5cd8f8d1a220/requirements.yml">https://gist.githubusercontent.com/fao89/bace0360f7f1756cae026573b4732dd5/raw/8ee2401bc311053e02d6ff0168bc5cd8f8d1a220/requirements.yml</a></div><div>[2] <a href="https://pulp.plan.io/issues/5745">https://pulp.plan.io/issues/5745</a></div><div>[3] <a href="https://galaxy.ansible.com/api/v2/collections/rshad/collection_demo/">https://galaxy.ansible.com/api/v2/collections/rshad/collection_demo/</a></div><div>[4] <a href="https://bobcares.com/blog/tips-to-fix-cloudflare-520-error/">https://bobcares.com/blog/tips-to-fix-cloudflare-520-error/</a></div><div>[5] <a href="https://pulp.plan.io/issues/7827">https://pulp.plan.io/issues/7827</a></div><div>[6] <a href="https://pulp.plan.io/issues/7826">https://pulp.plan.io/issues/7826</a></div><div>[7] <a href="https://pulp.plan.io/issues/7830">https://pulp.plan.io/issues/7830</a></div><div>[8] <a href="https://pulp.plan.io/issues/7831">https://pulp.plan.io/issues/7831</a><br clear="all"><div><div dir="ltr" class="gmail_signature" data-smartmail="gmail_signature"><div dir="ltr"><div dir="ltr"><br>Best regards,</div><div dir="ltr"><span style="color:rgb(0,0,0);font-family:RedHatText,sans-serif;font-size:14px;font-weight:700;text-transform:capitalize">Fabricio</span><span style="color:rgb(0,0,0);font-family:RedHatText,sans-serif;font-size:14px;font-weight:700;text-transform:capitalize"> </span><span style="color:rgb(0,0,0);font-family:RedHatText,sans-serif;font-size:14px;font-weight:700;text-transform:capitalize">Aguiar</span><div>Software Engineer, Pulp Project</div><div><a href="https://www.redhat.com/" style="color:rgb(0,136,206);font-family:RedHatText,sans-serif;font-size:12px;margin:0px" target="_blank">Red Hat Brazil - Latam</a><br></div><div>+55 11 999652368</div><div><img src="https://marketing-outfit-prod-images.s3-us-west-2.amazonaws.com/f5445ae0c9ddafd5b2f1836854d7416a/Logo-RedHat-Email.png" width="96" height="22"></div></div></div></div></div></div></div>