[Pulp-list] pulp 3 list packages

[Brian Bouterse]

What do the logs say about why the gunicorn process serving pulp-api is
dying? Would you want to file an issue https://pulp.plan.io/issues/new so
we can do some testing?

As an aside, I recommend using paging when pulling so many items from an
API. You could decompose your large request to more, smaller requests like:

http GET localhost/pulp/api/v3/content/rpm/packages/ offset=0 limit==10000
repository_version==/pulp/api/v3/repositories/rpm/rpm/2f46d319-7997-4e86-b159-8babee4aba19/versions/1/
--timeout=200
http GET localhost/pulp/api/v3/content/rpm/packages/ offset=10000
limit==10000
repository_version==/pulp/api/v3/repositories/rpm/rpm/2f46d319-7997-4e86-b159-8babee4aba19/versions/1/
--timeout=200

What's interesting about more, smaller requests is you can likely get the
data out of Pulp a lot faster since you can engage more gunicorn processes
in parallel. Conceptually one large query is attractive though, so maybe we
could improve that if you file it.

Another idea is to limit which fields are being returned to get at the data
you need faster.

All the best,
Brian



On Thu, Jan 30, 2020 at 2:46 PM Bin Li (BLOOMBERG/ 120 PARK) <
bli111 at bloomberg.net> wrote:

> The rhel 7 servers rpm repo has more than 26k packages. I got an "502 Bad
> Gateway" error if I tried to list all of them
>
> http GET localhost/pulp/api/v3/content/rpm/packages/ limit==20000
> repository_version==/pulp/api/v3/repositories/rpm/rpm/2f46d319-7997-4e86-b159-8babee4aba19/versions/1/
> --timeout=200
>
> What could cause this? Is there a fix?
>
> Thanks
> _______________________________________________
> Pulp-list mailing list
> Pulp-list at redhat.com
> https://www.redhat.com/mailman/listinfo/pulp-list
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/pulp-list/attachments/20200131/73a63bbb/attachment.htm>