[Pulp-list] pulp 3 list packages

Tue Feb 4 19:52:58 UTC 2020

Thanks Mike. This works. Set the timeout to 120s. The query of all packages in rhel7 actually took less than a minute.  Default 30s seems a little too short.

From: mikedep333 at redhat.com At: 02/04/20 13:32:26To:  dkliban at redhat.com
Cc:  Bin Li (BLOOMBERG/ 120 PARK ) ,  pulp-list at redhat.com
Subject: Re: [Pulp-list] pulp 3 list packages

Hi Bin,

Did this work for you?

if so, it sounds like we should look into always having ansible-pulp / the unit file pass a value to gunicorn (and make it configurable.)

-Mike

On Fri, Jan 31, 2020 at 10:27 AM Dennis Kliban <dkliban at redhat.com> wrote:

This can be modified using the systemd unit file[0]. Gunicorn takes a -t argument. It defaults to 30 seconds. 

[0] https://github.com/pulp/ansible-pulp/blob/master/roles/pulp/templates/pulpcore-api.service.j2#L13

On Fri, Jan 31, 2020 at 9:26 AM Bin Li (BLOOMBERG/ 120 PARK) <bli111 at bloomberg.net> wrote:

Thanks Brian.

It looks like the worker timeout.

Jan 31 09:19:30 pulpp-ob-581 gunicorn[147883]: [2020-01-31 09:19:30 -0500] [147883] [CRITICAL] WORKER TIMEOUT (pid:147898)
Jan 31 09:19:30 pulpp-ob-581 gunicorn[147883]: [2020-01-31 14:19:30 +0000] [147898] [INFO] Worker exiting (pid: 147898)
Jan 31 09:19:31 pulpp-ob-581 gunicorn[147883]: [2020-01-31 09:19:31 -0500] [161199] [INFO] Booting worker with pid: 161199

How do I change the timeout setting in settings.py?

From: bmbouter at redhat.com At: 01/31/20 04:18:14To:  Bin Li (BLOOMBERG/ 120 PARK ) 
Cc:  pulp-list at redhat.com
Subject: Re: [Pulp-list] pulp 3 list packages

What do the logs say about why the gunicorn process serving pulp-api is dying? Would you want to file an issue https://pulp.plan.io/issues/new so we can do some testing?

As an aside, I recommend using paging when pulling so many items from an API. You could decompose your large request to more, smaller requests like:

http GET localhost/pulp/api/v3/content/rpm/packages/ offset=0 limit==10000 repository_version==/pulp/api/v3/repositories/rpm/rpm/2f46d319-7997-4e86-b159-8babee4aba19/versions/1/ --timeout=200
http GET localhost/pulp/api/v3/content/rpm/packages/ offset=10000 limit==10000 repository_version==/pulp/api/v3/repositories/rpm/rpm/2f46d319-7997-4e86-b159-8babee4aba19/versions/1/ --timeout=200

What's interesting about more, smaller requests is you can likely get the data out of Pulp a lot faster since you can engage more gunicorn processes in parallel. Conceptually one large query is attractive though, so maybe we could improve that if you file it.

Another idea is to limit which fields are being returned to get at the data you need faster.

All the best,
Brian

On Thu, Jan 30, 2020 at 2:46 PM Bin Li (BLOOMBERG/ 120 PARK) <bli111 at bloomberg.net> wrote:

The rhel 7 servers rpm repo has more than 26k packages. I got an "502 Bad Gateway" error if I tried to list all of them

http GET localhost/pulp/api/v3/content/rpm/packages/ limit==20000 repository_version==/pulp/api/v3/repositories/rpm/rpm/2f46d319-7997-4e86-b159-8babee4aba19/versions/1/ --timeout=200

What could cause this? Is there a fix?

Thanks_______________________________________________
Pulp-list mailing list
Pulp-list at redhat.com
https://www.redhat.com/mailman/listinfo/pulp-list

_______________________________________________
Pulp-list mailing list
Pulp-list at redhat.com
https://www.redhat.com/mailman/listinfo/pulp-list
 _______________________________________________
Pulp-list mailing list
Pulp-list at redhat.com
https://www.redhat.com/mailman/listinfo/pulp-list

-- 

Mike DePaulo
He / Him / His
Service Reliability Engineer, Pulp
Red Hat
IM: mikedep333
GPG: 51745404

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/pulp-list/attachments/20200204/8e07ec9c/attachment.htm>