[Pulp-list] pulp 3 list packages

[Mike DePaulo]

Hi Bin,

Did this work for you?

if so, it sounds like we should look into always having ansible-pulp / the
unit file pass a value to gunicorn (and make it configurable.)

-Mike

On Fri, Jan 31, 2020 at 10:27 AM Dennis Kliban <dkliban at redhat.com> wrote:

> This can be modified using the systemd unit file[0]. Gunicorn takes a -t
> argument. It defaults to 30 seconds.
>
> [0]
> https://github.com/pulp/ansible-pulp/blob/master/roles/pulp/templates/pulpcore-api.service.j2#L13
>
>
> On Fri, Jan 31, 2020 at 9:26 AM Bin Li (BLOOMBERG/ 120 PARK) <
> bli111 at bloomberg.net> wrote:
>
>> Thanks Brian.
>>
>> It looks like the worker timeout.
>>
>> Jan 31 09:19:30 pulpp-ob-581 gunicorn[147883]: [2020-01-31 09:19:30
>> -0500] [147883] [CRITICAL] WORKER TIMEOUT (pid:147898)
>> Jan 31 09:19:30 pulpp-ob-581 gunicorn[147883]: [2020-01-31 14:19:30
>> +0000] [147898] [INFO] Worker exiting (pid: 147898)
>> Jan 31 09:19:31 pulpp-ob-581 gunicorn[147883]: [2020-01-31 09:19:31
>> -0500] [161199] [INFO] Booting worker with pid: 161199
>>
>> How do I change the timeout setting in settings.py?
>>
>>
>> From: bmbouter at redhat.com At: 01/31/20 04:18:14
>> To: Bin Li (BLOOMBERG/ 120 PARK ) <bli111 at bloomberg.net>
>> Cc: pulp-list at redhat.com
>> Subject: Re: [Pulp-list] pulp 3 list packages
>>
>> What do the logs say about why the gunicorn process serving pulp-api is
>> dying? Would you want to file an issue https://pulp.plan.io/issues/new
>> so we can do some testing?
>>
>> As an aside, I recommend using paging when pulling so many items from an
>> API. You could decompose your large request to more, smaller requests like:
>>
>> http GET localhost/pulp/api/v3/content/rpm/packages/ offset=0
>> limit==10000
>> repository_version==/pulp/api/v3/repositories/rpm/rpm/2f46d319-7997-4e86-b159-8babee4aba19/versions/1/
>> --timeout=200
>> http GET localhost/pulp/api/v3/content/rpm/packages/ offset=10000
>> limit==10000
>> repository_version==/pulp/api/v3/repositories/rpm/rpm/2f46d319-7997-4e86-b159-8babee4aba19/versions/1/
>> --timeout=200
>>
>> What's interesting about more, smaller requests is you can likely get the
>> data out of Pulp a lot faster since you can engage more gunicorn processes
>> in parallel. Conceptually one large query is attractive though, so maybe we
>> could improve that if you file it.
>>
>> Another idea is to limit which fields are being returned to get at the
>> data you need faster.
>>
>> All the best,
>> Brian
>>
>>
>>
>> On Thu, Jan 30, 2020 at 2:46 PM Bin Li (BLOOMBERG/ 120 PARK) <
>> bli111 at bloomberg.net> wrote:
>>
>>> The rhel 7 servers rpm repo has more than 26k packages. I got an "502
>>> Bad Gateway" error if I tried to list all of them
>>>
>>> http GET localhost/pulp/api/v3/content/rpm/packages/ limit==20000
>>> repository_version==/pulp/api/v3/repositories/rpm/rpm/2f46d319-7997-4e86-b159-8babee4aba19/versions/1/
>>> --timeout=200
>>>
>>> What could cause this? Is there a fix?
>>>
>>> Thanks
>>> _______________________________________________
>>> Pulp-list mailing list
>>> Pulp-list at redhat.com
>>> https://www.redhat.com/mailman/listinfo/pulp-list
>>
>>
>> _______________________________________________
>> Pulp-list mailing list
>> Pulp-list at redhat.com
>> https://www.redhat.com/mailman/listinfo/pulp-list
>
> _______________________________________________
> Pulp-list mailing list
> Pulp-list at redhat.com
> https://www.redhat.com/mailman/listinfo/pulp-list



-- 

Mike DePaulo

He / Him / His

Service Reliability Engineer, Pulp

Red Hat <https://www.redhat.com/>

IM: mikedep333

GPG: 51745404
<https://www.redhat.com/>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/pulp-list/attachments/20200204/a178918e/attachment.htm>