[Pulp-list] pypi sync no longer working after pypi.org migration with pulp 2.14 and pulp-python-plugins 2.0.2-1

[Bihan Zhang]

Dustin,
Would you mind opening a pulp issue with a description of the issue and the
patch? https://pulp.plan.io/issues/new
I'm not too familiar with the tasking system, and it would be good for the
issue to go through the regular triage process.


Cheers,
Bihan

On Wed, Jul 25, 2018 at 8:18 AM, Dustin McNabb <dustin.mcnabb at gmail.com>
wrote:

> Bihan -
>
> Thanks for the fast response. However, I’ve upgraded my lab from pulp 2.14
> to this beta and it introduces a new problem by upgrading to celery
> 4.0.2-4. Specifically, we now have this issue https://github.com/
> celery/celery/issues/3620. It was fixed in celery 4.1. You need multiple
> tasks in the queue upon initial startup to reproduce this issue. You can
> also see more discussion about this issue here: https://github.com/
> celery/celery/pull/3752 and here https://github.com/celery/
> celery/issues/4109#issuecomment-314367686. I implemented this fix in my
> lab environment and it resolved my issue.
>
> # Problem with pulp_resource_manager service
> Jul 25 06:10:03 pulp-lab pulp[17698]: celery.worker:CRITICAL:
> (17698-58784)   File "/usr/lib64/python2.7/site-packages/billiard/pool.py",
> line 1486, in apply_async
> Jul 25 06:10:03 pulp-lab  pulp[17698]: celery.worker:CRITICAL:
> (17698-58784)     self._quick_put((TASK, (result._job, None, func, args,
> kwds)))
> Jul 25 06:10:03 pulp-lab pulp[17698]: celery.worker:CRITICAL:
> (17698-58784) TypeError: 'NoneType' object is not callable
>
> # Here is the fix that was ultimately implemented which moves the
> consumption step below the object registration in worker/loops.py.
> -    consumer.consume()
> -    obj.on_ready()
>      obj.controller.register_with_event_loop(hub)
>      obj.register_with_event_loop(hub)
> +    consumer.consume()
> +    obj.on_ready()
>
>
> Thanks
> Dustin McNabb
>
>
>
>
> On Jul 24, 2018, at 2:23 PM, Bihan Zhang <bizhang at redhat.com> wrote:
>
> Dustin,
>
> These fixes were implemented[0] and were just release in pulp 2.16.4 beta
> and pulp_python 2.0.3 beta today.
> Here's our release schedule in case you want to keep an eye out on when
> this is going to go GA: https://pulp.plan.io/projects/pulp/wiki/2164_
> Release_Status
>
>
> [0] https://pulp.plan.io/issues/3578
>
> On Tue, Jul 24, 2018 at 4:50 PM, Dustin McNabb <dustin.mcnabb at gmail.com>
> wrote:
>
>> I have pulp-server 2.14.3-1 and pulp-python-plugins 2.0.2-1 and I can no
>> longer sync form python.pypi.org. I realize they switched to pypi.org and
>> files.pythonhosted.org, but the old site still redirects requests.
>> However, the URLs where pulp-python-plugins expects to find packages no
>> longer exist. Also, the pulp-python packages have not been updated since
>> 2017. Does this mean that we don’t have support for the new pypi.org warehouse
>> using pulp2? Or is there a legacy feed URL I can use?
>>
>> Jul 24 20:33:55 pulp-test-server pulp: nectar.downloaders.threaded:INFO:
>> Download failed: Download of https://pypi.org/simple/pypi/tinydb/json/
>> failed with code 404: Not Found
>> Jul 24 20:33:55 pulp-test-server pulp: requests.packages.urllib3.connectionpool:INFO:
>> Starting new HTTPS connection (37): pypi.org
>> Jul 24 20:33:55 pulp-test-server pulp: nectar.downloaders.threaded:INFO:
>> Download failed: Download of https://pypi.org/simple/pypi/s
>> etuptools/json/ failed with code 404: Not Found
>> Jul 24 20:33:55 pulp-test-server pulp: requests.packages.urllib3.connectionpool:INFO:
>> Starting new HTTPS connection (38): pypi.org
>> Jul 24 20:33:55 pulp-test-server pulp: nectar.downloaders.threaded:INFO:
>> Download failed: Download of https://pypi.org/simple/pypi/urllib3/json/
>> failed with code 404: Not Found
>> Jul 24 20:33:55 pulp-test-server pulp: requests.packages.urllib3.connectionpool:INFO:
>> Starting new HTTPS connection (39): pypi.org
>> Jul 24 20:33:55 pulp-test-server pulp: nectar.downloaders.threaded:INFO:
>> Download failed: Download of https://pypi.org/simple/pypi/chardet/json/
>> failed with code 404: Not Found
>> Jul 24 20:33:55 pulp-test-server pulp: requests.packages.urllib3.connectionpool:INFO:
>> Starting new HTTPS connection (40): pypi.org
>> Jul 24 20:33:55 pulp-test-server pulp: nectar.downloaders.threaded:INFO:
>> Download failed: Download of https://pypi.org/simple/pypi/passlib/json/
>> failed with code 404: Not Found
>> Jul 24 20:33:55 pulp-test-server pulp: requests.packages.urllib3.connectionpool:INFO:
>> Starting new HTTPS connection (41): pypi.org
>> Jul 24 20:33:55 pulp-test-server pulp: nectar.downloaders.threaded:INFO:
>> Download failed: Download of https://pypi.org/simple/pypi/s
>> etuptools/json/ failed with code 404: Not Found
>> Jul 24 20:33:55 pulp-test-server pulp: requests.packages.urllib3.connectionpool:INFO:
>> Starting new HTTPS connection (42): pypi.org
>> Jul 24 20:33:55 pulp-test-server pulp: nectar.downloaders.threaded:INFO:
>> Download failed: Download of https://pypi.org/simple/pypi/gunicorn/json/
>> failed with code 404: Not Found
>> Jul 24 20:33:55 pulp-test-server pulp: nectar.downloaders.threaded:INFO:
>> Download failed: Download of https://pypi.org/simple/pypi/idna/json/
>> failed with code 404: Not Found
>>
>> Thanks
>>
>> Dustin McNabb
>>
>>
>>
>>
>>
>> _______________________________________________
>> Pulp-list mailing list
>> Pulp-list at redhat.com
>> https://www.redhat.com/mailman/listinfo/pulp-list
>>
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/pulp-list/attachments/20180725/e244f8cc/attachment.htm>