[Pulp-list] pypi sync no longer working after pypi.org migration with pulp 2.14 and pulp-python-plugins 2.0.2-1

Wed Jul 25 17:58:11 UTC 2018

Dustin,

Thanks for the heads up. We're looking into it now and will have a fix soon.

David

On Wed, Jul 25, 2018 at 1:29 PM Dustin McNabb <dustin.mcnabb at gmail.com>
wrote:

> Apparently this essential pypi.org fix at the github link below was left
> out of the pulp-python-2.0.3-0.1 release for the pulp 2.16.4 beta. It is
> essential for pypi.org support as path is no longer an option and the url
> formation logic in models.py no longer applies. I have opened another issue
> at https://pulp.plan.io/issues/3884
>
>
> https://github.com/pulp/pulp_python/commit/79b46411a049e63fc24ba97a60b51fe0574b268c
>
> Dustin McNabb
>
>
>
> On Jul 25, 2018, at 7:32 AM, Brian Bouterse <bbouters at redhat.com> wrote:
>
> Thank you for investigating and reporting this. We are going to try to
> reproduce and patch the version of Celery we distribute with Pulp.
>
> On Wed, Jul 25, 2018 at 10:14 AM, Dustin McNabb <dustin.mcnabb at gmail.com>
> wrote:
>
>> Sure: https://pulp.plan.io/issues/3882
>>
>> Dustin McNabb
>>
>>
>>
>>
>> On Jul 25, 2018, at 6:55 AM, Bihan Zhang <bizhang at redhat.com> wrote:
>>
>> Dustin,
>> Would you mind opening a pulp issue with a description of the issue and
>> the patch? https://pulp.plan.io/issues/new
>> I'm not too familiar with the tasking system, and it would be good for
>> the issue to go through the regular triage process.
>>
>>
>> Cheers,
>> Bihan
>>
>> On Wed, Jul 25, 2018 at 8:18 AM, Dustin McNabb <dustin.mcnabb at gmail.com>
>> wrote:
>>
>>> Bihan -
>>>
>>> Thanks for the fast response. However, I’ve upgraded my lab from pulp
>>> 2.14 to this beta and it introduces a new problem by upgrading to celery
>>> 4.0.2-4. Specifically, we now have this issue
>>> https://github.com/celery/celery/issues/3620. It was fixed in celery
>>> 4.1. You need multiple tasks in the queue upon initial startup to reproduce
>>> this issue. You can also see more discussion about this issue here:
>>> https://github.com/celery/celery/pull/3752 and here
>>> https://github.com/celery/celery/issues/4109#issuecomment-314367686. I
>>> implemented this fix in my lab environment and it resolved my issue.
>>>
>>> # Problem with pulp_resource_manager service
>>> Jul 25 06:10:03 pulp-lab pulp[17698]: celery.worker:CRITICAL:
>>> (17698-58784)   File "/usr/lib64/python2.7/site-packages/billiard/pool.py",
>>> line 1486, in apply_async
>>> Jul 25 06:10:03 pulp-lab  pulp[17698]: celery.worker:CRITICAL:
>>> (17698-58784)     self._quick_put((TASK, (result._job, None, func, args,
>>> kwds)))
>>> Jul 25 06:10:03 pulp-lab pulp[17698]: celery.worker:CRITICAL:
>>> (17698-58784) TypeError: 'NoneType' object is not callable
>>>
>>> # Here is the fix that was ultimately implemented which moves the
>>> consumption step below the object registration in worker/loops.py.
>>> -    consumer.consume()
>>> -    obj.on_ready()
>>>      obj.controller.register_with_event_loop(hub)
>>>      obj.register_with_event_loop(hub)
>>> +    consumer.consume()
>>> +    obj.on_ready()
>>>
>>>
>>> Thanks
>>> Dustin McNabb
>>>
>>>
>>>
>>>
>>> On Jul 24, 2018, at 2:23 PM, Bihan Zhang <bizhang at redhat.com> wrote:
>>>
>>> Dustin,
>>>
>>> These fixes were implemented[0] and were just release in pulp 2.16.4
>>> beta and pulp_python 2.0.3 beta today.
>>> Here's our release schedule in case you want to keep an eye out on when
>>> this is going to go GA:
>>> https://pulp.plan.io/projects/pulp/wiki/2164_Release_Status
>>>
>>>
>>> [0] https://pulp.plan.io/issues/3578
>>>
>>> On Tue, Jul 24, 2018 at 4:50 PM, Dustin McNabb <dustin.mcnabb at gmail.com>
>>> wrote:
>>>
>>>> I have pulp-server 2.14.3-1 and pulp-python-plugins 2.0.2-1 and I can
>>>> no longer sync form python.pypi.org. I realize they switched to
>>>> pypi.org and files.pythonhosted.org, but the old site still redirects
>>>> requests. However, the URLs where pulp-python-plugins expects to find
>>>> packages no longer exist. Also, the pulp-python packages have not been
>>>> updated since 2017. Does this mean that we don’t have support for the new
>>>> pypi.org warehouse using pulp2? Or is there a legacy feed URL I can
>>>> use?
>>>>
>>>> Jul 24 20:33:55 pulp-test-server pulp:
>>>> nectar.downloaders.threaded:INFO: Download failed: Download of
>>>> https://pypi.org/simple/pypi/tinydb/json/ failed with code 404: Not
>>>> Found
>>>> Jul 24 20:33:55 pulp-test-server pulp:
>>>> requests.packages.urllib3.connectionpool:INFO: Starting new HTTPS
>>>> connection (37): pypi.org
>>>> Jul 24 20:33:55 pulp-test-server pulp:
>>>> nectar.downloaders.threaded:INFO: Download failed: Download of
>>>> https://pypi.org/simple/pypi/setuptools/json/ failed with code 404:
>>>> Not Found
>>>> Jul 24 20:33:55 pulp-test-server pulp:
>>>> requests.packages.urllib3.connectionpool:INFO: Starting new HTTPS
>>>> connection (38): pypi.org
>>>> Jul 24 20:33:55 pulp-test-server pulp:
>>>> nectar.downloaders.threaded:INFO: Download failed: Download of
>>>> https://pypi.org/simple/pypi/urllib3/json/ failed with code 404: Not
>>>> Found
>>>> Jul 24 20:33:55 pulp-test-server pulp:
>>>> requests.packages.urllib3.connectionpool:INFO: Starting new HTTPS
>>>> connection (39): pypi.org
>>>> Jul 24 20:33:55 pulp-test-server pulp:
>>>> nectar.downloaders.threaded:INFO: Download failed: Download of
>>>> https://pypi.org/simple/pypi/chardet/json/ failed with code 404: Not
>>>> Found
>>>> Jul 24 20:33:55 pulp-test-server pulp:
>>>> requests.packages.urllib3.connectionpool:INFO: Starting new HTTPS
>>>> connection (40): pypi.org
>>>> Jul 24 20:33:55 pulp-test-server pulp:
>>>> nectar.downloaders.threaded:INFO: Download failed: Download of
>>>> https://pypi.org/simple/pypi/passlib/json/ failed with code 404: Not
>>>> Found
>>>> Jul 24 20:33:55 pulp-test-server pulp:
>>>> requests.packages.urllib3.connectionpool:INFO: Starting new HTTPS
>>>> connection (41): pypi.org
>>>> Jul 24 20:33:55 pulp-test-server pulp:
>>>> nectar.downloaders.threaded:INFO: Download failed: Download of
>>>> https://pypi.org/simple/pypi/setuptools/json/ failed with code 404:
>>>> Not Found
>>>> Jul 24 20:33:55 pulp-test-server pulp:
>>>> requests.packages.urllib3.connectionpool:INFO: Starting new HTTPS
>>>> connection (42): pypi.org
>>>> Jul 24 20:33:55 pulp-test-server pulp:
>>>> nectar.downloaders.threaded:INFO: Download failed: Download of
>>>> https://pypi.org/simple/pypi/gunicorn/json/ failed with code 404: Not
>>>> Found
>>>> Jul 24 20:33:55 pulp-test-server pulp:
>>>> nectar.downloaders.threaded:INFO: Download failed: Download of
>>>> https://pypi.org/simple/pypi/idna/json/ failed with code 404: Not Found
>>>>
>>>> Thanks
>>>>
>>>> Dustin McNabb
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> _______________________________________________
>>>> Pulp-list mailing list
>>>> Pulp-list at redhat.com
>>>> https://www.redhat.com/mailman/listinfo/pulp-list
>>>>
>>>
>>>
>>>
>>
>>
>> _______________________________________________
>> Pulp-list mailing list
>> Pulp-list at redhat.com
>> https://www.redhat.com/mailman/listinfo/pulp-list
>>
>
>
> _______________________________________________
> Pulp-list mailing list
> Pulp-list at redhat.com
> https://www.redhat.com/mailman/listinfo/pulp-list
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/pulp-list/attachments/20180725/dbd6677f/attachment.htm>