[Pulp-dev] Port Pulp3 to use RQ

Dana Walker dawalker at redhat.com
Tue Mar 20 20:27:40 UTC 2018


Between your initial layout of the problems with Celery and answers to the
questions pointing out problems with RQ, I'm curious if we should consider
alternatives to these two big names?

Have you looked into Kuyruk or Dramatiq which both still support RabbitMQ
for instance?  See a list here:
https://www.fullstackpython.com/task-queues.html

--Dana

Dana Walker

Associate Software Engineer

Red Hat

<https://www.redhat.com>
<https://red.ht/sig>

On Tue, Mar 20, 2018 at 4:18 PM, Brian Bouterse <bbouters at redhat.com> wrote:

> Thank you for the questions. I put some responses inline.
>
> On Tue, Mar 20, 2018 at 4:03 PM, Austin Macdonald <austin at redhat.com>
> wrote:
>
>> Not being familiar with RQ, I have questions (but no opinion).
>>
>> Will we also be replacing RabbitMQ with Redis?
>>
> Yes this would replace RabbitMQ/Qpid with Redis. RQ only works with Redis.
>
>
>> Does anyone on the team have experience with RQ? In production?
>>
> I do not have production experience. I've run it locally to see if it runs
> as easy as they say it does. I also reviewed their code some.
>
>
>> How well does RQ scale?
>>
> I started up several workers on my machine and they processed some dummy
> tasks I dispatched well. Pulp has very low throughput needs, maybe a few
> tasks per minute per worker.
>
>
>> Is RQ's use of `pickle` a problem? https://pulp.plan.io/issues/23
>>
> It's not preferred, but I think it's ok to go out with that for Pulp3 at
> the start. At some point we could swap it for a json serializer which would
> be safer.
>
>
>> RQ doesn't work on Windows. Is that a problem? (jk)
>>
> Pulp could maybe run on Windows one day so this isn't ideal.
>
>
>>
>>
>> On Tue, Mar 20, 2018 at 3:35 PM, Brian Bouterse <bbouters at redhat.com>
>> wrote:
>>
>>> Motivation:
>>> 1. Celery causes many bugs and issues for Pulp2 and 3 users and there is
>>> no end in sight.
>>>
>>> 2. The Pulp core team spends a lot of effort fixing Celery bugs. It's
>>> often times just us doing it with little or no assistance from the upstream
>>> communities. It's across 4 projects: celery, kombu, billiard, and pyamqp.
>>>
>>> 3. Celery will never allow a coverage report to be generated when
>>> pulp-smash runs because Celery forked the multiprocessing library into
>>> something called billiard. This will limit Pulp forever.
>>>
>>
>>> 4. I don't want to work with Celery anymore and I think the other
>>> maintainers (@dalley, @daviddavis) may feel the same. It's an endless
>>> headache. Even basic things don't work in Celery regularly.
>>>
>>> Proposed change: Replace Pulp3's usage of Celery with RQ (
>>> http://python-rq.org/)
>>>
>>> We would keep the exact same design of a resource manager with n
>>> workers, each worker pulling it's work exclusively from a dedicated queue.
>>> I've looked into porting pulp3 to it and it's doable because all the same
>>> concepts are there. There are a few details to work out, but I wanted to
>>> start the "should we" discussion before we do all-out technical planning.
>>>
>>> When would we do this? I'm proposing soon. It doesn't need to block the
>>> beta, but soon would be good. I don't think users will care much except for
>>> their systemd files, but it is fundamental and important to pulp3 so we
>>> want to get it testing sooner.
>>>
>>> Ideas, comments, questions are welcome!
>>>
>>> Thanks,
>>> Brian
>>>
>>> _______________________________________________
>>> Pulp-dev mailing list
>>> Pulp-dev at redhat.com
>>> https://www.redhat.com/mailman/listinfo/pulp-dev
>>>
>>>
>>
>
> _______________________________________________
> Pulp-dev mailing list
> Pulp-dev at redhat.com
> https://www.redhat.com/mailman/listinfo/pulp-dev
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/pulp-dev/attachments/20180320/0fc7a5ee/attachment.htm>


More information about the Pulp-dev mailing list