[Pulp-dev] Port Pulp3 to use RQ

[Brian Bouterse]

Thank you for the questions. I put some responses inline.

On Tue, Mar 20, 2018 at 4:03 PM, Austin Macdonald <austin at redhat.com> wrote:

> Not being familiar with RQ, I have questions (but no opinion).
>
> Will we also be replacing RabbitMQ with Redis?
>
Yes this would replace RabbitMQ/Qpid with Redis. RQ only works with Redis.


> Does anyone on the team have experience with RQ? In production?
>
I do not have production experience. I've run it locally to see if it runs
as easy as they say it does. I also reviewed their code some.


> How well does RQ scale?
>
I started up several workers on my machine and they processed some dummy
tasks I dispatched well. Pulp has very low throughput needs, maybe a few
tasks per minute per worker.


> Is RQ's use of `pickle` a problem? https://pulp.plan.io/issues/23
>
It's not preferred, but I think it's ok to go out with that for Pulp3 at
the start. At some point we could swap it for a json serializer which would
be safer.


> RQ doesn't work on Windows. Is that a problem? (jk)
>
Pulp could maybe run on Windows one day so this isn't ideal.


>
>
> On Tue, Mar 20, 2018 at 3:35 PM, Brian Bouterse <bbouters at redhat.com>
> wrote:
>
>> Motivation:
>> 1. Celery causes many bugs and issues for Pulp2 and 3 users and there is
>> no end in sight.
>>
>> 2. The Pulp core team spends a lot of effort fixing Celery bugs. It's
>> often times just us doing it with little or no assistance from the upstream
>> communities. It's across 4 projects: celery, kombu, billiard, and pyamqp.
>>
>> 3. Celery will never allow a coverage report to be generated when
>> pulp-smash runs because Celery forked the multiprocessing library into
>> something called billiard. This will limit Pulp forever.
>>
>
>> 4. I don't want to work with Celery anymore and I think the other
>> maintainers (@dalley, @daviddavis) may feel the same. It's an endless
>> headache. Even basic things don't work in Celery regularly.
>>
>> Proposed change: Replace Pulp3's usage of Celery with RQ (
>> http://python-rq.org/)
>>
>> We would keep the exact same design of a resource manager with n workers,
>> each worker pulling it's work exclusively from a dedicated queue. I've
>> looked into porting pulp3 to it and it's doable because all the same
>> concepts are there. There are a few details to work out, but I wanted to
>> start the "should we" discussion before we do all-out technical planning.
>>
>> When would we do this? I'm proposing soon. It doesn't need to block the
>> beta, but soon would be good. I don't think users will care much except for
>> their systemd files, but it is fundamental and important to pulp3 so we
>> want to get it testing sooner.
>>
>> Ideas, comments, questions are welcome!
>>
>> Thanks,
>> Brian
>>
>> _______________________________________________
>> Pulp-dev mailing list
>> Pulp-dev at redhat.com
>> https://www.redhat.com/mailman/listinfo/pulp-dev
>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/pulp-dev/attachments/20180320/bc12ee58/attachment.htm>