[Pulp-dev] Port Pulp3 to use RQ

Brian Bouterse bbouters at redhat.com
Tue Mar 20 20:18:47 UTC 2018

Thank you for the questions. I put some responses inline.

On Tue, Mar 20, 2018 at 4:03 PM, Austin Macdonald <austin at redhat.com> wrote:

> Not being familiar with RQ, I have questions (but no opinion).
> Will we also be replacing RabbitMQ with Redis?
Yes this would replace RabbitMQ/Qpid with Redis. RQ only works with Redis.

> Does anyone on the team have experience with RQ? In production?
I do not have production experience. I've run it locally to see if it runs
as easy as they say it does. I also reviewed their code some.

> How well does RQ scale?
I started up several workers on my machine and they processed some dummy
tasks I dispatched well. Pulp has very low throughput needs, maybe a few
tasks per minute per worker.

> Is RQ's use of `pickle` a problem? https://pulp.plan.io/issues/23
It's not preferred, but I think it's ok to go out with that for Pulp3 at
the start. At some point we could swap it for a json serializer which would
be safer.

> RQ doesn't work on Windows. Is that a problem? (jk)
Pulp could maybe run on Windows one day so this isn't ideal.

> On Tue, Mar 20, 2018 at 3:35 PM, Brian Bouterse <bbouters at redhat.com>
> wrote:
>> Motivation:
>> 1. Celery causes many bugs and issues for Pulp2 and 3 users and there is
>> no end in sight.
>> 2. The Pulp core team spends a lot of effort fixing Celery bugs. It's
>> often times just us doing it with little or no assistance from the upstream
>> communities. It's across 4 projects: celery, kombu, billiard, and pyamqp.
>> 3. Celery will never allow a coverage report to be generated when
>> pulp-smash runs because Celery forked the multiprocessing library into
>> something called billiard. This will limit Pulp forever.
>> 4. I don't want to work with Celery anymore and I think the other
>> maintainers (@dalley, @daviddavis) may feel the same. It's an endless
>> headache. Even basic things don't work in Celery regularly.
>> Proposed change: Replace Pulp3's usage of Celery with RQ (
>> http://python-rq.org/)
>> We would keep the exact same design of a resource manager with n workers,
>> each worker pulling it's work exclusively from a dedicated queue. I've
>> looked into porting pulp3 to it and it's doable because all the same
>> concepts are there. There are a few details to work out, but I wanted to
>> start the "should we" discussion before we do all-out technical planning.
>> When would we do this? I'm proposing soon. It doesn't need to block the
>> beta, but soon would be good. I don't think users will care much except for
>> their systemd files, but it is fundamental and important to pulp3 so we
>> want to get it testing sooner.
>> Ideas, comments, questions are welcome!
>> Thanks,
>> Brian
>> _______________________________________________
>> Pulp-dev mailing list
>> Pulp-dev at redhat.com
>> https://www.redhat.com/mailman/listinfo/pulp-dev
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/pulp-dev/attachments/20180320/bc12ee58/attachment.htm>

More information about the Pulp-dev mailing list