[Pulp-dev] Port Pulp3 to use RQ

[Austin Macdonald]

Not being familiar with RQ, I have questions (but no opinion).

Will we also be replacing RabbitMQ with Redis?
Does anyone on the team have experience with RQ? In production?
How well does RQ scale?
Is RQ's use of `pickle` a problem? https://pulp.plan.io/issues/23
RQ doesn't work on Windows. Is that a problem? (jk)


On Tue, Mar 20, 2018 at 3:35 PM, Brian Bouterse <bbouters at redhat.com> wrote:

> Motivation:
> 1. Celery causes many bugs and issues for Pulp2 and 3 users and there is
> no end in sight.
>
> 2. The Pulp core team spends a lot of effort fixing Celery bugs. It's
> often times just us doing it with little or no assistance from the upstream
> communities. It's across 4 projects: celery, kombu, billiard, and pyamqp.
>
> 3. Celery will never allow a coverage report to be generated when
> pulp-smash runs because Celery forked the multiprocessing library into
> something called billiard. This will limit Pulp forever.
>

> 4. I don't want to work with Celery anymore and I think the other
> maintainers (@dalley, @daviddavis) may feel the same. It's an endless
> headache. Even basic things don't work in Celery regularly.
>
> Proposed change: Replace Pulp3's usage of Celery with RQ (
> http://python-rq.org/)
>
> We would keep the exact same design of a resource manager with n workers,
> each worker pulling it's work exclusively from a dedicated queue. I've
> looked into porting pulp3 to it and it's doable because all the same
> concepts are there. There are a few details to work out, but I wanted to
> start the "should we" discussion before we do all-out technical planning.
>
> When would we do this? I'm proposing soon. It doesn't need to block the
> beta, but soon would be good. I don't think users will care much except for
> their systemd files, but it is fundamental and important to pulp3 so we
> want to get it testing sooner.
>
> Ideas, comments, questions are welcome!
>
> Thanks,
> Brian
>
> _______________________________________________
> Pulp-dev mailing list
> Pulp-dev at redhat.com
> https://www.redhat.com/mailman/listinfo/pulp-dev
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/pulp-dev/attachments/20180320/b19ba294/attachment.htm>