[Pulp-dev] Port Pulp3 to use RQ

[Brian Bouterse]

Motivation:
1. Celery causes many bugs and issues for Pulp2 and 3 users and there is no
end in sight.

2. The Pulp core team spends a lot of effort fixing Celery bugs. It's often
times just us doing it with little or no assistance from the upstream
communities. It's across 4 projects: celery, kombu, billiard, and pyamqp.

3. Celery will never allow a coverage report to be generated when
pulp-smash runs because Celery forked the multiprocessing library into
something called billiard. This will limit Pulp forever.

4. I don't want to work with Celery anymore and I think the other
maintainers (@dalley, @daviddavis) may feel the same. It's an endless
headache. Even basic things don't work in Celery regularly.

Proposed change: Replace Pulp3's usage of Celery with RQ (
http://python-rq.org/)

We would keep the exact same design of a resource manager with n workers,
each worker pulling it's work exclusively from a dedicated queue. I've
looked into porting pulp3 to it and it's doable because all the same
concepts are there. There are a few details to work out, but I wanted to
start the "should we" discussion before we do all-out technical planning.

When would we do this? I'm proposing soon. It doesn't need to block the
beta, but soon would be good. I don't think users will care much except for
their systemd files, but it is fundamental and important to pulp3 so we
want to get it testing sooner.

Ideas, comments, questions are welcome!

Thanks,
Brian
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/pulp-dev/attachments/20180320/f918395c/attachment.htm>