<div dir="ltr"><div><div>Thanks for all the feedback and discussion. I'll try to give some responses.<br></div><div><br>I should have sent over this technical outline yesterday. I want to dig deeper into seeing if RQ has the features Pulp needs or not.<br><br></div><div>* RQ has a worker discovery mechanism built-in already in the <a href="https://github.com/rq/rq/blob/master/rq/worker.py#L534">heartbeat()</a> method. We can override this heartbeat method to have heartbeats run <a href="https://github.com/pulp/pulp/blob/3.0-dev/pulpcore/pulpcore/tasking/celery_app.py#L94-L114">our heartbeat code</a> also, the same way to write the worker records to the DB like they do today.<br></div><div>* We can hook the Task status transitions when tasks start, succeed, or fail with a custom worker subclass <a href="https://github.com/rq/rq/blob/master/rq/worker.py#L737">here</a>. This will handle task state transitions similar to how we do today with Celery. Custom worker objects are supported, see "Custom Worker classes" on <a href="http://python-rq.org/docs/workers/">this page</a>.<br></div><div>* We can cancel and/or delete a task (RQ calls a job) using these <a href="https://github.com/rq/rq/blob/master/rq/job.py#L522-L577">cancel() or delete() instance methods</a> or the helper function <a href="https://github.com/rq/rq/blob/master/rq/job.py#L58">cancel_job()</a> which does it by job ID. I'm not sure why this isn't clearly shown in the docs for RQ. This looks equivalent to what we do with Celery.<br></div><div>* We can have Task records get created when RQ jobs are dispatched using a custom Job object as well which is supported, see "Custom Job classes" on <a href="http://python-rq.org/docs/workers/">this page</a><br></div><div>* We can have <a href="https://github.com/pulp/pulp/blob/3.0-dev/pulpcore/pulpcore/tasking/util.py#L82">code that detects which task id it's running in</a> use this RQ method <a href="https://github.com/rq/rq/blob/master/rq/job.py#L75">that does the same</a>.<br></div><div>* Also RQ makes the key assumption that Pulp needs which is that each worker only processes 1 thing. From the RQ docs: "Each worker will process a single job at a time. Within a worker, there is no
concurrent processing going on."<br></div><div>* plugin writer's can make their RQ jobs just like they made Celery jobs. They work in an equivalent way.<br></div><div><br></div><div>I'm hoping to first determine if we think there are gaps. Some help on validating ^ would be good. If there are no gaps discovered, I estimate a prototype PR could be made within a day or two. I think it could be that easy.<br></div><br></div><div>In terms of what library to pick, what I like about RQ is that it seems active, healthy, and mature. It's over 5 year old with 124 committers, 44 releases, 111 open issues, 430 closed issues, and daily commits. I think TaskTiger is also a fine choice and very similar but isn't as old and established.<br><br>I have not looked into Kuyruk or Dramatiq because I want to get off of RabbitMQ altogether not just Celery. If we support RabbitMQ we also have to support Qpid which creates issues for users since they work pretty differently. I didn't state that before so it's probably good I state that too.<br><br></div><div>Please send more ideas, questions, and concerns!<br><br></div><div>-Brian<br></div><div><br></div><div><br></div><br></div><div class="gmail_extra"><br><div class="gmail_quote">On Wed, Mar 21, 2018 at 8:13 AM, Ina Panova <span dir="ltr"><<a href="mailto:ipanova@redhat.com" target="_blank">ipanova@redhat.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div><div><div>+1 what said dalley.<br><br></div>Whatever we'd decide to replace celery with, should not go before beta that's for sure.<br></div>I am +10000 to get rid of celery, but with something that would not have other limitations which would bring just different kind of pain. [0]<br></div>Let's keep searching and evaluating alternatives.<br><br>[0] <a href="https://www.youtube.com/watch?v=Qmhc7tZ6ElQ" target="_blank">https://www.youtube.com/watch?<wbr>v=Qmhc7tZ6ElQ</a><br></div><div class="gmail_extra"><br clear="all"><div><div class="m_-4551024805956888907gmail_signature" data-smartmail="gmail_signature"><div dir="ltr"><br><br>--------<br>Regards,<br><br>Ina Panova<br>Software Engineer| Pulp| Red Hat Inc.<br><br>"Do not go where the path may lead,<br> go instead where there is no path and leave a trail."<br></div></div></div><div><div class="h5">
<br><div class="gmail_quote">On Tue, Mar 20, 2018 at 9:52 PM, Daniel Alley <span dir="ltr"><<a href="mailto:dalley@redhat.com" target="_blank">dalley@redhat.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div><div>Another option is TaskTiger (<a href="https://github.com/closeio/tasktiger" target="_blank">https://github.com/closeio/ta<wbr>sktiger</a>) which really hooked me with their tagline.<br><br></div>But I really just don't see how we could pull this off responsibly in the next month (or even 3 months). Assuming the functionality gaps can be worked out, it then becomes a question of whether that amount of change would be acceptable in the interim period between betas.<br></div></div><div class="m_-4551024805956888907HOEnZb"><div class="m_-4551024805956888907h5"><div class="gmail_extra"><br><div class="gmail_quote">On Tue, Mar 20, 2018 at 4:39 PM, Daniel Alley <span dir="ltr"><<a href="mailto:dalley@redhat.com" target="_blank">dalley@redhat.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div>As Brian said, Celery has a lot of limitations and drawbacks, a lot of code complexity, and an upstream that is not terribly responsive. I, too, would love to see us move away from Celery at some point.<br><br></div><div>But having done a little bit of research over the last few hours since it was mentioned, I have some concerns about the gaps between Celery and RQ, and I don't think that changing Pulp to use RQ would be as trivial as we hope. <br><br></div><div>I'll start with the benefits of RQ, from what I've read so far.<br><br><ul><li>It has task prioritization that *actually works*, which would help resolve the issue where reserved resource work tasks get choked out by less important tasks like applicability. The officially recommended solution that Celery provides for this is... have dedicated workers for each priority level. Not ideal.</li><li>The documentation is pretty good, from what I can tell. The Celery documentation is usually OK but sometimes... lacking.</li><li>RQ is a lot more straightforwards and less complex to use, from what I can tell</li></ul><p>But, problems:</p><ul><li>RQ does not support revoking tasks. If you send the worker a SIGINT, it will finish the task and then stop processing new ones. If you send the worker SIGKILL, it will stop immediately, but I don't think it gracefully handles this circumstance.</li><ul><li>People have rolled their own revoke functionality, but we should really look at this.</li></ul><li>When a RQ task fails, it does not provide a mechanism to automatically run a piece of code. It puts the task on a "failed" queue and the python handle for it will have is_failed set to True. this means we would have to redesign how failed tasks are cleaned up</li><li>I have no idea what happens when RQ loses connection to Redis, I couldn't find that info anywhere. Celery (in theory, at least, reality is mushy) will try to reconnect to the broker.</li><li>I have no idea how well RQ deals with persistence<br></li></ul>Also... we have shaped large parts of our API around what Celery does. Undoing this would be very... nontrivial and I don't think it is possible before the beta date, and definitely not if we want to guarantee some level of stability.<br><br></div><div>I'll keep looking but as much as I despise working with Celery I don't think we can make this move without a lot more research to make sure these problems are solvable.<br></div></div><div class="gmail_extra"><br><div class="gmail_quote"><span>On Tue, Mar 20, 2018 at 4:03 PM, Austin Macdonald <span dir="ltr"><<a href="mailto:austin@redhat.com" target="_blank">austin@redhat.com</a>></span> wrote:<br></span><div><div class="m_-4551024805956888907m_3216760859568464737h5"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">Not being familiar with RQ, I have questions (but no opinion).<div><br></div><div>Will we also be replacing RabbitMQ with Redis?</div><div>Does anyone on the team have experience with RQ? In production?</div><div>How well does RQ scale?</div><div>Is RQ's use of `pickle` a problem? <a href="https://pulp.plan.io/issues/23" target="_blank">https://pulp.plan.io/<wbr>issues/23</a></div><div>RQ doesn't work on Windows. Is that a problem? (jk)</div><div><br></div><div class="gmail_extra"><br><div class="gmail_quote"><div><div class="m_-4551024805956888907m_3216760859568464737m_-4189850825409554847h5">On Tue, Mar 20, 2018 at 3:35 PM, Brian Bouterse <span dir="ltr"><<a href="mailto:bbouters@redhat.com" target="_blank">bbouters@redhat.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div><div>Motivation:<br>1. Celery causes many bugs and issues for Pulp2 and 3 users and there is no end in sight.<br><br></div><div>2. The Pulp core team spends a lot of effort fixing Celery bugs. It's often times just us doing it with little or no assistance from the upstream communities. It's across 4 projects: celery, kombu, billiard, and pyamqp.<br></div><div><br>3. Celery will never allow a coverage report to be generated when pulp-smash runs because Celery forked the multiprocessing library into something called billiard. This will limit Pulp forever. </div></div></div></blockquote></div></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div><div class="m_-4551024805956888907m_3216760859568464737m_-4189850825409554847h5"><div dir="ltr"><div><div></div></div><div><br>4. I don't want to work with Celery anymore and I think the other maintainers (@dalley, @daviddavis) may feel the same. It's an endless headache. Even basic things don't work in Celery regularly.<br></div><div><br></div><div>Proposed change: Replace Pulp3's usage of Celery with RQ (<a href="http://python-rq.org/" target="_blank">http://python-rq.org/</a>)<br><br></div><div>We would keep the exact same design of a resource manager with n workers, each worker pulling it's work exclusively from a dedicated queue. I've looked into porting pulp3 to it and it's doable because all the same concepts are there. There are a few details to work out, but I wanted to start the "should we" discussion before we do all-out technical planning.<br><br></div><div>When would we do this? I'm proposing soon. It doesn't need to block the beta, but soon would be good. I don't think users will care much except for their systemd files, but it is fundamental and important to pulp3 so we want to get it testing sooner.<br><br></div><div>Ideas, comments, questions are welcome!<br><br></div><div>Thanks,<br></div><div>Brian<br></div></div>
<br></div></div>______________________________<wbr>_________________<br>
Pulp-dev mailing list<br>
<a href="mailto:Pulp-dev@redhat.com" target="_blank">Pulp-dev@redhat.com</a><br>
<a href="https://www.redhat.com/mailman/listinfo/pulp-dev" rel="noreferrer" target="_blank">https://www.redhat.com/mailman<wbr>/listinfo/pulp-dev</a><br>
<br></blockquote></div><br></div></div>
<br>______________________________<wbr>_________________<br>
Pulp-dev mailing list<br>
<a href="mailto:Pulp-dev@redhat.com" target="_blank">Pulp-dev@redhat.com</a><br>
<a href="https://www.redhat.com/mailman/listinfo/pulp-dev" rel="noreferrer" target="_blank">https://www.redhat.com/mailman<wbr>/listinfo/pulp-dev</a><br>
<br></blockquote></div></div></div><br></div>
</blockquote></div><br></div>
</div></div><br>______________________________<wbr>_________________<br>
Pulp-dev mailing list<br>
<a href="mailto:Pulp-dev@redhat.com" target="_blank">Pulp-dev@redhat.com</a><br>
<a href="https://www.redhat.com/mailman/listinfo/pulp-dev" rel="noreferrer" target="_blank">https://www.redhat.com/mailman<wbr>/listinfo/pulp-dev</a><br>
<br></blockquote></div><br></div></div></div>
</blockquote></div><br></div>