<div dir="ltr"><div><div>Another option is TaskTiger (<a href="https://github.com/closeio/tasktiger">https://github.com/closeio/tasktiger</a>) which really hooked me with their tagline.<br><br></div>But I really just don't see how we could pull this off responsibly in the next month (or even 3 months). Assuming the functionality gaps can be worked out, it then becomes a question of whether that amount of change would be acceptable in the interim period between betas.<br></div></div><div class="gmail_extra"><br><div class="gmail_quote">On Tue, Mar 20, 2018 at 4:39 PM, Daniel Alley <span dir="ltr"><<a href="mailto:dalley@redhat.com" target="_blank">dalley@redhat.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div>As Brian said, Celery has a lot of limitations and drawbacks, a lot of code complexity, and an upstream that is not terribly responsive. I, too, would love to see us move away from Celery at some point.<br><br></div><div>But having done a little bit of research over the last few hours since it was mentioned, I have some concerns about the gaps between Celery and RQ, and I don't think that changing Pulp to use RQ would be as trivial as we hope. <br><br></div><div>I'll start with the benefits of RQ, from what I've read so far.<br><br><ul><li>It has task prioritization that *actually works*, which would help resolve the issue where reserved resource work tasks get choked out by less important tasks like applicability. The officially recommended solution that Celery provides for this is... have dedicated workers for each priority level. Not ideal.</li><li>The documentation is pretty good, from what I can tell. The Celery documentation is usually OK but sometimes... lacking.</li><li>RQ is a lot more straightforwards and less complex to use, from what I can tell</li></ul><p>But, problems:</p><ul><li>RQ does not support revoking tasks. If you send the worker a SIGINT, it will finish the task and then stop processing new ones. If you send the worker SIGKILL, it will stop immediately, but I don't think it gracefully handles this circumstance.</li><ul><li>People have rolled their own revoke functionality, but we should really look at this.</li></ul><li>When a RQ task fails, it does not provide a mechanism to automatically run a piece of code. It puts the task on a "failed" queue and the python handle for it will have is_failed set to True. this means we would have to redesign how failed tasks are cleaned up</li><li>I have no idea what happens when RQ loses connection to Redis, I couldn't find that info anywhere. Celery (in theory, at least, reality is mushy) will try to reconnect to the broker.</li><li>I have no idea how well RQ deals with persistence<br></li></ul>Also... we have shaped large parts of our API around what Celery does. Undoing this would be very... nontrivial and I don't think it is possible before the beta date, and definitely not if we want to guarantee some level of stability.<br><br></div><div>I'll keep looking but as much as I despise working with Celery I don't think we can make this move without a lot more research to make sure these problems are solvable.<br></div></div><div class="gmail_extra"><br><div class="gmail_quote"><span class="">On Tue, Mar 20, 2018 at 4:03 PM, Austin Macdonald <span dir="ltr"><<a href="mailto:austin@redhat.com" target="_blank">austin@redhat.com</a>></span> wrote:<br></span><div><div class="h5"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">Not being familiar with RQ, I have questions (but no opinion).<div><br></div><div>Will we also be replacing RabbitMQ with Redis?</div><div>Does anyone on the team have experience with RQ? In production?</div><div>How well does RQ scale?</div><div>Is RQ's use of `pickle` a problem? <a href="https://pulp.plan.io/issues/23" target="_blank">https://pulp.plan.io/<wbr>issues/23</a></div><div>RQ doesn't work on Windows. Is that a problem? (jk)</div><div><br></div><div class="gmail_extra"><br><div class="gmail_quote"><div><div class="m_-4189850825409554847h5">On Tue, Mar 20, 2018 at 3:35 PM, Brian Bouterse <span dir="ltr"><<a href="mailto:bbouters@redhat.com" target="_blank">bbouters@redhat.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div><div>Motivation:<br>1. Celery causes many bugs and issues for Pulp2 and 3 users and there is no end in sight.<br><br></div><div>2. The Pulp core team spends a lot of effort fixing Celery bugs. It's often times just us doing it with little or no assistance from the upstream communities. It's across 4 projects: celery, kombu, billiard, and pyamqp.<br></div><div><br>3. Celery will never allow a coverage report to be generated when pulp-smash runs because Celery forked the multiprocessing library into something called billiard. This will limit Pulp forever. </div></div></div></blockquote></div></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div><div class="m_-4189850825409554847h5"><div dir="ltr"><div><div></div></div><div><br>4. I don't want to work with Celery anymore and I think the other maintainers (@dalley, @daviddavis) may feel the same. It's an endless headache. Even basic things don't work in Celery regularly.<br></div><div><br></div><div>Proposed change: Replace Pulp3's usage of Celery with RQ (<a href="http://python-rq.org/" target="_blank">http://python-rq.org/</a>)<br><br></div><div>We would keep the exact same design of a resource manager with n workers, each worker pulling it's work exclusively from a dedicated queue. I've looked into porting pulp3 to it and it's doable because all the same concepts are there. There are a few details to work out, but I wanted to start the "should we" discussion before we do all-out technical planning.<br><br></div><div>When would we do this? I'm proposing soon. It doesn't need to block the beta, but soon would be good. I don't think users will care much except for their systemd files, but it is fundamental and important to pulp3 so we want to get it testing sooner.<br><br></div><div>Ideas, comments, questions are welcome!<br><br></div><div>Thanks,<br></div><div>Brian<br></div></div>
<br></div></div>______________________________<wbr>_________________<br>
Pulp-dev mailing list<br>
<a href="mailto:Pulp-dev@redhat.com" target="_blank">Pulp-dev@redhat.com</a><br>
<a href="https://www.redhat.com/mailman/listinfo/pulp-dev" rel="noreferrer" target="_blank">https://www.redhat.com/mailman<wbr>/listinfo/pulp-dev</a><br>
<br></blockquote></div><br></div></div>
<br>______________________________<wbr>_________________<br>
Pulp-dev mailing list<br>
<a href="mailto:Pulp-dev@redhat.com" target="_blank">Pulp-dev@redhat.com</a><br>
<a href="https://www.redhat.com/mailman/listinfo/pulp-dev" rel="noreferrer" target="_blank">https://www.redhat.com/mailman<wbr>/listinfo/pulp-dev</a><br>
<br></blockquote></div></div></div><br></div>
</blockquote></div><br></div>