<div dir="ltr"><div><div>Between your initial layout of the problems with Celery and answers to the questions pointing out problems with RQ, I'm curious if we should consider alternatives to these two big names?<br><br></div>Have you looked into Kuyruk or Dramatiq which both still support RabbitMQ for instance? See a list here: <a href="https://www.fullstackpython.com/task-queues.html">https://www.fullstackpython.com/task-queues.html</a><br><br></div>--Dana<br></div><div class="gmail_extra"><br clear="all"><div><div class="gmail_signature" data-smartmail="gmail_signature"><div dir="ltr"><div>
<p style="font-weight:bold;margin:0;padding:0;font-size:14px;text-transform:uppercase;margin-bottom:0"><span>Dana</span> <span>Walker</span></p>
<p style="font-weight:normal;font-size:10px;margin:0px 0px 4px;text-transform:uppercase"><span>Associate Software Engineer</span><span style="font-weight:normal;color:#aaa;margin:0"></span></p>
<p style="font-weight:normal;margin:0;font-size:10px;color:#999"><a style="color:#0088ce;font-size:10px;margin:0;text-decoration:none;font-family:'overpass',sans-serif" href="https://www.redhat.com" target="_blank">Red Hat <span><br><br></span></a></p>
<table border="0"><tbody><tr><td width="100px"><a href="https://red.ht/sig" target="_blank"> <img src="https://www.redhat.com/files/brand/email/sig-redhat.png" width="90" height="auto"></a> </td>
</tr></tbody></table>
</div></div></div></div>
<br><div class="gmail_quote">On Tue, Mar 20, 2018 at 4:18 PM, Brian Bouterse <span dir="ltr"><<a href="mailto:bbouters@redhat.com" target="_blank">bbouters@redhat.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">Thank you for the questions. I put some responses inline.<br><div class="gmail_extra"><br><div class="gmail_quote"><span class="">On Tue, Mar 20, 2018 at 4:03 PM, Austin Macdonald <span dir="ltr"><<a href="mailto:austin@redhat.com" target="_blank">austin@redhat.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">Not being familiar with RQ, I have questions (but no opinion).<div><br></div><div>Will we also be replacing RabbitMQ with Redis?</div></div></blockquote></span><div>Yes this would replace RabbitMQ/Qpid with Redis. RQ only works with Redis.<br> </div><span class=""><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div>Does anyone on the team have experience with RQ? In production?</div></div></blockquote></span><div>I do not have production experience. I've run it locally to see if it runs as easy as they say it does. I also reviewed their code some.<br> <br></div><span class=""><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div>How well does RQ scale?</div></div></blockquote></span><div>I started up several workers on my machine and they processed some dummy tasks I dispatched well. Pulp has very low throughput needs, maybe a few tasks per minute per worker.<br> <br></div><span class=""><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div>Is RQ's use of `pickle` a problem? <a href="https://pulp.plan.io/issues/23" target="_blank">https://pulp.plan.io/<wbr>issues/23</a></div></div></blockquote></span><div>It's not preferred, but I think it's ok to go out with that for Pulp3 at the start. At some point we could swap it for a json serializer which would be safer.<br></div><span class=""><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div>RQ doesn't work on Windows. Is that a problem? (jk)</div></div></blockquote></span><div>Pulp could maybe run on Windows one day so this isn't ideal.<br> <br></div><span class=""><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div><br></div><div class="gmail_extra"><br><div class="gmail_quote"><div><div class="m_8319539337385681874h5">On Tue, Mar 20, 2018 at 3:35 PM, Brian Bouterse <span dir="ltr"><<a href="mailto:bbouters@redhat.com" target="_blank">bbouters@redhat.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div><div>Motivation:<br>1. Celery causes many bugs and issues for Pulp2 and 3 users and there is no end in sight.<br><br></div><div>2. The Pulp core team spends a lot of effort fixing Celery bugs. It's often times just us doing it with little or no assistance from the upstream communities. It's across 4 projects: celery, kombu, billiard, and pyamqp.<br></div><div><br>3. Celery will never allow a coverage report to be generated when pulp-smash runs because Celery forked the multiprocessing library into something called billiard. This will limit Pulp forever. </div></div></div></blockquote></div></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div><div class="m_8319539337385681874h5"><div dir="ltr"><div><div></div></div><div><br>4. I don't want to work with Celery anymore and I think the other maintainers (@dalley, @daviddavis) may feel the same. It's an endless headache. Even basic things don't work in Celery regularly.<br></div><div><br></div><div>Proposed change: Replace Pulp3's usage of Celery with RQ (<a href="http://python-rq.org/" target="_blank">http://python-rq.org/</a>)<br><br></div><div>We would keep the exact same design of a resource manager with n workers, each worker pulling it's work exclusively from a dedicated queue. I've looked into porting pulp3 to it and it's doable because all the same concepts are there. There are a few details to work out, but I wanted to start the "should we" discussion before we do all-out technical planning.<br><br></div><div>When would we do this? I'm proposing soon. It doesn't need to block the beta, but soon would be good. I don't think users will care much except for their systemd files, but it is fundamental and important to pulp3 so we want to get it testing sooner.<br><br></div><div>Ideas, comments, questions are welcome!<br><br></div><div>Thanks,<br></div><div>Brian<br></div></div>
<br></div></div>______________________________<wbr>_________________<br>
Pulp-dev mailing list<br>
<a href="mailto:Pulp-dev@redhat.com" target="_blank">Pulp-dev@redhat.com</a><br>
<a href="https://www.redhat.com/mailman/listinfo/pulp-dev" rel="noreferrer" target="_blank">https://www.redhat.com/mailman<wbr>/listinfo/pulp-dev</a><br>
<br></blockquote></div><br></div></div>
</blockquote></span></div><br></div></div>
<br>______________________________<wbr>_________________<br>
Pulp-dev mailing list<br>
<a href="mailto:Pulp-dev@redhat.com">Pulp-dev@redhat.com</a><br>
<a href="https://www.redhat.com/mailman/listinfo/pulp-dev" rel="noreferrer" target="_blank">https://www.redhat.com/<wbr>mailman/listinfo/pulp-dev</a><br>
<br></blockquote></div><br></div>