<div dir="ltr"><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><span class="gmail-im"><div></div></span><div>Are there any benefits to
improving RQ vs the invented here method? I'm just curious about the
cost of maintaining a tasking system versus being part of a community
built one. This feels like the kind of problem that many other
applications should have in the Python world -- or are there elements of
Pulp's deployment architecture that make it unique here?</div></blockquote><div><br></div><div>This shouldn't be viewed through an "invented here" lens, because most of what the proposal does would actually reduce our dependence on "invented here". <br></div><div><br></div><div>Basically there is a fundamental problem with having task state split between the database, and some external service (RQ), which is incredibly difficult to keep consistent. There is a lot of existing complexity around resource locking that would completely go away if we just switch to keeping the "task queue" in the database, and using normal transactions and row/table locks rather than separate "lock objects" in additional tables, etc.<br></div><div><br></div><div>The idea is, we already have all of this information about the tasks in the database (reporting what happened and so on), and if we just store 2 extra pieces of information - the function to execute, and the parameters to execute with - we will essentially have the "front half" of a task queue, that we can much more easily keep in a consistent state with everything else.<br></div><div><br></div><div>This actually is a fairly common problem - it's called the outbox pattern: <a href="https://microservices.io/patterns/data/transactional-outbox.html">https://microservices.io/patterns/data/transactional-outbox.html</a></div><div><br></div><div>Regarding the "back half", which is dealing with the actual process of spawning the process, I'm less certain. Maybe Matthias can explain what the plan is, there. IMO even if we continued using RQ for that portion (part 3 in the diagram in the link), the change to the "front half" (everything up to and including the pulp resource manager") makes a lot of sense and would be a significant net reduction in complexity.<br></div><div><br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div><div>This is sort of an aside to this general change. Are Pulp tasks cleaned up from the database today?</div></div></blockquote><div><br></div><div>They aren't. We don't clean up anything automatically, cleanup is user-driven. <br></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Tue, Apr 13, 2021 at 11:18 AM Eric Helms <<a href="mailto:ehelms@redhat.com">ehelms@redhat.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div dir="ltr"><br></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Thu, Apr 8, 2021 at 5:24 PM Daniel Alley <<a href="mailto:dalley@redhat.com" target="_blank">dalley@redhat.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div>Eric,</div><div><br></div><div>* The idea is to move away from RQ entirely. RQ is fine (and vastly better than Celery IMO), but managing task state across both 1) the database and 2) a separate, external registry is still problematic. If all of the information can simply be kept in the database, then it will be much easier to maintain consistent state.<br></div></div></blockquote><div><br></div><div>Are there any benefits to improving RQ vs the invented here method? I'm just curious about the cost of maintaining a tasking system versus being part of a community built one. This feels like the kind of problem that many other applications should have in the Python world -- or are there elements of Pulp's deployment architecture that make it unique here?</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div></div><div>* <i>Maybe</i>. We're considering using Redis as a cache to improve content serving performance (after all, caching is one of the primary uses of Redis). If we do, then Redis would remain in the architecture, but it could potentially be an optional component and would be easier to remove at some point in the future.<br></div><div><div>* We'd just be adding a small amount of information to each task record, and it wouldn't prevent cleanup later. </div></div></div></blockquote><div><br></div><div>This is sort of an aside to this general change. Are Pulp tasks cleaned up from the database today?</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div><div><br></div><div><br></div></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Thu, Apr 8, 2021 at 4:42 PM Eric Helms <<a href="mailto:ehelms@redhat.com" target="_blank">ehelms@redhat.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">A few initial questions that get a bit into the stack but will help the Foreman project think on the proposed changes:<div><br></div><div> * Does this move away from RQ entirely or just RQ workers?</div><div> * Do the new workers remove Pulp 3's use of Redis all together?</div><div> * Will using the database result in any additional build up of tasking information that can impact performance over time? (Or does all task data get cleaned up eventually?)</div><div><br></div><div>Thanks for sending this along early.</div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Fri, Apr 2, 2021 at 4:43 PM Brian Bouterse <<a href="mailto:bmbouter@redhat.com" target="_blank">bmbouter@redhat.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div>FYI, @mdellweg and I have been collaborating on the tasking system changes. This email is to share some info to transition the work to @mdellweg while I'm out. With the new-style disabled by default I am hoping it can go into 3.13.<br></div><div><br></div><div>## The PoC and ticket info</div><div><br></div><div>The PoC is basically functional, but it's a PoC: <a href="https://github.com/pulp/pulpcore/pull/1222/" target="_blank">https://github.com/pulp/pulpcore/pull/1222/</a></div><div><br></div><div><div>* The epic is being tracked here which recaps why we're doing this and the high level approach. The sub-tasks capture the various detailed changes. <a href="https://pulp.plan.io/issues/8495" target="_blank">https://pulp.plan.io/issues/8495</a></div><div><br></div><div>* This is totally separate from the RQ workers you use today, and those will continue to be available for a while.</div><div><br></div><div>## Next Steps</div><div><br></div></div><div>* @mdellweg will continue the work and hopefully merge the PoC while I'm out</div><div><br></div><div>* Once it's demo-able I've asked @mdellweg to give a 20 minute, public (hopefully recorded) technical demo. While it is designed to be a drop-in replacement from a user perspective, we think sharing the internals will be helpful to get feedback and increase the list of those who understand the work.</div><div><br></div><div>All the best,</div><div>Brian</div><div><br></div></div>
_______________________________________________<br>
Pulp-dev mailing list<br>
<a href="mailto:Pulp-dev@redhat.com" target="_blank">Pulp-dev@redhat.com</a><br>
<a href="https://listman.redhat.com/mailman/listinfo/pulp-dev" rel="noreferrer" target="_blank">https://listman.redhat.com/mailman/listinfo/pulp-dev</a><br>
</blockquote></div><br clear="all"><div><br></div>-- <br><div dir="ltr"><div dir="ltr"><div><div dir="ltr">Eric Helms<div>Principal Software Engineer</div><div>Satellite</div></div></div></div></div>
_______________________________________________<br>
Pulp-dev mailing list<br>
<a href="mailto:Pulp-dev@redhat.com" target="_blank">Pulp-dev@redhat.com</a><br>
<a href="https://listman.redhat.com/mailman/listinfo/pulp-dev" rel="noreferrer" target="_blank">https://listman.redhat.com/mailman/listinfo/pulp-dev</a><br>
</blockquote></div>
</blockquote></div><br clear="all"><div><br></div>-- <br><div dir="ltr"><div dir="ltr"><div><div dir="ltr">Eric Helms<div>Principal Software Engineer</div><div>Satellite</div></div></div></div></div></div>
</blockquote></div>