[Pulp-dev] Tasking System Changes and Feedback
ehelms at redhat.com
Tue Apr 13 15:18:01 UTC 2021
On Thu, Apr 8, 2021 at 5:24 PM Daniel Alley <dalley at redhat.com> wrote:
> * The idea is to move away from RQ entirely. RQ is fine (and vastly
> better than Celery IMO), but managing task state across both 1) the
> database and 2) a separate, external registry is still problematic. If all
> of the information can simply be kept in the database, then it will be much
> easier to maintain consistent state.
Are there any benefits to improving RQ vs the invented here method? I'm
just curious about the cost of maintaining a tasking system versus being
part of a community built one. This feels like the kind of problem that
many other applications should have in the Python world -- or are there
elements of Pulp's deployment architecture that make it unique here?
> * *Maybe*. We're considering using Redis as a cache to improve content
> serving performance (after all, caching is one of the primary uses of
> Redis). If we do, then Redis would remain in the architecture, but it could
> potentially be an optional component and would be easier to remove at some
> point in the future.
> * We'd just be adding a small amount of information to each task record,
> and it wouldn't prevent cleanup later.
This is sort of an aside to this general change. Are Pulp tasks cleaned up
from the database today?
> On Thu, Apr 8, 2021 at 4:42 PM Eric Helms <ehelms at redhat.com> wrote:
>> A few initial questions that get a bit into the stack but will help the
>> Foreman project think on the proposed changes:
>> * Does this move away from RQ entirely or just RQ workers?
>> * Do the new workers remove Pulp 3's use of Redis all together?
>> * Will using the database result in any additional build up of tasking
>> information that can impact performance over time? (Or does all task data
>> get cleaned up eventually?)
>> Thanks for sending this along early.
>> On Fri, Apr 2, 2021 at 4:43 PM Brian Bouterse <bmbouter at redhat.com>
>>> FYI, @mdellweg and I have been collaborating on the tasking system
>>> changes. This email is to share some info to transition the work to
>>> @mdellweg while I'm out. With the new-style disabled by default I am hoping
>>> it can go into 3.13.
>>> ## The PoC and ticket info
>>> The PoC is basically functional, but it's a PoC:
>>> * The epic is being tracked here which recaps why we're doing this and
>>> the high level approach. The sub-tasks capture the various detailed
>>> changes. https://pulp.plan.io/issues/8495
>>> * This is totally separate from the RQ workers you use today, and those
>>> will continue to be available for a while.
>>> ## Next Steps
>>> * @mdellweg will continue the work and hopefully merge the PoC while I'm
>>> * Once it's demo-able I've asked @mdellweg to give a 20 minute, public
>>> (hopefully recorded) technical demo. While it is designed to be a drop-in
>>> replacement from a user perspective, we think sharing the internals will be
>>> helpful to get feedback and increase the list of those who understand the
>>> All the best,
>>> Pulp-dev mailing list
>>> Pulp-dev at redhat.com
>> Eric Helms
>> Principal Software Engineer
>> Pulp-dev mailing list
>> Pulp-dev at redhat.com
Principal Software Engineer
-------------- next part --------------
An HTML attachment was scrubbed...
More information about the Pulp-dev