[Pulp-dev] Tasking System Changes and Feedback

[Daniel Alley]

Eric,

* The idea is to move away from RQ entirely.  RQ is fine (and vastly better
than Celery IMO), but managing task state across both 1) the database and
2) a separate, external registry is still problematic.  If all of the
information can simply be kept in the database, then it will be much easier
to maintain consistent state.
* *Maybe*.  We're considering using Redis as a cache to improve content
serving performance (after all, caching is one of the primary uses of
Redis). If we do, then Redis would remain in the architecture, but it could
potentially be an optional component and would be easier to remove at some
point in the future.
* We'd just be adding a small amount of information to each task record,
and it wouldn't prevent cleanup later.



On Thu, Apr 8, 2021 at 4:42 PM Eric Helms <ehelms at redhat.com> wrote:

> A few initial questions that get a bit into the stack but will help the
> Foreman project think on the proposed changes:
>
>  * Does this move away from RQ entirely or just RQ workers?
>  * Do the new workers remove Pulp 3's use of Redis all together?
>  * Will using the database result in any additional build up of tasking
> information that can impact performance over time? (Or does all task data
> get cleaned up eventually?)
>
> Thanks for sending this along early.
>
> On Fri, Apr 2, 2021 at 4:43 PM Brian Bouterse <bmbouter at redhat.com> wrote:
>
>> FYI, @mdellweg and I have been collaborating on the tasking system
>> changes. This email is to share some info to transition the work to
>> @mdellweg while I'm out. With the new-style disabled by default I am hoping
>> it can go into 3.13.
>>
>> ## The PoC and ticket info
>>
>> The PoC is basically functional, but it's a PoC:
>> https://github.com/pulp/pulpcore/pull/1222/
>>
>> * The epic is being tracked here which recaps why we're doing this and
>> the high level approach. The sub-tasks capture the various detailed
>> changes. https://pulp.plan.io/issues/8495
>>
>> * This is totally separate from the RQ workers you use today, and those
>> will continue to be available for a while.
>>
>> ## Next Steps
>>
>> * @mdellweg will continue the work and hopefully merge the PoC while I'm
>> out
>>
>> * Once it's demo-able I've asked @mdellweg to give a 20 minute, public
>> (hopefully recorded) technical demo. While it is designed to be a drop-in
>> replacement from a user perspective, we think sharing the internals will be
>> helpful to get feedback and increase the list of those who understand the
>> work.
>>
>> All the best,
>> Brian
>>
>> _______________________________________________
>> Pulp-dev mailing list
>> Pulp-dev at redhat.com
>> https://listman.redhat.com/mailman/listinfo/pulp-dev
>>
>
>
> --
> Eric Helms
> Principal Software Engineer
> Satellite
> _______________________________________________
> Pulp-dev mailing list
> Pulp-dev at redhat.com
> https://listman.redhat.com/mailman/listinfo/pulp-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/pulp-dev/attachments/20210408/c3dca7ef/attachment.htm>