[Pulp-dev] Task tagging in Pulp 3

Tue Nov 14 20:24:59 UTC 2017

On 11/14/2017 01:24 PM, Brian Bouterse wrote:
> Thanks for all the discussion! I agree there are improvements to be made here.
> 
> I don't think either of these proposals solve all the problems without creating a few new ones. Rather than
> saying +1 to either, I want to talk about the goals and use cases a bit more. Here is a list of 3 related use
> cases that Pulp currently *can't* do, along with some commentary on the state of why. Once we decide on what
> we think users need, figuring out how to make it should be straightforward.
> 
> 1) As a user, I can know if a task is current locking a worker or not. Say they need to take a worker offline.
> They have no way of knowing if that is safe to do at this moment without this info. Pulp internally knows this
> information, but I don't believe this is visible to the user currently. This is useful info for debugging that
> we regularly have to pull from the db. Regardless of either proposal, we still need to decide if this will be
> included in the Task viewset. I'm +1 to adding this use case to the MVP explicitly. Do people feel this use
> case should be added to the MVP?

+1

> 
> 2) As a user, I can filter for tasks by the resource locked, e.g. repo 'foo', without forming a special search
> string to search by. Currently the 'resource' field in the TaskTag model stores a string like
> 'repository:foo'. Even if you know the name 'foo' you need to search via substring (inefficient and maybe
> dangerous).You also can't search by other properties like the UUID, feed, etc because to Pulp it's just a
> string 'repository:foo'. It doesn't know that is actually repository xxxxx-yyyy-yyyy-zzzzzzz with a name='foo'.

I would like to see this use case expanded (into several cases) to include a description of why a user wants
to do this.  What are they trying to accomplish.  Like: "As a user, I want to search for tasks pending for a
repository because I'm trying to understand why my task isn't running yet."

This applies to #3 as well.

> 
> 3) As a user, I can filter for tasks by operation type (e.g. sync). Currently we have no way to do this. The
> data model doesn't even have a field to capture this information. This feature seems simple from a high level,
> but determining the specific taxonomy of those operation types it can get messy. We have 'sync' and 'publish',
> those are pretty clear. What about 'update' like a publisher/importer/repo attribute update? How about 'add'
> and 'remove' content? What if both add and remove happen in the same operation? Is that two tags or a new one?
> If we're going to talk about this feature we need to call out the use cases more specifically. A series of use
> cases like: "As a user, I can filter for tasks labeled with the 'sync' operation" could work.
> 
> Another way to accomplish use case (3) is to record the actual task name as a string, e.g.
> 'pulpcore.app.tasks.importer.sync'. This won't work well either because we DRY up our tasks, especially
> update, so I think the simple taxonomy is still the way forward for that feature. https://pulp.plan.io/issues/3038

Agreed.  Exposing and relying on an implementation detail like "pulpcore.app.tasks.importer.sync" would be bad.

> 
> Each proposal affects these use cases, but neither of them totally enables all of them. Aside from solving
> them, what do others thing about these use cases and the current state of Pulp3 w.r.t them? Thanks for all the
> discussion.
> 
> -Brian
> 
> 
> On Thu, Nov 9, 2017 at 3:43 PM, Dennis Kliban <dkliban at redhat.com <mailto:dkliban at redhat.com>> wrote:
> 
>     On Mon, Nov 6, 2017 at 2:17 PM, David Davis <daviddavis at redhat.com <mailto:daviddavis at redhat.com>> wrote:
> 
>         Originally I scheduled a meeting for tomorrow but on second thought, I figured a pulp-dev thread would
>         be more inclusive than a meeting. I hope to get this resolved by the end of this week and if not then
>         maybe we can have a meeting. 
> 
>         This is to design out the replacement of task tags in Pulp 3. I’ve got feedback from a few other
>         developers in terms of how to do that so I wrote up a sort of outline of the problem and two possible
>         proposals. Looking for feedback/questions/etc on what people prefer.
> 
> 
>         Background
>         ---
> 
>         In Pulp 2, tasks have tags that either provide a task name/description or info on what resources a
>         task acts on. Tasks also have reserved resources, which provide a way for tasks to lock a particular
>         resource.
> 
>         In Pulp 3, we have models TaskTag and ReservedResource[0]. Tasks are associated with the resources
>         they work on via TaskTag. If a resource is locked, a ReservedResource record is created in the db and
>         then removed from the db once the resource is unlocked.
> 
> 
>         Problem
>         ---
> 
>         The task tag model doesn't really fit Pulp 3. It's perhaps too generic and totally unnecessary (see
>         Proposal 1), or it could be redesigned to accomodate other things (see Proposal 2).
> 
>         Also, we need to support created resources (e.g. publications) with tasks. Refactoring task tags might
>         provide an opportunity to do so.
> 
> 
>         User stories
>         ---
> 
>         As an authenticated user, I can see what resource(s) a task acted on.
>         As an authenticated user, I can search for a tasks based on what resource they acted on.
> 
> 
>         Proposal 1
>         ---
> 
>         Since tags and reserved resources in Pulp 3 will only store information about a particular repository
>         (not 100% sure here), it should be possible to simplify the data model. We could ditch both TaskTag
>         and ReservedResource models and just have a direct relationship between Tasks and Repositories (e.g.
>         TaskRepository). This model could also have some sort of field to indicate whether a particular field
>         is locked (e.g. is_locked). Unlike ReservedResource, this relationship would be persisted—only the
>         is_locked field would be updated when a task is done.
> 
> 
>         Proposal 2
>         ---
> 
>         We could keep the TaskTag relationship (perhaps even rename it to TaskResource) and we could add a
>         field to indicate the nature of the relationship between task and resource (e.g. created, updated,
>         etc). This field could not only capture what TaskTag is currently used for but also stuff like created
>         resources (e.g. publications). We could also have a field to indicate which task resources are locked
>         (e.g. is_locked).
> 
> 
>     I like proposal number 2.
> 
>      
> 
>         This would be useful for https://pulp.plan.io/issues/3033 <https://pulp.plan.io/issues/3033>.
> 
> 
>         Questions
>         ---
> 
>         - What proposal do we want to adopt?
>         - When do we need to address these changes?
>         - Do we really need to allow users to search tasks by a resource/repo at all?
> 
> 
>         [0] https://git.io/vF8iH
> 
>         David
> 
>         _______________________________________________
>         Pulp-dev mailing list
>         Pulp-dev at redhat.com <mailto:Pulp-dev at redhat.com>
>         https://www.redhat.com/mailman/listinfo/pulp-dev <https://www.redhat.com/mailman/listinfo/pulp-dev>
> 
> 
> 
>     _______________________________________________
>     Pulp-dev mailing list
>     Pulp-dev at redhat.com <mailto:Pulp-dev at redhat.com>
>     https://www.redhat.com/mailman/listinfo/pulp-dev <https://www.redhat.com/mailman/listinfo/pulp-dev>
> 
> 
> 
> 
> _______________________________________________
> Pulp-dev mailing list
> Pulp-dev at redhat.com
> https://www.redhat.com/mailman/listinfo/pulp-dev
> 

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 847 bytes
Desc: OpenPGP digital signature
URL: <http://listman.redhat.com/archives/pulp-dev/attachments/20171114/7efa712c/attachment.sig>