[Pulp-dev] Task tagging in Pulp 3

Tue Nov 14 19:24:03 UTC 2017

Thanks for all the discussion! I agree there are improvements to be made
here.

I don't think either of these proposals solve all the problems without
creating a few new ones. Rather than saying +1 to either, I want to talk
about the goals and use cases a bit more. Here is a list of 3 related use
cases that Pulp currently *can't* do, along with some commentary on the
state of why. Once we decide on what we think users need, figuring out how
to make it should be straightforward.

1) As a user, I can know if a task is current locking a worker or not. Say
they need to take a worker offline. They have no way of knowing if that is
safe to do at this moment without this info. Pulp internally knows this
information, but I don't believe this is visible to the user currently.
This is useful info for debugging that we regularly have to pull from the
db. Regardless of either proposal, we still need to decide if this will be
included in the Task viewset. I'm +1 to adding this use case to the MVP
explicitly. Do people feel this use case should be added to the MVP?

2) As a user, I can filter for tasks by the resource locked, e.g. repo
'foo', without forming a special search string to search by. Currently the
'resource' field in the TaskTag model stores a string like
'repository:foo'. Even if you know the name 'foo' you need to search via
substring (inefficient and maybe dangerous).You also can't search by other
properties like the UUID, feed, etc because to Pulp it's just a string
'repository:foo'. It doesn't know that is actually repository
xxxxx-yyyy-yyyy-zzzzzzz with a name='foo'.

3) As a user, I can filter for tasks by operation type (e.g. sync).
Currently we have no way to do this. The data model doesn't even have a
field to capture this information. This feature seems simple from a high
level, but determining the specific taxonomy of those operation types it
can get messy. We have 'sync' and 'publish', those are pretty clear. What
about 'update' like a publisher/importer/repo attribute update? How about
'add' and 'remove' content? What if both add and remove happen in the same
operation? Is that two tags or a new one? If we're going to talk about this
feature we need to call out the use cases more specifically. A series of
use cases like: "As a user, I can filter for tasks labeled with the 'sync'
operation" could work.

Another way to accomplish use case (3) is to record the actual task name as
a string, e.g. 'pulpcore.app.tasks.importer.sync'. This won't work well
either because we DRY up our tasks, especially update, so I think the
simple taxonomy is still the way forward for that feature.
https://pulp.plan.io/issues/3038

Each proposal affects these use cases, but neither of them totally enables
all of them. Aside from solving them, what do others thing about these use
cases and the current state of Pulp3 w.r.t them? Thanks for all the
discussion.

-Brian

On Thu, Nov 9, 2017 at 3:43 PM, Dennis Kliban <dkliban at redhat.com> wrote:

> On Mon, Nov 6, 2017 at 2:17 PM, David Davis <daviddavis at redhat.com> wrote:
>
>> Originally I scheduled a meeting for tomorrow but on second thought, I
>> figured a pulp-dev thread would be more inclusive than a meeting. I hope to
>> get this resolved by the end of this week and if not then maybe we can have
>> a meeting.
>>
>> This is to design out the replacement of task tags in Pulp 3. I’ve got
>> feedback from a few other developers in terms of how to do that so I wrote
>> up a sort of outline of the problem and two possible proposals. Looking for
>> feedback/questions/etc on what people prefer.
>>
>>
>> Background
>> ---
>>
>> In Pulp 2, tasks have tags that either provide a task name/description or
>> info on what resources a task acts on. Tasks also have reserved resources,
>> which provide a way for tasks to lock a particular resource.
>>
>> In Pulp 3, we have models TaskTag and ReservedResource[0]. Tasks are
>> associated with the resources they work on via TaskTag. If a resource is
>> locked, a ReservedResource record is created in the db and then removed
>> from the db once the resource is unlocked.
>>
>>
>> Problem
>> ---
>>
>> The task tag model doesn't really fit Pulp 3. It's perhaps too generic
>> and totally unnecessary (see Proposal 1), or it could be redesigned to
>> accomodate other things (see Proposal 2).
>>
>> Also, we need to support created resources (e.g. publications) with
>> tasks. Refactoring task tags might provide an opportunity to do so.
>>
>>
>> User stories
>> ---
>>
>> As an authenticated user, I can see what resource(s) a task acted on.
>> As an authenticated user, I can search for a tasks based on what resource
>> they acted on.
>>
>>
>> Proposal 1
>> ---
>>
>> Since tags and reserved resources in Pulp 3 will only store information
>> about a particular repository (not 100% sure here), it should be possible
>> to simplify the data model. We could ditch both TaskTag and
>> ReservedResource models and just have a direct relationship between Tasks
>> and Repositories (e.g. TaskRepository). This model could also have some
>> sort of field to indicate whether a particular field is locked (e.g.
>> is_locked). Unlike ReservedResource, this relationship would be
>> persisted—only the is_locked field would be updated when a task is done.
>>
>>
>> Proposal 2
>> ---
>>
>> We could keep the TaskTag relationship (perhaps even rename it to
>> TaskResource) and we could add a field to indicate the nature of the
>> relationship between task and resource (e.g. created, updated, etc). This
>> field could not only capture what TaskTag is currently used for but also
>> stuff like created resources (e.g. publications). We could also have a
>> field to indicate which task resources are locked (e.g. is_locked).
>>
>>
> I like proposal number 2.
>
>
>
>> This would be useful for https://pulp.plan.io/issues/3033.
>>
>>
>> Questions
>> ---
>>
>> - What proposal do we want to adopt?
>> - When do we need to address these changes?
>> - Do we really need to allow users to search tasks by a resource/repo at
>> all?
>>
>>
>> [0] https://git.io/vF8iH
>>
>> David
>>
>> _______________________________________________
>> Pulp-dev mailing list
>> Pulp-dev at redhat.com
>> https://www.redhat.com/mailman/listinfo/pulp-dev
>>
>>
>
> _______________________________________________
> Pulp-dev mailing list
> Pulp-dev at redhat.com
> https://www.redhat.com/mailman/listinfo/pulp-dev
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/pulp-dev/attachments/20171114/fafdee69/attachment.htm>