[Pulp-dev] Plugin relationship to tasks

Dennis Kliban dkliban at redhat.com
Mon Mar 26 18:30:07 UTC 2018


This proposal does not make the plugin writer's job any easier. This simply
changes where the plugin writer is providing validation logic, in a
serializer vs. in a view.

The other problem I see is that complexity for users is increased. Instead
of having 1 resource for tracking task progress, there would be an infinite
number of resources for tracking progress of tasks.

I don't see the value added by the proposed change.

On Fri, Mar 23, 2018 at 1:27 PM, Austin Macdonald <austin at redhat.com> wrote:

> To make sure this was feasable, I've created proof of concept PRs for
> pulpcore and pulp_file. All of the proposed changes are proofed, but not
> all work is done. For example, sync is implemented, but not publish.
>
>    - https://github.com/pulp/pulp/pull/3394
>    - https://github.com/pulp/pulp_file/pull/61
>
> I am very pleased with how it turned out, and I think it will make plugin
> writing even easier. I documented the design on a redmine issue. I
> encourage feedback/discussion on the issue.
>
> https://pulp.plan.io/issues/3522
>
> On Tue, Mar 20, 2018 at 7:46 AM, Milan Kovacik <mkovacik at redhat.com>
> wrote:
>
>> On Fri, Mar 16, 2018 at 4:55 PM, Milan Kovacik <mkovacik at redhat.com>
>> wrote:
>> >
>> >
>> >
>> > On Thu, Mar 15, 2018 at 9:21 PM, Austin Macdonald <austin at redhat.com>
>> wrote:
>> >>
>> >> I spoke with daviddavis about this and I would like to narrow the
>> scope a bit. This discussion should be limited to endpoints that deploy
>> tasks. The possibility for collisions that David pointed out regarding
>> v3/content/<type>/ is real, but should be discussed separately because
>> Content objects should follow the pattern of the other plugin defined
>> objects. All of them are master/detail so the namespacing
>> v3/plugin/<type>/content/ would go deeply against the grain.
>> >>
>> >>
>> >> On Mar 15, 2018 10:53, "David Davis" <daviddavis at redhat.com> wrote:
>> >>>
>> >>> I have an amendment to this proposal. Instead of namespacing just the
>> plugin task routes, I’d propose we namespace all plugin routes. Thus all
>> plugin routes would be namespaced under something like
>> “/api/v3/plugin/<plugin name>/“. For example, “/api/v3/content/file/” gets
>> moved to “/api/v3/plugin/file/content/” and so on.
>> >>>
>> >>>
>> >>> For task routes, plugins could still route to a pulpcore
>> viewset/action:
>> >>>
>> >>> User route: POST /api/v3/plugin/file/repository_version/
>> add_content=[…]
>> >>>
>> >>> Routes to: POST /api/v3/tasks plugin=file
>> task=create_repository_version add_content=[…]
>> >>
>> >>
>> >> AFAIK, we don't have a good mechanism for validating and re-routing
>> endpoints quite like this. Instead, a view or viewset would be created,
>> validation performed, and then a task is deployed. The task itself could be
>> a vanilla task from pulpcore though. Another issue is that a POST to
>> v3/plugin/file/repository_version/ implies that the resultant
>> repository_version is typed in some way. The created resource href would
>> still take the form v3/repositories/1234/versions/2/, so I think this is
>> a little misleading.
>> >
>> >
>> > +1 to generic Pulp concepts/objects staying outside of a plug-in
>> namespace/path
>> >
>> >>
>> >>
>> >> For the other endpoint:
>> >>  /api/v3/tasks plugin=file task=create_repository_version
>> add_content=[…]
>> >>
>> >> There is a still the problem is that POST requests should not contain
>> parameters that influence the allowable parameter set. For instance,
>> plugin=python might require an additional parameter that is not allowed for
>> the file plugin. In particular, this affects the auto-documented REST API.
>> >>
>> >>
>> >> However, the concept in general could work. I've adapted David's
>> suggestion and will present it side by side with the original idea. Note,
>> this only applies to "actions", which are sync, publish, add/remove, and
>> plugin-specific actions (including rich-dependency adding).
>> >>
>> >> The difference between the two ideas is based in semantics. Both would
>> deploy the same task functions.
>> >>
>> >> "Action Based"
>> >> POST v3/plugin/file/sync/
>> >> Note that "sync" is singular. This is a file-plugin action to sync
>> using the body parameters.
>> >
>> >
>> > - 1 as this might imply mixing <synchronous> and <asynchronous> actions
>> while all the time, asynchronous actions, such as the sync call, always
>> create a task object; the user would be creating the task objects at one
>> endpoint but deleting, filtering, tracking those at another.
>> >
>> >>
>> >>
>> >> "Task Object Based"
>> >> POST v3/tasks/file/syncs/
>> >> Note that "syncs" is plural. This is a POST that creates a file-sync
>> task object,.
>>
>> Btw an interesting consequence of tasks not being objects in a file
>> importer and publisher are these two issues:
>>  - The parameters in the API schema for the sync and publish endpoints
>> are incorrect [1]
>>  - Not able to access documentation endpoint [2]
>>
>> In the former issue, the DRF isn't able to automatically figure out
>> schema for e.g the `sync` endpoint of the file plug-in importer.
>> The reason for it is even though the `sync` verb is attached to an
>> importer object, the parameters passed in aren't supposed to be
>> importer details so they shouldn't use the importer serializer to get
>> e.g auto-documented.
>> Moreover, the result of a call to this endpoint is supposed to be a
>> task (json object), schema and serialization of which has to be
>> repeated for the `sync` endpoint to some extent.
>> Last but not least, it isn't clear whether the `sync` verb should
>> belong to an importer or a repo, or something else as the call
>> "responsibility" is spread somewhat between both and none.
>>
>> The latter issue is a consequence of the lack of support in DRF for
>> what the `sync` verb requires documenting thru its schema, which IMO
>> underlines the `sync` endpoint being odd, a misplaced task probably.
>>
>> Cheers,
>> milan
>>
>>
>>
>> [1] https://pulp.plan.io/issues/3351
>> [2] https://pulp.plan.io/issues/3420
>>
>> >
>> > + 1 for typed objects
>> >
>> >     # adding content to a repo version (already mentioned by Austin but
>> I like it)
>> >     POST at v3/tasks/docker/content_additions/
>> >
>> >     # synchronize the Zoo repo
>> >     pulp-admin create tasks rpm syncs --repo-id zoo --importer-id
>> zoo-importer --sync-config-override '{"ssl": "false"}' --format=json | tee
>> /tmp/inttest01_sync_task.json | pulp-admin track tasks --format json -
>> >
>> >     # get all docker content additions tasks since yesterday
>> >     pulp-admin list tasks docker content_additions --format=csv
>> --fields=id,worker_id --filter=started_at__gt=<yesterday>
>> >
>> >     # get all tasks since yesterday
>> >     pulp-admin list tasks  --filter=started_at__gt=<yesterday>
>> --format=yaml
>> >
>> >>
>> >> Discussion:
>> >> Another consideration is "Where will a Live API live?". A benefit of
>> David's proposal is that we would be providing plugins with a namespaced
>> area to do whatever they need, from live api to actions. We probably would
>> want to create this namespace for things like live apis even if we go in
>> the tasks/file/syncs/ direction. The only downside of this part of David's
>> proposal is that our endpoints will still be "action" verbs instead of
>> nouns, and I am ok with that.
>> >
>> >
>> > I'd suggest live API actions that create task object subclasses
>> (generic Pulp concept) reside in namespaced tasks path; this would still
>> allow the user to apply generic concepts
>> >     POST@/v3/tasks/file/frobnications/
>> >
>> >     pulp-admin create tasks file frobications --foo=bar --format json |
>> pulp-admin track tasks --format json --
>> >
>> > Other live API endpoints (not creating task objects) might be kept
>> under namespaces
>> >     POST at v3/plugins/file/shake/<{level: strong}
>> >
>> > pulp-admin plugins file shake --level=strong
>> >
>> >>
>> >> _______________________________________________
>> >> Pulp-dev mailing list
>> >> Pulp-dev at redhat.com
>> >> https://www.redhat.com/mailman/listinfo/pulp-dev
>> >>
>> >
>> > Cheers,
>> > milan
>> >
>>
>
>
> _______________________________________________
> Pulp-dev mailing list
> Pulp-dev at redhat.com
> https://www.redhat.com/mailman/listinfo/pulp-dev
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/pulp-dev/attachments/20180326/18016ae4/attachment.htm>


More information about the Pulp-dev mailing list