[Pulp-dev] Plugin relationship to tasks

Fri Mar 23 17:27:58 UTC 2018

To make sure this was feasable, I've created proof of concept PRs for
pulpcore and pulp_file. All of the proposed changes are proofed, but not
all work is done. For example, sync is implemented, but not publish.

   - https://github.com/pulp/pulp/pull/3394
   - https://github.com/pulp/pulp_file/pull/61

I am very pleased with how it turned out, and I think it will make plugin
writing even easier. I documented the design on a redmine issue. I
encourage feedback/discussion on the issue.

https://pulp.plan.io/issues/3522

On Tue, Mar 20, 2018 at 7:46 AM, Milan Kovacik <mkovacik at redhat.com> wrote:

> On Fri, Mar 16, 2018 at 4:55 PM, Milan Kovacik <mkovacik at redhat.com>
> wrote:
> >
> >
> >
> > On Thu, Mar 15, 2018 at 9:21 PM, Austin Macdonald <austin at redhat.com>
> wrote:
> >>
> >> I spoke with daviddavis about this and I would like to narrow the scope
> a bit. This discussion should be limited to endpoints that deploy tasks.
> The possibility for collisions that David pointed out regarding
> v3/content/<type>/ is real, but should be discussed separately because
> Content objects should follow the pattern of the other plugin defined
> objects. All of them are master/detail so the namespacing
> v3/plugin/<type>/content/ would go deeply against the grain.
> >>
> >>
> >> On Mar 15, 2018 10:53, "David Davis" <daviddavis at redhat.com> wrote:
> >>>
> >>> I have an amendment to this proposal. Instead of namespacing just the
> plugin task routes, I’d propose we namespace all plugin routes. Thus all
> plugin routes would be namespaced under something like
> “/api/v3/plugin/<plugin name>/“. For example, “/api/v3/content/file/” gets
> moved to “/api/v3/plugin/file/content/” and so on.
> >>>
> >>>
> >>> For task routes, plugins could still route to a pulpcore
> viewset/action:
> >>>
> >>> User route: POST /api/v3/plugin/file/repository_version/
> add_content=[…]
> >>>
> >>> Routes to: POST /api/v3/tasks plugin=file
> task=create_repository_version add_content=[…]
> >>
> >>
> >> AFAIK, we don't have a good mechanism for validating and re-routing
> endpoints quite like this. Instead, a view or viewset would be created,
> validation performed, and then a task is deployed. The task itself could be
> a vanilla task from pulpcore though. Another issue is that a POST to
> v3/plugin/file/repository_version/ implies that the resultant
> repository_version is typed in some way. The created resource href would
> still take the form v3/repositories/1234/versions/2/, so I think this is
> a little misleading.
> >
> >
> > +1 to generic Pulp concepts/objects staying outside of a plug-in
> namespace/path
> >
> >>
> >>
> >> For the other endpoint:
> >>  /api/v3/tasks plugin=file task=create_repository_version
> add_content=[…]
> >>
> >> There is a still the problem is that POST requests should not contain
> parameters that influence the allowable parameter set. For instance,
> plugin=python might require an additional parameter that is not allowed for
> the file plugin. In particular, this affects the auto-documented REST API.
> >>
> >>
> >> However, the concept in general could work. I've adapted David's
> suggestion and will present it side by side with the original idea. Note,
> this only applies to "actions", which are sync, publish, add/remove, and
> plugin-specific actions (including rich-dependency adding).
> >>
> >> The difference between the two ideas is based in semantics. Both would
> deploy the same task functions.
> >>
> >> "Action Based"
> >> POST v3/plugin/file/sync/
> >> Note that "sync" is singular. This is a file-plugin action to sync
> using the body parameters.
> >
> >
> > - 1 as this might imply mixing <synchronous> and <asynchronous> actions
> while all the time, asynchronous actions, such as the sync call, always
> create a task object; the user would be creating the task objects at one
> endpoint but deleting, filtering, tracking those at another.
> >
> >>
> >>
> >> "Task Object Based"
> >> POST v3/tasks/file/syncs/
> >> Note that "syncs" is plural. This is a POST that creates a file-sync
> task object,.
>
> Btw an interesting consequence of tasks not being objects in a file
> importer and publisher are these two issues:
>  - The parameters in the API schema for the sync and publish endpoints
> are incorrect [1]
>  - Not able to access documentation endpoint [2]
>
> In the former issue, the DRF isn't able to automatically figure out
> schema for e.g the `sync` endpoint of the file plug-in importer.
> The reason for it is even though the `sync` verb is attached to an
> importer object, the parameters passed in aren't supposed to be
> importer details so they shouldn't use the importer serializer to get
> e.g auto-documented.
> Moreover, the result of a call to this endpoint is supposed to be a
> task (json object), schema and serialization of which has to be
> repeated for the `sync` endpoint to some extent.
> Last but not least, it isn't clear whether the `sync` verb should
> belong to an importer or a repo, or something else as the call
> "responsibility" is spread somewhat between both and none.
>
> The latter issue is a consequence of the lack of support in DRF for
> what the `sync` verb requires documenting thru its schema, which IMO
> underlines the `sync` endpoint being odd, a misplaced task probably.
>
> Cheers,
> milan
>
>
>
> [1] https://pulp.plan.io/issues/3351
> [2] https://pulp.plan.io/issues/3420
>
> >
> > + 1 for typed objects
> >
> >     # adding content to a repo version (already mentioned by Austin but
> I like it)
> >     POST at v3/tasks/docker/content_additions/
> >
> >     # synchronize the Zoo repo
> >     pulp-admin create tasks rpm syncs --repo-id zoo --importer-id
> zoo-importer --sync-config-override '{"ssl": "false"}' --format=json | tee
> /tmp/inttest01_sync_task.json | pulp-admin track tasks --format json -
> >
> >     # get all docker content additions tasks since yesterday
> >     pulp-admin list tasks docker content_additions --format=csv
> --fields=id,worker_id --filter=started_at__gt=<yesterday>
> >
> >     # get all tasks since yesterday
> >     pulp-admin list tasks  --filter=started_at__gt=<yesterday>
> --format=yaml
> >
> >>
> >> Discussion:
> >> Another consideration is "Where will a Live API live?". A benefit of
> David's proposal is that we would be providing plugins with a namespaced
> area to do whatever they need, from live api to actions. We probably would
> want to create this namespace for things like live apis even if we go in
> the tasks/file/syncs/ direction. The only downside of this part of David's
> proposal is that our endpoints will still be "action" verbs instead of
> nouns, and I am ok with that.
> >
> >
> > I'd suggest live API actions that create task object subclasses (generic
> Pulp concept) reside in namespaced tasks path; this would still allow the
> user to apply generic concepts
> >     POST@/v3/tasks/file/frobnications/
> >
> >     pulp-admin create tasks file frobications --foo=bar --format json |
> pulp-admin track tasks --format json --
> >
> > Other live API endpoints (not creating task objects) might be kept under
> namespaces
> >     POST at v3/plugins/file/shake/<{level: strong}
> >
> > pulp-admin plugins file shake --level=strong
> >
> >>
> >> _______________________________________________
> >> Pulp-dev mailing list
> >> Pulp-dev at redhat.com
> >> https://www.redhat.com/mailman/listinfo/pulp-dev
> >>
> >
> > Cheers,
> > milan
> >
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/pulp-dev/attachments/20180323/52f45529/attachment.htm>