[Pulp-dev] Plugin relationship to tasks
Dennis Kliban
dkliban at redhat.com
Mon Apr 2 14:26:21 UTC 2018
Here is another idea that would remove the verbs from the REST API and
would provide a way to namespace plugin provided tasks.
We could add 2 additional resource types for plugin writers to provide:
versionaction and publicationaction. These REST resources would only
support GET requests. The response would contain a description of the
action and a description of the parameters this action accepts. Plugin
writers would create these resources by extending an ActionView with a
custom get() method for providing users with a description and also an
action() method which would contain the code that needs to run
asynchronously.
Every repository version would be created using the repository version API.
POST requests to the repository version API would accept a 'versionaction'
href along with a dictionary of parameters. Users would then receive a 202
response with a task href that can be used to monitor the progress of the
asynchronous repository version creation.
Every publication would be created using the publication API. POST requests
to the publication API would accept a 'publicationaction' href along with a
dictionary of parameters. Users would then receive a 202 response with a
task href that can be used to monitor the progress of the asynchronous
publication creation.
Here are some REST API changes this would bring:
*/api/v3/versionactions/* - returns a list of all the action descriptions
for creating repository versions
*/api/v3/versionactions/**pulpcore/addremove/* - returns a description of
the addremove action provided by pulpcore. Something like this:
{
'href': 'http://localhost/api/v3/versionactions/pulpcore/addremove/',
'description': 'Creates a repository version by adding and removing
content units from the latest repository version.',
'parameters': {
'add_content_units': 'list of content to add',
'remove_content_units': 'list of content to
remove'
}
}
*/api/v3/versionactions/<**plugin>/sync/ *- accepts GET and returns a
description of the sync action of the plugin. Something like this:
{
'href': 'http://localhost/api/v3/versionactions/file/sync/',
'description': 'Creates a repository version by syncing content from a
remote file repository.',
'parameters': {
'remote': 'href of the remote to sync from'
}
}
*/api/v3/publicationactions/ *- returns a list of all the actions for
creating publications
*/api/v3/publicationactions/<**plugin>/publish/* - accepts GET and returns
a description of the publish action for the file plugin. Something like
this:
{
'href': 'http://localhost/api/v3/versionactions/file/publish/',
'description': 'Create a publication from a file repository version.',
'parameters': {
'repository': 'href of the repository to
publish. The latest repository version will be published. Can only be
specified if repository_version is not specified.'
'repository_version': 'href of the repository
version to publish. Can only be specified if repository_version is not
specified. Can only be specified if repository is not specified.'
}
}
*/api/v3/repository/<uuid>/versions/ *- accepts POST requests with
'versionaction' href and 'parameters' dictionary. Returns 202 with a task
href.
*/api/v3/publications/ *- accepts POST requests with 'publicationaction'
href and 'parameters' dictionary. Returns 202 with a task href.
On Thu, Mar 29, 2018 at 4:18 PM, Milan Kovacik <mkovacik at redhat.com> wrote:
> Folks,
>
> I'd like to summarize where we've gotten so far. Our problem statement
> is threefold:
>
> 1) Content operations have to be routed thru particular plug-ins i.e
> no repository version can be created without a plug-in involvement, to
> avoid repository content consistency/correctness issues
> 2) API specialization i.e keeping core concepts above plug-in specific
> concepts; e.g CRUD@/v3/repositories/ v.s CRUD at docker/adds/
> 3) API consistency i.e preferably where I create it is where I manage
> it; e.g. w/r/t asynchronous action endpoints creating Task resources
> vs a Task management endpoint
>
> Since there is a significant intersection between the asynchronous
> action endpoints and endpoints manipulating repository versions
> asynchronously, me and Austin propose to convert the asynchronous
> action endpoints to Task management endpoints, in full control of
> plug-ins, in particular:
>
> Asynchronous Repository Version Management Endpoints:
>
> Repository synchronisation and publication and addition(deletion) of
> content into(from) a repository version would all be treated as Task
> objects primarily, residing in the plug-in namespace:
> POST@/v3/tasks/<plug-in>/[syncs, publishes, additions, deletions]/
> because all these operations share similar call signature, every one
> of these actions:
>
> * creates a Task object
> * the task, while executing, creates a new repository version and
> locks similar resources
> * has to be controlled by a particular plug-in i.e may require
> additional, plug-in specific settings
> * requires either a repository URI or repository version URI to operate
> on
> * requires generic action configuration e.g almost every sync
> requires (at least one) importer
>
> As a result:
> * the asynchronous content management Task is now the nexus where
> the configuration (repository, plug-in, origin) meets the action
> * these complex tasks that actually do the heavy lifting have a
> clear management endpoint
> * this change effectively renders the
> @/v3/repositories/<UUID>/versions/<#nr>/ endpoint read-only and
> documents the intention that no repository version is ever created
> without an actual plug-in involvement
> * a plug-in can expose custom asynchronous content endpoints, that
> manipulate repository versions, aligned with the more generic actions:
> POST@/v3/tasks/<plug-in>/[<custom asynchronous repository
> version management action>,...]/
> * the design aligns better with the DRF allowing (nested)
> serializers to control (and document) the endpoints in the direction
> from the more generic to the more specific implementation
>
> Asynchronous Core Object Management Endpoints:
>
> Additionally, for Asynchronous Core object management endpoints, we
> suggest to move Remote/Importer, Publisher, Exporter, Repository
> operations under a new core task namespace:
> POST@/v3/tasks/core/[importers, publishers,
> repositories]/[deletes, updates]/
> to be consistent with the asynchronous content actions. This however
> seems to conflict with 3).
>
> Caveats
>
> Both the current and the proposed design don't conceptually prevent a
> repository to have content provided by multiple plug-ins.
> Folks don't like Pulp becoming a task management system.
>
> Alternatives
>
> Dennis proposes to achieve similar result by adding plug-in content
> model hooks:
> "add_to_repository_version" and "remove_from_repository_version"
> This approach however doesn't seem to solve plug-ins requiring
> multiple or no importers and leaves multiple spots in the API, where
> repository versions can be created (failing 3); custom plug-in
> asynchronous content management endpoints and importers/remotes sync
> vs POST at v3/repository/UUID/versions/<#nr>/
>
> David proposes to namespace all objects under plug-in specific
> endpoints. This solves the repository--plug-in purity caveat, allows
> great plug-in flexibility and prevents endpoint name collisions.
> This approach may allow the plug-ins UX to drift from each other
> though, and solves neither the multiple/no importers case, nor the
> scattering of repository versions creation endpoints and fails 2)&3).
>
> I believe this Master/Detail Tasks design to be a reasonable
> compromise on the requirements in 1) 2) 3).
>
> Cheers,
> milan
>
> On Wed, Mar 28, 2018 at 9:38 PM, Dennis Kliban <dkliban at redhat.com> wrote:
> > On Wed, Mar 28, 2018 at 12:20 PM, Ina Panova <ipanova at redhat.com> wrote:
> >>
> >> I do not think that it's a valid argument to ban the proposal just
> because
> >> the proposal will bring changes to the existing plugins. Because:
> >>
> >> 1) i think it would be fair to think of other plugins and find a
> solution
> >> all plugins will be happy with --> so we are back to the generic
> correctness
> >> problem
> >> 2) we are not GA, not even Beta, we are free and eligible to make
> changes,
> >> no promises made yet.
> >> 3) if we bring changes *this* exactly is the time to do because we have
> >> just 2 plugins working with basic functionality :)
> >>
> >> W/r to the "Pulp turning into a task running system" ..and yet that's a
> >> system based on distributed task system
> >
> >
> > Pulp may rely on a tasking system to get work done, but we don't want
> users
> > thinking of it as a tasking system. We want our users to think of Pulp
> as a
> > repository management system.
> >
> >>
> >>
> >> I suggest to organize a meeting with an agenda in advance that would
> list
> >> 1) concerns 2) questions
> >> I also think it would be valuable to hear Jeff's feedback.
> >>
> >> If the team finds that meeting idea is beneficial with the believe that
> >> after the meeting we will:
> >> 1) i am not naive to believe that we would reach consensus but at least
> we
> >> would move forward
> >> 2) we will decrease the frustration level, because face2face
> conversation
> >> is more profitable, since we do see faces, have social interaction and
> not
> >> type onto keyboard and stare at the text on the monitor.
> >>
> >> then...I will take the action item and schedule the meeting.
> >>
> >> Meanwhile we'll have time to chew on Easter eggs and give to the
> proposal
> >> a deeper, unbiased, full of protein (that helps brain thinking) thought.
> >>
> >
> > I am against having a meeting to discuss this. The discussion on the list
> > has gotten traction and I'd like it to continue on here.
> >
> >>
> >>
> >>
> >>
> >> --------
> >> Regards,
> >>
> >> Ina Panova
> >> Software Engineer| Pulp| Red Hat Inc.
> >>
> >> "Do not go where the path may lead,
> >> go instead where there is no path and leave a trail."
> >>
> >> On Wed, Mar 28, 2018 at 12:11 AM, Brian Bouterse <bbouters at redhat.com>
> >> wrote:
> >>>
> >>> In terms of removing POST /api/v3/repositories/<uuid>/versions/ from
> >>> core, I want to bring it back to the MVP language and REST which drove
> the
> >>> original design. The MVP has: "As an authenticated user, I can create
> a new
> >>> version by adding or removing content to the latest version." To
> facilitate
> >>> that in a generic way, we need a core endpoint to do that, i.e.
> >>> /api/v3/repositories/<uuid>/versions/. My concern is that removing it
> would
> >>> cause us to not fulfill our use cases without requiring more code from
> some
> >>> plugin writers. Also in terms of REST philosophy, POSTing to the
> >>> RespotoryVersion resource to create a new RepositoryVersion is the
> >>> traditional url design.
> >>>
> >>> For pulp_ansible, for example, the generic add/remove functionality at
> >>> core endpoint ^ would be meet all of the pulp_ansible user's needs
> because
> >>> of the way the pulp_ansible content is modelled. So removing this
> endpoint
> >>> means more work for pulp_ansible developers w.r.t creating repo
> versions and
> >>> providing tasks and endpoints. I don't see this extra responsibility on
> >>> plugin writers coming with a clear benefit.
> >>>
> >>> One idea I liked in this discussion is to have a documented convention
> >>> that encourages plugin writers to put their viewsets in a namespaced
> area.
> >>> They still need the ability to put them anywhere due to live API
> >>> goals/requirements so this would only be a convention for those tasks
> they
> >>> are offering directly to their users.
> >>>
> >>>
> >>>
> >>>
> >>>
> >>> On Tue, Mar 27, 2018 at 5:40 PM, David Davis <daviddavis at redhat.com>
> >>> wrote:
> >>>>
> >>>> I like Austin’s task proposal in that the plugin writers can focus on
> >>>> serializers and tasks that can be easily integrated with core. That
> said, I
> >>>> agree on the counterpoints raise about the new resource endpoints and
> >>>> turning much of pulp into a task running system. So I am a bit mixed
> on
> >>>> which approach is better.
> >>>>
> >>>> I do think that we should remove POST
> >>>> /api/v3/repositories/<uuid>/versions/ from core.
> >>>>
> >>>>
> >>>> David
> >>>>
> >>>> On Tue, Mar 27, 2018 at 3:49 PM, Austin Macdonald <
> amacdona at redhat.com>
> >>>> wrote:
> >>>>>>
> >>>>>> /api/v3/repositories/<uuid>/versions/ endpoint does not perform
> plugin
> >>>>>> specific validation which can lead to "broken" repository versions.
> >>>>>> Plugin authors don't have any convention to follow when creating
> >>>>>> custom REST API endpoints for creating repository versions.
> >>>>>> As a result of ^, a user will have a hard time identifying
> repository
> >>>>>> version creation APIs in different plugins.
> >>>>>
> >>>>>
> >>>>> I agree with these points.
> >>>>>>
> >>>>>> My first inclination is to disable the ability to POST to
> >>>>>> /api/v3/repositories/<uuid>/versions/ and require users to use the
> plugin
> >>>>>> specific APIs for creating repository versions. However, I think
> that
> >>>>>> integrators of build systems that produce a variety of content
> types would
> >>>>>> have a lot more flexibility if they could use a single generic API
> endpoint
> >>>>>> to create a repository version independent of the content type.
> >>>>>>
> >>>>>> Let's continue this discussion by answering the following question:
> >>>>>>
> >>>>>> Should we disable the ability to POST to
> >>>>>> /api/v3/repositories/<uuid>/versions/ and require users to always
> use a
> >>>>>> plugin specific repository version creation API?
> >>>>>
> >>>>> Yes, I think we should disable POST to
> >>>>> /api/v3/repositories/<uuid>/versions/
> >>>>>
> >>>>> Simplifying integration is important, but we should not sacrifice
> >>>>> correctness enforcement.
> >>>>>
> >>>>> _______________________________________________
> >>>>> Pulp-dev mailing list
> >>>>> Pulp-dev at redhat.com
> >>>>> https://www.redhat.com/mailman/listinfo/pulp-dev
> >>>>>
> >>>>
> >>>>
> >>>> _______________________________________________
> >>>> Pulp-dev mailing list
> >>>> Pulp-dev at redhat.com
> >>>> https://www.redhat.com/mailman/listinfo/pulp-dev
> >>>>
> >>>
> >>>
> >>> _______________________________________________
> >>> Pulp-dev mailing list
> >>> Pulp-dev at redhat.com
> >>> https://www.redhat.com/mailman/listinfo/pulp-dev
> >>>
> >>
> >>
> >> _______________________________________________
> >> Pulp-dev mailing list
> >> Pulp-dev at redhat.com
> >> https://www.redhat.com/mailman/listinfo/pulp-dev
> >>
> >
> >
> > _______________________________________________
> > Pulp-dev mailing list
> > Pulp-dev at redhat.com
> > https://www.redhat.com/mailman/listinfo/pulp-dev
> >
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/pulp-dev/attachments/20180402/1548be73/attachment.htm>
More information about the Pulp-dev
mailing list