[Pulp-dev] Plugin relationship to tasks
Dennis Kliban
dkliban at redhat.com
Mon Apr 2 18:37:11 UTC 2018
On Mon, Apr 2, 2018 at 12:41 PM, Austin Macdonald <amacdona at redhat.com>
wrote:
> I'm concerned that this conversation is too broad. I'm willing to
> brainstorm ideas because this is an important topic, but I also hope that
> at some point we can come back to the proposal for master/detail tasks.
>
> This idea is interesting, because it provides a novel way to connect the
> plugins to pulpcore calls. The views that plugins implement would not be
> normal views, they would operate more like OPTIONS schema views. I'll try
> to show how I think this would look from a user's perspective:
>
> *File Sync:*
>
> The user doesn't know how to sync, so they do a GET request:
> *GET /v3/versionactions/*
>
> This returns a list of possible actions, one of them being
> v3/versionactions/file/sync. To see how to use that they do another GET
> request:
> *GET /v3versionactions/file/sync/*
>
> This returns the description of the file-sync action and the parameters
> necessary (importer, repository)
>
> *POST v3/repositories/1234/versions/* importer=importer_href,
> repository=repository_href, action=v3/versionactions/file/sync/
>
> I have some practical questions:
>
> - What does the documentation for v3/repositories/1234/versions/ look
> like?
> - Are the parameters listed, or does the user have to use
> v3/versionactions/sync/ to retrieve their schema?
>
> The schema for this API would be
'versionaction': 'An href for the action to be used for creating the new
repository version. List of available actions and their parameters is
available at /api/v3/versionactions/.
'parameters': 'A JSON object that contains parameter names and
corresponding values. The required parameters for each action can be
obtained using the versionactions API.
>
> - How does the RepositoryVersionsViewSet use the versionaction?
> - How are the params `importer`, and `repository` validated?
> - Synchronously
> - Asynchronously
>
> Django's utils would be used to convert the versionaction href into an
instance of the view provided by the plugin.
We could provide a validate() interface in the ActionView for plugin
writers. When implemented, the method is used to perform validation on
parameters at the REST API level. Any asynchronous validation would be part
of the action() method.
>
> - How is the celery task connected and dispatched?
>
> I am not sure of implementation details, but the goal is to have pulpcore
handle this and require just the action() method that will be turned in to
a task.
> Here are my initial thoughts:
>
> - I like that this brings us back to "to create a new
> RepositoryVersion, POST to versions".
> - I especially like that all versions are created at a single
> endpoint.
> - I don't like that the plugins are creating views that are not
> intended to be used as endpoints. These views are only "hit" for
> documentation, and they are used as parameters to other tasks.
> - This could be confusing for users
> - This is a different type of work than the rest of plugin writing,
> so it will have its own learning curve
> - This would not (speculation) play nice with auto-documentation.
> The autodocumentation would not know about the possible parameters for the
> actions, so the users would have to user GET v3/versionactions/ *instead
> *of docs.
> - This would not (speculation) play nice with auto-bindings and
> clients. Same problem as ^
> - GET request does not retrieve objects, it retrieves schema
>
> I think of these actions as 2 new types of resource in Pulp. Unlike
Remotes(Importers currently) and Content, these resources are singletons
that each plugin provides. Since users don't need to create new instances
of these resources it makes sense that they are represented by a View
instead of a ViewSet. Though we don't even need to explain that to plugin
writers. We just need to tell them that all operations that their plugin
provides for users need to be sublassed from ActionView.
>From the users point of view a GET on /api/v3/versionsactions/ to find a
versionaction href is the same as performing a GET on /api/v3/remotes/ to
find the href for a specific remote to sync from.
Auto documentation would work perfectly.
Each */api/v3/versionsactions/<plugin>/<action name>* endpoint would be
documented with "GET returns detailed information about SomeAction. This
information includes the parameters that need to be specified when this
action is used to create a repository version."
The */api/v3/repositories/<uuid>/versions/ *endpoint would be documented
with "POST accepts a 'versionaction': 'href for action that will be used to
create a repository version' and 'parameters': 'Object where the keys are
parameter names and values are their values. These parameters are used by
the action to create a repository version. The required keys for parameters
can be obtained by performing a GET on the versionaction href'. Returns 202
with a task href."
>
> On Mon, Apr 2, 2018 at 10:26 AM, Dennis Kliban <dkliban at redhat.com> wrote:
>
>> Here is another idea that would remove the verbs from the REST API and
>> would provide a way to namespace plugin provided tasks.
>>
>> We could add 2 additional resource types for plugin writers to provide:
>> versionaction and publicationaction. These REST resources would only
>> support GET requests. The response would contain a description of the
>> action and a description of the parameters this action accepts. Plugin
>> writers would create these resources by extending an ActionView with a
>> custom get() method for providing users with a description and also an
>> action() method which would contain the code that needs to run
>> asynchronously.
>>
>> Every repository version would be created using the repository version
>> API. POST requests to the repository version API would accept a
>> 'versionaction' href along with a dictionary of parameters. Users would
>> then receive a 202 response with a task href that can be used to monitor
>> the progress of the asynchronous repository version creation.
>>
>> Every publication would be created using the publication API. POST
>> requests to the publication API would accept a 'publicationaction' href
>> along with a dictionary of parameters. Users would then receive a 202
>> response with a task href that can be used to monitor the progress of the
>> asynchronous publication creation.
>>
>> Here are some REST API changes this would bring:
>>
>> */api/v3/versionactions/* - returns a list of all the action
>> descriptions for creating repository versions
>>
>> */api/v3/versionactions/**pulpcore/addremove/* - returns a description
>> of the addremove action provided by pulpcore. Something like this:
>>
>> {
>> 'href': 'http://localhost/api/v3/versionactions/pulpcore/addremove/',
>> 'description': 'Creates a repository version by adding and removing
>> content units from the latest repository version.',
>> 'parameters': {
>> 'add_content_units': 'list of content to
>> add',
>> 'remove_content_units': 'list of content to
>> remove'
>> }
>> }
>>
>>
>> */api/v3/versionactions/<**plugin>/sync/ *- accepts GET and returns a
>> description of the sync action of the plugin. Something like this:
>>
>> {
>> 'href': 'http://localhost/api/v3/versionactions/file/sync/',
>> 'description': 'Creates a repository version by syncing content from a
>> remote file repository.',
>> 'parameters': {
>> 'remote': 'href of the remote to sync from'
>> }
>> }
>>
>>
>> */api/v3/publicationactions/ *- returns a list of all the actions for
>> creating publications
>>
>> */api/v3/publicationactions/<**plugin>/publish/* - accepts GET and
>> returns a description of the publish action for the file plugin. Something
>> like this:
>>
>> {
>> 'href': 'http://localhost/api/v3/versionactions/file/publish/',
>> 'description': 'Create a publication from a file repository version.',
>> 'parameters': {
>> 'repository': 'href of the repository to
>> publish. The latest repository version will be published. Can only be
>> specified if repository_version is not specified.'
>> 'repository_version': 'href of the
>> repository version to publish. Can only be specified if repository_version
>> is not specified. Can only be specified if repository is not specified.'
>> }
>> }
>>
>> */api/v3/repository/<uuid>/versions/ *- accepts POST requests with
>> 'versionaction' href and 'parameters' dictionary. Returns 202 with a task
>> href.
>>
>> */api/v3/publications/ *- accepts POST requests with 'publicationaction'
>> href and 'parameters' dictionary. Returns 202 with a task href.
>>
>>
>> On Thu, Mar 29, 2018 at 4:18 PM, Milan Kovacik <mkovacik at redhat.com>
>> wrote:
>>
>>> Folks,
>>>
>>> I'd like to summarize where we've gotten so far. Our problem statement
>>> is threefold:
>>>
>>> 1) Content operations have to be routed thru particular plug-ins i.e
>>> no repository version can be created without a plug-in involvement, to
>>> avoid repository content consistency/correctness issues
>>> 2) API specialization i.e keeping core concepts above plug-in specific
>>> concepts; e.g CRUD@/v3/repositories/ v.s CRUD at docker/adds/
>>> 3) API consistency i.e preferably where I create it is where I manage
>>> it; e.g. w/r/t asynchronous action endpoints creating Task resources
>>> vs a Task management endpoint
>>>
>>> Since there is a significant intersection between the asynchronous
>>> action endpoints and endpoints manipulating repository versions
>>> asynchronously, me and Austin propose to convert the asynchronous
>>> action endpoints to Task management endpoints, in full control of
>>> plug-ins, in particular:
>>>
>>> Asynchronous Repository Version Management Endpoints:
>>>
>>> Repository synchronisation and publication and addition(deletion) of
>>> content into(from) a repository version would all be treated as Task
>>> objects primarily, residing in the plug-in namespace:
>>> POST@/v3/tasks/<plug-in>/[syncs, publishes, additions, deletions]/
>>> because all these operations share similar call signature, every one
>>> of these actions:
>>>
>>> * creates a Task object
>>> * the task, while executing, creates a new repository version and
>>> locks similar resources
>>> * has to be controlled by a particular plug-in i.e may require
>>> additional, plug-in specific settings
>>> * requires either a repository URI or repository version URI to
>>> operate on
>>> * requires generic action configuration e.g almost every sync
>>> requires (at least one) importer
>>>
>>> As a result:
>>> * the asynchronous content management Task is now the nexus where
>>> the configuration (repository, plug-in, origin) meets the action
>>> * these complex tasks that actually do the heavy lifting have a
>>> clear management endpoint
>>> * this change effectively renders the
>>> @/v3/repositories/<UUID>/versions/<#nr>/ endpoint read-only and
>>> documents the intention that no repository version is ever created
>>> without an actual plug-in involvement
>>> * a plug-in can expose custom asynchronous content endpoints, that
>>> manipulate repository versions, aligned with the more generic actions:
>>> POST@/v3/tasks/<plug-in>/[<custom asynchronous repository
>>> version management action>,...]/
>>> * the design aligns better with the DRF allowing (nested)
>>> serializers to control (and document) the endpoints in the direction
>>> from the more generic to the more specific implementation
>>>
>>> Asynchronous Core Object Management Endpoints:
>>>
>>> Additionally, for Asynchronous Core object management endpoints, we
>>> suggest to move Remote/Importer, Publisher, Exporter, Repository
>>> operations under a new core task namespace:
>>> POST@/v3/tasks/core/[importers, publishers,
>>> repositories]/[deletes, updates]/
>>> to be consistent with the asynchronous content actions. This however
>>> seems to conflict with 3).
>>>
>>> Caveats
>>>
>>> Both the current and the proposed design don't conceptually prevent a
>>> repository to have content provided by multiple plug-ins.
>>> Folks don't like Pulp becoming a task management system.
>>>
>>> Alternatives
>>>
>>> Dennis proposes to achieve similar result by adding plug-in content
>>> model hooks:
>>> "add_to_repository_version" and "remove_from_repository_version"
>>> This approach however doesn't seem to solve plug-ins requiring
>>> multiple or no importers and leaves multiple spots in the API, where
>>> repository versions can be created (failing 3); custom plug-in
>>> asynchronous content management endpoints and importers/remotes sync
>>> vs POST at v3/repository/UUID/versions/<#nr>/
>>>
>>> David proposes to namespace all objects under plug-in specific
>>> endpoints. This solves the repository--plug-in purity caveat, allows
>>> great plug-in flexibility and prevents endpoint name collisions.
>>> This approach may allow the plug-ins UX to drift from each other
>>> though, and solves neither the multiple/no importers case, nor the
>>> scattering of repository versions creation endpoints and fails 2)&3).
>>>
>>> I believe this Master/Detail Tasks design to be a reasonable
>>> compromise on the requirements in 1) 2) 3).
>>>
>>> Cheers,
>>> milan
>>>
>>> On Wed, Mar 28, 2018 at 9:38 PM, Dennis Kliban <dkliban at redhat.com>
>>> wrote:
>>> > On Wed, Mar 28, 2018 at 12:20 PM, Ina Panova <ipanova at redhat.com>
>>> wrote:
>>> >>
>>> >> I do not think that it's a valid argument to ban the proposal just
>>> because
>>> >> the proposal will bring changes to the existing plugins. Because:
>>> >>
>>> >> 1) i think it would be fair to think of other plugins and find a
>>> solution
>>> >> all plugins will be happy with --> so we are back to the generic
>>> correctness
>>> >> problem
>>> >> 2) we are not GA, not even Beta, we are free and eligible to make
>>> changes,
>>> >> no promises made yet.
>>> >> 3) if we bring changes *this* exactly is the time to do because we
>>> have
>>> >> just 2 plugins working with basic functionality :)
>>> >>
>>> >> W/r to the "Pulp turning into a task running system" ..and yet that's
>>> a
>>> >> system based on distributed task system
>>> >
>>> >
>>> > Pulp may rely on a tasking system to get work done, but we don't want
>>> users
>>> > thinking of it as a tasking system. We want our users to think of Pulp
>>> as a
>>> > repository management system.
>>> >
>>> >>
>>> >>
>>> >> I suggest to organize a meeting with an agenda in advance that would
>>> list
>>> >> 1) concerns 2) questions
>>> >> I also think it would be valuable to hear Jeff's feedback.
>>> >>
>>> >> If the team finds that meeting idea is beneficial with the believe
>>> that
>>> >> after the meeting we will:
>>> >> 1) i am not naive to believe that we would reach consensus but at
>>> least we
>>> >> would move forward
>>> >> 2) we will decrease the frustration level, because face2face
>>> conversation
>>> >> is more profitable, since we do see faces, have social interaction
>>> and not
>>> >> type onto keyboard and stare at the text on the monitor.
>>> >>
>>> >> then...I will take the action item and schedule the meeting.
>>> >>
>>> >> Meanwhile we'll have time to chew on Easter eggs and give to the
>>> proposal
>>> >> a deeper, unbiased, full of protein (that helps brain thinking)
>>> thought.
>>> >>
>>> >
>>> > I am against having a meeting to discuss this. The discussion on the
>>> list
>>> > has gotten traction and I'd like it to continue on here.
>>> >
>>> >>
>>> >>
>>> >>
>>> >>
>>> >> --------
>>> >> Regards,
>>> >>
>>> >> Ina Panova
>>> >> Software Engineer| Pulp| Red Hat Inc.
>>> >>
>>> >> "Do not go where the path may lead,
>>> >> go instead where there is no path and leave a trail."
>>> >>
>>> >> On Wed, Mar 28, 2018 at 12:11 AM, Brian Bouterse <bbouters at redhat.com
>>> >
>>> >> wrote:
>>> >>>
>>> >>> In terms of removing POST /api/v3/repositories/<uuid>/versions/ from
>>> >>> core, I want to bring it back to the MVP language and REST which
>>> drove the
>>> >>> original design. The MVP has: "As an authenticated user, I can
>>> create a new
>>> >>> version by adding or removing content to the latest version." To
>>> facilitate
>>> >>> that in a generic way, we need a core endpoint to do that, i.e.
>>> >>> /api/v3/repositories/<uuid>/versions/. My concern is that removing
>>> it would
>>> >>> cause us to not fulfill our use cases without requiring more code
>>> from some
>>> >>> plugin writers. Also in terms of REST philosophy, POSTing to the
>>> >>> RespotoryVersion resource to create a new RepositoryVersion is the
>>> >>> traditional url design.
>>> >>>
>>> >>> For pulp_ansible, for example, the generic add/remove functionality
>>> at
>>> >>> core endpoint ^ would be meet all of the pulp_ansible user's needs
>>> because
>>> >>> of the way the pulp_ansible content is modelled. So removing this
>>> endpoint
>>> >>> means more work for pulp_ansible developers w.r.t creating repo
>>> versions and
>>> >>> providing tasks and endpoints. I don't see this extra responsibility
>>> on
>>> >>> plugin writers coming with a clear benefit.
>>> >>>
>>> >>> One idea I liked in this discussion is to have a documented
>>> convention
>>> >>> that encourages plugin writers to put their viewsets in a namespaced
>>> area.
>>> >>> They still need the ability to put them anywhere due to live API
>>> >>> goals/requirements so this would only be a convention for those
>>> tasks they
>>> >>> are offering directly to their users.
>>> >>>
>>> >>>
>>> >>>
>>> >>>
>>> >>>
>>> >>> On Tue, Mar 27, 2018 at 5:40 PM, David Davis <daviddavis at redhat.com>
>>> >>> wrote:
>>> >>>>
>>> >>>> I like Austin’s task proposal in that the plugin writers can focus
>>> on
>>> >>>> serializers and tasks that can be easily integrated with core. That
>>> said, I
>>> >>>> agree on the counterpoints raise about the new resource endpoints
>>> and
>>> >>>> turning much of pulp into a task running system. So I am a bit
>>> mixed on
>>> >>>> which approach is better.
>>> >>>>
>>> >>>> I do think that we should remove POST
>>> >>>> /api/v3/repositories/<uuid>/versions/ from core.
>>> >>>>
>>> >>>>
>>> >>>> David
>>> >>>>
>>> >>>> On Tue, Mar 27, 2018 at 3:49 PM, Austin Macdonald <
>>> amacdona at redhat.com>
>>> >>>> wrote:
>>> >>>>>>
>>> >>>>>> /api/v3/repositories/<uuid>/versions/ endpoint does not perform
>>> plugin
>>> >>>>>> specific validation which can lead to "broken" repository
>>> versions.
>>> >>>>>> Plugin authors don't have any convention to follow when creating
>>> >>>>>> custom REST API endpoints for creating repository versions.
>>> >>>>>> As a result of ^, a user will have a hard time identifying
>>> repository
>>> >>>>>> version creation APIs in different plugins.
>>> >>>>>
>>> >>>>>
>>> >>>>> I agree with these points.
>>> >>>>>>
>>> >>>>>> My first inclination is to disable the ability to POST to
>>> >>>>>> /api/v3/repositories/<uuid>/versions/ and require users to use
>>> the plugin
>>> >>>>>> specific APIs for creating repository versions. However, I think
>>> that
>>> >>>>>> integrators of build systems that produce a variety of content
>>> types would
>>> >>>>>> have a lot more flexibility if they could use a single generic
>>> API endpoint
>>> >>>>>> to create a repository version independent of the content type.
>>> >>>>>>
>>> >>>>>> Let's continue this discussion by answering the following
>>> question:
>>> >>>>>>
>>> >>>>>> Should we disable the ability to POST to
>>> >>>>>> /api/v3/repositories/<uuid>/versions/ and require users to
>>> always use a
>>> >>>>>> plugin specific repository version creation API?
>>> >>>>>
>>> >>>>> Yes, I think we should disable POST to
>>> >>>>> /api/v3/repositories/<uuid>/versions/
>>> >>>>>
>>> >>>>> Simplifying integration is important, but we should not sacrifice
>>> >>>>> correctness enforcement.
>>> >>>>>
>>> >>>>> _______________________________________________
>>> >>>>> Pulp-dev mailing list
>>> >>>>> Pulp-dev at redhat.com
>>> >>>>> https://www.redhat.com/mailman/listinfo/pulp-dev
>>> >>>>>
>>> >>>>
>>> >>>>
>>> >>>> _______________________________________________
>>> >>>> Pulp-dev mailing list
>>> >>>> Pulp-dev at redhat.com
>>> >>>> https://www.redhat.com/mailman/listinfo/pulp-dev
>>> >>>>
>>> >>>
>>> >>>
>>> >>> _______________________________________________
>>> >>> Pulp-dev mailing list
>>> >>> Pulp-dev at redhat.com
>>> >>> https://www.redhat.com/mailman/listinfo/pulp-dev
>>> >>>
>>> >>
>>> >>
>>> >> _______________________________________________
>>> >> Pulp-dev mailing list
>>> >> Pulp-dev at redhat.com
>>> >> https://www.redhat.com/mailman/listinfo/pulp-dev
>>> >>
>>> >
>>> >
>>> > _______________________________________________
>>> > Pulp-dev mailing list
>>> > Pulp-dev at redhat.com
>>> > https://www.redhat.com/mailman/listinfo/pulp-dev
>>> >
>>>
>>
>>
>> _______________________________________________
>> Pulp-dev mailing list
>> Pulp-dev at redhat.com
>> https://www.redhat.com/mailman/listinfo/pulp-dev
>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/pulp-dev/attachments/20180402/2a164c76/attachment.htm>
More information about the Pulp-dev
mailing list