<div dir="ltr">I'm concerned that this conversation is too broad. I'm willing to brainstorm ideas because this is an important topic, but I also hope that at some point we can come back to the proposal for master/detail tasks.<div> </div><div>This idea is interesting, because it provides a novel way to connect the plugins to pulpcore calls. The views that plugins implement would not be normal views, they would operate more like OPTIONS schema views. I'll try to show how I think this would look from a user's perspective:</div><div> </div><div>File Sync:</div><div> </div><div>The user doesn't know how to sync, so they do a GET request:</div><div>GET /v3/versionactions/</div><div> </div><div>This returns a list of possible actions, one of them being v3/versionactions/file/sync. To see how to use that they do another GET request:</div><div>GET /v3versionactions/file/sync/</div><div> </div><div>This returns the description of the file-sync action and the parameters necessary (importer, repository)</div><div> </div><div>POST v3/repositories/1234/versions/ importer=importer_href, repository=repository_href, action=v3/versionactions/file/sync/ </div><div> </div><div>I have some practical questions:</div><div><ul><li>What does the documentation for v3/repositories/1234/versions/ look like?</li><ul><li>Are the parameters listed, or does the user have to use v3/versionactions/sync/ to retrieve their schema?</li></ul><li>How does the RepositoryVersionsViewSet use the versionaction?</li><ul><li>How are the params `importer`, and `repository` validated? </li><ul><li>Synchronously</li><li>Asynchronously</li></ul><li>How is the celery task connected and dispatched?</li></ul></ul><div>Here are my initial thoughts:</div></div><div><ul><li>I like that this brings us back to "to create a new RepositoryVersion, POST to versions". </li><ul><li>I especially like that all versions are created at a single endpoint.</li></ul><li>I don't like that the plugins are creating views that are not intended to be used as endpoints. These views are only "hit" for documentation, and they are used as parameters to other tasks. </li><ul><li>This could be confusing for users</li><li>This is a different type of work than the rest of plugin writing, so it will have its own learning curve</li><li>This would not (speculation) play nice with auto-documentation. The autodocumentation would not know about the possible parameters for the actions, so the users would have to user GET v3/versionactions/ instead of docs.</li><li>This would not <span style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:small;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:left;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline">(speculation) play nice with auto-bindings and clients. Same problem as ^</li><li>GET request does not retrieve objects, it retrieves schema</li></ul></ul></div></div><div class="gmail_extra"> <div class="gmail_quote">On Mon, Apr 2, 2018 at 10:26 AM, Dennis Kliban <<a href="mailto:dkliban@redhat.com" target="_blank">dkliban@redhat.com</a>> wrote: <blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div><div><div><div>Here is another idea that would remove the verbs from the REST API and would provide a way to namespace plugin provided tasks. </div><div> We could add 2 additional resource types for plugin writers to provide: versionaction and publicationaction. These REST resources would only support GET requests. The response would contain a description of the action and a description of the parameters this action accepts. Plugin writers would create these resources by extending an ActionView with a custom get() method for providing users with a description and also an action() method which would contain the code that needs to run asynchronously. Every repository version would be created using the repository version API. POST requests to the repository version API would accept a 'versionaction' href along with a dictionary of parameters. Users would then receive a 202 response with a task href that can be used to monitor the progress of the asynchronous repository version creation. Every publication would be created using the publication API. POST requests to the publication API would accept a 'publicationaction' href along with a dictionary of parameters. Users would then receive a 202 response with a task href that can be used to monitor the progress of the asynchronous publication creation. </div><div>Here are some REST API changes this would bring: </div><div> /api/v3/versionactions/ - returns a list of all the action descriptions for creating repository versions </div><div>/api/v3/versionactions/pulpcore/addremove/ - returns a description of the addremove action provided by pulpcore. Something like this: { 'href': '<a href="http://localhost/api/v3/versionactions/pulpcore/addremove/" target="_blank">http://localhost/api/v3/versionactions/pulpcore/addremove/</a>', 'description': 'Creates a repository version by adding and removing content units from the latest repository version.', 'parameters': { 'add_content_units': 'list of content to add', </div><div> 'remove_content_units': 'list of content to remove' } } </div>/api/v3/versionactions/<plugin>/sync/ - accepts GET and returns a description of the sync action of the plugin. Something like this: <div>{ 'href': '<a href="http://localhost/api/v3/versionactions/file/sync/" target="_blank">http://localhost/api/v3/versionactions/file/sync/</a>', 'description': 'Creates a repository version by syncing content from a remote file repository.', 'parameters': { 'remote': 'href of the remote to sync from' } }</div> </div>/api/v3/publicationactions/ - returns a list of all the actions for creating publications </div>/api/v3/publicationactions/<plugin>/publish/ - accepts GET and returns a description of the publish action for the file plugin. Something like this: { 'href': '<a href="http://localhost/api/v3/versionactions/file/publish/" target="_blank">http://localhost/api/v3/versionactions/file/publish/</a>', 'description': 'Create a publication from a file repository version.', 'parameters': { 'repository': 'href of the repository to publish. The latest repository version will be published. Can only be specified if repository_version is not specified.' 'repository_version': 'href of the repository version to publish. Can only be specified if repository_version is not specified. Can only be specified if repository is not specified.' } } </div><div>/api/v3/repository/<uuid>/versions/ - accepts POST requests with 'versionaction' href and 'parameters' dictionary. Returns 202 with a task href. </div><div>/api/v3/publications/ - accepts POST requests with 'publicationaction' href and 'parameters' dictionary. Returns 202 with a task href. </div> </div><div class="HOEnZb"><div class="h5"><div class="gmail_extra"> <div class="gmail_quote">On Thu, Mar 29, 2018 at 4:18 PM, Milan Kovacik <<a href="mailto:mkovacik@redhat.com" target="_blank">mkovacik@redhat.com</a>> wrote: <blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Folks, I'd like to summarize where we've gotten so far. Our problem statement is threefold: 1) Content operations have to be routed thru particular plug-ins i.e no repository version can be created without a plug-in involvement, to avoid repository content consistency/correctness issues 2) API specialization i.e keeping core concepts above plug-in specific concepts; e.g CRUD@/v3/repositories/ v.s CRUD@docker/adds/ 3) API consistency i.e preferably where I create it is where I manage it; e.g. w/r/t asynchronous action endpoints creating Task resources vs a Task management endpoint Since there is a significant intersection between the asynchronous action endpoints and endpoints manipulating repository versions asynchronously, me and Austin propose to convert the asynchronous action endpoints to Task management endpoints, in full control of plug-ins, in particular: Asynchronous Repository Version Management Endpoints: Repository synchronisation and publication and addition(deletion) of content into(from) a repository version would all be treated as Task objects primarily, residing in the plug-in namespace: POST@/v3/tasks/<plug-in>/[syncs, publishes, additions, deletions]/ because all these operations share similar call signature, every one of these actions: * creates a Task object * the task, while executing, creates a new repository version and locks similar resources * has to be controlled by a particular plug-in i.e may require additional, plug-in specific settings * requires either a repository URI or repository version URI to operate on * requires generic action configuration e.g almost every sync requires (at least one) importer As a result: * the asynchronous content management Task is now the nexus where the configuration (repository, plug-in, origin) meets the action * these complex tasks that actually do the heavy lifting have a clear management endpoint * this change effectively renders the @/v3/repositories/<UUID>/versions/<#nr>/ endpoint read-only and documents the intention that no repository version is ever created without an actual plug-in involvement * a plug-in can expose custom asynchronous content endpoints, that manipulate repository versions, aligned with the more generic actions: POST@/v3/tasks/<plug-in>/[<custom asynchronous repository version management action>,...]/ * the design aligns better with the DRF allowing (nested) serializers to control (and document) the endpoints in the direction from the more generic to the more specific implementation Asynchronous Core Object Management Endpoints: Additionally, for Asynchronous Core object management endpoints, we suggest to move Remote/Importer, Publisher, Exporter, Repository operations under a new core task namespace: POST@/v3/tasks/core/[importers, publishers, repositories]/[deletes, updates]/ to be consistent with the asynchronous content actions. This however seems to conflict with 3). Caveats Both the current and the proposed design don't conceptually prevent a repository to have content provided by multiple plug-ins. Folks don't like Pulp becoming a task management system. Alternatives Dennis proposes to achieve similar result by adding plug-in content model hooks: "add_to_repository_version" and "remove_from_repository_version" This approach however doesn't seem to solve plug-ins requiring multiple or no importers and leaves multiple spots in the API, where repository versions can be created (failing 3); custom plug-in asynchronous content management endpoints and importers/remotes sync vs POST@v3/repository/UUID/versions/<#nr>/ David proposes to namespace all objects under plug-in specific endpoints. This solves the repository--plug-in purity caveat, allows great plug-in flexibility and prevents endpoint name collisions. This approach may allow the plug-ins UX to drift from each other though, and solves neither the multiple/no importers case, nor the scattering of repository versions creation endpoints and fails 2)&3). I believe this Master/Detail Tasks design to be a reasonable compromise on the requirements in 1) 2) 3). Cheers, milan <div class="m_-6736672718274711553HOEnZb"><div class="m_-6736672718274711553h5"> On Wed, Mar 28, 2018 at 9:38 PM, Dennis Kliban <<a href="mailto:dkliban@redhat.com" target="_blank">dkliban@redhat.com</a>> wrote: > On Wed, Mar 28, 2018 at 12:20 PM, Ina Panova <<a href="mailto:ipanova@redhat.com" target="_blank">ipanova@redhat.com</a>> wrote: >> >> I do not think that it's a valid argument to ban the proposal just because >> the proposal will bring changes to the existing plugins. Because: >> >> 1) i think it would be fair to think of other plugins and find a solution >> all plugins will be happy with --> so we are back to the generic correctness >> problem >> 2) we are not GA, not even Beta, we are free and eligible to make changes, >> no promises made yet. >> 3) if we bring changes *this* exactly is the time to do because we have >> just 2 plugins working with basic functionality :) >> >> W/r to the "Pulp turning into a task running system" ..and yet that's a >> system based on distributed task system > > > Pulp may rely on a tasking system to get work done, but we don't want users > thinking of it as a tasking system. We want our users to think of Pulp as a > repository management system. > >> >> >> I suggest to organize a meeting with an agenda in advance that would list >> 1) concerns 2) questions >> I also think it would be valuable to hear Jeff's feedback. >> >> If the team finds that meeting idea is beneficial with the believe that >> after the meeting we will: >> 1) i am not naive to believe that we would reach consensus but at least we >> would move forward >> 2) we will decrease the frustration level, because face2face conversation >> is more profitable, since we do see faces, have social interaction and not >> type onto keyboard and stare at the text on the monitor. >> >> then...I will take the action item and schedule the meeting. >> >> Meanwhile we'll have time to chew on Easter eggs and give to the proposal >> a deeper, unbiased, full of protein (that helps brain thinking) thought. >> > > I am against having a meeting to discuss this. The discussion on the list > has gotten traction and I'd like it to continue on here. > >> >> >> >> >> -------- >> Regards, >> >> Ina Panova >> Software Engineer| Pulp| Red Hat Inc. >> >> "Do not go where the path may lead, >> go instead where there is no path and leave a trail." >> >> On Wed, Mar 28, 2018 at 12:11 AM, Brian Bouterse <<a href="mailto:bbouters@redhat.com" target="_blank">bbouters@redhat.com</a>> >> wrote: >>> >>> In terms of removing POST /api/v3/repositories/<uuid>/versions/ from >>> core, I want to bring it back to the MVP language and REST which drove the >>> original design. The MVP has: "As an authenticated user, I can create a new >>> version by adding or removing content to the latest version." To facilitate >>> that in a generic way, we need a core endpoint to do that, i.e. >>> /api/v3/repositories/<uuid>/versions/. My concern is that removing it would >>> cause us to not fulfill our use cases without requiring more code from some >>> plugin writers. Also in terms of REST philosophy, POSTing to the >>> RespotoryVersion resource to create a new RepositoryVersion is the >>> traditional url design. >>> >>> For pulp_ansible, for example, the generic add/remove functionality at >>> core endpoint ^ would be meet all of the pulp_ansible user's needs because >>> of the way the pulp_ansible content is modelled. So removing this endpoint >>> means more work for pulp_ansible developers w.r.t creating repo versions and >>> providing tasks and endpoints. I don't see this extra responsibility on >>> plugin writers coming with a clear benefit. >>> >>> One idea I liked in this discussion is to have a documented convention >>> that encourages plugin writers to put their viewsets in a namespaced area. >>> They still need the ability to put them anywhere due to live API >>> goals/requirements so this would only be a convention for those tasks they >>> are offering directly to their users. >>> >>> >>> >>> >>> >>> On Tue, Mar 27, 2018 at 5:40 PM, David Davis <<a href="mailto:daviddavis@redhat.com" target="_blank">daviddavis@redhat.com</a>> >>> wrote: >>>> >>>> I like Austin’s task proposal in that the plugin writers can focus on >>>> serializers and tasks that can be easily integrated with core. That said, I >>>> agree on the counterpoints raise about the new resource endpoints and >>>> turning much of pulp into a task running system. So I am a bit mixed on >>>> which approach is better. >>>> >>>> I do think that we should remove POST >>>> /api/v3/repositories/<uuid>/versions/ from core. >>>> >>>> >>>> David >>>> >>>> On Tue, Mar 27, 2018 at 3:49 PM, Austin Macdonald <<a href="mailto:amacdona@redhat.com" target="_blank">amacdona@redhat.com</a>> >>>> wrote: >>>>>> >>>>>> /api/v3/repositories/<uuid>/versions/ endpoint does not perform plugin >>>>>> specific validation which can lead to "broken" repository versions. >>>>>> Plugin authors don't have any convention to follow when creating >>>>>> custom REST API endpoints for creating repository versions. >>>>>> As a result of ^, a user will have a hard time identifying repository >>>>>> version creation APIs in different plugins. >>>>> >>>>> >>>>> I agree with these points. >>>>>> >>>>>> My first inclination is to disable the ability to POST to >>>>>> /api/v3/repositories/<uuid>/versions/ and require users to use the plugin >>>>>> specific APIs for creating repository versions. However, I think that >>>>>> integrators of build systems that produce a variety of content types would >>>>>> have a lot more flexibility if they could use a single generic API endpoint >>>>>> to create a repository version independent of the content type. >>>>>> >>>>>> Let's continue this discussion by answering the following question: >>>>>> >>>>>> Should we disable the ability to POST to >>>>>> /api/v3/repositories/<uuid>/versions/ and require users to always use a >>>>>> plugin specific repository version creation API? >>>>> >>>>> Yes, I think we should disable POST to >>>>> /api/v3/repositories/<uuid>/versions/ >>>>> >>>>> Simplifying integration is important, but we should not sacrifice >>>>> correctness enforcement. >>>>> >>>>> _______________________________________________ >>>>> Pulp-dev mailing list >>>>> <a href="mailto:Pulp-dev@redhat.com" target="_blank">Pulp-dev@redhat.com</a> >>>>> <a href="https://www.redhat.com/mailman/listinfo/pulp-dev" rel="noreferrer" target="_blank">https://www.redhat.com/mailman/listinfo/pulp-dev</a> >>>>> >>>> >>>> >>>> _______________________________________________ >>>> Pulp-dev mailing list >>>> <a href="mailto:Pulp-dev@redhat.com" target="_blank">Pulp-dev@redhat.com</a> >>>> <a href="https://www.redhat.com/mailman/listinfo/pulp-dev" rel="noreferrer" target="_blank">https://www.redhat.com/mailman/listinfo/pulp-dev</a> >>>> >>> >>> >>> _______________________________________________ >>> Pulp-dev mailing list >>> <a href="mailto:Pulp-dev@redhat.com" target="_blank">Pulp-dev@redhat.com</a> >>> <a href="https://www.redhat.com/mailman/listinfo/pulp-dev" rel="noreferrer" target="_blank">https://www.redhat.com/mailman/listinfo/pulp-dev</a> >>> >> >> >> _______________________________________________ >> Pulp-dev mailing list >> <a href="mailto:Pulp-dev@redhat.com" target="_blank">Pulp-dev@redhat.com</a> >> <a href="https://www.redhat.com/mailman/listinfo/pulp-dev" rel="noreferrer" target="_blank">https://www.redhat.com/mailman/listinfo/pulp-dev</a> >> > > > _______________________________________________ > Pulp-dev mailing list > <a href="mailto:Pulp-dev@redhat.com" target="_blank">Pulp-dev@redhat.com</a> > <a href="https://www.redhat.com/mailman/listinfo/pulp-dev" rel="noreferrer" target="_blank">https://www.redhat.com/mailman/listinfo/pulp-dev</a> > </div></div></blockquote></div> </div> </div></div> _______________________________________________ Pulp-dev mailing list <a href="mailto:Pulp-dev@redhat.com">Pulp-dev@redhat.com</a> <a href="https://www.redhat.com/mailman/listinfo/pulp-dev" rel="noreferrer" target="_blank">https://www.redhat.com/mailman/listinfo/pulp-dev</a> </blockquote></div> </div>