[Pulp-dev] Plugin relationship to tasks

Milan Kovacik mkovacik at redhat.com
Thu Mar 29 20:18:37 UTC 2018 

Folks,

I'd like to summarize where we've gotten so far. Our problem statement
is threefold:

1) Content operations have to be routed thru particular plug-ins i.e
no repository version can be created without a plug-in involvement, to
avoid repository content consistency/correctness issues
2) API specialization i.e keeping core concepts above plug-in specific
concepts; e.g CRUD@/v3/repositories/ v.s CRUD at docker/adds/
3) API consistency i.e preferably where I create it is where I manage
it; e.g. w/r/t asynchronous action endpoints creating Task resources
vs a Task management endpoint

Since there is a significant intersection between the asynchronous
action endpoints and endpoints manipulating repository versions
asynchronously, me and Austin propose to convert the asynchronous
action endpoints to Task management endpoints, in full control of
plug-ins, in particular:

Asynchronous Repository Version Management Endpoints:

  Repository synchronisation and publication and addition(deletion) of
content into(from) a repository version would all be treated as Task
objects primarily, residing in the plug-in namespace:
      POST@/v3/tasks/<plug-in>/[syncs, publishes, additions, deletions]/
  because all these operations share similar call signature, every one
of these actions:

  * creates a Task object
  * the task, while executing, creates a new repository version and
locks similar resources
  * has to be controlled by a particular plug-in i.e may require
additional, plug-in specific settings
  * requires either a repository URI or repository version URI to operate on
  * requires generic action configuration e.g almost every sync
requires (at least one) importer

 As a result:
   * the asynchronous content management Task is now the nexus where
the configuration (repository, plug-in, origin) meets the action
   * these complex tasks that actually do the heavy lifting have a
clear management endpoint
   * this change effectively renders the
@/v3/repositories/<UUID>/versions/<#nr>/ endpoint read-only and
documents the intention that no repository version is ever created
without an actual plug-in involvement
   * a plug-in can expose custom asynchronous content endpoints, that
manipulate repository versions, aligned with the more generic actions:
           POST@/v3/tasks/<plug-in>/[<custom asynchronous repository
version management action>,...]/
   * the design aligns better with the DRF allowing (nested)
serializers to control (and document) the endpoints in the direction
from the more generic to the more specific implementation

Asynchronous Core Object Management Endpoints:

  Additionally, for Asynchronous Core object management endpoints, we
suggest to move Remote/Importer, Publisher, Exporter, Repository
operations under a new core task namespace:
       POST@/v3/tasks/core/[importers, publishers,
repositories]/[deletes, updates]/
  to be consistent with the asynchronous content actions. This however
seems to conflict with 3).

Caveats

Both the current and the proposed design don't conceptually prevent a
repository to have content provided by multiple plug-ins.
Folks don't like Pulp becoming a task management system.

Alternatives

  Dennis proposes to achieve similar result by adding plug-in content
model hooks:
      "add_to_repository_version" and "remove_from_repository_version"
  This approach however doesn't seem to solve plug-ins requiring
multiple or no importers and leaves multiple spots in the API, where
repository versions can be created (failing 3); custom plug-in
asynchronous content management endpoints and importers/remotes sync
vs POST at v3/repository/UUID/versions/<#nr>/

  David proposes to namespace all objects under plug-in specific
endpoints. This solves the repository--plug-in purity caveat, allows
great plug-in flexibility and prevents endpoint name collisions.
 This approach may allow the plug-ins UX to drift from each other
though, and solves neither the multiple/no importers case, nor the
scattering of repository versions creation endpoints and fails 2)&3).

I believe this Master/Detail Tasks design to be a reasonable
compromise on the requirements in 1) 2) 3).

Cheers,
milan

On Wed, Mar 28, 2018 at 9:38 PM, Dennis Kliban <dkliban at redhat.com> wrote:
> On Wed, Mar 28, 2018 at 12:20 PM, Ina Panova <ipanova at redhat.com> wrote:
>>
>> I do not think that it's a valid argument to ban the proposal just because
>> the proposal will bring changes to the existing plugins. Because:
>>
>> 1) i think it would be fair to think of other plugins and find a solution
>> all plugins will be happy with --> so we are back to the generic correctness
>> problem
>> 2) we are not GA, not even Beta, we are free and eligible to make changes,
>> no promises made yet.
>> 3) if we bring changes *this* exactly is the time to do because we have
>> just 2 plugins working with basic functionality :)
>>
>> W/r to the "Pulp turning into a task running system" ..and yet that's a
>> system based on distributed task system
>
>
> Pulp may rely on a tasking system to get work done, but we don't want users
> thinking of it as a tasking system. We want our users to think of Pulp as a
> repository management system.
>
>>
>>
>> I suggest to organize a meeting with an agenda in advance that would list
>> 1) concerns 2) questions
>> I also think it would be valuable to hear Jeff's feedback.
>>
>> If the team finds that meeting idea is beneficial with the believe that
>> after the meeting we will:
>> 1) i am not naive to believe that we would reach consensus but at least we
>> would move forward
>> 2) we will decrease the frustration level, because face2face conversation
>> is more profitable, since we do see faces, have social interaction and not
>> type onto keyboard and stare at the text on the monitor.
>>
>> then...I will take the action item and schedule the meeting.
>>
>> Meanwhile we'll have time to chew on Easter eggs and give to the proposal
>> a deeper, unbiased, full of protein (that helps brain thinking) thought.
>>
>
> I am against having a meeting to discuss this. The discussion on the list
> has gotten traction and I'd like it to continue on here.
>
>>
>>
>>
>>
>> --------
>> Regards,
>>
>> Ina Panova
>> Software Engineer| Pulp| Red Hat Inc.
>>
>> "Do not go where the path may lead,
>>  go instead where there is no path and leave a trail."
>>
>> On Wed, Mar 28, 2018 at 12:11 AM, Brian Bouterse <bbouters at redhat.com>
>> wrote:
>>>
>>> In terms of removing POST /api/v3/repositories/<uuid>/versions/ from
>>> core, I want to bring it back to the MVP language and REST which drove the
>>> original design. The MVP has: "As an authenticated user, I can create a new
>>> version by adding or removing content to the latest version." To facilitate
>>> that in a generic way, we need a core endpoint to do that, i.e.
>>> /api/v3/repositories/<uuid>/versions/. My concern is that removing it would
>>> cause us to not fulfill our use cases without requiring more code from some
>>> plugin writers. Also in terms of REST philosophy, POSTing to the
>>> RespotoryVersion resource to create a new RepositoryVersion is the
>>> traditional url design.
>>>
>>> For pulp_ansible, for example, the generic add/remove functionality at
>>> core endpoint ^ would be meet all of the pulp_ansible user's needs because
>>> of the way the pulp_ansible content is modelled. So removing this endpoint
>>> means more work for pulp_ansible developers w.r.t creating repo versions and
>>> providing tasks and endpoints. I don't see this extra responsibility on
>>> plugin writers coming with a clear benefit.
>>>
>>> One idea I liked in this discussion is to have a documented convention
>>> that encourages plugin writers to put their viewsets in a namespaced area.
>>> They still need the ability to put them anywhere due to live API
>>> goals/requirements so this would only be a convention for those tasks they
>>> are offering directly to their users.
>>>
>>>
>>>
>>>
>>>
>>> On Tue, Mar 27, 2018 at 5:40 PM, David Davis <daviddavis at redhat.com>
>>> wrote:
>>>>
>>>> I like Austin’s task proposal in that the plugin writers can focus on
>>>> serializers and tasks that can be easily integrated with core. That said, I
>>>> agree on the counterpoints raise about the new resource endpoints and
>>>> turning much of pulp into a task running system. So I am a bit mixed on
>>>> which approach is better.
>>>>
>>>> I do think that we should remove POST
>>>> /api/v3/repositories/<uuid>/versions/ from core.
>>>>
>>>>
>>>> David
>>>>
>>>> On Tue, Mar 27, 2018 at 3:49 PM, Austin Macdonald <amacdona at redhat.com>
>>>> wrote:
>>>>>>
>>>>>> /api/v3/repositories/<uuid>/versions/ endpoint does not perform plugin
>>>>>> specific validation which can lead to "broken" repository versions.
>>>>>> Plugin authors don't have any convention to follow when creating
>>>>>> custom REST API endpoints for creating repository versions.
>>>>>> As a result of ^, a user will have a hard time identifying repository
>>>>>> version creation APIs in different plugins.
>>>>>
>>>>>
>>>>> I agree with these points.
>>>>>>
>>>>>> My first inclination is to disable the ability to POST to
>>>>>> /api/v3/repositories/<uuid>/versions/ and require users to use the plugin
>>>>>> specific APIs for creating repository versions. However, I think that
>>>>>> integrators of build systems that produce a variety of content types would
>>>>>> have a lot more flexibility if they could use a single generic API endpoint
>>>>>> to create a repository version independent of the content type.
>>>>>>
>>>>>> Let's continue this discussion by answering the following question:
>>>>>>
>>>>>> Should we disable the ability to POST to
>>>>>> /api/v3/repositories/<uuid>/versions/ and require users to always use a
>>>>>> plugin specific repository version creation API?
>>>>>
>>>>> Yes, I think we should disable POST to
>>>>> /api/v3/repositories/<uuid>/versions/
>>>>>
>>>>> Simplifying integration is important, but we should not sacrifice
>>>>> correctness enforcement.
>>>>>
>>>>> _______________________________________________
>>>>> Pulp-dev mailing list
>>>>> Pulp-dev at redhat.com
>>>>> https://www.redhat.com/mailman/listinfo/pulp-dev
>>>>>
>>>>
>>>>
>>>> _______________________________________________
>>>> Pulp-dev mailing list
>>>> Pulp-dev at redhat.com
>>>> https://www.redhat.com/mailman/listinfo/pulp-dev
>>>>
>>>
>>>
>>> _______________________________________________
>>> Pulp-dev mailing list
>>> Pulp-dev at redhat.com
>>> https://www.redhat.com/mailman/listinfo/pulp-dev
>>>
>>
>>
>> _______________________________________________
>> Pulp-dev mailing list
>> Pulp-dev at redhat.com
>> https://www.redhat.com/mailman/listinfo/pulp-dev
>>
>
>
> _______________________________________________
> Pulp-dev mailing list
> Pulp-dev at redhat.com
> https://www.redhat.com/mailman/listinfo/pulp-dev
>

More information about the Pulp-dev mailing list