[Pulp-dev] Pulp 3 REST API Challenges

David Davis daviddavis at redhat.com
Tue Apr 17 14:48:20 UTC 2018 

+1 to option 2.

David

On Tue, Apr 17, 2018 at 9:39 AM, Brian Bouterse <bbouters at redhat.com> wrote:

> I believe option 2 would be a good improvement for Pulp's API. It would
> resolve the current problem that a RepoVersion and/or a Publication can be
> created in multiple places in the REST API.
>
> On Fri, Apr 13, 2018 at 3:55 PM, Jeff Ortel <jortel at redhat.com> wrote:
>
>>
>>
>> On 04/12/2018 04:49 PM, Dennis Kliban wrote:
>>
>> On Thu, Apr 12, 2018 at 2:49 PM, Jeff Ortel <jortel at redhat.com> wrote:
>>
>>>
>>>
>>> On 04/11/2018 01:13 PM, Dennis Kliban wrote:
>>>
>>> On Tue, Apr 10, 2018 at 6:44 PM, Jeff Ortel <jortel at redhat.com> wrote:
>>>
>>>>
>>>>
>>>> On 04/10/2018 04:15 PM, Dennis Kliban wrote:
>>>>
>>>> On Tue, Apr 10, 2018 at 2:04 PM, Brian Bouterse <bbouters at redhat.com>
>>>> wrote:
>>>>
>>>>> These are good problem statements. I didn't understand all of the
>>>>> aspects of it, so I put some inline questions.
>>>>>
>>>>> My overall question is: are these related problems? To share my answer
>>>>> to this, I believe the first two problems are related and the third is
>>>>> separate. The classic divide and conquor approach we could use here is to
>>>>> confirm that the problems are unrelated and focus on resolving one of them
>>>>> first.
>>>>>
>>>>>
>>>> I don't think all 3 are related problems. The motivation for grouping
>>>> all together is that a subset of the action endpoints from problem 1 are
>>>> used to create repository versions and Problem 3 is a problem with the
>>>> repository version creation API.
>>>>
>>>>
>>>>>
>>>>> On Mon, Apr 9, 2018 at 3:18 PM, Austin Macdonald <austin at redhat.com>
>>>>> wrote:
>>>>>
>>>>>> Folks,
>>>>>>
>>>>>> Austin, Dennis, and Milan have identified the following issues with
>>>>>> current Pulp3 REST API design:
>>>>>>
>>>>>>    - Action endpoints are problematic.
>>>>>>    - Example POST@/importers/<plugin>/sync/
>>>>>>       - They are non-RESTful and would make client code tightly
>>>>>>       coupled with the server code.
>>>>>>       - These endpoints are inconsistent with the other parts of the
>>>>>>       REST API.
>>>>>>
>>>>>> Is self-consistency really a goal? I think it's a placeholder for
>>>>> consistency for REST since the "rest" of the API is RESTful. After reading
>>>>> parts of Roy Fielding's writeup of the definition of REST I believe "action
>>>>> endpoints are not RESTful" to be a true statement. Maybe "Action endpoints
>>>>> are problematic" should be replaced with "Action endpoints are not RESTful"
>>>>> perhaps and have the self-consistency bullet removed?
>>>>>
>>>>
>>>> +1 to "Action endpoints are not RESTful"
>>>> +1 to removing the self-consistency language
>>>>
>>>>
>>>>>
>>>>>>    - DRF is not being used as intended for action endpoints so we
>>>>>>       have to implement extra code. (against the grain)
>>>>>>
>>>>>> I don't know much about this. Where is the extra code?
>>>>>
>>>>>
>>>>>>
>>>>>>    - We don't have a convention for where plug-in-specific, custom
>>>>>>    repository version creation endpoints.
>>>>>>    - example POST@/api/v3/<where?>/docker/add/
>>>>>>       - needs to be discoverable through the schema
>>>>>>
>>>>>> What does discoverable via the schema ^ mean? Aren't all urls listed
>>>>> in the schema?
>>>>>
>>>>> I think of ^ problem somewhat differently. Yes all urls need to be
>>>>> discoverable (a REST property), but isn't it more of an issue that the urls
>>>>> which produce repo versions can't be identified distinctly from any other
>>>>> plugin-contributed url? To paraphrase this perspective: making a repo
>>>>> version is strewn about throughout the API in random places which is a bad
>>>>> user experience. Is that what is motivation url discovery?
>>>>>
>>>>>
>>>>
>>>> Yes. I envision a CLI that can discover new plugin
>>>> repository-version-creating functionality without having to install new
>>>> client packages. Allowing plugin writers to add endpoints in arbitrary
>>>> places for creating repository versions will make it impossible for the
>>>> client to know what all the possible ways of creating a repository version
>>>> are.
>>>>
>>>>
>>>>>
>>>>>>    - For direct repository version creation, plugins are not
>>>>>>    involved.
>>>>>>    - validation correctness problem: https://pulp.plan.io/issues/3541
>>>>>>       - example: POST@/api/v3/repositories/<repository_id>/versions/
>>>>>>
>>>>>> I agree with this problem statement. In terms of scope it affects
>>>>> some plugin writers but not all.
>>>>>
>>>>
>>>> I think it affects all plugin writers. Even the File plugin needs to
>>>> provide some validation when creating a repository version. Right now you
>>>> can add a FileContent with the same relative path as another FileContent in
>>>> the repository version. This should not be possible because it's not a
>>>> valid combination of FileContent units in the same repository version.
>>>>
>>>>
>>>> Not necessarily.  Two files with the same relative path will have
>>>> different digest (different content).  The assumption that they both cannot
>>>> be in the same repository makes assumptions about how the repository is
>>>> used which I don't think is a good idea.  Image two different versions of
>>>> abc.iso.
>>>>
>>>>
>>> Why is it bad to assume that a repository version is going to be
>>> published? What are the other ways to use a repository version?
>>>
>>>
>>> The repository may not be publish and/or may not be published by the
>>> FilePublisher in the file plugin project.  A user may want to sync and
>>> store many version of an iso in the repository and then selectively
>>> *add* a specific version to another repository for promotion work
>>> flows.  Also, the user could use another (custom) publisher that deals
>>> differently with multiple files with the same path in the repository.  The
>>> FilePublisher currently publishes the newest.  My point being, we, really
>>> cannot assume how the repository will be used or which publisher *may*
>>> publish it.
>>>
>>>
>> The problem was initially stated as "For direct repository version
>> creation, plugins are not involved". It sounds like you disagree that this
>> is a problem.
>>
>>
>> Yes.  Definitely, agreed.
>>
>> Can you confirm this by telling us if plugins should be able to provide
>> validation for this API provided by core?
>>
>>
>> Plugins participating in core endpoints is different, broader discussion.
>>
>> The following is not aimed at you dkliban :)
>>
>> We need to decide if we want to return to the pulp2 pattern whereby the
>> core delegates behavior to plugins via the plugin API.  Or, continue down
>> the pulp3 path whereby operations involving plugins are contributed to the
>> API by each plugin (not making a value judgment). Also, I value consistency
>> in APIs and don't think these approaches should be mixed (with the
>> exception of content related live-API).  Consistency in APIs reflect both a
>> thoughtful, mature design and provides a better user experience. I'm sure
>> everyone has cursed APIs that did things every-which-way. I don't think
>> there is any difference between creating a repository version via sync or
>> creating a version with a list of content to add/remove.  And to a lesser
>> degree publishing.  We should either POST to the /publications/ endpoint
>> for creating a publication (core API), *or* users should POST to the
>> plugin contributed endpoint (as currently) for publishing.
>>
>> Seems to me, there are 2 high-level choices:
>>
>> *1.  Core endpoints do not delegate/redirect to plugins.*
>>      - POST to /RepositoryVersion/ is removed.
>>      - POST to /Publications/  (stays gone)
>>      - Plugins provide endpoints for sync and other to create new
>> repository versions.
>>      - Plugins provide endpoints for creating Publications (publishing).
>>
>> *2. Core delegates behavior to plugins for those endpoints requiring
>> plugin participation.*
>>     - POST to /RepositoryVersion/ is the only way to create a repository
>> version.
>>     - POST to /Publications/  is the only way to create a Publication
>> (publish).
>>     - The POST parameters or body includes enough information so that
>> core can select a plugin.
>>     - Either the entire POST is passed along to the plugin, *or* the
>> plugin implements an API that's used by
>>       core for pre-defined participation.
>>
>> There have been proposals on how both #1 and #2 can be achieved. 'm
>> wondering if we can even agree on one of these two approaches.
>>
>>
>> A File repository version cannot be properly published if it contains 2
>>> FileContent units that both have the same relative path. The publisher has
>>> to decide which FileContent to publish at the relative path. That decision
>>> cannot be made intelligently by the publisher. The decision on which
>>> content unit to include in the repository version rests with the user that
>>> is creating the repository version. When a user tries to create a
>>> repository version with a FileContent that has the same relative path as
>>> another FileContent that was added previously, Pulp needs to inform the
>>> user that there is a conflict (and not create the repositiory version).
>>> This validation can only be provided by the File plugin.
>>>
>>>
>>>>
>>>>
>>>>>
>>>>>
>>>>>> We would like to get feedback on these issues being sound and worth
>>>>>> resolving before we resume particular solution discussion[1].
>>>>>>
>>>>>> Thanks,
>>>>>> Austin, Dennis, and Milan
>>>>>>
>>>>>> [1] https://www.redhat.com/archives/pulp-dev/2018-March/msg00066.html
>>>>>>
>>>>>>
>>>>>> _______________________________________________
>>>>>> Pulp-dev mailing list
>>>>>> Pulp-dev at redhat.com
>>>>>> https://www.redhat.com/mailman/listinfo/pulp-dev
>>>>>>
>>>>>>
>>>>>
>>>>> _______________________________________________
>>>>> Pulp-dev mailing list
>>>>> Pulp-dev at redhat.com
>>>>> https://www.redhat.com/mailman/listinfo/pulp-dev
>>>>>
>>>>>
>>>>
>>>>
>>>> _______________________________________________
>>>> Pulp-dev mailing listPulp-dev at redhat.comhttps://www.redhat.com/mailman/listinfo/pulp-dev
>>>>
>>>>
>>>>
>>>> _______________________________________________
>>>> Pulp-dev mailing list
>>>> Pulp-dev at redhat.com
>>>> https://www.redhat.com/mailman/listinfo/pulp-dev
>>>>
>>>>
>>>
>>>
>>> _______________________________________________
>>> Pulp-dev mailing list
>>> Pulp-dev at redhat.com
>>> https://www.redhat.com/mailman/listinfo/pulp-dev
>>>
>>>
>>
>>
>> _______________________________________________
>> Pulp-dev mailing list
>> Pulp-dev at redhat.com
>> https://www.redhat.com/mailman/listinfo/pulp-dev
>>
>>
>
> _______________________________________________
> Pulp-dev mailing list
> Pulp-dev at redhat.com
> https://www.redhat.com/mailman/listinfo/pulp-dev
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/pulp-dev/attachments/20180417/4e2d6fc0/attachment.htm>

More information about the Pulp-dev mailing list