[Pulp-dev] Pulp api seemingly incompatible with generated bindings

Dennis Kliban dkliban at redhat.com
Wed Jul 11 20:03:19 UTC 2018

On Mon, Jul 9, 2018 at 9:16 PM, Justin Sherrill <jsherril at redhat.com> wrote:

> Hi all,
> I finally had some time to play with these and I do have some concerns.
> First i'd say that Dennis' work to get the swagger generation working with
> href's seemed to work really well!  I did find one small bug(1) that may be
> isolated to the ruby bindings, but seems like a serialization issue
> potentially.
Thank you for filing the issue. We are going to add it to our next sprint
on Friday.

> However I do have some concerns.  According to Dennis, upstream swagger
> does not seem open to accepting this change that utilizes href's in its api
> calls.  This leaves us all in a bit of a pickle as:
> 1) pulp seems to suggest to use hrefs (and may remove ids in the future?)
> 2) swagger does not (and will never?) support referencing objects by
> href's in the future
I would like to clear up the language we are using.

swagger[0] is the former name for OpenAPI 2.0, which is a standard for
richly defining REST API schema.

Pulp 3 has a REST API endpoint that serves it's OpenAPI 2.0 schema. We are
constantly working to improve the accuracy and level of detail of this
OpenAPI 2.0 schema. We use the schema to generate our REST API
documentation. Our CLI is going to rely on the schema to inform the user
what all interactions with the server are available and what parameters are
needed. We have recently decided to start using pyswagger[1] as the client
library in the CLI. I have not been able to find a similar client for Ruby.
I welcome others to search for one.

Another solution for taking advantage of the OpenAPI 2.0 schema definition
on the client side is to generate a client using swagger-codegen[2].
swagger-codegen is a tool that takes OpenAPI 2.0 schema definition as an
input and produces a client library (bindings) in a language of your
choice. It does so by taking parameters it finds in the schema and inserts
them into templates for the specific language requested. The templates
bundled with swagger-codegen assume that the user will use "path
parameters" to identify resources. This behavior is not compatible with
Pulp because Pulp identifies resources using URIs.

I asked in #swagger on freenode about changing the behavior of
swagger-codegen and I was recommended to update the templates for Ruby,
Python, and any other languages we want to support. That is how users of
swagger-codegen are supposed to customize the behavior of swagger-codegen.

Instead of modifying each language's template, I found it easier to modify
the behavior of swagger-codegen. As a result, we can generate working
bindings in all the languages supported by swagger-codegen.

[0] https://github.com/OAI/OpenAPI-Specification/blob/master/versions/2.0.md
[1] https://github.com/pyopenapi/pyswagger
[2] https://swagger.io/tools/swagger-codegen/

> Thus for katello to use href's, we would need to use an altered version of
> swagger that someone maintains.  While the patch is small, I'm not sure
> either of our teams are super well positioned to maintain something like
> this long term.
I based my changes on the master branch, which will be the next, 2.4.0,
release of swagger-codegen. I have also checked that a similar patch can be
used for modifying the behavior on the 3.0.0 branch. We should not have
compatibility problems with swagger-codegen 2.y and a similar approach can
be used with 3.y when that becomes the stable release.

> It might make sense for the pulp team to maintain this, as the
> requirements are being suggested by the pulp team, however it doesn't seem
> like ya'll will actually be using any generated bindings which doesn't seem
> like a good dependency.
We want to make the modified version of swagger-codegen available to all
Pulp users. Our users will then have an option of using an unpatched
version of swagger-codegen to generate bindings that require the user to
remember both UUIDs and HREFs or using our version of swagger-codegen that
will produce bindings that use URI as the identifier in all cases.

> It might make sense for the Katello team but I would say we're only using
> hrefs because they seem to be what was recommended.  In both cases its a
> different technology (java) with a different built system.  I could see 3
> years from now some issue coming up that requires us to update to a newer
> swagger version and untangle this when no one that was involved in this
> decision was around.  I don't really see either of our teams maintaining a
> fork of the swagger code generation as a good long term solution.

The Pulp team will have ownership of the modified swagger-codegen as it
will be providing value to all users that want to integrate with Pulp 3.

We do not intend to publish bindings for any particular language because
each Pulp installation will require a different set of bindings based on
which plugins are installed.

> May I suggest that we either:
>  a) Work harder to get the code change upstream
> or if that does not seem to be going well
>  b) commit to continue responding with ids (in addition to hrefs) in all
> object responses
> (1)  https://pulp.plan.io/issues/3843
I don't think we plan to stop responding with IDs. However, our REST API
will continue to expect HREFs in body parameters that reference resources.
This leaves you with 3 options:

 - Write your own client for interacting with Pulp 3

 - use the upstream version of swagger-codegen to generate bindings and
keep track of IDs and HREFs

 - use the version of swagger-codegen shipped by the Pulp team and only
keep track of HREFs

> On 05/17/2018 06:26 PM, Dennis Kliban wrote:
> I have been able modify the default behavior of swagger-codegen to produce
> bindings that use relative URLs to identify resources. The required changes
> can be seen here[0]. The compiled jar is available for download here[1].
> The following command can be used to generate python bindings. Replace
> 'python' with 'ruby' and you'll get ruby bindings. The json file is the API
> schema returned by the Pulp server at http://localhost:8000/api/v3/d
> ocs/api.json.
> java -jar ./swagger-codegen-cli.jar generate -i ~/Downloads/pulp3-api.json
> -l python -o some_directory_name
> [0] https://github.com/dkliban/swagger-codegen/commit/e31e967698
> 64b73b06bd99f5e20e4720562539b9
> [1] https://repos.fedorapeople.org/pulp/pulp/swagger/swagger-cod
> egen-cli.jar
> On Tue, May 1, 2018 at 11:18 AM, David Davis <daviddavis at redhat.com>
> wrote:
>> We’ve exposed IDs and I think the next goal will be to somehow let
>> Katello use these IDs. I’ve opened a new story to solicit feedback on how
>> this’ll work:
>> https://pulp.plan.io/issues/3636
>> David
>> On Tue, May 1, 2018 at 9:14 AM, Brian Bouterse <bbouters at redhat.com>
>> wrote:
>>> On Tue, May 1, 2018 at 7:59 AM, Bryan Kearney <bkearney at redhat.com>
>>> wrote:
>>>> On 04/30/2018 04:08 PM, Brian Bouterse wrote:
>>>> > @asmacdo, checking in on the why is great. I want to try to articulate
>>>> > the benefits as I see them. Other perspectives and discussion are
>>>> welcome.
>>>> >
>>>> > The design of using URLs for referring things is a design whose goal
>>>> is
>>>> > to minimize complexity as the # of resources grows. The Internet is a
>>>> > useful analogy here. When someone wants to tell me how to find
>>>> something
>>>> > on Instagram, if the article's name is 'cat_pic432642' and that's all
>>>> I
>>>> > know, I'm going to have a hard time figuring out the actual URL to ask
>>>> > Instagram's servers for, e.g. /thepostsarehere/cat_pic432642. Even
>>>> if I
>>>> > did know how Instram's url space was laid out, I have to think about
>>>> it
>>>> > different from all other web services I interact with; now they're all
>>>> > different. This is the power of the URL itself. All clients and all
>>>> > servers can use the uniform resource locator to refer to things. I
>>>> think
>>>> > this was the main contribution from Tim Berners-Lee that allowed him
>>>> to
>>>> > implement HTTP which has URIs at its heart.
>>>> But, to Austins Point, if folks are going to interact with Pulp either
>>>> from the cli or from Go|rust|python libraries, they will not see the
>>>> urls. In your analogy, you would most likely google cat_pic432642  and
>>>> never know the url you are going to.
>>> I agree there will be much binding-based usage, and CLI usage, but there
>>> will always be users who will use httpie with some bash scripting calls
>>> instead. So from that perspective, the goals remain the same they have for
>>> years; we need to put forth the best, lowest-complexity REST API.
>>>> -- bk
>>> _______________________________________________
>>> Pulp-dev mailing list
>>> Pulp-dev at redhat.com
>>> https://www.redhat.com/mailman/listinfo/pulp-dev
>> _______________________________________________
>> Pulp-dev mailing list
>> Pulp-dev at redhat.com
>> https://www.redhat.com/mailman/listinfo/pulp-dev
> _______________________________________________
> Pulp-dev mailing listPulp-dev at redhat.comhttps://www.redhat.com/mailman/listinfo/pulp-dev
> _______________________________________________
> Pulp-dev mailing list
> Pulp-dev at redhat.com
> https://www.redhat.com/mailman/listinfo/pulp-dev
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/pulp-dev/attachments/20180711/2ac468e6/attachment.htm>

More information about the Pulp-dev mailing list