[Pulp-dev] Changes in the Pulp 3 Upload story

[Brian Bouterse]

On Fri, Feb 22, 2019 at 9:36 AM Justin Sherrill <jsherril at redhat.com> wrote:

>
> On 2/18/19 2:41 PM, Austin Macdonald wrote:
>
> Originally, our upload story was as follows:
> The user will upload a new file to Pulp via POST to /artifacts/ (provided
> by core)
> The user will create a new plugin specific Content via POST to
> /path/to/plugin/content/, referencing whatever artifacts that are
> contained, and whatever fields are expected for the new content.
> The user will add the new content to a repository via POST to
> /repositories/1/versions/
>
> However, this is somewhat cumbersome to the user with 3 API calls to
> accomplish something that only took one call in Pulp 2.
>
> How would you do this with one call in pulp2?
> https://docs.pulpproject.org/dev-guide/integration/rest-api/content/upload.html
> seems to suggest 3-4 calls.
>
Some plugins implemented the pulp2 equivalent of a one-shot uploader. Those
docs are for pulp2's core which don't include the plugin's docs.

>
> There are a couple of different paths plugins have taken to improve the
> user experience:
> The Python plugin follows the above workflow, but reads the Artifact file
> to determine the values for the fields. The RPM plugin has gone even
> farther and created a new endpoint for "one shot" upload that perform all
> of this in a single call. I think it is likely that the Python plugin will
> move more in the "one shot" direction, and other plugins will probably
> follow.
>
> How does the RPM one shot api work?  Will it be compatible with whatever
> solution https://pulp.plan.io/issues/4196 arrives at?
>
You would upload the Artifact as binary data along with what content type
it is and what relative path it uses and Pulp creates the Artifact, Content
unit, ContentArtifact. It should be compatible with issue 4196 because
django's binary form data should allow for parallel uploading before
calling the view handler. It may take 2 calls though. The issue to me isn't
about the number of calls as it is the client data payload complexity.

> I would hate for all our plugins to move to One shot methods which users
> can't even rely on.
>
I don't think we're taking the "generic" uploading away. You can always
rely on that. The issue w/ one-shot is that it's not possible (literally)
for many content types, e.g. Artifact-less content. It's also hard for
multi-artifact Content so that would probably still be something plugin
writers would provide as a custom thing for their content type. Regardless
it's just not possible to have consistency in this area.

> My problem with single api calls to upload files is that we cannot
> reliably use them due to limitation in request sizes.  We have to be
> prepared to use multiple calls to upload files regardless.  Maybe if a user
> is using some plugin that never has super large files (ansible?) you could
> be confident you would never hit a request size limitation.   But file,
> docker, and yum all would require multiple calls to get the physical data
> to the server.
>
I believe arbitrarily large files can be uploaded either through multi-part
form data or through the django-chunked interface. We'll see what happens
with 4196, but I expect arbitrary payload size to be a requirement for Pulp
users.

> I care more about having a consistent method for uploading files than
> having fewer api calls.   If we need a some content specific api, that's
> fine, but please make it a consistent part of the process.
>
It sounds like the 4-call interface is the only choice then if consistency
is a must. There isn't a way to offer consistency for one-shot uploaders.
Is it ok that Katello will have to fill out all of the field data when you
post the content type? What could be better?

> I feel like we may be chasing the wrong goal here (fewer calls vs a more
> consistent experience).
>
>
> That said, I think we should discuss this as a community to encourage
> plugins to behave similarly, and because there may also be a possibility
> for sharing some of code. It is my hope that a "one shot upload" could do 2
> things: 1) Upload and create Content. 2) Optionally add that content to
> repositories.
>
> _______________________________________________
> Pulp-dev mailing listPulp-dev at redhat.comhttps://www.redhat.com/mailman/listinfo/pulp-dev
>
> _______________________________________________
> Pulp-dev mailing list
> Pulp-dev at redhat.com
> https://www.redhat.com/mailman/listinfo/pulp-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/pulp-dev/attachments/20190222/c60320cb/attachment.htm>