<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
</head>
<body text="#000000" bgcolor="#FFFFFF">
<p><br>
</p>
<div class="moz-cite-prefix">On 2/22/19 12:07 PM, Brian Bouterse
wrote:<br>
</div>
<blockquote type="cite"
cite="mid:CAAcvrTEnS8qYJwfQFxw9cFnh5ksFqNhEuCnRN-Y=ZUUz9xzYrA@mail.gmail.com">
<meta http-equiv="content-type" content="text/html; charset=UTF-8">
<div dir="ltr">
<div dir="ltr"><br>
</div>
<br>
<div class="gmail_quote">
<div dir="ltr" class="gmail_attr">On Fri, Feb 22, 2019 at 9:36
AM Justin Sherrill <<a href="mailto:jsherril@redhat.com"
moz-do-not-send="true">jsherril@redhat.com</a>> wrote:<br>
</div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px
0.8ex;border-left:1px solid
rgb(204,204,204);padding-left:1ex">
<div bgcolor="#FFFFFF">
<p><br>
</p>
<div class="gmail-m_1484403417921773583moz-cite-prefix">On
2/18/19 2:41 PM, Austin Macdonald wrote:<br>
</div>
<blockquote type="cite">
<div dir="ltr">
<div>Originally, our upload story was as follows:</div>
<div>The user will upload a new file to Pulp via POST
to /artifacts/ (provided by core)<br>
</div>
<div>The user will create a new plugin specific
Content via POST to /path/to/plugin/content/,
referencing whatever artifacts that are contained,
and whatever fields are expected for the new
content.<br>
</div>
<div>The user will add the new content to a repository
via POST to /repositories/1/versions/<br>
</div>
<div><br>
</div>
<div>However, this is somewhat cumbersome to the user
with 3 API calls to accomplish something that only
took one call in Pulp 2. <br>
</div>
</div>
</blockquote>
<p>How would you do this with one call in pulp2?
<a
class="gmail-m_1484403417921773583moz-txt-link-freetext"
href="https://docs.pulpproject.org/dev-guide/integration/rest-api/content/upload.html"
target="_blank" moz-do-not-send="true">https://docs.pulpproject.org/dev-guide/integration/rest-api/content/upload.html</a>
seems to suggest 3-4 calls.<br>
</p>
</div>
</blockquote>
<div>Some plugins implemented the pulp2 equivalent of a
one-shot uploader. Those docs are for pulp2's core which
don't include the plugin's docs. <br>
</div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px
0.8ex;border-left:1px solid
rgb(204,204,204);padding-left:1ex">
<div bgcolor="#FFFFFF">
<p> </p>
<blockquote type="cite">
<div dir="ltr">
<div><br>
</div>
<div>There are a couple of different paths plugins
have taken to improve the user experience:</div>
<div>The Python plugin follows the above workflow, but
reads the Artifact file to determine the values for
the fields. The RPM plugin has gone even farther and
created a new endpoint for "one shot" upload that
perform all of this in a single call. I think it is
likely that the Python plugin will move more in the
"one shot" direction, and other plugins will
probably follow.<br>
</div>
</div>
</blockquote>
<p>How does the RPM one shot api work? Will it be
compatible with whatever solution <a
class="gmail-m_1484403417921773583moz-txt-link-freetext"
href="https://pulp.plan.io/issues/4196"
target="_blank" moz-do-not-send="true">https://pulp.plan.io/issues/4196</a>
arrives at? </p>
</div>
</blockquote>
<div>You would upload the Artifact as binary data along with
what content type it is and what relative path it uses and
Pulp creates the Artifact, Content unit, ContentArtifact. It
should be compatible with issue 4196 because django's binary
form data should allow for parallel uploading before calling
the view handler. It may take 2 calls though. The issue to
me isn't about the number of calls as it is the client data
payload complexity. <br>
</div>
</div>
</div>
</blockquote>
If i'm having to chunk up data, i already have quite a bit of client
data payload complexity. In pulp 2 this was most of the complexity!
<br>
<blockquote type="cite"
cite="mid:CAAcvrTEnS8qYJwfQFxw9cFnh5ksFqNhEuCnRN-Y=ZUUz9xzYrA@mail.gmail.com">
<div dir="ltr">
<div class="gmail_quote">
<blockquote class="gmail_quote" style="margin:0px 0px 0px
0.8ex;border-left:1px solid
rgb(204,204,204);padding-left:1ex">
<div bgcolor="#FFFFFF">
<p>I would hate for all our plugins to move to One shot
methods which users can't even rely on.<br>
</p>
</div>
</blockquote>
<div>I don't think we're taking the "generic" uploading away.
You can always rely on that. The issue w/ one-shot is that
it's not possible (literally) for many content types, e.g.
Artifact-less content. It's also hard for multi-artifact
Content so that would probably still be something plugin
writers would provide as a custom thing for their content
type. Regardless it's just not possible to have consistency
in this area. <br>
</div>
</div>
</div>
</blockquote>
<p>Why is it not possible to create a one-shot upload for
artifact-less content? (maybe we're defining what a one-shot
upload actually is differently, i'm reading it as something that
combines multiple steps into one)<br>
</p>
<p>Why is consistency not possible? I guess i don't see a huge
variation of upload scenarios beyond:</p>
<p>1. upload Zero to many files as artifacts<br>
</p>
<p>2. Provide some metadata about the zero or more artifacts or let
the plugin parse it out themselves (or maybe even a combination of
the two)<br>
</p>
<p>3. Import that unit into a repository. <br>
</p>
<p>I can see it being difficult as a user to go through all of those
steps (even if 2 & 3 were combined into one), and the desire
is to simplify the process, but uploading arbitrary files is not
simple. Why do i need to give up the plugin's ability to parse
the unit's details because i'm using the consistent api?</p>
<p>Keep in mind all my questions are coming from a very ignorant
perspective with respect of pulp3 internals, and more from a user
perspective. <br>
</p>
<blockquote type="cite"
cite="mid:CAAcvrTEnS8qYJwfQFxw9cFnh5ksFqNhEuCnRN-Y=ZUUz9xzYrA@mail.gmail.com">
<div dir="ltr">
<div class="gmail_quote">
<blockquote class="gmail_quote" style="margin:0px 0px 0px
0.8ex;border-left:1px solid
rgb(204,204,204);padding-left:1ex">
<div bgcolor="#FFFFFF">
<p> </p>
<p>My problem with single api calls to upload files is
that we cannot reliably use them due to limitation in
request sizes. We have to be prepared to use multiple
calls to upload files regardless. Maybe if a user is
using some plugin that never has super large files
(ansible?) you could be confident you would never hit a
request size limitation. But file, docker, and yum all
would require multiple calls to get the physical data to
the server.<br>
</p>
</div>
</blockquote>
<div>I believe arbitrarily large files can be uploaded either
through multi-part form data or through the django-chunked
interface. We'll see what happens with 4196, but I expect
arbitrary payload size to be a requirement for Pulp users.<br>
</div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px
0.8ex;border-left:1px solid
rgb(204,204,204);padding-left:1ex">
<div bgcolor="#FFFFFF">
<p> </p>
<p>I care more about having a consistent method for
uploading files than having fewer api calls. If we
need a some content specific api, that's fine, but
please make it a consistent part of the process. <br>
</p>
</div>
</blockquote>
<div>It sounds like the 4-call interface is the only choice
then if consistency is a must. There isn't a way to offer
consistency for one-shot uploaders. Is it ok that Katello
will have to fill out all of the field data when you post
the content type? What could be better?<br>
</div>
</div>
</div>
</blockquote>
<p>I'll reserve my comments here based on the discussion above.</p>
<p>Thanks!<br>
</p>
<p>Justin<br>
</p>
<p><br>
</p>
<blockquote type="cite"
cite="mid:CAAcvrTEnS8qYJwfQFxw9cFnh5ksFqNhEuCnRN-Y=ZUUz9xzYrA@mail.gmail.com">
<div dir="ltr">
<div class="gmail_quote">
<blockquote class="gmail_quote" style="margin:0px 0px 0px
0.8ex;border-left:1px solid
rgb(204,204,204);padding-left:1ex">
<div bgcolor="#FFFFFF">
<p> </p>
<p>I feel like we may be chasing the wrong goal here
(fewer calls vs a more consistent experience). <br>
</p>
<blockquote type="cite">
<div dir="ltr">
<div><br>
</div>
<div>That said, I think we should discuss this as a
community to encourage plugins to behave similarly,
and because there may also be a possibility for
sharing some of code. It is my hope that a "one shot
upload" could do 2 things: 1) Upload and create
Content. 2) Optionally add that content to
repositories.</div>
</div>
<br>
<fieldset
class="gmail-m_1484403417921773583mimeAttachmentHeader"></fieldset>
<pre class="gmail-m_1484403417921773583moz-quote-pre">_______________________________________________
Pulp-dev mailing list
<a class="gmail-m_1484403417921773583moz-txt-link-abbreviated" href="mailto:Pulp-dev@redhat.com" target="_blank" moz-do-not-send="true">Pulp-dev@redhat.com</a>
<a class="gmail-m_1484403417921773583moz-txt-link-freetext" href="https://www.redhat.com/mailman/listinfo/pulp-dev" target="_blank" moz-do-not-send="true">https://www.redhat.com/mailman/listinfo/pulp-dev</a>
</pre>
</blockquote>
</div>
_______________________________________________<br>
Pulp-dev mailing list<br>
<a href="mailto:Pulp-dev@redhat.com" target="_blank"
moz-do-not-send="true">Pulp-dev@redhat.com</a><br>
<a href="https://www.redhat.com/mailman/listinfo/pulp-dev"
rel="noreferrer" target="_blank" moz-do-not-send="true">https://www.redhat.com/mailman/listinfo/pulp-dev</a><br>
</blockquote>
</div>
</div>
</blockquote>
</body>
</html>