<div dir="ltr"><div dir="ltr"><div dir="ltr"><div>I'm so glad we're talking it through on the list. Thank you for your writing!</div><div><br></div><div>I wrote some inline, but overall I'm wondering:</div><div><br></div><div>how much do users value reducing the 3 calls into 1?</div><div>how much do users value the server inspecting the asset to figure out the content's metadata?</div><div>how much do users want to upload many artifacts at once? (Can we do this?)</div><div>how much do users value association at the same time they are uploading an artifact?</div><div>how much do users value association at the same time they are uploading many artifacts?</div><div><br></div></div>Thank you,</div><div>Brian</div><div><br></div><div dir="ltr"><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Fri, Feb 22, 2019 at 1:05 PM Justin Sherrill <<a href="mailto:jsherril@redhat.com">jsherril@redhat.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div bgcolor="#FFFFFF">
<p><br>
</p>
<div class="gmail-m_-2123531035532370686moz-cite-prefix">On 2/22/19 12:07 PM, Brian Bouterse
wrote:<br>
</div>
<blockquote type="cite">
<div dir="ltr">
<div dir="ltr"><br>
</div>
<br>
<div class="gmail_quote">
<div dir="ltr" class="gmail_attr">On Fri, Feb 22, 2019 at 9:36
AM Justin Sherrill <<a href="mailto:jsherril@redhat.com" target="_blank">jsherril@redhat.com</a>> wrote:<br>
</div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div bgcolor="#FFFFFF">
<p><br>
</p>
<div class="gmail-m_-2123531035532370686gmail-m_1484403417921773583moz-cite-prefix">On
2/18/19 2:41 PM, Austin Macdonald wrote:<br>
</div>
<blockquote type="cite">
<div dir="ltr">
<div>Originally, our upload story was as follows:</div>
<div>The user will upload a new file to Pulp via POST
to /artifacts/ (provided by core)<br>
</div>
<div>The user will create a new plugin specific
Content via POST to /path/to/plugin/content/,
referencing whatever artifacts that are contained,
and whatever fields are expected for the new
content.<br>
</div>
<div>The user will add the new content to a repository
via POST to /repositories/1/versions/<br>
</div>
<div><br>
</div>
<div>However, this is somewhat cumbersome to the user
with 3 API calls to accomplish something that only
took one call in Pulp 2. <br>
</div>
</div>
</blockquote>
<p>How would you do this with one call in pulp2?
<a class="gmail-m_-2123531035532370686gmail-m_1484403417921773583moz-txt-link-freetext" href="https://docs.pulpproject.org/dev-guide/integration/rest-api/content/upload.html" target="_blank">https://docs.pulpproject.org/dev-guide/integration/rest-api/content/upload.html</a>
seems to suggest 3-4 calls.<br>
</p>
</div>
</blockquote>
<div>Some plugins implemented the pulp2 equivalent of a
one-shot uploader. Those docs are for pulp2's core which
don't include the plugin's docs. <br>
</div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div bgcolor="#FFFFFF">
<p> </p>
<blockquote type="cite">
<div dir="ltr">
<div><br>
</div>
<div>There are a couple of different paths plugins
have taken to improve the user experience:</div>
<div>The Python plugin follows the above workflow, but
reads the Artifact file to determine the values for
the fields. The RPM plugin has gone even farther and
created a new endpoint for "one shot" upload that
perform all of this in a single call. I think it is
likely that the Python plugin will move more in the
"one shot" direction, and other plugins will
probably follow.<br>
</div>
</div>
</blockquote>
<p>How does the RPM one shot api work? Will it be
compatible with whatever solution <a class="gmail-m_-2123531035532370686gmail-m_1484403417921773583moz-txt-link-freetext" href="https://pulp.plan.io/issues/4196" target="_blank">https://pulp.plan.io/issues/4196</a>
arrives at? </p>
</div>
</blockquote>
<div>You would upload the Artifact as binary data along with
what content type it is and what relative path it uses and
Pulp creates the Artifact, Content unit, ContentArtifact. It
should be compatible with issue 4196 because django's binary
form data should allow for parallel uploading before calling
the view handler. It may take 2 calls though. The issue to
me isn't about the number of calls as it is the client data
payload complexity. <br>
</div>
</div>
</div>
</blockquote>
If i'm having to chunk up data, i already have quite a bit of client
data payload complexity. In pulp 2 this was most of the complexity!
<br>
<blockquote type="cite">
<div dir="ltr">
<div class="gmail_quote">
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div bgcolor="#FFFFFF">
<p>I would hate for all our plugins to move to One shot
methods which users can't even rely on.<br>
</p>
</div>
</blockquote>
<div>I don't think we're taking the "generic" uploading away.
You can always rely on that. The issue w/ one-shot is that
it's not possible (literally) for many content types, e.g.
Artifact-less content. It's also hard for multi-artifact
Content so that would probably still be something plugin
writers would provide as a custom thing for their content
type. Regardless it's just not possible to have consistency
in this area. <br>
</div>
</div>
</div>
</blockquote>
<p>Why is it not possible to create a one-shot upload for
artifact-less content? (maybe we're defining what a one-shot
upload actually is differently, i'm reading it as something that
combines multiple steps into one)<br></p></div></blockquote><div>I'm expecting the default viewset for Content to accept kwargs and try to save them onto the Content unit it's creating as attributes. Plugin writers can always override that though so it could be inconsistent. <br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div bgcolor="#FFFFFF"><p>
</p>
<p>Why is consistency not possible? I guess i don't see a huge
variation of upload scenarios beyond:</p></div></blockquote><div>The main reason is because the plugin writer is providing the viewset for the creation of the Content unit. Even if we make something in core that provides a "generic" functionality it will need plugin code provided somehow to parse the Artifact. For example the artifact specific parsing code is here: <a href="https://github.com/pulp/pulp_rpm/blob/master/pulp_rpm/app/viewsets.py#L92">https://github.com/pulp/pulp_rpm/blob/master/pulp_rpm/app/viewsets.py#L92</a><br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div bgcolor="#FFFFFF">
<p>1. upload Zero to many files as artifacts<br></p></div></blockquote><div>You get this generically. Post to the core-provided Artifact URL. <br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div bgcolor="#FFFFFF"><p>
</p>
<p>2. Provide some metadata about the zero or more artifacts or let
the plugin parse it out themselves (or maybe even a combination of
the two)<br></p></div></blockquote><div>Either the client provides the metadata 100% or the plugin has to be involved somehow. Not all plugins will do this, and we can't require them to. This is the inconsistent part even if we introduce a core-based one-shot uploader. <br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div bgcolor="#FFFFFF"><p>
</p>
<p>3. Import that unit into a repository. <br></p></div></blockquote><div>The issue is that creating a new repo-version with each version uploaded is problematic.I think users want to associate several units, so association is probably useful if we could have it accept many Artifacts at once. Is that what people imagined out of this feature?<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div bgcolor="#FFFFFF"><p>
</p>
<p>I can see it being difficult as a user to go through all of those
steps (even if 2 & 3 were combined into one), and the desire
is to simplify the process, but uploading arbitrary files is not
simple. Why do i need to give up the plugin's ability to parse
the unit's details because i'm using the consistent api?</p>
<p>Keep in mind all my questions are coming from a very ignorant
perspective with respect of pulp3 internals, and more from a user
perspective. <br>
</p>
<blockquote type="cite">
<div dir="ltr">
<div class="gmail_quote">
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div bgcolor="#FFFFFF">
<p> </p>
<p>My problem with single api calls to upload files is
that we cannot reliably use them due to limitation in
request sizes. We have to be prepared to use multiple
calls to upload files regardless. Maybe if a user is
using some plugin that never has super large files
(ansible?) you could be confident you would never hit a
request size limitation. But file, docker, and yum all
would require multiple calls to get the physical data to
the server.<br>
</p>
</div>
</blockquote>
<div>I believe arbitrarily large files can be uploaded either
through multi-part form data or through the django-chunked
interface. We'll see what happens with 4196, but I expect
arbitrary payload size to be a requirement for Pulp users.<br>
</div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div bgcolor="#FFFFFF">
<p> </p>
<p>I care more about having a consistent method for
uploading files than having fewer api calls. If we
need a some content specific api, that's fine, but
please make it a consistent part of the process. <br>
</p>
</div>
</blockquote>
<div>It sounds like the 4-call interface is the only choice
then if consistency is a must. There isn't a way to offer
consistency for one-shot uploaders. Is it ok that Katello
will have to fill out all of the field data when you post
the content type? What could be better?<br>
</div>
</div>
</div>
</blockquote>
<p>I'll reserve my comments here based on the discussion above.</p>
<p>Thanks!<br>
</p>
<p>Justin<br>
</p>
<p><br>
</p>
<blockquote type="cite">
<div dir="ltr">
<div class="gmail_quote">
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div bgcolor="#FFFFFF">
<p> </p>
<p>I feel like we may be chasing the wrong goal here
(fewer calls vs a more consistent experience). <br>
</p>
<blockquote type="cite">
<div dir="ltr">
<div><br>
</div>
<div>That said, I think we should discuss this as a
community to encourage plugins to behave similarly,
and because there may also be a possibility for
sharing some of code. It is my hope that a "one shot
upload" could do 2 things: 1) Upload and create
Content. 2) Optionally add that content to
repositories.</div>
</div>
<br>
<fieldset class="gmail-m_-2123531035532370686gmail-m_1484403417921773583mimeAttachmentHeader"></fieldset>
<pre class="gmail-m_-2123531035532370686gmail-m_1484403417921773583moz-quote-pre">_______________________________________________
Pulp-dev mailing list
<a class="gmail-m_-2123531035532370686gmail-m_1484403417921773583moz-txt-link-abbreviated" href="mailto:Pulp-dev@redhat.com" target="_blank">Pulp-dev@redhat.com</a>
<a class="gmail-m_-2123531035532370686gmail-m_1484403417921773583moz-txt-link-freetext" href="https://www.redhat.com/mailman/listinfo/pulp-dev" target="_blank">https://www.redhat.com/mailman/listinfo/pulp-dev</a>
</pre>
</blockquote>
</div>
_______________________________________________<br>
Pulp-dev mailing list<br>
<a href="mailto:Pulp-dev@redhat.com" target="_blank">Pulp-dev@redhat.com</a><br>
<a href="https://www.redhat.com/mailman/listinfo/pulp-dev" rel="noreferrer" target="_blank">https://www.redhat.com/mailman/listinfo/pulp-dev</a><br>
</blockquote>
</div>
</div>
</blockquote>
</div>
</blockquote></div></div></div>