<div dir="ltr"><div></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">What i do not see is, how it is supposed to work for<br> multiple-artifact content (does it really exist? Docker?). Can one<br> upload multiple files in one api-call?</blockquote><div><br></div><div>Well, as I mention, I don't think one-shot upload would be a good fit there. I'm *not* suggesting we try to cram everything through a one-shot upload pattern that would completely replace what we do currently. It's the same with no-artifact content such as errata, I think sending a POST to the content viewset is still the best solution there.<br></div><div><br></div><div>But for plugins that have a single artifact and metadata stored internally (which will be most of them), normal "create" meets their needs rather poorly, so the question is thus:</div><div><br></div><div>- Should we have a second semi-official pattern, one-shot upload, and should Pulp do anything to facilitate this workflow if possible<br></div><div>- For plugins that implement one-shot upload, should they *also* try to implement the normal "create" endpoints, even though it's un-idiomatic and the implementation is by necessity much more complicated than one-shot while also being less flexible. <br></div><div><br></div><div>But doing these things wouldn't prevent any plugin from doing things the way they're done currently. The question is, should *plugins* support both ways of uploading for "consistency" [2], not whether Pulp should do so. [3]</div><div><br></div><div><br></div><div>1. unless you count the digest *of* the file, but we don't expose that via the serializer, so still doesn't expose that particular ergonomic issue</div><div>2. <i>endpoint</i> consistency, perhaps, but because of the issues I mentioned, you may or may not need to pass in the original filename of the artifact, which isn't a field on the serializer at all. This breaks with all the expectations of what a viewset is supposed to do<br></div><div>3. Pulp doesn't actually "support" this to begin with, it's 100% provided by Django REST Framework<br></div><div><br></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Tue, Feb 19, 2019 at 12:07 PM Matthias Dellweg <<a href="mailto:dellweg@atix.de">dellweg@atix.de</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">If you parse all the metadata from the file / existing artifact, i see<br> no real difference (from the users perspective). One call would accept<br> a file upload, the other a reference to an existing artifact. The<br> latter might impose a problem internally if the artifacts are stored<br> *somewhere*, as you say. In the case of the file plugin, you need to<br> specify a file / artifact_ref and a relative_path. So no big difference<br> again.<br> What i do not see is, how it is supposed to work for<br> multiple-artifact content (does it really exist? Docker?). Can one<br> upload multiple files in one api-call?<br> <br> Does the internal copying around hurt so much as to not wanting to<br> implement the create endpoint?<br> <br> I know, the orphaning problem is really only circumvented by an atomic<br> one shot into the repo version endpoint. But i thought the problem is<br> related to this story.<br> <br> On Tue, 19 Feb 2019 11:33:18 -0500<br> Daniel Alley <<a href="mailto:dalley@redhat.com" target="_blank">dalley@redhat.com</a>> wrote:<br> <br> > @Matthias why would you prefer to keep the normal create? As you<br> > mention, the "orphan cleanup" issue applies to both, so there's no<br> > advantage given to the former.<br> > <br> > The normal "create" ( POST .../content/plugin/type/ ...) is<br> > unidiomatic and inconsistent, because the fields needed to upload a<br> > content are different from the fields on the actual serializer. Most<br> > content types keep metadata inside the packages themselves, so you<br> > can't let the user specify the content field values, you have to<br> > contort everything so that instead of hydrating the serializer from<br> > user input, it does so by parsing the content.<br> > <br> > There's also the issue that the libraries we're using to parse the<br> > (Python and RPM) packages do some validation on the filename to see<br> > that it has the right extension and so forth, and since artifacts are<br> > content-addressed and don't store that information, with normal<br> > create you have to pass the filename of the original artifact *back<br> > in* at the time you create it, and then copy the file from Pulp<br> > content storage into a temp directory under a filename which will<br> > validate properly, which is highly unfortunate.<br> > <br> > With one-shot upload, you avoid both of those problems, because<br> > there's no broken expectations as to what fields should be accepted,<br> > and because it should be possible (though I haven't tried it) to<br> > parse the original file *before* saving it as an artifact, thus<br> > avoiding a lot of mess. And you also get the option to add it<br> > straight into a repository. In my opinion, it's a straight upgrade.<br> > <br> > On Tue, Feb 19, 2019 at 10:57 AM Matthias Dellweg <<a href="mailto:dellweg@atix.de" target="_blank">dellweg@atix.de</a>><br> > wrote:<br> > <br> > > I have no objections to having the "one shot upload" or even "one<br> > > shot upload into repositoryversion". I think, i would like to keep<br> > > the 'traditional' create anyway.<br> > > The problem i see with create and one shot upload is, that another<br> > > thing could have triggered 'delete orphans' at the wrong time, and<br> > > you shiny new content unit disappears, before you can add it to a<br> > > repository version.<br> > ><br> > > On Mon, 18 Feb 2019 14:41:54 -0500<br> > > Austin Macdonald <<a href="mailto:austin@redhat.com" target="_blank">austin@redhat.com</a>> wrote:<br> > > <br> > > > Originally, our upload story was as follows:<br> > > > The user will upload a new file to Pulp via POST to /artifacts/<br> > > > (provided by core)<br> > > > The user will create a new plugin specific Content via POST to<br> > > > /path/to/plugin/content/, referencing whatever artifacts that are<br> > > > contained, and whatever fields are expected for the new content.<br> > > > The user will add the new content to a repository via POST to<br> > > > /repositories/1/versions/<br> > > ><br> > > > However, this is somewhat cumbersome to the user with 3 API calls<br> > > > to accomplish something that only took one call in Pulp 2.<br> > > ><br> > > > There are a couple of different paths plugins have taken to<br> > > > improve the user experience:<br> > > > The Python plugin follows the above workflow, but reads the<br> > > > Artifact file to determine the values for the fields. The RPM<br> > > > plugin has gone even farther and created a new endpoint for "one<br> > > > shot" upload that perform all of this in a single call. I think<br> > > > it is likely that the Python plugin will move more in the "one<br> > > > shot" direction, and other plugins will probably follow.<br> > > ><br> > > > That said, I think we should discuss this as a community to<br> > > > encourage plugins to behave similarly, and because there may also<br> > > > be a possibility for sharing some of code. It is my hope that a<br> > > > "one shot upload" could do 2 things: 1) Upload and create<br> > > > Content. 2) Optionally add that content to repositories. <br> > > _______________________________________________<br> > > Pulp-dev mailing list<br> > > <a href="mailto:Pulp-dev@redhat.com" target="_blank">Pulp-dev@redhat.com</a><br> > > <a href="https://www.redhat.com/mailman/listinfo/pulp-dev" rel="noreferrer" target="_blank">https://www.redhat.com/mailman/listinfo/pulp-dev</a><br> > > <br> <br> <br> <br> <br> Herzliche Grüße aus München<br> <br> Matthias Dellweg<br> ______________________________________________________ <br> Dr. Matthias M. Dellweg<br> <br> (Open Source Software Engineer)<br> <br> Tel: +49 (0)89 452 35 38-12<br> Fax: +49 (0)89 452 35 38-290<br> E-Mail: <a href="mailto:dellweg@atix.de" target="_blank">dellweg@atix.de</a><br> <br> ATIX - The Linux & Open Source Company<br> <br> ATIX Informationstechnologie und Consulting AG<br> Parkring 15<br> 85748 Garching bei München<br> <a href="http://www.atix.de" rel="noreferrer" target="_blank">www.atix.de</a><br> <br> <br> Registergericht: Amtsgericht München, Registernummer: HRB 168930<br> USt.-Id.: DE209485962<br> Vorstand: Thomas Merz (Vors.), Mark Hlawatschek<br> Vorsitzender des Aufsichtsrats: Dr. Martin Buss<br> </blockquote></div>