[Pulp-dev] Changes in the Pulp 3 Upload story

Fri Feb 22 16:55:47 UTC 2019

On Fri, Feb 22, 2019 at 7:56 AM Matthias Dellweg <dellweg at atix.de> wrote:

> If you parse all the metadata from the file / existing artifact, i see
> no real difference (from the users perspective). One call would accept
> a file upload, the other a reference to an existing artifact. The
> latter might impose a problem internally if the artifacts are stored
> *somewhere*, as you say. In the case of the file plugin, you need to
> specify a file / artifact_ref and a relative_path. So no big difference
> again.
>
There is one difference which highlights the main value of the one-shot
uploader. With the one-shot uploader, the client/user never needs to know
the data needed by the content's attributes. For example in RPM there are a
lot of attributes (
https://github.com/pulp/pulp_rpm/blob/master/pulp_rpm/app/models.py#L113 )
for the client to keep track of and then submit. The current uploader also
presupposes that the client wants to "deeply" know about the content. Some
would say that is what they want Pulp for in the first place. The one-shot
uploader lets the user hand Pulp an Artifact and not worry about the
details.

What i do not see is, how it is supposed to work for
> multiple-artifact content (does it really exist? Docker?). Can one
> upload multiple files in one api-call?
>
> Does the internal copying around hurt so much as to not wanting to
> implement the create endpoint?
>
> I know, the orphaning problem is really only circumvented by an atomic
> one shot into the repo version endpoint. But i thought the problem is
> related to this story.
>
> On Tue, 19 Feb 2019 11:33:18 -0500
> Daniel Alley <dalley at redhat.com> wrote:
>
> > @Matthias why would you prefer to keep the normal create?  As you
> > mention, the "orphan cleanup" issue applies to both, so there's no
> > advantage given to the former.
> >
> > The normal "create" ( POST .../content/plugin/type/ ...) is
> > unidiomatic and inconsistent, because the fields needed to upload a
> > content are different from the fields on the actual serializer.  Most
> > content types keep metadata inside the packages themselves, so you
> > can't let the user specify the content field values, you have to
> > contort everything so that instead of hydrating the serializer from
> > user input, it does so by parsing the content.
> >
> > There's also the issue that the libraries we're using to parse the
> > (Python and RPM) packages do some validation on the filename to see
> > that it has the right extension and so forth, and since artifacts are
> > content-addressed and don't store that information, with normal
> > create you have to pass the filename of the original artifact *back
> > in* at the time you create it, and then copy the file from Pulp
> > content storage into a temp directory under a filename which will
> > validate properly, which is highly unfortunate.
> >
> > With one-shot upload, you avoid both of those problems, because
> > there's no broken expectations as to what fields should be accepted,
> > and because it should be possible (though I haven't tried it) to
> > parse the original file *before* saving it as an artifact, thus
> > avoiding a lot of mess. And you also get the option to add it
> > straight into a repository. In my opinion, it's a straight upgrade.
> >
> > On Tue, Feb 19, 2019 at 10:57 AM Matthias Dellweg <dellweg at atix.de>
> > wrote:
> >
> > > I have no objections to having the "one shot upload" or even "one
> > > shot upload into repositoryversion". I think, i would like to keep
> > > the 'traditional' create anyway.
> > > The problem i see with create and one shot upload is, that another
> > > thing could have triggered 'delete orphans' at the wrong time, and
> > > you shiny new content unit disappears, before you can add it to a
> > > repository version.
> > >
> > > On Mon, 18 Feb 2019 14:41:54 -0500
> > > Austin Macdonald <austin at redhat.com> wrote:
> > >
> > > > Originally, our upload story was as follows:
> > > > The user will upload a new file to Pulp via POST to /artifacts/
> > > > (provided by core)
> > > > The user will create a new plugin specific Content via POST to
> > > > /path/to/plugin/content/, referencing whatever artifacts that are
> > > > contained, and whatever fields are expected for the new content.
> > > > The user will add the new content to a repository via POST to
> > > > /repositories/1/versions/
> > > >
> > > > However, this is somewhat cumbersome to the user with 3 API calls
> > > > to accomplish something that only took one call in Pulp 2.
> > > >
> > > > There are a couple of different paths plugins have taken to
> > > > improve the user experience:
> > > > The Python plugin follows the above workflow, but reads the
> > > > Artifact file to determine the values for the fields. The RPM
> > > > plugin has gone even farther and created a new endpoint for "one
> > > > shot" upload that perform all of this in a single call. I think
> > > > it is likely that the Python plugin will move more in the "one
> > > > shot" direction, and other plugins will probably follow.
> > > >
> > > > That said, I think we should discuss this as a community to
> > > > encourage plugins to behave similarly, and because there may also
> > > > be a possibility for sharing some of code. It is my hope that a
> > > > "one shot upload" could do 2 things: 1) Upload and create
> > > > Content. 2) Optionally add that content to repositories.
> > > _______________________________________________
> > > Pulp-dev mailing list
> > > Pulp-dev at redhat.com
> > > https://www.redhat.com/mailman/listinfo/pulp-dev
> > >
>
>
>
>
> Herzliche Grüße aus München
>
> Matthias Dellweg
> ______________________________________________________
> Dr. Matthias M. Dellweg
>
> (Open Source Software Engineer)
>
> Tel: +49 (0)89 452 35 38-12
> Fax: +49 (0)89 452 35 38-290
> E-Mail: dellweg at atix.de
>
> ATIX - The Linux & Open Source Company
>
> ATIX Informationstechnologie und Consulting AG
> Parkring 15
> 85748 Garching bei München
> www.atix.de
>
>
> Registergericht: Amtsgericht München, Registernummer: HRB 168930
> USt.-Id.: DE209485962
> Vorstand: Thomas Merz (Vors.), Mark Hlawatschek
> Vorsitzender des Aufsichtsrats: Dr. Martin Buss
> _______________________________________________
> Pulp-dev mailing list
> Pulp-dev at redhat.com
> https://www.redhat.com/mailman/listinfo/pulp-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/pulp-dev/attachments/20190222/8908b42b/attachment.htm>