[Pulp-dev] Changes in the Pulp 3 Upload story

Tue Feb 19 18:07:55 UTC 2019

> What i do not see is, how it is supposed to work for
> multiple-artifact content (does it really exist? Docker?). Can one
> upload multiple files in one api-call?

Well, as I mention, I don't think one-shot upload would be a good fit
there.  I'm *not* suggesting we try to cram everything through a one-shot
upload pattern that would completely replace what we do currently.  It's
the same with no-artifact content such as errata, I think sending a POST to
the content viewset is still the best solution there.

But for plugins that have a single artifact and metadata stored internally
(which will be most of them), normal "create" meets their needs rather
poorly, so the question is thus:

- Should we have a second semi-official pattern, one-shot upload, and
should Pulp do anything to facilitate this workflow  if possible
- For plugins that implement one-shot upload, should they *also* try to
implement the normal "create" endpoints, even though it's un-idiomatic and
the implementation is by necessity much more complicated than one-shot
while also being less flexible.

But doing these things wouldn't prevent any plugin from doing things the
way they're done currently.  The question is, should *plugins* support both
ways of uploading for "consistency" [2], not whether Pulp should do so. [3]

1. unless you count the digest *of* the file, but we don't expose that via
the serializer, so still doesn't expose that particular ergonomic issue
2. *endpoint* consistency, perhaps, but because of the issues I mentioned,
you may or may not need to pass in the original filename of the artifact,
which isn't a field on the serializer at all.  This breaks with all the
expectations of what a viewset is supposed to do
3. Pulp doesn't actually "support" this to begin with, it's 100% provided
by Django REST Framework

On Tue, Feb 19, 2019 at 12:07 PM Matthias Dellweg <dellweg at atix.de> wrote:

> If you parse all the metadata from the file / existing artifact, i see
> no real difference (from the users perspective). One call would accept
> a file upload, the other a reference to an existing artifact. The
> latter might impose a problem internally if the artifacts are stored
> *somewhere*, as you say. In the case of the file plugin, you need to
> specify a file / artifact_ref and a relative_path. So no big difference
> again.
> What i do not see is, how it is supposed to work for
> multiple-artifact content (does it really exist? Docker?). Can one
> upload multiple files in one api-call?
>
> Does the internal copying around hurt so much as to not wanting to
> implement the create endpoint?
>
> I know, the orphaning problem is really only circumvented by an atomic
> one shot into the repo version endpoint. But i thought the problem is
> related to this story.
>
> On Tue, 19 Feb 2019 11:33:18 -0500
> Daniel Alley <dalley at redhat.com> wrote:
>
> > @Matthias why would you prefer to keep the normal create?  As you
> > mention, the "orphan cleanup" issue applies to both, so there's no
> > advantage given to the former.
> >
> > The normal "create" ( POST .../content/plugin/type/ ...) is
> > unidiomatic and inconsistent, because the fields needed to upload a
> > content are different from the fields on the actual serializer.  Most
> > content types keep metadata inside the packages themselves, so you
> > can't let the user specify the content field values, you have to
> > contort everything so that instead of hydrating the serializer from
> > user input, it does so by parsing the content.
> >
> > There's also the issue that the libraries we're using to parse the
> > (Python and RPM) packages do some validation on the filename to see
> > that it has the right extension and so forth, and since artifacts are
> > content-addressed and don't store that information, with normal
> > create you have to pass the filename of the original artifact *back
> > in* at the time you create it, and then copy the file from Pulp
> > content storage into a temp directory under a filename which will
> > validate properly, which is highly unfortunate.
> >
> > With one-shot upload, you avoid both of those problems, because
> > there's no broken expectations as to what fields should be accepted,
> > and because it should be possible (though I haven't tried it) to
> > parse the original file *before* saving it as an artifact, thus
> > avoiding a lot of mess. And you also get the option to add it
> > straight into a repository. In my opinion, it's a straight upgrade.
> >
> > On Tue, Feb 19, 2019 at 10:57 AM Matthias Dellweg <dellweg at atix.de>
> > wrote:
> >
> > > I have no objections to having the "one shot upload" or even "one
> > > shot upload into repositoryversion". I think, i would like to keep
> > > the 'traditional' create anyway.
> > > The problem i see with create and one shot upload is, that another
> > > thing could have triggered 'delete orphans' at the wrong time, and
> > > you shiny new content unit disappears, before you can add it to a
> > > repository version.
> > >
> > > On Mon, 18 Feb 2019 14:41:54 -0500
> > > Austin Macdonald <austin at redhat.com> wrote:
> > >
> > > > Originally, our upload story was as follows:
> > > > The user will upload a new file to Pulp via POST to /artifacts/
> > > > (provided by core)
> > > > The user will create a new plugin specific Content via POST to
> > > > /path/to/plugin/content/, referencing whatever artifacts that are
> > > > contained, and whatever fields are expected for the new content.
> > > > The user will add the new content to a repository via POST to
> > > > /repositories/1/versions/
> > > >
> > > > However, this is somewhat cumbersome to the user with 3 API calls
> > > > to accomplish something that only took one call in Pulp 2.
> > > >
> > > > There are a couple of different paths plugins have taken to
> > > > improve the user experience:
> > > > The Python plugin follows the above workflow, but reads the
> > > > Artifact file to determine the values for the fields. The RPM
> > > > plugin has gone even farther and created a new endpoint for "one
> > > > shot" upload that perform all of this in a single call. I think
> > > > it is likely that the Python plugin will move more in the "one
> > > > shot" direction, and other plugins will probably follow.
> > > >
> > > > That said, I think we should discuss this as a community to
> > > > encourage plugins to behave similarly, and because there may also
> > > > be a possibility for sharing some of code. It is my hope that a
> > > > "one shot upload" could do 2 things: 1) Upload and create
> > > > Content. 2) Optionally add that content to repositories.
> > > _______________________________________________
> > > Pulp-dev mailing list
> > > Pulp-dev at redhat.com
> > > https://www.redhat.com/mailman/listinfo/pulp-dev
> > >
>
>
>
>
> Herzliche Grüße aus München
>
> Matthias Dellweg
> ______________________________________________________
> Dr. Matthias M. Dellweg
>
> (Open Source Software Engineer)
>
> Tel: +49 (0)89 452 35 38-12
> Fax: +49 (0)89 452 35 38-290
> E-Mail: dellweg at atix.de
>
> ATIX - The Linux & Open Source Company
>
> ATIX Informationstechnologie und Consulting AG
> Parkring 15
> 85748 Garching bei München
> www.atix.de
>
>
> Registergericht: Amtsgericht München, Registernummer: HRB 168930
> USt.-Id.: DE209485962
> Vorstand: Thomas Merz (Vors.), Mark Hlawatschek
> Vorsitzender des Aufsichtsrats: Dr. Martin Buss
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/pulp-dev/attachments/20190219/d253bcc5/attachment.htm>