[Pulp-dev] Changes in the Pulp 3 Upload story
Justin Sherrill
jsherril at redhat.com
Fri Feb 22 15:07:55 UTC 2019
On 2/19/19 12:30 PM, Brian Bouterse wrote:
> Overall +1, we should make a 1-shot uploader. It will need a bit of
> code from plugin writers to make it work though.
>
> I believe the "manual and generic" feature offerings we have now are
> pretty good. I don't think we should take them away.
>
> To describe yet another related use case; consider the uploading of a
> tarball with a lot of units in it (maybe a whole repo). For instance
> bunch of Ansible roles in a tarball is what a user would want to
> upload to Pulp, so you would want to upload the whole tarball at once.
> I see three cases:
>
> a) The generic method (what we already have). The user supplies all
> metadata because we can't know the right attributes without them
> telling it to us.
> b) The one-shot uploader to create a single content unit from a single
> Artifact. I think this is what is being proposed here. +1 to this.
> c) Uploading a bunch of content units in one binary, e.g. a tarball.
> We should leave this to plugin writers to implement for now.
Could b & c be the same api? The api could (depending on what the
plugin wants to implement) accept either one unit file or a tar file
with a bunch of different units.
>
> In terms of implementing (b) I believe it would be useful to have a
> Content unit provide an optional class method that will instantiate a
> content unit (but not save it) from an Artifact. Then core can offer
> an API that accepts a single Artifact binary and what content unit
> type it is and it should create+save the Artifact and then create+save
> the Content unit from that Artifact.
>
> This doesn't address adding it to/from repositories. That would be ok
> too as an additional option, but you probably want to do it to a whole
> bunch at once somehow. Also it will need to go through the tasking
> system for correctness.
>
>
> On Tue, Feb 19, 2019 at 11:52 AM David Davis <daviddavis at redhat.com
> <mailto:daviddavis at redhat.com>> wrote:
>
> What about the case of multi-artifact content? Don’t they require
> separate artifact creation and content creation routes?
>
> David
>
>
> On Tue, Feb 19, 2019 at 11:40 AM Austin Macdonald
> <austin at redhat.com <mailto:austin at redhat.com>> wrote:
>
> I think the key question to ask is:
> What circumstances would require the creation of Content that
> would not be met by a one-shot upload?
>
> On Tue, Feb 19, 2019 at 11:34 AM Daniel Alley
> <dalley at redhat.com <mailto:dalley at redhat.com>> wrote:
>
> @Matthias why would you prefer to keep the normal create?
> As you mention, the "orphan cleanup" issue applies to
> both, so there's no advantage given to the former.
>
> The normal "create" ( POST .../content/plugin/type/ ...)
> is unidiomatic and inconsistent, because the fields needed
> to upload a content are different from the fields on the
> actual serializer. Most content types keep metadata
> inside the packages themselves, so you can't let the user
> specify the content field values, you have to contort
> everything so that instead of hydrating the serializer
> from user input, it does so by parsing the content.
>
> There's also the issue that the libraries we're using to
> parse the (Python and RPM) packages do some validation on
> the filename to see that it has the right extension and so
> forth, and since artifacts are content-addressed and don't
> store that information, with normal create you have to
> pass the filename of the original artifact *back in* at
> the time you create it, and then copy the file from Pulp
> content storage into a temp directory under a filename
> which will validate properly, which is highly unfortunate.
>
> With one-shot upload, you avoid both of those problems,
> because there's no broken expectations as to what fields
> should be accepted, and because it should be possible
> (though I haven't tried it) to parse the original file
> *before* saving it as an artifact, thus avoiding a lot of
> mess. And you also get the option to add it straight into
> a repository. In my opinion, it's a straight upgrade.
>
> On Tue, Feb 19, 2019 at 10:57 AM Matthias Dellweg
> <dellweg at atix.de <mailto:dellweg at atix.de>> wrote:
>
> I have no objections to having the "one shot upload"
> or even "one shot
> upload into repositoryversion". I think, i would like
> to keep the
> 'traditional' create anyway.
> The problem i see with create and one shot upload is,
> that another
> thing could have triggered 'delete orphans' at the
> wrong time, and you
> shiny new content unit disappears, before you can add
> it to a
> repository version.
>
> On Mon, 18 Feb 2019 14:41:54 -0500
> Austin Macdonald <austin at redhat.com
> <mailto:austin at redhat.com>> wrote:
>
> > Originally, our upload story was as follows:
> > The user will upload a new file to Pulp via POST to
> /artifacts/
> > (provided by core)
> > The user will create a new plugin specific Content
> via POST to
> > /path/to/plugin/content/, referencing whatever
> artifacts that are
> > contained, and whatever fields are expected for the
> new content.
> > The user will add the new content to a repository
> via POST to
> > /repositories/1/versions/
> >
> > However, this is somewhat cumbersome to the user
> with 3 API calls to
> > accomplish something that only took one call in Pulp 2.
> >
> > There are a couple of different paths plugins have
> taken to improve
> > the user experience:
> > The Python plugin follows the above workflow, but
> reads the Artifact
> > file to determine the values for the fields. The RPM
> plugin has gone
> > even farther and created a new endpoint for "one
> shot" upload that
> > perform all of this in a single call. I think it is
> likely that the
> > Python plugin will move more in the "one shot"
> direction, and other
> > plugins will probably follow.
> >
> > That said, I think we should discuss this as a
> community to encourage
> > plugins to behave similarly, and because there may
> also be a
> > possibility for sharing some of code. It is my hope
> that a "one shot
> > upload" could do 2 things: 1) Upload and create
> Content. 2)
> > Optionally add that content to repositories.
> _______________________________________________
> Pulp-dev mailing list
> Pulp-dev at redhat.com <mailto:Pulp-dev at redhat.com>
> https://www.redhat.com/mailman/listinfo/pulp-dev
>
> _______________________________________________
> Pulp-dev mailing list
> Pulp-dev at redhat.com <mailto:Pulp-dev at redhat.com>
> https://www.redhat.com/mailman/listinfo/pulp-dev
>
> _______________________________________________
> Pulp-dev mailing list
> Pulp-dev at redhat.com <mailto:Pulp-dev at redhat.com>
> https://www.redhat.com/mailman/listinfo/pulp-dev
>
>
> _______________________________________________
> Pulp-dev mailing list
> Pulp-dev at redhat.com
> https://www.redhat.com/mailman/listinfo/pulp-dev
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/pulp-dev/attachments/20190222/93f0cf31/attachment.htm>
More information about the Pulp-dev
mailing list