[Pulp-dev] Changes in the Pulp 3 Upload story

Wed Feb 20 16:38:02 UTC 2019

I am +1. Question: if we decide to keep (a), is method (b) mostly an
additive change that can be addressed in pulpcore post-RC?

I’m imagining that during the RC phase plugins can write their own one-shot
uploaders and then after that, we can standardize on a solution we can move
into pulpcore.

David

On Tue, Feb 19, 2019 at 12:30 PM Brian Bouterse <bbouters at redhat.com> wrote:

> Overall +1, we should make a 1-shot uploader. It will need a bit of code
> from plugin writers to make it work though.
>
> I believe the "manual and generic" feature offerings we have now are
> pretty good. I don't think we should take them away.
>
> To describe yet another related use case; consider the uploading of a
> tarball with a lot of units in it (maybe a whole repo). For instance bunch
> of Ansible roles in a tarball is what a user would want to upload to Pulp,
> so you would want to upload the whole tarball at once. I see three cases:
>
> a) The generic method (what we already have). The user supplies all
> metadata because we can't know the right attributes without them telling it
> to us.
> b) The one-shot uploader to create a single content unit from a single
> Artifact. I think this is what is being proposed here. +1 to this.
> c) Uploading a bunch of content units in one binary, e.g. a tarball. We
> should leave this to plugin writers to implement for now.
>
> In terms of implementing (b) I believe it would be useful to have a
> Content unit provide an optional class method that will instantiate a
> content unit (but not save it) from an Artifact. Then core can offer an API
> that accepts a single Artifact binary and what content unit type it is and
> it should create+save the Artifact and then create+save the Content unit
> from that Artifact.
>
> This doesn't address adding it to/from repositories. That would be ok too
> as an additional option, but you probably want to do it to a whole bunch at
> once somehow. Also it will need to go through the tasking system for
> correctness.
>
>
> On Tue, Feb 19, 2019 at 11:52 AM David Davis <daviddavis at redhat.com>
> wrote:
>
>> What about the case of multi-artifact content? Don’t they require
>> separate artifact creation and content creation routes?
>>
>> David
>>
>>
>> On Tue, Feb 19, 2019 at 11:40 AM Austin Macdonald <austin at redhat.com>
>> wrote:
>>
>>> I think the key question to ask is:
>>> What circumstances would require the creation of Content that would not
>>> be met by a one-shot upload?
>>>
>>> On Tue, Feb 19, 2019 at 11:34 AM Daniel Alley <dalley at redhat.com> wrote:
>>>
>>>> @Matthias why would you prefer to keep the normal create?  As you
>>>> mention, the "orphan cleanup" issue applies to both, so there's no
>>>> advantage given to the former.
>>>>
>>>> The normal "create" ( POST .../content/plugin/type/ ...) is unidiomatic
>>>> and inconsistent, because the fields needed to upload a content are
>>>> different from the fields on the actual serializer.  Most content types
>>>> keep metadata inside the packages themselves, so you can't let the user
>>>> specify the content field values, you have to contort everything so that
>>>> instead of hydrating the serializer from user input, it does so by parsing
>>>> the content.
>>>>
>>>> There's also the issue that the libraries we're using to parse the
>>>> (Python and RPM) packages do some validation on the filename to see that it
>>>> has the right extension and so forth, and since artifacts are
>>>> content-addressed and don't store that information, with normal create you
>>>> have to pass the filename of the original artifact *back in* at the time
>>>> you create it, and then copy the file from Pulp content storage into a temp
>>>> directory under a filename which will validate properly, which is highly
>>>> unfortunate.
>>>>
>>>> With one-shot upload, you avoid both of those problems, because there's
>>>> no broken expectations as to what fields should be accepted, and because it
>>>> should be possible (though I haven't tried it) to parse the original file
>>>> *before* saving it as an artifact, thus avoiding a lot of mess. And you
>>>> also get the option to add it straight into a repository. In my opinion,
>>>> it's a straight upgrade.
>>>>
>>>> On Tue, Feb 19, 2019 at 10:57 AM Matthias Dellweg <dellweg at atix.de>
>>>> wrote:
>>>>
>>>>> I have no objections to having the "one shot upload" or even "one shot
>>>>> upload into repositoryversion". I think, i would like to keep the
>>>>> 'traditional' create anyway.
>>>>> The problem i see with create and one shot upload is, that another
>>>>> thing could have triggered 'delete orphans' at the wrong time, and you
>>>>> shiny new content unit disappears, before you can add it to a
>>>>> repository version.
>>>>>
>>>>> On Mon, 18 Feb 2019 14:41:54 -0500
>>>>> Austin Macdonald <austin at redhat.com> wrote:
>>>>>
>>>>> > Originally, our upload story was as follows:
>>>>> > The user will upload a new file to Pulp via POST to /artifacts/
>>>>> > (provided by core)
>>>>> > The user will create a new plugin specific Content via POST to
>>>>> > /path/to/plugin/content/, referencing whatever artifacts that are
>>>>> > contained, and whatever fields are expected for the new content.
>>>>> > The user will add the new content to a repository via POST to
>>>>> > /repositories/1/versions/
>>>>> >
>>>>> > However, this is somewhat cumbersome to the user with 3 API calls to
>>>>> > accomplish something that only took one call in Pulp 2.
>>>>> >
>>>>> > There are a couple of different paths plugins have taken to improve
>>>>> > the user experience:
>>>>> > The Python plugin follows the above workflow, but reads the Artifact
>>>>> > file to determine the values for the fields. The RPM plugin has gone
>>>>> > even farther and created a new endpoint for "one shot" upload that
>>>>> > perform all of this in a single call. I think it is likely that the
>>>>> > Python plugin will move more in the "one shot" direction, and other
>>>>> > plugins will probably follow.
>>>>> >
>>>>> > That said, I think we should discuss this as a community to encourage
>>>>> > plugins to behave similarly, and because there may also be a
>>>>> > possibility for sharing some of code. It is my hope that a "one shot
>>>>> > upload" could do 2 things: 1) Upload and create Content. 2)
>>>>> > Optionally add that content to repositories.
>>>>> _______________________________________________
>>>>> Pulp-dev mailing list
>>>>> Pulp-dev at redhat.com
>>>>> https://www.redhat.com/mailman/listinfo/pulp-dev
>>>>>
>>>> _______________________________________________
>>> Pulp-dev mailing list
>>> Pulp-dev at redhat.com
>>> https://www.redhat.com/mailman/listinfo/pulp-dev
>>>
>> _______________________________________________
>> Pulp-dev mailing list
>> Pulp-dev at redhat.com
>> https://www.redhat.com/mailman/listinfo/pulp-dev
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/pulp-dev/attachments/20190220/78345062/attachment.htm>