[Pulp-dev] Changes in the Pulp 3 Upload story

Wed Feb 20 17:16:05 UTC 2019

The only backwards incompatible change would be that plugin writers would
have to add "CreateModelMixin" themselves and ContentViewSet wouldn't
inherit from it by default.

On Wed, Feb 20, 2019 at 11:39 AM David Davis <daviddavis at redhat.com> wrote:

> I am +1. Question: if we decide to keep (a), is method (b) mostly an
> additive change that can be addressed in pulpcore post-RC?
>
> I’m imagining that during the RC phase plugins can write their own
> one-shot uploaders and then after that, we can standardize on a solution we
> can move into pulpcore.
>
> David
>
>
> On Tue, Feb 19, 2019 at 12:30 PM Brian Bouterse <bbouters at redhat.com>
> wrote:
>
>> Overall +1, we should make a 1-shot uploader. It will need a bit of code
>> from plugin writers to make it work though.
>>
>> I believe the "manual and generic" feature offerings we have now are
>> pretty good. I don't think we should take them away.
>>
>> To describe yet another related use case; consider the uploading of a
>> tarball with a lot of units in it (maybe a whole repo). For instance bunch
>> of Ansible roles in a tarball is what a user would want to upload to Pulp,
>> so you would want to upload the whole tarball at once. I see three cases:
>>
>> a) The generic method (what we already have). The user supplies all
>> metadata because we can't know the right attributes without them telling it
>> to us.
>> b) The one-shot uploader to create a single content unit from a single
>> Artifact. I think this is what is being proposed here. +1 to this.
>> c) Uploading a bunch of content units in one binary, e.g. a tarball. We
>> should leave this to plugin writers to implement for now.
>>
>> In terms of implementing (b) I believe it would be useful to have a
>> Content unit provide an optional class method that will instantiate a
>> content unit (but not save it) from an Artifact. Then core can offer an API
>> that accepts a single Artifact binary and what content unit type it is and
>> it should create+save the Artifact and then create+save the Content unit
>> from that Artifact.
>>
>> This doesn't address adding it to/from repositories. That would be ok too
>> as an additional option, but you probably want to do it to a whole bunch at
>> once somehow. Also it will need to go through the tasking system for
>> correctness.
>>
>>
>> On Tue, Feb 19, 2019 at 11:52 AM David Davis <daviddavis at redhat.com>
>> wrote:
>>
>>> What about the case of multi-artifact content? Don’t they require
>>> separate artifact creation and content creation routes?
>>>
>>> David
>>>
>>>
>>> On Tue, Feb 19, 2019 at 11:40 AM Austin Macdonald <austin at redhat.com>
>>> wrote:
>>>
>>>> I think the key question to ask is:
>>>> What circumstances would require the creation of Content that would not
>>>> be met by a one-shot upload?
>>>>
>>>> On Tue, Feb 19, 2019 at 11:34 AM Daniel Alley <dalley at redhat.com>
>>>> wrote:
>>>>
>>>>> @Matthias why would you prefer to keep the normal create?  As you
>>>>> mention, the "orphan cleanup" issue applies to both, so there's no
>>>>> advantage given to the former.
>>>>>
>>>>> The normal "create" ( POST .../content/plugin/type/ ...) is
>>>>> unidiomatic and inconsistent, because the fields needed to upload a content
>>>>> are different from the fields on the actual serializer.  Most content types
>>>>> keep metadata inside the packages themselves, so you can't let the user
>>>>> specify the content field values, you have to contort everything so that
>>>>> instead of hydrating the serializer from user input, it does so by parsing
>>>>> the content.
>>>>>
>>>>> There's also the issue that the libraries we're using to parse the
>>>>> (Python and RPM) packages do some validation on the filename to see that it
>>>>> has the right extension and so forth, and since artifacts are
>>>>> content-addressed and don't store that information, with normal create you
>>>>> have to pass the filename of the original artifact *back in* at the time
>>>>> you create it, and then copy the file from Pulp content storage into a temp
>>>>> directory under a filename which will validate properly, which is highly
>>>>> unfortunate.
>>>>>
>>>>> With one-shot upload, you avoid both of those problems, because
>>>>> there's no broken expectations as to what fields should be accepted, and
>>>>> because it should be possible (though I haven't tried it) to parse the
>>>>> original file *before* saving it as an artifact, thus avoiding a lot of
>>>>> mess. And you also get the option to add it straight into a repository. In
>>>>> my opinion, it's a straight upgrade.
>>>>>
>>>>> On Tue, Feb 19, 2019 at 10:57 AM Matthias Dellweg <dellweg at atix.de>
>>>>> wrote:
>>>>>
>>>>>> I have no objections to having the "one shot upload" or even "one shot
>>>>>> upload into repositoryversion". I think, i would like to keep the
>>>>>> 'traditional' create anyway.
>>>>>> The problem i see with create and one shot upload is, that another
>>>>>> thing could have triggered 'delete orphans' at the wrong time, and you
>>>>>> shiny new content unit disappears, before you can add it to a
>>>>>> repository version.
>>>>>>
>>>>>> On Mon, 18 Feb 2019 14:41:54 -0500
>>>>>> Austin Macdonald <austin at redhat.com> wrote:
>>>>>>
>>>>>> > Originally, our upload story was as follows:
>>>>>> > The user will upload a new file to Pulp via POST to /artifacts/
>>>>>> > (provided by core)
>>>>>> > The user will create a new plugin specific Content via POST to
>>>>>> > /path/to/plugin/content/, referencing whatever artifacts that are
>>>>>> > contained, and whatever fields are expected for the new content.
>>>>>> > The user will add the new content to a repository via POST to
>>>>>> > /repositories/1/versions/
>>>>>> >
>>>>>> > However, this is somewhat cumbersome to the user with 3 API calls to
>>>>>> > accomplish something that only took one call in Pulp 2.
>>>>>> >
>>>>>> > There are a couple of different paths plugins have taken to improve
>>>>>> > the user experience:
>>>>>> > The Python plugin follows the above workflow, but reads the Artifact
>>>>>> > file to determine the values for the fields. The RPM plugin has gone
>>>>>> > even farther and created a new endpoint for "one shot" upload that
>>>>>> > perform all of this in a single call. I think it is likely that the
>>>>>> > Python plugin will move more in the "one shot" direction, and other
>>>>>> > plugins will probably follow.
>>>>>> >
>>>>>> > That said, I think we should discuss this as a community to
>>>>>> encourage
>>>>>> > plugins to behave similarly, and because there may also be a
>>>>>> > possibility for sharing some of code. It is my hope that a "one shot
>>>>>> > upload" could do 2 things: 1) Upload and create Content. 2)
>>>>>> > Optionally add that content to repositories.
>>>>>> _______________________________________________
>>>>>> Pulp-dev mailing list
>>>>>> Pulp-dev at redhat.com
>>>>>> https://www.redhat.com/mailman/listinfo/pulp-dev
>>>>>>
>>>>> _______________________________________________
>>>> Pulp-dev mailing list
>>>> Pulp-dev at redhat.com
>>>> https://www.redhat.com/mailman/listinfo/pulp-dev
>>>>
>>> _______________________________________________
>>> Pulp-dev mailing list
>>> Pulp-dev at redhat.com
>>> https://www.redhat.com/mailman/listinfo/pulp-dev
>>>
>> _______________________________________________
> Pulp-dev mailing list
> Pulp-dev at redhat.com
> https://www.redhat.com/mailman/listinfo/pulp-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/pulp-dev/attachments/20190220/02035527/attachment.htm>