[Pulp-dev] Changes in the Pulp 3 Upload story

Tue Feb 19 17:30:22 UTC 2019

Overall +1, we should make a 1-shot uploader. It will need a bit of code
from plugin writers to make it work though.

I believe the "manual and generic" feature offerings we have now are pretty
good. I don't think we should take them away.

To describe yet another related use case; consider the uploading of a
tarball with a lot of units in it (maybe a whole repo). For instance bunch
of Ansible roles in a tarball is what a user would want to upload to Pulp,
so you would want to upload the whole tarball at once. I see three cases:

a) The generic method (what we already have). The user supplies all
metadata because we can't know the right attributes without them telling it
to us.
b) The one-shot uploader to create a single content unit from a single
Artifact. I think this is what is being proposed here. +1 to this.
c) Uploading a bunch of content units in one binary, e.g. a tarball. We
should leave this to plugin writers to implement for now.

In terms of implementing (b) I believe it would be useful to have a Content
unit provide an optional class method that will instantiate a content unit
(but not save it) from an Artifact. Then core can offer an API that accepts
a single Artifact binary and what content unit type it is and it should
create+save the Artifact and then create+save the Content unit from that
Artifact.

This doesn't address adding it to/from repositories. That would be ok too
as an additional option, but you probably want to do it to a whole bunch at
once somehow. Also it will need to go through the tasking system for
correctness.

On Tue, Feb 19, 2019 at 11:52 AM David Davis <daviddavis at redhat.com> wrote:

> What about the case of multi-artifact content? Don’t they require separate
> artifact creation and content creation routes?
>
> David
>
>
> On Tue, Feb 19, 2019 at 11:40 AM Austin Macdonald <austin at redhat.com>
> wrote:
>
>> I think the key question to ask is:
>> What circumstances would require the creation of Content that would not
>> be met by a one-shot upload?
>>
>> On Tue, Feb 19, 2019 at 11:34 AM Daniel Alley <dalley at redhat.com> wrote:
>>
>>> @Matthias why would you prefer to keep the normal create?  As you
>>> mention, the "orphan cleanup" issue applies to both, so there's no
>>> advantage given to the former.
>>>
>>> The normal "create" ( POST .../content/plugin/type/ ...) is unidiomatic
>>> and inconsistent, because the fields needed to upload a content are
>>> different from the fields on the actual serializer.  Most content types
>>> keep metadata inside the packages themselves, so you can't let the user
>>> specify the content field values, you have to contort everything so that
>>> instead of hydrating the serializer from user input, it does so by parsing
>>> the content.
>>>
>>> There's also the issue that the libraries we're using to parse the
>>> (Python and RPM) packages do some validation on the filename to see that it
>>> has the right extension and so forth, and since artifacts are
>>> content-addressed and don't store that information, with normal create you
>>> have to pass the filename of the original artifact *back in* at the time
>>> you create it, and then copy the file from Pulp content storage into a temp
>>> directory under a filename which will validate properly, which is highly
>>> unfortunate.
>>>
>>> With one-shot upload, you avoid both of those problems, because there's
>>> no broken expectations as to what fields should be accepted, and because it
>>> should be possible (though I haven't tried it) to parse the original file
>>> *before* saving it as an artifact, thus avoiding a lot of mess. And you
>>> also get the option to add it straight into a repository. In my opinion,
>>> it's a straight upgrade.
>>>
>>> On Tue, Feb 19, 2019 at 10:57 AM Matthias Dellweg <dellweg at atix.de>
>>> wrote:
>>>
>>>> I have no objections to having the "one shot upload" or even "one shot
>>>> upload into repositoryversion". I think, i would like to keep the
>>>> 'traditional' create anyway.
>>>> The problem i see with create and one shot upload is, that another
>>>> thing could have triggered 'delete orphans' at the wrong time, and you
>>>> shiny new content unit disappears, before you can add it to a
>>>> repository version.
>>>>
>>>> On Mon, 18 Feb 2019 14:41:54 -0500
>>>> Austin Macdonald <austin at redhat.com> wrote:
>>>>
>>>> > Originally, our upload story was as follows:
>>>> > The user will upload a new file to Pulp via POST to /artifacts/
>>>> > (provided by core)
>>>> > The user will create a new plugin specific Content via POST to
>>>> > /path/to/plugin/content/, referencing whatever artifacts that are
>>>> > contained, and whatever fields are expected for the new content.
>>>> > The user will add the new content to a repository via POST to
>>>> > /repositories/1/versions/
>>>> >
>>>> > However, this is somewhat cumbersome to the user with 3 API calls to
>>>> > accomplish something that only took one call in Pulp 2.
>>>> >
>>>> > There are a couple of different paths plugins have taken to improve
>>>> > the user experience:
>>>> > The Python plugin follows the above workflow, but reads the Artifact
>>>> > file to determine the values for the fields. The RPM plugin has gone
>>>> > even farther and created a new endpoint for "one shot" upload that
>>>> > perform all of this in a single call. I think it is likely that the
>>>> > Python plugin will move more in the "one shot" direction, and other
>>>> > plugins will probably follow.
>>>> >
>>>> > That said, I think we should discuss this as a community to encourage
>>>> > plugins to behave similarly, and because there may also be a
>>>> > possibility for sharing some of code. It is my hope that a "one shot
>>>> > upload" could do 2 things: 1) Upload and create Content. 2)
>>>> > Optionally add that content to repositories.
>>>> _______________________________________________
>>>> Pulp-dev mailing list
>>>> Pulp-dev at redhat.com
>>>> https://www.redhat.com/mailman/listinfo/pulp-dev
>>>>
>>> _______________________________________________
>> Pulp-dev mailing list
>> Pulp-dev at redhat.com
>> https://www.redhat.com/mailman/listinfo/pulp-dev
>>
> _______________________________________________
> Pulp-dev mailing list
> Pulp-dev at redhat.com
> https://www.redhat.com/mailman/listinfo/pulp-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/pulp-dev/attachments/20190219/36d87d65/attachment.htm>