[Pulp-dev] Pulp 3 content import

Dennis Kliban dkliban at redhat.com
Wed Apr 18 17:12:19 UTC 2018


Thank you Austin and David for correcting me.

It sounds like each plugin will implement how to handle creating
SomeContent from 0 or more artifacts. This will include having the server
inspect the Artifacts and generate required metadata. In this case the
client will be able to import content into Pulp in 2 steps:

1) Upload the Artifact using Artifact creation REST API
2) Create content using the SomeContent creation REST API that accepts a
list of Artifacts to use for creating SomeContent.





On Wed, Apr 18, 2018 at 12:46 PM, David Davis <daviddavis at redhat.com> wrote:

> I actually wrote this code[0] in Pulp 2 for the puppet plugin so maybe I
> can chime in.
>
> > As you pointed out, it requires the REST API client to know all the
> metadata required to create a piece of Content from an Artifact.
>
> This not quite true. The pulp_file plugin currently extracts metadata (ie
> digest) from the artifact when a content unit is created:
>
> https://github.com/pulp/pulp_file/blob/master/pulp_file/
> app/viewsets.py#L78
>
> I think something similar could be done for other content units.
>
> To handle both upload and sync, one option is to use a django signal that
> will extract the metadata from the puppet module:
>
> https://docs.djangoproject.com/en/2.0/topics/signals/
>
> I’d probably create a model method though on your Content model that both
> the sync and content unit creation code can call.
>
> [0] https://github.com/pulp/pulp_puppet/pull/126
>
>
> David
>
> On Wed, Apr 18, 2018 at 12:03 PM, Dennis Kliban <dkliban at redhat.com>
> wrote:
>
>> On Wed, Apr 18, 2018 at 11:02 AM, Simon Baatz <gmbnomis at gmail.com> wrote:
>>
>>> I am looking at Pulp 3 for the first time and have a question on
>>> creating/importing content units.
>>>
>>> In the file plugin, a file content unit can be created using
>>> POST at .../content/file/ by passing the meta-data ('relative_path') and
>>> the
>>> artifact (from which 'digest' is obtained as additional meta-data).  The
>>> POST is synchronous (returning a 201 CREATED if successful).
>>>
>>> However, for other artifact types there is more meta-data in the
>>> artifact(s)
>>> themselves and it may be "harder" to obtain.  For example, Puppet
>>> modules,
>>> Chef cookbooks, and Helm charts are tarballs with specific meta-data
>>> files.
>>> In Pulp 2, this meta-data extraction seems to be part of the
>>> import_uploaded_unit task.  According to the code comments, the puppet
>>> module unpacks the entire module to get the meta-data.
>>>
>>> If I got this correctly, Pulp 3 splits this up into content unit
>>> creation (sync) and "importing" (async, add content units using
>>> POST at ...versions/).  Where should the meta-data extraction be done
>>> given this split?  (the first step is synchronous, but perhaps it is
>>> acceptable to do some work there nevertheless. The second step
>>> looks too late)
>>>
>>>
>> The REST API provided by pulpcore in Pulp 3 is very generic. As you
>> pointed out, it requires the REST API client to know all the metadata
>> required to create a piece of Content from an Artifact. So I picture this
>> process as a 3 step process in the client:
>>
>> 1) Inspect the artifact to extract all needed metadata
>> 2) Use Artifact creation REST API to upload Artifact to Pulp
>> 3) Use SomeContent creation REST API to create SomeContent from the
>> uploaded Artifact and the metadata extracted in step 1
>>
>> With all that said, each plugin is also be able to provide additional
>> REST API endpoints. I suspect that some of these endpoints will support
>> "smart upload" that will be aware of the content type that is being
>> uploaded. These endpoints will allow users to upload an Artifact of a
>> specific type and the server will extract all the needed metadata to create
>> the SomeContent in Pulp.
>>
>>
>>> - Simon
>>>
>>> _______________________________________________
>>> Pulp-dev mailing list
>>> Pulp-dev at redhat.com
>>> https://www.redhat.com/mailman/listinfo/pulp-dev
>>>
>>
>>
>> _______________________________________________
>> Pulp-dev mailing list
>> Pulp-dev at redhat.com
>> https://www.redhat.com/mailman/listinfo/pulp-dev
>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/pulp-dev/attachments/20180418/2560af35/attachment.htm>


More information about the Pulp-dev mailing list