[Pulp-dev] Pulp 3 content import

[David Davis]

I actually wrote this code[0] in Pulp 2 for the puppet plugin so maybe I
can chime in.

> As you pointed out, it requires the REST API client to know all the
metadata required to create a piece of Content from an Artifact.

This not quite true. The pulp_file plugin currently extracts metadata (ie
digest) from the artifact when a content unit is created:

https://github.com/pulp/pulp_file/blob/master/pulp_file/app/viewsets.py#L78

I think something similar could be done for other content units.

To handle both upload and sync, one option is to use a django signal that
will extract the metadata from the puppet module:

https://docs.djangoproject.com/en/2.0/topics/signals/

I’d probably create a model method though on your Content model that both
the sync and content unit creation code can call.

[0] https://github.com/pulp/pulp_puppet/pull/126


David

On Wed, Apr 18, 2018 at 12:03 PM, Dennis Kliban <dkliban at redhat.com> wrote:

> On Wed, Apr 18, 2018 at 11:02 AM, Simon Baatz <gmbnomis at gmail.com> wrote:
>
>> I am looking at Pulp 3 for the first time and have a question on
>> creating/importing content units.
>>
>> In the file plugin, a file content unit can be created using
>> POST at .../content/file/ by passing the meta-data ('relative_path') and the
>> artifact (from which 'digest' is obtained as additional meta-data).  The
>> POST is synchronous (returning a 201 CREATED if successful).
>>
>> However, for other artifact types there is more meta-data in the
>> artifact(s)
>> themselves and it may be "harder" to obtain.  For example, Puppet modules,
>> Chef cookbooks, and Helm charts are tarballs with specific meta-data
>> files.
>> In Pulp 2, this meta-data extraction seems to be part of the
>> import_uploaded_unit task.  According to the code comments, the puppet
>> module unpacks the entire module to get the meta-data.
>>
>> If I got this correctly, Pulp 3 splits this up into content unit
>> creation (sync) and "importing" (async, add content units using
>> POST at ...versions/).  Where should the meta-data extraction be done
>> given this split?  (the first step is synchronous, but perhaps it is
>> acceptable to do some work there nevertheless. The second step
>> looks too late)
>>
>>
> The REST API provided by pulpcore in Pulp 3 is very generic. As you
> pointed out, it requires the REST API client to know all the metadata
> required to create a piece of Content from an Artifact. So I picture this
> process as a 3 step process in the client:
>
> 1) Inspect the artifact to extract all needed metadata
> 2) Use Artifact creation REST API to upload Artifact to Pulp
> 3) Use SomeContent creation REST API to create SomeContent from the
> uploaded Artifact and the metadata extracted in step 1
>
> With all that said, each plugin is also be able to provide additional REST
> API endpoints. I suspect that some of these endpoints will support "smart
> upload" that will be aware of the content type that is being uploaded.
> These endpoints will allow users to upload an Artifact of a specific type
> and the server will extract all the needed metadata to create the
> SomeContent in Pulp.
>
>
>> - Simon
>>
>> _______________________________________________
>> Pulp-dev mailing list
>> Pulp-dev at redhat.com
>> https://www.redhat.com/mailman/listinfo/pulp-dev
>>
>
>
> _______________________________________________
> Pulp-dev mailing list
> Pulp-dev at redhat.com
> https://www.redhat.com/mailman/listinfo/pulp-dev
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/pulp-dev/attachments/20180418/18d77a47/attachment.htm>