[Pulp-dev] Pulp 3 plugin for Chef cookbooks

[Simon Baatz]

I created the  beginnings of a Pulp 3 plugin to manage Chef cookbooks
[1].  Currently, it supports to create repos, create cookbook content
units, and publish repos.  A published & distributed repo will offer
a "universe" API endpoint for tools like Berkshelf.

I did not implement sync yet. I am waiting for "PendingVersion" to be available
first.

I ran into a couple of problems/uncertainties described below (sorry for the
lengthy mail). I am new to Django, DRF, and, obviously, Pulp 3 so any remarks or
suggestions are welcome:

- Create Content: The plugin reads as much meta-data as possible from the actual
  cookbook Artifact when creating a content unit. The motivation for this is:

  - One doesn't need a special tool to upload content, which makes uploading by e.g.
    a CI job easier.
  - It ensures consistency between metadata stored in Pulp and the actual
    metadata in the cookbook.

  However, this requires to extract a metadata file from a gzipped tar archive.
  Content unit creation is synchronous and doing this work in a synchronous call
  might not be optimal (we had a discussion in this topic on the pulp-dev
  mailing list already).

- Publication/Distribution: The metadata file ("universe") for a published
  cookbook repository contains absolute URLs for download (i.e. these point
  to published artifacts in a distribution).

  The current publication/distribution concept seems to have the underlying
  assumption that a Publication is fully relocatable: PublishedMetadata
  artifacts are created by the publishing task and creating a Distribution is a
  synchronous call that determines the base path of the published artifacts.

  This causes a problem with said "universe" file. Ideally, it could be
  pre-computed (it lists all artifacts in the repo).  However, this can't be
  done AFAIK since the base path is unknown at publication time and one can't
  generate additional metadata artifacts for a specific distribution later.

  The best solution I came up with was to implement a dynamic API. To reduce the
  amount of work to be done, the API does a simple string replacement: During
  publication, the universe file is pre-computed using placeholders. In the
  dynamic API these placeholders are replaced with the actual base URL of the
  distribution.

  However, I would prefer not to be forced to implement a dynamic API for static
  information. Is there a way to solve this differently?

- Content-Type Header: The "universe" file is JSON and must have a corresponding
  "Content-Type" HTTP header.

  However, content type of the development server seems to be "text/html" by
  default for all artifacts. Apparently, I can't set the content-type of a
  (meta-data) artifact(?)

- Getting the base url of a distribution in the dynamic API is surprisingly
  complicated and depends on the inner structure of pulp core (I took the
  implementation from 'pulp_ansible'). IMHO, a well defined way to obtain it
  should be part of the plugin API.

- "Content" class: The way to use only a single artifact in Content (like done
  in pulp_file) seems to require in-depth knowledge of the
  Content/ContentSerializer class and its inner workings.

  The downside of this can already be experienced in the "pulp_file" plugin: The
  fields "id" and "created" are missing, since the implementation there just
  overrides the 'fields' in the serializer).

  I think two Content types should be part of the plugin API: one with
  multiple artifacts, and a simpler one with a single artifact

- Uploading an Artifact that already exists returns an error, which is
  annoying if you use http/curl to import artifacts. Suppose some other user
  uploaded an artifact in the past. You won't get useful
  information from the POST request uploading the same artifact:

  HTTP/1.1 400 Bad Request
  Allow: GET, POST, HEAD, OPTIONS
  Content-Type: application/json
  Date: Sat, 12 May 2018 17:50:54 GMT
  Server: WSGIServer/0.2 CPython/3.6.2
  Vary: Accept

  {
      "non_field_errors": [
          "sha512 checksum must be unique."
      ]
  }

  This forced me to do something like:

    ...
    sha256=$(sha256sum "$targz" | awk '{print $1}')
    ARTIFACT_HREF=$(http :8000/pulp/api/v3/artifacts/?sha256=$sha256 | jq -r '.results[0]._href')
    if [[ $ARTIFACT_HREF == "null" ]]; then
        echo uploading artifact $cookbook_name sha256: $sha256
        http --form POST http://localhost:8000/pulp/api/v3/artifacts/ file@$targz
        ARTIFACT_HREF=$(http :8000/pulp/api/v3/artifacts/?sha256=$sha256 | jq -r '.results[0]._href')
        ...

  Perhaps a "303 See Other" to the existing artifact would help here.



[1]: https://github.com/gmbnomis/pulp_cookbook