[Pulp-dev] proposing changes to pulp 3 upload API
Jeff Ortel
jortel at redhat.com
Tue Jun 27 15:27:54 UTC 2017
On 06/27/2017 08:36 AM, Brian Bouterse wrote:
> I thought that we pulled out the chunking uploads from the MVP. IIRC, @jortel and I thought since that use
> case was for high performing (parallel) uploads and it should be on the 3.1+ page.
>
> +1 to just sending data without having a file handle. If the entire file is delivered in one request then
> having a file ID to upload to in a second request is just cumbersome.
> +1 to having the handler receiving that file just make it an Artifact() right away. This will work better with
> how Django handles file uploads.
A few things about uploading directly to an Artifact:
- The artifact FK to a content unit would need to become optional.
- Need to add use cases for cleaning up artifacts not associated with a content unit.
- The upload API would need additional information needed to create an artifact. Like relative path, size,
checksums etc.
- Since (I assume) you are proposing uploading/writing directly to artifact storage (not staging in a working
dir), the flow would need to involve (optional) validation. If validation fails, the artifact must not be
inserted into the DB.
-jeff
>
> I also think we can skip making one Artifact from another. That is not going to be a commonly used use case I
> think. So removing that use case and chunking that would be:
>
> * As an authenticated user, I can upload a file which becomes an Artifact. At the end up the of upload, the
> server returns the JSON representation of the created Artifact.
> * As an authenticated user, I can create a content unit by providing the content type, its Artifacts using
> IDs for each Artifact, and the metadata supplied in the POST body. This call is atomic, content unit is
> created in the database and on the filesystem or not at all.
>
> The biggest reason I think to do this adjustment is to aligns with the users desire to have uploads take fewer
> calls. This removes at least two calls from the workflow. It also avoids having to save the data multiple
> times which I don't think we can do practically.
>
> Thoughts or ideas?
>
> -Brian
>
> On Tue, Jun 27, 2017 at 8:55 AM, Dennis Kliban <dkliban at redhat.com <mailto:dkliban at redhat.com>> wrote:
>
> My motivations for writing this email include: recent discussion about pulp 2 upload API in #pulp and
> django's documentation on file uploads.
>
> Files uploaded to Django are initially stored in memory (if under 2.5 mb) or Python's tempfile module is
> used to write it to /tmp/ directory. The file created in /tmp is deleted when and if the last file handle
> is closed.
>
> If we implement the upload API as described in the MVP doc[0], then according to Django docs[1] we will be
> performing a write to disk 2 or 3 times for each upload. In cases where a file is bigger than 2.5mb in
> size, it will be first written to /tmp. The same file will then be written to /var/lib/pulp/uploads (or
> similar location) when the FileUpload model is saved. A third write will occur when an artifact is created
> using the FileUpload. This third write will likely be a move though.
>
> I propose that we eliminate writing the uploaded file to /var/lib/pulp/upload and go directly to creating
> an artifact. The use cases can then be rewritten as the following:
>
> * As an authenticated user, I can upload a file with an optional chunk size, and an optional offset. At
> the end up the of upload the server returns the JSON representation of the artifact.
>
>
> * As an authenticated user, I can create a new artifact by specifying an existing artifact id.
>
>
> * As an authenticated user, I can create a content unit by providing the content type, its Artifacts
> using IDs for each Artifact, and the metadata supplied in the POST body. This call is atomic, content
> unit is created in the database and on the filesystem or not at all.
>
>
>
>
> [0] https://pulp.plan.io/projects/pulp/wiki/Pulp_3_Minimum_Viable_Product#Upload-amp-Copy
> <https://pulp.plan.io/projects/pulp/wiki/Pulp_3_Minimum_Viable_Product#Upload-amp-Copy>
> [1] https://docs.djangoproject.com/en/1.9/topics/http/file-uploads/#handling-uploaded-files-with-a-model
> <https://docs.djangoproject.com/en/1.9/topics/http/file-uploads/#handling-uploaded-files-with-a-model>
>
> _______________________________________________
> Pulp-dev mailing list
> Pulp-dev at redhat.com <mailto:Pulp-dev at redhat.com>
> https://www.redhat.com/mailman/listinfo/pulp-dev <https://www.redhat.com/mailman/listinfo/pulp-dev>
>
>
>
>
> _______________________________________________
> Pulp-dev mailing list
> Pulp-dev at redhat.com
> https://www.redhat.com/mailman/listinfo/pulp-dev
>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 847 bytes
Desc: OpenPGP digital signature
URL: <http://listman.redhat.com/archives/pulp-dev/attachments/20170627/187aebb6/attachment.sig>
More information about the Pulp-dev
mailing list