<div dir="ltr"><div><div><div><div><div><div><div>For a file to be received and saved in the right place once, we need the view saving the file to have all the info to form the complete path. After talking w/ @jortel, I think we should store Artifacts at the following path:<br><br>MEDIA_ROOT/content/units/digest[0:2]/digest[2:]/<rel_path><br><br></div>Note that digest is the Artifact's sha256 digest. This is different from pulp2 which used the digest of the content unit. Note that <rel_path> would be provided by the user along with <size> and/or <checksum_digest>.<br><br></div>Note that this will cause an Artifact to live in exactly one place which means Artifacts are now unique by digest and would need to be able to be associated with multiple content units. I'm not sure why we didn't do this before, so I'm interested in exploring issues associated with this.<br><br></div>It would be a good workflow. For a single file content unit (e.g.) rpm upload would be a two step process.<br><br></div>1. POST/PUT the file's binary data and the <relative_path> and <size> and/or <checksum_digest> as GET parameters<br></div>2. Create a content unit with the unit metadata, and 0 .. n Artifacts referred to by ID. This could optionally associate the new unit with one repository as part of the atomic unit creation.<br><br></div>Thoughts/Ideas?<br><br></div>-Brian<br><div><div><div><div><div><div><br></div></div></div></div></div></div></div><div class="gmail_extra"><br><div class="gmail_quote">On Tue, Jun 27, 2017 at 4:16 PM, Dennis Kliban <span dir="ltr"><<a href="mailto:dkliban@redhat.com" target="_blank">dkliban@redhat.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div class="gmail_extra"><div class="gmail_quote"><span class="">On Tue, Jun 27, 2017 at 3:31 PM, Michael Hrivnak <span dir="ltr"><<a href="mailto:mhrivnak@redhat.com" target="_blank">mhrivnak@redhat.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">Could you re-summarize what problem would be solved by not having a FileUpload model, and giving the Artifact model the ability to have partial data and no Content foreign key?<div><br></div><div>I understand the concern about where on the filesystem the data gets written and how many times, but I'm not seeing how that's related to whether we have a FileUpload model or not. Are we discussing two separate issues? 1) filesystem locations and copy efficiency, and 2) API design? Or is this discussion trying to connect them in a way I'm not seeing?</div></div></blockquote><div><br></div></span><div>There were two concerns: 1) Filesystem location and copy efficiency 2) API design<br><br>The first one has been addressed. Thank you for pointing out that a second write will be a move operation.<br><br>However, I am still concerned about the complexity of the API. A relatively small file should not require an upload session to be uploaded. A single API call to the Artifacts API should be enough to upload a file and create an Artifact from it. In Pulp 3.1+ we can introduce the FileUpload model to support chunked uploads. At the same time we would extend the Artifact API to accept a FileUpload id for creating an Artifact. <br></div><div><div class="h5"><div><br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div class="m_-5778278824315326441m_-780941529446452707HOEnZb"><div class="m_-5778278824315326441m_-780941529446452707h5"><div class="gmail_extra"><br><div class="gmail_quote">On Tue, Jun 27, 2017 at 3:20 PM, Dennis Kliban <span dir="ltr"><<a href="mailto:dkliban@redhat.com" target="_blank">dkliban@redhat.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div class="gmail_extra"><div class="gmail_quote"><span>On Tue, Jun 27, 2017 at 2:56 PM, Brian Bouterse <span dir="ltr"><<a href="mailto:bbouters@redhat.com" target="_blank">bbouters@redhat.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div>Picking up from @jortel's observations...<br><br><div>+1 to allowing Artifacts to have an optional FK.<br><br></div><div>If we have an Artifacts endpoint then we can allow for the deleting of a single artifact if it has no FK. I think we want to disallow the removal of an Artifact that has a foreign key. Also filtering should allow a single operation to clean up all unassociated artifacts by searching for FK=None or similar.<br><br></div><div>Yes, we will need to allow the single call delivering a file to also specify the relative path, size, checksums etc. Since the POST body contains binary data we either need to accept this data as GET style params or use a multi-part MIME upload [0]. Note that this creation of an Artifact does not change the repository contents and therefore can be handled synchronously outside of the tasking system.<br><br></div><div>+1 to the saving of an Artifact to perform validation<br></div><br>[0]: <a href="https://www.w3.org/Protocols/rfc1341/7_2_Multipart.html" target="_blank">https://www.w3.org/Protocols/r<wbr>fc1341/7_2_Multipart.html</a><span class="m_-5778278824315326441m_-780941529446452707m_4609881861521469005m_-5142219127422626088m_-57592284970400827HOEnZb"></span><br> </div></div></blockquote><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div><span class="m_-5778278824315326441m_-780941529446452707m_4609881861521469005m_-5142219127422626088m_-57592284970400827HOEnZb"><font color="#888888"><br></font></span></div><span class="m_-5778278824315326441m_-780941529446452707m_4609881861521469005m_-5142219127422626088m_-57592284970400827HOEnZb"><font color="#888888">-Brian<br></font></span></div></blockquote><div><br></div></span><div>I also support this optional FK for Artifacts and validation on save. We should probably stick with accepting GET parameters for the MVP. Though multi-part MIME support would be good to consider for 3.1+.<br></div><div><div class="m_-5778278824315326441m_-780941529446452707m_4609881861521469005h5"><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><span class="m_-5778278824315326441m_-780941529446452707m_4609881861521469005m_-5142219127422626088m_-57592284970400827HOEnZb"><font color="#888888"></font></span></div><div class="m_-5778278824315326441m_-780941529446452707m_4609881861521469005m_-5142219127422626088m_-57592284970400827HOEnZb"><div class="m_-5778278824315326441m_-780941529446452707m_4609881861521469005m_-5142219127422626088m_-57592284970400827h5"><div class="gmail_extra"><br><div class="gmail_quote">On Tue, Jun 27, 2017 at 2:44 PM, Dennis Kliban <span dir="ltr"><<a href="mailto:dkliban@redhat.com" target="_blank">dkliban@redhat.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div class="gmail_extra"><div class="gmail_quote"><span>On Tue, Jun 27, 2017 at 1:24 PM, Michael Hrivnak <span dir="ltr"><<a href="mailto:mhrivnak@redhat.com" target="_blank">mhrivnak@redhat.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div><span class="m_-5778278824315326441m_-780941529446452707m_4609881861521469005m_-5142219127422626088m_-57592284970400827m_8458294198575060412m_-7313271489355059228gmail-"><br><div class="gmail_quote">On Tue, Jun 27, 2017 at 11:27 AM, Jeff Ortel <span dir="ltr"><<a href="mailto:jortel@redhat.com" target="_blank">jortel@redhat.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><br>
- The artifact FK to a content unit would need to become optional.<br>
<br>
- Need to add use cases for cleaning up artifacts not associated with a content unit.<br>
<br>
- The upload API would need additional information needed to create an artifact. Like relative path, size,<br>
checksums etc.<br>
<br>
- Since (I assume) you are proposing uploading/writing directly to artifact storage (not staging in a working<br>
dir), the flow would need to involve (optional) validation. If validation fails, the artifact must not be<br>
inserted into the DB.</blockquote></div><br></span>Perhaps a decent middle ground would be to stick with the plan of keeping uploaded (or partially uploaded) files as a separate model until they are ready to be turned into a Content instance plus artifacts, and save their file data directly to somewhere within /var/lib/pulp/. It would be some path distinct from where Artifacts are stored. That's what I had imagined we would do anyway. Then as Dennis pointed out, turning that into an Artifact would only require a move operation on the same filesystem, which is super-cheap.<br> </div></div></blockquote><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div class="gmail_extra">Would that address all the concerns? We'd write the data just once, and then move it once on the same filesystem. I haven't looked at django's support for this recently, but it seems like it should be doable.<span class="m_-5778278824315326441m_-780941529446452707m_4609881861521469005m_-5142219127422626088m_-57592284970400827m_8458294198575060412m_-7313271489355059228gmail-HOEnZb"><font color="#888888"><br clear="all"><div><br></div></font></span></div></div></blockquote></span><div>I was just looking at the dropbox API and noticed that they provide two separate API endpoints for regular file uploads[0] (< 150mb) and large file uploads[1]. It is the latter that supports chunking and requires using an upload id. For the most common case they support uploading a file with one API call. Our original proposal requires 2 for the same use case. Pulp API users would appreciate having to only make one API call to upload a file. <br><br>[0] <a href="https://www.dropbox.com/developers-v1/core/docs#files_put" target="_blank">https://www.dropbox.com/develo<wbr>pers-v1/core/docs#files_put</a><br>[1] <a href="https://www.dropbox.com/developers-v1/core/docs#chunked-upload" target="_blank">https://www.dropbox.com/develo<wbr>pers-v1/core/docs#chunked-uplo<wbr>ad</a> <br></div><div><br> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><span><div dir="ltr"><div class="gmail_extra"><span class="m_-5778278824315326441m_-780941529446452707m_4609881861521469005m_-5142219127422626088m_-57592284970400827m_8458294198575060412m_-7313271489355059228gmail-HOEnZb"><font color="#888888"><div></div>-- <br><div class="m_-5778278824315326441m_-780941529446452707m_4609881861521469005m_-5142219127422626088m_-57592284970400827m_8458294198575060412m_-7313271489355059228gmail-m_4750842144497045598gmail_signature"><div dir="ltr"><p style="color:rgb(0,0,0);font-family:overpass-mono,monospace;font-size:10px;margin:0px;padding:0px"><span style="margin:0px;padding:0px">Michael</span> <span style="margin:0px;padding:0px">Hrivnak</span></p><p style="color:rgb(0,0,0);font-family:overpass-mono,monospace;font-size:10px;margin:0px;padding:0px"></p><span style="color:rgb(0,0,0);font-family:overpass-mono,monospace;font-size:10px;margin:0px;padding:0px"><span style="margin:0px;padding:0px">Principal Software Engineer</span><span style="margin:0px;padding:0px">, <span style="margin:0px;padding:0px">RHCE</span></span> </span><span style="color:rgb(0,0,0);font-family:overpass-mono,monospace;font-size:10px"></span><br style="color:rgb(0,0,0);font-family:overpass-mono,monospace;font-size:10px;margin:0px;padding:0px"><p style="color:rgb(0,0,0);font-family:overpass-mono,monospace;font-size:10px;margin:0px;padding:0px">Red Hat</p></div></div>
</font></span></div></div>
<br></span><span>______________________________<wbr>_________________<br>
Pulp-dev mailing list<br>
<a href="mailto:Pulp-dev@redhat.com" target="_blank">Pulp-dev@redhat.com</a><br>
<a href="https://www.redhat.com/mailman/listinfo/pulp-dev" rel="noreferrer" target="_blank">https://www.redhat.com/mailman<wbr>/listinfo/pulp-dev</a><br>
<br></span></blockquote></div><br></div></div>
<br>______________________________<wbr>_________________<br>
Pulp-dev mailing list<br>
<a href="mailto:Pulp-dev@redhat.com" target="_blank">Pulp-dev@redhat.com</a><br>
<a href="https://www.redhat.com/mailman/listinfo/pulp-dev" rel="noreferrer" target="_blank">https://www.redhat.com/mailman<wbr>/listinfo/pulp-dev</a><br>
<br></blockquote></div><br></div>
</div></div></blockquote></div></div></div><br></div></div>
</blockquote></div><br><br clear="all"><div><br></div>-- <br><div class="m_-5778278824315326441m_-780941529446452707m_4609881861521469005gmail_signature" data-smartmail="gmail_signature"><div dir="ltr"><p style="color:rgb(0,0,0);font-family:overpass-mono,monospace;font-size:10px;margin:0px!important;padding:0px!important"><span style="margin:0px!important;padding:0px!important">Michael</span> <span style="margin:0px!important;padding:0px!important">Hrivnak</span></p><p style="color:rgb(0,0,0);font-family:overpass-mono,monospace;font-size:10px;margin:0px!important;padding:0px!important"></p><span style="color:rgb(0,0,0);font-family:overpass-mono,monospace;font-size:10px;margin:0px!important;padding:0px!important"><span style="margin:0px!important;padding:0px!important">Principal Software Engineer</span><span style="margin:0px!important;padding:0px!important">, <span style="margin:0px!important;padding:0px!important">RHCE</span></span> </span><span style="color:rgb(0,0,0);font-family:overpass-mono,monospace;font-size:10px"></span><br style="color:rgb(0,0,0);font-family:overpass-mono,monospace;font-size:10px;margin:0px!important;padding:0px!important"><p style="color:rgb(0,0,0);font-family:overpass-mono,monospace;font-size:10px;margin:0px!important;padding:0px!important">Red Hat</p></div></div>
</div>
</div></div></blockquote></div></div></div><br></div></div>
</blockquote></div><br></div>