<div dir="ltr"><br><div class="gmail_extra"><br><div class="gmail_quote">On Wed, Jun 28, 2017 at 11:55 PM, Michael Hrivnak <span dir="ltr"><<a href="mailto:mhrivnak@redhat.com" target="_blank">mhrivnak@redhat.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">For a unit like a Distribution, the relative path of the file does matter with respect to other files associated with the same content unit and needs to be preserved. That content type consists of a collection of arbitrary files in an arbitrary directory structure that we have to just preserve. I suppose it could be a field on the many-to-many table between an artifact and its content. If it were a field on the artifact, and it were part of the uniqueness constraint together with the checksum, that could also work. But it would oppose the goal of deduplicating files.</div></blockquote><div><br></div><div>I definitely misspoke in my last email about the relative path belonging to the content unit. As you pointed it out it belongs with the ContentArtifact which represents the relationship between an artifact and a content unit. <br></div><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div><br></div><div>Speaking of, is that the goal of making artifacts and content a many-to-many relationship? Otherwise could someone re-summarize why that's being proposed?</div></div></blockquote><div><br><br></div><div>The many to many relationship is between Artifact and ContentArtifact. This allows a content unit to have multiple Artifacts associated with it. <br></div><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div class="HOEnZb"><div class="h5"><div class="gmail_extra"><br><div class="gmail_quote">On Wed, Jun 28, 2017 at 6:38 PM, Dennis Kliban <span dir="ltr"><<a href="mailto:dkliban@redhat.com" target="_blank">dkliban@redhat.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div class="gmail_extra"><div class="gmail_quote"><span>On Wed, Jun 28, 2017 at 1:10 PM, Jeff Ortel <span dir="ltr"><<a href="mailto:jortel@redhat.com" target="_blank">jortel@redhat.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><span class="m_5692025364579063713m_2191123044746210965gmail-"><br>
<br>
On 06/28/2017 11:44 AM, Brian Bouterse wrote:<br>
> For a file to be received and saved in the right place once, we need the view saving the file to have all the<br>
> info to form the complete path. After talking w/ @jortel, I think we should store Artifacts at the following path:<br>
><br>
> MEDIA_ROOT/content/units/diges<wbr>t[0:2]/digest[2:]/<rel_path><br>
<br>
</span>Consider:<br>
MEDIA_ROOT/artifact/digest[0:2<wbr>]/digest[2:]/<rel_path><br>
<br>
Since artifact would have an optional association with content. And, given the many-to-many relationship, the<br>
content_id FK would not longer exist in the Artifact table. Also, I have more plans for Artifacts in a<br>
"Publishing" proposal I'm writing to pulp-dev (spoiler alert).<br>
<br>
We would also want to enforce the same CAS (content addressed storage) uniqueness in the DB using a unique<br>
constraint on the Artifact. Eg: unique (sha256, rel_path). This ensure that each unique artifact (file) has<br>
exactly 1 DB record.<br>
<span class="m_5692025364579063713m_2191123044746210965gmail-"><br></span></blockquote><div><br></div></span><div>I don't think it makes sense for an Artifact to have a rel_path. It is just a file. A ContentUnit should have the rel_path that will be used at publish time to make the file backing the Artifact available at that rel_path. Is my understanding of the rel_path correct? In that case the only thing that should be unique is the sha256 digest. I've written a story that outlines this use case: <a href="https://pulp.plan.io/issues/2843" target="_blank">https://pulp.plan.io/issues/28<wbr>43</a><br></div><div><div class="m_5692025364579063713h5"><div><br><br><br> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><span class="m_5692025364579063713m_2191123044746210965gmail-">
><br>
> Note that digest is the Artifact's sha256 digest. This is different from pulp2 which used the digest of the<br>
> content unit. Note that <rel_path> would be provided by the user along with <size> and/or <checksum_digest>.<br>
><br>
> Note that this will cause an Artifact to live in exactly one place which means Artifacts are now unique by<br>
> digest and would need to be able to be associated with multiple content units. I'm not sure why we didn't do<br>
> this before, so I'm interested in exploring issues associated with this.<br>
><br>
> It would be a good workflow. For a single file content unit (e.g.) rpm upload would be a two step process.<br>
><br>
> 1. POST/PUT the file's binary data and the <relative_path> and <size> and/or <checksum_digest> as GET parameters<br>
> 2. Create a content unit with the unit metadata, and 0 .. n Artifacts referred to by ID. This could optionally<br>
> associate the new unit with one repository as part of the atomic unit creation.<br>
><br>
> Thoughts/Ideas?<br>
><br>
> -Brian<br>
><br>
><br>
</span>> On Tue, Jun 27, 2017 at 4:16 PM, Dennis Kliban <<a href="mailto:dkliban@redhat.com" target="_blank">dkliban@redhat.com</a> <mailto:<a href="mailto:dkliban@redhat.com" target="_blank">dkliban@redhat.com</a>>> wrote:<br>
<span class="m_5692025364579063713m_2191123044746210965gmail-">><br>
> On Tue, Jun 27, 2017 at 3:31 PM, Michael Hrivnak <<a href="mailto:mhrivnak@redhat.com" target="_blank">mhrivnak@redhat.com</a> <mailto:<a href="mailto:mhrivnak@redhat.com" target="_blank">mhrivnak@redhat.com</a>>> wrote:<br>
><br>
> Could you re-summarize what problem would be solved by not having a FileUpload model, and giving the<br>
> Artifact model the ability to have partial data and no Content foreign key?<br>
><br>
> I understand the concern about where on the filesystem the data gets written and how many times, but<br>
> I'm not seeing how that's related to whether we have a FileUpload model or not. Are we discussing two<br>
> separate issues? 1) filesystem locations and copy efficiency, and 2) API design? Or is this discussion<br>
> trying to connect them in a way I'm not seeing?<br>
><br>
><br>
> There were two concerns: 1) Filesystem location and copy efficiency 2) API design<br>
><br>
> The first one has been addressed. Thank you for pointing out that a second write will be a move operation.<br>
><br>
> However, I am still concerned about the complexity of the API. A relatively small file should not require<br>
> an upload session to be uploaded. A single API call to the Artifacts API should be enough to upload a file<br>
> and create an Artifact from it. In Pulp 3.1+ we can introduce the FileUpload model to support chunked<br>
> uploads. At the same time we would extend the Artifact API to accept a FileUpload id for creating an<br>
> Artifact.<br>
><br>
><br>
</span>> On Tue, Jun 27, 2017 at 3:20 PM, Dennis Kliban <<a href="mailto:dkliban@redhat.com" target="_blank">dkliban@redhat.com</a> <mailto:<a href="mailto:dkliban@redhat.com" target="_blank">dkliban@redhat.com</a>>> wrote:<br>
><br>
> On Tue, Jun 27, 2017 at 2:56 PM, Brian Bouterse <<a href="mailto:bbouters@redhat.com" target="_blank">bbouters@redhat.com</a> <mailto:<a href="mailto:bbouters@redhat.com" target="_blank">bbouters@redhat.com</a>>><br>
<span class="m_5692025364579063713m_2191123044746210965gmail-">> wrote:<br>
><br>
> Picking up from @jortel's observations...<br>
><br>
> +1 to allowing Artifacts to have an optional FK.<br>
><br>
> If we have an Artifacts endpoint then we can allow for the deleting of a single artifact if it<br>
> has no FK. I think we want to disallow the removal of an Artifact that has a foreign key. Also<br>
> filtering should allow a single operation to clean up all unassociated artifacts by searching<br>
> for FK=None or similar.<br>
><br>
> Yes, we will need to allow the single call delivering a file to also specify the relative<br>
> path, size, checksums etc. Since the POST body contains binary data we either need to accept<br>
> this data as GET style params or use a multi-part MIME upload [0]. Note that this creation of<br>
> an Artifact does not change the repository contents and therefore can be handled synchronously<br>
> outside of the tasking system.<br>
><br>
> +1 to the saving of an Artifact to perform validation<br>
><br>
> [0]: <a href="https://www.w3.org/Protocols/rfc1341/7_2_Multipart.html" rel="noreferrer" target="_blank">https://www.w3.org/Protocols/r<wbr>fc1341/7_2_Multipart.html</a><br>
> <<a href="https://www.w3.org/Protocols/rfc1341/7_2_Multipart.html" rel="noreferrer" target="_blank">https://www.w3.org/Protocols<wbr>/rfc1341/7_2_Multipart.html</a>><br>
><br>
><br>
><br>
> -Brian<br>
><br>
><br>
> I also support this optional FK for Artifacts and validation on save. We should probably stick<br>
> with accepting GET parameters for the MVP. Though multi-part MIME support would be good to<br>
> consider for 3.1+.<br>
><br>
><br>
><br>
> On Tue, Jun 27, 2017 at 2:44 PM, Dennis Kliban <<a href="mailto:dkliban@redhat.com" target="_blank">dkliban@redhat.com</a><br>
</span><span class="m_5692025364579063713m_2191123044746210965gmail-">> <mailto:<a href="mailto:dkliban@redhat.com" target="_blank">dkliban@redhat.com</a>>> wrote:<br>
><br>
> On Tue, Jun 27, 2017 at 1:24 PM, Michael Hrivnak <<a href="mailto:mhrivnak@redhat.com" target="_blank">mhrivnak@redhat.com</a><br>
</span><span class="m_5692025364579063713m_2191123044746210965gmail-">> <mailto:<a href="mailto:mhrivnak@redhat.com" target="_blank">mhrivnak@redhat.com</a>>> wrote:<br>
><br>
><br>
> On Tue, Jun 27, 2017 at 11:27 AM, Jeff Ortel <<a href="mailto:jortel@redhat.com" target="_blank">jortel@redhat.com</a><br>
</span><div><div class="m_5692025364579063713m_2191123044746210965gmail-h5">> <mailto:<a href="mailto:jortel@redhat.com" target="_blank">jortel@redhat.com</a>>> wrote:<br>
><br>
><br>
> - The artifact FK to a content unit would need to become optional.<br>
><br>
> - Need to add use cases for cleaning up artifacts not associated with a content unit.<br>
><br>
> - The upload API would need additional information needed to create an artifact.<br>
> Like relative path, size,<br>
> checksums etc.<br>
><br>
> - Since (I assume) you are proposing uploading/writing directly to artifact<br>
> storage (not staging in a working<br>
> dir), the flow would need to involve (optional) validation. If validation fails,<br>
> the artifact must not be<br>
> inserted into the DB.<br>
><br>
><br>
> Perhaps a decent middle ground would be to stick with the plan of keeping uploaded (or<br>
> partially uploaded) files as a separate model until they are ready to be turned into a<br>
> Content instance plus artifacts, and save their file data directly to somewhere within<br>
> /var/lib/pulp/. It would be some path distinct from where Artifacts are stored. That's<br>
> what I had imagined we would do anyway. Then as Dennis pointed out, turning that into<br>
> an Artifact would only require a move operation on the same filesystem, which is<br>
> super-cheap.<br>
><br>
><br>
> Would that address all the concerns? We'd write the data just once, and then move it<br>
> once on the same filesystem. I haven't looked at django's support for this recently,<br>
> but it seems like it should be doable.<br>
><br>
> I was just looking at the dropbox API and noticed that they provide two separate API<br>
> endpoints for regular file uploads[0] (< 150mb) and large file uploads[1]. It is the<br>
> latter that supports chunking and requires using an upload id. For the most common case<br>
> they support uploading a file with one API call. Our original proposal requires 2 for the<br>
> same use case. Pulp API users would appreciate having to only make one API call to upload<br>
> a file.<br>
><br>
> [0] <a href="https://www.dropbox.com/developers-v1/core/docs#files_put" rel="noreferrer" target="_blank">https://www.dropbox.com/develo<wbr>pers-v1/core/docs#files_put</a><br>
> <<a href="https://www.dropbox.com/developers-v1/core/docs#files_put" rel="noreferrer" target="_blank">https://www.dropbox.com/deve<wbr>lopers-v1/core/docs#files_put</a>><br>
> [1] <a href="https://www.dropbox.com/developers-v1/core/docs#chunked-upload" rel="noreferrer" target="_blank">https://www.dropbox.com/develo<wbr>pers-v1/core/docs#chunked-uplo<wbr>ad</a><br>
> <<a href="https://www.dropbox.com/developers-v1/core/docs#chunked-upload" rel="noreferrer" target="_blank">https://www.dropbox.com/deve<wbr>lopers-v1/core/docs#chunked-up<wbr>load</a>><br>
><br>
><br>
><br>
> --<br>
><br>
> Michael Hrivnak<br>
><br>
> Principal Software Engineer, RHCE<br>
><br>
> Red Hat<br>
><br>
><br>
> _____________________________<wbr>__________________<br>
> Pulp-dev mailing list<br>
</div></div>> <a href="mailto:Pulp-dev@redhat.com" target="_blank">Pulp-dev@redhat.com</a> <mailto:<a href="mailto:Pulp-dev@redhat.com" target="_blank">Pulp-dev@redhat.com</a>><br>
> <a href="https://www.redhat.com/mailman/listinfo/pulp-dev" rel="noreferrer" target="_blank">https://www.redhat.com/mailma<wbr>n/listinfo/pulp-dev</a><br>
<span class="m_5692025364579063713m_2191123044746210965gmail-">> <<a href="https://www.redhat.com/mailman/listinfo/pulp-dev" rel="noreferrer" target="_blank">https://www.redhat.com/mailm<wbr>an/listinfo/pulp-dev</a>><br>
><br>
><br>
><br>
> _____________________________<wbr>__________________<br>
> Pulp-dev mailing list<br>
</span>> <a href="mailto:Pulp-dev@redhat.com" target="_blank">Pulp-dev@redhat.com</a> <mailto:<a href="mailto:Pulp-dev@redhat.com" target="_blank">Pulp-dev@redhat.com</a>><br>
> <a href="https://www.redhat.com/mailman/listinfo/pulp-dev" rel="noreferrer" target="_blank">https://www.redhat.com/mailma<wbr>n/listinfo/pulp-dev</a><br>
<span class="m_5692025364579063713m_2191123044746210965gmail-im m_5692025364579063713m_2191123044746210965gmail-HOEnZb">> <<a href="https://www.redhat.com/mailman/listinfo/pulp-dev" rel="noreferrer" target="_blank">https://www.redhat.com/mailm<wbr>an/listinfo/pulp-dev</a>><br>
><br>
><br>
><br>
><br>
><br>
><br>
> --<br>
><br>
> Michael Hrivnak<br>
><br>
> Principal Software Engineer, RHCE<br>
><br>
> Red Hat<br>
><br>
><br>
><br>
><br>
><br>
</span><div class="m_5692025364579063713m_2191123044746210965gmail-HOEnZb"><div class="m_5692025364579063713m_2191123044746210965gmail-h5">> ______________________________<wbr>_________________<br>
> Pulp-dev mailing list<br>
> <a href="mailto:Pulp-dev@redhat.com" target="_blank">Pulp-dev@redhat.com</a><br>
> <a href="https://www.redhat.com/mailman/listinfo/pulp-dev" rel="noreferrer" target="_blank">https://www.redhat.com/mailman<wbr>/listinfo/pulp-dev</a><br>
><br>
<br>
</div></div><br>______________________________<wbr>_________________<br>
Pulp-dev mailing list<br>
<a href="mailto:Pulp-dev@redhat.com" target="_blank">Pulp-dev@redhat.com</a><br>
<a href="https://www.redhat.com/mailman/listinfo/pulp-dev" rel="noreferrer" target="_blank">https://www.redhat.com/mailman<wbr>/listinfo/pulp-dev</a><br>
<br></blockquote></div></div></div><br></div></div>
<br>______________________________<wbr>_________________<br>
Pulp-dev mailing list<br>
<a href="mailto:Pulp-dev@redhat.com" target="_blank">Pulp-dev@redhat.com</a><br>
<a href="https://www.redhat.com/mailman/listinfo/pulp-dev" rel="noreferrer" target="_blank">https://www.redhat.com/mailman<wbr>/listinfo/pulp-dev</a><br>
<br></blockquote></div><br><br clear="all"><div><br></div>-- <br><div class="m_5692025364579063713gmail_signature" data-smartmail="gmail_signature"><div dir="ltr"><p style="color:rgb(0,0,0);font-family:overpass-mono,monospace;font-size:10px;margin:0px!important;padding:0px!important"><span style="margin:0px!important;padding:0px!important">Michael</span> <span style="margin:0px!important;padding:0px!important">Hrivnak</span></p><p style="color:rgb(0,0,0);font-family:overpass-mono,monospace;font-size:10px;margin:0px!important;padding:0px!important"></p><span style="color:rgb(0,0,0);font-family:overpass-mono,monospace;font-size:10px;margin:0px!important;padding:0px!important"><span style="margin:0px!important;padding:0px!important">Principal Software Engineer</span><span style="margin:0px!important;padding:0px!important">, <span style="margin:0px!important;padding:0px!important">RHCE</span></span> </span><span style="color:rgb(0,0,0);font-family:overpass-mono,monospace;font-size:10px"></span><br style="color:rgb(0,0,0);font-family:overpass-mono,monospace;font-size:10px;margin:0px!important;padding:0px!important"><p style="color:rgb(0,0,0);font-family:overpass-mono,monospace;font-size:10px;margin:0px!important;padding:0px!important">Red Hat</p></div></div>
</div>
</div></div></blockquote></div><br></div></div>