<div dir="ltr"><div>Tanya,</div><div>Thank you for summarizing the proposal. I left comments and questions directly on the story.<br></div><div><div><div dir="ltr" class="gmail_signature" data-smartmail="gmail_signature"><div dir="ltr"><div><div dir="ltr"><br><br>--------<br>Regards,<br><br>Ina Panova<br>Senior Software Engineer| Pulp| Red Hat Inc.<br><br>"Do not go where the path may lead,<br> go instead where there is no path and leave a trail."<br></div></div></div></div></div><br></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Mon, Oct 26, 2020 at 7:00 PM Tatiana Tereshchenko <<a href="mailto:ttereshc@redhat.com">ttereshc@redhat.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">Updated the redmine story <a href="https://pulp.plan.io/issues/6353" target="_blank">https://pulp.plan.io/issues/6353</a> with this proposal. Feel free to comment there as well.<div>If there are no objections, I'll start on a PoC once I finish the items I'm working on right now.</div><div><br></div><div>Thanks, </div><div>Tanya</div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Wed, Oct 21, 2020 at 8:48 PM Tatiana Tereshchenko <<a href="mailto:ttereshc@redhat.com" target="_blank">ttereshc@redhat.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">TL;DR: An attempt to propose the least invasive option to solve the case when remote repository metadata needs to be mirrored. Please provide feedback if you are interested in the outcome.<div><br></div><div>There have been multiple attempts and discussions to solve the relative_path problem in a general way which covers multiple use cases. </div><div>They all look very invasive and only possible to be done in Pulp 4+ due to the amount and significance of changes that needs to be made, to the data models and/or to the API. </div><div><br></div><div>The following proposal solves only this use case: As a user, I can mirror remote repository metadata as is. </div><div>An additional goal is to avoid backward incompatible changes and ideally leave a way for further improvement to solve the problem in a more general way.</div><div>(The following proposal does NOT solve a use case: As a user, I can have the same content under different relative paths in any repository.)</div><div><br></div><div>Proposal:</div><div>- Have a way to distinguish between repositories with managed content and with the exact mirror (e.g. create a repository with exact_mirror=True or a new dedicated repository type)</div><div> - For such repos, create a publication at sync time (includes published artifacts and metadata).</div><div> - For such repos, publish is no-op and always returns the existing publication for the requested repo version.</div><div> - For such repos, no modifications are allowed except the sync in mirror mode. (At least for RPM plugin, I believe we can't allow discrepancies between metadata and content in a repo, especially if some content is removed.)</div><div><br></div><div>Pros:</div><div> - non-invasive, only additive model changes</div><div> - can be implemented in a plugin which needs it or it can be moved to the pulpcore if it allows plugin input at certain points.</div><div> - leaves a way for further improvement to handle a more general case, see the full proposal here <a href="https://hackmd.io/02KBjCD3Q0WP7p4ALwzhJw#relative_path-in-PublishedArtifact-only" target="_blank">https://hackmd.io/02KBjCD3Q0WP7p4ALwzhJw#relative_path-in-PublishedArtifact-only</a></div><div><br></div><div>Cons:</div><div> - doesn't solve the problem of various relative paths for the same content in general way</div><div> - a separate code path (at times) to handle this type of repositories.</div><div><br></div><div>For reference:</div><div> - hackmd doc with all the considered proposals and the summary of the potentially valid ones <a href="https://hackmd.io/02KBjCD3Q0WP7p4ALwzhJw" target="_blank">https://hackmd.io/02KBjCD3Q0WP7p4ALwzhJw</a></div><div> - pulpcon video with the discussion of the proposals <a href="https://www.youtube.com/watch?v=7IzxAQYr5-I" target="_blank">https://www.youtube.com/watch?v=7IzxAQYr5-I</a></div><div><br></div><div>Thanks,</div><div>Tanya</div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Thu, May 7, 2020 at 2:07 PM Brian Bouterse <<a href="mailto:bmbouter@redhat.com" target="_blank">bmbouter@redhat.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div>I agree with that problem statement. pulp_file may want to have the same Content at two different paths in different RepositoryVersions (or even the same RepositoryVersion). Without this capability a user could never "move" where content lives in a RepositoryVersion if its already been placed in any other RepositoryVersion.</div><div><br></div><div>Additionally pulp_maven may need to sync two repositories in the wild that already contain the same content in two locations. I offer this as example not to pile-on, but because it's a multi-content artifact which I believe we will need to consider also as we work towards a solution.</div><div><br></div><div>I've been spending time on developing a solution, but it needs more work so it's not ready yet. Also other katello and galaxy_ng work continues to pre-empt this, so it could take a while.<br></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Thu, May 7, 2020 at 3:39 AM Matthias Dellweg <<a href="mailto:mdellweg@redhat.com" target="_blank">mdellweg@redhat.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div>> Users need to be able to store the same content unit at different relative paths in different repository versions. This problem is not unique to the RPM plugin. Do we agree about that?</div><div>Yes, we agree. In pulp_deb relative_path is part of the contents natural_key to circumvent this problem. So this creates two content units that only differ in relativ_path. At least they share the artifact.<br></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Thu, May 7, 2020 at 2:06 AM Dennis Kliban <<a href="mailto:dkliban@redhat.com" target="_blank">dkliban@redhat.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div>I'd like to provide a little bit more context for my previous email by going back to the original problem statement:</div><div><br></div><div><div dir="ltr" class="gmail_attr">On Wed, Apr 1, 2020 at 9:23 AM Daniel Alley <<a href="mailto:dalley@redhat.com" target="_blank">dalley@redhat.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><h3 id="gmail-m_1367049509803992740gmail-m_2595405513893244917gmail-m_-8032171694291597312gmail-m_-7667069988061076738gmail-m_3047469861454645021m_-138247769918178766gmail-m_-5310193286646399830gmail-Problem">Problem:</h3><p>Currently, a relative_path is tied to content in Pulp. This means that if a content unit exists in two places within a repository or across repositories, it has to be stored as two separate content units. This creates redundant data and potential confusion for users.</p><p>As a specific example, we need <a href="https://pulp.plan.io/issues/6353" rel="noopener" target="_blank">to support mirroring content in pulp_rpm</a>. Currently, for each location at which a single package is stored, we’ll need to create a content unit. We could end up with several records representing a single package. Users may be confused about why they see multiple records for a package and they may have trouble for example deciding which content unit to copy.</p></div></blockquote><div>Users need to be able to store the same content unit at different relative paths in different repository versions. This problem is not unique to the RPM plugin. Do we agree about that?</div><div><br></div><div>I've been working on a potential solution that solves this problem in a document[0]. It is a complicated change and the document does not fully capture the plan yet. Feedback and help on the design is welcome. <br></div><div><br></div><div>[0] <a href="https://hackmd.io/02KBjCD3Q0WP7p4ALwzhJw?edit" target="_blank">https://hackmd.io/02KBjCD3Q0WP7p4ALwzhJw?edit</a></div><div> </div></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Mon, May 4, 2020 at 4:11 PM Dennis Kliban <<a href="mailto:dkliban@redhat.com" target="_blank">dkliban@redhat.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div>I've reached two conclusions while trying to formulate a solution:</div><div><br></div><div>This problem needs to be solved at the repository version level. Repository membership needs to be tracked at the artifact level, and not content level as it is now. <br></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Thu, Apr 30, 2020 at 1:11 PM Daniel Alley <<a href="mailto:dalley@redhat.com" target="_blank">dalley@redhat.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div>Cool, so the only difference is whether to try to store the relationship in the DB, or leverage the fact that we already have the metadata and just re-parse it.</div><div><br></div><div>I know the latter approach has yet to be written up, but my concern there is that adding another layer of indirection between "repository version" and "content" is going to have an adverse impact on performance, since it is already the most complex and demanding query we issue to the DB and one of the most common and important.<br></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Thu, Apr 30, 2020 at 12:50 PM David Davis <<a href="mailto:daviddavis@redhat.com" target="_blank">daviddavis@redhat.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">Yes but I was imagining the mapping would be stored not as Content but as a separate object. So we wouldn't use filename for the mapping (rather we'd use ContentArtifact pk) and we wouldn't need to change ContentArtifact's relative_path at all. That said, I think your solution captures the idea though and is better in some ways.<div><br></div><div>Changing the RepositoryContent model to point to ContentArtifacts and store relative_paths is probably the best and most correct solution in theory. However, it's going to be painful to implement for both core and plugins.</div><div><br></div><div><div><div dir="ltr"><div dir="ltr"><div><div dir="ltr"><div dir="ltr"><div dir="ltr"><div>David</div></div></div></div></div></div></div></div><br></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Thu, Apr 30, 2020 at 12:33 PM Daniel Alley <<a href="mailto:dalley@redhat.com" target="_blank">dalley@redhat.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div><a class="gmail_plusreply" id="gmail-m_1367049509803992740gmail-m_2595405513893244917gmail-m_-8032171694291597312gmail-m_-7667069988061076738gmail-m_3047469861454645021gmail-m_-138247769918178766gmail-m_6731388314282964846gmail-m_-4718932104004480805gmail-m_5600417745597894402gmail-m_-2993930236119733805plusReplyChip-0" href="mailto:daviddavis@redhat.com" target="_blank">@David Davis</a> so this proposal would go something like this, correct?:<br></div><br><div>* For the signed metadata / exact mirror use-case we need to store the repository metadata itself as a content unit inside the RepositoryVersion anyway (because the hash must be equal)</div><div>* Because we have this metadata lying around, we can reference it at publish time to discover the appropriate PublishedArtifact.relative_path</div><div> * Create a map of "filename" -> "location_href" and look up the filename of each RPM package to find the appropriate path<br></div><div> * This should be pretty fast for the RPM plugin since createrepo_c is doing all the hard work</div><div>* Data migration to ensure ContentArtifact.relative_path is only storing the filename (and I would suggest we also change the name to "filename")</div><div>* If metadata isn't present in the RepositoryVersion, then just tweak the PublishedArtifact.relative_path so that it uses whichever our default repo layout is<br></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Tue, Apr 28, 2020 at 11:41 AM David Davis <<a href="mailto:daviddavis@redhat.com" target="_blank">daviddavis@redhat.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">Yes, that's correct. During our meeting we discussed two options: the first was to extend RepositoryContent to store relative path per ContentArtifact as storing a relative_path per Content won't work for multi-Artifact Content units.<div><br></div><div>An alternative that I pitched was to have plugins (or maybe even core someday) store this information outside RepositoryContent and then use this information during publishing to set relative_path on PublishedArtifacts. We'd have to modify the content app if we wanted to support pass through publications but I think asking plugins to use published artifacts in this case is warranted. That said, I don't think anyone else was keen on this idea though.<br clear="all"><div><div dir="ltr"><div dir="ltr"><div><div dir="ltr"><div dir="ltr"><div dir="ltr"><div><br></div><div>David</div></div></div></div></div></div></div></div><br></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Tue, Apr 28, 2020 at 10:30 AM Matthias Dellweg <<a href="mailto:mdellweg@redhat.com" target="_blank">mdellweg@redhat.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">That is only used for passthrough publication afaik. If you publish each content unit "by hand", you create a new relative path for each published artifact. That is, why it can be empty and still the content can be published.<br></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Tue, Apr 28, 2020 at 4:09 PM Daniel Alley <<a href="mailto:dalley@redhat.com" target="_blank">dalley@redhat.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div>We realized in our discussion that the original proposal described in my email will not work, because "relative_path" ultimately describes the path of the published <i>artifacts</i> (not content), and for content types with multiple artifacts, storing this information in a field on RepositoryContent would not be possible.</div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Mon, Apr 27, 2020 at 6:08 PM Daniel Alley <<a href="mailto:dalley@redhat.com" target="_blank">dalley@redhat.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div>There is a video call scheduled to discuss this issue tomorrow (Tuesday April 28th) at 13:30 UTC (please convert to your local time). <a href="https://meet.google.com/scy-csbx-qiu" target="_blank">https://meet.google.com/scy-csbx-qiu</a></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Sat, Apr 25, 2020 at 7:02 AM David Davis <<a href="mailto:daviddavis@redhat.com" target="_blank">daviddavis@redhat.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div><div><div dir="ltr"><div dir="ltr"><div><div dir="ltr"><div dir="ltr"><div dir="ltr"><div>I had a chance to think about this some more yesterday and wanted to email out my thoughts. I also think that this change sounds scary and will have a big impact on plugin writers so I thought of a couple alternatives:</div><div><br></div><div>First, we could add a relative_path field to RepositoryContent instead of moving it there. This would be an optional field. It would be up to plugins to manage this field and they would still need to populate the relative_path field on ContentArtifact. But plugins could use this optional field to store relative paths per repository and then use this field when generating publications.</div><div><br></div><div>The second alternative is one that is already laid out in the original email but to call it out again: it would be to not solve this in pulpcore. RPM would create its own object that would map content in a repository to relative_paths.</div><div><br></div><div>David</div></div></div></div></div></div></div></div><br></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Tue, Apr 21, 2020 at 9:22 AM Quirin Pamp <<a href="mailto:pamp@atix.de" target="_blank">pamp@atix.de</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"> <div dir="ltr"> <div id="gmail-m_1367049509803992740gmail-m_2595405513893244917gmail-m_-8032171694291597312gmail-m_-7667069988061076738gmail-m_3047469861454645021gmail-m_-138247769918178766gmail-m_6731388314282964846gmail-m_-4718932104004480805gmail-m_5600417745597894402gmail-m_-2993930236119733805gmail-m_-3345932667523395000gmail-m_-6871900733992814578gmail-m_5823751309584409495gmail-m_7710659920473041485gmail-m_202872833858503243gmail-m_-6499401100905295274gmail-m_4543847866100918455gmail-m_-6286089361145814873gmail-m_4329576662106291799gmail-m_7275620558784531060divtagdefaultwrapper" style="font-size:12pt;color:rgb(0,0,0);font-family:Calibri,Helvetica,sans-serif" dir="ltr"> <p style="margin-top:0px;margin-bottom:0px">Hi,</p> <p style="margin-top:0px;margin-bottom:0px"><br> </p> <p style="margin-top:0px;margin-bottom:0px">I am not currently very well versed in the classes involved, but moving relative_path around sounds slightly scary with the potential to break things.</p> <p style="margin-top:0px;margin-bottom:0px"><br> </p> <p style="margin-top:0px;margin-bottom:0px">As such, I would be interested to be kept in the loop as this moves forward. (Mailing list once there is some movement is entirely sufficient <span>😉</span>)</p> <p style="margin-top:0px;margin-bottom:0px"><br> </p> <p style="margin-top:0px;margin-bottom:0px">Thanks,</p> <p style="margin-top:0px;margin-bottom:0px">Quirin Pamp<br> </p> </div> <hr style="display:inline-block;width:98%"> <div id="gmail-m_1367049509803992740gmail-m_2595405513893244917gmail-m_-8032171694291597312gmail-m_-7667069988061076738gmail-m_3047469861454645021gmail-m_-138247769918178766gmail-m_6731388314282964846gmail-m_-4718932104004480805gmail-m_5600417745597894402gmail-m_-2993930236119733805gmail-m_-3345932667523395000gmail-m_-6871900733992814578gmail-m_5823751309584409495gmail-m_7710659920473041485gmail-m_202872833858503243gmail-m_-6499401100905295274gmail-m_4543847866100918455gmail-m_-6286089361145814873gmail-m_4329576662106291799gmail-m_7275620558784531060divRplyFwdMsg" dir="ltr"><font style="font-size:11pt" face="Calibri, sans-serif" color="#000000"><b>From:</b> <a href="mailto:pulp-dev-bounces@redhat.com" target="_blank">pulp-dev-bounces@redhat.com</a> <<a href="mailto:pulp-dev-bounces@redhat.com" target="_blank">pulp-dev-bounces@redhat.com</a>> on behalf of Ina Panova <<a href="mailto:ipanova@redhat.com" target="_blank">ipanova@redhat.com</a>><br> <b>Sent:</b> 21 April 2020 14:07:13<br> <b>To:</b> Daniel Alley <<a href="mailto:dalley@redhat.com" target="_blank">dalley@redhat.com</a>><br> <b>Cc:</b> Pulp-dev <<a href="mailto:pulp-dev@redhat.com" target="_blank">pulp-dev@redhat.com</a>><br> <b>Subject:</b> Re: [Pulp-dev] the "relative path" problem</font> <div> </div> </div> <div> <div dir="ltr"> <div>Daniel,</div> <div><br> </div> <div>how about setting up a meeting and brainstorm the alternatives, pros/cons there?<br> </div> <div> <div> <div dir="ltr"> <div dir="ltr"> <div> <div dir="ltr"><br> <br> --------<br> Regards,<br> <br> Ina Panova<br> Senior Software Engineer| Pulp| Red Hat Inc.<br> <br> "Do not go where the path may lead,<br> go instead where there is no path and leave a trail."<br> </div> </div> </div> </div> </div> <br> </div> </div> <br> <div> <div dir="ltr">On Fri, Apr 17, 2020 at 5:57 PM Daniel Alley <<a href="mailto:dalley@redhat.com" target="_blank">dalley@redhat.com</a>> wrote:<br> </div> <blockquote style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"> <div dir="ltr">Bump, this item needs to move forwards soon. Does anyone have any thoughts?<br> </div> <br> <div> <div dir="ltr">On Wed, Apr 1, 2020 at 9:40 AM Pavel Picka <<a href="mailto:ppicka@redhat.com" target="_blank">ppicka@redhat.com</a>> wrote:<br> </div> <blockquote style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"> <div dir="ltr">Hi, <div>I'd like to add one more question to this topic. Do you think it is a blocker for PRs [0] & [1] as by testing [2] this features I haven't run into real world example where two really same name packages appears. </div> <div>I think this is a 'must have' feature but until we solve/decide it we can have two features working may with warning in docs for users that can happen in some 'special' repositories. </div> <div><br> </div> <div>To follow topic directly I like proposed move to 'RepositoryContent' and add it to its uniqueness constraint (if I understand well).</div> <div><br> </div> <div>[0] <a href="https://github.com/pulp/pulp_rpm/pull/1657" target="_blank">https://github.com/pulp/pulp_rpm/pull/1657</a></div> <div>[1] <a href="https://github.com/pulp/pulp_rpm/pull/1642" target="_blank">https://github.com/pulp/pulp_rpm/pull/1642</a></div> <div>[2] tested with centos 7, 8, opensuse and SLE repositories</div> </div> <br> <div> <div dir="ltr">On Wed, Apr 1, 2020 at 3:22 PM Daniel Alley <<a href="mailto:dalley@redhat.com" target="_blank">dalley@redhat.com</a>> wrote:<br> </div> <blockquote style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"> <div dir="ltr"> <h3 id="gmail-m_1367049509803992740gmail-m_2595405513893244917gmail-m_-8032171694291597312gmail-m_-7667069988061076738gmail-m_3047469861454645021gmail-m_-138247769918178766gmail-m_6731388314282964846gmail-m_-4718932104004480805gmail-m_5600417745597894402gmail-m_-2993930236119733805gmail-m_-3345932667523395000gmail-m_-6871900733992814578gmail-m_5823751309584409495gmail-m_7710659920473041485gmail-m_202872833858503243gmail-m_-6499401100905295274gmail-m_4543847866100918455gmail-m_-6286089361145814873gmail-m_4329576662106291799gmail-m_7275620558784531060x_gmail-m_-8146024664432765745gmail-m_-8995103163427396541gmail-m_870933347578052004gmail-Problem"> <font size="2"><span style="font-weight:normal">We'd like to start a discussion on the "relative path problem" identified recently.</span></font><br> </h3> <h3 id="gmail-m_1367049509803992740gmail-m_2595405513893244917gmail-m_-8032171694291597312gmail-m_-7667069988061076738gmail-m_3047469861454645021gmail-m_-138247769918178766gmail-m_6731388314282964846gmail-m_-4718932104004480805gmail-m_5600417745597894402gmail-m_-2993930236119733805gmail-m_-3345932667523395000gmail-m_-6871900733992814578gmail-m_5823751309584409495gmail-m_7710659920473041485gmail-m_202872833858503243gmail-m_-6499401100905295274gmail-m_4543847866100918455gmail-m_-6286089361145814873gmail-m_4329576662106291799gmail-m_7275620558784531060x_gmail-m_-8146024664432765745gmail-m_-8995103163427396541gmail-m_870933347578052004gmail-Problem"> Problem:</h3> <p>Currently, a relative_path is tied to content in Pulp. This means that if a content unit exists in two places within a repository or across repositories, it has to be stored as two separate content units. This creates redundant data and potential confusion for users.</p> <p>As a specific example, we need <a href="https://pulp.plan.io/issues/6353" rel="noopener" target="_blank"> to support mirroring content in pulp_rpm</a>. Currently, for each location at which a single package is stored, we’ll need to create a content unit. We could end up with several records representing a single package. Users may be confused about why they see multiple records for a package and they may have trouble for example deciding which content unit to copy.</p> <h3 id="gmail-m_1367049509803992740gmail-m_2595405513893244917gmail-m_-8032171694291597312gmail-m_-7667069988061076738gmail-m_3047469861454645021gmail-m_-138247769918178766gmail-m_6731388314282964846gmail-m_-4718932104004480805gmail-m_5600417745597894402gmail-m_-2993930236119733805gmail-m_-3345932667523395000gmail-m_-6871900733992814578gmail-m_5823751309584409495gmail-m_7710659920473041485gmail-m_202872833858503243gmail-m_-6499401100905295274gmail-m_4543847866100918455gmail-m_-6286089361145814873gmail-m_4329576662106291799gmail-m_7275620558784531060x_gmail-m_-8146024664432765745gmail-m_-8995103163427396541gmail-m_870933347578052004gmail-Proposed-Solution"> Proposed Solution:</h3> <p>Move “relative_path” from its current location on ContentArtifact, to RepositoryContent. This will require a sizable data migration. It is possibly the case that in rare cases, repository versions may change slightly due to deduplication.</p> <p>A repository-version-wide uniqueness constraint will be present on “relative_path”, independently of any other repository uniquness constraints (repo_key_fields) defined by the plugin writer.</p> <p>Modify the Stages API so that the relative_path can be processed in the correct location – instead of “DeclarativeArtifact” it will likely need to go on “DeclarativeContent”</p> <p>Remove “location_href” from the RPM Package content model – it was never a true part of the RPM (file) metadata, it is derived from the repository metadata. So storing it as a part of the Content unit doesn’t entirely make sense.</p> <h3 id="gmail-m_1367049509803992740gmail-m_2595405513893244917gmail-m_-8032171694291597312gmail-m_-7667069988061076738gmail-m_3047469861454645021gmail-m_-138247769918178766gmail-m_6731388314282964846gmail-m_-4718932104004480805gmail-m_5600417745597894402gmail-m_-2993930236119733805gmail-m_-3345932667523395000gmail-m_-6871900733992814578gmail-m_5823751309584409495gmail-m_7710659920473041485gmail-m_202872833858503243gmail-m_-6499401100905295274gmail-m_4543847866100918455gmail-m_-6286089361145814873gmail-m_4329576662106291799gmail-m_7275620558784531060x_gmail-m_-8146024664432765745gmail-m_-8995103163427396541gmail-m_870933347578052004gmail-Alternatives"> Alternatives</h3> <p>In most cases, a content unit will have a single relative path for a content unit. Creating a general solution to solve a one-off problem is usually not a good idea. As an alternative, we could look at another solution for mirroring content. One example might be to create a new object (e.g. RpmRepoMirrorContentMapping) that maps content to specific paths within a repo or repo version.</p> <h3 id="gmail-m_1367049509803992740gmail-m_2595405513893244917gmail-m_-8032171694291597312gmail-m_-7667069988061076738gmail-m_3047469861454645021gmail-m_-138247769918178766gmail-m_6731388314282964846gmail-m_-4718932104004480805gmail-m_5600417745597894402gmail-m_-2993930236119733805gmail-m_-3345932667523395000gmail-m_-6871900733992814578gmail-m_5823751309584409495gmail-m_7710659920473041485gmail-m_202872833858503243gmail-m_-6499401100905295274gmail-m_4543847866100918455gmail-m_-6286089361145814873gmail-m_4329576662106291799gmail-m_7275620558784531060x_gmail-m_-8146024664432765745gmail-m_-8995103163427396541gmail-m_870933347578052004gmail-Questions"> Questions</h3> <ul> <li>How do we handle this in pulp_file? How are content units identified in pulp_file without relative_path?</li><ul> <li>Checksum?<br> </li></ul> <li>How was this problem handled in Pulp 2?</li></ul> <div><br> </div> <div>Please weigh in if you have any input on potential problems with the proposal, potential alternate solutions, or other insights or questions!<br> </div> </div> _______________________________________________<br> Pulp-dev mailing list<br> <a href="mailto:Pulp-dev@redhat.com" target="_blank">Pulp-dev@redhat.com</a><br> <a href="https://www.redhat.com/mailman/listinfo/pulp-dev" rel="noreferrer" target="_blank">https://www.redhat.com/mailman/listinfo/pulp-dev</a><br> </blockquote> </div> <br clear="all"> <div><br> </div> -- <br> <div dir="ltr"> <div dir="ltr"> <div>Pavel Picka</div> <div>Red Hat<br> </div> </div> </div> </blockquote> </div> _______________________________________________<br> Pulp-dev mailing list<br> <a href="mailto:Pulp-dev@redhat.com" target="_blank">Pulp-dev@redhat.com</a><br> <a href="https://www.redhat.com/mailman/listinfo/pulp-dev" rel="noreferrer" target="_blank">https://www.redhat.com/mailman/listinfo/pulp-dev</a><br> </blockquote> </div> </div> </div> _______________________________________________<br> Pulp-dev mailing list<br> <a href="mailto:Pulp-dev@redhat.com" target="_blank">Pulp-dev@redhat.com</a><br> <a href="https://www.redhat.com/mailman/listinfo/pulp-dev" rel="noreferrer" target="_blank">https://www.redhat.com/mailman/listinfo/pulp-dev</a><br> </blockquote></div> </blockquote></div> </blockquote></div> _______________________________________________<br> Pulp-dev mailing list<br> <a href="mailto:Pulp-dev@redhat.com" target="_blank">Pulp-dev@redhat.com</a><br> <a href="https://www.redhat.com/mailman/listinfo/pulp-dev" rel="noreferrer" target="_blank">https://www.redhat.com/mailman/listinfo/pulp-dev</a><br> </blockquote></div> </blockquote></div> </blockquote></div> </blockquote></div> </blockquote></div> _______________________________________________<br> Pulp-dev mailing list<br> <a href="mailto:Pulp-dev@redhat.com" target="_blank">Pulp-dev@redhat.com</a><br> <a href="https://www.redhat.com/mailman/listinfo/pulp-dev" rel="noreferrer" target="_blank">https://www.redhat.com/mailman/listinfo/pulp-dev</a><br> </blockquote></div> </blockquote></div> _______________________________________________<br> Pulp-dev mailing list<br> <a href="mailto:Pulp-dev@redhat.com" target="_blank">Pulp-dev@redhat.com</a><br> <a href="https://www.redhat.com/mailman/listinfo/pulp-dev" rel="noreferrer" target="_blank">https://www.redhat.com/mailman/listinfo/pulp-dev</a><br> </blockquote></div> _______________________________________________<br> Pulp-dev mailing list<br> <a href="mailto:Pulp-dev@redhat.com" target="_blank">Pulp-dev@redhat.com</a><br> <a href="https://www.redhat.com/mailman/listinfo/pulp-dev" rel="noreferrer" target="_blank">https://www.redhat.com/mailman/listinfo/pulp-dev</a><br> </blockquote></div> _______________________________________________<br> Pulp-dev mailing list<br> <a href="mailto:Pulp-dev@redhat.com" target="_blank">Pulp-dev@redhat.com</a><br> <a href="https://www.redhat.com/mailman/listinfo/pulp-dev" rel="noreferrer" target="_blank">https://www.redhat.com/mailman/listinfo/pulp-dev</a><br> </blockquote></div> </blockquote></div> _______________________________________________<br> Pulp-dev mailing list<br> <a href="mailto:Pulp-dev@redhat.com" target="_blank">Pulp-dev@redhat.com</a><br> <a href="https://www.redhat.com/mailman/listinfo/pulp-dev" rel="noreferrer" target="_blank">https://www.redhat.com/mailman/listinfo/pulp-dev</a><br> </blockquote></div>