<div dir="ltr">Thanks for raising this issue. The pulp_file also suffers from this problem in that files with duplicate names can be added to repo versions but they probably shouldn't be:<div><br></div><div><a href="https://pulp.plan.io/issues/4028">https://pulp.plan.io/issues/4028</a><br><div><br></div><div>@Simon I like the idea behind the repo_key solution you came up with. Can you be more specific around cases you think that it couldn't handle? I imagine that plugin writers could use properties or denormailzation (ie additional database columns) to solve cases where they need uniqueness across data that isn't in the database. In a worst case scenario, they can't use the pulpcore solution and just have to roll their own.</div><div><br><div><div dir="ltr" class="gmail_signature" data-smartmail="gmail_signature"><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div>David<br></div></div></div></div></div></div></div></div><br></div></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Fri, May 31, 2019 at 3:27 PM Simon Baatz <<a href="mailto:gmbnomis@gmail.com">gmbnomis@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">On Fri, May 31, 2019 at 01:12:58PM +0200, Tatiana Tereshchenko wrote:<br>
> A while ago RemoveDuplicates stage [0] was introduced to solve the<br>
> problem of enforcing uniqueness constraints within a repository version<br>
> at sync time.<br>
> The same problem ought to be solved when content which already exists<br>
> in Pulp is added to a repository. E.g. Content was uploaded, or content<br>
> was synced as a part of other repo. And now you want to add/copy it to<br>
> your repo.<br>
> RPM plugin has to solve this problem (specific examples can be seen in<br>
> this issue [1]).<br>
> It would be great if other plugins can share if the same problem exists<br>
> for them and if it's valuable to add some mechanism to the pulpcore.<br>
> I believe, if you use RemoveDuplicates stage during sync, then your<br>
> plugin is impacted by the described problem.<br>
<br>
Yes, the problem exists also for pulp_cookbook (although it does not<br>
use the RemoveDuplicates stage). Currently, the implementation to<br>
avoid duplicates in pulp_cookbook has the following components:<br>
<br>
- Content defines a 'repo_key' [0] similar to a unit_key. This key<br>
must be unique within a repo version (and not globally like the<br>
unit_key)<br>
<br>
- Cookbook metadata obtained during a sync does not contain<br>
digests. Therefore pulp_cookbook uses a custom stage<br>
QueryExistingRepoContentAndArtifacts [1] to identify existing<br>
content within the repo version the sync is based on. Content is<br>
queried using the repo key in the base repo version (and duplicates<br>
need not to be removed after the fact).<br>
<br>
(However, something like repo_key might be useful in the<br>
RemoveDuplicates stage for other plugins.)<br>
<br>
- As I found no way to ensure repo_key uniqueness on content<br>
association, it is done at publication time [2] based on the repo_key. <br>
However, this feels like a workaround. I think it should be<br>
enforced on repo version creation.<br>
<br>
> My personal opinion: if RemoveDuplicates stage was worth adding to the<br>
> pulpcore (stages API in pulpcore-plugin), a mechanism to ensure<br>
> uniqueness constraints within a repo version at association time makes<br>
> sense to add as well.<br>
<br>
I fully agree. I don't think the repo_key approach used by<br>
pulp_cookbook is general enough. It works well with Cookbooks, but<br>
other content types might have uniqueness constraints that<br>
can't be mapped directly to a composite key on repo versions.<br>
<br>
<br>
[0] <a href="https://github.com/gmbnomis/pulp_cookbook/blob/573e1813bd33c0d09d44cf2cab8634f0e4d10fd4/pulp_cookbook/app/models.py#L70" rel="noreferrer" target="_blank">https://github.com/gmbnomis/pulp_cookbook/blob/573e1813bd33c0d09d44cf2cab8634f0e4d10fd4/pulp_cookbook/app/models.py#L70</a><br>
[1] <a href="https://github.com/gmbnomis/pulp_cookbook/blob/573e1813bd33c0d09d44cf2cab8634f0e4d10fd4/pulp_cookbook/app/tasks/synchronizing.py#L61" rel="noreferrer" target="_blank">https://github.com/gmbnomis/pulp_cookbook/blob/573e1813bd33c0d09d44cf2cab8634f0e4d10fd4/pulp_cookbook/app/tasks/synchronizing.py#L61</a><br>
[2] <a href="https://github.com/gmbnomis/pulp_cookbook/blob/573e1813bd33c0d09d44cf2cab8634f0e4d10fd4/pulp_cookbook/app/tasks/publishing.py#L63" rel="noreferrer" target="_blank">https://github.com/gmbnomis/pulp_cookbook/blob/573e1813bd33c0d09d44cf2cab8634f0e4d10fd4/pulp_cookbook/app/tasks/publishing.py#L63</a><br>
<br>
_______________________________________________<br>
Pulp-dev mailing list<br>
<a href="mailto:Pulp-dev@redhat.com" target="_blank">Pulp-dev@redhat.com</a><br>
<a href="https://www.redhat.com/mailman/listinfo/pulp-dev" rel="noreferrer" target="_blank">https://www.redhat.com/mailman/listinfo/pulp-dev</a><br>
</blockquote></div>