<div dir="ltr"><div>I discussed this a little bit on the #<a href="http://rpm.org" target="_blank">rpm.org</a> channel. Here is the gist of that discussion</div><div></div><div><ul><li>The metadata is "crazy, but technically valid"</li><li>"the entire SUSE ecosystem tends to do this a lot, anything using OBS, including nvidia and dell and friends"</li><li>"also, SUSE packages can have the same NEVRA with being completely different packages because of how their build system makes packages"</li></ul><div>I'm not sure what the best means to fix it would be. Perhaps the uniqueness constraint should be on the location_href, instead of on the NEVRA? Or on NEVRA + location_href?<br></div></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Wed, Mar 18, 2020 at 9:47 AM Ina Panova <<a href="mailto:ipanova@redhat.com" target="_blank">ipanova@redhat.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div>Pavel, <br></div><div>I meant to say, that pulp3 does not have such limitation as pulp2 had ( saving rpms on the filesystem with same nevra).</div><div>The error is raised in pulp3 [0] when a repo version is created, because of the repo key[1], we cannot have 2 rpms with save NEVRA.</div><div><br></div><div>We can enable that, if we decide to, by adding location_href to the repo_key, *but* this needs to be evaluated, it can have side effects and we should involve our stakeholders to weigh in.</div><div><br></div><div dir="ltr">[0] <a href="https://github.com/pulp/pulpcore/blob/master/pulpcore/app/models/repository.py#L570" target="_blank">https://github.com/pulp/pulpcore/blob/master/pulpcore/app/models/repository.py#L570</a></div><div dir="ltr">[1] <a href="https://github.com/pulp/pulp_rpm/blob/master/pulp_rpm/app/models/package.py#L188" target="_blank">https://github.com/pulp/pulp_rpm/blob/master/pulp_rpm/app/models/package.py#L188</a></div><div dir="ltr"><br></div><div dir="ltr">--------<br>Regards,<br><br>Ina Panova<br>Senior Software Engineer| Pulp| Red Hat Inc.<br><br>"Do not go where the path may lead,<br> go instead where there is no path and leave a trail."<br></div><div><br></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Wed, Mar 18, 2020 at 2:24 PM Pavel Picka <<a href="mailto:ppicka@redhat.com" target="_blank">ppicka@redhat.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">True in opensuse repository there are two possibilities 'src' and 'nosrc' (this one should be legacy without source code), both are recognized by createrepo_c as arch 'src'. <div><br></div><div>To point the pulp2 code I mentioned I found here [0] (base rpm package what I understood).</div><div><br></div><div>The rise of error in pulp3 happening here [1] in pulpcore when adding packages to repository version. </div><div>So as Ina mentioned it doesn't have to be an issue with packages itself than the logic in sync. </div><div><br></div><div>[0] <a href="https://github.com/pulp/pulp_rpm/blob/2-master/plugins/pulp_rpm/plugins/db/models.py#L779" target="_blank">https://github.com/pulp/pulp_rpm/blob/2-master/plugins/pulp_rpm/plugins/db/models.py#L779</a></div><div>[1] <a href="https://github.com/pulp/pulpcore/blob/master/pulpcore/app/models/repository.py#L570" target="_blank">https://github.com/pulp/pulpcore/blob/master/pulpcore/app/models/repository.py#L570</a></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Wed, Mar 18, 2020 at 1:55 PM Ina Panova <<a href="mailto:ipanova@redhat.com" target="_blank">ipanova@redhat.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div>Tanya and Pavel,</div><div>in this issue it is explained why we cannot keep 2 packages with same NEVRA but different checksums within a repo <a href="https://pulp.plan.io/issues/494" target="_blank">https://pulp.plan.io/issues/494</a></div><div><br></div><div>Pulp2 had a limitation where it was not able to save on the filesystem 2 rpms with same filename, it lead to the primary.xml that could have pointed to the rpm that did not actually get saved.</div><div>I believe in Pulp3 we could allow having rpm with same NEVRA if they have different location_href within a repo.<br></div><div dir="ltr"><br>--------<br>Regards,<br><br>Ina Panova<br>Senior Software Engineer| Pulp| Red Hat Inc.<br><br>"Do not go where the path may lead,<br> go instead where there is no path and leave a trail."<br></div><div><br></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Wed, Mar 18, 2020 at 10:47 AM Tatiana Tereshchenko <<a href="mailto:ttereshc@redhat.com" target="_blank">ttereshc@redhat.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div dir="ltr">Hi Pavel,<br></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Tue, Mar 17, 2020 at 7:31 PM Pavel Picka <<a href="mailto:ppicka@redhat.com" target="_blank">ppicka@redhat.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">Hello, would like to ask you how to proceed with issue with duplicate (but not really) packages. <br><br>I am syncing suse repository (opensuse42 and SLE12) and get and duplicate error. But when checking the packages [0](from primary.xml) glibc and glibc they got same nevra but different checksum (and a few more as size..) so doesn't look like real duplicates.<br></div></blockquote><div>Those are weird, the have the same nevra but see the location_href, one is src and the other one is nosrc! :/ :</div><div><location href="nosrc/glibc-2.19-20.3.nosrc.rpm"/><br></div><div><location href="src/glibc-2.19-20.3.src.rpm"/><br></div><div> </div><div>It looks like something OpenSUSE specific. I'm not sure if it's a valid way to create a repo with such metadata, we need to figure it out at some point.</div><div><br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><br>I've checked Pulp2 and there is used nevra+sum for repository uniqueness. In pulp3 we use only nevra. <br></div></blockquote><div>Why do you think that in pulp 2 we use NEVRA + checksum? have you tested it? please point to the code.</div><div>I believe in Pulp 2 as well as in Pulp 3 we allow to have packages with different checksums in Pulp storage.</div><div>I don't think we allow having the same packages with different checksums in the same repo. </div><div>FWIW, in pulp 2 the most recently added package is chosen to stay in a repo, no packages with duplicate NEVRA left after sync, see <a href="https://github.com/pulp/pulp_rpm/blob/2-master/plugins/pulp_rpm/plugins/importers/yum/purge.py#L285-L333" target="_blank">https://github.com/pulp/pulp_rpm/blob/2-master/plugins/pulp_rpm/plugins/importers/yum/purge.py#L285-L333</a></div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><br>My suggestion is to extend repo_key_fields for rpm package as is in pulp2 with pkgId (checksum). As I don't think they are really duplicates and other software can rely on specific version of package.</div></blockquote><div><br></div><div>Unfortunately, I don't remember the main reason to remove duplicates based on nevra. Was it because some tooling will complain, or was it just to avoid duplicates at resync time? Does anyone know?</div><div>We should not change it unless we know for sure that it's needed + we would need to have an agreement from all our stakeholders for that change.</div><div><br></div><div>For now, I think we can move on and ensure that no duplicates are in a repo version. To my understanding, the behaviour will be the same as in pulp 2.</div><div>Feel free to share where you get duplicate error to see if it's a bug or not. I wonder why duplicates are not removed automatically. Maybe because the first version contains duplicates due to this bug <a href="https://pulp.plan.io/issues/6217" target="_blank">https://pulp.plan.io/issues/6217</a> ?</div><div><br></div><div>Tanya</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div><br></div><div>What do you think?<br><br><br>[0]<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><font face="monospace"><package type="rpm"><br> <name>glibc</name><br> <arch>src</arch><br> <version epoch="0" ver="2.19" rel="20.3"/><br> <checksum type="sha256" pkgid="YES">00d36c0f741b0c01a77ce318a2bbcfa59cb4dd0b24ce61f57c6205e4fa1bb310</checksum><br> <summary>Standard Shared Libraries (from the GNU C Library)</summary><br> <description>The GNU C Library provides the most important standard libraries used<br>by nearly all programs: the standard C library, the standard math<br>library, and the POSIX thread library. A system is not functional<br>without these libraries.</description><br> <packager><a href="https://www.suse.com/" target="_blank">https://www.suse.com/</a></packager><br> <url><a href="http://www.gnu.org/software/libc/libc.html" target="_blank">http://www.gnu.org/software/libc/libc.html</a></url><br> <time file="1426696882" build="1425645307"/><br> <size package="591662" installed="13047428" archive="974464"/><br><location href="nosrc/glibc-2.19-20.3.nosrc.rpm"/><br> <format><br> <rpm:license>LGPL-2.1+ and SUSE-LGPL-2.1+-with-GCC-exception and GPL-2.0+</rpm:license><br> <rpm:vendor>SUSE LLC <<a href="https://www.suse.com/&gt" target="_blank">https://www.suse.com/&gt</a>;</rpm:vendor><br> <rpm:group>System/Libraries</rpm:group><br> <rpm:buildhost>sheep16</rpm:buildhost><br> <rpm:sourcerpm/><br> <rpm:header-range start="872" end="144403"/><br> <rpm:requires><br> <rpm:entry name="pwdutils"/><br> <rpm:entry name="xz"/><br> <rpm:entry name="fdupes"/><br> <rpm:entry name="systemd-rpm-macros"/><br> <rpm:entry name="libselinux-devel"/><br> <rpm:entry name="makeinfo"/><br> </rpm:requires><br> </format><br></package><br><br><package type="rpm"><br> <name>glibc</name><br> <arch>src</arch><br> <version epoch="0" ver="2.19" rel="20.3"/><br> <checksum type="sha256" pkgid="YES">353e1dc85eab8d434be83160eca4fcee11a72eec345385df125ca0835abd6068</checksum><br> <summary>Standard Shared Libraries (from the GNU C Library)</summary><br> <description>The GNU C Library provides the most important standard libraries used<br>by nearly all programs: the standard C library, the standard math<br>library, and the POSIX thread library. A system is not functional<br>without these libraries.</description><br> <packager><a href="https://www.suse.com/" target="_blank">https://www.suse.com/</a></packager><br> <url><a href="http://www.gnu.org/software/libc/libc.html" target="_blank">http://www.gnu.org/software/libc/libc.html</a></url><br> <time file="1426696883" build="1423750734"/><br> <size package="12678975" installed="13047285" archive="13057760"/><br><location href="src/glibc-2.19-20.3.src.rpm"/><br> <format><br> <rpm:license>LGPL-2.1+ and SUSE-LGPL-2.1+-with-GCC-exception and GPL-2.0+</rpm:license><br> <rpm:vendor>SUSE LLC <<a href="https://www.suse.com/&gt" target="_blank">https://www.suse.com/&gt</a>;</rpm:vendor><br> <rpm:group>System/Libraries</rpm:group><br> <rpm:buildhost>sheep02</rpm:buildhost><br> <rpm:sourcerpm/><br> <rpm:header-range start="872" end="144334"/><br> <rpm:requires><br> <rpm:entry name="pwdutils"/><br> <rpm:entry name="xz"/><br> <rpm:entry name="fdupes"/><br> <rpm:entry name="systemd-rpm-macros"/><br> <rpm:entry name="libselinux-devel"/><br> <rpm:entry name="makeinfo"/><br> </rpm:requires><br> </format><br></package></font></blockquote><br>-- <br>Pavel Picka<br>Red Hat</div></div> _______________________________________________<br> Pulp-dev mailing list<br> <a href="mailto:Pulp-dev@redhat.com" target="_blank">Pulp-dev@redhat.com</a><br> <a href="https://www.redhat.com/mailman/listinfo/pulp-dev" rel="noreferrer" target="_blank">https://www.redhat.com/mailman/listinfo/pulp-dev</a><br> </blockquote></div></div> _______________________________________________<br> Pulp-dev mailing list<br> <a href="mailto:Pulp-dev@redhat.com" target="_blank">Pulp-dev@redhat.com</a><br> <a href="https://www.redhat.com/mailman/listinfo/pulp-dev" rel="noreferrer" target="_blank">https://www.redhat.com/mailman/listinfo/pulp-dev</a><br> </blockquote></div> </blockquote></div><br clear="all"><div><br></div>-- <br><div dir="ltr"><div dir="ltr"><div>Pavel Picka</div><div>Red Hat<br></div></div></div> _______________________________________________<br> Pulp-dev mailing list<br> <a href="mailto:Pulp-dev@redhat.com" target="_blank">Pulp-dev@redhat.com</a><br> <a href="https://www.redhat.com/mailman/listinfo/pulp-dev" rel="noreferrer" target="_blank">https://www.redhat.com/mailman/listinfo/pulp-dev</a><br> </blockquote></div> _______________________________________________<br> Pulp-dev mailing list<br> <a href="mailto:Pulp-dev@redhat.com" target="_blank">Pulp-dev@redhat.com</a><br> <a href="https://www.redhat.com/mailman/listinfo/pulp-dev" rel="noreferrer" target="_blank">https://www.redhat.com/mailman/listinfo/pulp-dev</a><br> </blockquote></div>