[Pulp-dev] Duplicate nevra but not pkgId (suse repos)

Tatiana Tereshchenko ttereshc at redhat.com
Wed Mar 18 09:46:40 UTC 2020


Hi Pavel,

On Tue, Mar 17, 2020 at 7:31 PM Pavel Picka <ppicka at redhat.com> wrote:

> Hello, would like to ask you how to proceed with issue with duplicate (but
> not really) packages.
>
> I am syncing suse repository (opensuse42 and SLE12) and get and duplicate
> error. But when checking the packages [0](from primary.xml) glibc and glibc
> they got same nevra but different checksum (and a few more as size..) so
> doesn't look like real duplicates.
>
Those are weird, the have the same nevra but see the location_href, one is
src and the other one is nosrc! :/ :
<location href="nosrc/glibc-2.19-20.3.nosrc.rpm"/>
<location href="src/glibc-2.19-20.3.src.rpm"/>

It looks like something OpenSUSE specific. I'm not sure if it's a valid way
to create a repo with such metadata, we need to figure it out at some point.


> I've checked Pulp2 and there is used nevra+sum for repository uniqueness.
> In pulp3 we use only nevra.
>
Why do you think that in pulp 2 we use NEVRA + checksum? have you tested
it?  please point to the code.
I believe in Pulp 2 as well as in Pulp 3 we allow to have packages with
different checksums in Pulp storage.
I don't think we allow having the same packages with different checksums in
the same repo.
FWIW, in pulp 2 the most recently added package is chosen to stay in a
repo, no packages with duplicate NEVRA left after sync, see
https://github.com/pulp/pulp_rpm/blob/2-master/plugins/pulp_rpm/plugins/importers/yum/purge.py#L285-L333


>
> My suggestion is to extend repo_key_fields for rpm package as is in pulp2
> with pkgId (checksum). As I don't think they are really duplicates and
> other software can rely on specific version of package.
>

Unfortunately, I don't remember the main reason to remove duplicates based
on nevra. Was it because some tooling will complain, or was it just to
avoid duplicates at resync time? Does anyone know?
We should not change it unless we know for sure that it's needed + we would
need to have an agreement from all our stakeholders for that change.

For now, I think we can move on and ensure that no duplicates are in a repo
version. To my understanding, the behaviour will be the same as in pulp 2.
Feel free to share where you get duplicate error to see if it's a bug or
not. I wonder why duplicates are not removed automatically. Maybe because
the first version contains duplicates due to this bug
https://pulp.plan.io/issues/6217 ?

Tanya


>
> What do you think?
>
>
> [0]
>
>> <package type="rpm">
>>   <name>glibc</name>
>>   <arch>src</arch>
>>   <version epoch="0" ver="2.19" rel="20.3"/>
>>   <checksum type="sha256"
>> pkgid="YES">00d36c0f741b0c01a77ce318a2bbcfa59cb4dd0b24ce61f57c6205e4fa1bb310</checksum>
>>   <summary>Standard Shared Libraries (from the GNU C Library)</summary>
>>   <description>The GNU C Library provides the most important standard
>> libraries used
>> by nearly all programs: the standard C library, the standard math
>> library, and the POSIX thread library. A system is not functional
>> without these libraries.</description>
>>   <packager>https://www.suse.com/</packager>
>>   <url>http://www.gnu.org/software/libc/libc.html</url>
>>   <time file="1426696882" build="1425645307"/>
>>   <size package="591662" installed="13047428" archive="974464"/>
>> <location href="nosrc/glibc-2.19-20.3.nosrc.rpm"/>
>>   <format>
>>     <rpm:license>LGPL-2.1+ and SUSE-LGPL-2.1+-with-GCC-exception and
>> GPL-2.0+</rpm:license>
>>     <rpm:vendor>SUSE LLC <https://www.suse.com/></rpm:vendor>
>>     <rpm:group>System/Libraries</rpm:group>
>>     <rpm:buildhost>sheep16</rpm:buildhost>
>>     <rpm:sourcerpm/>
>>     <rpm:header-range start="872" end="144403"/>
>>     <rpm:requires>
>>       <rpm:entry name="pwdutils"/>
>>       <rpm:entry name="xz"/>
>>       <rpm:entry name="fdupes"/>
>>       <rpm:entry name="systemd-rpm-macros"/>
>>       <rpm:entry name="libselinux-devel"/>
>>       <rpm:entry name="makeinfo"/>
>>     </rpm:requires>
>>   </format>
>> </package>
>>
>> <package type="rpm">
>>   <name>glibc</name>
>>   <arch>src</arch>
>>   <version epoch="0" ver="2.19" rel="20.3"/>
>>   <checksum type="sha256"
>> pkgid="YES">353e1dc85eab8d434be83160eca4fcee11a72eec345385df125ca0835abd6068</checksum>
>>   <summary>Standard Shared Libraries (from the GNU C Library)</summary>
>>   <description>The GNU C Library provides the most important standard
>> libraries used
>> by nearly all programs: the standard C library, the standard math
>> library, and the POSIX thread library. A system is not functional
>> without these libraries.</description>
>>   <packager>https://www.suse.com/</packager>
>>   <url>http://www.gnu.org/software/libc/libc.html</url>
>>   <time file="1426696883" build="1423750734"/>
>>   <size package="12678975" installed="13047285" archive="13057760"/>
>> <location href="src/glibc-2.19-20.3.src.rpm"/>
>>   <format>
>>     <rpm:license>LGPL-2.1+ and SUSE-LGPL-2.1+-with-GCC-exception and
>> GPL-2.0+</rpm:license>
>>     <rpm:vendor>SUSE LLC <https://www.suse.com/></rpm:vendor>
>>     <rpm:group>System/Libraries</rpm:group>
>>     <rpm:buildhost>sheep02</rpm:buildhost>
>>     <rpm:sourcerpm/>
>>     <rpm:header-range start="872" end="144334"/>
>>     <rpm:requires>
>>       <rpm:entry name="pwdutils"/>
>>       <rpm:entry name="xz"/>
>>       <rpm:entry name="fdupes"/>
>>       <rpm:entry name="systemd-rpm-macros"/>
>>       <rpm:entry name="libselinux-devel"/>
>>       <rpm:entry name="makeinfo"/>
>>     </rpm:requires>
>>   </format>
>> </package>
>
>
> --
> Pavel Picka
> Red Hat
> _______________________________________________
> Pulp-dev mailing list
> Pulp-dev at redhat.com
> https://www.redhat.com/mailman/listinfo/pulp-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/pulp-dev/attachments/20200318/62896d01/attachment.htm>


More information about the Pulp-dev mailing list