[Pulp-dev] Duplicate nevra but not pkgId (suse repos)
Pavel Picka
ppicka at redhat.com
Wed Mar 18 13:24:07 UTC 2020
True in opensuse repository there are two possibilities 'src' and 'nosrc'
(this one should be legacy without source code), both are recognized by
createrepo_c as arch 'src'.
To point the pulp2 code I mentioned I found here [0] (base rpm package what
I understood).
The rise of error in pulp3 happening here [1] in pulpcore when adding
packages to repository version.
So as Ina mentioned it doesn't have to be an issue with packages itself
than the logic in sync.
[0]
https://github.com/pulp/pulp_rpm/blob/2-master/plugins/pulp_rpm/plugins/db/models.py#L779
[1]
https://github.com/pulp/pulpcore/blob/master/pulpcore/app/models/repository.py#L570
On Wed, Mar 18, 2020 at 1:55 PM Ina Panova <ipanova at redhat.com> wrote:
> Tanya and Pavel,
> in this issue it is explained why we cannot keep 2 packages with same
> NEVRA but different checksums within a repo
> https://pulp.plan.io/issues/494
>
> Pulp2 had a limitation where it was not able to save on the filesystem 2
> rpms with same filename, it lead to the primary.xml that could have pointed
> to the rpm that did not actually get saved.
> I believe in Pulp3 we could allow having rpm with same NEVRA if they have
> different location_href within a repo.
>
> --------
> Regards,
>
> Ina Panova
> Senior Software Engineer| Pulp| Red Hat Inc.
>
> "Do not go where the path may lead,
> go instead where there is no path and leave a trail."
>
>
> On Wed, Mar 18, 2020 at 10:47 AM Tatiana Tereshchenko <ttereshc at redhat.com>
> wrote:
>
>> Hi Pavel,
>>
>> On Tue, Mar 17, 2020 at 7:31 PM Pavel Picka <ppicka at redhat.com> wrote:
>>
>>> Hello, would like to ask you how to proceed with issue with duplicate
>>> (but not really) packages.
>>>
>>> I am syncing suse repository (opensuse42 and SLE12) and get and
>>> duplicate error. But when checking the packages [0](from primary.xml) glibc
>>> and glibc they got same nevra but different checksum (and a few more as
>>> size..) so doesn't look like real duplicates.
>>>
>> Those are weird, the have the same nevra but see the location_href, one
>> is src and the other one is nosrc! :/ :
>> <location href="nosrc/glibc-2.19-20.3.nosrc.rpm"/>
>> <location href="src/glibc-2.19-20.3.src.rpm"/>
>>
>> It looks like something OpenSUSE specific. I'm not sure if it's a valid
>> way to create a repo with such metadata, we need to figure it out at some
>> point.
>>
>>
>>> I've checked Pulp2 and there is used nevra+sum for repository
>>> uniqueness. In pulp3 we use only nevra.
>>>
>> Why do you think that in pulp 2 we use NEVRA + checksum? have you tested
>> it? please point to the code.
>> I believe in Pulp 2 as well as in Pulp 3 we allow to have packages with
>> different checksums in Pulp storage.
>> I don't think we allow having the same packages with different checksums
>> in the same repo.
>> FWIW, in pulp 2 the most recently added package is chosen to stay in a
>> repo, no packages with duplicate NEVRA left after sync, see
>> https://github.com/pulp/pulp_rpm/blob/2-master/plugins/pulp_rpm/plugins/importers/yum/purge.py#L285-L333
>>
>>
>>>
>>> My suggestion is to extend repo_key_fields for rpm package as is in
>>> pulp2 with pkgId (checksum). As I don't think they are really duplicates
>>> and other software can rely on specific version of package.
>>>
>>
>> Unfortunately, I don't remember the main reason to remove duplicates
>> based on nevra. Was it because some tooling will complain, or was it just
>> to avoid duplicates at resync time? Does anyone know?
>> We should not change it unless we know for sure that it's needed + we
>> would need to have an agreement from all our stakeholders for that change.
>>
>> For now, I think we can move on and ensure that no duplicates are in a
>> repo version. To my understanding, the behaviour will be the same as in
>> pulp 2.
>> Feel free to share where you get duplicate error to see if it's a bug or
>> not. I wonder why duplicates are not removed automatically. Maybe because
>> the first version contains duplicates due to this bug
>> https://pulp.plan.io/issues/6217 ?
>>
>> Tanya
>>
>>
>>>
>>> What do you think?
>>>
>>>
>>> [0]
>>>
>>>> <package type="rpm">
>>>> <name>glibc</name>
>>>> <arch>src</arch>
>>>> <version epoch="0" ver="2.19" rel="20.3"/>
>>>> <checksum type="sha256"
>>>> pkgid="YES">00d36c0f741b0c01a77ce318a2bbcfa59cb4dd0b24ce61f57c6205e4fa1bb310</checksum>
>>>> <summary>Standard Shared Libraries (from the GNU C Library)</summary>
>>>> <description>The GNU C Library provides the most important standard
>>>> libraries used
>>>> by nearly all programs: the standard C library, the standard math
>>>> library, and the POSIX thread library. A system is not functional
>>>> without these libraries.</description>
>>>> <packager>https://www.suse.com/</packager>
>>>> <url>http://www.gnu.org/software/libc/libc.html</url>
>>>> <time file="1426696882" build="1425645307"/>
>>>> <size package="591662" installed="13047428" archive="974464"/>
>>>> <location href="nosrc/glibc-2.19-20.3.nosrc.rpm"/>
>>>> <format>
>>>> <rpm:license>LGPL-2.1+ and SUSE-LGPL-2.1+-with-GCC-exception and
>>>> GPL-2.0+</rpm:license>
>>>> <rpm:vendor>SUSE LLC <https://www.suse.com/></rpm:vendor>
>>>> <rpm:group>System/Libraries</rpm:group>
>>>> <rpm:buildhost>sheep16</rpm:buildhost>
>>>> <rpm:sourcerpm/>
>>>> <rpm:header-range start="872" end="144403"/>
>>>> <rpm:requires>
>>>> <rpm:entry name="pwdutils"/>
>>>> <rpm:entry name="xz"/>
>>>> <rpm:entry name="fdupes"/>
>>>> <rpm:entry name="systemd-rpm-macros"/>
>>>> <rpm:entry name="libselinux-devel"/>
>>>> <rpm:entry name="makeinfo"/>
>>>> </rpm:requires>
>>>> </format>
>>>> </package>
>>>>
>>>> <package type="rpm">
>>>> <name>glibc</name>
>>>> <arch>src</arch>
>>>> <version epoch="0" ver="2.19" rel="20.3"/>
>>>> <checksum type="sha256"
>>>> pkgid="YES">353e1dc85eab8d434be83160eca4fcee11a72eec345385df125ca0835abd6068</checksum>
>>>> <summary>Standard Shared Libraries (from the GNU C Library)</summary>
>>>> <description>The GNU C Library provides the most important standard
>>>> libraries used
>>>> by nearly all programs: the standard C library, the standard math
>>>> library, and the POSIX thread library. A system is not functional
>>>> without these libraries.</description>
>>>> <packager>https://www.suse.com/</packager>
>>>> <url>http://www.gnu.org/software/libc/libc.html</url>
>>>> <time file="1426696883" build="1423750734"/>
>>>> <size package="12678975" installed="13047285" archive="13057760"/>
>>>> <location href="src/glibc-2.19-20.3.src.rpm"/>
>>>> <format>
>>>> <rpm:license>LGPL-2.1+ and SUSE-LGPL-2.1+-with-GCC-exception and
>>>> GPL-2.0+</rpm:license>
>>>> <rpm:vendor>SUSE LLC <https://www.suse.com/></rpm:vendor>
>>>> <rpm:group>System/Libraries</rpm:group>
>>>> <rpm:buildhost>sheep02</rpm:buildhost>
>>>> <rpm:sourcerpm/>
>>>> <rpm:header-range start="872" end="144334"/>
>>>> <rpm:requires>
>>>> <rpm:entry name="pwdutils"/>
>>>> <rpm:entry name="xz"/>
>>>> <rpm:entry name="fdupes"/>
>>>> <rpm:entry name="systemd-rpm-macros"/>
>>>> <rpm:entry name="libselinux-devel"/>
>>>> <rpm:entry name="makeinfo"/>
>>>> </rpm:requires>
>>>> </format>
>>>> </package>
>>>
>>>
>>> --
>>> Pavel Picka
>>> Red Hat
>>> _______________________________________________
>>> Pulp-dev mailing list
>>> Pulp-dev at redhat.com
>>> https://www.redhat.com/mailman/listinfo/pulp-dev
>>>
>> _______________________________________________
>> Pulp-dev mailing list
>> Pulp-dev at redhat.com
>> https://www.redhat.com/mailman/listinfo/pulp-dev
>>
>
--
Pavel Picka
Red Hat
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/pulp-dev/attachments/20200318/fdd831bf/attachment.htm>
More information about the Pulp-dev
mailing list