[Pulp-dev] Duplicate nevra but not pkgId (suse repos)

Pavel Picka ppicka at redhat.com
Fri Mar 20 08:45:21 UTC 2020


I think we should keep nevra as unique constraint, but as I mentioned
before (above in this thread) your idea is similar to mine as my suggestion
was NEVRA + checksum (pkgId).
With pkgId I've already tested it and working good.

On Fri, Mar 20, 2020 at 5:43 AM Daniel Alley <dalley at redhat.com> wrote:

> I discussed this a little bit on the #rpm.org channel.  Here is the gist
> of that discussion
>
>    - The metadata is "crazy, but technically valid"
>    - "the entire SUSE ecosystem tends to do this a lot, anything using
>    OBS, including nvidia and dell and friends"
>    - "also, SUSE packages can have the same NEVRA with being completely
>    different packages because of how their build system makes packages"
>
> I'm not sure what the best means to fix it would be.  Perhaps the
> uniqueness constraint should be on the location_href, instead of on the
> NEVRA?  Or on NEVRA + location_href?
>
> On Wed, Mar 18, 2020 at 9:47 AM Ina Panova <ipanova at redhat.com> wrote:
>
>> Pavel,
>> I meant to say, that pulp3 does not have such limitation as pulp2 had (
>> saving rpms on the filesystem with same nevra).
>> The error is raised in pulp3 [0] when a repo version is created, because
>> of the repo key[1], we cannot have 2 rpms with save NEVRA.
>>
>> We can enable that, if we decide to, by adding location_href to the
>> repo_key, *but* this needs to be evaluated, it can have side effects and we
>> should involve our stakeholders to weigh in.
>>
>> [0]
>> https://github.com/pulp/pulpcore/blob/master/pulpcore/app/models/repository.py#L570
>> [1]
>> https://github.com/pulp/pulp_rpm/blob/master/pulp_rpm/app/models/package.py#L188
>>
>> --------
>> Regards,
>>
>> Ina Panova
>> Senior Software Engineer| Pulp| Red Hat Inc.
>>
>> "Do not go where the path may lead,
>>  go instead where there is no path and leave a trail."
>>
>>
>> On Wed, Mar 18, 2020 at 2:24 PM Pavel Picka <ppicka at redhat.com> wrote:
>>
>>> True in opensuse repository there are two possibilities 'src' and
>>> 'nosrc' (this one should be legacy without source code), both are
>>> recognized by createrepo_c as arch 'src'.
>>>
>>> To point the pulp2 code I mentioned I found here [0] (base rpm package
>>> what I understood).
>>>
>>> The rise of error in pulp3 happening here [1] in pulpcore when adding
>>> packages to repository version.
>>> So as Ina mentioned it doesn't have to be an issue with packages itself
>>> than the logic in sync.
>>>
>>> [0]
>>> https://github.com/pulp/pulp_rpm/blob/2-master/plugins/pulp_rpm/plugins/db/models.py#L779
>>> [1]
>>> https://github.com/pulp/pulpcore/blob/master/pulpcore/app/models/repository.py#L570
>>>
>>> On Wed, Mar 18, 2020 at 1:55 PM Ina Panova <ipanova at redhat.com> wrote:
>>>
>>>> Tanya and Pavel,
>>>> in this issue it is explained why we cannot keep 2 packages with same
>>>> NEVRA but different checksums within a repo
>>>> https://pulp.plan.io/issues/494
>>>>
>>>> Pulp2 had a limitation where it was not able to save on the filesystem
>>>> 2 rpms with same filename, it lead to the primary.xml that could have
>>>> pointed to the rpm that did not actually get saved.
>>>> I believe in Pulp3 we could allow having rpm with same NEVRA if they
>>>> have different location_href within a repo.
>>>>
>>>> --------
>>>> Regards,
>>>>
>>>> Ina Panova
>>>> Senior Software Engineer| Pulp| Red Hat Inc.
>>>>
>>>> "Do not go where the path may lead,
>>>>  go instead where there is no path and leave a trail."
>>>>
>>>>
>>>> On Wed, Mar 18, 2020 at 10:47 AM Tatiana Tereshchenko <
>>>> ttereshc at redhat.com> wrote:
>>>>
>>>>> Hi Pavel,
>>>>>
>>>>> On Tue, Mar 17, 2020 at 7:31 PM Pavel Picka <ppicka at redhat.com> wrote:
>>>>>
>>>>>> Hello, would like to ask you how to proceed with issue with duplicate
>>>>>> (but not really) packages.
>>>>>>
>>>>>> I am syncing suse repository (opensuse42 and SLE12) and get and
>>>>>> duplicate error. But when checking the packages [0](from primary.xml) glibc
>>>>>> and glibc they got same nevra but different checksum (and a few more as
>>>>>> size..) so doesn't look like real duplicates.
>>>>>>
>>>>> Those are weird, the have the same nevra but see the location_href,
>>>>> one is src and the other one is nosrc! :/ :
>>>>> <location href="nosrc/glibc-2.19-20.3.nosrc.rpm"/>
>>>>> <location href="src/glibc-2.19-20.3.src.rpm"/>
>>>>>
>>>>> It looks like something OpenSUSE specific. I'm not sure if it's a
>>>>> valid way to create a repo with such metadata, we need to figure it out at
>>>>> some point.
>>>>>
>>>>>
>>>>>> I've checked Pulp2 and there is used nevra+sum for repository
>>>>>> uniqueness. In pulp3 we use only nevra.
>>>>>>
>>>>> Why do you think that in pulp 2 we use NEVRA + checksum? have you
>>>>> tested it?  please point to the code.
>>>>> I believe in Pulp 2 as well as in Pulp 3 we allow to have packages
>>>>> with different checksums in Pulp storage.
>>>>> I don't think we allow having the same packages with different
>>>>> checksums in the same repo.
>>>>> FWIW, in pulp 2 the most recently added package is chosen to stay in a
>>>>> repo, no packages with duplicate NEVRA left after sync, see
>>>>> https://github.com/pulp/pulp_rpm/blob/2-master/plugins/pulp_rpm/plugins/importers/yum/purge.py#L285-L333
>>>>>
>>>>>
>>>>>>
>>>>>> My suggestion is to extend repo_key_fields for rpm package as is in
>>>>>> pulp2 with pkgId (checksum). As I don't think they are really duplicates
>>>>>> and other software can rely on specific version of package.
>>>>>>
>>>>>
>>>>> Unfortunately, I don't remember the main reason to remove duplicates
>>>>> based on nevra. Was it because some tooling will complain, or was it just
>>>>> to avoid duplicates at resync time? Does anyone know?
>>>>> We should not change it unless we know for sure that it's needed + we
>>>>> would need to have an agreement from all our stakeholders for that change.
>>>>>
>>>>> For now, I think we can move on and ensure that no duplicates are in a
>>>>> repo version. To my understanding, the behaviour will be the same as in
>>>>> pulp 2.
>>>>> Feel free to share where you get duplicate error to see if it's a bug
>>>>> or not. I wonder why duplicates are not removed automatically. Maybe
>>>>> because the first version contains duplicates due to this bug
>>>>> https://pulp.plan.io/issues/6217 ?
>>>>>
>>>>> Tanya
>>>>>
>>>>>
>>>>>>
>>>>>> What do you think?
>>>>>>
>>>>>>
>>>>>> [0]
>>>>>>
>>>>>>> <package type="rpm">
>>>>>>>   <name>glibc</name>
>>>>>>>   <arch>src</arch>
>>>>>>>   <version epoch="0" ver="2.19" rel="20.3"/>
>>>>>>>   <checksum type="sha256"
>>>>>>> pkgid="YES">00d36c0f741b0c01a77ce318a2bbcfa59cb4dd0b24ce61f57c6205e4fa1bb310</checksum>
>>>>>>>   <summary>Standard Shared Libraries (from the GNU C
>>>>>>> Library)</summary>
>>>>>>>   <description>The GNU C Library provides the most important
>>>>>>> standard libraries used
>>>>>>> by nearly all programs: the standard C library, the standard math
>>>>>>> library, and the POSIX thread library. A system is not functional
>>>>>>> without these libraries.</description>
>>>>>>>   <packager>https://www.suse.com/</packager>
>>>>>>>   <url>http://www.gnu.org/software/libc/libc.html</url>
>>>>>>>   <time file="1426696882" build="1425645307"/>
>>>>>>>   <size package="591662" installed="13047428" archive="974464"/>
>>>>>>> <location href="nosrc/glibc-2.19-20.3.nosrc.rpm"/>
>>>>>>>   <format>
>>>>>>>     <rpm:license>LGPL-2.1+ and SUSE-LGPL-2.1+-with-GCC-exception and
>>>>>>> GPL-2.0+</rpm:license>
>>>>>>>     <rpm:vendor>SUSE LLC <https://www.suse.com/></rpm:vendor>
>>>>>>>     <rpm:group>System/Libraries</rpm:group>
>>>>>>>     <rpm:buildhost>sheep16</rpm:buildhost>
>>>>>>>     <rpm:sourcerpm/>
>>>>>>>     <rpm:header-range start="872" end="144403"/>
>>>>>>>     <rpm:requires>
>>>>>>>       <rpm:entry name="pwdutils"/>
>>>>>>>       <rpm:entry name="xz"/>
>>>>>>>       <rpm:entry name="fdupes"/>
>>>>>>>       <rpm:entry name="systemd-rpm-macros"/>
>>>>>>>       <rpm:entry name="libselinux-devel"/>
>>>>>>>       <rpm:entry name="makeinfo"/>
>>>>>>>     </rpm:requires>
>>>>>>>   </format>
>>>>>>> </package>
>>>>>>>
>>>>>>> <package type="rpm">
>>>>>>>   <name>glibc</name>
>>>>>>>   <arch>src</arch>
>>>>>>>   <version epoch="0" ver="2.19" rel="20.3"/>
>>>>>>>   <checksum type="sha256"
>>>>>>> pkgid="YES">353e1dc85eab8d434be83160eca4fcee11a72eec345385df125ca0835abd6068</checksum>
>>>>>>>   <summary>Standard Shared Libraries (from the GNU C
>>>>>>> Library)</summary>
>>>>>>>   <description>The GNU C Library provides the most important
>>>>>>> standard libraries used
>>>>>>> by nearly all programs: the standard C library, the standard math
>>>>>>> library, and the POSIX thread library. A system is not functional
>>>>>>> without these libraries.</description>
>>>>>>>   <packager>https://www.suse.com/</packager>
>>>>>>>   <url>http://www.gnu.org/software/libc/libc.html</url>
>>>>>>>   <time file="1426696883" build="1423750734"/>
>>>>>>>   <size package="12678975" installed="13047285" archive="13057760"/>
>>>>>>> <location href="src/glibc-2.19-20.3.src.rpm"/>
>>>>>>>   <format>
>>>>>>>     <rpm:license>LGPL-2.1+ and SUSE-LGPL-2.1+-with-GCC-exception and
>>>>>>> GPL-2.0+</rpm:license>
>>>>>>>     <rpm:vendor>SUSE LLC <https://www.suse.com/></rpm:vendor>
>>>>>>>     <rpm:group>System/Libraries</rpm:group>
>>>>>>>     <rpm:buildhost>sheep02</rpm:buildhost>
>>>>>>>     <rpm:sourcerpm/>
>>>>>>>     <rpm:header-range start="872" end="144334"/>
>>>>>>>     <rpm:requires>
>>>>>>>       <rpm:entry name="pwdutils"/>
>>>>>>>       <rpm:entry name="xz"/>
>>>>>>>       <rpm:entry name="fdupes"/>
>>>>>>>       <rpm:entry name="systemd-rpm-macros"/>
>>>>>>>       <rpm:entry name="libselinux-devel"/>
>>>>>>>       <rpm:entry name="makeinfo"/>
>>>>>>>     </rpm:requires>
>>>>>>>   </format>
>>>>>>> </package>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> Pavel Picka
>>>>>> Red Hat
>>>>>> _______________________________________________
>>>>>> Pulp-dev mailing list
>>>>>> Pulp-dev at redhat.com
>>>>>> https://www.redhat.com/mailman/listinfo/pulp-dev
>>>>>>
>>>>> _______________________________________________
>>>>> Pulp-dev mailing list
>>>>> Pulp-dev at redhat.com
>>>>> https://www.redhat.com/mailman/listinfo/pulp-dev
>>>>>
>>>>
>>>
>>> --
>>> Pavel Picka
>>> Red Hat
>>> _______________________________________________
>>> Pulp-dev mailing list
>>> Pulp-dev at redhat.com
>>> https://www.redhat.com/mailman/listinfo/pulp-dev
>>>
>> _______________________________________________
>> Pulp-dev mailing list
>> Pulp-dev at redhat.com
>> https://www.redhat.com/mailman/listinfo/pulp-dev
>>
>

-- 
Pavel Picka
Red Hat
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/pulp-dev/attachments/20200320/ec6d1ffe/attachment.htm>


More information about the Pulp-dev mailing list