[Pulp-list] epel syncs all advisories every time

Wed Feb 10 14:34:25 UTC 2021

Great, it confirms that the advisory is recreated for updated ones only.
Thanks for the feedback and for the filed issue.

Tanya

On Wed, Feb 10, 2021 at 3:18 PM Winberg Adam <Adam.Winberg at smhi.se> wrote:

> > As for the recreating of advisories, if it happens to *all* of them for
> every sync, it seems like a Pulp RPM bug to me.
>
> You are correct, it does not happen to all of them but the majority.
>
>
> > If you try to sync two times in row, do you still see all advisories
> being recreated?
> No, two syncs in a row does not render any changes, it's only happening on
> a new day (i.e. after epel repo metadata has been updated).
>
> I created an issue for this, https://pulp.plan.io/issues/8229, since I
> don't think an advisory should be replaced in Pulp if the only changed
> metadata is 'updated_date'. But I might not see the whole picture.
>
> //Adam
>
> ------------------------------
> *From:* Tanya Tereshchenko <ttereshc at redhat.com>
> *Sent:* 10 February 2021 14:45
> *To:* Winberg Adam
> *Cc:* pulp-list at redhat.com
> *Subject:* Re: [Pulp-list] epel syncs all advisories every time
>
> Hi Adam,
>
> If you see this AdvisoryConflict for the migrated repos, please file an
> issue https://pulp.plan.io/projects/migration/issues/new
> Pulp2 managed advisories in a completely different way so there could be
> pkglist discrepancies when trying to update them (because Fedora advisories
> often refer to packages which are no longer in a repo)
> We'll see if there is something we can suggest to fix those or if there is
> a need to have a way in RPM plugin to overcome those in some user friendly
> way.
>
> As for the recreating of advisories, if it happens to *all* of them for
> every sync, it seems like a Pulp RPM bug to me.
> If it happens to many of them and you sync once a day at most, it seems to
> be related to what you pointed out - the updated date in EPEL repos changes
> very often.
> I checked EPEL7 and EPEL8 and indeed, not all but the majority of
> advisories there have today as their updated date.
> If you try to sync two times in row, do you still see all advisories being
> recreated?
>
> Thanks,
> Tanya
>
> On Mon, Feb 8, 2021 at 8:37 PM Winberg Adam <Adam.Winberg at smhi.se> wrote:
>
>> Looking a bit more on this the copy issue, with the error
>>
>>
>> pulp_rpm.app.exceptions.AdvisoryConflict: Incoming and existing
>> advisories have the same id and timestamp but different and intersecting
>> package lists. At least one of them is wrong. Advisory id:
>> FEDORA-EPEL-2019-927a9446df
>>
>> seems to be due to the advisory metadata in my frozen epel repos being
>> broken, the package lists are incomplete. These repos are migrated from
>> pulp2, but I don't know if the metadata was broken already in pulp2 or if
>> it broke during the migration. My pulp2 environment is no longer with me so
>> there's no way for me to find out.
>>
>> So I need to fix those repos. Once thats done the issue with EPEL syncs
>> re-adding all advisories are probably not that big, but it does make
>> syncing take quite some time and also makes it difficult for me to filter
>> newly added advisories (since they are added at every sync). So if anyone
>> has any idea how to solve this I would be grateful. Comparing the
>> advisories that are added with the ones that are removed shows that they
>> are identical except for a newer 'updated_date' on the added one (the
>> updated_date is todays date).
>>
>> Maybe an advisory shouldnt be considered for addition if the only change
>> is the updated_date and no other metadata is changed? And why is EPEL
>> setting a new 'update_date' on their advisories at every rebuild/re-index
>> of their repos?
>>
>> //Adam
>>
>>
>> ------------------------------
>> *From:* pulp-list-bounces at redhat.com <pulp-list-bounces at redhat.com> on
>> behalf of Winberg Adam <Adam.Winberg at smhi.se>
>> *Sent:* 08 February 2021 16:07
>> *To:* pulp-list at redhat.com
>> *Subject:* Re: [Pulp-list] epel syncs all advisories every time
>>
>>
>> The patch was also for copying, not syncing, so I was totally confused.
>> Sorry about that...
>>
>>
>> ------------------------------
>> *From:* pulp-list-bounces at redhat.com <pulp-list-bounces at redhat.com> on
>> behalf of Winberg Adam <Adam.Winberg at smhi.se>
>> *Sent:* 08 February 2021 15:58
>> *To:* pulp-list at redhat.com
>> *Subject:* Re: [Pulp-list] epel syncs all advisories every time
>>
>>
>> Did a quick test and the patch i mentioned is not involved.
>>
>>
>> Note that I only see this problem with EPEL, not with RHEL repos which
>> also publishes erratas/advisories. So the problem is probably on the EPEL
>> side, or..?
>>
>>
>> //Adam
>>
>>
>> ------------------------------
>> *From:* pulp-list-bounces at redhat.com <pulp-list-bounces at redhat.com> on
>> behalf of Winberg Adam <Adam.Winberg at smhi.se>
>> *Sent:* 08 February 2021 10:55
>> *To:* pulp-list at redhat.com
>> *Subject:* Re: [Pulp-list] epel syncs all advisories every time
>>
>>
>> I'm not sure, but I think this behaviour might have started after I
>> applied the following patch:
>>
>>
>> https://github.com/pulp/pulp_rpm/commit/1652026913308e8348543af6f62c3b5c5f89985b#diff-0b195d23762f04b205940bafb5889ddf96181afde122ead35f8c65fe03527647
>>
>>
>> //Adam
>>
>>
>> ------------------------------
>> *From:* pulp-list-bounces at redhat.com <pulp-list-bounces at redhat.com> on
>> behalf of Winberg Adam <Adam.Winberg at smhi.se>
>> *Sent:* 08 February 2021 10:39
>> *To:* pulp-list at redhat.com
>> *Subject:* [Pulp-list] epel syncs all advisories every time
>>
>>
>> I sync the rhel8 and rhel7 epel repo every day, and for some reason all
>> advisories are removed and added each time:
>>
>>
>>     "content_summary": {
>>         "added": {
>>             "rpm.advisory": {
>>                 "count": 2540,
>>                 "href":
>> "/pulp/api/v3/content/rpm/advisories/?repository_version_added=/pulp/api/v3/repositories/rpm/rpm/10e51ae6-65c7-42aa-8ab1-ffebdf752500/versions/19/"
>>             },
>>             "rpm.package": {
>>                 "count": 6,
>>                 "href":
>> "/pulp/api/v3/content/rpm/packages/?repository_version_added=/pulp/api/v3/repositories/rpm/rpm/10e51ae6-65c7-42aa-8ab1-ffebdf752500/versions/19/"
>>             }
>>         },
>>         "removed": {
>>             "rpm.advisory": {
>>                 "count": 2536,
>>                 "href":
>> "/pulp/api/v3/content/rpm/advisories/?repository_version_removed=/pulp/api/v3/repositories/rpm/rpm/10e51ae6-65c7-42aa-8ab1-ffebdf752500/versions/19/"
>>             }
>>         }
>>     },
>>
>>
>> This also leads to problem when I want to copy new advisories to my
>> frozen epel repo:
>> pulp_rpm.app.exceptions.AdvisoryConflict: Incoming and existing
>> advisories have the same id and timestamp but different and intersecting
>> package lists. At least one of them is wrong. Advisory id:
>> FEDORA-EPEL-2019-927a9446df
>>
>> So it seems that the advisory has actually changed, which seems weird.
>> And changed every day? Is this some quirk in the EPEL sources or a bug
>> in Pulp?
>>
>> //Adam
>>
>>
>>
>>
>> _______________________________________________
>> Pulp-list mailing list
>> Pulp-list at redhat.com
>> https://www.redhat.com/mailman/listinfo/pulp-list
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/pulp-list/attachments/20210210/7ff52ccd/attachment.htm>