[Pulp-dev] Replicate metadata

Wed May 16 20:19:39 UTC 2018

I think it would resolve like this:

If repository.exact_mirror == True:
  # sync as if sync_mode='mirror' and also save the remote metadata as a
unit also so the NoMetadataPublisher can generically publish it
else:
  # sync according to sync_mode either 'additive' or 'mirror'. This mirror
will *not* save the remote metadata so that's how it's feature different

I can write this up in redmine and send it to the list tomorrow.

On Wed, May 16, 2018 at 2:58 PM, David Davis <daviddavis at redhat.com> wrote:

> I had a similar idea in mind. How would this work with the sync_mode
> option though?
>
> +1 to adding this to redmine.
>
>
> David
>
> On Wed, May 16, 2018 at 2:40 PM, Brian Bouterse <bbouters at redhat.com>
> wrote:
>
>> I was thinking about these use cases. I'll use the same "first" and
>> "second" that @dkliban described.
>>
>> I recognize the "first" use case as a valid one. I think we could bring
>> it to bear pretty easily with a little bit of plugin involvement. We could
>> have a repository attribute boolen called exact_mirror (or similar) which
>> when False would do the following:
>> (a) allow a plugin writer to implement the exact-saving of metadata as an
>> Artifact
>> (b) disable add/remove of content using the always-raise an exception
>> with this proposal here:  https://pulp.plan.io/issues/3541#note-3
>> (c) have the plugin API offer a generic publisher called
>> NoMetdataPublisher that blindly publishes all Artifacts associated with a
>> RepoVersion
>>
>> This would cause the repo and its metadata to get published exactly, and
>> prevent core from corrupting it by changing the other content since we
>> can't also change the metadata.
>>
>> For the "second" use case, I think you could fulfil it by making an exact
>> mirror of the orignal publish so that it's only published once. This adds a
>> nice mirroring capability to Pulp.
>>
>> Should we move these plans into Redmine? What else do we need to know?
>>
>>
>> On Mon, May 14, 2018 at 3:12 PM, Dennis Kliban <dkliban at redhat.com>
>> wrote:
>>
>>> On Mon, May 14, 2018 at 1:27 PM, Justin Sherrill <jsherril at redhat.com>
>>> wrote:
>>>
>>>> I think it kind depends in how pulp would do it.  Thinking of the yum
>>>> example, all the information is there in an upstream yum repo for pulp to
>>>> import the 'publication' as is.  If it can be done 'naturally' with a yum
>>>> repo, there wouldn't be anything special around pulp -> pulp, it'd just be
>>>> yum_repo -> pulp.  However we'd need a pulp dev to chime in here :)
>>>>
>>>> This workflow is two use cases in Pulp 3:
>>>
>>>    - "As a user, I can create a repository version that is an exact
>>> mirror of a remote."
>>>    - "As a user, I can publish a repository version without generating
>>> metadata."
>>>
>>> The first use case would be satisfied by having the plugin provide code
>>> that downloads the metadata from the remote and saves it as 'metadata
>>> content' that's part of the repository version.
>>>
>>> The second use case could be satisfied by pulpcore providing a generic
>>> publisher that pubilshes everything in a repository version. The metadata
>>> would just be treated like any other files in the repo.
>>>
>>>
>>>
>>>> Justin
>>>>
>>>> On 05/14/2018 12:08 PM, Bryan Kearney wrote:
>>>>
>>>> @Justin, I think that makes sense for Pulp->Pulp. For Matthias, I think
>>>> he needs Native->Pulp which would not have a publication.
>>>>
>>>> -- bk
>>>>
>>>> On 05/14/2018 11:42 AM, Matthias Dellweg wrote:
>>>>
>>>> Mirroring the metedata exactly is also very important for Debian
>>>> Repositories, because of the way the metadata is signed in lieu of the
>>>> whole content. So it would be very beneficial, if this could be
>>>> provided as a 'service' of the pulp core platform somehow.
>>>>
>>>> Matthias
>>>>
>>>> On Mon, 14 May 2018 11:31:52 -0400
>>>> Justin Sherrill <jsherril at redhat.com> <jsherril at redhat.com> wrote:
>>>>
>>>>
>>>>  From my understanding of pulp 3, this would maybe involve the
>>>> ability to 'import' a publication.  Would that make sense?
>>>>
>>>> Justin
>>>>
>>>>
>>>> On 05/09/2018 08:22 AM, Bryan Kearney wrote:
>>>>
>>>> One of the themes I heard yesterday at the Red Hat Summit was around
>>>> having a pulp server mirror the upstream RPM repo metadata exactly.
>>>> The use case is that two pulp servers are behind a load balancer
>>>> mirroring the same repo. The users would like to be able to flip a
>>>> yum client acrross the two servers. Running createrepo to make
>>>> unique repos causes issues for the clients that appear to be
>>>> errors. I assume this pattern would not be unique for other package
>>>> clients that cache metadata.
>>>>
>>>> So, when looking ahead to pulp 3 I would ask that this be taken into
>>>> consideration. I can provide more info / use cases if necessary.
>>>>
>>>> -- bk
>>>>
>>>>
>>>>
>>>> _______________________________________________
>>>> Pulp-dev mailing listPulp-dev at redhat.comhttps://www.redhat.com/mailman/listinfo/pulp-dev
>>>>
>>>> _______________________________________________
>>>> Pulp-dev mailing listPulp-dev at redhat.comhttps://www.redhat.com/mailman/listinfo/pulp-dev
>>>>
>>>>
>>>>
>>>> _______________________________________________
>>>> Pulp-dev mailing listPulp-dev at redhat.comhttps://www.redhat.com/mailman/listinfo/pulp-dev
>>>>
>>>>
>>>>
>>>> _______________________________________________
>>>> Pulp-dev mailing list
>>>> Pulp-dev at redhat.com
>>>> https://www.redhat.com/mailman/listinfo/pulp-dev
>>>>
>>>>
>>>
>>> _______________________________________________
>>> Pulp-dev mailing list
>>> Pulp-dev at redhat.com
>>> https://www.redhat.com/mailman/listinfo/pulp-dev
>>>
>>>
>>
>> _______________________________________________
>> Pulp-dev mailing list
>> Pulp-dev at redhat.com
>> https://www.redhat.com/mailman/listinfo/pulp-dev
>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/pulp-dev/attachments/20180516/d5d70890/attachment.htm>