[Pulp-list] Using Pulp to "merge" multiple RPM repos into one?

[Nick Coghlan]

On 11 September 2015 at 23:09, Michael Hrivnak <mhrivnak at redhat.com> wrote:
> I think your plan is spot-on. In usually makes sense to have a 1-1 mapping
> of remote repos to pulp repos, and to keep the pulp repo as a simple mirror
> of that remote repo. From there, you can copy out of the pulp-hosted mirrors
> to compose new repos with whatever mix of content you like.

I've even come up with names I like for the RepoFunnel data model:
TrackingRepo and MergeRepo, and then the funnels are an N:1 mapping of
TrackingRepo event listeners to the target merge repo.

One feature that Pulp doesn't have yet that could be valuable for this
use case is the notion of a "metadata only" repo, where we don't
actually download the artifacts themsleve, but instead store just the
repo metadata, and the original *URLs* for the artifacts.

With appropriate publisher plugins, RepoFunnel could then be used an
input filter for an object storage based system like pinrepo:
https://github.com/pinterest/pinrepo

Regards,
Nick.

>
> Michael
>
> On Fri, Sep 11, 2015 at 5:42 AM, Nick Coghlan <ncoghlan at gmail.com> wrote:
>>
>> Hi folks,
>>
>> As part of a development workflow idea for the Fedora Environments &
>> Stacks working group [1], I'm looking to build a service that lets
>> people select multiple COPR repos, and have them automatically
>> integrated into a single downstream repo.
>>
>> As a starting point, I'm aiming to build the simplest possible proof
>> of concept: take two existing COPR repos, and configure Pulp to
>> download and republish all of their content as a single combined repo.
>>
>> I mistakenly thought I could do this just by adding multiple importers
>> to a single Pulp repository, but discovered today that Pulp doesn't
>> actually support doing that - the importer:repository mapping is 1:1.
>> Finding out I didn't know Pulp's capability's as well as I thought
>> made me realise I should ask here for advice before proceeding further
>> :)
>>
>> My current thinking is that my architecture will need to look something
>> like:
>>
>> 1. For any COPR repo I want to merge, configure a local mirror in Pulp
>> that imports the content from that repo. These would be system
>> managed, so there's only ever one local mirror per remote repo.
>> 2. For each funnel, configure a dedicated target repo, and create
>> event listeners on the relevant mirror repos that trigger a content
>> unit copy whenever the mirror repos are updated
>>
>> Does that general approach sound reasonable? Are there simpler
>> alternatives that I've missed?
>>
>> Regards,
>> Nick.
>>
>> [1]
>> https://fedoraproject.org/wiki/Env_and_Stacks/Projects/SoftwareComponentPipeline
>>
>> --
>> Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
>>
>> _______________________________________________
>> Pulp-list mailing list
>> Pulp-list at redhat.com
>> https://www.redhat.com/mailman/listinfo/pulp-list
>
>



-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia