[Pulp-dev] Composed Repositories

Tue May 15 14:42:25 UTC 2018

On 05/15/2018 05:58 AM, Austin Macdonald wrote:
> Here's another complexity, how do 2 plugins create a single publication?

The plugin API could make this seamless.

> We basically have the same problem of 2 parallel operations creating 
> content from a single source.

I don't think so.  plugins should not manipulate content outside of 
their domain (other plugins content) so either serial or parallel should 
be safe.

>
> On Tue, May 15, 2018, 06:27 Ina Panova <ipanova at redhat.com 
> <mailto:ipanova at redhat.com>> wrote:
>
>     +1 on not introducing dependencies between plugins.
>
>     What will be the behavior in case there is a composed repo of rpm
>     and ks trees but just the rpm plugin is installed?
>
>     Do we fail and say we cannot sync this repo at all or we just sync
>     the rpm part?
>
>
> Assuming plugins do not depend on each other, I think that when each 
> plugin looks at the upstream repo, they will only "see" the content of 
> that type. Conceptually, we will have 2 remotes, so it will feel like 
> we are syncing from 2 totally distinct repositories.
>
> The solution I've been imagining is a lot like 2. Each plugin would 
> sync to a *separate repository.* These separate repositories are then 
> published creating *separate publications*. This approach allows the 
> plugins to live completely in ignorance of each other.
>
> The final step is to associate *both publications to one 
> distribution*, which composes the publications as they are served.
>
> The downside is that we have to sync and publish twice, and that the 
> resulting versions and publications aren't locked together. But I 
> think this is better than leaving versions and publications unfinished 
> with the assumption that another plugin will finish the job. Maybe 
> linking them together could be a good use of the notes field.

Pulp should support repositories with composed (mixed) content for the 
same reason RH does.  The repository is a collection of content that 
users want to manage together.  Consider the promotion cases: dev, test, 
prod.

>
>
>     Depends how we plan this ^ i guess we'll decide which option 1 or
>     2 fits better.
>
>     Don't want to go wild, but what if notion of composed repos will
>     be so popular in the future that's its amount will increase? I
>     think we do want to at least partially being able to sync it and
>     not take the approach all or nothing?
>
>     #2 speaks to me more for now.
>
>
>
>
>     --------
>     Regards,
>
>     Ina Panova
>     Software Engineer| Pulp| Red Hat Inc.
>
>     "Do not go where the path may lead,
>      go instead where there is no path and leave a trail."
>
>     On Mon, May 14, 2018 at 9:44 PM, Jeff Ortel <jortel at redhat.com
>     <mailto:jortel at redhat.com>> wrote:
>
>         Let's brainstorm on something.
>
>         Pulp needs to deal with remote repositories that are composed
>         of multiple content types which may span the domain of a
>         single plugin.  Here are a few examples.  Some Red Hat RPM
>         repositories are composed of: RPMs, DRPMs, , ISOs and
>         Kickstart Trees.  Some OSTree repositories are composed of
>         OSTrees & Kickstart Trees. This raises a question:
>
>         How can pulp3 best support syncing with remote repositories
>         that are composed of multiple (unrelated) content types in a
>         way that doesn't result in plugins duplicating support for
>         content types?
>
>         Few approaches come to mind:
>
>         1. Multiple plugins (Remotes) participate in the sync flow to
>         produce a new repository version.
>         2. Multiple plugins (Remotes) are sync'd successively each
>         producing a new version of a repository.  Only the last
>         version contains the fully sync'd composition.
>         3. Plugins share code.
>         4. Other?
>
>
>         Option #1: Sync would be orchestrated by core or the user so
>         that multiple plugins (Remotes) participate in populating a
>         new repository version.  For example: the RPM plugin (Remote)
>         and the Kickstart Tree plugin (Remote) would both be sync'd
>         against the same remote repository that is composed of both
>         types.  The new repository version would be composed of the
>         result of both plugin (Remote) syncs.  To support this, we'd
>         need to provide a way for each plugin to operate seamlessly on
>         the same (new) repository version.  Perhaps something internal
>         to the RepositoryVersion.  The repository version would not be
>         marked "complete" until the last plugin (Remote) sync has
>         succeeded.  More complicated than #2 but results in only
>         creating truly complete versions or nothing.  No idea how this
>         would work with current REST API whereby plugins provide sync
>         endpoints.
>
>         Option #2: Sync would be orchestrated by core or the user so
>         that multiple plugins (Remotes) create successive repository
>         versions.  For example: the RPM plugin (Remote) and the
>         Kickstart Tree plugin (Remote) would both be sync'd against
>         the same remote repository that is a composition including
>         both types.  The intermediate versions would be incomplete.
>         Only the last version contains the fully sync'd composition. 
>         This approach can be supported by core today :) but will
>         produce incomplete repository versions that are marked
>         complete=True.  This /seems/ undesirable, right?  This may not
>         be a problem for distribution since I would imaging that only
>         the last (fully composed) version would be published.  But
>         what about other usages of the repository's "latest" version?
>
>         Option #3: requires a plugin to be aware of specific
>         repository composition(s); other plugins and creates a code
>         dependency between plugins. For example, the RPM plugin could
>         delegate ISOs to the File plugin and Kickstart Trees to the
>         KickStart Tree plugin.
>
>         For all options, plugins (Remotes) need to limit sync to
>         affect only those content types within their domain.  For
>         example, the RPM (Remote) sync cannot add/remove ISO or KS Trees.
>
>         I am an advocate of some from of options #1 or #2.  Combining
>         plugins (Remotes) as needed to deal with arbitrary
>         combinations within remote repositories seems very powerful;
>         does not impose complexity on plugin writers; and does not
>         introduce code dependencies between plugins.
>
>         Thoughts?
>
>         _______________________________________________
>         Pulp-dev mailing list
>         Pulp-dev at redhat.com <mailto:Pulp-dev at redhat.com>
>         https://www.redhat.com/mailman/listinfo/pulp-dev
>
>
>     _______________________________________________
>     Pulp-dev mailing list
>     Pulp-dev at redhat.com <mailto:Pulp-dev at redhat.com>
>     https://www.redhat.com/mailman/listinfo/pulp-dev
>

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/pulp-dev/attachments/20180515/7c24cfbe/attachment.htm>