[Pulp-dev] Composed Repositories

Tue May 15 14:36:59 UTC 2018

On 05/15/2018 05:26 AM, Ina Panova wrote:
> +1 on not introducing dependencies between plugins.
>
> What will be the behavior in case there is a composed repo of rpm and 
> ks trees but just the rpm plugin is installed?

I would expect the result would be to only sync the rpm content into the 
pulp repository.

> Do we fail and say we cannot sync this repo at all or we just sync the 
> rpm part?

No, I think it would be expected to succeed since the user has only 
installed the rpm plugin and requested that only rpm content be sync'd.  
The remote repository is composed of multiple content types out of 
convenience for managing the content.  Pulp should not be bound to the 
organization of remote repositories.

>
> Depends how we plan this ^ i guess we'll decide which option 1 or 2 
> fits better.
>
> Don't want to go wild, but what if notion of composed repos will be so 
> popular in the future that's its amount will increase? I think we do 
> want to at least partially being able to sync it and not take the 
> approach all or nothing?
>
> #2 speaks to me more for now.

#2 will create repository version with partial content which are 
complete=True.  Given users can choose which version to publish, do you 
see this as a problem.  What about cases where the "latest" version is, 
at times, partial?

>
>
>
>
> --------
> Regards,
>
> Ina Panova
> Software Engineer| Pulp| Red Hat Inc.
>
> "Do not go where the path may lead,
>  go instead where there is no path and leave a trail."
>
> On Mon, May 14, 2018 at 9:44 PM, Jeff Ortel <jortel at redhat.com 
> <mailto:jortel at redhat.com>> wrote:
>
>     Let's brainstorm on something.
>
>     Pulp needs to deal with remote repositories that are composed of
>     multiple content types which may span the domain of a single
>     plugin.  Here are a few examples. Some Red Hat RPM repositories
>     are composed of: RPMs, DRPMs, , ISOs and Kickstart Trees.  Some
>     OSTree repositories are composed of OSTrees & Kickstart Trees.
>     This raises a question:
>
>     How can pulp3 best support syncing with remote repositories that
>     are composed of multiple (unrelated) content types in a way that
>     doesn't result in plugins duplicating support for content types?
>
>     Few approaches come to mind:
>
>     1. Multiple plugins (Remotes) participate in the sync flow to
>     produce a new repository version.
>     2. Multiple plugins (Remotes) are sync'd successively each
>     producing a new version of a repository.  Only the last version
>     contains the fully sync'd composition.
>     3. Plugins share code.
>     4. Other?
>
>
>     Option #1: Sync would be orchestrated by core or the user so that
>     multiple plugins (Remotes) participate in populating a new
>     repository version.  For example: the RPM plugin (Remote) and the
>     Kickstart Tree plugin (Remote) would both be sync'd against the
>     same remote repository that is composed of both types.  The new
>     repository version would be composed of the result of both plugin
>     (Remote) syncs.  To support this, we'd need to provide a way for
>     each plugin to operate seamlessly on the same (new) repository
>     version.  Perhaps something internal to the RepositoryVersion. 
>     The repository version would not be marked "complete" until the
>     last plugin (Remote) sync has succeeded.  More complicated than #2
>     but results in only creating truly complete versions or nothing. 
>     No idea how this would work with current REST API whereby plugins
>     provide sync endpoints.
>
>     Option #2: Sync would be orchestrated by core or the user so that
>     multiple plugins (Remotes) create successive repository versions. 
>     For example: the RPM plugin (Remote) and the Kickstart Tree plugin
>     (Remote) would both be sync'd against the same remote repository
>     that is a composition including both types.  The intermediate
>     versions would be incomplete. Only the last version contains the
>     fully sync'd composition.  This approach can be supported by core
>     today :) but will produce incomplete repository versions that are
>     marked complete=True.  This /seems/ undesirable, right? This may
>     not be a problem for distribution since I would imaging that only
>     the last (fully composed) version would be published.  But what
>     about other usages of the repository's "latest" version?
>
>     Option #3: requires a plugin to be aware of specific repository
>     composition(s); other plugins and creates a code dependency
>     between plugins.  For example, the RPM plugin could delegate ISOs
>     to the File plugin and Kickstart Trees to the KickStart Tree plugin.
>
>     For all options, plugins (Remotes) need to limit sync to affect
>     only those content types within their domain. For example, the RPM
>     (Remote) sync cannot add/remove ISO or KS Trees.
>
>     I am an advocate of some from of options #1 or #2. Combining
>     plugins (Remotes) as needed to deal with arbitrary combinations
>     within remote repositories seems very powerful; does not impose
>     complexity on plugin writers; and does not introduce code
>     dependencies between plugins.
>
>     Thoughts?
>
>     _______________________________________________
>     Pulp-dev mailing list
>     Pulp-dev at redhat.com <mailto:Pulp-dev at redhat.com>
>     https://www.redhat.com/mailman/listinfo/pulp-dev
>     <https://www.redhat.com/mailman/listinfo/pulp-dev>
>
>

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/pulp-dev/attachments/20180515/18a98dca/attachment.htm>