[Pulp-dev] proposal: global importer settings

Michael Hrivnak mhrivnak at redhat.com
Fri Nov 3 17:28:27 UTC 2017

On Fri, Nov 3, 2017 at 11:58 AM, Jeff Ortel <jortel at redhat.com> wrote:

> On 11/02/2017 10:25 AM, Michael Hrivnak wrote:
> > I've been working on a planning task for how Pulp 3 will handle global
> importer settings. As part of that,
> > I've collected feedback from a number of stakeholders. You can view the
> planning task here:
> >
> > https://pulp.plan.io/issues/2373
> >
> > The aspect that everyone seems to agree on is that proxies should be
> configured once in one place, and that
> > other download-related settings are a good fit for the same mechanism. I
> wrote up a story for that here, and I
> > would appreciate feedback and grooming:
> >
> > https://pulp.plan.io/issues/3108
> >
> > There is much less clarity around other settings. This is because most
> other settings that we would consider
> > for global scope would need the ability to be overridden at the
> individual importer level. That multi-layered
> > approach where an individual importer's settings take precedence over
> the global settings is how Pulp 2 works,
> > but it is not generally liked. Katello for example only uses that
> feature in Pulp 2 to define download-related
> > settings; proxy, concurrency, and bandwidth limits. Many stakeholders
> expressed concern about retaining a
> > multi-layered approach to configuring importers. Doing so also would add
> substantial complexity to how Pulp 3
> > implements settings, and it adds complexity to the user experience.
> Can you elaborate on what pulp2 users don't like about the "multi-layered"
> approach in pulp2?
> IIRC, pulp2 only supports the "override config" thing (which I agree, few
> users like) and not something
> persistent.  I suspect that users don't like the pulp2 implementation but
> may find the more sophisticated
> profile concept appealing even with a precedence model.  Not supporting a
> model where the importer attributes
> take precedence over the profile will likely limit the properties in the
> profile to just the proxy.
> This could be a more powerful feature.  I'm imagining cases where users
> would find it useful for the profile
> to include properties beyond proxy URL (or other download properties).
> Perhaps even the download or sync
> policies.  Who knows.
> I do recognize that we really only have a concrete use case for the proxy
> but seems short sighted to design
> what seems to be intended as a more general concept that would be so
> limited.
> I don't perceive a precedence model as being particularly complex.

Pulp 2 has a three-layer approach. You can define importer settings
globally in a json file on disk, on an importer record in the DB, and in an
override config passed at the time of requesting an individual sync. That
contributes to it being difficult to know what settings actually got used
for any particular sync. And since the same setting can be set in multiple
places, users are sometimes surprised that a sync doesn't behave the way
they expected.

I think there is a place for having global importer settings that influence
how the imports happen. The download policy (immediate, on-demand, etc) is
a good example of something a user may want to set globally. I think we
could figure out a reasonable 2-layer approach to doing that. But since
importers are a master/detail model, that introduces some complexity. And I
do not see any particular reason we would need to deliver this feature in

Download settings are different, because I think the best way to model them
is in one place without any layering. And from a use-case standpoint, users
will logically group download settings differently than importer settings.
For example:

I want to use a proxy for external traffic, a different one for traffic to
a data center, and none for internal traffic. It's all about physical
network characteristics, and not at all about content and repos.

I want to set a download policy, or other importer settings, based on other
factors. Certain vendors, or certain products/repos within a specific
vendor, should get the same download policy based on things like how likely
I think it is that I'll need all the content in the repo, how reliable I
think that vendor will be at providing it on-demand, whether I expect to
test it up-front, etc.

So based on that, I think these are two different kinds of data that we
should handle separately. We can make a DownloadProfile with a well-defined
purpose, and that has the single source of data for any attributes it
stores. Later we could figure out how to make importer settings
configurable globally by adding a second layer, enabling master/detail with
that layer, etc.


Michael Hrivnak

Principal Software Engineer, RHCE

Red Hat
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/pulp-dev/attachments/20171103/a6212f86/attachment.htm>

More information about the Pulp-dev mailing list