[Pulp-dev] proposal: global importer settings

[Michael Hrivnak]

Double-bump. Thanks in advance for your help getting this wrapped up!

On Tue, Nov 14, 2017 at 1:32 PM, Michael Hrivnak <mhrivnak at redhat.com>
wrote:

> Bump. Any more questions on this? I'd like to get something agreed on and
> settled.
>
> Thanks!
>
>
>
> On Fri, Nov 3, 2017 at 1:28 PM, Michael Hrivnak <mhrivnak at redhat.com>
> wrote:
>
>>
>>
>> On Fri, Nov 3, 2017 at 11:58 AM, Jeff Ortel <jortel at redhat.com> wrote:
>>
>>>
>>>
>>> On 11/02/2017 10:25 AM, Michael Hrivnak wrote:
>>> > I've been working on a planning task for how Pulp 3 will handle global
>>> importer settings. As part of that,
>>> > I've collected feedback from a number of stakeholders. You can view
>>> the planning task here:
>>> >
>>> > https://pulp.plan.io/issues/2373
>>> >
>>> > The aspect that everyone seems to agree on is that proxies should be
>>> configured once in one place, and that
>>> > other download-related settings are a good fit for the same mechanism.
>>> I wrote up a story for that here, and I
>>> > would appreciate feedback and grooming:
>>> >
>>> > https://pulp.plan.io/issues/3108
>>> >
>>> > There is much less clarity around other settings. This is because most
>>> other settings that we would consider
>>> > for global scope would need the ability to be overridden at the
>>> individual importer level. That multi-layered
>>> > approach where an individual importer's settings take precedence over
>>> the global settings is how Pulp 2 works,
>>> > but it is not generally liked. Katello for example only uses that
>>> feature in Pulp 2 to define download-related
>>> > settings; proxy, concurrency, and bandwidth limits. Many stakeholders
>>> expressed concern about retaining a
>>> > multi-layered approach to configuring importers. Doing so also would
>>> add substantial complexity to how Pulp 3
>>> > implements settings, and it adds complexity to the user experience.
>>>
>>> Can you elaborate on what pulp2 users don't like about the
>>> "multi-layered" approach in pulp2?
>>>
>>> IIRC, pulp2 only supports the "override config" thing (which I agree,
>>> few users like) and not something
>>> persistent.  I suspect that users don't like the pulp2 implementation
>>> but may find the more sophisticated
>>> profile concept appealing even with a precedence model.  Not supporting
>>> a model where the importer attributes
>>> take precedence over the profile will likely limit the properties in the
>>> profile to just the proxy.
>>>
>>> This could be a more powerful feature.  I'm imagining cases where users
>>> would find it useful for the profile
>>> to include properties beyond proxy URL (or other download properties).
>>> Perhaps even the download or sync
>>> policies.  Who knows.
>>>
>>> I do recognize that we really only have a concrete use case for the
>>> proxy but seems short sighted to design
>>> what seems to be intended as a more general concept that would be so
>>> limited.
>>>
>>> I don't perceive a precedence model as being particularly complex.
>>
>>
>> Pulp 2 has a three-layer approach. You can define importer settings
>> globally in a json file on disk, on an importer record in the DB, and in an
>> override config passed at the time of requesting an individual sync. That
>> contributes to it being difficult to know what settings actually got used
>> for any particular sync. And since the same setting can be set in multiple
>> places, users are sometimes surprised that a sync doesn't behave the way
>> they expected.
>>
>> I think there is a place for having global importer settings that
>> influence how the imports happen. The download policy (immediate,
>> on-demand, etc) is a good example of something a user may want to set
>> globally. I think we could figure out a reasonable 2-layer approach to
>> doing that. But since importers are a master/detail model, that introduces
>> some complexity. And I do not see any particular reason we would need to
>> deliver this feature in 3.0.
>>
>> Download settings are different, because I think the best way to model
>> them is in one place without any layering. And from a use-case standpoint,
>> users will logically group download settings differently than importer
>> settings. For example:
>>
>> I want to use a proxy for external traffic, a different one for traffic
>> to a data center, and none for internal traffic. It's all about physical
>> network characteristics, and not at all about content and repos.
>>
>> I want to set a download policy, or other importer settings, based on
>> other factors. Certain vendors, or certain products/repos within a specific
>> vendor, should get the same download policy based on things like how likely
>> I think it is that I'll need all the content in the repo, how reliable I
>> think that vendor will be at providing it on-demand, whether I expect to
>> test it up-front, etc.
>>
>> So based on that, I think these are two different kinds of data that we
>> should handle separately. We can make a DownloadProfile with a well-defined
>> purpose, and that has the single source of data for any attributes it
>> stores. Later we could figure out how to make importer settings
>> configurable globally by adding a second layer, enabling master/detail with
>> that layer, etc.
>>
>> --
>>
>> Michael Hrivnak
>>
>> Principal Software Engineer, RHCE
>>
>> Red Hat
>>
>
>
>
> --
>
> Michael Hrivnak
>
> Principal Software Engineer, RHCE
>
> Red Hat
>



-- 

Michael Hrivnak

Principal Software Engineer, RHCE

Red Hat
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/pulp-dev/attachments/20171127/6b3d88f9/attachment.htm>