[Pulp-list] configuration files, validation and standards

[Jeff Ortel]

Never mind.
Looks like these things have already been decided.  Purging gc_config 
and associated unit tests mentioned below.

On 05/22/2012 12:27 PM, Jeff Ortel wrote:
>
>
> On 05/22/2012 10:57 AM, Jeff Ortel wrote:
>> All,
>>
>> So, I got derailed on something yesterday (related to this, but not
>> tihs) and decided "why waste a perfectly hosed up day" and decided to
>> finish up the day on something that has been bugging me about pulp v1.
>> Jay, don't kill me .. I mostly worked on this last night ;)
>>
>> One area in which Pulp is inconsistent is in configuration file handling
>> (reading/parsing/accessing/converting). The server uses a module based
>> on ConfigParser and the client (originally) took the route of classes
>> derived from INIConfig. These subclasses represented the configuration
>> file and defined a schema used for self validation. The validation
>> provided in common/config.py.
>>
>> As we move forward in v2, I would like to consolidate on a single
>> approach. That we pass configuration as (dict) objects instead of
>> INIConfig or ConfigParser objects. Further, that we consistently
>> leverage the validation that we invested in long ago but only adopted
>> (completely) in the client. This approach has several benefits.
>>
>> 1. Configuration file parsing/formatting is independent of how
>> configuration is used by our code.
>>
>> 2. By representing/passing configuration as a dictionary (actually a
>> dict of dict), cobbling up configuration for testing is easier.
>>
>> 3. Consistent usage of (existing) configuration validation provides the
>> following benefits:
>> - define/document configuration in one place.
>> - validate and report errors in one place with consistent error
>> messages instead of splattering validation everywhere the
>> configuration(s) are used.
>> - combined usage of validation and companion type conversion functions
>> makes for safe (and easy) conversion. Eg: values for bool
>> properties are consistently verified and converted.
>>
>> 4. A dict is well documented and understood. It can be easily pickled
>> and converted to json.
>>
>> I GC content handlers (in the agent) follow this approach. When loaded,
>> the handler configuration is passed as a dictionary and the loader using
>> validation to ensure that /standard/ parts of the configuration
>> (descriptor) are valid. I would suggest the same for server plugin
>> configurations.
>>
>> I suppose Im guilty of adding yet one more way of dealing with
>> configuration files but unlike some of the one-off solutions that have
>> been popping up (I think), this is an attempt to standardize on a single
>> approach. Just something to consider as we move forward.
>>
>> --------------------
>>
>>
>> To help facilitate (and demonstrate) this approach:
>>
>> In common/gc_config.py, I added a Config class that can be used to
>> easily read INI config files into a dict (graph). It supports
>> construction with a variety of inputs that are merged together to
>> provide: a composite representation; property/section overrides; easy
>> defaults. I also converted the validation to work on dictionaries.
>> Further, I added a method that renders an object graph representation of
>> the dict (graph) for easy access using (.) dot notation for those of you
>> that liked INIConfig for this reason.
>>
>> A Few Examples:
>>
>> This just show validation and simple dict access to the configuration.
>>
>> <snip>
>> [server]
>> host=myhost
>> port=80
>> </snip>
>>
>> >>> config = Config(path)
>> >>> config.validate(schema)
>> >>> print config['server']['port']
>> 80
>>
>
> Syntatic sugar and may distract from the central idea. That being that
> the config (isa) dict.
>
> <ignore>
>
>> # graph
>> >>> config = Config(path)
>> >>> obj = config.graph()
>> >>> print obj.server.port
>> 80
>>
>> >>> print obj.server.port
>> 80
>>
>> # not defined
>>
>> >>> print obj.server.notdefined
>> None
>>
>> >>> obj = config.graph(strict=True)
>> >>> print obj.server.notdefined
>> KeyError('notdefinded')
>
> </ignore>
>
>>
>> ----
>>
>> This example shows configuration defaults (dict) that is overlaid with
>> file at path (A) and then file at path (B). The precedence defined by
>> ordering.
>>
>> defaults = { 'port' : 443 }
>>
>> pathA = /etc/pulp/admin/admin.conf
>> <snip>
>> [server]
>> host=myhost
>> </snip>
>>
>> pathB = ~/.pulp/admin.conf
>> <snip>
>> [server]
>> host=redhat.com
>> </snip>
>>
>> >>> config = Config(defaults, pathA, pathB)
>> >>> config.validate(schema)
>> server = config['server']
>> >>> print server['host']
>> redhat.com
>>
>> >>> print server['port']
>> 443
>>
>> ----
>>
>> Section filtering provided but not demonstrated.
>>
>> _______________________________________________
>> Pulp-list mailing list
>> Pulp-list at redhat.com
>> https://www.redhat.com/mailman/listinfo/pulp-list
>
> _______________________________________________
> Pulp-list mailing list
> Pulp-list at redhat.com
> https://www.redhat.com/mailman/listinfo/pulp-list