[Freeipa-devel] [DESIGN] Text-based rules for CSR autogeneration using Jinja2

[Ben Lipton]

On 07/20/2016 12:21 PM, Simo Sorce wrote:
> On Wed, 2016-07-20 at 12:14 -0400, Ben Lipton wrote:
>> On 07/20/2016 10:37 AM, Simo Sorce wrote:
>>> On Wed, 2016-07-20 at 10:17 -0400, Ben Lipton wrote:
>>>> On 07/20/2016 06:27 AM, Simo Sorce wrote:
>>>>> On Tue, 2016-07-19 at 16:20 -0400, Ben Lipton wrote:
>>>>>> Hi,
>>>>>>
>>>>>> I have updated the design page
>>>>>> http://www.freeipa.org/page/V4/Automatic_Certificate_Request_
>>>>>> Gene
>>>>>> rati
>>>>>> on/Mapping_Rules
>>>>>> with my plan for implementing user-configurable rules for
>>>>>> mapping
>>>>>> IPA
>>>>>> data into certificate requests. In brief: we will use Jinja2
>>>>>> for
>>>>>> templating. Data rules (which map individual data items) and
>>>>>> syntax
>>>>>> rules (which group them into certificate fields) will both be
>>>>>> snippets
>>>>>> of Jinja2 markup. The formatting process will be as follows:
>>>>>> 1. Syntax rules will be rendered using Jinja2. Data rules
>>>>>> (rule
>>>>>> text,
>>>>>> not rendered) will be passed as the datarules attribute.
>>>>>> 2. Rendered syntax rules will be processed by the Formatter
>>>>>> class
>>>>>> for
>>>>>> the selected CSR generation helper (e.g. openssl or
>>>>>> certutil).
>>>>>> The
>>>>>> formatter combines these partial rules into a full template
>>>>>> for
>>>>>> the
>>>>>> config.
>>>>>> 3. The template will be rendered using Jinja2. Relevant data
>>>>>> from
>>>>>> the
>>>>>> IPA database will be available in the context for this
>>>>>> rendering.
>>>>>> 4. The final rendered template will be returned to the
>>>>>> caller,
>>>>>> labeled
>>>>>> with its function (e.g. a command line or a config file).
>>>>>>
>>>>>> Are there any comments or objections to this approach? Here's
>>>>>> an
>>>>>> example
>>>>>> to show what it might look like in practice.
>>>>>>
>>>>>> Example data rules:
>>>>>> email={{subject.email}}
>>>>>> O={{config.ipacertificatesubjectbase}}\nCN={{subject.username
>>>>>> }}
>>>>>>
>>>>>> Example syntax rule:
>>>>>> subjectAltName=@{% section %}{{datarules|join('\n')}}{%
>>>>>> endsection %}
>>>>>>
>>>>>> Example composed config template:
>>>>>> [ req ]
>>>>>> prompt = no
>>>>>> encrypt_key = no
>>>>>>
>>>>>> distinguished_name = {% section
>>>>>> %}O={{config.ipacertificatesubjectbase}}
>>>>>> CN={{subject.username}}{% endsection %}
>>>>>>
>>>>>> req_extensions = exts
>>>>>>
>>>>>> [ exts ]
>>>>>> subjectAltName=@{% section %}email={{subject.email}}{%
>>>>>> endsection
>>>>>> %}
>>>>>>
>>>>>> There's a lot more information about the thinking behind this
>>>>>> at
>>>>>> http://blog.benjaminlipton.com/2016/07/19/csr-generation-temp
>>>>>> lati
>>>>>> ng.h
>>>>>> tml
>>>>>> if you're interested, as well.
>>>>> Nice work Ben,
>>>>> it's been really nice to be able to follow your notes on the
>>>>> blog
>>>>> post,
>>>>> one question remains lingering in my head, why jinja2 ?
>>>>> I know that engine relatively well as I used it in ipsilon, so
>>>>> I am
>>>>> not
>>>>> questioning the choice just asking why specifically jinja2 and
>>>>> not
>>>>> something else, potentially language agnostic.
>>>>>
>>>>> Simo.
>>>> Honestly, my reasoning didn't go very far beyond that it seems to
>>>> be
>>>> widely used and is compatible with python, which is the language
>>>> where
>>>> the implementation is taking place (in the IPA RPC server). I
>>>> thought
>>>> about using the built-in python format strings or creating a
>>>> simple
>>>> domain-specific language, but the likelihood of wanting the
>>>> built-in
>>>> text processing features (join, replace, maybe even for loops)
>>>> seemed
>>>> high, and I didn't want to reimplement those features.
>>>>
>>>> Will the additional package dependency be a problem?
>>> I am more concerned a out the ability to process the data (which I
>>> guess is stored in LDAP) by another client, or in the CLI.
>>> Other than that the dependency does not concern me too much
>>> provided
>>> jinja2 templating is stable and has some guarantee that it will be
>>> supportable long term.
>>>
>>> If that is not guaranteed it is a problem, we cannot easily swap
>>> out
>>> one language for another once data is stored and used by the
>>> server.
>>> So the most important consideration for me is whether we are
>>> locking
>>> ourselves into something that will be hard to deal with later or
>>> not.
>>>
>>> Should the jinja2 project fail by the wayside next year would we be
>>> able to easily replace it with another engine without changing the
>>> templates as stored ?
>>>
>>> Simo.
>>>
>> Ah, ok, I understand the concern. For now, the plan is that the
>> server
>> will do all the text processing, so I don't really forsee a need for
>> any
>> other client to read the mapping rules from LDAP. However, it's true
>> that templates written in jinja2 would probably need at least minor
>> changes to be compatible with another templating engine. (Same goes
>> for
>> any other choice - a lot of these engines seem to have very similar,
>> but
>> not exactly compatible, syntax). I don't really know how to judge
>> the
>> long-term viability of the jinja2 project, though it seems to be
>> recognized by lots of projects (ansible[1], openstack[2], flask[3],
>> even
>> django[4] which has its own templating engine).
>>
>> In any case, if the team prefers it, I'd be comfortable going with a
>> more minimal DSL that only has the features we know we need. It
>> might
>> slightly limit the types of certs that can be generated, but that can
>> be
>> iterated on. But it would be another thing to design, build and
>> maintain. Let me know what you think.
> I am ok using jinja2 as long as we realize we may be on the hook for
> maintaining it ourselves in the long term. It's probably easier to do
> that than to write our own anyway.
>
> Simo.
It might also be worth pointing out that although the examples in the 
document are based on jinja2, the overall approach should be pretty 
compatible with other (existing or custom) template languages. So we're 
not locked into jinja2 in that way, it's just the rules in deployed 
databases that would be tough to convert.

Thanks for reading and commenting!

Ben