[Freeipa-devel] [DESIGN] Text-based rules for CSR autogeneration using Jinja2

[Petr Spacek]

On 20.7.2016 19:25, Ben Lipton wrote:
> On 07/20/2016 12:21 PM, Simo Sorce wrote:
>> On Wed, 2016-07-20 at 12:14 -0400, Ben Lipton wrote:
>>> On 07/20/2016 10:37 AM, Simo Sorce wrote:
>>>> On Wed, 2016-07-20 at 10:17 -0400, Ben Lipton wrote:
>>>>> On 07/20/2016 06:27 AM, Simo Sorce wrote:
>>>>>> On Tue, 2016-07-19 at 16:20 -0400, Ben Lipton wrote:
>>>>>>> Hi,
>>>>>>>
>>>>>>> I have updated the design page
>>>>>>> http://www.freeipa.org/page/V4/Automatic_Certificate_Request_
>>>>>>> Gene
>>>>>>> rati
>>>>>>> on/Mapping_Rules
>>>>>>> with my plan for implementing user-configurable rules for
>>>>>>> mapping
>>>>>>> IPA
>>>>>>> data into certificate requests. In brief: we will use Jinja2
>>>>>>> for
>>>>>>> templating. Data rules (which map individual data items) and
>>>>>>> syntax
>>>>>>> rules (which group them into certificate fields) will both be
>>>>>>> snippets
>>>>>>> of Jinja2 markup. The formatting process will be as follows:

I've finally found some time to read the sub-page Mapping_Rules and for me it
is kind of hard to follow. It would not be understandable without this e-mail
and the blog posts (BTW the blog articles are among best I have seen!).

Most importantly, the explanations in brackets above ["Data rules (which map
individual data items) and (which group them into certificate fields)"] are
missing in the wiki page itself :-)

Could you fold relevant parts of the e-mails and blogs back into the wiki page
so it is self-contained?

Besides this nit,
http://www.freeipa.org/page/V4/Automatic_Certificate_Request_Generation/Mapping_Rules#Planned_implementation
sounds reasonable. I like how it prevents bad data from template-injection.

Regarding
http://www.freeipa.org/page/V4/Automatic_Certificate_Request_Generation/Schema#Option_A
, I prefer Option A with separate object for each helper. It is somehow
cleaner and it might be useful to use distinct object classes for each helper etc.


API for ipa cert-get-requestdata sounds good.
API for ipa cert-request makes sense to me as well.

In any case I would recommend you to consult API design with Jan Cholasta
<jcholast at redhat.com> - he is our API custodian.


BTW I very much like "Alternatives considered" sections, we should have this
for each design!

Good work, I really like the dutiful analysis!



>>>>>>> 1. Syntax rules will be rendered using Jinja2. Data rules
>>>>>>> (rule
>>>>>>> text,
>>>>>>> not rendered) will be passed as the datarules attribute.
>>>>>>> 2. Rendered syntax rules will be processed by the Formatter
>>>>>>> class
>>>>>>> for
>>>>>>> the selected CSR generation helper (e.g. openssl or
>>>>>>> certutil).
>>>>>>> The
>>>>>>> formatter combines these partial rules into a full template
>>>>>>> for
>>>>>>> the
>>>>>>> config.
>>>>>>> 3. The template will be rendered using Jinja2. Relevant data
>>>>>>> from
>>>>>>> the
>>>>>>> IPA database will be available in the context for this
>>>>>>> rendering.
>>>>>>> 4. The final rendered template will be returned to the
>>>>>>> caller,
>>>>>>> labeled
>>>>>>> with its function (e.g. a command line or a config file).
>>>>>>>
>>>>>>> Are there any comments or objections to this approach? Here's
>>>>>>> an
>>>>>>> example
>>>>>>> to show what it might look like in practice.
>>>>>>>
>>>>>>> Example data rules:
>>>>>>> email={{subject.email}}
>>>>>>> O={{config.ipacertificatesubjectbase}}\nCN={{subject.username
>>>>>>> }}
>>>>>>>
>>>>>>> Example syntax rule:
>>>>>>> subjectAltName=@{% section %}{{datarules|join('\n')}}{%
>>>>>>> endsection %}
>>>>>>>
>>>>>>> Example composed config template:
>>>>>>> [ req ]
>>>>>>> prompt = no
>>>>>>> encrypt_key = no
>>>>>>>
>>>>>>> distinguished_name = {% section
>>>>>>> %}O={{config.ipacertificatesubjectbase}}
>>>>>>> CN={{subject.username}}{% endsection %}
>>>>>>>
>>>>>>> req_extensions = exts
>>>>>>>
>>>>>>> [ exts ]
>>>>>>> subjectAltName=@{% section %}email={{subject.email}}{%
>>>>>>> endsection
>>>>>>> %}
>>>>>>>
>>>>>>> There's a lot more information about the thinking behind this
>>>>>>> at
>>>>>>> http://blog.benjaminlipton.com/2016/07/19/csr-generation-temp
>>>>>>> lati
>>>>>>> ng.h
>>>>>>> tml
>>>>>>> if you're interested, as well.
>>>>>> Nice work Ben,
>>>>>> it's been really nice to be able to follow your notes on the
>>>>>> blog
>>>>>> post,
>>>>>> one question remains lingering in my head, why jinja2 ?
>>>>>> I know that engine relatively well as I used it in ipsilon, so
>>>>>> I am
>>>>>> not
>>>>>> questioning the choice just asking why specifically jinja2 and
>>>>>> not
>>>>>> something else, potentially language agnostic.
>>>>>>
>>>>>> Simo.
>>>>> Honestly, my reasoning didn't go very far beyond that it seems to
>>>>> be
>>>>> widely used and is compatible with python, which is the language
>>>>> where
>>>>> the implementation is taking place (in the IPA RPC server). I
>>>>> thought
>>>>> about using the built-in python format strings or creating a
>>>>> simple
>>>>> domain-specific language, but the likelihood of wanting the
>>>>> built-in
>>>>> text processing features (join, replace, maybe even for loops)
>>>>> seemed
>>>>> high, and I didn't want to reimplement those features.
>>>>>
>>>>> Will the additional package dependency be a problem?
>>>> I am more concerned a out the ability to process the data (which I
>>>> guess is stored in LDAP) by another client, or in the CLI.
>>>> Other than that the dependency does not concern me too much
>>>> provided
>>>> jinja2 templating is stable and has some guarantee that it will be
>>>> supportable long term.
>>>>
>>>> If that is not guaranteed it is a problem, we cannot easily swap
>>>> out
>>>> one language for another once data is stored and used by the
>>>> server.
>>>> So the most important consideration for me is whether we are
>>>> locking
>>>> ourselves into something that will be hard to deal with later or
>>>> not.
>>>>
>>>> Should the jinja2 project fail by the wayside next year would we be
>>>> able to easily replace it with another engine without changing the
>>>> templates as stored ?
>>>>
>>>> Simo.
>>>>
>>> Ah, ok, I understand the concern. For now, the plan is that the
>>> server
>>> will do all the text processing, so I don't really forsee a need for
>>> any
>>> other client to read the mapping rules from LDAP. However, it's true
>>> that templates written in jinja2 would probably need at least minor
>>> changes to be compatible with another templating engine. (Same goes
>>> for
>>> any other choice - a lot of these engines seem to have very similar,
>>> but
>>> not exactly compatible, syntax). I don't really know how to judge
>>> the
>>> long-term viability of the jinja2 project, though it seems to be
>>> recognized by lots of projects (ansible[1], openstack[2], flask[3],
>>> even
>>> django[4] which has its own templating engine).

Other big users are exactly the reason why I consider Jinja2 to be a good
choice. After all, if Jinja2 dies upstream we will not be alone :-)

Petr^2 Spacek

>>> In any case, if the team prefers it, I'd be comfortable going with a
>>> more minimal DSL that only has the features we know we need. It
>>> might
>>> slightly limit the types of certs that can be generated, but that can
>>> be
>>> iterated on. But it would be another thing to design, build and
>>> maintain. Let me know what you think.
>> I am ok using jinja2 as long as we realize we may be on the hook for
>> maintaining it ourselves in the long term. It's probably easier to do
>> that than to write our own anyway.
>>
>> Simo.
> It might also be worth pointing out that although the examples in the document
> are based on jinja2, the overall approach should be pretty compatible with
> other (existing or custom) template languages. So we're not locked into jinja2
> in that way, it's just the rules in deployed databases that would be tough to
> convert.
-- 
Petr^2 Spacek