[Freeipa-devel] [DESIGN] Text-based rules for CSR autogeneration using Jinja2

[Jan Cholasta]

On 21.7.2016 17:43, Petr Spacek wrote:
> On 20.7.2016 19:25, Ben Lipton wrote:
>> On 07/20/2016 12:21 PM, Simo Sorce wrote:
>>> On Wed, 2016-07-20 at 12:14 -0400, Ben Lipton wrote:
>>>> On 07/20/2016 10:37 AM, Simo Sorce wrote:
>>>>> On Wed, 2016-07-20 at 10:17 -0400, Ben Lipton wrote:
>>>>>> On 07/20/2016 06:27 AM, Simo Sorce wrote:
>>>>>>> On Tue, 2016-07-19 at 16:20 -0400, Ben Lipton wrote:
>>>>>>>> Hi,
>>>>>>>>
>>>>>>>> I have updated the design page
>>>>>>>> http://www.freeipa.org/page/V4/Automatic_Certificate_Request_
>>>>>>>> Gene
>>>>>>>> rati
>>>>>>>> on/Mapping_Rules
>>>>>>>> with my plan for implementing user-configurable rules for
>>>>>>>> mapping
>>>>>>>> IPA
>>>>>>>> data into certificate requests. In brief: we will use Jinja2
>>>>>>>> for
>>>>>>>> templating. Data rules (which map individual data items) and
>>>>>>>> syntax
>>>>>>>> rules (which group them into certificate fields) will both be
>>>>>>>> snippets
>>>>>>>> of Jinja2 markup. The formatting process will be as follows:
>
> I've finally found some time to read the sub-page Mapping_Rules and for me it
> is kind of hard to follow. It would not be understandable without this e-mail
> and the blog posts (BTW the blog articles are among best I have seen!).
>
> Most importantly, the explanations in brackets above ["Data rules (which map
> individual data items) and (which group them into certificate fields)"] are
> missing in the wiki page itself :-)
>
> Could you fold relevant parts of the e-mails and blogs back into the wiki page
> so it is self-contained?
>
> Besides this nit,
> http://www.freeipa.org/page/V4/Automatic_Certificate_Request_Generation/Mapping_Rules#Planned_implementation
> sounds reasonable. I like how it prevents bad data from template-injection.
>
> Regarding
> http://www.freeipa.org/page/V4/Automatic_Certificate_Request_Generation/Schema#Option_A
> , I prefer Option A with separate object for each helper. It is somehow
> cleaner and it might be useful to use distinct object classes for each helper etc.
>
>
> API for ipa cert-get-requestdata sounds good.
> API for ipa cert-request makes sense to me as well.
>
> In any case I would recommend you to consult API design with Jan Cholasta
> <jcholast at redhat.com> - he is our API custodian.

See the other thread for my comments: 
<https://www.redhat.com/archives/freeipa-devel/2016-July/msg00321.html>.

>
>
> BTW I very much like "Alternatives considered" sections, we should have this
> for each design!
>
> Good work, I really like the dutiful analysis!
>
>
>
>>>>>>>> 1. Syntax rules will be rendered using Jinja2. Data rules
>>>>>>>> (rule
>>>>>>>> text,
>>>>>>>> not rendered) will be passed as the datarules attribute.
>>>>>>>> 2. Rendered syntax rules will be processed by the Formatter
>>>>>>>> class
>>>>>>>> for
>>>>>>>> the selected CSR generation helper (e.g. openssl or
>>>>>>>> certutil).
>>>>>>>> The
>>>>>>>> formatter combines these partial rules into a full template
>>>>>>>> for
>>>>>>>> the
>>>>>>>> config.
>>>>>>>> 3. The template will be rendered using Jinja2. Relevant data
>>>>>>>> from
>>>>>>>> the
>>>>>>>> IPA database will be available in the context for this
>>>>>>>> rendering.
>>>>>>>> 4. The final rendered template will be returned to the
>>>>>>>> caller,
>>>>>>>> labeled
>>>>>>>> with its function (e.g. a command line or a config file).
>>>>>>>>
>>>>>>>> Are there any comments or objections to this approach? Here's
>>>>>>>> an
>>>>>>>> example
>>>>>>>> to show what it might look like in practice.
>>>>>>>>
>>>>>>>> Example data rules:
>>>>>>>> email={{subject.email}}
>>>>>>>> O={{config.ipacertificatesubjectbase}}\nCN={{subject.username
>>>>>>>> }}
>>>>>>>>
>>>>>>>> Example syntax rule:
>>>>>>>> subjectAltName=@{% section %}{{datarules|join('\n')}}{%
>>>>>>>> endsection %}
>>>>>>>>
>>>>>>>> Example composed config template:
>>>>>>>> [ req ]
>>>>>>>> prompt = no
>>>>>>>> encrypt_key = no
>>>>>>>>
>>>>>>>> distinguished_name = {% section
>>>>>>>> %}O={{config.ipacertificatesubjectbase}}
>>>>>>>> CN={{subject.username}}{% endsection %}
>>>>>>>>
>>>>>>>> req_extensions = exts
>>>>>>>>
>>>>>>>> [ exts ]
>>>>>>>> subjectAltName=@{% section %}email={{subject.email}}{%
>>>>>>>> endsection
>>>>>>>> %}
>>>>>>>>
>>>>>>>> There's a lot more information about the thinking behind this
>>>>>>>> at
>>>>>>>> http://blog.benjaminlipton.com/2016/07/19/csr-generation-temp
>>>>>>>> lati
>>>>>>>> ng.h
>>>>>>>> tml
>>>>>>>> if you're interested, as well.
>>>>>>> Nice work Ben,
>>>>>>> it's been really nice to be able to follow your notes on the
>>>>>>> blog
>>>>>>> post,
>>>>>>> one question remains lingering in my head, why jinja2 ?
>>>>>>> I know that engine relatively well as I used it in ipsilon, so
>>>>>>> I am
>>>>>>> not
>>>>>>> questioning the choice just asking why specifically jinja2 and
>>>>>>> not
>>>>>>> something else, potentially language agnostic.
>>>>>>>
>>>>>>> Simo.
>>>>>> Honestly, my reasoning didn't go very far beyond that it seems to
>>>>>> be
>>>>>> widely used and is compatible with python, which is the language
>>>>>> where
>>>>>> the implementation is taking place (in the IPA RPC server). I
>>>>>> thought
>>>>>> about using the built-in python format strings or creating a
>>>>>> simple
>>>>>> domain-specific language, but the likelihood of wanting the
>>>>>> built-in
>>>>>> text processing features (join, replace, maybe even for loops)
>>>>>> seemed
>>>>>> high, and I didn't want to reimplement those features.
>>>>>>
>>>>>> Will the additional package dependency be a problem?
>>>>> I am more concerned a out the ability to process the data (which I
>>>>> guess is stored in LDAP) by another client, or in the CLI.
>>>>> Other than that the dependency does not concern me too much
>>>>> provided
>>>>> jinja2 templating is stable and has some guarantee that it will be
>>>>> supportable long term.
>>>>>
>>>>> If that is not guaranteed it is a problem, we cannot easily swap
>>>>> out
>>>>> one language for another once data is stored and used by the
>>>>> server.
>>>>> So the most important consideration for me is whether we are
>>>>> locking
>>>>> ourselves into something that will be hard to deal with later or
>>>>> not.
>>>>>
>>>>> Should the jinja2 project fail by the wayside next year would we be
>>>>> able to easily replace it with another engine without changing the
>>>>> templates as stored ?
>>>>>
>>>>> Simo.
>>>>>
>>>> Ah, ok, I understand the concern. For now, the plan is that the
>>>> server
>>>> will do all the text processing, so I don't really forsee a need for
>>>> any
>>>> other client to read the mapping rules from LDAP. However, it's true
>>>> that templates written in jinja2 would probably need at least minor
>>>> changes to be compatible with another templating engine. (Same goes
>>>> for
>>>> any other choice - a lot of these engines seem to have very similar,
>>>> but
>>>> not exactly compatible, syntax). I don't really know how to judge
>>>> the
>>>> long-term viability of the jinja2 project, though it seems to be
>>>> recognized by lots of projects (ansible[1], openstack[2], flask[3],
>>>> even
>>>> django[4] which has its own templating engine).
>
> Other big users are exactly the reason why I consider Jinja2 to be a good
> choice. After all, if Jinja2 dies upstream we will not be alone :-)
>
> Petr^2 Spacek
>
>>>> In any case, if the team prefers it, I'd be comfortable going with a
>>>> more minimal DSL that only has the features we know we need. It
>>>> might
>>>> slightly limit the types of certs that can be generated, but that can
>>>> be
>>>> iterated on. But it would be another thing to design, build and
>>>> maintain. Let me know what you think.
>>> I am ok using jinja2 as long as we realize we may be on the hook for
>>> maintaining it ourselves in the long term. It's probably easier to do
>>> that than to write our own anyway.
>>>
>>> Simo.
>> It might also be worth pointing out that although the examples in the document
>> are based on jinja2, the overall approach should be pretty compatible with
>> other (existing or custom) template languages. So we're not locked into jinja2
>> in that way, it's just the rules in deployed databases that would be tough to
>> convert.


-- 
Jan Cholasta