[Freeipa-devel] DNSSEC support design considerations

[Petr Spacek]

Hello,

I found that we (probably) misunderstood each other. The sky-high level 
overview of the proposal follow:

NO CHANGE:
1) LDAP stores all *unsigned* data.

2)
NO CHANGE:
a) bind-dyndb-ldap *on each server* fetches all unsigned data from LDAP and 
store them in *in memory* database (we do it now)

THE DIFFERENCE:
b) All data will be stored in BIND's native RBT-database (RBTDB) instead of 
our own in-memory database.

NEW PIECES:
3)
Mechanisms implemented in BIND's RBTDB will do DNSSEC signing etc. for us. The 
BIND's feature is called 'in-line signing' and it can do all key/signature 
maintenance for us, including periodical zone re-signing etc.


The whole point of this proposal is about code-reusage. I'm trying to avoid 
re-inventing of the wheel.

Note that DNSSEC implementation in BIND has ~ 150 kiB of C code, stand-alone 
signing utilities add another ~ 200 kiB of code (~ 7000 lines) . I really 
don't want to re-write it again when it's not reasonable.

Further comments are in-line.


On 20.5.2013 14:07, Simo Sorce wrote:
> On Wed, 2013-05-15 at 17:11 +0200, Petr Spacek wrote:
>> On 15.5.2013 10:29, Simo Sorce wrote:
>>>> I investigated various scenarios for DNSSEC integration and I would like to
>>>> hear your opinions about proposed approach and it's effects.
>>>>
>>>>
>>>> The most important finding is that bind-dyndb-ldap can't support DNSSEC
>>>> without rewrite of the 'in-memory database' component.
>>>
>>> Can you elaborate why a rewrite would be needed ? What constraint we do not meet ?
>>
>> We have three main problems - partially with data structures and mostly with
>> the way how we work with the 'internal database':
>>
>> 1) DNSSEC requires strict record ordering, i.e. each record in database has to
>> have predecessor and successor (ordering by name and then by record data).
>> This can be done relatively simply, but it requires a full dump of the database.
>>
>> 2) On-line record signing requires a lot of data stored
>> per-record+per-signature. This would require bigger effort than point 1),
>> because many data structures and respective APIs and locking protocols have to
>> be re-designed.
>>
>> 3) Our current 'internal database' acts as a 'cache', i.e. records can appear
>> and disappear dynamically and the 'cache' is not considered as authoritative
>> source of data: LDAP search is conducted each time when some data are not
>> found etc. The result is that the same data can disappear and then appear
>> again in the cache etc.
>>
>> Typical update scenario, with persistent search enabled:
>> a) DNS UPDATE from client is received by BIND
>> b) New data are written to LDAP
>> c) DN of modified object is received via persistent search
>> d) All RRs under the *updated name* are discarded from the cache
>> <-- now the cache is not consistent with data in LDAP
>> e) Object from LDAP is fetched by plugin
>> <-- a query for the updated name will enforce instant cache refresh, because
>> we know that the cache is not authoritative
>> f) All RRs in the object are updated (in cache)
>>
>> The problem is that the cache in intermediate states (between <-- marks) can't
>> be used as authoritative source and will produce incorrect signatures. The
>> text below contains more details.
>>
>> Database's in BIND has concept of 'versions' ('transactions') which our
>> internal cache do not implement ... It could be solved by proper locking, of
>> course, but it will not be a piece of cake. We need to take care of many
>> parallel updates, parallel queries and parallel re-signing at the same time.
>>
>> I don't say that it is impossible to implement our own backend with same
>> properties as BIND's database, but I don't see the value (and I can see a lot
>> of bugs :-).
>
> Well, we do not necessarily need all the same properties of bind's
> database, only those that allow us to properly handle DNSSEC, so let's
> try to uncover what those constrains are first, so I can understand why
> you propose this solution as better than something else.
>
>>>> Fortunately, it seems
>>>> that we can drop our own implementation of the internal DNS database
>>>> (ldap_driver.c and cache.c) and re-use the database from BIND (so called
>>>> RBTDB).
>>>>
>>>> I'm trying to reach Adam Tkac with the question "Why we decided to implement
>>>> it again rather than re-use BIND's code?".
>>>>
>>>>
>>>> Re-usage of BIND's implementation will have following properties:
>>>>
>>>>
>>>> == Advantages ==
>>>> - Big part of DNSSEC implementation from BIND9 can be reused.
>>>> - Overall plugin implementation will be simpler - we can drop many lines of
>>>> our code and bugs.
>>>> - Run-time performance could be much much better.
>>>>
>>>> - We will get implementation for these tickets "for free":
>>>> -- #95  wildcard CNAME does NOT work
>>>> -- #64 	IXFR support (IMHO this is important!)
>>>> -- #6 	Cache non-existing records
>>>>
>>>> And partially:
>>>> -- #7 	Allow limiting of the cache
>>>
>>> Sounds very interesting.
>>>
>>>
>>>> == Disadvantages ==
>>>> - Support for configurations without persistent search will complicate things
>>>> a lot.
>>>> -- Proposal => Make persistent search obligatory. OpenLDAP supports LDAP
>>>> SyncRepl, so it should be possible to make plugin compatible with 389 and
>>>> OpenLDAP at the same time. I would defer this to somebody from users/OpenLDAP
>>>> community.
>>>
>>> Why the persistent search would be required ?
>> As I mentioned above - you need database dump, because DNSSEC requires strict
>> name and record ordering.
>>
>> It is possible to do incremental changes when the 'starting snapshot' is
>> established, but it means that we need information about each particular
>> change => that is what persistent search provides.
>
> Ok, so it is to have a complete view of the databse, I assume to reduce
> the number of re-computations needed for DNSSEC.

I think it's inevitable, because you *need* the strict resource record 
ordering (see point 1 above) for the NSEC/NSEC3 record generation [1] - i.e. 
from the very beginning.

Partial re-signing is inevitable after each change, but you can't start the 
signing process without the strict resource record ordering.

[1] http://tools.ietf.org/html/rfc4034#section-6

>>>> - Data from LDAP have to be dumped to memory (or to file) before the server
>>>> will start replying to queries.
>>>> => This is not nice, but servers usually are not restarted often. IMHO it is
>>>> a
>>>> good compromise between complexity and performance.
>>>
>>> I am not sure I understand what this means. Does it mean you cannot change single
>>> cache entries on the fly when a change happens in LDAP ? Or something else ?
>> Sorry, I didn't explained this part in it's full depth.
>>
>> You can change everything run-time, but there are small details which
>> complicates loading of the zone and run-time changes:
>>
>> 1) A normal zones requires SOA + NS + A/AAAA records (for NSs) to load. It is
>> (hypothetically) possible to create empty zone, fill it with SOA, NS and A
>> records and then incrementally add rest of the records.
>>
>> The problem is that you need to re-implement DNS resolution algorithm to find
>> which records you need at the beginning (SOA, NS, A/AAAA) and then load the rest.
>>
>> I would like to avoid this re-implementation. It is not possible to re-use
>> BIND's implementation because it is tied to the DB implementation ... but we
>> can't load the database because it is missing SOA, NS and A/AAAA records.
>> Chicken-egg problem.
>
> To be honest I am not sure I understand what's your point here.
I'm not sure how to explain it without describing implementation in BIND. It 
is implementation detail, we can discuss it when we agree on all other things.

>> 2) The second reason why I want to make persistent search obligatory is that
>> each change in DNSSEC signed zone requires a lot of work, so it is not a good
>> idea to wait with the work to time when somebody asks for particular record.
>>
>> How it works without persistent search (now):
>> 1) Query from a client is received by BIND
>> 2) Internal cache is consulted
>> 3) Record is not found in the cache - LDAP search is done
>> 4) Fetched records in saved to the cache
>> 5) Reply to client is constructed
>>
>> It is hard to work in the same way when DNSSEC is in place. Each change
>> implicates re-signing of the particular RRset and it's neighbours, i.e.:
>> 1) Query from a client is received by BIND
>> 2) Internal cache is consulted
>> 3) Record is not found in the cache - LDAP search is done
>> 4) Fetched records in saved to the cache
>> * 4b) New RRset is re-signed
>> * 4c) Records neighbouring with the new RR has to be updated and re-signed
>> 5) Reply to client is constructed
>
> Ok so the point here is that we want to do the signing at store time
> rather than read time. That is understandable.
> However we have 2 ways to look at it.
> 1. bind does the work
> 2. DS does the work
>
> I haven't seen any reasoning from you why letting Bind do this work is
> a better idea.
Simply said - because all the code is already in BIND (the feature is called 
'in-line signing', as I mentioned above).

> I actually see some security reasons why putting this into a DS plugin
> can have quite some advantages instead. Have you considered doing this
It could improve the security a bit, I agree. But I don't think that it is so 
big advantage. BIND already has all the facilities for key material handling, 
so the only thing we have to solve is how to distribute keys from LDAP to 
running BIND.

> work in a DS plugin at all ? If you haven and have discarded the idea,
> can you say why ?
1) It would require pulling ~ 200 kiB (~ 7000 lines) of DNSSEC signing code 
into 389.

2) It would require pulling 'text->DNS wire format' parser into 389 (because 
our LDAP stores plain text data but the signing process works with DNS wire 
format).

3) It simplifies bind-dyndb-ldap, but we still need to re-implement DNS search 
algorithm which takes DNSSEC oddities into account. (Note that the DNS search 
algorithm is part of the database implementation. Bugs/limitations in our 
implementation are the reason why wildard records are not supported...)

4) I'm not sure how it will work with replication. How to ensure that new 
record will not appear in the zone until the associated RRset is (re)computed 
by DS? (BIND has transaction mechanism built-in to the internal RBTDB.)

>> The point is that you *can* do changes run-time, but you need to know about
>> the changes as soon as possible because each change requires significant
>> amount of work (and magic/mana :-).
>>
>> It opens a lot of opportunities for race condition problems.
>
> Yes, I am really concerned about the race conditions of course, however
> I really wonder whether doing signing in bind is really a good idea.
> We need to synchronize these signatures to all masters right ?
No, because signatures are computed and stored only in memory - and forgotten 
after BIND shutdown. Yes, it requires re-computing on each load, this is 
definitely disadvantage.

> Doesn't that mean we need to store this data back in LDAP ?
No, only 'normal' DNS updates containing unsigned data will be written back to 
LDAP. RRSIG and NSEC records will never reach LDAP.

> That means more round-trips before the data ends up being usable, and we
> do not have transactions in LDAP, so I am worried that doing the signing
> in Bind may not be the best way to go.
I'm proposing to re-use BIND's transaction mechanism built in internal 
database implementation.

>>>> => It should be possible to save old database to disk (during BIND shutdown
>>>> or
>>>> periodically) and re-use this old database during server startup. I.e. server
>>>> will start replying immediately from 'old' database and then the server will
>>>> switch to the new database when dump from LDAP is finished.
>>>
>>>
>>> This look like an advantage ? Why is it a disadvantage ?
>> It was mentioned as 'proposed remedy' for the disadvantage above.
>
> I think having dual authoritative data sources may not be a good thing.
Consistency is a reason why I want to make persistent search mandatory.

IMHO persistent storage could save the day if LDAP is down for some reason. 
Old data in DNS are much better than no data in DNS.

>>>> => As a side effect, BIND can start even if connection to LDAP server is down
>>>> - this can improve infrastructure resiliency a lot!
>>>
>>> Same as above ?
>> The same here, it was mentioned as 'proposed remedy' for the disadvantage above.
>
> When it comes to DNSSEC starting w/o LDAP may just mean that you have
> different signatures for the same records on different masters. Is that
> 'legale' according to DNSSEC ?
1) You will have same signatures as long as records in LDAP and saved copy of 
the database (on the disk) are equal.

2) I didn't find any new limitation imposed by DNSSEC. AFAIK some 
inconsistency between servers is normal state in DNS, because zone transfers 
take some time and the tree structure have many levels.

The problems arise when data *in single database* (i.e. on one server) are 
inconsistent (e.g. signature != data in unsigned records). BIND solves this 
with it's built-in transaction mechanisms.

>>>> == Uncertain effects ==
>>>> - Memory consumption will change, but I'm not sure in which direction.
>>>> - SOA serial number maintenance is a open question.
>>>
>>> Why SOA serial is a problem ?
>> It simply needs more investigation. BIND's RBTDB maintains SOA serial
>> internally (it is intertwined to transactions in the DB), so the write-back to
>> LDAP could be very delicate operation.
>
> It means all masters will often be out of sync, this is not very good.
I don't think so. BIND can use timestamp-based serials in exactly same way as 
we do. The only problem is how to implement 'read from internal DB'->'write to 
LDAP' operation. It still needs more investigation.

>>>> Decision if persistent search is a 'requirement' or not will have significant
>>>> impact on the design, so I will write the design document when this decision
>>>> is made.
>>>
>>> I would like to know more details about the reasons before I can usefully comment.
>>
>> I forgot to one another 'Uncertain effect':
>> - Support for dynamically generated '_location' records will be a big
>> adventure. It probably means no change from the state without persistent
>> search :-) After basic exploration it seems doable, but still a bit uncertain.
>
> I need more info here, does it mean you have to store _location records
> when they are generated ?
I tend to do _location record generation during zone-load, so everything will 
be prepared when the query comes in. As a benefit it will allow zone transfers 
even for signed zones. This still needs more investigation.

 > Maybe we can use the internal bind database
> just for _location "zone" ?
I don't think that it is possible.

If _location.client-a and _location.client-b reside in the single database 
then client-a and client-b have to reside in the same database. (The reason is 
that _location.client-a and _location.client-b do not have immediate common 
ancestor.)

>> My personal conclusion is that re-using of BIND's backend will save a huge
>> amount of work/code to maintain/bugs.
>
> I can see that, unfortunately I fear it will make multi-master a lot
> more difficult at the same time. And given we do want to have
> multi-master properties we need to analyze that problem more carefully.
I agree. It is a delicate change and we should not hurry.

> Also by welding ourselves to internal Bind infrastructure too much, it
> will make it a lot more difficult for us to change the DNS
> infrastructure. Bind10 will be completely different internally, and we
> may simply decide to even not use bind10 at all and use a completely
> different engine going forward. So I am quite wary of welding ourselves
> even more to bind 9 internals.
Ehm ... how to say that ... 'to late'. I wasn't around when DNS design was 
made, so I don't know all the reasons behind the decision, but IMHO we use 
completely non-standard/obscure hacks all the time.

The proposal above doesn't extend our dependency on BIND, because we already 
depend on BIND9 *completely*. It is about dropping our own internal database 
implementation (buggy, incomplete, standard non-compliant) with the code from 
the original BIND (which is at least standard compliant).

<sarcasm>
Do you want to go back to 'light side of the force'? So we should start with 
designing some LDAP->nsupdate gateway and use that for zone maintenance. It 
doesn't solve adding/reconfiguring of zones on run-time, but it could be 
handled by some stand-alone daemon with an abstraction layer at proper place.
</sacrasm>

-- 
Petr^2 Spacek