[Freeipa-users] bind-dyndb-ldap vs DLZ

[Petr Spacek]

Hello Jorgen and list,

On 26.3.2015 01:25, Jorgen Lundman wrote:
> Thanks for your email, (I have included the ML in this reply)
> 
> We run Solaris with bind+DLZ+ldap on the DNS servers, and are looking at
> better performance. Which means evaluating bind-dyndb-ldap. I did some
> minor tweaks to compile bind-dyndb on Solaris.

Perfect! I can merge your changes upstream if you send me a patch with your
changes. It would make your life easier later when you need to pick new code.

> Since we already have large systems running in production, with
> provisioning, and support tools, it is unlikely we would change the schema.
> It would probably be less work for me to fork bind-dyndb and massage it
> into handling DLZ schema directly.

Hmm, that might be a challenge. bind-dyndb-ldap code implicitly assumes that
there is 1:1 mapping between DNS name<->LDAP DN. This makes implementation of
dynamic updates much easier.

Anyway, please be so kind and send your patches to us too. It is well possible
that some chunks will be applicable to bind-dyndb-ldap too.

Hopefully this approach would improve code quality upstream and at the same
time allow you to maintain minimal patch set and pull new code from upstream
often and with minimal pain.

BTW there is outstanding patch set which touches record-parsing logic:
https://github.com/pspacek/bind-dyndb-ldap/tree/unknown_record_types

It is not a final version yet but it might be a good idea to base your patches
on that to save some patch rebasing.

> But before that, we are just evaluating bind-dyndb to decide if it fits
> with our systems.
> 
> I loaded the 300k zones we have in DLZ-LDAP, into bind-dyndb schema (Just
> SOA records with a single ARecord).
> 
> [1]
> The first issue is that to start named takes about 50 mins. This is a bit
> of an issue as there is no resolving while it is starting. This seems to be

It is expected that bind-dyndb-ldap does not serve anything until whole zone
is loaded:

Basically the driver downloads all data from LDAP and feeds the data into
BIND's RBTDB implementation. This is the reason why it has read performance on
nearly same level as plain BIND but it also adds the same limitations - all
the zone data have to be in memory before the zone can be loaded.

(Also, the driver supports DNSSEC in-line signing and it does not make sense
to sign the zone before it is fully loaded.)

Unfortunately there is no way to detect that one zone is fully loaded because
SyncRepl is running on the whole dns sub-tree so we have to wait until initial
synchronization is done.

> the same delay each time I start it. slapd is running on localhost, and is
> otherwise idle. 

Hmm, that stinks! I would be happy to look into it if you can provide me with
output from a profiler of your choice. (It might be a good idea to profile
bind-dyndb-ldap together with whole named process to see all the interactions.)

> It is just a large syncrepl for 300k records, always
> starting from epoch...

This is a limitation of current implementation and it can be fixed, see below.

> [2]
> Is it supposed to be able to use the "dump-file" on exit for faster loads?

Yes, something like that is planned but not implemented yet.

You are basically interested in resolving following two tickets:
https://fedorahosted.org/bind-dyndb-ldap/ticket/124
https://fedorahosted.org/bind-dyndb-ldap/ticket/125

With these two in place it should be fairly easy to read all data from disk,
start serving the zone and do synchronization with LDAP server in background.

A pre-requisite for this are basically these:
1) Answering questions on
https://fedorahosted.org/bind-dyndb-ldap/wiki/BIND9/Design/RBTDB#Filesystemcachemaintenance
2) Implementation of meta-database for LDAP UUID->DNS node mapping
https://fedorahosted.org/bind-dyndb-ldap/ticket/151


Trac Report #3 shows currently planned work ordered by priority:
https://fedorahosted.org/bind-dyndb-ldap/report/3

As you might see, the necessary meta-database is pretty high on the list (it
should be done in ~ 1 month) but start-up performance has very low priority at
the moment and is very unlikely to be ready in Fedora 22 time frame.

If you want to get better start-up performace earlier I would suggest you to
work with us on it. Patches are more than welcome! :-)

We can provide you guidance but currently we do not have capacity to implement
it ourselves in near future, sorry!

> Perhaps I broke something while porting it to Solaris. I see it create

No you did not, it is not implemented yet :-)

> directories for each zone, only to create an empty "keys" directory. In
> addition to that, having 300k entries in one directory might need hashing.
> ZFS is ok with it, but it tends to slow down.
>
> However, once it is loaded, it is much faster than DLZ+LDAP. Previous
> system would see about 300qps. (All zones, plus
> /usr/share/dict/words+".com" with dnsperf)
> 
> bind-dyndb gave the same benchmarking test 18796qps.
> 
> Now, interestingly, I can also define a DLZ+LDAP line in named.conf after
> the bind-dyndb. This has the effect that while bind-dyndb is taking 50 mins
> to start up, it will resolve names using DLZ+LDAP. Sure, at the worse qps
> of course, but it is at least resolving.
> 
> [3]
> However, that has a side-effect that, once bind-dyndb is loaded, it will
> also query DLZ+LDAP on negative entries. Thus decreasing the performance to
> 3884qps.  Wonder if there is a way to have bind-dyndb ignore DLZ+LDAP /once
> it has loaded/.

Wow, I'm surprised that this combination actually works! I expect that there
will be some nasty surprises in there, especially when it comes to dynamic
updates.

> That is our current situation.
> 
> Sincerely,
> 
> Lund
> 
> ps. There are some minor faults in the example.ldif, like
> "idnsName=example.com"'s idnsName is "ipatest.com". And is missing NSRecord.

Oh, thank you for catching this! It is now fixed in master branch:
https://git.fedorahosted.org/cgit/bind-dyndb-ldap.git/commit/?id=0bf04034675c55178c7dd885e9cdfadd33ba441b

Petr^2 Spacek

> Petr Spacek wrote:
>> On 25.3.2015 04:45, Jorgen Lundman wrote:
>>>
>>> Hey Petr,
>>>
>>> Your name came up on the DLZ list a while back and I was interested in
>>> bind-dyndb-ldap. We run DLZ-LDAP here, and after this weeks DDOS
>>> firefighting we are looking at increasing our qps.
>>>
>>> At the moment, each BIND-DLZ-LDAP server can do about 200qps, which is
>>> pretty low. But for authoritative it has been sufficient until now. I have
>>> fixed DLZ-LDAP, enabling MULTITHREADED (it is off by default for some
>>> reason) but still, it has to query the DB each time. The thing refuses to
>>> use more than 130MB of cache, each server has 32GB so that is a bit
>>> annoying. :)
>>>
>>> Biggest issue with bind-dyndb-ldap is the schema difference to DLZ's
>>> schema.  Most items can probably be handled with straight keyword rename,
>>> but one of the larger differences is that bind-dyndb-ldap handles multiple
>>> "ARecord" for one entry, whereas DLZ uses that odd "A1,DNSHostName=foo" and
>>> "A2,DNSHostName=foo" to do multiples.
>>>
>>> Do you know if anyone has already looked at DLZ to bind-dyndb-ldap
>>> migration? I suppose as a test case (to see what qps I can get), I can
>>> convert the DLZ LDAP into zone files, then back to bind-dyndb-ldap schema
>>> using your supplied scripts.
>>>
>>> If you have time, drop me a line.
>>
>> Hello!
>>
>> As far as I know there is no DLZ->bind-dyndb-ldap convertor.
>>
>> I would say that an easiest way is to do AXFR from DLZ-backed zone, save it to
>> a file and feed the zone file into
>> https://github.com/pspacek/zone2dyndb-ldif
>>
>> IMHO it is not worth to develop special DLZ->bind-dyndb-ldap convertor because
>> we can always use zone file format as intermediate step.
>>
>> LDAP schema used by bind-dyndb-ldap is described on:
>> https://fedorahosted.org/bind-dyndb-ldap/wiki/LDAPSchema
>>
>>
>> If you have any further questions please contact freeipa-users at redhat.com
>> mailing list so everyone including search engines can see it :-)
>>
>> Thank you for understanding and have a nice day!
>>
> 


-- 
Petr^2 Spacek