[Freeipa-users] bind-dyndb-ldap vs DLZ

[Petr Spacek]

On 27.3.2015 02:02, Jorgen Lundman wrote:
> Petr Spacek wrote:
>> Perfect! I can merge your changes upstream if you send me a patch with your
>> changes. It would make your life easier later when you need to pick new code.
> 
> Not a problem, I need to figure out why Solaris mkdir returns -1, with
> errno = 0. Goes against the manpage and all logic. Checking for ||errno==0
> after mkdir "fixes" it but it is still odd.

That is really weird! I'm bit worried that it would break something on some
other system so I'm not rushing to merge it upstream.

> I also compiled without krb5. It basically compiled, but running would fail
> to find gss/mech_krb5.so  - no idea how that is supposed to link, but since
> it was a quick test, I just took it out.

Interesting. I guess that the missing file is something like Kerberos module
for GSS-API library on Solaris.

Anyway, feel free to send fixes or a patch with --disable-kerberos option for
configure.

> Otherwise it compiled smoothly, some minor linux-ism with linker flags.

Could you be more specific, please? I can certainly fix linker flags if you
tell me how :-)

>> Hmm, that might be a challenge. bind-dyndb-ldap code implicitly assumes that
>> there is 1:1 mapping between DNS name<->LDAP DN. This makes implementation of
>> dynamic updates much easier.
> 
> Actually yes, I did mean to ask about this too. DLZ-LDAP is pretty much a
> read-only setup (for us), so a quick scan of the sources led me to believe
> that simple string replacement of "idnsName" with "DNSZoneName" (and of
> course, all the others), should get "a large chunk" of it done. The schemas
> are quite close, on a whole..

Yeah, changes for read-only mode could be pretty easy. See mainly
src/ldap_entry.c a src/ldap_convert.c.

> But I was surprised to see bind-dyndb actually modify the LDAP data, for my
> test, just "idnsSOAserial". What is the purpose of this? I would guess for
> zone xfers? By why update LDAP (and not just internal DNS memory), if you
> essentially do not use it on restart (just gets updated again on restart,
> from the looks of it?)

This is to make sure that SOA serial is incremented even if the server
crashed. The problem is that data in LDAP could be modified while server is
down so you have to increment SOA serial in all cases.

This could be optimized further when we finally have persistent copy of data
on disk, we could simply store syncrepl cookie somewhere and increment SOA
serial only if something was changed.

>>> the same delay each time I start it. slapd is running on localhost, and is
>>> otherwise idle. 
>>
>> Hmm, that stinks! I would be happy to look into it if you can provide me with
>> output from a profiler of your choice. (It might be a good idea to profile
>> bind-dyndb-ldap together with whole named process to see all the interactions.)
> 
> It is in line with what we experience with LDAP generally. If I blow away
> the DNS-ldap tree, a full syncrepl update takes about 1 hour. If I remove
> the whole LDAP tree, about 4 hours. It is not something that goes faster
> with more threads afaik.

Here I'm merging both you e-mails to one:

> Hold those horses. I must admit this timing didn't sit right with me, the
> more I thought about it. Since my test was all new, I created a slapd.conf
> from scratch. Forgot all about entryUUID index!

Interesting. Could you tell us how long it took to start-up with indexed
entryUUID but without the other changes?

Or more specifically, could you please measure impact of separate changes
mentioned below?

> I took out the creations of the empty master/$domain/keys directories. We
> don't use any of that.

This effectively breaks dynamic updates, IXFR and DNSSEC in-line signing but
it might not be a problem for you if you do not use these features.

> I noticed that its not the syncrepl that is slow, it is the MOD update of
> all idnsSOAserial.  Unfortunately, they are all on one connection, so
> increasing
>         arg "connections 15";
> does not help.

Interesting. I believe that this is caused by internal event serialization in
bind-dyndb-ldap. All changes to idnsZone objects are processed only by single
thread (mainly purpose is AFAIK simplification of locking). It certainly can
be fixed and this fix should enable parallelization of zone loading.

BTW my guess is that you are testing the worst case - a lot of small zones. It
should scale better with bigger zones (but no guarantees here! :-).

> I took out the updates of idnsSOAserial by making
> ldap_replace_serial(return ISC_R_SUCCESS);
> 
> Startup time is just under 2 minutes. I can certainly live with that. Just
> go to work out if we can actually use it without idnsSOAserial or not. We

It has ~ no purpose if you do not do DNS zone transfers or in-line signing
(which depends on internal zone transfer).

> have just ldap master using syncrepl to each DNS server's local slapd. Then
> named to use that. No xfers, no slaves etc.

You probably do not need it.

> Shutdown is still slow, but that appears to be a Solaris bug, after all
> zones are shutdown and listening ports released, it sits around calling
> lwp_park and unpark for 10 minutes. I can debug that later.

If you are really using read-only zones feel free to kill -9 the named process :-)

>>> Is it supposed to be able to use the "dump-file" on exit for faster loads?
>>
>> Yes, something like that is planned but not implemented yet.
> 
> Ah ok great, good to know it is planned. We could probably live with 50
> mins startup time, as there are 28 servers in the DNS cluster. Just needs
> some care in case they are restarted, or worse, some bug causes coredump
> which affects all of them

Yes, simultaneous crash/coredump happens from time to time when we hit some
nasty bind-dyndb-ldap bug.

>>> [3]
>>> However, that has a side-effect that, once bind-dyndb is loaded, it will
>>> also query DLZ+LDAP on negative entries. Thus decreasing the performance to
>>> 3884qps.  Wonder if there is a way to have bind-dyndb ignore DLZ+LDAP /once
>>> it has loaded/.
>>
>> Wow, I'm surprised that this combination actually works! I expect that there
>> will be some nasty surprises in there, especially when it comes to dynamic
>> updates.
> 
> I must admit it would be tempting to see if bind-dyndb could issue a
> definitive reply (once loaded) for both positive and negative lookups,
> thereby causing DLZ-LDAP to be inert. 

bind-dyndb-ldap does answer authoritatively for all queries. I believe that
what you see is side-effect/"feature" of DLZ interface.

> Or making it a feature of bind-dyndb
> to use DLZ-LDAP during load. But only in the case that they both use the
> same schema and source tree in LDAP.
> 
> Feels a bit hackish but just evaluating possible solutions.

To me it sounds like a terrible hack. Do not even try to think about
interaction with DNSSEC in-line signing. Yuck!

Have a nice day (and send us patches! :-).

-- 
Petr^2 Spacek