[Freeipa-devel] Thoughts on the new LDAP backend plugin

[Jason Gerard DeRose]

Pavel is going to work on writing a new LDAP backend plugin, so now is
the time for everyone to voice their two cents.  Pavel's new backend
plugin will coexist with the current till it's mature enough to replace
it.

Here are some of my thoughts on the new ldap plugin.  I think we should:

1. Keep the command plugins completely isolated from the python-ldap
bindings: for command plugins, Backend.ldap is their black box into the
LDAP world.  They should never need to import anything from python-ldap
(even constants) and we should be able to use something besides the
python-ldap bindings under the hood without the other plugins knowing
the difference.  This makes testing easy because we can plugin in a
dummy Backend.ldap plugin.  Plus it just leads to a more manageable,
well-layered architecture.

2. Keep it simple: Backend.ldap needs to be easy to use and high level.
Its methods should take as arguments simple scalar and compound values,
and should return the same.  We should put the smarts and the heavy
lifting in Backend.ldap so we make the lives of our 69 (just counted)
command plugins easier.

3. Keep it generalized: we want Backend.ldap to serve the needs of
arbitrary 3rd-party plugins, so the API can't be in terms of adding
users, adding groups, etc.  We need to think in terms of adding
*entries*.  I know that I'm guilty for starting the infamous
make_foo_dn() series of methods, but we've all seen their shortcomings,
so now is our chance to fix it.

4. Design for testability... aka, write tests as we go.  Everyone knows
I love to beat this dead horse, but discipled test-driven development is
the fastest way to develop.  Now people may say, "but it takes time to
write unit tests, so obviously it slows development".  But people don't
realize how much time they spend on their ephemeral tests.  You know,
add a bunch print statements, try some different inputs, and then once
it seems like it's working, delete all these print statements.  In that
time, you could write a formal unit tests which will live on in the code
and keep everyone more productive.  Okay, stepping down from my dead
horse, I mean soapbox...

5. Design for Unicode/UTF-8 correctness: Backend.ldap is one of those
entry/exit points where we need explicitly to decode text on the way in,
and explicit to encode text on the way out.  Because text in DS (LDAP)
will always be UTF-8 encoded (AFAIK), it's tempting only to decode the
text when we first need to do text-stuff with it.  But by that point, we
have no idea that the text (stored in the `str` type) originated in
DS... we have a multi-input, multi-output plugable framework.  That
`str` might be Unicode in some encoding other that UTF-8 or it might
just as well be true binary data.  Pavel missed out on the big internal
Python-Unicode email discussion John and I had, but here is a diagram
from one of my emails:

  ===> Entry from network, file, broken C extension, etc.:
  ===>   * What is the source encoding?
  ===>   * internal_unicode = input_str.decode(source_encoding)

       ###########################################
       Internal processing in pure Python:
         * character data is always `unicode`
         * `str` is only used for true binary data
       ###########################################

  <=== Exit to network, file, broken C extension, etc.:
  <===   * What encoding does the destination expect?
  <===   * output_str = internal_unicode.encode(destination_encoding)

In freeIPA v2, we're taking a very Python-3.0-like approach to Unicode.
Internally we use the Python `unicode` type for all text and only use
`str` for true binary data, and we do explicit decoding and encoding.
In Python 3.0, the `str` type has been renamed to `bytes`, the `unicode`
type has been renamed to `str`, and the implicit decoding/encoding
features have been removed.  Or that's the gist of it.  See:

    http://docs.python.org/3.0/whatsnew/3.0.html


That's all for now!

Cheers,
Jason