[Freeipa-devel] Provisioning throughput

[thierry bordaz]

On 05/31/2016 02:02 PM, Petr Vobornik wrote:
> On 05/04/2016 02:20 PM, thierry bordaz wrote:
>> Hello,
>>
>>      I have been doing some tests/measures using
>>      https://github.com/freeipa/freeipa-tools/blob/master/create-test-data.py.
>>      The tool creates a set of typical users/hosts/groups... to import with a
>>      ldapadd.
>>
>>      I wrote down some finding in
>>      http://www.freeipa.org/page/V4/Performance_Improvements#Provisioning_throughput_and_DS_plugins.
>>      I still have to do some cleanup around the performance but the basic of a
>>      possible improvement is to do provisioning in several steps (disabling
>>      plugins, provisioning, enabling plugin, running fixup tasks).
>>
>>      Before going further in the design I wanted to share those ideas and know if
>>      it raise any concern.
>>
>>      thanks
>>      thierry
>>
> Let me conclude the previous sub-thread and also continue with
> discussion we had over phone.
>
> The subthread ended with proposal to go with offline provisioning by
> disabling LDAP ports similar to ipa-server-upgrade tool and then using
> LDAPI to add records by using ipalib in server mode e.g., as in
> ipa_otptoken_import.py
>
> So these are the tasks to solve/investigate:
>
> 1. Provide guidance/write script which would disable memberof plugin and
> other plugins. Disable ldap and ldaps port

I started describing the operations 
http://www.freeipa.org/page/V4/Performance_Improvements#Algorithm.
It needs to be updated with disabling regular ports and accessing 
through ldapi
>
> 2. Provide guidance/script how to use ipalib in server mode and how to
> import date. This could be even script which would load file in some
> format(e.g. JSON) and executed commands from the file. Basically what
> was Alexander proposing and I was against it. After some thought, I
> agree that the tool could be easy to write but I'd rather avoid adding
> it to 4.4 release maybe even to future releases. Because:
>   - It's almost the same as `[RFE] run multiple CLI commands in a batch`
> <https://fedorahosted.org/freeipa/ticket/5821> With the distinction that
> it connects directly LDAPI and not public API. First I'd like to see
> #5821 and then we can think of using same logic(input) to work in
> "migration" mode.
> - I don't want to include a quickly baked tool so late in 4.4
> development. It will have design flaws which will be harder to fix
> later. I don't want to loose time with discussion about design of the
> tool in this phase of 4.4.
Investigating a ldap bulk load, there is a quite limited set of 
operations to prepare the instance, run a ldap bulk load and run fixup.
However, starting yesterday looking at ipa-otptoken-import I am still 
unable to connect through ldapi and do those really simple operations... 
so even it could be easy to write tool my progress start slowly

Regarding the batch commands, it looks quite different than bulk import 
because batch commands (like user-add) run  several ldap ops 
(SRCH/ADD/MOD) and also batch commands does expect that DS plugins like 
memberof are enabled.

>
> 3. Provide guidance/script to revert #1 and run memberof fixup task.
>
> 4. Investigate how replication of so many records with member attributes
> will affect other replicas. I.e. if it would not brick entire topology.
>
> *Right now #4 seems to be the most important*.

This was shortly described in 
http://www.freeipa.org/page/V4/Performance_Improvements#Memberof_plugin 
and 
http://www.freeipa.org/page/V4/Performance_Improvements#Replication_being_late.

The topology will slowly converge and eventually all provisioned entries 
will be available everywhere.
But slowly can mean hours or even days before it converges. During that 
period, a provisioned rules can grant some rights on one replica (where 
it was imported) but will not grant it on an other where the rule is not 
yet replicated.

If the topology is a production topology, it can be better to rely on 
total init of the replicas than on replication.



>
> Then #1, #2, #3 could be delivered as sample script included in
> documentation and/or blog post/github. It would allow us to change it
> later. I would not focus on #2 before other core 4.4 items are finished.
> A lot of guidance is already written in tree design page but it will
> need to be transform to easily consumable form for admins. The scripts
> can be prepared even when 4.4 is out.
>
> In other words I would not create any official new ipa utility yet.