[Freeipa-devel] Provisioning throughput

Wed May 25 18:49:09 UTC 2016

thierry bordaz wrote:
>
> Hello,
>
> Thanks for all the feedbacks. I updated the design accordingly and with
> additional tests results
> (http://www.freeipa.org/page/V4/Performance_Improvements#Proposed_improvements)
> Several improvements can be done, in particular in DS plugins (memberof,
> retroCL), but for "easy" benefit provisioning will be done with memberof
> disabled followed by fixup.
>
> It remains some aspects that are not clear to me:
>
>   * For best performance, DS tuning and provisioning/fixup would
>     preferably be done under 'directory manager'
>     That means prompting DM password and writing it into temporary file.
>     Is that a concern ?
>   * Fixup requires that we know the filters matching the provisioned
>     entries. For example :
>       o (objectClass=inetorgperson)
>       o (objectClass=ipausergroup)
>       o (objectClass=ipahost)
>       o (objectClass=ipahostgroup)
>       o (objectClass=ipasudorule)
>       o (objectClass=ipahbacrule)
>
>         The set of objectclass could be hardcode or provided in the
>         provisioning CLI option
>         What to do if an entry in in the provision file does not match
>         any of those filter ? Should it stop without starting the
>         provisioning ?
>   * The CLI doing the provisioning could be something like 'ipa
>     provision <options>' or should it be a separated command e.g.
>     ipa-bulk-load ?

It depends. There is a migration command now, ipa migrate-ds, that adds 
records and is impacted by this. There is also the possibility of 
looping calls to ipa [user|group|etc]-add.

Would it be reasonable to require bulk import to be done on an IPA 
master so we can leverage the ldapi socket?

rob

>
> thanks
> thierry
>
> On 05/13/2016 10:18 AM, Ludwig Krispenz wrote:
>>
>> On 05/13/2016 09:42 AM, Petr Spacek wrote:
>>> On 13.5.2016 09:26, Martin Kosek wrote:
>>>> On 05/12/2016 04:16 PM, Ludwig Krispenz wrote:
>>>>> On 05/12/2016 03:45 PM, Ludwig Krispenz wrote:
>>>>>> On 05/12/2016 02:16 PM, Petr Vobornik wrote:
>>>>>>> On 05/10/2016 05:50 PM, thierry bordaz wrote:
>>>>>>>> On 05/05/2016 03:44 PM, Petr Vobornik wrote:
>>>>>>>>> On 05/04/2016 02:20 PM, thierry bordaz wrote:
>>>>>>>>>> Hello,
>>>>>>>>>>
>>>>>>>>>>        I have been doing some tests/measures using
>>>>>>>>>> https://github.com/freeipa/freeipa-tools/blob/master/create-test-data.py.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>        The tool creates a set of typical users/hosts/groups... to
>>>>>>>>>> import with a
>>>>>>>>>>        ldapadd.
>>>>>>>>>>
>>>>>>>>>>        I wrote down some finding in
>>>>>>>>>> http://www.freeipa.org/page/V4/Performance_Improvements#Provisioning_throughput_and_DS_plugins.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>        I still have to do some cleanup around the performance
>>>>>>>>>> but the
>>>>>>>>>> basic of a
>>>>>>>>>>        possible improvement is to do provisioning in several
>>>>>>>>>> steps
>>>>>>>>>> (disabling
>>>>>>>>>>        plugins, provisioning, enabling plugin, running fixup
>>>>>>>>>> tasks).
>>>>>>>>>>
>>>>>>>>>>        Before going further in the design I wanted to share
>>>>>>>>>> those ideas
>>>>>>>>>> and know if
>>>>>>>>>>        it raise any concern.
>>>>>>>>>>
>>>>>>>>>>        thanks
>>>>>>>>>>        thierry
>>>>>>>>>>
>>>>>>>>> Hi Thierry,
>>>>>>>>>
>>>>>>>>> Thanks for the analysis. Very nice.
>>>>>>>>>
>>>>>>>>> Knowing this will help us suggesting workarounds also for old
>>>>>>>>> releases.
>>>>>>>>>
>>>>>>>>> Couple questions:
>>>>>>>>>
>>>>>>>>> Have you tested retrCL disabled with memberOf enabled. It seems
>>>>>>>>> that it
>>>>>>>>> would eliminate 550K adds and 0.8M searches. What would be the
>>>>>>>>> time
>>>>>>>>> improvement?
>>>>>>>>>
>>>>>>>>> Do you know what is the time when memberof is enabled but
>>>>>>>>> slapi-nis and
>>>>>>>>> retroCL are disabled?
>>>>>>>> The culprit of the performance issue is very likely related to SRCH
>>>>>>>> (internal) triggered by memberof.
>>>>>>>>
>>>>>>>> If retroCL is disabled and memberof enabled, #SRCH is 13.8M.
>>>>>>>> If retroCL is disabled, slapi-nis disabled and memberof enabled
>>>>>>>> #SRCH is
>>>>>>>> 14.8
>>>>>>>> When all of them are enabled the #SRCH is 15M.
>>>>>>>>
>>>>>>>> You are right if retroCL is disabled the #ADD drops but it has no
>>>>>>>> significant effect on the duration.
>>>>>>> ok, thanks for the analysis
>>>>>>>
>>>>>>>> Regarding the duration of the provisioning, values are not
>>>>>>>> really stable
>>>>>>>> as performance of VM fluctuates. But as soon as memberof is
>>>>>>>> enabled the
>>>>>>>> provisioning lasts > 4hours where the same provisioning lasts
>>>>>>>> 6mins as
>>>>>>>> soon as memberof is disabled.
>>>>>>>>
>>>>>>>> I need to confirm the average time for internal searches but
>>>>>>>> assuming
>>>>>>>> 1ms per SRCH it consumes >90% of the provisioning.
>>>>>>>>
>>>>>>>>
>>>>>>>>>    From the text it was not clear to me, if you find or
>>>>>>>>> investigate
>>>>>>>>> possible improvements in memberof plugin which would improve the
>>>>>>>>> performance without stopping and starting DS.
>>>>>>> As was discussed at mtg, have you tried if the DS restart is really
>>>>>>> necessary?
>>>>>> memberof plugin can be enabled and disabled while the server is
>>>>>> running, BUT
>>>>>> to achieve this the "enable-dynamic-plugins" feature has to be
>>>>>> turned on. And
>>>>>> then any enable/disable of a plugin would try to do it dynamically
>>>>>> an dnot
>>>>>> wait for the restart.
>>>>>> And I think not all plugins are able to handle this, TomasB was
>>>>>> once working
>>>>>> on it for IPA plugins, but it was not completed as far as I know
>>>>> but enabling dynamic plugins can be done without restart, so what
>>>>> can be done is.
>>>>> - enable dynamic plugins
>>>>> - disable memberof
>>>>> - do some work
>>>>> - enable memberof
>>>>> - disable dynamic plugins
>>>> Please see
>>>> https://fedorahosted.org/freeipa/ticket/4203#comment:9
>>>> I do not think this will be that easy. We would first need to invest
>>>> into
>>>> updating FreeIPA plugins to work with dynamic plugins setting and
>>>> then we could
>>>> do things alike above.
>>>>
>>>> It looks like that for FreeIPA 4.4, we will need to live with DS
>>>> restart unless
>>>> there is some workaround...
>> couldn't the scenario I outline above with enabling dynamic plugins
>> only temporary work, are there any attempts to enable/disable plugins
>> during provisioning ? If that would be the case that would also
>> require a restart
>>> One more thing:
>>>
>>> How does it affect topologies with replicas?
>>>
>>> I might be wrong, but if memberOf is always computed locally then we
>>> have to
>>> disable it on *all* replicas.
>>>
>>> If we disabled it only on one replica and not others, the chosen
>>> replica would
>>> be way faster than rest of the topology and I'm not sure what would
>>> happen
>>> later on.
>> good point. we exclude memberof from replication as it is regenerated
>> on every server, so each replica would suffer from the performance
>> problem
>>>
>>> Thierry, Ludwig, can you comment on this?
>>>
>>
>
>
>