[Freeipa-devel] Provisioning throughput

Tue May 17 09:48:01 UTC 2016

On 05/13/2016 10:18 AM, Ludwig Krispenz wrote:
>
> On 05/13/2016 09:42 AM, Petr Spacek wrote:
>> On 13.5.2016 09:26, Martin Kosek wrote:
>>> On 05/12/2016 04:16 PM, Ludwig Krispenz wrote:
>>>> On 05/12/2016 03:45 PM, Ludwig Krispenz wrote:
>>>>> On 05/12/2016 02:16 PM, Petr Vobornik wrote:
>>>>>> On 05/10/2016 05:50 PM, thierry bordaz wrote:
>>>>>>> On 05/05/2016 03:44 PM, Petr Vobornik wrote:
>>>>>>>> On 05/04/2016 02:20 PM, thierry bordaz wrote:
>>>>>>>>> Hello,
>>>>>>>>>
>>>>>>>>>        I have been doing some tests/measures using
>>>>>>>>> https://github.com/freeipa/freeipa-tools/blob/master/create-test-data.py. 
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>        The tool creates a set of typical users/hosts/groups... to
>>>>>>>>> import with a
>>>>>>>>>        ldapadd.
>>>>>>>>>
>>>>>>>>>        I wrote down some finding in
>>>>>>>>> http://www.freeipa.org/page/V4/Performance_Improvements#Provisioning_throughput_and_DS_plugins. 
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>        I still have to do some cleanup around the performance 
>>>>>>>>> but the
>>>>>>>>> basic of a
>>>>>>>>>        possible improvement is to do provisioning in several 
>>>>>>>>> steps
>>>>>>>>> (disabling
>>>>>>>>>        plugins, provisioning, enabling plugin, running fixup 
>>>>>>>>> tasks).
>>>>>>>>>
>>>>>>>>>        Before going further in the design I wanted to share 
>>>>>>>>> those ideas
>>>>>>>>> and know if
>>>>>>>>>        it raise any concern.
>>>>>>>>>
>>>>>>>>>        thanks
>>>>>>>>>        thierry
>>>>>>>>>
>>>>>>>> Hi Thierry,
>>>>>>>>
>>>>>>>> Thanks for the analysis. Very nice.
>>>>>>>>
>>>>>>>> Knowing this will help us suggesting workarounds also for old 
>>>>>>>> releases.
>>>>>>>>
>>>>>>>> Couple questions:
>>>>>>>>
>>>>>>>> Have you tested retrCL disabled with memberOf enabled. It seems 
>>>>>>>> that it
>>>>>>>> would eliminate 550K adds and 0.8M searches. What would be the 
>>>>>>>> time
>>>>>>>> improvement?
>>>>>>>>
>>>>>>>> Do you know what is the time when memberof is enabled but 
>>>>>>>> slapi-nis and
>>>>>>>> retroCL are disabled?
>>>>>>> The culprit of the performance issue is very likely related to SRCH
>>>>>>> (internal) triggered by memberof.
>>>>>>>
>>>>>>> If retroCL is disabled and memberof enabled, #SRCH is 13.8M.
>>>>>>> If retroCL is disabled, slapi-nis disabled and memberof enabled 
>>>>>>> #SRCH is
>>>>>>> 14.8
>>>>>>> When all of them are enabled the #SRCH is 15M.
>>>>>>>
>>>>>>> You are right if retroCL is disabled the #ADD drops but it has no
>>>>>>> significant effect on the duration.
>>>>>> ok, thanks for the analysis
>>>>>>
>>>>>>> Regarding the duration of the provisioning, values are not 
>>>>>>> really stable
>>>>>>> as performance of VM fluctuates. But as soon as memberof is 
>>>>>>> enabled the
>>>>>>> provisioning lasts > 4hours where the same provisioning lasts 
>>>>>>> 6mins as
>>>>>>> soon as memberof is disabled.
>>>>>>>
>>>>>>> I need to confirm the average time for internal searches but 
>>>>>>> assuming
>>>>>>> 1ms per SRCH it consumes >90% of the provisioning.
>>>>>>>
>>>>>>>
>>>>>>>>    From the text it was not clear to me, if you find or 
>>>>>>>> investigate
>>>>>>>> possible improvements in memberof plugin which would improve the
>>>>>>>> performance without stopping and starting DS.
>>>>>> As was discussed at mtg, have you tried if the DS restart is really
>>>>>> necessary?
>>>>> memberof plugin can be enabled and disabled while the server is 
>>>>> running, BUT
>>>>> to achieve this the "enable-dynamic-plugins" feature has to be 
>>>>> turned on. And
>>>>> then any enable/disable of a plugin would try to do it dynamically 
>>>>> an dnot
>>>>> wait for the restart.
>>>>> And I think not all plugins are able to handle this, TomasB was 
>>>>> once working
>>>>> on it for IPA plugins, but it was not completed as far as I know
>>>> but enabling dynamic plugins can be done without restart, so what 
>>>> can be done is.
>>>> - enable dynamic plugins
>>>> - disable memberof
>>>> - do some work
>>>> - enable memberof
>>>> - disable dynamic plugins
>>> Please see
>>> https://fedorahosted.org/freeipa/ticket/4203#comment:9
>>> I do not think this will be that easy. We would first need to invest 
>>> into
>>> updating FreeIPA plugins to work with dynamic plugins setting and 
>>> then we could
>>> do things alike above.
>>>
>>> It looks like that for FreeIPA 4.4, we will need to live with DS 
>>> restart unless
>>> there is some workaround...
> couldn't the scenario I outline above with enabling dynamic plugins 
> only temporary work, are there any attempts to enable/disable plugins 
> during provisioning ? If that would be the case that would also 
> require a restart
>> One more thing:
>>
>> How does it affect topologies with replicas?
>>
>> I might be wrong, but if memberOf is always computed locally then we 
>> have to
>> disable it on *all* replicas.
>>
>> If we disabled it only on one replica and not others, the chosen 
>> replica would
>> be way faster than rest of the topology and I'm not sure what would 
>> happen
>> later on.
> good point. we exclude memberof from replication as it is regenerated 
> on every server, so each replica would suffer from the performance 
> problem

Right, that is a very good point.
Provisioning will be slow (through direct update or replicated update) 
as soon as memberof is enabled.
An option is to disable memberof only on the server receiving direct 
upates, provisioning will be fast, then let the topology converge with 
slow replication of the provisioned entries (fixup updates will not be 
replicated).
An other option is to disable memberof on all replica, do the 
provisioning. Then run fixup on all the replicas. That means we have a 
mechanism to detect that all provisioned entries have been replicated 
before running fixup.

>>
>> Thierry, Ludwig, can you comment on this?
>>
>