[Freeipa-devel] Provisioning throughput

[Ludwig Krispenz]

On 05/13/2016 09:42 AM, Petr Spacek wrote:
> On 13.5.2016 09:26, Martin Kosek wrote:
>> On 05/12/2016 04:16 PM, Ludwig Krispenz wrote:
>>> On 05/12/2016 03:45 PM, Ludwig Krispenz wrote:
>>>> On 05/12/2016 02:16 PM, Petr Vobornik wrote:
>>>>> On 05/10/2016 05:50 PM, thierry bordaz wrote:
>>>>>> On 05/05/2016 03:44 PM, Petr Vobornik wrote:
>>>>>>> On 05/04/2016 02:20 PM, thierry bordaz wrote:
>>>>>>>> Hello,
>>>>>>>>
>>>>>>>>        I have been doing some tests/measures using
>>>>>>>> https://github.com/freeipa/freeipa-tools/blob/master/create-test-data.py.
>>>>>>>>
>>>>>>>>        The tool creates a set of typical users/hosts/groups... to
>>>>>>>> import with a
>>>>>>>>        ldapadd.
>>>>>>>>
>>>>>>>>        I wrote down some finding in
>>>>>>>> http://www.freeipa.org/page/V4/Performance_Improvements#Provisioning_throughput_and_DS_plugins.
>>>>>>>>
>>>>>>>>
>>>>>>>>        I still have to do some cleanup around the performance but the
>>>>>>>> basic of a
>>>>>>>>        possible improvement is to do provisioning in several steps
>>>>>>>> (disabling
>>>>>>>>        plugins, provisioning, enabling plugin, running fixup tasks).
>>>>>>>>
>>>>>>>>        Before going further in the design I wanted to share those ideas
>>>>>>>> and know if
>>>>>>>>        it raise any concern.
>>>>>>>>
>>>>>>>>        thanks
>>>>>>>>        thierry
>>>>>>>>
>>>>>>> Hi Thierry,
>>>>>>>
>>>>>>> Thanks for the analysis. Very nice.
>>>>>>>
>>>>>>> Knowing this will help us suggesting workarounds also for old releases.
>>>>>>>
>>>>>>> Couple questions:
>>>>>>>
>>>>>>> Have you tested retrCL disabled with memberOf enabled. It seems that it
>>>>>>> would eliminate 550K adds and 0.8M searches. What would be the time
>>>>>>> improvement?
>>>>>>>
>>>>>>> Do you know what is the time when memberof is enabled but slapi-nis and
>>>>>>> retroCL are disabled?
>>>>>> The culprit of the performance issue is very likely related to SRCH
>>>>>> (internal) triggered by memberof.
>>>>>>
>>>>>> If retroCL is disabled and memberof enabled, #SRCH is 13.8M.
>>>>>> If retroCL is disabled, slapi-nis disabled and memberof enabled #SRCH is
>>>>>> 14.8
>>>>>> When all of them are enabled the #SRCH is 15M.
>>>>>>
>>>>>> You are right if retroCL is disabled the #ADD drops but it has no
>>>>>> significant effect on the duration.
>>>>> ok, thanks for the analysis
>>>>>
>>>>>> Regarding the duration of the provisioning, values are not really stable
>>>>>> as performance of VM fluctuates. But as soon as memberof is enabled the
>>>>>> provisioning lasts > 4hours where the same provisioning lasts 6mins as
>>>>>> soon as memberof is disabled.
>>>>>>
>>>>>> I need to confirm the average time for internal searches but assuming
>>>>>> 1ms per SRCH it consumes >90% of the provisioning.
>>>>>>
>>>>>>
>>>>>>>    From the text it was not clear to me, if you find or investigate
>>>>>>> possible improvements in memberof plugin which would improve the
>>>>>>> performance without stopping and starting DS.
>>>>> As was discussed at mtg, have you tried if the DS restart is really
>>>>> necessary?
>>>> memberof plugin can be enabled and disabled while the server is running, BUT
>>>> to achieve this the "enable-dynamic-plugins" feature has to be turned on. And
>>>> then any enable/disable of a plugin would try to do it dynamically an dnot
>>>> wait for the restart.
>>>> And I think not all plugins are able to handle this, TomasB was once working
>>>> on it for IPA plugins, but it was not completed as far as I know
>>> but enabling dynamic plugins can be done without restart, so what can be done is.
>>> - enable dynamic plugins
>>> - disable memberof
>>> - do some work
>>> - enable memberof
>>> - disable dynamic plugins
>> Please see
>> https://fedorahosted.org/freeipa/ticket/4203#comment:9
>> I do not think this will be that easy. We would first need to invest into
>> updating FreeIPA plugins to work with dynamic plugins setting and then we could
>> do things alike above.
>>
>> It looks like that for FreeIPA 4.4, we will need to live with DS restart unless
>> there is some workaround...
couldn't the scenario I outline above with enabling dynamic plugins only 
temporary work, are there any attempts to enable/disable plugins during 
provisioning ? If that would be the case that would also require a restart
> One more thing:
>
> How does it affect topologies with replicas?
>
> I might be wrong, but if memberOf is always computed locally then we have to
> disable it on *all* replicas.
>
> If we disabled it only on one replica and not others, the chosen replica would
> be way faster than rest of the topology and I'm not sure what would happen
> later on.
good point. we exclude memberof from replication as it is regenerated on 
every server, so each replica would suffer from the performance problem
>
> Thierry, Ludwig, can you comment on this?
>

-- 
Red Hat GmbH, http://www.de.redhat.com/, Registered seat: Grasbrunn,
Commercial register: Amtsgericht Muenchen, HRB 153243,
Managing Directors: Paul Argiry, Charles Cachera, Michael Cunningham, Michael O'Neill