[Freeipa-devel] Provisioning throughput

[Ludwig Krispenz]

On 05/12/2016 02:16 PM, Petr Vobornik wrote:
> On 05/10/2016 05:50 PM, thierry bordaz wrote:
>>
>> On 05/05/2016 03:44 PM, Petr Vobornik wrote:
>>> On 05/04/2016 02:20 PM, thierry bordaz wrote:
>>>> Hello,
>>>>
>>>>       I have been doing some tests/measures using
>>>>      
>>>> https://github.com/freeipa/freeipa-tools/blob/master/create-test-data.py.
>>>>
>>>>       The tool creates a set of typical users/hosts/groups... to
>>>> import with a
>>>>       ldapadd.
>>>>
>>>>       I wrote down some finding in
>>>>      
>>>> http://www.freeipa.org/page/V4/Performance_Improvements#Provisioning_throughput_and_DS_plugins.
>>>>
>>>>       I still have to do some cleanup around the performance but the
>>>> basic of a
>>>>       possible improvement is to do provisioning in several steps
>>>> (disabling
>>>>       plugins, provisioning, enabling plugin, running fixup tasks).
>>>>
>>>>       Before going further in the design I wanted to share those ideas
>>>> and know if
>>>>       it raise any concern.
>>>>
>>>>       thanks
>>>>       thierry
>>>>
>>> Hi Thierry,
>>>
>>> Thanks for the analysis. Very nice.
>>>
>>> Knowing this will help us suggesting workarounds also for old releases.
>>>
>>> Couple questions:
>>>
>>> Have you tested retrCL disabled with memberOf enabled. It seems that it
>>> would eliminate 550K adds and 0.8M searches. What would be the time
>>> improvement?
>>>
>>> Do you know what is the time when memberof is enabled but slapi-nis and
>>> retroCL are disabled?
>> The culprit of the performance issue is very likely related to SRCH
>> (internal) triggered by memberof.
>>
>> If retroCL is disabled and memberof enabled, #SRCH is 13.8M.
>> If retroCL is disabled, slapi-nis disabled and memberof enabled #SRCH is
>> 14.8
>> When all of them are enabled the #SRCH is 15M.
>>
>> You are right if retroCL is disabled the #ADD drops but it has no
>> significant effect on the duration.
> ok, thanks for the analysis
>
>> Regarding the duration of the provisioning, values are not really stable
>> as performance of VM fluctuates. But as soon as memberof is enabled the
>> provisioning lasts > 4hours where the same provisioning lasts 6mins as
>> soon as memberof is disabled.
>>
>> I need to confirm the average time for internal searches but assuming
>> 1ms per SRCH it consumes >90% of the provisioning.
>>
>>
>>>   From the text it was not clear to me, if you find or investigate
>>> possible improvements in memberof plugin which would improve the
>>> performance without stopping and starting DS.
> As was discussed at mtg, have you tried if the DS restart is really
> necessary?
memberof plugin can be enabled and disabled while the server is running, BUT
to achieve this the "enable-dynamic-plugins" feature has to be turned 
on. And then any enable/disable of a plugin would try to do it 
dynamically an dnot wait for the restart.
And I think not all plugins are able to handle this, TomasB was once 
working on it for IPA plugins, but it was not completed as far as I know
>
> And if it is required, what would be needed to not require restart.
>
> The workaround should be easy to use.

-- 
Red Hat GmbH, http://www.de.redhat.com/, Registered seat: Grasbrunn,
Commercial register: Amtsgericht Muenchen, HRB 153243,
Managing Directors: Paul Argiry, Charles Cachera, Michael Cunningham, Michael O'Neill