[Freeipa-devel] Provisioning throughput

[Ludwig Krispenz]

On 05/12/2016 03:45 PM, Ludwig Krispenz wrote:
>
> On 05/12/2016 02:16 PM, Petr Vobornik wrote:
>> On 05/10/2016 05:50 PM, thierry bordaz wrote:
>>>
>>> On 05/05/2016 03:44 PM, Petr Vobornik wrote:
>>>> On 05/04/2016 02:20 PM, thierry bordaz wrote:
>>>>> Hello,
>>>>>
>>>>>       I have been doing some tests/measures using
>>>>> https://github.com/freeipa/freeipa-tools/blob/master/create-test-data.py.
>>>>>
>>>>>       The tool creates a set of typical users/hosts/groups... to
>>>>> import with a
>>>>>       ldapadd.
>>>>>
>>>>>       I wrote down some finding in
>>>>> http://www.freeipa.org/page/V4/Performance_Improvements#Provisioning_throughput_and_DS_plugins.
>>>>>
>>>>>       I still have to do some cleanup around the performance but the
>>>>> basic of a
>>>>>       possible improvement is to do provisioning in several steps
>>>>> (disabling
>>>>>       plugins, provisioning, enabling plugin, running fixup tasks).
>>>>>
>>>>>       Before going further in the design I wanted to share those 
>>>>> ideas
>>>>> and know if
>>>>>       it raise any concern.
>>>>>
>>>>>       thanks
>>>>>       thierry
>>>>>
>>>> Hi Thierry,
>>>>
>>>> Thanks for the analysis. Very nice.
>>>>
>>>> Knowing this will help us suggesting workarounds also for old 
>>>> releases.
>>>>
>>>> Couple questions:
>>>>
>>>> Have you tested retrCL disabled with memberOf enabled. It seems 
>>>> that it
>>>> would eliminate 550K adds and 0.8M searches. What would be the time
>>>> improvement?
>>>>
>>>> Do you know what is the time when memberof is enabled but slapi-nis 
>>>> and
>>>> retroCL are disabled?
>>> The culprit of the performance issue is very likely related to SRCH
>>> (internal) triggered by memberof.
>>>
>>> If retroCL is disabled and memberof enabled, #SRCH is 13.8M.
>>> If retroCL is disabled, slapi-nis disabled and memberof enabled 
>>> #SRCH is
>>> 14.8
>>> When all of them are enabled the #SRCH is 15M.
>>>
>>> You are right if retroCL is disabled the #ADD drops but it has no
>>> significant effect on the duration.
>> ok, thanks for the analysis
>>
>>> Regarding the duration of the provisioning, values are not really 
>>> stable
>>> as performance of VM fluctuates. But as soon as memberof is enabled the
>>> provisioning lasts > 4hours where the same provisioning lasts 6mins as
>>> soon as memberof is disabled.
>>>
>>> I need to confirm the average time for internal searches but assuming
>>> 1ms per SRCH it consumes >90% of the provisioning.
>>>
>>>
>>>>   From the text it was not clear to me, if you find or investigate
>>>> possible improvements in memberof plugin which would improve the
>>>> performance without stopping and starting DS.
>> As was discussed at mtg, have you tried if the DS restart is really
>> necessary?
> memberof plugin can be enabled and disabled while the server is 
> running, BUT
> to achieve this the "enable-dynamic-plugins" feature has to be turned 
> on. And then any enable/disable of a plugin would try to do it 
> dynamically an dnot wait for the restart.
> And I think not all plugins are able to handle this, TomasB was once 
> working on it for IPA plugins, but it was not completed as far as I know
but enabling dynamic plugins can be done without restart, so what can be 
done is.
- enable dynamic plugins
- disable memberof
- do some work
- enable memberof
- disable dynamic plugins

>>
>> And if it is required, what would be needed to not require restart.
>>
>> The workaround should be easy to use.
>

-- 
Red Hat GmbH, http://www.de.redhat.com/, Registered seat: Grasbrunn,
Commercial register: Amtsgericht Muenchen, HRB 153243,
Managing Directors: Paul Argiry, Charles Cachera, Michael Cunningham, Michael O'Neill