[Freeipa-devel] Provisioning throughput
thierry bordaz
tbordaz at redhat.com
Wed May 25 19:25:20 UTC 2016
On 05/25/2016 08:49 PM, Rob Crittenden wrote:
> thierry bordaz wrote:
>>
>> Hello,
>>
>> Thanks for all the feedbacks. I updated the design accordingly and with
>> additional tests results
>> (http://www.freeipa.org/page/V4/Performance_Improvements#Proposed_improvements)
>>
>> Several improvements can be done, in particular in DS plugins (memberof,
>> retroCL), but for "easy" benefit provisioning will be done with memberof
>> disabled followed by fixup.
>>
>> It remains some aspects that are not clear to me:
>>
>> * For best performance, DS tuning and provisioning/fixup would
>> preferably be done under 'directory manager'
>> That means prompting DM password and writing it into temporary file.
>> Is that a concern ?
>> * Fixup requires that we know the filters matching the provisioned
>> entries. For example :
>> o (objectClass=inetorgperson)
>> o (objectClass=ipausergroup)
>> o (objectClass=ipahost)
>> o (objectClass=ipahostgroup)
>> o (objectClass=ipasudorule)
>> o (objectClass=ipahbacrule)
>>
>> The set of objectclass could be hardcode or provided in the
>> provisioning CLI option
>> What to do if an entry in in the provision file does not match
>> any of those filter ? Should it stop without starting the
>> provisioning ?
>> * The CLI doing the provisioning could be something like 'ipa
>> provision <options>' or should it be a separated command e.g.
>> ipa-bulk-load ?
>
> It depends. There is a migration command now, ipa migrate-ds, that
> adds records and is impacted by this. There is also the possibility of
> looping calls to ipa [user|group|etc]-add.
I agree that migration and bulk load can be linked. If migration
dump/update a set of entries before filling them into a new instance it
could use bulk load.
For set loop of ipa <object>-add, I think they add many others direct
operations (mainly SRCH) before doing the ADD in order to check
coherency. bulk load looks more straightforward.
>
> Would it be reasonable to require bulk import to be done on an IPA
> master so we can leverage the ldapi socket?
Do you mean using ldapi to reduce network latency or automember or
something else ?
thanks
theirry
>
> rob
>
>>
>> thanks
>> thierry
>>
>> On 05/13/2016 10:18 AM, Ludwig Krispenz wrote:
>>>
>>> On 05/13/2016 09:42 AM, Petr Spacek wrote:
>>>> On 13.5.2016 09:26, Martin Kosek wrote:
>>>>> On 05/12/2016 04:16 PM, Ludwig Krispenz wrote:
>>>>>> On 05/12/2016 03:45 PM, Ludwig Krispenz wrote:
>>>>>>> On 05/12/2016 02:16 PM, Petr Vobornik wrote:
>>>>>>>> On 05/10/2016 05:50 PM, thierry bordaz wrote:
>>>>>>>>> On 05/05/2016 03:44 PM, Petr Vobornik wrote:
>>>>>>>>>> On 05/04/2016 02:20 PM, thierry bordaz wrote:
>>>>>>>>>>> Hello,
>>>>>>>>>>>
>>>>>>>>>>> I have been doing some tests/measures using
>>>>>>>>>>> https://github.com/freeipa/freeipa-tools/blob/master/create-test-data.py.
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> The tool creates a set of typical
>>>>>>>>>>> users/hosts/groups... to
>>>>>>>>>>> import with a
>>>>>>>>>>> ldapadd.
>>>>>>>>>>>
>>>>>>>>>>> I wrote down some finding in
>>>>>>>>>>> http://www.freeipa.org/page/V4/Performance_Improvements#Provisioning_throughput_and_DS_plugins.
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> I still have to do some cleanup around the performance
>>>>>>>>>>> but the
>>>>>>>>>>> basic of a
>>>>>>>>>>> possible improvement is to do provisioning in several
>>>>>>>>>>> steps
>>>>>>>>>>> (disabling
>>>>>>>>>>> plugins, provisioning, enabling plugin, running fixup
>>>>>>>>>>> tasks).
>>>>>>>>>>>
>>>>>>>>>>> Before going further in the design I wanted to share
>>>>>>>>>>> those ideas
>>>>>>>>>>> and know if
>>>>>>>>>>> it raise any concern.
>>>>>>>>>>>
>>>>>>>>>>> thanks
>>>>>>>>>>> thierry
>>>>>>>>>>>
>>>>>>>>>> Hi Thierry,
>>>>>>>>>>
>>>>>>>>>> Thanks for the analysis. Very nice.
>>>>>>>>>>
>>>>>>>>>> Knowing this will help us suggesting workarounds also for old
>>>>>>>>>> releases.
>>>>>>>>>>
>>>>>>>>>> Couple questions:
>>>>>>>>>>
>>>>>>>>>> Have you tested retrCL disabled with memberOf enabled. It seems
>>>>>>>>>> that it
>>>>>>>>>> would eliminate 550K adds and 0.8M searches. What would be the
>>>>>>>>>> time
>>>>>>>>>> improvement?
>>>>>>>>>>
>>>>>>>>>> Do you know what is the time when memberof is enabled but
>>>>>>>>>> slapi-nis and
>>>>>>>>>> retroCL are disabled?
>>>>>>>>> The culprit of the performance issue is very likely related to
>>>>>>>>> SRCH
>>>>>>>>> (internal) triggered by memberof.
>>>>>>>>>
>>>>>>>>> If retroCL is disabled and memberof enabled, #SRCH is 13.8M.
>>>>>>>>> If retroCL is disabled, slapi-nis disabled and memberof enabled
>>>>>>>>> #SRCH is
>>>>>>>>> 14.8
>>>>>>>>> When all of them are enabled the #SRCH is 15M.
>>>>>>>>>
>>>>>>>>> You are right if retroCL is disabled the #ADD drops but it has no
>>>>>>>>> significant effect on the duration.
>>>>>>>> ok, thanks for the analysis
>>>>>>>>
>>>>>>>>> Regarding the duration of the provisioning, values are not
>>>>>>>>> really stable
>>>>>>>>> as performance of VM fluctuates. But as soon as memberof is
>>>>>>>>> enabled the
>>>>>>>>> provisioning lasts > 4hours where the same provisioning lasts
>>>>>>>>> 6mins as
>>>>>>>>> soon as memberof is disabled.
>>>>>>>>>
>>>>>>>>> I need to confirm the average time for internal searches but
>>>>>>>>> assuming
>>>>>>>>> 1ms per SRCH it consumes >90% of the provisioning.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>> From the text it was not clear to me, if you find or
>>>>>>>>>> investigate
>>>>>>>>>> possible improvements in memberof plugin which would improve the
>>>>>>>>>> performance without stopping and starting DS.
>>>>>>>> As was discussed at mtg, have you tried if the DS restart is
>>>>>>>> really
>>>>>>>> necessary?
>>>>>>> memberof plugin can be enabled and disabled while the server is
>>>>>>> running, BUT
>>>>>>> to achieve this the "enable-dynamic-plugins" feature has to be
>>>>>>> turned on. And
>>>>>>> then any enable/disable of a plugin would try to do it dynamically
>>>>>>> an dnot
>>>>>>> wait for the restart.
>>>>>>> And I think not all plugins are able to handle this, TomasB was
>>>>>>> once working
>>>>>>> on it for IPA plugins, but it was not completed as far as I know
>>>>>> but enabling dynamic plugins can be done without restart, so what
>>>>>> can be done is.
>>>>>> - enable dynamic plugins
>>>>>> - disable memberof
>>>>>> - do some work
>>>>>> - enable memberof
>>>>>> - disable dynamic plugins
>>>>> Please see
>>>>> https://fedorahosted.org/freeipa/ticket/4203#comment:9
>>>>> I do not think this will be that easy. We would first need to invest
>>>>> into
>>>>> updating FreeIPA plugins to work with dynamic plugins setting and
>>>>> then we could
>>>>> do things alike above.
>>>>>
>>>>> It looks like that for FreeIPA 4.4, we will need to live with DS
>>>>> restart unless
>>>>> there is some workaround...
>>> couldn't the scenario I outline above with enabling dynamic plugins
>>> only temporary work, are there any attempts to enable/disable plugins
>>> during provisioning ? If that would be the case that would also
>>> require a restart
>>>> One more thing:
>>>>
>>>> How does it affect topologies with replicas?
>>>>
>>>> I might be wrong, but if memberOf is always computed locally then we
>>>> have to
>>>> disable it on *all* replicas.
>>>>
>>>> If we disabled it only on one replica and not others, the chosen
>>>> replica would
>>>> be way faster than rest of the topology and I'm not sure what would
>>>> happen
>>>> later on.
>>> good point. we exclude memberof from replication as it is regenerated
>>> on every server, so each replica would suffer from the performance
>>> problem
>>>>
>>>> Thierry, Ludwig, can you comment on this?
>>>>
>>>
>>
>>
>>
>
More information about the Freeipa-devel
mailing list