<html>
<head>
<meta content="text/html; charset=windows-1252"
http-equiv="Content-Type">
</head>
<body text="#000000" bgcolor="#FFFFFF">
<br>
Hello,<br>
<br>
Thanks for all the feedbacks. I updated the design accordingly and
with additional tests results
(<a class="moz-txt-link-freetext" href="http://www.freeipa.org/page/V4/Performance_Improvements#Proposed_improvements">http://www.freeipa.org/page/V4/Performance_Improvements#Proposed_improvements</a>)<br>
Several improvements can be done, in particular in DS plugins
(memberof, retroCL), but for "easy" benefit provisioning will be
done with memberof disabled followed by fixup.<br>
<br>
It remains some aspects that are not clear to me:<br>
<ul>
<li>For best performance, DS tuning and provisioning/fixup would
preferably be done under 'directory manager'<br>
That means prompting DM password and writing it into temporary
file. Is that a concern ?</li>
<li>Fixup requires that we know the filters matching the
provisioned entries. For example : <br>
</li>
<ul>
<li>(objectClass=inetorgperson)</li>
<li>(objectClass=ipausergroup) <br>
</li>
<li>(objectClass=ipahost) <br>
</li>
<li>(objectClass=ipahostgroup) <br>
</li>
<li>(objectClass=ipasudorule) <br>
</li>
<li>(objectClass=ipahbacrule)<br>
<br>
The set of objectclass could be hardcode or provided in the
provisioning CLI option<br>
What to do if an entry in in the provision file does not match
any of those filter ? Should it stop without starting the
provisioning ?</li>
</ul>
<li>The CLI doing the provisioning could be something like 'ipa
provision <options>' or should it be a separated command
e.g. ipa-bulk-load ?</li>
</ul>
<p>thanks<br>
thierry<br>
</p>
<div class="moz-cite-prefix">On 05/13/2016 10:18 AM, Ludwig Krispenz
wrote:<br>
</div>
<blockquote cite="mid:57358DC7.9030905@redhat.com" type="cite">
<br>
On 05/13/2016 09:42 AM, Petr Spacek wrote:
<br>
<blockquote type="cite">On 13.5.2016 09:26, Martin Kosek wrote:
<br>
<blockquote type="cite">On 05/12/2016 04:16 PM, Ludwig Krispenz
wrote:
<br>
<blockquote type="cite">On 05/12/2016 03:45 PM, Ludwig
Krispenz wrote:
<br>
<blockquote type="cite">On 05/12/2016 02:16 PM, Petr
Vobornik wrote:
<br>
<blockquote type="cite">On 05/10/2016 05:50 PM, thierry
bordaz wrote:
<br>
<blockquote type="cite">On 05/05/2016 03:44 PM, Petr
Vobornik wrote:
<br>
<blockquote type="cite">On 05/04/2016 02:20 PM,
thierry bordaz wrote:
<br>
<blockquote type="cite">Hello,
<br>
<br>
I have been doing some tests/measures using
<br>
<a class="moz-txt-link-freetext" href="https://github.com/freeipa/freeipa-tools/blob/master/create-test-data.py">https://github.com/freeipa/freeipa-tools/blob/master/create-test-data.py</a>.
<br>
<br>
The tool creates a set of typical
users/hosts/groups... to
<br>
import with a
<br>
ldapadd.
<br>
<br>
I wrote down some finding in
<br>
<a class="moz-txt-link-freetext" href="http://www.freeipa.org/page/V4/Performance_Improvements#Provisioning_throughput_and_DS_plugins">http://www.freeipa.org/page/V4/Performance_Improvements#Provisioning_throughput_and_DS_plugins</a>.
<br>
<br>
<br>
I still have to do some cleanup around the
performance but the
<br>
basic of a
<br>
possible improvement is to do provisioning
in several steps
<br>
(disabling
<br>
plugins, provisioning, enabling plugin,
running fixup tasks).
<br>
<br>
Before going further in the design I wanted
to share those ideas
<br>
and know if
<br>
it raise any concern.
<br>
<br>
thanks
<br>
thierry
<br>
<br>
</blockquote>
Hi Thierry,
<br>
<br>
Thanks for the analysis. Very nice.
<br>
<br>
Knowing this will help us suggesting workarounds
also for old releases.
<br>
<br>
Couple questions:
<br>
<br>
Have you tested retrCL disabled with memberOf
enabled. It seems that it
<br>
would eliminate 550K adds and 0.8M searches. What
would be the time
<br>
improvement?
<br>
<br>
Do you know what is the time when memberof is
enabled but slapi-nis and
<br>
retroCL are disabled?
<br>
</blockquote>
The culprit of the performance issue is very likely
related to SRCH
<br>
(internal) triggered by memberof.
<br>
<br>
If retroCL is disabled and memberof enabled, #SRCH is
13.8M.
<br>
If retroCL is disabled, slapi-nis disabled and
memberof enabled #SRCH is
<br>
14.8
<br>
When all of them are enabled the #SRCH is 15M.
<br>
<br>
You are right if retroCL is disabled the #ADD drops
but it has no
<br>
significant effect on the duration.
<br>
</blockquote>
ok, thanks for the analysis
<br>
<br>
<blockquote type="cite">Regarding the duration of the
provisioning, values are not really stable
<br>
as performance of VM fluctuates. But as soon as
memberof is enabled the
<br>
provisioning lasts > 4hours where the same
provisioning lasts 6mins as
<br>
soon as memberof is disabled.
<br>
<br>
I need to confirm the average time for internal
searches but assuming
<br>
1ms per SRCH it consumes >90% of the provisioning.
<br>
<br>
<br>
<blockquote type="cite"> From the text it was not
clear to me, if you find or investigate
<br>
possible improvements in memberof plugin which would
improve the
<br>
performance without stopping and starting DS.
<br>
</blockquote>
</blockquote>
As was discussed at mtg, have you tried if the DS
restart is really
<br>
necessary?
<br>
</blockquote>
memberof plugin can be enabled and disabled while the
server is running, BUT
<br>
to achieve this the "enable-dynamic-plugins" feature has
to be turned on. And
<br>
then any enable/disable of a plugin would try to do it
dynamically an dnot
<br>
wait for the restart.
<br>
And I think not all plugins are able to handle this,
TomasB was once working
<br>
on it for IPA plugins, but it was not completed as far as
I know
<br>
</blockquote>
but enabling dynamic plugins can be done without restart, so
what can be done is.
<br>
- enable dynamic plugins
<br>
- disable memberof
<br>
- do some work
<br>
- enable memberof
<br>
- disable dynamic plugins
<br>
</blockquote>
Please see
<br>
<a class="moz-txt-link-freetext" href="https://fedorahosted.org/freeipa/ticket/4203#comment:9">https://fedorahosted.org/freeipa/ticket/4203#comment:9</a>
<br>
I do not think this will be that easy. We would first need to
invest into
<br>
updating FreeIPA plugins to work with dynamic plugins setting
and then we could
<br>
do things alike above.
<br>
<br>
It looks like that for FreeIPA 4.4, we will need to live with
DS restart unless
<br>
there is some workaround...
<br>
</blockquote>
</blockquote>
couldn't the scenario I outline above with enabling dynamic
plugins only temporary work, are there any attempts to
enable/disable plugins during provisioning ? If that would be the
case that would also require a restart
<br>
<blockquote type="cite">One more thing:
<br>
<br>
How does it affect topologies with replicas?
<br>
<br>
I might be wrong, but if memberOf is always computed locally
then we have to
<br>
disable it on *all* replicas.
<br>
<br>
If we disabled it only on one replica and not others, the chosen
replica would
<br>
be way faster than rest of the topology and I'm not sure what
would happen
<br>
later on.
<br>
</blockquote>
good point. we exclude memberof from replication as it is
regenerated on every server, so each replica would suffer from the
performance problem
<br>
<blockquote type="cite">
<br>
Thierry, Ludwig, can you comment on this?
<br>
<br>
</blockquote>
<br>
</blockquote>
<br>
</body>
</html>