[Freeipa-devel] user-* commands performance issues

[Martin Basti]

On 21.03.2016 12:44, Jan Cholasta wrote:
> On 21.3.2016 11:00, Petr Vobornik wrote:
>> On 03/17/2016 04:09 PM, Martin Basti wrote:
>>> Hello all,
>>>
>>> I would like to discuss the way how we should improve the speed of
>>> user-find commands (and other commands too if possible):
>>>
>>> 0)
>>> Do not do extra search for ipasshpubkey. This is clear, patch posted 
>>> for
>>> review.
>>> https://fedorahosted.org/freeipa/ticket/3376
>>>
>>> commands: user, stageuser, host, idview
>>>
>>> 1)
>>> make --no-members option visible in CLI
>>> https://fedorahosted.org/freeipa/ticket/4995
>>
>> There was a discussion around devconf that --no-members should be a
>> default behavior of xxx-find commands and I'm for it.
>
> +1, although we should be backward compatible with old clients which 
> expect the attributes to be there.
Ok, I agree to have --no-members as default for *-find commands, but it 
doesn't contradict to exposing --no-member option for all commands.

>
>>
>> Reasoning: use case: 'find me all groups which satisfy this filter'.
>> Showing members clutters the output(one group with >500 member makes it
>> unusable) and makes things slow(both on server and CLI side).
>>
>> For xxx-show commands it is a question where I don't have a strong 
>> opinion.
>
> I think it shouldn't hurt to keep them in -show commands, as there is 
> always only a single entry to process.
+1

>
>>
>>>
>>> I don't think we should implement also --no-indirect-members, I think
>>> that this kind of granularity is not needed.
>>> If --no-members is used, then indirect members will be ignored too.
>>
>> +1
>
> +1
>
>>
>>>
>>> commands: all which use members
>>>
>>> 2)
>>> Limit the amount of searches for memberof[indirect] (group, netgroup,
>>> role, hbacrule, sudorule) and search for each dn only once in find
>>> commands.
>>>
>>> We can have configurable option in default.conf (for example
>>> memberof_search_limit=100 (0 unlimited)). Find commands will get 
>>> members
>>> only for specified amount and if this limit is exceeded a warning
>>> message is shown.
>>> I do not like this idea much, I think it should be all or nothing, I
>>> prefer to not do this.
>
> +1
>
>>>
>>> However I like the idea of temporary caching inside find commands, 
>>> where
>>> each memberof DN is resolved just once and results are cached in a map
>>> and reused in current context of command. This should be improvement
>>> mainly for indirect searches, but cache should be faster for direct
>>> members than doing internal calls of framework objects. This part is
>>> backward compatible, the first part is not.
>>>
>>> https://fedorahosted.org/freeipa/ticket/5282
>>
>> What parts of the ticket can be solved with deref plugin? I guess we can
>> get the CNs, but not what is a direct member. Maybe it should be
>> discussed separately.
>
> Indirect members are already resolved by a single LDAP search. What 
> kind of additional optimization would you like to do for them?

We can use deref plugin to get pkeys from one search in case that pkeys 
are not part of DN. (I have to investigate if it is worth to do for 
user-find, I'm not sure if any memberof attributes have pkey that is not 
part of DN)

For indirect members, it is one search per entry, but for 1000 users, it 
is 1000 searches and I would like to have just one for the particular 
indirect member.


>
>>
>>>
>>> commands: user-find, stageuser-find, possibly all find commands
>>>
>>> 3)
>>> Remove userPassword, krbPrincipalKey from search results
>>> This change is not backward compatible, can we do this?
>>>
>>> https://fedorahosted.org/freeipa/ticket/5281
>>>
>>> commands: user-find
>>
>> I'm for it, would like to hear other opinions.
>>
>> Note: it should be only in user-find commands. 'show' has to display it.
>
> +1
>