[Freeipa-devel] user-* commands performance issues

[Martin Basti]

On 21.03.2016 15:50, Petr Vobornik wrote:
> On 03/21/2016 01:02 PM, Martin Basti wrote:
>>
>>
>> On 21.03.2016 12:44, Jan Cholasta wrote:
>>> On 21.3.2016 11:00, Petr Vobornik wrote:
>>>> On 03/17/2016 04:09 PM, Martin Basti wrote:
>>>>> Hello all,
>>>>>
>>>>> I would like to discuss the way how we should improve the speed of
>>>>> user-find commands (and other commands too if possible):
>>>>>
>>>>> 0)
>>>>> Do not do extra search for ipasshpubkey. This is clear, patch posted
>>>>> for
>>>>> review.
>>>>> https://fedorahosted.org/freeipa/ticket/3376
>>>>>
>>>>> commands: user, stageuser, host, idview
>>>>>
>>>>> 1)
>>>>> make --no-members option visible in CLI
>>>>> https://fedorahosted.org/freeipa/ticket/4995
>>>>
>>>> There was a discussion around devconf that --no-members should be a
>>>> default behavior of xxx-find commands and I'm for it.
>>>
>>> +1, although we should be backward compatible with old clients which
>>> expect the attributes to be there.
>> Ok, I agree to have --no-members as default for *-find commands, but it
>> doesn't contradict to exposing --no-member option for all commands.
>
> +1, xxx-show  can have --no-members
>
>>
>>>
>>>>
>>>> Reasoning: use case: 'find me all groups which satisfy this filter'.
>>>> Showing members clutters the output(one group with >500 member 
>>>> makes it
>>>> unusable) and makes things slow(both on server and CLI side).
>>>>
>>>> For xxx-show commands it is a question where I don't have a strong
>>>> opinion.
>>>
>>> I think it shouldn't hurt to keep them in -show commands, as there is
>>> always only a single entry to process.
>> +1
>>
>>>
>>>>
>>>>>
>>>>> I don't think we should implement also --no-indirect-members, I think
>>>>> that this kind of granularity is not needed.
>>>>> If --no-members is used, then indirect members will be ignored too.
>>>>
>>>> +1
>>>
>>> +1
>>>
>>>>
>>>>>
>>>>> commands: all which use members
>>>>>
>>>>> 2)
>>>>> Limit the amount of searches for memberof[indirect] (group, netgroup,
>>>>> role, hbacrule, sudorule) and search for each dn only once in find
>>>>> commands.
>>>>>
>>>>> We can have configurable option in default.conf (for example
>>>>> memberof_search_limit=100 (0 unlimited)). Find commands will get
>>>>> members
>>>>> only for specified amount and if this limit is exceeded a warning
>>>>> message is shown.
>>>>> I do not like this idea much, I think it should be all or nothing, I
>>>>> prefer to not do this.
>>>
>>> +1
>
> I'd also avoid anything special here. But there are sometimes cases 
> when the behavior is not good. E.g. a command fails because something 
> is not able to get members and you actually don't care about the 
> members. Not sure if it is was "fixed"(sizelimit=0). But with new 
> member handling it might not be a big issue.
>
>>>
>>>>>
>>>>> However I like the idea of temporary caching inside find commands,
>>>>> where
>>>>> each memberof DN is resolved just once and results are cached in a 
>>>>> map
>>>>> and reused in current context of command. This should be improvement
>>>>> mainly for indirect searches, but cache should be faster for direct
>>>>> members than doing internal calls of framework objects. This part is
>>>>> backward compatible, the first part is not.
>>>>>
>>>>> https://fedorahosted.org/freeipa/ticket/5282
>>>>
>>>> What parts of the ticket can be solved with deref plugin? I guess 
>>>> we can
>>>> get the CNs, but not what is a direct member. Maybe it should be
>>>> discussed separately.
>>>
>>> Indirect members are already resolved by a single LDAP search. What
>>> kind of additional optimization would you like to do for them?
>>
>> We can use deref plugin to get pkeys from one search in case that pkeys
>> are not part of DN. (I have to investigate if it is worth to do for
>> user-find, I'm not sure if any memberof attributes have pkey that is not
>> part of DN)
>
> sudo rule, hbac rule
>
>>
>> For indirect members, it is one search per entry, but for 1000 users, it
>> is 1000 searches and I would like to have just one for the particular
>> indirect member.
>
> are we talking about user-find? If so then it is mostly solved with 
> default --no-member style behavior.
>
> But if a user or a group is  directly/indirectly a member of a lot of 
> groups(1000) then it might become slow. But caching won't probably 
> help much, not sure.
In case that you have one user in 1000 indirect groups, it will not 
help, but if you have 500 users in the same 1000 indirect groups, it 
will be ~500 times faster than without caching.

Also we have to find out how to solve case that a user wants to show 
members for *-find commands.
is enough to show members when --all flag is used, or should we add 
separate option to show members in find command (--members, --get-members)?

Martin^2
>
>>
>>
>>>
>>>>
>>>>>
>>>>> commands: user-find, stageuser-find, possibly all find commands
>>>>>
>>>>> 3)
>>>>> Remove userPassword, krbPrincipalKey from search results
>>>>> This change is not backward compatible, can we do this?
>>>>>
>>>>> https://fedorahosted.org/freeipa/ticket/5281
>>>>>
>>>>> commands: user-find
>>>>
>>>> I'm for it, would like to hear other opinions.
>>>>
>>>> Note: it should be only in user-find commands. 'show' has to 
>>>> display it.
>>>
>>> +1
>>>
>>
>
>