[Freeipa-devel] user-* commands performance issues

[Petr Vobornik]

On 03/21/2016 01:02 PM, Martin Basti wrote:
>
>
> On 21.03.2016 12:44, Jan Cholasta wrote:
>> On 21.3.2016 11:00, Petr Vobornik wrote:
>>> On 03/17/2016 04:09 PM, Martin Basti wrote:
>>>> Hello all,
>>>>
>>>> I would like to discuss the way how we should improve the speed of
>>>> user-find commands (and other commands too if possible):
>>>>
>>>> 0)
>>>> Do not do extra search for ipasshpubkey. This is clear, patch posted
>>>> for
>>>> review.
>>>> https://fedorahosted.org/freeipa/ticket/3376
>>>>
>>>> commands: user, stageuser, host, idview
>>>>
>>>> 1)
>>>> make --no-members option visible in CLI
>>>> https://fedorahosted.org/freeipa/ticket/4995
>>>
>>> There was a discussion around devconf that --no-members should be a
>>> default behavior of xxx-find commands and I'm for it.
>>
>> +1, although we should be backward compatible with old clients which
>> expect the attributes to be there.
> Ok, I agree to have --no-members as default for *-find commands, but it
> doesn't contradict to exposing --no-member option for all commands.

+1, xxx-show  can have --no-members

>
>>
>>>
>>> Reasoning: use case: 'find me all groups which satisfy this filter'.
>>> Showing members clutters the output(one group with >500 member makes it
>>> unusable) and makes things slow(both on server and CLI side).
>>>
>>> For xxx-show commands it is a question where I don't have a strong
>>> opinion.
>>
>> I think it shouldn't hurt to keep them in -show commands, as there is
>> always only a single entry to process.
> +1
>
>>
>>>
>>>>
>>>> I don't think we should implement also --no-indirect-members, I think
>>>> that this kind of granularity is not needed.
>>>> If --no-members is used, then indirect members will be ignored too.
>>>
>>> +1
>>
>> +1
>>
>>>
>>>>
>>>> commands: all which use members
>>>>
>>>> 2)
>>>> Limit the amount of searches for memberof[indirect] (group, netgroup,
>>>> role, hbacrule, sudorule) and search for each dn only once in find
>>>> commands.
>>>>
>>>> We can have configurable option in default.conf (for example
>>>> memberof_search_limit=100 (0 unlimited)). Find commands will get
>>>> members
>>>> only for specified amount and if this limit is exceeded a warning
>>>> message is shown.
>>>> I do not like this idea much, I think it should be all or nothing, I
>>>> prefer to not do this.
>>
>> +1

I'd also avoid anything special here. But there are sometimes cases when 
the behavior is not good. E.g. a command fails because something is not 
able to get members and you actually don't care about the members. Not 
sure if it is was "fixed"(sizelimit=0). But with new member handling it 
might not be a big issue.

>>
>>>>
>>>> However I like the idea of temporary caching inside find commands,
>>>> where
>>>> each memberof DN is resolved just once and results are cached in a map
>>>> and reused in current context of command. This should be improvement
>>>> mainly for indirect searches, but cache should be faster for direct
>>>> members than doing internal calls of framework objects. This part is
>>>> backward compatible, the first part is not.
>>>>
>>>> https://fedorahosted.org/freeipa/ticket/5282
>>>
>>> What parts of the ticket can be solved with deref plugin? I guess we can
>>> get the CNs, but not what is a direct member. Maybe it should be
>>> discussed separately.
>>
>> Indirect members are already resolved by a single LDAP search. What
>> kind of additional optimization would you like to do for them?
>
> We can use deref plugin to get pkeys from one search in case that pkeys
> are not part of DN. (I have to investigate if it is worth to do for
> user-find, I'm not sure if any memberof attributes have pkey that is not
> part of DN)

sudo rule, hbac rule

>
> For indirect members, it is one search per entry, but for 1000 users, it
> is 1000 searches and I would like to have just one for the particular
> indirect member.

are we talking about user-find? If so then it is mostly solved with 
default --no-member style behavior.

But if a user or a group is  directly/indirectly a member of a lot of 
groups(1000) then it might become slow. But caching won't probably help 
much, not sure.

>
>
>>
>>>
>>>>
>>>> commands: user-find, stageuser-find, possibly all find commands
>>>>
>>>> 3)
>>>> Remove userPassword, krbPrincipalKey from search results
>>>> This change is not backward compatible, can we do this?
>>>>
>>>> https://fedorahosted.org/freeipa/ticket/5281
>>>>
>>>> commands: user-find
>>>
>>> I'm for it, would like to hear other opinions.
>>>
>>> Note: it should be only in user-find commands. 'show' has to display it.
>>
>> +1
>>
>


-- 
Petr Vobornik