[Freeipa-users] Slow ipa performance -- why so many ldap lookups ?

[Jan-Frode Myklebust]

On Tue, Mar 19, 2013 at 10:01:16PM +0100, Jakub Hrozek wrote:
> Hello Jan,
> I'm sorry you're seeing performance problems. 

We have been struggeling with performance and crashes for a while now.
Have had one crash were a replica dropped it's whole database, and a
couple of hangs probably caused by clients flooding 389ds. 
(ref ticket 00799719 and 00800931)..

> You should really use the ipa backend for better performance as it uses
> the memberof attribute (and a couple of other shortcuts to be able to
> tell if a missing member is a user or a group based on the format of the
> DN for example).

Sure, we very much want to use the IPA backend, but the LDAP backend seems
to have been working better for us. More robust. It might have been caused
by one of the ipa-servers running with too high error-log-level, but we've
too often seen users not getting their groups populated with the IPA
backend -- while this has never happened with LDAP backend. We fixed the
error-log-level today, and have moved our lab-servers over to
ipa-backend. Will see in a few days if the problem is fixed now.

> 
> > What we find very strange in the trace is:
> > 
> > 	- how many ldap searches are done (144!)
> 
> The number really depends on the group structure and nesting levels.

We have a few nested groups.. but sgallagh explained to me that this
large number of lookups was caused by me testing using "id", which
"calls 'initgroups()' followed by a loop of 'getgrgid()' for every group
you are a member of" and the getgrgid() needs to fetch all members of
each group.

> 
> > 	- that nesting is handled by the client, instead of using
> > 	  "memberOf".
> 
> I'm sorry, I don't quite understand the problem here. If ipa backend was
> used, then all groups would be resolved in a single search by fetching
> the objects the memberof attribute points at.

I was expecting the same to be done with RFC2307bis, but found no way of
telling it to use memberof instead of un-nesting all groups by itself.

> 
> On the other hand, the RFC2307bis schema does not guarantee there is a
> memberof attribute at all, so the client has to perform multiple queries
> based on the member attribute. This is one of the prime reasons to stick to
> the ipa backend as opposed to the LDAP back end with the RFC2307bis schema.

Ok.

> 
> > 	- that all group members are searched individually, and multiple
> > 	  times if they're members of multiple groups
> > 
> 
> They shouldn't be fetched multiple times, sounds like a bug to me. How
> did you measure this metric? Wireshark lookups? 

Wireshark lookups. Ref. trace attached to my previous message.

> 
> Can you tell us a little bit about your nesting structure? How many
> users, how many groups, how deep is the nesting?

305 users.
58 groups.
I belive there are max 2 levels of nesting group1(group2(group3)).


> By the way, the "id" command is not really a fair benchmark as, contrary
> to the initgroups() operation that happens during a login, also fetches
> all the group members. If you are seeing slow logins, then the best way
> to benchmark the initgroups is "id -G", not "id".

"sudo" will typically hang for many seconds before giving the password-prompt,
and this delay seems to have been approximately the same as the delay we
see with "id". Guess that's why I found it to be a good benchmark for
the performance problems we see.



  -jf