[Freeipa-users] Slow ipa performance -- why so many ldap lookups ?
Jakub Hrozek
jhrozek at redhat.com
Wed Mar 20 09:44:10 UTC 2013
On Tue, Mar 19, 2013 at 11:05:14PM +0100, Jan-Frode Myklebust wrote:
> On Tue, Mar 19, 2013 at 10:01:16PM +0100, Jakub Hrozek wrote:
> > Hello Jan,
> > I'm sorry you're seeing performance problems.
>
> We have been struggeling with performance and crashes for a while now.
> Have had one crash were a replica dropped it's whole database, and a
> couple of hangs probably caused by clients flooding 389ds.
> (ref ticket 00799719 and 00800931)..
>
> > You should really use the ipa backend for better performance as it uses
> > the memberof attribute (and a couple of other shortcuts to be able to
> > tell if a missing member is a user or a group based on the format of the
> > DN for example).
>
> Sure, we very much want to use the IPA backend, but the LDAP backend seems
> to have been working better for us. More robust. It might have been caused
> by one of the ipa-servers running with too high error-log-level, but we've
> too often seen users not getting their groups populated with the IPA
> backend -- while this has never happened with LDAP backend. We fixed the
> error-log-level today, and have moved our lab-servers over to
> ipa-backend. Will see in a few days if the problem is fixed now.
>
This really sounds like a bug. If you encounter a situation like this,
where a group does not show all its members, feel free to open a bug.
> >
> > > What we find very strange in the trace is:
> > >
> > > - how many ldap searches are done (144!)
> >
> > The number really depends on the group structure and nesting levels.
>
> We have a few nested groups.. but sgallagh explained to me that this
> large number of lookups was caused by me testing using "id", which
> "calls 'initgroups()' followed by a loop of 'getgrgid()' for every group
> you are a member of" and the getgrgid() needs to fetch all members of
> each group.
Yes, on the other hand, with a cold cache, the SSSD should be able to
just download all group members in one go using the dereference control.
I'm not completely sure, but I would presume even this dereference
lookup would show up as a single search in wireshark, not look them up
individually.
>
> >
> > > - that nesting is handled by the client, instead of using
> > > "memberOf".
> >
> > I'm sorry, I don't quite understand the problem here. If ipa backend was
> > used, then all groups would be resolved in a single search by fetching
> > the objects the memberof attribute points at.
>
> I was expecting the same to be done with RFC2307bis, but found no way of
> telling it to use memberof instead of un-nesting all groups by itself.
>
> >
> > On the other hand, the RFC2307bis schema does not guarantee there is a
> > memberof attribute at all, so the client has to perform multiple queries
> > based on the member attribute. This is one of the prime reasons to stick to
> > the ipa backend as opposed to the LDAP back end with the RFC2307bis schema.
>
> Ok.
>
> >
> > > - that all group members are searched individually, and multiple
> > > times if they're members of multiple groups
> > >
> >
> > They shouldn't be fetched multiple times, sounds like a bug to me. How
> > did you measure this metric? Wireshark lookups?
>
> Wireshark lookups. Ref. trace attached to my previous message.
>
Yes, sorry, I saw the attachment after I sent my initial reply :-)
> >
> > Can you tell us a little bit about your nesting structure? How many
> > users, how many groups, how deep is the nesting?
>
> 305 users.
> 58 groups.
> I belive there are max 2 levels of nesting group1(group2(group3)).
>
>
> > By the way, the "id" command is not really a fair benchmark as, contrary
> > to the initgroups() operation that happens during a login, also fetches
> > all the group members. If you are seeing slow logins, then the best way
> > to benchmark the initgroups is "id -G", not "id".
>
> "sudo" will typically hang for many seconds before giving the password-prompt,
> and this delay seems to have been approximately the same as the delay we
> see with "id". Guess that's why I found it to be a good benchmark for
> the performance problems we see.
OK, sudo might need the group members, fair point. I was thinking
logins previously.
Bottom line, if you are seeing inconsistent results with ipa backend,
please open a bug. This is something that would need fixing right away.
More information about the Freeipa-users
mailing list