[Freeipa-users] Large slow down when using IPA

[Simo Sorce]

On Mon, 2012-01-02 at 15:52 -0900, Erinn Looney-Triggs wrote:
> On 01/02/2012 01:47 PM, Simo Sorce wrote:
>  
> > Hi Erinn,
> > can you please tell what's the baseline you are comparing against ?
> > 
> > Is it nss_ldap ? With or without nscd ?
> > 
> > Simo.
> > 
> 
> 
> Here is what I am comparing, and hoping not to make myself look like an
> idiot in the process :).

First of all, thanks for the full explanation, it makes the situation
much more clear.

> ipa-client system without excludes configured in sssd.
> ipa-client system with excludes for users and groups < id 500 configured
> in sssd.
> Finally a non ipa-client system, as in local only.
> 
> These tests are all being done on the same system. All I have at this
> point is the fact that I started rolling out IPA (and thus sssd) to
> clients and AIDE checks bombed in terms of the amount of time they took.
> That and accurate bacula backups failed because it was taking too long
> to compare the local files to the db. Now it could be unrelated, but I
> hope not (in the not looking like an idiot category).
> 
> Now of course network traffic and lag times are involved here, but the
> one key point that made me take notice here is that the times for an
> AIDE check also bombed (at about the same ratio) on the IPA servers
> themselves, thus pointing me to the idea that perhaps this wasn't all
> network related.

Indeed it is not all network related, sssd caches a lot of information
so that most lookups should be 'local'.

Now here comes the gotcha in comparing against local files access.
The way sssd currently works each request goes through a pipe to the
sssd_nss process, while with local files glibc just reads entries off
the /etc/passwd file.

I wouldn't find it surprising that comparing sssd to local files you can
get a 3x perf hit on user lookups.
But I wouldn't expect a 3x nor a 2x slowdown on any program that doesn't
just do user lookups. Some programs that heavily do user lokups may see
a small degradation but it shouldn't be as evident as what you report,
just a few percents at most.

This is a know performance issue that we are going to solve with a local
fast shared memory cache, I planned to add this feature for a few years
now and I should be able to do it quite soon, but it is not there yet.

If you had compared it to nss_ldap (without nscd) and got wildly
different results then I'd be surprised :)

> Feel free to let me know if there is something smarter that can be done
> here, or if you believe I am just barking up the wrong tree. It would be
> interesting to test against nns_ldap etc. to see if things are about the
> same. I can also test clients on the same network segment as the ipa
> server (right now they are showing similar slowdowns as more physically
> remote systems).

The ipa server just uses sssd against the ldap server, although the
communication to the ldap server does not involve the network you still
have the perf hit of the sssd elaboration if you compare it against
local files. I would expect an ipa server to be faster on the first user
lookup, but not faster than any other client on following lookups as
they are cached in sssd.

> On the other hand the whole sssd link may be moot with the weird lastlog
> thing I just found (see other message), if AIDE and bacula are trying to
> process a 438GB file that could certainly slow them down. Nevertheless,
> this is what I am doing.

Yeah eliminating this file and then comparing results again would be a
very good idea.

HTH,
Simo.

-- 
Simo Sorce * Red Hat, Inc * New York