[Freeipa-users] Large slow down when using IPA

[Erinn Looney-Triggs]

On 01/02/2012 07:43 PM, Simo Sorce wrote:
> On Mon, 2012-01-02 at 15:52 -0900, Erinn Looney-Triggs wrote:
>> On 01/02/2012 01:47 PM, Simo Sorce wrote:
>>  
>>> Hi Erinn,
>>> can you please tell what's the baseline you are comparing against ?
>>>
>>> Is it nss_ldap ? With or without nscd ?
>>>
>>> Simo.
>>>
>>
>>
>> Here is what I am comparing, and hoping not to make myself look like an
>> idiot in the process :).
> 
> First of all, thanks for the full explanation, it makes the situation
> much more clear.
> 
>> ipa-client system without excludes configured in sssd.
>> ipa-client system with excludes for users and groups < id 500 configured
>> in sssd.
>> Finally a non ipa-client system, as in local only.
>>
>> These tests are all being done on the same system. All I have at this
>> point is the fact that I started rolling out IPA (and thus sssd) to
>> clients and AIDE checks bombed in terms of the amount of time they took.
>> That and accurate bacula backups failed because it was taking too long
>> to compare the local files to the db. Now it could be unrelated, but I
>> hope not (in the not looking like an idiot category).
>>
>> Now of course network traffic and lag times are involved here, but the
>> one key point that made me take notice here is that the times for an
>> AIDE check also bombed (at about the same ratio) on the IPA servers
>> themselves, thus pointing me to the idea that perhaps this wasn't all
>> network related.
> 
> Indeed it is not all network related, sssd caches a lot of information
> so that most lookups should be 'local'.
> 
> Now here comes the gotcha in comparing against local files access.
> The way sssd currently works each request goes through a pipe to the
> sssd_nss process, while with local files glibc just reads entries off
> the /etc/passwd file.
> 
> I wouldn't find it surprising that comparing sssd to local files you can
> get a 3x perf hit on user lookups.
> But I wouldn't expect a 3x nor a 2x slowdown on any program that doesn't
> just do user lookups. Some programs that heavily do user lokups may see
> a small degradation but it shouldn't be as evident as what you report,
> just a few percents at most.
> 
> This is a know performance issue that we are going to solve with a local
> fast shared memory cache, I planned to add this feature for a few years
> now and I should be able to do it quite soon, but it is not there yet.
> 
> If you had compared it to nss_ldap (without nscd) and got wildly
> different results then I'd be surprised :)
> 
>> Feel free to let me know if there is something smarter that can be done
>> here, or if you believe I am just barking up the wrong tree. It would be
>> interesting to test against nns_ldap etc. to see if things are about the
>> same. I can also test clients on the same network segment as the ipa
>> server (right now they are showing similar slowdowns as more physically
>> remote systems).
> 
> The ipa server just uses sssd against the ldap server, although the
> communication to the ldap server does not involve the network you still
> have the perf hit of the sssd elaboration if you compare it against
> local files. I would expect an ipa server to be faster on the first user
> lookup, but not faster than any other client on following lookups as
> they are cached in sssd.
> 
>> On the other hand the whole sssd link may be moot with the weird lastlog
>> thing I just found (see other message), if AIDE and bacula are trying to
>> process a 438GB file that could certainly slow them down. Nevertheless,
>> this is what I am doing.
> 
> Yeah eliminating this file and then comparing results again would be a
> very good idea.
> 
> HTH,
> Simo.
> 

Simo,
Thanks for the information, it looks like the real culprit in all of
this was the lastlog file, after excluding it from checks I am now back
to just about normal for aide runs.

Course aide probably should be smarter about sparse files but who knows.

Thanks again for the information, and time.

-Erinn


-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 554 bytes
Desc: OpenPGP digital signature
URL: <http://listman.redhat.com/archives/freeipa-users/attachments/20120102/aa6eb4a8/attachment.sig>