[Freeipa-users] Large slow down when using IPA

[Erinn Looney-Triggs]

On 01/02/2012 11:40 AM, Jakub Hrozek wrote:
> On Mon, Jan 02, 2012 at 12:53:29PM -0500, Simo Sorce wrote:
>> On Mon, 2012-01-02 at 17:29 +0100, Jakub Hrozek wrote:
>>> On Mon, Jan 02, 2012 at 10:00:02AM -0500, Simo Sorce wrote:
>>>> On Sat, 2011-12-31 at 01:35 -0900, Erinn Looney-Triggs wrote:
>>>>> On 12/30/2011 07:19 PM, JR Aquino wrote:
>>>>>>
>>>>>> On Dec 30, 2011, at 5:45 PM, Erinn Looney-Triggs wrote:
>>>>>>
>>>>>>> I have been slowly rolling out FreeIPA to my systems, trying to track
>>>>>>> differences/changes. One of the most noticeable has been a large slow
>>>>>>> down in file access times.
>>>>>>>
>>>>>>> Let me explain as best as I can. I use AIDE to track the file system
>>>>>>> (think tripwire) and it runs checks once a day. During these checks it
>>>>>>> is scanning (almost) the entire file system and comparing it to a stored
>>>>>>> database. On a moderately powered system with ~151k files, an AIDE run
>>>>>>> will usually take ~30 minutes. After the system becomes an IPA client
>>>>>>> the same run will generally take ~90-120 minutes. Un-install the
>>>>>>> ipa-client, back to ~30 minutes for an AIDE run.
>>>>>>>
>>>>>>> Now clearly a lot of lookups are being done for user names and group
>>>>>>> names, and this will have a performance hit that is dependant on the
>>>>>>> network. However, the odd thing is that even when running on the IPA
>>>>>>> server itself the slowdown is still the same.
>>>>>>>
>>>>>>> Not sure if this is an IPA problem, an SSSD problem, a bit of both, or
>>>>>>> neither, perhaps it is just the way it is, but a slowdown of 3-4x seems
>>>>>>> a bit much to me. Clearly the results are not scientific, however, they
>>>>>>> have been generally reproducible since I started rolling IPA out.
>>>>>>>
>>>>>>> As a side note this slowdown has also broken bacula backups, as the
>>>>>>> bacula client is scanning the filesystem for change (using accurate
>>>>>>> backups) the director times out.
>>>>>>>
>>>>>>> Any thoughts, or opinions? Workarounds etc? I have checked to make sure
>>>>>>> that SSSD caching is enabled, and functional.
>>>>>>>
>>>>>>> Thanks,
>>>>>>>
>>>>>>> -Erinn
>>>>>>
>>>>>> I am assuming that these are all running as local users.
>>>>>>
>>>>>> From the sssd.conf man page in the nss section:
>>>>>>
>>>>>> filter_users, filter_groups (string)
>>>>>>            Exclude certain users from being fetched from the sss NSS database. This is particularly useful for system accounts. This option can also be set per-domain or include fully-qualified names to filter only users from the
>>>>>>            particular domain.
>>>>>>
>>>>>>            Default: root
>>>>>>
>>>>>>
>>>>>> Try adding this to your sssd.conf:
>>>>>>
>>>>>> [nss]
>>>>>>            filter_groups = root,bacula,aide,otherdaemonuser <-as needed
>>>>>>            filter_users = root,bacula,aide,otherdaemonuser <- as needed
>>>>>>
>>>>>> Let me know if that solves your issue.
>>>>>>
>>>>>
>>>>> Thanks for pointing that out, completely missed that option! Wouldn't it
>>>>> be sweet to have an option that say looked at /etc/login.defs and just
>>>>> didn't lookup anything under MIN_UID, on the assumption that those are
>>>>> system accounts? Certainly would stop a lot of lookups I imagine.
>>>>
>>>> We already have range options (min_id/max_id), but unfortunately that
>>>> doesn't help when an application asks for information by name.
>>>> You either permanently blacklist such name or you have to do the lookup
>>>> and then find out it either a) does not exist, or b) it has to be
>>>> filtered out.
>>>>
>>>>> Of course you would have to leave it as an option and probably default
>>>>> it to off given the odd things people do with their systems.
>>>>
>>>> Indeed sssd used to enforce a min id range of 1000 and we turned it off
>>>> in the default configuration due to issues with weird configurations.
>>>>
>>>> Can you try using both min_id and filter_users and see if it makes any
>>>> difference in your case ?
>>>>
>>>> Simo.
>>>>
>>>
>>> Even when performing getpwuid() calls, SSSD first looks up the user
>>> information, reads the UID LDAP attribute and then checks the UID value
>>> from LDAP against min_id/max_id values.
>>
>> Not according to my reading of the sources, if you look into
>> nss_cmd_getpwuid_search() you'll see that we proceed only if we first
>> pass the min_id/max_id range check, otherwise we return ENOENT.
>>
>> Simo.
> 
> Sorry, you're right and I need to warm up my brain a little more after
> the Christmas break.
> 
> Thanks!

I am going through some testing now to try and get you folks something
more definitive. However, from an early test adding users/groups to
filter_* seemed to reduce the performance hit slightly, but it did not
take it anywhere near the levels it was at before sssd was in place.

Like I said I will continue to test and get you folks some more
definitive results, probably later today. Thanks for all the info and
feedback.

-Erinn


-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 554 bytes
Desc: OpenPGP digital signature
URL: <http://listman.redhat.com/archives/freeipa-users/attachments/20120102/5ce5c92c/attachment.sig>