[Freeipa-users] caching of lookups / performance problem

[Sullivan, Daniel [CRI]]

Hi,

I figured out what was going on with this issue.  Basically cache timeouts were causing a large number of uid numbers in an arbitrarily-timed directory listing to have expired cache records, which causes those records to be looked up again by the data provider (and thus blocking ‘ls -l’).  To work around this issue now we currently setting the entry_cache_timeout to something arbitrarily high, i.e. 999999, I’m questioning whether or not this is the best approach.  I’d like to use something like refresh_expired_interval, although based on my testing it appears that this does not update records for a trusted AD domain.  I’ve also tried using enumeration, and that doesn’t seem to work either.

I suppose my question is this; is there a preferred method to keep cache records up-to-date for a trusted AD domain?  Right now I am thinking about cron-tabbing an ‘ls -l’ of /home and allowing entry_cache_nowait_percentage to fill this function, although that seems hacky to me.

Any advisement that could be provided would be greatly appreciated.

Best,

Dan Sullivan

> On Jan 30, 2017, at 10:52 AM, Sullivan, Daniel [CRI] <dsullivan2 at bsd.uchicago.edu> wrote:
> 
> Hi,
> 
> I have another question about sssd performance.  I’m having a difficult time doing a regularly performant ‘ls -l’ operation against /home, a mounted NFS share of all of our users home directories.  There are 667 entries in this folder, and all of them have IDs that are resolvable via freeipa/sssd.  We are using an AD trusted domain.
> 
> It is clear to me why an initial invocation of this lookup should take some time (populating the local ldb cache).   And it does.  Usually around 5-10 minutes, but sometimes longer.  After the initial lookups are complete, the output of ‘ls -l' renders fine, and I can inspect the local filesystem cache using ldbsearch and see that it is populated.  The issue is that if I wait a while, or restart sssd, it appears that I have to go through all of these lookups again to render the directory listing.
> 
> I am trying to find an optimal configuration for sssd.conf that will allow a performant ‘ls -l’ listing of a directory with a large number of different id numbers assigned to filesystem objects to always return results immediately from the local cache (after an initial invocation of this command for any given directory).  I think basically what I want is to have the ldb cache always ‘up-to-date’, or at least have sssd willing to immediately dump what it has without having to do a bunch of lookups while blocking the ‘ls -l’ thread.  If possible, whatever solution implemented should also survive a restart of the sssd process.  In short, aside from an initial invocation, I never want ‘ls -l’ to take more than a few seconds.
> 
> The issue described above is somewhat problematic because it appears to cause contention on the sssd process effectively allowing a user doing ls -l /home to inadvertently degrade system performance for another user.
> 
> So far I have tried:
> 
> 1)  Implementing 'enumeration = true' for the [domain] section .  This seems to have no impact.  It might be worthwhile to note that we are using an AD trusted domain.
> 2)  Using the refresh_expired_interval configuration for the [domain] section
> 
> I have read the following two documents in a decent level of detail:
> 
> https://jhrozek.wordpress.com/2015/08/19/performance-tuning-sssd-for-large-ipa-ad-trust-deployments/
> https://jhrozek.wordpress.com/2015/03/11/anatomy-of-sssd-user-lookup/
> 
> It almost seems to me like the answer to this would be to keep the LDB cache valid indefinitely (step 4 on https://jhrozek.wordpress.com/2015/03/11/anatomy-of-sssd-user-lookup/).
> 
> Presumably this is a problem that somebody has seen before.  Would somebody be able to advise on the best way to deal with this?  I appreciate your help.
> 
> Thank you,
> 
> Dan
> 
> -- 
> Manage your subscription for the Freeipa-users mailing list:
> https://www.redhat.com/mailman/listinfo/freeipa-users
> Go to http://freeipa.org for more info on the project