[Freeipa-users] Needs help understand this timeout issue

Mon Feb 6 13:24:36 UTC 2017

Have you looked at the ignore_group_members option?  Maybe this is the problem you are seeing?

https://jhrozek.wordpress.com/2015/08/19/performance-tuning-sssd-for-large-ipa-ad-trust-deployments/

==snip==

ignore_group_members
Normally the most data-intensive operation is downloading the groups including their members. Usually, we are interested in what groups a user is a member of (id aduser at ad_domain) as the initial step rather than what members do specific groups include (getent group adgroup at ad_domain). Setting the ignore_group_members option to True makes all groups appear as empty, thus downloading only information about the group objects themselves and not their members, providing a significant performance boost. Please note that id aduser at ad_domain would still return all the correct groups!
==snip==

Dan

On Feb 6, 2017, at 1:59 AM, Troels Hansen <th at casalogic.dk> wrote:

Hi

I'm aware of the anatomy of how the lookup is done, but I would suppose a valid cache on the IPA server would result in the cache from the IPA server being used?

I have been debugging this issue some more, and can confirm is the client have its sssd cache invalidated by "sss_cache -E" and the IPA server have a valid cache, when the client asks for the user, the IPA server still asks the AD for the entire group info?

I can see, that even though the cache is refreshed the attribute initgrExpireTimestamp (in the ldb cache) isn't updated.
I have been unable to find out exactly what this controls?

lastUpdate and dataExpireTimestamp is updated to the time stamp of when the refresh ran.

----- On Feb 1, 2017, at 2:27 PM, Sullivan, Daniel [CRI] dsullivan2 at bsd.uchicago.edu wrote:

Have you checked to see if the user is expired in the cache, or if it is
impacted by entry_cache_nowait_percentage (ref sssd.conf). The default entry
timeout is only 90 minutes and entry_cache_nowait_percentage default is 50.

ldbsearch -H  /var/lib/sss/db/timestamps_xxx.xxx.uchicago.edu.ldb >
~/timestamps.txt
ldbsearch -H  /var/lib/sss/db/cache_xxx.xxx.uchicago.edu.ldb > ~/cache.txt

Might be able to provide more info.

Again, I’d really familiarize yourself with the anatomy of an sssd lookup, if
you get to know the diagram and steps 1-7 here it will be a big help to your
question(s).
https://jhrozek.wordpress.com/2015/03/11/anatomy-of-sssd-user-lookup/

Dan

On Feb 1, 2017, at 4:32 AM, Troels Hansen <th at casalogic.dk> wrote:

>From looking af at TCP dump, I can see that if a client requests a AD user from
IPA, IPA does a full user lookup in AD, even though the IPA server have the
user in local cache?

It looks like a single group generates a LOT of traffic to AD like:
client -> IPA
IPA -> client
IPA -> AD
AD -> IPA
IPA -> AD
IPA -> Second AD
Second AD -> IPA
IPA -> Second AD
IPA -> AD
AD -> IPA
IPA -> AD
IPA -> client
client -> IPA
IPA -> Client

Something similar continues for every group the user has.

Soo, I guess the question that I haven't been able to find documented anywhere.
Isn't the SSSD group cache on the IPA servers supposed to be used then a sssd
client requests a user?

----- On Feb 1, 2017, at 9:53 AM, Troels Hansen th at casalogic.dk wrote:

Hmm, suspect its happening on the server...... thous I haven't been able to
pinpoint a log entry that confirms my suspecting.

I have pinpointed the timeout to happen after 58 seconds after completely
removing the SSSD cache and restaring SSSD, which leads me to think my issue is
related to ldap_enumaration_search_timeout which defaults to 60.

rm -fr /var/lib/sss/{mc,db}/* && systemctl restart sssd && time id shja

However, I'm unable to crank it up on the server as it seems to be adjusted Down
to 60 Again?

Wed Feb  1 09:40:56 2017) [sssd[be[lx.dr.dk]]] [dp_get_options] (0x0400): Option
ldap_enumeration_search_timeout has value 120
(Wed Feb  1 09:40:56 2017) [sssd[be[lx.dr.dk]]] [dp_copy_options_ex] (0x0400):
Option ldap_enumeration_search_timeout has value 60
(Wed Feb  1 09:40:56 2017) [sssd[be[lx.dr.dk]]] [dp_copy_options_ex] (0x0400):
Option ldap_enumeration_search_timeout has value 60

LDAP seems speedy enough, not timeouts while querying it manually while a client
is doing a user lookup.

----- On Jan 30, 2017, at 6:06 PM, Sullivan, Daniel [CRI]
dsullivan2 at bsd.uchicago.edu wrote:

If the timeout is occurring on the server, I would start by increasing one or
both of these values:

ldap_opt_timeout
ldap_search_timeout

If that doesn’t work I’d take look to see if the 389 server is under high load
when you are performing this operation.  The easiest way I have found to do
this is to just execute an LDAP query directly against the IPA server when you
are performing an id lookup, for example:

ldapsearch -D "cn=Directory Manager" -w <pw> -s base -b "cn=config"
"(objectclass=*)”

If the LDAP server is not responsive you probably need to increase the number of
worker threads for 389ds.  Also, you might want to disable referrals, check out
the man pages for this;

ldap_referrals = false

Also, FWIW, if you crank up debug logging on the sssd client, you should be able
to see the amount of seconds of timeout assigned to the operation, and witness
the fact that the operation is actually timing out by inspecting the logs
themselves.  The logs can get a little verbose but the data is there.

--
Manage your subscription for the Freeipa-users mailing list:
https://www.redhat.com/mailman/listinfo/freeipa-users
Go to http://freeipa.org for more info on the project

--
Med venlig hilsen

Troels Hansen

Systemkonsulent

Casalogic A/S

T (+45) 70 20 10 63

M (+45) 22 43 71 57

Red Hat, SUSE, VMware, Citrix, Novell, Yellowfin BI, EnterpriseDB, Sophos og
meget mere.

--
Manage your subscription for the Freeipa-users mailing list:
https://www.redhat.com/mailman/listinfo/freeipa-users
Go to http://freeipa.org for more info on the project

--
Med venlig hilsen

Troels Hansen

Systemkonsulent

Casalogic A/S

T (+45) 70 20 10 63

M (+45) 22 43 71 57

Red Hat, SUSE, VMware, Citrix, Novell, Yellowfin BI, EnterpriseDB, Sophos og meget mere.