[Freeipa-users] IPA HBAC access using SSSD for user in trusted AD domain (RHEL 6.8)

Fri Jul 15 16:35:54 UTC 2016

Jakub,

Thank you for replying to me.  Before I forget I will say that I am still on sssd 1.13 on the domain controller; I didn’t upgrade it because I haven’t had any problems logging into that system yet.  That being said:

Thank you, but did this command return "No such user” ?

Yes.  Whenever this occurs "No such user" is the result from the id command executed on the client.

If it did, was the user cached previously (iow, was there a successfull
lookup before) ?

No, this is the first time the user has ever been looked up.  As far as I know the user has never been successfully entered into the cache.  Similarly, the user has never logged in to the IPA server via an SSSD client.

Here is an example of a failed lookup from a client:

[root at cri-kcriwebgdp1 problem]# id hahsan
id: hahsan: No such user

The DC logs for this operation are
NSS - https://gist.github.com/dsulli99/01715234efab09772e8236a13e4f4ef5
IPA - https://gist.github.com/dsulli99/f3cc92d7c32061fd4676a83a039c31b1

I can lookup this user fine on the DC:

[root at cri-ksysipadcp2 sssd]# id  hahsan
uid=339741696(hahsan at bsdad.uchicago.edu<mailto:hahsan at bsdad.uchicago.edu>) gid=339741696(hahsan at bsdad.uchicago.edu<mailto:hahsan at bsdad.uchicago.edu>) groups=339741696(hahsan at bsdad.uchicago.edu<mailto:hahsan at bsdad.uchicago.edu>),788655857
<truncated output>.

I appreciate your help with this.

Best,

Dan

On Jul 15, 2016, at 10:20 AM, Jakub Hrozek <jhrozek at redhat.com<mailto:jhrozek at redhat.com>> wrote:

On Fri, Jul 15, 2016 at 01:22:07PM +0000, Sullivan, Daniel [AAA] wrote:
Jakub,

Sure, no problem, I am happy to provide the output that you are requesting.  Thank you for taking the time to help me.

To answer your question, no record is returned (not missing groups). For example, the output of the failure was:

[root at cri-kcriwebgdp1 log]# id mjarsulic
id: mjarsulic: No such user

As per your request I have attached domain and nss logs for a lookup on the
user ‘spott’ (command invoked ‘id spott’ on the client). (immediately
after executing 'sss_cache -E; service sssd stop ; rm -rf /var/log/sssd/*;
service sssd start;’ on the client):

Thank you, but did this command return "No such user" ?

If it did, was the user cached previously (iow, was there a successfull
lookup before) ? The thing I'm confused about is that even if the back
end request failed (indicated by the "s2n exop request failed" message),
I would expect the NSS process to still return data from the cache.

As per why the request failed, you need to look into the matching logs
on the server side around that time the s2n request failed to see if
there was some issue with lookups.

The double-qualification is just an annoying debug message.
In 1.14, we store all usernames fully qualified (that's the first one
you see) but also append the domain name in some functions when printing
debug messages (that's the second one).

IPA - https://gist.github.com/dsulli99/4e45faa39474b9131be811e4a0779c40
NSS - https://gist.github.com/dsulli99/e2e10da34ff860ec15e56ea521eb8315

Not every record fails, and the behavior is inconsistent between lookups (i.e. sometimes a user will lookup correctly, sometimes it will not), but it appears that in some situations a timeout is occurring in the nss logs (not in the failure above).   In these situations it looks to me like the query is dispatched to the DC, and the lookup times out.  If I wait a little bit and perform the lookup on the same user again,  the record is returned (presumably because the DC eventually resolved and cached the query?).  We are migrating from CentrifyDC and have loaded 2000+ custom ID overrides into our Default Trust ID View; perhaps we will need to implement the tempfs caching for the /var/lib/sss/db on the DC as described in your performance tuning document (https://jhrozek.wordpress.com/2015/08/19/performance-tuning-sssd-for-large-ipa-ad-trust-deployments/).  These timeouts look like:

(Fri Jul 15 07:21:04 2016) [sssd[nss]] [get_dp_name_and_id] (0x0400): Not a LOCAL view, continuing with provided values.
(Fri Jul 15 07:21:04 2016) [sssd[nss]] [sss_dp_issue_request] (0x0400): Issuing request for [0x41e750:1:bson at bsdad.uchicago.edu<mailto:bson at bsdad.uchicago.edu><mailto:bson at bsdad.uchicago.edu>@bsdad.uchicago.edu<http://bsdad.uchicago.edu>]
(Fri Jul 15 07:21:04 2016) [sssd[nss]] [sss_dp_get_account_msg] (0x0400): Creating request for [bsdad.uchicago.edu<http://bsdad.uchicago.edu><http://bsdad.uchicago.edu>][0x1][BE_REQ_USER][1][name=bson at bsdad.uchicago.edu<mailto:name=bson at bsdad.uchicago.edu><mailto:name=bson at bsdad.uchicago.edu>:-]
(Fri Jul 15 07:21:04 2016) [sssd[nss]] [sbus_add_timeout] (0x2000): 0x1fa9020
(Fri Jul 15 07:21:04 2016) [sssd[nss]] [sss_dp_internal_get_send] (0x0400): Entering request [0x41e750:1:bson at bsdad.uchicago.edu<mailto:bson at bsdad.uchicago.edu><mailto:bson at bsdad.uchicago.edu>@bsdad.uchicago.edu<http://bsdad.uchicago.edu>]
(Fri Jul 15 07:21:17 2016) [sssd[nss]] [sbus_remove_timeout] (0x2000): 0x1fa9020
(Fri Jul 15 07:21:17 2016) [sssd[nss]] [sbus_dispatch] (0x4000): dbus conn: 0x1fa0730
(Fri Jul 15 07:21:17 2016) [sssd[nss]] [sbus_dispatch] (0x4000): Dispatching.
(Fri Jul 15 07:21:17 2016) [sssd[nss]] [sss_dp_get_reply] (0x1000): Got reply from Data Provider - DP error code: 3 errno: 110 error message: Connection timed out
(Fri Jul 15 07:21:17 2016) [sssd[nss]] [nss_cmd_getby_dp_callback] (0x0040): Unable to get information from Data Provider
Error: 3, 110, Connection timed out
Will try to return what we have in cache
(Fri Jul 15 07:21:17 2016) [sssd[nss]] [sss_dp_req_destructor] (0x0400): Deleting request: [0x41e750:1:bson at bsdad.uchicago.edu<mailto:bson at bsdad.uchicago.edu><mailto:bson at bsdad.uchicago.edu>@bsdad.uchicago.edu<http://bsdad.uchicago.edu>]
(Fri Jul 15 07:21:17 2016) [sssd[nss]] [reset_idle_timer] (0x4000): Idle timer re-set for client [0x1fa7fc0][22]
(Fri Jul 15 07:21:17 2016) [sssd[nss]] [reset_idle_timer] (0x4000): Idle timer re-set for client [0x1fa7fc0][22]
(Fri Jul 15 07:21:17 2016) [sssd[nss]] [client_recv] (0x0200): Client disconnected!
(Fri Jul 15 07:21:17 2016) [sssd[nss]] [client_close_fn] (0x2000): Terminated client [0x1fa7fc0][22]

I’m going to implement tmpfs caching on the DC, hopefully this will address at least a subset of these lookup failures.  I’ll report back with my findings.

Thank you again for your help.

Best,

Dan Sullivan

On Jul 15, 2016, at 7:12 AM, Jakub Hrozek <jhrozek at redhat.com<mailto:jhrozek at redhat.com><mailto:jhrozek at redhat.com>> wrote:

On Fri, Jul 15, 2016 at 12:00:56PM +0000, Sullivan, Daniel [AAA] wrote:
Lukas,

Thank you for your reply and inquiry.

First, to answer your question; yes, we have been using the default_domain_suffix for some time.  I am not sure what you mean by previously, but it is currently implemented and has been implemented prior to our 1.13 -> 1.14 upgrade.

And yes, I am assessing a possible software regression at the
current moment. It might be related to the default_domain_suffix
you are inquiring about.  Basically I am getting inconsistent
results on invocation of the id command with specifying the username
as ‘username’ or ‘username at fqdn’ on a client running 1.14
against a DC running 1.13 (i.e. no way to reliably invoke id against a
trusted domain account).  Sometimes the command will return a result,
and sometimes it will not.

No result or missing groups?

Looking at nss debug logs it appears that
a duplicate fqdn is being appended to the nss query as show here (as
@bsdad.uchicago.edu at bsdad.uchicago.edu<mailto:bsdad.uchicago.edu at bsdad.uchicago.edu><mailto:bsdad.uchicago.edu at bsdad.uchicago.edu><mailto:bsdad.uchicago.edu at bsdad.uchicago.edu>).
This lookup fails.

Yes, this is wrong, can you send me the full NSS and domain logs please?

--
Manage your subscription for the Freeipa-users mailing list:
https://www.redhat.com/mailman/listinfo/freeipa-users
Go to http://freeipa.org for more info on the project

********************************************************************************
This e-mail is intended only for the use of the individual or entity to which
it is addressed and may contain information that is privileged and confidential.
If the reader of this e-mail message is not the intended recipient, you are
hereby notified that any dissemination, distribution or copying of this
communication is prohibited. If you have received this e-mail in error, please
notify the sender and destroy all copies of the transmittal.

Thank you
University of Chicago Medicine and Biological Sciences
********************************************************************************

********************************************************************************
This e-mail is intended only for the use of the individual or entity to which
it is addressed and may contain information that is privileged and confidential.
If the reader of this e-mail message is not the intended recipient, you are 
hereby notified that any dissemination, distribution or copying of this
communication is prohibited. If you have received this e-mail in error, please 
notify the sender and destroy all copies of the transmittal. 

Thank you
University of Chicago Medicine and Biological Sciences 
********************************************************************************