[Freeipa-users] This again :) - ssh authentication for users in complex AD forest - where am I going wrong?

Tue Nov 22 15:37:06 UTC 2016

Upfront
  - I know this question is fairly common and I do read the list and 
archives, honest!
  - I'm following the SSSD troubleshooting wiki and running with debug 
settings for PAM and SSH
  - Still not quite sure where my config problem lies

  - I see "Server not found in kerberos database" in /var/log/messages 
so I think I have a simple /etc/krb5.conf error; possibly a very simple 
root cause like my client can't use DNS to autodiscover a KDC. Not 100% 
sure how to confirm that

Setup:

  - We run an IPA server at COMPANY-IDM.ORG with the goal of using it as 
"glue" for multiple Active Directory relationships
  - Successful trusts made with a number of test AD forest and domains, 
including SSH logins all working fine
  - Got the Trust set up to the real COMPANY.ORG forest last night
  - Trust to COMPANY.ORG went in just fine
  - We can fetch trusted domains through COMPANY.ORG and see all the 
children we expect to see (excellent!)

Situation:

  - I can resolve username at NAFTA.COMPANY.ORG on IPA server and bound client
  - I can "kinit username at NAFTA.COMPANY.ORG" on the IPA server and ipa 
managed client
  - From root I can "sudo username at nafta.company.org" on IPA and client 
server and end up as proper user in proper homedir
  - I can login via SSH to IPA server and client machines as 
user at TESTDOMAIN.ORG
  - ping COMPANY.ORG and NAFTA.COMPANY.ORG resolves to the remote AD 
servers so I think DNS forwarding is OK

BUT -- any sort of "ssh username at nafta.company.org" fails, client sees 
variations of "permission denied"; nothing super useful so far in 
security, ssh or sssd logs

So basically password checking is broken for the actual COMPANY.ORG 
trust we set up last night.

When I had this issue with our test AD domains I think the answer was 
that "client could not discover which KDC to contact for password 
checking" so our response was to customize the krb5.conf file to 
explicitly enable DNS lookups..

This feels to me like either I've messed up sssd.conf or perhaps more 
likely I'm missing a config setting in krb5.conf that is specific to 
password checking for COMPANY.ORG and NAFTA.COMPANY.org

We are running in AWS with VPC Flow Logs enabled and there are no 
obvious REJECT logs showing blockage of traffic to KDC or Domain Controllers

Seeking tips and any guidance people can provide!

Without burying people in log and config data, here is what I think the 
relevant info on our side is:

/etc/krb5.conf (IPA client)
---------------------------------

[libdefaults]

   default_realm = COMPANY-IDM.ORG
   dns_lookup_realm = true
   dns_lookup_kdc = true
   rdns = false
   ticket_lifetime = 24h
   forwardable = yes
   udp_preference_limit = 0
   default_ccache_name = KEYRING:persistent:%{uid}

[realms]

   COMPANY-IDM.ORG = {
     kdc = usaeilidmp001.company-idm.org:88
     master_kdc = usaeilidmp001.company-idm.org:88
     admin_server = usaeilidmp001.company-idm.org:749
     default_domain = company-idm.org
     pkinit_anchors = FILE:/etc/ipa/ca.crt

   }

[domain_realm]

ipa-client.company-aws.org = COMPANY-IDM.ORG

[capaths]

COMPANY-AWS.ORG = {

   COMPANY-IDM.ORG = COMPANY-AWS.ORG

}

COMPANY-IDM.ORG = {

   COMPANY-AWS.ORG = COMPANY-AWS.ORG

}

Here is /etc/sssd/sssd.conf from an IPA client:
---------------------------------------------------------

[domain/company-idm.org]
cache_credentials = True
krb5_store_password_if_offline = True
ipa_domain = company-idm.org
id_provider = ipa
auth_provider = ipa
access_provider = ipa
ldap_tls_cacert = /etc/ipa/ca.crt
ipa_hostname =  client.company-aws.org
chpass_provider = ipa
ipa_server = _srv_, usaeilidmp001.company-idm.org
dns_discovery_domain = company-idm.org

[sssd]

debug_level = 6
services = nss, sudo, pam, ssh
config_file_version = 2

domains = company-idm.org

[nss]

homedir_substring = /home

[pam]

debug_level = 10

[sudo]

[autofs]

[ssh]

debug_level = 6

[pac]

[ifp]

And finally after turning on debug here is some output from sssd_pam.log 
with debug mode set:
-----------

(Tue Nov 22 14:55:07 2016) [sssd[pam]] [pam_check_user_search] (0x0400): 
Returning info for user [username at NAFTA.COMPANY.ORG]
(Tue Nov 22 14:55:07 2016) [sssd[pam]] [pd_set_primary_name] (0x0400): 
User's primary name is username at NAFTA.COMPANY.ORG (Tue Nov 22 14:55:07 
2016) [sssd[pam]] [pam_initgr_cache_set] (0x2000): 
[username at nafta.COMPANY.ORG] added to PAM initgroup cache
(Tue Nov 22 14:55:07 2016) [sssd[pam]] [pam_dp_send_req] (0x0100): 
Sending request with the following data:
(Tue Nov 22 14:55:07 2016) [sssd[pam]] [pam_print_data] (0x0100): 
command: PAM_OPEN_SESSION
(Tue Nov 22 14:55:07 2016) [sssd[pam]] [pam_print_data] (0x0100): 
domain: NAFTA.COMPANY.ORG (Tue Nov 22 14:55:07 2016) [sssd[pam]] 
[pam_print_data] (0x0100): user: username at NAFTA.COMPANY.ORG (Tue Nov 22 
14:55:07 2016) [sssd[pam]] [pam_print_data] (0x0100): service: su-l
(Tue Nov 22 14:55:07 2016) [sssd[pam]] [pam_print_data] (0x0100): tty: 
pts/0
(Tue Nov 22 14:55:07 2016) [sssd[pam]] [pam_print_data] (0x0100): ruser: 
root
(Tue Nov 22 14:55:07 2016) [sssd[pam]] [pam_print_data] (0x0100): rhost: 
not set
(Tue Nov 22 14:55:07 2016) [sssd[pam]] [pam_print_data] (0x0100): 
authtok type: 0
(Tue Nov 22 14:55:07 2016) [sssd[pam]] [pam_print_data] (0x0100): 
newauthtok type: 0
(Tue Nov 22 14:55:07 2016) [sssd[pam]] [pam_print_data] (0x0100): priv: 1
(Tue Nov 22 14:55:07 2016) [sssd[pam]] [pam_print_data] (0x0100): 
cli_pid: 3939
(Tue Nov 22 14:55:07 2016) [sssd[pam]] [pam_print_data] (0x0100): logon 
name: username at nafta.COMPANY.ORG
  (Tue Nov 22 14:55:07 2016) [sssd[pam]] [sbus_add_timeout] (0x2000): 
0x7f98ae9cd8d0
(Tue Nov 22 14:55:07 2016) [sssd[pam]] [pam_dom_forwarder] (0x0100): 
pam_dp_send_req returned 0
  (Tue Nov 22 14:55:07 2016) [sssd[pam]] [sss_dp_req_destructor] 
(0x0400): Deleting request: [0x7f98ac9eb090:3:username at NAFTA.COMPANY.ORG]
  (Tue Nov 22 14:55:07 2016) [sssd[pam]] [sbus_remove_timeout] (0x2000): 
0x7f98ae9cd8d0
  (Tue Nov 22 14:55:07 2016) [sssd[pam]] [sbus_dispatch] (0x4000): dbus 
conn: 0x7f98ae9c6ce0
  (Tue Nov 22 14:55:07 2016) [sssd[pam]] [sbus_dispatch] (0x4000): 
Dispatching.
(Tue Nov 22 14:55:07 2016) [sssd[pam]] [pam_dp_process_reply] (0x0200): 
received: [0 (Success)][NAFTA.COMPANY.ORG]
  (Tue Nov 22 14:55:07 2016) [sssd[pam]] [pam_reply] (0x0200): pam_reply 
called with result [0]: Success.
  (Tue Nov 22 14:55:07 2016) [sssd[pam]] [pam_reply] (0x0200): blen: 35
  (Tue Nov 22 14:55:07 2016) [sssd[pam]] [reset_idle_timer] (0x4000): 
Idle timer re-set for client [0x7f98ae9ccf20][19]
(Tue Nov 22 14:55:12 2016) [sssd[pam]] [pam_initgr_cache_remove] 
(0x2000): [username at nafta.COMPANY.ORG] removed from PAM initgroup cache

pam_sshd has this to say:<mailto:dag at syngentaaws.org>

Nov 22 15:01:25 usaeilvdip001 sshd[4041]: pam_sss(sshd:auth): 
authentication failure; logname= uid=0 euid=0 tty=ssh ruser= 
rhost=usaeilvdip001.syngentaaws.org user=username at nafta.company.org 
<mailto:t859531 at nafta.syngenta.org>
Nov 22 15:01:25 usaeilvdip001 sshd[4041]: pam_sss(sshd:auth): received 
for user username at nafta.company.org <mailto:t859531 at nafta.syngenta.org>: 
4 (System error)
Nov 22 15:01:25 usaeilvdip001 sshd[4041]: Failed password for 
username at nafta.company.org <mailto:t859531 at nafta.syngenta.org> from 
10.127.64.12 port 33812 ssh2
Nov 22 15:01:29 usaeilvdip001 sshd[4041]: pam_sss(sshd:auth): 
authentication failure; logname= uid=0 euid=0 tty=ssh ruser= 
rhost=usaeilvdip001.syngentaaws.org user=username at nafta.company.org 
<mailto:t859531 at nafta.syngenta.org>
Nov 22 15:01:29 usaeilvdip001 sshd[4041]: pam_sss(sshd:auth): received 
for user username at nafta.company.org <mailto:t859531 at nafta.syngenta.org>: 
4 (System error)
Nov 22 15:01:29 usaeilvdip001 sshd[4041]: Failed password for 
username at nafta.company.org <mailto:t859531 at nafta.syngenta.org> from 
10.127.64.12 port 33812 ssh2
Nov 22 15:01:31 usaeilvdip001 sshd[4041]: Connection closed by 
10.127.64.12 [preauth]

And this seems pretty clear from /var/log/messages right after I fail 
with SSH as a NAFTA.COMPANY.ORG user:

Nov 22 15:29:43 usaeilvdip001 [sssd[krb5_child[4099]]]: Server not found 
in Kerberos database
Nov 22 15:29:43 usaeilvdip001 [sssd[krb5_child[4099]]]: Server not found 
in Kerberos database
Nov 22 15:29:54 usaeilvdip001 [sssd[krb5_child[4102]]]: Server not found 
in Kerberos database
Nov 22 15:29:54 usaeilvdip001 [sssd[krb5_child[4102]]]: Server not found 
in Kerberos database

I've played with simple edits to /etc/krb5.conf to explicitly set a 
realm for COMPANY.ORG so I could list kdc entries but it is either not 
working or I've got a syntax misunderstanding.

For instance I've tried to add this to krb5.conf on the client 
underneath the IPA REALM entry:

   COMPANY.ORG = {

     kdc = company.org:88

     master_kdc = company.org:88

     admin_server = company.org

   }

Any tips / help greatly appreciated

Regards,
Chris