[Freeipa-users] This again :) - ssh authentication for users in complex AD forest - where am I going wrong?

Tue Nov 22 15:49:30 UTC 2016

On Tue, Nov 22, 2016 at 10:37:06AM -0500, Chris Dagdigian wrote:
> Upfront
>  - I know this question is fairly common and I do read the list and
> archives, honest!
>  - I'm following the SSSD troubleshooting wiki and running with debug
> settings for PAM and SSH
>  - Still not quite sure where my config problem lies
> 
>  - I see "Server not found in kerberos database" in /var/log/messages so I
> think I have a simple /etc/krb5.conf error; possibly a very simple root
> cause like my client can't use DNS to autodiscover a KDC. Not 100% sure how
> to confirm that

Please send the full krb5_child.log with debug_level=10 in the
[domain/...] section of sssd.conf. My current guess is the ticket
validation fails. Which version of SSSD are you using?

bye,
Sumit

> 
> 
> Setup:
> 
>  - We run an IPA server at COMPANY-IDM.ORG with the goal of using it as
> "glue" for multiple Active Directory relationships
>  - Successful trusts made with a number of test AD forest and domains,
> including SSH logins all working fine
>  - Got the Trust set up to the real COMPANY.ORG forest last night
>  - Trust to COMPANY.ORG went in just fine
>  - We can fetch trusted domains through COMPANY.ORG and see all the children
> we expect to see (excellent!)
> 
> Situation:
> 
>  - I can resolve username at NAFTA.COMPANY.ORG on IPA server and bound client
>  - I can "kinit username at NAFTA.COMPANY.ORG" on the IPA server and ipa
> managed client
>  - From root I can "sudo username at nafta.company.org" on IPA and client
> server and end up as proper user in proper homedir
>  - I can login via SSH to IPA server and client machines as
> user at TESTDOMAIN.ORG
>  - ping COMPANY.ORG and NAFTA.COMPANY.ORG resolves to the remote AD servers
> so I think DNS forwarding is OK
> 
> BUT -- any sort of "ssh username at nafta.company.org" fails, client sees
> variations of "permission denied"; nothing super useful so far in security,
> ssh or sssd logs
> 
> So basically password checking is broken for the actual COMPANY.ORG trust we
> set up last night.
> 
> When I had this issue with our test AD domains I think the answer was that
> "client could not discover which KDC to contact for password checking" so
> our response was to customize the krb5.conf file to explicitly enable DNS
> lookups..
> 
> This feels to me like either I've messed up sssd.conf or perhaps more likely
> I'm missing a config setting in krb5.conf that is specific to password
> checking for COMPANY.ORG and NAFTA.COMPANY.org
> 
> 
> We are running in AWS with VPC Flow Logs enabled and there are no obvious
> REJECT logs showing blockage of traffic to KDC or Domain Controllers
> 
> 
> Seeking tips and any guidance people can provide!
> 
> Without burying people in log and config data, here is what I think the
> relevant info on our side is:
> 
> /etc/krb5.conf (IPA client)
> ---------------------------------
> 
> [libdefaults]
> 
>   default_realm = COMPANY-IDM.ORG
>   dns_lookup_realm = true
>   dns_lookup_kdc = true
>   rdns = false
>   ticket_lifetime = 24h
>   forwardable = yes
>   udp_preference_limit = 0
>   default_ccache_name = KEYRING:persistent:%{uid}
> 
> [realms]
> 
>   COMPANY-IDM.ORG = {
>     kdc = usaeilidmp001.company-idm.org:88
>     master_kdc = usaeilidmp001.company-idm.org:88
>     admin_server = usaeilidmp001.company-idm.org:749
>     default_domain = company-idm.org
>     pkinit_anchors = FILE:/etc/ipa/ca.crt
> 
>   }
> 
> [domain_realm]
> 
> ipa-client.company-aws.org = COMPANY-IDM.ORG
> 
> [capaths]
> 
> COMPANY-AWS.ORG = {
> 
>   COMPANY-IDM.ORG = COMPANY-AWS.ORG
> 
> }
> 
> COMPANY-IDM.ORG = {
> 
>   COMPANY-AWS.ORG = COMPANY-AWS.ORG
> 
> }
> 
> 
> 
> Here is /etc/sssd/sssd.conf from an IPA client:
> ---------------------------------------------------------
> 
> [domain/company-idm.org]
> cache_credentials = True
> krb5_store_password_if_offline = True
> ipa_domain = company-idm.org
> id_provider = ipa
> auth_provider = ipa
> access_provider = ipa
> ldap_tls_cacert = /etc/ipa/ca.crt
> ipa_hostname =  client.company-aws.org
> chpass_provider = ipa
> ipa_server = _srv_, usaeilidmp001.company-idm.org
> dns_discovery_domain = company-idm.org
> 
> [sssd]
> 
> debug_level = 6
> services = nss, sudo, pam, ssh
> config_file_version = 2
> 
> domains = company-idm.org
> 
> [nss]
> 
> homedir_substring = /home
> 
> [pam]
> 
> debug_level = 10
> 
> [sudo]
> 
> [autofs]
> 
> [ssh]
> 
> debug_level = 6
> 
> [pac]
> 
> [ifp]
> 
> 
> 
> And finally after turning on debug here is some output from sssd_pam.log
> with debug mode set:
> -----------
> 
> (Tue Nov 22 14:55:07 2016) [sssd[pam]] [pam_check_user_search] (0x0400):
> Returning info for user [username at NAFTA.COMPANY.ORG]
> (Tue Nov 22 14:55:07 2016) [sssd[pam]] [pd_set_primary_name] (0x0400):
> User's primary name is username at NAFTA.COMPANY.ORG (Tue Nov 22 14:55:07 2016)
> [sssd[pam]] [pam_initgr_cache_set] (0x2000): [username at nafta.COMPANY.ORG]
> added to PAM initgroup cache
> (Tue Nov 22 14:55:07 2016) [sssd[pam]] [pam_dp_send_req] (0x0100): Sending
> request with the following data:
> (Tue Nov 22 14:55:07 2016) [sssd[pam]] [pam_print_data] (0x0100): command:
> PAM_OPEN_SESSION
> (Tue Nov 22 14:55:07 2016) [sssd[pam]] [pam_print_data] (0x0100): domain:
> NAFTA.COMPANY.ORG (Tue Nov 22 14:55:07 2016) [sssd[pam]] [pam_print_data]
> (0x0100): user: username at NAFTA.COMPANY.ORG (Tue Nov 22 14:55:07 2016)
> [sssd[pam]] [pam_print_data] (0x0100): service: su-l
> (Tue Nov 22 14:55:07 2016) [sssd[pam]] [pam_print_data] (0x0100): tty: pts/0
> (Tue Nov 22 14:55:07 2016) [sssd[pam]] [pam_print_data] (0x0100): ruser:
> root
> (Tue Nov 22 14:55:07 2016) [sssd[pam]] [pam_print_data] (0x0100): rhost: not
> set
> (Tue Nov 22 14:55:07 2016) [sssd[pam]] [pam_print_data] (0x0100): authtok
> type: 0
> (Tue Nov 22 14:55:07 2016) [sssd[pam]] [pam_print_data] (0x0100): newauthtok
> type: 0
> (Tue Nov 22 14:55:07 2016) [sssd[pam]] [pam_print_data] (0x0100): priv: 1
> (Tue Nov 22 14:55:07 2016) [sssd[pam]] [pam_print_data] (0x0100): cli_pid:
> 3939
> (Tue Nov 22 14:55:07 2016) [sssd[pam]] [pam_print_data] (0x0100): logon
> name: username at nafta.COMPANY.ORG
>  (Tue Nov 22 14:55:07 2016) [sssd[pam]] [sbus_add_timeout] (0x2000):
> 0x7f98ae9cd8d0
> (Tue Nov 22 14:55:07 2016) [sssd[pam]] [pam_dom_forwarder] (0x0100):
> pam_dp_send_req returned 0
>  (Tue Nov 22 14:55:07 2016) [sssd[pam]] [sss_dp_req_destructor] (0x0400):
> Deleting request: [0x7f98ac9eb090:3:username at NAFTA.COMPANY.ORG]
>  (Tue Nov 22 14:55:07 2016) [sssd[pam]] [sbus_remove_timeout] (0x2000):
> 0x7f98ae9cd8d0
>  (Tue Nov 22 14:55:07 2016) [sssd[pam]] [sbus_dispatch] (0x4000): dbus conn:
> 0x7f98ae9c6ce0
>  (Tue Nov 22 14:55:07 2016) [sssd[pam]] [sbus_dispatch] (0x4000):
> Dispatching.
> (Tue Nov 22 14:55:07 2016) [sssd[pam]] [pam_dp_process_reply] (0x0200):
> received: [0 (Success)][NAFTA.COMPANY.ORG]
>  (Tue Nov 22 14:55:07 2016) [sssd[pam]] [pam_reply] (0x0200): pam_reply
> called with result [0]: Success.
>  (Tue Nov 22 14:55:07 2016) [sssd[pam]] [pam_reply] (0x0200): blen: 35
>  (Tue Nov 22 14:55:07 2016) [sssd[pam]] [reset_idle_timer] (0x4000): Idle
> timer re-set for client [0x7f98ae9ccf20][19]
> (Tue Nov 22 14:55:12 2016) [sssd[pam]] [pam_initgr_cache_remove] (0x2000):
> [username at nafta.COMPANY.ORG] removed from PAM initgroup cache
> 
> 
> pam_sshd has this to say:<mailto:dag at syngentaaws.org>
> 
> Nov 22 15:01:25 usaeilvdip001 sshd[4041]: pam_sss(sshd:auth): authentication
> failure; logname= uid=0 euid=0 tty=ssh ruser=
> rhost=usaeilvdip001.syngentaaws.org user=username at nafta.company.org
> <mailto:t859531 at nafta.syngenta.org>
> Nov 22 15:01:25 usaeilvdip001 sshd[4041]: pam_sss(sshd:auth): received for
> user username at nafta.company.org <mailto:t859531 at nafta.syngenta.org>: 4
> (System error)
> Nov 22 15:01:25 usaeilvdip001 sshd[4041]: Failed password for
> username at nafta.company.org <mailto:t859531 at nafta.syngenta.org> from
> 10.127.64.12 port 33812 ssh2
> Nov 22 15:01:29 usaeilvdip001 sshd[4041]: pam_sss(sshd:auth): authentication
> failure; logname= uid=0 euid=0 tty=ssh ruser=
> rhost=usaeilvdip001.syngentaaws.org user=username at nafta.company.org
> <mailto:t859531 at nafta.syngenta.org>
> Nov 22 15:01:29 usaeilvdip001 sshd[4041]: pam_sss(sshd:auth): received for
> user username at nafta.company.org <mailto:t859531 at nafta.syngenta.org>: 4
> (System error)
> Nov 22 15:01:29 usaeilvdip001 sshd[4041]: Failed password for
> username at nafta.company.org <mailto:t859531 at nafta.syngenta.org> from
> 10.127.64.12 port 33812 ssh2
> Nov 22 15:01:31 usaeilvdip001 sshd[4041]: Connection closed by 10.127.64.12
> [preauth]
> 
> 
> 
> And this seems pretty clear from /var/log/messages right after I fail with
> SSH as a NAFTA.COMPANY.ORG user:
> 
> Nov 22 15:29:43 usaeilvdip001 [sssd[krb5_child[4099]]]: Server not found in
> Kerberos database
> Nov 22 15:29:43 usaeilvdip001 [sssd[krb5_child[4099]]]: Server not found in
> Kerberos database
> Nov 22 15:29:54 usaeilvdip001 [sssd[krb5_child[4102]]]: Server not found in
> Kerberos database
> Nov 22 15:29:54 usaeilvdip001 [sssd[krb5_child[4102]]]: Server not found in
> Kerberos database
> 
> 
> I've played with simple edits to /etc/krb5.conf to explicitly set a realm
> for COMPANY.ORG so I could list kdc entries but it is either not working or
> I've got a syntax misunderstanding.
> 
> For instance I've tried to add this to krb5.conf on the client underneath
> the IPA REALM entry:
> 
>   COMPANY.ORG = {
> 
>     kdc = company.org:88
> 
>     master_kdc = company.org:88
> 
>     admin_server = company.org
> 
>   }
> 
> 
> Any tips / help greatly appreciated
> 
> 
> Regards,
> Chris
> 
> 
> 
> 
> 
> 
> 
> -- 
> Manage your subscription for the Freeipa-users mailing list:
> https://www.redhat.com/mailman/listinfo/freeipa-users
> Go to http://freeipa.org for more info on the project