[Freeipa-users] errors when one ipa server down

Mon Sep 17 14:19:08 UTC 2012

On 2012-09-08, at 11:03 AM, Dmitri Pal wrote:

> On 09/07/2012 04:50 PM, Rob Crittenden wrote:
>> Michael Mercier wrote:
>>> 
>>> On 2012-09-07, at 2:47 PM, Dmitri Pal wrote:
>>> 
>>>> On 09/07/2012 12:42 PM, Michael Mercier wrote:
>>>>> On 2012-09-07, at 12:14 PM, Dmitri Pal wrote:
>>>>> 
>>>>>> On 09/06/2012 10:40 AM, Michael Mercier wrote:
>>>>>>> Hello,
>>>>>>> 
>>>>>>> I have experienced some odd connectivity issues using MMR with
>>>>>>> FreeIPA (all systems CentOS 6.3).  I have 2 ipa servers
>>>>>>> (ipaserver / ipaserver2) setup using MMR.
>>>>>>> 
>>>>>>> [root at ipaserver ~]#ipa-replica-manage list
>>>>>>> ipaserver.mpls.local: master
>>>>>>> ipaserver2.mpls.local: master
>>>>>>> [root at ipaserver ~]# rpm -qa|grep ipa
>>>>>>> libipa_hbac-1.8.0-32.el6.x86_64
>>>>>>> ipa-admintools-2.2.0-16.el6.x86_64
>>>>>>> ipa-server-2.2.0-16.el6.x86_64
>>>>>>> ipa-pki-ca-theme-9.0.3-7.el6.noarch
>>>>>>> libipa_hbac-python-1.8.0-32.el6.x86_64
>>>>>>> ipa-client-2.2.0-16.el6.x86_64
>>>>>>> ipa-server-selinux-2.2.0-16.el6.x86_64
>>>>>>> ipa-pki-common-theme-9.0.3-7.el6.noarch
>>>>>>> python-iniparse-0.3.1-2.1.el6.noarch
>>>>>>> ipa-python-2.2.0-16.el6.x86_64
>>>>>>> 
>>>>>>> 
>>>>>>> [root at ipaserver2 ~]#ipa-replica-manage list
>>>>>>> ipaserver.mpls.local: master
>>>>>>> ipaserver2.mpls.local: master
>>>>>>> [root at ipaserver2 ~]# rpm -qa|grep ipa
>>>>>>> ipa-client-2.2.0-16.el6.x86_64
>>>>>>> ipa-server-2.2.0-16.el6.x86_64
>>>>>>> ipa-pki-ca-theme-9.0.3-7.el6.noarch
>>>>>>> ipa-python-2.2.0-16.el6.x86_64
>>>>>>> libipa_hbac-1.8.0-32.el6.x86_64
>>>>>>> python-iniparse-0.3.1-2.1.el6.noarch
>>>>>>> libipa_hbac-python-1.8.0-32.el6.x86_64
>>>>>>> ipa-admintools-2.2.0-16.el6.x86_64
>>>>>>> ipa-server-selinux-2.2.0-16.el6.x86_64
>>>>>>> ipa-pki-common-theme-9.0.3-7.el6.noarch
>>>>>>> 
>>>>>>> 
>>>>>>> [mike at ipaclient ~]$ rpm -qa|grep ipa
>>>>>>> ipa-admintools-2.2.0-16.el6.x86_64
>>>>>>> python-iniparse-0.3.1-2.1.el6.noarch
>>>>>>> ipa-python-2.2.0-16.el6.x86_64
>>>>>>> libipa_hbac-python-1.8.0-32.el6.x86_64
>>>>>>> ipa-client-2.2.0-16.el6.x86_64
>>>>>>> libipa_hbac-1.8.0-32.el6.x86_64
>>>>>>> 
>>>>>>> 
>>>>>>> I have a webserver (zenoss) using kerberos authentication.
>>>>>>> 
>>>>>>> [root at zenoss ~]# rpm -qa|grep ipa
>>>>>>> libipa_hbac-1.8.0-32.el6.x86_64
>>>>>>> libipa_hbac-python-1.8.0-32.el6.x86_64
>>>>>>> ipa-python-2.2.0-16.el6.x86_64
>>>>>>> ipa-client-2.2.0-16.el6.x86_64
>>>>>>> python-iniparse-0.3.1-2.1.el6.noarch
>>>>>>> ipa-admintools-2.2.0-16.el6.x86_64
>>>>>>> 
>>>>>>> <Location />
>>>>>>>  SSLRequireSSL
>>>>>>>  AuthType Kerberos
>>>>>>>  AuthName "Kerberos Login"
>>>>>>> 
>>>>>>>  KrbMethodK5Passwd Off
>>>>>>>  KrbAuthRealms MPLS.LOCAL
>>>>>>>  KrbSaveCredentials on
>>>>>>>  KrbServiceName HTTP
>>>>>>>  Krb5KeyTab /etc/http/conf.d/http.keytab
>>>>>>> 
>>>>>>>  AuthLDAPUrl "ldap://ipaserver.mpls.local
>>>>>>> ipaserver2.mpls.local/dc=mpls,dc=local?krbPrincipalName"
>>>>>>>  RequestHeader set X_REMOTE_USER %{remoteUser}e
>>>>>>>  require ldap-group
>>>>>>> cn=zenuser,cn=groups,cn=accounts,dc=mpls,dc=local
>>>>>>> </Location>
>>>>>>> 
>>>>>>> 
>>>>>>> With both ipaserver and ipaserver2 'up', if I connect to
>>>>>>> https://zenoss.mpls.local from ipaclient using firefox, I am
>>>>>>> successfully connected.  If on ipaserver I do a 'ifdown eth0' and
>>>>>>> attempt another connection, it fails.  I have also noticed the
>>>>>>> following:
>>>>>>> 
>>>>>>> 1. I am unable to use the ipaserver2 management interface when
>>>>>>> ipaserver is unavailable.
>>>>>>> 2. It takes a longer period of time to do a kinit
>>>>>>> 
>>>>>>> If the I then perform:
>>>>>>> [root at ipaserver ~]#ifup eth0
>>>>>>> 
>>>>>>> [root at ipaserver2 ~]#ifdown eth0
>>>>>>> 
>>>>>>> [mike at ipaclient ~]$kinit
>>>>>>> kinit: Cannot contact any KDC for realm 'MPLS.LOCAL' while
>>>>>>> getting initial credentials
>>>>>>> 
>>>>>>> [root at ipaserver2 ~]#ifup eth0
>>>>>>> 
>>>>>>> [mike at ipaclient ~]$ kinit
>>>>>>> Password for mike at MPLS.LOCAL:
>>>>>>> [mike at ipaclient ~]$
>>>>>>> 
>>>>>>> [root at ipaserver2 ~]#ifdown eth0
>>>>>>> 
>>>>>>> .. wait number of minutes
>>>>>>> 
>>>>>>> ipaclient screen locks - type password - after a short delay (~7
>>>>>>> seconds) screen unlock compeletes
>>>>>>> 
>>>>>>> [mike at ipaclient ~]$kinit
>>>>>>> Password for mike at MPLS.LOCAL:
>>>>>>> [mike at ipaclient ~]$
>>>>>>> 
>>>>>>> Any ideas?
>>>>>>> 
>>>>>>> Thanks,
>>>>>>> Mike
>>>>>> This seems to be some DNS problem.
>>>>>> You client does not see the second replica and might have some name
>>>>>> resolution timeouts.
>>>>>> 
>>>>>> Please check your dns setup and krb5.conf on the client.
>>>>>> 
>>>>>> To help more we need more details about you client configuration
>>>>>> DNS and
>>>>>> kerberos.
>>>>> Hi,
>>>>> 
>>>>> Additional information...
>>>>> 
>>>>> [root at zenoss ~]#more /etc/resolv.conf
>>>>> search mpls.local
>>>>> domain mpls.local
>>>>> nameserver 172.16.112.5
>>>>> nameserver 172.16.112.8
>>>>> 
>>>>> [root at zenoss ~]# more /etc/krb5.conf
>>>>> #File modified by ipa-client-install
>>>>> 
>>>>> [libdefaults]
>>>>>  default_realm = MPLS.LOCAL
>>>>>  dns_lookup_realm = true
>>>>>  dns_lookup_kdc = true
>>>>>  rdns = false
>>>>>  ticket_lifetime = 24h
>>>>>  forwardable = yes
>>>>> 
>>>>> [realms]
>>>>>  MPLS.LOCAL = {
>>>>>    pkinit_anchors = FILE:/etc/ipa/ca.crt
>>>>>  }
>>>>> 
>>>>> [domain_realm]
>>>>>  .mpls.local = MPLS.LOCAL
>>>>>  mpls.local = MPLS.LOCAL
>>>>> 
>>>>> [root at ipaclient ~]# more /etc/resolv.conf
>>>>> # Generated by NetworkManager
>>>>> search mpls.local
>>>>> nameserver 172.16.112.5
>>>>> nameserver 172.16.112.8
>>>>> 
>>>>> [root at ipaclient ~]# more /etc/krb5.conf
>>>>> #File modified by ipa-client-install
>>>>> 
>>>>> [libdefaults]
>>>>>  default_realm = MPLS.LOCAL
>>>>>  dns_lookup_realm = true
>>>>>  dns_lookup_kdc = true
>>>>>  rdns = false
>>>>>  ticket_lifetime = 24h
>>>>>  forwardable = yes
>>>>> 
>>>>> [realms]
>>>>>  MPLS.LOCAL = {
>>>>>    pkinit_anchors = FILE:/etc/ipa/ca.crt
>>>>>  }
>>>>> 
>>>>> [domain_realm]
>>>>>  .mpls.local = MPLS.LOCAL
>>>>>  mpls.local = MPLS.LOCAL
>>>>> 
>>>>> [root at ipaclient ~]# nslookup ipaserver
>>>>> Server:        172.16.112.5
>>>>> Address:    172.16.112.5#53
>>>>> 
>>>>> Name:    ipaserver.mpls.local
>>>>> Address: 172.16.112.5
>>>>> 
>>>>> [root at ipaserver ~]#ifdown eth0
>>>>> 
>>>>> [root at ipaclient ~]# nslookup ipaserver
>>>>> Server:        172.16.112.8
>>>>> Address:    172.16.112.8#53
>>>>> 
>>>>> Name:    ipaserver.mpls.local
>>>>> Address: 172.16.112.5
>>>>> 
>>>>> [root at ipaclient ~]# nslookup ipaserver2
>>>>> Server:        172.16.112.8
>>>>> Address:    172.16.112.8#53
>>>>> 
>>>>> Name:    ipaserver2.mpls.local
>>>>> Address: 172.16.112.8
>>>>> 
>>>>> Copy/paste from the DNS page on ipaserver/ipaserver2
>>>>> 
>>>>> @ NS ipaserver.mpls.local.
>>>>>     NS ipaserver2.mpls.local.
>>>>> _kerberos TXT MPLS.LOCAL
>>>>> _kerberos-master._tcp SRV 0 100 88 ipaserver
>>>>>                                         SRV 0 100 88 ipaserver2
>>>>> _kerberos-master._udp SRV 0 100 88 ipaserver
>>>>>                                           SRV 0 100 88 ipaserver2
>>>>> _kerberos._tcp SRV 0 100 88 ipaserver
>>>>>                            SRV 0 100 88 ipaserver2
>>>>> _kerberos._udp SRV 0 100 88 ipaserver
>>>>>                         SRV 0 100 88 ipaserver2
>>>>> _kpasswd._tcp SRV 0 100 464 ipaserver
>>>>>                        SRV 0 100 464 ipaserver2
>>>>> _kpasswd._udp SRV 0 100 464 ipaserver
>>>>>                         SRV 0 100 464 ipaserver2
>>>>> _ldap._tcp SRV 0 100 389 ipaserver
>>>>>                SRV 0 100 389 ipaserver2
>>>>> _ntp._udp SRV 0 100 123 ipaserver
>>>>>               SRV 0 100 123 ipaserver2
>>>>> ipaclient A 172.16.112.9
>>>>> ipaclient2 A 172.16.112.145
>>>>> ipaserver A 172.16.112.5
>>>>> ipaserver2 A 172.16.112.8
>>>>> zenoss A 172.16.112.6
>>>>> 
>>>>> Thanks,
>>>>> Mike
>>>>> 
>>>> I noticed that there is no domain line in the resolv.conf on the
>>>> client.
>>>> AFAIU in this case it would determine the domain by the gethostname and
>>>> in case of network being down it will fail over to the hosts file.
>>>> I wonder what is in your /etc/hosts?
>>>> Dose it have just a short host name?
>>> 
>>> [root at ipaclient ~]# more /etc/hosts
>>> 127.0.0.1    localhost.localdomain    localhost
>>> ::1    localhost6.localdomain6    localhost6
>>> 
>>> 
>>> Add domain mpls.local to /etc/resolv.conf
>>> 
>>> [root at ipaserver ~]#ifdown eth0
>>> 
>>> [root at ipaclient ~]# kinit mike
>>> kinit: Cannot contact any KDC for realm 'MPLS.LOCAL' while getting
>>> initial credentials
>>> [root at ipaclient ~]# nslookup ipaserver
>>> Server:        172.16.112.8
>>> Address:    172.16.112.8#53
>>> 
>>> Name:    ipaserver.mpls.local
>>> Address: 172.16.112.5
>>> 
>>> [root at ipaclient ~]# nslookup ipaserver2
>>> Server:        172.16.112.8
>>> Address:    172.16.112.8#53
>>> 
>>> Name:    ipaserver2.mpls.local
>>> Address: 172.16.112.8
>>> 
>>> add '172.16.112.9 ipaclient.mpls.local ipaclient' to /etc/hosts
>>> 
>>> [root at ipaserver ~]#ifup eth0
>>> 
>>> [root at ipaclient ~]# kinit mike
>>> Password for mike at MPLS.LOCAL:
>>> 
>>> [root at ipaserver ~]#ifdown eth0
>>> 
>>> [root at ipaclient ~]# kinit mike
>>> kinit: Cannot contact any KDC for realm 'MPLS.LOCAL' while getting
>>> initial credentials
>>> [root at ipaclient ~]# nslookup -type=srv _kerberos-master._tcp
>>> Server:        172.16.112.8
>>> Address:    172.16.112.8#53
>>> 
>>> _kerberos-master._tcp.mpls.local    service = 0 100 88
>>> ipaserver2.mpls.local.
>>> _kerberos-master._tcp.mpls.local    service = 0 100 88
>>> ipaserver.mpls.local.
>>> 
>>> [root at ipaclient ~]# nslookup -type=srv _kerberos-master._udp
>>> Server:        172.16.112.5
>>> Address:    172.16.112.5#53
>>> 
>>> _kerberos-master._udp.mpls.local    service = 0 100 88
>>> ipaserver.mpls.local.
>>> _kerberos-master._udp.mpls.local    service = 0 100 88
>>> ipaserver2.mpls.local.
>>> 
>>> 
>>> [root at ipaclient ~]# kinit mike
>>> kinit: Cannot contact any KDC for realm 'MPLS.LOCAL' while getting
>>> initial credentials
>>> 
>>> [root at ipaserver ~]#ifup eth0
>>> 
>>> [root at ipaclient ~]# kinit mike
>>> Password for mike at MPLS.LOCAL:
>> 
>> I'd start with the sssd logs. Is it seeing the main server go offline
>> and not switching to the second one? Or is it going into offline mode?
>> 
>> Do you have _srv_ or both servers listed in ipa_server in
>> /etc/sssd/sssd.conf?
>> 
>> rob
>> 
> Rob, may be I am missing something but how SSSD is related in this case?
> The test is done using kinit not SSSD.
> 
> It would actually be an interesting test to try the same via SSSD for
> example do su to mike instead of kinit and see what would happen (watch
> SSSD logs with high debug level, 8 for example).
> If that works it would probably mean that kinit does not fail over
> properly. So this would be a Kerberos kinit bug not IPA/SSSD bug.

Hello,

[root at ipaclient ~]# su mike
[mike at ipaclient root]$ exit
exit

[root at ipaserver ~]ifdown eth0

[root at ipaclient ~]# su mike
[mike at ipaclient root]$ exit
exit
[root at ipaclient ~]# 

debug_level = 8 for [sssd]
output while running above commands

(Mon Sep 17 10:16:20 2012) [sssd] [service_send_ping] (0x0100): Pinging nss
(Mon Sep 17 10:16:20 2012) [sssd] [sbus_add_timeout] (0x2000): 0x1539200
(Mon Sep 17 10:16:20 2012) [sssd] [service_send_ping] (0x0100): Pinging pam
(Mon Sep 17 10:16:20 2012) [sssd] [sbus_add_timeout] (0x2000): 0x15386a0
(Mon Sep 17 10:16:20 2012) [sssd] [sbus_remove_timeout] (0x2000): 0x1539200
(Mon Sep 17 10:16:20 2012) [sssd] [ping_check] (0x0100): Service nss replied to ping
(Mon Sep 17 10:16:20 2012) [sssd] [sbus_remove_timeout] (0x2000): 0x15386a0
(Mon Sep 17 10:16:20 2012) [sssd] [ping_check] (0x0100): Service pam replied to ping
(Mon Sep 17 10:16:29 2012) [sssd] [service_send_ping] (0x0100): Pinging mpls.local
(Mon Sep 17 10:16:29 2012) [sssd] [sbus_add_timeout] (0x2000): 0x15386a0
(Mon Sep 17 10:16:29 2012) [sssd] [sbus_remove_timeout] (0x2000): 0x15386a0
(Mon Sep 17 10:16:29 2012) [sssd] [ping_check] (0x0100): Service mpls.local replied to ping
(Mon Sep 17 10:16:30 2012) [sssd] [service_send_ping] (0x0100): Pinging nss
(Mon Sep 17 10:16:30 2012) [sssd] [sbus_add_timeout] (0x2000): 0x15386a0
(Mon Sep 17 10:16:30 2012) [sssd] [service_send_ping] (0x0100): Pinging pam
(Mon Sep 17 10:16:30 2012) [sssd] [sbus_add_timeout] (0x2000): 0x1539200
(Mon Sep 17 10:16:30 2012) [sssd] [sbus_remove_timeout] (0x2000): 0x15386a0
(Mon Sep 17 10:16:30 2012) [sssd] [ping_check] (0x0100): Service nss replied to ping
(Mon Sep 17 10:16:30 2012) [sssd] [sbus_remove_timeout] (0x2000): 0x1539200
(Mon Sep 17 10:16:30 2012) [sssd] [ping_check] (0x0100): Service pam replied to ping
(Mon Sep 17 10:16:39 2012) [sssd] [service_send_ping] (0x0100): Pinging mpls.local
(Mon Sep 17 10:16:39 2012) [sssd] [sbus_add_timeout] (0x2000): 0x1539200
(Mon Sep 17 10:16:39 2012) [sssd] [sbus_remove_timeout] (0x2000): 0x1539200
(Mon Sep 17 10:16:39 2012) [sssd] [ping_check] (0x0100): Service mpls.local replied to ping
(Mon Sep 17 10:16:40 2012) [sssd] [service_send_ping] (0x0100): Pinging nss
(Mon Sep 17 10:16:40 2012) [sssd] [sbus_add_timeout] (0x2000): 0x1539200
(Mon Sep 17 10:16:40 2012) [sssd] [service_send_ping] (0x0100): Pinging pam
(Mon Sep 17 10:16:40 2012) [sssd] [sbus_add_timeout] (0x2000): 0x15386a0
(Mon Sep 17 10:16:40 2012) [sssd] [sbus_remove_timeout] (0x2000): 0x1539200
(Mon Sep 17 10:16:40 2012) [sssd] [ping_check] (0x0100): Service nss replied to ping
(Mon Sep 17 10:16:40 2012) [sssd] [sbus_remove_timeout] (0x2000): 0x15386a0
(Mon Sep 17 10:16:40 2012) [sssd] [ping_check] (0x0100): Service pam replied to ping
(Mon Sep 17 10:16:49 2012) [sssd] [service_send_ping] (0x0100): Pinging mpls.local
(Mon Sep 17 10:16:49 2012) [sssd] [sbus_add_timeout] (0x2000): 0x15386a0
(Mon Sep 17 10:16:49 2012) [sssd] [sbus_remove_timeout] (0x2000): 0x15386a0
(Mon Sep 17 10:16:49 2012) [sssd] [ping_check] (0x0100): Service mpls.local replied to ping
(Mon Sep 17 10:16:50 2012) [sssd] [service_send_ping] (0x0100): Pinging nss
(Mon Sep 17 10:16:50 2012) [sssd] [sbus_add_timeout] (0x2000): 0x15386a0
(Mon Sep 17 10:16:50 2012) [sssd] [service_send_ping] (0x0100): Pinging pam
(Mon Sep 17 10:16:50 2012) [sssd] [sbus_add_timeout] (0x2000): 0x1539200
(Mon Sep 17 10:16:50 2012) [sssd] [sbus_remove_timeout] (0x2000): 0x15386a0
(Mon Sep 17 10:16:50 2012) [sssd] [ping_check] (0x0100): Service nss replied to ping
(Mon Sep 17 10:16:50 2012) [sssd] [sbus_remove_timeout] (0x2000): 0x1539200
(Mon Sep 17 10:16:50 2012) [sssd] [ping_check] (0x0100): Service pam replied to ping

Thanks,
Mike

> 
> -- 
> Thank you,
> Dmitri Pal
> 
> Sr. Engineering Manager for IdM portfolio
> Red Hat Inc.
> 
> 
> -------------------------------
> Looking to carve out IT costs?
> www.redhat.com/carveoutcosts/
> 
> 
> 