[Freeipa-users] errors when one ipa server down
Michael Mercier
mmercier at gmail.com
Mon Sep 17 14:27:39 UTC 2012
On 2012-09-10, at 4:35 AM, Petr Spacek wrote:
> On 09/08/2012 05:03 PM, Dmitri Pal wrote:
>> On 09/07/2012 04:50 PM, Rob Crittenden wrote:
>>> Michael Mercier wrote:
>>>>
>>>> On 2012-09-07, at 2:47 PM, Dmitri Pal wrote:
>>>>
>>>>> On 09/07/2012 12:42 PM, Michael Mercier wrote:
>>>>>> On 2012-09-07, at 12:14 PM, Dmitri Pal wrote:
>>>>>>
>>>>>>> On 09/06/2012 10:40 AM, Michael Mercier wrote:
>>>>>>>> Hello,
>>>>>>>>
>>>>>>>> I have experienced some odd connectivity issues using MMR with
>>>>>>>> FreeIPA (all systems CentOS 6.3). I have 2 ipa servers
>>>>>>>> (ipaserver / ipaserver2) setup using MMR.
>>>>>>>>
>>>>>>>> [root at ipaserver ~]#ipa-replica-manage list
>>>>>>>> ipaserver.mpls.local: master
>>>>>>>> ipaserver2.mpls.local: master
>>>>>>>> [root at ipaserver ~]# rpm -qa|grep ipa
>>>>>>>> libipa_hbac-1.8.0-32.el6.x86_64
>>>>>>>> ipa-admintools-2.2.0-16.el6.x86_64
>>>>>>>> ipa-server-2.2.0-16.el6.x86_64
>>>>>>>> ipa-pki-ca-theme-9.0.3-7.el6.noarch
>>>>>>>> libipa_hbac-python-1.8.0-32.el6.x86_64
>>>>>>>> ipa-client-2.2.0-16.el6.x86_64
>>>>>>>> ipa-server-selinux-2.2.0-16.el6.x86_64
>>>>>>>> ipa-pki-common-theme-9.0.3-7.el6.noarch
>>>>>>>> python-iniparse-0.3.1-2.1.el6.noarch
>>>>>>>> ipa-python-2.2.0-16.el6.x86_64
>>>>>>>>
>>>>>>>>
>>>>>>>> [root at ipaserver2 ~]#ipa-replica-manage list
>>>>>>>> ipaserver.mpls.local: master
>>>>>>>> ipaserver2.mpls.local: master
>>>>>>>> [root at ipaserver2 ~]# rpm -qa|grep ipa
>>>>>>>> ipa-client-2.2.0-16.el6.x86_64
>>>>>>>> ipa-server-2.2.0-16.el6.x86_64
>>>>>>>> ipa-pki-ca-theme-9.0.3-7.el6.noarch
>>>>>>>> ipa-python-2.2.0-16.el6.x86_64
>>>>>>>> libipa_hbac-1.8.0-32.el6.x86_64
>>>>>>>> python-iniparse-0.3.1-2.1.el6.noarch
>>>>>>>> libipa_hbac-python-1.8.0-32.el6.x86_64
>>>>>>>> ipa-admintools-2.2.0-16.el6.x86_64
>>>>>>>> ipa-server-selinux-2.2.0-16.el6.x86_64
>>>>>>>> ipa-pki-common-theme-9.0.3-7.el6.noarch
>>>>>>>>
>>>>>>>>
>>>>>>>> [mike at ipaclient ~]$ rpm -qa|grep ipa
>>>>>>>> ipa-admintools-2.2.0-16.el6.x86_64
>>>>>>>> python-iniparse-0.3.1-2.1.el6.noarch
>>>>>>>> ipa-python-2.2.0-16.el6.x86_64
>>>>>>>> libipa_hbac-python-1.8.0-32.el6.x86_64
>>>>>>>> ipa-client-2.2.0-16.el6.x86_64
>>>>>>>> libipa_hbac-1.8.0-32.el6.x86_64
>>>>>>>>
>>>>>>>>
>>>>>>>> I have a webserver (zenoss) using kerberos authentication.
>>>>>>>>
>>>>>>>> [root at zenoss ~]# rpm -qa|grep ipa
>>>>>>>> libipa_hbac-1.8.0-32.el6.x86_64
>>>>>>>> libipa_hbac-python-1.8.0-32.el6.x86_64
>>>>>>>> ipa-python-2.2.0-16.el6.x86_64
>>>>>>>> ipa-client-2.2.0-16.el6.x86_64
>>>>>>>> python-iniparse-0.3.1-2.1.el6.noarch
>>>>>>>> ipa-admintools-2.2.0-16.el6.x86_64
>>>>>>>>
>>>>>>>> <Location />
>>>>>>>> SSLRequireSSL
>>>>>>>> AuthType Kerberos
>>>>>>>> AuthName "Kerberos Login"
>>>>>>>>
>>>>>>>> KrbMethodK5Passwd Off
>>>>>>>> KrbAuthRealms MPLS.LOCAL
>>>>>>>> KrbSaveCredentials on
>>>>>>>> KrbServiceName HTTP
>>>>>>>> Krb5KeyTab /etc/http/conf.d/http.keytab
>>>>>>>>
>>>>>>>> AuthLDAPUrl "ldap://ipaserver.mpls.local
>>>>>>>> ipaserver2.mpls.local/dc=mpls,dc=local?krbPrincipalName"
>>>>>>>> RequestHeader set X_REMOTE_USER %{remoteUser}e
>>>>>>>> require ldap-group
>>>>>>>> cn=zenuser,cn=groups,cn=accounts,dc=mpls,dc=local
>>>>>>>> </Location>
>>>>>>>>
>>>>>>>>
>>>>>>>> With both ipaserver and ipaserver2 'up', if I connect to
>>>>>>>> https://zenoss.mpls.local from ipaclient using firefox, I am
>>>>>>>> successfully connected. If on ipaserver I do a 'ifdown eth0' and
>>>>>>>> attempt another connection, it fails. I have also noticed the
>>>>>>>> following:
>>>>>>>>
>>>>>>>> 1. I am unable to use the ipaserver2 management interface when
>>>>>>>> ipaserver is unavailable.
>>>>>>>> 2. It takes a longer period of time to do a kinit
>>>>>>>>
>>>>>>>> If the I then perform:
>>>>>>>> [root at ipaserver ~]#ifup eth0
>>>>>>>>
>>>>>>>> [root at ipaserver2 ~]#ifdown eth0
>>>>>>>>
>>>>>>>> [mike at ipaclient ~]$kinit
>>>>>>>> kinit: Cannot contact any KDC for realm 'MPLS.LOCAL' while
>>>>>>>> getting initial credentials
>>>>>>>>
>>>>>>>> [root at ipaserver2 ~]#ifup eth0
>>>>>>>>
>>>>>>>> [mike at ipaclient ~]$ kinit
>>>>>>>> Password for mike at MPLS.LOCAL:
>>>>>>>> [mike at ipaclient ~]$
>>>>>>>>
>>>>>>>> [root at ipaserver2 ~]#ifdown eth0
>>>>>>>>
>>>>>>>> .. wait number of minutes
>>>>>>>>
>>>>>>>> ipaclient screen locks - type password - after a short delay (~7
>>>>>>>> seconds) screen unlock compeletes
>>>>>>>>
>>>>>>>> [mike at ipaclient ~]$kinit
>>>>>>>> Password for mike at MPLS.LOCAL:
>>>>>>>> [mike at ipaclient ~]$
>>>>>>>>
>>>>>>>> Any ideas?
>>>>>>>>
>>>>>>>> Thanks,
>>>>>>>> Mike
>>>>>>> This seems to be some DNS problem.
>>>>>>> You client does not see the second replica and might have some name
>>>>>>> resolution timeouts.
>>>>>>>
>>>>>>> Please check your dns setup and krb5.conf on the client.
>>>>>>>
>>>>>>> To help more we need more details about you client configuration
>>>>>>> DNS and
>>>>>>> kerberos.
>>>>>> Hi,
>>>>>>
>>>>>> Additional information...
>>>>>>
>>>>>> [root at zenoss ~]#more /etc/resolv.conf
>>>>>> search mpls.local
>>>>>> domain mpls.local
>>>>>> nameserver 172.16.112.5
>>>>>> nameserver 172.16.112.8
>>>>>>
>>>>>> [root at zenoss ~]# more /etc/krb5.conf
>>>>>> #File modified by ipa-client-install
>>>>>>
>>>>>> [libdefaults]
>>>>>> default_realm = MPLS.LOCAL
>>>>>> dns_lookup_realm = true
>>>>>> dns_lookup_kdc = true
>>>>>> rdns = false
>>>>>> ticket_lifetime = 24h
>>>>>> forwardable = yes
>>>>>>
>>>>>> [realms]
>>>>>> MPLS.LOCAL = {
>>>>>> pkinit_anchors = FILE:/etc/ipa/ca.crt
>>>>>> }
>>>>>>
>>>>>> [domain_realm]
>>>>>> .mpls.local = MPLS.LOCAL
>>>>>> mpls.local = MPLS.LOCAL
>>>>>>
>>>>>> [root at ipaclient ~]# more /etc/resolv.conf
>>>>>> # Generated by NetworkManager
>>>>>> search mpls.local
>>>>>> nameserver 172.16.112.5
>>>>>> nameserver 172.16.112.8
>>>>>>
>>>>>> [root at ipaclient ~]# more /etc/krb5.conf
>>>>>> #File modified by ipa-client-install
>>>>>>
>>>>>> [libdefaults]
>>>>>> default_realm = MPLS.LOCAL
>>>>>> dns_lookup_realm = true
>>>>>> dns_lookup_kdc = true
>>>>>> rdns = false
>>>>>> ticket_lifetime = 24h
>>>>>> forwardable = yes
>>>>>>
>>>>>> [realms]
>>>>>> MPLS.LOCAL = {
>>>>>> pkinit_anchors = FILE:/etc/ipa/ca.crt
>>>>>> }
>>>>>>
>>>>>> [domain_realm]
>>>>>> .mpls.local = MPLS.LOCAL
>>>>>> mpls.local = MPLS.LOCAL
>>>>>>
>>>>>> [root at ipaclient ~]# nslookup ipaserver
>>>>>> Server: 172.16.112.5
>>>>>> Address: 172.16.112.5#53
>>>>>>
>>>>>> Name: ipaserver.mpls.local
>>>>>> Address: 172.16.112.5
>>>>>>
>>>>>> [root at ipaserver ~]#ifdown eth0
>>>>>>
>>>>>> [root at ipaclient ~]# nslookup ipaserver
>>>>>> Server: 172.16.112.8
>>>>>> Address: 172.16.112.8#53
>>>>>>
>>>>>> Name: ipaserver.mpls.local
>>>>>> Address: 172.16.112.5
>>>>>>
>>>>>> [root at ipaclient ~]# nslookup ipaserver2
>>>>>> Server: 172.16.112.8
>>>>>> Address: 172.16.112.8#53
>>>>>>
>>>>>> Name: ipaserver2.mpls.local
>>>>>> Address: 172.16.112.8
>>>>>>
>>>>>> Copy/paste from the DNS page on ipaserver/ipaserver2
>>>>>>
>>>>>> @ NS ipaserver.mpls.local.
>>>>>> NS ipaserver2.mpls.local.
>>>>>> _kerberos TXT MPLS.LOCAL
>>>>>> _kerberos-master._tcp SRV 0 100 88 ipaserver
>>>>>> SRV 0 100 88 ipaserver2
>>>>>> _kerberos-master._udp SRV 0 100 88 ipaserver
>>>>>> SRV 0 100 88 ipaserver2
>>>>>> _kerberos._tcp SRV 0 100 88 ipaserver
>>>>>> SRV 0 100 88 ipaserver2
>>>>>> _kerberos._udp SRV 0 100 88 ipaserver
>>>>>> SRV 0 100 88 ipaserver2
>>>>>> _kpasswd._tcp SRV 0 100 464 ipaserver
>>>>>> SRV 0 100 464 ipaserver2
>>>>>> _kpasswd._udp SRV 0 100 464 ipaserver
>>>>>> SRV 0 100 464 ipaserver2
>>>>>> _ldap._tcp SRV 0 100 389 ipaserver
>>>>>> SRV 0 100 389 ipaserver2
>>>>>> _ntp._udp SRV 0 100 123 ipaserver
>>>>>> SRV 0 100 123 ipaserver2
>>>>>> ipaclient A 172.16.112.9
>>>>>> ipaclient2 A 172.16.112.145
>>>>>> ipaserver A 172.16.112.5
>>>>>> ipaserver2 A 172.16.112.8
>>>>>> zenoss A 172.16.112.6
>>>>>>
>>>>>> Thanks,
>>>>>> Mike
>>>>>>
>>>>> I noticed that there is no domain line in the resolv.conf on the
>>>>> client.
>>>>> AFAIU in this case it would determine the domain by the gethostname and
>>>>> in case of network being down it will fail over to the hosts file.
>>>>> I wonder what is in your /etc/hosts?
>>>>> Dose it have just a short host name?
>>>>
>>>> [root at ipaclient ~]# more /etc/hosts
>>>> 127.0.0.1 localhost.localdomain localhost
>>>> ::1 localhost6.localdomain6 localhost6
>>>>
>>>>
>>>> Add domain mpls.local to /etc/resolv.conf
>>>>
>>>> [root at ipaserver ~]#ifdown eth0
>>>>
>>>> [root at ipaclient ~]# kinit mike
>>>> kinit: Cannot contact any KDC for realm 'MPLS.LOCAL' while getting
>>>> initial credentials
>>>> [root at ipaclient ~]# nslookup ipaserver
>>>> Server: 172.16.112.8
>>>> Address: 172.16.112.8#53
>>>>
>>>> Name: ipaserver.mpls.local
>>>> Address: 172.16.112.5
>>>>
>>>> [root at ipaclient ~]# nslookup ipaserver2
>>>> Server: 172.16.112.8
>>>> Address: 172.16.112.8#53
>>>>
>>>> Name: ipaserver2.mpls.local
>>>> Address: 172.16.112.8
>>>>
>>>> add '172.16.112.9 ipaclient.mpls.local ipaclient' to /etc/hosts
>>>>
>>>> [root at ipaserver ~]#ifup eth0
>>>>
>>>> [root at ipaclient ~]# kinit mike
>>>> Password for mike at MPLS.LOCAL:
>>>>
>>>> [root at ipaserver ~]#ifdown eth0
>>>>
>>>> [root at ipaclient ~]# kinit mike
>>>> kinit: Cannot contact any KDC for realm 'MPLS.LOCAL' while getting
>>>> initial credentials
>>>> [root at ipaclient ~]# nslookup -type=srv _kerberos-master._tcp
>>>> Server: 172.16.112.8
>>>> Address: 172.16.112.8#53
>>>>
>>>> _kerberos-master._tcp.mpls.local service = 0 100 88
>>>> ipaserver2.mpls.local.
>>>> _kerberos-master._tcp.mpls.local service = 0 100 88
>>>> ipaserver.mpls.local.
>>>>
>>>> [root at ipaclient ~]# nslookup -type=srv _kerberos-master._udp
>>>> Server: 172.16.112.5
>>>> Address: 172.16.112.5#53
>>>>
>>>> _kerberos-master._udp.mpls.local service = 0 100 88
>>>> ipaserver.mpls.local.
>>>> _kerberos-master._udp.mpls.local service = 0 100 88
>>>> ipaserver2.mpls.local.
>>>>
>>>>
>>>> [root at ipaclient ~]# kinit mike
>>>> kinit: Cannot contact any KDC for realm 'MPLS.LOCAL' while getting
>>>> initial credentials
>>>>
>>>> [root at ipaserver ~]#ifup eth0
>>>>
>>>> [root at ipaclient ~]# kinit mike
>>>> Password for mike at MPLS.LOCAL:
>>>
>>> I'd start with the sssd logs. Is it seeing the main server go offline
>>> and not switching to the second one? Or is it going into offline mode?
>>>
>>> Do you have _srv_ or both servers listed in ipa_server in
>>> /etc/sssd/sssd.conf?
>>>
>>> rob
>>>
>> Rob, may be I am missing something but how SSSD is related in this case?
>> The test is done using kinit not SSSD.
>>
>> It would actually be an interesting test to try the same via SSSD for
>> example do su to mike instead of kinit and see what would happen (watch
>> SSSD logs with high debug level, 8 for example).
>> If that works it would probably mean that kinit does not fail over
>> properly. So this would be a Kerberos kinit bug not IPA/SSSD bug.
>>
>
> AFAIK there is "sssd_krb5_locator_plugin". This plugin changes Kerberos servers dynamically at library level, so kinit should select same server as SSSD.
>
> Manual page sssd_krb5_locator_plugin says:
> If the environment variable SSSD_KRB5_LOCATOR_DEBUG is set to any value debug messages will be sent to stderr.
>
> You can execute
> SSSD_KRB5_LOCATOR_DEBUG=1 kinit ...
Hello,
[root at ipaclient ~]# SSSD_KRB5_LOCATOR_DEBUG=1 kinit mike
[sssd_krb5_locator] sssd_krb5_locator_init called
[sssd_krb5_locator] Found [172.16.112.8] in [/var/lib/sss/pubconf/kdcinfo.MPLS.LOCAL].
[sssd_krb5_locator] sssd_realm[MPLS.LOCAL] requested realm[MPLS.LOCAL] family[0] socktype[2] locate_service[1]
[sssd_krb5_locator] addr[172.16.112.8:88] family[2] socktype[2]
[sssd_krb5_locator] [172.16.112.8] used
[sssd_krb5_locator] sssd_krb5_locator_close called
[sssd_krb5_locator] sssd_krb5_locator_init called
[sssd_krb5_locator] Found [172.16.112.8] in [/var/lib/sss/pubconf/kdcinfo.MPLS.LOCAL].
[sssd_krb5_locator] sssd_realm[MPLS.LOCAL] requested realm[MPLS.LOCAL] family[0] socktype[1] locate_service[1]
[sssd_krb5_locator] addr[172.16.112.8:88] family[2] socktype[1]
[sssd_krb5_locator] [172.16.112.8] used
[sssd_krb5_locator] sssd_krb5_locator_close called
[sssd_krb5_locator] sssd_krb5_locator_init called
[sssd_krb5_locator] Found [172.16.112.8] in [/var/lib/sss/pubconf/kdcinfo.MPLS.LOCAL].
[sssd_krb5_locator] open failed [2][No such file or directory].
[sssd_krb5_locator] reading kpasswd address failed, using kdc address.
[sssd_krb5_locator] sssd_realm[MPLS.LOCAL] requested realm[MPLS.LOCAL] family[2] socktype[2] locate_service[2]
[sssd_krb5_locator] addr[172.16.112.8:88] family[2] socktype[2]
[sssd_krb5_locator] [172.16.112.8] used
[sssd_krb5_locator] sssd_krb5_locator_close called
Password for mike at MPLS.LOCAL:
[sssd_krb5_locator] sssd_krb5_locator_init called
[sssd_krb5_locator] Found [172.16.112.8] in [/var/lib/sss/pubconf/kdcinfo.MPLS.LOCAL].
[sssd_krb5_locator] open failed [2][No such file or directory].
[sssd_krb5_locator] reading kpasswd address failed, using kdc address.
[sssd_krb5_locator] sssd_realm[MPLS.LOCAL] requested realm[MPLS.LOCAL] family[0] socktype[2] locate_service[2]
[sssd_krb5_locator] addr[172.16.112.8:88] family[2] socktype[2]
[sssd_krb5_locator] [172.16.112.8] used
[sssd_krb5_locator] sssd_krb5_locator_close called
[sssd_krb5_locator] sssd_krb5_locator_init called
[sssd_krb5_locator] Found [172.16.112.8] in [/var/lib/sss/pubconf/kdcinfo.MPLS.LOCAL].
[sssd_krb5_locator] open failed [2][No such file or directory].
[sssd_krb5_locator] reading kpasswd address failed, using kdc address.
[sssd_krb5_locator] sssd_realm[MPLS.LOCAL] requested realm[MPLS.LOCAL] family[0] socktype[1] locate_service[2]
[sssd_krb5_locator] addr[172.16.112.8:88] family[2] socktype[1]
[sssd_krb5_locator] [172.16.112.8] used
[sssd_krb5_locator] sssd_krb5_locator_close called
[root at ipaserver2 ~]ifdown eth0 # NOTE: ipaserver2 is 172.16.112.8
[root at ipaclient ~]# SSSD_KRB5_LOCATOR_DEBUG=1 kinit mike
[sssd_krb5_locator] sssd_krb5_locator_init called
[sssd_krb5_locator] Found [172.16.112.8] in [/var/lib/sss/pubconf/kdcinfo.MPLS.LOCAL].
[sssd_krb5_locator] sssd_realm[MPLS.LOCAL] requested realm[MPLS.LOCAL] family[0] socktype[2] locate_service[1]
[sssd_krb5_locator] addr[172.16.112.8:88] family[2] socktype[2]
[sssd_krb5_locator] [172.16.112.8] used
[sssd_krb5_locator] sssd_krb5_locator_close called
[sssd_krb5_locator] sssd_krb5_locator_init called
[sssd_krb5_locator] Found [172.16.112.8] in [/var/lib/sss/pubconf/kdcinfo.MPLS.LOCAL].
[sssd_krb5_locator] sssd_realm[MPLS.LOCAL] requested realm[MPLS.LOCAL] family[0] socktype[1] locate_service[1]
[sssd_krb5_locator] addr[172.16.112.8:88] family[2] socktype[1]
[sssd_krb5_locator] [172.16.112.8] used
[sssd_krb5_locator] sssd_krb5_locator_close called
kinit: Cannot contact any KDC for realm 'MPLS.LOCAL' while getting initial credentials
Thanks,
Mike
More information about the Freeipa-users
mailing list