[Freeipa-users] errors when one ipa server down
Sumit Bose
sbose at redhat.com
Mon Sep 10 14:36:51 UTC 2012
On Mon, Sep 10, 2012 at 10:07:26AM -0400, Simo Sorce wrote:
> On Mon, 2012-09-10 at 15:20 +0200, Jakub Hrozek wrote:
> > On Mon, Sep 10, 2012 at 09:08:07AM -0400, Rob Crittenden wrote:
> > > Dmitri Pal wrote:
> > > >On 09/07/2012 04:50 PM, Rob Crittenden wrote:
> > > >>Michael Mercier wrote:
> > > >>>
> > > >>>On 2012-09-07, at 2:47 PM, Dmitri Pal wrote:
> > > >>>
> > > >>>>On 09/07/2012 12:42 PM, Michael Mercier wrote:
> > > >>>>>On 2012-09-07, at 12:14 PM, Dmitri Pal wrote:
> > > >>>>>
> > > >>>>>>On 09/06/2012 10:40 AM, Michael Mercier wrote:
> > > >>>>>>>Hello,
> > > >>>>>>>
> > > >>>>>>>I have experienced some odd connectivity issues using MMR with
> > > >>>>>>>FreeIPA (all systems CentOS 6.3). I have 2 ipa servers
> > > >>>>>>>(ipaserver / ipaserver2) setup using MMR.
> > > >>>>>>>
> > > >>>>>>>[root at ipaserver ~]#ipa-replica-manage list
> > > >>>>>>>ipaserver.mpls.local: master
> > > >>>>>>>ipaserver2.mpls.local: master
> > > >>>>>>>[root at ipaserver ~]# rpm -qa|grep ipa
> > > >>>>>>>libipa_hbac-1.8.0-32.el6.x86_64
> > > >>>>>>>ipa-admintools-2.2.0-16.el6.x86_64
> > > >>>>>>>ipa-server-2.2.0-16.el6.x86_64
> > > >>>>>>>ipa-pki-ca-theme-9.0.3-7.el6.noarch
> > > >>>>>>>libipa_hbac-python-1.8.0-32.el6.x86_64
> > > >>>>>>>ipa-client-2.2.0-16.el6.x86_64
> > > >>>>>>>ipa-server-selinux-2.2.0-16.el6.x86_64
> > > >>>>>>>ipa-pki-common-theme-9.0.3-7.el6.noarch
> > > >>>>>>>python-iniparse-0.3.1-2.1.el6.noarch
> > > >>>>>>>ipa-python-2.2.0-16.el6.x86_64
> > > >>>>>>>
> > > >>>>>>>
> > > >>>>>>>[root at ipaserver2 ~]#ipa-replica-manage list
> > > >>>>>>>ipaserver.mpls.local: master
> > > >>>>>>>ipaserver2.mpls.local: master
> > > >>>>>>>[root at ipaserver2 ~]# rpm -qa|grep ipa
> > > >>>>>>>ipa-client-2.2.0-16.el6.x86_64
> > > >>>>>>>ipa-server-2.2.0-16.el6.x86_64
> > > >>>>>>>ipa-pki-ca-theme-9.0.3-7.el6.noarch
> > > >>>>>>>ipa-python-2.2.0-16.el6.x86_64
> > > >>>>>>>libipa_hbac-1.8.0-32.el6.x86_64
> > > >>>>>>>python-iniparse-0.3.1-2.1.el6.noarch
> > > >>>>>>>libipa_hbac-python-1.8.0-32.el6.x86_64
> > > >>>>>>>ipa-admintools-2.2.0-16.el6.x86_64
> > > >>>>>>>ipa-server-selinux-2.2.0-16.el6.x86_64
> > > >>>>>>>ipa-pki-common-theme-9.0.3-7.el6.noarch
> > > >>>>>>>
> > > >>>>>>>
> > > >>>>>>>[mike at ipaclient ~]$ rpm -qa|grep ipa
> > > >>>>>>>ipa-admintools-2.2.0-16.el6.x86_64
> > > >>>>>>>python-iniparse-0.3.1-2.1.el6.noarch
> > > >>>>>>>ipa-python-2.2.0-16.el6.x86_64
> > > >>>>>>>libipa_hbac-python-1.8.0-32.el6.x86_64
> > > >>>>>>>ipa-client-2.2.0-16.el6.x86_64
> > > >>>>>>>libipa_hbac-1.8.0-32.el6.x86_64
> > > >>>>>>>
> > > >>>>>>>
> > > >>>>>>>I have a webserver (zenoss) using kerberos authentication.
> > > >>>>>>>
> > > >>>>>>>[root at zenoss ~]# rpm -qa|grep ipa
> > > >>>>>>>libipa_hbac-1.8.0-32.el6.x86_64
> > > >>>>>>>libipa_hbac-python-1.8.0-32.el6.x86_64
> > > >>>>>>>ipa-python-2.2.0-16.el6.x86_64
> > > >>>>>>>ipa-client-2.2.0-16.el6.x86_64
> > > >>>>>>>python-iniparse-0.3.1-2.1.el6.noarch
> > > >>>>>>>ipa-admintools-2.2.0-16.el6.x86_64
> > > >>>>>>>
> > > >>>>>>><Location />
> > > >>>>>>> SSLRequireSSL
> > > >>>>>>> AuthType Kerberos
> > > >>>>>>> AuthName "Kerberos Login"
> > > >>>>>>>
> > > >>>>>>> KrbMethodK5Passwd Off
> > > >>>>>>> KrbAuthRealms MPLS.LOCAL
> > > >>>>>>> KrbSaveCredentials on
> > > >>>>>>> KrbServiceName HTTP
> > > >>>>>>> Krb5KeyTab /etc/http/conf.d/http.keytab
> > > >>>>>>>
> > > >>>>>>> AuthLDAPUrl "ldap://ipaserver.mpls.local
> > > >>>>>>>ipaserver2.mpls.local/dc=mpls,dc=local?krbPrincipalName"
> > > >>>>>>> RequestHeader set X_REMOTE_USER %{remoteUser}e
> > > >>>>>>> require ldap-group
> > > >>>>>>>cn=zenuser,cn=groups,cn=accounts,dc=mpls,dc=local
> > > >>>>>>></Location>
> > > >>>>>>>
> > > >>>>>>>
> > > >>>>>>>With both ipaserver and ipaserver2 'up', if I connect to
> > > >>>>>>>https://zenoss.mpls.local from ipaclient using firefox, I am
> > > >>>>>>>successfully connected. If on ipaserver I do a 'ifdown eth0' and
> > > >>>>>>>attempt another connection, it fails. I have also noticed the
> > > >>>>>>>following:
> > > >>>>>>>
> > > >>>>>>>1. I am unable to use the ipaserver2 management interface when
> > > >>>>>>>ipaserver is unavailable.
> > > >>>>>>>2. It takes a longer period of time to do a kinit
> > > >>>>>>>
> > > >>>>>>>If the I then perform:
> > > >>>>>>>[root at ipaserver ~]#ifup eth0
> > > >>>>>>>
> > > >>>>>>>[root at ipaserver2 ~]#ifdown eth0
> > > >>>>>>>
> > > >>>>>>>[mike at ipaclient ~]$kinit
> > > >>>>>>>kinit: Cannot contact any KDC for realm 'MPLS.LOCAL' while
> > > >>>>>>>getting initial credentials
> > > >>>>>>>
> > > >>>>>>>[root at ipaserver2 ~]#ifup eth0
> > > >>>>>>>
> > > >>>>>>>[mike at ipaclient ~]$ kinit
> > > >>>>>>>Password for mike at MPLS.LOCAL:
> > > >>>>>>>[mike at ipaclient ~]$
> > > >>>>>>>
> > > >>>>>>>[root at ipaserver2 ~]#ifdown eth0
> > > >>>>>>>
> > > >>>>>>>.. wait number of minutes
> > > >>>>>>>
> > > >>>>>>>ipaclient screen locks - type password - after a short delay (~7
> > > >>>>>>>seconds) screen unlock compeletes
> > > >>>>>>>
> > > >>>>>>>[mike at ipaclient ~]$kinit
> > > >>>>>>>Password for mike at MPLS.LOCAL:
> > > >>>>>>>[mike at ipaclient ~]$
> > > >>>>>>>
> > > >>>>>>>Any ideas?
> > > >>>>>>>
> > > >>>>>>>Thanks,
> > > >>>>>>>Mike
> > > >>>>>>This seems to be some DNS problem.
> > > >>>>>>You client does not see the second replica and might have some name
> > > >>>>>>resolution timeouts.
> > > >>>>>>
> > > >>>>>>Please check your dns setup and krb5.conf on the client.
> > > >>>>>>
> > > >>>>>>To help more we need more details about you client configuration
> > > >>>>>>DNS and
> > > >>>>>>kerberos.
> > > >>>>>Hi,
> > > >>>>>
> > > >>>>>Additional information...
> > > >>>>>
> > > >>>>>[root at zenoss ~]#more /etc/resolv.conf
> > > >>>>>search mpls.local
> > > >>>>>domain mpls.local
> > > >>>>>nameserver 172.16.112.5
> > > >>>>>nameserver 172.16.112.8
> > > >>>>>
> > > >>>>>[root at zenoss ~]# more /etc/krb5.conf
> > > >>>>>#File modified by ipa-client-install
> > > >>>>>
> > > >>>>>[libdefaults]
> > > >>>>> default_realm = MPLS.LOCAL
> > > >>>>> dns_lookup_realm = true
> > > >>>>> dns_lookup_kdc = true
> > > >>>>> rdns = false
> > > >>>>> ticket_lifetime = 24h
> > > >>>>> forwardable = yes
> > > >>>>>
> > > >>>>>[realms]
> > > >>>>> MPLS.LOCAL = {
> > > >>>>> pkinit_anchors = FILE:/etc/ipa/ca.crt
> > > >>>>> }
> > > >>>>>
> > > >>>>>[domain_realm]
> > > >>>>> .mpls.local = MPLS.LOCAL
> > > >>>>> mpls.local = MPLS.LOCAL
> > > >>>>>
> > > >>>>>[root at ipaclient ~]# more /etc/resolv.conf
> > > >>>>># Generated by NetworkManager
> > > >>>>>search mpls.local
> > > >>>>>nameserver 172.16.112.5
> > > >>>>>nameserver 172.16.112.8
> > > >>>>>
> > > >>>>>[root at ipaclient ~]# more /etc/krb5.conf
> > > >>>>>#File modified by ipa-client-install
> > > >>>>>
> > > >>>>>[libdefaults]
> > > >>>>> default_realm = MPLS.LOCAL
> > > >>>>> dns_lookup_realm = true
> > > >>>>> dns_lookup_kdc = true
> > > >>>>> rdns = false
> > > >>>>> ticket_lifetime = 24h
> > > >>>>> forwardable = yes
> > > >>>>>
> > > >>>>>[realms]
> > > >>>>> MPLS.LOCAL = {
> > > >>>>> pkinit_anchors = FILE:/etc/ipa/ca.crt
> > > >>>>> }
> > > >>>>>
> > > >>>>>[domain_realm]
> > > >>>>> .mpls.local = MPLS.LOCAL
> > > >>>>> mpls.local = MPLS.LOCAL
> > > >>>>>
> > > >>>>>[root at ipaclient ~]# nslookup ipaserver
> > > >>>>>Server: 172.16.112.5
> > > >>>>>Address: 172.16.112.5#53
> > > >>>>>
> > > >>>>>Name: ipaserver.mpls.local
> > > >>>>>Address: 172.16.112.5
> > > >>>>>
> > > >>>>>[root at ipaserver ~]#ifdown eth0
> > > >>>>>
> > > >>>>>[root at ipaclient ~]# nslookup ipaserver
> > > >>>>>Server: 172.16.112.8
> > > >>>>>Address: 172.16.112.8#53
> > > >>>>>
> > > >>>>>Name: ipaserver.mpls.local
> > > >>>>>Address: 172.16.112.5
> > > >>>>>
> > > >>>>>[root at ipaclient ~]# nslookup ipaserver2
> > > >>>>>Server: 172.16.112.8
> > > >>>>>Address: 172.16.112.8#53
> > > >>>>>
> > > >>>>>Name: ipaserver2.mpls.local
> > > >>>>>Address: 172.16.112.8
> > > >>>>>
> > > >>>>>Copy/paste from the DNS page on ipaserver/ipaserver2
> > > >>>>>
> > > >>>>>@ NS ipaserver.mpls.local.
> > > >>>>> NS ipaserver2.mpls.local.
> > > >>>>>_kerberos TXT MPLS.LOCAL
> > > >>>>>_kerberos-master._tcp SRV 0 100 88 ipaserver
> > > >>>>> SRV 0 100 88 ipaserver2
> > > >>>>>_kerberos-master._udp SRV 0 100 88 ipaserver
> > > >>>>> SRV 0 100 88 ipaserver2
> > > >>>>>_kerberos._tcp SRV 0 100 88 ipaserver
> > > >>>>> SRV 0 100 88 ipaserver2
> > > >>>>>_kerberos._udp SRV 0 100 88 ipaserver
> > > >>>>> SRV 0 100 88 ipaserver2
> > > >>>>>_kpasswd._tcp SRV 0 100 464 ipaserver
> > > >>>>> SRV 0 100 464 ipaserver2
> > > >>>>>_kpasswd._udp SRV 0 100 464 ipaserver
> > > >>>>> SRV 0 100 464 ipaserver2
> > > >>>>>_ldap._tcp SRV 0 100 389 ipaserver
> > > >>>>> SRV 0 100 389 ipaserver2
> > > >>>>>_ntp._udp SRV 0 100 123 ipaserver
> > > >>>>> SRV 0 100 123 ipaserver2
> > > >>>>>ipaclient A 172.16.112.9
> > > >>>>>ipaclient2 A 172.16.112.145
> > > >>>>>ipaserver A 172.16.112.5
> > > >>>>>ipaserver2 A 172.16.112.8
> > > >>>>>zenoss A 172.16.112.6
> > > >>>>>
> > > >>>>>Thanks,
> > > >>>>>Mike
> > > >>>>>
> > > >>>>I noticed that there is no domain line in the resolv.conf on the
> > > >>>>client.
> > > >>>>AFAIU in this case it would determine the domain by the gethostname and
> > > >>>>in case of network being down it will fail over to the hosts file.
> > > >>>>I wonder what is in your /etc/hosts?
> > > >>>>Dose it have just a short host name?
> > > >>>
> > > >>>[root at ipaclient ~]# more /etc/hosts
> > > >>>127.0.0.1 localhost.localdomain localhost
> > > >>>::1 localhost6.localdomain6 localhost6
> > > >>>
> > > >>>
> > > >>>Add domain mpls.local to /etc/resolv.conf
> > > >>>
> > > >>>[root at ipaserver ~]#ifdown eth0
> > > >>>
> > > >>>[root at ipaclient ~]# kinit mike
> > > >>>kinit: Cannot contact any KDC for realm 'MPLS.LOCAL' while getting
> > > >>>initial credentials
> > > >>>[root at ipaclient ~]# nslookup ipaserver
> > > >>>Server: 172.16.112.8
> > > >>>Address: 172.16.112.8#53
> > > >>>
> > > >>>Name: ipaserver.mpls.local
> > > >>>Address: 172.16.112.5
> > > >>>
> > > >>>[root at ipaclient ~]# nslookup ipaserver2
> > > >>>Server: 172.16.112.8
> > > >>>Address: 172.16.112.8#53
> > > >>>
> > > >>>Name: ipaserver2.mpls.local
> > > >>>Address: 172.16.112.8
> > > >>>
> > > >>>add '172.16.112.9 ipaclient.mpls.local ipaclient' to /etc/hosts
> > > >>>
> > > >>>[root at ipaserver ~]#ifup eth0
> > > >>>
> > > >>>[root at ipaclient ~]# kinit mike
> > > >>>Password for mike at MPLS.LOCAL:
> > > >>>
> > > >>>[root at ipaserver ~]#ifdown eth0
> > > >>>
> > > >>>[root at ipaclient ~]# kinit mike
> > > >>>kinit: Cannot contact any KDC for realm 'MPLS.LOCAL' while getting
> > > >>>initial credentials
> > > >>>[root at ipaclient ~]# nslookup -type=srv _kerberos-master._tcp
> > > >>>Server: 172.16.112.8
> > > >>>Address: 172.16.112.8#53
> > > >>>
> > > >>>_kerberos-master._tcp.mpls.local service = 0 100 88
> > > >>>ipaserver2.mpls.local.
> > > >>>_kerberos-master._tcp.mpls.local service = 0 100 88
> > > >>>ipaserver.mpls.local.
> > > >>>
> > > >>>[root at ipaclient ~]# nslookup -type=srv _kerberos-master._udp
> > > >>>Server: 172.16.112.5
> > > >>>Address: 172.16.112.5#53
> > > >>>
> > > >>>_kerberos-master._udp.mpls.local service = 0 100 88
> > > >>>ipaserver.mpls.local.
> > > >>>_kerberos-master._udp.mpls.local service = 0 100 88
> > > >>>ipaserver2.mpls.local.
> > > >>>
> > > >>>
> > > >>>[root at ipaclient ~]# kinit mike
> > > >>>kinit: Cannot contact any KDC for realm 'MPLS.LOCAL' while getting
> > > >>>initial credentials
> > > >>>
> > > >>>[root at ipaserver ~]#ifup eth0
> > > >>>
> > > >>>[root at ipaclient ~]# kinit mike
> > > >>>Password for mike at MPLS.LOCAL:
> > > >>
> > > >>I'd start with the sssd logs. Is it seeing the main server go offline
> > > >>and not switching to the second one? Or is it going into offline mode?
> > > >>
> > > >>Do you have _srv_ or both servers listed in ipa_server in
> > > >>/etc/sssd/sssd.conf?
> > > >>
> > > >>rob
> > > >>
> > > >Rob, may be I am missing something but how SSSD is related in this case?
> > > >The test is done using kinit not SSSD.
> > > >
> > > >It would actually be an interesting test to try the same via SSSD for
> > > >example do su to mike instead of kinit and see what would happen (watch
> > > >SSSD logs with high debug level, 8 for example).
> > > >If that works it would probably mean that kinit does not fail over
> > > >properly. So this would be a Kerberos kinit bug not IPA/SSSD bug.
> > > >
> > >
> > > SSSD controls the Kerberos locator. If SSSD isn't detecting that the
> > > KDC is down then it is going to point the user to a non-working
> > > server.
> > >
> > > rob
> >
> > The SSSD only creates the file used by the locator when the first auth
> > request comes in trough the SSSD (in the case of IPA backed even an
> > identity lookup would do because it's GSSAPI-encrypted).
> >
> > Bottom line, just logging in as root and performing kinit is not enough,
> > kinit completely bypasses the SSSD and talks to the Kerberos server
> > directly.
>
> We have been discussing with Stephen about changing how the locator
> plugin works.
> Currently it is completely passive, ie it only reads a file and acts on
> it.
>
> We discussed about making the locator plugin able to 'ping' sssd and
> ask it to refresh the status of the file.
> However this is trickier than it sounds because we do not want to
> contact sssd every single time DNS resolution is needed, so we may have
> to put expiration timestamps or similar. We also need to properly back
> off if sssd is not responding and so on.
What about defining a task in the SSSD krb5 provider instead of pinging
it from the locator plugin. The task can run at a configurable interval
or never and checks if the current KDC is available. If not it tries the
next until it goes offline if no reachable KDC can be found and updates
or deletes the info file for the locator plugin..
This leave us with the question how to ping a KDC properly, but this we
have to find out for either case.
bye,
Sumit
>
> Requires some careful design to avoid turning it into a worst case for
> every resolution instead of an annoyances only once in a while.
>
> Simo.
>
> --
> Simo Sorce * Red Hat, Inc * New York
>
> _______________________________________________
> Freeipa-users mailing list
> Freeipa-users at redhat.com
> https://www.redhat.com/mailman/listinfo/freeipa-users
More information about the Freeipa-users
mailing list