[Freeipa-users] Fw: ssh problem with migrated FreeIPA client on EL7.1 -->Solved
Dmitri Pal
dpal at redhat.com
Sun Jun 7 20:30:28 UTC 2015
On 06/05/2015 01:47 PM, Prasun Gera wrote:
> I had faced a similar issue a month ago, for which I had created a
> ticket. https://fedorahosted.org/freeipa/ticket/4956
>
> On Fri, Jun 5, 2015 at 7:30 AM, Alexander Bokovoy <abokovoy at redhat.com
> <mailto:abokovoy at redhat.com>> wrote:
>
> On Fri, 05 Jun 2015, Christopher Lamb wrote:
>
> Hi Martin
>
> Thanks for updating the documenation!
>
> The suggested solution works not only my test servers, but
> also "in the
> real world". This morning I migrated the last production
> server (ipa host)
> to the new FreeIPA KDC.
>
> Just out of idle curiosity, why is the rm -f
> /var/lib/sss/db/* step
> required on our EL 7.1 + ipa-client 4.1 boxes, but not on our
> older EL 6.5
> + ipa-client 3.3.3 machines?
>
> Is the problem down to sssd? (on the EL 6.5 machines we are
> running sssd
> 1.9.2, while on EL 7.1 we have sssd 1.12.2
>
> I think there are more object types supported by newer SSSD versions
> which aren't invalidated like users or groups.
>
>
>
> Cheers
>
> Chris
>
>
>
> From: Martin Kosek <mkosek at redhat.com
> <mailto:mkosek at redhat.com>>
> To: Christopher Lamb/Switzerland/IBM at IBMCH, Rob Crittenden
> <rcritten at redhat.com <mailto:rcritten at redhat.com>>,
> freeipa-users at redhat.com <mailto:freeipa-users at redhat.com>
> Cc: Jakub Hrozek <jhrozek at redhat.com
> <mailto:jhrozek at redhat.com>>
> Date: 05.06.2015 08:06
> Subject: Re: [Freeipa-users] Fw: ssh problem with
> migrated FreeIPA
> client on EL7.1 -->Solved
>
>
>
> On 06/04/2015 07:34 PM, Christopher Lamb wrote:
>
> Hi All
>
> I can now report back success (at least on my throwaway
> EL7.1 test VM).
>
> To switch an EL 7.1 + ipa-client 4.1 host from an old
> FreeIPA 3.3.3 KDC
>
> to
>
> a new FreeIPA 4.1 KDC 3 steps are required:
>
> 1) ipa-client-install --uninstall
>
> 2) rm -f /var/lib/sss/db/*
>
> 3) ipa-client-install --server ldap.my.example.com
> <http://ldap.my.example.com> --domain
>
> my.example.com <http://my.example.com>
>
> -N
>
> Having done this, my free-ipa user successfully
> authenticates (e.g. ssh
> remote login with free-ipa user / password
>
>
> To switch EL 6.5 + ipa-client 3.3.3 hosts step 2) was not
> required.
>
> Kudos and thanks go to Rob C for suggesting step 2. (Note
> that the
> directory to be purged is /var/lib/sss/db/, not
> /var/lib/sssd/db/ as
> suggested earlier in this thread.
>
>
> Cool! Thanks for reaching back. I added this advice to the FreeIPA
> Troubleshooting guide too:
>
> http://www.freeipa.org/page/Troubleshooting#Cannot_authenticate_on_client
>
>
> Cheers
>
> Chris
>
>
>
>
> From: Martin Kosek <mkosek at redhat.com
> <mailto:mkosek at redhat.com>>
> To: Christopher Lamb/Switzerland/IBM at IBMCH,
> freeipa-users at redhat.com <mailto:freeipa-users at redhat.com>
> Cc: Jakub Hrozek <jhrozek at redhat.com
> <mailto:jhrozek at redhat.com>>, Rob Crittenden
> <rcritten at redhat.com
> <mailto:rcritten at redhat.com>>
> Date: 03.06.2015 10:39
> Subject: Re: [Freeipa-users] Fw: ssh
> problem with migrated
>
> FreeIPA
>
> client on EL7.1 -->Not Solved
>
>
>
> On 06/03/2015 10:30 AM, Christopher Lamb wrote:
>
> Hi all
>
> This is a quick(ish) note to bring everybody up to
> speed on this issue.
> Yesterday we had some private mail exchange on this
> issue as I did not
>
> wish
>
> to broadcast the krb5 and ipa install logs to the user
> list.
>
> The basic situation is that we are in the process of
> migrating from an
> FreeIPA 3.3.3 Server (KDC) to a new FreeIPA 4.1 Server
> (KDC). As
>
> discussed
>
> in a thread some weeks ago we did not do this by
> replicating (as perhaps
>
> we
>
> should have done). Instead we migrated the users across.
>
> We have 30+ servers that are IPA clients ("Hosts" in
> ipa-speak) joined
>
> to
>
> the old KDC. We are now in the process of migrating
> these hosts to the
>
> new
>
> 4.1 KDC.
>
> Most of the hosts run EL 6.5 + ipa-client 3.3.3. For
> all of these
>
> joining
>
> to the new KDC was trouble free, taking a few minutes
> each. After
>
> joining
>
> the new KDC FreeIPA users authenticated properly.
>
> We also had a small number of new EL 7.1 + ipa-client
> 4.1 hosts that
>
> were
>
> joined direct to the new 4.1 KDC, never having been
> joined of the 3.3.3
> KDC. These were also trouble free.
>
> The problem occurs with a handful of existing EL 7.1
> +ipa-client 4.1
>
> hosts
>
> that were originally joined to the 3.3.3 KDC, and must
> be moved to join
>
> the
>
> 4.1 KDC. These machines no longer authenticate valid
> FreeIPA users. I
>
> have
>
> been able to reproduce this behaviour with a freshly
> setup VM joined
>
> first
>
> to the 3.3.3 KDC, then moved to the 4.1 KDC.
>
> While the errors show in the krb5 child logs indicate
> that the password
>
> is
>
> incorrect, the same user / password is happily
> accepted by all the other
> hosts.
>
> It seems that in the process of moving / migrating the
> EL 7.1 /
>
> ipa-client
>
> 4.1 from the old KDC to the new KDC, "something" is
> left behind that
>
> causes
>
> problems. We have seen indications in the install logs
> that the kinit
>
> steps
>
> called during ipa-client install are getting responses
> from the wrong
>
> (old)
>
> KDC, and not from the new KDC.
>
> Frustratingly. over the weekend i managed to get one
> of the problem EL
>
> 7.1
>
> boxes to work. However I can't work out exactly what I
> was that I did
>
> that
>
> did the trick. However it seems that some kind of
> major de-install /
> cleanup + reinstall of the ipa-client may be needed.
>
> Rob has suggested that as part of such a cleanup I
> should do "rm
> -f /var/lib/sssd/db/*". I will test this later today
> and report back.
>
> Thanks to Rob, Jakub, Martin, Alexander et al for
> their help and
> suggestions so far.
>
> Chris
>
>
> Thanks for the background. The pain you are getting is
> exactly the reason
> why
> migration via replication to RHEL-7.1 is a better choice
> :-) Please let
>
> us
>
> know
> the result, I am curious how this works out.
>
>
>
>
>
> From: Martin Kosek
> <mkosek at redhat.com <mailto:mkosek at redhat.com>>
> To: Christopher
> Lamb/Switzerland/IBM at IBMCH,
> freeipa-users at redhat.com
> <mailto:freeipa-users at redhat.com>, Jakub Hrozek
> <jhrozek at redhat.com <mailto:jhrozek at redhat.com>>
> Date: 03.06.2015 09:34
> Subject: Re:
> [Freeipa-users] Fw: ssh problem with
>
> migrated
>
> FreeIPA
>
> client on EL7.1 -->Not Solved
>
>
>
> On 06/02/2015 06:15 PM, Christopher Lamb wrote:
>
>
> Hi
>
> Earlier today I setup 2 throwaway EL7.1 VMs to
> help narrow down the
>
> cause
>
> of this problem. Let's call them HOST09 and HOST10
>
> Both are mimimum installs of EL7.1, with NTPD
> installed and configured.
>
> HOST09 had ipa-client 4.1 installed via yum, and
> was configured to use
>
> our
>
> new FreeIPA 4.1 server, right from the start. -->
> My FreeIPA user
> authenticates successfully against this machine.
>
> HOST10 had ipa-client 4.1 installed as a
> dependency of one of our
>
> standard
>
> config packages, and was first set to use our old
> FreeIPA 3.3.3 server.
>
> -->
>
> My FreeIPA user authenticates successfully.
> against this machine.
>
> I then de-registered HOST10 from the FreeIPA 3.1
> server, and registered
> against the new FreeIPA 4.1 server --> My FreeIPA
> users does NOT
> authenticate successfully.
>
> This replicates well the behaviour I saw with my
> production servers,
>
> namely
>
> a) EL 7.1 hosts with ipa-client 4.1 registered
> directly against the new
>
> 4.1
>
> FreeIPA server authenticate properly.
>
> b) EL 7.1 hosts with ipa-client 4.1 first
> registered against the old
>
> 3.3.3
>
> FreeIPA server, then reregistered with the new 4.1
> FreeIPA server do
>
> NOT
>
> authenticate properly
>
> Chris
>
>
> Hello,
>
> This is really strange. What I do not fully understand
> is what is the
> "registration against a FreeIPA server". What server
> you install IPA
>
> client
>
> should matter if the deployment is set up properly.
> The host enrollment
> entry
> should simply replicate to whole infrastructure. The
> only thing that
>
> will
>
> probably differ is sssd.conf and krb5.conf as they
> will have different
> primary
> server set up, based on what your DNS setup is.
>
> It rather seems that the "reregistration" is what
> causes the issue. It
> looks
> like something cleanup problem during the process. I
> will let Jakub to
>
> help
>
> here, I would suggest including the SSSD logs from the
> failed login, it
>
> may
>
> help.
>
>
>
>
> ----- Forwarded by Christopher
> Lamb/Switzerland/IBM on 02.06.2015 16:52
> -----
>
> From: Christopher
>
> Lamb/Switzerland/IBM at IBMCH
>
> To: Jakub Hrozek
>
> <jhrozek at redhat.com <mailto:jhrozek at redhat.com>>
>
> Cc:
>
> freeipa-users at redhat.com <mailto:freeipa-users at redhat.com>
>
> Date: 02.06.2015 10:40
> Subject: Re:
>
> [Freeipa-users] Fw: ssh problem with
>
> migrated
>
> FreeIPA
>
> client on EL7.1 -->Not Solved
> Sent by:
>
> freeipa-users-bounces at redhat.com
> <mailto:freeipa-users-bounces at redhat.com>
>
>
>
>
> Hi Jakub
>
> Yes root login works, that's how I've been getting
> into the box.
>
> Surprisingly, kinit with my user seems to work on
> that box. After
>
> entering
>
> my password when prompted, it returns to the
> commandline without error.
>
> However if I try kinit with another FreeIPA user,
> then instead of
>
> prompting
>
> for a password, it gives "Generic
> preauthentication failure while
>
> getting
>
> initial credentials" error.
>
> Having set debug_level=10, when I try and ssh in
> with my FreeIPA user,
>
> I
>
> find errors like
>
> "Retrieving host .... with result: .. Matching
> credential not found"
>
> "Received error from KDC ... Additional
> pre-authentication required"
>
> "Received error from KDC... Decrypt integrity
> check failed"
>
> "Received error code 1432158219"
>
> Cheers
>
> Chris
>
>
>
>
>
> From:
>
> Jakub Hrozek
>
> <jhrozek at redhat.com <mailto:jhrozek at redhat.com>>
>
> To:
>
> Christopher
>
> Lamb/Switzerland/IBM at IBMCH
>
> Cc:
>
> freeipa-users at redhat.com <mailto:freeipa-users at redhat.com>
>
> Date:
>
> 02.06.2015 09:50
>
> Subject:
>
> Re:
>
> [Freeipa-users] Fw: ssh problem with
>
> migrated
>
> FreeIPA
> client on EL7.1 -->Not Solved
>
>
>
> On Tue, Jun 02, 2015 at 09:43:48AM +0200,
> Christopher Lamb wrote:
>
> Hi Jakub
>
> The same user / password works with all our
> FreeIPA hosts - just this
>
> one
>
> box is the problem. So the password should be
> good. Of course a type
>
> is
>
> always possible (especially for strong
> passwords), but I have tried
>
> many
>
> times which should eliminate the odd password
> typo. The user /
>
> password
>
> should also be good for both the old and the
> new FreeIPA Server.
>
>
> Interesting, can you add debug_level=10 to the
> domain section of
> sssd.conf? Then krb5_child.log should show
> Kerberos tracing info
> including which exact KDC SSSD was talking to.
>
>
> As I can neither log in direct, or via ssh to
> this box with my FreeIPA
> user, I assume Kinit with my user won't work-
> i will try later in the
>
> day.
>
> Well, login as a UNIX user (root) should work..
>
>
> My working assumption is that the problem is
> related in some way to
>
> the
>
> fact the host originally was a FreeIPA 3.3.3
> client, updated to
>
> FreeIPA
>
> 4.1, and switched between 2 FreeIPA servers. I
> am currently setting up
>
> 2
>
> throwaway EL 7.1 VMs to better test this. On
> one I will first install
> 3.3.3, then upgrade to 4.1. The second will
> have a direct install of
>
> 4.1
>
> client.
>
> Cheers
>
> Chris
>
>
>
> From:
>
> Jakub Hrozek
>
> <jhrozek at redhat.com <mailto:jhrozek at redhat.com>>
>
> To:
>
> freeipa-users at redhat.com <mailto:freeipa-users at redhat.com>
>
> Date:
>
> 02.06.2015 09:22
>
> Subject:
>
> Re:
>
> [Freeipa-users] Fw: ssh problem with
>
> migrated
> FreeIPA
>
> client on EL7.1 -->Not Solved
> Sent by:
>
> freeipa-users-bounces at redhat.com
> <mailto:freeipa-users-bounces at redhat.com>
>
>
>
>
> On Mon, Jun 01, 2015 at 07:35:11PM +0200,
> Christopher Lamb wrote:
>
>
> Hi All
>
> Bad news.
>
> Over the weekend I was able to get the
> original problem EL7.1 /
>
> FreeIPA
>
> 4.1
>
> host (FreeIPA client) to authenticate
> FreeiPA users (my test being
>
> ssh
>
> remote login with FreeIPA user and password).
>
> Today I tried a second machine, and had
> the same problem, ssh
>
> connections
>
> with FreeIPA user cause
> "[sssd[krb5_child[3445]]]: Decrypt integrity
>
> check
>
> failed"
>
>
> This really just means wrong password, can you
> kinit as that user
>
> using
>
> the same password?
>
>
> Ahh I thought, I have a solution for that:
> just remove ipa-client and
> reinstall via yum, register with the new
> FreeIPA server ....
>
> Only with this second machine I still
> can't ssh in with a FreeIPA
>
> user.
>
> Argg.....
>
> b.t.w, as this machine is a real physical
> server, I was able to try
>
> logging
>
> in direct with my FreeIPA user -->
> "Authentication Failure"
>
> I now have
> * a whole bunch of EL6.5 / FreeIPA 3.3.3
> hosts that migrated from the
>
> old
>
> FreeIPA server to the new without a hitch
> (i.e. they successfully
> authenticate FreeIPA users.)
> * one migrated EL7.1 / FreeIPA 4.1 host
> that I was able to migrate,
>
> but
>
> with problems
> * one migrated EL7.1 / FreeIPA 4.1 host
> that so far defies all
>
> attempts
>
> to
>
> authenticate with a FreeIPA user
> * one EL7.1 / FreeIPA 4.1 host that was
> only ever registered with the
>
> new
>
> FreeIPA server, and successfully
> authenticates FreeIPA users.
>
> Any ideas?
>
> Chris
>
>
> ----- Forwarded by Christopher
> Lamb/Switzerland/IBM on 01.06.2015
>
> 19:17
>
> -----
>
> From:
>
>
> Christopher
>
> Lamb/Switzerland/IBM at IBMCH
>
> To:
>
>
> Alexander Bokovoy
>
> <abokovoy at redhat.com <mailto:abokovoy at redhat.com>>,
>
> freeipa-users at redhat.com
> <mailto:freeipa-users at redhat.com>
> Date:
>
>
> 30.05.2015 18:52
>
> Subject:
>
>
> Re:
>
> [Freeipa-users] ssh problem with
> migrated FreeIPA
>
> client on
>
> EL7.1 --> Solved
> Sent by:
>
> freeipa-users-bounces at redhat.com
> <mailto:freeipa-users-bounces at redhat.com>
>
>
>
>
> Hi All
>
> It gives me pleasure to report the problem
> is solved - a minute ago I
>
> was
>
> able to login via ssh with my FreeIPA user
> to the problem server,
>
> while
>
> sitting on my terrace with a glass of wine!
>
> Thanks to Alexander for his helpful advice
> - we had some mail
>
> exchange
>
> outside the user list as I did not wish to
> broadcast content of keys,
> config files etc.
>
> Regardless of what I did with commands
> like klist, kvno everything
>
> seemed
>
> "ok", but I still could not ssh in. Even a
> ipa-getkeytab did not
>
> help.
>
>
> Therefore I decided to opt for brute force
> and (partial) ignorance. I
> completely uninstalled the FreeIPA client,
> and then reinstalled,
>
> configured
>
> - ét voilà I could ssh in!
>
> This leaves the enigma: what caused the
> problem? I suspect the
>
> following:
>
>
> The host is an EL 7.1, but the first
> FreeIPA client installed was
>
> version
>
> 3.3.3 (installed as set of standard
> packages that we bung on all our
> servers).
>
> This worked fine to authenticate against
> our "old" 3.x FreeIPA
>
> server,
>
> but
>
> did not work against the "new" 4.1 FreeIPA
> Server.
>
> When I realised I could not ssh in, one of
> the first things I did was
>
> to
>
> yum update the FreeIPA client from 3.3.3
> to 4.1 - but that did not
>
> help.
>
> The solution was to yum remove the FreeIPA
> client, then yum install
>
> the
>
> 4.1
>
> client.
>
> I have some more EL 7.1 servers with the
> FreeIPA 3.3.3 client
>
> installed,
>
> so
>
> it will be interesting to see it the
> problem can be reproduced.
>
> Keep up the good work,
>
> Chris
>
>
>
>
>
>
>
>
> From:
>
>
> Alexander Bokovoy
>
> <abokovoy at redhat.com <mailto:abokovoy at redhat.com>>
>
> To:
>
>
> Christopher
>
> Lamb/Switzerland/IBM at IBMCH
>
> Cc:
>
> freeipa-users at redhat.com
> <mailto:freeipa-users at redhat.com>
>
> Date:
>
>
> 29.05.2015 18:04
>
> Subject:
>
>
>
> Re:
>
> [Freeipa-users] ssh problem with
>
> migrated FreeIPA
>
> client on
> EL7.1
>
>
>
> On Fri, 29 May 2015, Christopher Lamb wrote:
>
>
> Hi All
>
> Some weeks ago I setup a new FreeIPA
> 4.1.0 on an OEL 7.1 server to
>
> replace
>
> the existing FreeIPA 3.0.0 running on
> OEL 6.5, and successfully
>
> migrated
>
> across the users.
>
> We have 50 odd Servers that are
> FreeIPA clients. Today I started
>
> migrating
>
> these one-by-one from the old FreeIPA
> 3.x server to the new FreeIPA
>
> 4
>
> server by doing an ipa-client-install
> --uninstall from the old, and
> ipa-client-install to register with
> the new 4.1.0 server.
>
> Most of the FreeIPA clients are
> running OEL 6.5, and for these the
> migration process above worked
> perfectly. After migrating the
>
> server,
>
> I
>
> could ssh in with my FreeIPA user.
>
> Then I migrated an OEL 7.1 server. The
> migration itself seemed to
>
> work,
>
> and
>
> getent passwd was successful for my
> FreeIPA user. However when I try
>
> and
>
> ssh in, my FreeIPA user / password is
> not accepted.
>
> Before the migration I could ssh into
> the problem server (though
>
> evidently
>
> it was using my FreeIPA user from the
> old FreeIPA server).
>
> I can ssh in with a local (non ldap)
> user, so ssh is running and
>
> working.
>
>
> >From user root I can successfully su
> to my FreeIPA user.
>
> Further investigation showed that
> version of ipa-client installed
>
> was
>
> 3.3.3, so I yum updated this to 4.1.0.
>
> However I still cannot ssh into the
> OEL 7.1 box with my FreeIPA
>
> user.
>
> The
>
> same user continues to work for the
> 6.5 boxes.
>
> A colleague tried to ssh in with his
> FreeIPA user, and was also
>
> rejected,
>
> so the problem is not my user, but is
> probably for all FreeIPA
>
> users.
>
>
> A failed ssh login attempt causes the
> following error
>
> in /var/log/messages
>
>
> [sssd[krb5_child[5393]]]: Decrypt
> integrity check failed
>
> It means /etc/krb5.keytab contains keys
> from older system and SSSD
> picks them up.
> Can you show output of 'klist -kKet'?
> --
> / Alexander Bokovoy
>
>
>
>
>
> --
> Manage your subscription for the
> Freeipa-users mailing list:
> https://www.redhat.com/mailman/listinfo/freeipa-users
> Go to http://freeipa.org for more info on
> the project
>
>
>
>
> --
> Manage your subscription for the
> Freeipa-users mailing list:
> https://www.redhat.com/mailman/listinfo/freeipa-users
> Go to http://freeipa.org for more info on
> the project
>
>
> --
> Manage your subscription for the Freeipa-users
> mailing list:
> https://www.redhat.com/mailman/listinfo/freeipa-users
> Go to http://freeipa.org for more info on the
> project
>
>
>
>
>
>
>
>
>
> --
> Manage your subscription for the Freeipa-users
> mailing list:
> https://www.redhat.com/mailman/listinfo/freeipa-users
> Go to http://freeipa.org for more info on the project
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
> --
> Manage your subscription for the Freeipa-users mailing list:
> https://www.redhat.com/mailman/listinfo/freeipa-users
> Go to http://freeipa.org for more info on the project
>
>
> --
> / Alexander Bokovoy
>
> --
> Manage your subscription for the Freeipa-users mailing list:
> https://www.redhat.com/mailman/listinfo/freeipa-users
> Go to http://freeipa.org for more info on the project
>
>
>
>
FYI https://fedorahosted.org/freeipa/ticket/5050
--
Thank you,
Dmitri Pal
Director of Engineering for IdM portfolio
Red Hat, Inc.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/freeipa-users/attachments/20150607/13bd5122/attachment.htm>
More information about the Freeipa-users
mailing list