[Freeipa-users] Fw: ssh problem with migrated FreeIPA client on EL7.1 -->Solved
Christopher Lamb
christopher.lamb at ch.ibm.com
Fri Jun 5 14:25:09 UTC 2015
Hi Martin
Thanks for updating the documenation!
The suggested solution works not only my test servers, but also "in the
real world". This morning I migrated the last production server (ipa host)
to the new FreeIPA KDC.
Just out of idle curiosity, why is the rm -f /var/lib/sss/db/* step
required on our EL 7.1 + ipa-client 4.1 boxes, but not on our older EL 6.5
+ ipa-client 3.3.3 machines?
Is the problem down to sssd? (on the EL 6.5 machines we are running sssd
1.9.2, while on EL 7.1 we have sssd 1.12.2
Cheers
Chris
From: Martin Kosek <mkosek at redhat.com>
To: Christopher Lamb/Switzerland/IBM at IBMCH, Rob Crittenden
<rcritten at redhat.com>, freeipa-users at redhat.com
Cc: Jakub Hrozek <jhrozek at redhat.com>
Date: 05.06.2015 08:06
Subject: Re: [Freeipa-users] Fw: ssh problem with migrated FreeIPA
client on EL7.1 -->Solved
On 06/04/2015 07:34 PM, Christopher Lamb wrote:
> Hi All
>
> I can now report back success (at least on my throwaway EL7.1 test VM).
>
> To switch an EL 7.1 + ipa-client 4.1 host from an old FreeIPA 3.3.3 KDC
to
> a new FreeIPA 4.1 KDC 3 steps are required:
>
> 1) ipa-client-install --uninstall
>
> 2) rm -f /var/lib/sss/db/*
>
> 3) ipa-client-install --server ldap.my.example.com --domain
my.example.com
> -N
>
> Having done this, my free-ipa user successfully authenticates (e.g. ssh
> remote login with free-ipa user / password
>
>
> To switch EL 6.5 + ipa-client 3.3.3 hosts step 2) was not required.
>
> Kudos and thanks go to Rob C for suggesting step 2. (Note that the
> directory to be purged is /var/lib/sss/db/, not /var/lib/sssd/db/ as
> suggested earlier in this thread.
Cool! Thanks for reaching back. I added this advice to the FreeIPA
Troubleshooting guide too:
http://www.freeipa.org/page/Troubleshooting#Cannot_authenticate_on_client
>
> Cheers
>
> Chris
>
>
>
>
> From: Martin Kosek <mkosek at redhat.com>
> To: Christopher Lamb/Switzerland/IBM at IBMCH,
> freeipa-users at redhat.com
> Cc: Jakub Hrozek <jhrozek at redhat.com>, Rob Crittenden
> <rcritten at redhat.com>
> Date: 03.06.2015 10:39
> Subject: Re: [Freeipa-users] Fw: ssh problem with migrated
FreeIPA
> client on EL7.1 -->Not Solved
>
>
>
> On 06/03/2015 10:30 AM, Christopher Lamb wrote:
>> Hi all
>>
>> This is a quick(ish) note to bring everybody up to speed on this issue.
>> Yesterday we had some private mail exchange on this issue as I did not
> wish
>> to broadcast the krb5 and ipa install logs to the user list.
>>
>> The basic situation is that we are in the process of migrating from an
>> FreeIPA 3.3.3 Server (KDC) to a new FreeIPA 4.1 Server (KDC). As
> discussed
>> in a thread some weeks ago we did not do this by replicating (as perhaps
> we
>> should have done). Instead we migrated the users across.
>>
>> We have 30+ servers that are IPA clients ("Hosts" in ipa-speak) joined
to
>> the old KDC. We are now in the process of migrating these hosts to the
> new
>> 4.1 KDC.
>>
>> Most of the hosts run EL 6.5 + ipa-client 3.3.3. For all of these
> joining
>> to the new KDC was trouble free, taking a few minutes each. After
joining
>> the new KDC FreeIPA users authenticated properly.
>>
>> We also had a small number of new EL 7.1 + ipa-client 4.1 hosts that
were
>> joined direct to the new 4.1 KDC, never having been joined of the 3.3.3
>> KDC. These were also trouble free.
>>
>> The problem occurs with a handful of existing EL 7.1 +ipa-client 4.1
> hosts
>> that were originally joined to the 3.3.3 KDC, and must be moved to join
> the
>> 4.1 KDC. These machines no longer authenticate valid FreeIPA users. I
> have
>> been able to reproduce this behaviour with a freshly setup VM joined
> first
>> to the 3.3.3 KDC, then moved to the 4.1 KDC.
>>
>> While the errors show in the krb5 child logs indicate that the password
> is
>> incorrect, the same user / password is happily accepted by all the other
>> hosts.
>>
>> It seems that in the process of moving / migrating the EL 7.1 /
> ipa-client
>> 4.1 from the old KDC to the new KDC, "something" is left behind that
> causes
>> problems. We have seen indications in the install logs that the kinit
> steps
>> called during ipa-client install are getting responses from the wrong
> (old)
>> KDC, and not from the new KDC.
>>
>> Frustratingly. over the weekend i managed to get one of the problem EL
> 7.1
>> boxes to work. However I can't work out exactly what I was that I did
> that
>> did the trick. However it seems that some kind of major de-install /
>> cleanup + reinstall of the ipa-client may be needed.
>>
>> Rob has suggested that as part of such a cleanup I should do "rm
>> -f /var/lib/sssd/db/*". I will test this later today and report back.
>>
>> Thanks to Rob, Jakub, Martin, Alexander et al for their help and
>> suggestions so far.
>>
>> Chris
>
> Thanks for the background. The pain you are getting is exactly the reason
> why
> migration via replication to RHEL-7.1 is a better choice :-) Please let
us
> know
> the result, I am curious how this works out.
>
>>
>>
>>
>>
>> From: Martin Kosek <mkosek at redhat.com>
>> To: Christopher Lamb/Switzerland/IBM at IBMCH,
>> freeipa-users at redhat.com, Jakub Hrozek <jhrozek at redhat.com>
>> Date: 03.06.2015 09:34
>> Subject: Re: [Freeipa-users] Fw: ssh problem with
migrated
> FreeIPA
>> client on EL7.1 -->Not Solved
>>
>>
>>
>> On 06/02/2015 06:15 PM, Christopher Lamb wrote:
>>>
>>> Hi
>>>
>>> Earlier today I setup 2 throwaway EL7.1 VMs to help narrow down the
> cause
>>> of this problem. Let's call them HOST09 and HOST10
>>>
>>> Both are mimimum installs of EL7.1, with NTPD installed and configured.
>>>
>>> HOST09 had ipa-client 4.1 installed via yum, and was configured to use
>> our
>>> new FreeIPA 4.1 server, right from the start. --> My FreeIPA user
>>> authenticates successfully against this machine.
>>>
>>> HOST10 had ipa-client 4.1 installed as a dependency of one of our
>> standard
>>> config packages, and was first set to use our old FreeIPA 3.3.3 server.
>> -->
>>> My FreeIPA user authenticates successfully. against this machine.
>>>
>>> I then de-registered HOST10 from the FreeIPA 3.1 server, and registered
>>> against the new FreeIPA 4.1 server --> My FreeIPA users does NOT
>>> authenticate successfully.
>>>
>>> This replicates well the behaviour I saw with my production servers,
>> namely
>>> a) EL 7.1 hosts with ipa-client 4.1 registered directly against the new
>> 4.1
>>> FreeIPA server authenticate properly.
>>>
>>> b) EL 7.1 hosts with ipa-client 4.1 first registered against the old
>> 3.3.3
>>> FreeIPA server, then reregistered with the new 4.1 FreeIPA server do
NOT
>>> authenticate properly
>>>
>>> Chris
>>
>> Hello,
>>
>> This is really strange. What I do not fully understand is what is the
>> "registration against a FreeIPA server". What server you install IPA
> client
>> should matter if the deployment is set up properly. The host enrollment
>> entry
>> should simply replicate to whole infrastructure. The only thing that
will
>> probably differ is sssd.conf and krb5.conf as they will have different
>> primary
>> server set up, based on what your DNS setup is.
>>
>> It rather seems that the "reregistration" is what causes the issue. It
>> looks
>> like something cleanup problem during the process. I will let Jakub to
> help
>> here, I would suggest including the SSSD logs from the failed login, it
> may
>> help.
>>
>>>
>>>
>>>
>>> ----- Forwarded by Christopher Lamb/Switzerland/IBM on 02.06.2015 16:52
>>> -----
>>>
>>> From: Christopher
Lamb/Switzerland/IBM at IBMCH
>>> To: Jakub Hrozek
<jhrozek at redhat.com>
>>> Cc:
freeipa-users at redhat.com
>>> Date: 02.06.2015 10:40
>>> Subject: Re:
[Freeipa-users] Fw: ssh problem with
> migrated
>> FreeIPA
>>> client on EL7.1 -->Not Solved
>>> Sent by:
freeipa-users-bounces at redhat.com
>>>
>>>
>>>
>>> Hi Jakub
>>>
>>> Yes root login works, that's how I've been getting into the box.
>>>
>>> Surprisingly, kinit with my user seems to work on that box. After
>> entering
>>> my password when prompted, it returns to the commandline without error.
>>>
>>> However if I try kinit with another FreeIPA user, then instead of
>> prompting
>>> for a password, it gives "Generic preauthentication failure while
> getting
>>> initial credentials" error.
>>>
>>> Having set debug_level=10, when I try and ssh in with my FreeIPA user,
I
>>> find errors like
>>>
>>> "Retrieving host .... with result: .. Matching credential not found"
>>>
>>> "Received error from KDC ... Additional pre-authentication required"
>>>
>>> "Received error from KDC... Decrypt integrity check failed"
>>>
>>> "Received error code 1432158219"
>>>
>>> Cheers
>>>
>>> Chris
>>>
>>>
>>>
>>>
>>>
>>> From:
Jakub Hrozek
> <jhrozek at redhat.com>
>>> To:
Christopher
> Lamb/Switzerland/IBM at IBMCH
>>> Cc:
> freeipa-users at redhat.com
>>> Date:
02.06.2015 09:50
>>> Subject:
Re:
> [Freeipa-users] Fw: ssh problem with
>> migrated
>>> FreeIPA
>>> client on EL7.1 -->Not Solved
>>>
>>>
>>>
>>> On Tue, Jun 02, 2015 at 09:43:48AM +0200, Christopher Lamb wrote:
>>>> Hi Jakub
>>>>
>>>> The same user / password works with all our FreeIPA hosts - just this
>> one
>>>> box is the problem. So the password should be good. Of course a type
is
>>>> always possible (especially for strong passwords), but I have tried
> many
>>>> times which should eliminate the odd password typo. The user /
password
>>>> should also be good for both the old and the new FreeIPA Server.
>>>
>>> Interesting, can you add debug_level=10 to the domain section of
>>> sssd.conf? Then krb5_child.log should show Kerberos tracing info
>>> including which exact KDC SSSD was talking to.
>>>
>>>>
>>>> As I can neither log in direct, or via ssh to this box with my FreeIPA
>>>> user, I assume Kinit with my user won't work- i will try later in the
>>> day.
>>>
>>> Well, login as a UNIX user (root) should work..
>>>
>>>>
>>>> My working assumption is that the problem is related in some way to
the
>>>> fact the host originally was a FreeIPA 3.3.3 client, updated to
FreeIPA
>>>> 4.1, and switched between 2 FreeIPA servers. I am currently setting up
> 2
>>>> throwaway EL 7.1 VMs to better test this. On one I will first install
>>>> 3.3.3, then upgrade to 4.1. The second will have a direct install of
> 4.1
>>>> client.
>>>>
>>>> Cheers
>>>>
>>>> Chris
>>>>
>>>>
>>>>
>>>> From:
> Jakub Hrozek
>> <jhrozek at redhat.com>
>>>> To:
>> freeipa-users at redhat.com
>>>> Date:
> 02.06.2015 09:22
>>>> Subject:
> Re:
>> [Freeipa-users] Fw: ssh problem with
>>> migrated
>>> FreeIPA
>>>> client on EL7.1 -->Not Solved
>>>> Sent by:
>> freeipa-users-bounces at redhat.com
>>>>
>>>>
>>>>
>>>> On Mon, Jun 01, 2015 at 07:35:11PM +0200, Christopher Lamb wrote:
>>>>>
>>>>> Hi All
>>>>>
>>>>> Bad news.
>>>>>
>>>>> Over the weekend I was able to get the original problem EL7.1 /
> FreeIPA
>>>> 4.1
>>>>> host (FreeIPA client) to authenticate FreeiPA users (my test being
ssh
>>>>> remote login with FreeIPA user and password).
>>>>>
>>>>> Today I tried a second machine, and had the same problem, ssh
>>> connections
>>>>> with FreeIPA user cause "[sssd[krb5_child[3445]]]: Decrypt integrity
>>>> check
>>>>> failed"
>>>>
>>>> This really just means wrong password, can you kinit as that user
using
>>>> the same password?
>>>>
>>>>>
>>>>> Ahh I thought, I have a solution for that: just remove ipa-client and
>>>>> reinstall via yum, register with the new FreeIPA server ....
>>>>>
>>>>> Only with this second machine I still can't ssh in with a FreeIPA
> user.
>>>>> Argg.....
>>>>>
>>>>> b.t.w, as this machine is a real physical server, I was able to try
>>>> logging
>>>>> in direct with my FreeIPA user --> "Authentication Failure"
>>>>>
>>>>> I now have
>>>>> * a whole bunch of EL6.5 / FreeIPA 3.3.3 hosts that migrated from the
>>> old
>>>>> FreeIPA server to the new without a hitch (i.e. they successfully
>>>>> authenticate FreeIPA users.)
>>>>> * one migrated EL7.1 / FreeIPA 4.1 host that I was able to migrate,
> but
>>>>> with problems
>>>>> * one migrated EL7.1 / FreeIPA 4.1 host that so far defies all
> attempts
>>>> to
>>>>> authenticate with a FreeIPA user
>>>>> * one EL7.1 / FreeIPA 4.1 host that was only ever registered with the
>>> new
>>>>> FreeIPA server, and successfully authenticates FreeIPA users.
>>>>>
>>>>> Any ideas?
>>>>>
>>>>> Chris
>>>>>
>>>>>
>>>>> ----- Forwarded by Christopher Lamb/Switzerland/IBM on 01.06.2015
> 19:17
>>>>> -----
>>>>>
>>>>> From:
>>
Christopher
>>> Lamb/Switzerland/IBM at IBMCH
>>>>> To:
>>
Alexander Bokovoy
>>> <abokovoy at redhat.com>,
>>>>> freeipa-users at redhat.com
>>>>> Date:
>>
30.05.2015 18:52
>>>>> Subject:
>>
Re:
>>> [Freeipa-users] ssh problem with
>>> migrated FreeIPA
>>>> client on
>>>>> EL7.1 --> Solved
>>>>> Sent by:
>>> freeipa-users-bounces at redhat.com
>>>>>
>>>>>
>>>>>
>>>>> Hi All
>>>>>
>>>>> It gives me pleasure to report the problem is solved - a minute ago I
>>> was
>>>>> able to login via ssh with my FreeIPA user to the problem server,
> while
>>>>> sitting on my terrace with a glass of wine!
>>>>>
>>>>> Thanks to Alexander for his helpful advice - we had some mail
exchange
>>>>> outside the user list as I did not wish to broadcast content of keys,
>>>>> config files etc.
>>>>>
>>>>> Regardless of what I did with commands like klist, kvno everything
>>> seemed
>>>>> "ok", but I still could not ssh in. Even a ipa-getkeytab did not
help.
>>>>>
>>>>> Therefore I decided to opt for brute force and (partial) ignorance. I
>>>>> completely uninstalled the FreeIPA client, and then reinstalled,
>>>> configured
>>>>> - ét voilà I could ssh in!
>>>>>
>>>>> This leaves the enigma: what caused the problem? I suspect the
>>> following:
>>>>>
>>>>> The host is an EL 7.1, but the first FreeIPA client installed was
>>> version
>>>>> 3.3.3 (installed as set of standard packages that we bung on all our
>>>>> servers).
>>>>>
>>>>> This worked fine to authenticate against our "old" 3.x FreeIPA
server,
>>>> but
>>>>> did not work against the "new" 4.1 FreeIPA Server.
>>>>>
>>>>> When I realised I could not ssh in, one of the first things I did was
>>> to
>>>>> yum update the FreeIPA client from 3.3.3 to 4.1 - but that did not
>>> help.
>>>>> The solution was to yum remove the FreeIPA client, then yum install
> the
>>>> 4.1
>>>>> client.
>>>>>
>>>>> I have some more EL 7.1 servers with the FreeIPA 3.3.3 client
>>> installed,
>>>> so
>>>>> it will be interesting to see it the problem can be reproduced.
>>>>>
>>>>> Keep up the good work,
>>>>>
>>>>> Chris
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> From:
>>>
> Alexander Bokovoy
>>> <abokovoy at redhat.com>
>>>>> To:
>>>
> Christopher
>>> Lamb/Switzerland/IBM at IBMCH
>>>>> Cc:
>>> freeipa-users at redhat.com
>>>>> Date:
>>>
> 29.05.2015 18:04
>>>>> Subject:
>>>
>
Re:
>>> [Freeipa-users] ssh problem with
>>>> migrated FreeIPA
>>>>> client on
>>>>> EL7.1
>>>>>
>>>>>
>>>>>
>>>>> On Fri, 29 May 2015, Christopher Lamb wrote:
>>>>>>
>>>>>> Hi All
>>>>>>
>>>>>> Some weeks ago I setup a new FreeIPA 4.1.0 on an OEL 7.1 server to
>>>> replace
>>>>>> the existing FreeIPA 3.0.0 running on OEL 6.5, and successfully
>>> migrated
>>>>>> across the users.
>>>>>>
>>>>>> We have 50 odd Servers that are FreeIPA clients. Today I started
>>>> migrating
>>>>>> these one-by-one from the old FreeIPA 3.x server to the new FreeIPA
4
>>>>>> server by doing an ipa-client-install --uninstall from the old, and
>>>>>> ipa-client-install to register with the new 4.1.0 server.
>>>>>>
>>>>>> Most of the FreeIPA clients are running OEL 6.5, and for these the
>>>>>> migration process above worked perfectly. After migrating the
server,
>>> I
>>>>>> could ssh in with my FreeIPA user.
>>>>>>
>>>>>> Then I migrated an OEL 7.1 server. The migration itself seemed to
>>> work,
>>>>> and
>>>>>> getent passwd was successful for my FreeIPA user. However when I try
>>> and
>>>>>> ssh in, my FreeIPA user / password is not accepted.
>>>>>>
>>>>>> Before the migration I could ssh into the problem server (though
>>>> evidently
>>>>>> it was using my FreeIPA user from the old FreeIPA server).
>>>>>>
>>>>>> I can ssh in with a local (non ldap) user, so ssh is running and
>>>> working.
>>>>>>
>>>>>> >From user root I can successfully su to my FreeIPA user.
>>>>>>
>>>>>> Further investigation showed that version of ipa-client installed
was
>>>>>> 3.3.3, so I yum updated this to 4.1.0.
>>>>>>
>>>>>> However I still cannot ssh into the OEL 7.1 box with my FreeIPA
user.
>>>> The
>>>>>> same user continues to work for the 6.5 boxes.
>>>>>>
>>>>>> A colleague tried to ssh in with his FreeIPA user, and was also
>>>> rejected,
>>>>>> so the problem is not my user, but is probably for all FreeIPA
users.
>>>>>>
>>>>>> A failed ssh login attempt causes the following error
>>>> in /var/log/messages
>>>>>>
>>>>>> [sssd[krb5_child[5393]]]: Decrypt integrity check failed
>>>>> It means /etc/krb5.keytab contains keys from older system and SSSD
>>>>> picks them up.
>>>>> Can you show output of 'klist -kKet'?
>>>>> --
>>>>> / Alexander Bokovoy
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Manage your subscription for the Freeipa-users mailing list:
>>>>> https://www.redhat.com/mailman/listinfo/freeipa-users
>>>>> Go to http://freeipa.org for more info on the project
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Manage your subscription for the Freeipa-users mailing list:
>>>>> https://www.redhat.com/mailman/listinfo/freeipa-users
>>>>> Go to http://freeipa.org for more info on the project
>>>>
>>>> --
>>>> Manage your subscription for the Freeipa-users mailing list:
>>>> https://www.redhat.com/mailman/listinfo/freeipa-users
>>>> Go to http://freeipa.org for more info on the project
>>>>
>>>>
>>>>
>>>>
>>>
>>>
>>>
>>>
>>>
>>> --
>>> Manage your subscription for the Freeipa-users mailing list:
>>> https://www.redhat.com/mailman/listinfo/freeipa-users
>>> Go to http://freeipa.org for more info on the project
>>>
>>>
>>>
>>>
>>
>>
>>
>>
>
>
>
>
More information about the Freeipa-users
mailing list