[Freeipa-users] Fw: ssh problem with migrated FreeIPA client on EL7.1 -->Solved
Alexander Bokovoy
abokovoy at redhat.com
Fri Jun 5 14:30:12 UTC 2015
On Fri, 05 Jun 2015, Christopher Lamb wrote:
>Hi Martin
>
>Thanks for updating the documenation!
>
>The suggested solution works not only my test servers, but also "in the
>real world". This morning I migrated the last production server (ipa host)
>to the new FreeIPA KDC.
>
>Just out of idle curiosity, why is the rm -f /var/lib/sss/db/* step
>required on our EL 7.1 + ipa-client 4.1 boxes, but not on our older EL 6.5
>+ ipa-client 3.3.3 machines?
>
>Is the problem down to sssd? (on the EL 6.5 machines we are running sssd
>1.9.2, while on EL 7.1 we have sssd 1.12.2
I think there are more object types supported by newer SSSD versions
which aren't invalidated like users or groups.
>
>Cheers
>
>Chris
>
>
>
>From: Martin Kosek <mkosek at redhat.com>
>To: Christopher Lamb/Switzerland/IBM at IBMCH, Rob Crittenden
> <rcritten at redhat.com>, freeipa-users at redhat.com
>Cc: Jakub Hrozek <jhrozek at redhat.com>
>Date: 05.06.2015 08:06
>Subject: Re: [Freeipa-users] Fw: ssh problem with migrated FreeIPA
> client on EL7.1 -->Solved
>
>
>
>On 06/04/2015 07:34 PM, Christopher Lamb wrote:
>> Hi All
>>
>> I can now report back success (at least on my throwaway EL7.1 test VM).
>>
>> To switch an EL 7.1 + ipa-client 4.1 host from an old FreeIPA 3.3.3 KDC
>to
>> a new FreeIPA 4.1 KDC 3 steps are required:
>>
>> 1) ipa-client-install --uninstall
>>
>> 2) rm -f /var/lib/sss/db/*
>>
>> 3) ipa-client-install --server ldap.my.example.com --domain
>my.example.com
>> -N
>>
>> Having done this, my free-ipa user successfully authenticates (e.g. ssh
>> remote login with free-ipa user / password
>>
>>
>> To switch EL 6.5 + ipa-client 3.3.3 hosts step 2) was not required.
>>
>> Kudos and thanks go to Rob C for suggesting step 2. (Note that the
>> directory to be purged is /var/lib/sss/db/, not /var/lib/sssd/db/ as
>> suggested earlier in this thread.
>
>Cool! Thanks for reaching back. I added this advice to the FreeIPA
>Troubleshooting guide too:
>
>http://www.freeipa.org/page/Troubleshooting#Cannot_authenticate_on_client
>
>>
>> Cheers
>>
>> Chris
>>
>>
>>
>>
>> From: Martin Kosek <mkosek at redhat.com>
>> To: Christopher Lamb/Switzerland/IBM at IBMCH,
>> freeipa-users at redhat.com
>> Cc: Jakub Hrozek <jhrozek at redhat.com>, Rob Crittenden
>> <rcritten at redhat.com>
>> Date: 03.06.2015 10:39
>> Subject: Re: [Freeipa-users] Fw: ssh problem with migrated
>FreeIPA
>> client on EL7.1 -->Not Solved
>>
>>
>>
>> On 06/03/2015 10:30 AM, Christopher Lamb wrote:
>>> Hi all
>>>
>>> This is a quick(ish) note to bring everybody up to speed on this issue.
>>> Yesterday we had some private mail exchange on this issue as I did not
>> wish
>>> to broadcast the krb5 and ipa install logs to the user list.
>>>
>>> The basic situation is that we are in the process of migrating from an
>>> FreeIPA 3.3.3 Server (KDC) to a new FreeIPA 4.1 Server (KDC). As
>> discussed
>>> in a thread some weeks ago we did not do this by replicating (as perhaps
>> we
>>> should have done). Instead we migrated the users across.
>>>
>>> We have 30+ servers that are IPA clients ("Hosts" in ipa-speak) joined
>to
>>> the old KDC. We are now in the process of migrating these hosts to the
>> new
>>> 4.1 KDC.
>>>
>>> Most of the hosts run EL 6.5 + ipa-client 3.3.3. For all of these
>> joining
>>> to the new KDC was trouble free, taking a few minutes each. After
>joining
>>> the new KDC FreeIPA users authenticated properly.
>>>
>>> We also had a small number of new EL 7.1 + ipa-client 4.1 hosts that
>were
>>> joined direct to the new 4.1 KDC, never having been joined of the 3.3.3
>>> KDC. These were also trouble free.
>>>
>>> The problem occurs with a handful of existing EL 7.1 +ipa-client 4.1
>> hosts
>>> that were originally joined to the 3.3.3 KDC, and must be moved to join
>> the
>>> 4.1 KDC. These machines no longer authenticate valid FreeIPA users. I
>> have
>>> been able to reproduce this behaviour with a freshly setup VM joined
>> first
>>> to the 3.3.3 KDC, then moved to the 4.1 KDC.
>>>
>>> While the errors show in the krb5 child logs indicate that the password
>> is
>>> incorrect, the same user / password is happily accepted by all the other
>>> hosts.
>>>
>>> It seems that in the process of moving / migrating the EL 7.1 /
>> ipa-client
>>> 4.1 from the old KDC to the new KDC, "something" is left behind that
>> causes
>>> problems. We have seen indications in the install logs that the kinit
>> steps
>>> called during ipa-client install are getting responses from the wrong
>> (old)
>>> KDC, and not from the new KDC.
>>>
>>> Frustratingly. over the weekend i managed to get one of the problem EL
>> 7.1
>>> boxes to work. However I can't work out exactly what I was that I did
>> that
>>> did the trick. However it seems that some kind of major de-install /
>>> cleanup + reinstall of the ipa-client may be needed.
>>>
>>> Rob has suggested that as part of such a cleanup I should do "rm
>>> -f /var/lib/sssd/db/*". I will test this later today and report back.
>>>
>>> Thanks to Rob, Jakub, Martin, Alexander et al for their help and
>>> suggestions so far.
>>>
>>> Chris
>>
>> Thanks for the background. The pain you are getting is exactly the reason
>> why
>> migration via replication to RHEL-7.1 is a better choice :-) Please let
>us
>> know
>> the result, I am curious how this works out.
>>
>>>
>>>
>>>
>>>
>>> From: Martin Kosek <mkosek at redhat.com>
>>> To: Christopher Lamb/Switzerland/IBM at IBMCH,
>>> freeipa-users at redhat.com, Jakub Hrozek <jhrozek at redhat.com>
>>> Date: 03.06.2015 09:34
>>> Subject: Re: [Freeipa-users] Fw: ssh problem with
>migrated
>> FreeIPA
>>> client on EL7.1 -->Not Solved
>>>
>>>
>>>
>>> On 06/02/2015 06:15 PM, Christopher Lamb wrote:
>>>>
>>>> Hi
>>>>
>>>> Earlier today I setup 2 throwaway EL7.1 VMs to help narrow down the
>> cause
>>>> of this problem. Let's call them HOST09 and HOST10
>>>>
>>>> Both are mimimum installs of EL7.1, with NTPD installed and configured.
>>>>
>>>> HOST09 had ipa-client 4.1 installed via yum, and was configured to use
>>> our
>>>> new FreeIPA 4.1 server, right from the start. --> My FreeIPA user
>>>> authenticates successfully against this machine.
>>>>
>>>> HOST10 had ipa-client 4.1 installed as a dependency of one of our
>>> standard
>>>> config packages, and was first set to use our old FreeIPA 3.3.3 server.
>>> -->
>>>> My FreeIPA user authenticates successfully. against this machine.
>>>>
>>>> I then de-registered HOST10 from the FreeIPA 3.1 server, and registered
>>>> against the new FreeIPA 4.1 server --> My FreeIPA users does NOT
>>>> authenticate successfully.
>>>>
>>>> This replicates well the behaviour I saw with my production servers,
>>> namely
>>>> a) EL 7.1 hosts with ipa-client 4.1 registered directly against the new
>>> 4.1
>>>> FreeIPA server authenticate properly.
>>>>
>>>> b) EL 7.1 hosts with ipa-client 4.1 first registered against the old
>>> 3.3.3
>>>> FreeIPA server, then reregistered with the new 4.1 FreeIPA server do
>NOT
>>>> authenticate properly
>>>>
>>>> Chris
>>>
>>> Hello,
>>>
>>> This is really strange. What I do not fully understand is what is the
>>> "registration against a FreeIPA server". What server you install IPA
>> client
>>> should matter if the deployment is set up properly. The host enrollment
>>> entry
>>> should simply replicate to whole infrastructure. The only thing that
>will
>>> probably differ is sssd.conf and krb5.conf as they will have different
>>> primary
>>> server set up, based on what your DNS setup is.
>>>
>>> It rather seems that the "reregistration" is what causes the issue. It
>>> looks
>>> like something cleanup problem during the process. I will let Jakub to
>> help
>>> here, I would suggest including the SSSD logs from the failed login, it
>> may
>>> help.
>>>
>>>>
>>>>
>>>>
>>>> ----- Forwarded by Christopher Lamb/Switzerland/IBM on 02.06.2015 16:52
>>>> -----
>>>>
>>>> From: Christopher
>Lamb/Switzerland/IBM at IBMCH
>>>> To: Jakub Hrozek
><jhrozek at redhat.com>
>>>> Cc:
>freeipa-users at redhat.com
>>>> Date: 02.06.2015 10:40
>>>> Subject: Re:
>[Freeipa-users] Fw: ssh problem with
>> migrated
>>> FreeIPA
>>>> client on EL7.1 -->Not Solved
>>>> Sent by:
>freeipa-users-bounces at redhat.com
>>>>
>>>>
>>>>
>>>> Hi Jakub
>>>>
>>>> Yes root login works, that's how I've been getting into the box.
>>>>
>>>> Surprisingly, kinit with my user seems to work on that box. After
>>> entering
>>>> my password when prompted, it returns to the commandline without error.
>>>>
>>>> However if I try kinit with another FreeIPA user, then instead of
>>> prompting
>>>> for a password, it gives "Generic preauthentication failure while
>> getting
>>>> initial credentials" error.
>>>>
>>>> Having set debug_level=10, when I try and ssh in with my FreeIPA user,
>I
>>>> find errors like
>>>>
>>>> "Retrieving host .... with result: .. Matching credential not found"
>>>>
>>>> "Received error from KDC ... Additional pre-authentication required"
>>>>
>>>> "Received error from KDC... Decrypt integrity check failed"
>>>>
>>>> "Received error code 1432158219"
>>>>
>>>> Cheers
>>>>
>>>> Chris
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> From:
> Jakub Hrozek
>> <jhrozek at redhat.com>
>>>> To:
> Christopher
>> Lamb/Switzerland/IBM at IBMCH
>>>> Cc:
>> freeipa-users at redhat.com
>>>> Date:
> 02.06.2015 09:50
>>>> Subject:
> Re:
>> [Freeipa-users] Fw: ssh problem with
>>> migrated
>>>> FreeIPA
>>>> client on EL7.1 -->Not Solved
>>>>
>>>>
>>>>
>>>> On Tue, Jun 02, 2015 at 09:43:48AM +0200, Christopher Lamb wrote:
>>>>> Hi Jakub
>>>>>
>>>>> The same user / password works with all our FreeIPA hosts - just this
>>> one
>>>>> box is the problem. So the password should be good. Of course a type
>is
>>>>> always possible (especially for strong passwords), but I have tried
>> many
>>>>> times which should eliminate the odd password typo. The user /
>password
>>>>> should also be good for both the old and the new FreeIPA Server.
>>>>
>>>> Interesting, can you add debug_level=10 to the domain section of
>>>> sssd.conf? Then krb5_child.log should show Kerberos tracing info
>>>> including which exact KDC SSSD was talking to.
>>>>
>>>>>
>>>>> As I can neither log in direct, or via ssh to this box with my FreeIPA
>>>>> user, I assume Kinit with my user won't work- i will try later in the
>>>> day.
>>>>
>>>> Well, login as a UNIX user (root) should work..
>>>>
>>>>>
>>>>> My working assumption is that the problem is related in some way to
>the
>>>>> fact the host originally was a FreeIPA 3.3.3 client, updated to
>FreeIPA
>>>>> 4.1, and switched between 2 FreeIPA servers. I am currently setting up
>> 2
>>>>> throwaway EL 7.1 VMs to better test this. On one I will first install
>>>>> 3.3.3, then upgrade to 4.1. The second will have a direct install of
>> 4.1
>>>>> client.
>>>>>
>>>>> Cheers
>>>>>
>>>>> Chris
>>>>>
>>>>>
>>>>>
>>>>> From:
>> Jakub Hrozek
>>> <jhrozek at redhat.com>
>>>>> To:
>>> freeipa-users at redhat.com
>>>>> Date:
>> 02.06.2015 09:22
>>>>> Subject:
>> Re:
>>> [Freeipa-users] Fw: ssh problem with
>>>> migrated
>>>> FreeIPA
>>>>> client on EL7.1 -->Not Solved
>>>>> Sent by:
>>> freeipa-users-bounces at redhat.com
>>>>>
>>>>>
>>>>>
>>>>> On Mon, Jun 01, 2015 at 07:35:11PM +0200, Christopher Lamb wrote:
>>>>>>
>>>>>> Hi All
>>>>>>
>>>>>> Bad news.
>>>>>>
>>>>>> Over the weekend I was able to get the original problem EL7.1 /
>> FreeIPA
>>>>> 4.1
>>>>>> host (FreeIPA client) to authenticate FreeiPA users (my test being
>ssh
>>>>>> remote login with FreeIPA user and password).
>>>>>>
>>>>>> Today I tried a second machine, and had the same problem, ssh
>>>> connections
>>>>>> with FreeIPA user cause "[sssd[krb5_child[3445]]]: Decrypt integrity
>>>>> check
>>>>>> failed"
>>>>>
>>>>> This really just means wrong password, can you kinit as that user
>using
>>>>> the same password?
>>>>>
>>>>>>
>>>>>> Ahh I thought, I have a solution for that: just remove ipa-client and
>>>>>> reinstall via yum, register with the new FreeIPA server ....
>>>>>>
>>>>>> Only with this second machine I still can't ssh in with a FreeIPA
>> user.
>>>>>> Argg.....
>>>>>>
>>>>>> b.t.w, as this machine is a real physical server, I was able to try
>>>>> logging
>>>>>> in direct with my FreeIPA user --> "Authentication Failure"
>>>>>>
>>>>>> I now have
>>>>>> * a whole bunch of EL6.5 / FreeIPA 3.3.3 hosts that migrated from the
>>>> old
>>>>>> FreeIPA server to the new without a hitch (i.e. they successfully
>>>>>> authenticate FreeIPA users.)
>>>>>> * one migrated EL7.1 / FreeIPA 4.1 host that I was able to migrate,
>> but
>>>>>> with problems
>>>>>> * one migrated EL7.1 / FreeIPA 4.1 host that so far defies all
>> attempts
>>>>> to
>>>>>> authenticate with a FreeIPA user
>>>>>> * one EL7.1 / FreeIPA 4.1 host that was only ever registered with the
>>>> new
>>>>>> FreeIPA server, and successfully authenticates FreeIPA users.
>>>>>>
>>>>>> Any ideas?
>>>>>>
>>>>>> Chris
>>>>>>
>>>>>>
>>>>>> ----- Forwarded by Christopher Lamb/Switzerland/IBM on 01.06.2015
>> 19:17
>>>>>> -----
>>>>>>
>>>>>> From:
>>>
> Christopher
>>>> Lamb/Switzerland/IBM at IBMCH
>>>>>> To:
>>>
> Alexander Bokovoy
>>>> <abokovoy at redhat.com>,
>>>>>> freeipa-users at redhat.com
>>>>>> Date:
>>>
> 30.05.2015 18:52
>>>>>> Subject:
>>>
> Re:
>>>> [Freeipa-users] ssh problem with
>>>> migrated FreeIPA
>>>>> client on
>>>>>> EL7.1 --> Solved
>>>>>> Sent by:
>>>> freeipa-users-bounces at redhat.com
>>>>>>
>>>>>>
>>>>>>
>>>>>> Hi All
>>>>>>
>>>>>> It gives me pleasure to report the problem is solved - a minute ago I
>>>> was
>>>>>> able to login via ssh with my FreeIPA user to the problem server,
>> while
>>>>>> sitting on my terrace with a glass of wine!
>>>>>>
>>>>>> Thanks to Alexander for his helpful advice - we had some mail
>exchange
>>>>>> outside the user list as I did not wish to broadcast content of keys,
>>>>>> config files etc.
>>>>>>
>>>>>> Regardless of what I did with commands like klist, kvno everything
>>>> seemed
>>>>>> "ok", but I still could not ssh in. Even a ipa-getkeytab did not
>help.
>>>>>>
>>>>>> Therefore I decided to opt for brute force and (partial) ignorance. I
>>>>>> completely uninstalled the FreeIPA client, and then reinstalled,
>>>>> configured
>>>>>> - ét voilà I could ssh in!
>>>>>>
>>>>>> This leaves the enigma: what caused the problem? I suspect the
>>>> following:
>>>>>>
>>>>>> The host is an EL 7.1, but the first FreeIPA client installed was
>>>> version
>>>>>> 3.3.3 (installed as set of standard packages that we bung on all our
>>>>>> servers).
>>>>>>
>>>>>> This worked fine to authenticate against our "old" 3.x FreeIPA
>server,
>>>>> but
>>>>>> did not work against the "new" 4.1 FreeIPA Server.
>>>>>>
>>>>>> When I realised I could not ssh in, one of the first things I did was
>>>> to
>>>>>> yum update the FreeIPA client from 3.3.3 to 4.1 - but that did not
>>>> help.
>>>>>> The solution was to yum remove the FreeIPA client, then yum install
>> the
>>>>> 4.1
>>>>>> client.
>>>>>>
>>>>>> I have some more EL 7.1 servers with the FreeIPA 3.3.3 client
>>>> installed,
>>>>> so
>>>>>> it will be interesting to see it the problem can be reproduced.
>>>>>>
>>>>>> Keep up the good work,
>>>>>>
>>>>>> Chris
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> From:
>>>>
>> Alexander Bokovoy
>>>> <abokovoy at redhat.com>
>>>>>> To:
>>>>
>> Christopher
>>>> Lamb/Switzerland/IBM at IBMCH
>>>>>> Cc:
>>>> freeipa-users at redhat.com
>>>>>> Date:
>>>>
>> 29.05.2015 18:04
>>>>>> Subject:
>>>>
>>
> Re:
>>>> [Freeipa-users] ssh problem with
>>>>> migrated FreeIPA
>>>>>> client on
>>>>>> EL7.1
>>>>>>
>>>>>>
>>>>>>
>>>>>> On Fri, 29 May 2015, Christopher Lamb wrote:
>>>>>>>
>>>>>>> Hi All
>>>>>>>
>>>>>>> Some weeks ago I setup a new FreeIPA 4.1.0 on an OEL 7.1 server to
>>>>> replace
>>>>>>> the existing FreeIPA 3.0.0 running on OEL 6.5, and successfully
>>>> migrated
>>>>>>> across the users.
>>>>>>>
>>>>>>> We have 50 odd Servers that are FreeIPA clients. Today I started
>>>>> migrating
>>>>>>> these one-by-one from the old FreeIPA 3.x server to the new FreeIPA
>4
>>>>>>> server by doing an ipa-client-install --uninstall from the old, and
>>>>>>> ipa-client-install to register with the new 4.1.0 server.
>>>>>>>
>>>>>>> Most of the FreeIPA clients are running OEL 6.5, and for these the
>>>>>>> migration process above worked perfectly. After migrating the
>server,
>>>> I
>>>>>>> could ssh in with my FreeIPA user.
>>>>>>>
>>>>>>> Then I migrated an OEL 7.1 server. The migration itself seemed to
>>>> work,
>>>>>> and
>>>>>>> getent passwd was successful for my FreeIPA user. However when I try
>>>> and
>>>>>>> ssh in, my FreeIPA user / password is not accepted.
>>>>>>>
>>>>>>> Before the migration I could ssh into the problem server (though
>>>>> evidently
>>>>>>> it was using my FreeIPA user from the old FreeIPA server).
>>>>>>>
>>>>>>> I can ssh in with a local (non ldap) user, so ssh is running and
>>>>> working.
>>>>>>>
>>>>>>> >From user root I can successfully su to my FreeIPA user.
>>>>>>>
>>>>>>> Further investigation showed that version of ipa-client installed
>was
>>>>>>> 3.3.3, so I yum updated this to 4.1.0.
>>>>>>>
>>>>>>> However I still cannot ssh into the OEL 7.1 box with my FreeIPA
>user.
>>>>> The
>>>>>>> same user continues to work for the 6.5 boxes.
>>>>>>>
>>>>>>> A colleague tried to ssh in with his FreeIPA user, and was also
>>>>> rejected,
>>>>>>> so the problem is not my user, but is probably for all FreeIPA
>users.
>>>>>>>
>>>>>>> A failed ssh login attempt causes the following error
>>>>> in /var/log/messages
>>>>>>>
>>>>>>> [sssd[krb5_child[5393]]]: Decrypt integrity check failed
>>>>>> It means /etc/krb5.keytab contains keys from older system and SSSD
>>>>>> picks them up.
>>>>>> Can you show output of 'klist -kKet'?
>>>>>> --
>>>>>> / Alexander Bokovoy
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> Manage your subscription for the Freeipa-users mailing list:
>>>>>> https://www.redhat.com/mailman/listinfo/freeipa-users
>>>>>> Go to http://freeipa.org for more info on the project
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> Manage your subscription for the Freeipa-users mailing list:
>>>>>> https://www.redhat.com/mailman/listinfo/freeipa-users
>>>>>> Go to http://freeipa.org for more info on the project
>>>>>
>>>>> --
>>>>> Manage your subscription for the Freeipa-users mailing list:
>>>>> https://www.redhat.com/mailman/listinfo/freeipa-users
>>>>> Go to http://freeipa.org for more info on the project
>>>>>
>>>>>
>>>>>
>>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> Manage your subscription for the Freeipa-users mailing list:
>>>> https://www.redhat.com/mailman/listinfo/freeipa-users
>>>> Go to http://freeipa.org for more info on the project
>>>>
>>>>
>>>>
>>>>
>>>
>>>
>>>
>>>
>>
>>
>>
>>
>
>
>
>
>
>--
>Manage your subscription for the Freeipa-users mailing list:
>https://www.redhat.com/mailman/listinfo/freeipa-users
>Go to http://freeipa.org for more info on the project
--
/ Alexander Bokovoy
More information about the Freeipa-users
mailing list