[Freeipa-users] Recovering from data-only backup doesn't recover Kerberos keys properly

Wed Feb 24 22:28:58 UTC 2016

> Are you just toying with this or did something go horribly wrong and
you're trying to restore a production environment?

This. :-(

I have actually rebuilt the environment from scratch, then wrote a
perl script that just recreated all users from the ldif using ipa
user-add and reset password for everyone.

After the fresh install the following command was used for each user:
ipa user-add --first='John' --last='Doe' --uid=1603600001
--gid=1603600001 --email='john.doe at contoso.com' --sshpubkey='ssh-rsa
<keyhere>' --random john.doe

I had to force uids/gids, so that users don't lose access to their home folders.

I have regenerated keytabs on all client hosts, but now there is some
weird behavior is demonstrated by sssd: users intermittently fail to
login. This is a log from a client machine (Amazon Linux 2015.09):

(Wed Feb 24 22:08:49 2016) [sssd[ssh]] [accept_fd_handler] (0x0400):
Client connected!
(Wed Feb 24 22:08:49 2016) [sssd[ssh]] [sss_cmd_get_version] (0x0200):
Received client version [0].
(Wed Feb 24 22:08:49 2016) [sssd[ssh]] [sss_cmd_get_version] (0x0200):
Offered version [0].
(Wed Feb 24 22:08:49 2016) [sssd[ssh]] [ssh_cmd_parse_request]
(0x0400): Requested domain [<ALL>]
(Wed Feb 24 22:08:49 2016) [sssd[ssh]] [ssh_cmd_parse_request]
(0x0400): Parsing name [marat.vyshegorodtsev][<ALL>]
(Wed Feb 24 22:08:49 2016) [sssd[ssh]] [sss_parse_name_for_domains]
(0x0200): name 'marat.vyshegorodtsev' matched without domain, user is
marat.vyshegorodtsev
(Wed Feb 24 22:08:49 2016) [sssd[ssh]] [sss_ssh_cmd_get_user_pubkeys]
(0x0400): Requesting SSH user public keys for [marat.vyshegorodtsev]
from [<ALL>]
(Wed Feb 24 22:08:49 2016) [sssd[ssh]] [sss_dp_issue_request]
(0x0400): Issuing request for
[0x40b2d0:1:marat.vyshegorodtsev at contoso.com]
(Wed Feb 24 22:08:49 2016) [sssd[ssh]] [sss_dp_get_account_msg]
(0x0400): Creating request for
[contoso.com][1][1][name=marat.vyshegorodtsev]
(Wed Feb 24 22:08:49 2016) [sssd[ssh]] [sbus_add_timeout] (0x2000): 0xb99c10
(Wed Feb 24 22:08:49 2016) [sssd[ssh]] [sss_dp_internal_get_send]
(0x0400): Entering request
[0x40b2d0:1:marat.vyshegorodtsev at contoso.com]
(Wed Feb 24 22:08:49 2016) [sssd[ssh]] [sbus_remove_timeout] (0x2000): 0xb99c10
(Wed Feb 24 22:08:49 2016) [sssd[ssh]] [sss_dp_get_reply] (0x1000):
Got reply from Data Provider - DP error code: 1 errno: 11 error
message: Offline
(Wed Feb 24 22:08:49 2016) [sssd[ssh]]
[ssh_user_pubkeys_search_dp_callback] (0x0040): Unable to get
information from Data Provider Error: 1, 11, Offline
(Wed Feb 24 22:08:49 2016) [sssd[ssh]] [ssh_user_pubkeys_search_next]
(0x0400): Requesting SSH user public keys for
[marat.vyshegorodtsev at contoso.com]
(Wed Feb 24 22:08:49 2016) [sssd[ssh]] [sss_dp_req_destructor]
(0x0400): Deleting request:
[0x40b2d0:1:marat.vyshegorodtsev at contoso.com]
(Wed Feb 24 22:08:49 2016) [sssd[ssh]] [client_recv] (0x0200): Client
disconnected!
(Wed Feb 24 22:08:49 2016) [sssd[ssh]] [client_destructor] (0x2000):
Terminated client [0xb90dd0][17]

This was produced in response to command
/usr/bin/sss_ssh_authorizedkeys marat.vyshegorodtsev

The whole log file has a lot of garbage in it like this:
(Wed Feb 24 22:28:09 2016) [sssd[ssh]] [sbus_get_sender_id_send]
(0x2000): Not a sysbus message, quit
(Wed Feb 24 22:28:19 2016) [sssd[ssh]] [sbus_message_handler]
(0x2000): Received SBUS method org.freedesktop.sssd.service.ping on
path /org/freedesktop/sssd/service

I can do a ldapsearch on the IPA server successfully using
login/password, but sssd just doesn't work.
The server runs Fedora 23 with FreeIPA 4.2.3-2.fc23.

Best regards, Marat

On Wed, Feb 24, 2016 at 6:07 AM, Rob Crittenden <rcritten at redhat.com> wrote:
> David Kupka wrote:
>> On 23/02/16 20:21, Marat Vyshegorodtsev wrote:
>>> Hi!
>>>
>>> I've been doing backups using the tool like this:
>>> ipa-backup --data --online
>>>
>>> I didn't want any configuration to be backed up, since it is managed
>>> from a chef recipe.
>>>
>>> However, when I tried to recover the backup to a fresh FreeIPA
>>> install, Kerberos (GSSAPI) broke — I can't authenticate myself
>>> anywhere using Kerberos: CLI, HTTP, etc.
>>>
>>> LDAP password-based authentication works alright.
>>>
>>> After some googling and reading through the mailing list, I followed
>>> this manual and updated all keytabs for all services — dirsrv, httpd,
>>> kadmin:
>>> http://www.freeipa.org/page/V3/Backup_and_Restore#Backup.2C_uninstall.2C_reinstall.2C_restore_JUST_the_LDAP_server
>>>
>>>
>>> Then it broke  in a different way: for a correct session it says that
>>> my session is expired or just does nothing, for an incorrect password
>>> it responds with "password incorrect" (see screenshot).
>>> https://yadi.sk/i/WVe8u1_ZpNh3w
>>>
>>> For CLI it just says that the credentials are incorrect regardless of
>>> what credentials I provide.
>>>
>>> I suppose that all krbPrincipalKey fields are tied to some other
>>> encryption key that is not included in data-only backup.
>>>
>>> Could you please let me know how to regenerate krbPrincipalKey for all
>>> users or how to work around this issue?
>>>
>>> Best regards,
>>> Marat
>>>
>>
>> Hello Marat,
>> I would say that this is expected. During freeipa-server installation
>> all service and host kerberos keys are generated randomly, stored in
>> Directory Server and in keytab accessible to the host/service.
>> When you reinstall freeipa-server all keys are regenerated and no longer
>> matches the ones stored in your backup.
>>
>> You can use ipa-getkeytab(1) with Directory Manager credentials to
>> retrieve new keys but think it's not enough to make it work again.
>> Hopefully, someone, who understand kerberos better will advice.
>>
>
> It sounds like he already re-generated those keytabs.
>
> The Kerberos master key is stored in LDAP so you should already have it.
> Seeing the KDC and/or httpd logs might be useful.
>
> Are you just toying with this or did something go horribly wrong and
> you're trying to restore a production environment?
>
> The instructions you used were strictly a brain dump, something I goofed
> around with as an interesting thought project but didn't entirely nail
> down. It is quite possible I didn't document some important step in there.
>
> rob