[Freeipa-users] SSSD startup failures on ipa clients
Mark Heslin
mheslin at redhat.com
Mon Jul 28 11:28:22 UTC 2014
Hi Jakub,
I've added the output of 'sssd -i -d4' below:
On 07/28/2014 03:39 AM, Jakub Hrozek wrote:
> On Sun, Jul 27, 2014 at 10:42:34PM -0400, Mark Heslin wrote:
>> Folks,
>>
>> I just stumbled on an odd issue. I have an OpenShift deployment with 2
>> brokers, 2 nodes, 1 rhc client
>> all running RHEL 6.5. I also have 2 IPA servers (1 server, 1 replica), 1 IPA
>> admin (tools) client all running RHEL 7.0.
>> All OpenShift hosts, client and IPA client are members of IPA domain
>> 'interop.example.com'.
>>
>> After creating ssh public keys on the IPA admin client for user 'ose-admin1'
>> and uploading them into IPA,
>> I am able to ssh with the key to all IPA domain hosts as user 'ose-admin1'
>> except the 2 node hosts.
>> In looking closer at the 2 node hosts I noticed that SSSD keeps failing on
>> start:
>>
>> # service sssd restart
>> Stopping sssd: cat: /var/run/sssd.pid: No such file or directory
>> [FAILED]
>> Starting sssd: [FAILED]
>>
>> Starting with debug mode shows:
>>
>> [root at node1/2 ~]# sssd -d9
>> (Sun Jul 27 22:12:29:527689 2014) [sssd] [check_file] (0x0400): lstat for
>> [/var/run/nscd/socket] failed: [2][No such file or directory].
>> (Sun Jul 27 22:12:29:529293 2014) [sssd] [ldb] (0x0400):
>> server_sort:Unable to register control with rootdse!
>> (Sun Jul 27 22:12:29:529596 2014) [sssd] [confdb_get_domain_internal]
>> (0x0400): No enumeration for [interop.example.com]!
>> (Sun Jul 27 22:12:29:529646 2014) [sssd] [confdb_get_domain_internal]
>> (0x1000): pwd_expiration_warning is -1
>> (Sun Jul 27 22:12:29:529686 2014) [sssd] [server_setup] (0x0040): Becoming
>> a daemon.
> At this point sssd became a deamon and detached from the terminal, so no
> more debug info was printed. Can you run sssd again, adding "-i"
> (interactive) this time?
[root at node2 ~]# sssd -i -d4
(Mon Jul 28 07:25:20 2014) [sssd] [get_ping_config] (0x0100): Time
between service pings for [interop.example.com]: [10]
(Mon Jul 28 07:25:20 2014) [sssd] [get_ping_config] (0x0100): Time
between SIGTERM and SIGKILL for [interop.example.com]: [60]
(Mon Jul 28 07:25:20 2014) [sssd] [start_service] (0x0100): Queueing
service interop.example.com for startup
/usr/libexec/sssd/sssd_be: error while loading shared libraries:
libcares.so.2: cannot open shared object file: No such file or directory
(Mon Jul 28 07:25:20 2014) [sssd] [mt_svc_exit_handler] (0x0040): Child
[interop.example.com] exited with code [127]
(Mon Jul 28 07:25:20 2014) [sssd] [get_ping_config] (0x0100): Time
between service pings for [interop.example.com]: [10]
(Mon Jul 28 07:25:20 2014) [sssd] [get_ping_config] (0x0100): Time
between SIGTERM and SIGKILL for [interop.example.com]: [60]
(Mon Jul 28 07:25:20 2014) [sssd] [start_service] (0x0100): Queueing
service interop.example.com for startup
/usr/libexec/sssd/sssd_be: error while loading shared libraries:
libcares.so.2: cannot open shared object file: No such file or directory
(Mon Jul 28 07:25:20 2014) [sssd] [mt_svc_exit_handler] (0x0040): Child
[interop.example.com] exited with code [127]
(Mon Jul 28 07:25:22 2014) [sssd] [get_ping_config] (0x0100): Time
between service pings for [interop.example.com]: [10]
(Mon Jul 28 07:25:22 2014) [sssd] [get_ping_config] (0x0100): Time
between SIGTERM and SIGKILL for [interop.example.com]: [60]
(Mon Jul 28 07:25:22 2014) [sssd] [start_service] (0x0100): Queueing
service interop.example.com for startup
/usr/libexec/sssd/sssd_be: error while loading shared libraries:
libcares.so.2: cannot open shared object file: No such file or directory
(Mon Jul 28 07:25:22 2014) [sssd] [mt_svc_exit_handler] (0x0040): Child
[interop.example.com] exited with code [127]
(Mon Jul 28 07:25:25 2014) [sssd] [services_startup_timeout] (0x0020):
Providers did not start in time, forcing services startup!
(Mon Jul 28 07:25:25 2014) [sssd] [services_startup_timeout] (0x0100):
Now starting services!
(Mon Jul 28 07:25:25 2014) [sssd] [get_ping_config] (0x0100): Time
between service pings for [nss]: [10]
(Mon Jul 28 07:25:25 2014) [sssd] [get_ping_config] (0x0100): Time
between SIGTERM and SIGKILL for [nss]: [60]
(Mon Jul 28 07:25:25 2014) [sssd] [start_service] (0x0100): Queueing
service nss for startup
(Mon Jul 28 07:25:25 2014) [sssd] [get_ping_config] (0x0100): Time
between service pings for [pam]: [10]
(Mon Jul 28 07:25:25 2014) [sssd] [get_ping_config] (0x0100): Time
between SIGTERM and SIGKILL for [pam]: [60]
(Mon Jul 28 07:25:25 2014) [sssd] [start_service] (0x0100): Queueing
service pam for startup
(Mon Jul 28 07:25:25 2014) [sssd] [get_ping_config] (0x0100): Time
between service pings for [ssh]: [10]
(Mon Jul 28 07:25:25 2014) [sssd] [get_ping_config] (0x0100): Time
between SIGTERM and SIGKILL for [ssh]: [60]
(Mon Jul 28 07:25:25 2014) [sssd] [start_service] (0x0100): Queueing
service ssh for startup
(Mon Jul 28 07:25:25 2014) [sssd] [get_ping_config] (0x0100): Time
between service pings for [pac]: [10]
(Mon Jul 28 07:25:25 2014) [sssd] [get_ping_config] (0x0100): Time
between SIGTERM and SIGKILL for [pac]: [60]
(Mon Jul 28 07:25:25 2014) [sssd] [start_service] (0x0100): Queueing
service pac for startup
(Mon Jul 28 07:25:25 2014) [sssd[nss]] [monitor_common_send_id]
(0x0100): Sending ID: (nss,1)
(Mon Jul 28 07:25:25 2014) [sssd[pam]] [monitor_common_send_id]
(0x0100): Sending ID: (pam,1)
(Mon Jul 28 07:25:25 2014) [sssd[pam]] [sss_names_init] (0x0100): Using
re
[(((?P<domain>[^\\]+)\\(?P<name>.+$))|((?P<name>[^@]+)@(?P<domain>.+$))|(^(?P<name>[^@\\]+)$))].
(Mon Jul 28 07:25:25 2014) [sssd[pam]] [sbus_client_init] (0x0020):
check_file failed for
[/var/lib/sss/pipes/private/sbus-dp_interop.example.com].
(Mon Jul 28 07:25:25 2014) [sssd[pam]] [sss_dp_init] (0x0010): Failed to
connect to monitor services.
(Mon Jul 28 07:25:25 2014) [sssd[pam]] [sss_process_init] (0x0010):
fatal error setting up backend connector
(Mon Jul 28 07:25:25 2014) [sssd] [sbus_dispatch] (0x0080): Connection
is not open for dispatching.
(Mon Jul 28 07:25:25 2014) [sssd] [mt_svc_exit_handler] (0x0040): Child
[pam] exited with code [3]
(Mon Jul 28 07:25:25 2014) [sssd] [get_ping_config] (0x0100): Time
between service pings for [pam]: [10]
(Mon Jul 28 07:25:25 2014) [sssd] [get_ping_config] (0x0100): Time
between SIGTERM and SIGKILL for [pam]: [60]
(Mon Jul 28 07:25:25 2014) [sssd] [start_service] (0x0100): Queueing
service pam for startup
(Mon Jul 28 07:25:25 2014) [sssd[nss]] [sss_names_init] (0x0100): Using
re
[(((?P<domain>[^\\]+)\\(?P<name>.+$))|((?P<name>[^@]+)@(?P<domain>.+$))|(^(?P<name>[^@\\]+)$))].
(Mon Jul 28 07:25:25 2014) [sssd[nss]] [sbus_client_init] (0x0020):
check_file failed for
[/var/lib/sss/pipes/private/sbus-dp_interop.example.com].
(Mon Jul 28 07:25:25 2014) [sssd[nss]] [sss_dp_init] (0x0010): Failed to
connect to monitor services.
(Mon Jul 28 07:25:25 2014) [sssd[nss]] [sss_process_init] (0x0010):
fatal error setting up backend connector
(Mon Jul 28 07:25:25 2014) [sssd] [sbus_dispatch] (0x0080): Connection
is not open for dispatching.
(Mon Jul 28 07:25:25 2014) [sssd] [mt_svc_exit_handler] (0x0040): Child
[nss] exited with code [3]
(Mon Jul 28 07:25:25 2014) [sssd] [get_ping_config] (0x0100): Time
between service pings for [nss]: [10]
(Mon Jul 28 07:25:25 2014) [sssd] [get_ping_config] (0x0100): Time
between SIGTERM and SIGKILL for [nss]: [60]
(Mon Jul 28 07:25:25 2014) [sssd] [start_service] (0x0100): Queueing
service nss for startup
(Mon Jul 28 07:25:25 2014) [sssd[pac]] [monitor_common_send_id]
(0x0100): Sending ID: (pac,1)
(Mon Jul 28 07:25:25 2014) [sssd[pac]] [sss_names_init] (0x0100): Using
re
[(((?P<domain>[^\\]+)\\(?P<name>.+$))|((?P<name>[^@]+)@(?P<domain>.+$))|(^(?P<name>[^@\\]+)$))].
(Mon Jul 28 07:25:25 2014) [sssd[pac]] [sbus_client_init] (0x0020):
check_file failed for
[/var/lib/sss/pipes/private/sbus-dp_interop.example.com].
(Mon Jul 28 07:25:25 2014) [sssd[pac]] [sss_dp_init] (0x0010): Failed to
connect to monitor services.
(Mon Jul 28 07:25:25 2014) [sssd[pac]] [sss_process_init] (0x0010):
fatal error setting up backend connector
(Mon Jul 28 07:25:25 2014) [sssd] [mt_svc_exit_handler] (0x0040): Child
[pac] exited with code [3]
(Mon Jul 28 07:25:25 2014) [sssd] [sbus_dispatch] (0x0080): Connection
is not open for dispatching.
(Mon Jul 28 07:25:25 2014) [sssd] [get_ping_config] (0x0100): Time
between service pings for [pac]: [10]
(Mon Jul 28 07:25:25 2014) [sssd] [get_ping_config] (0x0100): Time
between SIGTERM and SIGKILL for [pac]: [60]
(Mon Jul 28 07:25:25 2014) [sssd] [start_service] (0x0100): Queueing
service pac for startup
(Mon Jul 28 07:25:25 2014) [sssd[ssh]] [monitor_common_send_id]
(0x0100): Sending ID: (ssh,1)
(Mon Jul 28 07:25:25 2014) [sssd[ssh]] [sss_names_init] (0x0100): Using
re
[(((?P<domain>[^\\]+)\\(?P<name>.+$))|((?P<name>[^@]+)@(?P<domain>.+$))|(^(?P<name>[^@\\]+)$))].
(Mon Jul 28 07:25:25 2014) [sssd[ssh]] [sbus_client_init] (0x0020):
check_file failed for
[/var/lib/sss/pipes/private/sbus-dp_interop.example.com].
(Mon Jul 28 07:25:25 2014) [sssd[ssh]] [sss_dp_init] (0x0010): Failed to
connect to monitor services.
(Mon Jul 28 07:25:25 2014) [sssd[ssh]] [sss_process_init] (0x0010):
fatal error setting up backend connector
(Mon Jul 28 07:25:25 2014) [sssd] [sbus_dispatch] (0x0080): Connection
is not open for dispatching.
(Mon Jul 28 07:25:25 2014) [sssd] [mt_svc_exit_handler] (0x0040): Child
[ssh] exited with code [3]
(Mon Jul 28 07:25:25 2014) [sssd] [get_ping_config] (0x0100): Time
between service pings for [ssh]: [10]
(Mon Jul 28 07:25:25 2014) [sssd] [get_ping_config] (0x0100): Time
between SIGTERM and SIGKILL for [ssh]: [60]
(Mon Jul 28 07:25:25 2014) [sssd] [start_service] (0x0100): Queueing
service ssh for startup
(Mon Jul 28 07:25:25 2014) [sssd[pam]] [monitor_common_send_id]
(0x0100): Sending ID: (pam,1)
(Mon Jul 28 07:25:25 2014) [sssd[pam]] [sss_names_init] (0x0100): Using
re
[(((?P<domain>[^\\]+)\\(?P<name>.+$))|((?P<name>[^@]+)@(?P<domain>.+$))|(^(?P<name>[^@\\]+)$))].
(Mon Jul 28 07:25:25 2014) [sssd[pam]] [sbus_client_init] (0x0020):
check_file failed for
[/var/lib/sss/pipes/private/sbus-dp_interop.example.com].
(Mon Jul 28 07:25:25 2014) [sssd[pam]] [sss_dp_init] (0x0010): Failed to
connect to monitor services.
(Mon Jul 28 07:25:25 2014) [sssd[pam]] [sss_process_init] (0x0010):
fatal error setting up backend connector
(Mon Jul 28 07:25:25 2014) [sssd] [mt_svc_exit_handler] (0x0040): Child
[pam] exited with code [3]
(Mon Jul 28 07:25:25 2014) [sssd] [sbus_dispatch] (0x0080): Connection
is not open for dispatching.
(Mon Jul 28 07:25:25 2014) [sssd[ssh]] [monitor_common_send_id]
(0x0100): Sending ID: (ssh,1)
(Mon Jul 28 07:25:25 2014) [sssd[ssh]] [sss_names_init] (0x0100): Using
re
[(((?P<domain>[^\\]+)\\(?P<name>.+$))|((?P<name>[^@]+)@(?P<domain>.+$))|(^(?P<name>[^@\\]+)$))].
(Mon Jul 28 07:25:25 2014) [sssd[ssh]] [sbus_client_init] (0x0020):
check_file failed for
[/var/lib/sss/pipes/private/sbus-dp_interop.example.com].
(Mon Jul 28 07:25:25 2014) [sssd[ssh]] [sss_dp_init] (0x0010): Failed to
connect to monitor services.
(Mon Jul 28 07:25:25 2014) [sssd[ssh]] [sss_process_init] (0x0010):
fatal error setting up backend connector
(Mon Jul 28 07:25:25 2014) [sssd] [sbus_dispatch] (0x0080): Connection
is not open for dispatching.
(Mon Jul 28 07:25:25 2014) [sssd] [mt_svc_exit_handler] (0x0040): Child
[ssh] exited with code [3]
(Mon Jul 28 07:25:25 2014) [sssd[pac]] [monitor_common_send_id]
(0x0100): Sending ID: (pac,1)
(Mon Jul 28 07:25:25 2014) [sssd[pac]] [sss_names_init] (0x0100): Using
re
[(((?P<domain>[^\\]+)\\(?P<name>.+$))|((?P<name>[^@]+)@(?P<domain>.+$))|(^(?P<name>[^@\\]+)$))].
(Mon Jul 28 07:25:25 2014) [sssd[pac]] [sbus_client_init] (0x0020):
check_file failed for
[/var/lib/sss/pipes/private/sbus-dp_interop.example.com].
(Mon Jul 28 07:25:25 2014) [sssd[pac]] [sss_dp_init] (0x0010): Failed to
connect to monitor services.
(Mon Jul 28 07:25:25 2014) [sssd[pac]] [sss_process_init] (0x0010):
fatal error setting up backend connector
(Mon Jul 28 07:25:25 2014) [sssd] [sbus_dispatch] (0x0080): Connection
is not open for dispatching.
(Mon Jul 28 07:25:25 2014) [sssd] [mt_svc_exit_handler] (0x0040): Child
[pac] exited with code [3]
(Mon Jul 28 07:25:25 2014) [sssd[nss]] [monitor_common_send_id]
(0x0100): Sending ID: (nss,1)
(Mon Jul 28 07:25:25 2014) [sssd[nss]] [sss_names_init] (0x0100): Using
re
[(((?P<domain>[^\\]+)\\(?P<name>.+$))|((?P<name>[^@]+)@(?P<domain>.+$))|(^(?P<name>[^@\\]+)$))].
(Mon Jul 28 07:25:25 2014) [sssd[nss]] [sbus_client_init] (0x0020):
check_file failed for
[/var/lib/sss/pipes/private/sbus-dp_interop.example.com].
(Mon Jul 28 07:25:25 2014) [sssd[nss]] [sss_dp_init] (0x0010): Failed to
connect to monitor services.
(Mon Jul 28 07:25:25 2014) [sssd[nss]] [sss_process_init] (0x0010):
fatal error setting up backend connector
(Mon Jul 28 07:25:25 2014) [sssd] [mt_svc_exit_handler] (0x0040): Child
[nss] exited with code [3]
(Mon Jul 28 07:25:25 2014) [sssd] [sbus_dispatch] (0x0080): Connection
is not open for dispatching.
(Mon Jul 28 07:25:26 2014) [sssd] [get_ping_config] (0x0100): Time
between service pings for [interop.example.com]: [10]
(Mon Jul 28 07:25:26 2014) [sssd] [get_ping_config] (0x0100): Time
between SIGTERM and SIGKILL for [interop.example.com]: [60]
(Mon Jul 28 07:25:26 2014) [sssd] [start_service] (0x0100): Queueing
service interop.example.com for startup
/usr/libexec/sssd/sssd_be: error while loading shared libraries:
libcares.so.2: cannot open shared object file: No such file or directory
(Mon Jul 28 07:25:26 2014) [sssd] [mt_svc_exit_handler] (0x0040): Child
[interop.example.com] exited with code [127]
(Mon Jul 28 07:25:26 2014) [sssd] [mt_svc_exit_handler] (0x0010):
Process [interop.example.com], definitely stopped!
(Mon Jul 28 07:25:26 2014) [sssd] [monitor_quit] (0x0040): Returned with: 1
(Mon Jul 28 07:25:26 2014) [sssd] [monitor_quit] (0x0020): Terminating
[ssh][10518]
(Mon Jul 28 07:25:26 2014) [sssd] [monitor_quit] (0x0020): Couldn't kill
[ssh][10518]: [No such process]
(Mon Jul 28 07:25:26 2014) [sssd] [monitor_quit] (0x0020): Terminating
[pac][10517]
(Mon Jul 28 07:25:26 2014) [sssd] [monitor_quit] (0x0020): Couldn't kill
[pac][10517]: [No such process]
(Mon Jul 28 07:25:26 2014) [sssd] [monitor_quit] (0x0020): Terminating
[nss][10516]
(Mon Jul 28 07:25:26 2014) [sssd] [monitor_quit] (0x0020): Couldn't kill
[nss][10516]: [No such process]
(Mon Jul 28 07:25:26 2014) [sssd] [monitor_quit] (0x0020): Terminating
[pam][10515]
(Mon Jul 28 07:25:26 2014) [sssd] [monitor_quit] (0x0020): Couldn't kill
[pam][10515]: [No such process]
>> The logs show show nothing useful but this problem started during the
>> ipa-client-install - the log shows:
>>
>> 2014-07-23T18:40:22Z DEBUG args=/usr/sbin/authconfig --enablesssdauth
>> --enablemkhomedir --update --enablesssd
>> 2014-07-23T18:40:22Z DEBUG stdout=Starting oddjobd: [ OK ]
>> 2014-07-23T18:40:22Z DEBUG stderr=
>> 2014-07-23T18:40:22Z INFO SSSD enabled
>> 2014-07-23T18:40:29Z DEBUG args=/sbin/service sssd restart
>> 2014-07-23T18:40:29Z DEBUG stdout=Stopping sssd: [FAILED]
>> Starting sssd: [FAILED]
>>
>> 2014-07-23T18:40:29Z DEBUG stderr=cat: /var/run/sssd.pid: No such file or
>> directory
>>
>> 2014-07-23T18:40:29Z WARNING SSSD service restart was unsuccessful.
>> 2014-07-23T18:40:29Z DEBUG args=/sbin/chkconfig sssd on
>> 2014-07-23T18:40:29Z DEBUG stdout=
>>
>> Any ideas? Have we seen this before? I suppose I could uninstall the ipa
>> client and re-install but I didn't want
>> to touch anything until I hear back.
>>
>> Thanks!
>>
>> -m
>>
>> btw - All systems have been updated as of this evening. Kerberos works fine
>> but anything requiring
>> lookups is toast.
>>
>>
>>
>>
>>
>> --
>> Manage your subscription for the Freeipa-users mailing list:
>> https://www.redhat.com/mailman/listinfo/freeipa-users
>> Go To http://freeipa.org for more info on the project
More information about the Freeipa-users
mailing list