[Freeipa-users] ipactl start fails for no apparent reason
Dmitri Pal
dpal at redhat.com
Wed Apr 1 12:06:44 UTC 2015
On 04/01/2015 07:52 AM, Traiano Welcome wrote:
> Hi Dmitri
>
>
> On Wed, Apr 1, 2015 at 2:23 PM, Dmitri Pal <dpal at redhat.com> wrote:
>> On 04/01/2015 04:14 AM, Traiano Welcome wrote:
>>> Hi Martin
>>>
>>> Thanks for the response. Check results inline:
>>>
>>>
>>> On Wed, Apr 1, 2015 at 10:37 AM, Martin Babinsky <mbabinsk at redhat.com>
>>> wrote:
>>>> On 04/01/2015 09:20 AM, Traiano Welcome wrote:
>>>>> Some information from the dirsrv error log (sanitized: XYZ = realm):
>>>>>
>>>>> [01/Apr/2015:11:01:49 +0300] - 389-Directory/1.3.1.6 B2014.160.2139
>>>>> starting up
>>>>> [01/Apr/2015:11:01:49 +0300] schema-compat-plugin - warning: no
>>>>> entries set up under cn=computers, cn=compat,dc=idm,dc=local
>>>>> [01/Apr/2015:11:01:49 +0300] - Skipping CoS Definition cn=Password
>>>>> Policy,cn=accounts,dc=idm,dc=local--no CoS Templates found, which
>>>>> should be added before the CoS Definition.
>>>>> [01/Apr/2015:11:01:49 +0300] NSMMReplicationPlugin - CleanAllRUV Task:
>>>>> cleanAllRUV task found, resuming the cleaning of rid(6)...
>>>>> [01/Apr/2015:11:01:49 +0300] - Skipping CoS Definition cn=Password
>>>>> Policy,cn=accounts,dc=idm,dc=local--no CoS Templates found, which
>>>>> should be added before the CoS Definition.
>>>>> [01/Apr/2015:11:01:49 +0300] - slapd started. Listening on All
>>>>> Interfaces port 389 for LDAP requests
>>>>> [01/Apr/2015:11:01:49 +0300] - Listening on All Interfaces port 636
>>>>> for LDAPS requests
>>>>> [01/Apr/2015:11:01:49 +0300] - Listening on
>>>>> /var/run/slapd-IDM-LOCAL.socket for LDAPI requests
>>>>> [01/Apr/2015:11:01:49 +0300] set_krb5_creds - Could not get initial
>>>>> credentials for principal [ldap/kwtpr-idm-mstr@] in keytab
>>>>> [FILE:/etc/dirsrv/ds.keytab]: -1765328203 (Key table entry not found)
>>>>> [01/Apr/2015:11:01:49 +0300] set_krb5_creds - Could not get initial
>>>>> credentials for principal [ldap/kwtpr-idm-mstr@] in keytab
>>>>> [FILE:/etc/dirsrv/ds.keytab]: -1765328203 (Key table entry not found)
>>>>> [01/Apr/2015:11:01:49 +0300] set_krb5_creds - Could not get initial
>>>>> credentials for principal [ldap/kwtpr-idm-mstr@] in keytab
>>>>> [FILE:/etc/dirsrv/ds.keytab]: -1765328203 (Key table entry not found)
>>>>> [01/Apr/2015:11:01:49 +0300] set_krb5_creds - Could not get initial
>>>>> credentials for principal [ldap/kwtpr-idm-mstr@] in keytab
>>>>> [FILE:/etc/dirsrv/ds.keytab]: -1765328203 (Key table entry not found)
>>>>> [01/Apr/2015:11:01:49 +0300] set_krb5_creds - Could not get initial
>>>>> credentials for principal [ldap/kwtpr-idm-mstr@] in keytab
>>>>> [FILE:/etc/dirsrv/ds.keytab]: -1765328203 (Key table entry not found)
>>>>> [01/Apr/2015:11:01:49 +0300] slapd_ldap_sasl_interactive_bind - Error:
>>>>> could not perform interactive bind for id [] mech [GSSAPI]: LDAP error
>>>>> -2 (Local error) (SASL(-1): generic failure: GSSAPI Error: Unspecified
>>>>> GSS failure. Minor code may provide more information (No Kerberos
>>>>> credentials available)) errno 0 (Success)
>>>>> [01/Apr/2015:11:01:49 +0300] slapi_ldap_bind - Error: could not
>>>>> perform interactive bind for id [] authentication mechanism [GSSAPI]:
>>>>> error -2 (Local error)
>>>>> [01/Apr/2015:11:01:49 +0300] NSMMReplicationPlugin -
>>>>> agmt="cn=meTokwtard-idm-slve.idm.local" (kwtard-idm-slve:389):
>>>>> Replication bind with GSSAPI auth failed: LDAP error -2 (Local error)
>>>>> (SASL(-1): generic failure: GSSAPI Error: Unspecified GSS failure.
>>>>> Minor code may provide more information (No Kerberos credentials
>>>>> available))
>>>>> [01/Apr/2015:11:01:49 +0300] slapd_ldap_sasl_interactive_bind - Error:
>>>>> could not perform interactive bind for id [] mech [GSSAPI]: LDAP error
>>>>> -2 (Local error) (SASL(-1): generic failure: GSSAPI Error: Unspecified
>>>>> GSS failure. Minor code may provide more information (No Kerberos
>>>>> credentials available)) errno 0 (Success)
>>>>> [01/Apr/2015:11:01:49 +0300] slapi_ldap_bind - Error: could not
>>>>> perform interactive bind for id [] authentication mechanism [GSSAPI]:
>>>>> error -2 (Local error)
>>>>> [01/Apr/2015:11:01:49 +0300] NSMMReplicationPlugin -
>>>>> agmt="cn=meToindpr-idm-slve.idm.local" (indpr-idm-slve:389):
>>>>> Replication bind with GSSAPI auth failed: LDAP error -2 (Local error)
>>>>> (SASL(-1): generic failure: GSSAPI Error: Unspecified GSS failure.
>>>>> Minor code may provide more information (No Kerberos credentials
>>>>> available))
>>>>> [01/Apr/2015:11:01:50 +0300] - slapd shutting down - signaling operation
>>>>> threads
>>>>> [01/Apr/2015:11:01:50 +0300] - slapd shutting down - waiting for 27
>>>>> threads to terminate
>>>>> [01/Apr/2015:11:01:50 +0300] - slapd shutting down - closing down
>>>>> internal subsystems and plugins
>>>>> [01/Apr/2015:11:01:58 +0300] NSMMReplicationPlugin - CleanAllRUV Task:
>>>>> Cleaning rid (6)...
>>>>> [01/Apr/2015:11:01:58 +0300] NSMMReplicationPlugin - CleanAllRUV Task:
>>>>> Waiting to process all the updates from the deleted replica...
>>>>> [01/Apr/2015:11:01:58 +0300] NSMMReplicationPlugin - CleanAllRUV Task:
>>>>> Waiting for all the replicas to be online...
>>>>> [01/Apr/2015:11:01:58 +0300] NSMMReplicationPlugin - CleanAllRUV Task:
>>>>> Server shutting down. Process will resume at server startup
>>>>> [01/Apr/2015:11:02:09 +0300] slapd_ldap_sasl_interactive_bind - Error:
>>>>> could not perform interactive bind for id [] mech [GSSAPI]: LDAP error
>>>>> -1 (Can't contact LDAP server) ((null)) errno 110 (Connection timed
>>>>> out)
>>>>> [01/Apr/2015:11:02:09 +0300] slapi_ldap_bind - Error: could not
>>>>> perform interactive bind for id [] authentication mechanism [GSSAPI]:
>>>>> error -1 (Can't contact LDAP server)
>>>>> [01/Apr/2015:11:02:09 +0300] NSMMReplicationPlugin -
>>>>> agmt="cn=meTokwtospr-idm-slve.idm.local" (kwtospr-idm-slve:389):
>>>>> Replication bind with GSSAPI auth failed: LDAP error -1 (Can't contact
>>>>> LDAP server) ()
>>>>> [01/Apr/2015:11:02:09 +0300] slapd_ldap_sasl_interactive_bind - Error:
>>>>> could not perform interactive bind for id [] mech [GSSAPI]: LDAP error
>>>>> -2 (Local error) (SASL(-1): generic failure: GSSAPI Error: Unspecified
>>>>> GSS failure. Minor code may provide more information (No Kerberos
>>>>> credentials available)) errno 0 (Success)
>>>>> [01/Apr/2015:11:02:09 +0300] slapi_ldap_bind - Error: could not
>>>>> perform interactive bind for id [] authentication mechanism [GSSAPI]:
>>>>> error -2 (Local error)
>>>>> [01/Apr/2015:11:02:09 +0300] NSMMReplicationPlugin -
>>>>> agmt="cn=meTokwtpr-idm-slve.idm.local" (kwtpr-idm-slve:389):
>>>>> Replication bind with GSSAPI auth failed: LDAP error -2 (Local error)
>>>>> (SASL(-1): generic failure: GSSAPI Error: Unspecified GSS failure.
>>>>> Minor code may provide more information (No Kerberos credentials
>>>>> available))
>>>>> errors
>>>>> [01/Apr/2015:11:02:09 +0300] slapd_ldap_sasl_interactive_bind - Error:
>>>>> could not perform interactive bind for id [] mech [GSSAPI]: LDAP error
>>>>> -2 (Local error) (SASL(-1): generic failure: GSSAPI Error: Unspecified
>>>>> GSS failure. Minor code may provide more information (No Kerberos
>>>>> credentials available)) errno 0 (Success)
>>>>> [01/Apr/2015:11:02:09 +0300] slapi_ldap_bind - Error: could not
>>>>> perform interactive bind for id [] authentication mechanism [GSSAPI]:
>>>>> error -2 (Local error)
>>>>> [01/Apr/2015:11:02:09 +0300] NSMMReplicationPlugin -
>>>>> agmt="cn=meToukpr-idm-slve.idm.local" (ukpr-idm-slve:389): Replication
>>>>> bind with GSSAPI auth failed: LDAP error -2 (Local error) (SASL(-1):
>>>>> generic failure: GSSAPI Error: Unspecified GSS failure. Minor code
>>>>> may provide more information (No Kerberos credentials available))
>>>>> [01/Apr/2015:11:02:09 +0300] - Waiting for 4 database threads to stop
>>>>> [01/Apr/2015:11:02:10 +0300] - All database threads now stopped
>>>>> [01/Apr/2015:11:02:10 +0300] - slapd stopped.
>>>>> [01/Apr/2015:10:15:39 +0300] - 389-Directory/1.3.1.6 B2014.160.2139
>>>>> starting up
>>>>> [01/Apr/2015:10:15:39 +0300] schema-compat-plugin - warning: no
>>>>> entries set up under cn=computers, cn=compat,dc=idm,dc=local
>>>>> [01/Apr/2015:10:15:39 +0300] - Skipping CoS Definition cn=Password
>>>>> Policy,cn=accounts,dc=idm,dc=local--no CoS Templates found, which
>>>>> should be added before the CoS Definition.
>>>>> [01/Apr/2015:10:15:39 +0300] NSMMReplicationPlugin - CleanAllRUV Task:
>>>>> cleanAllRUV task found, resuming the cleaning of rid(6)...
>>>>> [01/Apr/2015:10:15:39 +0300] set_krb5_creds - Could not get initial
>>>>> credentials for principal [ldap/kwtpr-idm-mstr@] in keytab
>>>>> [FILE:/etc/dirsrv/ds.keytab]: -1765328203 (Key table entry not found)
>>>>> [01/Apr/2015:10:15:39 +0300] set_krb5_creds - Could not get initial
>>>>> credentials for principal [ldap/kwtpr-idm-mstr@] in keytab
>>>>> [FILE:/etc/dirsrv/ds.keytab]: -1765328203 (Key table entry not found)
>>>>> [01/Apr/2015:10:15:39 +0300] - Skipping CoS Definition cn=Password
>>>>> Policy,cn=accounts,dc=idm,dc=local--no CoS Templates found, which
>>>>> should be added before the CoS Definition.
>>>>> [01/Apr/2015:10:15:39 +0300] set_krb5_creds - Could not get initial
>>>>> credentials for principal [ldap/kwtpr-idm-mstr@] in keytab
>>>>> [FILE:/etc/dirsrv/ds.keytab]: -1765328203 (Key table entry not found)
>>>>> [01/Apr/2015:10:15:39 +0300] slapd_ldap_sasl_interactive_bind - Error:
>>>>> could not perform interactive bind for id [] mech [GSSAPI]: LDAP error
>>>>> -2 (Local error) (SASL(-1): generic failure: GSSAPI Error: Unspecified
>>>>> GSS failure. Minor code may provide more information (No Kerberos
>>>>> credentials available)) errno 2 (No such file or directory)
>>>>> [01/Apr/2015:10:15:39 +0300] slapi_ldap_bind - Error: could not
>>>>> perform interactive bind for id [] authentication mechanism [GSSAPI]:
>>>>> error -2 (Local error)
>>>>> [01/Apr/2015:10:15:39 +0300] set_krb5_creds - Could not get initial
>>>>> credentials for principal [ldap/kwtpr-idm-mstr@] in keytab
>>>>> [FILE:/etc/dirsrv/ds.keytab]: -1765328203 (Key table entry not found)
>>>>> [01/Apr/2015:10:15:39 +0300] csngen_new_csn - Warning: too much time
>>>>> skew (-2771 secs). Current seqnum=3
>>>>> [01/Apr/2015:10:15:39 +0300] NSMMReplicationPlugin -
>>>>> agmt="cn=meTokwtard-idm-slve.idm.local" (kwtard-idm-slve:389):
>>>>> Replication bind with GSSAPI auth failed: LDAP error -2 (Local error)
>>>>> (SASL(-1): generic failure: GSSAPI Error: Unspecified GSS failure.
>>>>> Minor code may provide more information (No Kerberos credentials
>>>>> available))
>>>>> [01/Apr/2015:10:15:39 +0300] set_krb5_creds - Could not get initial
>>>>> credentials for principal [ldap/kwtpr-idm-mstr@] in keytab
>>>>> [FILE:/etc/dirsrv/ds.keytab]: -1765328203 (Key table entry not found)
>>>>> [01/Apr/2015:10:15:39 +0300] csngen_new_csn - Warning: too much time
>>>>> skew (-2770 secs). Current seqnum=1
>>>>> [01/Apr/2015:10:15:39 +0300] - slapd started. Listening on All
>>>>> Interfaces port 389 for LDAP requests
>>>>> [01/Apr/2015:10:15:39 +0300] - Listening on All Interfaces port 636
>>>>> for LDAPS requests
>>>>> [01/Apr/2015:10:15:39 +0300] - Listening on
>>>>> /var/run/slapd-IDM-LOCAL.socket for LDAPI requests
>>>>> [01/Apr/2015:10:15:39 +0300] slapd_ldap_sasl_interactive_bind - Error:
>>>>> could not perform interactive bind for id [] mech [GSSAPI]: LDAP error
>>>>> -2 (Local error) (SASL(-1): generic failure: GSSAPI Error: Unspecified
>>>>> GSS failure. Minor code may provide more information (No Kerberos
>>>>> credentials available)) errno 0 (Success)
>>>>> [01/Apr/2015:10:15:39 +0300] slapi_ldap_bind - Error: could not
>>>>> perform interactive bind for id [] authentication mechanism [GSSAPI]:
>>>>> error -2 (Local error)
>>>>> [01/Apr/2015:10:15:39 +0300] NSMMReplicationPlugin -
>>>>> agmt="cn=meToindpr-idm-slve.idm.local" (indpr-idm-slve:389):
>>>>> Replication bind with GSSAPI auth failed: LDAP error -2 (Local error)
>>>>> (SASL(-1): generic failure: GSSAPI Error: Unspecified GSS failure.
>>>>> Minor code may provide more information (No Kerberos credentials
>>>>> available))
>>>>> [01/Apr/2015:10:15:40 +0300] csngen_new_csn - Warning: too much time
>>>>> skew (-2771 secs). Current seqnum=1
>>>>> [01/Apr/2015:10:15:41 +0300] - slapd shutting down - signaling operation
>>>>> threads
>>>>> [01/Apr/2015:10:15:41 +0300] - slapd shutting down - waiting for 28
>>>>> threads to terminate
>>>>> [01/Apr/2015:10:15:41 +0300] - slapd shutting down - closing down
>>>>> internal subsystems and plugins
>>>>> [01/Apr/2015:10:15:48 +0300] NSMMReplicationPlugin - CleanAllRUV Task:
>>>>> Cleaning rid (6)...
>>>>> [01/Apr/2015:10:15:48 +0300] NSMMReplicationPlugin - CleanAllRUV Task:
>>>>> Waiting to process all the updates from the deleted replica...
>>>>> [01/Apr/2015:10:15:48 +0300] NSMMReplicationPlugin - CleanAllRUV Task:
>>>>> Waiting for all the replicas to be online...
>>>>> [01/Apr/2015:10:15:48 +0300] NSMMReplicationPlugin - CleanAllRUV Task:
>>>>> Server shutting down. Process will resume at server startup
>>>>> [01/Apr/2015:10:15:58 +0300] slapd_ldap_sasl_interactive_bind - Error:
>>>>> could not perform interactive bind for id [] mech [GSSAPI]: LDAP error
>>>>> -1 (Can't contact LDAP server) ((null)) errno 110 (Connection timed
>>>>> out)
>>>>> [01/Apr/2015:10:15:58 +0300] slapi_ldap_bind - Error: could not
>>>>> perform interactive bind for id [] authentication mechanism [GSSAPI]:
>>>>> error -1 (Can't contact LDAP server)
>>>>> [01/Apr/2015:10:15:58 +0300] NSMMReplicationPlugin -
>>>>> agmt="cn=meTokwtospr-idm-slve.idm.local" (kwtospr-idm-slve:389):
>>>>> Replication bind with GSSAPI auth failed: LDAP error -1 (Can't contact
>>>>> LDAP server) ()
>>>>> [01/Apr/2015:10:15:58 +0300] slapd_ldap_sasl_interactive_bind - Error:
>>>>> could not perform interactive bind for id [] mech [GSSAPI]: LDAP error
>>>>> -2 (Local error) (SASL(-1): generic failure: GSSAPI Error: Unspecified
>>>>> GSS failure. Minor code may provide more information (No Kerberos
>>>>> credentials available)) errno 0 (Success)
>>>>> [01/Apr/2015:10:15:58 +0300] slapi_ldap_bind - Error: could not
>>>>> perform interactive bind for id [] authentication mechanism [GSSAPI]:
>>>>> error -2 (Local error)
>>>>> [01/Apr/2015:10:15:58 +0300] NSMMReplicationPlugin -
>>>>> agmt="cn=meTokwtpr-idm-slve.idm.local" (kwtpr-idm-slve:389):
>>>>> Replication bind with GSSAPI auth failed: LDAP error -2 (Local error)
>>>>> (SASL(-1): generic failure: GSSAPI Error: Unspecified GSS failure.
>>>>> Minor code may provide more information (No Kerberos credentials
>>>>> available))
>>>>> [01/Apr/2015:10:15:59 +0300] slapd_ldap_sasl_interactive_bind - Error:
>>>>> could not perform interactive bind for id [] mech [GSSAPI]: LDAP error
>>>>> -2 (Local error) (SASL(-1): generic failure: GSSAPI Error: Unspecified
>>>>> GSS failure. Minor code may provide more information (No Kerberos
>>>>> credentials available)) errno 0 (Success)
>>>>> [01/Apr/2015:10:15:59 +0300] slapi_ldap_bind - Error: could not
>>>>> perform interactive bind for id [] authentication mechanism [GSSAPI]:
>>>>> error -2 (Local error)
>>>>> [01/Apr/2015:10:15:59 +0300] NSMMReplicationPlugin -
>>>>> agmt="cn=meToukpr-idm-slve.idm.local" (ukpr-idm-slve:389): Replication
>>>>> bind with GSSAPI auth failed: LDAP error -2 (Local error) (SASL(-1):
>>>>> generic failure: GSSAPI Error: Unspecified GSS failure. Minor code
>>>>> may provide more information (No Kerberos credentials available))
>>>>> [01/Apr/2015:10:15:59 +0300] - Waiting for 4 database threads to stop
>>>>> [01/Apr/2015:10:16:00 +0300] - All database threads now stopped
>>>>> [01/Apr/2015:10:16:00 +0300] - slapd stopped.
>>>>>
>>>>> On Wed, Apr 1, 2015 at 9:56 AM, Traiano Welcome <traiano at gmail.com>
>>>>> wrote:
>>>>>> Hi List
>>>>>>
>>>>>> I've just tried to restart my IPA services after recently adding a new
>>>>>> replica (0 configuration changes on the IPA server otherwise!), but
>>>>>> ipactl fails when starting up named:
>>>>>>
>>>>>> ---
>>>>>> [root at lolpr-xyz-mstr slapd-XYZ-LOCAL]# ipactl start
>>>>>> Starting Directory Service
>>>>>> Starting krb5kdc Service
>>>>>> Starting kadmin Service
>>>>>> Starting named Service
>>>>>> Job for named.service failed. See 'systemctl status named.service' and
>>>>>> 'journalctl -xn' for details.
>>>>>> Failed to start named Service
>>>>>> Shutting down
>>>>>> Aborting ipactl
>>>>>> ---
>>>>>>
>>>>>> I then manual start named service and try again, but then smb service
>>>>>> fails:
>>>>>>
>>>>>> ---
>>>>>> [root at lolpr-xyz-mstr ~]# ipactl start
>>>>>> Existing service file detected!
>>>>>> Assuming stale, cleaning and proceeding
>>>>>> Starting Directory Service
>>>>>> Starting krb5kdc Service
>>>>>> Starting kadmin Service
>>>>>> Starting named Service
>>>>>> Starting ipa_memcached Service
>>>>>> Starting httpd Service
>>>>>> Starting pki-tomcatd Service
>>>>>> Starting smb Service
>>>>>> Job for smb.service failed. See 'systemctl status smb.service' and
>>>>>> 'journalctl -xn' for details.
>>>>>> Failed to start smb Service
>>>>>> Shutting down
>>>>>> Aborting ipactl
>>>>>> ---
>>>>>>
>>>>>> systemctl status shows the following output for smb.service:
>>>>>>
>>>>>> ---
>>>>>> [root at lolpr-xyz-mstr ~]# systemctl -l status smb.service
>>>>>> smb.service - Samba SMB Daemon
>>>>>> Loaded: loaded (/usr/lib/systemd/system/smb.service; disabled)
>>>>>> Active: failed (Result: exit-code) since Wed 2015-04-01 09:21:10
>>>>>> AST; 1min 14s ago
>>>>>> Process: 4662 ExecStart=/usr/sbin/smbd $SMBDOPTIONS (code=exited,
>>>>>> status=1/FAILURE)
>>>>>> Main PID: 4662 (code=exited, status=1/FAILURE)
>>>>>> Status: "Starting process..."
>>>>>> CGroup: /system.slice/smb.service
>>>>>>
>>>>>> Apr 01 09:21:09 lolpr-xyz-mstr.xyz.local smbd[4662]: GSSAPI client step
>>>>>> 1
>>>>>> Apr 01 09:21:09 lolpr-xyz-mstr.xyz.local smbd[4662]: GSSAPI Error:
>>>>>> Unspecified GSS failure. Minor code may provide more information
>>>>>> (Server ldap/lolpr-xyz-mstr at XYZ.LOCAL not found in Kerberos database)
>>>>>> Apr 01 09:21:10 lolpr-xyz-mstr.xyz.local smbd[4662]: [2015/04/01
>>>>>> 09:21:10.211028, 0] ipa_sam.c:4440(pdb_init_ipasam)
>>>>>> Apr 01 09:21:10 lolpr-xyz-mstr.xyz.local smbd[4662]: Failed to get base
>>>>>> DN.
>>>>>> Apr 01 09:21:10 lolpr-xyz-mstr.xyz.local smbd[4662]: [2015/04/01
>>>>>> 09:21:10.211210, 0]
>>>>>> ../source3/passdb/pdb_interface.c:178(make_pdb_method_name)
>>>>>> Apr 01 09:21:10 lolpr-xyz-mstr.xyz.local smbd[4662]: pdb backend
>>>>>> ipasam:ldapi://%2fvar%2frun%2fslapd-XYZ-LOCAL.socket did not correctly
>>>>>> init (error was NT_STATUS_UNSUCCESSFUL)
>>>>>> Apr 01 09:21:10 lolpr-xyz-mstr.xyz.local systemd[1]: smb.service: main
>>>>>> process exited, code=exited, status=1/FAILURE
>>>>>> Apr 01 09:21:10 lolpr-xyz-mstr.xyz.local systemd[1]: Failed to start
>>>>>> Samba SMB Daemon.
>>>>>> Apr 01 09:21:10 lolpr-xyz-mstr.xyz.local systemd[1]: Unit smb.service
>>>>>> entered failed state.
>>>>>> Apr 01 09:21:12 lolpr-xyz-mstr.xyz.local systemd[1]: Stopped Samba SMB
>>>>>> Daemon.
>>>>>> ---
>>>>>>
>>>>>>
>>>>>> I manually try to start the smb service as follows, but can't (Of
>>>>>> course the directory service is not up, so there's a little catch22
>>>>>> there and this many not mean much):
>>>>>>
>>>>>>
>>>>>> ---
>>>>>>
>>>>>> [root at lolpr-xyz-mstr slapd-XYZ-LOCAL]# systemctl status smb.service
>>>>>> smb.service - Samba SMB Daemon
>>>>>> Loaded: loaded (/usr/lib/systemd/system/smb.service; disabled)
>>>>>> Active: failed (Result: exit-code) since Wed 2015-04-01 09:50:38
>>>>>> AST;
>>>>>> 57s ago
>>>>>> Process: 8089 ExecStart=/usr/sbin/smbd $SMBDOPTIONS (code=exited,
>>>>>> status=1/FAILURE)
>>>>>> Main PID: 8089 (code=exited, status=1/FAILURE)
>>>>>> Status: "Starting process..."
>>>>>>
>>>>>> Apr 01 09:50:36 lolpr-xyz-mstr.xyz.local smbd[8089]: kerberos error:
>>>>>> code=-1765328228, message=Cannot contact any KDC for realm 'XYZ.LOCAL'
>>>>>> Apr 01 09:50:37 lolpr-xyz-mstr.xyz.local smbd[8089]: [2015/04/01
>>>>>> 09:50:37.573772, 0] ipa_sam.c:4128(bind_callback_cleanup)
>>>>>> Apr 01 09:50:37 lolpr-xyz-mstr.xyz.local smbd[8089]: kerberos error:
>>>>>> code=-1765328228, message=Cannot contact any KDC for realm 'XYZ.LOCAL'
>>>>>> Apr 01 09:50:38 lolpr-xyz-mstr.xyz.local smbd[8089]: [2015/04/01
>>>>>> 09:50:38.574722, 0] ipa_sam.c:4440(pdb_init_ipasam)
>>>>>> Apr 01 09:50:38 lolpr-xyz-mstr.xyz.local smbd[8089]: Failed to get base
>>>>>> DN.
>>>>>> Apr 01 09:50:38 lolpr-xyz-mstr.xyz.local smbd[8089]: [2015/04/01
>>>>>> 09:50:38.574903, 0]
>>>>>> ../source3/passdb/pdb_interface.c:178(make_pdb_method_name)
>>>>>> Apr 01 09:50:38 lolpr-xyz-mstr.xyz.local smbd[8089]: pdb backend
>>>>>> ipasam:ldapi://%2fvar%2frun%2fslapd-XYZ-LOCAL.socket did not correctly
>>>>>> init (error was NT_STATUS_UNSUCCESSFUL)
>>>>>> Apr 01 09:50:38 lolpr-xyz-mstr.xyz.local systemd[1]: smb.service: main
>>>>>> process exited, code=exited, status=1/FAILURE
>>>>>> Apr 01 09:50:38 lolpr-xyz-mstr.xyz.local systemd[1]: Failed to start
>>>>>> Samba SMB Daemon.
>>>>>> Apr 01 09:50:38 lolpr-xyz-mstr.xyz.local systemd[1]: Unit smb.service
>>>>>> entered failed state.
>>>>>> [root at lolpr-xyz-mstr slapd-XYZ-LOCAL]#
>>>>>>
>>>>>> ---
>>>>>>
>>>>>> Please could someone advise me on how to drill deeper into debugging
>>>>>> this issue to get ipactl to start ?
>>>>>>
>>>>>> NOTES:
>>>>>>
>>>>>> - This server is successfully in a Trust relationship with
>>>>>> ActiveDirectory.
>>>>>> - There are a number of replicas established which have been working
>>>>>> fine til this morning
>>>>>> - Another replica was added around the time of the failure using the
>>>>>> same steps as usual (not sure how this could be related)
>>>>>>
>>>>>>
>>>>>> Many thanks in advance,
>>>>>> Traiano
>>>>>
>>>> Hi Traiano,
>>>>
>>>> it seems like there is some problem with Kerberos keytab for DS service.
>>>>
>>>> Take a look at this guide:
>>>>
>>>> http://www.freeipa.org/page/Troubleshooting#Service_does_not_start
>>>>
>>>> and check whether there is something wrong with DS keytab and that the
>>>> service principal is set up correctly.
>>>>
>>>
>>> Walking through this pedantically:
>>>
>>> Service does not start:
>>>
>>> 1) See service log of the respective service for the exact error text.
>>> For example, the Directory Server stores the log in
>>> /var/log/dirsrv/slapd-REALM-NAME/errors
>>>
>>> check
>>>
>>> 2) Make sure that the server the service is running on has a fully
>>> qualified domain name
>>>
>>> ---
>>> [root at lolpr-xyz-mstr ~]# hostname
>>> lolpr-xyz-mstr.xyz.local
>>> [root at lolpr-xyz-mstr ~]# host `hostname`
>>> lolpr-xyz-mstr.xyz.local has address 172.16.100.68
>>> [root at lolpr-xyz-mstr ~]# host 172.16.100.68
>>> 68.100.16.172.in-addr.arpa domain name pointer lolpr-xyz-mstr.xyz.local.
>>> [root at lolpr-xyz-mstr ~]#
>>> ---
>>>
>>> 3) See what keys are in the keytab used for authentication of the service,
>>> e.g.:
>>> # klist -kt /etc/dirsrv/ds.keytab
>>>
>>>
>>> ---
>>> [root at lolpr-xyz-mstr slapd-XYZ-LOCAL]# klist -kt /etc/dirsrv/ds.keytab
>>> Keytab name: FILE:/etc/dirsrv/ds.keytab
>>> KVNO Timestamp Principal
>>> ---- -------------------
>>> ------------------------------------------------------
>>> 2 11/06/2014 13:13:06 ldap/lolpr-xyz-mstr.xyz.local at XYZ.LOCAL
>>> 2 11/06/2014 13:13:06 ldap/lolpr-xyz-mstr.xyz.local at XYZ.LOCAL
>>> 2 11/06/2014 13:13:06 ldap/lolpr-xyz-mstr.xyz.local at XYZ.LOCAL
>>> 2 11/06/2014 13:13:06 ldap/lolpr-xyz-mstr.xyz.local at XYZ.LOCAL
>>> ---
>>>
>>> 4) Make sure that the stored principals match the system FQDN system name
>>>
>>> check:
>>>
>>> ---
>>> [root at lolpr-xyz-mstr ~]# host lolpr-xyz-mstr.xyz.local
>>> lolpr-xyz-mstr.xyz.local has address 172.16.100.68
>>> [root at lolpr-xyz-mstr ~]#
>>> ---
>>>
>>> 5) Make sure that the version of the keys (KVNO) stored in the keytab
>>> and in the FreeIPA server match:
>>> $ kvno ldap/ipa.example.com at EXAMPLE.COM
>>>
>>>
>>> check ... This is unusual:
>>>
>>> ---
>>> [root at lolpr-xyz-mstr ~]# kvno ldap/lolpr-xyz-mstr.xyz.local at XYZ.LOCAL
>>> kvno: Credentials cache keyring 'persistent:0:0' not found while
>>> getting client principal name
>>> ---
>>>
>>> Now, when I look at my krb5.conf, I see the file has had a recent
>>> change ... yet, I'm sure this file was never edited: Does the
>>> krb5.conf below look correct for a standard IPA primary server?:
>>>
>>> ---
>>> [root at lolpr-xyz-mstr ~]# ls -l /etc/krb5.conf
>>> -rw-r--r-- 1 root root 811 Apr 1 11:01 /etc/krb5.conf
>>> ---
>>>
>>>
>>> ---
>>> [root at lolpr-xyz-mstr ~]# cat /etc/krb5.conf
>>> includedir /var/lib/sss/pubconf/krb5.include.d/
>>>
>>> [logging]
>>> default = FILE:/var/log/krb5libs.log
>>> kdc = FILE:/var/log/krb5kdc.log
>>> admin_server = FILE:/var/log/kadmind.log
>>>
>>> [libdefaults]
>>> default_realm = XYZ.LOCAL
>>> dns_lookup_realm = false
>>> dns_lookup_kdc = true
>>> rdns = false
>>> ticket_lifetime = 24h
>>> forwardable = yes
>>> default_ccache_name = KEYRING:persistent:%{uid}
>>>
>>> [realms]
>>> XYZ.LOCAL = {
>>> kdc = lolpr-xyz-mstr.xyz.local:88
>>> master_kdc = lolpr-xyz-mstr.xyz.local:88
>>> admin_server = lolpr-xyz-mstr.xyz.local:749
>>> default_domain = xyz.local
>>> pkinit_anchors = FILE:/etc/ipa/ca.crt
>>> auth_to_local =
>>> RULE:[1:$1@$0](^.*@WINDOM.LOCAL$)s/@WINDOM.LOCAL/@windom.local/
>>> auth_to_local = DEFAULT
>>> }
>>>
>>> [domain_realm]
>>> .xyz.local = XYZ.LOCAL
>>> xyz.local = XYZ.LOCAL
>>>
>>> [dbmodules]
>>> XYZ.LOCAL = {
>>> db_library = ipadb.so
>>> }
>>> ---
>>
>>
>> I do not see any glaring problems in this file.
>> This seems to be 4.1 bits.
>
> IPA 3.3 on CentOS release 7.0.1406 (Core)
>
>
>> There is definitely something wrong with the Kerberos part though.
>> And the fact that you can't access credential cache is pointing to a
>> problem.
> Yes. Trying to start the krb5kdc service manually:
>
>
> ---
> job for krb5kdc.service failed. See 'systemctl status krb5kdc.service'
> and 'journalctl -xn' for details.
> ---
>
> Checking the krb5kdc.service status:
>
> ---
> [root at lolpr-xyz-mstr log]# systemctl status krb5kdc.service
> krb5kdc.service - Kerberos 5 KDC
> Loaded: loaded (/usr/lib/systemd/system/krb5kdc.service; disabled)
> Active: failed (Result: exit-code) since Wed 2015-04-01 14:42:15 AST; 7s ago
> Process: 3884 ExecStart=/usr/sbin/krb5kdc -P /var/run/krb5kdc.pid
> $KRB5KDC_ARGS (code=exited, status=1/FAILURE)
>
> Apr 01 14:42:15 lolpr-xyz-mstr.xyz.local systemd[1]: Starting Kerberos 5 KDC...
> Apr 01 14:42:15 lolpr-xyz-mstr.xyz.local krb5kdc[3884]: krb5kdc:
> cannot initialize realm XYZ.LOCAL - see log file for details
> Apr 01 14:42:15 lolpr-xyz-mstr.xyz.local systemd[1]: krb5kdc.service:
> control process exited, code=exited status=1
> Apr 01 14:42:15 lolpr-xyz-mstr.xyz.local systemd[1]: Failed to start
> Kerberos 5 KDC.
> Apr 01 14:42:15 lolpr-xyz-mstr.xyz.local systemd[1]: Unit
> krb5kdc.service entered failed state.
> ---
>
>
> Checking the logs:
>
> ---
> [root at lolpr-xyz-mstr log]# cat krb5kdc.log
> krb5kdc: Server error - while fetching master key K/M for realm XYZ.LOCAL
> ---
>
>
>
>> Do you see any selinux denials?
> Selinux has been disabled for months. I see this is still so in
> selinux conf: SELINUX=disabled
>
>
>
>> If the file was touched may be it was touched by recent update or
>> installation of some other package on the system.
>> The update/install might have set wrong context on the cred cache causing
>> problems like this.
> I've been careful to disable all external repos on the system since
> installation, so I'm only using packages on the original installation
> iso. It's a hermetically sealed system from the package point of view:
>
> [root at lolpr-xyz-mstr yum.repos.d]# ls -l
> total 4
> -rw-r--r--. 1 root root 133 Nov 5 19:06 CentOS-Local.repo
> [root at lolpr-xyz-mstr yum.repos.d]#
> [root at lolpr-xyz-mstr yum.repos.d]#
> [root at lolpr-xyz-mstr yum.repos.d]# cat CentOS-Local.repo
> [LocalRepo]
> name=Local Repository
> baseurl=file:///repo
> enabled=1
> gpgcheck=1
> gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-7
> [root at lolpr-xyz-mstr yum.repos.d]#
>
>
>> Anything interesting in the KDC log?
>>
>
> This looks like a clue:
>
> krb5kdc: Server error - while fetching master key K/M for realm XYZ.LOCAL
>
> ... But I'm not sure how to interpret this usefully ...
This means that DS has not started as master key is in DS.
Can you check the DS server logs?
>
>
>
>>> 6) Make sure that there are no DNS Issues and both forward and reverse
>>> DNS records of the are OK and match the system name and the stored
>>> principal keys
>>>
>>> check. DNS works.
>>>
>>> 7) Make sure that the system time difference on the host and FreeIPA
>>> server is not greater than 5 minutes
>>>
>>> They're one and the same in this case.
>>>
>>>> --
>>>> Martin^3 Babinsky
>>> Thanks,
>>> Traiano
>>>
>>
>> --
>> Thank you,
>> Dmitri Pal
>>
>> Sr. Engineering Manager IdM portfolio
>> Red Hat, Inc.
>>
>> --
>> Manage your subscription for the Freeipa-users mailing list:
>> https://www.redhat.com/mailman/listinfo/freeipa-users
>> Go to http://freeipa.org for more info on the project
--
Thank you,
Dmitri Pal
Sr. Engineering Manager IdM portfolio
Red Hat, Inc.
More information about the Freeipa-users
mailing list