[Freeipa-users] ipactl start fails for no apparent reason

Wed Apr 1 07:20:34 UTC 2015

Some information from the dirsrv error log (sanitized: XYZ = realm):

[01/Apr/2015:11:01:49 +0300] - 389-Directory/1.3.1.6 B2014.160.2139 starting up
[01/Apr/2015:11:01:49 +0300] schema-compat-plugin - warning: no
entries set up under cn=computers, cn=compat,dc=idm,dc=local
[01/Apr/2015:11:01:49 +0300] - Skipping CoS Definition cn=Password
Policy,cn=accounts,dc=idm,dc=local--no CoS Templates found, which
should be added before the CoS Definition.
[01/Apr/2015:11:01:49 +0300] NSMMReplicationPlugin - CleanAllRUV Task:
cleanAllRUV task found, resuming the cleaning of rid(6)...
[01/Apr/2015:11:01:49 +0300] - Skipping CoS Definition cn=Password
Policy,cn=accounts,dc=idm,dc=local--no CoS Templates found, which
should be added before the CoS Definition.
[01/Apr/2015:11:01:49 +0300] - slapd started.  Listening on All
Interfaces port 389 for LDAP requests
[01/Apr/2015:11:01:49 +0300] - Listening on All Interfaces port 636
for LDAPS requests
[01/Apr/2015:11:01:49 +0300] - Listening on
/var/run/slapd-IDM-LOCAL.socket for LDAPI requests
[01/Apr/2015:11:01:49 +0300] set_krb5_creds - Could not get initial
credentials for principal [ldap/kwtpr-idm-mstr@] in keytab
[FILE:/etc/dirsrv/ds.keytab]: -1765328203 (Key table entry not found)
[01/Apr/2015:11:01:49 +0300] set_krb5_creds - Could not get initial
credentials for principal [ldap/kwtpr-idm-mstr@] in keytab
[FILE:/etc/dirsrv/ds.keytab]: -1765328203 (Key table entry not found)
[01/Apr/2015:11:01:49 +0300] set_krb5_creds - Could not get initial
credentials for principal [ldap/kwtpr-idm-mstr@] in keytab
[FILE:/etc/dirsrv/ds.keytab]: -1765328203 (Key table entry not found)
[01/Apr/2015:11:01:49 +0300] set_krb5_creds - Could not get initial
credentials for principal [ldap/kwtpr-idm-mstr@] in keytab
[FILE:/etc/dirsrv/ds.keytab]: -1765328203 (Key table entry not found)
[01/Apr/2015:11:01:49 +0300] set_krb5_creds - Could not get initial
credentials for principal [ldap/kwtpr-idm-mstr@] in keytab
[FILE:/etc/dirsrv/ds.keytab]: -1765328203 (Key table entry not found)
[01/Apr/2015:11:01:49 +0300] slapd_ldap_sasl_interactive_bind - Error:
could not perform interactive bind for id [] mech [GSSAPI]: LDAP error
-2 (Local error) (SASL(-1): generic failure: GSSAPI Error: Unspecified
GSS failure.  Minor code may provide more information (No Kerberos
credentials available)) errno 0 (Success)
[01/Apr/2015:11:01:49 +0300] slapi_ldap_bind - Error: could not
perform interactive bind for id [] authentication mechanism [GSSAPI]:
error -2 (Local error)
[01/Apr/2015:11:01:49 +0300] NSMMReplicationPlugin -
agmt="cn=meTokwtard-idm-slve.idm.local" (kwtard-idm-slve:389):
Replication bind with GSSAPI auth failed: LDAP error -2 (Local error)
(SASL(-1): generic failure: GSSAPI Error: Unspecified GSS failure.
Minor code may provide more information (No Kerberos credentials
available))
[01/Apr/2015:11:01:49 +0300] slapd_ldap_sasl_interactive_bind - Error:
could not perform interactive bind for id [] mech [GSSAPI]: LDAP error
-2 (Local error) (SASL(-1): generic failure: GSSAPI Error: Unspecified
GSS failure.  Minor code may provide more information (No Kerberos
credentials available)) errno 0 (Success)
[01/Apr/2015:11:01:49 +0300] slapi_ldap_bind - Error: could not
perform interactive bind for id [] authentication mechanism [GSSAPI]:
error -2 (Local error)
[01/Apr/2015:11:01:49 +0300] NSMMReplicationPlugin -
agmt="cn=meToindpr-idm-slve.idm.local" (indpr-idm-slve:389):
Replication bind with GSSAPI auth failed: LDAP error -2 (Local error)
(SASL(-1): generic failure: GSSAPI Error: Unspecified GSS failure.
Minor code may provide more information (No Kerberos credentials
available))
[01/Apr/2015:11:01:50 +0300] - slapd shutting down - signaling operation threads
[01/Apr/2015:11:01:50 +0300] - slapd shutting down - waiting for 27
threads to terminate
[01/Apr/2015:11:01:50 +0300] - slapd shutting down - closing down
internal subsystems and plugins
[01/Apr/2015:11:01:58 +0300] NSMMReplicationPlugin - CleanAllRUV Task:
Cleaning rid (6)...
[01/Apr/2015:11:01:58 +0300] NSMMReplicationPlugin - CleanAllRUV Task:
Waiting to process all the updates from the deleted replica...
[01/Apr/2015:11:01:58 +0300] NSMMReplicationPlugin - CleanAllRUV Task:
Waiting for all the replicas to be online...
[01/Apr/2015:11:01:58 +0300] NSMMReplicationPlugin - CleanAllRUV Task:
Server shutting down.  Process will resume at server startup
[01/Apr/2015:11:02:09 +0300] slapd_ldap_sasl_interactive_bind - Error:
could not perform interactive bind for id [] mech [GSSAPI]: LDAP error
-1 (Can't contact LDAP server) ((null)) errno 110 (Connection timed
out)
[01/Apr/2015:11:02:09 +0300] slapi_ldap_bind - Error: could not
perform interactive bind for id [] authentication mechanism [GSSAPI]:
error -1 (Can't contact LDAP server)
[01/Apr/2015:11:02:09 +0300] NSMMReplicationPlugin -
agmt="cn=meTokwtospr-idm-slve.idm.local" (kwtospr-idm-slve:389):
Replication bind with GSSAPI auth failed: LDAP error -1 (Can't contact
LDAP server) ()
[01/Apr/2015:11:02:09 +0300] slapd_ldap_sasl_interactive_bind - Error:
could not perform interactive bind for id [] mech [GSSAPI]: LDAP error
-2 (Local error) (SASL(-1): generic failure: GSSAPI Error: Unspecified
GSS failure.  Minor code may provide more information (No Kerberos
credentials available)) errno 0 (Success)
[01/Apr/2015:11:02:09 +0300] slapi_ldap_bind - Error: could not
perform interactive bind for id [] authentication mechanism [GSSAPI]:
error -2 (Local error)
[01/Apr/2015:11:02:09 +0300] NSMMReplicationPlugin -
agmt="cn=meTokwtpr-idm-slve.idm.local" (kwtpr-idm-slve:389):
Replication bind with GSSAPI auth failed: LDAP error -2 (Local error)
(SASL(-1): generic failure: GSSAPI Error: Unspecified GSS failure.
Minor code may provide more information (No Kerberos credentials
available))
errors
[01/Apr/2015:11:02:09 +0300] slapd_ldap_sasl_interactive_bind - Error:
could not perform interactive bind for id [] mech [GSSAPI]: LDAP error
-2 (Local error) (SASL(-1): generic failure: GSSAPI Error: Unspecified
GSS failure.  Minor code may provide more information (No Kerberos
credentials available)) errno 0 (Success)
[01/Apr/2015:11:02:09 +0300] slapi_ldap_bind - Error: could not
perform interactive bind for id [] authentication mechanism [GSSAPI]:
error -2 (Local error)
[01/Apr/2015:11:02:09 +0300] NSMMReplicationPlugin -
agmt="cn=meToukpr-idm-slve.idm.local" (ukpr-idm-slve:389): Replication
bind with GSSAPI auth failed: LDAP error -2 (Local error) (SASL(-1):
generic failure: GSSAPI Error: Unspecified GSS failure.  Minor code
may provide more information (No Kerberos credentials available))
[01/Apr/2015:11:02:09 +0300] - Waiting for 4 database threads to stop
[01/Apr/2015:11:02:10 +0300] - All database threads now stopped
[01/Apr/2015:11:02:10 +0300] - slapd stopped.
[01/Apr/2015:10:15:39 +0300] - 389-Directory/1.3.1.6 B2014.160.2139 starting up
[01/Apr/2015:10:15:39 +0300] schema-compat-plugin - warning: no
entries set up under cn=computers, cn=compat,dc=idm,dc=local
[01/Apr/2015:10:15:39 +0300] - Skipping CoS Definition cn=Password
Policy,cn=accounts,dc=idm,dc=local--no CoS Templates found, which
should be added before the CoS Definition.
[01/Apr/2015:10:15:39 +0300] NSMMReplicationPlugin - CleanAllRUV Task:
cleanAllRUV task found, resuming the cleaning of rid(6)...
[01/Apr/2015:10:15:39 +0300] set_krb5_creds - Could not get initial
credentials for principal [ldap/kwtpr-idm-mstr@] in keytab
[FILE:/etc/dirsrv/ds.keytab]: -1765328203 (Key table entry not found)
[01/Apr/2015:10:15:39 +0300] set_krb5_creds - Could not get initial
credentials for principal [ldap/kwtpr-idm-mstr@] in keytab
[FILE:/etc/dirsrv/ds.keytab]: -1765328203 (Key table entry not found)
[01/Apr/2015:10:15:39 +0300] - Skipping CoS Definition cn=Password
Policy,cn=accounts,dc=idm,dc=local--no CoS Templates found, which
should be added before the CoS Definition.
[01/Apr/2015:10:15:39 +0300] set_krb5_creds - Could not get initial
credentials for principal [ldap/kwtpr-idm-mstr@] in keytab
[FILE:/etc/dirsrv/ds.keytab]: -1765328203 (Key table entry not found)
[01/Apr/2015:10:15:39 +0300] slapd_ldap_sasl_interactive_bind - Error:
could not perform interactive bind for id [] mech [GSSAPI]: LDAP error
-2 (Local error) (SASL(-1): generic failure: GSSAPI Error: Unspecified
GSS failure.  Minor code may provide more information (No Kerberos
credentials available)) errno 2 (No such file or directory)
[01/Apr/2015:10:15:39 +0300] slapi_ldap_bind - Error: could not
perform interactive bind for id [] authentication mechanism [GSSAPI]:
error -2 (Local error)
[01/Apr/2015:10:15:39 +0300] set_krb5_creds - Could not get initial
credentials for principal [ldap/kwtpr-idm-mstr@] in keytab
[FILE:/etc/dirsrv/ds.keytab]: -1765328203 (Key table entry not found)
[01/Apr/2015:10:15:39 +0300] csngen_new_csn - Warning: too much time
skew (-2771 secs). Current seqnum=3
[01/Apr/2015:10:15:39 +0300] NSMMReplicationPlugin -
agmt="cn=meTokwtard-idm-slve.idm.local" (kwtard-idm-slve:389):
Replication bind with GSSAPI auth failed: LDAP error -2 (Local error)
(SASL(-1): generic failure: GSSAPI Error: Unspecified GSS failure.
Minor code may provide more information (No Kerberos credentials
available))
[01/Apr/2015:10:15:39 +0300] set_krb5_creds - Could not get initial
credentials for principal [ldap/kwtpr-idm-mstr@] in keytab
[FILE:/etc/dirsrv/ds.keytab]: -1765328203 (Key table entry not found)
[01/Apr/2015:10:15:39 +0300] csngen_new_csn - Warning: too much time
skew (-2770 secs). Current seqnum=1
[01/Apr/2015:10:15:39 +0300] - slapd started.  Listening on All
Interfaces port 389 for LDAP requests
[01/Apr/2015:10:15:39 +0300] - Listening on All Interfaces port 636
for LDAPS requests
[01/Apr/2015:10:15:39 +0300] - Listening on
/var/run/slapd-IDM-LOCAL.socket for LDAPI requests
[01/Apr/2015:10:15:39 +0300] slapd_ldap_sasl_interactive_bind - Error:
could not perform interactive bind for id [] mech [GSSAPI]: LDAP error
-2 (Local error) (SASL(-1): generic failure: GSSAPI Error: Unspecified
GSS failure.  Minor code may provide more information (No Kerberos
credentials available)) errno 0 (Success)
[01/Apr/2015:10:15:39 +0300] slapi_ldap_bind - Error: could not
perform interactive bind for id [] authentication mechanism [GSSAPI]:
error -2 (Local error)
[01/Apr/2015:10:15:39 +0300] NSMMReplicationPlugin -
agmt="cn=meToindpr-idm-slve.idm.local" (indpr-idm-slve:389):
Replication bind with GSSAPI auth failed: LDAP error -2 (Local error)
(SASL(-1): generic failure: GSSAPI Error: Unspecified GSS failure.
Minor code may provide more information (No Kerberos credentials
available))
[01/Apr/2015:10:15:40 +0300] csngen_new_csn - Warning: too much time
skew (-2771 secs). Current seqnum=1
[01/Apr/2015:10:15:41 +0300] - slapd shutting down - signaling operation threads
[01/Apr/2015:10:15:41 +0300] - slapd shutting down - waiting for 28
threads to terminate
[01/Apr/2015:10:15:41 +0300] - slapd shutting down - closing down
internal subsystems and plugins
[01/Apr/2015:10:15:48 +0300] NSMMReplicationPlugin - CleanAllRUV Task:
Cleaning rid (6)...
[01/Apr/2015:10:15:48 +0300] NSMMReplicationPlugin - CleanAllRUV Task:
Waiting to process all the updates from the deleted replica...
[01/Apr/2015:10:15:48 +0300] NSMMReplicationPlugin - CleanAllRUV Task:
Waiting for all the replicas to be online...
[01/Apr/2015:10:15:48 +0300] NSMMReplicationPlugin - CleanAllRUV Task:
Server shutting down.  Process will resume at server startup
[01/Apr/2015:10:15:58 +0300] slapd_ldap_sasl_interactive_bind - Error:
could not perform interactive bind for id [] mech [GSSAPI]: LDAP error
-1 (Can't contact LDAP server) ((null)) errno 110 (Connection timed
out)
[01/Apr/2015:10:15:58 +0300] slapi_ldap_bind - Error: could not
perform interactive bind for id [] authentication mechanism [GSSAPI]:
error -1 (Can't contact LDAP server)
[01/Apr/2015:10:15:58 +0300] NSMMReplicationPlugin -
agmt="cn=meTokwtospr-idm-slve.idm.local" (kwtospr-idm-slve:389):
Replication bind with GSSAPI auth failed: LDAP error -1 (Can't contact
LDAP server) ()
[01/Apr/2015:10:15:58 +0300] slapd_ldap_sasl_interactive_bind - Error:
could not perform interactive bind for id [] mech [GSSAPI]: LDAP error
-2 (Local error) (SASL(-1): generic failure: GSSAPI Error: Unspecified
GSS failure.  Minor code may provide more information (No Kerberos
credentials available)) errno 0 (Success)
[01/Apr/2015:10:15:58 +0300] slapi_ldap_bind - Error: could not
perform interactive bind for id [] authentication mechanism [GSSAPI]:
error -2 (Local error)
[01/Apr/2015:10:15:58 +0300] NSMMReplicationPlugin -
agmt="cn=meTokwtpr-idm-slve.idm.local" (kwtpr-idm-slve:389):
Replication bind with GSSAPI auth failed: LDAP error -2 (Local error)
(SASL(-1): generic failure: GSSAPI Error: Unspecified GSS failure.
Minor code may provide more information (No Kerberos credentials
available))
[01/Apr/2015:10:15:59 +0300] slapd_ldap_sasl_interactive_bind - Error:
could not perform interactive bind for id [] mech [GSSAPI]: LDAP error
-2 (Local error) (SASL(-1): generic failure: GSSAPI Error: Unspecified
GSS failure.  Minor code may provide more information (No Kerberos
credentials available)) errno 0 (Success)
[01/Apr/2015:10:15:59 +0300] slapi_ldap_bind - Error: could not
perform interactive bind for id [] authentication mechanism [GSSAPI]:
error -2 (Local error)
[01/Apr/2015:10:15:59 +0300] NSMMReplicationPlugin -
agmt="cn=meToukpr-idm-slve.idm.local" (ukpr-idm-slve:389): Replication
bind with GSSAPI auth failed: LDAP error -2 (Local error) (SASL(-1):
generic failure: GSSAPI Error: Unspecified GSS failure.  Minor code
may provide more information (No Kerberos credentials available))
[01/Apr/2015:10:15:59 +0300] - Waiting for 4 database threads to stop
[01/Apr/2015:10:16:00 +0300] - All database threads now stopped
[01/Apr/2015:10:16:00 +0300] - slapd stopped.

On Wed, Apr 1, 2015 at 9:56 AM, Traiano Welcome <traiano at gmail.com> wrote:
> Hi List
>
> I've just tried to restart my IPA services after recently adding a new
> replica (0 configuration changes on the IPA server otherwise!), but
> ipactl fails when starting up named:
>
> ---
> [root at lolpr-xyz-mstr slapd-XYZ-LOCAL]# ipactl start
> Starting Directory Service
> Starting krb5kdc Service
> Starting kadmin Service
> Starting named Service
> Job for named.service failed. See 'systemctl status named.service' and
> 'journalctl -xn' for details.
> Failed to start named Service
> Shutting down
> Aborting ipactl
> ---
>
> I then manual start named service and try again, but then smb service fails:
>
> ---
> [root at lolpr-xyz-mstr ~]# ipactl start
> Existing service file detected!
> Assuming stale, cleaning and proceeding
> Starting Directory Service
> Starting krb5kdc Service
> Starting kadmin Service
> Starting named Service
> Starting ipa_memcached Service
> Starting httpd Service
> Starting pki-tomcatd Service
> Starting smb Service
> Job for smb.service failed. See 'systemctl status smb.service' and
> 'journalctl -xn' for details.
> Failed to start smb Service
> Shutting down
> Aborting ipactl
> ---
>
> systemctl status shows the following output for smb.service:
>
> ---
> [root at lolpr-xyz-mstr ~]# systemctl -l status smb.service
> smb.service - Samba SMB Daemon
>    Loaded: loaded (/usr/lib/systemd/system/smb.service; disabled)
>    Active: failed (Result: exit-code) since Wed 2015-04-01 09:21:10
> AST; 1min 14s ago
>   Process: 4662 ExecStart=/usr/sbin/smbd $SMBDOPTIONS (code=exited,
> status=1/FAILURE)
>  Main PID: 4662 (code=exited, status=1/FAILURE)
>    Status: "Starting process..."
>    CGroup: /system.slice/smb.service
>
> Apr 01 09:21:09 lolpr-xyz-mstr.xyz.local smbd[4662]: GSSAPI client step 1
> Apr 01 09:21:09 lolpr-xyz-mstr.xyz.local smbd[4662]: GSSAPI Error:
> Unspecified GSS failure.  Minor code may provide more information
> (Server ldap/lolpr-xyz-mstr at XYZ.LOCAL not found in Kerberos database)
> Apr 01 09:21:10 lolpr-xyz-mstr.xyz.local smbd[4662]: [2015/04/01
> 09:21:10.211028,  0] ipa_sam.c:4440(pdb_init_ipasam)
> Apr 01 09:21:10 lolpr-xyz-mstr.xyz.local smbd[4662]: Failed to get base DN.
> Apr 01 09:21:10 lolpr-xyz-mstr.xyz.local smbd[4662]: [2015/04/01
> 09:21:10.211210,  0]
> ../source3/passdb/pdb_interface.c:178(make_pdb_method_name)
> Apr 01 09:21:10 lolpr-xyz-mstr.xyz.local smbd[4662]: pdb backend
> ipasam:ldapi://%2fvar%2frun%2fslapd-XYZ-LOCAL.socket did not correctly
> init (error was NT_STATUS_UNSUCCESSFUL)
> Apr 01 09:21:10 lolpr-xyz-mstr.xyz.local systemd[1]: smb.service: main
> process exited, code=exited, status=1/FAILURE
> Apr 01 09:21:10 lolpr-xyz-mstr.xyz.local systemd[1]: Failed to start
> Samba SMB Daemon.
> Apr 01 09:21:10 lolpr-xyz-mstr.xyz.local systemd[1]: Unit smb.service
> entered failed state.
> Apr 01 09:21:12 lolpr-xyz-mstr.xyz.local systemd[1]: Stopped Samba SMB Daemon.
> ---
>
>
> I manually try to start the smb service as follows, but can't (Of
> course the directory service is not up, so there's a little catch22
> there and this many not mean much):
>
>
> ---
>
> [root at lolpr-xyz-mstr slapd-XYZ-LOCAL]# systemctl status smb.service
> smb.service - Samba SMB Daemon
>    Loaded: loaded (/usr/lib/systemd/system/smb.service; disabled)
>    Active: failed (Result: exit-code) since Wed 2015-04-01 09:50:38 AST; 57s ago
>   Process: 8089 ExecStart=/usr/sbin/smbd $SMBDOPTIONS (code=exited,
> status=1/FAILURE)
>  Main PID: 8089 (code=exited, status=1/FAILURE)
>    Status: "Starting process..."
>
> Apr 01 09:50:36 lolpr-xyz-mstr.xyz.local smbd[8089]: kerberos error:
> code=-1765328228, message=Cannot contact any KDC for realm 'XYZ.LOCAL'
> Apr 01 09:50:37 lolpr-xyz-mstr.xyz.local smbd[8089]: [2015/04/01
> 09:50:37.573772,  0] ipa_sam.c:4128(bind_callback_cleanup)
> Apr 01 09:50:37 lolpr-xyz-mstr.xyz.local smbd[8089]: kerberos error:
> code=-1765328228, message=Cannot contact any KDC for realm 'XYZ.LOCAL'
> Apr 01 09:50:38 lolpr-xyz-mstr.xyz.local smbd[8089]: [2015/04/01
> 09:50:38.574722,  0] ipa_sam.c:4440(pdb_init_ipasam)
> Apr 01 09:50:38 lolpr-xyz-mstr.xyz.local smbd[8089]: Failed to get base DN.
> Apr 01 09:50:38 lolpr-xyz-mstr.xyz.local smbd[8089]: [2015/04/01
> 09:50:38.574903,  0]
> ../source3/passdb/pdb_interface.c:178(make_pdb_method_name)
> Apr 01 09:50:38 lolpr-xyz-mstr.xyz.local smbd[8089]: pdb backend
> ipasam:ldapi://%2fvar%2frun%2fslapd-XYZ-LOCAL.socket did not correctly
> init (error was NT_STATUS_UNSUCCESSFUL)
> Apr 01 09:50:38 lolpr-xyz-mstr.xyz.local systemd[1]: smb.service: main
> process exited, code=exited, status=1/FAILURE
> Apr 01 09:50:38 lolpr-xyz-mstr.xyz.local systemd[1]: Failed to start
> Samba SMB Daemon.
> Apr 01 09:50:38 lolpr-xyz-mstr.xyz.local systemd[1]: Unit smb.service
> entered failed state.
> [root at lolpr-xyz-mstr slapd-XYZ-LOCAL]#
>
> ---
>
> Please could someone advise me on how to drill deeper into debugging
> this issue to get ipactl to start ?
>
> NOTES:
>
> - This server is successfully in a Trust relationship with ActiveDirectory.
> - There are a number of replicas established which have been working
> fine til this morning
> - Another replica was added around the time of the failure using the
> same steps as usual (not sure how this could be related)
>
>
> Many thanks in advance,
> Traiano