[Freeipa-users] IPA wont start, all services fail

Mon Jan 18 00:54:19 UTC 2016

> -----Original Message-----
> From: Alexander Bokovoy [mailto:abokovoy at redhat.com]
> >This is from the smb log:
> >
> >It's hard to tell why they won't start, but it looks a little like
> >Kerberos won't start because there aren't any values in LDAP, and LDAP
> >won't start because Kerberos isn't started?
> No, LDAP server startup is not tied to Kerberos. It can perfectly start without that,
> as Kerberos in 389-ds is only needed for replication to happen.

Great - thanks.

> Samba is failing because it cannot get access to LDAP server using GSSAPI,
> that's right.
> 
> KDC is failing because LDAP server is not available, that's right too.
> ... 
> You may ignore ACL's plugin output as it just mentions that there are ACLs
> against entries which don't exist -- this is normal, because we still have ACLs in
> place for cn=dns,$SUFFIX even if you don't configure integrated DNS. These
> messages have nothing to do with your problem.

ok, thanks.

> None of the above is revealing an issue.
> 
> Follow http://www.port389.org/docs/389ds/FAQ/faq.html#debugging-crashes
> to enable crashdumps for ns-slapd to see what happens in reality (check
> systemd-enabled systems' recipes).

Here is where things got interesting - I was 20 minutes in before I realised I had 
no dirsrv core dumps.

New things I learnt while doing this though:

 - I have 2.5 GB of core files in /var/log/samba/cores/winbindd ? To the best of my 
knowledge I was using SSSD, I have no idea what winbind is doing there. Can I just 
delete (yum remove samba-winbind*) it? From the look of it, I'm getting a new winbind 
core dump every 5 minutes.Could this be stopping samba from running?

 - /etc/nsswitch.conf is all "files sss" - there's no winbind anywhere.

- while following the instructions to "set ulimit -c unlimited" on system I found things 
that *really* confused me:

As noted in the original email, this was in the failed list of systemctld:

 dirsrv at unix.co.org.au.service

and it continues to fail this morning. So I tried running 

sc start dirsrv.target

and that worked:

[root at vmts-linuxidm samba]# sc status dirsrv.target
● dirsrv.target - 389 Directory Server
   Loaded: loaded (/usr/lib/systemd/system/dirsrv.target; enabled; vendor preset: disabled)
   Active: active since Mon 2016-01-18 09:58:14 AEDT; 1h 20min ago

Jan 18 09:58:14 vmts-linuxidm.unix.co.org.au systemd[1]: Reached target 389 Directory Server.
Jan 18 09:58:14 vmts-linuxidm.unix.co.org.au systemd[1]: Starting 389 Directory Server.

So I stopped it and started dirsrv at unix.co.org.au just to confirm, and yes it's failing. 
After some testing, I discovered that *this* would work:

sc start dirsrv at UNIX-CO-ORG-AU

My syntax was all wrong. (Does anyone know how can I clear out bad syntax from the 
systemctld output?)

Anyway, I have a running dirsrv, but SMB still fails, and it's failing on winbind first (see 
notes below). It looks like it's because there's no Kerberos server available. Indeed, 
kinit admin is still failing. I think that when I ran ipa-adtrust-install I said no to creating 
sids for local users. 

I'm beginning to think that is the root error, but have a feeling that winbind isn't helping 
either.

Does this seem more likely?

Cheers
L.

Notes:

Running DIRSRV 

[root at vmts-linuxidm samba]# sc status dirsrv at UNIX-CO-ORG-AU.service
● dirsrv at UNIX-CO-ORG-AU.service - 389 Directory Server UNIX-CO-ORG-AU.
   Loaded: loaded (/usr/lib/systemd/system/dirsrv at .service; enabled; vendor preset: disabled)
   Active: active (running) since Mon 2016-01-18 11:21:25 AEDT; 5min ago
  Process: 11655 ExecStart=/usr/sbin/ns-slapd -D /etc/dirsrv/slapd-%i -i /var/run/dirsrv/slapd-%i.pid -w /var/run/dirsrv/slapd-%i.startpid (code=exited, status=0/SUCCESS)
 Main PID: 11656 (ns-slapd)
   CGroup: /system.slice/system-dirsrv.slice/dirsrv at UNIX-CO-ORG-AU.service
           └─11656 /usr/sbin/ns-slapd -D /etc/dirsrv/slapd-UNIX-CO-ORG-AU -i /var/run/dirsrv/slapd-UNIX-CO-OR...

Jan 18 11:21:25 vmts-linuxidm.unix.co.org.au ns-slapd[11655]: [18/Jan/2016:11:21:25 +1100] - SSL alert:         ...led
Jan 18 11:21:25 vmts-linuxidm.unix.co.org.au ns-slapd[11655]: [18/Jan/2016:11:21:25 +1100] - SSL alert:         ...led
Jan 18 11:21:25 vmts-linuxidm.unix.co.org.au ns-slapd[11655]: [18/Jan/2016:11:21:25 +1100] - SSL alert:         ...led
Jan 18 11:21:25 vmts-linuxidm.unix.co.org.au ns-slapd[11655]: [18/Jan/2016:11:21:25 +1100] - SSL alert:         ...led
Jan 18 11:21:25 vmts-linuxidm.unix.co.org.au ns-slapd[11655]: [18/Jan/2016:11:21:25 +1100] - SSL alert:         ...led
Jan 18 11:21:25 vmts-linuxidm.unix.co.org.au ns-slapd[11655]: [18/Jan/2016:11:21:25 +1100] - SSL alert:         ...led
Jan 18 11:21:25 vmts-linuxidm.unix.co.org.au ns-slapd[11655]: [18/Jan/2016:11:21:25 +1100] SSL Initialization - ...1.2
Jan 18 11:25:06 vmts-linuxidm.unix.co.org.au ns-slapd[11656]: GSSAPI server step 1
Jan 18 11:25:06 vmts-linuxidm.unix.co.org.au ns-slapd[11656]: GSSAPI server step 2
Jan 18 11:25:06 vmts-linuxidm.unix.co.org.au ns-slapd[11656]: GSSAPI server step 3

When samba fails, from journalctl -xe (I'm from Ubuntu land, I'm still getting used to Centos)

vmts-linuxidm.unix.co.org.au winbindd[11717]: [2016/01/18 11:25:02.359848,  0] ipa_sam.c:4208(bind_callback_cleanup)
vmts-linuxidm.unix.co.org.au winbindd[11717]:   kerberos error: code=-1765328228, message=Cannot contact any KDC for realm 'UNIX.CO.ORG.AU'
vmts-linuxidm.unix.co.org.au winbindd[11717]: [2016/01/18 11:25:02.359949,  0] ../source3/lib/smbldap.c:998(smbldap_connect_system)
vmts-linuxidm.unix.co.org.au winbindd[11717]:   failed to bind to server ldapi://%2fvar%2frun%2fslapd-UNIX-CO-ORG-AU.socket with dn="[Anonymous bind]" Error: Local error
vmts-linuxidm.unix.co.org.au winbindd[11717]:           (unknown)
vmts-linuxidm.unix.co.org.au winbindd[11717]: [2016/01/18 11:25:03.361039,  0] ipa_sam.c:4208(bind_callback_cleanup)
vmts-linuxidm.unix.co.org.au winbindd[11717]:   kerberos error: code=-1765328228, message=Cannot contact any KDC for realm 'UNIX.CO.ORG.AU'
vmts-linuxidm.unix.co.org.au winbindd[11717]: [2016/01/18 11:25:04.361894,  0] ipa_sam.c:4208(bind_callback_cleanup)
vmts-linuxidm.unix.co.org.au winbindd[11717]:   kerberos error: code=-1765328228, message=Cannot contact any KDC for realm 'UNIX.CO.ORG.AU'
vmts-linuxidm.unix.co.org.au polkitd[660]: Registered Authentication Agent for unix-process:11718:525588 (system bus name :1.40 [/usr/bin/pkttyagent --notify-fd 5 --fallback], object path /org/freedesktop/PolicyKit1/AuthenticationAgent, locale en_AU.UTF-8)
vmts-linuxidm.unix.co.org.au polkitd[660]: Unregistered Authentication Agent for unix-process:11718:525588 (system bus name :1.40, object path /org/freedesktop/PolicyKit1/AuthenticationAgent, locale en_AU.UTF-8) (disconnected from bus)
vmts-linuxidm.unix.co.org.au winbindd[11717]: [2016/01/18 11:25:05.362765,  0] ipa_sam.c:4208(bind_callback_cleanup)
vmts-linuxidm.unix.co.org.au winbindd[11717]:   kerberos error: code=-1765328228, message=Cannot contact any KDC for realm 'UNIX.CO.ORG.AU'
vmts-linuxidm.unix.co.org.au polkitd[660]: Registered Authentication Agent for unix-process:11723:525731 (system bus name :1.41 [/usr/bin/pkttyagent --notify-fd 5 --fallback], object path /org/freedesktop/PolicyKit1/AuthenticationAgent, locale en_AU.UTF-8)
vmts-linuxidm.unix.co.org.au systemd[1]: Starting Samba SMB Daemon...
Subject: Unit smb.service has begun start-up
Defined-By: systemd
Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel

Unit smb.service has begun starting up.
vmts-linuxidm.unix.co.org.au smbd[11729]: GSSAPI client step 1
vmts-linuxidm.unix.co.org.au smbd[11729]: GSSAPI client step 1
vmts-linuxidm.unix.co.org.au ns-slapd[11656]: GSSAPI server step 1
vmts-linuxidm.unix.co.org.au smbd[11729]: GSSAPI client step 1
vmts-linuxidm.unix.co.org.au ns-slapd[11656]: GSSAPI server step 2
vmts-linuxidm.unix.co.org.au smbd[11729]: GSSAPI client step 2
vmts-linuxidm.unix.co.org.au ns-slapd[11656]: GSSAPI server step 3
vmts-linuxidm.unix.co.org.au smbd[11729]: [2016/01/18 11:25:06.183597,  0] ipa_sam.c:3654(get_fallback_group_sid)
vmts-linuxidm.unix.co.org.au smbd[11729]:   Missing mandatory attribute ipaNTSecurityIdentifier.
vmts-linuxidm.unix.co.org.au smbd[11729]: [2016/01/18 11:25:06.183642,  0] ipa_sam.c:4606(pdb_init_ipasam)
vmts-linuxidm.unix.co.org.au smbd[11729]:   Cannot find SID of fallback group.
vmts-linuxidm.unix.co.org.au smbd[11729]: [2016/01/18 11:25:06.183659,  0] ../source3/passdb/pdb_interface.c:179(make_pdb_method_name)
vmts-linuxidm.unix.co.org.au smbd[11729]:   pdb backend ipasam:ldapi://%2fvar%2frun%2fslapd-UNIX-CO-ORG-AU.socket did not correctly init (error was NT_STATUS_INVALID_PARAMETER)
vmts-linuxidm.unix.co.org.au polkitd[660]: Unregistered Authentication Agent for unix-process:11723:525731 (system bus name :1.41, object path /org/freedesktop/PolicyKit1/AuthenticationAgent, locale en_AU.UTF-8) (disconnected from bus)
vmts-linuxidm.unix.co.org.au systemd[1]: smb.service: main process exited, code=exited, status=1/FAILURE
vmts-linuxidm.unix.co.org.au systemd[1]: Failed to start Samba SMB Daemon.
Subject: Unit smb.service has failed
Defined-By: systemd
Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel

Unit smb.service has failed.

The result is failed.
vmts-linuxidm.unix.co.org.au systemd[1]: Unit smb.service entered failed state.
vmts-linuxidm.unix.co.org.au systemd[1]: smb.service failed.
vmts-linuxidm.unix.co.org.au winbindd[11717]: [2016/01/18 11:25:06.363629,  0] ipa_sam.c:4208(bind_callback_cleanup)
vmts-linuxidm.unix.co.org.au winbindd[11717]:   kerberos error: code=-1765328228, message=Cannot contact any KDC for realm 'UNIX.CO.ORG.AU'

This email (including any attachments or links) may contain 
confidential and/or legally privileged information and is 
intended only to be read or used by the addressee.  If you 
are not the intended addressee, any use, distribution, 
disclosure or copying of this email is strictly 
prohibited.  
Confidentiality and legal privilege attached to this email 
(including any attachments) are not waived or lost by 
reason of its mistaken delivery to you.
If you have received this email in error, please delete it 
and notify us immediately by telephone or email.  Peter 
MacCallum Cancer Centre provides no guarantee that this 
transmission is free of virus or that it has not been 
intercepted or altered and will not be liable for any delay 
in its receipt.