[Freeipa-users] FreeIPA web UI Freezing up

Mon Jun 8 16:07:09 UTC 2015

On 06/08/2015 10:02 AM, nathan at nathanpeters.com wrote:
>> On 06/05/2015 03:31 PM, nathan at nathanpeters.com wrote:
>>>> I have noticed that happen a couple times in the last few days.
>>>> FreeIPA
>>>> server 4.1.3 on CentOS 7 with a sync relationship to a Windows server
>>>> 2008R2 domain controller.
>>>>
>>>> The web ui will stop working and just show a blank page.
>>>>
>>>> When I try to do a ipactl status the command just freezes and does
>>>> nothing.
>>>>
>>>> In the exmaple I paste below, there was 5 minutes between when I
>>>> entered
>>>> the command and when I did ctrl-c after getting tired of waiting for
>>>> nothing to happen.
>>>> After the ipactl command failed to work at all, I decided to restart
>>>> the
>>>> httpd service manually, and then saw a whole pile of strange errors
>>>> around
>>>> failing to bind to ldap server and generic kerberos errors.
>>>>
>>>> Rebooting the server seems to work for 24 hours or so until things go
>>>> wonky again.
>>>>
>>>> [username at dc1 ~]$ sudo su -
>>>> Last login: Fri Jun  5 16:05:55 UTC 2015 on pts/0
>>>> [root at dc1 ~]# ipactl status
>>>> ^CCancelled.
>>>> [root at dc1 ~]# ipactl restart
>>>> ^CCancelled.
>>>> [root at dc1 ~]# ipactl restart
>>>> ^CCancelled.
>>>> [root at dc1 ~]# systemctl restart httpd
>>>> [root at dc1 ~]#
>>>>
>>>>
>>>> Jun 05 21:02:32 dc1.mydomain.net systemd[1]: Stopping The Apache HTTP
>>>> Server...
>>>> Jun 05 21:03:01 dc1.mydomain.net winbindd[2171]: GSSAPI client step 1
>>>> Jun 05 21:03:01 dc1.mydomain.net winbindd[2171]: GSSAPI client step 1
>>>> Jun 05 21:03:19 dc1.mydomain.net systemd[1]: Created slice
>>>> user-0.slice.
>>>> Jun 05 21:03:19 dc1.mydomain.net systemd[1]: Starting Session 161 of
>>>> user
>>>> root.
>>>> Jun 05 21:03:19 dc1.mydomain.net systemd-logind[604]: New session 161
>>>> of
>>>> user root.
>>>> Jun 05 21:03:19 dc1.mydomain.net systemd[1]: Started Session 161 of
>>>> user
>>>> root.
>>>> Jun 05 21:03:19 dc1.mydomain.net login[614]: pam_unix(login:session):
>>>> session opened for user root by LOGIN(uid=0)
>>>> Jun 05 21:03:19 dc1.mydomain.net login[614]: ROOT LOGIN ON tty1
>>>> Jun 05 21:03:22 dc1.mydomain.net winbindd[2171]: [2015/06/05
>>>> 21:03:22.932855,  0] ipa_sam.c:4144(bind_callback_cleanup)
>>>> Jun 05 21:03:22 dc1.mydomain.net winbindd[2171]: kerberos error:
>>>> code=-1765328324, message=Generic error (see e-text)
>>>> Jun 05 21:03:22 dc1.mydomain.net winbindd[2171]: GSSAPI client step 1
>>>> Jun 05 21:03:22 dc1.mydomain.net winbindd[2171]: GSSAPI client step 1
>>>> Jun 05 21:03:43 dc1.mydomain.net winbindd[2171]: [2015/06/05
>>>> 21:03:43.935800,  0] ipa_sam.c:4144(bind_callback_cleanup)
>>>> Jun 05 21:03:43 dc1.mydomain.net winbindd[2171]: kerberos error:
>>>> code=-1765328324, message=Generic error (see e-text)
>>>> Jun 05 21:03:46 dc1.mydomain.net smbd[2208]: GSSAPI client step 1
>>>> Jun 05 21:03:46 dc1.mydomain.net smbd[2208]: GSSAPI client step 1
>>>> Jun 05 21:04:02 dc1.mydomain.net systemd[1]: httpd.service stopping
>>>> timed
>>>> out. Killing.
>>>> Jun 05 21:04:02 dc1.mydomain.net systemd[1]: httpd.service: main
>>>> process
>>>> exited, code=killed, status=9/KILL
>>>> Jun 05 21:04:02 dc1.mydomain.net systemd[1]: Unit httpd.service entered
>>>> failed state.
>>>> Jun 05 21:04:02 dc1.mydomain.net systemd[1]: Starting The Apache HTTP
>>>> Server...
>>>> Jun 05 21:04:02 dc1.mydomain.net systemd[1]: Started The Apache HTTP
>>>> Server.
>>>> Jun 05 21:04:07 dc1.mydomain.net smbd[2208]: [2015/06/05
>>>> 21:04:07.152666,
>>>> 0] ipa_sam.c:4144(bind_callback_cleanup)
>>>> Jun 05 21:04:07 dc1.mydomain.net smbd[2208]: kerberos error:
>>>> code=-1765328324, message=Generic error (see e-text)
>>>> Jun 05 21:04:07 dc1.mydomain.net smbd[2208]: [2015/06/05
>>>> 21:04:07.152995,
>>>> 0] ../source3/lib/smbldap.c:998(smbldap_connect_system)
>>>> Jun 05 21:04:07 dc1.mydomain.net smbd[2208]: failed to bind to server
>>>> ldapi://%2fvar%2frun%2fslapd-MYDOMAIN-NET.socket with dn="[Anonymous
>>>> bind]" Error: Local error
>>>> Jun 05 21:04:07 dc1.mydomain.net smbd[2208]: (unknown)
>>>> Jun 05 21:04:07 dc1.mydomain.net smbd[2208]: [2015/06/05
>>>> 21:04:07.153407,
>>>> 0]
>>>> ../source3/rpc_server/netlogon/srv_netlog_nt.c:975(_netr_ServerAuthenticate3)
>>>> Jun 05 21:04:07 dc1.mydomain.net smbd[2208]: _netr_ServerAuthenticate3:
>>>> failed to get machine password for account office.mydomain.net.:
>>>> NT_STATUS_NONE_MAPPED
>>>> Jun 05 21:08:02 dc1.mydomain.net winbindd[2171]: GSSAPI client step 1
>>>> Jun 05 21:08:02 dc1.mydomain.net winbindd[2171]: GSSAPI client step 1
>>>> Jun 05 21:08:23 dc1.mydomain.net winbindd[2171]: [2015/06/05
>>>> 21:08:23.034001,  0] ipa_sam.c:4144(bind_callback_cleanup)
>>>> Jun 05 21:08:23 dc1.mydomain.net winbindd[2171]: kerberos error:
>>>> code=-1765328324, message=Generic error (see e-text)
>>>> Jun 05 21:08:23 dc1.mydomain.net winbindd[2171]: GSSAPI client step 1
>>>> Jun 05 21:08:23 dc1.mydomain.net winbindd[2171]: GSSAPI client step 1
>>>>
>>>> I also got this error from the web ui after restarting httpd:
>>>>
>>>> Runtime error
>>>>
>>>> Web UI got in unrecoverable state during "metadata" phase
>>>>
>>>>
>>>> --
>>>> Manage your subscription for the Freeipa-users mailing list:
>>>> https://www.redhat.com/mailman/listinfo/freeipa-users
>>>> Go to http://freeipa.org for more info on the project
>>>>
>>> Further information : restarting the httpd service didn't help, but
>>> restarting the dirsrv service allowed me to once again login to the
>>> webui
>>> and the ipactl command started working again after the restart of
>>> dirsrv.
>>>
>>> Is there something I can look for in my logs next time this happens. I
>>> have a feeling it *will* happen again this is a critical server I'm in
>>> charge of so it will not be good if I cannot come up with a solid
>>> explanation or bug report on why this server spontaneously stops
>>> working.
>>>
>>> [root at dc1 ~]# ipactl restart
>>> (waiting 3 or 4 minutes with nothing happening)
>>> ^CCancelled.
>>> [root at dc1 ~]# systemctl restart dirsrv at MYDOMAIN-NET
>>> [root at dc1 ~]# ipactl status
>>> Directory Service: RUNNING
>>> krb5kdc Service: RUNNING
>>> kadmin Service: RUNNING
>>> named Service: RUNNING
>>> ipa_memcached Service: RUNNING
>>> httpd Service: RUNNING
>>> pki-tomcatd Service: RUNNING
>>> smb Service: RUNNING
>>> winbind Service: RUNNING
>>> ipa-otpd Service: RUNNING
>>> ipa-dnskeysyncd Service: RUNNING
>>> ipa: INFO: The ipactl command was successful
>>> [root at dc1 ~]#
>>>
>>> Here are some additional entries from my /var/log/dirsrv/slapd-MYDOMAIN
>>> logs.  Strange error messages about non initialized replica.
>>>
>>> However, I know the windows machine is properly syncing data because I
>>> have over 300 synced users and when I update them in AD the updated
>>> attributes sync to IPA.
>> Is it possible this is an old winsync agreement that is no longer valid?
> I have only ever made a single winsync agreement on this server that I
> know of.  How would I tell if an agreement is no longer valid?
>
>

ldapsearch -xLLL -D "cn=directory manager" -W -b cn=config 
objectclass=nsDSWindowsReplicationAgreement