[Freeipa-users] FreeIPA web UI Freezing up

Mon Jun 8 16:02:55 UTC 2015

> On 06/05/2015 03:31 PM, nathan at nathanpeters.com wrote:
>>> I have noticed that happen a couple times in the last few days.
>>> FreeIPA
>>> server 4.1.3 on CentOS 7 with a sync relationship to a Windows server
>>> 2008R2 domain controller.
>>>
>>> The web ui will stop working and just show a blank page.
>>>
>>> When I try to do a ipactl status the command just freezes and does
>>> nothing.
>>>
>>> In the exmaple I paste below, there was 5 minutes between when I
>>> entered
>>> the command and when I did ctrl-c after getting tired of waiting for
>>> nothing to happen.
>>> After the ipactl command failed to work at all, I decided to restart
>>> the
>>> httpd service manually, and then saw a whole pile of strange errors
>>> around
>>> failing to bind to ldap server and generic kerberos errors.
>>>
>>> Rebooting the server seems to work for 24 hours or so until things go
>>> wonky again.
>>>
>>> [username at dc1 ~]$ sudo su -
>>> Last login: Fri Jun  5 16:05:55 UTC 2015 on pts/0
>>> [root at dc1 ~]# ipactl status
>>> ^CCancelled.
>>> [root at dc1 ~]# ipactl restart
>>> ^CCancelled.
>>> [root at dc1 ~]# ipactl restart
>>> ^CCancelled.
>>> [root at dc1 ~]# systemctl restart httpd
>>> [root at dc1 ~]#
>>>
>>>
>>> Jun 05 21:02:32 dc1.mydomain.net systemd[1]: Stopping The Apache HTTP
>>> Server...
>>> Jun 05 21:03:01 dc1.mydomain.net winbindd[2171]: GSSAPI client step 1
>>> Jun 05 21:03:01 dc1.mydomain.net winbindd[2171]: GSSAPI client step 1
>>> Jun 05 21:03:19 dc1.mydomain.net systemd[1]: Created slice
>>> user-0.slice.
>>> Jun 05 21:03:19 dc1.mydomain.net systemd[1]: Starting Session 161 of
>>> user
>>> root.
>>> Jun 05 21:03:19 dc1.mydomain.net systemd-logind[604]: New session 161
>>> of
>>> user root.
>>> Jun 05 21:03:19 dc1.mydomain.net systemd[1]: Started Session 161 of
>>> user
>>> root.
>>> Jun 05 21:03:19 dc1.mydomain.net login[614]: pam_unix(login:session):
>>> session opened for user root by LOGIN(uid=0)
>>> Jun 05 21:03:19 dc1.mydomain.net login[614]: ROOT LOGIN ON tty1
>>> Jun 05 21:03:22 dc1.mydomain.net winbindd[2171]: [2015/06/05
>>> 21:03:22.932855,  0] ipa_sam.c:4144(bind_callback_cleanup)
>>> Jun 05 21:03:22 dc1.mydomain.net winbindd[2171]: kerberos error:
>>> code=-1765328324, message=Generic error (see e-text)
>>> Jun 05 21:03:22 dc1.mydomain.net winbindd[2171]: GSSAPI client step 1
>>> Jun 05 21:03:22 dc1.mydomain.net winbindd[2171]: GSSAPI client step 1
>>> Jun 05 21:03:43 dc1.mydomain.net winbindd[2171]: [2015/06/05
>>> 21:03:43.935800,  0] ipa_sam.c:4144(bind_callback_cleanup)
>>> Jun 05 21:03:43 dc1.mydomain.net winbindd[2171]: kerberos error:
>>> code=-1765328324, message=Generic error (see e-text)
>>> Jun 05 21:03:46 dc1.mydomain.net smbd[2208]: GSSAPI client step 1
>>> Jun 05 21:03:46 dc1.mydomain.net smbd[2208]: GSSAPI client step 1
>>> Jun 05 21:04:02 dc1.mydomain.net systemd[1]: httpd.service stopping
>>> timed
>>> out. Killing.
>>> Jun 05 21:04:02 dc1.mydomain.net systemd[1]: httpd.service: main
>>> process
>>> exited, code=killed, status=9/KILL
>>> Jun 05 21:04:02 dc1.mydomain.net systemd[1]: Unit httpd.service entered
>>> failed state.
>>> Jun 05 21:04:02 dc1.mydomain.net systemd[1]: Starting The Apache HTTP
>>> Server...
>>> Jun 05 21:04:02 dc1.mydomain.net systemd[1]: Started The Apache HTTP
>>> Server.
>>> Jun 05 21:04:07 dc1.mydomain.net smbd[2208]: [2015/06/05
>>> 21:04:07.152666,
>>> 0] ipa_sam.c:4144(bind_callback_cleanup)
>>> Jun 05 21:04:07 dc1.mydomain.net smbd[2208]: kerberos error:
>>> code=-1765328324, message=Generic error (see e-text)
>>> Jun 05 21:04:07 dc1.mydomain.net smbd[2208]: [2015/06/05
>>> 21:04:07.152995,
>>> 0] ../source3/lib/smbldap.c:998(smbldap_connect_system)
>>> Jun 05 21:04:07 dc1.mydomain.net smbd[2208]: failed to bind to server
>>> ldapi://%2fvar%2frun%2fslapd-MYDOMAIN-NET.socket with dn="[Anonymous
>>> bind]" Error: Local error
>>> Jun 05 21:04:07 dc1.mydomain.net smbd[2208]: (unknown)
>>> Jun 05 21:04:07 dc1.mydomain.net smbd[2208]: [2015/06/05
>>> 21:04:07.153407,
>>> 0]
>>> ../source3/rpc_server/netlogon/srv_netlog_nt.c:975(_netr_ServerAuthenticate3)
>>> Jun 05 21:04:07 dc1.mydomain.net smbd[2208]: _netr_ServerAuthenticate3:
>>> failed to get machine password for account office.mydomain.net.:
>>> NT_STATUS_NONE_MAPPED
>>> Jun 05 21:08:02 dc1.mydomain.net winbindd[2171]: GSSAPI client step 1
>>> Jun 05 21:08:02 dc1.mydomain.net winbindd[2171]: GSSAPI client step 1
>>> Jun 05 21:08:23 dc1.mydomain.net winbindd[2171]: [2015/06/05
>>> 21:08:23.034001,  0] ipa_sam.c:4144(bind_callback_cleanup)
>>> Jun 05 21:08:23 dc1.mydomain.net winbindd[2171]: kerberos error:
>>> code=-1765328324, message=Generic error (see e-text)
>>> Jun 05 21:08:23 dc1.mydomain.net winbindd[2171]: GSSAPI client step 1
>>> Jun 05 21:08:23 dc1.mydomain.net winbindd[2171]: GSSAPI client step 1
>>>
>>> I also got this error from the web ui after restarting httpd:
>>>
>>> Runtime error
>>>
>>> Web UI got in unrecoverable state during "metadata" phase
>>>
>>>
>>> --
>>> Manage your subscription for the Freeipa-users mailing list:
>>> https://www.redhat.com/mailman/listinfo/freeipa-users
>>> Go to http://freeipa.org for more info on the project
>>>
>> Further information : restarting the httpd service didn't help, but
>> restarting the dirsrv service allowed me to once again login to the
>> webui
>> and the ipactl command started working again after the restart of
>> dirsrv.
>>
>> Is there something I can look for in my logs next time this happens. I
>> have a feeling it *will* happen again this is a critical server I'm in
>> charge of so it will not be good if I cannot come up with a solid
>> explanation or bug report on why this server spontaneously stops
>> working.
>>
>> [root at dc1 ~]# ipactl restart
>> (waiting 3 or 4 minutes with nothing happening)
>> ^CCancelled.
>> [root at dc1 ~]# systemctl restart dirsrv at MYDOMAIN-NET
>> [root at dc1 ~]# ipactl status
>> Directory Service: RUNNING
>> krb5kdc Service: RUNNING
>> kadmin Service: RUNNING
>> named Service: RUNNING
>> ipa_memcached Service: RUNNING
>> httpd Service: RUNNING
>> pki-tomcatd Service: RUNNING
>> smb Service: RUNNING
>> winbind Service: RUNNING
>> ipa-otpd Service: RUNNING
>> ipa-dnskeysyncd Service: RUNNING
>> ipa: INFO: The ipactl command was successful
>> [root at dc1 ~]#
>>
>> Here are some additional entries from my /var/log/dirsrv/slapd-MYDOMAIN
>> logs.  Strange error messages about non initialized replica.
>>
>> However, I know the windows machine is properly syncing data because I
>> have over 300 synced users and when I update them in AD the updated
>> attributes sync to IPA.
>
> Is it possible this is an old winsync agreement that is no longer valid?

I have only ever made a single winsync agreement on this server that I
know of.  How would I tell if an agreement is no longer valid?