[Freeipa-users] Freeipa 4.2.0 hangs intermittently

Tue Aug 23 13:11:45 UTC 2016

On 23.8.2016 15:07, Rakesh Rajasekharan wrote:
> I was able to fix that may be temporarily... when i checked the network..
> there was another process that was running and consuming a lot of network (
> i have no idea who did that. I need to seriously start restricting people
> access to this machine )
> 
> after killing that perfomance improved drastically
> 
> But now, suddenly I started experiencing the same hang.
> 
> This time , I gert the following error when checked dmesg
> 
> [  301.236976] ns-slapd[3124]: segfault at 0 ip 00007f1de416951c sp
> 00007f1dee1dba70 error 4 in libcos-plugin.so[7f1de4166000+b000]
> [ 1116.248431] TCP: request_sock_TCP: Possible SYN flooding on port 88.
> Sending cookies.  Check SNMP counters.
> [11831.397037] ns-slapd[22550]: segfault at 0 ip 00007f533d82251c sp
> 00007f5347894a70 error 4 in libcos-plugin.so[7f533d81f000+b000]
> [11832.727989] ns-slapd[22606]: segfault at 0 ip 00007f6231eb951c sp
> 00007f623bf2ba70 error 4 in libcos-plugin.so[7f6231eb6000+b00

Okay, this one is serious. The LDAP server crashed.

1. Make sure all your packages are up-to-date.

Please see
http://directory.fedoraproject.org/docs/389ds/FAQ/faq.html#debugging-crashes
for further instructions how to debug this.

Petr^2 Spacek

> 
> and in /var/log/dirsrv/example-com/errors
> 
> [23/Aug/2016:12:49:36 +0000] DSRetroclPlugin - delete_changerecord: could
> not delete change record 3291138 (rc: 32)
> [23/Aug/2016:12:49:36 +0000] DSRetroclPlugin - delete_changerecord: could
> not delete change record 3291139 (rc: 32)
> [23/Aug/2016:12:49:36 +0000] DSRetroclPlugin - delete_changerecord: could
> not delete change record 3291140 (rc: 32)
> [23/Aug/2016:12:49:36 +0000] DSRetroclPlugin - delete_changerecord: could
> not delete change record 3291141 (rc: 32)
> [23/Aug/2016:12:49:36 +0000] DSRetroclPlugin - delete_changerecord: could
> not delete change record 3291142 (rc: 32)
> [23/Aug/2016:12:49:36 +0000] DSRetroclPlugin - delete_changerecord: could
> not delete change record 3291143 (rc: 32)
> [23/Aug/2016:12:49:36 +0000] DSRetroclPlugin - delete_changerecord: could
> not delete change record 3291144 (rc: 32)
> [23/Aug/2016:12:49:36 +0000] DSRetroclPlugin - delete_changerecord: could
> not delete change record 3291145 (rc: 32)
> [23/Aug/2016:12:49:50 +0000] - Retry count exceeded in delete
> [23/Aug/2016:12:49:50 +0000] DSRetroclPlugin - delete_changerecord: could
> not delete change record 3292734 (rc: 51)
> 
> 
> Can  i do something about this error.. I treid to restart ipa a couple of
> time but that did not help
> 
> Thanks
> Rakesh
> 
> On Mon, Aug 22, 2016 at 2:27 PM, Petr Spacek <pspacek at redhat.com> wrote:
> 
>> On 19.8.2016 19:32, Rakesh Rajasekharan wrote:
>>> I am running my set up on AWS cloud, and entropy is low at around 180 .
>>>
>>> I plan to increase it bu installing haveged . But, would low entropy by
>> any
>>> chance cause this issue of intermittent hang .
>>> Also, the hang is mostly observed when registering around 20 clients
>>> together
>>
>> Possibly, I'm not sure. If you want to dig into this, I would do this:
>> 1. look what process hangs on client (using pstree command or so)
>> $ pstree
>>
>> 2. look to what server and port is the hanging client connected to
>> $ lsof -p <PID of the hanging process>
>>
>> 3. jump to server and see what process is bound to the target port
>> $ netstat -pn
>>
>> 4. see where the process if hanging
>> $ strace -p <PID of the hanging process>
>>
>> I hope it helps.
>>
>> Petr^2 Spacek
>>
>>> On Fri, Aug 19, 2016 at 7:24 PM, Rakesh Rajasekharan <
>>> rakesh.rajasekharan at gmail.com> wrote:
>>>
>>>> yes there seems to be something thats worrying.. I have faced this today
>>>> as well.
>>>> There are few hosts around 280 odd left and when i try adding them to
>> IPA
>>>> , the slowness begins..
>>>>
>>>> all the ipa commands like ipa user-find.. etc becomes very slow in
>>>> responding.
>>>>
>>>> the SYNC_RECV are not many though just around 80-90 and today that was
>>>> around 20 only
>>>>
>>>>
>>>> I have for now increased tcp_max_syn_backlog to 5000.
>>>> For now the slowness seems to have gone.. but I will do a try adding the
>>>> clients again tomorrow and see how it goes
>>>>
>>>> Thanks
>>>> Rakesh
>>>>
>>>> The issues
>>>>
>>>> On Fri, Aug 19, 2016 at 12:58 PM, Petr Spacek <pspacek at redhat.com>
>> wrote:
>>>>
>>>>> On 18.8.2016 17:23, Rakesh Rajasekharan wrote:
>>>>>> Hi
>>>>>>
>>>>>> I am migrating to freeipa from openldap and have around 4000 clients
>>>>>>
>>>>>> I had openned a another thread on that, but chose to start a new one
>>>>> here
>>>>>> as its a separate issue
>>>>>>
>>>>>> I was able to change the nssslapd-maxdescriptors adding an ldif file
>>>>>>
>>>>>> cat nsslapd-modify.ldif
>>>>>> dn: cn=config
>>>>>> changetype: modify
>>>>>> replace: nsslapd-maxdescriptors
>>>>>> nsslapd-maxdescriptors: 17000
>>>>>>
>>>>>> and running the ldapmodify command
>>>>>>
>>>>>> I have now started moving clients running an openldap to Freeipa and
>>>>> have
>>>>>> today moved close to 2000 clients
>>>>>>
>>>>>> However, I have noticed that IPA hangs intermittently.
>>>>>>
>>>>>> running a kinit admin returns the below error
>>>>>> kinit: Generic error (see e-text) while getting initial credentials
>>>>>>
>>>>>> from the /var/log/messages, I see this entry
>>>>>>
>>>>>>  prod-ipa-master-int kernel: [104090.315801] TCP: request_sock_TCP:
>>>>>> Possible SYN flooding on port 88. Sending cookies.  Check SNMP
>> counters.
>>>>>
>>>>> I would be worried about this message. Maybe kernel/firewall is doing
>>>>> something fishy behind your back and blocking some connections or so.
>>>>>
>>>>> Petr^2 Spacek
>>>>>
>>>>>
>>>>>> Aug 18 13:00:01 prod-ipa-master-int systemd[1]: Started Session 4885
>> of
>>>>>> user root.
>>>>>> Aug 18 13:00:01 prod-ipa-master-int systemd[1]: Starting Session 4885
>> of
>>>>>> user root.
>>>>>> Aug 18 13:01:01 prod-ipa-master-int systemd[1]: Started Session 4886
>> of
>>>>>> user root.
>>>>>> Aug 18 13:01:01 prod-ipa-master-int systemd[1]: Starting Session 4886
>> of
>>>>>> user root.
>>>>>> Aug 18 13:02:40 prod-ipa-master-int python[28984]: ansible-command
>>>>> Invoked
>>>>>> with creates=None executable=None shell=True args= removes=None
>>>>> warn=True
>>>>>> chdir=None
>>>>>> Aug 18 13:04:37 prod-ipa-master-int sssd_be: GSSAPI Error: Unspecified
>>>>> GSS
>>>>>> failure.  Minor code may provide more information (KDC returned error
>>>>>> string: PROCESS_TGS)
>>>>>>
>>>>>> Could it be possible that its due to the initial load of adding the
>>>>> clients
>>>>>> or is there something else that I need to take care of.
>>
> 

-- 
Petr Spacek  @  Red Hat