<div dir="ltr"><div>All of the troubleshooting seems fine. </div><div>However, Running <a href="http://libconv.pl">libconv.pl</a> gives me this output ----- Recommendations ----- 1. You have unindexed components, this can be caused from a search on an unindexed attribute, or your returned results exceeded the allidsthreshold. Unindexed components are not recommended. To refuse unindexed searches, switch 'nsslapd-require-index' to 'on' under your database entry (e.g. cn=UserRoot,cn=ldbm database,cn=plugins,cn=config). 2. You have a significant difference between binds and unbinds. You may want to investigate this difference. </div><div> </div><div>I feel, this could be a pointer to things going slow.. and IPA hanging. I think i now have something that I can try and nail down this issue. On a sidenote, I was earlier running openldap and migrated over to Freeipa, </div><div>Thanks </div><div>Rakesh </div><div> </div></div><div class="gmail_extra"> <div class="gmail_quote">On Wed, Aug 24, 2016 at 12:38 PM, Petr Spacek <<a href="mailto:pspacek@redhat.com" target="_blank">pspacek@redhat.com</a>> wrote: <blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">On 23.8.2016 18:44, Rakesh Rajasekharan wrote: > I think thers something seriously wrong with my system > > not able to run any IPA commands > > klist > Ticket cache: KEYRING:persistent:0:0 > Default principal: <a href="mailto:admin@XYZ.COM">admin@XYZ.COM</a> > > Valid starting Expires Service principal > 2016-08-23T16:26:36 2016-08-24T16:26:22 krbtgt/<a href="mailto:XYZ.COM@XYZ.COM">XYZ.COM@XYZ.COM</a> > > > [root@prod-ipa-master-1a :~] ipactl status > Directory Service: RUNNING > krb5kdc Service: RUNNING > kadmin Service: RUNNING > ipa_memcached Service: RUNNING > httpd Service: RUNNING > pki-tomcatd Service: RUNNING > ipa-otpd Service: RUNNING > ipa: INFO: The ipactl command was successful > > > > [root@prod-ipa-master :~] ipa user-find p-testuser > ipa: ERROR: Kerberos error: ('Unspecified GSS failure. Minor code may > provide more information', 851968)/("Cannot contact any KDC for realm ' > <a href="http://XYZ.COM" rel="noreferrer" target="_blank">XYZ.COM</a>'", -1765328228) > This is weird because the server seems to be up. Please follow <a href="http://www.freeipa.org/page/Troubleshooting#Authentication.2FKerberos" rel="noreferrer" target="_blank">http://www.freeipa.org/page/Troubleshooting#Authentication.2FKerberos</a> Petr^2 Spacek <div class="HOEnZb"><div class="h5"> > > > Thanks > > Rakesh > > On Tue, Aug 23, 2016 at 10:01 PM, Rakesh Rajasekharan < > <a href="mailto:rakesh.rajasekharan@gmail.com">rakesh.rajasekharan@gmail.com</a>> wrote: > >> i changed the loggin level to 4 . Modifying nsslapd-accesslog-level >> >> But, the hang is still there. though I dont see the sigfault now >> >> >> >> >> On Tue, Aug 23, 2016 at 9:02 PM, Rakesh Rajasekharan < >> <a href="mailto:rakesh.rajasekharan@gmail.com">rakesh.rajasekharan@gmail.com</a>> wrote: >> >>> My disk was getting filled too fast >>> >>> logs under /var/log/dirsrv was coming around 5 gb quickly filling up >>> >>> Is there a way to make the logging less verbose >>> >>> >>> >>> On Tue, Aug 23, 2016 at 6:41 PM, Petr Spacek <<a href="mailto:pspacek@redhat.com">pspacek@redhat.com</a>> wrote: >>> >>>> On 23.8.2016 15:07, Rakesh Rajasekharan wrote: >>>>> I was able to fix that may be temporarily... when i checked the >>>> network.. >>>>> there was another process that was running and consuming a lot of >>>> network ( >>>>> i have no idea who did that. I need to seriously start restricting >>>> people >>>>> access to this machine ) >>>>> >>>>> after killing that perfomance improved drastically >>>>> >>>>> But now, suddenly I started experiencing the same hang. >>>>> >>>>> This time , I gert the following error when checked dmesg >>>>> >>>>> [ 301.236976] ns-slapd[3124]: segfault at 0 ip 00007f1de416951c sp >>>>> 00007f1dee1dba70 error 4 in libcos-plugin.so[7f1de4166000+b000] >>>>> [ 1116.248431] TCP: request_sock_TCP: Possible SYN flooding on port 88. >>>>> Sending cookies. Check SNMP counters. >>>>> [11831.397037] ns-slapd[22550]: segfault at 0 ip 00007f533d82251c sp >>>>> 00007f5347894a70 error 4 in libcos-plugin.so[7f533d81f000+b000] >>>>> [11832.727989] ns-slapd[22606]: segfault at 0 ip 00007f6231eb951c sp >>>>> 00007f623bf2ba70 error 4 in libcos-plugin.so[7f6231eb6000+b00 >>>> >>>> Okay, this one is serious. The LDAP server crashed. >>>> >>>> 1. Make sure all your packages are up-to-date. >>>> >>>> Please see >>>> <a href="http://directory.fedoraproject.org/docs/389ds/FAQ/faq.html#d" rel="noreferrer" target="_blank">http://directory.fedoraproject.org/docs/389ds/FAQ/faq.html#d</a> >>>> ebugging-crashes >>>> for further instructions how to debug this. >>>> >>>> Petr^2 Spacek >>>> >>>>> >>>>> and in /var/log/dirsrv/example-com/errors >>>>> >>>>> [23/Aug/2016:12:49:36 +0000] DSRetroclPlugin - delete_changerecord: >>>> could >>>>> not delete change record 3291138 (rc: 32) >>>>> [23/Aug/2016:12:49:36 +0000] DSRetroclPlugin - delete_changerecord: >>>> could >>>>> not delete change record 3291139 (rc: 32) >>>>> [23/Aug/2016:12:49:36 +0000] DSRetroclPlugin - delete_changerecord: >>>> could >>>>> not delete change record 3291140 (rc: 32) >>>>> [23/Aug/2016:12:49:36 +0000] DSRetroclPlugin - delete_changerecord: >>>> could >>>>> not delete change record 3291141 (rc: 32) >>>>> [23/Aug/2016:12:49:36 +0000] DSRetroclPlugin - delete_changerecord: >>>> could >>>>> not delete change record 3291142 (rc: 32) >>>>> [23/Aug/2016:12:49:36 +0000] DSRetroclPlugin - delete_changerecord: >>>> could >>>>> not delete change record 3291143 (rc: 32) >>>>> [23/Aug/2016:12:49:36 +0000] DSRetroclPlugin - delete_changerecord: >>>> could >>>>> not delete change record 3291144 (rc: 32) >>>>> [23/Aug/2016:12:49:36 +0000] DSRetroclPlugin - delete_changerecord: >>>> could >>>>> not delete change record 3291145 (rc: 32) >>>>> [23/Aug/2016:12:49:50 +0000] - Retry count exceeded in delete >>>>> [23/Aug/2016:12:49:50 +0000] DSRetroclPlugin - delete_changerecord: >>>> could >>>>> not delete change record 3292734 (rc: 51) >>>>> >>>>> >>>>> Can i do something about this error.. I treid to restart ipa a couple >>>> of >>>>> time but that did not help >>>>> >>>>> Thanks >>>>> Rakesh >>>>> >>>>> On Mon, Aug 22, 2016 at 2:27 PM, Petr Spacek <<a href="mailto:pspacek@redhat.com">pspacek@redhat.com</a>> >>>> wrote: >>>>> >>>>>> On 19.8.2016 19:32, Rakesh Rajasekharan wrote: >>>>>>> I am running my set up on AWS cloud, and entropy is low at around >>>> 180 . >>>>>>> >>>>>>> I plan to increase it bu installing haveged . But, would low entropy >>>> by >>>>>> any >>>>>>> chance cause this issue of intermittent hang . >>>>>>> Also, the hang is mostly observed when registering around 20 clients >>>>>>> together >>>>>> >>>>>> Possibly, I'm not sure. If you want to dig into this, I would do this: >>>>>> 1. look what process hangs on client (using pstree command or so) >>>>>> $ pstree >>>>>> >>>>>> 2. look to what server and port is the hanging client connected to >>>>>> $ lsof -p <PID of the hanging process> >>>>>> >>>>>> 3. jump to server and see what process is bound to the target port >>>>>> $ netstat -pn >>>>>> >>>>>> 4. see where the process if hanging >>>>>> $ strace -p <PID of the hanging process> >>>>>> >>>>>> I hope it helps. >>>>>> >>>>>> Petr^2 Spacek >>>>>> >>>>>>> On Fri, Aug 19, 2016 at 7:24 PM, Rakesh Rajasekharan < >>>>>>> <a href="mailto:rakesh.rajasekharan@gmail.com">rakesh.rajasekharan@gmail.com</a>> wrote: >>>>>>> >>>>>>>> yes there seems to be something thats worrying.. I have faced this >>>> today >>>>>>>> as well. >>>>>>>> There are few hosts around 280 odd left and when i try adding them >>>> to >>>>>> IPA >>>>>>>> , the slowness begins.. >>>>>>>> >>>>>>>> all the ipa commands like ipa user-find.. etc becomes very slow in >>>>>>>> responding. >>>>>>>> >>>>>>>> the SYNC_RECV are not many though just around 80-90 and today that >>>> was >>>>>>>> around 20 only >>>>>>>> >>>>>>>> >>>>>>>> I have for now increased tcp_max_syn_backlog to 5000. >>>>>>>> For now the slowness seems to have gone.. but I will do a try >>>> adding the >>>>>>>> clients again tomorrow and see how it goes >>>>>>>> >>>>>>>> Thanks >>>>>>>> Rakesh >>>>>>>> >>>>>>>> The issues >>>>>>>> >>>>>>>> On Fri, Aug 19, 2016 at 12:58 PM, Petr Spacek <<a href="mailto:pspacek@redhat.com">pspacek@redhat.com</a>> >>>>>> wrote: >>>>>>>> >>>>>>>>> On 18.8.2016 17:23, Rakesh Rajasekharan wrote: >>>>>>>>>> Hi >>>>>>>>>> >>>>>>>>>> I am migrating to freeipa from openldap and have around 4000 >>>> clients >>>>>>>>>> >>>>>>>>>> I had openned a another thread on that, but chose to start a new >>>> one >>>>>>>>> here >>>>>>>>>> as its a separate issue >>>>>>>>>> >>>>>>>>>> I was able to change the nssslapd-maxdescriptors adding an ldif >>>> file >>>>>>>>>> >>>>>>>>>> cat nsslapd-modify.ldif >>>>>>>>>> dn: cn=config >>>>>>>>>> changetype: modify >>>>>>>>>> replace: nsslapd-maxdescriptors >>>>>>>>>> nsslapd-maxdescriptors: 17000 >>>>>>>>>> >>>>>>>>>> and running the ldapmodify command >>>>>>>>>> >>>>>>>>>> I have now started moving clients running an openldap to Freeipa >>>> and >>>>>>>>> have >>>>>>>>>> today moved close to 2000 clients >>>>>>>>>> >>>>>>>>>> However, I have noticed that IPA hangs intermittently. >>>>>>>>>> >>>>>>>>>> running a kinit admin returns the below error >>>>>>>>>> kinit: Generic error (see e-text) while getting initial >>>> credentials >>>>>>>>>> >>>>>>>>>> from the /var/log/messages, I see this entry >>>>>>>>>> >>>>>>>>>> prod-ipa-master-int kernel: [104090.315801] TCP: >>>> request_sock_TCP: >>>>>>>>>> Possible SYN flooding on port 88. Sending cookies. Check SNMP >>>>>> counters. >>>>>>>>> >>>>>>>>> I would be worried about this message. Maybe kernel/firewall is >>>> doing >>>>>>>>> something fishy behind your back and blocking some connections or >>>> so. >>>>>>>>> >>>>>>>>> Petr^2 Spacek >>>>>>>>> >>>>>>>>> >>>>>>>>>> Aug 18 13:00:01 prod-ipa-master-int systemd[1]: Started Session >>>> 4885 >>>>>> of >>>>>>>>>> user root. >>>>>>>>>> Aug 18 13:00:01 prod-ipa-master-int systemd[1]: Starting Session >>>> 4885 >>>>>> of >>>>>>>>>> user root. >>>>>>>>>> Aug 18 13:01:01 prod-ipa-master-int systemd[1]: Started Session >>>> 4886 >>>>>> of >>>>>>>>>> user root. >>>>>>>>>> Aug 18 13:01:01 prod-ipa-master-int systemd[1]: Starting Session >>>> 4886 >>>>>> of >>>>>>>>>> user root. >>>>>>>>>> Aug 18 13:02:40 prod-ipa-master-int python[28984]: ansible-command >>>>>>>>> Invoked >>>>>>>>>> with creates=None executable=None shell=True args= removes=None >>>>>>>>> warn=True >>>>>>>>>> chdir=None >>>>>>>>>> Aug 18 13:04:37 prod-ipa-master-int sssd_be: GSSAPI Error: >>>> Unspecified >>>>>>>>> GSS >>>>>>>>>> failure. Minor code may provide more information (KDC returned >>>> error >>>>>>>>>> string: PROCESS_TGS) >>>>>>>>>> >>>>>>>>>> Could it be possible that its due to the initial load of adding >>>> the >>>>>>>>> clients >>>>>>>>>> or is there something else that I need to take care of. </div></div></blockquote></div> </div>