[Freeipa-users] Freeipa 4.2.0 hangs intermittently

Tue Aug 23 15:32:54 UTC 2016

My disk was getting filled too fast

logs under /var/log/dirsrv was coming around 5 gb quickly filling up

Is there a way to make the logging less verbose

On Tue, Aug 23, 2016 at 6:41 PM, Petr Spacek <pspacek at redhat.com> wrote:

> On 23.8.2016 15:07, Rakesh Rajasekharan wrote:
> > I was able to fix that may be temporarily... when i checked the network..
> > there was another process that was running and consuming a lot of
> network (
> > i have no idea who did that. I need to seriously start restricting people
> > access to this machine )
> >
> > after killing that perfomance improved drastically
> >
> > But now, suddenly I started experiencing the same hang.
> >
> > This time , I gert the following error when checked dmesg
> >
> > [  301.236976] ns-slapd[3124]: segfault at 0 ip 00007f1de416951c sp
> > 00007f1dee1dba70 error 4 in libcos-plugin.so[7f1de4166000+b000]
> > [ 1116.248431] TCP: request_sock_TCP: Possible SYN flooding on port 88.
> > Sending cookies.  Check SNMP counters.
> > [11831.397037] ns-slapd[22550]: segfault at 0 ip 00007f533d82251c sp
> > 00007f5347894a70 error 4 in libcos-plugin.so[7f533d81f000+b000]
> > [11832.727989] ns-slapd[22606]: segfault at 0 ip 00007f6231eb951c sp
> > 00007f623bf2ba70 error 4 in libcos-plugin.so[7f6231eb6000+b00
>
> Okay, this one is serious. The LDAP server crashed.
>
> 1. Make sure all your packages are up-to-date.
>
> Please see
> http://directory.fedoraproject.org/docs/389ds/
> FAQ/faq.html#debugging-crashes
> for further instructions how to debug this.
>
> Petr^2 Spacek
>
> >
> > and in /var/log/dirsrv/example-com/errors
> >
> > [23/Aug/2016:12:49:36 +0000] DSRetroclPlugin - delete_changerecord: could
> > not delete change record 3291138 (rc: 32)
> > [23/Aug/2016:12:49:36 +0000] DSRetroclPlugin - delete_changerecord: could
> > not delete change record 3291139 (rc: 32)
> > [23/Aug/2016:12:49:36 +0000] DSRetroclPlugin - delete_changerecord: could
> > not delete change record 3291140 (rc: 32)
> > [23/Aug/2016:12:49:36 +0000] DSRetroclPlugin - delete_changerecord: could
> > not delete change record 3291141 (rc: 32)
> > [23/Aug/2016:12:49:36 +0000] DSRetroclPlugin - delete_changerecord: could
> > not delete change record 3291142 (rc: 32)
> > [23/Aug/2016:12:49:36 +0000] DSRetroclPlugin - delete_changerecord: could
> > not delete change record 3291143 (rc: 32)
> > [23/Aug/2016:12:49:36 +0000] DSRetroclPlugin - delete_changerecord: could
> > not delete change record 3291144 (rc: 32)
> > [23/Aug/2016:12:49:36 +0000] DSRetroclPlugin - delete_changerecord: could
> > not delete change record 3291145 (rc: 32)
> > [23/Aug/2016:12:49:50 +0000] - Retry count exceeded in delete
> > [23/Aug/2016:12:49:50 +0000] DSRetroclPlugin - delete_changerecord: could
> > not delete change record 3292734 (rc: 51)
> >
> >
> > Can  i do something about this error.. I treid to restart ipa a couple of
> > time but that did not help
> >
> > Thanks
> > Rakesh
> >
> > On Mon, Aug 22, 2016 at 2:27 PM, Petr Spacek <pspacek at redhat.com> wrote:
> >
> >> On 19.8.2016 19:32, Rakesh Rajasekharan wrote:
> >>> I am running my set up on AWS cloud, and entropy is low at around 180 .
> >>>
> >>> I plan to increase it bu installing haveged . But, would low entropy by
> >> any
> >>> chance cause this issue of intermittent hang .
> >>> Also, the hang is mostly observed when registering around 20 clients
> >>> together
> >>
> >> Possibly, I'm not sure. If you want to dig into this, I would do this:
> >> 1. look what process hangs on client (using pstree command or so)
> >> $ pstree
> >>
> >> 2. look to what server and port is the hanging client connected to
> >> $ lsof -p <PID of the hanging process>
> >>
> >> 3. jump to server and see what process is bound to the target port
> >> $ netstat -pn
> >>
> >> 4. see where the process if hanging
> >> $ strace -p <PID of the hanging process>
> >>
> >> I hope it helps.
> >>
> >> Petr^2 Spacek
> >>
> >>> On Fri, Aug 19, 2016 at 7:24 PM, Rakesh Rajasekharan <
> >>> rakesh.rajasekharan at gmail.com> wrote:
> >>>
> >>>> yes there seems to be something thats worrying.. I have faced this
> today
> >>>> as well.
> >>>> There are few hosts around 280 odd left and when i try adding them to
> >> IPA
> >>>> , the slowness begins..
> >>>>
> >>>> all the ipa commands like ipa user-find.. etc becomes very slow in
> >>>> responding.
> >>>>
> >>>> the SYNC_RECV are not many though just around 80-90 and today that was
> >>>> around 20 only
> >>>>
> >>>>
> >>>> I have for now increased tcp_max_syn_backlog to 5000.
> >>>> For now the slowness seems to have gone.. but I will do a try adding
> the
> >>>> clients again tomorrow and see how it goes
> >>>>
> >>>> Thanks
> >>>> Rakesh
> >>>>
> >>>> The issues
> >>>>
> >>>> On Fri, Aug 19, 2016 at 12:58 PM, Petr Spacek <pspacek at redhat.com>
> >> wrote:
> >>>>
> >>>>> On 18.8.2016 17:23, Rakesh Rajasekharan wrote:
> >>>>>> Hi
> >>>>>>
> >>>>>> I am migrating to freeipa from openldap and have around 4000 clients
> >>>>>>
> >>>>>> I had openned a another thread on that, but chose to start a new one
> >>>>> here
> >>>>>> as its a separate issue
> >>>>>>
> >>>>>> I was able to change the nssslapd-maxdescriptors adding an ldif file
> >>>>>>
> >>>>>> cat nsslapd-modify.ldif
> >>>>>> dn: cn=config
> >>>>>> changetype: modify
> >>>>>> replace: nsslapd-maxdescriptors
> >>>>>> nsslapd-maxdescriptors: 17000
> >>>>>>
> >>>>>> and running the ldapmodify command
> >>>>>>
> >>>>>> I have now started moving clients running an openldap to Freeipa and
> >>>>> have
> >>>>>> today moved close to 2000 clients
> >>>>>>
> >>>>>> However, I have noticed that IPA hangs intermittently.
> >>>>>>
> >>>>>> running a kinit admin returns the below error
> >>>>>> kinit: Generic error (see e-text) while getting initial credentials
> >>>>>>
> >>>>>> from the /var/log/messages, I see this entry
> >>>>>>
> >>>>>>  prod-ipa-master-int kernel: [104090.315801] TCP: request_sock_TCP:
> >>>>>> Possible SYN flooding on port 88. Sending cookies.  Check SNMP
> >> counters.
> >>>>>
> >>>>> I would be worried about this message. Maybe kernel/firewall is doing
> >>>>> something fishy behind your back and blocking some connections or so.
> >>>>>
> >>>>> Petr^2 Spacek
> >>>>>
> >>>>>
> >>>>>> Aug 18 13:00:01 prod-ipa-master-int systemd[1]: Started Session 4885
> >> of
> >>>>>> user root.
> >>>>>> Aug 18 13:00:01 prod-ipa-master-int systemd[1]: Starting Session
> 4885
> >> of
> >>>>>> user root.
> >>>>>> Aug 18 13:01:01 prod-ipa-master-int systemd[1]: Started Session 4886
> >> of
> >>>>>> user root.
> >>>>>> Aug 18 13:01:01 prod-ipa-master-int systemd[1]: Starting Session
> 4886
> >> of
> >>>>>> user root.
> >>>>>> Aug 18 13:02:40 prod-ipa-master-int python[28984]: ansible-command
> >>>>> Invoked
> >>>>>> with creates=None executable=None shell=True args= removes=None
> >>>>> warn=True
> >>>>>> chdir=None
> >>>>>> Aug 18 13:04:37 prod-ipa-master-int sssd_be: GSSAPI Error:
> Unspecified
> >>>>> GSS
> >>>>>> failure.  Minor code may provide more information (KDC returned
> error
> >>>>>> string: PROCESS_TGS)
> >>>>>>
> >>>>>> Could it be possible that its due to the initial load of adding the
> >>>>> clients
> >>>>>> or is there something else that I need to take care of.
> >>
> >
>
>
> --
> Petr Spacek  @  Red Hat
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/freeipa-users/attachments/20160823/121c1ed8/attachment.htm>