<html>
  <head>
    <meta content="text/html; charset=utf-8" http-equiv="Content-Type">
  </head>
  <body bgcolor="#FFFFFF" text="#000000">
    <div class="moz-cite-prefix">On 11/12/2014 05:42 AM, Walter van
      Lille wrote:<br>
    </div>
    <blockquote
cite="mid:CAMqGCT8OJojr+a_3-cPfKX=mHdWMVfOA1bDCb2EfKv+5mC8osA@mail.gmail.com"
      type="cite">
      <div dir="ltr">Thanks again for the assistance guys. I have saved
        two files and included it here. Hope it tells you more than it
        does me.</div>
    </blockquote>
    <br>
    These stack traces contain no useful symbols.<br>
    <br>
    Please read
    <a class="moz-txt-link-freetext" href="http://www.port389.org/docs/389ds/FAQ/faq.html#debugging-crashes">http://www.port389.org/docs/389ds/FAQ/faq.html#debugging-crashes</a> to
    find out how to install the correct debuginfo packages on your
    system.  For IPA you will also need debuginfo packages for ipa and
    slapi-nis<br>
    <br>
    If debuginfo-install is not working, see
    <a class="moz-txt-link-freetext" href="https://www.centos.org/forums/viewtopic.php?t=1710">https://www.centos.org/forums/viewtopic.php?t=1710</a><br>
    <br>
    If you simply cannot figure out how to install the debuginfo
    packages, please email me with the following information:<br>
    <br>
    $ rpm -q ipa-server slapi-nis 389-ds-base openldap db4 nss nspr
    glibc<br>
    <br>
    and I will make them available to you<br>
    <br>
    NOTE: With debuginfo packages, the version of the debuginfo package
    _must exactly match_ the version of the associated package e.g. for
    <span class="">389-ds-base.x86_64   1.2.11.15-34.el6_5 you must have
    </span><br>
    <span class="">389-ds-base-debuginfo.x86_64   1.2.11.15-34.el6_5<br>
      <br>
      If the versions do not match exactly, you will not get the
      symbols.<br>
      <br>
    </span>
    <blockquote
cite="mid:CAMqGCT8OJojr+a_3-cPfKX=mHdWMVfOA1bDCb2EfKv+5mC8osA@mail.gmail.com"
      type="cite">
      <div dir="ltr">
        <div><br>
        </div>
        <div>Regards,</div>
        <div><br>
        </div>
        <div>Wallter</div>
      </div>
      <div class="gmail_extra"><br>
        <div class="gmail_quote">On Tue, Nov 11, 2014 at 8:03 PM, Rich
          Megginson <span dir="ltr"><<a moz-do-not-send="true"
              href="mailto:rmeggins@redhat.com" target="_blank">rmeggins@redhat.com</a>></span>
          wrote:<br>
          <blockquote class="gmail_quote" style="margin:0 0 0
            .8ex;border-left:1px #ccc solid;padding-left:1ex">
            <div bgcolor="#FFFFFF" text="#000000"><span class="">
                <div>On 11/11/2014 10:37 AM, Martin Basti wrote:<br>
                </div>
                <blockquote type="cite">
                  <div>On 11/11/14 15:58, Rich Megginson wrote:<br>
                  </div>
                  <blockquote type="cite">
                    <div>On 11/11/2014 06:20 AM, Ludwig Krispenz wrote:<br>
                    </div>
                    <blockquote type="cite"> <br>
                      <div>On 11/11/2014 02:14 PM, Martin Basti wrote:<br>
                      </div>
                      <blockquote type="cite">
                        <div>Ludiwg (CCed) this seems like old (fixed?)
                          DS bug.<br>
                        </div>
                      </blockquote>
                      hmm, it says limit is 2097152, so it already has
                      the new setting, but the error message says the
                      packet is 800MB<b><br>
                      </b></blockquote>
                    <br>
                    <b>Right.  That usually means the server was
                      expecting an encrypted SASL buffer from the
                      client, but instead the client thinks SASL
                      encryption negotiation failed and just sent a
                      plain LDAP buffer.  What version of 389-ds-base
                      are you using?  rpm -q 389-ds-base<br>
                      <br>
                      <a moz-do-not-send="true"
                        href="https://fedorahosted.org/389/ticket/47416"
                        target="_blank">https://fedorahosted.org/389/ticket/47416</a><br>
                      <br>
                      So, DO NOT increase your sasl io buffer size - it
                      will not fix the problem, and it will leave you
                      open to DoS attacks.<br>
                    </b></blockquote>
                  <br>
                  He is using <br>
                  <div><br>
                  </div>
                  <div>CentOS release 6.5 (Final)<br>
                  </div>
                  <div>389-ds-base.x86_64   1.2.11.15-34.el6_5</div>
                </blockquote>
                <br>
                <br>
              </span> Ignore the "SASL encrypted packet length exceeds
              maximum allowed limit" error.  All it means is the client
              has some error and is canceling the connection.<br>
              <br>
              The bugzilla associated with <b><a moz-do-not-send="true"
                  href="https://fedorahosted.org/389/ticket/47416"
                  target="_blank">47416</a> is targeted for RHEL 7.1. 
                The problem is that we were never able to figure out
                what error the client was getting that caused the client
                to stop establishing the </b>SASL encryption, and the
              original customer who reported the problem dropped the
              case - everything just started working, no further
              errors.  Note that 47416 doesn't really fix the problem or
              address the root cause - the root cause is some error on
              the client side that causes it to stop doing encrypted
              SASL I/O and send an LDAP unbind request.<br>
              <br>
              Let's get back to the actual problem - what is the actual
              problem?  The ldap server is hanging or unresponsive?  If
              so, then see <a moz-do-not-send="true"
                href="http://www.port389.org/docs/389ds/FAQ/faq.html#debugging-hangs"
                target="_blank">http://www.port389.org/docs/389ds/FAQ/faq.html#debugging-hangs</a>. 

              Let's get some dirsrv stack traces during the period of
              time when it appears to be unresponsive.
              <div>
                <div class="h5"><br>
                  <br>
                  <blockquote type="cite"> <br>
                    <blockquote type="cite"><b> <br>
                      </b>
                      <blockquote type="cite"><b> </b>
                        <blockquote type="cite">
                          <div> <br>
                            On 11/11/14 13:13, Walter van Lille wrote:<br>
                          </div>
                          <blockquote type="cite">
                            <div dir="ltr">
                              <div>
                                <div>I've just cleaned out a ton of
                                  slapd_poll timed out messages from the
                                  output and changed the names to
                                  protect the innocent, :-)</div>
                                <div>Here is the output as requested:</div>
                                <div><br>
                                </div>
                                <div><br>
                                </div>
                                <div><b>[05/Nov/2014:11:44:05 +0200] -
                                    SASL encrypted packet length exceeds
                                    maximum allowed limit
                                    (length=805565, limit=2097152). 
                                    Change the nsslapd-maxsasliosize
                                    attribute in cn=config to increase
                                    limit.</b></div>
                              </div>
                              <div><b><br>
                                </b></div>
                              <div><b>[10/Nov/2014:14:45:19 +0200] -
                                  slapd_poll(115) timed out</b></div>
                              <div><b>[10/Nov/2014:14:45:19 +0200]
                                  sasl_io_enable - Cannot enable SASL
                                  security on connection in CLOSING
                                  state</b></div>
                              <div><b>[10/Nov/2014:14:45:19 +0200] -
                                  Error: could not add/remove IO layers
                                  from connection</b></div>
                              <div>
                                <div><b>[11/Nov/2014:11:48:09 +0200] -
                                    slapd shutting down - signaling
                                    operation threads</b></div>
                                <div><b>[11/Nov/2014:11:48:09 +0200] -
                                    slapd shutting down - waiting for 30
                                    threads to terminate</b></div>
                              </div>
                              <div>
                                <div><b>[11/Nov/2014:13:14:12 +0200] -
                                    slapd shutting down - closing down
                                    internal subsystems and plugins</b></div>
                                <div><b>[11/Nov/2014:13:14:12 +0200] -
                                    Waiting for 4 database threads to
                                    stop</b></div>
                                <div><b>[11/Nov/2014:13:14:13 +0200] -
                                    All database threads now stopped</b></div>
                                <div><b>[11/Nov/2014:13:14:13 +0200] -
                                    slapd stopped.</b></div>
                                <div><b>[11/Nov/2014:13:26:35 +0200] -
                                    389-Directory/<a
                                      moz-do-not-send="true"
                                      href="http://1.2.11.15"
                                      target="_blank">1.2.11.15</a>
                                    B2014.219.179 starting up</b></div>
                                <div><b>[11/Nov/2014:13:26:35 +0200]
                                    schema-compat-plugin - warning: no
                                    entries set up under cn=computers,
                                    cn=compat,dc=sample,dc=example</b></div>
                                <div><b>[11/Nov/2014:13:26:36 +0200] -
                                    Skipping CoS Definition cn=Password
                                    Policy,cn=accounts,dc=sample,dc=example--no
                                    CoS Templates found, which should be
                                    added before the CoS Definition.</b></div>
                                <div><b>[11/Nov/2014:13:26:36 +0200] -
                                    Skipping CoS Definition cn=Password
                                    Policy,cn=accounts,dc=sample,dc=example--no
                                    CoS Templates found, which should be
                                    added before the CoS Definition.</b></div>
                                <div><b>[11/Nov/2014:13:26:36 +0200] -
                                    slapd started.  Listening on All
                                    Interfaces port 389 for LDAP
                                    requests</b></div>
                                <div><b>[11/Nov/2014:13:26:36 +0200] -
                                    Listening on All Interfaces port 636
                                    for LDAPS requests</b></div>
                                <div><b>[11/Nov/2014:13:26:36 +0200] -
                                    Listening on
                                    /var/run/slapd-SAMPLE-EXAMPLE.socket
                                    for LDAPI requests</b></div>
                                <div><b>[11/Nov/2014:13:57:08 +0200] -
                                    slapd_poll(78) timed out</b></div>
                              </div>
                              <div><b><br>
                                </b></div>
                              <div><b><br>
                                </b></div>
                              <div><b><br>
                                </b></div>
                              <div><br>
                              </div>
                              <div class="gmail_extra"><br>
                                <div class="gmail_quote">On Tue, Nov 11,
                                  2014 at 1:19 PM, Martin Basti <span
                                    dir="ltr"><<a
                                      moz-do-not-send="true"
                                      href="mailto:mbasti@redhat.com"
                                      target="_blank">mbasti@redhat.com</a>></span>
                                  wrote:<br>
                                  <blockquote class="gmail_quote"
                                    style="margin:0 0 0
                                    .8ex;border-left:1px #ccc
                                    solid;padding-left:1ex">
                                    <div bgcolor="#FFFFFF"
                                      text="#000000">
                                      <div>IMHO It's DS bug, can you
                                        share DS error log?<br>
                                        pspacek CCed to examine named
                                        logs.<br>
                                        <br>
                                        Martin^2
                                        <div>
                                          <div><br>
                                            <br>
                                            On 11/11/14 12:13, Walter
                                            van Lille wrote:<br>
                                          </div>
                                        </div>
                                      </div>
                                      <div>
                                        <div>
                                          <blockquote type="cite">
                                            <div dir="ltr">Hi Martin,
                                              thanks for the reply.
                                              <div>My version:
                                                bind-dyndb-ldap-2.3-5.el6.x86_64</div>
                                              <div>The server doesn't
                                                have journalctl
                                                installed but I have the
                                                outputs from the
                                                messages and named.run
                                                files that I included
                                                here:</div>
                                              <div><br>
                                              </div>
                                              <div>Messages:</div>
                                              <div><br>
                                              </div>
                                              <div>
                                                <div><b>Nov 11 12:30:13
                                                    freeipa named[1481]:
                                                    error (network
                                                    unreachable)
                                                    resolving
                                                    'example.example.com.10.123.123.123/A/IN':
                                                    2001:500:2f::f#53</b></div>
                                                <div><b>Nov 11 12:30:23
                                                    freeipa named[1481]:
                                                    LDAP query timed
                                                    out. Try to adjust
                                                    "timeout" parameter</b></div>
                                                <div><b>Nov 11 12:30:23
                                                    freeipa named[1481]:
                                                    LDAP query timed
                                                    out. Try to adjust
                                                    "timeout" parameter</b></div>
                                                <div><b>Nov 11 12:30:33
                                                    freeipa named[1481]:
                                                    LDAP query timed
                                                    out. Try to adjust
                                                    "timeout" parameter</b></div>
                                                <div><b>Nov 11 12:30:33
                                                    freeipa named[1481]:
                                                    LDAP query timed
                                                    out. Try to adjust
                                                    "timeout" parameter</b></div>
                                              </div>
                                              <div><br>
                                              </div>
                                              <div>Named.run:</div>
                                              <div><br>
                                              </div>
                                              <div>
                                                <div><b>client
                                                    10.123.123.123#42639:
                                                    transfer of
                                                    'example.example/IN':
                                                    AXFR-style IXFR
                                                    started</b></div>
                                                <div><b>client
                                                    10.123.123.123#42639:
                                                    transfer of
                                                    ''example.example/IN':
                                                    AXFR-style IXFR
                                                    ended</b></div>
                                                <div><b>client
                                                    10.123.123.123#46912:
                                                    transfer of
                                                    '10.123.123.123.in-addr.arpa/IN':
                                                    AXFR-style IXFR
                                                    started</b></div>
                                                <div><b>client
                                                    10.123.123.123#46912:
                                                    transfer of
                                                    '10.123.123.123.in-addr.arpa/IN':
                                                    AXFR-style IXFR
                                                    ended</b></div>
                                                <div><b>LDAP query timed
                                                    out. Try to adjust
                                                    "timeout" parameter</b></div>
                                                <div><b>LDAP query timed
                                                    out. Try to adjust
                                                    "timeout" parameter</b></div>
                                                <div><b>LDAP query timed
                                                    out. Try to adjust
                                                    "timeout" parameter</b></div>
                                              </div>
                                              <div><br>
                                              </div>
                                              <div>I just replaced the
                                                IPs and the actual names
                                                with something more
                                                generic.</div>
                                              <div><br>
                                              </div>
                                              <div>Regards,</div>
                                              <div><br>
                                              </div>
                                              <div>Walter</div>
                                              <div class="gmail_extra"><br>
                                                <div class="gmail_quote">On
                                                  Thu, Nov 6, 2014 at
                                                  5:00 PM, Martin Basti
                                                  <span dir="ltr"><<a
moz-do-not-send="true" href="mailto:mbasti@redhat.com" target="_blank">mbasti@redhat.com</a>></span>
                                                  wrote:<br>
                                                  <blockquote
                                                    class="gmail_quote"
                                                    style="margin:0 0 0
                                                    .8ex;border-left:1px
                                                    #ccc
                                                    solid;padding-left:1ex">
                                                    <div
                                                      bgcolor="#FFFFFF"
                                                      text="#000000">
                                                      <div>
                                                        <div>
                                                          <div>On
                                                          06/11/14
                                                          14:58, Walter
                                                          van Lille
                                                          wrote:<br>
                                                          </div>
                                                          <blockquote
                                                          type="cite">
                                                          <div dir="ltr">Hi,

                                                          <div><br>
                                                          </div>
                                                          <div>I need
                                                          some
                                                          assistance
                                                          please.</div>
                                                          <div>I've
                                                          taken over an
                                                          IPA server to
                                                          manage a few
                                                          months ago,
                                                          and it was
                                                          working fine
                                                          until recently
                                                          when it
                                                          started acting
                                                          up seemingly
                                                          off its own
                                                          accord.</div>
                                                          <div>When I do
                                                          an ipactl
                                                          status it
                                                          basically
                                                          gives an
                                                          output as
                                                          shown below:</div>
                                                          <div><br>
                                                          </div>
                                                          <div><br>
                                                          </div>
                                                          <div><b>Directory

                                                          Service:
                                                          RUNNING<br>
                                                          </b></div>
                                                          <div><b><br>
                                                          </b></div>
                                                          <div>
                                                          <div><b>Loooooooooooooooooooooooooooooooooooooooooooooooooong







                                                          pause... (To
                                                          the tune of 7
                                                          minutes
                                                          sometimes)</b></div>
                                                          </div>
                                                          <div><b><br>
                                                          </b></div>
                                                          <div>
                                                          <div><b>KDC
                                                          Service:
                                                          RUNNING</b></div>
                                                          <div><b>KPASSWD

                                                          Service:
                                                          RUNNING</b></div>
                                                          <div><b>DNS
                                                          Service:
                                                          RUNNING</b></div>
                                                          <div><b>MEMCACHE

                                                          Service:
                                                          RUNNING</b></div>
                                                          <div><b>HTTP
                                                          Service:
                                                          RUNNING</b></div>
                                                          <div><b>CA
                                                          Service:
                                                          RUNNING</b></div>
                                                          <div><b>ADTRUST

                                                          Service:
                                                          RUNNING</b></div>
                                                          <div><b>EXTID
                                                          Service:
                                                          RUNNING</b></div>
                                                          </div>
                                                          <div><br>
                                                          </div>
                                                          <div>Running
                                                          top showed
                                                          that ns-slapd
                                                          was munching
                                                          almost all my
                                                          resources, but
                                                          I got that
                                                          fixed by
                                                          upping the
                                                          cache.
                                                          Unfortunately
                                                          this did not
                                                          correct the
                                                          issue and it
                                                          still reacts
                                                          in the same
                                                          fashion,
                                                          although the
                                                          resources have
                                                          been freed up
                                                          now.</div>
                                                          <div>I've
                                                          noticed that
                                                          when I run dig
                                                          on either the
                                                          local server
                                                          or a remote
                                                          machine that
                                                          the query
                                                          basically just
                                                          times out as
                                                          shown here:</div>
                                                          <div><br>
                                                          </div>
                                                          <div>
                                                          <div> <b>dig
                                                          freeipa.myexample.sample</b></div>
                                                          <div><b><br>
                                                          </b></div>
                                                          <div><b>;
                                                          <<>>
                                                          DiG
                                                          9.8.2rc1-RedHat-9.8.2-0.23.rc1.el6_5.1
                                                          <<>>

freeipa.myexample.sample</b></div>
                                                          <div><b>;;
                                                          global
                                                          options: +cmd</b></div>
                                                          <div><b>;;
                                                          connection
                                                          timed out; no
                                                          servers could
                                                          be reached</b></div>
                                                          </div>
                                                          <div><br>
                                                          </div>
                                                          <div>When the
                                                          KDC service
                                                          fails to
                                                          start, then
                                                          name lookups
                                                          seem OK, but
                                                          authentication
                                                          fails.
                                                          otherwise it's
                                                          dead in the
                                                          water.</div>
                                                          <div><br>
                                                          </div>
                                                          <div>This also
                                                          happens:</div>
                                                          <div>
                                                          <div><br>
                                                          </div>
                                                          <div><b>sudo
                                                          ipactl status</b></div>
                                                          <div><b>Directory

                                                          Service:
                                                          RUNNING</b></div>
                                                          <div><b>Unknown
                                                          error when
                                                          retrieving
                                                          list of
                                                          services from
                                                          LDAP:</b></div>
                                                          </div>
                                                          <div><b><br>
                                                          </b></div>
                                                          <div>My
                                                          software setup
                                                          is as follows:</div>
                                                          <div><br>
                                                          </div>
                                                          <div><b>CentOS
                                                          release 6.5
                                                          (Final)<br>
                                                          </b></div>
                                                          <div><b>389-ds-base.x86_64


                                                           
                                                          1.2.11.15-34.el6_5<br>
                                                          </b></div>
                                                          <div><b>bind.x86_64
                                                                 
 32:9.8.2-0.23.rc1.el6_5.1<br>
                                                          </b></div>
                                                          <div>
                                                          <div><b>bind-dyndb-ldap.x86_64</b></div>
                                                          <div><b>bind-libs.x86_64


                                                             
32:9.8.2-0.23.rc1.el6_5.1</b></div>
                                                          <div><b>bind-utils.x86_64



                                                           
 32:9.8.2-0.23.rc1.el6_5.1</b></div>
                                                          <div><b>rpcbind.x86_64

                                                               
                                                          0.2.0-11.el6  
                                                             
                                                          @anaconda-CentOS-201311291202.x86_64/6.5</b></div>
                                                          <div><b>samba4-winbind.x86_64</b></div>
                                                          </div>
                                                          <div><b>krb5-server.x86_64


                                                           
                                                          1.10.3-15.el6_5.1<br>
                                                          </b></div>
                                                          <div><b><br>
                                                          </b></div>
                                                          <div><b>Linux
                                                          2.6.32-431.29.2.el6.x86_64

                                                          #1 SMP Tue Sep
                                                          9 21:36:05 UTC
                                                          2014 x86_64
                                                          x86_64 x86_64
                                                          GNU/Linux<br>
                                                          </b></div>
                                                          <div><br>
                                                          </div>
                                                          <div>It's not
                                                          a permanent
                                                          situation as
                                                          it sometimes
                                                          runs 100% for
                                                          a while, but
                                                          80% of the
                                                          time it is
                                                          unusable. If
                                                          anybody can
                                                          assist me,
                                                          please be so
                                                          kind.</div>
                                                          <div><br>
                                                          </div>
                                                          <div>Regards,</div>
                                                          <div><br>
                                                          </div>
                                                          <div>Walter</div>
                                                          <div><br>
                                                          </div>
                                                          </div>
                                                          </blockquote>
                                                        </div>
                                                      </div>
                                                      Hello please which
                                                      version of
                                                      bind-dyndb-ldap do
                                                      you use?<br>
                                                      I had similar
                                                      issue with
                                                      bind-dyndb-ldap,
                                                      but it was
                                                      development
                                                      version, I'm not
                                                      sure if this is
                                                      your case.<br>
                                                      When named was
                                                      failing, dirserv
                                                      was really slow.<br>
                                                      <br>
                                                      Can you send
                                                      journalctl -b -u
                                                      named log when dig
                                                      doesn't work??<span><font
color="#888888"><br>
                                                          <br>
                                                          <pre cols="72">-- 
Martin Basti</pre>
                                                        </font></span></div>
                                                  </blockquote>
                                                </div>
                                                <br>
                                              </div>
                                            </div>
                                          </blockquote>
                                          <br>
                                          <br>
                                        </div>
                                      </div>
                                      <span><font color="#888888">
                                          <pre cols="72">-- 
Martin Basti</pre>
                                        </font></span></div>
                                  </blockquote>
                                </div>
                                <br>
                              </div>
                            </div>
                          </blockquote>
                          <br>
                          <br>
                          <pre cols="72">-- 
Martin Basti</pre>
                        </blockquote>
                        <br>
                        <br>
                        <fieldset></fieldset>
                        <br>
                      </blockquote>
                      <br>
                      <br>
                      <fieldset></fieldset>
                      <br>
                    </blockquote>
                    <br>
                    <br>
                    <pre cols="72">-- 
Martin Basti</pre>
                    <br>
                    <fieldset></fieldset>
                    <br>
                  </blockquote>
                  <br>
                </div>
              </div>
            </div>
            <br>
            --<br>
            Manage your subscription for the Freeipa-users mailing list:<br>
            <a moz-do-not-send="true"
              href="https://www.redhat.com/mailman/listinfo/freeipa-users"
              target="_blank">https://www.redhat.com/mailman/listinfo/freeipa-users</a><br>
            Go To <a moz-do-not-send="true" href="http://freeipa.org"
              target="_blank">http://freeipa.org</a> for more info on
            the project<br>
          </blockquote>
        </div>
        <br>
      </div>
    </blockquote>
    <br>
  </body>
</html>