<html>
  <head>
    <meta content="text/html; charset=windows-1252"
      http-equiv="Content-Type">
  </head>
  <body bgcolor="#FFFFFF" text="#000000">
    <div class="moz-cite-prefix">On 02/19/2015 12:11 PM, Jim Richard
      wrote:<br>
    </div>
    <blockquote
      cite="mid:09887AA8-5F57-495E-9AFC-2B34969DBB66@placeiq.com"
      type="cite">
      <meta http-equiv="Content-Type" content="text/html;
        charset=windows-1252">
      Hi Rich, here’s what all 4 of my IPA servers look like right now.
      You can see that SSO-107’s CPU usage is much higher than the other
      3 and it spikes to over 100% often. And what I see over time is
      that the higher and higher cpu usage will happen between two of my
      four servers, one will drop off and the other increases and each
      time this cycle happens, the cpu usage on the one server that is
      spiking will get a little bit higher.
      <div class=""><br class="">
      </div>
      <div class="">The two servers that show this behavior are SSO-107
        and SSO-109.</div>
    </blockquote>
    <br>
    SSO-107 is almost entirely idle except for 1 thread doing
    replication updates, and the poll thread.  There are 255 descriptors
    it is polling on.<br>
    <br>
    SSO-109 is entirely idle except for the poll thread.  There are 230
    descriptors it is polling on.<br>
    <br>
    You might try using top with the ns-slapd process and the  'H' -
    Threads mode flag.  It would be very interesting to see the cpu
    usage breakdown by thread.<br>
    <br>
    If it is indeed the poll thread that is consuming all of the cpu,
    there's not much that can be done.  CPU usage in the poll thread is
    a function of the number of connections, but since there is not much
    difference between 230 and 255, I would not expect a large CPU usage
    difference between 107 and 109 based solely on number of
    connections.<br>
    <br>
    Are you seeing timeouts or application failures or poor performance
    that seems to be due to high CPU usage?  If so, and these are
    virtual machines, you might consider adding more virtual CPUs to
    give the server more processing power for the worker threads to
    compensate for monopolization by the poll thread.<br>
    <br>
    <blockquote
      cite="mid:09887AA8-5F57-495E-9AFC-2B34969DBB66@placeiq.com"
      type="cite">
      <div class=""><br class="">
      </div>
      <div class="">I’ve attached some more detailed stack trace as
        well.</div>
      <div class=""><br class="">
      </div>
      <br>
      <fieldset class="mimeAttachmentHeader"></fieldset>
      <br>
      <meta http-equiv="Content-Type" content="text/html;
        charset=windows-1252">
      <br>
      <fieldset class="mimeAttachmentHeader"></fieldset>
      <br>
      <meta http-equiv="Content-Type" content="text/html;
        charset=windows-1252">
      <div class=""><br class="">
      </div>
      <div class=""><br class="">
      </div>
      <div class=""><br class="">
      </div>
      <div class=""><br class="">
      </div>
      <div class="">Here’s what my replication agreements look like:</div>
      <div class=""><br class="">
      </div>
      <div class="">
        <div class="">[root@sso-107 (NY) ~]$ ipa-replica-manage list</div>
        <div class=""><a moz-do-not-send="true"
            href="http://sso-108.nym1.placeiq.net" class="">sso-108.nym1.placeiq.net</a>:
          master</div>
        <div class=""><a moz-do-not-send="true"
            href="http://sso-110.nym1.placeiq.net" class="">sso-110.nym1.placeiq.net</a>:
          master</div>
        <div class=""><a moz-do-not-send="true"
            href="http://sso-107.nym1.placeiq.net" class="">sso-107.nym1.placeiq.net</a>:
          master</div>
        <div class=""><a moz-do-not-send="true"
            href="http://sso-109.nym1.placeiq.net" class="">sso-109.nym1.placeiq.net</a>:
          master</div>
        <div class=""><br class="">
        </div>
        <div class="">[root@sso-107 (NY) ~]$ ipa-replica-manage list <a
            moz-do-not-send="true"
            href="http://sso-107.nym1.placeiq.net" class="">sso-107.nym1.placeiq.net</a></div>
        <div class=""><a moz-do-not-send="true"
            href="http://sso-108.nym1.placeiq.net" class="">sso-108.nym1.placeiq.net</a>:
          replica</div>
        <div class=""><a moz-do-not-send="true"
            href="http://sso-110.nym1.placeiq.net" class="">sso-110.nym1.placeiq.net</a>:
          replica</div>
        <div class=""><br class="">
        </div>
        <div class="">[root@sso-107 (NY) ~]$ ipa-replica-manage list <a
            moz-do-not-send="true"
            href="http://sso-108.nym1.placeiq.net" class="">sso-108.nym1.placeiq.net</a></div>
        <div class=""><a moz-do-not-send="true"
            href="http://sso-107.nym1.placeiq.net" class="">sso-107.nym1.placeiq.net</a>:
          replica</div>
        <div class=""><a moz-do-not-send="true"
            href="http://sso-109.nym1.placeiq.net" class="">sso-109.nym1.placeiq.net</a>:
          replica</div>
        <div class=""><br class="">
        </div>
        <div class="">[root@sso-107 (NY) ~]$ ipa-replica-manage list <a
            moz-do-not-send="true"
            href="http://sso-109.nym1.placeiq.net" class="">sso-109.nym1.placeiq.net</a></div>
        <div class=""><a moz-do-not-send="true"
            href="http://sso-108.nym1.placeiq.net" class="">sso-108.nym1.placeiq.net</a>:
          replica</div>
        <div class=""><a moz-do-not-send="true"
            href="http://sso-110.nym1.placeiq.net" class="">sso-110.nym1.placeiq.net</a>:
          replica</div>
        <div class=""><br class="">
        </div>
        <div class="">[root@sso-107 (NY) ~]$ ipa-replica-manage list <a
            moz-do-not-send="true"
            href="http://sso-110.nym1.placeiq.net" class="">sso-110.nym1.placeiq.net</a></div>
        <div class=""><a moz-do-not-send="true"
            href="http://sso-107.nym1.placeiq.net" class="">sso-107.nym1.placeiq.net</a>:
          replica</div>
        <div class=""><a moz-do-not-send="true"
            href="http://sso-109.nym1.placeiq.net" class="">sso-109.nym1.placeiq.net</a>:
          replica</div>
      </div>
      <div class=""><br class="">
      </div>
      <div class=""><br class="">
      </div>
      <div class=""><br class="">
        <div class="">
          <div class=""><br class="">
          </div>
          <div class=""><br class="">
          </div>
          <div class="">
            <div class="">SSO-107</div>
            <div class=""><br class="">
            </div>
            <div class="">top - 15:58:08 up 2 days, 10:00,  1 user,
               load average: 0.00, 0.03, 0.06</div>
            <div class="">Tasks:   1 total,   0 running,   1 sleeping,  
              0 stopped,   0 zombie</div>
            <div class="">Cpu(s): 12.2%us,  1.1%sy,  0.0%ni, 86.7%id,
               0.1%wa,  0.0%hi,  0.0%si,  0.0%st</div>
            <div class="">Mem:   2952788k total,  2160216k used,  
              792572k free,   182584k buffers</div>
            <div class="">Swap:  4094972k total,        0k used,
               4094972k free,   678292k cached</div>
            <div class=""><br class="">
            </div>
            <div class="">  PID USER      PR  NI  VIRT  RES  SHR S %CPU
              %MEM    TIME+  COMMAND</div>
            <div class="">11615 dirsrv    20   0 2063m 843m  19m S 25.5
              29.3 403:53.56 ns-slapd</div>
            <div class=""><br class="">
            </div>
            <div class="">[root@sso-107 (NY)
              /var/log/dirsrv/slapd-PLACEIQ-NET]$ ls -al /proc/`cat
              /var/run/dirsrv/slapd-PLACEIQ-NET.pid`/fd|grep socket|wc
              -l</div>
            <div class="">245</div>
            <div class=""><br class="">
            </div>
            <div class=""><br class="">
            </div>
            <div class=""><br class="">
            </div>
            <div class=""><br class="">
            </div>
            <div class="">SSO-108</div>
            <div class=""><br class="">
            </div>
            <div class="">top - 15:57:26 up 3 days, 17:25,  1 user,
               load average: 0.03, 0.03, 0.00</div>
            <div class="">Tasks:   1 total,   0 running,   1 sleeping,  
              0 stopped,   0 zombie</div>
            <div class="">Cpu(s):  0.3%us,  0.2%sy,  0.0%ni, 99.4%id,
               0.1%wa,  0.0%hi,  0.0%si,  0.0%st</div>
            <div class="">Mem:   2952788k total,  2200792k used,  
              751996k free,   182084k buffers</div>
            <div class="">Swap:  4094972k total,        0k used,
               4094972k free,   713848k cached</div>
            <div class=""><br class="">
            </div>
            <div class="">  PID USER      PR  NI  VIRT  RES  SHR S %CPU
              %MEM    TIME+  COMMAND</div>
            <div class="">24399 dirsrv    20   0 2055m 819m  19m S  0.8
              28.4  54:48.53 ns-slapd</div>
            <div class=""><br class="">
            </div>
            <div class="">[root@sso-108 (NY)
              /var/log/dirsrv/slapd-PLACEIQ-NET]$ ls -al /proc/`cat
              /var/run/dirsrv/slapd-PLACEIQ-NET.pid`/fd|grep socket|wc
              -l</div>
            <div class="">232</div>
            <div class=""><br class="">
            </div>
            <div class=""><br class="">
            </div>
            <div class=""><br class="">
            </div>
            <div class=""><br class="">
            </div>
            <div class="">SSO-109</div>
            <div class=""><br class="">
            </div>
            <div class="">top - 16:00:05 up 4 days,  9:10,  1 user,
               load average: 0.06, 0.32, 0.35</div>
            <div class="">Tasks:   1 total,   0 running,   1 sleeping,  
              0 stopped,   0 zombie</div>
            <div class="">Cpu(s):  0.7%us,  0.3%sy,  0.0%ni, 98.9%id,
               0.2%wa,  0.0%hi,  0.0%si,  0.0%st</div>
            <div class="">Mem:   2952788k total,  2422572k used,  
              530216k free,   235472k buffers</div>
            <div class="">Swap:  4094972k total,        0k used,
               4094972k free,   906080k cached</div>
            <div class=""><br class="">
            </div>
            <div class="">  PID USER      PR  NI  VIRT  RES  SHR S %CPU
              %MEM    TIME+  COMMAND</div>
            <div class="">22522 dirsrv    20   0 2065m 772m  19m S  1.2
              26.8 308:13.07 ns-slapd</div>
            <div class=""><br class="">
            </div>
            <div class="">[root@sso-109 (NY) ~]$ ls -al /proc/`cat
              /var/run/dirsrv/slapd-PLACEIQ-NET.pid`/fd|grep socket|wc
              -l</div>
            <div class="">219</div>
            <div class=""><br class="">
            </div>
            <div class=""><br class="">
            </div>
            <div class=""><br class="">
            </div>
            <div class=""><br class="">
            </div>
            <div class=""><br class="">
            </div>
            <div class="">SSO-110</div>
            <div class=""><br class="">
            </div>
            <div class="">top - 16:07:54 up 14 days, 18:03,  1 user,
               load average: 0.00, 0.01, 0.00</div>
            <div class="">Tasks:   1 total,   0 running,   1 sleeping,  
              0 stopped,   0 zombie</div>
            <div class="">Cpu(s):  2.0%us,  1.0%sy,  0.0%ni, 96.7%id,
               0.3%wa,  0.0%hi,  0.0%si,  0.0%st</div>
            <div class="">Mem:   2952788k total,  2304556k used,  
              648232k free,   155216k buffers</div>
            <div class="">Swap:  4094972k total,       64k used,
               4094908k free,   748972k cached</div>
            <div class=""><br class="">
            </div>
            <div class="">  PID USER      PR  NI  VIRT  RES  SHR S %CPU
              %MEM    TIME+  COMMAND</div>
            <div class=""> 2401 dirsrv    20   0 2074m 839m  18m S  4.8
              29.1  48:25.58 ns-slapd</div>
            <div class=""> </div>
            <div class="">[root@sso-110 (NY)
              /var/log/dirsrv/slapd-PLACEIQ-NET]$ ls -al /proc/`cat
              /var/run/dirsrv/slapd-PLACEIQ-NET.pid`/fd|grep socket|wc
              -l</div>
            <div class="">257</div>
          </div>
          <div class=""><br class="">
          </div>
          <div class=""><br class="">
          </div>
          <div class=""><br class="">
          </div>
          <div class=""><br class="">
          </div>
          <div class=""><br class="">
            <div apple-content-edited="true" class="">
              <div style="color: rgb(0, 0, 0); font-family: Verdana;
                font-size: 11px; font-style: normal; font-variant:
                normal; font-weight: normal; letter-spacing: normal;
                line-height: normal; orphans: auto; text-align: start;
                text-indent: 0px; text-transform: none; white-space:
                normal; widows: auto; word-spacing: 0px;
                -webkit-text-stroke-width: 0px;" class=""><br
                  class="Apple-interchange-newline">
                <br class="">
              </div>
              <div style="color: rgb(0, 0, 0); font-family: Verdana;
                font-size: 11px; font-style: normal; font-variant:
                normal; font-weight: normal; letter-spacing: normal;
                line-height: normal; orphans: auto; text-align: start;
                text-indent: 0px; text-transform: none; white-space:
                normal; widows: auto; word-spacing: 0px;
                -webkit-text-stroke-width: 0px;" class="">
                <table style="color: rgb(34, 34, 34); margin: 0px;
                  font-size: 13.63636302948px; font-family: Times;
                  background-color: rgb(255, 255, 255);" class=""
                  cellpadding="0" cellspacing="0">
                  <tbody class="">
                    <tr class="" height="3">
                      <td style="font-family: arial, sans-serif; margin:
                        0px; vertical-align: top;" class=""><br>
                      </td>
                    </tr>
                  </tbody><tbody class="">
                    <tr class="" height="3">
                      <td style="font-family: arial, sans-serif; margin:
                        0px; vertical-align: top; background-color:
                        rgb(119, 117, 171);" class="" width="5"><br>
                      </td>
                      <td style="font-family: arial, sans-serif; margin:
                        0px; vertical-align: top; background-color:
                        rgb(68, 54, 124);" class="" width="5"><br>
                      </td>
                      <td style="font-family: arial, sans-serif; margin:
                        0px; vertical-align: top; background-color:
                        rgb(99, 80, 178);" class="" width="5"><br>
                      </td>
                      <td style="font-family: arial, sans-serif; margin:
                        0px; vertical-align: top; background-color:
                        rgb(119, 117, 171);" class="" width="5"><br>
                      </td>
                      <td style="font-family: arial, sans-serif; margin:
                        0px; vertical-align: top; background-color:
                        rgb(148, 145, 145);" class="" width="5"><br>
                      </td>
                      <td style="font-family: arial, sans-serif; margin:
                        0px; vertical-align: top; background-color:
                        rgb(203, 203, 203);" class="" width="5"><br>
                      </td>
                      <td style="font-family: arial, sans-serif; margin:
                        0px; vertical-align: top; background-color:
                        rgb(148, 145, 145);" class="" width="5"><br>
                      </td>
                      <td style="font-family: arial, sans-serif; margin:
                        0px; vertical-align: top; background-color:
                        rgb(119, 117, 171);" class="" width="5"><br>
                      </td>
                      <td style="font-family: arial, sans-serif; margin:
                        0px; vertical-align: top; background-color:
                        rgb(99, 80, 178);" class="" width="5"><br>
                      </td>
                      <td style="font-family: arial, sans-serif; margin:
                        0px; vertical-align: top; background-color:
                        rgb(68, 54, 124);" class="" width="5"><br>
                      </td>
                      <td style="font-family: arial, sans-serif; margin:
                        0px; vertical-align: top; background-color:
                        rgb(99, 80, 178);" class="" width="5"><br>
                      </td>
                      <td style="font-family: arial, sans-serif; margin:
                        0px; vertical-align: top; background-color:
                        rgb(119, 117, 171);" class="" width="5"><br>
                      </td>
                      <td style="font-family: arial, sans-serif; margin:
                        0px; vertical-align: top; background-color:
                        rgb(148, 145, 145);" class="" width="5"><br>
                      </td>
                      <td style="font-family: arial, sans-serif; margin:
                        0px; vertical-align: top; background-color:
                        rgb(203, 203, 203);" class="" width="5"><br>
                      </td>
                      <td style="font-family: arial, sans-serif; margin:
                        0px; vertical-align: top; background-color:
                        rgb(148, 145, 145);" class="" width="5"><br>
                      </td>
                      <td style="font-family: arial, sans-serif; margin:
                        0px; vertical-align: top; background-color:
                        rgb(119, 117, 171);" class="" width="5"><br>
                      </td>
                      <td style="font-family: arial, sans-serif; margin:
                        0px; vertical-align: top; background-color:
                        rgb(99, 80, 178);" class="" width="5"><br>
                      </td>
                      <td style="font-family: arial, sans-serif; margin:
                        0px; vertical-align: top; background-color:
                        rgb(119, 117, 171);" class="" width="5"><br>
                      </td>
                      <td style="font-family: arial, sans-serif; margin:
                        0px; vertical-align: top; background-color:
                        rgb(119, 117, 171);" class="" width="5"><br>
                      </td>
                      <td style="font-family: arial, sans-serif; margin:
                        0px; vertical-align: top; background-color:
                        rgb(119, 117, 171);" class="" width="5"><br>
                      </td>
                      <td style="font-family: arial, sans-serif; margin:
                        0px; vertical-align: top; background-color:
                        rgb(119, 117, 171);" class="" width="5"><br>
                      </td>
                      <td style="font-family: arial, sans-serif; margin:
                        0px; vertical-align: top; background-color:
                        rgb(119, 117, 171);" class="" width="5"><br>
                      </td>
                      <td style="font-family: arial, sans-serif; margin:
                        0px; vertical-align: top; background-color:
                        rgb(119, 117, 171);" class="" width="5"><br>
                      </td>
                      <td style="font-family: arial, sans-serif; margin:
                        0px; vertical-align: top; background-color:
                        rgb(119, 117, 171);" class="" width="5"><br>
                      </td>
                      <td style="font-family: arial, sans-serif; margin:
                        0px; vertical-align: top; background-color:
                        rgb(119, 117, 171);" class="" width="5"><br>
                      </td>
                      <td style="font-family: arial, sans-serif; margin:
                        0px; vertical-align: top; background-color:
                        rgb(119, 117, 171);" class="" width="5"><br>
                      </td>
                      <td style="font-family: arial, sans-serif; margin:
                        0px; vertical-align: top; background-color:
                        rgb(119, 117, 171);" class="" width="5"><br>
                      </td>
                      <td style="font-family: arial, sans-serif; margin:
                        0px; vertical-align: top; background-color:
                        rgb(119, 117, 171);" class="" width="5"><br>
                      </td>
                      <td style="font-family: arial, sans-serif; margin:
                        0px; vertical-align: top; background-color:
                        rgb(119, 117, 171);" class="" width="5"><br>
                      </td>
                      <td style="font-family: arial, sans-serif; margin:
                        0px; vertical-align: top; background-color:
                        rgb(119, 117, 171);" class="" width="5"><br>
                      </td>
                      <td style="font-family: arial, sans-serif; margin:
                        0px; vertical-align: top; background-color:
                        rgb(119, 117, 171);" class="" width="5"><br>
                      </td>
                      <td style="font-family: arial, sans-serif; margin:
                        0px; vertical-align: top; background-color:
                        rgb(119, 117, 171);" class="" width="5"><br>
                      </td>
                      <td style="font-family: arial, sans-serif; margin:
                        0px; vertical-align: top; background-color:
                        rgb(119, 117, 171);" class="" width="5"><br>
                      </td>
                      <td style="font-family: arial, sans-serif; margin:
                        0px; vertical-align: top; background-color:
                        rgb(119, 117, 171);" class="" width="5"><br>
                      </td>
                      <td style="font-family: arial, sans-serif; margin:
                        0px; vertical-align: top; background-color:
                        rgb(119, 117, 171);" class="" width="5"><br>
                      </td>
                      <td style="font-family: arial, sans-serif; margin:
                        0px; vertical-align: top; background-color:
                        rgb(119, 117, 171);" class=""><br>
                      </td>
                    </tr>
                  </tbody><tbody class="">
                    <tr class="" height="10">
                      <td colspan="36" style="font-family: Georgia,
                        serif; margin: 0px; vertical-align: top;
                        padding-top: 5px; font-style: italic; font-size:
                        10px; letter-spacing: 0.1em; line-height: 1.5em;
                        color: rgb(117, 107, 172);" class=""><span
                          style="color: rgb(70, 70, 70);" class="">Jim
                          Richard  |  </span><a moz-do-not-send="true"
href="http://www.google.com/url?q=http%3A%2F%2Fwww.placeiq.com%2F&sa=D&sntz=1&usg=AFrqEzcYjZpDPyqW7feNK9EgLq-c9JlHiw"
                          target="_blank" style="color: rgb(117, 107,
                          172); text-decoration: none;" class="">PlaceIQ</a>  |
                         Systems Administrator  |  jrichard<a
                          moz-do-not-send="true"
                          href="mailto:name@placeiq.com" target="_blank"
                          style="color: rgb(117, 107, 172);
                          text-decoration: none;" class="">@placeiq.com</a>  |  <a
                          moz-do-not-send="true" style="color: rgb(17,
                          85, 204);" class="">+1 (646) 338-8905</a></td>
                    </tr>
                  </tbody>
                </table>
                <div class=""><br class="">
                </div>
              </div>
              <br class="Apple-interchange-newline">
            </div>
            <br class="">
            <div>
              <blockquote type="cite" class="">
                <div class="">On Feb 19, 2015, at 9:33 AM, Rich
                  Megginson <<a moz-do-not-send="true"
                    href="mailto:rmeggins@redhat.com" class="">rmeggins@redhat.com</a>>
                  wrote:</div>
                <br class="Apple-interchange-newline">
                <div class="">
                  <meta content="text/html; charset=windows-1252"
                    http-equiv="Content-Type" class="">
                  <div bgcolor="#FFFFFF" text="#000000" class="">
                    <div class="moz-cite-prefix">On 02/18/2015 11:05 PM,
                      Jatin Nansi wrote:<br class="">
                    </div>
                    <blockquote cite="mid:54E57D28.103@redhat.com"
                      type="cite" class="">
                      <meta content="text/html; charset=windows-1252"
                        http-equiv="Content-Type" class="">
                      Check the ns-slapd access and error logs of the DS
                      instance hosting the IPA instance. The strace
                      output indicates that ns-slapd spent most of its
                      time waiting for network activity to happen
                      (poll), which is normal for ns-slapd.<br class="">
                    </blockquote>
                    <br class="">
                    The number of open connections correlates to the CPU
                    usage.  Do this:<br class="">
                    <br class="">
                    # ls -al /proc/`cat
                    /var/run/dirsrv/slapd-MY-DOMAIN.pid`/fd|grep
                    socket|wc -l<br class="">
                    <br class="">
                    How many socket connections do you have?<br class="">
                    <br class="">
                    Also, it will be very useful to get some stack
                    traces of the running server to see what the various
                    threads are doing.  See <a moz-do-not-send="true"
                      class="moz-txt-link-freetext"
                      href="http://www.port389.org/docs/389ds/FAQ/faq.html#debugging-hangs">http://www.port389.org/docs/389ds/FAQ/faq.html#debugging-hangs</a><br
                      class="">
                    <br class="">
                    <blockquote cite="mid:54E57D28.103@redhat.com"
                      type="cite" class=""> <br class="">
                      Jatin<br class="">
                      <div class="moz-cite-prefix">On 19/02/15 15:52,
                        Jim Richard wrote:<br class="">
                      </div>
                      <blockquote
                        cite="mid:30BE1A3D-7170-485D-9A91-C868682450D9@placeiq.com"
                        type="cite" class="">
                        <meta http-equiv="Content-Type"
                          content="text/html; charset=windows-1252"
                          class="">
                        I’ve got 4 Redhat IDM masters in a multi-master
                        config. 3.0.0-42.el6.centos is the IPA version,
                        389-ds-base version 1.2.11.15-48.el6_6, Centos
                        6.6
                        <div class=""><br class="">
                        </div>
                        <div class="">Monitoring established connections
                          on port 389 and dsInOps over time shows a
                          consistent/even level of activity however 2 of
                          the 4 IPA servers show ever increasing CPI
                          usage by ns-slapd. One ns-slapd process will
                          start to show increased CPU for a time, then
                          drop off as another then increases. This cycle
                          continues with each switch seeing more and
                          more total SPU usage by ns-slapd.</div>
                        <div class=""><br class="">
                        </div>
                        <div class="">strace timing for the offending
                          ns-slapd looks like the following:</div>
                        <div class=""><br class="">
                        </div>
                        <div class=""><br class="">
                        </div>
                      </blockquote>
                    </blockquote>
                  </div>
                </div>
              </blockquote>
              <br class="">
              <blockquote type="cite" class="">
                <div class="">
                  <div bgcolor="#FFFFFF" text="#000000" class="">
                    <blockquote cite="mid:54E57D28.103@redhat.com"
                      type="cite" class="">
                      <blockquote
                        cite="mid:30BE1A3D-7170-485D-9A91-C868682450D9@placeiq.com"
                        type="cite" class="">
                        <div class=""> </div>
                        <div class="">
                          <div class="">% time     seconds  usecs/call  
                              calls    errors syscall</div>
                          <div class="">------ ----------- -----------
                            --------- --------- ----------------</div>
                          <div class=""> 96.12    9.342272        1133  
                               8243           poll</div>
                          <div class="">  3.86    0.375457          53  
                               7066        41 futex</div>
                          <div class="">  0.01    0.000668           0  
                               8244      8244 getpeername</div>
                          <div class="">  0.00    0.000374           0  
                                929           close</div>
                          <div class="">  0.00    0.000368           0  
                               3201           read</div>
                          <div class="">  0.00    0.000151           0  
                                882           setsockopt</div>
                          <div class="">  0.00    0.000095           2  
                                 42           access</div>
                          <div class="">  0.00    0.000033           0  
                               1365           fcntl</div>
                          <div class="">  0.00    0.000000           0  
                                 42           open</div>
                          <div class="">  0.00    0.000000           0  
                                 39           stat</div>
                          <div class="">  0.00    0.000000           0  
                                 42           fstat</div>
                          <div class="">  0.00    0.000000           0  
                                  1           madvise</div>
                          <div class="">  0.00    0.000000           0  
                                441           accept</div>
                          <div class="">  0.00    0.000000           0  
                                441           getsockname</div>
                          <div class="">  0.00    0.000000           0  
                                  1           restart_syscall</div>
                          <div class="">------ ----------- -----------
                            --------- --------- ----------------</div>
                          <div class="">100.00    9.719418              
                              30979      8285 total</div>
                        </div>
                        <div class=""><br class="">
                        </div>
                        <div class=""><br class="">
                        </div>
                        <div class="">I have carefully reviewed
                          cn=config settings on all four master servers
                          to confirm that they match.</div>
                        <div class=""><br class="">
                        </div>
                        <div class="">Based on this strace output can
                          you perhaps point me in the right direction,
                          give me a clue on what I should be looking at.</div>
                        <div class=""><br class="">
                        </div>
                        <div class="">Here’s a screen shot of my Zabbix
                          reporting to help describe the problem. Note
                          the graph in the bottom right corner.</div>
                        <div class=""><br class="">
                        </div>
                        <div class="">The problem is most certainly
                          related to replication but I just don’t know
                          what specifically to look at.</div>
                        <div class=""><br class="">
                        </div>
                        <div class=""><span
                            id="cid:part1.04040701.00090201@redhat.com"><Mail
                            Attachment.png></span></div>
                        <div class=""><br class="">
                        </div>
                        <div class=""><br class="">
                        </div>
                        <div class=""><br class="">
                        </div>
                        <div class="">Thanks in advance for any clues
                          you can provide.<br class="">
                          <div class="">
                            <div style="font-family: Verdana; font-size:
                              11px; font-style: normal; font-variant:
                              normal; font-weight: normal;
                              letter-spacing: normal; line-height:
                              normal; orphans: auto; text-align: start;
                              text-indent: 0px; text-transform: none;
                              white-space: normal; widows: auto;
                              word-spacing: 0px;
                              -webkit-text-stroke-width: 0px;" class=""><br
                                class="Apple-interchange-newline">
                              <br class="">
                            </div>
                            <div style="font-family: Verdana; font-size:
                              11px; font-style: normal; font-variant:
                              normal; font-weight: normal;
                              letter-spacing: normal; line-height:
                              normal; orphans: auto; text-align: start;
                              text-indent: 0px; text-transform: none;
                              white-space: normal; widows: auto;
                              word-spacing: 0px;
                              -webkit-text-stroke-width: 0px;" class="">
                              <table style="color: rgb(34, 34, 34);
                                margin: 0px; font-size:
                                13.63636302948px; font-family: Times;
                                background-color: rgb(255, 255, 255);"
                                class="" cellpadding="0" cellspacing="0">
                                <tbody class="">
                                  <tr class="" height="3">
                                    <td style="font-family: arial,
                                      sans-serif; margin: 0px;
                                      vertical-align: top;" class=""><br
                                        class="">
                                    </td>
                                  </tr>
                                </tbody><tbody class="">
                                  <tr class="" height="3">
                                    <td style="font-family: arial,
                                      sans-serif; margin: 0px;
                                      vertical-align: top;
                                      background-color: rgb(119, 117,
                                      171);" class="" width="5"><br
                                        class="">
                                    </td>
                                    <td style="font-family: arial,
                                      sans-serif; margin: 0px;
                                      vertical-align: top;
                                      background-color: rgb(68, 54,
                                      124);" class="" width="5"><br
                                        class="">
                                    </td>
                                    <td style="font-family: arial,
                                      sans-serif; margin: 0px;
                                      vertical-align: top;
                                      background-color: rgb(99, 80,
                                      178);" class="" width="5"><br
                                        class="">
                                    </td>
                                    <td style="font-family: arial,
                                      sans-serif; margin: 0px;
                                      vertical-align: top;
                                      background-color: rgb(119, 117,
                                      171);" class="" width="5"><br
                                        class="">
                                    </td>
                                    <td style="font-family: arial,
                                      sans-serif; margin: 0px;
                                      vertical-align: top;
                                      background-color: rgb(148, 145,
                                      145);" class="" width="5"><br
                                        class="">
                                    </td>
                                    <td style="font-family: arial,
                                      sans-serif; margin: 0px;
                                      vertical-align: top;
                                      background-color: rgb(203, 203,
                                      203);" class="" width="5"><br
                                        class="">
                                    </td>
                                    <td style="font-family: arial,
                                      sans-serif; margin: 0px;
                                      vertical-align: top;
                                      background-color: rgb(148, 145,
                                      145);" class="" width="5"><br
                                        class="">
                                    </td>
                                    <td style="font-family: arial,
                                      sans-serif; margin: 0px;
                                      vertical-align: top;
                                      background-color: rgb(119, 117,
                                      171);" class="" width="5"><br
                                        class="">
                                    </td>
                                    <td style="font-family: arial,
                                      sans-serif; margin: 0px;
                                      vertical-align: top;
                                      background-color: rgb(99, 80,
                                      178);" class="" width="5"><br
                                        class="">
                                    </td>
                                    <td style="font-family: arial,
                                      sans-serif; margin: 0px;
                                      vertical-align: top;
                                      background-color: rgb(68, 54,
                                      124);" class="" width="5"><br
                                        class="">
                                    </td>
                                    <td style="font-family: arial,
                                      sans-serif; margin: 0px;
                                      vertical-align: top;
                                      background-color: rgb(99, 80,
                                      178);" class="" width="5"><br
                                        class="">
                                    </td>
                                    <td style="font-family: arial,
                                      sans-serif; margin: 0px;
                                      vertical-align: top;
                                      background-color: rgb(119, 117,
                                      171);" class="" width="5"><br
                                        class="">
                                    </td>
                                    <td style="font-family: arial,
                                      sans-serif; margin: 0px;
                                      vertical-align: top;
                                      background-color: rgb(148, 145,
                                      145);" class="" width="5"><br
                                        class="">
                                    </td>
                                    <td style="font-family: arial,
                                      sans-serif; margin: 0px;
                                      vertical-align: top;
                                      background-color: rgb(203, 203,
                                      203);" class="" width="5"><br
                                        class="">
                                    </td>
                                    <td style="font-family: arial,
                                      sans-serif; margin: 0px;
                                      vertical-align: top;
                                      background-color: rgb(148, 145,
                                      145);" class="" width="5"><br
                                        class="">
                                    </td>
                                    <td style="font-family: arial,
                                      sans-serif; margin: 0px;
                                      vertical-align: top;
                                      background-color: rgb(119, 117,
                                      171);" class="" width="5"><br
                                        class="">
                                    </td>
                                    <td style="font-family: arial,
                                      sans-serif; margin: 0px;
                                      vertical-align: top;
                                      background-color: rgb(99, 80,
                                      178);" class="" width="5"><br
                                        class="">
                                    </td>
                                    <td style="font-family: arial,
                                      sans-serif; margin: 0px;
                                      vertical-align: top;
                                      background-color: rgb(119, 117,
                                      171);" class="" width="5"><br
                                        class="">
                                    </td>
                                    <td style="font-family: arial,
                                      sans-serif; margin: 0px;
                                      vertical-align: top;
                                      background-color: rgb(119, 117,
                                      171);" class="" width="5"><br
                                        class="">
                                    </td>
                                    <td style="font-family: arial,
                                      sans-serif; margin: 0px;
                                      vertical-align: top;
                                      background-color: rgb(119, 117,
                                      171);" class="" width="5"><br
                                        class="">
                                    </td>
                                    <td style="font-family: arial,
                                      sans-serif; margin: 0px;
                                      vertical-align: top;
                                      background-color: rgb(119, 117,
                                      171);" class="" width="5"><br
                                        class="">
                                    </td>
                                    <td style="font-family: arial,
                                      sans-serif; margin: 0px;
                                      vertical-align: top;
                                      background-color: rgb(119, 117,
                                      171);" class="" width="5"><br
                                        class="">
                                    </td>
                                    <td style="font-family: arial,
                                      sans-serif; margin: 0px;
                                      vertical-align: top;
                                      background-color: rgb(119, 117,
                                      171);" class="" width="5"><br
                                        class="">
                                    </td>
                                    <td style="font-family: arial,
                                      sans-serif; margin: 0px;
                                      vertical-align: top;
                                      background-color: rgb(119, 117,
                                      171);" class="" width="5"><br
                                        class="">
                                    </td>
                                    <td style="font-family: arial,
                                      sans-serif; margin: 0px;
                                      vertical-align: top;
                                      background-color: rgb(119, 117,
                                      171);" class="" width="5"><br
                                        class="">
                                    </td>
                                    <td style="font-family: arial,
                                      sans-serif; margin: 0px;
                                      vertical-align: top;
                                      background-color: rgb(119, 117,
                                      171);" class="" width="5"><br
                                        class="">
                                    </td>
                                    <td style="font-family: arial,
                                      sans-serif; margin: 0px;
                                      vertical-align: top;
                                      background-color: rgb(119, 117,
                                      171);" class="" width="5"><br
                                        class="">
                                    </td>
                                    <td style="font-family: arial,
                                      sans-serif; margin: 0px;
                                      vertical-align: top;
                                      background-color: rgb(119, 117,
                                      171);" class="" width="5"><br
                                        class="">
                                    </td>
                                    <td style="font-family: arial,
                                      sans-serif; margin: 0px;
                                      vertical-align: top;
                                      background-color: rgb(119, 117,
                                      171);" class="" width="5"><br
                                        class="">
                                    </td>
                                    <td style="font-family: arial,
                                      sans-serif; margin: 0px;
                                      vertical-align: top;
                                      background-color: rgb(119, 117,
                                      171);" class="" width="5"><br
                                        class="">
                                    </td>
                                    <td style="font-family: arial,
                                      sans-serif; margin: 0px;
                                      vertical-align: top;
                                      background-color: rgb(119, 117,
                                      171);" class="" width="5"><br
                                        class="">
                                    </td>
                                    <td style="font-family: arial,
                                      sans-serif; margin: 0px;
                                      vertical-align: top;
                                      background-color: rgb(119, 117,
                                      171);" class="" width="5"><br
                                        class="">
                                    </td>
                                    <td style="font-family: arial,
                                      sans-serif; margin: 0px;
                                      vertical-align: top;
                                      background-color: rgb(119, 117,
                                      171);" class="" width="5"><br
                                        class="">
                                    </td>
                                    <td style="font-family: arial,
                                      sans-serif; margin: 0px;
                                      vertical-align: top;
                                      background-color: rgb(119, 117,
                                      171);" class="" width="5"><br
                                        class="">
                                    </td>
                                    <td style="font-family: arial,
                                      sans-serif; margin: 0px;
                                      vertical-align: top;
                                      background-color: rgb(119, 117,
                                      171);" class="" width="5"><br
                                        class="">
                                    </td>
                                    <td style="font-family: arial,
                                      sans-serif; margin: 0px;
                                      vertical-align: top;
                                      background-color: rgb(119, 117,
                                      171);" class=""><br class="">
                                    </td>
                                  </tr>
                                </tbody><tbody class="">
                                  <tr class="" height="10">
                                    <td colspan="36" style="font-family:
                                      Georgia, serif; margin: 0px;
                                      vertical-align: top; padding-top:
                                      5px; font-style: italic;
                                      font-size: 10px; letter-spacing:
                                      0.1em; line-height: 1.5em; color:
                                      rgb(117, 107, 172);" class=""><span
                                        style="color: rgb(70, 70, 70);"
                                        class="">Jim Richard  |  </span><a
                                        moz-do-not-send="true"
href="http://www.google.com/url?q=http%3A%2F%2Fwww.placeiq.com%2F&sa=D&sntz=1&usg=AFrqEzcYjZpDPyqW7feNK9EgLq-c9JlHiw"
                                        target="_blank" style="color:
                                        rgb(117, 107, 172);
                                        text-decoration: none;" class="">PlaceIQ</a>  |


                                       Systems Administrator  |
                                       jrichard<a moz-do-not-send="true"
                                        href="mailto:name@placeiq.com"
                                        target="_blank" style="color:
                                        rgb(117, 107, 172);
                                        text-decoration: none;" class="">@placeiq.com</a>  |  <a
                                        moz-do-not-send="true"
                                        style="color: rgb(17, 85, 204);"
                                        class="">+1 (646) 338-8905</a></td>
                                  </tr>
                                </tbody>
                              </table>
                              <div class=""><br class="">
                              </div>
                            </div>
                            <br class="Apple-interchange-newline">
                          </div>
                          <br class="">
                        </div>
                        <br class="">
                        <fieldset class="mimeAttachmentHeader"></fieldset>
                        <br class="">
                      </blockquote>
                      <br class="">
                      <br class="">
                      <fieldset class="mimeAttachmentHeader"></fieldset>
                      <br class="">
                    </blockquote>
                    <br class="">
                  </div>
                  -- <br class="">
                  Manage your subscription for the Freeipa-users mailing
                  list:<br class="">
                  <a moz-do-not-send="true"
                    href="https://www.redhat.com/mailman/listinfo/freeipa-users"
                    class="">https://www.redhat.com/mailman/listinfo/freeipa-users</a><br
                    class="">
                  Go To <a class="moz-txt-link-freetext" href="http://freeipa.org">http://freeipa.org</a> for more info on the project</div>
              </blockquote>
            </div>
            <br class="">
          </div>
        </div>
      </div>
    </blockquote>
    <br>
  </body>
</html>